2024-04-17T10:08:20.390 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-17T10:08:20.390 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_neesingh@teuthology --description fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/scrub} --name neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi --verbose --archive /home/teuthworker/archive/neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi/7660164 -- /home/teuthworker/archive/neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi/7660164/orig.config.yaml 2024-04-17T10:08:20.436 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-17T10:08:20.517 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi/7660164 branch: wip-neesingh-testing-240416 description: fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/scrub} email: neesingh@redhat.com first_in_suite: false job_id: '7660164' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-neesingh-testing-240416 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds cache max size: 1000 mds debug frag: true mds debug scatterstat: true mds log max segments: 1 mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - Replacing daemon mds - Scrub error on inode - Behind on trimming - Metadata damage detected - bad backtrace on inode - overall HEALTH_ - \(MDS_TRIM\) sha1: 97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec kclient: syntax: v2 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-neesingh-testing-240416 sha1: 97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec owner: scheduled_neesingh@teuthology priority: 10 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 558 sha1: 97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec sleep_before_teardown: 0 suite: fs:functional suite_branch: wip-neesingh-testing-240416 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKLCPHVKYlyBx3k3l7SXqRQ6otBkXAMeHzA5g6VnnLHyI2KFfcB0TNtDgws/3HGN8Q+sYxZiSuaJ3MlKUNWKtA0= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLM9TefWVy1Os86VRIbo5G/yblWpdj3KdzfMm0byC7iyhNA3wQp7cXTqkLg39esSvEU4st03zx7g85jcOr8ZANY= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_scrub_checks - tasks.cephfs.test_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-17_09:52:10 tube: smithi user: neesingh verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-17T10:08:20.517 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/qa; will attempt to use it 2024-04-17T10:08:20.518 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/qa/tasks 2024-04-17T10:08:20.518 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-17T10:08:20.520 INFO:teuthology.task.internal:Checking packages... 2024-04-17T10:08:20.544 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec' 2024-04-17T10:08:20.544 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-17T10:08:20.544 INFO:teuthology.packaging:ref: None 2024-04-17T10:08:20.544 INFO:teuthology.packaging:tag: None 2024-04-17T10:08:20.544 INFO:teuthology.packaging:branch: wip-neesingh-testing-240416 2024-04-17T10:08:20.544 INFO:teuthology.packaging:sha1: 97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec 2024-04-17T10:08:20.544 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-neesingh-testing-240416 2024-04-17T10:08:20.790 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:08:20.792 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-17T10:08:20.801 INFO:teuthology.task.internal:no buildpackages task found 2024-04-17T10:08:20.801 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-17T10:08:20.810 INFO:teuthology.task.internal:Saving configuration 2024-04-17T10:08:20.827 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-17T10:08:20.832 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-17T10:08:20.858 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi/7660164', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 10:02:00.010557', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGMLKp8KpVegh/Vxz1eYroBDqBaWF0ILPxAx0Fuo5eyU'} 2024-04-17T10:08:20.878 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi/7660164', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-17 10:02:00.012243', 'locked_by': 'scheduled_neesingh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINxUh3pNYtmtVYQJ1TAAeNhCGhuiuosC7ykfY/0PE13C'} 2024-04-17T10:08:20.878 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-17T10:08:20.885 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-17T10:08:20.885 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-17T10:08:20.885 INFO:teuthology.run_tasks:Running task console_log... 2024-04-17T10:08:20.971 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9956c3c4c0>, signals=[15]) 2024-04-17T10:08:20.971 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-17T10:08:20.978 INFO:teuthology.task.internal:Opening connections... 2024-04-17T10:08:20.978 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:08:20.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:08:21.063 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:08:21.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:08:21.142 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-17T10:08:21.149 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-04-17T10:08:21.154 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-04-17T10:08:21.155 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-04-17T10:08:21.200 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-17T10:08:21.200 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Ubuntu" 2024-04-17T10:08:21.200 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="22.04" 2024-04-17T10:08:21.200 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_CODENAME=jammy 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:ID=ubuntu 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE=debian 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-17T10:08:21.201 INFO:teuthology.orchestra.run.smithi067.stdout:UBUNTU_CODENAME=jammy 2024-04-17T10:08:21.202 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-04-17T10:08:21.225 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-04-17T10:08:21.230 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-04-17T10:08:21.230 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="Ubuntu" 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="22.04" 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_CODENAME=jammy 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:ID=ubuntu 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE=debian 2024-04-17T10:08:21.277 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-17T10:08:21.278 INFO:teuthology.orchestra.run.smithi086.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-17T10:08:21.278 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-17T10:08:21.278 INFO:teuthology.orchestra.run.smithi086.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-17T10:08:21.278 INFO:teuthology.orchestra.run.smithi086.stdout:UBUNTU_CODENAME=jammy 2024-04-17T10:08:21.278 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-04-17T10:08:21.300 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-17T10:08:21.309 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-17T10:08:21.315 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-17T10:08:21.316 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-04-17T10:08:21.318 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-04-17T10:08:21.324 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-17T10:08:21.330 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-17T10:08:21.330 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-04-17T10:08:21.365 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-04-17T10:08:21.377 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-17T10:08:21.429 INFO:teuthology.run_tasks:Running task kernel... 2024-04-17T10:08:21.444 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-17T10:08:21.444 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-04-17T10:08:21.444 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-04-17T10:08:21.444 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-17T10:08:21.445 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-17T10:08:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:08:21.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2024-04-17T10:08:21.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:08:21.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2024-04-17T10:08:21.627 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T10:08:21.627 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-17T10:08:21.627 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-17T10:08:21.633 INFO:teuthology.orchestra.run.smithi067.stdout:5.15.0-56-generic 2024-04-17T10:08:21.633 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T10:08:21.633 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-17T10:08:21.633 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2024-04-17T10:08:21.634 DEBUG:teuthology.orchestra.run.smithi067:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-04-17T10:08:21.692 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-04-17T10:08:21.712 DEBUG:teuthology.task.kernel:sha1 for client.1 is 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T10:08:21.712 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8"... 2024-04-17T10:08:21.712 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-04-17T10:08:21.718 INFO:teuthology.orchestra.run.smithi086.stdout:5.15.0-56-generic 2024-04-17T10:08:21.718 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T10:08:21.718 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-17T10:08:21.718 INFO:teuthology.task.kernel:Installing linux-firmware on client.1... 2024-04-17T10:08:21.718 DEBUG:teuthology.orchestra.run.smithi086:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2024-04-17T10:08:21.775 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: git config --global init.defaultBranch 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: 2024-04-17T10:08:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:hint: git branch -m 2024-04-17T10:08:21.846 INFO:teuthology.orchestra.run.smithi067.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-04-17T10:08:21.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: is subject to change. To configure the initial branch name to use in all 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: of your new repositories, which will suppress this warning, call: 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: git config --global init.defaultBranch 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-04-17T10:08:21.901 INFO:teuthology.orchestra.run.smithi086.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2024-04-17T10:08:21.902 INFO:teuthology.orchestra.run.smithi086.stderr:hint: 2024-04-17T10:08:21.902 INFO:teuthology.orchestra.run.smithi086.stderr:hint: git branch -m 2024-04-17T10:08:21.902 INFO:teuthology.orchestra.run.smithi086.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2024-04-17T10:08:21.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2024-04-17T10:08:21.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-04-17T10:08:21.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2024-04-17T10:08:21.970 DEBUG:teuthology.orchestra.run.smithi067:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-04-17T10:08:22.020 DEBUG:teuthology.orchestra.run.smithi086:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2024-04-17T10:10:01.738 INFO:teuthology.orchestra.run.smithi067.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-04-17T10:10:01.738 INFO:teuthology.orchestra.run.smithi067.stderr: * [new branch] main -> origin/main 2024-04-17T10:10:01.773 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20190312 -> 20190312 2024-04-17T10:10:01.773 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20190416 -> 20190416 2024-04-17T10:10:01.773 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20190514 -> 20190514 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20190618 -> 20190618 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20190717 -> 20190717 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20190815 -> 20190815 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20190923 -> 20190923 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20191022 -> 20191022 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20191215 -> 20191215 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200122 -> 20200122 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200316 -> 20200316 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200421 -> 20200421 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200519 -> 20200519 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200619 -> 20200619 2024-04-17T10:10:01.774 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200721 -> 20200721 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200817 -> 20200817 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20200918 -> 20200918 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20201022 -> 20201022 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20201118 -> 20201118 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20201218 -> 20201218 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20210208 -> 20210208 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20210315 -> 20210315 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20210511 -> 20210511 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20210716 -> 20210716 2024-04-17T10:10:01.778 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20210818 -> 20210818 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20210919 -> 20210919 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20211027 -> 20211027 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20211216 -> 20211216 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220209 -> 20220209 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220310 -> 20220310 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220411 -> 20220411 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220509 -> 20220509 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220610 -> 20220610 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220708 -> 20220708 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220815 -> 20220815 2024-04-17T10:10:01.779 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20220913 -> 20220913 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20221012 -> 20221012 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20221109 -> 20221109 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20221214 -> 20221214 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230117 -> 20230117 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230210 -> 20230210 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230310 -> 20230310 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230404 -> 20230404 2024-04-17T10:10:01.780 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230515 -> 20230515 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230625 -> 20230625 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230804 -> 20230804 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20230919 -> 20230919 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20231030 -> 20231030 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20231111 -> 20231111 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20231211 -> 20231211 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20240115 -> 20240115 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20240220 -> 20240220 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20240312 -> 20240312 2024-04-17T10:10:01.782 INFO:teuthology.orchestra.run.smithi067.stderr: * [new tag] 20240410 -> 20240410 2024-04-17T10:10:06.178 INFO:teuthology.orchestra.run.smithi067.stderr:Updating files: 48% (1744/3602) Updating files: 49% (1765/3602) Updating files: 50% (1801/3602) Updating files: 51% (1838/3602) Updating files: 52% (1874/3602) Updating files: 53% (1910/3602) Updating files: 54% (1946/3602) Updating files: 55% (1982/3602) Updating files: 56% (2018/3602) Updating files: 57% (2054/3602) Updating files: 58% (2090/3602) Updating files: 59% (2126/3602) Updating files: 60% (2162/3602) Updating files: 61% (2198/3602) Updating files: 61% (2218/3602) Updating files: 62% (2234/3602) Updating files: 63% (2270/3602) Updating files: 64% (2306/3602) Updating files: 65% (2342/3602) Updating files: 66% (2378/3602) Updating files: 67% (2414/3602) Updating files: 68% (2450/3602) Updating files: 69% (2486/3602) Updating files: 70% (2522/3602) Updating files: 71% (2558/3602) Updating files: 71% (2584/3602) Updating files: 72% (2594/3602) Updating files: 73% (2630/3602) Updating files: 74% (2666/3602) Updating files: 75% (2702/3602) Updating files: 76% (2738/3602) Updating files: 77% (2774/3602) Updating files: 78% (2810/3602) Updating files: 79% (2846/3602) Updating files: 80% (2882/3602) Updating files: 81% (2918/3602) Updating files: 82% (2954/3602) Updating files: 83% (2990/3602) Updating files: 83% (3012/3602) Updating files: 84% (3026/3602) Updating files: 85% (3062/3602) Updating files: 86% (3098/3602) Updating files: 87% (3134/3602) Updating files: 88% (3170/3602) Updating files: 89% (3206/3602) Updating files: 90% (3242/3602) Updating files: 91% (3278/3602) Updating files: 92% (3314/3602) Updating files: 93% (3350/3602) Updating files: 94% (3386/3602) Updating files: 95% (3422/3602) Updating files: 96% (3458/3602) Updating files: 97% (3494/3602) Updating files: 98% (3530/3602) Updating files: 99% (3566/3602) Updating files: 100% (3602/3602) Updating files: 100% (3602/3602), done. 2024-04-17T10:10:06.184 INFO:teuthology.orchestra.run.smithi067.stdout:HEAD is now at 5da74b16 Merge branch 'robot/pr-0-1712895838' into 'main' 2024-04-17T10:10:06.206 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-17T10:10:06.207 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T10:10:06.431 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T10:10:06.431 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /tmp/linux-image.deb && echo linux-image-6.9.0-rc3-g3f60ed14f7c1_6.9.0-rc3-g3f60ed14f7c1-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/ --input-file=- 2024-04-17T10:10:06.434 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi067.front.sepia.ceph.com to complete... 2024-04-17T10:10:07.678 INFO:teuthology.orchestra.run.smithi086.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2024-04-17T10:10:07.678 INFO:teuthology.orchestra.run.smithi086.stderr: * [new branch] main -> origin/main 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20190312 -> 20190312 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20190416 -> 20190416 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20190514 -> 20190514 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20190618 -> 20190618 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20190717 -> 20190717 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20190815 -> 20190815 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20190923 -> 20190923 2024-04-17T10:10:07.711 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20191022 -> 20191022 2024-04-17T10:10:07.712 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20191215 -> 20191215 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200122 -> 20200122 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200316 -> 20200316 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200421 -> 20200421 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200519 -> 20200519 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200619 -> 20200619 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200721 -> 20200721 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200817 -> 20200817 2024-04-17T10:10:07.713 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20200918 -> 20200918 2024-04-17T10:10:07.715 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20201022 -> 20201022 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20201118 -> 20201118 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20201218 -> 20201218 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20210208 -> 20210208 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20210315 -> 20210315 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20210511 -> 20210511 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20210716 -> 20210716 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20210818 -> 20210818 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20210919 -> 20210919 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20211027 -> 20211027 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20211216 -> 20211216 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220209 -> 20220209 2024-04-17T10:10:07.716 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220310 -> 20220310 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220411 -> 20220411 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220509 -> 20220509 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220610 -> 20220610 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220708 -> 20220708 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220815 -> 20220815 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20220913 -> 20220913 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20221012 -> 20221012 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20221109 -> 20221109 2024-04-17T10:10:07.717 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20221214 -> 20221214 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230117 -> 20230117 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230210 -> 20230210 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230310 -> 20230310 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230404 -> 20230404 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230515 -> 20230515 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230625 -> 20230625 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230804 -> 20230804 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20230919 -> 20230919 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20231030 -> 20231030 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20231111 -> 20231111 2024-04-17T10:10:07.719 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20231211 -> 20231211 2024-04-17T10:10:07.720 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20240115 -> 20240115 2024-04-17T10:10:07.720 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20240220 -> 20240220 2024-04-17T10:10:07.720 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20240312 -> 20240312 2024-04-17T10:10:07.720 INFO:teuthology.orchestra.run.smithi086.stderr: * [new tag] 20240410 -> 20240410 2024-04-17T10:10:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17 10:10:08 URL:https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/linux-image-6.9.0-rc3-g3f60ed14f7c1_6.9.0-rc3-g3f60ed14f7c1-1_amd64.deb [44839714/44839714] -> "/tmp/linux-image.deb" [1] 2024-04-17T10:10:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:FINISHED --2024-04-17 10:10:08-- 2024-04-17T10:10:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:Total wall clock time: 2.0s 2024-04-17T10:10:08.457 INFO:teuthology.orchestra.run.smithi067.stderr:Downloaded: 1 files, 43M in 1.8s (23.7 MB/s) 2024-04-17T10:10:08.461 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.0... 2024-04-17T10:10:08.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo dpkg -i /tmp/linux-image.deb 2024-04-17T10:10:08.754 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package linux-image-6.9.0-rc3-g3f60ed14f7c1. 2024-04-17T10:10:10.698 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-04-17T10:10:10.702 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-04-17T10:10:10.825 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking linux-image-6.9.0-rc3-g3f60ed14f7c1 (6.9.0-rc3-g3f60ed14f7c1-1) ... 2024-04-17T10:10:11.882 INFO:teuthology.orchestra.run.smithi086.stderr:Updating files: 50% (1822/3602) Updating files: 51% (1838/3602) Updating files: 52% (1874/3602) Updating files: 53% (1910/3602) Updating files: 54% (1946/3602) Updating files: 55% (1982/3602) Updating files: 56% (2018/3602) Updating files: 57% (2054/3602) Updating files: 58% (2090/3602) Updating files: 59% (2126/3602) Updating files: 60% (2162/3602) Updating files: 61% (2198/3602) Updating files: 62% (2234/3602) Updating files: 63% (2270/3602) Updating files: 63% (2298/3602) Updating files: 64% (2306/3602) Updating files: 65% (2342/3602) Updating files: 66% (2378/3602) Updating files: 67% (2414/3602) Updating files: 68% (2450/3602) Updating files: 69% (2486/3602) Updating files: 70% (2522/3602) Updating files: 71% (2558/3602) Updating files: 72% (2594/3602) Updating files: 73% (2630/3602) Updating files: 74% (2666/3602) Updating files: 75% (2702/3602) Updating files: 76% (2738/3602) Updating files: 77% (2774/3602) Updating files: 77% (2789/3602) Updating files: 78% (2810/3602) Updating files: 79% (2846/3602) Updating files: 80% (2882/3602) Updating files: 81% (2918/3602) Updating files: 82% (2954/3602) Updating files: 83% (2990/3602) Updating files: 84% (3026/3602) Updating files: 85% (3062/3602) Updating files: 86% (3098/3602) Updating files: 87% (3134/3602) Updating files: 88% (3170/3602) Updating files: 89% (3206/3602) Updating files: 90% (3242/3602) Updating files: 91% (3278/3602) Updating files: 92% (3314/3602) Updating files: 93% (3350/3602) Updating files: 94% (3386/3602) Updating files: 94% (3394/3602) Updating files: 95% (3422/3602) Updating files: 96% (3458/3602) Updating files: 97% (3494/3602) Updating files: 98% (3530/3602) Updating files: 99% (3566/3602) Updating files: 100% (3602/3602) Updating files: 100% (3602/3602), done. 2024-04-17T10:10:11.890 INFO:teuthology.orchestra.run.smithi086.stdout:HEAD is now at 5da74b16 Merge branch 'robot/pr-0-1712895838' into 'main' 2024-04-17T10:10:11.907 INFO:teuthology.task.kernel:Downloading kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.1... 2024-04-17T10:10:11.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 2024-04-17T10:10:12.057 INFO:teuthology.task.kernel:fetching, builder baseurl is https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T10:10:12.057 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /tmp/linux-image.deb && echo linux-image-6.9.0-rc3-g3f60ed14f7c1_6.9.0-rc3-g3f60ed14f7c1-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/ --input-file=- 2024-04-17T10:10:12.060 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi086.front.sepia.ceph.com to complete... 2024-04-17T10:10:14.003 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17 10:10:13 URL:https://1.chacra.ceph.com/r/kernel/testing/3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8/ubuntu/jammy/flavors/default/pool/main/l/linux-6.9.0-rc3-g3f60ed14f7c1/linux-image-6.9.0-rc3-g3f60ed14f7c1_6.9.0-rc3-g3f60ed14f7c1-1_amd64.deb [44839714/44839714] -> "/tmp/linux-image.deb" [1] 2024-04-17T10:10:14.003 INFO:teuthology.orchestra.run.smithi086.stderr:FINISHED --2024-04-17 10:10:13-- 2024-04-17T10:10:14.003 INFO:teuthology.orchestra.run.smithi086.stderr:Total wall clock time: 1.9s 2024-04-17T10:10:14.003 INFO:teuthology.orchestra.run.smithi086.stderr:Downloaded: 1 files, 43M in 1.7s (25.9 MB/s) 2024-04-17T10:10:14.007 INFO:teuthology.task.kernel:Installing kernel 3f60ed14f7c110c7ce3c8d83dd2ed0d08e80d0c8 on client.1... 2024-04-17T10:10:14.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo dpkg -i /tmp/linux-image.deb 2024-04-17T10:10:14.260 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package linux-image-6.9.0-rc3-g3f60ed14f7c1. 2024-04-17T10:10:14.638 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up linux-image-6.9.0-rc3-g3f60ed14f7c1 (6.9.0-rc3-g3f60ed14f7c1-1) ... 2024-04-17T10:10:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:update-initramfs: Generating /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:16.200 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... 129542 files and directories currently installed.) 2024-04-17T10:10:16.203 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack /tmp/linux-image.deb ... 2024-04-17T10:10:16.310 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking linux-image-6.9.0-rc3-g3f60ed14f7c1 (6.9.0-rc3-g3f60ed14f7c1-1) ... 2024-04-17T10:10:20.420 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up linux-image-6.9.0-rc3-g3f60ed14f7c1 (6.9.0-rc3-g3f60ed14f7c1-1) ... 2024-04-17T10:10:20.519 INFO:teuthology.orchestra.run.smithi086.stderr:update-initramfs: Generating /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:35.802 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2024-04-17T10:10:35.809 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-17T10:10:35.830 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-04-17T10:10:36.116 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:36.133 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:36.408 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-17T10:10:36.417 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-17T10:10:36.480 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-17T10:10:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-17T10:10:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-17T10:10:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-17T10:10:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-17T10:10:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-04-17T10:10:36.873 DEBUG:teuthology.orchestra.run.smithi067:> dpkg-deb -c /tmp/linux-image.deb 2024-04-17T10:10:36.881 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./ 2024-04-17T10:10:36.881 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./boot/ 2024-04-17T10:10:36.881 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7002012 2024-04-11 18:50 ./boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:36.900 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 172073 2024-04-11 18:50 ./boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:36.901 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13120512 2024-04-11 18:50 ./boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:36.921 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/ 2024-04-17T10:10:36.921 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/ 2024-04-17T10:10:36.921 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/postinst.d/ 2024-04-17T10:10:36.921 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/postrm.d/ 2024-04-17T10:10:36.921 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/preinst.d/ 2024-04-17T10:10:36.921 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/prerm.d/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ 2024-04-17T10:10:36.922 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 507569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/aesni-intel.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 97953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 109905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 155249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 87009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-17T10:10:36.925 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/ 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/ 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/ 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-17T10:10:36.926 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpuid.ko 2024-04-17T10:10:36.927 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/msr.ko 2024-04-17T10:10:36.927 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/ 2024-04-17T10:10:36.927 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 444425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-17T10:10:36.927 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1035881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-17T10:10:36.927 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 2849609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/af_alg.ko 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/algif_hash.ko 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/algif_skcipher.ko 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-17T10:10:36.942 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/asymmetric_keys/ 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/ 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-17T10:10:36.943 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cryptd.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_simd.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-17T10:10:36.944 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lrw.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-17T10:10:36.945 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 169065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xts.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ 2024-04-17T10:10:36.946 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_extlog.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/apei/ 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/apei/einj.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 119033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-17T10:10:36.947 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 96825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 107737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-17T10:10:36.948 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_legacy.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-17T10:10:36.949 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_radisys.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rz1000.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-17T10:10:36.950 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 102121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-17T10:10:36.951 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/ 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 163969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/ 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/ 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 140865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/ 2024-04-17T10:10:36.952 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1241657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-17T10:10:36.991 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 203289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-17T10:10:36.991 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/ 2024-04-17T10:10:36.991 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 115265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 143433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/ 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 126617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 119873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 313217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/ 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 117233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/ 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/applicom.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-17T10:10:36.992 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/ 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 186921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 179561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-17T10:10:36.993 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/ 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 103409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/nvram.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/ 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-04-17T10:10:36.994 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/ 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_class.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_core.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/ 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-palmas.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 57289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-si5351.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-twl6040.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-wm831x.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/ 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-17T10:10:36.995 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 89257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 333657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/ 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/ 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/ 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 751713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/ 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 90441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-17T10:10:36.996 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/ 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/ 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 146769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 96145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 51105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-17T10:10:36.997 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 64161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 68697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/ 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-gpio.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max14577.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max77693.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max8997.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-palmas.ko 2024-04-17T10:10:36.998 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-rt8973a.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-sm5502.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/ 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/dmi-sysfs.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/efi/ 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/efi/efi-pstore.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/ 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-adp5520.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-amd8111.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-arizona.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-da9052.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-da9055.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-dln2.ko 2024-04-17T10:10:36.999 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 81073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-f7188x.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-generic.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-kempld.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-lp3943.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max7300.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max730x.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max732x.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-pca953x.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-pcf857x.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-rdc321x.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-sch.ko 2024-04-17T10:10:37.000 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-sch311x.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-tps65912.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-twl4030.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-twl6040.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-vx855.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm831x.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm8350.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm8994.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/ 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-a4tech.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-apple.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-17T10:10:37.001 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-belkin.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cherry.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-chicony.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 74393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cp2112.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cypress.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ezkey.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-17T10:10:37.002 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-generic.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kensington.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-17T10:10:37.003 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lg-g15.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 68081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 121369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 97953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-magicmouse.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-microsoft.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-monterey.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ntrig.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-17T10:10:37.004 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-17T10:10:37.005 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-17T10:10:37.006 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 81113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 184745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 319921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/ 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 153633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbhid.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbkbd.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbmouse.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 261945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-17T10:10:37.007 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/ 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 75065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 110681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 382033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 77153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 67009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 115057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-17T10:10:37.008 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 89025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 81473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 68865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 69009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-17T10:10:37.009 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/da9052-hwmon.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/da9055-hwmon.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 113201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc2103.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 108961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-17T10:10:37.010 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 56313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hih6130.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-17T10:10:37.011 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 156465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/jc42.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 57769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 86497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-17T10:10:37.012 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 131505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-17T10:10:37.013 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mc13783-adc.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 56505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 141353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 72737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-17T10:10:37.014 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 67841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 97777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 101233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-17T10:10:37.015 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-17T10:10:37.016 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 71169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 96577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-17T10:10:37.017 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 75129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 83425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 116777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 134441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/ 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/ 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/ 2024-04-17T10:10:37.018 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 64233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-04-17T10:10:37.019 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 76969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 57217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-17T10:10:37.020 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/ 2024-04-17T10:10:37.021 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 329697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1119913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 83905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 408433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 120921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 347217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 84945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-17T10:10:37.022 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/ 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 590929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/ 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 514641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/ 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 911665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 375545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ocrdma/ 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 197121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/usnic/ 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 212553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-04-17T10:10:37.023 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 334401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 120673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 132585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 157937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ 2024-04-17T10:10:37.024 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 156673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-17T10:10:37.049 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/ 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/evbug.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/ff-memless.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/ 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/ 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-04-17T10:10:37.050 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lkkbd.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lm8323.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lm8333.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/newtonkbd.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/qt1070.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/qt2160.ko 2024-04-17T10:10:37.051 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/stowaway.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/sunkbd.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/xtkbd.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/matrix-keymap.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/ 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 160017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 114273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-17T10:10:37.052 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/gpio_mouse.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 428185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/psmouse.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/ 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 140969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ct82c710.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/parkbd.ko 2024-04-17T10:10:37.053 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/pcips2.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ps2mult.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serport.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/ 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-88pm860x.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-adp5520.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 77265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-bd2802.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-da903x.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-da9052.ko 2024-04-17T10:10:37.054 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-gpio.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3533.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm355x.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3642.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp8788.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-max8997.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-mc13783.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-menf21bmc.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca9532.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca955x.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca963x.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-tca6507.ko 2024-04-17T10:10:37.055 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-wm831x-status.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-wm8350.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/ 2024-04-17T10:10:37.056 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/ 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 790753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 116825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bufio.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 58281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 199041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 134945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 67497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-era.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-mirror.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 97209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-17T10:10:37.057 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-17T10:10:37.058 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-17T10:10:37.058 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-region-hash.ko 2024-04-17T10:10:37.058 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-17T10:10:37.058 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-17T10:10:37.058 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 139817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-snapshot.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 234457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-zero.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/ 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 264329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 125009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 157681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-17T10:10:37.059 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 444665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/ 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/ 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 214841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 77985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 150761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 97281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 63873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/ 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm800.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm805.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm80x.ko 2024-04-17T10:10:37.060 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/arizona-i2c.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 148633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/arizona.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/bcm590xx.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/cros_ec_dev.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/dln2.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/janz-cmodio.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/kempld-core.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lm3533-core.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lp3943.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/max8907.ko 2024-04-17T10:10:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mc13xxx-core.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/menf21bmc.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633-adc.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633-gpio.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/retu-mfd.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 99425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/si476x-core.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 58057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/tps65010.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/tps6507x.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-17T10:10:37.062 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wm8994.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 64265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ad525x_dpot.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/ 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 57521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 58633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/ 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/core.ko 2024-04-17T10:10:37.063 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/ 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ds1682.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/dummy-irq.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/echo/ 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/echo/echo.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/ 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/genwqe/ 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 191649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hmc6352.ko 2024-04-17T10:10:37.064 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ibmasm/ 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 92345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ics932s401.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/ 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/ 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 154169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 69809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 389033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-17T10:10:37.065 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/phantom.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ti-st/ 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ti-st/st_drv.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/ 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 234905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/ 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 620961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/ 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/ 2024-04-17T10:10:37.066 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 139817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/ 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 587065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/ 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 158649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-17T10:10:37.067 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 73865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 109521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/ 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 64705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/ 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 84329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/ 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/ 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/ 2024-04-17T10:10:37.068 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 56057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 115881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/ 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/ 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 124497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 128409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 110345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/ 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 101857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-17T10:10:37.069 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 88953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/ 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 105633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 228985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1733353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 167185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/ 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 161417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 440625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/ 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/ 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 432201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/ 2024-04-17T10:10:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 257201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 446177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1311433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 206689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 269313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/ 2024-04-17T10:10:37.071 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/ 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 77457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 169089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 76209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/ 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 75945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ec_bhf.ko 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/ 2024-04-17T10:10:37.072 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/ 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 356377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 63185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fujitsu/ 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 51513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 115569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/ 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 366217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/ 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 755793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/ 2024-04-17T10:10:37.073 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 324577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/ 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1196297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/ 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 362593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/ 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 671801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/ 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 121537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 966705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 193585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-17T10:10:37.074 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 108633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/ 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 126881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 150777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/ 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 905033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 344145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-04-17T10:10:37.075 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1412417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-17T10:10:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ 2024-04-17T10:10:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-04-17T10:10:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-04-17T10:10:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-04-17T10:10:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 117353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-17T10:10:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/ 2024-04-17T10:10:37.109 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 131425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 56689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/ 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 181081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/ 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 171417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/ 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 58249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/ 2024-04-17T10:10:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 300817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 105689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 885129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/ 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/ 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 81641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 100689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 264161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-17T10:10:37.111 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/ 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 142449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/ 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1350385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/ 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 57953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/ 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 80065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/ 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 74289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 79457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-17T10:10:37.112 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 100785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 83441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/ 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 644537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/ 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 128497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 207993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 82633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-17T10:10:37.113 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/ 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/ 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 110729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/ 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 98689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 102513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/ 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/ 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 69913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-17T10:10:37.114 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/ 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/defxx.ko 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/skfp/ 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 275801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/skfp/skfp.ko 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/ 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 290441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/ 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 108329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-17T10:10:37.115 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/ 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-gpio.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/ 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-17T10:10:37.116 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 103377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 96025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 121105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-17T10:10:37.117 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/plip/ 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/plip/plip.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-17T10:10:37.118 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sb1000.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/ 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 80161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/ 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 137321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 108089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/ 2024-04-17T10:10:37.119 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 176497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/ 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 383593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/ 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 175537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/ 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/ 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 335857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 126553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/ 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 168145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 103889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-17T10:10:37.120 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/ 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/ 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pci-stub.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/ 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 179961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia_core.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-04-17T10:10:37.121 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 133753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/ 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/broadcom/ 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/samsung/ 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/ 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/ 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/ 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/ 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec.ko 2024-04-17T10:10:37.122 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/ 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 87481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 73865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 164769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-17T10:10:37.123 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 51881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/ibm_rtl.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/ 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 51897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-17T10:10:37.124 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/ 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/88pm860x_battery.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/88pm860x_charger.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 68833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq2415x_charger.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 73073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq24190_charger.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq24735-charger.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/da9030_battery.ko 2024-04-17T10:10:37.125 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/da9052-battery.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2760_battery.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2780_battery.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2781_battery.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2782_battery.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/gpio-charger.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/isp1704_charger.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/lp8727_charger.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max14577_charger.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max17042_battery.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max8903_charger.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max8925_power.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/pcf50633-charger.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/sbs-battery.ko 2024-04-17T10:10:37.126 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/test_power.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/tps65090-charger.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm831x_backup.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm831x_power.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm8350_power.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/ 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 73297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/ 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/ 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-17T10:10:37.127 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/ 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/ 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/ 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 67513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/ 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-88pm80x.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-88pm860x.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9052.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9055.ko 2024-04-17T10:10:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9063.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 82225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-17T10:10:37.129 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-lp8788.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t86.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8907.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8925.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8997.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8998.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-palmas.ko 2024-04-17T10:10:37.130 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 56777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf50633.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rc5t583.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-s35390a.ko 2024-04-17T10:10:37.131 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-tps6586x.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-tps65910.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-wm831x.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-wm8350.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 90377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 69593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 164569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/ 2024-04-17T10:10:37.132 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 340721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 120337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/ 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 468081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 380673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic94xx/ 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 243705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/ 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 135617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/ 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 304993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/ 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1527113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-17T10:10:37.133 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/ 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 297801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/ 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 144401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/ 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 390713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/ 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 88881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 117481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 173017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-17T10:10:37.134 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/ 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 267577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/ 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 95329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 194017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fdomain.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/ 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 366529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 251505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 57097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-17T10:10:37.135 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/imm.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 327841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 129785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/ 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 397785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 63121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/ 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 386849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 178177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 64809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/ 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 274009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-17T10:10:37.136 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/ 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 2736001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/ 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 88529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 389177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 104769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/ 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 836945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/ 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 158729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 96537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/ 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 101625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-04-17T10:10:37.137 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/ 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 443745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 152033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ppa.ko 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/ 2024-04-17T10:10:37.138 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 2324809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-17T10:10:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 103601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-17T10:10:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/ 2024-04-17T10:10:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 697337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-17T10:10:37.177 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qlogicfas408.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 322865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 219857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 357153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 121185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 93481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 138121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 57401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/ 2024-04-17T10:10:37.178 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 205025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/xen-scsifront.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 174913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/ 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/ 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 920169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/ 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1170033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 51897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-17T10:10:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 149721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/ 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 90057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/ 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/ 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/ 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-17T10:10:37.180 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 99313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-17T10:10:37.181 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 69729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/moxa.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/mxser.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 134401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/ 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/ 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-04-17T10:10:37.182 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/altera_uart.ko 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/ 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 120577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/rp2.ko 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/sc16is7xx.ko 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/ 2024-04-17T10:10:37.186 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 693081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/ 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_dmem_genirq.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_mf624.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_netx.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-17T10:10:37.187 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/ 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/c67x00/ 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 90321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/c67x00/c67x00.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 189209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/ 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 101545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-17T10:10:37.188 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 63073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 56721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 107417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/common/ 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/common/ulpi.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc2/ 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/ 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 574593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/ 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/ 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 72249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/u_ether.ko 2024-04-17T10:10:37.189 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/u_serial.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 131345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 146617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 101249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-04-17T10:10:37.190 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 71257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 84937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/ 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-04-17T10:10:37.191 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 204049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/libcomposite.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/ 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/bdc/ 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 182561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 69313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 90313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/net2272.ko 2024-04-17T10:10:37.192 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 90361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/net2280.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 94841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 80521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 87465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 95441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 195969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/ 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/bcma-hcd.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 108321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/isp116x-hcd.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 104377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 102273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811_cs.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/ssb-hcd.ko 2024-04-17T10:10:37.193 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/ 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/cytherm.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ehset.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-17T10:10:37.194 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/lvstest.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/ 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-17T10:10:37.195 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 84265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbtest.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/mon/ 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 91529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/mon/usbmon.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/musb/ 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 396209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/musb/musb_hdrc.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/ 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-isp1301.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-tahvo.ko 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ 2024-04-17T10:10:37.196 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 79681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 58329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/f81232.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 188609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 152305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 142833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-17T10:10:37.197 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 69217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 122329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/metro-usb.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 79681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 89969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mxuport.ko 2024-04-17T10:10:37.198 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 193857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-17T10:10:37.199 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 151137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usbserial.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/wishbone-serial.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 64049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-17T10:10:37.200 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 201193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/ 2024-04-17T10:10:37.201 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 76041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 122321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/ 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/ 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 206921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 134265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 93673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/ 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 138721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 71281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-17T10:10:37.202 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 107905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/ 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/ 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/aat2870_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp5520_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp8860_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 51745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp8870_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/as3711_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/bd6107.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/da903x_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/da9052_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/gpio_backlight.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/kb3886_bl.ko 2024-04-17T10:10:37.203 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lm3533_bl.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lm3639_bl.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lv5207lp.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/max8925_bl.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/pandora_bl.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/wm831x_bl.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/ 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-17T10:10:37.204 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/matrox_w1.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/w1-gpio.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/ 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-17T10:10:37.205 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 129433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/acquirewdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/advantechwdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/cpu5wdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9052_wdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9055_wdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9063_wdt.ko 2024-04-17T10:10:37.206 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/dw_wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/eurotechwdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/kempld_wdt.ko 2024-04-17T10:10:37.207 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/mena21_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pc87413_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/retu_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sc1200wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-04-17T10:10:37.208 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/twl4030_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wafer5823wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wm831x_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wm8350_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/ 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-17T10:10:37.209 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/ 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 213913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-scsiback.ko 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/ 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/ 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 192617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/adfs/ 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 78153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/adfs/adfs.ko 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/ 2024-04-17T10:10:37.210 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 197849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/afs/ 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1261113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/afs/kafs.ko 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/autofs/ 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 127377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/autofs/autofs4.ko 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/ 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 80521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/bfs/ 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/bfs/bfs.ko 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/ 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 4489593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/ 2024-04-17T10:10:37.211 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 376857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1586089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/ 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 130073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/configfs/ 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 139081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/configfs/configfs.ko 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/ 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/ 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 813225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/efs/ 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/efs/efs.ko 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/ 2024-04-17T10:10:37.212 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1906817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/ 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/freevxfs/ 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/freevxfs/freevxfs.ko 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/ 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/ 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1486521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/ 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 183305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/ 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 282097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hpfs/ 2024-04-17T10:10:37.213 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 249009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hpfs/hpfs.ko 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/ 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 93033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/ 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 533073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/ 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 361369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/ 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 112609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/ 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 591585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/ 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/ 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 102561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-17T10:10:37.214 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/ 2024-04-17T10:10:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 74353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-17T10:10:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/ 2024-04-17T10:10:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 147689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-17T10:10:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1320249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-17T10:10:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 82105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv2.ko 2024-04-17T10:10:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 151641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-17T10:10:37.215 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 2518505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-17T10:10:37.258 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/ 2024-04-17T10:10:37.258 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-17T10:10:37.258 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-17T10:10:37.258 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/ 2024-04-17T10:10:37.258 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 2024049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-17T10:10:37.258 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/ 2024-04-17T10:10:37.258 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 734025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/ 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-17T10:10:37.259 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ascii.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-17T10:10:37.260 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 95625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 144209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 167137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 119513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-17T10:10:37.261 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-17T10:10:37.262 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 302073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 548169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 3245617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 22113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/omfs/ 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 67489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/omfs/omfs.ko 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/ 2024-04-17T10:10:37.263 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 462873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 67977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx4/ 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx4/qnx4.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx6/ 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 56097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx6/qnx6.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/ 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_tree.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_v1.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_v2.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/ 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 684841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-17T10:10:37.264 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/ 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/ 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/ 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 3319041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/ 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/ 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 164185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/ 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 115977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/ 2024-04-17T10:10:37.265 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 310017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 198233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/ 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 5192705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/ 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/locking/ 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 97625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/locking/locktorture.ko 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/time/ 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/time/test_udelay.ko 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 72665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-17T10:10:37.266 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc7.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/ 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/interval_tree_test.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/ 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/ 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-17T10:10:37.267 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/memory-notifier-error-inject.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/notifier-error-inject.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/percpu_test.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/pm-notifier-error-inject.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/ 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 152025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/rbtree_test.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/ 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test-kstrtox.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 4912441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_bpf.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 61169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_firmware.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_module.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_user_copy.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-17T10:10:37.268 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/xz/ 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/xz/xz_dec_test.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/ 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/ 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-17T10:10:37.269 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/ 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/ 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 106793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/ 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 236793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-17T10:10:37.270 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/ 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 879697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/ 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 84761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1030873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-17T10:10:37.271 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-17T10:10:37.272 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/ 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1243505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/ 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 153801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/ 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 244889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 71033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 74993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-17T10:10:37.273 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/ 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 507705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/hsr/ 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 128873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/hsr/hsr.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 357425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 59393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ 2024-04-17T10:10:37.274 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 75489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 77721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-17T10:10:37.275 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ip_tables.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-17T10:10:37.276 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-17T10:10:37.277 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 54441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ 2024-04-17T10:10:37.278 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 47433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 103177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_gre.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 118881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 71089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/mip6.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-17T10:10:37.279 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-17T10:10:37.280 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 29817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 94089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 30297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/ 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 106369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/ 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 153217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-17T10:10:37.281 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 66241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/lapb/ 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 63345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/lapb/lapb.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/ 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 332889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc2.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/ 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 388873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/ 2024-04-17T10:10:37.282 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 147745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 124041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 124345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 126681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 126473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 137833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 72353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 133681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-17T10:10:37.283 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 139073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 138265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 136177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 139897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 534113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-17T10:10:37.284 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 55217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 496817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 134377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-17T10:10:37.285 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 140801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 79873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 117201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 46105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 118369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-17T10:10:37.286 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 807985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 65145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 50657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-17T10:10:37.287 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_osf.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-17T10:10:37.288 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 24993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_synproxy.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tproxy.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 108393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/x_tables.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 21865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-17T10:10:37.289 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-17T10:10:37.290 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-17T10:10:37.291 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 52817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 16161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-17T10:10:37.292 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 11881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 19777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-17T10:10:37.293 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpudp.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/ 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 17233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/ 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 14073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/ 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 488825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 10873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-17T10:10:37.294 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/ 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/ 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 142025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/phonet.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 68817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/pn_pep.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/ 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 318617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 202305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 85105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rxrpc/ 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1027897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rxrpc/rxrpc.ko 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/ 2024-04-17T10:10:37.295 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 28721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 27593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 53465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 45673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 41457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_cgroup.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 37305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-17T10:10:37.296 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 49321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 79145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 15385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 48785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 8377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 25097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 43153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 39177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq_codel.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 42889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 62817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-17T10:10:37.297 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 81833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 36625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 44793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 32833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 13505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 31801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 60249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 40889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 33433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 38329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-17T10:10:37.298 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 35777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 34161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/ 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1187313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/ 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/ 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 432409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1940889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/ 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 890729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/ 2024-04-17T10:10:37.299 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 1122337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-17T10:10:37.302 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/ 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 18801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/ 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 105609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 90265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 123769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 12905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/x25/ 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 165841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/x25/x25.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/ 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 26729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_algo.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 131865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_user.ko 2024-04-17T10:10:37.303 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 644769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 628542 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 7706 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 23609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 9941 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 70990 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 146959 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 232835 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 210 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 72602 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 879 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 258450 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 313019 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/ 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/ 2024-04-17T10:10:37.304 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/doc/ 2024-04-17T10:10:37.305 INFO:teuthology.orchestra.run.smithi067.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T10:10:37.305 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 174 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/changelog.Debian.gz 2024-04-17T10:10:37.305 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r-- root/root 691 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/copyright 2024-04-17T10:10:37.310 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:37.310 INFO:teuthology.task.kernel:searching for kernel 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:37.311 DEBUG:teuthology.orchestra.run.smithi067:> egrep '(submenu|menuentry.*6.9.0-rc3-g3f60ed14f7c1).*{' /boot/grub/grub.cfg 2024-04-17T10:10:37.352 INFO:teuthology.orchestra.run.smithi067.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T10:10:37.352 INFO:teuthology.orchestra.run.smithi067.stdout: menuentry 'Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-rc3-g3f60ed14f7c1-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T10:10:37.352 INFO:teuthology.orchestra.run.smithi067.stdout: menuentry 'Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-rc3-g3f60ed14f7c1-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T10:10:37.353 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-04-17T10:10:37.353 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:37.353 DEBUG:teuthology.orchestra.run.smithi067:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-04-17T10:10:37.395 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi067.front.sepia.ceph.com to complete... 2024-04-17T10:10:37.768 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub' 2024-04-17T10:10:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-17T10:10:37.793 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2024-04-17T10:10:37.957 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:37.964 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:38.179 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-17T10:10:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-17T10:10:38.227 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-17T10:10:38.232 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-17T10:10:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-17T10:10:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-17T10:10:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-17T10:10:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:done 2024-04-17T10:10:41.962 INFO:teuthology.orchestra.run.smithi086.stderr:Sourcing file `/etc/default/grub' 2024-04-17T10:10:41.980 INFO:teuthology.orchestra.run.smithi086.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-17T10:10:42.006 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-04-17T10:10:42.309 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:42.326 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:42.622 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-17T10:10:42.635 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-17T10:10:42.698 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-17T10:10:42.703 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-17T10:10:42.867 INFO:teuthology.orchestra.run.smithi086.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-17T10:10:42.867 INFO:teuthology.orchestra.run.smithi086.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-17T10:10:42.867 INFO:teuthology.orchestra.run.smithi086.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-17T10:10:42.910 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-04-17T10:10:43.095 DEBUG:teuthology.orchestra.run.smithi086:> dpkg-deb -c /tmp/linux-image.deb 2024-04-17T10:10:43.103 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./ 2024-04-17T10:10:43.103 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./boot/ 2024-04-17T10:10:43.103 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7002012 2024-04-11 18:50 ./boot/System.map-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:43.120 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 172073 2024-04-11 18:50 ./boot/config-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:43.120 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13120512 2024-04-11 18:50 ./boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/postinst.d/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/postrm.d/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/preinst.d/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./etc/kernel/prerm.d/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/ 2024-04-17T10:10:43.140 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/ 2024-04-17T10:10:43.141 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/ 2024-04-17T10:10:43.141 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ 2024-04-17T10:10:43.141 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 507569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/aesni-intel.ko 2024-04-17T10:10:43.141 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 97953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 109905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 155249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 87009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-17T10:10:43.144 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/ 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/ 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/ 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/cpuid.ko 2024-04-17T10:10:43.145 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kernel/msr.ko 2024-04-17T10:10:43.146 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/ 2024-04-17T10:10:43.146 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 444425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-17T10:10:43.146 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1035881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-17T10:10:43.146 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 2849609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/arch/x86/kvm/kvm.ko 2024-04-17T10:10:43.159 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ 2024-04-17T10:10:43.159 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/af_alg.ko 2024-04-17T10:10:43.159 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/algif_hash.ko 2024-04-17T10:10:43.159 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/algif_skcipher.ko 2024-04-17T10:10:43.159 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ansi_cprng.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/anubis.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/arc4.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/asymmetric_keys/ 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/ 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_memcpy.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_pq.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_tx.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/async_xor.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/async_tx/raid6test.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authenc.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/authencesn.ko 2024-04-17T10:10:43.160 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blake2b_generic.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_common.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/blowfish_generic.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/camellia_generic.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast5_generic.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast6_generic.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cast_common.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/ccm.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cmac.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crc32_generic.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cryptd.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_engine.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_simd.ko 2024-04-17T10:10:43.161 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/crypto_user.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/cts.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/des_generic.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/dh_generic.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/echainiv.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/essiv.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/fcrypt.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/khazad.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lrw.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/lz4hc.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/md4.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/michael_mic.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcbc.ko 2024-04-17T10:10:43.162 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/pcrypt.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/rmd160.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/seed.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/serpent_generic.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 169065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tcrypt.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/tea.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_common.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/twofish_generic.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/vmac.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/wp512.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xcbc.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xor.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xts.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/crypto/xxhash_generic.ko 2024-04-17T10:10:43.163 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_extlog.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/acpi_pad.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/apei/ 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/apei/einj.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/ec_sys.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/platform_profile.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbs.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/sbshc.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 119033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/acpi/video.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/acard-ahci.ko 2024-04-17T10:10:43.164 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 96825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/ahci_platform.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 107737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/libahci_platform.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_acpi.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ali.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_amd.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_artop.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atiixp.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_atp867x.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd640.ko 2024-04-17T10:10:43.165 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cmd64x.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_cypress.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_efar.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt366.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt37x.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it8213.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_it821x.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_jmicron.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_legacy.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_marvell.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_mpiix.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_netcell.ko 2024-04-17T10:10:43.166 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ninja32.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87410.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_ns87415.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_oldpiix.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_opti.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_optidma.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pcmcia.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_piccolo.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_radisys.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rdc.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_rz1000.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sch.ko 2024-04-17T10:10:43.167 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_serverworks.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sil680.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_sl82c105.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_triflex.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pata_via.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/pdc_adma.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_inic162x.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 102121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_mv.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_nv.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_promise.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_qstor.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sil24.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sis.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_svw.ko 2024-04-17T10:10:43.168 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_sx4.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_uli.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_via.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ata/sata_vsc.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/auxdisplay/ks0108.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/ 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 163969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/bcma/bcma.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/ 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/ 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 140865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/aoe/aoe.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/ 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1241657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/drbd/drbd.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 203289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/floppy.ko 2024-04-17T10:10:43.169 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/ 2024-04-17T10:10:43.170 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 115265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-17T10:10:43.170 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 143433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/nbd.ko 2024-04-17T10:10:43.170 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/ 2024-04-17T10:10:43.170 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 126617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/null_blk/null_blk.ko 2024-04-17T10:10:43.170 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 119873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/pktcdvd.ko 2024-04-17T10:10:43.170 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 313217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/rbd.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/ 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 117233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/ 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/block/zram/zram.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/applicom.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hangcheck-timer.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/ 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-17T10:10:43.227 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/via-rng.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 186921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 179561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/lp.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/ 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 103409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/mwave/mwave.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/nvram.ko 2024-04-17T10:10:43.228 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/ppdev.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tlclk.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/ 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/tpm/xen-tpmfront.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/ 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_class.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_core.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/char/xillybus/xillybus_pcie.ko 2024-04-17T10:10:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/ 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-palmas.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 57289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-si5351.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-twl6040.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/clk/clk-wm831x.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/ 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 89257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 333657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/ccp/ccp.ko 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/ 2024-04-17T10:10:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/ 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/ 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 751713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-aes.ko 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/padlock-sha.ko 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/ 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 90441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/ 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dca/dca.ko 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/ 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-17T10:10:43.231 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ 2024-04-17T10:10:43.232 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 146769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-17T10:10:43.233 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ 2024-04-17T10:10:43.233 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 96145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/amd64_edac.ko 2024-04-17T10:10:43.233 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/e752x_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 51105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/edac_mce_amd.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3000_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i3200_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5100_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i5400_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7300_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 64161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i7core_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/i82975x_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/ie31200_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 68697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/sb_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/edac/x38_edac.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/ 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-gpio.ko 2024-04-17T10:10:43.234 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max14577.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max77693.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-max8997.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-palmas.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-rt8973a.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/extcon/extcon-sm5502.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/ 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/dmi-sysfs.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/efi/ 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/efi/efi-pstore.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/ 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-adp5520.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-amd8111.ko 2024-04-17T10:10:43.235 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-arizona.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-da9052.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-da9055.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-dln2.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 81073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-f7188x.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-generic.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ich.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-janz-ttl.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-kempld.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-lp3943.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max7300.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max730x.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-max732x.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-ml-ioh.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-pca953x.ko 2024-04-17T10:10:43.236 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-pcf857x.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-rdc321x.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-sch.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-sch311x.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-tps65912.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-twl4030.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-twl6040.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-vx855.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm831x.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm8350.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/gpio/gpio-wm8994.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/ 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-a4tech.ko 2024-04-17T10:10:43.237 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-apple.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-appleir.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-aureal.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-axff.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-belkin.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cherry.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-chicony.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 74393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cp2112.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-cypress.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-dr.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elecom.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-elo.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-emsff.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ezkey.ko 2024-04-17T10:10:43.238 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gaff.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-generic.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gt683r.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-gyration.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-holtekff.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-hyperv.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-icade.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kensington.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-keytouch.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-kye.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lcpower.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-led.ko 2024-04-17T10:10:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lenovo.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-lg-g15.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 68081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 121369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 97953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-logitech.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-magicmouse.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-microsoft.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-monterey.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-multitouch.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ntrig.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-ortek.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-penmount.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-petalynx.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-picolcd.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-pl.ko 2024-04-17T10:10:43.240 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-plantronics.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-primax.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-rmi.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-common.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-17T10:10:43.241 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-roccat.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-saitek.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-samsung.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sjoy.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sony.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-speedlink.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-steelseries.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-sunplus.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tivo.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-tmff.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-topseed.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-twinhan.ko 2024-04-17T10:10:43.242 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 81113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-uclogic.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-waltop.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 184745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-wiimote.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-xinmo.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zpff.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid-zydacron.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 319921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/hid.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/uhid.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/ 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 153633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbhid.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbkbd.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/usbhid/usbmouse.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 261945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hid/wacom.ko 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/ 2024-04-17T10:10:43.243 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 75065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_balloon.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 110681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_utils.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 382033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hv/hv_vmbus.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 77153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 67009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/abituguru3.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7414.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ad7418.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adc128d818.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1025.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 115057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1026.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1029.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm1031.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adm9240.ko 2024-04-17T10:10:43.244 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ads7828.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7410.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7411.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 89025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7462.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7470.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 81473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7475.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/adt7x10.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/amc6821.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 68865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/applesmc.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asb100.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 69009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asc7621.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/atxp1.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/coretemp.ko 2024-04-17T10:10:43.245 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/da9052-hwmon.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/da9055-hwmon.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 113201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/dme1737.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds1621.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ds620.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc1403.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc2103.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/emc6w201.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71805f.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 108961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f71882fg.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/f75375s.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fam15h_power.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/fschmd.ko 2024-04-17T10:10:43.246 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g760a.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/g762.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl518sm.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 56313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/gl520sm.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hih6130.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5500_temp.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/i5k_amb.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmaem.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ibmpex.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina209.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ina2xx.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 156465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/it87.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/jc42.ko 2024-04-17T10:10:43.247 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k10temp.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/k8temp.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lineage-pem.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 57769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm63.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm73.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm75.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm77.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm78.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm80.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm83.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 86497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm85.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm87.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm90.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm92.ko 2024-04-17T10:10:43.248 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 131505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm93.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95234.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95241.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/lm95245.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc2945.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4151.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4215.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4222.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4245.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4260.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/ltc4261.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max16065.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1619.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max1668.ko 2024-04-17T10:10:43.249 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max197.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6639.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6650.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/max6697.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mc13783-adc.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/mcp3021.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 56505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6683.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 141353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775-core.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 72737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct6775.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 67841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/nct7802.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 97777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87360.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pc87427.ko 2024-04-17T10:10:43.250 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pcf8591.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 101233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/powr1220.ko 2024-04-17T10:10:43.251 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5627.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch5636.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht15.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sht21.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/shtc1.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/sis5595.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47b397.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m1.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/smsc47m192.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/thmc50.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp102.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp103.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp401.ko 2024-04-17T10:10:43.252 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/tmp421.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via-cputemp.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/via686a.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt1211.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/vt8231.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 71169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627ehf.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 96577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83627hf.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83781d.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 75129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83791d.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 83425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83792d.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 116777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83793.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 134441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83795.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l785ts.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/w83l786ng.ko 2024-04-17T10:10:43.253 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/wm831x-hwmon.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/hwmon/wm8350-hwmon.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/ 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/ 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/ 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali1535.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali1563.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2024-04-17T10:10:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 64233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-dln2.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-gpio.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 76969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-kempld.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-17T10:10:43.255 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-ocores.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 57217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis5595.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis630.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-17T10:10:43.256 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/busses/i2c-xiic.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-mux.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-smbus.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/i2c-stub.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/ 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 329697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1119913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_core.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 83905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-17T10:10:43.257 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 408433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 120921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 347217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 84945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/ 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 590929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/ 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 514641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/ 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 911665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 375545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-17T10:10:43.258 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ocrdma/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 197121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/usnic/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 212553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 334401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 120673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 132585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 157937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 156673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-17T10:10:43.259 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/ 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/evbug.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/ff-memless.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/ 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/gameport/gameport.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/joydev.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/ 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5520-keys.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5588-keys.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/adp5589-keys.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-17T10:10:43.260 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lkkbd.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lm8323.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/lm8333.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/matrix_keypad.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/max7359_keypad.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/mcs_touchkey.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/newtonkbd.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/opencores-kbd.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/qt1070.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/qt2160.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/samsung-keypad.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/stowaway.ko 2024-04-17T10:10:43.261 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/sunkbd.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/tca6416-keypad.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/tca8418_keypad.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/twl4030_keypad.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/keyboard/xtkbd.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/matrix-keymap.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/ 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/appletouch.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/bcm5974.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 160017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/cyapatp.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 114273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/gpio_mouse.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 428185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/psmouse.ko 2024-04-17T10:10:43.262 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/sermouse.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/ 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 140969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/altera_ps2.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/arc_ps2.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ct82c710.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/parkbd.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/pcips2.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/ps2mult.ko 2024-04-17T10:10:43.263 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serio_raw.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/serio/serport.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/input/sparse-keymap.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/ 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-88pm860x.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-adp5520.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 77265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-bd2802.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-blinkm.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-da903x.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-da9052.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-gpio.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3530.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3533.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm355x.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lm3642.ko 2024-04-17T10:10:43.264 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp3944.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-lp8788.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-max8997.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-mc13783.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-menf21bmc.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca9532.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca955x.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-pca963x.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-ss4200.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-tca6507.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-wm831x-status.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/leds-wm8350.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-17T10:10:43.265 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/ 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/ 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 790753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/bcache/bcache.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bio-prison.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 116825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-bufio.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 58281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache-smq.ko 2024-04-17T10:10:43.266 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 199041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-cache.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 134945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-crypt.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-delay.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 67497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-era.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-flakey.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log-userspace.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-log.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-mirror.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 97209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-multipath.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-queue-length.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-raid.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-region-hash.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-round-robin.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-service-time.ko 2024-04-17T10:10:43.267 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 139817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-snapshot.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-switch.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 234457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-thin-pool.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-verity.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/dm-zero.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/ 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 264329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid0.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 125009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid1.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 157681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid10.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 444665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/md/raid456.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/ 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/ 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 214841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptbase.ko 2024-04-17T10:10:43.268 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 77985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptctl.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptfc.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptlan.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 150761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptsas.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 97281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptscsih.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 63873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/message/fusion/mptspi.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/ 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm800.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm805.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/88pm80x.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/arizona-i2c.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 148633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/arizona.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/bcm590xx.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/cros_ec_dev.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/dln2.ko 2024-04-17T10:10:43.269 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/janz-cmodio.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/kempld-core.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lm3533-core.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lm3533-ctrlbank.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lp3943.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_ich.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/lpc_sch.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/max8907.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mc13xxx-core.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/mc13xxx-i2c.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/menf21bmc.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633-adc.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633-gpio.ko 2024-04-17T10:10:43.270 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/pcf50633.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/rdc321x-southbridge.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/retu-mfd.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 99425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/si476x-core.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 58057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/sm501.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/tps65010.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/tps6507x.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/viperboard.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/vx855.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wl1273-core.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/mfd/wm8994.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ad525x_dpot-i2c.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 64265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ad525x_dpot.ko 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/ 2024-04-17T10:10:43.271 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 57521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds9802als.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/apds990x.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 58633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/bh1770glc.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/ 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/c2port/core.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/ 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/cb710/cb710.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ds1682.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/dummy-irq.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/echo/ 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/echo/echo.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/ 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/at24.ko 2024-04-17T10:10:43.272 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/eeprom/max6875.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/enclosure.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/genwqe/ 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 191649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/genwqe/genwqe_card.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hmc6352.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/hpilo.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ibmasm/ 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 92345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ibmasm/ibmasm.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ics932s401.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29003.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/isl29020.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/ 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-17T10:10:43.273 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/ 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 154169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-me.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 69809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei-txe.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 389033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/mei/mei.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/phantom.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ti-st/ 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/ti-st/st_drv.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_7xx1.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tifm_core.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/tsl2550.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_balloon.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/ 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 234905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ 2024-04-17T10:10:43.274 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/ 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 620961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/bonding/bonding.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/ 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/ 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 139817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/ 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 587065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/dummy.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/eql.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/ 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-17T10:10:43.275 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 158649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 73865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 109521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/ 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 64705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/ 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 84329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/ 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-17T10:10:43.276 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/ 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/ 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 56057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 115881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/ 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/ 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 124497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/ 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 128409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/ 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 110345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/ 2024-04-17T10:10:43.277 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 101857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 88953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/ 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 105633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 228985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1733353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 167185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/ 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 161417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 440625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/ 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/ 2024-04-17T10:10:43.278 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 432201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-17T10:10:43.357 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/ 2024-04-17T10:10:43.357 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 257201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 446177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1311433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 206689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 269313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/ 2024-04-17T10:10:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/ 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 77457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 169089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 76209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/ 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 75945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/dnet.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ec_bhf.ko 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/ 2024-04-17T10:10:43.359 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/ 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 356377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ethoc.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 63185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fealnx.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fujitsu/ 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 51513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 115569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/ 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 366217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/ 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 755793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/ 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 324577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-17T10:10:43.360 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/ 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1196297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/ 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 362593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/ 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 671801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/ 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 121537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 966705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 193585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 108633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/jme.ko 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/ 2024-04-17T10:10:43.361 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 126881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 150777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/ 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/ 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 905033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 344145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/ 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1412417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8842.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2024-04-17T10:10:43.362 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 117353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/ 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/ 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 131425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 56689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/ 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 181081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/ 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 171417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/ 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 58249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-17T10:10:43.363 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/ 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/ 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 300817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 105689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 885129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/ 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/ 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 81641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 100689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 264161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/ 2024-04-17T10:10:43.364 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/sxgbe/ 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 142449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/ 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1350385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/ 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 57953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/ 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 80065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/ 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 74289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 79457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 100785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 83441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-17T10:10:43.365 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/ 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/ 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 644537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/ 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 128497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 207993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 82633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/ 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-17T10:10:43.366 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/ 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 110729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/ 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 98689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 102513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/ 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/ 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 69913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/ 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/defxx.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/skfp/ 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 275801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/fddi/skfp/skfp.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/geneve.ko 2024-04-17T10:10:43.367 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/ 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 290441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/ 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ifb.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 108329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvlan.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/macvtap.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/ 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-gpio.ko 2024-04-17T10:10:43.368 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mdio.ko 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/mii.ko 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/netconsole.ko 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/nlmon.ko 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/ 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-17T10:10:43.369 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/amd.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/bcm87xx.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/broadcom.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/cicada.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/davicom.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/et1011c.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/icplus.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/lxt.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 103377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/marvell.ko 2024-04-17T10:10:43.370 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 96025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/micrel.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/national.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 121105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/phylink.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/qsemi.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/realtek.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/smsc.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/ste10Xp.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/phy/vitesse.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/plip/ 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/plip/plip.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_async.ko 2024-04-17T10:10:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppoe.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pppox.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/ppp/pptp.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sb1000.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/ 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/slip/slip.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/sungem_phy.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 80161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/tap.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/ 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 137321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team.ko 2024-04-17T10:10:43.372 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_random.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 108089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/veth.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/ 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 176497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/ 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 383593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/vxlan/vxlan.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/ 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 175537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ntb/ntb.ko 2024-04-17T10:10:43.373 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/ 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/ 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 335857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme-core.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 126553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/nvme/host/nvme.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/ 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 168145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_cs.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 103889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_pc.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/parport/parport_serial.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/ 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/ 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2024-04-17T10:10:43.374 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/pci-stub.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pci/xen-pcifront.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/ 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/i82092.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 179961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia_core.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/pd6729.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 133753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/ 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/broadcom/ 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2024-04-17T10:10:43.375 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/samsung/ 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/ 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/ 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/ 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/ 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/ 2024-04-17T10:10:43.376 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 87481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/acerhdf.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 73865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 164769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 51881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/hdaps.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/ibm_rtl.ko 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/ 2024-04-17T10:10:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/intel_ips.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 51897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/platform/x86/wmi.ko 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/ 2024-04-17T10:10:43.378 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/88pm860x_battery.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/88pm860x_charger.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 68833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq2415x_charger.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 73073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq24190_charger.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/bq24735-charger.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/cros_peripheral_charger.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/da9030_battery.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/da9052-battery.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2760_battery.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2780_battery.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2781_battery.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/ds2782_battery.ko 2024-04-17T10:10:43.379 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/gpio-charger.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/isp1704_charger.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/lp8727_charger.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max14577_charger.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max17042_battery.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max8903_charger.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/max8925_power.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/pcf50633-charger.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/sbs-battery.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/test_power.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/tps65090-charger.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm831x_backup.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm831x_power.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/power/supply/wm8350_power.ko 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/ 2024-04-17T10:10:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 73297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/ 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/ 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/clients/pps_parport.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/pps/pps_core.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ptp/ptp_kvm.ko 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/ 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/ 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/ 2024-04-17T10:10:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 67513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ras/amd/fmpm.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/ 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-88pm80x.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-88pm860x.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9052.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9055.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-da9063.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 82225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-17T10:10:43.382 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-em3027.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-lp8788.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-m48t86.ko 2024-04-17T10:10:43.383 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max6900.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8907.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8925.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8997.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-max8998.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-mc13xxx.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-palmas.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 56777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf50633.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-17T10:10:43.384 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rc5t583.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-s35390a.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-tps6586x.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-tps65910.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-wm831x.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-wm8350.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/rtc/rtc-x1205.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 90377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-9xxx.ko 2024-04-17T10:10:43.385 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 69593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-sas.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/3w-xxxx.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 164569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/BusLogic.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/a100u2w.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/ 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 340721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 120337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/advansys.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/ 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 468081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 380673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic94xx/ 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 243705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/aic94xx/aic94xx.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/am53c974.ko 2024-04-17T10:10:43.386 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/ 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 135617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/atp870u.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/ 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 304993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/ 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1527113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bfa/bfa.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/ 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 297801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/ 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 144401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ch.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/ 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 390713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/ 2024-04-17T10:10:43.387 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/ 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 88881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/ 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 117481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 173017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dc395x.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/dmx3191d.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/ 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 267577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/esp_scsi.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/ 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 95329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 194017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fdomain.ko 2024-04-17T10:10:43.388 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/ 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 366529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/fnic/fnic.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 251505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hpsa.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hptiop.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 57097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/hv_storvsc.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/imm.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/initio.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 327841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ipr.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 129785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ips.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/ 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 397785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/isci/isci.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 63121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/ 2024-04-17T10:10:43.389 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 386849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libfc/libfc.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 178177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 64809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/ 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 274009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/libsas/libsas.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/ 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 2736001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/ 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 88529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 389177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 104769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/megaraid.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/ 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 836945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-17T10:10:43.390 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/ 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 158729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 96537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/mvumi.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/ 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 101625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/ 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 443745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 152033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/pmcraid.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ppa.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla1280.ko 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/ 2024-04-17T10:10:43.391 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 2324809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 103601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/ 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 697337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/qlogicfas408.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/raid_class.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 322865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_debug.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 219857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 357153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 121185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 93481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/ses.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 138121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/st.ko 2024-04-17T10:10:43.392 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 57401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/stex.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/ 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 205025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/virtio_scsi.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/wd719x.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/scsi/xen-scsifront.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 174913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ssb/ssb.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/ 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/ 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 920169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/ 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-17T10:10:43.393 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_file.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_iblock.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1170033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_mod.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 51897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_pscsi.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 149721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/target_core_user.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/ 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 90057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/ 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/ 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/ 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-17T10:10:43.394 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_powerclamp.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-17T10:10:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 99313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 69729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/moxa.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/mxser.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 134401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_gsm.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/n_hdlc.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/nozomi.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/ 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/ 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/8250_dw.ko 2024-04-17T10:10:43.396 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/altera_jtaguart.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/altera_uart.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/arc_uart.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/fsl_lpuart.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/ 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 120577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/rp2.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/tty/serial/sc16is7xx.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/ 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 693081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-17T10:10:43.397 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/ 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_aec.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_cif.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_dmem_genirq.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_mf624.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_netx.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pci_generic.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_pdrv_genirq.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/uio/uio_sercos3.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/ 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/c67x00/ 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 90321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/c67x00/c67x00.ko 2024-04-17T10:10:43.398 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 189209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_npcm.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/ 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 101545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-acm.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 63073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 56721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usblp.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 107417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/class/usbtmc.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/common/ 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/common/ulpi.ko 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc2/ 2024-04-17T10:10:43.399 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc2/dwc2_pci.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/ 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3-haps.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3-pci.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 574593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/dwc3/dwc3.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/ 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/ 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 72249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/u_ether.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/u_serial.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 131345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2024-04-17T10:10:43.400 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 146617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 101249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 71257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 84937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/ 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ether.ko 2024-04-17T10:10:43.401 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_hid.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_printer.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_serial.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/g_zero.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 204049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/libcomposite.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/ 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/bdc/ 2024-04-17T10:10:43.402 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 182561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/goku_udc.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/gr_udc.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 69313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/mv_udc.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 90313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/net2272.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 90361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/net2280.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 94841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/pch_udc.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 80521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 87465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 95441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 195969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/gadget/udc/udc-core.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/ 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/bcma-hcd.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 108321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/isp116x-hcd.ko 2024-04-17T10:10:43.403 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 104377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/oxu210hp-hcd.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 102273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/r8a66597-hcd.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/sl811_cs.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/ssb-hcd.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/host/xhci-plat-hcd.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/ 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/mdc800.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/image/microtek.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/adutux.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/appledisplay.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/cypress_cy7c63.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/cytherm.ko 2024-04-17T10:10:43.404 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ehset.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi26.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/emi62.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ezusb.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/idmouse.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/iowarrior.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/ldusb.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/legousbtower.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/lvstest.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/ 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usb3503.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usblcd.ko 2024-04-17T10:10:43.405 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 84265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/usbtest.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/uss720.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/misc/yurex.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/mon/ 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 91529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/mon/usbmon.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/musb/ 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 396209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/musb/musb_hdrc.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/ 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-generic.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-isp1301.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/phy/phy-tahvo.ko 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ 2024-04-17T10:10:43.406 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/aircable.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ark3116.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ch341.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 79681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cp210x.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cyberjack.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 58329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/empeg.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/f81232.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 188609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 152305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 142833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/io_ti.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 69217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipaq.ko 2024-04-17T10:10:43.407 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ipw.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ir-usb.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 122329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mct_u232.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/metro-usb.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 79681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7720.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 89969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mos7840.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/mxuport.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/navman.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/omninet.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/opticon.ko 2024-04-17T10:10:43.408 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 193857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/option.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/oti6858.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/pl2303.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcaux.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/qcserial.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/quatech2.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/safe_serial.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/sierra.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ssu100.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/symbolserial.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70497 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_debug.ko 2024-04-17T10:10:43.409 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 151137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/usbserial.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/visor.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/whiteheat.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/wishbone-serial.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 64049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/uas.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-17T10:10:43.410 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-karma.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 201193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/storage/usb-storage.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/ 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 76041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 122321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/ 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/ 2024-04-17T10:10:43.411 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 206921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 134265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 93673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/ 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 138721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 71281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_net.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 107905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/vhost/vhost_scsi.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/ 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/ 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/88pm860x_bl.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/aat2870_bl.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp5520_bl.ko 2024-04-17T10:10:43.412 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp8860_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 51745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/adp8870_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/apple_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/as3711_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/bd6107.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/da903x_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/da9052_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/gpio_backlight.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/kb3886_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lcd.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lm3533_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lm3639_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/lv5207lp.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/max8925_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/pandora_bl.ko 2024-04-17T10:10:43.413 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/pcf50633-backlight.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/video/backlight/wm831x_bl.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/ 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2482.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/ds2490.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/matrox_w1.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/masters/w1-gpio.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/ 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-17T10:10:43.414 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 129433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/w1/wire.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/acquirewdt.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/advantechwdt.ko 2024-04-17T10:10:43.415 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/cpu5wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9052_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9055_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/da9063_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/dw_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/eurotechwdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/hpwdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/i6300esb.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ib700wdt.ko 2024-04-17T10:10:43.416 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ibmasr.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/it87_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/kempld_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/machzwd.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/mena21_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/menf21bmc_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/nv_tco.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/of_xilinx_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pc87413_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/retu_wdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc60xxwdt.ko 2024-04-17T10:10:43.417 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_epx_c3.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sc1200wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/smsc37b787_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/softdog.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/twl4030_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/via_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wafer5823wdt.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wdt_pci.ko 2024-04-17T10:10:43.418 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wm831x_wdt.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/wm8350_wdt.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/watchdog/xen_wdt.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/ 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-evtchn.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntalloc.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-gntdev.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/ 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 213913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-privcmd.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xen-scsiback.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/ 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ 2024-04-17T10:10:43.419 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/ 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 192617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/9p/9p.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/adfs/ 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 78153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/adfs/adfs.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/ 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 197849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/affs/affs.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/afs/ 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1261113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/afs/kafs.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/autofs/ 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 127377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/autofs/autofs4.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/ 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 80521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/befs/befs.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/bfs/ 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/bfs/bfs.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/binfmt_misc.ko 2024-04-17T10:10:43.420 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 4489593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/btrfs/btrfs.ko 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 376857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cachefiles/cachefiles.ko 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1586089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ceph/ceph.ko 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 130073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/coda/coda.ko 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/configfs/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 139081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/configfs/configfs.ko 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/cramfs/cramfs.ko 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 813225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/dlm/dlm.ko 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/efs/ 2024-04-17T10:10:43.421 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/efs/efs.ko 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/ 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1906817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/f2fs/f2fs.ko 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/ 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fat/msdos.ko 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/freevxfs/ 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/freevxfs/freevxfs.ko 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/ 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/fuse/cuse.ko 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/ 2024-04-17T10:10:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1486521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/gfs2/gfs2.ko 2024-04-17T10:10:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/ 2024-04-17T10:10:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 183305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfs/hfs.ko 2024-04-17T10:10:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/ 2024-04-17T10:10:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 282097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hfsplus/hfsplus.ko 2024-04-17T10:10:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hpfs/ 2024-04-17T10:10:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 249009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/hpfs/hpfs.ko 2024-04-17T10:10:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 93033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/isofs/isofs.ko 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 533073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/jfs/jfs.ko 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 361369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/lockd/lockd.ko 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 112609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/minix/minix.ko 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 591585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/netfs/netfs.ko 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 102561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/ 2024-04-17T10:10:43.478 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 74353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/ 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 147689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1320249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfs.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 82105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv2.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 151641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv3.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 2518505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs/nfsv4.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/ 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/grace.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfs_common/nfs_acl.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/ 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 2024049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nfsd/nfsd.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/ 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 734025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nilfs2/nilfs2.ko 2024-04-17T10:10:43.479 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/ 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-celtic.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-centeuro.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-croatian.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-cyrillic.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-gaelic.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-greek.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-iceland.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-inuit.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-roman.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-romanian.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/mac-turkish.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ascii.ko 2024-04-17T10:10:43.480 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1250.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1251.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp1255.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp737.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp775.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp850.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp852.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp855.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp857.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp860.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp861.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp862.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp863.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp864.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp865.ko 2024-04-17T10:10:43.481 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp866.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp869.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp874.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 95625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp932.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 144209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp936.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 167137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp949.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 119513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_cp950.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_euc-jp.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-1.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-13.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-14.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-15.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-2.ko 2024-04-17T10:10:43.482 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-3.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-4.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-5.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-6.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-7.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_iso8859-9.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-r.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-ru.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_koi8-u.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/nls/nls_utf8.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 302073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-17T10:10:43.483 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 548169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 3245617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 22113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/omfs/ 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 67489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/omfs/omfs.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/ 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 462873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/overlayfs/overlay.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 67977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/pstore/ramoops.ko 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx4/ 2024-04-17T10:10:43.484 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx4/qnx4.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx6/ 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 56097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/qnx6/qnx6.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/ 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_tree.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_v1.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/quota/quota_v2.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/ 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 684841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/reiserfs/reiserfs.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/ 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/romfs/romfs.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/ 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/ 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 3319041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/client/cifs.ko 2024-04-17T10:10:43.485 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/ 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_arc4.ko 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/smb/common/cifs_md4.ko 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/ 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 164185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/squashfs/squashfs.ko 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/ 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 115977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/sysv/sysv.ko 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/ 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 310017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/udf/udf.ko 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 198233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/ufs/ufs.ko 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/ 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 5192705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/fs/xfs/xfs.ko 2024-04-17T10:10:43.486 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/ 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/locking/ 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 97625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/locking/locktorture.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/time/ 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/time/test_udelay.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 72665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/kernel/torture.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc-itu-t.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc7.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crc8.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/ 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libarc4.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/crypto/libdes.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/interval_tree_test.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/libcrc32c.ko 2024-04-17T10:10:43.487 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lru_cache.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/ 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4_compress.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/lz4/lz4hc_compress.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/ 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7281 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/math/cordic.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/memory-notifier-error-inject.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/notifier-error-inject.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/percpu_test.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/pm-notifier-error-inject.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/ 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 152025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/raid6/raid6_pq.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/rbtree_test.ko 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/ 2024-04-17T10:10:43.488 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test-kstrtox.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 4912441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_bpf.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 61169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_firmware.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7121 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_module.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/test_user_copy.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_bm.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_fsm.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/ts_kmp.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/xz/ 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/lib/xz/xz_dec_test.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/ 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/mm/hwpoison-inject.ko 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ 2024-04-17T10:10:43.489 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/ 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/6lowpan.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_dest.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_fragment.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_hop.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_mobility.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_routing.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/6lowpan/nhc_udp.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/ 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/garp.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/mrp.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/p8022.ko 2024-04-17T10:10:43.490 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/psnap.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13537 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/802/stp.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/ 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 106793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/8021q/8021q.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/ 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 236793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_fd.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_rdma.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/9p/9pnet_virtio.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/ 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 879697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/batman-adv/batman-adv.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/ 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 84761 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/br_netfilter.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1030873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/bridge.ko 2024-04-17T10:10:43.491 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9569 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-17T10:10:43.492 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70529 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/ebtables.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/ 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1243505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ceph/libceph.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/ 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 153801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/core/pktgen.ko 2024-04-17T10:10:43.493 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/ 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 244889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_diag.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 71033 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv4.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 74993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dccp/dccp_ipv6.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/ 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 507705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/dsa_core.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_brcm.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_dsa.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_none.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/dsa/tag_trailer.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/hsr/ 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 128873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/hsr/hsr.ko 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ 2024-04-17T10:10:43.494 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 357425 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 59393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ah4.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/esp4.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/fou.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/gre.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/inet_diag.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 75489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_gre.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 77721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_tunnel.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ip_vti.ko 2024-04-17T10:10:43.495 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipcomp.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/ipip.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50169 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ip_tables.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_filter.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-17T10:10:43.496 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_bic.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_dctcp.ko 2024-04-17T10:10:43.497 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_diag.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_highspeed.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_htcp.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11665 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_hybla.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_illinois.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11737 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_lp.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_scalable.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_vegas.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_veno.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_westwood.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tcp_yeah.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/tunnel4.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_diag.ko 2024-04-17T10:10:43.498 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 54441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/udp_tunnel.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ah6.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 47433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/esp6.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/fou6.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 103177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_gre.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 118881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_tunnel.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 71089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ip6_vti.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/ipcomp6.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/mip6.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-17T10:10:43.499 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-17T10:10:43.500 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 29817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 94089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/sit.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 30297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/tunnel6.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/ 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 106369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/key/af_key.ko 2024-04-17T10:10:43.501 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/ 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 153217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_core.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_eth.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ip6.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_netlink.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 66241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/l2tp/l2tp_ppp.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/lapb/ 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 63345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/lapb/lapb.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/ 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 332889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/llc/llc2.ko 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/ 2024-04-17T10:10:43.502 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 388873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mac802154/mac802154.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/ 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/mpls/mpls_gso.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 147745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 124041 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 124345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 126681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 126473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 137833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-17T10:10:43.503 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 72353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 133681 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 139073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 138265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 136177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 139897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 534113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20953 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24577 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-17T10:10:43.504 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12649 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15977 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12809 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 55217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conncount.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 496817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 134377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24513 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-17T10:10:43.505 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 140801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 79873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 117201 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 46105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_log_syslog.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 118369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14185 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-17T10:10:43.506 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13345 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_irc.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_sip.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 807985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nf_tables.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53049 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_cttimeout.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_log.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 65145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_chain_nat.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_compat.ko 2024-04-17T10:10:43.507 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 50657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_ct.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11081 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fib.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_hash.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_limit.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17921 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_log.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_masq.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_nat.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16473 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_numgen.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13945 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_osf.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_queue.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_redir.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject.ko 2024-04-17T10:10:43.508 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_reject_inet.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15057 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_socket.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 24993 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_synproxy.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17641 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/nft_tproxy.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 108393 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/x_tables.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_AUDIT.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 21865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_CT.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_DSCP.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HL.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15745 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_HMARK.ko 2024-04-17T10:10:43.509 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45217 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15937 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LED.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13017 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_LOG.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NETMAP.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10857 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFLOG.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15361 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23089 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_RATEEST.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16689 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_SECMARK.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18985 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20697 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TEE.ko 2024-04-17T10:10:43.510 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TPROXY.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_TRACE.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15289 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_addrtype.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16633 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_bpf.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cgroup.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cluster.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_comment.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11961 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connbytes.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlabel.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connlimit.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_connmark.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14209 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_conntrack.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_cpu.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dccp.ko 2024-04-17T10:10:43.511 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_devgroup.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10257 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_dscp.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12657 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ecn.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_esp.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 52817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hashlimit.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_helper.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_hl.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipcomp.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_iprange.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_ipvs.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19193 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_l2tp.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9137 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_length.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11521 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_limit.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mac.ko 2024-04-17T10:10:43.512 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_mark.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_multiport.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 16161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nat.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_nfacct.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_osf.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11561 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_owner.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_physdev.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_pkttype.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13553 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_policy.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_quota.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 11881 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_rateest.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9273 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_realm.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_recent.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_sctp.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36489 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_set.ko 2024-04-17T10:10:43.513 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 19777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_socket.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10753 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_state.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10849 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_statistic.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_string.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpmss.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18449 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_tcpudp.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12441 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_time.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netfilter/xt_u32.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/ 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 17233 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/netlink/netlink_diag.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/ 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 14073 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/nsh/nsh.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/ 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 488825 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/openvswitch.ko 2024-04-17T10:10:43.514 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12009 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-geneve.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 10873 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-gre.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/openvswitch/vport-vxlan.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/ 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15929 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/packet/af_packet_diag.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/ 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 142025 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/phonet.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 68817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/phonet/pn_pep.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/ 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 318617 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 202305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_rdma.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 85105 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rds/rds_tcp.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rxrpc/ 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1027897 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/rxrpc/rxrpc.ko 2024-04-17T10:10:43.515 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/ 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39001 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_csum.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 28721 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_gact.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_mirred.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 27593 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_nat.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 53465 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_pedit.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38705 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_police.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23969 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_simple.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43417 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_skbedit.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 45673 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/act_vlan.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32129 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_basic.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 41457 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_bpf.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23225 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_cgroup.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 37305 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_flow.ko 2024-04-17T10:10:43.516 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39601 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_fw.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 49321 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_route.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 79145 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/cls_u32.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8713 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_cmp.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 15385 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_ipset.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 48785 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_meta.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9241 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_nbyte.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13369 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_text.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 8377 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/em_u32.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_choke.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 25097 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_codel.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43481 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_drr.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 43153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 39177 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_fq_codel.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 42889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_gred.ko 2024-04-17T10:10:43.517 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 62817 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hfsc.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26585 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_hhf.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 81833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_htb.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33545 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_ingress.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 36625 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12297 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_mqprio_lib.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33065 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_multiq.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 44793 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_netem.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 32833 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_pie.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 13505 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_plug.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 31801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_prio.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 60249 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_qfq.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 40889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_red.ko 2024-04-17T10:10:43.518 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 33433 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfb.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 38329 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_sfq.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 35777 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_tbf.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 34161 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sched/sch_teql.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/ 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1187313 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23913 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sctp/sctp_diag.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/ 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/ 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 432409 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70113 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1940889 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/sunrpc.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/ 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 890729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/ 2024-04-17T10:10:43.519 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 1122337 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12153 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/tipc/tipc_diag.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/ 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 18801 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/unix/unix_diag.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/ 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 105609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 90265 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 123769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 12905 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 20401 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/x25/ 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 165841 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/x25/x25.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/ 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 26729 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_algo.ko 2024-04-17T10:10:43.520 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23353 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 131865 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/kernel/net/xfrm/xfrm_user.ko 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 644769 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 628542 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.alias.bin 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 7706 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 23609 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.alias.bin 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 9941 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.bin 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 70990 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.builtin.modinfo 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 146959 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 232835 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.dep.bin 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 210 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.devname 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 72602 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.order 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 879 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.softdep 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 258450 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 313019 2024-04-11 18:50 ./lib/modules/6.9.0-rc3-g3f60ed14f7c1/modules.symbols.bin 2024-04-17T10:10:43.521 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/ 2024-04-17T10:10:43.522 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/ 2024-04-17T10:10:43.522 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/doc/ 2024-04-17T10:10:43.522 INFO:teuthology.orchestra.run.smithi086.stdout:drwxr-xr-x root/root 0 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/ 2024-04-17T10:10:43.522 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 174 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/changelog.Debian.gz 2024-04-17T10:10:43.522 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r-- root/root 691 2024-04-11 18:50 ./usr/share/doc/linux-image-6.9.0-rc3-g3f60ed14f7c1/copyright 2024-04-17T10:10:43.526 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:43.526 INFO:teuthology.task.kernel:searching for kernel 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:43.527 DEBUG:teuthology.orchestra.run.smithi086:> egrep '(submenu|menuentry.*6.9.0-rc3-g3f60ed14f7c1).*{' /boot/grub/grub.cfg 2024-04-17T10:10:43.533 INFO:teuthology.orchestra.run.smithi086.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T10:10:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: menuentry 'Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-rc3-g3f60ed14f7c1-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T10:10:43.533 INFO:teuthology.orchestra.run.smithi086.stdout: menuentry 'Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.9.0-rc3-g3f60ed14f7c1-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2024-04-17T10:10:43.534 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2024-04-17T10:10:43.534 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:43.534 DEBUG:teuthology.orchestra.run.smithi086:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2024-04-17T10:10:43.576 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi086.front.sepia.ceph.com to complete... 2024-04-17T10:10:43.961 INFO:teuthology.orchestra.run.smithi086.stderr:Sourcing file `/etc/default/grub' 2024-04-17T10:10:43.962 INFO:teuthology.orchestra.run.smithi086.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-17T10:10:43.986 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-04-17T10:10:44.168 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:44.182 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:10:44.390 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-17T10:10:44.395 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-17T10:10:44.433 INFO:teuthology.orchestra.run.smithi086.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-17T10:10:44.438 INFO:teuthology.orchestra.run.smithi086.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-17T10:10:44.578 INFO:teuthology.orchestra.run.smithi086.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-17T10:10:44.578 INFO:teuthology.orchestra.run.smithi086.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-17T10:10:44.578 INFO:teuthology.orchestra.run.smithi086.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-17T10:10:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-04-17T10:10:45.730 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-04-17T10:11:09.518 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-17T10:11:09.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T10:11:09.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:11:15.736 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-04-17T10:11:15.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-17T10:11:15.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:11:44.031 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2024-04-17T10:11:50.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-04-17T10:11:53.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T10:11:53.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:11:59.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-17T10:11:59.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:12:17.827 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-04-17T10:12:29.838 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-17T10:12:29.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:12:30.226 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-04-17T10:12:31.553 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-17T10:12:31.553 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-17T10:12:31.554 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-04-17T10:12:31.606 INFO:teuthology.orchestra.run.smithi086.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:12:31.606 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:12:31.606 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-17T10:12:31.607 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-17T10:12:32.608 DEBUG:teuthology.parallel:result is None 2024-04-17T10:12:53.037 DEBUG:teuthology.orchestra.remote:timed out 2024-04-17T10:13:05.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T10:13:05.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:13:05.261 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-17T10:13:05.902 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T10:13:05.902 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g3f60ed14f7c1"... 2024-04-17T10:13:05.903 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-04-17T10:13:05.954 INFO:teuthology.orchestra.run.smithi067.stdout:6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:13:05.954 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g3f60ed14f7c1 vs 6.9.0-rc3-g3f60ed14f7c1 2024-04-17T10:13:05.954 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-17T10:13:05.954 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-17T10:13:06.955 DEBUG:teuthology.parallel:result is None 2024-04-17T10:13:06.956 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-17T10:13:06.963 INFO:teuthology.task.internal:Creating test directory... 2024-04-17T10:13:06.964 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-17T10:13:06.967 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-17T10:13:06.976 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-17T10:13:06.988 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-17T10:13:06.994 INFO:teuthology.task.internal:Creating archive directory... 2024-04-17T10:13:06.994 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-17T10:13:07.017 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-17T10:13:07.046 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-17T10:13:07.052 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-17T10:13:07.053 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-17T10:13:07.081 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-17T10:13:07.156 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T10:13:07.181 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T10:13:07.196 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T10:13:07.223 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-17T10:13:07.225 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-17T10:13:07.235 INFO:teuthology.task.internal:Configuring sudo... 2024-04-17T10:13:07.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-17T10:13:07.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-17T10:13:07.260 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-17T10:13:07.271 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-17T10:13:07.272 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-17T10:13:07.301 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-17T10:13:07.309 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T10:13:07.359 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T10:13:07.411 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:13:07.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-17T10:13:07.483 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-17T10:13:07.492 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-17T10:13:07.542 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:13:07.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-17T10:13:07.611 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-04-17T10:13:07.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-04-17T10:13:07.739 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-17T10:13:07.753 INFO:teuthology.task.internal:Starting timer... 2024-04-17T10:13:07.753 INFO:teuthology.run_tasks:Running task pcp... 2024-04-17T10:13:07.769 INFO:teuthology.run_tasks:Running task selinux... 2024-04-17T10:13:07.779 DEBUG:teuthology.task.selinux:Excluding smithi067: OS 'ubuntu' does not support SELinux 2024-04-17T10:13:07.779 DEBUG:teuthology.task.selinux:Excluding smithi086: OS 'ubuntu' does not support SELinux 2024-04-17T10:13:07.779 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-17T10:13:07.779 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-17T10:13:07.779 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-17T10:13:07.780 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-17T10:13:07.789 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-17T10:13:07.795 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-17T10:13:07.924 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-17T10:13:07.938 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-17T10:13:07.939 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-17T10:20:53.696 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-04-17T10:20:53.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T10:20:53.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:20:53.825 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-04-17T10:20:53.948 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-04-17T10:20:53.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-17T10:20:53.949 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-17T10:20:54.029 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-04-17T10:20:54.165 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-04-17T10:20:54.165 INFO:teuthology.run_tasks:Running task clock... 2024-04-17T10:20:54.176 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-17T10:20:54.176 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-17T10:20:54.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T10:20:54.180 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-17T10:20:54.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-17T10:20:54.218 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-17T10:20:54.219 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Command line: ntpd -gq 2024-04-17T10:20:54.219 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: ---------------------------------------------------- 2024-04-17T10:20:54.219 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: ntp-4 is maintained by Network Time Foundation, 2024-04-17T10:20:54.219 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-17T10:20:54.219 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: corporation. Support and training for ntp-4 are 2024-04-17T10:20:54.219 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: available at https://www.nwtime.org/support 2024-04-17T10:20:54.219 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: ---------------------------------------------------- 2024-04-17T10:20:54.220 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: proto: precision = 0.042 usec (-24) 2024-04-17T10:20:54.220 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: basedate set to 2022-02-04 2024-04-17T10:20:54.220 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: gps base set to 2022-02-06 (week 2196) 2024-04-17T10:20:54.220 INFO:teuthology.orchestra.run.smithi067.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T10:20:54.220 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T10:20:54.221 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: restrict ::: KOD does nothing without LIMITED. 2024-04-17T10:20:54.221 INFO:teuthology.orchestra.run.smithi067.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-17T10:20:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Listen and drop on 0 v6wildcard [::]:123 2024-04-17T10:20:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-17T10:20:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Listen normally on 2 lo 127.0.0.1:123 2024-04-17T10:20:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Listen normally on 3 enp3s0f1 172.21.15.67:123 2024-04-17T10:20:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Listen normally on 4 lo [::1]:123 2024-04-17T10:20:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:fc0b%5]:123 2024-04-17T10:20:54.222 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:20:54 ntpd[17183]: Listening on routing socket on fd #22 for interface updates 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Command line: ntpd -gq 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: ---------------------------------------------------- 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: ntp-4 is maintained by Network Time Foundation, 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: corporation. Support and training for ntp-4 are 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: available at https://www.nwtime.org/support 2024-04-17T10:20:54.245 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: ---------------------------------------------------- 2024-04-17T10:20:54.246 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: proto: precision = 0.044 usec (-24) 2024-04-17T10:20:54.246 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: basedate set to 2022-02-04 2024-04-17T10:20:54.246 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: gps base set to 2022-02-06 (week 2196) 2024-04-17T10:20:54.246 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T10:20:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-17T10:20:54.247 INFO:teuthology.orchestra.run.smithi086.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-17T10:20:54.247 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: restrict ::: KOD does nothing without LIMITED. 2024-04-17T10:20:54.247 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Listen and drop on 0 v6wildcard [::]:123 2024-04-17T10:20:54.247 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-17T10:20:54.248 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Listen normally on 2 lo 127.0.0.1:123 2024-04-17T10:20:54.248 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Listen normally on 3 enp3s0f1 172.21.15.86:123 2024-04-17T10:20:54.248 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Listen normally on 4 lo [::1]:123 2024-04-17T10:20:54.248 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d809%5]:123 2024-04-17T10:20:54.248 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:20:54 ntpd[17214]: Listening on routing socket on fd #22 for interface updates 2024-04-17T10:20:55.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:55 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:55.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:55 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:55 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:55.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:55 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:56 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:56 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:56.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:56 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:56.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:56 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:56.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:56 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:56.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:56 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:56.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:56.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:56.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:56.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:56.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:56.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:56.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:56.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:56 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:57 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:57 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:57 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:57.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:57 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:57.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:57 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:57.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:57 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:57.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:57 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:57.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:57 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:57.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:57 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:57.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:57 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:57.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:57 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:57.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:57 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:58.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:58 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:58 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:58 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:58 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:58 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:58 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:58.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:58.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:58.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:58.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:58.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:58.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:58 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:59.221 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:59.222 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:59.222 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:59.222 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:59.222 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:59.222 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:59.224 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:20:59 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:20:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:59 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:59.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:59 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:59 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:20:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:59 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:20:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:59 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:20:59.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:20:59 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:00 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:00 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:00 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:00 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:00 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:00.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:00 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:00.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:00.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:00 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:01 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:01 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:01 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:01 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:01.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:01 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:01.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:01 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:01.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:01 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:02.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:02.219 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:02.220 INFO:teuthology.orchestra.run.smithi067.stdout:17 Apr 10:21:02 ntpd[17183]: ntpd: time slew -0.002361 s 2024-04-17T10:21:02.220 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew -0.002361s 2024-04-17T10:21:02.221 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-17T10:21:02.221 INFO:teuthology.orchestra.run.smithi067.stderr:17 Apr 10:21:02 ntpd[17183]: can't open /var/log/ntpstats/loopstats.20240417: Permission denied 2024-04-17T10:21:02.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:02.245 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: can't open /var/log/ntpstats/rawstats.20240417: Permission denied 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: can't open /var/log/ntpstats/peerstats.20240417: Permission denied 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:17 Apr 10:21:02 ntpd[17214]: ntpd: time slew -0.001299 s 2024-04-17T10:21:02.246 INFO:teuthology.orchestra.run.smithi086.stdout:ntpd: time slew -0.001299s 2024-04-17T10:21:02.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-17T10:21:02.247 INFO:teuthology.orchestra.run.smithi086.stderr:17 Apr 10:21:02 ntpd[17214]: can't open /var/log/ntpstats/loopstats.20240417: Permission denied 2024-04-17T10:21:02.362 INFO:teuthology.orchestra.run.smithi086.stdout: remote refid st t when poll reach delay offset jitter 2024-04-17T10:21:02.362 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================== 2024-04-17T10:21:02.362 INFO:teuthology.orchestra.run.smithi086.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.362 INFO:teuthology.orchestra.run.smithi086.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.363 INFO:teuthology.orchestra.run.smithi086.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.363 INFO:teuthology.orchestra.run.smithi086.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.456 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2024-04-17T10:21:02.457 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2024-04-17T10:21:02.457 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.457 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.457 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.457 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-17T10:21:02.458 INFO:teuthology.run_tasks:Running task install... 2024-04-17T10:21:02.467 DEBUG:teuthology.task.install:project ceph 2024-04-17T10:21:02.467 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec'}} 2024-04-17T10:21:02.468 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec'} 2024-04-17T10:21:02.468 INFO:teuthology.task.install:Using flavor: default 2024-04-17T10:21:02.475 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-17T10:21:02.475 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-17T10:21:02.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-key list | grep Ceph 2024-04-17T10:21:02.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-key list | grep Ceph 2024-04-17T10:21:02.580 INFO:teuthology.orchestra.run.smithi086.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-17T10:21:02.609 INFO:teuthology.orchestra.run.smithi067.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-17T10:21:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-17T10:21:02.631 INFO:teuthology.orchestra.run.smithi086.stdout:uid [ unknown] Ceph.com (release key) 2024-04-17T10:21:02.632 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-17T10:21:02.632 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-17T10:21:02.633 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec 2024-04-17T10:21:02.669 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-17T10:21:02.669 INFO:teuthology.orchestra.run.smithi067.stdout:uid [ unknown] Ceph.com (release key) 2024-04-17T10:21:02.669 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-17T10:21:02.669 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-17T10:21:02.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec 2024-04-17T10:21:02.786 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default/ 2024-04-17T10:21:02.786 INFO:teuthology.task.install.deb:Package version is 19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:21:02.896 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default/ 2024-04-17T10:21:02.896 INFO:teuthology.task.install.deb:Package version is 19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:21:02.960 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:21:02.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-17T10:21:02.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo apt-get update 2024-04-17T10:21:03.006 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:21:03.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-17T10:21:03.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo apt-get update 2024-04-17T10:21:03.189 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-17T10:21:03.212 INFO:teuthology.orchestra.run.smithi086.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy InRelease 2024-04-17T10:21:03.219 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-17T10:21:03.219 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-17T10:21:03.229 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-17T10:21:03.233 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-17T10:21:03.249 INFO:teuthology.orchestra.run.smithi067.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-17T10:21:03.254 INFO:teuthology.orchestra.run.smithi086.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-17T10:21:03.262 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy InRelease 2024-04-17T10:21:03.275 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,398 B] 2024-04-17T10:21:03.283 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-17T10:21:03.286 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-17T10:21:03.305 INFO:teuthology.orchestra.run.smithi067.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-17T10:21:03.307 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-17T10:21:03.326 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-17T10:21:03.359 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,398 B] 2024-04-17T10:21:03.359 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-17T10:21:03.441 INFO:teuthology.orchestra.run.smithi086.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-17T10:21:03.773 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 28.9 kB in 1s (50.3 kB/s) 2024-04-17T10:21:03.828 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 28.9 kB in 1s (50.3 kB/s) 2024-04-17T10:21:04.740 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-04-17T10:21:04.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1628-g97ae6cc9-1jammy cephadm=19.0.0-1628-g97ae6cc9-1jammy ceph-mds=19.0.0-1628-g97ae6cc9-1jammy ceph-mgr=19.0.0-1628-g97ae6cc9-1jammy ceph-common=19.0.0-1628-g97ae6cc9-1jammy ceph-fuse=19.0.0-1628-g97ae6cc9-1jammy ceph-test=19.0.0-1628-g97ae6cc9-1jammy ceph-volume=19.0.0-1628-g97ae6cc9-1jammy radosgw=19.0.0-1628-g97ae6cc9-1jammy python3-rados=19.0.0-1628-g97ae6cc9-1jammy python3-rgw=19.0.0-1628-g97ae6cc9-1jammy python3-cephfs=19.0.0-1628-g97ae6cc9-1jammy python3-rbd=19.0.0-1628-g97ae6cc9-1jammy libcephfs2=19.0.0-1628-g97ae6cc9-1jammy libcephfs-dev=19.0.0-1628-g97ae6cc9-1jammy librados2=19.0.0-1628-g97ae6cc9-1jammy librbd1=19.0.0-1628-g97ae6cc9-1jammy rbd-fuse=19.0.0-1628-g97ae6cc9-1jammy python3-cephfs=19.0.0-1628-g97ae6cc9-1jammy cephfs-shell=19.0.0-1628-g97ae6cc9-1jammy cephfs-top=19.0.0-1628-g97ae6cc9-1jammy cephfs-mirror=19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:21:04.828 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-04-17T10:21:04.834 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-17T10:21:04.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-1628-g97ae6cc9-1jammy cephadm=19.0.0-1628-g97ae6cc9-1jammy ceph-mds=19.0.0-1628-g97ae6cc9-1jammy ceph-mgr=19.0.0-1628-g97ae6cc9-1jammy ceph-common=19.0.0-1628-g97ae6cc9-1jammy ceph-fuse=19.0.0-1628-g97ae6cc9-1jammy ceph-test=19.0.0-1628-g97ae6cc9-1jammy ceph-volume=19.0.0-1628-g97ae6cc9-1jammy radosgw=19.0.0-1628-g97ae6cc9-1jammy python3-rados=19.0.0-1628-g97ae6cc9-1jammy python3-rgw=19.0.0-1628-g97ae6cc9-1jammy python3-cephfs=19.0.0-1628-g97ae6cc9-1jammy python3-rbd=19.0.0-1628-g97ae6cc9-1jammy libcephfs2=19.0.0-1628-g97ae6cc9-1jammy libcephfs-dev=19.0.0-1628-g97ae6cc9-1jammy librados2=19.0.0-1628-g97ae6cc9-1jammy librbd1=19.0.0-1628-g97ae6cc9-1jammy rbd-fuse=19.0.0-1628-g97ae6cc9-1jammy python3-cephfs=19.0.0-1628-g97ae6cc9-1jammy cephfs-shell=19.0.0-1628-g97ae6cc9-1jammy cephfs-top=19.0.0-1628-g97ae6cc9-1jammy cephfs-mirror=19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:21:04.925 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-17T10:21:05.000 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2024-04-17T10:21:05.001 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2024-04-17T10:21:05.101 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-17T10:21:05.101 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-17T10:21:05.171 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T10:21:05.171 INFO:teuthology.orchestra.run.smithi086.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T10:21:05.171 INFO:teuthology.orchestra.run.smithi086.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T10:21:05.171 INFO:teuthology.orchestra.run.smithi086.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T10:21:05.171 INFO:teuthology.orchestra.run.smithi086.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-17T10:21:05.172 INFO:teuthology.orchestra.run.smithi086.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-17T10:21:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T10:21:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:The following additional packages will be installed: 2024-04-17T10:21:05.172 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-17T10:21:05.172 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-17T10:21:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T10:21:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-17T10:21:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:Suggested packages: 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-17T10:21:05.176 INFO:teuthology.orchestra.run.smithi086.stdout:Recommended packages: 2024-04-17T10:21:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: btrfs-tools 2024-04-17T10:21:05.279 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T10:21:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T10:21:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T10:21:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T10:21:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-17T10:21:05.280 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-17T10:21:05.280 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T10:21:05.281 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-04-17T10:21:05.281 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-17T10:21:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-17T10:21:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T10:21:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-17T10:21:05.282 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-17T10:21:05.283 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-17T10:21:05.284 INFO:teuthology.orchestra.run.smithi067.stdout:Recommended packages: 2024-04-17T10:21:05.285 INFO:teuthology.orchestra.run.smithi067.stdout: btrfs-tools 2024-04-17T10:21:05.325 INFO:teuthology.orchestra.run.smithi086.stdout:The following NEW packages will be installed: 2024-04-17T10:21:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-17T10:21:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-17T10:21:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-17T10:21:05.325 INFO:teuthology.orchestra.run.smithi086.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-17T10:21:05.326 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-17T10:21:05.327 INFO:teuthology.orchestra.run.smithi086.stdout: socat unzip xmlstarlet zip 2024-04-17T10:21:05.440 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-04-17T10:21:05.440 INFO:teuthology.orchestra.run.smithi067.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-17T10:21:05.440 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-17T10:21:05.440 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-17T10:21:05.441 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-17T10:21:05.441 INFO:teuthology.orchestra.run.smithi067.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-17T10:21:05.441 INFO:teuthology.orchestra.run.smithi067.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-17T10:21:05.441 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-17T10:21:05.442 INFO:teuthology.orchestra.run.smithi067.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-17T10:21:05.442 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-17T10:21:05.442 INFO:teuthology.orchestra.run.smithi067.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-17T10:21:05.442 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-17T10:21:05.443 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-17T10:21:05.444 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-17T10:21:05.444 INFO:teuthology.orchestra.run.smithi067.stdout: socat unzip xmlstarlet zip 2024-04-17T10:21:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:0 upgraded, 124 newly installed, 0 to remove and 305 not upgraded. 2024-04-17T10:21:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:Need to get 176 MB of archives. 2024-04-17T10:21:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:After this operation, 732 MB of additional disk space will be used. 2024-04-17T10:21:05.475 INFO:teuthology.orchestra.run.smithi086.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1628-g97ae6cc9-1jammy [3,533 kB] 2024-04-17T10:21:05.521 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-17T10:21:05.533 INFO:teuthology.orchestra.run.smithi067.stdout:0 upgraded, 124 newly installed, 0 to remove and 305 not upgraded. 2024-04-17T10:21:05.533 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 176 MB of archives. 2024-04-17T10:21:05.533 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 732 MB of additional disk space will be used. 2024-04-17T10:21:05.533 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-17T10:21:05.560 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-17T10:21:05.577 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-17T10:21:05.591 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-1628-g97ae6cc9-1jammy [3,533 kB] 2024-04-17T10:21:05.603 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-17T10:21:05.646 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-17T10:21:05.649 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-17T10:21:05.682 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-17T10:21:05.688 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-17T10:21:05.711 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-17T10:21:05.713 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1628-g97ae6cc9-1jammy [3,217 kB] 2024-04-17T10:21:05.767 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-17T10:21:05.771 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-17T10:21:05.785 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-17T10:21:05.788 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-17T10:21:05.789 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-17T10:21:05.795 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-17T10:21:05.797 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-17T10:21:05.799 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-17T10:21:05.800 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-17T10:21:05.801 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-17T10:21:05.818 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-17T10:21:05.821 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-17T10:21:05.823 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-17T10:21:05.823 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-17T10:21:05.824 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-17T10:21:05.839 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-17T10:21:05.839 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-17T10:21:05.839 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-17T10:21:05.840 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-17T10:21:05.840 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-17T10:21:05.850 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-17T10:21:05.858 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-17T10:21:05.859 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-17T10:21:05.861 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-17T10:21:05.862 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-17T10:21:05.862 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-17T10:21:05.878 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-17T10:21:05.879 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-17T10:21:05.879 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-17T10:21:05.880 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-17T10:21:05.881 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-17T10:21:05.889 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1628-g97ae6cc9-1jammy [709 kB] 2024-04-17T10:21:05.898 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-17T10:21:05.898 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-17T10:21:05.904 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-17T10:21:05.904 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-17T10:21:05.905 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-17T10:21:05.918 INFO:teuthology.orchestra.run.smithi067.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-17T10:21:05.919 INFO:teuthology.orchestra.run.smithi067.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-17T10:21:05.921 INFO:teuthology.orchestra.run.smithi067.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-17T10:21:05.922 INFO:teuthology.orchestra.run.smithi067.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-17T10:21:05.923 INFO:teuthology.orchestra.run.smithi067.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-17T10:21:05.926 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1628-g97ae6cc9-1jammy [356 kB] 2024-04-17T10:21:05.937 INFO:teuthology.orchestra.run.smithi067.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-17T10:21:05.938 INFO:teuthology.orchestra.run.smithi086.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1628-g97ae6cc9-1jammy [32.8 kB] 2024-04-17T10:21:05.938 INFO:teuthology.orchestra.run.smithi086.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1628-g97ae6cc9-1jammy [180 kB] 2024-04-17T10:21:05.940 INFO:teuthology.orchestra.run.smithi067.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-17T10:21:05.940 INFO:teuthology.orchestra.run.smithi067.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-17T10:21:05.941 INFO:teuthology.orchestra.run.smithi067.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-17T10:21:05.944 INFO:teuthology.orchestra.run.smithi067.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-17T10:21:05.944 INFO:teuthology.orchestra.run.smithi086.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-17T10:21:05.944 INFO:teuthology.orchestra.run.smithi086.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1628-g97ae6cc9-1jammy [64.8 kB] 2024-04-17T10:21:05.946 INFO:teuthology.orchestra.run.smithi086.stdout:Get:13 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1628-g97ae6cc9-1jammy [330 kB] 2024-04-17T10:21:05.949 INFO:teuthology.orchestra.run.smithi067.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-1628-g97ae6cc9-1jammy [3,217 kB] 2024-04-17T10:21:05.957 INFO:teuthology.orchestra.run.smithi086.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-17T10:21:05.976 INFO:teuthology.orchestra.run.smithi086.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1628-g97ae6cc9-1jammy [6,575 kB] 2024-04-17T10:21:06.008 INFO:teuthology.orchestra.run.smithi086.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-17T10:21:06.011 INFO:teuthology.orchestra.run.smithi067.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-17T10:21:06.012 INFO:teuthology.orchestra.run.smithi067.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-17T10:21:06.013 INFO:teuthology.orchestra.run.smithi067.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-17T10:21:06.014 INFO:teuthology.orchestra.run.smithi067.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-17T10:21:06.015 INFO:teuthology.orchestra.run.smithi067.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-17T10:21:06.039 INFO:teuthology.orchestra.run.smithi067.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-17T10:21:06.041 INFO:teuthology.orchestra.run.smithi067.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-17T10:21:06.042 INFO:teuthology.orchestra.run.smithi067.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-17T10:21:06.068 INFO:teuthology.orchestra.run.smithi067.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-17T10:21:06.068 INFO:teuthology.orchestra.run.smithi067.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-17T10:21:06.069 INFO:teuthology.orchestra.run.smithi067.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-17T10:21:06.070 INFO:teuthology.orchestra.run.smithi067.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-17T10:21:06.070 INFO:teuthology.orchestra.run.smithi067.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-17T10:21:06.071 INFO:teuthology.orchestra.run.smithi067.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-17T10:21:06.087 INFO:teuthology.orchestra.run.smithi067.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-17T10:21:06.088 INFO:teuthology.orchestra.run.smithi067.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-17T10:21:06.092 INFO:teuthology.orchestra.run.smithi067.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-17T10:21:06.107 INFO:teuthology.orchestra.run.smithi067.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-17T10:21:06.109 INFO:teuthology.orchestra.run.smithi067.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-17T10:21:06.110 INFO:teuthology.orchestra.run.smithi067.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-17T10:21:06.114 INFO:teuthology.orchestra.run.smithi067.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-17T10:21:06.117 INFO:teuthology.orchestra.run.smithi067.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-17T10:21:06.117 INFO:teuthology.orchestra.run.smithi067.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-17T10:21:06.118 INFO:teuthology.orchestra.run.smithi067.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-17T10:21:06.128 INFO:teuthology.orchestra.run.smithi067.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-17T10:21:06.133 INFO:teuthology.orchestra.run.smithi067.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-17T10:21:06.134 INFO:teuthology.orchestra.run.smithi067.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-17T10:21:06.149 INFO:teuthology.orchestra.run.smithi067.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-17T10:21:06.150 INFO:teuthology.orchestra.run.smithi067.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-17T10:21:06.151 INFO:teuthology.orchestra.run.smithi067.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-17T10:21:06.151 INFO:teuthology.orchestra.run.smithi067.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-17T10:21:06.153 INFO:teuthology.orchestra.run.smithi067.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-17T10:21:06.154 INFO:teuthology.orchestra.run.smithi067.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-17T10:21:06.162 INFO:teuthology.orchestra.run.smithi067.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-17T10:21:06.173 INFO:teuthology.orchestra.run.smithi067.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-17T10:21:06.173 INFO:teuthology.orchestra.run.smithi067.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-17T10:21:06.176 INFO:teuthology.orchestra.run.smithi067.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-17T10:21:06.182 INFO:teuthology.orchestra.run.smithi086.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-17T10:21:06.191 INFO:teuthology.orchestra.run.smithi086.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-17T10:21:06.192 INFO:teuthology.orchestra.run.smithi067.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-17T10:21:06.203 INFO:teuthology.orchestra.run.smithi067.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-1628-g97ae6cc9-1jammy [709 kB] 2024-04-17T10:21:06.218 INFO:teuthology.orchestra.run.smithi086.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-17T10:21:06.227 INFO:teuthology.orchestra.run.smithi086.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-17T10:21:06.259 INFO:teuthology.orchestra.run.smithi067.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-1628-g97ae6cc9-1jammy [356 kB] 2024-04-17T10:21:06.271 INFO:teuthology.orchestra.run.smithi067.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-1628-g97ae6cc9-1jammy [32.8 kB] 2024-04-17T10:21:06.272 INFO:teuthology.orchestra.run.smithi067.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-1628-g97ae6cc9-1jammy [180 kB] 2024-04-17T10:21:06.278 INFO:teuthology.orchestra.run.smithi067.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-1628-g97ae6cc9-1jammy [64.8 kB] 2024-04-17T10:21:06.281 INFO:teuthology.orchestra.run.smithi067.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-1628-g97ae6cc9-1jammy [330 kB] 2024-04-17T10:21:06.327 INFO:teuthology.orchestra.run.smithi067.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-1628-g97ae6cc9-1jammy [6,575 kB] 2024-04-17T10:21:06.354 INFO:teuthology.orchestra.run.smithi086.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1628-g97ae6cc9-1jammy [112 kB] 2024-04-17T10:21:06.355 INFO:teuthology.orchestra.run.smithi086.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1628-g97ae6cc9-1jammy [467 kB] 2024-04-17T10:21:06.371 INFO:teuthology.orchestra.run.smithi086.stdout:Get:23 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1628-g97ae6cc9-1jammy [24.1 MB] 2024-04-17T10:21:06.391 INFO:teuthology.orchestra.run.smithi086.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-17T10:21:06.867 INFO:teuthology.orchestra.run.smithi067.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-1628-g97ae6cc9-1jammy [112 kB] 2024-04-17T10:21:06.868 INFO:teuthology.orchestra.run.smithi067.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-1628-g97ae6cc9-1jammy [467 kB] 2024-04-17T10:21:06.890 INFO:teuthology.orchestra.run.smithi067.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-1628-g97ae6cc9-1jammy [24.1 MB] 2024-04-17T10:21:06.979 INFO:teuthology.orchestra.run.smithi086.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-17T10:21:06.994 INFO:teuthology.orchestra.run.smithi086.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-17T10:21:07.010 INFO:teuthology.orchestra.run.smithi086.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-17T10:21:07.019 INFO:teuthology.orchestra.run.smithi086.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-17T10:21:07.019 INFO:teuthology.orchestra.run.smithi086.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-17T10:21:07.036 INFO:teuthology.orchestra.run.smithi086.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-17T10:21:07.073 INFO:teuthology.orchestra.run.smithi086.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-17T10:21:07.087 INFO:teuthology.orchestra.run.smithi086.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-17T10:21:07.090 INFO:teuthology.orchestra.run.smithi086.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-17T10:21:07.091 INFO:teuthology.orchestra.run.smithi086.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-17T10:21:07.261 INFO:teuthology.orchestra.run.smithi086.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-17T10:21:07.344 INFO:teuthology.orchestra.run.smithi086.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-17T10:21:07.367 INFO:teuthology.orchestra.run.smithi086.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-17T10:21:07.425 INFO:teuthology.orchestra.run.smithi086.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-17T10:21:07.473 INFO:teuthology.orchestra.run.smithi086.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-17T10:21:07.495 INFO:teuthology.orchestra.run.smithi086.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-17T10:21:07.516 INFO:teuthology.orchestra.run.smithi086.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-17T10:21:07.691 INFO:teuthology.orchestra.run.smithi086.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-17T10:21:07.707 INFO:teuthology.orchestra.run.smithi086.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-17T10:21:07.714 INFO:teuthology.orchestra.run.smithi086.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-17T10:21:07.719 INFO:teuthology.orchestra.run.smithi086.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-17T10:21:07.777 INFO:teuthology.orchestra.run.smithi086.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1628-g97ae6cc9-1jammy [5,076 kB] 2024-04-17T10:21:07.903 INFO:teuthology.orchestra.run.smithi086.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-17T10:21:07.985 INFO:teuthology.orchestra.run.smithi086.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-17T10:21:08.001 INFO:teuthology.orchestra.run.smithi086.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-17T10:21:08.078 INFO:teuthology.orchestra.run.smithi086.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1628-g97ae6cc9-1jammy [243 kB] 2024-04-17T10:21:08.084 INFO:teuthology.orchestra.run.smithi086.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1628-g97ae6cc9-1jammy [124 kB] 2024-04-17T10:21:08.096 INFO:teuthology.orchestra.run.smithi086.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1628-g97ae6cc9-1jammy [1,520 kB] 2024-04-17T10:21:08.186 INFO:teuthology.orchestra.run.smithi086.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-17T10:21:08.195 INFO:teuthology.orchestra.run.smithi086.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1628-g97ae6cc9-1jammy [6,237 kB] 2024-04-17T10:21:08.237 INFO:teuthology.orchestra.run.smithi086.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-17T10:21:08.249 INFO:teuthology.orchestra.run.smithi086.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-17T10:21:08.407 INFO:teuthology.orchestra.run.smithi086.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-17T10:21:08.408 INFO:teuthology.orchestra.run.smithi086.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-17T10:21:08.414 INFO:teuthology.orchestra.run.smithi086.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-17T10:21:08.418 INFO:teuthology.orchestra.run.smithi086.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-17T10:21:08.432 INFO:teuthology.orchestra.run.smithi086.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-17T10:21:08.549 INFO:teuthology.orchestra.run.smithi086.stdout:Get:62 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1628-g97ae6cc9-1jammy [22.7 MB] 2024-04-17T10:21:08.618 INFO:teuthology.orchestra.run.smithi086.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-17T10:21:08.661 INFO:teuthology.orchestra.run.smithi067.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-1628-g97ae6cc9-1jammy [5,076 kB] 2024-04-17T10:21:08.879 INFO:teuthology.orchestra.run.smithi086.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-17T10:21:09.010 INFO:teuthology.orchestra.run.smithi086.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-17T10:21:09.054 INFO:teuthology.orchestra.run.smithi086.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-17T10:21:09.054 INFO:teuthology.orchestra.run.smithi067.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-1628-g97ae6cc9-1jammy [243 kB] 2024-04-17T10:21:09.062 INFO:teuthology.orchestra.run.smithi067.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-1628-g97ae6cc9-1jammy [124 kB] 2024-04-17T10:21:09.066 INFO:teuthology.orchestra.run.smithi067.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-1628-g97ae6cc9-1jammy [1,520 kB] 2024-04-17T10:21:09.144 INFO:teuthology.orchestra.run.smithi086.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-17T10:21:09.153 INFO:teuthology.orchestra.run.smithi086.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-17T10:21:09.166 INFO:teuthology.orchestra.run.smithi086.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-17T10:21:09.195 INFO:teuthology.orchestra.run.smithi067.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-1628-g97ae6cc9-1jammy [6,237 kB] 2024-04-17T10:21:09.208 INFO:teuthology.orchestra.run.smithi086.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-17T10:21:09.470 INFO:teuthology.orchestra.run.smithi086.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-17T10:21:09.470 INFO:teuthology.orchestra.run.smithi086.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-17T10:21:09.472 INFO:teuthology.orchestra.run.smithi086.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-17T10:21:09.636 INFO:teuthology.orchestra.run.smithi086.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-17T10:21:09.651 INFO:teuthology.orchestra.run.smithi067.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-1628-g97ae6cc9-1jammy [22.7 MB] 2024-04-17T10:21:09.893 INFO:teuthology.orchestra.run.smithi086.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1628-g97ae6cc9-1jammy [14.0 kB] 2024-04-17T10:21:09.893 INFO:teuthology.orchestra.run.smithi086.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1628-g97ae6cc9-1jammy [866 kB] 2024-04-17T10:21:09.917 INFO:teuthology.orchestra.run.smithi086.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1628-g97ae6cc9-1jammy [2,191 kB] 2024-04-17T10:21:09.976 INFO:teuthology.orchestra.run.smithi086.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-17T10:21:10.054 INFO:teuthology.orchestra.run.smithi086.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1628-g97ae6cc9-1jammy [776 kB] 2024-04-17T10:21:10.110 INFO:teuthology.orchestra.run.smithi086.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1628-g97ae6cc9-1jammy [144 kB] 2024-04-17T10:21:10.116 INFO:teuthology.orchestra.run.smithi086.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1628-g97ae6cc9-1jammy [1,921 kB] 2024-04-17T10:21:10.213 INFO:teuthology.orchestra.run.smithi086.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1628-g97ae6cc9-1jammy [8,625 kB] 2024-04-17T10:21:10.381 INFO:teuthology.orchestra.run.smithi086.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-17T10:21:10.390 INFO:teuthology.orchestra.run.smithi086.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-17T10:21:10.391 INFO:teuthology.orchestra.run.smithi086.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-17T10:21:10.461 INFO:teuthology.orchestra.run.smithi086.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-17T10:21:10.461 INFO:teuthology.orchestra.run.smithi086.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-17T10:21:10.462 INFO:teuthology.orchestra.run.smithi086.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-17T10:21:10.463 INFO:teuthology.orchestra.run.smithi086.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-17T10:21:10.463 INFO:teuthology.orchestra.run.smithi086.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-17T10:21:10.464 INFO:teuthology.orchestra.run.smithi086.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-17T10:21:10.629 INFO:teuthology.orchestra.run.smithi086.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-17T10:21:10.730 INFO:teuthology.orchestra.run.smithi086.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1628-g97ae6cc9-1jammy [14.2 kB] 2024-04-17T10:21:10.730 INFO:teuthology.orchestra.run.smithi086.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1628-g97ae6cc9-1jammy [49.9 MB] 2024-04-17T10:21:10.840 INFO:teuthology.orchestra.run.smithi086.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-17T10:21:11.168 INFO:teuthology.orchestra.run.smithi086.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-17T10:21:11.215 INFO:teuthology.orchestra.run.smithi086.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-17T10:21:11.242 INFO:teuthology.orchestra.run.smithi086.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-17T10:21:11.251 INFO:teuthology.orchestra.run.smithi086.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-17T10:21:11.296 INFO:teuthology.orchestra.run.smithi086.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-17T10:21:11.322 INFO:teuthology.orchestra.run.smithi086.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-17T10:21:11.324 INFO:teuthology.orchestra.run.smithi086.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-17T10:21:11.331 INFO:teuthology.orchestra.run.smithi086.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-17T10:21:11.334 INFO:teuthology.orchestra.run.smithi086.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-17T10:21:11.376 INFO:teuthology.orchestra.run.smithi067.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-1628-g97ae6cc9-1jammy [14.0 kB] 2024-04-17T10:21:11.377 INFO:teuthology.orchestra.run.smithi067.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-1628-g97ae6cc9-1jammy [866 kB] 2024-04-17T10:21:11.408 INFO:teuthology.orchestra.run.smithi067.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-1628-g97ae6cc9-1jammy [2,191 kB] 2024-04-17T10:21:11.530 INFO:teuthology.orchestra.run.smithi086.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-17T10:21:11.547 INFO:teuthology.orchestra.run.smithi067.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-1628-g97ae6cc9-1jammy [776 kB] 2024-04-17T10:21:11.601 INFO:teuthology.orchestra.run.smithi067.stdout:Get:113 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-1628-g97ae6cc9-1jammy [144 kB] 2024-04-17T10:21:11.606 INFO:teuthology.orchestra.run.smithi067.stdout:Get:114 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-1628-g97ae6cc9-1jammy [1,921 kB] 2024-04-17T10:21:11.735 INFO:teuthology.orchestra.run.smithi067.stdout:Get:115 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-1628-g97ae6cc9-1jammy [8,625 kB] 2024-04-17T10:21:11.789 INFO:teuthology.orchestra.run.smithi086.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-17T10:21:12.064 INFO:teuthology.orchestra.run.smithi086.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-17T10:21:12.066 INFO:teuthology.orchestra.run.smithi086.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-17T10:21:12.085 INFO:teuthology.orchestra.run.smithi086.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-17T10:21:12.089 INFO:teuthology.orchestra.run.smithi086.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-17T10:21:12.129 INFO:teuthology.orchestra.run.smithi086.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-17T10:21:12.143 INFO:teuthology.orchestra.run.smithi086.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-17T10:21:12.234 INFO:teuthology.orchestra.run.smithi086.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-17T10:21:12.235 INFO:teuthology.orchestra.run.smithi086.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-17T10:21:12.236 INFO:teuthology.orchestra.run.smithi086.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-17T10:21:12.290 INFO:teuthology.orchestra.run.smithi067.stdout:Get:116 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-1628-g97ae6cc9-1jammy [14.2 kB] 2024-04-17T10:21:12.290 INFO:teuthology.orchestra.run.smithi067.stdout:Get:117 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-1628-g97ae6cc9-1jammy [49.9 MB] 2024-04-17T10:21:12.416 INFO:teuthology.orchestra.run.smithi086.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-17T10:21:12.689 INFO:teuthology.orchestra.run.smithi086.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-17T10:21:13.958 INFO:teuthology.orchestra.run.smithi086.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1628-g97ae6cc9-1jammy [121 kB] 2024-04-17T10:21:13.959 INFO:teuthology.orchestra.run.smithi086.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-1628-g97ae6cc9-1jammy [228 kB] 2024-04-17T10:21:13.961 INFO:teuthology.orchestra.run.smithi086.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-1628-g97ae6cc9-1jammy [34.0 kB] 2024-04-17T10:21:13.962 INFO:teuthology.orchestra.run.smithi086.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-1628-g97ae6cc9-1jammy [23.4 kB] 2024-04-17T10:21:13.962 INFO:teuthology.orchestra.run.smithi086.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1628-g97ae6cc9-1jammy [38.4 kB] 2024-04-17T10:21:13.963 INFO:teuthology.orchestra.run.smithi086.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1628-g97ae6cc9-1jammy [13.0 MB] 2024-04-17T10:21:14.807 INFO:teuthology.orchestra.run.smithi086.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1628-g97ae6cc9-1jammy [91.5 kB] 2024-04-17T10:21:15.217 INFO:teuthology.orchestra.run.smithi067.stdout:Get:118 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-1628-g97ae6cc9-1jammy [121 kB] 2024-04-17T10:21:15.218 INFO:teuthology.orchestra.run.smithi067.stdout:Get:119 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-1628-g97ae6cc9-1jammy [228 kB] 2024-04-17T10:21:15.221 INFO:teuthology.orchestra.run.smithi067.stdout:Get:120 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-1628-g97ae6cc9-1jammy [34.0 kB] 2024-04-17T10:21:15.221 INFO:teuthology.orchestra.run.smithi067.stdout:Get:121 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-1628-g97ae6cc9-1jammy [23.4 kB] 2024-04-17T10:21:15.221 INFO:teuthology.orchestra.run.smithi067.stdout:Get:122 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-1628-g97ae6cc9-1jammy [38.4 kB] 2024-04-17T10:21:15.222 INFO:teuthology.orchestra.run.smithi067.stdout:Get:123 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-1628-g97ae6cc9-1jammy [13.0 MB] 2024-04-17T10:21:15.444 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 176 MB in 9s (18.6 MB/s) 2024-04-17T10:21:15.605 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-17T10:21:15.644 INFO:teuthology.orchestra.run.smithi067.stdout:Get:124 https://2.chacra.ceph.com/r/ceph/wip-neesingh-testing-240416/97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-1628-g97ae6cc9-1jammy [91.5 kB] 2024-04-17T10:21:15.671 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132922 files and directories currently installed.) 2024-04-17T10:21:15.675 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-17T10:21:15.718 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-17T10:21:16.021 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-17T10:21:16.034 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-17T10:21:16.112 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T10:21:16.320 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 176 MB in 10s (17.3 MB/s) 2024-04-17T10:21:16.432 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cmd2. 2024-04-17T10:21:16.445 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-17T10:21:16.450 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-17T10:21:16.490 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-17T10:21:16.518 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132922 files and directories currently installed.) 2024-04-17T10:21:16.522 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-17T10:21:16.557 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-17T10:21:16.877 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-17T10:21:16.891 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-17T10:21:16.976 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T10:21:17.047 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-17T10:21:17.062 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-17T10:21:17.227 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T10:21:17.297 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-17T10:21:17.311 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-17T10:21:17.346 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T10:21:17.538 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cmd2. 2024-04-17T10:21:17.554 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-17T10:21:17.604 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-17T10:21:17.699 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-17T10:21:17.714 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-17T10:21:17.766 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T10:21:18.033 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-17T10:21:18.048 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-17T10:21:18.149 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T10:21:18.253 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-17T10:21:18.268 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T10:21:18.311 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:21:18.545 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-17T10:21:18.560 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-17T10:21:18.643 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T10:21:18.664 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-17T10:21:18.678 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T10:21:18.714 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:21:18.981 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-17T10:21:18.996 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-17T10:21:19.039 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T10:21:19.109 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-17T10:21:19.124 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T10:21:19.167 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:21:19.492 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-17T10:21:19.508 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T10:21:19.554 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-17T10:21:19.559 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:21:19.568 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-17T10:21:19.612 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T10:21:19.890 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librados2. 2024-04-17T10:21:19.905 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../010-librados2_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:19.965 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librados2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-17T10:21:20.036 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T10:21:20.087 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:21:20.435 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librbd1. 2024-04-17T10:21:20.450 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../011-librbd1_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:20.456 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-17T10:21:20.472 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-17T10:21:20.493 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librbd1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:20.515 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:21:20.860 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-17T10:21:20.875 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-17T10:21:20.918 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T10:21:21.137 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libcephfs2. 2024-04-17T10:21:21.152 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:21.195 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libcephfs2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:21.329 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librados2. 2024-04-17T10:21:21.344 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../010-librados2_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:21.395 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librados2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:21.473 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rados. 2024-04-17T10:21:21.488 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../013-python3-rados_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:21.523 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rados (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:21.819 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-17T10:21:21.834 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:21.848 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librbd1. 2024-04-17T10:21:21.864 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../011-librbd1_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:21.886 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-ceph-argparse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:21.932 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librbd1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:22.138 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cephfs. 2024-04-17T10:21:22.153 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:22.196 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cephfs (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:22.335 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs2. 2024-04-17T10:21:22.350 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:22.393 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:22.541 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-17T10:21:22.555 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:22.590 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:22.712 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rados. 2024-04-17T10:21:22.727 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../013-python3-rados_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:22.778 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rados (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:22.910 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-prettytable. 2024-04-17T10:21:22.924 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-17T10:21:22.959 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-17T10:21:23.147 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-17T10:21:23.163 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:23.179 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rbd. 2024-04-17T10:21:23.194 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:23.214 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-argparse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:23.237 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rbd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:23.508 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cephfs. 2024-04-17T10:21:23.524 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:23.567 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cephfs (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:23.632 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-17T10:21:23.647 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-17T10:21:23.690 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T10:21:23.878 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-17T10:21:23.893 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:23.936 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:24.001 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-17T10:21:24.015 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-17T10:21:24.058 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T10:21:24.255 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-prettytable. 2024-04-17T10:21:24.271 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-17T10:21:24.314 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-17T10:21:24.403 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-17T10:21:24.418 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-17T10:21:24.453 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T10:21:24.541 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rbd. 2024-04-17T10:21:24.556 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:24.591 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rbd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:24.731 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua5.1. 2024-04-17T10:21:24.746 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-17T10:21:24.797 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-17T10:21:24.919 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-17T10:21:24.934 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-17T10:21:24.985 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T10:21:25.109 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua-any. 2024-04-17T10:21:25.124 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-17T10:21:25.184 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-17T10:21:25.280 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-17T10:21:25.295 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-17T10:21:25.338 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T10:21:25.487 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package zip. 2024-04-17T10:21:25.501 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-17T10:21:25.536 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking zip (3.0-12build2) ... 2024-04-17T10:21:25.633 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-17T10:21:25.648 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-17T10:21:25.691 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T10:21:25.873 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package unzip. 2024-04-17T10:21:25.887 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-17T10:21:25.931 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-17T10:21:26.027 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua5.1. 2024-04-17T10:21:26.043 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-17T10:21:26.085 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-17T10:21:26.359 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package luarocks. 2024-04-17T10:21:26.371 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-any. 2024-04-17T10:21:26.374 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-17T10:21:26.387 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-17T10:21:26.418 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-17T10:21:26.430 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-17T10:21:26.682 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package zip. 2024-04-17T10:21:26.698 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-17T10:21:26.740 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking zip (3.0-12build2) ... 2024-04-17T10:21:26.841 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package librgw2. 2024-04-17T10:21:26.855 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../027-librgw2_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:26.900 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking librgw2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:27.010 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package unzip. 2024-04-17T10:21:27.027 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-17T10:21:27.194 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-17T10:21:27.360 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rgw. 2024-04-17T10:21:27.375 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:27.410 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:27.614 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package luarocks. 2024-04-17T10:21:27.629 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-17T10:21:27.672 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-17T10:21:27.755 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-17T10:21:27.770 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-17T10:21:27.813 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T10:21:28.073 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package librgw2. 2024-04-17T10:21:28.087 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../027-librgw2_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:28.124 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libradosstriper1. 2024-04-17T10:21:28.132 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking librgw2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:28.139 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:28.199 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libradosstriper1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:28.559 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rgw. 2024-04-17T10:21:28.573 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:28.577 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-common. 2024-04-17T10:21:28.592 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../031-ceph-common_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:28.609 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:28.668 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:28.920 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-17T10:21:28.934 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-17T10:21:28.978 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T10:21:29.315 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libradosstriper1. 2024-04-17T10:21:29.329 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:29.373 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libradosstriper1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:29.734 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-common. 2024-04-17T10:21:29.748 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../031-ceph-common_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:29.784 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:29.815 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-base. 2024-04-17T10:21:29.830 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../032-ceph-base_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:29.874 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-base (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:30.470 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-17T10:21:30.483 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-17T10:21:30.519 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-17T10:21:30.839 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cheroot. 2024-04-17T10:21:30.853 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-17T10:21:30.897 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T10:21:30.931 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-base. 2024-04-17T10:21:30.947 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../032-ceph-base_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:30.998 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-base (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:31.441 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-17T10:21:31.456 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-17T10:21:31.499 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-17T10:21:31.577 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-17T10:21:31.591 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-17T10:21:31.635 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-17T10:21:31.802 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-17T10:21:31.817 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-17T10:21:31.852 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-17T10:21:31.938 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cheroot. 2024-04-17T10:21:31.953 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-17T10:21:31.996 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T10:21:32.155 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-17T10:21:32.170 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-17T10:21:32.213 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-17T10:21:32.341 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-17T10:21:32.357 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-17T10:21:32.391 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-17T10:21:32.516 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-tempora. 2024-04-17T10:21:32.530 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-17T10:21:32.594 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-17T10:21:32.669 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-17T10:21:32.684 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-17T10:21:32.727 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-17T10:21:32.902 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-portend. 2024-04-17T10:21:32.916 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-17T10:21:32.960 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-17T10:21:33.055 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-17T10:21:33.070 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-17T10:21:33.113 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-17T10:21:33.288 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-17T10:21:33.303 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-17T10:21:33.338 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-17T10:21:33.416 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempora. 2024-04-17T10:21:33.431 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-17T10:21:33.474 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-17T10:21:33.649 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-17T10:21:33.664 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-17T10:21:33.698 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-17T10:21:33.736 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-portend. 2024-04-17T10:21:33.751 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-17T10:21:33.785 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-17T10:21:34.052 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-natsort. 2024-04-17T10:21:34.067 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-17T10:21:34.110 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-17T10:21:34.113 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-17T10:21:34.129 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-17T10:21:34.171 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-17T10:21:34.422 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-logutils. 2024-04-17T10:21:34.437 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-17T10:21:34.458 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-17T10:21:34.474 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-17T10:21:34.480 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-17T10:21:34.516 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-17T10:21:34.749 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-17T10:21:34.764 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-17T10:21:34.799 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-17T10:21:34.895 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-natsort. 2024-04-17T10:21:34.910 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-17T10:21:34.953 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-17T10:21:35.135 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-mako. 2024-04-17T10:21:35.150 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-17T10:21:35.193 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T10:21:35.298 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-logutils. 2024-04-17T10:21:35.313 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-17T10:21:35.355 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-17T10:21:35.572 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-17T10:21:35.587 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-17T10:21:35.630 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-17T10:21:35.683 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-17T10:21:35.699 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-17T10:21:35.742 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-17T10:21:35.866 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-17T10:21:35.882 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-17T10:21:35.915 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-17T10:21:35.995 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-mako. 2024-04-17T10:21:36.010 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-17T10:21:36.044 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T10:21:36.235 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-webob. 2024-04-17T10:21:36.250 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-17T10:21:36.293 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-17T10:21:36.339 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-17T10:21:36.355 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-17T10:21:36.389 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-17T10:21:36.679 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-waitress. 2024-04-17T10:21:36.692 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-17T10:21:36.694 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-17T10:21:36.707 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-17T10:21:36.740 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T10:21:36.750 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-17T10:21:37.070 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webob. 2024-04-17T10:21:37.073 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-tempita. 2024-04-17T10:21:37.086 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-17T10:21:37.089 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-17T10:21:37.128 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-17T10:21:37.131 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T10:21:37.468 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-paste. 2024-04-17T10:21:37.483 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-17T10:21:37.517 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T10:21:37.673 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-waitress. 2024-04-17T10:21:37.688 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-17T10:21:37.733 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T10:21:37.821 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-17T10:21:37.835 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-17T10:21:37.904 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T10:21:38.067 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-tempita. 2024-04-17T10:21:38.083 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-17T10:21:38.125 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T10:21:38.198 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-17T10:21:38.214 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-17T10:21:38.915 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-17T10:21:39.112 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-paste. 2024-04-17T10:21:39.128 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-17T10:21:39.168 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-webtest. 2024-04-17T10:21:39.170 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T10:21:39.182 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-17T10:21:39.226 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-17T10:21:39.515 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-17T10:21:39.531 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-17T10:21:39.546 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pecan. 2024-04-17T10:21:39.560 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-17T10:21:39.573 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T10:21:39.604 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T10:21:39.876 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-17T10:21:39.890 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-17T10:21:39.934 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-17T10:21:39.983 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-certifi. 2024-04-17T10:21:39.998 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-17T10:21:40.042 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-17T10:21:40.204 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-webtest. 2024-04-17T10:21:40.219 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-17T10:21:40.254 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-17T10:21:40.402 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-idna. 2024-04-17T10:21:40.417 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-17T10:21:40.460 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-17T10:21:40.565 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pecan. 2024-04-17T10:21:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-17T10:21:40.623 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T10:21:40.814 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-urllib3. 2024-04-17T10:21:40.829 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-17T10:21:40.872 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T10:21:40.961 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-certifi. 2024-04-17T10:21:40.977 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-17T10:21:41.020 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-17T10:21:41.225 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-requests. 2024-04-17T10:21:41.239 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-17T10:21:41.274 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T10:21:41.372 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-idna. 2024-04-17T10:21:41.388 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-17T10:21:41.430 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-17T10:21:41.561 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-17T10:21:41.575 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-17T10:21:41.610 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T10:21:41.742 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-urllib3. 2024-04-17T10:21:41.757 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-17T10:21:41.800 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T10:21:41.947 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-17T10:21:41.962 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:42.005 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:42.153 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests. 2024-04-17T10:21:42.169 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-17T10:21:42.211 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T10:21:42.418 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-17T10:21:42.432 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:42.476 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:42.514 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-17T10:21:42.530 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-17T10:21:42.564 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T10:21:42.821 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-17T10:21:42.835 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-17T10:21:42.867 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-17T10:21:42.878 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-17T10:21:42.883 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:42.925 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-modules-core (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:43.190 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-openssl. 2024-04-17T10:21:43.204 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-17T10:21:43.239 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-17T10:21:43.314 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-17T10:21:43.329 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:43.372 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:43.584 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr. 2024-04-17T10:21:43.599 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:43.642 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:43.708 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-17T10:21:43.724 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-17T10:21:43.766 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-17T10:21:44.061 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mon. 2024-04-17T10:21:44.076 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:44.077 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-openssl. 2024-04-17T10:21:44.093 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-17T10:21:44.120 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mon (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:44.127 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-17T10:21:44.489 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr. 2024-04-17T10:21:44.504 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:44.546 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:44.715 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-osd. 2024-04-17T10:21:44.729 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:44.765 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-osd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:44.933 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mon. 2024-04-17T10:21:44.949 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:44.991 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mon (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:45.535 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph. 2024-04-17T10:21:45.549 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../068-ceph_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:45.562 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-osd. 2024-04-17T10:21:45.577 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:45.593 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:45.611 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-osd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:45.904 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-fuse. 2024-04-17T10:21:45.918 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:45.995 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:46.365 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph. 2024-04-17T10:21:46.373 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mds. 2024-04-17T10:21:46.379 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../068-ceph_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:46.386 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:46.423 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:46.631 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mds (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:46.742 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-fuse. 2024-04-17T10:21:46.757 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:46.817 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:46.984 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package cephadm. 2024-04-17T10:21:46.997 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../071-cephadm_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:47.059 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:47.187 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mds. 2024-04-17T10:21:47.202 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:47.245 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mds (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:47.353 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-17T10:21:47.365 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-17T10:21:47.412 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-17T10:21:47.631 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephadm. 2024-04-17T10:21:47.647 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../071-cephadm_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:21:47.681 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:47.773 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-17T10:21:47.788 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T10:21:47.831 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T10:21:47.967 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-17T10:21:47.982 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-17T10:21:48.016 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-17T10:21:48.328 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-17T10:21:48.343 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T10:21:48.411 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T10:21:48.844 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-babel. 2024-04-17T10:21:48.855 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T10:21:48.910 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T10:21:49.272 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-jinja2. 2024-04-17T10:21:49.283 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-17T10:21:49.321 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T10:21:49.399 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-babel. 2024-04-17T10:21:49.414 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-17T10:21:49.457 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T10:21:49.716 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-17T10:21:49.728 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:49.766 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:49.777 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-jinja2. 2024-04-17T10:21:49.791 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-17T10:21:49.835 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T10:21:50.136 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-17T10:21:50.147 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-17T10:21:50.185 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-17T10:21:50.213 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-17T10:21:50.228 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:50.271 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:50.505 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-routes. 2024-04-17T10:21:50.517 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-17T10:21:50.554 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T10:21:50.600 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-17T10:21:50.615 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-17T10:21:50.658 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-17T10:21:50.925 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-17T10:21:50.936 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:50.974 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:50.977 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-routes. 2024-04-17T10:21:50.993 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-17T10:21:51.035 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T10:21:51.353 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-17T10:21:51.364 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-17T10:21:51.372 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-17T10:21:51.388 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:51.419 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T10:21:51.430 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-dashboard (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:51.834 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-17T10:21:51.849 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-17T10:21:51.892 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T10:21:51.947 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-joblib. 2024-04-17T10:21:51.959 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-17T10:21:51.997 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T10:21:52.367 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-17T10:21:52.379 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-joblib. 2024-04-17T10:21:52.379 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-17T10:21:52.394 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-17T10:21:52.453 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T10:21:52.502 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-17T10:21:52.780 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-sklearn. 2024-04-17T10:21:52.792 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-17T10:21:52.854 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T10:21:53.007 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-17T10:21:53.023 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-17T10:21:53.142 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-17T10:21:53.470 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-sklearn. 2024-04-17T10:21:53.486 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-17T10:21:53.522 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-17T10:21:53.534 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:53.536 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T10:21:53.571 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:54.270 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-17T10:21:54.285 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:54.328 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:54.599 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-cachetools. 2024-04-17T10:21:54.612 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-17T10:21:54.649 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-17T10:21:54.969 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-17T10:21:54.982 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-17T10:21:55.027 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-17T10:21:55.307 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-cachetools. 2024-04-17T10:21:55.323 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-17T10:21:55.339 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-17T10:21:55.353 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-17T10:21:55.365 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-17T10:21:55.397 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T10:21:55.633 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-rsa. 2024-04-17T10:21:55.646 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-17T10:21:55.683 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-17T10:21:55.701 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-17T10:21:55.717 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-17T10:21:55.759 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-17T10:21:56.028 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-google-auth. 2024-04-17T10:21:56.040 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-17T10:21:56.071 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-17T10:21:56.078 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-17T10:21:56.087 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-17T10:21:56.121 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T10:21:56.415 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-17T10:21:56.429 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-17T10:21:56.441 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-rsa. 2024-04-17T10:21:56.456 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-17T10:21:56.472 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T10:21:56.499 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-17T10:21:56.809 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-websocket. 2024-04-17T10:21:56.819 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-google-auth. 2024-04-17T10:21:56.822 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-17T10:21:56.835 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-17T10:21:56.859 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-17T10:21:56.877 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-17T10:21:57.230 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-17T10:21:57.246 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-17T10:21:57.289 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T10:21:57.395 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-17T10:21:57.408 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-17T10:21:57.436 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T10:21:57.600 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-websocket. 2024-04-17T10:21:57.616 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-17T10:21:57.658 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-17T10:21:58.137 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-17T10:21:58.153 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-17T10:21:58.167 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-17T10:21:58.178 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:58.186 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T10:21:58.216 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:58.686 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-17T10:21:58.697 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-17T10:21:58.736 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T10:21:59.017 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-17T10:21:59.032 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:21:59.075 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:21:59.089 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-17T10:21:59.101 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T10:21:59.139 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T10:21:59.394 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-17T10:21:59.409 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-17T10:21:59.416 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package jq. 2024-04-17T10:21:59.429 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T10:21:59.452 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T10:21:59.466 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-17T10:21:59.702 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package socat. 2024-04-17T10:21:59.715 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-17T10:21:59.768 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-17T10:21:59.806 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-17T10:21:59.821 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T10:21:59.856 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T10:22:00.038 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package xmlstarlet. 2024-04-17T10:22:00.050 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-17T10:22:00.089 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-17T10:22:00.100 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package jq. 2024-04-17T10:22:00.115 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-17T10:22:00.158 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-17T10:22:00.436 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package socat. 2024-04-17T10:22:00.451 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-17T10:22:00.477 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-test. 2024-04-17T10:22:00.489 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../099-ceph-test_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:00.494 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-17T10:22:00.526 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-test (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:00.789 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package xmlstarlet. 2024-04-17T10:22:00.805 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-17T10:22:00.847 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-17T10:22:01.243 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-test. 2024-04-17T10:22:01.258 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../099-ceph-test_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:01.301 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-test (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:02.539 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ceph-volume. 2024-04-17T10:22:02.552 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:22:02.589 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ceph-volume (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:03.001 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-17T10:22:03.013 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:03.051 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking cephfs-mirror (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:03.180 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ceph-volume. 2024-04-17T10:22:03.197 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:22:03.238 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ceph-volume (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:03.345 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-colorama. 2024-04-17T10:22:03.357 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-17T10:22:03.387 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-17T10:22:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package cephfs-shell. 2024-04-17T10:22:03.609 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-17T10:22:03.621 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:22:03.626 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:03.688 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephfs-mirror (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:03.697 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking cephfs-shell (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:03.967 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package cephfs-top. 2024-04-17T10:22:03.980 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:22:04.017 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking cephfs-top (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:04.020 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-colorama. 2024-04-17T10:22:04.036 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-17T10:22:04.078 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-17T10:22:04.294 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-17T10:22:04.308 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:04.339 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephfs-shell. 2024-04-17T10:22:04.352 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libcephfs-dev (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:04.356 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:22:04.398 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephfs-shell (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:04.667 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package cephfs-top. 2024-04-17T10:22:04.683 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-1628-g97ae6cc9-1jammy_all.deb ... 2024-04-17T10:22:04.705 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-17T10:22:04.718 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-17T10:22:04.725 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking cephfs-top (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:04.763 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T10:22:04.952 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-17T10:22:04.969 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:05.003 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libcephfs-dev (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:05.084 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-17T10:22:05.098 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-17T10:22:05.150 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-17T10:22:05.322 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-17T10:22:05.339 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-17T10:22:05.381 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T10:22:05.504 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package nvme-cli. 2024-04-17T10:22:05.518 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-17T10:22:05.562 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T10:22:05.668 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-17T10:22:05.684 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-17T10:22:05.726 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-17T10:22:06.009 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package pkg-config. 2024-04-17T10:22:06.023 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-17T10:22:06.084 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T10:22:06.113 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package nvme-cli. 2024-04-17T10:22:06.129 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-17T10:22:06.171 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T10:22:06.328 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-17T10:22:06.342 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-17T10:22:06.386 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-17T10:22:06.593 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package pkg-config. 2024-04-17T10:22:06.609 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-17T10:22:06.651 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T10:22:06.847 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-17T10:22:06.861 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-17T10:22:06.905 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-17T10:22:06.938 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-17T10:22:06.954 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-17T10:22:06.998 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-17T10:22:07.225 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pastescript. 2024-04-17T10:22:07.239 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-17T10:22:07.283 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-17T10:22:07.409 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-17T10:22:07.422 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-17T10:22:07.459 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-17T10:22:07.620 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pluggy. 2024-04-17T10:22:07.633 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-17T10:22:07.669 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-17T10:22:07.762 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pastescript. 2024-04-17T10:22:07.777 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-17T10:22:07.811 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-17T10:22:07.955 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-psutil. 2024-04-17T10:22:07.969 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-17T10:22:08.030 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-17T10:22:08.098 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pluggy. 2024-04-17T10:22:08.110 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-17T10:22:08.156 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-17T10:22:08.358 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-py. 2024-04-17T10:22:08.372 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-17T10:22:08.408 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-17T10:22:08.443 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-psutil. 2024-04-17T10:22:08.455 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-17T10:22:08.517 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-17T10:22:08.711 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pygments. 2024-04-17T10:22:08.725 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-17T10:22:08.769 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T10:22:08.870 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-py. 2024-04-17T10:22:08.882 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-17T10:22:08.920 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-17T10:22:09.248 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pygments. 2024-04-17T10:22:09.261 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-17T10:22:09.265 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-17T10:22:09.277 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-17T10:22:09.314 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-17T10:22:09.332 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T10:22:09.593 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-toml. 2024-04-17T10:22:09.605 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-17T10:22:09.634 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-17T10:22:09.752 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-17T10:22:09.767 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-17T10:22:09.810 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-17T10:22:09.945 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-pytest. 2024-04-17T10:22:09.959 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-17T10:22:10.003 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T10:22:10.138 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-toml. 2024-04-17T10:22:10.151 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-17T10:22:10.188 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-17T10:22:10.323 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package python3-simplejson. 2024-04-17T10:22:10.337 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-17T10:22:10.404 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-17T10:22:10.491 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-pytest. 2024-04-17T10:22:10.504 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-17T10:22:10.549 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T10:22:10.642 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-17T10:22:10.656 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-17T10:22:10.692 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-17T10:22:10.911 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package python3-simplejson. 2024-04-17T10:22:10.924 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-17T10:22:10.969 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-17T10:22:11.238 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-17T10:22:11.252 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-17T10:22:11.296 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-17T10:22:11.396 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package radosgw. 2024-04-17T10:22:11.408 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../122-radosgw_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:11.437 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking radosgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:11.942 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package radosgw. 2024-04-17T10:22:11.954 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../122-radosgw_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:11.983 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking radosgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:12.090 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package rbd-fuse. 2024-04-17T10:22:12.104 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:12.148 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking rbd-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:12.483 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-17T10:22:12.595 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package rbd-fuse. 2024-04-17T10:22:12.608 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-1628-g97ae6cc9-1jammy_amd64.deb ... 2024-04-17T10:22:12.653 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking rbd-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:12.778 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T10:22:12.946 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T10:22:12.947 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-17T10:22:13.226 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-17T10:22:13.367 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-17T10:22:13.492 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-17T10:22:13.624 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-17T10:22:13.791 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-17T10:22:14.219 INFO:teuthology.orchestra.run.smithi086.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-17T10:22:14.222 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-17T10:22:14.273 INFO:teuthology.orchestra.run.smithi086.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-04-17T10:22:14.323 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:14.817 INFO:teuthology.orchestra.run.smithi086.stdout:Adding system user cephadm....done 2024-04-17T10:22:14.834 INFO:teuthology.orchestra.run.smithi067.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-17T10:22:14.889 INFO:teuthology.orchestra.run.smithi067.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2024-04-17T10:22:14.960 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:14.970 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T10:22:15.272 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-17T10:22:15.537 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user cephadm....done 2024-04-17T10:22:15.539 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-17T10:22:15.665 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-17T10:22:15.691 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-17T10:22:15.950 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-17T10:22:16.018 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-17T10:22:16.218 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T10:22:16.285 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-17T10:22:16.318 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-17T10:22:16.412 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-17T10:22:16.611 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T10:22:16.696 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-17T10:22:16.947 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-17T10:22:17.013 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-17T10:22:17.065 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-17T10:22:17.281 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-17T10:22:17.391 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-17T10:22:17.716 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-17T10:22:17.768 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-17T10:22:18.009 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-17T10:22:18.045 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-17T10:22:18.329 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-17T10:22:18.418 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-17T10:22:18.597 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-17T10:22:18.745 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-17T10:22:18.815 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-17T10:22:19.021 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-17T10:22:19.150 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-17T10:22:19.330 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-ceph-argparse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:19.443 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-17T10:22:19.639 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T10:22:19.735 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-argparse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:19.799 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T10:22:19.950 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-17T10:22:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-17T10:22:20.137 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-17T10:22:20.234 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T10:22:20.238 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-17T10:22:20.377 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-17T10:22:20.489 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-17T10:22:20.632 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-17T10:22:20.636 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-17T10:22:20.900 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-17T10:22:21.017 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-17T10:22:21.075 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-17T10:22:21.148 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T10:22:21.273 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-17T10:22:21.305 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-17T10:22:21.364 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-17T10:22:21.428 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-17T10:22:21.546 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-17T10:22:21.658 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T10:22:21.759 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-17T10:22:21.905 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-17T10:22:22.006 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-17T10:22:22.052 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T10:22:22.291 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-17T10:22:22.311 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-17T10:22:22.567 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-17T10:22:22.629 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up zip (3.0-12build2) ... 2024-04-17T10:22:22.777 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T10:22:22.885 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up zip (3.0-12build2) ... 2024-04-17T10:22:23.036 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-17T10:22:23.490 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T10:22:23.763 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-17T10:22:23.766 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T10:22:23.917 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-17T10:22:24.014 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-17T10:22:24.118 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T10:22:24.131 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-17T10:22:24.257 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-17T10:22:24.477 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-17T10:22:24.583 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-17T10:22:24.887 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-17T10:22:25.001 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-17T10:22:25.188 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T10:22:25.269 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-17T10:22:25.598 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T10:22:25.687 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-17T10:22:25.932 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-17T10:22:25.997 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-17T10:22:26.267 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T10:22:26.315 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-17T10:22:26.635 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-17T10:22:26.683 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-17T10:22:26.886 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-17T10:22:26.950 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-17T10:22:27.020 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:27.076 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:27.363 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-idna (3.3-1) ... 2024-04-17T10:22:27.394 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-idna (3.3-1) ... 2024-04-17T10:22:27.706 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T10:22:27.754 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-17T10:22:28.961 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T10:22:29.026 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-17T10:22:29.196 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:22:29.261 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:22:29.364 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-17T10:22:29.412 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-17T10:22:29.640 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T10:22:29.680 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-17T10:22:29.781 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-17T10:22:29.799 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-17T10:22:29.925 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T10:22:29.957 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-17T10:22:30.235 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-17T10:22:30.267 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-17T10:22:30.502 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-17T10:22:30.576 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-17T10:22:30.770 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:22:30.844 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:22:30.921 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-17T10:22:30.987 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-17T10:22:31.256 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-17T10:22:31.313 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-17T10:22:31.549 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-17T10:22:31.564 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-17T10:22:31.824 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-17T10:22:31.850 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-17T10:22:32.092 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-17T10:22:32.150 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-17T10:22:32.359 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-17T10:22:32.401 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-17T10:22:32.644 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T10:22:32.669 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-17T10:22:32.770 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-17T10:22:32.812 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-17T10:22:33.062 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T10:22:33.121 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-17T10:22:33.163 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T10:22:33.247 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-17T10:22:33.448 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-17T10:22:33.582 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-17T10:22:33.766 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-17T10:22:33.916 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-17T10:22:34.034 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T10:22:34.159 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-17T10:22:34.193 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-17T10:22:34.259 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-17T10:22:34.319 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T10:22:34.369 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-17T10:22:34.408 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-17T10:22:34.458 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-17T10:22:34.662 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T10:22:34.696 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-17T10:22:34.763 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T10:22:34.846 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-17T10:22:35.156 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-17T10:22:35.265 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-17T10:22:35.433 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-17T10:22:35.549 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-17T10:22:35.558 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-17T10:22:35.667 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-17T10:22:35.826 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:22:35.918 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-17T10:22:35.952 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T10:22:36.052 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-17T10:22:36.287 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T10:22:36.378 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-17T10:22:36.596 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T10:22:36.729 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-17T10:22:36.855 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-17T10:22:37.014 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-17T10:22:37.042 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-17T10:22:37.267 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-17T10:22:37.352 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-17T10:22:37.560 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-17T10:22:37.628 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T10:22:37.827 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-17T10:22:37.904 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-17T10:22:38.137 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-17T10:22:38.281 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-17T10:22:38.539 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-17T10:22:38.574 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T10:22:38.874 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-17T10:22:38.952 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T10:22:39.077 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librados2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:39.203 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:39.243 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-17T10:22:39.313 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-17T10:22:39.418 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librados2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:39.544 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libsqlite3-mod-ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:39.607 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:39.687 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-17T10:22:39.758 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libradosstriper1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:39.884 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-17T10:22:40.106 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libcephfs2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:40.186 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librbd1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:40.299 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libradosstriper1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:40.312 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-modules-core (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:40.438 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:40.450 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-17T10:22:40.663 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T10:22:40.664 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T10:22:40.785 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librbd1 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:40.911 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-modules-core (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:41.037 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:41.247 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T10:22:41.247 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-17T10:22:41.289 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libcephfs-dev (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:41.432 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rados (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:41.558 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up librgw2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:41.684 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephfs-top (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:41.810 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T10:22:41.905 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libcephfs-dev (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:42.065 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rados (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:42.233 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up librgw2 (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:42.359 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up cephfs-top (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:42.518 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-17T10:22:43.164 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rbd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:43.264 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up rbd-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:43.399 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-rgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:43.525 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up python3-cephfs (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:43.651 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephfs-shell (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:43.817 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rbd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:43.819 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:43.943 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up rbd-fuse (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:44.169 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-rgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:44.295 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up python3-cephfs (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:44.372 INFO:teuthology.orchestra.run.smithi067.stdout:Adding group ceph....done 2024-04-17T10:22:44.396 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up cephfs-shell (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:44.505 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-common (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:44.849 INFO:teuthology.orchestra.run.smithi067.stdout:Adding system user ceph....done 2024-04-17T10:22:45.050 INFO:teuthology.orchestra.run.smithi086.stdout:Adding group ceph....done 2024-04-17T10:22:45.126 INFO:teuthology.orchestra.run.smithi067.stdout:Setting system user ceph properties....done 2024-04-17T10:22:45.144 INFO:teuthology.orchestra.run.smithi067.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-17T10:22:45.299 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-17T10:22:45.434 INFO:teuthology.orchestra.run.smithi086.stdout:Adding system user ceph....done 2024-04-17T10:22:45.603 INFO:teuthology.orchestra.run.smithi086.stdout:Setting system user ceph properties....done 2024-04-17T10:22:45.621 INFO:teuthology.orchestra.run.smithi086.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-17T10:22:45.722 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-17T10:22:45.764 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-17T10:22:46.152 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-17T10:22:46.403 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up cephfs-mirror (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:46.611 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T10:22:46.611 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T10:22:46.831 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up cephfs-mirror (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:47.057 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T10:22:47.057 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-17T10:22:47.313 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-test (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:47.439 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up radosgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:47.691 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-test (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:47.816 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up radosgw (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:47.960 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T10:22:47.960 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T10:22:48.355 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T10:22:48.355 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-17T10:22:48.625 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-base (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:48.968 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-base (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:48.992 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-17T10:22:49.467 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-17T10:22:49.652 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mds (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:49.873 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T10:22:49.874 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T10:22:50.120 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mds (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:50.333 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T10:22:50.334 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-17T10:22:50.554 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:50.774 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T10:22:50.774 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T10:22:50.972 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:51.200 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T10:22:51.201 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-17T10:22:51.422 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-osd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:51.737 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T10:22:51.737 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T10:22:51.848 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-osd (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:52.211 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T10:22:52.212 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-17T10:22:52.432 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:52.591 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:52.734 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mon (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:52.891 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-k8sevents (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:52.949 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T10:22:52.949 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T10:22:53.017 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:53.160 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mon (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:53.369 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T10:22:53.370 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-17T10:22:53.619 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:53.761 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:53.887 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-mgr-dashboard (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:54.011 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-cephadm (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:54.030 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ceph-volume (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:54.178 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:54.363 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-mgr-dashboard (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:54.492 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T10:22:54.592 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T10:22:54.638 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ceph-volume (19.0.0-1628-g97ae6cc9-1jammy) ... 2024-04-17T10:22:54.744 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T10:22:55.002 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T10:22:55.111 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T10:22:55.296 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T10:22:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T10:22:56.380 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T10:22:56.601 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T10:22:56.606 DEBUG:teuthology.orchestra.run.smithi067:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-17T10:22:56.682 INFO:teuthology.orchestra.run.smithi067.stdout:Reading package lists... 2024-04-17T10:22:56.859 INFO:teuthology.orchestra.run.smithi067.stdout:Building dependency tree... 2024-04-17T10:22:56.860 INFO:teuthology.orchestra.run.smithi067.stdout:Reading state information... 2024-04-17T10:22:57.034 INFO:teuthology.orchestra.run.smithi067.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-17T10:22:57.035 INFO:teuthology.orchestra.run.smithi067.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T10:22:57.036 INFO:teuthology.orchestra.run.smithi067.stdout:The following additional packages will be installed: 2024-04-17T10:22:57.036 INFO:teuthology.orchestra.run.smithi067.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-17T10:22:57.036 INFO:teuthology.orchestra.run.smithi067.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-17T10:22:57.036 INFO:teuthology.orchestra.run.smithi067.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-17T10:22:57.037 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-17T10:22:57.037 INFO:teuthology.orchestra.run.smithi067.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T10:22:57.037 INFO:teuthology.orchestra.run.smithi067.stdout:Suggested packages: 2024-04-17T10:22:57.037 INFO:teuthology.orchestra.run.smithi067.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-17T10:22:57.037 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-17T10:22:57.037 INFO:teuthology.orchestra.run.smithi067.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-17T10:22:57.093 INFO:teuthology.orchestra.run.smithi067.stdout:The following NEW packages will be installed: 2024-04-17T10:22:57.094 INFO:teuthology.orchestra.run.smithi067.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-17T10:22:57.094 INFO:teuthology.orchestra.run.smithi067.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-17T10:22:57.094 INFO:teuthology.orchestra.run.smithi067.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-17T10:22:57.094 INFO:teuthology.orchestra.run.smithi067.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-17T10:22:57.095 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-17T10:22:57.095 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-17T10:22:57.095 INFO:teuthology.orchestra.run.smithi067.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T10:22:57.096 INFO:teuthology.orchestra.run.smithi086.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T10:22:57.097 INFO:teuthology.orchestra.run.smithi067.stdout:The following packages will be upgraded: 2024-04-17T10:22:57.097 INFO:teuthology.orchestra.run.smithi067.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-17T10:22:57.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-17T10:22:57.177 INFO:teuthology.orchestra.run.smithi086.stdout:Reading package lists... 2024-04-17T10:22:57.335 INFO:teuthology.orchestra.run.smithi067.stdout:4 upgraded, 39 newly installed, 0 to remove and 301 not upgraded. 2024-04-17T10:22:57.335 INFO:teuthology.orchestra.run.smithi067.stdout:Need to get 55.7 MB of archives. 2024-04-17T10:22:57.335 INFO:teuthology.orchestra.run.smithi067.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-17T10:22:57.335 INFO:teuthology.orchestra.run.smithi067.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-17T10:22:57.354 INFO:teuthology.orchestra.run.smithi086.stdout:Building dependency tree... 2024-04-17T10:22:57.355 INFO:teuthology.orchestra.run.smithi086.stdout:Reading state information... 2024-04-17T10:22:57.528 INFO:teuthology.orchestra.run.smithi086.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-17T10:22:57.528 INFO:teuthology.orchestra.run.smithi086.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-17T10:22:57.528 INFO:teuthology.orchestra.run.smithi086.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-17T10:22:57.528 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages were automatically installed and are no longer required: 2024-04-17T10:22:57.528 INFO:teuthology.orchestra.run.smithi086.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-17T10:22:57.529 INFO:teuthology.orchestra.run.smithi086.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-17T10:22:57.529 INFO:teuthology.orchestra.run.smithi086.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-17T10:22:57.529 INFO:teuthology.orchestra.run.smithi086.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-17T10:22:57.529 INFO:teuthology.orchestra.run.smithi086.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-17T10:22:57.529 INFO:teuthology.orchestra.run.smithi086.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-17T10:22:57.530 INFO:teuthology.orchestra.run.smithi086.stdout:The following additional packages will be installed: 2024-04-17T10:22:57.530 INFO:teuthology.orchestra.run.smithi086.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-17T10:22:57.530 INFO:teuthology.orchestra.run.smithi086.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-17T10:22:57.531 INFO:teuthology.orchestra.run.smithi086.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-17T10:22:57.531 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-17T10:22:57.531 INFO:teuthology.orchestra.run.smithi086.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T10:22:57.531 INFO:teuthology.orchestra.run.smithi086.stdout:Suggested packages: 2024-04-17T10:22:57.531 INFO:teuthology.orchestra.run.smithi086.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-17T10:22:57.531 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-17T10:22:57.532 INFO:teuthology.orchestra.run.smithi086.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-17T10:22:57.584 INFO:teuthology.orchestra.run.smithi086.stdout:The following NEW packages will be installed: 2024-04-17T10:22:57.585 INFO:teuthology.orchestra.run.smithi086.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-17T10:22:57.586 INFO:teuthology.orchestra.run.smithi086.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-17T10:22:57.586 INFO:teuthology.orchestra.run.smithi086.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-17T10:22:57.586 INFO:teuthology.orchestra.run.smithi086.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-17T10:22:57.586 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-17T10:22:57.586 INFO:teuthology.orchestra.run.smithi086.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-17T10:22:57.586 INFO:teuthology.orchestra.run.smithi086.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-17T10:22:57.587 INFO:teuthology.orchestra.run.smithi086.stdout:The following packages will be upgraded: 2024-04-17T10:22:57.587 INFO:teuthology.orchestra.run.smithi086.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-17T10:22:57.772 INFO:teuthology.orchestra.run.smithi086.stdout:4 upgraded, 39 newly installed, 0 to remove and 301 not upgraded. 2024-04-17T10:22:57.772 INFO:teuthology.orchestra.run.smithi086.stdout:Need to get 55.7 MB of archives. 2024-04-17T10:22:57.773 INFO:teuthology.orchestra.run.smithi086.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-17T10:22:57.773 INFO:teuthology.orchestra.run.smithi086.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-17T10:22:57.978 INFO:teuthology.orchestra.run.smithi067.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-17T10:22:57.994 INFO:teuthology.orchestra.run.smithi067.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-17T10:22:58.121 INFO:teuthology.orchestra.run.smithi067.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-17T10:22:58.121 INFO:teuthology.orchestra.run.smithi067.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-17T10:22:58.122 INFO:teuthology.orchestra.run.smithi067.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-17T10:22:58.151 INFO:teuthology.orchestra.run.smithi067.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-17T10:22:58.152 INFO:teuthology.orchestra.run.smithi067.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-17T10:22:58.156 INFO:teuthology.orchestra.run.smithi067.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-17T10:22:58.158 INFO:teuthology.orchestra.run.smithi067.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-17T10:22:58.159 INFO:teuthology.orchestra.run.smithi067.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-17T10:22:58.324 INFO:teuthology.orchestra.run.smithi067.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-17T10:22:58.414 INFO:teuthology.orchestra.run.smithi086.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-17T10:22:58.430 INFO:teuthology.orchestra.run.smithi086.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-17T10:22:58.557 INFO:teuthology.orchestra.run.smithi086.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-17T10:22:58.557 INFO:teuthology.orchestra.run.smithi086.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-17T10:22:58.558 INFO:teuthology.orchestra.run.smithi086.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-17T10:22:58.585 INFO:teuthology.orchestra.run.smithi086.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-17T10:22:58.586 INFO:teuthology.orchestra.run.smithi086.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-17T10:22:58.590 INFO:teuthology.orchestra.run.smithi086.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-17T10:22:58.592 INFO:teuthology.orchestra.run.smithi086.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-17T10:22:58.593 INFO:teuthology.orchestra.run.smithi086.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-17T10:22:58.618 INFO:teuthology.orchestra.run.smithi067.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-17T10:22:58.627 INFO:teuthology.orchestra.run.smithi067.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-17T10:22:58.632 INFO:teuthology.orchestra.run.smithi067.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-17T10:22:58.717 INFO:teuthology.orchestra.run.smithi067.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-17T10:22:58.759 INFO:teuthology.orchestra.run.smithi086.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-17T10:22:59.054 INFO:teuthology.orchestra.run.smithi086.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-17T10:22:59.063 INFO:teuthology.orchestra.run.smithi086.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-17T10:22:59.068 INFO:teuthology.orchestra.run.smithi086.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-17T10:22:59.157 INFO:teuthology.orchestra.run.smithi086.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-17T10:22:59.802 INFO:teuthology.orchestra.run.smithi067.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-17T10:22:59.804 INFO:teuthology.orchestra.run.smithi067.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-17T10:22:59.804 INFO:teuthology.orchestra.run.smithi067.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-17T10:22:59.807 INFO:teuthology.orchestra.run.smithi067.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-17T10:22:59.807 INFO:teuthology.orchestra.run.smithi067.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-17T10:22:59.812 INFO:teuthology.orchestra.run.smithi067.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-17T10:22:59.978 INFO:teuthology.orchestra.run.smithi067.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-17T10:23:00.156 INFO:teuthology.orchestra.run.smithi086.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-17T10:23:00.158 INFO:teuthology.orchestra.run.smithi086.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-17T10:23:00.158 INFO:teuthology.orchestra.run.smithi086.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-17T10:23:00.161 INFO:teuthology.orchestra.run.smithi086.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-17T10:23:00.161 INFO:teuthology.orchestra.run.smithi086.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-17T10:23:00.165 INFO:teuthology.orchestra.run.smithi086.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-17T10:23:00.331 INFO:teuthology.orchestra.run.smithi086.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-17T10:23:00.585 INFO:teuthology.orchestra.run.smithi067.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-17T10:23:00.585 INFO:teuthology.orchestra.run.smithi067.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-17T10:23:00.765 INFO:teuthology.orchestra.run.smithi067.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-17T10:23:00.766 INFO:teuthology.orchestra.run.smithi067.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-17T10:23:00.804 INFO:teuthology.orchestra.run.smithi067.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-17T10:23:00.865 INFO:teuthology.orchestra.run.smithi067.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-17T10:23:00.871 INFO:teuthology.orchestra.run.smithi067.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-17T10:23:00.871 INFO:teuthology.orchestra.run.smithi067.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-17T10:23:00.872 INFO:teuthology.orchestra.run.smithi067.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-17T10:23:00.873 INFO:teuthology.orchestra.run.smithi067.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-17T10:23:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-17T10:23:00.940 INFO:teuthology.orchestra.run.smithi086.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-17T10:23:01.079 INFO:teuthology.orchestra.run.smithi067.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-17T10:23:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-17T10:23:01.120 INFO:teuthology.orchestra.run.smithi086.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-17T10:23:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-17T10:23:01.216 INFO:teuthology.orchestra.run.smithi086.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-17T10:23:01.221 INFO:teuthology.orchestra.run.smithi086.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-17T10:23:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-17T10:23:01.222 INFO:teuthology.orchestra.run.smithi086.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-17T10:23:01.223 INFO:teuthology.orchestra.run.smithi086.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-17T10:23:01.429 INFO:teuthology.orchestra.run.smithi086.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-17T10:23:01.504 INFO:teuthology.orchestra.run.smithi067.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-17T10:23:01.850 INFO:teuthology.orchestra.run.smithi086.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-17T10:23:02.254 INFO:teuthology.orchestra.run.smithi067.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-17T10:23:02.254 INFO:teuthology.orchestra.run.smithi067.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-17T10:23:02.254 INFO:teuthology.orchestra.run.smithi067.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-17T10:23:02.255 INFO:teuthology.orchestra.run.smithi067.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-17T10:23:02.255 INFO:teuthology.orchestra.run.smithi067.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-17T10:23:02.255 INFO:teuthology.orchestra.run.smithi067.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-17T10:23:02.285 INFO:teuthology.orchestra.run.smithi067.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-17T10:23:02.285 INFO:teuthology.orchestra.run.smithi067.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-17T10:23:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-17T10:23:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-17T10:23:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-17T10:23:02.600 INFO:teuthology.orchestra.run.smithi086.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-17T10:23:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-17T10:23:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-17T10:23:02.634 INFO:teuthology.orchestra.run.smithi086.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-17T10:23:02.635 INFO:teuthology.orchestra.run.smithi086.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-17T10:23:02.680 INFO:teuthology.orchestra.run.smithi067.stdout:Preconfiguring packages ... 2024-04-17T10:23:02.846 INFO:teuthology.orchestra.run.smithi067.stdout:Fetched 55.7 MB in 5s (10.8 MB/s) 2024-04-17T10:23:03.010 INFO:teuthology.orchestra.run.smithi086.stdout:Preconfiguring packages ... 2024-04-17T10:23:03.120 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140582 files and directories currently installed.) 2024-04-17T10:23:03.124 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T10:23:03.171 INFO:teuthology.orchestra.run.smithi086.stdout:Fetched 55.7 MB in 5s (11.1 MB/s) 2024-04-17T10:23:03.270 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T10:23:03.416 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140582 files and directories currently installed.) 2024-04-17T10:23:03.419 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T10:23:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T10:23:03.913 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-17T10:23:04.261 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-17T10:23:04.993 INFO:teuthology.orchestra.run.smithi067.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-17T10:23:05.293 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package flex. 2024-04-17T10:23:05.367 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140582 files and directories currently installed.) 2024-04-17T10:23:05.370 INFO:teuthology.orchestra.run.smithi086.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-17T10:23:05.371 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-17T10:23:05.416 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-17T10:23:05.660 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package flex. 2024-04-17T10:23:05.733 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140582 files and directories currently installed.) 2024-04-17T10:23:05.736 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-17T10:23:05.775 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-17T10:23:05.911 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T10:23:06.089 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-17T10:23:06.471 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T10:23:06.480 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T10:23:06.747 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-17T10:23:07.044 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140655 files and directories currently installed.) 2024-04-17T10:23:07.048 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T10:23:07.172 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T10:23:07.568 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T10:23:07.595 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T10:23:07.695 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T10:23:08.062 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T10:23:08.185 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140655 files and directories currently installed.) 2024-04-17T10:23:08.189 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T10:23:08.356 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T10:23:08.420 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package bison. 2024-04-17T10:23:08.500 INFO:teuthology.orchestra.run.smithi067.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140655 files and directories currently installed.) 2024-04-17T10:23:08.504 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-17T10:23:08.582 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T10:23:08.760 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-17T10:23:08.887 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-17T10:23:08.990 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-17T10:23:09.005 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-17T10:23:09.048 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T10:23:09.202 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T10:23:09.410 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package dump. 2024-04-17T10:23:09.426 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-17T10:23:09.543 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking dump (0.4b47-3) ... 2024-04-17T10:23:09.554 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package bison. 2024-04-17T10:23:09.631 INFO:teuthology.orchestra.run.smithi086.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 140655 files and directories currently installed.) 2024-04-17T10:23:09.636 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-17T10:23:09.707 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T10:23:09.830 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package indent. 2024-04-17T10:23:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:09.880 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T10:23:10.116 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-17T10:23:10.132 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-17T10:23:10.190 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T10:23:10.249 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-17T10:23:10.264 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-17T10:23:10.299 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T10:23:10.554 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-17T10:23:10.571 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-17T10:23:10.612 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T10:23:10.619 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package dump. 2024-04-17T10:23:10.635 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-17T10:23:10.686 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking dump (0.4b47-3) ... 2024-04-17T10:23:10.848 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-17T10:23:10.865 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-17T10:23:10.898 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T10:23:10.956 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package indent. 2024-04-17T10:23:10.972 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:11.014 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T10:23:11.252 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-17T10:23:11.269 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-17T10:23:11.317 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-17T10:23:11.318 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:11.333 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-17T10:23:11.375 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T10:23:11.529 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-17T10:23:11.546 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-17T10:23:11.579 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:11.721 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-17T10:23:11.737 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-17T10:23:11.771 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T10:23:11.849 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libjson-perl. 2024-04-17T10:23:11.865 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-17T10:23:11.907 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-17T10:23:12.024 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-17T10:23:12.039 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-17T10:23:12.123 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T10:23:12.210 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-17T10:23:12.226 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-17T10:23:12.268 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T10:23:12.469 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-17T10:23:12.485 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-17T10:23:12.527 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:12.813 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-17T10:23:12.829 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-17T10:23:12.871 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:13.141 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libjson-perl. 2024-04-17T10:23:13.156 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-17T10:23:13.190 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-17T10:23:13.555 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-17T10:23:13.561 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-17T10:23:13.571 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T10:23:13.576 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-17T10:23:13.605 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:13.619 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T10:23:13.815 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-17T10:23:13.832 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T10:23:13.874 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:14.218 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-17T10:23:14.235 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T10:23:14.276 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:14.621 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-17T10:23:14.637 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-17T10:23:14.679 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T10:23:14.990 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-17T10:23:15.006 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T10:23:15.048 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T10:23:15.377 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-17T10:23:15.393 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:15.435 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:15.490 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-17T10:23:15.506 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T10:23:15.548 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:15.796 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-17T10:23:15.809 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-17T10:23:15.812 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:15.825 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-17T10:23:15.854 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:15.859 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:16.107 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-17T10:23:16.123 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:16.157 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:16.229 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-17T10:23:16.244 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T10:23:16.287 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:16.409 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-17T10:23:16.426 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T10:23:16.468 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T10:23:16.657 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-17T10:23:16.672 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-17T10:23:16.715 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T10:23:16.988 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-17T10:23:16.994 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-17T10:23:17.004 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-17T10:23:17.008 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T10:23:17.046 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T10:23:17.059 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T10:23:17.408 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package modemmanager. 2024-04-17T10:23:17.424 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T10:23:17.446 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-17T10:23:17.461 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:17.466 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:17.504 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:17.882 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-17T10:23:17.897 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:17.940 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:18.097 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package network-manager. 2024-04-17T10:23:18.114 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T10:23:18.181 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-17T10:23:18.251 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-17T10:23:18.267 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:18.301 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:18.529 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-17T10:23:18.545 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-17T10:23:18.612 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T10:23:18.914 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package ppp. 2024-04-17T10:23:18.930 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-17T10:23:19.149 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-17T10:23:19.164 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-17T10:23:19.207 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T10:23:19.324 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T10:23:19.585 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package modemmanager. 2024-04-17T10:23:19.601 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-17T10:23:19.644 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:19.652 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package pptp-linux. 2024-04-17T10:23:19.666 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-17T10:23:19.710 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-17T10:23:19.946 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-17T10:23:19.961 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-17T10:23:19.996 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-17T10:23:20.249 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-17T10:23:20.264 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-17T10:23:20.275 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package network-manager. 2024-04-17T10:23:20.291 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-17T10:23:20.308 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-client-common (238) ... 2024-04-17T10:23:20.341 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-17T10:23:20.645 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-17T10:23:20.661 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:20.703 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:21.032 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package ppp. 2024-04-17T10:23:21.048 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-17T10:23:21.266 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-common. 2024-04-17T10:23:21.282 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-17T10:23:21.376 INFO:teuthology.orchestra.run.smithi067.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-17T10:23:21.406 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-common (238) ... 2024-04-17T10:23:21.418 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T10:23:21.779 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package pptp-linux. 2024-04-17T10:23:21.794 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-17T10:23:21.837 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-17T10:23:21.865 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-14. 2024-04-17T10:23:21.881 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:21.926 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:22.107 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-17T10:23:22.121 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-17T10:23:22.165 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-17T10:23:22.468 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-17T10:23:22.482 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-17T10:23:22.518 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking postgresql-client-common (238) ... 2024-04-17T10:23:22.822 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-17T10:23:22.836 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:22.901 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:23.151 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql. 2024-04-17T10:23:23.166 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-17T10:23:23.209 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql (14+238) ... 2024-04-17T10:23:23.443 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package postgresql-common. 2024-04-17T10:23:23.452 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-client. 2024-04-17T10:23:23.457 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-17T10:23:23.468 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-17T10:23:23.510 INFO:teuthology.orchestra.run.smithi086.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-17T10:23:23.528 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-client (14+238) ... 2024-04-17T10:23:23.541 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking postgresql-common (238) ... 2024-04-17T10:23:23.772 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-17T10:23:23.787 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-17T10:23:23.830 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-17T10:23:24.025 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package postgresql-14. 2024-04-17T10:23:24.039 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-17T10:23:24.085 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:24.149 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package tcl8.6. 2024-04-17T10:23:24.166 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-17T10:23:24.207 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T10:23:24.493 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package tcl. 2024-04-17T10:23:24.510 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-17T10:23:24.582 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-17T10:23:24.904 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package wpasupplicant. 2024-04-17T10:23:24.921 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-17T10:23:24.954 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T10:23:25.292 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-17T10:23:25.308 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-17T10:23:25.337 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package postgresql. 2024-04-17T10:23:25.351 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-17T10:23:25.383 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-17T10:23:25.395 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking postgresql (14+238) ... 2024-04-17T10:23:25.622 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package postgresql-client. 2024-04-17T10:23:25.637 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-17T10:23:25.672 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking postgresql-client (14+238) ... 2024-04-17T10:23:25.761 INFO:teuthology.orchestra.run.smithi067.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-17T10:23:25.778 INFO:teuthology.orchestra.run.smithi067.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-17T10:23:25.828 INFO:teuthology.orchestra.run.smithi067.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T10:23:25.907 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-17T10:23:25.922 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-17T10:23:25.965 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-17T10:23:26.131 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-client-common (238) ... 2024-04-17T10:23:26.310 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package tcl8.6. 2024-04-17T10:23:26.326 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-17T10:23:26.359 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-17T10:23:26.368 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T10:23:26.485 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T10:23:26.611 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:26.663 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package tcl. 2024-04-17T10:23:26.679 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-17T10:23:26.745 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T10:23:26.770 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-17T10:23:26.863 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T10:23:27.014 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:27.049 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package wpasupplicant. 2024-04-17T10:23:27.065 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-17T10:23:27.098 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T10:23:27.469 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-17T10:23:27.485 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-17T10:23:27.536 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-17T10:23:27.939 INFO:teuthology.orchestra.run.smithi086.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-17T10:23:27.955 INFO:teuthology.orchestra.run.smithi086.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-17T10:23:27.988 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-17T10:23:28.005 INFO:teuthology.orchestra.run.smithi086.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T10:23:28.103 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T10:23:28.312 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up postgresql-client-common (238) ... 2024-04-17T10:23:28.455 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T10:23:28.456 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T10:23:28.503 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-17T10:23:28.629 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-17T10:23:28.789 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:28.915 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-17T10:23:29.041 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-17T10:23:29.156 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T10:23:29.200 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:29.273 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T10:23:29.408 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T10:23:29.534 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-17T10:23:29.660 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T10:23:29.786 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:29.920 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T10:23:30.071 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T10:23:30.141 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-17T10:23:30.506 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-17T10:23:30.818 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T10:23:30.918 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T10:23:30.919 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-17T10:23:30.969 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up dump (0.4b47-3) ... 2024-04-17T10:23:31.064 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-17T10:23:31.137 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:31.239 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T10:23:31.340 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-17T10:23:31.440 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-17T10:23:31.567 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T10:23:31.608 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-17T10:23:31.656 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-17T10:23:31.734 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-17T10:23:31.760 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T10:23:31.844 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-17T10:23:31.886 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T10:23:31.961 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-17T10:23:32.062 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T10:23:32.096 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-17T10:23:32.172 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:32.273 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-17T10:23:32.355 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:32.407 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-client (14+238) ... 2024-04-17T10:23:32.501 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-17T10:23:32.567 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-17T10:23:32.657 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-17T10:23:33.156 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:33.156 INFO:teuthology.orchestra.run.smithi067.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-17T10:23:33.157 INFO:teuthology.orchestra.run.smithi067.stdout:which means they are currently configured by ifupdown: 2024-04-17T10:23:33.157 INFO:teuthology.orchestra.run.smithi067.stdout:- enp3s0f1 2024-04-17T10:23:33.157 INFO:teuthology.orchestra.run.smithi067.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-17T10:23:33.157 INFO:teuthology.orchestra.run.smithi067.stdout:remove their configuration from /etc/network/interfaces. 2024-04-17T10:23:33.157 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:33.296 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-17T10:23:33.304 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-17T10:23:33.422 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up dump (0.4b47-3) ... 2024-04-17T10:23:33.550 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-17T10:23:33.615 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:33.693 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-17T10:23:33.741 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-17T10:23:33.867 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-17T10:23:34.018 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-17T10:23:34.100 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-17T10:23:34.144 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-17T10:23:34.225 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-17T10:23:34.296 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-17T10:23:34.415 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-17T10:23:34.589 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-17T10:23:34.715 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:34.819 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:34.841 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-17T10:23:34.996 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:35.034 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up postgresql-client (14+238) ... 2024-04-17T10:23:35.097 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-17T10:23:35.160 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-17T10:23:35.198 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-common (238) ... 2024-04-17T10:23:35.777 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:35.777 INFO:teuthology.orchestra.run.smithi086.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-17T10:23:35.777 INFO:teuthology.orchestra.run.smithi086.stdout:which means they are currently configured by ifupdown: 2024-04-17T10:23:35.777 INFO:teuthology.orchestra.run.smithi086.stdout:- enp3s0f1 2024-04-17T10:23:35.777 INFO:teuthology.orchestra.run.smithi086.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-17T10:23:35.778 INFO:teuthology.orchestra.run.smithi086.stdout:remove their configuration from /etc/network/interfaces. 2024-04-17T10:23:35.778 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:35.909 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-17T10:23:36.154 INFO:teuthology.orchestra.run.smithi067.stdout:Adding user postgres to group ssl-cert 2024-04-17T10:23:36.309 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-17T10:23:36.405 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:36.405 INFO:teuthology.orchestra.run.smithi067.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-17T10:23:36.670 INFO:teuthology.orchestra.run.smithi067.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-17T10:23:36.671 INFO:teuthology.orchestra.run.smithi067.stdout:Removing obsolete dictionary files: 2024-04-17T10:23:36.709 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-17T10:23:37.390 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-17T10:23:37.421 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-17T10:23:37.556 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-17T10:23:37.715 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-17T10:23:37.866 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up postgresql-common (238) ... 2024-04-17T10:23:37.991 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:38.344 INFO:teuthology.orchestra.run.smithi067.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-17T10:23:38.361 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-17T10:23:38.415 INFO:teuthology.orchestra.run.smithi067.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-17T10:23:38.415 INFO:teuthology.orchestra.run.smithi067.stdout:This user must also own the server process. 2024-04-17T10:23:38.415 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:38.415 INFO:teuthology.orchestra.run.smithi067.stdout:The database cluster will be initialized with locale "en_US". 2024-04-17T10:23:38.416 INFO:teuthology.orchestra.run.smithi067.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-17T10:23:38.416 INFO:teuthology.orchestra.run.smithi067.stdout:The default text search configuration will be set to "english". 2024-04-17T10:23:38.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:38.416 INFO:teuthology.orchestra.run.smithi067.stdout:Data page checksums are disabled. 2024-04-17T10:23:38.416 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:38.416 INFO:teuthology.orchestra.run.smithi067.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-17T10:23:38.417 INFO:teuthology.orchestra.run.smithi067.stdout:creating subdirectories ... ok 2024-04-17T10:23:38.417 INFO:teuthology.orchestra.run.smithi067.stdout:selecting dynamic shared memory implementation ... posix 2024-04-17T10:23:38.441 INFO:teuthology.orchestra.run.smithi067.stdout:selecting default max_connections ... 100 2024-04-17T10:23:38.476 INFO:teuthology.orchestra.run.smithi067.stdout:selecting default shared_buffers ... 128MB 2024-04-17T10:23:38.490 INFO:teuthology.orchestra.run.smithi067.stdout:selecting default time zone ... Etc/UTC 2024-04-17T10:23:38.491 INFO:teuthology.orchestra.run.smithi067.stdout:creating configuration files ... ok 2024-04-17T10:23:38.663 INFO:teuthology.orchestra.run.smithi067.stdout:running bootstrap script ... ok 2024-04-17T10:23:38.839 INFO:teuthology.orchestra.run.smithi086.stdout:Adding user postgres to group ssl-cert 2024-04-17T10:23:39.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:39.084 INFO:teuthology.orchestra.run.smithi086.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-17T10:23:39.304 INFO:teuthology.orchestra.run.smithi067.stdout:performing post-bootstrap initialization ... ok 2024-04-17T10:23:39.323 INFO:teuthology.orchestra.run.smithi086.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-17T10:23:39.323 INFO:teuthology.orchestra.run.smithi086.stdout:Removing obsolete dictionary files: 2024-04-17T10:23:39.866 INFO:teuthology.orchestra.run.smithi067.stdout:syncing data to disk ... ok 2024-04-17T10:23:40.072 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-17T10:23:40.363 INFO:teuthology.orchestra.run.smithi067.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-17T10:23:40.544 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-17T10:23:41.056 INFO:teuthology.orchestra.run.smithi086.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-17T10:23:41.080 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout:This user must also own the server process. 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout:The database cluster will be initialized with locale "en_US". 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout:The default text search configuration will be set to "english". 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:41.129 INFO:teuthology.orchestra.run.smithi086.stdout:Data page checksums are disabled. 2024-04-17T10:23:41.130 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:41.130 INFO:teuthology.orchestra.run.smithi086.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-17T10:23:41.130 INFO:teuthology.orchestra.run.smithi086.stdout:creating subdirectories ... ok 2024-04-17T10:23:41.130 INFO:teuthology.orchestra.run.smithi086.stdout:selecting dynamic shared memory implementation ... posix 2024-04-17T10:23:41.154 INFO:teuthology.orchestra.run.smithi086.stdout:selecting default max_connections ... 100 2024-04-17T10:23:41.186 INFO:teuthology.orchestra.run.smithi086.stdout:selecting default shared_buffers ... 128MB 2024-04-17T10:23:41.200 INFO:teuthology.orchestra.run.smithi086.stdout:selecting default time zone ... Etc/UTC 2024-04-17T10:23:41.201 INFO:teuthology.orchestra.run.smithi086.stdout:creating configuration files ... ok 2024-04-17T10:23:41.365 INFO:teuthology.orchestra.run.smithi086.stdout:running bootstrap script ... ok 2024-04-17T10:23:41.978 INFO:teuthology.orchestra.run.smithi086.stdout:performing post-bootstrap initialization ... ok 2024-04-17T10:23:42.578 INFO:teuthology.orchestra.run.smithi086.stdout:syncing data to disk ... ok 2024-04-17T10:23:43.097 INFO:teuthology.orchestra.run.smithi086.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-17T10:23:43.340 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:43.441 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-17T10:23:43.567 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up postgresql (14+238) ... 2024-04-17T10:23:43.693 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:43.844 INFO:teuthology.orchestra.run.smithi067.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:44.098 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T10:23:44.098 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T10:23:44.824 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T10:23:44.908 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-17T10:23:45.218 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T10:23:45.412 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T10:23:46.060 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:46.169 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-17T10:23:46.303 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up postgresql (14+238) ... 2024-04-17T10:23:46.421 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-17T10:23:46.547 INFO:teuthology.orchestra.run.smithi086.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-17T10:23:46.806 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T10:23:46.806 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-17T10:23:46.890 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-17T10:23:46.999 INFO:teuthology.orchestra.run.smithi067.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T10:23:47.528 INFO:teuthology.orchestra.run.smithi067.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T10:23:47.537 DEBUG:teuthology.parallel:result is None 2024-04-17T10:23:47.549 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-17T10:23:47.642 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-17T10:23:47.977 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-17T10:23:48.120 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-17T10:23:49.990 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-17T10:23:50.090 INFO:teuthology.orchestra.run.smithi086.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-17T10:23:50.637 INFO:teuthology.orchestra.run.smithi086.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-17T10:23:50.643 DEBUG:teuthology.parallel:result is None 2024-04-17T10:23:50.644 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec 2024-04-17T10:23:50.953 DEBUG:teuthology.orchestra.run.smithi067:> dpkg-query -W -f '${Version}' ceph 2024-04-17T10:23:50.978 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:23:50.979 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:23:50.979 INFO:teuthology.task.install:The correct ceph version 19.0.0-1628-g97ae6cc9-1jammy is installed. 2024-04-17T10:23:50.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec 2024-04-17T10:23:51.121 DEBUG:teuthology.orchestra.run.smithi086:> dpkg-query -W -f '${Version}' ceph 2024-04-17T10:23:51.143 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:23:51.143 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1628-g97ae6cc9-1jammy 2024-04-17T10:23:51.143 INFO:teuthology.task.install:The correct ceph version 19.0.0-1628-g97ae6cc9-1jammy is installed. 2024-04-17T10:23:51.145 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-17T10:23:51.145 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:51.145 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-17T10:23:51.168 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:51.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-17T10:23:51.209 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-17T10:23:51.210 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:51.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-04-17T10:23:51.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-17T10:23:51.294 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:51.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-04-17T10:23:51.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-17T10:23:51.372 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-17T10:23:51.373 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:51.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-17T10:23:51.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-17T10:23:51.458 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:51.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-17T10:23:51.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-17T10:23:51.536 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-17T10:23:51.537 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:51.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-04-17T10:23:51.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-17T10:23:51.623 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:51.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-04-17T10:23:51.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-17T10:23:51.709 INFO:teuthology.run_tasks:Running task ceph... 2024-04-17T10:23:51.794 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-17T10:23:51.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2024-04-17T10:23:51.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2024-04-17T10:23:51.816 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-17T10:23:51.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-17T10:23:51.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-17T10:23:51.879 INFO:tasks.ceph:Creating extra log directories... 2024-04-17T10:23:51.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-17T10:23:51.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-17T10:23:51.951 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-17T10:23:51.951 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-17T10:23:51.951 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-17T10:23:51.951 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:51.951 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-17T10:23:51.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-17T10:23:52.088 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:52.088 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-17T10:23:52.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-17T10:23:52.189 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-17T10:23:52.189 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', 'Scrub error on inode', 'Behind on trimming', 'Metadata damage detected', 'bad backtrace on inode', 'overall HEALTH_', '\\(MDS_TRIM\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-17T10:23:52.189 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi/7660164', 'branch': 'wip-neesingh-testing-240416', 'description': 'fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/scrub}', 'email': 'neesingh@redhat.com', 'first_in_suite': False, 'job_id': '7660164', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'neesingh-2024-04-17_09:52:10-fs:functional-wip-neesingh-testing-240416-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-neesingh-testing-240416'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', 'Scrub error on inode', 'Behind on trimming', 'Metadata damage detected', 'bad backtrace on inode', 'overall HEALTH_', '\\(MDS_TRIM\\)'], 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec'}}, 'kclient': {'syntax': 'v2'}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-neesingh-testing-240416', 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec'}}, 'owner': 'scheduled_neesingh@teuthology', 'priority': 10, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 558, 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec', 'sleep_before_teardown': 0, 'suite': 'fs:functional', 'suite_branch': 'wip-neesingh-testing-240416', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKLCPHVKYlyBx3k3l7SXqRQ6otBkXAMeHzA5g6VnnLHyI2KFfcB0TNtDgws/3HGN8Q+sYxZiSuaJ3MlKUNWKtA0=', 'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLM9TefWVy1Os86VRIbo5G/yblWpdj3KdzfMm0byC7iyhNA3wQp7cXTqkLg39esSvEU4st03zx7g85jcOr8ZANY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'Replacing daemon mds', 'Scrub error on inode', 'Behind on trimming', 'Metadata damage detected', 'bad backtrace on inode', 'overall HEALTH_', '\\(MDS_TRIM\\)'], 'sha1': '97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_scrub_checks', 'tasks.cephfs.test_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-17_09:52:10', 'tube': 'smithi', 'user': 'neesingh', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-17T10:23:52.189 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-17T10:23:52.193 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-17T10:23:52.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2024-04-17T10:23:52.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2024-04-17T10:23:52.268 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:52.268 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-04-17T10:23:52.319 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-17T10:23:52.319 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-04-17T10:23:52.370 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-17T10:23:52.371 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:52.371 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 898 Links: 1 2024-04-17T10:23:52.371 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:52.371 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 10:17:19.625517331 +0000 2024-04-17T10:23:52.371 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 10:17:19.357519726 +0000 2024-04-17T10:23:52.371 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 10:17:19.357519726 +0000 2024-04-17T10:23:52.371 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 10:17:19.357519726 +0000 2024-04-17T10:23:52.371 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-17T10:23:52.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:23:52.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:23:52.433 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000403675 s, 1.3 MB/s 2024-04-17T10:23:52.435 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-17T10:23:52.488 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 915 Links: 1 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 10:17:20.193512259 +0000 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 10:17:19.905514831 +0000 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 10:17:19.905514831 +0000 2024-04-17T10:23:52.539 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 10:17:19.905514831 +0000 2024-04-17T10:23:52.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-17T10:23:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:23:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:23:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000374855 s, 1.4 MB/s 2024-04-17T10:23:52.604 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-17T10:23:52.657 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-04-17T10:23:52.706 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-17T10:23:52.707 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:52.707 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 933 Links: 1 2024-04-17T10:23:52.707 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:52.707 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 10:17:20.781507006 +0000 2024-04-17T10:23:52.707 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 10:17:20.497509544 +0000 2024-04-17T10:23:52.707 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 10:17:20.497509544 +0000 2024-04-17T10:23:52.707 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 10:17:20.497509544 +0000 2024-04-17T10:23:52.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-17T10:23:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:23:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:23:52.769 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000377026 s, 1.4 MB/s 2024-04-17T10:23:52.771 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-17T10:23:52.824 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-04-17T10:23:52.874 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-17T10:23:52.875 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:52.875 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 948 Links: 1 2024-04-17T10:23:52.875 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:52.875 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-04-17 10:17:21.353501898 +0000 2024-04-17T10:23:52.875 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-04-17 10:17:21.101504148 +0000 2024-04-17T10:23:52.875 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-04-17 10:17:21.101504148 +0000 2024-04-17T10:23:52.875 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-04-17 10:17:21.101504148 +0000 2024-04-17T10:23:52.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-17T10:23:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:23:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:23:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000380057 s, 1.3 MB/s 2024-04-17T10:23:52.939 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-17T10:23:52.993 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T10:23:52.993 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:52.993 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-04-17T10:23:53.002 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-17T10:23:53.002 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 887 Links: 1 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-17 10:17:18.376012771 +0000 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-17 10:17:18.092015307 +0000 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-17 10:17:18.092015307 +0000 2024-04-17T10:23:53.055 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-04-17 10:17:18.092015307 +0000 2024-04-17T10:23:53.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-17T10:23:53.120 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-17T10:23:53.121 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-17T10:23:53.121 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000397113 s, 1.3 MB/s 2024-04-17T10:23:53.123 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-17T10:23:53.180 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-04-17T10:23:53.234 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-17T10:23:53.235 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:53.235 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 901 Links: 1 2024-04-17T10:23:53.235 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:53.235 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-17 10:17:18.932007804 +0000 2024-04-17T10:23:53.235 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-17 10:17:18.668010162 +0000 2024-04-17T10:23:53.235 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-17 10:17:18.668010162 +0000 2024-04-17T10:23:53.235 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-04-17 10:17:18.668010162 +0000 2024-04-17T10:23:53.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-17T10:23:53.300 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-17T10:23:53.301 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-17T10:23:53.301 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000384064 s, 1.3 MB/s 2024-04-17T10:23:53.303 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-17T10:23:53.356 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 919 Links: 1 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-17 10:17:19.512002624 +0000 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-17 10:17:19.228005161 +0000 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-17 10:17:19.228005161 +0000 2024-04-17T10:23:53.411 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-04-17 10:17:19.228005161 +0000 2024-04-17T10:23:53.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-17T10:23:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-17T10:23:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-17T10:23:53.477 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000373347 s, 1.4 MB/s 2024-04-17T10:23:53.479 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-17T10:23:53.536 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-04-17T10:23:53.586 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-17T10:23:53.586 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-17T10:23:53.586 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 6h/6d Inode: 935 Links: 1 2024-04-17T10:23:53.586 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-17T10:23:53.586 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-04-17 10:17:20.055997765 +0000 2024-04-17T10:23:53.586 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-04-17 10:17:19.796000087 +0000 2024-04-17T10:23:53.587 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-04-17 10:17:19.796000087 +0000 2024-04-17T10:23:53.587 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-04-17 10:17:19.796000087 +0000 2024-04-17T10:23:53.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-17T10:23:53.653 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-04-17T10:23:53.653 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-04-17T10:23:53.653 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000390208 s, 1.3 MB/s 2024-04-17T10:23:53.655 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-17T10:23:53.708 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T10:23:53.709 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-17T10:23:53.709 INFO:tasks.ceph:Generating config... 2024-04-17T10:23:53.710 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-17T10:23:53.710 INFO:tasks.ceph:[client] debug client = 20 2024-04-17T10:23:53.710 INFO:tasks.ceph:[client] debug ms = 1 2024-04-17T10:23:53.710 INFO:tasks.ceph:[client] pid file = 2024-04-17T10:23:53.710 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-17T10:23:53.710 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-17T10:23:53.711 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] mds cache max size = 1000 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] mds log max segments = 1 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-17T10:23:53.711 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-17T10:23:53.712 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-17T10:23:53.712 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-17T10:23:53.712 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-17T10:23:53.712 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-17T10:23:53.712 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-17T10:23:53.713 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-17T10:23:53.713 INFO:tasks.ceph:Setting up mon.a... 2024-04-17T10:23:53.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-17T10:23:53.760 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2024-04-17T10:23:53.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-17T10:23:53.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T10:23:53.929 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.b', '172.21.15.86'), ('mon.c', '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]')] 2024-04-17T10:23:53.930 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-17T10:23:53.931 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:53.931 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-17T10:23:53.982 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --add b 172.21.15.86 --addv c '[v2:172.21.15.86:3301,v1:172.21.15.86:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:23:54.077 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 8e8d1faf-cb51-4015-918b-f894d5d28e99 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 8e8d1faf-cb51-4015-918b-f894d5d28e99 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2024-04-17T10:23:54.082991+0000 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-04-17T10:23:54.082991+0000 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 17 (quincy) 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] mon.c 2024-04-17T10:23:54.078 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-17T10:23:54.080 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-17T10:23:54.089 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8e8d1faf-cb51-4015-918b-f894d5d28e99... 2024-04-17T10:23:54.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-17T10:23:54.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2024-04-17T10:23:54.173 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.174 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.175 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]" 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 8e8d1faf-cb51-4015-918b-f894d5d28e99 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2024-04-17T10:23:54.176 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.177 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 20 2024-04-17T10:23:54.178 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 20 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2024-04-17T10:23:54.179 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2024-04-17T10:23:54.180 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2024-04-17T10:23:54.181 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2024-04-17T10:23:54.182 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds balancer = 20 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2024-04-17T10:23:54.183 INFO:teuthology.orchestra.run.smithi067.stdout: mds cache max size = 1000 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: mds log max segments = 1 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2024-04-17T10:23:54.184 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.185 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2024-04-17T10:23:54.186 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2024-04-17T10:23:54.187 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "172.21.15.67,172.21.15.86,[v2:172.21.15.86:3301,v1:172.21.15.86:6790]" 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on skipped message = False 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = 8e8d1faf-cb51-4015-918b-f894d5d28e99 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2024-04-17T10:23:54.188 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2024-04-17T10:23:54.189 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug trim objects = true 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2024-04-17T10:23:54.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.191 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 20 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 20 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2024-04-17T10:23:54.192 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2024-04-17T10:23:54.193 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2024-04-17T10:23:54.193 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2024-04-17T10:23:54.193 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2024-04-17T10:23:54.193 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2024-04-17T10:23:54.193 INFO:teuthology.orchestra.run.smithi086.stdout: osd op complaint time = 180 2024-04-17T10:23:54.193 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.193 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2024-04-17T10:23:54.194 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-04-17T10:23:54.195 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2024-04-17T10:23:54.196 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: mon op complaint time = 120 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-04-17T10:23:54.197 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: client mount timeout = 600 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: debug client = 20 2024-04-17T10:23:54.198 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = "" 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout: rados mon op timeout = 900 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout: rados osd op timeout = 900 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.a] 2024-04-17T10:23:54.199 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.c] 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.b] 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.d] 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout:[mds] 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout: debug mds = 20 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout: debug mds balancer = 20 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout: mds cache max size = 1000 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout: mds debug frag = True 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout: mds debug scatterstat = True 2024-04-17T10:23:54.200 INFO:teuthology.orchestra.run.smithi086.stdout: mds log max segments = 1 2024-04-17T10:23:54.201 INFO:teuthology.orchestra.run.smithi086.stdout: mds op complaint time = 180 2024-04-17T10:23:54.201 INFO:teuthology.orchestra.run.smithi086.stdout: mds verify scatter = True 2024-04-17T10:23:54.201 INFO:teuthology.orchestra.run.smithi086.stdout: osd op complaint time = 180 2024-04-17T10:23:54.201 INFO:teuthology.orchestra.run.smithi086.stdout: rados mon op timeout = 900 2024-04-17T10:23:54.201 INFO:teuthology.orchestra.run.smithi086.stdout: rados osd op timeout = 900 2024-04-17T10:23:54.203 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-17T10:23:54.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-17T10:23:54.303 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-17T10:23:54.303 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:54.303 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-17T10:23:54.312 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:54.313 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-17T10:23:54.365 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:23:54.365 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:54.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-17T10:23:54.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T10:23:54.437 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:23:54.437 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:23:54.489 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:23:54.490 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:54.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-17T10:23:54.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-17T10:23:54.525 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:23:54.525 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:23:54.578 INFO:tasks.ceph:Setting up mon nodes... 2024-04-17T10:23:54.579 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-17T10:23:54.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-17T10:23:54.634 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-17T10:23:54.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-17T10:23:54.698 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-17T10:23:54.713 INFO:tasks.ceph:Setting up mds nodes... 2024-04-17T10:23:54.713 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-17T10:23:54.769 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-17T10:23:54.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-17T10:23:54.794 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-17T10:23:54.892 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-17T10:23:54.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-17T10:23:54.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-17T10:23:54.975 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-17T10:23:54.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-17T10:23:54.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-17T10:23:55.096 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-17T10:23:55.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-17T10:23:55.120 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-17T10:23:55.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-17T10:23:55.163 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-17T10:23:55.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-17T10:23:55.263 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-17T10:23:55.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-17T10:23:55.321 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-17T10:23:55.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-17T10:23:55.422 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-17T10:23:55.438 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-17T10:23:55.439 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-17T10:23:55.439 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-17T10:23:55.459 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T10:23:55.459 INFO:tasks.ceph:role: osd.0 2024-04-17T10:23:55.459 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:23:55.459 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-17T10:23:55.528 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:23:55.529 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:23:55.928 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T10:23:55.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T10:23:55.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-17T10:23:56.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-17T10:23:56.051 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:23:56.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:23:56.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-17T10:23:56.114 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T10:23:56.114 INFO:tasks.ceph:role: osd.1 2024-04-17T10:23:56.115 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:23:56.115 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:23:56.186 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:23:56.187 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:23:56.580 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T10:23:56.582 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T10:23:56.583 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-17T10:23:56.705 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-17T10:23:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:23:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:23:56.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-17T10:23:56.787 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T10:23:56.787 INFO:tasks.ceph:role: osd.2 2024-04-17T10:23:56.787 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:23:56.787 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-17T10:23:56.857 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:23:56.858 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:23:57.269 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T10:23:57.272 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T10:23:57.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-17T10:23:57.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-17T10:23:57.408 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:23:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:23:57.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-17T10:23:57.471 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-17T10:23:57.471 INFO:tasks.ceph:role: osd.3 2024-04-17T10:23:57.471 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:23:57.471 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-17T10:23:57.542 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:23:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:23:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:23:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:23:57.543 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:23:57.543 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:23:57.544 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:23:57.544 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:23:57.544 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:23:57.544 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:23:57.909 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2024-04-17T10:23:57.911 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2024-04-17T10:23:57.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-17T10:23:57.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-17T10:23:58.054 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:23:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:23:58.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:23:58.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:23:58.220+0000 7f984ea127c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-17T10:23:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:23:58.224+0000 7f984ea127c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-17T10:23:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:23:58.224+0000 7f984ea127c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T10:23:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:23:58.224+0000 7f984ea127c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T10:23:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:23:58.224+0000 7f984ea127c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-17T10:23:58.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:23:58.228+0000 7f984ea127c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-17T10:24:01.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-17T10:24:01.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:01.396+0000 7f43b3d137c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-17T10:24:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:01.396+0000 7f43b3d137c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-17T10:24:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:01.396+0000 7f43b3d137c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T10:24:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:01.396+0000 7f43b3d137c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T10:24:01.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:01.396+0000 7f43b3d137c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-17T10:24:01.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:01.396+0000 7f43b3d137c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-17T10:24:04.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-17T10:24:04.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:04.544+0000 7fd991c727c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-17T10:24:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:04.544+0000 7fd991c727c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-17T10:24:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:04.544+0000 7fd991c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T10:24:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:04.544+0000 7fd991c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T10:24:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:04.544+0000 7fd991c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-17T10:24:04.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:04.548+0000 7fd991c727c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-17T10:24:07.591 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-17T10:24:07.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:07.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:07.704+0000 7fb704ef77c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-17T10:24:07.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:07.708+0000 7fb704ef77c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-17T10:24:07.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:07.708+0000 7fb704ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T10:24:07.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:07.708+0000 7fb704ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T10:24:07.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:07.708+0000 7fb704ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-17T10:24:07.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:07.708+0000 7fb704ef77c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-17T10:24:10.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-17T10:24:10.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-17T10:24:10.827 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T10:24:10.828 INFO:tasks.ceph:role: osd.4 2024-04-17T10:24:10.828 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:24:10.828 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-17T10:24:10.901 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:24:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:24:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:24:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:24:10.901 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:24:10.901 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:24:10.901 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:24:10.902 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:24:10.902 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:24:10.902 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:24:11.314 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-04-17T10:24:11.316 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-04-17T10:24:11.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-17T10:24:11.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-17T10:24:11.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:24:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:24:11.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-17T10:24:11.533 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T10:24:11.533 INFO:tasks.ceph:role: osd.5 2024-04-17T10:24:11.533 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:24:11.533 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:24:11.601 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:24:11.602 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:24:11.602 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:24:11.961 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-04-17T10:24:11.964 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-04-17T10:24:11.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-17T10:24:12.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-17T10:24:12.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:24:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:24:12.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-17T10:24:12.162 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T10:24:12.162 INFO:tasks.ceph:role: osd.6 2024-04-17T10:24:12.162 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:24:12.162 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-17T10:24:12.237 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:24:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:24:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:24:12.237 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:24:12.237 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:24:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:24:12.238 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:24:12.238 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:24:12.238 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:24:12.238 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:24:12.638 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-04-17T10:24:12.640 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-04-17T10:24:12.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-17T10:24:12.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-17T10:24:12.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:24:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:24:12.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-17T10:24:12.825 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-17T10:24:12.826 INFO:tasks.ceph:role: osd.7 2024-04-17T10:24:12.826 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:24:12.826 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-17T10:24:12.898 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-17T10:24:12.899 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-17T10:24:12.899 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-17T10:24:13.291 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-04-17T10:24:13.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-04-17T10:24:13.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-17T10:24:13.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-17T10:24:13.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /sbin/restorecon: command not found 2024-04-17T10:24:13.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:24:13.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:13.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:13.545+0000 7f4cc0f037c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-17T10:24:13.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:13.553+0000 7f4cc0f037c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-17T10:24:13.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:13.553+0000 7f4cc0f037c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T10:24:13.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:13.553+0000 7f4cc0f037c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T10:24:13.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:13.553+0000 7f4cc0f037c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-17T10:24:13.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:13.561+0000 7f4cc0f037c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-17T10:24:16.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-17T10:24:16.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:16.718 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:16.721+0000 7f28be93c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-17T10:24:16.719 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:16.721+0000 7f28be93c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-17T10:24:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:16.721+0000 7f28be93c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T10:24:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:16.721+0000 7f28be93c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T10:24:16.720 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:16.721+0000 7f28be93c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-17T10:24:16.721 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:16.721+0000 7f28be93c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-17T10:24:19.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-17T10:24:19.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:19.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:19.897+0000 7fdcd02467c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-17T10:24:19.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:19.901+0000 7fdcd02467c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-17T10:24:19.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:19.901+0000 7fdcd02467c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T10:24:19.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:19.901+0000 7fdcd02467c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T10:24:19.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:19.901+0000 7fdcd02467c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-17T10:24:19.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:19.901+0000 7fdcd02467c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-17T10:24:22.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:24:22.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:24:22.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-17T10:24:22.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:23.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:23.069+0000 7f2dccd017c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-17T10:24:23.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:23.069+0000 7f2dccd017c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-17T10:24:23.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:23.069+0000 7f2dccd017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T10:24:23.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:23.069+0000 7f2dccd017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T10:24:23.068 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:23.069+0000 7f2dccd017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-17T10:24:23.069 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:23.069+0000 7f2dccd017c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-17T10:24:26.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-17T10:24:26.111 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-17T10:24:26.111 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-17T10:24:26.134 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-17T10:24:26.199 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.199 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-17T10:24:26.262 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-17T10:24:26.326 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-17T10:24:26.390 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-17T10:24:26.454 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-17T10:24:26.518 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-17T10:24:26.539 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-17T10:24:26.606 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-17T10:24:26.671 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-17T10:24:26.734 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-17T10:24:26.798 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-17T10:24:26.867 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-17T10:24:26.931 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.931 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-17T10:24:26.941 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:26.941 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-17T10:24:26.991 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:26.991 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-17T10:24:27.000 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:27.001 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-17T10:24:27.051 INFO:tasks.ceph:Adding keys to all mons... 2024-04-17T10:24:27.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-17T10:24:27.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-17T10:24:27.098 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2024-04-17T10:24:27.099 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw== 2024-04-17T10:24:27.099 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2024-04-17T10:24:27.099 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g== 2024-04-17T10:24:27.099 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2024-04-17T10:24:27.099 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ== 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ== 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig== 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA== 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw== 2024-04-17T10:24:27.100 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2024-04-17T10:24:27.101 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg== 2024-04-17T10:24:27.101 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2024-04-17T10:24:27.101 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw== 2024-04-17T10:24:27.101 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2024-04-17T10:24:27.101 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ== 2024-04-17T10:24:27.101 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2024-04-17T10:24:27.102 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ== 2024-04-17T10:24:27.102 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2024-04-17T10:24:27.102 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g== 2024-04-17T10:24:27.102 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2024-04-17T10:24:27.102 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw== 2024-04-17T10:24:27.102 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2024-04-17T10:24:27.103 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ== 2024-04-17T10:24:27.103 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:24:27.103 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:24:27.103 INFO:teuthology.orchestra.run.smithi067.stdout:[client.2] 2024-04-17T10:24:27.103 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA== 2024-04-17T10:24:27.103 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2024-04-17T10:24:27.104 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA== 2024-04-17T10:24:27.104 INFO:teuthology.orchestra.run.smithi067.stdout:[client.3] 2024-04-17T10:24:27.104 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g== 2024-04-17T10:24:27.112 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.y] 2024-04-17T10:24:27.112 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.a] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.c] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg== 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.b] 2024-04-17T10:24:27.113 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw== 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout:[mds.d] 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ== 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ== 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g== 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.6] 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw== 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.7] 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ== 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:24:27.114 INFO:teuthology.orchestra.run.smithi086.stdout:[client.2] 2024-04-17T10:24:27.115 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA== 2024-04-17T10:24:27.115 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-04-17T10:24:27.115 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA== 2024-04-17T10:24:27.115 INFO:teuthology.orchestra.run.smithi086.stdout:[client.3] 2024-04-17T10:24:27.115 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g== 2024-04-17T10:24:27.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T10:24:27.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T10:24:27.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T10:24:27.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-17T10:24:27.621 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-17T10:24:27.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.854 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-17T10:24:27.968 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:27.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:28.027 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:28.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:28.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:28.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:28.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:28.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-17T10:24:28.205 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-17T10:24:28.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-17T10:24:28.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T10:24:28.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-17T10:24:28.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-17T10:24:28.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T10:24:28.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-17T10:24:28.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-17T10:24:28.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-17T10:24:28.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-17T10:24:28.690 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:28.694 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-17T10:24:28.745 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-17T10:24:28.746 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-17T10:24:28.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-17T10:24:28.749 INFO:tasks.ceph.mon.a:Started 2024-04-17T10:24:28.749 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-17T10:24:28.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-17T10:24:28.788 INFO:tasks.ceph.mon.b:Started 2024-04-17T10:24:28.788 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-17T10:24:28.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-17T10:24:28.791 INFO:tasks.ceph.mon.c:Started 2024-04-17T10:24:28.791 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-17T10:24:28.792 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-17T10:24:28.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-17T10:24:28.795 INFO:tasks.ceph.mgr.y:Started 2024-04-17T10:24:28.795 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-17T10:24:28.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-17T10:24:28.798 INFO:tasks.ceph.mgr.x:Started 2024-04-17T10:24:28.798 INFO:tasks.ceph:Setting crush tunables to default 2024-04-17T10:24:28.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2024-04-17T10:24:28.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77e4c00640 1 Processor -- start 2024-04-17T10:24:28.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77e4c00640 1 -- start start 2024-04-17T10:24:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77e4c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77e015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77e4c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 0x7f77e0099ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77e4c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e015f450 con 0x7f77e015f900 2024-04-17T10:24:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77e4c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e00cccb0 con 0x7f77e015f020 2024-04-17T10:24:28.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77e4c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e00ccf90 con 0x7f77e00b2560 2024-04-17T10:24:28.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77df400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f77e015f900 0x7f77e015fce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38288/0 (socket says 172.21.15.67:38288) 2024-04-17T10:24:28.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77df400640 1 -- 172.21.15.67:0/1803220848 learned_addr learned my addr 172.21.15.67:0/1803220848 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77d7e00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77e015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.940+0000 7f77dfe00640 1 --2- 172.21.15.67:0/1803220848 >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 0x7f77e0099ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77d7e00640 1 -- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 msgr2=0x7f77e015eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77d7e00640 1 -- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 msgr2=0x7f77e015eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77d7e00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77e015eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77d7e00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77e015eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77dfe00640 1 -- 172.21.15.67:0/1803220848 >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 msgr2=0x7f77e0099ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77dfe00640 1 -- 172.21.15.67:0/1803220848 >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 msgr2=0x7f77e0099ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77dfe00640 1 --2- 172.21.15.67:0/1803220848 >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 0x7f77e0099ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-17T10:24:28.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:28.944+0000 7f77dfe00640 1 --2- 172.21.15.67:0/1803220848 >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 0x7f77e0099ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-17T10:24:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110480473 0 0) 0x7f77d0001120 con 0x7f77e015f900 2024-04-17T10:24:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c4003530 con 0x7f77e015f900 2024-04-17T10:24:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 80139646 0 0) 0x7f77c4003530 con 0x7f77e015f900 2024-04-17T10:24:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 msgr2=0x7f77e015eae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T10:24:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77e015eae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 msgr2=0x7f77e0099ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T10:24:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 --2- 172.21.15.67:0/1803220848 >> v2:172.21.15.86:3300/0 conn(0x7f77e015f020 0x7f77e0099ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e00cd270 con 0x7f77e015f900 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f77d0001850 con 0x7f77e015f900 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 >> v1:172.21.15.67:6789/0 conn(0x7f77e015f900 legacy=0x7f77e015fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77c40039c0 0x7f77c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77c4004810 0x7f77c4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e00cccb0 con 0x7f77c4004810 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e00ccf90 con 0x7f77c40039c0 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c400bb00 con 0x7f77e00b2560 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77d7e00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77c4004810 0x7f77c4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77df400640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77c40039c0 0x7f77c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dfe00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77df400640 1 -- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 msgr2=0x7f77c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77df400640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77c40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77df400640 1 -- 172.21.15.67:0/1803220848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77c4004810 msgr2=0x7f77c4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77df400640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77c4004810 0x7f77c4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77df400640 1 -- 172.21.15.67:0/1803220848 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c4003530 con 0x7f77c40039c0 2024-04-17T10:24:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77d7e00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77c4004810 0x7f77c4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.108+0000 7f77dfe00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77e00b2560 0x7f77c40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:29.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77df400640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77c40039c0 0x7f77c4003de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f77d0001060 tx=0x7f77d001f2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9de448f67555a841 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d000c0e0 con 0x7f77c40039c0 2024-04-17T10:24:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d0005ae0 con 0x7f77c40039c0 2024-04-17T10:24:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77dea00640 1 -- 172.21.15.67:0/1803220848 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0007070 con 0x7f77c40039c0 2024-04-17T10:24:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77e4c00640 1 -- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77c40039c0 msgr2=0x7f77c4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77e4c00640 1 --2- 172.21.15.67:0/1803220848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77c40039c0 0x7f77c4003de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f77d0001060 tx=0x7f77d001f2f0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77dfe00640 1 -- 172.21.15.67:0/1803220848 reap_dead start 2024-04-17T10:24:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77e4c00640 1 -- 172.21.15.67:0/1803220848 shutdown_connections 2024-04-17T10:24:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77e4c00640 1 -- 172.21.15.67:0/1803220848 >> 172.21.15.67:0/1803220848 conn(0x7f77e00b3180 msgr2=0x7f77e00b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77e4c00640 1 -- 172.21.15.67:0/1803220848 shutdown_connections 2024-04-17T10:24:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.112+0000 7f77e4c00640 1 -- 172.21.15.67:0/1803220848 wait complete. 2024-04-17T10:24:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 Processor -- start 2024-04-17T10:24:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 -- start start 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77c40039c0 0x7f77e00cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e00cb850 0x7f77e0159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e015a440 0x7f77e0070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e01603f0 con 0x7f77e00cb850 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e00cd550 con 0x7f77e015a440 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77e4c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e0160570 con 0x7f77c40039c0 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77c40039c0 0x7f77e00cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e00cb850 0x7f77e0159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77c40039c0 0x7f77e00cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:59494/0 (socket says 172.21.15.67:59494) 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 -- 172.21.15.67:0/2332673449 learned_addr learned my addr 172.21.15.67:0/2332673449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77dfe00640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e015a440 0x7f77e0070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 -- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e015a440 msgr2=0x7f77e0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e015a440 0x7f77e0070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 -- 172.21.15.67:0/2332673449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e00cb850 msgr2=0x7f77e0159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e00cb850 0x7f77e0159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77df400640 1 -- 172.21.15.67:0/2332673449 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e015d0a0 con 0x7f77c40039c0 2024-04-17T10:24:29.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.116+0000 7f77d7e00640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e00cb850 0x7f77e0159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77df400640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77c40039c0 0x7f77e00cb310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77d001f120 tx=0x7f77c4005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77d7400640 1 -- 172.21.15.67:0/2332673449 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0005c90 con 0x7f77c40039c0 2024-04-17T10:24:29.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77d7400640 1 -- 172.21.15.67:0/2332673449 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d0005e30 con 0x7f77c40039c0 2024-04-17T10:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77d7400640 1 -- 172.21.15.67:0/2332673449 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0004a10 con 0x7f77c40039c0 2024-04-17T10:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e015d380 con 0x7f77c40039c0 2024-04-17T10:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e015d940 con 0x7f77c40039c0 2024-04-17T10:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77d7400640 1 -- 172.21.15.67:0/2332673449 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f77d00b0420 con 0x7f77c40039c0 2024-04-17T10:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77d7400640 1 -- 172.21.15.67:0/2332673449 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f77d00bcac0 con 0x7f77c40039c0 2024-04-17T10:24:29.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.120+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e004f780 con 0x7f77c40039c0 2024-04-17T10:24:29.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.132+0000 7f77d7400640 1 -- 172.21.15.67:0/2332673449 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f77d00c1070 con 0x7f77c40039c0 2024-04-17T10:24:29.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.192+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f77e00cdb20 con 0x7f77c40039c0 2024-04-17T10:24:29.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77d7400640 1 -- 172.21.15.67:0/2332673449 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f77e015f450 con 0x7f77c40039c0 2024-04-17T10:24:29.202 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-04-17T10:24:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77c40039c0 msgr2=0x7f77e00cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77c40039c0 0x7f77e00cb310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77d001f120 tx=0x7f77c4005840 comp rx=0 tx=0).stop 2024-04-17T10:24:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 shutdown_connections 2024-04-17T10:24:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77e015a440 0x7f77e0070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e00cb850 0x7f77e0159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 --2- 172.21.15.67:0/2332673449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77c40039c0 0x7f77e00cb310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 >> 172.21.15.67:0/2332673449 conn(0x7f77e00b3180 msgr2=0x7f77e00cfd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:29.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 shutdown_connections 2024-04-17T10:24:29.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:29.204+0000 7f77e4c00640 1 -- 172.21.15.67:0/2332673449 wait complete. 2024-04-17T10:24:29.223 INFO:tasks.ceph:check_enable_crimson: False 2024-04-17T10:24:29.224 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-17T10:24:29.224 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:29.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-17T10:24:29.244 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:29.244 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-17T10:24:29.303 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:29.304+0000 7f385702e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T10:24:29.305 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:29.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-17T10:24:29.312 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:29.316+0000 7fedc4d5a180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-17T10:24:29.364 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:24:29.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-17T10:24:29.428 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:29.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-17T10:24:29.449 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:29.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-17T10:24:29.509 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:29.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-17T10:24:29.569 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:24:29.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-17T10:24:29.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new e7c33b47-11fd-4bd1-86a2-18d1179d12e4 0 2024-04-17T10:24:29.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dc0e00640 1 Processor -- start 2024-04-17T10:24:29.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dc0e00640 1 -- start start 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dc0e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dc0e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5dbc0ad080 0x7f5dbc0ad510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dc0e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dbc0b36c0 con 0x7f5dbc06c310 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dc0e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dbc15ce20 con 0x7f5dbc0ad080 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dc0e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0b4d80 con 0x7f5dbc06bf30 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5db3e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5dbc06c310 0x7f5dbc0b4130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51978/0 (socket says 172.21.15.86:51978) 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5db3e00640 1 -- 172.21.15.86:0/1418052509 learned_addr learned my addr 172.21.15.86:0/1418052509 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-04-17T10:24:29.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5db9e00640 1 --2- 172.21.15.86:0/1418052509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.812+0000 7f5dba800640 1 --2- 172.21.15.86:0/1418052509 >> v2:172.21.15.67:3300/0 conn(0x7f5dbc0ad080 0x7f5dbc0ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9400640 1 -- 172.21.15.86:0/1418052509 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1140658497 0 0) 0x7f5dbc0b36c0 con 0x7f5dbc06c310 2024-04-17T10:24:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9400640 1 -- 172.21.15.86:0/1418052509 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da0003530 con 0x7f5dbc06c310 2024-04-17T10:24:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9e00640 1 -- 172.21.15.86:0/1418052509 >> v2:172.21.15.67:3300/0 conn(0x7f5dbc0ad080 msgr2=0x7f5dbc0ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9e00640 1 --2- 172.21.15.86:0/1418052509 >> v2:172.21.15.67:3300/0 conn(0x7f5dbc0ad080 0x7f5dbc0ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9e00640 1 -- 172.21.15.86:0/1418052509 >> v1:172.21.15.86:6789/0 conn(0x7f5dbc06c310 legacy=0x7f5dbc0b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9e00640 1 -- 172.21.15.86:0/1418052509 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc0b5020 con 0x7f5dbc06bf30 2024-04-17T10:24:29.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9e00640 1 --2- 172.21.15.86:0/1418052509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc15c8e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0097c0 tx=0x7f5dac01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c62a999594eb4179 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:29.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9400640 1 -- 172.21.15.86:0/1418052509 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0b2070 con 0x7f5dbc06bf30 2024-04-17T10:24:29.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9400640 1 -- 172.21.15.86:0/1418052509 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac0ad640 con 0x7f5dbc06bf30 2024-04-17T10:24:29.814 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.816+0000 7f5db9400640 1 -- 172.21.15.86:0/1418052509 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0ad9f0 con 0x7f5dbc06bf30 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1418052509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 msgr2=0x7f5dbc15c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 --2- 172.21.15.86:0/1418052509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc15c8e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0097c0 tx=0x7f5dac01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1418052509 shutdown_connections 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 --2- 172.21.15.86:0/1418052509 >> v2:172.21.15.67:3300/0 conn(0x7f5dbc0ad080 0x7f5dbc0ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 --2- 172.21.15.86:0/1418052509 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc15c8e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1418052509 >> 172.21.15.86:0/1418052509 conn(0x7f5dbc0b2710 msgr2=0x7f5dbc0cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1418052509 shutdown_connections 2024-04-17T10:24:29.815 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1418052509 wait complete. 2024-04-17T10:24:29.816 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 Processor -- start 2024-04-17T10:24:29.817 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- start start 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dbc06c310 0x7f5dbc1462f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dbc0ad080 0x7f5dbc157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0b3160 con 0x7f5dbc0ad080 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc0b2fe0 con 0x7f5dbc06c310 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dc0e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc06b4a0 con 0x7f5dbc06bf30 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dbc06c310 0x7f5dbc1462f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc1526c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57870/0 (socket says 172.21.15.86:57870) 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 -- 172.21.15.86:0/1376025310 learned_addr learned my addr 172.21.15.86:0/1376025310 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5dba800640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dbc0ad080 0x7f5dbc157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 -- 172.21.15.86:0/1376025310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dbc06c310 msgr2=0x7f5dbc1462f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dbc06c310 0x7f5dbc1462f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 -- 172.21.15.86:0/1376025310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dbc0ad080 msgr2=0x7f5dbc157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dbc0ad080 0x7f5dbc157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.820+0000 7f5db9e00640 1 -- 172.21.15.86:0/1376025310 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc151620 con 0x7f5dbc06bf30 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5db9e00640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc1526c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5dac01a0d0 tx=0x7f5dac019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5db2000640 1 -- 172.21.15.86:0/1376025310 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac0b2040 con 0x7f5dbc06bf30 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5db2000640 1 -- 172.21.15.86:0/1376025310 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac0ad440 con 0x7f5dbc06bf30 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5db2000640 1 -- 172.21.15.86:0/1376025310 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac006e20 con 0x7f5dbc06bf30 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dbc151900 con 0x7f5dbc06bf30 2024-04-17T10:24:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dbc151ec0 con 0x7f5dbc06bf30 2024-04-17T10:24:29.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5db2000640 1 -- 172.21.15.86:0/1376025310 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5dac0be020 con 0x7f5dbc06bf30 2024-04-17T10:24:29.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5db2000640 1 -- 172.21.15.86:0/1376025310 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5dac0c07b0 con 0x7f5dbc06bf30 2024-04-17T10:24:29.821 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.824+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dbc067f10 con 0x7f5dbc06bf30 2024-04-17T10:24:29.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.828+0000 7f5db2000640 1 -- 172.21.15.86:0/1376025310 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5dac016090 con 0x7f5dbc06bf30 2024-04-17T10:24:29.823 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:29.824+0000 7f385702e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T10:24:29.848 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:29.852+0000 7fedc4d5a180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-17T10:24:29.889 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.892+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e7c33b47-11fd-4bd1-86a2-18d1179d12e4", "id": 0} v 0) v1 -- 0x7f5dbc152c00 con 0x7f5dbc06bf30 2024-04-17T10:24:29.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.900+0000 7f5db2000640 1 -- 172.21.15.86:0/1376025310 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7c33b47-11fd-4bd1-86a2-18d1179d12e4", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5dac0ad5f0 con 0x7f5dbc06bf30 2024-04-17T10:24:29.898 INFO:teuthology.orchestra.run.smithi086.stdout:0 2024-04-17T10:24:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 msgr2=0x7f5dbc1526c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc1526c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5dac01a0d0 tx=0x7f5dac019240 comp rx=0 tx=0).stop 2024-04-17T10:24:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 shutdown_connections 2024-04-17T10:24:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5dbc0ad080 0x7f5dbc157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5dbc06c310 0x7f5dbc1462f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 --2- 172.21.15.86:0/1376025310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5dbc06bf30 0x7f5dbc1526c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 >> 172.21.15.86:0/1376025310 conn(0x7f5dbc0b2710 msgr2=0x7f5dbc159a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 shutdown_connections 2024-04-17T10:24:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:29.904+0000 7f5dc0e00640 1 -- 172.21.15.86:0/1376025310 wait complete. 2024-04-17T10:24:29.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 526adc26-4eee-4b2c-86e3-6e55d021ca82 1 2024-04-17T10:24:30.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcdcc00640 1 Processor -- start 2024-04-17T10:24:30.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcdcc00640 1 -- start start 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcdcc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd80ce210 0x7fbcd80cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcdcc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcd80ce5f0 0x7fbcd815ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcdcc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcd80b7430 con 0x7fbcd815bfb0 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcdcc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd815c560 con 0x7fbcd80ce5f0 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcdcc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd815c840 con 0x7fbcd80ce210 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd5e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd80ce210 0x7fbcd80cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd6800640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbcd815bfb0 0x7fbcd80cf440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51988/0 (socket says 172.21.15.86:51988) 2024-04-17T10:24:30.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd6800640 1 -- 172.21.15.86:0/3008938198 learned_addr learned my addr 172.21.15.86:0/3008938198 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-04-17T10:24:30.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbccfe00640 1 --2- 172.21.15.86:0/3008938198 >> v2:172.21.15.67:3300/0 conn(0x7fbcd80ce5f0 0x7fbcd815ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd5e00640 1 -- 172.21.15.86:0/3008938198 >> v2:172.21.15.67:3300/0 conn(0x7fbcd80ce5f0 msgr2=0x7fbcd815ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:30.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd5e00640 1 --2- 172.21.15.86:0/3008938198 >> v2:172.21.15.67:3300/0 conn(0x7fbcd80ce5f0 0x7fbcd815ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd5e00640 1 -- 172.21.15.86:0/3008938198 >> v1:172.21.15.86:6789/0 conn(0x7fbcd815bfb0 legacy=0x7fbcd80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd5e00640 1 -- 172.21.15.86:0/3008938198 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd815cb20 con 0x7fbcd80ce210 2024-04-17T10:24:30.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbccfe00640 1 --2- 172.21.15.86:0/3008938198 >> v2:172.21.15.67:3300/0 conn(0x7fbcd80ce5f0 0x7fbcd815ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:24:30.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.056+0000 7fbcd5400640 1 -- 172.21.15.86:0/3008938198 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031844337 0 0) 0x7fbcd00014a0 con 0x7fbcd815bfb0 2024-04-17T10:24:30.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcd5e00640 1 --2- 172.21.15.86:0/3008938198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd80ce210 0x7fbcd80cc8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80097c0 tx=0x7fbcc801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7dfa5fc38994c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcd5400640 1 -- 172.21.15.86:0/3008938198 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc80b1070 con 0x7fbcd80ce210 2024-04-17T10:24:30.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcd5400640 1 -- 172.21.15.86:0/3008938198 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc80ac410 con 0x7fbcd80ce210 2024-04-17T10:24:30.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcd5400640 1 -- 172.21.15.86:0/3008938198 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc80ac7e0 con 0x7fbcd80ce210 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 -- 172.21.15.86:0/3008938198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd80ce210 msgr2=0x7fbcd80cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 --2- 172.21.15.86:0/3008938198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd80ce210 0x7fbcd80cc8d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80097c0 tx=0x7fbcc801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 -- 172.21.15.86:0/3008938198 shutdown_connections 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 --2- 172.21.15.86:0/3008938198 >> v2:172.21.15.67:3300/0 conn(0x7fbcd80ce5f0 0x7fbcd815ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 --2- 172.21.15.86:0/3008938198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd80ce210 0x7fbcd80cc8d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 -- 172.21.15.86:0/3008938198 >> 172.21.15.86:0/3008938198 conn(0x7fbcd80b3460 msgr2=0x7fbcd80b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 -- 172.21.15.86:0/3008938198 shutdown_connections 2024-04-17T10:24:30.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.060+0000 7fbcdcc00640 1 -- 172.21.15.86:0/3008938198 wait complete. 2024-04-17T10:24:30.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 Processor -- start 2024-04-17T10:24:30.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 -- start start 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd80ce210 0x7fbcd80c92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd80ce5f0 0x7fbcd81654e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd815bfb0 0x7fbcd806cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8168900 con 0x7fbcd80ce5f0 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8168780 con 0x7fbcd80ce210 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcdcc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8168a80 con 0x7fbcd815bfb0 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd80ce210 0x7fbcd80c92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd80ce210 0x7fbcd80c92e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38036/0 (socket says 172.21.15.86:38036) 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 -- 172.21.15.86:0/2739105576 learned_addr learned my addr 172.21.15.86:0/2739105576 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd6800640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd815bfb0 0x7fbcd806cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.064 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbccfe00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd80ce5f0 0x7fbcd81654e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 -- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd815bfb0 msgr2=0x7fbcd806cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd815bfb0 0x7fbcd806cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 -- 172.21.15.86:0/2739105576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd80ce5f0 msgr2=0x7fbcd81654e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd80ce5f0 0x7fbcd81654e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.064+0000 7fbcd5e00640 1 -- 172.21.15.86:0/2739105576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd806c2a0 con 0x7fbcd80ce210 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcd5e00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd80ce210 0x7fbcd80c92e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80af040 tx=0x7fbcc8019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcce000640 1 -- 172.21.15.86:0/2739105576 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc80b1040 con 0x7fbcd80ce210 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcce000640 1 -- 172.21.15.86:0/2739105576 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc80bb070 con 0x7fbcd80ce210 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcce000640 1 -- 172.21.15.86:0/2739105576 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc80b5670 con 0x7fbcd80ce210 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd806c580 con 0x7fbcd80ce210 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd8150e40 con 0x7fbcd80ce210 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcce000640 1 -- 172.21.15.86:0/2739105576 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fbcc80b5810 con 0x7fbcd80ce210 2024-04-17T10:24:30.065 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcce000640 1 -- 172.21.15.86:0/2739105576 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fbcc80c0ad0 con 0x7fbcd80ce210 2024-04-17T10:24:30.066 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd804ef30 con 0x7fbcd80ce210 2024-04-17T10:24:30.066 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:30.068+0000 7fedc4d5a180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T10:24:30.067 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.072+0000 7fbcce000640 1 -- 172.21.15.86:0/2739105576 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fbcc8016090 con 0x7fbcd80ce210 2024-04-17T10:24:30.076 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:30.076+0000 7f385702e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-17T10:24:30.134 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.136+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "526adc26-4eee-4b2c-86e3-6e55d021ca82", "id": 1} v 0) v1 -- 0x7fbcd8151010 con 0x7fbcd80ce210 2024-04-17T10:24:30.142 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.144+0000 7fbcce000640 1 -- 172.21.15.86:0/2739105576 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "526adc26-4eee-4b2c-86e3-6e55d021ca82", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbcc80c0cd0 con 0x7fbcd80ce210 2024-04-17T10:24:30.142 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd80ce210 msgr2=0x7fbcd80c92e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd80ce210 0x7fbcd80c92e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbcc80af040 tx=0x7fbcc8019240 comp rx=0 tx=0).stop 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 shutdown_connections 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcd815bfb0 0x7fbcd806cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcd80ce5f0 0x7fbcd81654e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 --2- 172.21.15.86:0/2739105576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcd80ce210 0x7fbcd80c92e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 >> 172.21.15.86:0/2739105576 conn(0x7fbcd80b3460 msgr2=0x7fbcd8157b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 shutdown_connections 2024-04-17T10:24:30.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.148+0000 7fbcdcc00640 1 -- 172.21.15.86:0/2739105576 wait complete. 2024-04-17T10:24:30.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 65a857d0-8abf-46ac-9056-b12b997ec561 2 2024-04-17T10:24:30.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d8600640 1 Processor -- start 2024-04-17T10:24:30.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d8600640 1 -- start start 2024-04-17T10:24:30.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d8600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0d015eda0 0x7fe0d00aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d8600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d00b3230 con 0x7fe0d00b4ea0 2024-04-17T10:24:30.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d8600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d00b3510 con 0x7fe0d015a7f0 2024-04-17T10:24:30.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d8600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d00b37f0 con 0x7fe0d015eda0 2024-04-17T10:24:30.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d6600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0d015eda0 0x7fe0d00aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d5c00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe0d015a7f0 0x7fe0d00b4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51992/0 (socket says 172.21.15.86:51992) 2024-04-17T10:24:30.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d5c00640 1 -- 172.21.15.86:0/3556755420 learned_addr learned my addr 172.21.15.86:0/3556755420 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-04-17T10:24:30.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d6600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0d015eda0 0x7fe0d00aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38046/0 (socket says 172.21.15.86:38046) 2024-04-17T10:24:30.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d6600640 1 -- 172.21.15.86:0/3556755420 >> v1:172.21.15.86:6789/0 conn(0x7fe0d015a7f0 legacy=0x7fe0d00b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d6600640 1 -- 172.21.15.86:0/3556755420 >> v1:172.21.15.67:6789/0 conn(0x7fe0d00b4ea0 legacy=0x7fe0d00ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.300+0000 7fe0d6600640 1 -- 172.21.15.86:0/3556755420 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d00b3ad0 con 0x7fe0d015eda0 2024-04-17T10:24:30.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d6600640 1 --2- 172.21.15.86:0/3556755420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015eda0 0x7fe0d00aeb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800bfa0 tx=0x7fe0c801eb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3781a1fce15e065e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0cfe00640 1 -- 172.21.15.86:0/3556755420 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c800b2d0 con 0x7fe0d015eda0 2024-04-17T10:24:30.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0cfe00640 1 -- 172.21.15.86:0/3556755420 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c80a7930 con 0x7fe0d015eda0 2024-04-17T10:24:30.301 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0cfe00640 1 -- 172.21.15.86:0/3556755420 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c80a7d00 con 0x7fe0d015eda0 2024-04-17T10:24:30.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d8600640 1 -- 172.21.15.86:0/3556755420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015eda0 msgr2=0x7fe0d00aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d8600640 1 --2- 172.21.15.86:0/3556755420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015eda0 0x7fe0d00aeb10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe0c800bfa0 tx=0x7fe0c801eb40 comp rx=0 tx=0).stop 2024-04-17T10:24:30.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d8600640 1 -- 172.21.15.86:0/3556755420 shutdown_connections 2024-04-17T10:24:30.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d8600640 1 --2- 172.21.15.86:0/3556755420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015eda0 0x7fe0d00aeb10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d8600640 1 -- 172.21.15.86:0/3556755420 >> 172.21.15.86:0/3556755420 conn(0x7fe0d00b2710 msgr2=0x7fe0d00cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d8600640 1 -- 172.21.15.86:0/3556755420 shutdown_connections 2024-04-17T10:24:30.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.304+0000 7fe0d8600640 1 -- 172.21.15.86:0/3556755420 wait complete. 2024-04-17T10:24:30.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 Processor -- start 2024-04-17T10:24:30.304 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 -- start start 2024-04-17T10:24:30.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d00b4ea0 0x7fe0d014c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015a7f0 0x7fe0d014a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d015eda0 0x7fe0d01517e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00b3010 con 0x7fe0d015eda0 2024-04-17T10:24:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00b2e90 con 0x7fe0d015a7f0 2024-04-17T10:24:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d006c450 con 0x7fe0d00b4ea0 2024-04-17T10:24:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015a7f0 0x7fe0d014a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d5c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d00b4ea0 0x7fe0d014c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015a7f0 0x7fe0d014a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38058/0 (socket says 172.21.15.86:38058) 2024-04-17T10:24:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d5200640 1 -- 172.21.15.86:0/2309853692 learned_addr learned my addr 172.21.15.86:0/2309853692 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:30.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d6600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d015eda0 0x7fe0d01517e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d6600640 1 -- 172.21.15.86:0/2309853692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d00b4ea0 msgr2=0x7fe0d014c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d6600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d00b4ea0 0x7fe0d014c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d6600640 1 -- 172.21.15.86:0/2309853692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015a7f0 msgr2=0x7fe0d014a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d6600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015a7f0 0x7fe0d014a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d6600640 1 -- 172.21.15.86:0/2309853692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d014cf40 con 0x7fe0d015eda0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d5200640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015a7f0 0x7fe0d014a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d6600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d015eda0 0x7fe0d01517e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80ad000 tx=0x7fe0c801b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0cd600640 1 -- 172.21.15.86:0/2309853692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8009750 con 0x7fe0d015eda0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0cd600640 1 -- 172.21.15.86:0/2309853692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c8009910 con 0x7fe0d015eda0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d00af500 con 0x7fe0d015eda0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.308+0000 7fe0cd600640 1 -- 172.21.15.86:0/2309853692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8009d00 con 0x7fe0d015eda0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.312+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d00af7a0 con 0x7fe0d015eda0 2024-04-17T10:24:30.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.312+0000 7fe0cd600640 1 -- 172.21.15.86:0/2309853692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe0c80c0020 con 0x7fe0d015eda0 2024-04-17T10:24:30.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.312+0000 7fe0cd600640 1 -- 172.21.15.86:0/2309853692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe0c8009310 con 0x7fe0d015eda0 2024-04-17T10:24:30.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.312+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d004ef30 con 0x7fe0d015eda0 2024-04-17T10:24:30.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.332+0000 7fe0cd600640 1 -- 172.21.15.86:0/2309853692 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe0c80c7020 con 0x7fe0d015eda0 2024-04-17T10:24:30.396 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.400+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd new", "uuid": "65a857d0-8abf-46ac-9056-b12b997ec561", "id": 2} v 0) v1 -- 0x7fe0d0154bd0 con 0x7fe0d015eda0 2024-04-17T10:24:30.399 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0cd600640 1 -- 172.21.15.86:0/2309853692 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "65a857d0-8abf-46ac-9056-b12b997ec561", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe0c8018760 con 0x7fe0d015eda0 2024-04-17T10:24:30.400 INFO:teuthology.orchestra.run.smithi086.stdout:2 2024-04-17T10:24:30.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d015eda0 msgr2=0x7fe0d01517e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d015eda0 0x7fe0d01517e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe0c80ad000 tx=0x7fe0c801b8e0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 shutdown_connections 2024-04-17T10:24:30.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0d015eda0 0x7fe0d01517e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0d015a7f0 0x7fe0d014a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 --2- 172.21.15.86:0/2309853692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0d00b4ea0 0x7fe0d014c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 >> 172.21.15.86:0/2309853692 conn(0x7fe0d00b2710 msgr2=0x7fe0d006e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 shutdown_connections 2024-04-17T10:24:30.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.404+0000 7fe0d8600640 1 -- 172.21.15.86:0/2309853692 wait complete. 2024-04-17T10:24:30.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 7da79f1b-4b06-4704-9fbb-4100785acd2a 3 2024-04-17T10:24:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.556+0000 7fd2da000640 1 Processor -- start 2024-04-17T10:24:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.556+0000 7fd2da000640 1 -- start start 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2da000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd2d406f2b0 0x7fd2d406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2da000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d40ad8f0 con 0x7fd2d40b3550 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2da000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d406f850 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2da000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d406fb30 con 0x7fd2d406f2b0 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2d3400640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd2d40b3aa0 0x7fd2d40ae670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51994/0 (socket says 172.21.15.86:51994) 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2d3400640 1 -- 172.21.15.86:0/1347022286 learned_addr learned my addr 172.21.15.86:0/1347022286 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2d3e00640 1 --2- 172.21.15.86:0/1347022286 >> v2:172.21.15.86:3300/0 conn(0x7fd2d406f2b0 0x7fd2d406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2d3e00640 1 -- 172.21.15.86:0/1347022286 >> v1:172.21.15.86:6789/0 conn(0x7fd2d40b3aa0 legacy=0x7fd2d40ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2d3e00640 1 -- 172.21.15.86:0/1347022286 >> v1:172.21.15.67:6789/0 conn(0x7fd2d40b3550 legacy=0x7fd2d40b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.556 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.560+0000 7fd2d3e00640 1 -- 172.21.15.86:0/1347022286 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40adb90 con 0x7fd2d406f2b0 2024-04-17T10:24:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2d3e00640 1 --2- 172.21.15.86:0/1347022286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d406f2b0 0x7fd2d406ac90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009d90 tx=0x7fd2cc01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed39af261750d874 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2d2a00640 1 -- 172.21.15.86:0/1347022286 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0b2070 con 0x7fd2d406f2b0 2024-04-17T10:24:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2d2a00640 1 -- 172.21.15.86:0/1347022286 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc0a57a0 con 0x7fd2d406f2b0 2024-04-17T10:24:30.559 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2d2a00640 1 -- 172.21.15.86:0/1347022286 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0a5b70 con 0x7fd2d406f2b0 2024-04-17T10:24:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- 172.21.15.86:0/1347022286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d406f2b0 msgr2=0x7fd2d406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 --2- 172.21.15.86:0/1347022286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d406f2b0 0x7fd2d406ac90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009d90 tx=0x7fd2cc01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- 172.21.15.86:0/1347022286 shutdown_connections 2024-04-17T10:24:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 --2- 172.21.15.86:0/1347022286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d406f2b0 0x7fd2d406ac90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- 172.21.15.86:0/1347022286 >> 172.21.15.86:0/1347022286 conn(0x7fd2d40b2710 msgr2=0x7fd2d40cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- 172.21.15.86:0/1347022286 shutdown_connections 2024-04-17T10:24:30.560 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- 172.21.15.86:0/1347022286 wait complete. 2024-04-17T10:24:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 Processor -- start 2024-04-17T10:24:30.562 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- start start 2024-04-17T10:24:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2d406f2b0 0x7fd2d4150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.564 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d40b3550 0x7fd2d414b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d40b3aa0 0x7fd2d4151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d406b760 con 0x7fd2d40b3550 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d406b5e0 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.564+0000 7fd2da000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4168d10 con 0x7fd2d406f2b0 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d40b3aa0 0x7fd2d4151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2d406f2b0 0x7fd2d4150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d40b3aa0 0x7fd2d4151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38068/0 (socket says 172.21.15.86:38068) 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 -- 172.21.15.86:0/3948032052 learned_addr learned my addr 172.21.15.86:0/3948032052 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2cbe00640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d40b3550 0x7fd2d414b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 -- 172.21.15.86:0/3948032052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2d406f2b0 msgr2=0x7fd2d4150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2d406f2b0 0x7fd2d4150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 -- 172.21.15.86:0/3948032052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d40b3550 msgr2=0x7fd2d414b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d40b3550 0x7fd2d414b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 -- 172.21.15.86:0/3948032052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d414f930 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.565 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2d3e00640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d40b3aa0 0x7fd2d4151d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0b0040 tx=0x7fd2cc019680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2cb400640 1 -- 172.21.15.86:0/3948032052 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0b2050 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2cb400640 1 -- 172.21.15.86:0/3948032052 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc0be070 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2cb400640 1 -- 172.21.15.86:0/3948032052 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc0ada90 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d414fc10 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d41597c0 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2cb400640 1 -- 172.21.15.86:0/3948032052 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd2cc0adc30 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2cb400640 1 -- 172.21.15.86:0/3948032052 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fd2cc0a56a0 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.566 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.568+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d4067f10 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.569 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.572+0000 7fd2cb400640 1 -- 172.21.15.86:0/3948032052 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd2cc016560 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.640+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7da79f1b-4b06-4704-9fbb-4100785acd2a", "id": 3} v 0) v1 -- 0x7fd2d406ae30 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2cb400640 1 -- 172.21.15.86:0/3948032052 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7da79f1b-4b06-4704-9fbb-4100785acd2a", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd2cc0c3460 con 0x7fd2d40b3aa0 2024-04-17T10:24:30.644 INFO:teuthology.orchestra.run.smithi086.stdout:3 2024-04-17T10:24:30.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d40b3aa0 msgr2=0x7fd2d4151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d40b3aa0 0x7fd2d4151d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc0b0040 tx=0x7fd2cc019680 comp rx=0 tx=0).stop 2024-04-17T10:24:30.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 shutdown_connections 2024-04-17T10:24:30.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2d40b3aa0 0x7fd2d4151d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d40b3550 0x7fd2d414b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 --2- 172.21.15.86:0/3948032052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2d406f2b0 0x7fd2d4150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 >> 172.21.15.86:0/3948032052 conn(0x7fd2d40b2710 msgr2=0x7fd2d4073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 shutdown_connections 2024-04-17T10:24:30.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.648+0000 7fd2da000640 1 -- 172.21.15.86:0/3948032052 wait complete. 2024-04-17T10:24:30.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new d77c3c57-bcdc-40aa-8383-1e284bf39505 4 2024-04-17T10:24:30.694 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:30.696+0000 7fedc4d5a180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T10:24:30.725 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:30.724+0000 7f385702e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-17T10:24:30.758 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:30.760+0000 7fedc4d5a180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T10:24:30.790 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:30.792+0000 7f385702e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-17T10:24:30.797 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.800+0000 7fcfbe400640 1 Processor -- start 2024-04-17T10:24:30.798 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.800+0000 7fcfbe400640 1 -- start start 2024-04-17T10:24:30.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfbe400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfb815f020 0x7fcfb815c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfbe400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb80cfe60 0x7fcfb8099ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfbe400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb80b1980 con 0x7fcfb815ec40 2024-04-17T10:24:30.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfbe400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb815d780 con 0x7fcfb815f020 2024-04-17T10:24:30.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfbe400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb80b22b0 con 0x7fcfb80cfe60 2024-04-17T10:24:30.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7400640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcfb815ec40 0x7fcfb815d070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:52002/0 (socket says 172.21.15.86:52002) 2024-04-17T10:24:30.799 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7400640 1 -- 172.21.15.86:0/3264261367 learned_addr learned my addr 172.21.15.86:0/3264261367 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-04-17T10:24:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb80cfe60 0x7fcfb8099ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfafe00640 1 --2- 172.21.15.86:0/3264261367 >> v2:172.21.15.67:3300/0 conn(0x7fcfb815f020 0x7fcfb815c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb6a00640 1 -- 172.21.15.86:0/3264261367 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260884229 0 0) 0x7fcfb80b1980 con 0x7fcfb815ec40 2024-04-17T10:24:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb6a00640 1 -- 172.21.15.86:0/3264261367 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf9c003530 con 0x7fcfb815ec40 2024-04-17T10:24:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7e00640 1 -- 172.21.15.86:0/3264261367 >> v2:172.21.15.67:3300/0 conn(0x7fcfb815f020 msgr2=0x7fcfb815c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:30.800 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7e00640 1 --2- 172.21.15.86:0/3264261367 >> v2:172.21.15.67:3300/0 conn(0x7fcfb815f020 0x7fcfb815c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7e00640 1 -- 172.21.15.86:0/3264261367 >> v1:172.21.15.86:6789/0 conn(0x7fcfb815ec40 legacy=0x7fcfb815d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7e00640 1 -- 172.21.15.86:0/3264261367 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb80b1c60 con 0x7fcfb80cfe60 2024-04-17T10:24:30.801 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfafe00640 1 --2- 172.21.15.86:0/3264261367 >> v2:172.21.15.67:3300/0 conn(0x7fcfb815f020 0x7fcfb815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:30.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.804+0000 7fcfb7e00640 1 --2- 172.21.15.86:0/3264261367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb80cfe60 0x7fcfb8099ba0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009d90 tx=0x7fcfb001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771cda6a321d64b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfb6a00640 1 -- 172.21.15.86:0/3264261367 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00b2070 con 0x7fcfb80cfe60 2024-04-17T10:24:30.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfb6a00640 1 -- 172.21.15.86:0/3264261367 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb00a57a0 con 0x7fcfb80cfe60 2024-04-17T10:24:30.803 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfb6a00640 1 -- 172.21.15.86:0/3264261367 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00a5b70 con 0x7fcfb80cfe60 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- 172.21.15.86:0/3264261367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb80cfe60 msgr2=0x7fcfb8099ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 --2- 172.21.15.86:0/3264261367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb80cfe60 0x7fcfb8099ba0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009d90 tx=0x7fcfb001c8e0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- 172.21.15.86:0/3264261367 shutdown_connections 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 --2- 172.21.15.86:0/3264261367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb80cfe60 0x7fcfb8099ba0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 --2- 172.21.15.86:0/3264261367 >> v2:172.21.15.67:3300/0 conn(0x7fcfb815f020 0x7fcfb815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- 172.21.15.86:0/3264261367 >> 172.21.15.86:0/3264261367 conn(0x7fcfb80b3410 msgr2=0x7fcfb80b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- 172.21.15.86:0/3264261367 shutdown_connections 2024-04-17T10:24:30.804 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- 172.21.15.86:0/3264261367 wait complete. 2024-04-17T10:24:30.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 Processor -- start 2024-04-17T10:24:30.805 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- start start 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb80cfe60 0x7fcfb80b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfb815ec40 0x7fcfb80b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb815f020 0x7fcfb80c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb80cebd0 con 0x7fcfb80cfe60 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb80cea50 con 0x7fcfb815ec40 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfbe400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb80b2720 con 0x7fcfb815f020 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfb7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb815f020 0x7fcfb80c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfafe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfb815ec40 0x7fcfb80b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.807 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfb7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb80cfe60 0x7fcfb80b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfb7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb815f020 0x7fcfb80c5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57916/0 (socket says 172.21.15.86:57916) 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.808+0000 7fcfb7e00640 1 -- 172.21.15.86:0/2780248712 learned_addr learned my addr 172.21.15.86:0/2780248712 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfb7e00640 1 -- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfb815ec40 msgr2=0x7fcfb80b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfb7e00640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfb815ec40 0x7fcfb80b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfb7e00640 1 -- 172.21.15.86:0/2780248712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb80cfe60 msgr2=0x7fcfb80b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfb7e00640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb80cfe60 0x7fcfb80b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfb7e00640 1 -- 172.21.15.86:0/2780248712 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb80c67b0 con 0x7fcfb815f020 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfb7400640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb80cfe60 0x7fcfb80b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:30.808 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfafe00640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfb815ec40 0x7fcfb80b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:30.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfb7e00640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb815f020 0x7fcfb80c5af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfb001af90 tx=0x7fcfb0019680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:30.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfaf400640 1 -- 172.21.15.86:0/2780248712 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00b2050 con 0x7fcfb815f020 2024-04-17T10:24:30.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfaf400640 1 -- 172.21.15.86:0/2780248712 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb00be070 con 0x7fcfb815f020 2024-04-17T10:24:30.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfaf400640 1 -- 172.21.15.86:0/2780248712 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb00ad850 con 0x7fcfb815f020 2024-04-17T10:24:30.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb80c6a90 con 0x7fcfb815f020 2024-04-17T10:24:30.809 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb80c7050 con 0x7fcfb815f020 2024-04-17T10:24:30.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfaf400640 1 -- 172.21.15.86:0/2780248712 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fcfb00ad9f0 con 0x7fcfb815f020 2024-04-17T10:24:30.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfaf400640 1 -- 172.21.15.86:0/2780248712 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fcfb00adce0 con 0x7fcfb815f020 2024-04-17T10:24:30.810 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.812+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb804ef30 con 0x7fcfb815f020 2024-04-17T10:24:30.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.816+0000 7fcfaf400640 1 -- 172.21.15.86:0/2780248712 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fcfb0016560 con 0x7fcfb815f020 2024-04-17T10:24:30.822 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:30.824+0000 7fedc4d5a180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T10:24:30.854 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:30.856+0000 7f385702e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-17T10:24:30.877 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.880+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d77c3c57-bcdc-40aa-8383-1e284bf39505", "id": 4} v 0) v1 -- 0x7fcfb8167620 con 0x7fcfb815f020 2024-04-17T10:24:30.883 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfaf400640 1 -- 172.21.15.86:0/2780248712 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d77c3c57-bcdc-40aa-8383-1e284bf39505", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcfb00c3460 con 0x7fcfb815f020 2024-04-17T10:24:30.884 INFO:teuthology.orchestra.run.smithi086.stdout:4 2024-04-17T10:24:30.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb815f020 msgr2=0x7fcfb80c5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:30.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb815f020 0x7fcfb80c5af0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfb001af90 tx=0x7fcfb0019680 comp rx=0 tx=0).stop 2024-04-17T10:24:30.885 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 shutdown_connections 2024-04-17T10:24:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfb815f020 0x7fcfb80c5af0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfb815ec40 0x7fcfb80b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 --2- 172.21.15.86:0/2780248712 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfb80cfe60 0x7fcfb80b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 >> 172.21.15.86:0/2780248712 conn(0x7fcfb80b3410 msgr2=0x7fcfb80b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 shutdown_connections 2024-04-17T10:24:30.886 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:30.888+0000 7fcfbe400640 1 -- 172.21.15.86:0/2780248712 wait complete. 2024-04-17T10:24:30.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new c2c739f2-870b-4bc2-8861-6f788264da32 5 2024-04-17T10:24:30.914 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:30.916+0000 7fedc4d5a180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T10:24:30.948 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:30.948+0000 7f385702e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-17T10:24:30.977 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:30.980+0000 7fedc4d5a180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T10:24:31.012 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:31.012+0000 7f385702e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-17T10:24:31.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.052+0000 7f09ed600640 1 Processor -- start 2024-04-17T10:24:31.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.052+0000 7f09ed600640 1 -- start start 2024-04-17T10:24:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.052+0000 7f09ed600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09e806ac90 0x7f09e815c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.052+0000 7f09ed600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e806b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09ed600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e81564f0 con 0x7f09e815d3e0 2024-04-17T10:24:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09ed600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e806b5b0 con 0x7f09e806ac90 2024-04-17T10:24:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09ed600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e80cc520 con 0x7f09e80c3de0 2024-04-17T10:24:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09e7400640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f09e815d3e0 0x7f09e80cd5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:52006/0 (socket says 172.21.15.86:52006) 2024-04-17T10:24:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09e7400640 1 -- 172.21.15.86:0/1615241291 learned_addr learned my addr 172.21.15.86:0/1615241291 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-04-17T10:24:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e806b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09e6a00640 1 --2- 172.21.15.86:0/1615241291 >> v2:172.21.15.67:3300/0 conn(0x7f09e806ac90 0x7f09e815c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09e6000640 1 -- 172.21.15.86:0/1615241291 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051621345 0 0) 0x7f09e00014a0 con 0x7f09e815d3e0 2024-04-17T10:24:31.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09dfe00640 1 -- 172.21.15.86:0/1615241291 >> v2:172.21.15.67:3300/0 conn(0x7f09e806ac90 msgr2=0x7f09e815c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:31.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09dfe00640 1 --2- 172.21.15.86:0/1615241291 >> v2:172.21.15.67:3300/0 conn(0x7f09e806ac90 0x7f09e815c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09dfe00640 1 -- 172.21.15.86:0/1615241291 >> v1:172.21.15.86:6789/0 conn(0x7f09e815d3e0 legacy=0x7f09e80cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09dfe00640 1 -- 172.21.15.86:0/1615241291 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e80cc800 con 0x7f09e80c3de0 2024-04-17T10:24:31.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09e6a00640 1 --2- 172.21.15.86:0/1615241291 >> v2:172.21.15.67:3300/0 conn(0x7f09e806ac90 0x7f09e815c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09dfe00640 1 --2- 172.21.15.86:0/1615241291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e806b070 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097f0 tx=0x7f09d001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5f05bf2b68b071f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.056+0000 7f09e6000640 1 -- 172.21.15.86:0/1615241291 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d00a7070 con 0x7f09e80c3de0 2024-04-17T10:24:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09e6000640 1 -- 172.21.15.86:0/1615241291 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d00b0070 con 0x7f09e80c3de0 2024-04-17T10:24:31.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09e6000640 1 -- 172.21.15.86:0/1615241291 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d00ab890 con 0x7f09e80c3de0 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- 172.21.15.86:0/1615241291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 msgr2=0x7f09e806b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 --2- 172.21.15.86:0/1615241291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e806b070 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097f0 tx=0x7f09d001c470 comp rx=0 tx=0).stop 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- 172.21.15.86:0/1615241291 shutdown_connections 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 --2- 172.21.15.86:0/1615241291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e806b070 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 --2- 172.21.15.86:0/1615241291 >> v2:172.21.15.67:3300/0 conn(0x7f09e806ac90 0x7f09e815c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- 172.21.15.86:0/1615241291 >> 172.21.15.86:0/1615241291 conn(0x7f09e80b22a0 msgr2=0x7f09e80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- 172.21.15.86:0/1615241291 shutdown_connections 2024-04-17T10:24:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- 172.21.15.86:0/1615241291 wait complete. 2024-04-17T10:24:31.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 Processor -- start 2024-04-17T10:24:31.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- start start 2024-04-17T10:24:31.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e806ac90 0x7f09e80cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e814eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e815d3e0 0x7f09e814f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e81691b0 con 0x7f09e806ac90 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e806bad0 con 0x7f09e815d3e0 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09ed600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e8169330 con 0x7f09e80c3de0 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e814eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09e7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e815d3e0 0x7f09e814f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09dfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e814eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57932/0 (socket says 172.21.15.86:57932) 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09dfe00640 1 -- 172.21.15.86:0/758429170 learned_addr learned my addr 172.21.15.86:0/758429170 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.060+0000 7f09e6a00640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e806ac90 0x7f09e80cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dfe00640 1 -- 172.21.15.86:0/758429170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e815d3e0 msgr2=0x7f09e814f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dfe00640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e815d3e0 0x7f09e814f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dfe00640 1 -- 172.21.15.86:0/758429170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e806ac90 msgr2=0x7f09e80cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dfe00640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e806ac90 0x7f09e80cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dfe00640 1 -- 172.21.15.86:0/758429170 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e80d0160 con 0x7f09e80c3de0 2024-04-17T10:24:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dfe00640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e814eb20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097f0 tx=0x7f09d001acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dea00640 1 -- 172.21.15.86:0/758429170 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d00abe60 con 0x7f09e80c3de0 2024-04-17T10:24:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dea00640 1 -- 172.21.15.86:0/758429170 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d0004240 con 0x7f09e80c3de0 2024-04-17T10:24:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dea00640 1 -- 172.21.15.86:0/758429170 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0003620 con 0x7f09e80c3de0 2024-04-17T10:24:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e815ac60 con 0x7f09e80c3de0 2024-04-17T10:24:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e815b220 con 0x7f09e80c3de0 2024-04-17T10:24:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dea00640 1 -- 172.21.15.86:0/758429170 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f09d0003b50 con 0x7f09e80c3de0 2024-04-17T10:24:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09dea00640 1 -- 172.21.15.86:0/758429170 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f09d00bfc40 con 0x7f09e80c3de0 2024-04-17T10:24:31.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.064+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e804ef30 con 0x7f09e80c3de0 2024-04-17T10:24:31.063 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.068+0000 7f09dea00640 1 -- 172.21.15.86:0/758429170 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f09d0016090 con 0x7f09e80c3de0 2024-04-17T10:24:31.076 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:31.080+0000 7fedc4d5a180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T10:24:31.105 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:31.104+0000 7f385702e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-17T10:24:31.140 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.144+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c2c739f2-870b-4bc2-8861-6f788264da32", "id": 5} v 0) v1 -- 0x7f09e80b3070 con 0x7f09e80c3de0 2024-04-17T10:24:31.141 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:31.144+0000 7fedc4d5a180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T10:24:31.144 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09dea00640 1 -- 172.21.15.86:0/758429170 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c2c739f2-870b-4bc2-8861-6f788264da32", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f09d0003e40 con 0x7f09e80c3de0 2024-04-17T10:24:31.145 INFO:teuthology.orchestra.run.smithi086.stdout:5 2024-04-17T10:24:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 msgr2=0x7f09e814eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e814eb20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f09d00097f0 tx=0x7f09d001acb0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 shutdown_connections 2024-04-17T10:24:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09e815d3e0 0x7f09e814f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09e80c3de0 0x7f09e814eb20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 --2- 172.21.15.86:0/758429170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09e806ac90 0x7f09e80cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.146 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 >> 172.21.15.86:0/758429170 conn(0x7f09e80b22a0 msgr2=0x7f09e815bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 shutdown_connections 2024-04-17T10:24:31.147 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.148+0000 7f09ed600640 1 -- 172.21.15.86:0/758429170 wait complete. 2024-04-17T10:24:31.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 21849e19-c989-4e79-81fa-3a0c1b2d5656 6 2024-04-17T10:24:31.170 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:31.172+0000 7f385702e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-17T10:24:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94ffc00640 1 Processor -- start 2024-04-17T10:24:31.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94ffc00640 1 -- start start 2024-04-17T10:24:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94ffc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f815dcf0 0x7f94f80cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94ffc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94f80ce7a0 0x7f94f80b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.305 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94ffc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f80d0040 con 0x7f94f806cac0 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94ffc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f80cfde0 con 0x7f94f80ce7a0 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94ffc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80d0320 con 0x7f94f815dcf0 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fd200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f815dcf0 0x7f94f80cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fd200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f815dcf0 0x7f94f80cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57946/0 (socket says 172.21.15.86:57946) 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fd200640 1 -- 172.21.15.86:0/3561518353 learned_addr learned my addr 172.21.15.86:0/3561518353 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fdc00640 1 --2- 172.21.15.86:0/3561518353 >> v2:172.21.15.67:3300/0 conn(0x7f94f80ce7a0 0x7f94f80b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fd200640 1 -- 172.21.15.86:0/3561518353 >> v2:172.21.15.67:3300/0 conn(0x7f94f80ce7a0 msgr2=0x7f94f80b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fd200640 1 --2- 172.21.15.86:0/3561518353 >> v2:172.21.15.67:3300/0 conn(0x7f94f80ce7a0 0x7f94f80b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fd200640 1 -- 172.21.15.86:0/3561518353 >> v1:172.21.15.67:6789/0 conn(0x7f94f806cac0 legacy=0x7f94f80ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.308+0000 7f94fd200640 1 -- 172.21.15.86:0/3561518353 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f80d0600 con 0x7f94f815dcf0 2024-04-17T10:24:31.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94fd200640 1 --2- 172.21.15.86:0/3561518353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f815dcf0 0x7f94f80cf8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94e80098a0 tx=0x7f94e801c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1d99bbbc3d7810 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94f7e00640 1 -- 172.21.15.86:0/3561518353 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e80aa670 con 0x7f94f815dcf0 2024-04-17T10:24:31.308 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94f7e00640 1 -- 172.21.15.86:0/3561518353 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e80af070 con 0x7f94f815dcf0 2024-04-17T10:24:31.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94f7e00640 1 -- 172.21.15.86:0/3561518353 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e80aaa30 con 0x7f94f815dcf0 2024-04-17T10:24:31.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 -- 172.21.15.86:0/3561518353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f815dcf0 msgr2=0x7f94f80cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 --2- 172.21.15.86:0/3561518353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f815dcf0 0x7f94f80cf8a0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f94e80098a0 tx=0x7f94e801c510 comp rx=0 tx=0).stop 2024-04-17T10:24:31.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 -- 172.21.15.86:0/3561518353 shutdown_connections 2024-04-17T10:24:31.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 --2- 172.21.15.86:0/3561518353 >> v2:172.21.15.67:3300/0 conn(0x7f94f80ce7a0 0x7f94f80b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 --2- 172.21.15.86:0/3561518353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f815dcf0 0x7f94f80cf8a0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 -- 172.21.15.86:0/3561518353 >> 172.21.15.86:0/3561518353 conn(0x7f94f80b33a0 msgr2=0x7f94f80b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 -- 172.21.15.86:0/3561518353 shutdown_connections 2024-04-17T10:24:31.310 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.312+0000 7f94ffc00640 1 -- 172.21.15.86:0/3561518353 wait complete. 2024-04-17T10:24:31.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 Processor -- start 2024-04-17T10:24:31.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 -- start start 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f806cac0 0x7f94f8156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94f80ce7a0 0x7f94f815f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f815dcf0 0x7f94f815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80b2350 con 0x7f94f80ce7a0 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80b21d0 con 0x7f94f815dcf0 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80b24d0 con 0x7f94f806cac0 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f806cac0 0x7f94f8156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f806cac0 0x7f94f8156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57954/0 (socket says 172.21.15.86:57954) 2024-04-17T10:24:31.313 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 -- 172.21.15.86:0/2385593286 learned_addr learned my addr 172.21.15.86:0/2385593286 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fdc00640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f815dcf0 0x7f94f815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94efe00640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94f80ce7a0 0x7f94f815f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 -- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f815dcf0 msgr2=0x7f94f815bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f815dcf0 0x7f94f815bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 -- 172.21.15.86:0/2385593286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94f80ce7a0 msgr2=0x7f94f815f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94f80ce7a0 0x7f94f815f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 -- 172.21.15.86:0/2385593286 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f80c5b20 con 0x7f94f806cac0 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94efe00640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94f80ce7a0 0x7f94f815f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:31.314 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94fd200640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f806cac0 0x7f94f8156e40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94e80ac040 tx=0x7f94e80192b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94f5600640 1 -- 172.21.15.86:0/2385593286 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e80be610 con 0x7f94f806cac0 2024-04-17T10:24:31.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94f5600640 1 -- 172.21.15.86:0/2385593286 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e80be7b0 con 0x7f94f806cac0 2024-04-17T10:24:31.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94f5600640 1 -- 172.21.15.86:0/2385593286 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e80beba0 con 0x7f94f806cac0 2024-04-17T10:24:31.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f8157420 con 0x7f94f806cac0 2024-04-17T10:24:31.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f81579e0 con 0x7f94f806cac0 2024-04-17T10:24:31.315 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94f5600640 1 -- 172.21.15.86:0/2385593286 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f94e80bc020 con 0x7f94f806cac0 2024-04-17T10:24:31.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94f5600640 1 -- 172.21.15.86:0/2385593286 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f94e80c7a30 con 0x7f94f806cac0 2024-04-17T10:24:31.316 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.316+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f804ef30 con 0x7f94f806cac0 2024-04-17T10:24:31.317 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.320+0000 7f94f5600640 1 -- 172.21.15.86:0/2385593286 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f94e8016130 con 0x7f94f806cac0 2024-04-17T10:24:31.319 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:31.324+0000 7fedc4d5a180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T10:24:31.346 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:31.348+0000 7f385702e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-17T10:24:31.383 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.388+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "21849e19-c989-4e79-81fa-3a0c1b2d5656", "id": 6} v 0) v1 -- 0x7f94f806e5a0 con 0x7f94f806cac0 2024-04-17T10:24:31.387 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94f5600640 1 -- 172.21.15.86:0/2385593286 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "21849e19-c989-4e79-81fa-3a0c1b2d5656", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f94e80c7c80 con 0x7f94f806cac0 2024-04-17T10:24:31.387 INFO:teuthology.orchestra.run.smithi086.stdout:6 2024-04-17T10:24:31.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f806cac0 msgr2=0x7f94f8156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.388 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f806cac0 0x7f94f8156e40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f94e80ac040 tx=0x7f94e80192b0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 shutdown_connections 2024-04-17T10:24:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94f815dcf0 0x7f94f815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94f80ce7a0 0x7f94f815f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 --2- 172.21.15.86:0/2385593286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94f806cac0 0x7f94f8156e40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 >> 172.21.15.86:0/2385593286 conn(0x7f94f80b33a0 msgr2=0x7f94f80708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 shutdown_connections 2024-04-17T10:24:31.389 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.392+0000 7f94ffc00640 1 -- 172.21.15.86:0/2385593286 wait complete. 2024-04-17T10:24:31.392 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:31.396+0000 7fedc4d5a180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T10:24:31.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 743537a0-cacc-4f52-8d54-524ded1ed61e 7 2024-04-17T10:24:31.422 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:31.424+0000 7f385702e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-17T10:24:31.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969a00640 1 Processor -- start 2024-04-17T10:24:31.541 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969a00640 1 -- start start 2024-04-17T10:24:31.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa96415ce90 0x7fa9640cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.542 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa96409acd0 con 0x7fa96409a710 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9640cc360 con 0x7fa964099a50 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9640cc640 con 0x7fa96415ce90 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa963e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa96415ce90 0x7fa9640cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969000640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa964099a50 0x7fa96415d4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:52022/0 (socket says 172.21.15.86:52022) 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.544+0000 7fa969000640 1 -- 172.21.15.86:0/1222809237 learned_addr learned my addr 172.21.15.86:0/1222809237 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa963e00640 1 -- 172.21.15.86:0/1222809237 >> v1:172.21.15.86:6789/0 conn(0x7fa964099a50 legacy=0x7fa96415d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa963e00640 1 -- 172.21.15.86:0/1222809237 >> v1:172.21.15.67:6789/0 conn(0x7fa96409a710 legacy=0x7fa9640cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa963e00640 1 -- 172.21.15.86:0/1222809237 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9640cc920 con 0x7fa96415ce90 2024-04-17T10:24:31.545 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa963e00640 1 --2- 172.21.15.86:0/1222809237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa96415ce90 0x7fa9640cf200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa9540097f0 tx=0x7fa95401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4fea356ff337295 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa963400640 1 -- 172.21.15.86:0/1222809237 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540b2070 con 0x7fa96415ce90 2024-04-17T10:24:31.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa963400640 1 -- 172.21.15.86:0/1222809237 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9540a9070 con 0x7fa96415ce90 2024-04-17T10:24:31.546 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa963400640 1 -- 172.21.15.86:0/1222809237 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540ad880 con 0x7fa96415ce90 2024-04-17T10:24:31.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa969a00640 1 -- 172.21.15.86:0/1222809237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa96415ce90 msgr2=0x7fa9640cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.547 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa969a00640 1 --2- 172.21.15.86:0/1222809237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa96415ce90 0x7fa9640cf200 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa9540097f0 tx=0x7fa95401c450 comp rx=0 tx=0).stop 2024-04-17T10:24:31.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa969a00640 1 -- 172.21.15.86:0/1222809237 shutdown_connections 2024-04-17T10:24:31.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa969a00640 1 --2- 172.21.15.86:0/1222809237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa96415ce90 0x7fa9640cf200 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.548+0000 7fa969a00640 1 -- 172.21.15.86:0/1222809237 >> 172.21.15.86:0/1222809237 conn(0x7fa9640d1630 msgr2=0x7fa9640d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 -- 172.21.15.86:0/1222809237 shutdown_connections 2024-04-17T10:24:31.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 -- 172.21.15.86:0/1222809237 wait complete. 2024-04-17T10:24:31.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 Processor -- start 2024-04-17T10:24:31.548 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 -- start start 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa964099a50 0x7fa9640b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96409a710 0x7fa96415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa96415ce90 0x7fa964150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96409a270 con 0x7fa96415ce90 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96416a0a0 con 0x7fa96409a710 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9641f84e0 con 0x7fa964099a50 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa964099a50 0x7fa9640b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.551 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa95be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96409a710 0x7fa96415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa964099a50 0x7fa9640b42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.86:57970/0 (socket says 172.21.15.86:57970) 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa95be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96409a710 0x7fa96415bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.86:38096/0 (socket says 172.21.15.86:38096) 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 -- 172.21.15.86:0/2137255407 learned_addr learned my addr 172.21.15.86:0/2137255407 (peer_addr_for_me v2:172.21.15.86:0/0) 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa969000640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa96415ce90 0x7fa964150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 -- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96409a710 msgr2=0x7fa96415bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96409a710 0x7fa96415bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 -- 172.21.15.86:0/2137255407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa96415ce90 msgr2=0x7fa964150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa96415ce90 0x7fa964150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 -- 172.21.15.86:0/2137255407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9641f87c0 con 0x7fa964099a50 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa95be00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96409a710 0x7fa96415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa963e00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa964099a50 0x7fa9640b42b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa95401ab00 tx=0x7fa95401ac90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.552+0000 7fa960c00640 1 -- 172.21.15.86:0/2137255407 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540b2050 con 0x7fa964099a50 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa960c00640 1 -- 172.21.15.86:0/2137255407 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9540be070 con 0x7fa964099a50 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9641f8aa0 con 0x7fa964099a50 2024-04-17T10:24:31.552 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa960c00640 1 -- 172.21.15.86:0/2137255407 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9540b9690 con 0x7fa964099a50 2024-04-17T10:24:31.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9640b4950 con 0x7fa964099a50 2024-04-17T10:24:31.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa960c00640 1 -- 172.21.15.86:0/2137255407 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa9540ab070 con 0x7fa964099a50 2024-04-17T10:24:31.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa960c00640 1 -- 172.21.15.86:0/2137255407 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fa9540b7400 con 0x7fa964099a50 2024-04-17T10:24:31.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96404f780 con 0x7fa964099a50 2024-04-17T10:24:31.553 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.556+0000 7fa960c00640 1 -- 172.21.15.86:0/2137255407 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fa9540160d0 con 0x7fa964099a50 2024-04-17T10:24:31.624 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.628+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd new", "uuid": "743537a0-cacc-4f52-8d54-524ded1ed61e", "id": 7} v 0) v1 -- 0x7fa9640d2390 con 0x7fa964099a50 2024-04-17T10:24:31.628 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa960c00640 1 -- 172.21.15.86:0/2137255407 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "743537a0-cacc-4f52-8d54-524ded1ed61e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa9540059a0 con 0x7fa964099a50 2024-04-17T10:24:31.628 INFO:teuthology.orchestra.run.smithi086.stdout:7 2024-04-17T10:24:31.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa964099a50 msgr2=0x7fa9640b42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa964099a50 0x7fa9640b42b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa95401ab00 tx=0x7fa95401ac90 comp rx=0 tx=0).stop 2024-04-17T10:24:31.629 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 shutdown_connections 2024-04-17T10:24:31.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa96415ce90 0x7fa964150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa96409a710 0x7fa96415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 --2- 172.21.15.86:0/2137255407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa964099a50 0x7fa9640b42b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 >> 172.21.15.86:0/2137255407 conn(0x7fa9640d1630 msgr2=0x7fa9640c55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 shutdown_connections 2024-04-17T10:24:31.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-04-17T10:24:31.632+0000 7fa969a00640 1 -- 172.21.15.86:0/2137255407 wait complete. 2024-04-17T10:24:31.647 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-17T10:24:31.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-17T10:24:31.651 INFO:tasks.ceph.osd.0:Started 2024-04-17T10:24:31.651 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-17T10:24:31.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-17T10:24:31.654 INFO:tasks.ceph.osd.1:Started 2024-04-17T10:24:31.654 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-17T10:24:31.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-17T10:24:31.657 INFO:tasks.ceph.osd.2:Started 2024-04-17T10:24:31.657 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-17T10:24:31.658 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-17T10:24:31.660 INFO:tasks.ceph.osd.3:Started 2024-04-17T10:24:31.661 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-17T10:24:31.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-17T10:24:31.664 INFO:tasks.ceph.osd.4:Started 2024-04-17T10:24:31.664 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-17T10:24:31.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-17T10:24:31.667 INFO:tasks.ceph.osd.5:Started 2024-04-17T10:24:31.667 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-17T10:24:31.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-17T10:24:31.670 INFO:tasks.ceph.osd.6:Started 2024-04-17T10:24:31.670 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-17T10:24:31.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-17T10:24:31.675 INFO:tasks.ceph.osd.7:Started 2024-04-17T10:24:31.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:24:31.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd26000640 1 Processor -- start 2024-04-17T10:24:31.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd26000640 1 -- start start 2024-04-17T10:24:31.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd26000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdd200cf7f0 0x7fdd20161ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd26000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 0x7fdd200cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd26000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2006ff10 con 0x7fdd20162410 2024-04-17T10:24:31.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd26000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd200cfbd0 con 0x7fdd200cf7f0 2024-04-17T10:24:31.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd26000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd20161760 con 0x7fdd2006e5f0 2024-04-17T10:24:31.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd17e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdd20162410 0x7fdd200c55b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57454/0 (socket says 172.21.15.67:57454) 2024-04-17T10:24:31.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd17e00640 1 -- 172.21.15.67:0/2074262822 learned_addr learned my addr 172.21.15.67:0/2074262822 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:24:31.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1fe00640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 0x7fdd200cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285934460 0 0) 0x7fdd2006ff10 con 0x7fdd20162410 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1f400640 1 --2- 172.21.15.67:0/2074262822 >> v2:172.21.15.67:3300/0 conn(0x7fdd200cf7f0 0x7fdd20161ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd04003530 con 0x7fdd20162410 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129262715 0 0) 0x7fdd04003530 con 0x7fdd20162410 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 msgr2=0x7fdd200cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 0x7fdd200cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 >> v2:172.21.15.67:3300/0 conn(0x7fdd200cf7f0 msgr2=0x7fdd20161ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 --2- 172.21.15.67:0/2074262822 >> v2:172.21.15.67:3300/0 conn(0x7fdd200cf7f0 0x7fdd20161ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd200701b0 con 0x7fdd20162410 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fdd08001680 con 0x7fdd20162410 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.880+0000 7fdd1f400640 1 --2- 172.21.15.67:0/2074262822 >> v2:172.21.15.67:3300/0 conn(0x7fdd200cf7f0 0x7fdd20161ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 >> v1:172.21.15.67:6789/0 conn(0x7fdd20162410 legacy=0x7fdd200c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd040039c0 0x7fdd04003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 0x7fdd04004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd04004d50 0x7fdd040051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd20161760 con 0x7fdd040039c0 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0400ba90 con 0x7fdd04004d50 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0400bc90 con 0x7fdd2006e5f0 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1f400640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd040039c0 0x7fdd04003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd17e00640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 0x7fdd04004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1fe00640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd04004d50 0x7fdd040051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1f400640 1 -- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 msgr2=0x7fdd04004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1f400640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd2006e5f0 0x7fdd04004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1f400640 1 -- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd04004d50 msgr2=0x7fdd040051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1f400640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd04004d50 0x7fdd040051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1f400640 1 -- 172.21.15.67:0/2074262822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04003530 con 0x7fdd040039c0 2024-04-17T10:24:31.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1f400640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd040039c0 0x7fdd04003de0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdd100098d0 tx=0x7fdd1001c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e2b23ef2bf8774 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1000f040 con 0x7fdd040039c0 2024-04-17T10:24:31.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd100a9650 con 0x7fdd040039c0 2024-04-17T10:24:31.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.884+0000 7fdd1ea00640 1 -- 172.21.15.67:0/2074262822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd100a9a00 con 0x7fdd040039c0 2024-04-17T10:24:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- 172.21.15.67:0/2074262822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd040039c0 msgr2=0x7fdd04003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 --2- 172.21.15.67:0/2074262822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd040039c0 0x7fdd04003de0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdd100098d0 tx=0x7fdd1001c650 comp rx=0 tx=0).stop 2024-04-17T10:24:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1fe00640 1 -- 172.21.15.67:0/2074262822 reap_dead start 2024-04-17T10:24:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- 172.21.15.67:0/2074262822 shutdown_connections 2024-04-17T10:24:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- 172.21.15.67:0/2074262822 >> 172.21.15.67:0/2074262822 conn(0x7fdd200b2340 msgr2=0x7fdd200b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- 172.21.15.67:0/2074262822 shutdown_connections 2024-04-17T10:24:31.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- 172.21.15.67:0/2074262822 wait complete. 2024-04-17T10:24:31.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 Processor -- start 2024-04-17T10:24:31.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- start start 2024-04-17T10:24:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040039c0 0x7fdd201607e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd20152540 0x7fdd201f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd20148410 0x7fdd200ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200d0290 con 0x7fdd20148410 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd20161a40 con 0x7fdd040039c0 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd26000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200d0410 con 0x7fdd20152540 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040039c0 0x7fdd201607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd20148410 0x7fdd200ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040039c0 0x7fdd201607e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:48408/0 (socket says 172.21.15.67:48408) 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 -- 172.21.15.67:0/908983045 learned_addr learned my addr 172.21.15.67:0/908983045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd17e00640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd20152540 0x7fdd201f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:31.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 -- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd20152540 msgr2=0x7fdd201f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd20152540 0x7fdd201f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 -- 172.21.15.67:0/908983045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd20148410 msgr2=0x7fdd200ce2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd20148410 0x7fdd200ce2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1f400640 1 -- 172.21.15.67:0/908983045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd201f7f80 con 0x7fdd040039c0 2024-04-17T10:24:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.888+0000 7fdd1fe00640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd20148410 0x7fdd200ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:31.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd1f400640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040039c0 0x7fdd201607e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd1001c480 tx=0x7fdd04005840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd17400640 1 -- 172.21.15.67:0/908983045 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1000f000 con 0x7fdd040039c0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd17400640 1 -- 172.21.15.67:0/908983045 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd100a9450 con 0x7fdd040039c0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd17400640 1 -- 172.21.15.67:0/908983045 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd100b1820 con 0x7fdd040039c0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2015f740 con 0x7fdd040039c0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2015fd00 con 0x7fdd040039c0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd17400640 1 -- 172.21.15.67:0/908983045 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fdd100bc420 con 0x7fdd040039c0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd17400640 1 -- 172.21.15.67:0/908983045 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fdd100a9600 con 0x7fdd040039c0 2024-04-17T10:24:31.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.892+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2009e9a0 con 0x7fdd040039c0 2024-04-17T10:24:31.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.896+0000 7fdd17400640 1 -- 172.21.15.67:0/908983045 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fdd201f7f80 con 0x7fdd040039c0 2024-04-17T10:24:31.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd20074e30 con 0x7fdd040039c0 2024-04-17T10:24:31.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd17400640 1 -- 172.21.15.67:0/908983045 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fdd100ad020 con 0x7fdd040039c0 2024-04-17T10:24:31.980 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:24:31.980 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:24:31.631356+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040039c0 msgr2=0x7fdd201607e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040039c0 0x7fdd201607e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd1001c480 tx=0x7fdd04005840 comp rx=0 tx=0).stop 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 shutdown_connections 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdd20148410 0x7fdd200ce2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdd20152540 0x7fdd201f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 --2- 172.21.15.67:0/908983045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdd040039c0 0x7fdd201607e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 >> 172.21.15.67:0/908983045 conn(0x7fdd200b2340 msgr2=0x7fdd200cacf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 shutdown_connections 2024-04-17T10:24:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:31.980+0000 7fdd26000640 1 -- 172.21.15.67:0/908983045 wait complete. 2024-04-17T10:24:31.999 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-17T10:24:31.999 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-17T10:24:32.063 INFO:tasks.ceph.mgr.x.smithi086.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-17T10:24:32.063 INFO:tasks.ceph.mgr.x.smithi086.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-17T10:24:32.063 INFO:tasks.ceph.mgr.x.smithi086.stderr: from numpy import show_config as show_numpy_config 2024-04-17T10:24:32.102 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:32.104+0000 7fedc4d5a180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T10:24:32.150 INFO:tasks.ceph.mgr.y.smithi067.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-17T10:24:32.150 INFO:tasks.ceph.mgr.y.smithi067.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-17T10:24:32.150 INFO:tasks.ceph.mgr.y.smithi067.stderr: from numpy import show_config as show_numpy_config 2024-04-17T10:24:32.204 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:32.204+0000 7f385702e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-17T10:24:32.249 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:32.252+0000 7fedc4d5a180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T10:24:32.283 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T10:24:32.284+0000 7f209483b7c0 -1 Falling back to public interface 2024-04-17T10:24:32.314 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T10:24:32.316+0000 7f2702c547c0 -1 Falling back to public interface 2024-04-17T10:24:32.315 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T10:24:32.316+0000 7f7e5b0267c0 -1 Falling back to public interface 2024-04-17T10:24:32.344 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-17T10:24:32.348+0000 7f4c6c9207c0 -1 Falling back to public interface 2024-04-17T10:24:32.350 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T10:24:32.352+0000 7f5b91a657c0 -1 Falling back to public interface 2024-04-17T10:24:32.366 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:32.368+0000 7f385702e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-17T10:24:32.368 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-04-17T10:24:32.372+0000 7f29c159b7c0 -1 Falling back to public interface 2024-04-17T10:24:32.376 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-17T10:24:32.380+0000 7fd8ad76e7c0 -1 Falling back to public interface 2024-04-17T10:24:32.392 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-04-17T10:24:32.396+0000 7ff2a70a27c0 -1 Falling back to public interface 2024-04-17T10:24:32.430 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:32.432+0000 7fedc4d5a180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T10:24:32.502 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:32.504+0000 7fedc4d5a180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T10:24:32.552 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:32.552+0000 7f385702e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-17T10:24:32.629 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:32.628+0000 7f385702e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-17T10:24:32.649 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:32.652+0000 7fedc4d5a180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T10:24:32.786 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:32.788+0000 7f385702e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-17T10:24:32.805 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T10:24:32.936 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.936+0000 7f3d5ca00640 1 Processor -- start 2024-04-17T10:24:32.937 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.936+0000 7f3d5ca00640 1 -- start start 2024-04-17T10:24:32.938 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d5ca00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d5815ecb0 0x7f3d580712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:32.938 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d5ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d580cdfc0 0x7f3d58071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:32.938 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d5ca00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5815f940 con 0x7f3d5815f3f0 2024-04-17T10:24:32.938 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d5ca00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5815fc20 con 0x7f3d5815ecb0 2024-04-17T10:24:32.938 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d5ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5815ff00 con 0x7f3d580cdfc0 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d55e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d5815ecb0 0x7f3d580712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d56800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d580cdfc0 0x7f3d58071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d55e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d5815ecb0 0x7f3d580712e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:48420/0 (socket says 172.21.15.67:48420) 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d55e00640 1 -- 172.21.15.67:0/1142887182 learned_addr learned my addr 172.21.15.67:0/1142887182 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d55e00640 1 -- 172.21.15.67:0/1142887182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d580cdfc0 msgr2=0x7f3d58071820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1142887182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d580cdfc0 0x7f3d58071820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d55e00640 1 -- 172.21.15.67:0/1142887182 >> v1:172.21.15.86:6789/0 conn(0x7f3d5815f3f0 legacy=0x7f3d580cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:32.939 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.940+0000 7f3d55e00640 1 -- 172.21.15.67:0/1142887182 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580cd390 con 0x7f3d5815ecb0 2024-04-17T10:24:32.942 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1142887182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815ecb0 0x7f3d580712e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009870 tx=0x7f3d4801ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4654e50d45c12b7f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:32.942 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d55400640 1 -- 172.21.15.67:0/1142887182 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480b5070 con 0x7f3d5815ecb0 2024-04-17T10:24:32.942 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d55400640 1 -- 172.21.15.67:0/1142887182 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d480a8a50 con 0x7f3d5815ecb0 2024-04-17T10:24:32.942 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d55400640 1 -- 172.21.15.67:0/1142887182 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480a8e20 con 0x7f3d5815ecb0 2024-04-17T10:24:32.943 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1142887182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815ecb0 msgr2=0x7f3d580712e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:32.943 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 --2- 172.21.15.67:0/1142887182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815ecb0 0x7f3d580712e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3d48009870 tx=0x7f3d4801ed50 comp rx=0 tx=0).stop 2024-04-17T10:24:32.943 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1142887182 shutdown_connections 2024-04-17T10:24:32.943 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 --2- 172.21.15.67:0/1142887182 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d580cdfc0 0x7f3d58071820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:32.943 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 --2- 172.21.15.67:0/1142887182 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815ecb0 0x7f3d580712e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:32.943 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1142887182 >> 172.21.15.67:0/1142887182 conn(0x7f3d58067200 msgr2=0x7f3d5806ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:32.943 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1142887182 shutdown_connections 2024-04-17T10:24:32.944 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1142887182 wait complete. 2024-04-17T10:24:32.944 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 Processor -- start 2024-04-17T10:24:32.944 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- start start 2024-04-17T10:24:32.947 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d580cdfc0 0x7f3d5815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:32.947 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5815ecb0 0x7f3d5813c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:32.947 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815f3f0 0x7f3d58158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:32.947 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d58144620 con 0x7f3d580cdfc0 2024-04-17T10:24:32.947 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5815f090 con 0x7f3d5815f3f0 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.944+0000 7f3d5ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d581447a0 con 0x7f3d5815ecb0 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d580cdfc0 0x7f3d5815a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d580cdfc0 0x7f3d5815a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36370/0 (socket says 172.21.15.67:36370) 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 -- 172.21.15.67:0/1834061954 learned_addr learned my addr 172.21.15.67:0/1834061954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d4fe00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5815ecb0 0x7f3d5813c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d56800640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815f3f0 0x7f3d58158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 -- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5815ecb0 msgr2=0x7f3d5813c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5815ecb0 0x7f3d5813c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 -- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815f3f0 msgr2=0x7f3d58158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815f3f0 0x7f3d58158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 -- 172.21.15.67:0/1834061954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5815b760 con 0x7f3d580cdfc0 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d4fe00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5815ecb0 0x7f3d5813c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d55e00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d580cdfc0 0x7f3d5815a8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d480b3040 tx=0x7f3d4801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:32.948 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d4e000640 1 -- 172.21.15.67:0/1834061954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480b5070 con 0x7f3d580cdfc0 2024-04-17T10:24:32.949 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d4e000640 1 -- 172.21.15.67:0/1834061954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d480b0de0 con 0x7f3d580cdfc0 2024-04-17T10:24:32.949 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d4e000640 1 -- 172.21.15.67:0/1834061954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d480bb670 con 0x7f3d580cdfc0 2024-04-17T10:24:32.949 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5815aec0 con 0x7f3d580cdfc0 2024-04-17T10:24:32.949 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5815b480 con 0x7f3d580cdfc0 2024-04-17T10:24:32.949 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d4e000640 1 -- 172.21.15.67:0/1834061954 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3d480c2060 con 0x7f3d580cdfc0 2024-04-17T10:24:32.949 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d4e000640 1 -- 172.21.15.67:0/1834061954 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f3d480c4400 con 0x7f3d580cdfc0 2024-04-17T10:24:32.949 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.948+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d58074e20 con 0x7f3d580cdfc0 2024-04-17T10:24:32.950 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:32.952+0000 7f3d4e000640 1 -- 172.21.15.67:0/1834061954 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f3d480dbdd0 con 0x7f3d580cdfc0 2024-04-17T10:24:33.009 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.008+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d580aee10 con 0x7f3d580cdfc0 2024-04-17T10:24:33.010 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d4e000640 1 -- 172.21.15.67:0/1834061954 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f3d480ca070 con 0x7f3d580cdfc0 2024-04-17T10:24:33.010 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-17T10:24:33.010 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:24:31.631356+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:24:33.011 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d580cdfc0 msgr2=0x7f3d5815a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:33.011 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d580cdfc0 0x7f3d5815a8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d480b3040 tx=0x7f3d4801baf0 comp rx=0 tx=0).stop 2024-04-17T10:24:33.012 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 shutdown_connections 2024-04-17T10:24:33.012 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5815f3f0 0x7f3d58158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:33.012 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5815ecb0 0x7f3d5813c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:33.012 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 --2- 172.21.15.67:0/1834061954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d580cdfc0 0x7f3d5815a8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:33.012 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 >> 172.21.15.67:0/1834061954 conn(0x7f3d58067200 msgr2=0x7f3d580cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:33.012 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 shutdown_connections 2024-04-17T10:24:33.012 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:33.012+0000 7f3d5ca00640 1 -- 172.21.15.67:0/1834061954 wait complete. 2024-04-17T10:24:33.027 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-17T10:24:33.338 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:33.340+0000 7fedc4d5a180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T10:24:33.425 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:33.428+0000 7fedc4d5a180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T10:24:33.509 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:33.508+0000 7f385702e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-17T10:24:33.600 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:33.600+0000 7f385702e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-17T10:24:33.625 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:33.628+0000 7fedc4d5a180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T10:24:33.709 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:33.712+0000 7fedc4d5a180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T10:24:33.812 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:33.812+0000 7f385702e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-17T10:24:33.902 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:33.904+0000 7f385702e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-17T10:24:34.035 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:34.040+0000 7fedc4d5a180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T10:24:34.106 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:34.108+0000 7fedc4d5a180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T10:24:34.255 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:34.256+0000 7f385702e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-17T10:24:34.330 INFO:tasks.ceph.mgr.y.smithi067.stderr:2024-04-17T10:24:34.332+0000 7f385702e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-17T10:24:35.127 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:35.132+0000 7fed6ae00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-17T10:24:35.128 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-04-17T10:24:35.132+0000 7fed6ae00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-17T10:24:36.236 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T10:24:36.236+0000 7f209483b7c0 -1 osd.0 0 log_to_monitors true 2024-04-17T10:24:36.300 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-17T10:24:36.304+0000 7fd8ad76e7c0 -1 osd.5 0 log_to_monitors true 2024-04-17T10:24:36.321 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-04-17T10:24:36.324+0000 7ff2a70a27c0 -1 osd.6 0 log_to_monitors true 2024-04-17T10:24:36.351 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T10:24:36.352+0000 7f7e5b0267c0 -1 osd.2 0 log_to_monitors true 2024-04-17T10:24:36.369 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T10:24:36.368+0000 7f2702c547c0 -1 osd.1 0 log_to_monitors true 2024-04-17T10:24:36.400 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T10:24:36.400+0000 7f5b91a657c0 -1 osd.3 0 log_to_monitors true 2024-04-17T10:24:36.604 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-04-17T10:24:36.608+0000 7f29c159b7c0 -1 osd.7 0 log_to_monitors true 2024-04-17T10:24:36.615 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-17T10:24:36.616+0000 7f4c6c9207c0 -1 osd.4 0 log_to_monitors true 2024-04-17T10:24:39.835 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T10:24:40.045 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa13e600640 1 Processor -- start 2024-04-17T10:24:40.045 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa13e600640 1 -- start start 2024-04-17T10:24:40.045 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa13e600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa13806c270 0x7fa1380c4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:40.045 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa13e600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa138166340 0x7fa13816be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:40.045 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa13e600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa138162690 con 0x7fa1380c4cd0 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa13e600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa138166980 con 0x7fa13806c270 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa13e600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1380c5130 con 0x7fa138166340 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa136a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa1380c4cd0 0x7fa138165c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57460/0 (socket says 172.21.15.67:57460) 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa136a00640 1 -- 172.21.15.67:0/3723177858 learned_addr learned my addr 172.21.15.67:0/3723177858 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa137e00640 1 --2- 172.21.15.67:0/3723177858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa138166340 0x7fa13816be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.044+0000 7fa137400640 1 --2- 172.21.15.67:0/3723177858 >> v2:172.21.15.86:3300/0 conn(0x7fa13806c270 0x7fa1380c4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.048+0000 7fa136000640 1 -- 172.21.15.67:0/3723177858 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015948400 0 0) 0x7fa138162690 con 0x7fa1380c4cd0 2024-04-17T10:24:40.046 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.048+0000 7fa136000640 1 -- 172.21.15.67:0/3723177858 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11c003530 con 0x7fa1380c4cd0 2024-04-17T10:24:40.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.048+0000 7fa137e00640 1 -- 172.21.15.67:0/3723177858 >> v2:172.21.15.86:3300/0 conn(0x7fa13806c270 msgr2=0x7fa1380c4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:40.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.048+0000 7fa137e00640 1 --2- 172.21.15.67:0/3723177858 >> v2:172.21.15.86:3300/0 conn(0x7fa13806c270 0x7fa1380c4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.048+0000 7fa137e00640 1 -- 172.21.15.67:0/3723177858 >> v1:172.21.15.67:6789/0 conn(0x7fa1380c4cd0 legacy=0x7fa138165c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:40.049 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.048+0000 7fa137e00640 1 -- 172.21.15.67:0/3723177858 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa138162970 con 0x7fa138166340 2024-04-17T10:24:40.050 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa137e00640 1 --2- 172.21.15.67:0/3723177858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa138166340 0x7fa13816be00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa130009d90 tx=0x7fa13001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe89fd7430ae3c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:40.051 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa136000640 1 -- 172.21.15.67:0/3723177858 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1300b2070 con 0x7fa138166340 2024-04-17T10:24:40.051 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa136000640 1 -- 172.21.15.67:0/3723177858 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1300a5740 con 0x7fa138166340 2024-04-17T10:24:40.051 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa136000640 1 -- 172.21.15.67:0/3723177858 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1300b2070 con 0x7fa138166340 2024-04-17T10:24:40.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 -- 172.21.15.67:0/3723177858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa138166340 msgr2=0x7fa13816be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:40.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 --2- 172.21.15.67:0/3723177858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa138166340 0x7fa13816be00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa130009d90 tx=0x7fa13001c990 comp rx=0 tx=0).stop 2024-04-17T10:24:40.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 -- 172.21.15.67:0/3723177858 shutdown_connections 2024-04-17T10:24:40.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 --2- 172.21.15.67:0/3723177858 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa138166340 0x7fa13816be00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 --2- 172.21.15.67:0/3723177858 >> v2:172.21.15.86:3300/0 conn(0x7fa13806c270 0x7fa1380c4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.052 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 -- 172.21.15.67:0/3723177858 >> 172.21.15.67:0/3723177858 conn(0x7fa1380b32b0 msgr2=0x7fa1380b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:40.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 -- 172.21.15.67:0/3723177858 shutdown_connections 2024-04-17T10:24:40.053 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.052+0000 7fa13e600640 1 -- 172.21.15.67:0/3723177858 wait complete. 2024-04-17T10:24:40.054 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 Processor -- start 2024-04-17T10:24:40.055 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 -- start start 2024-04-17T10:24:40.056 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa13806c270 0x7fa1380b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:40.056 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380c4cd0 0x7fa13814ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:40.056 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa138166340 0x7fa13814efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:40.057 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13816ccd0 con 0x7fa138166340 2024-04-17T10:24:40.057 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13816cb50 con 0x7fa13806c270 2024-04-17T10:24:40.057 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa13e600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13816ce50 con 0x7fa1380c4cd0 2024-04-17T10:24:40.057 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa136a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380c4cd0 0x7fa13814ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:40.057 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa136a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380c4cd0 0x7fa13814ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:44110/0 (socket says 172.21.15.67:44110) 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.056+0000 7fa136a00640 1 -- 172.21.15.67:0/1967610004 learned_addr learned my addr 172.21.15.67:0/1967610004 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa137400640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa13806c270 0x7fa1380b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa137e00640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa138166340 0x7fa13814efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa136a00640 1 -- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa13806c270 msgr2=0x7fa1380b26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa136a00640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa13806c270 0x7fa1380b26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa136a00640 1 -- 172.21.15.67:0/1967610004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa138166340 msgr2=0x7fa13814efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa136a00640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa138166340 0x7fa13814efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.058 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa136a00640 1 -- 172.21.15.67:0/1967610004 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa138163e30 con 0x7fa1380c4cd0 2024-04-17T10:24:40.059 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa136a00640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380c4cd0 0x7fa13814ea80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c450 tx=0x7fa12400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:40.059 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa12f400640 1 -- 172.21.15.67:0/1967610004 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124017070 con 0x7fa1380c4cd0 2024-04-17T10:24:40.059 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa12f400640 1 -- 172.21.15.67:0/1967610004 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa124005db0 con 0x7fa1380c4cd0 2024-04-17T10:24:40.060 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa12f400640 1 -- 172.21.15.67:0/1967610004 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa124007290 con 0x7fa1380c4cd0 2024-04-17T10:24:40.060 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa13814f630 con 0x7fa1380c4cd0 2024-04-17T10:24:40.060 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1380c79d0 con 0x7fa1380c4cd0 2024-04-17T10:24:40.060 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.060+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa138074e20 con 0x7fa1380c4cd0 2024-04-17T10:24:40.063 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.064+0000 7fa12f400640 1 -- 172.21.15.67:0/1967610004 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 4) v1 ==== 86671+0+0 (secure 0 0 0) 0x7fa12401a070 con 0x7fa1380c4cd0 2024-04-17T10:24:40.064 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.064+0000 7fa12f400640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa118066f00 0x7fa1180693c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:40.064 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.064+0000 7fa12f400640 1 -- 172.21.15.67:0/1967610004 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fa124083410 con 0x7fa1380c4cd0 2024-04-17T10:24:40.066 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.068+0000 7fa137400640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa118066f00 0x7fa1180693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:40.067 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.068+0000 7fa137400640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa118066f00 0x7fa1180693c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa128009b60 tx=0x7fa128009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:40.076 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.076+0000 7fa12f400640 1 -- 172.21.15.67:0/1967610004 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa124054c30 con 0x7fa1380c4cd0 2024-04-17T10:24:40.422 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.424+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1381640a0 con 0x7fa1380c4cd0 2024-04-17T10:24:40.425 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.424+0000 7fa12f400640 1 -- 172.21.15.67:0/1967610004 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7fa124058cd0 con 0x7fa1380c4cd0 2024-04-17T10:24:40.425 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-17T10:24:40.425 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":12,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:24:38.179381+0000","last_up_change":"0.000000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:24:40.428 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.428+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa118066f00 msgr2=0x7fa1180693c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:40.429 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.428+0000 7fa13e600640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa118066f00 0x7fa1180693c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa128009b60 tx=0x7fa128009190 comp rx=0 tx=0).stop 2024-04-17T10:24:40.429 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.428+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380c4cd0 msgr2=0x7fa13814ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:40.429 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.428+0000 7fa13e600640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380c4cd0 0x7fa13814ea80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa12400c450 tx=0x7fa12400c910 comp rx=0 tx=0).stop 2024-04-17T10:24:40.431 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 shutdown_connections 2024-04-17T10:24:40.431 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa118066f00 0x7fa1180693c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.431 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa138166340 0x7fa13814efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.431 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1380c4cd0 0x7fa13814ea80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.431 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 --2- 172.21.15.67:0/1967610004 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa13806c270 0x7fa1380b26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:40.431 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 >> 172.21.15.67:0/1967610004 conn(0x7fa1380b32b0 msgr2=0x7fa1381645c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:40.433 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 shutdown_connections 2024-04-17T10:24:40.433 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:40.432+0000 7fa13e600640 1 -- 172.21.15.67:0/1967610004 wait complete. 2024-04-17T10:24:40.450 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T10:24:40.452+0000 7f5b8ca00640 -1 osd.3 0 waiting for initial osdmap 2024-04-17T10:24:40.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-17T10:24:40.497 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T10:24:40.500+0000 7f26fdc00640 -1 osd.1 0 waiting for initial osdmap 2024-04-17T10:24:40.502 INFO:tasks.ceph.osd.3.smithi067.stderr:2024-04-17T10:24:40.504+0000 7f5b82a00640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:40.522 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T10:24:40.524+0000 7f7e55e00640 -1 osd.2 0 waiting for initial osdmap 2024-04-17T10:24:40.533 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-04-17T10:24:40.535+0000 7ff2a2a00640 -1 osd.6 0 waiting for initial osdmap 2024-04-17T10:24:40.546 INFO:tasks.ceph.osd.1.smithi067.stderr:2024-04-17T10:24:40.548+0000 7f26f3c00640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:40.553 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-17T10:24:40.555+0000 7fd8a8600640 -1 osd.5 0 waiting for initial osdmap 2024-04-17T10:24:40.556 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-04-17T10:24:40.559+0000 7f29bce00640 -1 osd.7 0 waiting for initial osdmap 2024-04-17T10:24:40.569 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-04-17T10:24:40.571+0000 7ff298000640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:40.571 INFO:tasks.ceph.osd.2.smithi067.stderr:2024-04-17T10:24:40.572+0000 7f7e4be00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:40.585 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-04-17T10:24:40.587+0000 7fd89e600640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:40.588 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-17T10:24:40.591+0000 7f4c68200640 -1 osd.4 0 waiting for initial osdmap 2024-04-17T10:24:40.596 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-04-17T10:24:40.599+0000 7f29b2400640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:40.628 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-04-17T10:24:40.631+0000 7f4c5d800640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:40.644 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T10:24:40.644+0000 7f208f600640 -1 osd.0 0 waiting for initial osdmap 2024-04-17T10:24:40.686 INFO:tasks.ceph.osd.0.smithi067.stderr:2024-04-17T10:24:40.688+0000 7f2085600640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-17T10:24:47.269 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-17T10:24:47.411 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c8a000640 1 Processor -- start 2024-04-17T10:24:47.411 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c8a000640 1 -- start start 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c8a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c840719f0 0x7f0c840cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415f3f0 0x7f0c840cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c8a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840ce330 con 0x7f0c8415ecb0 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c8a000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8415f940 con 0x7f0c840719f0 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c8a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8415fc20 con 0x7f0c8415f3f0 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c7be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c840719f0 0x7f0c840cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c83400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0c8415ecb0 0x7f0c840712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39574/0 (socket says 172.21.15.67:39574) 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c83400640 1 -- 172.21.15.67:0/1301617141 learned_addr learned my addr 172.21.15.67:0/1301617141 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c83e00640 1 --2- 172.21.15.67:0/1301617141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415f3f0 0x7f0c840cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:47.413 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c7be00640 1 -- 172.21.15.67:0/1301617141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415f3f0 msgr2=0x7f0c840cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:47.414 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c7be00640 1 --2- 172.21.15.67:0/1301617141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415f3f0 0x7f0c840cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.414 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c7be00640 1 -- 172.21.15.67:0/1301617141 >> v1:172.21.15.67:6789/0 conn(0x7f0c8415ecb0 legacy=0x7f0c840712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:47.414 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c7be00640 1 -- 172.21.15.67:0/1301617141 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8415ff00 con 0x7f0c840719f0 2024-04-17T10:24:47.414 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.412+0000 7f0c82a00640 1 -- 172.21.15.67:0/1301617141 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299518327 0 0) 0x7f0c74001120 con 0x7f0c8415ecb0 2024-04-17T10:24:47.416 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c7be00640 1 --2- 172.21.15.67:0/1301617141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c840cd8b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0097c0 tx=0x7f0c6c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0b36fe38643a9d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:47.416 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c82a00640 1 -- 172.21.15.67:0/1301617141 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0b5070 con 0x7f0c840719f0 2024-04-17T10:24:47.416 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c82a00640 1 -- 172.21.15.67:0/1301617141 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c0a8990 con 0x7f0c840719f0 2024-04-17T10:24:47.416 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c82a00640 1 -- 172.21.15.67:0/1301617141 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c0a8d60 con 0x7f0c840719f0 2024-04-17T10:24:47.417 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 -- 172.21.15.67:0/1301617141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 msgr2=0x7f0c840cd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:47.417 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 --2- 172.21.15.67:0/1301617141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c840cd8b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0097c0 tx=0x7f0c6c01ec90 comp rx=0 tx=0).stop 2024-04-17T10:24:47.417 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 -- 172.21.15.67:0/1301617141 shutdown_connections 2024-04-17T10:24:47.417 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 --2- 172.21.15.67:0/1301617141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415f3f0 0x7f0c840cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.417 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 --2- 172.21.15.67:0/1301617141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c840cd8b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.417 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 -- 172.21.15.67:0/1301617141 >> 172.21.15.67:0/1301617141 conn(0x7f0c84067200 msgr2=0x7f0c8406ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:47.417 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 -- 172.21.15.67:0/1301617141 shutdown_connections 2024-04-17T10:24:47.418 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.416+0000 7f0c8a000640 1 -- 172.21.15.67:0/1301617141 wait complete. 2024-04-17T10:24:47.418 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 Processor -- start 2024-04-17T10:24:47.418 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 -- start start 2024-04-17T10:24:47.419 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c8413c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415ecb0 0x7f0c84156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c8415f3f0 0x7f0c84158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8415f090 con 0x7f0c840719f0 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c841601e0 con 0x7f0c8415f3f0 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84144620 con 0x7f0c8415ecb0 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c8413c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c8413c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51060/0 (socket says 172.21.15.67:51060) 2024-04-17T10:24:47.420 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 -- 172.21.15.67:0/831908075 learned_addr learned my addr 172.21.15.67:0/831908075 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:47.421 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 -- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415ecb0 msgr2=0x7f0c84156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:47.421 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415ecb0 0x7f0c84156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.421 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 -- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c8415f3f0 msgr2=0x7f0c84158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:47.421 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c8415f3f0 0x7f0c84158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.421 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 -- 172.21.15.67:0/831908075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8413d7d0 con 0x7f0c840719f0 2024-04-17T10:24:47.421 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c83400640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c8413c8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c7b0 tx=0x7f0c7400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:47.421 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c7b400640 1 -- 172.21.15.67:0/831908075 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74018070 con 0x7f0c840719f0 2024-04-17T10:24:47.422 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c7b400640 1 -- 172.21.15.67:0/831908075 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c740072f0 con 0x7f0c840719f0 2024-04-17T10:24:47.422 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8413cf30 con 0x7f0c840719f0 2024-04-17T10:24:47.422 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c7b400640 1 -- 172.21.15.67:0/831908075 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c740047f0 con 0x7f0c840719f0 2024-04-17T10:24:47.422 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.420+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8413d4f0 con 0x7f0c840719f0 2024-04-17T10:24:47.422 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.424+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8406bbd0 con 0x7f0c840719f0 2024-04-17T10:24:47.423 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.424+0000 7f0c7b400640 1 -- 172.21.15.67:0/831908075 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0c7401b070 con 0x7f0c840719f0 2024-04-17T10:24:47.423 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.424+0000 7f0c7b400640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c60066f60 0x7f0c60069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.423 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.424+0000 7f0c7b400640 1 -- 172.21.15.67:0/831908075 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0c74084100 con 0x7f0c840719f0 2024-04-17T10:24:47.424 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.424+0000 7f0c7be00640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c60066f60 0x7f0c60069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:47.424 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.424+0000 7f0c7be00640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c60066f60 0x7f0c60069420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0b2000 tx=0x7f0c6c004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:47.428 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.428+0000 7f0c7b400640 1 -- 172.21.15.67:0/831908075 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0c840ce330 con 0x7f0c840719f0 2024-04-17T10:24:47.602 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c84074e20 con 0x7f0c840719f0 2024-04-17T10:24:47.603 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c7b400640 1 -- 172.21.15.67:0/831908075 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12836 (secure 0 0 0) 0x7f0c74055450 con 0x7f0c840719f0 2024-04-17T10:24:47.603 INFO:teuthology.misc.health.smithi067.stdout: 2024-04-17T10:24:47.603 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:24:44.530466+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:24:47.605 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c60066f60 msgr2=0x7f0c60069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:47.605 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c60066f60 0x7f0c60069420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c0b2000 tx=0x7f0c6c004740 comp rx=0 tx=0).stop 2024-04-17T10:24:47.605 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 msgr2=0x7f0c8413c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:47.605 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c8413c8c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c7b0 tx=0x7f0c7400cc70 comp rx=0 tx=0).stop 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 shutdown_connections 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c60066f60 0x7f0c60069420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c8415f3f0 0x7f0c84158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c8415ecb0 0x7f0c84156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 --2- 172.21.15.67:0/831908075 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c840719f0 0x7f0c8413c8c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.604+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 >> 172.21.15.67:0/831908075 conn(0x7f0c84067200 msgr2=0x7f0c8406ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.608+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 shutdown_connections 2024-04-17T10:24:47.606 INFO:teuthology.misc.health.smithi067.stderr:2024-04-17T10:24:47.608+0000 7f0c8a000640 1 -- 172.21.15.67:0/831908075 wait complete. 2024-04-17T10:24:47.623 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-17T10:24:47.623 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-17T10:24:47.623 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:24:47.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:24:47.626 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:24:47.626 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:24:47.626 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:24:47.629 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:24:47.629 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:24:47.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:24:47.633 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:24:47.633 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:24:47.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:24:47.636 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:24:47.637 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-17T10:24:47.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:24:47.738 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:24:47.739 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:24:47.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2469400640 1 Processor -- start 2024-04-17T10:24:47.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2469400640 1 -- start start 2024-04-17T10:24:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2469400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2464145040 0x7f246406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2469400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f246406ef10 0x7f2464071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2469400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246406e2f0 con 0x7f2464071850 2024-04-17T10:24:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2469400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24640a86c0 con 0x7f2464145040 2024-04-17T10:24:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2469400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246415bf10 con 0x7f246406ef10 2024-04-17T10:24:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2462800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2464145040 0x7f246406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2462800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2464145040 0x7f246406e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51062/0 (socket says 172.21.15.67:51062) 2024-04-17T10:24:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2462800640 1 -- 172.21.15.67:0/1074750393 learned_addr learned my addr 172.21.15.67:0/1074750393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2459e00640 1 --2- 172.21.15.67:0/1074750393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f246406ef10 0x7f2464071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:47.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2462800640 1 -- 172.21.15.67:0/1074750393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f246406ef10 msgr2=0x7f2464071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2462800640 1 --2- 172.21.15.67:0/1074750393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f246406ef10 0x7f2464071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2462800640 1 -- 172.21.15.67:0/1074750393 >> v1:172.21.15.86:6789/0 conn(0x7f2464071850 legacy=0x7f246415f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:47.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.780+0000 7f2462800640 1 -- 172.21.15.67:0/1074750393 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246406d6e0 con 0x7f2464145040 2024-04-17T10:24:47.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:47.784+0000 7f2462800640 1 --2- 172.21.15.67:0/1074750393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464145040 0x7f246406e9d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2454009e50 tx=0x7f245401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346a13d9eed30414 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2461e00640 1 -- 172.21.15.67:0/1074750393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24540b4070 con 0x7f2464145040 2024-04-17T10:24:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2461e00640 1 -- 172.21.15.67:0/1074750393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24540af400 con 0x7f2464145040 2024-04-17T10:24:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2461e00640 1 -- 172.21.15.67:0/1074750393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24540af7d0 con 0x7f2464145040 2024-04-17T10:24:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 -- 172.21.15.67:0/1074750393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464145040 msgr2=0x7f246406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 --2- 172.21.15.67:0/1074750393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464145040 0x7f246406e9d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2454009e50 tx=0x7f245401f210 comp rx=0 tx=0).stop 2024-04-17T10:24:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 -- 172.21.15.67:0/1074750393 shutdown_connections 2024-04-17T10:24:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 --2- 172.21.15.67:0/1074750393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f246406ef10 0x7f2464071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 --2- 172.21.15.67:0/1074750393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464145040 0x7f246406e9d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 -- 172.21.15.67:0/1074750393 >> 172.21.15.67:0/1074750393 conn(0x7f246406ac90 msgr2=0x7f246406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 -- 172.21.15.67:0/1074750393 shutdown_connections 2024-04-17T10:24:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.480+0000 7f2469400640 1 -- 172.21.15.67:0/1074750393 wait complete. 2024-04-17T10:24:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 Processor -- start 2024-04-17T10:24:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 -- start start 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f246406ef10 0x7f2464138f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464071850 0x7f246413d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2464145040 0x7f246413ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246406dca0 con 0x7f2464071850 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2464071ca0 con 0x7f246406ef10 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2469400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246406df80 con 0x7f2464145040 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464071850 0x7f246413d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464071850 0x7f246413d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51074/0 (socket says 172.21.15.67:51074) 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 -- 172.21.15.67:0/2128903706 learned_addr learned my addr 172.21.15.67:0/2128903706 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2462800640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f246406ef10 0x7f2464138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2463200640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2464145040 0x7f246413ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 -- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2464145040 msgr2=0x7f246413ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2464145040 0x7f246413ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 -- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f246406ef10 msgr2=0x7f2464138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f246406ef10 0x7f2464138f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2459e00640 1 -- 172.21.15.67:0/2128903706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24641394a0 con 0x7f2464071850 2024-04-17T10:24:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2462800640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f246406ef10 0x7f2464138f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.484+0000 7f2463200640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2464145040 0x7f246413ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:24:48.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.488+0000 7f2459e00640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464071850 0x7f246413d990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f244c00d5f0 tx=0x7f244c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.488+0000 7f245b400640 1 -- 172.21.15.67:0/2128903706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c0125f0 con 0x7f2464071850 2024-04-17T10:24:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.488+0000 7f245b400640 1 -- 172.21.15.67:0/2128903706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244c019070 con 0x7f2464071850 2024-04-17T10:24:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.488+0000 7f245b400640 1 -- 172.21.15.67:0/2128903706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c0129b0 con 0x7f2464071850 2024-04-17T10:24:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.488+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246413be60 con 0x7f2464071850 2024-04-17T10:24:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.488+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246413eaf0 con 0x7f2464071850 2024-04-17T10:24:48.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.488+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2464072d50 con 0x7f2464071850 2024-04-17T10:24:48.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.492+0000 7f245b400640 1 -- 172.21.15.67:0/2128903706 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f244c012b50 con 0x7f2464071850 2024-04-17T10:24:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.492+0000 7f245b400640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2440066fb0 0x7f2440069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.492+0000 7f245b400640 1 -- 172.21.15.67:0/2128903706 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f244c085070 con 0x7f2464071850 2024-04-17T10:24:48.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.492+0000 7f2462800640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2440066fb0 0x7f2440069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.496+0000 7f2462800640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2440066fb0 0x7f2440069470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24540b1000 tx=0x7f24540aa000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:48.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.496+0000 7f245b400640 1 -- 172.21.15.67:0/2128903706 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f244c0563b0 con 0x7f2464071850 2024-04-17T10:24:48.551 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:24:48.553 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:24:48.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2464074e30 con 0x7f2464071850 2024-04-17T10:24:48.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f245b400640 1 -- 172.21.15.67:0/2128903706 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12836 (secure 0 0 0) 0x7f244c055cb0 con 0x7f2464071850 2024-04-17T10:24:48.674 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:24:48.675 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:24:44.530466+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:24:48.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2440066fb0 msgr2=0x7f2440069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2440066fb0 0x7f2440069470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f24540b1000 tx=0x7f24540aa000 comp rx=0 tx=0).stop 2024-04-17T10:24:48.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464071850 msgr2=0x7f246413d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464071850 0x7f246413d990 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f244c00d5f0 tx=0x7f244c00dab0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 shutdown_connections 2024-04-17T10:24:48.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2440066fb0 0x7f2440069470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2464145040 0x7f246413ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2464071850 0x7f246413d990 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 --2- 172.21.15.67:0/2128903706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f246406ef10 0x7f2464138f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 >> 172.21.15.67:0/2128903706 conn(0x7f246406ac90 msgr2=0x7f246415f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:48.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 shutdown_connections 2024-04-17T10:24:48.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.676+0000 7f2469400640 1 -- 172.21.15.67:0/2128903706 wait complete. 2024-04-17T10:24:48.695 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:24:48.695 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:24:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed22400640 1 Processor -- start 2024-04-17T10:24:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed22400640 1 -- start start 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed22400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c162160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed22400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed1c15f200 0x7fed1c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed22400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1c15f680 con 0x7fed1c0717c0 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed22400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1c06d6a0 con 0x7fed1c15f200 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed22400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c06d8d0 con 0x7fed1c0713e0 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed1c15f200 0x7fed1c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed1c15f200 0x7fed1c06d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51096/0 (socket says 172.21.15.67:51096) 2024-04-17T10:24:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1aa00640 1 -- 172.21.15.67:0/3262461345 learned_addr learned my addr 172.21.15.67:0/3262461345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1b400640 1 --2- 172.21.15.67:0/3262461345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1aa00640 1 -- 172.21.15.67:0/3262461345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 msgr2=0x7fed1c162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1aa00640 1 --2- 172.21.15.67:0/3262461345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c162160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1aa00640 1 -- 172.21.15.67:0/3262461345 >> v1:172.21.15.86:6789/0 conn(0x7fed1c0717c0 legacy=0x7fed1c0ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.840+0000 7fed1aa00640 1 -- 172.21.15.67:0/3262461345 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c06dbb0 con 0x7fed1c15f200 2024-04-17T10:24:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed1aa00640 1 --2- 172.21.15.67:0/3262461345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c06d160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed08009870 tx=0x7fed0801c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a12364d7b0e095b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed1a000640 1 -- 172.21.15.67:0/3262461345 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080a95f0 con 0x7fed1c15f200 2024-04-17T10:24:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed1a000640 1 -- 172.21.15.67:0/3262461345 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed080a9790 con 0x7fed1c15f200 2024-04-17T10:24:48.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed1a000640 1 -- 172.21.15.67:0/3262461345 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080a9b60 con 0x7fed1c15f200 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 -- 172.21.15.67:0/3262461345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 msgr2=0x7fed1c06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 --2- 172.21.15.67:0/3262461345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c06d160 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fed08009870 tx=0x7fed0801c530 comp rx=0 tx=0).stop 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 -- 172.21.15.67:0/3262461345 shutdown_connections 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 --2- 172.21.15.67:0/3262461345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c06d160 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 --2- 172.21.15.67:0/3262461345 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c162160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 -- 172.21.15.67:0/3262461345 >> 172.21.15.67:0/3262461345 conn(0x7fed1c0b1740 msgr2=0x7fed1c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 -- 172.21.15.67:0/3262461345 shutdown_connections 2024-04-17T10:24:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 -- 172.21.15.67:0/3262461345 wait complete. 2024-04-17T10:24:48.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.844+0000 7fed22400640 1 Processor -- start 2024-04-17T10:24:48.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 -- start start 2024-04-17T10:24:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c0cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1c0717c0 0x7fed1c13d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c0ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c071d50 con 0x7fed1c15f200 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c071bd0 con 0x7fed1c0717c0 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1c147640 con 0x7fed1c0713e0 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c0ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c0ce1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51108/0 (socket says 172.21.15.67:51108) 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 -- 172.21.15.67:0/2036096580 learned_addr learned my addr 172.21.15.67:0/2036096580 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1b400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c0cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1aa00640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1c0717c0 0x7fed1c13d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 -- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 msgr2=0x7fed1c0cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c0cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 -- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1c0717c0 msgr2=0x7fed1c13d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1c0717c0 0x7fed1c13d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 -- 172.21.15.67:0/2036096580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c13f480 con 0x7fed1c15f200 2024-04-17T10:24:48.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed1be00640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c0ce1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed1400c730 tx=0x7fed1400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:48.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed13400640 1 -- 172.21.15.67:0/2036096580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14003b80 con 0x7fed1c15f200 2024-04-17T10:24:48.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed13400640 1 -- 172.21.15.67:0/2036096580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed14003d20 con 0x7fed1c15f200 2024-04-17T10:24:48.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed13400640 1 -- 172.21.15.67:0/2036096580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1400aa30 con 0x7fed1c15f200 2024-04-17T10:24:48.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c15db30 con 0x7fed1c15f200 2024-04-17T10:24:48.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1c13f720 con 0x7fed1c15f200 2024-04-17T10:24:48.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.848+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1c04ef30 con 0x7fed1c15f200 2024-04-17T10:24:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.852+0000 7fed13400640 1 -- 172.21.15.67:0/2036096580 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fed1400abd0 con 0x7fed1c15f200 2024-04-17T10:24:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.852+0000 7fed13400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecfc066f60 0x7fecfc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.852+0000 7fed13400640 1 -- 172.21.15.67:0/2036096580 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fed14084aa0 con 0x7fed1c15f200 2024-04-17T10:24:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.852+0000 7fed1b400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecfc066f60 0x7fecfc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.852+0000 7fed1b400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecfc066f60 0x7fecfc069420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0079c0 tx=0x7fed0c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:48.856+0000 7fed13400640 1 -- 172.21.15.67:0/2036096580 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fed1c15f680 con 0x7fed1c15f200 2024-04-17T10:24:49.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.020+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed1c0b00b0 con 0x7fed1c15f200 2024-04-17T10:24:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.020+0000 7fed13400640 1 -- 172.21.15.67:0/2036096580 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed14055710 con 0x7fed1c15f200 2024-04-17T10:24:49.021 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecfc066f60 msgr2=0x7fecfc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecfc066f60 0x7fecfc069420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0079c0 tx=0x7fed0c008040 comp rx=0 tx=0).stop 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 msgr2=0x7fed1c0ce1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c0ce1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed1400c730 tx=0x7fed1400cbf0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 shutdown_connections 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecfc066f60 0x7fecfc069420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1c15f200 0x7fed1c0ce1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1c0717c0 0x7fed1c13d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 --2- 172.21.15.67:0/2036096580 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed1c0713e0 0x7fed1c0cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 >> 172.21.15.67:0/2036096580 conn(0x7fed1c0b1740 msgr2=0x7fed1c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:49.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 shutdown_connections 2024-04-17T10:24:49.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.024+0000 7fed22400640 1 -- 172.21.15.67:0/2036096580 wait complete. 2024-04-17T10:24:49.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:24:49.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.180+0000 7fcf90400640 1 Processor -- start 2024-04-17T10:24:49.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.180+0000 7fcf90400640 1 -- start start 2024-04-17T10:24:49.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf90400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf880ce1b0 0x7fcf880ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:49.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf90400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815fbd0 0x7fcf880c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:49.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf90400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf880cead0 con 0x7fcf8815cc00 2024-04-17T10:24:49.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf90400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8815be60 con 0x7fcf880ce1b0 2024-04-17T10:24:49.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf90400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88160230 con 0x7fcf8815fbd0 2024-04-17T10:24:49.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8da00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf880ce1b0 0x7fcf880ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:49.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8e400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815fbd0 0x7fcf880c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:49.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8da00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf880ce1b0 0x7fcf880ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:56556/0 (socket says 172.21.15.67:56556) 2024-04-17T10:24:49.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8da00640 1 -- 172.21.15.67:0/3941950878 learned_addr learned my addr 172.21.15.67:0/3941950878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:49.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8d000640 1 --1- 172.21.15.67:0/3941950878 >> v1:172.21.15.86:6789/0 conn(0x7fcf8815cc00 0x7fcf8815f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:40152/0 (socket says 172.21.15.67:40152) 2024-04-17T10:24:49.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8da00640 1 -- 172.21.15.67:0/3941950878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815fbd0 msgr2=0x7fcf880c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:49.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8da00640 1 --2- 172.21.15.67:0/3941950878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815fbd0 0x7fcf880c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8da00640 1 -- 172.21.15.67:0/3941950878 >> v1:172.21.15.86:6789/0 conn(0x7fcf8815cc00 legacy=0x7fcf8815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.184+0000 7fcf8da00640 1 -- 172.21.15.67:0/3941950878 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8815c120 con 0x7fcf880ce1b0 2024-04-17T10:24:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf8da00640 1 --2- 172.21.15.67:0/3941950878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf880ce1b0 0x7fcf880ce590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009870 tx=0x7fcf7c01ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9b6ba0f100a309c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf87e00640 1 -- 172.21.15.67:0/3941950878 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0b5070 con 0x7fcf880ce1b0 2024-04-17T10:24:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf87e00640 1 -- 172.21.15.67:0/3941950878 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c0a8a50 con 0x7fcf880ce1b0 2024-04-17T10:24:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf87e00640 1 -- 172.21.15.67:0/3941950878 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0a8e20 con 0x7fcf880ce1b0 2024-04-17T10:24:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 -- 172.21.15.67:0/3941950878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf880ce1b0 msgr2=0x7fcf880ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 --2- 172.21.15.67:0/3941950878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf880ce1b0 0x7fcf880ce590 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c009870 tx=0x7fcf7c01ed50 comp rx=0 tx=0).stop 2024-04-17T10:24:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 -- 172.21.15.67:0/3941950878 shutdown_connections 2024-04-17T10:24:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 --2- 172.21.15.67:0/3941950878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815fbd0 0x7fcf880c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 --2- 172.21.15.67:0/3941950878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf880ce1b0 0x7fcf880ce590 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 -- 172.21.15.67:0/3941950878 >> 172.21.15.67:0/3941950878 conn(0x7fcf880b33f0 msgr2=0x7fcf880b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:49.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 -- 172.21.15.67:0/3941950878 shutdown_connections 2024-04-17T10:24:49.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 -- 172.21.15.67:0/3941950878 wait complete. 2024-04-17T10:24:49.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 Processor -- start 2024-04-17T10:24:49.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.188+0000 7fcf90400640 1 -- start start 2024-04-17T10:24:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf880ce1b0 0x7fcf880c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815cc00 0x7fcf880ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8815fbd0 0x7fcf880ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88159730 con 0x7fcf880ce1b0 2024-04-17T10:24:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8815c6e0 con 0x7fcf8815fbd0 2024-04-17T10:24:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881598b0 con 0x7fcf8815cc00 2024-04-17T10:24:49.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf880ce1b0 0x7fcf880c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf880ce1b0 0x7fcf880c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51110/0 (socket says 172.21.15.67:51110) 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 -- 172.21.15.67:0/1963024017 learned_addr learned my addr 172.21.15.67:0/1963024017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8e400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8815fbd0 0x7fcf880ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 -- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815cc00 msgr2=0x7fcf880ca850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8d000640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815cc00 0x7fcf880ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815cc00 0x7fcf880ca850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 -- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8815fbd0 msgr2=0x7fcf880ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8815fbd0 0x7fcf880ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 -- 172.21.15.67:0/1963024017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880cb3f0 con 0x7fcf880ce1b0 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8d000640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815cc00 0x7fcf880ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8e400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8815fbd0 0x7fcf880ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf8da00640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf880ce1b0 0x7fcf880c99c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c01d400 tx=0x7fcf7c01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:49.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf85600640 1 -- 172.21.15.67:0/1963024017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0b5070 con 0x7fcf880ce1b0 2024-04-17T10:24:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf85600640 1 -- 172.21.15.67:0/1963024017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c0b0de0 con 0x7fcf880ce1b0 2024-04-17T10:24:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf85600640 1 -- 172.21.15.67:0/1963024017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0bb670 con 0x7fcf880ce1b0 2024-04-17T10:24:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf880cb110 con 0x7fcf880ce1b0 2024-04-17T10:24:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf880c91c0 con 0x7fcf880ce1b0 2024-04-17T10:24:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf88070ab0 con 0x7fcf880ce1b0 2024-04-17T10:24:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.192+0000 7fcf85600640 1 -- 172.21.15.67:0/1963024017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcf7c0c2060 con 0x7fcf880ce1b0 2024-04-17T10:24:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.196+0000 7fcf85600640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf68066f60 0x7fcf68069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.196+0000 7fcf85600640 1 -- 172.21.15.67:0/1963024017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcf7c122220 con 0x7fcf880ce1b0 2024-04-17T10:24:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.196+0000 7fcf8d000640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf68066f60 0x7fcf68069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:49.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.196+0000 7fcf8d000640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf68066f60 0x7fcf68069420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf74007fd0 tx=0x7fcf74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.200+0000 7fcf85600640 1 -- 172.21.15.67:0/1963024017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcf7c0188b0 con 0x7fcf880ce1b0 2024-04-17T10:24:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf88074e30 con 0x7fcf880ce1b0 2024-04-17T10:24:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf85600640 1 -- 172.21.15.67:0/1963024017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7fcf7c120e20 con 0x7fcf880ce1b0 2024-04-17T10:24:49.394 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:24:49.394 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-04-17T10:24:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf68066f60 msgr2=0x7fcf68069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf68066f60 0x7fcf68069420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf74007fd0 tx=0x7fcf74009040 comp rx=0 tx=0).stop 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf880ce1b0 msgr2=0x7fcf880c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf880ce1b0 0x7fcf880c99c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c01d400 tx=0x7fcf7c01baf0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 shutdown_connections 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf68066f60 0x7fcf68069420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8815fbd0 0x7fcf880ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8815cc00 0x7fcf880ca850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 --2- 172.21.15.67:0/1963024017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf880ce1b0 0x7fcf880c99c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:49.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 >> 172.21.15.67:0/1963024017 conn(0x7fcf880b33f0 msgr2=0x7fcf880b06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:49.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 shutdown_connections 2024-04-17T10:24:49.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:49.396+0000 7fcf90400640 1 -- 172.21.15.67:0/1963024017 wait complete. 2024-04-17T10:24:51.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:24:51.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.568+0000 7fabe3600640 1 Processor -- start 2024-04-17T10:24:51.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.568+0000 7fabe3600640 1 -- start start 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabe3600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabdc071310 0x7fabdc071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabe3600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc0cc300 0x7fabdc0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabe3600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabdc06e2f0 con 0x7fabdc145040 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabe3600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabdc0a86c0 con 0x7fabdc071310 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabe3600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc071cb0 con 0x7fabdc0cc300 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabdc071310 0x7fabdc071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabdc071310 0x7fabdc071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50116/0 (socket says 172.21.15.67:50116) 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 -- 172.21.15.67:0/3851581663 learned_addr learned my addr 172.21.15.67:0/3851581663 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabe1600640 1 --2- 172.21.15.67:0/3851581663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc0cc300 0x7fabdc0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 -- 172.21.15.67:0/3851581663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc0cc300 msgr2=0x7fabdc0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:51.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 --2- 172.21.15.67:0/3851581663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc0cc300 0x7fabdc0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 -- 172.21.15.67:0/3851581663 >> v1:172.21.15.86:6789/0 conn(0x7fabdc145040 legacy=0x7fabdc06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 -- 172.21.15.67:0/3851581663 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc06d6e0 con 0x7fabdc071310 2024-04-17T10:24:51.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabd3e00640 1 --2- 172.21.15.67:0/3851581663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc071310 0x7fabdc071770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009870 tx=0x7fabc401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f633fcedc841de server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:51.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabdbe00640 1 -- 172.21.15.67:0/3851581663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc40b5070 con 0x7fabdc071310 2024-04-17T10:24:51.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabdbe00640 1 -- 172.21.15.67:0/3851581663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc40a8a50 con 0x7fabdc071310 2024-04-17T10:24:51.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.572+0000 7fabdbe00640 1 -- 172.21.15.67:0/3851581663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc40a8e20 con 0x7fabdc071310 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- 172.21.15.67:0/3851581663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc071310 msgr2=0x7fabdc071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 --2- 172.21.15.67:0/3851581663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc071310 0x7fabdc071770 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009870 tx=0x7fabc401ed50 comp rx=0 tx=0).stop 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- 172.21.15.67:0/3851581663 shutdown_connections 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 --2- 172.21.15.67:0/3851581663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc0cc300 0x7fabdc0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 --2- 172.21.15.67:0/3851581663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc071310 0x7fabdc071770 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- 172.21.15.67:0/3851581663 >> 172.21.15.67:0/3851581663 conn(0x7fabdc06ac90 msgr2=0x7fabdc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- 172.21.15.67:0/3851581663 shutdown_connections 2024-04-17T10:24:51.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- 172.21.15.67:0/3851581663 wait complete. 2024-04-17T10:24:51.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 Processor -- start 2024-04-17T10:24:51.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- start start 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc071310 0x7fabdc136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabdc0cc300 0x7fabdc13c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc145040 0x7fabdc14e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc06df80 con 0x7fabdc145040 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc06dca0 con 0x7fabdc0cc300 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe3600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc06d370 con 0x7fabdc071310 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc145040 0x7fabdc14e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc145040 0x7fabdc14e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50132/0 (socket says 172.21.15.67:50132) 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 -- 172.21.15.67:0/305714992 learned_addr learned my addr 172.21.15.67:0/305714992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabd3e00640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabdc0cc300 0x7fabdc13c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe0c00640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc071310 0x7fabdc136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 -- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc071310 msgr2=0x7fabdc136000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc071310 0x7fabdc136000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 -- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabdc0cc300 msgr2=0x7fabdc13c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabdc0cc300 0x7fabdc13c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe1600640 1 -- 172.21.15.67:0/305714992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc136eb0 con 0x7fabdc145040 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.576+0000 7fabe0c00640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc071310 0x7fabdc136000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabe1600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc145040 0x7fabdc14e5b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fabd400efc0 tx=0x7fabd400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabd9600640 1 -- 172.21.15.67:0/305714992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd4018020 con 0x7fabdc145040 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabd9600640 1 -- 172.21.15.67:0/305714992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd4014480 con 0x7fabdc145040 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabd9600640 1 -- 172.21.15.67:0/305714992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd4014870 con 0x7fabdc145040 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc137190 con 0x7fabdc145040 2024-04-17T10:24:51.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc137750 con 0x7fabdc145040 2024-04-17T10:24:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc072d50 con 0x7fabdc145040 2024-04-17T10:24:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabd9600640 1 -- 172.21.15.67:0/305714992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fabd401b020 con 0x7fabdc145040 2024-04-17T10:24:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabd9600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb8066f60 0x7fabb8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabd9600640 1 -- 172.21.15.67:0/305714992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fabd4085b20 con 0x7fabdc145040 2024-04-17T10:24:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.580+0000 7fabe0c00640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb8066f60 0x7fabb8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.584+0000 7fabe0c00640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb8066f60 0x7fabb8069420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fabcc001670 tx=0x7fabcc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:51.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.584+0000 7fabd9600640 1 -- 172.21.15.67:0/305714992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fabd4022030 con 0x7fabdc145040 2024-04-17T10:24:51.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.780+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabdc074e30 con 0x7fabdc145040 2024-04-17T10:24:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.780+0000 7fabd9600640 1 -- 172.21.15.67:0/305714992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7fabd4010040 con 0x7fabdc145040 2024-04-17T10:24:51.780 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:24:51.780 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-04-17T10:24:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-04-17T10:24:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb8066f60 msgr2=0x7fabb8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb8066f60 0x7fabb8069420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fabcc001670 tx=0x7fabcc009040 comp rx=0 tx=0).stop 2024-04-17T10:24:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc145040 msgr2=0x7fabdc14e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc145040 0x7fabdc14e5b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fabd400efc0 tx=0x7fabd400c490 comp rx=0 tx=0).stop 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 shutdown_connections 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb8066f60 0x7fabb8069420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabdc145040 0x7fabdc14e5b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabdc0cc300 0x7fabdc13c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 --2- 172.21.15.67:0/305714992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabdc071310 0x7fabdc136000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 >> 172.21.15.67:0/305714992 conn(0x7fabdc06ac90 msgr2=0x7fabdc15e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 shutdown_connections 2024-04-17T10:24:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.784+0000 7fabe3600640 1 -- 172.21.15.67:0/305714992 wait complete. 2024-04-17T10:24:51.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:24:51.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.940+0000 7fba42200640 1 Processor -- start 2024-04-17T10:24:51.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.940+0000 7fba42200640 1 -- start start 2024-04-17T10:24:51.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba42200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c145040 0x7fba3c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba42200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba3c06ef10 0x7fba3c071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba42200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c06e2f0 con 0x7fba3c071850 2024-04-17T10:24:51.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba42200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3c0a86c0 con 0x7fba3c06ef10 2024-04-17T10:24:51.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba42200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15bf10 con 0x7fba3c145040 2024-04-17T10:24:51.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fba3c071850 0x7fba3c15f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43158/0 (socket says 172.21.15.67:43158) 2024-04-17T10:24:51.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3be00640 1 -- 172.21.15.67:0/310481205 learned_addr learned my addr 172.21.15.67:0/310481205 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:24:51.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba32a00640 1 --2- 172.21.15.67:0/310481205 >> v2:172.21.15.86:3300/0 conn(0x7fba3c06ef10 0x7fba3c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3b400640 1 --2- 172.21.15.67:0/310481205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c145040 0x7fba3c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3aa00640 1 -- 172.21.15.67:0/310481205 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728093943 0 0) 0x7fba3c06e2f0 con 0x7fba3c071850 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3aa00640 1 -- 172.21.15.67:0/310481205 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba20003530 con 0x7fba3c071850 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3aa00640 1 -- 172.21.15.67:0/310481205 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3828951800 0 0) 0x7fba20003530 con 0x7fba3c071850 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3b400640 1 -- 172.21.15.67:0/310481205 >> v2:172.21.15.86:3300/0 conn(0x7fba3c06ef10 msgr2=0x7fba3c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3b400640 1 --2- 172.21.15.67:0/310481205 >> v2:172.21.15.86:3300/0 conn(0x7fba3c06ef10 0x7fba3c071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3b400640 1 -- 172.21.15.67:0/310481205 >> v1:172.21.15.67:6789/0 conn(0x7fba3c071850 legacy=0x7fba3c15f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba3b400640 1 -- 172.21.15.67:0/310481205 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c06d6e0 con 0x7fba3c145040 2024-04-17T10:24:51.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.944+0000 7fba32a00640 1 --2- 172.21.15.67:0/310481205 >> v2:172.21.15.86:3300/0 conn(0x7fba3c06ef10 0x7fba3c071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:24:51.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba3b400640 1 --2- 172.21.15.67:0/310481205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c145040 0x7fba3c06e9d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0097c0 tx=0x7fba2c01ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba1d1f06b34e591f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba3aa00640 1 -- 172.21.15.67:0/310481205 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0b5070 con 0x7fba3c145040 2024-04-17T10:24:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba3aa00640 1 -- 172.21.15.67:0/310481205 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c0a85f0 con 0x7fba3c145040 2024-04-17T10:24:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba3aa00640 1 -- 172.21.15.67:0/310481205 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34001a00 con 0x7fba3c145040 2024-04-17T10:24:51.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 -- 172.21.15.67:0/310481205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c145040 msgr2=0x7fba3c06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 --2- 172.21.15.67:0/310481205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c145040 0x7fba3c06e9d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0097c0 tx=0x7fba2c01ebb0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 -- 172.21.15.67:0/310481205 shutdown_connections 2024-04-17T10:24:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 --2- 172.21.15.67:0/310481205 >> v2:172.21.15.86:3300/0 conn(0x7fba3c06ef10 0x7fba3c071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 --2- 172.21.15.67:0/310481205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c145040 0x7fba3c06e9d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 -- 172.21.15.67:0/310481205 >> 172.21.15.67:0/310481205 conn(0x7fba3c06ac90 msgr2=0x7fba3c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 -- 172.21.15.67:0/310481205 shutdown_connections 2024-04-17T10:24:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.948+0000 7fba42200640 1 -- 172.21.15.67:0/310481205 wait complete. 2024-04-17T10:24:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 Processor -- start 2024-04-17T10:24:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 -- start start 2024-04-17T10:24:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ef10 0x7fba3c13a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c071850 0x7fba3c15aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c145040 0x7fba3c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c06dca0 con 0x7fba3c06ef10 2024-04-17T10:24:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c071ca0 con 0x7fba3c145040 2024-04-17T10:24:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c06df80 con 0x7fba3c071850 2024-04-17T10:24:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ef10 0x7fba3c13a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ef10 0x7fba3c13a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50146/0 (socket says 172.21.15.67:50146) 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 -- 172.21.15.67:0/2398004743 learned_addr learned my addr 172.21.15.67:0/2398004743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba32a00640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c071850 0x7fba3c15aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3be00640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c145040 0x7fba3c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 -- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c071850 msgr2=0x7fba3c15aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c071850 0x7fba3c15aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 -- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c145040 msgr2=0x7fba3c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c145040 0x7fba3c15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 -- 172.21.15.67:0/2398004743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c138b90 con 0x7fba3c06ef10 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba3b400640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ef10 0x7fba3c13a390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fba2c000c10 tx=0x7fba2c01b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba33e00640 1 -- 172.21.15.67:0/2398004743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0bd020 con 0x7fba3c06ef10 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba33e00640 1 -- 172.21.15.67:0/2398004743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c0b0640 con 0x7fba3c06ef10 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba33e00640 1 -- 172.21.15.67:0/2398004743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0b5040 con 0x7fba3c06ef10 2024-04-17T10:24:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c137f10 con 0x7fba3c06ef10 2024-04-17T10:24:51.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c1384d0 con 0x7fba3c06ef10 2024-04-17T10:24:51.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.952+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c072d50 con 0x7fba3c06ef10 2024-04-17T10:24:51.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.956+0000 7fba33e00640 1 -- 172.21.15.67:0/2398004743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fba2c0c0020 con 0x7fba3c06ef10 2024-04-17T10:24:51.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.956+0000 7fba33e00640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f60 0x7fba18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:51.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.956+0000 7fba33e00640 1 -- 172.21.15.67:0/2398004743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba2c121d80 con 0x7fba3c06ef10 2024-04-17T10:24:51.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.956+0000 7fba32a00640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f60 0x7fba18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:51.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.956+0000 7fba32a00640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f60 0x7fba18069420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fba24009580 tx=0x7fba24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:51.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:51.960+0000 7fba33e00640 1 -- 172.21.15.67:0/2398004743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fba2c0c7030 con 0x7fba3c06ef10 2024-04-17T10:24:52.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.124+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba3c074e30 con 0x7fba3c06ef10 2024-04-17T10:24:52.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.124+0000 7fba33e00640 1 -- 172.21.15.67:0/2398004743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12836 (secure 0 0 0) 0x7fba2c018710 con 0x7fba3c06ef10 2024-04-17T10:24:52.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:24:52.125 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":16,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:24:44.530466+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f60 msgr2=0x7fba18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f60 0x7fba18069420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fba24009580 tx=0x7fba24008040 comp rx=0 tx=0).stop 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ef10 msgr2=0x7fba3c13a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ef10 0x7fba3c13a390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fba2c000c10 tx=0x7fba2c01b950 comp rx=0 tx=0).stop 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 shutdown_connections 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f60 0x7fba18069420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c145040 0x7fba3c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c071850 0x7fba3c15aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 --2- 172.21.15.67:0/2398004743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ef10 0x7fba3c13a390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 >> 172.21.15.67:0/2398004743 conn(0x7fba3c06ac90 msgr2=0x7fba3c06b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 shutdown_connections 2024-04-17T10:24:52.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.128+0000 7fba42200640 1 -- 172.21.15.67:0/2398004743 wait complete. 2024-04-17T10:24:52.146 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:24:52.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:24:52.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:24:52.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:24:52.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e8400640 1 Processor -- start 2024-04-17T10:24:52.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e8400640 1 -- start start 2024-04-17T10:24:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e8400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35e00ce3b0 0x7f35e006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e8400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00b1680 con 0x7f35e00b0cd0 2024-04-17T10:24:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e8400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00b1960 con 0x7f35e006b1d0 2024-04-17T10:24:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e8400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e00b1430 con 0x7f35e00ce3b0 2024-04-17T10:24:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35e00ce3b0 0x7f35e006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35e00ce3b0 0x7f35e006ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50156/0 (socket says 172.21.15.67:50156) 2024-04-17T10:24:52.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e5000640 1 -- 172.21.15.67:0/1343965166 learned_addr learned my addr 172.21.15.67:0/1343965166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e5000640 1 -- 172.21.15.67:0/1343965166 >> v1:172.21.15.86:6789/0 conn(0x7f35e006b1d0 legacy=0x7f35e0161cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e5000640 1 -- 172.21.15.67:0/1343965166 >> v1:172.21.15.67:6789/0 conn(0x7f35e00b0cd0 legacy=0x7f35e00cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.296+0000 7f35e5000640 1 -- 172.21.15.67:0/1343965166 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00b1c40 con 0x7f35e00ce3b0 2024-04-17T10:24:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e5000640 1 --2- 172.21.15.67:0/1343965166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e006ac90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7df09bc107429237 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35dfe00640 1 -- 172.21.15.67:0/1343965166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc0b1070 con 0x7f35e00ce3b0 2024-04-17T10:24:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35dfe00640 1 -- 172.21.15.67:0/1343965166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc0a8050 con 0x7f35e00ce3b0 2024-04-17T10:24:52.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35dfe00640 1 -- 172.21.15.67:0/1343965166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc0ac860 con 0x7f35e00ce3b0 2024-04-17T10:24:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 -- 172.21.15.67:0/1343965166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 msgr2=0x7f35e006ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 --2- 172.21.15.67:0/1343965166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e006ac90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009870 tx=0x7f35cc01c4d0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 -- 172.21.15.67:0/1343965166 shutdown_connections 2024-04-17T10:24:52.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 --2- 172.21.15.67:0/1343965166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e006ac90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 -- 172.21.15.67:0/1343965166 >> 172.21.15.67:0/1343965166 conn(0x7f35e00b22a0 msgr2=0x7f35e00b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:52.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 -- 172.21.15.67:0/1343965166 shutdown_connections 2024-04-17T10:24:52.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 -- 172.21.15.67:0/1343965166 wait complete. 2024-04-17T10:24:52.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 Processor -- start 2024-04-17T10:24:52.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.300+0000 7f35e8400640 1 -- start start 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35e006b1d0 0x7f35e0157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35e00b0cd0 0x7f35e0153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e0159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e016c5b0 con 0x7f35e00ce3b0 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e006b5e0 con 0x7f35e00b0cd0 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e016c730 con 0x7f35e006b1d0 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e0159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e0159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50164/0 (socket says 172.21.15.67:50164) 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 -- 172.21.15.67:0/3786272816 learned_addr learned my addr 172.21.15.67:0/3786272816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e5000640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35e00b0cd0 0x7f35e0153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e5a00640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35e006b1d0 0x7f35e0157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 -- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35e006b1d0 msgr2=0x7f35e0157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35e006b1d0 0x7f35e0157680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 -- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35e00b0cd0 msgr2=0x7f35e0153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35e00b0cd0 0x7f35e0153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 -- 172.21.15.67:0/3786272816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00ad720 con 0x7f35e00ce3b0 2024-04-17T10:24:52.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e6400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e0159360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35d800c770 tx=0x7f35d800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35dd600640 1 -- 172.21.15.67:0/3786272816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8003b80 con 0x7f35e00ce3b0 2024-04-17T10:24:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35dd600640 1 -- 172.21.15.67:0/3786272816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d8003d20 con 0x7f35e00ce3b0 2024-04-17T10:24:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35dd600640 1 -- 172.21.15.67:0/3786272816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d800aa90 con 0x7f35e00ce3b0 2024-04-17T10:24:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e0153d60 con 0x7f35e00ce3b0 2024-04-17T10:24:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e00adca0 con 0x7f35e00ce3b0 2024-04-17T10:24:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.304+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e004ef30 con 0x7f35e00ce3b0 2024-04-17T10:24:52.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.308+0000 7f35dd600640 1 -- 172.21.15.67:0/3786272816 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f35d800ac30 con 0x7f35e00ce3b0 2024-04-17T10:24:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.308+0000 7f35dd600640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35c0066f60 0x7f35c0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.308+0000 7f35dd600640 1 -- 172.21.15.67:0/3786272816 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f35d8084b30 con 0x7f35e00ce3b0 2024-04-17T10:24:52.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.308+0000 7f35e5a00640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35c0066f60 0x7f35c0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.308+0000 7f35e5a00640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35c0066f60 0x7f35c0069420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35e00cd6e0 tx=0x7f35d4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:52.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.312+0000 7f35dd600640 1 -- 172.21.15.67:0/3786272816 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f35d80557e0 con 0x7f35e00ce3b0 2024-04-17T10:24:52.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.472+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35e00b66d0 con 0x7f35e00ce3b0 2024-04-17T10:24:52.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.472+0000 7f35dd600640 1 -- 172.21.15.67:0/3786272816 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35d8059880 con 0x7f35e00ce3b0 2024-04-17T10:24:52.473 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35c0066f60 msgr2=0x7f35c0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35c0066f60 0x7f35c0069420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f35e00cd6e0 tx=0x7f35d4009040 comp rx=0 tx=0).stop 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 msgr2=0x7f35e0159360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e0159360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f35d800c770 tx=0x7f35d800cc30 comp rx=0 tx=0).stop 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 shutdown_connections 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35c0066f60 0x7f35c0069420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e00ce3b0 0x7f35e0159360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35e00b0cd0 0x7f35e0153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 --2- 172.21.15.67:0/3786272816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35e006b1d0 0x7f35e0157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 >> 172.21.15.67:0/3786272816 conn(0x7f35e00b22a0 msgr2=0x7f35e015f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 shutdown_connections 2024-04-17T10:24:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.476+0000 7f35e8400640 1 -- 172.21.15.67:0/3786272816 wait complete. 2024-04-17T10:24:52.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:24:52.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0c5000640 1 Processor -- start 2024-04-17T10:24:52.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0c5000640 1 -- start start 2024-04-17T10:24:52.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0c5000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c00d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0c5000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0c0071840 0x7fb0c00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0c5000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c01456d0 con 0x7fb0c01446c0 2024-04-17T10:24:52.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0c5000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c00b51a0 con 0x7fb0c0071840 2024-04-17T10:24:52.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0c5000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00d1390 con 0x7fb0c0144aa0 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0b5400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c00d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0be800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0c0071840 0x7fb0c00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0be800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb0c0071840 0x7fb0c00d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:51392/0 (socket says 172.21.15.67:51392) 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0be800640 1 -- 172.21.15.67:0/1615991435 learned_addr learned my addr 172.21.15.67:0/1615991435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0bde00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb0c01446c0 0x7fb0c0144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:40188/0 (socket says 172.21.15.67:40188) 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0be800640 1 -- 172.21.15.67:0/1615991435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 msgr2=0x7fb0c00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0be800640 1 --2- 172.21.15.67:0/1615991435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c00d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0be800640 1 -- 172.21.15.67:0/1615991435 >> v1:172.21.15.86:6789/0 conn(0x7fb0c01446c0 legacy=0x7fb0c0144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.636+0000 7fb0be800640 1 -- 172.21.15.67:0/1615991435 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c00b6640 con 0x7fb0c0071840 2024-04-17T10:24:52.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0be800640 1 --2- 172.21.15.67:0/1615991435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c0071840 0x7fb0c00d0e50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00b5320 tx=0x7fb0b801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c702abb4e4a69df2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0bd400640 1 -- 172.21.15.67:0/1615991435 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80ab660 con 0x7fb0c0071840 2024-04-17T10:24:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0bd400640 1 -- 172.21.15.67:0/1615991435 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b80b0070 con 0x7fb0c0071840 2024-04-17T10:24:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0bd400640 1 -- 172.21.15.67:0/1615991435 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b80aba20 con 0x7fb0c0071840 2024-04-17T10:24:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 -- 172.21.15.67:0/1615991435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c0071840 msgr2=0x7fb0c00d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1615991435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c0071840 0x7fb0c00d0e50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb0c00b5320 tx=0x7fb0b801c460 comp rx=0 tx=0).stop 2024-04-17T10:24:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 -- 172.21.15.67:0/1615991435 shutdown_connections 2024-04-17T10:24:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1615991435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c0071840 0x7fb0c00d0e50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1615991435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c00d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 -- 172.21.15.67:0/1615991435 >> 172.21.15.67:0/1615991435 conn(0x7fb0c00b34d0 msgr2=0x7fb0c00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 -- 172.21.15.67:0/1615991435 shutdown_connections 2024-04-17T10:24:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.640+0000 7fb0c5000640 1 -- 172.21.15.67:0/1615991435 wait complete. 2024-04-17T10:24:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 Processor -- start 2024-04-17T10:24:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 -- start start 2024-04-17T10:24:52.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0071840 0x7fb0c015ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c01446c0 0x7fb0c013b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c013cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00b5e20 con 0x7fb0c0071840 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c015c0d0 con 0x7fb0c01446c0 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c00b5fa0 con 0x7fb0c0144aa0 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0071840 0x7fb0c015ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0071840 0x7fb0c015ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50176/0 (socket says 172.21.15.67:50176) 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 -- 172.21.15.67:0/1376049828 learned_addr learned my addr 172.21.15.67:0/1376049828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0b5400640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c01446c0 0x7fb0c013b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0be800640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c013cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 -- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 msgr2=0x7fb0c013cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c013cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 -- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c01446c0 msgr2=0x7fb0c013b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c01446c0 0x7fb0c013b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 -- 172.21.15.67:0/1376049828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c013dc50 con 0x7fb0c0071840 2024-04-17T10:24:52.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0be800640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c013cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:52.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0bde00640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0071840 0x7fb0c015ad80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c420 tx=0x7fb0b000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:52.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0b6a00640 1 -- 172.21.15.67:0/1376049828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00074e0 con 0x7fb0c0071840 2024-04-17T10:24:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0b6a00640 1 -- 172.21.15.67:0/1376049828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b0007680 con 0x7fb0c0071840 2024-04-17T10:24:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0b6a00640 1 -- 172.21.15.67:0/1376049828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0004a30 con 0x7fb0c0071840 2024-04-17T10:24:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.644+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c013df30 con 0x7fb0c0071840 2024-04-17T10:24:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.648+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c015cfd0 con 0x7fb0c0071840 2024-04-17T10:24:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.648+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c0070bb0 con 0x7fb0c0071840 2024-04-17T10:24:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.648+0000 7fb0b6a00640 1 -- 172.21.15.67:0/1376049828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb0b001a020 con 0x7fb0c0071840 2024-04-17T10:24:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.648+0000 7fb0b6a00640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb09c066f90 0x7fb09c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.648+0000 7fb0b6a00640 1 -- 172.21.15.67:0/1376049828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb0b0084570 con 0x7fb0c0071840 2024-04-17T10:24:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.648+0000 7fb0b5400640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb09c066f90 0x7fb09c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.652+0000 7fb0b5400640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb09c066f90 0x7fb09c069450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0c013cb60 tx=0x7fb0a800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:52.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.652+0000 7fb0b6a00640 1 -- 172.21.15.67:0/1376049828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb0b00557a0 con 0x7fb0c0071840 2024-04-17T10:24:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.824+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb0c0071620 con 0x7fb0c0071840 2024-04-17T10:24:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.824+0000 7fb0b6a00640 1 -- 172.21.15.67:0/1376049828 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb0b0084020 con 0x7fb0c0071840 2024-04-17T10:24:52.825 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:24:52.825 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:24:52.825 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:24:52.825 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:24:52.825 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:24:52.825 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:24:52.825 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:24:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb09c066f90 msgr2=0x7fb09c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb09c066f90 0x7fb09c069450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb0c013cb60 tx=0x7fb0a800b040 comp rx=0 tx=0).stop 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0071840 msgr2=0x7fb0c015ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0071840 0x7fb0c015ad80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000c420 tx=0x7fb0b000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 shutdown_connections 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb09c066f90 0x7fb09c069450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb0c0144aa0 0x7fb0c013cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb0c01446c0 0x7fb0c013b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 --2- 172.21.15.67:0/1376049828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb0c0071840 0x7fb0c015ad80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 >> 172.21.15.67:0/1376049828 conn(0x7fb0c00b34d0 msgr2=0x7fb0c00b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 shutdown_connections 2024-04-17T10:24:52.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.828+0000 7fb0c5000640 1 -- 172.21.15.67:0/1376049828 wait complete. 2024-04-17T10:24:52.846 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:24:52.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:24:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa99c400640 1 Processor -- start 2024-04-17T10:24:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa99c400640 1 -- start start 2024-04-17T10:24:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa99c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa994144100 0x7fa9940cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa99c400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa994071360 0x7fa99406e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa99c400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa994071bb0 con 0x7fa99406ea10 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa99c400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9940b3140 con 0x7fa994071360 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa99c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9940b3420 con 0x7fa994144100 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa994071360 0x7fa99406e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa994071360 0x7fa99406e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50182/0 (socket says 172.21.15.67:50182) 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999000640 1 -- 172.21.15.67:0/3933658352 learned_addr learned my addr 172.21.15.67:0/3933658352 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999a00640 1 --2- 172.21.15.67:0/3933658352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa994144100 0x7fa9940cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999000640 1 -- 172.21.15.67:0/3933658352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa994144100 msgr2=0x7fa9940cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999000640 1 --2- 172.21.15.67:0/3933658352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa994144100 0x7fa9940cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999000640 1 -- 172.21.15.67:0/3933658352 >> v1:172.21.15.67:6789/0 conn(0x7fa99406ea10 legacy=0x7fa99415e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.992+0000 7fa999000640 1 -- 172.21.15.67:0/3933658352 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa994071e90 con 0x7fa994071360 2024-04-17T10:24:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa999000640 1 --2- 172.21.15.67:0/3933658352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994071360 0x7fa99406e4d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa98000bb10 tx=0x7fa98001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8f2855c108ce9b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa993e00640 1 -- 172.21.15.67:0/3933658352 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9800b4070 con 0x7fa994071360 2024-04-17T10:24:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa993e00640 1 -- 172.21.15.67:0/3933658352 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9800a7c10 con 0x7fa994071360 2024-04-17T10:24:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa993e00640 1 -- 172.21.15.67:0/3933658352 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9800af440 con 0x7fa994071360 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 -- 172.21.15.67:0/3933658352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994071360 msgr2=0x7fa99406e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 --2- 172.21.15.67:0/3933658352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994071360 0x7fa99406e4d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa98000bb10 tx=0x7fa98001ec30 comp rx=0 tx=0).stop 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 -- 172.21.15.67:0/3933658352 shutdown_connections 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 --2- 172.21.15.67:0/3933658352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994071360 0x7fa99406e4d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 --2- 172.21.15.67:0/3933658352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa994144100 0x7fa9940cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 -- 172.21.15.67:0/3933658352 >> 172.21.15.67:0/3933658352 conn(0x7fa99406ac90 msgr2=0x7fa99406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 -- 172.21.15.67:0/3933658352 shutdown_connections 2024-04-17T10:24:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:52.996+0000 7fa99c400640 1 -- 172.21.15.67:0/3933658352 wait complete. 2024-04-17T10:24:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 Processor -- start 2024-04-17T10:24:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 -- start start 2024-04-17T10:24:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa99406ea10 0x7fa9941f7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa994071360 0x7fa994159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994144100 0x7fa9940ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99406d540 con 0x7fa994144100 2024-04-17T10:24:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99406d3c0 con 0x7fa994071360 2024-04-17T10:24:53.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99406d6c0 con 0x7fa99406ea10 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994144100 0x7fa9940ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994144100 0x7fa9940ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50184/0 (socket says 172.21.15.67:50184) 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 -- 172.21.15.67:0/432347439 learned_addr learned my addr 172.21.15.67:0/432347439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa999a00640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa99406ea10 0x7fa9941f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa999000640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa994071360 0x7fa994159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 -- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa99406ea10 msgr2=0x7fa9941f7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa99406ea10 0x7fa9941f7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 -- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa994071360 msgr2=0x7fa994159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa994071360 0x7fa994159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 -- 172.21.15.67:0/432347439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9940ace00 con 0x7fa994144100 2024-04-17T10:24:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99a400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994144100 0x7fa9940ac8c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00c770 tx=0x7fa98c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa991600640 1 -- 172.21.15.67:0/432347439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c003b80 con 0x7fa994144100 2024-04-17T10:24:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa991600640 1 -- 172.21.15.67:0/432347439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c003d20 con 0x7fa994144100 2024-04-17T10:24:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa991600640 1 -- 172.21.15.67:0/432347439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00aa30 con 0x7fa994144100 2024-04-17T10:24:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99415a6c0 con 0x7fa994144100 2024-04-17T10:24:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.000+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99415cde0 con 0x7fa994144100 2024-04-17T10:24:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.004+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa994072da0 con 0x7fa994144100 2024-04-17T10:24:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.004+0000 7fa991600640 1 -- 172.21.15.67:0/432347439 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa98c00abd0 con 0x7fa994144100 2024-04-17T10:24:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.004+0000 7fa991600640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa974066f60 0x7fa974069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.004+0000 7fa991600640 1 -- 172.21.15.67:0/432347439 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa98c084a20 con 0x7fa994144100 2024-04-17T10:24:53.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.004+0000 7fa999a00640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa974066f60 0x7fa974069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.004+0000 7fa999a00640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa974066f60 0x7fa974069420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9940ae3d0 tx=0x7fa988008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:53.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.008+0000 7fa991600640 1 -- 172.21.15.67:0/432347439 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa98c00ce20 con 0x7fa994144100 2024-04-17T10:24:53.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.172+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa994074e30 con 0x7fa994144100 2024-04-17T10:24:53.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.564+0000 7fa991600640 1 -- 172.21.15.67:0/432347439 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa98c055710 con 0x7fa994144100 2024-04-17T10:24:53.565 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:24:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.568+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa974066f60 msgr2=0x7fa974069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.568+0000 7fa99c400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa974066f60 0x7fa974069420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa9940ae3d0 tx=0x7fa988008040 comp rx=0 tx=0).stop 2024-04-17T10:24:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.568+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994144100 msgr2=0x7fa9940ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.568+0000 7fa99c400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994144100 0x7fa9940ac8c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00c770 tx=0x7fa98c00cc30 comp rx=0 tx=0).stop 2024-04-17T10:24:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 shutdown_connections 2024-04-17T10:24:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa974066f60 0x7fa974069420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa994144100 0x7fa9940ac8c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa994071360 0x7fa994159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 --2- 172.21.15.67:0/432347439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa99406ea10 0x7fa9941f7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 >> 172.21.15.67:0/432347439 conn(0x7fa99406ac90 msgr2=0x7fa99413f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:53.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 shutdown_connections 2024-04-17T10:24:53.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.572+0000 7fa99c400640 1 -- 172.21.15.67:0/432347439 wait complete. 2024-04-17T10:24:53.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-17T10:24:53.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb982a00640 1 Processor -- start 2024-04-17T10:24:53.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb982a00640 1 -- start start 2024-04-17T10:24:53.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb982a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 0x7fb97c0b1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb982a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97c0b2f60 con 0x7fb97c15f020 2024-04-17T10:24:53.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb982a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb97c0cf980 con 0x7fb97c099a50 2024-04-17T10:24:53.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb982a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c0cfc80 con 0x7fb97c0cdd40 2024-04-17T10:24:53.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb981600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb97c15f020 0x7fb97c0cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43174/0 (socket says 172.21.15.67:43174) 2024-04-17T10:24:53.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.744+0000 7fb981600640 1 -- 172.21.15.67:0/2842436679 learned_addr learned my addr 172.21.15.67:0/2842436679 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:24:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb980c00640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 0x7fb97c0b1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209895046 0 0) 0x7fb970001120 con 0x7fb97c15f020 2024-04-17T10:24:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb964003530 con 0x7fb97c15f020 2024-04-17T10:24:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178824520 0 0) 0x7fb97c0cf980 con 0x7fb97c099a50 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb970001120 con 0x7fb97c099a50 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866607344 0 0) 0x7fb964003530 con 0x7fb97c15f020 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 msgr2=0x7fb97c0b1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 0x7fb97c0b1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 >> v1:172.21.15.86:6789/0 conn(0x7fb97c099a50 legacy=0x7fb97c1667d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c0cfed0 con 0x7fb97c15f020 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fb970001850 con 0x7fb97c15f020 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 >> v1:172.21.15.67:6789/0 conn(0x7fb97c15f020 legacy=0x7fb97c0cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb964003b80 0x7fb964003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 0x7fb964004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb964004d50 0x7fb9640051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c0cfc80 con 0x7fb964003b80 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96400ba80 con 0x7fb964004d50 2024-04-17T10:24:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb96400bc80 con 0x7fb97c0cdd40 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb981600640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb964003b80 0x7fb964003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb980c00640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 0x7fb964004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb982000640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb964004d50 0x7fb9640051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb981600640 1 -- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 msgr2=0x7fb964004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb981600640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb97c0cdd40 0x7fb964004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb981600640 1 -- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb964004d50 msgr2=0x7fb9640051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb981600640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb964004d50 0x7fb9640051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.748+0000 7fb981600640 1 -- 172.21.15.67:0/2842436679 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb964003530 con 0x7fb964003b80 2024-04-17T10:24:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb981600640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb964003b80 0x7fb964003f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb97000c320 tx=0x7fb97001cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f524a74f9e39d912 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970007dd0 con 0x7fb964003b80 2024-04-17T10:24:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9700a9070 con 0x7fb964003b80 2024-04-17T10:24:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb97be00640 1 -- 172.21.15.67:0/2842436679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb970005d20 con 0x7fb964003b80 2024-04-17T10:24:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- 172.21.15.67:0/2842436679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb964003b80 msgr2=0x7fb964003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 --2- 172.21.15.67:0/2842436679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb964003b80 0x7fb964003f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb97000c320 tx=0x7fb97001cb30 comp rx=0 tx=0).stop 2024-04-17T10:24:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982000640 1 -- 172.21.15.67:0/2842436679 reap_dead start 2024-04-17T10:24:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- 172.21.15.67:0/2842436679 shutdown_connections 2024-04-17T10:24:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- 172.21.15.67:0/2842436679 >> 172.21.15.67:0/2842436679 conn(0x7fb97c0b3190 msgr2=0x7fb97c0b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- 172.21.15.67:0/2842436679 shutdown_connections 2024-04-17T10:24:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- 172.21.15.67:0/2842436679 wait complete. 2024-04-17T10:24:53.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 Processor -- start 2024-04-17T10:24:53.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- start start 2024-04-17T10:24:53.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb964003b80 0x7fb97c1f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb97c0b6bf0 0x7fb97c15c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb97c1620f0 0x7fb97c15cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c09a270 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c0b2910 con 0x7fb97c1620f0 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.752+0000 7fb982a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97c1f7f70 con 0x7fb964003b80 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb97c0b6bf0 0x7fb97c15c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb97c0b6bf0 0x7fb97c15c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50212/0 (socket says 172.21.15.67:50212) 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 -- 172.21.15.67:0/736837488 learned_addr learned my addr 172.21.15.67:0/736837488 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb981600640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb964003b80 0x7fb97c1f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb982000640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb97c1620f0 0x7fb97c15cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 -- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb964003b80 msgr2=0x7fb97c1f75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb964003b80 0x7fb97c1f75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 -- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb97c1620f0 msgr2=0x7fb97c15cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb97c1620f0 0x7fb97c15cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 -- 172.21.15.67:0/736837488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c066de0 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb982000640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb97c1620f0 0x7fb97c15cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb981600640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb964003b80 0x7fb97c1f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb980c00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb97c0b6bf0 0x7fb97c15c620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9680119d0 tx=0x7fb968011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb979600640 1 -- 172.21.15.67:0/736837488 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96800fb50 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb979600640 1 -- 172.21.15.67:0/736837488 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb96800fcf0 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97c071180 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb979600640 1 -- 172.21.15.67:0/736837488 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96800fea0 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb97c1f8530 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb97c074dc0 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.756+0000 7fb979600640 1 -- 172.21.15.67:0/736837488 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb968012040 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.760+0000 7fb979600640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb95c066f60 0x7fb95c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:53.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.760+0000 7fb979600640 1 -- 172.21.15.67:0/736837488 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb968085bd0 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.760+0000 7fb981600640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb95c066f60 0x7fb95c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:53.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.760+0000 7fb981600640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb95c066f60 0x7fb95c069420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ab000 tx=0x7fb9700a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:53.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.764+0000 7fb979600640 1 -- 172.21.15.67:0/736837488 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb96802b020 con 0x7fb97c0b6bf0 2024-04-17T10:24:53.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:53.924+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fb97c0c4f90 con 0x7fb97c0b6bf0 2024-04-17T10:24:54.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.572+0000 7fb979600640 1 -- 172.21.15.67:0/736837488 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7fb968019030 con 0x7fb97c0b6bf0 2024-04-17T10:24:54.572 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:24:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb95c066f60 msgr2=0x7fb95c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb95c066f60 0x7fb95c069420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb9700ab000 tx=0x7fb9700a7070 comp rx=0 tx=0).stop 2024-04-17T10:24:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb97c0b6bf0 msgr2=0x7fb97c15c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb97c0b6bf0 0x7fb97c15c620 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb9680119d0 tx=0x7fb968011e90 comp rx=0 tx=0).stop 2024-04-17T10:24:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 shutdown_connections 2024-04-17T10:24:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb95c066f60 0x7fb95c069420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb97c1620f0 0x7fb97c15cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb97c0b6bf0 0x7fb97c15c620 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 --2- 172.21.15.67:0/736837488 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb964003b80 0x7fb97c1f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 >> 172.21.15.67:0/736837488 conn(0x7fb97c0b3190 msgr2=0x7fb97c15e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:54.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 shutdown_connections 2024-04-17T10:24:54.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.576+0000 7fb982a00640 1 -- 172.21.15.67:0/736837488 wait complete. 2024-04-17T10:24:54.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:24:54.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.760+0000 7f7821e00640 1 Processor -- start 2024-04-17T10:24:54.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.760+0000 7f7821e00640 1 -- start start 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7821e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f781c144450 0x7f781c0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7821e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f781c072ab0 0x7f781c0d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7821e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f781c15af60 con 0x7f781c0d0c30 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7821e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781c0b3fe0 con 0x7f781c072ab0 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7821e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f781c15bce0 con 0x7f781c144450 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f781c072ab0 0x7f781c0d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f781c072ab0 0x7f781c0d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50220/0 (socket says 172.21.15.67:50220) 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 -- 172.21.15.67:0/414814331 learned_addr learned my addr 172.21.15.67:0/414814331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f781aa00640 1 --2- 172.21.15.67:0/414814331 >> v2:172.21.15.86:3300/0 conn(0x7f781c144450 0x7f781c0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:54.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 -- 172.21.15.67:0/414814331 >> v2:172.21.15.86:3300/0 conn(0x7f781c144450 msgr2=0x7f781c0b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:54.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 --2- 172.21.15.67:0/414814331 >> v2:172.21.15.86:3300/0 conn(0x7f781c144450 0x7f781c0b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 -- 172.21.15.67:0/414814331 >> v1:172.21.15.67:6789/0 conn(0x7f781c0d0c30 legacy=0x7f781c15bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:54.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 -- 172.21.15.67:0/414814331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c15b200 con 0x7f781c072ab0 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.764+0000 7f7812000640 1 --2- 172.21.15.67:0/414814331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c072ab0 0x7f781c0d06f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f781c0b63d0 tx=0x7f780401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46add6bc63239570 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f781a000640 1 -- 172.21.15.67:0/414814331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78040b1070 con 0x7f781c072ab0 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f781a000640 1 -- 172.21.15.67:0/414814331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78040ac600 con 0x7f781c072ab0 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f781a000640 1 -- 172.21.15.67:0/414814331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78040ac9d0 con 0x7f781c072ab0 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- 172.21.15.67:0/414814331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c072ab0 msgr2=0x7f781c0d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 --2- 172.21.15.67:0/414814331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c072ab0 0x7f781c0d06f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f781c0b63d0 tx=0x7f780401c490 comp rx=0 tx=0).stop 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- 172.21.15.67:0/414814331 shutdown_connections 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 --2- 172.21.15.67:0/414814331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c072ab0 0x7f781c0d06f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 --2- 172.21.15.67:0/414814331 >> v2:172.21.15.86:3300/0 conn(0x7f781c144450 0x7f781c0b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- 172.21.15.67:0/414814331 >> 172.21.15.67:0/414814331 conn(0x7f781c0b5ef0 msgr2=0x7f781c099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:54.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- 172.21.15.67:0/414814331 shutdown_connections 2024-04-17T10:24:54.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- 172.21.15.67:0/414814331 wait complete. 2024-04-17T10:24:54.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 Processor -- start 2024-04-17T10:24:54.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- start start 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781c072ab0 0x7f781c151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781c0d0c30 0x7f781c151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c144450 0x7f781c0c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c167390 con 0x7f781c144450 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781c0d1060 con 0x7f781c0d0c30 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7821e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f781c167510 con 0x7f781c072ab0 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f781b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c144450 0x7f781c0c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:54.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f781b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c144450 0x7f781c0c9fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50232/0 (socket says 172.21.15.67:50232) 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f781b400640 1 -- 172.21.15.67:0/215419824 learned_addr learned my addr 172.21.15.67:0/215419824 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f781aa00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781c072ab0 0x7f781c151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.768+0000 7f7812000640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781c0d0c30 0x7f781c151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f781b400640 1 -- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781c072ab0 msgr2=0x7f781c151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f781b400640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781c072ab0 0x7f781c151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f781b400640 1 -- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781c0d0c30 msgr2=0x7f781c151f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f781b400640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781c0d0c30 0x7f781c151f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f781b400640 1 -- 172.21.15.67:0/215419824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781c143040 con 0x7f781c144450 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7812000640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781c0d0c30 0x7f781c151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f781b400640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c144450 0x7f781c0c9fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f781400c760 tx=0x7f781400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7813e00640 1 -- 172.21.15.67:0/215419824 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7814003b80 con 0x7f781c144450 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7813e00640 1 -- 172.21.15.67:0/215419824 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7814003d20 con 0x7f781c144450 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781c143320 con 0x7f781c144450 2024-04-17T10:24:54.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7813e00640 1 -- 172.21.15.67:0/215419824 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781400aa30 con 0x7f781c144450 2024-04-17T10:24:54.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781c1438e0 con 0x7f781c144450 2024-04-17T10:24:54.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781c04ef30 con 0x7f781c144450 2024-04-17T10:24:54.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7813e00640 1 -- 172.21.15.67:0/215419824 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f781400abd0 con 0x7f781c144450 2024-04-17T10:24:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7813e00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f8066f60 0x7f77f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.772+0000 7f7813e00640 1 -- 172.21.15.67:0/215419824 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f78140858c0 con 0x7f781c144450 2024-04-17T10:24:54.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.776+0000 7f781aa00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f8066f60 0x7f77f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:54.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.776+0000 7f781aa00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f8066f60 0x7f77f8069420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f780c009b60 tx=0x7f780c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.776+0000 7f7813e00640 1 -- 172.21.15.67:0/215419824 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7814056910 con 0x7f781c144450 2024-04-17T10:24:54.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:54.972+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f781c13e930 con 0x7f781c144450 2024-04-17T10:24:55.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.576+0000 7f7813e00640 1 -- 172.21.15.67:0/215419824 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7814056210 con 0x7f781c144450 2024-04-17T10:24:55.575 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.580+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f8066f60 msgr2=0x7f77f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.580+0000 7f7821e00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f8066f60 0x7f77f8069420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f780c009b60 tx=0x7f780c009190 comp rx=0 tx=0).stop 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.580+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c144450 msgr2=0x7f781c0c9fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.580+0000 7f7821e00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c144450 0x7f781c0c9fc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f781400c760 tx=0x7f781400cc20 comp rx=0 tx=0).stop 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.584+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 shutdown_connections 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.584+0000 7f7821e00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f8066f60 0x7f77f8069420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.584+0000 7f7821e00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f781c144450 0x7f781c0c9fc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.584+0000 7f7821e00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781c0d0c30 0x7f781c151f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.584+0000 7f7821e00640 1 --2- 172.21.15.67:0/215419824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781c072ab0 0x7f781c151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.584+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 >> 172.21.15.67:0/215419824 conn(0x7f781c0b5ef0 msgr2=0x7f781c0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.588+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 shutdown_connections 2024-04-17T10:24:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.588+0000 7f7821e00640 1 -- 172.21.15.67:0/215419824 wait complete. 2024-04-17T10:24:55.640 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-17T10:24:55.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-17T10:24:55.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.788+0000 7f3b10000640 1 Processor -- start 2024-04-17T10:24:55.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.788+0000 7f3b10000640 1 -- start start 2024-04-17T10:24:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.788+0000 7f3b10000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b08071a70 0x7f3b081665d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.788+0000 7f3b10000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0806d9a0 con 0x7f3b081447e0 2024-04-17T10:24:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.788+0000 7f3b10000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0806dc80 con 0x7f3b0806d3c0 2024-04-17T10:24:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.788+0000 7f3b10000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b080cc310 con 0x7f3b08071a70 2024-04-17T10:24:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b0d600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3b081447e0 0x7f3b08144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43186/0 (socket says 172.21.15.67:43186) 2024-04-17T10:24:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b0d600640 1 -- 172.21.15.67:0/212214124 learned_addr learned my addr 172.21.15.67:0/212214124 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:24:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b0e000640 1 --2- 172.21.15.67:0/212214124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b08071a70 0x7f3b081665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979671740 0 0) 0x7f3b0806d9a0 con 0x7f3b081447e0 2024-04-17T10:24:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af0003530 con 0x7f3b081447e0 2024-04-17T10:24:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018317108 0 0) 0x7f3b0806dc80 con 0x7f3b0806d3c0 2024-04-17T10:24:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b0806d9a0 con 0x7f3b0806d3c0 2024-04-17T10:24:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b0e000640 1 -- 172.21.15.67:0/212214124 >> v1:172.21.15.86:6789/0 conn(0x7f3b0806d3c0 legacy=0x7f3b08071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b0e000640 1 -- 172.21.15.67:0/212214124 >> v1:172.21.15.67:6789/0 conn(0x7f3b081447e0 legacy=0x7f3b08144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:55.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797043674 0 0) 0x7f3af0003530 con 0x7f3b081447e0 2024-04-17T10:24:55.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b0e000640 1 -- 172.21.15.67:0/212214124 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0806df60 con 0x7f3b08071a70 2024-04-17T10:24:55.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b0e000640 1 --2- 172.21.15.67:0/212214124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b08071a70 0x7f3b081665d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009dc0 tx=0x7f3b0001c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b066a917063667c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:55.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.792+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000b2070 con 0x7f3b08071a70 2024-04-17T10:24:55.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b000a58a0 con 0x7f3b08071a70 2024-04-17T10:24:55.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b07e00640 1 -- 172.21.15.67:0/212214124 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b000a5c70 con 0x7f3b08071a70 2024-04-17T10:24:55.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- 172.21.15.67:0/212214124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b08071a70 msgr2=0x7f3b081665d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:55.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 --2- 172.21.15.67:0/212214124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b08071a70 0x7f3b081665d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009dc0 tx=0x7f3b0001c8d0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- 172.21.15.67:0/212214124 shutdown_connections 2024-04-17T10:24:55.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 --2- 172.21.15.67:0/212214124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b08071a70 0x7f3b081665d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- 172.21.15.67:0/212214124 >> 172.21.15.67:0/212214124 conn(0x7f3b0806ac90 msgr2=0x7f3b0806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:55.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- 172.21.15.67:0/212214124 shutdown_connections 2024-04-17T10:24:55.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- 172.21.15.67:0/212214124 wait complete. 2024-04-17T10:24:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 Processor -- start 2024-04-17T10:24:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- start start 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b0806d3c0 0x7f3b0815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b08071a70 0x7f3b0813e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b081447e0 0x7f3b081585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08169ad0 con 0x7f3b0806d3c0 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08169950 con 0x7f3b08071a70 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b10000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b08169c50 con 0x7f3b081447e0 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b0806d3c0 0x7f3b0815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b0806d3c0 0x7f3b0815b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50236/0 (socket says 172.21.15.67:50236) 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 -- 172.21.15.67:0/1683093346 learned_addr learned my addr 172.21.15.67:0/1683093346 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0cc00640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b08071a70 0x7f3b0813e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0e000640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b081447e0 0x7f3b081585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 -- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b081447e0 msgr2=0x7f3b081585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b081447e0 0x7f3b081585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 -- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b08071a70 msgr2=0x7f3b0813e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b08071a70 0x7f3b0813e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0d600640 1 -- 172.21.15.67:0/1683093346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b080aa030 con 0x7f3b0806d3c0 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.796+0000 7f3b0cc00640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b08071a70 0x7f3b0813e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b0d600640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b0806d3c0 0x7f3b0815b2b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0011f0 tx=0x7f3afc00c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:55.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b05600640 1 -- 172.21.15.67:0/1683093346 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc017070 con 0x7f3b0806d3c0 2024-04-17T10:24:55.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b05600640 1 -- 172.21.15.67:0/1683093346 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc005cf0 con 0x7f3b0806d3c0 2024-04-17T10:24:55.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0813eea0 con 0x7f3b0806d3c0 2024-04-17T10:24:55.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b05600640 1 -- 172.21.15.67:0/1683093346 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0072d0 con 0x7f3b0806d3c0 2024-04-17T10:24:55.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0813f460 con 0x7f3b0806d3c0 2024-04-17T10:24:55.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b08072da0 con 0x7f3b0806d3c0 2024-04-17T10:24:55.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b05600640 1 -- 172.21.15.67:0/1683093346 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3afc01a070 con 0x7f3b0806d3c0 2024-04-17T10:24:55.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b05600640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ae8066f60 0x7f3ae8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:55.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b05600640 1 -- 172.21.15.67:0/1683093346 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3afc084370 con 0x7f3b0806d3c0 2024-04-17T10:24:55.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.800+0000 7f3b0cc00640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ae8066f60 0x7f3ae8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:55.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.804+0000 7f3b0cc00640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ae8066f60 0x7f3ae8069420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3b0813baa0 tx=0x7f3af4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:55.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.804+0000 7f3b05600640 1 -- 172.21.15.67:0/1683093346 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3afc054c90 con 0x7f3b0806d3c0 2024-04-17T10:24:55.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:55.976+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f3b08140a90 con 0x7f3b0806d3c0 2024-04-17T10:24:56.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.591+0000 7f3b05600640 1 -- 172.21.15.67:0/1683093346 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) v1 ==== 157+0+0 (secure 0 0 0) 0x7f3afc058d30 con 0x7f3b0806d3c0 2024-04-17T10:24:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ae8066f60 msgr2=0x7f3ae8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ae8066f60 0x7f3ae8069420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3b0813baa0 tx=0x7f3af4009040 comp rx=0 tx=0).stop 2024-04-17T10:24:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b0806d3c0 msgr2=0x7f3b0815b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b0806d3c0 0x7f3b0815b2b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0011f0 tx=0x7f3afc00c870 comp rx=0 tx=0).stop 2024-04-17T10:24:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 shutdown_connections 2024-04-17T10:24:56.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ae8066f60 0x7f3ae8069420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b081447e0 0x7f3b081585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b08071a70 0x7f3b0813e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 --2- 172.21.15.67:0/1683093346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b0806d3c0 0x7f3b0815b2b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 >> 172.21.15.67:0/1683093346 conn(0x7f3b0806ac90 msgr2=0x7f3b0813c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 shutdown_connections 2024-04-17T10:24:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.595+0000 7f3b10000640 1 -- 172.21.15.67:0/1683093346 wait complete. 2024-04-17T10:24:56.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:24:56.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff15b400640 1 Processor -- start 2024-04-17T10:24:56.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff15b400640 1 -- start start 2024-04-17T10:24:56.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff15b400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff15415f230 0x7ff15409a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:56.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff15b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1540cc2e0 0x7ff154099cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:56.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff15b400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1540b3290 con 0x7ff1540cc770 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff15b400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff15415f610 con 0x7ff1540cc2e0 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff15b400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1540cf6b0 con 0x7ff15415f230 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff14be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1540cc2e0 0x7ff154099cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff14be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff1540cc2e0 0x7ff154099cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50240/0 (socket says 172.21.15.67:50240) 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff14be00640 1 -- 172.21.15.67:0/3961785335 learned_addr learned my addr 172.21.15.67:0/3961785335 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff158a00640 1 --2- 172.21.15.67:0/3961785335 >> v2:172.21.15.86:3300/0 conn(0x7ff15415f230 0x7ff15409a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff14be00640 1 -- 172.21.15.67:0/3961785335 >> v2:172.21.15.86:3300/0 conn(0x7ff15415f230 msgr2=0x7ff15409a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff14be00640 1 --2- 172.21.15.67:0/3961785335 >> v2:172.21.15.86:3300/0 conn(0x7ff15415f230 0x7ff15409a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff14be00640 1 -- 172.21.15.67:0/3961785335 >> v1:172.21.15.86:6789/0 conn(0x7ff1540cc770 legacy=0x7ff1540b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.771+0000 7ff14be00640 1 -- 172.21.15.67:0/3961785335 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540b0fc0 con 0x7ff1540cc2e0 2024-04-17T10:24:56.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff14be00640 1 --2- 172.21.15.67:0/3961785335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154099cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff15409fa50 tx=0x7ff13c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0641ffc396da70 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:56.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff153e00640 1 -- 172.21.15.67:0/3961785335 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0b0070 con 0x7ff1540cc2e0 2024-04-17T10:24:56.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff153e00640 1 -- 172.21.15.67:0/3961785335 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13c0ab600 con 0x7ff1540cc2e0 2024-04-17T10:24:56.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff153e00640 1 -- 172.21.15.67:0/3961785335 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0ab9d0 con 0x7ff1540cc2e0 2024-04-17T10:24:56.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 -- 172.21.15.67:0/3961785335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 msgr2=0x7ff154099cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:56.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 --2- 172.21.15.67:0/3961785335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154099cb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff15409fa50 tx=0x7ff13c01c420 comp rx=0 tx=0).stop 2024-04-17T10:24:56.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 -- 172.21.15.67:0/3961785335 shutdown_connections 2024-04-17T10:24:56.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 --2- 172.21.15.67:0/3961785335 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154099cb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 --2- 172.21.15.67:0/3961785335 >> v2:172.21.15.86:3300/0 conn(0x7ff15415f230 0x7ff15409a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 -- 172.21.15.67:0/3961785335 >> 172.21.15.67:0/3961785335 conn(0x7ff1540b34a0 msgr2=0x7ff1540b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:56.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 -- 172.21.15.67:0/3961785335 shutdown_connections 2024-04-17T10:24:56.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.775+0000 7ff15b400640 1 -- 172.21.15.67:0/3961785335 wait complete. 2024-04-17T10:24:56.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 Processor -- start 2024-04-17T10:24:56.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 -- start start 2024-04-17T10:24:56.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:56.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1540cc770 0x7ff1541f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:56.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff15415f230 0x7ff15406c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:56.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540b3fc0 con 0x7ff1540cc2e0 2024-04-17T10:24:56.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15409a600 con 0x7ff15415f230 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1540b4140 con 0x7ff1540cc770 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50252/0 (socket says 172.21.15.67:50252) 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 -- 172.21.15.67:0/1339427584 learned_addr learned my addr 172.21.15.67:0/1339427584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff14be00640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1540cc770 0x7ff1541f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff159400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff15415f230 0x7ff15406c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 -- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1540cc770 msgr2=0x7ff1541f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1540cc770 0x7ff1541f8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 -- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff15415f230 msgr2=0x7ff15406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff15415f230 0x7ff15406c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 -- 172.21.15.67:0/1339427584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff154163c40 con 0x7ff1540cc2e0 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff159400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff15415f230 0x7ff15406c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:56.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff158a00640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154157fa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff14400d800 tx=0x7ff14400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff151600640 1 -- 172.21.15.67:0/1339427584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400b6c0 con 0x7ff1540cc2e0 2024-04-17T10:24:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff151600640 1 -- 172.21.15.67:0/1339427584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff144019070 con 0x7ff1540cc2e0 2024-04-17T10:24:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff151600640 1 -- 172.21.15.67:0/1339427584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14400ba60 con 0x7ff1540cc2e0 2024-04-17T10:24:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff154163f20 con 0x7ff1540cc2e0 2024-04-17T10:24:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.779+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1541644e0 con 0x7ff1540cc2e0 2024-04-17T10:24:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.783+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15404ef30 con 0x7ff1540cc2e0 2024-04-17T10:24:56.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.783+0000 7ff151600640 1 -- 172.21.15.67:0/1339427584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff14400bc00 con 0x7ff1540cc2e0 2024-04-17T10:24:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.783+0000 7ff151600640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff130066f60 0x7ff130069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.783+0000 7ff151600640 1 -- 172.21.15.67:0/1339427584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6007+0+0 (secure 0 0 0) 0x7ff144085640 con 0x7ff1540cc2e0 2024-04-17T10:24:56.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.783+0000 7ff14be00640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff130066f60 0x7ff130069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.783+0000 7ff14be00640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff130066f60 0x7ff130069420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0ad000 tx=0x7ff13c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:56.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.787+0000 7ff151600640 1 -- 172.21.15.67:0/1339427584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff144056480 con 0x7ff1540cc2e0 2024-04-17T10:24:56.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:56.951+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff154099d20 con 0x7ff1540cc2e0 2024-04-17T10:24:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.611+0000 7ff151600640 1 -- 172.21.15.67:0/1339427584 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) v1 ==== 228+0+0 (secure 0 0 0) 0x7ff144055d80 con 0x7ff1540cc2e0 2024-04-17T10:24:58.611 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data_ec' created 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff130066f60 msgr2=0x7ff130069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff130066f60 0x7ff130069420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff13c0ad000 tx=0x7ff13c008040 comp rx=0 tx=0).stop 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 msgr2=0x7ff154157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154157fa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff14400d800 tx=0x7ff14400dcc0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 shutdown_connections 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff130066f60 0x7ff130069420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff15415f230 0x7ff15406c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff1540cc770 0x7ff1541f8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 --2- 172.21.15.67:0/1339427584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff1540cc2e0 0x7ff154157fa0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 >> 172.21.15.67:0/1339427584 conn(0x7ff1540b34a0 msgr2=0x7ff1540ce190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:58.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 shutdown_connections 2024-04-17T10:24:58.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.619+0000 7ff15b400640 1 -- 172.21.15.67:0/1339427584 wait complete. 2024-04-17T10:24:58.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-17T10:24:58.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.795+0000 7f898d000640 1 Processor -- start 2024-04-17T10:24:58.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.795+0000 7f898d000640 1 -- start start 2024-04-17T10:24:58.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f898d000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898806d3c0 0x7f8988071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:58.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f898d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89880718a0 0x7f89880cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f898d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898806d9a0 con 0x7f89881447e0 2024-04-17T10:24:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f898d000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8988071d10 con 0x7f89880718a0 2024-04-17T10:24:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f898d000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89880cc310 con 0x7f898806d3c0 2024-04-17T10:24:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f8986800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89880718a0 0x7f89880cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f8986800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89880718a0 0x7f89880cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50262/0 (socket says 172.21.15.67:50262) 2024-04-17T10:24:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f8986800640 1 -- 172.21.15.67:0/4270224308 learned_addr learned my addr 172.21.15.67:0/4270224308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:58.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f897fe00640 1 --2- 172.21.15.67:0/4270224308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898806d3c0 0x7f8988071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:58.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f8986800640 1 -- 172.21.15.67:0/4270224308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898806d3c0 msgr2=0x7f8988071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:58.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f8986800640 1 --2- 172.21.15.67:0/4270224308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898806d3c0 0x7f8988071360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f8986800640 1 -- 172.21.15.67:0/4270224308 >> v1:172.21.15.67:6789/0 conn(0x7f89881447e0 legacy=0x7f8988144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:58.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.799+0000 7f8986800640 1 -- 172.21.15.67:0/4270224308 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898806dc80 con 0x7f89880718a0 2024-04-17T10:24:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f8986800640 1 --2- 172.21.15.67:0/4270224308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89880718a0 0x7f89880cc7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8980009e50 tx=0x7f898001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef89ce00c2656f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f8985400640 1 -- 172.21.15.67:0/4270224308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89800a5740 con 0x7f89880718a0 2024-04-17T10:24:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f8985400640 1 -- 172.21.15.67:0/4270224308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89800a58e0 con 0x7f89880718a0 2024-04-17T10:24:58.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f8985400640 1 -- 172.21.15.67:0/4270224308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89800a5cb0 con 0x7f89880718a0 2024-04-17T10:24:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- 172.21.15.67:0/4270224308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89880718a0 msgr2=0x7f89880cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 --2- 172.21.15.67:0/4270224308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89880718a0 0x7f89880cc7a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8980009e50 tx=0x7f898001c970 comp rx=0 tx=0).stop 2024-04-17T10:24:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- 172.21.15.67:0/4270224308 shutdown_connections 2024-04-17T10:24:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 --2- 172.21.15.67:0/4270224308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89880718a0 0x7f89880cc7a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 --2- 172.21.15.67:0/4270224308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f898806d3c0 0x7f8988071360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- 172.21.15.67:0/4270224308 >> 172.21.15.67:0/4270224308 conn(0x7f898806ac90 msgr2=0x7f898806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:58.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- 172.21.15.67:0/4270224308 shutdown_connections 2024-04-17T10:24:58.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- 172.21.15.67:0/4270224308 wait complete. 2024-04-17T10:24:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 Processor -- start 2024-04-17T10:24:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- start start 2024-04-17T10:24:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f898806d3c0 0x7f89880b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89880718a0 0x7f898815ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89881447e0 0x7f89880b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89880cd8c0 con 0x7f898806d3c0 2024-04-17T10:24:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89880cd740 con 0x7f89881447e0 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f898d000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89880cda40 con 0x7f89880718a0 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.803+0000 7f8985e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f898806d3c0 0x7f89880b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f898806d3c0 0x7f89880b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50274/0 (socket says 172.21.15.67:50274) 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 -- 172.21.15.67:0/2532261661 learned_addr learned my addr 172.21.15.67:0/2532261661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f897fe00640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89880718a0 0x7f898815ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8986800640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89881447e0 0x7f89880b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 -- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89880718a0 msgr2=0x7f898815ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89880718a0 0x7f898815ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 -- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89881447e0 msgr2=0x7f89880b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89881447e0 0x7f89880b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 -- 172.21.15.67:0/2532261661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89880cb100 con 0x7f898806d3c0 2024-04-17T10:24:58.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8986800640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89881447e0 0x7f89880b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f8985e00640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f898806d3c0 0x7f89880b1510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f897800c450 tx=0x7f897800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f897e000640 1 -- 172.21.15.67:0/2532261661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978016020 con 0x7f898806d3c0 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f897e000640 1 -- 172.21.15.67:0/2532261661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89780052d0 con 0x7f898806d3c0 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f897e000640 1 -- 172.21.15.67:0/2532261661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978007260 con 0x7f898806d3c0 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89880cb3e0 con 0x7f898806d3c0 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8988140ac0 con 0x7f898806d3c0 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8988072da0 con 0x7f898806d3c0 2024-04-17T10:24:58.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f897e000640 1 -- 172.21.15.67:0/2532261661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8978007400 con 0x7f898806d3c0 2024-04-17T10:24:58.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.807+0000 7f897e000640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8964066f60 0x7f8964069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:58.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.811+0000 7f897e000640 1 -- 172.21.15.67:0/2532261661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f8978084ce0 con 0x7f898806d3c0 2024-04-17T10:24:58.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.811+0000 7f897fe00640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8964066f60 0x7f8964069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:58.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.811+0000 7f897fe00640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8964066f60 0x7f8964069420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89880aea70 tx=0x7f8970008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:58.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.815+0000 7f897e000640 1 -- 172.21.15.67:0/2532261661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8978020020 con 0x7f898806d3c0 2024-04-17T10:24:58.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:58.975+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f8988074e30 con 0x7f898806d3c0 2024-04-17T10:24:59.491 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T10:24:59.491+0000 7fb6eee00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:24:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.615+0000 7f897e000640 1 -- 172.21.15.67:0/2532261661 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8978005480 con 0x7f898806d3c0 2024-04-17T10:24:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:set pool 4 allow_ec_overwrites to true 2024-04-17T10:24:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8964066f60 msgr2=0x7f8964069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8964066f60 0x7f8964069420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f89880aea70 tx=0x7f8970008040 comp rx=0 tx=0).stop 2024-04-17T10:24:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f898806d3c0 msgr2=0x7f89880b1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f898806d3c0 0x7f89880b1510 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f897800c450 tx=0x7f897800c910 comp rx=0 tx=0).stop 2024-04-17T10:24:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 shutdown_connections 2024-04-17T10:24:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8964066f60 0x7f8964069420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89881447e0 0x7f89880b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89880718a0 0x7f898815ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 --2- 172.21.15.67:0/2532261661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f898806d3c0 0x7f89880b1510 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f897800c450 tx=0x7f897800c910 comp rx=0 tx=0).stop 2024-04-17T10:24:59.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.623+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 >> 172.21.15.67:0/2532261661 conn(0x7f898806ac90 msgr2=0x7f898813bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.627+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 shutdown_connections 2024-04-17T10:24:59.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.627+0000 7f898d000640 1 -- 172.21.15.67:0/2532261661 wait complete. 2024-04-17T10:24:59.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-17T10:24:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5526000640 1 Processor -- start 2024-04-17T10:24:59.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5526000640 1 -- start start 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5526000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f552015dd60 0x7f5520161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5526000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f552009adf0 0x7f5520161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5526000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552015e140 con 0x7f55200cff90 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5526000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55201626d0 con 0x7f552015dd60 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5526000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5520099d50 con 0x7f552009adf0 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f551f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f552015dd60 0x7f5520161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f551f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f552015dd60 0x7f5520161880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50278/0 (socket says 172.21.15.67:50278) 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f551f400640 1 -- 172.21.15.67:0/3120340624 learned_addr learned my addr 172.21.15.67:0/3120340624 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f5516a00640 1 --2- 172.21.15.67:0/3120340624 >> v2:172.21.15.86:3300/0 conn(0x7f552009adf0 0x7f5520161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f551f400640 1 -- 172.21.15.67:0/3120340624 >> v2:172.21.15.86:3300/0 conn(0x7f552009adf0 msgr2=0x7f5520161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:24:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f551f400640 1 --2- 172.21.15.67:0/3120340624 >> v2:172.21.15.86:3300/0 conn(0x7f552009adf0 0x7f5520161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f551f400640 1 -- 172.21.15.67:0/3120340624 >> v1:172.21.15.86:6789/0 conn(0x7f55200cff90 legacy=0x7f55200d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.819+0000 7f551f400640 1 -- 172.21.15.67:0/3120340624 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552009a030 con 0x7f552015dd60 2024-04-17T10:24:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f551f400640 1 --2- 172.21.15.67:0/3120340624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552015dd60 0x7f5520161880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5520073040 tx=0x7f551001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76018f36e8be4c30 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f551ea00640 1 -- 172.21.15.67:0/3120340624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100aa650 con 0x7f552015dd60 2024-04-17T10:24:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f551ea00640 1 -- 172.21.15.67:0/3120340624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55100af070 con 0x7f552015dd60 2024-04-17T10:24:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f551ea00640 1 -- 172.21.15.67:0/3120340624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100aaa10 con 0x7f552015dd60 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 -- 172.21.15.67:0/3120340624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552015dd60 msgr2=0x7f5520161880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 --2- 172.21.15.67:0/3120340624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552015dd60 0x7f5520161880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5520073040 tx=0x7f551001c4f0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 -- 172.21.15.67:0/3120340624 shutdown_connections 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 --2- 172.21.15.67:0/3120340624 >> v2:172.21.15.86:3300/0 conn(0x7f552009adf0 0x7f5520161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 --2- 172.21.15.67:0/3120340624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552015dd60 0x7f5520161880 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 -- 172.21.15.67:0/3120340624 >> 172.21.15.67:0/3120340624 conn(0x7f55200b34a0 msgr2=0x7f55200b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 -- 172.21.15.67:0/3120340624 shutdown_connections 2024-04-17T10:24:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 -- 172.21.15.67:0/3120340624 wait complete. 2024-04-17T10:24:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 Processor -- start 2024-04-17T10:24:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.823+0000 7f5526000640 1 -- start start 2024-04-17T10:24:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552009adf0 0x7f55200ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55200cff90 0x7f552015b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552015dd60 0x7f552006fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552006c730 con 0x7f552009adf0 2024-04-17T10:24:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552006c5b0 con 0x7f55200cff90 2024-04-17T10:24:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552006c8b0 con 0x7f552015dd60 2024-04-17T10:24:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552009adf0 0x7f55200ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552009adf0 0x7f55200ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50284/0 (socket says 172.21.15.67:50284) 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 -- 172.21.15.67:0/2623561877 learned_addr learned my addr 172.21.15.67:0/2623561877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551fe00640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552015dd60 0x7f552006fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5516a00640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55200cff90 0x7f552015b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 -- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552015dd60 msgr2=0x7f552006fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552015dd60 0x7f552006fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 -- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55200cff90 msgr2=0x7f552015b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55200cff90 0x7f552015b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 -- 172.21.15.67:0/2623561877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552015c4d0 con 0x7f552009adf0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f551f400640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552009adf0 0x7f55200ce3b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5510019100 tx=0x7f551001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5517e00640 1 -- 172.21.15.67:0/2623561877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100bd5f0 con 0x7f552009adf0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5517e00640 1 -- 172.21.15.67:0/2623561877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55100a6040 con 0x7f552009adf0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55200cd8d0 con 0x7f552009adf0 2024-04-17T10:24:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5517e00640 1 -- 172.21.15.67:0/2623561877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55100bd9d0 con 0x7f552009adf0 2024-04-17T10:24:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55200cde90 con 0x7f552009adf0 2024-04-17T10:24:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552004ef30 con 0x7f552009adf0 2024-04-17T10:24:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.827+0000 7f5517e00640 1 -- 172.21.15.67:0/2623561877 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f55100b33f0 con 0x7f552009adf0 2024-04-17T10:24:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.831+0000 7f5517e00640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54fc066f60 0x7f54fc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:24:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.831+0000 7f5517e00640 1 -- 172.21.15.67:0/2623561877 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6500+0+0 (secure 0 0 0) 0x7f551011f9e0 con 0x7f552009adf0 2024-04-17T10:24:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.831+0000 7f5516a00640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54fc066f60 0x7f54fc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:24:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.831+0000 7f5516a00640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54fc066f60 0x7f54fc069420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f55200cbf60 tx=0x7f5508008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:24:59.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:24:59.835+0000 7f5517e00640 1 -- 172.21.15.67:0/2623561877 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f55100c3060 con 0x7f552009adf0 2024-04-17T10:25:00.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.031+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f55200b42d0 con 0x7f552009adf0 2024-04-17T10:25:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.631+0000 7f5517e00640 1 -- 172.21.15.67:0/2623561877 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f5510016110 con 0x7f552009adf0 2024-04-17T10:25:00.632 INFO:teuthology.orchestra.run.smithi067.stderr:added data pool 4 to fsmap 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54fc066f60 msgr2=0x7f54fc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54fc066f60 0x7f54fc069420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f55200cbf60 tx=0x7f5508008040 comp rx=0 tx=0).stop 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552009adf0 msgr2=0x7f55200ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552009adf0 0x7f55200ce3b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5510019100 tx=0x7f551001ad30 comp rx=0 tx=0).stop 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 shutdown_connections 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54fc066f60 0x7f54fc069420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f552015dd60 0x7f552006fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55200cff90 0x7f552015b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 --2- 172.21.15.67:0/2623561877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f552009adf0 0x7f55200ce3b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 >> 172.21.15.67:0/2623561877 conn(0x7f55200b34a0 msgr2=0x7f55200b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 shutdown_connections 2024-04-17T10:25:00.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.635+0000 7f5526000640 1 -- 172.21.15.67:0/2623561877 wait complete. 2024-04-17T10:25:00.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:25:00.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a3800640 1 Processor -- start 2024-04-17T10:25:00.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a3800640 1 -- start start 2024-04-17T10:25:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a3800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c1620a0 0x7f699c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a3800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f699c15e550 0x7f699c0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a3800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699c15eb50 con 0x7f699c0ce020 2024-04-17T10:25:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a3800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c161650 con 0x7f699c15e550 2024-04-17T10:25:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a3800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c15ee30 con 0x7f699c1620a0 2024-04-17T10:25:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a0e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f699c0ce020 0x7f699c0ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39130/0 (socket says 172.21.15.67:39130) 2024-04-17T10:25:00.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a0e00640 1 -- 172.21.15.67:0/2915919898 learned_addr learned my addr 172.21.15.67:0/2915919898 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f699be00640 1 --2- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c1620a0 0x7f699c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a1800640 1 --2- 172.21.15.67:0/2915919898 >> v2:172.21.15.86:3300/0 conn(0x7f699c15e550 0x7f699c0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f699b400640 1 -- 172.21.15.67:0/2915919898 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752766587 0 0) 0x7f699c15eb50 con 0x7f699c0ce020 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f699b400640 1 -- 172.21.15.67:0/2915919898 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6984003530 con 0x7f699c0ce020 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a1800640 1 -- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c1620a0 msgr2=0x7f699c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a1800640 1 --2- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c1620a0 0x7f699c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a1800640 1 -- 172.21.15.67:0/2915919898 >> v1:172.21.15.67:6789/0 conn(0x7f699c0ce020 legacy=0x7f699c0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f699b400640 1 -- 172.21.15.67:0/2915919898 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3242127007 0 0) 0x7f6984003530 con 0x7f699c0ce020 2024-04-17T10:25:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.795+0000 7f69a1800640 1 -- 172.21.15.67:0/2915919898 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c1618a0 con 0x7f699c15e550 2024-04-17T10:25:00.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a1800640 1 --2- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c15e550 0x7f699c0cf510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699401c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a10dbc6215614cb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:00.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f699b400640 1 -- 172.21.15.67:0/2915919898 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69940b2070 con 0x7f699c15e550 2024-04-17T10:25:00.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f699b400640 1 -- 172.21.15.67:0/2915919898 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69940a5570 con 0x7f699c15e550 2024-04-17T10:25:00.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f699b400640 1 -- 172.21.15.67:0/2915919898 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990001680 con 0x7f699c15e550 2024-04-17T10:25:00.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 -- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c15e550 msgr2=0x7f699c0cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 --2- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c15e550 0x7f699c0cf510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6994009d90 tx=0x7f699401c860 comp rx=0 tx=0).stop 2024-04-17T10:25:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 -- 172.21.15.67:0/2915919898 shutdown_connections 2024-04-17T10:25:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 --2- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c15e550 0x7f699c0cf510 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 --2- 172.21.15.67:0/2915919898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c1620a0 0x7f699c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 -- 172.21.15.67:0/2915919898 >> 172.21.15.67:0/2915919898 conn(0x7f699c0b33f0 msgr2=0x7f699c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 -- 172.21.15.67:0/2915919898 shutdown_connections 2024-04-17T10:25:00.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.799+0000 7f69a3800640 1 -- 172.21.15.67:0/2915919898 wait complete. 2024-04-17T10:25:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 Processor -- start 2024-04-17T10:25:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 -- start start 2024-04-17T10:25:00.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c0ce020 0x7f699c0b70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:00.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f699c15e550 0x7f699c1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c1620a0 0x7f699c0b5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c0c8630 con 0x7f699c15e550 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c0c84b0 con 0x7f699c1620a0 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c16a9f0 con 0x7f699c0ce020 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f699c15e550 0x7f699c1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f699c15e550 0x7f699c1598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42992/0 (socket says 172.21.15.67:42992) 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 -- 172.21.15.67:0/3882487845 learned_addr learned my addr 172.21.15.67:0/3882487845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a0e00640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c0ce020 0x7f699c0b70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a1800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c1620a0 0x7f699c0b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 -- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c0ce020 msgr2=0x7f699c0b70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c0ce020 0x7f699c0b70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 -- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c1620a0 msgr2=0x7f699c0b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:00.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c1620a0 0x7f699c0b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 -- 172.21.15.67:0/3882487845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c15a010 con 0x7f699c15e550 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a1800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c1620a0 0x7f699c0b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f699be00640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f699c15e550 0x7f699c1598a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f698800e870 tx=0x7f698800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f6998c00640 1 -- 172.21.15.67:0/3882487845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988017070 con 0x7f699c15e550 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f6998c00640 1 -- 172.21.15.67:0/3882487845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698800c940 con 0x7f699c15e550 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f6998c00640 1 -- 172.21.15.67:0/3882487845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698800ccf0 con 0x7f699c15e550 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c15a2f0 con 0x7f699c15e550 2024-04-17T10:25:00.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.803+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c15a8b0 con 0x7f699c15e550 2024-04-17T10:25:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.807+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c070ab0 con 0x7f699c15e550 2024-04-17T10:25:00.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.807+0000 7f6998c00640 1 -- 172.21.15.67:0/3882487845 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f698801a070 con 0x7f699c15e550 2024-04-17T10:25:00.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.807+0000 7f6998c00640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f697c066f60 0x7f697c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:00.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.807+0000 7f6998c00640 1 -- 172.21.15.67:0/3882487845 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f69880855f0 con 0x7f699c15e550 2024-04-17T10:25:00.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.807+0000 7f69a0e00640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f697c066f60 0x7f697c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:00.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.807+0000 7f69a0e00640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f697c066f60 0x7f697c069420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6990001ec0 tx=0x7f69900016e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:00.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:00.811+0000 7f6998c00640 1 -- 172.21.15.67:0/3882487845 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6988055bd0 con 0x7f699c15e550 2024-04-17T10:25:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f699c0b3190 con 0x7f699c15e550 2024-04-17T10:25:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f6998c00640 1 -- 172.21.15.67:0/3882487845 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f6988059c70 con 0x7f699c15e550 2024-04-17T10:25:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:01.011 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:00.039461+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-17T10:25:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f697c066f60 msgr2=0x7f697c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f697c066f60 0x7f697c069420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6990001ec0 tx=0x7f69900016e0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f699c15e550 msgr2=0x7f699c1598a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f699c15e550 0x7f699c1598a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f698800e870 tx=0x7f698800ec70 comp rx=0 tx=0).stop 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 shutdown_connections 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f697c066f60 0x7f697c069420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f699c1620a0 0x7f699c0b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f699c15e550 0x7f699c1598a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 --2- 172.21.15.67:0/3882487845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f699c0ce020 0x7f699c0b70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 >> 172.21.15.67:0/3882487845 conn(0x7f699c0b33f0 msgr2=0x7f699c0d0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 shutdown_connections 2024-04-17T10:25:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.011+0000 7f69a3800640 1 -- 172.21.15.67:0/3882487845 wait complete. 2024-04-17T10:25:01.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:01.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 Processor -- start 2024-04-17T10:25:01.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 -- start start 2024-04-17T10:25:01.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d881446c0 0x7f6d88144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d88144aa0 0x7f6d880d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d88071840 0x7f6d880d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d880b51a0 con 0x7f6d88071840 2024-04-17T10:25:01.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d880d1390 con 0x7f6d88144aa0 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d8dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d881456d0 con 0x7f6d881446c0 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d88071840 0x7f6d880d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6d88071840 0x7f6d880d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43008/0 (socket says 172.21.15.67:43008) 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 -- 172.21.15.67:0/52774855 learned_addr learned my addr 172.21.15.67:0/52774855 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d86000640 1 --2- 172.21.15.67:0/52774855 >> v2:172.21.15.86:3300/0 conn(0x7f6d88144aa0 0x7f6d880d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d86a00640 1 --2- 172.21.15.67:0/52774855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d881446c0 0x7f6d88144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 -- 172.21.15.67:0/52774855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d881446c0 msgr2=0x7f6d88144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 --2- 172.21.15.67:0/52774855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d881446c0 0x7f6d88144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 -- 172.21.15.67:0/52774855 >> v2:172.21.15.86:3300/0 conn(0x7f6d88144aa0 msgr2=0x7f6d880d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 --2- 172.21.15.67:0/52774855 >> v2:172.21.15.86:3300/0 conn(0x7f6d88144aa0 0x7f6d880d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d87400640 1 -- 172.21.15.67:0/52774855 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d880b4870 con 0x7f6d88071840 2024-04-17T10:25:01.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.175+0000 7f6d86a00640 1 --2- 172.21.15.67:0/52774855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d881446c0 0x7f6d88144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d87400640 1 --2- 172.21.15.67:0/52774855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d880d0e50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d88068530 tx=0x7f6d8001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d2bf10c6b25cf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d85600640 1 -- 172.21.15.67:0/52774855 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800a57b0 con 0x7f6d88071840 2024-04-17T10:25:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d85600640 1 -- 172.21.15.67:0/52774855 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d800a5950 con 0x7f6d88071840 2024-04-17T10:25:01.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d85600640 1 -- 172.21.15.67:0/52774855 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800a5d20 con 0x7f6d88071840 2024-04-17T10:25:01.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 -- 172.21.15.67:0/52774855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 msgr2=0x7f6d880d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/52774855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d880d0e50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6d88068530 tx=0x7f6d8001c930 comp rx=0 tx=0).stop 2024-04-17T10:25:01.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 -- 172.21.15.67:0/52774855 shutdown_connections 2024-04-17T10:25:01.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/52774855 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d880d0e50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/52774855 >> v2:172.21.15.86:3300/0 conn(0x7f6d88144aa0 0x7f6d880d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/52774855 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d881446c0 0x7f6d88144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 -- 172.21.15.67:0/52774855 >> 172.21.15.67:0/52774855 conn(0x7f6d880b34d0 msgr2=0x7f6d880b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:01.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 -- 172.21.15.67:0/52774855 shutdown_connections 2024-04-17T10:25:01.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.179+0000 7f6d8dc00640 1 -- 172.21.15.67:0/52774855 wait complete. 2024-04-17T10:25:01.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 Processor -- start 2024-04-17T10:25:01.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 -- start start 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d88158be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d881446c0 0x7f6d880affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88144aa0 0x7f6d8813ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8815fed0 con 0x7f6d88071840 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8815c0d0 con 0x7f6d881446c0 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d88160050 con 0x7f6d88144aa0 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d88158be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d88158be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43014/0 (socket says 172.21.15.67:43014) 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 -- 172.21.15.67:0/3875908450 learned_addr learned my addr 172.21.15.67:0/3875908450 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86000640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d881446c0 0x7f6d880affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d87400640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88144aa0 0x7f6d8813ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 -- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88144aa0 msgr2=0x7f6d8813ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88144aa0 0x7f6d8813ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 -- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d881446c0 msgr2=0x7f6d880affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d881446c0 0x7f6d880affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 -- 172.21.15.67:0/3875908450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8813ee30 con 0x7f6d88071840 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d87400640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88144aa0 0x7f6d8813ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d86a00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d88158be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800e810 tx=0x7f6d7800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d7ea00640 1 -- 172.21.15.67:0/3875908450 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7800f040 con 0x7f6d88071840 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d7ea00640 1 -- 172.21.15.67:0/3875908450 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d78004030 con 0x7f6d88071840 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8813f0b0 con 0x7f6d88071840 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d7ea00640 1 -- 172.21.15.67:0/3875908450 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78004400 con 0x7f6d88071840 2024-04-17T10:25:01.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.183+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8813f670 con 0x7f6d88071840 2024-04-17T10:25:01.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.187+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d88070bb0 con 0x7f6d88071840 2024-04-17T10:25:01.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.187+0000 7f6d7ea00640 1 -- 172.21.15.67:0/3875908450 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6d7800c6d0 con 0x7f6d88071840 2024-04-17T10:25:01.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.187+0000 7f6d7ea00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d68066f60 0x7f6d68069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.187+0000 7f6d7ea00640 1 -- 172.21.15.67:0/3875908450 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6d78084f60 con 0x7f6d88071840 2024-04-17T10:25:01.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.187+0000 7f6d86000640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d68066f60 0x7f6d68069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.187+0000 7f6d86000640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d68066f60 0x7f6d68069420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d88140490 tx=0x7f6d74009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.191+0000 7f6d7ea00640 1 -- 172.21.15.67:0/3875908450 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6d8813ee30 con 0x7f6d88071840 2024-04-17T10:25:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.359+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d880b41a0 con 0x7f6d88071840 2024-04-17T10:25:01.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.359+0000 7f6d7ea00640 1 -- 172.21.15.67:0/3875908450 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+18479 (secure 0 0 0) 0x7f6d78017020 con 0x7f6d88071840 2024-04-17T10:25:01.360 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:01.361 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":24,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:00.039187+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d68066f60 msgr2=0x7f6d68069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d68066f60 0x7f6d68069420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d88140490 tx=0x7f6d74009190 comp rx=0 tx=0).stop 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 msgr2=0x7f6d88158be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d88158be0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800e810 tx=0x7f6d7800ecd0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 shutdown_connections 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d68066f60 0x7f6d68069420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d88144aa0 0x7f6d8813ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d881446c0 0x7f6d880affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 --2- 172.21.15.67:0/3875908450 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d88071840 0x7f6d88158be0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 >> 172.21.15.67:0/3875908450 conn(0x7f6d880b34d0 msgr2=0x7f6d880b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 shutdown_connections 2024-04-17T10:25:01.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.363+0000 7f6d8dc00640 1 -- 172.21.15.67:0/3875908450 wait complete. 2024-04-17T10:25:01.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:01.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd29200640 1 Processor -- start 2024-04-17T10:25:01.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd29200640 1 -- start start 2024-04-17T10:25:01.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd29200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcd240b56b0 0x7fcd24070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd29200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd24071200 0x7fcd2406c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd29200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd24161f70 con 0x7fcd2415dd40 2024-04-17T10:25:01.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd29200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd240b5b10 con 0x7fcd24071200 2024-04-17T10:25:01.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd29200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd24071650 con 0x7fcd240b56b0 2024-04-17T10:25:01.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd22800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd24071200 0x7fcd2406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd22800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd24071200 0x7fcd2406c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43018/0 (socket says 172.21.15.67:43018) 2024-04-17T10:25:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd22800640 1 -- 172.21.15.67:0/912938501 learned_addr learned my addr 172.21.15.67:0/912938501 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd1be00640 1 --2- 172.21.15.67:0/912938501 >> v2:172.21.15.86:3300/0 conn(0x7fcd240b56b0 0x7fcd24070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd22800640 1 -- 172.21.15.67:0/912938501 >> v2:172.21.15.86:3300/0 conn(0x7fcd240b56b0 msgr2=0x7fcd24070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd22800640 1 --2- 172.21.15.67:0/912938501 >> v2:172.21.15.86:3300/0 conn(0x7fcd240b56b0 0x7fcd24070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd22800640 1 -- 172.21.15.67:0/912938501 >> v1:172.21.15.67:6789/0 conn(0x7fcd2415dd40 legacy=0x7fcd24161760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.523+0000 7fcd22800640 1 -- 172.21.15.67:0/912938501 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd24162250 con 0x7fcd24071200 2024-04-17T10:25:01.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd22800640 1 --2- 172.21.15.67:0/912938501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24071200 0x7fcd2406c510 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcd240b44b0 tx=0x7fcd1c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9bdebb1e0b7465 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd21400640 1 -- 172.21.15.67:0/912938501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0a5620 con 0x7fcd24071200 2024-04-17T10:25:01.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd21400640 1 -- 172.21.15.67:0/912938501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c0a57c0 con 0x7fcd24071200 2024-04-17T10:25:01.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd21400640 1 -- 172.21.15.67:0/912938501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0a5b90 con 0x7fcd24071200 2024-04-17T10:25:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 -- 172.21.15.67:0/912938501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24071200 msgr2=0x7fcd2406c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 --2- 172.21.15.67:0/912938501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24071200 0x7fcd2406c510 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcd240b44b0 tx=0x7fcd1c01c960 comp rx=0 tx=0).stop 2024-04-17T10:25:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 -- 172.21.15.67:0/912938501 shutdown_connections 2024-04-17T10:25:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 --2- 172.21.15.67:0/912938501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd24071200 0x7fcd2406c510 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 --2- 172.21.15.67:0/912938501 >> v2:172.21.15.86:3300/0 conn(0x7fcd240b56b0 0x7fcd24070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 -- 172.21.15.67:0/912938501 >> 172.21.15.67:0/912938501 conn(0x7fcd240b33f0 msgr2=0x7fcd240b37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:01.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 -- 172.21.15.67:0/912938501 shutdown_connections 2024-04-17T10:25:01.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.527+0000 7fcd29200640 1 -- 172.21.15.67:0/912938501 wait complete. 2024-04-17T10:25:01.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 Processor -- start 2024-04-17T10:25:01.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 -- start start 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd24071200 0x7fcd240cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd240b56b0 0x7fcd241561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd2415dd40 0x7fcd240ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2406d670 con 0x7fcd2415dd40 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2406d4f0 con 0x7fcd24071200 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2406d7f0 con 0x7fcd240b56b0 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd2415dd40 0x7fcd240ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd2415dd40 0x7fcd240ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43022/0 (socket says 172.21.15.67:43022) 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 -- 172.21.15.67:0/1366449090 learned_addr learned my addr 172.21.15.67:0/1366449090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd1be00640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd240b56b0 0x7fcd241561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd21e00640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd24071200 0x7fcd240cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 -- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd240b56b0 msgr2=0x7fcd241561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd240b56b0 0x7fcd241561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 -- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd24071200 msgr2=0x7fcd240cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd24071200 0x7fcd240cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 -- 172.21.15.67:0/1366449090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd24162880 con 0x7fcd2415dd40 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd22800640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd2415dd40 0x7fcd240ccfe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c01a590 tx=0x7fcd1c019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd1a000640 1 -- 172.21.15.67:0/1366449090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0bc020 con 0x7fcd2415dd40 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd1a000640 1 -- 172.21.15.67:0/1366449090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c0b88a0 con 0x7fcd2415dd40 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd1a000640 1 -- 172.21.15.67:0/1366449090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c0a5620 con 0x7fcd2415dd40 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd240cac10 con 0x7fcd2415dd40 2024-04-17T10:25:01.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2415b350 con 0x7fcd2415dd40 2024-04-17T10:25:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.531+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd24070ab0 con 0x7fcd2415dd40 2024-04-17T10:25:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.535+0000 7fcd1a000640 1 -- 172.21.15.67:0/1366449090 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcd1c0bf020 con 0x7fcd2415dd40 2024-04-17T10:25:01.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.535+0000 7fcd1a000640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd00066f60 0x7fcd00069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.535+0000 7fcd1a000640 1 -- 172.21.15.67:0/1366449090 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcd1c120440 con 0x7fcd2415dd40 2024-04-17T10:25:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.535+0000 7fcd21e00640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd00066f60 0x7fcd00069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.535+0000 7fcd21e00640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd00066f60 0x7fcd00069420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd24066ef0 tx=0x7fcd14009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.539+0000 7fcd1a000640 1 -- 172.21.15.67:0/1366449090 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcd1c0c6030 con 0x7fcd2415dd40 2024-04-17T10:25:01.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.703+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd24074e30 con 0x7fcd2415dd40 2024-04-17T10:25:01.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd1a000640 1 -- 172.21.15.67:0/1366449090 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7fcd24162880 con 0x7fcd2415dd40 2024-04-17T10:25:01.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:01.706 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd00066f60 msgr2=0x7fcd00069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd00066f60 0x7fcd00069420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcd24066ef0 tx=0x7fcd14009040 comp rx=0 tx=0).stop 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd2415dd40 msgr2=0x7fcd240ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd2415dd40 0x7fcd240ccfe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c01a590 tx=0x7fcd1c019700 comp rx=0 tx=0).stop 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 shutdown_connections 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd00066f60 0x7fcd00069420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd2415dd40 0x7fcd240ccfe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd240b56b0 0x7fcd241561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 --2- 172.21.15.67:0/1366449090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd24071200 0x7fcd240cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 >> 172.21.15.67:0/1366449090 conn(0x7fcd240b33f0 msgr2=0x7fcd2415e120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 shutdown_connections 2024-04-17T10:25:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.707+0000 7fcd29200640 1 -- 172.21.15.67:0/1366449090 wait complete. 2024-04-17T10:25:01.742 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:25:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f79c00640 1 Processor -- start 2024-04-17T10:25:01.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f79c00640 1 -- start start 2024-04-17T10:25:01.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f79c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f7415cc00 0x7f9f7415f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f79c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7415be60 con 0x7f9f740ce1b0 2024-04-17T10:25:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f79c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f74160230 con 0x7f9f7415fa00 2024-04-17T10:25:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f79c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7415c120 con 0x7f9f7415cc00 2024-04-17T10:25:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f6a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f7415cc00 0x7f9f7415f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f6a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f7415cc00 0x7f9f7415f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43028/0 (socket says 172.21.15.67:43028) 2024-04-17T10:25:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.883+0000 7f9f6a000640 1 -- 172.21.15.67:0/1756010771 learned_addr learned my addr 172.21.15.67:0/1756010771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f6a000640 1 -- 172.21.15.67:0/1756010771 >> v1:172.21.15.86:6789/0 conn(0x7f9f7415fa00 legacy=0x7f9f740c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:01.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f6a000640 1 -- 172.21.15.67:0/1756010771 >> v1:172.21.15.67:6789/0 conn(0x7f9f740ce1b0 legacy=0x7f9f740ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f6a000640 1 -- 172.21.15.67:0/1756010771 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7415c400 con 0x7f9f7415cc00 2024-04-17T10:25:01.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f6a000640 1 --2- 172.21.15.67:0/1756010771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415cc00 0x7f9f7415f4c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009870 tx=0x7f9f5c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d27b06564bd180e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f72000640 1 -- 172.21.15.67:0/1756010771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c0b5070 con 0x7f9f7415cc00 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f72000640 1 -- 172.21.15.67:0/1756010771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c0a8a10 con 0x7f9f7415cc00 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f72000640 1 -- 172.21.15.67:0/1756010771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c0a8de0 con 0x7f9f7415cc00 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f79c00640 1 -- 172.21.15.67:0/1756010771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415cc00 msgr2=0x7f9f7415f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f79c00640 1 --2- 172.21.15.67:0/1756010771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415cc00 0x7f9f7415f4c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009870 tx=0x7f9f5c01ed10 comp rx=0 tx=0).stop 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f79c00640 1 -- 172.21.15.67:0/1756010771 shutdown_connections 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f79c00640 1 --2- 172.21.15.67:0/1756010771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415cc00 0x7f9f7415f4c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f79c00640 1 -- 172.21.15.67:0/1756010771 >> 172.21.15.67:0/1756010771 conn(0x7f9f740b33f0 msgr2=0x7f9f740b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f79c00640 1 -- 172.21.15.67:0/1756010771 shutdown_connections 2024-04-17T10:25:01.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.887+0000 7f9f79c00640 1 -- 172.21.15.67:0/1756010771 wait complete. 2024-04-17T10:25:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 Processor -- start 2024-04-17T10:25:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 -- start start 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f740ce1b0 0x7f9f741f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f7415cc00 0x7f9f74155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415fa00 0x7f9f74155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f741598b0 con 0x7f9f7415fa00 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f74159730 con 0x7f9f740ce1b0 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7415fe30 con 0x7f9f7415cc00 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415fa00 0x7f9f74155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415fa00 0x7f9f74155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43030/0 (socket says 172.21.15.67:43030) 2024-04-17T10:25:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 -- 172.21.15.67:0/2335914961 learned_addr learned my addr 172.21.15.67:0/2335914961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 -- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f7415cc00 msgr2=0x7f9f74155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f7415cc00 0x7f9f74155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 -- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f740ce1b0 msgr2=0x7f9f741f3ab0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f740ce1b0 0x7f9f741f3ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 -- 172.21.15.67:0/2335914961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f74143ae0 con 0x7f9f7415fa00 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f73400640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415fa00 0x7f9f74155870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c410 tx=0x7f9f6c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f6b400640 1 -- 172.21.15.67:0/2335914961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c017070 con 0x7f9f7415fa00 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f6b400640 1 -- 172.21.15.67:0/2335914961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c003600 con 0x7f9f7415fa00 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f6b400640 1 -- 172.21.15.67:0/2335914961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c003a30 con 0x7f9f7415fa00 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f74154850 con 0x7f9f7415fa00 2024-04-17T10:25:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.891+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f74154e10 con 0x7f9f7415fa00 2024-04-17T10:25:01.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.895+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f741f4050 con 0x7f9f7415fa00 2024-04-17T10:25:01.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.895+0000 7f9f6b400640 1 -- 172.21.15.67:0/2335914961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9f6c01a070 con 0x7f9f7415fa00 2024-04-17T10:25:01.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.895+0000 7f9f6b400640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f50066f60 0x7f9f50069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:01.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.895+0000 7f9f6b400640 1 -- 172.21.15.67:0/2335914961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9f6c0847b0 con 0x7f9f7415fa00 2024-04-17T10:25:01.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.895+0000 7f9f72a00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f50066f60 0x7f9f50069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:01.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.895+0000 7f9f72a00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f50066f60 0x7f9f50069420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f64007fd0 tx=0x7f9f64007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:01.899+0000 7f9f6b400640 1 -- 172.21.15.67:0/2335914961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9f6c0553e0 con 0x7f9f7415fa00 2024-04-17T10:25:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.099+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f74070ab0 con 0x7f9f7415fa00 2024-04-17T10:25:02.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.099+0000 7f9f6b400640 1 -- 172.21.15.67:0/2335914961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f9f6c054ce0 con 0x7f9f7415fa00 2024-04-17T10:25:02.100 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:02.100 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:00.039461+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-04-17T10:25:02.101 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 7 2024-04-17T10:25:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f50066f60 msgr2=0x7f9f50069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f50066f60 0x7f9f50069420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9f64007fd0 tx=0x7f9f64007d60 comp rx=0 tx=0).stop 2024-04-17T10:25:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415fa00 msgr2=0x7f9f74155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415fa00 0x7f9f74155870 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c00c410 tx=0x7f9f6c00c8d0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 shutdown_connections 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f50066f60 0x7f9f50069420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f7415fa00 0x7f9f74155870 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f7415cc00 0x7f9f74155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 --2- 172.21.15.67:0/2335914961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f740ce1b0 0x7f9f741f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 >> 172.21.15.67:0/2335914961 conn(0x7f9f740b33f0 msgr2=0x7f9f740b19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 shutdown_connections 2024-04-17T10:25:02.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.103+0000 7f9f79c00640 1 -- 172.21.15.67:0/2335914961 wait complete. 2024-04-17T10:25:02.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:02.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d7200640 1 Processor -- start 2024-04-17T10:25:02.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d7200640 1 -- start start 2024-04-17T10:25:02.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d7200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff7d80d07d0 0x7ff7d81686d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d80b79f0 con 0x7ff7d815dd70 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d7200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d80d23b0 con 0x7ff7d815d990 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d7200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d80d2fc0 con 0x7ff7d80d07d0 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7cd400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7d815dd70 0x7ff7d80d00c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39162/0 (socket says 172.21.15.67:39162) 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7cd400640 1 -- 172.21.15.67:0/189752854 learned_addr learned my addr 172.21.15.67:0/189752854 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d6800640 1 --2- 172.21.15.67:0/189752854 >> v2:172.21.15.86:3300/0 conn(0x7ff7d80d07d0 0x7ff7d81686d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888354584 0 0) 0x7ff7d80b79f0 con 0x7ff7d815dd70 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc003530 con 0x7ff7d815dd70 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295838667 0 0) 0x7ff7d80d23b0 con 0x7ff7d815d990 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d80b79f0 con 0x7ff7d815d990 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3637580392 0 0) 0x7ff7bc003530 con 0x7ff7d815dd70 2024-04-17T10:25:02.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 >> v2:172.21.15.86:3300/0 conn(0x7ff7d80d07d0 msgr2=0x7ff7d81686d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 --2- 172.21.15.67:0/189752854 >> v2:172.21.15.86:3300/0 conn(0x7ff7d80d07d0 0x7ff7d81686d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 >> v1:172.21.15.86:6789/0 conn(0x7ff7d815d990 legacy=0x7ff7d80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d80d2690 con 0x7ff7d815dd70 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7ff7c0001680 con 0x7ff7d815dd70 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 >> v1:172.21.15.67:6789/0 conn(0x7ff7d815dd70 legacy=0x7ff7d80d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7bc003b50 0x7ff7bc003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7bc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7bc0051d0 0x7ff7bc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc00c0d0 con 0x7ff7bc004810 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc00c280 con 0x7ff7bc0051d0 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc00c480 con 0x7ff7bc003b50 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.263+0000 7ff7cd400640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7bc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d5e00640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7bc003b50 0x7ff7bc003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d6800640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7bc0051d0 0x7ff7bc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7cd400640 1 -- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7bc003b50 msgr2=0x7ff7bc003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:02.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7cd400640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7bc003b50 0x7ff7bc003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7cd400640 1 -- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7bc0051d0 msgr2=0x7ff7bc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:02.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7cd400640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7bc0051d0 0x7ff7bc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7cd400640 1 -- 172.21.15.67:0/189752854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc003530 con 0x7ff7bc004810 2024-04-17T10:25:02.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d6800640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7bc0051d0 0x7ff7bc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:02.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d6800640 1 -- 172.21.15.67:0/189752854 reap_dead start 2024-04-17T10:25:02.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7cd400640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7bc004c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000c190 tx=0x7ff7c001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e40f4855be95c8d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0007cb0 con 0x7ff7bc004810 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00a9020 con 0x7ff7bc004810 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d5400640 1 -- 172.21.15.67:0/189752854 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0005ae0 con 0x7ff7bc004810 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d7200640 1 -- 172.21.15.67:0/189752854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 msgr2=0x7ff7bc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d7200640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7bc004c90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff7c000c190 tx=0x7ff7c001cc00 comp rx=0 tx=0).stop 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d7200640 1 -- 172.21.15.67:0/189752854 shutdown_connections 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d7200640 1 --2- 172.21.15.67:0/189752854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7bc004c90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.267+0000 7ff7d7200640 1 -- 172.21.15.67:0/189752854 >> 172.21.15.67:0/189752854 conn(0x7ff7d80b53c0 msgr2=0x7ff7d80b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 -- 172.21.15.67:0/189752854 shutdown_connections 2024-04-17T10:25:02.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 -- 172.21.15.67:0/189752854 wait complete. 2024-04-17T10:25:02.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 Processor -- start 2024-04-17T10:25:02.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 -- start start 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7d8161880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d80c9660 0x7ff7d80c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d814cb90 0x7ff7d814d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80d35b0 con 0x7ff7bc004810 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80d3430 con 0x7ff7d814cb90 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d80d00c0 con 0x7ff7d80c9660 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7d8161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7d8161880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43050/0 (socket says 172.21.15.67:43050) 2024-04-17T10:25:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 -- 172.21.15.67:0/3760502017 learned_addr learned my addr 172.21.15.67:0/3760502017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d80c9660 0x7ff7d80c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 -- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d80c9660 msgr2=0x7ff7d80c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d6800640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d814cb90 0x7ff7d814d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d80c9660 0x7ff7d80c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 -- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d814cb90 msgr2=0x7ff7d814d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d814cb90 0x7ff7d814d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 -- 172.21.15.67:0/3760502017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d80c7080 con 0x7ff7bc004810 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d80c9660 0x7ff7d80c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d6800640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d814cb90 0x7ff7d814d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d5e00640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7d8161880 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800fb40 tx=0x7ff7c800d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7cea00640 1 -- 172.21.15.67:0/3760502017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8003b90 con 0x7ff7bc004810 2024-04-17T10:25:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7cea00640 1 -- 172.21.15.67:0/3760502017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c8018070 con 0x7ff7bc004810 2024-04-17T10:25:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7cea00640 1 -- 172.21.15.67:0/3760502017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c800d920 con 0x7ff7bc004810 2024-04-17T10:25:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.271+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d80c7360 con 0x7ff7bc004810 2024-04-17T10:25:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.275+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d80c7920 con 0x7ff7bc004810 2024-04-17T10:25:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.275+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d8074dc0 con 0x7ff7bc004810 2024-04-17T10:25:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.275+0000 7ff7cea00640 1 -- 172.21.15.67:0/3760502017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff7c800dac0 con 0x7ff7bc004810 2024-04-17T10:25:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.275+0000 7ff7cea00640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7b4066f90 0x7ff7b4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.275+0000 7ff7cea00640 1 -- 172.21.15.67:0/3760502017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7ff7c8085680 con 0x7ff7bc004810 2024-04-17T10:25:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.275+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7b4066f90 0x7ff7b4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.275+0000 7ff7cd400640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7b4066f90 0x7ff7b4069450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00aa000 tx=0x7ff7c00019d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:02.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.279+0000 7ff7cea00640 1 -- 172.21.15.67:0/3760502017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff7c8056330 con 0x7ff7bc004810 2024-04-17T10:25:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.443+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d80b4030 con 0x7ff7bc004810 2024-04-17T10:25:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.443+0000 7ff7cea00640 1 -- 172.21.15.67:0/3760502017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7ff7c8082840 con 0x7ff7bc004810 2024-04-17T10:25:02.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:02.444 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7b4066f90 msgr2=0x7ff7b4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7b4066f90 0x7ff7b4069450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00aa000 tx=0x7ff7c00019d0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 msgr2=0x7ff7d8161880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7d8161880 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7c800fb40 tx=0x7ff7c800d150 comp rx=0 tx=0).stop 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 shutdown_connections 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7b4066f90 0x7ff7b4069450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d814cb90 0x7ff7d814d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d80c9660 0x7ff7d80c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 --2- 172.21.15.67:0/3760502017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7bc004810 0x7ff7d8161880 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 >> 172.21.15.67:0/3760502017 conn(0x7ff7d80b53c0 msgr2=0x7ff7d80cc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 shutdown_connections 2024-04-17T10:25:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:02.447+0000 7ff7d7200640 1 -- 172.21.15.67:0/3760502017 wait complete. 2024-04-17T10:25:02.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:24:54.978691+0000', 'modified': '2024-04-17T10:25:00.039461+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/1394098337', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 1394098337}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 1394098337}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-04-17T10:25:02.869 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:25:02.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T10:25:03.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.011+0000 7f3952e00640 1 Processor -- start 2024-04-17T10:25:03.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.011+0000 7f3952e00640 1 -- start start 2024-04-17T10:25:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.011+0000 7f3952e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 0x7f394c168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.011+0000 7f3952e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394c0b5980 con 0x7f394c15dd10 2024-04-17T10:25:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.011+0000 7f3952e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f394c0b0cc0 con 0x7f394c0b07b0 2024-04-17T10:25:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.011+0000 7f3952e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c0b1280 con 0x7f394c0714d0 2024-04-17T10:25:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f394c15dd10 0x7f394c0b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39164/0 (socket says 172.21.15.67:39164) 2024-04-17T10:25:03.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394be00640 1 -- 172.21.15.67:0/902473188 learned_addr learned my addr 172.21.15.67:0/902473188 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708487147 0 0) 0x7f394c0b5980 con 0x7f394c15dd10 2024-04-17T10:25:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3930003530 con 0x7f394c15dd10 2024-04-17T10:25:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f3950e00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 0x7f394c168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3808175790 0 0) 0x7f3930003530 con 0x7f394c15dd10 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 msgr2=0x7f394c168c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 0x7f394c168c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 >> v1:172.21.15.86:6789/0 conn(0x7f394c0b07b0 legacy=0x7f394c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c0b0fa0 con 0x7f394c15dd10 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f393c001680 con 0x7f394c15dd10 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 >> v1:172.21.15.67:6789/0 conn(0x7f394c15dd10 legacy=0x7f394c0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39300039f0 0x7f3930003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3930004810 0x7f3930004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 0x7f3930005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c0b1280 con 0x7f3930004810 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393000bb90 con 0x7f39300039f0 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393000bd90 con 0x7f394c0714d0 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394b400640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3930004810 0x7f3930004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394be00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39300039f0 0x7f3930003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f3950e00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 0x7f3930005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394b400640 1 -- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 msgr2=0x7f3930005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394b400640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 0x7f3930005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394b400640 1 -- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39300039f0 msgr2=0x7f3930003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394b400640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f39300039f0 0x7f3930003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f394b400640 1 -- 172.21.15.67:0/902473188 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3930003530 con 0x7f3930004810 2024-04-17T10:25:03.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.015+0000 7f3950e00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0714d0 0x7f3930005280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f394b400640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3930004810 0x7f3930004c90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3938001750 tx=0x7f393801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71461b941e627137 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39380013a0 con 0x7f3930004810 2024-04-17T10:25:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39380052f0 con 0x7f3930004810 2024-04-17T10:25:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f394aa00640 1 -- 172.21.15.67:0/902473188 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938003840 con 0x7f3930004810 2024-04-17T10:25:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 -- 172.21.15.67:0/902473188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3930004810 msgr2=0x7f3930004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 --2- 172.21.15.67:0/902473188 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3930004810 0x7f3930004c90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3938001750 tx=0x7f393801c3b0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3950e00640 1 -- 172.21.15.67:0/902473188 reap_dead start 2024-04-17T10:25:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 -- 172.21.15.67:0/902473188 shutdown_connections 2024-04-17T10:25:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 -- 172.21.15.67:0/902473188 >> 172.21.15.67:0/902473188 conn(0x7f394c0b34d0 msgr2=0x7f394c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 -- 172.21.15.67:0/902473188 shutdown_connections 2024-04-17T10:25:03.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 -- 172.21.15.67:0/902473188 wait complete. 2024-04-17T10:25:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 Processor -- start 2024-04-17T10:25:03.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.019+0000 7f3952e00640 1 -- start start 2024-04-17T10:25:03.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3930004810 0x7f394c158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0cd660 0x7f394c15a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c15ac10 0x7f394c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c1695e0 con 0x7f394c15ac10 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c169460 con 0x7f3930004810 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c169760 con 0x7f394c0cd660 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c15ac10 0x7f394c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c15ac10 0x7f394c15b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43066/0 (socket says 172.21.15.67:43066) 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 -- 172.21.15.67:0/3759324545 learned_addr learned my addr 172.21.15.67:0/3759324545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f394be00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3930004810 0x7f394c158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f394b400640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0cd660 0x7f394c15a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 -- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0cd660 msgr2=0x7f394c15a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0cd660 0x7f394c15a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 -- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3930004810 msgr2=0x7f394c158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3930004810 0x7f394c158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 -- 172.21.15.67:0/3759324545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394c15b5b0 con 0x7f394c15ac10 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f394b400640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0cd660 0x7f394c15a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3950e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c15ac10 0x7f394c15b070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3944009f50 tx=0x7f39440115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3943e00640 1 -- 172.21.15.67:0/3759324545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394401a020 con 0x7f394c15ac10 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3943e00640 1 -- 172.21.15.67:0/3759324545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3944016470 con 0x7f394c15ac10 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f394c0cb590 con 0x7f394c15ac10 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3943e00640 1 -- 172.21.15.67:0/3759324545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944016840 con 0x7f394c15ac10 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394c15b880 con 0x7f394c15ac10 2024-04-17T10:25:03.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.023+0000 7f3943e00640 1 -- 172.21.15.67:0/3759324545 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f394401d020 con 0x7f394c15ac10 2024-04-17T10:25:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3943e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f392c066f60 0x7f392c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3943e00640 1 -- 172.21.15.67:0/3759324545 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3944085fe0 con 0x7f394c15ac10 2024-04-17T10:25:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f394be00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f392c066f60 0x7f392c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 Processor -- start 2024-04-17T10:25:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 -- start start 2024-04-17T10:25:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f394be00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f392c066f60 0x7f392c069420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f393c001c10 tx=0x7f393c001ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bc240 0x55893f8719f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f871f30 0x55893f872310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893f7595f0 con 0x55893f871f30 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893f6c0da0 con 0x55893f8bc240 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55893f649150 con 0x55893f8bb920 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f871f30 0x55893f872310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f871f30 0x55893f872310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43078/0 (socket says 172.21.15.67:43078) 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 -- 172.21.15.67:0/2291036955 learned_addr learned my addr 172.21.15.67:0/2291036955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f394b400640 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bc240 0x55893f8719f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f394be00640 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 -- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 msgr2=0x55893f8bbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8bbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 -- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bc240 msgr2=0x55893f8719f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bc240 0x55893f8719f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 -- 172.21.15.67:0/2291036955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55893f662ca0 con 0x55893f871f30 2024-04-17T10:25:03.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3950e00640 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f871f30 0x55893f872310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3944008010 tx=0x7f3944056d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa01a1971a5fd55 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3940c00640 1 -- 172.21.15.67:0/2291036955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394405ecb0 con 0x55893f871f30 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3940c00640 1 -- 172.21.15.67:0/2291036955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39440548a0 con 0x55893f871f30 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3940c00640 1 -- 172.21.15.67:0/2291036955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944052c10 con 0x55893f871f30 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 -- 172.21.15.67:0/2291036955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f871f30 msgr2=0x55893f872310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.027+0000 7f3956349000 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f871f30 0x55893f872310 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3944008010 tx=0x7f3944056d20 comp rx=0 tx=0).stop 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- 172.21.15.67:0/2291036955 shutdown_connections 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f871f30 0x55893f872310 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bc240 0x55893f8719f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 --2- 172.21.15.67:0/2291036955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8bbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- 172.21.15.67:0/2291036955 >> 172.21.15.67:0/2291036955 conn(0x55893f8782a0 msgr2=0x55893f8702d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- 172.21.15.67:0/2291036955 shutdown_connections 2024-04-17T10:25:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- 172.21.15.67:0/2291036955 wait complete. 2024-04-17T10:25:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- start start 2024-04-17T10:25:03.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 Processor -- start 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8c3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f875aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893f67b990 con 0x55893f8bc240 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893f6bce40 con 0x55893f871f30 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55893f64c900 con 0x55893f8bb920 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 10 client.0 _ll_register_callbacks cb 0x55893f7741c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f875aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f875aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43090/0 (socket says 172.21.15.67:43090) 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 -- 172.21.15.67:0/2693514848 learned_addr learned my addr 172.21.15.67:0/2693514848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f394b400640 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8c3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f394be00640 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 -- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 msgr2=0x55893f8c3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8c3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 -- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 msgr2=0x55893f8c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 -- 172.21.15.67:0/2693514848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55893f679630 con 0x55893f8bc240 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3950e00640 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f875aa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3944056e50 tx=0x7f3944059070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7f84e5d3e1e02f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3935600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3935600640 1 -- 172.21.15.67:0/2693514848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3944055bf0 con 0x55893f8bc240 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3935600640 1 -- 172.21.15.67:0/2693514848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f394407db20 con 0x55893f8bc240 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39440548a0 con 0x55893f8bc240 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3935600640 1 -- 172.21.15.67:0/2693514848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f394407dd20 con 0x55893f8bc240 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 10 client.4395 fetch_fsmap learned FSMap version 7 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55893f6493a0 con 0x55893f8bc240 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3935600640 1 -- 172.21.15.67:0/2693514848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39440548a0 con 0x55893f8bc240 2024-04-17T10:25:03.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3935600640 1 -- 172.21.15.67:0/2693514848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f39440522f0 con 0x55893f8bc240 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 10 client.4395 fetch_fsmap finished waiting for FSMap version 7 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 10 client.4395 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 20 client.4395 populate_metadata read hostname 'smithi067' 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f877160 0x55893f8ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 4 client.4395 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 --> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55893f65ca50 con 0x55893f877160 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.031+0000 7f394be00640 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f877160 0x55893f8ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3935600640 10 client.4395 ms_handle_connect on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f394be00640 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f877160 0x55893f8ccea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e4f04c152481dd4 server_cookie=da302e3a3998bcb5 in_seq=0 out_seq=0 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3935600640 1 -- 172.21.15.67:0/2693514848 <== mds.0 v2:172.21.15.67:6834/1394098337 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55893f65ca50 con 0x55893f877160 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3935600640 10 client.4395 handle_command_reply: tid=0 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 client.4395 shutdown 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 msgr2=0x55893f875aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f875aa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3944056e50 tx=0x7f3944059070 comp rx=0 tx=0).stop 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 shutdown_connections 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f877160 0x55893f8ccea0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f875aa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8c3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 --2- 172.21.15.67:0/2693514848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3935600640 0 client.4395 ms_handle_reset on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 >> 172.21.15.67:0/2693514848 conn(0x55893f8782a0 msgr2=0x55893f8709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 shutdown_connections 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 1 -- 172.21.15.67:0/2693514848 wait complete. 2024-04-17T10:25:03.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.035+0000 7f3956349000 20 client.4395 trim_cache size 0 max 16384 2024-04-17T10:25:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 Processor -- start 2024-04-17T10:25:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 -- start start 2024-04-17T10:25:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f871f30 0x55893f8752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bb920 0x55893f875800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f8c4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c203e20 con 0x55893f8bc240 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394c203220 con 0x55893f8bb920 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394c203820 con 0x55893f871f30 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f8c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f8c4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43092/0 (socket says 172.21.15.67:43092) 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 -- 172.21.15.67:0/4255607020 learned_addr learned my addr 172.21.15.67:0/4255607020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f394be00640 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f871f30 0x55893f8752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f394b400640 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bb920 0x55893f875800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 -- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f871f30 msgr2=0x55893f8752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f871f30 0x55893f8752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 -- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bb920 msgr2=0x55893f875800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bb920 0x55893f875800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 -- 172.21.15.67:0/4255607020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55893f8bc070 con 0x55893f8bc240 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3950e00640 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f8c4950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3944000f90 tx=0x7f3944018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47babafc49aa5104 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3937e00640 1 -- 172.21.15.67:0/4255607020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394407db20 con 0x55893f8bc240 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3937e00640 1 -- 172.21.15.67:0/4255607020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39440548a0 con 0x55893f8bc240 2024-04-17T10:25:03.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.051+0000 7f3937e00640 1 -- 172.21.15.67:0/4255607020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39440559f0 con 0x55893f8bc240 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/4255607020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 msgr2=0x55893f8c4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f8c4950 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3944000f90 tx=0x7f3944018000 comp rx=0 tx=0).stop 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/4255607020 shutdown_connections 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f8c4950 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f8bb920 0x55893f875800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- 172.21.15.67:0/4255607020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f871f30 0x55893f8752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/4255607020 >> 172.21.15.67:0/4255607020 conn(0x55893f8782a0 msgr2=0x55893f874bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/4255607020 shutdown_connections 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/4255607020 wait complete. 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- start start 2024-04-17T10:25:03.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 Processor -- start 2024-04-17T10:25:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8ffdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f900300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893f6c53d0 con 0x55893f8bc240 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893f77cd50 con 0x55893f871f30 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55893f65b980 con 0x55893f8bb920 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 10 client.0 _ll_register_callbacks cb 0x7f3944011ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f900300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f900300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43094/0 (socket says 172.21.15.67:43094) 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 -- 172.21.15.67:0/3039293546 learned_addr learned my addr 172.21.15.67:0/3039293546 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f394b400640 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8ffdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f394be00640 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 -- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 msgr2=0x55893f8ffdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8ffdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 -- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 msgr2=0x55893f8c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 -- 172.21.15.67:0/3039293546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55893f742600 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3950e00640 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f900300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3944056e50 tx=0x7f3944052840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c712b0541cc612 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3935600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3935600640 1 -- 172.21.15.67:0/3039293546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394405a160 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3935600640 1 -- 172.21.15.67:0/3039293546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f394405a300 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39440548a0 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3935600640 1 -- 172.21.15.67:0/3039293546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3944056370 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 10 client.4401 fetch_fsmap learned FSMap version 7 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55893f6559b0 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3935600640 1 -- 172.21.15.67:0/3039293546 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39440548a0 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3935600640 1 -- 172.21.15.67:0/3039293546 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 7) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f3944055460 con 0x55893f8bc240 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 10 client.4401 fetch_fsmap finished waiting for FSMap version 7 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 10 client.4401 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:25:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 20 client.4401 populate_metadata read hostname 'smithi067' 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f900840 0x55893f91c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 4 client.4401 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.055+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 --> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55893f6ce2e0 con 0x55893f900840 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f394be00640 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f900840 0x55893f91c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f394be00640 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f900840 0x55893f91c270 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fb4594913d3e4ed server_cookie=562b98063766a45e in_seq=0 out_seq=0 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3935600640 10 client.4401 ms_handle_connect on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3935600640 1 -- 172.21.15.67:0/3039293546 <== mds.0 v2:172.21.15.67:6834/1394098337 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55893f6ce2e0 con 0x55893f900840 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3935600640 10 client.4401 handle_command_reply: tid=0 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 client.4401 shutdown 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 msgr2=0x55893f900300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f900300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3944056e50 tx=0x7f3944052840 comp rx=0 tx=0).stop 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 shutdown_connections 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x55893f900840 0x55893f91c270 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55893f8bc240 0x55893f900300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55893f8bb920 0x55893f8ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 --2- 172.21.15.67:0/3039293546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55893f871f30 0x55893f8c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3935600640 0 client.4401 ms_handle_reset on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 >> 172.21.15.67:0/3039293546 conn(0x55893f8782a0 msgr2=0x55893f8752c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 shutdown_connections 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 1 -- 172.21.15.67:0/3039293546 wait complete. 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3956349000 20 client.4401 trim_cache size 0 max 16384 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:25:03.062 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4275, 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 7.4825959040000001, 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 7, 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 25, 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 24, 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 14.500737405000001 2024-04-17T10:25:03.063 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f392c066f60 msgr2=0x7f392c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f392c066f60 0x7f392c069420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f393c001c10 tx=0x7f393c001ba0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c15ac10 msgr2=0x7f394c15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c15ac10 0x7f394c15b070 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3944009f50 tx=0x7f39440115a0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 shutdown_connections 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f392c066f60 0x7f392c069420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f394c15ac10 0x7f394c15b070 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f394c0cd660 0x7f394c15a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 --2- 172.21.15.67:0/3759324545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3930004810 0x7f394c158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 >> 172.21.15.67:0/3759324545 conn(0x7f394c0b34d0 msgr2=0x7f394c0b4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 shutdown_connections 2024-04-17T10:25:03.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.059+0000 7f3952e00640 1 -- 172.21.15.67:0/3759324545 wait complete. 2024-04-17T10:25:03.077 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T10:25:03.077 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:03.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:25:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.219+0000 7fbb49000640 1 Processor -- start 2024-04-17T10:25:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.219+0000 7fbb49000640 1 -- start start 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb49000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb440ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb49000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb4406eac0 0x7fbb440ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb49000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4415ae40 con 0x7fbb44143ae0 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb49000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb440cf430 con 0x7fbb4406eac0 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb49000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb44071440 con 0x7fbb4406e6e0 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb3be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb4406eac0 0x7fbb440ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb41e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb440ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb3be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb4406eac0 0x7fbb440ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:49044/0 (socket says 172.21.15.67:49044) 2024-04-17T10:25:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb3be00640 1 -- 172.21.15.67:0/81636943 learned_addr learned my addr 172.21.15.67:0/81636943 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb41e00640 1 -- 172.21.15.67:0/81636943 >> v2:172.21.15.86:3300/0 conn(0x7fbb4406eac0 msgr2=0x7fbb440ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb41e00640 1 --2- 172.21.15.67:0/81636943 >> v2:172.21.15.86:3300/0 conn(0x7fbb4406eac0 0x7fbb440ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb41e00640 1 -- 172.21.15.67:0/81636943 >> v1:172.21.15.86:6789/0 conn(0x7fbb44143ae0 legacy=0x7fbb44143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.223+0000 7fbb41e00640 1 -- 172.21.15.67:0/81636943 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb44071720 con 0x7fbb4406e6e0 2024-04-17T10:25:03.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb41e00640 1 --2- 172.21.15.67:0/81636943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb440ce9b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb340097c0 tx=0x7fbb3401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0c28fe107e06c21 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb41400640 1 -- 172.21.15.67:0/81636943 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb340b1070 con 0x7fbb4406e6e0 2024-04-17T10:25:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb41400640 1 -- 172.21.15.67:0/81636943 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb340a8050 con 0x7fbb4406e6e0 2024-04-17T10:25:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb41400640 1 -- 172.21.15.67:0/81636943 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb340ac860 con 0x7fbb4406e6e0 2024-04-17T10:25:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- 172.21.15.67:0/81636943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 msgr2=0x7fbb440ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 --2- 172.21.15.67:0/81636943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb440ce9b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb340097c0 tx=0x7fbb3401c3e0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- 172.21.15.67:0/81636943 shutdown_connections 2024-04-17T10:25:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 --2- 172.21.15.67:0/81636943 >> v2:172.21.15.86:3300/0 conn(0x7fbb4406eac0 0x7fbb440ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 --2- 172.21.15.67:0/81636943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb440ce9b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- 172.21.15.67:0/81636943 >> 172.21.15.67:0/81636943 conn(0x7fbb4406ac90 msgr2=0x7fbb4406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- 172.21.15.67:0/81636943 shutdown_connections 2024-04-17T10:25:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- 172.21.15.67:0/81636943 wait complete. 2024-04-17T10:25:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 Processor -- start 2024-04-17T10:25:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- start start 2024-04-17T10:25:03.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb441599a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb4406eac0 0x7fbb440ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb44143ae0 0x7fbb440ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440b3a40 con 0x7fbb4406eac0 2024-04-17T10:25:03.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440b38c0 con 0x7fbb44143ae0 2024-04-17T10:25:03.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.227+0000 7fbb49000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb440b3bc0 con 0x7fbb4406e6e0 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb4406eac0 0x7fbb440ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb41e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb441599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb4406eac0 0x7fbb440ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43106/0 (socket says 172.21.15.67:43106) 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 -- 172.21.15.67:0/1804972801 learned_addr learned my addr 172.21.15.67:0/1804972801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb42800640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb44143ae0 0x7fbb440ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 -- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 msgr2=0x7fbb441599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb441599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 -- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb44143ae0 msgr2=0x7fbb440ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb44143ae0 0x7fbb440ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 -- 172.21.15.67:0/1804972801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4413abb0 con 0x7fbb4406eac0 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb41e00640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb441599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3be00640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb4406eac0 0x7fbb440ac8c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00e890 tx=0x7fbb2c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3a000640 1 -- 172.21.15.67:0/1804972801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c017070 con 0x7fbb4406eac0 2024-04-17T10:25:03.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3a000640 1 -- 172.21.15.67:0/1804972801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2c00c940 con 0x7fbb4406eac0 2024-04-17T10:25:03.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3a000640 1 -- 172.21.15.67:0/1804972801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c00cd30 con 0x7fbb4406eac0 2024-04-17T10:25:03.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4413ae90 con 0x7fbb4406eac0 2024-04-17T10:25:03.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4413b450 con 0x7fbb4406eac0 2024-04-17T10:25:03.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb44072d50 con 0x7fbb4406eac0 2024-04-17T10:25:03.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.231+0000 7fbb3a000640 1 -- 172.21.15.67:0/1804972801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbb2c01a070 con 0x7fbb4406eac0 2024-04-17T10:25:03.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.235+0000 7fbb3a000640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb20066f60 0x7fbb20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:03.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.235+0000 7fbb3a000640 1 -- 172.21.15.67:0/1804972801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbb2c0852e0 con 0x7fbb4406eac0 2024-04-17T10:25:03.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.235+0000 7fbb41e00640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb20066f60 0x7fbb20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:03.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.235+0000 7fbb41e00640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb20066f60 0x7fbb20069420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbb340ae000 tx=0x7fbb340a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.239+0000 7fbb3a000640 1 -- 172.21.15.67:0/1804972801 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbb2c0558c0 con 0x7fbb4406eac0 2024-04-17T10:25:03.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.439+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbb4415a8d0 con 0x7fbb4406eac0 2024-04-17T10:25:03.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.439+0000 7fbb3a000640 1 -- 172.21.15.67:0/1804972801 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7fbb2c059960 con 0x7fbb4406eac0 2024-04-17T10:25:03.440 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-17T10:25:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb20066f60 msgr2=0x7fbb20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb20066f60 0x7fbb20069420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbb340ae000 tx=0x7fbb340a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb4406eac0 msgr2=0x7fbb440ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb4406eac0 0x7fbb440ac8c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c00e890 tx=0x7fbb2c00ed50 comp rx=0 tx=0).stop 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 shutdown_connections 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb20066f60 0x7fbb20069420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb44143ae0 0x7fbb440ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb4406eac0 0x7fbb440ac8c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 --2- 172.21.15.67:0/1804972801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb4406e6e0 0x7fbb441599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 >> 172.21.15.67:0/1804972801 conn(0x7fbb4406ac90 msgr2=0x7fbb44140460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 shutdown_connections 2024-04-17T10:25:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:03.443+0000 7fbb49000640 1 -- 172.21.15.67:0/1804972801 wait complete. 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:25:03.460 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-17T10:25:03.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.461 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:03.470 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42443sec preferred_lft 42443sec 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:25:03.471 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:03.472 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-17T10:25:03.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.472 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:03.472 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:03.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-04-17T10:25:03.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-04-17T10:25:03.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-04-17T10:25:03.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-17T10:25:03.473 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:03.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.581 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:25:03.600 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:25:03.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.602 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:25:03.673 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:25:03.673 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:25:03.673 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:25:03.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.675 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:03.675 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:03.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:25:03.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:25:03.675 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:25:03.675 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:03.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.767 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:25:03.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.777 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-17T10:25:03.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.829 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:03.829 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:03.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:03.829 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-17T10:25:03.829 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:03.915 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-17T10:25:03.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:03.915 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:03.915 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:03.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-17T10:25:03.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-17T10:25:03.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-17T10:25:03.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-17T10:25:03.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-17T10:25:03.916 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:04.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:04.084 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:04.084 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:04.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:25:04.084 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-04-17T10:25:04.084 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:04.123 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-17T10:25:04.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:04.123 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:04.173 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-17T10:25:04.173 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:04.173 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:04.224 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-17T10:25:04.285 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-17T10:25:04.342 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-17T10:25:04.342 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-17T10:25:04.342 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/run/lock 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-17T10:25:04.343 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/fs/nfsd 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/run/rpc_pipefs 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:04.344 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:04.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:04.345 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-17T10:25:04.396 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-17T10:25:04.397 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-17T10:25:04.447 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-17T10:25:04.467 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-17T10:25:04.518 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-17T10:25:04.518 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-17T10:25:04.518 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-17T10:25:04.518 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-17T10:25:04.518 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-17T10:25:04.518 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/run/lock 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/fs/nfsd 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-17T10:25:04.519 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/run/rpc_pipefs 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:04.520 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:04.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:04.521 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-17T10:25:04.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:2024-04-17T10:25:04.547+0000 7f1847a4d580 -1 init, newargv = 0x559739d49760 newargc=15 2024-04-17T10:25:04.546 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:ceph-fuse[33691]: starting ceph client 2024-04-17T10:25:04.555 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:ceph-fuse[33691]: starting fuse 2024-04-17T10:25:05.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo modprobe fuse 2024-04-17T10:25:05.680 DEBUG:teuthology.orchestra.run.smithi067:> cat /proc/self/mounts | awk '{print $2}' 2024-04-17T10:25:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:/sys 2024-04-17T10:25:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:/proc 2024-04-17T10:25:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:/dev 2024-04-17T10:25:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/pts 2024-04-17T10:25:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:/run 2024-04-17T10:25:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:/ 2024-04-17T10:25:05.731 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/security 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/shm 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/run/lock 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/cgroup 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/pstore 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/bpf 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/sys/fs/binfmt_misc 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/hugepages 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/dev/mqueue 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/debug 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/tracing 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/proc/fs/nfsd 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/fs/fuse/connections 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/sys/kernel/config 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/run/credentials/systemd-sysusers.service 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/run/rpc_pipefs 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/run/user/1000 2024-04-17T10:25:05.732 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-0 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-1 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-2 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/var/lib/ceph/osd/ceph-3 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:05.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:05.734 DEBUG:teuthology.orchestra.run.smithi067:> ls /sys/fs/fuse/connections 2024-04-17T10:25:05.785 INFO:teuthology.orchestra.run.smithi067.stdout:44 2024-04-17T10:25:05.785 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [44] 2024-04-17T10:25:05.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo stdin-killer -- python3 -c ' 2024-04-17T10:25:05.785 DEBUG:teuthology.orchestra.run.smithi067:> import glob 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> import re 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> import subprocess 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> def _find_admin_socket(client_name): 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> files = glob.glob(asok_path) 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> # Given a non-glob path, it better be there 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> if "*" not in asok_path: 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> assert(len(files) == 1) 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> return files[0] 2024-04-17T10:25:05.786 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> for f in files: 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> contents = proc_f.read() 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> if mountpoint in contents: 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> return f 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> print(_find_admin_socket("client.admin")) 2024-04-17T10:25:05.787 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:25:05.920 INFO:teuthology.orchestra.run.smithi067.stdout:/var/run/ceph/ceph-client.admin.33691.asok 2024-04-17T10:25:05.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T10:25:05.932 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.33691.asok 2024-04-17T10:25:05.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:05.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.33691.asok status 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "metadata": { 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_sha1": "97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec", 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-1628-g97ae6cc9 (97ae6cc94daeb493fb1bfb6c82833f1485dfb1ec) squid (dev)", 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "entity_id": "admin", 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "hostname": "smithi067", 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "pid": "33691", 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "root": "/" 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:25:06.110 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_count": 0, 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "dentry_pinned_count": 0, 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4423, 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "inst": { 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "name": { 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "client", 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "num": 4423 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:0", 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2849379759 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": { 2024-04-17T10:25:06.111 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "v1", 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "addr": "172.21.15.67:0", 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "nonce": 2849379759 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "inst_str": "client.4423 172.21.15.67:0/2849379759", 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "addr_str": "172.21.15.67:0/2849379759", 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "inode_count": 1, 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_epoch": 7, 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch": 25, 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_epoch_barrier": 0, 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "blocklisted": false, 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs" 2024-04-17T10:25:06.112 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:25:06.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:06.125 DEBUG:teuthology.orchestra.run.smithi067:> stat --file-system '--printf=%T 2024-04-17T10:25:06.125 DEBUG:teuthology.orchestra.run.smithi067:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:06.134 INFO:teuthology.orchestra.run.smithi067.stdout:fuseblk 2024-04-17T10:25:06.135 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:06.135 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:06.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:06.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:06.197 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-17T10:25:06.274 INFO:teuthology.orchestra.run.smithi067.stdout:# file: . 2024-04-17T10:25:06.274 INFO:teuthology.orchestra.run.smithi067.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-17T10:25:06.274 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:06.275 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:06.275 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:06.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:06.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a87200640 1 Processor -- start 2024-04-17T10:25:06.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a87200640 1 -- start start 2024-04-17T10:25:06.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a87200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a80072370 0x7f3a8006f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a87200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a800ad8f0 con 0x7f3a8015dbc0 2024-04-17T10:25:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a87200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a800adbd0 con 0x7f3a8006c410 2024-04-17T10:25:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a87200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8015dfa0 con 0x7f3a80072370 2024-04-17T10:25:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a7fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a8015dbc0 0x7f3a8006db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39180/0 (socket says 172.21.15.67:39180) 2024-04-17T10:25:06.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.511+0000 7f3a7fe00640 1 -- 172.21.15.67:0/815502443 learned_addr learned my addr 172.21.15.67:0/815502443 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a85200640 1 --2- 172.21.15.67:0/815502443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a80072370 0x7f3a8006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193256811 0 0) 0x7f3a800ad8f0 con 0x7f3a8015dbc0 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a64003530 con 0x7f3a8015dbc0 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096316364 0 0) 0x7f3a800adbd0 con 0x7f3a8006c410 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a800ad8f0 con 0x7f3a8006c410 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2513923761 0 0) 0x7f3a64003530 con 0x7f3a8015dbc0 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a85200640 1 -- 172.21.15.67:0/815502443 >> v1:172.21.15.86:6789/0 conn(0x7f3a8006c410 legacy=0x7f3a80071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a85200640 1 -- 172.21.15.67:0/815502443 >> v1:172.21.15.67:6789/0 conn(0x7f3a8015dbc0 legacy=0x7f3a8006db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a85200640 1 -- 172.21.15.67:0/815502443 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a800adeb0 con 0x7f3a80072370 2024-04-17T10:25:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a85200640 1 --2- 172.21.15.67:0/815502443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a80072370 0x7f3a8006f560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009dc0 tx=0x7f3a7801c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=120018f7c0ce73ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a780ac070 con 0x7f3a80072370 2024-04-17T10:25:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a780a6470 con 0x7f3a80072370 2024-04-17T10:25:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.515+0000 7f3a7f400640 1 -- 172.21.15.67:0/815502443 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70001680 con 0x7f3a80072370 2024-04-17T10:25:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- 172.21.15.67:0/815502443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a80072370 msgr2=0x7f3a8006f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 --2- 172.21.15.67:0/815502443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a80072370 0x7f3a8006f560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009dc0 tx=0x7f3a7801c820 comp rx=0 tx=0).stop 2024-04-17T10:25:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- 172.21.15.67:0/815502443 shutdown_connections 2024-04-17T10:25:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 --2- 172.21.15.67:0/815502443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a80072370 0x7f3a8006f560 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- 172.21.15.67:0/815502443 >> 172.21.15.67:0/815502443 conn(0x7f3a800af670 msgr2=0x7f3a800b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- 172.21.15.67:0/815502443 shutdown_connections 2024-04-17T10:25:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- 172.21.15.67:0/815502443 wait complete. 2024-04-17T10:25:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 Processor -- start 2024-04-17T10:25:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- start start 2024-04-17T10:25:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006c410 0x7f3a80138790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a80072370 0x7f3a801492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a8015dbc0 0x7f3a80149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8016b110 con 0x7f3a8006c410 2024-04-17T10:25:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8006af70 con 0x7f3a80072370 2024-04-17T10:25:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a87200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8016b290 con 0x7f3a8015dbc0 2024-04-17T10:25:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006c410 0x7f3a80138790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006c410 0x7f3a80138790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43108/0 (socket says 172.21.15.67:43108) 2024-04-17T10:25:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 -- 172.21.15.67:0/3613444203 learned_addr learned my addr 172.21.15.67:0/3613444203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a85200640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a8015dbc0 0x7f3a80149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a77400640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a80072370 0x7f3a801492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 -- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a8015dbc0 msgr2=0x7f3a80149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a8015dbc0 0x7f3a80149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 -- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a80072370 msgr2=0x7f3a801492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a80072370 0x7f3a801492c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.519+0000 7f3a7fe00640 1 -- 172.21.15.67:0/3613444203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8014c080 con 0x7f3a8006c410 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006c410 0x7f3a80138790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000c600 tx=0x7f3a7000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a7cc00640 1 -- 172.21.15.67:0/3613444203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a70017070 con 0x7f3a8006c410 2024-04-17T10:25:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a7cc00640 1 -- 172.21.15.67:0/3613444203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a70005d10 con 0x7f3a8006c410 2024-04-17T10:25:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a7cc00640 1 -- 172.21.15.67:0/3613444203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a700072b0 con 0x7f3a8006c410 2024-04-17T10:25:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a80153360 con 0x7f3a8006c410 2024-04-17T10:25:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8014bda0 con 0x7f3a8006c410 2024-04-17T10:25:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8009e260 con 0x7f3a8006c410 2024-04-17T10:25:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a7cc00640 1 -- 172.21.15.67:0/3613444203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3a7001a070 con 0x7f3a8006c410 2024-04-17T10:25:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a7cc00640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a5c066f90 0x7f3a5c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.523+0000 7f3a7cc00640 1 -- 172.21.15.67:0/3613444203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3a700847e0 con 0x7f3a8006c410 2024-04-17T10:25:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.527+0000 7f3a77400640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a5c066f90 0x7f3a5c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.527+0000 7f3a77400640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a5c066f90 0x7f3a5c069450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3a80168370 tx=0x7f3a68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.527+0000 7f3a7cc00640 1 -- 172.21.15.67:0/3613444203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3a70054e50 con 0x7f3a8006c410 2024-04-17T10:25:06.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.695+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a80074e30 con 0x7f3a8006c410 2024-04-17T10:25:06.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.695+0000 7f3a7cc00640 1 -- 172.21.15.67:0/3613444203 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7f3a70058ef0 con 0x7f3a8006c410 2024-04-17T10:25:06.696 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:06.696 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a5c066f90 msgr2=0x7f3a5c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a5c066f90 0x7f3a5c069450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3a80168370 tx=0x7f3a68009040 comp rx=0 tx=0).stop 2024-04-17T10:25:06.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006c410 msgr2=0x7f3a80138790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006c410 0x7f3a80138790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3a7000c600 tx=0x7f3a7000cac0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 shutdown_connections 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a5c066f90 0x7f3a5c069450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a8015dbc0 0x7f3a80149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a80072370 0x7f3a801492c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 --2- 172.21.15.67:0/3613444203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a8006c410 0x7f3a80138790 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 >> 172.21.15.67:0/3613444203 conn(0x7f3a800af670 msgr2=0x7f3a800b3730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 shutdown_connections 2024-04-17T10:25:06.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.699+0000 7f3a87200640 1 -- 172.21.15.67:0/3613444203 wait complete. 2024-04-17T10:25:06.718 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:06.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:06.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d3200640 1 Processor -- start 2024-04-17T10:25:06.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d3200640 1 -- start start 2024-04-17T10:25:06.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d3200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 0x7f75cc15d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0d03c0 con 0x7f75cc06f8b0 2024-04-17T10:25:06.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d3200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75cc0d06a0 con 0x7f75cc06d810 2024-04-17T10:25:06.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d3200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0d0980 con 0x7f75cc0707c0 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75c3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75cc06f8b0 0x7f75cc0700b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39186/0 (socket says 172.21.15.67:39186) 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75c3400640 1 -- 172.21.15.67:0/1752743937 learned_addr learned my addr 172.21.15.67:0/1752743937 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d1200640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 0x7f75cc15d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272529101 0 0) 0x7f75cc0d03c0 con 0x7f75cc06f8b0 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b0003530 con 0x7f75cc06f8b0 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2547078905 0 0) 0x7f75b0003530 con 0x7f75cc06f8b0 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 msgr2=0x7f75cc15d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 0x7f75cc15d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 >> v1:172.21.15.86:6789/0 conn(0x7f75cc06d810 legacy=0x7f75cc0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc0d0c60 con 0x7f75cc06f8b0 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f75b4001680 con 0x7f75cc06f8b0 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 >> v1:172.21.15.67:6789/0 conn(0x7f75cc06f8b0 legacy=0x7f75cc0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b0004810 0x7f75b0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 0x7f75b00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc0d0980 con 0x7f75b00039f0 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b000bae0 con 0x7f75b0004810 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b000bce0 con 0x7f75cc0707c0 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75cbe00640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75c3400640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b0004810 0x7f75b0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.863+0000 7f75d1200640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 0x7f75b00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cbe00640 1 -- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 msgr2=0x7f75b00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cbe00640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc0707c0 0x7f75b00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cbe00640 1 -- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b0004810 msgr2=0x7f75b0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:06.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cbe00640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b0004810 0x7f75b0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cbe00640 1 -- 172.21.15.67:0/1752743937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b0003530 con 0x7f75b00039f0 2024-04-17T10:25:06.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cbe00640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75b0003e10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f75bc001750 tx=0x7f75bc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3d3f21c1067ab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:06.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc0013a0 con 0x7f75b00039f0 2024-04-17T10:25:06.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc0052f0 con 0x7f75b00039f0 2024-04-17T10:25:06.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75cb400640 1 -- 172.21.15.67:0/1752743937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc06cfc0 con 0x7f75b00039f0 2024-04-17T10:25:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75d3200640 1 -- 172.21.15.67:0/1752743937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 msgr2=0x7f75b0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75d3200640 1 --2- 172.21.15.67:0/1752743937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75b0003e10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f75bc001750 tx=0x7f75bc01c3c0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75d1200640 1 -- 172.21.15.67:0/1752743937 reap_dead start 2024-04-17T10:25:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75d3200640 1 -- 172.21.15.67:0/1752743937 shutdown_connections 2024-04-17T10:25:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75d3200640 1 -- 172.21.15.67:0/1752743937 >> 172.21.15.67:0/1752743937 conn(0x7f75cc0d1800 msgr2=0x7f75cc0d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75d3200640 1 -- 172.21.15.67:0/1752743937 shutdown_connections 2024-04-17T10:25:06.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.867+0000 7f75d3200640 1 -- 172.21.15.67:0/1752743937 wait complete. 2024-04-17T10:25:06.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 Processor -- start 2024-04-17T10:25:06.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 -- start start 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75cc154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75cc158690 0x7f75cc155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc155e50 0x7f75cc0b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc15de00 con 0x7f75b00039f0 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc15dc80 con 0x7f75cc158690 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc15df80 con 0x7f75cc155e50 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75cc154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75cc154920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43132/0 (socket says 172.21.15.67:43132) 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 -- 172.21.15.67:0/4145969475 learned_addr learned my addr 172.21.15.67:0/4145969475 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75c3400640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75cc158690 0x7f75cc155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d1200640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc155e50 0x7f75cc0b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 -- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc155e50 msgr2=0x7f75cc0b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc155e50 0x7f75cc0b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 -- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75cc158690 msgr2=0x7f75cc155910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75cc158690 0x7f75cc155910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:06.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 -- 172.21.15.67:0/4145969475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc155180 con 0x7f75b00039f0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75cbe00640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75cc154920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005040 tx=0x7f75b0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75c8c00640 1 -- 172.21.15.67:0/4145969475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc007a40 con 0x7f75b00039f0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75c8c00640 1 -- 172.21.15.67:0/4145969475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75bc0bb070 con 0x7f75b00039f0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75c8c00640 1 -- 172.21.15.67:0/4145969475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75bc0013a0 con 0x7f75b00039f0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.871+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc158b10 con 0x7f75b00039f0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.875+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc158d90 con 0x7f75b00039f0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.875+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc04ef30 con 0x7f75b00039f0 2024-04-17T10:25:06.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.875+0000 7f75c8c00640 1 -- 172.21.15.67:0/4145969475 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f75bc0b4040 con 0x7f75b00039f0 2024-04-17T10:25:06.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.875+0000 7f75c8c00640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75a8066f60 0x7f75a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.875+0000 7f75c8c00640 1 -- 172.21.15.67:0/4145969475 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f75bc11f130 con 0x7f75b00039f0 2024-04-17T10:25:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.875+0000 7f75c3400640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75a8066f60 0x7f75a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.875+0000 7f75c3400640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75a8066f60 0x7f75a8069420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75b4000c10 tx=0x7f75b4000ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:06.879+0000 7f75c8c00640 1 -- 172.21.15.67:0/4145969475 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f75cc158b10 con 0x7f75b00039f0 2024-04-17T10:25:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75cc0b7780 con 0x7f75b00039f0 2024-04-17T10:25:07.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75c8c00640 1 -- 172.21.15.67:0/4145969475 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75bc0160d0 con 0x7f75b00039f0 2024-04-17T10:25:07.042 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75a8066f60 msgr2=0x7f75a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75a8066f60 0x7f75a8069420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f75b4000c10 tx=0x7f75b4000ee0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 msgr2=0x7f75cc154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75cc154920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f75bc005040 tx=0x7f75b0005840 comp rx=0 tx=0).stop 2024-04-17T10:25:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 shutdown_connections 2024-04-17T10:25:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75a8066f60 0x7f75a8069420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75cc155e50 0x7f75cc0b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75cc158690 0x7f75cc155910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 --2- 172.21.15.67:0/4145969475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00039f0 0x7f75cc154920 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 >> 172.21.15.67:0/4145969475 conn(0x7f75cc0d1800 msgr2=0x7f75cc0700b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 shutdown_connections 2024-04-17T10:25:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.043+0000 7f75d3200640 1 -- 172.21.15.67:0/4145969475 wait complete. 2024-04-17T10:25:07.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 Processor -- start 2024-04-17T10:25:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 -- start start 2024-04-17T10:25:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15d815dd60 0x7f15d8161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f15d809adf0 0x7f15d8161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d80cff90 0x7f15d80d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d81626d0 con 0x7f15d815dd60 2024-04-17T10:25:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d815e140 con 0x7f15d809adf0 2024-04-17T10:25:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15dcc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d80d0960 con 0x7f15d80cff90 2024-04-17T10:25:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15d815dd60 0x7f15d8161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f15d815dd60 0x7f15d8161880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43138/0 (socket says 172.21.15.67:43138) 2024-04-17T10:25:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 -- 172.21.15.67:0/3552291035 learned_addr learned my addr 172.21.15.67:0/3552291035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15cfe00640 1 --2- 172.21.15.67:0/3552291035 >> v2:172.21.15.86:3300/0 conn(0x7f15d809adf0 0x7f15d8161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d6800640 1 --2- 172.21.15.67:0/3552291035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d80cff90 0x7f15d80d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 -- 172.21.15.67:0/3552291035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d80cff90 msgr2=0x7f15d80d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 --2- 172.21.15.67:0/3552291035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d80cff90 0x7f15d80d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 -- 172.21.15.67:0/3552291035 >> v2:172.21.15.86:3300/0 conn(0x7f15d809adf0 msgr2=0x7f15d8161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 --2- 172.21.15.67:0/3552291035 >> v2:172.21.15.86:3300/0 conn(0x7f15d809adf0 0x7f15d8161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d5e00640 1 -- 172.21.15.67:0/3552291035 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d8099d50 con 0x7f15d815dd60 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15d6800640 1 --2- 172.21.15.67:0/3552291035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d80cff90 0x7f15d80d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.207+0000 7f15cfe00640 1 --2- 172.21.15.67:0/3552291035 >> v2:172.21.15.86:3300/0 conn(0x7f15d809adf0 0x7f15d8161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15d5e00640 1 --2- 172.21.15.67:0/3552291035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d815dd60 0x7f15d8161880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f15d8073040 tx=0x7f15c801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300c1dcf96c28110 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15d5400640 1 -- 172.21.15.67:0/3552291035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c80aa610 con 0x7f15d815dd60 2024-04-17T10:25:07.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15d5400640 1 -- 172.21.15.67:0/3552291035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c80af070 con 0x7f15d815dd60 2024-04-17T10:25:07.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15d5400640 1 -- 172.21.15.67:0/3552291035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c80aa9d0 con 0x7f15d815dd60 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 -- 172.21.15.67:0/3552291035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d815dd60 msgr2=0x7f15d8161880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 --2- 172.21.15.67:0/3552291035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d815dd60 0x7f15d8161880 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f15d8073040 tx=0x7f15c801c4f0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 -- 172.21.15.67:0/3552291035 shutdown_connections 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 --2- 172.21.15.67:0/3552291035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d80cff90 0x7f15d80d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 --2- 172.21.15.67:0/3552291035 >> v2:172.21.15.86:3300/0 conn(0x7f15d809adf0 0x7f15d8161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 --2- 172.21.15.67:0/3552291035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d815dd60 0x7f15d8161880 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 -- 172.21.15.67:0/3552291035 >> 172.21.15.67:0/3552291035 conn(0x7f15d80b34a0 msgr2=0x7f15d80b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 -- 172.21.15.67:0/3552291035 shutdown_connections 2024-04-17T10:25:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.211+0000 7f15dcc00640 1 -- 172.21.15.67:0/3552291035 wait complete. 2024-04-17T10:25:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 Processor -- start 2024-04-17T10:25:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 -- start start 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15d809adf0 0x7f15d80cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d80cff90 0x7f15d81591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d815dd60 0x7f15d80c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d806c5b0 con 0x7f15d80cff90 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d809a8d0 con 0x7f15d809adf0 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d806c730 con 0x7f15d815dd60 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d80cff90 0x7f15d81591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d80cff90 0x7f15d81591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43144/0 (socket says 172.21.15.67:43144) 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15d5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15d809adf0 0x7f15d80cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 -- 172.21.15.67:0/2022968049 learned_addr learned my addr 172.21.15.67:0/2022968049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15d6800640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d815dd60 0x7f15d80c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 -- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d815dd60 msgr2=0x7f15d80c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d815dd60 0x7f15d80c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 -- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15d809adf0 msgr2=0x7f15d80cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15d809adf0 0x7f15d80cb3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 -- 172.21.15.67:0/2022968049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d80ca770 con 0x7f15d80cff90 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15cfe00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d80cff90 0x7f15d81591b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f15c000c7c0 tx=0x7f15c000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15ce000640 1 -- 172.21.15.67:0/2022968049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0015070 con 0x7f15d80cff90 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15ce000640 1 -- 172.21.15.67:0/2022968049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c0007a30 con 0x7f15d80cff90 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15ce000640 1 -- 172.21.15.67:0/2022968049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c0007e00 con 0x7f15d80cff90 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d80caa50 con 0x7f15d80cff90 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d80c61d0 con 0x7f15d80cff90 2024-04-17T10:25:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.215+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d804ef30 con 0x7f15d80cff90 2024-04-17T10:25:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.219+0000 7f15ce000640 1 -- 172.21.15.67:0/2022968049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f15c0018070 con 0x7f15d80cff90 2024-04-17T10:25:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.219+0000 7f15ce000640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b4066f60 0x7f15b4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.219+0000 7f15ce000640 1 -- 172.21.15.67:0/2022968049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f15c0085260 con 0x7f15d80cff90 2024-04-17T10:25:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.219+0000 7f15d5e00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b4066f60 0x7f15b4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.219+0000 7f15d5e00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b4066f60 0x7f15b4069420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15c80ae000 tx=0x7f15c80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.223+0000 7f15ce000640 1 -- 172.21.15.67:0/2022968049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f15c0055840 con 0x7f15d80cff90 2024-04-17T10:25:07.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f15d80b42d0 con 0x7f15d80cff90 2024-04-17T10:25:07.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15ce000640 1 -- 172.21.15.67:0/2022968049 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f15c00598e0 con 0x7f15d80cff90 2024-04-17T10:25:07.386 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b4066f60 msgr2=0x7f15b4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b4066f60 0x7f15b4069420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f15c80ae000 tx=0x7f15c80a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d80cff90 msgr2=0x7f15d81591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d80cff90 0x7f15d81591b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f15c000c7c0 tx=0x7f15c000cc80 comp rx=0 tx=0).stop 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 shutdown_connections 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b4066f60 0x7f15b4069420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15d815dd60 0x7f15d80c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d80cff90 0x7f15d81591b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 --2- 172.21.15.67:0/2022968049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15d809adf0 0x7f15d80cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 >> 172.21.15.67:0/2022968049 conn(0x7f15d80b34a0 msgr2=0x7f15d8161f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:07.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 shutdown_connections 2024-04-17T10:25:07.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.387+0000 7f15dcc00640 1 -- 172.21.15.67:0/2022968049 wait complete. 2024-04-17T10:25:07.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:07.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c9000640 1 Processor -- start 2024-04-17T10:25:07.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c9000640 1 -- start start 2024-04-17T10:25:07.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c9000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7c409a3c0 0x7fd7c40b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c9000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40b3290 con 0x7fd7c409a900 2024-04-17T10:25:07.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c9000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c415f610 con 0x7fd7c415f230 2024-04-17T10:25:07.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c9000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c40cf6b0 con 0x7fd7c409a3c0 2024-04-17T10:25:07.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7bbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd7c409a900 0x7fd7c4099cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39190/0 (socket says 172.21.15.67:39190) 2024-04-17T10:25:07.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7bbe00640 1 -- 172.21.15.67:0/3816746722 learned_addr learned my addr 172.21.15.67:0/3816746722 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:07.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c2800640 1 --2- 172.21.15.67:0/3816746722 >> v2:172.21.15.86:3300/0 conn(0x7fd7c409a3c0 0x7fd7c40b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879863177 0 0) 0x7fd7c40b3290 con 0x7fd7c409a900 2024-04-17T10:25:07.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a8003530 con 0x7fd7c409a900 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 646112566 0 0) 0x7fd7a8003530 con 0x7fd7c409a900 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 >> v2:172.21.15.86:3300/0 conn(0x7fd7c409a3c0 msgr2=0x7fd7c40b17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 --2- 172.21.15.67:0/3816746722 >> v2:172.21.15.86:3300/0 conn(0x7fd7c409a3c0 0x7fd7c40b17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 >> v1:172.21.15.86:6789/0 conn(0x7fd7c415f230 legacy=0x7fd7c40cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40b0fc0 con 0x7fd7c409a900 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fd7ac001680 con 0x7fd7c409a900 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 >> v1:172.21.15.67:6789/0 conn(0x7fd7c409a900 legacy=0x7fd7c4099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7a80039c0 0x7fd7a8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7a8004810 0x7fd7a8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80051d0 0x7fd7a800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40cf6b0 con 0x7fd7a80039c0 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a800c170 con 0x7fd7a8004810 2024-04-17T10:25:07.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a800c370 con 0x7fd7a80051d0 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c1e00640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7a80039c0 0x7fd7a8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7bbe00640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7a8004810 0x7fd7a8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.551+0000 7fd7c2800640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80051d0 0x7fd7a800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1e00640 1 -- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80051d0 msgr2=0x7fd7a800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1e00640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80051d0 0x7fd7a800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1e00640 1 -- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7a8004810 msgr2=0x7fd7a8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1e00640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7a8004810 0x7fd7a8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1e00640 1 -- 172.21.15.67:0/3816746722 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a8003530 con 0x7fd7a80039c0 2024-04-17T10:25:07.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c2800640 1 -- 172.21.15.67:0/3816746722 reap_dead start 2024-04-17T10:25:07.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1e00640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7a80039c0 0x7fd7a8003de0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4001780 tx=0x7fd7b401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11593143a82829d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40013a0 con 0x7fd7a80039c0 2024-04-17T10:25:07.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b40052f0 con 0x7fd7a80039c0 2024-04-17T10:25:07.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c1400640 1 -- 172.21.15.67:0/3816746722 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4003840 con 0x7fd7a80039c0 2024-04-17T10:25:07.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c9000640 1 -- 172.21.15.67:0/3816746722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7a80039c0 msgr2=0x7fd7a8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7a80039c0 0x7fd7a8003de0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4001780 tx=0x7fd7b401c400 comp rx=0 tx=0).stop 2024-04-17T10:25:07.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c9000640 1 -- 172.21.15.67:0/3816746722 shutdown_connections 2024-04-17T10:25:07.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3816746722 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7a80039c0 0x7fd7a8003de0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c9000640 1 -- 172.21.15.67:0/3816746722 >> 172.21.15.67:0/3816746722 conn(0x7fd7c40b34a0 msgr2=0x7fd7c40b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:07.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c9000640 1 -- 172.21.15.67:0/3816746722 shutdown_connections 2024-04-17T10:25:07.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.555+0000 7fd7c9000640 1 -- 172.21.15.67:0/3816746722 wait complete. 2024-04-17T10:25:07.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 Processor -- start 2024-04-17T10:25:07.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 -- start start 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80039c0 0x7fd7c40af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4159070 0x7fd7c41594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c406d050 0x7fd7c406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40b3050 con 0x7fd7c406d050 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40b2ed0 con 0x7fd7c4159070 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40b3fc0 con 0x7fd7a80039c0 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c406d050 0x7fd7c406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c406d050 0x7fd7c406d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43164/0 (socket says 172.21.15.67:43164) 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 -- 172.21.15.67:0/3320832316 learned_addr learned my addr 172.21.15.67:0/3320832316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7bbe00640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4159070 0x7fd7c41594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c1e00640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80039c0 0x7fd7c40af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 -- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80039c0 msgr2=0x7fd7c40af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80039c0 0x7fd7c40af9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 -- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4159070 msgr2=0x7fd7c41594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4159070 0x7fd7c41594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 -- 172.21.15.67:0/3320832316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c40cddb0 con 0x7fd7c406d050 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7bbe00640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4159070 0x7fd7c41594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c2800640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c406d050 0x7fd7c406d4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7c406e380 tx=0x7fd7bc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7ba000640 1 -- 172.21.15.67:0/3320832316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc00fdb0 con 0x7fd7c406d050 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7ba000640 1 -- 172.21.15.67:0/3320832316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc01d070 con 0x7fd7c406d050 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c40c54e0 con 0x7fd7c406d050 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7ba000640 1 -- 172.21.15.67:0/3320832316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc015670 con 0x7fd7c406d050 2024-04-17T10:25:07.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.559+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c41f7020 con 0x7fd7c406d050 2024-04-17T10:25:07.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.563+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c404ef30 con 0x7fd7c406d050 2024-04-17T10:25:07.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.563+0000 7fd7ba000640 1 -- 172.21.15.67:0/3320832316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd7bc015810 con 0x7fd7c406d050 2024-04-17T10:25:07.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.563+0000 7fd7ba000640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd7a0066f60 0x7fd7a0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.563+0000 7fd7ba000640 1 -- 172.21.15.67:0/3320832316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd7bc085f40 con 0x7fd7c406d050 2024-04-17T10:25:07.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.563+0000 7fd7c1e00640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd7a0066f60 0x7fd7a0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.567+0000 7fd7c1e00640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd7a0066f60 0x7fd7a0069420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4001750 tx=0x7fd7b40011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.567+0000 7fd7ba000640 1 -- 172.21.15.67:0/3320832316 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd7bc027020 con 0x7fd7c406d050 2024-04-17T10:25:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.731+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd7c40b43b0 con 0x7fd7c406d050 2024-04-17T10:25:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7ba000640 1 -- 172.21.15.67:0/3320832316 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd7c40c54e0 con 0x7fd7c406d050 2024-04-17T10:25:07.733 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:07.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd7a0066f60 msgr2=0x7fd7a0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd7a0066f60 0x7fd7a0069420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4001750 tx=0x7fd7b40011a0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c406d050 msgr2=0x7fd7c406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c406d050 0x7fd7c406d4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7c406e380 tx=0x7fd7bc00f580 comp rx=0 tx=0).stop 2024-04-17T10:25:07.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 shutdown_connections 2024-04-17T10:25:07.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd7a0066f60 0x7fd7a0069420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7c406d050 0x7fd7c406d4d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd7c4159070 0x7fd7c41594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 --2- 172.21.15.67:0/3320832316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7a80039c0 0x7fd7c40af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 >> 172.21.15.67:0/3320832316 conn(0x7fd7c40b34a0 msgr2=0x7fd7c40b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:07.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 shutdown_connections 2024-04-17T10:25:07.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.735+0000 7fd7c9000640 1 -- 172.21.15.67:0/3320832316 wait complete. 2024-04-17T10:25:07.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:07.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f7e00640 1 Processor -- start 2024-04-17T10:25:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f7e00640 1 -- start start 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f7e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f20f006cba0 0x7f20f00d0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f00cfad0 con 0x7f20f0099b30 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f7e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20f006c250 con 0x7f20f00cd9b0 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f7e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20f015eac0 con 0x7f20f006cba0 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20f0099b30 0x7f20f00d07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39192/0 (socket says 172.21.15.67:39192) 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 -- 172.21.15.67:0/2733755187 learned_addr learned my addr 172.21.15.67:0/2733755187 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f4a00640 1 --2- 172.21.15.67:0/2733755187 >> v2:172.21.15.86:3300/0 conn(0x7f20f006cba0 0x7f20f00d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597347463 0 0) 0x7f20f00cfad0 con 0x7f20f0099b30 2024-04-17T10:25:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20d8003530 con 0x7f20f0099b30 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1036873972 0 0) 0x7f20d8003530 con 0x7f20f0099b30 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 >> v2:172.21.15.86:3300/0 conn(0x7f20f006cba0 msgr2=0x7f20f00d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 --2- 172.21.15.67:0/2733755187 >> v2:172.21.15.86:3300/0 conn(0x7f20f006cba0 0x7f20f00d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 >> v1:172.21.15.86:6789/0 conn(0x7f20f00cd9b0 legacy=0x7f20f006c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f015eda0 con 0x7f20f0099b30 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f20e8001a00 con 0x7f20f0099b30 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 >> v1:172.21.15.67:6789/0 conn(0x7f20f0099b30 legacy=0x7f20f00d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20d8003a10 0x7f20d8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d8004810 0x7f20d8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80051d0 0x7f20d800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f015eac0 con 0x7f20d80051d0 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d800c0d0 con 0x7f20d8004810 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d800c2d0 con 0x7f20d8003a10 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80051d0 0x7f20d800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5400640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20d8003a10 0x7f20d8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f4a00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d8004810 0x7f20d8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 -- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20d8003a10 msgr2=0x7f20d8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20d8003a10 0x7f20d8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 -- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d8004810 msgr2=0x7f20d8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d8004810 0x7f20d8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.899+0000 7f20f5e00640 1 -- 172.21.15.67:0/2733755187 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d8003530 con 0x7f20d80051d0 2024-04-17T10:25:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f5e00640 1 -- 172.21.15.67:0/2733755187 reap_dead start 2024-04-17T10:25:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f5e00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80051d0 0x7f20d800b9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f20e800a7b0 tx=0x7f20e801d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776cd13f3489bd9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e800a3d0 con 0x7f20d80051d0 2024-04-17T10:25:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e80ad650 con 0x7f20d80051d0 2024-04-17T10:25:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20efe00640 1 -- 172.21.15.67:0/2733755187 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80ada40 con 0x7f20d80051d0 2024-04-17T10:25:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f7e00640 1 -- 172.21.15.67:0/2733755187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80051d0 msgr2=0x7f20d800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f7e00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80051d0 0x7f20d800b9d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f20e800a7b0 tx=0x7f20e801d2e0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f7e00640 1 -- 172.21.15.67:0/2733755187 shutdown_connections 2024-04-17T10:25:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f7e00640 1 --2- 172.21.15.67:0/2733755187 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20d80051d0 0x7f20d800b9d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f7e00640 1 -- 172.21.15.67:0/2733755187 >> 172.21.15.67:0/2733755187 conn(0x7f20f00d1890 msgr2=0x7f20f00d1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f7e00640 1 -- 172.21.15.67:0/2733755187 shutdown_connections 2024-04-17T10:25:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.903+0000 7f20f7e00640 1 -- 172.21.15.67:0/2733755187 wait complete. 2024-04-17T10:25:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 Processor -- start 2024-04-17T10:25:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 -- start start 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d80051d0 0x7f20f00c6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0153920 0x7f20f0153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f015f3a0 0x7f20f015f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f006c490 con 0x7f20f015f3a0 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f00cfd70 con 0x7f20d80051d0 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f006c610 con 0x7f20f0153920 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f015f3a0 0x7f20f015f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f015f3a0 0x7f20f015f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43188/0 (socket says 172.21.15.67:43188) 2024-04-17T10:25:07.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 -- 172.21.15.67:0/1258943082 learned_addr learned my addr 172.21.15.67:0/1258943082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5400640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d80051d0 0x7f20f00c6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f4a00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0153920 0x7f20f0153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 -- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0153920 msgr2=0x7f20f0153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0153920 0x7f20f0153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 -- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d80051d0 msgr2=0x7f20f00c6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d80051d0 0x7f20f00c6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 -- 172.21.15.67:0/1258943082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f0160c90 con 0x7f20f015f3a0 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5400640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d80051d0 0x7f20f00c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f4a00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0153920 0x7f20f0153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f5e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f015f3a0 0x7f20f015f820 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f20e801d110 tx=0x7f20d8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20ed600640 1 -- 172.21.15.67:0/1258943082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8001940 con 0x7f20f015f3a0 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20ed600640 1 -- 172.21.15.67:0/1258943082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e80b4070 con 0x7f20f015f3a0 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20f0160110 con 0x7f20f015f3a0 2024-04-17T10:25:07.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20ed600640 1 -- 172.21.15.67:0/1258943082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80ada00 con 0x7f20f015f3a0 2024-04-17T10:25:07.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.907+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20f01606d0 con 0x7f20f015f3a0 2024-04-17T10:25:07.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.911+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20f004ef30 con 0x7f20f015f3a0 2024-04-17T10:25:07.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.911+0000 7f20ed600640 1 -- 172.21.15.67:0/1258943082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f20e80b9470 con 0x7f20f015f3a0 2024-04-17T10:25:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.911+0000 7f20ed600640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20d0066f60 0x7f20d0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.911+0000 7f20ed600640 1 -- 172.21.15.67:0/1258943082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f20e811fc10 con 0x7f20f015f3a0 2024-04-17T10:25:07.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.911+0000 7f20f5400640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20d0066f60 0x7f20d0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:07.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.911+0000 7f20f5400640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20d0066f60 0x7f20d0069420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f20e4000c10 tx=0x7f20e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:07.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:07.915+0000 7f20ed600640 1 -- 172.21.15.67:0/1258943082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f20e8016ff0 con 0x7f20f015f3a0 2024-04-17T10:25:08.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f20f00d26a0 con 0x7f20f015f3a0 2024-04-17T10:25:08.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20ed600640 1 -- 172.21.15.67:0/1258943082 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f20e80f6020 con 0x7f20f015f3a0 2024-04-17T10:25:08.075 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:08.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20d0066f60 msgr2=0x7f20d0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20d0066f60 0x7f20d0069420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f20e4000c10 tx=0x7f20e4009040 comp rx=0 tx=0).stop 2024-04-17T10:25:08.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f015f3a0 msgr2=0x7f20f015f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f015f3a0 0x7f20f015f820 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f20e801d110 tx=0x7f20d8005840 comp rx=0 tx=0).stop 2024-04-17T10:25:08.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 shutdown_connections 2024-04-17T10:25:08.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20d0066f60 0x7f20d0069420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f015f3a0 0x7f20f015f820 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f0153920 0x7f20f0153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 --2- 172.21.15.67:0/1258943082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20d80051d0 0x7f20f00c6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.075+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 >> 172.21.15.67:0/1258943082 conn(0x7f20f00d1890 msgr2=0x7f20f00b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.079+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 shutdown_connections 2024-04-17T10:25:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.079+0000 7f20f7e00640 1 -- 172.21.15.67:0/1258943082 wait complete. 2024-04-17T10:25:08.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:25:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f980b200640 1 Processor -- start 2024-04-17T10:25:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f980b200640 1 -- start start 2024-04-17T10:25:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f980b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f980c154c70 0x7f980c072350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f980b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c072890 0x7f980c0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f980b200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f980c15ad00 con 0x7f980c0cdb30 2024-04-17T10:25:08.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f980b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f980c15b910 con 0x7f980c154c70 2024-04-17T10:25:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f980b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c15afe0 con 0x7f980c072890 2024-04-17T10:25:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f980c154c70 0x7f980c072350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f980c154c70 0x7f980c072350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43196/0 (socket says 172.21.15.67:43196) 2024-04-17T10:25:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809e00640 1 -- 172.21.15.67:0/2287349603 learned_addr learned my addr 172.21.15.67:0/2287349603 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809400640 1 --2- 172.21.15.67:0/2287349603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c072890 0x7f980c0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809e00640 1 -- 172.21.15.67:0/2287349603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c072890 msgr2=0x7f980c0cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809e00640 1 --2- 172.21.15.67:0/2287349603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c072890 0x7f980c0cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809e00640 1 -- 172.21.15.67:0/2287349603 >> v1:172.21.15.86:6789/0 conn(0x7f980c0cdb30 legacy=0x7f980c06fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809e00640 1 -- 172.21.15.67:0/2287349603 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c15b2c0 con 0x7f980c154c70 2024-04-17T10:25:08.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.243+0000 7f9809400640 1 --2- 172.21.15.67:0/2287349603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c072890 0x7f980c0cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f9809e00640 1 --2- 172.21.15.67:0/2287349603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c072350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aeaa18cb87ff226 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f9808a00640 1 -- 172.21.15.67:0/2287349603 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0b2070 con 0x7f980c154c70 2024-04-17T10:25:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f9808a00640 1 -- 172.21.15.67:0/2287349603 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97fc0a9070 con 0x7f980c154c70 2024-04-17T10:25:08.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f9808a00640 1 -- 172.21.15.67:0/2287349603 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97fc0ad880 con 0x7f980c154c70 2024-04-17T10:25:08.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 -- 172.21.15.67:0/2287349603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 msgr2=0x7f980c072350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 --2- 172.21.15.67:0/2287349603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c072350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc01c560 comp rx=0 tx=0).stop 2024-04-17T10:25:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 -- 172.21.15.67:0/2287349603 shutdown_connections 2024-04-17T10:25:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 --2- 172.21.15.67:0/2287349603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c072890 0x7f980c0cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 --2- 172.21.15.67:0/2287349603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c072350 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 -- 172.21.15.67:0/2287349603 >> 172.21.15.67:0/2287349603 conn(0x7f980c0b23f0 msgr2=0x7f980c0b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 -- 172.21.15.67:0/2287349603 shutdown_connections 2024-04-17T10:25:08.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.247+0000 7f980b200640 1 -- 172.21.15.67:0/2287349603 wait complete. 2024-04-17T10:25:08.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 Processor -- start 2024-04-17T10:25:08.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 -- start start 2024-04-17T10:25:08.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c072890 0x7f980c1564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0cdb30 0x7f980c14c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c167130 con 0x7f980c154c70 2024-04-17T10:25:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0ce150 con 0x7f980c072890 2024-04-17T10:25:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c1672b0 con 0x7f980c0cdb30 2024-04-17T10:25:08.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f9809e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c072890 0x7f980c1564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f9809e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c072890 0x7f980c1564b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:49166/0 (socket says 172.21.15.67:49166) 2024-04-17T10:25:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f9809e00640 1 -- 172.21.15.67:0/269922388 learned_addr learned my addr 172.21.15.67:0/269922388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f9809400640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0cdb30 0x7f980c14c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980a800640 1 -- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0cdb30 msgr2=0x7f980c14c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980a800640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0cdb30 0x7f980c14c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980a800640 1 -- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c072890 msgr2=0x7f980c1564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980a800640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c072890 0x7f980c1564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980a800640 1 -- 172.21.15.67:0/269922388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c14ced0 con 0x7f980c154c70 2024-04-17T10:25:08.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980a800640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c145640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f980400c700 tx=0x7f980400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f9802000640 1 -- 172.21.15.67:0/269922388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9804003b80 con 0x7f980c154c70 2024-04-17T10:25:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f9802000640 1 -- 172.21.15.67:0/269922388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9804003d20 con 0x7f980c154c70 2024-04-17T10:25:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f980c14d170 con 0x7f980c154c70 2024-04-17T10:25:08.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f9802000640 1 -- 172.21.15.67:0/269922388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f980400aa30 con 0x7f980c154c70 2024-04-17T10:25:08.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.251+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f980c14d730 con 0x7f980c154c70 2024-04-17T10:25:08.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.255+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f980c074dc0 con 0x7f980c154c70 2024-04-17T10:25:08.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.255+0000 7f9802000640 1 -- 172.21.15.67:0/269922388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f980400abd0 con 0x7f980c154c70 2024-04-17T10:25:08.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.255+0000 7f9802000640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97ec066f60 0x7f97ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.255+0000 7f9802000640 1 -- 172.21.15.67:0/269922388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9804085140 con 0x7f980c154c70 2024-04-17T10:25:08.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.255+0000 7f9809e00640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97ec066f60 0x7f97ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.255+0000 7f9809e00640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97ec066f60 0x7f97ec069420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:08.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.259+0000 7f9802000640 1 -- 172.21.15.67:0/269922388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9804055720 con 0x7f980c154c70 2024-04-17T10:25:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.435+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f980c145b80 con 0x7f980c154c70 2024-04-17T10:25:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.435+0000 7f9802000640 1 -- 172.21.15.67:0/269922388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f98040597c0 con 0x7f980c154c70 2024-04-17T10:25:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:25:08.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97ec066f60 msgr2=0x7f97ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97ec066f60 0x7f97ec069420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f97fc009870 tx=0x7f97fc0a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 msgr2=0x7f980c145640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c145640 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f980400c700 tx=0x7f980400cbc0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 shutdown_connections 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97ec066f60 0x7f97ec069420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f980c154c70 0x7f980c145640 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f980c0cdb30 0x7f980c14c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 --2- 172.21.15.67:0/269922388 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f980c072890 0x7f980c1564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 >> 172.21.15.67:0/269922388 conn(0x7f980c0b23f0 msgr2=0x7f980c0cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 shutdown_connections 2024-04-17T10:25:08.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.439+0000 7f980b200640 1 -- 172.21.15.67:0/269922388 wait complete. 2024-04-17T10:25:08.458 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:08.459 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:08.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8fa5200640 1 Processor -- start 2024-04-17T10:25:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8fa5200640 1 -- start start 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8fa5200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8fa015de50 0x7f8fa00cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8fa5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8fa006b6a0 0x7f8fa006f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8fa5200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00700a0 con 0x7f8fa00cf920 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8fa5200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00cad20 con 0x7f8fa006b6a0 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8fa5200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa0070380 con 0x7f8fa015de50 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f95400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8fa006b6a0 0x7f8fa006f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f95400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8fa006b6a0 0x7f8fa006f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43214/0 (socket says 172.21.15.67:43214) 2024-04-17T10:25:08.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f95400640 1 -- 172.21.15.67:0/2755847774 learned_addr learned my addr 172.21.15.67:0/2755847774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f9de00640 1 --2- 172.21.15.67:0/2755847774 >> v2:172.21.15.86:3300/0 conn(0x7f8fa015de50 0x7f8fa00cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f95400640 1 -- 172.21.15.67:0/2755847774 >> v2:172.21.15.86:3300/0 conn(0x7f8fa015de50 msgr2=0x7f8fa00cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f95400640 1 --2- 172.21.15.67:0/2755847774 >> v2:172.21.15.86:3300/0 conn(0x7f8fa015de50 0x7f8fa00cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f95400640 1 -- 172.21.15.67:0/2755847774 >> v1:172.21.15.86:6789/0 conn(0x7f8fa00cf920 legacy=0x7f8fa00cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.655+0000 7f8f95400640 1 -- 172.21.15.67:0/2755847774 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00caf70 con 0x7f8fa006b6a0 2024-04-17T10:25:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8f95400640 1 --2- 172.21.15.67:0/2755847774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa006f970 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00d1770 tx=0x7f8f8801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27ed62c804572a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8f9d400640 1 -- 172.21.15.67:0/2755847774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880b0070 con 0x7f8fa006b6a0 2024-04-17T10:25:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8f9d400640 1 -- 172.21.15.67:0/2755847774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f880ab600 con 0x7f8fa006b6a0 2024-04-17T10:25:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8f9d400640 1 -- 172.21.15.67:0/2755847774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880ab9d0 con 0x7f8fa006b6a0 2024-04-17T10:25:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 -- 172.21.15.67:0/2755847774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 msgr2=0x7f8fa006f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 --2- 172.21.15.67:0/2755847774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa006f970 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00d1770 tx=0x7f8f8801c490 comp rx=0 tx=0).stop 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 -- 172.21.15.67:0/2755847774 shutdown_connections 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 --2- 172.21.15.67:0/2755847774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa006f970 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 --2- 172.21.15.67:0/2755847774 >> v2:172.21.15.86:3300/0 conn(0x7f8fa015de50 0x7f8fa00cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 -- 172.21.15.67:0/2755847774 >> 172.21.15.67:0/2755847774 conn(0x7f8fa00b22a0 msgr2=0x7f8fa00b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 -- 172.21.15.67:0/2755847774 shutdown_connections 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 -- 172.21.15.67:0/2755847774 wait complete. 2024-04-17T10:25:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 Processor -- start 2024-04-17T10:25:08.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.659+0000 7f8fa5200640 1 -- start start 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa00c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fa00cf920 0x7f8fa0156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fa015de50 0x7f8fa00c45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00cc2f0 con 0x7f8fa006b6a0 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa00cc170 con 0x7f8fa00cf920 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa016a7f0 con 0x7f8fa015de50 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa00c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa00c6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43218/0 (socket says 172.21.15.67:43218) 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 -- 172.21.15.67:0/1417560787 learned_addr learned my addr 172.21.15.67:0/1417560787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f95400640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fa00cf920 0x7f8fa0156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9e800640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fa015de50 0x7f8fa00c45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 -- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fa015de50 msgr2=0x7f8fa00c45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fa015de50 0x7f8fa00c45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 -- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fa00cf920 msgr2=0x7f8fa0156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fa00cf920 0x7f8fa0156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 -- 172.21.15.67:0/1417560787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00b5740 con 0x7f8fa006b6a0 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f9de00640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa00c6890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000e930 tx=0x7f8f9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f96a00640 1 -- 172.21.15.67:0/1417560787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f90017070 con 0x7f8fa006b6a0 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f96a00640 1 -- 172.21.15.67:0/1417560787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9000ca00 con 0x7f8fa006b6a0 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f96a00640 1 -- 172.21.15.67:0/1417560787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9000cdb0 con 0x7f8fa006b6a0 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa00b5a20 con 0x7f8fa006b6a0 2024-04-17T10:25:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa00b5fe0 con 0x7f8fa006b6a0 2024-04-17T10:25:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa009e900 con 0x7f8fa006b6a0 2024-04-17T10:25:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.663+0000 7f8f96a00640 1 -- 172.21.15.67:0/1417560787 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8f9001a070 con 0x7f8fa006b6a0 2024-04-17T10:25:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.667+0000 7f8f96a00640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f7c066f60 0x7f8f7c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.667+0000 7f8f96a00640 1 -- 172.21.15.67:0/1417560787 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8f900856d0 con 0x7f8fa006b6a0 2024-04-17T10:25:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.667+0000 7f8f95400640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f7c066f60 0x7f8f7c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.667+0000 7f8f95400640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f7c066f60 0x7f8f7c069420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f880ad000 tx=0x7f8f88008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.671+0000 7f8f96a00640 1 -- 172.21.15.67:0/1417560787 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8f90055e00 con 0x7f8fa006b6a0 2024-04-17T10:25:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.831+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fa015d680 con 0x7f8fa006b6a0 2024-04-17T10:25:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8f96a00640 1 -- 172.21.15.67:0/1417560787 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7f8f90059ea0 con 0x7f8fa006b6a0 2024-04-17T10:25:08.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:08.834 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f7c066f60 msgr2=0x7f8f7c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f7c066f60 0x7f8f7c069420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8f880ad000 tx=0x7f8f88008040 comp rx=0 tx=0).stop 2024-04-17T10:25:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 msgr2=0x7f8fa00c6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa00c6890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8f9000e930 tx=0x7f8f9000edf0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 shutdown_connections 2024-04-17T10:25:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f7c066f60 0x7f8f7c069420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fa015de50 0x7f8fa00c45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fa00cf920 0x7f8fa0156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 --2- 172.21.15.67:0/1417560787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fa006b6a0 0x7f8fa00c6890 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 >> 172.21.15.67:0/1417560787 conn(0x7f8fa00b22a0 msgr2=0x7f8fa00cd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 shutdown_connections 2024-04-17T10:25:08.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.835+0000 7f8fa5200640 1 -- 172.21.15.67:0/1417560787 wait complete. 2024-04-17T10:25:08.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:08.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:08.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4edd400640 1 Processor -- start 2024-04-17T10:25:08.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4edd400640 1 -- start start 2024-04-17T10:25:09.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4edd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ed8071af0 0x7f4ed80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4edd400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4edd400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed815cb30 con 0x7f4ed8159440 2024-04-17T10:25:09.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4edd400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80a8650 con 0x7f4ed8071af0 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4edd400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed815ce10 con 0x7f4ed815be70 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ed6800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ed8159440 0x7f4ed80713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39204/0 (socket says 172.21.15.67:39204) 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ed6800640 1 -- 172.21.15.67:0/639013921 learned_addr learned my addr 172.21.15.67:0/639013921 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ecde00640 1 --2- 172.21.15.67:0/639013921 >> v2:172.21.15.67:3300/0 conn(0x7f4ed8071af0 0x7f4ed80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ed7200640 1 --2- 172.21.15.67:0/639013921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ecde00640 1 -- 172.21.15.67:0/639013921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 msgr2=0x7f4ed815c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ecde00640 1 --2- 172.21.15.67:0/639013921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed815c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ecde00640 1 -- 172.21.15.67:0/639013921 >> v1:172.21.15.67:6789/0 conn(0x7f4ed8159440 legacy=0x7f4ed80713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:08.999+0000 7f4ecde00640 1 -- 172.21.15.67:0/639013921 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80cd610 con 0x7f4ed8071af0 2024-04-17T10:25:09.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4ecde00640 1 --2- 172.21.15.67:0/639013921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8071af0 0x7f4ed80ce0f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097c0 tx=0x7f4ec001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714536ff29281f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4ed5e00640 1 -- 172.21.15.67:0/639013921 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00aa600 con 0x7f4ed8071af0 2024-04-17T10:25:09.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4ed5e00640 1 -- 172.21.15.67:0/639013921 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec00af070 con 0x7f4ed8071af0 2024-04-17T10:25:09.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4ed5e00640 1 -- 172.21.15.67:0/639013921 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00aa9c0 con 0x7f4ed8071af0 2024-04-17T10:25:09.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 -- 172.21.15.67:0/639013921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8071af0 msgr2=0x7f4ed80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 --2- 172.21.15.67:0/639013921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8071af0 0x7f4ed80ce0f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097c0 tx=0x7f4ec001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 -- 172.21.15.67:0/639013921 shutdown_connections 2024-04-17T10:25:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 --2- 172.21.15.67:0/639013921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed815c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 --2- 172.21.15.67:0/639013921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8071af0 0x7f4ed80ce0f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 -- 172.21.15.67:0/639013921 >> 172.21.15.67:0/639013921 conn(0x7f4ed80b1740 msgr2=0x7f4ed80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 -- 172.21.15.67:0/639013921 shutdown_connections 2024-04-17T10:25:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.003+0000 7f4edd400640 1 -- 172.21.15.67:0/639013921 wait complete. 2024-04-17T10:25:09.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 Processor -- start 2024-04-17T10:25:09.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 -- start start 2024-04-17T10:25:09.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed8071af0 0x7f4ed813c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8159440 0x7f4ed815a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed80b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed815d840 con 0x7f4ed8159440 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed815c800 con 0x7f4ed8071af0 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed815d9c0 con 0x7f4ed815be70 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8159440 0x7f4ed815a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8159440 0x7f4ed815a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43238/0 (socket says 172.21.15.67:43238) 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 -- 172.21.15.67:0/2574216125 learned_addr learned my addr 172.21.15.67:0/2574216125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ed6800640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed8071af0 0x7f4ed813c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ed7200640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed80b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 -- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 msgr2=0x7f4ed80b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed80b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 -- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed8071af0 msgr2=0x7f4ed813c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed8071af0 0x7f4ed813c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 -- 172.21.15.67:0/2574216125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed815ad10 con 0x7f4ed8159440 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecde00640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8159440 0x7f4ed815a4a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ec001a0d0 tx=0x7f4ec001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecf400640 1 -- 172.21.15.67:0/2574216125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00be610 con 0x7f4ed8159440 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecf400640 1 -- 172.21.15.67:0/2574216125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec00be7d0 con 0x7f4ed8159440 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4ecf400640 1 -- 172.21.15.67:0/2574216125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00bebc0 con 0x7f4ed8159440 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed815aff0 con 0x7f4ed8159440 2024-04-17T10:25:09.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.007+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed80b2970 con 0x7f4ed8159440 2024-04-17T10:25:09.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.011+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed804ef30 con 0x7f4ed8159440 2024-04-17T10:25:09.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.011+0000 7f4ecf400640 1 -- 172.21.15.67:0/2574216125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4ec00bc020 con 0x7f4ed8159440 2024-04-17T10:25:09.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.011+0000 7f4ecf400640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4eb4066f60 0x7f4eb4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.011+0000 7f4ecf400640 1 -- 172.21.15.67:0/2574216125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4ec011f5c0 con 0x7f4ed8159440 2024-04-17T10:25:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.011+0000 7f4ed6800640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4eb4066f60 0x7f4eb4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.011+0000 7f4ed6800640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4eb4066f60 0x7f4eb4069420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8000f70 tx=0x7f4ec8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.015+0000 7f4ecf400640 1 -- 172.21.15.67:0/2574216125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4ec00c5030 con 0x7f4ed8159440 2024-04-17T10:25:09.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4ed815dda0 con 0x7f4ed8159440 2024-04-17T10:25:09.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4ecf400640 1 -- 172.21.15.67:0/2574216125 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ec0016090 con 0x7f4ed8159440 2024-04-17T10:25:09.179 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4eb4066f60 msgr2=0x7f4eb4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4eb4066f60 0x7f4eb4069420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8000f70 tx=0x7f4ec8009040 comp rx=0 tx=0).stop 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8159440 msgr2=0x7f4ed815a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8159440 0x7f4ed815a4a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4ec001a0d0 tx=0x7f4ec001ace0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 shutdown_connections 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4eb4066f60 0x7f4eb4069420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ed815be70 0x7f4ed80b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ed8159440 0x7f4ed815a4a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 --2- 172.21.15.67:0/2574216125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ed8071af0 0x7f4ed813c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 >> 172.21.15.67:0/2574216125 conn(0x7f4ed80b1740 msgr2=0x7f4ed80afe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 shutdown_connections 2024-04-17T10:25:09.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.179+0000 7f4edd400640 1 -- 172.21.15.67:0/2574216125 wait complete. 2024-04-17T10:25:09.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:09.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.343+0000 7f6926000640 1 Processor -- start 2024-04-17T10:25:09.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.343+0000 7f6926000640 1 -- start start 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f6926000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69200aef60 0x7f69200cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f6926000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f692006e980 0x7f69200b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f6926000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69200b4990 con 0x7f69200cce60 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f6926000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69200af340 con 0x7f692006e980 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f6926000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f692015ae90 con 0x7f69200aef60 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f6916a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f69200cce60 0x7f692006e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39216/0 (socket says 172.21.15.67:39216) 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f6916a00640 1 -- 172.21.15.67:0/427421141 learned_addr learned my addr 172.21.15.67:0/427421141 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f691fe00640 1 --2- 172.21.15.67:0/427421141 >> v2:172.21.15.67:3300/0 conn(0x7f692006e980 0x7f69200b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f691f400640 1 --2- 172.21.15.67:0/427421141 >> v2:172.21.15.86:3300/0 conn(0x7f69200aef60 0x7f69200cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f691fe00640 1 -- 172.21.15.67:0/427421141 >> v2:172.21.15.86:3300/0 conn(0x7f69200aef60 msgr2=0x7f69200cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:09.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f691fe00640 1 --2- 172.21.15.67:0/427421141 >> v2:172.21.15.86:3300/0 conn(0x7f69200aef60 0x7f69200cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f691fe00640 1 -- 172.21.15.67:0/427421141 >> v1:172.21.15.67:6789/0 conn(0x7f69200cce60 legacy=0x7f692006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f691fe00640 1 -- 172.21.15.67:0/427421141 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692006ee80 con 0x7f692006e980 2024-04-17T10:25:09.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.347+0000 7f691f400640 1 --2- 172.21.15.67:0/427421141 >> v2:172.21.15.86:3300/0 conn(0x7f69200aef60 0x7f69200cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f691fe00640 1 --2- 172.21.15.67:0/427421141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692006e980 0x7f69200b3120 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f692006bdc0 tx=0x7f691801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4c4a36a408fc1f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f691ea00640 1 -- 172.21.15.67:0/427421141 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180b2070 con 0x7f692006e980 2024-04-17T10:25:09.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f691ea00640 1 -- 172.21.15.67:0/427421141 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69180a5760 con 0x7f692006e980 2024-04-17T10:25:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f691ea00640 1 -- 172.21.15.67:0/427421141 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180a5b30 con 0x7f692006e980 2024-04-17T10:25:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- 172.21.15.67:0/427421141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692006e980 msgr2=0x7f69200b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 --2- 172.21.15.67:0/427421141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692006e980 0x7f69200b3120 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f692006bdc0 tx=0x7f691801c8a0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- 172.21.15.67:0/427421141 shutdown_connections 2024-04-17T10:25:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 --2- 172.21.15.67:0/427421141 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692006e980 0x7f69200b3120 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 --2- 172.21.15.67:0/427421141 >> v2:172.21.15.86:3300/0 conn(0x7f69200aef60 0x7f69200cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- 172.21.15.67:0/427421141 >> 172.21.15.67:0/427421141 conn(0x7f69200b1740 msgr2=0x7f69200b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- 172.21.15.67:0/427421141 shutdown_connections 2024-04-17T10:25:09.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- 172.21.15.67:0/427421141 wait complete. 2024-04-17T10:25:09.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 Processor -- start 2024-04-17T10:25:09.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- start start 2024-04-17T10:25:09.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f692006e980 0x7f6920154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69200aef60 0x7f692014dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69200cce60 0x7f692014e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69200b3ff0 con 0x7f69200cce60 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69200b3e70 con 0x7f69200aef60 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.351+0000 7f6926000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69201673c0 con 0x7f692006e980 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69200cce60 0x7f692014e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69200cce60 0x7f692014e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43266/0 (socket says 172.21.15.67:43266) 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 -- 172.21.15.67:0/290779034 learned_addr learned my addr 172.21.15.67:0/290779034 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6916a00640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69200aef60 0x7f692014dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691f400640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f692006e980 0x7f6920154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 -- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f692006e980 msgr2=0x7f6920154a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f692006e980 0x7f6920154a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 -- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69200aef60 msgr2=0x7f692014dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69200aef60 0x7f692014dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 -- 172.21.15.67:0/290779034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69201528a0 con 0x7f69200cce60 2024-04-17T10:25:09.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f691fe00640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69200cce60 0x7f692014e160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69180194b0 tx=0x7f6918019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6917e00640 1 -- 172.21.15.67:0/290779034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180b2050 con 0x7f69200cce60 2024-04-17T10:25:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6917e00640 1 -- 172.21.15.67:0/290779034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69180be070 con 0x7f69200cce60 2024-04-17T10:25:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6917e00640 1 -- 172.21.15.67:0/290779034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69180ad890 con 0x7f69200cce60 2024-04-17T10:25:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6920153620 con 0x7f69200cce60 2024-04-17T10:25:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6920152e20 con 0x7f69200cce60 2024-04-17T10:25:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692004ef30 con 0x7f69200cce60 2024-04-17T10:25:09.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.355+0000 7f6917e00640 1 -- 172.21.15.67:0/290779034 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f69180ada30 con 0x7f69200cce60 2024-04-17T10:25:09.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.359+0000 7f6917e00640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f68fc066f60 0x7f68fc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.359+0000 7f6917e00640 1 -- 172.21.15.67:0/290779034 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6918120110 con 0x7f69200cce60 2024-04-17T10:25:09.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.359+0000 7f691f400640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f68fc066f60 0x7f68fc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.359+0000 7f691f400640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f68fc066f60 0x7f68fc069420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6910009580 tx=0x7f6910009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.363+0000 7f6917e00640 1 -- 172.21.15.67:0/290779034 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6918016520 con 0x7f69200cce60 2024-04-17T10:25:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.523+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f692015a300 con 0x7f69200cce60 2024-04-17T10:25:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6917e00640 1 -- 172.21.15.67:0/290779034 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69180f0940 con 0x7f69200cce60 2024-04-17T10:25:09.525 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f68fc066f60 msgr2=0x7f68fc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f68fc066f60 0x7f68fc069420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6910009580 tx=0x7f6910009190 comp rx=0 tx=0).stop 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69200cce60 msgr2=0x7f692014e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69200cce60 0x7f692014e160 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f69180194b0 tx=0x7f6918019640 comp rx=0 tx=0).stop 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 shutdown_connections 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f68fc066f60 0x7f68fc069420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69200cce60 0x7f692014e160 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69200aef60 0x7f692014dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 --2- 172.21.15.67:0/290779034 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f692006e980 0x7f6920154a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 >> 172.21.15.67:0/290779034 conn(0x7f69200b1740 msgr2=0x7f69200cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 shutdown_connections 2024-04-17T10:25:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.527+0000 7f6926000640 1 -- 172.21.15.67:0/290779034 wait complete. 2024-04-17T10:25:09.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc79e000640 1 Processor -- start 2024-04-17T10:25:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc79e000640 1 -- start start 2024-04-17T10:25:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc79e000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7981448a0 0x7fc79806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc79e000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc798144dc0 0x7fc79806b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc79e000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7980d1210 con 0x7fc798145310 2024-04-17T10:25:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc79e000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc798072e50 con 0x7fc7981448a0 2024-04-17T10:25:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc79e000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7980af060 con 0x7fc798144dc0 2024-04-17T10:25:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc797400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7981448a0 0x7fc79806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc78ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc798144dc0 0x7fc79806b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc797400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc7981448a0 0x7fc79806ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:49206/0 (socket says 172.21.15.67:49206) 2024-04-17T10:25:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc78ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc798144dc0 0x7fc79806b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:50164/0 (socket says 172.21.15.67:50164) 2024-04-17T10:25:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc797400640 1 -- 172.21.15.67:0/3992860341 learned_addr learned my addr 172.21.15.67:0/3992860341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc78ea00640 1 -- 172.21.15.67:0/3992860341 >> v2:172.21.15.86:3300/0 conn(0x7fc7981448a0 msgr2=0x7fc79806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc78ea00640 1 --2- 172.21.15.67:0/3992860341 >> v2:172.21.15.86:3300/0 conn(0x7fc7981448a0 0x7fc79806ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc78ea00640 1 -- 172.21.15.67:0/3992860341 >> v1:172.21.15.86:6789/0 conn(0x7fc798145310 legacy=0x7fc79815f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.691+0000 7fc78ea00640 1 -- 172.21.15.67:0/3992860341 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798072b70 con 0x7fc798144dc0 2024-04-17T10:25:09.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc78ea00640 1 --2- 172.21.15.67:0/3992860341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc798144dc0 0x7fc79806b1d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7800097c0 tx=0x7fc78001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0bbb49e23b2d8bd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc796a00640 1 -- 172.21.15.67:0/3992860341 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7800b1070 con 0x7fc798144dc0 2024-04-17T10:25:09.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc796a00640 1 -- 172.21.15.67:0/3992860341 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7800ac410 con 0x7fc798144dc0 2024-04-17T10:25:09.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc796a00640 1 -- 172.21.15.67:0/3992860341 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7800ac7e0 con 0x7fc798144dc0 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 -- 172.21.15.67:0/3992860341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc798144dc0 msgr2=0x7fc79806b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 --2- 172.21.15.67:0/3992860341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc798144dc0 0x7fc79806b1d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc7800097c0 tx=0x7fc78001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 -- 172.21.15.67:0/3992860341 shutdown_connections 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 --2- 172.21.15.67:0/3992860341 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc798144dc0 0x7fc79806b1d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 --2- 172.21.15.67:0/3992860341 >> v2:172.21.15.86:3300/0 conn(0x7fc7981448a0 0x7fc79806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 -- 172.21.15.67:0/3992860341 >> 172.21.15.67:0/3992860341 conn(0x7fc7980b5e50 msgr2=0x7fc798099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 -- 172.21.15.67:0/3992860341 shutdown_connections 2024-04-17T10:25:09.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.695+0000 7fc79e000640 1 -- 172.21.15.67:0/3992860341 wait complete. 2024-04-17T10:25:09.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 Processor -- start 2024-04-17T10:25:09.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 -- start start 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7981448a0 0x7fc79815acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc798144dc0 0x7fc7981435e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc798145310 0x7fc798159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc798168ca0 con 0x7fc798145310 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc798168b20 con 0x7fc798144dc0 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc798168e20 con 0x7fc7981448a0 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc798145310 0x7fc798159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc798145310 0x7fc798159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43280/0 (socket says 172.21.15.67:43280) 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 -- 172.21.15.67:0/2978006505 learned_addr learned my addr 172.21.15.67:0/2978006505 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7981448a0 0x7fc79815acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc78ea00640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc798144dc0 0x7fc7981435e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 -- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7981448a0 msgr2=0x7fc79815acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7981448a0 0x7fc79815acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 -- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc798144dc0 msgr2=0x7fc7981435e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc798144dc0 0x7fc7981435e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 -- 172.21.15.67:0/2978006505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7981595d0 con 0x7fc798145310 2024-04-17T10:25:09.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc797e00640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc798145310 0x7fc798159090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c730 tx=0x7fc79000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc78fe00640 1 -- 172.21.15.67:0/2978006505 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc790003b80 con 0x7fc798145310 2024-04-17T10:25:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc78fe00640 1 -- 172.21.15.67:0/2978006505 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc790003d20 con 0x7fc798145310 2024-04-17T10:25:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc78fe00640 1 -- 172.21.15.67:0/2978006505 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79000aa30 con 0x7fc798145310 2024-04-17T10:25:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc79815b4b0 con 0x7fc798145310 2024-04-17T10:25:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7981f83b0 con 0x7fc798145310 2024-04-17T10:25:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.699+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc79804ef30 con 0x7fc798145310 2024-04-17T10:25:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.703+0000 7fc78fe00640 1 -- 172.21.15.67:0/2978006505 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc79000abd0 con 0x7fc798145310 2024-04-17T10:25:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.703+0000 7fc78fe00640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc774066f60 0x7fc774069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.703+0000 7fc78fe00640 1 -- 172.21.15.67:0/2978006505 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc790085060 con 0x7fc798145310 2024-04-17T10:25:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.703+0000 7fc797400640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc774066f60 0x7fc774069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.703+0000 7fc797400640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc774066f60 0x7fc774069420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc79815b230 tx=0x7fc788009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.707+0000 7fc78fe00640 1 -- 172.21.15.67:0/2978006505 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc7981595d0 con 0x7fc798145310 2024-04-17T10:25:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.867+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc79809e6b0 con 0x7fc798145310 2024-04-17T10:25:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.867+0000 7fc78fe00640 1 -- 172.21.15.67:0/2978006505 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc790055770 con 0x7fc798145310 2024-04-17T10:25:09.868 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:09.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc774066f60 msgr2=0x7fc774069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc774066f60 0x7fc774069420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc79815b230 tx=0x7fc788009240 comp rx=0 tx=0).stop 2024-04-17T10:25:09.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc798145310 msgr2=0x7fc798159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:09.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc798145310 0x7fc798159090 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc79000c730 tx=0x7fc79000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 shutdown_connections 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc774066f60 0x7fc774069420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc798145310 0x7fc798159090 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc798144dc0 0x7fc7981435e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 --2- 172.21.15.67:0/2978006505 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7981448a0 0x7fc79815acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 >> 172.21.15.67:0/2978006505 conn(0x7fc7980b5e50 msgr2=0x7fc79815e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 shutdown_connections 2024-04-17T10:25:09.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:09.871+0000 7fc79e000640 1 -- 172.21.15.67:0/2978006505 wait complete. 2024-04-17T10:25:09.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:10.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb56e00640 1 Processor -- start 2024-04-17T10:25:10.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb56e00640 1 -- start start 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb56e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb500aef60 0x7fcb500cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb56e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb500cce60 0x7fcb5006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb56e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500b4990 con 0x7fcb5006e7b0 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb56e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb500cd2c0 con 0x7fcb500aef60 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb56e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500af340 con 0x7fcb500cce60 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb4fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb500aef60 0x7fcb500cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb4fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb500aef60 0x7fcb500cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43294/0 (socket says 172.21.15.67:43294) 2024-04-17T10:25:10.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb4fe00640 1 -- 172.21.15.67:0/2302062265 learned_addr learned my addr 172.21.15.67:0/2302062265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:10.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb47400640 1 --2- 172.21.15.67:0/2302062265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb500cce60 0x7fcb5006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb4fe00640 1 -- 172.21.15.67:0/2302062265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb500cce60 msgr2=0x7fcb5006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:10.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb4fe00640 1 --2- 172.21.15.67:0/2302062265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb500cce60 0x7fcb5006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb4fe00640 1 -- 172.21.15.67:0/2302062265 >> v1:172.21.15.67:6789/0 conn(0x7fcb5006e7b0 legacy=0x7fcb500713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb4fe00640 1 -- 172.21.15.67:0/2302062265 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5015ae90 con 0x7fcb500aef60 2024-04-17T10:25:10.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.035+0000 7fcb47400640 1 --2- 172.21.15.67:0/2302062265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb500cce60 0x7fcb5006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:10.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb4fe00640 1 --2- 172.21.15.67:0/2302062265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500aef60 0x7fcb500cc920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000bb10 tx=0x7fcb4001eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f5cd9ead473a18 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:10.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb4f400640 1 -- 172.21.15.67:0/2302062265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400b4070 con 0x7fcb500aef60 2024-04-17T10:25:10.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb4f400640 1 -- 172.21.15.67:0/2302062265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb400a7990 con 0x7fcb500aef60 2024-04-17T10:25:10.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb4f400640 1 -- 172.21.15.67:0/2302062265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400a7d60 con 0x7fcb500aef60 2024-04-17T10:25:10.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 -- 172.21.15.67:0/2302062265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500aef60 msgr2=0x7fcb500cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 --2- 172.21.15.67:0/2302062265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500aef60 0x7fcb500cc920 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb4000bb10 tx=0x7fcb4001eba0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 -- 172.21.15.67:0/2302062265 shutdown_connections 2024-04-17T10:25:10.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 --2- 172.21.15.67:0/2302062265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb500cce60 0x7fcb5006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 --2- 172.21.15.67:0/2302062265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500aef60 0x7fcb500cc920 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 -- 172.21.15.67:0/2302062265 >> 172.21.15.67:0/2302062265 conn(0x7fcb500b1740 msgr2=0x7fcb500b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:10.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 -- 172.21.15.67:0/2302062265 shutdown_connections 2024-04-17T10:25:10.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 -- 172.21.15.67:0/2302062265 wait complete. 2024-04-17T10:25:10.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 Processor -- start 2024-04-17T10:25:10.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.039+0000 7fcb56e00640 1 -- start start 2024-04-17T10:25:10.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb5006e7b0 0x7fcb50134cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb500aef60 0x7fcb501373f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500cce60 0x7fcb50159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5015bc10 con 0x7fcb500cce60 2024-04-17T10:25:10.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5015b410 con 0x7fcb500aef60 2024-04-17T10:25:10.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb501358e0 con 0x7fcb5006e7b0 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500cce60 0x7fcb50159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500cce60 0x7fcb50159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43302/0 (socket says 172.21.15.67:43302) 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 -- 172.21.15.67:0/3037425366 learned_addr learned my addr 172.21.15.67:0/3037425366 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb4fe00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb5006e7b0 0x7fcb50134cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb47400640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb500aef60 0x7fcb501373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 -- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb5006e7b0 msgr2=0x7fcb50134cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb5006e7b0 0x7fcb50134cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 -- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb500aef60 msgr2=0x7fcb501373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb500aef60 0x7fcb501373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 -- 172.21.15.67:0/3037425366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb50137eb0 con 0x7fcb500cce60 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb47400640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb500aef60 0x7fcb501373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb54e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500cce60 0x7fcb50159850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c760 tx=0x7fcb4800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb4cc00640 1 -- 172.21.15.67:0/3037425366 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb48003b80 con 0x7fcb500cce60 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb4cc00640 1 -- 172.21.15.67:0/3037425366 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb48003d20 con 0x7fcb500cce60 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb4cc00640 1 -- 172.21.15.67:0/3037425366 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4800aa30 con 0x7fcb500cce60 2024-04-17T10:25:10.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb50138190 con 0x7fcb500cce60 2024-04-17T10:25:10.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5013b620 con 0x7fcb500cce60 2024-04-17T10:25:10.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.043+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb50135360 con 0x7fcb500cce60 2024-04-17T10:25:10.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.047+0000 7fcb4cc00640 1 -- 172.21.15.67:0/3037425366 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcb4800abd0 con 0x7fcb500cce60 2024-04-17T10:25:10.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.047+0000 7fcb4cc00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb2c066f90 0x7fcb2c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.047+0000 7fcb4cc00640 1 -- 172.21.15.67:0/3037425366 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fcb48085180 con 0x7fcb500cce60 2024-04-17T10:25:10.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.047+0000 7fcb4fe00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb2c066f90 0x7fcb2c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.047+0000 7fcb4fe00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb2c066f90 0x7fcb2c069450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb400b1000 tx=0x7fcb40008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:10.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.051+0000 7fcb4cc00640 1 -- 172.21.15.67:0/3037425366 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcb500b4990 con 0x7fcb500cce60 2024-04-17T10:25:10.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.211+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcb5004ef30 con 0x7fcb500cce60 2024-04-17T10:25:10.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.211+0000 7fcb4cc00640 1 -- 172.21.15.67:0/3037425366 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcb48055db0 con 0x7fcb500cce60 2024-04-17T10:25:10.212 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:10.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb2c066f90 msgr2=0x7fcb2c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb2c066f90 0x7fcb2c069450 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcb400b1000 tx=0x7fcb40008040 comp rx=0 tx=0).stop 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500cce60 msgr2=0x7fcb50159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500cce60 0x7fcb50159850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcb4800c760 tx=0x7fcb4800cc20 comp rx=0 tx=0).stop 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 shutdown_connections 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb2c066f90 0x7fcb2c069450 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb500cce60 0x7fcb50159850 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb500aef60 0x7fcb501373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 --2- 172.21.15.67:0/3037425366 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb5006e7b0 0x7fcb50134cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 >> 172.21.15.67:0/3037425366 conn(0x7fcb500b1740 msgr2=0x7fcb5015bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 shutdown_connections 2024-04-17T10:25:10.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.215+0000 7fcb56e00640 1 -- 172.21.15.67:0/3037425366 wait complete. 2024-04-17T10:25:10.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:25:10.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1368400640 1 Processor -- start 2024-04-17T10:25:10.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1368400640 1 -- start start 2024-04-17T10:25:10.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1368400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1360145040 0x7f136006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1368400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f136006ef10 0x7f1360071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1368400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136006e2f0 con 0x7f1360071850 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1368400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13600a86c0 con 0x7f1360145040 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1368400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136015bf10 con 0x7f136006ef10 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1360145040 0x7f136006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1360145040 0x7f136006e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57958/0 (socket says 172.21.15.67:57958) 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365a00640 1 -- 172.21.15.67:0/3555676324 learned_addr learned my addr 172.21.15.67:0/3555676324 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365000640 1 --2- 172.21.15.67:0/3555676324 >> v2:172.21.15.86:3300/0 conn(0x7f136006ef10 0x7f1360071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365a00640 1 -- 172.21.15.67:0/3555676324 >> v2:172.21.15.86:3300/0 conn(0x7f136006ef10 msgr2=0x7f1360071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365a00640 1 --2- 172.21.15.67:0/3555676324 >> v2:172.21.15.86:3300/0 conn(0x7f136006ef10 0x7f1360071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365a00640 1 -- 172.21.15.67:0/3555676324 >> v1:172.21.15.86:6789/0 conn(0x7f1360071850 legacy=0x7f136015f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.375+0000 7f1365a00640 1 -- 172.21.15.67:0/3555676324 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360071c60 con 0x7f1360145040 2024-04-17T10:25:10.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1365a00640 1 --2- 172.21.15.67:0/3555676324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1360145040 0x7f136006e9d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f13600727b0 tx=0x7f135401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea108a5c9f991677 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f135fe00640 1 -- 172.21.15.67:0/3555676324 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13540b5070 con 0x7f1360145040 2024-04-17T10:25:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f135fe00640 1 -- 172.21.15.67:0/3555676324 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13540a8a10 con 0x7f1360145040 2024-04-17T10:25:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f135fe00640 1 -- 172.21.15.67:0/3555676324 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13540a8de0 con 0x7f1360145040 2024-04-17T10:25:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 -- 172.21.15.67:0/3555676324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1360145040 msgr2=0x7f136006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 --2- 172.21.15.67:0/3555676324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1360145040 0x7f136006e9d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f13600727b0 tx=0x7f135401ed10 comp rx=0 tx=0).stop 2024-04-17T10:25:10.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 -- 172.21.15.67:0/3555676324 shutdown_connections 2024-04-17T10:25:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 --2- 172.21.15.67:0/3555676324 >> v2:172.21.15.86:3300/0 conn(0x7f136006ef10 0x7f1360071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 --2- 172.21.15.67:0/3555676324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1360145040 0x7f136006e9d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 -- 172.21.15.67:0/3555676324 >> 172.21.15.67:0/3555676324 conn(0x7f136006ac90 msgr2=0x7f136006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 -- 172.21.15.67:0/3555676324 shutdown_connections 2024-04-17T10:25:10.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.379+0000 7f1368400640 1 -- 172.21.15.67:0/3555676324 wait complete. 2024-04-17T10:25:10.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 Processor -- start 2024-04-17T10:25:10.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 -- start start 2024-04-17T10:25:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136006ef10 0x7f136015b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1360071850 0x7f13601f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1360145040 0x7f136015d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136006df80 con 0x7f136006ef10 2024-04-17T10:25:10.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136006dca0 con 0x7f1360145040 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f136006d370 con 0x7f1360071850 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136006ef10 0x7f136015b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136006ef10 0x7f136015b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57968/0 (socket says 172.21.15.67:57968) 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 -- 172.21.15.67:0/2422434792 learned_addr learned my addr 172.21.15.67:0/2422434792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1366400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1360145040 0x7f136015d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365000640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1360071850 0x7f13601f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 -- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1360071850 msgr2=0x7f13601f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1360071850 0x7f13601f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 -- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1360145040 msgr2=0x7f136015d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1360145040 0x7f136015d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 -- 172.21.15.67:0/2422434792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13600b28e0 con 0x7f136006ef10 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1366400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1360145040 0x7f136015d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1365a00640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136006ef10 0x7f136015b060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f135401b920 tx=0x7f135401bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f135d600640 1 -- 172.21.15.67:0/2422434792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13540b5070 con 0x7f136006ef10 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f135d600640 1 -- 172.21.15.67:0/2422434792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13540b0b30 con 0x7f136006ef10 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13600b2b80 con 0x7f136006ef10 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f135d600640 1 -- 172.21.15.67:0/2422434792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13540bb410 con 0x7f136006ef10 2024-04-17T10:25:10.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.383+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136015c800 con 0x7f136006ef10 2024-04-17T10:25:10.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.387+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1360072d50 con 0x7f136006ef10 2024-04-17T10:25:10.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.387+0000 7f135d600640 1 -- 172.21.15.67:0/2422434792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f13540c2060 con 0x7f136006ef10 2024-04-17T10:25:10.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.387+0000 7f135d600640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1340066f60 0x7f1340069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:10.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.387+0000 7f135d600640 1 -- 172.21.15.67:0/2422434792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f1354122550 con 0x7f136006ef10 2024-04-17T10:25:10.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.387+0000 7f1365000640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1340066f60 0x7f1340069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:10.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.387+0000 7f1365000640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1340066f60 0x7f1340069420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f136013f160 tx=0x7f134c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:10.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.391+0000 7f135d600640 1 -- 172.21.15.67:0/2422434792 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1360071c60 con 0x7f136006ef10 2024-04-17T10:25:10.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.563+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1360074e30 con 0x7f136006ef10 2024-04-17T10:25:10.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.563+0000 7f135d600640 1 -- 172.21.15.67:0/2422434792 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1354018870 con 0x7f136006ef10 2024-04-17T10:25:10.564 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:25:10.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.563+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1340066f60 msgr2=0x7f1340069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.563+0000 7f1368400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1340066f60 0x7f1340069420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f136013f160 tx=0x7f134c008040 comp rx=0 tx=0).stop 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.563+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136006ef10 msgr2=0x7f136015b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.563+0000 7f1368400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136006ef10 0x7f136015b060 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f135401b920 tx=0x7f135401bab0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 shutdown_connections 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1340066f60 0x7f1340069420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1360145040 0x7f136015d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1360071850 0x7f13601f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 --2- 172.21.15.67:0/2422434792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f136006ef10 0x7f136015b060 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 >> 172.21.15.67:0/2422434792 conn(0x7f136006ac90 msgr2=0x7f136013f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:10.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 shutdown_connections 2024-04-17T10:25:10.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:10.567+0000 7f1368400640 1 -- 172.21.15.67:0/2422434792 wait complete. 2024-04-17T10:25:10.584 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi067.front.sepia.ceph.com... 2024-04-17T10:25:10.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:10.584 DEBUG:teuthology.orchestra.run.smithi067:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-17T10:25:10.613 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi067.stderr:ceph-fuse[33691]: fuse finished with error 0 and tester_r 0 2024-04-17T10:25:10.617 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:25:16.719 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:25:16.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:16.719 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-17T10:25:16.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:16.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.931+0000 7f888ea00640 1 Processor -- start 2024-04-17T10:25:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.931+0000 7f888ea00640 1 -- start start 2024-04-17T10:25:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.931+0000 7f888ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 0x7f8888071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.931+0000 7f888ea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888806e2f0 con 0x7f88880cc300 2024-04-17T10:25:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.931+0000 7f888ea00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888806d6e0 con 0x7f8888145040 2024-04-17T10:25:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.931+0000 7f888ea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88880a86c0 con 0x7f8888071310 2024-04-17T10:25:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f888ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f88880cc300 0x7f88880cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52536/0 (socket says 172.21.15.67:52536) 2024-04-17T10:25:16.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f888ca00640 1 -- 172.21.15.67:0/2871448569 learned_addr learned my addr 172.21.15.67:0/2871448569 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:16.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f887fe00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 0x7f8888071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292523535 0 0) 0x7f888806e2f0 con 0x7f88880cc300 2024-04-17T10:25:16.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c003530 con 0x7f88880cc300 2024-04-17T10:25:16.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57508871 0 0) 0x7f888806d6e0 con 0x7f8888145040 2024-04-17T10:25:16.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888806e2f0 con 0x7f8888145040 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3266375738 0 0) 0x7f886c003530 con 0x7f88880cc300 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 msgr2=0x7f8888071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 0x7f8888071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 >> v1:172.21.15.86:6789/0 conn(0x7f8888145040 legacy=0x7f888806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888806d9c0 con 0x7f88880cc300 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f8880001a00 con 0x7f88880cc300 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 >> v1:172.21.15.67:6789/0 conn(0x7f88880cc300 legacy=0x7f88880cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f886c003b80 0x7f886c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f886c004810 0x7f886c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 0x7f886c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00ba80 con 0x7f886c003b80 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00bc00 con 0x7f886c004810 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f886c00be00 con 0x7f8888071310 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887e00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f886c003b80 0x7f886c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f887fe00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f886c004810 0x7f886c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f888ca00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 0x7f886c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887e00640 1 -- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 msgr2=0x7f886c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887e00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 0x7f886c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887e00640 1 -- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f886c004810 msgr2=0x7f886c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887e00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f886c004810 0x7f886c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f8887e00640 1 -- 172.21.15.67:0/2871448569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f886c003530 con 0x7f886c003b80 2024-04-17T10:25:16.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.935+0000 7f888ca00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8888071310 0x7f886c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f8887e00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f886c003b80 0x7f886c003f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f88780018d0 tx=0x7f887801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c6a9cd8e5151296 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8878001520 con 0x7f886c003b80 2024-04-17T10:25:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88780052f0 con 0x7f886c003b80 2024-04-17T10:25:16.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f8887400640 1 -- 172.21.15.67:0/2871448569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88780a8950 con 0x7f886c003b80 2024-04-17T10:25:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 -- 172.21.15.67:0/2871448569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f886c003b80 msgr2=0x7f886c003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 --2- 172.21.15.67:0/2871448569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f886c003b80 0x7f886c003f80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f88780018d0 tx=0x7f887801ecf0 comp rx=0 tx=0).stop 2024-04-17T10:25:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ca00640 1 -- 172.21.15.67:0/2871448569 reap_dead start 2024-04-17T10:25:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 -- 172.21.15.67:0/2871448569 shutdown_connections 2024-04-17T10:25:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 -- 172.21.15.67:0/2871448569 >> 172.21.15.67:0/2871448569 conn(0x7f888806ac90 msgr2=0x7f888806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 -- 172.21.15.67:0/2871448569 shutdown_connections 2024-04-17T10:25:16.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 -- 172.21.15.67:0/2871448569 wait complete. 2024-04-17T10:25:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 Processor -- start 2024-04-17T10:25:16.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.939+0000 7f888ea00640 1 -- start start 2024-04-17T10:25:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f886c003b80 0x7f8888142540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f888815ac40 0x7f8888143c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888813c1f0 0x7f88880b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888806df80 con 0x7f888813c1f0 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888815bf10 con 0x7f888815ac40 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888806d370 con 0x7f886c003b80 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888813c1f0 0x7f88880b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888813c1f0 0x7f88880b3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57978/0 (socket says 172.21.15.67:57978) 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 -- 172.21.15.67:0/602698050 learned_addr learned my addr 172.21.15.67:0/602698050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f887fe00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f888815ac40 0x7f8888143c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f8887e00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f886c003b80 0x7f8888142540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 -- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f886c003b80 msgr2=0x7f8888142540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f886c003b80 0x7f8888142540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 -- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f888815ac40 msgr2=0x7f8888143c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f888815ac40 0x7f8888143c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 -- 172.21.15.67:0/602698050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8888142b90 con 0x7f888813c1f0 2024-04-17T10:25:16.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ca00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888813c1f0 0x7f88880b3e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f88880b2530 tx=0x7f8880007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f8884c00640 1 -- 172.21.15.67:0/602698050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888000eb00 con 0x7f888813c1f0 2024-04-17T10:25:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f8884c00640 1 -- 172.21.15.67:0/602698050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888000eca0 con 0x7f888813c1f0 2024-04-17T10:25:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8888142e70 con 0x7f888813c1f0 2024-04-17T10:25:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f8884c00640 1 -- 172.21.15.67:0/602698050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888000ee90 con 0x7f888813c1f0 2024-04-17T10:25:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8888143430 con 0x7f888813c1f0 2024-04-17T10:25:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8888072d50 con 0x7f888813c1f0 2024-04-17T10:25:16.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.943+0000 7f8884c00640 1 -- 172.21.15.67:0/602698050 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f888001a020 con 0x7f888813c1f0 2024-04-17T10:25:16.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.947+0000 7f8884c00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8864067100 0x7f88640695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.947+0000 7f8884c00640 1 -- 172.21.15.67:0/602698050 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8880084dc0 con 0x7f888813c1f0 2024-04-17T10:25:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.947+0000 7f8887e00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8864067100 0x7f88640695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:16.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.947+0000 7f8887e00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8864067100 0x7f88640695c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f88780b0000 tx=0x7f88780aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:16.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:16.951+0000 7f8884c00640 1 -- 172.21.15.67:0/602698050 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8880021020 con 0x7f888813c1f0 2024-04-17T10:25:17.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.111+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88880cef10 con 0x7f888813c1f0 2024-04-17T10:25:17.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f8884c00640 1 -- 172.21.15.67:0/602698050 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7f88800559f0 con 0x7f888813c1f0 2024-04-17T10:25:17.114 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:17.114 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8864067100 msgr2=0x7f88640695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8864067100 0x7f88640695c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f88780b0000 tx=0x7f88780aa070 comp rx=0 tx=0).stop 2024-04-17T10:25:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888813c1f0 msgr2=0x7f88880b3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888813c1f0 0x7f88880b3e40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f88880b2530 tx=0x7f8880007bc0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 shutdown_connections 2024-04-17T10:25:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8864067100 0x7f88640695c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f888813c1f0 0x7f88880b3e40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f888815ac40 0x7f8888143c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 --2- 172.21.15.67:0/602698050 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f886c003b80 0x7f8888142540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 >> 172.21.15.67:0/602698050 conn(0x7f888806ac90 msgr2=0x7f888806e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 shutdown_connections 2024-04-17T10:25:17.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.115+0000 7f888ea00640 1 -- 172.21.15.67:0/602698050 wait complete. 2024-04-17T10:25:17.136 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.275+0000 7f7b13600640 1 Processor -- start 2024-04-17T10:25:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.275+0000 7f7b13600640 1 -- start start 2024-04-17T10:25:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b13600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c1620a0 0x7f7b0c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b13600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0c15eb50 con 0x7f7b0c0ce020 2024-04-17T10:25:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b13600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0c15ee30 con 0x7f7b0c15e550 2024-04-17T10:25:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b13600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c161650 con 0x7f7b0c1620a0 2024-04-17T10:25:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b10c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b0c0ce020 0x7f7b0c0ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52538/0 (socket says 172.21.15.67:52538) 2024-04-17T10:25:17.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b10c00640 1 -- 172.21.15.67:0/2057323072 learned_addr learned my addr 172.21.15.67:0/2057323072 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:17.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b03e00640 1 --2- 172.21.15.67:0/2057323072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c1620a0 0x7f7b0c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b0be00640 1 -- 172.21.15.67:0/2057323072 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430381668 0 0) 0x7f7b0c15eb50 con 0x7f7b0c0ce020 2024-04-17T10:25:17.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b0be00640 1 -- 172.21.15.67:0/2057323072 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af0003530 con 0x7f7b0c0ce020 2024-04-17T10:25:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b0be00640 1 -- 172.21.15.67:0/2057323072 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787779426 0 0) 0x7f7b0c15ee30 con 0x7f7b0c15e550 2024-04-17T10:25:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b0be00640 1 -- 172.21.15.67:0/2057323072 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c15eb50 con 0x7f7b0c15e550 2024-04-17T10:25:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b03e00640 1 -- 172.21.15.67:0/2057323072 >> v1:172.21.15.86:6789/0 conn(0x7f7b0c15e550 legacy=0x7f7b0c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b03e00640 1 -- 172.21.15.67:0/2057323072 >> v1:172.21.15.67:6789/0 conn(0x7f7b0c0ce020 legacy=0x7f7b0c0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.279+0000 7f7b03e00640 1 -- 172.21.15.67:0/2057323072 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c1618a0 con 0x7f7b0c1620a0 2024-04-17T10:25:17.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b03e00640 1 --2- 172.21.15.67:0/2057323072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c1620a0 0x7f7b0c15e010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7af40097f0 tx=0x7f7af401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeedb74fb58872b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:17.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b0be00640 1 -- 172.21.15.67:0/2057323072 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af40b2070 con 0x7f7b0c1620a0 2024-04-17T10:25:17.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b0be00640 1 -- 172.21.15.67:0/2057323072 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af40ad680 con 0x7f7b0c1620a0 2024-04-17T10:25:17.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b0be00640 1 -- 172.21.15.67:0/2057323072 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af40ada50 con 0x7f7b0c1620a0 2024-04-17T10:25:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 -- 172.21.15.67:0/2057323072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c1620a0 msgr2=0x7f7b0c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 --2- 172.21.15.67:0/2057323072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c1620a0 0x7f7b0c15e010 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7af40097f0 tx=0x7f7af401c470 comp rx=0 tx=0).stop 2024-04-17T10:25:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 -- 172.21.15.67:0/2057323072 shutdown_connections 2024-04-17T10:25:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 --2- 172.21.15.67:0/2057323072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c1620a0 0x7f7b0c15e010 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 -- 172.21.15.67:0/2057323072 >> 172.21.15.67:0/2057323072 conn(0x7f7b0c0b33f0 msgr2=0x7f7b0c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 -- 172.21.15.67:0/2057323072 shutdown_connections 2024-04-17T10:25:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 -- 172.21.15.67:0/2057323072 wait complete. 2024-04-17T10:25:17.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.283+0000 7f7b13600640 1 Processor -- start 2024-04-17T10:25:17.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 -- start start 2024-04-17T10:25:17.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0ce020 0x7f7b0c15c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c15e550 0x7f7b0c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b0c1620a0 0x7f7b0c15f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c0c8630 con 0x7f7b0c0ce020 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c0c84b0 con 0x7f7b0c1620a0 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c16ba00 con 0x7f7b0c15e550 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0ce020 0x7f7b0c15c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0ce020 0x7f7b0c15c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57980/0 (socket says 172.21.15.67:57980) 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 -- 172.21.15.67:0/2505307892 learned_addr learned my addr 172.21.15.67:0/2505307892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b11600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b0c1620a0 0x7f7b0c15f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b03e00640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c15e550 0x7f7b0c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 -- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c15e550 msgr2=0x7f7b0c1509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c15e550 0x7f7b0c1509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 -- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b0c1620a0 msgr2=0x7f7b0c15f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b0c1620a0 0x7f7b0c15f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 -- 172.21.15.67:0/2505307892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c151ea0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b10c00640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0ce020 0x7f7b0c15c620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00c450 tx=0x7f7afc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b09600640 1 -- 172.21.15.67:0/2505307892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc017070 con 0x7f7b0c0ce020 2024-04-17T10:25:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b09600640 1 -- 172.21.15.67:0/2505307892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc005cf0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c152180 con 0x7f7b0c0ce020 2024-04-17T10:25:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b09600640 1 -- 172.21.15.67:0/2505307892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc0072b0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c152740 con 0x7f7b0c0ce020 2024-04-17T10:25:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.287+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c14c8d0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.291+0000 7f7b09600640 1 -- 172.21.15.67:0/2505307892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7afc01a070 con 0x7f7b0c0ce020 2024-04-17T10:25:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.291+0000 7f7b09600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ae8066f60 0x7f7ae8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.291+0000 7f7b09600640 1 -- 172.21.15.67:0/2505307892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7afc0847b0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.291+0000 7f7b03e00640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ae8066f60 0x7f7ae8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.291+0000 7f7b03e00640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ae8066f60 0x7f7ae8069420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7af40af000 tx=0x7f7af40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:17.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.295+0000 7f7b09600640 1 -- 172.21.15.67:0/2505307892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7afc0553e0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.455+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b0c070ab0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b09600640 1 -- 172.21.15.67:0/2505307892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7f7afc054ce0 con 0x7f7b0c0ce020 2024-04-17T10:25:17.457 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:17.457 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ae8066f60 msgr2=0x7f7ae8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ae8066f60 0x7f7ae8069420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7af40af000 tx=0x7f7af40a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0ce020 msgr2=0x7f7b0c15c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0ce020 0x7f7b0c15c620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7afc00c450 tx=0x7f7afc00c910 comp rx=0 tx=0).stop 2024-04-17T10:25:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 shutdown_connections 2024-04-17T10:25:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ae8066f60 0x7f7ae8069420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b0c1620a0 0x7f7b0c15f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b0c15e550 0x7f7b0c1509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 --2- 172.21.15.67:0/2505307892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b0c0ce020 0x7f7b0c15c620 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 >> 172.21.15.67:0/2505307892 conn(0x7f7b0c0b33f0 msgr2=0x7f7b0c073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 shutdown_connections 2024-04-17T10:25:17.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.459+0000 7f7b13600640 1 -- 172.21.15.67:0/2505307892 wait complete. 2024-04-17T10:25:17.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:25:17.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0165400640 1 Processor -- start 2024-04-17T10:25:17.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0165400640 1 -- start start 2024-04-17T10:25:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0165400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f01600cd030 0x7f016006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0165400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f016006e7b0 0x7f01600b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0165400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01600b4990 con 0x7f01600aef60 2024-04-17T10:25:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0165400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01600af340 con 0x7f01600cd030 2024-04-17T10:25:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0165400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016006ec00 con 0x7f016006e7b0 2024-04-17T10:25:17.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0157e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f01600cd030 0x7f016006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0157e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f01600cd030 0x7f016006e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57994/0 (socket says 172.21.15.67:57994) 2024-04-17T10:25:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0157e00640 1 -- 172.21.15.67:0/1808068913 learned_addr learned my addr 172.21.15.67:0/1808068913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f015f200640 1 --2- 172.21.15.67:0/1808068913 >> v2:172.21.15.86:3300/0 conn(0x7f016006e7b0 0x7f01600b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0157e00640 1 -- 172.21.15.67:0/1808068913 >> v2:172.21.15.86:3300/0 conn(0x7f016006e7b0 msgr2=0x7f01600b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0157e00640 1 --2- 172.21.15.67:0/1808068913 >> v2:172.21.15.86:3300/0 conn(0x7f016006e7b0 0x7f01600b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0157e00640 1 -- 172.21.15.67:0/1808068913 >> v1:172.21.15.67:6789/0 conn(0x7f01600aef60 legacy=0x7f01600cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.623+0000 7f0157e00640 1 -- 172.21.15.67:0/1808068913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016015ae90 con 0x7f01600cd030 2024-04-17T10:25:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0157e00640 1 --2- 172.21.15.67:0/1808068913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600cd030 0x7f016006e270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f01600b2e60 tx=0x7f014801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1a1311554b9fab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f015de00640 1 -- 172.21.15.67:0/1808068913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480b2070 con 0x7f01600cd030 2024-04-17T10:25:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f015de00640 1 -- 172.21.15.67:0/1808068913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01480ad680 con 0x7f01600cd030 2024-04-17T10:25:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f015de00640 1 -- 172.21.15.67:0/1808068913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480ada50 con 0x7f01600cd030 2024-04-17T10:25:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 -- 172.21.15.67:0/1808068913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600cd030 msgr2=0x7f016006e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 --2- 172.21.15.67:0/1808068913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600cd030 0x7f016006e270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f01600b2e60 tx=0x7f014801c490 comp rx=0 tx=0).stop 2024-04-17T10:25:17.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 -- 172.21.15.67:0/1808068913 shutdown_connections 2024-04-17T10:25:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 --2- 172.21.15.67:0/1808068913 >> v2:172.21.15.86:3300/0 conn(0x7f016006e7b0 0x7f01600b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 --2- 172.21.15.67:0/1808068913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600cd030 0x7f016006e270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 -- 172.21.15.67:0/1808068913 >> 172.21.15.67:0/1808068913 conn(0x7f01600b1740 msgr2=0x7f01600b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 -- 172.21.15.67:0/1808068913 shutdown_connections 2024-04-17T10:25:17.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.627+0000 7f0165400640 1 -- 172.21.15.67:0/1808068913 wait complete. 2024-04-17T10:25:17.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 Processor -- start 2024-04-17T10:25:17.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 -- start start 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016006e7b0 0x7f0160157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600aef60 0x7f016013b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600cd030 0x7f0160150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016015bc10 con 0x7f01600aef60 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01600b3e70 con 0x7f016006e7b0 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016013b600 con 0x7f01600cd030 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600aef60 0x7f016013b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600aef60 0x7f016013b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58000/0 (socket says 172.21.15.67:58000) 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 -- 172.21.15.67:0/2062216217 learned_addr learned my addr 172.21.15.67:0/2062216217 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f015f200640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600cd030 0x7f0160150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f015e800640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016006e7b0 0x7f0160157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 -- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600cd030 msgr2=0x7f0160150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600cd030 0x7f0160150580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 -- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016006e7b0 msgr2=0x7f0160157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016006e7b0 0x7f0160157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 -- 172.21.15.67:0/2062216217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0160155440 con 0x7f01600aef60 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0157e00640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600aef60 0x7f016013b0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f014801a0c0 tx=0x7f014801acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0156a00640 1 -- 172.21.15.67:0/2062216217 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480b2040 con 0x7f01600aef60 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0156a00640 1 -- 172.21.15.67:0/2062216217 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01480adc60 con 0x7f01600aef60 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f016014a040 con 0x7f01600aef60 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0156a00640 1 -- 172.21.15.67:0/2062216217 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480b8670 con 0x7f01600aef60 2024-04-17T10:25:17.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.631+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016014a600 con 0x7f01600aef60 2024-04-17T10:25:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.635+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016004ef30 con 0x7f01600aef60 2024-04-17T10:25:17.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.635+0000 7f0156a00640 1 -- 172.21.15.67:0/2062216217 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f01480bf030 con 0x7f01600aef60 2024-04-17T10:25:17.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.635+0000 7f0156a00640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f013c066f60 0x7f013c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:17.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.635+0000 7f0156a00640 1 -- 172.21.15.67:0/2062216217 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f014811eba0 con 0x7f01600aef60 2024-04-17T10:25:17.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.635+0000 7f015e800640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f013c066f60 0x7f013c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:17.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.635+0000 7f015e800640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f013c066f60 0x7f013c069420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0150001650 tx=0x7f0150009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.639+0000 7f0156a00640 1 -- 172.21.15.67:0/2062216217 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0148016110 con 0x7f01600aef60 2024-04-17T10:25:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:17.835+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f016006da90 con 0x7f01600aef60 2024-04-17T10:25:18.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.683+0000 7f0156a00640 1 -- 172.21.15.67:0/2062216217 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f01480ef860 con 0x7f01600aef60 2024-04-17T10:25:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f013c066f60 msgr2=0x7f013c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f013c066f60 0x7f013c069420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0150001650 tx=0x7f0150009040 comp rx=0 tx=0).stop 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600aef60 msgr2=0x7f016013b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600aef60 0x7f016013b0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f014801a0c0 tx=0x7f014801acd0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 shutdown_connections 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f013c066f60 0x7f013c069420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01600cd030 0x7f0160150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01600aef60 0x7f016013b0c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 --2- 172.21.15.67:0/2062216217 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f016006e7b0 0x7f0160157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 >> 172.21.15.67:0/2062216217 conn(0x7f01600b1740 msgr2=0x7f016006dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 shutdown_connections 2024-04-17T10:25:18.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.687+0000 7f0165400640 1 -- 172.21.15.67:0/2062216217 wait complete. 2024-04-17T10:25:18.710 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-17T10:25:18.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:25:18.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3b2a00640 1 Processor -- start 2024-04-17T10:25:18.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3b2a00640 1 -- start start 2024-04-17T10:25:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3b2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac159440 0x7fe3ac0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3b2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3ac0713e0 0x7fe3ac071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3b2a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac15cb30 con 0x7fe3ac15be70 2024-04-17T10:25:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3b2a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ac0a8650 con 0x7fe3ac0713e0 2024-04-17T10:25:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3b2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac071d80 con 0x7fe3ac159440 2024-04-17T10:25:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3ab400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3ac0713e0 0x7fe3ac071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3ab400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe3ac0713e0 0x7fe3ac071840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58016/0 (socket says 172.21.15.67:58016) 2024-04-17T10:25:18.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3ab400640 1 -- 172.21.15.67:0/3132937913 learned_addr learned my addr 172.21.15.67:0/3132937913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:18.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3abe00640 1 --2- 172.21.15.67:0/3132937913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac159440 0x7fe3ac0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:18.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3ab400640 1 -- 172.21.15.67:0/3132937913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac159440 msgr2=0x7fe3ac0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:18.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3ab400640 1 --2- 172.21.15.67:0/3132937913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac159440 0x7fe3ac0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3ab400640 1 -- 172.21.15.67:0/3132937913 >> v1:172.21.15.86:6789/0 conn(0x7fe3ac15be70 legacy=0x7fe3ac15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:18.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.859+0000 7fe3ab400640 1 -- 172.21.15.67:0/3132937913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac0cd610 con 0x7fe3ac0713e0 2024-04-17T10:25:18.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3ab400640 1 --2- 172.21.15.67:0/3132937913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac0713e0 0x7fe3ac071840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe398009870 tx=0x7fe39801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899bf54095973ed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:18.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3aaa00640 1 -- 172.21.15.67:0/3132937913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980a7070 con 0x7fe3ac0713e0 2024-04-17T10:25:18.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3aaa00640 1 -- 172.21.15.67:0/3132937913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3980b0070 con 0x7fe3ac0713e0 2024-04-17T10:25:18.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3aaa00640 1 -- 172.21.15.67:0/3132937913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980ab890 con 0x7fe3ac0713e0 2024-04-17T10:25:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3132937913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac0713e0 msgr2=0x7fe3ac071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3132937913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac0713e0 0x7fe3ac071840 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe398009870 tx=0x7fe39801c560 comp rx=0 tx=0).stop 2024-04-17T10:25:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3132937913 shutdown_connections 2024-04-17T10:25:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3132937913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac0713e0 0x7fe3ac071840 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3132937913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac159440 0x7fe3ac0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3132937913 >> 172.21.15.67:0/3132937913 conn(0x7fe3ac0b1740 msgr2=0x7fe3ac0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:18.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3132937913 shutdown_connections 2024-04-17T10:25:18.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.863+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3132937913 wait complete. 2024-04-17T10:25:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 Processor -- start 2024-04-17T10:25:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 -- start start 2024-04-17T10:25:18.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3ac0713e0 0x7fe3ac0ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:18.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac159440 0x7fe3ac0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:18.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac15be70 0x7fe3ac1570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:18.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac15d9c0 con 0x7fe3ac159440 2024-04-17T10:25:18.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac15d840 con 0x7fe3ac0713e0 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac15db40 con 0x7fe3ac15be70 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac159440 0x7fe3ac0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac159440 0x7fe3ac0b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58020/0 (socket says 172.21.15.67:58020) 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 -- 172.21.15.67:0/3989469360 learned_addr learned my addr 172.21.15.67:0/3989469360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3abe00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3ac0713e0 0x7fe3ac0ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b0a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac15be70 0x7fe3ac1570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 -- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac15be70 msgr2=0x7fe3ac1570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac15be70 0x7fe3ac1570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 -- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3ac0713e0 msgr2=0x7fe3ac0ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3ac0713e0 0x7fe3ac0ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 -- 172.21.15.67:0/3989469360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac1563d0 con 0x7fe3ac159440 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b0a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac15be70 0x7fe3ac1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3ab400640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac159440 0x7fe3ac0b4450 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe3980ad040 tx=0x7fe39801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3a3e00640 1 -- 172.21.15.67:0/3989469360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980be5f0 con 0x7fe3ac159440 2024-04-17T10:25:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3a3e00640 1 -- 172.21.15.67:0/3989469360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3980be790 con 0x7fe3ac159440 2024-04-17T10:25:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3a3e00640 1 -- 172.21.15.67:0/3989469360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980beb80 con 0x7fe3ac159440 2024-04-17T10:25:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac0b4990 con 0x7fe3ac159440 2024-04-17T10:25:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac0ad270 con 0x7fe3ac159440 2024-04-17T10:25:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.867+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac04ef30 con 0x7fe3ac159440 2024-04-17T10:25:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.871+0000 7fe3a3e00640 1 -- 172.21.15.67:0/3989469360 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe3980b43f0 con 0x7fe3ac159440 2024-04-17T10:25:18.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.871+0000 7fe3a3e00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe38c066f60 0x7fe38c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:18.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.871+0000 7fe3a3e00640 1 -- 172.21.15.67:0/3989469360 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe39811fd40 con 0x7fe3ac159440 2024-04-17T10:25:18.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.871+0000 7fe3abe00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe38c066f60 0x7fe38c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:18.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.871+0000 7fe3abe00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe38c066f60 0x7fe38c069420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe39c007a60 tx=0x7fe39c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:18.875+0000 7fe3a3e00640 1 -- 172.21.15.67:0/3989469360 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe398016150 con 0x7fe3ac159440 2024-04-17T10:25:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.071+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe3ac15e740 con 0x7fe3ac159440 2024-04-17T10:25:19.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.687+0000 7fe3a3e00640 1 -- 172.21.15.67:0/3989469360 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe3980f04d0 con 0x7fe3ac159440 2024-04-17T10:25:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe38c066f60 msgr2=0x7fe38c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe38c066f60 0x7fe38c069420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe39c007a60 tx=0x7fe39c008040 comp rx=0 tx=0).stop 2024-04-17T10:25:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac159440 msgr2=0x7fe3ac0b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac159440 0x7fe3ac0b4450 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe3980ad040 tx=0x7fe39801ada0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 shutdown_connections 2024-04-17T10:25:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe38c066f60 0x7fe38c069420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3ac15be70 0x7fe3ac1570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3ac159440 0x7fe3ac0b4450 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 --2- 172.21.15.67:0/3989469360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3ac0713e0 0x7fe3ac0ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 >> 172.21.15.67:0/3989469360 conn(0x7fe3ac0b1740 msgr2=0x7fe3ac0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 shutdown_connections 2024-04-17T10:25:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.691+0000 7fe3b2a00640 1 -- 172.21.15.67:0/3989469360 wait complete. 2024-04-17T10:25:19.712 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:25:19.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:25:19.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 Processor -- start 2024-04-17T10:25:19.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 -- start start 2024-04-17T10:25:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57940ce1b0 0x7f57940ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f579415cc00 0x7f579415f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:19.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57940cead0 con 0x7f579415fa00 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f579415be60 con 0x7f57940ce1b0 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5794160230 con 0x7f579415cc00 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57940ce1b0 0x7f57940ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f57940ce1b0 0x7f57940ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58026/0 (socket says 172.21.15.67:58026) 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 -- 172.21.15.67:0/633096578 learned_addr learned my addr 172.21.15.67:0/633096578 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5792a00640 1 --2- 172.21.15.67:0/633096578 >> v2:172.21.15.86:3300/0 conn(0x7f579415cc00 0x7f579415f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 -- 172.21.15.67:0/633096578 >> v2:172.21.15.86:3300/0 conn(0x7f579415cc00 msgr2=0x7f579415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 --2- 172.21.15.67:0/633096578 >> v2:172.21.15.86:3300/0 conn(0x7f579415cc00 0x7f579415f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 -- 172.21.15.67:0/633096578 >> v1:172.21.15.67:6789/0 conn(0x7f579415fa00 legacy=0x7f5794070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 -- 172.21.15.67:0/633096578 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579415c120 con 0x7f57940ce1b0 2024-04-17T10:25:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5793400640 1 --2- 172.21.15.67:0/633096578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57940ce1b0 0x7f57940ce590 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5794072de0 tx=0x7f578401ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794c828e36ee6d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5792000640 1 -- 172.21.15.67:0/633096578 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57840b5070 con 0x7f57940ce1b0 2024-04-17T10:25:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5792000640 1 -- 172.21.15.67:0/633096578 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57840a8a10 con 0x7f57940ce1b0 2024-04-17T10:25:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f5792000640 1 -- 172.21.15.67:0/633096578 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57840a8de0 con 0x7f57940ce1b0 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 -- 172.21.15.67:0/633096578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57940ce1b0 msgr2=0x7f57940ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.859+0000 7f579a200640 1 --2- 172.21.15.67:0/633096578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57940ce1b0 0x7f57940ce590 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5794072de0 tx=0x7f578401ed10 comp rx=0 tx=0).stop 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- 172.21.15.67:0/633096578 shutdown_connections 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 --2- 172.21.15.67:0/633096578 >> v2:172.21.15.86:3300/0 conn(0x7f579415cc00 0x7f579415f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 --2- 172.21.15.67:0/633096578 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57940ce1b0 0x7f57940ce590 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- 172.21.15.67:0/633096578 >> 172.21.15.67:0/633096578 conn(0x7f57940b33f0 msgr2=0x7f57940b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- 172.21.15.67:0/633096578 shutdown_connections 2024-04-17T10:25:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- 172.21.15.67:0/633096578 wait complete. 2024-04-17T10:25:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 Processor -- start 2024-04-17T10:25:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- start start 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57940ce1b0 0x7f57941e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f579415cc00 0x7f57940cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f579415fa00 0x7f579414f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5794159730 con 0x7f579415fa00 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579415c6e0 con 0x7f57940ce1b0 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f579a200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57941598b0 con 0x7f579415cc00 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f579415fa00 0x7f579414f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f579415fa00 0x7f579414f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58036/0 (socket says 172.21.15.67:58036) 2024-04-17T10:25:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 -- 172.21.15.67:0/2181238436 learned_addr learned my addr 172.21.15.67:0/2181238436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5792a00640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f579415cc00 0x7f57940cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 -- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f579415cc00 msgr2=0x7f57940cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f579415cc00 0x7f57940cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 -- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57940ce1b0 msgr2=0x7f57941e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57940ce1b0 0x7f57941e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 -- 172.21.15.67:0/2181238436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57941e63a0 con 0x7f579415fa00 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5792a00640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f579415cc00 0x7f57940cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f5793e00640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f579415fa00 0x7f579414f200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f578c00c760 tx=0x7f578c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f578b400640 1 -- 172.21.15.67:0/2181238436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c003b80 con 0x7f579415fa00 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.863+0000 7f578b400640 1 -- 172.21.15.67:0/2181238436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c003d20 con 0x7f579415fa00 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57941e6680 con 0x7f579415fa00 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f578b400640 1 -- 172.21.15.67:0/2181238436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c00aa30 con 0x7f579415fa00 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57941e6c40 con 0x7f579415fa00 2024-04-17T10:25:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5794070ab0 con 0x7f579415fa00 2024-04-17T10:25:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f578b400640 1 -- 172.21.15.67:0/2181238436 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f578c00abd0 con 0x7f579415fa00 2024-04-17T10:25:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f578b400640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5774066f60 0x7f5774069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f578b400640 1 -- 172.21.15.67:0/2181238436 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f578c085220 con 0x7f579415fa00 2024-04-17T10:25:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f5793400640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5774066f60 0x7f5774069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.867+0000 7f5793400640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5774066f60 0x7f5774069420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f57840b2000 tx=0x7f5784004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:19.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:19.871+0000 7f578b400640 1 -- 172.21.15.67:0/2181238436 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f578c055ee0 con 0x7f579415fa00 2024-04-17T10:25:20.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5794074e30 con 0x7f579415fa00 2024-04-17T10:25:20.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f578b400640 1 -- 172.21.15.67:0/2181238436 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f578c0557e0 con 0x7f579415fa00 2024-04-17T10:25:20.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:20.070 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:25:20.070 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5774066f60 msgr2=0x7f5774069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5774066f60 0x7f5774069420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f57840b2000 tx=0x7f5784004740 comp rx=0 tx=0).stop 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f579415fa00 msgr2=0x7f579414f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f579415fa00 0x7f579414f200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f578c00c760 tx=0x7f578c00cc20 comp rx=0 tx=0).stop 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 shutdown_connections 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5774066f60 0x7f5774069420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f579415fa00 0x7f579414f200 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f579415cc00 0x7f57940cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 --2- 172.21.15.67:0/2181238436 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57940ce1b0 0x7f57941e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 >> 172.21.15.67:0/2181238436 conn(0x7f57940b33f0 msgr2=0x7f579415d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 shutdown_connections 2024-04-17T10:25:20.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.071+0000 7f579a200640 1 -- 172.21.15.67:0/2181238436 wait complete. 2024-04-17T10:25:20.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:20.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe8b200640 1 Processor -- start 2024-04-17T10:25:20.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe8b200640 1 -- start start 2024-04-17T10:25:20.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe8b200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efe841466e0 0x7efe8406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe8b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe84071a20 0x7efe840adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe8b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe84161740 con 0x7efe84146da0 2024-04-17T10:25:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe8b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe84161a20 con 0x7efe84071a20 2024-04-17T10:25:20.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe8b200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe84161d00 con 0x7efe841466e0 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe84071a20 0x7efe840adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe84071a20 0x7efe840adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58042/0 (socket says 172.21.15.67:58042) 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 -- 172.21.15.67:0/973189633 learned_addr learned my addr 172.21.15.67:0/973189633 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe83e00640 1 --2- 172.21.15.67:0/973189633 >> v2:172.21.15.86:3300/0 conn(0x7efe841466e0 0x7efe8406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 -- 172.21.15.67:0/973189633 >> v2:172.21.15.86:3300/0 conn(0x7efe841466e0 msgr2=0x7efe8406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 --2- 172.21.15.67:0/973189633 >> v2:172.21.15.86:3300/0 conn(0x7efe841466e0 0x7efe8406eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 -- 172.21.15.67:0/973189633 >> v1:172.21.15.67:6789/0 conn(0x7efe84146da0 legacy=0x7efe84071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 -- 172.21.15.67:0/973189633 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84161fe0 con 0x7efe84071a20 2024-04-17T10:25:20.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe83e00640 1 --2- 172.21.15.67:0/973189633 >> v2:172.21.15.86:3300/0 conn(0x7efe841466e0 0x7efe8406eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe89200640 1 --2- 172.21.15.67:0/973189633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe84071a20 0x7efe840adde0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efe840b2530 tx=0x7efe7c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60b9e89117eb686 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.235+0000 7efe83400640 1 -- 172.21.15.67:0/973189633 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0b0070 con 0x7efe84071a20 2024-04-17T10:25:20.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe83400640 1 -- 172.21.15.67:0/973189633 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c0ab600 con 0x7efe84071a20 2024-04-17T10:25:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe83400640 1 -- 172.21.15.67:0/973189633 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0ab9d0 con 0x7efe84071a20 2024-04-17T10:25:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- 172.21.15.67:0/973189633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe84071a20 msgr2=0x7efe840adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 --2- 172.21.15.67:0/973189633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe84071a20 0x7efe840adde0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efe840b2530 tx=0x7efe7c01c520 comp rx=0 tx=0).stop 2024-04-17T10:25:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- 172.21.15.67:0/973189633 shutdown_connections 2024-04-17T10:25:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 --2- 172.21.15.67:0/973189633 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe84071a20 0x7efe840adde0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 --2- 172.21.15.67:0/973189633 >> v2:172.21.15.86:3300/0 conn(0x7efe841466e0 0x7efe8406eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- 172.21.15.67:0/973189633 >> 172.21.15.67:0/973189633 conn(0x7efe8406ac90 msgr2=0x7efe8406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- 172.21.15.67:0/973189633 shutdown_connections 2024-04-17T10:25:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- 172.21.15.67:0/973189633 wait complete. 2024-04-17T10:25:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 Processor -- start 2024-04-17T10:25:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- start start 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe84071a20 0x7efe84158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe841466e0 0x7efe841417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe84146da0 0x7efe84141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840aeec0 con 0x7efe841466e0 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840aed40 con 0x7efe84071a20 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe8b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe840af040 con 0x7efe84146da0 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe7b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe841466e0 0x7efe841417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe7b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe841466e0 0x7efe841417c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58050/0 (socket says 172.21.15.67:58050) 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe7b400640 1 -- 172.21.15.67:0/4054819782 learned_addr learned my addr 172.21.15.67:0/4054819782 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe83e00640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe84071a20 0x7efe84158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe89200640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe84146da0 0x7efe84141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.239+0000 7efe7b400640 1 -- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe84146da0 msgr2=0x7efe84141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe7b400640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe84146da0 0x7efe84141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe7b400640 1 -- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe84071a20 msgr2=0x7efe84158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe7b400640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe84071a20 0x7efe84158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe7b400640 1 -- 172.21.15.67:0/4054819782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe840b1b30 con 0x7efe841466e0 2024-04-17T10:25:20.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe7b400640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe841466e0 0x7efe841417c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00c730 tx=0x7efe6c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe80c00640 1 -- 172.21.15.67:0/4054819782 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c016020 con 0x7efe841466e0 2024-04-17T10:25:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe80c00640 1 -- 172.21.15.67:0/4054819782 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6c006d60 con 0x7efe841466e0 2024-04-17T10:25:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe80c00640 1 -- 172.21.15.67:0/4054819782 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c003bc0 con 0x7efe841466e0 2024-04-17T10:25:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe840b1570 con 0x7efe841466e0 2024-04-17T10:25:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8413e2d0 con 0x7efe841466e0 2024-04-17T10:25:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe84072d50 con 0x7efe841466e0 2024-04-17T10:25:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe80c00640 1 -- 172.21.15.67:0/4054819782 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efe6c00a850 con 0x7efe841466e0 2024-04-17T10:25:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe80c00640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe60066f90 0x7efe60069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.243+0000 7efe80c00640 1 -- 172.21.15.67:0/4054819782 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efe6c084e80 con 0x7efe841466e0 2024-04-17T10:25:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.247+0000 7efe83e00640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe60066f90 0x7efe60069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.247+0000 7efe83e00640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe60066f90 0x7efe60069450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efe74009950 tx=0x7efe74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.247+0000 7efe80c00640 1 -- 172.21.15.67:0/4054819782 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efe6c020020 con 0x7efe841466e0 2024-04-17T10:25:20.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.411+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe8406e740 con 0x7efe841466e0 2024-04-17T10:25:20.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.411+0000 7efe80c00640 1 -- 172.21.15.67:0/4054819782 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7efe6c0554f0 con 0x7efe841466e0 2024-04-17T10:25:20.412 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:20.412 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe60066f90 msgr2=0x7efe60069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe60066f90 0x7efe60069450 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efe74009950 tx=0x7efe74008040 comp rx=0 tx=0).stop 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe841466e0 msgr2=0x7efe841417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe841466e0 0x7efe841417c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efe6c00c730 tx=0x7efe6c00cbf0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 shutdown_connections 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe60066f90 0x7efe60069450 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe84146da0 0x7efe84141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe841466e0 0x7efe841417c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 --2- 172.21.15.67:0/4054819782 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe84071a20 0x7efe84158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 >> 172.21.15.67:0/4054819782 conn(0x7efe8406ac90 msgr2=0x7efe8413d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 shutdown_connections 2024-04-17T10:25:20.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.415+0000 7efe8b200640 1 -- 172.21.15.67:0/4054819782 wait complete. 2024-04-17T10:25:20.433 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:25:20.434 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:25:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb567e00640 1 Processor -- start 2024-04-17T10:25:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb567e00640 1 -- start start 2024-04-17T10:25:20.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb567e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb560146da0 0x7fb560071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb567e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb560161740 con 0x7fb5601466e0 2024-04-17T10:25:20.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb567e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb560161a20 con 0x7fb560071850 2024-04-17T10:25:20.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb567e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb560161d00 con 0x7fb560146da0 2024-04-17T10:25:20.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb565400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5601466e0 0x7fb56006eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54284/0 (socket says 172.21.15.67:54284) 2024-04-17T10:25:20.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb565400640 1 -- 172.21.15.67:0/497769260 learned_addr learned my addr 172.21.15.67:0/497769260 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:20.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb564a00640 1 --2- 172.21.15.67:0/497769260 >> v2:172.21.15.86:3300/0 conn(0x7fb560146da0 0x7fb560071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408499963 0 0) 0x7fb560161740 con 0x7fb5601466e0 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb548003530 con 0x7fb5601466e0 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2347852821 0 0) 0x7fb548003530 con 0x7fb5601466e0 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 >> v2:172.21.15.86:3300/0 conn(0x7fb560146da0 msgr2=0x7fb560071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 --2- 172.21.15.67:0/497769260 >> v2:172.21.15.86:3300/0 conn(0x7fb560146da0 0x7fb560071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 >> v1:172.21.15.86:6789/0 conn(0x7fb560071850 legacy=0x7fb5600adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb560161fe0 con 0x7fb5601466e0 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fb554001680 con 0x7fb5601466e0 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 >> v1:172.21.15.67:6789/0 conn(0x7fb5601466e0 legacy=0x7fb56006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480039c0 0x7fb548003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5480043c0 0x7fb548004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb548005090 0x7fb54800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb560161d00 con 0x7fb5480043c0 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54800bff0 con 0x7fb5480039c0 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54800c1f0 con 0x7fb548005090 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb564a00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5480043c0 0x7fb548004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.575+0000 7fb565400640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480039c0 0x7fb548003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb565e00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb548005090 0x7fb54800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb564a00640 1 -- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb548005090 msgr2=0x7fb54800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb564a00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb548005090 0x7fb54800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb564a00640 1 -- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480039c0 msgr2=0x7fb548003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb564a00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480039c0 0x7fb548003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb564a00640 1 -- 172.21.15.67:0/497769260 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb548003530 con 0x7fb5480043c0 2024-04-17T10:25:20.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb565e00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb548005090 0x7fb54800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:20.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb565e00640 1 -- 172.21.15.67:0/497769260 reap_dead start 2024-04-17T10:25:20.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb564a00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5480043c0 0x7fb548004890 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009ae0 tx=0x7fb54c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc70fc858a2d55 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0b2070 con 0x7fb5480043c0 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c0a5660 con 0x7fb5480043c0 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb55fe00640 1 -- 172.21.15.67:0/497769260 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c0096b0 con 0x7fb5480043c0 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb567e00640 1 -- 172.21.15.67:0/497769260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5480043c0 msgr2=0x7fb548004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb567e00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5480043c0 0x7fb548004890 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009ae0 tx=0x7fb54c01ca70 comp rx=0 tx=0).stop 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb567e00640 1 -- 172.21.15.67:0/497769260 shutdown_connections 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb567e00640 1 --2- 172.21.15.67:0/497769260 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5480043c0 0x7fb548004890 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb567e00640 1 -- 172.21.15.67:0/497769260 >> 172.21.15.67:0/497769260 conn(0x7fb56006ac90 msgr2=0x7fb56006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.579+0000 7fb567e00640 1 -- 172.21.15.67:0/497769260 shutdown_connections 2024-04-17T10:25:20.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 -- 172.21.15.67:0/497769260 wait complete. 2024-04-17T10:25:20.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 Processor -- start 2024-04-17T10:25:20.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 -- start start 2024-04-17T10:25:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480043c0 0x7fb5600b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5600afaa0 0x7fb5600adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5601f3f60 0x7fb5600cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb560071490 con 0x7fb5601f3f60 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb560071310 con 0x7fb5480043c0 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb560071610 con 0x7fb5600afaa0 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5601f3f60 0x7fb5600cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5601f3f60 0x7fb5600cd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57864/0 (socket says 172.21.15.67:57864) 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 -- 172.21.15.67:0/1079344084 learned_addr learned my addr 172.21.15.67:0/1079344084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb564a00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5600afaa0 0x7fb5600adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565400640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480043c0 0x7fb5600b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 -- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5600afaa0 msgr2=0x7fb5600adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5600afaa0 0x7fb5600adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 -- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480043c0 msgr2=0x7fb5600b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480043c0 0x7fb5600b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 -- 172.21.15.67:0/1079344084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5600ae2e0 con 0x7fb5601f3f60 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb565e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5601f3f60 0x7fb5600cd480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb5580059b0 tx=0x7fb55800f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb55d600640 1 -- 172.21.15.67:0/1079344084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558010040 con 0x7fb5601f3f60 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb55d600640 1 -- 172.21.15.67:0/1079344084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55800fdc0 con 0x7fb5601f3f60 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb55d600640 1 -- 172.21.15.67:0/1079344084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558015650 con 0x7fb5601f3f60 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.583+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56015f260 con 0x7fb5601f3f60 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.587+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56015f820 con 0x7fb5601f3f60 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.587+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb560072d50 con 0x7fb5601f3f60 2024-04-17T10:25:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.587+0000 7fb55d600640 1 -- 172.21.15.67:0/1079344084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb558015a00 con 0x7fb5601f3f60 2024-04-17T10:25:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.587+0000 7fb55d600640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb540066f90 0x7fb540069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.587+0000 7fb55d600640 1 -- 172.21.15.67:0/1079344084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb558088b80 con 0x7fb5601f3f60 2024-04-17T10:25:20.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.587+0000 7fb565400640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb540066f90 0x7fb540069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.587+0000 7fb565400640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb540066f90 0x7fb540069450 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb554001060 tx=0x7fb554007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.591+0000 7fb55d600640 1 -- 172.21.15.67:0/1079344084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb5580591f0 con 0x7fb5601f3f60 2024-04-17T10:25:20.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.751+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb560074e30 con 0x7fb540066f90 2024-04-17T10:25:20.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.759+0000 7fb55d600640 1 -- 172.21.15.67:0/1079344084 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332537 (secure 0 0 0) 0x7fb560074e30 con 0x7fb540066f90 2024-04-17T10:25:20.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:20.766 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:25:20.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.763+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb540066f90 msgr2=0x7fb540069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.763+0000 7fb567e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb540066f90 0x7fb540069450 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb554001060 tx=0x7fb554007900 comp rx=0 tx=0).stop 2024-04-17T10:25:20.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.763+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5601f3f60 msgr2=0x7fb5600cd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.763+0000 7fb567e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5601f3f60 0x7fb5600cd480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb5580059b0 tx=0x7fb55800f540 comp rx=0 tx=0).stop 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 shutdown_connections 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb540066f90 0x7fb540069450 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5601f3f60 0x7fb5600cd480 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5600afaa0 0x7fb5600adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 --2- 172.21.15.67:0/1079344084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5480043c0 0x7fb5600b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 >> 172.21.15.67:0/1079344084 conn(0x7fb56006ac90 msgr2=0x7fb56006eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 shutdown_connections 2024-04-17T10:25:20.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.767+0000 7fb567e00640 1 -- 172.21.15.67:0/1079344084 wait complete. 2024-04-17T10:25:20.784 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-17T10:25:20.140208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003333"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155309+0000","last_change":"2024-04-17T10:24:54.602648+0000","last_active":"2024-04-17T10:25:03.155309+0000","last_peered":"2024-04-17T10:25:03.155309+0000","last_clean":"2024-04-17T10:25:03.155309+0000","last_became_active":"2024-04-17T10:24:54.602381+0000","last_became_peered":"2024-04-17T10:24:54.602381+0000","last_unstale":"2024-04-17T10:25:03.155309+0000","last_undegraded":"2024-04-17T10:25:03.155309+0000","last_fullsized":"2024-04-17T10:25:03.155309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638270+0000","last_change":"2024-04-17T10:24:55.617102+0000","last_active":"2024-04-17T10:25:01.638270+0000","last_peered":"2024-04-17T10:25:01.638270+0000","last_clean":"2024-04-17T10:25:01.638270+0000","last_became_active":"2024-04-17T10:24:55.615401+0000","last_became_peered":"2024-04-17T10:24:55.615401+0000","last_unstale":"2024-04-17T10:25:01.638270+0000","last_undegraded":"2024-04-17T10:25:01.638270+0000","last_fullsized":"2024-04-17T10:25:01.638270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637780+0000","last_change":"2024-04-17T10:25:00.246846+0000","last_active":"2024-04-17T10:25:01.637780+0000","last_peered":"2024-04-17T10:25:01.637780+0000","last_clean":"2024-04-17T10:25:01.637780+0000","last_became_active":"2024-04-17T10:25:00.246379+0000","last_became_peered":"2024-04-17T10:25:00.246379+0000","last_unstale":"2024-04-17T10:25:01.637780+0000","last_undegraded":"2024-04-17T10:25:01.637780+0000","last_fullsized":"2024-04-17T10:25:01.637780+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:55:05.249462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638369+0000","last_change":"2024-04-17T10:24:54.607859+0000","last_active":"2024-04-17T10:25:01.638369+0000","last_peered":"2024-04-17T10:25:01.638369+0000","last_clean":"2024-04-17T10:25:01.638369+0000","last_became_active":"2024-04-17T10:24:54.607576+0000","last_became_peered":"2024-04-17T10:24:54.607576+0000","last_unstale":"2024-04-17T10:25:01.638369+0000","last_undegraded":"2024-04-17T10:25:01.638369+0000","last_fullsized":"2024-04-17T10:25:01.638369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156289+0000","last_change":"2024-04-17T10:24:55.628197+0000","last_active":"2024-04-17T10:25:03.156289+0000","last_peered":"2024-04-17T10:25:03.156289+0000","last_clean":"2024-04-17T10:25:03.156289+0000","last_became_active":"2024-04-17T10:24:55.627976+0000","last_became_peered":"2024-04-17T10:24:55.627976+0000","last_unstale":"2024-04-17T10:25:03.156289+0000","last_undegraded":"2024-04-17T10:25:03.156289+0000","last_fullsized":"2024-04-17T10:25:03.156289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636562+0000","last_change":"2024-04-17T10:24:59.651727+0000","last_active":"2024-04-17T10:25:01.636562+0000","last_peered":"2024-04-17T10:25:01.636562+0000","last_clean":"2024-04-17T10:25:01.636562+0000","last_became_active":"2024-04-17T10:24:59.651358+0000","last_became_peered":"2024-04-17T10:24:59.651358+0000","last_unstale":"2024-04-17T10:25:01.636562+0000","last_undegraded":"2024-04-17T10:25:01.636562+0000","last_fullsized":"2024-04-17T10:25:01.636562+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423903+0000","last_change":"2024-04-17T10:24:54.611808+0000","last_active":"2024-04-17T10:25:02.423903+0000","last_peered":"2024-04-17T10:25:02.423903+0000","last_clean":"2024-04-17T10:25:02.423903+0000","last_became_active":"2024-04-17T10:24:54.610903+0000","last_became_peered":"2024-04-17T10:24:54.610903+0000","last_unstale":"2024-04-17T10:25:02.423903+0000","last_undegraded":"2024-04-17T10:25:02.423903+0000","last_fullsized":"2024-04-17T10:25:02.423903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:09.085810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268578+0000","last_change":"2024-04-17T10:24:55.618364+0000","last_active":"2024-04-17T10:25:03.268578+0000","last_peered":"2024-04-17T10:25:03.268578+0000","last_clean":"2024-04-17T10:25:03.268578+0000","last_became_active":"2024-04-17T10:24:55.618013+0000","last_became_peered":"2024-04-17T10:24:55.618013+0000","last_unstale":"2024-04-17T10:25:03.268578+0000","last_undegraded":"2024-04-17T10:25:03.268578+0000","last_fullsized":"2024-04-17T10:25:03.268578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636861+0000","last_change":"2024-04-17T10:24:59.647051+0000","last_active":"2024-04-17T10:25:01.636861+0000","last_peered":"2024-04-17T10:25:01.636861+0000","last_clean":"2024-04-17T10:25:01.636861+0000","last_became_active":"2024-04-17T10:24:59.646611+0000","last_became_peered":"2024-04-17T10:24:59.646611+0000","last_unstale":"2024-04-17T10:25:01.636861+0000","last_undegraded":"2024-04-17T10:25:01.636861+0000","last_fullsized":"2024-04-17T10:25:01.636861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423712+0000","last_change":"2024-04-17T10:24:54.611301+0000","last_active":"2024-04-17T10:25:02.423712+0000","last_peered":"2024-04-17T10:25:02.423712+0000","last_clean":"2024-04-17T10:25:02.423712+0000","last_became_active":"2024-04-17T10:24:54.610243+0000","last_became_peered":"2024-04-17T10:24:54.610243+0000","last_unstale":"2024-04-17T10:25:02.423712+0000","last_undegraded":"2024-04-17T10:25:02.423712+0000","last_fullsized":"2024-04-17T10:25:02.423712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:36.054430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268618+0000","last_change":"2024-04-17T10:24:55.618112+0000","last_active":"2024-04-17T10:25:03.268618+0000","last_peered":"2024-04-17T10:25:03.268618+0000","last_clean":"2024-04-17T10:25:03.268618+0000","last_became_active":"2024-04-17T10:24:55.617127+0000","last_became_peered":"2024-04-17T10:24:55.617127+0000","last_unstale":"2024-04-17T10:25:03.268618+0000","last_undegraded":"2024-04-17T10:25:03.268618+0000","last_fullsized":"2024-04-17T10:25:03.268618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636698+0000","last_change":"2024-04-17T10:24:59.641046+0000","last_active":"2024-04-17T10:25:01.636698+0000","last_peered":"2024-04-17T10:25:01.636698+0000","last_clean":"2024-04-17T10:25:01.636698+0000","last_became_active":"2024-04-17T10:24:59.640622+0000","last_became_peered":"2024-04-17T10:24:59.640622+0000","last_unstale":"2024-04-17T10:25:01.636698+0000","last_undegraded":"2024-04-17T10:25:01.636698+0000","last_fullsized":"2024-04-17T10:25:01.636698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537179+0000","last_change":"2024-04-17T10:24:55.617061+0000","last_active":"2024-04-17T10:25:02.537179+0000","last_peered":"2024-04-17T10:25:02.537179+0000","last_clean":"2024-04-17T10:25:02.537179+0000","last_became_active":"2024-04-17T10:24:55.616809+0000","last_became_peered":"2024-04-17T10:24:55.616809+0000","last_unstale":"2024-04-17T10:25:02.537179+0000","last_undegraded":"2024-04-17T10:25:02.537179+0000","last_fullsized":"2024-04-17T10:25:02.537179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423815+0000","last_change":"2024-04-17T10:24:54.607974+0000","last_active":"2024-04-17T10:25:02.423815+0000","last_peered":"2024-04-17T10:25:02.423815+0000","last_clean":"2024-04-17T10:25:02.423815+0000","last_became_active":"2024-04-17T10:24:54.606303+0000","last_became_peered":"2024-04-17T10:24:54.606303+0000","last_unstale":"2024-04-17T10:25:02.423815+0000","last_undegraded":"2024-04-17T10:25:02.423815+0000","last_fullsized":"2024-04-17T10:25:02.423815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:52.381704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637318+0000","last_change":"2024-04-17T10:24:59.645993+0000","last_active":"2024-04-17T10:25:01.637318+0000","last_peered":"2024-04-17T10:25:01.637318+0000","last_clean":"2024-04-17T10:25:01.637318+0000","last_became_active":"2024-04-17T10:24:59.645305+0000","last_became_peered":"2024-04-17T10:24:59.645305+0000","last_unstale":"2024-04-17T10:25:01.637318+0000","last_undegraded":"2024-04-17T10:25:01.637318+0000","last_fullsized":"2024-04-17T10:25:01.637318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638467+0000","last_change":"2024-04-17T10:24:55.614771+0000","last_active":"2024-04-17T10:25:01.638467+0000","last_peered":"2024-04-17T10:25:01.638467+0000","last_clean":"2024-04-17T10:25:01.638467+0000","last_became_active":"2024-04-17T10:24:55.614517+0000","last_became_peered":"2024-04-17T10:24:55.614517+0000","last_unstale":"2024-04-17T10:25:01.638467+0000","last_undegraded":"2024-04-17T10:25:01.638467+0000","last_fullsized":"2024-04-17T10:25:01.638467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636723+0000","last_change":"2024-04-17T10:24:54.608749+0000","last_active":"2024-04-17T10:25:01.636723+0000","last_peered":"2024-04-17T10:25:01.636723+0000","last_clean":"2024-04-17T10:25:01.636723+0000","last_became_active":"2024-04-17T10:24:54.606617+0000","last_became_peered":"2024-04-17T10:24:54.606617+0000","last_unstale":"2024-04-17T10:25:01.636723+0000","last_undegraded":"2024-04-17T10:25:01.636723+0000","last_fullsized":"2024-04-17T10:25:01.636723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636946+0000","last_change":"2024-04-17T10:24:59.645457+0000","last_active":"2024-04-17T10:25:01.636946+0000","last_peered":"2024-04-17T10:25:01.636946+0000","last_clean":"2024-04-17T10:25:01.636946+0000","last_became_active":"2024-04-17T10:24:59.644747+0000","last_became_peered":"2024-04-17T10:24:59.644747+0000","last_unstale":"2024-04-17T10:25:01.636946+0000","last_undegraded":"2024-04-17T10:25:01.636946+0000","last_fullsized":"2024-04-17T10:25:01.636946+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637820+0000","last_change":"2024-04-17T10:24:55.630638+0000","last_active":"2024-04-17T10:25:01.637820+0000","last_peered":"2024-04-17T10:25:01.637820+0000","last_clean":"2024-04-17T10:25:01.637820+0000","last_became_active":"2024-04-17T10:24:55.630351+0000","last_became_peered":"2024-04-17T10:24:55.630351+0000","last_unstale":"2024-04-17T10:25:01.637820+0000","last_undegraded":"2024-04-17T10:25:01.637820+0000","last_fullsized":"2024-04-17T10:25:01.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636842+0000","last_change":"2024-04-17T10:24:54.609635+0000","last_active":"2024-04-17T10:25:01.636842+0000","last_peered":"2024-04-17T10:25:01.636842+0000","last_clean":"2024-04-17T10:25:01.636842+0000","last_became_active":"2024-04-17T10:24:54.607298+0000","last_became_peered":"2024-04-17T10:24:54.607298+0000","last_unstale":"2024-04-17T10:25:01.636842+0000","last_undegraded":"2024-04-17T10:25:01.636842+0000","last_fullsized":"2024-04-17T10:25:01.636842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637200+0000","last_change":"2024-04-17T10:25:00.243112+0000","last_active":"2024-04-17T10:25:01.637200+0000","last_peered":"2024-04-17T10:25:01.637200+0000","last_clean":"2024-04-17T10:25:01.637200+0000","last_became_active":"2024-04-17T10:25:00.242669+0000","last_became_peered":"2024-04-17T10:25:00.242669+0000","last_unstale":"2024-04-17T10:25:01.637200+0000","last_undegraded":"2024-04-17T10:25:01.637200+0000","last_fullsized":"2024-04-17T10:25:01.637200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637134+0000","last_change":"2024-04-17T10:24:55.608101+0000","last_active":"2024-04-17T10:25:01.637134+0000","last_peered":"2024-04-17T10:25:01.637134+0000","last_clean":"2024-04-17T10:25:01.637134+0000","last_became_active":"2024-04-17T10:24:55.607668+0000","last_became_peered":"2024-04-17T10:24:55.607668+0000","last_unstale":"2024-04-17T10:25:01.637134+0000","last_undegraded":"2024-04-17T10:25:01.637134+0000","last_fullsized":"2024-04-17T10:25:01.637134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:35.913082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423205+0000","last_change":"2024-04-17T10:24:54.607981+0000","last_active":"2024-04-17T10:25:02.423205+0000","last_peered":"2024-04-17T10:25:02.423205+0000","last_clean":"2024-04-17T10:25:02.423205+0000","last_became_active":"2024-04-17T10:24:54.607150+0000","last_became_peered":"2024-04-17T10:24:54.607150+0000","last_unstale":"2024-04-17T10:25:02.423205+0000","last_undegraded":"2024-04-17T10:25:02.423205+0000","last_fullsized":"2024-04-17T10:25:02.423205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636873+0000","last_change":"2024-04-17T10:24:59.645029+0000","last_active":"2024-04-17T10:25:01.636873+0000","last_peered":"2024-04-17T10:25:01.636873+0000","last_clean":"2024-04-17T10:25:01.636873+0000","last_became_active":"2024-04-17T10:24:59.644540+0000","last_became_peered":"2024-04-17T10:24:59.644540+0000","last_unstale":"2024-04-17T10:25:01.636873+0000","last_undegraded":"2024-04-17T10:25:01.636873+0000","last_fullsized":"2024-04-17T10:25:01.636873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156445+0000","last_change":"2024-04-17T10:24:55.610920+0000","last_active":"2024-04-17T10:25:03.156445+0000","last_peered":"2024-04-17T10:25:03.156445+0000","last_clean":"2024-04-17T10:25:03.156445+0000","last_became_active":"2024-04-17T10:24:55.610566+0000","last_became_peered":"2024-04-17T10:24:55.610566+0000","last_unstale":"2024-04-17T10:25:03.156445+0000","last_undegraded":"2024-04-17T10:25:03.156445+0000","last_fullsized":"2024-04-17T10:25:03.156445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537952+0000","last_change":"2024-04-17T10:24:54.609866+0000","last_active":"2024-04-17T10:25:02.537952+0000","last_peered":"2024-04-17T10:25:02.537952+0000","last_clean":"2024-04-17T10:25:02.537952+0000","last_became_active":"2024-04-17T10:24:54.607310+0000","last_became_peered":"2024-04-17T10:24:54.607310+0000","last_unstale":"2024-04-17T10:25:02.537952+0000","last_undegraded":"2024-04-17T10:25:02.537952+0000","last_fullsized":"2024-04-17T10:25:02.537952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637020+0000","last_change":"2024-04-17T10:25:00.242151+0000","last_active":"2024-04-17T10:25:01.637020+0000","last_peered":"2024-04-17T10:25:01.637020+0000","last_clean":"2024-04-17T10:25:01.637020+0000","last_became_active":"2024-04-17T10:25:00.241789+0000","last_became_peered":"2024-04-17T10:25:00.241789+0000","last_unstale":"2024-04-17T10:25:01.637020+0000","last_undegraded":"2024-04-17T10:25:01.637020+0000","last_fullsized":"2024-04-17T10:25:01.637020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638591+0000","last_change":"2024-04-17T10:24:55.611037+0000","last_active":"2024-04-17T10:25:01.638591+0000","last_peered":"2024-04-17T10:25:01.638591+0000","last_clean":"2024-04-17T10:25:01.638591+0000","last_became_active":"2024-04-17T10:24:55.610713+0000","last_became_peered":"2024-04-17T10:24:55.610713+0000","last_unstale":"2024-04-17T10:25:01.638591+0000","last_undegraded":"2024-04-17T10:25:01.638591+0000","last_fullsized":"2024-04-17T10:25:01.638591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636990+0000","last_change":"2024-04-17T10:24:54.614299+0000","last_active":"2024-04-17T10:25:01.636990+0000","last_peered":"2024-04-17T10:25:01.636990+0000","last_clean":"2024-04-17T10:25:01.636990+0000","last_became_active":"2024-04-17T10:24:54.613957+0000","last_became_peered":"2024-04-17T10:24:54.613957+0000","last_unstale":"2024-04-17T10:25:01.636990+0000","last_undegraded":"2024-04-17T10:25:01.636990+0000","last_fullsized":"2024-04-17T10:25:01.636990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637684+0000","last_change":"2024-04-17T10:25:00.241327+0000","last_active":"2024-04-17T10:25:01.637684+0000","last_peered":"2024-04-17T10:25:01.637684+0000","last_clean":"2024-04-17T10:25:01.637684+0000","last_became_active":"2024-04-17T10:25:00.240884+0000","last_became_peered":"2024-04-17T10:25:00.240884+0000","last_unstale":"2024-04-17T10:25:01.637684+0000","last_undegraded":"2024-04-17T10:25:01.637684+0000","last_fullsized":"2024-04-17T10:25:01.637684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:46.933770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.624025+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.623725+0000","last_became_peered":"2024-04-17T10:24:55.623725+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424055+0000","last_change":"2024-04-17T10:24:54.610613+0000","last_active":"2024-04-17T10:25:02.424055+0000","last_peered":"2024-04-17T10:25:02.424055+0000","last_clean":"2024-04-17T10:25:02.424055+0000","last_became_active":"2024-04-17T10:24:54.609405+0000","last_became_peered":"2024-04-17T10:24:54.609405+0000","last_unstale":"2024-04-17T10:25:02.424055+0000","last_undegraded":"2024-04-17T10:25:02.424055+0000","last_fullsized":"2024-04-17T10:25:02.424055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:06.248042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155827+0000","last_change":"2024-04-17T10:25:00.243396+0000","last_active":"2024-04-17T10:25:03.155827+0000","last_peered":"2024-04-17T10:25:03.155827+0000","last_clean":"2024-04-17T10:25:03.155827+0000","last_became_active":"2024-04-17T10:25:00.243086+0000","last_became_peered":"2024-04-17T10:25:00.243086+0000","last_unstale":"2024-04-17T10:25:03.155827+0000","last_undegraded":"2024-04-17T10:25:03.155827+0000","last_fullsized":"2024-04-17T10:25:03.155827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268735+0000","last_change":"2024-04-17T10:24:55.611872+0000","last_active":"2024-04-17T10:25:03.268735+0000","last_peered":"2024-04-17T10:25:03.268735+0000","last_clean":"2024-04-17T10:25:03.268735+0000","last_became_active":"2024-04-17T10:24:55.607201+0000","last_became_peered":"2024-04-17T10:24:55.607201+0000","last_unstale":"2024-04-17T10:25:03.268735+0000","last_undegraded":"2024-04-17T10:25:03.268735+0000","last_fullsized":"2024-04-17T10:25:03.268735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637222+0000","last_change":"2024-04-17T10:24:54.598254+0000","last_active":"2024-04-17T10:25:01.637222+0000","last_peered":"2024-04-17T10:25:01.637222+0000","last_clean":"2024-04-17T10:25:01.637222+0000","last_became_active":"2024-04-17T10:24:54.597869+0000","last_became_peered":"2024-04-17T10:24:54.597869+0000","last_unstale":"2024-04-17T10:25:01.637222+0000","last_undegraded":"2024-04-17T10:25:01.637222+0000","last_fullsized":"2024-04-17T10:25:01.637222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636567+0000","last_change":"2024-04-17T10:24:59.656986+0000","last_active":"2024-04-17T10:25:01.636567+0000","last_peered":"2024-04-17T10:25:01.636567+0000","last_clean":"2024-04-17T10:25:01.636567+0000","last_became_active":"2024-04-17T10:24:59.656124+0000","last_became_peered":"2024-04-17T10:24:59.656124+0000","last_unstale":"2024-04-17T10:25:01.636567+0000","last_undegraded":"2024-04-17T10:25:01.636567+0000","last_fullsized":"2024-04-17T10:25:01.636567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637349+0000","last_change":"2024-04-17T10:24:55.625808+0000","last_active":"2024-04-17T10:25:01.637349+0000","last_peered":"2024-04-17T10:25:01.637349+0000","last_clean":"2024-04-17T10:25:01.637349+0000","last_became_active":"2024-04-17T10:24:55.625551+0000","last_became_peered":"2024-04-17T10:24:55.625551+0000","last_unstale":"2024-04-17T10:25:01.637349+0000","last_undegraded":"2024-04-17T10:25:01.637349+0000","last_fullsized":"2024-04-17T10:25:01.637349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267481+0000","last_change":"2024-04-17T10:24:54.603615+0000","last_active":"2024-04-17T10:25:03.267481+0000","last_peered":"2024-04-17T10:25:03.267481+0000","last_clean":"2024-04-17T10:25:03.267481+0000","last_became_active":"2024-04-17T10:24:54.600774+0000","last_became_peered":"2024-04-17T10:24:54.600774+0000","last_unstale":"2024-04-17T10:25:03.267481+0000","last_undegraded":"2024-04-17T10:25:03.267481+0000","last_fullsized":"2024-04-17T10:25:03.267481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637164+0000","last_change":"2024-04-17T10:24:59.646484+0000","last_active":"2024-04-17T10:25:01.637164+0000","last_peered":"2024-04-17T10:25:01.637164+0000","last_clean":"2024-04-17T10:25:01.637164+0000","last_became_active":"2024-04-17T10:24:59.646061+0000","last_became_peered":"2024-04-17T10:24:59.646061+0000","last_unstale":"2024-04-17T10:25:01.637164+0000","last_undegraded":"2024-04-17T10:25:01.637164+0000","last_fullsized":"2024-04-17T10:25:01.637164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636232+0000","last_change":"2024-04-17T10:24:55.624915+0000","last_active":"2024-04-17T10:25:01.636232+0000","last_peered":"2024-04-17T10:25:01.636232+0000","last_clean":"2024-04-17T10:25:01.636232+0000","last_became_active":"2024-04-17T10:24:55.624230+0000","last_became_peered":"2024-04-17T10:24:55.624230+0000","last_unstale":"2024-04-17T10:25:01.636232+0000","last_undegraded":"2024-04-17T10:25:01.636232+0000","last_fullsized":"2024-04-17T10:25:01.636232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624893+0000","last_change":"2024-04-17T10:24:54.599766+0000","last_active":"2024-04-17T10:25:02.624893+0000","last_peered":"2024-04-17T10:25:02.624893+0000","last_clean":"2024-04-17T10:25:02.624893+0000","last_became_active":"2024-04-17T10:24:54.599476+0000","last_became_peered":"2024-04-17T10:24:54.599476+0000","last_unstale":"2024-04-17T10:25:02.624893+0000","last_undegraded":"2024-04-17T10:25:02.624893+0000","last_fullsized":"2024-04-17T10:25:02.624893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637263+0000","last_change":"2024-04-17T10:24:59.656597+0000","last_active":"2024-04-17T10:25:01.637263+0000","last_peered":"2024-04-17T10:25:01.637263+0000","last_clean":"2024-04-17T10:25:01.637263+0000","last_became_active":"2024-04-17T10:24:59.656135+0000","last_became_peered":"2024-04-17T10:24:59.656135+0000","last_unstale":"2024-04-17T10:25:01.637263+0000","last_undegraded":"2024-04-17T10:25:01.637263+0000","last_fullsized":"2024-04-17T10:25:01.637263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155614+0000","last_change":"2024-04-17T10:24:55.591986+0000","last_active":"2024-04-17T10:25:03.155614+0000","last_peered":"2024-04-17T10:25:03.155614+0000","last_clean":"2024-04-17T10:25:03.155614+0000","last_became_active":"2024-04-17T10:24:55.591672+0000","last_became_peered":"2024-04-17T10:24:55.591672+0000","last_unstale":"2024-04-17T10:25:03.155614+0000","last_undegraded":"2024-04-17T10:25:03.155614+0000","last_fullsized":"2024-04-17T10:25:03.155614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423263+0000","last_change":"2024-04-17T10:24:54.614030+0000","last_active":"2024-04-17T10:25:02.423263+0000","last_peered":"2024-04-17T10:25:02.423263+0000","last_clean":"2024-04-17T10:25:02.423263+0000","last_became_active":"2024-04-17T10:24:54.613752+0000","last_became_peered":"2024-04-17T10:24:54.613752+0000","last_unstale":"2024-04-17T10:25:02.423263+0000","last_undegraded":"2024-04-17T10:25:02.423263+0000","last_fullsized":"2024-04-17T10:25:02.423263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156369+0000","last_change":"2024-04-17T10:25:00.244705+0000","last_active":"2024-04-17T10:25:03.156369+0000","last_peered":"2024-04-17T10:25:03.156369+0000","last_clean":"2024-04-17T10:25:03.156369+0000","last_became_active":"2024-04-17T10:25:00.243853+0000","last_became_peered":"2024-04-17T10:25:00.243853+0000","last_unstale":"2024-04-17T10:25:03.156369+0000","last_undegraded":"2024-04-17T10:25:03.156369+0000","last_fullsized":"2024-04-17T10:25:03.156369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638050+0000","last_change":"2024-04-17T10:24:55.622500+0000","last_active":"2024-04-17T10:25:01.638050+0000","last_peered":"2024-04-17T10:25:01.638050+0000","last_clean":"2024-04-17T10:25:01.638050+0000","last_became_active":"2024-04-17T10:24:55.622206+0000","last_became_peered":"2024-04-17T10:24:55.622206+0000","last_unstale":"2024-04-17T10:25:01.638050+0000","last_undegraded":"2024-04-17T10:25:01.638050+0000","last_fullsized":"2024-04-17T10:25:01.638050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268327+0000","last_change":"2024-04-17T10:24:54.601255+0000","last_active":"2024-04-17T10:25:03.268327+0000","last_peered":"2024-04-17T10:25:03.268327+0000","last_clean":"2024-04-17T10:25:03.268327+0000","last_became_active":"2024-04-17T10:24:54.600968+0000","last_became_peered":"2024-04-17T10:24:54.600968+0000","last_unstale":"2024-04-17T10:25:03.268327+0000","last_undegraded":"2024-04-17T10:25:03.268327+0000","last_fullsized":"2024-04-17T10:25:03.268327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423358+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:25:02.423358+0000","last_peered":"2024-04-17T10:25:02.423358+0000","last_clean":"2024-04-17T10:25:02.423358+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:25:02.423358+0000","last_undegraded":"2024-04-17T10:25:02.423358+0000","last_fullsized":"2024-04-17T10:25:02.423358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537629+0000","last_change":"2024-04-17T10:25:00.247011+0000","last_active":"2024-04-17T10:25:02.537629+0000","last_peered":"2024-04-17T10:25:02.537629+0000","last_clean":"2024-04-17T10:25:02.537629+0000","last_became_active":"2024-04-17T10:25:00.245780+0000","last_became_peered":"2024-04-17T10:25:00.245780+0000","last_unstale":"2024-04-17T10:25:02.537629+0000","last_undegraded":"2024-04-17T10:25:02.537629+0000","last_fullsized":"2024-04-17T10:25:02.537629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423766+0000","last_change":"2024-04-17T10:24:55.613528+0000","last_active":"2024-04-17T10:25:02.423766+0000","last_peered":"2024-04-17T10:25:02.423766+0000","last_clean":"2024-04-17T10:25:02.423766+0000","last_became_active":"2024-04-17T10:24:55.613203+0000","last_became_peered":"2024-04-17T10:24:55.613203+0000","last_unstale":"2024-04-17T10:25:02.423766+0000","last_undegraded":"2024-04-17T10:25:02.423766+0000","last_fullsized":"2024-04-17T10:25:02.423766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:15.389666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625258+0000","last_change":"2024-04-17T10:24:54.605199+0000","last_active":"2024-04-17T10:25:02.625258+0000","last_peered":"2024-04-17T10:25:02.625258+0000","last_clean":"2024-04-17T10:25:02.625258+0000","last_became_active":"2024-04-17T10:24:54.604881+0000","last_became_peered":"2024-04-17T10:24:54.604881+0000","last_unstale":"2024-04-17T10:25:02.625258+0000","last_undegraded":"2024-04-17T10:25:02.625258+0000","last_fullsized":"2024-04-17T10:25:02.625258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537487+0000","last_change":"2024-04-17T10:25:00.242786+0000","last_active":"2024-04-17T10:25:02.537487+0000","last_peered":"2024-04-17T10:25:02.537487+0000","last_clean":"2024-04-17T10:25:02.537487+0000","last_became_active":"2024-04-17T10:25:00.242222+0000","last_became_peered":"2024-04-17T10:25:00.242222+0000","last_unstale":"2024-04-17T10:25:02.537487+0000","last_undegraded":"2024-04-17T10:25:02.537487+0000","last_fullsized":"2024-04-17T10:25:02.537487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:01.409834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268628+0000","last_change":"2024-04-17T10:24:55.606154+0000","last_active":"2024-04-17T10:25:03.268628+0000","last_peered":"2024-04-17T10:25:03.268628+0000","last_clean":"2024-04-17T10:25:03.268628+0000","last_became_active":"2024-04-17T10:24:55.605854+0000","last_became_peered":"2024-04-17T10:24:55.605854+0000","last_unstale":"2024-04-17T10:25:03.268628+0000","last_undegraded":"2024-04-17T10:25:03.268628+0000","last_fullsized":"2024-04-17T10:25:03.268628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636665+0000","last_change":"2024-04-17T10:24:54.605549+0000","last_active":"2024-04-17T10:25:01.636665+0000","last_peered":"2024-04-17T10:25:01.636665+0000","last_clean":"2024-04-17T10:25:01.636665+0000","last_became_active":"2024-04-17T10:24:54.605197+0000","last_became_peered":"2024-04-17T10:24:54.605197+0000","last_unstale":"2024-04-17T10:25:01.636665+0000","last_undegraded":"2024-04-17T10:25:01.636665+0000","last_fullsized":"2024-04-17T10:25:01.636665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155960+0000","last_change":"2024-04-17T10:24:59.652908+0000","last_active":"2024-04-17T10:25:03.155960+0000","last_peered":"2024-04-17T10:25:03.155960+0000","last_clean":"2024-04-17T10:25:03.155960+0000","last_became_active":"2024-04-17T10:24:59.652604+0000","last_became_peered":"2024-04-17T10:24:59.652604+0000","last_unstale":"2024-04-17T10:25:03.155960+0000","last_undegraded":"2024-04-17T10:25:03.155960+0000","last_fullsized":"2024-04-17T10:25:03.155960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424244+0000","last_change":"2024-04-17T10:24:55.612380+0000","last_active":"2024-04-17T10:25:02.424244+0000","last_peered":"2024-04-17T10:25:02.424244+0000","last_clean":"2024-04-17T10:25:02.424244+0000","last_became_active":"2024-04-17T10:24:55.611511+0000","last_became_peered":"2024-04-17T10:24:55.611511+0000","last_unstale":"2024-04-17T10:25:02.424244+0000","last_undegraded":"2024-04-17T10:25:02.424244+0000","last_fullsized":"2024-04-17T10:25:02.424244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:49.176910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624922+0000","last_change":"2024-04-17T10:24:54.601414+0000","last_active":"2024-04-17T10:25:02.624922+0000","last_peered":"2024-04-17T10:25:02.624922+0000","last_clean":"2024-04-17T10:25:02.624922+0000","last_became_active":"2024-04-17T10:24:54.600297+0000","last_became_peered":"2024-04-17T10:24:54.600297+0000","last_unstale":"2024-04-17T10:25:02.624922+0000","last_undegraded":"2024-04-17T10:25:02.624922+0000","last_fullsized":"2024-04-17T10:25:02.624922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267910+0000","last_change":"2024-04-17T10:25:00.242716+0000","last_active":"2024-04-17T10:25:03.267910+0000","last_peered":"2024-04-17T10:25:03.267910+0000","last_clean":"2024-04-17T10:25:03.267910+0000","last_became_active":"2024-04-17T10:25:00.242357+0000","last_became_peered":"2024-04-17T10:25:00.242357+0000","last_unstale":"2024-04-17T10:25:03.267910+0000","last_undegraded":"2024-04-17T10:25:03.267910+0000","last_fullsized":"2024-04-17T10:25:03.267910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424088+0000","last_change":"2024-04-17T10:24:55.604575+0000","last_active":"2024-04-17T10:25:02.424088+0000","last_peered":"2024-04-17T10:25:02.424088+0000","last_clean":"2024-04-17T10:25:02.424088+0000","last_became_active":"2024-04-17T10:24:55.604252+0000","last_became_peered":"2024-04-17T10:24:55.604252+0000","last_unstale":"2024-04-17T10:25:02.424088+0000","last_undegraded":"2024-04-17T10:25:02.424088+0000","last_fullsized":"2024-04-17T10:25:02.424088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:02.059745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637103+0000","last_change":"2024-04-17T10:24:54.595839+0000","last_active":"2024-04-17T10:25:01.637103+0000","last_peered":"2024-04-17T10:25:01.637103+0000","last_clean":"2024-04-17T10:25:01.637103+0000","last_became_active":"2024-04-17T10:24:54.594547+0000","last_became_peered":"2024-04-17T10:24:54.594547+0000","last_unstale":"2024-04-17T10:25:01.637103+0000","last_undegraded":"2024-04-17T10:25:01.637103+0000","last_fullsized":"2024-04-17T10:25:01.637103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636525+0000","last_change":"2024-04-17T10:24:59.644365+0000","last_active":"2024-04-17T10:25:01.636525+0000","last_peered":"2024-04-17T10:25:01.636525+0000","last_clean":"2024-04-17T10:25:01.636525+0000","last_became_active":"2024-04-17T10:24:59.643969+0000","last_became_peered":"2024-04-17T10:24:59.643969+0000","last_unstale":"2024-04-17T10:25:01.636525+0000","last_undegraded":"2024-04-17T10:25:01.636525+0000","last_fullsized":"2024-04-17T10:25:01.636525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423432+0000","last_change":"2024-04-17T10:24:55.598627+0000","last_active":"2024-04-17T10:25:02.423432+0000","last_peered":"2024-04-17T10:25:02.423432+0000","last_clean":"2024-04-17T10:25:02.423432+0000","last_became_active":"2024-04-17T10:24:55.598319+0000","last_became_peered":"2024-04-17T10:24:55.598319+0000","last_unstale":"2024-04-17T10:25:02.423432+0000","last_undegraded":"2024-04-17T10:25:02.423432+0000","last_fullsized":"2024-04-17T10:25:02.423432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624900+0000","last_change":"2024-04-17T10:24:54.600358+0000","last_active":"2024-04-17T10:25:02.624900+0000","last_peered":"2024-04-17T10:25:02.624900+0000","last_clean":"2024-04-17T10:25:02.624900+0000","last_became_active":"2024-04-17T10:24:54.600064+0000","last_became_peered":"2024-04-17T10:24:54.600064+0000","last_unstale":"2024-04-17T10:25:02.624900+0000","last_undegraded":"2024-04-17T10:25:02.624900+0000","last_fullsized":"2024-04-17T10:25:02.624900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156321+0000","last_change":"2024-04-17T10:24:59.651264+0000","last_active":"2024-04-17T10:25:03.156321+0000","last_peered":"2024-04-17T10:25:03.156321+0000","last_clean":"2024-04-17T10:25:03.156321+0000","last_became_active":"2024-04-17T10:24:59.650654+0000","last_became_peered":"2024-04-17T10:24:59.650654+0000","last_unstale":"2024-04-17T10:25:03.156321+0000","last_undegraded":"2024-04-17T10:25:03.156321+0000","last_fullsized":"2024-04-17T10:25:03.156321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537254+0000","last_change":"2024-04-17T10:24:55.608832+0000","last_active":"2024-04-17T10:25:02.537254+0000","last_peered":"2024-04-17T10:25:02.537254+0000","last_clean":"2024-04-17T10:25:02.537254+0000","last_became_active":"2024-04-17T10:24:55.608402+0000","last_became_peered":"2024-04-17T10:24:55.608402+0000","last_unstale":"2024-04-17T10:25:02.537254+0000","last_undegraded":"2024-04-17T10:25:02.537254+0000","last_fullsized":"2024-04-17T10:25:02.537254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537354+0000","last_change":"2024-04-17T10:24:54.601967+0000","last_active":"2024-04-17T10:25:02.537354+0000","last_peered":"2024-04-17T10:25:02.537354+0000","last_clean":"2024-04-17T10:25:02.537354+0000","last_became_active":"2024-04-17T10:24:54.601643+0000","last_became_peered":"2024-04-17T10:24:54.601643+0000","last_unstale":"2024-04-17T10:25:02.537354+0000","last_undegraded":"2024-04-17T10:25:02.537354+0000","last_fullsized":"2024-04-17T10:25:02.537354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638493+0000","last_change":"2024-04-17T10:25:00.243903+0000","last_active":"2024-04-17T10:25:01.638493+0000","last_peered":"2024-04-17T10:25:01.638493+0000","last_clean":"2024-04-17T10:25:01.638493+0000","last_became_active":"2024-04-17T10:25:00.243126+0000","last_became_peered":"2024-04-17T10:25:00.243126+0000","last_unstale":"2024-04-17T10:25:01.638493+0000","last_undegraded":"2024-04-17T10:25:01.638493+0000","last_fullsized":"2024-04-17T10:25:01.638493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:44.715063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423617+0000","last_change":"2024-04-17T10:24:55.598632+0000","last_active":"2024-04-17T10:25:02.423617+0000","last_peered":"2024-04-17T10:25:02.423617+0000","last_clean":"2024-04-17T10:25:02.423617+0000","last_became_active":"2024-04-17T10:24:55.598302+0000","last_became_peered":"2024-04-17T10:24:55.598302+0000","last_unstale":"2024-04-17T10:25:02.423617+0000","last_undegraded":"2024-04-17T10:25:02.423617+0000","last_fullsized":"2024-04-17T10:25:02.423617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267532+0000","last_change":"2024-04-17T10:24:54.603334+0000","last_active":"2024-04-17T10:25:03.267532+0000","last_peered":"2024-04-17T10:25:03.267532+0000","last_clean":"2024-04-17T10:25:03.267532+0000","last_became_active":"2024-04-17T10:24:54.603066+0000","last_became_peered":"2024-04-17T10:24:54.603066+0000","last_unstale":"2024-04-17T10:25:03.267532+0000","last_undegraded":"2024-04-17T10:25:03.267532+0000","last_fullsized":"2024-04-17T10:25:03.267532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639048+0000","last_change":"2024-04-17T10:25:00.247479+0000","last_active":"2024-04-17T10:25:01.639048+0000","last_peered":"2024-04-17T10:25:01.639048+0000","last_clean":"2024-04-17T10:25:01.639048+0000","last_became_active":"2024-04-17T10:25:00.247066+0000","last_became_peered":"2024-04-17T10:25:00.247066+0000","last_unstale":"2024-04-17T10:25:01.639048+0000","last_undegraded":"2024-04-17T10:25:01.639048+0000","last_fullsized":"2024-04-17T10:25:01.639048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:10:54.855530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155768+0000","last_change":"2024-04-17T10:24:55.616361+0000","last_active":"2024-04-17T10:25:03.155768+0000","last_peered":"2024-04-17T10:25:03.155768+0000","last_clean":"2024-04-17T10:25:03.155768+0000","last_became_active":"2024-04-17T10:24:55.615902+0000","last_became_peered":"2024-04-17T10:24:55.615902+0000","last_unstale":"2024-04-17T10:25:03.155768+0000","last_undegraded":"2024-04-17T10:25:03.155768+0000","last_fullsized":"2024-04-17T10:25:03.155768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:53.388065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155191+0000","last_change":"2024-04-17T10:24:54.599143+0000","last_active":"2024-04-17T10:25:03.155191+0000","last_peered":"2024-04-17T10:25:03.155191+0000","last_clean":"2024-04-17T10:25:03.155191+0000","last_became_active":"2024-04-17T10:24:54.598831+0000","last_became_peered":"2024-04-17T10:24:54.598831+0000","last_unstale":"2024-04-17T10:25:03.155191+0000","last_undegraded":"2024-04-17T10:25:03.155191+0000","last_fullsized":"2024-04-17T10:25:03.155191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268682+0000","last_change":"2024-04-17T10:25:00.243074+0000","last_active":"2024-04-17T10:25:03.268682+0000","last_peered":"2024-04-17T10:25:03.268682+0000","last_clean":"2024-04-17T10:25:03.268682+0000","last_became_active":"2024-04-17T10:25:00.242532+0000","last_became_peered":"2024-04-17T10:25:00.242532+0000","last_unstale":"2024-04-17T10:25:03.268682+0000","last_undegraded":"2024-04-17T10:25:03.268682+0000","last_fullsized":"2024-04-17T10:25:03.268682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423646+0000","last_change":"2024-04-17T10:24:55.603471+0000","last_active":"2024-04-17T10:25:02.423646+0000","last_peered":"2024-04-17T10:25:02.423646+0000","last_clean":"2024-04-17T10:25:02.423646+0000","last_became_active":"2024-04-17T10:24:55.603177+0000","last_became_peered":"2024-04-17T10:24:55.603177+0000","last_unstale":"2024-04-17T10:25:02.423646+0000","last_undegraded":"2024-04-17T10:25:02.423646+0000","last_fullsized":"2024-04-17T10:25:02.423646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:52.938391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:10.622101+0000","last_change":"2024-04-17T10:24:54.599941+0000","last_active":"2024-04-17T10:25:10.622101+0000","last_peered":"2024-04-17T10:25:10.622101+0000","last_clean":"2024-04-17T10:25:10.622101+0000","last_became_active":"2024-04-17T10:24:54.599658+0000","last_became_peered":"2024-04-17T10:24:54.599658+0000","last_unstale":"2024-04-17T10:25:10.622101+0000","last_undegraded":"2024-04-17T10:25:10.622101+0000","last_fullsized":"2024-04-17T10:25:10.622101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637141+0000","last_change":"2024-04-17T10:24:59.657550+0000","last_active":"2024-04-17T10:25:01.637141+0000","last_peered":"2024-04-17T10:25:01.637141+0000","last_clean":"2024-04-17T10:25:01.637141+0000","last_became_active":"2024-04-17T10:24:59.657179+0000","last_became_peered":"2024-04-17T10:24:59.657179+0000","last_unstale":"2024-04-17T10:25:01.637141+0000","last_undegraded":"2024-04-17T10:25:01.637141+0000","last_fullsized":"2024-04-17T10:25:01.637141+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637399+0000","last_change":"2024-04-17T10:24:55.609933+0000","last_active":"2024-04-17T10:25:01.637399+0000","last_peered":"2024-04-17T10:25:01.637399+0000","last_clean":"2024-04-17T10:25:01.637399+0000","last_became_active":"2024-04-17T10:24:55.602879+0000","last_became_peered":"2024-04-17T10:24:55.602879+0000","last_unstale":"2024-04-17T10:25:01.637399+0000","last_undegraded":"2024-04-17T10:25:01.637399+0000","last_fullsized":"2024-04-17T10:25:01.637399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:51.084300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636629+0000","last_change":"2024-04-17T10:24:54.607532+0000","last_active":"2024-04-17T10:25:01.636629+0000","last_peered":"2024-04-17T10:25:01.636629+0000","last_clean":"2024-04-17T10:25:01.636629+0000","last_became_active":"2024-04-17T10:24:54.606527+0000","last_became_peered":"2024-04-17T10:24:54.606527+0000","last_unstale":"2024-04-17T10:25:01.636629+0000","last_undegraded":"2024-04-17T10:25:01.636629+0000","last_fullsized":"2024-04-17T10:25:01.636629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637464+0000","last_change":"2024-04-17T10:24:59.644478+0000","last_active":"2024-04-17T10:25:01.637464+0000","last_peered":"2024-04-17T10:25:01.637464+0000","last_clean":"2024-04-17T10:25:01.637464+0000","last_became_active":"2024-04-17T10:24:59.644083+0000","last_became_peered":"2024-04-17T10:24:59.644083+0000","last_unstale":"2024-04-17T10:25:01.637464+0000","last_undegraded":"2024-04-17T10:25:01.637464+0000","last_fullsized":"2024-04-17T10:25:01.637464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156377+0000","last_change":"2024-04-17T10:24:55.627359+0000","last_active":"2024-04-17T10:25:03.156377+0000","last_peered":"2024-04-17T10:25:03.156377+0000","last_clean":"2024-04-17T10:25:03.156377+0000","last_became_active":"2024-04-17T10:24:55.627094+0000","last_became_peered":"2024-04-17T10:24:55.627094+0000","last_unstale":"2024-04-17T10:25:03.156377+0000","last_undegraded":"2024-04-17T10:25:03.156377+0000","last_fullsized":"2024-04-17T10:25:03.156377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267575+0000","last_change":"2024-04-17T10:24:54.613073+0000","last_active":"2024-04-17T10:25:03.267575+0000","last_peered":"2024-04-17T10:25:03.267575+0000","last_clean":"2024-04-17T10:25:03.267575+0000","last_became_active":"2024-04-17T10:24:54.612771+0000","last_became_peered":"2024-04-17T10:24:54.612771+0000","last_unstale":"2024-04-17T10:25:03.267575+0000","last_undegraded":"2024-04-17T10:25:03.267575+0000","last_fullsized":"2024-04-17T10:25:03.267575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637853+0000","last_change":"2024-04-17T10:24:59.653549+0000","last_active":"2024-04-17T10:25:01.637853+0000","last_peered":"2024-04-17T10:25:01.637853+0000","last_clean":"2024-04-17T10:25:01.637853+0000","last_became_active":"2024-04-17T10:24:59.652615+0000","last_became_peered":"2024-04-17T10:24:59.652615+0000","last_unstale":"2024-04-17T10:25:01.637853+0000","last_undegraded":"2024-04-17T10:25:01.637853+0000","last_fullsized":"2024-04-17T10:25:01.637853+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:43.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268787+0000","last_change":"2024-04-17T10:24:55.612177+0000","last_active":"2024-04-17T10:25:03.268787+0000","last_peered":"2024-04-17T10:25:03.268787+0000","last_clean":"2024-04-17T10:25:03.268787+0000","last_became_active":"2024-04-17T10:24:55.611847+0000","last_became_peered":"2024-04-17T10:24:55.611847+0000","last_unstale":"2024-04-17T10:25:03.268787+0000","last_undegraded":"2024-04-17T10:25:03.268787+0000","last_fullsized":"2024-04-17T10:25:03.268787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:00.256447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537797+0000","last_change":"2024-04-17T10:24:54.614002+0000","last_active":"2024-04-17T10:25:02.537797+0000","last_peered":"2024-04-17T10:25:02.537797+0000","last_clean":"2024-04-17T10:25:02.537797+0000","last_became_active":"2024-04-17T10:24:54.613675+0000","last_became_peered":"2024-04-17T10:24:54.613675+0000","last_unstale":"2024-04-17T10:25:02.537797+0000","last_undegraded":"2024-04-17T10:25:02.537797+0000","last_fullsized":"2024-04-17T10:25:02.537797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637973+0000","last_change":"2024-04-17T10:25:00.244266+0000","last_active":"2024-04-17T10:25:01.637973+0000","last_peered":"2024-04-17T10:25:01.637973+0000","last_clean":"2024-04-17T10:25:01.637973+0000","last_became_active":"2024-04-17T10:25:00.243271+0000","last_became_peered":"2024-04-17T10:25:00.243271+0000","last_unstale":"2024-04-17T10:25:01.637973+0000","last_undegraded":"2024-04-17T10:25:01.637973+0000","last_fullsized":"2024-04-17T10:25:01.637973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:37:59.616402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156261+0000","last_change":"2024-04-17T10:25:00.244894+0000","last_active":"2024-04-17T10:25:03.156261+0000","last_peered":"2024-04-17T10:25:03.156261+0000","last_clean":"2024-04-17T10:25:03.156261+0000","last_became_active":"2024-04-17T10:25:00.244366+0000","last_became_peered":"2024-04-17T10:25:00.244366+0000","last_unstale":"2024-04-17T10:25:03.156261+0000","last_undegraded":"2024-04-17T10:25:03.156261+0000","last_fullsized":"2024-04-17T10:25:03.156261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636436+0000","last_change":"2024-04-17T10:24:55.610258+0000","last_active":"2024-04-17T10:25:01.636436+0000","last_peered":"2024-04-17T10:25:01.636436+0000","last_clean":"2024-04-17T10:25:01.636436+0000","last_became_active":"2024-04-17T10:24:55.609904+0000","last_became_peered":"2024-04-17T10:24:55.609904+0000","last_unstale":"2024-04-17T10:25:01.636436+0000","last_undegraded":"2024-04-17T10:25:01.636436+0000","last_fullsized":"2024-04-17T10:25:01.636436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267470+0000","last_change":"2024-04-17T10:24:54.605205+0000","last_active":"2024-04-17T10:25:03.267470+0000","last_peered":"2024-04-17T10:25:03.267470+0000","last_clean":"2024-04-17T10:25:03.267470+0000","last_became_active":"2024-04-17T10:24:54.601015+0000","last_became_peered":"2024-04-17T10:24:54.601015+0000","last_unstale":"2024-04-17T10:25:03.267470+0000","last_undegraded":"2024-04-17T10:25:03.267470+0000","last_fullsized":"2024-04-17T10:25:03.267470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537785+0000","last_change":"2024-04-17T10:24:59.648566+0000","last_active":"2024-04-17T10:25:02.537785+0000","last_peered":"2024-04-17T10:25:02.537785+0000","last_clean":"2024-04-17T10:25:02.537785+0000","last_became_active":"2024-04-17T10:24:59.648170+0000","last_became_peered":"2024-04-17T10:24:59.648170+0000","last_unstale":"2024-04-17T10:25:02.537785+0000","last_undegraded":"2024-04-17T10:25:02.537785+0000","last_fullsized":"2024-04-17T10:25:02.537785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637759+0000","last_change":"2024-04-17T10:24:55.611737+0000","last_active":"2024-04-17T10:25:01.637759+0000","last_peered":"2024-04-17T10:25:01.637759+0000","last_clean":"2024-04-17T10:25:01.637759+0000","last_became_active":"2024-04-17T10:24:55.611353+0000","last_became_peered":"2024-04-17T10:24:55.611353+0000","last_unstale":"2024-04-17T10:25:01.637759+0000","last_undegraded":"2024-04-17T10:25:01.637759+0000","last_fullsized":"2024-04-17T10:25:01.637759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638133+0000","last_change":"2024-04-17T10:24:54.601357+0000","last_active":"2024-04-17T10:25:01.638133+0000","last_peered":"2024-04-17T10:25:01.638133+0000","last_clean":"2024-04-17T10:25:01.638133+0000","last_became_active":"2024-04-17T10:24:54.601111+0000","last_became_peered":"2024-04-17T10:24:54.601111+0000","last_unstale":"2024-04-17T10:25:01.638133+0000","last_undegraded":"2024-04-17T10:25:01.638133+0000","last_fullsized":"2024-04-17T10:25:01.638133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537510+0000","last_change":"2024-04-17T10:24:54.601574+0000","last_active":"2024-04-17T10:25:02.537510+0000","last_peered":"2024-04-17T10:25:02.537510+0000","last_clean":"2024-04-17T10:25:02.537510+0000","last_became_active":"2024-04-17T10:24:54.601261+0000","last_became_peered":"2024-04-17T10:24:54.601261+0000","last_unstale":"2024-04-17T10:25:02.537510+0000","last_undegraded":"2024-04-17T10:25:02.537510+0000","last_fullsized":"2024-04-17T10:25:02.537510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156049+0000","last_change":"2024-04-17T10:24:55.604048+0000","last_active":"2024-04-17T10:25:03.156049+0000","last_peered":"2024-04-17T10:25:03.156049+0000","last_clean":"2024-04-17T10:25:03.156049+0000","last_became_active":"2024-04-17T10:24:55.592091+0000","last_became_peered":"2024-04-17T10:24:55.592091+0000","last_unstale":"2024-04-17T10:25:03.156049+0000","last_undegraded":"2024-04-17T10:25:03.156049+0000","last_fullsized":"2024-04-17T10:25:03.156049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625146+0000","last_change":"2024-04-17T10:24:59.656792+0000","last_active":"2024-04-17T10:25:02.625146+0000","last_peered":"2024-04-17T10:25:02.625146+0000","last_clean":"2024-04-17T10:25:02.625146+0000","last_became_active":"2024-04-17T10:24:59.656244+0000","last_became_peered":"2024-04-17T10:24:59.656244+0000","last_unstale":"2024-04-17T10:25:02.625146+0000","last_undegraded":"2024-04-17T10:25:02.625146+0000","last_fullsized":"2024-04-17T10:25:02.625146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:57.427708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638566+0000","last_change":"2024-04-17T10:24:54.609412+0000","last_active":"2024-04-17T10:25:01.638566+0000","last_peered":"2024-04-17T10:25:01.638566+0000","last_clean":"2024-04-17T10:25:01.638566+0000","last_became_active":"2024-04-17T10:24:54.609119+0000","last_became_peered":"2024-04-17T10:24:54.609119+0000","last_unstale":"2024-04-17T10:25:01.638566+0000","last_undegraded":"2024-04-17T10:25:01.638566+0000","last_fullsized":"2024-04-17T10:25:01.638566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636952+0000","last_change":"2024-04-17T10:24:55.614982+0000","last_active":"2024-04-17T10:25:01.636952+0000","last_peered":"2024-04-17T10:25:01.636952+0000","last_clean":"2024-04-17T10:25:01.636952+0000","last_became_active":"2024-04-17T10:24:55.614619+0000","last_became_peered":"2024-04-17T10:24:55.614619+0000","last_unstale":"2024-04-17T10:25:01.636952+0000","last_undegraded":"2024-04-17T10:25:01.636952+0000","last_fullsized":"2024-04-17T10:25:01.636952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:32.258402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638839+0000","last_change":"2024-04-17T10:24:59.647972+0000","last_active":"2024-04-17T10:25:01.638839+0000","last_peered":"2024-04-17T10:25:01.638839+0000","last_clean":"2024-04-17T10:25:01.638839+0000","last_became_active":"2024-04-17T10:24:59.646737+0000","last_became_peered":"2024-04-17T10:24:59.646737+0000","last_unstale":"2024-04-17T10:25:01.638839+0000","last_undegraded":"2024-04-17T10:25:01.638839+0000","last_fullsized":"2024-04-17T10:25:01.638839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:59.695291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537104+0000","last_change":"2024-04-17T10:24:55.608996+0000","last_active":"2024-04-17T10:25:02.537104+0000","last_peered":"2024-04-17T10:25:02.537104+0000","last_clean":"2024-04-17T10:25:02.537104+0000","last_became_active":"2024-04-17T10:24:55.608621+0000","last_became_peered":"2024-04-17T10:24:55.608621+0000","last_unstale":"2024-04-17T10:25:02.537104+0000","last_undegraded":"2024-04-17T10:25:02.537104+0000","last_fullsized":"2024-04-17T10:25:02.537104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625124+0000","last_change":"2024-04-17T10:24:55.616866+0000","last_active":"2024-04-17T10:25:02.625124+0000","last_peered":"2024-04-17T10:25:02.625124+0000","last_clean":"2024-04-17T10:25:02.625124+0000","last_became_active":"2024-04-17T10:24:55.616582+0000","last_became_peered":"2024-04-17T10:24:55.616582+0000","last_unstale":"2024-04-17T10:25:02.625124+0000","last_undegraded":"2024-04-17T10:25:02.625124+0000","last_fullsized":"2024-04-17T10:25:02.625124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268379+0000","last_change":"2024-04-17T10:24:55.613768+0000","last_active":"2024-04-17T10:25:03.268379+0000","last_peered":"2024-04-17T10:25:03.268379+0000","last_clean":"2024-04-17T10:25:03.268379+0000","last_became_active":"2024-04-17T10:24:55.612427+0000","last_became_peered":"2024-04-17T10:24:55.612427+0000","last_unstale":"2024-04-17T10:25:03.268379+0000","last_undegraded":"2024-04-17T10:25:03.268379+0000","last_fullsized":"2024-04-17T10:25:03.268379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155940+0000","last_change":"2024-04-17T10:24:55.613422+0000","last_active":"2024-04-17T10:25:03.155940+0000","last_peered":"2024-04-17T10:25:03.155940+0000","last_clean":"2024-04-17T10:25:03.155940+0000","last_became_active":"2024-04-17T10:24:55.612627+0000","last_became_peered":"2024-04-17T10:24:55.612627+0000","last_unstale":"2024-04-17T10:25:03.155940+0000","last_undegraded":"2024-04-17T10:25:03.155940+0000","last_fullsized":"2024-04-17T10:25:03.155940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639265+0000","last_change":"2024-04-17T10:24:55.617532+0000","last_active":"2024-04-17T10:25:01.639265+0000","last_peered":"2024-04-17T10:25:01.639265+0000","last_clean":"2024-04-17T10:25:01.639265+0000","last_became_active":"2024-04-17T10:24:55.617215+0000","last_became_peered":"2024-04-17T10:24:55.617215+0000","last_unstale":"2024-04-17T10:25:01.639265+0000","last_undegraded":"2024-04-17T10:25:01.639265+0000","last_fullsized":"2024-04-17T10:25:01.639265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636559+0000","last_change":"2024-04-17T10:24:55.631795+0000","last_active":"2024-04-17T10:25:01.636559+0000","last_peered":"2024-04-17T10:25:01.636559+0000","last_clean":"2024-04-17T10:25:01.636559+0000","last_became_active":"2024-04-17T10:24:55.629826+0000","last_became_peered":"2024-04-17T10:24:55.629826+0000","last_unstale":"2024-04-17T10:25:01.636559+0000","last_undegraded":"2024-04-17T10:25:01.636559+0000","last_fullsized":"2024-04-17T10:25:01.636559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537858+0000","last_change":"2024-04-17T10:24:55.606321+0000","last_active":"2024-04-17T10:25:02.537858+0000","last_peered":"2024-04-17T10:25:02.537858+0000","last_clean":"2024-04-17T10:25:02.537858+0000","last_became_active":"2024-04-17T10:24:55.605895+0000","last_became_peered":"2024-04-17T10:24:55.605895+0000","last_unstale":"2024-04-17T10:25:02.537858+0000","last_undegraded":"2024-04-17T10:25:02.537858+0000","last_fullsized":"2024-04-17T10:25:02.537858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638540+0000","last_change":"2024-04-17T10:24:55.607147+0000","last_active":"2024-04-17T10:25:01.638540+0000","last_peered":"2024-04-17T10:25:01.638540+0000","last_clean":"2024-04-17T10:25:01.638540+0000","last_became_active":"2024-04-17T10:24:55.606931+0000","last_became_peered":"2024-04-17T10:24:55.606931+0000","last_unstale":"2024-04-17T10:25:01.638540+0000","last_undegraded":"2024-04-17T10:25:01.638540+0000","last_fullsized":"2024-04-17T10:25:01.638540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268051+0000","last_change":"2024-04-17T10:24:55.609356+0000","last_active":"2024-04-17T10:25:03.268051+0000","last_peered":"2024-04-17T10:25:03.268051+0000","last_clean":"2024-04-17T10:25:03.268051+0000","last_became_active":"2024-04-17T10:24:55.607716+0000","last_became_peered":"2024-04-17T10:24:55.607716+0000","last_unstale":"2024-04-17T10:25:03.268051+0000","last_undegraded":"2024-04-17T10:25:03.268051+0000","last_fullsized":"2024-04-17T10:25:03.268051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638038+0000","last_change":"2024-04-17T10:24:55.605676+0000","last_active":"2024-04-17T10:25:01.638038+0000","last_peered":"2024-04-17T10:25:01.638038+0000","last_clean":"2024-04-17T10:25:01.638038+0000","last_became_active":"2024-04-17T10:24:55.605431+0000","last_became_peered":"2024-04-17T10:24:55.605431+0000","last_unstale":"2024-04-17T10:25:01.638038+0000","last_undegraded":"2024-04-17T10:25:01.638038+0000","last_fullsized":"2024-04-17T10:25:01.638038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268286+0000","last_change":"2024-04-17T10:24:55.615398+0000","last_active":"2024-04-17T10:25:03.268286+0000","last_peered":"2024-04-17T10:25:03.268286+0000","last_clean":"2024-04-17T10:25:03.268286+0000","last_became_active":"2024-04-17T10:24:55.615034+0000","last_became_peered":"2024-04-17T10:24:55.615034+0000","last_unstale":"2024-04-17T10:25:03.268286+0000","last_undegraded":"2024-04-17T10:25:03.268286+0000","last_fullsized":"2024-04-17T10:25:03.268286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155772+0000","last_change":"2024-04-17T10:24:55.613776+0000","last_active":"2024-04-17T10:25:03.155772+0000","last_peered":"2024-04-17T10:25:03.155772+0000","last_clean":"2024-04-17T10:25:03.155772+0000","last_became_active":"2024-04-17T10:24:55.613106+0000","last_became_peered":"2024-04-17T10:24:55.613106+0000","last_unstale":"2024-04-17T10:25:03.155772+0000","last_undegraded":"2024-04-17T10:25:03.155772+0000","last_fullsized":"2024-04-17T10:25:03.155772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537535+0000","last_change":"2024-04-17T10:24:55.619933+0000","last_active":"2024-04-17T10:25:02.537535+0000","last_peered":"2024-04-17T10:25:02.537535+0000","last_clean":"2024-04-17T10:25:02.537535+0000","last_became_active":"2024-04-17T10:24:55.619666+0000","last_became_peered":"2024-04-17T10:24:55.619666+0000","last_unstale":"2024-04-17T10:25:02.537535+0000","last_undegraded":"2024-04-17T10:25:02.537535+0000","last_fullsized":"2024-04-17T10:25:02.537535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423902+0000","last_change":"2024-04-17T10:24:55.608090+0000","last_active":"2024-04-17T10:25:02.423902+0000","last_peered":"2024-04-17T10:25:02.423902+0000","last_clean":"2024-04-17T10:25:02.423902+0000","last_became_active":"2024-04-17T10:24:55.607715+0000","last_became_peered":"2024-04-17T10:24:55.607715+0000","last_unstale":"2024-04-17T10:25:02.423902+0000","last_undegraded":"2024-04-17T10:25:02.423902+0000","last_fullsized":"2024-04-17T10:25:02.423902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:07.835599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625372+0000","last_change":"2024-04-17T10:24:55.616021+0000","last_active":"2024-04-17T10:25:02.625372+0000","last_peered":"2024-04-17T10:25:02.625372+0000","last_clean":"2024-04-17T10:25:02.625372+0000","last_became_active":"2024-04-17T10:24:55.615736+0000","last_became_peered":"2024-04-17T10:24:55.615736+0000","last_unstale":"2024-04-17T10:25:02.625372+0000","last_undegraded":"2024-04-17T10:25:02.625372+0000","last_fullsized":"2024-04-17T10:25:02.625372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268965+0000","last_change":"2024-04-17T10:24:55.614983+0000","last_active":"2024-04-17T10:25:03.268965+0000","last_peered":"2024-04-17T10:25:03.268965+0000","last_clean":"2024-04-17T10:25:03.268965+0000","last_became_active":"2024-04-17T10:24:55.614651+0000","last_became_peered":"2024-04-17T10:24:55.614651+0000","last_unstale":"2024-04-17T10:25:03.268965+0000","last_undegraded":"2024-04-17T10:25:03.268965+0000","last_fullsized":"2024-04-17T10:25:03.268965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267935+0000","last_change":"2024-04-17T10:24:55.626976+0000","last_active":"2024-04-17T10:25:03.267935+0000","last_peered":"2024-04-17T10:25:03.267935+0000","last_clean":"2024-04-17T10:25:03.267935+0000","last_became_active":"2024-04-17T10:24:55.626743+0000","last_became_peered":"2024-04-17T10:24:55.626743+0000","last_unstale":"2024-04-17T10:25:03.267935+0000","last_undegraded":"2024-04-17T10:25:03.267935+0000","last_fullsized":"2024-04-17T10:25:03.267935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636737+0000","last_change":"2024-04-17T10:24:55.618634+0000","last_active":"2024-04-17T10:25:01.636737+0000","last_peered":"2024-04-17T10:25:01.636737+0000","last_clean":"2024-04-17T10:25:01.636737+0000","last_became_active":"2024-04-17T10:24:55.618343+0000","last_became_peered":"2024-04-17T10:24:55.618343+0000","last_unstale":"2024-04-17T10:25:01.636737+0000","last_undegraded":"2024-04-17T10:25:01.636737+0000","last_fullsized":"2024-04-17T10:25:01.636737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155747+0000","last_change":"2024-04-17T10:24:55.616149+0000","last_active":"2024-04-17T10:25:03.155747+0000","last_peered":"2024-04-17T10:25:03.155747+0000","last_clean":"2024-04-17T10:25:03.155747+0000","last_became_active":"2024-04-17T10:24:55.615561+0000","last_became_peered":"2024-04-17T10:24:55.615561+0000","last_unstale":"2024-04-17T10:25:03.155747+0000","last_undegraded":"2024-04-17T10:25:03.155747+0000","last_fullsized":"2024-04-17T10:25:03.155747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:37.060791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638375+0000","last_change":"2024-04-17T10:24:55.612680+0000","last_active":"2024-04-17T10:25:01.638375+0000","last_peered":"2024-04-17T10:25:01.638375+0000","last_clean":"2024-04-17T10:25:01.638375+0000","last_became_active":"2024-04-17T10:24:55.612280+0000","last_became_peered":"2024-04-17T10:24:55.612280+0000","last_unstale":"2024-04-17T10:25:01.638375+0000","last_undegraded":"2024-04-17T10:25:01.638375+0000","last_fullsized":"2024-04-17T10:25:01.638375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.610235+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.609934+0000","last_became_peered":"2024-04-17T10:24:55.609934+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537674+0000","last_change":"2024-04-17T10:24:55.613780+0000","last_active":"2024-04-17T10:25:02.537674+0000","last_peered":"2024-04-17T10:25:02.537674+0000","last_clean":"2024-04-17T10:25:02.537674+0000","last_became_active":"2024-04-17T10:24:55.613360+0000","last_became_peered":"2024-04-17T10:24:55.613360+0000","last_unstale":"2024-04-17T10:25:02.537674+0000","last_undegraded":"2024-04-17T10:25:02.537674+0000","last_fullsized":"2024-04-17T10:25:02.537674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537823+0000","last_change":"2024-04-17T10:24:55.617826+0000","last_active":"2024-04-17T10:25:02.537823+0000","last_peered":"2024-04-17T10:25:02.537823+0000","last_clean":"2024-04-17T10:25:02.537823+0000","last_became_active":"2024-04-17T10:24:55.617446+0000","last_became_peered":"2024-04-17T10:24:55.617446+0000","last_unstale":"2024-04-17T10:25:02.537823+0000","last_undegraded":"2024-04-17T10:25:02.537823+0000","last_fullsized":"2024-04-17T10:25:02.537823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424189+0000","last_change":"2024-04-17T10:24:55.626063+0000","last_active":"2024-04-17T10:25:02.424189+0000","last_peered":"2024-04-17T10:25:02.424189+0000","last_clean":"2024-04-17T10:25:02.424189+0000","last_became_active":"2024-04-17T10:24:55.625720+0000","last_became_peered":"2024-04-17T10:24:55.625720+0000","last_unstale":"2024-04-17T10:25:02.424189+0000","last_undegraded":"2024-04-17T10:25:02.424189+0000","last_fullsized":"2024-04-17T10:25:02.424189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:39.746376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637649+0000","last_change":"2024-04-17T10:24:55.622013+0000","last_active":"2024-04-17T10:25:01.637649+0000","last_peered":"2024-04-17T10:25:01.637649+0000","last_clean":"2024-04-17T10:25:01.637649+0000","last_became_active":"2024-04-17T10:24:55.618726+0000","last_became_peered":"2024-04-17T10:24:55.618726+0000","last_unstale":"2024-04-17T10:25:01.637649+0000","last_undegraded":"2024-04-17T10:25:01.637649+0000","last_fullsized":"2024-04-17T10:25:01.637649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423351+0000","last_change":"2024-04-17T10:24:55.614704+0000","last_active":"2024-04-17T10:25:02.423351+0000","last_peered":"2024-04-17T10:25:02.423351+0000","last_clean":"2024-04-17T10:25:02.423351+0000","last_became_active":"2024-04-17T10:24:55.614425+0000","last_became_peered":"2024-04-17T10:24:55.614425+0000","last_unstale":"2024-04-17T10:25:02.423351+0000","last_undegraded":"2024-04-17T10:25:02.423351+0000","last_fullsized":"2024-04-17T10:25:02.423351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268264+0000","last_change":"2024-04-17T10:24:55.630019+0000","last_active":"2024-04-17T10:25:03.268264+0000","last_peered":"2024-04-17T10:25:03.268264+0000","last_clean":"2024-04-17T10:25:03.268264+0000","last_became_active":"2024-04-17T10:24:55.629824+0000","last_became_peered":"2024-04-17T10:24:55.629824+0000","last_unstale":"2024-04-17T10:25:03.268264+0000","last_undegraded":"2024-04-17T10:25:03.268264+0000","last_fullsized":"2024-04-17T10:25:03.268264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423511+0000","last_change":"2024-04-17T10:24:55.612346+0000","last_active":"2024-04-17T10:25:02.423511+0000","last_peered":"2024-04-17T10:25:02.423511+0000","last_clean":"2024-04-17T10:25:02.423511+0000","last_became_active":"2024-04-17T10:24:55.610079+0000","last_became_peered":"2024-04-17T10:24:55.610079+0000","last_unstale":"2024-04-17T10:25:02.423511+0000","last_undegraded":"2024-04-17T10:25:02.423511+0000","last_fullsized":"2024-04-17T10:25:02.423511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268502+0000","last_change":"2024-04-17T10:24:55.612221+0000","last_active":"2024-04-17T10:25:03.268502+0000","last_peered":"2024-04-17T10:25:03.268502+0000","last_clean":"2024-04-17T10:25:03.268502+0000","last_became_active":"2024-04-17T10:24:55.611946+0000","last_became_peered":"2024-04-17T10:24:55.611946+0000","last_unstale":"2024-04-17T10:25:03.268502+0000","last_undegraded":"2024-04-17T10:25:03.268502+0000","last_fullsized":"2024-04-17T10:25:03.268502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:56.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156183+0000","last_change":"2024-04-17T10:24:55.617011+0000","last_active":"2024-04-17T10:25:03.156183+0000","last_peered":"2024-04-17T10:25:03.156183+0000","last_clean":"2024-04-17T10:25:03.156183+0000","last_became_active":"2024-04-17T10:24:55.616709+0000","last_became_peered":"2024-04-17T10:24:55.616709+0000","last_unstale":"2024-04-17T10:25:03.156183+0000","last_undegraded":"2024-04-17T10:25:03.156183+0000","last_fullsized":"2024-04-17T10:25:03.156183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:10.092171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636934+0000","last_change":"2024-04-17T10:24:55.627305+0000","last_active":"2024-04-17T10:25:01.636934+0000","last_peered":"2024-04-17T10:25:01.636934+0000","last_clean":"2024-04-17T10:25:01.636934+0000","last_became_active":"2024-04-17T10:24:55.626978+0000","last_became_peered":"2024-04-17T10:24:55.626978+0000","last_unstale":"2024-04-17T10:25:01.636934+0000","last_undegraded":"2024-04-17T10:25:01.636934+0000","last_fullsized":"2024-04-17T10:25:01.636934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424104+0000","last_change":"2024-04-17T10:24:55.613165+0000","last_active":"2024-04-17T10:25:02.424104+0000","last_peered":"2024-04-17T10:25:02.424104+0000","last_clean":"2024-04-17T10:25:02.424104+0000","last_became_active":"2024-04-17T10:24:55.611929+0000","last_became_peered":"2024-04-17T10:24:55.611929+0000","last_unstale":"2024-04-17T10:25:02.424104+0000","last_undegraded":"2024-04-17T10:25:02.424104+0000","last_fullsized":"2024-04-17T10:25:02.424104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:48.582360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:25:20.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:25:20.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.967+0000 7f6d32a00640 1 Processor -- start 2024-04-17T10:25:20.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.967+0000 7f6d32a00640 1 -- start start 2024-04-17T10:25:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d32a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 0x7f6d2c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d32a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d2c15cc00 0x7f6d2c15f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d32a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2c0cead0 con 0x7f6d2c15fa00 2024-04-17T10:25:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d32a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d2c15be60 con 0x7f6d2c15cc00 2024-04-17T10:25:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d32a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c160230 con 0x7f6d2c0ce1b0 2024-04-17T10:25:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6d2c15fa00 0x7f6d2c070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54288/0 (socket says 172.21.15.67:54288) 2024-04-17T10:25:20.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 -- 172.21.15.67:0/75612150 learned_addr learned my addr 172.21.15.67:0/75612150 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:20.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d23e00640 1 --2- 172.21.15.67:0/75612150 >> v2:172.21.15.86:3300/0 conn(0x7f6d2c15cc00 0x7f6d2c15f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2be00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 0x7f6d2c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233003149 0 0) 0x7f6d2c0cead0 con 0x7f6d2c15fa00 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d10003530 con 0x7f6d2c15fa00 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2014803180 0 0) 0x7f6d10003530 con 0x7f6d2c15fa00 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 msgr2=0x7f6d2c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 0x7f6d2c0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 >> v2:172.21.15.86:3300/0 conn(0x7f6d2c15cc00 msgr2=0x7f6d2c15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 --2- 172.21.15.67:0/75612150 >> v2:172.21.15.86:3300/0 conn(0x7f6d2c15cc00 0x7f6d2c15f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c15c120 con 0x7f6d2c15fa00 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f6d24001a00 con 0x7f6d2c15fa00 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 >> v1:172.21.15.67:6789/0 conn(0x7f6d2c15fa00 legacy=0x7f6d2c070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 0x7f6d100039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d10003f90 0x7f6d10004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d10004d50 0x7f6d100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15be60 con 0x7f6d10004d50 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c160230 con 0x7f6d10003f90 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1000ba50 con 0x7f6d2c0ce1b0 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d10004d50 0x7f6d100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d2be00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 0x7f6d100039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d23e00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d10003f90 0x7f6d10004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 -- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 msgr2=0x7f6d100039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d2c0ce1b0 0x7f6d100039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 -- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d10003f90 msgr2=0x7f6d10004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d10003f90 0x7f6d10004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.971+0000 7f6d30a00640 1 -- 172.21.15.67:0/75612150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10003530 con 0x7f6d10004d50 2024-04-17T10:25:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d30a00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d10004d50 0x7f6d100051d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400a7b0 tx=0x7f6d2401d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9495e749ab8cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2400a3d0 con 0x7f6d10004d50 2024-04-17T10:25:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d240ad650 con 0x7f6d10004d50 2024-04-17T10:25:20.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d2b400640 1 -- 172.21.15.67:0/75612150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d240ada40 con 0x7f6d10004d50 2024-04-17T10:25:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d32a00640 1 -- 172.21.15.67:0/75612150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d10004d50 msgr2=0x7f6d100051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d32a00640 1 --2- 172.21.15.67:0/75612150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d10004d50 0x7f6d100051d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400a7b0 tx=0x7f6d2401d2e0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d30a00640 1 -- 172.21.15.67:0/75612150 reap_dead start 2024-04-17T10:25:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d32a00640 1 -- 172.21.15.67:0/75612150 shutdown_connections 2024-04-17T10:25:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d32a00640 1 -- 172.21.15.67:0/75612150 >> 172.21.15.67:0/75612150 conn(0x7f6d2c0b33f0 msgr2=0x7f6d2c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d32a00640 1 -- 172.21.15.67:0/75612150 shutdown_connections 2024-04-17T10:25:20.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d32a00640 1 -- 172.21.15.67:0/75612150 wait complete. 2024-04-17T10:25:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.975+0000 7f6d32a00640 1 Processor -- start 2024-04-17T10:25:20.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 -- start start 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d10004d50 0x7f6d2c141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2c0c8950 0x7f6d2c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d2c15b510 0x7f6d2c0c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15fe70 con 0x7f6d2c0c8950 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15c6e0 con 0x7f6d2c15b510 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2c15fff0 con 0x7f6d10004d50 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2c0c8950 0x7f6d2c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2c0c8950 0x7f6d2c159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57880/0 (socket says 172.21.15.67:57880) 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 -- 172.21.15.67:0/921221635 learned_addr learned my addr 172.21.15.67:0/921221635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d2be00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d10004d50 0x7f6d2c141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d30a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d2c15b510 0x7f6d2c0c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 -- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d10004d50 msgr2=0x7f6d2c141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d10004d50 0x7f6d2c141060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 -- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d2c15b510 msgr2=0x7f6d2c0c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d2c15b510 0x7f6d2c0c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 -- 172.21.15.67:0/921221635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c15a470 con 0x7f6d2c0c8950 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d30a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d2c15b510 0x7f6d2c0c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d23e00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2c0c8950 0x7f6d2c159b00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d14011a00 tx=0x7f6d14011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d28c00640 1 -- 172.21.15.67:0/921221635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d1400faf0 con 0x7f6d2c0c8950 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d28c00640 1 -- 172.21.15.67:0/921221635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d1400fc90 con 0x7f6d2c0c8950 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d28c00640 1 -- 172.21.15.67:0/921221635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d14014440 con 0x7f6d2c0c8950 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d2c15a750 con 0x7f6d2c0c8950 2024-04-17T10:25:20.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d2c15ad10 con 0x7f6d2c0c8950 2024-04-17T10:25:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.979+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d2c070ab0 con 0x7f6d2c0c8950 2024-04-17T10:25:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.983+0000 7f6d28c00640 1 -- 172.21.15.67:0/921221635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6d14012040 con 0x7f6d2c0c8950 2024-04-17T10:25:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.983+0000 7f6d28c00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d08066f60 0x7f6d08069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.983+0000 7f6d28c00640 1 -- 172.21.15.67:0/921221635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6d14085ab0 con 0x7f6d2c0c8950 2024-04-17T10:25:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.983+0000 7f6d2be00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d08066f60 0x7f6d08069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:20.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.983+0000 7f6d2be00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d08066f60 0x7f6d08069420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009be0 tx=0x7f6d1c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:20.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:20.987+0000 7f6d28c00640 1 -- 172.21.15.67:0/921221635 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6d14019030 con 0x7f6d2c0c8950 2024-04-17T10:25:21.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.147+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d2c0cf040 con 0x7f6d08066f60 2024-04-17T10:25:21.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d28c00640 1 -- 172.21.15.67:0/921221635 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332537 (secure 0 0 0) 0x7f6d2c0cf040 con 0x7f6d08066f60 2024-04-17T10:25:21.157 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d08066f60 msgr2=0x7f6d08069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d08066f60 0x7f6d08069420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009be0 tx=0x7f6d1c009190 comp rx=0 tx=0).stop 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2c0c8950 msgr2=0x7f6d2c159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2c0c8950 0x7f6d2c159b00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6d14011a00 tx=0x7f6d14011e00 comp rx=0 tx=0).stop 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 shutdown_connections 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d08066f60 0x7f6d08069420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d2c15b510 0x7f6d2c0c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2c0c8950 0x7f6d2c159b00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 --2- 172.21.15.67:0/921221635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d10004d50 0x7f6d2c141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 >> 172.21.15.67:0/921221635 conn(0x7f6d2c0b33f0 msgr2=0x7f6d2c0b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 shutdown_connections 2024-04-17T10:25:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.159+0000 7f6d32a00640 1 -- 172.21.15.67:0/921221635 wait complete. 2024-04-17T10:25:21.177 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-17T10:25:20.140208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003333"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155309+0000","last_change":"2024-04-17T10:24:54.602648+0000","last_active":"2024-04-17T10:25:03.155309+0000","last_peered":"2024-04-17T10:25:03.155309+0000","last_clean":"2024-04-17T10:25:03.155309+0000","last_became_active":"2024-04-17T10:24:54.602381+0000","last_became_peered":"2024-04-17T10:24:54.602381+0000","last_unstale":"2024-04-17T10:25:03.155309+0000","last_undegraded":"2024-04-17T10:25:03.155309+0000","last_fullsized":"2024-04-17T10:25:03.155309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638270+0000","last_change":"2024-04-17T10:24:55.617102+0000","last_active":"2024-04-17T10:25:01.638270+0000","last_peered":"2024-04-17T10:25:01.638270+0000","last_clean":"2024-04-17T10:25:01.638270+0000","last_became_active":"2024-04-17T10:24:55.615401+0000","last_became_peered":"2024-04-17T10:24:55.615401+0000","last_unstale":"2024-04-17T10:25:01.638270+0000","last_undegraded":"2024-04-17T10:25:01.638270+0000","last_fullsized":"2024-04-17T10:25:01.638270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637780+0000","last_change":"2024-04-17T10:25:00.246846+0000","last_active":"2024-04-17T10:25:01.637780+0000","last_peered":"2024-04-17T10:25:01.637780+0000","last_clean":"2024-04-17T10:25:01.637780+0000","last_became_active":"2024-04-17T10:25:00.246379+0000","last_became_peered":"2024-04-17T10:25:00.246379+0000","last_unstale":"2024-04-17T10:25:01.637780+0000","last_undegraded":"2024-04-17T10:25:01.637780+0000","last_fullsized":"2024-04-17T10:25:01.637780+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:55:05.249462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638369+0000","last_change":"2024-04-17T10:24:54.607859+0000","last_active":"2024-04-17T10:25:01.638369+0000","last_peered":"2024-04-17T10:25:01.638369+0000","last_clean":"2024-04-17T10:25:01.638369+0000","last_became_active":"2024-04-17T10:24:54.607576+0000","last_became_peered":"2024-04-17T10:24:54.607576+0000","last_unstale":"2024-04-17T10:25:01.638369+0000","last_undegraded":"2024-04-17T10:25:01.638369+0000","last_fullsized":"2024-04-17T10:25:01.638369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156289+0000","last_change":"2024-04-17T10:24:55.628197+0000","last_active":"2024-04-17T10:25:03.156289+0000","last_peered":"2024-04-17T10:25:03.156289+0000","last_clean":"2024-04-17T10:25:03.156289+0000","last_became_active":"2024-04-17T10:24:55.627976+0000","last_became_peered":"2024-04-17T10:24:55.627976+0000","last_unstale":"2024-04-17T10:25:03.156289+0000","last_undegraded":"2024-04-17T10:25:03.156289+0000","last_fullsized":"2024-04-17T10:25:03.156289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636562+0000","last_change":"2024-04-17T10:24:59.651727+0000","last_active":"2024-04-17T10:25:01.636562+0000","last_peered":"2024-04-17T10:25:01.636562+0000","last_clean":"2024-04-17T10:25:01.636562+0000","last_became_active":"2024-04-17T10:24:59.651358+0000","last_became_peered":"2024-04-17T10:24:59.651358+0000","last_unstale":"2024-04-17T10:25:01.636562+0000","last_undegraded":"2024-04-17T10:25:01.636562+0000","last_fullsized":"2024-04-17T10:25:01.636562+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423903+0000","last_change":"2024-04-17T10:24:54.611808+0000","last_active":"2024-04-17T10:25:02.423903+0000","last_peered":"2024-04-17T10:25:02.423903+0000","last_clean":"2024-04-17T10:25:02.423903+0000","last_became_active":"2024-04-17T10:24:54.610903+0000","last_became_peered":"2024-04-17T10:24:54.610903+0000","last_unstale":"2024-04-17T10:25:02.423903+0000","last_undegraded":"2024-04-17T10:25:02.423903+0000","last_fullsized":"2024-04-17T10:25:02.423903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:09.085810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268578+0000","last_change":"2024-04-17T10:24:55.618364+0000","last_active":"2024-04-17T10:25:03.268578+0000","last_peered":"2024-04-17T10:25:03.268578+0000","last_clean":"2024-04-17T10:25:03.268578+0000","last_became_active":"2024-04-17T10:24:55.618013+0000","last_became_peered":"2024-04-17T10:24:55.618013+0000","last_unstale":"2024-04-17T10:25:03.268578+0000","last_undegraded":"2024-04-17T10:25:03.268578+0000","last_fullsized":"2024-04-17T10:25:03.268578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636861+0000","last_change":"2024-04-17T10:24:59.647051+0000","last_active":"2024-04-17T10:25:01.636861+0000","last_peered":"2024-04-17T10:25:01.636861+0000","last_clean":"2024-04-17T10:25:01.636861+0000","last_became_active":"2024-04-17T10:24:59.646611+0000","last_became_peered":"2024-04-17T10:24:59.646611+0000","last_unstale":"2024-04-17T10:25:01.636861+0000","last_undegraded":"2024-04-17T10:25:01.636861+0000","last_fullsized":"2024-04-17T10:25:01.636861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423712+0000","last_change":"2024-04-17T10:24:54.611301+0000","last_active":"2024-04-17T10:25:02.423712+0000","last_peered":"2024-04-17T10:25:02.423712+0000","last_clean":"2024-04-17T10:25:02.423712+0000","last_became_active":"2024-04-17T10:24:54.610243+0000","last_became_peered":"2024-04-17T10:24:54.610243+0000","last_unstale":"2024-04-17T10:25:02.423712+0000","last_undegraded":"2024-04-17T10:25:02.423712+0000","last_fullsized":"2024-04-17T10:25:02.423712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:36.054430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268618+0000","last_change":"2024-04-17T10:24:55.618112+0000","last_active":"2024-04-17T10:25:03.268618+0000","last_peered":"2024-04-17T10:25:03.268618+0000","last_clean":"2024-04-17T10:25:03.268618+0000","last_became_active":"2024-04-17T10:24:55.617127+0000","last_became_peered":"2024-04-17T10:24:55.617127+0000","last_unstale":"2024-04-17T10:25:03.268618+0000","last_undegraded":"2024-04-17T10:25:03.268618+0000","last_fullsized":"2024-04-17T10:25:03.268618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636698+0000","last_change":"2024-04-17T10:24:59.641046+0000","last_active":"2024-04-17T10:25:01.636698+0000","last_peered":"2024-04-17T10:25:01.636698+0000","last_clean":"2024-04-17T10:25:01.636698+0000","last_became_active":"2024-04-17T10:24:59.640622+0000","last_became_peered":"2024-04-17T10:24:59.640622+0000","last_unstale":"2024-04-17T10:25:01.636698+0000","last_undegraded":"2024-04-17T10:25:01.636698+0000","last_fullsized":"2024-04-17T10:25:01.636698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537179+0000","last_change":"2024-04-17T10:24:55.617061+0000","last_active":"2024-04-17T10:25:02.537179+0000","last_peered":"2024-04-17T10:25:02.537179+0000","last_clean":"2024-04-17T10:25:02.537179+0000","last_became_active":"2024-04-17T10:24:55.616809+0000","last_became_peered":"2024-04-17T10:24:55.616809+0000","last_unstale":"2024-04-17T10:25:02.537179+0000","last_undegraded":"2024-04-17T10:25:02.537179+0000","last_fullsized":"2024-04-17T10:25:02.537179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423815+0000","last_change":"2024-04-17T10:24:54.607974+0000","last_active":"2024-04-17T10:25:02.423815+0000","last_peered":"2024-04-17T10:25:02.423815+0000","last_clean":"2024-04-17T10:25:02.423815+0000","last_became_active":"2024-04-17T10:24:54.606303+0000","last_became_peered":"2024-04-17T10:24:54.606303+0000","last_unstale":"2024-04-17T10:25:02.423815+0000","last_undegraded":"2024-04-17T10:25:02.423815+0000","last_fullsized":"2024-04-17T10:25:02.423815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:52.381704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637318+0000","last_change":"2024-04-17T10:24:59.645993+0000","last_active":"2024-04-17T10:25:01.637318+0000","last_peered":"2024-04-17T10:25:01.637318+0000","last_clean":"2024-04-17T10:25:01.637318+0000","last_became_active":"2024-04-17T10:24:59.645305+0000","last_became_peered":"2024-04-17T10:24:59.645305+0000","last_unstale":"2024-04-17T10:25:01.637318+0000","last_undegraded":"2024-04-17T10:25:01.637318+0000","last_fullsized":"2024-04-17T10:25:01.637318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638467+0000","last_change":"2024-04-17T10:24:55.614771+0000","last_active":"2024-04-17T10:25:01.638467+0000","last_peered":"2024-04-17T10:25:01.638467+0000","last_clean":"2024-04-17T10:25:01.638467+0000","last_became_active":"2024-04-17T10:24:55.614517+0000","last_became_peered":"2024-04-17T10:24:55.614517+0000","last_unstale":"2024-04-17T10:25:01.638467+0000","last_undegraded":"2024-04-17T10:25:01.638467+0000","last_fullsized":"2024-04-17T10:25:01.638467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636723+0000","last_change":"2024-04-17T10:24:54.608749+0000","last_active":"2024-04-17T10:25:01.636723+0000","last_peered":"2024-04-17T10:25:01.636723+0000","last_clean":"2024-04-17T10:25:01.636723+0000","last_became_active":"2024-04-17T10:24:54.606617+0000","last_became_peered":"2024-04-17T10:24:54.606617+0000","last_unstale":"2024-04-17T10:25:01.636723+0000","last_undegraded":"2024-04-17T10:25:01.636723+0000","last_fullsized":"2024-04-17T10:25:01.636723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636946+0000","last_change":"2024-04-17T10:24:59.645457+0000","last_active":"2024-04-17T10:25:01.636946+0000","last_peered":"2024-04-17T10:25:01.636946+0000","last_clean":"2024-04-17T10:25:01.636946+0000","last_became_active":"2024-04-17T10:24:59.644747+0000","last_became_peered":"2024-04-17T10:24:59.644747+0000","last_unstale":"2024-04-17T10:25:01.636946+0000","last_undegraded":"2024-04-17T10:25:01.636946+0000","last_fullsized":"2024-04-17T10:25:01.636946+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637820+0000","last_change":"2024-04-17T10:24:55.630638+0000","last_active":"2024-04-17T10:25:01.637820+0000","last_peered":"2024-04-17T10:25:01.637820+0000","last_clean":"2024-04-17T10:25:01.637820+0000","last_became_active":"2024-04-17T10:24:55.630351+0000","last_became_peered":"2024-04-17T10:24:55.630351+0000","last_unstale":"2024-04-17T10:25:01.637820+0000","last_undegraded":"2024-04-17T10:25:01.637820+0000","last_fullsized":"2024-04-17T10:25:01.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636842+0000","last_change":"2024-04-17T10:24:54.609635+0000","last_active":"2024-04-17T10:25:01.636842+0000","last_peered":"2024-04-17T10:25:01.636842+0000","last_clean":"2024-04-17T10:25:01.636842+0000","last_became_active":"2024-04-17T10:24:54.607298+0000","last_became_peered":"2024-04-17T10:24:54.607298+0000","last_unstale":"2024-04-17T10:25:01.636842+0000","last_undegraded":"2024-04-17T10:25:01.636842+0000","last_fullsized":"2024-04-17T10:25:01.636842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637200+0000","last_change":"2024-04-17T10:25:00.243112+0000","last_active":"2024-04-17T10:25:01.637200+0000","last_peered":"2024-04-17T10:25:01.637200+0000","last_clean":"2024-04-17T10:25:01.637200+0000","last_became_active":"2024-04-17T10:25:00.242669+0000","last_became_peered":"2024-04-17T10:25:00.242669+0000","last_unstale":"2024-04-17T10:25:01.637200+0000","last_undegraded":"2024-04-17T10:25:01.637200+0000","last_fullsized":"2024-04-17T10:25:01.637200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637134+0000","last_change":"2024-04-17T10:24:55.608101+0000","last_active":"2024-04-17T10:25:01.637134+0000","last_peered":"2024-04-17T10:25:01.637134+0000","last_clean":"2024-04-17T10:25:01.637134+0000","last_became_active":"2024-04-17T10:24:55.607668+0000","last_became_peered":"2024-04-17T10:24:55.607668+0000","last_unstale":"2024-04-17T10:25:01.637134+0000","last_undegraded":"2024-04-17T10:25:01.637134+0000","last_fullsized":"2024-04-17T10:25:01.637134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:35.913082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423205+0000","last_change":"2024-04-17T10:24:54.607981+0000","last_active":"2024-04-17T10:25:02.423205+0000","last_peered":"2024-04-17T10:25:02.423205+0000","last_clean":"2024-04-17T10:25:02.423205+0000","last_became_active":"2024-04-17T10:24:54.607150+0000","last_became_peered":"2024-04-17T10:24:54.607150+0000","last_unstale":"2024-04-17T10:25:02.423205+0000","last_undegraded":"2024-04-17T10:25:02.423205+0000","last_fullsized":"2024-04-17T10:25:02.423205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636873+0000","last_change":"2024-04-17T10:24:59.645029+0000","last_active":"2024-04-17T10:25:01.636873+0000","last_peered":"2024-04-17T10:25:01.636873+0000","last_clean":"2024-04-17T10:25:01.636873+0000","last_became_active":"2024-04-17T10:24:59.644540+0000","last_became_peered":"2024-04-17T10:24:59.644540+0000","last_unstale":"2024-04-17T10:25:01.636873+0000","last_undegraded":"2024-04-17T10:25:01.636873+0000","last_fullsized":"2024-04-17T10:25:01.636873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156445+0000","last_change":"2024-04-17T10:24:55.610920+0000","last_active":"2024-04-17T10:25:03.156445+0000","last_peered":"2024-04-17T10:25:03.156445+0000","last_clean":"2024-04-17T10:25:03.156445+0000","last_became_active":"2024-04-17T10:24:55.610566+0000","last_became_peered":"2024-04-17T10:24:55.610566+0000","last_unstale":"2024-04-17T10:25:03.156445+0000","last_undegraded":"2024-04-17T10:25:03.156445+0000","last_fullsized":"2024-04-17T10:25:03.156445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537952+0000","last_change":"2024-04-17T10:24:54.609866+0000","last_active":"2024-04-17T10:25:02.537952+0000","last_peered":"2024-04-17T10:25:02.537952+0000","last_clean":"2024-04-17T10:25:02.537952+0000","last_became_active":"2024-04-17T10:24:54.607310+0000","last_became_peered":"2024-04-17T10:24:54.607310+0000","last_unstale":"2024-04-17T10:25:02.537952+0000","last_undegraded":"2024-04-17T10:25:02.537952+0000","last_fullsized":"2024-04-17T10:25:02.537952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637020+0000","last_change":"2024-04-17T10:25:00.242151+0000","last_active":"2024-04-17T10:25:01.637020+0000","last_peered":"2024-04-17T10:25:01.637020+0000","last_clean":"2024-04-17T10:25:01.637020+0000","last_became_active":"2024-04-17T10:25:00.241789+0000","last_became_peered":"2024-04-17T10:25:00.241789+0000","last_unstale":"2024-04-17T10:25:01.637020+0000","last_undegraded":"2024-04-17T10:25:01.637020+0000","last_fullsized":"2024-04-17T10:25:01.637020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638591+0000","last_change":"2024-04-17T10:24:55.611037+0000","last_active":"2024-04-17T10:25:01.638591+0000","last_peered":"2024-04-17T10:25:01.638591+0000","last_clean":"2024-04-17T10:25:01.638591+0000","last_became_active":"2024-04-17T10:24:55.610713+0000","last_became_peered":"2024-04-17T10:24:55.610713+0000","last_unstale":"2024-04-17T10:25:01.638591+0000","last_undegraded":"2024-04-17T10:25:01.638591+0000","last_fullsized":"2024-04-17T10:25:01.638591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636990+0000","last_change":"2024-04-17T10:24:54.614299+0000","last_active":"2024-04-17T10:25:01.636990+0000","last_peered":"2024-04-17T10:25:01.636990+0000","last_clean":"2024-04-17T10:25:01.636990+0000","last_became_active":"2024-04-17T10:24:54.613957+0000","last_became_peered":"2024-04-17T10:24:54.613957+0000","last_unstale":"2024-04-17T10:25:01.636990+0000","last_undegraded":"2024-04-17T10:25:01.636990+0000","last_fullsized":"2024-04-17T10:25:01.636990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637684+0000","last_change":"2024-04-17T10:25:00.241327+0000","last_active":"2024-04-17T10:25:01.637684+0000","last_peered":"2024-04-17T10:25:01.637684+0000","last_clean":"2024-04-17T10:25:01.637684+0000","last_became_active":"2024-04-17T10:25:00.240884+0000","last_became_peered":"2024-04-17T10:25:00.240884+0000","last_unstale":"2024-04-17T10:25:01.637684+0000","last_undegraded":"2024-04-17T10:25:01.637684+0000","last_fullsized":"2024-04-17T10:25:01.637684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:46.933770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.624025+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.623725+0000","last_became_peered":"2024-04-17T10:24:55.623725+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424055+0000","last_change":"2024-04-17T10:24:54.610613+0000","last_active":"2024-04-17T10:25:02.424055+0000","last_peered":"2024-04-17T10:25:02.424055+0000","last_clean":"2024-04-17T10:25:02.424055+0000","last_became_active":"2024-04-17T10:24:54.609405+0000","last_became_peered":"2024-04-17T10:24:54.609405+0000","last_unstale":"2024-04-17T10:25:02.424055+0000","last_undegraded":"2024-04-17T10:25:02.424055+0000","last_fullsized":"2024-04-17T10:25:02.424055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:06.248042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155827+0000","last_change":"2024-04-17T10:25:00.243396+0000","last_active":"2024-04-17T10:25:03.155827+0000","last_peered":"2024-04-17T10:25:03.155827+0000","last_clean":"2024-04-17T10:25:03.155827+0000","last_became_active":"2024-04-17T10:25:00.243086+0000","last_became_peered":"2024-04-17T10:25:00.243086+0000","last_unstale":"2024-04-17T10:25:03.155827+0000","last_undegraded":"2024-04-17T10:25:03.155827+0000","last_fullsized":"2024-04-17T10:25:03.155827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268735+0000","last_change":"2024-04-17T10:24:55.611872+0000","last_active":"2024-04-17T10:25:03.268735+0000","last_peered":"2024-04-17T10:25:03.268735+0000","last_clean":"2024-04-17T10:25:03.268735+0000","last_became_active":"2024-04-17T10:24:55.607201+0000","last_became_peered":"2024-04-17T10:24:55.607201+0000","last_unstale":"2024-04-17T10:25:03.268735+0000","last_undegraded":"2024-04-17T10:25:03.268735+0000","last_fullsized":"2024-04-17T10:25:03.268735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637222+0000","last_change":"2024-04-17T10:24:54.598254+0000","last_active":"2024-04-17T10:25:01.637222+0000","last_peered":"2024-04-17T10:25:01.637222+0000","last_clean":"2024-04-17T10:25:01.637222+0000","last_became_active":"2024-04-17T10:24:54.597869+0000","last_became_peered":"2024-04-17T10:24:54.597869+0000","last_unstale":"2024-04-17T10:25:01.637222+0000","last_undegraded":"2024-04-17T10:25:01.637222+0000","last_fullsized":"2024-04-17T10:25:01.637222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636567+0000","last_change":"2024-04-17T10:24:59.656986+0000","last_active":"2024-04-17T10:25:01.636567+0000","last_peered":"2024-04-17T10:25:01.636567+0000","last_clean":"2024-04-17T10:25:01.636567+0000","last_became_active":"2024-04-17T10:24:59.656124+0000","last_became_peered":"2024-04-17T10:24:59.656124+0000","last_unstale":"2024-04-17T10:25:01.636567+0000","last_undegraded":"2024-04-17T10:25:01.636567+0000","last_fullsized":"2024-04-17T10:25:01.636567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637349+0000","last_change":"2024-04-17T10:24:55.625808+0000","last_active":"2024-04-17T10:25:01.637349+0000","last_peered":"2024-04-17T10:25:01.637349+0000","last_clean":"2024-04-17T10:25:01.637349+0000","last_became_active":"2024-04-17T10:24:55.625551+0000","last_became_peered":"2024-04-17T10:24:55.625551+0000","last_unstale":"2024-04-17T10:25:01.637349+0000","last_undegraded":"2024-04-17T10:25:01.637349+0000","last_fullsized":"2024-04-17T10:25:01.637349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267481+0000","last_change":"2024-04-17T10:24:54.603615+0000","last_active":"2024-04-17T10:25:03.267481+0000","last_peered":"2024-04-17T10:25:03.267481+0000","last_clean":"2024-04-17T10:25:03.267481+0000","last_became_active":"2024-04-17T10:24:54.600774+0000","last_became_peered":"2024-04-17T10:24:54.600774+0000","last_unstale":"2024-04-17T10:25:03.267481+0000","last_undegraded":"2024-04-17T10:25:03.267481+0000","last_fullsized":"2024-04-17T10:25:03.267481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637164+0000","last_change":"2024-04-17T10:24:59.646484+0000","last_active":"2024-04-17T10:25:01.637164+0000","last_peered":"2024-04-17T10:25:01.637164+0000","last_clean":"2024-04-17T10:25:01.637164+0000","last_became_active":"2024-04-17T10:24:59.646061+0000","last_became_peered":"2024-04-17T10:24:59.646061+0000","last_unstale":"2024-04-17T10:25:01.637164+0000","last_undegraded":"2024-04-17T10:25:01.637164+0000","last_fullsized":"2024-04-17T10:25:01.637164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636232+0000","last_change":"2024-04-17T10:24:55.624915+0000","last_active":"2024-04-17T10:25:01.636232+0000","last_peered":"2024-04-17T10:25:01.636232+0000","last_clean":"2024-04-17T10:25:01.636232+0000","last_became_active":"2024-04-17T10:24:55.624230+0000","last_became_peered":"2024-04-17T10:24:55.624230+0000","last_unstale":"2024-04-17T10:25:01.636232+0000","last_undegraded":"2024-04-17T10:25:01.636232+0000","last_fullsized":"2024-04-17T10:25:01.636232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624893+0000","last_change":"2024-04-17T10:24:54.599766+0000","last_active":"2024-04-17T10:25:02.624893+0000","last_peered":"2024-04-17T10:25:02.624893+0000","last_clean":"2024-04-17T10:25:02.624893+0000","last_became_active":"2024-04-17T10:24:54.599476+0000","last_became_peered":"2024-04-17T10:24:54.599476+0000","last_unstale":"2024-04-17T10:25:02.624893+0000","last_undegraded":"2024-04-17T10:25:02.624893+0000","last_fullsized":"2024-04-17T10:25:02.624893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637263+0000","last_change":"2024-04-17T10:24:59.656597+0000","last_active":"2024-04-17T10:25:01.637263+0000","last_peered":"2024-04-17T10:25:01.637263+0000","last_clean":"2024-04-17T10:25:01.637263+0000","last_became_active":"2024-04-17T10:24:59.656135+0000","last_became_peered":"2024-04-17T10:24:59.656135+0000","last_unstale":"2024-04-17T10:25:01.637263+0000","last_undegraded":"2024-04-17T10:25:01.637263+0000","last_fullsized":"2024-04-17T10:25:01.637263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155614+0000","last_change":"2024-04-17T10:24:55.591986+0000","last_active":"2024-04-17T10:25:03.155614+0000","last_peered":"2024-04-17T10:25:03.155614+0000","last_clean":"2024-04-17T10:25:03.155614+0000","last_became_active":"2024-04-17T10:24:55.591672+0000","last_became_peered":"2024-04-17T10:24:55.591672+0000","last_unstale":"2024-04-17T10:25:03.155614+0000","last_undegraded":"2024-04-17T10:25:03.155614+0000","last_fullsized":"2024-04-17T10:25:03.155614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423263+0000","last_change":"2024-04-17T10:24:54.614030+0000","last_active":"2024-04-17T10:25:02.423263+0000","last_peered":"2024-04-17T10:25:02.423263+0000","last_clean":"2024-04-17T10:25:02.423263+0000","last_became_active":"2024-04-17T10:24:54.613752+0000","last_became_peered":"2024-04-17T10:24:54.613752+0000","last_unstale":"2024-04-17T10:25:02.423263+0000","last_undegraded":"2024-04-17T10:25:02.423263+0000","last_fullsized":"2024-04-17T10:25:02.423263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156369+0000","last_change":"2024-04-17T10:25:00.244705+0000","last_active":"2024-04-17T10:25:03.156369+0000","last_peered":"2024-04-17T10:25:03.156369+0000","last_clean":"2024-04-17T10:25:03.156369+0000","last_became_active":"2024-04-17T10:25:00.243853+0000","last_became_peered":"2024-04-17T10:25:00.243853+0000","last_unstale":"2024-04-17T10:25:03.156369+0000","last_undegraded":"2024-04-17T10:25:03.156369+0000","last_fullsized":"2024-04-17T10:25:03.156369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638050+0000","last_change":"2024-04-17T10:24:55.622500+0000","last_active":"2024-04-17T10:25:01.638050+0000","last_peered":"2024-04-17T10:25:01.638050+0000","last_clean":"2024-04-17T10:25:01.638050+0000","last_became_active":"2024-04-17T10:24:55.622206+0000","last_became_peered":"2024-04-17T10:24:55.622206+0000","last_unstale":"2024-04-17T10:25:01.638050+0000","last_undegraded":"2024-04-17T10:25:01.638050+0000","last_fullsized":"2024-04-17T10:25:01.638050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268327+0000","last_change":"2024-04-17T10:24:54.601255+0000","last_active":"2024-04-17T10:25:03.268327+0000","last_peered":"2024-04-17T10:25:03.268327+0000","last_clean":"2024-04-17T10:25:03.268327+0000","last_became_active":"2024-04-17T10:24:54.600968+0000","last_became_peered":"2024-04-17T10:24:54.600968+0000","last_unstale":"2024-04-17T10:25:03.268327+0000","last_undegraded":"2024-04-17T10:25:03.268327+0000","last_fullsized":"2024-04-17T10:25:03.268327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423358+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:25:02.423358+0000","last_peered":"2024-04-17T10:25:02.423358+0000","last_clean":"2024-04-17T10:25:02.423358+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:25:02.423358+0000","last_undegraded":"2024-04-17T10:25:02.423358+0000","last_fullsized":"2024-04-17T10:25:02.423358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537629+0000","last_change":"2024-04-17T10:25:00.247011+0000","last_active":"2024-04-17T10:25:02.537629+0000","last_peered":"2024-04-17T10:25:02.537629+0000","last_clean":"2024-04-17T10:25:02.537629+0000","last_became_active":"2024-04-17T10:25:00.245780+0000","last_became_peered":"2024-04-17T10:25:00.245780+0000","last_unstale":"2024-04-17T10:25:02.537629+0000","last_undegraded":"2024-04-17T10:25:02.537629+0000","last_fullsized":"2024-04-17T10:25:02.537629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423766+0000","last_change":"2024-04-17T10:24:55.613528+0000","last_active":"2024-04-17T10:25:02.423766+0000","last_peered":"2024-04-17T10:25:02.423766+0000","last_clean":"2024-04-17T10:25:02.423766+0000","last_became_active":"2024-04-17T10:24:55.613203+0000","last_became_peered":"2024-04-17T10:24:55.613203+0000","last_unstale":"2024-04-17T10:25:02.423766+0000","last_undegraded":"2024-04-17T10:25:02.423766+0000","last_fullsized":"2024-04-17T10:25:02.423766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:15.389666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625258+0000","last_change":"2024-04-17T10:24:54.605199+0000","last_active":"2024-04-17T10:25:02.625258+0000","last_peered":"2024-04-17T10:25:02.625258+0000","last_clean":"2024-04-17T10:25:02.625258+0000","last_became_active":"2024-04-17T10:24:54.604881+0000","last_became_peered":"2024-04-17T10:24:54.604881+0000","last_unstale":"2024-04-17T10:25:02.625258+0000","last_undegraded":"2024-04-17T10:25:02.625258+0000","last_fullsized":"2024-04-17T10:25:02.625258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537487+0000","last_change":"2024-04-17T10:25:00.242786+0000","last_active":"2024-04-17T10:25:02.537487+0000","last_peered":"2024-04-17T10:25:02.537487+0000","last_clean":"2024-04-17T10:25:02.537487+0000","last_became_active":"2024-04-17T10:25:00.242222+0000","last_became_peered":"2024-04-17T10:25:00.242222+0000","last_unstale":"2024-04-17T10:25:02.537487+0000","last_undegraded":"2024-04-17T10:25:02.537487+0000","last_fullsized":"2024-04-17T10:25:02.537487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:01.409834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268628+0000","last_change":"2024-04-17T10:24:55.606154+0000","last_active":"2024-04-17T10:25:03.268628+0000","last_peered":"2024-04-17T10:25:03.268628+0000","last_clean":"2024-04-17T10:25:03.268628+0000","last_became_active":"2024-04-17T10:24:55.605854+0000","last_became_peered":"2024-04-17T10:24:55.605854+0000","last_unstale":"2024-04-17T10:25:03.268628+0000","last_undegraded":"2024-04-17T10:25:03.268628+0000","last_fullsized":"2024-04-17T10:25:03.268628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636665+0000","last_change":"2024-04-17T10:24:54.605549+0000","last_active":"2024-04-17T10:25:01.636665+0000","last_peered":"2024-04-17T10:25:01.636665+0000","last_clean":"2024-04-17T10:25:01.636665+0000","last_became_active":"2024-04-17T10:24:54.605197+0000","last_became_peered":"2024-04-17T10:24:54.605197+0000","last_unstale":"2024-04-17T10:25:01.636665+0000","last_undegraded":"2024-04-17T10:25:01.636665+0000","last_fullsized":"2024-04-17T10:25:01.636665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155960+0000","last_change":"2024-04-17T10:24:59.652908+0000","last_active":"2024-04-17T10:25:03.155960+0000","last_peered":"2024-04-17T10:25:03.155960+0000","last_clean":"2024-04-17T10:25:03.155960+0000","last_became_active":"2024-04-17T10:24:59.652604+0000","last_became_peered":"2024-04-17T10:24:59.652604+0000","last_unstale":"2024-04-17T10:25:03.155960+0000","last_undegraded":"2024-04-17T10:25:03.155960+0000","last_fullsized":"2024-04-17T10:25:03.155960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424244+0000","last_change":"2024-04-17T10:24:55.612380+0000","last_active":"2024-04-17T10:25:02.424244+0000","last_peered":"2024-04-17T10:25:02.424244+0000","last_clean":"2024-04-17T10:25:02.424244+0000","last_became_active":"2024-04-17T10:24:55.611511+0000","last_became_peered":"2024-04-17T10:24:55.611511+0000","last_unstale":"2024-04-17T10:25:02.424244+0000","last_undegraded":"2024-04-17T10:25:02.424244+0000","last_fullsized":"2024-04-17T10:25:02.424244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:49.176910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624922+0000","last_change":"2024-04-17T10:24:54.601414+0000","last_active":"2024-04-17T10:25:02.624922+0000","last_peered":"2024-04-17T10:25:02.624922+0000","last_clean":"2024-04-17T10:25:02.624922+0000","last_became_active":"2024-04-17T10:24:54.600297+0000","last_became_peered":"2024-04-17T10:24:54.600297+0000","last_unstale":"2024-04-17T10:25:02.624922+0000","last_undegraded":"2024-04-17T10:25:02.624922+0000","last_fullsized":"2024-04-17T10:25:02.624922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267910+0000","last_change":"2024-04-17T10:25:00.242716+0000","last_active":"2024-04-17T10:25:03.267910+0000","last_peered":"2024-04-17T10:25:03.267910+0000","last_clean":"2024-04-17T10:25:03.267910+0000","last_became_active":"2024-04-17T10:25:00.242357+0000","last_became_peered":"2024-04-17T10:25:00.242357+0000","last_unstale":"2024-04-17T10:25:03.267910+0000","last_undegraded":"2024-04-17T10:25:03.267910+0000","last_fullsized":"2024-04-17T10:25:03.267910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424088+0000","last_change":"2024-04-17T10:24:55.604575+0000","last_active":"2024-04-17T10:25:02.424088+0000","last_peered":"2024-04-17T10:25:02.424088+0000","last_clean":"2024-04-17T10:25:02.424088+0000","last_became_active":"2024-04-17T10:24:55.604252+0000","last_became_peered":"2024-04-17T10:24:55.604252+0000","last_unstale":"2024-04-17T10:25:02.424088+0000","last_undegraded":"2024-04-17T10:25:02.424088+0000","last_fullsized":"2024-04-17T10:25:02.424088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:02.059745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637103+0000","last_change":"2024-04-17T10:24:54.595839+0000","last_active":"2024-04-17T10:25:01.637103+0000","last_peered":"2024-04-17T10:25:01.637103+0000","last_clean":"2024-04-17T10:25:01.637103+0000","last_became_active":"2024-04-17T10:24:54.594547+0000","last_became_peered":"2024-04-17T10:24:54.594547+0000","last_unstale":"2024-04-17T10:25:01.637103+0000","last_undegraded":"2024-04-17T10:25:01.637103+0000","last_fullsized":"2024-04-17T10:25:01.637103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636525+0000","last_change":"2024-04-17T10:24:59.644365+0000","last_active":"2024-04-17T10:25:01.636525+0000","last_peered":"2024-04-17T10:25:01.636525+0000","last_clean":"2024-04-17T10:25:01.636525+0000","last_became_active":"2024-04-17T10:24:59.643969+0000","last_became_peered":"2024-04-17T10:24:59.643969+0000","last_unstale":"2024-04-17T10:25:01.636525+0000","last_undegraded":"2024-04-17T10:25:01.636525+0000","last_fullsized":"2024-04-17T10:25:01.636525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423432+0000","last_change":"2024-04-17T10:24:55.598627+0000","last_active":"2024-04-17T10:25:02.423432+0000","last_peered":"2024-04-17T10:25:02.423432+0000","last_clean":"2024-04-17T10:25:02.423432+0000","last_became_active":"2024-04-17T10:24:55.598319+0000","last_became_peered":"2024-04-17T10:24:55.598319+0000","last_unstale":"2024-04-17T10:25:02.423432+0000","last_undegraded":"2024-04-17T10:25:02.423432+0000","last_fullsized":"2024-04-17T10:25:02.423432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624900+0000","last_change":"2024-04-17T10:24:54.600358+0000","last_active":"2024-04-17T10:25:02.624900+0000","last_peered":"2024-04-17T10:25:02.624900+0000","last_clean":"2024-04-17T10:25:02.624900+0000","last_became_active":"2024-04-17T10:24:54.600064+0000","last_became_peered":"2024-04-17T10:24:54.600064+0000","last_unstale":"2024-04-17T10:25:02.624900+0000","last_undegraded":"2024-04-17T10:25:02.624900+0000","last_fullsized":"2024-04-17T10:25:02.624900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156321+0000","last_change":"2024-04-17T10:24:59.651264+0000","last_active":"2024-04-17T10:25:03.156321+0000","last_peered":"2024-04-17T10:25:03.156321+0000","last_clean":"2024-04-17T10:25:03.156321+0000","last_became_active":"2024-04-17T10:24:59.650654+0000","last_became_peered":"2024-04-17T10:24:59.650654+0000","last_unstale":"2024-04-17T10:25:03.156321+0000","last_undegraded":"2024-04-17T10:25:03.156321+0000","last_fullsized":"2024-04-17T10:25:03.156321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537254+0000","last_change":"2024-04-17T10:24:55.608832+0000","last_active":"2024-04-17T10:25:02.537254+0000","last_peered":"2024-04-17T10:25:02.537254+0000","last_clean":"2024-04-17T10:25:02.537254+0000","last_became_active":"2024-04-17T10:24:55.608402+0000","last_became_peered":"2024-04-17T10:24:55.608402+0000","last_unstale":"2024-04-17T10:25:02.537254+0000","last_undegraded":"2024-04-17T10:25:02.537254+0000","last_fullsized":"2024-04-17T10:25:02.537254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537354+0000","last_change":"2024-04-17T10:24:54.601967+0000","last_active":"2024-04-17T10:25:02.537354+0000","last_peered":"2024-04-17T10:25:02.537354+0000","last_clean":"2024-04-17T10:25:02.537354+0000","last_became_active":"2024-04-17T10:24:54.601643+0000","last_became_peered":"2024-04-17T10:24:54.601643+0000","last_unstale":"2024-04-17T10:25:02.537354+0000","last_undegraded":"2024-04-17T10:25:02.537354+0000","last_fullsized":"2024-04-17T10:25:02.537354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638493+0000","last_change":"2024-04-17T10:25:00.243903+0000","last_active":"2024-04-17T10:25:01.638493+0000","last_peered":"2024-04-17T10:25:01.638493+0000","last_clean":"2024-04-17T10:25:01.638493+0000","last_became_active":"2024-04-17T10:25:00.243126+0000","last_became_peered":"2024-04-17T10:25:00.243126+0000","last_unstale":"2024-04-17T10:25:01.638493+0000","last_undegraded":"2024-04-17T10:25:01.638493+0000","last_fullsized":"2024-04-17T10:25:01.638493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:44.715063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423617+0000","last_change":"2024-04-17T10:24:55.598632+0000","last_active":"2024-04-17T10:25:02.423617+0000","last_peered":"2024-04-17T10:25:02.423617+0000","last_clean":"2024-04-17T10:25:02.423617+0000","last_became_active":"2024-04-17T10:24:55.598302+0000","last_became_peered":"2024-04-17T10:24:55.598302+0000","last_unstale":"2024-04-17T10:25:02.423617+0000","last_undegraded":"2024-04-17T10:25:02.423617+0000","last_fullsized":"2024-04-17T10:25:02.423617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267532+0000","last_change":"2024-04-17T10:24:54.603334+0000","last_active":"2024-04-17T10:25:03.267532+0000","last_peered":"2024-04-17T10:25:03.267532+0000","last_clean":"2024-04-17T10:25:03.267532+0000","last_became_active":"2024-04-17T10:24:54.603066+0000","last_became_peered":"2024-04-17T10:24:54.603066+0000","last_unstale":"2024-04-17T10:25:03.267532+0000","last_undegraded":"2024-04-17T10:25:03.267532+0000","last_fullsized":"2024-04-17T10:25:03.267532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639048+0000","last_change":"2024-04-17T10:25:00.247479+0000","last_active":"2024-04-17T10:25:01.639048+0000","last_peered":"2024-04-17T10:25:01.639048+0000","last_clean":"2024-04-17T10:25:01.639048+0000","last_became_active":"2024-04-17T10:25:00.247066+0000","last_became_peered":"2024-04-17T10:25:00.247066+0000","last_unstale":"2024-04-17T10:25:01.639048+0000","last_undegraded":"2024-04-17T10:25:01.639048+0000","last_fullsized":"2024-04-17T10:25:01.639048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:10:54.855530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155768+0000","last_change":"2024-04-17T10:24:55.616361+0000","last_active":"2024-04-17T10:25:03.155768+0000","last_peered":"2024-04-17T10:25:03.155768+0000","last_clean":"2024-04-17T10:25:03.155768+0000","last_became_active":"2024-04-17T10:24:55.615902+0000","last_became_peered":"2024-04-17T10:24:55.615902+0000","last_unstale":"2024-04-17T10:25:03.155768+0000","last_undegraded":"2024-04-17T10:25:03.155768+0000","last_fullsized":"2024-04-17T10:25:03.155768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:53.388065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155191+0000","last_change":"2024-04-17T10:24:54.599143+0000","last_active":"2024-04-17T10:25:03.155191+0000","last_peered":"2024-04-17T10:25:03.155191+0000","last_clean":"2024-04-17T10:25:03.155191+0000","last_became_active":"2024-04-17T10:24:54.598831+0000","last_became_peered":"2024-04-17T10:24:54.598831+0000","last_unstale":"2024-04-17T10:25:03.155191+0000","last_undegraded":"2024-04-17T10:25:03.155191+0000","last_fullsized":"2024-04-17T10:25:03.155191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268682+0000","last_change":"2024-04-17T10:25:00.243074+0000","last_active":"2024-04-17T10:25:03.268682+0000","last_peered":"2024-04-17T10:25:03.268682+0000","last_clean":"2024-04-17T10:25:03.268682+0000","last_became_active":"2024-04-17T10:25:00.242532+0000","last_became_peered":"2024-04-17T10:25:00.242532+0000","last_unstale":"2024-04-17T10:25:03.268682+0000","last_undegraded":"2024-04-17T10:25:03.268682+0000","last_fullsized":"2024-04-17T10:25:03.268682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423646+0000","last_change":"2024-04-17T10:24:55.603471+0000","last_active":"2024-04-17T10:25:02.423646+0000","last_peered":"2024-04-17T10:25:02.423646+0000","last_clean":"2024-04-17T10:25:02.423646+0000","last_became_active":"2024-04-17T10:24:55.603177+0000","last_became_peered":"2024-04-17T10:24:55.603177+0000","last_unstale":"2024-04-17T10:25:02.423646+0000","last_undegraded":"2024-04-17T10:25:02.423646+0000","last_fullsized":"2024-04-17T10:25:02.423646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:52.938391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:10.622101+0000","last_change":"2024-04-17T10:24:54.599941+0000","last_active":"2024-04-17T10:25:10.622101+0000","last_peered":"2024-04-17T10:25:10.622101+0000","last_clean":"2024-04-17T10:25:10.622101+0000","last_became_active":"2024-04-17T10:24:54.599658+0000","last_became_peered":"2024-04-17T10:24:54.599658+0000","last_unstale":"2024-04-17T10:25:10.622101+0000","last_undegraded":"2024-04-17T10:25:10.622101+0000","last_fullsized":"2024-04-17T10:25:10.622101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637141+0000","last_change":"2024-04-17T10:24:59.657550+0000","last_active":"2024-04-17T10:25:01.637141+0000","last_peered":"2024-04-17T10:25:01.637141+0000","last_clean":"2024-04-17T10:25:01.637141+0000","last_became_active":"2024-04-17T10:24:59.657179+0000","last_became_peered":"2024-04-17T10:24:59.657179+0000","last_unstale":"2024-04-17T10:25:01.637141+0000","last_undegraded":"2024-04-17T10:25:01.637141+0000","last_fullsized":"2024-04-17T10:25:01.637141+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637399+0000","last_change":"2024-04-17T10:24:55.609933+0000","last_active":"2024-04-17T10:25:01.637399+0000","last_peered":"2024-04-17T10:25:01.637399+0000","last_clean":"2024-04-17T10:25:01.637399+0000","last_became_active":"2024-04-17T10:24:55.602879+0000","last_became_peered":"2024-04-17T10:24:55.602879+0000","last_unstale":"2024-04-17T10:25:01.637399+0000","last_undegraded":"2024-04-17T10:25:01.637399+0000","last_fullsized":"2024-04-17T10:25:01.637399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:51.084300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636629+0000","last_change":"2024-04-17T10:24:54.607532+0000","last_active":"2024-04-17T10:25:01.636629+0000","last_peered":"2024-04-17T10:25:01.636629+0000","last_clean":"2024-04-17T10:25:01.636629+0000","last_became_active":"2024-04-17T10:24:54.606527+0000","last_became_peered":"2024-04-17T10:24:54.606527+0000","last_unstale":"2024-04-17T10:25:01.636629+0000","last_undegraded":"2024-04-17T10:25:01.636629+0000","last_fullsized":"2024-04-17T10:25:01.636629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637464+0000","last_change":"2024-04-17T10:24:59.644478+0000","last_active":"2024-04-17T10:25:01.637464+0000","last_peered":"2024-04-17T10:25:01.637464+0000","last_clean":"2024-04-17T10:25:01.637464+0000","last_became_active":"2024-04-17T10:24:59.644083+0000","last_became_peered":"2024-04-17T10:24:59.644083+0000","last_unstale":"2024-04-17T10:25:01.637464+0000","last_undegraded":"2024-04-17T10:25:01.637464+0000","last_fullsized":"2024-04-17T10:25:01.637464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156377+0000","last_change":"2024-04-17T10:24:55.627359+0000","last_active":"2024-04-17T10:25:03.156377+0000","last_peered":"2024-04-17T10:25:03.156377+0000","last_clean":"2024-04-17T10:25:03.156377+0000","last_became_active":"2024-04-17T10:24:55.627094+0000","last_became_peered":"2024-04-17T10:24:55.627094+0000","last_unstale":"2024-04-17T10:25:03.156377+0000","last_undegraded":"2024-04-17T10:25:03.156377+0000","last_fullsized":"2024-04-17T10:25:03.156377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267575+0000","last_change":"2024-04-17T10:24:54.613073+0000","last_active":"2024-04-17T10:25:03.267575+0000","last_peered":"2024-04-17T10:25:03.267575+0000","last_clean":"2024-04-17T10:25:03.267575+0000","last_became_active":"2024-04-17T10:24:54.612771+0000","last_became_peered":"2024-04-17T10:24:54.612771+0000","last_unstale":"2024-04-17T10:25:03.267575+0000","last_undegraded":"2024-04-17T10:25:03.267575+0000","last_fullsized":"2024-04-17T10:25:03.267575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637853+0000","last_change":"2024-04-17T10:24:59.653549+0000","last_active":"2024-04-17T10:25:01.637853+0000","last_peered":"2024-04-17T10:25:01.637853+0000","last_clean":"2024-04-17T10:25:01.637853+0000","last_became_active":"2024-04-17T10:24:59.652615+0000","last_became_peered":"2024-04-17T10:24:59.652615+0000","last_unstale":"2024-04-17T10:25:01.637853+0000","last_undegraded":"2024-04-17T10:25:01.637853+0000","last_fullsized":"2024-04-17T10:25:01.637853+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:43.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268787+0000","last_change":"2024-04-17T10:24:55.612177+0000","last_active":"2024-04-17T10:25:03.268787+0000","last_peered":"2024-04-17T10:25:03.268787+0000","last_clean":"2024-04-17T10:25:03.268787+0000","last_became_active":"2024-04-17T10:24:55.611847+0000","last_became_peered":"2024-04-17T10:24:55.611847+0000","last_unstale":"2024-04-17T10:25:03.268787+0000","last_undegraded":"2024-04-17T10:25:03.268787+0000","last_fullsized":"2024-04-17T10:25:03.268787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:00.256447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537797+0000","last_change":"2024-04-17T10:24:54.614002+0000","last_active":"2024-04-17T10:25:02.537797+0000","last_peered":"2024-04-17T10:25:02.537797+0000","last_clean":"2024-04-17T10:25:02.537797+0000","last_became_active":"2024-04-17T10:24:54.613675+0000","last_became_peered":"2024-04-17T10:24:54.613675+0000","last_unstale":"2024-04-17T10:25:02.537797+0000","last_undegraded":"2024-04-17T10:25:02.537797+0000","last_fullsized":"2024-04-17T10:25:02.537797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637973+0000","last_change":"2024-04-17T10:25:00.244266+0000","last_active":"2024-04-17T10:25:01.637973+0000","last_peered":"2024-04-17T10:25:01.637973+0000","last_clean":"2024-04-17T10:25:01.637973+0000","last_became_active":"2024-04-17T10:25:00.243271+0000","last_became_peered":"2024-04-17T10:25:00.243271+0000","last_unstale":"2024-04-17T10:25:01.637973+0000","last_undegraded":"2024-04-17T10:25:01.637973+0000","last_fullsized":"2024-04-17T10:25:01.637973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:37:59.616402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156261+0000","last_change":"2024-04-17T10:25:00.244894+0000","last_active":"2024-04-17T10:25:03.156261+0000","last_peered":"2024-04-17T10:25:03.156261+0000","last_clean":"2024-04-17T10:25:03.156261+0000","last_became_active":"2024-04-17T10:25:00.244366+0000","last_became_peered":"2024-04-17T10:25:00.244366+0000","last_unstale":"2024-04-17T10:25:03.156261+0000","last_undegraded":"2024-04-17T10:25:03.156261+0000","last_fullsized":"2024-04-17T10:25:03.156261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636436+0000","last_change":"2024-04-17T10:24:55.610258+0000","last_active":"2024-04-17T10:25:01.636436+0000","last_peered":"2024-04-17T10:25:01.636436+0000","last_clean":"2024-04-17T10:25:01.636436+0000","last_became_active":"2024-04-17T10:24:55.609904+0000","last_became_peered":"2024-04-17T10:24:55.609904+0000","last_unstale":"2024-04-17T10:25:01.636436+0000","last_undegraded":"2024-04-17T10:25:01.636436+0000","last_fullsized":"2024-04-17T10:25:01.636436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267470+0000","last_change":"2024-04-17T10:24:54.605205+0000","last_active":"2024-04-17T10:25:03.267470+0000","last_peered":"2024-04-17T10:25:03.267470+0000","last_clean":"2024-04-17T10:25:03.267470+0000","last_became_active":"2024-04-17T10:24:54.601015+0000","last_became_peered":"2024-04-17T10:24:54.601015+0000","last_unstale":"2024-04-17T10:25:03.267470+0000","last_undegraded":"2024-04-17T10:25:03.267470+0000","last_fullsized":"2024-04-17T10:25:03.267470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537785+0000","last_change":"2024-04-17T10:24:59.648566+0000","last_active":"2024-04-17T10:25:02.537785+0000","last_peered":"2024-04-17T10:25:02.537785+0000","last_clean":"2024-04-17T10:25:02.537785+0000","last_became_active":"2024-04-17T10:24:59.648170+0000","last_became_peered":"2024-04-17T10:24:59.648170+0000","last_unstale":"2024-04-17T10:25:02.537785+0000","last_undegraded":"2024-04-17T10:25:02.537785+0000","last_fullsized":"2024-04-17T10:25:02.537785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637759+0000","last_change":"2024-04-17T10:24:55.611737+0000","last_active":"2024-04-17T10:25:01.637759+0000","last_peered":"2024-04-17T10:25:01.637759+0000","last_clean":"2024-04-17T10:25:01.637759+0000","last_became_active":"2024-04-17T10:24:55.611353+0000","last_became_peered":"2024-04-17T10:24:55.611353+0000","last_unstale":"2024-04-17T10:25:01.637759+0000","last_undegraded":"2024-04-17T10:25:01.637759+0000","last_fullsized":"2024-04-17T10:25:01.637759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638133+0000","last_change":"2024-04-17T10:24:54.601357+0000","last_active":"2024-04-17T10:25:01.638133+0000","last_peered":"2024-04-17T10:25:01.638133+0000","last_clean":"2024-04-17T10:25:01.638133+0000","last_became_active":"2024-04-17T10:24:54.601111+0000","last_became_peered":"2024-04-17T10:24:54.601111+0000","last_unstale":"2024-04-17T10:25:01.638133+0000","last_undegraded":"2024-04-17T10:25:01.638133+0000","last_fullsized":"2024-04-17T10:25:01.638133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537510+0000","last_change":"2024-04-17T10:24:54.601574+0000","last_active":"2024-04-17T10:25:02.537510+0000","last_peered":"2024-04-17T10:25:02.537510+0000","last_clean":"2024-04-17T10:25:02.537510+0000","last_became_active":"2024-04-17T10:24:54.601261+0000","last_became_peered":"2024-04-17T10:24:54.601261+0000","last_unstale":"2024-04-17T10:25:02.537510+0000","last_undegraded":"2024-04-17T10:25:02.537510+0000","last_fullsized":"2024-04-17T10:25:02.537510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156049+0000","last_change":"2024-04-17T10:24:55.604048+0000","last_active":"2024-04-17T10:25:03.156049+0000","last_peered":"2024-04-17T10:25:03.156049+0000","last_clean":"2024-04-17T10:25:03.156049+0000","last_became_active":"2024-04-17T10:24:55.592091+0000","last_became_peered":"2024-04-17T10:24:55.592091+0000","last_unstale":"2024-04-17T10:25:03.156049+0000","last_undegraded":"2024-04-17T10:25:03.156049+0000","last_fullsized":"2024-04-17T10:25:03.156049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625146+0000","last_change":"2024-04-17T10:24:59.656792+0000","last_active":"2024-04-17T10:25:02.625146+0000","last_peered":"2024-04-17T10:25:02.625146+0000","last_clean":"2024-04-17T10:25:02.625146+0000","last_became_active":"2024-04-17T10:24:59.656244+0000","last_became_peered":"2024-04-17T10:24:59.656244+0000","last_unstale":"2024-04-17T10:25:02.625146+0000","last_undegraded":"2024-04-17T10:25:02.625146+0000","last_fullsized":"2024-04-17T10:25:02.625146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:57.427708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638566+0000","last_change":"2024-04-17T10:24:54.609412+0000","last_active":"2024-04-17T10:25:01.638566+0000","last_peered":"2024-04-17T10:25:01.638566+0000","last_clean":"2024-04-17T10:25:01.638566+0000","last_became_active":"2024-04-17T10:24:54.609119+0000","last_became_peered":"2024-04-17T10:24:54.609119+0000","last_unstale":"2024-04-17T10:25:01.638566+0000","last_undegraded":"2024-04-17T10:25:01.638566+0000","last_fullsized":"2024-04-17T10:25:01.638566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636952+0000","last_change":"2024-04-17T10:24:55.614982+0000","last_active":"2024-04-17T10:25:01.636952+0000","last_peered":"2024-04-17T10:25:01.636952+0000","last_clean":"2024-04-17T10:25:01.636952+0000","last_became_active":"2024-04-17T10:24:55.614619+0000","last_became_peered":"2024-04-17T10:24:55.614619+0000","last_unstale":"2024-04-17T10:25:01.636952+0000","last_undegraded":"2024-04-17T10:25:01.636952+0000","last_fullsized":"2024-04-17T10:25:01.636952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:32.258402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638839+0000","last_change":"2024-04-17T10:24:59.647972+0000","last_active":"2024-04-17T10:25:01.638839+0000","last_peered":"2024-04-17T10:25:01.638839+0000","last_clean":"2024-04-17T10:25:01.638839+0000","last_became_active":"2024-04-17T10:24:59.646737+0000","last_became_peered":"2024-04-17T10:24:59.646737+0000","last_unstale":"2024-04-17T10:25:01.638839+0000","last_undegraded":"2024-04-17T10:25:01.638839+0000","last_fullsized":"2024-04-17T10:25:01.638839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:59.695291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537104+0000","last_change":"2024-04-17T10:24:55.608996+0000","last_active":"2024-04-17T10:25:02.537104+0000","last_peered":"2024-04-17T10:25:02.537104+0000","last_clean":"2024-04-17T10:25:02.537104+0000","last_became_active":"2024-04-17T10:24:55.608621+0000","last_became_peered":"2024-04-17T10:24:55.608621+0000","last_unstale":"2024-04-17T10:25:02.537104+0000","last_undegraded":"2024-04-17T10:25:02.537104+0000","last_fullsized":"2024-04-17T10:25:02.537104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625124+0000","last_change":"2024-04-17T10:24:55.616866+0000","last_active":"2024-04-17T10:25:02.625124+0000","last_peered":"2024-04-17T10:25:02.625124+0000","last_clean":"2024-04-17T10:25:02.625124+0000","last_became_active":"2024-04-17T10:24:55.616582+0000","last_became_peered":"2024-04-17T10:24:55.616582+0000","last_unstale":"2024-04-17T10:25:02.625124+0000","last_undegraded":"2024-04-17T10:25:02.625124+0000","last_fullsized":"2024-04-17T10:25:02.625124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268379+0000","last_change":"2024-04-17T10:24:55.613768+0000","last_active":"2024-04-17T10:25:03.268379+0000","last_peered":"2024-04-17T10:25:03.268379+0000","last_clean":"2024-04-17T10:25:03.268379+0000","last_became_active":"2024-04-17T10:24:55.612427+0000","last_became_peered":"2024-04-17T10:24:55.612427+0000","last_unstale":"2024-04-17T10:25:03.268379+0000","last_undegraded":"2024-04-17T10:25:03.268379+0000","last_fullsized":"2024-04-17T10:25:03.268379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155940+0000","last_change":"2024-04-17T10:24:55.613422+0000","last_active":"2024-04-17T10:25:03.155940+0000","last_peered":"2024-04-17T10:25:03.155940+0000","last_clean":"2024-04-17T10:25:03.155940+0000","last_became_active":"2024-04-17T10:24:55.612627+0000","last_became_peered":"2024-04-17T10:24:55.612627+0000","last_unstale":"2024-04-17T10:25:03.155940+0000","last_undegraded":"2024-04-17T10:25:03.155940+0000","last_fullsized":"2024-04-17T10:25:03.155940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639265+0000","last_change":"2024-04-17T10:24:55.617532+0000","last_active":"2024-04-17T10:25:01.639265+0000","last_peered":"2024-04-17T10:25:01.639265+0000","last_clean":"2024-04-17T10:25:01.639265+0000","last_became_active":"2024-04-17T10:24:55.617215+0000","last_became_peered":"2024-04-17T10:24:55.617215+0000","last_unstale":"2024-04-17T10:25:01.639265+0000","last_undegraded":"2024-04-17T10:25:01.639265+0000","last_fullsized":"2024-04-17T10:25:01.639265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636559+0000","last_change":"2024-04-17T10:24:55.631795+0000","last_active":"2024-04-17T10:25:01.636559+0000","last_peered":"2024-04-17T10:25:01.636559+0000","last_clean":"2024-04-17T10:25:01.636559+0000","last_became_active":"2024-04-17T10:24:55.629826+0000","last_became_peered":"2024-04-17T10:24:55.629826+0000","last_unstale":"2024-04-17T10:25:01.636559+0000","last_undegraded":"2024-04-17T10:25:01.636559+0000","last_fullsized":"2024-04-17T10:25:01.636559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537858+0000","last_change":"2024-04-17T10:24:55.606321+0000","last_active":"2024-04-17T10:25:02.537858+0000","last_peered":"2024-04-17T10:25:02.537858+0000","last_clean":"2024-04-17T10:25:02.537858+0000","last_became_active":"2024-04-17T10:24:55.605895+0000","last_became_peered":"2024-04-17T10:24:55.605895+0000","last_unstale":"2024-04-17T10:25:02.537858+0000","last_undegraded":"2024-04-17T10:25:02.537858+0000","last_fullsized":"2024-04-17T10:25:02.537858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638540+0000","last_change":"2024-04-17T10:24:55.607147+0000","last_active":"2024-04-17T10:25:01.638540+0000","last_peered":"2024-04-17T10:25:01.638540+0000","last_clean":"2024-04-17T10:25:01.638540+0000","last_became_active":"2024-04-17T10:24:55.606931+0000","last_became_peered":"2024-04-17T10:24:55.606931+0000","last_unstale":"2024-04-17T10:25:01.638540+0000","last_undegraded":"2024-04-17T10:25:01.638540+0000","last_fullsized":"2024-04-17T10:25:01.638540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268051+0000","last_change":"2024-04-17T10:24:55.609356+0000","last_active":"2024-04-17T10:25:03.268051+0000","last_peered":"2024-04-17T10:25:03.268051+0000","last_clean":"2024-04-17T10:25:03.268051+0000","last_became_active":"2024-04-17T10:24:55.607716+0000","last_became_peered":"2024-04-17T10:24:55.607716+0000","last_unstale":"2024-04-17T10:25:03.268051+0000","last_undegraded":"2024-04-17T10:25:03.268051+0000","last_fullsized":"2024-04-17T10:25:03.268051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638038+0000","last_change":"2024-04-17T10:24:55.605676+0000","last_active":"2024-04-17T10:25:01.638038+0000","last_peered":"2024-04-17T10:25:01.638038+0000","last_clean":"2024-04-17T10:25:01.638038+0000","last_became_active":"2024-04-17T10:24:55.605431+0000","last_became_peered":"2024-04-17T10:24:55.605431+0000","last_unstale":"2024-04-17T10:25:01.638038+0000","last_undegraded":"2024-04-17T10:25:01.638038+0000","last_fullsized":"2024-04-17T10:25:01.638038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268286+0000","last_change":"2024-04-17T10:24:55.615398+0000","last_active":"2024-04-17T10:25:03.268286+0000","last_peered":"2024-04-17T10:25:03.268286+0000","last_clean":"2024-04-17T10:25:03.268286+0000","last_became_active":"2024-04-17T10:24:55.615034+0000","last_became_peered":"2024-04-17T10:24:55.615034+0000","last_unstale":"2024-04-17T10:25:03.268286+0000","last_undegraded":"2024-04-17T10:25:03.268286+0000","last_fullsized":"2024-04-17T10:25:03.268286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155772+0000","last_change":"2024-04-17T10:24:55.613776+0000","last_active":"2024-04-17T10:25:03.155772+0000","last_peered":"2024-04-17T10:25:03.155772+0000","last_clean":"2024-04-17T10:25:03.155772+0000","last_became_active":"2024-04-17T10:24:55.613106+0000","last_became_peered":"2024-04-17T10:24:55.613106+0000","last_unstale":"2024-04-17T10:25:03.155772+0000","last_undegraded":"2024-04-17T10:25:03.155772+0000","last_fullsized":"2024-04-17T10:25:03.155772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537535+0000","last_change":"2024-04-17T10:24:55.619933+0000","last_active":"2024-04-17T10:25:02.537535+0000","last_peered":"2024-04-17T10:25:02.537535+0000","last_clean":"2024-04-17T10:25:02.537535+0000","last_became_active":"2024-04-17T10:24:55.619666+0000","last_became_peered":"2024-04-17T10:24:55.619666+0000","last_unstale":"2024-04-17T10:25:02.537535+0000","last_undegraded":"2024-04-17T10:25:02.537535+0000","last_fullsized":"2024-04-17T10:25:02.537535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423902+0000","last_change":"2024-04-17T10:24:55.608090+0000","last_active":"2024-04-17T10:25:02.423902+0000","last_peered":"2024-04-17T10:25:02.423902+0000","last_clean":"2024-04-17T10:25:02.423902+0000","last_became_active":"2024-04-17T10:24:55.607715+0000","last_became_peered":"2024-04-17T10:24:55.607715+0000","last_unstale":"2024-04-17T10:25:02.423902+0000","last_undegraded":"2024-04-17T10:25:02.423902+0000","last_fullsized":"2024-04-17T10:25:02.423902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:07.835599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625372+0000","last_change":"2024-04-17T10:24:55.616021+0000","last_active":"2024-04-17T10:25:02.625372+0000","last_peered":"2024-04-17T10:25:02.625372+0000","last_clean":"2024-04-17T10:25:02.625372+0000","last_became_active":"2024-04-17T10:24:55.615736+0000","last_became_peered":"2024-04-17T10:24:55.615736+0000","last_unstale":"2024-04-17T10:25:02.625372+0000","last_undegraded":"2024-04-17T10:25:02.625372+0000","last_fullsized":"2024-04-17T10:25:02.625372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268965+0000","last_change":"2024-04-17T10:24:55.614983+0000","last_active":"2024-04-17T10:25:03.268965+0000","last_peered":"2024-04-17T10:25:03.268965+0000","last_clean":"2024-04-17T10:25:03.268965+0000","last_became_active":"2024-04-17T10:24:55.614651+0000","last_became_peered":"2024-04-17T10:24:55.614651+0000","last_unstale":"2024-04-17T10:25:03.268965+0000","last_undegraded":"2024-04-17T10:25:03.268965+0000","last_fullsized":"2024-04-17T10:25:03.268965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267935+0000","last_change":"2024-04-17T10:24:55.626976+0000","last_active":"2024-04-17T10:25:03.267935+0000","last_peered":"2024-04-17T10:25:03.267935+0000","last_clean":"2024-04-17T10:25:03.267935+0000","last_became_active":"2024-04-17T10:24:55.626743+0000","last_became_peered":"2024-04-17T10:24:55.626743+0000","last_unstale":"2024-04-17T10:25:03.267935+0000","last_undegraded":"2024-04-17T10:25:03.267935+0000","last_fullsized":"2024-04-17T10:25:03.267935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636737+0000","last_change":"2024-04-17T10:24:55.618634+0000","last_active":"2024-04-17T10:25:01.636737+0000","last_peered":"2024-04-17T10:25:01.636737+0000","last_clean":"2024-04-17T10:25:01.636737+0000","last_became_active":"2024-04-17T10:24:55.618343+0000","last_became_peered":"2024-04-17T10:24:55.618343+0000","last_unstale":"2024-04-17T10:25:01.636737+0000","last_undegraded":"2024-04-17T10:25:01.636737+0000","last_fullsized":"2024-04-17T10:25:01.636737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155747+0000","last_change":"2024-04-17T10:24:55.616149+0000","last_active":"2024-04-17T10:25:03.155747+0000","last_peered":"2024-04-17T10:25:03.155747+0000","last_clean":"2024-04-17T10:25:03.155747+0000","last_became_active":"2024-04-17T10:24:55.615561+0000","last_became_peered":"2024-04-17T10:24:55.615561+0000","last_unstale":"2024-04-17T10:25:03.155747+0000","last_undegraded":"2024-04-17T10:25:03.155747+0000","last_fullsized":"2024-04-17T10:25:03.155747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:37.060791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638375+0000","last_change":"2024-04-17T10:24:55.612680+0000","last_active":"2024-04-17T10:25:01.638375+0000","last_peered":"2024-04-17T10:25:01.638375+0000","last_clean":"2024-04-17T10:25:01.638375+0000","last_became_active":"2024-04-17T10:24:55.612280+0000","last_became_peered":"2024-04-17T10:24:55.612280+0000","last_unstale":"2024-04-17T10:25:01.638375+0000","last_undegraded":"2024-04-17T10:25:01.638375+0000","last_fullsized":"2024-04-17T10:25:01.638375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.610235+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.609934+0000","last_became_peered":"2024-04-17T10:24:55.609934+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537674+0000","last_change":"2024-04-17T10:24:55.613780+0000","last_active":"2024-04-17T10:25:02.537674+0000","last_peered":"2024-04-17T10:25:02.537674+0000","last_clean":"2024-04-17T10:25:02.537674+0000","last_became_active":"2024-04-17T10:24:55.613360+0000","last_became_peered":"2024-04-17T10:24:55.613360+0000","last_unstale":"2024-04-17T10:25:02.537674+0000","last_undegraded":"2024-04-17T10:25:02.537674+0000","last_fullsized":"2024-04-17T10:25:02.537674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537823+0000","last_change":"2024-04-17T10:24:55.617826+0000","last_active":"2024-04-17T10:25:02.537823+0000","last_peered":"2024-04-17T10:25:02.537823+0000","last_clean":"2024-04-17T10:25:02.537823+0000","last_became_active":"2024-04-17T10:24:55.617446+0000","last_became_peered":"2024-04-17T10:24:55.617446+0000","last_unstale":"2024-04-17T10:25:02.537823+0000","last_undegraded":"2024-04-17T10:25:02.537823+0000","last_fullsized":"2024-04-17T10:25:02.537823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424189+0000","last_change":"2024-04-17T10:24:55.626063+0000","last_active":"2024-04-17T10:25:02.424189+0000","last_peered":"2024-04-17T10:25:02.424189+0000","last_clean":"2024-04-17T10:25:02.424189+0000","last_became_active":"2024-04-17T10:24:55.625720+0000","last_became_peered":"2024-04-17T10:24:55.625720+0000","last_unstale":"2024-04-17T10:25:02.424189+0000","last_undegraded":"2024-04-17T10:25:02.424189+0000","last_fullsized":"2024-04-17T10:25:02.424189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:39.746376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637649+0000","last_change":"2024-04-17T10:24:55.622013+0000","last_active":"2024-04-17T10:25:01.637649+0000","last_peered":"2024-04-17T10:25:01.637649+0000","last_clean":"2024-04-17T10:25:01.637649+0000","last_became_active":"2024-04-17T10:24:55.618726+0000","last_became_peered":"2024-04-17T10:24:55.618726+0000","last_unstale":"2024-04-17T10:25:01.637649+0000","last_undegraded":"2024-04-17T10:25:01.637649+0000","last_fullsized":"2024-04-17T10:25:01.637649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423351+0000","last_change":"2024-04-17T10:24:55.614704+0000","last_active":"2024-04-17T10:25:02.423351+0000","last_peered":"2024-04-17T10:25:02.423351+0000","last_clean":"2024-04-17T10:25:02.423351+0000","last_became_active":"2024-04-17T10:24:55.614425+0000","last_became_peered":"2024-04-17T10:24:55.614425+0000","last_unstale":"2024-04-17T10:25:02.423351+0000","last_undegraded":"2024-04-17T10:25:02.423351+0000","last_fullsized":"2024-04-17T10:25:02.423351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268264+0000","last_change":"2024-04-17T10:24:55.630019+0000","last_active":"2024-04-17T10:25:03.268264+0000","last_peered":"2024-04-17T10:25:03.268264+0000","last_clean":"2024-04-17T10:25:03.268264+0000","last_became_active":"2024-04-17T10:24:55.629824+0000","last_became_peered":"2024-04-17T10:24:55.629824+0000","last_unstale":"2024-04-17T10:25:03.268264+0000","last_undegraded":"2024-04-17T10:25:03.268264+0000","last_fullsized":"2024-04-17T10:25:03.268264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423511+0000","last_change":"2024-04-17T10:24:55.612346+0000","last_active":"2024-04-17T10:25:02.423511+0000","last_peered":"2024-04-17T10:25:02.423511+0000","last_clean":"2024-04-17T10:25:02.423511+0000","last_became_active":"2024-04-17T10:24:55.610079+0000","last_became_peered":"2024-04-17T10:24:55.610079+0000","last_unstale":"2024-04-17T10:25:02.423511+0000","last_undegraded":"2024-04-17T10:25:02.423511+0000","last_fullsized":"2024-04-17T10:25:02.423511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268502+0000","last_change":"2024-04-17T10:24:55.612221+0000","last_active":"2024-04-17T10:25:03.268502+0000","last_peered":"2024-04-17T10:25:03.268502+0000","last_clean":"2024-04-17T10:25:03.268502+0000","last_became_active":"2024-04-17T10:24:55.611946+0000","last_became_peered":"2024-04-17T10:24:55.611946+0000","last_unstale":"2024-04-17T10:25:03.268502+0000","last_undegraded":"2024-04-17T10:25:03.268502+0000","last_fullsized":"2024-04-17T10:25:03.268502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:56.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156183+0000","last_change":"2024-04-17T10:24:55.617011+0000","last_active":"2024-04-17T10:25:03.156183+0000","last_peered":"2024-04-17T10:25:03.156183+0000","last_clean":"2024-04-17T10:25:03.156183+0000","last_became_active":"2024-04-17T10:24:55.616709+0000","last_became_peered":"2024-04-17T10:24:55.616709+0000","last_unstale":"2024-04-17T10:25:03.156183+0000","last_undegraded":"2024-04-17T10:25:03.156183+0000","last_fullsized":"2024-04-17T10:25:03.156183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:10.092171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636934+0000","last_change":"2024-04-17T10:24:55.627305+0000","last_active":"2024-04-17T10:25:01.636934+0000","last_peered":"2024-04-17T10:25:01.636934+0000","last_clean":"2024-04-17T10:25:01.636934+0000","last_became_active":"2024-04-17T10:24:55.626978+0000","last_became_peered":"2024-04-17T10:24:55.626978+0000","last_unstale":"2024-04-17T10:25:01.636934+0000","last_undegraded":"2024-04-17T10:25:01.636934+0000","last_fullsized":"2024-04-17T10:25:01.636934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424104+0000","last_change":"2024-04-17T10:24:55.613165+0000","last_active":"2024-04-17T10:25:02.424104+0000","last_peered":"2024-04-17T10:25:02.424104+0000","last_clean":"2024-04-17T10:25:02.424104+0000","last_became_active":"2024-04-17T10:24:55.611929+0000","last_became_peered":"2024-04-17T10:24:55.611929+0000","last_unstale":"2024-04-17T10:25:02.424104+0000","last_undegraded":"2024-04-17T10:25:02.424104+0000","last_fullsized":"2024-04-17T10:25:02.424104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:48.582360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:25:21.181 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:25:21.182 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-17T10:25:21.182 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-17T10:25:21.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-17T10:25:21.183 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-17T10:25:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.363+0000 7f9348400640 1 Processor -- start 2024-04-17T10:25:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.363+0000 7f9348400640 1 -- start start 2024-04-17T10:25:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9348400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 0x7f93401620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9348400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934015eb50 con 0x7f93400cf510 2024-04-17T10:25:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9348400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f934015ee30 con 0x7f93400ce400 2024-04-17T10:25:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9348400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9340161650 con 0x7f93400ce020 2024-04-17T10:25:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9346400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f93400cf510 0x7f93400cf9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54302/0 (socket says 172.21.15.67:54302) 2024-04-17T10:25:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9346400640 1 -- 172.21.15.67:0/918308775 learned_addr learned my addr 172.21.15.67:0/918308775 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345a00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 0x7f93401620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165741900 0 0) 0x7f934015eb50 con 0x7f93400cf510 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9328003530 con 0x7f93400cf510 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570543567 0 0) 0x7f9328003530 con 0x7f93400cf510 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 msgr2=0x7f93401620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 0x7f93401620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 >> v1:172.21.15.86:6789/0 conn(0x7f93400ce400 legacy=0x7f934015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93401618a0 con 0x7f93400cf510 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f9338001a00 con 0x7f93400cf510 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 >> v1:172.21.15.67:6789/0 conn(0x7f93400cf510 legacy=0x7f93400cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93280039f0 0x7f9328003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 0x7f9328004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9328004d50 0x7f93280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9340161650 con 0x7f93280039f0 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932800b9d0 con 0x7f9328004d50 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932800bbd0 con 0x7f93400ce020 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345a00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93280039f0 0x7f9328003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345000640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 0x7f9328004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9346400640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9328004d50 0x7f93280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345a00640 1 -- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 msgr2=0x7f9328004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345a00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f93400ce020 0x7f9328004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345a00640 1 -- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9328004d50 msgr2=0x7f93280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345a00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9328004d50 0x7f93280051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.367+0000 7f9345a00640 1 -- 172.21.15.67:0/918308775 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9328003530 con 0x7f93280039f0 2024-04-17T10:25:21.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9345a00640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93280039f0 0x7f9328003dd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9334009ae0 tx=0x7f933401c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d513616085de34 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:21.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340b2070 con 0x7f93280039f0 2024-04-17T10:25:21.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93340a55a0 con 0x7f93280039f0 2024-04-17T10:25:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f933fe00640 1 -- 172.21.15.67:0/918308775 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93340096b0 con 0x7f93280039f0 2024-04-17T10:25:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9348400640 1 -- 172.21.15.67:0/918308775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93280039f0 msgr2=0x7f9328003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9348400640 1 --2- 172.21.15.67:0/918308775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f93280039f0 0x7f9328003dd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9334009ae0 tx=0x7f933401c9b0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9346400640 1 -- 172.21.15.67:0/918308775 reap_dead start 2024-04-17T10:25:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9348400640 1 -- 172.21.15.67:0/918308775 shutdown_connections 2024-04-17T10:25:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9348400640 1 -- 172.21.15.67:0/918308775 >> 172.21.15.67:0/918308775 conn(0x7f93400b33f0 msgr2=0x7f93400b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9348400640 1 -- 172.21.15.67:0/918308775 shutdown_connections 2024-04-17T10:25:21.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.371+0000 7f9348400640 1 -- 172.21.15.67:0/918308775 wait complete. 2024-04-17T10:25:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 Processor -- start 2024-04-17T10:25:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 -- start start 2024-04-17T10:25:21.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93280039f0 0x7f93400b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934015d650 0x7f9340159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340149e40 0x7f934014a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934006bf70 con 0x7f9340149e40 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93400ce860 con 0x7f93280039f0 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f934006c0f0 con 0x7f934015d650 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340149e40 0x7f934014a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340149e40 0x7f934014a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57900/0 (socket says 172.21.15.67:57900) 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 -- 172.21.15.67:0/2624502487 learned_addr learned my addr 172.21.15.67:0/2624502487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9345000640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934015d650 0x7f9340159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9345a00640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93280039f0 0x7f93400b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 -- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934015d650 msgr2=0x7f9340159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934015d650 0x7f9340159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 -- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93280039f0 msgr2=0x7f93400b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93280039f0 0x7f93400b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 -- 172.21.15.67:0/2624502487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93400b75e0 con 0x7f9340149e40 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9346400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340149e40 0x7f934014a2c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f93400b4410 tx=0x7f9338007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f933d600640 1 -- 172.21.15.67:0/2624502487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933800abc0 con 0x7f9340149e40 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f933d600640 1 -- 172.21.15.67:0/2624502487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9338018070 con 0x7f9340149e40 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93400b6d40 con 0x7f9340149e40 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f933d600640 1 -- 172.21.15.67:0/2624502487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93380133f0 con 0x7f9340149e40 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.375+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93400b7300 con 0x7f9340149e40 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.379+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9340070ab0 con 0x7f9340149e40 2024-04-17T10:25:21.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.379+0000 7f933d600640 1 -- 172.21.15.67:0/2624502487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9338013590 con 0x7f9340149e40 2024-04-17T10:25:21.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.379+0000 7f933d600640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9320066f60 0x7f9320069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.379+0000 7f933d600640 1 -- 172.21.15.67:0/2624502487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f93380859e0 con 0x7f9340149e40 2024-04-17T10:25:21.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.379+0000 7f9345a00640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9320066f60 0x7f9320069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.379+0000 7f9345a00640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9320066f60 0x7f9320069420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f93340af000 tx=0x7f93340046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:21.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.383+0000 7f933d600640 1 -- 172.21.15.67:0/2624502487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9338056610 con 0x7f9340149e40 2024-04-17T10:25:21.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.587+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f9340074e30 con 0x7f9340149e40 2024-04-17T10:25:21.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.591+0000 7f933d600640 1 -- 172.21.15.67:0/2624502487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+165733 (secure 0 0 0) 0x7f9338055f10 con 0x7f9340149e40 2024-04-17T10:25:21.592 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9320066f60 msgr2=0x7f9320069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9320066f60 0x7f9320069420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f93340af000 tx=0x7f93340046d0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340149e40 msgr2=0x7f934014a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340149e40 0x7f934014a2c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f93400b4410 tx=0x7f9338007b20 comp rx=0 tx=0).stop 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 shutdown_connections 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9320066f60 0x7f9320069420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9340149e40 0x7f934014a2c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f934015d650 0x7f9340159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 --2- 172.21.15.67:0/2624502487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f93280039f0 0x7f93400b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 >> 172.21.15.67:0/2624502487 conn(0x7f93400b33f0 msgr2=0x7f93400d0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 shutdown_connections 2024-04-17T10:25:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.595+0000 7f9348400640 1 -- 172.21.15.67:0/2624502487 wait complete. 2024-04-17T10:25:21.613 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":4206025909},{"type":"v1","addr":"172.21.15.86:6833","nonce":4206025909}]},"active_addr":"172.21.15.86:6833/4206025909","active_change":"2024-04-17T10:24:34.114970+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2712299134}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2445210757}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1007225792}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1183760120}]}]} 2024-04-17T10:25:21.616 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-17T10:25:21.616 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-17T10:25:21.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f421dc00640 1 Processor -- start 2024-04-17T10:25:21.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f421dc00640 1 -- start start 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f421dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42181446c0 0x7f4218144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f421dc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4218071840 0x7f42180b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f421dc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42181456d0 con 0x7f4218144aa0 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f421dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42180b51a0 con 0x7f42181446c0 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f421dc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42180b4870 con 0x7f4218071840 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4216a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42181446c0 0x7f4218144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4216a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42181446c0 0x7f4218144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57910/0 (socket says 172.21.15.67:57910) 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4216a00640 1 -- 172.21.15.67:0/4192285378 learned_addr learned my addr 172.21.15.67:0/4192285378 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:21.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4217400640 1 --2- 172.21.15.67:0/4192285378 >> v2:172.21.15.86:3300/0 conn(0x7f4218071840 0x7f42180b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4216a00640 1 -- 172.21.15.67:0/4192285378 >> v2:172.21.15.86:3300/0 conn(0x7f4218071840 msgr2=0x7f42180b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4216a00640 1 --2- 172.21.15.67:0/4192285378 >> v2:172.21.15.86:3300/0 conn(0x7f4218071840 0x7f42180b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4216a00640 1 -- 172.21.15.67:0/4192285378 >> v1:172.21.15.67:6789/0 conn(0x7f4218144aa0 legacy=0x7f42180d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.759+0000 7f4216a00640 1 -- 172.21.15.67:0/4192285378 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180b4b50 con 0x7f42181446c0 2024-04-17T10:25:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f4216a00640 1 --2- 172.21.15.67:0/4192285378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42181446c0 0x7f4218144ea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f42180b5320 tx=0x7f420801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6909f44674e263de server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f4215600640 1 -- 172.21.15.67:0/4192285378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080ab600 con 0x7f42181446c0 2024-04-17T10:25:21.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f4215600640 1 -- 172.21.15.67:0/4192285378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42080b0070 con 0x7f42181446c0 2024-04-17T10:25:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f4215600640 1 -- 172.21.15.67:0/4192285378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080ab9c0 con 0x7f42181446c0 2024-04-17T10:25:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 -- 172.21.15.67:0/4192285378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42181446c0 msgr2=0x7f4218144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 --2- 172.21.15.67:0/4192285378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42181446c0 0x7f4218144ea0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f42180b5320 tx=0x7f420801c4f0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 -- 172.21.15.67:0/4192285378 shutdown_connections 2024-04-17T10:25:21.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 --2- 172.21.15.67:0/4192285378 >> v2:172.21.15.86:3300/0 conn(0x7f4218071840 0x7f42180b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 --2- 172.21.15.67:0/4192285378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42181446c0 0x7f4218144ea0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 -- 172.21.15.67:0/4192285378 >> 172.21.15.67:0/4192285378 conn(0x7f42180b34d0 msgr2=0x7f42180b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 -- 172.21.15.67:0/4192285378 shutdown_connections 2024-04-17T10:25:21.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.763+0000 7f421dc00640 1 -- 172.21.15.67:0/4192285378 wait complete. 2024-04-17T10:25:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 Processor -- start 2024-04-17T10:25:21.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 -- start start 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4218071840 0x7f42181403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42181446c0 0x7f42180b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4218144aa0 0x7f42180b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180b6500 con 0x7f4218071840 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180b6380 con 0x7f4218144aa0 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180b6680 con 0x7f42181446c0 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4218071840 0x7f42181403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4218071840 0x7f42181403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57916/0 (socket says 172.21.15.67:57916) 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 -- 172.21.15.67:0/2768069971 learned_addr learned my addr 172.21.15.67:0/2768069971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:21.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4217400640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4218144aa0 0x7f42180b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216000640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42181446c0 0x7f42180b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 -- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42181446c0 msgr2=0x7f42180b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42181446c0 0x7f42180b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 -- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4218144aa0 msgr2=0x7f42180b2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4218144aa0 0x7f42180b2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 -- 172.21.15.67:0/2768069971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218140930 con 0x7f4218071840 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f4216a00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4218071840 0x7f42181403f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f420801a120 tx=0x7f420801ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f420ea00640 1 -- 172.21.15.67:0/2768069971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080be5f0 con 0x7f4218071840 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f420ea00640 1 -- 172.21.15.67:0/2768069971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42080be790 con 0x7f4218071840 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f420ea00640 1 -- 172.21.15.67:0/2768069971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080beb60 con 0x7f4218071840 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42181fa860 con 0x7f4218071840 2024-04-17T10:25:21.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.771+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42181fae20 con 0x7f4218071840 2024-04-17T10:25:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.775+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421806df00 con 0x7f4218071840 2024-04-17T10:25:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.775+0000 7f420ea00640 1 -- 172.21.15.67:0/2768069971 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f42080aa070 con 0x7f4218071840 2024-04-17T10:25:21.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.775+0000 7f420ea00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41f8066f90 0x7f41f8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.775+0000 7f420ea00640 1 -- 172.21.15.67:0/2768069971 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f420811f1c0 con 0x7f4218071840 2024-04-17T10:25:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.775+0000 7f4216000640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41f8066f90 0x7f41f8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:21.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.775+0000 7f4216000640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41f8066f90 0x7f41f8069450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f421815b770 tx=0x7f4204009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:21.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.775+0000 7f420ea00640 1 -- 172.21.15.67:0/2768069971 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4208016110 con 0x7f4218071840 2024-04-17T10:25:21.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.935+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4218074e30 con 0x7f4218071840 2024-04-17T10:25:21.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f420ea00640 1 -- 172.21.15.67:0/2768069971 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7f42080ef830 con 0x7f4218071840 2024-04-17T10:25:21.937 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:21.937 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41f8066f90 msgr2=0x7f41f8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41f8066f90 0x7f41f8069450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f421815b770 tx=0x7f4204009040 comp rx=0 tx=0).stop 2024-04-17T10:25:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4218071840 msgr2=0x7f42181403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4218071840 0x7f42181403f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f420801a120 tx=0x7f420801ad30 comp rx=0 tx=0).stop 2024-04-17T10:25:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 shutdown_connections 2024-04-17T10:25:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41f8066f90 0x7f41f8069450 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4218144aa0 0x7f42180b2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42181446c0 0x7f42180b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 --2- 172.21.15.67:0/2768069971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4218071840 0x7f42181403f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 >> 172.21.15.67:0/2768069971 conn(0x7f42180b34d0 msgr2=0x7f42180b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 shutdown_connections 2024-04-17T10:25:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:21.943+0000 7f421dc00640 1 -- 172.21.15.67:0/2768069971 wait complete. 2024-04-17T10:25:21.958 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-17T10:25:21.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3592000640 1 Processor -- start 2024-04-17T10:25:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3592000640 1 -- start start 2024-04-17T10:25:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3592000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3592000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358c06e2f0 con 0x7f358c0cc300 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3592000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358c06d6e0 con 0x7f358c145040 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3592000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c0a86c0 con 0x7f358c071310 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f358be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f358c0cc300 0x7f358c0cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54318/0 (socket says 172.21.15.67:54318) 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f358be00640 1 -- 172.21.15.67:0/773301768 learned_addr learned my addr 172.21.15.67:0/773301768 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3582a00640 1 --2- 172.21.15.67:0/773301768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f358aa00640 1 -- 172.21.15.67:0/773301768 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741025391 0 0) 0x7f358c06e2f0 con 0x7f358c0cc300 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f358aa00640 1 -- 172.21.15.67:0/773301768 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3570003530 con 0x7f358c0cc300 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f358aa00640 1 -- 172.21.15.67:0/773301768 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378960114 0 0) 0x7f358c06d6e0 con 0x7f358c145040 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f358aa00640 1 -- 172.21.15.67:0/773301768 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c06e2f0 con 0x7f358c145040 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3582a00640 1 -- 172.21.15.67:0/773301768 >> v1:172.21.15.86:6789/0 conn(0x7f358c145040 legacy=0x7f358c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3582a00640 1 -- 172.21.15.67:0/773301768 >> v1:172.21.15.67:6789/0 conn(0x7f358c0cc300 legacy=0x7f358c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.103+0000 7f3582a00640 1 -- 172.21.15.67:0/773301768 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c06d9c0 con 0x7f358c071310 2024-04-17T10:25:22.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3582a00640 1 --2- 172.21.15.67:0/773301768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c071770 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35740097f0 tx=0x7f357401ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1cb2d42fce7b5a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f358aa00640 1 -- 172.21.15.67:0/773301768 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35740b5070 con 0x7f358c071310 2024-04-17T10:25:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f358aa00640 1 -- 172.21.15.67:0/773301768 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35740a89d0 con 0x7f358c071310 2024-04-17T10:25:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f358aa00640 1 -- 172.21.15.67:0/773301768 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35740a8da0 con 0x7f358c071310 2024-04-17T10:25:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3592000640 1 -- 172.21.15.67:0/773301768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 msgr2=0x7f358c071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3592000640 1 --2- 172.21.15.67:0/773301768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c071770 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35740097f0 tx=0x7f357401ecd0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3592000640 1 -- 172.21.15.67:0/773301768 shutdown_connections 2024-04-17T10:25:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3592000640 1 --2- 172.21.15.67:0/773301768 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c071770 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3592000640 1 -- 172.21.15.67:0/773301768 >> 172.21.15.67:0/773301768 conn(0x7f358c06ac90 msgr2=0x7f358c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3592000640 1 -- 172.21.15.67:0/773301768 shutdown_connections 2024-04-17T10:25:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.107+0000 7f3592000640 1 -- 172.21.15.67:0/773301768 wait complete. 2024-04-17T10:25:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 Processor -- start 2024-04-17T10:25:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 -- start start 2024-04-17T10:25:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c1391c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f358c0cc300 0x7f358c13b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f358c145040 0x7f358c13bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c15bf10 con 0x7f358c145040 2024-04-17T10:25:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c06df80 con 0x7f358c0cc300 2024-04-17T10:25:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c06d370 con 0x7f358c071310 2024-04-17T10:25:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f358c145040 0x7f358c13bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f358c145040 0x7f358c13bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57926/0 (socket says 172.21.15.67:57926) 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 -- 172.21.15.67:0/2163752167 learned_addr learned my addr 172.21.15.67:0/2163752167 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3582a00640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f358c0cc300 0x7f358c13b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358b400640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c1391c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 -- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 msgr2=0x7f358c1391c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c1391c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 -- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f358c0cc300 msgr2=0x7f358c13b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f358c0cc300 0x7f358c13b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 -- 172.21.15.67:0/2163752167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0b2a10 con 0x7f358c145040 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f358be00640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f358c145040 0x7f358c13bd40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f358400cbd0 tx=0x7f358400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3583e00640 1 -- 172.21.15.67:0/2163752167 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584017070 con 0x7f358c145040 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3583e00640 1 -- 172.21.15.67:0/2163752167 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358400ace0 con 0x7f358c145040 2024-04-17T10:25:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3583e00640 1 -- 172.21.15.67:0/2163752167 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584012620 con 0x7f358c145040 2024-04-17T10:25:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358c13c3c0 con 0x7f358c145040 2024-04-17T10:25:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.111+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358c13c980 con 0x7f358c145040 2024-04-17T10:25:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.115+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f358c0b2be0 con 0x7f358c145040 2024-04-17T10:25:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.115+0000 7f3583e00640 1 -- 172.21.15.67:0/2163752167 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f358401a070 con 0x7f358c145040 2024-04-17T10:25:22.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.115+0000 7f3583e00640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3568066f60 0x7f3568069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.115+0000 7f3583e00640 1 -- 172.21.15.67:0/2163752167 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3584085380 con 0x7f358c145040 2024-04-17T10:25:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.115+0000 7f358b400640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3568066f60 0x7f3568069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.119+0000 7f358b400640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3568066f60 0x7f3568069420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f357c0011f0 tx=0x7f357c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.119+0000 7f3583e00640 1 -- 172.21.15.67:0/2163752167 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3584055960 con 0x7f358c145040 2024-04-17T10:25:22.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.283+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f358c13cbc0 con 0x7f358c145040 2024-04-17T10:25:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3583e00640 1 -- 172.21.15.67:0/2163752167 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7f3584059a00 con 0x7f358c145040 2024-04-17T10:25:22.282 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:22.282 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3568066f60 msgr2=0x7f3568069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3568066f60 0x7f3568069420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f357c0011f0 tx=0x7f357c009040 comp rx=0 tx=0).stop 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f358c145040 msgr2=0x7f358c13bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f358c145040 0x7f358c13bd40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f358400cbd0 tx=0x7f358400a430 comp rx=0 tx=0).stop 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 shutdown_connections 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3568066f60 0x7f3568069420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f358c145040 0x7f358c13bd40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f358c0cc300 0x7f358c13b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 --2- 172.21.15.67:0/2163752167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f358c071310 0x7f358c1391c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 >> 172.21.15.67:0/2163752167 conn(0x7f358c06ac90 msgr2=0x7f358c15fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 shutdown_connections 2024-04-17T10:25:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.287+0000 7f3592000640 1 -- 172.21.15.67:0/2163752167 wait complete. 2024-04-17T10:25:22.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-17T10:25:22.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-17T10:25:22.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-17T10:25:22.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-17T10:25:22.305 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-17T10:25:22.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-17T10:25:22.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-17T10:25:22.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-17T10:25:22.397 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:25:22.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:25:22.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.555+0000 7f7d68000640 1 Processor -- start 2024-04-17T10:25:22.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d68000640 1 -- start start 2024-04-17T10:25:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d68000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d6006ac90 0x7f7d600cff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d65600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7d60161ff0 0x7f7d60166280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54332/0 (socket says 172.21.15.67:54332) 2024-04-17T10:25:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d64c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d6006ac90 0x7f7d600cff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d64c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d6006ac90 0x7f7d600cff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57934/0 (socket says 172.21.15.67:57934) 2024-04-17T10:25:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d65600640 1 -- 172.21.15.67:0/1249545396 learned_addr learned my addr 172.21.15.67:0/1249545396 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d68000640 1 --2- 172.21.15.67:0/1249545396 >> v2:172.21.15.86:3300/0 conn(0x7f7d600ce2c0 0x7f7d600ce750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d600b3180 con 0x7f7d60161ff0 2024-04-17T10:25:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d60165df0 con 0x7f7d6006ac90 2024-04-17T10:25:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.559+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6006b270 con 0x7f7d600ce2c0 2024-04-17T10:25:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f7d64c00640 1 -- 172.21.15.67:0/1249545396 >> v2:172.21.15.86:3300/0 conn(0x7f7d600ce2c0 msgr2=0x7f7d600ce750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T10:25:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f7d64c00640 1 --2- 172.21.15.67:0/1249545396 >> v2:172.21.15.86:3300/0 conn(0x7f7d600ce2c0 0x7f7d600ce750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f7d64c00640 1 -- 172.21.15.67:0/1249545396 >> v1:172.21.15.67:6789/0 conn(0x7f7d60161ff0 legacy=0x7f7d60166280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f7d64c00640 1 -- 172.21.15.67:0/1249545396 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6006b550 con 0x7f7d6006ac90 2024-04-17T10:25:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f3e19200640 1 Processor -- start 2024-04-17T10:25:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f2f60200640 1 Processor -- start 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f70db200640 1 Processor -- start 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f70db200640 1 -- start start 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f70db200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d415aaa0 0x7f70d40cd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f70db200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d40ccb10 0x7f70d40cd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f70db200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d415ae80 con 0x7f70d40ce2d0 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f70db200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d4070180 con 0x7f70d415aaa0 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f70db200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d4070460 con 0x7f70d40ccb10 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f3e19200640 1 -- start start 2024-04-17T10:25:22.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f3e19200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140b4970 0x7f3e14163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.563+0000 7f2f60200640 1 -- start start 2024-04-17T10:25:22.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d415aaa0 0x7f70d40cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70d415aaa0 0x7f70d40cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57936/0 (socket says 172.21.15.67:57936) 2024-04-17T10:25:22.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3e00640 1 -- 172.21.15.67:0/1379583127 learned_addr learned my addr 172.21.15.67:0/1379583127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e0be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140b4970 0x7f3e14163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e0be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140b4970 0x7f3e14163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49292/0 (socket says 172.21.15.67:49292) 2024-04-17T10:25:22.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e19200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e14071720 con 0x7f3e140cef80 2024-04-17T10:25:22.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e0be00640 1 -- 172.21.15.67:0/106551080 learned_addr learned my addr 172.21.15.67:0/106551080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e19200640 1 -- 172.21.15.67:0/106551080 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e140719a0 con 0x7f3e14071310 2024-04-17T10:25:22.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e19200640 1 -- 172.21.15.67:0/106551080 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e14071c80 con 0x7f3e140b4970 2024-04-17T10:25:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3400640 1 --2- 172.21.15.67:0/1379583127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d40ccb10 0x7f70d40cd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f60200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f58159bb0 0x7f2f580b3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3e00640 1 -- 172.21.15.67:0/1379583127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d40ccb10 msgr2=0x7f70d40cd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3e00640 1 --2- 172.21.15.67:0/1379583127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d40ccb10 0x7f70d40cd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3e00640 1 -- 172.21.15.67:0/1379583127 >> v1:172.21.15.86:6789/0 conn(0x7f70d40ce2d0 legacy=0x7f70d406e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d2a00640 1 -- 172.21.15.67:0/1379583127 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253231061 0 0) 0x7f70d415ae80 con 0x7f70d40ce2d0 2024-04-17T10:25:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e11400640 1 -- 172.21.15.67:0/106551080 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27079496 0 0) 0x7f3e140719a0 con 0x7f3e14071310 2024-04-17T10:25:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e11400640 1 -- 172.21.15.67:0/106551080 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df8003530 con 0x7f3e14071310 2024-04-17T10:25:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e11400640 1 -- 172.21.15.67:0/106551080 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958426927 0 0) 0x7f3e04001120 con 0x7f3e140cef80 2024-04-17T10:25:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e11400640 1 -- 172.21.15.67:0/106551080 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df80048f0 con 0x7f3e140cef80 2024-04-17T10:25:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f60200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f58071310 0x7f2f580717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f7d64c00640 1 --2- 172.21.15.67:0/1249545396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6006ac90 0x7f7d600cff70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7d600707b0 tx=0x7f7d4c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e56a61b5c0de049 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f5d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f58159bb0 0x7f2f580b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f5d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f58159bb0 0x7f2f580b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57952/0 (socket says 172.21.15.67:57952) 2024-04-17T10:25:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f5e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f58071310 0x7f2f580717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f5e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f58071310 0x7f2f580717c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49306/0 (socket says 172.21.15.67:49306) 2024-04-17T10:25:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3e00640 1 -- 172.21.15.67:0/1379583127 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d40cec70 con 0x7f70d415aaa0 2024-04-17T10:25:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f70d3400640 1 --2- 172.21.15.67:0/1379583127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d40ccb10 0x7f70d40cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f5ce00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2f5815e1f0 0x7f2f580b4200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54342/0 (socket says 172.21.15.67:54342) 2024-04-17T10:25:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f5d800640 1 -- 172.21.15.67:0/306717862 learned_addr learned my addr 172.21.15.67:0/306717862 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f7d5fe00640 1 -- 172.21.15.67:0/1249545396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0b1070 con 0x7f7d6006ac90 2024-04-17T10:25:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580b4cc0 con 0x7f2f5815e1f0 2024-04-17T10:25:22.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f5815e680 con 0x7f2f58159bb0 2024-04-17T10:25:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e0be00640 1 -- 172.21.15.67:0/106551080 >> v1:172.21.15.86:6789/0 conn(0x7f3e14071310 legacy=0x7f3e140b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f3e0be00640 1 -- 172.21.15.67:0/106551080 >> v1:172.21.15.67:6789/0 conn(0x7f3e140cef80 legacy=0x7f3e140b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.567+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f58159f90 con 0x7f2f58071310 2024-04-17T10:25:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f7d5fe00640 1 -- 172.21.15.67:0/1249545396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c0ac620 con 0x7f7d6006ac90 2024-04-17T10:25:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f7d5fe00640 1 -- 172.21.15.67:0/1249545396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0ac9f0 con 0x7f7d6006ac90 2024-04-17T10:25:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f3e0be00640 1 -- 172.21.15.67:0/106551080 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14162580 con 0x7f3e140b4970 2024-04-17T10:25:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2fad400640 1 Processor -- start 2024-04-17T10:25:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2fad400640 1 -- start start 2024-04-17T10:25:22.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6006ac90 msgr2=0x7f7d600cff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f7d68000640 1 --2- 172.21.15.67:0/1249545396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6006ac90 0x7f7d600cff70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7d600707b0 tx=0x7f7d4c01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7fa9b9600640 1 Processor -- start 2024-04-17T10:25:22.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2f5e200640 1 -- 172.21.15.67:0/306717862 >> v2:172.21.15.67:3300/0 conn(0x7f2f58159bb0 msgr2=0x7f2f580b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2f5e200640 1 --2- 172.21.15.67:0/306717862 >> v2:172.21.15.67:3300/0 conn(0x7f2f58159bb0 0x7f2f580b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2f5e200640 1 -- 172.21.15.67:0/306717862 >> v1:172.21.15.67:6789/0 conn(0x7f2f5815e1f0 legacy=0x7f2f580b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7fa9b9600640 1 -- start start 2024-04-17T10:25:22.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2fad400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2fa806feb0 0x7f2fa806e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2fad400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fa80b4970 0x7f2fa806ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2fad400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa80acfa0 con 0x7f2fa806bca0 2024-04-17T10:25:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2fad400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa80ad280 con 0x7f2fa80b4970 2024-04-17T10:25:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2fad400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa80ad560 con 0x7f2fa806feb0 2024-04-17T10:25:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2f9fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2fa806feb0 0x7f2fa806e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2f9fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2fa806feb0 0x7f2fa806e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37076/0 (socket says 172.21.15.67:37076) 2024-04-17T10:25:22.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2f9fe00640 1 -- 172.21.15.67:0/2920126816 learned_addr learned my addr 172.21.15.67:0/2920126816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7fa9b9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa9b415dd30 0x7fa9b40d1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7fa9b9600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa9b40d3510 0x7fa9b40b9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b9600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d07f0 0x7fa9b40d0c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 shutdown_connections 2024-04-17T10:25:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f7d68000640 1 --2- 172.21.15.67:0/1249545396 >> v2:172.21.15.86:3300/0 conn(0x7f7d600ce2c0 0x7f7d600ce750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f7d68000640 1 --2- 172.21.15.67:0/1249545396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d6006ac90 0x7f7d600cff70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 >> 172.21.15.67:0/1249545396 conn(0x7f7d600b2340 msgr2=0x7f7d600b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b9600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b40b9a50 con 0x7fa9b415dd30 2024-04-17T10:25:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b9600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b40d1180 con 0x7fa9b40d3510 2024-04-17T10:25:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b9600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b40d3080 con 0x7fa9b40d07f0 2024-04-17T10:25:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.571+0000 7f2f5e200640 1 -- 172.21.15.67:0/306717862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580b49e0 con 0x7f2f58071310 2024-04-17T10:25:22.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2fa7200640 1 --2- 172.21.15.67:0/2920126816 >> v2:172.21.15.67:3300/0 conn(0x7f2fa80b4970 0x7f2fa806ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9aa000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa9b40d3510 0x7fa9b40b9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9aa000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa9b40d3510 0x7fa9b40b9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37086/0 (socket says 172.21.15.67:37086) 2024-04-17T10:25:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9aa000640 1 -- 172.21.15.67:0/2125181088 learned_addr learned my addr 172.21.15.67:0/2125181088 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b3400640 1 --2- 172.21.15.67:0/2125181088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d07f0 0x7fa9b40d0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b2a00640 1 --2- 172.21.15.67:0/2125181088 >> v2:172.21.15.67:3300/0 conn(0x7fa9b415dd30 0x7fa9b40d1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 shutdown_connections 2024-04-17T10:25:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f9fe00640 1 -- 172.21.15.67:0/2920126816 >> v2:172.21.15.67:3300/0 conn(0x7f2fa80b4970 msgr2=0x7f2fa806ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f9fe00640 1 --2- 172.21.15.67:0/2920126816 >> v2:172.21.15.67:3300/0 conn(0x7f2fa80b4970 0x7f2fa806ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f9fe00640 1 -- 172.21.15.67:0/2920126816 >> v1:172.21.15.86:6789/0 conn(0x7f2fa806bca0 legacy=0x7f2fa806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f5e200640 1 --2- 172.21.15.67:0/306717862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f58071310 0x7f2f580717c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009d90 tx=0x7f2f5001daa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=128c6910b65a6698 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f9fe00640 1 -- 172.21.15.67:0/2920126816 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80ad840 con 0x7f2fa806feb0 2024-04-17T10:25:22.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2fa7200640 1 --2- 172.21.15.67:0/2920126816 >> v2:172.21.15.67:3300/0 conn(0x7f2fa80b4970 0x7f2fa806ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:22.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b3400640 1 -- 172.21.15.67:0/2125181088 >> v2:172.21.15.86:3300/0 conn(0x7fa9b40d3510 msgr2=0x7fa9b40b9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b3400640 1 --2- 172.21.15.67:0/2125181088 >> v2:172.21.15.86:3300/0 conn(0x7fa9b40d3510 0x7fa9b40b9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b3400640 1 -- 172.21.15.67:0/2125181088 >> v2:172.21.15.67:3300/0 conn(0x7fa9b415dd30 msgr2=0x7fa9b40d1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b3400640 1 --2- 172.21.15.67:0/2125181088 >> v2:172.21.15.67:3300/0 conn(0x7fa9b415dd30 0x7fa9b40d1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7fa9b3400640 1 -- 172.21.15.67:0/2125181088 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b40d2470 con 0x7fa9b40d07f0 2024-04-17T10:25:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f7d68000640 1 -- 172.21.15.67:0/1249545396 wait complete. 2024-04-17T10:25:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f57e00640 1 -- 172.21.15.67:0/306717862 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500b3070 con 0x7f2f58071310 2024-04-17T10:25:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f57e00640 1 -- 172.21.15.67:0/306717862 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f500a6a30 con 0x7f2f58071310 2024-04-17T10:25:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e0be00640 1 --2- 172.21.15.67:0/106551080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140b4970 0x7f3e14163060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0098a0 tx=0x7f3dfc01c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dff3e8688c7eaeb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f57e00640 1 -- 172.21.15.67:0/306717862 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f500a6e00 con 0x7f2f58071310 2024-04-17T10:25:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f70d3e00640 1 --2- 172.21.15.67:0/1379583127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d415aaa0 0x7f70d40cd110 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f70c400bb10 tx=0x7f70c401eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ecffaa2b584557 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e11400640 1 -- 172.21.15.67:0/106551080 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0a9610 con 0x7f3e140b4970 2024-04-17T10:25:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e11400640 1 -- 172.21.15.67:0/106551080 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc0a97b0 con 0x7f3e140b4970 2024-04-17T10:25:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e11400640 1 -- 172.21.15.67:0/106551080 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0a9610 con 0x7f3e140b4970 2024-04-17T10:25:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f70d2a00640 1 -- 172.21.15.67:0/1379583127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c40b4070 con 0x7f70d415aaa0 2024-04-17T10:25:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f58071310 msgr2=0x7f2f580717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f2f60200640 1 --2- 172.21.15.67:0/306717862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f58071310 0x7f2f580717c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2f50009d90 tx=0x7f2f5001daa0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f7d68000640 1 Processor -- start 2024-04-17T10:25:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f70d2a00640 1 -- 172.21.15.67:0/1379583127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c40a7b60 con 0x7f70d415aaa0 2024-04-17T10:25:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f70d2a00640 1 -- 172.21.15.67:0/1379583127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c40af440 con 0x7f70d415aaa0 2024-04-17T10:25:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e19200640 1 -- 172.21.15.67:0/106551080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140b4970 msgr2=0x7f3e14163060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e19200640 1 --2- 172.21.15.67:0/106551080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140b4970 0x7f3e14163060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc0098a0 tx=0x7f3dfc01c550 comp rx=0 tx=0).stop 2024-04-17T10:25:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e19200640 1 -- 172.21.15.67:0/106551080 shutdown_connections 2024-04-17T10:25:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e19200640 1 --2- 172.21.15.67:0/106551080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140b4970 0x7f3e14163060 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.575+0000 7f3e19200640 1 -- 172.21.15.67:0/106551080 >> 172.21.15.67:0/106551080 conn(0x7f3e1406ac90 msgr2=0x7f3e1406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d68000640 1 -- start start 2024-04-17T10:25:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f3e19200640 1 -- 172.21.15.67:0/106551080 shutdown_connections 2024-04-17T10:25:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f3e19200640 1 -- 172.21.15.67:0/106551080 wait complete. 2024-04-17T10:25:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 shutdown_connections 2024-04-17T10:25:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 --2- 172.21.15.67:0/306717862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f58071310 0x7f2f580717c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 --2- 172.21.15.67:0/306717862 >> v2:172.21.15.67:3300/0 conn(0x7f2f58159bb0 0x7f2f580b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 >> 172.21.15.67:0/306717862 conn(0x7f2f5806ac90 msgr2=0x7f2f5806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 shutdown_connections 2024-04-17T10:25:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 -- 172.21.15.67:0/306717862 wait complete. 2024-04-17T10:25:22.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d68000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d6006ac90 0x7f7d600ad620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d68000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d600ce2c0 0x7f7d601554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b3400640 1 --2- 172.21.15.67:0/2125181088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d07f0 0x7fa9b40d0c40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40b60b0 tx=0x7fa9ac01c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=371c097346f35d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d65600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d6006ac90 0x7f7d600ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b2000640 1 -- 172.21.15.67:0/2125181088 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0b2070 con 0x7fa9b40d07f0 2024-04-17T10:25:22.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d65600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d6006ac90 0x7f7d600ad620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49322/0 (socket says 172.21.15.67:49322) 2024-04-17T10:25:22.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 -- 172.21.15.67:0/1379583127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d415aaa0 msgr2=0x7f70d40cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d65600640 1 -- 172.21.15.67:0/494440585 learned_addr learned my addr 172.21.15.67:0/494440585 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 --2- 172.21.15.67:0/1379583127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d415aaa0 0x7f70d40cd110 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f70c400bb10 tx=0x7f70c401eb70 comp rx=0 tx=0).stop 2024-04-17T10:25:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 -- 172.21.15.67:0/1379583127 shutdown_connections 2024-04-17T10:25:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 --2- 172.21.15.67:0/1379583127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d40ccb10 0x7f70d40cd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 --2- 172.21.15.67:0/1379583127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d415aaa0 0x7f70d40cd110 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 -- 172.21.15.67:0/1379583127 >> 172.21.15.67:0/1379583127 conn(0x7f70d40b23b0 msgr2=0x7f70d40b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d68000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d60161ff0 0x7f7d6015ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d64c00640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d600ce2c0 0x7f7d601554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f3e19200640 1 Processor -- start 2024-04-17T10:25:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 -- 172.21.15.67:0/1379583127 shutdown_connections 2024-04-17T10:25:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f3e19200640 1 -- start start 2024-04-17T10:25:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b2000640 1 -- 172.21.15.67:0/2125181088 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ac0a58f0 con 0x7fa9b40d07f0 2024-04-17T10:25:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d66000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d60161ff0 0x7f7d6015ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b2000640 1 -- 172.21.15.67:0/2125181088 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac0b2070 con 0x7fa9b40d07f0 2024-04-17T10:25:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 Processor -- start 2024-04-17T10:25:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f60200640 1 -- start start 2024-04-17T10:25:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b9600640 1 -- 172.21.15.67:0/2125181088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d07f0 msgr2=0x7fa9b40d0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b9600640 1 --2- 172.21.15.67:0/2125181088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d07f0 0x7fa9b40d0c40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40b60b0 tx=0x7fa9ac01c870 comp rx=0 tx=0).stop 2024-04-17T10:25:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f2f9fe00640 1 --2- 172.21.15.67:0/2920126816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806feb0 0x7f2fa806e6c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2fa806c8c0 tx=0x7f2f9801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f3d3dbdbe4d2927 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6016f0a0 con 0x7f7d60161ff0 2024-04-17T10:25:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6016ef20 con 0x7f7d600ce2c0 2024-04-17T10:25:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6016f220 con 0x7f7d6006ac90 2024-04-17T10:25:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b9600640 1 -- 172.21.15.67:0/2125181088 shutdown_connections 2024-04-17T10:25:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b9600640 1 --2- 172.21.15.67:0/2125181088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d07f0 0x7fa9b40d0c40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b9600640 1 --2- 172.21.15.67:0/2125181088 >> v2:172.21.15.86:3300/0 conn(0x7fa9b40d3510 0x7fa9b40b9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b9600640 1 --2- 172.21.15.67:0/2125181088 >> v2:172.21.15.67:3300/0 conn(0x7fa9b415dd30 0x7fa9b40d1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7fa9b9600640 1 -- 172.21.15.67:0/2125181088 >> 172.21.15.67:0/2125181088 conn(0x7fa9b4072590 msgr2=0x7fa9b40729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fa5e00640 1 -- 172.21.15.67:0/2920126816 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f980ab680 con 0x7f2fa806feb0 2024-04-17T10:25:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fa5e00640 1 -- 172.21.15.67:0/2920126816 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f980b0070 con 0x7f2fa806feb0 2024-04-17T10:25:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fa5e00640 1 -- 172.21.15.67:0/2920126816 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f980aba40 con 0x7f2fa806feb0 2024-04-17T10:25:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7fa9b9600640 1 -- 172.21.15.67:0/2125181088 shutdown_connections 2024-04-17T10:25:22.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d64c00640 1 -- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d6006ac90 msgr2=0x7f7d600ad620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d64c00640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d6006ac90 0x7f7d600ad620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d64c00640 1 -- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d60161ff0 msgr2=0x7f7d6015ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f60200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f58071310 0x7f2f58142d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d64c00640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d60161ff0 0x7f7d6015ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d64c00640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d601fb580 con 0x7f7d600ce2c0 2024-04-17T10:25:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f60200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58159bb0 0x7f2f58147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f60200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f5815e1f0 0x7f2f5814b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f60200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580ce470 con 0x7f2f58159bb0 2024-04-17T10:25:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f60200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580ce2f0 con 0x7f2f58071310 2024-04-17T10:25:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f60200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580ce5f0 con 0x7f2f5815e1f0 2024-04-17T10:25:22.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 -- 172.21.15.67:0/2920126816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806feb0 msgr2=0x7f2fa806e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 --2- 172.21.15.67:0/2920126816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806feb0 0x7f2fa806e6c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2fa806c8c0 tx=0x7f2f9801c460 comp rx=0 tx=0).stop 2024-04-17T10:25:22.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d66000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d60161ff0 0x7f7d6015ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:22.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f58071310 0x7f2f58142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f58071310 0x7f2f58142d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37124/0 (socket says 172.21.15.67:37124) 2024-04-17T10:25:22.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 -- 172.21.15.67:0/1168157306 learned_addr learned my addr 172.21.15.67:0/1168157306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7fa9b9600640 1 -- 172.21.15.67:0/2125181088 wait complete. 2024-04-17T10:25:22.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5e200640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f5815e1f0 0x7f2f5814b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d64c00640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d600ce2c0 0x7f7d601554c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0ac000 tx=0x7f7d4c01ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d65600640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d6006ac90 0x7f7d600ad620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.579+0000 7f70db200640 1 -- 172.21.15.67:0/1379583127 wait complete. 2024-04-17T10:25:22.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7fbb0d800640 1 Processor -- start 2024-04-17T10:25:22.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5ce00640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58159bb0 0x7f2f58147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0b1050 con 0x7f7d600ce2c0 2024-04-17T10:25:22.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7fbb0d800640 1 -- start start 2024-04-17T10:25:22.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c0bd070 con 0x7f7d600ce2c0 2024-04-17T10:25:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e14071310 0x7f3e14150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0b8670 con 0x7f7d600ce2c0 2024-04-17T10:25:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 -- 172.21.15.67:0/2920126816 shutdown_connections 2024-04-17T10:25:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 --2- 172.21.15.67:0/2920126816 >> v2:172.21.15.67:3300/0 conn(0x7f2fa80b4970 0x7f2fa806ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 --2- 172.21.15.67:0/2920126816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806feb0 0x7f2fa806e6c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 -- 172.21.15.67:0/2920126816 >> 172.21.15.67:0/2920126816 conn(0x7f2fa8071c80 msgr2=0x7f2fa8072090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 -- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f5815e1f0 msgr2=0x7f2f5814b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f5815e1f0 0x7f2f5814b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 -- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58159bb0 msgr2=0x7f2f58147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f58159bb0 0x7f2f58147940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 -- 172.21.15.67:0/1168157306 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f58162e70 con 0x7f2f58071310 2024-04-17T10:25:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d601fef80 con 0x7f7d600ce2c0 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e140b4970 0x7f3e1415e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140cef80 0x7f3e140b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140cf4e0 con 0x7f3e14071310 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e140cf360 con 0x7f3e140b4970 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1415f040 con 0x7f3e140cef80 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e140b4970 0x7f3e1415e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e140b4970 0x7f3e1415e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37112/0 (socket says 172.21.15.67:37112) 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 -- 172.21.15.67:0/763649812 learned_addr learned my addr 172.21.15.67:0/763649812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e12800640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140cef80 0x7f3e140b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 -- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140cef80 msgr2=0x7f3e140b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140cef80 0x7f3e140b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 -- 172.21.15.67:0/763649812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e14071310 msgr2=0x7f3e14150000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e14071310 0x7f3e14150000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 -- 172.21.15.67:0/763649812 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14150540 con 0x7f3e140b4970 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e12800640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e140cef80 0x7f3e140b10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0be00640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e140b4970 0x7f3e1415e260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc01ac00 tx=0x7f3dfc01ad90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0a9610 con 0x7f3e140b4970 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc0c2500 con 0x7f3e140b4970 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0c28b0 con 0x7f3e140b4970 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1415f720 con 0x7f3e140b4970 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1415f9c0 con 0x7f3e140b4970 2024-04-17T10:25:22.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 -- 172.21.15.67:0/2920126816 shutdown_connections 2024-04-17T10:25:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7fa9b9600640 1 Processor -- start 2024-04-17T10:25:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2f5d800640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f58071310 0x7f2f58142d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e890 tx=0x7f2f4400ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7fa9b9600640 1 -- start start 2024-04-17T10:25:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f2fad400640 1 -- 172.21.15.67:0/2920126816 wait complete. 2024-04-17T10:25:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb0d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb080ceb30 0x7fbb0815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb0d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb0815f3e0 0x7fbb0815f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e14072d50 con 0x7f3e140b4970 2024-04-17T10:25:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f44017070 con 0x7f2f58071310 2024-04-17T10:25:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb0d800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb08160150 con 0x7fbb081450b0 2024-04-17T10:25:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb0d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb0806d6e0 con 0x7fbb0815f3e0 2024-04-17T10:25:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb0d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0806d930 con 0x7fbb080ceb30 2024-04-17T10:25:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb07400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb080ceb30 0x7fbb0815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb07400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb080ceb30 0x7fbb0815bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49356/0 (socket says 172.21.15.67:49356) 2024-04-17T10:25:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb07400640 1 -- 172.21.15.67:0/812744085 learned_addr learned my addr 172.21.15.67:0/812744085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4400c940 con 0x7f2f58071310 2024-04-17T10:25:22.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4400cd10 con 0x7f2f58071310 2024-04-17T10:25:22.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f58163150 con 0x7f2f58071310 2024-04-17T10:25:22.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb07400640 1 -- 172.21.15.67:0/812744085 >> v2:172.21.15.67:3300/0 conn(0x7fbb0815f3e0 msgr2=0x7fbb0815f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb07400640 1 --2- 172.21.15.67:0/812744085 >> v2:172.21.15.67:3300/0 conn(0x7fbb0815f3e0 0x7fbb0815f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb07400640 1 -- 172.21.15.67:0/812744085 >> v1:172.21.15.86:6789/0 conn(0x7fbb081450b0 legacy=0x7fbb080ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f58163710 con 0x7f2f58071310 2024-04-17T10:25:22.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f58072d50 con 0x7f2f58071310 2024-04-17T10:25:22.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.583+0000 7f70db200640 1 Processor -- start 2024-04-17T10:25:22.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f70db200640 1 -- start start 2024-04-17T10:25:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d601ff540 con 0x7f7d600ce2c0 2024-04-17T10:25:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7d4c0b8810 con 0x7f7d600ce2c0 2024-04-17T10:25:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3dfc0c2a50 con 0x7f3e140b4970 2024-04-17T10:25:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f2fad400640 1 Processor -- start 2024-04-17T10:25:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fa9b9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b40d07f0 0x7fa9b4159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fa9b9600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d3510 0x7fa9b4160bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fbb07400640 1 -- 172.21.15.67:0/812744085 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0806dc10 con 0x7fbb080ceb30 2024-04-17T10:25:22.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7fa9b9600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9b415dd30 0x7fa9b4157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b40d4350 con 0x7fa9b40d07f0 2024-04-17T10:25:22.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b9600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b40d41d0 con 0x7fa9b415dd30 2024-04-17T10:25:22.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b9600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b40d44d0 con 0x7fa9b40d3510 2024-04-17T10:25:22.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9aa000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d3510 0x7fa9b4160bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e0a000640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3df0066fb0 0x7f3df0069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f3dfc0ff360 con 0x7f3e140b4970 2024-04-17T10:25:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9b415dd30 0x7fa9b4157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9b415dd30 0x7fa9b4157b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37136/0 (socket says 172.21.15.67:37136) 2024-04-17T10:25:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b3400640 1 -- 172.21.15.67:0/1408848866 learned_addr learned my addr 172.21.15.67:0/1408848866 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b2a00640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b40d07f0 0x7fa9b4159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7d6009e9a0 con 0x7f7d600ce2c0 2024-04-17T10:25:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f70db200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d40ccb10 0x7f70d4159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f70db200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70d40ce2d0 0x7f70d40adaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f70db200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d415aaa0 0x7f70d4156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f70db200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d40cc380 con 0x7f70d40ccb10 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f70db200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d40cc200 con 0x7f70d40ce2d0 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.587+0000 7f70db200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d406ac90 con 0x7f70d415aaa0 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70d40ce2d0 0x7f70d40adaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70d40ce2d0 0x7f70d40adaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37128/0 (socket says 172.21.15.67:37128) 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 -- 172.21.15.67:0/1895002253 learned_addr learned my addr 172.21.15.67:0/1895002253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e11e00640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3df0066fb0 0x7f3df0069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e0a000640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.67:6824/3611711736,v1:172.21.15.67:6825/3611711736] conn(0x7f3df0071440 0x7f3df00738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d5d600640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d40066fe0 0x7f7d400694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 --> [v2:172.21.15.67:6824/3611711736,v1:172.21.15.67:6825/3611711736] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3df0073fb0 con 0x7f3df0071440 2024-04-17T10:25:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dfc0fdcf0 con 0x7f3e140b4970 2024-04-17T10:25:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9aa000640 1 -- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9b415dd30 msgr2=0x7fa9b4157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9aa000640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9b415dd30 0x7fa9b4157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9aa000640 1 -- 172.21.15.67:0/1408848866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b40d07f0 msgr2=0x7fa9b4159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9aa000640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b40d07f0 0x7fa9b4159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2fad400640 1 -- start start 2024-04-17T10:25:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2f4401a070 con 0x7f2f58071310 2024-04-17T10:25:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7d4c0edb10 con 0x7f7d600ce2c0 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d5d600640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:6809/2698936194,v1:172.21.15.67:6811/2698936194] conn(0x7f7d400706e0 0x7f7d40072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.67:6809/2698936194,v1:172.21.15.67:6811/2698936194] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d400731f0 con 0x7f7d400706e0 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d65600640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d40066fe0 0x7f7d400694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d4c124060 con 0x7f7d600ce2c0 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e11e00640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3df0066fb0 0x7f3df0069470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e040014d0 tx=0x7f3e04001180 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9aa000640 1 -- 172.21.15.67:0/1408848866 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b4159f60 con 0x7fa9b40d3510 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b2a00640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9b40d07f0 0x7fa9b4159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9aa000640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d3510 0x7fa9b4160bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40d3910 tx=0x7fa9a400c5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d9200640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d415aaa0 0x7f70d4156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 -- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d415aaa0 msgr2=0x7f70d4156420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70d415aaa0 0x7f70d4156420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 -- 172.21.15.67:0/1895002253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d40ccb10 msgr2=0x7f70d4159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70d40ccb10 0x7f70d4159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 -- 172.21.15.67:0/1895002253 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d4159ac0 con 0x7f70d40ce2d0 2024-04-17T10:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e12800640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.67:6824/3611711736,v1:172.21.15.67:6825/3611711736] conn(0x7f3df0071440 0x7f3df00738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70d3400640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70d40ce2d0 0x7f70d40adaf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c7c0 tx=0x7f70c000cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c0007a30 con 0x7f70d40ce2d0 2024-04-17T10:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c0007bd0 con 0x7f70d40ce2d0 2024-04-17T10:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c00113f0 con 0x7f70d40ce2d0 2024-04-17T10:25:22.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4023070 con 0x7fa9b40d3510 2024-04-17T10:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a4015be0 con 0x7fa9b40d3510 2024-04-17T10:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4016030 con 0x7fa9b40d3510 2024-04-17T10:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d40ae590 con 0x7f70d40ce2d0 2024-04-17T10:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d40aeb50 con 0x7f70d40ce2d0 2024-04-17T10:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70d40af210 con 0x7f70d40ce2d0 2024-04-17T10:25:22.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2f55600640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f38067060 0x7f2f38069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2f44085550 con 0x7f2f58071310 2024-04-17T10:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2f55600640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x7f2f380707e0 0x7f2f38072c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 --> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f44014e50 con 0x7f2f380707e0 2024-04-17T10:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f4408c050 con 0x7f2f58071310 2024-04-17T10:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fbb07400640 1 --2- 172.21.15.67:0/812744085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb080ceb30 0x7fbb0815bd80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009870 tx=0x7fbaf401c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9aa5c4b1879419fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f3e12800640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.67:6824/3611711736,v1:172.21.15.67:6825/3611711736] conn(0x7f3df0071440 0x7f3df00738e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f2f5ce00640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f38067060 0x7f2f38069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7f7d66000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:6809/2698936194,v1:172.21.15.67:6811/2698936194] conn(0x7f7d400706e0 0x7f7d40072b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb06a00640 1 -- 172.21.15.67:0/812744085 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40b2070 con 0x7fbb080ceb30 2024-04-17T10:25:22.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f7d65600640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d40066fe0 0x7f7d400694a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005890 tx=0x7f7d54009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb06a00640 1 -- 172.21.15.67:0/812744085 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf40a9070 con 0x7fbb080ceb30 2024-04-17T10:25:22.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb06a00640 1 -- 172.21.15.67:0/812744085 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40ad880 con 0x7fbb080ceb30 2024-04-17T10:25:22.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2f5e200640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x7f2f380707e0 0x7f2f38072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806bca0 0x7f2fa80b0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f7d66000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:6809/2698936194,v1:172.21.15.67:6811/2698936194] conn(0x7f7d400706e0 0x7f7d40072b20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa806feb0 0x7f2fa80afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa80b4970 0x7f2fa81558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80ae040 con 0x7f2fa806feb0 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa806f760 con 0x7f2fa806bca0 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80ae1c0 con 0x7f2fa80b4970 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa80b4970 0x7f2fa81558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa80b4970 0x7f2fa81558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49400/0 (socket says 172.21.15.67:49400) 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 -- 172.21.15.67:0/4132065901 learned_addr learned my addr 172.21.15.67:0/4132065901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2f9fe00640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa806feb0 0x7f2fa80afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.591+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b415a240 con 0x7fa9b40d3510 2024-04-17T10:25:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b415a800 con 0x7fa9b40d3510 2024-04-17T10:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 -- 172.21.15.67:0/812744085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb080ceb30 msgr2=0x7fbb0815bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 --2- 172.21.15.67:0/812744085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb080ceb30 0x7fbb0815bd80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4009870 tx=0x7fbaf401c490 comp rx=0 tx=0).stop 2024-04-17T10:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 -- 172.21.15.67:0/812744085 shutdown_connections 2024-04-17T10:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 --2- 172.21.15.67:0/812744085 >> v2:172.21.15.67:3300/0 conn(0x7fbb0815f3e0 0x7fbb0815f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 --2- 172.21.15.67:0/812744085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb080ceb30 0x7fbb0815bd80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 -- 172.21.15.67:0/812744085 >> 172.21.15.67:0/812744085 conn(0x7fbb08067070 msgr2=0x7fbb0806ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa6800640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806bca0 0x7f2fa80b0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== osd.3 v2:172.21.15.67:6824/3611711736 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3df0073fb0 con 0x7f3df0071440 2024-04-17T10:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 -- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806bca0 msgr2=0x7f2fa80b0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2f5ce00640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f38067060 0x7f2f38069520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f5815e150 tx=0x7f2f4c007cf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806bca0 0x7f2fa80b0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 -- 172.21.15.67:0/4132065901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa806feb0 msgr2=0x7f2fa80afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa806feb0 0x7f2fa80afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 -- 172.21.15.67:0/4132065901 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80b0d10 con 0x7f2fa80b4970 2024-04-17T10:25:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa6800640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fa806bca0 0x7f2fa80b0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 -- 172.21.15.67:0/812744085 shutdown_connections 2024-04-17T10:25:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2f9fe00640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fa806feb0 0x7f2fa80afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fa7200640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa80b4970 0x7f2fa81558a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0007ef0 tx=0x7f2fa000d4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== osd.2 v2:172.21.15.67:6809/2698936194 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7d400731f0 con 0x7f7d400706e0 2024-04-17T10:25:22.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f70c0011590 con 0x7f70d40ce2d0 2024-04-17T10:25:22.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa001a070 con 0x7f2fa80b4970 2024-04-17T10:25:22.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f70cbe00640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70b4067030 0x7f70b40694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa0003ba0 con 0x7f2fa80b4970 2024-04-17T10:25:22.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0004030 con 0x7f2fa80b4970 2024-04-17T10:25:22.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa80b0ff0 con 0x7f2fa80b4970 2024-04-17T10:25:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa80b15b0 con 0x7f2fa80b4970 2024-04-17T10:25:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f70c0085ab0 con 0x7f70d40ce2d0 2024-04-17T10:25:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f70cbe00640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x7f70b40707b0 0x7f70b4072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 --> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70c0025ea0 con 0x7f70b40707b0 2024-04-17T10:25:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70c0085ea0 con 0x7f70d40ce2d0 2024-04-17T10:25:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f2f5e200640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x7f2f380707e0 0x7f2f38072c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f70d9200640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x7f70b40707b0 0x7f70b4072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f70d3e00640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70b4067030 0x7f70b40694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa9b408afe0 con 0x7fa9b40d3510 2024-04-17T10:25:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.595+0000 7fbb0d800640 1 -- 172.21.15.67:0/812744085 wait complete. 2024-04-17T10:25:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2fa809e1f0 con 0x7f2fa80b4970 2024-04-17T10:25:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f70d9200640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x7f70b40707b0 0x7f70b4072bf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa9a4026070 con 0x7fa9b40d3510 2024-04-17T10:25:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== osd.1 v2:172.21.15.67:6808/2092989796 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f70c0025ea0 con 0x7f70b40707b0 2024-04-17T10:25:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9ab400640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa990067030 0x7fa9900694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== osd.0 v2:172.21.15.67:6800/3376065543 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2f44014e50 con 0x7f2f380707e0 2024-04-17T10:25:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f70d3e00640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70b4067030 0x7f70b40694f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f70c40b1000 tx=0x7f70c4004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa9a408d110 con 0x7fa9b40d3510 2024-04-17T10:25:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9b2a00640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa990067030 0x7fa9900694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9ab400640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6816/1567645357,v1:172.21.15.86:6817/1567645357] conn(0x7fa990070720 0x7fa990072b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 --> [v2:172.21.15.86:6816/1567645357,v1:172.21.15.86:6817/1567645357] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9a400ae50 con 0x7fa990070720 2024-04-17T10:25:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9a4094050 con 0x7fa9b40d3510 2024-04-17T10:25:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2fa001d070 con 0x7f2fa80b4970 2024-04-17T10:25:22.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9b3400640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6816/1567645357,v1:172.21.15.86:6817/1567645357] conn(0x7fa990070720 0x7fa990072b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.599+0000 7fa9b2a00640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa990067030 0x7fa9900694f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0097a0 tx=0x7fa99c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fbb0d800640 1 Processor -- start 2024-04-17T10:25:22.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fbb0d800640 1 -- start start 2024-04-17T10:25:22.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fa9b3400640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6816/1567645357,v1:172.21.15.86:6817/1567645357] conn(0x7fa990070720 0x7fa990072b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2f9ea00640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f84067030 0x7f2f840694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2fa0085600 con 0x7f2fa80b4970 2024-04-17T10:25:22.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2f9ea00640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x7f2f84070730 0x7f2f84072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 --> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f84073240 con 0x7f2f84070730 2024-04-17T10:25:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fa008c050 con 0x7f2fa80b4970 2024-04-17T10:25:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2fa6800640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f84067030 0x7f2f840694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2f9fe00640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x7f2f84070730 0x7f2f84072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== osd.5 v2:172.21.15.86:6816/1567645357 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fa9a400ae50 con 0x7fa990070720 2024-04-17T10:25:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fd06d400640 1 Processor -- start 2024-04-17T10:25:22.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fbb0d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb080ceb30 0x7fbb08163100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2fa6800640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f84067030 0x7f2f840694f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2f90000f70 tx=0x7f2f90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7f2f9fe00640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x7f2f84070730 0x7f2f84072b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fbb0d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb081450b0 0x7fbb0813bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fbb0d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0815f3e0 0x7fbb0815e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fbb0d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0806ec20 con 0x7fbb080ceb30 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb081450b0 0x7fbb0813bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb081450b0 0x7fbb0813bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37150/0 (socket says 172.21.15.67:37150) 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 -- 172.21.15.67:0/3914855748 learned_addr learned my addr 172.21.15.67:0/3914855748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb0ce00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0815f3e0 0x7fbb0815e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb0d800640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0815ff30 con 0x7fbb081450b0 2024-04-17T10:25:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb0d800640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0806eda0 con 0x7fbb0815f3e0 2024-04-17T10:25:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== osd.4 v2:172.21.15.86:6800/710871084 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2f84073240 con 0x7f2f84070730 2024-04-17T10:25:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07e00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb080ceb30 0x7fbb08163100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 -- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0815f3e0 msgr2=0x7fbb0815e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0815f3e0 0x7fbb0815e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 -- 172.21.15.67:0/3914855748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb080ceb30 msgr2=0x7fbb08163100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.603+0000 7fd06d400640 1 -- start start 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd06d400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd0680cef80 0x7fd068163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd06d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0680719a0 con 0x7fd068071310 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd06d400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd068071c80 con 0x7fd0680b4970 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd06d400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd068162580 con 0x7fd0680cef80 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb080ceb30 0x7fbb08163100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb081ff210 con 0x7fbb081450b0 2024-04-17T10:25:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07e00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb080ceb30 0x7fbb08163100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd066800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd0680cef80 0x7fd068163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd066800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd0680cef80 0x7fd068163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37152/0 (socket says 172.21.15.67:37152) 2024-04-17T10:25:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd066800640 1 -- 172.21.15.67:0/1652278663 learned_addr learned my addr 172.21.15.67:0/1652278663 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb07400640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb081450b0 0x7fbb0813bc80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbaf401cd30 tx=0x7fbaf4019230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730906248 0 0) 0x7fd068071c80 con 0x7fd0680b4970 2024-04-17T10:25:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd04c003530 con 0x7fd0680b4970 2024-04-17T10:25:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40bb020 con 0x7fbb081450b0 2024-04-17T10:25:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf40b3040 con 0x7fbb081450b0 2024-04-17T10:25:22.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb0d800640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb081fe690 con 0x7fbb081450b0 2024-04-17T10:25:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673582816 0 0) 0x7fd0680719a0 con 0x7fd068071310 2024-04-17T10:25:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd068071c80 con 0x7fd068071310 2024-04-17T10:25:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf40b2050 con 0x7fbb081450b0 2024-04-17T10:25:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.607+0000 7fbb0d800640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb081fec50 con 0x7fbb081450b0 2024-04-17T10:25:22.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.611+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2132405102 0 0) 0x7fd04c003530 con 0x7fd0680b4970 2024-04-17T10:25:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.611+0000 7fd066800640 1 -- 172.21.15.67:0/1652278663 >> v1:172.21.15.86:6789/0 conn(0x7fd0680b4970 legacy=0x7fd0680b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.611+0000 7fd066800640 1 -- 172.21.15.67:0/1652278663 >> v1:172.21.15.67:6789/0 conn(0x7fd068071310 legacy=0x7fd0680b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.611+0000 7fd066800640 1 -- 172.21.15.67:0/1652278663 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd068162860 con 0x7fd0680cef80 2024-04-17T10:25:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.611+0000 7fbb0d800640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbb08074dc0 con 0x7fbb081450b0 2024-04-17T10:25:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbaf40be020 con 0x7fbb081450b0 2024-04-17T10:25:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbaffe00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae8067030 0x7fbae80694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbaf411f080 con 0x7fbb081450b0 2024-04-17T10:25:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fd066800640 1 --2- 172.21.15.67:0/1652278663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0680cef80 0x7fd068163060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0580097f0 tx=0x7fd05801c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=647335167d798b49 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbaffe00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6824/4003922523,v1:172.21.15.86:6825/4003922523] conn(0x7fbae80707b0 0x7fbae8072bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:6824/4003922523,v1:172.21.15.86:6825/4003922523] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbaf40c1e50 con 0x7fbae80707b0 2024-04-17T10:25:22.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbaf4126050 con 0x7fbb081450b0 2024-04-17T10:25:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0580b0070 con 0x7fd0680cef80 2024-04-17T10:25:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbb0ce00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6824/4003922523,v1:172.21.15.86:6825/4003922523] conn(0x7fbae80707b0 0x7fbae8072bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0580ab400 con 0x7fd0680cef80 2024-04-17T10:25:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fd065e00640 1 -- 172.21.15.67:0/1652278663 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060001a00 con 0x7fd0680cef80 2024-04-17T10:25:22.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.615+0000 7fbb0ce00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6824/4003922523,v1:172.21.15.86:6825/4003922523] conn(0x7fbae80707b0 0x7fbae8072bf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fd06d400640 1 -- 172.21.15.67:0/1652278663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0680cef80 msgr2=0x7fd068163060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fbb07e00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae8067030 0x7fbae80694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fd06d400640 1 --2- 172.21.15.67:0/1652278663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0680cef80 0x7fd068163060 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0580097f0 tx=0x7fd05801c420 comp rx=0 tx=0).stop 2024-04-17T10:25:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fd06d400640 1 -- 172.21.15.67:0/1652278663 shutdown_connections 2024-04-17T10:25:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fd06d400640 1 --2- 172.21.15.67:0/1652278663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0680cef80 0x7fd068163060 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fd06d400640 1 -- 172.21.15.67:0/1652278663 >> 172.21.15.67:0/1652278663 conn(0x7fd06806ac90 msgr2=0x7fd06806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== osd.6 v2:172.21.15.86:6824/4003922523 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fbaf40c1e50 con 0x7fbae80707b0 2024-04-17T10:25:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fd06d400640 1 -- 172.21.15.67:0/1652278663 shutdown_connections 2024-04-17T10:25:22.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fd06d400640 1 -- 172.21.15.67:0/1652278663 wait complete. 2024-04-17T10:25:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.619+0000 7fbb07e00640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae8067030 0x7fbae80694f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8003e40 tx=0x7fbaf8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 Processor -- start 2024-04-17T10:25:22.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 -- start start 2024-04-17T10:25:22.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 --> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f70d409ea10 con 0x7f70b40707b0 2024-04-17T10:25:22.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7f70cbe00640 1 -- 172.21.15.67:0/1895002253 <== osd.1 v2:172.21.15.67:6808/2092989796 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f70d409ea10 con 0x7f70b40707b0 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd068071310 0x7fd068150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0680b4970 0x7fd068155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0680cef80 0x7fd068155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd066800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd068071310 0x7fd068150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd066800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd068071310 0x7fd068150c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37154/0 (socket says 172.21.15.67:37154) 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd066800640 1 -- 172.21.15.67:0/1849290369 learned_addr learned my addr 172.21.15.67:0/1849290369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06815f1c0 con 0x7fd0680cef80 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06815f040 con 0x7fd068071310 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.623+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd06815f340 con 0x7fd0680b4970 2024-04-17T10:25:22.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd067200640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0680cef80 0x7fd068155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x7f70b40707b0 msgr2=0x7f70b4072bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x7f70b40707b0 0x7f70b4072bf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd066800640 1 -- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0680b4970 msgr2=0x7fd068155310 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T10:25:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd066800640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0680b4970 0x7fd068155310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd066800640 1 -- 172.21.15.67:0/1849290369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0680cef80 msgr2=0x7fd068155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd066800640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0680cef80 0x7fd068155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd066800640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0680b2c20 con 0x7fd068071310 2024-04-17T10:25:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70b4067030 msgr2=0x7f70b40694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70b4067030 0x7f70b40694f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f70c40b1000 tx=0x7f70c4004740 comp rx=0 tx=0).stop 2024-04-17T10:25:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70d40ce2d0 msgr2=0x7f70d40adaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 --2- 172.21.15.67:0/1895002253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70d40ce2d0 0x7f70d40adaf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c7c0 tx=0x7f70c000cc80 comp rx=0 tx=0).stop 2024-04-17T10:25:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70d9200640 1 -- 172.21.15.67:0/1895002253 reap_dead start 2024-04-17T10:25:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 shutdown_connections 2024-04-17T10:25:22.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 >> 172.21.15.67:0/1895002253 conn(0x7f70d40b23b0 msgr2=0x7f70d40b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd066800640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd068071310 0x7fd068150c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd0580ae040 tx=0x7fd05801ac60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 shutdown_connections 2024-04-17T10:25:22.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f70db200640 1 -- 172.21.15.67:0/1895002253 wait complete. 2024-04-17T10:25:22.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0580bb020 con 0x7fd068071310 2024-04-17T10:25:22.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0680af150 con 0x7fd068071310 2024-04-17T10:25:22.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0680af710 con 0x7fd068071310 2024-04-17T10:25:22.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd058004020 con 0x7fd068071310 2024-04-17T10:25:22.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0580b0040 con 0x7fd068071310 2024-04-17T10:25:22.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 --> [v2:172.21.15.67:6809/2698936194,v1:172.21.15.67:6811/2698936194] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7d60074e30 con 0x7f7d400706e0 2024-04-17T10:25:22.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.627+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd040001040 con 0x7fd068071310 2024-04-17T10:25:22.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d5d600640 1 -- 172.21.15.67:0/494440585 <== osd.2 v2:172.21.15.67:6809/2698936194 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7d60074e30 con 0x7f7d400706e0 2024-04-17T10:25:22.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:6809/2698936194,v1:172.21.15.67:6811/2698936194] conn(0x7f7d400706e0 msgr2=0x7f7d40072b20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 --> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2f58074e30 con 0x7f2f380707e0 2024-04-17T10:25:22.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.67:6809/2698936194,v1:172.21.15.67:6811/2698936194] conn(0x7f7d400706e0 0x7f7d40072b20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d40066fe0 msgr2=0x7f7d400694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d40066fe0 0x7f7d400694a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005890 tx=0x7f7d54009040 comp rx=0 tx=0).stop 2024-04-17T10:25:22.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d600ce2c0 msgr2=0x7f7d601554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 --2- 172.21.15.67:0/494440585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d600ce2c0 0x7f7d601554c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c0ac000 tx=0x7f7d4c01ace0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d66000640 1 -- 172.21.15.67:0/494440585 reap_dead start 2024-04-17T10:25:22.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f2f55600640 1 -- 172.21.15.67:0/1168157306 <== osd.0 v2:172.21.15.67:6800/3376065543 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2f58074e30 con 0x7f2f380707e0 2024-04-17T10:25:22.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 --> [v2:172.21.15.67:6824/3611711736,v1:172.21.15.67:6825/3611711736] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3e14074e30 con 0x7f3df0071440 2024-04-17T10:25:22.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd0580be070 con 0x7fd068071310 2024-04-17T10:25:22.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 shutdown_connections 2024-04-17T10:25:22.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 >> 172.21.15.67:0/494440585 conn(0x7f7d600b2340 msgr2=0x7f7d600b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 shutdown_connections 2024-04-17T10:25:22.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.631+0000 7f7d68000640 1 -- 172.21.15.67:0/494440585 wait complete. 2024-04-17T10:25:22.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x7f2f380707e0 msgr2=0x7f2f38072c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x7f2f380707e0 0x7f2f38072c20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e0a000640 1 -- 172.21.15.67:0/763649812 <== osd.3 v2:172.21.15.67:6824/3611711736 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3e14074e30 con 0x7f3df0071440 2024-04-17T10:25:22.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd05ea00640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd044066f60 0x7fd044069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd05811e950 con 0x7fd068071310 2024-04-17T10:25:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd05fe00640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd044066f60 0x7fd044069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f38067060 msgr2=0x7f2f38069520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f38067060 0x7f2f38069520 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2f5815e150 tx=0x7f2f4c007cf0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f58071310 msgr2=0x7f2f58142d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 --2- 172.21.15.67:0/1168157306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f58071310 0x7f2f58142d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2f4400e890 tx=0x7f2f4400ed50 comp rx=0 tx=0).stop 2024-04-17T10:25:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f5e200640 1 -- 172.21.15.67:0/1168157306 reap_dead start 2024-04-17T10:25:22.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd05ea00640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6808/2202166568,v1:172.21.15.86:6809/2202166568] conn(0x7fd0440705e0 0x7fd044072a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:6808/2202166568,v1:172.21.15.86:6809/2202166568] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0440730f0 con 0x7fd0440705e0 2024-04-17T10:25:22.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd05811ede0 con 0x7fd068071310 2024-04-17T10:25:22.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 shutdown_connections 2024-04-17T10:25:22.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 >> 172.21.15.67:0/1168157306 conn(0x7f2f5806ac90 msgr2=0x7f2f580af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 shutdown_connections 2024-04-17T10:25:22.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 >> [v2:172.21.15.67:6824/3611711736,v1:172.21.15.67:6825/3611711736] conn(0x7f3df0071440 msgr2=0x7f3df00738e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd067200640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6808/2202166568,v1:172.21.15.86:6809/2202166568] conn(0x7fd0440705e0 0x7fd044072a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd05fe00640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd044066f60 0x7fd044069420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd068154140 tx=0x7fd0500078b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f2f60200640 1 -- 172.21.15.67:0/1168157306 wait complete. 2024-04-17T10:25:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.67:6824/3611711736,v1:172.21.15.67:6825/3611711736] conn(0x7f3df0071440 0x7f3df00738e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 --> [v2:172.21.15.86:6816/1567645357,v1:172.21.15.86:6817/1567645357] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa9b40c6d30 con 0x7fa990070720 2024-04-17T10:25:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3df0066fb0 msgr2=0x7f3df0069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3df0066fb0 0x7f3df0069470 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f3e040014d0 tx=0x7f3e04001180 comp rx=0 tx=0).stop 2024-04-17T10:25:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e140b4970 msgr2=0x7f3e1415e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 --2- 172.21.15.67:0/763649812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e140b4970 0x7f3e1415e260 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc01ac00 tx=0x7f3dfc01ad90 comp rx=0 tx=0).stop 2024-04-17T10:25:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e12800640 1 -- 172.21.15.67:0/763649812 reap_dead start 2024-04-17T10:25:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fd067200640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6808/2202166568,v1:172.21.15.86:6809/2202166568] conn(0x7fd0440705e0 0x7fd044072a20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 shutdown_connections 2024-04-17T10:25:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 >> 172.21.15.67:0/763649812 conn(0x7f3e1406ac90 msgr2=0x7f3e141657d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 shutdown_connections 2024-04-17T10:25:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7f3e19200640 1 -- 172.21.15.67:0/763649812 wait complete. 2024-04-17T10:25:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.635+0000 7fa9ab400640 1 -- 172.21.15.67:0/1408848866 <== osd.5 v2:172.21.15.86:6816/1567645357 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa9b40c6d30 con 0x7fa990070720 2024-04-17T10:25:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== osd.7 v2:172.21.15.86:6808/2202166568 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd0440730f0 con 0x7fd0440705e0 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6816/1567645357,v1:172.21.15.86:6817/1567645357] conn(0x7fa990070720 msgr2=0x7fa990072b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fa9b9600640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6816/1567645357,v1:172.21.15.86:6817/1567645357] conn(0x7fa990070720 0x7fa990072b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa990067030 msgr2=0x7fa9900694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fa9b9600640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa990067030 0x7fa9900694f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0097a0 tx=0x7fa99c008040 comp rx=0 tx=0).stop 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d3510 msgr2=0x7fa9b4160bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fa9b9600640 1 --2- 172.21.15.67:0/1408848866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9b40d3510 0x7fa9b4160bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40d3910 tx=0x7fa9a400c5f0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7fa9b3400640 1 -- 172.21.15.67:0/1408848866 reap_dead start 2024-04-17T10:25:22.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.639+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 --> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2fa8074e30 con 0x7f2f84070730 2024-04-17T10:25:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 shutdown_connections 2024-04-17T10:25:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2f9ea00640 1 -- 172.21.15.67:0/4132065901 <== osd.4 v2:172.21.15.86:6800/710871084 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2fa8074e30 con 0x7f2f84070730 2024-04-17T10:25:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 >> 172.21.15.67:0/1408848866 conn(0x7fa9b4072590 msgr2=0x7fa9b40b4190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x7f2f84070730 msgr2=0x7f2f84072b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 shutdown_connections 2024-04-17T10:25:22.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7fa9b9600640 1 -- 172.21.15.67:0/1408848866 wait complete. 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x7f2f84070730 0x7f2f84072b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f84067030 msgr2=0x7f2f840694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f84067030 0x7f2f840694f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2f90000f70 tx=0x7f2f90009040 comp rx=0 tx=0).stop 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa80b4970 msgr2=0x7f2fa81558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 --2- 172.21.15.67:0/4132065901 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fa80b4970 0x7f2fa81558a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2fa0007ef0 tx=0x7f2fa000d4a0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fa7200640 1 -- 172.21.15.67:0/4132065901 reap_dead start 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 shutdown_connections 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 >> 172.21.15.67:0/4132065901 conn(0x7f2fa8071c80 msgr2=0x7f2fa806c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 shutdown_connections 2024-04-17T10:25:22.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.643+0000 7f2fad400640 1 -- 172.21.15.67:0/4132065901 wait complete. 2024-04-17T10:25:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.647+0000 7fbb0d800640 1 -- 172.21.15.67:0/3914855748 --> [v2:172.21.15.86:6824/4003922523,v1:172.21.15.86:6825/4003922523] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbb080b0920 con 0x7fbae80707b0 2024-04-17T10:25:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.647+0000 7fbaffe00640 1 -- 172.21.15.67:0/3914855748 <== osd.6 v2:172.21.15.86:6824/4003922523 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbb080b0920 con 0x7fbae80707b0 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 -- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6824/4003922523,v1:172.21.15.86:6825/4003922523] conn(0x7fbae80707b0 msgr2=0x7fbae8072bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6824/4003922523,v1:172.21.15.86:6825/4003922523] conn(0x7fbae80707b0 0x7fbae8072bf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 -- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae8067030 msgr2=0x7fbae80694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae8067030 0x7fbae80694f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8003e40 tx=0x7fbaf8009040 comp rx=0 tx=0).stop 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 -- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb081450b0 msgr2=0x7fbb0813bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 --2- 172.21.15.67:0/3914855748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb081450b0 0x7fbb0813bc80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbaf401cd30 tx=0x7fbaf4019230 comp rx=0 tx=0).stop 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbb0ce00640 1 -- 172.21.15.67:0/3914855748 reap_dead start 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 -- 172.21.15.67:0/3914855748 shutdown_connections 2024-04-17T10:25:22.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 -- 172.21.15.67:0/3914855748 >> 172.21.15.67:0/3914855748 conn(0x7fbb08067070 msgr2=0x7fbb0815cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 -- 172.21.15.67:0/3914855748 shutdown_connections 2024-04-17T10:25:22.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.651+0000 7fbafd600640 1 -- 172.21.15.67:0/3914855748 wait complete. 2024-04-17T10:25:22.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.667+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 --> [v2:172.21.15.86:6808/2202166568,v1:172.21.15.86:6809/2202166568] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd040002de0 con 0x7fd0440705e0 2024-04-17T10:25:22.676 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-04-17T10:25:22.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-17T10:25:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.667+0000 7fd05ea00640 1 -- 172.21.15.67:0/1849290369 <== osd.7 v2:172.21.15.86:6808/2202166568 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd040002de0 con 0x7fd0440705e0 2024-04-17T10:25:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.671+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6808/2202166568,v1:172.21.15.86:6809/2202166568] conn(0x7fd0440705e0 msgr2=0x7fd044072a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.671+0000 7fd06d400640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6808/2202166568,v1:172.21.15.86:6809/2202166568] conn(0x7fd0440705e0 0x7fd044072a20 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.671+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd044066f60 msgr2=0x7fd044069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.671+0000 7fd06d400640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd044066f60 0x7fd044069420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd068154140 tx=0x7fd0500078b0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.671+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd068071310 msgr2=0x7fd068150c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.671+0000 7fd06d400640 1 --2- 172.21.15.67:0/1849290369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd068071310 0x7fd068150c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd0580ae040 tx=0x7fd05801ac60 comp rx=0 tx=0).stop 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.675+0000 7fd067200640 1 -- 172.21.15.67:0/1849290369 reap_dead start 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.675+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 shutdown_connections 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.675+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 >> 172.21.15.67:0/1849290369 conn(0x7fd06806ac90 msgr2=0x7fd06815a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.675+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 shutdown_connections 2024-04-17T10:25:22.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.675+0000 7fd06d400640 1 -- 172.21.15.67:0/1849290369 wait complete. 2024-04-17T10:25:22.679 INFO:teuthology.orchestra.run.smithi067.stdout:55834574860 2024-04-17T10:25:22.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-17T10:25:22.680 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:22.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-17T10:25:22.681 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:22.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-17T10:25:22.682 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:22.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-17T10:25:22.683 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:22.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-17T10:25:22.685 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-04-17T10:25:22.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-17T10:25:22.693 INFO:teuthology.orchestra.run.smithi067.stdout:55834574860 2024-04-17T10:25:22.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-17T10:25:22.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9c0000640 1 Processor -- start 2024-04-17T10:25:22.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9c0000640 1 -- start start 2024-04-17T10:25:22.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9c0000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0033e10 0x7fc9b00eb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9c0000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9b0005ec0 0x7fc9b0006310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9bcc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc9b0047bf0 0x7fc9b00eb690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54354/0 (socket says 172.21.15.67:54354) 2024-04-17T10:25:22.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9c0000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b00341f0 con 0x7fc9b0047bf0 2024-04-17T10:25:22.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9c0000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b00501e0 con 0x7fc9b0005ec0 2024-04-17T10:25:22.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9c0000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b0037b90 con 0x7fc9b0033e10 2024-04-17T10:25:22.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9be000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9b0005ec0 0x7fc9b0006310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9bd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0033e10 0x7fc9b00eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.939+0000 7fc9bcc00640 1 -- 172.21.15.67:0/2173651309 learned_addr learned my addr 172.21.15.67:0/2173651309 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7fc9bd600640 1 -- 172.21.15.67:0/2173651309 >> v2:172.21.15.67:3300/0 conn(0x7fc9b0005ec0 msgr2=0x7fc9b0006310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7f9ec2a00640 1 Processor -- start 2024-04-17T10:25:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7fc9bd600640 1 --2- 172.21.15.67:0/2173651309 >> v2:172.21.15.67:3300/0 conn(0x7fc9b0005ec0 0x7fc9b0006310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7f9ec2a00640 1 -- start start 2024-04-17T10:25:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7fc9bd600640 1 -- 172.21.15.67:0/2173651309 >> v1:172.21.15.67:6789/0 conn(0x7fc9b0047bf0 legacy=0x7fc9b00eb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7fc9bd600640 1 -- 172.21.15.67:0/2173651309 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b004fa00 con 0x7fc9b0033e10 2024-04-17T10:25:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7f9ec2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ebc0ad690 0x7f9ebc0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7fa31e200640 1 Processor -- start 2024-04-17T10:25:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7fa31e200640 1 -- start start 2024-04-17T10:25:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7f9ec2a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ebc0cc3f0 con 0x7f9ebc0d00e0 2024-04-17T10:25:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7f9ec2a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ebc0cf150 con 0x7f9ebc144920 2024-04-17T10:25:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.943+0000 7f9ec2a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ebc0cc660 con 0x7f9ebc0ad690 2024-04-17T10:25:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebbe00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9ebc144920 0x7f9ebc0cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54768/0 (socket says 172.21.15.67:54768) 2024-04-17T10:25:22.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebbe00640 1 -- 172.21.15.67:0/256543837 learned_addr learned my addr 172.21.15.67:0/256543837 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebb400640 1 --2- 172.21.15.67:0/256543837 >> v2:172.21.15.67:3300/0 conn(0x7f9ebc0ad690 0x7f9ebc0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fc9bd600640 1 --2- 172.21.15.67:0/2173651309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0033e10 0x7fc9b00eb150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80649d0 tx=0x7fc9b8099b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a62fc4757b59a21f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024500949 0 0) 0x7f9ebc0cf150 con 0x7f9ebc144920 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea0007940 con 0x7f9ebc144920 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 62139996 0 0) 0x7f9ea0007940 con 0x7f9ebc144920 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 >> v2:172.21.15.67:3300/0 conn(0x7f9ebc0ad690 msgr2=0x7f9ebc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 --2- 172.21.15.67:0/256543837 >> v2:172.21.15.67:3300/0 conn(0x7f9ebc0ad690 0x7f9ebc0ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 >> v1:172.21.15.67:6789/0 conn(0x7f9ebc0d00e0 legacy=0x7f9ebc0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc0cc940 con 0x7f9ebc144920 2024-04-17T10:25:22.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f9ea4001680 con 0x7f9ebc144920 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 >> v1:172.21.15.86:6789/0 conn(0x7f9ebc144920 legacy=0x7f9ebc0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ea0007e20 0x7f9ea0008240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0008c20 0x7f9ea00090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ea00095e0 0x7f9ea00145d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc0cc660 con 0x7f9ea0008c20 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea00076d0 con 0x7f9ea0007e20 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0014b90 con 0x7f9ea00095e0 2024-04-17T10:25:22.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebbe00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ea0007e20 0x7f9ea0008240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fc9b7e00640 1 -- 172.21.15.67:0/2173651309 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b81237f0 con 0x7fc9b0033e10 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fc9b7e00640 1 -- 172.21.15.67:0/2173651309 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8123990 con 0x7fc9b0033e10 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fc9b7e00640 1 -- 172.21.15.67:0/2173651309 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b8123e00 con 0x7fc9b0033e10 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa31e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa318161e80 0x7fa3180d07a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa31e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa3180d0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa31e200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa318166880 con 0x7fa318163270 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa31e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa318162790 con 0x7fa318161e80 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa31e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa318162a70 con 0x7fa318162260 2024-04-17T10:25:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa316a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa3180d0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa316a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa3180d0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49438/0 (socket says 172.21.15.67:49438) 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa316a00640 1 -- 172.21.15.67:0/368865522 learned_addr learned my addr 172.21.15.67:0/368865522 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa317400640 1 --2- 172.21.15.67:0/368865522 >> v2:172.21.15.67:3300/0 conn(0x7fa318161e80 0x7fa3180d07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa316a00640 1 -- 172.21.15.67:0/368865522 >> v2:172.21.15.67:3300/0 conn(0x7fa318161e80 msgr2=0x7fa3180d07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa316a00640 1 --2- 172.21.15.67:0/368865522 >> v2:172.21.15.67:3300/0 conn(0x7fa318161e80 0x7fa3180d07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa316a00640 1 -- 172.21.15.67:0/368865522 >> v1:172.21.15.67:6789/0 conn(0x7fa318163270 legacy=0x7fa318165b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7fa316a00640 1 -- 172.21.15.67:0/368865522 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa318162d50 con 0x7fa318162260 2024-04-17T10:25:22.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.947+0000 7f9ebb400640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0008c20 0x7f9ea00090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 -- 172.21.15.67:0/2173651309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0033e10 msgr2=0x7fc9b00eb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 --2- 172.21.15.67:0/2173651309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0033e10 0x7fc9b00eb150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc9b80649d0 tx=0x7fc9b8099b30 comp rx=0 tx=0).stop 2024-04-17T10:25:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 -- 172.21.15.67:0/2173651309 shutdown_connections 2024-04-17T10:25:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 --2- 172.21.15.67:0/2173651309 >> v2:172.21.15.67:3300/0 conn(0x7fc9b0005ec0 0x7fc9b0006310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 --2- 172.21.15.67:0/2173651309 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0033e10 0x7fc9b00eb150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 -- 172.21.15.67:0/2173651309 >> 172.21.15.67:0/2173651309 conn(0x7fc9b0009f90 msgr2=0x7fc9b000a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 -- 172.21.15.67:0/2173651309 shutdown_connections 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 -- 172.21.15.67:0/2173651309 wait complete. 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7f9ebbe00640 1 -- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ea00095e0 msgr2=0x7f9ea00145d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7f9ebbe00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ea00095e0 0x7f9ea00145d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7f9ebbe00640 1 -- 172.21.15.67:0/256543837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0008c20 msgr2=0x7f9ea00090a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7f9ebbe00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0008c20 0x7f9ea00090a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7f9ebbe00640 1 -- 172.21.15.67:0/256543837 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc0cc940 con 0x7f9ea0007e20 2024-04-17T10:25:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7f9ec0a00640 1 -- 172.21.15.67:0/256543837 reap_dead start 2024-04-17T10:25:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.951+0000 7fc9c0000640 1 Processor -- start 2024-04-17T10:25:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9c0000640 1 -- start start 2024-04-17T10:25:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f78da600640 1 Processor -- start 2024-04-17T10:25:22.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa316a00640 1 --2- 172.21.15.67:0/368865522 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa3180d0ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa304009870 tx=0x7fa30401c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5b9efe50d843e61 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f6f5ce00640 1 Processor -- start 2024-04-17T10:25:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f6f5ce00640 1 -- start start 2024-04-17T10:25:22.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ebbe00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ea0007e20 0x7f9ea0008240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001060 tx=0x7f9ea401f3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d87744509a725f4c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9c0000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0005ec0 0x7fc9b00e0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9c0000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9b0033e10 0x7fc9b00de610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea4007d20 con 0x7f9ea0007e20 2024-04-17T10:25:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f6cbbe00640 1 Processor -- start 2024-04-17T10:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9c0000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0047bf0 0x7fc9b0036940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9c0000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b004fd50 con 0x7fc9b0005ec0 2024-04-17T10:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9c0000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b004f060 con 0x7fc9b0033e10 2024-04-17T10:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9c0000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b004f590 con 0x7fc9b0047bf0 2024-04-17T10:25:22.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ebaa00640 1 -- 172.21.15.67:0/256543837 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea4005ae0 con 0x7f9ea0007e20 2024-04-17T10:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa316000640 1 -- 172.21.15.67:0/368865522 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3040ab5f0 con 0x7fa318162260 2024-04-17T10:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa316000640 1 -- 172.21.15.67:0/368865522 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3040b0070 con 0x7fa318162260 2024-04-17T10:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa316000640 1 -- 172.21.15.67:0/368865522 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3040ab9b0 con 0x7fa318162260 2024-04-17T10:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0005ec0 0x7fc9b00e0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9b0033e10 0x7fc9b00de610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9b0033e10 0x7fc9b00de610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37168/0 (socket says 172.21.15.67:37168) 2024-04-17T10:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 -- 172.21.15.67:0/2358208108 learned_addr learned my addr 172.21.15.67:0/2358208108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa31e200640 1 -- 172.21.15.67:0/368865522 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 msgr2=0x7fa3180d0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa31e200640 1 --2- 172.21.15.67:0/368865522 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa3180d0ce0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa304009870 tx=0x7fa30401c520 comp rx=0 tx=0).stop 2024-04-17T10:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ec2a00640 1 -- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ea0007e20 msgr2=0x7f9ea0008240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ea0007e20 0x7f9ea0008240 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9ea4001060 tx=0x7f9ea401f3f0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f6cbbe00640 1 -- start start 2024-04-17T10:25:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa31e200640 1 -- 172.21.15.67:0/368865522 shutdown_connections 2024-04-17T10:25:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa31e200640 1 --2- 172.21.15.67:0/368865522 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa3180d0ce0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa31e200640 1 --2- 172.21.15.67:0/368865522 >> v2:172.21.15.67:3300/0 conn(0x7fa318161e80 0x7fa3180d07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa31e200640 1 -- 172.21.15.67:0/368865522 >> 172.21.15.67:0/368865522 conn(0x7fa318072590 msgr2=0x7fa3180729a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 -- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0047bf0 msgr2=0x7fc9b0036940 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T10:25:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0047bf0 0x7fc9b0036940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 -- 172.21.15.67:0/2358208108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0005ec0 msgr2=0x7fc9b00e0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0005ec0 0x7fc9b00e0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fc9bcc00640 1 -- 172.21.15.67:0/2358208108 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b00e08b0 con 0x7fc9b0033e10 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f78da600640 1 -- start start 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f78da600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f78d40b4970 0x7f78d4163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f78da600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d40719a0 con 0x7f78d4071310 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f78da600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d4071c80 con 0x7f78d40cef80 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f78da600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d4071760 con 0x7f78d40b4970 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ec2a00640 1 -- 172.21.15.67:0/256543837 shutdown_connections 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/256543837 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ea0007e20 0x7f9ea0008240 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7f9ec2a00640 1 -- 172.21.15.67:0/256543837 >> 172.21.15.67:0/256543837 conn(0x7f9ebc0b23b0 msgr2=0x7f9ebc0b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.955+0000 7fa31e200640 1 -- 172.21.15.67:0/368865522 shutdown_connections 2024-04-17T10:25:22.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fc9bcc00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9b0033e10 0x7fc9b00de610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400c370 tx=0x7fc9a400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fa31e200640 1 -- 172.21.15.67:0/368865522 wait complete. 2024-04-17T10:25:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f9ec2a00640 1 -- 172.21.15.67:0/256543837 shutdown_connections 2024-04-17T10:25:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f9ec2a00640 1 -- 172.21.15.67:0/256543837 wait complete. 2024-04-17T10:25:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f78d4071310 0x7f78d40b2770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54372/0 (socket says 172.21.15.67:54372) 2024-04-17T10:25:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d3400640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f78d40cef80 0x7f78d40b4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54776/0 (socket says 172.21.15.67:54776) 2024-04-17T10:25:22.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d3400640 1 -- 172.21.15.67:0/2805180095 learned_addr learned my addr 172.21.15.67:0/2805180095 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:22.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6f5ce00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f580b47b0 0x7f6f5815e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6f5ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5815e630 con 0x7f6f580b3bb0 2024-04-17T10:25:22.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6f5ce00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5815a0f0 con 0x7f6f58071a20 2024-04-17T10:25:22.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6f5ce00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f5815a3b0 con 0x7f6f580b47b0 2024-04-17T10:25:22.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6f56800640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6f58071a20 0x7f6f58153800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54786/0 (socket says 172.21.15.67:54786) 2024-04-17T10:25:22.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6f56800640 1 -- 172.21.15.67:0/541096988 learned_addr learned my addr 172.21.15.67:0/541096988 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:22.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6cbbe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40b47b0 0x7f6cb415e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fa31e200640 1 Processor -- start 2024-04-17T10:25:22.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6cbbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cb40b3bb0 0x7f6cb4071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175747064 0 0) 0x7f78d4071c80 con 0x7f78d40cef80 2024-04-17T10:25:22.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b8003530 con 0x7f78d40cef80 2024-04-17T10:25:22.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fc9b5600640 1 -- 172.21.15.67:0/2358208108 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4017070 con 0x7fc9b0033e10 2024-04-17T10:25:22.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fc9b5600640 1 -- 172.21.15.67:0/2358208108 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a4005cf0 con 0x7fc9b0033e10 2024-04-17T10:25:22.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fc9b5600640 1 -- 172.21.15.67:0/2358208108 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40072f0 con 0x7fc9b0033e10 2024-04-17T10:25:22.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fc9c0000640 1 -- 172.21.15.67:0/2358208108 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b0034660 con 0x7fc9b0033e10 2024-04-17T10:25:22.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6cbbe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6cb40cc1b0 0x7f6cb4071850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7fc9c0000640 1 -- 172.21.15.67:0/2358208108 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b0034c20 con 0x7fc9b0033e10 2024-04-17T10:25:22.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6cbbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb415e630 con 0x7f6cb40b3bb0 2024-04-17T10:25:22.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6cbbe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb415a0f0 con 0x7f6cb40cc1b0 2024-04-17T10:25:22.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6cbbe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb415a3b0 con 0x7f6cb40b47b0 2024-04-17T10:25:22.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6f55e00640 1 --2- 172.21.15.67:0/541096988 >> v2:172.21.15.86:3300/0 conn(0x7f6f580b47b0 0x7f6f5815e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1874684796 0 0) 0x7f78b8003530 con 0x7f78d40cef80 2024-04-17T10:25:22.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 >> v2:172.21.15.67:3300/0 conn(0x7f78d40b4970 msgr2=0x7f78d4163060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2805180095 >> v2:172.21.15.67:3300/0 conn(0x7f78d40b4970 0x7f78d4163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 --2- 172.21.15.67:0/2805180095 >> v2:172.21.15.67:3300/0 conn(0x7f78d40b4970 0x7f78d4163060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 >> v1:172.21.15.67:6789/0 conn(0x7f78d4071310 legacy=0x7f78d40b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d4162580 con 0x7f78d40cef80 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f78bc001680 con 0x7f78d40cef80 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 >> v1:172.21.15.86:6789/0 conn(0x7f78d40cef80 legacy=0x7f78d40b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f78d2a00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8003a10 0x7f78b8003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f6cb8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cb40b3bb0 0x7f6cb4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cb40b3bb0 0x7f6cb4071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58112/0 (socket says 172.21.15.67:58112) 2024-04-17T10:25:22.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb8a00640 1 -- 172.21.15.67:0/3987820057 learned_addr learned my addr 172.21.15.67:0/3987820057 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.959+0000 7f9ec2a00640 1 Processor -- start 2024-04-17T10:25:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78d2a00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78b8004810 0x7f78b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78d2a00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b80053d0 0x7f78b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336765834 0 0) 0x7f6f5815a0f0 con 0x7f6f58071a20 2024-04-17T10:25:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f3c003530 con 0x7f6f58071a20 2024-04-17T10:25:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102278042 0 0) 0x7f6f500014a0 con 0x7f6f580b3bb0 2024-04-17T10:25:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f5815a0f0 con 0x7f6f580b3bb0 2024-04-17T10:25:22.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc98c0052a0 con 0x7fc9b0033e10 2024-04-17T10:25:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb9e00640 1 --2- 172.21.15.67:0/3987820057 >> v2:172.21.15.86:3300/0 conn(0x7f6cb40cc1b0 0x7f6cb4071850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb8a00640 1 -- 172.21.15.67:0/3987820057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40b47b0 msgr2=0x7f6cb415e0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-17T10:25:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78d3e00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b80053d0 0x7f78b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7fa31e200640 1 -- start start 2024-04-17T10:25:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d4071760 con 0x7f78b80053d0 2024-04-17T10:25:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b800c240 con 0x7f78b8003a10 2024-04-17T10:25:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b800c440 con 0x7f78b8004810 2024-04-17T10:25:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2805180095 >> v2:172.21.15.67:3300/0 conn(0x7f78d40b4970 0x7f78d4163060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78d3400640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8003a10 0x7f78b8003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f9ec2a00640 1 -- start start 2024-04-17T10:25:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78b8004810 0x7f78b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1782856399 0 0) 0x7f6f3c003530 con 0x7f6f58071a20 2024-04-17T10:25:22.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 >> v2:172.21.15.86:3300/0 conn(0x7f6f580b47b0 msgr2=0x7f6f5815e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb8a00640 1 --2- 172.21.15.67:0/3987820057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40b47b0 0x7f6cb415e0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb8a00640 1 -- 172.21.15.67:0/3987820057 >> v2:172.21.15.86:3300/0 conn(0x7f6cb40cc1b0 msgr2=0x7f6cb4071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb8a00640 1 --2- 172.21.15.67:0/3987820057 >> v2:172.21.15.86:3300/0 conn(0x7f6cb40cc1b0 0x7f6cb4071850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6cb8a00640 1 -- 172.21.15.67:0/3987820057 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb415a690 con 0x7f6cb40b3bb0 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f9ec2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0007e20 0x7f9ebc0b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f9ec2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ebc13fc10 0x7f9ebc1f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f9ec2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ebc140070 0x7f9ebc1f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f9ec2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc0cfb50 con 0x7f9ea0007e20 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f9ec2a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc0cf9d0 con 0x7f9ebc140070 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f9ec2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ebc0cfcd0 con 0x7f9ebc13fc10 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ebc13fc10 0x7f9ebc1f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ebc13fc10 0x7f9ebc1f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49466/0 (socket says 172.21.15.67:49466) 2024-04-17T10:25:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cb8a00640 1 --2- 172.21.15.67:0/3987820057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b3bb0 0x7f6cb4071310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6cb406e750 tx=0x7f6ca001e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27ce22ddfd95d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 -- 172.21.15.67:0/75292019 learned_addr learned my addr 172.21.15.67:0/75292019 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ec0a00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ebc140070 0x7f9ebc1f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 -- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ebc140070 msgr2=0x7f9ebc1f6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ebc140070 0x7f9ebc1f6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 -- 172.21.15.67:0/75292019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0007e20 msgr2=0x7f9ebc0b5ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T10:25:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7fa31e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa318161e80 0x7fa3180d2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa31e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa318156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa31e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa318163270 0x7fa3180cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0007e20 0x7f9ebc0b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 -- 172.21.15.67:0/75292019 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc15d360 con 0x7f9ebc13fc10 2024-04-17T10:25:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa317400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa318161e80 0x7fa3180d2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa317400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa318161e80 0x7fa3180d2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37222/0 (socket says 172.21.15.67:37222) 2024-04-17T10:25:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fc9b5600640 1 -- 172.21.15.67:0/2358208108 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc9a401a070 con 0x7fc9b0033e10 2024-04-17T10:25:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ebb400640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ebc13fc10 0x7f9ebc1f66a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9eac010910 tx=0x7f9eac010dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9eb3e00640 1 -- 172.21.15.67:0/75292019 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac01a020 con 0x7f9ebc13fc10 2024-04-17T10:25:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9eb3e00640 1 -- 172.21.15.67:0/75292019 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eac00ea80 con 0x7f9ebc13fc10 2024-04-17T10:25:22.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78cbe00640 1 -- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8003a10 msgr2=0x7f78b8003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8003a10 0x7f78b8003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78cbe00640 1 -- 172.21.15.67:0/2805180095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b80053d0 msgr2=0x7f78b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b80053d0 0x7f78b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78cbe00640 1 -- 172.21.15.67:0/2805180095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b8003530 con 0x7f78b8004810 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78d3e00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78b80053d0 0x7f78b800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78d3e00640 1 -- 172.21.15.67:0/2805180095 reap_dead start 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 --2- 172.21.15.67:0/541096988 >> v2:172.21.15.86:3300/0 conn(0x7f6f580b47b0 0x7f6f5815e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 >> v1:172.21.15.67:6789/0 conn(0x7f6f580b3bb0 legacy=0x7f6f58071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5815a690 con 0x7f6f58071a20 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f6f48001680 con 0x7f6f58071a20 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 >> v1:172.21.15.86:6789/0 conn(0x7f6f58071a20 legacy=0x7f6f58153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c004d30 0x7f6f3c0037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f3c003e20 0x7f6f3c005130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f3c0043d0 0x7f6f3c005740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5815a3b0 con 0x7f6f3c004d30 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c006020 con 0x7f6f3c003e20 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c006220 con 0x7f6f3c0043d0 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f55400640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f3c003e20 0x7f6f3c005130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f55400640 1 -- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f3c0043d0 msgr2=0x7f6f3c005740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f55400640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f3c0043d0 0x7f6f3c005740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f55400640 1 -- 172.21.15.67:0/541096988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c004d30 msgr2=0x7f6f3c0037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f55400640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c004d30 0x7f6f3c0037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f55400640 1 -- 172.21.15.67:0/541096988 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f5815a690 con 0x7f6f3c003e20 2024-04-17T10:25:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.963+0000 7f6f56800640 1 -- 172.21.15.67:0/541096988 reap_dead start 2024-04-17T10:25:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fecbd600640 1 Processor -- start 2024-04-17T10:25:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fecbd600640 1 -- start start 2024-04-17T10:25:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cb3e00640 1 -- 172.21.15.67:0/3987820057 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca00ae070 con 0x7f6cb40b3bb0 2024-04-17T10:25:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cb3e00640 1 -- 172.21.15.67:0/3987820057 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca0008be0 con 0x7f6cb40b3bb0 2024-04-17T10:25:22.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cb3e00640 1 -- 172.21.15.67:0/3987820057 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca000a0d0 con 0x7f6cb40b3bb0 2024-04-17T10:25:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9eb3e00640 1 -- 172.21.15.67:0/75292019 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eac00ee70 con 0x7f9ebc13fc10 2024-04-17T10:25:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ebc0b6220 con 0x7f9ebc13fc10 2024-04-17T10:25:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ebc140560 con 0x7f9ebc13fc10 2024-04-17T10:25:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fc9b5600640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998067060 0x7fc998069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f78d3400640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8003a10 0x7f78b8003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:22.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fc9b5600640 1 -- 172.21.15.67:0/2358208108 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc9a4084b30 con 0x7fc9b0033e10 2024-04-17T10:25:22.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 -- 172.21.15.67:0/3987820057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b3bb0 msgr2=0x7f6cb4071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/3987820057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b3bb0 0x7f6cb4071310 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6cb406e750 tx=0x7f6ca001e660 comp rx=0 tx=0).stop 2024-04-17T10:25:22.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fc9bd600640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998067060 0x7fc998069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 -- 172.21.15.67:0/3987820057 shutdown_connections 2024-04-17T10:25:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/3987820057 >> v2:172.21.15.86:3300/0 conn(0x7f6cb40cc1b0 0x7f6cb4071850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/3987820057 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b3bb0 0x7f6cb4071310 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/3987820057 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40b47b0 0x7f6cb415e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 -- 172.21.15.67:0/3987820057 >> 172.21.15.67:0/3987820057 conn(0x7f6cb406ac90 msgr2=0x7f6cb406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa316a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa318156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa317e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa318163270 0x7fa3180cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa316a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa318156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49472/0 (socket says 172.21.15.67:49472) 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa317e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa318163270 0x7fa3180cb040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58130/0 (socket says 172.21.15.67:58130) 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7fa317400640 1 -- 172.21.15.67:0/3859030732 learned_addr learned my addr 172.21.15.67:0/3859030732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3181463b0 con 0x7fa318163270 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318146230 con 0x7fa318161e80 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa318146530 con 0x7fa318162260 2024-04-17T10:25:22.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f6f55400640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f3c003e20 0x7f6f3c005130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f580b3fd0 tx=0x7f6f5001c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8b1bc0d169e7185 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecbd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb8163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.967+0000 7f6cbbe00640 1 -- 172.21.15.67:0/3987820057 shutdown_connections 2024-04-17T10:25:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f6cbbe00640 1 -- 172.21.15.67:0/3987820057 wait complete. 2024-04-17T10:25:22.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9b4a00640 1 Processor -- start 2024-04-17T10:25:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecbd600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fecb80b4970 0x7fecb8166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9b4a00640 1 -- start start 2024-04-17T10:25:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa317400640 1 -- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 msgr2=0x7fa318156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa317400640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa318156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa317400640 1 -- 172.21.15.67:0/3859030732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa318163270 msgr2=0x7fa3180cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa317400640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa318163270 0x7fa3180cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa317400640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3180cb940 con 0x7fa318161e80 2024-04-17T10:25:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fc9b5600640 1 -- 172.21.15.67:0/2358208108 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc9a4021020 con 0x7fc9b0033e10 2024-04-17T10:25:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e940052a0 con 0x7f9ebc13fc10 2024-04-17T10:25:22.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50001d60 con 0x7f6f3c003e20 2024-04-17T10:25:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f50003b80 con 0x7f6f3c003e20 2024-04-17T10:25:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f6f54a00640 1 -- 172.21.15.67:0/541096988 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500050d0 con 0x7f6f3c003e20 2024-04-17T10:25:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecbd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb80719a0 con 0x7fecb8071310 2024-04-17T10:25:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecbd600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb8071c80 con 0x7fecb80b4970 2024-04-17T10:25:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecbd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb8162580 con 0x7fecb80cef80 2024-04-17T10:25:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecb6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb8163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa317400640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa318161e80 0x7fa3180d2dc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa30800c7f0 tx=0x7fa30800ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fc9bd600640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998067060 0x7fc998069520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc9b00079b0 tx=0x7fc9b8125070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f9eb3e00640 1 -- 172.21.15.67:0/75292019 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9eac01d020 con 0x7f9ebc13fc10 2024-04-17T10:25:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecb6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb8163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49484/0 (socket says 172.21.15.67:49484) 2024-04-17T10:25:22.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecb6a00640 1 -- 172.21.15.67:0/357281787 learned_addr learned my addr 172.21.15.67:0/357281787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa30f400640 1 -- 172.21.15.67:0/3859030732 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308015070 con 0x7fa318161e80 2024-04-17T10:25:22.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa30f400640 1 -- 172.21.15.67:0/3859030732 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa308004240 con 0x7fa318161e80 2024-04-17T10:25:22.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa30f400640 1 -- 172.21.15.67:0/3859030732 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308003620 con 0x7fa318161e80 2024-04-17T10:25:22.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f6cbbe00640 1 Processor -- start 2024-04-17T10:25:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3180cbc20 con 0x7fa318161e80 2024-04-17T10:25:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3180cc1e0 con 0x7fa318161e80 2024-04-17T10:25:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecb6a00640 1 -- 172.21.15.67:0/357281787 >> v2:172.21.15.86:3300/0 conn(0x7fecb80b4970 msgr2=0x7fecb8166430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-17T10:25:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecb6a00640 1 --2- 172.21.15.67:0/357281787 >> v2:172.21.15.86:3300/0 conn(0x7fecb80b4970 0x7fecb8166430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecb6a00640 1 -- 172.21.15.67:0/357281787 >> v1:172.21.15.67:6789/0 conn(0x7fecb8071310 legacy=0x7fecb80b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:22.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fecb6a00640 1 -- 172.21.15.67:0/357281787 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb8162860 con 0x7fecb80cef80 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f9eb3e00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e98067030 0x7f9e980694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7f9eb3e00640 1 -- 172.21.15.67:0/75292019 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9eac085d00 con 0x7f9ebc13fc10 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9b4a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe9b0071310 0x7fe9b00717a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9b4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b015c9b0 con 0x7fe9b015da50 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9b4a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b015cc90 con 0x7fe9b0145210 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9b4a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b00a8650 con 0x7fe9b0071310 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9ae800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9b015da50 0x7fe9b015de80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54396/0 (socket says 172.21.15.67:54396) 2024-04-17T10:25:22.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.971+0000 7fe9ae800640 1 -- 172.21.15.67:0/3859020893 learned_addr learned my addr 172.21.15.67:0/3859020893 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:22.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31808afe0 con 0x7fa318161e80 2024-04-17T10:25:22.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 -- start start 2024-04-17T10:25:22.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6f5ce00640 1 -- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f3c003e20 msgr2=0x7f6f3c005130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f3c003e20 0x7f6f3c005130 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f580b3fd0 tx=0x7f6f5001c820 comp rx=0 tx=0).stop 2024-04-17T10:25:22.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f9ebbe00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e98067030 0x7f9e980694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ade00640 1 --2- 172.21.15.67:0/3859020893 >> v2:172.21.15.86:3300/0 conn(0x7fe9b0071310 0x7fe9b00717a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102945517 0 0) 0x7fe9b015c9b0 con 0x7fe9b015da50 2024-04-17T10:25:22.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe994003530 con 0x7fe9b015da50 2024-04-17T10:25:22.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6f5ce00640 1 -- 172.21.15.67:0/541096988 shutdown_connections 2024-04-17T10:25:22.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/541096988 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f3c003e20 0x7f6f3c005130 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:22.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6f5ce00640 1 -- 172.21.15.67:0/541096988 >> 172.21.15.67:0/541096988 conn(0x7f6f5806ac90 msgr2=0x7f6f5806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809296256 0 0) 0x7fe9b015cc90 con 0x7fe9b0145210 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b015c9b0 con 0x7fe9b0145210 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb40b3bb0 0x7f6cb4154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b47b0 0x7f6cb414a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40cc1b0 0x7f6cb40b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb4154190 con 0x7f6cb40b47b0 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb4154010 con 0x7f6cb40b3bb0 2024-04-17T10:25:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb4154310 con 0x7f6cb40cc1b0 2024-04-17T10:25:22.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1931349957 0 0) 0x7fe994003530 con 0x7fe9b015da50 2024-04-17T10:25:22.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b47b0 0x7f6cb414a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:22.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b47b0 0x7f6cb414a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58144/0 (socket says 172.21.15.67:58144) 2024-04-17T10:25:22.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 -- 172.21.15.67:0/4204852245 learned_addr learned my addr 172.21.15.67:0/4204852245 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:22.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ade00640 1 -- 172.21.15.67:0/3859020893 >> v1:172.21.15.86:6789/0 conn(0x7fe9b0145210 legacy=0x7fe9b015bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ade00640 1 -- 172.21.15.67:0/3859020893 >> v1:172.21.15.67:6789/0 conn(0x7fe9b015da50 legacy=0x7fe9b015de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:22.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fe9ade00640 1 -- 172.21.15.67:0/3859020893 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b015cf70 con 0x7fe9b0071310 2024-04-17T10:25:23.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb9400640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb40b3bb0 0x7f6cb4154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7fa30f400640 1 -- 172.21.15.67:0/3859030732 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa3080039d0 con 0x7fa318161e80 2024-04-17T10:25:23.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 -- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40cc1b0 msgr2=0x7f6cb40b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40cc1b0 0x7f6cb40b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 -- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb40b3bb0 msgr2=0x7f6cb4154760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb40b3bb0 0x7f6cb4154760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 -- 172.21.15.67:0/4204852245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb4153340 con 0x7f6cb40b47b0 2024-04-17T10:25:23.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6f5ce00640 1 -- 172.21.15.67:0/541096988 shutdown_connections 2024-04-17T10:25:23.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb8a00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b47b0 0x7f6cb414a5a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6ca001ef00 tx=0x7f6ca001b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6f5ce00640 1 -- 172.21.15.67:0/541096988 wait complete. 2024-04-17T10:25:23.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78b8004810 0x7f78b8004c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f78d40b4d80 tx=0x7f78c401c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99cfdaa7d2fb95e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb1600640 1 -- 172.21.15.67:0/4204852245 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca00ae040 con 0x7f6cb40b47b0 2024-04-17T10:25:23.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb1600640 1 -- 172.21.15.67:0/4204852245 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca00bc070 con 0x7f6cb40b47b0 2024-04-17T10:25:23.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cb1600640 1 -- 172.21.15.67:0/4204852245 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca000a8e0 con 0x7f6cb40b47b0 2024-04-17T10:25:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb4151600 con 0x7f6cb40b47b0 2024-04-17T10:25:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fa30f400640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa2f8066fb0 0x7fa2f8069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fa30f400640 1 -- 172.21.15.67:0/3859030732 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa308085240 con 0x7fa318161e80 2024-04-17T10:25:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fa316a00640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa2f8066fb0 0x7fa2f8069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f9ebbe00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e98067030 0x7f9e980694f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40af000 tx=0x7f9ea4000ff0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400a670 con 0x7f78b8004810 2024-04-17T10:25:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c40a9670 con 0x7f78b8004810 2024-04-17T10:25:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78d2a00640 1 -- 172.21.15.67:0/2805180095 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c400a670 con 0x7f78b8004810 2024-04-17T10:25:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecb6a00640 1 --2- 172.21.15.67:0/357281787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb8163060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feca8009870 tx=0x7feca801c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30e0df133327e9ed server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecb5600640 1 -- 172.21.15.67:0/357281787 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80b0070 con 0x7fecb80cef80 2024-04-17T10:25:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecb5600640 1 -- 172.21.15.67:0/357281787 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca80ab600 con 0x7fecb80cef80 2024-04-17T10:25:23.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecb5600640 1 -- 172.21.15.67:0/357281787 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80ab9d0 con 0x7fecb80cef80 2024-04-17T10:25:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.975+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb41518a0 con 0x7f6cb40b47b0 2024-04-17T10:25:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78da600640 1 -- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78b8004810 msgr2=0x7f78b8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fa316a00640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa2f8066fb0 0x7fa2f8069470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa3040af000 tx=0x7fa3040a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78da600640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78b8004810 0x7f78b8004c90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f78d40b4d80 tx=0x7f78c401c5d0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78da600640 1 -- 172.21.15.67:0/2805180095 shutdown_connections 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78da600640 1 --2- 172.21.15.67:0/2805180095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78b8004810 0x7f78b8004c90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78da600640 1 -- 172.21.15.67:0/2805180095 >> 172.21.15.67:0/2805180095 conn(0x7f78d406ac90 msgr2=0x7f78d406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78da600640 1 -- 172.21.15.67:0/2805180095 shutdown_connections 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f78da600640 1 -- 172.21.15.67:0/2805180095 wait complete. 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 -- 172.21.15.67:0/357281787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 msgr2=0x7fecb8163060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 --2- 172.21.15.67:0/357281787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb8163060 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feca8009870 tx=0x7feca801c540 comp rx=0 tx=0).stop 2024-04-17T10:25:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f6f5ce00640 1 Processor -- start 2024-04-17T10:25:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f6f5ce00640 1 -- start start 2024-04-17T10:25:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 -- 172.21.15.67:0/357281787 shutdown_connections 2024-04-17T10:25:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 --2- 172.21.15.67:0/357281787 >> v2:172.21.15.86:3300/0 conn(0x7fecb80b4970 0x7fecb8166430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 --2- 172.21.15.67:0/357281787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb8163060 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 -- 172.21.15.67:0/357281787 >> 172.21.15.67:0/357281787 conn(0x7fecb806ac90 msgr2=0x7fecb806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f9eb3e00640 1 -- 172.21.15.67:0/75292019 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9eac024020 con 0x7f9ebc13fc10 2024-04-17T10:25:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 -- 172.21.15.67:0/357281787 shutdown_connections 2024-04-17T10:25:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7fecbd600640 1 -- 172.21.15.67:0/357281787 wait complete. 2024-04-17T10:25:23.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.979+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb4072d50 con 0x7f6cb40b47b0 2024-04-17T10:25:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 Processor -- start 2024-04-17T10:25:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 Processor -- start 2024-04-17T10:25:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 -- start start 2024-04-17T10:25:23.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 -- start start 2024-04-17T10:25:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f5ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c003e20 0x7f6f581525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f5ce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f581f30f0 0x7f6f581f3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f5ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f581e7110 0x7f6f581e7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f5ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5815e270 con 0x7f6f3c003e20 2024-04-17T10:25:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f5ce00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5815e0f0 con 0x7f6f581e7110 2024-04-17T10:25:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f5ce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5815e3f0 con 0x7f6f581f30f0 2024-04-17T10:25:23.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f55400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f581f30f0 0x7f6f581f3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6cb1600640 1 -- 172.21.15.67:0/4204852245 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6ca00b1070 con 0x7f6cb40b47b0 2024-04-17T10:25:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f56800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f581e7110 0x7f6f581e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f55400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f581f30f0 0x7f6f581f3550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49492/0 (socket says 172.21.15.67:49492) 2024-04-17T10:25:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f55400640 1 -- 172.21.15.67:0/2482607349 learned_addr learned my addr 172.21.15.67:0/2482607349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fe9ade00640 1 --2- 172.21.15.67:0/3859020893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0071310 0x7fe9b00717a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe998009740 tx=0x7fe99801eaf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4b84f0fadbf2341 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f55e00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c003e20 0x7f6f581525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9980b5070 con 0x7fe9b0071310 2024-04-17T10:25:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9980a85b0 con 0x7fe9b0071310 2024-04-17T10:25:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fe9ad400640 1 -- 172.21.15.67:0/3859020893 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8001a00 con 0x7fe9b0071310 2024-04-17T10:25:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8071310 0x7fecb8154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecb80b4970 0x7fecb8152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb815e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b3390 con 0x7fecb80b4970 2024-04-17T10:25:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b3210 con 0x7fecb8071310 2024-04-17T10:25:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecbd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b3510 con 0x7fecb80cef80 2024-04-17T10:25:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6cb1600640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c94066f60 0x7f6c94069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8004810 0x7f78d414a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d40b0660 0x7f78d40afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40b1bc0 0x7f78d4159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d40cf360 con 0x7f78d40b1bc0 2024-04-17T10:25:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d4162b40 con 0x7f78b8004810 2024-04-17T10:25:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78da600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d40cf4e0 con 0x7f78d40b0660 2024-04-17T10:25:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f56800640 1 -- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f581f30f0 msgr2=0x7f6f581f3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f56800640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f581f30f0 0x7f6f581f3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f56800640 1 -- 172.21.15.67:0/2482607349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c003e20 msgr2=0x7f6f581525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6cb1600640 1 -- 172.21.15.67:0/4204852245 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6ca011ff00 con 0x7f6cb40b47b0 2024-04-17T10:25:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f56800640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c003e20 0x7f6f581525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f56800640 1 -- 172.21.15.67:0/2482607349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f58156980 con 0x7f6f581e7110 2024-04-17T10:25:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f55e00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c003e20 0x7f6f581525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecb7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb815e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecb7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb815e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49494/0 (socket says 172.21.15.67:49494) 2024-04-17T10:25:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecb7400640 1 -- 172.21.15.67:0/2152907384 learned_addr learned my addr 172.21.15.67:0/2152907384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecb6a00640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8071310 0x7fecb8154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fecb6000640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecb80b4970 0x7fecb8152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7fa30f400640 1 -- 172.21.15.67:0/3859030732 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa3080559a0 con 0x7fa318161e80 2024-04-17T10:25:23.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f6f56800640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f581e7110 0x7f6f581e7550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6f580cc0d0 tx=0x7f6f4800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40b1bc0 0x7f78d4159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40b1bc0 0x7f78d4159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58178/0 (socket says 172.21.15.67:58178) 2024-04-17T10:25:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.983+0000 7f78d3e00640 1 -- 172.21.15.67:0/2643021838 learned_addr learned my addr 172.21.15.67:0/2643021838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78d3400640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8004810 0x7f78d414a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecb6a00640 1 -- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 msgr2=0x7fecb815e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecb6a00640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb815e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecb6a00640 1 -- 172.21.15.67:0/2152907384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecb80b4970 msgr2=0x7fecb8152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecb6a00640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecb80b4970 0x7fecb8152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecb6a00640 1 -- 172.21.15.67:0/2152907384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb81527b0 con 0x7fecb8071310 2024-04-17T10:25:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d40b0660 0x7f78d40afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cbe00640 1 -- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8004810 msgr2=0x7f78d414a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8004810 0x7f78d414a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cbe00640 1 -- 172.21.15.67:0/2643021838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40b1bc0 msgr2=0x7f78d4159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40b1bc0 0x7f78d4159790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cbe00640 1 -- 172.21.15.67:0/2643021838 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d415e1d0 con 0x7f78d40b0660 2024-04-17T10:25:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78d3e00640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40b1bc0 0x7f78d4159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6f4e000640 1 -- 172.21.15.67:0/2482607349 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48007440 con 0x7f6f581e7110 2024-04-17T10:25:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6f4e000640 1 -- 172.21.15.67:0/2482607349 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f480075e0 con 0x7f6f581e7110 2024-04-17T10:25:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6f4e000640 1 -- 172.21.15.67:0/2482607349 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f48004a10 con 0x7f6f581e7110 2024-04-17T10:25:23.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecb6a00640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8071310 0x7fecb8154920 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feca80ab000 tx=0x7feca80192e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6cb9400640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c94066f60 0x7f6c94069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fe9b4a00640 1 -- 172.21.15.67:0/3859020893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0071310 msgr2=0x7fe9b00717a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/3859020893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0071310 0x7fe9b00717a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe998009740 tx=0x7fe99801eaf0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fe9b4a00640 1 -- 172.21.15.67:0/3859020893 shutdown_connections 2024-04-17T10:25:23.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/3859020893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b0071310 0x7fe9b00717a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fe9b4a00640 1 -- 172.21.15.67:0/3859020893 >> 172.21.15.67:0/3859020893 conn(0x7fe9b006ac90 msgr2=0x7fe9b006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f58156c60 con 0x7f6f581e7110 2024-04-17T10:25:23.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f58152ae0 con 0x7f6f581e7110 2024-04-17T10:25:23.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6cb9400640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c94066f60 0x7f6c94069420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009f60 tx=0x7f6ca80094f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cbe00640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d40b0660 0x7f78d40afbc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f78c40aa000 tx=0x7f78b8005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecaea00640 1 -- 172.21.15.67:0/2152907384 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80b9020 con 0x7fecb8071310 2024-04-17T10:25:23.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecaea00640 1 -- 172.21.15.67:0/2152907384 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca80b1040 con 0x7fecb8071310 2024-04-17T10:25:23.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecaea00640 1 -- 172.21.15.67:0/2152907384 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80b0050 con 0x7fecb8071310 2024-04-17T10:25:23.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cb400640 1 -- 172.21.15.67:0/2643021838 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c40c1070 con 0x7f78d40b0660 2024-04-17T10:25:23.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb815f670 con 0x7fecb8071310 2024-04-17T10:25:23.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cb400640 1 -- 172.21.15.67:0/2643021838 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78c40bc420 con 0x7f78d40b0660 2024-04-17T10:25:23.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78cb400640 1 -- 172.21.15.67:0/2643021838 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78c40c1070 con 0x7f78d40b0660 2024-04-17T10:25:23.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb8161d20 con 0x7fecb8071310 2024-04-17T10:25:23.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d40b0100 con 0x7f78d40b0660 2024-04-17T10:25:23.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d40b0e00 con 0x7f78d40b0660 2024-04-17T10:25:23.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.987+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f58072d50 con 0x7f6f581e7110 2024-04-17T10:25:23.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb8072d50 con 0x7fecb8071310 2024-04-17T10:25:23.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7f6f4e000640 1 -- 172.21.15.67:0/2482607349 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6f48004bb0 con 0x7f6f581e7110 2024-04-17T10:25:23.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7fe9b4a00640 1 -- 172.21.15.67:0/3859020893 shutdown_connections 2024-04-17T10:25:23.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7fe9b4a00640 1 -- 172.21.15.67:0/3859020893 wait complete. 2024-04-17T10:25:23.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7f6f4e000640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f38067030 0x7f6f380694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78ac0052a0 con 0x7f78d40b0660 2024-04-17T10:25:23.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7f6cb1600640 1 -- 172.21.15.67:0/4204852245 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6ca0018250 con 0x7f6cb40b47b0 2024-04-17T10:25:23.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7f6f4e000640 1 -- 172.21.15.67:0/2482607349 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f6f480847f0 con 0x7f6f581e7110 2024-04-17T10:25:23.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7fecaea00640 1 -- 172.21.15.67:0/2152907384 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7feca80bc020 con 0x7fecb8071310 2024-04-17T10:25:23.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7fecaea00640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec98066fe0 0x7fec980694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7fecaea00640 1 -- 172.21.15.67:0/2152907384 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7feca811eb70 con 0x7fecb8071310 2024-04-17T10:25:23.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.991+0000 7f78cb400640 1 -- 172.21.15.67:0/2643021838 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f78c40a94f0 con 0x7f78d40b0660 2024-04-17T10:25:23.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 Processor -- start 2024-04-17T10:25:23.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7f78cb400640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78b0067160 0x7f78b0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 -- start start 2024-04-17T10:25:23.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7f78cb400640 1 -- 172.21.15.67:0/2643021838 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f78c411f800 con 0x7f78d40b0660 2024-04-17T10:25:23.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7f78d3400640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78b0067160 0x7f78b0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7f6f55e00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f38067030 0x7f6f380694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7f6f55e00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f38067030 0x7f6f380694f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6f58050870 tx=0x7f6f44009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7f6f4e000640 1 -- 172.21.15.67:0/2482607349 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6f48054e50 con 0x7f6f581e7110 2024-04-17T10:25:23.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fecb6000640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec98066fe0 0x7fec980694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7f78d3400640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78b0067160 0x7f78b0069620 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78bc001ec0 tx=0x7f78bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0071310 0x7fe9b0140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9b0145210 0x7fe9b00c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b015da50 0x7fe9b015b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b006e820 con 0x7fe9b0145210 2024-04-17T10:25:23.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b015d5c0 con 0x7fe9b015da50 2024-04-17T10:25:23.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9b4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b006e9a0 con 0x7fe9b0071310 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.995+0000 7fe9ae800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b015da50 0x7fe9b015b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b015da50 0x7fe9b015b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37306/0 (socket says 172.21.15.67:37306) 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 -- 172.21.15.67:0/1934821144 learned_addr learned my addr 172.21.15.67:0/1934821144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ade00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9b0145210 0x7fe9b00c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 -- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0071310 msgr2=0x7fe9b0140bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0071310 0x7fe9b0140bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 -- 172.21.15.67:0/1934821144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9b0145210 msgr2=0x7fe9b00c6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9b0145210 0x7fe9b00c6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 -- 172.21.15.67:0/1934821144 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b00c8800 con 0x7fe9b015da50 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9ae800640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b015da50 0x7fe9b015b270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800c810 tx=0x7fe9a800ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fecaea00640 1 -- 172.21.15.67:0/2152907384 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7feca8016130 con 0x7fecb8071310 2024-04-17T10:25:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9a6a00640 1 -- 172.21.15.67:0/1934821144 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8018070 con 0x7fe9b015da50 2024-04-17T10:25:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9a6a00640 1 -- 172.21.15.67:0/1934821144 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a800a640 con 0x7fe9b015da50 2024-04-17T10:25:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9a6a00640 1 -- 172.21.15.67:0/1934821144 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a800aa30 con 0x7fe9b015da50 2024-04-17T10:25:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b00c7760 con 0x7fe9b015da50 2024-04-17T10:25:23.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:22.999+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b00c7d20 con 0x7fe9b015da50 2024-04-17T10:25:23.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.003+0000 7f78cb400640 1 -- 172.21.15.67:0/2643021838 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f78c40ad020 con 0x7f78d40b0660 2024-04-17T10:25:23.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.003+0000 7fecb6000640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec98066fe0 0x7fec980694a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fecb815dff0 tx=0x7feca4009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.003+0000 7fe9a6a00640 1 -- 172.21.15.67:0/1934821144 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe9a801b070 con 0x7fe9b015da50 2024-04-17T10:25:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.003+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b0072d50 con 0x7fe9b015da50 2024-04-17T10:25:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.007+0000 7fe9a6a00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe98c066fb0 0x7fe98c069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.007+0000 7fe9a6a00640 1 -- 172.21.15.67:0/1934821144 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fe9a8084aa0 con 0x7fe9b015da50 2024-04-17T10:25:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.007+0000 7fe9a5e00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe98c066fb0 0x7fe98c069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.007+0000 7fe9a5e00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe98c066fb0 0x7fe98c069470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0001ec0 tx=0x7fe9a00010e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.011+0000 7fe9a6a00640 1 -- 172.21.15.67:0/1934821144 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe9a8055100 con 0x7fe9b015da50 2024-04-17T10:25:23.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.319+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f9e94005590 con 0x7f9ebc13fc10 2024-04-17T10:25:23.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.319+0000 7f9eb3e00640 1 -- 172.21.15.67:0/75292019 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9eac0569b0 con 0x7f9ebc13fc10 2024-04-17T10:25:23.316 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-04-17T10:25:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e98067030 msgr2=0x7f9e980694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e98067030 0x7f9e980694f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f9ea40af000 tx=0x7f9ea4000ff0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ebc13fc10 msgr2=0x7f9ebc1f66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ebc13fc10 0x7f9ebc1f66a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9eac010910 tx=0x7f9eac010dd0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 shutdown_connections 2024-04-17T10:25:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e98067030 0x7f9e980694f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ebc140070 0x7f9ebc1f6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ebc13fc10 0x7f9ebc1f66a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 --2- 172.21.15.67:0/75292019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ea0007e20 0x7f9ebc0b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 >> 172.21.15.67:0/75292019 conn(0x7f9ebc0b23b0 msgr2=0x7f9ebc1427a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 shutdown_connections 2024-04-17T10:25:23.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.323+0000 7f9ec2a00640 1 -- 172.21.15.67:0/75292019 wait complete. 2024-04-17T10:25:23.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.327+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc98c005860 con 0x7fc9b0033e10 2024-04-17T10:25:23.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.327+0000 7fc9b5600640 1 -- 172.21.15.67:0/2358208108 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc9a4055870 con 0x7fc9b0033e10 2024-04-17T10:25:23.326 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-17T10:25:23.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998067060 msgr2=0x7fc998069520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998067060 0x7fc998069520 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc9b00079b0 tx=0x7fc9b8125070 comp rx=0 tx=0).stop 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9b0033e10 msgr2=0x7fc9b00de610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9b0033e10 0x7fc9b00de610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc9a400c370 tx=0x7fc9a400c830 comp rx=0 tx=0).stop 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 shutdown_connections 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998067060 0x7fc998069520 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc9b00079b0 tx=0x7fc9b8125070 comp rx=0 tx=0).stop 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9b0047bf0 0x7fc9b0036940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9b0033e10 0x7fc9b00de610 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 --2- 172.21.15.67:0/2358208108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9b0005ec0 0x7fc9b00e0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 >> 172.21.15.67:0/2358208108 conn(0x7fc9b0009f90 msgr2=0x7fc9b00056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 shutdown_connections 2024-04-17T10:25:23.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fc9aaa00640 1 -- 172.21.15.67:0/2358208108 wait complete. 2024-04-17T10:25:23.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa3180b4680 con 0x7fa318161e80 2024-04-17T10:25:23.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.335+0000 7fa30f400640 1 -- 172.21.15.67:0/3859030732 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa308059a40 con 0x7fa318161e80 2024-04-17T10:25:23.336 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-17T10:25:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa2f8066fb0 msgr2=0x7fa2f8069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa2f8066fb0 0x7fa2f8069470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa3040af000 tx=0x7fa3040a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa318161e80 msgr2=0x7fa3180d2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa318161e80 0x7fa3180d2dc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa30800c7f0 tx=0x7fa30800ccb0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 shutdown_connections 2024-04-17T10:25:23.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa2f8066fb0 0x7fa2f8069470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa318163270 0x7fa3180cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa318162260 0x7fa318156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 --2- 172.21.15.67:0/3859030732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa318161e80 0x7fa3180d2dc0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa30800c7f0 tx=0x7fa30800ccb0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 >> 172.21.15.67:0/3859030732 conn(0x7fa318072590 msgr2=0x7fa3181605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 shutdown_connections 2024-04-17T10:25:23.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fa31e200640 1 -- 172.21.15.67:0/3859030732 wait complete. 2024-04-17T10:25:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6cb414e600 con 0x7f6cb40b47b0 2024-04-17T10:25:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fecb8161800 con 0x7fecb8071310 2024-04-17T10:25:23.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7f6cb1600640 1 -- 172.21.15.67:0/4204852245 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ca0018250 con 0x7f6cb40b47b0 2024-04-17T10:25:23.340 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-17T10:25:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6f58074e30 con 0x7f6f581e7110 2024-04-17T10:25:23.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.343+0000 7fecaea00640 1 -- 172.21.15.67:0/2152907384 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feca80ef360 con 0x7fecb8071310 2024-04-17T10:25:23.341 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-04-17T10:25:23.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6f4e000640 1 -- 172.21.15.67:0/2482607349 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6f48058ef0 con 0x7f6f581e7110 2024-04-17T10:25:23.342 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-17T10:25:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c94066f60 msgr2=0x7f6c94069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c94066f60 0x7f6c94069420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6ca8009f60 tx=0x7f6ca80094f0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b47b0 msgr2=0x7f6cb414a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b47b0 0x7f6cb414a5a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6ca001ef00 tx=0x7f6ca001b400 comp rx=0 tx=0).stop 2024-04-17T10:25:23.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec98066fe0 msgr2=0x7fec980694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7fecbd600640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec98066fe0 0x7fec980694a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fecb815dff0 tx=0x7feca4009060 comp rx=0 tx=0).stop 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8071310 msgr2=0x7fecb8154920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7fecbd600640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8071310 0x7fecb8154920 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feca80ab000 tx=0x7feca80192e0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 shutdown_connections 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c94066f60 0x7f6c94069420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cb40cc1b0 0x7f6cb40b1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cb40b47b0 0x7f6cb414a5a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 --2- 172.21.15.67:0/4204852245 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cb40b3bb0 0x7f6cb4154760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.347+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 >> 172.21.15.67:0/4204852245 conn(0x7f6cb406ac90 msgr2=0x7f6cb406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 shutdown_connections 2024-04-17T10:25:23.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec98066fe0 0x7fec980694a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 shutdown_connections 2024-04-17T10:25:23.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecb80cef80 0x7fecb815e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecb80b4970 0x7fecb8152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 --2- 172.21.15.67:0/2152907384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecb8071310 0x7fecb8154920 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 >> 172.21.15.67:0/2152907384 conn(0x7fecb806ac90 msgr2=0x7fecb81657d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 shutdown_connections 2024-04-17T10:25:23.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6cbbe00640 1 -- 172.21.15.67:0/4204852245 wait complete. 2024-04-17T10:25:23.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7fecbd600640 1 -- 172.21.15.67:0/2152907384 wait complete. 2024-04-17T10:25:23.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f38067030 msgr2=0x7f6f380694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f38067030 0x7f6f380694f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6f58050870 tx=0x7f6f44009190 comp rx=0 tx=0).stop 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f581e7110 msgr2=0x7f6f581e7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f581e7110 0x7f6f581e7550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6f580cc0d0 tx=0x7f6f4800c940 comp rx=0 tx=0).stop 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 shutdown_connections 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f38067030 0x7f6f380694f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f581e7110 0x7f6f581e7550 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f581f30f0 0x7f6f581f3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 --2- 172.21.15.67:0/2482607349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f3c003e20 0x7f6f581525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 >> 172.21.15.67:0/2482607349 conn(0x7f6f5806ac90 msgr2=0x7f6f5806b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 shutdown_connections 2024-04-17T10:25:23.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.351+0000 7f6f5ce00640 1 -- 172.21.15.67:0/2482607349 wait complete. 2024-04-17T10:25:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.355+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f78ac005590 con 0x7f78d40b0660 2024-04-17T10:25:23.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.355+0000 7f78cb400640 1 -- 172.21.15.67:0/2643021838 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f78c4016300 con 0x7f78d40b0660 2024-04-17T10:25:23.353 INFO:teuthology.orchestra.run.smithi067.stdout:55834574857 2024-04-17T10:25:23.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe9b00722b0 con 0x7fe9b015da50 2024-04-17T10:25:23.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7fe9a6a00640 1 -- 172.21.15.67:0/1934821144 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe9a80591a0 con 0x7fe9b015da50 2024-04-17T10:25:23.356 INFO:teuthology.orchestra.run.smithi067.stdout:55834574856 2024-04-17T10:25:23.356 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.2 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78b0067160 msgr2=0x7f78b0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78b0067160 0x7f78b0069620 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f78bc001ec0 tx=0x7f78bc009040 comp rx=0 tx=0).stop 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d40b0660 msgr2=0x7f78d40afbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d40b0660 0x7f78d40afbc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f78c40aa000 tx=0x7f78b8005840 comp rx=0 tx=0).stop 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 shutdown_connections 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78b0067160 0x7f78b0069620 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78d40b1bc0 0x7f78d4159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78d40b0660 0x7f78d40afbc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 --2- 172.21.15.67:0/2643021838 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78b8004810 0x7f78d414a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.359+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 >> 172.21.15.67:0/2643021838 conn(0x7f78d406ac90 msgr2=0x7f78d4165eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.363+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 shutdown_connections 2024-04-17T10:25:23.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.363+0000 7f78da600640 1 -- 172.21.15.67:0/2643021838 wait complete. 2024-04-17T10:25:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.363+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe98c066fb0 msgr2=0x7fe98c069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.363+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe98c066fb0 0x7fe98c069470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe9a0001ec0 tx=0x7fe9a00010e0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.363+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b015da50 msgr2=0x7fe9b015b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:23.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.363+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b015da50 0x7fe9b015b270 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800c810 tx=0x7fe9a800ccd0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.367+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 shutdown_connections 2024-04-17T10:25:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.367+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe98c066fb0 0x7fe98c069470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.367+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9b015da50 0x7fe9b015b270 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.367+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9b0145210 0x7fe9b00c6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.367+0000 7fe9b4a00640 1 --2- 172.21.15.67:0/1934821144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9b0071310 0x7fe9b0140bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:23.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.367+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 >> 172.21.15.67:0/1934821144 conn(0x7fe9b006ac90 msgr2=0x7fe9b015de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:23.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.371+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 shutdown_connections 2024-04-17T10:25:23.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:23.371+0000 7fe9b4a00640 1 -- 172.21.15.67:0/1934821144 wait complete. 2024-04-17T10:25:23.371 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.1 2024-04-17T10:25:23.378 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.0 2024-04-17T10:25:23.382 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.5 2024-04-17T10:25:23.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574858 for osd.7 2024-04-17T10:25:23.392 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.4 2024-04-17T10:25:23.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.3 2024-04-17T10:25:23.396 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.6 2024-04-17T10:25:24.358 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-17T10:25:24.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-17T10:25:24.378 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-17T10:25:24.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-17T10:25:24.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-17T10:25:24.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-17T10:25:24.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-17T10:25:24.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-17T10:25:24.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb930c00640 1 Processor -- start 2024-04-17T10:25:24.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb930c00640 1 -- start start 2024-04-17T10:25:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb930c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb92c1597b0 0x7fb92c0b2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb930c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c0ce6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb930c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb92c06fac0 con 0x7fb92c159b90 2024-04-17T10:25:24.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb929e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb92c1597b0 0x7fb92c0b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb92a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb92c159b90 0x7fb92c06e340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54408/0 (socket says 172.21.15.67:54408) 2024-04-17T10:25:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.591+0000 7fb92a800640 1 -- 172.21.15.67:0/178537273 learned_addr learned my addr 172.21.15.67:0/178537273 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.595+0000 7fb930c00640 1 -- 172.21.15.67:0/178537273 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb92c0aebe0 con 0x7fb92c1597b0 2024-04-17T10:25:24.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.595+0000 7fb930c00640 1 -- 172.21.15.67:0/178537273 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c0aee30 con 0x7fb92c0ae720 2024-04-17T10:25:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.595+0000 7fb929e00640 1 -- 172.21.15.67:0/178537273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 msgr2=0x7fb92c0ce6b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-17T10:25:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.595+0000 7fb929e00640 1 --2- 172.21.15.67:0/178537273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c0ce6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.595+0000 7fb929e00640 1 -- 172.21.15.67:0/178537273 >> v1:172.21.15.67:6789/0 conn(0x7fb92c159b90 legacy=0x7fb92c06e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.595+0000 7fb929e00640 1 -- 172.21.15.67:0/178537273 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c0af110 con 0x7fb92c1597b0 2024-04-17T10:25:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb929e00640 1 --2- 172.21.15.67:0/178537273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c1597b0 0x7fb92c0b2930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0097c0 tx=0x7fb91c01c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee383fcaa4ead686 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb928a00640 1 -- 172.21.15.67:0/178537273 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c0b0070 con 0x7fb92c1597b0 2024-04-17T10:25:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb928a00640 1 -- 172.21.15.67:0/178537273 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91c0ab600 con 0x7fb92c1597b0 2024-04-17T10:25:24.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb928a00640 1 -- 172.21.15.67:0/178537273 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91c0ab9d0 con 0x7fb92c1597b0 2024-04-17T10:25:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 -- 172.21.15.67:0/178537273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c1597b0 msgr2=0x7fb92c0b2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 --2- 172.21.15.67:0/178537273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c1597b0 0x7fb92c0b2930 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb91c0097c0 tx=0x7fb91c01c430 comp rx=0 tx=0).stop 2024-04-17T10:25:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 -- 172.21.15.67:0/178537273 shutdown_connections 2024-04-17T10:25:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 --2- 172.21.15.67:0/178537273 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c0ce6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 --2- 172.21.15.67:0/178537273 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c1597b0 0x7fb92c0b2930 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 -- 172.21.15.67:0/178537273 >> 172.21.15.67:0/178537273 conn(0x7fb92c071260 msgr2=0x7fb92c071670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 -- 172.21.15.67:0/178537273 shutdown_connections 2024-04-17T10:25:24.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.603+0000 7fb930c00640 1 -- 172.21.15.67:0/178537273 wait complete. 2024-04-17T10:25:24.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.607+0000 7fb930c00640 1 Processor -- start 2024-04-17T10:25:24.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.607+0000 7fb930c00640 1 -- start start 2024-04-17T10:25:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb930c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c13f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb930c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb92c1597b0 0x7fb92c157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb930c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c159b90 0x7fb92c1507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb92a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c159b90 0x7fb92c1507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb92c1597b0 0x7fb92c157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c13f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb92a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c159b90 0x7fb92c1507f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58206/0 (socket says 172.21.15.67:58206) 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c13f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49532/0 (socket says 172.21.15.67:49532) 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb92c1597b0 0x7fb92c157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37320/0 (socket says 172.21.15.67:37320) 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb92a800640 1 -- 172.21.15.67:0/2384808723 learned_addr learned my addr 172.21.15.67:0/2384808723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb930c00640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c06ebd0 con 0x7fb92c159b90 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb930c00640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c06ea50 con 0x7fb92c1597b0 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb930c00640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb92c071d80 con 0x7fb92c0ae720 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 -- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 msgr2=0x7fb92c13f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c13f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 -- 172.21.15.67:0/2384808723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c159b90 msgr2=0x7fb92c1507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c159b90 0x7fb92c1507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb92c155ff0 con 0x7fb92c1597b0 2024-04-17T10:25:24.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb92a800640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c159b90 0x7fb92c1507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:24.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929e00640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c13f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:24.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.611+0000 7fb929400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb92c1597b0 0x7fb92c157090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb924009f70 tx=0x7fb92400bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.615+0000 7fb922000640 1 -- 172.21.15.67:0/2384808723 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924015710 con 0x7fb92c1597b0 2024-04-17T10:25:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.615+0000 7fb922000640 1 -- 172.21.15.67:0/2384808723 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9240158f0 con 0x7fb92c1597b0 2024-04-17T10:25:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.615+0000 7fb922000640 1 -- 172.21.15.67:0/2384808723 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb924015ce0 con 0x7fb92c1597b0 2024-04-17T10:25:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.615+0000 7fb930c00640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92c1562d0 con 0x7fb92c1597b0 2024-04-17T10:25:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.615+0000 7fb930c00640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92c156890 con 0x7fb92c1597b0 2024-04-17T10:25:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.619+0000 7fb922000640 1 -- 172.21.15.67:0/2384808723 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb924020400 con 0x7fb92c1597b0 2024-04-17T10:25:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.623+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c074dc0 con 0x7fb92c1597b0 2024-04-17T10:25:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.623+0000 7fb922000640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb908067030 0x7fb9080694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.623+0000 7fb922000640 1 -- 172.21.15.67:0/2384808723 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fb92408a0d0 con 0x7fb92c1597b0 2024-04-17T10:25:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.623+0000 7fb929e00640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb908067030 0x7fb9080694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.623+0000 7fb929e00640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb908067030 0x7fb9080694f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb92c06fda0 tx=0x7fb91c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.631+0000 7fb922000640 1 -- 172.21.15.67:0/2384808723 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb92405a730 con 0x7fb92c1597b0 2024-04-17T10:25:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.659+0000 7f482dc00640 1 Processor -- start 2024-04-17T10:25:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.659+0000 7f482dc00640 1 -- start start 2024-04-17T10:25:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.659+0000 7fd798a00640 1 Processor -- start 2024-04-17T10:25:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.659+0000 7fbddde00640 1 Processor -- start 2024-04-17T10:25:24.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.659+0000 7fd798a00640 1 -- start start 2024-04-17T10:25:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f482dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4828070ef0 0x7f48280d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f482dc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48280d0950 0x7f4828099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f482dc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482806c110 con 0x7f4828099f90 2024-04-17T10:25:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f482dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482809a400 con 0x7f4828070ef0 2024-04-17T10:25:24.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f482dc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48280712d0 con 0x7f48280d0950 2024-04-17T10:25:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd798a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7940cef80 0x7fd794163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd798a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd794071310 0x7fd794166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd798a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd7940b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd791e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7940cef80 0x7fd794163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd798a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7940b4bb0 con 0x7fd794071310 2024-04-17T10:25:24.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd798a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7940719a0 con 0x7fd7940cef80 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd791e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7940cef80 0x7fd794163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37324/0 (socket says 172.21.15.67:37324) 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd791e00640 1 -- 172.21.15.67:0/269692149 learned_addr learned my addr 172.21.15.67:0/269692149 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd798a00640 1 -- 172.21.15.67:0/269692149 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd794071c80 con 0x7fd7940b4260 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fbddde00640 1 -- start start 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f4827e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4828070ef0 0x7f48280d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f482d200640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4828099f90 0x7f482815d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54802/0 (socket says 172.21.15.67:54802) 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f482d200640 1 -- 172.21.15.67:0/4090535799 learned_addr learned my addr 172.21.15.67:0/4090535799 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7f4827400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48280d0950 0x7f4828099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd792800640 1 --2- 172.21.15.67:0/269692149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd7940b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd792800640 1 -- 172.21.15.67:0/269692149 >> v2:172.21.15.86:3300/0 conn(0x7fd7940cef80 msgr2=0x7fd794163060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fbddde00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd806b3a0 0x7fbdd80b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fbddde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbdd80b2660 0x7fbdd80b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fbddde00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd815aec0 con 0x7fbdd80b48a0 2024-04-17T10:25:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fbddde00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd815a340 con 0x7fbdd80b2660 2024-04-17T10:25:24.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fbddde00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd815a620 con 0x7fbdd806b3a0 2024-04-17T10:25:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fbdd6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbdd80b2660 0x7fbdd80b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd792800640 1 --2- 172.21.15.67:0/269692149 >> v2:172.21.15.86:3300/0 conn(0x7fd7940cef80 0x7fd794163060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd792800640 1 -- 172.21.15.67:0/269692149 >> v2:172.21.15.67:3300/0 conn(0x7fd794071310 msgr2=0x7fd794166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.663+0000 7fd792800640 1 --2- 172.21.15.67:0/269692149 >> v2:172.21.15.67:3300/0 conn(0x7fd794071310 0x7fd794166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7f4826a00640 1 -- 172.21.15.67:0/4090535799 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345591244 0 0) 0x7f482806c110 con 0x7f4828099f90 2024-04-17T10:25:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7f4826a00640 1 -- 172.21.15.67:0/4090535799 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f480c003530 con 0x7f4828099f90 2024-04-17T10:25:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fd792800640 1 -- 172.21.15.67:0/269692149 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd794162580 con 0x7fd7940b4260 2024-04-17T10:25:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fd791e00640 1 --2- 172.21.15.67:0/269692149 >> v2:172.21.15.86:3300/0 conn(0x7fd7940cef80 0x7fd794163060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7f4827400640 1 -- 172.21.15.67:0/4090535799 >> v2:172.21.15.67:3300/0 conn(0x7f4828070ef0 msgr2=0x7f48280d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7f4827400640 1 --2- 172.21.15.67:0/4090535799 >> v2:172.21.15.67:3300/0 conn(0x7f4828070ef0 0x7f48280d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7f4827400640 1 -- 172.21.15.67:0/4090535799 >> v1:172.21.15.86:6789/0 conn(0x7f4828099f90 legacy=0x7f482815d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7f4827400640 1 -- 172.21.15.67:0/4090535799 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4828070480 con 0x7f48280d0950 2024-04-17T10:25:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fbdd7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbdd80b48a0 0x7fbdd8071280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54410/0 (socket says 172.21.15.67:54410) 2024-04-17T10:25:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fbdd7400640 1 -- 172.21.15.67:0/3146520218 learned_addr learned my addr 172.21.15.67:0/3146520218 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fbdd6a00640 1 --2- 172.21.15.67:0/3146520218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd806b3a0 0x7fbdd80b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fbdd6a00640 1 -- 172.21.15.67:0/3146520218 >> v2:172.21.15.67:3300/0 conn(0x7fbdd80b2660 msgr2=0x7fbdd80b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fbdd6a00640 1 --2- 172.21.15.67:0/3146520218 >> v2:172.21.15.67:3300/0 conn(0x7fbdd80b2660 0x7fbdd80b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fbdd6a00640 1 -- 172.21.15.67:0/3146520218 >> v1:172.21.15.67:6789/0 conn(0x7fbdd80b48a0 legacy=0x7fbdd8071280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.667+0000 7fbdd6a00640 1 -- 172.21.15.67:0/3146520218 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd815a900 con 0x7fbdd806b3a0 2024-04-17T10:25:24.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7facd9400640 1 Processor -- start 2024-04-17T10:25:24.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f4827400640 1 --2- 172.21.15.67:0/4090535799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48280d0950 0x7f4828099a50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48140097c0 tx=0x7f481401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=beda134d42ed0ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd792800640 1 --2- 172.21.15.67:0/269692149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd7940b4670 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd79406bc50 tx=0x7fd78c01c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44963719319f052d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f4826a00640 1 -- 172.21.15.67:0/4090535799 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48140ab430 con 0x7f48280d0950 2024-04-17T10:25:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f4826a00640 1 -- 172.21.15.67:0/4090535799 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48140b0070 con 0x7f48280d0950 2024-04-17T10:25:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f4826a00640 1 -- 172.21.15.67:0/4090535799 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48140ab7f0 con 0x7f48280d0950 2024-04-17T10:25:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd790a00640 1 -- 172.21.15.67:0/269692149 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0b2070 con 0x7fd7940b4260 2024-04-17T10:25:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd790a00640 1 -- 172.21.15.67:0/269692149 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c0a57b0 con 0x7fd7940b4260 2024-04-17T10:25:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd790a00640 1 -- 172.21.15.67:0/269692149 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0a5b80 con 0x7fd7940b4260 2024-04-17T10:25:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7facd9400640 1 -- start start 2024-04-17T10:25:24.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 -- 172.21.15.67:0/4090535799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48280d0950 msgr2=0x7f4828099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 --2- 172.21.15.67:0/4090535799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48280d0950 0x7f4828099a50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48140097c0 tx=0x7f481401c450 comp rx=0 tx=0).stop 2024-04-17T10:25:24.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f5f7d800640 1 Processor -- start 2024-04-17T10:25:24.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fbdd6a00640 1 --2- 172.21.15.67:0/3146520218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd806b3a0 0x7fbdd80b3520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40097c0 tx=0x7fbdc401c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5afeb03ea20b8e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 -- 172.21.15.67:0/269692149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 msgr2=0x7fd7940b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 --2- 172.21.15.67:0/269692149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd7940b4670 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd79406bc50 tx=0x7fd78c01c930 comp rx=0 tx=0).stop 2024-04-17T10:25:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 -- 172.21.15.67:0/4090535799 shutdown_connections 2024-04-17T10:25:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 --2- 172.21.15.67:0/4090535799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48280d0950 0x7f4828099a50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 --2- 172.21.15.67:0/4090535799 >> v2:172.21.15.67:3300/0 conn(0x7f4828070ef0 0x7f48280d0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 -- 172.21.15.67:0/4090535799 >> 172.21.15.67:0/4090535799 conn(0x7f48280d1610 msgr2=0x7f48280d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 -- 172.21.15.67:0/269692149 shutdown_connections 2024-04-17T10:25:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 --2- 172.21.15.67:0/269692149 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd7940b4670 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 --2- 172.21.15.67:0/269692149 >> v2:172.21.15.67:3300/0 conn(0x7fd794071310 0x7fd794166430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 --2- 172.21.15.67:0/269692149 >> v2:172.21.15.86:3300/0 conn(0x7fd7940cef80 0x7fd794163060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 -- 172.21.15.67:0/269692149 >> 172.21.15.67:0/269692149 conn(0x7fd79406ac90 msgr2=0x7fd79406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 -- 172.21.15.67:0/4090535799 shutdown_connections 2024-04-17T10:25:24.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 -- 172.21.15.67:0/269692149 shutdown_connections 2024-04-17T10:25:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7fd798a00640 1 -- 172.21.15.67:0/269692149 wait complete. 2024-04-17T10:25:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f5f7d800640 1 -- start start 2024-04-17T10:25:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7f482dc00640 1 -- 172.21.15.67:0/4090535799 wait complete. 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7facd9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7facd4159bb0 0x7facd40b3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7facd9400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7facd415e1f0 0x7facd4071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7facd9400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd40b4cc0 con 0x7facd406b7b0 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7facd9400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd415e680 con 0x7facd4159bb0 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.671+0000 7facd9400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd40b49e0 con 0x7facd415e1f0 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd3200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7facd415e1f0 0x7facd4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd3200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7facd415e1f0 0x7facd4071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37348/0 (socket says 172.21.15.67:37348) 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd3200640 1 -- 172.21.15.67:0/875656255 learned_addr learned my addr 172.21.15.67:0/875656255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd1e00640 1 --2- 172.21.15.67:0/875656255 >> v2:172.21.15.67:3300/0 conn(0x7facd4159bb0 0x7facd40b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7efc10c00640 1 Processor -- start 2024-04-17T10:25:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd1400640 1 -- 172.21.15.67:0/875656255 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332904269 0 0) 0x7facd40b4cc0 con 0x7facd406b7b0 2024-04-17T10:25:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd1400640 1 -- 172.21.15.67:0/875656255 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb8003530 con 0x7facd406b7b0 2024-04-17T10:25:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7efc10c00640 1 -- start start 2024-04-17T10:25:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f7d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f7806b7b0 0x7f5f780b3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f7d800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f7815e1f0 0x7f5f780b4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f7d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f78071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f7d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7815e680 con 0x7f5f7806b7b0 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f7d800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f780b4740 con 0x7f5f7815e1f0 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f7d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f780b49e0 con 0x7f5f78159950 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbdd5600640 1 -- 172.21.15.67:0/3146520218 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40b0070 con 0x7fbdd806b3a0 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbdd5600640 1 -- 172.21.15.67:0/3146520218 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc40ab600 con 0x7fbdd806b3a0 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbdd5600640 1 -- 172.21.15.67:0/3146520218 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc40ab9d0 con 0x7fbdd806b3a0 2024-04-17T10:25:24.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbddde00640 1 -- 172.21.15.67:0/3146520218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd806b3a0 msgr2=0x7fbdd80b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbddde00640 1 --2- 172.21.15.67:0/3146520218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd806b3a0 0x7fbdd80b3520 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40097c0 tx=0x7fbdc401c430 comp rx=0 tx=0).stop 2024-04-17T10:25:24.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f482dc00640 1 Processor -- start 2024-04-17T10:25:24.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd3200640 1 -- 172.21.15.67:0/875656255 >> v2:172.21.15.67:3300/0 conn(0x7facd4159bb0 msgr2=0x7facd40b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd3200640 1 --2- 172.21.15.67:0/875656255 >> v2:172.21.15.67:3300/0 conn(0x7facd4159bb0 0x7facd40b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd3200640 1 -- 172.21.15.67:0/875656255 >> v1:172.21.15.86:6789/0 conn(0x7facd406b7b0 legacy=0x7facd40b4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7facd3200640 1 -- 172.21.15.67:0/875656255 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd40b2880 con 0x7facd415e1f0 2024-04-17T10:25:24.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fd798a00640 1 Processor -- start 2024-04-17T10:25:24.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fd798a00640 1 -- start start 2024-04-17T10:25:24.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f77400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f78071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f482dc00640 1 -- start start 2024-04-17T10:25:24.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f6fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f7815e1f0 0x7f5f780b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f6fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f7815e1f0 0x7f5f780b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37362/0 (socket says 172.21.15.67:37362) 2024-04-17T10:25:24.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f6fe00640 1 -- 172.21.15.67:0/179857500 learned_addr learned my addr 172.21.15.67:0/179857500 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f5f76a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5f7806b7b0 0x7f5f780b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f1189200640 1 Processor -- start 2024-04-17T10:25:24.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7f1189200640 1 -- start start 2024-04-17T10:25:24.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbddde00640 1 -- 172.21.15.67:0/3146520218 shutdown_connections 2024-04-17T10:25:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbddde00640 1 --2- 172.21.15.67:0/3146520218 >> v2:172.21.15.67:3300/0 conn(0x7fbdd80b2660 0x7fbdd80b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbddde00640 1 --2- 172.21.15.67:0/3146520218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd806b3a0 0x7fbdd80b3520 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.675+0000 7fbddde00640 1 -- 172.21.15.67:0/3146520218 >> 172.21.15.67:0/3146520218 conn(0x7fbdd8066e10 msgr2=0x7fbdd806ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f5f6fe00640 1 -- 172.21.15.67:0/179857500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 msgr2=0x7f5f78071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fbddde00640 1 -- 172.21.15.67:0/3146520218 shutdown_connections 2024-04-17T10:25:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fbddde00640 1 -- 172.21.15.67:0/3146520218 wait complete. 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd798a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd794071310 0x7fd7941436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd798a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd794143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd798a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7940cef80 0x7fd79415a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd798a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940b3100 con 0x7fd7940cef80 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd798a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940b2f80 con 0x7fd794071310 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd798a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7940b3280 con 0x7fd7940b4260 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f5f6fe00640 1 --2- 172.21.15.67:0/179857500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f78071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f5f6fe00640 1 -- 172.21.15.67:0/179857500 >> v2:172.21.15.67:3300/0 conn(0x7f5f7806b7b0 msgr2=0x7f5f780b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f5f6fe00640 1 --2- 172.21.15.67:0/179857500 >> v2:172.21.15.67:3300/0 conn(0x7f5f7806b7b0 0x7f5f780b3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1189200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f117c0502d0 0x7f117c0e9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1183400640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f117c0defd0 0x7f117c006d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54822/0 (socket says 172.21.15.67:54822) 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1183400640 1 -- 172.21.15.67:0/1486079370 learned_addr learned my addr 172.21.15.67:0/1486079370 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c007550 con 0x7f117c03ab40 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c007830 con 0x7f117c0defd0 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc10c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efc0c0b47b0 0x7efc0c15e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc10c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0b3bb0 0x7efc0c071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c034670 con 0x7f117c0502d0 2024-04-17T10:25:24.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc10c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0c15e630 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc10c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc0c15a0f0 con 0x7efc0c0b47b0 2024-04-17T10:25:24.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc10c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c15a3b0 con 0x7efc0c0b3bb0 2024-04-17T10:25:24.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7940cef80 0x7fd79415a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1183e00640 1 --2- 172.21.15.67:0/1486079370 >> v2:172.21.15.86:3300/0 conn(0x7f117c0502d0 0x7f117c0e9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f482dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4828070ef0 0x7f4828159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f482dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4828099f90 0x7f48281717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f482dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48280d0950 0x7f4828203600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f482dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482806d1b0 con 0x7f48280d0950 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f482dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482806cc90 con 0x7f4828099f90 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f482dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482806bda0 con 0x7f4828070ef0 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc0a800640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efc0c0cc1b0 0x7efc0c153800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54832/0 (socket says 172.21.15.67:54832) 2024-04-17T10:25:24.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc0a800640 1 -- 172.21.15.67:0/2182890367 learned_addr learned my addr 172.21.15.67:0/2182890367 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7940cef80 0x7fd79415a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58270/0 (socket says 172.21.15.67:58270) 2024-04-17T10:25:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 -- 172.21.15.67:0/726210271 learned_addr learned my addr 172.21.15.67:0/726210271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd791400640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd794143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc09e00640 1 --2- 172.21.15.67:0/2182890367 >> v2:172.21.15.86:3300/0 conn(0x7efc0c0b47b0 0x7efc0c15e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd791e00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd794071310 0x7fd7941436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 -- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 msgr2=0x7fd794143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd794143f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 -- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd794071310 msgr2=0x7fd7941436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4828070ef0 0x7f4828159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4828070ef0 0x7f4828159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49604/0 (socket says 172.21.15.67:49604) 2024-04-17T10:25:24.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 -- 172.21.15.67:0/138459920 learned_addr learned my addr 172.21.15.67:0/138459920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f482d200640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48280d0950 0x7f4828203600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f5f6fe00640 1 -- 172.21.15.67:0/179857500 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f780b2880 con 0x7f5f7815e1f0 2024-04-17T10:25:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757853194 0 0) 0x7f117c007830 con 0x7f117c0defd0 2024-04-17T10:25:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f116c005740 con 0x7f117c0defd0 2024-04-17T10:25:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc09400640 1 --2- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0b3bb0 0x7efc0c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 -- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4828099f90 msgr2=0x7f48281717b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T10:25:24.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4828099f90 0x7f48281717b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 -- 172.21.15.67:0/138459920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48280d0950 msgr2=0x7f4828203600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48280d0950 0x7f4828203600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428290439 0 0) 0x7f117c007550 con 0x7f117c03ab40 2024-04-17T10:25:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f117c007830 con 0x7f117c03ab40 2024-04-17T10:25:24.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 -- 172.21.15.67:0/138459920 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482815c730 con 0x7f4828070ef0 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 776128729 0 0) 0x7f116c005740 con 0x7f117c0defd0 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 >> v2:172.21.15.86:3300/0 conn(0x7f117c0502d0 msgr2=0x7f117c0e9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 --2- 172.21.15.67:0/1486079370 >> v2:172.21.15.86:3300/0 conn(0x7f117c0502d0 0x7f117c0e9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 >> v1:172.21.15.67:6789/0 conn(0x7f117c03ab40 legacy=0x7f117c04fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c007b10 con 0x7f117c0defd0 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f1178001680 con 0x7f117c0defd0 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 >> v1:172.21.15.86:6789/0 conn(0x7f117c0defd0 legacy=0x7f117c006d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f116c005d60 0x7f116c006180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c006a20 0x7f116c006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c0073e0 0x7f116c0147d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c034670 con 0x7f116c006a20 2024-04-17T10:25:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f116c00bc30 con 0x7f116c0073e0 2024-04-17T10:25:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116c00be30 con 0x7f116c005d60 2024-04-17T10:25:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f4827e00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4828070ef0 0x7f4828159be0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4820007e40 tx=0x7f482000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd794071310 0x7fd7941436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 -- 172.21.15.67:0/726210271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7940b0700 con 0x7fd7940cef80 2024-04-17T10:25:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd791e00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd794071310 0x7fd7941436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:24.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd792800640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7940cef80 0x7fd79415a1a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd78c01d1d0 tx=0x7fd78c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc08a00640 1 -- 172.21.15.67:0/2182890367 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051397353 0 0) 0x7efbfc004e40 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc08a00640 1 -- 172.21.15.67:0/2182890367 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf0003530 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f1182a00640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c006a20 0x7f116c006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f481fe00640 1 -- 172.21.15.67:0/138459920 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4820014620 con 0x7f4828070ef0 2024-04-17T10:25:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd78a000640 1 -- 172.21.15.67:0/726210271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0bb020 con 0x7fd7940cef80 2024-04-17T10:25:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f481fe00640 1 -- 172.21.15.67:0/138459920 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4820019070 con 0x7f4828070ef0 2024-04-17T10:25:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7f481fe00640 1 -- 172.21.15.67:0/138459920 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4820014a00 con 0x7f4828070ef0 2024-04-17T10:25:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd78a000640 1 -- 172.21.15.67:0/726210271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c0ad640 con 0x7fd7940cef80 2024-04-17T10:25:24.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd78a000640 1 -- 172.21.15.67:0/726210271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0b2040 con 0x7fd7940cef80 2024-04-17T10:25:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc09e00640 1 -- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0b3bb0 msgr2=0x7efc0c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc09e00640 1 --2- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0b3bb0 0x7efc0c071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7efc09e00640 1 -- 172.21.15.67:0/2182890367 >> v1:172.21.15.86:6789/0 conn(0x7efc0c0cc1b0 legacy=0x7efc0c153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7efc09e00640 1 -- 172.21.15.67:0/2182890367 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c15a690 con 0x7efc0c0b47b0 2024-04-17T10:25:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.679+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7940b09e0 con 0x7fd7940cef80 2024-04-17T10:25:24.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1183400640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f116c005d60 0x7f116c006180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7940b0fa0 con 0x7fd7940cef80 2024-04-17T10:25:24.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1183e00640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c0073e0 0x7f116c0147d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7fbddde00640 1 Processor -- start 2024-04-17T10:25:24.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482815ca10 con 0x7f4828070ef0 2024-04-17T10:25:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7efc09400640 1 --2- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0b3bb0 0x7efc0c071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1183e00640 1 -- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f116c005d60 msgr2=0x7f116c006180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1183e00640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f116c005d60 0x7f116c006180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1183e00640 1 -- 172.21.15.67:0/1486079370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c006a20 msgr2=0x7f116c006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1183e00640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c006a20 0x7f116c006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1183e00640 1 -- 172.21.15.67:0/1486079370 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c007b10 con 0x7f116c0073e0 2024-04-17T10:25:24.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48280c6ac0 con 0x7f4828070ef0 2024-04-17T10:25:24.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f1182a00640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c006a20 0x7f116c006ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:24.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd794072d50 con 0x7fd7940cef80 2024-04-17T10:25:24.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48280c6cc0 con 0x7f4828070ef0 2024-04-17T10:25:24.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7facd3200640 1 --2- 172.21.15.67:0/875656255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd415e1f0 0x7facd4071310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7facd406e6c0 tx=0x7faccc01ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1a796849c7a1338 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7fbddde00640 1 -- start start 2024-04-17T10:25:24.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7f5f6fe00640 1 --2- 172.21.15.67:0/179857500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f780b4200 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5f780af3b0 tx=0x7f5f6001ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e6b2fbae46b4da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7facd1400640 1 -- 172.21.15.67:0/875656255 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0a56c0 con 0x7facd415e1f0 2024-04-17T10:25:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7facd1400640 1 -- 172.21.15.67:0/875656255 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faccc0a5860 con 0x7facd415e1f0 2024-04-17T10:25:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7facd1400640 1 -- 172.21.15.67:0/875656255 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faccc0a5c30 con 0x7facd415e1f0 2024-04-17T10:25:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7fbddde00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd806b3a0 0x7fbdd80af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7fbddde00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80b2660 0x7fbdd8149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.683+0000 7fbddde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdd80b48a0 0x7fbdd80b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbddde00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd8071b10 con 0x7fbdd80b48a0 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbddde00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd8071990 con 0x7fbdd806b3a0 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbddde00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd806de20 con 0x7fbdd80b2660 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdd80b48a0 0x7fbdd80b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdd80b48a0 0x7fbdd80b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58296/0 (socket says 172.21.15.67:58296) 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 -- 172.21.15.67:0/1725326259 learned_addr learned my addr 172.21.15.67:0/1725326259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 -- 172.21.15.67:0/875656255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd415e1f0 msgr2=0x7facd4071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 --2- 172.21.15.67:0/875656255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd415e1f0 0x7facd4071310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7facd406e6c0 tx=0x7faccc01ca30 comp rx=0 tx=0).stop 2024-04-17T10:25:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fd78a000640 1 -- 172.21.15.67:0/726210271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd78c0be020 con 0x7fd7940cef80 2024-04-17T10:25:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f5f76000640 1 -- 172.21.15.67:0/179857500 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f600b5070 con 0x7f5f7815e1f0 2024-04-17T10:25:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 -- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80b2660 msgr2=0x7fbdd8149940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T10:25:24.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80b2660 0x7fbdd8149940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 -- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd806b3a0 msgr2=0x7fbdd80af330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd806b3a0 0x7fbdd80af330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 -- 172.21.15.67:0/1725326259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8149e80 con 0x7fbdd80b48a0 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f5f76000640 1 -- 172.21.15.67:0/179857500 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f600a89d0 con 0x7f5f7815e1f0 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f5f76000640 1 -- 172.21.15.67:0/179857500 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f600a8da0 con 0x7f5f7815e1f0 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 -- 172.21.15.67:0/875656255 shutdown_connections 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 --2- 172.21.15.67:0/875656255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd415e1f0 0x7facd4071310 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 --2- 172.21.15.67:0/875656255 >> v2:172.21.15.67:3300/0 conn(0x7facd4159bb0 0x7facd40b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 -- 172.21.15.67:0/875656255 >> 172.21.15.67:0/875656255 conn(0x7facd406ac90 msgr2=0x7facd406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1183e00640 1 -- 172.21.15.67:0/1486079370 reap_dead start 2024-04-17T10:25:24.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fbdd7400640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdd80b48a0 0x7fbdd80b1030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c550 tx=0x7fbdc800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1183e00640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c0073e0 0x7f116c0147d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1184064760 tx=0x7f11840976f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db64b3fb73392abb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 -- 172.21.15.67:0/875656255 shutdown_connections 2024-04-17T10:25:24.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7facd9400640 1 -- 172.21.15.67:0/875656255 wait complete. 2024-04-17T10:25:24.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118412d070 con 0x7f116c0073e0 2024-04-17T10:25:24.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1182000640 1 -- 172.21.15.67:0/1486079370 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1184128410 con 0x7f116c0073e0 2024-04-17T10:25:24.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f5f7d800640 1 -- 172.21.15.67:0/179857500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 msgr2=0x7f5f780b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f5f7d800640 1 --2- 172.21.15.67:0/179857500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f780b4200 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5f780af3b0 tx=0x7f5f6001ed10 comp rx=0 tx=0).stop 2024-04-17T10:25:24.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f481fe00640 1 -- 172.21.15.67:0/138459920 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4820014ba0 con 0x7f4828070ef0 2024-04-17T10:25:24.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c0073e0 msgr2=0x7f116c0147d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1189200640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c0073e0 0x7f116c0147d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1184064760 tx=0x7f11840976f0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fd78a000640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd774067060 0x7fd774069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fd78a000640 1 -- 172.21.15.67:0/726210271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd78c11f9e0 con 0x7fd7940cef80 2024-04-17T10:25:24.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7fd791e00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd774067060 0x7fd774069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 shutdown_connections 2024-04-17T10:25:24.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1189200640 1 --2- 172.21.15.67:0/1486079370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f116c0073e0 0x7f116c0147d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.687+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 >> 172.21.15.67:0/1486079370 conn(0x7f117c054ef0 msgr2=0x7f117c0552d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 shutdown_connections 2024-04-17T10:25:24.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 -- 172.21.15.67:0/179857500 shutdown_connections 2024-04-17T10:25:24.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 --2- 172.21.15.67:0/179857500 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f78071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 --2- 172.21.15.67:0/179857500 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f780b4200 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 --2- 172.21.15.67:0/179857500 >> v2:172.21.15.67:3300/0 conn(0x7f5f7806b7b0 0x7f5f780b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 -- 172.21.15.67:0/179857500 >> 172.21.15.67:0/179857500 conn(0x7f5f7806ac90 msgr2=0x7f5f7806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 -- 172.21.15.67:0/179857500 shutdown_connections 2024-04-17T10:25:24.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f1189200640 1 -- 172.21.15.67:0/1486079370 wait complete. 2024-04-17T10:25:24.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 -- 172.21.15.67:0/179857500 wait complete. 2024-04-17T10:25:24.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 Processor -- start 2024-04-17T10:25:24.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7fd791e00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd774067060 0x7fd774069520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd78400d940 tx=0x7fd784008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 -- start start 2024-04-17T10:25:24.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc09e00640 1 --2- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c15e0f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efbf800a960 tx=0x7efbf801d890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a232679dad9a3f97 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc08a00640 1 -- 172.21.15.67:0/2182890367 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf80b3070 con 0x7efc0c0b47b0 2024-04-17T10:25:24.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc08a00640 1 -- 172.21.15.67:0/2182890367 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf80a6810 con 0x7efc0c0b47b0 2024-04-17T10:25:24.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc08a00640 1 -- 172.21.15.67:0/2182890367 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf800b040 con 0x7efc0c0b47b0 2024-04-17T10:25:24.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f481fe00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4808067030 0x7f48080694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f481fe00640 1 -- 172.21.15.67:0/138459920 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4820086a90 con 0x7f4828070ef0 2024-04-17T10:25:24.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f4827400640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4808067030 0x7f48080694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f1189200640 1 Processor -- start 2024-04-17T10:25:24.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 -- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 msgr2=0x7efc0c15e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 --2- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c15e0f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efbf800a960 tx=0x7efbf801d890 comp rx=0 tx=0).stop 2024-04-17T10:25:24.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd406b7b0 0x7facd415a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facd4159bb0 0x7facd415af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd406b7b0 0x7facd415a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 Processor -- start 2024-04-17T10:25:24.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd406b7b0 0x7facd415a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37414/0 (socket says 172.21.15.67:37414) 2024-04-17T10:25:24.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f1189200640 1 -- start start 2024-04-17T10:25:24.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f5f7d800640 1 -- start start 2024-04-17T10:25:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facd4159bb0 0x7facd415af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facd4159bb0 0x7facd415af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58304/0 (socket says 172.21.15.67:58304) 2024-04-17T10:25:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd2800640 1 -- 172.21.15.67:0/4012577471 learned_addr learned my addr 172.21.15.67:0/4012577471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd415e1f0 0x7facd415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f4827400640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4808067030 0x7f48080694f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48140097c0 tx=0x7f48140a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd40ce680 con 0x7facd4159bb0 2024-04-17T10:25:24.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd40ce500 con 0x7facd406b7b0 2024-04-17T10:25:24.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd40ce800 con 0x7facd415e1f0 2024-04-17T10:25:24.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7facd3200640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd415e1f0 0x7facd415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7f481fe00640 1 -- 172.21.15.67:0/138459920 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f48280c6cc0 con 0x7f4828070ef0 2024-04-17T10:25:24.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 -- 172.21.15.67:0/2182890367 shutdown_connections 2024-04-17T10:25:24.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 --2- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0b3bb0 0x7efc0c071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 --2- 172.21.15.67:0/2182890367 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c15e0f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 -- 172.21.15.67:0/2182890367 >> 172.21.15.67:0/2182890367 conn(0x7efc0c06ac90 msgr2=0x7efc0c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 -- 172.21.15.67:0/2182890367 shutdown_connections 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.691+0000 7efc10c00640 1 -- 172.21.15.67:0/2182890367 wait complete. 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7806b7b0 0x7f5f7815c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f7813b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f78157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78159de0 con 0x7f5f7806b7b0 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f780b2e40 con 0x7f5f7815e1f0 2024-04-17T10:25:24.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f78159f60 con 0x7f5f78159950 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0073e0 0x7f117c0da680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c04a660 0x7f117c0536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c053c20 0x7f117c177fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0df3b0 con 0x7f116c0073e0 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0532c0 con 0x7f117c053c20 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0498c0 con 0x7f117c04a660 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c04a660 0x7f117c0536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c04a660 0x7f117c0536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49640/0 (socket says 172.21.15.67:49640) 2024-04-17T10:25:24.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 -- 172.21.15.67:0/1331845255 learned_addr learned my addr 172.21.15.67:0/1331845255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f76a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7806b7b0 0x7f5f7815c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f76a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7806b7b0 0x7f5f7815c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58308/0 (socket says 172.21.15.67:58308) 2024-04-17T10:25:24.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f76a00640 1 -- 172.21.15.67:0/3084313277 learned_addr learned my addr 172.21.15.67:0/3084313277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f77400640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f78157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6fe00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f7813b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7facd1e00640 1 -- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd415e1f0 msgr2=0x7facd415b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7facd1e00640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd415e1f0 0x7facd415b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7facd1e00640 1 -- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd406b7b0 msgr2=0x7facd415a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7facd1e00640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd406b7b0 0x7facd415a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7facd1e00640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd414ed90 con 0x7facd4159bb0 2024-04-17T10:25:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7facd1e00640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facd4159bb0 0x7facd415af80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7facc000e870 tx=0x7facc000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7facd2800640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd406b7b0 0x7facd415a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6fe00640 1 -- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 msgr2=0x7f5f78157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6fe00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f78157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6fe00640 1 -- 172.21.15.67:0/3084313277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7806b7b0 msgr2=0x7f5f7815c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6fe00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7806b7b0 0x7f5f7815c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6fe00640 1 -- 172.21.15.67:0/3084313277 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f78142d50 con 0x7f5f78159950 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f77400640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f78157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f76a00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7806b7b0 0x7f5f7815c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1183400640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0073e0 0x7f117c0da680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 -- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c053c20 msgr2=0x7f117c177fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c053c20 0x7f117c177fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 -- 172.21.15.67:0/1331845255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0073e0 msgr2=0x7f117c0da680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0073e0 0x7f117c0da680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 -- 172.21.15.67:0/1331845255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c178d10 con 0x7f117c04a660 2024-04-17T10:25:24.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1182a00640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c04a660 0x7f117c0536e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f117000e810 tx=0x7f117000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1177400640 1 -- 172.21.15.67:0/1331845255 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700050d0 con 0x7f117c04a660 2024-04-17T10:25:24.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1177400640 1 -- 172.21.15.67:0/1331845255 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1170018070 con 0x7f117c04a660 2024-04-17T10:25:24.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1177400640 1 -- 172.21.15.67:0/1331845255 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11700054b0 con 0x7f117c04a660 2024-04-17T10:25:24.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6fe00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f7813b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f780b48c0 tx=0x7f5f6001bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c049b80 con 0x7f117c04a660 2024-04-17T10:25:24.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c04a140 con 0x7f117c04a660 2024-04-17T10:25:24.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6ea00640 1 -- 172.21.15.67:0/3084313277 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f600b5040 con 0x7f5f78159950 2024-04-17T10:25:24.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6ea00640 1 -- 172.21.15.67:0/3084313277 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f600c1070 con 0x7f5f78159950 2024-04-17T10:25:24.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f6ea00640 1 -- 172.21.15.67:0/3084313277 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f600b1c40 con 0x7f5f78159950 2024-04-17T10:25:24.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f78143030 con 0x7f5f78159950 2024-04-17T10:25:24.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f781435f0 con 0x7f5f78159950 2024-04-17T10:25:24.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7efc10c00640 1 Processor -- start 2024-04-17T10:25:24.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.695+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c008e00 con 0x7f117c04a660 2024-04-17T10:25:24.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc10c00640 1 -- start start 2024-04-17T10:25:24.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f78072d50 con 0x7f5f78159950 2024-04-17T10:25:24.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7fbdcf400640 1 -- 172.21.15.67:0/1725326259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8017070 con 0x7fbdd80b48a0 2024-04-17T10:25:24.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7fbddde00640 1 -- 172.21.15.67:0/1725326259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd80afa30 con 0x7fbdd80b48a0 2024-04-17T10:25:24.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7fbddde00640 1 -- 172.21.15.67:0/1725326259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd8150430 con 0x7fbdd80b48a0 2024-04-17T10:25:24.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7fd78a000640 1 -- 172.21.15.67:0/726210271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd78c0c5030 con 0x7fd7940cef80 2024-04-17T10:25:24.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc10c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc0c0b3bb0 0x7efc0c157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc10c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c14db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7faccaa00640 1 -- 172.21.15.67:0/4012577471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc0017070 con 0x7facd4159bb0 2024-04-17T10:25:24.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7faccaa00640 1 -- 172.21.15.67:0/4012577471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc000c940 con 0x7facd4159bb0 2024-04-17T10:25:24.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7faccaa00640 1 -- 172.21.15.67:0/4012577471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc000ccf0 con 0x7facd4159bb0 2024-04-17T10:25:24.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc10c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0cc1b0 0x7efc0c158580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc09400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c14db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7f1177400640 1 -- 172.21.15.67:0/1331845255 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f117000c660 con 0x7f117c04a660 2024-04-17T10:25:24.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7f1177400640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1164067130 0x7f11640695f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7f1177400640 1 -- 172.21.15.67:0/1331845255 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f11700850b0 con 0x7f117c04a660 2024-04-17T10:25:24.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd414f070 con 0x7facd4159bb0 2024-04-17T10:25:24.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd414f630 con 0x7facd4159bb0 2024-04-17T10:25:24.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc09400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c14db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37454/0 (socket says 172.21.15.67:37454) 2024-04-17T10:25:24.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc09400640 1 -- 172.21.15.67:0/1207441336 learned_addr learned my addr 172.21.15.67:0/1207441336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:24.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7f5f6ea00640 1 -- 172.21.15.67:0/3084313277 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5f600a8890 con 0x7f5f78159950 2024-04-17T10:25:24.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7f5f6ea00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f54066fb0 0x7f5f54069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c0b2920 con 0x7efc0c0b3bb0 2024-04-17T10:25:24.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.699+0000 7efc0a800640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0cc1b0 0x7efc0c158580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7f5f6ea00640 1 -- 172.21.15.67:0/3084313277 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5f60122c30 con 0x7f5f78159950 2024-04-17T10:25:24.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c071c60 con 0x7efc0c0b47b0 2024-04-17T10:25:24.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc0c0b2aa0 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd4072d50 con 0x7facd4159bb0 2024-04-17T10:25:24.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc0a800640 1 -- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 msgr2=0x7efc0c14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc09e00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc0c0b3bb0 0x7efc0c157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc0a800640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c14db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc0a800640 1 -- 172.21.15.67:0/1207441336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc0c0b3bb0 msgr2=0x7efc0c157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7f1183400640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1164067130 0x7f11640695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc0a800640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc0c0b3bb0 0x7efc0c157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc0a800640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0c14e520 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc09400640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc09e00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc0c0b3bb0 0x7efc0c157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7efc0a800640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0cc1b0 0x7efc0c158580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c700 tx=0x7efbfc00cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7f5f76a00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f54066fb0 0x7f5f54069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7f1183400640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1164067130 0x7f11640695f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1178004880 tx=0x7f1178009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7f1177400640 1 -- 172.21.15.67:0/1331845255 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1170055ce0 con 0x7f117c04a660 2024-04-17T10:25:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb00052a0 con 0x7fbdd80b48a0 2024-04-17T10:25:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7fbdcf400640 1 -- 172.21.15.67:0/1725326259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc8005ac0 con 0x7fbdd80b48a0 2024-04-17T10:25:24.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.703+0000 7fbdcf400640 1 -- 172.21.15.67:0/1725326259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8007070 con 0x7fbdd80b48a0 2024-04-17T10:25:24.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.707+0000 7f5f76a00640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f54066fb0 0x7f5f54069470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800af80 tx=0x7f5f68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.707+0000 7fbdcf400640 1 -- 172.21.15.67:0/1725326259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbdc801a070 con 0x7fbdd80b48a0 2024-04-17T10:25:24.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.707+0000 7faccaa00640 1 -- 172.21.15.67:0/4012577471 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7facc001a070 con 0x7facd4159bb0 2024-04-17T10:25:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.707+0000 7fbdcf400640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdb8066f60 0x7fbdb8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.707+0000 7faccaa00640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facb4066fb0 0x7facb4069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.707+0000 7faccaa00640 1 -- 172.21.15.67:0/4012577471 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7facc0085960 con 0x7facd4159bb0 2024-04-17T10:25:24.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.707+0000 7f5f6ea00640 1 -- 172.21.15.67:0/3084313277 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5f60018870 con 0x7f5f78159950 2024-04-17T10:25:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7facd2800640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facb4066fb0 0x7facb4069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7fbdd6a00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdb8066f60 0x7fbdb8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7facd2800640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facb4066fb0 0x7facb4069470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7facc40011f0 tx=0x7facc4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7efc02000640 1 -- 172.21.15.67:0/1207441336 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc007250 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7efc02000640 1 -- 172.21.15.67:0/1207441336 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc017070 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7efc02000640 1 -- 172.21.15.67:0/1207441336 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc00a880 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0c14e800 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc0c14edc0 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.711+0000 7fbdd6a00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdb8066f60 0x7fbdb8069420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40ad000 tx=0x7fbdc40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.715+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc0c072d50 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.715+0000 7fbdcf400640 1 -- 172.21.15.67:0/1725326259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbdc8082540 con 0x7fbdd80b48a0 2024-04-17T10:25:24.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.715+0000 7faccaa00640 1 -- 172.21.15.67:0/4012577471 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7facc00560c0 con 0x7facd4159bb0 2024-04-17T10:25:24.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.715+0000 7efc02000640 1 -- 172.21.15.67:0/1207441336 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efbfc00aa20 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.719+0000 7efc02000640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbec067060 0x7efbec069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:24.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.719+0000 7efc02000640 1 -- 172.21.15.67:0/1207441336 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7efbfc0852f0 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.719+0000 7efc09e00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbec067060 0x7efbec069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:24.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.719+0000 7efc09e00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbec067060 0x7efbec069520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efbf80b0000 tx=0x7efbf800c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:24.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.719+0000 7fbdcf400640 1 -- 172.21.15.67:0/1725326259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbdc8054f10 con 0x7fbdd80b48a0 2024-04-17T10:25:24.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.723+0000 7efc02000640 1 -- 172.21.15.67:0/1207441336 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efc0c15e630 con 0x7efc0c0cc1b0 2024-04-17T10:25:24.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.971+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb92c06f020 con 0x7fb92c1597b0 2024-04-17T10:25:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.975+0000 7fb922000640 1 -- 172.21.15.67:0/2384808723 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb92405e7d0 con 0x7fb92c1597b0 2024-04-17T10:25:24.969 INFO:teuthology.orchestra.run.smithi067.stdout:55834574860 2024-04-17T10:25:24.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.975+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb908067030 msgr2=0x7fb9080694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.975+0000 7fb91b400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb908067030 0x7fb9080694f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb92c06fda0 tx=0x7fb91c0a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.975+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb92c1597b0 msgr2=0x7fb92c157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb92c1597b0 0x7fb92c157090 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb924009f70 tx=0x7fb92400bee0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 shutdown_connections 2024-04-17T10:25:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb908067030 0x7fb9080694f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb92c159b90 0x7fb92c1507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb92c1597b0 0x7fb92c157090 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 --2- 172.21.15.67:0/2384808723 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb92c0ae720 0x7fb92c13f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 >> 172.21.15.67:0/2384808723 conn(0x7fb92c071260 msgr2=0x7fb92c153650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 shutdown_connections 2024-04-17T10:25:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:24.979+0000 7fb91b400640 1 -- 172.21.15.67:0/2384808723 wait complete. 2024-04-17T10:25:25.005 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.2 2024-04-17T10:25:25.005 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.035+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7efc0c0cbaa0 con 0x7efc0c0cc1b0 2024-04-17T10:25:25.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc02000640 1 -- 172.21.15.67:0/1207441336 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efbfc055fa0 con 0x7efc0c0cc1b0 2024-04-17T10:25:25.034 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-04-17T10:25:25.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbec067060 msgr2=0x7efbec069520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbec067060 0x7efbec069520 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efbf80b0000 tx=0x7efbf800c040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0cc1b0 msgr2=0x7efc0c158580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0cc1b0 0x7efc0c158580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efbfc00c700 tx=0x7efbfc00cbc0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 shutdown_connections 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbec067060 0x7efbec069520 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efbf80b0000 tx=0x7efbf800c040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc0c0cc1b0 0x7efc0c158580 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc0c0b47b0 0x7efc0c14db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 --2- 172.21.15.67:0/1207441336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc0c0b3bb0 0x7efc0c157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 >> 172.21.15.67:0/1207441336 conn(0x7efc0c06ac90 msgr2=0x7efc0c157d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 shutdown_connections 2024-04-17T10:25:25.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.039+0000 7efc10c00640 1 -- 172.21.15.67:0/1207441336 wait complete. 2024-04-17T10:25:25.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.043+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f117c054020 con 0x7f117c04a660 2024-04-17T10:25:25.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.043+0000 7f1177400640 1 -- 172.21.15.67:0/1331845255 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11700555e0 con 0x7f117c04a660 2024-04-17T10:25:25.039 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:25.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4828074dc0 con 0x7f4828070ef0 2024-04-17T10:25:25.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f481fe00640 1 -- 172.21.15.67:0/138459920 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4820057740 con 0x7f4828070ef0 2024-04-17T10:25:25.042 INFO:teuthology.orchestra.run.smithi067.stdout:55834574858 2024-04-17T10:25:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd794166750 con 0x7fd7940cef80 2024-04-17T10:25:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1164067130 msgr2=0x7f11640695f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1164067130 0x7f11640695f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1178004880 tx=0x7f1178009040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c04a660 msgr2=0x7f117c0536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c04a660 0x7f117c0536e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f117000e810 tx=0x7f117000ecd0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 shutdown_connections 2024-04-17T10:25:25.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1164067130 0x7f11640695f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c053c20 0x7f117c177fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c04a660 0x7f117c0536e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 --2- 172.21.15.67:0/1331845255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f116c0073e0 0x7f117c0da680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 >> 172.21.15.67:0/1331845255 conn(0x7f117c054ef0 msgr2=0x7f117c038170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7fd78a000640 1 -- 172.21.15.67:0/726210271 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd78c016520 con 0x7fd7940cef80 2024-04-17T10:25:25.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 shutdown_connections 2024-04-17T10:25:25.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.047+0000 7f1189200640 1 -- 172.21.15.67:0/1331845255 wait complete. 2024-04-17T10:25:25.045 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4808067030 msgr2=0x7f48080694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4808067030 0x7f48080694f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f48140097c0 tx=0x7f48140a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4828070ef0 msgr2=0x7f4828159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4828070ef0 0x7f4828159be0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4820007e40 tx=0x7f482000e4b0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 shutdown_connections 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4808067030 0x7f48080694f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48280d0950 0x7f4828203600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4828099f90 0x7f48281717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 --2- 172.21.15.67:0/138459920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4828070ef0 0x7f4828159be0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 >> 172.21.15.67:0/138459920 conn(0x7f48280d1610 msgr2=0x7f48280cf8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 shutdown_connections 2024-04-17T10:25:25.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7f482dc00640 1 -- 172.21.15.67:0/138459920 wait complete. 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd774067060 msgr2=0x7fd774069520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd774067060 0x7fd774069520 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd78400d940 tx=0x7fd784008040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7940cef80 msgr2=0x7fd79415a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7940cef80 0x7fd79415a1a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd78c01d1d0 tx=0x7fd78c0196d0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 shutdown_connections 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd774067060 0x7fd774069520 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd7940cef80 0x7fd79415a1a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7940b4260 0x7fd794143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 --2- 172.21.15.67:0/726210271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd794071310 0x7fd7941436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 >> 172.21.15.67:0/726210271 conn(0x7fd79406ac90 msgr2=0x7fd794162f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 shutdown_connections 2024-04-17T10:25:25.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.051+0000 7fd798a00640 1 -- 172.21.15.67:0/726210271 wait complete. 2024-04-17T10:25:25.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.055+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5f78074e30 con 0x7f5f78159950 2024-04-17T10:25:25.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.055+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7facd4074e30 con 0x7facd4159bb0 2024-04-17T10:25:25.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.055+0000 7f5f6ea00640 1 -- 172.21.15.67:0/3084313277 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f600f38e0 con 0x7f5f78159950 2024-04-17T10:25:25.051 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:25.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7faccaa00640 1 -- 172.21.15.67:0/4012577471 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7facc005a160 con 0x7facd4159bb0 2024-04-17T10:25:25.054 INFO:teuthology.orchestra.run.smithi067.stdout:55834574859 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f54066fb0 msgr2=0x7f5f54069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f54066fb0 0x7f5f54069470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f6800af80 tx=0x7f5f68008040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 msgr2=0x7f5f7813b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f7813b430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5f780b48c0 tx=0x7f5f6001bab0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 shutdown_connections 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f54066fb0 0x7f5f54069470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7815e1f0 0x7f5f78157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f78159950 0x7f5f7813b430 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 --2- 172.21.15.67:0/3084313277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7806b7b0 0x7f5f7815c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 >> 172.21.15.67:0/3084313277 conn(0x7f5f7806ac90 msgr2=0x7f5f7815da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.059+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 shutdown_connections 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facb4066fb0 msgr2=0x7facb4069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facb4066fb0 0x7facb4069470 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7facc40011f0 tx=0x7facc4009040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facd4159bb0 msgr2=0x7facd415af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facd4159bb0 0x7facd415af80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7facc000e870 tx=0x7facc000ed30 comp rx=0 tx=0).stop 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 shutdown_connections 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facb4066fb0 0x7facb4069470 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7facd415e1f0 0x7facd415b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7facd4159bb0 0x7facd415af80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 --2- 172.21.15.67:0/4012577471 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7facd406b7b0 0x7facd415a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 >> 172.21.15.67:0/4012577471 conn(0x7facd406ac90 msgr2=0x7facd406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7f5f7d800640 1 -- 172.21.15.67:0/3084313277 wait complete. 2024-04-17T10:25:25.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 shutdown_connections 2024-04-17T10:25:25.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.063+0000 7facd9400640 1 -- 172.21.15.67:0/4012577471 wait complete. 2024-04-17T10:25:25.064 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.6 2024-04-17T10:25:25.064 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.075+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbdb0005860 con 0x7fbdd80b48a0 2024-04-17T10:25:25.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.075+0000 7fbdcf400640 1 -- 172.21.15.67:0/1725326259 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbdc8054810 con 0x7fbdd80b48a0 2024-04-17T10:25:25.072 INFO:teuthology.orchestra.run.smithi067.stdout:55834574860 2024-04-17T10:25:25.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdb8066f60 msgr2=0x7fbdb8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdb8066f60 0x7fbdb8069420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbdc40ad000 tx=0x7fbdc40a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdd80b48a0 msgr2=0x7fbdd80b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdd80b48a0 0x7fbdd80b1030 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c550 tx=0x7fbdc800ca10 comp rx=0 tx=0).stop 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 shutdown_connections 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdb8066f60 0x7fbdb8069420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbdd80b48a0 0x7fbdd80b1030 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbdd80b2660 0x7fbdd8149940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 --2- 172.21.15.67:0/1725326259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbdd806b3a0 0x7fbdd80af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 >> 172.21.15.67:0/1725326259 conn(0x7fbdd8066e10 msgr2=0x7fbdd80cb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 shutdown_connections 2024-04-17T10:25:25.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.079+0000 7fbdccc00640 1 -- 172.21.15.67:0/1725326259 wait complete. 2024-04-17T10:25:25.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.0 2024-04-17T10:25:25.080 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.083 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-04-17T10:25:25.084 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.092 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.3 2024-04-17T10:25:25.093 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.093 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-04-17T10:25:25.093 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.096 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.5 2024-04-17T10:25:25.096 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.100 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.7 2024-04-17T10:25:25.100 DEBUG:teuthology.parallel:result is None 2024-04-17T10:25:25.100 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-17T10:25:25.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:25:25.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.283+0000 7f2ed5000640 1 Processor -- start 2024-04-17T10:25:25.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.283+0000 7f2ed5000640 1 -- start start 2024-04-17T10:25:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ed5000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ed01620a0 0x7f2ed015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ed5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ed015e550 0x7f2ed00cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ed5000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed015eb50 con 0x7f2ed00ce020 2024-04-17T10:25:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ed5000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed0161650 con 0x7f2ed015e550 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ed5000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed015ee30 con 0x7f2ed01620a0 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ece800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ed015e550 0x7f2ed00cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ece800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ed015e550 0x7f2ed00cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58340/0 (socket says 172.21.15.67:58340) 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ece800640 1 -- 172.21.15.67:0/2923149791 learned_addr learned my addr 172.21.15.67:0/2923149791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ec5400640 1 --2- 172.21.15.67:0/2923149791 >> v2:172.21.15.86:3300/0 conn(0x7f2ed01620a0 0x7f2ed015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ece800640 1 -- 172.21.15.67:0/2923149791 >> v2:172.21.15.86:3300/0 conn(0x7f2ed01620a0 msgr2=0x7f2ed015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ece800640 1 --2- 172.21.15.67:0/2923149791 >> v2:172.21.15.86:3300/0 conn(0x7f2ed01620a0 0x7f2ed015e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ece800640 1 -- 172.21.15.67:0/2923149791 >> v1:172.21.15.86:6789/0 conn(0x7f2ed00ce020 legacy=0x7f2ed00ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:25.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.287+0000 7f2ece800640 1 -- 172.21.15.67:0/2923149791 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed01618a0 con 0x7f2ed015e550 2024-04-17T10:25:25.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ece800640 1 --2- 172.21.15.67:0/2923149791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed015e550 0x7f2ed00cf510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009e50 tx=0x7f2ec801c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d9c6c2a3aab9c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:25.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ecd400640 1 -- 172.21.15.67:0/2923149791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80a5860 con 0x7f2ed015e550 2024-04-17T10:25:25.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ecd400640 1 -- 172.21.15.67:0/2923149791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec80a5a00 con 0x7f2ed015e550 2024-04-17T10:25:25.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ecd400640 1 -- 172.21.15.67:0/2923149791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80a5dd0 con 0x7f2ed015e550 2024-04-17T10:25:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 -- 172.21.15.67:0/2923149791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed015e550 msgr2=0x7f2ed00cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2923149791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed015e550 0x7f2ed00cf510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009e50 tx=0x7f2ec801c980 comp rx=0 tx=0).stop 2024-04-17T10:25:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 -- 172.21.15.67:0/2923149791 shutdown_connections 2024-04-17T10:25:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2923149791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed015e550 0x7f2ed00cf510 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2923149791 >> v2:172.21.15.86:3300/0 conn(0x7f2ed01620a0 0x7f2ed015e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 -- 172.21.15.67:0/2923149791 >> 172.21.15.67:0/2923149791 conn(0x7f2ed00b33f0 msgr2=0x7f2ed00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 -- 172.21.15.67:0/2923149791 shutdown_connections 2024-04-17T10:25:25.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 -- 172.21.15.67:0/2923149791 wait complete. 2024-04-17T10:25:25.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 Processor -- start 2024-04-17T10:25:25.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.291+0000 7f2ed5000640 1 -- start start 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed00ce020 0x7f2ed0160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ed015e550 0x7f2ed0151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed01620a0 0x7f2ed0152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00c8630 con 0x7f2ed01620a0 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00c84b0 con 0x7f2ed00ce020 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed016a9f0 con 0x7f2ed015e550 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed01620a0 0x7f2ed0152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed01620a0 0x7f2ed0152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58352/0 (socket says 172.21.15.67:58352) 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 -- 172.21.15.67:0/2509880189 learned_addr learned my addr 172.21.15.67:0/2509880189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:25.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ec5400640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ed015e550 0x7f2ed0151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ecde00640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed00ce020 0x7f2ed0160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 -- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ed015e550 msgr2=0x7f2ed0151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ed015e550 0x7f2ed0151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 -- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed00ce020 msgr2=0x7f2ed0160580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed00ce020 0x7f2ed0160580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 -- 172.21.15.67:0/2509880189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0153430 con 0x7f2ed01620a0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ecde00640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed00ce020 0x7f2ed0160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ece800640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed01620a0 0x7f2ed0152470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009e50 tx=0x7f2ec8019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ec6a00640 1 -- 172.21.15.67:0/2509880189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80bc020 con 0x7f2ed01620a0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ec6a00640 1 -- 172.21.15.67:0/2509880189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec80b8670 con 0x7f2ed01620a0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ec6a00640 1 -- 172.21.15.67:0/2509880189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80b8a60 con 0x7f2ed01620a0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed0153710 con 0x7f2ed01620a0 2024-04-17T10:25:25.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed0153cd0 con 0x7f2ed01620a0 2024-04-17T10:25:25.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed0070ab0 con 0x7f2ed01620a0 2024-04-17T10:25:25.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.295+0000 7f2ec6a00640 1 -- 172.21.15.67:0/2509880189 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2ec80bf020 con 0x7f2ed01620a0 2024-04-17T10:25:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.299+0000 7f2ec6a00640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2eac066f60 0x7f2eac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.299+0000 7f2ec6a00640 1 -- 172.21.15.67:0/2509880189 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2ec8120690 con 0x7f2ed01620a0 2024-04-17T10:25:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.299+0000 7f2ecde00640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2eac066f60 0x7f2eac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.299+0000 7f2ecde00640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2eac066f60 0x7f2eac069420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0001760 tx=0x7f2ec0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:25.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.303+0000 7f2ec6a00640 1 -- 172.21.15.67:0/2509880189 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2ec80c6030 con 0x7f2ed01620a0 2024-04-17T10:25:25.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.459+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ed0074e30 con 0x7f2eac066f60 2024-04-17T10:25:25.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.467+0000 7f2ec6a00640 1 -- 172.21.15.67:0/2509880189 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332537 (secure 0 0 0) 0x7f2ed0074e30 con 0x7f2eac066f60 2024-04-17T10:25:25.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:25.469 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2eac066f60 msgr2=0x7f2eac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2eac066f60 0x7f2eac069420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0001760 tx=0x7f2ec0009040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed01620a0 msgr2=0x7f2ed0152470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed01620a0 0x7f2ed0152470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8009e50 tx=0x7f2ec8019720 comp rx=0 tx=0).stop 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 shutdown_connections 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2eac066f60 0x7f2eac069420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ed01620a0 0x7f2ed0152470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ed015e550 0x7f2ed0151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 --2- 172.21.15.67:0/2509880189 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ed00ce020 0x7f2ed0160580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 >> 172.21.15.67:0/2509880189 conn(0x7f2ed00b33f0 msgr2=0x7f2ed0154ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 shutdown_connections 2024-04-17T10:25:25.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.471+0000 7f2ed5000640 1 -- 172.21.15.67:0/2509880189 wait complete. 2024-04-17T10:25:25.486 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-17T10:25:24.141029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003106"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155309+0000","last_change":"2024-04-17T10:24:54.602648+0000","last_active":"2024-04-17T10:25:03.155309+0000","last_peered":"2024-04-17T10:25:03.155309+0000","last_clean":"2024-04-17T10:25:03.155309+0000","last_became_active":"2024-04-17T10:24:54.602381+0000","last_became_peered":"2024-04-17T10:24:54.602381+0000","last_unstale":"2024-04-17T10:25:03.155309+0000","last_undegraded":"2024-04-17T10:25:03.155309+0000","last_fullsized":"2024-04-17T10:25:03.155309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638270+0000","last_change":"2024-04-17T10:24:55.617102+0000","last_active":"2024-04-17T10:25:01.638270+0000","last_peered":"2024-04-17T10:25:01.638270+0000","last_clean":"2024-04-17T10:25:01.638270+0000","last_became_active":"2024-04-17T10:24:55.615401+0000","last_became_peered":"2024-04-17T10:24:55.615401+0000","last_unstale":"2024-04-17T10:25:01.638270+0000","last_undegraded":"2024-04-17T10:25:01.638270+0000","last_fullsized":"2024-04-17T10:25:01.638270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637780+0000","last_change":"2024-04-17T10:25:00.246846+0000","last_active":"2024-04-17T10:25:01.637780+0000","last_peered":"2024-04-17T10:25:01.637780+0000","last_clean":"2024-04-17T10:25:01.637780+0000","last_became_active":"2024-04-17T10:25:00.246379+0000","last_became_peered":"2024-04-17T10:25:00.246379+0000","last_unstale":"2024-04-17T10:25:01.637780+0000","last_undegraded":"2024-04-17T10:25:01.637780+0000","last_fullsized":"2024-04-17T10:25:01.637780+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:55:05.249462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638369+0000","last_change":"2024-04-17T10:24:54.607859+0000","last_active":"2024-04-17T10:25:01.638369+0000","last_peered":"2024-04-17T10:25:01.638369+0000","last_clean":"2024-04-17T10:25:01.638369+0000","last_became_active":"2024-04-17T10:24:54.607576+0000","last_became_peered":"2024-04-17T10:24:54.607576+0000","last_unstale":"2024-04-17T10:25:01.638369+0000","last_undegraded":"2024-04-17T10:25:01.638369+0000","last_fullsized":"2024-04-17T10:25:01.638369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156289+0000","last_change":"2024-04-17T10:24:55.628197+0000","last_active":"2024-04-17T10:25:03.156289+0000","last_peered":"2024-04-17T10:25:03.156289+0000","last_clean":"2024-04-17T10:25:03.156289+0000","last_became_active":"2024-04-17T10:24:55.627976+0000","last_became_peered":"2024-04-17T10:24:55.627976+0000","last_unstale":"2024-04-17T10:25:03.156289+0000","last_undegraded":"2024-04-17T10:25:03.156289+0000","last_fullsized":"2024-04-17T10:25:03.156289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636562+0000","last_change":"2024-04-17T10:24:59.651727+0000","last_active":"2024-04-17T10:25:01.636562+0000","last_peered":"2024-04-17T10:25:01.636562+0000","last_clean":"2024-04-17T10:25:01.636562+0000","last_became_active":"2024-04-17T10:24:59.651358+0000","last_became_peered":"2024-04-17T10:24:59.651358+0000","last_unstale":"2024-04-17T10:25:01.636562+0000","last_undegraded":"2024-04-17T10:25:01.636562+0000","last_fullsized":"2024-04-17T10:25:01.636562+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423903+0000","last_change":"2024-04-17T10:24:54.611808+0000","last_active":"2024-04-17T10:25:02.423903+0000","last_peered":"2024-04-17T10:25:02.423903+0000","last_clean":"2024-04-17T10:25:02.423903+0000","last_became_active":"2024-04-17T10:24:54.610903+0000","last_became_peered":"2024-04-17T10:24:54.610903+0000","last_unstale":"2024-04-17T10:25:02.423903+0000","last_undegraded":"2024-04-17T10:25:02.423903+0000","last_fullsized":"2024-04-17T10:25:02.423903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:09.085810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268578+0000","last_change":"2024-04-17T10:24:55.618364+0000","last_active":"2024-04-17T10:25:03.268578+0000","last_peered":"2024-04-17T10:25:03.268578+0000","last_clean":"2024-04-17T10:25:03.268578+0000","last_became_active":"2024-04-17T10:24:55.618013+0000","last_became_peered":"2024-04-17T10:24:55.618013+0000","last_unstale":"2024-04-17T10:25:03.268578+0000","last_undegraded":"2024-04-17T10:25:03.268578+0000","last_fullsized":"2024-04-17T10:25:03.268578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636861+0000","last_change":"2024-04-17T10:24:59.647051+0000","last_active":"2024-04-17T10:25:01.636861+0000","last_peered":"2024-04-17T10:25:01.636861+0000","last_clean":"2024-04-17T10:25:01.636861+0000","last_became_active":"2024-04-17T10:24:59.646611+0000","last_became_peered":"2024-04-17T10:24:59.646611+0000","last_unstale":"2024-04-17T10:25:01.636861+0000","last_undegraded":"2024-04-17T10:25:01.636861+0000","last_fullsized":"2024-04-17T10:25:01.636861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423712+0000","last_change":"2024-04-17T10:24:54.611301+0000","last_active":"2024-04-17T10:25:02.423712+0000","last_peered":"2024-04-17T10:25:02.423712+0000","last_clean":"2024-04-17T10:25:02.423712+0000","last_became_active":"2024-04-17T10:24:54.610243+0000","last_became_peered":"2024-04-17T10:24:54.610243+0000","last_unstale":"2024-04-17T10:25:02.423712+0000","last_undegraded":"2024-04-17T10:25:02.423712+0000","last_fullsized":"2024-04-17T10:25:02.423712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:36.054430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268618+0000","last_change":"2024-04-17T10:24:55.618112+0000","last_active":"2024-04-17T10:25:03.268618+0000","last_peered":"2024-04-17T10:25:03.268618+0000","last_clean":"2024-04-17T10:25:03.268618+0000","last_became_active":"2024-04-17T10:24:55.617127+0000","last_became_peered":"2024-04-17T10:24:55.617127+0000","last_unstale":"2024-04-17T10:25:03.268618+0000","last_undegraded":"2024-04-17T10:25:03.268618+0000","last_fullsized":"2024-04-17T10:25:03.268618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636698+0000","last_change":"2024-04-17T10:24:59.641046+0000","last_active":"2024-04-17T10:25:01.636698+0000","last_peered":"2024-04-17T10:25:01.636698+0000","last_clean":"2024-04-17T10:25:01.636698+0000","last_became_active":"2024-04-17T10:24:59.640622+0000","last_became_peered":"2024-04-17T10:24:59.640622+0000","last_unstale":"2024-04-17T10:25:01.636698+0000","last_undegraded":"2024-04-17T10:25:01.636698+0000","last_fullsized":"2024-04-17T10:25:01.636698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537179+0000","last_change":"2024-04-17T10:24:55.617061+0000","last_active":"2024-04-17T10:25:02.537179+0000","last_peered":"2024-04-17T10:25:02.537179+0000","last_clean":"2024-04-17T10:25:02.537179+0000","last_became_active":"2024-04-17T10:24:55.616809+0000","last_became_peered":"2024-04-17T10:24:55.616809+0000","last_unstale":"2024-04-17T10:25:02.537179+0000","last_undegraded":"2024-04-17T10:25:02.537179+0000","last_fullsized":"2024-04-17T10:25:02.537179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423815+0000","last_change":"2024-04-17T10:24:54.607974+0000","last_active":"2024-04-17T10:25:02.423815+0000","last_peered":"2024-04-17T10:25:02.423815+0000","last_clean":"2024-04-17T10:25:02.423815+0000","last_became_active":"2024-04-17T10:24:54.606303+0000","last_became_peered":"2024-04-17T10:24:54.606303+0000","last_unstale":"2024-04-17T10:25:02.423815+0000","last_undegraded":"2024-04-17T10:25:02.423815+0000","last_fullsized":"2024-04-17T10:25:02.423815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:52.381704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637318+0000","last_change":"2024-04-17T10:24:59.645993+0000","last_active":"2024-04-17T10:25:01.637318+0000","last_peered":"2024-04-17T10:25:01.637318+0000","last_clean":"2024-04-17T10:25:01.637318+0000","last_became_active":"2024-04-17T10:24:59.645305+0000","last_became_peered":"2024-04-17T10:24:59.645305+0000","last_unstale":"2024-04-17T10:25:01.637318+0000","last_undegraded":"2024-04-17T10:25:01.637318+0000","last_fullsized":"2024-04-17T10:25:01.637318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638467+0000","last_change":"2024-04-17T10:24:55.614771+0000","last_active":"2024-04-17T10:25:01.638467+0000","last_peered":"2024-04-17T10:25:01.638467+0000","last_clean":"2024-04-17T10:25:01.638467+0000","last_became_active":"2024-04-17T10:24:55.614517+0000","last_became_peered":"2024-04-17T10:24:55.614517+0000","last_unstale":"2024-04-17T10:25:01.638467+0000","last_undegraded":"2024-04-17T10:25:01.638467+0000","last_fullsized":"2024-04-17T10:25:01.638467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636723+0000","last_change":"2024-04-17T10:24:54.608749+0000","last_active":"2024-04-17T10:25:01.636723+0000","last_peered":"2024-04-17T10:25:01.636723+0000","last_clean":"2024-04-17T10:25:01.636723+0000","last_became_active":"2024-04-17T10:24:54.606617+0000","last_became_peered":"2024-04-17T10:24:54.606617+0000","last_unstale":"2024-04-17T10:25:01.636723+0000","last_undegraded":"2024-04-17T10:25:01.636723+0000","last_fullsized":"2024-04-17T10:25:01.636723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636946+0000","last_change":"2024-04-17T10:24:59.645457+0000","last_active":"2024-04-17T10:25:01.636946+0000","last_peered":"2024-04-17T10:25:01.636946+0000","last_clean":"2024-04-17T10:25:01.636946+0000","last_became_active":"2024-04-17T10:24:59.644747+0000","last_became_peered":"2024-04-17T10:24:59.644747+0000","last_unstale":"2024-04-17T10:25:01.636946+0000","last_undegraded":"2024-04-17T10:25:01.636946+0000","last_fullsized":"2024-04-17T10:25:01.636946+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637820+0000","last_change":"2024-04-17T10:24:55.630638+0000","last_active":"2024-04-17T10:25:01.637820+0000","last_peered":"2024-04-17T10:25:01.637820+0000","last_clean":"2024-04-17T10:25:01.637820+0000","last_became_active":"2024-04-17T10:24:55.630351+0000","last_became_peered":"2024-04-17T10:24:55.630351+0000","last_unstale":"2024-04-17T10:25:01.637820+0000","last_undegraded":"2024-04-17T10:25:01.637820+0000","last_fullsized":"2024-04-17T10:25:01.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636842+0000","last_change":"2024-04-17T10:24:54.609635+0000","last_active":"2024-04-17T10:25:01.636842+0000","last_peered":"2024-04-17T10:25:01.636842+0000","last_clean":"2024-04-17T10:25:01.636842+0000","last_became_active":"2024-04-17T10:24:54.607298+0000","last_became_peered":"2024-04-17T10:24:54.607298+0000","last_unstale":"2024-04-17T10:25:01.636842+0000","last_undegraded":"2024-04-17T10:25:01.636842+0000","last_fullsized":"2024-04-17T10:25:01.636842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637200+0000","last_change":"2024-04-17T10:25:00.243112+0000","last_active":"2024-04-17T10:25:01.637200+0000","last_peered":"2024-04-17T10:25:01.637200+0000","last_clean":"2024-04-17T10:25:01.637200+0000","last_became_active":"2024-04-17T10:25:00.242669+0000","last_became_peered":"2024-04-17T10:25:00.242669+0000","last_unstale":"2024-04-17T10:25:01.637200+0000","last_undegraded":"2024-04-17T10:25:01.637200+0000","last_fullsized":"2024-04-17T10:25:01.637200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637134+0000","last_change":"2024-04-17T10:24:55.608101+0000","last_active":"2024-04-17T10:25:01.637134+0000","last_peered":"2024-04-17T10:25:01.637134+0000","last_clean":"2024-04-17T10:25:01.637134+0000","last_became_active":"2024-04-17T10:24:55.607668+0000","last_became_peered":"2024-04-17T10:24:55.607668+0000","last_unstale":"2024-04-17T10:25:01.637134+0000","last_undegraded":"2024-04-17T10:25:01.637134+0000","last_fullsized":"2024-04-17T10:25:01.637134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:35.913082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423205+0000","last_change":"2024-04-17T10:24:54.607981+0000","last_active":"2024-04-17T10:25:02.423205+0000","last_peered":"2024-04-17T10:25:02.423205+0000","last_clean":"2024-04-17T10:25:02.423205+0000","last_became_active":"2024-04-17T10:24:54.607150+0000","last_became_peered":"2024-04-17T10:24:54.607150+0000","last_unstale":"2024-04-17T10:25:02.423205+0000","last_undegraded":"2024-04-17T10:25:02.423205+0000","last_fullsized":"2024-04-17T10:25:02.423205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636873+0000","last_change":"2024-04-17T10:24:59.645029+0000","last_active":"2024-04-17T10:25:01.636873+0000","last_peered":"2024-04-17T10:25:01.636873+0000","last_clean":"2024-04-17T10:25:01.636873+0000","last_became_active":"2024-04-17T10:24:59.644540+0000","last_became_peered":"2024-04-17T10:24:59.644540+0000","last_unstale":"2024-04-17T10:25:01.636873+0000","last_undegraded":"2024-04-17T10:25:01.636873+0000","last_fullsized":"2024-04-17T10:25:01.636873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156445+0000","last_change":"2024-04-17T10:24:55.610920+0000","last_active":"2024-04-17T10:25:03.156445+0000","last_peered":"2024-04-17T10:25:03.156445+0000","last_clean":"2024-04-17T10:25:03.156445+0000","last_became_active":"2024-04-17T10:24:55.610566+0000","last_became_peered":"2024-04-17T10:24:55.610566+0000","last_unstale":"2024-04-17T10:25:03.156445+0000","last_undegraded":"2024-04-17T10:25:03.156445+0000","last_fullsized":"2024-04-17T10:25:03.156445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537952+0000","last_change":"2024-04-17T10:24:54.609866+0000","last_active":"2024-04-17T10:25:02.537952+0000","last_peered":"2024-04-17T10:25:02.537952+0000","last_clean":"2024-04-17T10:25:02.537952+0000","last_became_active":"2024-04-17T10:24:54.607310+0000","last_became_peered":"2024-04-17T10:24:54.607310+0000","last_unstale":"2024-04-17T10:25:02.537952+0000","last_undegraded":"2024-04-17T10:25:02.537952+0000","last_fullsized":"2024-04-17T10:25:02.537952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637020+0000","last_change":"2024-04-17T10:25:00.242151+0000","last_active":"2024-04-17T10:25:01.637020+0000","last_peered":"2024-04-17T10:25:01.637020+0000","last_clean":"2024-04-17T10:25:01.637020+0000","last_became_active":"2024-04-17T10:25:00.241789+0000","last_became_peered":"2024-04-17T10:25:00.241789+0000","last_unstale":"2024-04-17T10:25:01.637020+0000","last_undegraded":"2024-04-17T10:25:01.637020+0000","last_fullsized":"2024-04-17T10:25:01.637020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638591+0000","last_change":"2024-04-17T10:24:55.611037+0000","last_active":"2024-04-17T10:25:01.638591+0000","last_peered":"2024-04-17T10:25:01.638591+0000","last_clean":"2024-04-17T10:25:01.638591+0000","last_became_active":"2024-04-17T10:24:55.610713+0000","last_became_peered":"2024-04-17T10:24:55.610713+0000","last_unstale":"2024-04-17T10:25:01.638591+0000","last_undegraded":"2024-04-17T10:25:01.638591+0000","last_fullsized":"2024-04-17T10:25:01.638591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636990+0000","last_change":"2024-04-17T10:24:54.614299+0000","last_active":"2024-04-17T10:25:01.636990+0000","last_peered":"2024-04-17T10:25:01.636990+0000","last_clean":"2024-04-17T10:25:01.636990+0000","last_became_active":"2024-04-17T10:24:54.613957+0000","last_became_peered":"2024-04-17T10:24:54.613957+0000","last_unstale":"2024-04-17T10:25:01.636990+0000","last_undegraded":"2024-04-17T10:25:01.636990+0000","last_fullsized":"2024-04-17T10:25:01.636990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637684+0000","last_change":"2024-04-17T10:25:00.241327+0000","last_active":"2024-04-17T10:25:01.637684+0000","last_peered":"2024-04-17T10:25:01.637684+0000","last_clean":"2024-04-17T10:25:01.637684+0000","last_became_active":"2024-04-17T10:25:00.240884+0000","last_became_peered":"2024-04-17T10:25:00.240884+0000","last_unstale":"2024-04-17T10:25:01.637684+0000","last_undegraded":"2024-04-17T10:25:01.637684+0000","last_fullsized":"2024-04-17T10:25:01.637684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:46.933770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.624025+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.623725+0000","last_became_peered":"2024-04-17T10:24:55.623725+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424055+0000","last_change":"2024-04-17T10:24:54.610613+0000","last_active":"2024-04-17T10:25:02.424055+0000","last_peered":"2024-04-17T10:25:02.424055+0000","last_clean":"2024-04-17T10:25:02.424055+0000","last_became_active":"2024-04-17T10:24:54.609405+0000","last_became_peered":"2024-04-17T10:24:54.609405+0000","last_unstale":"2024-04-17T10:25:02.424055+0000","last_undegraded":"2024-04-17T10:25:02.424055+0000","last_fullsized":"2024-04-17T10:25:02.424055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:06.248042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155827+0000","last_change":"2024-04-17T10:25:00.243396+0000","last_active":"2024-04-17T10:25:03.155827+0000","last_peered":"2024-04-17T10:25:03.155827+0000","last_clean":"2024-04-17T10:25:03.155827+0000","last_became_active":"2024-04-17T10:25:00.243086+0000","last_became_peered":"2024-04-17T10:25:00.243086+0000","last_unstale":"2024-04-17T10:25:03.155827+0000","last_undegraded":"2024-04-17T10:25:03.155827+0000","last_fullsized":"2024-04-17T10:25:03.155827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268735+0000","last_change":"2024-04-17T10:24:55.611872+0000","last_active":"2024-04-17T10:25:03.268735+0000","last_peered":"2024-04-17T10:25:03.268735+0000","last_clean":"2024-04-17T10:25:03.268735+0000","last_became_active":"2024-04-17T10:24:55.607201+0000","last_became_peered":"2024-04-17T10:24:55.607201+0000","last_unstale":"2024-04-17T10:25:03.268735+0000","last_undegraded":"2024-04-17T10:25:03.268735+0000","last_fullsized":"2024-04-17T10:25:03.268735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637222+0000","last_change":"2024-04-17T10:24:54.598254+0000","last_active":"2024-04-17T10:25:01.637222+0000","last_peered":"2024-04-17T10:25:01.637222+0000","last_clean":"2024-04-17T10:25:01.637222+0000","last_became_active":"2024-04-17T10:24:54.597869+0000","last_became_peered":"2024-04-17T10:24:54.597869+0000","last_unstale":"2024-04-17T10:25:01.637222+0000","last_undegraded":"2024-04-17T10:25:01.637222+0000","last_fullsized":"2024-04-17T10:25:01.637222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636567+0000","last_change":"2024-04-17T10:24:59.656986+0000","last_active":"2024-04-17T10:25:01.636567+0000","last_peered":"2024-04-17T10:25:01.636567+0000","last_clean":"2024-04-17T10:25:01.636567+0000","last_became_active":"2024-04-17T10:24:59.656124+0000","last_became_peered":"2024-04-17T10:24:59.656124+0000","last_unstale":"2024-04-17T10:25:01.636567+0000","last_undegraded":"2024-04-17T10:25:01.636567+0000","last_fullsized":"2024-04-17T10:25:01.636567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637349+0000","last_change":"2024-04-17T10:24:55.625808+0000","last_active":"2024-04-17T10:25:01.637349+0000","last_peered":"2024-04-17T10:25:01.637349+0000","last_clean":"2024-04-17T10:25:01.637349+0000","last_became_active":"2024-04-17T10:24:55.625551+0000","last_became_peered":"2024-04-17T10:24:55.625551+0000","last_unstale":"2024-04-17T10:25:01.637349+0000","last_undegraded":"2024-04-17T10:25:01.637349+0000","last_fullsized":"2024-04-17T10:25:01.637349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267481+0000","last_change":"2024-04-17T10:24:54.603615+0000","last_active":"2024-04-17T10:25:03.267481+0000","last_peered":"2024-04-17T10:25:03.267481+0000","last_clean":"2024-04-17T10:25:03.267481+0000","last_became_active":"2024-04-17T10:24:54.600774+0000","last_became_peered":"2024-04-17T10:24:54.600774+0000","last_unstale":"2024-04-17T10:25:03.267481+0000","last_undegraded":"2024-04-17T10:25:03.267481+0000","last_fullsized":"2024-04-17T10:25:03.267481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637164+0000","last_change":"2024-04-17T10:24:59.646484+0000","last_active":"2024-04-17T10:25:01.637164+0000","last_peered":"2024-04-17T10:25:01.637164+0000","last_clean":"2024-04-17T10:25:01.637164+0000","last_became_active":"2024-04-17T10:24:59.646061+0000","last_became_peered":"2024-04-17T10:24:59.646061+0000","last_unstale":"2024-04-17T10:25:01.637164+0000","last_undegraded":"2024-04-17T10:25:01.637164+0000","last_fullsized":"2024-04-17T10:25:01.637164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636232+0000","last_change":"2024-04-17T10:24:55.624915+0000","last_active":"2024-04-17T10:25:01.636232+0000","last_peered":"2024-04-17T10:25:01.636232+0000","last_clean":"2024-04-17T10:25:01.636232+0000","last_became_active":"2024-04-17T10:24:55.624230+0000","last_became_peered":"2024-04-17T10:24:55.624230+0000","last_unstale":"2024-04-17T10:25:01.636232+0000","last_undegraded":"2024-04-17T10:25:01.636232+0000","last_fullsized":"2024-04-17T10:25:01.636232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624893+0000","last_change":"2024-04-17T10:24:54.599766+0000","last_active":"2024-04-17T10:25:02.624893+0000","last_peered":"2024-04-17T10:25:02.624893+0000","last_clean":"2024-04-17T10:25:02.624893+0000","last_became_active":"2024-04-17T10:24:54.599476+0000","last_became_peered":"2024-04-17T10:24:54.599476+0000","last_unstale":"2024-04-17T10:25:02.624893+0000","last_undegraded":"2024-04-17T10:25:02.624893+0000","last_fullsized":"2024-04-17T10:25:02.624893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637263+0000","last_change":"2024-04-17T10:24:59.656597+0000","last_active":"2024-04-17T10:25:01.637263+0000","last_peered":"2024-04-17T10:25:01.637263+0000","last_clean":"2024-04-17T10:25:01.637263+0000","last_became_active":"2024-04-17T10:24:59.656135+0000","last_became_peered":"2024-04-17T10:24:59.656135+0000","last_unstale":"2024-04-17T10:25:01.637263+0000","last_undegraded":"2024-04-17T10:25:01.637263+0000","last_fullsized":"2024-04-17T10:25:01.637263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155614+0000","last_change":"2024-04-17T10:24:55.591986+0000","last_active":"2024-04-17T10:25:03.155614+0000","last_peered":"2024-04-17T10:25:03.155614+0000","last_clean":"2024-04-17T10:25:03.155614+0000","last_became_active":"2024-04-17T10:24:55.591672+0000","last_became_peered":"2024-04-17T10:24:55.591672+0000","last_unstale":"2024-04-17T10:25:03.155614+0000","last_undegraded":"2024-04-17T10:25:03.155614+0000","last_fullsized":"2024-04-17T10:25:03.155614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423263+0000","last_change":"2024-04-17T10:24:54.614030+0000","last_active":"2024-04-17T10:25:02.423263+0000","last_peered":"2024-04-17T10:25:02.423263+0000","last_clean":"2024-04-17T10:25:02.423263+0000","last_became_active":"2024-04-17T10:24:54.613752+0000","last_became_peered":"2024-04-17T10:24:54.613752+0000","last_unstale":"2024-04-17T10:25:02.423263+0000","last_undegraded":"2024-04-17T10:25:02.423263+0000","last_fullsized":"2024-04-17T10:25:02.423263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156369+0000","last_change":"2024-04-17T10:25:00.244705+0000","last_active":"2024-04-17T10:25:03.156369+0000","last_peered":"2024-04-17T10:25:03.156369+0000","last_clean":"2024-04-17T10:25:03.156369+0000","last_became_active":"2024-04-17T10:25:00.243853+0000","last_became_peered":"2024-04-17T10:25:00.243853+0000","last_unstale":"2024-04-17T10:25:03.156369+0000","last_undegraded":"2024-04-17T10:25:03.156369+0000","last_fullsized":"2024-04-17T10:25:03.156369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638050+0000","last_change":"2024-04-17T10:24:55.622500+0000","last_active":"2024-04-17T10:25:01.638050+0000","last_peered":"2024-04-17T10:25:01.638050+0000","last_clean":"2024-04-17T10:25:01.638050+0000","last_became_active":"2024-04-17T10:24:55.622206+0000","last_became_peered":"2024-04-17T10:24:55.622206+0000","last_unstale":"2024-04-17T10:25:01.638050+0000","last_undegraded":"2024-04-17T10:25:01.638050+0000","last_fullsized":"2024-04-17T10:25:01.638050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268327+0000","last_change":"2024-04-17T10:24:54.601255+0000","last_active":"2024-04-17T10:25:03.268327+0000","last_peered":"2024-04-17T10:25:03.268327+0000","last_clean":"2024-04-17T10:25:03.268327+0000","last_became_active":"2024-04-17T10:24:54.600968+0000","last_became_peered":"2024-04-17T10:24:54.600968+0000","last_unstale":"2024-04-17T10:25:03.268327+0000","last_undegraded":"2024-04-17T10:25:03.268327+0000","last_fullsized":"2024-04-17T10:25:03.268327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423358+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:25:02.423358+0000","last_peered":"2024-04-17T10:25:02.423358+0000","last_clean":"2024-04-17T10:25:02.423358+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:25:02.423358+0000","last_undegraded":"2024-04-17T10:25:02.423358+0000","last_fullsized":"2024-04-17T10:25:02.423358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537629+0000","last_change":"2024-04-17T10:25:00.247011+0000","last_active":"2024-04-17T10:25:02.537629+0000","last_peered":"2024-04-17T10:25:02.537629+0000","last_clean":"2024-04-17T10:25:02.537629+0000","last_became_active":"2024-04-17T10:25:00.245780+0000","last_became_peered":"2024-04-17T10:25:00.245780+0000","last_unstale":"2024-04-17T10:25:02.537629+0000","last_undegraded":"2024-04-17T10:25:02.537629+0000","last_fullsized":"2024-04-17T10:25:02.537629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423766+0000","last_change":"2024-04-17T10:24:55.613528+0000","last_active":"2024-04-17T10:25:02.423766+0000","last_peered":"2024-04-17T10:25:02.423766+0000","last_clean":"2024-04-17T10:25:02.423766+0000","last_became_active":"2024-04-17T10:24:55.613203+0000","last_became_peered":"2024-04-17T10:24:55.613203+0000","last_unstale":"2024-04-17T10:25:02.423766+0000","last_undegraded":"2024-04-17T10:25:02.423766+0000","last_fullsized":"2024-04-17T10:25:02.423766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:15.389666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625258+0000","last_change":"2024-04-17T10:24:54.605199+0000","last_active":"2024-04-17T10:25:02.625258+0000","last_peered":"2024-04-17T10:25:02.625258+0000","last_clean":"2024-04-17T10:25:02.625258+0000","last_became_active":"2024-04-17T10:24:54.604881+0000","last_became_peered":"2024-04-17T10:24:54.604881+0000","last_unstale":"2024-04-17T10:25:02.625258+0000","last_undegraded":"2024-04-17T10:25:02.625258+0000","last_fullsized":"2024-04-17T10:25:02.625258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537487+0000","last_change":"2024-04-17T10:25:00.242786+0000","last_active":"2024-04-17T10:25:02.537487+0000","last_peered":"2024-04-17T10:25:02.537487+0000","last_clean":"2024-04-17T10:25:02.537487+0000","last_became_active":"2024-04-17T10:25:00.242222+0000","last_became_peered":"2024-04-17T10:25:00.242222+0000","last_unstale":"2024-04-17T10:25:02.537487+0000","last_undegraded":"2024-04-17T10:25:02.537487+0000","last_fullsized":"2024-04-17T10:25:02.537487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:01.409834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268628+0000","last_change":"2024-04-17T10:24:55.606154+0000","last_active":"2024-04-17T10:25:03.268628+0000","last_peered":"2024-04-17T10:25:03.268628+0000","last_clean":"2024-04-17T10:25:03.268628+0000","last_became_active":"2024-04-17T10:24:55.605854+0000","last_became_peered":"2024-04-17T10:24:55.605854+0000","last_unstale":"2024-04-17T10:25:03.268628+0000","last_undegraded":"2024-04-17T10:25:03.268628+0000","last_fullsized":"2024-04-17T10:25:03.268628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636665+0000","last_change":"2024-04-17T10:24:54.605549+0000","last_active":"2024-04-17T10:25:01.636665+0000","last_peered":"2024-04-17T10:25:01.636665+0000","last_clean":"2024-04-17T10:25:01.636665+0000","last_became_active":"2024-04-17T10:24:54.605197+0000","last_became_peered":"2024-04-17T10:24:54.605197+0000","last_unstale":"2024-04-17T10:25:01.636665+0000","last_undegraded":"2024-04-17T10:25:01.636665+0000","last_fullsized":"2024-04-17T10:25:01.636665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155960+0000","last_change":"2024-04-17T10:24:59.652908+0000","last_active":"2024-04-17T10:25:03.155960+0000","last_peered":"2024-04-17T10:25:03.155960+0000","last_clean":"2024-04-17T10:25:03.155960+0000","last_became_active":"2024-04-17T10:24:59.652604+0000","last_became_peered":"2024-04-17T10:24:59.652604+0000","last_unstale":"2024-04-17T10:25:03.155960+0000","last_undegraded":"2024-04-17T10:25:03.155960+0000","last_fullsized":"2024-04-17T10:25:03.155960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424244+0000","last_change":"2024-04-17T10:24:55.612380+0000","last_active":"2024-04-17T10:25:02.424244+0000","last_peered":"2024-04-17T10:25:02.424244+0000","last_clean":"2024-04-17T10:25:02.424244+0000","last_became_active":"2024-04-17T10:24:55.611511+0000","last_became_peered":"2024-04-17T10:24:55.611511+0000","last_unstale":"2024-04-17T10:25:02.424244+0000","last_undegraded":"2024-04-17T10:25:02.424244+0000","last_fullsized":"2024-04-17T10:25:02.424244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:49.176910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624922+0000","last_change":"2024-04-17T10:24:54.601414+0000","last_active":"2024-04-17T10:25:02.624922+0000","last_peered":"2024-04-17T10:25:02.624922+0000","last_clean":"2024-04-17T10:25:02.624922+0000","last_became_active":"2024-04-17T10:24:54.600297+0000","last_became_peered":"2024-04-17T10:24:54.600297+0000","last_unstale":"2024-04-17T10:25:02.624922+0000","last_undegraded":"2024-04-17T10:25:02.624922+0000","last_fullsized":"2024-04-17T10:25:02.624922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267910+0000","last_change":"2024-04-17T10:25:00.242716+0000","last_active":"2024-04-17T10:25:03.267910+0000","last_peered":"2024-04-17T10:25:03.267910+0000","last_clean":"2024-04-17T10:25:03.267910+0000","last_became_active":"2024-04-17T10:25:00.242357+0000","last_became_peered":"2024-04-17T10:25:00.242357+0000","last_unstale":"2024-04-17T10:25:03.267910+0000","last_undegraded":"2024-04-17T10:25:03.267910+0000","last_fullsized":"2024-04-17T10:25:03.267910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424088+0000","last_change":"2024-04-17T10:24:55.604575+0000","last_active":"2024-04-17T10:25:02.424088+0000","last_peered":"2024-04-17T10:25:02.424088+0000","last_clean":"2024-04-17T10:25:02.424088+0000","last_became_active":"2024-04-17T10:24:55.604252+0000","last_became_peered":"2024-04-17T10:24:55.604252+0000","last_unstale":"2024-04-17T10:25:02.424088+0000","last_undegraded":"2024-04-17T10:25:02.424088+0000","last_fullsized":"2024-04-17T10:25:02.424088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:02.059745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637103+0000","last_change":"2024-04-17T10:24:54.595839+0000","last_active":"2024-04-17T10:25:01.637103+0000","last_peered":"2024-04-17T10:25:01.637103+0000","last_clean":"2024-04-17T10:25:01.637103+0000","last_became_active":"2024-04-17T10:24:54.594547+0000","last_became_peered":"2024-04-17T10:24:54.594547+0000","last_unstale":"2024-04-17T10:25:01.637103+0000","last_undegraded":"2024-04-17T10:25:01.637103+0000","last_fullsized":"2024-04-17T10:25:01.637103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636525+0000","last_change":"2024-04-17T10:24:59.644365+0000","last_active":"2024-04-17T10:25:01.636525+0000","last_peered":"2024-04-17T10:25:01.636525+0000","last_clean":"2024-04-17T10:25:01.636525+0000","last_became_active":"2024-04-17T10:24:59.643969+0000","last_became_peered":"2024-04-17T10:24:59.643969+0000","last_unstale":"2024-04-17T10:25:01.636525+0000","last_undegraded":"2024-04-17T10:25:01.636525+0000","last_fullsized":"2024-04-17T10:25:01.636525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423432+0000","last_change":"2024-04-17T10:24:55.598627+0000","last_active":"2024-04-17T10:25:02.423432+0000","last_peered":"2024-04-17T10:25:02.423432+0000","last_clean":"2024-04-17T10:25:02.423432+0000","last_became_active":"2024-04-17T10:24:55.598319+0000","last_became_peered":"2024-04-17T10:24:55.598319+0000","last_unstale":"2024-04-17T10:25:02.423432+0000","last_undegraded":"2024-04-17T10:25:02.423432+0000","last_fullsized":"2024-04-17T10:25:02.423432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624900+0000","last_change":"2024-04-17T10:24:54.600358+0000","last_active":"2024-04-17T10:25:02.624900+0000","last_peered":"2024-04-17T10:25:02.624900+0000","last_clean":"2024-04-17T10:25:02.624900+0000","last_became_active":"2024-04-17T10:24:54.600064+0000","last_became_peered":"2024-04-17T10:24:54.600064+0000","last_unstale":"2024-04-17T10:25:02.624900+0000","last_undegraded":"2024-04-17T10:25:02.624900+0000","last_fullsized":"2024-04-17T10:25:02.624900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156321+0000","last_change":"2024-04-17T10:24:59.651264+0000","last_active":"2024-04-17T10:25:03.156321+0000","last_peered":"2024-04-17T10:25:03.156321+0000","last_clean":"2024-04-17T10:25:03.156321+0000","last_became_active":"2024-04-17T10:24:59.650654+0000","last_became_peered":"2024-04-17T10:24:59.650654+0000","last_unstale":"2024-04-17T10:25:03.156321+0000","last_undegraded":"2024-04-17T10:25:03.156321+0000","last_fullsized":"2024-04-17T10:25:03.156321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537254+0000","last_change":"2024-04-17T10:24:55.608832+0000","last_active":"2024-04-17T10:25:02.537254+0000","last_peered":"2024-04-17T10:25:02.537254+0000","last_clean":"2024-04-17T10:25:02.537254+0000","last_became_active":"2024-04-17T10:24:55.608402+0000","last_became_peered":"2024-04-17T10:24:55.608402+0000","last_unstale":"2024-04-17T10:25:02.537254+0000","last_undegraded":"2024-04-17T10:25:02.537254+0000","last_fullsized":"2024-04-17T10:25:02.537254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537354+0000","last_change":"2024-04-17T10:24:54.601967+0000","last_active":"2024-04-17T10:25:02.537354+0000","last_peered":"2024-04-17T10:25:02.537354+0000","last_clean":"2024-04-17T10:25:02.537354+0000","last_became_active":"2024-04-17T10:24:54.601643+0000","last_became_peered":"2024-04-17T10:24:54.601643+0000","last_unstale":"2024-04-17T10:25:02.537354+0000","last_undegraded":"2024-04-17T10:25:02.537354+0000","last_fullsized":"2024-04-17T10:25:02.537354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638493+0000","last_change":"2024-04-17T10:25:00.243903+0000","last_active":"2024-04-17T10:25:01.638493+0000","last_peered":"2024-04-17T10:25:01.638493+0000","last_clean":"2024-04-17T10:25:01.638493+0000","last_became_active":"2024-04-17T10:25:00.243126+0000","last_became_peered":"2024-04-17T10:25:00.243126+0000","last_unstale":"2024-04-17T10:25:01.638493+0000","last_undegraded":"2024-04-17T10:25:01.638493+0000","last_fullsized":"2024-04-17T10:25:01.638493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:44.715063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423617+0000","last_change":"2024-04-17T10:24:55.598632+0000","last_active":"2024-04-17T10:25:02.423617+0000","last_peered":"2024-04-17T10:25:02.423617+0000","last_clean":"2024-04-17T10:25:02.423617+0000","last_became_active":"2024-04-17T10:24:55.598302+0000","last_became_peered":"2024-04-17T10:24:55.598302+0000","last_unstale":"2024-04-17T10:25:02.423617+0000","last_undegraded":"2024-04-17T10:25:02.423617+0000","last_fullsized":"2024-04-17T10:25:02.423617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267532+0000","last_change":"2024-04-17T10:24:54.603334+0000","last_active":"2024-04-17T10:25:03.267532+0000","last_peered":"2024-04-17T10:25:03.267532+0000","last_clean":"2024-04-17T10:25:03.267532+0000","last_became_active":"2024-04-17T10:24:54.603066+0000","last_became_peered":"2024-04-17T10:24:54.603066+0000","last_unstale":"2024-04-17T10:25:03.267532+0000","last_undegraded":"2024-04-17T10:25:03.267532+0000","last_fullsized":"2024-04-17T10:25:03.267532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639048+0000","last_change":"2024-04-17T10:25:00.247479+0000","last_active":"2024-04-17T10:25:01.639048+0000","last_peered":"2024-04-17T10:25:01.639048+0000","last_clean":"2024-04-17T10:25:01.639048+0000","last_became_active":"2024-04-17T10:25:00.247066+0000","last_became_peered":"2024-04-17T10:25:00.247066+0000","last_unstale":"2024-04-17T10:25:01.639048+0000","last_undegraded":"2024-04-17T10:25:01.639048+0000","last_fullsized":"2024-04-17T10:25:01.639048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:10:54.855530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155768+0000","last_change":"2024-04-17T10:24:55.616361+0000","last_active":"2024-04-17T10:25:03.155768+0000","last_peered":"2024-04-17T10:25:03.155768+0000","last_clean":"2024-04-17T10:25:03.155768+0000","last_became_active":"2024-04-17T10:24:55.615902+0000","last_became_peered":"2024-04-17T10:24:55.615902+0000","last_unstale":"2024-04-17T10:25:03.155768+0000","last_undegraded":"2024-04-17T10:25:03.155768+0000","last_fullsized":"2024-04-17T10:25:03.155768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:53.388065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155191+0000","last_change":"2024-04-17T10:24:54.599143+0000","last_active":"2024-04-17T10:25:03.155191+0000","last_peered":"2024-04-17T10:25:03.155191+0000","last_clean":"2024-04-17T10:25:03.155191+0000","last_became_active":"2024-04-17T10:24:54.598831+0000","last_became_peered":"2024-04-17T10:24:54.598831+0000","last_unstale":"2024-04-17T10:25:03.155191+0000","last_undegraded":"2024-04-17T10:25:03.155191+0000","last_fullsized":"2024-04-17T10:25:03.155191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268682+0000","last_change":"2024-04-17T10:25:00.243074+0000","last_active":"2024-04-17T10:25:03.268682+0000","last_peered":"2024-04-17T10:25:03.268682+0000","last_clean":"2024-04-17T10:25:03.268682+0000","last_became_active":"2024-04-17T10:25:00.242532+0000","last_became_peered":"2024-04-17T10:25:00.242532+0000","last_unstale":"2024-04-17T10:25:03.268682+0000","last_undegraded":"2024-04-17T10:25:03.268682+0000","last_fullsized":"2024-04-17T10:25:03.268682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423646+0000","last_change":"2024-04-17T10:24:55.603471+0000","last_active":"2024-04-17T10:25:02.423646+0000","last_peered":"2024-04-17T10:25:02.423646+0000","last_clean":"2024-04-17T10:25:02.423646+0000","last_became_active":"2024-04-17T10:24:55.603177+0000","last_became_peered":"2024-04-17T10:24:55.603177+0000","last_unstale":"2024-04-17T10:25:02.423646+0000","last_undegraded":"2024-04-17T10:25:02.423646+0000","last_fullsized":"2024-04-17T10:25:02.423646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:52.938391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:10.622101+0000","last_change":"2024-04-17T10:24:54.599941+0000","last_active":"2024-04-17T10:25:10.622101+0000","last_peered":"2024-04-17T10:25:10.622101+0000","last_clean":"2024-04-17T10:25:10.622101+0000","last_became_active":"2024-04-17T10:24:54.599658+0000","last_became_peered":"2024-04-17T10:24:54.599658+0000","last_unstale":"2024-04-17T10:25:10.622101+0000","last_undegraded":"2024-04-17T10:25:10.622101+0000","last_fullsized":"2024-04-17T10:25:10.622101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637141+0000","last_change":"2024-04-17T10:24:59.657550+0000","last_active":"2024-04-17T10:25:01.637141+0000","last_peered":"2024-04-17T10:25:01.637141+0000","last_clean":"2024-04-17T10:25:01.637141+0000","last_became_active":"2024-04-17T10:24:59.657179+0000","last_became_peered":"2024-04-17T10:24:59.657179+0000","last_unstale":"2024-04-17T10:25:01.637141+0000","last_undegraded":"2024-04-17T10:25:01.637141+0000","last_fullsized":"2024-04-17T10:25:01.637141+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637399+0000","last_change":"2024-04-17T10:24:55.609933+0000","last_active":"2024-04-17T10:25:01.637399+0000","last_peered":"2024-04-17T10:25:01.637399+0000","last_clean":"2024-04-17T10:25:01.637399+0000","last_became_active":"2024-04-17T10:24:55.602879+0000","last_became_peered":"2024-04-17T10:24:55.602879+0000","last_unstale":"2024-04-17T10:25:01.637399+0000","last_undegraded":"2024-04-17T10:25:01.637399+0000","last_fullsized":"2024-04-17T10:25:01.637399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:51.084300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636629+0000","last_change":"2024-04-17T10:24:54.607532+0000","last_active":"2024-04-17T10:25:01.636629+0000","last_peered":"2024-04-17T10:25:01.636629+0000","last_clean":"2024-04-17T10:25:01.636629+0000","last_became_active":"2024-04-17T10:24:54.606527+0000","last_became_peered":"2024-04-17T10:24:54.606527+0000","last_unstale":"2024-04-17T10:25:01.636629+0000","last_undegraded":"2024-04-17T10:25:01.636629+0000","last_fullsized":"2024-04-17T10:25:01.636629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637464+0000","last_change":"2024-04-17T10:24:59.644478+0000","last_active":"2024-04-17T10:25:01.637464+0000","last_peered":"2024-04-17T10:25:01.637464+0000","last_clean":"2024-04-17T10:25:01.637464+0000","last_became_active":"2024-04-17T10:24:59.644083+0000","last_became_peered":"2024-04-17T10:24:59.644083+0000","last_unstale":"2024-04-17T10:25:01.637464+0000","last_undegraded":"2024-04-17T10:25:01.637464+0000","last_fullsized":"2024-04-17T10:25:01.637464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156377+0000","last_change":"2024-04-17T10:24:55.627359+0000","last_active":"2024-04-17T10:25:03.156377+0000","last_peered":"2024-04-17T10:25:03.156377+0000","last_clean":"2024-04-17T10:25:03.156377+0000","last_became_active":"2024-04-17T10:24:55.627094+0000","last_became_peered":"2024-04-17T10:24:55.627094+0000","last_unstale":"2024-04-17T10:25:03.156377+0000","last_undegraded":"2024-04-17T10:25:03.156377+0000","last_fullsized":"2024-04-17T10:25:03.156377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267575+0000","last_change":"2024-04-17T10:24:54.613073+0000","last_active":"2024-04-17T10:25:03.267575+0000","last_peered":"2024-04-17T10:25:03.267575+0000","last_clean":"2024-04-17T10:25:03.267575+0000","last_became_active":"2024-04-17T10:24:54.612771+0000","last_became_peered":"2024-04-17T10:24:54.612771+0000","last_unstale":"2024-04-17T10:25:03.267575+0000","last_undegraded":"2024-04-17T10:25:03.267575+0000","last_fullsized":"2024-04-17T10:25:03.267575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637853+0000","last_change":"2024-04-17T10:24:59.653549+0000","last_active":"2024-04-17T10:25:01.637853+0000","last_peered":"2024-04-17T10:25:01.637853+0000","last_clean":"2024-04-17T10:25:01.637853+0000","last_became_active":"2024-04-17T10:24:59.652615+0000","last_became_peered":"2024-04-17T10:24:59.652615+0000","last_unstale":"2024-04-17T10:25:01.637853+0000","last_undegraded":"2024-04-17T10:25:01.637853+0000","last_fullsized":"2024-04-17T10:25:01.637853+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:43.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268787+0000","last_change":"2024-04-17T10:24:55.612177+0000","last_active":"2024-04-17T10:25:03.268787+0000","last_peered":"2024-04-17T10:25:03.268787+0000","last_clean":"2024-04-17T10:25:03.268787+0000","last_became_active":"2024-04-17T10:24:55.611847+0000","last_became_peered":"2024-04-17T10:24:55.611847+0000","last_unstale":"2024-04-17T10:25:03.268787+0000","last_undegraded":"2024-04-17T10:25:03.268787+0000","last_fullsized":"2024-04-17T10:25:03.268787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:00.256447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537797+0000","last_change":"2024-04-17T10:24:54.614002+0000","last_active":"2024-04-17T10:25:02.537797+0000","last_peered":"2024-04-17T10:25:02.537797+0000","last_clean":"2024-04-17T10:25:02.537797+0000","last_became_active":"2024-04-17T10:24:54.613675+0000","last_became_peered":"2024-04-17T10:24:54.613675+0000","last_unstale":"2024-04-17T10:25:02.537797+0000","last_undegraded":"2024-04-17T10:25:02.537797+0000","last_fullsized":"2024-04-17T10:25:02.537797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637973+0000","last_change":"2024-04-17T10:25:00.244266+0000","last_active":"2024-04-17T10:25:01.637973+0000","last_peered":"2024-04-17T10:25:01.637973+0000","last_clean":"2024-04-17T10:25:01.637973+0000","last_became_active":"2024-04-17T10:25:00.243271+0000","last_became_peered":"2024-04-17T10:25:00.243271+0000","last_unstale":"2024-04-17T10:25:01.637973+0000","last_undegraded":"2024-04-17T10:25:01.637973+0000","last_fullsized":"2024-04-17T10:25:01.637973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:37:59.616402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156261+0000","last_change":"2024-04-17T10:25:00.244894+0000","last_active":"2024-04-17T10:25:03.156261+0000","last_peered":"2024-04-17T10:25:03.156261+0000","last_clean":"2024-04-17T10:25:03.156261+0000","last_became_active":"2024-04-17T10:25:00.244366+0000","last_became_peered":"2024-04-17T10:25:00.244366+0000","last_unstale":"2024-04-17T10:25:03.156261+0000","last_undegraded":"2024-04-17T10:25:03.156261+0000","last_fullsized":"2024-04-17T10:25:03.156261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636436+0000","last_change":"2024-04-17T10:24:55.610258+0000","last_active":"2024-04-17T10:25:01.636436+0000","last_peered":"2024-04-17T10:25:01.636436+0000","last_clean":"2024-04-17T10:25:01.636436+0000","last_became_active":"2024-04-17T10:24:55.609904+0000","last_became_peered":"2024-04-17T10:24:55.609904+0000","last_unstale":"2024-04-17T10:25:01.636436+0000","last_undegraded":"2024-04-17T10:25:01.636436+0000","last_fullsized":"2024-04-17T10:25:01.636436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267470+0000","last_change":"2024-04-17T10:24:54.605205+0000","last_active":"2024-04-17T10:25:03.267470+0000","last_peered":"2024-04-17T10:25:03.267470+0000","last_clean":"2024-04-17T10:25:03.267470+0000","last_became_active":"2024-04-17T10:24:54.601015+0000","last_became_peered":"2024-04-17T10:24:54.601015+0000","last_unstale":"2024-04-17T10:25:03.267470+0000","last_undegraded":"2024-04-17T10:25:03.267470+0000","last_fullsized":"2024-04-17T10:25:03.267470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537785+0000","last_change":"2024-04-17T10:24:59.648566+0000","last_active":"2024-04-17T10:25:02.537785+0000","last_peered":"2024-04-17T10:25:02.537785+0000","last_clean":"2024-04-17T10:25:02.537785+0000","last_became_active":"2024-04-17T10:24:59.648170+0000","last_became_peered":"2024-04-17T10:24:59.648170+0000","last_unstale":"2024-04-17T10:25:02.537785+0000","last_undegraded":"2024-04-17T10:25:02.537785+0000","last_fullsized":"2024-04-17T10:25:02.537785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637759+0000","last_change":"2024-04-17T10:24:55.611737+0000","last_active":"2024-04-17T10:25:01.637759+0000","last_peered":"2024-04-17T10:25:01.637759+0000","last_clean":"2024-04-17T10:25:01.637759+0000","last_became_active":"2024-04-17T10:24:55.611353+0000","last_became_peered":"2024-04-17T10:24:55.611353+0000","last_unstale":"2024-04-17T10:25:01.637759+0000","last_undegraded":"2024-04-17T10:25:01.637759+0000","last_fullsized":"2024-04-17T10:25:01.637759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638133+0000","last_change":"2024-04-17T10:24:54.601357+0000","last_active":"2024-04-17T10:25:01.638133+0000","last_peered":"2024-04-17T10:25:01.638133+0000","last_clean":"2024-04-17T10:25:01.638133+0000","last_became_active":"2024-04-17T10:24:54.601111+0000","last_became_peered":"2024-04-17T10:24:54.601111+0000","last_unstale":"2024-04-17T10:25:01.638133+0000","last_undegraded":"2024-04-17T10:25:01.638133+0000","last_fullsized":"2024-04-17T10:25:01.638133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537510+0000","last_change":"2024-04-17T10:24:54.601574+0000","last_active":"2024-04-17T10:25:02.537510+0000","last_peered":"2024-04-17T10:25:02.537510+0000","last_clean":"2024-04-17T10:25:02.537510+0000","last_became_active":"2024-04-17T10:24:54.601261+0000","last_became_peered":"2024-04-17T10:24:54.601261+0000","last_unstale":"2024-04-17T10:25:02.537510+0000","last_undegraded":"2024-04-17T10:25:02.537510+0000","last_fullsized":"2024-04-17T10:25:02.537510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156049+0000","last_change":"2024-04-17T10:24:55.604048+0000","last_active":"2024-04-17T10:25:03.156049+0000","last_peered":"2024-04-17T10:25:03.156049+0000","last_clean":"2024-04-17T10:25:03.156049+0000","last_became_active":"2024-04-17T10:24:55.592091+0000","last_became_peered":"2024-04-17T10:24:55.592091+0000","last_unstale":"2024-04-17T10:25:03.156049+0000","last_undegraded":"2024-04-17T10:25:03.156049+0000","last_fullsized":"2024-04-17T10:25:03.156049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625146+0000","last_change":"2024-04-17T10:24:59.656792+0000","last_active":"2024-04-17T10:25:02.625146+0000","last_peered":"2024-04-17T10:25:02.625146+0000","last_clean":"2024-04-17T10:25:02.625146+0000","last_became_active":"2024-04-17T10:24:59.656244+0000","last_became_peered":"2024-04-17T10:24:59.656244+0000","last_unstale":"2024-04-17T10:25:02.625146+0000","last_undegraded":"2024-04-17T10:25:02.625146+0000","last_fullsized":"2024-04-17T10:25:02.625146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:57.427708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638566+0000","last_change":"2024-04-17T10:24:54.609412+0000","last_active":"2024-04-17T10:25:01.638566+0000","last_peered":"2024-04-17T10:25:01.638566+0000","last_clean":"2024-04-17T10:25:01.638566+0000","last_became_active":"2024-04-17T10:24:54.609119+0000","last_became_peered":"2024-04-17T10:24:54.609119+0000","last_unstale":"2024-04-17T10:25:01.638566+0000","last_undegraded":"2024-04-17T10:25:01.638566+0000","last_fullsized":"2024-04-17T10:25:01.638566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636952+0000","last_change":"2024-04-17T10:24:55.614982+0000","last_active":"2024-04-17T10:25:01.636952+0000","last_peered":"2024-04-17T10:25:01.636952+0000","last_clean":"2024-04-17T10:25:01.636952+0000","last_became_active":"2024-04-17T10:24:55.614619+0000","last_became_peered":"2024-04-17T10:24:55.614619+0000","last_unstale":"2024-04-17T10:25:01.636952+0000","last_undegraded":"2024-04-17T10:25:01.636952+0000","last_fullsized":"2024-04-17T10:25:01.636952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:32.258402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638839+0000","last_change":"2024-04-17T10:24:59.647972+0000","last_active":"2024-04-17T10:25:01.638839+0000","last_peered":"2024-04-17T10:25:01.638839+0000","last_clean":"2024-04-17T10:25:01.638839+0000","last_became_active":"2024-04-17T10:24:59.646737+0000","last_became_peered":"2024-04-17T10:24:59.646737+0000","last_unstale":"2024-04-17T10:25:01.638839+0000","last_undegraded":"2024-04-17T10:25:01.638839+0000","last_fullsized":"2024-04-17T10:25:01.638839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:59.695291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537104+0000","last_change":"2024-04-17T10:24:55.608996+0000","last_active":"2024-04-17T10:25:02.537104+0000","last_peered":"2024-04-17T10:25:02.537104+0000","last_clean":"2024-04-17T10:25:02.537104+0000","last_became_active":"2024-04-17T10:24:55.608621+0000","last_became_peered":"2024-04-17T10:24:55.608621+0000","last_unstale":"2024-04-17T10:25:02.537104+0000","last_undegraded":"2024-04-17T10:25:02.537104+0000","last_fullsized":"2024-04-17T10:25:02.537104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625124+0000","last_change":"2024-04-17T10:24:55.616866+0000","last_active":"2024-04-17T10:25:02.625124+0000","last_peered":"2024-04-17T10:25:02.625124+0000","last_clean":"2024-04-17T10:25:02.625124+0000","last_became_active":"2024-04-17T10:24:55.616582+0000","last_became_peered":"2024-04-17T10:24:55.616582+0000","last_unstale":"2024-04-17T10:25:02.625124+0000","last_undegraded":"2024-04-17T10:25:02.625124+0000","last_fullsized":"2024-04-17T10:25:02.625124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268379+0000","last_change":"2024-04-17T10:24:55.613768+0000","last_active":"2024-04-17T10:25:03.268379+0000","last_peered":"2024-04-17T10:25:03.268379+0000","last_clean":"2024-04-17T10:25:03.268379+0000","last_became_active":"2024-04-17T10:24:55.612427+0000","last_became_peered":"2024-04-17T10:24:55.612427+0000","last_unstale":"2024-04-17T10:25:03.268379+0000","last_undegraded":"2024-04-17T10:25:03.268379+0000","last_fullsized":"2024-04-17T10:25:03.268379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155940+0000","last_change":"2024-04-17T10:24:55.613422+0000","last_active":"2024-04-17T10:25:03.155940+0000","last_peered":"2024-04-17T10:25:03.155940+0000","last_clean":"2024-04-17T10:25:03.155940+0000","last_became_active":"2024-04-17T10:24:55.612627+0000","last_became_peered":"2024-04-17T10:24:55.612627+0000","last_unstale":"2024-04-17T10:25:03.155940+0000","last_undegraded":"2024-04-17T10:25:03.155940+0000","last_fullsized":"2024-04-17T10:25:03.155940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639265+0000","last_change":"2024-04-17T10:24:55.617532+0000","last_active":"2024-04-17T10:25:01.639265+0000","last_peered":"2024-04-17T10:25:01.639265+0000","last_clean":"2024-04-17T10:25:01.639265+0000","last_became_active":"2024-04-17T10:24:55.617215+0000","last_became_peered":"2024-04-17T10:24:55.617215+0000","last_unstale":"2024-04-17T10:25:01.639265+0000","last_undegraded":"2024-04-17T10:25:01.639265+0000","last_fullsized":"2024-04-17T10:25:01.639265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636559+0000","last_change":"2024-04-17T10:24:55.631795+0000","last_active":"2024-04-17T10:25:01.636559+0000","last_peered":"2024-04-17T10:25:01.636559+0000","last_clean":"2024-04-17T10:25:01.636559+0000","last_became_active":"2024-04-17T10:24:55.629826+0000","last_became_peered":"2024-04-17T10:24:55.629826+0000","last_unstale":"2024-04-17T10:25:01.636559+0000","last_undegraded":"2024-04-17T10:25:01.636559+0000","last_fullsized":"2024-04-17T10:25:01.636559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537858+0000","last_change":"2024-04-17T10:24:55.606321+0000","last_active":"2024-04-17T10:25:02.537858+0000","last_peered":"2024-04-17T10:25:02.537858+0000","last_clean":"2024-04-17T10:25:02.537858+0000","last_became_active":"2024-04-17T10:24:55.605895+0000","last_became_peered":"2024-04-17T10:24:55.605895+0000","last_unstale":"2024-04-17T10:25:02.537858+0000","last_undegraded":"2024-04-17T10:25:02.537858+0000","last_fullsized":"2024-04-17T10:25:02.537858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638540+0000","last_change":"2024-04-17T10:24:55.607147+0000","last_active":"2024-04-17T10:25:01.638540+0000","last_peered":"2024-04-17T10:25:01.638540+0000","last_clean":"2024-04-17T10:25:01.638540+0000","last_became_active":"2024-04-17T10:24:55.606931+0000","last_became_peered":"2024-04-17T10:24:55.606931+0000","last_unstale":"2024-04-17T10:25:01.638540+0000","last_undegraded":"2024-04-17T10:25:01.638540+0000","last_fullsized":"2024-04-17T10:25:01.638540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268051+0000","last_change":"2024-04-17T10:24:55.609356+0000","last_active":"2024-04-17T10:25:03.268051+0000","last_peered":"2024-04-17T10:25:03.268051+0000","last_clean":"2024-04-17T10:25:03.268051+0000","last_became_active":"2024-04-17T10:24:55.607716+0000","last_became_peered":"2024-04-17T10:24:55.607716+0000","last_unstale":"2024-04-17T10:25:03.268051+0000","last_undegraded":"2024-04-17T10:25:03.268051+0000","last_fullsized":"2024-04-17T10:25:03.268051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638038+0000","last_change":"2024-04-17T10:24:55.605676+0000","last_active":"2024-04-17T10:25:01.638038+0000","last_peered":"2024-04-17T10:25:01.638038+0000","last_clean":"2024-04-17T10:25:01.638038+0000","last_became_active":"2024-04-17T10:24:55.605431+0000","last_became_peered":"2024-04-17T10:24:55.605431+0000","last_unstale":"2024-04-17T10:25:01.638038+0000","last_undegraded":"2024-04-17T10:25:01.638038+0000","last_fullsized":"2024-04-17T10:25:01.638038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268286+0000","last_change":"2024-04-17T10:24:55.615398+0000","last_active":"2024-04-17T10:25:03.268286+0000","last_peered":"2024-04-17T10:25:03.268286+0000","last_clean":"2024-04-17T10:25:03.268286+0000","last_became_active":"2024-04-17T10:24:55.615034+0000","last_became_peered":"2024-04-17T10:24:55.615034+0000","last_unstale":"2024-04-17T10:25:03.268286+0000","last_undegraded":"2024-04-17T10:25:03.268286+0000","last_fullsized":"2024-04-17T10:25:03.268286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155772+0000","last_change":"2024-04-17T10:24:55.613776+0000","last_active":"2024-04-17T10:25:03.155772+0000","last_peered":"2024-04-17T10:25:03.155772+0000","last_clean":"2024-04-17T10:25:03.155772+0000","last_became_active":"2024-04-17T10:24:55.613106+0000","last_became_peered":"2024-04-17T10:24:55.613106+0000","last_unstale":"2024-04-17T10:25:03.155772+0000","last_undegraded":"2024-04-17T10:25:03.155772+0000","last_fullsized":"2024-04-17T10:25:03.155772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537535+0000","last_change":"2024-04-17T10:24:55.619933+0000","last_active":"2024-04-17T10:25:02.537535+0000","last_peered":"2024-04-17T10:25:02.537535+0000","last_clean":"2024-04-17T10:25:02.537535+0000","last_became_active":"2024-04-17T10:24:55.619666+0000","last_became_peered":"2024-04-17T10:24:55.619666+0000","last_unstale":"2024-04-17T10:25:02.537535+0000","last_undegraded":"2024-04-17T10:25:02.537535+0000","last_fullsized":"2024-04-17T10:25:02.537535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423902+0000","last_change":"2024-04-17T10:24:55.608090+0000","last_active":"2024-04-17T10:25:02.423902+0000","last_peered":"2024-04-17T10:25:02.423902+0000","last_clean":"2024-04-17T10:25:02.423902+0000","last_became_active":"2024-04-17T10:24:55.607715+0000","last_became_peered":"2024-04-17T10:24:55.607715+0000","last_unstale":"2024-04-17T10:25:02.423902+0000","last_undegraded":"2024-04-17T10:25:02.423902+0000","last_fullsized":"2024-04-17T10:25:02.423902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:07.835599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625372+0000","last_change":"2024-04-17T10:24:55.616021+0000","last_active":"2024-04-17T10:25:02.625372+0000","last_peered":"2024-04-17T10:25:02.625372+0000","last_clean":"2024-04-17T10:25:02.625372+0000","last_became_active":"2024-04-17T10:24:55.615736+0000","last_became_peered":"2024-04-17T10:24:55.615736+0000","last_unstale":"2024-04-17T10:25:02.625372+0000","last_undegraded":"2024-04-17T10:25:02.625372+0000","last_fullsized":"2024-04-17T10:25:02.625372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268965+0000","last_change":"2024-04-17T10:24:55.614983+0000","last_active":"2024-04-17T10:25:03.268965+0000","last_peered":"2024-04-17T10:25:03.268965+0000","last_clean":"2024-04-17T10:25:03.268965+0000","last_became_active":"2024-04-17T10:24:55.614651+0000","last_became_peered":"2024-04-17T10:24:55.614651+0000","last_unstale":"2024-04-17T10:25:03.268965+0000","last_undegraded":"2024-04-17T10:25:03.268965+0000","last_fullsized":"2024-04-17T10:25:03.268965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267935+0000","last_change":"2024-04-17T10:24:55.626976+0000","last_active":"2024-04-17T10:25:03.267935+0000","last_peered":"2024-04-17T10:25:03.267935+0000","last_clean":"2024-04-17T10:25:03.267935+0000","last_became_active":"2024-04-17T10:24:55.626743+0000","last_became_peered":"2024-04-17T10:24:55.626743+0000","last_unstale":"2024-04-17T10:25:03.267935+0000","last_undegraded":"2024-04-17T10:25:03.267935+0000","last_fullsized":"2024-04-17T10:25:03.267935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636737+0000","last_change":"2024-04-17T10:24:55.618634+0000","last_active":"2024-04-17T10:25:01.636737+0000","last_peered":"2024-04-17T10:25:01.636737+0000","last_clean":"2024-04-17T10:25:01.636737+0000","last_became_active":"2024-04-17T10:24:55.618343+0000","last_became_peered":"2024-04-17T10:24:55.618343+0000","last_unstale":"2024-04-17T10:25:01.636737+0000","last_undegraded":"2024-04-17T10:25:01.636737+0000","last_fullsized":"2024-04-17T10:25:01.636737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155747+0000","last_change":"2024-04-17T10:24:55.616149+0000","last_active":"2024-04-17T10:25:03.155747+0000","last_peered":"2024-04-17T10:25:03.155747+0000","last_clean":"2024-04-17T10:25:03.155747+0000","last_became_active":"2024-04-17T10:24:55.615561+0000","last_became_peered":"2024-04-17T10:24:55.615561+0000","last_unstale":"2024-04-17T10:25:03.155747+0000","last_undegraded":"2024-04-17T10:25:03.155747+0000","last_fullsized":"2024-04-17T10:25:03.155747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:37.060791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638375+0000","last_change":"2024-04-17T10:24:55.612680+0000","last_active":"2024-04-17T10:25:01.638375+0000","last_peered":"2024-04-17T10:25:01.638375+0000","last_clean":"2024-04-17T10:25:01.638375+0000","last_became_active":"2024-04-17T10:24:55.612280+0000","last_became_peered":"2024-04-17T10:24:55.612280+0000","last_unstale":"2024-04-17T10:25:01.638375+0000","last_undegraded":"2024-04-17T10:25:01.638375+0000","last_fullsized":"2024-04-17T10:25:01.638375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.610235+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.609934+0000","last_became_peered":"2024-04-17T10:24:55.609934+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537674+0000","last_change":"2024-04-17T10:24:55.613780+0000","last_active":"2024-04-17T10:25:02.537674+0000","last_peered":"2024-04-17T10:25:02.537674+0000","last_clean":"2024-04-17T10:25:02.537674+0000","last_became_active":"2024-04-17T10:24:55.613360+0000","last_became_peered":"2024-04-17T10:24:55.613360+0000","last_unstale":"2024-04-17T10:25:02.537674+0000","last_undegraded":"2024-04-17T10:25:02.537674+0000","last_fullsized":"2024-04-17T10:25:02.537674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537823+0000","last_change":"2024-04-17T10:24:55.617826+0000","last_active":"2024-04-17T10:25:02.537823+0000","last_peered":"2024-04-17T10:25:02.537823+0000","last_clean":"2024-04-17T10:25:02.537823+0000","last_became_active":"2024-04-17T10:24:55.617446+0000","last_became_peered":"2024-04-17T10:24:55.617446+0000","last_unstale":"2024-04-17T10:25:02.537823+0000","last_undegraded":"2024-04-17T10:25:02.537823+0000","last_fullsized":"2024-04-17T10:25:02.537823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424189+0000","last_change":"2024-04-17T10:24:55.626063+0000","last_active":"2024-04-17T10:25:02.424189+0000","last_peered":"2024-04-17T10:25:02.424189+0000","last_clean":"2024-04-17T10:25:02.424189+0000","last_became_active":"2024-04-17T10:24:55.625720+0000","last_became_peered":"2024-04-17T10:24:55.625720+0000","last_unstale":"2024-04-17T10:25:02.424189+0000","last_undegraded":"2024-04-17T10:25:02.424189+0000","last_fullsized":"2024-04-17T10:25:02.424189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:39.746376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637649+0000","last_change":"2024-04-17T10:24:55.622013+0000","last_active":"2024-04-17T10:25:01.637649+0000","last_peered":"2024-04-17T10:25:01.637649+0000","last_clean":"2024-04-17T10:25:01.637649+0000","last_became_active":"2024-04-17T10:24:55.618726+0000","last_became_peered":"2024-04-17T10:24:55.618726+0000","last_unstale":"2024-04-17T10:25:01.637649+0000","last_undegraded":"2024-04-17T10:25:01.637649+0000","last_fullsized":"2024-04-17T10:25:01.637649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423351+0000","last_change":"2024-04-17T10:24:55.614704+0000","last_active":"2024-04-17T10:25:02.423351+0000","last_peered":"2024-04-17T10:25:02.423351+0000","last_clean":"2024-04-17T10:25:02.423351+0000","last_became_active":"2024-04-17T10:24:55.614425+0000","last_became_peered":"2024-04-17T10:24:55.614425+0000","last_unstale":"2024-04-17T10:25:02.423351+0000","last_undegraded":"2024-04-17T10:25:02.423351+0000","last_fullsized":"2024-04-17T10:25:02.423351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268264+0000","last_change":"2024-04-17T10:24:55.630019+0000","last_active":"2024-04-17T10:25:03.268264+0000","last_peered":"2024-04-17T10:25:03.268264+0000","last_clean":"2024-04-17T10:25:03.268264+0000","last_became_active":"2024-04-17T10:24:55.629824+0000","last_became_peered":"2024-04-17T10:24:55.629824+0000","last_unstale":"2024-04-17T10:25:03.268264+0000","last_undegraded":"2024-04-17T10:25:03.268264+0000","last_fullsized":"2024-04-17T10:25:03.268264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423511+0000","last_change":"2024-04-17T10:24:55.612346+0000","last_active":"2024-04-17T10:25:02.423511+0000","last_peered":"2024-04-17T10:25:02.423511+0000","last_clean":"2024-04-17T10:25:02.423511+0000","last_became_active":"2024-04-17T10:24:55.610079+0000","last_became_peered":"2024-04-17T10:24:55.610079+0000","last_unstale":"2024-04-17T10:25:02.423511+0000","last_undegraded":"2024-04-17T10:25:02.423511+0000","last_fullsized":"2024-04-17T10:25:02.423511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268502+0000","last_change":"2024-04-17T10:24:55.612221+0000","last_active":"2024-04-17T10:25:03.268502+0000","last_peered":"2024-04-17T10:25:03.268502+0000","last_clean":"2024-04-17T10:25:03.268502+0000","last_became_active":"2024-04-17T10:24:55.611946+0000","last_became_peered":"2024-04-17T10:24:55.611946+0000","last_unstale":"2024-04-17T10:25:03.268502+0000","last_undegraded":"2024-04-17T10:25:03.268502+0000","last_fullsized":"2024-04-17T10:25:03.268502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:56.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156183+0000","last_change":"2024-04-17T10:24:55.617011+0000","last_active":"2024-04-17T10:25:03.156183+0000","last_peered":"2024-04-17T10:25:03.156183+0000","last_clean":"2024-04-17T10:25:03.156183+0000","last_became_active":"2024-04-17T10:24:55.616709+0000","last_became_peered":"2024-04-17T10:24:55.616709+0000","last_unstale":"2024-04-17T10:25:03.156183+0000","last_undegraded":"2024-04-17T10:25:03.156183+0000","last_fullsized":"2024-04-17T10:25:03.156183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:10.092171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636934+0000","last_change":"2024-04-17T10:24:55.627305+0000","last_active":"2024-04-17T10:25:01.636934+0000","last_peered":"2024-04-17T10:25:01.636934+0000","last_clean":"2024-04-17T10:25:01.636934+0000","last_became_active":"2024-04-17T10:24:55.626978+0000","last_became_peered":"2024-04-17T10:24:55.626978+0000","last_unstale":"2024-04-17T10:25:01.636934+0000","last_undegraded":"2024-04-17T10:25:01.636934+0000","last_fullsized":"2024-04-17T10:25:01.636934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424104+0000","last_change":"2024-04-17T10:24:55.613165+0000","last_active":"2024-04-17T10:25:02.424104+0000","last_peered":"2024-04-17T10:25:02.424104+0000","last_clean":"2024-04-17T10:25:02.424104+0000","last_became_active":"2024-04-17T10:24:55.611929+0000","last_became_peered":"2024-04-17T10:24:55.611929+0000","last_unstale":"2024-04-17T10:25:02.424104+0000","last_undegraded":"2024-04-17T10:25:02.424104+0000","last_fullsized":"2024-04-17T10:25:02.424104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:48.582360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:25:25.491 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:25:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.671+0000 7f45b8400640 1 Processor -- start 2024-04-17T10:25:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.671+0000 7f45b8400640 1 -- start start 2024-04-17T10:25:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b8400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b8400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b015eb50 con 0x7f45b01620a0 2024-04-17T10:25:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b8400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b015ee30 con 0x7f45b00ce020 2024-04-17T10:25:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b8400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b0161650 con 0x7f45b00cf510 2024-04-17T10:25:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b5000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f45b01620a0 0x7f45b015e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54430/0 (socket says 172.21.15.67:54430) 2024-04-17T10:25:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b5000640 1 -- 172.21.15.67:0/2375981653 learned_addr learned my addr 172.21.15.67:0/2375981653 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b6400640 1 --2- 172.21.15.67:0/2375981653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45afe00640 1 -- 172.21.15.67:0/2375981653 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589844458 0 0) 0x7f45b015eb50 con 0x7f45b01620a0 2024-04-17T10:25:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45afe00640 1 -- 172.21.15.67:0/2375981653 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4598003530 con 0x7f45b01620a0 2024-04-17T10:25:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b6400640 1 -- 172.21.15.67:0/2375981653 >> v1:172.21.15.86:6789/0 conn(0x7f45b00ce020 legacy=0x7f45b00ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45afe00640 1 -- 172.21.15.67:0/2375981653 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287457992 0 0) 0x7f45b015ee30 con 0x7f45b00ce020 2024-04-17T10:25:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b6400640 1 -- 172.21.15.67:0/2375981653 >> v1:172.21.15.67:6789/0 conn(0x7f45b01620a0 legacy=0x7f45b015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.675+0000 7f45b6400640 1 -- 172.21.15.67:0/2375981653 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b01618a0 con 0x7f45b00cf510 2024-04-17T10:25:25.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b6400640 1 --2- 172.21.15.67:0/2375981653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00cf9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009df0 tx=0x7f45a801c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd3b505a0d475f8d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:25.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45afe00640 1 -- 172.21.15.67:0/2375981653 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a80b2070 con 0x7f45b00cf510 2024-04-17T10:25:25.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45afe00640 1 -- 172.21.15.67:0/2375981653 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a80a5930 con 0x7f45b00cf510 2024-04-17T10:25:25.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45afe00640 1 -- 172.21.15.67:0/2375981653 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a80a5d00 con 0x7f45b00cf510 2024-04-17T10:25:25.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 -- 172.21.15.67:0/2375981653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 msgr2=0x7f45b00cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 --2- 172.21.15.67:0/2375981653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00cf9a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f45a8009df0 tx=0x7f45a801c960 comp rx=0 tx=0).stop 2024-04-17T10:25:25.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 -- 172.21.15.67:0/2375981653 shutdown_connections 2024-04-17T10:25:25.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 --2- 172.21.15.67:0/2375981653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00cf9a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 -- 172.21.15.67:0/2375981653 >> 172.21.15.67:0/2375981653 conn(0x7f45b00b33f0 msgr2=0x7f45b00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 -- 172.21.15.67:0/2375981653 shutdown_connections 2024-04-17T10:25:25.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 -- 172.21.15.67:0/2375981653 wait complete. 2024-04-17T10:25:25.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 Processor -- start 2024-04-17T10:25:25.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.679+0000 7f45b8400640 1 -- start start 2024-04-17T10:25:25.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00ce020 0x7f45b0157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b01620a0 0x7f45b00b6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015e8a0 con 0x7f45b00ce020 2024-04-17T10:25:25.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b015e720 con 0x7f45b01620a0 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b00c84b0 con 0x7f45b00cf510 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00ce020 0x7f45b0157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00ce020 0x7f45b0157bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58362/0 (socket says 172.21.15.67:58362) 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 -- 172.21.15.67:0/4151820325 learned_addr learned my addr 172.21.15.67:0/4151820325 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5000640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b6400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b01620a0 0x7f45b00b6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 -- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 msgr2=0x7f45b00b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 -- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b01620a0 msgr2=0x7f45b00b6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b01620a0 0x7f45b00b6fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 -- 172.21.15.67:0/4151820325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b0158300 con 0x7f45b00ce020 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b6400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b01620a0 0x7f45b00b6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5000640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:25.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b5a00640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00ce020 0x7f45b0157bd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c4d0 tx=0x7f45a400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:25.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45ad600640 1 -- 172.21.15.67:0/4151820325 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4017070 con 0x7f45b00ce020 2024-04-17T10:25:25.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45ad600640 1 -- 172.21.15.67:0/4151820325 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a4005480 con 0x7f45b00ce020 2024-04-17T10:25:25.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45ad600640 1 -- 172.21.15.67:0/4151820325 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4003840 con 0x7f45b00ce020 2024-04-17T10:25:25.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b015c6c0 con 0x7f45b00ce020 2024-04-17T10:25:25.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b015cc80 con 0x7f45b00ce020 2024-04-17T10:25:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b0070ab0 con 0x7f45b00ce020 2024-04-17T10:25:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.683+0000 7f45ad600640 1 -- 172.21.15.67:0/4151820325 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f45a401a070 con 0x7f45b00ce020 2024-04-17T10:25:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.687+0000 7f45ad600640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4590066f90 0x7f4590069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.687+0000 7f45ad600640 1 -- 172.21.15.67:0/4151820325 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f45a4084720 con 0x7f45b00ce020 2024-04-17T10:25:25.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.687+0000 7f45b5000640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4590066f90 0x7f4590069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:25.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.687+0000 7f45b5000640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4590066f90 0x7f4590069450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f45b00b4790 tx=0x7f459c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:25.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.691+0000 7f45ad600640 1 -- 172.21.15.67:0/4151820325 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f45a40554a0 con 0x7f45b00ce020 2024-04-17T10:25:25.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.855+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45b0074e30 con 0x7f4590066f90 2024-04-17T10:25:25.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.863+0000 7f45ad600640 1 -- 172.21.15.67:0/4151820325 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+332537 (secure 0 0 0) 0x7f45b0074e30 con 0x7f4590066f90 2024-04-17T10:25:25.861 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:25.864 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4590066f90 msgr2=0x7f4590069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4590066f90 0x7f4590069450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f45b00b4790 tx=0x7f459c009040 comp rx=0 tx=0).stop 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00ce020 msgr2=0x7f45b0157bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00ce020 0x7f45b0157bd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f45a400c4d0 tx=0x7f45a400c990 comp rx=0 tx=0).stop 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 shutdown_connections 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4590066f90 0x7f4590069450 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b01620a0 0x7f45b00b6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b00cf510 0x7f45b00b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 --2- 172.21.15.67:0/4151820325 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b00ce020 0x7f45b0157bd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:25.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 >> 172.21.15.67:0/4151820325 conn(0x7f45b00b33f0 msgr2=0x7f45b00d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:25.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 shutdown_connections 2024-04-17T10:25:25.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:25.867+0000 7f45b8400640 1 -- 172.21.15.67:0/4151820325 wait complete. 2024-04-17T10:25:25.882 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-17T10:25:24.141029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219472,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755248,"statfs":{"total":773094113280,"available":772869373952,"internally_reserved":0,"allocated":3383296,"data_stored":2049160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003106"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":28,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155309+0000","last_change":"2024-04-17T10:24:54.602648+0000","last_active":"2024-04-17T10:25:03.155309+0000","last_peered":"2024-04-17T10:25:03.155309+0000","last_clean":"2024-04-17T10:25:03.155309+0000","last_became_active":"2024-04-17T10:24:54.602381+0000","last_became_peered":"2024-04-17T10:24:54.602381+0000","last_unstale":"2024-04-17T10:25:03.155309+0000","last_undegraded":"2024-04-17T10:25:03.155309+0000","last_fullsized":"2024-04-17T10:25:03.155309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638270+0000","last_change":"2024-04-17T10:24:55.617102+0000","last_active":"2024-04-17T10:25:01.638270+0000","last_peered":"2024-04-17T10:25:01.638270+0000","last_clean":"2024-04-17T10:25:01.638270+0000","last_became_active":"2024-04-17T10:24:55.615401+0000","last_became_peered":"2024-04-17T10:24:55.615401+0000","last_unstale":"2024-04-17T10:25:01.638270+0000","last_undegraded":"2024-04-17T10:25:01.638270+0000","last_fullsized":"2024-04-17T10:25:01.638270+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637780+0000","last_change":"2024-04-17T10:25:00.246846+0000","last_active":"2024-04-17T10:25:01.637780+0000","last_peered":"2024-04-17T10:25:01.637780+0000","last_clean":"2024-04-17T10:25:01.637780+0000","last_became_active":"2024-04-17T10:25:00.246379+0000","last_became_peered":"2024-04-17T10:25:00.246379+0000","last_unstale":"2024-04-17T10:25:01.637780+0000","last_undegraded":"2024-04-17T10:25:01.637780+0000","last_fullsized":"2024-04-17T10:25:01.637780+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:55:05.249462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638369+0000","last_change":"2024-04-17T10:24:54.607859+0000","last_active":"2024-04-17T10:25:01.638369+0000","last_peered":"2024-04-17T10:25:01.638369+0000","last_clean":"2024-04-17T10:25:01.638369+0000","last_became_active":"2024-04-17T10:24:54.607576+0000","last_became_peered":"2024-04-17T10:24:54.607576+0000","last_unstale":"2024-04-17T10:25:01.638369+0000","last_undegraded":"2024-04-17T10:25:01.638369+0000","last_fullsized":"2024-04-17T10:25:01.638369+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156289+0000","last_change":"2024-04-17T10:24:55.628197+0000","last_active":"2024-04-17T10:25:03.156289+0000","last_peered":"2024-04-17T10:25:03.156289+0000","last_clean":"2024-04-17T10:25:03.156289+0000","last_became_active":"2024-04-17T10:24:55.627976+0000","last_became_peered":"2024-04-17T10:24:55.627976+0000","last_unstale":"2024-04-17T10:25:03.156289+0000","last_undegraded":"2024-04-17T10:25:03.156289+0000","last_fullsized":"2024-04-17T10:25:03.156289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636562+0000","last_change":"2024-04-17T10:24:59.651727+0000","last_active":"2024-04-17T10:25:01.636562+0000","last_peered":"2024-04-17T10:25:01.636562+0000","last_clean":"2024-04-17T10:25:01.636562+0000","last_became_active":"2024-04-17T10:24:59.651358+0000","last_became_peered":"2024-04-17T10:24:59.651358+0000","last_unstale":"2024-04-17T10:25:01.636562+0000","last_undegraded":"2024-04-17T10:25:01.636562+0000","last_fullsized":"2024-04-17T10:25:01.636562+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423903+0000","last_change":"2024-04-17T10:24:54.611808+0000","last_active":"2024-04-17T10:25:02.423903+0000","last_peered":"2024-04-17T10:25:02.423903+0000","last_clean":"2024-04-17T10:25:02.423903+0000","last_became_active":"2024-04-17T10:24:54.610903+0000","last_became_peered":"2024-04-17T10:24:54.610903+0000","last_unstale":"2024-04-17T10:25:02.423903+0000","last_undegraded":"2024-04-17T10:25:02.423903+0000","last_fullsized":"2024-04-17T10:25:02.423903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:09.085810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268578+0000","last_change":"2024-04-17T10:24:55.618364+0000","last_active":"2024-04-17T10:25:03.268578+0000","last_peered":"2024-04-17T10:25:03.268578+0000","last_clean":"2024-04-17T10:25:03.268578+0000","last_became_active":"2024-04-17T10:24:55.618013+0000","last_became_peered":"2024-04-17T10:24:55.618013+0000","last_unstale":"2024-04-17T10:25:03.268578+0000","last_undegraded":"2024-04-17T10:25:03.268578+0000","last_fullsized":"2024-04-17T10:25:03.268578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636861+0000","last_change":"2024-04-17T10:24:59.647051+0000","last_active":"2024-04-17T10:25:01.636861+0000","last_peered":"2024-04-17T10:25:01.636861+0000","last_clean":"2024-04-17T10:25:01.636861+0000","last_became_active":"2024-04-17T10:24:59.646611+0000","last_became_peered":"2024-04-17T10:24:59.646611+0000","last_unstale":"2024-04-17T10:25:01.636861+0000","last_undegraded":"2024-04-17T10:25:01.636861+0000","last_fullsized":"2024-04-17T10:25:01.636861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423712+0000","last_change":"2024-04-17T10:24:54.611301+0000","last_active":"2024-04-17T10:25:02.423712+0000","last_peered":"2024-04-17T10:25:02.423712+0000","last_clean":"2024-04-17T10:25:02.423712+0000","last_became_active":"2024-04-17T10:24:54.610243+0000","last_became_peered":"2024-04-17T10:24:54.610243+0000","last_unstale":"2024-04-17T10:25:02.423712+0000","last_undegraded":"2024-04-17T10:25:02.423712+0000","last_fullsized":"2024-04-17T10:25:02.423712+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:36.054430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268618+0000","last_change":"2024-04-17T10:24:55.618112+0000","last_active":"2024-04-17T10:25:03.268618+0000","last_peered":"2024-04-17T10:25:03.268618+0000","last_clean":"2024-04-17T10:25:03.268618+0000","last_became_active":"2024-04-17T10:24:55.617127+0000","last_became_peered":"2024-04-17T10:24:55.617127+0000","last_unstale":"2024-04-17T10:25:03.268618+0000","last_undegraded":"2024-04-17T10:25:03.268618+0000","last_fullsized":"2024-04-17T10:25:03.268618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636698+0000","last_change":"2024-04-17T10:24:59.641046+0000","last_active":"2024-04-17T10:25:01.636698+0000","last_peered":"2024-04-17T10:25:01.636698+0000","last_clean":"2024-04-17T10:25:01.636698+0000","last_became_active":"2024-04-17T10:24:59.640622+0000","last_became_peered":"2024-04-17T10:24:59.640622+0000","last_unstale":"2024-04-17T10:25:01.636698+0000","last_undegraded":"2024-04-17T10:25:01.636698+0000","last_fullsized":"2024-04-17T10:25:01.636698+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537179+0000","last_change":"2024-04-17T10:24:55.617061+0000","last_active":"2024-04-17T10:25:02.537179+0000","last_peered":"2024-04-17T10:25:02.537179+0000","last_clean":"2024-04-17T10:25:02.537179+0000","last_became_active":"2024-04-17T10:24:55.616809+0000","last_became_peered":"2024-04-17T10:24:55.616809+0000","last_unstale":"2024-04-17T10:25:02.537179+0000","last_undegraded":"2024-04-17T10:25:02.537179+0000","last_fullsized":"2024-04-17T10:25:02.537179+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423815+0000","last_change":"2024-04-17T10:24:54.607974+0000","last_active":"2024-04-17T10:25:02.423815+0000","last_peered":"2024-04-17T10:25:02.423815+0000","last_clean":"2024-04-17T10:25:02.423815+0000","last_became_active":"2024-04-17T10:24:54.606303+0000","last_became_peered":"2024-04-17T10:24:54.606303+0000","last_unstale":"2024-04-17T10:25:02.423815+0000","last_undegraded":"2024-04-17T10:25:02.423815+0000","last_fullsized":"2024-04-17T10:25:02.423815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:52.381704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637318+0000","last_change":"2024-04-17T10:24:59.645993+0000","last_active":"2024-04-17T10:25:01.637318+0000","last_peered":"2024-04-17T10:25:01.637318+0000","last_clean":"2024-04-17T10:25:01.637318+0000","last_became_active":"2024-04-17T10:24:59.645305+0000","last_became_peered":"2024-04-17T10:24:59.645305+0000","last_unstale":"2024-04-17T10:25:01.637318+0000","last_undegraded":"2024-04-17T10:25:01.637318+0000","last_fullsized":"2024-04-17T10:25:01.637318+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638467+0000","last_change":"2024-04-17T10:24:55.614771+0000","last_active":"2024-04-17T10:25:01.638467+0000","last_peered":"2024-04-17T10:25:01.638467+0000","last_clean":"2024-04-17T10:25:01.638467+0000","last_became_active":"2024-04-17T10:24:55.614517+0000","last_became_peered":"2024-04-17T10:24:55.614517+0000","last_unstale":"2024-04-17T10:25:01.638467+0000","last_undegraded":"2024-04-17T10:25:01.638467+0000","last_fullsized":"2024-04-17T10:25:01.638467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636723+0000","last_change":"2024-04-17T10:24:54.608749+0000","last_active":"2024-04-17T10:25:01.636723+0000","last_peered":"2024-04-17T10:25:01.636723+0000","last_clean":"2024-04-17T10:25:01.636723+0000","last_became_active":"2024-04-17T10:24:54.606617+0000","last_became_peered":"2024-04-17T10:24:54.606617+0000","last_unstale":"2024-04-17T10:25:01.636723+0000","last_undegraded":"2024-04-17T10:25:01.636723+0000","last_fullsized":"2024-04-17T10:25:01.636723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636946+0000","last_change":"2024-04-17T10:24:59.645457+0000","last_active":"2024-04-17T10:25:01.636946+0000","last_peered":"2024-04-17T10:25:01.636946+0000","last_clean":"2024-04-17T10:25:01.636946+0000","last_became_active":"2024-04-17T10:24:59.644747+0000","last_became_peered":"2024-04-17T10:24:59.644747+0000","last_unstale":"2024-04-17T10:25:01.636946+0000","last_undegraded":"2024-04-17T10:25:01.636946+0000","last_fullsized":"2024-04-17T10:25:01.636946+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637820+0000","last_change":"2024-04-17T10:24:55.630638+0000","last_active":"2024-04-17T10:25:01.637820+0000","last_peered":"2024-04-17T10:25:01.637820+0000","last_clean":"2024-04-17T10:25:01.637820+0000","last_became_active":"2024-04-17T10:24:55.630351+0000","last_became_peered":"2024-04-17T10:24:55.630351+0000","last_unstale":"2024-04-17T10:25:01.637820+0000","last_undegraded":"2024-04-17T10:25:01.637820+0000","last_fullsized":"2024-04-17T10:25:01.637820+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636842+0000","last_change":"2024-04-17T10:24:54.609635+0000","last_active":"2024-04-17T10:25:01.636842+0000","last_peered":"2024-04-17T10:25:01.636842+0000","last_clean":"2024-04-17T10:25:01.636842+0000","last_became_active":"2024-04-17T10:24:54.607298+0000","last_became_peered":"2024-04-17T10:24:54.607298+0000","last_unstale":"2024-04-17T10:25:01.636842+0000","last_undegraded":"2024-04-17T10:25:01.636842+0000","last_fullsized":"2024-04-17T10:25:01.636842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637200+0000","last_change":"2024-04-17T10:25:00.243112+0000","last_active":"2024-04-17T10:25:01.637200+0000","last_peered":"2024-04-17T10:25:01.637200+0000","last_clean":"2024-04-17T10:25:01.637200+0000","last_became_active":"2024-04-17T10:25:00.242669+0000","last_became_peered":"2024-04-17T10:25:00.242669+0000","last_unstale":"2024-04-17T10:25:01.637200+0000","last_undegraded":"2024-04-17T10:25:01.637200+0000","last_fullsized":"2024-04-17T10:25:01.637200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637134+0000","last_change":"2024-04-17T10:24:55.608101+0000","last_active":"2024-04-17T10:25:01.637134+0000","last_peered":"2024-04-17T10:25:01.637134+0000","last_clean":"2024-04-17T10:25:01.637134+0000","last_became_active":"2024-04-17T10:24:55.607668+0000","last_became_peered":"2024-04-17T10:24:55.607668+0000","last_unstale":"2024-04-17T10:25:01.637134+0000","last_undegraded":"2024-04-17T10:25:01.637134+0000","last_fullsized":"2024-04-17T10:25:01.637134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:35.913082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423205+0000","last_change":"2024-04-17T10:24:54.607981+0000","last_active":"2024-04-17T10:25:02.423205+0000","last_peered":"2024-04-17T10:25:02.423205+0000","last_clean":"2024-04-17T10:25:02.423205+0000","last_became_active":"2024-04-17T10:24:54.607150+0000","last_became_peered":"2024-04-17T10:24:54.607150+0000","last_unstale":"2024-04-17T10:25:02.423205+0000","last_undegraded":"2024-04-17T10:25:02.423205+0000","last_fullsized":"2024-04-17T10:25:02.423205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636873+0000","last_change":"2024-04-17T10:24:59.645029+0000","last_active":"2024-04-17T10:25:01.636873+0000","last_peered":"2024-04-17T10:25:01.636873+0000","last_clean":"2024-04-17T10:25:01.636873+0000","last_became_active":"2024-04-17T10:24:59.644540+0000","last_became_peered":"2024-04-17T10:24:59.644540+0000","last_unstale":"2024-04-17T10:25:01.636873+0000","last_undegraded":"2024-04-17T10:25:01.636873+0000","last_fullsized":"2024-04-17T10:25:01.636873+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156445+0000","last_change":"2024-04-17T10:24:55.610920+0000","last_active":"2024-04-17T10:25:03.156445+0000","last_peered":"2024-04-17T10:25:03.156445+0000","last_clean":"2024-04-17T10:25:03.156445+0000","last_became_active":"2024-04-17T10:24:55.610566+0000","last_became_peered":"2024-04-17T10:24:55.610566+0000","last_unstale":"2024-04-17T10:25:03.156445+0000","last_undegraded":"2024-04-17T10:25:03.156445+0000","last_fullsized":"2024-04-17T10:25:03.156445+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537952+0000","last_change":"2024-04-17T10:24:54.609866+0000","last_active":"2024-04-17T10:25:02.537952+0000","last_peered":"2024-04-17T10:25:02.537952+0000","last_clean":"2024-04-17T10:25:02.537952+0000","last_became_active":"2024-04-17T10:24:54.607310+0000","last_became_peered":"2024-04-17T10:24:54.607310+0000","last_unstale":"2024-04-17T10:25:02.537952+0000","last_undegraded":"2024-04-17T10:25:02.537952+0000","last_fullsized":"2024-04-17T10:25:02.537952+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637020+0000","last_change":"2024-04-17T10:25:00.242151+0000","last_active":"2024-04-17T10:25:01.637020+0000","last_peered":"2024-04-17T10:25:01.637020+0000","last_clean":"2024-04-17T10:25:01.637020+0000","last_became_active":"2024-04-17T10:25:00.241789+0000","last_became_peered":"2024-04-17T10:25:00.241789+0000","last_unstale":"2024-04-17T10:25:01.637020+0000","last_undegraded":"2024-04-17T10:25:01.637020+0000","last_fullsized":"2024-04-17T10:25:01.637020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638591+0000","last_change":"2024-04-17T10:24:55.611037+0000","last_active":"2024-04-17T10:25:01.638591+0000","last_peered":"2024-04-17T10:25:01.638591+0000","last_clean":"2024-04-17T10:25:01.638591+0000","last_became_active":"2024-04-17T10:24:55.610713+0000","last_became_peered":"2024-04-17T10:24:55.610713+0000","last_unstale":"2024-04-17T10:25:01.638591+0000","last_undegraded":"2024-04-17T10:25:01.638591+0000","last_fullsized":"2024-04-17T10:25:01.638591+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636990+0000","last_change":"2024-04-17T10:24:54.614299+0000","last_active":"2024-04-17T10:25:01.636990+0000","last_peered":"2024-04-17T10:25:01.636990+0000","last_clean":"2024-04-17T10:25:01.636990+0000","last_became_active":"2024-04-17T10:24:54.613957+0000","last_became_peered":"2024-04-17T10:24:54.613957+0000","last_unstale":"2024-04-17T10:25:01.636990+0000","last_undegraded":"2024-04-17T10:25:01.636990+0000","last_fullsized":"2024-04-17T10:25:01.636990+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:31.252041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637684+0000","last_change":"2024-04-17T10:25:00.241327+0000","last_active":"2024-04-17T10:25:01.637684+0000","last_peered":"2024-04-17T10:25:01.637684+0000","last_clean":"2024-04-17T10:25:01.637684+0000","last_became_active":"2024-04-17T10:25:00.240884+0000","last_became_peered":"2024-04-17T10:25:00.240884+0000","last_unstale":"2024-04-17T10:25:01.637684+0000","last_undegraded":"2024-04-17T10:25:01.637684+0000","last_fullsized":"2024-04-17T10:25:01.637684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:26:46.933770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.624025+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.623725+0000","last_became_peered":"2024-04-17T10:24:55.623725+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424055+0000","last_change":"2024-04-17T10:24:54.610613+0000","last_active":"2024-04-17T10:25:02.424055+0000","last_peered":"2024-04-17T10:25:02.424055+0000","last_clean":"2024-04-17T10:25:02.424055+0000","last_became_active":"2024-04-17T10:24:54.609405+0000","last_became_peered":"2024-04-17T10:24:54.609405+0000","last_unstale":"2024-04-17T10:25:02.424055+0000","last_undegraded":"2024-04-17T10:25:02.424055+0000","last_fullsized":"2024-04-17T10:25:02.424055+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:06.248042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155827+0000","last_change":"2024-04-17T10:25:00.243396+0000","last_active":"2024-04-17T10:25:03.155827+0000","last_peered":"2024-04-17T10:25:03.155827+0000","last_clean":"2024-04-17T10:25:03.155827+0000","last_became_active":"2024-04-17T10:25:00.243086+0000","last_became_peered":"2024-04-17T10:25:00.243086+0000","last_unstale":"2024-04-17T10:25:03.155827+0000","last_undegraded":"2024-04-17T10:25:03.155827+0000","last_fullsized":"2024-04-17T10:25:03.155827+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268735+0000","last_change":"2024-04-17T10:24:55.611872+0000","last_active":"2024-04-17T10:25:03.268735+0000","last_peered":"2024-04-17T10:25:03.268735+0000","last_clean":"2024-04-17T10:25:03.268735+0000","last_became_active":"2024-04-17T10:24:55.607201+0000","last_became_peered":"2024-04-17T10:24:55.607201+0000","last_unstale":"2024-04-17T10:25:03.268735+0000","last_undegraded":"2024-04-17T10:25:03.268735+0000","last_fullsized":"2024-04-17T10:25:03.268735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637222+0000","last_change":"2024-04-17T10:24:54.598254+0000","last_active":"2024-04-17T10:25:01.637222+0000","last_peered":"2024-04-17T10:25:01.637222+0000","last_clean":"2024-04-17T10:25:01.637222+0000","last_became_active":"2024-04-17T10:24:54.597869+0000","last_became_peered":"2024-04-17T10:24:54.597869+0000","last_unstale":"2024-04-17T10:25:01.637222+0000","last_undegraded":"2024-04-17T10:25:01.637222+0000","last_fullsized":"2024-04-17T10:25:01.637222+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636567+0000","last_change":"2024-04-17T10:24:59.656986+0000","last_active":"2024-04-17T10:25:01.636567+0000","last_peered":"2024-04-17T10:25:01.636567+0000","last_clean":"2024-04-17T10:25:01.636567+0000","last_became_active":"2024-04-17T10:24:59.656124+0000","last_became_peered":"2024-04-17T10:24:59.656124+0000","last_unstale":"2024-04-17T10:25:01.636567+0000","last_undegraded":"2024-04-17T10:25:01.636567+0000","last_fullsized":"2024-04-17T10:25:01.636567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637349+0000","last_change":"2024-04-17T10:24:55.625808+0000","last_active":"2024-04-17T10:25:01.637349+0000","last_peered":"2024-04-17T10:25:01.637349+0000","last_clean":"2024-04-17T10:25:01.637349+0000","last_became_active":"2024-04-17T10:24:55.625551+0000","last_became_peered":"2024-04-17T10:24:55.625551+0000","last_unstale":"2024-04-17T10:25:01.637349+0000","last_undegraded":"2024-04-17T10:25:01.637349+0000","last_fullsized":"2024-04-17T10:25:01.637349+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267481+0000","last_change":"2024-04-17T10:24:54.603615+0000","last_active":"2024-04-17T10:25:03.267481+0000","last_peered":"2024-04-17T10:25:03.267481+0000","last_clean":"2024-04-17T10:25:03.267481+0000","last_became_active":"2024-04-17T10:24:54.600774+0000","last_became_peered":"2024-04-17T10:24:54.600774+0000","last_unstale":"2024-04-17T10:25:03.267481+0000","last_undegraded":"2024-04-17T10:25:03.267481+0000","last_fullsized":"2024-04-17T10:25:03.267481+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637164+0000","last_change":"2024-04-17T10:24:59.646484+0000","last_active":"2024-04-17T10:25:01.637164+0000","last_peered":"2024-04-17T10:25:01.637164+0000","last_clean":"2024-04-17T10:25:01.637164+0000","last_became_active":"2024-04-17T10:24:59.646061+0000","last_became_peered":"2024-04-17T10:24:59.646061+0000","last_unstale":"2024-04-17T10:25:01.637164+0000","last_undegraded":"2024-04-17T10:25:01.637164+0000","last_fullsized":"2024-04-17T10:25:01.637164+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636232+0000","last_change":"2024-04-17T10:24:55.624915+0000","last_active":"2024-04-17T10:25:01.636232+0000","last_peered":"2024-04-17T10:25:01.636232+0000","last_clean":"2024-04-17T10:25:01.636232+0000","last_became_active":"2024-04-17T10:24:55.624230+0000","last_became_peered":"2024-04-17T10:24:55.624230+0000","last_unstale":"2024-04-17T10:25:01.636232+0000","last_undegraded":"2024-04-17T10:25:01.636232+0000","last_fullsized":"2024-04-17T10:25:01.636232+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624893+0000","last_change":"2024-04-17T10:24:54.599766+0000","last_active":"2024-04-17T10:25:02.624893+0000","last_peered":"2024-04-17T10:25:02.624893+0000","last_clean":"2024-04-17T10:25:02.624893+0000","last_became_active":"2024-04-17T10:24:54.599476+0000","last_became_peered":"2024-04-17T10:24:54.599476+0000","last_unstale":"2024-04-17T10:25:02.624893+0000","last_undegraded":"2024-04-17T10:25:02.624893+0000","last_fullsized":"2024-04-17T10:25:02.624893+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637263+0000","last_change":"2024-04-17T10:24:59.656597+0000","last_active":"2024-04-17T10:25:01.637263+0000","last_peered":"2024-04-17T10:25:01.637263+0000","last_clean":"2024-04-17T10:25:01.637263+0000","last_became_active":"2024-04-17T10:24:59.656135+0000","last_became_peered":"2024-04-17T10:24:59.656135+0000","last_unstale":"2024-04-17T10:25:01.637263+0000","last_undegraded":"2024-04-17T10:25:01.637263+0000","last_fullsized":"2024-04-17T10:25:01.637263+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155614+0000","last_change":"2024-04-17T10:24:55.591986+0000","last_active":"2024-04-17T10:25:03.155614+0000","last_peered":"2024-04-17T10:25:03.155614+0000","last_clean":"2024-04-17T10:25:03.155614+0000","last_became_active":"2024-04-17T10:24:55.591672+0000","last_became_peered":"2024-04-17T10:24:55.591672+0000","last_unstale":"2024-04-17T10:25:03.155614+0000","last_undegraded":"2024-04-17T10:25:03.155614+0000","last_fullsized":"2024-04-17T10:25:03.155614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423263+0000","last_change":"2024-04-17T10:24:54.614030+0000","last_active":"2024-04-17T10:25:02.423263+0000","last_peered":"2024-04-17T10:25:02.423263+0000","last_clean":"2024-04-17T10:25:02.423263+0000","last_became_active":"2024-04-17T10:24:54.613752+0000","last_became_peered":"2024-04-17T10:24:54.613752+0000","last_unstale":"2024-04-17T10:25:02.423263+0000","last_undegraded":"2024-04-17T10:25:02.423263+0000","last_fullsized":"2024-04-17T10:25:02.423263+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156369+0000","last_change":"2024-04-17T10:25:00.244705+0000","last_active":"2024-04-17T10:25:03.156369+0000","last_peered":"2024-04-17T10:25:03.156369+0000","last_clean":"2024-04-17T10:25:03.156369+0000","last_became_active":"2024-04-17T10:25:00.243853+0000","last_became_peered":"2024-04-17T10:25:00.243853+0000","last_unstale":"2024-04-17T10:25:03.156369+0000","last_undegraded":"2024-04-17T10:25:03.156369+0000","last_fullsized":"2024-04-17T10:25:03.156369+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638050+0000","last_change":"2024-04-17T10:24:55.622500+0000","last_active":"2024-04-17T10:25:01.638050+0000","last_peered":"2024-04-17T10:25:01.638050+0000","last_clean":"2024-04-17T10:25:01.638050+0000","last_became_active":"2024-04-17T10:24:55.622206+0000","last_became_peered":"2024-04-17T10:24:55.622206+0000","last_unstale":"2024-04-17T10:25:01.638050+0000","last_undegraded":"2024-04-17T10:25:01.638050+0000","last_fullsized":"2024-04-17T10:25:01.638050+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268327+0000","last_change":"2024-04-17T10:24:54.601255+0000","last_active":"2024-04-17T10:25:03.268327+0000","last_peered":"2024-04-17T10:25:03.268327+0000","last_clean":"2024-04-17T10:25:03.268327+0000","last_became_active":"2024-04-17T10:24:54.600968+0000","last_became_peered":"2024-04-17T10:24:54.600968+0000","last_unstale":"2024-04-17T10:25:03.268327+0000","last_undegraded":"2024-04-17T10:25:03.268327+0000","last_fullsized":"2024-04-17T10:25:03.268327+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":180,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423358+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:25:02.423358+0000","last_peered":"2024-04-17T10:25:02.423358+0000","last_clean":"2024-04-17T10:25:02.423358+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:25:02.423358+0000","last_undegraded":"2024-04-17T10:25:02.423358+0000","last_fullsized":"2024-04-17T10:25:02.423358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537629+0000","last_change":"2024-04-17T10:25:00.247011+0000","last_active":"2024-04-17T10:25:02.537629+0000","last_peered":"2024-04-17T10:25:02.537629+0000","last_clean":"2024-04-17T10:25:02.537629+0000","last_became_active":"2024-04-17T10:25:00.245780+0000","last_became_peered":"2024-04-17T10:25:00.245780+0000","last_unstale":"2024-04-17T10:25:02.537629+0000","last_undegraded":"2024-04-17T10:25:02.537629+0000","last_fullsized":"2024-04-17T10:25:02.537629+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423766+0000","last_change":"2024-04-17T10:24:55.613528+0000","last_active":"2024-04-17T10:25:02.423766+0000","last_peered":"2024-04-17T10:25:02.423766+0000","last_clean":"2024-04-17T10:25:02.423766+0000","last_became_active":"2024-04-17T10:24:55.613203+0000","last_became_peered":"2024-04-17T10:24:55.613203+0000","last_unstale":"2024-04-17T10:25:02.423766+0000","last_undegraded":"2024-04-17T10:25:02.423766+0000","last_fullsized":"2024-04-17T10:25:02.423766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:15.389666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625258+0000","last_change":"2024-04-17T10:24:54.605199+0000","last_active":"2024-04-17T10:25:02.625258+0000","last_peered":"2024-04-17T10:25:02.625258+0000","last_clean":"2024-04-17T10:25:02.625258+0000","last_became_active":"2024-04-17T10:24:54.604881+0000","last_became_peered":"2024-04-17T10:24:54.604881+0000","last_unstale":"2024-04-17T10:25:02.625258+0000","last_undegraded":"2024-04-17T10:25:02.625258+0000","last_fullsized":"2024-04-17T10:25:02.625258+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537487+0000","last_change":"2024-04-17T10:25:00.242786+0000","last_active":"2024-04-17T10:25:02.537487+0000","last_peered":"2024-04-17T10:25:02.537487+0000","last_clean":"2024-04-17T10:25:02.537487+0000","last_became_active":"2024-04-17T10:25:00.242222+0000","last_became_peered":"2024-04-17T10:25:00.242222+0000","last_unstale":"2024-04-17T10:25:02.537487+0000","last_undegraded":"2024-04-17T10:25:02.537487+0000","last_fullsized":"2024-04-17T10:25:02.537487+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:01.409834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268628+0000","last_change":"2024-04-17T10:24:55.606154+0000","last_active":"2024-04-17T10:25:03.268628+0000","last_peered":"2024-04-17T10:25:03.268628+0000","last_clean":"2024-04-17T10:25:03.268628+0000","last_became_active":"2024-04-17T10:24:55.605854+0000","last_became_peered":"2024-04-17T10:24:55.605854+0000","last_unstale":"2024-04-17T10:25:03.268628+0000","last_undegraded":"2024-04-17T10:25:03.268628+0000","last_fullsized":"2024-04-17T10:25:03.268628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636665+0000","last_change":"2024-04-17T10:24:54.605549+0000","last_active":"2024-04-17T10:25:01.636665+0000","last_peered":"2024-04-17T10:25:01.636665+0000","last_clean":"2024-04-17T10:25:01.636665+0000","last_became_active":"2024-04-17T10:24:54.605197+0000","last_became_peered":"2024-04-17T10:24:54.605197+0000","last_unstale":"2024-04-17T10:25:01.636665+0000","last_undegraded":"2024-04-17T10:25:01.636665+0000","last_fullsized":"2024-04-17T10:25:01.636665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:50.077939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155960+0000","last_change":"2024-04-17T10:24:59.652908+0000","last_active":"2024-04-17T10:25:03.155960+0000","last_peered":"2024-04-17T10:25:03.155960+0000","last_clean":"2024-04-17T10:25:03.155960+0000","last_became_active":"2024-04-17T10:24:59.652604+0000","last_became_peered":"2024-04-17T10:24:59.652604+0000","last_unstale":"2024-04-17T10:25:03.155960+0000","last_undegraded":"2024-04-17T10:25:03.155960+0000","last_fullsized":"2024-04-17T10:25:03.155960+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:06.099388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424244+0000","last_change":"2024-04-17T10:24:55.612380+0000","last_active":"2024-04-17T10:25:02.424244+0000","last_peered":"2024-04-17T10:25:02.424244+0000","last_clean":"2024-04-17T10:25:02.424244+0000","last_became_active":"2024-04-17T10:24:55.611511+0000","last_became_peered":"2024-04-17T10:24:55.611511+0000","last_unstale":"2024-04-17T10:25:02.424244+0000","last_undegraded":"2024-04-17T10:25:02.424244+0000","last_fullsized":"2024-04-17T10:25:02.424244+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:49.176910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624922+0000","last_change":"2024-04-17T10:24:54.601414+0000","last_active":"2024-04-17T10:25:02.624922+0000","last_peered":"2024-04-17T10:25:02.624922+0000","last_clean":"2024-04-17T10:25:02.624922+0000","last_became_active":"2024-04-17T10:24:54.600297+0000","last_became_peered":"2024-04-17T10:24:54.600297+0000","last_unstale":"2024-04-17T10:25:02.624922+0000","last_undegraded":"2024-04-17T10:25:02.624922+0000","last_fullsized":"2024-04-17T10:25:02.624922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267910+0000","last_change":"2024-04-17T10:25:00.242716+0000","last_active":"2024-04-17T10:25:03.267910+0000","last_peered":"2024-04-17T10:25:03.267910+0000","last_clean":"2024-04-17T10:25:03.267910+0000","last_became_active":"2024-04-17T10:25:00.242357+0000","last_became_peered":"2024-04-17T10:25:00.242357+0000","last_unstale":"2024-04-17T10:25:03.267910+0000","last_undegraded":"2024-04-17T10:25:03.267910+0000","last_fullsized":"2024-04-17T10:25:03.267910+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:34:53.216553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424088+0000","last_change":"2024-04-17T10:24:55.604575+0000","last_active":"2024-04-17T10:25:02.424088+0000","last_peered":"2024-04-17T10:25:02.424088+0000","last_clean":"2024-04-17T10:25:02.424088+0000","last_became_active":"2024-04-17T10:24:55.604252+0000","last_became_peered":"2024-04-17T10:24:55.604252+0000","last_unstale":"2024-04-17T10:25:02.424088+0000","last_undegraded":"2024-04-17T10:25:02.424088+0000","last_fullsized":"2024-04-17T10:25:02.424088+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:02.059745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637103+0000","last_change":"2024-04-17T10:24:54.595839+0000","last_active":"2024-04-17T10:25:01.637103+0000","last_peered":"2024-04-17T10:25:01.637103+0000","last_clean":"2024-04-17T10:25:01.637103+0000","last_became_active":"2024-04-17T10:24:54.594547+0000","last_became_peered":"2024-04-17T10:24:54.594547+0000","last_unstale":"2024-04-17T10:25:01.637103+0000","last_undegraded":"2024-04-17T10:25:01.637103+0000","last_fullsized":"2024-04-17T10:25:01.637103+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:19:57.725700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636525+0000","last_change":"2024-04-17T10:24:59.644365+0000","last_active":"2024-04-17T10:25:01.636525+0000","last_peered":"2024-04-17T10:25:01.636525+0000","last_clean":"2024-04-17T10:25:01.636525+0000","last_became_active":"2024-04-17T10:24:59.643969+0000","last_became_peered":"2024-04-17T10:24:59.643969+0000","last_unstale":"2024-04-17T10:25:01.636525+0000","last_undegraded":"2024-04-17T10:25:01.636525+0000","last_fullsized":"2024-04-17T10:25:01.636525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:56.978034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423432+0000","last_change":"2024-04-17T10:24:55.598627+0000","last_active":"2024-04-17T10:25:02.423432+0000","last_peered":"2024-04-17T10:25:02.423432+0000","last_clean":"2024-04-17T10:25:02.423432+0000","last_became_active":"2024-04-17T10:24:55.598319+0000","last_became_peered":"2024-04-17T10:24:55.598319+0000","last_unstale":"2024-04-17T10:25:02.423432+0000","last_undegraded":"2024-04-17T10:25:02.423432+0000","last_fullsized":"2024-04-17T10:25:02.423432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:25:12.768604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":32,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.624900+0000","last_change":"2024-04-17T10:24:54.600358+0000","last_active":"2024-04-17T10:25:02.624900+0000","last_peered":"2024-04-17T10:25:02.624900+0000","last_clean":"2024-04-17T10:25:02.624900+0000","last_became_active":"2024-04-17T10:24:54.600064+0000","last_became_peered":"2024-04-17T10:24:54.600064+0000","last_unstale":"2024-04-17T10:25:02.624900+0000","last_undegraded":"2024-04-17T10:25:02.624900+0000","last_fullsized":"2024-04-17T10:25:02.624900+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156321+0000","last_change":"2024-04-17T10:24:59.651264+0000","last_active":"2024-04-17T10:25:03.156321+0000","last_peered":"2024-04-17T10:25:03.156321+0000","last_clean":"2024-04-17T10:25:03.156321+0000","last_became_active":"2024-04-17T10:24:59.650654+0000","last_became_peered":"2024-04-17T10:24:59.650654+0000","last_unstale":"2024-04-17T10:25:03.156321+0000","last_undegraded":"2024-04-17T10:25:03.156321+0000","last_fullsized":"2024-04-17T10:25:03.156321+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:52.622003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537254+0000","last_change":"2024-04-17T10:24:55.608832+0000","last_active":"2024-04-17T10:25:02.537254+0000","last_peered":"2024-04-17T10:25:02.537254+0000","last_clean":"2024-04-17T10:25:02.537254+0000","last_became_active":"2024-04-17T10:24:55.608402+0000","last_became_peered":"2024-04-17T10:24:55.608402+0000","last_unstale":"2024-04-17T10:25:02.537254+0000","last_undegraded":"2024-04-17T10:25:02.537254+0000","last_fullsized":"2024-04-17T10:25:02.537254+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537354+0000","last_change":"2024-04-17T10:24:54.601967+0000","last_active":"2024-04-17T10:25:02.537354+0000","last_peered":"2024-04-17T10:25:02.537354+0000","last_clean":"2024-04-17T10:25:02.537354+0000","last_became_active":"2024-04-17T10:24:54.601643+0000","last_became_peered":"2024-04-17T10:24:54.601643+0000","last_unstale":"2024-04-17T10:25:02.537354+0000","last_undegraded":"2024-04-17T10:25:02.537354+0000","last_fullsized":"2024-04-17T10:25:02.537354+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638493+0000","last_change":"2024-04-17T10:25:00.243903+0000","last_active":"2024-04-17T10:25:01.638493+0000","last_peered":"2024-04-17T10:25:01.638493+0000","last_clean":"2024-04-17T10:25:01.638493+0000","last_became_active":"2024-04-17T10:25:00.243126+0000","last_became_peered":"2024-04-17T10:25:00.243126+0000","last_unstale":"2024-04-17T10:25:01.638493+0000","last_undegraded":"2024-04-17T10:25:01.638493+0000","last_fullsized":"2024-04-17T10:25:01.638493+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:44.715063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423617+0000","last_change":"2024-04-17T10:24:55.598632+0000","last_active":"2024-04-17T10:25:02.423617+0000","last_peered":"2024-04-17T10:25:02.423617+0000","last_clean":"2024-04-17T10:25:02.423617+0000","last_became_active":"2024-04-17T10:24:55.598302+0000","last_became_peered":"2024-04-17T10:24:55.598302+0000","last_unstale":"2024-04-17T10:25:02.423617+0000","last_undegraded":"2024-04-17T10:25:02.423617+0000","last_fullsized":"2024-04-17T10:25:02.423617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267532+0000","last_change":"2024-04-17T10:24:54.603334+0000","last_active":"2024-04-17T10:25:03.267532+0000","last_peered":"2024-04-17T10:25:03.267532+0000","last_clean":"2024-04-17T10:25:03.267532+0000","last_became_active":"2024-04-17T10:24:54.603066+0000","last_became_peered":"2024-04-17T10:24:54.603066+0000","last_unstale":"2024-04-17T10:25:03.267532+0000","last_undegraded":"2024-04-17T10:25:03.267532+0000","last_fullsized":"2024-04-17T10:25:03.267532+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:14.224161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639048+0000","last_change":"2024-04-17T10:25:00.247479+0000","last_active":"2024-04-17T10:25:01.639048+0000","last_peered":"2024-04-17T10:25:01.639048+0000","last_clean":"2024-04-17T10:25:01.639048+0000","last_became_active":"2024-04-17T10:25:00.247066+0000","last_became_peered":"2024-04-17T10:25:00.247066+0000","last_unstale":"2024-04-17T10:25:01.639048+0000","last_undegraded":"2024-04-17T10:25:01.639048+0000","last_fullsized":"2024-04-17T10:25:01.639048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:10:54.855530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155768+0000","last_change":"2024-04-17T10:24:55.616361+0000","last_active":"2024-04-17T10:25:03.155768+0000","last_peered":"2024-04-17T10:25:03.155768+0000","last_clean":"2024-04-17T10:25:03.155768+0000","last_became_active":"2024-04-17T10:24:55.615902+0000","last_became_peered":"2024-04-17T10:24:55.615902+0000","last_unstale":"2024-04-17T10:25:03.155768+0000","last_undegraded":"2024-04-17T10:25:03.155768+0000","last_fullsized":"2024-04-17T10:25:03.155768+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:53.388065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155191+0000","last_change":"2024-04-17T10:24:54.599143+0000","last_active":"2024-04-17T10:25:03.155191+0000","last_peered":"2024-04-17T10:25:03.155191+0000","last_clean":"2024-04-17T10:25:03.155191+0000","last_became_active":"2024-04-17T10:24:54.598831+0000","last_became_peered":"2024-04-17T10:24:54.598831+0000","last_unstale":"2024-04-17T10:25:03.155191+0000","last_undegraded":"2024-04-17T10:25:03.155191+0000","last_fullsized":"2024-04-17T10:25:03.155191+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268682+0000","last_change":"2024-04-17T10:25:00.243074+0000","last_active":"2024-04-17T10:25:03.268682+0000","last_peered":"2024-04-17T10:25:03.268682+0000","last_clean":"2024-04-17T10:25:03.268682+0000","last_became_active":"2024-04-17T10:25:00.242532+0000","last_became_peered":"2024-04-17T10:25:00.242532+0000","last_unstale":"2024-04-17T10:25:03.268682+0000","last_undegraded":"2024-04-17T10:25:03.268682+0000","last_fullsized":"2024-04-17T10:25:03.268682+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:43.233154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423646+0000","last_change":"2024-04-17T10:24:55.603471+0000","last_active":"2024-04-17T10:25:02.423646+0000","last_peered":"2024-04-17T10:25:02.423646+0000","last_clean":"2024-04-17T10:25:02.423646+0000","last_became_active":"2024-04-17T10:24:55.603177+0000","last_became_peered":"2024-04-17T10:24:55.603177+0000","last_unstale":"2024-04-17T10:25:02.423646+0000","last_undegraded":"2024-04-17T10:25:02.423646+0000","last_fullsized":"2024-04-17T10:25:02.423646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:52.938391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'10","reported_seq":34,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:10.622101+0000","last_change":"2024-04-17T10:24:54.599941+0000","last_active":"2024-04-17T10:25:10.622101+0000","last_peered":"2024-04-17T10:25:10.622101+0000","last_clean":"2024-04-17T10:25:10.622101+0000","last_became_active":"2024-04-17T10:24:54.599658+0000","last_became_peered":"2024-04-17T10:24:54.599658+0000","last_unstale":"2024-04-17T10:25:10.622101+0000","last_undegraded":"2024-04-17T10:25:10.622101+0000","last_fullsized":"2024-04-17T10:25:10.622101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:58.254984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637141+0000","last_change":"2024-04-17T10:24:59.657550+0000","last_active":"2024-04-17T10:25:01.637141+0000","last_peered":"2024-04-17T10:25:01.637141+0000","last_clean":"2024-04-17T10:25:01.637141+0000","last_became_active":"2024-04-17T10:24:59.657179+0000","last_became_peered":"2024-04-17T10:24:59.657179+0000","last_unstale":"2024-04-17T10:25:01.637141+0000","last_undegraded":"2024-04-17T10:25:01.637141+0000","last_fullsized":"2024-04-17T10:25:01.637141+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:43.786019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637399+0000","last_change":"2024-04-17T10:24:55.609933+0000","last_active":"2024-04-17T10:25:01.637399+0000","last_peered":"2024-04-17T10:25:01.637399+0000","last_clean":"2024-04-17T10:25:01.637399+0000","last_became_active":"2024-04-17T10:24:55.602879+0000","last_became_peered":"2024-04-17T10:24:55.602879+0000","last_unstale":"2024-04-17T10:25:01.637399+0000","last_undegraded":"2024-04-17T10:25:01.637399+0000","last_fullsized":"2024-04-17T10:25:01.637399+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:51.084300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636629+0000","last_change":"2024-04-17T10:24:54.607532+0000","last_active":"2024-04-17T10:25:01.636629+0000","last_peered":"2024-04-17T10:25:01.636629+0000","last_clean":"2024-04-17T10:25:01.636629+0000","last_became_active":"2024-04-17T10:24:54.606527+0000","last_became_peered":"2024-04-17T10:24:54.606527+0000","last_unstale":"2024-04-17T10:25:01.636629+0000","last_undegraded":"2024-04-17T10:25:01.636629+0000","last_fullsized":"2024-04-17T10:25:01.636629+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:55.276780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637464+0000","last_change":"2024-04-17T10:24:59.644478+0000","last_active":"2024-04-17T10:25:01.637464+0000","last_peered":"2024-04-17T10:25:01.637464+0000","last_clean":"2024-04-17T10:25:01.637464+0000","last_became_active":"2024-04-17T10:24:59.644083+0000","last_became_peered":"2024-04-17T10:24:59.644083+0000","last_unstale":"2024-04-17T10:25:01.637464+0000","last_undegraded":"2024-04-17T10:25:01.637464+0000","last_fullsized":"2024-04-17T10:25:01.637464+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:15:04.444750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156377+0000","last_change":"2024-04-17T10:24:55.627359+0000","last_active":"2024-04-17T10:25:03.156377+0000","last_peered":"2024-04-17T10:25:03.156377+0000","last_clean":"2024-04-17T10:25:03.156377+0000","last_became_active":"2024-04-17T10:24:55.627094+0000","last_became_peered":"2024-04-17T10:24:55.627094+0000","last_unstale":"2024-04-17T10:25:03.156377+0000","last_undegraded":"2024-04-17T10:25:03.156377+0000","last_fullsized":"2024-04-17T10:25:03.156377+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267575+0000","last_change":"2024-04-17T10:24:54.613073+0000","last_active":"2024-04-17T10:25:03.267575+0000","last_peered":"2024-04-17T10:25:03.267575+0000","last_clean":"2024-04-17T10:25:03.267575+0000","last_became_active":"2024-04-17T10:24:54.612771+0000","last_became_peered":"2024-04-17T10:24:54.612771+0000","last_unstale":"2024-04-17T10:25:03.267575+0000","last_undegraded":"2024-04-17T10:25:03.267575+0000","last_fullsized":"2024-04-17T10:25:03.267575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:44.176582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637853+0000","last_change":"2024-04-17T10:24:59.653549+0000","last_active":"2024-04-17T10:25:01.637853+0000","last_peered":"2024-04-17T10:25:01.637853+0000","last_clean":"2024-04-17T10:25:01.637853+0000","last_became_active":"2024-04-17T10:24:59.652615+0000","last_became_peered":"2024-04-17T10:24:59.652615+0000","last_unstale":"2024-04-17T10:25:01.637853+0000","last_undegraded":"2024-04-17T10:25:01.637853+0000","last_fullsized":"2024-04-17T10:25:01.637853+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:43.760818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268787+0000","last_change":"2024-04-17T10:24:55.612177+0000","last_active":"2024-04-17T10:25:03.268787+0000","last_peered":"2024-04-17T10:25:03.268787+0000","last_clean":"2024-04-17T10:25:03.268787+0000","last_became_active":"2024-04-17T10:24:55.611847+0000","last_became_peered":"2024-04-17T10:24:55.611847+0000","last_unstale":"2024-04-17T10:25:03.268787+0000","last_undegraded":"2024-04-17T10:25:03.268787+0000","last_fullsized":"2024-04-17T10:25:03.268787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:00.256447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537797+0000","last_change":"2024-04-17T10:24:54.614002+0000","last_active":"2024-04-17T10:25:02.537797+0000","last_peered":"2024-04-17T10:25:02.537797+0000","last_clean":"2024-04-17T10:25:02.537797+0000","last_became_active":"2024-04-17T10:24:54.613675+0000","last_became_peered":"2024-04-17T10:24:54.613675+0000","last_unstale":"2024-04-17T10:25:02.537797+0000","last_undegraded":"2024-04-17T10:25:02.537797+0000","last_fullsized":"2024-04-17T10:25:02.537797+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:26.758124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637973+0000","last_change":"2024-04-17T10:25:00.244266+0000","last_active":"2024-04-17T10:25:01.637973+0000","last_peered":"2024-04-17T10:25:01.637973+0000","last_clean":"2024-04-17T10:25:01.637973+0000","last_became_active":"2024-04-17T10:25:00.243271+0000","last_became_peered":"2024-04-17T10:25:00.243271+0000","last_unstale":"2024-04-17T10:25:01.637973+0000","last_undegraded":"2024-04-17T10:25:01.637973+0000","last_fullsized":"2024-04-17T10:25:01.637973+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:37:59.616402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156261+0000","last_change":"2024-04-17T10:25:00.244894+0000","last_active":"2024-04-17T10:25:03.156261+0000","last_peered":"2024-04-17T10:25:03.156261+0000","last_clean":"2024-04-17T10:25:03.156261+0000","last_became_active":"2024-04-17T10:25:00.244366+0000","last_became_peered":"2024-04-17T10:25:00.244366+0000","last_unstale":"2024-04-17T10:25:03.156261+0000","last_undegraded":"2024-04-17T10:25:03.156261+0000","last_fullsized":"2024-04-17T10:25:03.156261+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:11.875242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636436+0000","last_change":"2024-04-17T10:24:55.610258+0000","last_active":"2024-04-17T10:25:01.636436+0000","last_peered":"2024-04-17T10:25:01.636436+0000","last_clean":"2024-04-17T10:25:01.636436+0000","last_became_active":"2024-04-17T10:24:55.609904+0000","last_became_peered":"2024-04-17T10:24:55.609904+0000","last_unstale":"2024-04-17T10:25:01.636436+0000","last_undegraded":"2024-04-17T10:25:01.636436+0000","last_fullsized":"2024-04-17T10:25:01.636436+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267470+0000","last_change":"2024-04-17T10:24:54.605205+0000","last_active":"2024-04-17T10:25:03.267470+0000","last_peered":"2024-04-17T10:25:03.267470+0000","last_clean":"2024-04-17T10:25:03.267470+0000","last_became_active":"2024-04-17T10:24:54.601015+0000","last_became_peered":"2024-04-17T10:24:54.601015+0000","last_unstale":"2024-04-17T10:25:03.267470+0000","last_undegraded":"2024-04-17T10:25:03.267470+0000","last_fullsized":"2024-04-17T10:25:03.267470+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:08.462618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537785+0000","last_change":"2024-04-17T10:24:59.648566+0000","last_active":"2024-04-17T10:25:02.537785+0000","last_peered":"2024-04-17T10:25:02.537785+0000","last_clean":"2024-04-17T10:25:02.537785+0000","last_became_active":"2024-04-17T10:24:59.648170+0000","last_became_peered":"2024-04-17T10:24:59.648170+0000","last_unstale":"2024-04-17T10:25:02.537785+0000","last_undegraded":"2024-04-17T10:25:02.537785+0000","last_fullsized":"2024-04-17T10:25:02.537785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:08:19.429309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637759+0000","last_change":"2024-04-17T10:24:55.611737+0000","last_active":"2024-04-17T10:25:01.637759+0000","last_peered":"2024-04-17T10:25:01.637759+0000","last_clean":"2024-04-17T10:25:01.637759+0000","last_became_active":"2024-04-17T10:24:55.611353+0000","last_became_peered":"2024-04-17T10:24:55.611353+0000","last_unstale":"2024-04-17T10:25:01.637759+0000","last_undegraded":"2024-04-17T10:25:01.637759+0000","last_fullsized":"2024-04-17T10:25:01.637759+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:21:34.037393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638133+0000","last_change":"2024-04-17T10:24:54.601357+0000","last_active":"2024-04-17T10:25:01.638133+0000","last_peered":"2024-04-17T10:25:01.638133+0000","last_clean":"2024-04-17T10:25:01.638133+0000","last_became_active":"2024-04-17T10:24:54.601111+0000","last_became_peered":"2024-04-17T10:24:54.601111+0000","last_unstale":"2024-04-17T10:25:01.638133+0000","last_undegraded":"2024-04-17T10:25:01.638133+0000","last_fullsized":"2024-04-17T10:25:01.638133+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:33.735217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537510+0000","last_change":"2024-04-17T10:24:54.601574+0000","last_active":"2024-04-17T10:25:02.537510+0000","last_peered":"2024-04-17T10:25:02.537510+0000","last_clean":"2024-04-17T10:25:02.537510+0000","last_became_active":"2024-04-17T10:24:54.601261+0000","last_became_peered":"2024-04-17T10:24:54.601261+0000","last_unstale":"2024-04-17T10:25:02.537510+0000","last_undegraded":"2024-04-17T10:25:02.537510+0000","last_fullsized":"2024-04-17T10:25:02.537510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:29:34.906721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156049+0000","last_change":"2024-04-17T10:24:55.604048+0000","last_active":"2024-04-17T10:25:03.156049+0000","last_peered":"2024-04-17T10:25:03.156049+0000","last_clean":"2024-04-17T10:25:03.156049+0000","last_became_active":"2024-04-17T10:24:55.592091+0000","last_became_peered":"2024-04-17T10:24:55.592091+0000","last_unstale":"2024-04-17T10:25:03.156049+0000","last_undegraded":"2024-04-17T10:25:03.156049+0000","last_fullsized":"2024-04-17T10:25:03.156049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625146+0000","last_change":"2024-04-17T10:24:59.656792+0000","last_active":"2024-04-17T10:25:02.625146+0000","last_peered":"2024-04-17T10:25:02.625146+0000","last_clean":"2024-04-17T10:25:02.625146+0000","last_became_active":"2024-04-17T10:24:59.656244+0000","last_became_peered":"2024-04-17T10:24:59.656244+0000","last_unstale":"2024-04-17T10:25:02.625146+0000","last_undegraded":"2024-04-17T10:25:02.625146+0000","last_fullsized":"2024-04-17T10:25:02.625146+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:09:57.427708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638566+0000","last_change":"2024-04-17T10:24:54.609412+0000","last_active":"2024-04-17T10:25:01.638566+0000","last_peered":"2024-04-17T10:25:01.638566+0000","last_clean":"2024-04-17T10:25:01.638566+0000","last_became_active":"2024-04-17T10:24:54.609119+0000","last_became_peered":"2024-04-17T10:24:54.609119+0000","last_unstale":"2024-04-17T10:25:01.638566+0000","last_undegraded":"2024-04-17T10:25:01.638566+0000","last_fullsized":"2024-04-17T10:25:01.638566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:53.565410+0000","last_clean_scrub_stamp":"2024-04-17T10:24:53.565410+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:48.047038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636952+0000","last_change":"2024-04-17T10:24:55.614982+0000","last_active":"2024-04-17T10:25:01.636952+0000","last_peered":"2024-04-17T10:25:01.636952+0000","last_clean":"2024-04-17T10:25:01.636952+0000","last_became_active":"2024-04-17T10:24:55.614619+0000","last_became_peered":"2024-04-17T10:24:55.614619+0000","last_unstale":"2024-04-17T10:25:01.636952+0000","last_undegraded":"2024-04-17T10:25:01.636952+0000","last_fullsized":"2024-04-17T10:25:01.636952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:32.258402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638839+0000","last_change":"2024-04-17T10:24:59.647972+0000","last_active":"2024-04-17T10:25:01.638839+0000","last_peered":"2024-04-17T10:25:01.638839+0000","last_clean":"2024-04-17T10:25:01.638839+0000","last_became_active":"2024-04-17T10:24:59.646737+0000","last_became_peered":"2024-04-17T10:24:59.646737+0000","last_unstale":"2024-04-17T10:25:01.638839+0000","last_undegraded":"2024-04-17T10:25:01.638839+0000","last_fullsized":"2024-04-17T10:25:01.638839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:58.611414+0000","last_clean_scrub_stamp":"2024-04-17T10:24:58.611414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:59.695291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537104+0000","last_change":"2024-04-17T10:24:55.608996+0000","last_active":"2024-04-17T10:25:02.537104+0000","last_peered":"2024-04-17T10:25:02.537104+0000","last_clean":"2024-04-17T10:25:02.537104+0000","last_became_active":"2024-04-17T10:24:55.608621+0000","last_became_peered":"2024-04-17T10:24:55.608621+0000","last_unstale":"2024-04-17T10:25:02.537104+0000","last_undegraded":"2024-04-17T10:25:02.537104+0000","last_fullsized":"2024-04-17T10:25:02.537104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:10.506244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625124+0000","last_change":"2024-04-17T10:24:55.616866+0000","last_active":"2024-04-17T10:25:02.625124+0000","last_peered":"2024-04-17T10:25:02.625124+0000","last_clean":"2024-04-17T10:25:02.625124+0000","last_became_active":"2024-04-17T10:24:55.616582+0000","last_became_peered":"2024-04-17T10:24:55.616582+0000","last_unstale":"2024-04-17T10:25:02.625124+0000","last_undegraded":"2024-04-17T10:25:02.625124+0000","last_fullsized":"2024-04-17T10:25:02.625124+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:27:45.182942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268379+0000","last_change":"2024-04-17T10:24:55.613768+0000","last_active":"2024-04-17T10:25:03.268379+0000","last_peered":"2024-04-17T10:25:03.268379+0000","last_clean":"2024-04-17T10:25:03.268379+0000","last_became_active":"2024-04-17T10:24:55.612427+0000","last_became_peered":"2024-04-17T10:24:55.612427+0000","last_unstale":"2024-04-17T10:25:03.268379+0000","last_undegraded":"2024-04-17T10:25:03.268379+0000","last_fullsized":"2024-04-17T10:25:03.268379+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:51.322375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155940+0000","last_change":"2024-04-17T10:24:55.613422+0000","last_active":"2024-04-17T10:25:03.155940+0000","last_peered":"2024-04-17T10:25:03.155940+0000","last_clean":"2024-04-17T10:25:03.155940+0000","last_became_active":"2024-04-17T10:24:55.612627+0000","last_became_peered":"2024-04-17T10:24:55.612627+0000","last_unstale":"2024-04-17T10:25:03.155940+0000","last_undegraded":"2024-04-17T10:25:03.155940+0000","last_fullsized":"2024-04-17T10:25:03.155940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:17.677815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.639265+0000","last_change":"2024-04-17T10:24:55.617532+0000","last_active":"2024-04-17T10:25:01.639265+0000","last_peered":"2024-04-17T10:25:01.639265+0000","last_clean":"2024-04-17T10:25:01.639265+0000","last_became_active":"2024-04-17T10:24:55.617215+0000","last_became_peered":"2024-04-17T10:24:55.617215+0000","last_unstale":"2024-04-17T10:25:01.639265+0000","last_undegraded":"2024-04-17T10:25:01.639265+0000","last_fullsized":"2024-04-17T10:25:01.639265+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636559+0000","last_change":"2024-04-17T10:24:55.631795+0000","last_active":"2024-04-17T10:25:01.636559+0000","last_peered":"2024-04-17T10:25:01.636559+0000","last_clean":"2024-04-17T10:25:01.636559+0000","last_became_active":"2024-04-17T10:24:55.629826+0000","last_became_peered":"2024-04-17T10:24:55.629826+0000","last_unstale":"2024-04-17T10:25:01.636559+0000","last_undegraded":"2024-04-17T10:25:01.636559+0000","last_fullsized":"2024-04-17T10:25:01.636559+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537858+0000","last_change":"2024-04-17T10:24:55.606321+0000","last_active":"2024-04-17T10:25:02.537858+0000","last_peered":"2024-04-17T10:25:02.537858+0000","last_clean":"2024-04-17T10:25:02.537858+0000","last_became_active":"2024-04-17T10:24:55.605895+0000","last_became_peered":"2024-04-17T10:24:55.605895+0000","last_unstale":"2024-04-17T10:25:02.537858+0000","last_undegraded":"2024-04-17T10:25:02.537858+0000","last_fullsized":"2024-04-17T10:25:02.537858+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638540+0000","last_change":"2024-04-17T10:24:55.607147+0000","last_active":"2024-04-17T10:25:01.638540+0000","last_peered":"2024-04-17T10:25:01.638540+0000","last_clean":"2024-04-17T10:25:01.638540+0000","last_became_active":"2024-04-17T10:24:55.606931+0000","last_became_peered":"2024-04-17T10:24:55.606931+0000","last_unstale":"2024-04-17T10:25:01.638540+0000","last_undegraded":"2024-04-17T10:25:01.638540+0000","last_fullsized":"2024-04-17T10:25:01.638540+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268051+0000","last_change":"2024-04-17T10:24:55.609356+0000","last_active":"2024-04-17T10:25:03.268051+0000","last_peered":"2024-04-17T10:25:03.268051+0000","last_clean":"2024-04-17T10:25:03.268051+0000","last_became_active":"2024-04-17T10:24:55.607716+0000","last_became_peered":"2024-04-17T10:24:55.607716+0000","last_unstale":"2024-04-17T10:25:03.268051+0000","last_undegraded":"2024-04-17T10:25:03.268051+0000","last_fullsized":"2024-04-17T10:25:03.268051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638038+0000","last_change":"2024-04-17T10:24:55.605676+0000","last_active":"2024-04-17T10:25:01.638038+0000","last_peered":"2024-04-17T10:25:01.638038+0000","last_clean":"2024-04-17T10:25:01.638038+0000","last_became_active":"2024-04-17T10:24:55.605431+0000","last_became_peered":"2024-04-17T10:24:55.605431+0000","last_unstale":"2024-04-17T10:25:01.638038+0000","last_undegraded":"2024-04-17T10:25:01.638038+0000","last_fullsized":"2024-04-17T10:25:01.638038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268286+0000","last_change":"2024-04-17T10:24:55.615398+0000","last_active":"2024-04-17T10:25:03.268286+0000","last_peered":"2024-04-17T10:25:03.268286+0000","last_clean":"2024-04-17T10:25:03.268286+0000","last_became_active":"2024-04-17T10:24:55.615034+0000","last_became_peered":"2024-04-17T10:24:55.615034+0000","last_unstale":"2024-04-17T10:25:03.268286+0000","last_undegraded":"2024-04-17T10:25:03.268286+0000","last_fullsized":"2024-04-17T10:25:03.268286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:11.162543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155772+0000","last_change":"2024-04-17T10:24:55.613776+0000","last_active":"2024-04-17T10:25:03.155772+0000","last_peered":"2024-04-17T10:25:03.155772+0000","last_clean":"2024-04-17T10:25:03.155772+0000","last_became_active":"2024-04-17T10:24:55.613106+0000","last_became_peered":"2024-04-17T10:24:55.613106+0000","last_unstale":"2024-04-17T10:25:03.155772+0000","last_undegraded":"2024-04-17T10:25:03.155772+0000","last_fullsized":"2024-04-17T10:25:03.155772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537535+0000","last_change":"2024-04-17T10:24:55.619933+0000","last_active":"2024-04-17T10:25:02.537535+0000","last_peered":"2024-04-17T10:25:02.537535+0000","last_clean":"2024-04-17T10:25:02.537535+0000","last_became_active":"2024-04-17T10:24:55.619666+0000","last_became_peered":"2024-04-17T10:24:55.619666+0000","last_unstale":"2024-04-17T10:25:02.537535+0000","last_undegraded":"2024-04-17T10:25:02.537535+0000","last_fullsized":"2024-04-17T10:25:02.537535+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:57:49.810919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423902+0000","last_change":"2024-04-17T10:24:55.608090+0000","last_active":"2024-04-17T10:25:02.423902+0000","last_peered":"2024-04-17T10:25:02.423902+0000","last_clean":"2024-04-17T10:25:02.423902+0000","last_became_active":"2024-04-17T10:24:55.607715+0000","last_became_peered":"2024-04-17T10:24:55.607715+0000","last_unstale":"2024-04-17T10:25:02.423902+0000","last_undegraded":"2024-04-17T10:25:02.423902+0000","last_fullsized":"2024-04-17T10:25:02.423902+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:30:07.835599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.625372+0000","last_change":"2024-04-17T10:24:55.616021+0000","last_active":"2024-04-17T10:25:02.625372+0000","last_peered":"2024-04-17T10:25:02.625372+0000","last_clean":"2024-04-17T10:25:02.625372+0000","last_became_active":"2024-04-17T10:24:55.615736+0000","last_became_peered":"2024-04-17T10:24:55.615736+0000","last_unstale":"2024-04-17T10:25:02.625372+0000","last_undegraded":"2024-04-17T10:25:02.625372+0000","last_fullsized":"2024-04-17T10:25:02.625372+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268965+0000","last_change":"2024-04-17T10:24:55.614983+0000","last_active":"2024-04-17T10:25:03.268965+0000","last_peered":"2024-04-17T10:25:03.268965+0000","last_clean":"2024-04-17T10:25:03.268965+0000","last_became_active":"2024-04-17T10:24:55.614651+0000","last_became_peered":"2024-04-17T10:24:55.614651+0000","last_unstale":"2024-04-17T10:25:03.268965+0000","last_undegraded":"2024-04-17T10:25:03.268965+0000","last_fullsized":"2024-04-17T10:25:03.268965+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:57.370191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.267935+0000","last_change":"2024-04-17T10:24:55.626976+0000","last_active":"2024-04-17T10:25:03.267935+0000","last_peered":"2024-04-17T10:25:03.267935+0000","last_clean":"2024-04-17T10:25:03.267935+0000","last_became_active":"2024-04-17T10:24:55.626743+0000","last_became_peered":"2024-04-17T10:24:55.626743+0000","last_unstale":"2024-04-17T10:25:03.267935+0000","last_undegraded":"2024-04-17T10:25:03.267935+0000","last_fullsized":"2024-04-17T10:25:03.267935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636737+0000","last_change":"2024-04-17T10:24:55.618634+0000","last_active":"2024-04-17T10:25:01.636737+0000","last_peered":"2024-04-17T10:25:01.636737+0000","last_clean":"2024-04-17T10:25:01.636737+0000","last_became_active":"2024-04-17T10:24:55.618343+0000","last_became_peered":"2024-04-17T10:24:55.618343+0000","last_unstale":"2024-04-17T10:25:01.636737+0000","last_undegraded":"2024-04-17T10:25:01.636737+0000","last_fullsized":"2024-04-17T10:25:01.636737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.155747+0000","last_change":"2024-04-17T10:24:55.616149+0000","last_active":"2024-04-17T10:25:03.155747+0000","last_peered":"2024-04-17T10:25:03.155747+0000","last_clean":"2024-04-17T10:25:03.155747+0000","last_became_active":"2024-04-17T10:24:55.615561+0000","last_became_peered":"2024-04-17T10:24:55.615561+0000","last_unstale":"2024-04-17T10:25:03.155747+0000","last_undegraded":"2024-04-17T10:25:03.155747+0000","last_fullsized":"2024-04-17T10:25:03.155747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:37.060791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.638375+0000","last_change":"2024-04-17T10:24:55.612680+0000","last_active":"2024-04-17T10:25:01.638375+0000","last_peered":"2024-04-17T10:25:01.638375+0000","last_clean":"2024-04-17T10:25:01.638375+0000","last_became_active":"2024-04-17T10:24:55.612280+0000","last_became_peered":"2024-04-17T10:24:55.612280+0000","last_unstale":"2024-04-17T10:25:01.638375+0000","last_undegraded":"2024-04-17T10:25:01.638375+0000","last_fullsized":"2024-04-17T10:25:01.638375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637130+0000","last_change":"2024-04-17T10:24:55.610235+0000","last_active":"2024-04-17T10:25:01.637130+0000","last_peered":"2024-04-17T10:25:01.637130+0000","last_clean":"2024-04-17T10:25:01.637130+0000","last_became_active":"2024-04-17T10:24:55.609934+0000","last_became_peered":"2024-04-17T10:24:55.609934+0000","last_unstale":"2024-04-17T10:25:01.637130+0000","last_undegraded":"2024-04-17T10:25:01.637130+0000","last_fullsized":"2024-04-17T10:25:01.637130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:15.230521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537674+0000","last_change":"2024-04-17T10:24:55.613780+0000","last_active":"2024-04-17T10:25:02.537674+0000","last_peered":"2024-04-17T10:25:02.537674+0000","last_clean":"2024-04-17T10:25:02.537674+0000","last_became_active":"2024-04-17T10:24:55.613360+0000","last_became_peered":"2024-04-17T10:24:55.613360+0000","last_unstale":"2024-04-17T10:25:02.537674+0000","last_undegraded":"2024-04-17T10:25:02.537674+0000","last_fullsized":"2024-04-17T10:25:02.537674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:08:37.384869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.537823+0000","last_change":"2024-04-17T10:24:55.617826+0000","last_active":"2024-04-17T10:25:02.537823+0000","last_peered":"2024-04-17T10:25:02.537823+0000","last_clean":"2024-04-17T10:25:02.537823+0000","last_became_active":"2024-04-17T10:24:55.617446+0000","last_became_peered":"2024-04-17T10:24:55.617446+0000","last_unstale":"2024-04-17T10:25:02.537823+0000","last_undegraded":"2024-04-17T10:25:02.537823+0000","last_fullsized":"2024-04-17T10:25:02.537823+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:31:16.859036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424189+0000","last_change":"2024-04-17T10:24:55.626063+0000","last_active":"2024-04-17T10:25:02.424189+0000","last_peered":"2024-04-17T10:25:02.424189+0000","last_clean":"2024-04-17T10:25:02.424189+0000","last_became_active":"2024-04-17T10:24:55.625720+0000","last_became_peered":"2024-04-17T10:24:55.625720+0000","last_unstale":"2024-04-17T10:25:02.424189+0000","last_undegraded":"2024-04-17T10:25:02.424189+0000","last_fullsized":"2024-04-17T10:25:02.424189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:54:39.746376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.637649+0000","last_change":"2024-04-17T10:24:55.622013+0000","last_active":"2024-04-17T10:25:01.637649+0000","last_peered":"2024-04-17T10:25:01.637649+0000","last_clean":"2024-04-17T10:25:01.637649+0000","last_became_active":"2024-04-17T10:24:55.618726+0000","last_became_peered":"2024-04-17T10:24:55.618726+0000","last_unstale":"2024-04-17T10:25:01.637649+0000","last_undegraded":"2024-04-17T10:25:01.637649+0000","last_fullsized":"2024-04-17T10:25:01.637649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:36:07.254403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423351+0000","last_change":"2024-04-17T10:24:55.614704+0000","last_active":"2024-04-17T10:25:02.423351+0000","last_peered":"2024-04-17T10:25:02.423351+0000","last_clean":"2024-04-17T10:25:02.423351+0000","last_became_active":"2024-04-17T10:24:55.614425+0000","last_became_peered":"2024-04-17T10:24:55.614425+0000","last_unstale":"2024-04-17T10:25:02.423351+0000","last_undegraded":"2024-04-17T10:25:02.423351+0000","last_fullsized":"2024-04-17T10:25:02.423351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:53.209901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268264+0000","last_change":"2024-04-17T10:24:55.630019+0000","last_active":"2024-04-17T10:25:03.268264+0000","last_peered":"2024-04-17T10:25:03.268264+0000","last_clean":"2024-04-17T10:25:03.268264+0000","last_became_active":"2024-04-17T10:24:55.629824+0000","last_became_peered":"2024-04-17T10:24:55.629824+0000","last_unstale":"2024-04-17T10:25:03.268264+0000","last_undegraded":"2024-04-17T10:25:03.268264+0000","last_fullsized":"2024-04-17T10:25:03.268264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:41:13.901064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.423511+0000","last_change":"2024-04-17T10:24:55.612346+0000","last_active":"2024-04-17T10:25:02.423511+0000","last_peered":"2024-04-17T10:25:02.423511+0000","last_clean":"2024-04-17T10:25:02.423511+0000","last_became_active":"2024-04-17T10:24:55.610079+0000","last_became_peered":"2024-04-17T10:24:55.610079+0000","last_unstale":"2024-04-17T10:25:02.423511+0000","last_undegraded":"2024-04-17T10:25:02.423511+0000","last_fullsized":"2024-04-17T10:25:02.423511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:20.908017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.268502+0000","last_change":"2024-04-17T10:24:55.612221+0000","last_active":"2024-04-17T10:25:03.268502+0000","last_peered":"2024-04-17T10:25:03.268502+0000","last_clean":"2024-04-17T10:25:03.268502+0000","last_became_active":"2024-04-17T10:24:55.611946+0000","last_became_peered":"2024-04-17T10:24:55.611946+0000","last_unstale":"2024-04-17T10:25:03.268502+0000","last_undegraded":"2024-04-17T10:25:03.268502+0000","last_fullsized":"2024-04-17T10:25:03.268502+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:30:56.510382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:03.156183+0000","last_change":"2024-04-17T10:24:55.617011+0000","last_active":"2024-04-17T10:25:03.156183+0000","last_peered":"2024-04-17T10:25:03.156183+0000","last_clean":"2024-04-17T10:25:03.156183+0000","last_became_active":"2024-04-17T10:24:55.616709+0000","last_became_peered":"2024-04-17T10:24:55.616709+0000","last_unstale":"2024-04-17T10:25:03.156183+0000","last_undegraded":"2024-04-17T10:25:03.156183+0000","last_fullsized":"2024-04-17T10:25:03.156183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:22:10.092171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:01.636934+0000","last_change":"2024-04-17T10:24:55.627305+0000","last_active":"2024-04-17T10:25:01.636934+0000","last_peered":"2024-04-17T10:25:01.636934+0000","last_clean":"2024-04-17T10:25:01.636934+0000","last_became_active":"2024-04-17T10:24:55.626978+0000","last_became_peered":"2024-04-17T10:24:55.626978+0000","last_unstale":"2024-04-17T10:25:01.636934+0000","last_undegraded":"2024-04-17T10:25:01.636934+0000","last_fullsized":"2024-04-17T10:25:01.636934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:51.295571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-17T10:25:02.424104+0000","last_change":"2024-04-17T10:24:55.613165+0000","last_active":"2024-04-17T10:25:02.424104+0000","last_peered":"2024-04-17T10:25:02.424104+0000","last_clean":"2024-04-17T10:25:02.424104+0000","last_became_active":"2024-04-17T10:24:55.611929+0000","last_became_peered":"2024-04-17T10:24:55.611929+0000","last_unstale":"2024-04-17T10:25:02.424104+0000","last_undegraded":"2024-04-17T10:25:02.424104+0000","last_fullsized":"2024-04-17T10:25:02.424104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:54.571771+0000","last_clean_scrub_stamp":"2024-04-17T10:24:54.571771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:42:48.582360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":864256,"data_stored":698343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":107483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":107519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":111235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":107393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343968,"statfs":{"total":96636764160,"available":96608223232,"internally_reserved":0,"allocated":872448,"data_stored":698399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:25:25.887 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-17T10:25:25.887 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-17T10:25:25.887 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-17T10:25:25.888 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-17T10:25:26.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec68800640 1 Processor -- start 2024-04-17T10:25:26.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec68800640 1 -- start start 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec68800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec601620a0 0x7fec6015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec68800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec6015e550 0x7fec600cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec68800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6015eb50 con 0x7fec600ce020 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec68800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec60161650 con 0x7fec601620a0 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec68800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec6015ee30 con 0x7fec6015e550 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec65400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec601620a0 0x7fec6015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec65400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec601620a0 0x7fec6015e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58366/0 (socket says 172.21.15.67:58366) 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec65400640 1 -- 172.21.15.67:0/3973873913 learned_addr learned my addr 172.21.15.67:0/3973873913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.067+0000 7fec66800640 1 --2- 172.21.15.67:0/3973873913 >> v2:172.21.15.86:3300/0 conn(0x7fec6015e550 0x7fec600cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec65400640 1 -- 172.21.15.67:0/3973873913 >> v2:172.21.15.86:3300/0 conn(0x7fec6015e550 msgr2=0x7fec600cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec65400640 1 --2- 172.21.15.67:0/3973873913 >> v2:172.21.15.86:3300/0 conn(0x7fec6015e550 0x7fec600cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec65400640 1 -- 172.21.15.67:0/3973873913 >> v1:172.21.15.86:6789/0 conn(0x7fec600ce020 legacy=0x7fec600ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec65400640 1 -- 172.21.15.67:0/3973873913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec601618a0 con 0x7fec601620a0 2024-04-17T10:25:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec66800640 1 --2- 172.21.15.67:0/3973873913 >> v2:172.21.15.86:3300/0 conn(0x7fec6015e550 0x7fec600cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec65400640 1 --2- 172.21.15.67:0/3973873913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6015e010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec50009870 tx=0x7fec5001c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a391ef36f72982 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec64a00640 1 -- 172.21.15.67:0/3973873913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec500b1070 con 0x7fec601620a0 2024-04-17T10:25:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec64a00640 1 -- 172.21.15.67:0/3973873913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec500ac620 con 0x7fec601620a0 2024-04-17T10:25:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec64a00640 1 -- 172.21.15.67:0/3973873913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec500ac9f0 con 0x7fec601620a0 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 -- 172.21.15.67:0/3973873913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 msgr2=0x7fec6015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 --2- 172.21.15.67:0/3973873913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6015e010 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fec50009870 tx=0x7fec5001c4b0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 -- 172.21.15.67:0/3973873913 shutdown_connections 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 --2- 172.21.15.67:0/3973873913 >> v2:172.21.15.86:3300/0 conn(0x7fec6015e550 0x7fec600cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 --2- 172.21.15.67:0/3973873913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6015e010 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 -- 172.21.15.67:0/3973873913 >> 172.21.15.67:0/3973873913 conn(0x7fec600b33f0 msgr2=0x7fec600b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 -- 172.21.15.67:0/3973873913 shutdown_connections 2024-04-17T10:25:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.071+0000 7fec68800640 1 -- 172.21.15.67:0/3973873913 wait complete. 2024-04-17T10:25:26.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 Processor -- start 2024-04-17T10:25:26.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 -- start start 2024-04-17T10:25:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec600ce020 0x7fec601604e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6015e550 0x7fec6014f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6014f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec600c8630 con 0x7fec601620a0 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec600c84b0 con 0x7fec600ce020 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6016a9f0 con 0x7fec6015e550 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6014f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6014f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58376/0 (socket says 172.21.15.67:58376) 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 -- 172.21.15.67:0/226699510 learned_addr learned my addr 172.21.15.67:0/226699510 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec65e00640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec600ce020 0x7fec601604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec65400640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6015e550 0x7fec6014f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 -- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6015e550 msgr2=0x7fec6014f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6015e550 0x7fec6014f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 -- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec600ce020 msgr2=0x7fec601604e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec600ce020 0x7fec601604e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 -- 172.21.15.67:0/226699510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec60153740 con 0x7fec601620a0 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec65400640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6015e550 0x7fec6014f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec66800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6014f5e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00cf20 tx=0x7fec5c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec5a000640 1 -- 172.21.15.67:0/226699510 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c01a070 con 0x7fec601620a0 2024-04-17T10:25:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec5a000640 1 -- 172.21.15.67:0/226699510 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c00d840 con 0x7fec601620a0 2024-04-17T10:25:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec5a000640 1 -- 172.21.15.67:0/226699510 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c00dc30 con 0x7fec601620a0 2024-04-17T10:25:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec60153a20 con 0x7fec601620a0 2024-04-17T10:25:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.075+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec60153fe0 con 0x7fec601620a0 2024-04-17T10:25:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.079+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec60070ab0 con 0x7fec601620a0 2024-04-17T10:25:26.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.079+0000 7fec5a000640 1 -- 172.21.15.67:0/226699510 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fec5c01d070 con 0x7fec601620a0 2024-04-17T10:25:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.079+0000 7fec5a000640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec44066f60 0x7fec44069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.079+0000 7fec5a000640 1 -- 172.21.15.67:0/226699510 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fec5c085bf0 con 0x7fec601620a0 2024-04-17T10:25:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.079+0000 7fec65e00640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec44066f60 0x7fec44069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.083+0000 7fec65e00640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec44066f60 0x7fec44069420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fec54007fd0 tx=0x7fec54007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.083+0000 7fec5a000640 1 -- 172.21.15.67:0/226699510 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fec5c0561d0 con 0x7fec601620a0 2024-04-17T10:25:26.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.279+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fec60074e30 con 0x7fec601620a0 2024-04-17T10:25:26.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.279+0000 7fec5a000640 1 -- 172.21.15.67:0/226699510 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fec5c05a270 con 0x7fec601620a0 2024-04-17T10:25:26.276 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:26.276 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-17T10:25:26.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec44066f60 msgr2=0x7fec44069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec44066f60 0x7fec44069420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fec54007fd0 tx=0x7fec54007e40 comp rx=0 tx=0).stop 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 msgr2=0x7fec6014f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6014f5e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fec5c00cf20 tx=0x7fec5c00a7d0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 shutdown_connections 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec44066f60 0x7fec44069420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec601620a0 0x7fec6014f5e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec6015e550 0x7fec6014f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 --2- 172.21.15.67:0/226699510 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec600ce020 0x7fec601604e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 >> 172.21.15.67:0/226699510 conn(0x7fec600b33f0 msgr2=0x7fec600ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 shutdown_connections 2024-04-17T10:25:26.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.283+0000 7fec68800640 1 -- 172.21.15.67:0/226699510 wait complete. 2024-04-17T10:25:26.298 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-17T10:25:26.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:26.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.443+0000 7f4f19600640 1 Processor -- start 2024-04-17T10:25:26.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.443+0000 7f4f19600640 1 -- start start 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f19600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f14099a50 0x7f4f1415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f19600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4f140d03a0 0x7f4f14099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f19600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f1415f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f19600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f140cde20 con 0x7f4f14099a50 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f19600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1409a370 con 0x7f4f140d03a0 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f19600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f140cf300 con 0x7f4f1409a740 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f14099a50 0x7f4f1415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4f14099a50 0x7f4f1415c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58392/0 (socket says 172.21.15.67:58392) 2024-04-17T10:25:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 -- 172.21.15.67:0/4080070337 learned_addr learned my addr 172.21.15.67:0/4080070337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f0b400640 1 --2- 172.21.15.67:0/4080070337 >> v2:172.21.15.86:3300/0 conn(0x7f4f140d03a0 0x7f4f14099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f18c00640 1 --2- 172.21.15.67:0/4080070337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f1415f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 -- 172.21.15.67:0/4080070337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 msgr2=0x7f4f1415f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 --2- 172.21.15.67:0/4080070337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f1415f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 -- 172.21.15.67:0/4080070337 >> v2:172.21.15.86:3300/0 conn(0x7f4f140d03a0 msgr2=0x7f4f14099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 --2- 172.21.15.67:0/4080070337 >> v2:172.21.15.86:3300/0 conn(0x7f4f140d03a0 0x7f4f14099e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f13e00640 1 -- 172.21.15.67:0/4080070337 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140cf5e0 con 0x7f4f14099a50 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f0b400640 1 --2- 172.21.15.67:0/4080070337 >> v2:172.21.15.86:3300/0 conn(0x7f4f140d03a0 0x7f4f14099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.447+0000 7f4f18c00640 1 --2- 172.21.15.67:0/4080070337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f1415f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f13e00640 1 --2- 172.21.15.67:0/4080070337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f14099a50 0x7f4f1415c7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4f1409acd0 tx=0x7f4f0401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5128739f28dd22ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f13400640 1 -- 172.21.15.67:0/4080070337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f040b0070 con 0x7f4f14099a50 2024-04-17T10:25:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f13400640 1 -- 172.21.15.67:0/4080070337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f040ab600 con 0x7f4f14099a50 2024-04-17T10:25:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f13400640 1 -- 172.21.15.67:0/4080070337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f040ab9d0 con 0x7f4f14099a50 2024-04-17T10:25:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- 172.21.15.67:0/4080070337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f14099a50 msgr2=0x7f4f1415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 --2- 172.21.15.67:0/4080070337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f14099a50 0x7f4f1415c7a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4f1409acd0 tx=0x7f4f0401c260 comp rx=0 tx=0).stop 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- 172.21.15.67:0/4080070337 shutdown_connections 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 --2- 172.21.15.67:0/4080070337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f1415f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 --2- 172.21.15.67:0/4080070337 >> v2:172.21.15.86:3300/0 conn(0x7f4f140d03a0 0x7f4f14099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 --2- 172.21.15.67:0/4080070337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f14099a50 0x7f4f1415c7a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- 172.21.15.67:0/4080070337 >> 172.21.15.67:0/4080070337 conn(0x7f4f140d1690 msgr2=0x7f4f140d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- 172.21.15.67:0/4080070337 shutdown_connections 2024-04-17T10:25:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- 172.21.15.67:0/4080070337 wait complete. 2024-04-17T10:25:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 Processor -- start 2024-04-17T10:25:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- start start 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f14099a50 0x7f4f140b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f14165110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f140d03a0 0x7f4f1414aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f140cc320 con 0x7f4f140d03a0 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f140cc1a0 con 0x7f4f14099a50 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f19600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f140cc4a0 con 0x7f4f1409a740 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f18c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f140d03a0 0x7f4f1414aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f18c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f140d03a0 0x7f4f1414aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58400/0 (socket says 172.21.15.67:58400) 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.451+0000 7f4f18c00640 1 -- 172.21.15.67:0/2212742822 learned_addr learned my addr 172.21.15.67:0/2212742822 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f0b400640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f14165110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f13e00640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f14099a50 0x7f4f140b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f18c00640 1 -- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 msgr2=0x7f4f14165110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f18c00640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f14165110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f18c00640 1 -- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f14099a50 msgr2=0x7f4f140b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f18c00640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f14099a50 0x7f4f140b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f18c00640 1 -- 172.21.15.67:0/2212742822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14165650 con 0x7f4f140d03a0 2024-04-17T10:25:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f18c00640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f140d03a0 0x7f4f1414aab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00cef0 tx=0x7f4f0c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f10c00640 1 -- 172.21.15.67:0/2212742822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c01a070 con 0x7f4f140d03a0 2024-04-17T10:25:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f10c00640 1 -- 172.21.15.67:0/2212742822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c00d840 con 0x7f4f140d03a0 2024-04-17T10:25:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f10c00640 1 -- 172.21.15.67:0/2212742822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c00dc30 con 0x7f4f140d03a0 2024-04-17T10:25:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f14165900 con 0x7f4f140d03a0 2024-04-17T10:25:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f14165ec0 con 0x7f4f140d03a0 2024-04-17T10:25:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f14074dc0 con 0x7f4f140d03a0 2024-04-17T10:25:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f10c00640 1 -- 172.21.15.67:0/2212742822 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4f0c01d070 con 0x7f4f140d03a0 2024-04-17T10:25:26.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f10c00640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ef0066f60 0x7f4ef0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.455+0000 7f4f10c00640 1 -- 172.21.15.67:0/2212742822 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f4f0c085b60 con 0x7f4f140d03a0 2024-04-17T10:25:26.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.459+0000 7f4f13e00640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ef0066f60 0x7f4ef0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.459+0000 7f4f13e00640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ef0066f60 0x7f4ef0069420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4f040ad000 tx=0x7f4f040a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.459+0000 7f4f10c00640 1 -- 172.21.15.67:0/2212742822 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4f0c056830 con 0x7f4f140d03a0 2024-04-17T10:25:26.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.627+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f140d24a0 con 0x7f4f140d03a0 2024-04-17T10:25:26.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.627+0000 7f4f10c00640 1 -- 172.21.15.67:0/2212742822 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7f4f0c056130 con 0x7f4f140d03a0 2024-04-17T10:25:26.626 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:26.626 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ef0066f60 msgr2=0x7f4ef0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ef0066f60 0x7f4ef0069420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4f040ad000 tx=0x7f4f040a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f140d03a0 msgr2=0x7f4f1414aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f140d03a0 0x7f4f1414aab0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00cef0 tx=0x7f4f0c00a710 comp rx=0 tx=0).stop 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 shutdown_connections 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ef0066f60 0x7f4ef0069420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4f140d03a0 0x7f4f1414aab0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4f1409a740 0x7f4f14165110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 --2- 172.21.15.67:0/2212742822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4f14099a50 0x7f4f140b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 >> 172.21.15.67:0/2212742822 conn(0x7f4f140d1690 msgr2=0x7f4f140ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:26.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 shutdown_connections 2024-04-17T10:25:26.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.631+0000 7f4f19600640 1 -- 172.21.15.67:0/2212742822 wait complete. 2024-04-17T10:25:26.645 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:26.645 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 Processor -- start 2024-04-17T10:25:26.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 -- start start 2024-04-17T10:25:26.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe80cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fe80cce60 0x7f7fe806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7fe806e7b0 0x7f7fe80713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe80cd2c0 con 0x7f7fe80cce60 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe80af340 con 0x7f7fe806e7b0 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fef800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe80b4990 con 0x7f7fe80aef60 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fe80cce60 0x7f7fe806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fe80cce60 0x7f7fe806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58404/0 (socket says 172.21.15.67:58404) 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 -- 172.21.15.67:0/2687523503 learned_addr learned my addr 172.21.15.67:0/2687523503 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fece00640 1 --2- 172.21.15.67:0/2687523503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe80cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fed800640 1 --2- 172.21.15.67:0/2687523503 >> v2:172.21.15.86:3300/0 conn(0x7f7fe806e7b0 0x7f7fe80713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 -- 172.21.15.67:0/2687523503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 msgr2=0x7f7fe80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 --2- 172.21.15.67:0/2687523503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe80cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 -- 172.21.15.67:0/2687523503 >> v2:172.21.15.86:3300/0 conn(0x7f7fe806e7b0 msgr2=0x7f7fe80713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 --2- 172.21.15.67:0/2687523503 >> v2:172.21.15.86:3300/0 conn(0x7f7fe806e7b0 0x7f7fe80713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.795+0000 7f7fdfe00640 1 -- 172.21.15.67:0/2687523503 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe815ae90 con 0x7f7fe80cce60 2024-04-17T10:25:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fdfe00640 1 --2- 172.21.15.67:0/2687523503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe80cce60 0x7f7fe806e270 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80b2900 tx=0x7f7fd001e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea15928eb4735b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fe7e00640 1 -- 172.21.15.67:0/2687523503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd00b4070 con 0x7f7fe80cce60 2024-04-17T10:25:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fe7e00640 1 -- 172.21.15.67:0/2687523503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd00a7840 con 0x7f7fe80cce60 2024-04-17T10:25:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fe7e00640 1 -- 172.21.15.67:0/2687523503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd00a7c10 con 0x7f7fe80cce60 2024-04-17T10:25:26.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 -- 172.21.15.67:0/2687523503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe80cce60 msgr2=0x7f7fe806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 --2- 172.21.15.67:0/2687523503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe80cce60 0x7f7fe806e270 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7fe80b2900 tx=0x7f7fd001e8a0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 -- 172.21.15.67:0/2687523503 shutdown_connections 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 --2- 172.21.15.67:0/2687523503 >> v2:172.21.15.86:3300/0 conn(0x7f7fe806e7b0 0x7f7fe80713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 --2- 172.21.15.67:0/2687523503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe80cce60 0x7f7fe806e270 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 --2- 172.21.15.67:0/2687523503 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe80cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 -- 172.21.15.67:0/2687523503 >> 172.21.15.67:0/2687523503 conn(0x7f7fe80b1740 msgr2=0x7f7fe80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 -- 172.21.15.67:0/2687523503 shutdown_connections 2024-04-17T10:25:26.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.799+0000 7f7fef800640 1 -- 172.21.15.67:0/2687523503 wait complete. 2024-04-17T10:25:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 Processor -- start 2024-04-17T10:25:26.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 -- start start 2024-04-17T10:25:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe806e7b0 0x7f7fe80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe80cce60 0x7f7fe815db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe815bc10 con 0x7f7fe806e7b0 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe815b410 con 0x7f7fe80cce60 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe8148150 con 0x7f7fe80aef60 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe806e7b0 0x7f7fe80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe806e7b0 0x7f7fe80ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58406/0 (socket says 172.21.15.67:58406) 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 -- 172.21.15.67:0/3382896559 learned_addr learned my addr 172.21.15.67:0/3382896559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fed800640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe80cce60 0x7f7fe815db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fdfe00640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 -- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 msgr2=0x7f7fe815d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe815d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 -- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe80cce60 msgr2=0x7f7fe815db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe80cce60 0x7f7fe815db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 -- 172.21.15.67:0/3382896559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe8148430 con 0x7f7fe806e7b0 2024-04-17T10:25:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fece00640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe806e7b0 0x7f7fe80ade30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800d800 tx=0x7f7fd800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fe5600640 1 -- 172.21.15.67:0/3382896559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800b660 con 0x7f7fe806e7b0 2024-04-17T10:25:26.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fe5600640 1 -- 172.21.15.67:0/3382896559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8019070 con 0x7f7fe806e7b0 2024-04-17T10:25:26.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe8148710 con 0x7f7fe806e7b0 2024-04-17T10:25:26.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fe5600640 1 -- 172.21.15.67:0/3382896559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800ba00 con 0x7f7fe806e7b0 2024-04-17T10:25:26.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe8148cd0 con 0x7f7fe806e7b0 2024-04-17T10:25:26.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.803+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe804ef30 con 0x7f7fe806e7b0 2024-04-17T10:25:26.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.807+0000 7f7fe5600640 1 -- 172.21.15.67:0/3382896559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7fd800bba0 con 0x7f7fe806e7b0 2024-04-17T10:25:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.807+0000 7f7fe5600640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fc4066f60 0x7f7fc4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.807+0000 7f7fe5600640 1 -- 172.21.15.67:0/3382896559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7fd80856c0 con 0x7f7fe806e7b0 2024-04-17T10:25:26.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.807+0000 7f7fdfe00640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fc4066f60 0x7f7fc4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:26.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.807+0000 7f7fdfe00640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fc4066f60 0x7f7fc4069420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7fd00b1000 tx=0x7f7fd0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.811+0000 7f7fe5600640 1 -- 172.21.15.67:0/3382896559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7fd8055d30 con 0x7f7fe806e7b0 2024-04-17T10:25:26.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7fe815c0c0 con 0x7f7fe806e7b0 2024-04-17T10:25:26.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fe5600640 1 -- 172.21.15.67:0/3382896559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7fd8059dd0 con 0x7f7fe806e7b0 2024-04-17T10:25:26.978 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fc4066f60 msgr2=0x7f7fc4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fc4066f60 0x7f7fc4069420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7fd00b1000 tx=0x7f7fd0008040 comp rx=0 tx=0).stop 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe806e7b0 msgr2=0x7f7fe80ade30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe806e7b0 0x7f7fe80ade30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800d800 tx=0x7f7fd800dcc0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 shutdown_connections 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fc4066f60 0x7f7fc4069420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe80cce60 0x7f7fe815db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe80aef60 0x7f7fe815d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 --2- 172.21.15.67:0/3382896559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe806e7b0 0x7f7fe80ade30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:26.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 >> 172.21.15.67:0/3382896559 conn(0x7f7fe80b1740 msgr2=0x7f7fe80b0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 shutdown_connections 2024-04-17T10:25:26.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:26.983+0000 7f7fef800640 1 -- 172.21.15.67:0/3382896559 wait complete. 2024-04-17T10:25:26.998 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:27.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15e0a00640 1 Processor -- start 2024-04-17T10:25:27.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15e0a00640 1 -- start start 2024-04-17T10:25:27.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15e0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06d810 0x7f15dc0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15e0a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f15dc0707c0 0x7f15dc15d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15e0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc0d03c0 con 0x7f15dc06f8b0 2024-04-17T10:25:27.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15e0a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15dc0d06a0 con 0x7f15dc0707c0 2024-04-17T10:25:27.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15e0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc0d0980 con 0x7f15dc06d810 2024-04-17T10:25:27.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15d3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f15dc06f8b0 0x7f15dc0700b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54434/0 (socket says 172.21.15.67:54434) 2024-04-17T10:25:27.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15d3e00640 1 -- 172.21.15.67:0/3169294214 learned_addr learned my addr 172.21.15.67:0/3169294214 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:27.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15da800640 1 --2- 172.21.15.67:0/3169294214 >> v2:172.21.15.86:3300/0 conn(0x7f15dc0707c0 0x7f15dc15d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15d9e00640 1 --2- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06d810 0x7f15dc0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15d9400640 1 -- 172.21.15.67:0/3169294214 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017899882 0 0) 0x7f15dc0d03c0 con 0x7f15dc06f8b0 2024-04-17T10:25:27.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15d9400640 1 -- 172.21.15.67:0/3169294214 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c0003530 con 0x7f15dc06f8b0 2024-04-17T10:25:27.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15da800640 1 -- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06d810 msgr2=0x7f15dc0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:27.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15da800640 1 --2- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06d810 0x7f15dc0cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15da800640 1 -- 172.21.15.67:0/3169294214 >> v1:172.21.15.67:6789/0 conn(0x7f15dc06f8b0 legacy=0x7f15dc0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.147+0000 7f15da800640 1 -- 172.21.15.67:0/3169294214 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0d0c60 con 0x7f15dc0707c0 2024-04-17T10:25:27.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15da800640 1 --2- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc0707c0 0x7f15dc15d470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009d90 tx=0x7f15d401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0337636a461c75d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15d9400640 1 -- 172.21.15.67:0/3169294214 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d40b2070 con 0x7f15dc0707c0 2024-04-17T10:25:27.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15d9400640 1 -- 172.21.15.67:0/3169294214 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d40a5720 con 0x7f15dc0707c0 2024-04-17T10:25:27.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15d9400640 1 -- 172.21.15.67:0/3169294214 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d40a5af0 con 0x7f15dc0707c0 2024-04-17T10:25:27.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15e0a00640 1 -- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc0707c0 msgr2=0x7f15dc15d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15e0a00640 1 --2- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc0707c0 0x7f15dc15d470 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009d90 tx=0x7f15d401c940 comp rx=0 tx=0).stop 2024-04-17T10:25:27.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15e0a00640 1 -- 172.21.15.67:0/3169294214 shutdown_connections 2024-04-17T10:25:27.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15e0a00640 1 --2- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc0707c0 0x7f15dc15d470 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15e0a00640 1 --2- 172.21.15.67:0/3169294214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06d810 0x7f15dc0cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.151+0000 7f15e0a00640 1 -- 172.21.15.67:0/3169294214 >> 172.21.15.67:0/3169294214 conn(0x7f15dc0d1800 msgr2=0x7f15dc0d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:27.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 -- 172.21.15.67:0/3169294214 shutdown_connections 2024-04-17T10:25:27.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 -- 172.21.15.67:0/3169294214 wait complete. 2024-04-17T10:25:27.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 Processor -- start 2024-04-17T10:25:27.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 -- start start 2024-04-17T10:25:27.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc06d810 0x7f15dc153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06f8b0 0x7f15dc15c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0707c0 0x7f15dc1f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc06bda0 con 0x7f15dc0707c0 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc06de00 con 0x7f15dc06d810 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15dc06bf20 con 0x7f15dc06f8b0 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0707c0 0x7f15dc1f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0707c0 0x7f15dc1f9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58410/0 (socket says 172.21.15.67:58410) 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 -- 172.21.15.67:0/1172408821 learned_addr learned my addr 172.21.15.67:0/1172408821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15d9e00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc06d810 0x7f15dc153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15d3e00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06f8b0 0x7f15dc15c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 -- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06f8b0 msgr2=0x7f15dc15c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06f8b0 0x7f15dc15c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 -- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc06d810 msgr2=0x7f15dc153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc06d810 0x7f15dc153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 -- 172.21.15.67:0/1172408821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0685d0 con 0x7f15dc0707c0 2024-04-17T10:25:27.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15da800640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0707c0 0x7f15dc1f9e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009d90 tx=0x7f15d40196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15d2000640 1 -- 172.21.15.67:0/1172408821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d40b2050 con 0x7f15dc0707c0 2024-04-17T10:25:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15d2000640 1 -- 172.21.15.67:0/1172408821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d40be070 con 0x7f15dc0707c0 2024-04-17T10:25:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15d2000640 1 -- 172.21.15.67:0/1172408821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d4009450 con 0x7f15dc0707c0 2024-04-17T10:25:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.155+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15dc0b4b30 con 0x7f15dc0707c0 2024-04-17T10:25:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.159+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc0b50f0 con 0x7f15dc0707c0 2024-04-17T10:25:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.159+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15dc0b48a0 con 0x7f15dc0707c0 2024-04-17T10:25:27.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.159+0000 7f15d2000640 1 -- 172.21.15.67:0/1172408821 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f15d40ada90 con 0x7f15dc0707c0 2024-04-17T10:25:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.159+0000 7f15d2000640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b8066f60 0x7f15b8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.159+0000 7f15d2000640 1 -- 172.21.15.67:0/1172408821 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f15d41203f0 con 0x7f15dc0707c0 2024-04-17T10:25:27.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.159+0000 7f15d9e00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b8066f60 0x7f15b8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.159+0000 7f15d9e00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b8066f60 0x7f15b8069420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009b60 tx=0x7f15cc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.163+0000 7f15d2000640 1 -- 172.21.15.67:0/1172408821 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f15d4016560 con 0x7f15dc0707c0 2024-04-17T10:25:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f15dc0b7780 con 0x7f15dc0707c0 2024-04-17T10:25:27.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15d2000640 1 -- 172.21.15.67:0/1172408821 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f15d40f0b80 con 0x7f15dc0707c0 2024-04-17T10:25:27.321 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b8066f60 msgr2=0x7f15b8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b8066f60 0x7f15b8069420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f15cc009b60 tx=0x7f15cc009190 comp rx=0 tx=0).stop 2024-04-17T10:25:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0707c0 msgr2=0x7f15dc1f9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0707c0 0x7f15dc1f9e10 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f15d4009d90 tx=0x7f15d40196e0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 shutdown_connections 2024-04-17T10:25:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f15b8066f60 0x7f15b8069420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15dc0707c0 0x7f15dc1f9e10 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15dc06f8b0 0x7f15dc15c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 --2- 172.21.15.67:0/1172408821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15dc06d810 0x7f15dc153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 >> 172.21.15.67:0/1172408821 conn(0x7f15dc0d1800 msgr2=0x7f15dc0b6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 shutdown_connections 2024-04-17T10:25:27.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.327+0000 7f15e0a00640 1 -- 172.21.15.67:0/1172408821 wait complete. 2024-04-17T10:25:27.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:27.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571f800640 1 Processor -- start 2024-04-17T10:25:27.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571f800640 1 -- start start 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571f800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 0x7f57180c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571f800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f571806ac90 0x7f571806b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571f800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571815df90 con 0x7f57180d0690 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571f800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57180ce8e0 con 0x7f571806ac90 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571f800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f571806b610 con 0x7f571815dbb0 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571d800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57180d0690 0x7f57180d0b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54450/0 (socket says 172.21.15.67:54450) 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571d800640 1 -- 172.21.15.67:0/3297852563 learned_addr learned my addr 172.21.15.67:0/3297852563 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717e00640 1 --2- 172.21.15.67:0/3297852563 >> v2:172.21.15.86:3300/0 conn(0x7f571806ac90 0x7f571806b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f571ce00640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 0x7f57180c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814894185 0 0) 0x7f571815df90 con 0x7f57180d0690 2024-04-17T10:25:27.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5700003530 con 0x7f57180d0690 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2147130674 0 0) 0x7f5700003530 con 0x7f57180d0690 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 msgr2=0x7f57180c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 0x7f57180c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 >> v2:172.21.15.86:3300/0 conn(0x7f571806ac90 msgr2=0x7f571806b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 --2- 172.21.15.67:0/3297852563 >> v2:172.21.15.86:3300/0 conn(0x7f571806ac90 0x7f571806b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57180ca3b0 con 0x7f57180d0690 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f5710001a00 con 0x7f57180d0690 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 >> v1:172.21.15.67:6789/0 conn(0x7f57180d0690 legacy=0x7f57180d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 0x7f57000039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5700003f40 0x7f5700004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5700004d50 0x7f57000051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180ce8e0 con 0x7f5700004d50 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f571806b610 con 0x7f5700003f40 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.491+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f570000ba00 con 0x7f571815dbb0 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571d800640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5700004d50 0x7f57000051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571ce00640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 0x7f57000039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f5717e00640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5700003f40 0x7f5700004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571d800640 1 -- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 msgr2=0x7f57000039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571d800640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f571815dbb0 0x7f57000039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571d800640 1 -- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5700003f40 msgr2=0x7f5700004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571d800640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5700003f40 0x7f5700004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571d800640 1 -- 172.21.15.67:0/3297852563 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5700003530 con 0x7f5700004d50 2024-04-17T10:25:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f571d800640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5700004d50 0x7f57000051d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f571000a880 tx=0x7f571001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd13d387ccfe99b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571000a4a0 con 0x7f5700004d50 2024-04-17T10:25:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57100ad630 con 0x7f5700004d50 2024-04-17T10:25:27.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.495+0000 7f5717400640 1 -- 172.21.15.67:0/3297852563 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57100adaa0 con 0x7f5700004d50 2024-04-17T10:25:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- 172.21.15.67:0/3297852563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5700004d50 msgr2=0x7f57000051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 --2- 172.21.15.67:0/3297852563 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5700004d50 0x7f57000051d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f571000a880 tx=0x7f571001d3a0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571d800640 1 -- 172.21.15.67:0/3297852563 reap_dead start 2024-04-17T10:25:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- 172.21.15.67:0/3297852563 shutdown_connections 2024-04-17T10:25:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- 172.21.15.67:0/3297852563 >> 172.21.15.67:0/3297852563 conn(0x7f5718099a50 msgr2=0x7f5718099e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- 172.21.15.67:0/3297852563 shutdown_connections 2024-04-17T10:25:27.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- 172.21.15.67:0/3297852563 wait complete. 2024-04-17T10:25:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 Processor -- start 2024-04-17T10:25:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- start start 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5700004d50 0x7f57181537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f571815df90 0x7f571815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f571815f760 0x7f571814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180ce470 con 0x7f571815df90 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57180ca970 con 0x7f571815f760 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571f800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57180ce5f0 con 0x7f5700004d50 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f571815df90 0x7f571815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f571815df90 0x7f571815b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58426/0 (socket says 172.21.15.67:58426) 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 -- 172.21.15.67:0/2947396694 learned_addr learned my addr 172.21.15.67:0/2947396694 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571ce00640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5700004d50 0x7f57181537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571d800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f571815f760 0x7f571814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 -- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5700004d50 msgr2=0x7f57181537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5700004d50 0x7f57181537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 -- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f571815f760 msgr2=0x7f571814fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f571815f760 0x7f571814fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f5717e00640 1 -- 172.21.15.67:0/2947396694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571815e3f0 con 0x7f571815df90 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.499+0000 7f571d800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f571815f760 0x7f571814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f5717e00640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f571815df90 0x7f571815b890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5704011960 tx=0x7f5704011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f5714c00640 1 -- 172.21.15.67:0/2947396694 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f570400fa90 con 0x7f571815df90 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f5714c00640 1 -- 172.21.15.67:0/2947396694 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f570400fc30 con 0x7f571815df90 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571815e6c0 con 0x7f571815df90 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f5714c00640 1 -- 172.21.15.67:0/2947396694 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5704014440 con 0x7f571815df90 2024-04-17T10:25:27.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f571815ef60 con 0x7f571815df90 2024-04-17T10:25:27.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f571804ef30 con 0x7f571815df90 2024-04-17T10:25:27.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f5714c00640 1 -- 172.21.15.67:0/2947396694 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5704012040 con 0x7f571815df90 2024-04-17T10:25:27.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f5714c00640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56f8066f60 0x7f56f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f5714c00640 1 -- 172.21.15.67:0/2947396694 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f5704085ad0 con 0x7f571815df90 2024-04-17T10:25:27.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.503+0000 7f571ce00640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56f8066f60 0x7f56f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.507+0000 7f571ce00640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56f8066f60 0x7f56f8069420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f570c009b20 tx=0x7f570c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.507+0000 7f5714c00640 1 -- 172.21.15.67:0/2947396694 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5704019030 con 0x7f571815df90 2024-04-17T10:25:27.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.671+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f571815ae30 con 0x7f571815df90 2024-04-17T10:25:27.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.671+0000 7f5714c00640 1 -- 172.21.15.67:0/2947396694 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5704056020 con 0x7f571815df90 2024-04-17T10:25:27.668 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56f8066f60 msgr2=0x7f56f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56f8066f60 0x7f56f8069420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f570c009b20 tx=0x7f570c008040 comp rx=0 tx=0).stop 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f571815df90 msgr2=0x7f571815b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f571815df90 0x7f571815b890 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5704011960 tx=0x7f5704011e20 comp rx=0 tx=0).stop 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 shutdown_connections 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56f8066f60 0x7f56f8069420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f571815f760 0x7f571814fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f571815df90 0x7f571815b890 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 --2- 172.21.15.67:0/2947396694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5700004d50 0x7f57181537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 >> 172.21.15.67:0/2947396694 conn(0x7f5718099a50 msgr2=0x7f57180cf6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:27.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 shutdown_connections 2024-04-17T10:25:27.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.675+0000 7f571f800640 1 -- 172.21.15.67:0/2947396694 wait complete. 2024-04-17T10:25:27.689 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:27.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa3c00640 1 Processor -- start 2024-04-17T10:25:27.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa3c00640 1 -- start start 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa3c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a9c15c580 0x7f9a9c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa3c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c15cb30 con 0x7f9a9c071af0 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa3c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a9c0cd610 con 0x7f9a9c159440 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa3c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9c0a8650 con 0x7f9a9c15c580 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa1c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a9c15c580 0x7f9a9c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa1c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a9c15c580 0x7f9a9c0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58428/0 (socket says 172.21.15.67:58428) 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa1c00640 1 -- 172.21.15.67:0/666763508 learned_addr learned my addr 172.21.15.67:0/666763508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:27.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9a9be00640 1 --1- 172.21.15.67:0/666763508 >> v1:172.21.15.67:6789/0 conn(0x7f9a9c071af0 0x7f9a9c15be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54454/0 (socket says 172.21.15.67:54454) 2024-04-17T10:25:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa1c00640 1 -- 172.21.15.67:0/666763508 >> v1:172.21.15.86:6789/0 conn(0x7f9a9c159440 legacy=0x7f9a9c0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa1c00640 1 -- 172.21.15.67:0/666763508 >> v1:172.21.15.67:6789/0 conn(0x7f9a9c071af0 legacy=0x7f9a9c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.839+0000 7f9aa1c00640 1 -- 172.21.15.67:0/666763508 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c0cd8f0 con 0x7f9a9c15c580 2024-04-17T10:25:27.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa1c00640 1 --2- 172.21.15.67:0/666763508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c0ce0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965d5155379a25c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9a9b400640 1 -- 172.21.15.67:0/666763508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940a5810 con 0x7f9a9c15c580 2024-04-17T10:25:27.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9a9b400640 1 -- 172.21.15.67:0/666763508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a940a59b0 con 0x7f9a9c15c580 2024-04-17T10:25:27.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9a9b400640 1 -- 172.21.15.67:0/666763508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940a5d80 con 0x7f9a9c15c580 2024-04-17T10:25:27.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa3c00640 1 -- 172.21.15.67:0/666763508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 msgr2=0x7f9a9c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/666763508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c0ce0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9a94009e50 tx=0x7f9a9401ca30 comp rx=0 tx=0).stop 2024-04-17T10:25:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa3c00640 1 -- 172.21.15.67:0/666763508 shutdown_connections 2024-04-17T10:25:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/666763508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c0ce0f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa3c00640 1 -- 172.21.15.67:0/666763508 >> 172.21.15.67:0/666763508 conn(0x7f9a9c0b1740 msgr2=0x7f9a9c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa3c00640 1 -- 172.21.15.67:0/666763508 shutdown_connections 2024-04-17T10:25:27.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.843+0000 7f9aa3c00640 1 -- 172.21.15.67:0/666763508 wait complete. 2024-04-17T10:25:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 Processor -- start 2024-04-17T10:25:27.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 -- start start 2024-04-17T10:25:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a9c071af0 0x7f9a9c0b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c159440 0x7f9a9c1525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c15dc10 con 0x7f9a9c15c580 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c15da90 con 0x7f9a9c159440 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a9c15dd90 con 0x7f9a9c071af0 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58444/0 (socket says 172.21.15.67:58444) 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 -- 172.21.15.67:0/1148099082 learned_addr learned my addr 172.21.15.67:0/1148099082 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9a9be00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c159440 0x7f9a9c1525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1200640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a9c071af0 0x7f9a9c0b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 -- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a9c071af0 msgr2=0x7f9a9c0b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a9c071af0 0x7f9a9c0b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 -- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c159440 msgr2=0x7f9a9c1525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c159440 0x7f9a9c1525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 -- 172.21.15.67:0/1148099082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c14e760 con 0x7f9a9c15c580 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9a9be00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c159440 0x7f9a9c1525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa1c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c152b10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a940af040 tx=0x7f9a940197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9a98c00640 1 -- 172.21.15.67:0/1148099082 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940b8880 con 0x7f9a9c15c580 2024-04-17T10:25:27.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9a98c00640 1 -- 172.21.15.67:0/1148099082 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a940be070 con 0x7f9a9c15c580 2024-04-17T10:25:27.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9a98c00640 1 -- 172.21.15.67:0/1148099082 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a940b8c80 con 0x7f9a9c15c580 2024-04-17T10:25:27.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a9c15bab0 con 0x7f9a9c15c580 2024-04-17T10:25:27.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9c150240 con 0x7f9a9c15c580 2024-04-17T10:25:27.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.847+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a9c04ef30 con 0x7f9a9c15c580 2024-04-17T10:25:27.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.851+0000 7f9a98c00640 1 -- 172.21.15.67:0/1148099082 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9a940c3460 con 0x7f9a9c15c580 2024-04-17T10:25:27.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.851+0000 7f9a98c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a7c066f60 0x7f9a7c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:27.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.851+0000 7f9a98c00640 1 -- 172.21.15.67:0/1148099082 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f9a941205f0 con 0x7f9a9c15c580 2024-04-17T10:25:27.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.851+0000 7f9aa1200640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a7c066f60 0x7f9a7c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:27.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.851+0000 7f9aa1200640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a7c066f60 0x7f9a7c069420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c156a60 tx=0x7f9a90009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:27.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:27.855+0000 7f9a98c00640 1 -- 172.21.15.67:0/1148099082 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9a94016620 con 0x7f9a9c15c580 2024-04-17T10:25:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9a9c1575b0 con 0x7f9a9c15c580 2024-04-17T10:25:28.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9a98c00640 1 -- 172.21.15.67:0/1148099082 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9a940f1220 con 0x7f9a9c15c580 2024-04-17T10:25:28.011 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a7c066f60 msgr2=0x7f9a7c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a7c066f60 0x7f9a7c069420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c156a60 tx=0x7f9a90009040 comp rx=0 tx=0).stop 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 msgr2=0x7f9a9c152b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c152b10 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9a940af040 tx=0x7f9a940197d0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 shutdown_connections 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a7c066f60 0x7f9a7c069420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a9c15c580 0x7f9a9c152b10 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a9c159440 0x7f9a9c1525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 --2- 172.21.15.67:0/1148099082 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a9c071af0 0x7f9a9c0b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.015+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 >> 172.21.15.67:0/1148099082 conn(0x7f9a9c0b1740 msgr2=0x7f9a9c0ad360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.019+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 shutdown_connections 2024-04-17T10:25:28.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.019+0000 7f9aa3c00640 1 -- 172.21.15.67:0/1148099082 wait complete. 2024-04-17T10:25:28.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:25:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd526800640 1 Processor -- start 2024-04-17T10:25:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd526800640 1 -- start start 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd526800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd520071310 0x7fd520071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd526800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd5200cc300 0x7fd5200cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd526800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd52006e2f0 con 0x7fd520145040 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd526800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5200a86c0 con 0x7fd520071310 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd526800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd520071cb0 con 0x7fd5200cc300 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd516a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd520071310 0x7fd520071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd516a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd520071310 0x7fd520071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58458/0 (socket says 172.21.15.67:58458) 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd516a00640 1 -- 172.21.15.67:0/1281477399 learned_addr learned my addr 172.21.15.67:0/1281477399 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.175+0000 7fd51fe00640 1 --2- 172.21.15.67:0/1281477399 >> v2:172.21.15.86:3300/0 conn(0x7fd5200cc300 0x7fd5200cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd516a00640 1 -- 172.21.15.67:0/1281477399 >> v2:172.21.15.86:3300/0 conn(0x7fd5200cc300 msgr2=0x7fd5200cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd516a00640 1 --2- 172.21.15.67:0/1281477399 >> v2:172.21.15.86:3300/0 conn(0x7fd5200cc300 0x7fd5200cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd516a00640 1 -- 172.21.15.67:0/1281477399 >> v1:172.21.15.67:6789/0 conn(0x7fd520145040 legacy=0x7fd52006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd516a00640 1 -- 172.21.15.67:0/1281477399 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52006d6e0 con 0x7fd520071310 2024-04-17T10:25:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd516a00640 1 --2- 172.21.15.67:0/1281477399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd520071310 0x7fd520071770 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5200af390 tx=0x7fd50801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a341be8bd58a8e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd51ea00640 1 -- 172.21.15.67:0/1281477399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5080b5070 con 0x7fd520071310 2024-04-17T10:25:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd51ea00640 1 -- 172.21.15.67:0/1281477399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5080a8a10 con 0x7fd520071310 2024-04-17T10:25:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd51ea00640 1 -- 172.21.15.67:0/1281477399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5080a8de0 con 0x7fd520071310 2024-04-17T10:25:28.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 -- 172.21.15.67:0/1281477399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd520071310 msgr2=0x7fd520071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 --2- 172.21.15.67:0/1281477399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd520071310 0x7fd520071770 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5200af390 tx=0x7fd50801ed10 comp rx=0 tx=0).stop 2024-04-17T10:25:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 -- 172.21.15.67:0/1281477399 shutdown_connections 2024-04-17T10:25:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 --2- 172.21.15.67:0/1281477399 >> v2:172.21.15.86:3300/0 conn(0x7fd5200cc300 0x7fd5200cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 --2- 172.21.15.67:0/1281477399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd520071310 0x7fd520071770 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 -- 172.21.15.67:0/1281477399 >> 172.21.15.67:0/1281477399 conn(0x7fd52006ac90 msgr2=0x7fd52006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 -- 172.21.15.67:0/1281477399 shutdown_connections 2024-04-17T10:25:28.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.179+0000 7fd526800640 1 -- 172.21.15.67:0/1281477399 wait complete. 2024-04-17T10:25:28.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 Processor -- start 2024-04-17T10:25:28.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 -- start start 2024-04-17T10:25:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd520071310 0x7fd52015ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5200cc300 0x7fd520144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd520145040 0x7fd520137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52006df80 con 0x7fd5200cc300 2024-04-17T10:25:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd52006dca0 con 0x7fd520071310 2024-04-17T10:25:28.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd52006d370 con 0x7fd520145040 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5200cc300 0x7fd520144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5200cc300 0x7fd520144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58462/0 (socket says 172.21.15.67:58462) 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 -- 172.21.15.67:0/4029660659 learned_addr learned my addr 172.21.15.67:0/4029660659 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd51f400640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd520071310 0x7fd52015ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd51fe00640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd520145040 0x7fd520137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 -- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd520145040 msgr2=0x7fd520137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd520145040 0x7fd520137b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 -- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd520071310 msgr2=0x7fd52015ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd520071310 0x7fd52015ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 -- 172.21.15.67:0/4029660659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd520136af0 con 0x7fd5200cc300 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd516a00640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5200cc300 0x7fd520144090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd50801b920 tx=0x7fd50801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd517e00640 1 -- 172.21.15.67:0/4029660659 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5080b5070 con 0x7fd5200cc300 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd517e00640 1 -- 172.21.15.67:0/4029660659 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5080b0b30 con 0x7fd5200cc300 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd517e00640 1 -- 172.21.15.67:0/4029660659 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5080bb410 con 0x7fd5200cc300 2024-04-17T10:25:28.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd520136dd0 con 0x7fd5200cc300 2024-04-17T10:25:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.183+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd520137390 con 0x7fd5200cc300 2024-04-17T10:25:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.187+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd520072d50 con 0x7fd5200cc300 2024-04-17T10:25:28.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.187+0000 7fd517e00640 1 -- 172.21.15.67:0/4029660659 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd5080c2060 con 0x7fd5200cc300 2024-04-17T10:25:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.187+0000 7fd517e00640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4fc066f90 0x7fd4fc069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.187+0000 7fd517e00640 1 -- 172.21.15.67:0/4029660659 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fd508122450 con 0x7fd5200cc300 2024-04-17T10:25:28.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.187+0000 7fd51f400640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4fc066f90 0x7fd4fc069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.187+0000 7fd51f400640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4fc066f90 0x7fd4fc069450 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd510001650 tx=0x7fd510009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.191+0000 7fd517e00640 1 -- 172.21.15.67:0/4029660659 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd508018870 con 0x7fd5200cc300 2024-04-17T10:25:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.387+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd52015efe0 con 0x7fd5200cc300 2024-04-17T10:25:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.387+0000 7fd517e00640 1 -- 172.21.15.67:0/4029660659 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7fd5080f3110 con 0x7fd5200cc300 2024-04-17T10:25:28.383 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:28.384 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:25:28.384 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:25:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4fc066f90 msgr2=0x7fd4fc069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4fc066f90 0x7fd4fc069450 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fd510001650 tx=0x7fd510009040 comp rx=0 tx=0).stop 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5200cc300 msgr2=0x7fd520144090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5200cc300 0x7fd520144090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd50801b920 tx=0x7fd50801bab0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 shutdown_connections 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4fc066f90 0x7fd4fc069450 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd520145040 0x7fd520137b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd5200cc300 0x7fd520144090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 --2- 172.21.15.67:0/4029660659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd520071310 0x7fd52015ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 >> 172.21.15.67:0/4029660659 conn(0x7fd52006ac90 msgr2=0x7fd5200b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 shutdown_connections 2024-04-17T10:25:28.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.391+0000 7fd526800640 1 -- 172.21.15.67:0/4029660659 wait complete. 2024-04-17T10:25:28.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:28.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a3200640 1 Processor -- start 2024-04-17T10:25:28.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a3200640 1 -- start start 2024-04-17T10:25:28.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa39c0ce1b0 0x7fa39c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a3200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa39c15fbd0 0x7fa39c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa39c0cead0 con 0x7fa39c15cc00 2024-04-17T10:25:28.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c15ffe0 con 0x7fa39c0ce1b0 2024-04-17T10:25:28.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a3200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c15be60 con 0x7fa39c15fbd0 2024-04-17T10:25:28.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa39be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa39c0ce1b0 0x7fa39c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa39be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa39c0ce1b0 0x7fa39c0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58478/0 (socket says 172.21.15.67:58478) 2024-04-17T10:25:28.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa39be00640 1 -- 172.21.15.67:0/787074052 learned_addr learned my addr 172.21.15.67:0/787074052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:28.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a1200640 1 --2- 172.21.15.67:0/787074052 >> v2:172.21.15.86:3300/0 conn(0x7fa39c15fbd0 0x7fa39c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa39be00640 1 -- 172.21.15.67:0/787074052 >> v2:172.21.15.86:3300/0 conn(0x7fa39c15fbd0 msgr2=0x7fa39c0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:28.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa39be00640 1 --2- 172.21.15.67:0/787074052 >> v2:172.21.15.86:3300/0 conn(0x7fa39c15fbd0 0x7fa39c0c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa39be00640 1 -- 172.21.15.67:0/787074052 >> v1:172.21.15.67:6789/0 conn(0x7fa39c15cc00 legacy=0x7fa39c15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa39be00640 1 -- 172.21.15.67:0/787074052 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c160230 con 0x7fa39c0ce1b0 2024-04-17T10:25:28.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.551+0000 7fa3a1200640 1 --2- 172.21.15.67:0/787074052 >> v2:172.21.15.86:3300/0 conn(0x7fa39c15fbd0 0x7fa39c0c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa39be00640 1 --2- 172.21.15.67:0/787074052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c0ce590 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0d2840 tx=0x7fa38c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963f82386e00f37e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa39aa00640 1 -- 172.21.15.67:0/787074052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0b5070 con 0x7fa39c0ce1b0 2024-04-17T10:25:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa39aa00640 1 -- 172.21.15.67:0/787074052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c0a8a10 con 0x7fa39c0ce1b0 2024-04-17T10:25:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa39aa00640 1 -- 172.21.15.67:0/787074052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c0a8de0 con 0x7fa39c0ce1b0 2024-04-17T10:25:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 -- 172.21.15.67:0/787074052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c0ce1b0 msgr2=0x7fa39c0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 --2- 172.21.15.67:0/787074052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c0ce590 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0d2840 tx=0x7fa38c01ed10 comp rx=0 tx=0).stop 2024-04-17T10:25:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 -- 172.21.15.67:0/787074052 shutdown_connections 2024-04-17T10:25:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 --2- 172.21.15.67:0/787074052 >> v2:172.21.15.86:3300/0 conn(0x7fa39c15fbd0 0x7fa39c0c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 --2- 172.21.15.67:0/787074052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c0ce590 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 -- 172.21.15.67:0/787074052 >> 172.21.15.67:0/787074052 conn(0x7fa39c0b33f0 msgr2=0x7fa39c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:28.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 -- 172.21.15.67:0/787074052 shutdown_connections 2024-04-17T10:25:28.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.555+0000 7fa3a3200640 1 -- 172.21.15.67:0/787074052 wait complete. 2024-04-17T10:25:28.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 Processor -- start 2024-04-17T10:25:28.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 -- start start 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c1432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c15cc00 0x7fa39c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c15fbd0 0x7fa39c0ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c15c6e0 con 0x7fa39c15cc00 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c15c400 con 0x7fa39c0ce1b0 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c159730 con 0x7fa39c15fbd0 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c15cc00 0x7fa39c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c15cc00 0x7fa39c06d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58486/0 (socket says 172.21.15.67:58486) 2024-04-17T10:25:28.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 -- 172.21.15.67:0/304083033 learned_addr learned my addr 172.21.15.67:0/304083033 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:28.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39be00640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c1432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a1200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c15fbd0 0x7fa39c0ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 -- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c15fbd0 msgr2=0x7fa39c0ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c15fbd0 0x7fa39c0ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 -- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0ce1b0 msgr2=0x7fa39c1432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c1432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 -- 172.21.15.67:0/304083033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c1e6670 con 0x7fa39c15cc00 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a1200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c15fbd0 0x7fa39c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39be00640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c1432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa39b400640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c15cc00 0x7fa39c06d680 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c420 tx=0x7fa38800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa393e00640 1 -- 172.21.15.67:0/304083033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3880052d0 con 0x7fa39c15cc00 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa393e00640 1 -- 172.21.15.67:0/304083033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa388005470 con 0x7fa39c15cc00 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa393e00640 1 -- 172.21.15.67:0/304083033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388007260 con 0x7fa39c15cc00 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c1e6950 con 0x7fa39c15cc00 2024-04-17T10:25:28.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.559+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c1e6f10 con 0x7fa39c15cc00 2024-04-17T10:25:28.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.563+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c070ab0 con 0x7fa39c15cc00 2024-04-17T10:25:28.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.563+0000 7fa393e00640 1 -- 172.21.15.67:0/304083033 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa388007400 con 0x7fa39c15cc00 2024-04-17T10:25:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.563+0000 7fa393e00640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa37c066f90 0x7fa37c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.563+0000 7fa393e00640 1 -- 172.21.15.67:0/304083033 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa3880848c0 con 0x7fa39c15cc00 2024-04-17T10:25:28.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.563+0000 7fa39be00640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa37c066f90 0x7fa37c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.563+0000 7fa39be00640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa37c066f90 0x7fa37c069450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0b2000 tx=0x7fa38c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.567+0000 7fa393e00640 1 -- 172.21.15.67:0/304083033 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa3880554f0 con 0x7fa39c15cc00 2024-04-17T10:25:28.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.731+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa39c0d0910 con 0x7fa39c15cc00 2024-04-17T10:25:28.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.731+0000 7fa393e00640 1 -- 172.21.15.67:0/304083033 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7fa388054df0 con 0x7fa39c15cc00 2024-04-17T10:25:28.728 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:28.728 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:28.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa37c066f90 msgr2=0x7fa37c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa37c066f90 0x7fa37c069450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa38c0b2000 tx=0x7fa38c008040 comp rx=0 tx=0).stop 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c15cc00 msgr2=0x7fa39c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c15cc00 0x7fa39c06d680 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c420 tx=0x7fa38800c8e0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 shutdown_connections 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa37c066f90 0x7fa37c069450 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa39c15fbd0 0x7fa39c0ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa39c15cc00 0x7fa39c06d680 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 --2- 172.21.15.67:0/304083033 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa39c0ce1b0 0x7fa39c1432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 >> 172.21.15.67:0/304083033 conn(0x7fa39c0b33f0 msgr2=0x7fa39c0c6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 shutdown_connections 2024-04-17T10:25:28.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.735+0000 7fa3a3200640 1 -- 172.21.15.67:0/304083033 wait complete. 2024-04-17T10:25:28.750 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:28.751 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:28.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a9800640 1 Processor -- start 2024-04-17T10:25:28.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a9800640 1 -- start start 2024-04-17T10:25:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a9800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a41653b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a9800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a40b4990 con 0x7f96a40cd030 2024-04-17T10:25:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a9800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a415ae90 con 0x7f96a40aef60 2024-04-17T10:25:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a9800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a40af340 con 0x7f96a406e980 2024-04-17T10:25:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f969be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96a40cd030 0x7f96a406e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54484/0 (socket says 172.21.15.67:54484) 2024-04-17T10:25:28.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f969be00640 1 -- 172.21.15.67:0/1162217246 learned_addr learned my addr 172.21.15.67:0/1162217246 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a3400640 1 --2- 172.21.15.67:0/1162217246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a41653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889603669 0 0) 0x7f968c001120 con 0x7f96a40cd030 2024-04-17T10:25:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.899+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9688003530 con 0x7f96a40cd030 2024-04-17T10:25:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348261001 0 0) 0x7f96a415ae90 con 0x7f96a40aef60 2024-04-17T10:25:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f968c001120 con 0x7f96a40aef60 2024-04-17T10:25:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a3400640 1 -- 172.21.15.67:0/1162217246 >> v1:172.21.15.86:6789/0 conn(0x7f96a40aef60 legacy=0x7f96a40cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a3400640 1 -- 172.21.15.67:0/1162217246 >> v1:172.21.15.67:6789/0 conn(0x7f96a40cd030 legacy=0x7f96a406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298193720 0 0) 0x7f9688003530 con 0x7f96a40cd030 2024-04-17T10:25:28.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a3400640 1 -- 172.21.15.67:0/1162217246 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a406ee80 con 0x7f96a406e980 2024-04-17T10:25:28.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a3400640 1 --2- 172.21.15.67:0/1162217246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a41653b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f969c009dc0 tx=0x7f969c01c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65ab665418ea40d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c0ac070 con 0x7f96a406e980 2024-04-17T10:25:28.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c0a6470 con 0x7f96a406e980 2024-04-17T10:25:28.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a2000640 1 -- 172.21.15.67:0/1162217246 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c001870 con 0x7f96a406e980 2024-04-17T10:25:28.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a9800640 1 -- 172.21.15.67:0/1162217246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 msgr2=0x7f96a41653b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.903+0000 7f96a9800640 1 --2- 172.21.15.67:0/1162217246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a41653b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f969c009dc0 tx=0x7f969c01c820 comp rx=0 tx=0).stop 2024-04-17T10:25:28.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- 172.21.15.67:0/1162217246 shutdown_connections 2024-04-17T10:25:28.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 --2- 172.21.15.67:0/1162217246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a41653b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- 172.21.15.67:0/1162217246 >> 172.21.15.67:0/1162217246 conn(0x7f96a40b1740 msgr2=0x7f96a40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:28.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- 172.21.15.67:0/1162217246 shutdown_connections 2024-04-17T10:25:28.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- 172.21.15.67:0/1162217246 wait complete. 2024-04-17T10:25:28.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 Processor -- start 2024-04-17T10:25:28.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- start start 2024-04-17T10:25:28.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a414c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96a40aef60 0x7f96a41459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a40cd030 0x7f96a4145f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a406d2e0 con 0x7f96a40cd030 2024-04-17T10:25:28.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a406d160 con 0x7f96a40aef60 2024-04-17T10:25:28.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a9800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96a406d460 con 0x7f96a406e980 2024-04-17T10:25:28.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a40cd030 0x7f96a4145f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a40cd030 0x7f96a4145f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58498/0 (socket says 172.21.15.67:58498) 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 -- 172.21.15.67:0/719872899 learned_addr learned my addr 172.21.15.67:0/719872899 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a2a00640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a414c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f969be00640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96a40aef60 0x7f96a41459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 -- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 msgr2=0x7f96a414c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a414c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 -- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96a40aef60 msgr2=0x7f96a41459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96a40aef60 0x7f96a41459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.907+0000 7f96a3400640 1 -- 172.21.15.67:0/719872899 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a415c6e0 con 0x7f96a40cd030 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f96a3400640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a40cd030 0x7f96a4145f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f969c01cd00 tx=0x7f969c0195c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f969aa00640 1 -- 172.21.15.67:0/719872899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c0c0070 con 0x7f96a40cd030 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f969aa00640 1 -- 172.21.15.67:0/719872899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f969c0bb470 con 0x7f96a40cd030 2024-04-17T10:25:28.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f969aa00640 1 -- 172.21.15.67:0/719872899 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969c0ac050 con 0x7f96a40cd030 2024-04-17T10:25:28.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a415c9c0 con 0x7f96a40cd030 2024-04-17T10:25:28.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a415cf80 con 0x7f96a40cd030 2024-04-17T10:25:28.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a404ef30 con 0x7f96a40cd030 2024-04-17T10:25:28.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f969aa00640 1 -- 172.21.15.67:0/719872899 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f969c0c3070 con 0x7f96a40cd030 2024-04-17T10:25:28.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f969aa00640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9680066f60 0x7f9680069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:28.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f969aa00640 1 -- 172.21.15.67:0/719872899 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f969c11f970 con 0x7f96a40cd030 2024-04-17T10:25:28.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.911+0000 7f96a2a00640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9680066f60 0x7f9680069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:28.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.915+0000 7f96a2a00640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9680066f60 0x7f9680069420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f96940017c0 tx=0x7f9694009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:28.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:28.915+0000 7f969aa00640 1 -- 172.21.15.67:0/719872899 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f969c0164a0 con 0x7f96a40cd030 2024-04-17T10:25:29.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.079+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96a414bc70 con 0x7f96a40cd030 2024-04-17T10:25:29.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f969aa00640 1 -- 172.21.15.67:0/719872899 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f969c0effe0 con 0x7f96a40cd030 2024-04-17T10:25:29.077 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9680066f60 msgr2=0x7f9680069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9680066f60 0x7f9680069420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f96940017c0 tx=0x7f9694009040 comp rx=0 tx=0).stop 2024-04-17T10:25:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a40cd030 msgr2=0x7f96a4145f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a40cd030 0x7f96a4145f10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f969c01cd00 tx=0x7f969c0195c0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 shutdown_connections 2024-04-17T10:25:29.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9680066f60 0x7f9680069420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96a40cd030 0x7f96a4145f10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96a40aef60 0x7f96a41459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 --2- 172.21.15.67:0/719872899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96a406e980 0x7f96a414c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 >> 172.21.15.67:0/719872899 conn(0x7f96a40b1740 msgr2=0x7f96a40af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 shutdown_connections 2024-04-17T10:25:29.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.083+0000 7f96a9800640 1 -- 172.21.15.67:0/719872899 wait complete. 2024-04-17T10:25:29.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:29.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b8000640 1 Processor -- start 2024-04-17T10:25:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b8000640 1 -- start start 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b8000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73b006cba0 0x7f73b00d0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b8000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f73b0099b30 0x7f73b00d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b8000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73b00cfad0 con 0x7f73b00cd9b0 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b8000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b006c250 con 0x7f73b006cba0 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b8000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73b015eac0 con 0x7f73b0099b30 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73b006cba0 0x7f73b00d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f73b006cba0 0x7f73b00d0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58508/0 (socket says 172.21.15.67:58508) 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b4c00640 1 -- 172.21.15.67:0/3525023331 learned_addr learned my addr 172.21.15.67:0/3525023331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:29.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b6000640 1 --2- 172.21.15.67:0/3525023331 >> v2:172.21.15.86:3300/0 conn(0x7f73b0099b30 0x7f73b00d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b4c00640 1 -- 172.21.15.67:0/3525023331 >> v2:172.21.15.86:3300/0 conn(0x7f73b0099b30 msgr2=0x7f73b00d07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b4c00640 1 --2- 172.21.15.67:0/3525023331 >> v2:172.21.15.86:3300/0 conn(0x7f73b0099b30 0x7f73b00d07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b4c00640 1 -- 172.21.15.67:0/3525023331 >> v1:172.21.15.86:6789/0 conn(0x7f73b00cd9b0 legacy=0x7f73b006c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:29.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.247+0000 7f73b4c00640 1 -- 172.21.15.67:0/3525023331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b015eda0 con 0x7f73b006cba0 2024-04-17T10:25:29.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b4c00640 1 --2- 172.21.15.67:0/3525023331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b00d0290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f73b0067a60 tx=0x7f739c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab4c63699b28fab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73afe00640 1 -- 172.21.15.67:0/3525023331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c0b0070 con 0x7f73b006cba0 2024-04-17T10:25:29.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73afe00640 1 -- 172.21.15.67:0/3525023331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b00d1520 con 0x7f73b006cba0 2024-04-17T10:25:29.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73afe00640 1 -- 172.21.15.67:0/3525023331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c0ab820 con 0x7f73b006cba0 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 -- 172.21.15.67:0/3525023331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 msgr2=0x7f73b00d0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 --2- 172.21.15.67:0/3525023331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b00d0290 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f73b0067a60 tx=0x7f739c01c4e0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 -- 172.21.15.67:0/3525023331 shutdown_connections 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 --2- 172.21.15.67:0/3525023331 >> v2:172.21.15.86:3300/0 conn(0x7f73b0099b30 0x7f73b00d07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 --2- 172.21.15.67:0/3525023331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b00d0290 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 -- 172.21.15.67:0/3525023331 >> 172.21.15.67:0/3525023331 conn(0x7f73b00d1890 msgr2=0x7f73b00d1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 -- 172.21.15.67:0/3525023331 shutdown_connections 2024-04-17T10:25:29.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.251+0000 7f73b8000640 1 -- 172.21.15.67:0/3525023331 wait complete. 2024-04-17T10:25:29.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 Processor -- start 2024-04-17T10:25:29.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 -- start start 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b0159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b0099b30 0x7f73b00cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b00cd9b0 0x7f73b00c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b015d8d0 con 0x7f73b006cba0 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73b015d750 con 0x7f73b00cd9b0 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73b015da50 con 0x7f73b0099b30 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b0159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b0159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58520/0 (socket says 172.21.15.67:58520) 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 -- 172.21.15.67:0/449277292 learned_addr learned my addr 172.21.15.67:0/449277292 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b4c00640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b0099b30 0x7f73b00cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b6000640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b00cd9b0 0x7f73b00c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 -- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b0099b30 msgr2=0x7f73b00cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b0099b30 0x7f73b00cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 -- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b00cd9b0 msgr2=0x7f73b00c8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b00cd9b0 0x7f73b00c8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 -- 172.21.15.67:0/449277292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b00afb70 con 0x7f73b006cba0 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b4c00640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b0099b30 0x7f73b00cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b5600640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b0159950 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73a400b410 tx=0x7f73a400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73ad600640 1 -- 172.21.15.67:0/449277292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4015030 con 0x7f73b006cba0 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73ad600640 1 -- 172.21.15.67:0/449277292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73a4004020 con 0x7f73b006cba0 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73ad600640 1 -- 172.21.15.67:0/449277292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73a4004450 con 0x7f73b006cba0 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73b00afe50 con 0x7f73b006cba0 2024-04-17T10:25:29.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.255+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73b00b0410 con 0x7f73b006cba0 2024-04-17T10:25:29.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.259+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73b004ef30 con 0x7f73b006cba0 2024-04-17T10:25:29.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.259+0000 7f73ad600640 1 -- 172.21.15.67:0/449277292 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f73a4018070 con 0x7f73b006cba0 2024-04-17T10:25:29.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.259+0000 7f73ad600640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7390066f60 0x7f7390069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.259+0000 7f73ad600640 1 -- 172.21.15.67:0/449277292 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f73a4084f30 con 0x7f73b006cba0 2024-04-17T10:25:29.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.259+0000 7f73b4c00640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7390066f60 0x7f7390069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.259+0000 7f73b4c00640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7390066f60 0x7f7390069420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f739c0ad000 tx=0x7f739c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.263+0000 7f73ad600640 1 -- 172.21.15.67:0/449277292 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f73a4055bf0 con 0x7f73b006cba0 2024-04-17T10:25:29.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73b00d26a0 con 0x7f73b006cba0 2024-04-17T10:25:29.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73ad600640 1 -- 172.21.15.67:0/449277292 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73a40554f0 con 0x7f73b006cba0 2024-04-17T10:25:29.423 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7390066f60 msgr2=0x7f7390069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7390066f60 0x7f7390069420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f739c0ad000 tx=0x7f739c008040 comp rx=0 tx=0).stop 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 msgr2=0x7f73b0159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b0159950 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f73a400b410 tx=0x7f73a400b8d0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 shutdown_connections 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7390066f60 0x7f7390069420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f73b00cd9b0 0x7f73b00c8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f73b0099b30 0x7f73b00cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 --2- 172.21.15.67:0/449277292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f73b006cba0 0x7f73b0159950 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.427+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 >> 172.21.15.67:0/449277292 conn(0x7f73b00d1890 msgr2=0x7f73b0070dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.431+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 shutdown_connections 2024-04-17T10:25:29.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.431+0000 7f73b8000640 1 -- 172.21.15.67:0/449277292 wait complete. 2024-04-17T10:25:29.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:29.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3ff400640 1 Processor -- start 2024-04-17T10:25:29.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3ff400640 1 -- start start 2024-04-17T10:25:29.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3ff400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa3f8159440 0x7fa3f80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3ff400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f815cb30 con 0x7fa3f815c580 2024-04-17T10:25:29.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3ff400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f80cd610 con 0x7fa3f80713e0 2024-04-17T10:25:29.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3ff400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f80a8650 con 0x7fa3f8159440 2024-04-17T10:25:29.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3fd400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3f815c580 0x7fa3f815d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54492/0 (socket says 172.21.15.67:54492) 2024-04-17T10:25:29.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3fd400640 1 -- 172.21.15.67:0/3973150702 learned_addr learned my addr 172.21.15.67:0/3973150702 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:29.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.591+0000 7fa3fca00640 1 --2- 172.21.15.67:0/3973150702 >> v2:172.21.15.86:3300/0 conn(0x7fa3f8159440 0x7fa3f80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3f7400640 1 -- 172.21.15.67:0/3973150702 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155655156 0 0) 0x7fa3f815cb30 con 0x7fa3f815c580 2024-04-17T10:25:29.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3f7400640 1 -- 172.21.15.67:0/3973150702 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e0003530 con 0x7fa3f815c580 2024-04-17T10:25:29.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3fca00640 1 -- 172.21.15.67:0/3973150702 >> v1:172.21.15.86:6789/0 conn(0x7fa3f80713e0 legacy=0x7fa3f815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3fca00640 1 -- 172.21.15.67:0/3973150702 >> v1:172.21.15.67:6789/0 conn(0x7fa3f815c580 legacy=0x7fa3f815d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3fca00640 1 -- 172.21.15.67:0/3973150702 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f80cd8f0 con 0x7fa3f8159440 2024-04-17T10:25:29.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3fca00640 1 --2- 172.21.15.67:0/3973150702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80ce0f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0097c0 tx=0x7fa3f815df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85cd5bb6c3504ee server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3f7400640 1 -- 172.21.15.67:0/3973150702 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0b1070 con 0x7fa3f8159440 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3f7400640 1 -- 172.21.15.67:0/3973150702 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec0ac410 con 0x7fa3f8159440 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3f7400640 1 -- 172.21.15.67:0/3973150702 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0ac7e0 con 0x7fa3f8159440 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3ff400640 1 -- 172.21.15.67:0/3973150702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 msgr2=0x7fa3f80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.595+0000 7fa3ff400640 1 --2- 172.21.15.67:0/3973150702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80ce0f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0097c0 tx=0x7fa3f815df50 comp rx=0 tx=0).stop 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- 172.21.15.67:0/3973150702 shutdown_connections 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 --2- 172.21.15.67:0/3973150702 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80ce0f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- 172.21.15.67:0/3973150702 >> 172.21.15.67:0/3973150702 conn(0x7fa3f80b1740 msgr2=0x7fa3f80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- 172.21.15.67:0/3973150702 shutdown_connections 2024-04-17T10:25:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- 172.21.15.67:0/3973150702 wait complete. 2024-04-17T10:25:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 Processor -- start 2024-04-17T10:25:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- start start 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f80713e0 0x7fa3f81535f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3f815c580 0x7fa3f80b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8169640 con 0x7fa3f80713e0 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f81694c0 con 0x7fa3f8159440 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3ff400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f81697c0 con 0x7fa3f815c580 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f80713e0 0x7fa3f81535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f80713e0 0x7fa3f81535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58536/0 (socket says 172.21.15.67:58536) 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 -- 172.21.15.67:0/2104646965 learned_addr learned my addr 172.21.15.67:0/2104646965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3f7e00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fd400640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3f815c580 0x7fa3f80b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 -- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3f815c580 msgr2=0x7fa3f80b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3f815c580 0x7fa3f80b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 -- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 msgr2=0x7fa3f80b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 -- 172.21.15.67:0/2104646965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f80b39e0 con 0x7fa3f80713e0 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3f7e00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.599+0000 7fa3fca00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f80713e0 0x7fa3f81535f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0097c0 tx=0x7fa3ec019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3f4c00640 1 -- 172.21.15.67:0/2104646965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0b9020 con 0x7fa3f80713e0 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3f4c00640 1 -- 172.21.15.67:0/2104646965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec0acd70 con 0x7fa3f80713e0 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3f4c00640 1 -- 172.21.15.67:0/2104646965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec0b1040 con 0x7fa3f80713e0 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f80b3cc0 con 0x7fa3f80713e0 2024-04-17T10:25:29.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f80b4280 con 0x7fa3f80713e0 2024-04-17T10:25:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f804ef30 con 0x7fa3f80713e0 2024-04-17T10:25:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3f4c00640 1 -- 172.21.15.67:0/2104646965 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa3ec0bc020 con 0x7fa3f80713e0 2024-04-17T10:25:29.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3f4c00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3d8066f60 0x7fa3d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3f4c00640 1 -- 172.21.15.67:0/2104646965 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa3ec11ebd0 con 0x7fa3f80713e0 2024-04-17T10:25:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.603+0000 7fa3f7e00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3d8066f60 0x7fa3d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.607+0000 7fa3f7e00640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3d8066f60 0x7fa3d8069420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8159f20 tx=0x7fa3e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.607+0000 7fa3f4c00640 1 -- 172.21.15.67:0/2104646965 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa3ec0c3030 con 0x7fa3f80713e0 2024-04-17T10:25:29.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3f814d620 con 0x7fa3f80713e0 2024-04-17T10:25:29.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3f4c00640 1 -- 172.21.15.67:0/2104646965 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3ec016090 con 0x7fa3f80713e0 2024-04-17T10:25:29.766 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:29.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3d8066f60 msgr2=0x7fa3d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3d8066f60 0x7fa3d8069420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8159f20 tx=0x7fa3e4009040 comp rx=0 tx=0).stop 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f80713e0 msgr2=0x7fa3f81535f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f80713e0 0x7fa3f81535f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec0097c0 tx=0x7fa3ec019240 comp rx=0 tx=0).stop 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 shutdown_connections 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3d8066f60 0x7fa3d8069420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3f815c580 0x7fa3f80b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3f8159440 0x7fa3f80b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 --2- 172.21.15.67:0/2104646965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f80713e0 0x7fa3f81535f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 >> 172.21.15.67:0/2104646965 conn(0x7fa3f80b1740 msgr2=0x7fa3f80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 shutdown_connections 2024-04-17T10:25:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.771+0000 7fa3ff400640 1 -- 172.21.15.67:0/2104646965 wait complete. 2024-04-17T10:25:29.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:29.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbccc00640 1 Processor -- start 2024-04-17T10:25:29.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbccc00640 1 -- start start 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbccc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbccc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbc80d06a0 0x7fbbc8161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbccc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc815e140 con 0x7fbbc809adf0 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbccc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc81626d0 con 0x7fbbc80d06a0 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbccc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8099d50 con 0x7fbbc815dd60 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbc6800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbc80d06a0 0x7fbbc8161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbc5e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbc6800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbbc80d06a0 0x7fbbc8161dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37650/0 (socket says 172.21.15.67:37650) 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbc5e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8161880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:49830/0 (socket says 172.21.15.67:49830) 2024-04-17T10:25:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.935+0000 7fbbc6800640 1 -- 172.21.15.67:0/1481725486 learned_addr learned my addr 172.21.15.67:0/1481725486 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc6800640 1 -- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 msgr2=0x7fbbc8161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc6800640 1 --2- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc6800640 1 -- 172.21.15.67:0/1481725486 >> v1:172.21.15.86:6789/0 conn(0x7fbbc809adf0 legacy=0x7fbbc80cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc6800640 1 -- 172.21.15.67:0/1481725486 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc809a030 con 0x7fbbc80d06a0 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc5e00640 1 --2- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc5400640 1 -- 172.21.15.67:0/1481725486 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204327398 0 0) 0x7fbbc815e140 con 0x7fbbc809adf0 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc6800640 1 --2- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc8161dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009e50 tx=0x7fbbc001ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d37664d7b5d717d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc5400640 1 -- 172.21.15.67:0/1481725486 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc00a58f0 con 0x7fbbc80d06a0 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc5400640 1 -- 172.21.15.67:0/1481725486 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc00a5a90 con 0x7fbbc80d06a0 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbc5400640 1 -- 172.21.15.67:0/1481725486 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc00a5e60 con 0x7fbbc80d06a0 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbccc00640 1 -- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 msgr2=0x7fbbc8161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.939+0000 7fbbccc00640 1 --2- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc8161dc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0009e50 tx=0x7fbbc001ca00 comp rx=0 tx=0).stop 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- 172.21.15.67:0/1481725486 shutdown_connections 2024-04-17T10:25:29.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 --2- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc8161dc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 --2- 172.21.15.67:0/1481725486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- 172.21.15.67:0/1481725486 >> 172.21.15.67:0/1481725486 conn(0x7fbbc80b34a0 msgr2=0x7fbbc80b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- 172.21.15.67:0/1481725486 shutdown_connections 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- 172.21.15.67:0/1481725486 wait complete. 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 Processor -- start 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- start start 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc809adf0 0x7fbbc8151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc81518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc806c730 con 0x7fbbc809adf0 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc806c5b0 con 0x7fbbc80d06a0 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc806c8b0 con 0x7fbbc815dd60 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc809adf0 0x7fbbc8151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc809adf0 0x7fbbc8151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58540/0 (socket says 172.21.15.67:58540) 2024-04-17T10:25:29.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 -- 172.21.15.67:0/2071000895 learned_addr learned my addr 172.21.15.67:0/2071000895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbbd400640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc81518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc6800640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 -- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 msgr2=0x7fbbc8151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8151e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 -- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 msgr2=0x7fbbc81518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc81518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 -- 172.21.15.67:0/2071000895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc80af3f0 con 0x7fbbc809adf0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc6800640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbbd400640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc81518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbc5e00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc809adf0 0x7fbbc8151050 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800e980 tx=0x7fbbb800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbbea00640 1 -- 172.21.15.67:0/2071000895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8017070 con 0x7fbbc809adf0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbbea00640 1 -- 172.21.15.67:0/2071000895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb800ca00 con 0x7fbbc809adf0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbbea00640 1 -- 172.21.15.67:0/2071000895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800cdd0 con 0x7fbbc809adf0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.943+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc80af6d0 con 0x7fbbc809adf0 2024-04-17T10:25:29.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.947+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc80afc90 con 0x7fbbc809adf0 2024-04-17T10:25:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.947+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc804ef30 con 0x7fbbc809adf0 2024-04-17T10:25:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.947+0000 7fbbbea00640 1 -- 172.21.15.67:0/2071000895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbbb801a070 con 0x7fbbc809adf0 2024-04-17T10:25:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.947+0000 7fbbbea00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbba4066f60 0x7fbba4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.947+0000 7fbbbea00640 1 -- 172.21.15.67:0/2071000895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fbbb80856e0 con 0x7fbbc809adf0 2024-04-17T10:25:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.947+0000 7fbbbd400640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbba4066f60 0x7fbba4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.947+0000 7fbbbd400640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbba4066f60 0x7fbba4069420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80cf4a0 tx=0x7fbbb0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:29.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:29.951+0000 7fbbbea00640 1 -- 172.21.15.67:0/2071000895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbbb8056310 con 0x7fbbc809adf0 2024-04-17T10:25:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbbc80b42d0 con 0x7fbbc809adf0 2024-04-17T10:25:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbbea00640 1 -- 172.21.15.67:0/2071000895 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbbb8055c10 con 0x7fbbc809adf0 2024-04-17T10:25:30.109 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbba4066f60 msgr2=0x7fbba4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbba4066f60 0x7fbba4069420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbbc80cf4a0 tx=0x7fbbb0009040 comp rx=0 tx=0).stop 2024-04-17T10:25:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc809adf0 msgr2=0x7fbbc8151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc809adf0 0x7fbbc8151050 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800e980 tx=0x7fbbb800ee40 comp rx=0 tx=0).stop 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 shutdown_connections 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbba4066f60 0x7fbba4069420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbbc815dd60 0x7fbbc8151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbbc80d06a0 0x7fbbc81518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 --2- 172.21.15.67:0/2071000895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbc809adf0 0x7fbbc8151050 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 >> 172.21.15.67:0/2071000895 conn(0x7fbbc80b34a0 msgr2=0x7fbbc8161f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 shutdown_connections 2024-04-17T10:25:30.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.115+0000 7fbbccc00640 1 -- 172.21.15.67:0/2071000895 wait complete. 2024-04-17T10:25:30.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:25:30.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7436c00640 1 Processor -- start 2024-04-17T10:25:30.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7436c00640 1 -- start start 2024-04-17T10:25:30.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7436c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74300cf510 0x7f74300cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7436c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743015eb50 con 0x7f74301620a0 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7436c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f743015ee30 con 0x7f74300ce020 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7436c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7430161650 con 0x7f74300cf510 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7427e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74301620a0 0x7f743015e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54498/0 (socket says 172.21.15.67:54498) 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7427e00640 1 -- 172.21.15.67:0/4207421936 learned_addr learned my addr 172.21.15.67:0/4207421936 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f7434c00640 1 --2- 172.21.15.67:0/4207421936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74300cf510 0x7f74300cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f742f400640 1 -- 172.21.15.67:0/4207421936 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973487495 0 0) 0x7f743015eb50 con 0x7f74301620a0 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.275+0000 7f742f400640 1 -- 172.21.15.67:0/4207421936 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7414003530 con 0x7f74301620a0 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f742f400640 1 -- 172.21.15.67:0/4207421936 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895416369 0 0) 0x7f743015ee30 con 0x7f74300ce020 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7434c00640 1 -- 172.21.15.67:0/4207421936 >> v1:172.21.15.86:6789/0 conn(0x7f74300ce020 legacy=0x7f74300ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7434c00640 1 -- 172.21.15.67:0/4207421936 >> v1:172.21.15.67:6789/0 conn(0x7f74301620a0 legacy=0x7f743015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7434c00640 1 -- 172.21.15.67:0/4207421936 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74301618a0 con 0x7f74300cf510 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7434c00640 1 --2- 172.21.15.67:0/4207421936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74300cf510 0x7f74300cf9a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7428009df0 tx=0x7f742801c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb26ee95a564d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:30.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f742f400640 1 -- 172.21.15.67:0/4207421936 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280b2070 con 0x7f74300cf510 2024-04-17T10:25:30.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f742f400640 1 -- 172.21.15.67:0/4207421936 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74280a5930 con 0x7f74300cf510 2024-04-17T10:25:30.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f742f400640 1 -- 172.21.15.67:0/4207421936 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74280a5ce0 con 0x7f74300cf510 2024-04-17T10:25:30.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7436c00640 1 -- 172.21.15.67:0/4207421936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74300cf510 msgr2=0x7f74300cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7436c00640 1 --2- 172.21.15.67:0/4207421936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74300cf510 0x7f74300cf9a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7428009df0 tx=0x7f742801c960 comp rx=0 tx=0).stop 2024-04-17T10:25:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7436c00640 1 -- 172.21.15.67:0/4207421936 shutdown_connections 2024-04-17T10:25:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7436c00640 1 --2- 172.21.15.67:0/4207421936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74300cf510 0x7f74300cf9a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7436c00640 1 -- 172.21.15.67:0/4207421936 >> 172.21.15.67:0/4207421936 conn(0x7f74300b33f0 msgr2=0x7f74300b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7436c00640 1 -- 172.21.15.67:0/4207421936 shutdown_connections 2024-04-17T10:25:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.279+0000 7f7436c00640 1 -- 172.21.15.67:0/4207421936 wait complete. 2024-04-17T10:25:30.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 Processor -- start 2024-04-17T10:25:30.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 -- start start 2024-04-17T10:25:30.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74300ce020 0x7f74300cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300cf510 0x7f7430154040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74301620a0 0x7f7430158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743015e8a0 con 0x7f74300cf510 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743015e720 con 0x7f74300ce020 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74300c84b0 con 0x7f74301620a0 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300cf510 0x7f7430154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300cf510 0x7f7430154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58552/0 (socket says 172.21.15.67:58552) 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 -- 172.21.15.67:0/2636359683 learned_addr learned my addr 172.21.15.67:0/2636359683 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f742fe00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74300ce020 0x7f74300cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7434c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74301620a0 0x7f7430158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 -- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74301620a0 msgr2=0x7f7430158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74301620a0 0x7f7430158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 -- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74300ce020 msgr2=0x7f74300cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74300ce020 0x7f74300cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 -- 172.21.15.67:0/2636359683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7430154580 con 0x7f74300cf510 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7434c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74301620a0 0x7f7430158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f742fe00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74300ce020 0x7f74300cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7427e00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300cf510 0x7f7430154040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f741800c3f0 tx=0x7f741800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f742cc00640 1 -- 172.21.15.67:0/2636359683 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7418017070 con 0x7f74300cf510 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f742cc00640 1 -- 172.21.15.67:0/2636359683 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7418005cf0 con 0x7f74300cf510 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f742cc00640 1 -- 172.21.15.67:0/2636359683 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74180072f0 con 0x7f74300cf510 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74300cbb10 con 0x7f74300cf510 2024-04-17T10:25:30.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.283+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743015d4f0 con 0x7f74300cf510 2024-04-17T10:25:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.287+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7430070ab0 con 0x7f74300cf510 2024-04-17T10:25:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.287+0000 7f742cc00640 1 -- 172.21.15.67:0/2636359683 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f741801a070 con 0x7f74300cf510 2024-04-17T10:25:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.287+0000 7f742cc00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f740c066f60 0x7f740c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.287+0000 7f742cc00640 1 -- 172.21.15.67:0/2636359683 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f7418084790 con 0x7f74300cf510 2024-04-17T10:25:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.287+0000 7f742fe00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f740c066f60 0x7f740c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.287+0000 7f742fe00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f740c066f60 0x7f740c069420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7420000f70 tx=0x7f7420009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.291+0000 7f742cc00640 1 -- 172.21.15.67:0/2636359683 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7418055450 con 0x7f74300cf510 2024-04-17T10:25:30.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.487+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7430074e30 con 0x7f74300cf510 2024-04-17T10:25:30.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.487+0000 7f742cc00640 1 -- 172.21.15.67:0/2636359683 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f7418054d50 con 0x7f74300cf510 2024-04-17T10:25:30.483 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:30.483 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:25:30.483 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:25:30.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f740c066f60 msgr2=0x7f740c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f740c066f60 0x7f740c069420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7420000f70 tx=0x7f7420009040 comp rx=0 tx=0).stop 2024-04-17T10:25:30.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300cf510 msgr2=0x7f7430154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300cf510 0x7f7430154040 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f741800c3f0 tx=0x7f741800c8b0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 shutdown_connections 2024-04-17T10:25:30.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f740c066f60 0x7f740c069420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74301620a0 0x7f7430158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74300cf510 0x7f7430154040 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 --2- 172.21.15.67:0/2636359683 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74300ce020 0x7f74300cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 >> 172.21.15.67:0/2636359683 conn(0x7f74300b33f0 msgr2=0x7f743015dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:30.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 shutdown_connections 2024-04-17T10:25:30.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.491+0000 7f7436c00640 1 -- 172.21.15.67:0/2636359683 wait complete. 2024-04-17T10:25:30.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc265600640 1 Processor -- start 2024-04-17T10:25:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc265600640 1 -- start start 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc265600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2600cc250 0x7fc2600cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc265600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 0x7fc2600cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc265600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26006b100 con 0x7fc2600ce2a0 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc265600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2600cd150 con 0x7fc2600cc250 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc265600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26006b3e0 con 0x7fc2600cb260 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc25ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2600cc250 0x7fc2600cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc25ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc2600cc250 0x7fc2600cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59654/0 (socket says 172.21.15.67:59654) 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc25ea00640 1 -- 172.21.15.67:0/173268268 learned_addr learned my addr 172.21.15.67:0/173268268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc25ea00640 1 -- 172.21.15.67:0/173268268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 msgr2=0x7fc2600cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:30.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc25ea00640 1 --2- 172.21.15.67:0/173268268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 0x7fc2600cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc25ea00640 1 -- 172.21.15.67:0/173268268 >> v1:172.21.15.86:6789/0 conn(0x7fc2600ce2a0 legacy=0x7fc2600cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:30.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.651+0000 7fc25ea00640 1 -- 172.21.15.67:0/173268268 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26006b6c0 con 0x7fc2600cc250 2024-04-17T10:25:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc25ea00640 1 --2- 172.21.15.67:0/173268268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600cc250 0x7fc2600cad20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc250009870 tx=0x7fc25001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77fc2448b5759b6c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc25e000640 1 -- 172.21.15.67:0/173268268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2500b0070 con 0x7fc2600cc250 2024-04-17T10:25:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc25e000640 1 -- 172.21.15.67:0/173268268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2500ab600 con 0x7fc2600cc250 2024-04-17T10:25:30.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc25e000640 1 -- 172.21.15.67:0/173268268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2500ab9d0 con 0x7fc2600cc250 2024-04-17T10:25:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 -- 172.21.15.67:0/173268268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600cc250 msgr2=0x7fc2600cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 --2- 172.21.15.67:0/173268268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600cc250 0x7fc2600cad20 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc250009870 tx=0x7fc25001c520 comp rx=0 tx=0).stop 2024-04-17T10:25:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 -- 172.21.15.67:0/173268268 shutdown_connections 2024-04-17T10:25:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 --2- 172.21.15.67:0/173268268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 0x7fc2600cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 --2- 172.21.15.67:0/173268268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600cc250 0x7fc2600cad20 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 -- 172.21.15.67:0/173268268 >> 172.21.15.67:0/173268268 conn(0x7fc2600b22a0 msgr2=0x7fc2600b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 -- 172.21.15.67:0/173268268 shutdown_connections 2024-04-17T10:25:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.655+0000 7fc265600640 1 -- 172.21.15.67:0/173268268 wait complete. 2024-04-17T10:25:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 Processor -- start 2024-04-17T10:25:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 -- start start 2024-04-17T10:25:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 0x7fc260160c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2600cc250 0x7fc260150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600ce2a0 0x7fc2601510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600cdef0 con 0x7fc2600ce2a0 2024-04-17T10:25:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600cdd70 con 0x7fc2600cc250 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26016b100 con 0x7fc2600cb260 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600ce2a0 0x7fc2601510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600ce2a0 0x7fc2601510a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59668/0 (socket says 172.21.15.67:59668) 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 -- 172.21.15.67:0/1457381686 learned_addr learned my addr 172.21.15.67:0/1457381686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25ea00640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 0x7fc260160c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc257e00640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2600cc250 0x7fc260150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 -- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 msgr2=0x7fc260160c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 0x7fc260160c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 -- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2600cc250 msgr2=0x7fc260150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2600cc250 0x7fc260150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 -- 172.21.15.67:0/1457381686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26015f900 con 0x7fc2600ce2a0 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc257e00640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2600cc250 0x7fc260150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:30.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc25f400640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600ce2a0 0x7fc2601510a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc25800b930 tx=0x7fc25800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc256a00640 1 -- 172.21.15.67:0/1457381686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258019070 con 0x7fc2600ce2a0 2024-04-17T10:25:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc256a00640 1 -- 172.21.15.67:0/1457381686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25800cb60 con 0x7fc2600ce2a0 2024-04-17T10:25:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc256a00640 1 -- 172.21.15.67:0/1457381686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258014440 con 0x7fc2600ce2a0 2024-04-17T10:25:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26015fbe0 con 0x7fc2600ce2a0 2024-04-17T10:25:30.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.659+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2601601a0 con 0x7fc2600ce2a0 2024-04-17T10:25:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.663+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26006cd30 con 0x7fc2600ce2a0 2024-04-17T10:25:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.663+0000 7fc256a00640 1 -- 172.21.15.67:0/1457381686 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc25801c070 con 0x7fc2600ce2a0 2024-04-17T10:25:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.663+0000 7fc256a00640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc23c066f60 0x7fc23c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.663+0000 7fc256a00640 1 -- 172.21.15.67:0/1457381686 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fc258084f70 con 0x7fc2600ce2a0 2024-04-17T10:25:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.663+0000 7fc25ea00640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc23c066f60 0x7fc23c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.663+0000 7fc25ea00640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc23c066f60 0x7fc23c069420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc2500ad000 tx=0x7fc2500a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:30.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.667+0000 7fc256a00640 1 -- 172.21.15.67:0/1457381686 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc258055550 con 0x7fc2600ce2a0 2024-04-17T10:25:30.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.835+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc260074e30 con 0x7fc2600ce2a0 2024-04-17T10:25:30.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc256a00640 1 -- 172.21.15.67:0/1457381686 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18466 (secure 0 0 0) 0x7fc2580595f0 con 0x7fc2600ce2a0 2024-04-17T10:25:30.833 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:30.833 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:01.631784+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:30.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc23c066f60 msgr2=0x7fc23c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc23c066f60 0x7fc23c069420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc2500ad000 tx=0x7fc2500a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:30.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600ce2a0 msgr2=0x7fc2601510a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600ce2a0 0x7fc2601510a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc25800b930 tx=0x7fc25800bdf0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 shutdown_connections 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc23c066f60 0x7fc23c069420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc2600ce2a0 0x7fc2601510a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc2600cc250 0x7fc260150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 --2- 172.21.15.67:0/1457381686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc2600cb260 0x7fc260160c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 >> 172.21.15.67:0/1457381686 conn(0x7fc2600b22a0 msgr2=0x7fc2600d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 shutdown_connections 2024-04-17T10:25:30.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.839+0000 7fc265600640 1 -- 172.21.15.67:0/1457381686 wait complete. 2024-04-17T10:25:30.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:25:30.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f61200640 1 Processor -- start 2024-04-17T10:25:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f61200640 1 -- start start 2024-04-17T10:25:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f61200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f5c0cf0c0 0x7f8f5c1441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f61200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c15ae40 con 0x7f8f5c06eac0 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f61200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5c071440 con 0x7f8f5c06e6e0 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f61200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5c071720 con 0x7f8f5c0cf0c0 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f5a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f5c0cf0c0 0x7f8f5c1441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f5a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f5c0cf0c0 0x7f8f5c1441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59674/0 (socket says 172.21.15.67:59674) 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f5a800640 1 -- 172.21.15.67:0/3954871786 learned_addr learned my addr 172.21.15.67:0/3954871786 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f5a800640 1 -- 172.21.15.67:0/3954871786 >> v1:172.21.15.86:6789/0 conn(0x7f8f5c06e6e0 legacy=0x7f8f5c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:30.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f5a800640 1 -- 172.21.15.67:0/3954871786 >> v1:172.21.15.67:6789/0 conn(0x7f8f5c06eac0 legacy=0x7f8f5c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:30.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:30.999+0000 7f8f5a800640 1 -- 172.21.15.67:0/3954871786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c071a00 con 0x7f8f5c0cf0c0 2024-04-17T10:25:30.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f5a800640 1 --2- 172.21.15.67:0/3954871786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c0cf0c0 0x7f8f5c1441f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009e50 tx=0x7f8f5401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df98ba9849a56f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:30.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f59400640 1 -- 172.21.15.67:0/3954871786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540a57f0 con 0x7f8f5c0cf0c0 2024-04-17T10:25:30.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f59400640 1 -- 172.21.15.67:0/3954871786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f540a5990 con 0x7f8f5c0cf0c0 2024-04-17T10:25:30.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f59400640 1 -- 172.21.15.67:0/3954871786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540a5d60 con 0x7f8f5c0cf0c0 2024-04-17T10:25:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f61200640 1 -- 172.21.15.67:0/3954871786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c0cf0c0 msgr2=0x7f8f5c1441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f61200640 1 --2- 172.21.15.67:0/3954871786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c0cf0c0 0x7f8f5c1441f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009e50 tx=0x7f8f5401c930 comp rx=0 tx=0).stop 2024-04-17T10:25:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f61200640 1 -- 172.21.15.67:0/3954871786 shutdown_connections 2024-04-17T10:25:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f61200640 1 --2- 172.21.15.67:0/3954871786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c0cf0c0 0x7f8f5c1441f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f61200640 1 -- 172.21.15.67:0/3954871786 >> 172.21.15.67:0/3954871786 conn(0x7f8f5c06ac90 msgr2=0x7f8f5c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f61200640 1 -- 172.21.15.67:0/3954871786 shutdown_connections 2024-04-17T10:25:31.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.003+0000 7f8f61200640 1 -- 172.21.15.67:0/3954871786 wait complete. 2024-04-17T10:25:31.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 Processor -- start 2024-04-17T10:25:31.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 -- start start 2024-04-17T10:25:31.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f5c06e6e0 0x7f8f5c0b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c06eac0 0x7f8f5c13c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f5c0cf0c0 0x7f8f5c13dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c1680b0 con 0x7f8f5c06eac0 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c167f30 con 0x7f8f5c06e6e0 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c168230 con 0x7f8f5c0cf0c0 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c06eac0 0x7f8f5c13c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c06eac0 0x7f8f5c13c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59688/0 (socket says 172.21.15.67:59688) 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 -- 172.21.15.67:0/3793522719 learned_addr learned my addr 172.21.15.67:0/3793522719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f59e00640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f5c06e6e0 0x7f8f5c0b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f5a800640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f5c0cf0c0 0x7f8f5c13dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 -- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f5c0cf0c0 msgr2=0x7f8f5c13dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f5c0cf0c0 0x7f8f5c13dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 -- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f5c06e6e0 msgr2=0x7f8f5c0b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f5c06e6e0 0x7f8f5c0b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 -- 172.21.15.67:0/3793522719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5c0b0f40 con 0x7f8f5c06eac0 2024-04-17T10:25:31.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f53e00640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c06eac0 0x7f8f5c13c0f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400c420 tx=0x7f8f4400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f52000640 1 -- 172.21.15.67:0/3793522719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f440052d0 con 0x7f8f5c06eac0 2024-04-17T10:25:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f52000640 1 -- 172.21.15.67:0/3793522719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44005470 con 0x7f8f5c06eac0 2024-04-17T10:25:31.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f52000640 1 -- 172.21.15.67:0/3793522719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44007260 con 0x7f8f5c06eac0 2024-04-17T10:25:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5c0b1220 con 0x7f8f5c06eac0 2024-04-17T10:25:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.007+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5c0b17e0 con 0x7f8f5c06eac0 2024-04-17T10:25:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.011+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5c072d50 con 0x7f8f5c06eac0 2024-04-17T10:25:31.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.011+0000 7f8f52000640 1 -- 172.21.15.67:0/3793522719 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8f44007400 con 0x7f8f5c06eac0 2024-04-17T10:25:31.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.011+0000 7f8f52000640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f38066f60 0x7f8f38069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.011+0000 7f8f52000640 1 -- 172.21.15.67:0/3793522719 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f8f44084890 con 0x7f8f5c06eac0 2024-04-17T10:25:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.011+0000 7f8f59e00640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f38066f60 0x7f8f38069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.011+0000 7f8f59e00640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f38066f60 0x7f8f38069420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c1655e0 tx=0x7f8f4c009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.015+0000 7f8f52000640 1 -- 172.21.15.67:0/3793522719 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8f44054e70 con 0x7f8f5c06eac0 2024-04-17T10:25:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f5c074e30 con 0x7f8f5c06eac0 2024-04-17T10:25:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f52000640 1 -- 172.21.15.67:0/3793522719 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f8f44058f10 con 0x7f8f5c06eac0 2024-04-17T10:25:31.209 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:31.210 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:25:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:25:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f38066f60 msgr2=0x7f8f38069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f38066f60 0x7f8f38069420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c1655e0 tx=0x7f8f4c009060 comp rx=0 tx=0).stop 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c06eac0 msgr2=0x7f8f5c13c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c06eac0 0x7f8f5c13c0f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400c420 tx=0x7f8f4400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 shutdown_connections 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f38066f60 0x7f8f38069420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f5c0cf0c0 0x7f8f5c13dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f5c06eac0 0x7f8f5c13c0f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 --2- 172.21.15.67:0/3793522719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f5c06e6e0 0x7f8f5c0b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 >> 172.21.15.67:0/3793522719 conn(0x7f8f5c06ac90 msgr2=0x7f8f5c0adb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 shutdown_connections 2024-04-17T10:25:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.215+0000 7f8f61200640 1 -- 172.21.15.67:0/3793522719 wait complete. 2024-04-17T10:25:31.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:24:54.978691+0000', 'modified': '2024-04-17T10:25:19.079374+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4275}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4275': {'gid': 4275, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/1394098337', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6834', 'nonce': 1394098337}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 1394098337}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:25:31.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:25:31.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T10:25:31.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2747800640 1 Processor -- start 2024-04-17T10:25:31.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2747800640 1 -- start start 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2747800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27401620a0 0x7f274015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2747800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274015eb50 con 0x7f274015e550 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2747800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274015ee30 con 0x7f27400ce020 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2747800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2740161650 con 0x7f27401620a0 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2745800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f274015e550 0x7f27400cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40926/0 (socket says 172.21.15.67:40926) 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f2745800640 1 -- 172.21.15.67:0/2434667188 learned_addr learned my addr 172.21.15.67:0/2434667188 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f273fe00640 1 --2- 172.21.15.67:0/2434667188 >> v2:172.21.15.86:3300/0 conn(0x7f27401620a0 0x7f274015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.783+0000 7f273f400640 1 -- 172.21.15.67:0/2434667188 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039332104 0 0) 0x7f274015eb50 con 0x7f274015e550 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273f400640 1 -- 172.21.15.67:0/2434667188 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2728003530 con 0x7f274015e550 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273fe00640 1 -- 172.21.15.67:0/2434667188 >> v1:172.21.15.86:6789/0 conn(0x7f27400ce020 legacy=0x7f27400ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273fe00640 1 -- 172.21.15.67:0/2434667188 >> v1:172.21.15.67:6789/0 conn(0x7f274015e550 legacy=0x7f27400cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273fe00640 1 -- 172.21.15.67:0/2434667188 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27401618a0 con 0x7f27401620a0 2024-04-17T10:25:31.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273fe00640 1 --2- 172.21.15.67:0/2434667188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27401620a0 0x7f274015e010 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f272c0097f0 tx=0x7f272c01c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=458c49dae4597698 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273f400640 1 -- 172.21.15.67:0/2434667188 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0b0070 con 0x7f27401620a0 2024-04-17T10:25:31.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273f400640 1 -- 172.21.15.67:0/2434667188 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f272c0ab600 con 0x7f27401620a0 2024-04-17T10:25:31.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f273f400640 1 -- 172.21.15.67:0/2434667188 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272c0ab9d0 con 0x7f27401620a0 2024-04-17T10:25:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f2747800640 1 -- 172.21.15.67:0/2434667188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27401620a0 msgr2=0x7f274015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f2747800640 1 --2- 172.21.15.67:0/2434667188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27401620a0 0x7f274015e010 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f272c0097f0 tx=0x7f272c01c430 comp rx=0 tx=0).stop 2024-04-17T10:25:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f2747800640 1 -- 172.21.15.67:0/2434667188 shutdown_connections 2024-04-17T10:25:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f2747800640 1 --2- 172.21.15.67:0/2434667188 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27401620a0 0x7f274015e010 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.787+0000 7f2747800640 1 -- 172.21.15.67:0/2434667188 >> 172.21.15.67:0/2434667188 conn(0x7f27400b33f0 msgr2=0x7f27400b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 -- 172.21.15.67:0/2434667188 shutdown_connections 2024-04-17T10:25:31.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 -- 172.21.15.67:0/2434667188 wait complete. 2024-04-17T10:25:31.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 Processor -- start 2024-04-17T10:25:31.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 -- start start 2024-04-17T10:25:31.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27400ce020 0x7f2740160580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274015e550 0x7f27400b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27401620a0 0x7f27400b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400c8630 con 0x7f27400ce020 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400c84b0 con 0x7f274015e550 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274016ba00 con 0x7f27401620a0 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27400ce020 0x7f2740160580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27400ce020 0x7f2740160580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59704/0 (socket says 172.21.15.67:59704) 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 -- 172.21.15.67:0/1058244739 learned_addr learned my addr 172.21.15.67:0/1058244739 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f273fe00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274015e550 0x7f27400b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 -- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27401620a0 msgr2=0x7f27400b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27401620a0 0x7f27400b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 -- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274015e550 msgr2=0x7f27400b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274015e550 0x7f27400b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 -- 172.21.15.67:0/1058244739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27400b5190 con 0x7f27400ce020 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f273fe00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274015e550 0x7f27400b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2744e00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27400ce020 0x7f2740160580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f273400c530 tx=0x7f273400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f273cc00640 1 -- 172.21.15.67:0/1058244739 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734016020 con 0x7f27400ce020 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f273cc00640 1 -- 172.21.15.67:0/1058244739 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2734005220 con 0x7f27400ce020 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274015da20 con 0x7f27400ce020 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.791+0000 7f273cc00640 1 -- 172.21.15.67:0/1058244739 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734007030 con 0x7f27400ce020 2024-04-17T10:25:31.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.795+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27400b6cb0 con 0x7f27400ce020 2024-04-17T10:25:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.795+0000 7f273cc00640 1 -- 172.21.15.67:0/1058244739 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f27340071d0 con 0x7f27400ce020 2024-04-17T10:25:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.795+0000 7f273cc00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2720066f90 0x7f2720069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.795+0000 7f273cc00640 1 -- 172.21.15.67:0/1058244739 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f2734084da0 con 0x7f27400ce020 2024-04-17T10:25:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f273fe00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2720066f90 0x7f2720069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 Processor -- start 2024-04-17T10:25:31.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- start start 2024-04-17T10:25:31.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f273fe00640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2720066f90 0x7f2720069450 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f27400b3380 tx=0x7f272c0ad040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f18d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f19240 0x561443ece9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443ecf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561443db65f0 con 0x561443ecef30 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561443d1dda0 con 0x561443f19240 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x561443ca6150 con 0x561443f18920 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443ecf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443ecf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59718/0 (socket says 172.21.15.67:59718) 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 -- 172.21.15.67:0/3691582336 learned_addr learned my addr 172.21.15.67:0/3691582336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2744e00640 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f18d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 -- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 msgr2=0x561443f18d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f18d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f273fe00640 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f19240 0x561443ece9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 -- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f19240 msgr2=0x561443ece9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f19240 0x561443ece9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 -- 172.21.15.67:0/3691582336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561443cbfca0 con 0x561443ecef30 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2744e00640 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f18d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f273fe00640 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f19240 0x561443ece9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2745800640 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443ecf310 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2738002120 tx=0x7f273800c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a6fe5bf66f54ed server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2731600640 1 -- 172.21.15.67:0/3691582336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738017070 con 0x561443ecef30 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2731600640 1 -- 172.21.15.67:0/3691582336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27380072d0 con 0x561443ecef30 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f2731600640 1 -- 172.21.15.67:0/3691582336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27380076c0 con 0x561443ecef30 2024-04-17T10:25:31.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- 172.21.15.67:0/3691582336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 msgr2=0x561443ecf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443ecf310 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2738002120 tx=0x7f273800c4b0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- 172.21.15.67:0/3691582336 shutdown_connections 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443ecf310 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f19240 0x561443ece9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 --2- 172.21.15.67:0/3691582336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f18d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- 172.21.15.67:0/3691582336 >> 172.21.15.67:0/3691582336 conn(0x561443ed52a0 msgr2=0x561443ecd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- 172.21.15.67:0/3691582336 shutdown_connections 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.799+0000 7f274ad82000 1 -- 172.21.15.67:0/3691582336 wait complete. 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 -- start start 2024-04-17T10:25:31.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 Processor -- start 2024-04-17T10:25:31.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443f1fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f20230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443ed2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561443cd8990 con 0x561443f19240 2024-04-17T10:25:31.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561443d19e40 con 0x561443ecef30 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x561443ca9900 con 0x561443f18920 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 10 client.0 _ll_register_callbacks cb 0x561443dd11c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443ed2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2744e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443f1fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f273fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f20230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443ed2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59734/0 (socket says 172.21.15.67:59734) 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2744e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443f1fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:44646/0 (socket says 172.21.15.67:44646) 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 -- 172.21.15.67:0/2964283618 learned_addr learned my addr 172.21.15.67:0/2964283618 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 -- 172.21.15.67:0/2964283618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 msgr2=0x561443f20230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f20230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 -- 172.21.15.67:0/2964283618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 msgr2=0x561443f1fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443f1fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 -- 172.21.15.67:0/2964283618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561443cd6630 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2745800640 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443ed2aa0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2738015040 tx=0x7f2738005b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cff51cfe8ad724c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2726000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2726000640 1 -- 172.21.15.67:0/2964283618 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738022020 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2726000640 1 -- 172.21.15.67:0/2964283618 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f273801e400 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27380072d0 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2726000640 1 -- 172.21.15.67:0/2964283618 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2738017040 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 10 client.4800 fetch_fsmap learned FSMap version 9 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x561443ca63a0 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2726000640 1 -- 172.21.15.67:0/2964283618 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27380072d0 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2726000640 1 -- 172.21.15.67:0/2964283618 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f273801d040 con 0x561443f19240 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 10 client.4800 fetch_fsmap finished waiting for FSMap version 9 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 10 client.4800 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:25:31.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 20 client.4800 populate_metadata read hostname 'smithi067' 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443ed4160 0x561443f29d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 4 client.4800 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 --> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561443cb9a50 con 0x561443ed4160 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.803+0000 7f2744e00640 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443ed4160 0x561443f29d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f2744e00640 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443ed4160 0x561443f29d50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a75a5a0b9c34e46 server_cookie=5cb31fca750d35b8 in_seq=0 out_seq=0 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f2726000640 10 client.4800 ms_handle_connect on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f2726000640 1 -- 172.21.15.67:0/2964283618 <== mds.0 v2:172.21.15.67:6834/1394098337 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561443cb9a50 con 0x561443ed4160 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f2726000640 10 client.4800 handle_command_reply: tid=0 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 client.4800 shutdown 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 msgr2=0x561443ed2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443ed2aa0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2738015040 tx=0x7f2738005b80 comp rx=0 tx=0).stop 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 shutdown_connections 2024-04-17T10:25:31.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443ed4160 0x561443f29d50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443ed2aa0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443f20230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 --2- 172.21.15.67:0/2964283618 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443f1fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f2726000640 0 client.4800 ms_handle_reset on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 >> 172.21.15.67:0/2964283618 conn(0x561443ed52a0 msgr2=0x561443ecd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 shutdown_connections 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 1 -- 172.21.15.67:0/2964283618 wait complete. 2024-04-17T10:25:31.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.807+0000 7f274ad82000 20 client.4800 trim_cache size 0 max 16384 2024-04-17T10:25:31.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 Processor -- start 2024-04-17T10:25:31.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- start start 2024-04-17T10:25:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443ed2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443ed2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443f58c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2740203150 con 0x561443f19240 2024-04-17T10:25:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27400b2320 con 0x561443ecef30 2024-04-17T10:25:31.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2740202bb0 con 0x561443f18920 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443f58c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443f58c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59744/0 (socket says 172.21.15.67:59744) 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 -- 172.21.15.67:0/1455723957 learned_addr learned my addr 172.21.15.67:0/1455723957 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2744e00640 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443ed2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f273fe00640 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443ed2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 -- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 msgr2=0x561443ed2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443ed2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 -- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 msgr2=0x561443ed2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443ed2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 -- 172.21.15.67:0/1455723957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561443f19070 con 0x561443f19240 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2745800640 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443f58c30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f273800cf90 tx=0x7f273801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1cf61646507ccc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2730c00640 1 -- 172.21.15.67:0/1455723957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738022070 con 0x561443f19240 2024-04-17T10:25:31.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2730c00640 1 -- 172.21.15.67:0/1455723957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2738007a90 con 0x561443f19240 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f2730c00640 1 -- 172.21.15.67:0/1455723957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2738004b70 con 0x561443f19240 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- 172.21.15.67:0/1455723957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 msgr2=0x561443f58c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443f58c30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f273800cf90 tx=0x7f273801e000 comp rx=0 tx=0).stop 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- 172.21.15.67:0/1455723957 shutdown_connections 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443f19240 0x561443f58c30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f18920 0x561443ed2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 --2- 172.21.15.67:0/1455723957 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443ecef30 0x561443ed2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- 172.21.15.67:0/1455723957 >> 172.21.15.67:0/1455723957 conn(0x561443ed52a0 msgr2=0x561443f053c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- 172.21.15.67:0/1455723957 shutdown_connections 2024-04-17T10:25:31.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.823+0000 7f274ad82000 1 -- 172.21.15.67:0/1455723957 wait complete. 2024-04-17T10:25:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- start start 2024-04-17T10:25:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 Processor -- start 2024-04-17T10:25:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443f259a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f18920 0x561443f5c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f19240 0x561443f5c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561443d223d0 con 0x561443ecef30 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561443dd9d50 con 0x561443f18920 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x561443cb8980 con 0x561443f19240 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 10 client.0 _ll_register_callbacks cb 0x7f2738001970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443f259a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443f259a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59758/0 (socket says 172.21.15.67:59758) 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 -- 172.21.15.67:0/4246733067 learned_addr learned my addr 172.21.15.67:0/4246733067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2745800640 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f19240 0x561443f5c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f273fe00640 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f18920 0x561443f5c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 -- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f19240 msgr2=0x561443f5c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f19240 0x561443f5c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 -- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f18920 msgr2=0x561443f5c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f18920 0x561443f5c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 -- 172.21.15.67:0/4246733067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561443d9f600 con 0x561443ecef30 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2744e00640 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443f259a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2734000fc0 tx=0x7f273405da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b20d29f4344ee7f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 1 -- 172.21.15.67:0/4246733067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2734010d70 con 0x561443ecef30 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 1 -- 172.21.15.67:0/4246733067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f27340519d0 con 0x561443ecef30 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561443d2b2e0 con 0x561443ecef30 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 1 -- 172.21.15.67:0/4246733067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27340590b0 con 0x561443ecef30 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 10 client.4806 fetch_fsmap learned FSMap version 9 2024-04-17T10:25:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0x561443cb29b0 con 0x561443ecef30 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 1 -- 172.21.15.67:0/4246733067 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561443d2b2e0 con 0x561443ecef30 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 1 -- 172.21.15.67:0/4246733067 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 9) v1 ==== 3302+0+0 (secure 0 0 0) 0x7f273405d5e0 con 0x561443ecef30 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 10 client.4806 fetch_fsmap finished waiting for FSMap version 9 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 10 client.4806 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 20 client.4806 populate_metadata read hostname 'smithi067' 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443f5d090 0x561443f5d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 4 client.4806 mds_command: new command op to 4275 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 --> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561443d25630 con 0x561443f5d090 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f273fe00640 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443f5d090 0x561443f5d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f273fe00640 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443f5d090 0x561443f5d470 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=843f3d4d16a87274 server_cookie=1e6e3776b23e4e44 in_seq=0 out_seq=0 2024-04-17T10:25:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 10 client.4806 ms_handle_connect on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 1 -- 172.21.15.67:0/4246733067 <== mds.0 v2:172.21.15.67:6834/1394098337 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x561443d25630 con 0x561443f5d090 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f2726000640 10 client.4806 handle_command_reply: tid=0 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 client.4806 shutdown 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 msgr2=0x561443f259a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.827+0000 7f274ad82000 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443f259a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2734000fc0 tx=0x7f273405da70 comp rx=0 tx=0).stop 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 shutdown_connections 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:6834/1394098337,v1:172.21.15.67:6835/1394098337] conn(0x561443f5d090 0x561443f5d470 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561443f19240 0x561443f5c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561443f18920 0x561443f5c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 --2- 172.21.15.67:0/4246733067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561443ecef30 0x561443f259a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2726000640 0 client.4806 ms_handle_reset on v2:172.21.15.67:6834/1394098337 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 >> 172.21.15.67:0/4246733067 conn(0x561443ed52a0 msgr2=0x561443ed1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 shutdown_connections 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 1 -- 172.21.15.67:0/4246733067 wait complete. 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f274ad82000 20 client.4806 trim_cache size 0 max 16384 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4275, 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:25:31.831 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 36.249464142000001, 2024-04-17T10:25:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 9, 2024-04-17T10:25:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 25, 2024-04-17T10:25:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 25, 2024-04-17T10:25:31.832 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 43.267606469999997 2024-04-17T10:25:31.832 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2720066f90 msgr2=0x7f2720069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2720066f90 0x7f2720069450 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f27400b3380 tx=0x7f272c0ad040 comp rx=0 tx=0).stop 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27400ce020 msgr2=0x7f2740160580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27400ce020 0x7f2740160580 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f273400c530 tx=0x7f273400c9f0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 shutdown_connections 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2720066f90 0x7f2720069450 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27401620a0 0x7f27400b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274015e550 0x7f27400b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 --2- 172.21.15.67:0/1058244739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27400ce020 0x7f2740160580 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 >> 172.21.15.67:0/1058244739 conn(0x7f27400b33f0 msgr2=0x7f27400cc020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 shutdown_connections 2024-04-17T10:25:31.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:31.831+0000 7f2747800640 1 -- 172.21.15.67:0/1058244739 wait complete. 2024-04-17T10:25:31.844 INFO:teuthology.run_tasks:Running task kclient... 2024-04-17T10:25:31.859 INFO:tasks.kclient:Mounting kernel clients... 2024-04-17T10:25:31.859 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-17T10:25:31.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:31.859 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:25:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-17T10:25:31.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:31.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-17T10:25:31.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:31.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-17T10:25:32.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.146 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:25:32.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-04-17T10:25:32.222 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-04-17T10:25:32.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:25:32.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.225 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-04-17T10:25:32.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link delete ceph-brx 2024-04-17T10:25:32.301 INFO:teuthology.orchestra.run.smithi086.stderr:Cannot find device "ceph-brx" 2024-04-17T10:25:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:25:32.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.304 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-04-17T10:25:32.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link delete ceph-brx 2024-04-17T10:25:32.428 INFO:teuthology.orchestra.run.smithi086.stderr:Cannot find device "ceph-brx" 2024-04-17T10:25:32.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:25:32.430 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-17T10:25:32.430 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T10:25:32.430 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:25:32.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:25:32.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.575+0000 7fa1a0000640 1 Processor -- start 2024-04-17T10:25:32.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.575+0000 7fa1a0000640 1 -- start start 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa1a0000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1980cef80 0x7fa198163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa1a0000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa1980b4970 0x7fa198166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa1a0000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1980719a0 con 0x7fa198071310 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa1a0000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa198071c80 con 0x7fa1980cef80 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa1a0000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa198162580 con 0x7fa1980b4970 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1980cef80 0x7fa198163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa1980cef80 0x7fa198163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59764/0 (socket says 172.21.15.67:59764) 2024-04-17T10:25:32.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19d600640 1 -- 172.21.15.67:0/3939569773 learned_addr learned my addr 172.21.15.67:0/3939569773 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19e000640 1 --2- 172.21.15.67:0/3939569773 >> v2:172.21.15.86:3300/0 conn(0x7fa1980b4970 0x7fa198166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19d600640 1 -- 172.21.15.67:0/3939569773 >> v2:172.21.15.86:3300/0 conn(0x7fa1980b4970 msgr2=0x7fa198166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19d600640 1 --2- 172.21.15.67:0/3939569773 >> v2:172.21.15.86:3300/0 conn(0x7fa1980b4970 0x7fa198166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19d600640 1 -- 172.21.15.67:0/3939569773 >> v1:172.21.15.67:6789/0 conn(0x7fa198071310 legacy=0x7fa1980b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19d600640 1 -- 172.21.15.67:0/3939569773 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa198162860 con 0x7fa1980cef80 2024-04-17T10:25:32.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.579+0000 7fa19e000640 1 --2- 172.21.15.67:0/3939569773 >> v2:172.21.15.86:3300/0 conn(0x7fa1980b4970 0x7fa198166430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:32.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa19d600640 1 --2- 172.21.15.67:0/3939569773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1980cef80 0x7fa198163060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa19806eb20 tx=0x7fa18c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07066a0c50573d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:32.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa197e00640 1 -- 172.21.15.67:0/3939569773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0b0070 con 0x7fa1980cef80 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa197e00640 1 -- 172.21.15.67:0/3939569773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c0ab600 con 0x7fa1980cef80 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa197e00640 1 -- 172.21.15.67:0/3939569773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0ab9d0 con 0x7fa1980cef80 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- 172.21.15.67:0/3939569773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1980cef80 msgr2=0x7fa198163060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 --2- 172.21.15.67:0/3939569773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1980cef80 0x7fa198163060 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa19806eb20 tx=0x7fa18c01c520 comp rx=0 tx=0).stop 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- 172.21.15.67:0/3939569773 shutdown_connections 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 --2- 172.21.15.67:0/3939569773 >> v2:172.21.15.86:3300/0 conn(0x7fa1980b4970 0x7fa198166430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 --2- 172.21.15.67:0/3939569773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa1980cef80 0x7fa198163060 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- 172.21.15.67:0/3939569773 >> 172.21.15.67:0/3939569773 conn(0x7fa19806ac90 msgr2=0x7fa19806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- 172.21.15.67:0/3939569773 shutdown_connections 2024-04-17T10:25:32.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- 172.21.15.67:0/3939569773 wait complete. 2024-04-17T10:25:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 Processor -- start 2024-04-17T10:25:32.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- start start 2024-04-17T10:25:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198071310 0x7fa198152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980b4970 0x7fa1980af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980cef80 0x7fa198157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1980b3350 con 0x7fa198071310 2024-04-17T10:25:32.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1980b31d0 con 0x7fa1980b4970 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.583+0000 7fa1a0000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1980b34d0 con 0x7fa1980cef80 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198071310 0x7fa198152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198071310 0x7fa198152400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59768/0 (socket says 172.21.15.67:59768) 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 -- 172.21.15.67:0/1272676840 learned_addr learned my addr 172.21.15.67:0/1272676840 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19cc00640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980b4970 0x7fa1980af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 -- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980cef80 msgr2=0x7fa198157a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19e000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980cef80 0x7fa198157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980cef80 0x7fa198157a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 -- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980b4970 msgr2=0x7fa1980af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980b4970 0x7fa1980af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 -- 172.21.15.67:0/1272676840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19815a3b0 con 0x7fa198071310 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19cc00640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980b4970 0x7fa1980af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19e000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980cef80 0x7fa198157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa19d600640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198071310 0x7fa198152400 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0ae040 tx=0x7fa18c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa195600640 1 -- 172.21.15.67:0/1272676840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0b8020 con 0x7fa198071310 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa195600640 1 -- 172.21.15.67:0/1272676840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa18c0b4630 con 0x7fa198071310 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1980b1020 con 0x7fa198071310 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa195600640 1 -- 172.21.15.67:0/1272676840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18c0b0040 con 0x7fa198071310 2024-04-17T10:25:32.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1980b15e0 con 0x7fa198071310 2024-04-17T10:25:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa198072d50 con 0x7fa198071310 2024-04-17T10:25:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.587+0000 7fa195600640 1 -- 172.21.15.67:0/1272676840 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa18c0bb020 con 0x7fa198071310 2024-04-17T10:25:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.591+0000 7fa195600640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa178066f90 0x7fa178069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.591+0000 7fa195600640 1 -- 172.21.15.67:0/1272676840 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7fa18c11ef10 con 0x7fa198071310 2024-04-17T10:25:32.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.591+0000 7fa19cc00640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa178066f90 0x7fa178069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:32.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.591+0000 7fa19cc00640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa178066f90 0x7fa178069450 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa198159700 tx=0x7fa184009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:32.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.595+0000 7fa195600640 1 -- 172.21.15.67:0/1272676840 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa18c0c2030 con 0x7fa198071310 2024-04-17T10:25:32.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa198074e30 con 0x7fa198071310 2024-04-17T10:25:32.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa195600640 1 -- 172.21.15.67:0/1272676840 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7fa19815a3b0 con 0x7fa198071310 2024-04-17T10:25:32.786 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa178066f90 msgr2=0x7fa178069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa178066f90 0x7fa178069450 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa198159700 tx=0x7fa184009040 comp rx=0 tx=0).stop 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198071310 msgr2=0x7fa198152400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198071310 0x7fa198152400 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa18c0ae040 tx=0x7fa18c0192c0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 shutdown_connections 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa178066f90 0x7fa178069450 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa1980cef80 0x7fa198157a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa1980b4970 0x7fa1980af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 --2- 172.21.15.67:0/1272676840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198071310 0x7fa198152400 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 >> 172.21.15.67:0/1272676840 conn(0x7fa19806ac90 msgr2=0x7fa19815a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 shutdown_connections 2024-04-17T10:25:32.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:32.791+0000 7fa1a0000640 1 -- 172.21.15.67:0/1272676840 wait complete. 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:25:32.807 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:25:32.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.807 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:32.816 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42414sec preferred_lft 42414sec 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:25:32.817 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:32.818 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-17T10:25:32.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.819 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:32.819 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:32.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-04-17T10:25:32.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-04-17T10:25:32.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-04-17T10:25:32.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-17T10:25:32.819 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:32.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.930 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:25:32.949 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:25:32.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:32.952 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:25:33.003 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:25:33.003 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:25:33.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:33.005 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:33.005 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:33.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:25:33.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:25:33.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:25:33.005 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:33.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:33.094 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:25:33.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:33.104 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-17T10:25:33.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:33.153 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:33.153 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:33.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:25:33.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-17T10:25:33.153 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:33.235 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-17T10:25:33.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:33.235 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:33.235 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:33.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-17T10:25:33.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-17T10:25:33.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-17T10:25:33.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-17T10:25:33.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-17T10:25:33.236 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:33.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:33.373 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:33.373 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:33.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:25:33.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-04-17T10:25:33.373 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:33.413 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:33.413 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:25:33.464 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:25:33.465 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:33.465 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:25:33.516 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:25:33.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:33.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:33.605 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:33.605 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:34.217 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:34.217 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:34.217 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:25:34.217 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:25:34.217 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:25:34.221 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:25:34.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:34.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:25:34.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:34.241 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:34.295 INFO:teuthology.orchestra.run.smithi067.stdout:client4815 2024-04-17T10:25:34.296 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:34.296 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:34.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:34.345 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:34.399 INFO:teuthology.orchestra.run.smithi067.stdout:client4815 2024-04-17T10:25:34.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:34.399 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:34.451 INFO:teuthology.orchestra.run.smithi067.stdout:client4815 2024-04-17T10:25:34.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:34.451 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8e8d1faf-cb51-4015-918b-f894d5d28e99.client4815/status') 2024-04-17T10:25:34.515 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4815 (3)192.168.0.1:0/2052018029 2024-04-17T10:25:34.516 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-04-17T10:25:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-04-17T10:25:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-04-17T10:25:34.516 INFO:teuthology.orchestra.run.smithi067.stderr:69 bytes copied, 5.3064e-05 s, 1.3 MB/s 2024-04-17T10:25:34.518 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-04-17T10:25:34.518 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T10:25:34.518 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-17T10:25:34.518 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:25:34.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 Processor -- start 2024-04-17T10:25:34.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 -- start start 2024-04-17T10:25:34.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61e80ce020 0x7f61e81620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:34.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f61e80ce400 0x7f61e815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e815e550 0x7f61e80cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e8161650 con 0x7f61e80ce020 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e81625e0 con 0x7f61e80ce400 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61eee00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e815eb50 con 0x7f61e815e550 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61e80ce020 0x7f61e81620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f61e80ce020 0x7f61e81620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59778/0 (socket says 172.21.15.67:59778) 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 -- 172.21.15.67:0/3720939364 learned_addr learned my addr 172.21.15.67:0/3720939364 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61ece00640 1 --2- 172.21.15.67:0/3720939364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e815e550 0x7f61e80cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61dfe00640 1 --2- 172.21.15.67:0/3720939364 >> v2:172.21.15.86:3300/0 conn(0x7f61e80ce400 0x7f61e815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 -- 172.21.15.67:0/3720939364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e815e550 msgr2=0x7f61e80cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 --2- 172.21.15.67:0/3720939364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e815e550 0x7f61e80cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 -- 172.21.15.67:0/3720939364 >> v2:172.21.15.86:3300/0 conn(0x7f61e80ce400 msgr2=0x7f61e815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 --2- 172.21.15.67:0/3720939364 >> v2:172.21.15.86:3300/0 conn(0x7f61e80ce400 0x7f61e815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61e7e00640 1 -- 172.21.15.67:0/3720939364 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e815ee30 con 0x7f61e80ce020 2024-04-17T10:25:34.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.707+0000 7f61dfe00640 1 --2- 172.21.15.67:0/3720939364 >> v2:172.21.15.86:3300/0 conn(0x7f61e80ce400 0x7f61e815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:34.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61e7e00640 1 --2- 172.21.15.67:0/3720939364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce020 0x7f61e81620a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009870 tx=0x7f61d801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee989b280f861870 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:34.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61e7400640 1 -- 172.21.15.67:0/3720939364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d80b1070 con 0x7f61e80ce020 2024-04-17T10:25:34.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61e7400640 1 -- 172.21.15.67:0/3720939364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d80ac620 con 0x7f61e80ce020 2024-04-17T10:25:34.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61e7400640 1 -- 172.21.15.67:0/3720939364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d80ac9f0 con 0x7f61e80ce020 2024-04-17T10:25:34.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 -- 172.21.15.67:0/3720939364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce020 msgr2=0x7f61e81620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:34.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 --2- 172.21.15.67:0/3720939364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce020 0x7f61e81620a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009870 tx=0x7f61d801c4f0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 -- 172.21.15.67:0/3720939364 shutdown_connections 2024-04-17T10:25:34.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 --2- 172.21.15.67:0/3720939364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e815e550 0x7f61e80cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 --2- 172.21.15.67:0/3720939364 >> v2:172.21.15.86:3300/0 conn(0x7f61e80ce400 0x7f61e815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 --2- 172.21.15.67:0/3720939364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce020 0x7f61e81620a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 -- 172.21.15.67:0/3720939364 >> 172.21.15.67:0/3720939364 conn(0x7f61e80b33f0 msgr2=0x7f61e80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:34.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 -- 172.21.15.67:0/3720939364 shutdown_connections 2024-04-17T10:25:34.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.711+0000 7f61eee00640 1 -- 172.21.15.67:0/3720939364 wait complete. 2024-04-17T10:25:34.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 Processor -- start 2024-04-17T10:25:34.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 -- start start 2024-04-17T10:25:34.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e80ce020 0x7f61e81494c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:34.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce400 0x7f61e814ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:34.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e815e550 0x7f61e814b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:34.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e80ce990 con 0x7f61e80ce400 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e80ce810 con 0x7f61e815e550 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e80c84b0 con 0x7f61e80ce020 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce400 0x7f61e814ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce400 0x7f61e814ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59784/0 (socket says 172.21.15.67:59784) 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 -- 172.21.15.67:0/4254123742 learned_addr learned my addr 172.21.15.67:0/4254123742 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61e7e00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e80ce020 0x7f61e81494c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61ece00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e815e550 0x7f61e814b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 -- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e80ce020 msgr2=0x7f61e81494c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e80ce020 0x7f61e81494c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 -- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e815e550 msgr2=0x7f61e814b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e815e550 0x7f61e814b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 -- 172.21.15.67:0/4254123742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e815ff20 con 0x7f61e80ce400 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61dfe00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce400 0x7f61e814ae30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c7f0 tx=0x7f61d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61e4c00640 1 -- 172.21.15.67:0/4254123742 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0015070 con 0x7f61e80ce400 2024-04-17T10:25:34.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61e4c00640 1 -- 172.21.15.67:0/4254123742 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d0007a30 con 0x7f61e80ce400 2024-04-17T10:25:34.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61e4c00640 1 -- 172.21.15.67:0/4254123742 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0007e00 con 0x7f61e80ce400 2024-04-17T10:25:34.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e815bf20 con 0x7f61e80ce400 2024-04-17T10:25:34.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e815c4e0 con 0x7f61e80ce400 2024-04-17T10:25:34.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.715+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e8070ab0 con 0x7f61e80ce400 2024-04-17T10:25:34.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.719+0000 7f61e4c00640 1 -- 172.21.15.67:0/4254123742 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f61d0018070 con 0x7f61e80ce400 2024-04-17T10:25:34.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.719+0000 7f61e4c00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f61c4066f60 0x7f61c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:34.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.719+0000 7f61e4c00640 1 -- 172.21.15.67:0/4254123742 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6532+0+0 (secure 0 0 0) 0x7f61d0085240 con 0x7f61e80ce400 2024-04-17T10:25:34.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.719+0000 7f61e7e00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f61c4066f60 0x7f61c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:34.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.719+0000 7f61e7e00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f61c4066f60 0x7f61c4069420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f61d80ae000 tx=0x7f61d80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:34.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.723+0000 7f61e4c00640 1 -- 172.21.15.67:0/4254123742 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f61d0055820 con 0x7f61e80ce400 2024-04-17T10:25:34.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.927+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f61e8074e30 con 0x7f61e80ce400 2024-04-17T10:25:34.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.927+0000 7f61e4c00640 1 -- 172.21.15.67:0/4254123742 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f61d00598c0 con 0x7f61e80ce400 2024-04-17T10:25:34.923 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-17T10:25:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f61c4066f60 msgr2=0x7f61c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f61c4066f60 0x7f61c4069420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f61d80ae000 tx=0x7f61d80a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce400 msgr2=0x7f61e814ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce400 0x7f61e814ae30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f61d000c7f0 tx=0x7f61d000ccb0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 shutdown_connections 2024-04-17T10:25:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f61c4066f60 0x7f61c4069420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61e815e550 0x7f61e814b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61e80ce400 0x7f61e814ae30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 --2- 172.21.15.67:0/4254123742 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61e80ce020 0x7f61e81494c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 >> 172.21.15.67:0/4254123742 conn(0x7f61e80b33f0 msgr2=0x7f61e8070cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 shutdown_connections 2024-04-17T10:25:34.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:34.931+0000 7f61eee00640 1 -- 172.21.15.67:0/4254123742 wait complete. 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:25:34.944 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-17T10:25:34.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:34.945 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:25:34.954 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:34.954 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42411sec preferred_lft 42411sec 2024-04-17T10:25:34.955 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link tentative 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:25:34.956 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link tentative 2024-04-17T10:25:34.957 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:34.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:34.957 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:25:35.018 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:25:35.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.020 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:25:35.076 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:25:35.076 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:25:35.076 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:25:35.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.077 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:35.078 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:35.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:25:35.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:25:35.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:25:35.078 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:35.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.174 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:25:35.183 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:25:35.184 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.184 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-04-17T10:25:35.232 INFO:teuthology.orchestra.run.smithi067.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-17T10:25:35.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.233 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:35.234 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:35.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:25:35.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-17T10:25:35.234 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:35.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.314 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-17T10:25:35.376 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:35.376 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 12:74:5a:69:0b:79 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::1074:5aff:fe69:b79/64 scope link tentative 2024-04-17T10:25:35.377 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-17T10:25:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:35.440 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:25:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 12:74:5a:69:0b:79 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T10:25:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T10:25:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::1074:5aff:fe69:b79/64 scope link 2024-04-17T10:25:35.441 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:35.442 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-17T10:25:35.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-17T10:25:35.443 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:35.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.578 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:25:35.579 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:25:35.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.1 up 2024-04-17T10:25:35.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.1 master ceph-brx 2024-04-17T10:25:35.579 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:25:35.622 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:35.622 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-17T10:25:35.672 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-17T10:25:35.672 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:35.673 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-17T10:25:35.724 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-04-17T10:25:35.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:35.820 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:35.820 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:35.836 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:35.837 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:35.837 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:25:35.837 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 2@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:25:35.837 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:25:35.842 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:25:35.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-17T10:25:35.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.861 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:35.915 INFO:teuthology.orchestra.run.smithi067.stdout:client4610 2024-04-17T10:25:35.915 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:35.915 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:35.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:35.968 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:36.027 INFO:teuthology.orchestra.run.smithi067.stdout:client4610 2024-04-17T10:25:36.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.027 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:36.078 INFO:teuthology.orchestra.run.smithi067.stdout:client4610 2024-04-17T10:25:36.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.079 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8e8d1faf-cb51-4015-918b-f894d5d28e99.client4610/status') 2024-04-17T10:25:36.139 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.4610 (3)172.21.15.67:0/1382194029 2024-04-17T10:25:36.139 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-04-17T10:25:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-04-17T10:25:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-04-17T10:25:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:70 bytes copied, 5.2318e-05 s, 1.3 MB/s 2024-04-17T10:25:36.141 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-04-17T10:25:36.142 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T10:25:36.142 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-17T10:25:36.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:25:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.331+0000 7f34b2600640 1 Processor -- start 2024-04-17T10:25:36.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.331+0000 7f34b2600640 1 -- start start 2024-04-17T10:25:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34b2600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34ac0d03a0 0x7f34ac15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34b2600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac09a740 0x7f34ac0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34b2600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34ac0cde20 con 0x7f34ac099a50 2024-04-17T10:25:36.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34b2600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34ac0cf300 con 0x7f34ac0d03a0 2024-04-17T10:25:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34b2600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0cf5e0 con 0x7f34ac09a740 2024-04-17T10:25:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34abe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34ac0d03a0 0x7f34ac15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:36.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34b1c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac09a740 0x7f34ac0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34abe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f34ac0d03a0 0x7f34ac15c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:44704/0 (socket says 172.21.15.67:44704) 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34b1200640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f34ac099a50 0x7f34ac099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:34778/0 (socket says 172.21.15.67:34778) 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34abe00640 1 -- 172.21.15.67:0/3011390308 learned_addr learned my addr 172.21.15.67:0/3011390308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34abe00640 1 -- 172.21.15.67:0/3011390308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac09a740 msgr2=0x7f34ac0cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34abe00640 1 --2- 172.21.15.67:0/3011390308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac09a740 0x7f34ac0cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34abe00640 1 -- 172.21.15.67:0/3011390308 >> v1:172.21.15.86:6789/0 conn(0x7f34ac099a50 legacy=0x7f34ac099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34abe00640 1 -- 172.21.15.67:0/3011390308 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac0cf8c0 con 0x7f34ac0d03a0 2024-04-17T10:25:36.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.335+0000 7f34ab400640 1 -- 172.21.15.67:0/3011390308 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286029544 0 0) 0x7f34ac0cde20 con 0x7f34ac099a50 2024-04-17T10:25:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34abe00640 1 --2- 172.21.15.67:0/3011390308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac0d03a0 0x7f34ac15c7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3498009870 tx=0x7f349801c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c087fdb941c69b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34ab400640 1 -- 172.21.15.67:0/3011390308 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980b2070 con 0x7f34ac0d03a0 2024-04-17T10:25:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34ab400640 1 -- 172.21.15.67:0/3011390308 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34980a9070 con 0x7f34ac0d03a0 2024-04-17T10:25:36.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34ab400640 1 -- 172.21.15.67:0/3011390308 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980ad880 con 0x7f34ac0d03a0 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 -- 172.21.15.67:0/3011390308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac0d03a0 msgr2=0x7f34ac15c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 --2- 172.21.15.67:0/3011390308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac0d03a0 0x7f34ac15c7a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3498009870 tx=0x7f349801c560 comp rx=0 tx=0).stop 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 -- 172.21.15.67:0/3011390308 shutdown_connections 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 --2- 172.21.15.67:0/3011390308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac09a740 0x7f34ac0cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 --2- 172.21.15.67:0/3011390308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac0d03a0 0x7f34ac15c7a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 -- 172.21.15.67:0/3011390308 >> 172.21.15.67:0/3011390308 conn(0x7f34ac0d1690 msgr2=0x7f34ac0d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 -- 172.21.15.67:0/3011390308 shutdown_connections 2024-04-17T10:25:36.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 -- 172.21.15.67:0/3011390308 wait complete. 2024-04-17T10:25:36.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 Processor -- start 2024-04-17T10:25:36.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.339+0000 7f34b2600640 1 -- start start 2024-04-17T10:25:36.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac099a50 0x7f34ac1505a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:36.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac09a740 0x7f34ac154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:36.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac0d03a0 0x7f34ac06cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:36.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac1686b0 con 0x7f34ac09a740 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac0cd180 con 0x7f34ac099a50 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac168830 con 0x7f34ac0d03a0 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac09a740 0x7f34ac154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac09a740 0x7f34ac154ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59792/0 (socket says 172.21.15.67:59792) 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 -- 172.21.15.67:0/3360038133 learned_addr learned my addr 172.21.15.67:0/3360038133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b1c00640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac0d03a0 0x7f34ac06cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b1200640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac099a50 0x7f34ac1505a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 -- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac0d03a0 msgr2=0x7f34ac06cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac0d03a0 0x7f34ac06cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 -- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac099a50 msgr2=0x7f34ac1505a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac099a50 0x7f34ac1505a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 -- 172.21.15.67:0/3360038133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac1554a0 con 0x7f34ac09a740 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b1200640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac099a50 0x7f34ac1505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:36.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34abe00640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac09a740 0x7f34ac154ba0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f349801ac10 tx=0x7f3498019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34a8c00640 1 -- 172.21.15.67:0/3360038133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980b2050 con 0x7f34ac09a740 2024-04-17T10:25:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34a8c00640 1 -- 172.21.15.67:0/3360038133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34980b3040 con 0x7f34ac09a740 2024-04-17T10:25:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34a8c00640 1 -- 172.21.15.67:0/3360038133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980b9450 con 0x7f34ac09a740 2024-04-17T10:25:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac155780 con 0x7f34ac09a740 2024-04-17T10:25:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ac155d40 con 0x7f34ac09a740 2024-04-17T10:25:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ac074dc0 con 0x7f34ac09a740 2024-04-17T10:25:36.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.343+0000 7f34a8c00640 1 -- 172.21.15.67:0/3360038133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f34980c0020 con 0x7f34ac09a740 2024-04-17T10:25:36.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.347+0000 7f34a8c00640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f348c066f60 0x7f348c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:36.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.347+0000 7f34a8c00640 1 -- 172.21.15.67:0/3360038133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f349811f250 con 0x7f34ac09a740 2024-04-17T10:25:36.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.347+0000 7f34b1200640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f348c066f60 0x7f348c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:36.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.347+0000 7f34b1200640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f348c066f60 0x7f348c069420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f34a0000f70 tx=0x7f34a0007cb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:36.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.351+0000 7f34a8c00640 1 -- 172.21.15.67:0/3360038133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3498016150 con 0x7f34ac09a740 2024-04-17T10:25:36.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f34ac0d24a0 con 0x7f34ac09a740 2024-04-17T10:25:36.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34a8c00640 1 -- 172.21.15.67:0/3360038133 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f34980efe50 con 0x7f34ac09a740 2024-04-17T10:25:36.542 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-17T10:25:36.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f348c066f60 msgr2=0x7f348c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f348c066f60 0x7f348c069420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f34a0000f70 tx=0x7f34a0007cb0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac09a740 msgr2=0x7f34ac154ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac09a740 0x7f34ac154ba0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f349801ac10 tx=0x7f3498019300 comp rx=0 tx=0).stop 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 shutdown_connections 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f348c066f60 0x7f348c069420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34ac0d03a0 0x7f34ac06cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34ac09a740 0x7f34ac154ba0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 --2- 172.21.15.67:0/3360038133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34ac099a50 0x7f34ac1505a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 >> 172.21.15.67:0/3360038133 conn(0x7f34ac0d1690 msgr2=0x7f34ac0af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 shutdown_connections 2024-04-17T10:25:36.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:36.547+0000 7f34b2600640 1 -- 172.21.15.67:0/3360038133 wait complete. 2024-04-17T10:25:36.562 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T10:25:36.562 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:25:36.563 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi086.front.sepia.ceph.com 2024-04-17T10:25:36.563 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-17T10:25:36.563 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-17T10:25:36.563 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T10:25:36.563 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T10:25:36.563 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:25:36.563 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-17T10:25:36.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.563 DEBUG:teuthology.orchestra.run.smithi086:> ip addr 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f6 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f0 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:36.573 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f7 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f1 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:08 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:09 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: inet 172.21.15.86/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft 42409sec preferred_lft 42409sec 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 02:42:1c:4e:ee:6f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:25:36.574 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:36.575 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-17T10:25:36.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.575 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:36.575 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:36.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link add name ceph-brx type bridge 2024-04-17T10:25:36.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip addr flush dev ceph-brx 2024-04-17T10:25:36.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set ceph-brx up 2024-04-17T10:25:36.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-17T10:25:36.576 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:36.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.690 DEBUG:teuthology.orchestra.run.smithi086:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:25:36.708 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-04-17T10:25:36.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.711 DEBUG:teuthology.orchestra.run.smithi086:> route 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:Kernel IP routing table 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:25:36.783 INFO:teuthology.orchestra.run.smithi086.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:36.784 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:36.784 INFO:teuthology.orchestra.run.smithi086.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:36.784 INFO:teuthology.orchestra.run.smithi086.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:25:36.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.784 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:36.784 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:36.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:25:36.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:25:36.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:25:36.785 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:36.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.877 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-04-17T10:25:36.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.887 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list-id 2024-04-17T10:25:36.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:36.936 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:36.936 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:36.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-17T10:25:36.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-17T10:25:36.936 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:37.018 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-17T10:25:37.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:37.018 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:37.018 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:37.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-17T10:25:37.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-17T10:25:37.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-17T10:25:37.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-17T10:25:37.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-17T10:25:37.019 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:37.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:37.186 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:37.187 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:37.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set brx.0 up 2024-04-17T10:25:37.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set dev brx.0 master ceph-brx 2024-04-17T10:25:37.187 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:37.228 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:37.228 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-17T10:25:37.284 INFO:teuthology.orchestra.run.smithi086.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-17T10:25:37.284 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:37.284 DEBUG:teuthology.orchestra.run.smithi086:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-17T10:25:37.335 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-17T10:25:37.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:37.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:37.425 INFO:teuthology.orchestra.run.smithi086.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:37.425 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:38.134 INFO:teuthology.orchestra.run.smithi086.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:38.134 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:38.134 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:25:38.134 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: trying mount with new device syntax: 1@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:25:38.134 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:25:38.138 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:25:38.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-17T10:25:38.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.200 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:38.264 INFO:teuthology.orchestra.run.smithi086.stdout:client4619 2024-04-17T10:25:38.264 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:38.264 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:38.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.316 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:38.371 INFO:teuthology.orchestra.run.smithi086.stdout:client4619 2024-04-17T10:25:38.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.371 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:38.422 INFO:teuthology.orchestra.run.smithi086.stdout:client4619 2024-04-17T10:25:38.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.423 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8e8d1faf-cb51-4015-918b-f894d5d28e99.client4619/status') 2024-04-17T10:25:38.484 INFO:teuthology.orchestra.run.smithi086.stdout:instance: client.4619 (3)192.168.0.1:0/1025168763 2024-04-17T10:25:38.484 INFO:teuthology.orchestra.run.smithi086.stdout:blocklisted: false 2024-04-17T10:25:38.484 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2024-04-17T10:25:38.484 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2024-04-17T10:25:38.484 INFO:teuthology.orchestra.run.smithi086.stderr:69 bytes copied, 5.6824e-05 s, 1.2 MB/s 2024-04-17T10:25:38.485 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-04-17T10:25:38.485 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-17T10:25:38.485 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-17T10:25:38.485 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:25:38.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f803cc00640 1 Processor -- start 2024-04-17T10:25:38.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f803cc00640 1 -- start start 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f803cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80381446c0 0x7f8038144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f803cc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8038144aa0 0x7f80380d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f803cc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80380b6640 con 0x7f8038071840 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f803cc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80380b51a0 con 0x7f80381446c0 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f803cc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80381456d0 con 0x7f8038144aa0 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f8035e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80381446c0 0x7f8038144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f8035e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f80381446c0 0x7f8038144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59798/0 (socket says 172.21.15.67:59798) 2024-04-17T10:25:38.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f8035e00640 1 -- 172.21.15.67:0/100232286 learned_addr learned my addr 172.21.15.67:0/100232286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f802d400640 1 --2- 172.21.15.67:0/100232286 >> v2:172.21.15.86:3300/0 conn(0x7f8038144aa0 0x7f80380d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f8035e00640 1 -- 172.21.15.67:0/100232286 >> v2:172.21.15.86:3300/0 conn(0x7f8038144aa0 msgr2=0x7f80380d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f8035e00640 1 --2- 172.21.15.67:0/100232286 >> v2:172.21.15.86:3300/0 conn(0x7f8038144aa0 0x7f80380d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f8035e00640 1 -- 172.21.15.67:0/100232286 >> v1:172.21.15.86:6789/0 conn(0x7f8038071840 legacy=0x7f803815f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:38.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.643+0000 7f8035e00640 1 -- 172.21.15.67:0/100232286 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80380b4870 con 0x7f80381446c0 2024-04-17T10:25:38.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f8035e00640 1 --2- 172.21.15.67:0/100232286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80381446c0 0x7f8038144ea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f803815f620 tx=0x7f802801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169e885329967b7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:38.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f8035400640 1 -- 172.21.15.67:0/100232286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80280aa650 con 0x7f80381446c0 2024-04-17T10:25:38.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f8035400640 1 -- 172.21.15.67:0/100232286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80280af070 con 0x7f80381446c0 2024-04-17T10:25:38.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f8035400640 1 -- 172.21.15.67:0/100232286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80280aaa10 con 0x7f80381446c0 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 -- 172.21.15.67:0/100232286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80381446c0 msgr2=0x7f8038144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 --2- 172.21.15.67:0/100232286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80381446c0 0x7f8038144ea0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f803815f620 tx=0x7f802801c4f0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 -- 172.21.15.67:0/100232286 shutdown_connections 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 --2- 172.21.15.67:0/100232286 >> v2:172.21.15.86:3300/0 conn(0x7f8038144aa0 0x7f80380d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 --2- 172.21.15.67:0/100232286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80381446c0 0x7f8038144ea0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 -- 172.21.15.67:0/100232286 >> 172.21.15.67:0/100232286 conn(0x7f80380b34d0 msgr2=0x7f80380b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 -- 172.21.15.67:0/100232286 shutdown_connections 2024-04-17T10:25:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.647+0000 7f803cc00640 1 -- 172.21.15.67:0/100232286 wait complete. 2024-04-17T10:25:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 Processor -- start 2024-04-17T10:25:38.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 -- start start 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8038071840 0x7f803806c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80381446c0 0x7f8038153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8038144aa0 0x7f8038154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80380d1260 con 0x7f8038144aa0 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803815c0d0 con 0x7f8038071840 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8038168a40 con 0x7f80381446c0 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8038144aa0 0x7f8038154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8038144aa0 0x7f8038154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59800/0 (socket says 172.21.15.67:59800) 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 -- 172.21.15.67:0/2246680163 learned_addr learned my addr 172.21.15.67:0/2246680163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f802d400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80381446c0 0x7f8038153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8035e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8038071840 0x7f803806c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 -- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80381446c0 msgr2=0x7f8038153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80381446c0 0x7f8038153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 -- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8038071840 msgr2=0x7f803806c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8038071840 0x7f803806c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 -- 172.21.15.67:0/2246680163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803815b400 con 0x7f8038144aa0 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f8036800640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8038144aa0 0x7f8038154520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f803000c3c0 tx=0x7f803000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f802ea00640 1 -- 172.21.15.67:0/2246680163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8030017070 con 0x7f8038144aa0 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f802ea00640 1 -- 172.21.15.67:0/2246680163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80300052d0 con 0x7f8038144aa0 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f802ea00640 1 -- 172.21.15.67:0/2246680163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8030003860 con 0x7f8038144aa0 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803815b6e0 con 0x7f8038144aa0 2024-04-17T10:25:38.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.651+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803806d8d0 con 0x7f8038144aa0 2024-04-17T10:25:38.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.655+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8038070bb0 con 0x7f8038144aa0 2024-04-17T10:25:38.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.655+0000 7f802ea00640 1 -- 172.21.15.67:0/2246680163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f803001a070 con 0x7f8038144aa0 2024-04-17T10:25:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.655+0000 7f802ea00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8014066f90 0x7f8014069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.655+0000 7f802ea00640 1 -- 172.21.15.67:0/2246680163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f8030084690 con 0x7f8038144aa0 2024-04-17T10:25:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.655+0000 7f8035e00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8014066f90 0x7f8014069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.655+0000 7f8035e00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8014066f90 0x7f8014069450 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80280ae000 tx=0x7f80280a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.659+0000 7f802ea00640 1 -- 172.21.15.67:0/2246680163 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8030054c40 con 0x7f8038144aa0 2024-04-17T10:25:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.855+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8038074e30 con 0x7f8038144aa0 2024-04-17T10:25:38.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.855+0000 7f802ea00640 1 -- 172.21.15.67:0/2246680163 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f8030058ce0 con 0x7f8038144aa0 2024-04-17T10:25:38.852 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8014066f90 msgr2=0x7f8014069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8014066f90 0x7f8014069450 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80280ae000 tx=0x7f80280a7000 comp rx=0 tx=0).stop 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8038144aa0 msgr2=0x7f8038154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8038144aa0 0x7f8038154520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f803000c3c0 tx=0x7f803000c880 comp rx=0 tx=0).stop 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 shutdown_connections 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8014066f90 0x7f8014069450 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8038144aa0 0x7f8038154520 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f80381446c0 0x7f8038153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 --2- 172.21.15.67:0/2246680163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8038071840 0x7f803806c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 >> 172.21.15.67:0/2246680163 conn(0x7f80380b34d0 msgr2=0x7f8038140e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 shutdown_connections 2024-04-17T10:25:38.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:38.859+0000 7f803cc00640 1 -- 172.21.15.67:0/2246680163 wait complete. 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi086.front.sepia.ceph.com 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:25:38.873 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-17T10:25:38.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.873 DEBUG:teuthology.orchestra.run.smithi086:> ip addr 2024-04-17T10:25:38.883 INFO:teuthology.orchestra.run.smithi086.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:38.883 INFO:teuthology.orchestra.run.smithi086.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:38.883 INFO:teuthology.orchestra.run.smithi086.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:38.883 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:38.883 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:38.883 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f6 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f0 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:d8:da:f7 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: altname enp7s0f1 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:08 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0c:c4:7a:8f:d8:09 brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: inet 172.21.15.86/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft 42407sec preferred_lft 42407sec 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2024-04-17T10:25:38.884 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 02:42:1c:4e:ee:6f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:25:38.885 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:38.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.886 DEBUG:teuthology.orchestra.run.smithi086:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:25:38.947 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-04-17T10:25:38.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:38.950 DEBUG:teuthology.orchestra.run.smithi086:> route 2024-04-17T10:25:39.008 INFO:teuthology.orchestra.run.smithi086.stdout:Kernel IP routing table 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run.smithi086.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:25:39.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.010 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:39.010 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:39.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:25:39.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:25:39.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:25:39.010 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:39.104 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.105 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list 2024-04-17T10:25:39.114 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-17T10:25:39.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.114 DEBUG:teuthology.orchestra.run.smithi086:> ip netns list-id 2024-04-17T10:25:39.168 INFO:teuthology.orchestra.run.smithi086.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-17T10:25:39.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.169 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:39.169 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:39.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-17T10:25:39.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-17T10:25:39.169 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:39.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-17T10:25:39.316 INFO:teuthology.orchestra.run.smithi086.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.316 INFO:teuthology.orchestra.run.smithi086.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:25:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 6e:6a:89:09:46:34 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T10:25:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T10:25:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::6c6a:89ff:fe09:4634/64 scope link 2024-04-17T10:25:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-17T10:25:39.379 INFO:teuthology.orchestra.run.smithi086.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:25:39.379 INFO:teuthology.orchestra.run.smithi086.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:25:39.379 INFO:teuthology.orchestra.run.smithi086.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 ::1/128 scope host 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: link/ether 6e:6a:89:09:46:34 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: inet6 fe80::6c6a:89ff:fe09:4634/64 scope link 2024-04-17T10:25:39.380 INFO:teuthology.orchestra.run.smithi086.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:25:39.381 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-17T10:25:39.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-17T10:25:39.382 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:39.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.517 DEBUG:teuthology.orchestra.run.smithi086:> (cd / && exec bash -c ' 2024-04-17T10:25:39.517 DEBUG:teuthology.orchestra.run.smithi086:> set -e 2024-04-17T10:25:39.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set brx.1 up 2024-04-17T10:25:39.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo ip link set dev brx.1 master ceph-brx 2024-04-17T10:25:39.517 DEBUG:teuthology.orchestra.run.smithi086:> ') 2024-04-17T10:25:39.555 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:39.555 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-17T10:25:39.604 INFO:teuthology.orchestra.run.smithi086.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-17T10:25:39.604 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:25:39.604 DEBUG:teuthology.orchestra.run.smithi086:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-17T10:25:39.655 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-04-17T10:25:39.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:39.742 INFO:teuthology.orchestra.run.smithi086.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:39.743 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:39.757 INFO:teuthology.orchestra.run.smithi086.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:25:39.757 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:25:39.757 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:25:39.757 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: trying mount with new device syntax: 3@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:25:39.757 INFO:teuthology.orchestra.run.smithi086.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:25:39.762 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:25:39.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-17T10:25:39.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.781 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:39.835 INFO:teuthology.orchestra.run.smithi086.stdout:client4810 2024-04-17T10:25:39.835 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:39.835 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:39.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.884 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:39.938 INFO:teuthology.orchestra.run.smithi086.stdout:client4810 2024-04-17T10:25:39.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.939 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:25:39.990 INFO:teuthology.orchestra.run.smithi086.stdout:client4810 2024-04-17T10:25:39.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:39.991 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/8e8d1faf-cb51-4015-918b-f894d5d28e99.client4810/status') 2024-04-17T10:25:40.050 INFO:teuthology.orchestra.run.smithi086.stdout:instance: client.4810 (3)192.168.0.2:0/546471928 2024-04-17T10:25:40.051 INFO:teuthology.orchestra.run.smithi086.stdout:blocklisted: false 2024-04-17T10:25:40.051 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records in 2024-04-17T10:25:40.051 INFO:teuthology.orchestra.run.smithi086.stderr:0+1 records out 2024-04-17T10:25:40.051 INFO:teuthology.orchestra.run.smithi086.stderr:68 bytes copied, 4.971e-05 s, 1.4 MB/s 2024-04-17T10:25:40.053 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-17T10:25:40.069 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:40.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.223+0000 7f5164600640 1 Processor -- start 2024-04-17T10:25:40.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.223+0000 7f5164600640 1 -- start start 2024-04-17T10:25:40.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.223+0000 7f5164600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 0x7f515c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.223+0000 7f5164600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515c15eb50 con 0x7f515c0ce020 2024-04-17T10:25:40.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5164600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515c15ee30 con 0x7f515c15e550 2024-04-17T10:25:40.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5164600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c161650 con 0x7f515c1620a0 2024-04-17T10:25:40.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f515c0ce020 0x7f515c0ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40944/0 (socket says 172.21.15.67:40944) 2024-04-17T10:25:40.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161c00640 1 -- 172.21.15.67:0/3076886362 learned_addr learned my addr 172.21.15.67:0/3076886362 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:40.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161200640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 0x7f515c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293334946 0 0) 0x7f515c15eb50 con 0x7f515c0ce020 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5144003530 con 0x7f515c0ce020 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 763358206 0 0) 0x7f5144003530 con 0x7f515c0ce020 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 msgr2=0x7f515c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 0x7f515c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 >> v1:172.21.15.86:6789/0 conn(0x7f515c15e550 legacy=0x7f515c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c1618a0 con 0x7f515c0ce020 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f5150001680 con 0x7f515c0ce020 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 >> v1:172.21.15.67:6789/0 conn(0x7f515c0ce020 legacy=0x7f515c0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 0x7f51440039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f5144004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5144004d50 0x7f51440051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c161650 con 0x7f5144003f30 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514400ba30 con 0x7f5144004d50 2024-04-17T10:25:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514400bc30 con 0x7f515c1620a0 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161200640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f5144004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5162600640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5144004d50 0x7f51440051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161c00640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 0x7f51440039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161200640 1 -- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 msgr2=0x7f51440039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161200640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c1620a0 0x7f51440039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161200640 1 -- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5144004d50 msgr2=0x7f51440051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161200640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5144004d50 0x7f51440051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5161200640 1 -- 172.21.15.67:0/3076886362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5144003530 con 0x7f5144003f30 2024-04-17T10:25:40.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.227+0000 7f5162600640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5144004d50 0x7f51440051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:40.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5161200640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f5144004810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5148009ae0 tx=0x7f514801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8421e5a70a532a5a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480b2070 con 0x7f5144003f30 2024-04-17T10:25:40.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51480a55a0 con 0x7f5144003f30 2024-04-17T10:25:40.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f515be00640 1 -- 172.21.15.67:0/3076886362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51480096b0 con 0x7f5144003f30 2024-04-17T10:25:40.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 -- 172.21.15.67:0/3076886362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 msgr2=0x7f5144004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 --2- 172.21.15.67:0/3076886362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f5144004810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5148009ae0 tx=0x7f514801c9b0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5162600640 1 -- 172.21.15.67:0/3076886362 reap_dead start 2024-04-17T10:25:40.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 -- 172.21.15.67:0/3076886362 shutdown_connections 2024-04-17T10:25:40.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 -- 172.21.15.67:0/3076886362 >> 172.21.15.67:0/3076886362 conn(0x7f515c0b33f0 msgr2=0x7f515c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:40.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 -- 172.21.15.67:0/3076886362 shutdown_connections 2024-04-17T10:25:40.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 -- 172.21.15.67:0/3076886362 wait complete. 2024-04-17T10:25:40.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 Processor -- start 2024-04-17T10:25:40.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.231+0000 7f5164600640 1 -- start start 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f515c148320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f515c15bed0 0x7f515c15d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c148d80 0x7f515c149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c06bf70 con 0x7f5144003f30 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c162500 con 0x7f515c15bed0 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c06c0f0 con 0x7f515c148d80 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f515c148320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f515c148320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59820/0 (socket says 172.21.15.67:59820) 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f515c15bed0 0x7f515c15d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 -- 172.21.15.67:0/3446400078 learned_addr learned my addr 172.21.15.67:0/3446400078 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5162600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c148d80 0x7f515c149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 -- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c148d80 msgr2=0x7f515c149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c148d80 0x7f515c149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 -- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f515c15bed0 msgr2=0x7f515c15d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f515c15bed0 0x7f515c15d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 -- 172.21.15.67:0/3446400078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c148860 con 0x7f5144003f30 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5161c00640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f515c148320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f515000e710 tx=0x7f515000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5159600640 1 -- 172.21.15.67:0/3446400078 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515000c640 con 0x7f5144003f30 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5159600640 1 -- 172.21.15.67:0/3446400078 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150018070 con 0x7f5144003f30 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c15c360 con 0x7f5144003f30 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5159600640 1 -- 172.21.15.67:0/3446400078 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515000ca40 con 0x7f5144003f30 2024-04-17T10:25:40.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c15c920 con 0x7f5144003f30 2024-04-17T10:25:40.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c070ab0 con 0x7f5144003f30 2024-04-17T10:25:40.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.235+0000 7f5159600640 1 -- 172.21.15.67:0/3446400078 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f515001d460 con 0x7f5144003f30 2024-04-17T10:25:40.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.239+0000 7f5159600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f513c066f60 0x7f513c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.239+0000 7f5159600640 1 -- 172.21.15.67:0/3446400078 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f5150085340 con 0x7f5144003f30 2024-04-17T10:25:40.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.239+0000 7f5161200640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f513c066f60 0x7f513c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.239+0000 7f5161200640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f513c066f60 0x7f513c069420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f51480af000 tx=0x7f51480046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.243+0000 7f5159600640 1 -- 172.21.15.67:0/3446400078 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5150055860 con 0x7f5144003f30 2024-04-17T10:25:40.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.407+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f515c074e30 con 0x7f5144003f30 2024-04-17T10:25:40.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.407+0000 7f5159600640 1 -- 172.21.15.67:0/3446400078 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f5150059900 con 0x7f5144003f30 2024-04-17T10:25:40.403 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:40.403 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:40.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f513c066f60 msgr2=0x7f513c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f513c066f60 0x7f513c069420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f51480af000 tx=0x7f51480046d0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 msgr2=0x7f515c148320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f515c148320 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f515000e710 tx=0x7f515000ebd0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 shutdown_connections 2024-04-17T10:25:40.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f513c066f60 0x7f513c069420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f515c148d80 0x7f515c149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f515c15bed0 0x7f515c15d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 --2- 172.21.15.67:0/3446400078 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5144003f30 0x7f515c148320 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 >> 172.21.15.67:0/3446400078 conn(0x7f515c0b33f0 msgr2=0x7f515c160980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:40.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 shutdown_connections 2024-04-17T10:25:40.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.411+0000 7f5164600640 1 -- 172.21.15.67:0/3446400078 wait complete. 2024-04-17T10:25:40.425 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:40.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07e1800640 1 Processor -- start 2024-04-17T10:25:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07e1800640 1 -- start start 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07e1800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07dc161f90 0x7f07dc0cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07e1800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f07dc09adf0 0x7f07dc0d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07e1800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07dc15e140 con 0x7f07dc15dd60 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07e1800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07dc1626d0 con 0x7f07dc161f90 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07e1800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07dc099d50 con 0x7f07dc09adf0 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07d2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f07dc161f90 0x7f07dc0cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07daa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f07dc15dd60 0x7f07dc161880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37194/0 (socket says 172.21.15.67:37194) 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07daa00640 1 -- 172.21.15.67:0/67843841 learned_addr learned my addr 172.21.15.67:0/67843841 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07db400640 1 --2- 172.21.15.67:0/67843841 >> v2:172.21.15.86:3300/0 conn(0x7f07dc09adf0 0x7f07dc0d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07d2000640 1 -- 172.21.15.67:0/67843841 >> v2:172.21.15.86:3300/0 conn(0x7f07dc09adf0 msgr2=0x7f07dc0d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07d2000640 1 --2- 172.21.15.67:0/67843841 >> v2:172.21.15.86:3300/0 conn(0x7f07dc09adf0 0x7f07dc0d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07d2000640 1 -- 172.21.15.67:0/67843841 >> v1:172.21.15.67:6789/0 conn(0x7f07dc15dd60 legacy=0x7f07dc161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.575+0000 7f07d2000640 1 -- 172.21.15.67:0/67843841 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc09a030 con 0x7f07dc161f90 2024-04-17T10:25:40.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07d2000640 1 --2- 172.21.15.67:0/67843841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc0cff90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0c8a10 tx=0x7f07c401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf405ed559260f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07da000640 1 -- 172.21.15.67:0/67843841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40b0070 con 0x7f07dc161f90 2024-04-17T10:25:40.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07da000640 1 -- 172.21.15.67:0/67843841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c40ab600 con 0x7f07dc161f90 2024-04-17T10:25:40.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07da000640 1 -- 172.21.15.67:0/67843841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40ab9d0 con 0x7f07dc161f90 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 -- 172.21.15.67:0/67843841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 msgr2=0x7f07dc0cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 --2- 172.21.15.67:0/67843841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc0cff90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0c8a10 tx=0x7f07c401c480 comp rx=0 tx=0).stop 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 -- 172.21.15.67:0/67843841 shutdown_connections 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 --2- 172.21.15.67:0/67843841 >> v2:172.21.15.86:3300/0 conn(0x7f07dc09adf0 0x7f07dc0d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 --2- 172.21.15.67:0/67843841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc0cff90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 -- 172.21.15.67:0/67843841 >> 172.21.15.67:0/67843841 conn(0x7f07dc0b34a0 msgr2=0x7f07dc0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 -- 172.21.15.67:0/67843841 shutdown_connections 2024-04-17T10:25:40.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.579+0000 7f07e1800640 1 -- 172.21.15.67:0/67843841 wait complete. 2024-04-17T10:25:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 Processor -- start 2024-04-17T10:25:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 -- start start 2024-04-17T10:25:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07dc09adf0 0x7f07dc15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07dc15dd60 0x7f07dc156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc06c730 con 0x7f07dc161f90 2024-04-17T10:25:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc06c5b0 con 0x7f07dc15dd60 2024-04-17T10:25:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07dc06c8b0 con 0x7f07dc09adf0 2024-04-17T10:25:40.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56722/0 (socket says 172.21.15.67:56722) 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 -- 172.21.15.67:0/896205249 learned_addr learned my addr 172.21.15.67:0/896205249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07daa00640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07dc09adf0 0x7f07dc15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07d2000640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07dc15dd60 0x7f07dc156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 -- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07dc09adf0 msgr2=0x7f07dc15a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07dc09adf0 0x7f07dc15a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 -- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07dc15dd60 msgr2=0x7f07dc156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07dc15dd60 0x7f07dc156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 -- 172.21.15.67:0/896205249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07dc15bdd0 con 0x7f07dc161f90 2024-04-17T10:25:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07db400640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc158030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009dd0 tx=0x7f07d400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07d3400640 1 -- 172.21.15.67:0/896205249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4018020 con 0x7f07dc161f90 2024-04-17T10:25:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07d3400640 1 -- 172.21.15.67:0/896205249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d4014480 con 0x7f07dc161f90 2024-04-17T10:25:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07d3400640 1 -- 172.21.15.67:0/896205249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d4014870 con 0x7f07dc161f90 2024-04-17T10:25:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07dc15ce30 con 0x7f07dc161f90 2024-04-17T10:25:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07dc15c350 con 0x7f07dc161f90 2024-04-17T10:25:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.583+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07dc04ef30 con 0x7f07dc161f90 2024-04-17T10:25:40.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.587+0000 7f07d3400640 1 -- 172.21.15.67:0/896205249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f07d401b020 con 0x7f07dc161f90 2024-04-17T10:25:40.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.587+0000 7f07d3400640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b8066f60 0x7f07b8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.587+0000 7f07d3400640 1 -- 172.21.15.67:0/896205249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f07d4085e30 con 0x7f07dc161f90 2024-04-17T10:25:40.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.587+0000 7f07daa00640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b8066f60 0x7f07b8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.587+0000 7f07daa00640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b8066f60 0x7f07b8069420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f07dc168bc0 tx=0x7f07cc007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.591+0000 7f07d3400640 1 -- 172.21.15.67:0/896205249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f07d4022070 con 0x7f07dc161f90 2024-04-17T10:25:40.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.751+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f07dc0b42d0 con 0x7f07dc161f90 2024-04-17T10:25:40.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.751+0000 7f07d3400640 1 -- 172.21.15.67:0/896205249 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f07d4010040 con 0x7f07dc161f90 2024-04-17T10:25:40.748 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b8066f60 msgr2=0x7f07b8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b8066f60 0x7f07b8069420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f07dc168bc0 tx=0x7f07cc007e90 comp rx=0 tx=0).stop 2024-04-17T10:25:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 msgr2=0x7f07dc158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc158030 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f07d4009dd0 tx=0x7f07d400c490 comp rx=0 tx=0).stop 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 shutdown_connections 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b8066f60 0x7f07b8069420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07dc161f90 0x7f07dc158030 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07dc15dd60 0x7f07dc156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 --2- 172.21.15.67:0/896205249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07dc09adf0 0x7f07dc15a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 >> 172.21.15.67:0/896205249 conn(0x7f07dc0b34a0 msgr2=0x7f07dc0b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 shutdown_connections 2024-04-17T10:25:40.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.755+0000 7f07e1800640 1 -- 172.21.15.67:0/896205249 wait complete. 2024-04-17T10:25:40.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:40.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7384c00640 1 Processor -- start 2024-04-17T10:25:40.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7384c00640 1 -- start start 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7384c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7380161f90 0x7f73800cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7384c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738009adf0 0x7f73800d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7384c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738015e140 con 0x7f738015dd60 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7384c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73801626d0 con 0x7f7380161f90 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7384c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7380099d50 con 0x7f738009adf0 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f737de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f738015dd60 0x7f7380161880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37204/0 (socket says 172.21.15.67:37204) 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f737de00640 1 -- 172.21.15.67:0/1807238485 learned_addr learned my addr 172.21.15.67:0/1807238485 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:40.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7375400640 1 --2- 172.21.15.67:0/1807238485 >> v2:172.21.15.67:3300/0 conn(0x7f7380161f90 0x7f73800cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f737e800640 1 --2- 172.21.15.67:0/1807238485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738009adf0 0x7f73800d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7375400640 1 -- 172.21.15.67:0/1807238485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738009adf0 msgr2=0x7f73800d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:40.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7375400640 1 --2- 172.21.15.67:0/1807238485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738009adf0 0x7f73800d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7375400640 1 -- 172.21.15.67:0/1807238485 >> v1:172.21.15.67:6789/0 conn(0x7f738015dd60 legacy=0x7f7380161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f737d400640 1 -- 172.21.15.67:0/1807238485 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070122838 0 0) 0x7f738015e140 con 0x7f738015dd60 2024-04-17T10:25:40.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f7375400640 1 -- 172.21.15.67:0/1807238485 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738009a030 con 0x7f7380161f90 2024-04-17T10:25:40.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.919+0000 7f737e800640 1 --2- 172.21.15.67:0/1807238485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738009adf0 0x7f73800d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:40.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7375400640 1 --2- 172.21.15.67:0/1807238485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7380161f90 0x7f73800cff90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f73680097c0 tx=0x7f736801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa17b8086e3880c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f737d400640 1 -- 172.21.15.67:0/1807238485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73680b2070 con 0x7f7380161f90 2024-04-17T10:25:40.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f737d400640 1 -- 172.21.15.67:0/1807238485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73680ad640 con 0x7f7380161f90 2024-04-17T10:25:40.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f737d400640 1 -- 172.21.15.67:0/1807238485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73680ada10 con 0x7f7380161f90 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 -- 172.21.15.67:0/1807238485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7380161f90 msgr2=0x7f73800cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 --2- 172.21.15.67:0/1807238485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7380161f90 0x7f73800cff90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f73680097c0 tx=0x7f736801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 -- 172.21.15.67:0/1807238485 shutdown_connections 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 --2- 172.21.15.67:0/1807238485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738009adf0 0x7f73800d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 --2- 172.21.15.67:0/1807238485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7380161f90 0x7f73800cff90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 -- 172.21.15.67:0/1807238485 >> 172.21.15.67:0/1807238485 conn(0x7f73800b34a0 msgr2=0x7f73800b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 -- 172.21.15.67:0/1807238485 shutdown_connections 2024-04-17T10:25:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 -- 172.21.15.67:0/1807238485 wait complete. 2024-04-17T10:25:40.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 Processor -- start 2024-04-17T10:25:40.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.923+0000 7f7384c00640 1 -- start start 2024-04-17T10:25:40.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f738009adf0 0x7f73800b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738015dd60 0x7f7380157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7380161f90 0x7f73800b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738006c730 con 0x7f738009adf0 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738006c5b0 con 0x7f7380161f90 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738006c8b0 con 0x7f738015dd60 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f738009adf0 0x7f73800b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f738009adf0 0x7f73800b73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56740/0 (socket says 172.21.15.67:56740) 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 -- 172.21.15.67:0/494947920 learned_addr learned my addr 172.21.15.67:0/494947920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737e800640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7380161f90 0x7f73800b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7375400640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738015dd60 0x7f7380157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 -- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738015dd60 msgr2=0x7f7380157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738015dd60 0x7f7380157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 -- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7380161f90 msgr2=0x7f73800b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7380161f90 0x7f73800b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 -- 172.21.15.67:0/494947920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7380070700 con 0x7f738009adf0 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737e800640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7380161f90 0x7f73800b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7375400640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738015dd60 0x7f7380157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f737de00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f738009adf0 0x7f73800b73b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f737000c890 tx=0x7f737000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7376a00640 1 -- 172.21.15.67:0/494947920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370018070 con 0x7f738009adf0 2024-04-17T10:25:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7376a00640 1 -- 172.21.15.67:0/494947920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737000a640 con 0x7f738009adf0 2024-04-17T10:25:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738015ec60 con 0x7f738009adf0 2024-04-17T10:25:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7376a00640 1 -- 172.21.15.67:0/494947920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737000aa30 con 0x7f738009adf0 2024-04-17T10:25:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73800c9940 con 0x7f738009adf0 2024-04-17T10:25:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.927+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73800c9d20 con 0x7f738009adf0 2024-04-17T10:25:40.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.931+0000 7f7376a00640 1 -- 172.21.15.67:0/494947920 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f737001b070 con 0x7f738009adf0 2024-04-17T10:25:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.931+0000 7f7376a00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f735c066f60 0x7f735c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.931+0000 7f7376a00640 1 -- 172.21.15.67:0/494947920 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7370084a60 con 0x7f738009adf0 2024-04-17T10:25:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.931+0000 7f7375400640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f735c066f60 0x7f735c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.931+0000 7f7375400640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f735c066f60 0x7f735c069420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f73680af000 tx=0x7f73680a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:40.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:40.935+0000 7f7376a00640 1 -- 172.21.15.67:0/494947920 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f73700556e0 con 0x7f738009adf0 2024-04-17T10:25:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f738004ef30 con 0x7f738009adf0 2024-04-17T10:25:41.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7376a00640 1 -- 172.21.15.67:0/494947920 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7370054fe0 con 0x7f738009adf0 2024-04-17T10:25:41.097 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f735c066f60 msgr2=0x7f735c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f735c066f60 0x7f735c069420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f73680af000 tx=0x7f73680a7070 comp rx=0 tx=0).stop 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f738009adf0 msgr2=0x7f73800b73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f738009adf0 0x7f73800b73b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f737000c890 tx=0x7f737000cd50 comp rx=0 tx=0).stop 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 shutdown_connections 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f735c066f60 0x7f735c069420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7380161f90 0x7f73800b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f738015dd60 0x7f7380157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 --2- 172.21.15.67:0/494947920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f738009adf0 0x7f73800b73b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 >> 172.21.15.67:0/494947920 conn(0x7f73800b34a0 msgr2=0x7f73800cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 shutdown_connections 2024-04-17T10:25:41.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.103+0000 7f7384c00640 1 -- 172.21.15.67:0/494947920 wait complete. 2024-04-17T10:25:41.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47fe00640 1 Processor -- start 2024-04-17T10:25:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47fe00640 1 -- start start 2024-04-17T10:25:41.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe48006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe48006b1d0 0x7fe48015f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48015bf50 con 0x7fe48013e7a0 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48006b620 con 0x7fe48006b1d0 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48013f0d0 con 0x7fe48013ec90 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe48013e7a0 0x7fe480070710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37218/0 (socket says 172.21.15.67:37218) 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47ea00640 1 -- 172.21.15.67:0/3961350711 learned_addr learned my addr 172.21.15.67:0/3961350711 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47f400640 1 --2- 172.21.15.67:0/3961350711 >> v2:172.21.15.67:3300/0 conn(0x7fe48006b1d0 0x7fe48015f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe477e00640 1 --2- 172.21.15.67:0/3961350711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe48006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47e000640 1 -- 172.21.15.67:0/3961350711 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361840984 0 0) 0x7fe48015bf50 con 0x7fe48013e7a0 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47e000640 1 -- 172.21.15.67:0/3961350711 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe464003530 con 0x7fe48013e7a0 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47f400640 1 -- 172.21.15.67:0/3961350711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 msgr2=0x7fe48006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47f400640 1 --2- 172.21.15.67:0/3961350711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe48006ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47f400640 1 -- 172.21.15.67:0/3961350711 >> v1:172.21.15.67:6789/0 conn(0x7fe48013e7a0 legacy=0x7fe480070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe47f400640 1 -- 172.21.15.67:0/3961350711 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48015c230 con 0x7fe48006b1d0 2024-04-17T10:25:41.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.267+0000 7fe477e00640 1 --2- 172.21.15.67:0/3961350711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe48006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:41.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47f400640 1 --2- 172.21.15.67:0/3961350711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48006b1d0 0x7fe48015f470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe47801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acd9f62b8007b3c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47e000640 1 -- 172.21.15.67:0/3961350711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780b4070 con 0x7fe48006b1d0 2024-04-17T10:25:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47e000640 1 -- 172.21.15.67:0/3961350711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4780af600 con 0x7fe48006b1d0 2024-04-17T10:25:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47e000640 1 -- 172.21.15.67:0/3961350711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780af9d0 con 0x7fe48006b1d0 2024-04-17T10:25:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 -- 172.21.15.67:0/3961350711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48006b1d0 msgr2=0x7fe48015f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 --2- 172.21.15.67:0/3961350711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48006b1d0 0x7fe48015f470 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe47801f150 comp rx=0 tx=0).stop 2024-04-17T10:25:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 -- 172.21.15.67:0/3961350711 shutdown_connections 2024-04-17T10:25:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 --2- 172.21.15.67:0/3961350711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe48006b1d0 0x7fe48015f470 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 --2- 172.21.15.67:0/3961350711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe48006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 -- 172.21.15.67:0/3961350711 >> 172.21.15.67:0/3961350711 conn(0x7fe4800b2310 msgr2=0x7fe4800b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 -- 172.21.15.67:0/3961350711 shutdown_connections 2024-04-17T10:25:41.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.271+0000 7fe47fe00640 1 -- 172.21.15.67:0/3961350711 wait complete. 2024-04-17T10:25:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 Processor -- start 2024-04-17T10:25:41.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 -- start start 2024-04-17T10:25:41.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe48006b1d0 0x7fe48015aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe4800b14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe480158aa0 0x7fe4800b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48015cad0 con 0x7fe480158aa0 2024-04-17T10:25:41.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48015c7f0 con 0x7fe48006b1d0 2024-04-17T10:25:41.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48015bc40 con 0x7fe48013ec90 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe480158aa0 0x7fe4800b19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe480158aa0 0x7fe4800b19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56758/0 (socket says 172.21.15.67:56758) 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 -- 172.21.15.67:0/4249009263 learned_addr learned my addr 172.21.15.67:0/4249009263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47ea00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe48006b1d0 0x7fe48015aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe477e00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe4800b14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 -- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 msgr2=0x7fe4800b14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe4800b14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 -- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe48006b1d0 msgr2=0x7fe48015aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe48006b1d0 0x7fe48015aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 -- 172.21.15.67:0/4249009263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe480159010 con 0x7fe480158aa0 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47f400640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe480158aa0 0x7fe4800b19e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe47801bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe476a00640 1 -- 172.21.15.67:0/4249009263 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780b4040 con 0x7fe480158aa0 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe476a00640 1 -- 172.21.15.67:0/4249009263 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4780c1070 con 0x7fe480158aa0 2024-04-17T10:25:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe476a00640 1 -- 172.21.15.67:0/4249009263 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478009450 con 0x7fe480158aa0 2024-04-17T10:25:41.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4801592f0 con 0x7fe480158aa0 2024-04-17T10:25:41.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.275+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4801598b0 con 0x7fe480158aa0 2024-04-17T10:25:41.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.279+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48004f780 con 0x7fe480158aa0 2024-04-17T10:25:41.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.279+0000 7fe476a00640 1 -- 172.21.15.67:0/4249009263 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe4780ba6b0 con 0x7fe480158aa0 2024-04-17T10:25:41.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.279+0000 7fe476a00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe45c066f60 0x7fe45c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.279+0000 7fe476a00640 1 -- 172.21.15.67:0/4249009263 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fe478123cf0 con 0x7fe480158aa0 2024-04-17T10:25:41.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.279+0000 7fe47ea00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe45c066f60 0x7fe45c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.279+0000 7fe47ea00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe45c066f60 0x7fe45c069420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe470007fd0 tx=0x7fe470009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.283+0000 7fe476a00640 1 -- 172.21.15.67:0/4249009263 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe478018cb0 con 0x7fe480158aa0 2024-04-17T10:25:41.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe48006fb00 con 0x7fe480158aa0 2024-04-17T10:25:41.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe476a00640 1 -- 172.21.15.67:0/4249009263 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe4780f42a0 con 0x7fe480158aa0 2024-04-17T10:25:41.442 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe45c066f60 msgr2=0x7fe45c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe45c066f60 0x7fe45c069420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe470007fd0 tx=0x7fe470009040 comp rx=0 tx=0).stop 2024-04-17T10:25:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe480158aa0 msgr2=0x7fe4800b19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe480158aa0 0x7fe4800b19e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe478009d90 tx=0x7fe47801bef0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 shutdown_connections 2024-04-17T10:25:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe45c066f60 0x7fe45c069420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe480158aa0 0x7fe4800b19e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe48013ec90 0x7fe4800b14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 --2- 172.21.15.67:0/4249009263 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe48006b1d0 0x7fe48015aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 >> 172.21.15.67:0/4249009263 conn(0x7fe4800b2310 msgr2=0x7fe4800ccff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 shutdown_connections 2024-04-17T10:25:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.447+0000 7fe47fe00640 1 -- 172.21.15.67:0/4249009263 wait complete. 2024-04-17T10:25:41.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:41.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.611+0000 7f07d8800640 1 Processor -- start 2024-04-17T10:25:41.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.611+0000 7f07d8800640 1 -- start start 2024-04-17T10:25:41.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d8800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f07d00cd030 0x7f07d006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d8800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d00b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d8800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d00b4990 con 0x7f07d00aef60 2024-04-17T10:25:41.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d8800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d00af340 con 0x7f07d00cd030 2024-04-17T10:25:41.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d8800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d006ec00 con 0x7f07d006e7b0 2024-04-17T10:25:41.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d5400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f07d00cd030 0x7f07d006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d00b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d5e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f07d00aef60 0x7f07d00cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:34040/0 (socket says 172.21.15.67:34040) 2024-04-17T10:25:41.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d5e00640 1 -- 172.21.15.67:0/1391498923 learned_addr learned my addr 172.21.15.67:0/1391498923 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d5400640 1 -- 172.21.15.67:0/1391498923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 msgr2=0x7f07d00b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d5400640 1 --2- 172.21.15.67:0/1391498923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d00b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d5400640 1 -- 172.21.15.67:0/1391498923 >> v1:172.21.15.86:6789/0 conn(0x7f07d00aef60 legacy=0x7f07d00cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.615+0000 7f07d5400640 1 -- 172.21.15.67:0/1391498923 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d015ae90 con 0x7f07d00cd030 2024-04-17T10:25:41.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d5400640 1 --2- 172.21.15.67:0/1391498923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00cd030 0x7f07d006e270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f07c00097c0 tx=0x7f07c001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=420643834a5cd730 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d4a00640 1 -- 172.21.15.67:0/1391498923 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c00b0070 con 0x7f07d00cd030 2024-04-17T10:25:41.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d4a00640 1 -- 172.21.15.67:0/1391498923 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c00ab600 con 0x7f07d00cd030 2024-04-17T10:25:41.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d4a00640 1 -- 172.21.15.67:0/1391498923 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c00ab9d0 con 0x7f07d00cd030 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 -- 172.21.15.67:0/1391498923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00cd030 msgr2=0x7f07d006e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 --2- 172.21.15.67:0/1391498923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00cd030 0x7f07d006e270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f07c00097c0 tx=0x7f07c001c410 comp rx=0 tx=0).stop 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 -- 172.21.15.67:0/1391498923 shutdown_connections 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 --2- 172.21.15.67:0/1391498923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d00b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 --2- 172.21.15.67:0/1391498923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00cd030 0x7f07d006e270 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 -- 172.21.15.67:0/1391498923 >> 172.21.15.67:0/1391498923 conn(0x7f07d00b1740 msgr2=0x7f07d00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 -- 172.21.15.67:0/1391498923 shutdown_connections 2024-04-17T10:25:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 -- 172.21.15.67:0/1391498923 wait complete. 2024-04-17T10:25:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.619+0000 7f07d8800640 1 Processor -- start 2024-04-17T10:25:41.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 -- start start 2024-04-17T10:25:41.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d013a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00aef60 0x7f07d0149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d00cd030 0x7f07d014f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d015bc10 con 0x7f07d00cd030 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00b3e70 con 0x7f07d00aef60 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d01f7800 con 0x7f07d006e7b0 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d00cd030 0x7f07d014f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d00cd030 0x7f07d014f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56760/0 (socket says 172.21.15.67:56760) 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d6800640 1 -- 172.21.15.67:0/2677931081 learned_addr learned my addr 172.21.15.67:0/2677931081 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5400640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00aef60 0x7f07d0149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5e00640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d013a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5400640 1 -- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 msgr2=0x7f07d013a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5400640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d013a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5400640 1 -- 172.21.15.67:0/2677931081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d00cd030 msgr2=0x7f07d014f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5400640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d00cd030 0x7f07d014f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5400640 1 -- 172.21.15.67:0/2677931081 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d013a580 con 0x7f07d00aef60 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d6800640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d00cd030 0x7f07d014f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d5400640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00aef60 0x7f07d0149f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07d00723f0 tx=0x7f07c00191b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07ca000640 1 -- 172.21.15.67:0/2677931081 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c00b8020 con 0x7f07d00aef60 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07ca000640 1 -- 172.21.15.67:0/2677931081 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c00b4630 con 0x7f07d00aef60 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07ca000640 1 -- 172.21.15.67:0/2677931081 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c00b0040 con 0x7f07d00aef60 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d01f4750 con 0x7f07d00aef60 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d01f4d10 con 0x7f07d00aef60 2024-04-17T10:25:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.623+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d004ef30 con 0x7f07d00aef60 2024-04-17T10:25:41.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.627+0000 7f07ca000640 1 -- 172.21.15.67:0/2677931081 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f07c00bb020 con 0x7f07d00aef60 2024-04-17T10:25:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.627+0000 7f07ca000640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b4066f60 0x7f07b4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.627+0000 7f07ca000640 1 -- 172.21.15.67:0/2677931081 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f07c011f400 con 0x7f07d00aef60 2024-04-17T10:25:41.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.627+0000 7f07d5e00640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b4066f60 0x7f07b4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.627+0000 7f07d5e00640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b4066f60 0x7f07b4069420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07c4000f70 tx=0x7f07c4007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.631+0000 7f07ca000640 1 -- 172.21.15.67:0/2677931081 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f07c00c2030 con 0x7f07d00aef60 2024-04-17T10:25:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.791+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f07d006dca0 con 0x7f07d00aef60 2024-04-17T10:25:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.791+0000 7f07ca000640 1 -- 172.21.15.67:0/2677931081 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f07c0016090 con 0x7f07d00aef60 2024-04-17T10:25:41.787 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:41.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b4066f60 msgr2=0x7f07b4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b4066f60 0x7f07b4069420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07c4000f70 tx=0x7f07c4007e90 comp rx=0 tx=0).stop 2024-04-17T10:25:41.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00aef60 msgr2=0x7f07d0149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00aef60 0x7f07d0149f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07d00723f0 tx=0x7f07c00191b0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 shutdown_connections 2024-04-17T10:25:41.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07b4066f60 0x7f07b4069420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f07d00cd030 0x7f07d014f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f07d00aef60 0x7f07d0149f30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 --2- 172.21.15.67:0/2677931081 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f07d006e7b0 0x7f07d013a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 >> 172.21.15.67:0/2677931081 conn(0x7f07d00b1740 msgr2=0x7f07d00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:41.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 shutdown_connections 2024-04-17T10:25:41.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.795+0000 7f07d8800640 1 -- 172.21.15.67:0/2677931081 wait complete. 2024-04-17T10:25:41.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:41.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 Processor -- start 2024-04-17T10:25:41.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 -- start start 2024-04-17T10:25:41.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27e415de50 0x7f27e40cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e406f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27e40cf920 0x7f27e40cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e40cad20 con 0x7f27e415de50 2024-04-17T10:25:41.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e40700a0 con 0x7f27e40cf920 2024-04-17T10:25:41.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27ebe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e4070380 con 0x7f27e406b6a0 2024-04-17T10:25:41.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27e415de50 0x7f27e40cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27e415de50 0x7f27e40cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56766/0 (socket says 172.21.15.67:56766) 2024-04-17T10:25:41.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 -- 172.21.15.67:0/152110981 learned_addr learned my addr 172.21.15.67:0/152110981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9e00640 1 --2- 172.21.15.67:0/152110981 >> v2:172.21.15.86:3300/0 conn(0x7f27e40cf920 0x7f27e40cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e8a00640 1 --2- 172.21.15.67:0/152110981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e406f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 -- 172.21.15.67:0/152110981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 msgr2=0x7f27e406f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 --2- 172.21.15.67:0/152110981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e406f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 -- 172.21.15.67:0/152110981 >> v2:172.21.15.86:3300/0 conn(0x7f27e40cf920 msgr2=0x7f27e40cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 --2- 172.21.15.67:0/152110981 >> v2:172.21.15.86:3300/0 conn(0x7f27e40cf920 0x7f27e40cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9400640 1 -- 172.21.15.67:0/152110981 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40caf70 con 0x7f27e415de50 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e9e00640 1 --2- 172.21.15.67:0/152110981 >> v2:172.21.15.86:3300/0 conn(0x7f27e40cf920 0x7f27e40cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:41.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.955+0000 7f27e8a00640 1 --2- 172.21.15.67:0/152110981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e406f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:41.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27e9400640 1 --2- 172.21.15.67:0/152110981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40cc7d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f27e40b3340 tx=0x7f27d801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1475c8032f65f227 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27e3e00640 1 -- 172.21.15.67:0/152110981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d80b0070 con 0x7f27e415de50 2024-04-17T10:25:41.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27e3e00640 1 -- 172.21.15.67:0/152110981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d80ab600 con 0x7f27e415de50 2024-04-17T10:25:41.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27e3e00640 1 -- 172.21.15.67:0/152110981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d80ab9d0 con 0x7f27e415de50 2024-04-17T10:25:41.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 -- 172.21.15.67:0/152110981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 msgr2=0x7f27e40cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 --2- 172.21.15.67:0/152110981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40cc7d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f27e40b3340 tx=0x7f27d801c490 comp rx=0 tx=0).stop 2024-04-17T10:25:41.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 -- 172.21.15.67:0/152110981 shutdown_connections 2024-04-17T10:25:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 --2- 172.21.15.67:0/152110981 >> v2:172.21.15.86:3300/0 conn(0x7f27e40cf920 0x7f27e40cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 --2- 172.21.15.67:0/152110981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e406f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 --2- 172.21.15.67:0/152110981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40cc7d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 -- 172.21.15.67:0/152110981 >> 172.21.15.67:0/152110981 conn(0x7f27e40b22a0 msgr2=0x7f27e40b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 -- 172.21.15.67:0/152110981 shutdown_connections 2024-04-17T10:25:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.959+0000 7f27ebe00640 1 -- 172.21.15.67:0/152110981 wait complete. 2024-04-17T10:25:41.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 Processor -- start 2024-04-17T10:25:41.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 -- start start 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e41539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40cf920 0x7f27e415bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40cc310 con 0x7f27e415de50 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40cc190 con 0x7f27e40cf920 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e406e4a0 con 0x7f27e406b6a0 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56772/0 (socket says 172.21.15.67:56772) 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 -- 172.21.15.67:0/3094279756 learned_addr learned my addr 172.21.15.67:0/3094279756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e8a00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40cf920 0x7f27e415bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9400640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e41539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 -- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 msgr2=0x7f27e41539a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e41539a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 -- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40cf920 msgr2=0x7f27e415bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40cf920 0x7f27e415bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 -- 172.21.15.67:0/3094279756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40afbf0 con 0x7f27e415de50 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9400640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e41539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e8a00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40cf920 0x7f27e415bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e9e00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40af3a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27dc00cf20 tx=0x7f27dc00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e1600640 1 -- 172.21.15.67:0/3094279756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc01a070 con 0x7f27e415de50 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e1600640 1 -- 172.21.15.67:0/3094279756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc00d840 con 0x7f27e415de50 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27e1600640 1 -- 172.21.15.67:0/3094279756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc00dc30 con 0x7f27e415de50 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e40afed0 con 0x7f27e415de50 2024-04-17T10:25:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.963+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e415b3f0 con 0x7f27e415de50 2024-04-17T10:25:41.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.967+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e409e900 con 0x7f27e415de50 2024-04-17T10:25:41.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.967+0000 7f27e1600640 1 -- 172.21.15.67:0/3094279756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f27dc01d070 con 0x7f27e415de50 2024-04-17T10:25:41.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.967+0000 7f27e1600640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27c4066f60 0x7f27c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:41.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.967+0000 7f27e1600640 1 -- 172.21.15.67:0/3094279756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f27dc085d60 con 0x7f27e415de50 2024-04-17T10:25:41.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.967+0000 7f27e9400640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27c4066f60 0x7f27c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:41.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.967+0000 7f27e9400640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27c4066f60 0x7f27c4069420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f27d80ad000 tx=0x7f27d80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:41.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:41.971+0000 7f27e1600640 1 -- 172.21.15.67:0/3094279756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f27dc0568d0 con 0x7f27e415de50 2024-04-17T10:25:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.135+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27e4074e30 con 0x7f27e415de50 2024-04-17T10:25:42.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.135+0000 7f27e1600640 1 -- 172.21.15.67:0/3094279756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f27dc0561d0 con 0x7f27e415de50 2024-04-17T10:25:42.131 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:42.131 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27c4066f60 msgr2=0x7f27c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27c4066f60 0x7f27c4069420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f27d80ad000 tx=0x7f27d80a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 msgr2=0x7f27e40af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40af3a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f27dc00cf20 tx=0x7f27dc00a7d0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 shutdown_connections 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27c4066f60 0x7f27c4069420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27e415de50 0x7f27e40af3a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27e40cf920 0x7f27e415bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 --2- 172.21.15.67:0/3094279756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27e406b6a0 0x7f27e41539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 >> 172.21.15.67:0/3094279756 conn(0x7f27e40b22a0 msgr2=0x7f27e4157130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 shutdown_connections 2024-04-17T10:25:42.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.139+0000 7f27ebe00640 1 -- 172.21.15.67:0/3094279756 wait complete. 2024-04-17T10:25:42.152 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:42.153 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:42.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2009400640 1 Processor -- start 2024-04-17T10:25:42.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2009400640 1 -- start start 2024-04-17T10:25:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2009400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f200415a560 0x7f20040cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2009400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200415a9c0 con 0x7f20040722b0 2024-04-17T10:25:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2009400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200415ac40 con 0x7f2004071ed0 2024-04-17T10:25:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2009400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20040ccbf0 con 0x7f200415a560 2024-04-17T10:25:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2003200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f200415a560 0x7f20040cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f1ffbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20040722b0 0x7f200415e210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37234/0 (socket says 172.21.15.67:37234) 2024-04-17T10:25:42.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f1ffbe00640 1 -- 172.21.15.67:0/3837468411 learned_addr learned my addr 172.21.15.67:0/3837468411 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2003200640 1 -- 172.21.15.67:0/3837468411 >> v1:172.21.15.86:6789/0 conn(0x7f2004071ed0 legacy=0x7f20040aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2003200640 1 -- 172.21.15.67:0/3837468411 >> v1:172.21.15.67:6789/0 conn(0x7f20040722b0 legacy=0x7f200415e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.303+0000 7f2003200640 1 -- 172.21.15.67:0/3837468411 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f200415af20 con 0x7f200415a560 2024-04-17T10:25:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2003200640 1 --2- 172.21.15.67:0/3837468411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200415a560 0x7f20040cf020 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00aea0 tx=0x7f1ffc01d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac68688143aaf653 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2001e00640 1 -- 172.21.15.67:0/3837468411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0a6780 con 0x7f200415a560 2024-04-17T10:25:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2001e00640 1 -- 172.21.15.67:0/3837468411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ffc0a6920 con 0x7f200415a560 2024-04-17T10:25:42.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2001e00640 1 -- 172.21.15.67:0/3837468411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ffc0a6cf0 con 0x7f200415a560 2024-04-17T10:25:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 -- 172.21.15.67:0/3837468411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200415a560 msgr2=0x7f20040cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 --2- 172.21.15.67:0/3837468411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200415a560 0x7f20040cf020 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc00aea0 tx=0x7f1ffc01d900 comp rx=0 tx=0).stop 2024-04-17T10:25:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 -- 172.21.15.67:0/3837468411 shutdown_connections 2024-04-17T10:25:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 --2- 172.21.15.67:0/3837468411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f200415a560 0x7f20040cf020 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 -- 172.21.15.67:0/3837468411 >> 172.21.15.67:0/3837468411 conn(0x7f20040b1740 msgr2=0x7f20040b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 -- 172.21.15.67:0/3837468411 shutdown_connections 2024-04-17T10:25:42.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 -- 172.21.15.67:0/3837468411 wait complete. 2024-04-17T10:25:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 Processor -- start 2024-04-17T10:25:42.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.307+0000 7f2009400640 1 -- start start 2024-04-17T10:25:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2004071ed0 0x7f2004140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040722b0 0x7f20040b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200415a560 0x7f20040b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20040cf560 con 0x7f2004071ed0 2024-04-17T10:25:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200415dcf0 con 0x7f20040722b0 2024-04-17T10:25:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200415e920 con 0x7f200415a560 2024-04-17T10:25:42.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2004071ed0 0x7f2004140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2004071ed0 0x7f2004140410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56784/0 (socket says 172.21.15.67:56784) 2024-04-17T10:25:42.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 -- 172.21.15.67:0/181453581 learned_addr learned my addr 172.21.15.67:0/181453581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:42.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040722b0 0x7f20040b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2003200640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200415a560 0x7f20040b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 -- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200415a560 msgr2=0x7f20040b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200415a560 0x7f20040b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 -- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040722b0 msgr2=0x7f20040b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040722b0 0x7f20040b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 -- 172.21.15.67:0/181453581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20041524f0 con 0x7f2004071ed0 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2002800640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2004071ed0 0x7f2004140410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400c500 tx=0x7f1ff400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f1ffaa00640 1 -- 172.21.15.67:0/181453581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4016020 con 0x7f2004071ed0 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f1ffaa00640 1 -- 172.21.15.67:0/181453581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4005220 con 0x7f2004071ed0 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f1ffaa00640 1 -- 172.21.15.67:0/181453581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4007030 con 0x7f2004071ed0 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2004141770 con 0x7f2004071ed0 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2004141d30 con 0x7f2004071ed0 2024-04-17T10:25:42.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f200404ef30 con 0x7f2004071ed0 2024-04-17T10:25:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.311+0000 7f1ffaa00640 1 -- 172.21.15.67:0/181453581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1ff40071f0 con 0x7f2004071ed0 2024-04-17T10:25:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.315+0000 7f1ffaa00640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1fe0066f90 0x7f1fe0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.315+0000 7f1ffaa00640 1 -- 172.21.15.67:0/181453581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f1ff4084cd0 con 0x7f2004071ed0 2024-04-17T10:25:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.315+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1fe0066f90 0x7f1fe0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.315+0000 7f1ffbe00640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1fe0066f90 0x7f1fe0069450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f20041528e0 tx=0x7f1fec00aed0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.319+0000 7f1ffaa00640 1 -- 172.21.15.67:0/181453581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1ff408b080 con 0x7f2004071ed0 2024-04-17T10:25:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f200406ef80 con 0x7f2004071ed0 2024-04-17T10:25:42.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f1ffaa00640 1 -- 172.21.15.67:0/181453581 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ff4020020 con 0x7f2004071ed0 2024-04-17T10:25:42.474 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1fe0066f90 msgr2=0x7f1fe0069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1fe0066f90 0x7f1fe0069450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f20041528e0 tx=0x7f1fec00aed0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2004071ed0 msgr2=0x7f2004140410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2004071ed0 0x7f2004140410 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1ff400c500 tx=0x7f1ff400c9c0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 shutdown_connections 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1fe0066f90 0x7f1fe0069450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f200415a560 0x7f20040b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20040722b0 0x7f20040b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 --2- 172.21.15.67:0/181453581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2004071ed0 0x7f2004140410 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 >> 172.21.15.67:0/181453581 conn(0x7f20040b1740 msgr2=0x7f200415cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 shutdown_connections 2024-04-17T10:25:42.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.479+0000 7f2009400640 1 -- 172.21.15.67:0/181453581 wait complete. 2024-04-17T10:25:42.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.643+0000 7fb163600640 1 Processor -- start 2024-04-17T10:25:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.643+0000 7fb163600640 1 -- start start 2024-04-17T10:25:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb163600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb15c071af0 0x7fb15c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb163600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15c15cb30 con 0x7fb15c15be70 2024-04-17T10:25:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb163600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15c0cd610 con 0x7fb15c159440 2024-04-17T10:25:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb163600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15c0a8650 con 0x7fb15c071af0 2024-04-17T10:25:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb161600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb15c15be70 0x7fb15c15c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37246/0 (socket says 172.21.15.67:37246) 2024-04-17T10:25:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb161600640 1 -- 172.21.15.67:0/1786028155 learned_addr learned my addr 172.21.15.67:0/1786028155 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15be00640 1 --2- 172.21.15.67:0/1786028155 >> v2:172.21.15.86:3300/0 conn(0x7fb15c071af0 0x7fb15c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806869267 0 0) 0x7fb15c15cb30 con 0x7fb15c15be70 2024-04-17T10:25:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144003530 con 0x7fb15c15be70 2024-04-17T10:25:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539158455 0 0) 0x7fb144003530 con 0x7fb15c15be70 2024-04-17T10:25:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 >> v2:172.21.15.86:3300/0 conn(0x7fb15c071af0 msgr2=0x7fb15c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 --2- 172.21.15.67:0/1786028155 >> v2:172.21.15.86:3300/0 conn(0x7fb15c071af0 0x7fb15c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 >> v1:172.21.15.86:6789/0 conn(0x7fb15c159440 legacy=0x7fb15c0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c0cd8f0 con 0x7fb15c15be70 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fb154001a00 con 0x7fb15c15be70 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 >> v1:172.21.15.67:6789/0 conn(0x7fb15c15be70 legacy=0x7fb15c15c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1440039c0 0x7fb144003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb144004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb144005090 0x7fb14400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400bfc0 con 0x7fb1440043c0 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400c170 con 0x7fb144005090 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14400c370 con 0x7fb1440039c0 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15be00640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb144004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb161600640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb144005090 0x7fb14400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb160c00640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1440039c0 0x7fb144003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15be00640 1 -- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1440039c0 msgr2=0x7fb144003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15be00640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb1440039c0 0x7fb144003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15be00640 1 -- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb144005090 msgr2=0x7fb14400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15be00640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb144005090 0x7fb14400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb15be00640 1 -- 172.21.15.67:0/1786028155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb144003530 con 0x7fb1440043c0 2024-04-17T10:25:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.647+0000 7fb161600640 1 -- 172.21.15.67:0/1786028155 reap_dead start 2024-04-17T10:25:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb15be00640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb144004890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb148009ab0 tx=0x7fb14801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d049b2fbec9d23 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1480b2070 con 0x7fb1440043c0 2024-04-17T10:25:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1480a5640 con 0x7fb1440043c0 2024-04-17T10:25:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb15b400640 1 -- 172.21.15.67:0/1786028155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1480096b0 con 0x7fb1440043c0 2024-04-17T10:25:42.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 -- 172.21.15.67:0/1786028155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 msgr2=0x7fb144004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb144004890 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb148009ab0 tx=0x7fb14801ca50 comp rx=0 tx=0).stop 2024-04-17T10:25:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 -- 172.21.15.67:0/1786028155 shutdown_connections 2024-04-17T10:25:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 --2- 172.21.15.67:0/1786028155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb144004890 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 -- 172.21.15.67:0/1786028155 >> 172.21.15.67:0/1786028155 conn(0x7fb15c0b1740 msgr2=0x7fb15c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 -- 172.21.15.67:0/1786028155 shutdown_connections 2024-04-17T10:25:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 -- 172.21.15.67:0/1786028155 wait complete. 2024-04-17T10:25:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.651+0000 7fb163600640 1 Processor -- start 2024-04-17T10:25:42.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 -- start start 2024-04-17T10:25:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb15c15da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb15c1501f0 0x7fb15c1f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb15c1f6e90 0x7fb15c0b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c159820 con 0x7fb1440043c0 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c15ce10 con 0x7fb15c1501f0 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c0ce0f0 con 0x7fb15c1f6e90 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb15c15da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb15c15da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56800/0 (socket says 172.21.15.67:56800) 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 -- 172.21.15.67:0/2092816909 learned_addr learned my addr 172.21.15.67:0/2092816909 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb161600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb15c1f6e90 0x7fb15c0b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb15be00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb15c1501f0 0x7fb15c1f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 -- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb15c1f6e90 msgr2=0x7fb15c0b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb15c1f6e90 0x7fb15c0b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 -- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb15c1501f0 msgr2=0x7fb15c1f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb15c1501f0 0x7fb15c1f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 -- 172.21.15.67:0/2092816909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c15cf90 con 0x7fb1440043c0 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb161600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb15c1f6e90 0x7fb15c0b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb15be00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb15c1501f0 0x7fb15c1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb160c00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb15c15da30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb150011e30 tx=0x7fb15000f540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb158c00640 1 -- 172.21.15.67:0/2092816909 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150010040 con 0x7fb1440043c0 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb158c00640 1 -- 172.21.15.67:0/2092816909 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15001a070 con 0x7fb1440043c0 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb158c00640 1 -- 172.21.15.67:0/2092816909 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150015630 con 0x7fb1440043c0 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c1f7410 con 0x7fb1440043c0 2024-04-17T10:25:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15c15d230 con 0x7fb1440043c0 2024-04-17T10:25:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.655+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15c04ef30 con 0x7fb1440043c0 2024-04-17T10:25:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.659+0000 7fb158c00640 1 -- 172.21.15.67:0/2092816909 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb1500157d0 con 0x7fb1440043c0 2024-04-17T10:25:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.659+0000 7fb158c00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb13c066f60 0x7fb13c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.659+0000 7fb158c00640 1 -- 172.21.15.67:0/2092816909 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb150088c80 con 0x7fb1440043c0 2024-04-17T10:25:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.659+0000 7fb15be00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb13c066f60 0x7fb13c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.659+0000 7fb15be00640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb13c066f60 0x7fb13c069420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb1480af000 tx=0x7fb148008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.663+0000 7fb158c00640 1 -- 172.21.15.67:0/2092816909 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb1500597f0 con 0x7fb1440043c0 2024-04-17T10:25:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.823+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb15c06dfd0 con 0x7fb1440043c0 2024-04-17T10:25:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.823+0000 7fb158c00640 1 -- 172.21.15.67:0/2092816909 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb1500590f0 con 0x7fb1440043c0 2024-04-17T10:25:42.820 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb13c066f60 msgr2=0x7fb13c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb13c066f60 0x7fb13c069420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb1480af000 tx=0x7fb148008040 comp rx=0 tx=0).stop 2024-04-17T10:25:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 msgr2=0x7fb15c15da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb15c15da30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb150011e30 tx=0x7fb15000f540 comp rx=0 tx=0).stop 2024-04-17T10:25:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 shutdown_connections 2024-04-17T10:25:42.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb13c066f60 0x7fb13c069420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb15c1f6e90 0x7fb15c0b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb15c1501f0 0x7fb15c1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 --2- 172.21.15.67:0/2092816909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1440043c0 0x7fb15c15da30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 >> 172.21.15.67:0/2092816909 conn(0x7fb15c0b1740 msgr2=0x7fb15c06d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 shutdown_connections 2024-04-17T10:25:42.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.827+0000 7fb163600640 1 -- 172.21.15.67:0/2092816909 wait complete. 2024-04-17T10:25:42.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:42.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.987+0000 7f239f400640 1 Processor -- start 2024-04-17T10:25:42.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f239f400640 1 -- start start 2024-04-17T10:25:42.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f239f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23980cd030 0x7f239806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f239f400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f239806e7b0 0x7f23980b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f239f400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980b4990 con 0x7f23980aef60 2024-04-17T10:25:42.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f239f400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980af340 con 0x7f23980cd030 2024-04-17T10:25:42.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f239f400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239806ec00 con 0x7f239806e7b0 2024-04-17T10:25:42.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f238fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23980cd030 0x7f239806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f238fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23980cd030 0x7f239806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56810/0 (socket says 172.21.15.67:56810) 2024-04-17T10:25:42.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f238fe00640 1 -- 172.21.15.67:0/1130729425 learned_addr learned my addr 172.21.15.67:0/1130729425 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:42.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f239d400640 1 --2- 172.21.15.67:0/1130729425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f239806e7b0 0x7f23980b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f238fe00640 1 -- 172.21.15.67:0/1130729425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f239806e7b0 msgr2=0x7f23980b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:42.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f238fe00640 1 --2- 172.21.15.67:0/1130729425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f239806e7b0 0x7f23980b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f238fe00640 1 -- 172.21.15.67:0/1130729425 >> v1:172.21.15.86:6789/0 conn(0x7f23980aef60 legacy=0x7f23980cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:42.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.991+0000 7f238fe00640 1 -- 172.21.15.67:0/1130729425 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239815ae90 con 0x7f23980cd030 2024-04-17T10:25:42.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f238fe00640 1 --2- 172.21.15.67:0/1130729425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980cd030 0x7f239806e270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f238000bb10 tx=0x7f238001eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=987b51cb2e832f47 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f2397e00640 1 -- 172.21.15.67:0/1130729425 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800b4070 con 0x7f23980cd030 2024-04-17T10:25:42.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f2397e00640 1 -- 172.21.15.67:0/1130729425 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23800a7b60 con 0x7f23980cd030 2024-04-17T10:25:42.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f2397e00640 1 -- 172.21.15.67:0/1130729425 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800af440 con 0x7f23980cd030 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 -- 172.21.15.67:0/1130729425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980cd030 msgr2=0x7f239806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 --2- 172.21.15.67:0/1130729425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980cd030 0x7f239806e270 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f238000bb10 tx=0x7f238001eba0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 -- 172.21.15.67:0/1130729425 shutdown_connections 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 --2- 172.21.15.67:0/1130729425 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f239806e7b0 0x7f23980b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 --2- 172.21.15.67:0/1130729425 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980cd030 0x7f239806e270 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 -- 172.21.15.67:0/1130729425 >> 172.21.15.67:0/1130729425 conn(0x7f23980b1740 msgr2=0x7f23980b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 -- 172.21.15.67:0/1130729425 shutdown_connections 2024-04-17T10:25:42.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 -- 172.21.15.67:0/1130729425 wait complete. 2024-04-17T10:25:42.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.995+0000 7f239f400640 1 Processor -- start 2024-04-17T10:25:42.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 -- start start 2024-04-17T10:25:42.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f239806e7b0 0x7f2398144a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980aef60 0x7f23981f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23980cd030 0x7f23981f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239815bc10 con 0x7f23980aef60 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23980b3e70 con 0x7f239806e7b0 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2398145220 con 0x7f23980cd030 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980aef60 0x7f23981f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239d400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23980cd030 0x7f23981f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f239806e7b0 0x7f2398144a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980aef60 0x7f23981f5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56816/0 (socket says 172.21.15.67:56816) 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 -- 172.21.15.67:0/436567114 learned_addr learned my addr 172.21.15.67:0/436567114 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 -- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23980cd030 msgr2=0x7f23981f6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23980cd030 0x7f23981f6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 -- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f239806e7b0 msgr2=0x7f2398144a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f239806e7b0 0x7f2398144a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 -- 172.21.15.67:0/436567114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2398145500 con 0x7f23980aef60 2024-04-17T10:25:42.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f238fe00640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980aef60 0x7f23981f5be0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f238000b2c0 tx=0x7f238001b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f2395600640 1 -- 172.21.15.67:0/436567114 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800b4040 con 0x7f23980aef60 2024-04-17T10:25:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f2395600640 1 -- 172.21.15.67:0/436567114 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23800a7960 con 0x7f23980aef60 2024-04-17T10:25:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f2395600640 1 -- 172.21.15.67:0/436567114 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800ba6a0 con 0x7f23980aef60 2024-04-17T10:25:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23981457e0 con 0x7f23980aef60 2024-04-17T10:25:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2398145fe0 con 0x7f23980aef60 2024-04-17T10:25:42.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:42.999+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239804ef30 con 0x7f23980aef60 2024-04-17T10:25:42.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.003+0000 7f2395600640 1 -- 172.21.15.67:0/436567114 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f23800c1020 con 0x7f23980aef60 2024-04-17T10:25:42.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.003+0000 7f2395600640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2374066f60 0x7f2374069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:42.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.003+0000 7f2395600640 1 -- 172.21.15.67:0/436567114 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f2380122080 con 0x7f23980aef60 2024-04-17T10:25:42.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.003+0000 7f239ca00640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2374066f60 0x7f2374069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:42.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.003+0000 7f239ca00640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2374066f60 0x7f2374069420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2388007fd0 tx=0x7f2388009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:43.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.007+0000 7f2395600640 1 -- 172.21.15.67:0/436567114 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f23800c8020 con 0x7f23980aef60 2024-04-17T10:25:43.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.167+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2398072590 con 0x7f23980aef60 2024-04-17T10:25:43.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.167+0000 7f2395600640 1 -- 172.21.15.67:0/436567114 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2380018820 con 0x7f23980aef60 2024-04-17T10:25:43.163 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2374066f60 msgr2=0x7f2374069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2374066f60 0x7f2374069420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2388007fd0 tx=0x7f2388009040 comp rx=0 tx=0).stop 2024-04-17T10:25:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980aef60 msgr2=0x7f23981f5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980aef60 0x7f23981f5be0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f238000b2c0 tx=0x7f238001b940 comp rx=0 tx=0).stop 2024-04-17T10:25:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 shutdown_connections 2024-04-17T10:25:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2374066f60 0x7f2374069420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23980cd030 0x7f23981f6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23980aef60 0x7f23981f5be0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 --2- 172.21.15.67:0/436567114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f239806e7b0 0x7f2398144a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 >> 172.21.15.67:0/436567114 conn(0x7f23980b1740 msgr2=0x7f2398149db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 shutdown_connections 2024-04-17T10:25:43.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.171+0000 7f239f400640 1 -- 172.21.15.67:0/436567114 wait complete. 2024-04-17T10:25:43.183 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:43.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.331+0000 7f4582400640 1 Processor -- start 2024-04-17T10:25:43.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.331+0000 7f4582400640 1 -- start start 2024-04-17T10:25:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.331+0000 7f4582400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f457c0aef60 0x7f457c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.331+0000 7f4582400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c0b4990 con 0x7f457c06e980 2024-04-17T10:25:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.331+0000 7f4582400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457c15ae90 con 0x7f457c0cce60 2024-04-17T10:25:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f4582400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457c0af340 con 0x7f457c0aef60 2024-04-17T10:25:43.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f457c06e980 0x7f457c0b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37250/0 (socket says 172.21.15.67:37250) 2024-04-17T10:25:43.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457be00640 1 -- 172.21.15.67:0/3090599007 learned_addr learned my addr 172.21.15.67:0/3090599007 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:43.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457b400640 1 --2- 172.21.15.67:0/3090599007 >> v2:172.21.15.67:3300/0 conn(0x7f457c0aef60 0x7f457c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457aa00640 1 -- 172.21.15.67:0/3090599007 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960858393 0 0) 0x7f45740014a0 con 0x7f457c06e980 2024-04-17T10:25:43.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457b400640 1 -- 172.21.15.67:0/3090599007 >> v1:172.21.15.86:6789/0 conn(0x7f457c0cce60 legacy=0x7f457c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457b400640 1 -- 172.21.15.67:0/3090599007 >> v1:172.21.15.67:6789/0 conn(0x7f457c06e980 legacy=0x7f457c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457b400640 1 -- 172.21.15.67:0/3090599007 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c06ee80 con 0x7f457c0aef60 2024-04-17T10:25:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457b400640 1 --2- 172.21.15.67:0/3090599007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0aef60 0x7f457c0cc920 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f45640097c0 tx=0x7f456401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbf594e3c0312805 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457aa00640 1 -- 172.21.15.67:0/3090599007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45640b0070 con 0x7f457c0aef60 2024-04-17T10:25:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.335+0000 7f457aa00640 1 -- 172.21.15.67:0/3090599007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45640ab600 con 0x7f457c0aef60 2024-04-17T10:25:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457aa00640 1 -- 172.21.15.67:0/3090599007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45640ab9d0 con 0x7f457c0aef60 2024-04-17T10:25:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- 172.21.15.67:0/3090599007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0aef60 msgr2=0x7f457c0cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 --2- 172.21.15.67:0/3090599007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0aef60 0x7f457c0cc920 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f45640097c0 tx=0x7f456401c410 comp rx=0 tx=0).stop 2024-04-17T10:25:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- 172.21.15.67:0/3090599007 shutdown_connections 2024-04-17T10:25:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 --2- 172.21.15.67:0/3090599007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0aef60 0x7f457c0cc920 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- 172.21.15.67:0/3090599007 >> 172.21.15.67:0/3090599007 conn(0x7f457c0b1740 msgr2=0x7f457c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- 172.21.15.67:0/3090599007 shutdown_connections 2024-04-17T10:25:43.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- 172.21.15.67:0/3090599007 wait complete. 2024-04-17T10:25:43.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 Processor -- start 2024-04-17T10:25:43.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- start start 2024-04-17T10:25:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f457c06e980 0x7f457c146490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f457c0aef60 0x7f457c146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0cce60 0x7f457c147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c168550 con 0x7f457c0cce60 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457c1683d0 con 0x7f457c0aef60 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4582400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457c1686d0 con 0x7f457c06e980 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0cce60 0x7f457c147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0cce60 0x7f457c147170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56828/0 (socket says 172.21.15.67:56828) 2024-04-17T10:25:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 -- 172.21.15.67:0/1105903825 learned_addr learned my addr 172.21.15.67:0/1105903825 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:43.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457b400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f457c06e980 0x7f457c146490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f4572a00640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f457c0aef60 0x7f457c146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 -- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f457c06e980 msgr2=0x7f457c146490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f457c06e980 0x7f457c146490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 -- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f457c0aef60 msgr2=0x7f457c146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f457c0aef60 0x7f457c146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457be00640 1 -- 172.21.15.67:0/1105903825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c1586a0 con 0x7f457c0cce60 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.339+0000 7f457b400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f457c06e980 0x7f457c146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f457be00640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0cce60 0x7f457c147170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f457400c700 tx=0x7f457400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4573e00640 1 -- 172.21.15.67:0/1105903825 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574018070 con 0x7f457c0cce60 2024-04-17T10:25:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4573e00640 1 -- 172.21.15.67:0/1105903825 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457400ce10 con 0x7f457c0cce60 2024-04-17T10:25:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457c148430 con 0x7f457c0cce60 2024-04-17T10:25:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4573e00640 1 -- 172.21.15.67:0/1105903825 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400a880 con 0x7f457c0cce60 2024-04-17T10:25:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457c147a60 con 0x7f457c0cce60 2024-04-17T10:25:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457c04ef30 con 0x7f457c0cce60 2024-04-17T10:25:43.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4573e00640 1 -- 172.21.15.67:0/1105903825 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f457401b070 con 0x7f457c0cce60 2024-04-17T10:25:43.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4573e00640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4558066f60 0x7f4558069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f4573e00640 1 -- 172.21.15.67:0/1105903825 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f4574084a00 con 0x7f457c0cce60 2024-04-17T10:25:43.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.343+0000 7f457b400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4558066f60 0x7f4558069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.347+0000 7f457b400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4558066f60 0x7f4558069420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45640ad000 tx=0x7f45640a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.347+0000 7f4573e00640 1 -- 172.21.15.67:0/1105903825 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f457c1586a0 con 0x7f457c0cce60 2024-04-17T10:25:43.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.507+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f457c14eef0 con 0x7f457c0cce60 2024-04-17T10:25:43.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4573e00640 1 -- 172.21.15.67:0/1105903825 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f457c0b4990 con 0x7f457c0cce60 2024-04-17T10:25:43.505 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:43.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4558066f60 msgr2=0x7f4558069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4558066f60 0x7f4558069420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f45640ad000 tx=0x7f45640a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0cce60 msgr2=0x7f457c147170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0cce60 0x7f457c147170 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f457400c700 tx=0x7f457400cbc0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 shutdown_connections 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4558066f60 0x7f4558069420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f457c0cce60 0x7f457c147170 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f457c0aef60 0x7f457c146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 --2- 172.21.15.67:0/1105903825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f457c06e980 0x7f457c146490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 >> 172.21.15.67:0/1105903825 conn(0x7f457c0b1740 msgr2=0x7f457c0af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 shutdown_connections 2024-04-17T10:25:43.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.511+0000 7f4582400640 1 -- 172.21.15.67:0/1105903825 wait complete. 2024-04-17T10:25:43.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb6200640 1 Processor -- start 2024-04-17T10:25:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb6200640 1 -- start start 2024-04-17T10:25:43.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb6200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3bb0161160 0x7f3bb0099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb6200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb00d0760 con 0x7f3bb0099f90 2024-04-17T10:25:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb6200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb01624e0 con 0x7f3bb00b2570 2024-04-17T10:25:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb6200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb00cfc70 con 0x7f3bb0161160 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb5800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3bb0099f90 0x7f3bb0161dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37252/0 (socket says 172.21.15.67:37252) 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb5800640 1 -- 172.21.15.67:0/3863634379 learned_addr learned my addr 172.21.15.67:0/3863634379 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb4e00640 1 --2- 172.21.15.67:0/3863634379 >> v2:172.21.15.86:3300/0 conn(0x7f3bb0161160 0x7f3bb0099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3baf400640 1 -- 172.21.15.67:0/3863634379 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446130888 0 0) 0x7f3bb00d0760 con 0x7f3bb0099f90 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3baf400640 1 -- 172.21.15.67:0/3863634379 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b98003530 con 0x7f3bb0099f90 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb4e00640 1 -- 172.21.15.67:0/3863634379 >> v1:172.21.15.86:6789/0 conn(0x7f3bb00b2570 legacy=0x7f3bb01616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb4e00640 1 -- 172.21.15.67:0/3863634379 >> v1:172.21.15.67:6789/0 conn(0x7f3bb0099f90 legacy=0x7f3bb0161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3baf400640 1 -- 172.21.15.67:0/3863634379 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866091227 0 0) 0x7f3b98003530 con 0x7f3bb0099f90 2024-04-17T10:25:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.675+0000 7f3bb4e00640 1 -- 172.21.15.67:0/3863634379 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb00cc2a0 con 0x7f3bb0161160 2024-04-17T10:25:43.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb4e00640 1 --2- 172.21.15.67:0/3863634379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0161160 0x7f3bb0099a50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40097c0 tx=0x7f3ba401c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b6b9e6b5f6064b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:43.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3baf400640 1 -- 172.21.15.67:0/3863634379 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40b0070 con 0x7f3bb0161160 2024-04-17T10:25:43.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3baf400640 1 -- 172.21.15.67:0/3863634379 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba40ab400 con 0x7f3bb0161160 2024-04-17T10:25:43.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3baf400640 1 -- 172.21.15.67:0/3863634379 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8001a00 con 0x7f3bb0161160 2024-04-17T10:25:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 -- 172.21.15.67:0/3863634379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0161160 msgr2=0x7f3bb0099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 --2- 172.21.15.67:0/3863634379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0161160 0x7f3bb0099a50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40097c0 tx=0x7f3ba401c3c0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 -- 172.21.15.67:0/3863634379 shutdown_connections 2024-04-17T10:25:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 --2- 172.21.15.67:0/3863634379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0161160 0x7f3bb0099a50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 -- 172.21.15.67:0/3863634379 >> 172.21.15.67:0/3863634379 conn(0x7f3bb00d1690 msgr2=0x7f3bb00d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 -- 172.21.15.67:0/3863634379 shutdown_connections 2024-04-17T10:25:43.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 -- 172.21.15.67:0/3863634379 wait complete. 2024-04-17T10:25:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.679+0000 7f3bb6200640 1 Processor -- start 2024-04-17T10:25:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 -- start start 2024-04-17T10:25:43.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0099f90 0x7f3bb00ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb00b2570 0x7f3bb00b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bb0161160 0x7f3bb014bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb009a3c0 con 0x7f3bb00b2570 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb00cc5d0 con 0x7f3bb0099f90 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb00b2bc0 con 0x7f3bb0161160 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb00b2570 0x7f3bb00b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb00b2570 0x7f3bb00b6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56838/0 (socket says 172.21.15.67:56838) 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 -- 172.21.15.67:0/594619916 learned_addr learned my addr 172.21.15.67:0/594619916 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb4e00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0099f90 0x7f3bb00ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb5800640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bb0161160 0x7f3bb014bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 -- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bb0161160 msgr2=0x7f3bb014bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bb0161160 0x7f3bb014bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 -- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0099f90 msgr2=0x7f3bb00ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0099f90 0x7f3bb00ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 -- 172.21.15.67:0/594619916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb00c5360 con 0x7f3bb00b2570 2024-04-17T10:25:43.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb4e00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0099f90 0x7f3bb00ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bafe00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb00b2570 0x7f3bb00b6e60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c3c0 tx=0x7f3b9c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bacc00640 1 -- 172.21.15.67:0/594619916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c017070 con 0x7f3bb00b2570 2024-04-17T10:25:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bacc00640 1 -- 172.21.15.67:0/594619916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c0052d0 con 0x7f3bb00b2570 2024-04-17T10:25:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bacc00640 1 -- 172.21.15.67:0/594619916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c003840 con 0x7f3bb00b2570 2024-04-17T10:25:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb00ca8a0 con 0x7f3bb00b2570 2024-04-17T10:25:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb014c090 con 0x7f3bb00b2570 2024-04-17T10:25:43.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.683+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb0074dc0 con 0x7f3bb00b2570 2024-04-17T10:25:43.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.687+0000 7f3bacc00640 1 -- 172.21.15.67:0/594619916 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3b9c01a070 con 0x7f3bb00b2570 2024-04-17T10:25:43.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.687+0000 7f3bacc00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b90066f90 0x7f3b90069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:43.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.687+0000 7f3bacc00640 1 -- 172.21.15.67:0/594619916 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f3b9c0847e0 con 0x7f3bb00b2570 2024-04-17T10:25:43.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.687+0000 7f3bb4e00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b90066f90 0x7f3b90069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:43.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.687+0000 7f3bb4e00640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b90066f90 0x7f3b90069450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40ad000 tx=0x7f3ba40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.691+0000 7f3bacc00640 1 -- 172.21.15.67:0/594619916 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3b9c005480 con 0x7f3bb00b2570 2024-04-17T10:25:43.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.851+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bb00cadc0 con 0x7f3bb00b2570 2024-04-17T10:25:43.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bacc00640 1 -- 172.21.15.67:0/594619916 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f3b9c055350 con 0x7f3bb00b2570 2024-04-17T10:25:43.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:43.849 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:43.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b90066f90 msgr2=0x7f3b90069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b90066f90 0x7f3b90069450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40ad000 tx=0x7f3ba40a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb00b2570 msgr2=0x7f3bb00b6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb00b2570 0x7f3bb00b6e60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c3c0 tx=0x7f3b9c00c880 comp rx=0 tx=0).stop 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 shutdown_connections 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b90066f90 0x7f3b90069450 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3bb0161160 0x7f3bb014bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3bb00b2570 0x7f3bb00b6e60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 --2- 172.21.15.67:0/594619916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3bb0099f90 0x7f3bb00ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 >> 172.21.15.67:0/594619916 conn(0x7f3bb00d1690 msgr2=0x7f3bb0070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 shutdown_connections 2024-04-17T10:25:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:43.855+0000 7f3bb6200640 1 -- 172.21.15.67:0/594619916 wait complete. 2024-04-17T10:25:43.871 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:43.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:44.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.019+0000 7f2114c00640 1 Processor -- start 2024-04-17T10:25:44.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.019+0000 7f2114c00640 1 -- start start 2024-04-17T10:25:44.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.019+0000 7f2114c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 0x7f21100ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.019+0000 7f2114c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21100713e0 0x7f2110071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.019+0000 7f2114c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211015cb30 con 0x7f211015be70 2024-04-17T10:25:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f2114c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21100a8650 con 0x7f21100713e0 2024-04-17T10:25:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f2114c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2110071d80 con 0x7f2110159440 2024-04-17T10:25:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f211015be70 0x7f211015c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37266/0 (socket says 172.21.15.67:37266) 2024-04-17T10:25:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210e800640 1 -- 172.21.15.67:0/3675381154 learned_addr learned my addr 172.21.15.67:0/3675381154 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210de00640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 0x7f21100ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f2105400640 1 --2- 172.21.15.67:0/3675381154 >> v2:172.21.15.86:3300/0 conn(0x7f21100713e0 0x7f2110071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824524879 0 0) 0x7f211015cb30 con 0x7f211015be70 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20f4003530 con 0x7f211015be70 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129223522 0 0) 0x7f20f4003530 con 0x7f211015be70 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 msgr2=0x7f21100ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 0x7f21100ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 >> v2:172.21.15.86:3300/0 conn(0x7f21100713e0 msgr2=0x7f2110071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 --2- 172.21.15.67:0/3675381154 >> v2:172.21.15.86:3300/0 conn(0x7f21100713e0 0x7f2110071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21100cd610 con 0x7f211015be70 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f2108001a00 con 0x7f211015be70 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 >> v1:172.21.15.67:6789/0 conn(0x7f211015be70 legacy=0x7f211015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f40039c0 0x7f20f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4004810 0x7f20f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 0x7f20f40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f400b9d0 con 0x7f20f40039c0 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20f400bb50 con 0x7f20f4004810 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20f400bcd0 con 0x7f2110159440 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f2105400640 1 --2- 172.21.15.67:0/3675381154 >> v2:172.21.15.86:3300/0 conn(0x7f21100713e0 0x7f2110071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210de00640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f40039c0 0x7f20f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210e800640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 0x7f20f40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f2105400640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4004810 0x7f20f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210de00640 1 -- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 msgr2=0x7f20f40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:44.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210de00640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2110159440 0x7f20f40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210de00640 1 -- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4004810 msgr2=0x7f20f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:44.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210de00640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f20f4004810 0x7f20f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.023+0000 7f210de00640 1 -- 172.21.15.67:0/3675381154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20f4003530 con 0x7f20f40039c0 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f210de00640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f40039c0 0x7f20f4003de0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2100009ac0 tx=0x7f210001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3bc5848b5c3f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2100009690 con 0x7f20f40039c0 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21000a6460 con 0x7f20f40039c0 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f210d400640 1 -- 172.21.15.67:0/3675381154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21000a6830 con 0x7f20f40039c0 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 -- 172.21.15.67:0/3675381154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f40039c0 msgr2=0x7f20f4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 --2- 172.21.15.67:0/3675381154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20f40039c0 0x7f20f4003de0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2100009ac0 tx=0x7f210001ca50 comp rx=0 tx=0).stop 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f210e800640 1 -- 172.21.15.67:0/3675381154 reap_dead start 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 -- 172.21.15.67:0/3675381154 shutdown_connections 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 -- 172.21.15.67:0/3675381154 >> 172.21.15.67:0/3675381154 conn(0x7f21100b1740 msgr2=0x7f21100b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 -- 172.21.15.67:0/3675381154 shutdown_connections 2024-04-17T10:25:44.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 -- 172.21.15.67:0/3675381154 wait complete. 2024-04-17T10:25:44.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 Processor -- start 2024-04-17T10:25:44.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.027+0000 7f2114c00640 1 -- start start 2024-04-17T10:25:44.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f40039c0 0x7f211014e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21100ae320 0x7f21100cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21100cc790 0x7f2110155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211015d840 con 0x7f21100ae320 2024-04-17T10:25:44.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211015ce10 con 0x7f21100cc790 2024-04-17T10:25:44.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211015d9c0 con 0x7f20f40039c0 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21100ae320 0x7f21100cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21100ae320 0x7f21100cc250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56852/0 (socket says 172.21.15.67:56852) 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 -- 172.21.15.67:0/1698460012 learned_addr learned my addr 172.21.15.67:0/1698460012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f210de00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f40039c0 0x7f211014e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f210e800640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21100cc790 0x7f2110155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 -- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f40039c0 msgr2=0x7f211014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f40039c0 0x7f211014e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 -- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21100cc790 msgr2=0x7f2110155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21100cc790 0x7f2110155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 -- 172.21.15.67:0/1698460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2110157080 con 0x7f21100ae320 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2105400640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21100ae320 0x7f21100cc250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f20f8011960 tx=0x7f20f8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2106a00640 1 -- 172.21.15.67:0/1698460012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f800fbd0 con 0x7f21100ae320 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2106a00640 1 -- 172.21.15.67:0/1698460012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f800fd70 con 0x7f21100ae320 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2106a00640 1 -- 172.21.15.67:0/1698460012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f8015660 con 0x7f21100ae320 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2110157360 con 0x7f21100ae320 2024-04-17T10:25:44.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2110157920 con 0x7f21100ae320 2024-04-17T10:25:44.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211004ef30 con 0x7f21100ae320 2024-04-17T10:25:44.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.031+0000 7f2106a00640 1 -- 172.21.15.67:0/1698460012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f20f8015800 con 0x7f21100ae320 2024-04-17T10:25:44.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.035+0000 7f2106a00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20ec066f60 0x7f20ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.035+0000 7f2106a00640 1 -- 172.21.15.67:0/1698460012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f20f8086060 con 0x7f21100ae320 2024-04-17T10:25:44.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.035+0000 7f210de00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20ec066f60 0x7f20ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.035+0000 7f210de00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20ec066f60 0x7f20ec069420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21000ad000 tx=0x7f21000b2040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.039+0000 7f2106a00640 1 -- 172.21.15.67:0/1698460012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f20f8019030 con 0x7f21100ae320 2024-04-17T10:25:44.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.199+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f21100b11c0 con 0x7f21100ae320 2024-04-17T10:25:44.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.199+0000 7f2106a00640 1 -- 172.21.15.67:0/1698460012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f20f8056b40 con 0x7f21100ae320 2024-04-17T10:25:44.195 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20ec066f60 msgr2=0x7f20ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20ec066f60 0x7f20ec069420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f21000ad000 tx=0x7f21000b2040 comp rx=0 tx=0).stop 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21100ae320 msgr2=0x7f21100cc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21100ae320 0x7f21100cc250 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f20f8011960 tx=0x7f20f8011e20 comp rx=0 tx=0).stop 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 shutdown_connections 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20ec066f60 0x7f20ec069420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21100cc790 0x7f2110155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21100ae320 0x7f21100cc250 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 --2- 172.21.15.67:0/1698460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f20f40039c0 0x7f211014e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 >> 172.21.15.67:0/1698460012 conn(0x7f21100b1740 msgr2=0x7f211006b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:44.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 shutdown_connections 2024-04-17T10:25:44.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.203+0000 7f2114c00640 1 -- 172.21.15.67:0/1698460012 wait complete. 2024-04-17T10:25:44.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:44.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.363+0000 7f890c600640 1 Processor -- start 2024-04-17T10:25:44.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.363+0000 7f890c600640 1 -- start start 2024-04-17T10:25:44.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890c600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f890406e980 0x7f89041653b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890c600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89040b4990 con 0x7f89040cd030 2024-04-17T10:25:44.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890c600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890415ae90 con 0x7f89040aef60 2024-04-17T10:25:44.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890c600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89040af340 con 0x7f890406e980 2024-04-17T10:25:44.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f8909200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f89040cd030 0x7f890406e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37276/0 (socket says 172.21.15.67:37276) 2024-04-17T10:25:44.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f8909200640 1 -- 172.21.15.67:0/210074212 learned_addr learned my addr 172.21.15.67:0/210074212 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:44.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890a600640 1 --2- 172.21.15.67:0/210074212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f890406e980 0x7f89041653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849882911 0 0) 0x7f88f0001120 con 0x7f89040cd030 2024-04-17T10:25:44.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88ec003530 con 0x7f89040cd030 2024-04-17T10:25:44.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842252369 0 0) 0x7f890415ae90 con 0x7f89040aef60 2024-04-17T10:25:44.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88f0001120 con 0x7f89040aef60 2024-04-17T10:25:44.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890a600640 1 -- 172.21.15.67:0/210074212 >> v1:172.21.15.86:6789/0 conn(0x7f89040aef60 legacy=0x7f89040cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890a600640 1 -- 172.21.15.67:0/210074212 >> v1:172.21.15.67:6789/0 conn(0x7f89040cd030 legacy=0x7f890406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 752391722 0 0) 0x7f88ec003530 con 0x7f89040cd030 2024-04-17T10:25:44.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.367+0000 7f890a600640 1 -- 172.21.15.67:0/210074212 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890406ee80 con 0x7f890406e980 2024-04-17T10:25:44.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890a600640 1 --2- 172.21.15.67:0/210074212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f890406e980 0x7f89041653b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009dc0 tx=0x7f88fc01c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2feb5b4449bd23c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc0ac070 con 0x7f890406e980 2024-04-17T10:25:44.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc0a6470 con 0x7f890406e980 2024-04-17T10:25:44.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f8903e00640 1 -- 172.21.15.67:0/210074212 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0001870 con 0x7f890406e980 2024-04-17T10:25:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 -- 172.21.15.67:0/210074212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f890406e980 msgr2=0x7f89041653b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 --2- 172.21.15.67:0/210074212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f890406e980 0x7f89041653b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f88fc009dc0 tx=0x7f88fc01c820 comp rx=0 tx=0).stop 2024-04-17T10:25:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 -- 172.21.15.67:0/210074212 shutdown_connections 2024-04-17T10:25:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 --2- 172.21.15.67:0/210074212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f890406e980 0x7f89041653b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 -- 172.21.15.67:0/210074212 >> 172.21.15.67:0/210074212 conn(0x7f89040b1740 msgr2=0x7f89040b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 -- 172.21.15.67:0/210074212 shutdown_connections 2024-04-17T10:25:44.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 -- 172.21.15.67:0/210074212 wait complete. 2024-04-17T10:25:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 Processor -- start 2024-04-17T10:25:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.371+0000 7f890c600640 1 -- start start 2024-04-17T10:25:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f890406e980 0x7f8904155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89040aef60 0x7f89041562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89040cd030 0x7f8904148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89041686a0 con 0x7f89040aef60 2024-04-17T10:25:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8904168520 con 0x7f890406e980 2024-04-17T10:25:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8904168820 con 0x7f89040cd030 2024-04-17T10:25:44.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89040aef60 0x7f89041562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89040aef60 0x7f89041562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56858/0 (socket says 172.21.15.67:56858) 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 -- 172.21.15.67:0/1905939828 learned_addr learned my addr 172.21.15.67:0/1905939828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909c00640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f890406e980 0x7f8904155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890a600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89040cd030 0x7f8904148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 -- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89040cd030 msgr2=0x7f8904148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89040cd030 0x7f8904148ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 -- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f890406e980 msgr2=0x7f8904155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f890406e980 0x7f8904155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 -- 172.21.15.67:0/1905939828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8904158a50 con 0x7f89040aef60 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8909200640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89040aef60 0x7f89041562d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88f000caf0 tx=0x7f88f0007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8901600640 1 -- 172.21.15.67:0/1905939828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0017070 con 0x7f89040aef60 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8901600640 1 -- 172.21.15.67:0/1905939828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0005cf0 con 0x7f89040aef60 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890406e370 con 0x7f89040aef60 2024-04-17T10:25:44.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8901600640 1 -- 172.21.15.67:0/1905939828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f00072b0 con 0x7f89040aef60 2024-04-17T10:25:44.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8904158cd0 con 0x7f89040aef60 2024-04-17T10:25:44.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890404ef30 con 0x7f89040aef60 2024-04-17T10:25:44.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.375+0000 7f8901600640 1 -- 172.21.15.67:0/1905939828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f88f001a070 con 0x7f89040aef60 2024-04-17T10:25:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.379+0000 7f8901600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88e4066f60 0x7f88e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.379+0000 7f8901600640 1 -- 172.21.15.67:0/1905939828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f88f00849e0 con 0x7f89040aef60 2024-04-17T10:25:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.379+0000 7f8909c00640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88e4066f60 0x7f88e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.379+0000 7f8909c00640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88e4066f60 0x7f88e4069420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f88f80011f0 tx=0x7f88f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.379+0000 7f8901600640 1 -- 172.21.15.67:0/1905939828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8904158a50 con 0x7f89040aef60 2024-04-17T10:25:44.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.539+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8904158f10 con 0x7f89040aef60 2024-04-17T10:25:44.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f8901600640 1 -- 172.21.15.67:0/1905939828 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89040b4990 con 0x7f89040aef60 2024-04-17T10:25:44.537 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88e4066f60 msgr2=0x7f88e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88e4066f60 0x7f88e4069420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f88f80011f0 tx=0x7f88f8009040 comp rx=0 tx=0).stop 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89040aef60 msgr2=0x7f89041562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89040aef60 0x7f89041562d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f88f000caf0 tx=0x7f88f0007e60 comp rx=0 tx=0).stop 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 shutdown_connections 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88e4066f60 0x7f88e4069420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89040cd030 0x7f8904148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89040aef60 0x7f89041562d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 --2- 172.21.15.67:0/1905939828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f890406e980 0x7f8904155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 >> 172.21.15.67:0/1905939828 conn(0x7f89040b1740 msgr2=0x7f890414ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 shutdown_connections 2024-04-17T10:25:44.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.543+0000 7f890c600640 1 -- 172.21.15.67:0/1905939828 wait complete. 2024-04-17T10:25:44.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:44.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb6e800640 1 Processor -- start 2024-04-17T10:25:44.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb6e800640 1 -- start start 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb6e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb68071af0 0x7fbb680ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb6e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb6815be70 0x7fbb6815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb6e800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6815cb30 con 0x7fbb68159440 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb6e800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb680a8650 con 0x7fbb68071af0 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb6e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6815ce10 con 0x7fbb6815be70 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb5ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb68071af0 0x7fbb680ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb5ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb68071af0 0x7fbb680ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56864/0 (socket says 172.21.15.67:56864) 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1535857719 learned_addr learned my addr 172.21.15.67:0/1535857719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb67e00640 1 --2- 172.21.15.67:0/1535857719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb6815be70 0x7fbb6815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1535857719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb6815be70 msgr2=0x7fbb6815c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb5ea00640 1 --2- 172.21.15.67:0/1535857719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb6815be70 0x7fbb6815c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1535857719 >> v1:172.21.15.86:6789/0 conn(0x7fbb68159440 legacy=0x7fbb680713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:44.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1535857719 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb680cd610 con 0x7fbb68071af0 2024-04-17T10:25:44.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.707+0000 7fbb67e00640 1 --2- 172.21.15.67:0/1535857719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb6815be70 0x7fbb6815c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb5ea00640 1 --2- 172.21.15.67:0/1535857719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68071af0 0x7fbb680ce0f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009870 tx=0x7fbb5001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598da11d5c640d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb66a00640 1 -- 172.21.15.67:0/1535857719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb500a9620 con 0x7fbb68071af0 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb66a00640 1 -- 172.21.15.67:0/1535857719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb500a97c0 con 0x7fbb68071af0 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb66a00640 1 -- 172.21.15.67:0/1535857719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb500a9b90 con 0x7fbb68071af0 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 -- 172.21.15.67:0/1535857719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68071af0 msgr2=0x7fbb680ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1535857719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68071af0 0x7fbb680ce0f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbb50009870 tx=0x7fbb5001c560 comp rx=0 tx=0).stop 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 -- 172.21.15.67:0/1535857719 shutdown_connections 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1535857719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb6815be70 0x7fbb6815c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1535857719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68071af0 0x7fbb680ce0f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 -- 172.21.15.67:0/1535857719 >> 172.21.15.67:0/1535857719 conn(0x7fbb680b1740 msgr2=0x7fbb680b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 -- 172.21.15.67:0/1535857719 shutdown_connections 2024-04-17T10:25:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.711+0000 7fbb6e800640 1 -- 172.21.15.67:0/1535857719 wait complete. 2024-04-17T10:25:44.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 Processor -- start 2024-04-17T10:25:44.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 -- start start 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb68071af0 0x7fbb6813dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68159440 0x7fbb68150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb6815be70 0x7fbb68151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6815d840 con 0x7fbb68159440 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6815c800 con 0x7fbb6815be70 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6815d9c0 con 0x7fbb68071af0 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68159440 0x7fbb68150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68159440 0x7fbb68150e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56876/0 (socket says 172.21.15.67:56876) 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1024389310 learned_addr learned my addr 172.21.15.67:0/1024389310 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb67400640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb68071af0 0x7fbb6813dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb67e00640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb6815be70 0x7fbb68151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb68071af0 msgr2=0x7fbb6813dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb68071af0 0x7fbb6813dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb6815be70 msgr2=0x7fbb68151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb6815be70 0x7fbb68151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 -- 172.21.15.67:0/1024389310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68151fb0 con 0x7fbb68159440 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5ea00640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68159440 0x7fbb68150e30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbb500ab040 tx=0x7fbb5001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5fe00640 1 -- 172.21.15.67:0/1024389310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb500bc450 con 0x7fbb68159440 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5fe00640 1 -- 172.21.15.67:0/1024389310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb500bc5f0 con 0x7fbb68159440 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb5fe00640 1 -- 172.21.15.67:0/1024389310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb500bc9e0 con 0x7fbb68159440 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb681519f0 con 0x7fbb68159440 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.715+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb680b2a10 con 0x7fbb68159440 2024-04-17T10:25:44.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.719+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6804ef30 con 0x7fbb68159440 2024-04-17T10:25:44.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.719+0000 7fbb5fe00640 1 -- 172.21.15.67:0/1024389310 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbb500bcb80 con 0x7fbb68159440 2024-04-17T10:25:44.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.719+0000 7fbb5fe00640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb44066f90 0x7fbb44069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:44.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.719+0000 7fbb5fe00640 1 -- 172.21.15.67:0/1024389310 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fbb5011f710 con 0x7fbb68159440 2024-04-17T10:25:44.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.719+0000 7fbb67400640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb44066f90 0x7fbb44069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:44.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.719+0000 7fbb67400640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb44066f90 0x7fbb44069450 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbb58001ea0 tx=0x7fbb58009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:44.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.723+0000 7fbb5fe00640 1 -- 172.21.15.67:0/1024389310 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbb500c3030 con 0x7fbb68159440 2024-04-17T10:25:44.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbb68156470 con 0x7fbb68159440 2024-04-17T10:25:44.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb5fe00640 1 -- 172.21.15.67:0/1024389310 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbb500ad020 con 0x7fbb68159440 2024-04-17T10:25:44.881 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb44066f90 msgr2=0x7fbb44069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb44066f90 0x7fbb44069450 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbb58001ea0 tx=0x7fbb58009040 comp rx=0 tx=0).stop 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68159440 msgr2=0x7fbb68150e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68159440 0x7fbb68150e30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fbb500ab040 tx=0x7fbb5001ada0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 shutdown_connections 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb44066f90 0x7fbb44069450 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb6815be70 0x7fbb68151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb68159440 0x7fbb68150e30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 --2- 172.21.15.67:0/1024389310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb68071af0 0x7fbb6813dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:44.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 >> 172.21.15.67:0/1024389310 conn(0x7fbb680b1740 msgr2=0x7fbb6815a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:44.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 shutdown_connections 2024-04-17T10:25:44.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:44.887+0000 7fbb6e800640 1 -- 172.21.15.67:0/1024389310 wait complete. 2024-04-17T10:25:44.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197e00640 1 Processor -- start 2024-04-17T10:25:45.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197e00640 1 -- start start 2024-04-17T10:25:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f11980b5d80 0x7f11980b6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11980b9850 con 0x7f119815db00 2024-04-17T10:25:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11980d1c80 con 0x7f11980b6950 2024-04-17T10:25:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11980d2850 con 0x7f11980b5d80 2024-04-17T10:25:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f119815db00 0x7f11980b9140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37292/0 (socket says 172.21.15.67:37292) 2024-04-17T10:25:45.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196a00640 1 -- 172.21.15.67:0/927994900 learned_addr learned my addr 172.21.15.67:0/927994900 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:45.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 --2- 172.21.15.67:0/927994900 >> v2:172.21.15.86:3300/0 conn(0x7f11980b5d80 0x7f11980b6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746890504 0 0) 0x7f1188001120 con 0x7f119815db00 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f117c003530 con 0x7f119815db00 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192012062 0 0) 0x7f11980d1c80 con 0x7f11980b6950 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1188001120 con 0x7f11980b6950 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2647680730 0 0) 0x7f117c003530 con 0x7f119815db00 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 >> v2:172.21.15.86:3300/0 conn(0x7f11980b5d80 msgr2=0x7f11980b6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 --2- 172.21.15.67:0/927994900 >> v2:172.21.15.86:3300/0 conn(0x7f11980b5d80 0x7f11980b6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 >> v1:172.21.15.86:6789/0 conn(0x7f11980b6950 legacy=0x7f11980b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119815dee0 con 0x7f119815db00 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f1188001850 con 0x7f119815db00 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 >> v1:172.21.15.67:6789/0 conn(0x7f119815db00 legacy=0x7f11980b9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c003ba0 0x7f117c003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c004810 0x7f117c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f117c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11980d2850 con 0x7f117c0051d0 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c00bff0 con 0x7f117c003ba0 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c00c1f0 con 0x7f117c004810 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 --2- 172.21.15.67:0/927994900 >> v2:172.21.15.86:3300/0 conn(0x7f11980b5d80 0x7f11980b6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196a00640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c003ba0 0x7f117c003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f117c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f118fe00640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c004810 0x7f117c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 -- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c004810 msgr2=0x7f117c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f117c004810 0x7f117c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 -- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c003ba0 msgr2=0x7f117c003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c003ba0 0x7f117c003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1197400640 1 -- 172.21.15.67:0/927994900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c003530 con 0x7f117c0051d0 2024-04-17T10:25:45.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.051+0000 7f1196a00640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f117c003ba0 0x7f117c003fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197400640 1 -- 172.21.15.67:0/927994900 reap_dead start 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197400640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f117c00b9d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f119808cd00 tx=0x7f119001ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f8945ddbc7b2b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11900b2070 con 0x7f117c0051d0 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11900a5e30 con 0x7f117c0051d0 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1196000640 1 -- 172.21.15.67:0/927994900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1188007dc0 con 0x7f117c0051d0 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197e00640 1 -- 172.21.15.67:0/927994900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 msgr2=0x7f117c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197e00640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f117c00b9d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f119808cd00 tx=0x7f119001ce20 comp rx=0 tx=0).stop 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197e00640 1 -- 172.21.15.67:0/927994900 shutdown_connections 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197e00640 1 --2- 172.21.15.67:0/927994900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f117c00b9d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197e00640 1 -- 172.21.15.67:0/927994900 >> 172.21.15.67:0/927994900 conn(0x7f11980d0710 msgr2=0x7f11980d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197e00640 1 -- 172.21.15.67:0/927994900 shutdown_connections 2024-04-17T10:25:45.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.055+0000 7f1197e00640 1 -- 172.21.15.67:0/927994900 wait complete. 2024-04-17T10:25:45.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 Processor -- start 2024-04-17T10:25:45.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 -- start start 2024-04-17T10:25:45.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f11981f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1198153130 0x7f1198153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f119815b450 0x7f119815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11980d2200 con 0x7f117c0051d0 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11980d1ae0 con 0x7f1198153130 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11980d24e0 con 0x7f119815b450 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f11981f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f11981f3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56902/0 (socket says 172.21.15.67:56902) 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 -- 172.21.15.67:0/1881658904 learned_addr learned my addr 172.21.15.67:0/1881658904 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197400640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f119815b450 0x7f119815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f118fe00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1198153130 0x7f1198153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 -- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f119815b450 msgr2=0x7f119815b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f119815b450 0x7f119815b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 -- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1198153130 msgr2=0x7f1198153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1198153130 0x7f1198153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 -- 172.21.15.67:0/1881658904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11981548f0 con 0x7f117c0051d0 2024-04-17T10:25:45.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f118fe00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1198153130 0x7f1198153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1196a00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f11981f3f20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f11880072c0 tx=0x7f1188007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f118ea00640 1 -- 172.21.15.67:0/1881658904 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118801a070 con 0x7f117c0051d0 2024-04-17T10:25:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f118ea00640 1 -- 172.21.15.67:0/1881658904 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118800f9a0 con 0x7f117c0051d0 2024-04-17T10:25:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f118ea00640 1 -- 172.21.15.67:0/1881658904 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118800fd70 con 0x7f117c0051d0 2024-04-17T10:25:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1198154050 con 0x7f117c0051d0 2024-04-17T10:25:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.059+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1198154610 con 0x7f117c0051d0 2024-04-17T10:25:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.063+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f119804f780 con 0x7f117c0051d0 2024-04-17T10:25:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.063+0000 7f118ea00640 1 -- 172.21.15.67:0/1881658904 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1188014450 con 0x7f117c0051d0 2024-04-17T10:25:45.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.063+0000 7f118ea00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1174066f90 0x7f1174069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.063+0000 7f118ea00640 1 -- 172.21.15.67:0/1881658904 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f1188084d90 con 0x7f117c0051d0 2024-04-17T10:25:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.063+0000 7f118fe00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1174066f90 0x7f1174069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.063+0000 7f118fe00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1174066f90 0x7f1174069450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1198154ff0 tx=0x7f1180009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.067+0000 7f118ea00640 1 -- 172.21.15.67:0/1881658904 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f11980b9850 con 0x7f117c0051d0 2024-04-17T10:25:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.227+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f11980d1c60 con 0x7f117c0051d0 2024-04-17T10:25:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.227+0000 7f118ea00640 1 -- 172.21.15.67:0/1881658904 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f11880553c0 con 0x7f117c0051d0 2024-04-17T10:25:45.225 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1174066f90 msgr2=0x7f1174069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1174066f90 0x7f1174069450 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1198154ff0 tx=0x7f1180009040 comp rx=0 tx=0).stop 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 msgr2=0x7f11981f3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f11981f3f20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f11880072c0 tx=0x7f1188007780 comp rx=0 tx=0).stop 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 shutdown_connections 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1174066f90 0x7f1174069450 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f119815b450 0x7f119815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1198153130 0x7f1198153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 --2- 172.21.15.67:0/1881658904 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f117c0051d0 0x7f11981f3f20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 >> 172.21.15.67:0/1881658904 conn(0x7f11980d0710 msgr2=0x7f11980b7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 shutdown_connections 2024-04-17T10:25:45.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.231+0000 7f1197e00640 1 -- 172.21.15.67:0/1881658904 wait complete. 2024-04-17T10:25:45.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 Processor -- start 2024-04-17T10:25:45.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 -- start start 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748145040 0x7fa74806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa74806ef10 0x7fa748071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa748071850 0x7fa74815f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7480a86c0 con 0x7fa748071850 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74815bf10 con 0x7fa74806ef10 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.391+0000 7fa750200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa748071ca0 con 0x7fa748145040 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa748071850 0x7fa74815f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa748071850 0x7fa74815f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56910/0 (socket says 172.21.15.67:56910) 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 -- 172.21.15.67:0/2114273912 learned_addr learned my addr 172.21.15.67:0/2114273912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74d800640 1 --2- 172.21.15.67:0/2114273912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748145040 0x7fa74806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74ce00640 1 --2- 172.21.15.67:0/2114273912 >> v2:172.21.15.86:3300/0 conn(0x7fa74806ef10 0x7fa748071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 -- 172.21.15.67:0/2114273912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748145040 msgr2=0x7fa74806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:45.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 --2- 172.21.15.67:0/2114273912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748145040 0x7fa74806e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 -- 172.21.15.67:0/2114273912 >> v2:172.21.15.86:3300/0 conn(0x7fa74806ef10 msgr2=0x7fa748071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:45.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 --2- 172.21.15.67:0/2114273912 >> v2:172.21.15.86:3300/0 conn(0x7fa74806ef10 0x7fa748071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 -- 172.21.15.67:0/2114273912 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74806e2f0 con 0x7fa748071850 2024-04-17T10:25:45.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa74e200640 1 --2- 172.21.15.67:0/2114273912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa748071850 0x7fa74815f510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa74806bac0 tx=0x7fa74001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e5a1c3d6296d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa747e00640 1 -- 172.21.15.67:0/2114273912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7400b4070 con 0x7fa748071850 2024-04-17T10:25:45.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa747e00640 1 -- 172.21.15.67:0/2114273912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7400af400 con 0x7fa748071850 2024-04-17T10:25:45.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.395+0000 7fa747e00640 1 -- 172.21.15.67:0/2114273912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7400af7d0 con 0x7fa748071850 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- 172.21.15.67:0/2114273912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa748071850 msgr2=0x7fa74815f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 --2- 172.21.15.67:0/2114273912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa748071850 0x7fa74815f510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa74806bac0 tx=0x7fa74001f1e0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- 172.21.15.67:0/2114273912 shutdown_connections 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 --2- 172.21.15.67:0/2114273912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa748071850 0x7fa74815f510 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 --2- 172.21.15.67:0/2114273912 >> v2:172.21.15.86:3300/0 conn(0x7fa74806ef10 0x7fa748071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 --2- 172.21.15.67:0/2114273912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748145040 0x7fa74806e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- 172.21.15.67:0/2114273912 >> 172.21.15.67:0/2114273912 conn(0x7fa74806ac90 msgr2=0x7fa74806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- 172.21.15.67:0/2114273912 shutdown_connections 2024-04-17T10:25:45.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- 172.21.15.67:0/2114273912 wait complete. 2024-04-17T10:25:45.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 Processor -- start 2024-04-17T10:25:45.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- start start 2024-04-17T10:25:45.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74806ef10 0x7fa7480b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748071850 0x7fa7480b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa748145040 0x7fa7481f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74806dca0 con 0x7fa74806ef10 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74806d9c0 con 0x7fa748145040 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa750200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74806df80 con 0x7fa748071850 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74806ef10 0x7fa7480b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74806ef10 0x7fa7480b43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56926/0 (socket says 172.21.15.67:56926) 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 -- 172.21.15.67:0/4094905193 learned_addr learned my addr 172.21.15.67:0/4094905193 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74ce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748071850 0x7fa7480b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74e200640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa748145040 0x7fa7481f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 -- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748071850 msgr2=0x7fa7480b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748071850 0x7fa7480b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 -- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa748145040 msgr2=0x7fa7481f3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa748145040 0x7fa7481f3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.399+0000 7fa74d800640 1 -- 172.21.15.67:0/4094905193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7480addb0 con 0x7fa74806ef10 2024-04-17T10:25:45.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa74d800640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74806ef10 0x7fa7480b43b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00e870 tx=0x7fa73c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa745600640 1 -- 172.21.15.67:0/4094905193 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c017070 con 0x7fa74806ef10 2024-04-17T10:25:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa745600640 1 -- 172.21.15.67:0/4094905193 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c00c940 con 0x7fa74806ef10 2024-04-17T10:25:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa745600640 1 -- 172.21.15.67:0/4094905193 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c00ccf0 con 0x7fa74806ef10 2024-04-17T10:25:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7480ae090 con 0x7fa74806ef10 2024-04-17T10:25:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7480ae650 con 0x7fa74806ef10 2024-04-17T10:25:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa748072d50 con 0x7fa74806ef10 2024-04-17T10:25:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa745600640 1 -- 172.21.15.67:0/4094905193 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa73c01a070 con 0x7fa74806ef10 2024-04-17T10:25:45.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa745600640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa745600640 1 -- 172.21.15.67:0/4094905193 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fa73c0855b0 con 0x7fa74806ef10 2024-04-17T10:25:45.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.403+0000 7fa74ce00640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.407+0000 7fa74ce00640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7480ad490 tx=0x7fa734008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.407+0000 7fa745600640 1 -- 172.21.15.67:0/4094905193 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa73c055b60 con 0x7fa74806ef10 2024-04-17T10:25:45.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.571+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa748074e30 con 0x7fa74806ef10 2024-04-17T10:25:45.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.571+0000 7fa745600640 1 -- 172.21.15.67:0/4094905193 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7fa73c059c00 con 0x7fa74806ef10 2024-04-17T10:25:45.568 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:45.568 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 msgr2=0x7fa728069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa7480ad490 tx=0x7fa734008040 comp rx=0 tx=0).stop 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74806ef10 msgr2=0x7fa7480b43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74806ef10 0x7fa7480b43b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00e870 tx=0x7fa73c00ec70 comp rx=0 tx=0).stop 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 shutdown_connections 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa748145040 0x7fa7481f3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa748071850 0x7fa7480b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 --2- 172.21.15.67:0/4094905193 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74806ef10 0x7fa7480b43b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 >> 172.21.15.67:0/4094905193 conn(0x7fa74806ac90 msgr2=0x7fa74813f7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:45.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 shutdown_connections 2024-04-17T10:25:45.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.575+0000 7fa750200640 1 -- 172.21.15.67:0/4094905193 wait complete. 2024-04-17T10:25:45.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:45.590 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:45.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06ff400640 1 Processor -- start 2024-04-17T10:25:45.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06ff400640 1 -- start start 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06f8159440 0x7f06f80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06ff400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f815cb30 con 0x7f06f815c580 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06ff400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80cd610 con 0x7f06f80713e0 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06ff400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80a8650 con 0x7f06f8159440 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06fca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f06f8159440 0x7f06f80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06fd400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06f815c580 0x7f06f815d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37308/0 (socket says 172.21.15.67:37308) 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06fd400640 1 -- 172.21.15.67:0/619418499 learned_addr learned my addr 172.21.15.67:0/619418499 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06fca00640 1 -- 172.21.15.67:0/619418499 >> v1:172.21.15.86:6789/0 conn(0x7f06f80713e0 legacy=0x7f06f815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06fca00640 1 -- 172.21.15.67:0/619418499 >> v1:172.21.15.67:6789/0 conn(0x7f06f815c580 legacy=0x7f06f815d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06fca00640 1 -- 172.21.15.67:0/619418499 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80cd8f0 con 0x7f06f8159440 2024-04-17T10:25:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.739+0000 7f06f7400640 1 -- 172.21.15.67:0/619418499 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112545515 0 0) 0x7f06f815cb30 con 0x7f06f815c580 2024-04-17T10:25:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06fca00640 1 --2- 172.21.15.67:0/619418499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f8159440 0x7f06f80ce0f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0097c0 tx=0x7f06f815df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a06bdaad92b2b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06f7400640 1 -- 172.21.15.67:0/619418499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0b0070 con 0x7f06f8159440 2024-04-17T10:25:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06f7400640 1 -- 172.21.15.67:0/619418499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec0ab600 con 0x7f06f8159440 2024-04-17T10:25:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06f7400640 1 -- 172.21.15.67:0/619418499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0ab9d0 con 0x7f06f8159440 2024-04-17T10:25:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06ff400640 1 -- 172.21.15.67:0/619418499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f8159440 msgr2=0x7f06f80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06ff400640 1 --2- 172.21.15.67:0/619418499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f8159440 0x7f06f80ce0f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f06ec0097c0 tx=0x7f06f815df50 comp rx=0 tx=0).stop 2024-04-17T10:25:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06ff400640 1 -- 172.21.15.67:0/619418499 shutdown_connections 2024-04-17T10:25:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06ff400640 1 --2- 172.21.15.67:0/619418499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f8159440 0x7f06f80ce0f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06ff400640 1 -- 172.21.15.67:0/619418499 >> 172.21.15.67:0/619418499 conn(0x7f06f80b1740 msgr2=0x7f06f80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06ff400640 1 -- 172.21.15.67:0/619418499 shutdown_connections 2024-04-17T10:25:45.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.743+0000 7f06ff400640 1 -- 172.21.15.67:0/619418499 wait complete. 2024-04-17T10:25:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 Processor -- start 2024-04-17T10:25:45.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 -- start start 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80713e0 0x7f06f81519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f8159440 0x7f06f81504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815c580 0x7f06f8154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f8169640 con 0x7f06f80713e0 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f81694c0 con 0x7f06f815c580 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f81697c0 con 0x7f06f8159440 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80713e0 0x7f06f81519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80713e0 0x7f06f81519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56944/0 (socket says 172.21.15.67:56944) 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 -- 172.21.15.67:0/3720369953 learned_addr learned my addr 172.21.15.67:0/3720369953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fd400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815c580 0x7f06f8154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06f7e00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f8159440 0x7f06f81504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 -- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f8159440 msgr2=0x7f06f81504c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f8159440 0x7f06f81504c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 -- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815c580 msgr2=0x7f06f8154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815c580 0x7f06f8154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 -- 172.21.15.67:0/3720369953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80b2a10 con 0x7f06f80713e0 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06f7e00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f8159440 0x7f06f81504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fd400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815c580 0x7f06f8154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06fca00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80713e0 0x7f06f81519c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f06ec019070 tx=0x7f06ec01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06f4c00640 1 -- 172.21.15.67:0/3720369953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0b0050 con 0x7f06f80713e0 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06f4c00640 1 -- 172.21.15.67:0/3720369953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec0bc070 con 0x7f06f80713e0 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f80b2cf0 con 0x7f06f80713e0 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06f4c00640 1 -- 172.21.15.67:0/3720369953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0b5a50 con 0x7f06f80713e0 2024-04-17T10:25:45.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f80b4340 con 0x7f06f80713e0 2024-04-17T10:25:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.747+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f804ef30 con 0x7f06f80713e0 2024-04-17T10:25:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.751+0000 7f06f4c00640 1 -- 172.21.15.67:0/3720369953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f06ec0b5bf0 con 0x7f06f80713e0 2024-04-17T10:25:45.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.751+0000 7f06f4c00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.751+0000 7f06f4c00640 1 -- 172.21.15.67:0/3720369953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f06ec11fc70 con 0x7f06f80713e0 2024-04-17T10:25:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.751+0000 7f06f7e00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:45.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.751+0000 7f06f7e00640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06f81554d0 tx=0x7f06e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:45.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.755+0000 7f06f4c00640 1 -- 172.21.15.67:0/3720369953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f06ec016050 con 0x7f06f80713e0 2024-04-17T10:25:45.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.915+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06f814eb60 con 0x7f06f80713e0 2024-04-17T10:25:45.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.915+0000 7f06f4c00640 1 -- 172.21.15.67:0/3720369953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06ec0f0880 con 0x7f06f80713e0 2024-04-17T10:25:45.911 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 msgr2=0x7f06d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06f81554d0 tx=0x7f06e4008040 comp rx=0 tx=0).stop 2024-04-17T10:25:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80713e0 msgr2=0x7f06f81519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80713e0 0x7f06f81519c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f06ec019070 tx=0x7f06ec01aca0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 shutdown_connections 2024-04-17T10:25:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815c580 0x7f06f8154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f8159440 0x7f06f81504c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 --2- 172.21.15.67:0/3720369953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80713e0 0x7f06f81519c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:45.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 >> 172.21.15.67:0/3720369953 conn(0x7f06f80b1740 msgr2=0x7f06f80af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:45.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 shutdown_connections 2024-04-17T10:25:45.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:45.919+0000 7f06ff400640 1 -- 172.21.15.67:0/3720369953 wait complete. 2024-04-17T10:25:45.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.079+0000 7fb58f600640 1 Processor -- start 2024-04-17T10:25:46.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.079+0000 7fb58f600640 1 -- start start 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.079+0000 7fb58f600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb5881474c0 0x7fb5880b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.079+0000 7fb58f600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb58806b3a0 0x7fb5880d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58f600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5881616a0 con 0x7fb5880b3920 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58f600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb588161940 con 0x7fb58806b3a0 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58f600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb588161c20 con 0x7fb5881474c0 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb58806b3a0 0x7fb5880d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb57fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5880b3920 0x7fb58806ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37324/0 (socket says 172.21.15.67:37324) 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb57fe00640 1 -- 172.21.15.67:0/2561659498 learned_addr learned my addr 172.21.15.67:0/2561659498 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58cc00640 1 --2- 172.21.15.67:0/2561659498 >> v2:172.21.15.86:3300/0 conn(0x7fb5881474c0 0x7fb5880b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58d600640 1 -- 172.21.15.67:0/2561659498 >> v2:172.21.15.86:3300/0 conn(0x7fb5881474c0 msgr2=0x7fb5880b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58d600640 1 --2- 172.21.15.67:0/2561659498 >> v2:172.21.15.86:3300/0 conn(0x7fb5881474c0 0x7fb5880b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58d600640 1 -- 172.21.15.67:0/2561659498 >> v1:172.21.15.67:6789/0 conn(0x7fb5880b3920 legacy=0x7fb58806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58d600640 1 -- 172.21.15.67:0/2561659498 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb588161f00 con 0x7fb58806b3a0 2024-04-17T10:25:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58cc00640 1 --2- 172.21.15.67:0/2561659498 >> v2:172.21.15.86:3300/0 conn(0x7fb5881474c0 0x7fb5880b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb587e00640 1 -- 172.21.15.67:0/2561659498 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807963249 0 0) 0x7fb5881616a0 con 0x7fb5880b3920 2024-04-17T10:25:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb58d600640 1 --2- 172.21.15.67:0/2561659498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb58806b3a0 0x7fb5880d0480 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5880b63d0 tx=0x7fb58001f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d1c24e43121a5e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb587e00640 1 -- 172.21.15.67:0/2561659498 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800b4070 con 0x7fb58806b3a0 2024-04-17T10:25:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb587e00640 1 -- 172.21.15.67:0/2561659498 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5800af600 con 0x7fb58806b3a0 2024-04-17T10:25:46.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.083+0000 7fb587e00640 1 -- 172.21.15.67:0/2561659498 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800af9d0 con 0x7fb58806b3a0 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- 172.21.15.67:0/2561659498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb58806b3a0 msgr2=0x7fb5880d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 --2- 172.21.15.67:0/2561659498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb58806b3a0 0x7fb5880d0480 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5880b63d0 tx=0x7fb58001f110 comp rx=0 tx=0).stop 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- 172.21.15.67:0/2561659498 shutdown_connections 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 --2- 172.21.15.67:0/2561659498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb58806b3a0 0x7fb5880d0480 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 --2- 172.21.15.67:0/2561659498 >> v2:172.21.15.86:3300/0 conn(0x7fb5881474c0 0x7fb5880b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- 172.21.15.67:0/2561659498 >> 172.21.15.67:0/2561659498 conn(0x7fb5880b5ef0 msgr2=0x7fb588099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- 172.21.15.67:0/2561659498 shutdown_connections 2024-04-17T10:25:46.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- 172.21.15.67:0/2561659498 wait complete. 2024-04-17T10:25:46.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 Processor -- start 2024-04-17T10:25:46.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- start start 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb58806b3a0 0x7fb58813f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880b3920 0x7fb5880cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5881474c0 0x7fb5880cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb588162420 con 0x7fb5880b3920 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880b3d70 con 0x7fb58806b3a0 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58f600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5880d09c0 con 0x7fb5881474c0 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880b3920 0x7fb5880cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880b3920 0x7fb5880cecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56966/0 (socket says 172.21.15.67:56966) 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 -- 172.21.15.67:0/1732214314 learned_addr learned my addr 172.21.15.67:0/1732214314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58cc00640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb58806b3a0 0x7fb58813f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58d600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5881474c0 0x7fb5880cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 -- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5881474c0 msgr2=0x7fb5880cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5881474c0 0x7fb5880cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 -- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb58806b3a0 msgr2=0x7fb58813f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb58806b3a0 0x7fb58813f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb57fe00640 1 -- 172.21.15.67:0/1732214314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880ae230 con 0x7fb5880b3920 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.087+0000 7fb58d600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5881474c0 0x7fb5880cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb57fe00640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880b3920 0x7fb5880cecf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5700011f0 tx=0x7fb57000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb585600640 1 -- 172.21.15.67:0/1732214314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570018070 con 0x7fb5880b3920 2024-04-17T10:25:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb585600640 1 -- 172.21.15.67:0/1732214314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5700074e0 con 0x7fb5880b3920 2024-04-17T10:25:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb585600640 1 -- 172.21.15.67:0/1732214314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570004a10 con 0x7fb5880b3920 2024-04-17T10:25:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58813fc10 con 0x7fb5880b3920 2024-04-17T10:25:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5881401d0 con 0x7fb5880b3920 2024-04-17T10:25:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58804ef30 con 0x7fb5880b3920 2024-04-17T10:25:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb585600640 1 -- 172.21.15.67:0/1732214314 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb570004bb0 con 0x7fb5880b3920 2024-04-17T10:25:46.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb585600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb564066f60 0x7fb564069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb585600640 1 -- 172.21.15.67:0/1732214314 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb5700846b0 con 0x7fb5880b3920 2024-04-17T10:25:46.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.091+0000 7fb58cc00640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb564066f60 0x7fb564069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.095+0000 7fb58cc00640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb564066f60 0x7fb564069420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb578006fd0 tx=0x7fb578008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.095+0000 7fb585600640 1 -- 172.21.15.67:0/1732214314 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb5880ae230 con 0x7fb5880b3920 2024-04-17T10:25:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.255+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5880727e0 con 0x7fb5880b3920 2024-04-17T10:25:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.255+0000 7fb585600640 1 -- 172.21.15.67:0/1732214314 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb570054bd0 con 0x7fb5880b3920 2024-04-17T10:25:46.252 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb564066f60 msgr2=0x7fb564069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb564066f60 0x7fb564069420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb578006fd0 tx=0x7fb578008040 comp rx=0 tx=0).stop 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880b3920 msgr2=0x7fb5880cecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880b3920 0x7fb5880cecf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb5700011f0 tx=0x7fb57000c9b0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 shutdown_connections 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb564066f60 0x7fb564069420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5881474c0 0x7fb5880cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5880b3920 0x7fb5880cecf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 --2- 172.21.15.67:0/1732214314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb58806b3a0 0x7fb58813f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 >> 172.21.15.67:0/1732214314 conn(0x7fb5880b5ef0 msgr2=0x7fb58806f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 shutdown_connections 2024-04-17T10:25:46.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.259+0000 7fb58f600640 1 -- 172.21.15.67:0/1732214314 wait complete. 2024-04-17T10:25:46.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:46.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f903cc00640 1 Processor -- start 2024-04-17T10:25:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f903cc00640 1 -- start start 2024-04-17T10:25:46.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f903cc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f903815dd60 0x7f9038161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f903cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90380d06a0 0x7f9038161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f903cc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903815e140 con 0x7f903809adf0 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f903cc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90381626d0 con 0x7f903815dd60 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f903cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9038099d50 con 0x7f90380d06a0 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9035400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f903809adf0 0x7f90380cff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37332/0 (socket says 172.21.15.67:37332) 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9035400640 1 -- 172.21.15.67:0/465843304 learned_addr learned my addr 172.21.15.67:0/465843304 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9035e00640 1 --2- 172.21.15.67:0/465843304 >> v2:172.21.15.86:3300/0 conn(0x7f903815dd60 0x7f9038161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9036800640 1 --2- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90380d06a0 0x7f9038161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9034a00640 1 -- 172.21.15.67:0/465843304 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157011820 0 0) 0x7f903815e140 con 0x7f903809adf0 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9034a00640 1 -- 172.21.15.67:0/465843304 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f901c003530 con 0x7f903809adf0 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9035e00640 1 -- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90380d06a0 msgr2=0x7f9038161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9035e00640 1 --2- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90380d06a0 0x7f9038161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9035e00640 1 -- 172.21.15.67:0/465843304 >> v1:172.21.15.67:6789/0 conn(0x7f903809adf0 legacy=0x7f90380cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.423+0000 7f9035e00640 1 -- 172.21.15.67:0/465843304 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903809a030 con 0x7f903815dd60 2024-04-17T10:25:46.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f9035e00640 1 --2- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f903815dd60 0x7f9038161880 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f90280097c0 tx=0x7f902801c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9445f077a6470cad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f9034a00640 1 -- 172.21.15.67:0/465843304 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280a7070 con 0x7f903815dd60 2024-04-17T10:25:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f9034a00640 1 -- 172.21.15.67:0/465843304 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90280b0070 con 0x7f903815dd60 2024-04-17T10:25:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f9034a00640 1 -- 172.21.15.67:0/465843304 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280ab890 con 0x7f903815dd60 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 -- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f903815dd60 msgr2=0x7f9038161880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 --2- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f903815dd60 0x7f9038161880 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f90280097c0 tx=0x7f902801c470 comp rx=0 tx=0).stop 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 -- 172.21.15.67:0/465843304 shutdown_connections 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 --2- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f90380d06a0 0x7f9038161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 --2- 172.21.15.67:0/465843304 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f903815dd60 0x7f9038161880 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 -- 172.21.15.67:0/465843304 >> 172.21.15.67:0/465843304 conn(0x7f90380b34a0 msgr2=0x7f90380b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 -- 172.21.15.67:0/465843304 shutdown_connections 2024-04-17T10:25:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 -- 172.21.15.67:0/465843304 wait complete. 2024-04-17T10:25:46.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 Processor -- start 2024-04-17T10:25:46.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.427+0000 7f903cc00640 1 -- start start 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f903809adf0 0x7f9038149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90380d06a0 0x7f90380b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f903815dd60 0x7f903815f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903806c730 con 0x7f903809adf0 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903806c5b0 con 0x7f90380d06a0 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f903806c8b0 con 0x7f903815dd60 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f903809adf0 0x7f9038149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f903809adf0 0x7f9038149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56972/0 (socket says 172.21.15.67:56972) 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 -- 172.21.15.67:0/2122679414 learned_addr learned my addr 172.21.15.67:0/2122679414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9036800640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f903815dd60 0x7f903815f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035400640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90380d06a0 0x7f90380b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 -- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f903815dd60 msgr2=0x7f903815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f903815dd60 0x7f903815f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 -- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90380d06a0 msgr2=0x7f90380b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90380d06a0 0x7f90380b5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 -- 172.21.15.67:0/2122679414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90380b64d0 con 0x7f903809adf0 2024-04-17T10:25:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f9035e00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f903809adf0 0x7f9038149c00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f90280ad040 tx=0x7f902801acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f902e000640 1 -- 172.21.15.67:0/2122679414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280bf610 con 0x7f903809adf0 2024-04-17T10:25:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f902e000640 1 -- 172.21.15.67:0/2122679414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90280abe50 con 0x7f903809adf0 2024-04-17T10:25:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f902e000640 1 -- 172.21.15.67:0/2122679414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280bfa10 con 0x7f903809adf0 2024-04-17T10:25:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90380b67b0 con 0x7f903809adf0 2024-04-17T10:25:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90380b6d70 con 0x7f903809adf0 2024-04-17T10:25:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.431+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903804ef30 con 0x7f903809adf0 2024-04-17T10:25:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.435+0000 7f902e000640 1 -- 172.21.15.67:0/2122679414 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f90280bd020 con 0x7f903809adf0 2024-04-17T10:25:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.435+0000 7f902e000640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9018066f60 0x7f9018069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.435+0000 7f902e000640 1 -- 172.21.15.67:0/2122679414 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f902811fcf0 con 0x7f903809adf0 2024-04-17T10:25:46.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.435+0000 7f9035400640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9018066f60 0x7f9018069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.435+0000 7f9035400640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9018066f60 0x7f9018069420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f90380c8810 tx=0x7f9024009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.439+0000 7f902e000640 1 -- 172.21.15.67:0/2122679414 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9028016090 con 0x7f903809adf0 2024-04-17T10:25:46.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.603+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f90380d0130 con 0x7f903809adf0 2024-04-17T10:25:46.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.603+0000 7f902e000640 1 -- 172.21.15.67:0/2122679414 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f90280f0a10 con 0x7f903809adf0 2024-04-17T10:25:46.599 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:46.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.603+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9018066f60 msgr2=0x7f9018069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.603+0000 7f903cc00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9018066f60 0x7f9018069420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f90380c8810 tx=0x7f9024009040 comp rx=0 tx=0).stop 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.603+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f903809adf0 msgr2=0x7f9038149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.603+0000 7f903cc00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f903809adf0 0x7f9038149c00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f90280ad040 tx=0x7f902801acb0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 shutdown_connections 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9018066f60 0x7f9018069420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f903815dd60 0x7f903815f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90380d06a0 0x7f90380b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 --2- 172.21.15.67:0/2122679414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f903809adf0 0x7f9038149c00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 >> 172.21.15.67:0/2122679414 conn(0x7f90380b34a0 msgr2=0x7f90380b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 shutdown_connections 2024-04-17T10:25:46.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.607+0000 7f903cc00640 1 -- 172.21.15.67:0/2122679414 wait complete. 2024-04-17T10:25:46.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ec2a00640 1 Processor -- start 2024-04-17T10:25:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ec2a00640 1 -- start start 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ec2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ebc0aef60 0x7f6ebc0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0cce60 0x7f6ebc06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ec2a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc0b4990 con 0x7f6ebc06e7b0 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ec2a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc0cd2c0 con 0x7f6ebc0aef60 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ec2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc0af340 con 0x7f6ebc0cce60 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ebc0aef60 0x7f6ebc0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ebc0aef60 0x7f6ebc0cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56974/0 (socket says 172.21.15.67:56974) 2024-04-17T10:25:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebbe00640 1 -- 172.21.15.67:0/4228487411 learned_addr learned my addr 172.21.15.67:0/4228487411 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:46.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6eb3400640 1 --2- 172.21.15.67:0/4228487411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0cce60 0x7f6ebc06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebbe00640 1 -- 172.21.15.67:0/4228487411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0cce60 msgr2=0x7f6ebc06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:46.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/4228487411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0cce60 0x7f6ebc06e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebbe00640 1 -- 172.21.15.67:0/4228487411 >> v1:172.21.15.86:6789/0 conn(0x7f6ebc06e7b0 legacy=0x7f6ebc0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebbe00640 1 -- 172.21.15.67:0/4228487411 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc15ae90 con 0x7f6ebc0aef60 2024-04-17T10:25:46.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.767+0000 7f6ebb400640 1 -- 172.21.15.67:0/4228487411 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421616973 0 0) 0x7f6eb40014a0 con 0x7f6ebc06e7b0 2024-04-17T10:25:46.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/4228487411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc0cc920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009870 tx=0x7f6eac01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8ee62feb1cf019 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ebb400640 1 -- 172.21.15.67:0/4228487411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac0b0070 con 0x7f6ebc0aef60 2024-04-17T10:25:46.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ebb400640 1 -- 172.21.15.67:0/4228487411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac0ab600 con 0x7f6ebc0aef60 2024-04-17T10:25:46.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ebb400640 1 -- 172.21.15.67:0/4228487411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac0ab9d0 con 0x7f6ebc0aef60 2024-04-17T10:25:46.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 -- 172.21.15.67:0/4228487411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0aef60 msgr2=0x7f6ebc0cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/4228487411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc0cc920 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009870 tx=0x7f6eac01c4d0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 -- 172.21.15.67:0/4228487411 shutdown_connections 2024-04-17T10:25:46.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/4228487411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0cce60 0x7f6ebc06e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/4228487411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc0cc920 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 -- 172.21.15.67:0/4228487411 >> 172.21.15.67:0/4228487411 conn(0x7f6ebc0b1740 msgr2=0x7f6ebc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 -- 172.21.15.67:0/4228487411 shutdown_connections 2024-04-17T10:25:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.771+0000 7f6ec2a00640 1 -- 172.21.15.67:0/4228487411 wait complete. 2024-04-17T10:25:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 Processor -- start 2024-04-17T10:25:46.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 -- start start 2024-04-17T10:25:46.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc06e7b0 0x7f6ebc141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0cce60 0x7f6ebc14f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc15bc10 con 0x7f6ebc0cce60 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc15b410 con 0x7f6ebc0aef60 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc14db60 con 0x7f6ebc06e7b0 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0cce60 0x7f6ebc14f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0cce60 0x7f6ebc14f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56988/0 (socket says 172.21.15.67:56988) 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 -- 172.21.15.67:0/1701460012 learned_addr learned my addr 172.21.15.67:0/1701460012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6eb3400640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc06e7b0 0x7f6ebc141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 -- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc06e7b0 msgr2=0x7f6ebc141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc06e7b0 0x7f6ebc141440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 -- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0aef60 msgr2=0x7f6ebc141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 -- 172.21.15.67:0/1701460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc0ccc30 con 0x7f6ebc0cce60 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6eb3400640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc06e7b0 0x7f6ebc141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec0a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0cce60 0x7f6ebc14f550 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c780 tx=0x7f6eb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1701460012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4003db0 con 0x7f6ebc0cce60 2024-04-17T10:25:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1701460012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb400ce30 con 0x7f6ebc0cce60 2024-04-17T10:25:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1701460012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4005290 con 0x7f6ebc0cce60 2024-04-17T10:25:46.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc14d5c0 con 0x7f6ebc0cce60 2024-04-17T10:25:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.775+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc14d880 con 0x7f6ebc0cce60 2024-04-17T10:25:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.779+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc04ef30 con 0x7f6ebc0cce60 2024-04-17T10:25:46.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.779+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1701460012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6eb4005430 con 0x7f6ebc0cce60 2024-04-17T10:25:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.779+0000 7f6eb8c00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f90 0x7f6e98069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.779+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1701460012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6eb4085290 con 0x7f6ebc0cce60 2024-04-17T10:25:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.779+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f90 0x7f6e98069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:46.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.779+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f90 0x7f6e98069450 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0ad000 tx=0x7f6eac0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.783+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1701460012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6ebc0b4990 con 0x7f6ebc0cce60 2024-04-17T10:25:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.943+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6ebc148ab0 con 0x7f6ebc0cce60 2024-04-17T10:25:46.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.943+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1701460012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6eb4055e00 con 0x7f6ebc0cce60 2024-04-17T10:25:46.940 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f90 msgr2=0x7f6e98069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f90 0x7f6e98069450 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0ad000 tx=0x7f6eac0a6000 comp rx=0 tx=0).stop 2024-04-17T10:25:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0cce60 msgr2=0x7f6ebc14f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0cce60 0x7f6ebc14f550 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c780 tx=0x7f6eb400cc40 comp rx=0 tx=0).stop 2024-04-17T10:25:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 shutdown_connections 2024-04-17T10:25:46.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f90 0x7f6e98069450 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc0cce60 0x7f6ebc14f550 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0aef60 0x7f6ebc141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 --2- 172.21.15.67:0/1701460012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc06e7b0 0x7f6ebc141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 >> 172.21.15.67:0/1701460012 conn(0x7f6ebc0b1740 msgr2=0x7f6ebc15a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 shutdown_connections 2024-04-17T10:25:46.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:46.947+0000 7f6ec2a00640 1 -- 172.21.15.67:0/1701460012 wait complete. 2024-04-17T10:25:46.973 INFO:tasks.cephfs_test_runner:Starting test: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) test flag scrub_mdsdir 2024-04-17T10:25:46.973 INFO:tasks.cephfs_test_runner:test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-04-17T10:25:46.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.119+0000 7f45bda00640 1 Processor -- start 2024-04-17T10:25:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.119+0000 7f45bda00640 1 -- start start 2024-04-17T10:25:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45bda00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 0x7f45b80cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45bda00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b806e2f0 con 0x7f45b8145040 2024-04-17T10:25:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45bda00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45b806d6e0 con 0x7f45b8071310 2024-04-17T10:25:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45bda00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b80a86c0 con 0x7f45b8071770 2024-04-17T10:25:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f45b8145040 0x7f45b806e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37338/0 (socket says 172.21.15.67:37338) 2024-04-17T10:25:47.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 -- 172.21.15.67:0/502768546 learned_addr learned my addr 172.21.15.67:0/502768546 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b7400640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 0x7f45b80cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375898685 0 0) 0x7f45b806e2f0 con 0x7f45b8145040 2024-04-17T10:25:47.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f459c003530 con 0x7f45b8145040 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2030817705 0 0) 0x7f459c003530 con 0x7f45b8145040 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 msgr2=0x7f45b80cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 0x7f45b80cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 >> v1:172.21.15.86:6789/0 conn(0x7f45b8071310 legacy=0x7f45b80cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b806d9c0 con 0x7f45b8145040 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f45a8001680 con 0x7f45b8145040 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b7400640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 0x7f45b80cca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 >> v1:172.21.15.67:6789/0 conn(0x7f45b8145040 legacy=0x7f45b806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f459c0039f0 0x7f459c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 0x7f459c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f459c004d50 0x7f459c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c00b9d0 con 0x7f459c0039f0 2024-04-17T10:25:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c00bb50 con 0x7f459c004d50 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c00bd50 con 0x7f45b8071770 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f459c0039f0 0x7f459c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45ae000640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 0x7f459c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 -- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 msgr2=0x7f459c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f45b8071770 0x7f459c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 -- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f459c004d50 msgr2=0x7f459c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f459c004d50 0x7f459c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.123+0000 7f45b6a00640 1 -- 172.21.15.67:0/502768546 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c003530 con 0x7f459c0039f0 2024-04-17T10:25:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45b6a00640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f459c0039f0 0x7f459c003dd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f45a800c110 tx=0x7f45a801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a4e7682619bc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8007c50 con 0x7f459c0039f0 2024-04-17T10:25:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a80a9020 con 0x7f459c0039f0 2024-04-17T10:25:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45b6000640 1 -- 172.21.15.67:0/502768546 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8005b20 con 0x7f459c0039f0 2024-04-17T10:25:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45bda00640 1 -- 172.21.15.67:0/502768546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f459c0039f0 msgr2=0x7f459c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45bda00640 1 --2- 172.21.15.67:0/502768546 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f459c0039f0 0x7f459c003dd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f45a800c110 tx=0x7f45a801cbc0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45b7400640 1 -- 172.21.15.67:0/502768546 reap_dead start 2024-04-17T10:25:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45bda00640 1 -- 172.21.15.67:0/502768546 shutdown_connections 2024-04-17T10:25:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45bda00640 1 -- 172.21.15.67:0/502768546 >> 172.21.15.67:0/502768546 conn(0x7f45b806ac90 msgr2=0x7f45b806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45bda00640 1 -- 172.21.15.67:0/502768546 shutdown_connections 2024-04-17T10:25:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45bda00640 1 -- 172.21.15.67:0/502768546 wait complete. 2024-04-17T10:25:47.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.127+0000 7f45bda00640 1 Processor -- start 2024-04-17T10:25:47.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 -- start start 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f459c0039f0 0x7f45b815b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b81f35a0 0x7f45b80ad500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b80ada40 0x7f45b8143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b806df80 con 0x7f45b80ada40 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b815bf10 con 0x7f45b81f35a0 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45b8071ba0 con 0x7f459c0039f0 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b80ada40 0x7f45b8143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b80ada40 0x7f45b8143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56994/0 (socket says 172.21.15.67:56994) 2024-04-17T10:25:47.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 -- 172.21.15.67:0/1409235575 learned_addr learned my addr 172.21.15.67:0/1409235575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b6a00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f459c0039f0 0x7f45b815b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45ae000640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b81f35a0 0x7f45b80ad500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 -- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f459c0039f0 msgr2=0x7f45b815b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f459c0039f0 0x7f45b815b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 -- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b81f35a0 msgr2=0x7f45b80ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b81f35a0 0x7f45b80ad500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 -- 172.21.15.67:0/1409235575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b8140f30 con 0x7f45b80ada40 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b6a00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f459c0039f0 0x7f45b815b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45ae000640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b81f35a0 0x7f45b80ad500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:47.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45b7400640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b80ada40 0x7f45b8143730 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f45b80b2530 tx=0x7f45b000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45af400640 1 -- 172.21.15.67:0/1409235575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b001b070 con 0x7f45b80ada40 2024-04-17T10:25:47.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45af400640 1 -- 172.21.15.67:0/1409235575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b000bce0 con 0x7f45b80ada40 2024-04-17T10:25:47.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45af400640 1 -- 172.21.15.67:0/1409235575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0003be0 con 0x7f45b80ada40 2024-04-17T10:25:47.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b8141210 con 0x7f45b80ada40 2024-04-17T10:25:47.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b81417d0 con 0x7f45b80ada40 2024-04-17T10:25:47.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.131+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b8072d50 con 0x7f45b80ada40 2024-04-17T10:25:47.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.135+0000 7f45af400640 1 -- 172.21.15.67:0/1409235575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f45b001e070 con 0x7f45b80ada40 2024-04-17T10:25:47.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.135+0000 7f45af400640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4594066f60 0x7f4594069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.135+0000 7f45af400640 1 -- 172.21.15.67:0/1409235575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f45b0084e60 con 0x7f45b80ada40 2024-04-17T10:25:47.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.135+0000 7f45b6a00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4594066f60 0x7f4594069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.135+0000 7f45b6a00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4594066f60 0x7f4594069420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45a80aa000 tx=0x7f45b806d8b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.139+0000 7f45af400640 1 -- 172.21.15.67:0/1409235575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f45b00559d0 con 0x7f45b80ada40 2024-04-17T10:25:47.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.299+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45b8074e30 con 0x7f45b80ada40 2024-04-17T10:25:47.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45af400640 1 -- 172.21.15.67:0/1409235575 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f45b00552d0 con 0x7f45b80ada40 2024-04-17T10:25:47.298 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:47.298 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4594066f60 msgr2=0x7f4594069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4594066f60 0x7f4594069420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f45a80aa000 tx=0x7f45b806d8b0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b80ada40 msgr2=0x7f45b8143730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b80ada40 0x7f45b8143730 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f45b80b2530 tx=0x7f45b000b480 comp rx=0 tx=0).stop 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 shutdown_connections 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4594066f60 0x7f4594069420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45b80ada40 0x7f45b8143730 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45b81f35a0 0x7f45b80ad500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 --2- 172.21.15.67:0/1409235575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f459c0039f0 0x7f45b815b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 >> 172.21.15.67:0/1409235575 conn(0x7f45b806ac90 msgr2=0x7f45b80cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 shutdown_connections 2024-04-17T10:25:47.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.303+0000 7f45bda00640 1 -- 172.21.15.67:0/1409235575 wait complete. 2024-04-17T10:25:47.319 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:47.319 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:47.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc896e00640 1 Processor -- start 2024-04-17T10:25:47.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc896e00640 1 -- start start 2024-04-17T10:25:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc896e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900aef60 0x7fc8900cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc896e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc8900cce60 0x7fc89006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc896e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8900b4990 con 0x7fc89006e7b0 2024-04-17T10:25:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc896e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8900cd2c0 con 0x7fc8900cce60 2024-04-17T10:25:47.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc896e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900af340 con 0x7fc8900aef60 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc88f400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc8900cce60 0x7fc89006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc88fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900aef60 0x7fc8900cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc894e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc89006e7b0 0x7fc8900713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:34170/0 (socket says 172.21.15.67:34170) 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc894e00640 1 -- 172.21.15.67:0/1860265219 learned_addr learned my addr 172.21.15.67:0/1860265219 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc88f400640 1 -- 172.21.15.67:0/1860265219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900aef60 msgr2=0x7fc8900cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc88f400640 1 --2- 172.21.15.67:0/1860265219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900aef60 0x7fc8900cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc88f400640 1 -- 172.21.15.67:0/1860265219 >> v1:172.21.15.86:6789/0 conn(0x7fc89006e7b0 legacy=0x7fc8900713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.471+0000 7fc88f400640 1 -- 172.21.15.67:0/1860265219 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89015ae90 con 0x7fc8900cce60 2024-04-17T10:25:47.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc88f400640 1 --2- 172.21.15.67:0/1860265219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900cce60 0x7fc89006e270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00ba60 tx=0x7fc87c01eae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff9cea892f640094 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc88ea00640 1 -- 172.21.15.67:0/1860265219 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0b4070 con 0x7fc8900cce60 2024-04-17T10:25:47.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc88ea00640 1 -- 172.21.15.67:0/1860265219 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c0a78d0 con 0x7fc8900cce60 2024-04-17T10:25:47.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc88ea00640 1 -- 172.21.15.67:0/1860265219 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0a7ca0 con 0x7fc8900cce60 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 -- 172.21.15.67:0/1860265219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900cce60 msgr2=0x7fc89006e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 --2- 172.21.15.67:0/1860265219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900cce60 0x7fc89006e270 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00ba60 tx=0x7fc87c01eae0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 -- 172.21.15.67:0/1860265219 shutdown_connections 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 --2- 172.21.15.67:0/1860265219 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8900cce60 0x7fc89006e270 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 --2- 172.21.15.67:0/1860265219 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900aef60 0x7fc8900cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 -- 172.21.15.67:0/1860265219 >> 172.21.15.67:0/1860265219 conn(0x7fc8900b1740 msgr2=0x7fc8900b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 -- 172.21.15.67:0/1860265219 shutdown_connections 2024-04-17T10:25:47.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.475+0000 7fc896e00640 1 -- 172.21.15.67:0/1860265219 wait complete. 2024-04-17T10:25:47.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 Processor -- start 2024-04-17T10:25:47.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 -- start start 2024-04-17T10:25:47.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc89006e7b0 0x7fc8901480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900aef60 0x7fc890151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900cce60 0x7fc8901473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89015bc10 con 0x7fc8900aef60 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89015b410 con 0x7fc89006e7b0 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901487b0 con 0x7fc8900cce60 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900aef60 0x7fc890151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900aef60 0x7fc890151eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57004/0 (socket says 172.21.15.67:57004) 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 -- 172.21.15.67:0/261633625 learned_addr learned my addr 172.21.15.67:0/261633625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88fe00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc89006e7b0 0x7fc8901480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc894e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900cce60 0x7fc8901473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 -- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900cce60 msgr2=0x7fc8901473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900cce60 0x7fc8901473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 -- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc89006e7b0 msgr2=0x7fc8901480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc89006e7b0 0x7fc8901480b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 -- 172.21.15.67:0/261633625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8900afde0 con 0x7fc8900aef60 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc88f400640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900aef60 0x7fc890151eb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0b2040 tx=0x7fc87c01b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc887e00640 1 -- 172.21.15.67:0/261633625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0bd070 con 0x7fc8900aef60 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc887e00640 1 -- 172.21.15.67:0/261633625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c0afb70 con 0x7fc8900aef60 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc887e00640 1 -- 172.21.15.67:0/261633625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0b4040 con 0x7fc8900aef60 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc890152b60 con 0x7fc8900aef60 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc890153120 con 0x7fc8900aef60 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.479+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89004ef30 con 0x7fc8900aef60 2024-04-17T10:25:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.483+0000 7fc887e00640 1 -- 172.21.15.67:0/261633625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc87c0c0070 con 0x7fc8900aef60 2024-04-17T10:25:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.483+0000 7fc887e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc870066f90 0x7fc870069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.483+0000 7fc887e00640 1 -- 172.21.15.67:0/261633625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fc87c11f100 con 0x7fc8900aef60 2024-04-17T10:25:47.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.483+0000 7fc88fe00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc870066f90 0x7fc870069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.483+0000 7fc88fe00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc870066f90 0x7fc870069450 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8901f9c80 tx=0x7fc880009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.487+0000 7fc887e00640 1 -- 172.21.15.67:0/261633625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc87c018760 con 0x7fc8900aef60 2024-04-17T10:25:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.647+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc89006c3e0 con 0x7fc8900aef60 2024-04-17T10:25:47.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.647+0000 7fc887e00640 1 -- 172.21.15.67:0/261633625 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc87c0ef620 con 0x7fc8900aef60 2024-04-17T10:25:47.643 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc870066f90 msgr2=0x7fc870069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc870066f90 0x7fc870069450 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc8901f9c80 tx=0x7fc880009190 comp rx=0 tx=0).stop 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900aef60 msgr2=0x7fc890151eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900aef60 0x7fc890151eb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0b2040 tx=0x7fc87c01b880 comp rx=0 tx=0).stop 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 shutdown_connections 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc870066f90 0x7fc870069450 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8900cce60 0x7fc8901473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8900aef60 0x7fc890151eb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 --2- 172.21.15.67:0/261633625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc89006e7b0 0x7fc8901480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 >> 172.21.15.67:0/261633625 conn(0x7fc8900b1740 msgr2=0x7fc8900b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 shutdown_connections 2024-04-17T10:25:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.651+0000 7fc896e00640 1 -- 172.21.15.67:0/261633625 wait complete. 2024-04-17T10:25:47.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:47.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.811+0000 7f2fb6000640 1 Processor -- start 2024-04-17T10:25:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fb6000640 1 -- start start 2024-04-17T10:25:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fb6000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb01474c0 0x7f2fb00b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fb6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fb00b3920 0x7f2fb006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fb6000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb01616a0 con 0x7f2fb006b1d0 2024-04-17T10:25:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fb6000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb006b640 con 0x7f2fb00b3920 2024-04-17T10:25:47.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fb6000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00d1490 con 0x7f2fb01474c0 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fa6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fb00b3920 0x7f2fb006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fafe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2fb006b1d0 0x7f2fb015dfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37352/0 (socket says 172.21.15.67:37352) 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fafe00640 1 -- 172.21.15.67:0/556374364 learned_addr learned my addr 172.21.15.67:0/556374364 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2faf400640 1 --2- 172.21.15.67:0/556374364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb01474c0 0x7f2fb00b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fa6a00640 1 -- 172.21.15.67:0/556374364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb01474c0 msgr2=0x7f2fb00b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fa6a00640 1 --2- 172.21.15.67:0/556374364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb01474c0 0x7f2fb00b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fa6a00640 1 -- 172.21.15.67:0/556374364 >> v1:172.21.15.67:6789/0 conn(0x7f2fb006b1d0 legacy=0x7f2fb015dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2faea00640 1 -- 172.21.15.67:0/556374364 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938217554 0 0) 0x7f2fb01616a0 con 0x7f2fb006b1d0 2024-04-17T10:25:47.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.815+0000 7f2fa6a00640 1 -- 172.21.15.67:0/556374364 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb0161940 con 0x7f2fb00b3920 2024-04-17T10:25:47.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fa6a00640 1 --2- 172.21.15.67:0/556374364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb00b3920 0x7f2fb006ac90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2f980097c0 tx=0x7f2f9801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acdc3987291b7486 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2faea00640 1 -- 172.21.15.67:0/556374364 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f980b5070 con 0x7f2fb00b3920 2024-04-17T10:25:47.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2faea00640 1 -- 172.21.15.67:0/556374364 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f980a8990 con 0x7f2fb00b3920 2024-04-17T10:25:47.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2faea00640 1 -- 172.21.15.67:0/556374364 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f980a8d60 con 0x7f2fb00b3920 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 -- 172.21.15.67:0/556374364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb00b3920 msgr2=0x7f2fb006ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 --2- 172.21.15.67:0/556374364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb00b3920 0x7f2fb006ac90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2f980097c0 tx=0x7f2f9801ec90 comp rx=0 tx=0).stop 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 -- 172.21.15.67:0/556374364 shutdown_connections 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 --2- 172.21.15.67:0/556374364 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb00b3920 0x7f2fb006ac90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 --2- 172.21.15.67:0/556374364 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb01474c0 0x7f2fb00b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 -- 172.21.15.67:0/556374364 >> 172.21.15.67:0/556374364 conn(0x7f2fb00b5ef0 msgr2=0x7f2fb0099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 -- 172.21.15.67:0/556374364 shutdown_connections 2024-04-17T10:25:47.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.819+0000 7f2fb6000640 1 -- 172.21.15.67:0/556374364 wait complete. 2024-04-17T10:25:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 Processor -- start 2024-04-17T10:25:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 -- start start 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb006b1d0 0x7f2fb015cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb00b3920 0x7f2fb00add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb01474c0 0x7f2fb0140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb00b2250 con 0x7f2fb01474c0 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0161f00 con 0x7f2fb006b1d0 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0162420 con 0x7f2fb00b3920 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb01474c0 0x7f2fb0140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb01474c0 0x7f2fb0140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57016/0 (socket says 172.21.15.67:57016) 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 -- 172.21.15.67:0/2705009393 learned_addr learned my addr 172.21.15.67:0/2705009393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2faf400640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb006b1d0 0x7f2fb015cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fa6a00640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb00b3920 0x7f2fb00add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 -- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb00b3920 msgr2=0x7f2fb00add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb00b3920 0x7f2fb00add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 -- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb006b1d0 msgr2=0x7f2fb015cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb006b1d0 0x7f2fb015cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 -- 172.21.15.67:0/2705009393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb00ae2d0 con 0x7f2fb01474c0 2024-04-17T10:25:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fafe00640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb01474c0 0x7f2fb0140520 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800caf0 tx=0x7f2fa800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fa7e00640 1 -- 172.21.15.67:0/2705009393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa800aca0 con 0x7f2fb01474c0 2024-04-17T10:25:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fa7e00640 1 -- 172.21.15.67:0/2705009393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa800ae40 con 0x7f2fb01474c0 2024-04-17T10:25:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fa7e00640 1 -- 172.21.15.67:0/2705009393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa8012640 con 0x7f2fb01474c0 2024-04-17T10:25:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb015f290 con 0x7f2fb01474c0 2024-04-17T10:25:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.823+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb015f850 con 0x7f2fb01474c0 2024-04-17T10:25:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.827+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb004ef30 con 0x7f2fb01474c0 2024-04-17T10:25:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.827+0000 7f2fa7e00640 1 -- 172.21.15.67:0/2705009393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2fa80128a0 con 0x7f2fb01474c0 2024-04-17T10:25:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.827+0000 7f2fa7e00640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f8c066f60 0x7f2f8c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.827+0000 7f2fa7e00640 1 -- 172.21.15.67:0/2705009393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f2fa80859c0 con 0x7f2fb01474c0 2024-04-17T10:25:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.827+0000 7f2faf400640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f8c066f60 0x7f2f8c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.827+0000 7f2faf400640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f8c066f60 0x7f2f8c069420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2fa00079c0 tx=0x7f2fa0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.831+0000 7f2fa7e00640 1 -- 172.21.15.67:0/2705009393 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2fb00ae2d0 con 0x7f2fb01474c0 2024-04-17T10:25:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.995+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2fb00727e0 con 0x7f2fb01474c0 2024-04-17T10:25:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fa7e00640 1 -- 172.21.15.67:0/2705009393 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2fa8055ee0 con 0x7f2fb01474c0 2024-04-17T10:25:47.993 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f8c066f60 msgr2=0x7f2f8c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f8c066f60 0x7f2f8c069420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f2fa00079c0 tx=0x7f2fa0008040 comp rx=0 tx=0).stop 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb01474c0 msgr2=0x7f2fb0140520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb01474c0 0x7f2fb0140520 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f2fa800caf0 tx=0x7f2fa800a060 comp rx=0 tx=0).stop 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 shutdown_connections 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f8c066f60 0x7f2f8c069420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb01474c0 0x7f2fb0140520 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb00b3920 0x7f2fb00add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 --2- 172.21.15.67:0/2705009393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb006b1d0 0x7f2fb015cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 >> 172.21.15.67:0/2705009393 conn(0x7f2fb00b5ef0 msgr2=0x7f2fb00b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 shutdown_connections 2024-04-17T10:25:47.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:47.999+0000 7f2fb6000640 1 -- 172.21.15.67:0/2705009393 wait complete. 2024-04-17T10:25:48.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:48.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f63a0a00640 1 Processor -- start 2024-04-17T10:25:48.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f63a0a00640 1 -- start start 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f63a0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f639c159440 0x7f639c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f63a0a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f639c15c580 0x7f639c071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f63a0a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639c15cb30 con 0x7f639c0713e0 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f63a0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f639c0a8650 con 0x7f639c159440 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f63a0a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f639c15ce10 con 0x7f639c15c580 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f6399e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f639c159440 0x7f639c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f6399e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f639c159440 0x7f639c0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57018/0 (socket says 172.21.15.67:57018) 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f6399e00640 1 -- 172.21.15.67:0/83841295 learned_addr learned my addr 172.21.15.67:0/83841295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f639a800640 1 --2- 172.21.15.67:0/83841295 >> v2:172.21.15.86:3300/0 conn(0x7f639c15c580 0x7f639c071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f6399e00640 1 -- 172.21.15.67:0/83841295 >> v2:172.21.15.86:3300/0 conn(0x7f639c15c580 msgr2=0x7f639c071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f6399e00640 1 --2- 172.21.15.67:0/83841295 >> v2:172.21.15.86:3300/0 conn(0x7f639c15c580 0x7f639c071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f6399e00640 1 -- 172.21.15.67:0/83841295 >> v1:172.21.15.86:6789/0 conn(0x7f639c0713e0 legacy=0x7f639c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.163+0000 7f6399e00640 1 -- 172.21.15.67:0/83841295 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c0cd610 con 0x7f639c159440 2024-04-17T10:25:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f6399e00640 1 --2- 172.21.15.67:0/83841295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c159440 0x7f639c0ce0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f639c0ce660 tx=0x7f638c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c781d6fb1e4a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f6399400640 1 -- 172.21.15.67:0/83841295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c0ab430 con 0x7f639c159440 2024-04-17T10:25:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f6399400640 1 -- 172.21.15.67:0/83841295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c0b0070 con 0x7f639c159440 2024-04-17T10:25:48.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f6399400640 1 -- 172.21.15.67:0/83841295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c0ab7f0 con 0x7f639c159440 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 -- 172.21.15.67:0/83841295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c159440 msgr2=0x7f639c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 --2- 172.21.15.67:0/83841295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c159440 0x7f639c0ce0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f639c0ce660 tx=0x7f638c01c520 comp rx=0 tx=0).stop 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 -- 172.21.15.67:0/83841295 shutdown_connections 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 --2- 172.21.15.67:0/83841295 >> v2:172.21.15.86:3300/0 conn(0x7f639c15c580 0x7f639c071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 --2- 172.21.15.67:0/83841295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c159440 0x7f639c0ce0f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 -- 172.21.15.67:0/83841295 >> 172.21.15.67:0/83841295 conn(0x7f639c0b1740 msgr2=0x7f639c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 -- 172.21.15.67:0/83841295 shutdown_connections 2024-04-17T10:25:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.167+0000 7f63a0a00640 1 -- 172.21.15.67:0/83841295 wait complete. 2024-04-17T10:25:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 Processor -- start 2024-04-17T10:25:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 -- start start 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c0713e0 0x7f639c14dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f639c159440 0x7f639c14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f639c15c580 0x7f639c150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c15d9c0 con 0x7f639c0713e0 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f639c15d840 con 0x7f639c15c580 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639c15db40 con 0x7f639c159440 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c0713e0 0x7f639c14dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c0713e0 0x7f639c14dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57034/0 (socket says 172.21.15.67:57034) 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 -- 172.21.15.67:0/355774924 learned_addr learned my addr 172.21.15.67:0/355774924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f639a800640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f639c15c580 0x7f639c150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6391400640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f639c159440 0x7f639c14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 -- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f639c159440 msgr2=0x7f639c14f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f639c159440 0x7f639c14f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 -- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f639c15c580 msgr2=0x7f639c150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f639c15c580 0x7f639c150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 -- 172.21.15.67:0/355774924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c1f7270 con 0x7f639c0713e0 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6399e00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c0713e0 0x7f639c14dc80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f638c0ad040 tx=0x7f638c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6392a00640 1 -- 172.21.15.67:0/355774924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c0be5f0 con 0x7f639c0713e0 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6392a00640 1 -- 172.21.15.67:0/355774924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638c0be790 con 0x7f639c0713e0 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f6392a00640 1 -- 172.21.15.67:0/355774924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638c0beb60 con 0x7f639c0713e0 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639c14ed20 con 0x7f639c0713e0 2024-04-17T10:25:48.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639c14f2e0 con 0x7f639c0713e0 2024-04-17T10:25:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.171+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639c04ef30 con 0x7f639c0713e0 2024-04-17T10:25:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.175+0000 7f6392a00640 1 -- 172.21.15.67:0/355774924 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f638c0b43f0 con 0x7f639c0713e0 2024-04-17T10:25:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.175+0000 7f6392a00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6378066f90 0x7f6378069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.175+0000 7f6392a00640 1 -- 172.21.15.67:0/355774924 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f638c0a8070 con 0x7f639c0713e0 2024-04-17T10:25:48.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.175+0000 7f6391400640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6378066f90 0x7f6378069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.175+0000 7f6391400640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6378066f90 0x7f6378069450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f639c071b80 tx=0x7f6384009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.179+0000 7f6392a00640 1 -- 172.21.15.67:0/355774924 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f638c016110 con 0x7f639c0713e0 2024-04-17T10:25:48.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.343+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f639c14ead0 con 0x7f639c0713e0 2024-04-17T10:25:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.343+0000 7f6392a00640 1 -- 172.21.15.67:0/355774924 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f638c0f06f0 con 0x7f639c0713e0 2024-04-17T10:25:48.339 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6378066f90 msgr2=0x7f6378069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6378066f90 0x7f6378069450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f639c071b80 tx=0x7f6384009040 comp rx=0 tx=0).stop 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c0713e0 msgr2=0x7f639c14dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c0713e0 0x7f639c14dc80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f638c0ad040 tx=0x7f638c01ad60 comp rx=0 tx=0).stop 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 shutdown_connections 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6378066f90 0x7f6378069450 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f639c15c580 0x7f639c150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f639c159440 0x7f639c14f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 --2- 172.21.15.67:0/355774924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f639c0713e0 0x7f639c14dc80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 >> 172.21.15.67:0/355774924 conn(0x7f639c0b1740 msgr2=0x7f639c0acbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 shutdown_connections 2024-04-17T10:25:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.347+0000 7f63a0a00640 1 -- 172.21.15.67:0/355774924 wait complete. 2024-04-17T10:25:48.360 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:48.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.507+0000 7f24bc200640 1 Processor -- start 2024-04-17T10:25:48.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.507+0000 7f24bc200640 1 -- start start 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24bc200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b4161f90 0x7f24b40cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24bc200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24b409adf0 0x7f24b40d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24bc200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b415e140 con 0x7f24b415dd60 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24bc200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b41626d0 con 0x7f24b409adf0 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24bc200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b4099d50 con 0x7f24b4161f90 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24ba200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24b409adf0 0x7f24b40d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24ba200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24b409adf0 0x7f24b40d04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57048/0 (socket says 172.21.15.67:57048) 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24ba200640 1 -- 172.21.15.67:0/3166986177 learned_addr learned my addr 172.21.15.67:0/3166986177 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24ba200640 1 -- 172.21.15.67:0/3166986177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b4161f90 msgr2=0x7f24b40cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:48.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24b8e00640 1 --2- 172.21.15.67:0/3166986177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b4161f90 0x7f24b40cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24ba200640 1 --2- 172.21.15.67:0/3166986177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b4161f90 0x7f24b40cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24ba200640 1 -- 172.21.15.67:0/3166986177 >> v1:172.21.15.86:6789/0 conn(0x7f24b415dd60 legacy=0x7f24b4161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:48.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24ba200640 1 -- 172.21.15.67:0/3166986177 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b409a030 con 0x7f24b409adf0 2024-04-17T10:25:48.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.511+0000 7f24b8e00640 1 --2- 172.21.15.67:0/3166986177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b4161f90 0x7f24b40cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:48.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24ba200640 1 --2- 172.21.15.67:0/3166986177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b409adf0 0x7f24b40d04d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009e50 tx=0x7f24ac01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8942ba64d2ada6f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24b3e00640 1 -- 172.21.15.67:0/3166986177 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0a57f0 con 0x7f24b409adf0 2024-04-17T10:25:48.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24b3e00640 1 -- 172.21.15.67:0/3166986177 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac0a5990 con 0x7f24b409adf0 2024-04-17T10:25:48.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24b3e00640 1 -- 172.21.15.67:0/3166986177 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0a5d60 con 0x7f24b409adf0 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 -- 172.21.15.67:0/3166986177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b409adf0 msgr2=0x7f24b40d04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 --2- 172.21.15.67:0/3166986177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b409adf0 0x7f24b40d04d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f24ac009e50 tx=0x7f24ac01ca30 comp rx=0 tx=0).stop 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 -- 172.21.15.67:0/3166986177 shutdown_connections 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 --2- 172.21.15.67:0/3166986177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b409adf0 0x7f24b40d04d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 --2- 172.21.15.67:0/3166986177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b4161f90 0x7f24b40cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 -- 172.21.15.67:0/3166986177 >> 172.21.15.67:0/3166986177 conn(0x7f24b40b34a0 msgr2=0x7f24b40b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 -- 172.21.15.67:0/3166986177 shutdown_connections 2024-04-17T10:25:48.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 -- 172.21.15.67:0/3166986177 wait complete. 2024-04-17T10:25:48.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 Processor -- start 2024-04-17T10:25:48.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.515+0000 7f24bc200640 1 -- start start 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b409adf0 0x7f24b4160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b415dd60 0x7f24b4158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b4161f90 0x7f24b4158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b406c730 con 0x7f24b415dd60 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b406c5b0 con 0x7f24b4161f90 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b406c8b0 con 0x7f24b409adf0 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b415dd60 0x7f24b4158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b415dd60 0x7f24b4158660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57062/0 (socket says 172.21.15.67:57062) 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 -- 172.21.15.67:0/2845891927 learned_addr learned my addr 172.21.15.67:0/2845891927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b9800640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b409adf0 0x7f24b4160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24ba200640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b4161f90 0x7f24b4158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 -- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b409adf0 msgr2=0x7f24b4160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b409adf0 0x7f24b4160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 -- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b4161f90 msgr2=0x7f24b4158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b4161f90 0x7f24b4158ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 -- 172.21.15.67:0/2845891927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b4159220 con 0x7f24b415dd60 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b8e00640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b415dd60 0x7f24b4158660 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f24a000c7f0 tx=0x7f24a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b1600640 1 -- 172.21.15.67:0/2845891927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0015070 con 0x7f24b415dd60 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b1600640 1 -- 172.21.15.67:0/2845891927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a0007820 con 0x7f24b415dd60 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b1600640 1 -- 172.21.15.67:0/2845891927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0007c10 con 0x7f24b415dd60 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b4159500 con 0x7f24b415dd60 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b4159ac0 con 0x7f24b415dd60 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b41f6160 con 0x7f24b415dd60 2024-04-17T10:25:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.519+0000 7f24b1600640 1 -- 172.21.15.67:0/2845891927 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f24a0018070 con 0x7f24b415dd60 2024-04-17T10:25:48.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.523+0000 7f24b1600640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2494066f90 0x7f2494069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.523+0000 7f24b1600640 1 -- 172.21.15.67:0/2845891927 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f24a00854d0 con 0x7f24b415dd60 2024-04-17T10:25:48.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.523+0000 7f24b9800640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2494066f90 0x7f2494069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.523+0000 7f24b9800640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2494066f90 0x7f2494069450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24a8001810 tx=0x7f24a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.527+0000 7f24b1600640 1 -- 172.21.15.67:0/2845891927 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f24b41f6160 con 0x7f24b415dd60 2024-04-17T10:25:48.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.687+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f24b404ef30 con 0x7f24b415dd60 2024-04-17T10:25:48.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.687+0000 7f24b1600640 1 -- 172.21.15.67:0/2845891927 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f24a0055a70 con 0x7f24b415dd60 2024-04-17T10:25:48.684 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2494066f90 msgr2=0x7f2494069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2494066f90 0x7f2494069450 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f24a8001810 tx=0x7f24a8009040 comp rx=0 tx=0).stop 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b415dd60 msgr2=0x7f24b4158660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b415dd60 0x7f24b4158660 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f24a000c7f0 tx=0x7f24a000ccb0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 shutdown_connections 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2494066f90 0x7f2494069450 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24b4161f90 0x7f24b4158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24b415dd60 0x7f24b4158660 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 --2- 172.21.15.67:0/2845891927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24b409adf0 0x7f24b4160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 >> 172.21.15.67:0/2845891927 conn(0x7f24b40b34a0 msgr2=0x7f24b40cc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 shutdown_connections 2024-04-17T10:25:48.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.691+0000 7f24bc200640 1 -- 172.21.15.67:0/2845891927 wait complete. 2024-04-17T10:25:48.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir' 2024-04-17T10:25:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.851+0000 7fb3c3c00640 1 Processor -- start 2024-04-17T10:25:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.851+0000 7fb3c3c00640 1 -- start start 2024-04-17T10:25:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c3c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3bc06e6e0 0x7fb3bc0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c3c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3bc06eac0 0x7fb3bc0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c3c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc143ae0 0x7fb3bc143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c3c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc0cf430 con 0x7fb3bc06eac0 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c3c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc15ae40 con 0x7fb3bc06e6e0 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c3c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc071440 con 0x7fb3bc143ae0 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3bc06eac0 0x7fb3bc0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3bc06eac0 0x7fb3bc0ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57070/0 (socket says 172.21.15.67:57070) 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 -- 172.21.15.67:0/1065758881 learned_addr learned my addr 172.21.15.67:0/1065758881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c1200640 1 --2- 172.21.15.67:0/1065758881 >> v2:172.21.15.86:3300/0 conn(0x7fb3bc06e6e0 0x7fb3bc0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c1c00640 1 --2- 172.21.15.67:0/1065758881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc143ae0 0x7fb3bc143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 -- 172.21.15.67:0/1065758881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc143ae0 msgr2=0x7fb3bc143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 --2- 172.21.15.67:0/1065758881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc143ae0 0x7fb3bc143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 -- 172.21.15.67:0/1065758881 >> v2:172.21.15.86:3300/0 conn(0x7fb3bc06e6e0 msgr2=0x7fb3bc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 --2- 172.21.15.67:0/1065758881 >> v2:172.21.15.86:3300/0 conn(0x7fb3bc06e6e0 0x7fb3bc0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3b3e00640 1 -- 172.21.15.67:0/1065758881 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc071720 con 0x7fb3bc06eac0 2024-04-17T10:25:48.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c1200640 1 --2- 172.21.15.67:0/1065758881 >> v2:172.21.15.86:3300/0 conn(0x7fb3bc06e6e0 0x7fb3bc0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:48.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.855+0000 7fb3c1c00640 1 --2- 172.21.15.67:0/1065758881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc143ae0 0x7fb3bc143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:48.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3b3e00640 1 --2- 172.21.15.67:0/1065758881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc0ceef0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0cf620 tx=0x7fb3a401eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fb8617c65314274 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3bbe00640 1 -- 172.21.15.67:0/1065758881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40b4070 con 0x7fb3bc06eac0 2024-04-17T10:25:48.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3bbe00640 1 -- 172.21.15.67:0/1065758881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a40a7bc0 con 0x7fb3bc06eac0 2024-04-17T10:25:48.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3bbe00640 1 -- 172.21.15.67:0/1065758881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40af440 con 0x7fb3bc06eac0 2024-04-17T10:25:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 -- 172.21.15.67:0/1065758881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc06eac0 msgr2=0x7fb3bc0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/1065758881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc0ceef0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc0cf620 tx=0x7fb3a401eb30 comp rx=0 tx=0).stop 2024-04-17T10:25:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 -- 172.21.15.67:0/1065758881 shutdown_connections 2024-04-17T10:25:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/1065758881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc143ae0 0x7fb3bc143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/1065758881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc0ceef0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/1065758881 >> v2:172.21.15.86:3300/0 conn(0x7fb3bc06e6e0 0x7fb3bc0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 -- 172.21.15.67:0/1065758881 >> 172.21.15.67:0/1065758881 conn(0x7fb3bc06ac90 msgr2=0x7fb3bc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:48.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 -- 172.21.15.67:0/1065758881 shutdown_connections 2024-04-17T10:25:48.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 -- 172.21.15.67:0/1065758881 wait complete. 2024-04-17T10:25:48.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 Processor -- start 2024-04-17T10:25:48.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.859+0000 7fb3c3c00640 1 -- start start 2024-04-17T10:25:48.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc06e6e0 0x7fb3bc13ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc13b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc143ae0 0x7fb3bc1577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc15ebf0 con 0x7fb3bc143ae0 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc1444b0 con 0x7fb3bc06eac0 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc15ed70 con 0x7fb3bc06e6e0 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc143ae0 0x7fb3bc1577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc143ae0 0x7fb3bc1577b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57086/0 (socket says 172.21.15.67:57086) 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 -- 172.21.15.67:0/320001762 learned_addr learned my addr 172.21.15.67:0/320001762 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1200640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc06e6e0 0x7fb3bc13ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3b3e00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc13b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 -- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc06e6e0 msgr2=0x7fb3bc13ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc06e6e0 0x7fb3bc13ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 -- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc06eac0 msgr2=0x7fb3bc13b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc13b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 -- 172.21.15.67:0/320001762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc13d2f0 con 0x7fb3bc143ae0 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3b3e00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc13b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:48.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c1c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc143ae0 0x7fb3bc1577b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400cd50 tx=0x7fb3b400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3b9600640 1 -- 172.21.15.67:0/320001762 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400f680 con 0x7fb3bc143ae0 2024-04-17T10:25:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3b9600640 1 -- 172.21.15.67:0/320001762 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3b400f840 con 0x7fb3bc143ae0 2024-04-17T10:25:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3b9600640 1 -- 172.21.15.67:0/320001762 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b400fc30 con 0x7fb3bc143ae0 2024-04-17T10:25:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc141490 con 0x7fb3bc143ae0 2024-04-17T10:25:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc13dd90 con 0x7fb3bc143ae0 2024-04-17T10:25:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc072d50 con 0x7fb3bc143ae0 2024-04-17T10:25:48.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.863+0000 7fb3b9600640 1 -- 172.21.15.67:0/320001762 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb3b400e3e0 con 0x7fb3bc143ae0 2024-04-17T10:25:48.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.867+0000 7fb3b9600640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb398066f60 0x7fb398069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:48.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.867+0000 7fb3b9600640 1 -- 172.21.15.67:0/320001762 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fb3b4086140 con 0x7fb3bc143ae0 2024-04-17T10:25:48.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.867+0000 7fb3c1200640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb398066f60 0x7fb398069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:48.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.867+0000 7fb3c1200640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb398066f60 0x7fb398069420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00a9a0 tx=0x7fb3ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:48.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:48.871+0000 7fb3b9600640 1 -- 172.21.15.67:0/320001762 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb3b4014030 con 0x7fb3bc143ae0 2024-04-17T10:25:49.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.087+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]} v 0) v1 -- 0x7fb3bc074e30 con 0x7fb3bc143ae0 2024-04-17T10:25:49.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.823+0000 7fb3b9600640 1 -- 172.21.15.67:0/320001762 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]}]=0 v71) v1 ==== 151+0+0 (secure 0 0 0) 0x7fb3b4056d40 con 0x7fb3bc143ae0 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb398066f60 msgr2=0x7fb398069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb398066f60 0x7fb398069420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00a9a0 tx=0x7fb3ac008040 comp rx=0 tx=0).stop 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc143ae0 msgr2=0x7fb3bc1577b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc143ae0 0x7fb3bc1577b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb3b400cd50 tx=0x7fb3b400a650 comp rx=0 tx=0).stop 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 shutdown_connections 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb398066f60 0x7fb398069420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3bc143ae0 0x7fb3bc1577b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3bc06eac0 0x7fb3bc13b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 --2- 172.21.15.67:0/320001762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3bc06e6e0 0x7fb3bc13ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 >> 172.21.15.67:0/320001762 conn(0x7fb3bc06ac90 msgr2=0x7fb3bc0ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:49.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 shutdown_connections 2024-04-17T10:25:49.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.827+0000 7fb3c3c00640 1 -- 172.21.15.67:0/320001762 wait complete. 2024-04-17T10:25:49.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:25:49.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6433800640 1 Processor -- start 2024-04-17T10:25:49.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6433800640 1 -- start start 2024-04-17T10:25:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6433800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6433800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f642c0b3920 0x7f642c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6433800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c1616a0 con 0x7f642c06b1d0 2024-04-17T10:25:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6433800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c06b640 con 0x7f642c0b3920 2024-04-17T10:25:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6433800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0d1490 con 0x7f642c1474c0 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6423e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f642c0b3920 0x7f642c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6423e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f642c0b3920 0x7f642c06ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57092/0 (socket says 172.21.15.67:57092) 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6423e00640 1 -- 172.21.15.67:0/353951948 learned_addr learned my addr 172.21.15.67:0/353951948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6430e00640 1 --2- 172.21.15.67:0/353951948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6423e00640 1 -- 172.21.15.67:0/353951948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 msgr2=0x7f642c0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6423e00640 1 --2- 172.21.15.67:0/353951948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c0b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6423e00640 1 -- 172.21.15.67:0/353951948 >> v1:172.21.15.86:6789/0 conn(0x7f642c06b1d0 legacy=0x7f642c15dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:49.999+0000 7f6423e00640 1 -- 172.21.15.67:0/353951948 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c161940 con 0x7f642c0b3920 2024-04-17T10:25:49.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.003+0000 7f6423e00640 1 --2- 172.21.15.67:0/353951948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c06ac90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6414009870 tx=0x7f641401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445a8c0997d6a3f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:49.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.003+0000 7f642be00640 1 -- 172.21.15.67:0/353951948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64140b5070 con 0x7f642c0b3920 2024-04-17T10:25:49.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.003+0000 7f642be00640 1 -- 172.21.15.67:0/353951948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64140a8a50 con 0x7f642c0b3920 2024-04-17T10:25:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.003+0000 7f642be00640 1 -- 172.21.15.67:0/353951948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64140a8e20 con 0x7f642c0b3920 2024-04-17T10:25:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- 172.21.15.67:0/353951948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 msgr2=0x7f642c06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 --2- 172.21.15.67:0/353951948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c06ac90 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f6414009870 tx=0x7f641401ed50 comp rx=0 tx=0).stop 2024-04-17T10:25:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- 172.21.15.67:0/353951948 shutdown_connections 2024-04-17T10:25:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 --2- 172.21.15.67:0/353951948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c06ac90 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 --2- 172.21.15.67:0/353951948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c0b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- 172.21.15.67:0/353951948 >> 172.21.15.67:0/353951948 conn(0x7f642c0b5ef0 msgr2=0x7f642c099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- 172.21.15.67:0/353951948 shutdown_connections 2024-04-17T10:25:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- 172.21.15.67:0/353951948 wait complete. 2024-04-17T10:25:50.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 Processor -- start 2024-04-17T10:25:50.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- start start 2024-04-17T10:25:50.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c06b1d0 0x7f642c160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c1606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c152ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0b2250 con 0x7f642c0b3920 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c161f00 con 0x7f642c06b1d0 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6433800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c162420 con 0x7f642c1474c0 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c1606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c1606d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57108/0 (socket says 172.21.15.67:57108) 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 -- 172.21.15.67:0/855645758 learned_addr learned my addr 172.21.15.67:0/855645758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6430e00640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c06b1d0 0x7f642c160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6431800640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c152ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 -- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 msgr2=0x7f642c152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c152ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 -- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c06b1d0 msgr2=0x7f642c160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c06b1d0 0x7f642c160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 -- 172.21.15.67:0/855645758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c0cb430 con 0x7f642c0b3920 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.007+0000 7f6423e00640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c1606d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f64140b3040 tx=0x7f641401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6429600640 1 -- 172.21.15.67:0/855645758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64140c0070 con 0x7f642c0b3920 2024-04-17T10:25:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6429600640 1 -- 172.21.15.67:0/855645758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64140ae550 con 0x7f642c0b3920 2024-04-17T10:25:50.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6429600640 1 -- 172.21.15.67:0/855645758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64140b5070 con 0x7f642c0b3920 2024-04-17T10:25:50.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c0cb710 con 0x7f642c0b3920 2024-04-17T10:25:50.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c144ac0 con 0x7f642c0b3920 2024-04-17T10:25:50.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c04ef30 con 0x7f642c0b3920 2024-04-17T10:25:50.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6429600640 1 -- 172.21.15.67:0/855645758 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f64140b9070 con 0x7f642c0b3920 2024-04-17T10:25:50.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6429600640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6408066f60 0x7f6408069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6429600640 1 -- 172.21.15.67:0/855645758 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6414122c20 con 0x7f642c0b3920 2024-04-17T10:25:50.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.011+0000 7f6430e00640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6408066f60 0x7f6408069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.015+0000 7f6430e00640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6408066f60 0x7f6408069420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f641c00b930 tx=0x7f641c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:50.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.015+0000 7f6429600640 1 -- 172.21.15.67:0/855645758 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f64140188b0 con 0x7f642c0b3920 2024-04-17T10:25:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.179+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f642c09a310 con 0x7f642c0b3920 2024-04-17T10:25:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6429600640 1 -- 172.21.15.67:0/855645758 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f64140f3790 con 0x7f642c0b3920 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6408066f60 msgr2=0x7f6408069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6408066f60 0x7f6408069420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f641c00b930 tx=0x7f641c008040 comp rx=0 tx=0).stop 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 msgr2=0x7f642c1606d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c1606d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f64140b3040 tx=0x7f641401baf0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 shutdown_connections 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6408066f60 0x7f6408069420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642c1474c0 0x7f642c152ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642c0b3920 0x7f642c1606d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 --2- 172.21.15.67:0/855645758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f642c06b1d0 0x7f642c160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 >> 172.21.15.67:0/855645758 conn(0x7f642c0b5ef0 msgr2=0x7f642c0b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:50.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 shutdown_connections 2024-04-17T10:25:50.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.183+0000 7f6433800640 1 -- 172.21.15.67:0/855645758 wait complete. 2024-04-17T10:25:50.198 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:25:50.198 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:25:50.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:25:50.351 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:25:50.351 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:50.351 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:50.360 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:50.361 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:50.412 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:50.412 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:50.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:50.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cefe00640 1 Processor -- start 2024-04-17T10:25:50.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cefe00640 1 -- start start 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf01450b0 0x7f9cf015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cefe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9cf015faf0 0x7f9cf00ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cefe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf0160150 con 0x7f9cf00ce420 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cefe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf015ff20 con 0x7f9cf015faf0 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cefe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf00cedc0 con 0x7f9cf01450b0 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cee000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9cf00ce420 0x7f9cf015f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33060/0 (socket says 172.21.15.67:33060) 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cee000640 1 -- 172.21.15.67:0/3620636183 learned_addr learned my addr 172.21.15.67:0/3620636183 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cef400640 1 --2- 172.21.15.67:0/3620636183 >> v2:172.21.15.67:3300/0 conn(0x7f9cf015faf0 0x7f9cf00ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9ceea00640 1 --2- 172.21.15.67:0/3620636183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf01450b0 0x7f9cf015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cef400640 1 -- 172.21.15.67:0/3620636183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf01450b0 msgr2=0x7f9cf015bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cef400640 1 --2- 172.21.15.67:0/3620636183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf01450b0 0x7f9cf015bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cef400640 1 -- 172.21.15.67:0/3620636183 >> v1:172.21.15.67:6789/0 conn(0x7f9cf00ce420 legacy=0x7f9cf015f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9cef400640 1 -- 172.21.15.67:0/3620636183 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf006d930 con 0x7f9cf015faf0 2024-04-17T10:25:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.655+0000 7f9ced600640 1 -- 172.21.15.67:0/3620636183 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218159877 0 0) 0x7f9cf0160150 con 0x7f9cf00ce420 2024-04-17T10:25:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cef400640 1 --2- 172.21.15.67:0/3620636183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf015faf0 0x7f9cf00ce880 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009d90 tx=0x7f9ce801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fa3518b17f981e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9ced600640 1 -- 172.21.15.67:0/3620636183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce80b2070 con 0x7f9cf015faf0 2024-04-17T10:25:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9ced600640 1 -- 172.21.15.67:0/3620636183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce80a57a0 con 0x7f9cf015faf0 2024-04-17T10:25:50.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9ced600640 1 -- 172.21.15.67:0/3620636183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce80a5b70 con 0x7f9cf015faf0 2024-04-17T10:25:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 -- 172.21.15.67:0/3620636183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf015faf0 msgr2=0x7f9cf00ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 --2- 172.21.15.67:0/3620636183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf015faf0 0x7f9cf00ce880 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8009d90 tx=0x7f9ce801c8e0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 -- 172.21.15.67:0/3620636183 shutdown_connections 2024-04-17T10:25:50.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 --2- 172.21.15.67:0/3620636183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf015faf0 0x7f9cf00ce880 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 --2- 172.21.15.67:0/3620636183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf01450b0 0x7f9cf015bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 -- 172.21.15.67:0/3620636183 >> 172.21.15.67:0/3620636183 conn(0x7f9cf0067070 msgr2=0x7f9cf006ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 -- 172.21.15.67:0/3620636183 shutdown_connections 2024-04-17T10:25:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 -- 172.21.15.67:0/3620636183 wait complete. 2024-04-17T10:25:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 Processor -- start 2024-04-17T10:25:50.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.659+0000 7f9cefe00640 1 -- start start 2024-04-17T10:25:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf00ce420 0x7f9cf015b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf01450b0 0x7f9cf013b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf015faf0 0x7f9cf013c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf006eda0 con 0x7f9cf01450b0 2024-04-17T10:25:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf006ec20 con 0x7f9cf00ce420 2024-04-17T10:25:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf006ef20 con 0x7f9cf015faf0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf01450b0 0x7f9cf013b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf01450b0 0x7f9cf013b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36960/0 (socket says 172.21.15.67:36960) 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 -- 172.21.15.67:0/2304418216 learned_addr learned my addr 172.21.15.67:0/2304418216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cef400640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf015faf0 0x7f9cf013c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9ceea00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf00ce420 0x7f9cf015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 -- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf015faf0 msgr2=0x7f9cf013c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf015faf0 0x7f9cf013c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 -- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf00ce420 msgr2=0x7f9cf015b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf00ce420 0x7f9cf015b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 -- 172.21.15.67:0/2304418216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00cd560 con 0x7f9cf01450b0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9ceea00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf00ce420 0x7f9cf015b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cee000640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf01450b0 0x7f9cf013b7c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c520 tx=0x7f9cdc00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9ce6a00640 1 -- 172.21.15.67:0/2304418216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc018070 con 0x7f9cf01450b0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9ce6a00640 1 -- 172.21.15.67:0/2304418216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cdc0074e0 con 0x7f9cf01450b0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9ce6a00640 1 -- 172.21.15.67:0/2304418216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cdc0049f0 con 0x7f9cf01450b0 2024-04-17T10:25:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf00cd840 con 0x7f9cf01450b0 2024-04-17T10:25:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf00cde00 con 0x7f9cf01450b0 2024-04-17T10:25:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf0074dc0 con 0x7f9cf01450b0 2024-04-17T10:25:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.663+0000 7f9ce6a00640 1 -- 172.21.15.67:0/2304418216 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9cdc004b90 con 0x7f9cf01450b0 2024-04-17T10:25:50.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.667+0000 7f9ce6a00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9cd0066f60 0x7f9cd0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.667+0000 7f9ce6a00640 1 -- 172.21.15.67:0/2304418216 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9cdc084700 con 0x7f9cf01450b0 2024-04-17T10:25:50.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.667+0000 7f9ceea00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9cd0066f60 0x7f9cd0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.667+0000 7f9ceea00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9cd0066f60 0x7f9cd0069420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009bb0 tx=0x7f9ce0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:50.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.671+0000 7f9ce6a00640 1 -- 172.21.15.67:0/2304418216 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9cdc054cb0 con 0x7f9cf01450b0 2024-04-17T10:25:50.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cf0140f80 con 0x7f9cf01450b0 2024-04-17T10:25:50.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9ce6a00640 1 -- 172.21.15.67:0/2304418216 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f9cdc058d50 con 0x7f9cf01450b0 2024-04-17T10:25:50.830 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:50.830 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9cd0066f60 msgr2=0x7f9cd0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9cd0066f60 0x7f9cd0069420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0009bb0 tx=0x7f9ce0008040 comp rx=0 tx=0).stop 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf01450b0 msgr2=0x7f9cf013b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf01450b0 0x7f9cf013b7c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc00c520 tx=0x7f9cdc00c9e0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 shutdown_connections 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9cd0066f60 0x7f9cd0069420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9cf015faf0 0x7f9cf013c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf01450b0 0x7f9cf013b7c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 --2- 172.21.15.67:0/2304418216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9cf00ce420 0x7f9cf015b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 >> 172.21.15.67:0/2304418216 conn(0x7f9cf0067070 msgr2=0x7f9cf0142340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 shutdown_connections 2024-04-17T10:25:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.835+0000 7f9cefe00640 1 -- 172.21.15.67:0/2304418216 wait complete. 2024-04-17T10:25:50.852 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:50.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.999+0000 7f3e78e00640 1 Processor -- start 2024-04-17T10:25:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:50.999+0000 7f3e78e00640 1 -- start start 2024-04-17T10:25:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e78e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e78e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e740cd2c0 0x7f3e740cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e78e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e740cc500 con 0x7f3e7415b930 2024-04-17T10:25:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e78e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e740cc7e0 con 0x7f3e740cd2c0 2024-04-17T10:25:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e78e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e740ccac0 con 0x7f3e7409a160 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e740cd2c0 0x7f3e740cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e740cd2c0 0x7f3e740cf200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36974/0 (socket says 172.21.15.67:36974) 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 -- 172.21.15.67:0/1099116809 learned_addr learned my addr 172.21.15.67:0/1099116809 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e6be00640 1 --2- 172.21.15.67:0/1099116809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 -- 172.21.15.67:0/1099116809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 msgr2=0x7f3e7415c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 --2- 172.21.15.67:0/1099116809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 -- 172.21.15.67:0/1099116809 >> v1:172.21.15.86:6789/0 conn(0x7f3e7415b930 legacy=0x7f3e74099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:50.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 -- 172.21.15.67:0/1099116809 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e740cde60 con 0x7f3e740cd2c0 2024-04-17T10:25:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.003+0000 7f3e73e00640 1 --2- 172.21.15.67:0/1099116809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e740cf200 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009e50 tx=0x7f3e6c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ace47132e3ea6c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e72a00640 1 -- 172.21.15.67:0/1099116809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0a5700 con 0x7f3e740cd2c0 2024-04-17T10:25:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e72a00640 1 -- 172.21.15.67:0/1099116809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6c0a58a0 con 0x7f3e740cd2c0 2024-04-17T10:25:51.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e72a00640 1 -- 172.21.15.67:0/1099116809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c0a5c70 con 0x7f3e740cd2c0 2024-04-17T10:25:51.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- 172.21.15.67:0/1099116809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 msgr2=0x7f3e740cf200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 --2- 172.21.15.67:0/1099116809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e740cf200 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c009e50 tx=0x7f3e6c01ca30 comp rx=0 tx=0).stop 2024-04-17T10:25:51.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- 172.21.15.67:0/1099116809 shutdown_connections 2024-04-17T10:25:51.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 --2- 172.21.15.67:0/1099116809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e740cf200 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 --2- 172.21.15.67:0/1099116809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- 172.21.15.67:0/1099116809 >> 172.21.15.67:0/1099116809 conn(0x7f3e740d1630 msgr2=0x7f3e740d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:51.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- 172.21.15.67:0/1099116809 shutdown_connections 2024-04-17T10:25:51.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- 172.21.15.67:0/1099116809 wait complete. 2024-04-17T10:25:51.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 Processor -- start 2024-04-17T10:25:51.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- start start 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e741f7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e7415b930 0x7f3e7415d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74168ab0 con 0x7f3e740cd2c0 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74168930 con 0x7f3e7415b930 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e78e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e74168c30 con 0x7f3e7409a160 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e741f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e741f7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36978/0 (socket says 172.21.15.67:36978) 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e6be00640 1 -- 172.21.15.67:0/34246536 learned_addr learned my addr 172.21.15.67:0/34246536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:51.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e73400640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.007+0000 7f3e73e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e7415b930 0x7f3e7415d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6be00640 1 -- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 msgr2=0x7f3e7415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6be00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6be00640 1 -- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e7415b930 msgr2=0x7f3e7415d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6be00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e7415b930 0x7f3e7415d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6be00640 1 -- 172.21.15.67:0/34246536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e740c8950 con 0x7f3e740cd2c0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e73e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e7415b930 0x7f3e7415d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6be00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e741f7040 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c00d5f0 tx=0x7f3e5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6b400640 1 -- 172.21.15.67:0/34246536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c0125f0 con 0x7f3e740cd2c0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6b400640 1 -- 172.21.15.67:0/34246536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c019070 con 0x7f3e740cd2c0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6b400640 1 -- 172.21.15.67:0/34246536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c0129b0 con 0x7f3e740cd2c0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e740c8c30 con 0x7f3e740cd2c0 2024-04-17T10:25:51.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e740c91f0 con 0x7f3e740cd2c0 2024-04-17T10:25:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e7404f780 con 0x7f3e740cd2c0 2024-04-17T10:25:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6b400640 1 -- 172.21.15.67:0/34246536 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3e5c012b50 con 0x7f3e740cd2c0 2024-04-17T10:25:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6b400640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e50066f60 0x7f3e50069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.011+0000 7f3e6b400640 1 -- 172.21.15.67:0/34246536 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f3e5c0857c0 con 0x7f3e740cd2c0 2024-04-17T10:25:51.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.015+0000 7f3e73400640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e50066f60 0x7f3e50069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.015+0000 7f3e73400640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e50066f60 0x7f3e50069420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3e64001810 tx=0x7f3e64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.015+0000 7f3e6b400640 1 -- 172.21.15.67:0/34246536 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3e5c056330 con 0x7f3e740cd2c0 2024-04-17T10:25:51.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e741664f0 con 0x7f3e740cd2c0 2024-04-17T10:25:51.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e6b400640 1 -- 172.21.15.67:0/34246536 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e5c055c30 con 0x7f3e740cd2c0 2024-04-17T10:25:51.174 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:51.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e50066f60 msgr2=0x7f3e50069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e50066f60 0x7f3e50069420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3e64001810 tx=0x7f3e64009040 comp rx=0 tx=0).stop 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 msgr2=0x7f3e741f7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e741f7040 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c00d5f0 tx=0x7f3e5c00dab0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 shutdown_connections 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e50066f60 0x7f3e50069420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e7415b930 0x7f3e7415d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e740cd2c0 0x7f3e741f7040 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 --2- 172.21.15.67:0/34246536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e7409a160 0x7f3e7415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.179+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 >> 172.21.15.67:0/34246536 conn(0x7f3e740d1630 msgr2=0x7f3e740b6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.183+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 shutdown_connections 2024-04-17T10:25:51.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.183+0000 7f3e78e00640 1 -- 172.21.15.67:0/34246536 wait complete. 2024-04-17T10:25:51.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:51.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.343+0000 7f6d9c400640 1 Processor -- start 2024-04-17T10:25:51.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.343+0000 7f6d9c400640 1 -- start start 2024-04-17T10:25:51.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d9c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d9415a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d9c400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d940722b0 0x7f6d940aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d9c400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d940af2e0 con 0x7f6d9415e210 2024-04-17T10:25:51.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d9c400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d940ccbf0 con 0x7f6d940722b0 2024-04-17T10:25:51.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d9c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9415ac40 con 0x7f6d94071ed0 2024-04-17T10:25:51.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d9415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d940722b0 0x7f6d940aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d9415a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:47764/0 (socket says 172.21.15.67:47764) 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99a00640 1 -- 172.21.15.67:0/4246706200 learned_addr learned my addr 172.21.15.67:0/4246706200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99000640 1 -- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 msgr2=0x7f6d9415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99000640 1 --2- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d9415a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d93e00640 1 -- 172.21.15.67:0/4246706200 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223506242 0 0) 0x7f6d940af2e0 con 0x7f6d9415e210 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99000640 1 -- 172.21.15.67:0/4246706200 >> v1:172.21.15.86:6789/0 conn(0x7f6d9415e210 legacy=0x7f6d941508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99000640 1 -- 172.21.15.67:0/4246706200 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9415af20 con 0x7f6d940722b0 2024-04-17T10:25:51.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.347+0000 7f6d99a00640 1 --2- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d9415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:25:51.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d99000640 1 --2- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d940722b0 0x7f6d940aeda0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6d800097c0 tx=0x7f6d8001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfb25b90ee6d20dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d93e00640 1 -- 172.21.15.67:0/4246706200 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800b0070 con 0x7f6d940722b0 2024-04-17T10:25:51.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d93e00640 1 -- 172.21.15.67:0/4246706200 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d800ab600 con 0x7f6d940722b0 2024-04-17T10:25:51.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d93e00640 1 -- 172.21.15.67:0/4246706200 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800ab9d0 con 0x7f6d940722b0 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 -- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d940722b0 msgr2=0x7f6d940aeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 --2- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d940722b0 0x7f6d940aeda0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6d800097c0 tx=0x7f6d8001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 -- 172.21.15.67:0/4246706200 shutdown_connections 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 --2- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d940722b0 0x7f6d940aeda0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 --2- 172.21.15.67:0/4246706200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d9415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 -- 172.21.15.67:0/4246706200 >> 172.21.15.67:0/4246706200 conn(0x7f6d940b1740 msgr2=0x7f6d940b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 -- 172.21.15.67:0/4246706200 shutdown_connections 2024-04-17T10:25:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 -- 172.21.15.67:0/4246706200 wait complete. 2024-04-17T10:25:51.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 Processor -- start 2024-04-17T10:25:51.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.351+0000 7f6d9c400640 1 -- start start 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d94143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d940722b0 0x7f6d94149bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9415e210 0x7f6d9414bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940ce3a0 con 0x7f6d940722b0 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940ce220 con 0x7f6d9415e210 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d940ce520 con 0x7f6d94071ed0 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d940722b0 0x7f6d94149bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d940722b0 0x7f6d94149bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36992/0 (socket says 172.21.15.67:36992) 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 -- 172.21.15.67:0/3116817719 learned_addr learned my addr 172.21.15.67:0/3116817719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:51.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99a00640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d94143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9a400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9415e210 0x7f6d9414bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 -- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 msgr2=0x7f6d94143a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d94143a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 -- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9415e210 msgr2=0x7f6d9414bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9415e210 0x7f6d9414bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 -- 172.21.15.67:0/3116817719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9414ab80 con 0x7f6d940722b0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9a400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9415e210 0x7f6d9414bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d99000640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d940722b0 0x7f6d94149bd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6d800ae040 tx=0x7f6d80019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d91600640 1 -- 172.21.15.67:0/3116817719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800ba070 con 0x7f6d940722b0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d91600640 1 -- 172.21.15.67:0/3116817719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d800b43f0 con 0x7f6d940722b0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d91600640 1 -- 172.21.15.67:0/3116817719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d800b0040 con 0x7f6d940722b0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9414ae60 con 0x7f6d940722b0 2024-04-17T10:25:51.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9414b700 con 0x7f6d940722b0 2024-04-17T10:25:51.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9414b250 con 0x7f6d940722b0 2024-04-17T10:25:51.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.355+0000 7f6d91600640 1 -- 172.21.15.67:0/3116817719 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6d800bd070 con 0x7f6d940722b0 2024-04-17T10:25:51.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.359+0000 7f6d91600640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d74066f60 0x7f6d74069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.359+0000 7f6d91600640 1 -- 172.21.15.67:0/3116817719 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6d8011e300 con 0x7f6d940722b0 2024-04-17T10:25:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.359+0000 7f6d99a00640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d74066f60 0x7f6d74069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.359+0000 7f6d99a00640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d74066f60 0x7f6d74069420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d9414d960 tx=0x7f6d8800a350 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.363+0000 7f6d91600640 1 -- 172.21.15.67:0/3116817719 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6d80016090 con 0x7f6d940722b0 2024-04-17T10:25:51.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.523+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6d9404ef30 con 0x7f6d940722b0 2024-04-17T10:25:51.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.523+0000 7f6d91600640 1 -- 172.21.15.67:0/3116817719 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6d800ee820 con 0x7f6d940722b0 2024-04-17T10:25:51.519 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d74066f60 msgr2=0x7f6d74069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d74066f60 0x7f6d74069420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6d9414d960 tx=0x7f6d8800a350 comp rx=0 tx=0).stop 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d940722b0 msgr2=0x7f6d94149bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d940722b0 0x7f6d94149bd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6d800ae040 tx=0x7f6d80019240 comp rx=0 tx=0).stop 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 shutdown_connections 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6d74066f60 0x7f6d74069420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d9415e210 0x7f6d9414bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d940722b0 0x7f6d94149bd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 --2- 172.21.15.67:0/3116817719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d94071ed0 0x7f6d94143a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 >> 172.21.15.67:0/3116817719 conn(0x7f6d940b1740 msgr2=0x7f6d940af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 shutdown_connections 2024-04-17T10:25:51.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.527+0000 7f6d9c400640 1 -- 172.21.15.67:0/3116817719 wait complete. 2024-04-17T10:25:51.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:51.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 Processor -- start 2024-04-17T10:25:51.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 -- start start 2024-04-17T10:25:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55100aef60 0x7f55100cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100cce60 0x7f551006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f551006e7b0 0x7f55100713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55100cd2c0 con 0x7f551006e7b0 2024-04-17T10:25:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55100af340 con 0x7f55100aef60 2024-04-17T10:25:51.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5517a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55100b4990 con 0x7f55100cce60 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.691+0000 7f5515a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f551006e7b0 0x7f55100713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f551006e7b0 0x7f55100713e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36994/0 (socket says 172.21.15.67:36994) 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 -- 172.21.15.67:0/1205067529 learned_addr learned my addr 172.21.15.67:0/1205067529 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f550fe00640 1 --2- 172.21.15.67:0/1205067529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100cce60 0x7f551006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515000640 1 --2- 172.21.15.67:0/1205067529 >> v2:172.21.15.86:3300/0 conn(0x7f55100aef60 0x7f55100cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 -- 172.21.15.67:0/1205067529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100cce60 msgr2=0x7f551006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 --2- 172.21.15.67:0/1205067529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100cce60 0x7f551006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 -- 172.21.15.67:0/1205067529 >> v2:172.21.15.86:3300/0 conn(0x7f55100aef60 msgr2=0x7f55100cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:51.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 --2- 172.21.15.67:0/1205067529 >> v2:172.21.15.86:3300/0 conn(0x7f55100aef60 0x7f55100cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 -- 172.21.15.67:0/1205067529 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551015ae90 con 0x7f551006e7b0 2024-04-17T10:25:51.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f5515a00640 1 --2- 172.21.15.67:0/1205067529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f55100713e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f551006bdc0 tx=0x7f550801c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e0c9bb24386e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f550f400640 1 -- 172.21.15.67:0/1205067529 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080a9430 con 0x7f551006e7b0 2024-04-17T10:25:51.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f550f400640 1 -- 172.21.15.67:0/1205067529 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55080ae070 con 0x7f551006e7b0 2024-04-17T10:25:51.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.695+0000 7f550f400640 1 -- 172.21.15.67:0/1205067529 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080a97f0 con 0x7f551006e7b0 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- 172.21.15.67:0/1205067529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 msgr2=0x7f55100713e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 --2- 172.21.15.67:0/1205067529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f55100713e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f551006bdc0 tx=0x7f550801c6a0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- 172.21.15.67:0/1205067529 shutdown_connections 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 --2- 172.21.15.67:0/1205067529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f55100713e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 --2- 172.21.15.67:0/1205067529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100cce60 0x7f551006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 --2- 172.21.15.67:0/1205067529 >> v2:172.21.15.86:3300/0 conn(0x7f55100aef60 0x7f55100cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- 172.21.15.67:0/1205067529 >> 172.21.15.67:0/1205067529 conn(0x7f55100b1740 msgr2=0x7f55100b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- 172.21.15.67:0/1205067529 shutdown_connections 2024-04-17T10:25:51.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- 172.21.15.67:0/1205067529 wait complete. 2024-04-17T10:25:51.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 Processor -- start 2024-04-17T10:25:51.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- start start 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f5510149f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100aef60 0x7f551014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55100cce60 0x7f551014b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551015bc10 con 0x7f551006e7b0 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100b3120 con 0x7f55100cce60 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5517a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5510153e90 con 0x7f55100aef60 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f5510149f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f5510149f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37002/0 (socket says 172.21.15.67:37002) 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 -- 172.21.15.67:0/4258858687 learned_addr learned my addr 172.21.15.67:0/4258858687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f550fe00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100aef60 0x7f551014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515a00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55100cce60 0x7f551014b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 -- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100aef60 msgr2=0x7f551014d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100aef60 0x7f551014d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 -- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55100cce60 msgr2=0x7f551014b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55100cce60 0x7f551014b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.699+0000 7f5515000640 1 -- 172.21.15.67:0/4258858687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5510154170 con 0x7f551006e7b0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f5515000640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f5510149f60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f550400e8b0 tx=0x7f550400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f550cc00640 1 -- 172.21.15.67:0/4258858687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5504017070 con 0x7f551006e7b0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f550cc00640 1 -- 172.21.15.67:0/4258858687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550400c940 con 0x7f551006e7b0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f550cc00640 1 -- 172.21.15.67:0/4258858687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550400cd10 con 0x7f551006e7b0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5510154450 con 0x7f551006e7b0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551014a4a0 con 0x7f551006e7b0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551004ef30 con 0x7f551006e7b0 2024-04-17T10:25:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f550cc00640 1 -- 172.21.15.67:0/4258858687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f550401a070 con 0x7f551006e7b0 2024-04-17T10:25:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f550cc00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54f0066f60 0x7f54f0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f550cc00640 1 -- 172.21.15.67:0/4258858687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f55040853a0 con 0x7f551006e7b0 2024-04-17T10:25:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.703+0000 7f550fe00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54f0066f60 0x7f54f0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.707+0000 7f550fe00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54f0066f60 0x7f54f0069420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f551014cfb0 tx=0x7f54fc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:51.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.707+0000 7f550cc00640 1 -- 172.21.15.67:0/4258858687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f55040558c0 con 0x7f551006e7b0 2024-04-17T10:25:51.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.871+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f551014ecb0 con 0x7f551006e7b0 2024-04-17T10:25:51.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.871+0000 7f550cc00640 1 -- 172.21.15.67:0/4258858687 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5504059960 con 0x7f551006e7b0 2024-04-17T10:25:51.867 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54f0066f60 msgr2=0x7f54f0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54f0066f60 0x7f54f0069420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f551014cfb0 tx=0x7f54fc009190 comp rx=0 tx=0).stop 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 msgr2=0x7f5510149f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f5510149f60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f550400e8b0 tx=0x7f550400ed70 comp rx=0 tx=0).stop 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 shutdown_connections 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f54f0066f60 0x7f54f0069420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55100cce60 0x7f551014b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55100aef60 0x7f551014d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 --2- 172.21.15.67:0/4258858687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f551006e7b0 0x7f5510149f60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 >> 172.21.15.67:0/4258858687 conn(0x7f55100b1740 msgr2=0x7f55100b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 shutdown_connections 2024-04-17T10:25:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:51.875+0000 7f5517a00640 1 -- 172.21.15.67:0/4258858687 wait complete. 2024-04-17T10:25:51.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:52.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 Processor -- start 2024-04-17T10:25:52.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 -- start start 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4b8415dd60 0x7f4b84161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b8409adf0 0x7f4b84161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b840cff90 0x7f4b840d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b841626d0 con 0x7f4b840cff90 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8415e140 con 0x7f4b8415dd60 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b8a200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b840d0960 con 0x7f4b8409adf0 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b840cff90 0x7f4b840d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4b840cff90 0x7f4b840d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37012/0 (socket says 172.21.15.67:37012) 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 -- 172.21.15.67:0/981272886 learned_addr learned my addr 172.21.15.67:0/981272886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83400640 1 --2- 172.21.15.67:0/981272886 >> v2:172.21.15.86:3300/0 conn(0x7f4b8415dd60 0x7f4b84161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b7be00640 1 --2- 172.21.15.67:0/981272886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b8409adf0 0x7f4b84161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 -- 172.21.15.67:0/981272886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b8409adf0 msgr2=0x7f4b84161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 --2- 172.21.15.67:0/981272886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b8409adf0 0x7f4b84161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 -- 172.21.15.67:0/981272886 >> v2:172.21.15.86:3300/0 conn(0x7f4b8415dd60 msgr2=0x7f4b84161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 --2- 172.21.15.67:0/981272886 >> v2:172.21.15.86:3300/0 conn(0x7f4b8415dd60 0x7f4b84161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.039+0000 7f4b83e00640 1 -- 172.21.15.67:0/981272886 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b84099d50 con 0x7f4b840cff90 2024-04-17T10:25:52.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b83e00640 1 --2- 172.21.15.67:0/981272886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b840cff90 0x7f4b840d0420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4b8406f620 tx=0x7f4b7c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbf88c2705adb01 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b82a00640 1 -- 172.21.15.67:0/981272886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0a5860 con 0x7f4b840cff90 2024-04-17T10:25:52.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b82a00640 1 -- 172.21.15.67:0/981272886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0a5a00 con 0x7f4b840cff90 2024-04-17T10:25:52.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b82a00640 1 -- 172.21.15.67:0/981272886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0a5dd0 con 0x7f4b840cff90 2024-04-17T10:25:52.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 -- 172.21.15.67:0/981272886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b840cff90 msgr2=0x7f4b840d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 --2- 172.21.15.67:0/981272886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b840cff90 0x7f4b840d0420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4b8406f620 tx=0x7f4b7c01c9c0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 -- 172.21.15.67:0/981272886 shutdown_connections 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 --2- 172.21.15.67:0/981272886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b840cff90 0x7f4b840d0420 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 --2- 172.21.15.67:0/981272886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b8409adf0 0x7f4b84161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 --2- 172.21.15.67:0/981272886 >> v2:172.21.15.86:3300/0 conn(0x7f4b8415dd60 0x7f4b84161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 -- 172.21.15.67:0/981272886 >> 172.21.15.67:0/981272886 conn(0x7f4b840b34a0 msgr2=0x7f4b840b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 -- 172.21.15.67:0/981272886 shutdown_connections 2024-04-17T10:25:52.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.043+0000 7f4b8a200640 1 -- 172.21.15.67:0/981272886 wait complete. 2024-04-17T10:25:52.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 Processor -- start 2024-04-17T10:25:52.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 -- start start 2024-04-17T10:25:52.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b8409adf0 0x7f4b8415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b840cff90 0x7f4b840b6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8415dd60 0x7f4b840b6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8406c5b0 con 0x7f4b8415dd60 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8409a8d0 con 0x7f4b8409adf0 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8406c730 con 0x7f4b840cff90 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8415dd60 0x7f4b840b6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8415dd60 0x7f4b840b6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37020/0 (socket says 172.21.15.67:37020) 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 -- 172.21.15.67:0/1403698779 learned_addr learned my addr 172.21.15.67:0/1403698779 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:52.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83400640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b8409adf0 0x7f4b8415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b7be00640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b840cff90 0x7f4b840b6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 -- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b840cff90 msgr2=0x7f4b840b6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b840cff90 0x7f4b840b6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 -- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b8409adf0 msgr2=0x7f4b8415a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b8409adf0 0x7f4b8415a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 -- 172.21.15.67:0/1403698779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840c6210 con 0x7f4b8415dd60 2024-04-17T10:25:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b83e00640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8415dd60 0x7f4b840b6540 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0af040 tx=0x7f4b7c019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b7b400640 1 -- 172.21.15.67:0/1403698779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0b8880 con 0x7f4b8415dd60 2024-04-17T10:25:52.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b7b400640 1 -- 172.21.15.67:0/1403698779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c0be070 con 0x7f4b8415dd60 2024-04-17T10:25:52.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b7b400640 1 -- 172.21.15.67:0/1403698779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0b8c60 con 0x7f4b8415dd60 2024-04-17T10:25:52.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b840c64f0 con 0x7f4b8415dd60 2024-04-17T10:25:52.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.047+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b840c6ab0 con 0x7f4b8415dd60 2024-04-17T10:25:52.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.051+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8404ef30 con 0x7f4b8415dd60 2024-04-17T10:25:52.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.051+0000 7f4b7b400640 1 -- 172.21.15.67:0/1403698779 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4b7c0c3460 con 0x7f4b8415dd60 2024-04-17T10:25:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.051+0000 7f4b7b400640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b60066f60 0x7f4b60069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.051+0000 7f4b7b400640 1 -- 172.21.15.67:0/1403698779 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f4b7c120620 con 0x7f4b8415dd60 2024-04-17T10:25:52.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.051+0000 7f4b83400640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b60066f60 0x7f4b60069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.051+0000 7f4b83400640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b60066f60 0x7f4b60069420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b740079c0 tx=0x7f4b74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.055+0000 7f4b7b400640 1 -- 172.21.15.67:0/1403698779 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4b7c0165e0 con 0x7f4b8415dd60 2024-04-17T10:25:52.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4b840cfd50 con 0x7f4b8415dd60 2024-04-17T10:25:52.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b7b400640 1 -- 172.21.15.67:0/1403698779 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4b7c0f0b40 con 0x7f4b8415dd60 2024-04-17T10:25:52.214 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b60066f60 msgr2=0x7f4b60069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b60066f60 0x7f4b60069420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b740079c0 tx=0x7f4b74008040 comp rx=0 tx=0).stop 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8415dd60 msgr2=0x7f4b840b6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8415dd60 0x7f4b840b6540 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c0af040 tx=0x7f4b7c019760 comp rx=0 tx=0).stop 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 shutdown_connections 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b60066f60 0x7f4b60069420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4b8415dd60 0x7f4b840b6540 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4b840cff90 0x7f4b840b6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 --2- 172.21.15.67:0/1403698779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4b8409adf0 0x7f4b8415a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 >> 172.21.15.67:0/1403698779 conn(0x7f4b840b34a0 msgr2=0x7f4b84161f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 shutdown_connections 2024-04-17T10:25:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.219+0000 7f4b8a200640 1 -- 172.21.15.67:0/1403698779 wait complete. 2024-04-17T10:25:52.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:25:52.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1921200640 1 Processor -- start 2024-04-17T10:25:52.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1921200640 1 -- start start 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1921200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f191c0cf0c0 0x7f191c1441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1921200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c15ae40 con 0x7f191c06e6e0 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1921200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f191c071440 con 0x7f191c06eac0 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1921200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f191c071720 con 0x7f191c0cf0c0 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1919e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f191c06e6e0 0x7f191c0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33076/0 (socket says 172.21.15.67:33076) 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.383+0000 7f1919e00640 1 -- 172.21.15.67:0/2275232953 learned_addr learned my addr 172.21.15.67:0/2275232953 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f191a800640 1 --2- 172.21.15.67:0/2275232953 >> v2:172.21.15.86:3300/0 conn(0x7f191c0cf0c0 0x7f191c1441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f1919400640 1 -- 172.21.15.67:0/2275232953 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311149465 0 0) 0x7f191c15ae40 con 0x7f191c06e6e0 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f1919400640 1 -- 172.21.15.67:0/2275232953 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1900005740 con 0x7f191c06e6e0 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f191a800640 1 -- 172.21.15.67:0/2275232953 >> v1:172.21.15.86:6789/0 conn(0x7f191c06eac0 legacy=0x7f191c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f1919400640 1 -- 172.21.15.67:0/2275232953 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933521647 0 0) 0x7f191c071440 con 0x7f191c06eac0 2024-04-17T10:25:52.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f191a800640 1 -- 172.21.15.67:0/2275232953 >> v1:172.21.15.67:6789/0 conn(0x7f191c06e6e0 legacy=0x7f191c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f191a800640 1 -- 172.21.15.67:0/2275232953 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c071a00 con 0x7f191c0cf0c0 2024-04-17T10:25:52.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f191a800640 1 --2- 172.21.15.67:0/2275232953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 0x7f191c1441f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1914009d90 tx=0x7f191401c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=912ed3a4edd8bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f1919400640 1 -- 172.21.15.67:0/2275232953 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19140b2070 con 0x7f191c0cf0c0 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f1919400640 1 -- 172.21.15.67:0/2275232953 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19140a5870 con 0x7f191c0cf0c0 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.387+0000 7f1919400640 1 -- 172.21.15.67:0/2275232953 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19140a5c20 con 0x7f191c0cf0c0 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- 172.21.15.67:0/2275232953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 msgr2=0x7f191c1441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 --2- 172.21.15.67:0/2275232953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 0x7f191c1441f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f1914009d90 tx=0x7f191401c8b0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- 172.21.15.67:0/2275232953 shutdown_connections 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 --2- 172.21.15.67:0/2275232953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 0x7f191c1441f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- 172.21.15.67:0/2275232953 >> 172.21.15.67:0/2275232953 conn(0x7f191c06ac90 msgr2=0x7f191c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- 172.21.15.67:0/2275232953 shutdown_connections 2024-04-17T10:25:52.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- 172.21.15.67:0/2275232953 wait complete. 2024-04-17T10:25:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 Processor -- start 2024-04-17T10:25:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- start start 2024-04-17T10:25:52.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191c06e6e0 0x7f191c13eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191c06eac0 0x7f191c1f5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 0x7f191c1f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c06dd00 con 0x7f191c06e6e0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191c06db80 con 0x7f191c0cf0c0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1921200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191c06de80 con 0x7f191c06eac0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191c06e6e0 0x7f191c13eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191c06e6e0 0x7f191c13eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37022/0 (socket says 172.21.15.67:37022) 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 -- 172.21.15.67:0/2167049915 learned_addr learned my addr 172.21.15.67:0/2167049915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 -- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191c06eac0 msgr2=0x7f191c1f5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191c06eac0 0x7f191c1f5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 -- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 msgr2=0x7f191c1f5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 0x7f191c1f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.391+0000 7f1919e00640 1 -- 172.21.15.67:0/2167049915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f191c1417c0 con 0x7f191c06e6e0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1919e00640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191c06e6e0 0x7f191c13eed0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c450 tx=0x7f190c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1912000640 1 -- 172.21.15.67:0/2167049915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c017070 con 0x7f191c06e6e0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1912000640 1 -- 172.21.15.67:0/2167049915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c005cf0 con 0x7f191c06e6e0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1912000640 1 -- 172.21.15.67:0/2167049915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c007290 con 0x7f191c06e6e0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f191c141aa0 con 0x7f191c06e6e0 2024-04-17T10:25:52.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f191c142060 con 0x7f191c06e6e0 2024-04-17T10:25:52.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f191c13c300 con 0x7f191c06e6e0 2024-04-17T10:25:52.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1912000640 1 -- 172.21.15.67:0/2167049915 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f190c01a070 con 0x7f191c06e6e0 2024-04-17T10:25:52.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1912000640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18f8066f60 0x7f18f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.395+0000 7f1912000640 1 -- 172.21.15.67:0/2167049915 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f190c084880 con 0x7f191c06e6e0 2024-04-17T10:25:52.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.399+0000 7f1913e00640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18f8066f60 0x7f18f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.399+0000 7f1913e00640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18f8066f60 0x7f18f8069420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1904007fd0 tx=0x7f1904007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.399+0000 7f1912000640 1 -- 172.21.15.67:0/2167049915 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f190c0553f0 con 0x7f191c06e6e0 2024-04-17T10:25:52.488 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:25:52.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:25:52.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.575+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f191c074e30 con 0x7f191c06e6e0 2024-04-17T10:25:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.575+0000 7f1912000640 1 -- 172.21.15.67:0/2167049915 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f190c054cf0 con 0x7f191c06e6e0 2024-04-17T10:25:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18f8066f60 msgr2=0x7f18f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18f8066f60 0x7f18f8069420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1904007fd0 tx=0x7f1904007900 comp rx=0 tx=0).stop 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191c06e6e0 msgr2=0x7f191c13eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191c06e6e0 0x7f191c13eed0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f190c00c450 tx=0x7f190c00c910 comp rx=0 tx=0).stop 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 shutdown_connections 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18f8066f60 0x7f18f8069420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f191c0cf0c0 0x7f191c1f5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f191c06eac0 0x7f191c1f5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 --2- 172.21.15.67:0/2167049915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f191c06e6e0 0x7f191c13eed0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 >> 172.21.15.67:0/2167049915 conn(0x7f191c06ac90 msgr2=0x7f191c0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:52.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 shutdown_connections 2024-04-17T10:25:52.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.579+0000 7f1921200640 1 -- 172.21.15.67:0/2167049915 wait complete. 2024-04-17T10:25:52.592 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:25:52.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:52.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-17T10:25:52.671 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:25:52.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:52.671 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:25:52.724 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:52.725 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:52.734 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:52.734 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:52.784 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:52.785 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:52.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.987+0000 7f82d9e00640 1 Processor -- start 2024-04-17T10:25:52.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d9e00640 1 -- start start 2024-04-17T10:25:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d9e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82d4071770 0x7f82d40cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d9e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d406e2f0 con 0x7f82d4071310 2024-04-17T10:25:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d9e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d406d6e0 con 0x7f82d4145040 2024-04-17T10:25:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d9e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d40a86c0 con 0x7f82d4071770 2024-04-17T10:25:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82ca000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82d4071310 0x7f82d40cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33082/0 (socket says 172.21.15.67:33082) 2024-04-17T10:25:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82ca000640 1 -- 172.21.15.67:0/3392211271 learned_addr learned my addr 172.21.15.67:0/3392211271 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d3400640 1 --2- 172.21.15.67:0/3392211271 >> v2:172.21.15.86:3300/0 conn(0x7f82d4071770 0x7f82d40cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938801249 0 0) 0x7f82d406e2f0 con 0x7f82d4071310 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b8003530 con 0x7f82d4071310 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986949691 0 0) 0x7f82d406d6e0 con 0x7f82d4145040 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82d406e2f0 con 0x7f82d4145040 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126482371 0 0) 0x7f82b8003530 con 0x7f82d4071310 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 >> v2:172.21.15.86:3300/0 conn(0x7f82d4071770 msgr2=0x7f82d40cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 --2- 172.21.15.67:0/3392211271 >> v2:172.21.15.86:3300/0 conn(0x7f82d4071770 0x7f82d40cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 >> v1:172.21.15.86:6789/0 conn(0x7f82d4145040 legacy=0x7f82d406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d406d9c0 con 0x7f82d4071310 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f82bc001680 con 0x7f82d4071310 2024-04-17T10:25:52.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 >> v1:172.21.15.67:6789/0 conn(0x7f82d4071310 legacy=0x7f82d40cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8003b50 0x7f82b8003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b8004810 0x7f82b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b80051d0 0x7f82b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b800bfc0 con 0x7f82b8003b50 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b800c140 con 0x7f82b80051d0 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b800c340 con 0x7f82b8004810 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2a00640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8003b50 0x7f82b8003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d3400640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b80051d0 0x7f82b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82ca000640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b8004810 0x7f82b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2a00640 1 -- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b8004810 msgr2=0x7f82b8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2a00640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b8004810 0x7f82b8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2a00640 1 -- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b80051d0 msgr2=0x7f82b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2a00640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b80051d0 0x7f82b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d2a00640 1 -- 172.21.15.67:0/3392211271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8003530 con 0x7f82b8003b50 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d3400640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b80051d0 0x7f82b800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.991+0000 7f82d3400640 1 -- 172.21.15.67:0/3392211271 reap_dead start 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d2a00640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8003b50 0x7f82b8003f50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f82c4007fd0 tx=0x7f82c401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb362ef2b7b36d00 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c4001520 con 0x7f82b8003b50 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c40052f0 con 0x7f82b8003b50 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d2000640 1 -- 172.21.15.67:0/3392211271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c40a88d0 con 0x7f82b8003b50 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d9e00640 1 -- 172.21.15.67:0/3392211271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8003b50 msgr2=0x7f82b8003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d9e00640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8003b50 0x7f82b8003f50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f82c4007fd0 tx=0x7f82c401ec70 comp rx=0 tx=0).stop 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d9e00640 1 -- 172.21.15.67:0/3392211271 shutdown_connections 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d9e00640 1 --2- 172.21.15.67:0/3392211271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8003b50 0x7f82b8003f50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d9e00640 1 -- 172.21.15.67:0/3392211271 >> 172.21.15.67:0/3392211271 conn(0x7f82d406ac90 msgr2=0x7f82d406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:52.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d9e00640 1 -- 172.21.15.67:0/3392211271 shutdown_connections 2024-04-17T10:25:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.995+0000 7f82d9e00640 1 -- 172.21.15.67:0/3392211271 wait complete. 2024-04-17T10:25:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 Processor -- start 2024-04-17T10:25:52.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 -- start start 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b8003b50 0x7f82d415b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d40a9710 0x7f82d4142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d40af3c0 0x7f82d40af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d406df80 con 0x7f82d40af3c0 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d415bf10 con 0x7f82b8003b50 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d4071b80 con 0x7f82d40a9710 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d40af3c0 0x7f82d40af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d40af3c0 0x7f82d40af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37048/0 (socket says 172.21.15.67:37048) 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 -- 172.21.15.67:0/799915924 learned_addr learned my addr 172.21.15.67:0/799915924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d2a00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b8003b50 0x7f82d415b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82ca000640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d40a9710 0x7f82d4142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 -- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d40a9710 msgr2=0x7f82d4142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d40a9710 0x7f82d4142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 -- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b8003b50 msgr2=0x7f82d415b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b8003b50 0x7f82d415b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 -- 172.21.15.67:0/799915924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d41435f0 con 0x7f82d40af3c0 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82ca000640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d40a9710 0x7f82d4142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d3400640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d40af3c0 0x7f82d40af840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f82d4071ff0 tx=0x7f82cc010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82cbe00640 1 -- 172.21.15.67:0/799915924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc010dd0 con 0x7f82d40af3c0 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82cbe00640 1 -- 172.21.15.67:0/799915924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82cc0039a0 con 0x7f82d40af3c0 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82cbe00640 1 -- 172.21.15.67:0/799915924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc003d90 con 0x7f82d40af3c0 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d41438d0 con 0x7f82d40af3c0 2024-04-17T10:25:52.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d4143e90 con 0x7f82d40af3c0 2024-04-17T10:25:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:52.999+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d4072d50 con 0x7f82d40af3c0 2024-04-17T10:25:52.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.003+0000 7f82cbe00640 1 -- 172.21.15.67:0/799915924 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f82cc01e020 con 0x7f82d40af3c0 2024-04-17T10:25:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.003+0000 7f82cbe00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f82b0066f60 0x7f82b0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.003+0000 7f82cbe00640 1 -- 172.21.15.67:0/799915924 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f82cc085b70 con 0x7f82d40af3c0 2024-04-17T10:25:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.003+0000 7f82d2a00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f82b0066f60 0x7f82b0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.003+0000 7f82d2a00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f82b0066f60 0x7f82b0069420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f82c4007fd0 tx=0x7f82c40aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:53.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.007+0000 7f82cbe00640 1 -- 172.21.15.67:0/799915924 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f82cc0566e0 con 0x7f82d40af3c0 2024-04-17T10:25:53.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.167+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82d4074e30 con 0x7f82d40af3c0 2024-04-17T10:25:53.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.167+0000 7f82cbe00640 1 -- 172.21.15.67:0/799915924 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f82cc055fe0 con 0x7f82d40af3c0 2024-04-17T10:25:53.164 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:53.164 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f82b0066f60 msgr2=0x7f82b0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f82b0066f60 0x7f82b0069420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f82c4007fd0 tx=0x7f82c40aa070 comp rx=0 tx=0).stop 2024-04-17T10:25:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d40af3c0 msgr2=0x7f82d40af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d40af3c0 0x7f82d40af840 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f82d4071ff0 tx=0x7f82cc010590 comp rx=0 tx=0).stop 2024-04-17T10:25:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 shutdown_connections 2024-04-17T10:25:53.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f82b0066f60 0x7f82b0069420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82d40af3c0 0x7f82d40af840 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82d40a9710 0x7f82d4142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 --2- 172.21.15.67:0/799915924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b8003b50 0x7f82d415b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 >> 172.21.15.67:0/799915924 conn(0x7f82d406ac90 msgr2=0x7f82d40b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 shutdown_connections 2024-04-17T10:25:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.171+0000 7f82d9e00640 1 -- 172.21.15.67:0/799915924 wait complete. 2024-04-17T10:25:53.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:53.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3de6200640 1 Processor -- start 2024-04-17T10:25:53.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3de6200640 1 -- start start 2024-04-17T10:25:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3de6200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3de00d06a0 0x7f3de006bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3de6200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de015e140 con 0x7f3de015dd60 2024-04-17T10:25:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3de6200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de0099d50 con 0x7f3de0161f90 2024-04-17T10:25:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3de6200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3de01626d0 con 0x7f3de00d06a0 2024-04-17T10:25:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3ddf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3de015dd60 0x7f3de0161880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33092/0 (socket says 172.21.15.67:33092) 2024-04-17T10:25:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3ddf400640 1 -- 172.21.15.67:0/3252035299 learned_addr learned my addr 172.21.15.67:0/3252035299 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:53.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.335+0000 7f3ddfe00640 1 --2- 172.21.15.67:0/3252035299 >> v2:172.21.15.86:3300/0 conn(0x7f3de00d06a0 0x7f3de006bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddea00640 1 -- 172.21.15.67:0/3252035299 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345512745 0 0) 0x7f3de015e140 con 0x7f3de015dd60 2024-04-17T10:25:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddea00640 1 -- 172.21.15.67:0/3252035299 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc4003530 con 0x7f3de015dd60 2024-04-17T10:25:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddfe00640 1 -- 172.21.15.67:0/3252035299 >> v1:172.21.15.86:6789/0 conn(0x7f3de0161f90 legacy=0x7f3de00cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddfe00640 1 -- 172.21.15.67:0/3252035299 >> v1:172.21.15.67:6789/0 conn(0x7f3de015dd60 legacy=0x7f3de0161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddfe00640 1 -- 172.21.15.67:0/3252035299 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de009a030 con 0x7f3de00d06a0 2024-04-17T10:25:53.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddfe00640 1 --2- 172.21.15.67:0/3252035299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de00d06a0 0x7f3de006bda0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009d90 tx=0x7f3dd801c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7e8f03bbc47a3b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:53.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddea00640 1 -- 172.21.15.67:0/3252035299 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80b2070 con 0x7f3de00d06a0 2024-04-17T10:25:53.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddea00640 1 -- 172.21.15.67:0/3252035299 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd80a5a00 con 0x7f3de00d06a0 2024-04-17T10:25:53.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.339+0000 7f3ddea00640 1 -- 172.21.15.67:0/3252035299 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80a5db0 con 0x7f3de00d06a0 2024-04-17T10:25:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- 172.21.15.67:0/3252035299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de00d06a0 msgr2=0x7f3de006bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 --2- 172.21.15.67:0/3252035299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de00d06a0 0x7f3de006bda0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009d90 tx=0x7f3dd801c940 comp rx=0 tx=0).stop 2024-04-17T10:25:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- 172.21.15.67:0/3252035299 shutdown_connections 2024-04-17T10:25:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 --2- 172.21.15.67:0/3252035299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de00d06a0 0x7f3de006bda0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- 172.21.15.67:0/3252035299 >> 172.21.15.67:0/3252035299 conn(0x7f3de00b34a0 msgr2=0x7f3de00b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- 172.21.15.67:0/3252035299 shutdown_connections 2024-04-17T10:25:53.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- 172.21.15.67:0/3252035299 wait complete. 2024-04-17T10:25:53.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 Processor -- start 2024-04-17T10:25:53.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- start start 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de00d06a0 0x7f3de00c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de015dd60 0x7f3de00c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3de0161f90 0x7f3de01577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de016bd90 con 0x7f3de0161f90 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de016bc10 con 0x7f3de015dd60 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3de6200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3de016bf10 con 0x7f3de00d06a0 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3de0161f90 0x7f3de01577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3de0161f90 0x7f3de01577c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37050/0 (socket says 172.21.15.67:37050) 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 -- 172.21.15.67:0/2913886822 learned_addr learned my addr 172.21.15.67:0/2913886822 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddf400640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de00d06a0 0x7f3de00c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3dd7e00640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de015dd60 0x7f3de00c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 -- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de00d06a0 msgr2=0x7f3de00c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de00d06a0 0x7f3de00c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 -- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de015dd60 msgr2=0x7f3de00c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de015dd60 0x7f3de00c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.343+0000 7f3ddfe00640 1 -- 172.21.15.67:0/2913886822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0156a00 con 0x7f3de0161f90 2024-04-17T10:25:53.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3ddfe00640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3de0161f90 0x7f3de01577c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009d90 tx=0x7f3dd80196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:53.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3dd7400640 1 -- 172.21.15.67:0/2913886822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80b2050 con 0x7f3de0161f90 2024-04-17T10:25:53.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3dd7400640 1 -- 172.21.15.67:0/2913886822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dd80be070 con 0x7f3de0161f90 2024-04-17T10:25:53.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3dd7400640 1 -- 172.21.15.67:0/2913886822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8004260 con 0x7f3de0161f90 2024-04-17T10:25:53.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de0156ce0 con 0x7f3de0161f90 2024-04-17T10:25:53.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de01572a0 con 0x7f3de0161f90 2024-04-17T10:25:53.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de004ef30 con 0x7f3de0161f90 2024-04-17T10:25:53.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3dd7400640 1 -- 172.21.15.67:0/2913886822 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3dd8004400 con 0x7f3de0161f90 2024-04-17T10:25:53.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3dd7400640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3dbc066f60 0x7f3dbc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3dd7400640 1 -- 172.21.15.67:0/2913886822 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f3dd8120650 con 0x7f3de0161f90 2024-04-17T10:25:53.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.347+0000 7f3ddf400640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3dbc066f60 0x7f3dbc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.351+0000 7f3ddf400640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3dbc066f60 0x7f3dbc069420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00011f0 tx=0x7f3dd0007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:53.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.351+0000 7f3dd7400640 1 -- 172.21.15.67:0/2913886822 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3dd8016560 con 0x7f3de0161f90 2024-04-17T10:25:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.515+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3de00b3b30 con 0x7f3de0161f90 2024-04-17T10:25:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.515+0000 7f3dd7400640 1 -- 172.21.15.67:0/2913886822 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3dd80f0ae0 con 0x7f3de0161f90 2024-04-17T10:25:53.511 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3dbc066f60 msgr2=0x7f3dbc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3dbc066f60 0x7f3dbc069420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3dd00011f0 tx=0x7f3dd0007d40 comp rx=0 tx=0).stop 2024-04-17T10:25:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3de0161f90 msgr2=0x7f3de01577c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3de0161f90 0x7f3de01577c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8009d90 tx=0x7f3dd80196e0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 shutdown_connections 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3dbc066f60 0x7f3dbc069420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3de0161f90 0x7f3de01577c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3de015dd60 0x7f3de00c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 --2- 172.21.15.67:0/2913886822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3de00d06a0 0x7f3de00c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 >> 172.21.15.67:0/2913886822 conn(0x7f3de00b34a0 msgr2=0x7f3de00ce430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 shutdown_connections 2024-04-17T10:25:53.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.519+0000 7f3de6200640 1 -- 172.21.15.67:0/2913886822 wait complete. 2024-04-17T10:25:53.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:53.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.679+0000 7f8170600640 1 Processor -- start 2024-04-17T10:25:53.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f8170600640 1 -- start start 2024-04-17T10:25:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f8170600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f81681583f0 0x7f8168072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f8170600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81680b5300 0x7f816806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f8170600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816815f360 con 0x7f81680728c0 2024-04-17T10:25:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f8170600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81680b3160 con 0x7f81680b5300 2024-04-17T10:25:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f8170600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81680b3440 con 0x7f81681583f0 2024-04-17T10:25:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81680b5300 0x7f816806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f81680b5300 0x7f816806ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37056/0 (socket says 172.21.15.67:37056) 2024-04-17T10:25:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816e600640 1 -- 172.21.15.67:0/2422766474 learned_addr learned my addr 172.21.15.67:0/2422766474 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816dc00640 1 --2- 172.21.15.67:0/2422766474 >> v2:172.21.15.86:3300/0 conn(0x7f81681583f0 0x7f8168072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816e600640 1 -- 172.21.15.67:0/2422766474 >> v2:172.21.15.86:3300/0 conn(0x7f81681583f0 msgr2=0x7f8168072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816e600640 1 --2- 172.21.15.67:0/2422766474 >> v2:172.21.15.86:3300/0 conn(0x7f81681583f0 0x7f8168072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816e600640 1 -- 172.21.15.67:0/2422766474 >> v1:172.21.15.67:6789/0 conn(0x7f81680728c0 legacy=0x7f81680b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.683+0000 7f816e600640 1 -- 172.21.15.67:0/2422766474 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680b3720 con 0x7f81680b5300 2024-04-17T10:25:53.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f816e600640 1 --2- 172.21.15.67:0/2422766474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816806ac90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8168066fe0 tx=0x7f816001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a9dd6cf76c8a53 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:53.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8167e00640 1 -- 172.21.15.67:0/2422766474 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81600a57a0 con 0x7f81680b5300 2024-04-17T10:25:53.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8167e00640 1 -- 172.21.15.67:0/2422766474 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81600a5940 con 0x7f81680b5300 2024-04-17T10:25:53.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8167e00640 1 -- 172.21.15.67:0/2422766474 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81600a5d10 con 0x7f81680b5300 2024-04-17T10:25:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 -- 172.21.15.67:0/2422766474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 msgr2=0x7f816806ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 --2- 172.21.15.67:0/2422766474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816806ac90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8168066fe0 tx=0x7f816001c9f0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 -- 172.21.15.67:0/2422766474 shutdown_connections 2024-04-17T10:25:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 --2- 172.21.15.67:0/2422766474 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816806ac90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 --2- 172.21.15.67:0/2422766474 >> v2:172.21.15.86:3300/0 conn(0x7f81681583f0 0x7f8168072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 -- 172.21.15.67:0/2422766474 >> 172.21.15.67:0/2422766474 conn(0x7f81680b2570 msgr2=0x7f8168071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:53.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 -- 172.21.15.67:0/2422766474 shutdown_connections 2024-04-17T10:25:53.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 -- 172.21.15.67:0/2422766474 wait complete. 2024-04-17T10:25:53.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 Processor -- start 2024-04-17T10:25:53.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.687+0000 7f8170600640 1 -- start start 2024-04-17T10:25:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81680728c0 0x7f8168149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816815f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81681583f0 0x7f8168160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816806b9e0 con 0x7f81680b5300 2024-04-17T10:25:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81680b4200 con 0x7f81680728c0 2024-04-17T10:25:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816806bb60 con 0x7f81681583f0 2024-04-17T10:25:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816815f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816815f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37068/0 (socket says 172.21.15.67:37068) 2024-04-17T10:25:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 -- 172.21.15.67:0/1409742827 learned_addr learned my addr 172.21.15.67:0/1409742827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816dc00640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81680728c0 0x7f8168149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816e600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81681583f0 0x7f8168160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 -- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81681583f0 msgr2=0x7f8168160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81681583f0 0x7f8168160050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 -- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81680728c0 msgr2=0x7f8168149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81680728c0 0x7f8168149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 -- 172.21.15.67:0/1409742827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168148890 con 0x7f81680b5300 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f816d200640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816815f530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f815400c370 tx=0x7f815400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8165600640 1 -- 172.21.15.67:0/1409742827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81540052d0 con 0x7f81680b5300 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8165600640 1 -- 172.21.15.67:0/1409742827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8154005470 con 0x7f81680b5300 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8165600640 1 -- 172.21.15.67:0/1409742827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154007260 con 0x7f81680b5300 2024-04-17T10:25:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8168148b70 con 0x7f81680b5300 2024-04-17T10:25:53.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8168149130 con 0x7f81680b5300 2024-04-17T10:25:53.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.691+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8168149330 con 0x7f81680b5300 2024-04-17T10:25:53.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.695+0000 7f8165600640 1 -- 172.21.15.67:0/1409742827 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8154007400 con 0x7f81680b5300 2024-04-17T10:25:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.695+0000 7f8165600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8148066f60 0x7f8148069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.695+0000 7f8165600640 1 -- 172.21.15.67:0/1409742827 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f8154084810 con 0x7f81680b5300 2024-04-17T10:25:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.695+0000 7f816dc00640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8148066f60 0x7f8148069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.695+0000 7f816dc00640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8148066f60 0x7f8148069420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f815c009b60 tx=0x7f815c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:53.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.699+0000 7f8165600640 1 -- 172.21.15.67:0/1409742827 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8154054dd0 con 0x7f81680b5300 2024-04-17T10:25:53.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.859+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f816804ef30 con 0x7f81680b5300 2024-04-17T10:25:53.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.859+0000 7f8165600640 1 -- 172.21.15.67:0/1409742827 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8154058e70 con 0x7f81680b5300 2024-04-17T10:25:53.856 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8148066f60 msgr2=0x7f8148069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8148066f60 0x7f8148069420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f815c009b60 tx=0x7f815c009190 comp rx=0 tx=0).stop 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 msgr2=0x7f816815f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816815f530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f815400c370 tx=0x7f815400c830 comp rx=0 tx=0).stop 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 shutdown_connections 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8148066f60 0x7f8148069420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f81681583f0 0x7f8168160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f81680b5300 0x7f816815f530 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 --2- 172.21.15.67:0/1409742827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f81680728c0 0x7f8168149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 >> 172.21.15.67:0/1409742827 conn(0x7f81680b2570 msgr2=0x7f8168071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 shutdown_connections 2024-04-17T10:25:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:53.863+0000 7f8170600640 1 -- 172.21.15.67:0/1409742827 wait complete. 2024-04-17T10:25:53.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 Processor -- start 2024-04-17T10:25:54.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 -- start start 2024-04-17T10:25:54.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa05c159440 0x7fa05c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa05c15be70 0x7fa05c15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c0a8650 con 0x7fa05c159440 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c071d80 con 0x7fa05c15be70 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa063000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c15ce10 con 0x7fa05c0713e0 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa05c159440 0x7fa05c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa05c159440 0x7fa05c0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37078/0 (socket says 172.21.15.67:37078) 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 -- 172.21.15.67:0/1105129418 learned_addr learned my addr 172.21.15.67:0/1105129418 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa053400640 1 --2- 172.21.15.67:0/1105129418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa061000640 1 --2- 172.21.15.67:0/1105129418 >> v2:172.21.15.86:3300/0 conn(0x7fa05c15be70 0x7fa05c15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 -- 172.21.15.67:0/1105129418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 msgr2=0x7fa05c071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 --2- 172.21.15.67:0/1105129418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 -- 172.21.15.67:0/1105129418 >> v2:172.21.15.86:3300/0 conn(0x7fa05c15be70 msgr2=0x7fa05c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 --2- 172.21.15.67:0/1105129418 >> v2:172.21.15.86:3300/0 conn(0x7fa05c15be70 0x7fa05c15c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa05be00640 1 -- 172.21.15.67:0/1105129418 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c15cb30 con 0x7fa05c159440 2024-04-17T10:25:54.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa053400640 1 --2- 172.21.15.67:0/1105129418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:54.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.027+0000 7fa061000640 1 --2- 172.21.15.67:0/1105129418 >> v2:172.21.15.86:3300/0 conn(0x7fa05c15be70 0x7fa05c15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa05be00640 1 --2- 172.21.15.67:0/1105129418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c159440 0x7fa05c0ce0f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0726f0 tx=0x7fa04c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e955563e1dd06966 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa05b400640 1 -- 172.21.15.67:0/1105129418 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0a5870 con 0x7fa05c159440 2024-04-17T10:25:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa05b400640 1 -- 172.21.15.67:0/1105129418 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c0a5a10 con 0x7fa05c159440 2024-04-17T10:25:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa05b400640 1 -- 172.21.15.67:0/1105129418 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0a5de0 con 0x7fa05c159440 2024-04-17T10:25:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 -- 172.21.15.67:0/1105129418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c159440 msgr2=0x7fa05c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 --2- 172.21.15.67:0/1105129418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c159440 0x7fa05c0ce0f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0726f0 tx=0x7fa04c01c9f0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 -- 172.21.15.67:0/1105129418 shutdown_connections 2024-04-17T10:25:54.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 --2- 172.21.15.67:0/1105129418 >> v2:172.21.15.86:3300/0 conn(0x7fa05c15be70 0x7fa05c15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 --2- 172.21.15.67:0/1105129418 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 --2- 172.21.15.67:0/1105129418 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c159440 0x7fa05c0ce0f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 -- 172.21.15.67:0/1105129418 >> 172.21.15.67:0/1105129418 conn(0x7fa05c0b1740 msgr2=0x7fa05c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 -- 172.21.15.67:0/1105129418 shutdown_connections 2024-04-17T10:25:54.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 -- 172.21.15.67:0/1105129418 wait complete. 2024-04-17T10:25:54.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 Processor -- start 2024-04-17T10:25:54.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.031+0000 7fa063000640 1 -- start start 2024-04-17T10:25:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa05c159440 0x7fa05c1f9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c15be70 0x7fa05c1fa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c15c7e0 con 0x7fa05c15be70 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c159820 con 0x7fa05c159440 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c15d840 con 0x7fa05c0713e0 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c15be70 0x7fa05c1fa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c15be70 0x7fa05c1fa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37080/0 (socket says 172.21.15.67:37080) 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 -- 172.21.15.67:0/3625561405 learned_addr learned my addr 172.21.15.67:0/3625561405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa053400640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa05c159440 0x7fa05c1f9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa05be00640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 -- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 msgr2=0x7fa05c159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 -- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa05c159440 msgr2=0x7fa05c1f9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa05c159440 0x7fa05c1f9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 -- 172.21.15.67:0/3625561405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c0b0b60 con 0x7fa05c15be70 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa061000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c15be70 0x7fa05c1fa230 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa05400d800 tx=0x7fa05400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa058c00640 1 -- 172.21.15.67:0/3625561405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400b6c0 con 0x7fa05c15be70 2024-04-17T10:25:54.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa058c00640 1 -- 172.21.15.67:0/3625561405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa054019070 con 0x7fa05c15be70 2024-04-17T10:25:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa058c00640 1 -- 172.21.15.67:0/3625561405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05400baa0 con 0x7fa05c15be70 2024-04-17T10:25:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c0affe0 con 0x7fa05c15be70 2024-04-17T10:25:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c0b05a0 con 0x7fa05c15be70 2024-04-17T10:25:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.035+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c04ef30 con 0x7fa05c15be70 2024-04-17T10:25:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.039+0000 7fa058c00640 1 -- 172.21.15.67:0/3625561405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa05400bc40 con 0x7fa05c15be70 2024-04-17T10:25:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.039+0000 7fa058c00640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa038066f60 0x7fa038069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.039+0000 7fa058c00640 1 -- 172.21.15.67:0/3625561405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fa054085810 con 0x7fa05c15be70 2024-04-17T10:25:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.039+0000 7fa05be00640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa038066f60 0x7fa038069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.039+0000 7fa05be00640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa038066f60 0x7fa038069420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0ae000 tx=0x7fa04c0a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.043+0000 7fa058c00640 1 -- 172.21.15.67:0/3625561405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa054055d30 con 0x7fa05c15be70 2024-04-17T10:25:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.203+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa05c158ad0 con 0x7fa05c15be70 2024-04-17T10:25:54.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.203+0000 7fa058c00640 1 -- 172.21.15.67:0/3625561405 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa054059dd0 con 0x7fa05c15be70 2024-04-17T10:25:54.200 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:54.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa038066f60 msgr2=0x7fa038069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa038066f60 0x7fa038069420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0ae000 tx=0x7fa04c0a7070 comp rx=0 tx=0).stop 2024-04-17T10:25:54.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c15be70 msgr2=0x7fa05c1fa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c15be70 0x7fa05c1fa230 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa05400d800 tx=0x7fa05400dcc0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 shutdown_connections 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa038066f60 0x7fa038069420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa05c15be70 0x7fa05c1fa230 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa05c159440 0x7fa05c1f9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 --2- 172.21.15.67:0/3625561405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa05c0713e0 0x7fa05c159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 >> 172.21.15.67:0/3625561405 conn(0x7fa05c0b1740 msgr2=0x7fa05c0ad790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 shutdown_connections 2024-04-17T10:25:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.207+0000 7fa063000640 1 -- 172.21.15.67:0/3625561405 wait complete. 2024-04-17T10:25:54.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6303200640 1 Processor -- start 2024-04-17T10:25:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6303200640 1 -- start start 2024-04-17T10:25:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6303200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63040aed70 0x7f630415f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6303200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f630415f860 0x7f630415e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6303200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630415cf40 con 0x7f630415b950 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6303200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630415dde0 con 0x7f630415f860 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6303200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040af190 con 0x7f63040aed70 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6302800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f630415f860 0x7f630415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6302800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f630415f860 0x7f630415e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37082/0 (socket says 172.21.15.67:37082) 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6302800640 1 -- 172.21.15.67:0/1212417013 learned_addr learned my addr 172.21.15.67:0/1212417013 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f62f9400640 1 --2- 172.21.15.67:0/1212417013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63040aed70 0x7f630415f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6302800640 1 -- 172.21.15.67:0/1212417013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63040aed70 msgr2=0x7f630415f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6302800640 1 --2- 172.21.15.67:0/1212417013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63040aed70 0x7f630415f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6302800640 1 -- 172.21.15.67:0/1212417013 >> v1:172.21.15.86:6789/0 conn(0x7f630415b950 legacy=0x7f630415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.371+0000 7f6302800640 1 -- 172.21.15.67:0/1212417013 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630415d1d0 con 0x7f630415f860 2024-04-17T10:25:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6302800640 1 --2- 172.21.15.67:0/1212417013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f860 0x7f630415e270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009870 tx=0x7f62fc01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3097dd4dc5b1e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6301400640 1 -- 172.21.15.67:0/1212417013 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0ab650 con 0x7f630415f860 2024-04-17T10:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6301400640 1 -- 172.21.15.67:0/1212417013 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc0b0070 con 0x7f630415f860 2024-04-17T10:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6301400640 1 -- 172.21.15.67:0/1212417013 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0aba10 con 0x7f630415f860 2024-04-17T10:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 -- 172.21.15.67:0/1212417013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f860 msgr2=0x7f630415e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 --2- 172.21.15.67:0/1212417013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f860 0x7f630415e270 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009870 tx=0x7f62fc01c560 comp rx=0 tx=0).stop 2024-04-17T10:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 -- 172.21.15.67:0/1212417013 shutdown_connections 2024-04-17T10:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 --2- 172.21.15.67:0/1212417013 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f860 0x7f630415e270 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 --2- 172.21.15.67:0/1212417013 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63040aed70 0x7f630415f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 -- 172.21.15.67:0/1212417013 >> 172.21.15.67:0/1212417013 conn(0x7f6304072a40 msgr2=0x7f6304099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 -- 172.21.15.67:0/1212417013 shutdown_connections 2024-04-17T10:25:54.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.375+0000 7f6303200640 1 -- 172.21.15.67:0/1212417013 wait complete. 2024-04-17T10:25:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 Processor -- start 2024-04-17T10:25:54.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 -- start start 2024-04-17T10:25:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63040aed70 0x7f6304146bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415b950 0x7f630414a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630415f860 0x7f63040b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304160120 con 0x7f63040aed70 2024-04-17T10:25:54.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630415ffa0 con 0x7f630415f860 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6304168660 con 0x7f630415b950 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63040aed70 0x7f6304146bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63040aed70 0x7f6304146bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37084/0 (socket says 172.21.15.67:37084) 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 -- 172.21.15.67:0/600976920 learned_addr learned my addr 172.21.15.67:0/600976920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6302800640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630415f860 0x7f63040b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f62f9400640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415b950 0x7f630414a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 -- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415b950 msgr2=0x7f630414a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415b950 0x7f630414a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 -- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630415f860 msgr2=0x7f63040b14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630415f860 0x7f63040b14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 -- 172.21.15.67:0/600976920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304148460 con 0x7f63040aed70 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6302800640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630415f860 0x7f63040b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:54.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6301e00640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63040aed70 0x7f6304146bb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f62f400c790 tx=0x7f62f400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f62faa00640 1 -- 172.21.15.67:0/600976920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4016020 con 0x7f63040aed70 2024-04-17T10:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f62faa00640 1 -- 172.21.15.67:0/600976920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4006d60 con 0x7f63040aed70 2024-04-17T10:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6304148740 con 0x7f63040aed70 2024-04-17T10:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f62faa00640 1 -- 172.21.15.67:0/600976920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4003bc0 con 0x7f63040aed70 2024-04-17T10:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.379+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6304148d00 con 0x7f63040aed70 2024-04-17T10:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.383+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630404f780 con 0x7f63040aed70 2024-04-17T10:25:54.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.383+0000 7f62faa00640 1 -- 172.21.15.67:0/600976920 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f62f400a850 con 0x7f63040aed70 2024-04-17T10:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.383+0000 7f62faa00640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e0066f60 0x7f62e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.383+0000 7f62faa00640 1 -- 172.21.15.67:0/600976920 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f62f4085180 con 0x7f63040aed70 2024-04-17T10:25:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.383+0000 7f62f9400640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e0066f60 0x7f62e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.383+0000 7f62f9400640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e0066f60 0x7f62e0069420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6304149a70 tx=0x7f62ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.387+0000 7f62faa00640 1 -- 172.21.15.67:0/600976920 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f62f4020020 con 0x7f63040aed70 2024-04-17T10:25:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.547+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f63040b5bc0 con 0x7f63040aed70 2024-04-17T10:25:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.547+0000 7f62faa00640 1 -- 172.21.15.67:0/600976920 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f62f4055d80 con 0x7f63040aed70 2024-04-17T10:25:54.544 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:54.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e0066f60 msgr2=0x7f62e0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e0066f60 0x7f62e0069420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6304149a70 tx=0x7f62ec008040 comp rx=0 tx=0).stop 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63040aed70 msgr2=0x7f6304146bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63040aed70 0x7f6304146bb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f62f400c790 tx=0x7f62f400cc50 comp rx=0 tx=0).stop 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 shutdown_connections 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e0066f60 0x7f62e0069420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f630415f860 0x7f63040b14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415b950 0x7f630414a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 --2- 172.21.15.67:0/600976920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63040aed70 0x7f6304146bb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 >> 172.21.15.67:0/600976920 conn(0x7f6304072a40 msgr2=0x7f63040b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 shutdown_connections 2024-04-17T10:25:54.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.551+0000 7f6303200640 1 -- 172.21.15.67:0/600976920 wait complete. 2024-04-17T10:25:54.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:25:54.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.711+0000 7ffbde400640 1 Processor -- start 2024-04-17T10:25:54.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.711+0000 7ffbde400640 1 -- start start 2024-04-17T10:25:54.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbde400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffbd81466e0 0x7ffbd806eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbde400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd80adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbde400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd8161740 con 0x7ffbd8146da0 2024-04-17T10:25:54.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbde400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd8161a20 con 0x7ffbd81466e0 2024-04-17T10:25:54.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbde400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd8161d00 con 0x7ffbd8071a20 2024-04-17T10:25:54.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffbd81466e0 0x7ffbd806eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd80adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffbd81466e0 0x7ffbd806eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:45692/0 (socket says 172.21.15.67:45692) 2024-04-17T10:25:54.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7400640 1 -- 172.21.15.67:0/3461872508 learned_addr learned my addr 172.21.15.67:0/3461872508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:54.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbcfe00640 1 --1- 172.21.15.67:0/3461872508 >> v1:172.21.15.86:6789/0 conn(0x7ffbd8146da0 0x7ffbd8071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:41804/0 (socket says 172.21.15.67:41804) 2024-04-17T10:25:54.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7400640 1 -- 172.21.15.67:0/3461872508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 msgr2=0x7ffbd80adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:54.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7400640 1 --2- 172.21.15.67:0/3461872508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd80adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7400640 1 -- 172.21.15.67:0/3461872508 >> v1:172.21.15.86:6789/0 conn(0x7ffbd8146da0 legacy=0x7ffbd8071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.715+0000 7ffbd7400640 1 -- 172.21.15.67:0/3461872508 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd8161fe0 con 0x7ffbd81466e0 2024-04-17T10:25:54.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbd7400640 1 --2- 172.21.15.67:0/3461872508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd81466e0 0x7ffbd806eb70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009870 tx=0x7ffbc801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1658766bba2bd7cb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbd6a00640 1 -- 172.21.15.67:0/3461872508 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc80b0070 con 0x7ffbd81466e0 2024-04-17T10:25:54.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbd6a00640 1 -- 172.21.15.67:0/3461872508 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc8004230 con 0x7ffbd81466e0 2024-04-17T10:25:54.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbd6a00640 1 -- 172.21.15.67:0/3461872508 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8003620 con 0x7ffbd81466e0 2024-04-17T10:25:54.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 -- 172.21.15.67:0/3461872508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd81466e0 msgr2=0x7ffbd806eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 --2- 172.21.15.67:0/3461872508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd81466e0 0x7ffbd806eb70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009870 tx=0x7ffbc801c3f0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 -- 172.21.15.67:0/3461872508 shutdown_connections 2024-04-17T10:25:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 --2- 172.21.15.67:0/3461872508 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd80adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 --2- 172.21.15.67:0/3461872508 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd81466e0 0x7ffbd806eb70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 -- 172.21.15.67:0/3461872508 >> 172.21.15.67:0/3461872508 conn(0x7ffbd806ac90 msgr2=0x7ffbd806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 -- 172.21.15.67:0/3461872508 shutdown_connections 2024-04-17T10:25:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 -- 172.21.15.67:0/3461872508 wait complete. 2024-04-17T10:25:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.719+0000 7ffbde400640 1 Processor -- start 2024-04-17T10:25:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 -- start start 2024-04-17T10:25:54.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd815d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd81466e0 0x7ffbd813d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8146da0 0x7ffbd8160430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd806f230 con 0x7ffbd81466e0 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd806f0b0 con 0x7ffbd8146da0 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd80ae320 con 0x7ffbd8071a20 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd81466e0 0x7ffbd813d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd81466e0 0x7ffbd813d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37100/0 (socket says 172.21.15.67:37100) 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 -- 172.21.15.67:0/1585280755 learned_addr learned my addr 172.21.15.67:0/1585280755 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbd7400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd815d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbd7e00640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8146da0 0x7ffbd8160430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 -- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 msgr2=0x7ffbd815d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd815d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 -- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8146da0 msgr2=0x7ffbd8160430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8146da0 0x7ffbd8160430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 -- 172.21.15.67:0/1585280755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd813dba0 con 0x7ffbd81466e0 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcfe00640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd81466e0 0x7ffbd813d250 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009e30 tx=0x7ffbc000c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcf400640 1 -- 172.21.15.67:0/1585280755 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0019070 con 0x7ffbd81466e0 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcf400640 1 -- 172.21.15.67:0/1585280755 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc0007830 con 0x7ffbd81466e0 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd813de80 con 0x7ffbd81466e0 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbcf400640 1 -- 172.21.15.67:0/1585280755 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0007c40 con 0x7ffbd81466e0 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd813e440 con 0x7ffbd81466e0 2024-04-17T10:25:54.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.723+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd8072d50 con 0x7ffbd81466e0 2024-04-17T10:25:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.727+0000 7ffbcf400640 1 -- 172.21.15.67:0/1585280755 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ffbc001c070 con 0x7ffbd81466e0 2024-04-17T10:25:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.727+0000 7ffbcf400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffbb4066f60 0x7ffbb4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.727+0000 7ffbcf400640 1 -- 172.21.15.67:0/1585280755 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ffbc00872a0 con 0x7ffbd81466e0 2024-04-17T10:25:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.727+0000 7ffbd7400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffbb4066f60 0x7ffbb4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:54.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.727+0000 7ffbd7400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffbb4066f60 0x7ffbb4069420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ad000 tx=0x7ffbc8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:54.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.731+0000 7ffbcf400640 1 -- 172.21.15.67:0/1585280755 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ffbc0057d90 con 0x7ffbd81466e0 2024-04-17T10:25:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffbd80cba80 con 0x7ffbd81466e0 2024-04-17T10:25:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbcf400640 1 -- 172.21.15.67:0/1585280755 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffbc0057690 con 0x7ffbd81466e0 2024-04-17T10:25:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffbb4066f60 msgr2=0x7ffbb4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffbb4066f60 0x7ffbb4069420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffbc80ad000 tx=0x7ffbc8009040 comp rx=0 tx=0).stop 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd81466e0 msgr2=0x7ffbd813d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd81466e0 0x7ffbd813d250 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffbc0009e30 tx=0x7ffbc000c430 comp rx=0 tx=0).stop 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 shutdown_connections 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffbb4066f60 0x7ffbb4069420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffbd8146da0 0x7ffbd8160430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffbd81466e0 0x7ffbd813d250 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 --2- 172.21.15.67:0/1585280755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffbd8071a20 0x7ffbd815d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 >> 172.21.15.67:0/1585280755 conn(0x7ffbd806ac90 msgr2=0x7ffbd806d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 shutdown_connections 2024-04-17T10:25:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:54.907+0000 7ffbde400640 1 -- 172.21.15.67:0/1585280755 wait complete. 2024-04-17T10:25:54.921 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-17T10:25:54.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:54.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-17T10:25:54.960 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:25:54.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:54.961 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-17T10:25:55.033 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:55.033 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:55.042 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:55.042 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:55.092 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:25:55.092 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:55.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.335+0000 7ff9aae00640 1 Processor -- start 2024-04-17T10:25:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.335+0000 7ff9aae00640 1 -- start start 2024-04-17T10:25:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9aae00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff9a406eac0 0x7ff9a4143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9aae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a415ae40 con 0x7ff9a406e6e0 2024-04-17T10:25:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9aae00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a4071440 con 0x7ff9a40cf0c0 2024-04-17T10:25:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9aae00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a4071720 con 0x7ff9a406eac0 2024-04-17T10:25:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff9a406e6e0 0x7ff9a40ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33108/0 (socket says 172.21.15.67:33108) 2024-04-17T10:25:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 -- 172.21.15.67:0/2688335274 learned_addr learned my addr 172.21.15.67:0/2688335274 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541157938 0 0) 0x7ff9a415ae40 con 0x7ff9a406e6e0 2024-04-17T10:25:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff988005740 con 0x7ff9a406e6e0 2024-04-17T10:25:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff99be00640 1 --2- 172.21.15.67:0/2688335274 >> v2:172.21.15.86:3300/0 conn(0x7ff9a406eac0 0x7ff9a4143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1651399883 0 0) 0x7ff988005740 con 0x7ff9a406e6e0 2024-04-17T10:25:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 >> v2:172.21.15.86:3300/0 conn(0x7ff9a406eac0 msgr2=0x7ff9a4143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 --2- 172.21.15.67:0/2688335274 >> v2:172.21.15.86:3300/0 conn(0x7ff9a406eac0 0x7ff9a4143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 >> v1:172.21.15.86:6789/0 conn(0x7ff9a40cf0c0 legacy=0x7ff9a4144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a4071a00 con 0x7ff9a406e6e0 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7ff994001680 con 0x7ff9a406e6e0 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 >> v1:172.21.15.67:6789/0 conn(0x7ff9a406e6e0 legacy=0x7ff9a40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff988005bd0 0x7ff988005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff988006a20 0x7ff988006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9880073e0 0x7ff98800dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a4071720 con 0x7ff988005bd0 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98800e380 con 0x7ff9880073e0 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98800e580 con 0x7ff988006a20 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff988005bd0 0x7ff988005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff99be00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff988006a20 0x7ff988006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a8e00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9880073e0 0x7ff98800dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 -- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff988006a20 msgr2=0x7ff988006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:55.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff988006a20 0x7ff988006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 -- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9880073e0 msgr2=0x7ff98800dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9880073e0 0x7ff98800dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a3e00640 1 -- 172.21.15.67:0/2688335274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff988005740 con 0x7ff988005bd0 2024-04-17T10:25:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.339+0000 7ff9a8e00640 1 -- 172.21.15.67:0/2688335274 reap_dead start 2024-04-17T10:25:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9a3e00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff988005bd0 0x7ff988005ff0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c240 tx=0x7ff99401cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e315906f2d284827 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994007d20 con 0x7ff988005bd0 2024-04-17T10:25:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940a9070 con 0x7ff988005bd0 2024-04-17T10:25:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9a3400640 1 -- 172.21.15.67:0/2688335274 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff994005d80 con 0x7ff988005bd0 2024-04-17T10:25:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9aae00640 1 -- 172.21.15.67:0/2688335274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff988005bd0 msgr2=0x7ff988005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9aae00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff988005bd0 0x7ff988005ff0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff99400c240 tx=0x7ff99401cc50 comp rx=0 tx=0).stop 2024-04-17T10:25:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9aae00640 1 -- 172.21.15.67:0/2688335274 shutdown_connections 2024-04-17T10:25:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9aae00640 1 --2- 172.21.15.67:0/2688335274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff988005bd0 0x7ff988005ff0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9aae00640 1 -- 172.21.15.67:0/2688335274 >> 172.21.15.67:0/2688335274 conn(0x7ff9a406ac90 msgr2=0x7ff9a406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9aae00640 1 -- 172.21.15.67:0/2688335274 shutdown_connections 2024-04-17T10:25:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.343+0000 7ff9aae00640 1 -- 172.21.15.67:0/2688335274 wait complete. 2024-04-17T10:25:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 Processor -- start 2024-04-17T10:25:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 -- start start 2024-04-17T10:25:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff988005bd0 0x7ff9a40b0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a4151f40 0x7ff9a40b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a40b35f0 0x7ff9a40b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a4144470 con 0x7ff9a40b35f0 2024-04-17T10:25:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a41442f0 con 0x7ff988005bd0 2024-04-17T10:25:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a415e3e0 con 0x7ff9a4151f40 2024-04-17T10:25:55.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a40b35f0 0x7ff9a40b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a40b35f0 0x7ff9a40b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37110/0 (socket says 172.21.15.67:37110) 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 -- 172.21.15.67:0/741764486 learned_addr learned my addr 172.21.15.67:0/741764486 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a3e00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff988005bd0 0x7ff9a40b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff99be00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a4151f40 0x7ff9a40b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 -- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a4151f40 msgr2=0x7ff9a40b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a4151f40 0x7ff9a40b30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 -- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff988005bd0 msgr2=0x7ff9a40b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff988005bd0 0x7ff9a40b0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 -- 172.21.15.67:0/741764486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a40aefe0 con 0x7ff9a40b35f0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a8e00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a40b35f0 0x7ff9a40b0e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff9a406ef50 tx=0x7ff99c00fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a0c00640 1 -- 172.21.15.67:0/741764486 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c01a070 con 0x7ff9a40b35f0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a0c00640 1 -- 172.21.15.67:0/741764486 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c006d70 con 0x7ff9a40b35f0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a40af2c0 con 0x7ff9a40b35f0 2024-04-17T10:25:55.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9a0c00640 1 -- 172.21.15.67:0/741764486 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c003ba0 con 0x7ff9a40b35f0 2024-04-17T10:25:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.347+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a40af880 con 0x7ff9a40b35f0 2024-04-17T10:25:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.351+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a40b3a50 con 0x7ff9a40b35f0 2024-04-17T10:25:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.351+0000 7ff9a0c00640 1 -- 172.21.15.67:0/741764486 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff99c01d070 con 0x7ff9a40b35f0 2024-04-17T10:25:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.351+0000 7ff9a0c00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff980066f90 0x7ff980069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.351+0000 7ff9a0c00640 1 -- 172.21.15.67:0/741764486 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff99c084880 con 0x7ff9a40b35f0 2024-04-17T10:25:55.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.351+0000 7ff9a3e00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff980066f90 0x7ff980069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.351+0000 7ff9a3e00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff980066f90 0x7ff980069450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff9940ab000 tx=0x7ff9940a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.355+0000 7ff9a0c00640 1 -- 172.21.15.67:0/741764486 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff99c054da0 con 0x7ff9a40b35f0 2024-04-17T10:25:55.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.515+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9a4074e30 con 0x7ff9a40b35f0 2024-04-17T10:25:55.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.515+0000 7ff9a0c00640 1 -- 172.21.15.67:0/741764486 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7ff99c058e40 con 0x7ff9a40b35f0 2024-04-17T10:25:55.554 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:55.554 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff980066f90 msgr2=0x7ff980069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff980066f90 0x7ff980069450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff9940ab000 tx=0x7ff9940a7070 comp rx=0 tx=0).stop 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a40b35f0 msgr2=0x7ff9a40b0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a40b35f0 0x7ff9a40b0e80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7ff9a406ef50 tx=0x7ff99c00fc00 comp rx=0 tx=0).stop 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 shutdown_connections 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff980066f90 0x7ff980069450 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a40b35f0 0x7ff9a40b0e80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a4151f40 0x7ff9a40b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 --2- 172.21.15.67:0/741764486 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff988005bd0 0x7ff9a40b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 >> 172.21.15.67:0/741764486 conn(0x7ff9a406ac90 msgr2=0x7ff9a40cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:55.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 shutdown_connections 2024-04-17T10:25:55.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.519+0000 7ff9aae00640 1 -- 172.21.15.67:0/741764486 wait complete. 2024-04-17T10:25:55.557 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:55.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:55.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.707+0000 7fa8b7e00640 1 Processor -- start 2024-04-17T10:25:55.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.707+0000 7fa8b7e00640 1 -- start start 2024-04-17T10:25:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8b815b930 0x7fa8b815c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b809a160 0x7fa8b80cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b80cc500 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b80cc7e0 con 0x7fa8b815b930 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b7e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b80ccac0 con 0x7fa8b809a160 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8b815b930 0x7fa8b815c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8b815b930 0x7fa8b815c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37114/0 (socket says 172.21.15.67:37114) 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b6a00640 1 -- 172.21.15.67:0/3255481687 learned_addr learned my addr 172.21.15.67:0/3255481687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b7400640 1 --2- 172.21.15.67:0/3255481687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b809a160 0x7fa8b80cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b6a00640 1 -- 172.21.15.67:0/3255481687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b809a160 msgr2=0x7fa8b80cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b6a00640 1 --2- 172.21.15.67:0/3255481687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b809a160 0x7fa8b80cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b6a00640 1 -- 172.21.15.67:0/3255481687 >> v1:172.21.15.67:6789/0 conn(0x7fa8b80cd2c0 legacy=0x7fa8b8099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b6a00640 1 -- 172.21.15.67:0/3255481687 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b80cde60 con 0x7fa8b815b930 2024-04-17T10:25:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.711+0000 7fa8b7400640 1 --2- 172.21.15.67:0/3255481687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b809a160 0x7fa8b80cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:55.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b6a00640 1 --2- 172.21.15.67:0/3255481687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b815b930 0x7fa8b815c810 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009e50 tx=0x7fa8a801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea13609d942248e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b6000640 1 -- 172.21.15.67:0/3255481687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a80a5700 con 0x7fa8b815b930 2024-04-17T10:25:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b6000640 1 -- 172.21.15.67:0/3255481687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a80a58a0 con 0x7fa8b815b930 2024-04-17T10:25:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b6000640 1 -- 172.21.15.67:0/3255481687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a80a5c70 con 0x7fa8b815b930 2024-04-17T10:25:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3255481687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b815b930 msgr2=0x7fa8b815c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3255481687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b815b930 0x7fa8b815c810 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa8a8009e50 tx=0x7fa8a801ca30 comp rx=0 tx=0).stop 2024-04-17T10:25:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3255481687 shutdown_connections 2024-04-17T10:25:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3255481687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b809a160 0x7fa8b80cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3255481687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b815b930 0x7fa8b815c810 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3255481687 >> 172.21.15.67:0/3255481687 conn(0x7fa8b80d1630 msgr2=0x7fa8b80d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3255481687 shutdown_connections 2024-04-17T10:25:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3255481687 wait complete. 2024-04-17T10:25:55.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 Processor -- start 2024-04-17T10:25:55.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- start start 2024-04-17T10:25:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8b809a160 0x7fa8b80cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80cd2c0 0x7fa8b8156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b815b930 0x7fa8b815cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8168ab0 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8168930 con 0x7fa8b809a160 2024-04-17T10:25:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.715+0000 7fa8b7e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b8168c30 con 0x7fa8b815b930 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80cd2c0 0x7fa8b8156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80cd2c0 0x7fa8b8156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37118/0 (socket says 172.21.15.67:37118) 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 -- 172.21.15.67:0/3734150552 learned_addr learned my addr 172.21.15.67:0/3734150552 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8b7400640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b815b930 0x7fa8b815cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8b6a00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8b809a160 0x7fa8b80cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 -- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b815b930 msgr2=0x7fa8b815cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b815b930 0x7fa8b815cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 -- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8b809a160 msgr2=0x7fa8b80cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8b809a160 0x7fa8b80cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 -- 172.21.15.67:0/3734150552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b81574b0 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8ae000640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80cd2c0 0x7fa8b8156b70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000c3a0 tx=0x7fa8a000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8af400640 1 -- 172.21.15.67:0/3734150552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a00052d0 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8af400640 1 -- 172.21.15.67:0/3734150552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a0005470 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b8156090 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8af400640 1 -- 172.21.15.67:0/3734150552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a0007260 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b8157790 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b804f780 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.719+0000 7fa8af400640 1 -- 172.21.15.67:0/3734150552 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa8a0007400 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.723+0000 7fa8af400640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa894066f60 0x7fa894069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.723+0000 7fa8af400640 1 -- 172.21.15.67:0/3734150552 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fa8a0084880 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.723+0000 7fa8b6a00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa894066f60 0x7fa894069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.723+0000 7fa8b6a00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa894066f60 0x7fa894069420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80ae000 tx=0x7fa8a80a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.727+0000 7fa8af400640 1 -- 172.21.15.67:0/3734150552 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa8a0055480 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8b815b0d0 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8af400640 1 -- 172.21.15.67:0/3734150552 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8a0054d80 con 0x7fa8b80cd2c0 2024-04-17T10:25:55.885 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa894066f60 msgr2=0x7fa894069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa894066f60 0x7fa894069420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80ae000 tx=0x7fa8a80a7070 comp rx=0 tx=0).stop 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80cd2c0 msgr2=0x7fa8b8156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80cd2c0 0x7fa8b8156b70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa8a000c3a0 tx=0x7fa8a000c860 comp rx=0 tx=0).stop 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 shutdown_connections 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa894066f60 0x7fa894069420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8b815b930 0x7fa8b815cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8b80cd2c0 0x7fa8b8156b70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 --2- 172.21.15.67:0/3734150552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8b809a160 0x7fa8b80cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 >> 172.21.15.67:0/3734150552 conn(0x7fa8b80d1630 msgr2=0x7fa8b8157a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 shutdown_connections 2024-04-17T10:25:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:55.891+0000 7fa8b7e00640 1 -- 172.21.15.67:0/3734150552 wait complete. 2024-04-17T10:25:55.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:56.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd019600640 1 Processor -- start 2024-04-17T10:25:56.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd019600640 1 -- start start 2024-04-17T10:25:56.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd019600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd0140cd2c0 0x7fd01415c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd019600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0140cc500 con 0x7fd01415b930 2024-04-17T10:25:56.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd019600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0140cc7e0 con 0x7fd01409a160 2024-04-17T10:25:56.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd019600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0140ccac0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd013e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd01415b930 0x7fd014099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33138/0 (socket says 172.21.15.67:33138) 2024-04-17T10:25:56.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd013e00640 1 -- 172.21.15.67:0/1818583053 learned_addr learned my addr 172.21.15.67:0/1818583053 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:56.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd018c00640 1 --2- 172.21.15.67:0/1818583053 >> v2:172.21.15.86:3300/0 conn(0x7fd0140cd2c0 0x7fd01415c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd013400640 1 -- 172.21.15.67:0/1818583053 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17875137 0 0) 0x7fd0140cc500 con 0x7fd01415b930 2024-04-17T10:25:56.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd013400640 1 -- 172.21.15.67:0/1818583053 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcff8003530 con 0x7fd01415b930 2024-04-17T10:25:56.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd018c00640 1 -- 172.21.15.67:0/1818583053 >> v1:172.21.15.86:6789/0 conn(0x7fd01409a160 legacy=0x7fd0140cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd018c00640 1 -- 172.21.15.67:0/1818583053 >> v1:172.21.15.67:6789/0 conn(0x7fd01415b930 legacy=0x7fd014099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.055+0000 7fd018c00640 1 -- 172.21.15.67:0/1818583053 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0140cde60 con 0x7fd0140cd2c0 2024-04-17T10:25:56.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd018c00640 1 --2- 172.21.15.67:0/1818583053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0140cd2c0 0x7fd01415c810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd00c009d90 tx=0x7fd00c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab898de2f960e9c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd013400640 1 -- 172.21.15.67:0/1818583053 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c0b2070 con 0x7fd0140cd2c0 2024-04-17T10:25:56.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd013400640 1 -- 172.21.15.67:0/1818583053 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00c0a5830 con 0x7fd0140cd2c0 2024-04-17T10:25:56.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd013400640 1 -- 172.21.15.67:0/1818583053 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00c0a5c00 con 0x7fd0140cd2c0 2024-04-17T10:25:56.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd019600640 1 -- 172.21.15.67:0/1818583053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0140cd2c0 msgr2=0x7fd01415c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd019600640 1 --2- 172.21.15.67:0/1818583053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0140cd2c0 0x7fd01415c810 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd00c009d90 tx=0x7fd00c01c970 comp rx=0 tx=0).stop 2024-04-17T10:25:56.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd019600640 1 -- 172.21.15.67:0/1818583053 shutdown_connections 2024-04-17T10:25:56.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd019600640 1 --2- 172.21.15.67:0/1818583053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0140cd2c0 0x7fd01415c810 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd019600640 1 -- 172.21.15.67:0/1818583053 >> 172.21.15.67:0/1818583053 conn(0x7fd0140d1630 msgr2=0x7fd0140d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:56.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd019600640 1 -- 172.21.15.67:0/1818583053 shutdown_connections 2024-04-17T10:25:56.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.059+0000 7fd019600640 1 -- 172.21.15.67:0/1818583053 wait complete. 2024-04-17T10:25:56.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 Processor -- start 2024-04-17T10:25:56.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 -- start start 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd01409a160 0x7fd01406c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0140cd2c0 0x7fd0140c7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd01415b930 0x7fd0140c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014169ac0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014169940 con 0x7fd01415b930 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd014169c40 con 0x7fd01409a160 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0140cd2c0 0x7fd0140c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0140cd2c0 0x7fd0140c7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37130/0 (socket says 172.21.15.67:37130) 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 -- 172.21.15.67:0/406479030 learned_addr learned my addr 172.21.15.67:0/406479030 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd013e00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd01409a160 0x7fd01406c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd018c00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd01415b930 0x7fd0140c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 -- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd01409a160 msgr2=0x7fd01406c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd01409a160 0x7fd01406c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 -- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd01415b930 msgr2=0x7fd0140c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd01415b930 0x7fd0140c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 -- 172.21.15.67:0/406479030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0140b63c0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd018c00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd01415b930 0x7fd0140c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd013e00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd01409a160 0x7fd01406c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd00be00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0140cd2c0 0x7fd0140c7330 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c370 tx=0x7fcffc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd010c00640 1 -- 172.21.15.67:0/406479030 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc0052d0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd010c00640 1 -- 172.21.15.67:0/406479030 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc005470 con 0x7fd0140cd2c0 2024-04-17T10:25:56.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd010c00640 1 -- 172.21.15.67:0/406479030 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc007260 con 0x7fd0140cd2c0 2024-04-17T10:25:56.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0140b66a0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0140b6c60 con 0x7fd0140cd2c0 2024-04-17T10:25:56.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.063+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01404f780 con 0x7fd0140cd2c0 2024-04-17T10:25:56.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.067+0000 7fd010c00640 1 -- 172.21.15.67:0/406479030 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcffc007400 con 0x7fd0140cd2c0 2024-04-17T10:25:56.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.067+0000 7fd010c00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcff0066f60 0x7fcff0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.067+0000 7fd010c00640 1 -- 172.21.15.67:0/406479030 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fcffc0848b0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.067+0000 7fd013e00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcff0066f60 0x7fcff0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.067+0000 7fd013e00640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcff0066f60 0x7fcff0069420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd0040011f0 tx=0x7fd004009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.071+0000 7fd010c00640 1 -- 172.21.15.67:0/406479030 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcffc0554a0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.231+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0140d2440 con 0x7fd0140cd2c0 2024-04-17T10:25:56.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.231+0000 7fd010c00640 1 -- 172.21.15.67:0/406479030 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcffc054da0 con 0x7fd0140cd2c0 2024-04-17T10:25:56.227 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:56.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcff0066f60 msgr2=0x7fcff0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcff0066f60 0x7fcff0069420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd0040011f0 tx=0x7fd004009040 comp rx=0 tx=0).stop 2024-04-17T10:25:56.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0140cd2c0 msgr2=0x7fd0140c7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0140cd2c0 0x7fd0140c7330 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcffc00c370 tx=0x7fcffc00c830 comp rx=0 tx=0).stop 2024-04-17T10:25:56.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 shutdown_connections 2024-04-17T10:25:56.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcff0066f60 0x7fcff0069420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd01415b930 0x7fd0140c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0140cd2c0 0x7fd0140c7330 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 --2- 172.21.15.67:0/406479030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd01409a160 0x7fd01406c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 >> 172.21.15.67:0/406479030 conn(0x7fd0140d1630 msgr2=0x7fd014099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:56.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 shutdown_connections 2024-04-17T10:25:56.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.235+0000 7fd019600640 1 -- 172.21.15.67:0/406479030 wait complete. 2024-04-17T10:25:56.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e2a800640 1 Processor -- start 2024-04-17T10:25:56.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e2a800640 1 -- start start 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e2a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240aef60 0x7f4e240cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e2a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e2406e980 0x7f4e240b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e2a800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e240b4990 con 0x7f4e240cce60 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e2a800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e240cd2c0 con 0x7f4e2406e980 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e2a800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e240af340 con 0x7f4e240aef60 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e1be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4e240cce60 0x7f4e2406e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33142/0 (socket says 172.21.15.67:33142) 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.399+0000 7f4e1be00640 1 -- 172.21.15.67:0/244403620 learned_addr learned my addr 172.21.15.67:0/244403620 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e23e00640 1 --2- 172.21.15.67:0/244403620 >> v2:172.21.15.67:3300/0 conn(0x7f4e2406e980 0x7f4e240b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e23400640 1 --2- 172.21.15.67:0/244403620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240aef60 0x7f4e240cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e23e00640 1 -- 172.21.15.67:0/244403620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240aef60 msgr2=0x7f4e240cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:56.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e23e00640 1 --2- 172.21.15.67:0/244403620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240aef60 0x7f4e240cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e23e00640 1 -- 172.21.15.67:0/244403620 >> v1:172.21.15.67:6789/0 conn(0x7f4e240cce60 legacy=0x7f4e2406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e22a00640 1 -- 172.21.15.67:0/244403620 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584472768 0 0) 0x7f4e0c001120 con 0x7f4e240cce60 2024-04-17T10:25:56.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e23e00640 1 -- 172.21.15.67:0/244403620 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2415ae90 con 0x7f4e2406e980 2024-04-17T10:25:56.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e23e00640 1 --2- 172.21.15.67:0/244403620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2406e980 0x7f4e240b3120 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009d90 tx=0x7f4e1c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f137481703e5376 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e22a00640 1 -- 172.21.15.67:0/244403620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c0b2070 con 0x7f4e2406e980 2024-04-17T10:25:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e22a00640 1 -- 172.21.15.67:0/244403620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e1c0a5890 con 0x7f4e2406e980 2024-04-17T10:25:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.403+0000 7f4e22a00640 1 -- 172.21.15.67:0/244403620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c0a5c60 con 0x7f4e2406e980 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- 172.21.15.67:0/244403620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2406e980 msgr2=0x7f4e240b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 --2- 172.21.15.67:0/244403620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2406e980 0x7f4e240b3120 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009d90 tx=0x7f4e1c01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- 172.21.15.67:0/244403620 shutdown_connections 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 --2- 172.21.15.67:0/244403620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2406e980 0x7f4e240b3120 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 --2- 172.21.15.67:0/244403620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240aef60 0x7f4e240cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- 172.21.15.67:0/244403620 >> 172.21.15.67:0/244403620 conn(0x7f4e240b1740 msgr2=0x7f4e240b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- 172.21.15.67:0/244403620 shutdown_connections 2024-04-17T10:25:56.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- 172.21.15.67:0/244403620 wait complete. 2024-04-17T10:25:56.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 Processor -- start 2024-04-17T10:25:56.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- start start 2024-04-17T10:25:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e2406e980 0x7f4e24142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e240aef60 0x7f4e240aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240cce60 0x7f4e240b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2415bc10 con 0x7f4e240aef60 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e240b3e70 con 0x7f4e2406e980 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e2a800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2414f0c0 con 0x7f4e240cce60 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e240aef60 0x7f4e240aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e240aef60 0x7f4e240aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37152/0 (socket says 172.21.15.67:37152) 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 -- 172.21.15.67:0/1126402046 learned_addr learned my addr 172.21.15.67:0/1126402046 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e23400640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e2406e980 0x7f4e24142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e23e00640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240cce60 0x7f4e240b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 -- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240cce60 msgr2=0x7f4e240b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240cce60 0x7f4e240b0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 -- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e2406e980 msgr2=0x7f4e24142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e2406e980 0x7f4e24142af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 -- 172.21.15.67:0/1126402046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24151cf0 con 0x7f4e240aef60 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.407+0000 7f4e1be00640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e240aef60 0x7f4e240aff10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00c4b0 tx=0x7f4e0c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e1b400640 1 -- 172.21.15.67:0/1126402046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c018070 con 0x7f4e240aef60 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e1b400640 1 -- 172.21.15.67:0/1126402046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0c007500 con 0x7f4e240aef60 2024-04-17T10:25:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e1b400640 1 -- 172.21.15.67:0/1126402046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c004a30 con 0x7f4e240aef60 2024-04-17T10:25:56.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2414f330 con 0x7f4e240aef60 2024-04-17T10:25:56.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2414f8f0 con 0x7f4e240aef60 2024-04-17T10:25:56.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2404ef30 con 0x7f4e240aef60 2024-04-17T10:25:56.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e1b400640 1 -- 172.21.15.67:0/1126402046 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4e0c004bd0 con 0x7f4e240aef60 2024-04-17T10:25:56.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e1b400640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e00066f60 0x7f4e00069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e1b400640 1 -- 172.21.15.67:0/1126402046 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f4e0c084620 con 0x7f4e240aef60 2024-04-17T10:25:56.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.411+0000 7f4e23400640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e00066f60 0x7f4e00069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.415+0000 7f4e23400640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e00066f60 0x7f4e00069420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400b9d0 tx=0x7f4e14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.415+0000 7f4e1b400640 1 -- 172.21.15.67:0/1126402046 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4e24151cf0 con 0x7f4e240aef60 2024-04-17T10:25:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.579+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4e240b1cc0 con 0x7f4e240aef60 2024-04-17T10:25:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.579+0000 7f4e1b400640 1 -- 172.21.15.67:0/1126402046 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4e240b4990 con 0x7f4e240aef60 2024-04-17T10:25:56.574 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.579+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e00066f60 msgr2=0x7f4e00069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.579+0000 7f4e2a800640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e00066f60 0x7f4e00069420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f4e1400b9d0 tx=0x7f4e14008040 comp rx=0 tx=0).stop 2024-04-17T10:25:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.579+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e240aef60 msgr2=0x7f4e240aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.579+0000 7f4e2a800640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e240aef60 0x7f4e240aff10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00c4b0 tx=0x7f4e0c00c970 comp rx=0 tx=0).stop 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 shutdown_connections 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e00066f60 0x7f4e00069420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e240cce60 0x7f4e240b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e240aef60 0x7f4e240aff10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 --2- 172.21.15.67:0/1126402046 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e2406e980 0x7f4e24142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 >> 172.21.15.67:0/1126402046 conn(0x7f4e240b1740 msgr2=0x7f4e2406b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 shutdown_connections 2024-04-17T10:25:56.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.583+0000 7f4e2a800640 1 -- 172.21.15.67:0/1126402046 wait complete. 2024-04-17T10:25:56.595 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:56.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.743+0000 7fd410400640 1 Processor -- start 2024-04-17T10:25:56.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.743+0000 7fd410400640 1 -- start start 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.743+0000 7fd410400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4080cd030 0x7fd40806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.743+0000 7fd410400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd40806e7b0 0x7fd4080b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.743+0000 7fd410400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4080b4990 con 0x7fd4080aef60 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.743+0000 7fd410400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4080af340 con 0x7fd4080cd030 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.743+0000 7fd410400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40806ec00 con 0x7fd40806e7b0 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40da00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd4080aef60 0x7fd4080cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33144/0 (socket says 172.21.15.67:33144) 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40da00640 1 -- 172.21.15.67:0/1860794582 learned_addr learned my addr 172.21.15.67:0/1860794582 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40d000640 1 --2- 172.21.15.67:0/1860794582 >> v2:172.21.15.67:3300/0 conn(0x7fd4080cd030 0x7fd40806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40e400640 1 --2- 172.21.15.67:0/1860794582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd40806e7b0 0x7fd4080b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40d000640 1 -- 172.21.15.67:0/1860794582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd40806e7b0 msgr2=0x7fd4080b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:56.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40d000640 1 --2- 172.21.15.67:0/1860794582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd40806e7b0 0x7fd4080b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40d000640 1 -- 172.21.15.67:0/1860794582 >> v1:172.21.15.67:6789/0 conn(0x7fd4080aef60 legacy=0x7fd4080cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40d000640 1 -- 172.21.15.67:0/1860794582 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40815ae90 con 0x7fd4080cd030 2024-04-17T10:25:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd40d000640 1 --2- 172.21.15.67:0/1860794582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080cd030 0x7fd40806e270 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40097c0 tx=0x7fd3f401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d821abbfb4a3dda server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd407e00640 1 -- 172.21.15.67:0/1860794582 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40b0070 con 0x7fd4080cd030 2024-04-17T10:25:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd407e00640 1 -- 172.21.15.67:0/1860794582 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f40ab600 con 0x7fd4080cd030 2024-04-17T10:25:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.747+0000 7fd407e00640 1 -- 172.21.15.67:0/1860794582 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40ab9d0 con 0x7fd4080cd030 2024-04-17T10:25:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- 172.21.15.67:0/1860794582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080cd030 msgr2=0x7fd40806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 --2- 172.21.15.67:0/1860794582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080cd030 0x7fd40806e270 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40097c0 tx=0x7fd3f401c410 comp rx=0 tx=0).stop 2024-04-17T10:25:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- 172.21.15.67:0/1860794582 shutdown_connections 2024-04-17T10:25:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 --2- 172.21.15.67:0/1860794582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd40806e7b0 0x7fd4080b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 --2- 172.21.15.67:0/1860794582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080cd030 0x7fd40806e270 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- 172.21.15.67:0/1860794582 >> 172.21.15.67:0/1860794582 conn(0x7fd4080b1740 msgr2=0x7fd4080b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- 172.21.15.67:0/1860794582 shutdown_connections 2024-04-17T10:25:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- 172.21.15.67:0/1860794582 wait complete. 2024-04-17T10:25:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 Processor -- start 2024-04-17T10:25:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- start start 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40806e7b0 0x7fd40814b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080aef60 0x7fd408159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4080cd030 0x7fd408154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40815bc10 con 0x7fd4080aef60 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4080b3e70 con 0x7fd40806e7b0 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd410400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4081520b0 con 0x7fd4080cd030 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080aef60 0x7fd408159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080aef60 0x7fd408159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37162/0 (socket says 172.21.15.67:37162) 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 -- 172.21.15.67:0/294742658 learned_addr learned my addr 172.21.15.67:0/294742658 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40e400640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4080cd030 0x7fd408154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40da00640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40806e7b0 0x7fd40814b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 -- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4080cd030 msgr2=0x7fd408154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4080cd030 0x7fd408154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 -- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40806e7b0 msgr2=0x7fd40814b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40806e7b0 0x7fd40814b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 -- 172.21.15.67:0/294742658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40814a7f0 con 0x7fd4080aef60 2024-04-17T10:25:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.751+0000 7fd40d000640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080aef60 0x7fd408159ab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40ae040 tx=0x7fd3f40191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd405600640 1 -- 172.21.15.67:0/294742658 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40b0040 con 0x7fd4080aef60 2024-04-17T10:25:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd405600640 1 -- 172.21.15.67:0/294742658 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3f40b9070 con 0x7fd4080aef60 2024-04-17T10:25:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd405600640 1 -- 172.21.15.67:0/294742658 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3f40aaa30 con 0x7fd4080aef60 2024-04-17T10:25:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40814aad0 con 0x7fd4080aef60 2024-04-17T10:25:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40814b090 con 0x7fd4080aef60 2024-04-17T10:25:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40804ef30 con 0x7fd4080aef60 2024-04-17T10:25:56.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd405600640 1 -- 172.21.15.67:0/294742658 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd3f40aabd0 con 0x7fd4080aef60 2024-04-17T10:25:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd405600640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd3e8066f60 0x7fd3e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:56.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd405600640 1 -- 172.21.15.67:0/294742658 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fd3f411f4b0 con 0x7fd4080aef60 2024-04-17T10:25:56.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.755+0000 7fd40da00640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd3e8066f60 0x7fd3e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:56.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.759+0000 7fd40da00640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd3e8066f60 0x7fd3e8069420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc000f70 tx=0x7fd3fc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:56.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.759+0000 7fd405600640 1 -- 172.21.15.67:0/294742658 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd3f4016090 con 0x7fd4080aef60 2024-04-17T10:25:56.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.919+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd40806dca0 con 0x7fd4080aef60 2024-04-17T10:25:56.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.919+0000 7fd405600640 1 -- 172.21.15.67:0/294742658 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd3f40efa60 con 0x7fd4080aef60 2024-04-17T10:25:56.916 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd3e8066f60 msgr2=0x7fd3e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd3e8066f60 0x7fd3e8069420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc000f70 tx=0x7fd3fc009040 comp rx=0 tx=0).stop 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080aef60 msgr2=0x7fd408159ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080aef60 0x7fd408159ab0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd3f40ae040 tx=0x7fd3f40191b0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 shutdown_connections 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd3e8066f60 0x7fd3e8069420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4080cd030 0x7fd408154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4080aef60 0x7fd408159ab0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 --2- 172.21.15.67:0/294742658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd40806e7b0 0x7fd40814b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 >> 172.21.15.67:0/294742658 conn(0x7fd4080b1740 msgr2=0x7fd4080b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 shutdown_connections 2024-04-17T10:25:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:56.923+0000 7fd410400640 1 -- 172.21.15.67:0/294742658 wait complete. 2024-04-17T10:25:56.938 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:25:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f93000640 1 Processor -- start 2024-04-17T10:25:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f93000640 1 -- start start 2024-04-17T10:25:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f93000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f8c0cdcf0 0x7f6f8c0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f93000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c06b100 con 0x7f6f8c0cc250 2024-04-17T10:25:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f93000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c06b3e0 con 0x7f6f8c0ce2a0 2024-04-17T10:25:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f93000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f8c0cd150 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f8be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6f8c0cc250 0x7f6f8c0cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33154/0 (socket says 172.21.15.67:33154) 2024-04-17T10:25:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f8be00640 1 -- 172.21.15.67:0/1256455878 learned_addr learned my addr 172.21.15.67:0/1256455878 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f91000640 1 --2- 172.21.15.67:0/1256455878 >> v2:172.21.15.67:3300/0 conn(0x7f6f8c0cdcf0 0x7f6f8c0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f8b400640 1 -- 172.21.15.67:0/1256455878 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086416513 0 0) 0x7f6f8c06b100 con 0x7f6f8c0cc250 2024-04-17T10:25:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f8b400640 1 -- 172.21.15.67:0/1256455878 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70003530 con 0x7f6f8c0cc250 2024-04-17T10:25:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f91000640 1 -- 172.21.15.67:0/1256455878 >> v1:172.21.15.86:6789/0 conn(0x7f6f8c0ce2a0 legacy=0x7f6f8c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f91000640 1 -- 172.21.15.67:0/1256455878 >> v1:172.21.15.67:6789/0 conn(0x7f6f8c0cc250 legacy=0x7f6f8c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.087+0000 7f6f91000640 1 -- 172.21.15.67:0/1256455878 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c06b6c0 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f91000640 1 --2- 172.21.15.67:0/1256455878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0cf9d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009d90 tx=0x7f6f8401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827afed427f676aa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f8b400640 1 -- 172.21.15.67:0/1256455878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f840b2070 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f8b400640 1 -- 172.21.15.67:0/1256455878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f840a5940 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f8b400640 1 -- 172.21.15.67:0/1256455878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f840a5cf0 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 -- 172.21.15.67:0/1256455878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 msgr2=0x7f6f8c0cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 --2- 172.21.15.67:0/1256455878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0cf9d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6f84009d90 tx=0x7f6f8401c970 comp rx=0 tx=0).stop 2024-04-17T10:25:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 -- 172.21.15.67:0/1256455878 shutdown_connections 2024-04-17T10:25:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 --2- 172.21.15.67:0/1256455878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0cf9d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 -- 172.21.15.67:0/1256455878 >> 172.21.15.67:0/1256455878 conn(0x7f6f8c0b22a0 msgr2=0x7f6f8c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 -- 172.21.15.67:0/1256455878 shutdown_connections 2024-04-17T10:25:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 -- 172.21.15.67:0/1256455878 wait complete. 2024-04-17T10:25:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 Processor -- start 2024-04-17T10:25:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.091+0000 7f6f93000640 1 -- start start 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f8c0cc250 0x7f6f8c0b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f8c0ce2a0 0x7f6f8c0b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c16c290 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c16c110 con 0x7f6f8c0cc250 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c16c410 con 0x7f6f8c0ce2a0 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37176/0 (socket says 172.21.15.67:37176) 2024-04-17T10:25:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 -- 172.21.15.67:0/2643114519 learned_addr learned my addr 172.21.15.67:0/2643114519 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:57.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f8be00640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f8c0cc250 0x7f6f8c0b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f91000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f8c0ce2a0 0x7f6f8c0b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 -- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f8c0ce2a0 msgr2=0x7f6f8c0b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f8c0ce2a0 0x7f6f8c0b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 -- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f8c0cc250 msgr2=0x7f6f8c0b5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f8c0cc250 0x7f6f8c0b5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 -- 172.21.15.67:0/2643114519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c0c6420 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f8be00640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f8c0cc250 0x7f6f8c0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f91000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f8c0ce2a0 0x7f6f8c0b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f83400640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0b3290 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400c340 tx=0x7f6f7400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f88c00640 1 -- 172.21.15.67:0/2643114519 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f740052d0 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f88c00640 1 -- 172.21.15.67:0/2643114519 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f74005470 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c0c5b80 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f88c00640 1 -- 172.21.15.67:0/2643114519 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74007260 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c0c6140 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.095+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c09e900 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.099+0000 7f6f88c00640 1 -- 172.21.15.67:0/2643114519 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6f74007400 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.099+0000 7f6f88c00640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f68066f60 0x7f6f68069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.099+0000 7f6f88c00640 1 -- 172.21.15.67:0/2643114519 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6f74084890 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.099+0000 7f6f8be00640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f68066f60 0x7f6f68069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.099+0000 7f6f8be00640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f68066f60 0x7f6f68069420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0011f0 tx=0x7f6f7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.103+0000 7f6f88c00640 1 -- 172.21.15.67:0/2643114519 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6f74055490 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6f8c0b2a90 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f88c00640 1 -- 172.21.15.67:0/2643114519 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6f74054d90 con 0x7f6f8c0cdcf0 2024-04-17T10:25:57.270 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:25:57.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f68066f60 msgr2=0x7f6f68069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f68066f60 0x7f6f68069420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0011f0 tx=0x7f6f7c009040 comp rx=0 tx=0).stop 2024-04-17T10:25:57.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 msgr2=0x7f6f8c0b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0b3290 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400c340 tx=0x7f6f7400c800 comp rx=0 tx=0).stop 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 shutdown_connections 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f68066f60 0x7f6f68069420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f8c0ce2a0 0x7f6f8c0b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f8c0cdcf0 0x7f6f8c0b3290 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 --2- 172.21.15.67:0/2643114519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f8c0cc250 0x7f6f8c0b5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.275+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 >> 172.21.15.67:0/2643114519 conn(0x7f6f8c0b22a0 msgr2=0x7f6f8c0ad5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.279+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 shutdown_connections 2024-04-17T10:25:57.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.279+0000 7f6f93000640 1 -- 172.21.15.67:0/2643114519 wait complete. 2024-04-17T10:25:57.292 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-17T10:25:57.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:57.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-17T10:25:57.316 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:25:57.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:57.316 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-17T10:25:57.369 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:57.369 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:57.380 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:57.380 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:57.429 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:25:57.429 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:25:57.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:25:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.623+0000 7fa6ba800640 1 Processor -- start 2024-04-17T10:25:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.623+0000 7fa6ba800640 1 -- start start 2024-04-17T10:25:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6ba800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6b4145040 0x7fa6b406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 0x7fa6b4071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6ba800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b406e2f0 con 0x7fa6b4071850 2024-04-17T10:25:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6ba800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b40a86c0 con 0x7fa6b4145040 2024-04-17T10:25:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6ba800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b415bf10 con 0x7fa6b406ef10 2024-04-17T10:25:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa6b4071850 0x7fa6b415f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33170/0 (socket says 172.21.15.67:33170) 2024-04-17T10:25:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b3e00640 1 -- 172.21.15.67:0/3432291209 learned_addr learned my addr 172.21.15.67:0/3432291209 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:57.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b3400640 1 --2- 172.21.15.67:0/3432291209 >> v2:172.21.15.86:3300/0 conn(0x7fa6b4145040 0x7fa6b406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6aaa00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 0x7fa6b4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989520347 0 0) 0x7fa6b406e2f0 con 0x7fa6b4071850 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698003530 con 0x7fa6b4071850 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1596061640 0 0) 0x7fa698003530 con 0x7fa6b4071850 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 msgr2=0x7fa6b4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 0x7fa6b4071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 >> v2:172.21.15.86:3300/0 conn(0x7fa6b4145040 msgr2=0x7fa6b406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 --2- 172.21.15.67:0/3432291209 >> v2:172.21.15.86:3300/0 conn(0x7fa6b4145040 0x7fa6b406e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b406d6e0 con 0x7fa6b4071850 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fa6ac001a00 con 0x7fa6b4071850 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 >> v1:172.21.15.67:6789/0 conn(0x7fa6b4071850 legacy=0x7fa6b415f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6980039c0 0x7fa698003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa698004810 0x7fa698004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 0x7fa6980051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b415bf10 con 0x7fa698004810 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69800b9d0 con 0x7fa6980039c0 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa69800bb50 con 0x7fa6b406ef10 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6aaa00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa698004810 0x7fa698004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b3400640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6980039c0 0x7fa698003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6b3e00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 0x7fa6980051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6aaa00640 1 -- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 msgr2=0x7fa6980051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6aaa00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6b406ef10 0x7fa6980051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6aaa00640 1 -- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6980039c0 msgr2=0x7fa698003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6aaa00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6980039c0 0x7fa698003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.627+0000 7fa6aaa00640 1 -- 172.21.15.67:0/3432291209 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698003530 con 0x7fa698004810 2024-04-17T10:25:57.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6aaa00640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa698004810 0x7fa698004c90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009ae0 tx=0x7fa69c01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8557026b1165c83f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c0b2070 con 0x7fa698004810 2024-04-17T10:25:57.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c0a5580 con 0x7fa698004810 2024-04-17T10:25:57.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6b2a00640 1 -- 172.21.15.67:0/3432291209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c0096b0 con 0x7fa698004810 2024-04-17T10:25:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6ba800640 1 -- 172.21.15.67:0/3432291209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa698004810 msgr2=0x7fa698004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6ba800640 1 --2- 172.21.15.67:0/3432291209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa698004810 0x7fa698004c90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009ae0 tx=0x7fa69c01ca70 comp rx=0 tx=0).stop 2024-04-17T10:25:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6b3e00640 1 -- 172.21.15.67:0/3432291209 reap_dead start 2024-04-17T10:25:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6ba800640 1 -- 172.21.15.67:0/3432291209 shutdown_connections 2024-04-17T10:25:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6ba800640 1 -- 172.21.15.67:0/3432291209 >> 172.21.15.67:0/3432291209 conn(0x7fa6b406ac90 msgr2=0x7fa6b406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6ba800640 1 -- 172.21.15.67:0/3432291209 shutdown_connections 2024-04-17T10:25:57.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.631+0000 7fa6ba800640 1 -- 172.21.15.67:0/3432291209 wait complete. 2024-04-17T10:25:57.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 Processor -- start 2024-04-17T10:25:57.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 -- start start 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa698004810 0x7fa6b41f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b413a0b0 0x7fa6b413a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6b413bd00 0x7fa6b413c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b406dca0 con 0x7fa6b413bd00 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b4071ca0 con 0x7fa6b413a0b0 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b406df80 con 0x7fa698004810 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6b413bd00 0x7fa6b413c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6b413bd00 0x7fa6b413c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37192/0 (socket says 172.21.15.67:37192) 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 -- 172.21.15.67:0/3814706769 learned_addr learned my addr 172.21.15.67:0/3814706769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3400640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa698004810 0x7fa6b41f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6aaa00640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b413a0b0 0x7fa6b413a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 -- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa698004810 msgr2=0x7fa6b41f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa698004810 0x7fa6b41f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 -- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b413a0b0 msgr2=0x7fa6b413a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b413a0b0 0x7fa6b413a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 -- 172.21.15.67:0/3814706769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b415e390 con 0x7fa6b413bd00 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6b3e00640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6b413bd00 0x7fa6b413c180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa6b406ba90 tx=0x7fa6ac00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6abe00640 1 -- 172.21.15.67:0/3814706769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac00ec10 con 0x7fa6b413bd00 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6abe00640 1 -- 172.21.15.67:0/3814706769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac00edd0 con 0x7fa6b413bd00 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6abe00640 1 -- 172.21.15.67:0/3814706769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac013420 con 0x7fa6b413bd00 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b415e670 con 0x7fa6b413bd00 2024-04-17T10:25:57.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b415ec30 con 0x7fa6b413bd00 2024-04-17T10:25:57.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.635+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6b4072d50 con 0x7fa6b413bd00 2024-04-17T10:25:57.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.639+0000 7fa6abe00640 1 -- 172.21.15.67:0/3814706769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa6ac01a020 con 0x7fa6b413bd00 2024-04-17T10:25:57.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.639+0000 7fa6abe00640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa690066f60 0x7fa690069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.639+0000 7fa6abe00640 1 -- 172.21.15.67:0/3814706769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fa6ac084ea0 con 0x7fa6b413bd00 2024-04-17T10:25:57.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.639+0000 7fa6b3400640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa690066f60 0x7fa690069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.639+0000 7fa6b3400640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa690066f60 0x7fa690069420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009ab0 tx=0x7fa6a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.643+0000 7fa6abe00640 1 -- 172.21.15.67:0/3814706769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa6ac021020 con 0x7fa6b413bd00 2024-04-17T10:25:57.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.807+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6b4140740 con 0x7fa6b413bd00 2024-04-17T10:25:57.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.807+0000 7fa6abe00640 1 -- 172.21.15.67:0/3814706769 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7fa6ac055a10 con 0x7fa6b413bd00 2024-04-17T10:25:57.803 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:25:57.803 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:25:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa690066f60 msgr2=0x7fa690069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa690066f60 0x7fa690069420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4009ab0 tx=0x7fa6a4008040 comp rx=0 tx=0).stop 2024-04-17T10:25:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6b413bd00 msgr2=0x7fa6b413c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6b413bd00 0x7fa6b413c180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fa6b406ba90 tx=0x7fa6ac00e530 comp rx=0 tx=0).stop 2024-04-17T10:25:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 shutdown_connections 2024-04-17T10:25:57.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa690066f60 0x7fa690069420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6b413bd00 0x7fa6b413c180 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6b413a0b0 0x7fa6b413a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 --2- 172.21.15.67:0/3814706769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa698004810 0x7fa6b41f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 >> 172.21.15.67:0/3814706769 conn(0x7fa6b406ac90 msgr2=0x7fa6b40cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 shutdown_connections 2024-04-17T10:25:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.811+0000 7fa6ba800640 1 -- 172.21.15.67:0/3814706769 wait complete. 2024-04-17T10:25:57.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:25:57.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:25:57.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.975+0000 7f7817200640 1 Processor -- start 2024-04-17T10:25:57.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.975+0000 7f7817200640 1 -- start start 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7817200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f781009a160 0x7f781015c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7817200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 0x7f78100cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7817200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78100cc500 con 0x7f781015b930 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7817200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78100cc7e0 con 0x7f781009a160 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7817200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78100ccac0 con 0x7f78100cd2c0 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f781015b930 0x7f7810099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33186/0 (socket says 172.21.15.67:33186) 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 -- 172.21.15.67:0/2909850824 learned_addr learned my addr 172.21.15.67:0/2909850824 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7816800640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 0x7f78100cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815400640 1 --2- 172.21.15.67:0/2909850824 >> v2:172.21.15.86:3300/0 conn(0x7f781009a160 0x7f781015c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299495407 0 0) 0x7f78100cc500 con 0x7f781015b930 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f8003530 con 0x7f781015b930 2024-04-17T10:25:57.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 85739439 0 0) 0x7f77f8003530 con 0x7f781015b930 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 msgr2=0x7f78100cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 0x7f78100cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 >> v2:172.21.15.86:3300/0 conn(0x7f781009a160 msgr2=0x7f781015c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 --2- 172.21.15.67:0/2909850824 >> v2:172.21.15.86:3300/0 conn(0x7f781009a160 0x7f781015c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78100cde60 con 0x7f781015b930 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f7804001680 con 0x7f781015b930 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 >> v1:172.21.15.67:6789/0 conn(0x7f781015b930 legacy=0x7f7810099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f77f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 0x7f77f8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f8004d50 0x7f77f80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78100cc7e0 con 0x7f77f80039c0 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78100ccac0 con 0x7f77f8004d50 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f800b9d0 con 0x7f78100cd2c0 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f77f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815400640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 0x7f77f8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7816800640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f8004d50 0x7f77f80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 -- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 msgr2=0x7f77f8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78100cd2c0 0x7f77f8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 -- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f8004d50 msgr2=0x7f77f80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77f8004d50 0x7f77f80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.979+0000 7f7815e00640 1 -- 172.21.15.67:0/2909850824 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f8003530 con 0x7f77f80039c0 2024-04-17T10:25:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7815e00640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f77f8003de0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f780400c270 tx=0x7f780401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de67a345d28578db server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804007d60 con 0x7f77f80039c0 2024-04-17T10:25:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78040a9020 con 0x7f77f80039c0 2024-04-17T10:25:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7814a00640 1 -- 172.21.15.67:0/2909850824 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804005ae0 con 0x7f77f80039c0 2024-04-17T10:25:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7817200640 1 -- 172.21.15.67:0/2909850824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 msgr2=0x7f77f8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7817200640 1 --2- 172.21.15.67:0/2909850824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f77f8003de0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f780400c270 tx=0x7f780401ccc0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7816800640 1 -- 172.21.15.67:0/2909850824 reap_dead start 2024-04-17T10:25:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7817200640 1 -- 172.21.15.67:0/2909850824 shutdown_connections 2024-04-17T10:25:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7817200640 1 -- 172.21.15.67:0/2909850824 >> 172.21.15.67:0/2909850824 conn(0x7f78100d1630 msgr2=0x7f78100d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7817200640 1 -- 172.21.15.67:0/2909850824 shutdown_connections 2024-04-17T10:25:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.983+0000 7f7817200640 1 -- 172.21.15.67:0/2909850824 wait complete. 2024-04-17T10:25:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 Processor -- start 2024-04-17T10:25:57.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 -- start start 2024-04-17T10:25:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f78100b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781009e740 0x7f7810159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781015a770 0x7f781009d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78100cfda0 con 0x7f77f80039c0 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78100cfc20 con 0x7f781009e740 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78100cff20 con 0x7f781015a770 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f78100b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f78100b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37210/0 (socket says 172.21.15.67:37210) 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 -- 172.21.15.67:0/158748637 learned_addr learned my addr 172.21.15.67:0/158748637 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815400640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781009e740 0x7f7810159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7816800640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781015a770 0x7f781009d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 -- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781015a770 msgr2=0x7f781009d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781015a770 0x7f781009d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 -- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781009e740 msgr2=0x7f7810159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781009e740 0x7f7810159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 -- 172.21.15.67:0/158748637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f781009b750 con 0x7f77f80039c0 2024-04-17T10:25:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815400640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781009e740 0x7f7810159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7815e00640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f78100b4580 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f780400c270 tx=0x7f77f8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f780a000640 1 -- 172.21.15.67:0/158748637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7804004c00 con 0x7f77f80039c0 2024-04-17T10:25:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f780a000640 1 -- 172.21.15.67:0/158748637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7804004da0 con 0x7f77f80039c0 2024-04-17T10:25:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f780a000640 1 -- 172.21.15.67:0/158748637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78040072d0 con 0x7f77f80039c0 2024-04-17T10:25:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f781009ba30 con 0x7f77f80039c0 2024-04-17T10:25:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f781009e220 con 0x7f77f80039c0 2024-04-17T10:25:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.987+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f781004f780 con 0x7f77f80039c0 2024-04-17T10:25:57.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.991+0000 7f780a000640 1 -- 172.21.15.67:0/158748637 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7804007470 con 0x7f77f80039c0 2024-04-17T10:25:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.991+0000 7f780a000640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f4066f90 0x7f77f4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.991+0000 7f780a000640 1 -- 172.21.15.67:0/158748637 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f780411ed80 con 0x7f77f80039c0 2024-04-17T10:25:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.991+0000 7f7815400640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f4066f90 0x7f77f4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.991+0000 7f7815400640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f4066f90 0x7f77f4069450 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7810070260 tx=0x7f7800008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:57.995+0000 7f780a000640 1 -- 172.21.15.67:0/158748637 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f78040be060 con 0x7f77f80039c0 2024-04-17T10:25:58.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78100d2440 con 0x7f77f80039c0 2024-04-17T10:25:58.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f780a000640 1 -- 172.21.15.67:0/158748637 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78040169d0 con 0x7f77f80039c0 2024-04-17T10:25:58.162 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:25:58.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f4066f90 msgr2=0x7f77f4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f4066f90 0x7f77f4069450 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7810070260 tx=0x7f7800008040 comp rx=0 tx=0).stop 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 msgr2=0x7f78100b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f78100b4580 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f780400c270 tx=0x7f77f8005840 comp rx=0 tx=0).stop 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 shutdown_connections 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f77f4066f90 0x7f77f4069450 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f781015a770 0x7f781009d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f781009e740 0x7f7810159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 --2- 172.21.15.67:0/158748637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77f80039c0 0x7f78100b4580 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 >> 172.21.15.67:0/158748637 conn(0x7f78100d1630 msgr2=0x7f781006f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 shutdown_connections 2024-04-17T10:25:58.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.167+0000 7f7817200640 1 -- 172.21.15.67:0/158748637 wait complete. 2024-04-17T10:25:58.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:25:58.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.331+0000 7ff9ed800640 1 Processor -- start 2024-04-17T10:25:58.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.331+0000 7ff9ed800640 1 -- start start 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9ed800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e8161f90 0x7ff9e80cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9ed800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff9e809adf0 0x7ff9e80d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9ed800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9e815e140 con 0x7ff9e815dd60 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9ed800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e81626d0 con 0x7ff9e809adf0 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9ed800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8099d50 con 0x7ff9e8161f90 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e7400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff9e809adf0 0x7ff9e80d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9dfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e8161f90 0x7ff9e80cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e7400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff9e809adf0 0x7ff9e80d04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:45840/0 (socket says 172.21.15.67:45840) 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9dfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e8161f90 0x7ff9e80cff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:48078/0 (socket says 172.21.15.67:48078) 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e7400640 1 -- 172.21.15.67:0/1642496153 learned_addr learned my addr 172.21.15.67:0/1642496153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:58.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e6a00640 1 --1- 172.21.15.67:0/1642496153 >> v1:172.21.15.86:6789/0 conn(0x7ff9e815dd60 0x7ff9e8161880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:41844/0 (socket says 172.21.15.67:41844) 2024-04-17T10:25:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e7400640 1 -- 172.21.15.67:0/1642496153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e8161f90 msgr2=0x7ff9e80cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e7400640 1 --2- 172.21.15.67:0/1642496153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e8161f90 0x7ff9e80cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e7400640 1 -- 172.21.15.67:0/1642496153 >> v1:172.21.15.86:6789/0 conn(0x7ff9e815dd60 legacy=0x7ff9e8161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e7400640 1 -- 172.21.15.67:0/1642496153 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e809a030 con 0x7ff9e809adf0 2024-04-17T10:25:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.335+0000 7ff9e6000640 1 -- 172.21.15.67:0/1642496153 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311238515 0 0) 0x7ff9e815e140 con 0x7ff9e815dd60 2024-04-17T10:25:58.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9e7400640 1 --2- 172.21.15.67:0/1642496153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 0x7ff9e80d04d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009e50 tx=0x7ff9e001ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e15edb8cafdb92c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9e6000640 1 -- 172.21.15.67:0/1642496153 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00a57f0 con 0x7ff9e809adf0 2024-04-17T10:25:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9e6000640 1 -- 172.21.15.67:0/1642496153 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e00a5990 con 0x7ff9e809adf0 2024-04-17T10:25:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9e6000640 1 -- 172.21.15.67:0/1642496153 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00a5d60 con 0x7ff9e809adf0 2024-04-17T10:25:58.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 -- 172.21.15.67:0/1642496153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 msgr2=0x7ff9e80d04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 --2- 172.21.15.67:0/1642496153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 0x7ff9e80d04d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009e50 tx=0x7ff9e001ca30 comp rx=0 tx=0).stop 2024-04-17T10:25:58.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 -- 172.21.15.67:0/1642496153 shutdown_connections 2024-04-17T10:25:58.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 --2- 172.21.15.67:0/1642496153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 0x7ff9e80d04d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 --2- 172.21.15.67:0/1642496153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e8161f90 0x7ff9e80cff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 -- 172.21.15.67:0/1642496153 >> 172.21.15.67:0/1642496153 conn(0x7ff9e80b34a0 msgr2=0x7ff9e80b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 -- 172.21.15.67:0/1642496153 shutdown_connections 2024-04-17T10:25:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 -- 172.21.15.67:0/1642496153 wait complete. 2024-04-17T10:25:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 Processor -- start 2024-04-17T10:25:58.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.339+0000 7ff9ed800640 1 -- start start 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 0x7ff9e8151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e815dd60 0x7ff9e8159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8161f90 0x7ff9e81554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e806c730 con 0x7ff9e8161f90 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e806c5b0 con 0x7ff9e809adf0 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e806c8b0 con 0x7ff9e815dd60 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8161f90 0x7ff9e81554d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8161f90 0x7ff9e81554d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37216/0 (socket says 172.21.15.67:37216) 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 -- 172.21.15.67:0/3393043607 learned_addr learned my addr 172.21.15.67:0/3393043607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e6a00640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 0x7ff9e8151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9dfe00640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e815dd60 0x7ff9e8159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 -- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e815dd60 msgr2=0x7ff9e8159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e815dd60 0x7ff9e8159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 -- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 msgr2=0x7ff9e8151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 0x7ff9e8151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 -- 172.21.15.67:0/3393043607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e815a080 con 0x7ff9e8161f90 2024-04-17T10:25:58.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9e7400640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8161f90 0x7ff9e81554d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff9e001a660 tx=0x7ff9e00197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:58.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9dea00640 1 -- 172.21.15.67:0/3393043607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00b8670 con 0x7ff9e8161f90 2024-04-17T10:25:58.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9dea00640 1 -- 172.21.15.67:0/3393043607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e00be070 con 0x7ff9e8161f90 2024-04-17T10:25:58.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e815a360 con 0x7ff9e8161f90 2024-04-17T10:25:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9dea00640 1 -- 172.21.15.67:0/3393043607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00b8a70 con 0x7ff9e8161f90 2024-04-17T10:25:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e80b0f60 con 0x7ff9e8161f90 2024-04-17T10:25:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9e804ef30 con 0x7ff9e8161f90 2024-04-17T10:25:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.343+0000 7ff9dea00640 1 -- 172.21.15.67:0/3393043607 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff9e00b8c10 con 0x7ff9e8161f90 2024-04-17T10:25:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.347+0000 7ff9dea00640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9c4066f60 0x7ff9c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.347+0000 7ff9dea00640 1 -- 172.21.15.67:0/3393043607 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff9e01205b0 con 0x7ff9e8161f90 2024-04-17T10:25:58.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.347+0000 7ff9e6a00640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9c4066f60 0x7ff9c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.347+0000 7ff9e6a00640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9c4066f60 0x7ff9c4069420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8007fd0 tx=0x7ff9d8007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.351+0000 7ff9dea00640 1 -- 172.21.15.67:0/3393043607 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff9e0016620 con 0x7ff9e8161f90 2024-04-17T10:25:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.511+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9e80b42d0 con 0x7ff9e8161f90 2024-04-17T10:25:58.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.511+0000 7ff9dea00640 1 -- 172.21.15.67:0/3393043607 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9e00a5960 con 0x7ff9e8161f90 2024-04-17T10:25:58.508 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9c4066f60 msgr2=0x7ff9c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9c4066f60 0x7ff9c4069420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8007fd0 tx=0x7ff9d8007d60 comp rx=0 tx=0).stop 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8161f90 msgr2=0x7ff9e81554d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8161f90 0x7ff9e81554d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff9e001a660 tx=0x7ff9e00197d0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 shutdown_connections 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9c4066f60 0x7ff9c4069420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9e8161f90 0x7ff9e81554d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9e815dd60 0x7ff9e8159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 --2- 172.21.15.67:0/3393043607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9e809adf0 0x7ff9e8151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 >> 172.21.15.67:0/3393043607 conn(0x7ff9e80b34a0 msgr2=0x7ff9e8156d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 shutdown_connections 2024-04-17T10:25:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.515+0000 7ff9ed800640 1 -- 172.21.15.67:0/3393043607 wait complete. 2024-04-17T10:25:58.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:25:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c26600640 1 Processor -- start 2024-04-17T10:25:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c26600640 1 -- start start 2024-04-17T10:25:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c26600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c26600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c2015be70 0x7f1c2015c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c26600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2015cb30 con 0x7f1c20159440 2024-04-17T10:25:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c26600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c200a8650 con 0x7f1c2015be70 2024-04-17T10:25:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c26600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2015ce10 con 0x7f1c20071af0 2024-04-17T10:25:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c2015be70 0x7f1c2015c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c16a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1c2015be70 0x7f1c2015c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:45868/0 (socket says 172.21.15.67:45868) 2024-04-17T10:25:58.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1fe00640 1 -- 172.21.15.67:0/2615155130 learned_addr learned my addr 172.21.15.67:0/2615155130 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1f400640 1 --1- 172.21.15.67:0/2615155130 >> v1:172.21.15.86:6789/0 conn(0x7f1c20159440 0x7f1c200713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:41854/0 (socket says 172.21.15.67:41854) 2024-04-17T10:25:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1fe00640 1 -- 172.21.15.67:0/2615155130 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 msgr2=0x7f1c200ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1fe00640 1 --2- 172.21.15.67:0/2615155130 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1fe00640 1 -- 172.21.15.67:0/2615155130 >> v1:172.21.15.86:6789/0 conn(0x7f1c20159440 legacy=0x7f1c200713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.679+0000 7f1c1fe00640 1 -- 172.21.15.67:0/2615155130 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c200cd610 con 0x7f1c2015be70 2024-04-17T10:25:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c1fe00640 1 --2- 172.21.15.67:0/2615155130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2015be70 0x7f1c2015c280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1c18009e50 tx=0x7f1c1801ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b20ac43079844b56 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c1ea00640 1 -- 172.21.15.67:0/2615155130 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180a5900 con 0x7f1c2015be70 2024-04-17T10:25:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c1ea00640 1 -- 172.21.15.67:0/2615155130 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c180a5aa0 con 0x7f1c2015be70 2024-04-17T10:25:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c1ea00640 1 -- 172.21.15.67:0/2615155130 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180ad3f0 con 0x7f1c2015be70 2024-04-17T10:25:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 -- 172.21.15.67:0/2615155130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2015be70 msgr2=0x7f1c2015c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 --2- 172.21.15.67:0/2615155130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2015be70 0x7f1c2015c280 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1c18009e50 tx=0x7f1c1801ca30 comp rx=0 tx=0).stop 2024-04-17T10:25:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 -- 172.21.15.67:0/2615155130 shutdown_connections 2024-04-17T10:25:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 --2- 172.21.15.67:0/2615155130 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c2015be70 0x7f1c2015c280 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 --2- 172.21.15.67:0/2615155130 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 -- 172.21.15.67:0/2615155130 >> 172.21.15.67:0/2615155130 conn(0x7f1c200b1740 msgr2=0x7f1c200b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:58.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 -- 172.21.15.67:0/2615155130 shutdown_connections 2024-04-17T10:25:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.683+0000 7f1c26600640 1 -- 172.21.15.67:0/2615155130 wait complete. 2024-04-17T10:25:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 Processor -- start 2024-04-17T10:25:58.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 -- start start 2024-04-17T10:25:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c20159440 0x7f1c20157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015be70 0x7f1c20140010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2015d840 con 0x7f1c2015be70 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2015c7c0 con 0x7f1c20159440 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2015d9c0 con 0x7f1c20071af0 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015be70 0x7f1c20140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015be70 0x7f1c20140010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37228/0 (socket says 172.21.15.67:37228) 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 -- 172.21.15.67:0/177024294 learned_addr learned my addr 172.21.15.67:0/177024294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1f400640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c16a00640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c20159440 0x7f1c20157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 -- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 msgr2=0x7f1c200a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200a92e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 -- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c20159440 msgr2=0x7f1c20157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c20159440 0x7f1c20157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 -- 172.21.15.67:0/177024294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c20158c60 con 0x7f1c2015be70 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c16a00640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c20159440 0x7f1c20157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1f400640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:25:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c1fe00640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015be70 0x7f1c20140010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1c180af040 tx=0x7f1c180197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c17e00640 1 -- 172.21.15.67:0/177024294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180b8470 con 0x7f1c2015be70 2024-04-17T10:25:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c17e00640 1 -- 172.21.15.67:0/177024294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c180b8630 con 0x7f1c2015be70 2024-04-17T10:25:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c17e00640 1 -- 172.21.15.67:0/177024294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c180b8a60 con 0x7f1c2015be70 2024-04-17T10:25:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c20157bc0 con 0x7f1c2015be70 2024-04-17T10:25:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.687+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c20158180 con 0x7f1c2015be70 2024-04-17T10:25:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.691+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2004ef30 con 0x7f1c2015be70 2024-04-17T10:25:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.691+0000 7f1c17e00640 1 -- 172.21.15.67:0/177024294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1c180c0030 con 0x7f1c2015be70 2024-04-17T10:25:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.691+0000 7f1c17e00640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bfc066f60 0x7f1bfc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.691+0000 7f1c17e00640 1 -- 172.21.15.67:0/177024294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f1c1811ff40 con 0x7f1c2015be70 2024-04-17T10:25:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.691+0000 7f1c1f400640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bfc066f60 0x7f1bfc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:58.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.691+0000 7f1c1f400640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bfc066f60 0x7f1bfc069420 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c10000f70 tx=0x7f1c10007d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.695+0000 7f1c17e00640 1 -- 172.21.15.67:0/177024294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1c180c7020 con 0x7f1c2015be70 2024-04-17T10:25:58.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.855+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1c2006f000 con 0x7f1c2015be70 2024-04-17T10:25:58.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c17e00640 1 -- 172.21.15.67:0/177024294 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1c18016620 con 0x7f1c2015be70 2024-04-17T10:25:58.852 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bfc066f60 msgr2=0x7f1bfc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bfc066f60 0x7f1bfc069420 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c10000f70 tx=0x7f1c10007d70 comp rx=0 tx=0).stop 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015be70 msgr2=0x7f1c20140010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015be70 0x7f1c20140010 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1c180af040 tx=0x7f1c180197d0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 shutdown_connections 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bfc066f60 0x7f1bfc069420 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c2015be70 0x7f1c20140010 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c20159440 0x7f1c20157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 --2- 172.21.15.67:0/177024294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c20071af0 0x7f1c200a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 >> 172.21.15.67:0/177024294 conn(0x7f1c200b1740 msgr2=0x7f1c2015eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 shutdown_connections 2024-04-17T10:25:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:58.859+0000 7f1c26600640 1 -- 172.21.15.67:0/177024294 wait complete. 2024-04-17T10:25:58.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:25:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f9282400640 1 Processor -- start 2024-04-17T10:25:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f9282400640 1 -- start start 2024-04-17T10:25:59.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f9282400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f9282400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f927c0d06a0 0x7f927c161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f9282400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c15e140 con 0x7f927c09adf0 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f9282400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927c1626d0 con 0x7f927c0d06a0 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f9282400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c099d50 con 0x7f927c15dd60 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f927c0d06a0 0x7f927c161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f927c0d06a0 0x7f927c161dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37244/0 (socket says 172.21.15.67:37244) 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927be00640 1 -- 172.21.15.67:0/2381382118 learned_addr learned my addr 172.21.15.67:0/2381382118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927b400640 1 --2- 172.21.15.67:0/2381382118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927be00640 1 -- 172.21.15.67:0/2381382118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 msgr2=0x7f927c161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927be00640 1 --2- 172.21.15.67:0/2381382118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927be00640 1 -- 172.21.15.67:0/2381382118 >> v1:172.21.15.67:6789/0 conn(0x7f927c09adf0 legacy=0x7f927c0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.023+0000 7f927be00640 1 -- 172.21.15.67:0/2381382118 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c09a030 con 0x7f927c0d06a0 2024-04-17T10:25:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f927be00640 1 --2- 172.21.15.67:0/2381382118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c161dc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9274009e50 tx=0x7f927401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2214b84a747f96b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f927aa00640 1 -- 172.21.15.67:0/2381382118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740a58f0 con 0x7f927c0d06a0 2024-04-17T10:25:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f927aa00640 1 -- 172.21.15.67:0/2381382118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92740a5a90 con 0x7f927c0d06a0 2024-04-17T10:25:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f927aa00640 1 -- 172.21.15.67:0/2381382118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740a5e60 con 0x7f927c0d06a0 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 -- 172.21.15.67:0/2381382118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 msgr2=0x7f927c161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 --2- 172.21.15.67:0/2381382118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c161dc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9274009e50 tx=0x7f927401ca00 comp rx=0 tx=0).stop 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 -- 172.21.15.67:0/2381382118 shutdown_connections 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 --2- 172.21.15.67:0/2381382118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c161dc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 --2- 172.21.15.67:0/2381382118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 -- 172.21.15.67:0/2381382118 >> 172.21.15.67:0/2381382118 conn(0x7f927c0b34a0 msgr2=0x7f927c0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 -- 172.21.15.67:0/2381382118 shutdown_connections 2024-04-17T10:25:59.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.027+0000 7f9282400640 1 -- 172.21.15.67:0/2381382118 wait complete. 2024-04-17T10:25:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 Processor -- start 2024-04-17T10:25:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 -- start start 2024-04-17T10:25:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c09adf0 0x7f927c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c1476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c06c730 con 0x7f927c0d06a0 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c06c5b0 con 0x7f927c09adf0 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c06c8b0 con 0x7f927c15dd60 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c1609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37256/0 (socket says 172.21.15.67:37256) 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 -- 172.21.15.67:0/2736033052 learned_addr learned my addr 172.21.15.67:0/2736033052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f927be00640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c1476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f927b400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c09adf0 0x7f927c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 -- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 msgr2=0x7f927c1476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c1476f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 -- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c09adf0 msgr2=0x7f927c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c09adf0 0x7f927c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 -- 172.21.15.67:0/2736033052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c159810 con 0x7f927c0d06a0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9272a00640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c1609f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f926400c450 tx=0x7f926400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9273e00640 1 -- 172.21.15.67:0/2736033052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264016020 con 0x7f927c0d06a0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9273e00640 1 -- 172.21.15.67:0/2736033052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92640052d0 con 0x7f927c0d06a0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9273e00640 1 -- 172.21.15.67:0/2736033052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264007260 con 0x7f927c0d06a0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c159af0 con 0x7f927c0d06a0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c15a0b0 con 0x7f927c0d06a0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.031+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927c04ef30 con 0x7f927c0d06a0 2024-04-17T10:25:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.035+0000 7f9273e00640 1 -- 172.21.15.67:0/2736033052 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9264007400 con 0x7f927c0d06a0 2024-04-17T10:25:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.035+0000 7f9273e00640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9258066f60 0x7f9258069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.035+0000 7f9273e00640 1 -- 172.21.15.67:0/2736033052 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f9264084c30 con 0x7f927c0d06a0 2024-04-17T10:25:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.035+0000 7f927b400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9258066f60 0x7f9258069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.035+0000 7f927b400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9258066f60 0x7f9258069420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f926c009ac0 tx=0x7f926c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.039+0000 7f9273e00640 1 -- 172.21.15.67:0/2736033052 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9264020020 con 0x7f927c0d06a0 2024-04-17T10:25:59.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.199+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f927c1624b0 con 0x7f927c0d06a0 2024-04-17T10:25:59.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.199+0000 7f9273e00640 1 -- 172.21.15.67:0/2736033052 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9264005480 con 0x7f927c0d06a0 2024-04-17T10:25:59.196 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9258066f60 msgr2=0x7f9258069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9258066f60 0x7f9258069420 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f926c009ac0 tx=0x7f926c009190 comp rx=0 tx=0).stop 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 msgr2=0x7f927c1609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c1609f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f926400c450 tx=0x7f926400c910 comp rx=0 tx=0).stop 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 shutdown_connections 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9258066f60 0x7f9258069420 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c15dd60 0x7f927c1476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c0d06a0 0x7f927c1609f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 --2- 172.21.15.67:0/2736033052 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c09adf0 0x7f927c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 >> 172.21.15.67:0/2736033052 conn(0x7f927c0b34a0 msgr2=0x7f927c0b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:59.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 shutdown_connections 2024-04-17T10:25:59.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.203+0000 7f9282400640 1 -- 172.21.15.67:0/2736033052 wait complete. 2024-04-17T10:25:59.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:25:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.363+0000 7f1dfd200640 1 Processor -- start 2024-04-17T10:25:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.363+0000 7f1dfd200640 1 -- start start 2024-04-17T10:25:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1dfd200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1df806e6e0 0x7f1df80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1dfd200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df806eac0 0x7f1df80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1dfd200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1df8143ae0 0x7f1df8143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1dfd200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df80cf430 con 0x7f1df8143ae0 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1dfd200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df815ae40 con 0x7f1df806e6e0 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1dfd200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8071440 con 0x7f1df806eac0 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1df8143ae0 0x7f1df8143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1df8143ae0 0x7f1df8143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37270/0 (socket says 172.21.15.67:37270) 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 -- 172.21.15.67:0/2402359667 learned_addr learned my addr 172.21.15.67:0/2402359667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df5e00640 1 --2- 172.21.15.67:0/2402359667 >> v2:172.21.15.86:3300/0 conn(0x7f1df806e6e0 0x7f1df80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1ded400640 1 --2- 172.21.15.67:0/2402359667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df806eac0 0x7f1df80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 -- 172.21.15.67:0/2402359667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df806eac0 msgr2=0x7f1df80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 --2- 172.21.15.67:0/2402359667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df806eac0 0x7f1df80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 -- 172.21.15.67:0/2402359667 >> v2:172.21.15.86:3300/0 conn(0x7f1df806e6e0 msgr2=0x7f1df80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 --2- 172.21.15.67:0/2402359667 >> v2:172.21.15.86:3300/0 conn(0x7f1df806e6e0 0x7f1df80ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 -- 172.21.15.67:0/2402359667 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8071720 con 0x7f1df8143ae0 2024-04-17T10:25:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df5e00640 1 --2- 172.21.15.67:0/2402359667 >> v2:172.21.15.86:3300/0 conn(0x7f1df806e6e0 0x7f1df80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1ded400640 1 --2- 172.21.15.67:0/2402359667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df806eac0 0x7f1df80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:25:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.367+0000 7f1df6800640 1 --2- 172.21.15.67:0/2402359667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df8143ae0 0x7f1df8143f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1df80cf620 tx=0x7f1df001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b472e0f37254672c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1df5400640 1 -- 172.21.15.67:0/2402359667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df00a57b0 con 0x7f1df8143ae0 2024-04-17T10:25:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1df5400640 1 -- 172.21.15.67:0/2402359667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df00a5950 con 0x7f1df8143ae0 2024-04-17T10:25:59.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1df5400640 1 -- 172.21.15.67:0/2402359667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df00a5d20 con 0x7f1df8143ae0 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- 172.21.15.67:0/2402359667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df8143ae0 msgr2=0x7f1df8143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 --2- 172.21.15.67:0/2402359667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df8143ae0 0x7f1df8143f70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f1df80cf620 tx=0x7f1df001c930 comp rx=0 tx=0).stop 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- 172.21.15.67:0/2402359667 shutdown_connections 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 --2- 172.21.15.67:0/2402359667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df8143ae0 0x7f1df8143f70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 --2- 172.21.15.67:0/2402359667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df806eac0 0x7f1df80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 --2- 172.21.15.67:0/2402359667 >> v2:172.21.15.86:3300/0 conn(0x7f1df806e6e0 0x7f1df80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- 172.21.15.67:0/2402359667 >> 172.21.15.67:0/2402359667 conn(0x7f1df806ac90 msgr2=0x7f1df806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- 172.21.15.67:0/2402359667 shutdown_connections 2024-04-17T10:25:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- 172.21.15.67:0/2402359667 wait complete. 2024-04-17T10:25:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 Processor -- start 2024-04-17T10:25:59.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- start start 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806e6e0 0x7f1df80b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1df806eac0 0x7f1df8161920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df8143ae0 0x7f1df8164b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df815ebf0 con 0x7f1df806e6e0 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df81444b0 con 0x7f1df806eac0 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1dfd200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df815ed70 con 0x7f1df8143ae0 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1df5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806e6e0 0x7f1df80b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1df5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806e6e0 0x7f1df80b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37284/0 (socket says 172.21.15.67:37284) 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1df5e00640 1 -- 172.21.15.67:0/705745008 learned_addr learned my addr 172.21.15.67:0/705745008 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1ded400640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1df806eac0 0x7f1df8161920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.371+0000 7f1df6800640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df8143ae0 0x7f1df8164b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1df5e00640 1 -- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df8143ae0 msgr2=0x7f1df8164b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1df5e00640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df8143ae0 0x7f1df8164b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1df5e00640 1 -- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1df806eac0 msgr2=0x7f1df8161920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1df5e00640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1df806eac0 0x7f1df8161920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1df5e00640 1 -- 172.21.15.67:0/705745008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8163970 con 0x7f1df806e6e0 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1df5e00640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806e6e0 0x7f1df80b1b40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c7f0 tx=0x7f1de800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1deea00640 1 -- 172.21.15.67:0/705745008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de8007a30 con 0x7f1df806e6e0 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1deea00640 1 -- 172.21.15.67:0/705745008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de8007bd0 con 0x7f1df806e6e0 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df8163c50 con 0x7f1df806e6e0 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1deea00640 1 -- 172.21.15.67:0/705745008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80113f0 con 0x7f1df806e6e0 2024-04-17T10:25:59.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df8164210 con 0x7f1df806e6e0 2024-04-17T10:25:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df8072d50 con 0x7f1df806e6e0 2024-04-17T10:25:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1deea00640 1 -- 172.21.15.67:0/705745008 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1de8011590 con 0x7f1df806e6e0 2024-04-17T10:25:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1deea00640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd4066f60 0x7f1dd4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.375+0000 7f1deea00640 1 -- 172.21.15.67:0/705745008 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f1de8085710 con 0x7f1df806e6e0 2024-04-17T10:25:59.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.379+0000 7f1ded400640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd4066f60 0x7f1dd4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.379+0000 7f1ded400640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd4066f60 0x7f1dd4069420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006fd0 tx=0x7f1de0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.379+0000 7f1deea00640 1 -- 172.21.15.67:0/705745008 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1de8015020 con 0x7f1df806e6e0 2024-04-17T10:25:59.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.555+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1df8164630 con 0x7f1df806e6e0 2024-04-17T10:25:59.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.555+0000 7f1deea00640 1 -- 172.21.15.67:0/705745008 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1de8056280 con 0x7f1df806e6e0 2024-04-17T10:25:59.551 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd4066f60 msgr2=0x7f1dd4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd4066f60 0x7f1dd4069420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1de0006fd0 tx=0x7f1de0008040 comp rx=0 tx=0).stop 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806e6e0 msgr2=0x7f1df80b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806e6e0 0x7f1df80b1b40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1de800c7f0 tx=0x7f1de800ccb0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 shutdown_connections 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd4066f60 0x7f1dd4069420 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1df8143ae0 0x7f1df8164b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1df806eac0 0x7f1df8161920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 --2- 172.21.15.67:0/705745008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1df806e6e0 0x7f1df80b1b40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 >> 172.21.15.67:0/705745008 conn(0x7f1df806ac90 msgr2=0x7f1df806b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 shutdown_connections 2024-04-17T10:25:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.559+0000 7f1dfd200640 1 -- 172.21.15.67:0/705745008 wait complete. 2024-04-17T10:25:59.572 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-17T10:25:59.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:59.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-17T10:25:59.599 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi086.front.sepia.ceph.com 2024-04-17T10:25:59.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:25:59.599 DEBUG:teuthology.orchestra.run.smithi086:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-17T10:25:59.649 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:25:59.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:25:59.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735be00640 1 Processor -- start 2024-04-17T10:25:59.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735be00640 1 -- start start 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c161160 0x7f735c099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f735c0b2570 0x7f735c099f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f735c0d0760 con 0x7f735c1616c0 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f735c0cfc70 con 0x7f735c0b2570 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735be00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0cc5d0 con 0x7f735c161160 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f7352000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f735c0b2570 0x7f735c099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f7352000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f735c0b2570 0x7f735c099f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37298/0 (socket says 172.21.15.67:37298) 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f7352000640 1 -- 172.21.15.67:0/3542754629 learned_addr learned my addr 172.21.15.67:0/3542754629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f735aa00640 1 --2- 172.21.15.67:0/3542754629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c161160 0x7f735c099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f7352000640 1 -- 172.21.15.67:0/3542754629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c161160 msgr2=0x7f735c099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:25:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f7352000640 1 --2- 172.21.15.67:0/3542754629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c161160 0x7f735c099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f7352000640 1 -- 172.21.15.67:0/3542754629 >> v1:172.21.15.67:6789/0 conn(0x7f735c1616c0 legacy=0x7f735c161b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.795+0000 7f7352000640 1 -- 172.21.15.67:0/3542754629 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0cc2a0 con 0x7f735c0b2570 2024-04-17T10:25:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f7352000640 1 --2- 172.21.15.67:0/3542754629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c0b2570 0x7f735c099f90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7344009870 tx=0x7f734401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d6ac26fe2674d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735a000640 1 -- 172.21.15.67:0/3542754629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73440a7070 con 0x7f735c0b2570 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735a000640 1 -- 172.21.15.67:0/3542754629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73440b0070 con 0x7f735c0b2570 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735a000640 1 -- 172.21.15.67:0/3542754629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73440ab890 con 0x7f735c0b2570 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 -- 172.21.15.67:0/3542754629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c0b2570 msgr2=0x7f735c099f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 --2- 172.21.15.67:0/3542754629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c0b2570 0x7f735c099f90 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7344009870 tx=0x7f734401c560 comp rx=0 tx=0).stop 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 -- 172.21.15.67:0/3542754629 shutdown_connections 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 --2- 172.21.15.67:0/3542754629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c0b2570 0x7f735c099f90 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 --2- 172.21.15.67:0/3542754629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c161160 0x7f735c099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 -- 172.21.15.67:0/3542754629 >> 172.21.15.67:0/3542754629 conn(0x7f735c0d1690 msgr2=0x7f735c0d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 -- 172.21.15.67:0/3542754629 shutdown_connections 2024-04-17T10:25:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.799+0000 7f735be00640 1 -- 172.21.15.67:0/3542754629 wait complete. 2024-04-17T10:25:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 Processor -- start 2024-04-17T10:25:59.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 -- start start 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c0b2570 0x7f735c0cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735c161160 0x7f735c0c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c1616c0 0x7f735c153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06bda0 con 0x7f735c1616c0 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c0b2bc0 con 0x7f735c161160 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f735c06bf20 con 0x7f735c0b2570 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c1616c0 0x7f735c153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c1616c0 0x7f735c153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37300/0 (socket says 172.21.15.67:37300) 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 -- 172.21.15.67:0/1319098608 learned_addr learned my addr 172.21.15.67:0/1319098608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:25:59.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735aa00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c0b2570 0x7f735c0cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f7352000640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735c161160 0x7f735c0c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 -- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c0b2570 msgr2=0x7f735c0cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c0b2570 0x7f735c0cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 -- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735c161160 msgr2=0x7f735c0c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:25:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735c161160 0x7f735c0c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:25:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 -- 172.21.15.67:0/1319098608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c0c5990 con 0x7f735c1616c0 2024-04-17T10:25:59.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735b400640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c1616c0 0x7f735c153f90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f735400c730 tx=0x7f735400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f7353e00640 1 -- 172.21.15.67:0/1319098608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354003b80 con 0x7f735c1616c0 2024-04-17T10:25:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f7353e00640 1 -- 172.21.15.67:0/1319098608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7354003d20 con 0x7f735c1616c0 2024-04-17T10:25:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c0c5c70 con 0x7f735c1616c0 2024-04-17T10:25:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f7353e00640 1 -- 172.21.15.67:0/1319098608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400aa30 con 0x7f735c1616c0 2024-04-17T10:25:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.803+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c14b7d0 con 0x7f735c1616c0 2024-04-17T10:25:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.807+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f735c14b5c0 con 0x7f735c1616c0 2024-04-17T10:25:59.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.807+0000 7f7353e00640 1 -- 172.21.15.67:0/1319098608 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f735400abd0 con 0x7f735c1616c0 2024-04-17T10:25:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.807+0000 7f7353e00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7338066f90 0x7f7338069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:25:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.807+0000 7f7353e00640 1 -- 172.21.15.67:0/1319098608 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7354085250 con 0x7f735c1616c0 2024-04-17T10:25:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.807+0000 7f735aa00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7338066f90 0x7f7338069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:25:59.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.807+0000 7f735aa00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7338066f90 0x7f7338069450 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f734c009ac0 tx=0x7f734c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:25:59.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:25:59.811+0000 7f7353e00640 1 -- 172.21.15.67:0/1319098608 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7354055e50 con 0x7f735c1616c0 2024-04-17T10:26:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.011+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f735c074dc0 con 0x7f735c1616c0 2024-04-17T10:26:00.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.011+0000 7f7353e00640 1 -- 172.21.15.67:0/1319098608 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f7354055750 con 0x7f735c1616c0 2024-04-17T10:26:00.006 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:00.007 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:26:00.007 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:26:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.011+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7338066f90 msgr2=0x7f7338069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7338066f90 0x7f7338069450 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f734c009ac0 tx=0x7f734c009190 comp rx=0 tx=0).stop 2024-04-17T10:26:00.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c1616c0 msgr2=0x7f735c153f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c1616c0 0x7f735c153f90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f735400c730 tx=0x7f735400cbf0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 shutdown_connections 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7338066f90 0x7f7338069450 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f735c1616c0 0x7f735c153f90 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f735c161160 0x7f735c0c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 --2- 172.21.15.67:0/1319098608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f735c0b2570 0x7f735c0cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 >> 172.21.15.67:0/1319098608 conn(0x7f735c0d1690 msgr2=0x7f735c15a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 shutdown_connections 2024-04-17T10:26:00.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.015+0000 7f735be00640 1 -- 172.21.15.67:0/1319098608 wait complete. 2024-04-17T10:26:00.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:26:00.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0680000640 1 Processor -- start 2024-04-17T10:26:00.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0680000640 1 -- start start 2024-04-17T10:26:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0680000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06780714d0 0x7f0678168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0680000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780b5980 con 0x7f06780b07b0 2024-04-17T10:26:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0680000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06780b0cc0 con 0x7f067815dd10 2024-04-17T10:26:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0680000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06780b1280 con 0x7f06780714d0 2024-04-17T10:26:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f067cc00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06780b07b0 0x7f0678070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33210/0 (socket says 172.21.15.67:33210) 2024-04-17T10:26:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f067cc00640 1 -- 172.21.15.67:0/3783520403 learned_addr learned my addr 172.21.15.67:0/3783520403 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f067e000640 1 --2- 172.21.15.67:0/3783520403 >> v2:172.21.15.86:3300/0 conn(0x7f06780714d0 0x7f0678168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0677e00640 1 -- 172.21.15.67:0/3783520403 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865787853 0 0) 0x7f06780b5980 con 0x7f06780b07b0 2024-04-17T10:26:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0677e00640 1 -- 172.21.15.67:0/3783520403 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0660003530 con 0x7f06780b07b0 2024-04-17T10:26:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0677e00640 1 -- 172.21.15.67:0/3783520403 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974457850 0 0) 0x7f06780b0cc0 con 0x7f067815dd10 2024-04-17T10:26:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f0677e00640 1 -- 172.21.15.67:0/3783520403 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06780b5980 con 0x7f067815dd10 2024-04-17T10:26:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f067e000640 1 -- 172.21.15.67:0/3783520403 >> v1:172.21.15.86:6789/0 conn(0x7f067815dd10 legacy=0x7f06780b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f067e000640 1 -- 172.21.15.67:0/3783520403 >> v1:172.21.15.67:6789/0 conn(0x7f06780b07b0 legacy=0x7f0678070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.175+0000 7f067e000640 1 -- 172.21.15.67:0/3783520403 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780b0fa0 con 0x7f06780714d0 2024-04-17T10:26:00.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f067e000640 1 --2- 172.21.15.67:0/3783520403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06780714d0 0x7f0678168c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0670009d90 tx=0x7f067001c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77a2df16ed3f049c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0677e00640 1 -- 172.21.15.67:0/3783520403 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700b2070 con 0x7f06780714d0 2024-04-17T10:26:00.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0677e00640 1 -- 172.21.15.67:0/3783520403 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06700a5970 con 0x7f06780714d0 2024-04-17T10:26:00.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0677e00640 1 -- 172.21.15.67:0/3783520403 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06700a5d40 con 0x7f06780714d0 2024-04-17T10:26:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0680000640 1 -- 172.21.15.67:0/3783520403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06780714d0 msgr2=0x7f0678168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0680000640 1 --2- 172.21.15.67:0/3783520403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06780714d0 0x7f0678168c50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0670009d90 tx=0x7f067001c8b0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0680000640 1 -- 172.21.15.67:0/3783520403 shutdown_connections 2024-04-17T10:26:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0680000640 1 --2- 172.21.15.67:0/3783520403 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06780714d0 0x7f0678168c50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0680000640 1 -- 172.21.15.67:0/3783520403 >> 172.21.15.67:0/3783520403 conn(0x7f06780b34d0 msgr2=0x7f06780b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0680000640 1 -- 172.21.15.67:0/3783520403 shutdown_connections 2024-04-17T10:26:00.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.179+0000 7f0680000640 1 -- 172.21.15.67:0/3783520403 wait complete. 2024-04-17T10:26:00.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 Processor -- start 2024-04-17T10:26:00.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 -- start start 2024-04-17T10:26:00.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780714d0 0x7f0678158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780b07b0 0x7f0678156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f067815dd10 0x7f0678156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067816bb30 con 0x7f06780b07b0 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067816b9b0 con 0x7f067815dd10 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067816bcb0 con 0x7f06780714d0 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780714d0 0x7f0678158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f067815dd10 0x7f0678156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780b07b0 0x7f0678156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f067815dd10 0x7f0678156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:45932/0 (socket says 172.21.15.67:45932) 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780714d0 0x7f0678158770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:48172/0 (socket says 172.21.15.67:48172) 2024-04-17T10:26:00.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067e000640 1 -- 172.21.15.67:0/1875786161 learned_addr learned my addr 172.21.15.67:0/1875786161 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 -- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f067815dd10 msgr2=0x7f0678156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f067815dd10 0x7f0678156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 -- 172.21.15.67:0/1875786161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780b07b0 msgr2=0x7f0678156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780b07b0 0x7f0678156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 -- 172.21.15.67:0/1875786161 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0678158ed0 con 0x7f06780714d0 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067e000640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f067815dd10 0x7f0678156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f067d600640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780714d0 0x7f0678158770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f066c00c3f0 tx=0x7f066c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0675600640 1 -- 172.21.15.67:0/1875786161 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c017070 con 0x7f06780714d0 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0675600640 1 -- 172.21.15.67:0/1875786161 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c0052d0 con 0x7f06780714d0 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0675600640 1 -- 172.21.15.67:0/1875786161 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f066c003840 con 0x7f06780714d0 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06781591b0 con 0x7f06780714d0 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.183+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0678159770 con 0x7f06780714d0 2024-04-17T10:26:00.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.187+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0678070bb0 con 0x7f06780714d0 2024-04-17T10:26:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.187+0000 7f0675600640 1 -- 172.21.15.67:0/1875786161 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f066c01a070 con 0x7f06780714d0 2024-04-17T10:26:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.187+0000 7f0675600640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0658066f60 0x7f0658069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.187+0000 7f0675600640 1 -- 172.21.15.67:0/1875786161 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f066c0846f0 con 0x7f06780714d0 2024-04-17T10:26:00.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.187+0000 7f067cc00640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0658066f60 0x7f0658069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.187+0000 7f067cc00640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0658066f60 0x7f0658069420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f06781524c0 tx=0x7f0664007d80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.199+0000 7f0675600640 1 -- 172.21.15.67:0/1875786161 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f066c005480 con 0x7f06780714d0 2024-04-17T10:26:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.363+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06780cdfd0 con 0x7f06780714d0 2024-04-17T10:26:00.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0675600640 1 -- 172.21.15.67:0/1875786161 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f066c054ca0 con 0x7f06780714d0 2024-04-17T10:26:00.361 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:00.361 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0658066f60 msgr2=0x7f0658069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0658066f60 0x7f0658069420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f06781524c0 tx=0x7f0664007d80 comp rx=0 tx=0).stop 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780714d0 msgr2=0x7f0678158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780714d0 0x7f0678158770 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f066c00c3f0 tx=0x7f066c00c8b0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 shutdown_connections 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0658066f60 0x7f0658069420 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f067815dd10 0x7f0678156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780b07b0 0x7f0678156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 --2- 172.21.15.67:0/1875786161 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780714d0 0x7f0678158770 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 >> 172.21.15.67:0/1875786161 conn(0x7f06780b34d0 msgr2=0x7f06780b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:00.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 shutdown_connections 2024-04-17T10:26:00.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.367+0000 7f0680000640 1 -- 172.21.15.67:0/1875786161 wait complete. 2024-04-17T10:26:00.382 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:24:53.220996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:24:53.975318+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-17T10:24:57.660058+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-17T10:26:00.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:26:00.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f705a200640 1 Processor -- start 2024-04-17T10:26:00.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f705a200640 1 -- start start 2024-04-17T10:26:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f705a200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f705415b980 0x7f70540cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f705a200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70540cca60 con 0x7f705415bd60 2024-04-17T10:26:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f705a200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70540ccd40 con 0x7f705409a240 2024-04-17T10:26:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f705a200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70540cd020 con 0x7f705415b980 2024-04-17T10:26:00.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f7053400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f705415b980 0x7f70540cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f7053400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f705415b980 0x7f70540cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35574/0 (socket says 172.21.15.67:35574) 2024-04-17T10:26:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f7053400640 1 -- 172.21.15.67:0/2488956990 learned_addr learned my addr 172.21.15.67:0/2488956990 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f704be00640 1 --1- 172.21.15.67:0/2488956990 >> v1:172.21.15.67:6789/0 conn(0x7f705415bd60 0x7f7054099b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58882/0 (socket says 172.21.15.67:58882) 2024-04-17T10:26:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f7053400640 1 -- 172.21.15.67:0/2488956990 >> v1:172.21.15.86:6789/0 conn(0x7f705409a240 legacy=0x7f70540cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f7053400640 1 -- 172.21.15.67:0/2488956990 >> v1:172.21.15.67:6789/0 conn(0x7f705415bd60 legacy=0x7f7054099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.531+0000 7f7053400640 1 -- 172.21.15.67:0/2488956990 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70540cd300 con 0x7f705415b980 2024-04-17T10:26:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f7053400640 1 --2- 172.21.15.67:0/2488956990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705415b980 0x7f70540cc1f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7044009870 tx=0x7f704401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf0c587f8adde629 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f7052a00640 1 -- 172.21.15.67:0/2488956990 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70440b5070 con 0x7f705415b980 2024-04-17T10:26:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f7052a00640 1 -- 172.21.15.67:0/2488956990 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70440a8a50 con 0x7f705415b980 2024-04-17T10:26:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f7052a00640 1 -- 172.21.15.67:0/2488956990 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70440a8e20 con 0x7f705415b980 2024-04-17T10:26:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f705a200640 1 -- 172.21.15.67:0/2488956990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705415b980 msgr2=0x7f70540cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f705a200640 1 --2- 172.21.15.67:0/2488956990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705415b980 0x7f70540cc1f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7044009870 tx=0x7f704401ed50 comp rx=0 tx=0).stop 2024-04-17T10:26:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f705a200640 1 -- 172.21.15.67:0/2488956990 shutdown_connections 2024-04-17T10:26:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f705a200640 1 --2- 172.21.15.67:0/2488956990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705415b980 0x7f70540cc1f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f705a200640 1 -- 172.21.15.67:0/2488956990 >> 172.21.15.67:0/2488956990 conn(0x7f70540b33a0 msgr2=0x7f70540b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f705a200640 1 -- 172.21.15.67:0/2488956990 shutdown_connections 2024-04-17T10:26:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.535+0000 7f705a200640 1 -- 172.21.15.67:0/2488956990 wait complete. 2024-04-17T10:26:00.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 Processor -- start 2024-04-17T10:26:00.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 -- start start 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705409a240 0x7f705415ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f705415b980 0x7f705414cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f705415bd60 0x7f70540b4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70540cfdf0 con 0x7f705409a240 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70540cc730 con 0x7f705415b980 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70540cff70 con 0x7f705415bd60 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705409a240 0x7f705415ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705409a240 0x7f705415ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35586/0 (socket says 172.21.15.67:35586) 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f704be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f705415b980 0x7f705414cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 -- 172.21.15.67:0/2199990185 learned_addr learned my addr 172.21.15.67:0/2199990185 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053e00640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f705415bd60 0x7f70540b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 -- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f705415bd60 msgr2=0x7f70540b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f705415bd60 0x7f70540b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 -- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f705415b980 msgr2=0x7f705414cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f705415b980 0x7f705414cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 -- 172.21.15.67:0/2199990185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705415d9d0 con 0x7f705409a240 2024-04-17T10:26:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f7053400640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705409a240 0x7f705415ea70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f704401b960 tx=0x7f704401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f704b400640 1 -- 172.21.15.67:0/2199990185 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70440c0070 con 0x7f705409a240 2024-04-17T10:26:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f704b400640 1 -- 172.21.15.67:0/2199990185 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70440ae770 con 0x7f705409a240 2024-04-17T10:26:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f704b400640 1 -- 172.21.15.67:0/2199990185 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70440b5070 con 0x7f705409a240 2024-04-17T10:26:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705415dcb0 con 0x7f705409a240 2024-04-17T10:26:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.539+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705415e270 con 0x7f705409a240 2024-04-17T10:26:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.543+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705404ef30 con 0x7f705409a240 2024-04-17T10:26:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.543+0000 7f704b400640 1 -- 172.21.15.67:0/2199990185 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f70440c3070 con 0x7f705409a240 2024-04-17T10:26:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.543+0000 7f704b400640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7030066f90 0x7f7030069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.543+0000 7f704b400640 1 -- 172.21.15.67:0/2199990185 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f7044122cd0 con 0x7f705409a240 2024-04-17T10:26:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.543+0000 7f704be00640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7030066f90 0x7f7030069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.543+0000 7f704be00640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7030066f90 0x7f7030069450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f70541f4cb0 tx=0x7f703c007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.547+0000 7f704b400640 1 -- 172.21.15.67:0/2199990185 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7044129080 con 0x7f705409a240 2024-04-17T10:26:00.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.707+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f70540b2350 con 0x7f705409a240 2024-04-17T10:26:00.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f704b400640 1 -- 172.21.15.67:0/2199990185 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f70440f9020 con 0x7f705409a240 2024-04-17T10:26:00.706 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:26:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7030066f90 msgr2=0x7f7030069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7030066f90 0x7f7030069450 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f70541f4cb0 tx=0x7f703c007e90 comp rx=0 tx=0).stop 2024-04-17T10:26:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705409a240 msgr2=0x7f705415ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705409a240 0x7f705415ea70 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f704401b960 tx=0x7f704401baf0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 shutdown_connections 2024-04-17T10:26:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7030066f90 0x7f7030069450 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f705415bd60 0x7f70540b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f705415b980 0x7f705414cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 --2- 172.21.15.67:0/2199990185 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f705409a240 0x7f705415ea70 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 >> 172.21.15.67:0/2199990185 conn(0x7f70540b33a0 msgr2=0x7f70540b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 shutdown_connections 2024-04-17T10:26:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.711+0000 7f705a200640 1 -- 172.21.15.67:0/2199990185 wait complete. 2024-04-17T10:26:00.725 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:26:00.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.871+0000 7f142f200640 1 Processor -- start 2024-04-17T10:26:00.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.871+0000 7f142f200640 1 -- start start 2024-04-17T10:26:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14300ad890 0x7f14300aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142f200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f14300af520 0x7f143006bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142f200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f143015c060 con 0x7f143006c1e0 2024-04-17T10:26:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142f200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143006c630 con 0x7f14300af520 2024-04-17T10:26:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143015c340 con 0x7f14300ad890 2024-04-17T10:26:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142d400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f14300af520 0x7f143006bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14300ad890 0x7f14300aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14300ad890 0x7f14300aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:33702/0 (socket says 172.21.15.67:33702) 2024-04-17T10:26:00.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142de00640 1 -- 172.21.15.67:0/2811773833 learned_addr learned my addr 172.21.15.67:0/2811773833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142de00640 1 -- 172.21.15.67:0/2811773833 >> v2:172.21.15.86:3300/0 conn(0x7f14300af520 msgr2=0x7f143006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142de00640 1 --2- 172.21.15.67:0/2811773833 >> v2:172.21.15.86:3300/0 conn(0x7f14300af520 0x7f143006bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142de00640 1 -- 172.21.15.67:0/2811773833 >> v1:172.21.15.86:6789/0 conn(0x7f143006c1e0 legacy=0x7f143015f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142de00640 1 -- 172.21.15.67:0/2811773833 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143015c620 con 0x7f14300ad890 2024-04-17T10:26:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142d400640 1 --2- 172.21.15.67:0/2811773833 >> v2:172.21.15.86:3300/0 conn(0x7f14300af520 0x7f143006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:00.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.875+0000 7f142ca00640 1 -- 172.21.15.67:0/2811773833 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1019101 0 0) 0x7f143015c060 con 0x7f143006c1e0 2024-04-17T10:26:00.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142de00640 1 --2- 172.21.15.67:0/2811773833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14300ad890 0x7f14300aefe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f142001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd2f7335f7aebedb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142ca00640 1 -- 172.21.15.67:0/2811773833 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14200ab610 con 0x7f14300ad890 2024-04-17T10:26:00.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142ca00640 1 -- 172.21.15.67:0/2811773833 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14200b0070 con 0x7f14300ad890 2024-04-17T10:26:00.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142ca00640 1 -- 172.21.15.67:0/2811773833 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14200ab9d0 con 0x7f14300ad890 2024-04-17T10:26:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 -- 172.21.15.67:0/2811773833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14300ad890 msgr2=0x7f14300aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 --2- 172.21.15.67:0/2811773833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14300ad890 0x7f14300aefe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1420009870 tx=0x7f142001c560 comp rx=0 tx=0).stop 2024-04-17T10:26:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 -- 172.21.15.67:0/2811773833 shutdown_connections 2024-04-17T10:26:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 --2- 172.21.15.67:0/2811773833 >> v2:172.21.15.86:3300/0 conn(0x7f14300af520 0x7f143006bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 --2- 172.21.15.67:0/2811773833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14300ad890 0x7f14300aefe0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 -- 172.21.15.67:0/2811773833 >> 172.21.15.67:0/2811773833 conn(0x7f1430071af0 msgr2=0x7f1430071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 -- 172.21.15.67:0/2811773833 shutdown_connections 2024-04-17T10:26:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 -- 172.21.15.67:0/2811773833 wait complete. 2024-04-17T10:26:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 Processor -- start 2024-04-17T10:26:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.879+0000 7f142f200640 1 -- start start 2024-04-17T10:26:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f143006c1e0 0x7f1430159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14300ad890 0x7f14300cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14300af520 0x7f14300ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14301688b0 con 0x7f14300af520 2024-04-17T10:26:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1430168730 con 0x7f14300ad890 2024-04-17T10:26:00.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1430168a30 con 0x7f143006c1e0 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14300af520 0x7f14300ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14300af520 0x7f14300ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35592/0 (socket says 172.21.15.67:35592) 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 -- 172.21.15.67:0/2377828 learned_addr learned my addr 172.21.15.67:0/2377828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142d400640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14300ad890 0x7f14300cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142de00640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f143006c1e0 0x7f1430159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 -- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f143006c1e0 msgr2=0x7f1430159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f143006c1e0 0x7f1430159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 -- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14300ad890 msgr2=0x7f14300cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14300ad890 0x7f14300cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 -- 172.21.15.67:0/2377828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143009d3a0 con 0x7f14300af520 2024-04-17T10:26:00.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142e800640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14300af520 0x7f14300ce060 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f142800c780 tx=0x7f142800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f1426000640 1 -- 172.21.15.67:0/2377828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428003db0 con 0x7f14300af520 2024-04-17T10:26:00.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f1426000640 1 -- 172.21.15.67:0/2377828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1428005070 con 0x7f14300af520 2024-04-17T10:26:00.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f1426000640 1 -- 172.21.15.67:0/2377828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428005460 con 0x7f14300af520 2024-04-17T10:26:00.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1430157fa0 con 0x7f14300af520 2024-04-17T10:26:00.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14301562e0 con 0x7f14300af520 2024-04-17T10:26:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143009d880 con 0x7f14300af520 2024-04-17T10:26:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.883+0000 7f1426000640 1 -- 172.21.15.67:0/2377828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1428005600 con 0x7f14300af520 2024-04-17T10:26:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.887+0000 7f1426000640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1410066f60 0x7f1410069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:00.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.887+0000 7f1426000640 1 -- 172.21.15.67:0/2377828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f14280852c0 con 0x7f14300af520 2024-04-17T10:26:00.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.887+0000 7f142de00640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1410066f60 0x7f1410069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:00.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.887+0000 7f142de00640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1410066f60 0x7f1410069420 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14200af000 tx=0x7f14200a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:00.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:00.891+0000 7f1426000640 1 -- 172.21.15.67:0/2377828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f14280557e0 con 0x7f14300af520 2024-04-17T10:26:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f143004f780 con 0x7f14300af520 2024-04-17T10:26:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f1426000640 1 -- 172.21.15.67:0/2377828 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1428059880 con 0x7f14300af520 2024-04-17T10:26:01.050 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:26:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1410066f60 msgr2=0x7f1410069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1410066f60 0x7f1410069420 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14200af000 tx=0x7f14200a7000 comp rx=0 tx=0).stop 2024-04-17T10:26:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14300af520 msgr2=0x7f14300ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14300af520 0x7f14300ce060 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f142800c780 tx=0x7f142800cc40 comp rx=0 tx=0).stop 2024-04-17T10:26:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 shutdown_connections 2024-04-17T10:26:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1410066f60 0x7f1410069420 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14300af520 0x7f14300ce060 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14300ad890 0x7f14300cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 --2- 172.21.15.67:0/2377828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f143006c1e0 0x7f1430159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.055+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 >> 172.21.15.67:0/2377828 conn(0x7f1430071af0 msgr2=0x7f143015a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.059+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 shutdown_connections 2024-04-17T10:26:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.059+0000 7f142f200640 1 -- 172.21.15.67:0/2377828 wait complete. 2024-04-17T10:26:01.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:26:01.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac3ae00640 1 Processor -- start 2024-04-17T10:26:01.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac3ae00640 1 -- start start 2024-04-17T10:26:01.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac3ae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac3ae00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac340707c0 0x7fac3415d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac3ae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac340d03c0 con 0x7fac3406f8b0 2024-04-17T10:26:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac3ae00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340d06a0 con 0x7fac340707c0 2024-04-17T10:26:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac3ae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac340d0980 con 0x7fac3406d810 2024-04-17T10:26:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac2be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac3406f8b0 0x7fac340700b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58884/0 (socket says 172.21.15.67:58884) 2024-04-17T10:26:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac2be00640 1 -- 172.21.15.67:0/3806535065 learned_addr learned my addr 172.21.15.67:0/3806535065 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac33e00640 1 --2- 172.21.15.67:0/3806535065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.219+0000 7fac38e00640 1 --2- 172.21.15.67:0/3806535065 >> v2:172.21.15.86:3300/0 conn(0x7fac340707c0 0x7fac3415d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33400640 1 -- 172.21.15.67:0/3806535065 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156560733 0 0) 0x7fac340d03c0 con 0x7fac3406f8b0 2024-04-17T10:26:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33400640 1 -- 172.21.15.67:0/3806535065 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac18003530 con 0x7fac3406f8b0 2024-04-17T10:26:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33e00640 1 -- 172.21.15.67:0/3806535065 >> v2:172.21.15.86:3300/0 conn(0x7fac340707c0 msgr2=0x7fac3415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33e00640 1 --2- 172.21.15.67:0/3806535065 >> v2:172.21.15.86:3300/0 conn(0x7fac340707c0 0x7fac3415d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33e00640 1 -- 172.21.15.67:0/3806535065 >> v1:172.21.15.67:6789/0 conn(0x7fac3406f8b0 legacy=0x7fac340700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33e00640 1 -- 172.21.15.67:0/3806535065 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac340d0c60 con 0x7fac3406d810 2024-04-17T10:26:01.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33e00640 1 --2- 172.21.15.67:0/3806535065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340cfad0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac240097c0 tx=0x7fac2401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6165536901353d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33400640 1 -- 172.21.15.67:0/3806535065 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac240b2070 con 0x7fac3406d810 2024-04-17T10:26:01.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33400640 1 -- 172.21.15.67:0/3806535065 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac240ad440 con 0x7fac3406d810 2024-04-17T10:26:01.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.223+0000 7fac33400640 1 -- 172.21.15.67:0/3806535065 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac240ad810 con 0x7fac3406d810 2024-04-17T10:26:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- 172.21.15.67:0/3806535065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 msgr2=0x7fac340cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 --2- 172.21.15.67:0/3806535065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340cfad0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fac240097c0 tx=0x7fac2401c470 comp rx=0 tx=0).stop 2024-04-17T10:26:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- 172.21.15.67:0/3806535065 shutdown_connections 2024-04-17T10:26:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 --2- 172.21.15.67:0/3806535065 >> v2:172.21.15.86:3300/0 conn(0x7fac340707c0 0x7fac3415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 --2- 172.21.15.67:0/3806535065 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340cfad0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- 172.21.15.67:0/3806535065 >> 172.21.15.67:0/3806535065 conn(0x7fac340d1800 msgr2=0x7fac340d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- 172.21.15.67:0/3806535065 shutdown_connections 2024-04-17T10:26:01.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- 172.21.15.67:0/3806535065 wait complete. 2024-04-17T10:26:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 Processor -- start 2024-04-17T10:26:01.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- start start 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac3406f8b0 0x7fac3414d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac340707c0 0x7fac3414dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3406bda0 con 0x7fac340707c0 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3406de00 con 0x7fac3406f8b0 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac3ae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3406bf20 con 0x7fac3406d810 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac340707c0 0x7fac3414dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac340707c0 0x7fac3414dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35598/0 (socket says 172.21.15.67:35598) 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 -- 172.21.15.67:0/1787055123 learned_addr learned my addr 172.21.15.67:0/1787055123 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac33e00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac2be00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac3406f8b0 0x7fac3414d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 -- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 msgr2=0x7fac340b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 -- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac3406f8b0 msgr2=0x7fac3414d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac3406f8b0 0x7fac3414d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 -- 172.21.15.67:0/1787055123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3414e140 con 0x7fac340707c0 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac2be00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac3406f8b0 0x7fac3414d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac38e00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac340707c0 0x7fac3414dc00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00ef90 tx=0x7fac2c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.227+0000 7fac30c00640 1 -- 172.21.15.67:0/1787055123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c018020 con 0x7fac340707c0 2024-04-17T10:26:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac30c00640 1 -- 172.21.15.67:0/1787055123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c014480 con 0x7fac340707c0 2024-04-17T10:26:01.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac340b2b00 con 0x7fac340707c0 2024-04-17T10:26:01.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac30c00640 1 -- 172.21.15.67:0/1787055123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c009e60 con 0x7fac340707c0 2024-04-17T10:26:01.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac340b1800 con 0x7fac340707c0 2024-04-17T10:26:01.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3404ef30 con 0x7fac340707c0 2024-04-17T10:26:01.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac30c00640 1 -- 172.21.15.67:0/1787055123 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fac2c01b020 con 0x7fac340707c0 2024-04-17T10:26:01.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac30c00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac30c00640 1 -- 172.21.15.67:0/1787055123 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7fac2c0861b0 con 0x7fac340707c0 2024-04-17T10:26:01.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.231+0000 7fac33e00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.235+0000 7fac33e00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fac240af000 tx=0x7fac24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.235+0000 7fac30c00640 1 -- 172.21.15.67:0/1787055123 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fac2c022030 con 0x7fac340707c0 2024-04-17T10:26:01.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.395+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fac340b7780 con 0x7fac340707c0 2024-04-17T10:26:01.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.395+0000 7fac30c00640 1 -- 172.21.15.67:0/1787055123 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fac3414e140 con 0x7fac340707c0 2024-04-17T10:26:01.391 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 msgr2=0x7fac10069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fac240af000 tx=0x7fac24008040 comp rx=0 tx=0).stop 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac340707c0 msgr2=0x7fac3414dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac340707c0 0x7fac3414dc00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00ef90 tx=0x7fac2c00c490 comp rx=0 tx=0).stop 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 shutdown_connections 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac340707c0 0x7fac3414dc00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac3406f8b0 0x7fac3414d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 --2- 172.21.15.67:0/1787055123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac3406d810 0x7fac340b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 >> 172.21.15.67:0/1787055123 conn(0x7fac340d1800 msgr2=0x7fac340af850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 shutdown_connections 2024-04-17T10:26:01.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.399+0000 7fac3ae00640 1 -- 172.21.15.67:0/1787055123 wait complete. 2024-04-17T10:26:01.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-17T10:26:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.559+0000 7f9283c00640 1 Processor -- start 2024-04-17T10:26:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9283c00640 1 -- start start 2024-04-17T10:26:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9283c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c06bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9283c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c15e140 con 0x7f927c161f90 2024-04-17T10:26:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9283c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f927c099d50 con 0x7f927c15dd60 2024-04-17T10:26:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9283c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c1626d0 con 0x7f927c0d06a0 2024-04-17T10:26:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f927be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f927c161f90 0x7f927c0cff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58896/0 (socket says 172.21.15.67:58896) 2024-04-17T10:26:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f927be00640 1 -- 172.21.15.67:0/14949900 learned_addr learned my addr 172.21.15.67:0/14949900 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9281c00640 1 --2- 172.21.15.67:0/14949900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c06bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101750337 0 0) 0x7f927c15e140 con 0x7f927c161f90 2024-04-17T10:26:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9264003530 con 0x7f927c161f90 2024-04-17T10:26:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910072288 0 0) 0x7f927c099d50 con 0x7f927c15dd60 2024-04-17T10:26:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f927c15e140 con 0x7f927c15dd60 2024-04-17T10:26:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9281c00640 1 -- 172.21.15.67:0/14949900 >> v1:172.21.15.86:6789/0 conn(0x7f927c15dd60 legacy=0x7f927c161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9281c00640 1 -- 172.21.15.67:0/14949900 >> v1:172.21.15.67:6789/0 conn(0x7f927c161f90 legacy=0x7f927c0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 928885238 0 0) 0x7f9264003530 con 0x7f927c161f90 2024-04-17T10:26:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.563+0000 7f9281c00640 1 -- 172.21.15.67:0/14949900 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c09a030 con 0x7f927c0d06a0 2024-04-17T10:26:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9281c00640 1 --2- 172.21.15.67:0/14949900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c06bda0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9274009dc0 tx=0x7f927401c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a767e27f0cf12e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740b2070 con 0x7f927c0d06a0 2024-04-17T10:26:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92740a56c0 con 0x7f927c0d06a0 2024-04-17T10:26:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f927b400640 1 -- 172.21.15.67:0/14949900 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268001680 con 0x7f927c0d06a0 2024-04-17T10:26:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9283c00640 1 -- 172.21.15.67:0/14949900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 msgr2=0x7f927c06bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9283c00640 1 --2- 172.21.15.67:0/14949900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c06bda0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9274009dc0 tx=0x7f927401c880 comp rx=0 tx=0).stop 2024-04-17T10:26:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9283c00640 1 -- 172.21.15.67:0/14949900 shutdown_connections 2024-04-17T10:26:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9283c00640 1 --2- 172.21.15.67:0/14949900 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c06bda0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9283c00640 1 -- 172.21.15.67:0/14949900 >> 172.21.15.67:0/14949900 conn(0x7f927c0b34a0 msgr2=0x7f927c0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9283c00640 1 -- 172.21.15.67:0/14949900 shutdown_connections 2024-04-17T10:26:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.567+0000 7f9283c00640 1 -- 172.21.15.67:0/14949900 wait complete. 2024-04-17T10:26:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 Processor -- start 2024-04-17T10:26:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 -- start start 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c0af740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c15dd60 0x7f927c14c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c161f90 0x7f927c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c16bd90 con 0x7f927c161f90 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c16bc10 con 0x7f927c15dd60 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c16bf10 con 0x7f927c0d06a0 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c161f90 0x7f927c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c161f90 0x7f927c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35608/0 (socket says 172.21.15.67:35608) 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 -- 172.21.15.67:0/1063160925 learned_addr learned my addr 172.21.15.67:0/1063160925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f927be00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c15dd60 0x7f927c14c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281200640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c0af740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 -- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 msgr2=0x7f927c0af740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c0af740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 -- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c15dd60 msgr2=0x7f927c14c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c15dd60 0x7f927c14c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 -- 172.21.15.67:0/1063160925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c155d40 con 0x7f927c161f90 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f927be00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c15dd60 0x7f927c14c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9281c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c161f90 0x7f927c0aeba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f927401cd60 tx=0x7f9274019620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9278c00640 1 -- 172.21.15.67:0/1063160925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740b2050 con 0x7f927c161f90 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9278c00640 1 -- 172.21.15.67:0/1063160925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92740be070 con 0x7f927c161f90 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9278c00640 1 -- 172.21.15.67:0/1063160925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92740ada50 con 0x7f927c161f90 2024-04-17T10:26:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c1516c0 con 0x7f927c161f90 2024-04-17T10:26:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c151c80 con 0x7f927c161f90 2024-04-17T10:26:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.571+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927c04ef30 con 0x7f927c161f90 2024-04-17T10:26:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.575+0000 7f9278c00640 1 -- 172.21.15.67:0/1063160925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f92740adbf0 con 0x7f927c161f90 2024-04-17T10:26:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.575+0000 7f9278c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f925c066f60 0x7f925c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.575+0000 7f9278c00640 1 -- 172.21.15.67:0/1063160925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f92741204e0 con 0x7f927c161f90 2024-04-17T10:26:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.575+0000 7f9281200640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f925c066f60 0x7f925c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.575+0000 7f9281200640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f925c066f60 0x7f925c069420 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f92700011f0 tx=0x7f9270009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.579+0000 7f9278c00640 1 -- 172.21.15.67:0/1063160925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f92740164a0 con 0x7f927c161f90 2024-04-17T10:26:01.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.739+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f927c0b42d0 con 0x7f927c161f90 2024-04-17T10:26:01.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.739+0000 7f9278c00640 1 -- 172.21.15.67:0/1063160925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f92740ab340 con 0x7f927c161f90 2024-04-17T10:26:01.734 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f925c066f60 msgr2=0x7f925c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f925c066f60 0x7f925c069420 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f92700011f0 tx=0x7f9270009040 comp rx=0 tx=0).stop 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c161f90 msgr2=0x7f927c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c161f90 0x7f927c0aeba0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f927401cd60 tx=0x7f9274019620 comp rx=0 tx=0).stop 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 shutdown_connections 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f925c066f60 0x7f925c069420 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f927c161f90 0x7f927c0aeba0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f927c15dd60 0x7f927c14c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 --2- 172.21.15.67:0/1063160925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f927c0d06a0 0x7f927c0af740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 >> 172.21.15.67:0/1063160925 conn(0x7f927c0b34a0 msgr2=0x7f927c0cfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:01.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 shutdown_connections 2024-04-17T10:26:01.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.743+0000 7f9283c00640 1 -- 172.21.15.67:0/1063160925 wait complete. 2024-04-17T10:26:01.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:26:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6254000640 1 Processor -- start 2024-04-17T10:26:01.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6254000640 1 -- start start 2024-04-17T10:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6254000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c071310 0x7f624c071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6254000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624c06e2f0 con 0x7f624c145040 2024-04-17T10:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6254000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f624c06d6e0 con 0x7f624c0cc300 2024-04-17T10:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6254000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624c0a86c0 con 0x7f624c071310 2024-04-17T10:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6251600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f624c145040 0x7f624c06e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58900/0 (socket says 172.21.15.67:58900) 2024-04-17T10:26:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6251600640 1 -- 172.21.15.67:0/1788340083 learned_addr learned my addr 172.21.15.67:0/1788340083 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6250c00640 1 --2- 172.21.15.67:0/1788340083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c071310 0x7f624c071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f624be00640 1 -- 172.21.15.67:0/1788340083 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56303689 0 0) 0x7f624c06e2f0 con 0x7f624c145040 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f624be00640 1 -- 172.21.15.67:0/1788340083 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6234003530 con 0x7f624c145040 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f624be00640 1 -- 172.21.15.67:0/1788340083 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136542000 0 0) 0x7f624c06d6e0 con 0x7f624c0cc300 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f624be00640 1 -- 172.21.15.67:0/1788340083 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f624c06e2f0 con 0x7f624c0cc300 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6250c00640 1 -- 172.21.15.67:0/1788340083 >> v1:172.21.15.86:6789/0 conn(0x7f624c0cc300 legacy=0x7f624c0cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6250c00640 1 -- 172.21.15.67:0/1788340083 >> v1:172.21.15.67:6789/0 conn(0x7f624c145040 legacy=0x7f624c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.903+0000 7f6250c00640 1 -- 172.21.15.67:0/1788340083 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c06d9c0 con 0x7f624c071310 2024-04-17T10:26:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6250c00640 1 --2- 172.21.15.67:0/1788340083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c071310 0x7f624c071770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f62380097f0 tx=0x7f623801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac475cd8d1269fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f624be00640 1 -- 172.21.15.67:0/1788340083 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62380b5070 con 0x7f624c071310 2024-04-17T10:26:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f624be00640 1 -- 172.21.15.67:0/1788340083 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62380a89d0 con 0x7f624c071310 2024-04-17T10:26:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f624be00640 1 -- 172.21.15.67:0/1788340083 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62380a8da0 con 0x7f624c071310 2024-04-17T10:26:01.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6254000640 1 -- 172.21.15.67:0/1788340083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c071310 msgr2=0x7f624c071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6254000640 1 --2- 172.21.15.67:0/1788340083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c071310 0x7f624c071770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f62380097f0 tx=0x7f623801ecd0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6254000640 1 -- 172.21.15.67:0/1788340083 shutdown_connections 2024-04-17T10:26:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6254000640 1 --2- 172.21.15.67:0/1788340083 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c071310 0x7f624c071770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6254000640 1 -- 172.21.15.67:0/1788340083 >> 172.21.15.67:0/1788340083 conn(0x7f624c06ac90 msgr2=0x7f624c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6254000640 1 -- 172.21.15.67:0/1788340083 shutdown_connections 2024-04-17T10:26:01.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.907+0000 7f6254000640 1 -- 172.21.15.67:0/1788340083 wait complete. 2024-04-17T10:26:01.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 Processor -- start 2024-04-17T10:26:01.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 -- start start 2024-04-17T10:26:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f624c071310 0x7f624c143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c0cc300 0x7f624c0b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f624c145040 0x7f624c15ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c15bf10 con 0x7f624c145040 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624c06df80 con 0x7f624c071310 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624c06d370 con 0x7f624c0cc300 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f624c145040 0x7f624c15ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f624c145040 0x7f624c15ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35620/0 (socket says 172.21.15.67:35620) 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 -- 172.21.15.67:0/66642765 learned_addr learned my addr 172.21.15.67:0/66642765 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6251600640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f624c071310 0x7f624c143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6250c00640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c0cc300 0x7f624c0b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 -- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c0cc300 msgr2=0x7f624c0b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c0cc300 0x7f624c0b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 -- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f624c071310 msgr2=0x7f624c143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:01.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f624c071310 0x7f624c143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 -- 172.21.15.67:0/66642765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f624c0ae4f0 con 0x7f624c145040 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6250c00640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c0cc300 0x7f624c0b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6251600640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f624c071310 0x7f624c143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6252000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f624c145040 0x7f624c15ae10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f624400c7c0 tx=0x7f624400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6249600640 1 -- 172.21.15.67:0/66642765 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6244003b80 con 0x7f624c145040 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6249600640 1 -- 172.21.15.67:0/66642765 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6244003d20 con 0x7f624c145040 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6249600640 1 -- 172.21.15.67:0/66642765 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624400aa50 con 0x7f624c145040 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f624c1432d0 con 0x7f624c145040 2024-04-17T10:26:01.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f624c0b1bf0 con 0x7f624c145040 2024-04-17T10:26:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.911+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f624c072d50 con 0x7f624c145040 2024-04-17T10:26:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.915+0000 7f6249600640 1 -- 172.21.15.67:0/66642765 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f624400abf0 con 0x7f624c145040 2024-04-17T10:26:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.915+0000 7f6249600640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f622c066f60 0x7f622c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:01.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.915+0000 7f6249600640 1 -- 172.21.15.67:0/66642765 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f6244085480 con 0x7f624c145040 2024-04-17T10:26:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.915+0000 7f6251600640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f622c066f60 0x7f622c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:01.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.915+0000 7f6251600640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f622c066f60 0x7f622c069420 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62400011f0 tx=0x7f6240009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:01.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:01.919+0000 7f6249600640 1 -- 172.21.15.67:0/66642765 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6244056080 con 0x7f624c145040 2024-04-17T10:26:02.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.111+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f624c074e30 con 0x7f624c145040 2024-04-17T10:26:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.111+0000 7f6249600640 1 -- 172.21.15.67:0/66642765 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f6244055980 con 0x7f624c145040 2024-04-17T10:26:02.108 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:02.108 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:26:02.108 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f622c066f60 msgr2=0x7f622c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f622c066f60 0x7f622c069420 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f62400011f0 tx=0x7f6240009040 comp rx=0 tx=0).stop 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f624c145040 msgr2=0x7f624c15ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f624c145040 0x7f624c15ae10 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f624400c7c0 tx=0x7f624400cc80 comp rx=0 tx=0).stop 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 shutdown_connections 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f622c066f60 0x7f622c069420 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f624c145040 0x7f624c15ae10 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f624c0cc300 0x7f624c0b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 --2- 172.21.15.67:0/66642765 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f624c071310 0x7f624c143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 >> 172.21.15.67:0/66642765 conn(0x7f624c06ac90 msgr2=0x7f624c0af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 shutdown_connections 2024-04-17T10:26:02.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.115+0000 7f6254000640 1 -- 172.21.15.67:0/66642765 wait complete. 2024-04-17T10:26:02.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:26:02.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.275+0000 7f777dc00640 1 Processor -- start 2024-04-17T10:26:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.275+0000 7f777dc00640 1 -- start start 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f777dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77780ce1b0 0x7f77780ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f777dc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f777815cc00 0x7f777815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f777dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f777815fa00 0x7f7778070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f777dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77780cead0 con 0x7f777815fa00 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f777dc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f777815be60 con 0x7f777815cc00 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f777dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7778160230 con 0x7f77780ce1b0 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f777815fa00 0x7f7778070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f777815fa00 0x7f7778070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35622/0 (socket says 172.21.15.67:35622) 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 -- 172.21.15.67:0/4205740664 learned_addr learned my addr 172.21.15.67:0/4205740664 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7776a00640 1 --2- 172.21.15.67:0/4205740664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77780ce1b0 0x7f77780ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f776fe00640 1 --2- 172.21.15.67:0/4205740664 >> v2:172.21.15.86:3300/0 conn(0x7f777815cc00 0x7f777815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 -- 172.21.15.67:0/4205740664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77780ce1b0 msgr2=0x7f77780ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 --2- 172.21.15.67:0/4205740664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77780ce1b0 0x7f77780ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 -- 172.21.15.67:0/4205740664 >> v2:172.21.15.86:3300/0 conn(0x7f777815cc00 msgr2=0x7f777815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 --2- 172.21.15.67:0/4205740664 >> v2:172.21.15.86:3300/0 conn(0x7f777815cc00 0x7f777815f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.279+0000 7f7777400640 1 -- 172.21.15.67:0/4205740664 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777815c120 con 0x7f777815fa00 2024-04-17T10:26:02.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f7777400640 1 --2- 172.21.15.67:0/4205740664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777815fa00 0x7f7778070cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f777806e1b0 tx=0x7f777001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff909422dbc245c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:02.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f7776000640 1 -- 172.21.15.67:0/4205740664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77700b4070 con 0x7f777815fa00 2024-04-17T10:26:02.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f7776000640 1 -- 172.21.15.67:0/4205740664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77700af400 con 0x7f777815fa00 2024-04-17T10:26:02.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f7776000640 1 -- 172.21.15.67:0/4205740664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77700af7d0 con 0x7f777815fa00 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 -- 172.21.15.67:0/4205740664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777815fa00 msgr2=0x7f7778070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 --2- 172.21.15.67:0/4205740664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777815fa00 0x7f7778070cc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f777806e1b0 tx=0x7f777001f1e0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 -- 172.21.15.67:0/4205740664 shutdown_connections 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 --2- 172.21.15.67:0/4205740664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f777815fa00 0x7f7778070cc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 --2- 172.21.15.67:0/4205740664 >> v2:172.21.15.86:3300/0 conn(0x7f777815cc00 0x7f777815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 --2- 172.21.15.67:0/4205740664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77780ce1b0 0x7f77780ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 -- 172.21.15.67:0/4205740664 >> 172.21.15.67:0/4205740664 conn(0x7f77780b33f0 msgr2=0x7f77780b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 -- 172.21.15.67:0/4205740664 shutdown_connections 2024-04-17T10:26:02.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 -- 172.21.15.67:0/4205740664 wait complete. 2024-04-17T10:26:02.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 Processor -- start 2024-04-17T10:26:02.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.283+0000 7f777dc00640 1 -- start start 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77780ce1b0 0x7f7778158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f777815cc00 0x7f77780b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f777815fa00 0x7f7778157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777815fe70 con 0x7f77780ce1b0 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777815c6e0 con 0x7f777815fa00 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f777815fff0 con 0x7f777815cc00 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77780ce1b0 0x7f7778158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77780ce1b0 0x7f7778158870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35630/0 (socket says 172.21.15.67:35630) 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 -- 172.21.15.67:0/2627783438 learned_addr learned my addr 172.21.15.67:0/2627783438 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:02.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f776fe00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f777815cc00 0x7f77780b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7777400640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f777815fa00 0x7f7778157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 -- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f777815cc00 msgr2=0x7f77780b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f777815cc00 0x7f77780b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 -- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f777815fa00 msgr2=0x7f7778157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f777815fa00 0x7f7778157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 -- 172.21.15.67:0/2627783438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7778158e50 con 0x7f77780ce1b0 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f7776a00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77780ce1b0 0x7f7778158870 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f776800d800 tx=0x7f776800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f776ea00640 1 -- 172.21.15.67:0/2627783438 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768018020 con 0x7f77780ce1b0 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f776ea00640 1 -- 172.21.15.67:0/2627783438 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77680096a0 con 0x7f77780ce1b0 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f776ea00640 1 -- 172.21.15.67:0/2627783438 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768009a50 con 0x7f77780ce1b0 2024-04-17T10:26:02.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7778156dd0 con 0x7f77780ce1b0 2024-04-17T10:26:02.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7778157390 con 0x7f77780ce1b0 2024-04-17T10:26:02.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7778070ab0 con 0x7f77780ce1b0 2024-04-17T10:26:02.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.287+0000 7f776ea00640 1 -- 172.21.15.67:0/2627783438 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f776801b020 con 0x7f77780ce1b0 2024-04-17T10:26:02.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.291+0000 7f776ea00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7754066f60 0x7f7754069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.291+0000 7f776ea00640 1 -- 172.21.15.67:0/2627783438 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f77680858a0 con 0x7f77780ce1b0 2024-04-17T10:26:02.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.291+0000 7f776fe00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7754066f60 0x7f7754069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.291+0000 7f776fe00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7754066f60 0x7f7754069420 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7778144ca0 tx=0x7f7760008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:02.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.295+0000 7f776ea00640 1 -- 172.21.15.67:0/2627783438 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7768022020 con 0x7f77780ce1b0 2024-04-17T10:26:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.459+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7778074e30 con 0x7f77780ce1b0 2024-04-17T10:26:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.459+0000 7f776ea00640 1 -- 172.21.15.67:0/2627783438 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f7768055e50 con 0x7f77780ce1b0 2024-04-17T10:26:02.454 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:02.454 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:26:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.459+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7754066f60 msgr2=0x7f7754069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.459+0000 7f777dc00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7754066f60 0x7f7754069420 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7778144ca0 tx=0x7f7760008040 comp rx=0 tx=0).stop 2024-04-17T10:26:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.459+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77780ce1b0 msgr2=0x7f7778158870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.459+0000 7f777dc00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77780ce1b0 0x7f7778158870 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f776800d800 tx=0x7f776800dcc0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 shutdown_connections 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7754066f60 0x7f7754069420 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f777815fa00 0x7f7778157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f777815cc00 0x7f77780b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 --2- 172.21.15.67:0/2627783438 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77780ce1b0 0x7f7778158870 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 >> 172.21.15.67:0/2627783438 conn(0x7f77780b33f0 msgr2=0x7f77780c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 shutdown_connections 2024-04-17T10:26:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.463+0000 7f777dc00640 1 -- 172.21.15.67:0/2627783438 wait complete. 2024-04-17T10:26:02.477 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:26:02.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:26:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.623+0000 7f1a35a00640 1 Processor -- start 2024-04-17T10:26:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.623+0000 7f1a35a00640 1 -- start start 2024-04-17T10:26:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.623+0000 7f1a35a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1a30071770 0x7f1a300cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.623+0000 7f1a35a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3006e2f0 con 0x7f1a30145040 2024-04-17T10:26:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.623+0000 7f1a35a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3006d6e0 con 0x7f1a30071310 2024-04-17T10:26:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.623+0000 7f1a35a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a300a86c0 con 0x7f1a30071770 2024-04-17T10:26:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1a30145040 0x7f1a3006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58910/0 (socket says 172.21.15.67:58910) 2024-04-17T10:26:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 -- 172.21.15.67:0/3445107689 learned_addr learned my addr 172.21.15.67:0/3445107689 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394418102 0 0) 0x7f1a3006e2f0 con 0x7f1a30145040 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2f400640 1 --2- 172.21.15.67:0/3445107689 >> v2:172.21.15.86:3300/0 conn(0x7f1a30071770 0x7f1a300cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a14003530 con 0x7f1a30145040 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066446011 0 0) 0x7f1a14003530 con 0x7f1a30145040 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 >> v2:172.21.15.86:3300/0 conn(0x7f1a30071770 msgr2=0x7f1a300cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 --2- 172.21.15.67:0/3445107689 >> v2:172.21.15.86:3300/0 conn(0x7f1a30071770 0x7f1a300cca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 >> v1:172.21.15.86:6789/0 conn(0x7f1a30071310 legacy=0x7f1a300cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3006d9c0 con 0x7f1a30145040 2024-04-17T10:26:02.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f1a20001680 con 0x7f1a30145040 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 >> v1:172.21.15.67:6789/0 conn(0x7f1a30145040 legacy=0x7f1a3006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140039c0 0x7f1a14003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a14004810 0x7f1a14004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a140051d0 0x7f1a1400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1400c050 con 0x7f1a140039c0 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1400c200 con 0x7f1a140051d0 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1400c400 con 0x7f1a14004810 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140039c0 0x7f1a14003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2f400640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a140051d0 0x7f1a1400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a14004810 0x7f1a14004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 -- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a14004810 msgr2=0x7f1a14004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a14004810 0x7f1a14004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 -- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a140051d0 msgr2=0x7f1a1400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a140051d0 0x7f1a1400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2ea00640 1 -- 172.21.15.67:0/3445107689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a14003530 con 0x7f1a140039c0 2024-04-17T10:26:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a2f400640 1 -- 172.21.15.67:0/3445107689 reap_dead start 2024-04-17T10:26:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.627+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a14004810 0x7f1a14004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:26:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a2ea00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140039c0 0x7f1a14003de0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000c1c0 tx=0x7f1a2001f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a3b30aaff04410c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20007d00 con 0x7f1a140039c0 2024-04-17T10:26:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20005b00 con 0x7f1a140039c0 2024-04-17T10:26:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a2e000640 1 -- 172.21.15.67:0/3445107689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20007070 con 0x7f1a140039c0 2024-04-17T10:26:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- 172.21.15.67:0/3445107689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140039c0 msgr2=0x7f1a14003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140039c0 0x7f1a14003de0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000c1c0 tx=0x7f1a2001f390 comp rx=0 tx=0).stop 2024-04-17T10:26:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- 172.21.15.67:0/3445107689 shutdown_connections 2024-04-17T10:26:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 --2- 172.21.15.67:0/3445107689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a140039c0 0x7f1a14003de0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- 172.21.15.67:0/3445107689 >> 172.21.15.67:0/3445107689 conn(0x7f1a3006ac90 msgr2=0x7f1a3006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- 172.21.15.67:0/3445107689 shutdown_connections 2024-04-17T10:26:02.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- 172.21.15.67:0/3445107689 wait complete. 2024-04-17T10:26:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 Processor -- start 2024-04-17T10:26:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- start start 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a140039c0 0x7f1a301f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a3015b280 0x7f1a3015e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3015eb80 0x7f1a301417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3006df80 con 0x7f1a3015eb80 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3015bf10 con 0x7f1a3015b280 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.631+0000 7f1a35a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a30071b80 con 0x7f1a140039c0 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3015eb80 0x7f1a301417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a140039c0 0x7f1a301f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3015eb80 0x7f1a301417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35648/0 (socket says 172.21.15.67:35648) 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 -- 172.21.15.67:0/116380891 learned_addr learned my addr 172.21.15.67:0/116380891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a27e00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a3015b280 0x7f1a3015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 -- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a140039c0 msgr2=0x7f1a301f3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a140039c0 0x7f1a301f3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 -- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a3015b280 msgr2=0x7f1a3015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a3015b280 0x7f1a3015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 -- 172.21.15.67:0/116380891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a300b2b30 con 0x7f1a3015eb80 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a2f400640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3015eb80 0x7f1a301417b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1a300cee70 tx=0x7f1a280115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a26a00640 1 -- 172.21.15.67:0/116380891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a28011db0 con 0x7f1a3015eb80 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a26a00640 1 -- 172.21.15.67:0/116380891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2801b070 con 0x7f1a3015eb80 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a300b2e10 con 0x7f1a3015eb80 2024-04-17T10:26:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a26a00640 1 -- 172.21.15.67:0/116380891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a280039a0 con 0x7f1a3015eb80 2024-04-17T10:26:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a300b33d0 con 0x7f1a3015eb80 2024-04-17T10:26:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a30072d50 con 0x7f1a3015eb80 2024-04-17T10:26:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.635+0000 7f1a26a00640 1 -- 172.21.15.67:0/116380891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1a28003b40 con 0x7f1a3015eb80 2024-04-17T10:26:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.639+0000 7f1a26a00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a0c066f60 0x7f1a0c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.639+0000 7f1a26a00640 1 -- 172.21.15.67:0/116380891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f1a28085bf0 con 0x7f1a3015eb80 2024-04-17T10:26:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.639+0000 7f1a2ea00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a0c066f60 0x7f1a0c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.639+0000 7f1a2ea00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a0c066f60 0x7f1a0c069420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a200ae000 tx=0x7f1a20007040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:02.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.643+0000 7f1a26a00640 1 -- 172.21.15.67:0/116380891 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1a28056110 con 0x7f1a3015eb80 2024-04-17T10:26:02.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.835+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a30074e30 con 0x7f1a3015eb80 2024-04-17T10:26:02.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.835+0000 7f1a26a00640 1 -- 172.21.15.67:0/116380891 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f1a2805a1b0 con 0x7f1a3015eb80 2024-04-17T10:26:02.832 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:02.832 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:26:02.832 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a0c066f60 msgr2=0x7f1a0c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a0c066f60 0x7f1a0c069420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a200ae000 tx=0x7f1a20007040 comp rx=0 tx=0).stop 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3015eb80 msgr2=0x7f1a301417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3015eb80 0x7f1a301417b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1a300cee70 tx=0x7f1a280115a0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 shutdown_connections 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a0c066f60 0x7f1a0c069420 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a3015eb80 0x7f1a301417b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a3015b280 0x7f1a3015e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 --2- 172.21.15.67:0/116380891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a140039c0 0x7f1a301f3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 >> 172.21.15.67:0/116380891 conn(0x7f1a3006ac90 msgr2=0x7f1a300af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:02.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 shutdown_connections 2024-04-17T10:26:02.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.839+0000 7f1a35a00640 1 -- 172.21.15.67:0/116380891 wait complete. 2024-04-17T10:26:02.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:26:02.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18cba00640 1 Processor -- start 2024-04-17T10:26:02.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18cba00640 1 -- start start 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18cba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c40b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18cba00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18c40b57b0 0x7f18c4070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18cba00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c40b0cc0 con 0x7f18c40b07b0 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18cba00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c40b1280 con 0x7f18c40b57b0 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18cba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c40b0fa0 con 0x7f18c415dd10 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18c3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18c40b57b0 0x7f18c4070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18c3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18c40b57b0 0x7f18c4070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35656/0 (socket says 172.21.15.67:35656) 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:02.999+0000 7f18c3e00640 1 -- 172.21.15.67:0/688577664 learned_addr learned my addr 172.21.15.67:0/688577664 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:02.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c9000640 1 --2- 172.21.15.67:0/688577664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c40b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:02.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3e00640 1 -- 172.21.15.67:0/688577664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 msgr2=0x7f18c40b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:02.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3e00640 1 --2- 172.21.15.67:0/688577664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c40b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:02.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3e00640 1 -- 172.21.15.67:0/688577664 >> v1:172.21.15.86:6789/0 conn(0x7f18c40b07b0 legacy=0x7f18c41618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:02.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3e00640 1 -- 172.21.15.67:0/688577664 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c406bed0 con 0x7f18c40b57b0 2024-04-17T10:26:02.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3e00640 1 --2- 172.21.15.67:0/688577664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b57b0 0x7f18c4070dc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009870 tx=0x7f18b001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b96210cac50473e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:02.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3400640 1 -- 172.21.15.67:0/688577664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b00ab680 con 0x7f18c40b57b0 2024-04-17T10:26:02.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3400640 1 -- 172.21.15.67:0/688577664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b00b0070 con 0x7f18c40b57b0 2024-04-17T10:26:02.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18c3400640 1 -- 172.21.15.67:0/688577664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b00aba40 con 0x7f18c40b57b0 2024-04-17T10:26:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18cba00640 1 -- 172.21.15.67:0/688577664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b57b0 msgr2=0x7f18c4070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18cba00640 1 --2- 172.21.15.67:0/688577664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b57b0 0x7f18c4070dc0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009870 tx=0x7f18b001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18cba00640 1 -- 172.21.15.67:0/688577664 shutdown_connections 2024-04-17T10:26:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.003+0000 7f18cba00640 1 --2- 172.21.15.67:0/688577664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b57b0 0x7f18c4070dc0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 --2- 172.21.15.67:0/688577664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c40b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- 172.21.15.67:0/688577664 >> 172.21.15.67:0/688577664 conn(0x7f18c40b34d0 msgr2=0x7f18c40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:03.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- 172.21.15.67:0/688577664 shutdown_connections 2024-04-17T10:26:03.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- 172.21.15.67:0/688577664 wait complete. 2024-04-17T10:26:03.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 Processor -- start 2024-04-17T10:26:03.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- start start 2024-04-17T10:26:03.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b07b0 0x7f18c41574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c4150100 0x7f18c40cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c416ae70 con 0x7f18c40b07b0 2024-04-17T10:26:03.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4071710 con 0x7f18c4150100 2024-04-17T10:26:03.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c416aff0 con 0x7f18c415dd10 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b07b0 0x7f18c41574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b07b0 0x7f18c41574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35664/0 (socket says 172.21.15.67:35664) 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 -- 172.21.15.67:0/2950183144 learned_addr learned my addr 172.21.15.67:0/2950183144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c3e00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9a00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c4150100 0x7f18c40cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 -- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 msgr2=0x7f18c414fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c414fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 -- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c4150100 msgr2=0x7f18c40cfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c4150100 0x7f18c40cfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 -- 172.21.15.67:0/2950183144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c40d0980 con 0x7f18c40b07b0 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c3e00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c414fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c9000640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b07b0 0x7f18c41574a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f18b800d5f0 tx=0x7f18b800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c0c00640 1 -- 172.21.15.67:0/2950183144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b80125f0 con 0x7f18c40b07b0 2024-04-17T10:26:03.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c0c00640 1 -- 172.21.15.67:0/2950183144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b8019070 con 0x7f18c40b07b0 2024-04-17T10:26:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18c0c00640 1 -- 172.21.15.67:0/2950183144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b80129b0 con 0x7f18c40b07b0 2024-04-17T10:26:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.007+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c40ce760 con 0x7f18c40b07b0 2024-04-17T10:26:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.011+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c40ced20 con 0x7f18c40b07b0 2024-04-17T10:26:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.011+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c4070bb0 con 0x7f18c40b07b0 2024-04-17T10:26:03.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.011+0000 7f18c0c00640 1 -- 172.21.15.67:0/2950183144 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f18b8012b50 con 0x7f18c40b07b0 2024-04-17T10:26:03.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.011+0000 7f18c0c00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18a4066f60 0x7f18a4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.011+0000 7f18c0c00640 1 -- 172.21.15.67:0/2950183144 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f18b8085800 con 0x7f18c40b07b0 2024-04-17T10:26:03.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.011+0000 7f18c3e00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18a4066f60 0x7f18a4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.011+0000 7f18c3e00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18a4066f60 0x7f18a4069420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f18b00af000 tx=0x7f18b00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.015+0000 7f18c0c00640 1 -- 172.21.15.67:0/2950183144 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f18b800b320 con 0x7f18c40b07b0 2024-04-17T10:26:03.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.211+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18c4074e30 con 0x7f18c40b07b0 2024-04-17T10:26:03.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.211+0000 7f18c0c00640 1 -- 172.21.15.67:0/2950183144 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4934 (secure 0 0 0) 0x7f18b8056380 con 0x7f18c40b07b0 2024-04-17T10:26:03.208 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:03.208 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2319029878","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2319029878},{"type":"v1","addr":"172.21.15.86:6836","nonce":2319029878}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/3401981535","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3401981535},{"type":"v1","addr":"172.21.15.67:6833","nonce":3401981535}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3988158653","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3988158653},{"type":"v1","addr":"172.21.15.86:6837","nonce":3988158653}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:24:54.978691+0000","modified":"2024-04-17T10:25:19.079374+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4275},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4275":{"gid":4275,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1394098337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1394098337},{"type":"v1","addr":"172.21.15.67:6835","nonce":1394098337}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-04-17T10:26:03.209 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 9 2024-04-17T10:26:03.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18a4066f60 msgr2=0x7f18a4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18a4066f60 0x7f18a4069420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f18b00af000 tx=0x7f18b00a7000 comp rx=0 tx=0).stop 2024-04-17T10:26:03.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b07b0 msgr2=0x7f18c41574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b07b0 0x7f18c41574a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f18b800d5f0 tx=0x7f18b800dab0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 shutdown_connections 2024-04-17T10:26:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18a4066f60 0x7f18a4069420 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18c4150100 0x7f18c40cfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18c415dd10 0x7f18c414fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 --2- 172.21.15.67:0/2950183144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18c40b07b0 0x7f18c41574a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 >> 172.21.15.67:0/2950183144 conn(0x7f18c40b34d0 msgr2=0x7f18c40d1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 shutdown_connections 2024-04-17T10:26:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.215+0000 7f18cba00640 1 -- 172.21.15.67:0/2950183144 wait complete. 2024-04-17T10:26:03.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:26:03.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.375+0000 7f99a8e00640 1 Processor -- start 2024-04-17T10:26:03.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.375+0000 7f99a8e00640 1 -- start start 2024-04-17T10:26:03.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.375+0000 7f99a8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99a40d03a0 0x7f99a415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f99a8e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a40cde20 con 0x7f99a409a740 2024-04-17T10:26:03.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f99a8e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a40cf300 con 0x7f99a4099a50 2024-04-17T10:26:03.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f99a8e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a40cf5e0 con 0x7f99a40d03a0 2024-04-17T10:26:03.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f999be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99a40d03a0 0x7f99a415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f999be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f99a40d03a0 0x7f99a415c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35672/0 (socket says 172.21.15.67:35672) 2024-04-17T10:26:03.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f999be00640 1 -- 172.21.15.67:0/716284473 learned_addr learned my addr 172.21.15.67:0/716284473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:03.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f999be00640 1 -- 172.21.15.67:0/716284473 >> v1:172.21.15.86:6789/0 conn(0x7f99a4099a50 legacy=0x7f99a4099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f999be00640 1 -- 172.21.15.67:0/716284473 >> v1:172.21.15.67:6789/0 conn(0x7f99a409a740 legacy=0x7f99a40cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f999be00640 1 -- 172.21.15.67:0/716284473 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a40cf8c0 con 0x7f99a40d03a0 2024-04-17T10:26:03.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.379+0000 7f999be00640 1 --2- 172.21.15.67:0/716284473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a415c7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f998c009870 tx=0x7f998c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72cee74b102bfab0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a2a00640 1 -- 172.21.15.67:0/716284473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b2070 con 0x7f99a40d03a0 2024-04-17T10:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a2a00640 1 -- 172.21.15.67:0/716284473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c0a9070 con 0x7f99a40d03a0 2024-04-17T10:26:03.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a2a00640 1 -- 172.21.15.67:0/716284473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0ad880 con 0x7f99a40d03a0 2024-04-17T10:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- 172.21.15.67:0/716284473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 msgr2=0x7f99a415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 --2- 172.21.15.67:0/716284473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a415c7a0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f998c009870 tx=0x7f998c01c560 comp rx=0 tx=0).stop 2024-04-17T10:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- 172.21.15.67:0/716284473 shutdown_connections 2024-04-17T10:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 --2- 172.21.15.67:0/716284473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a415c7a0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- 172.21.15.67:0/716284473 >> 172.21.15.67:0/716284473 conn(0x7f99a40d1690 msgr2=0x7f99a40d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- 172.21.15.67:0/716284473 shutdown_connections 2024-04-17T10:26:03.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- 172.21.15.67:0/716284473 wait complete. 2024-04-17T10:26:03.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 Processor -- start 2024-04-17T10:26:03.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- start start 2024-04-17T10:26:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a4099a50 0x7f99a40c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99a409a740 0x7f99a406cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a4169a50 con 0x7f99a40d03a0 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a41698d0 con 0x7f99a4099a50 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a8e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a4169bd0 con 0x7f99a409a740 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a406d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35688/0 (socket says 172.21.15.67:35688) 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a3e00640 1 -- 172.21.15.67:0/4058600113 learned_addr learned my addr 172.21.15.67:0/4058600113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f999be00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99a409a740 0x7f99a406cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.383+0000 7f99a3400640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a4099a50 0x7f99a40c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a3e00640 1 -- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99a409a740 msgr2=0x7f99a406cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a3e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99a409a740 0x7f99a406cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a3e00640 1 -- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a4099a50 msgr2=0x7f99a40c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a3e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a4099a50 0x7f99a40c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a3e00640 1 -- 172.21.15.67:0/4058600113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4142580 con 0x7f99a40d03a0 2024-04-17T10:26:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a3e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a406d330 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c7a0 tx=0x7f999c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f999b400640 1 -- 172.21.15.67:0/4058600113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c003b80 con 0x7f99a40d03a0 2024-04-17T10:26:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f999b400640 1 -- 172.21.15.67:0/4058600113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c003d20 con 0x7f99a40d03a0 2024-04-17T10:26:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f999b400640 1 -- 172.21.15.67:0/4058600113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c00aa90 con 0x7f99a40d03a0 2024-04-17T10:26:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a40b6c80 con 0x7f99a40d03a0 2024-04-17T10:26:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a4142b00 con 0x7f99a40d03a0 2024-04-17T10:26:03.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a4074dc0 con 0x7f99a40d03a0 2024-04-17T10:26:03.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f999b400640 1 -- 172.21.15.67:0/4058600113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f999c00ac30 con 0x7f99a40d03a0 2024-04-17T10:26:03.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f999b400640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9980066f90 0x7f9980069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f999b400640 1 -- 172.21.15.67:0/4058600113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7f999c085320 con 0x7f99a40d03a0 2024-04-17T10:26:03.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.387+0000 7f99a3400640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9980066f90 0x7f9980069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.391+0000 7f99a3400640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9980066f90 0x7f9980069450 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9994007fd0 tx=0x7f9994009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.391+0000 7f999b400640 1 -- 172.21.15.67:0/4058600113 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f999c0557b0 con 0x7f99a40d03a0 2024-04-17T10:26:03.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.555+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99a4167490 con 0x7f99a40d03a0 2024-04-17T10:26:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.555+0000 7f999b400640 1 -- 172.21.15.67:0/4058600113 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18854 (secure 0 0 0) 0x7f999c059850 con 0x7f99a40d03a0 2024-04-17T10:26:03.552 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:03.552 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":27,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:25:35.774064+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:24:53.220996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-17T10:24:53.975318+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-17T10:24:57.660058+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.5","mappings":[{"from":5,"to":2}]},{"pgid":"3.11","mappings":[{"from":7,"to":1}]},{"pgid":"3.22","mappings":[{"from":3,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9980066f90 msgr2=0x7f9980069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9980066f90 0x7f9980069450 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9994007fd0 tx=0x7f9994009040 comp rx=0 tx=0).stop 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 msgr2=0x7f99a406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a406d330 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f999c00c7a0 tx=0x7f999c00cc60 comp rx=0 tx=0).stop 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 shutdown_connections 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9980066f90 0x7f9980069450 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99a40d03a0 0x7f99a406d330 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f99a409a740 0x7f99a406cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 --2- 172.21.15.67:0/4058600113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f99a4099a50 0x7f99a40c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 >> 172.21.15.67:0/4058600113 conn(0x7f99a40d1690 msgr2=0x7f99a40afb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:03.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 shutdown_connections 2024-04-17T10:26:03.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.559+0000 7f99a8e00640 1 -- 172.21.15.67:0/4058600113 wait complete. 2024-04-17T10:26:03.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:26:03.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.719+0000 7ff912e00640 1 Processor -- start 2024-04-17T10:26:03.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.719+0000 7ff912e00640 1 -- start start 2024-04-17T10:26:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff912e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff90c0ce020 0x7ff90c1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff912e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce400 0x7ff90c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff912e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff90c15eb50 con 0x7ff90c15e550 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff912e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c161650 con 0x7ff90c0ce020 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff912e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c1625e0 con 0x7ff90c0ce400 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff90be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff90c0ce020 0x7ff90c1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff90be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff90c0ce020 0x7ff90c1620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35690/0 (socket says 172.21.15.67:35690) 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff90be00640 1 -- 172.21.15.67:0/2401542713 learned_addr learned my addr 172.21.15.67:0/2401542713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff903400640 1 --2- 172.21.15.67:0/2401542713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce400 0x7ff90c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff90be00640 1 -- 172.21.15.67:0/2401542713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce400 msgr2=0x7ff90c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff90be00640 1 --2- 172.21.15.67:0/2401542713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce400 0x7ff90c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff90be00640 1 -- 172.21.15.67:0/2401542713 >> v1:172.21.15.86:6789/0 conn(0x7ff90c15e550 legacy=0x7ff90c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:03.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.723+0000 7ff90be00640 1 -- 172.21.15.67:0/2401542713 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c15ee30 con 0x7ff90c0ce020 2024-04-17T10:26:03.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff90be00640 1 --2- 172.21.15.67:0/2401542713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce020 0x7ff90c1620a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009870 tx=0x7ff8fc01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c68ab3f1de2ff473 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff90b400640 1 -- 172.21.15.67:0/2401542713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc0ab600 con 0x7ff90c0ce020 2024-04-17T10:26:03.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff90b400640 1 -- 172.21.15.67:0/2401542713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8fc0b0070 con 0x7ff90c0ce020 2024-04-17T10:26:03.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff90b400640 1 -- 172.21.15.67:0/2401542713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8fc0ab9c0 con 0x7ff90c0ce020 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 -- 172.21.15.67:0/2401542713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce020 msgr2=0x7ff90c1620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 --2- 172.21.15.67:0/2401542713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce020 0x7ff90c1620a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009870 tx=0x7ff8fc01c4f0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 -- 172.21.15.67:0/2401542713 shutdown_connections 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 --2- 172.21.15.67:0/2401542713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce400 0x7ff90c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 --2- 172.21.15.67:0/2401542713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce020 0x7ff90c1620a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 -- 172.21.15.67:0/2401542713 >> 172.21.15.67:0/2401542713 conn(0x7ff90c0b33f0 msgr2=0x7ff90c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 -- 172.21.15.67:0/2401542713 shutdown_connections 2024-04-17T10:26:03.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 -- 172.21.15.67:0/2401542713 wait complete. 2024-04-17T10:26:03.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 Processor -- start 2024-04-17T10:26:03.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.727+0000 7ff912e00640 1 -- start start 2024-04-17T10:26:03.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce020 0x7ff90c13d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce400 0x7ff90c1586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff90c15e550 0x7ff90c156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0ce960 con 0x7ff90c0ce400 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0ce7e0 con 0x7ff90c15e550 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0c84b0 con 0x7ff90c0ce020 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce400 0x7ff90c1586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce400 0x7ff90c1586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35702/0 (socket says 172.21.15.67:35702) 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff90be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce020 0x7ff90c13d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 -- 172.21.15.67:0/2431514041 learned_addr learned my addr 172.21.15.67:0/2431514041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff910e00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff90c15e550 0x7ff90c156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 -- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce020 msgr2=0x7ff90c13d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce020 0x7ff90c13d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 -- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff90c15e550 msgr2=0x7ff90c156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff90c15e550 0x7ff90c156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 -- 172.21.15.67:0/2431514041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c156a70 con 0x7ff90c0ce400 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff90be00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce020 0x7ff90c13d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff903400640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce400 0x7ff90c1586b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff8f400d800 tx=0x7ff8f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff908c00640 1 -- 172.21.15.67:0/2431514041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f400b6c0 con 0x7ff90c0ce400 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff908c00640 1 -- 172.21.15.67:0/2431514041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f4019070 con 0x7ff90c0ce400 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff908c00640 1 -- 172.21.15.67:0/2431514041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f400ba60 con 0x7ff90c0ce400 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c13f610 con 0x7ff90c0ce400 2024-04-17T10:26:03.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c13fbd0 con 0x7ff90c0ce400 2024-04-17T10:26:03.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c070ab0 con 0x7ff90c0ce400 2024-04-17T10:26:03.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.731+0000 7ff908c00640 1 -- 172.21.15.67:0/2431514041 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff8f400bc00 con 0x7ff90c0ce400 2024-04-17T10:26:03.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.735+0000 7ff908c00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff8e8066f60 0x7ff8e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:03.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.735+0000 7ff908c00640 1 -- 172.21.15.67:0/2431514041 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6735+0+0 (secure 0 0 0) 0x7ff8f4085840 con 0x7ff90c0ce400 2024-04-17T10:26:03.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.735+0000 7ff90be00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff8e8066f60 0x7ff8e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:03.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.735+0000 7ff90be00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff8e8066f60 0x7ff8e8069420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0af000 tx=0x7ff8fc0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:03.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.739+0000 7ff908c00640 1 -- 172.21.15.67:0/2431514041 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff8f4055d60 con 0x7ff90c0ce400 2024-04-17T10:26:03.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.931+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff90c074e30 con 0x7ff90c0ce400 2024-04-17T10:26:03.928 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:26:03.931+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:26:03.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.935+0000 7ff908c00640 1 -- 172.21.15.67:0/2431514041 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff8f4059e00 con 0x7ff90c0ce400 2024-04-17T10:26:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:26:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff8e8066f60 msgr2=0x7ff8e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff8e8066f60 0x7ff8e8069420 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc0af000 tx=0x7ff8fc0a7000 comp rx=0 tx=0).stop 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce400 msgr2=0x7ff90c1586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce400 0x7ff90c1586b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff8f400d800 tx=0x7ff8f400dcc0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 shutdown_connections 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff8e8066f60 0x7ff8e8069420 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff90c15e550 0x7ff90c156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff90c0ce400 0x7ff90c1586b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 --2- 172.21.15.67:0/2431514041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff90c0ce020 0x7ff90c13d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 >> 172.21.15.67:0/2431514041 conn(0x7ff90c0b33f0 msgr2=0x7ff90c15faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 shutdown_connections 2024-04-17T10:26:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:03.939+0000 7ff912e00640 1 -- 172.21.15.67:0/2431514041 wait complete. 2024-04-17T10:26:03.943 INFO:tasks.ceph.mds.c.smithi067.stderr: -5435> 2024-04-17T10:24:59.491+0000 7fb6eee00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:26:03.956 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:26:03.993 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:26:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.107+0000 7efc24200640 1 Processor -- start 2024-04-17T10:26:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.107+0000 7efc24200640 1 -- start start 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc24200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efc1c0cd030 0x7efc1c06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc24200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c0b4990 con 0x7efc1c0aef60 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc24200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc1c06ec00 con 0x7efc1c06e7b0 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc24200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1c0af340 con 0x7efc1c0cd030 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc21800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efc1c0aef60 0x7efc1c0cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58948/0 (socket says 172.21.15.67:58948) 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc21800640 1 -- 172.21.15.67:0/932862998 learned_addr learned my addr 172.21.15.67:0/932862998 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc20e00640 1 --2- 172.21.15.67:0/932862998 >> v2:172.21.15.67:3300/0 conn(0x7efc1c0cd030 0x7efc1c06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc1be00640 1 -- 172.21.15.67:0/932862998 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252560274 0 0) 0x7efc10001120 con 0x7efc1c0aef60 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc1be00640 1 -- 172.21.15.67:0/932862998 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc04005740 con 0x7efc1c0aef60 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc20e00640 1 -- 172.21.15.67:0/932862998 >> v1:172.21.15.86:6789/0 conn(0x7efc1c06e7b0 legacy=0x7efc1c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc20e00640 1 -- 172.21.15.67:0/932862998 >> v1:172.21.15.67:6789/0 conn(0x7efc1c0aef60 legacy=0x7efc1c0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc20e00640 1 -- 172.21.15.67:0/932862998 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c15ae90 con 0x7efc1c0cd030 2024-04-17T10:26:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.111+0000 7efc20e00640 1 --2- 172.21.15.67:0/932862998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c0cd030 0x7efc1c06e270 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7efc080097c0 tx=0x7efc0801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cbe9190f3be482 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc1be00640 1 -- 172.21.15.67:0/932862998 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc080b0070 con 0x7efc1c0cd030 2024-04-17T10:26:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc1be00640 1 -- 172.21.15.67:0/932862998 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc080ab600 con 0x7efc1c0cd030 2024-04-17T10:26:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc1be00640 1 -- 172.21.15.67:0/932862998 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc080ab9d0 con 0x7efc1c0cd030 2024-04-17T10:26:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- 172.21.15.67:0/932862998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c0cd030 msgr2=0x7efc1c06e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 --2- 172.21.15.67:0/932862998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c0cd030 0x7efc1c06e270 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7efc080097c0 tx=0x7efc0801c410 comp rx=0 tx=0).stop 2024-04-17T10:26:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- 172.21.15.67:0/932862998 shutdown_connections 2024-04-17T10:26:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 --2- 172.21.15.67:0/932862998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c0cd030 0x7efc1c06e270 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- 172.21.15.67:0/932862998 >> 172.21.15.67:0/932862998 conn(0x7efc1c0b1740 msgr2=0x7efc1c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- 172.21.15.67:0/932862998 shutdown_connections 2024-04-17T10:26:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- 172.21.15.67:0/932862998 wait complete. 2024-04-17T10:26:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 Processor -- start 2024-04-17T10:26:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- start start 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c06e7b0 0x7efc1c157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc1c0aef60 0x7efc1c154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc1c0cd030 0x7efc1c1546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c15bc10 con 0x7efc1c06e7b0 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c06d160 con 0x7efc1c0cd030 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc24200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c159c40 con 0x7efc1c0aef60 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc21800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c06e7b0 0x7efc1c157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc21800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c06e7b0 0x7efc1c157150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35748/0 (socket says 172.21.15.67:35748) 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.115+0000 7efc21800640 1 -- 172.21.15.67:0/3032266921 learned_addr learned my addr 172.21.15.67:0/3032266921 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc22200640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc1c0cd030 0x7efc1c1546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc20e00640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc1c0aef60 0x7efc1c154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc21800640 1 -- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc1c0aef60 msgr2=0x7efc1c154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc21800640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc1c0aef60 0x7efc1c154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc21800640 1 -- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc1c0cd030 msgr2=0x7efc1c1546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc21800640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc1c0cd030 0x7efc1c1546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc21800640 1 -- 172.21.15.67:0/3032266921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc1c159f20 con 0x7efc1c06e7b0 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc20e00640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc1c0aef60 0x7efc1c154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc21800640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c06e7b0 0x7efc1c157150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7efc1000c730 tx=0x7efc1000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc19600640 1 -- 172.21.15.67:0/3032266921 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10017070 con 0x7efc1c06e7b0 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc19600640 1 -- 172.21.15.67:0/3032266921 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc10005ac0 con 0x7efc1c06e7b0 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc19600640 1 -- 172.21.15.67:0/3032266921 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10007070 con 0x7efc1c06e7b0 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c15a200 con 0x7efc1c06e7b0 2024-04-17T10:26:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c15a7c0 con 0x7efc1c06e7b0 2024-04-17T10:26:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c04ef30 con 0x7efc1c06e7b0 2024-04-17T10:26:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.119+0000 7efc19600640 1 -- 172.21.15.67:0/3032266921 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efc1001a070 con 0x7efc1c06e7b0 2024-04-17T10:26:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.123+0000 7efc19600640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbfc066f60 0x7efbfc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.123+0000 7efc19600640 1 -- 172.21.15.67:0/3032266921 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6821+0+0 (secure 0 0 0) 0x7efc100848e0 con 0x7efc1c06e7b0 2024-04-17T10:26:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.123+0000 7efc20e00640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbfc066f60 0x7efbfc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.123+0000 7efc20e00640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbfc066f60 0x7efbfc069420 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efc080ad000 tx=0x7efc08008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.127+0000 7efc19600640 1 -- 172.21.15.67:0/3032266921 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efc1c0b4990 con 0x7efc1c06e7b0 2024-04-17T10:26:04.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.327+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efc1c15ac70 con 0x7efc1c06e7b0 2024-04-17T10:26:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.939+0000 7efc19600640 1 -- 172.21.15.67:0/3032266921 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7efc10054da0 con 0x7efc1c06e7b0 2024-04-17T10:26:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbfc066f60 msgr2=0x7efbfc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbfc066f60 0x7efbfc069420 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efc080ad000 tx=0x7efc08008040 comp rx=0 tx=0).stop 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c06e7b0 msgr2=0x7efc1c157150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c06e7b0 0x7efc1c157150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7efc1000c730 tx=0x7efc1000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 shutdown_connections 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efbfc066f60 0x7efbfc069420 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efc1c0cd030 0x7efc1c1546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efc1c0aef60 0x7efc1c154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 --2- 172.21.15.67:0/3032266921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efc1c06e7b0 0x7efc1c157150 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 >> 172.21.15.67:0/3032266921 conn(0x7efc1c0b1740 msgr2=0x7efc1c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 shutdown_connections 2024-04-17T10:26:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:04.943+0000 7efc24200640 1 -- 172.21.15.67:0/3032266921 wait complete. 2024-04-17T10:26:04.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:26:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.159+0000 7f4117c00640 1 Processor -- start 2024-04-17T10:26:05.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.159+0000 7f4117c00640 1 -- start start 2024-04-17T10:26:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.159+0000 7f4117c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4110099b30 0x7f411006cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.159+0000 7f4117c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4110099f10 0x7f411015de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.159+0000 7f4117c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411015f150 con 0x7f411015e3d0 2024-04-17T10:26:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.159+0000 7f4117c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411006c0f0 con 0x7f4110099b30 2024-04-17T10:26:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.159+0000 7f4117c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41100cf980 con 0x7f4110099f10 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4107e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4110099f10 0x7f411015de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4110099b30 0x7f411006cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4110099b30 0x7f411006cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:44298/0 (socket says 172.21.15.67:44298) 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 -- 172.21.15.67:0/2985853234 learned_addr learned my addr 172.21.15.67:0/2985853234 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4107e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4110099f10 0x7f411015de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:33864/0 (socket says 172.21.15.67:33864) 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 -- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4110099f10 msgr2=0x7f411015de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 --2- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4110099f10 0x7f411015de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 -- 172.21.15.67:0/2985853234 >> v1:172.21.15.86:6789/0 conn(0x7f411015e3d0 legacy=0x7f4110161790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 -- 172.21.15.67:0/2985853234 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411015e8b0 con 0x7f4110099b30 2024-04-17T10:26:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4107e00640 1 --2- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4110099f10 0x7f411015de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f4115200640 1 --2- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099b30 0x7f411006cd70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41000098a0 tx=0x7f410001c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91db5176641790e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.163+0000 7f410fe00640 1 -- 172.21.15.67:0/2985853234 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000ab430 con 0x7f4110099b30 2024-04-17T10:26:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f410fe00640 1 -- 172.21.15.67:0/2985853234 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41000b0070 con 0x7f4110099b30 2024-04-17T10:26:05.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f410fe00640 1 -- 172.21.15.67:0/2985853234 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000ab7f0 con 0x7f4110099b30 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099b30 msgr2=0x7f411006cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 --2- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099b30 0x7f411006cd70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f41000098a0 tx=0x7f410001c510 comp rx=0 tx=0).stop 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- 172.21.15.67:0/2985853234 shutdown_connections 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 --2- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4110099f10 0x7f411015de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 --2- 172.21.15.67:0/2985853234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099b30 0x7f411006cd70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- 172.21.15.67:0/2985853234 >> 172.21.15.67:0/2985853234 conn(0x7f41100b32e0 msgr2=0x7f41100b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- 172.21.15.67:0/2985853234 shutdown_connections 2024-04-17T10:26:05.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- 172.21.15.67:0/2985853234 wait complete. 2024-04-17T10:26:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 Processor -- start 2024-04-17T10:26:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- start start 2024-04-17T10:26:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4110099b30 0x7f4110141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099f10 0x7f41100b1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f411015e3d0 0x7f41100b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411009a470 con 0x7f4110099b30 2024-04-17T10:26:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411009a2f0 con 0x7f4110099f10 2024-04-17T10:26:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4117c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41100d10c0 con 0x7f411015e3d0 2024-04-17T10:26:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4115200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4110099b30 0x7f4110141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4115200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4110099b30 0x7f4110141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35758/0 (socket says 172.21.15.67:35758) 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4115200640 1 -- 172.21.15.67:0/2845796874 learned_addr learned my addr 172.21.15.67:0/2845796874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4107e00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099f10 0x7f41100b1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.167+0000 7f4115c00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f411015e3d0 0x7f41100b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4115200640 1 -- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f411015e3d0 msgr2=0x7f41100b1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4115200640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f411015e3d0 0x7f41100b1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4115200640 1 -- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099f10 msgr2=0x7f41100b1820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4115200640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099f10 0x7f41100b1820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4115200640 1 -- 172.21.15.67:0/2845796874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110158490 con 0x7f4110099b30 2024-04-17T10:26:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4115200640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4110099b30 0x7f4110141b90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f41000ad040 tx=0x7f41000192b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f410d600640 1 -- 172.21.15.67:0/2845796874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000ba070 con 0x7f4110099b30 2024-04-17T10:26:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f410d600640 1 -- 172.21.15.67:0/2845796874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41000abe40 con 0x7f4110099b30 2024-04-17T10:26:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f410d600640 1 -- 172.21.15.67:0/2845796874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41000b4660 con 0x7f4110099b30 2024-04-17T10:26:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41101447f0 con 0x7f4110099b30 2024-04-17T10:26:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4110158a10 con 0x7f4110099b30 2024-04-17T10:26:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411009f900 con 0x7f4110099b30 2024-04-17T10:26:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.171+0000 7f410d600640 1 -- 172.21.15.67:0/2845796874 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f41000bd070 con 0x7f4110099b30 2024-04-17T10:26:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.175+0000 7f410d600640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40ec066f60 0x7f40ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.175+0000 7f410d600640 1 -- 172.21.15.67:0/2845796874 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6821+0+0 (secure 0 0 0) 0x7f410011f2c0 con 0x7f4110099b30 2024-04-17T10:26:05.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.175+0000 7f4107e00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40ec066f60 0x7f40ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:05.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.175+0000 7f4107e00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40ec066f60 0x7f40ec069420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009580 tx=0x7f40f8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:05.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.179+0000 7f410d600640 1 -- 172.21.15.67:0/2845796874 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4100016130 con 0x7f4110099b30 2024-04-17T10:26:05.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.343+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4110074e30 con 0x7f4110099b30 2024-04-17T10:26:05.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f410d600640 1 -- 172.21.15.67:0/2845796874 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f41000ef810 con 0x7f4110099b30 2024-04-17T10:26:05.937 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:26:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40ec066f60 msgr2=0x7f40ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40ec066f60 0x7f40ec069420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009580 tx=0x7f40f8009190 comp rx=0 tx=0).stop 2024-04-17T10:26:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4110099b30 msgr2=0x7f4110141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4110099b30 0x7f4110141b90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f41000ad040 tx=0x7f41000192b0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 shutdown_connections 2024-04-17T10:26:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40ec066f60 0x7f40ec069420 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f40f8009580 tx=0x7f40f8009190 comp rx=0 tx=0).stop 2024-04-17T10:26:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f411015e3d0 0x7f41100b1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4110099f10 0x7f41100b1820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 --2- 172.21.15.67:0/2845796874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4110099b30 0x7f4110141b90 secure :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f41000ad040 tx=0x7f41000192b0 comp rx=0 tx=0).stop 2024-04-17T10:26:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 >> 172.21.15.67:0/2845796874 conn(0x7f41100b32e0 msgr2=0x7f41101578a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.943+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 shutdown_connections 2024-04-17T10:26:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:05.947+0000 7f4117c00640 1 -- 172.21.15.67:0/2845796874 wait complete. 2024-04-17T10:26:05.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:26:06.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f019ac00640 1 Processor -- start 2024-04-17T10:26:06.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f019ac00640 1 -- start start 2024-04-17T10:26:06.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f019ac00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 0x7f019406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f019ac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019406d9a0 con 0x7f0194144f00 2024-04-17T10:26:06.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f019ac00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019406dc80 con 0x7f0194071a70 2024-04-17T10:26:06.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f019ac00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01940cc310 con 0x7f01941447e0 2024-04-17T10:26:06.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0194144f00 0x7f0194071360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58950/0 (socket says 172.21.15.67:58950) 2024-04-17T10:26:06.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 -- 172.21.15.67:0/1006363355 learned_addr learned my addr 172.21.15.67:0/1006363355 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193e00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 0x7f019406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106764826 0 0) 0x7f019406d9a0 con 0x7f0194144f00 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0178003530 con 0x7f0194144f00 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756529700 0 0) 0x7f019406dc80 con 0x7f0194071a70 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f019406d9a0 con 0x7f0194071a70 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 159900818 0 0) 0x7f0178003530 con 0x7f0194144f00 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 msgr2=0x7f019406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 0x7f019406d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 >> v1:172.21.15.86:6789/0 conn(0x7f0194071a70 legacy=0x7f01940cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019406df60 con 0x7f0194144f00 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f0180001680 con 0x7f0194144f00 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193e00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 0x7f019406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 >> v1:172.21.15.67:6789/0 conn(0x7f0194144f00 legacy=0x7f0194071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 0x7f0178003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0178004110 0x7f0178004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0178004d50 0x7f01780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940cc310 con 0x7f0178004110 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017800bb90 con 0x7f0178004d50 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017800bd90 con 0x7f01941447e0 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0178004110 0x7f0178004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193e00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 0x7f0178003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0198c00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0178004d50 0x7f01780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 -- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 msgr2=0x7f0178003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f01941447e0 0x7f0178003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 -- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0178004d50 msgr2=0x7f01780051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0178004d50 0x7f01780051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.119+0000 7f0193400640 1 -- 172.21.15.67:0/1006363355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0178003530 con 0x7f0178004110 2024-04-17T10:26:06.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f0193400640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0178004110 0x7f0178004810 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f018000c0e0 tx=0x7f018001cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18648d8e8181315d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0180007c50 con 0x7f0178004110 2024-04-17T10:26:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01800a9070 con 0x7f0178004110 2024-04-17T10:26:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f0192a00640 1 -- 172.21.15.67:0/1006363355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0180005da0 con 0x7f0178004110 2024-04-17T10:26:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f019ac00640 1 -- 172.21.15.67:0/1006363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0178004110 msgr2=0x7f0178004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f019ac00640 1 --2- 172.21.15.67:0/1006363355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0178004110 0x7f0178004810 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f018000c0e0 tx=0x7f018001cb10 comp rx=0 tx=0).stop 2024-04-17T10:26:06.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f0198c00640 1 -- 172.21.15.67:0/1006363355 reap_dead start 2024-04-17T10:26:06.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f019ac00640 1 -- 172.21.15.67:0/1006363355 shutdown_connections 2024-04-17T10:26:06.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f019ac00640 1 -- 172.21.15.67:0/1006363355 >> 172.21.15.67:0/1006363355 conn(0x7f019406ac90 msgr2=0x7f019406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:06.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f019ac00640 1 -- 172.21.15.67:0/1006363355 shutdown_connections 2024-04-17T10:26:06.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.123+0000 7f019ac00640 1 -- 172.21.15.67:0/1006363355 wait complete. 2024-04-17T10:26:06.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 Processor -- start 2024-04-17T10:26:06.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 -- start start 2024-04-17T10:26:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0178004110 0x7f0194159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01941f9080 0x7f019413a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01940b08f0 0x7f01940b0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940cda90 con 0x7f01940b08f0 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01940cd910 con 0x7f01941f9080 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0194145530 con 0x7f0178004110 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01940b08f0 0x7f01940b0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01940b08f0 0x7f01940b0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35778/0 (socket says 172.21.15.67:35778) 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 -- 172.21.15.67:0/691539116 learned_addr learned my addr 172.21.15.67:0/691539116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0193e00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0178004110 0x7f0194159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0193400640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01941f9080 0x7f019413a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 -- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0178004110 msgr2=0x7f0194159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0178004110 0x7f0194159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 -- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01941f9080 msgr2=0x7f019413a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01941f9080 0x7f019413a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 -- 172.21.15.67:0/691539116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019413afd0 con 0x7f01940b08f0 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0193400640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01941f9080 0x7f019413a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:06.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f0198c00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01940b08f0 0x7f01940b0d60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f019406bb40 tx=0x7f018c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f018be00640 1 -- 172.21.15.67:0/691539116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c003980 con 0x7f01940b08f0 2024-04-17T10:26:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f018be00640 1 -- 172.21.15.67:0/691539116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c018070 con 0x7f01940b08f0 2024-04-17T10:26:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f018be00640 1 -- 172.21.15.67:0/691539116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c003d80 con 0x7f01940b08f0 2024-04-17T10:26:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019413b2b0 con 0x7f01940b08f0 2024-04-17T10:26:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.127+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0194137390 con 0x7f01940b08f0 2024-04-17T10:26:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.131+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0194072da0 con 0x7f01940b08f0 2024-04-17T10:26:06.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.131+0000 7f018be00640 1 -- 172.21.15.67:0/691539116 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f018c0050d0 con 0x7f01940b08f0 2024-04-17T10:26:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.131+0000 7f018be00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0174066f90 0x7f0174069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.131+0000 7f018be00640 1 -- 172.21.15.67:0/691539116 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6383+0+0 (secure 0 0 0) 0x7f018c085080 con 0x7f01940b08f0 2024-04-17T10:26:06.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.131+0000 7f0193e00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0174066f90 0x7f0174069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.131+0000 7f0193e00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0174066f90 0x7f0174069450 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0184009af0 tx=0x7f0184009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:06.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.135+0000 7f018be00640 1 -- 172.21.15.67:0/691539116 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f018c055d50 con 0x7f01940b08f0 2024-04-17T10:26:06.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.303+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f019415a5e0 con 0x7f01940b08f0 2024-04-17T10:26:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.951+0000 7f018be00640 1 -- 172.21.15.67:0/691539116 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f018c055650 con 0x7f01940b08f0 2024-04-17T10:26:06.947 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:26:06.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0174066f90 msgr2=0x7f0174069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:06.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0174066f90 0x7f0174069450 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0184009af0 tx=0x7f0184009190 comp rx=0 tx=0).stop 2024-04-17T10:26:06.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01940b08f0 msgr2=0x7f01940b0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:06.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01940b08f0 0x7f01940b0d60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f019406bb40 tx=0x7f018c00eb90 comp rx=0 tx=0).stop 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 shutdown_connections 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0174066f90 0x7f0174069450 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f01940b08f0 0x7f01940b0d60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f01941f9080 0x7f019413a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 --2- 172.21.15.67:0/691539116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0178004110 0x7f0194159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 >> 172.21.15.67:0/691539116 conn(0x7f019406ac90 msgr2=0x7f0194142990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 shutdown_connections 2024-04-17T10:26:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:06.955+0000 7f019ac00640 1 -- 172.21.15.67:0/691539116 wait complete. 2024-04-17T10:26:06.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-17T10:26:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.123+0000 7f374ae00640 1 Processor -- start 2024-04-17T10:26:07.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.123+0000 7f374ae00640 1 -- start start 2024-04-17T10:26:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.123+0000 7f374ae00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f374406d3c0 0x7f3744071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.123+0000 7f374ae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3744071cb0 con 0x7f37441447e0 2024-04-17T10:26:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.123+0000 7f374ae00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374406d9a0 con 0x7f37440718a0 2024-04-17T10:26:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.123+0000 7f374ae00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37440cc310 con 0x7f374406d3c0 2024-04-17T10:26:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.131+0000 7f3748e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f37440718a0 0x7f37440cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:39010/0 (socket says 172.21.15.67:39010) 2024-04-17T10:26:07.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.131+0000 7f3748e00640 1 -- 172.21.15.67:0/1203523345 learned_addr learned my addr 172.21.15.67:0/1203523345 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:07.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.131+0000 7f3743400640 1 --2- 172.21.15.67:0/1203523345 >> v2:172.21.15.86:3300/0 conn(0x7f374406d3c0 0x7f3744071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:07.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.131+0000 7f3742a00640 1 -- 172.21.15.67:0/1203523345 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999232640 0 0) 0x7f374406d9a0 con 0x7f37440718a0 2024-04-17T10:26:07.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.131+0000 7f3742a00640 1 -- 172.21.15.67:0/1203523345 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3728003530 con 0x7f37440718a0 2024-04-17T10:26:07.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.131+0000 7f3742a00640 1 -- 172.21.15.67:0/1203523345 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463021351 0 0) 0x7f3734001050 con 0x7f37441447e0 2024-04-17T10:26:07.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.131+0000 7f3742a00640 1 -- 172.21.15.67:0/1203523345 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f374406d9a0 con 0x7f37441447e0 2024-04-17T10:26:07.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.135+0000 7f3743400640 1 -- 172.21.15.67:0/1203523345 >> v1:172.21.15.86:6789/0 conn(0x7f37440718a0 legacy=0x7f37440cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:07.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.135+0000 7f3743400640 1 -- 172.21.15.67:0/1203523345 >> v1:172.21.15.67:6789/0 conn(0x7f37441447e0 legacy=0x7f3744144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:07.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.135+0000 7f3743400640 1 -- 172.21.15.67:0/1203523345 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374406dc80 con 0x7f374406d3c0 2024-04-17T10:26:07.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.139+0000 7f3743400640 1 --2- 172.21.15.67:0/1203523345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374406d3c0 0x7f3744071360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3730009740 tx=0x7f373001c240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66f7190db540d843 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.139+0000 7f3742a00640 1 -- 172.21.15.67:0/1203523345 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300b0070 con 0x7f374406d3c0 2024-04-17T10:26:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.139+0000 7f3742a00640 1 -- 172.21.15.67:0/1203523345 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37300ab400 con 0x7f374406d3c0 2024-04-17T10:26:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.139+0000 7f3742a00640 1 -- 172.21.15.67:0/1203523345 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3734001680 con 0x7f374406d3c0 2024-04-17T10:26:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.143+0000 7f374ae00640 1 -- 172.21.15.67:0/1203523345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374406d3c0 msgr2=0x7f3744071360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.143+0000 7f374ae00640 1 --2- 172.21.15.67:0/1203523345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374406d3c0 0x7f3744071360 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3730009740 tx=0x7f373001c240 comp rx=0 tx=0).stop 2024-04-17T10:26:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.143+0000 7f374ae00640 1 -- 172.21.15.67:0/1203523345 shutdown_connections 2024-04-17T10:26:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.143+0000 7f374ae00640 1 --2- 172.21.15.67:0/1203523345 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f374406d3c0 0x7f3744071360 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.143+0000 7f374ae00640 1 -- 172.21.15.67:0/1203523345 >> 172.21.15.67:0/1203523345 conn(0x7f374406ac90 msgr2=0x7f374406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.143+0000 7f374ae00640 1 -- 172.21.15.67:0/1203523345 shutdown_connections 2024-04-17T10:26:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.143+0000 7f374ae00640 1 -- 172.21.15.67:0/1203523345 wait complete. 2024-04-17T10:26:07.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 Processor -- start 2024-04-17T10:26:07.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 -- start start 2024-04-17T10:26:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374406d3c0 0x7f374415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37440718a0 0x7f3744137600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37441447e0 0x7f37441420c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37440cda90 con 0x7f37440718a0 2024-04-17T10:26:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37440cd910 con 0x7f37441447e0 2024-04-17T10:26:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37440b2900 con 0x7f374406d3c0 2024-04-17T10:26:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37440718a0 0x7f3744137600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:07.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37440718a0 0x7f3744137600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35784/0 (socket says 172.21.15.67:35784) 2024-04-17T10:26:07.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 -- 172.21.15.67:0/749537658 learned_addr learned my addr 172.21.15.67:0/749537658 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:07.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743e00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374406d3c0 0x7f374415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:07.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3748e00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37441447e0 0x7f37441420c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 -- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374406d3c0 msgr2=0x7f374415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374406d3c0 0x7f374415ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 -- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37441447e0 msgr2=0x7f37441420c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37441447e0 0x7f37441420c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 -- 172.21.15.67:0/749537658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374415b0a0 con 0x7f37440718a0 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f3743400640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37440718a0 0x7f3744137600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f37300ae040 tx=0x7f373001aa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f373be00640 1 -- 172.21.15.67:0/749537658 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300b0050 con 0x7f37440718a0 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f373be00640 1 -- 172.21.15.67:0/749537658 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37300b1040 con 0x7f37440718a0 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f373be00640 1 -- 172.21.15.67:0/749537658 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c001a00 con 0x7f37440718a0 2024-04-17T10:26:07.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3744133670 con 0x7f37440718a0 2024-04-17T10:26:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.147+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3744133c30 con 0x7f37440718a0 2024-04-17T10:26:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.151+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3744072da0 con 0x7f37440718a0 2024-04-17T10:26:07.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.151+0000 7f373be00640 1 -- 172.21.15.67:0/749537658 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f37300be030 con 0x7f37440718a0 2024-04-17T10:26:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.151+0000 7f373be00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3724066fb0 0x7f3724069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.151+0000 7f373be00640 1 -- 172.21.15.67:0/749537658 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f373011e9a0 con 0x7f37440718a0 2024-04-17T10:26:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.151+0000 7f3743e00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3724066fb0 0x7f3724069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:07.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.151+0000 7f3743e00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3724066fb0 0x7f3724069470 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f37340018b0 tx=0x7f3734009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:07.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.155+0000 7f373be00640 1 -- 172.21.15.67:0/749537658 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f37300c5060 con 0x7f37440718a0 2024-04-17T10:26:07.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.319+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3744074e30 con 0x7f37440718a0 2024-04-17T10:26:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.959+0000 7f373be00640 1 -- 172.21.15.67:0/749537658 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7f3730015fb0 con 0x7f37440718a0 2024-04-17T10:26:07.954 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data_ec' removed 2024-04-17T10:26:07.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3724066fb0 msgr2=0x7f3724069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3724066fb0 0x7f3724069470 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f37340018b0 tx=0x7f3734009040 comp rx=0 tx=0).stop 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37440718a0 msgr2=0x7f3744137600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37440718a0 0x7f3744137600 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f37300ae040 tx=0x7f373001aa80 comp rx=0 tx=0).stop 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 shutdown_connections 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3724066fb0 0x7f3724069470 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37441447e0 0x7f37441420c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37440718a0 0x7f3744137600 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 --2- 172.21.15.67:0/749537658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f374406d3c0 0x7f374415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 >> 172.21.15.67:0/749537658 conn(0x7f374406ac90 msgr2=0x7f37440b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 shutdown_connections 2024-04-17T10:26:07.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:07.963+0000 7f374ae00640 1 -- 172.21.15.67:0/749537658 wait complete. 2024-04-17T10:26:07.985 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:26:07.985 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:26:07.985 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:26:07.985 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:26:07.985 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:26:07.985 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:26:07.986 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:26:07.986 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:26:07.986 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:26:07.986 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:26:07.986 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:26:07.986 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:26:07.987 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:26:07.987 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:26:07.987 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:26:07.987 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:26:07.999 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:26:07.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:26:08.000 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:26:08.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:26:08.001 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:26:08.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:26:08.002 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:26:08.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:26:08.042 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:26:08.043 DEBUG:teuthology.parallel:result is None 2024-04-17T10:26:08.044 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:26:08.045 DEBUG:teuthology.parallel:result is None 2024-04-17T10:26:08.051 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:26:08.051 DEBUG:teuthology.parallel:result is None 2024-04-17T10:26:08.052 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:26:08.053 DEBUG:teuthology.parallel:result is None 2024-04-17T10:26:08.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:26:08.131 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:26:08.131 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:26:08.139 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:26:08.141 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:26:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ea00640 1 Processor -- start 2024-04-17T10:26:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ea00640 1 -- start start 2024-04-17T10:26:08.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d68145040 0x7f1d6806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:08.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ea00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1d68071a20 0x7f1d680cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:08.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ea00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6806e2f0 con 0x7f1d6806ef10 2024-04-17T10:26:08.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ea00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d680a86c0 con 0x7f1d68071a20 2024-04-17T10:26:08.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6815bf10 con 0x7f1d68145040 2024-04-17T10:26:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ca00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1d68071a20 0x7f1d680cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d67e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d68145040 0x7f1d6806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:08.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ca00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1d68071a20 0x7f1d680cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:44358/0 (socket says 172.21.15.67:44358) 2024-04-17T10:26:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.207+0000 7f1d6ca00640 1 -- 172.21.15.67:0/2181348725 learned_addr learned my addr 172.21.15.67:0/2181348725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d6ca00640 1 -- 172.21.15.67:0/2181348725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d68145040 msgr2=0x7f1d6806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d6ca00640 1 --2- 172.21.15.67:0/2181348725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d68145040 0x7f1d6806e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d6ca00640 1 -- 172.21.15.67:0/2181348725 >> v1:172.21.15.86:6789/0 conn(0x7f1d6806ef10 legacy=0x7f1d68071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d67400640 1 -- 172.21.15.67:0/2181348725 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244514994 0 0) 0x7f1d6806e2f0 con 0x7f1d6806ef10 2024-04-17T10:26:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d6ca00640 1 -- 172.21.15.67:0/2181348725 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6806d6e0 con 0x7f1d68071a20 2024-04-17T10:26:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d6ca00640 1 --2- 172.21.15.67:0/2181348725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 0x7f1d680cc300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009e50 tx=0x7f1d6001f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=685db01b1c346c1a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d67400640 1 -- 172.21.15.67:0/2181348725 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d600b4070 con 0x7f1d68071a20 2024-04-17T10:26:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d67400640 1 -- 172.21.15.67:0/2181348725 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d600af400 con 0x7f1d68071a20 2024-04-17T10:26:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.211+0000 7f1d67400640 1 -- 172.21.15.67:0/2181348725 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d600af7d0 con 0x7f1d68071a20 2024-04-17T10:26:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- 172.21.15.67:0/2181348725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 msgr2=0x7f1d680cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/2181348725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 0x7f1d680cc300 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1d60009e50 tx=0x7f1d6001f210 comp rx=0 tx=0).stop 2024-04-17T10:26:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- 172.21.15.67:0/2181348725 shutdown_connections 2024-04-17T10:26:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/2181348725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 0x7f1d680cc300 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/2181348725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d68145040 0x7f1d6806e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- 172.21.15.67:0/2181348725 >> 172.21.15.67:0/2181348725 conn(0x7f1d6806ac90 msgr2=0x7f1d6806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- 172.21.15.67:0/2181348725 shutdown_connections 2024-04-17T10:26:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- 172.21.15.67:0/2181348725 wait complete. 2024-04-17T10:26:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 Processor -- start 2024-04-17T10:26:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- start start 2024-04-17T10:26:08.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6806ef10 0x7f1d6813e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 0x7f1d6813fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d68145040 0x7f1d6813c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6806df80 con 0x7f1d68145040 2024-04-17T10:26:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6806dca0 con 0x7f1d68071a20 2024-04-17T10:26:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6806d370 con 0x7f1d6806ef10 2024-04-17T10:26:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d68145040 0x7f1d6813c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:08.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d68145040 0x7f1d6813c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35826/0 (socket says 172.21.15.67:35826) 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 -- 172.21.15.67:0/4094250527 learned_addr learned my addr 172.21.15.67:0/4094250527 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d67e00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6806ef10 0x7f1d6813e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d5fe00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 0x7f1d6813fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 -- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6806ef10 msgr2=0x7f1d6813e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6806ef10 0x7f1d6813e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 -- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 msgr2=0x7f1d6813fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 0x7f1d6813fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.215+0000 7f1d6ca00640 1 -- 172.21.15.67:0/4094250527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d681f42f0 con 0x7f1d68145040 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d6ca00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d68145040 0x7f1d6813c360 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1d6001ce40 tx=0x7f1d6001bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d64c00640 1 -- 172.21.15.67:0/4094250527 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d600b4040 con 0x7f1d68145040 2024-04-17T10:26:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d64c00640 1 -- 172.21.15.67:0/4094250527 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d600090d0 con 0x7f1d68145040 2024-04-17T10:26:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d64c00640 1 -- 172.21.15.67:0/4094250527 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d60009510 con 0x7f1d68145040 2024-04-17T10:26:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d681f45d0 con 0x7f1d68145040 2024-04-17T10:26:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d681f4b90 con 0x7f1d68145040 2024-04-17T10:26:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d68072d50 con 0x7f1d68145040 2024-04-17T10:26:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d64c00640 1 -- 172.21.15.67:0/4094250527 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1d600c2020 con 0x7f1d68145040 2024-04-17T10:26:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d64c00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d44066f60 0x7f1d44069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d64c00640 1 -- 172.21.15.67:0/4094250527 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f1d60122890 con 0x7f1d68145040 2024-04-17T10:26:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.219+0000 7f1d67e00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d44066f60 0x7f1d44069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.223+0000 7f1d67e00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d44066f60 0x7f1d44069420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009ac0 tx=0x7f1d58009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.223+0000 7f1d64c00640 1 -- 172.21.15.67:0/4094250527 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1d600c9020 con 0x7f1d68145040 2024-04-17T10:26:08.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.407+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1d68074e30 con 0x7f1d68145040 2024-04-17T10:26:08.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d64c00640 1 -- 172.21.15.67:0/4094250527 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1d60018d70 con 0x7f1d68145040 2024-04-17T10:26:08.969 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d44066f60 msgr2=0x7f1d44069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d44066f60 0x7f1d44069420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f1d58009ac0 tx=0x7f1d58009190 comp rx=0 tx=0).stop 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d68145040 msgr2=0x7f1d6813c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d68145040 0x7f1d6813c360 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1d6001ce40 tx=0x7f1d6001bfb0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 shutdown_connections 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d44066f60 0x7f1d44069420 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d68145040 0x7f1d6813c360 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d68071a20 0x7f1d6813fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 --2- 172.21.15.67:0/4094250527 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6806ef10 0x7f1d6813e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 >> 172.21.15.67:0/4094250527 conn(0x7f1d6806ac90 msgr2=0x7f1d680cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 shutdown_connections 2024-04-17T10:26:08.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:08.975+0000 7f1d6ea00640 1 -- 172.21.15.67:0/4094250527 wait complete. 2024-04-17T10:26:08.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:26:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27fa200640 1 Processor -- start 2024-04-17T10:26:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27fa200640 1 -- start start 2024-04-17T10:26:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27fa200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27f409a160 0x7f27f40cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27fa200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40d0760 con 0x7f27f4161160 2024-04-17T10:26:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27fa200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f40cc2a0 con 0x7f27f40b2570 2024-04-17T10:26:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27fa200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f40cfc70 con 0x7f27f409a160 2024-04-17T10:26:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f8e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27f4161160 0x7f27f4099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58966/0 (socket says 172.21.15.67:58966) 2024-04-17T10:26:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f8e00640 1 -- 172.21.15.67:0/1166702995 learned_addr learned my addr 172.21.15.67:0/1166702995 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:09.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f3e00640 1 --2- 172.21.15.67:0/1166702995 >> v2:172.21.15.86:3300/0 conn(0x7f27f409a160 0x7f27f40cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f3400640 1 -- 172.21.15.67:0/1166702995 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186847082 0 0) 0x7f27f40d0760 con 0x7f27f4161160 2024-04-17T10:26:09.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f3400640 1 -- 172.21.15.67:0/1166702995 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27dc003530 con 0x7f27f4161160 2024-04-17T10:26:09.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f3e00640 1 -- 172.21.15.67:0/1166702995 >> v1:172.21.15.86:6789/0 conn(0x7f27f40b2570 legacy=0x7f27f40ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f3e00640 1 -- 172.21.15.67:0/1166702995 >> v1:172.21.15.67:6789/0 conn(0x7f27f4161160 legacy=0x7f27f4099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.147+0000 7f27f3e00640 1 -- 172.21.15.67:0/1166702995 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f40cfec0 con 0x7f27f409a160 2024-04-17T10:26:09.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27f3e00640 1 --2- 172.21.15.67:0/1166702995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f409a160 0x7f27f40cca60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097c0 tx=0x7f27e001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95a3090b2ec3e4dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27f3400640 1 -- 172.21.15.67:0/1166702995 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00b0070 con 0x7f27f409a160 2024-04-17T10:26:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27f3400640 1 -- 172.21.15.67:0/1166702995 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e00ab600 con 0x7f27f409a160 2024-04-17T10:26:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27f3400640 1 -- 172.21.15.67:0/1166702995 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00ab9d0 con 0x7f27f409a160 2024-04-17T10:26:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27fa200640 1 -- 172.21.15.67:0/1166702995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f409a160 msgr2=0x7f27f40cca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27fa200640 1 --2- 172.21.15.67:0/1166702995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f409a160 0x7f27f40cca60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097c0 tx=0x7f27e001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27fa200640 1 -- 172.21.15.67:0/1166702995 shutdown_connections 2024-04-17T10:26:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27fa200640 1 --2- 172.21.15.67:0/1166702995 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f409a160 0x7f27f40cca60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27fa200640 1 -- 172.21.15.67:0/1166702995 >> 172.21.15.67:0/1166702995 conn(0x7f27f40d1690 msgr2=0x7f27f40d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27fa200640 1 -- 172.21.15.67:0/1166702995 shutdown_connections 2024-04-17T10:26:09.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.151+0000 7f27fa200640 1 -- 172.21.15.67:0/1166702995 wait complete. 2024-04-17T10:26:09.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 Processor -- start 2024-04-17T10:26:09.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 -- start start 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f409a160 0x7f27f4144fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f40b2570 0x7f27f414dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f4161160 0x7f27f4154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f406bf20 con 0x7f27f40b2570 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f406bda0 con 0x7f27f4161160 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f40d23d0 con 0x7f27f409a160 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f40b2570 0x7f27f414dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f40b2570 0x7f27f414dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35838/0 (socket says 172.21.15.67:35838) 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 -- 172.21.15.67:0/4148239621 learned_addr learned my addr 172.21.15.67:0/4148239621 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f8e00640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f409a160 0x7f27f4144fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f9800640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f4161160 0x7f27f4154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 -- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f409a160 msgr2=0x7f27f4144fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f409a160 0x7f27f4144fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 -- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f4161160 msgr2=0x7f27f4154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f4161160 0x7f27f4154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 -- 172.21.15.67:0/4148239621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f41454e0 con 0x7f27f40b2570 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f3e00640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f40b2570 0x7f27f414dcd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f27e00ae040 tx=0x7f27e0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f0c00640 1 -- 172.21.15.67:0/4148239621 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00ba070 con 0x7f27f40b2570 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f0c00640 1 -- 172.21.15.67:0/4148239621 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e00b43f0 con 0x7f27f40b2570 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27f0c00640 1 -- 172.21.15.67:0/4148239621 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00b0040 con 0x7f27f40b2570 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f40b5160 con 0x7f27f40b2570 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f40c7380 con 0x7f27f40b2570 2024-04-17T10:26:09.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.155+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f40b5330 con 0x7f27f40b2570 2024-04-17T10:26:09.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.159+0000 7f27f0c00640 1 -- 172.21.15.67:0/4148239621 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f27e00bd070 con 0x7f27f40b2570 2024-04-17T10:26:09.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.159+0000 7f27f0c00640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27d4066f60 0x7f27d4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.159+0000 7f27f0c00640 1 -- 172.21.15.67:0/4148239621 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f27e011df00 con 0x7f27f40b2570 2024-04-17T10:26:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.159+0000 7f27f8e00640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27d4066f60 0x7f27d4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.159+0000 7f27f8e00640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27d4066f60 0x7f27d4069420 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f27e80011f0 tx=0x7f27e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.163+0000 7f27f0c00640 1 -- 172.21.15.67:0/4148239621 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f27e0016090 con 0x7f27f40b2570 2024-04-17T10:26:09.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.379+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f27f4074dc0 con 0x7f27f40b2570 2024-04-17T10:26:09.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27f0c00640 1 -- 172.21.15.67:0/4148239621 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f27e00ef0c0 con 0x7f27f40b2570 2024-04-17T10:26:09.376 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:09.376 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:26:09.376 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:26:09.377 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.378 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.379 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.380 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.381 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.382 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:26:09.383 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.384 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.385 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:26:09.386 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.387 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mds", 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBdox9mmFSrBRAAisAnkXNgCopMaYuVp4+OXQ==", 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mds" 2024-04-17T10:26:09.388 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-mgr", 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBdox9mBV+rBRAAbM0zYRTbLs1S9clth6Q8xg==", 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-osd", 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBdox9mnWerBRAAZKMGBmL9oNyXtPtyDZDy0A==", 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-osd" 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.389 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd", 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBdox9mvG+rBRAA8GWo5R+GJtf5nDlupDVZAw==", 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBdox9mIHirBRAAwOIobXOpT52/abmHOO6YUQ==", 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.390 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.bootstrap-rgw", 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBdox9mxYCrBRAAL5ibeuOMTW0+KCDLs72o6A==", 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.391 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:26:09.392 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:26:09.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27d4066f60 msgr2=0x7f27d4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27d4066f60 0x7f27d4069420 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f27e80011f0 tx=0x7f27e8009040 comp rx=0 tx=0).stop 2024-04-17T10:26:09.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f40b2570 msgr2=0x7f27f414dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f40b2570 0x7f27f414dcd0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f27e00ae040 tx=0x7f27e0019240 comp rx=0 tx=0).stop 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 shutdown_connections 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27d4066f60 0x7f27d4069420 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27f4161160 0x7f27f4154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27f40b2570 0x7f27f414dcd0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 --2- 172.21.15.67:0/4148239621 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27f409a160 0x7f27f4144fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 >> 172.21.15.67:0/4148239621 conn(0x7f27f40d1690 msgr2=0x7f27f406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 shutdown_connections 2024-04-17T10:26:09.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.383+0000 7f27fa200640 1 -- 172.21.15.67:0/4148239621 wait complete. 2024-04-17T10:26:09.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-17T10:26:09.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.543+0000 7f2106400640 1 Processor -- start 2024-04-17T10:26:09.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.543+0000 7f2106400640 1 -- start start 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f2106400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21000cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f2106400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f21000cdcf0 0x7f21000cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f2106400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210006b100 con 0x7f21000cb260 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f2106400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21000cd150 con 0x7f21000cdcf0 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f2106400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210006b3e0 con 0x7f21000cc250 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20f7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f21000cb260 0x7f21000cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58974/0 (socket says 172.21.15.67:58974) 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20f7e00640 1 -- 172.21.15.67:0/566917086 learned_addr learned my addr 172.21.15.67:0/566917086 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20ff400640 1 --2- 172.21.15.67:0/566917086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21000cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20ffe00640 1 --2- 172.21.15.67:0/566917086 >> v2:172.21.15.86:3300/0 conn(0x7f21000cdcf0 0x7f21000cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20fea00640 1 -- 172.21.15.67:0/566917086 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772853534 0 0) 0x7f210006b100 con 0x7f21000cb260 2024-04-17T10:26:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20fea00640 1 -- 172.21.15.67:0/566917086 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e4003530 con 0x7f21000cb260 2024-04-17T10:26:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20ff400640 1 -- 172.21.15.67:0/566917086 >> v2:172.21.15.86:3300/0 conn(0x7f21000cdcf0 msgr2=0x7f21000cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20ff400640 1 --2- 172.21.15.67:0/566917086 >> v2:172.21.15.86:3300/0 conn(0x7f21000cdcf0 0x7f21000cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20ff400640 1 -- 172.21.15.67:0/566917086 >> v1:172.21.15.67:6789/0 conn(0x7f21000cb260 legacy=0x7f21000cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20ff400640 1 -- 172.21.15.67:0/566917086 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210006b6c0 con 0x7f21000cc250 2024-04-17T10:26:09.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.547+0000 7f20ffe00640 1 --2- 172.21.15.67:0/566917086 >> v2:172.21.15.86:3300/0 conn(0x7f21000cdcf0 0x7f21000cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f20ff400640 1 --2- 172.21.15.67:0/566917086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21000cad20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f20f00097c0 tx=0x7f20f001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a555577c7687ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f20fea00640 1 -- 172.21.15.67:0/566917086 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f00b0070 con 0x7f21000cc250 2024-04-17T10:26:09.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f20fea00640 1 -- 172.21.15.67:0/566917086 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f00ab600 con 0x7f21000cc250 2024-04-17T10:26:09.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f20fea00640 1 -- 172.21.15.67:0/566917086 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f00ab9d0 con 0x7f21000cc250 2024-04-17T10:26:09.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- 172.21.15.67:0/566917086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 msgr2=0x7f21000cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 --2- 172.21.15.67:0/566917086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21000cad20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f20f00097c0 tx=0x7f20f001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- 172.21.15.67:0/566917086 shutdown_connections 2024-04-17T10:26:09.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 --2- 172.21.15.67:0/566917086 >> v2:172.21.15.86:3300/0 conn(0x7f21000cdcf0 0x7f21000cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 --2- 172.21.15.67:0/566917086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21000cad20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- 172.21.15.67:0/566917086 >> 172.21.15.67:0/566917086 conn(0x7f21000b22a0 msgr2=0x7f21000b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:09.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- 172.21.15.67:0/566917086 shutdown_connections 2024-04-17T10:26:09.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- 172.21.15.67:0/566917086 wait complete. 2024-04-17T10:26:09.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 Processor -- start 2024-04-17T10:26:09.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- start start 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21000cb260 0x7f21000ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21001469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21000cdcf0 0x7f2100146f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000d02e0 con 0x7f21000cb260 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21000d0160 con 0x7f21000cdcf0 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.551+0000 7f2106400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210016b100 con 0x7f21000cc250 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21000cb260 0x7f21000ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21000cb260 0x7f21000ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35848/0 (socket says 172.21.15.67:35848) 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 -- 172.21.15.67:0/325720535 learned_addr learned my addr 172.21.15.67:0/325720535 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ffe00640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21000cdcf0 0x7f2100146f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20f7e00640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21001469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 -- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 msgr2=0x7f21001469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21001469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 -- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21000cdcf0 msgr2=0x7f2100146f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21000cdcf0 0x7f2100146f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 -- 172.21.15.67:0/325720535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21000cad70 con 0x7f21000cb260 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20ff400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21000cb260 0x7f21000ca830 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20f00097c0 tx=0x7f20f0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20f7400640 1 -- 172.21.15.67:0/325720535 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f00b0040 con 0x7f21000cb260 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20f7400640 1 -- 172.21.15.67:0/325720535 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20f00ba070 con 0x7f21000cb260 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20f7400640 1 -- 172.21.15.67:0/325720535 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20f00b4850 con 0x7f21000cb260 2024-04-17T10:26:09.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21000b6510 con 0x7f21000cb260 2024-04-17T10:26:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f210015fc40 con 0x7f21000cb260 2024-04-17T10:26:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f210009e900 con 0x7f21000cb260 2024-04-17T10:26:09.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20f7400640 1 -- 172.21.15.67:0/325720535 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f20f00b3070 con 0x7f21000cb260 2024-04-17T10:26:09.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.555+0000 7f20f7400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20dc066f60 0x7f20dc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.559+0000 7f20f7400640 1 -- 172.21.15.67:0/325720535 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f20f011e7c0 con 0x7f21000cb260 2024-04-17T10:26:09.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.559+0000 7f20f7e00640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20dc066f60 0x7f20dc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.559+0000 7f20f7e00640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20dc066f60 0x7f20dc069420 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2100160a00 tx=0x7f20e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.559+0000 7f20f7400640 1 -- 172.21.15.67:0/325720535 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f20f0016090 con 0x7f21000cb260 2024-04-17T10:26:09.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.779+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f21000b2720 con 0x7f21000cb260 2024-04-17T10:26:09.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f20f7400640 1 -- 172.21.15.67:0/325720535 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f20f00ef990 con 0x7f21000cb260 2024-04-17T10:26:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20dc066f60 msgr2=0x7f20dc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20dc066f60 0x7f20dc069420 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2100160a00 tx=0x7f20e8009040 comp rx=0 tx=0).stop 2024-04-17T10:26:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21000cb260 msgr2=0x7f21000ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21000cb260 0x7f21000ca830 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f20f00097c0 tx=0x7f20f0019240 comp rx=0 tx=0).stop 2024-04-17T10:26:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 shutdown_connections 2024-04-17T10:26:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20dc066f60 0x7f20dc069420 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21000cdcf0 0x7f2100146f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f21000cc250 0x7f21001469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 --2- 172.21.15.67:0/325720535 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21000cb260 0x7f21000ca830 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 >> 172.21.15.67:0/325720535 conn(0x7f21000b22a0 msgr2=0x7f21000ad580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 shutdown_connections 2024-04-17T10:26:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.783+0000 7f2106400640 1 -- 172.21.15.67:0/325720535 wait complete. 2024-04-17T10:26:09.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-17T10:26:09.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.943+0000 7f4750c00640 1 Processor -- start 2024-04-17T10:26:09.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.943+0000 7f4750c00640 1 -- start start 2024-04-17T10:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.943+0000 7f4750c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f474c145040 0x7f474c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4750c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c06e2f0 con 0x7f474c071a20 2024-04-17T10:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4750c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f474c06d6e0 con 0x7f474c06ef10 2024-04-17T10:26:09.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4750c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474c0a86c0 con 0x7f474c145040 2024-04-17T10:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f474c145040 0x7f474c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f474a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f474c071a20 0x7f474c0cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58982/0 (socket says 172.21.15.67:58982) 2024-04-17T10:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f474a800640 1 -- 172.21.15.67:0/2816968471 learned_addr learned my addr 172.21.15.67:0/2816968471 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749e00640 1 -- 172.21.15.67:0/2816968471 >> v1:172.21.15.86:6789/0 conn(0x7f474c06ef10 legacy=0x7f474c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749e00640 1 -- 172.21.15.67:0/2816968471 >> v1:172.21.15.67:6789/0 conn(0x7f474c071a20 legacy=0x7f474c0cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749e00640 1 -- 172.21.15.67:0/2816968471 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c06d9c0 con 0x7f474c145040 2024-04-17T10:26:09.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749400640 1 -- 172.21.15.67:0/2816968471 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357942134 0 0) 0x7f474c06e2f0 con 0x7f474c071a20 2024-04-17T10:26:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749e00640 1 --2- 172.21.15.67:0/2816968471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c145040 0x7f474c06e9d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097c0 tx=0x7f473c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1180655c42fbf5d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749400640 1 -- 172.21.15.67:0/2816968471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c0b5070 con 0x7f474c145040 2024-04-17T10:26:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749400640 1 -- 172.21.15.67:0/2816968471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c0a8990 con 0x7f474c145040 2024-04-17T10:26:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.947+0000 7f4749400640 1 -- 172.21.15.67:0/2816968471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c0a8d60 con 0x7f474c145040 2024-04-17T10:26:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- 172.21.15.67:0/2816968471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c145040 msgr2=0x7f474c06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 --2- 172.21.15.67:0/2816968471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c145040 0x7f474c06e9d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f473c0097c0 tx=0x7f473c01ec90 comp rx=0 tx=0).stop 2024-04-17T10:26:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- 172.21.15.67:0/2816968471 shutdown_connections 2024-04-17T10:26:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 --2- 172.21.15.67:0/2816968471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c145040 0x7f474c06e9d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- 172.21.15.67:0/2816968471 >> 172.21.15.67:0/2816968471 conn(0x7f474c06ac90 msgr2=0x7f474c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- 172.21.15.67:0/2816968471 shutdown_connections 2024-04-17T10:26:09.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- 172.21.15.67:0/2816968471 wait complete. 2024-04-17T10:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 Processor -- start 2024-04-17T10:26:09.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- start start 2024-04-17T10:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f474c06ef10 0x7f474c13c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c071a20 0x7f474c1f3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f474c145040 0x7f474c1f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c06df80 con 0x7f474c071a20 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c15bf10 con 0x7f474c06ef10 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4750c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474c06d370 con 0x7f474c145040 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c071a20 0x7f474c1f3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c071a20 0x7f474c1f3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35868/0 (socket says 172.21.15.67:35868) 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 -- 172.21.15.67:0/3885066971 learned_addr learned my addr 172.21.15.67:0/3885066971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4749e00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f474c06ef10 0x7f474c13c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f474a800640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f474c145040 0x7f474c1f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 -- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f474c145040 msgr2=0x7f474c1f46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f474c145040 0x7f474c1f46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 -- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f474c06ef10 msgr2=0x7f474c13c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f474c06ef10 0x7f474c13c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4741400640 1 -- 172.21.15.67:0/3885066971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c1f3a40 con 0x7f474c071a20 2024-04-17T10:26:09.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.951+0000 7f4749e00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f474c06ef10 0x7f474c13c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4741400640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c071a20 0x7f474c1f3500 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f473400c500 tx=0x7f473400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4742a00640 1 -- 172.21.15.67:0/3885066971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734016020 con 0x7f474c071a20 2024-04-17T10:26:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4742a00640 1 -- 172.21.15.67:0/3885066971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4734005220 con 0x7f474c071a20 2024-04-17T10:26:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c1f3cc0 con 0x7f474c071a20 2024-04-17T10:26:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4742a00640 1 -- 172.21.15.67:0/3885066971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734007030 con 0x7f474c071a20 2024-04-17T10:26:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c1f4280 con 0x7f474c071a20 2024-04-17T10:26:09.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f474c072d50 con 0x7f474c071a20 2024-04-17T10:26:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4742a00640 1 -- 172.21.15.67:0/3885066971 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f47340071f0 con 0x7f474c071a20 2024-04-17T10:26:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4742a00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4728066f60 0x7f4728069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4742a00640 1 -- 172.21.15.67:0/3885066971 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4734084920 con 0x7f474c071a20 2024-04-17T10:26:09.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.955+0000 7f4749e00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4728066f60 0x7f4728069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.959+0000 7f4749e00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4728066f60 0x7f4728069420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f473c0b2000 tx=0x7f473c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:09.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:09.959+0000 7f4742a00640 1 -- 172.21.15.67:0/3885066971 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4734020020 con 0x7f474c071a20 2024-04-17T10:26:10.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.183+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f474c074e30 con 0x7f474c071a20 2024-04-17T10:26:10.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4742a00640 1 -- 172.21.15.67:0/3885066971 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f474c1f3a40 con 0x7f474c071a20 2024-04-17T10:26:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4728066f60 msgr2=0x7f4728069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4728066f60 0x7f4728069420 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f473c0b2000 tx=0x7f473c008040 comp rx=0 tx=0).stop 2024-04-17T10:26:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c071a20 msgr2=0x7f474c1f3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c071a20 0x7f474c1f3500 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f473400c500 tx=0x7f473400c9c0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 shutdown_connections 2024-04-17T10:26:10.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4728066f60 0x7f4728069420 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f474c145040 0x7f474c1f46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f474c071a20 0x7f474c1f3500 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 --2- 172.21.15.67:0/3885066971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f474c06ef10 0x7f474c13c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 >> 172.21.15.67:0/3885066971 conn(0x7f474c06ac90 msgr2=0x7f474c06b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:10.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 shutdown_connections 2024-04-17T10:26:10.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.187+0000 7f4750c00640 1 -- 172.21.15.67:0/3885066971 wait complete. 2024-04-17T10:26:10.203 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-17T10:26:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde99c00640 1 Processor -- start 2024-04-17T10:26:10.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde99c00640 1 -- start start 2024-04-17T10:26:10.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde99c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde9406e6e0 0x7fde940ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde99c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9415ae40 con 0x7fde941441f0 2024-04-17T10:26:10.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde99c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde94071440 con 0x7fde9406eac0 2024-04-17T10:26:10.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde99c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde94071720 con 0x7fde9406e6e0 2024-04-17T10:26:10.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde93400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fde941441f0 0x7fde940b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45316/0 (socket says 172.21.15.67:45316) 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde93400640 1 -- 172.21.15.67:0/1513804732 learned_addr learned my addr 172.21.15.67:0/1513804732 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde92a00640 1 --2- 172.21.15.67:0/1513804732 >> v2:172.21.15.86:3300/0 conn(0x7fde9406e6e0 0x7fde940ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde91600640 1 -- 172.21.15.67:0/1513804732 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 256742120 0 0) 0x7fde9415ae40 con 0x7fde941441f0 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde91600640 1 -- 172.21.15.67:0/1513804732 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78003530 con 0x7fde941441f0 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde92a00640 1 -- 172.21.15.67:0/1513804732 >> v1:172.21.15.86:6789/0 conn(0x7fde9406eac0 legacy=0x7fde94143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde92a00640 1 -- 172.21.15.67:0/1513804732 >> v1:172.21.15.67:6789/0 conn(0x7fde941441f0 legacy=0x7fde940b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde91600640 1 -- 172.21.15.67:0/1513804732 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522647982 0 0) 0x7fde94071440 con 0x7fde9406eac0 2024-04-17T10:26:10.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.351+0000 7fde92a00640 1 -- 172.21.15.67:0/1513804732 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde94071a00 con 0x7fde9406e6e0 2024-04-17T10:26:10.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde92a00640 1 --2- 172.21.15.67:0/1513804732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 0x7fde940ce9b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fde840097c0 tx=0x7fde8401c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a68e14fad9e2267 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:10.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde91600640 1 -- 172.21.15.67:0/1513804732 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde840b0070 con 0x7fde9406e6e0 2024-04-17T10:26:10.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde91600640 1 -- 172.21.15.67:0/1513804732 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde840ab600 con 0x7fde9406e6e0 2024-04-17T10:26:10.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde91600640 1 -- 172.21.15.67:0/1513804732 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde840ab9b0 con 0x7fde9406e6e0 2024-04-17T10:26:10.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde99c00640 1 -- 172.21.15.67:0/1513804732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 msgr2=0x7fde940ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde99c00640 1 --2- 172.21.15.67:0/1513804732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 0x7fde940ce9b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fde840097c0 tx=0x7fde8401c300 comp rx=0 tx=0).stop 2024-04-17T10:26:10.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde99c00640 1 -- 172.21.15.67:0/1513804732 shutdown_connections 2024-04-17T10:26:10.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde99c00640 1 --2- 172.21.15.67:0/1513804732 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 0x7fde940ce9b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde99c00640 1 -- 172.21.15.67:0/1513804732 >> 172.21.15.67:0/1513804732 conn(0x7fde9406ac90 msgr2=0x7fde9406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:10.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde99c00640 1 -- 172.21.15.67:0/1513804732 shutdown_connections 2024-04-17T10:26:10.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.355+0000 7fde99c00640 1 -- 172.21.15.67:0/1513804732 wait complete. 2024-04-17T10:26:10.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 Processor -- start 2024-04-17T10:26:10.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 -- start start 2024-04-17T10:26:10.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 0x7fde94159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9406eac0 0x7fde940cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde941441f0 0x7fde94141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde941688b0 con 0x7fde941441f0 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde94168730 con 0x7fde9406e6e0 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde94168a30 con 0x7fde9406eac0 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde941441f0 0x7fde94141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde941441f0 0x7fde94141bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59496/0 (socket says 172.21.15.67:59496) 2024-04-17T10:26:10.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 -- 172.21.15.67:0/3666125469 learned_addr learned my addr 172.21.15.67:0/3666125469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde92a00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 0x7fde94159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde92000640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9406eac0 0x7fde940cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 -- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9406eac0 msgr2=0x7fde940cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9406eac0 0x7fde940cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 -- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 msgr2=0x7fde94159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 0x7fde94159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 -- 172.21.15.67:0/3666125469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940c5f80 con 0x7fde941441f0 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde93400640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde941441f0 0x7fde94141bc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fde8c001fb0 tx=0x7fde8c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:10.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde8aa00640 1 -- 172.21.15.67:0/3666125469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c012400 con 0x7fde941441f0 2024-04-17T10:26:10.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde8aa00640 1 -- 172.21.15.67:0/3666125469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c0125a0 con 0x7fde941441f0 2024-04-17T10:26:10.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde8aa00640 1 -- 172.21.15.67:0/3666125469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c0129b0 con 0x7fde941441f0 2024-04-17T10:26:10.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde940c61f0 con 0x7fde941441f0 2024-04-17T10:26:10.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.359+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9415a040 con 0x7fde941441f0 2024-04-17T10:26:10.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.363+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde940c63c0 con 0x7fde941441f0 2024-04-17T10:26:10.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.363+0000 7fde8aa00640 1 -- 172.21.15.67:0/3666125469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fde8c012b50 con 0x7fde941441f0 2024-04-17T10:26:10.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.363+0000 7fde8aa00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde74066f60 0x7fde74069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.363+0000 7fde8aa00640 1 -- 172.21.15.67:0/3666125469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fde8c085660 con 0x7fde941441f0 2024-04-17T10:26:10.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.363+0000 7fde92a00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde74066f60 0x7fde74069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.363+0000 7fde92a00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde74066f60 0x7fde74069420 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fde840ad000 tx=0x7fde840a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:10.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.367+0000 7fde8aa00640 1 -- 172.21.15.67:0/3666125469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fde8c056140 con 0x7fde941441f0 2024-04-17T10:26:10.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.583+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fde940b3570 con 0x7fde941441f0 2024-04-17T10:26:10.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde8aa00640 1 -- 172.21.15.67:0/3666125469 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fde8c05a1e0 con 0x7fde941441f0 2024-04-17T10:26:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde74066f60 msgr2=0x7fde74069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde74066f60 0x7fde74069420 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fde840ad000 tx=0x7fde840a6000 comp rx=0 tx=0).stop 2024-04-17T10:26:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde941441f0 msgr2=0x7fde94141bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde941441f0 0x7fde94141bc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fde8c001fb0 tx=0x7fde8c00a430 comp rx=0 tx=0).stop 2024-04-17T10:26:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 shutdown_connections 2024-04-17T10:26:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde74066f60 0x7fde74069420 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde941441f0 0x7fde94141bc0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde9406eac0 0x7fde940cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 --2- 172.21.15.67:0/3666125469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde9406e6e0 0x7fde94159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.587+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 >> 172.21.15.67:0/3666125469 conn(0x7fde9406ac90 msgr2=0x7fde94142ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.591+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 shutdown_connections 2024-04-17T10:26:10.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.591+0000 7fde99c00640 1 -- 172.21.15.67:0/3666125469 wait complete. 2024-04-17T10:26:10.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-17T10:26:10.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.747+0000 7f75b5c00640 1 Processor -- start 2024-04-17T10:26:10.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.747+0000 7f75b5c00640 1 -- start start 2024-04-17T10:26:10.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75b5c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75b00714d0 0x7f75b0168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75b5c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b00b5980 con 0x7f75b015dd10 2024-04-17T10:26:10.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75b5c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b00b0cc0 con 0x7f75b00b07b0 2024-04-17T10:26:10.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75b5c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75b00b1280 con 0x7f75b00714d0 2024-04-17T10:26:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75aea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75b015dd10 0x7f75b00b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45324/0 (socket says 172.21.15.67:45324) 2024-04-17T10:26:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75aea00640 1 -- 172.21.15.67:0/1978616271 learned_addr learned my addr 172.21.15.67:0/1978616271 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:10.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75af400640 1 --2- 172.21.15.67:0/1978616271 >> v2:172.21.15.67:3300/0 conn(0x7f75b00714d0 0x7f75b0168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75ae000640 1 -- 172.21.15.67:0/1978616271 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 529274582 0 0) 0x7f75b00b5980 con 0x7f75b015dd10 2024-04-17T10:26:10.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75ae000640 1 -- 172.21.15.67:0/1978616271 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7594003530 con 0x7f75b015dd10 2024-04-17T10:26:10.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75af400640 1 -- 172.21.15.67:0/1978616271 >> v1:172.21.15.86:6789/0 conn(0x7f75b00b07b0 legacy=0x7f75b0070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75af400640 1 -- 172.21.15.67:0/1978616271 >> v1:172.21.15.67:6789/0 conn(0x7f75b015dd10 legacy=0x7f75b00b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.751+0000 7f75af400640 1 -- 172.21.15.67:0/1978616271 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b00b0fa0 con 0x7f75b00714d0 2024-04-17T10:26:10.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75af400640 1 --2- 172.21.15.67:0/1978616271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168c50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009d90 tx=0x7f75a801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c7d3faf740766a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75ae000640 1 -- 172.21.15.67:0/1978616271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a80b2070 con 0x7f75b00714d0 2024-04-17T10:26:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75ae000640 1 -- 172.21.15.67:0/1978616271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a80a5970 con 0x7f75b00714d0 2024-04-17T10:26:10.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75ae000640 1 -- 172.21.15.67:0/1978616271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a80a5d40 con 0x7f75b00714d0 2024-04-17T10:26:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 -- 172.21.15.67:0/1978616271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 msgr2=0x7f75b0168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 --2- 172.21.15.67:0/1978616271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168c50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f75a8009d90 tx=0x7f75a801c8b0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 -- 172.21.15.67:0/1978616271 shutdown_connections 2024-04-17T10:26:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 --2- 172.21.15.67:0/1978616271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168c50 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 -- 172.21.15.67:0/1978616271 >> 172.21.15.67:0/1978616271 conn(0x7f75b00b34d0 msgr2=0x7f75b00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 -- 172.21.15.67:0/1978616271 shutdown_connections 2024-04-17T10:26:10.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 -- 172.21.15.67:0/1978616271 wait complete. 2024-04-17T10:26:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 Processor -- start 2024-04-17T10:26:10.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.755+0000 7f75b5c00640 1 -- start start 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b00b07b0 0x7f75b00ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75b015dd10 0x7f75b00cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b016bb30 con 0x7f75b00714d0 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b016b9b0 con 0x7f75b00b07b0 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75b016bcb0 con 0x7f75b015dd10 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59524/0 (socket says 172.21.15.67:59524) 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 -- 172.21.15.67:0/3088252297 learned_addr learned my addr 172.21.15.67:0/3088252297 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:10.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75a6000640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b00b07b0 0x7f75b00ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75af400640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75b015dd10 0x7f75b00cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 -- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75b015dd10 msgr2=0x7f75b00cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75b015dd10 0x7f75b00cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 -- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b00b07b0 msgr2=0x7f75b00ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b00b07b0 0x7f75b00ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 -- 172.21.15.67:0/3088252297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b00d1400 con 0x7f75b00714d0 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75a6000640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b00b07b0 0x7f75b00ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:10.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75aea00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168830 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f75a000c420 tx=0x7f75a000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75a7400640 1 -- 172.21.15.67:0/3088252297 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a0017070 con 0x7f75b00714d0 2024-04-17T10:26:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75a7400640 1 -- 172.21.15.67:0/3088252297 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a0005cf0 con 0x7f75b00714d0 2024-04-17T10:26:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b00cd660 con 0x7f75b00714d0 2024-04-17T10:26:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75a7400640 1 -- 172.21.15.67:0/3088252297 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a00072d0 con 0x7f75b00714d0 2024-04-17T10:26:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b014ce80 con 0x7f75b00714d0 2024-04-17T10:26:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b014d1b0 con 0x7f75b00714d0 2024-04-17T10:26:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.759+0000 7f75a7400640 1 -- 172.21.15.67:0/3088252297 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f75a001a070 con 0x7f75b00714d0 2024-04-17T10:26:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.763+0000 7f75a7400640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f758c066f90 0x7f758c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.763+0000 7f75a7400640 1 -- 172.21.15.67:0/3088252297 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f75a0084110 con 0x7f75b00714d0 2024-04-17T10:26:10.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.763+0000 7f75a6000640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f758c066f90 0x7f758c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:10.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.763+0000 7f75a6000640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f758c066f90 0x7f758c069450 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f75b01574f0 tx=0x7f7598009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:10.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.767+0000 7f75a7400640 1 -- 172.21.15.67:0/3088252297 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f75a0054c80 con 0x7f75b00714d0 2024-04-17T10:26:10.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.983+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f75b0070bb0 con 0x7f75b00714d0 2024-04-17T10:26:10.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75a7400640 1 -- 172.21.15.67:0/3088252297 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f75a0058d20 con 0x7f75b00714d0 2024-04-17T10:26:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f758c066f90 msgr2=0x7f758c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f758c066f90 0x7f758c069450 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f75b01574f0 tx=0x7f7598009040 comp rx=0 tx=0).stop 2024-04-17T10:26:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 msgr2=0x7f75b0168830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168830 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f75a000c420 tx=0x7f75a000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 shutdown_connections 2024-04-17T10:26:10.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f758c066f90 0x7f758c069450 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75b015dd10 0x7f75b00cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75b00b07b0 0x7f75b00ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 --2- 172.21.15.67:0/3088252297 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75b00714d0 0x7f75b0168830 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:10.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 >> 172.21.15.67:0/3088252297 conn(0x7f75b00b34d0 msgr2=0x7f75b015b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:10.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 shutdown_connections 2024-04-17T10:26:10.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:10.987+0000 7f75b5c00640 1 -- 172.21.15.67:0/3088252297 wait complete. 2024-04-17T10:26:11.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-17T10:26:11.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19d1600640 1 Processor -- start 2024-04-17T10:26:11.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19d1600640 1 -- start start 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19d1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19cc0ce1b0 0x7f19cc0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19d1600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19cc15cc00 0x7f19cc15f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19d1600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19cc0cead0 con 0x7f19cc15fa00 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19d1600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc15be60 con 0x7f19cc0ce1b0 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19d1600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19cc160230 con 0x7f19cc15cc00 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19caa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19cc0ce1b0 0x7f19cc0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19caa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f19cc0ce1b0 0x7f19cc0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59530/0 (socket says 172.21.15.67:59530) 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19caa00640 1 -- 172.21.15.67:0/3103701962 learned_addr learned my addr 172.21.15.67:0/3103701962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19caa00640 1 -- 172.21.15.67:0/3103701962 >> v2:172.21.15.86:3300/0 conn(0x7f19cc15cc00 msgr2=0x7f19cc15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19c3e00640 1 --2- 172.21.15.67:0/3103701962 >> v2:172.21.15.86:3300/0 conn(0x7f19cc15cc00 0x7f19cc15f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19caa00640 1 --2- 172.21.15.67:0/3103701962 >> v2:172.21.15.86:3300/0 conn(0x7f19cc15cc00 0x7f19cc15f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19caa00640 1 -- 172.21.15.67:0/3103701962 >> v1:172.21.15.86:6789/0 conn(0x7f19cc15fa00 legacy=0x7f19cc070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19caa00640 1 -- 172.21.15.67:0/3103701962 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc15c120 con 0x7f19cc0ce1b0 2024-04-17T10:26:11.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.151+0000 7f19c3e00640 1 --2- 172.21.15.67:0/3103701962 >> v2:172.21.15.86:3300/0 conn(0x7f19cc15cc00 0x7f19cc15f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:26:11.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19caa00640 1 --2- 172.21.15.67:0/3103701962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0ce1b0 0x7f19cc0ce590 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f19cc072de0 tx=0x7f19bc01ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcae591481feb047 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19ca000640 1 -- 172.21.15.67:0/3103701962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0b5070 con 0x7f19cc0ce1b0 2024-04-17T10:26:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19ca000640 1 -- 172.21.15.67:0/3103701962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19bc0a89d0 con 0x7f19cc0ce1b0 2024-04-17T10:26:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19ca000640 1 -- 172.21.15.67:0/3103701962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19bc0a8da0 con 0x7f19cc0ce1b0 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 -- 172.21.15.67:0/3103701962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0ce1b0 msgr2=0x7f19cc0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 --2- 172.21.15.67:0/3103701962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0ce1b0 0x7f19cc0ce590 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f19cc072de0 tx=0x7f19bc01ecd0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 -- 172.21.15.67:0/3103701962 shutdown_connections 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 --2- 172.21.15.67:0/3103701962 >> v2:172.21.15.86:3300/0 conn(0x7f19cc15cc00 0x7f19cc15f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 --2- 172.21.15.67:0/3103701962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc0ce1b0 0x7f19cc0ce590 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 -- 172.21.15.67:0/3103701962 >> 172.21.15.67:0/3103701962 conn(0x7f19cc0b33f0 msgr2=0x7f19cc0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 -- 172.21.15.67:0/3103701962 shutdown_connections 2024-04-17T10:26:11.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.155+0000 7f19d1600640 1 -- 172.21.15.67:0/3103701962 wait complete. 2024-04-17T10:26:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 Processor -- start 2024-04-17T10:26:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 -- start start 2024-04-17T10:26:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19cc0ce1b0 0x7f19cc0b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19cc15cc00 0x7f19cc144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc15fa00 0x7f19cc1e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc159730 con 0x7f19cc15fa00 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc15c6e0 con 0x7f19cc15cc00 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19cc1598b0 con 0x7f19cc0ce1b0 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc15fa00 0x7f19cc1e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc15fa00 0x7f19cc1e6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59540/0 (socket says 172.21.15.67:59540) 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 -- 172.21.15.67:0/115880053 learned_addr learned my addr 172.21.15.67:0/115880053 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19caa00640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19cc0ce1b0 0x7f19cc0b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19c3e00640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19cc15cc00 0x7f19cc144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 -- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19cc0ce1b0 msgr2=0x7f19cc0b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19cc0ce1b0 0x7f19cc0b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 -- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19cc15cc00 msgr2=0x7f19cc144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19cc15cc00 0x7f19cc144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 -- 172.21.15.67:0/115880053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19cc0b5fd0 con 0x7f19cc15fa00 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19cb400640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc15fa00 0x7f19cc1e6d90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f19c400b9a0 tx=0x7f19c400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19c2a00640 1 -- 172.21.15.67:0/115880053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4019070 con 0x7f19cc15fa00 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19c2a00640 1 -- 172.21.15.67:0/115880053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c400cb40 con 0x7f19cc15fa00 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19cc0b62b0 con 0x7f19cc15fa00 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19c2a00640 1 -- 172.21.15.67:0/115880053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4014440 con 0x7f19cc15fa00 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19cc0b6870 con 0x7f19cc15fa00 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.159+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19cc070ab0 con 0x7f19cc15fa00 2024-04-17T10:26:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.163+0000 7f19c2a00640 1 -- 172.21.15.67:0/115880053 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f19c401c070 con 0x7f19cc15fa00 2024-04-17T10:26:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.163+0000 7f19c2a00640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19a8066f60 0x7f19a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.163+0000 7f19c2a00640 1 -- 172.21.15.67:0/115880053 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f19c4084ab0 con 0x7f19cc15fa00 2024-04-17T10:26:11.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.163+0000 7f19caa00640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19a8066f60 0x7f19a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.163+0000 7f19caa00640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19a8066f60 0x7f19a8069420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0b2000 tx=0x7f19bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.167+0000 7f19c2a00640 1 -- 172.21.15.67:0/115880053 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f19c4055500 con 0x7f19cc15fa00 2024-04-17T10:26:11.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.387+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f19cc0cf040 con 0x7f19cc15fa00 2024-04-17T10:26:11.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.387+0000 7f19c2a00640 1 -- 172.21.15.67:0/115880053 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f19c40595a0 con 0x7f19cc15fa00 2024-04-17T10:26:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19a8066f60 msgr2=0x7f19a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19a8066f60 0x7f19a8069420 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f19bc0b2000 tx=0x7f19bc009040 comp rx=0 tx=0).stop 2024-04-17T10:26:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc15fa00 msgr2=0x7f19cc1e6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc15fa00 0x7f19cc1e6d90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f19c400b9a0 tx=0x7f19c400be60 comp rx=0 tx=0).stop 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 shutdown_connections 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19a8066f60 0x7f19a8069420 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19cc15fa00 0x7f19cc1e6d90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19cc15cc00 0x7f19cc144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 --2- 172.21.15.67:0/115880053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f19cc0ce1b0 0x7f19cc0b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 >> 172.21.15.67:0/115880053 conn(0x7f19cc0b33f0 msgr2=0x7f19cc15e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 shutdown_connections 2024-04-17T10:26:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.391+0000 7f19d1600640 1 -- 172.21.15.67:0/115880053 wait complete. 2024-04-17T10:26:11.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-17T10:26:11.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.551+0000 7ff662000640 1 Processor -- start 2024-04-17T10:26:11.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.551+0000 7ff662000640 1 -- start start 2024-04-17T10:26:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.551+0000 7ff662000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff65c0ce1b0 0x7ff65c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.551+0000 7ff662000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.551+0000 7ff662000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c0cead0 con 0x7ff65c15cc00 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.551+0000 7ff662000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff65c15be60 con 0x7ff65c0ce1b0 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.551+0000 7ff662000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c160230 con 0x7ff65c15fbd0 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff65c0ce1b0 0x7ff65c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff65c0ce1b0 0x7ff65c0ce590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59542/0 (socket says 172.21.15.67:59542) 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff653e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff65c15cc00 0x7ff65c15f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45328/0 (socket says 172.21.15.67:45328) 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 -- 172.21.15.67:0/480204820 learned_addr learned my addr 172.21.15.67:0/480204820 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65be00640 1 --2- 172.21.15.67:0/480204820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 -- 172.21.15.67:0/480204820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 msgr2=0x7ff65c0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 --2- 172.21.15.67:0/480204820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 -- 172.21.15.67:0/480204820 >> v1:172.21.15.67:6789/0 conn(0x7ff65c15cc00 legacy=0x7ff65c15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 -- 172.21.15.67:0/480204820 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c15c120 con 0x7ff65c0ce1b0 2024-04-17T10:26:11.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65b400640 1 --2- 172.21.15.67:0/480204820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c0ce1b0 0x7ff65c0ce590 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009870 tx=0x7ff64c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be50d1985ea06ffd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65aa00640 1 -- 172.21.15.67:0/480204820 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0b5070 con 0x7ff65c0ce1b0 2024-04-17T10:26:11.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65aa00640 1 -- 172.21.15.67:0/480204820 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c0a8a50 con 0x7ff65c0ce1b0 2024-04-17T10:26:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.555+0000 7ff65aa00640 1 -- 172.21.15.67:0/480204820 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0a8e20 con 0x7ff65c0ce1b0 2024-04-17T10:26:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- 172.21.15.67:0/480204820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c0ce1b0 msgr2=0x7ff65c0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 --2- 172.21.15.67:0/480204820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c0ce1b0 0x7ff65c0ce590 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009870 tx=0x7ff64c01ed50 comp rx=0 tx=0).stop 2024-04-17T10:26:11.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- 172.21.15.67:0/480204820 shutdown_connections 2024-04-17T10:26:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 --2- 172.21.15.67:0/480204820 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 --2- 172.21.15.67:0/480204820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c0ce1b0 0x7ff65c0ce590 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- 172.21.15.67:0/480204820 >> 172.21.15.67:0/480204820 conn(0x7ff65c0b33f0 msgr2=0x7ff65c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- 172.21.15.67:0/480204820 shutdown_connections 2024-04-17T10:26:11.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- 172.21.15.67:0/480204820 wait complete. 2024-04-17T10:26:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 Processor -- start 2024-04-17T10:26:11.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- start start 2024-04-17T10:26:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c0ce1b0 0x7ff65c14d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c15cc00 0x7ff65c0ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c159730 con 0x7ff65c15cc00 2024-04-17T10:26:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c15c6e0 con 0x7ff65c0ce1b0 2024-04-17T10:26:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff662000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c1598b0 con 0x7ff65c15fbd0 2024-04-17T10:26:11.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c15cc00 0x7ff65c0ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c15cc00 0x7ff65c0ca730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59550/0 (socket says 172.21.15.67:59550) 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 -- 172.21.15.67:0/2533289218 learned_addr learned my addr 172.21.15.67:0/2533289218 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff65b400640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c0ce1b0 0x7ff65c14d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff65be00640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 -- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 msgr2=0x7ff65c0c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 -- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c0ce1b0 msgr2=0x7ff65c14d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c0ce1b0 0x7ff65c14d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff653e00640 1 -- 172.21.15.67:0/2533289218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c0cafb0 con 0x7ff65c15cc00 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.559+0000 7ff65be00640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff653e00640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c15cc00 0x7ff65c0ca730 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c420 tx=0x7ff64400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff653400640 1 -- 172.21.15.67:0/2533289218 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6440074e0 con 0x7ff65c15cc00 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff653400640 1 -- 172.21.15.67:0/2533289218 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff644007680 con 0x7ff65c15cc00 2024-04-17T10:26:11.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff653400640 1 -- 172.21.15.67:0/2533289218 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff644004a30 con 0x7ff65c15cc00 2024-04-17T10:26:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c0cb290 con 0x7ff65c15cc00 2024-04-17T10:26:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c0cc1a0 con 0x7ff65c15cc00 2024-04-17T10:26:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65c0cc3a0 con 0x7ff65c15cc00 2024-04-17T10:26:11.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff653400640 1 -- 172.21.15.67:0/2533289218 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff64401a020 con 0x7ff65c15cc00 2024-04-17T10:26:11.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff653400640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff638066f90 0x7ff638069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff653400640 1 -- 172.21.15.67:0/2533289218 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff6440845d0 con 0x7ff65c15cc00 2024-04-17T10:26:11.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.563+0000 7ff65b400640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff638066f90 0x7ff638069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.567+0000 7ff65b400640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff638066f90 0x7ff638069450 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0b2000 tx=0x7ff64c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.567+0000 7ff653400640 1 -- 172.21.15.67:0/2533289218 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff65c0cc3a0 con 0x7ff65c15cc00 2024-04-17T10:26:11.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.787+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7ff65c070ab0 con 0x7ff65c15cc00 2024-04-17T10:26:11.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.787+0000 7ff653400640 1 -- 172.21.15.67:0/2533289218 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff6440557a0 con 0x7ff65c15cc00 2024-04-17T10:26:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff638066f90 msgr2=0x7ff638069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff638066f90 0x7ff638069450 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0b2000 tx=0x7ff64c009040 comp rx=0 tx=0).stop 2024-04-17T10:26:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c15cc00 msgr2=0x7ff65c0ca730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c15cc00 0x7ff65c0ca730 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff64400c420 tx=0x7ff64400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 shutdown_connections 2024-04-17T10:26:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff638066f90 0x7ff638069450 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff65c15fbd0 0x7ff65c0c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff65c15cc00 0x7ff65c0ca730 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 --2- 172.21.15.67:0/2533289218 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff65c0ce1b0 0x7ff65c14d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 >> 172.21.15.67:0/2533289218 conn(0x7ff65c0b33f0 msgr2=0x7ff65c0cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 shutdown_connections 2024-04-17T10:26:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.791+0000 7ff662000640 1 -- 172.21.15.67:0/2533289218 wait complete. 2024-04-17T10:26:11.806 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:26:11.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.951+0000 7f4998200640 1 Processor -- start 2024-04-17T10:26:11.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.951+0000 7f4998200640 1 -- start start 2024-04-17T10:26:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4998200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f499006e6e0 0x7f49900ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4998200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499015ae40 con 0x7f49901441f0 2024-04-17T10:26:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4998200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4990071440 con 0x7f499006eac0 2024-04-17T10:26:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4998200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4990071720 con 0x7f499006e6e0 2024-04-17T10:26:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4996200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49901441f0 0x7f49900b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45336/0 (socket says 172.21.15.67:45336) 2024-04-17T10:26:11.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4996200640 1 -- 172.21.15.67:0/1692216830 learned_addr learned my addr 172.21.15.67:0/1692216830 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4995800640 1 --2- 172.21.15.67:0/1692216830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f499006e6e0 0x7f49900ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f498fe00640 1 -- 172.21.15.67:0/1692216830 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1086212501 0 0) 0x7f499015ae40 con 0x7f49901441f0 2024-04-17T10:26:11.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f498fe00640 1 -- 172.21.15.67:0/1692216830 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4978003530 con 0x7f49901441f0 2024-04-17T10:26:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f498fe00640 1 -- 172.21.15.67:0/1692216830 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784675063 0 0) 0x7f4990071440 con 0x7f499006eac0 2024-04-17T10:26:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4995800640 1 -- 172.21.15.67:0/1692216830 >> v1:172.21.15.86:6789/0 conn(0x7f499006eac0 legacy=0x7f4990143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4995800640 1 -- 172.21.15.67:0/1692216830 >> v1:172.21.15.67:6789/0 conn(0x7f49901441f0 legacy=0x7f49900b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.955+0000 7f4995800640 1 -- 172.21.15.67:0/1692216830 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990071a00 con 0x7f499006e6e0 2024-04-17T10:26:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4995800640 1 --2- 172.21.15.67:0/1692216830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f499006e6e0 0x7f49900ce9b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f49840097f0 tx=0x7f498401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98df7b1f8c9d1df5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f498fe00640 1 -- 172.21.15.67:0/1692216830 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840b0070 con 0x7f499006e6e0 2024-04-17T10:26:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f498fe00640 1 -- 172.21.15.67:0/1692216830 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49840ab600 con 0x7f499006e6e0 2024-04-17T10:26:11.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f498fe00640 1 -- 172.21.15.67:0/1692216830 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840ab9d0 con 0x7f499006e6e0 2024-04-17T10:26:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- 172.21.15.67:0/1692216830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f499006e6e0 msgr2=0x7f49900ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 --2- 172.21.15.67:0/1692216830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f499006e6e0 0x7f49900ce9b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f49840097f0 tx=0x7f498401c420 comp rx=0 tx=0).stop 2024-04-17T10:26:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- 172.21.15.67:0/1692216830 shutdown_connections 2024-04-17T10:26:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 --2- 172.21.15.67:0/1692216830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f499006e6e0 0x7f49900ce9b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- 172.21.15.67:0/1692216830 >> 172.21.15.67:0/1692216830 conn(0x7f499006ac90 msgr2=0x7f499006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- 172.21.15.67:0/1692216830 shutdown_connections 2024-04-17T10:26:11.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- 172.21.15.67:0/1692216830 wait complete. 2024-04-17T10:26:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 Processor -- start 2024-04-17T10:26:11.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- start start 2024-04-17T10:26:11.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499006e6e0 0x7f49900cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f499006eac0 0x7f4990159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49901441f0 0x7f499015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49901688b0 con 0x7f499006e6e0 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4990168730 con 0x7f499006eac0 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4998200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4990168a30 con 0x7f49901441f0 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4995800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499006e6e0 0x7f49900cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4995800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499006e6e0 0x7f49900cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59562/0 (socket says 172.21.15.67:59562) 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.959+0000 7f4995800640 1 -- 172.21.15.67:0/1121793572 learned_addr learned my addr 172.21.15.67:0/1121793572 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4994e00640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f499006eac0 0x7f4990159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4996200640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49901441f0 0x7f499015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4995800640 1 -- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49901441f0 msgr2=0x7f499015a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4995800640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49901441f0 0x7f499015a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4995800640 1 -- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f499006eac0 msgr2=0x7f4990159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:11.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4995800640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f499006eac0 0x7f4990159bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4995800640 1 -- 172.21.15.67:0/1121793572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499015a650 con 0x7f499006e6e0 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4994e00640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f499006eac0 0x7f4990159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4995800640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499006e6e0 0x7f49900cade0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f49840ae040 tx=0x7f498401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f498d600640 1 -- 172.21.15.67:0/1121793572 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49840b0050 con 0x7f499006e6e0 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f498d600640 1 -- 172.21.15.67:0/1121793572 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49840bc070 con 0x7f499006e6e0 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49900cb550 con 0x7f499006e6e0 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f498d600640 1 -- 172.21.15.67:0/1121793572 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4984006e20 con 0x7f499006e6e0 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49900cbb10 con 0x7f499006e6e0 2024-04-17T10:26:11.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4990072d50 con 0x7f499006e6e0 2024-04-17T10:26:11.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f498d600640 1 -- 172.21.15.67:0/1121793572 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f49840b5870 con 0x7f499006e6e0 2024-04-17T10:26:11.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f498d600640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4970066f60 0x7f4970069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:11.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.963+0000 7f498d600640 1 -- 172.21.15.67:0/1121793572 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f498411f620 con 0x7f499006e6e0 2024-04-17T10:26:11.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.967+0000 7f4994e00640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4970066f60 0x7f4970069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:11.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.967+0000 7f4994e00640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4970066f60 0x7f4970069420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f49900b2340 tx=0x7f497c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:11.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:11.967+0000 7f498d600640 1 -- 172.21.15.67:0/1121793572 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f49840160d0 con 0x7f499006e6e0 2024-04-17T10:26:12.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.131+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4990074e30 con 0x7f499006e6e0 2024-04-17T10:26:12.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f498d600640 1 -- 172.21.15.67:0/1121793572 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+13116 (secure 0 0 0) 0x7f49840f07f0 con 0x7f499006e6e0 2024-04-17T10:26:12.128 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:12.129 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":35,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:26:08.990459+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4970066f60 msgr2=0x7f4970069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4970066f60 0x7f4970069420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f49900b2340 tx=0x7f497c009040 comp rx=0 tx=0).stop 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499006e6e0 msgr2=0x7f49900cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499006e6e0 0x7f49900cade0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f49840ae040 tx=0x7f498401ac60 comp rx=0 tx=0).stop 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 shutdown_connections 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4970066f60 0x7f4970069420 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49901441f0 0x7f499015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f499006eac0 0x7f4990159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 --2- 172.21.15.67:0/1121793572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f499006e6e0 0x7f49900cade0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 >> 172.21.15.67:0/1121793572 conn(0x7f499006ac90 msgr2=0x7f49900a9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 shutdown_connections 2024-04-17T10:26:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.135+0000 7f4998200640 1 -- 172.21.15.67:0/1121793572 wait complete. 2024-04-17T10:26:12.150 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:26:12.150 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:26:12.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb637c00640 1 Processor -- start 2024-04-17T10:26:12.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb637c00640 1 -- start start 2024-04-17T10:26:12.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb637c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb630159440 0x7fb6300ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb637c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63015c580 0x7fb630071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb637c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63015cb30 con 0x7fb6300713e0 2024-04-17T10:26:12.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb637c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6300a8650 con 0x7fb630159440 2024-04-17T10:26:12.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb637c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63015ce10 con 0x7fb63015c580 2024-04-17T10:26:12.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb635200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb630159440 0x7fb6300ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb635200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb630159440 0x7fb6300ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59578/0 (socket says 172.21.15.67:59578) 2024-04-17T10:26:12.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb635200640 1 -- 172.21.15.67:0/2861456808 learned_addr learned my addr 172.21.15.67:0/2861456808 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:12.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.299+0000 7fb635c00640 1 --2- 172.21.15.67:0/2861456808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63015c580 0x7fb630071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb635200640 1 -- 172.21.15.67:0/2861456808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63015c580 msgr2=0x7fb630071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:12.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb635200640 1 --2- 172.21.15.67:0/2861456808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63015c580 0x7fb630071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb635200640 1 -- 172.21.15.67:0/2861456808 >> v1:172.21.15.86:6789/0 conn(0x7fb6300713e0 legacy=0x7fb63015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb635200640 1 -- 172.21.15.67:0/2861456808 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300cd610 con 0x7fb630159440 2024-04-17T10:26:12.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb635200640 1 --2- 172.21.15.67:0/2861456808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb630159440 0x7fb6300ce0f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb624009870 tx=0x7fb62401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cb9e4146d74d7bb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb62f400640 1 -- 172.21.15.67:0/2861456808 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6240a7070 con 0x7fb630159440 2024-04-17T10:26:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb62f400640 1 -- 172.21.15.67:0/2861456808 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6240b0070 con 0x7fb630159440 2024-04-17T10:26:12.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb62f400640 1 -- 172.21.15.67:0/2861456808 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6240ab890 con 0x7fb630159440 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb637c00640 1 -- 172.21.15.67:0/2861456808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb630159440 msgr2=0x7fb6300ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb637c00640 1 --2- 172.21.15.67:0/2861456808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb630159440 0x7fb6300ce0f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fb624009870 tx=0x7fb62401c560 comp rx=0 tx=0).stop 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb637c00640 1 -- 172.21.15.67:0/2861456808 shutdown_connections 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb637c00640 1 --2- 172.21.15.67:0/2861456808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb63015c580 0x7fb630071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb637c00640 1 --2- 172.21.15.67:0/2861456808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb630159440 0x7fb6300ce0f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.303+0000 7fb637c00640 1 -- 172.21.15.67:0/2861456808 >> 172.21.15.67:0/2861456808 conn(0x7fb6300b1740 msgr2=0x7fb6300b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 -- 172.21.15.67:0/2861456808 shutdown_connections 2024-04-17T10:26:12.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 -- 172.21.15.67:0/2861456808 wait complete. 2024-04-17T10:26:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 Processor -- start 2024-04-17T10:26:12.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 -- start start 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6300713e0 0x7fb63014b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb630159440 0x7fb63014cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb63015c580 0x7fb63015a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63015d9c0 con 0x7fb63015c580 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63015d840 con 0x7fb630159440 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb63015db40 con 0x7fb6300713e0 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb63015c580 0x7fb63015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb63015c580 0x7fb63015a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59588/0 (socket says 172.21.15.67:59588) 2024-04-17T10:26:12.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 -- 172.21.15.67:0/1766046324 learned_addr learned my addr 172.21.15.67:0/1766046324 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635200640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6300713e0 0x7fb63014b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb62fe00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb630159440 0x7fb63014cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 -- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6300713e0 msgr2=0x7fb63014b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6300713e0 0x7fb63014b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 -- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb630159440 msgr2=0x7fb63014cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb630159440 0x7fb63014cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 -- 172.21.15.67:0/1766046324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63015b2e0 con 0x7fb63015c580 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb635c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb63015c580 0x7fb63015a050 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb628007e90 tx=0x7fb62800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb62cc00640 1 -- 172.21.15.67:0/1766046324 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628014420 con 0x7fb63015c580 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb62cc00640 1 -- 172.21.15.67:0/1766046324 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628019070 con 0x7fb63015c580 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb62cc00640 1 -- 172.21.15.67:0/1766046324 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628014800 con 0x7fb63015c580 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.307+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb630152810 con 0x7fb63015c580 2024-04-17T10:26:12.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.311+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63014d2f0 con 0x7fb63015c580 2024-04-17T10:26:12.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.311+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb63004ef30 con 0x7fb63015c580 2024-04-17T10:26:12.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.311+0000 7fb62cc00640 1 -- 172.21.15.67:0/1766046324 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb6280149a0 con 0x7fb63015c580 2024-04-17T10:26:12.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.311+0000 7fb62cc00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb610066f90 0x7fb610069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.311+0000 7fb62cc00640 1 -- 172.21.15.67:0/1766046324 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb628085680 con 0x7fb63015c580 2024-04-17T10:26:12.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.311+0000 7fb635200640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb610066f90 0x7fb610069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.311+0000 7fb635200640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb610066f90 0x7fb610069450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb6240af000 tx=0x7fb6240a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.315+0000 7fb62cc00640 1 -- 172.21.15.67:0/1766046324 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb6280567b0 con 0x7fb63015c580 2024-04-17T10:26:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.475+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb63014fd80 con 0x7fb63015c580 2024-04-17T10:26:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb62cc00640 1 -- 172.21.15.67:0/1766046324 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6280560b0 con 0x7fb63015c580 2024-04-17T10:26:12.472 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:26:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb610066f90 msgr2=0x7fb610069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb610066f90 0x7fb610069450 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb6240af000 tx=0x7fb6240a7000 comp rx=0 tx=0).stop 2024-04-17T10:26:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb63015c580 msgr2=0x7fb63015a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb63015c580 0x7fb63015a050 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb628007e90 tx=0x7fb62800e4b0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 shutdown_connections 2024-04-17T10:26:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb610066f90 0x7fb610069450 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb63015c580 0x7fb63015a050 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb630159440 0x7fb63014cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 --2- 172.21.15.67:0/1766046324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6300713e0 0x7fb63014b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 >> 172.21.15.67:0/1766046324 conn(0x7fb6300b1740 msgr2=0x7fb6300b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 shutdown_connections 2024-04-17T10:26:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.479+0000 7fb637c00640 1 -- 172.21.15.67:0/1766046324 wait complete. 2024-04-17T10:26:12.493 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:26:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.639+0000 7f3004800640 1 Processor -- start 2024-04-17T10:26:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.639+0000 7f3004800640 1 -- start start 2024-04-17T10:26:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3004800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ffc071310 0x7f2ffc071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3004800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ffc0cc300 0x7f2ffc0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3004800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc06e2f0 con 0x7f2ffc145040 2024-04-17T10:26:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3004800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc0a86c0 con 0x7f2ffc071310 2024-04-17T10:26:12.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3004800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc071cb0 con 0x7f2ffc0cc300 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ffc071310 0x7f2ffc071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2ffc145040 0x7f2ffc06e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:40756/0 (socket says 172.21.15.67:40756) 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001e00640 1 -- 172.21.15.67:0/407779792 learned_addr learned my addr 172.21.15.67:0/407779792 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3002800640 1 --2- 172.21.15.67:0/407779792 >> v2:172.21.15.86:3300/0 conn(0x7f2ffc0cc300 0x7f2ffc0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001400640 1 -- 172.21.15.67:0/407779792 >> v2:172.21.15.86:3300/0 conn(0x7f2ffc0cc300 msgr2=0x7f2ffc0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001400640 1 --2- 172.21.15.67:0/407779792 >> v2:172.21.15.86:3300/0 conn(0x7f2ffc0cc300 0x7f2ffc0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001400640 1 -- 172.21.15.67:0/407779792 >> v1:172.21.15.86:6789/0 conn(0x7f2ffc145040 legacy=0x7f2ffc06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001400640 1 -- 172.21.15.67:0/407779792 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc06d6e0 con 0x7f2ffc071310 2024-04-17T10:26:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3000a00640 1 -- 172.21.15.67:0/407779792 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341895254 0 0) 0x7f2ffc06e2f0 con 0x7f2ffc145040 2024-04-17T10:26:12.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.643+0000 7f3001400640 1 --2- 172.21.15.67:0/407779792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc071310 0x7f2ffc071770 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0af390 tx=0x7f2fec01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a80a455dd75331 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3000a00640 1 -- 172.21.15.67:0/407779792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0b5070 con 0x7f2ffc071310 2024-04-17T10:26:12.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3000a00640 1 -- 172.21.15.67:0/407779792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec0a8970 con 0x7f2ffc071310 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3000a00640 1 -- 172.21.15.67:0/407779792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0a8d40 con 0x7f2ffc071310 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- 172.21.15.67:0/407779792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc071310 msgr2=0x7f2ffc071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 --2- 172.21.15.67:0/407779792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc071310 0x7f2ffc071770 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc0af390 tx=0x7f2fec01ec70 comp rx=0 tx=0).stop 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- 172.21.15.67:0/407779792 shutdown_connections 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 --2- 172.21.15.67:0/407779792 >> v2:172.21.15.86:3300/0 conn(0x7f2ffc0cc300 0x7f2ffc0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 --2- 172.21.15.67:0/407779792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc071310 0x7f2ffc071770 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- 172.21.15.67:0/407779792 >> 172.21.15.67:0/407779792 conn(0x7f2ffc06ac90 msgr2=0x7f2ffc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- 172.21.15.67:0/407779792 shutdown_connections 2024-04-17T10:26:12.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- 172.21.15.67:0/407779792 wait complete. 2024-04-17T10:26:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 Processor -- start 2024-04-17T10:26:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- start start 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc071310 0x7f2ffc142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ffc0cc300 0x7f2ffc13f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc145040 0x7f2ffc15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc06df80 con 0x7f2ffc145040 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc06dca0 con 0x7f2ffc071310 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3004800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc06d370 con 0x7f2ffc0cc300 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3002800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc145040 0x7f2ffc15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3002800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc145040 0x7f2ffc15a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59604/0 (socket says 172.21.15.67:59604) 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3002800640 1 -- 172.21.15.67:0/3307788663 learned_addr learned my addr 172.21.15.67:0/3307788663 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3001e00640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc071310 0x7f2ffc142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.647+0000 7f3001400640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ffc0cc300 0x7f2ffc13f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3002800640 1 -- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ffc0cc300 msgr2=0x7f2ffc13f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3002800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ffc0cc300 0x7f2ffc13f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3002800640 1 -- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc071310 msgr2=0x7f2ffc142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3002800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc071310 0x7f2ffc142520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3002800640 1 -- 172.21.15.67:0/3307788663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc13cba0 con 0x7f2ffc145040 2024-04-17T10:26:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3002800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc145040 0x7f2ffc15a950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800efc0 tx=0x7f2ff800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f2ff6000640 1 -- 172.21.15.67:0/3307788663 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8018020 con 0x7f2ffc145040 2024-04-17T10:26:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f2ff6000640 1 -- 172.21.15.67:0/3307788663 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff8014480 con 0x7f2ffc145040 2024-04-17T10:26:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f2ff6000640 1 -- 172.21.15.67:0/3307788663 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8014870 con 0x7f2ffc145040 2024-04-17T10:26:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc13ce80 con 0x7f2ffc145040 2024-04-17T10:26:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc13d440 con 0x7f2ffc145040 2024-04-17T10:26:12.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc072d50 con 0x7f2ffc145040 2024-04-17T10:26:12.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f2ff6000640 1 -- 172.21.15.67:0/3307788663 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2ff801b020 con 0x7f2ffc145040 2024-04-17T10:26:12.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f2ff6000640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe0066f60 0x7f2fe0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.651+0000 7f2ff6000640 1 -- 172.21.15.67:0/3307788663 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2ff8085870 con 0x7f2ffc145040 2024-04-17T10:26:12.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.655+0000 7f3001e00640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe0066f60 0x7f2fe0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.655+0000 7f3001e00640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe0066f60 0x7f2fe0069420 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0007fd0 tx=0x7f2ff0007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.655+0000 7f2ff6000640 1 -- 172.21.15.67:0/3307788663 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2ff8022070 con 0x7f2ffc145040 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2ffc0ce250 con 0x7f2ffc145040 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f2ff6000640 1 -- 172.21.15.67:0/3307788663 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2ff8010040 con 0x7f2ffc145040 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:26:12.817 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:26:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe0066f60 msgr2=0x7f2fe0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe0066f60 0x7f2fe0069420 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2ff0007fd0 tx=0x7f2ff0007d60 comp rx=0 tx=0).stop 2024-04-17T10:26:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc145040 msgr2=0x7f2ffc15a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc145040 0x7f2ffc15a950 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800efc0 tx=0x7f2ff800c490 comp rx=0 tx=0).stop 2024-04-17T10:26:12.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 shutdown_connections 2024-04-17T10:26:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe0066f60 0x7f2fe0069420 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ffc145040 0x7f2ffc15a950 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ffc0cc300 0x7f2ffc13f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 --2- 172.21.15.67:0/3307788663 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ffc071310 0x7f2ffc142520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 >> 172.21.15.67:0/3307788663 conn(0x7f2ffc06ac90 msgr2=0x7f2ffc0b0520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 shutdown_connections 2024-04-17T10:26:12.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.823+0000 7f3004800640 1 -- 172.21.15.67:0/3307788663 wait complete. 2024-04-17T10:26:12.839 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:26:12.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:26:12.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351c00640 1 Processor -- start 2024-04-17T10:26:12.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351c00640 1 -- start start 2024-04-17T10:26:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c15da60 0x7f434c0ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434c0b57a0 con 0x7f434c070020 2024-04-17T10:26:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434c0b34c0 con 0x7f434c0af440 2024-04-17T10:26:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0b37a0 con 0x7f434c15da60 2024-04-17T10:26:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f434c070020 0x7f434c06ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45344/0 (socket says 172.21.15.67:45344) 2024-04-17T10:26:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f4351200640 1 -- 172.21.15.67:0/336952576 learned_addr learned my addr 172.21.15.67:0/336952576 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:12.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f434be00640 1 --2- 172.21.15.67:0/336952576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c15da60 0x7f434c0ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f434aa00640 1 -- 172.21.15.67:0/336952576 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335484771 0 0) 0x7f434c0b57a0 con 0x7f434c070020 2024-04-17T10:26:12.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f434aa00640 1 -- 172.21.15.67:0/336952576 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4330003530 con 0x7f434c070020 2024-04-17T10:26:12.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f434be00640 1 -- 172.21.15.67:0/336952576 >> v1:172.21.15.86:6789/0 conn(0x7f434c0af440 legacy=0x7f434c06ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f434be00640 1 -- 172.21.15.67:0/336952576 >> v1:172.21.15.67:6789/0 conn(0x7f434c070020 legacy=0x7f434c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.991+0000 7f434be00640 1 -- 172.21.15.67:0/336952576 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c0b3a80 con 0x7f434c15da60 2024-04-17T10:26:12.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f434be00640 1 --2- 172.21.15.67:0/336952576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c15da60 0x7f434c0ad4a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f433c0097f0 tx=0x7f433c01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55147767110596a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f434aa00640 1 -- 172.21.15.67:0/336952576 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c0ab5f0 con 0x7f434c15da60 2024-04-17T10:26:12.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f434aa00640 1 -- 172.21.15.67:0/336952576 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433c0b0070 con 0x7f434c15da60 2024-04-17T10:26:12.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f434aa00640 1 -- 172.21.15.67:0/336952576 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433c0ab9b0 con 0x7f434c15da60 2024-04-17T10:26:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f4351c00640 1 -- 172.21.15.67:0/336952576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c15da60 msgr2=0x7f434c0ad4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/336952576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c15da60 0x7f434c0ad4a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f433c0097f0 tx=0x7f433c01c4e0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f4351c00640 1 -- 172.21.15.67:0/336952576 shutdown_connections 2024-04-17T10:26:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/336952576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c15da60 0x7f434c0ad4a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f4351c00640 1 -- 172.21.15.67:0/336952576 >> 172.21.15.67:0/336952576 conn(0x7f434c099a50 msgr2=0x7f434c0b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f4351c00640 1 -- 172.21.15.67:0/336952576 shutdown_connections 2024-04-17T10:26:12.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.995+0000 7f4351c00640 1 -- 172.21.15.67:0/336952576 wait complete. 2024-04-17T10:26:12.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 Processor -- start 2024-04-17T10:26:12.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 -- start start 2024-04-17T10:26:12.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f434c070020 0x7f434c147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c0af440 0x7f434c09df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f434c15da60 0x7f434c150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c06b520 con 0x7f434c15da60 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434c06b3a0 con 0x7f434c070020 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434c0afa40 con 0x7f434c0af440 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f434c15da60 0x7f434c150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f434c15da60 0x7f434c150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59606/0 (socket says 172.21.15.67:59606) 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 -- 172.21.15.67:0/2045432049 learned_addr learned my addr 172.21.15.67:0/2045432049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f434be00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f434c070020 0x7f434c147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f434b400640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c0af440 0x7f434c09df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 -- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c0af440 msgr2=0x7f434c09df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c0af440 0x7f434c09df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 -- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f434c070020 msgr2=0x7f434c147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f434c070020 0x7f434c147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 -- 172.21.15.67:0/2045432049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434c148340 con 0x7f434c15da60 2024-04-17T10:26:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f434b400640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c0af440 0x7f434c09df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351200640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f434c15da60 0x7f434c150f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f434400cb40 tx=0x7f434400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4343e00640 1 -- 172.21.15.67:0/2045432049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344017070 con 0x7f434c15da60 2024-04-17T10:26:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4343e00640 1 -- 172.21.15.67:0/2045432049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f434400aca0 con 0x7f434c15da60 2024-04-17T10:26:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4343e00640 1 -- 172.21.15.67:0/2045432049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344012620 con 0x7f434c15da60 2024-04-17T10:26:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f434c14fee0 con 0x7f434c15da60 2024-04-17T10:26:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:12.999+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f434c1504a0 con 0x7f434c15da60 2024-04-17T10:26:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.003+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f434c074dc0 con 0x7f434c15da60 2024-04-17T10:26:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.003+0000 7f4343e00640 1 -- 172.21.15.67:0/2045432049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f434401a070 con 0x7f434c15da60 2024-04-17T10:26:12.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.003+0000 7f4343e00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f432c066f60 0x7f432c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:12.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.003+0000 7f4343e00640 1 -- 172.21.15.67:0/2045432049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4344084c30 con 0x7f434c15da60 2024-04-17T10:26:12.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.003+0000 7f434be00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f432c066f60 0x7f432c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:12.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.003+0000 7f434be00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f432c066f60 0x7f432c069420 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f433c0af000 tx=0x7f433c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:13.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.007+0000 7f4343e00640 1 -- 172.21.15.67:0/2045432049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f434c148340 con 0x7f434c15da60 2024-04-17T10:26:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.167+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f434c06d290 con 0x7f434c15da60 2024-04-17T10:26:13.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4343e00640 1 -- 172.21.15.67:0/2045432049 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4344055710 con 0x7f434c15da60 2024-04-17T10:26:13.988 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:26:13.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f432c066f60 msgr2=0x7f432c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f432c066f60 0x7f432c069420 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f433c0af000 tx=0x7f433c0a7000 comp rx=0 tx=0).stop 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f434c15da60 msgr2=0x7f434c150f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f434c15da60 0x7f434c150f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f434400cb40 tx=0x7f434400a060 comp rx=0 tx=0).stop 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 shutdown_connections 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f432c066f60 0x7f432c069420 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f434c15da60 0x7f434c150f80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f434c0af440 0x7f434c09df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 --2- 172.21.15.67:0/2045432049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f434c070020 0x7f434c147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 >> 172.21.15.67:0/2045432049 conn(0x7f434c099a50 msgr2=0x7f434c0ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 shutdown_connections 2024-04-17T10:26:13.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:13.995+0000 7f4351c00640 1 -- 172.21.15.67:0/2045432049 wait complete. 2024-04-17T10:26:14.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:26:14.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f642e200640 1 Processor -- start 2024-04-17T10:26:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f642e200640 1 -- start start 2024-04-17T10:26:14.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f642e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f64280cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f642e200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f64280707c0 0x7f642815d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f642e200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64280d03c0 con 0x7f642806f8b0 2024-04-17T10:26:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f642e200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64280d06a0 con 0x7f64280707c0 2024-04-17T10:26:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f642e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64280d0980 con 0x7f642806d810 2024-04-17T10:26:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f641ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f642806f8b0 0x7f64280700b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45348/0 (socket says 172.21.15.67:45348) 2024-04-17T10:26:14.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f641ea00640 1 -- 172.21.15.67:0/3586250697 learned_addr learned my addr 172.21.15.67:0/3586250697 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6427400640 1 --2- 172.21.15.67:0/3586250697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f64280cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6427e00640 1 --2- 172.21.15.67:0/3586250697 >> v2:172.21.15.86:3300/0 conn(0x7f64280707c0 0x7f642815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6426a00640 1 -- 172.21.15.67:0/3586250697 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588838760 0 0) 0x7f64280d03c0 con 0x7f642806f8b0 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6426a00640 1 -- 172.21.15.67:0/3586250697 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f640c003530 con 0x7f642806f8b0 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6426a00640 1 -- 172.21.15.67:0/3586250697 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1891584141 0 0) 0x7f640c003530 con 0x7f642806f8b0 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6427400640 1 -- 172.21.15.67:0/3586250697 >> v2:172.21.15.86:3300/0 conn(0x7f64280707c0 msgr2=0x7f642815d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6427400640 1 --2- 172.21.15.67:0/3586250697 >> v2:172.21.15.86:3300/0 conn(0x7f64280707c0 0x7f642815d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6427400640 1 -- 172.21.15.67:0/3586250697 >> v1:172.21.15.67:6789/0 conn(0x7f642806f8b0 legacy=0x7f64280700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:14.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6427400640 1 -- 172.21.15.67:0/3586250697 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64280d0c60 con 0x7f642806d810 2024-04-17T10:26:14.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.171+0000 7f6427e00640 1 --2- 172.21.15.67:0/3586250697 >> v2:172.21.15.86:3300/0 conn(0x7f64280707c0 0x7f642815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f6427400640 1 --2- 172.21.15.67:0/3586250697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f64280cfad0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f64180097c0 tx=0x7f641801c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b92a77864320b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f6426a00640 1 -- 172.21.15.67:0/3586250697 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64180b0070 con 0x7f642806d810 2024-04-17T10:26:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f6426a00640 1 -- 172.21.15.67:0/3586250697 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64180ab400 con 0x7f642806d810 2024-04-17T10:26:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f6426a00640 1 -- 172.21.15.67:0/3586250697 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410001680 con 0x7f642806d810 2024-04-17T10:26:14.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 -- 172.21.15.67:0/3586250697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 msgr2=0x7f64280cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 --2- 172.21.15.67:0/3586250697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f64280cfad0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f64180097c0 tx=0x7f641801c390 comp rx=0 tx=0).stop 2024-04-17T10:26:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 -- 172.21.15.67:0/3586250697 shutdown_connections 2024-04-17T10:26:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 --2- 172.21.15.67:0/3586250697 >> v2:172.21.15.86:3300/0 conn(0x7f64280707c0 0x7f642815d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 --2- 172.21.15.67:0/3586250697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f64280cfad0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 -- 172.21.15.67:0/3586250697 >> 172.21.15.67:0/3586250697 conn(0x7f64280d1800 msgr2=0x7f64280d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 -- 172.21.15.67:0/3586250697 shutdown_connections 2024-04-17T10:26:14.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.175+0000 7f642e200640 1 -- 172.21.15.67:0/3586250697 wait complete. 2024-04-17T10:26:14.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 Processor -- start 2024-04-17T10:26:14.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 -- start start 2024-04-17T10:26:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f642815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642806f8b0 0x7f6428153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:14.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64280707c0 0x7f6428154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642806bf20 con 0x7f642806f8b0 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642806bda0 con 0x7f64280707c0 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642806c0a0 con 0x7f642806d810 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642806f8b0 0x7f6428153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642806f8b0 0x7f6428153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59620/0 (socket says 172.21.15.67:59620) 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 -- 172.21.15.67:0/669120730 learned_addr learned my addr 172.21.15.67:0/669120730 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f6427e00640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64280707c0 0x7f6428154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f6427400640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f642815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 -- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 msgr2=0x7f642815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f642815d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 -- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64280707c0 msgr2=0x7f6428154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64280707c0 0x7f6428154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 -- 172.21.15.67:0/669120730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642809d470 con 0x7f642806f8b0 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f6427e00640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64280707c0 0x7f6428154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641ea00640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642806f8b0 0x7f6428153380 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f641000c730 tx=0x7f641000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641fe00640 1 -- 172.21.15.67:0/669120730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410017070 con 0x7f642806f8b0 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641fe00640 1 -- 172.21.15.67:0/669120730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6410005ae0 con 0x7f642806f8b0 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f641fe00640 1 -- 172.21.15.67:0/669120730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6410007070 con 0x7f642806f8b0 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642809d750 con 0x7f642806f8b0 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.179+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642809dd10 con 0x7f642806f8b0 2024-04-17T10:26:14.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.183+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642804ef30 con 0x7f642806f8b0 2024-04-17T10:26:14.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.183+0000 7f641fe00640 1 -- 172.21.15.67:0/669120730 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f641001a070 con 0x7f642806f8b0 2024-04-17T10:26:14.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.183+0000 7f641fe00640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6404066f60 0x7f6404069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:14.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.183+0000 7f641fe00640 1 -- 172.21.15.67:0/669120730 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f6410084530 con 0x7f642806f8b0 2024-04-17T10:26:14.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.183+0000 7f6427400640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6404066f60 0x7f6404069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:14.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.183+0000 7f6427400640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6404066f60 0x7f6404069420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f64180ad000 tx=0x7f6418008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:14.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.187+0000 7f641fe00640 1 -- 172.21.15.67:0/669120730 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6410055590 con 0x7f642806f8b0 2024-04-17T10:26:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.351+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f64280b7780 con 0x7f642806f8b0 2024-04-17T10:26:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:14.999+0000 7f641fe00640 1 -- 172.21.15.67:0/669120730 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) v1 ==== 170+0+0 (secure 0 0 0) 0x7f6410054e90 con 0x7f642806f8b0 2024-04-17T10:26:14.995 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:26:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6404066f60 msgr2=0x7f6404069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6404066f60 0x7f6404069420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f64180ad000 tx=0x7f6418008040 comp rx=0 tx=0).stop 2024-04-17T10:26:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642806f8b0 msgr2=0x7f6428153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642806f8b0 0x7f6428153380 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f641000c730 tx=0x7f641000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 shutdown_connections 2024-04-17T10:26:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6404066f60 0x7f6404069420 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64280707c0 0x7f6428154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f642806f8b0 0x7f6428153380 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 --2- 172.21.15.67:0/669120730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f642806d810 0x7f642815d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 >> 172.21.15.67:0/669120730 conn(0x7f64280d1800 msgr2=0x7f64280b6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 shutdown_connections 2024-04-17T10:26:14.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.003+0000 7f642e200640 1 -- 172.21.15.67:0/669120730 wait complete. 2024-04-17T10:26:15.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:26:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 Processor -- start 2024-04-17T10:26:15.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 -- start start 2024-04-17T10:26:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09380aef60 0x7f09380cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09380cce60 0x7f093806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f093806e7b0 0x7f09380713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09380af340 con 0x7f09380cce60 2024-04-17T10:26:15.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09380b4990 con 0x7f09380aef60 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093815ae90 con 0x7f093806e7b0 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f092f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09380cce60 0x7f093806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f092f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09380cce60 0x7f093806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59632/0 (socket says 172.21.15.67:59632) 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f092f400640 1 -- 172.21.15.67:0/2222425601 learned_addr learned my addr 172.21.15.67:0/2222425601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f0937e00640 1 --2- 172.21.15.67:0/2222425601 >> v2:172.21.15.86:3300/0 conn(0x7f09380aef60 0x7f09380cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.187+0000 7f093d200640 1 --2- 172.21.15.67:0/2222425601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f093806e7b0 0x7f09380713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f092f400640 1 -- 172.21.15.67:0/2222425601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f093806e7b0 msgr2=0x7f09380713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f092f400640 1 --2- 172.21.15.67:0/2222425601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f093806e7b0 0x7f09380713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f092f400640 1 -- 172.21.15.67:0/2222425601 >> v2:172.21.15.86:3300/0 conn(0x7f09380aef60 msgr2=0x7f09380cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:15.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f092f400640 1 --2- 172.21.15.67:0/2222425601 >> v2:172.21.15.86:3300/0 conn(0x7f09380aef60 0x7f09380cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f092f400640 1 -- 172.21.15.67:0/2222425601 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093806ee80 con 0x7f09380cce60 2024-04-17T10:26:15.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f092f400640 1 --2- 172.21.15.67:0/2222425601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380cce60 0x7f093806e270 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f093806bdc0 tx=0x7f092001c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7a099b4c321ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f0937400640 1 -- 172.21.15.67:0/2222425601 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09200b0070 con 0x7f09380cce60 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f0937400640 1 -- 172.21.15.67:0/2222425601 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09200ab600 con 0x7f09380cce60 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f0937400640 1 -- 172.21.15.67:0/2222425601 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09200ab9d0 con 0x7f09380cce60 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 -- 172.21.15.67:0/2222425601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380cce60 msgr2=0x7f093806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 --2- 172.21.15.67:0/2222425601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380cce60 0x7f093806e270 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f093806bdc0 tx=0x7f092001c1d0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 -- 172.21.15.67:0/2222425601 shutdown_connections 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 --2- 172.21.15.67:0/2222425601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f093806e7b0 0x7f09380713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 --2- 172.21.15.67:0/2222425601 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380cce60 0x7f093806e270 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 --2- 172.21.15.67:0/2222425601 >> v2:172.21.15.86:3300/0 conn(0x7f09380aef60 0x7f09380cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 -- 172.21.15.67:0/2222425601 >> 172.21.15.67:0/2222425601 conn(0x7f09380b1740 msgr2=0x7f09380b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:15.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.191+0000 7f093f200640 1 -- 172.21.15.67:0/2222425601 shutdown_connections 2024-04-17T10:26:15.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 -- 172.21.15.67:0/2222425601 wait complete. 2024-04-17T10:26:15.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 Processor -- start 2024-04-17T10:26:15.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 -- start start 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f093806e7b0 0x7f0938135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380aef60 0x7f093806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09380cce60 0x7f093806b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093815bc10 con 0x7f09380aef60 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09380b3120 con 0x7f093806e7b0 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0938153320 con 0x7f09380cce60 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380aef60 0x7f093806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380aef60 0x7f093806ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59642/0 (socket says 172.21.15.67:59642) 2024-04-17T10:26:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 -- 172.21.15.67:0/730377545 learned_addr learned my addr 172.21.15.67:0/730377545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f0937e00640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f093806e7b0 0x7f0938135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093d200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09380cce60 0x7f093806b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 -- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09380cce60 msgr2=0x7f093806b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09380cce60 0x7f093806b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 -- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f093806e7b0 msgr2=0x7f0938135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f093806e7b0 0x7f0938135980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 -- 172.21.15.67:0/730377545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938153600 con 0x7f09380aef60 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093d200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09380cce60 0x7f093806b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f0937e00640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f093806e7b0 0x7f0938135980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f092f400640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380aef60 0x7f093806ac90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f09200ae040 tx=0x7f0920018f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:15.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f0934c00640 1 -- 172.21.15.67:0/730377545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09200b0040 con 0x7f09380aef60 2024-04-17T10:26:15.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f0934c00640 1 -- 172.21.15.67:0/730377545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09200b9070 con 0x7f09380aef60 2024-04-17T10:26:15.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f0934c00640 1 -- 172.21.15.67:0/730377545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09200b4880 con 0x7f09380aef60 2024-04-17T10:26:15.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.195+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09381538e0 con 0x7f09380aef60 2024-04-17T10:26:15.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.199+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093814c0d0 con 0x7f09380aef60 2024-04-17T10:26:15.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.199+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093804ef30 con 0x7f09380aef60 2024-04-17T10:26:15.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.199+0000 7f0934c00640 1 -- 172.21.15.67:0/730377545 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f09200ab4c0 con 0x7f09380aef60 2024-04-17T10:26:15.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.199+0000 7f0934c00640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0914066f60 0x7f0914069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:15.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.199+0000 7f0934c00640 1 -- 172.21.15.67:0/730377545 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f092011edb0 con 0x7f09380aef60 2024-04-17T10:26:15.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.199+0000 7f0937e00640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0914066f60 0x7f0914069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:15.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.203+0000 7f0937e00640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0914066f60 0x7f0914069420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0928009f50 tx=0x7f0928009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:15.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.203+0000 7f0934c00640 1 -- 172.21.15.67:0/730377545 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0920016110 con 0x7f09380aef60 2024-04-17T10:26:15.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:15.403+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f09380b1c50 con 0x7f09380aef60 2024-04-17T10:26:15.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.003+0000 7f0934c00640 1 -- 172.21.15.67:0/730377545 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f09200ef640 con 0x7f09380aef60 2024-04-17T10:26:15.996 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 5 and data pool 6 2024-04-17T10:26:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.003+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0914066f60 msgr2=0x7f0914069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.003+0000 7f093f200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0914066f60 0x7f0914069420 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f0928009f50 tx=0x7f0928009190 comp rx=0 tx=0).stop 2024-04-17T10:26:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.003+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380aef60 msgr2=0x7f093806ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:15.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.003+0000 7f093f200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380aef60 0x7f093806ac90 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f09200ae040 tx=0x7f0920018f70 comp rx=0 tx=0).stop 2024-04-17T10:26:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 shutdown_connections 2024-04-17T10:26:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0914066f60 0x7f0914069420 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09380cce60 0x7f093806b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09380aef60 0x7f093806ac90 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 --2- 172.21.15.67:0/730377545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f093806e7b0 0x7f0938135980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 >> 172.21.15.67:0/730377545 conn(0x7f09380b1740 msgr2=0x7f0938151cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:16.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 shutdown_connections 2024-04-17T10:26:16.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.007+0000 7f093f200640 1 -- 172.21.15.67:0/730377545 wait complete. 2024-04-17T10:26:16.042 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:26:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.187+0000 7f29cfe00640 1 Processor -- start 2024-04-17T10:26:16.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.187+0000 7f29cfe00640 1 -- start start 2024-04-17T10:26:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cfe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f29c81620a0 0x7f29c815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29c815e550 0x7f29c80cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cfe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c815eb50 con 0x7f29c80ce020 2024-04-17T10:26:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cfe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c8161650 con 0x7f29c815e550 2024-04-17T10:26:16.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cfe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c815ee30 con 0x7f29c81620a0 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29c815e550 0x7f29c80cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cd400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f29c80ce020 0x7f29c80ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45360/0 (socket says 172.21.15.67:45360) 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cd400640 1 -- 172.21.15.67:0/2295869917 learned_addr learned my addr 172.21.15.67:0/2295869917 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cca00640 1 --2- 172.21.15.67:0/2295869917 >> v2:172.21.15.86:3300/0 conn(0x7f29c81620a0 0x7f29c815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cde00640 1 -- 172.21.15.67:0/2295869917 >> v2:172.21.15.86:3300/0 conn(0x7f29c81620a0 msgr2=0x7f29c815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cde00640 1 --2- 172.21.15.67:0/2295869917 >> v2:172.21.15.86:3300/0 conn(0x7f29c81620a0 0x7f29c815e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cde00640 1 -- 172.21.15.67:0/2295869917 >> v1:172.21.15.67:6789/0 conn(0x7f29c80ce020 legacy=0x7f29c80ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29cde00640 1 -- 172.21.15.67:0/2295869917 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c81618a0 con 0x7f29c815e550 2024-04-17T10:26:16.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.191+0000 7f29c7e00640 1 -- 172.21.15.67:0/2295869917 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936167378 0 0) 0x7f29c815eb50 con 0x7f29c80ce020 2024-04-17T10:26:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cde00640 1 --2- 172.21.15.67:0/2295869917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c815e550 0x7f29c80cf510 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009d90 tx=0x7f29c001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754945c1e5e230a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29c7e00640 1 -- 172.21.15.67:0/2295869917 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c00b2070 con 0x7f29c815e550 2024-04-17T10:26:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29c7e00640 1 -- 172.21.15.67:0/2295869917 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c00a59e0 con 0x7f29c815e550 2024-04-17T10:26:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29c7e00640 1 -- 172.21.15.67:0/2295869917 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c00a5db0 con 0x7f29c815e550 2024-04-17T10:26:16.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 -- 172.21.15.67:0/2295869917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c815e550 msgr2=0x7f29c80cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 --2- 172.21.15.67:0/2295869917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c815e550 0x7f29c80cf510 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009d90 tx=0x7f29c001c900 comp rx=0 tx=0).stop 2024-04-17T10:26:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 -- 172.21.15.67:0/2295869917 shutdown_connections 2024-04-17T10:26:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 --2- 172.21.15.67:0/2295869917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c815e550 0x7f29c80cf510 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 --2- 172.21.15.67:0/2295869917 >> v2:172.21.15.86:3300/0 conn(0x7f29c81620a0 0x7f29c815e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 -- 172.21.15.67:0/2295869917 >> 172.21.15.67:0/2295869917 conn(0x7f29c80b33f0 msgr2=0x7f29c80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 -- 172.21.15.67:0/2295869917 shutdown_connections 2024-04-17T10:26:16.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 -- 172.21.15.67:0/2295869917 wait complete. 2024-04-17T10:26:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 Processor -- start 2024-04-17T10:26:16.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.195+0000 7f29cfe00640 1 -- start start 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c80ce020 0x7f29c815b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c815e550 0x7f29c815a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c81620a0 0x7f29c81524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80c8630 con 0x7f29c81620a0 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c80c84b0 con 0x7f29c815e550 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c816a9f0 con 0x7f29c80ce020 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c81620a0 0x7f29c81524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c81620a0 0x7f29c81524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59666/0 (socket says 172.21.15.67:59666) 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 -- 172.21.15.67:0/184368406 learned_addr learned my addr 172.21.15.67:0/184368406 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cca00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c815e550 0x7f29c815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cd400640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c80ce020 0x7f29c815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 -- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c80ce020 msgr2=0x7f29c815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c80ce020 0x7f29c815b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 -- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c815e550 msgr2=0x7f29c815a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c815e550 0x7f29c815a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 -- 172.21.15.67:0/184368406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c8159650 con 0x7f29c81620a0 2024-04-17T10:26:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cd400640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c80ce020 0x7f29c815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:16.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cde00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c81620a0 0x7f29c81524c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f29c00b0040 tx=0x7f29c00196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29c5600640 1 -- 172.21.15.67:0/184368406 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c00b2050 con 0x7f29c81620a0 2024-04-17T10:26:16.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29c5600640 1 -- 172.21.15.67:0/184368406 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c00b3040 con 0x7f29c81620a0 2024-04-17T10:26:16.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29c5600640 1 -- 172.21.15.67:0/184368406 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0003be0 con 0x7f29c81620a0 2024-04-17T10:26:16.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c8159930 con 0x7f29c81620a0 2024-04-17T10:26:16.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c8152dd0 con 0x7f29c81620a0 2024-04-17T10:26:16.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c8070ab0 con 0x7f29c81620a0 2024-04-17T10:26:16.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.199+0000 7f29c5600640 1 -- 172.21.15.67:0/184368406 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f29c00c0030 con 0x7f29c81620a0 2024-04-17T10:26:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.203+0000 7f29c5600640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a8066f60 0x7f29a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.203+0000 7f29c5600640 1 -- 172.21.15.67:0/184368406 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29c011f8e0 con 0x7f29c81620a0 2024-04-17T10:26:16.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.203+0000 7f29cd400640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a8066f60 0x7f29a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.203+0000 7f29cd400640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a8066f60 0x7f29a8069420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f29bc000f70 tx=0x7f29bc007c40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.207+0000 7f29c5600640 1 -- 172.21.15.67:0/184368406 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f29c00c7020 con 0x7f29c81620a0 2024-04-17T10:26:16.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.375+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29c8074e30 con 0x7f29c81620a0 2024-04-17T10:26:16.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.375+0000 7f29c5600640 1 -- 172.21.15.67:0/184368406 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16907 (secure 0 0 0) 0x7f29c0016520 con 0x7f29c81620a0 2024-04-17T10:26:16.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:16.371 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":38,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:26:15.407901+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:26:13.215866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-17T10:26:14.398708+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a8066f60 msgr2=0x7f29a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a8066f60 0x7f29a8069420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f29bc000f70 tx=0x7f29bc007c40 comp rx=0 tx=0).stop 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c81620a0 msgr2=0x7f29c81524c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c81620a0 0x7f29c81524c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f29c00b0040 tx=0x7f29c00196a0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 shutdown_connections 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a8066f60 0x7f29a8069420 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c81620a0 0x7f29c81524c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c815e550 0x7f29c815a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 --2- 172.21.15.67:0/184368406 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c80ce020 0x7f29c815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 >> 172.21.15.67:0/184368406 conn(0x7f29c80b33f0 msgr2=0x7f29c80cc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:16.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 shutdown_connections 2024-04-17T10:26:16.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.379+0000 7f29cfe00640 1 -- 172.21.15.67:0/184368406 wait complete. 2024-04-17T10:26:16.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:26:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f605d400640 1 Processor -- start 2024-04-17T10:26:16.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f605d400640 1 -- start start 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f605d400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f605d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60580d33d0 0x7f60580b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f605d400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580d2330 con 0x7f60580b5370 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f605d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60580b9910 con 0x7f60580d33d0 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f605d400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60580d2610 con 0x7f6058160f30 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60580d33d0 0x7f60580b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f60580d33d0 0x7f60580b93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59670/0 (socket says 172.21.15.67:59670) 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057400640 1 -- 172.21.15.67:0/2780286330 learned_addr learned my addr 172.21.15.67:0/2780286330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057e00640 1 --2- 172.21.15.67:0/2780286330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057400640 1 -- 172.21.15.67:0/2780286330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 msgr2=0x7f60580d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057400640 1 --2- 172.21.15.67:0/2780286330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057400640 1 -- 172.21.15.67:0/2780286330 >> v1:172.21.15.86:6789/0 conn(0x7f60580b5370 legacy=0x7f60580b6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.539+0000 7f6057400640 1 -- 172.21.15.67:0/2780286330 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580d28f0 con 0x7f60580d33d0 2024-04-17T10:26:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f6057400640 1 --2- 172.21.15.67:0/2780286330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580d33d0 0x7f60580b93d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6044009870 tx=0x7f604401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2adb1c6002dc0302 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f6056a00640 1 -- 172.21.15.67:0/2780286330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60440ab5f0 con 0x7f60580d33d0 2024-04-17T10:26:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f6056a00640 1 -- 172.21.15.67:0/2780286330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60440b0070 con 0x7f60580d33d0 2024-04-17T10:26:16.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f6056a00640 1 -- 172.21.15.67:0/2780286330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60440ab9b0 con 0x7f60580d33d0 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 -- 172.21.15.67:0/2780286330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580d33d0 msgr2=0x7f60580b93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 --2- 172.21.15.67:0/2780286330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580d33d0 0x7f60580b93d0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6044009870 tx=0x7f604401c520 comp rx=0 tx=0).stop 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 -- 172.21.15.67:0/2780286330 shutdown_connections 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 --2- 172.21.15.67:0/2780286330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580d33d0 0x7f60580b93d0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 --2- 172.21.15.67:0/2780286330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 -- 172.21.15.67:0/2780286330 >> 172.21.15.67:0/2780286330 conn(0x7f60580d0540 msgr2=0x7f60580d0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 -- 172.21.15.67:0/2780286330 shutdown_connections 2024-04-17T10:26:16.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 -- 172.21.15.67:0/2780286330 wait complete. 2024-04-17T10:26:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 Processor -- start 2024-04-17T10:26:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.543+0000 7f605d400640 1 -- start start 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580b5370 0x7f60580c4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60580d33d0 0x7f60580c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605816a830 con 0x7f60580b5370 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605816a6b0 con 0x7f60580d33d0 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605816a9b0 con 0x7f6058160f30 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580b5370 0x7f60580c4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580b5370 0x7f60580c4d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59676/0 (socket says 172.21.15.67:59676) 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 -- 172.21.15.67:0/4047704363 learned_addr learned my addr 172.21.15.67:0/4047704363 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60580d33d0 0x7f60580c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605ca00640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 -- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 msgr2=0x7f60580c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 -- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60580d33d0 msgr2=0x7f60580c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60580d33d0 0x7f60580c7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 -- 172.21.15.67:0/4047704363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580d1d80 con 0x7f60580b5370 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f6057e00640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580b5370 0x7f60580c4d50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f604800d830 tx=0x7f604800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f604fe00640 1 -- 172.21.15.67:0/4047704363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604800b6c0 con 0x7f60580b5370 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f604fe00640 1 -- 172.21.15.67:0/4047704363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6048019070 con 0x7f60580b5370 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f604fe00640 1 -- 172.21.15.67:0/4047704363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604800ba60 con 0x7f60580b5370 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60580c7660 con 0x7f60580b5370 2024-04-17T10:26:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605813bb70 con 0x7f60580b5370 2024-04-17T10:26:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6058074dc0 con 0x7f60580b5370 2024-04-17T10:26:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.547+0000 7f604fe00640 1 -- 172.21.15.67:0/4047704363 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f604801e460 con 0x7f60580b5370 2024-04-17T10:26:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.551+0000 7f604fe00640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6038066f60 0x7f6038069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.551+0000 7f604fe00640 1 -- 172.21.15.67:0/4047704363 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6048085770 con 0x7f60580b5370 2024-04-17T10:26:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.551+0000 7f6057400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6038066f60 0x7f6038069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.551+0000 7f6057400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6038066f60 0x7f6038069420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f60440af000 tx=0x7f60440a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.555+0000 7f604fe00640 1 -- 172.21.15.67:0/4047704363 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6048055f90 con 0x7f60580b5370 2024-04-17T10:26:16.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f605815d350 con 0x7f60580b5370 2024-04-17T10:26:16.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f604fe00640 1 -- 172.21.15.67:0/4047704363 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+16907 (secure 0 0 0) 0x7f604805a030 con 0x7f60580b5370 2024-04-17T10:26:16.714 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:16.714 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":38,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:26:15.407901+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:26:13.215866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-17T10:26:14.398708+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6038066f60 msgr2=0x7f6038069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6038066f60 0x7f6038069420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f60440af000 tx=0x7f60440a7000 comp rx=0 tx=0).stop 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580b5370 msgr2=0x7f60580c4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580b5370 0x7f60580c4d50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f604800d830 tx=0x7f604800dcf0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 shutdown_connections 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6038066f60 0x7f6038069420 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6058160f30 0x7f60580c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f60580d33d0 0x7f60580c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 --2- 172.21.15.67:0/4047704363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580b5370 0x7f60580c4d50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.719+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 >> 172.21.15.67:0/4047704363 conn(0x7f60580d0540 msgr2=0x7f605806c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:16.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.723+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 shutdown_connections 2024-04-17T10:26:16.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.723+0000 7f605d400640 1 -- 172.21.15.67:0/4047704363 wait complete. 2024-04-17T10:26:16.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:26:16.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75e0200640 1 Processor -- start 2024-04-17T10:26:16.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75e0200640 1 -- start start 2024-04-17T10:26:16.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75e0200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f75d8071af0 0x7f75d80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75e0200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75d815be70 0x7f75d815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75e0200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d815cb30 con 0x7f75d8159440 2024-04-17T10:26:16.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75e0200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d80a8650 con 0x7f75d815be70 2024-04-17T10:26:16.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75e0200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d815ce10 con 0x7f75d8071af0 2024-04-17T10:26:16.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.883+0000 7f75de200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75d815be70 0x7f75d815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75de200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75d815be70 0x7f75d815c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59680/0 (socket says 172.21.15.67:59680) 2024-04-17T10:26:16.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75de200640 1 -- 172.21.15.67:0/1374255795 learned_addr learned my addr 172.21.15.67:0/1374255795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:16.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75dce00640 1 --2- 172.21.15.67:0/1374255795 >> v2:172.21.15.86:3300/0 conn(0x7f75d8071af0 0x7f75d80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75de200640 1 -- 172.21.15.67:0/1374255795 >> v2:172.21.15.86:3300/0 conn(0x7f75d8071af0 msgr2=0x7f75d80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:16.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75de200640 1 --2- 172.21.15.67:0/1374255795 >> v2:172.21.15.86:3300/0 conn(0x7f75d8071af0 0x7f75d80ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75de200640 1 -- 172.21.15.67:0/1374255795 >> v1:172.21.15.86:6789/0 conn(0x7f75d8159440 legacy=0x7f75d80713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:16.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75de200640 1 -- 172.21.15.67:0/1374255795 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d80cd610 con 0x7f75d815be70 2024-04-17T10:26:16.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75de200640 1 --2- 172.21.15.67:0/1374255795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d815c2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f75d80cf770 tx=0x7f75d001c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962b7930a79780bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75d7e00640 1 -- 172.21.15.67:0/1374255795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00a5870 con 0x7f75d815be70 2024-04-17T10:26:16.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75d7e00640 1 -- 172.21.15.67:0/1374255795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d00a5a10 con 0x7f75d815be70 2024-04-17T10:26:16.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.887+0000 7f75d7e00640 1 -- 172.21.15.67:0/1374255795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00a5de0 con 0x7f75d815be70 2024-04-17T10:26:16.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- 172.21.15.67:0/1374255795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 msgr2=0x7f75d815c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 --2- 172.21.15.67:0/1374255795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d815c2c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f75d80cf770 tx=0x7f75d001c9b0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- 172.21.15.67:0/1374255795 shutdown_connections 2024-04-17T10:26:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 --2- 172.21.15.67:0/1374255795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d815c2c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 --2- 172.21.15.67:0/1374255795 >> v2:172.21.15.86:3300/0 conn(0x7f75d8071af0 0x7f75d80ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- 172.21.15.67:0/1374255795 >> 172.21.15.67:0/1374255795 conn(0x7f75d80b1740 msgr2=0x7f75d80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- 172.21.15.67:0/1374255795 shutdown_connections 2024-04-17T10:26:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- 172.21.15.67:0/1374255795 wait complete. 2024-04-17T10:26:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 Processor -- start 2024-04-17T10:26:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- start start 2024-04-17T10:26:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d8071af0 0x7f75d81515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d8159440 0x7f75d814c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d814f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d815d840 con 0x7f75d815be70 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d81598e0 con 0x7f75d8071af0 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75e0200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d815d9c0 con 0x7f75d8159440 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d814f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d814f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59690/0 (socket says 172.21.15.67:59690) 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 -- 172.21.15.67:0/4206745734 learned_addr learned my addr 172.21.15.67:0/4206745734 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75dce00640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d8159440 0x7f75d814c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75dd800640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d8071af0 0x7f75d81515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 -- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d8159440 msgr2=0x7f75d814c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d8159440 0x7f75d814c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 -- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d8071af0 msgr2=0x7f75d81515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d8071af0 0x7f75d81515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75de200640 1 -- 172.21.15.67:0/4206745734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8165b20 con 0x7f75d815be70 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75dce00640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d8159440 0x7f75d814c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.891+0000 7f75dd800640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d8071af0 0x7f75d81515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75de200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d814f950 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75d00af040 tx=0x7f75d0019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75d5600640 1 -- 172.21.15.67:0/4206745734 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00bc020 con 0x7f75d815be70 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75d5600640 1 -- 172.21.15.67:0/4206745734 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d00b8690 con 0x7f75d815be70 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d8165e00 con 0x7f75d815be70 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75d5600640 1 -- 172.21.15.67:0/4206745734 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00b8a80 con 0x7f75d815be70 2024-04-17T10:26:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d8157320 con 0x7f75d815be70 2024-04-17T10:26:16.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d804ef30 con 0x7f75d815be70 2024-04-17T10:26:16.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75d5600640 1 -- 172.21.15.67:0/4206745734 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f75d00bf020 con 0x7f75d815be70 2024-04-17T10:26:16.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75d5600640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b8066f60 0x7f75b8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:16.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.895+0000 7f75d5600640 1 -- 172.21.15.67:0/4206745734 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f75d0120290 con 0x7f75d815be70 2024-04-17T10:26:16.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.899+0000 7f75dd800640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b8066f60 0x7f75b8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:16.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.899+0000 7f75dd800640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b8066f60 0x7f75b8069420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f75cc003ec0 tx=0x7f75cc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:16.899+0000 7f75d5600640 1 -- 172.21.15.67:0/4206745734 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f75d00c6030 con 0x7f75d815be70 2024-04-17T10:26:17.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:17.103+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f75d80ce340 con 0x7f75d815be70 2024-04-17T10:26:18.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.019+0000 7f75d5600640 1 -- 172.21.15.67:0/4206745734 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f75d00165a0 con 0x7f75d815be70 2024-04-17T10:26:18.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b8066f60 msgr2=0x7f75b8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b8066f60 0x7f75b8069420 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f75cc003ec0 tx=0x7f75cc009040 comp rx=0 tx=0).stop 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 msgr2=0x7f75d814f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d814f950 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f75d00af040 tx=0x7f75d0019750 comp rx=0 tx=0).stop 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 shutdown_connections 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b8066f60 0x7f75b8069420 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d815be70 0x7f75d814f950 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d8159440 0x7f75d814c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 --2- 172.21.15.67:0/4206745734 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75d8071af0 0x7f75d81515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 >> 172.21.15.67:0/4206745734 conn(0x7f75d80b1740 msgr2=0x7f75d80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 shutdown_connections 2024-04-17T10:26:18.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.023+0000 7f75e0200640 1 -- 172.21.15.67:0/4206745734 wait complete. 2024-04-17T10:26:18.041 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:26:18.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:26:18.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.195+0000 7f1642a00640 1 Processor -- start 2024-04-17T10:26:18.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.195+0000 7f1642a00640 1 -- start start 2024-04-17T10:26:18.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.195+0000 7f1642a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f163c071ed0 0x7f163c15a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:18.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.195+0000 7f1642a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c0722b0 0x7f163c0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:18.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.195+0000 7f1642a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f163c0af2e0 con 0x7f163c15e210 2024-04-17T10:26:18.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.195+0000 7f1642a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f163c0ccbf0 con 0x7f163c071ed0 2024-04-17T10:26:18.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.195+0000 7f1642a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c15ac40 con 0x7f163c0722b0 2024-04-17T10:26:18.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f163be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f163c071ed0 0x7f163c15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:18.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f1633400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c0722b0 0x7f163c0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:18.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f163be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f163c071ed0 0x7f163c15a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:36536/0 (socket says 172.21.15.67:36536) 2024-04-17T10:26:18.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f163be00640 1 -- 172.21.15.67:0/201240162 learned_addr learned my addr 172.21.15.67:0/201240162 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:18.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f1640a00640 1 --1- 172.21.15.67:0/201240162 >> v1:172.21.15.86:6789/0 conn(0x7f163c15e210 0x7f163c1508d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:40798/0 (socket says 172.21.15.67:40798) 2024-04-17T10:26:18.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f163be00640 1 -- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c0722b0 msgr2=0x7f163c0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:18.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f163be00640 1 --2- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c0722b0 0x7f163c0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f163be00640 1 -- 172.21.15.67:0/201240162 >> v1:172.21.15.86:6789/0 conn(0x7f163c15e210 legacy=0x7f163c1508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:18.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f163be00640 1 -- 172.21.15.67:0/201240162 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c15af20 con 0x7f163c071ed0 2024-04-17T10:26:18.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.199+0000 7f1633400640 1 --2- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c0722b0 0x7f163c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:18.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f163be00640 1 --2- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c15a560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f162c00aa10 tx=0x7f162c01d930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92306b3208dbf3f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f163b400640 1 -- 172.21.15.67:0/201240162 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c0b3070 con 0x7f163c071ed0 2024-04-17T10:26:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f163b400640 1 -- 172.21.15.67:0/201240162 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162c0a6810 con 0x7f163c071ed0 2024-04-17T10:26:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f163b400640 1 -- 172.21.15.67:0/201240162 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162c00b040 con 0x7f163c071ed0 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 msgr2=0x7f163c15a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 --2- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c15a560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f162c00aa10 tx=0x7f162c01d930 comp rx=0 tx=0).stop 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- 172.21.15.67:0/201240162 shutdown_connections 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 --2- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c0722b0 0x7f163c0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 --2- 172.21.15.67:0/201240162 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c15a560 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- 172.21.15.67:0/201240162 >> 172.21.15.67:0/201240162 conn(0x7f163c0b1740 msgr2=0x7f163c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- 172.21.15.67:0/201240162 shutdown_connections 2024-04-17T10:26:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- 172.21.15.67:0/201240162 wait complete. 2024-04-17T10:26:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 Processor -- start 2024-04-17T10:26:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- start start 2024-04-17T10:26:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163c0722b0 0x7f163c0a96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c15e210 0x7f163c14f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0ce3e0 con 0x7f163c0722b0 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0ce260 con 0x7f163c071ed0 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.203+0000 7f1642a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163c0ce560 con 0x7f163c15e210 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163c0722b0 0x7f163c0a96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163c0722b0 0x7f163c0a96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59694/0 (socket says 172.21.15.67:59694) 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 -- 172.21.15.67:0/1076493211 learned_addr learned my addr 172.21.15.67:0/1076493211 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f163be00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1640a00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c15e210 0x7f163c14f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 -- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c15e210 msgr2=0x7f163c14f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c15e210 0x7f163c14f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 -- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 msgr2=0x7f163c158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 -- 172.21.15.67:0/1076493211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c158e50 con 0x7f163c0722b0 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f163be00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1633400640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163c0722b0 0x7f163c0a96a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f162400ebe0 tx=0x7f162400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1638c00640 1 -- 172.21.15.67:0/1076493211 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624017070 con 0x7f163c0722b0 2024-04-17T10:26:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1638c00640 1 -- 172.21.15.67:0/1076493211 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f162400cc30 con 0x7f163c0722b0 2024-04-17T10:26:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1638c00640 1 -- 172.21.15.67:0/1076493211 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1624013470 con 0x7f163c0722b0 2024-04-17T10:26:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f163c159130 con 0x7f163c0722b0 2024-04-17T10:26:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f163c1596f0 con 0x7f163c0722b0 2024-04-17T10:26:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163c04ef30 con 0x7f163c0722b0 2024-04-17T10:26:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.207+0000 7f1638c00640 1 -- 172.21.15.67:0/1076493211 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f162401a070 con 0x7f163c0722b0 2024-04-17T10:26:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.211+0000 7f1638c00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1618066f60 0x7f1618069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.211+0000 7f1638c00640 1 -- 172.21.15.67:0/1076493211 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f16240854d0 con 0x7f163c0722b0 2024-04-17T10:26:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.211+0000 7f163be00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1618066f60 0x7f1618069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.211+0000 7f163be00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1618066f60 0x7f1618069420 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f162c0b0000 tx=0x7f162c0a8070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.215+0000 7f1638c00640 1 -- 172.21.15.67:0/1076493211 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1624056340 con 0x7f163c0722b0 2024-04-17T10:26:18.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:18.411+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f163c1f4eb0 con 0x7f163c0722b0 2024-04-17T10:26:19.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.027+0000 7f1638c00640 1 -- 172.21.15.67:0/1076493211 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1624055c40 con 0x7f163c0722b0 2024-04-17T10:26:19.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1618066f60 msgr2=0x7f1618069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1618066f60 0x7f1618069420 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f162c0b0000 tx=0x7f162c0a8070 comp rx=0 tx=0).stop 2024-04-17T10:26:19.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163c0722b0 msgr2=0x7f163c0a96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163c0722b0 0x7f163c0a96a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f162400ebe0 tx=0x7f162400c520 comp rx=0 tx=0).stop 2024-04-17T10:26:19.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 shutdown_connections 2024-04-17T10:26:19.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1618066f60 0x7f1618069420 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f163c15e210 0x7f163c14f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f163c0722b0 0x7f163c0a96a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 --2- 172.21.15.67:0/1076493211 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f163c071ed0 0x7f163c158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 >> 172.21.15.67:0/1076493211 conn(0x7f163c0b1740 msgr2=0x7f163c0b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:19.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 shutdown_connections 2024-04-17T10:26:19.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.031+0000 7f1642a00640 1 -- 172.21.15.67:0/1076493211 wait complete. 2024-04-17T10:26:19.047 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:26:19.047 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:26:19.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc0d800640 1 Processor -- start 2024-04-17T10:26:19.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc0d800640 1 -- start start 2024-04-17T10:26:19.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc0d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc08161160 0x7fbc08099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc0d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc08099f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc0d800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc080d0760 con 0x7fbc081616c0 2024-04-17T10:26:19.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc0d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc080cfc70 con 0x7fbc08161160 2024-04-17T10:26:19.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc0d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080cc5d0 con 0x7fbc080b2570 2024-04-17T10:26:19.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc07e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc08161160 0x7fbc08099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc07e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbc08161160 0x7fbc08099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59700/0 (socket says 172.21.15.67:59700) 2024-04-17T10:26:19.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc07e00640 1 -- 172.21.15.67:0/2095084308 learned_addr learned my addr 172.21.15.67:0/2095084308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:19.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbbffe00640 1 --2- 172.21.15.67:0/2095084308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc08099f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc07e00640 1 -- 172.21.15.67:0/2095084308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 msgr2=0x7fbc08099f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:19.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc07e00640 1 --2- 172.21.15.67:0/2095084308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc08099f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc07e00640 1 -- 172.21.15.67:0/2095084308 >> v1:172.21.15.86:6789/0 conn(0x7fbc081616c0 legacy=0x7fbc08161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.195+0000 7fbc07e00640 1 -- 172.21.15.67:0/2095084308 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc080cc2a0 con 0x7fbc08161160 2024-04-17T10:26:19.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc07e00640 1 --2- 172.21.15.67:0/2095084308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc08099a50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009870 tx=0x7fbbf801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d334edeba30fb1b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc07400640 1 -- 172.21.15.67:0/2095084308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf80a7070 con 0x7fbc08161160 2024-04-17T10:26:19.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc07400640 1 -- 172.21.15.67:0/2095084308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf80b0070 con 0x7fbc08161160 2024-04-17T10:26:19.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc07400640 1 -- 172.21.15.67:0/2095084308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf80ab890 con 0x7fbc08161160 2024-04-17T10:26:19.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 -- 172.21.15.67:0/2095084308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 msgr2=0x7fbc08099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 --2- 172.21.15.67:0/2095084308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc08099a50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fbbf8009870 tx=0x7fbbf801c560 comp rx=0 tx=0).stop 2024-04-17T10:26:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 -- 172.21.15.67:0/2095084308 shutdown_connections 2024-04-17T10:26:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 --2- 172.21.15.67:0/2095084308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc08099f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 --2- 172.21.15.67:0/2095084308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc08099a50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 -- 172.21.15.67:0/2095084308 >> 172.21.15.67:0/2095084308 conn(0x7fbc080d1690 msgr2=0x7fbc080d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 -- 172.21.15.67:0/2095084308 shutdown_connections 2024-04-17T10:26:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.199+0000 7fbc0d800640 1 -- 172.21.15.67:0/2095084308 wait complete. 2024-04-17T10:26:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 Processor -- start 2024-04-17T10:26:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 -- start start 2024-04-17T10:26:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc080ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc0814cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc081616c0 0x7fbc080c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0806bda0 con 0x7fbc08161160 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc080b2bc0 con 0x7fbc081616c0 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0806bf20 con 0x7fbc080b2570 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc0814cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc0814cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59712/0 (socket says 172.21.15.67:59712) 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 -- 172.21.15.67:0/655588670 learned_addr learned my addr 172.21.15.67:0/655588670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc07e00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc080ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0ce00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc081616c0 0x7fbc080c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 -- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 msgr2=0x7fbc080ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc080ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 -- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc081616c0 msgr2=0x7fbc080c7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc081616c0 0x7fbc080c7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 -- 172.21.15.67:0/655588670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc080c8900 con 0x7fbc08161160 2024-04-17T10:26:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbbffe00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc0814cc60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000d620 tx=0x7fbbf000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc04c00640 1 -- 172.21.15.67:0/655588670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000b620 con 0x7fbc08161160 2024-04-17T10:26:19.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc04c00640 1 -- 172.21.15.67:0/655588670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbf0019070 con 0x7fbc08161160 2024-04-17T10:26:19.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc04c00640 1 -- 172.21.15.67:0/655588670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf000ba20 con 0x7fbc08161160 2024-04-17T10:26:19.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc080c8be0 con 0x7fbc08161160 2024-04-17T10:26:19.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc080c91a0 con 0x7fbc08161160 2024-04-17T10:26:19.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.203+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc08074dc0 con 0x7fbc08161160 2024-04-17T10:26:19.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.207+0000 7fbc04c00640 1 -- 172.21.15.67:0/655588670 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbbf000bbc0 con 0x7fbc08161160 2024-04-17T10:26:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.207+0000 7fbc04c00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbbe4066f60 0x7fbbe4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.207+0000 7fbc04c00640 1 -- 172.21.15.67:0/655588670 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbbf00854e0 con 0x7fbc08161160 2024-04-17T10:26:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.207+0000 7fbc07e00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbbe4066f60 0x7fbbe4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.207+0000 7fbc07e00640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbbe4066f60 0x7fbbe4069420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80af000 tx=0x7fbbf80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.211+0000 7fbc04c00640 1 -- 172.21.15.67:0/655588670 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbbf0055d00 con 0x7fbc08161160 2024-04-17T10:26:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc0809ac60 con 0x7fbc08161160 2024-04-17T10:26:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc04c00640 1 -- 172.21.15.67:0/655588670 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fbbf0059da0 con 0x7fbc08161160 2024-04-17T10:26:19.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:19.401 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1835906721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1835906721},{"type":"v1","addr":"172.21.15.86:6836","nonce":1835906721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2085506860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2085506860},{"type":"v1","addr":"172.21.15.86:6837","nonce":2085506860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2728147711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2728147711},{"type":"v1","addr":"172.21.15.67:6834","nonce":2728147711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:26:15.408252+0000","modified":"2024-04-17T10:26:18.413434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1409597842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1409597842},{"type":"v1","addr":"172.21.15.67:6835","nonce":1409597842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-17T10:26:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 18 2024-04-17T10:26:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbbe4066f60 msgr2=0x7fbbe4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbbe4066f60 0x7fbbe4069420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbbf80af000 tx=0x7fbbf80a7000 comp rx=0 tx=0).stop 2024-04-17T10:26:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 msgr2=0x7fbc0814cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc0814cc60 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000d620 tx=0x7fbbf000dae0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 shutdown_connections 2024-04-17T10:26:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbbe4066f60 0x7fbbe4069420 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbc081616c0 0x7fbc080c7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc08161160 0x7fbc0814cc60 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 --2- 172.21.15.67:0/655588670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbc080b2570 0x7fbc080ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 >> 172.21.15.67:0/655588670 conn(0x7fbc080d1690 msgr2=0x7fbc0806ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:19.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 shutdown_connections 2024-04-17T10:26:19.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.407+0000 7fbc0d800640 1 -- 172.21.15.67:0/655588670 wait complete. 2024-04-17T10:26:19.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:26:19.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.567+0000 7f17ffc00640 1 Processor -- start 2024-04-17T10:26:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.567+0000 7f17ffc00640 1 -- start start 2024-04-17T10:26:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17ffc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f17f81669e0 0x7f17f8168dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17ffc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f80b3ab0 con 0x7f17f815ecf0 2024-04-17T10:26:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17ffc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f8161700 con 0x7f17f8071790 2024-04-17T10:26:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17ffc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f81619e0 con 0x7f17f81669e0 2024-04-17T10:26:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17efe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17f815ecf0 0x7f17f806e670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45374/0 (socket says 172.21.15.67:45374) 2024-04-17T10:26:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17efe00640 1 -- 172.21.15.67:0/1784715483 learned_addr learned my addr 172.21.15.67:0/1784715483 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fdc00640 1 --2- 172.21.15.67:0/1784715483 >> v2:172.21.15.86:3300/0 conn(0x7f17f81669e0 0x7f17f8168dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1533344577 0 0) 0x7f17f80b3ab0 con 0x7f17f815ecf0 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17dc003530 con 0x7f17f815ecf0 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4268197423 0 0) 0x7f17dc003530 con 0x7f17f815ecf0 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 >> v2:172.21.15.86:3300/0 conn(0x7f17f81669e0 msgr2=0x7f17f8168dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 --2- 172.21.15.67:0/1784715483 >> v2:172.21.15.86:3300/0 conn(0x7f17f81669e0 0x7f17f8168dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 >> v1:172.21.15.86:6789/0 conn(0x7f17f8071790 legacy=0x7f17f8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f8161cc0 con 0x7f17f815ecf0 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f17e0001680 con 0x7f17f815ecf0 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 >> v1:172.21.15.67:6789/0 conn(0x7f17f815ecf0 legacy=0x7f17f806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17dc003a10 0x7f17dc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17dc004810 0x7f17dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f81619e0 con 0x7f17dc005220 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc00c170 con 0x7f17dc004810 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc00c370 con 0x7f17dc003a10 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fdc00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fd200640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17dc003a10 0x7f17dc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17efe00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17dc004810 0x7f17dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fdc00640 1 -- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17dc003a10 msgr2=0x7f17dc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fdc00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17dc003a10 0x7f17dc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fdc00640 1 -- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17dc004810 msgr2=0x7f17dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fdc00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17dc004810 0x7f17dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17fdc00640 1 -- 172.21.15.67:0/1784715483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc003530 con 0x7f17dc005220 2024-04-17T10:26:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.571+0000 7f17efe00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17dc004810 0x7f17dc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17fdc00640 1 -- 172.21.15.67:0/1784715483 reap_dead start 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17fdc00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17dc00b9d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009f90 tx=0x7f17f001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fa552873f069a60 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0009c10 con 0x7f17dc005220 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f0003ba0 con 0x7f17dc005220 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17f7e00640 1 -- 172.21.15.67:0/1784715483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f00a5980 con 0x7f17dc005220 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17ffc00640 1 -- 172.21.15.67:0/1784715483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 msgr2=0x7f17dc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17ffc00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17dc00b9d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f17f0009f90 tx=0x7f17f001cc80 comp rx=0 tx=0).stop 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17ffc00640 1 -- 172.21.15.67:0/1784715483 shutdown_connections 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17ffc00640 1 --2- 172.21.15.67:0/1784715483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17dc00b9d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17ffc00640 1 -- 172.21.15.67:0/1784715483 >> 172.21.15.67:0/1784715483 conn(0x7f17f80cfa30 msgr2=0x7f17f80cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17ffc00640 1 -- 172.21.15.67:0/1784715483 shutdown_connections 2024-04-17T10:26:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.575+0000 7f17ffc00640 1 -- 172.21.15.67:0/1784715483 wait complete. 2024-04-17T10:26:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 Processor -- start 2024-04-17T10:26:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 -- start start 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17f8152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17f8162830 0x7f17f8162cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8164f50 0x7f17f81653d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8071b70 con 0x7f17dc005220 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f81624c0 con 0x7f17f8164f50 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8071cf0 con 0x7f17f8162830 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17f8152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17f8152ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59736/0 (socket says 172.21.15.67:59736) 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 -- 172.21.15.67:0/530473136 learned_addr learned my addr 172.21.15.67:0/530473136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17efe00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17f8162830 0x7f17f8162cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fdc00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8164f50 0x7f17f81653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 -- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17f8162830 msgr2=0x7f17f8162cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17f8162830 0x7f17f8162cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 -- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8164f50 msgr2=0x7f17f81653d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8164f50 0x7f17f81653d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 -- 172.21.15.67:0/530473136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f81632f0 con 0x7f17dc005220 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17fd200640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17f8152ff0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f17e800f5f0 tx=0x7f17e800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17f5600640 1 -- 172.21.15.67:0/530473136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e80052e0 con 0x7f17dc005220 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17f5600640 1 -- 172.21.15.67:0/530473136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e8018070 con 0x7f17dc005220 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f81635d0 con 0x7f17dc005220 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17f5600640 1 -- 172.21.15.67:0/530473136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e8005480 con 0x7f17dc005220 2024-04-17T10:26:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f8163b90 con 0x7f17dc005220 2024-04-17T10:26:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.579+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f8165910 con 0x7f17dc005220 2024-04-17T10:26:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.583+0000 7f17f5600640 1 -- 172.21.15.67:0/530473136 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f17e8003820 con 0x7f17dc005220 2024-04-17T10:26:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.583+0000 7f17f5600640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17d4066f90 0x7f17d4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.583+0000 7f17f5600640 1 -- 172.21.15.67:0/530473136 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f17e8084f40 con 0x7f17dc005220 2024-04-17T10:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.583+0000 7f17efe00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17d4066f90 0x7f17d4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.583+0000 7f17efe00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17d4066f90 0x7f17d4069450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f17e0001ec0 tx=0x7f17e0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.587+0000 7f17f5600640 1 -- 172.21.15.67:0/530473136 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f17f8165910 con 0x7f17dc005220 2024-04-17T10:26:19.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.751+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17f8074e30 con 0x7f17dc005220 2024-04-17T10:26:19.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.751+0000 7f17f5600640 1 -- 172.21.15.67:0/530473136 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16894 (secure 0 0 0) 0x7f17e8055d20 con 0x7f17dc005220 2024-04-17T10:26:19.746 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:19.747 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":39,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:26:17.013394+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:26:13.215866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-17T10:26:14.398708+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17d4066f90 msgr2=0x7f17d4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17d4066f90 0x7f17d4069450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f17e0001ec0 tx=0x7f17e0009040 comp rx=0 tx=0).stop 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 msgr2=0x7f17f8152ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17f8152ff0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f17e800f5f0 tx=0x7f17e800fab0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 shutdown_connections 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17d4066f90 0x7f17d4069450 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17f8164f50 0x7f17f81653d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f17f8162830 0x7f17f8162cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 --2- 172.21.15.67:0/530473136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17dc005220 0x7f17f8152ff0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 >> 172.21.15.67:0/530473136 conn(0x7f17f80cfa30 msgr2=0x7f17f806ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 shutdown_connections 2024-04-17T10:26:19.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.755+0000 7f17ffc00640 1 -- 172.21.15.67:0/530473136 wait complete. 2024-04-17T10:26:19.767 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:26:19.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:26:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.911+0000 7f8216a00640 1 Processor -- start 2024-04-17T10:26:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8216a00640 1 -- start start 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8216a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82101446c0 0x7f8210144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8216a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8210144aa0 0x7f82100d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8216a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82100b6640 con 0x7f8210071840 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8216a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82100b51a0 con 0x7f8210144aa0 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8216a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82101456d0 con 0x7f82101446c0 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8207e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8210144aa0 0x7f82100d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8207e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8210144aa0 0x7f82100d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59742/0 (socket says 172.21.15.67:59742) 2024-04-17T10:26:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8207e00640 1 -- 172.21.15.67:0/1358416294 learned_addr learned my addr 172.21.15.67:0/1358416294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f820fe00640 1 --2- 172.21.15.67:0/1358416294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82101446c0 0x7f8210144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8207e00640 1 -- 172.21.15.67:0/1358416294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82101446c0 msgr2=0x7f8210144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8207e00640 1 --2- 172.21.15.67:0/1358416294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82101446c0 0x7f8210144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8207e00640 1 -- 172.21.15.67:0/1358416294 >> v1:172.21.15.86:6789/0 conn(0x7f8210071840 legacy=0x7f821015f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.915+0000 7f8207e00640 1 -- 172.21.15.67:0/1358416294 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82100b4870 con 0x7f8210144aa0 2024-04-17T10:26:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8207e00640 1 --2- 172.21.15.67:0/1358416294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100d0910 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009870 tx=0x7f81f801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2064a54addd7db9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f820f400640 1 -- 172.21.15.67:0/1358416294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f80aa600 con 0x7f8210144aa0 2024-04-17T10:26:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f820f400640 1 -- 172.21.15.67:0/1358416294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81f80af070 con 0x7f8210144aa0 2024-04-17T10:26:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f820f400640 1 -- 172.21.15.67:0/1358416294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f80aa9c0 con 0x7f8210144aa0 2024-04-17T10:26:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 -- 172.21.15.67:0/1358416294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 msgr2=0x7f82100d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 --2- 172.21.15.67:0/1358416294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100d0910 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009870 tx=0x7f81f801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 -- 172.21.15.67:0/1358416294 shutdown_connections 2024-04-17T10:26:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 --2- 172.21.15.67:0/1358416294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100d0910 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 --2- 172.21.15.67:0/1358416294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82101446c0 0x7f8210144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 -- 172.21.15.67:0/1358416294 >> 172.21.15.67:0/1358416294 conn(0x7f82100b34d0 msgr2=0x7f82100b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 -- 172.21.15.67:0/1358416294 shutdown_connections 2024-04-17T10:26:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 -- 172.21.15.67:0/1358416294 wait complete. 2024-04-17T10:26:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 Processor -- start 2024-04-17T10:26:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.919+0000 7f8216a00640 1 -- start start 2024-04-17T10:26:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8210071840 0x7f821015b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82101446c0 0x7f82100a8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100aa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82100d1260 con 0x7f8210144aa0 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821015c0d0 con 0x7f82101446c0 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8210168a40 con 0x7f8210071840 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100aa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100aa9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59748/0 (socket says 172.21.15.67:59748) 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 -- 172.21.15.67:0/1985087664 learned_addr learned my addr 172.21.15.67:0/1985087664 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f820fe00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8210071840 0x7f821015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8207e00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82101446c0 0x7f82100a8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 -- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8210071840 msgr2=0x7f821015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8210071840 0x7f821015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 -- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82101446c0 msgr2=0x7f82100a8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82101446c0 0x7f82100a8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 -- 172.21.15.67:0/1985087664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82100a8fe0 con 0x7f8210144aa0 2024-04-17T10:26:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8214a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100aa9e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f820800c770 tx=0x7f820800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f820cc00640 1 -- 172.21.15.67:0/1985087664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8208003b80 con 0x7f8210144aa0 2024-04-17T10:26:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f820cc00640 1 -- 172.21.15.67:0/1985087664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8208003d20 con 0x7f8210144aa0 2024-04-17T10:26:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f820cc00640 1 -- 172.21.15.67:0/1985087664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f820800aa90 con 0x7f8210144aa0 2024-04-17T10:26:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82100a92c0 con 0x7f8210144aa0 2024-04-17T10:26:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82100a9880 con 0x7f8210144aa0 2024-04-17T10:26:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8210074e30 con 0x7f8210144aa0 2024-04-17T10:26:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.923+0000 7f820cc00640 1 -- 172.21.15.67:0/1985087664 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f820800ac30 con 0x7f8210144aa0 2024-04-17T10:26:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.927+0000 7f820cc00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f81ec066f60 0x7f81ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.927+0000 7f820cc00640 1 -- 172.21.15.67:0/1985087664 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8208085000 con 0x7f8210144aa0 2024-04-17T10:26:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.927+0000 7f820fe00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f81ec066f60 0x7f81ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:19.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.927+0000 7f820fe00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f81ec066f60 0x7f81ec069420 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f821015ec30 tx=0x7f8200008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:19.931+0000 7f820cc00640 1 -- 172.21.15.67:0/1985087664 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8208055830 con 0x7f8210144aa0 2024-04-17T10:26:19.975 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T10:26:19.979+0000 7f6eacc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:26:20.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.091+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82100b41a0 con 0x7f81ec066f60 2024-04-17T10:26:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f820cc00640 1 -- 172.21.15.67:0/1985087664 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251956 (secure 0 0 0) 0x7f82100b41a0 con 0x7f81ec066f60 2024-04-17T10:26:20.095 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:20.097 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f81ec066f60 msgr2=0x7f81ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f81ec066f60 0x7f81ec069420 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f821015ec30 tx=0x7f8200008040 comp rx=0 tx=0).stop 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 msgr2=0x7f82100aa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100aa9e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f820800c770 tx=0x7f820800cc30 comp rx=0 tx=0).stop 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 shutdown_connections 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f81ec066f60 0x7f81ec069420 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8210144aa0 0x7f82100aa9e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82101446c0 0x7f82100a8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 --2- 172.21.15.67:0/1985087664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8210071840 0x7f821015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 >> 172.21.15.67:0/1985087664 conn(0x7f82100b34d0 msgr2=0x7f8210070fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 shutdown_connections 2024-04-17T10:26:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.099+0000 7f8216a00640 1 -- 172.21.15.67:0/1985087664 wait complete. 2024-04-17T10:26:20.115 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-04-17T10:26:18.153692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220864,"kb_used_data":4684,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753856,"statfs":{"total":773094113280,"available":772867948544,"internally_reserved":0,"allocated":4796416,"data_stored":2729416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.159968"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023976+0000","last_change":"2024-04-17T10:26:16.030068+0000","last_active":"2024-04-17T10:26:17.023976+0000","last_peered":"2024-04-17T10:26:17.023976+0000","last_clean":"2024-04-17T10:26:17.023976+0000","last_became_active":"2024-04-17T10:26:16.029727+0000","last_became_peered":"2024-04-17T10:26:16.029727+0000","last_unstale":"2024-04-17T10:26:17.023976+0000","last_undegraded":"2024-04-17T10:26:17.023976+0000","last_fullsized":"2024-04-17T10:26:17.023976+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:13.009368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023438+0000","last_change":"2024-04-17T10:26:16.037613+0000","last_active":"2024-04-17T10:26:17.023438+0000","last_peered":"2024-04-17T10:26:17.023438+0000","last_clean":"2024-04-17T10:26:17.023438+0000","last_became_active":"2024-04-17T10:26:16.033247+0000","last_became_peered":"2024-04-17T10:26:16.033247+0000","last_unstale":"2024-04-17T10:26:17.023438+0000","last_undegraded":"2024-04-17T10:26:17.023438+0000","last_fullsized":"2024-04-17T10:26:17.023438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:26:00.713000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129796+0000","last_change":"2024-04-17T10:26:16.043989+0000","last_active":"2024-04-17T10:26:17.129796+0000","last_peered":"2024-04-17T10:26:17.129796+0000","last_clean":"2024-04-17T10:26:17.129796+0000","last_became_active":"2024-04-17T10:26:16.043707+0000","last_became_peered":"2024-04-17T10:26:16.043707+0000","last_unstale":"2024-04-17T10:26:17.129796+0000","last_undegraded":"2024-04-17T10:26:17.129796+0000","last_fullsized":"2024-04-17T10:26:17.129796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:33.205243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130120+0000","last_change":"2024-04-17T10:26:16.021991+0000","last_active":"2024-04-17T10:26:17.130120+0000","last_peered":"2024-04-17T10:26:17.130120+0000","last_clean":"2024-04-17T10:26:17.130120+0000","last_became_active":"2024-04-17T10:26:16.019681+0000","last_became_peered":"2024-04-17T10:26:16.019681+0000","last_unstale":"2024-04-17T10:26:17.130120+0000","last_undegraded":"2024-04-17T10:26:17.130120+0000","last_fullsized":"2024-04-17T10:26:17.130120+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:38.924123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130060+0000","last_change":"2024-04-17T10:26:16.022947+0000","last_active":"2024-04-17T10:26:17.130060+0000","last_peered":"2024-04-17T10:26:17.130060+0000","last_clean":"2024-04-17T10:26:17.130060+0000","last_became_active":"2024-04-17T10:26:16.022391+0000","last_became_peered":"2024-04-17T10:26:16.022391+0000","last_unstale":"2024-04-17T10:26:17.130060+0000","last_undegraded":"2024-04-17T10:26:17.130060+0000","last_fullsized":"2024-04-17T10:26:17.130060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:41:35.936279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021654+0000","last_change":"2024-04-17T10:26:16.039470+0000","last_active":"2024-04-17T10:26:17.021654+0000","last_peered":"2024-04-17T10:26:17.021654+0000","last_clean":"2024-04-17T10:26:17.021654+0000","last_became_active":"2024-04-17T10:26:16.039208+0000","last_became_peered":"2024-04-17T10:26:16.039208+0000","last_unstale":"2024-04-17T10:26:17.021654+0000","last_undegraded":"2024-04-17T10:26:17.021654+0000","last_fullsized":"2024-04-17T10:26:17.021654+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:33.205243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128144+0000","last_change":"2024-04-17T10:26:16.036158+0000","last_active":"2024-04-17T10:26:17.128144+0000","last_peered":"2024-04-17T10:26:17.128144+0000","last_clean":"2024-04-17T10:26:17.128144+0000","last_became_active":"2024-04-17T10:26:16.035850+0000","last_became_peered":"2024-04-17T10:26:16.035850+0000","last_unstale":"2024-04-17T10:26:17.128144+0000","last_undegraded":"2024-04-17T10:26:17.128144+0000","last_fullsized":"2024-04-17T10:26:17.128144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:34.924217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022561+0000","last_change":"2024-04-17T10:26:16.033952+0000","last_active":"2024-04-17T10:26:17.022561+0000","last_peered":"2024-04-17T10:26:17.022561+0000","last_clean":"2024-04-17T10:26:17.022561+0000","last_became_active":"2024-04-17T10:26:16.033680+0000","last_became_peered":"2024-04-17T10:26:16.033680+0000","last_unstale":"2024-04-17T10:26:17.022561+0000","last_undegraded":"2024-04-17T10:26:17.022561+0000","last_fullsized":"2024-04-17T10:26:17.022561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:41.723249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129932+0000","last_change":"2024-04-17T10:26:16.045038+0000","last_active":"2024-04-17T10:26:17.129932+0000","last_peered":"2024-04-17T10:26:17.129932+0000","last_clean":"2024-04-17T10:26:17.129932+0000","last_became_active":"2024-04-17T10:26:16.044811+0000","last_became_peered":"2024-04-17T10:26:16.044811+0000","last_unstale":"2024-04-17T10:26:17.129932+0000","last_undegraded":"2024-04-17T10:26:17.129932+0000","last_fullsized":"2024-04-17T10:26:17.129932+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128770+0000","last_change":"2024-04-17T10:26:16.042109+0000","last_active":"2024-04-17T10:26:17.128770+0000","last_peered":"2024-04-17T10:26:17.128770+0000","last_clean":"2024-04-17T10:26:17.128770+0000","last_became_active":"2024-04-17T10:26:16.041786+0000","last_became_peered":"2024-04-17T10:26:16.041786+0000","last_unstale":"2024-04-17T10:26:17.128770+0000","last_undegraded":"2024-04-17T10:26:17.128770+0000","last_fullsized":"2024-04-17T10:26:17.128770+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:28.752753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128937+0000","last_change":"2024-04-17T10:26:16.031145+0000","last_active":"2024-04-17T10:26:17.128937+0000","last_peered":"2024-04-17T10:26:17.128937+0000","last_clean":"2024-04-17T10:26:17.128937+0000","last_became_active":"2024-04-17T10:26:16.030879+0000","last_became_peered":"2024-04-17T10:26:16.030879+0000","last_unstale":"2024-04-17T10:26:17.128937+0000","last_undegraded":"2024-04-17T10:26:17.128937+0000","last_fullsized":"2024-04-17T10:26:17.128937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:21.100099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129604+0000","last_change":"2024-04-17T10:26:16.024291+0000","last_active":"2024-04-17T10:26:17.129604+0000","last_peered":"2024-04-17T10:26:17.129604+0000","last_clean":"2024-04-17T10:26:17.129604+0000","last_became_active":"2024-04-17T10:26:16.023880+0000","last_became_peered":"2024-04-17T10:26:16.023880+0000","last_unstale":"2024-04-17T10:26:17.129604+0000","last_undegraded":"2024-04-17T10:26:17.129604+0000","last_fullsized":"2024-04-17T10:26:17.129604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:24:48.055839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129069+0000","last_change":"2024-04-17T10:26:16.042987+0000","last_active":"2024-04-17T10:26:17.129069+0000","last_peered":"2024-04-17T10:26:17.129069+0000","last_clean":"2024-04-17T10:26:17.129069+0000","last_became_active":"2024-04-17T10:26:16.041681+0000","last_became_peered":"2024-04-17T10:26:16.041681+0000","last_unstale":"2024-04-17T10:26:17.129069+0000","last_undegraded":"2024-04-17T10:26:17.129069+0000","last_fullsized":"2024-04-17T10:26:17.129069+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:52.680433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020007+0000","last_change":"2024-04-17T10:26:16.025657+0000","last_active":"2024-04-17T10:26:17.020007+0000","last_peered":"2024-04-17T10:26:17.020007+0000","last_clean":"2024-04-17T10:26:17.020007+0000","last_became_active":"2024-04-17T10:26:16.025140+0000","last_became_peered":"2024-04-17T10:26:16.025140+0000","last_unstale":"2024-04-17T10:26:17.020007+0000","last_undegraded":"2024-04-17T10:26:17.020007+0000","last_fullsized":"2024-04-17T10:26:17.020007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:33.205243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128352+0000","last_change":"2024-04-17T10:26:16.022415+0000","last_active":"2024-04-17T10:26:17.128352+0000","last_peered":"2024-04-17T10:26:17.128352+0000","last_clean":"2024-04-17T10:26:17.128352+0000","last_became_active":"2024-04-17T10:26:16.022034+0000","last_became_peered":"2024-04-17T10:26:16.022034+0000","last_unstale":"2024-04-17T10:26:17.128352+0000","last_undegraded":"2024-04-17T10:26:17.128352+0000","last_fullsized":"2024-04-17T10:26:17.128352+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:18.186055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127197+0000","last_change":"2024-04-17T10:26:16.034353+0000","last_active":"2024-04-17T10:26:17.127197+0000","last_peered":"2024-04-17T10:26:17.127197+0000","last_clean":"2024-04-17T10:26:17.127197+0000","last_became_active":"2024-04-17T10:26:16.034087+0000","last_became_peered":"2024-04-17T10:26:16.034087+0000","last_unstale":"2024-04-17T10:26:17.127197+0000","last_undegraded":"2024-04-17T10:26:17.127197+0000","last_fullsized":"2024-04-17T10:26:17.127197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128144+0000","last_change":"2024-04-17T10:26:16.034324+0000","last_active":"2024-04-17T10:26:17.128144+0000","last_peered":"2024-04-17T10:26:17.128144+0000","last_clean":"2024-04-17T10:26:17.128144+0000","last_became_active":"2024-04-17T10:26:16.033864+0000","last_became_peered":"2024-04-17T10:26:16.033864+0000","last_unstale":"2024-04-17T10:26:17.128144+0000","last_undegraded":"2024-04-17T10:26:17.128144+0000","last_fullsized":"2024-04-17T10:26:17.128144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:32:59.576436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129441+0000","last_change":"2024-04-17T10:26:16.017782+0000","last_active":"2024-04-17T10:26:17.129441+0000","last_peered":"2024-04-17T10:26:17.129441+0000","last_clean":"2024-04-17T10:26:17.129441+0000","last_became_active":"2024-04-17T10:26:16.017547+0000","last_became_peered":"2024-04-17T10:26:16.017547+0000","last_unstale":"2024-04-17T10:26:17.129441+0000","last_undegraded":"2024-04-17T10:26:17.129441+0000","last_fullsized":"2024-04-17T10:26:17.129441+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:24:48.055839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130145+0000","last_change":"2024-04-17T10:26:16.033700+0000","last_active":"2024-04-17T10:26:17.130145+0000","last_peered":"2024-04-17T10:26:17.130145+0000","last_clean":"2024-04-17T10:26:17.130145+0000","last_became_active":"2024-04-17T10:26:16.033446+0000","last_became_peered":"2024-04-17T10:26:16.033446+0000","last_unstale":"2024-04-17T10:26:17.130145+0000","last_undegraded":"2024-04-17T10:26:17.130145+0000","last_fullsized":"2024-04-17T10:26:17.130145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:35.619573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128867+0000","last_change":"2024-04-17T10:26:16.020772+0000","last_active":"2024-04-17T10:26:17.128867+0000","last_peered":"2024-04-17T10:26:17.128867+0000","last_clean":"2024-04-17T10:26:17.128867+0000","last_became_active":"2024-04-17T10:26:16.020479+0000","last_became_peered":"2024-04-17T10:26:16.020479+0000","last_unstale":"2024-04-17T10:26:17.128867+0000","last_undegraded":"2024-04-17T10:26:17.128867+0000","last_fullsized":"2024-04-17T10:26:17.128867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:01.722709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129011+0000","last_change":"2024-04-17T10:26:16.042143+0000","last_active":"2024-04-17T10:26:17.129011+0000","last_peered":"2024-04-17T10:26:17.129011+0000","last_clean":"2024-04-17T10:26:17.129011+0000","last_became_active":"2024-04-17T10:26:16.041653+0000","last_became_peered":"2024-04-17T10:26:16.041653+0000","last_unstale":"2024-04-17T10:26:17.129011+0000","last_undegraded":"2024-04-17T10:26:17.129011+0000","last_fullsized":"2024-04-17T10:26:17.129011+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:20.516273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020208+0000","last_change":"2024-04-17T10:26:16.025404+0000","last_active":"2024-04-17T10:26:17.020208+0000","last_peered":"2024-04-17T10:26:17.020208+0000","last_clean":"2024-04-17T10:26:17.020208+0000","last_became_active":"2024-04-17T10:26:16.024589+0000","last_became_peered":"2024-04-17T10:26:16.024589+0000","last_unstale":"2024-04-17T10:26:17.020208+0000","last_undegraded":"2024-04-17T10:26:17.020208+0000","last_fullsized":"2024-04-17T10:26:17.020208+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:34.924217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127674+0000","last_change":"2024-04-17T10:26:16.019330+0000","last_active":"2024-04-17T10:26:17.127674+0000","last_peered":"2024-04-17T10:26:17.127674+0000","last_clean":"2024-04-17T10:26:17.127674+0000","last_became_active":"2024-04-17T10:26:16.019027+0000","last_became_peered":"2024-04-17T10:26:16.019027+0000","last_unstale":"2024-04-17T10:26:17.127674+0000","last_undegraded":"2024-04-17T10:26:17.127674+0000","last_fullsized":"2024-04-17T10:26:17.127674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.321256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129818+0000","last_change":"2024-04-17T10:26:16.022468+0000","last_active":"2024-04-17T10:26:17.129818+0000","last_peered":"2024-04-17T10:26:17.129818+0000","last_clean":"2024-04-17T10:26:17.129818+0000","last_became_active":"2024-04-17T10:26:16.022174+0000","last_became_peered":"2024-04-17T10:26:16.022174+0000","last_unstale":"2024-04-17T10:26:17.129818+0000","last_undegraded":"2024-04-17T10:26:17.129818+0000","last_fullsized":"2024-04-17T10:26:17.129818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:17.422739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129415+0000","last_change":"2024-04-17T10:26:16.035114+0000","last_active":"2024-04-17T10:26:17.129415+0000","last_peered":"2024-04-17T10:26:17.129415+0000","last_clean":"2024-04-17T10:26:17.129415+0000","last_became_active":"2024-04-17T10:26:16.034619+0000","last_became_peered":"2024-04-17T10:26:16.034619+0000","last_unstale":"2024-04-17T10:26:17.129415+0000","last_undegraded":"2024-04-17T10:26:17.129415+0000","last_fullsized":"2024-04-17T10:26:17.129415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:32.972807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022434+0000","last_change":"2024-04-17T10:26:16.047228+0000","last_active":"2024-04-17T10:26:17.022434+0000","last_peered":"2024-04-17T10:26:17.022434+0000","last_clean":"2024-04-17T10:26:17.022434+0000","last_became_active":"2024-04-17T10:26:16.046864+0000","last_became_peered":"2024-04-17T10:26:16.046864+0000","last_unstale":"2024-04-17T10:26:17.022434+0000","last_undegraded":"2024-04-17T10:26:17.022434+0000","last_fullsized":"2024-04-17T10:26:17.022434+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.789876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129327+0000","last_change":"2024-04-17T10:26:16.015586+0000","last_active":"2024-04-17T10:26:17.129327+0000","last_peered":"2024-04-17T10:26:17.129327+0000","last_clean":"2024-04-17T10:26:17.129327+0000","last_became_active":"2024-04-17T10:26:16.013966+0000","last_became_peered":"2024-04-17T10:26:16.013966+0000","last_unstale":"2024-04-17T10:26:17.129327+0000","last_undegraded":"2024-04-17T10:26:17.129327+0000","last_fullsized":"2024-04-17T10:26:17.129327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:40.218457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128684+0000","last_change":"2024-04-17T10:26:16.037153+0000","last_active":"2024-04-17T10:26:17.128684+0000","last_peered":"2024-04-17T10:26:17.128684+0000","last_clean":"2024-04-17T10:26:17.128684+0000","last_became_active":"2024-04-17T10:26:16.036876+0000","last_became_peered":"2024-04-17T10:26:16.036876+0000","last_unstale":"2024-04-17T10:26:17.128684+0000","last_undegraded":"2024-04-17T10:26:17.128684+0000","last_fullsized":"2024-04-17T10:26:17.128684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:32:59.576436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128761+0000","last_change":"2024-04-17T10:26:16.039129+0000","last_active":"2024-04-17T10:26:17.128761+0000","last_peered":"2024-04-17T10:26:17.128761+0000","last_clean":"2024-04-17T10:26:17.128761+0000","last_became_active":"2024-04-17T10:26:16.038832+0000","last_became_peered":"2024-04-17T10:26:16.038832+0000","last_unstale":"2024-04-17T10:26:17.128761+0000","last_undegraded":"2024-04-17T10:26:17.128761+0000","last_fullsized":"2024-04-17T10:26:17.128761+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:41.855139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127939+0000","last_change":"2024-04-17T10:26:16.029526+0000","last_active":"2024-04-17T10:26:17.127939+0000","last_peered":"2024-04-17T10:26:17.127939+0000","last_clean":"2024-04-17T10:26:17.127939+0000","last_became_active":"2024-04-17T10:26:16.028988+0000","last_became_peered":"2024-04-17T10:26:16.028988+0000","last_unstale":"2024-04-17T10:26:17.127939+0000","last_undegraded":"2024-04-17T10:26:17.127939+0000","last_fullsized":"2024-04-17T10:26:17.127939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:34.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127629+0000","last_change":"2024-04-17T10:26:16.025147+0000","last_active":"2024-04-17T10:26:17.127629+0000","last_peered":"2024-04-17T10:26:17.127629+0000","last_clean":"2024-04-17T10:26:17.127629+0000","last_became_active":"2024-04-17T10:26:16.024610+0000","last_became_peered":"2024-04-17T10:26:16.024610+0000","last_unstale":"2024-04-17T10:26:17.127629+0000","last_undegraded":"2024-04-17T10:26:17.127629+0000","last_fullsized":"2024-04-17T10:26:17.127629+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:34.924217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021529+0000","last_change":"2024-04-17T10:26:16.032556+0000","last_active":"2024-04-17T10:26:17.021529+0000","last_peered":"2024-04-17T10:26:17.021529+0000","last_clean":"2024-04-17T10:26:17.021529+0000","last_became_active":"2024-04-17T10:26:16.032268+0000","last_became_peered":"2024-04-17T10:26:16.032268+0000","last_unstale":"2024-04-17T10:26:17.021529+0000","last_undegraded":"2024-04-17T10:26:17.021529+0000","last_fullsized":"2024-04-17T10:26:17.021529+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:39.610854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127590+0000","last_change":"2024-04-17T10:26:16.034205+0000","last_active":"2024-04-17T10:26:17.127590+0000","last_peered":"2024-04-17T10:26:17.127590+0000","last_clean":"2024-04-17T10:26:17.127590+0000","last_became_active":"2024-04-17T10:26:16.033881+0000","last_became_peered":"2024-04-17T10:26:16.033881+0000","last_unstale":"2024-04-17T10:26:17.127590+0000","last_undegraded":"2024-04-17T10:26:17.127590+0000","last_fullsized":"2024-04-17T10:26:17.127590+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.789876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129454+0000","last_change":"2024-04-17T10:26:15.021792+0000","last_active":"2024-04-17T10:26:17.129454+0000","last_peered":"2024-04-17T10:26:17.129454+0000","last_clean":"2024-04-17T10:26:17.129454+0000","last_became_active":"2024-04-17T10:26:15.021421+0000","last_became_peered":"2024-04-17T10:26:15.021421+0000","last_unstale":"2024-04-17T10:26:17.129454+0000","last_undegraded":"2024-04-17T10:26:17.129454+0000","last_fullsized":"2024-04-17T10:26:17.129454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:01.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022319+0000","last_change":"2024-04-17T10:26:16.034650+0000","last_active":"2024-04-17T10:26:17.022319+0000","last_peered":"2024-04-17T10:26:17.022319+0000","last_clean":"2024-04-17T10:26:17.022319+0000","last_became_active":"2024-04-17T10:26:16.034250+0000","last_became_peered":"2024-04-17T10:26:16.034250+0000","last_unstale":"2024-04-17T10:26:17.022319+0000","last_undegraded":"2024-04-17T10:26:17.022319+0000","last_fullsized":"2024-04-17T10:26:17.022319+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:34.289100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129888+0000","last_change":"2024-04-17T10:26:15.024021+0000","last_active":"2024-04-17T10:26:17.129888+0000","last_peered":"2024-04-17T10:26:17.129888+0000","last_clean":"2024-04-17T10:26:17.129888+0000","last_became_active":"2024-04-17T10:26:15.023550+0000","last_became_peered":"2024-04-17T10:26:15.023550+0000","last_unstale":"2024-04-17T10:26:17.129888+0000","last_undegraded":"2024-04-17T10:26:17.129888+0000","last_fullsized":"2024-04-17T10:26:17.129888+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:40.716005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020194+0000","last_change":"2024-04-17T10:26:16.021799+0000","last_active":"2024-04-17T10:26:17.020194+0000","last_peered":"2024-04-17T10:26:17.020194+0000","last_clean":"2024-04-17T10:26:17.020194+0000","last_became_active":"2024-04-17T10:26:16.021514+0000","last_became_peered":"2024-04-17T10:26:16.021514+0000","last_unstale":"2024-04-17T10:26:17.020194+0000","last_undegraded":"2024-04-17T10:26:17.020194+0000","last_fullsized":"2024-04-17T10:26:17.020194+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.789876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022361+0000","last_change":"2024-04-17T10:26:15.032062+0000","last_active":"2024-04-17T10:26:17.022361+0000","last_peered":"2024-04-17T10:26:17.022361+0000","last_clean":"2024-04-17T10:26:17.022361+0000","last_became_active":"2024-04-17T10:26:15.031663+0000","last_became_peered":"2024-04-17T10:26:15.031663+0000","last_unstale":"2024-04-17T10:26:17.022361+0000","last_undegraded":"2024-04-17T10:26:17.022361+0000","last_fullsized":"2024-04-17T10:26:17.022361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023991+0000","last_change":"2024-04-17T10:26:16.043046+0000","last_active":"2024-04-17T10:26:17.023991+0000","last_peered":"2024-04-17T10:26:17.023991+0000","last_clean":"2024-04-17T10:26:17.023991+0000","last_became_active":"2024-04-17T10:26:16.042810+0000","last_became_peered":"2024-04-17T10:26:16.042810+0000","last_unstale":"2024-04-17T10:26:17.023991+0000","last_undegraded":"2024-04-17T10:26:17.023991+0000","last_fullsized":"2024-04-17T10:26:17.023991+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:59.825316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.019923+0000","last_change":"2024-04-17T10:26:15.014570+0000","last_active":"2024-04-17T10:26:17.019923+0000","last_peered":"2024-04-17T10:26:17.019923+0000","last_clean":"2024-04-17T10:26:17.019923+0000","last_became_active":"2024-04-17T10:26:15.014304+0000","last_became_peered":"2024-04-17T10:26:15.014304+0000","last_unstale":"2024-04-17T10:26:17.019923+0000","last_undegraded":"2024-04-17T10:26:17.019923+0000","last_fullsized":"2024-04-17T10:26:17.019923+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129296+0000","last_change":"2024-04-17T10:26:16.044841+0000","last_active":"2024-04-17T10:26:17.129296+0000","last_peered":"2024-04-17T10:26:17.129296+0000","last_clean":"2024-04-17T10:26:17.129296+0000","last_became_active":"2024-04-17T10:26:16.044587+0000","last_became_peered":"2024-04-17T10:26:16.044587+0000","last_unstale":"2024-04-17T10:26:17.129296+0000","last_undegraded":"2024-04-17T10:26:17.129296+0000","last_fullsized":"2024-04-17T10:26:17.129296+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:40.218457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129168+0000","last_change":"2024-04-17T10:26:15.026920+0000","last_active":"2024-04-17T10:26:17.129168+0000","last_peered":"2024-04-17T10:26:17.129168+0000","last_clean":"2024-04-17T10:26:17.129168+0000","last_became_active":"2024-04-17T10:26:15.026278+0000","last_became_peered":"2024-04-17T10:26:15.026278+0000","last_unstale":"2024-04-17T10:26:17.129168+0000","last_undegraded":"2024-04-17T10:26:17.129168+0000","last_fullsized":"2024-04-17T10:26:17.129168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128542+0000","last_change":"2024-04-17T10:26:16.042948+0000","last_active":"2024-04-17T10:26:17.128542+0000","last_peered":"2024-04-17T10:26:17.128542+0000","last_clean":"2024-04-17T10:26:17.128542+0000","last_became_active":"2024-04-17T10:26:16.042667+0000","last_became_peered":"2024-04-17T10:26:16.042667+0000","last_unstale":"2024-04-17T10:26:17.128542+0000","last_undegraded":"2024-04-17T10:26:17.128542+0000","last_fullsized":"2024-04-17T10:26:17.128542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:03.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127725+0000","last_change":"2024-04-17T10:26:15.030585+0000","last_active":"2024-04-17T10:26:17.127725+0000","last_peered":"2024-04-17T10:26:17.127725+0000","last_clean":"2024-04-17T10:26:17.127725+0000","last_became_active":"2024-04-17T10:26:15.030322+0000","last_became_peered":"2024-04-17T10:26:15.030322+0000","last_unstale":"2024-04-17T10:26:17.127725+0000","last_undegraded":"2024-04-17T10:26:17.127725+0000","last_fullsized":"2024-04-17T10:26:17.127725+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:12:32.979655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020263+0000","last_change":"2024-04-17T10:26:16.030615+0000","last_active":"2024-04-17T10:26:17.020263+0000","last_peered":"2024-04-17T10:26:17.020263+0000","last_clean":"2024-04-17T10:26:17.020263+0000","last_became_active":"2024-04-17T10:26:16.030317+0000","last_became_peered":"2024-04-17T10:26:16.030317+0000","last_unstale":"2024-04-17T10:26:17.020263+0000","last_undegraded":"2024-04-17T10:26:17.020263+0000","last_fullsized":"2024-04-17T10:26:17.020263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:18.186055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021854+0000","last_change":"2024-04-17T10:26:15.029862+0000","last_active":"2024-04-17T10:26:17.021854+0000","last_peered":"2024-04-17T10:26:17.021854+0000","last_clean":"2024-04-17T10:26:17.021854+0000","last_became_active":"2024-04-17T10:26:15.029546+0000","last_became_peered":"2024-04-17T10:26:15.029546+0000","last_unstale":"2024-04-17T10:26:17.021854+0000","last_undegraded":"2024-04-17T10:26:17.021854+0000","last_fullsized":"2024-04-17T10:26:17.021854+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023446+0000","last_change":"2024-04-17T10:26:16.035881+0000","last_active":"2024-04-17T10:26:17.023446+0000","last_peered":"2024-04-17T10:26:17.023446+0000","last_clean":"2024-04-17T10:26:17.023446+0000","last_became_active":"2024-04-17T10:26:16.035556+0000","last_became_peered":"2024-04-17T10:26:16.035556+0000","last_unstale":"2024-04-17T10:26:17.023446+0000","last_undegraded":"2024-04-17T10:26:17.023446+0000","last_fullsized":"2024-04-17T10:26:17.023446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:15.886315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.019302+0000","last_change":"2024-04-17T10:26:15.022158+0000","last_active":"2024-04-17T10:26:17.019302+0000","last_peered":"2024-04-17T10:26:17.019302+0000","last_clean":"2024-04-17T10:26:17.019302+0000","last_became_active":"2024-04-17T10:26:15.021932+0000","last_became_peered":"2024-04-17T10:26:15.021932+0000","last_unstale":"2024-04-17T10:26:17.019302+0000","last_undegraded":"2024-04-17T10:26:17.019302+0000","last_fullsized":"2024-04-17T10:26:17.019302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130170+0000","last_change":"2024-04-17T10:26:16.022122+0000","last_active":"2024-04-17T10:26:17.130170+0000","last_peered":"2024-04-17T10:26:17.130170+0000","last_clean":"2024-04-17T10:26:17.130170+0000","last_became_active":"2024-04-17T10:26:16.019272+0000","last_became_peered":"2024-04-17T10:26:16.019272+0000","last_unstale":"2024-04-17T10:26:17.130170+0000","last_undegraded":"2024-04-17T10:26:17.130170+0000","last_fullsized":"2024-04-17T10:26:17.130170+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:02.272505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.024026+0000","last_change":"2024-04-17T10:26:15.022167+0000","last_active":"2024-04-17T10:26:17.024026+0000","last_peered":"2024-04-17T10:26:17.024026+0000","last_clean":"2024-04-17T10:26:17.024026+0000","last_became_active":"2024-04-17T10:26:15.021764+0000","last_became_peered":"2024-04-17T10:26:15.021764+0000","last_unstale":"2024-04-17T10:26:17.024026+0000","last_undegraded":"2024-04-17T10:26:17.024026+0000","last_fullsized":"2024-04-17T10:26:17.024026+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:24.783338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127892+0000","last_change":"2024-04-17T10:26:16.027057+0000","last_active":"2024-04-17T10:26:17.127892+0000","last_peered":"2024-04-17T10:26:17.127892+0000","last_clean":"2024-04-17T10:26:17.127892+0000","last_became_active":"2024-04-17T10:26:16.026783+0000","last_became_peered":"2024-04-17T10:26:16.026783+0000","last_unstale":"2024-04-17T10:26:17.127892+0000","last_undegraded":"2024-04-17T10:26:17.127892+0000","last_fullsized":"2024-04-17T10:26:17.127892+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:03.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129156+0000","last_change":"2024-04-17T10:26:15.025321+0000","last_active":"2024-04-17T10:26:17.129156+0000","last_peered":"2024-04-17T10:26:17.129156+0000","last_clean":"2024-04-17T10:26:17.129156+0000","last_became_active":"2024-04-17T10:26:15.024334+0000","last_became_peered":"2024-04-17T10:26:15.024334+0000","last_unstale":"2024-04-17T10:26:17.129156+0000","last_undegraded":"2024-04-17T10:26:17.129156+0000","last_fullsized":"2024-04-17T10:26:17.129156+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129653+0000","last_change":"2024-04-17T10:26:16.029809+0000","last_active":"2024-04-17T10:26:17.129653+0000","last_peered":"2024-04-17T10:26:17.129653+0000","last_clean":"2024-04-17T10:26:17.129653+0000","last_became_active":"2024-04-17T10:26:16.029556+0000","last_became_peered":"2024-04-17T10:26:16.029556+0000","last_unstale":"2024-04-17T10:26:17.129653+0000","last_undegraded":"2024-04-17T10:26:17.129653+0000","last_fullsized":"2024-04-17T10:26:17.129653+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:12:28.540793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129225+0000","last_change":"2024-04-17T10:26:15.025219+0000","last_active":"2024-04-17T10:26:17.129225+0000","last_peered":"2024-04-17T10:26:17.129225+0000","last_clean":"2024-04-17T10:26:17.129225+0000","last_became_active":"2024-04-17T10:26:15.024949+0000","last_became_peered":"2024-04-17T10:26:15.024949+0000","last_unstale":"2024-04-17T10:26:17.129225+0000","last_undegraded":"2024-04-17T10:26:17.129225+0000","last_fullsized":"2024-04-17T10:26:17.129225+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:40.848602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020112+0000","last_change":"2024-04-17T10:26:16.021248+0000","last_active":"2024-04-17T10:26:17.020112+0000","last_peered":"2024-04-17T10:26:17.020112+0000","last_clean":"2024-04-17T10:26:17.020112+0000","last_became_active":"2024-04-17T10:26:16.020951+0000","last_became_peered":"2024-04-17T10:26:16.020951+0000","last_unstale":"2024-04-17T10:26:17.020112+0000","last_undegraded":"2024-04-17T10:26:17.020112+0000","last_fullsized":"2024-04-17T10:26:17.020112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:34.289100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023631+0000","last_change":"2024-04-17T10:26:15.023890+0000","last_active":"2024-04-17T10:26:17.023631+0000","last_peered":"2024-04-17T10:26:17.023631+0000","last_clean":"2024-04-17T10:26:17.023631+0000","last_became_active":"2024-04-17T10:26:15.023658+0000","last_became_peered":"2024-04-17T10:26:15.023658+0000","last_unstale":"2024-04-17T10:26:17.023631+0000","last_undegraded":"2024-04-17T10:26:17.023631+0000","last_fullsized":"2024-04-17T10:26:17.023631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:33.282562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129304+0000","last_change":"2024-04-17T10:26:16.042733+0000","last_active":"2024-04-17T10:26:17.129304+0000","last_peered":"2024-04-17T10:26:17.129304+0000","last_clean":"2024-04-17T10:26:17.129304+0000","last_became_active":"2024-04-17T10:26:16.042443+0000","last_became_peered":"2024-04-17T10:26:16.042443+0000","last_unstale":"2024-04-17T10:26:17.129304+0000","last_undegraded":"2024-04-17T10:26:17.129304+0000","last_fullsized":"2024-04-17T10:26:17.129304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:32.972807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023562+0000","last_change":"2024-04-17T10:26:15.025596+0000","last_active":"2024-04-17T10:26:17.023562+0000","last_peered":"2024-04-17T10:26:17.023562+0000","last_clean":"2024-04-17T10:26:17.023562+0000","last_became_active":"2024-04-17T10:26:15.024806+0000","last_became_peered":"2024-04-17T10:26:15.024806+0000","last_unstale":"2024-04-17T10:26:17.023562+0000","last_undegraded":"2024-04-17T10:26:17.023562+0000","last_fullsized":"2024-04-17T10:26:17.023562+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:24.707986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023230+0000","last_change":"2024-04-17T10:26:16.030635+0000","last_active":"2024-04-17T10:26:17.023230+0000","last_peered":"2024-04-17T10:26:17.023230+0000","last_clean":"2024-04-17T10:26:17.023230+0000","last_became_active":"2024-04-17T10:26:16.030149+0000","last_became_peered":"2024-04-17T10:26:16.030149+0000","last_unstale":"2024-04-17T10:26:17.023230+0000","last_undegraded":"2024-04-17T10:26:17.023230+0000","last_fullsized":"2024-04-17T10:26:17.023230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:05:07.792336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021690+0000","last_change":"2024-04-17T10:26:15.029872+0000","last_active":"2024-04-17T10:26:17.021690+0000","last_peered":"2024-04-17T10:26:17.021690+0000","last_clean":"2024-04-17T10:26:17.021690+0000","last_became_active":"2024-04-17T10:26:15.029535+0000","last_became_peered":"2024-04-17T10:26:15.029535+0000","last_unstale":"2024-04-17T10:26:17.021690+0000","last_undegraded":"2024-04-17T10:26:17.021690+0000","last_fullsized":"2024-04-17T10:26:17.021690+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:33.378584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127634+0000","last_change":"2024-04-17T10:26:16.021498+0000","last_active":"2024-04-17T10:26:17.127634+0000","last_peered":"2024-04-17T10:26:17.127634+0000","last_clean":"2024-04-17T10:26:17.127634+0000","last_became_active":"2024-04-17T10:26:16.021206+0000","last_became_peered":"2024-04-17T10:26:16.021206+0000","last_unstale":"2024-04-17T10:26:17.127634+0000","last_undegraded":"2024-04-17T10:26:17.127634+0000","last_fullsized":"2024-04-17T10:26:17.127634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:18.186055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127547+0000","last_change":"2024-04-17T10:26:15.021680+0000","last_active":"2024-04-17T10:26:17.127547+0000","last_peered":"2024-04-17T10:26:17.127547+0000","last_clean":"2024-04-17T10:26:17.127547+0000","last_became_active":"2024-04-17T10:26:15.021333+0000","last_became_peered":"2024-04-17T10:26:15.021333+0000","last_unstale":"2024-04-17T10:26:17.127547+0000","last_undegraded":"2024-04-17T10:26:17.127547+0000","last_fullsized":"2024-04-17T10:26:17.127547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:37.917585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023347+0000","last_change":"2024-04-17T10:26:16.037275+0000","last_active":"2024-04-17T10:26:17.023347+0000","last_peered":"2024-04-17T10:26:17.023347+0000","last_clean":"2024-04-17T10:26:17.023347+0000","last_became_active":"2024-04-17T10:26:16.037009+0000","last_became_peered":"2024-04-17T10:26:16.037009+0000","last_unstale":"2024-04-17T10:26:17.023347+0000","last_undegraded":"2024-04-17T10:26:17.023347+0000","last_fullsized":"2024-04-17T10:26:17.023347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:43:19.406554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127231+0000","last_change":"2024-04-17T10:26:15.023024+0000","last_active":"2024-04-17T10:26:17.127231+0000","last_peered":"2024-04-17T10:26:17.127231+0000","last_clean":"2024-04-17T10:26:17.127231+0000","last_became_active":"2024-04-17T10:26:15.022759+0000","last_became_peered":"2024-04-17T10:26:15.022759+0000","last_unstale":"2024-04-17T10:26:17.127231+0000","last_undegraded":"2024-04-17T10:26:17.127231+0000","last_fullsized":"2024-04-17T10:26:17.127231+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023528+0000","last_change":"2024-04-17T10:26:16.031929+0000","last_active":"2024-04-17T10:26:17.023528+0000","last_peered":"2024-04-17T10:26:17.023528+0000","last_clean":"2024-04-17T10:26:17.023528+0000","last_became_active":"2024-04-17T10:26:16.031632+0000","last_became_peered":"2024-04-17T10:26:16.031632+0000","last_unstale":"2024-04-17T10:26:17.023528+0000","last_undegraded":"2024-04-17T10:26:17.023528+0000","last_fullsized":"2024-04-17T10:26:17.023528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:17.990861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128661+0000","last_change":"2024-04-17T10:26:15.029084+0000","last_active":"2024-04-17T10:26:17.128661+0000","last_peered":"2024-04-17T10:26:17.128661+0000","last_clean":"2024-04-17T10:26:17.128661+0000","last_became_active":"2024-04-17T10:26:15.028738+0000","last_became_peered":"2024-04-17T10:26:15.028738+0000","last_unstale":"2024-04-17T10:26:17.128661+0000","last_undegraded":"2024-04-17T10:26:17.128661+0000","last_fullsized":"2024-04-17T10:26:17.128661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:44:16.094451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129124+0000","last_change":"2024-04-17T10:26:16.042635+0000","last_active":"2024-04-17T10:26:17.129124+0000","last_peered":"2024-04-17T10:26:17.129124+0000","last_clean":"2024-04-17T10:26:17.129124+0000","last_became_active":"2024-04-17T10:26:16.041984+0000","last_became_peered":"2024-04-17T10:26:16.041984+0000","last_unstale":"2024-04-17T10:26:17.129124+0000","last_undegraded":"2024-04-17T10:26:17.129124+0000","last_fullsized":"2024-04-17T10:26:17.129124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:41.723249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020129+0000","last_change":"2024-04-17T10:26:15.024341+0000","last_active":"2024-04-17T10:26:17.020129+0000","last_peered":"2024-04-17T10:26:17.020129+0000","last_clean":"2024-04-17T10:26:17.020129+0000","last_became_active":"2024-04-17T10:26:15.023942+0000","last_became_peered":"2024-04-17T10:26:15.023942+0000","last_unstale":"2024-04-17T10:26:17.020129+0000","last_undegraded":"2024-04-17T10:26:17.020129+0000","last_fullsized":"2024-04-17T10:26:17.020129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:33.378584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020013+0000","last_change":"2024-04-17T10:26:16.037957+0000","last_active":"2024-04-17T10:26:17.020013+0000","last_peered":"2024-04-17T10:26:17.020013+0000","last_clean":"2024-04-17T10:26:17.020013+0000","last_became_active":"2024-04-17T10:26:16.037635+0000","last_became_peered":"2024-04-17T10:26:16.037635+0000","last_unstale":"2024-04-17T10:26:17.020013+0000","last_undegraded":"2024-04-17T10:26:17.020013+0000","last_fullsized":"2024-04-17T10:26:17.020013+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130130+0000","last_change":"2024-04-17T10:26:15.024041+0000","last_active":"2024-04-17T10:26:17.130130+0000","last_peered":"2024-04-17T10:26:17.130130+0000","last_clean":"2024-04-17T10:26:17.130130+0000","last_became_active":"2024-04-17T10:26:15.023764+0000","last_became_peered":"2024-04-17T10:26:15.023764+0000","last_unstale":"2024-04-17T10:26:17.130130+0000","last_undegraded":"2024-04-17T10:26:17.130130+0000","last_fullsized":"2024-04-17T10:26:17.130130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:12:10.234000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130327+0000","last_change":"2024-04-17T10:26:16.031381+0000","last_active":"2024-04-17T10:26:17.130327+0000","last_peered":"2024-04-17T10:26:17.130327+0000","last_clean":"2024-04-17T10:26:17.130327+0000","last_became_active":"2024-04-17T10:26:16.030975+0000","last_became_peered":"2024-04-17T10:26:16.030975+0000","last_unstale":"2024-04-17T10:26:17.130327+0000","last_undegraded":"2024-04-17T10:26:17.130327+0000","last_fullsized":"2024-04-17T10:26:17.130327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:41.722543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021882+0000","last_change":"2024-04-17T10:26:15.031919+0000","last_active":"2024-04-17T10:26:17.021882+0000","last_peered":"2024-04-17T10:26:17.021882+0000","last_clean":"2024-04-17T10:26:17.021882+0000","last_became_active":"2024-04-17T10:26:15.031610+0000","last_became_peered":"2024-04-17T10:26:15.031610+0000","last_unstale":"2024-04-17T10:26:17.021882+0000","last_undegraded":"2024-04-17T10:26:17.021882+0000","last_fullsized":"2024-04-17T10:26:17.021882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:44:16.094451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127368+0000","last_change":"2024-04-17T10:26:16.022774+0000","last_active":"2024-04-17T10:26:17.127368+0000","last_peered":"2024-04-17T10:26:17.127368+0000","last_clean":"2024-04-17T10:26:17.127368+0000","last_became_active":"2024-04-17T10:26:16.022523+0000","last_became_peered":"2024-04-17T10:26:16.022523+0000","last_unstale":"2024-04-17T10:26:17.127368+0000","last_undegraded":"2024-04-17T10:26:17.127368+0000","last_fullsized":"2024-04-17T10:26:17.127368+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:34.289100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130001+0000","last_change":"2024-04-17T10:26:15.022779+0000","last_active":"2024-04-17T10:26:17.130001+0000","last_peered":"2024-04-17T10:26:17.130001+0000","last_clean":"2024-04-17T10:26:17.130001+0000","last_became_active":"2024-04-17T10:26:15.022559+0000","last_became_peered":"2024-04-17T10:26:15.022559+0000","last_unstale":"2024-04-17T10:26:17.130001+0000","last_undegraded":"2024-04-17T10:26:17.130001+0000","last_fullsized":"2024-04-17T10:26:17.130001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:15.073761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129812+0000","last_change":"2024-04-17T10:26:16.024405+0000","last_active":"2024-04-17T10:26:17.129812+0000","last_peered":"2024-04-17T10:26:17.129812+0000","last_clean":"2024-04-17T10:26:17.129812+0000","last_became_active":"2024-04-17T10:26:16.024157+0000","last_became_peered":"2024-04-17T10:26:16.024157+0000","last_unstale":"2024-04-17T10:26:17.129812+0000","last_undegraded":"2024-04-17T10:26:17.129812+0000","last_fullsized":"2024-04-17T10:26:17.129812+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:29.545780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129834+0000","last_change":"2024-04-17T10:26:15.016970+0000","last_active":"2024-04-17T10:26:17.129834+0000","last_peered":"2024-04-17T10:26:17.129834+0000","last_clean":"2024-04-17T10:26:17.129834+0000","last_became_active":"2024-04-17T10:26:15.016647+0000","last_became_peered":"2024-04-17T10:26:15.016647+0000","last_unstale":"2024-04-17T10:26:17.129834+0000","last_undegraded":"2024-04-17T10:26:17.129834+0000","last_fullsized":"2024-04-17T10:26:17.129834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:53:00.093534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128359+0000","last_change":"2024-04-17T10:26:16.029523+0000","last_active":"2024-04-17T10:26:17.128359+0000","last_peered":"2024-04-17T10:26:17.128359+0000","last_clean":"2024-04-17T10:26:17.128359+0000","last_became_active":"2024-04-17T10:26:16.029274+0000","last_became_peered":"2024-04-17T10:26:16.029274+0000","last_unstale":"2024-04-17T10:26:17.128359+0000","last_undegraded":"2024-04-17T10:26:17.128359+0000","last_fullsized":"2024-04-17T10:26:17.128359+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.321256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127827+0000","last_change":"2024-04-17T10:26:15.024263+0000","last_active":"2024-04-17T10:26:17.127827+0000","last_peered":"2024-04-17T10:26:17.127827+0000","last_clean":"2024-04-17T10:26:17.127827+0000","last_became_active":"2024-04-17T10:26:15.023970+0000","last_became_peered":"2024-04-17T10:26:15.023970+0000","last_unstale":"2024-04-17T10:26:17.127827+0000","last_undegraded":"2024-04-17T10:26:17.127827+0000","last_fullsized":"2024-04-17T10:26:17.127827+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:22.083842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023491+0000","last_change":"2024-04-17T10:26:16.035718+0000","last_active":"2024-04-17T10:26:17.023491+0000","last_peered":"2024-04-17T10:26:17.023491+0000","last_clean":"2024-04-17T10:26:17.023491+0000","last_became_active":"2024-04-17T10:26:16.035440+0000","last_became_peered":"2024-04-17T10:26:16.035440+0000","last_unstale":"2024-04-17T10:26:17.023491+0000","last_undegraded":"2024-04-17T10:26:17.023491+0000","last_fullsized":"2024-04-17T10:26:17.023491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:07:53.620668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129620+0000","last_change":"2024-04-17T10:26:15.024351+0000","last_active":"2024-04-17T10:26:17.129620+0000","last_peered":"2024-04-17T10:26:17.129620+0000","last_clean":"2024-04-17T10:26:17.129620+0000","last_became_active":"2024-04-17T10:26:15.023926+0000","last_became_peered":"2024-04-17T10:26:15.023926+0000","last_unstale":"2024-04-17T10:26:17.129620+0000","last_undegraded":"2024-04-17T10:26:17.129620+0000","last_fullsized":"2024-04-17T10:26:17.129620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:39:14.994872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021703+0000","last_change":"2024-04-17T10:26:16.035712+0000","last_active":"2024-04-17T10:26:17.021703+0000","last_peered":"2024-04-17T10:26:17.021703+0000","last_clean":"2024-04-17T10:26:17.021703+0000","last_became_active":"2024-04-17T10:26:16.035401+0000","last_became_peered":"2024-04-17T10:26:16.035401+0000","last_unstale":"2024-04-17T10:26:17.021703+0000","last_undegraded":"2024-04-17T10:26:17.021703+0000","last_fullsized":"2024-04-17T10:26:17.021703+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:29.545780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128808+0000","last_change":"2024-04-17T10:26:15.024262+0000","last_active":"2024-04-17T10:26:17.128808+0000","last_peered":"2024-04-17T10:26:17.128808+0000","last_clean":"2024-04-17T10:26:17.128808+0000","last_became_active":"2024-04-17T10:26:15.023961+0000","last_became_peered":"2024-04-17T10:26:15.023961+0000","last_unstale":"2024-04-17T10:26:17.128808+0000","last_undegraded":"2024-04-17T10:26:17.128808+0000","last_fullsized":"2024-04-17T10:26:17.128808+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:33.378584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127918+0000","last_change":"2024-04-17T10:26:16.034983+0000","last_active":"2024-04-17T10:26:17.127918+0000","last_peered":"2024-04-17T10:26:17.127918+0000","last_clean":"2024-04-17T10:26:17.127918+0000","last_became_active":"2024-04-17T10:26:16.034774+0000","last_became_peered":"2024-04-17T10:26:16.034774+0000","last_unstale":"2024-04-17T10:26:17.127918+0000","last_undegraded":"2024-04-17T10:26:17.127918+0000","last_fullsized":"2024-04-17T10:26:17.127918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:07:18.673781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130255+0000","last_change":"2024-04-17T10:26:15.027630+0000","last_active":"2024-04-17T10:26:17.130255+0000","last_peered":"2024-04-17T10:26:17.130255+0000","last_clean":"2024-04-17T10:26:17.130255+0000","last_became_active":"2024-04-17T10:26:15.027316+0000","last_became_peered":"2024-04-17T10:26:15.027316+0000","last_unstale":"2024-04-17T10:26:17.130255+0000","last_undegraded":"2024-04-17T10:26:17.130255+0000","last_fullsized":"2024-04-17T10:26:17.130255+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:23.199598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023500+0000","last_change":"2024-04-17T10:26:16.037855+0000","last_active":"2024-04-17T10:26:17.023500+0000","last_peered":"2024-04-17T10:26:17.023500+0000","last_clean":"2024-04-17T10:26:17.023500+0000","last_became_active":"2024-04-17T10:26:16.037406+0000","last_became_peered":"2024-04-17T10:26:16.037406+0000","last_unstale":"2024-04-17T10:26:17.023500+0000","last_undegraded":"2024-04-17T10:26:17.023500+0000","last_fullsized":"2024-04-17T10:26:17.023500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:32.966599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129900+0000","last_change":"2024-04-17T10:26:15.030014+0000","last_active":"2024-04-17T10:26:17.129900+0000","last_peered":"2024-04-17T10:26:17.129900+0000","last_clean":"2024-04-17T10:26:17.129900+0000","last_became_active":"2024-04-17T10:26:15.029696+0000","last_became_peered":"2024-04-17T10:26:15.029696+0000","last_unstale":"2024-04-17T10:26:17.129900+0000","last_undegraded":"2024-04-17T10:26:17.129900+0000","last_fullsized":"2024-04-17T10:26:17.129900+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:01.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129782+0000","last_change":"2024-04-17T10:26:16.036209+0000","last_active":"2024-04-17T10:26:17.129782+0000","last_peered":"2024-04-17T10:26:17.129782+0000","last_clean":"2024-04-17T10:26:17.129782+0000","last_became_active":"2024-04-17T10:26:16.035931+0000","last_became_peered":"2024-04-17T10:26:16.035931+0000","last_unstale":"2024-04-17T10:26:17.129782+0000","last_undegraded":"2024-04-17T10:26:17.129782+0000","last_fullsized":"2024-04-17T10:26:17.129782+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:39.610854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129632+0000","last_change":"2024-04-17T10:26:15.023114+0000","last_active":"2024-04-17T10:26:17.129632+0000","last_peered":"2024-04-17T10:26:17.129632+0000","last_clean":"2024-04-17T10:26:17.129632+0000","last_became_active":"2024-04-17T10:26:15.022829+0000","last_became_peered":"2024-04-17T10:26:15.022829+0000","last_unstale":"2024-04-17T10:26:17.129632+0000","last_undegraded":"2024-04-17T10:26:17.129632+0000","last_fullsized":"2024-04-17T10:26:17.129632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:34.613035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021802+0000","last_change":"2024-04-17T10:26:16.043888+0000","last_active":"2024-04-17T10:26:17.021802+0000","last_peered":"2024-04-17T10:26:17.021802+0000","last_clean":"2024-04-17T10:26:17.021802+0000","last_became_active":"2024-04-17T10:26:16.043600+0000","last_became_peered":"2024-04-17T10:26:16.043600+0000","last_unstale":"2024-04-17T10:26:17.021802+0000","last_undegraded":"2024-04-17T10:26:17.021802+0000","last_fullsized":"2024-04-17T10:26:17.021802+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128056+0000","last_change":"2024-04-17T10:26:15.020460+0000","last_active":"2024-04-17T10:26:17.128056+0000","last_peered":"2024-04-17T10:26:17.128056+0000","last_clean":"2024-04-17T10:26:17.128056+0000","last_became_active":"2024-04-17T10:26:15.020183+0000","last_became_peered":"2024-04-17T10:26:15.020183+0000","last_unstale":"2024-04-17T10:26:17.128056+0000","last_undegraded":"2024-04-17T10:26:17.128056+0000","last_fullsized":"2024-04-17T10:26:17.128056+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:34.613035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020007+0000","last_change":"2024-04-17T10:26:16.029100+0000","last_active":"2024-04-17T10:26:17.020007+0000","last_peered":"2024-04-17T10:26:17.020007+0000","last_clean":"2024-04-17T10:26:17.020007+0000","last_became_active":"2024-04-17T10:26:16.028828+0000","last_became_peered":"2024-04-17T10:26:16.028828+0000","last_unstale":"2024-04-17T10:26:17.020007+0000","last_undegraded":"2024-04-17T10:26:17.020007+0000","last_fullsized":"2024-04-17T10:26:17.020007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:29.545780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.019428+0000","last_change":"2024-04-17T10:26:15.019637+0000","last_active":"2024-04-17T10:26:17.019428+0000","last_peered":"2024-04-17T10:26:17.019428+0000","last_clean":"2024-04-17T10:26:17.019428+0000","last_became_active":"2024-04-17T10:26:15.019383+0000","last_became_peered":"2024-04-17T10:26:15.019383+0000","last_unstale":"2024-04-17T10:26:17.019428+0000","last_undegraded":"2024-04-17T10:26:17.019428+0000","last_fullsized":"2024-04-17T10:26:17.019428+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:44:16.094451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128395+0000","last_change":"2024-04-17T10:26:16.027190+0000","last_active":"2024-04-17T10:26:17.128395+0000","last_peered":"2024-04-17T10:26:17.128395+0000","last_clean":"2024-04-17T10:26:17.128395+0000","last_became_active":"2024-04-17T10:26:16.026812+0000","last_became_peered":"2024-04-17T10:26:16.026812+0000","last_unstale":"2024-04-17T10:26:17.128395+0000","last_undegraded":"2024-04-17T10:26:17.128395+0000","last_fullsized":"2024-04-17T10:26:17.128395+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:28.752753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023056+0000","last_change":"2024-04-17T10:26:15.018797+0000","last_active":"2024-04-17T10:26:17.023056+0000","last_peered":"2024-04-17T10:26:17.023056+0000","last_clean":"2024-04-17T10:26:17.023056+0000","last_became_active":"2024-04-17T10:26:15.018534+0000","last_became_peered":"2024-04-17T10:26:15.018534+0000","last_unstale":"2024-04-17T10:26:17.023056+0000","last_undegraded":"2024-04-17T10:26:17.023056+0000","last_fullsized":"2024-04-17T10:26:17.023056+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:28.539242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128645+0000","last_change":"2024-04-17T10:26:16.038162+0000","last_active":"2024-04-17T10:26:17.128645+0000","last_peered":"2024-04-17T10:26:17.128645+0000","last_clean":"2024-04-17T10:26:17.128645+0000","last_became_active":"2024-04-17T10:26:16.037772+0000","last_became_peered":"2024-04-17T10:26:16.037772+0000","last_unstale":"2024-04-17T10:26:17.128645+0000","last_undegraded":"2024-04-17T10:26:17.128645+0000","last_fullsized":"2024-04-17T10:26:17.128645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:07:18.673781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128152+0000","last_change":"2024-04-17T10:26:15.027300+0000","last_active":"2024-04-17T10:26:17.128152+0000","last_peered":"2024-04-17T10:26:17.128152+0000","last_clean":"2024-04-17T10:26:17.128152+0000","last_became_active":"2024-04-17T10:26:15.026973+0000","last_became_peered":"2024-04-17T10:26:15.026973+0000","last_unstale":"2024-04-17T10:26:17.128152+0000","last_undegraded":"2024-04-17T10:26:17.128152+0000","last_fullsized":"2024-04-17T10:26:17.128152+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127311+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:26:17.127311+0000","last_peered":"2024-04-17T10:26:17.127311+0000","last_clean":"2024-04-17T10:26:17.127311+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:26:17.127311+0000","last_undegraded":"2024-04-17T10:26:17.127311+0000","last_fullsized":"2024-04-17T10:26:17.127311+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":782523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":191055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":197277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":191077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":198811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1036288,"data_stored":780875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:26:20.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:26:20.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.299+0000 7fda7b800640 1 Processor -- start 2024-04-17T10:26:20.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.299+0000 7fda7b800640 1 -- start start 2024-04-17T10:26:20.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda7b800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda740cef80 0x7fda74163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda7b800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda740b4970 0x7fda74166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda7b800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda740719a0 con 0x7fda74071310 2024-04-17T10:26:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda7b800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda74071c80 con 0x7fda740cef80 2024-04-17T10:26:20.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda7b800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda74162580 con 0x7fda740b4970 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda78e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda740cef80 0x7fda74163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda79800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda740b4970 0x7fda74166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda78e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda740cef80 0x7fda74163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:58760/0 (socket says 172.21.15.67:58760) 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda6be00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fda74071310 0x7fda740b4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:60700/0 (socket says 172.21.15.67:60700) 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda78e00640 1 -- 172.21.15.67:0/2211395684 learned_addr learned my addr 172.21.15.67:0/2211395684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda73e00640 1 -- 172.21.15.67:0/2211395684 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571516842 0 0) 0x7fda740719a0 con 0x7fda74071310 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda79800640 1 -- 172.21.15.67:0/2211395684 >> v2:172.21.15.86:3300/0 conn(0x7fda740cef80 msgr2=0x7fda74163060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda79800640 1 --2- 172.21.15.67:0/2211395684 >> v2:172.21.15.86:3300/0 conn(0x7fda740cef80 0x7fda74163060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda79800640 1 -- 172.21.15.67:0/2211395684 >> v1:172.21.15.86:6789/0 conn(0x7fda74071310 legacy=0x7fda740b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda79800640 1 -- 172.21.15.67:0/2211395684 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74162860 con 0x7fda740b4970 2024-04-17T10:26:20.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.303+0000 7fda78e00640 1 --2- 172.21.15.67:0/2211395684 >> v2:172.21.15.86:3300/0 conn(0x7fda740cef80 0x7fda74163060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:20.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda79800640 1 --2- 172.21.15.67:0/2211395684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda740b4970 0x7fda74166430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009d90 tx=0x7fda6c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce079ef1748daf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:20.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda73e00640 1 -- 172.21.15.67:0/2211395684 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0b2070 con 0x7fda740b4970 2024-04-17T10:26:20.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda73e00640 1 -- 172.21.15.67:0/2211395684 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda6c0a5830 con 0x7fda740b4970 2024-04-17T10:26:20.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda73e00640 1 -- 172.21.15.67:0/2211395684 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6c0a5c00 con 0x7fda740b4970 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 -- 172.21.15.67:0/2211395684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda740b4970 msgr2=0x7fda74166430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 --2- 172.21.15.67:0/2211395684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda740b4970 0x7fda74166430 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009d90 tx=0x7fda6c01c970 comp rx=0 tx=0).stop 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 -- 172.21.15.67:0/2211395684 shutdown_connections 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 --2- 172.21.15.67:0/2211395684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda740b4970 0x7fda74166430 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 --2- 172.21.15.67:0/2211395684 >> v2:172.21.15.86:3300/0 conn(0x7fda740cef80 0x7fda74163060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 -- 172.21.15.67:0/2211395684 >> 172.21.15.67:0/2211395684 conn(0x7fda7406ac90 msgr2=0x7fda7406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 -- 172.21.15.67:0/2211395684 shutdown_connections 2024-04-17T10:26:20.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 -- 172.21.15.67:0/2211395684 wait complete. 2024-04-17T10:26:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.307+0000 7fda7b800640 1 Processor -- start 2024-04-17T10:26:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 -- start start 2024-04-17T10:26:20.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda74071310 0x7fda74155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740b4970 0x7fda74154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda740cef80 0x7fda74159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda740b3310 con 0x7fda740b4970 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda740b3190 con 0x7fda740cef80 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda740b3490 con 0x7fda74071310 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740b4970 0x7fda74154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda78e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda74071310 0x7fda74155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740b4970 0x7fda74154c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32800/0 (socket says 172.21.15.67:32800) 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda78e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda74071310 0x7fda74155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:37634/0 (socket says 172.21.15.67:37634) 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 -- 172.21.15.67:0/1666219575 learned_addr learned my addr 172.21.15.67:0/1666219575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda79800640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda740cef80 0x7fda74159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 -- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda74071310 msgr2=0x7fda74155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda74071310 0x7fda74155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 -- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda740cef80 msgr2=0x7fda74159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda740cef80 0x7fda74159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 -- 172.21.15.67:0/1666219575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda74159c50 con 0x7fda740b4970 2024-04-17T10:26:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda78e00640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda74071310 0x7fda74155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:20.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda6be00640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740b4970 0x7fda74154c90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00c410 tx=0x7fda5c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:20.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda71600640 1 -- 172.21.15.67:0/1666219575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c0074e0 con 0x7fda740b4970 2024-04-17T10:26:20.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda71600640 1 -- 172.21.15.67:0/1666219575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda5c0047f0 con 0x7fda740b4970 2024-04-17T10:26:20.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda71600640 1 -- 172.21.15.67:0/1666219575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5c007680 con 0x7fda740b4970 2024-04-17T10:26:20.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda74159f30 con 0x7fda740b4970 2024-04-17T10:26:20.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7415f200 con 0x7fda740b4970 2024-04-17T10:26:20.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.311+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda74072d50 con 0x7fda740b4970 2024-04-17T10:26:20.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.315+0000 7fda71600640 1 -- 172.21.15.67:0/1666219575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fda5c011070 con 0x7fda740b4970 2024-04-17T10:26:20.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.315+0000 7fda71600640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda50066f60 0x7fda50069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.315+0000 7fda71600640 1 -- 172.21.15.67:0/1666219575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fda5c084db0 con 0x7fda740b4970 2024-04-17T10:26:20.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.315+0000 7fda78e00640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda50066f60 0x7fda50069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.315+0000 7fda78e00640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda50066f60 0x7fda50069420 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda640096f0 tx=0x7fda64009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:20.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.319+0000 7fda71600640 1 -- 172.21.15.67:0/1666219575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fda5c089070 con 0x7fda740b4970 2024-04-17T10:26:20.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.475+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fda74074e30 con 0x7fda50066f60 2024-04-17T10:26:20.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.483+0000 7fda71600640 1 -- 172.21.15.67:0/1666219575 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251956 (secure 0 0 0) 0x7fda74074e30 con 0x7fda50066f60 2024-04-17T10:26:20.479 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:20.482 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:26:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda50066f60 msgr2=0x7fda50069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda50066f60 0x7fda50069420 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fda640096f0 tx=0x7fda64009240 comp rx=0 tx=0).stop 2024-04-17T10:26:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740b4970 msgr2=0x7fda74154c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740b4970 0x7fda74154c90 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fda5c00c410 tx=0x7fda5c00c8d0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 shutdown_connections 2024-04-17T10:26:20.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda50066f60 0x7fda50069420 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda740cef80 0x7fda74159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda740b4970 0x7fda74154c90 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 --2- 172.21.15.67:0/1666219575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda74071310 0x7fda74155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 >> 172.21.15.67:0/1666219575 conn(0x7fda7406ac90 msgr2=0x7fda74162f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:20.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 shutdown_connections 2024-04-17T10:26:20.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.487+0000 7fda7b800640 1 -- 172.21.15.67:0/1666219575 wait complete. 2024-04-17T10:26:20.501 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-04-17T10:26:20.154133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":120,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220864,"kb_used_data":4684,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753856,"statfs":{"total":773094113280,"available":772867948544,"internally_reserved":0,"allocated":4796416,"data_stored":2729416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001030"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023976+0000","last_change":"2024-04-17T10:26:16.030068+0000","last_active":"2024-04-17T10:26:17.023976+0000","last_peered":"2024-04-17T10:26:17.023976+0000","last_clean":"2024-04-17T10:26:17.023976+0000","last_became_active":"2024-04-17T10:26:16.029727+0000","last_became_peered":"2024-04-17T10:26:16.029727+0000","last_unstale":"2024-04-17T10:26:17.023976+0000","last_undegraded":"2024-04-17T10:26:17.023976+0000","last_fullsized":"2024-04-17T10:26:17.023976+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:13.009368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023438+0000","last_change":"2024-04-17T10:26:16.037613+0000","last_active":"2024-04-17T10:26:17.023438+0000","last_peered":"2024-04-17T10:26:17.023438+0000","last_clean":"2024-04-17T10:26:17.023438+0000","last_became_active":"2024-04-17T10:26:16.033247+0000","last_became_peered":"2024-04-17T10:26:16.033247+0000","last_unstale":"2024-04-17T10:26:17.023438+0000","last_undegraded":"2024-04-17T10:26:17.023438+0000","last_fullsized":"2024-04-17T10:26:17.023438+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:26:00.713000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129796+0000","last_change":"2024-04-17T10:26:16.043989+0000","last_active":"2024-04-17T10:26:17.129796+0000","last_peered":"2024-04-17T10:26:17.129796+0000","last_clean":"2024-04-17T10:26:17.129796+0000","last_became_active":"2024-04-17T10:26:16.043707+0000","last_became_peered":"2024-04-17T10:26:16.043707+0000","last_unstale":"2024-04-17T10:26:17.129796+0000","last_undegraded":"2024-04-17T10:26:17.129796+0000","last_fullsized":"2024-04-17T10:26:17.129796+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:33.205243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130120+0000","last_change":"2024-04-17T10:26:16.021991+0000","last_active":"2024-04-17T10:26:17.130120+0000","last_peered":"2024-04-17T10:26:17.130120+0000","last_clean":"2024-04-17T10:26:17.130120+0000","last_became_active":"2024-04-17T10:26:16.019681+0000","last_became_peered":"2024-04-17T10:26:16.019681+0000","last_unstale":"2024-04-17T10:26:17.130120+0000","last_undegraded":"2024-04-17T10:26:17.130120+0000","last_fullsized":"2024-04-17T10:26:17.130120+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:38.924123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130060+0000","last_change":"2024-04-17T10:26:16.022947+0000","last_active":"2024-04-17T10:26:17.130060+0000","last_peered":"2024-04-17T10:26:17.130060+0000","last_clean":"2024-04-17T10:26:17.130060+0000","last_became_active":"2024-04-17T10:26:16.022391+0000","last_became_peered":"2024-04-17T10:26:16.022391+0000","last_unstale":"2024-04-17T10:26:17.130060+0000","last_undegraded":"2024-04-17T10:26:17.130060+0000","last_fullsized":"2024-04-17T10:26:17.130060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:41:35.936279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021654+0000","last_change":"2024-04-17T10:26:16.039470+0000","last_active":"2024-04-17T10:26:17.021654+0000","last_peered":"2024-04-17T10:26:17.021654+0000","last_clean":"2024-04-17T10:26:17.021654+0000","last_became_active":"2024-04-17T10:26:16.039208+0000","last_became_peered":"2024-04-17T10:26:16.039208+0000","last_unstale":"2024-04-17T10:26:17.021654+0000","last_undegraded":"2024-04-17T10:26:17.021654+0000","last_fullsized":"2024-04-17T10:26:17.021654+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:33.205243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128144+0000","last_change":"2024-04-17T10:26:16.036158+0000","last_active":"2024-04-17T10:26:17.128144+0000","last_peered":"2024-04-17T10:26:17.128144+0000","last_clean":"2024-04-17T10:26:17.128144+0000","last_became_active":"2024-04-17T10:26:16.035850+0000","last_became_peered":"2024-04-17T10:26:16.035850+0000","last_unstale":"2024-04-17T10:26:17.128144+0000","last_undegraded":"2024-04-17T10:26:17.128144+0000","last_fullsized":"2024-04-17T10:26:17.128144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:34.924217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022561+0000","last_change":"2024-04-17T10:26:16.033952+0000","last_active":"2024-04-17T10:26:17.022561+0000","last_peered":"2024-04-17T10:26:17.022561+0000","last_clean":"2024-04-17T10:26:17.022561+0000","last_became_active":"2024-04-17T10:26:16.033680+0000","last_became_peered":"2024-04-17T10:26:16.033680+0000","last_unstale":"2024-04-17T10:26:17.022561+0000","last_undegraded":"2024-04-17T10:26:17.022561+0000","last_fullsized":"2024-04-17T10:26:17.022561+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:41.723249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129932+0000","last_change":"2024-04-17T10:26:16.045038+0000","last_active":"2024-04-17T10:26:17.129932+0000","last_peered":"2024-04-17T10:26:17.129932+0000","last_clean":"2024-04-17T10:26:17.129932+0000","last_became_active":"2024-04-17T10:26:16.044811+0000","last_became_peered":"2024-04-17T10:26:16.044811+0000","last_unstale":"2024-04-17T10:26:17.129932+0000","last_undegraded":"2024-04-17T10:26:17.129932+0000","last_fullsized":"2024-04-17T10:26:17.129932+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128770+0000","last_change":"2024-04-17T10:26:16.042109+0000","last_active":"2024-04-17T10:26:17.128770+0000","last_peered":"2024-04-17T10:26:17.128770+0000","last_clean":"2024-04-17T10:26:17.128770+0000","last_became_active":"2024-04-17T10:26:16.041786+0000","last_became_peered":"2024-04-17T10:26:16.041786+0000","last_unstale":"2024-04-17T10:26:17.128770+0000","last_undegraded":"2024-04-17T10:26:17.128770+0000","last_fullsized":"2024-04-17T10:26:17.128770+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:28.752753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128937+0000","last_change":"2024-04-17T10:26:16.031145+0000","last_active":"2024-04-17T10:26:17.128937+0000","last_peered":"2024-04-17T10:26:17.128937+0000","last_clean":"2024-04-17T10:26:17.128937+0000","last_became_active":"2024-04-17T10:26:16.030879+0000","last_became_peered":"2024-04-17T10:26:16.030879+0000","last_unstale":"2024-04-17T10:26:17.128937+0000","last_undegraded":"2024-04-17T10:26:17.128937+0000","last_fullsized":"2024-04-17T10:26:17.128937+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:21.100099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129604+0000","last_change":"2024-04-17T10:26:16.024291+0000","last_active":"2024-04-17T10:26:17.129604+0000","last_peered":"2024-04-17T10:26:17.129604+0000","last_clean":"2024-04-17T10:26:17.129604+0000","last_became_active":"2024-04-17T10:26:16.023880+0000","last_became_peered":"2024-04-17T10:26:16.023880+0000","last_unstale":"2024-04-17T10:26:17.129604+0000","last_undegraded":"2024-04-17T10:26:17.129604+0000","last_fullsized":"2024-04-17T10:26:17.129604+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:24:48.055839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129069+0000","last_change":"2024-04-17T10:26:16.042987+0000","last_active":"2024-04-17T10:26:17.129069+0000","last_peered":"2024-04-17T10:26:17.129069+0000","last_clean":"2024-04-17T10:26:17.129069+0000","last_became_active":"2024-04-17T10:26:16.041681+0000","last_became_peered":"2024-04-17T10:26:16.041681+0000","last_unstale":"2024-04-17T10:26:17.129069+0000","last_undegraded":"2024-04-17T10:26:17.129069+0000","last_fullsized":"2024-04-17T10:26:17.129069+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:52.680433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020007+0000","last_change":"2024-04-17T10:26:16.025657+0000","last_active":"2024-04-17T10:26:17.020007+0000","last_peered":"2024-04-17T10:26:17.020007+0000","last_clean":"2024-04-17T10:26:17.020007+0000","last_became_active":"2024-04-17T10:26:16.025140+0000","last_became_peered":"2024-04-17T10:26:16.025140+0000","last_unstale":"2024-04-17T10:26:17.020007+0000","last_undegraded":"2024-04-17T10:26:17.020007+0000","last_fullsized":"2024-04-17T10:26:17.020007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:29:33.205243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128352+0000","last_change":"2024-04-17T10:26:16.022415+0000","last_active":"2024-04-17T10:26:17.128352+0000","last_peered":"2024-04-17T10:26:17.128352+0000","last_clean":"2024-04-17T10:26:17.128352+0000","last_became_active":"2024-04-17T10:26:16.022034+0000","last_became_peered":"2024-04-17T10:26:16.022034+0000","last_unstale":"2024-04-17T10:26:17.128352+0000","last_undegraded":"2024-04-17T10:26:17.128352+0000","last_fullsized":"2024-04-17T10:26:17.128352+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:18.186055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127197+0000","last_change":"2024-04-17T10:26:16.034353+0000","last_active":"2024-04-17T10:26:17.127197+0000","last_peered":"2024-04-17T10:26:17.127197+0000","last_clean":"2024-04-17T10:26:17.127197+0000","last_became_active":"2024-04-17T10:26:16.034087+0000","last_became_peered":"2024-04-17T10:26:16.034087+0000","last_unstale":"2024-04-17T10:26:17.127197+0000","last_undegraded":"2024-04-17T10:26:17.127197+0000","last_fullsized":"2024-04-17T10:26:17.127197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128144+0000","last_change":"2024-04-17T10:26:16.034324+0000","last_active":"2024-04-17T10:26:17.128144+0000","last_peered":"2024-04-17T10:26:17.128144+0000","last_clean":"2024-04-17T10:26:17.128144+0000","last_became_active":"2024-04-17T10:26:16.033864+0000","last_became_peered":"2024-04-17T10:26:16.033864+0000","last_unstale":"2024-04-17T10:26:17.128144+0000","last_undegraded":"2024-04-17T10:26:17.128144+0000","last_fullsized":"2024-04-17T10:26:17.128144+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:32:59.576436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129441+0000","last_change":"2024-04-17T10:26:16.017782+0000","last_active":"2024-04-17T10:26:17.129441+0000","last_peered":"2024-04-17T10:26:17.129441+0000","last_clean":"2024-04-17T10:26:17.129441+0000","last_became_active":"2024-04-17T10:26:16.017547+0000","last_became_peered":"2024-04-17T10:26:16.017547+0000","last_unstale":"2024-04-17T10:26:17.129441+0000","last_undegraded":"2024-04-17T10:26:17.129441+0000","last_fullsized":"2024-04-17T10:26:17.129441+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:24:48.055839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130145+0000","last_change":"2024-04-17T10:26:16.033700+0000","last_active":"2024-04-17T10:26:17.130145+0000","last_peered":"2024-04-17T10:26:17.130145+0000","last_clean":"2024-04-17T10:26:17.130145+0000","last_became_active":"2024-04-17T10:26:16.033446+0000","last_became_peered":"2024-04-17T10:26:16.033446+0000","last_unstale":"2024-04-17T10:26:17.130145+0000","last_undegraded":"2024-04-17T10:26:17.130145+0000","last_fullsized":"2024-04-17T10:26:17.130145+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:35.619573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128867+0000","last_change":"2024-04-17T10:26:16.020772+0000","last_active":"2024-04-17T10:26:17.128867+0000","last_peered":"2024-04-17T10:26:17.128867+0000","last_clean":"2024-04-17T10:26:17.128867+0000","last_became_active":"2024-04-17T10:26:16.020479+0000","last_became_peered":"2024-04-17T10:26:16.020479+0000","last_unstale":"2024-04-17T10:26:17.128867+0000","last_undegraded":"2024-04-17T10:26:17.128867+0000","last_fullsized":"2024-04-17T10:26:17.128867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:05:01.722709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129011+0000","last_change":"2024-04-17T10:26:16.042143+0000","last_active":"2024-04-17T10:26:17.129011+0000","last_peered":"2024-04-17T10:26:17.129011+0000","last_clean":"2024-04-17T10:26:17.129011+0000","last_became_active":"2024-04-17T10:26:16.041653+0000","last_became_peered":"2024-04-17T10:26:16.041653+0000","last_unstale":"2024-04-17T10:26:17.129011+0000","last_undegraded":"2024-04-17T10:26:17.129011+0000","last_fullsized":"2024-04-17T10:26:17.129011+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:20.516273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020208+0000","last_change":"2024-04-17T10:26:16.025404+0000","last_active":"2024-04-17T10:26:17.020208+0000","last_peered":"2024-04-17T10:26:17.020208+0000","last_clean":"2024-04-17T10:26:17.020208+0000","last_became_active":"2024-04-17T10:26:16.024589+0000","last_became_peered":"2024-04-17T10:26:16.024589+0000","last_unstale":"2024-04-17T10:26:17.020208+0000","last_undegraded":"2024-04-17T10:26:17.020208+0000","last_fullsized":"2024-04-17T10:26:17.020208+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:34.924217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127674+0000","last_change":"2024-04-17T10:26:16.019330+0000","last_active":"2024-04-17T10:26:17.127674+0000","last_peered":"2024-04-17T10:26:17.127674+0000","last_clean":"2024-04-17T10:26:17.127674+0000","last_became_active":"2024-04-17T10:26:16.019027+0000","last_became_peered":"2024-04-17T10:26:16.019027+0000","last_unstale":"2024-04-17T10:26:17.127674+0000","last_undegraded":"2024-04-17T10:26:17.127674+0000","last_fullsized":"2024-04-17T10:26:17.127674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.321256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129818+0000","last_change":"2024-04-17T10:26:16.022468+0000","last_active":"2024-04-17T10:26:17.129818+0000","last_peered":"2024-04-17T10:26:17.129818+0000","last_clean":"2024-04-17T10:26:17.129818+0000","last_became_active":"2024-04-17T10:26:16.022174+0000","last_became_peered":"2024-04-17T10:26:16.022174+0000","last_unstale":"2024-04-17T10:26:17.129818+0000","last_undegraded":"2024-04-17T10:26:17.129818+0000","last_fullsized":"2024-04-17T10:26:17.129818+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:17.422739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129415+0000","last_change":"2024-04-17T10:26:16.035114+0000","last_active":"2024-04-17T10:26:17.129415+0000","last_peered":"2024-04-17T10:26:17.129415+0000","last_clean":"2024-04-17T10:26:17.129415+0000","last_became_active":"2024-04-17T10:26:16.034619+0000","last_became_peered":"2024-04-17T10:26:16.034619+0000","last_unstale":"2024-04-17T10:26:17.129415+0000","last_undegraded":"2024-04-17T10:26:17.129415+0000","last_fullsized":"2024-04-17T10:26:17.129415+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:32.972807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022434+0000","last_change":"2024-04-17T10:26:16.047228+0000","last_active":"2024-04-17T10:26:17.022434+0000","last_peered":"2024-04-17T10:26:17.022434+0000","last_clean":"2024-04-17T10:26:17.022434+0000","last_became_active":"2024-04-17T10:26:16.046864+0000","last_became_peered":"2024-04-17T10:26:16.046864+0000","last_unstale":"2024-04-17T10:26:17.022434+0000","last_undegraded":"2024-04-17T10:26:17.022434+0000","last_fullsized":"2024-04-17T10:26:17.022434+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.789876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129327+0000","last_change":"2024-04-17T10:26:16.015586+0000","last_active":"2024-04-17T10:26:17.129327+0000","last_peered":"2024-04-17T10:26:17.129327+0000","last_clean":"2024-04-17T10:26:17.129327+0000","last_became_active":"2024-04-17T10:26:16.013966+0000","last_became_peered":"2024-04-17T10:26:16.013966+0000","last_unstale":"2024-04-17T10:26:17.129327+0000","last_undegraded":"2024-04-17T10:26:17.129327+0000","last_fullsized":"2024-04-17T10:26:17.129327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:40.218457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128684+0000","last_change":"2024-04-17T10:26:16.037153+0000","last_active":"2024-04-17T10:26:17.128684+0000","last_peered":"2024-04-17T10:26:17.128684+0000","last_clean":"2024-04-17T10:26:17.128684+0000","last_became_active":"2024-04-17T10:26:16.036876+0000","last_became_peered":"2024-04-17T10:26:16.036876+0000","last_unstale":"2024-04-17T10:26:17.128684+0000","last_undegraded":"2024-04-17T10:26:17.128684+0000","last_fullsized":"2024-04-17T10:26:17.128684+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:32:59.576436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128761+0000","last_change":"2024-04-17T10:26:16.039129+0000","last_active":"2024-04-17T10:26:17.128761+0000","last_peered":"2024-04-17T10:26:17.128761+0000","last_clean":"2024-04-17T10:26:17.128761+0000","last_became_active":"2024-04-17T10:26:16.038832+0000","last_became_peered":"2024-04-17T10:26:16.038832+0000","last_unstale":"2024-04-17T10:26:17.128761+0000","last_undegraded":"2024-04-17T10:26:17.128761+0000","last_fullsized":"2024-04-17T10:26:17.128761+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:41.855139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127939+0000","last_change":"2024-04-17T10:26:16.029526+0000","last_active":"2024-04-17T10:26:17.127939+0000","last_peered":"2024-04-17T10:26:17.127939+0000","last_clean":"2024-04-17T10:26:17.127939+0000","last_became_active":"2024-04-17T10:26:16.028988+0000","last_became_peered":"2024-04-17T10:26:16.028988+0000","last_unstale":"2024-04-17T10:26:17.127939+0000","last_undegraded":"2024-04-17T10:26:17.127939+0000","last_fullsized":"2024-04-17T10:26:17.127939+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:34.385122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127629+0000","last_change":"2024-04-17T10:26:16.025147+0000","last_active":"2024-04-17T10:26:17.127629+0000","last_peered":"2024-04-17T10:26:17.127629+0000","last_clean":"2024-04-17T10:26:17.127629+0000","last_became_active":"2024-04-17T10:26:16.024610+0000","last_became_peered":"2024-04-17T10:26:16.024610+0000","last_unstale":"2024-04-17T10:26:17.127629+0000","last_undegraded":"2024-04-17T10:26:17.127629+0000","last_fullsized":"2024-04-17T10:26:17.127629+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:34.924217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021529+0000","last_change":"2024-04-17T10:26:16.032556+0000","last_active":"2024-04-17T10:26:17.021529+0000","last_peered":"2024-04-17T10:26:17.021529+0000","last_clean":"2024-04-17T10:26:17.021529+0000","last_became_active":"2024-04-17T10:26:16.032268+0000","last_became_peered":"2024-04-17T10:26:16.032268+0000","last_unstale":"2024-04-17T10:26:17.021529+0000","last_undegraded":"2024-04-17T10:26:17.021529+0000","last_fullsized":"2024-04-17T10:26:17.021529+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:39.610854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127590+0000","last_change":"2024-04-17T10:26:16.034205+0000","last_active":"2024-04-17T10:26:17.127590+0000","last_peered":"2024-04-17T10:26:17.127590+0000","last_clean":"2024-04-17T10:26:17.127590+0000","last_became_active":"2024-04-17T10:26:16.033881+0000","last_became_peered":"2024-04-17T10:26:16.033881+0000","last_unstale":"2024-04-17T10:26:17.127590+0000","last_undegraded":"2024-04-17T10:26:17.127590+0000","last_fullsized":"2024-04-17T10:26:17.127590+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.789876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129454+0000","last_change":"2024-04-17T10:26:15.021792+0000","last_active":"2024-04-17T10:26:17.129454+0000","last_peered":"2024-04-17T10:26:17.129454+0000","last_clean":"2024-04-17T10:26:17.129454+0000","last_became_active":"2024-04-17T10:26:15.021421+0000","last_became_peered":"2024-04-17T10:26:15.021421+0000","last_unstale":"2024-04-17T10:26:17.129454+0000","last_undegraded":"2024-04-17T10:26:17.129454+0000","last_fullsized":"2024-04-17T10:26:17.129454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:01.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022319+0000","last_change":"2024-04-17T10:26:16.034650+0000","last_active":"2024-04-17T10:26:17.022319+0000","last_peered":"2024-04-17T10:26:17.022319+0000","last_clean":"2024-04-17T10:26:17.022319+0000","last_became_active":"2024-04-17T10:26:16.034250+0000","last_became_peered":"2024-04-17T10:26:16.034250+0000","last_unstale":"2024-04-17T10:26:17.022319+0000","last_undegraded":"2024-04-17T10:26:17.022319+0000","last_fullsized":"2024-04-17T10:26:17.022319+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:34.289100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129888+0000","last_change":"2024-04-17T10:26:15.024021+0000","last_active":"2024-04-17T10:26:17.129888+0000","last_peered":"2024-04-17T10:26:17.129888+0000","last_clean":"2024-04-17T10:26:17.129888+0000","last_became_active":"2024-04-17T10:26:15.023550+0000","last_became_peered":"2024-04-17T10:26:15.023550+0000","last_unstale":"2024-04-17T10:26:17.129888+0000","last_undegraded":"2024-04-17T10:26:17.129888+0000","last_fullsized":"2024-04-17T10:26:17.129888+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:40.716005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020194+0000","last_change":"2024-04-17T10:26:16.021799+0000","last_active":"2024-04-17T10:26:17.020194+0000","last_peered":"2024-04-17T10:26:17.020194+0000","last_clean":"2024-04-17T10:26:17.020194+0000","last_became_active":"2024-04-17T10:26:16.021514+0000","last_became_peered":"2024-04-17T10:26:16.021514+0000","last_unstale":"2024-04-17T10:26:17.020194+0000","last_undegraded":"2024-04-17T10:26:17.020194+0000","last_fullsized":"2024-04-17T10:26:17.020194+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:25.789876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.022361+0000","last_change":"2024-04-17T10:26:15.032062+0000","last_active":"2024-04-17T10:26:17.022361+0000","last_peered":"2024-04-17T10:26:17.022361+0000","last_clean":"2024-04-17T10:26:17.022361+0000","last_became_active":"2024-04-17T10:26:15.031663+0000","last_became_peered":"2024-04-17T10:26:15.031663+0000","last_unstale":"2024-04-17T10:26:17.022361+0000","last_undegraded":"2024-04-17T10:26:17.022361+0000","last_fullsized":"2024-04-17T10:26:17.022361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023991+0000","last_change":"2024-04-17T10:26:16.043046+0000","last_active":"2024-04-17T10:26:17.023991+0000","last_peered":"2024-04-17T10:26:17.023991+0000","last_clean":"2024-04-17T10:26:17.023991+0000","last_became_active":"2024-04-17T10:26:16.042810+0000","last_became_peered":"2024-04-17T10:26:16.042810+0000","last_unstale":"2024-04-17T10:26:17.023991+0000","last_undegraded":"2024-04-17T10:26:17.023991+0000","last_fullsized":"2024-04-17T10:26:17.023991+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:59.825316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.019923+0000","last_change":"2024-04-17T10:26:15.014570+0000","last_active":"2024-04-17T10:26:17.019923+0000","last_peered":"2024-04-17T10:26:17.019923+0000","last_clean":"2024-04-17T10:26:17.019923+0000","last_became_active":"2024-04-17T10:26:15.014304+0000","last_became_peered":"2024-04-17T10:26:15.014304+0000","last_unstale":"2024-04-17T10:26:17.019923+0000","last_undegraded":"2024-04-17T10:26:17.019923+0000","last_fullsized":"2024-04-17T10:26:17.019923+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129296+0000","last_change":"2024-04-17T10:26:16.044841+0000","last_active":"2024-04-17T10:26:17.129296+0000","last_peered":"2024-04-17T10:26:17.129296+0000","last_clean":"2024-04-17T10:26:17.129296+0000","last_became_active":"2024-04-17T10:26:16.044587+0000","last_became_peered":"2024-04-17T10:26:16.044587+0000","last_unstale":"2024-04-17T10:26:17.129296+0000","last_undegraded":"2024-04-17T10:26:17.129296+0000","last_fullsized":"2024-04-17T10:26:17.129296+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:40.218457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129168+0000","last_change":"2024-04-17T10:26:15.026920+0000","last_active":"2024-04-17T10:26:17.129168+0000","last_peered":"2024-04-17T10:26:17.129168+0000","last_clean":"2024-04-17T10:26:17.129168+0000","last_became_active":"2024-04-17T10:26:15.026278+0000","last_became_peered":"2024-04-17T10:26:15.026278+0000","last_unstale":"2024-04-17T10:26:17.129168+0000","last_undegraded":"2024-04-17T10:26:17.129168+0000","last_fullsized":"2024-04-17T10:26:17.129168+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128542+0000","last_change":"2024-04-17T10:26:16.042948+0000","last_active":"2024-04-17T10:26:17.128542+0000","last_peered":"2024-04-17T10:26:17.128542+0000","last_clean":"2024-04-17T10:26:17.128542+0000","last_became_active":"2024-04-17T10:26:16.042667+0000","last_became_peered":"2024-04-17T10:26:16.042667+0000","last_unstale":"2024-04-17T10:26:17.128542+0000","last_undegraded":"2024-04-17T10:26:17.128542+0000","last_fullsized":"2024-04-17T10:26:17.128542+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:03.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127725+0000","last_change":"2024-04-17T10:26:15.030585+0000","last_active":"2024-04-17T10:26:17.127725+0000","last_peered":"2024-04-17T10:26:17.127725+0000","last_clean":"2024-04-17T10:26:17.127725+0000","last_became_active":"2024-04-17T10:26:15.030322+0000","last_became_peered":"2024-04-17T10:26:15.030322+0000","last_unstale":"2024-04-17T10:26:17.127725+0000","last_undegraded":"2024-04-17T10:26:17.127725+0000","last_fullsized":"2024-04-17T10:26:17.127725+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:12:32.979655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020263+0000","last_change":"2024-04-17T10:26:16.030615+0000","last_active":"2024-04-17T10:26:17.020263+0000","last_peered":"2024-04-17T10:26:17.020263+0000","last_clean":"2024-04-17T10:26:17.020263+0000","last_became_active":"2024-04-17T10:26:16.030317+0000","last_became_peered":"2024-04-17T10:26:16.030317+0000","last_unstale":"2024-04-17T10:26:17.020263+0000","last_undegraded":"2024-04-17T10:26:17.020263+0000","last_fullsized":"2024-04-17T10:26:17.020263+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:18.186055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021854+0000","last_change":"2024-04-17T10:26:15.029862+0000","last_active":"2024-04-17T10:26:17.021854+0000","last_peered":"2024-04-17T10:26:17.021854+0000","last_clean":"2024-04-17T10:26:17.021854+0000","last_became_active":"2024-04-17T10:26:15.029546+0000","last_became_peered":"2024-04-17T10:26:15.029546+0000","last_unstale":"2024-04-17T10:26:17.021854+0000","last_undegraded":"2024-04-17T10:26:17.021854+0000","last_fullsized":"2024-04-17T10:26:17.021854+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023446+0000","last_change":"2024-04-17T10:26:16.035881+0000","last_active":"2024-04-17T10:26:17.023446+0000","last_peered":"2024-04-17T10:26:17.023446+0000","last_clean":"2024-04-17T10:26:17.023446+0000","last_became_active":"2024-04-17T10:26:16.035556+0000","last_became_peered":"2024-04-17T10:26:16.035556+0000","last_unstale":"2024-04-17T10:26:17.023446+0000","last_undegraded":"2024-04-17T10:26:17.023446+0000","last_fullsized":"2024-04-17T10:26:17.023446+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:15.886315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.019302+0000","last_change":"2024-04-17T10:26:15.022158+0000","last_active":"2024-04-17T10:26:17.019302+0000","last_peered":"2024-04-17T10:26:17.019302+0000","last_clean":"2024-04-17T10:26:17.019302+0000","last_became_active":"2024-04-17T10:26:15.021932+0000","last_became_peered":"2024-04-17T10:26:15.021932+0000","last_unstale":"2024-04-17T10:26:17.019302+0000","last_undegraded":"2024-04-17T10:26:17.019302+0000","last_fullsized":"2024-04-17T10:26:17.019302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130170+0000","last_change":"2024-04-17T10:26:16.022122+0000","last_active":"2024-04-17T10:26:17.130170+0000","last_peered":"2024-04-17T10:26:17.130170+0000","last_clean":"2024-04-17T10:26:17.130170+0000","last_became_active":"2024-04-17T10:26:16.019272+0000","last_became_peered":"2024-04-17T10:26:16.019272+0000","last_unstale":"2024-04-17T10:26:17.130170+0000","last_undegraded":"2024-04-17T10:26:17.130170+0000","last_fullsized":"2024-04-17T10:26:17.130170+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:02.272505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.024026+0000","last_change":"2024-04-17T10:26:15.022167+0000","last_active":"2024-04-17T10:26:17.024026+0000","last_peered":"2024-04-17T10:26:17.024026+0000","last_clean":"2024-04-17T10:26:17.024026+0000","last_became_active":"2024-04-17T10:26:15.021764+0000","last_became_peered":"2024-04-17T10:26:15.021764+0000","last_unstale":"2024-04-17T10:26:17.024026+0000","last_undegraded":"2024-04-17T10:26:17.024026+0000","last_fullsized":"2024-04-17T10:26:17.024026+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:24.783338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127892+0000","last_change":"2024-04-17T10:26:16.027057+0000","last_active":"2024-04-17T10:26:17.127892+0000","last_peered":"2024-04-17T10:26:17.127892+0000","last_clean":"2024-04-17T10:26:17.127892+0000","last_became_active":"2024-04-17T10:26:16.026783+0000","last_became_peered":"2024-04-17T10:26:16.026783+0000","last_unstale":"2024-04-17T10:26:17.127892+0000","last_undegraded":"2024-04-17T10:26:17.127892+0000","last_fullsized":"2024-04-17T10:26:17.127892+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:03.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129156+0000","last_change":"2024-04-17T10:26:15.025321+0000","last_active":"2024-04-17T10:26:17.129156+0000","last_peered":"2024-04-17T10:26:17.129156+0000","last_clean":"2024-04-17T10:26:17.129156+0000","last_became_active":"2024-04-17T10:26:15.024334+0000","last_became_peered":"2024-04-17T10:26:15.024334+0000","last_unstale":"2024-04-17T10:26:17.129156+0000","last_undegraded":"2024-04-17T10:26:17.129156+0000","last_fullsized":"2024-04-17T10:26:17.129156+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129653+0000","last_change":"2024-04-17T10:26:16.029809+0000","last_active":"2024-04-17T10:26:17.129653+0000","last_peered":"2024-04-17T10:26:17.129653+0000","last_clean":"2024-04-17T10:26:17.129653+0000","last_became_active":"2024-04-17T10:26:16.029556+0000","last_became_peered":"2024-04-17T10:26:16.029556+0000","last_unstale":"2024-04-17T10:26:17.129653+0000","last_undegraded":"2024-04-17T10:26:17.129653+0000","last_fullsized":"2024-04-17T10:26:17.129653+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:12:28.540793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129225+0000","last_change":"2024-04-17T10:26:15.025219+0000","last_active":"2024-04-17T10:26:17.129225+0000","last_peered":"2024-04-17T10:26:17.129225+0000","last_clean":"2024-04-17T10:26:17.129225+0000","last_became_active":"2024-04-17T10:26:15.024949+0000","last_became_peered":"2024-04-17T10:26:15.024949+0000","last_unstale":"2024-04-17T10:26:17.129225+0000","last_undegraded":"2024-04-17T10:26:17.129225+0000","last_fullsized":"2024-04-17T10:26:17.129225+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:40.848602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020112+0000","last_change":"2024-04-17T10:26:16.021248+0000","last_active":"2024-04-17T10:26:17.020112+0000","last_peered":"2024-04-17T10:26:17.020112+0000","last_clean":"2024-04-17T10:26:17.020112+0000","last_became_active":"2024-04-17T10:26:16.020951+0000","last_became_peered":"2024-04-17T10:26:16.020951+0000","last_unstale":"2024-04-17T10:26:17.020112+0000","last_undegraded":"2024-04-17T10:26:17.020112+0000","last_fullsized":"2024-04-17T10:26:17.020112+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:34.289100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023631+0000","last_change":"2024-04-17T10:26:15.023890+0000","last_active":"2024-04-17T10:26:17.023631+0000","last_peered":"2024-04-17T10:26:17.023631+0000","last_clean":"2024-04-17T10:26:17.023631+0000","last_became_active":"2024-04-17T10:26:15.023658+0000","last_became_peered":"2024-04-17T10:26:15.023658+0000","last_unstale":"2024-04-17T10:26:17.023631+0000","last_undegraded":"2024-04-17T10:26:17.023631+0000","last_fullsized":"2024-04-17T10:26:17.023631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:33.282562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129304+0000","last_change":"2024-04-17T10:26:16.042733+0000","last_active":"2024-04-17T10:26:17.129304+0000","last_peered":"2024-04-17T10:26:17.129304+0000","last_clean":"2024-04-17T10:26:17.129304+0000","last_became_active":"2024-04-17T10:26:16.042443+0000","last_became_peered":"2024-04-17T10:26:16.042443+0000","last_unstale":"2024-04-17T10:26:17.129304+0000","last_undegraded":"2024-04-17T10:26:17.129304+0000","last_fullsized":"2024-04-17T10:26:17.129304+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:32.972807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023562+0000","last_change":"2024-04-17T10:26:15.025596+0000","last_active":"2024-04-17T10:26:17.023562+0000","last_peered":"2024-04-17T10:26:17.023562+0000","last_clean":"2024-04-17T10:26:17.023562+0000","last_became_active":"2024-04-17T10:26:15.024806+0000","last_became_peered":"2024-04-17T10:26:15.024806+0000","last_unstale":"2024-04-17T10:26:17.023562+0000","last_undegraded":"2024-04-17T10:26:17.023562+0000","last_fullsized":"2024-04-17T10:26:17.023562+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:24.707986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023230+0000","last_change":"2024-04-17T10:26:16.030635+0000","last_active":"2024-04-17T10:26:17.023230+0000","last_peered":"2024-04-17T10:26:17.023230+0000","last_clean":"2024-04-17T10:26:17.023230+0000","last_became_active":"2024-04-17T10:26:16.030149+0000","last_became_peered":"2024-04-17T10:26:16.030149+0000","last_unstale":"2024-04-17T10:26:17.023230+0000","last_undegraded":"2024-04-17T10:26:17.023230+0000","last_fullsized":"2024-04-17T10:26:17.023230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:05:07.792336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021690+0000","last_change":"2024-04-17T10:26:15.029872+0000","last_active":"2024-04-17T10:26:17.021690+0000","last_peered":"2024-04-17T10:26:17.021690+0000","last_clean":"2024-04-17T10:26:17.021690+0000","last_became_active":"2024-04-17T10:26:15.029535+0000","last_became_peered":"2024-04-17T10:26:15.029535+0000","last_unstale":"2024-04-17T10:26:17.021690+0000","last_undegraded":"2024-04-17T10:26:17.021690+0000","last_fullsized":"2024-04-17T10:26:17.021690+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:33.378584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127634+0000","last_change":"2024-04-17T10:26:16.021498+0000","last_active":"2024-04-17T10:26:17.127634+0000","last_peered":"2024-04-17T10:26:17.127634+0000","last_clean":"2024-04-17T10:26:17.127634+0000","last_became_active":"2024-04-17T10:26:16.021206+0000","last_became_peered":"2024-04-17T10:26:16.021206+0000","last_unstale":"2024-04-17T10:26:17.127634+0000","last_undegraded":"2024-04-17T10:26:17.127634+0000","last_fullsized":"2024-04-17T10:26:17.127634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:18.186055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127547+0000","last_change":"2024-04-17T10:26:15.021680+0000","last_active":"2024-04-17T10:26:17.127547+0000","last_peered":"2024-04-17T10:26:17.127547+0000","last_clean":"2024-04-17T10:26:17.127547+0000","last_became_active":"2024-04-17T10:26:15.021333+0000","last_became_peered":"2024-04-17T10:26:15.021333+0000","last_unstale":"2024-04-17T10:26:17.127547+0000","last_undegraded":"2024-04-17T10:26:17.127547+0000","last_fullsized":"2024-04-17T10:26:17.127547+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:15:37.917585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023347+0000","last_change":"2024-04-17T10:26:16.037275+0000","last_active":"2024-04-17T10:26:17.023347+0000","last_peered":"2024-04-17T10:26:17.023347+0000","last_clean":"2024-04-17T10:26:17.023347+0000","last_became_active":"2024-04-17T10:26:16.037009+0000","last_became_peered":"2024-04-17T10:26:16.037009+0000","last_unstale":"2024-04-17T10:26:17.023347+0000","last_undegraded":"2024-04-17T10:26:17.023347+0000","last_fullsized":"2024-04-17T10:26:17.023347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:43:19.406554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127231+0000","last_change":"2024-04-17T10:26:15.023024+0000","last_active":"2024-04-17T10:26:17.127231+0000","last_peered":"2024-04-17T10:26:17.127231+0000","last_clean":"2024-04-17T10:26:17.127231+0000","last_became_active":"2024-04-17T10:26:15.022759+0000","last_became_peered":"2024-04-17T10:26:15.022759+0000","last_unstale":"2024-04-17T10:26:17.127231+0000","last_undegraded":"2024-04-17T10:26:17.127231+0000","last_fullsized":"2024-04-17T10:26:17.127231+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:50:48.203979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023528+0000","last_change":"2024-04-17T10:26:16.031929+0000","last_active":"2024-04-17T10:26:17.023528+0000","last_peered":"2024-04-17T10:26:17.023528+0000","last_clean":"2024-04-17T10:26:17.023528+0000","last_became_active":"2024-04-17T10:26:16.031632+0000","last_became_peered":"2024-04-17T10:26:16.031632+0000","last_unstale":"2024-04-17T10:26:17.023528+0000","last_undegraded":"2024-04-17T10:26:17.023528+0000","last_fullsized":"2024-04-17T10:26:17.023528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:23:17.990861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128661+0000","last_change":"2024-04-17T10:26:15.029084+0000","last_active":"2024-04-17T10:26:17.128661+0000","last_peered":"2024-04-17T10:26:17.128661+0000","last_clean":"2024-04-17T10:26:17.128661+0000","last_became_active":"2024-04-17T10:26:15.028738+0000","last_became_peered":"2024-04-17T10:26:15.028738+0000","last_unstale":"2024-04-17T10:26:17.128661+0000","last_undegraded":"2024-04-17T10:26:17.128661+0000","last_fullsized":"2024-04-17T10:26:17.128661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:44:16.094451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129124+0000","last_change":"2024-04-17T10:26:16.042635+0000","last_active":"2024-04-17T10:26:17.129124+0000","last_peered":"2024-04-17T10:26:17.129124+0000","last_clean":"2024-04-17T10:26:17.129124+0000","last_became_active":"2024-04-17T10:26:16.041984+0000","last_became_peered":"2024-04-17T10:26:16.041984+0000","last_unstale":"2024-04-17T10:26:17.129124+0000","last_undegraded":"2024-04-17T10:26:17.129124+0000","last_fullsized":"2024-04-17T10:26:17.129124+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:39:41.723249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020129+0000","last_change":"2024-04-17T10:26:15.024341+0000","last_active":"2024-04-17T10:26:17.020129+0000","last_peered":"2024-04-17T10:26:17.020129+0000","last_clean":"2024-04-17T10:26:17.020129+0000","last_became_active":"2024-04-17T10:26:15.023942+0000","last_became_peered":"2024-04-17T10:26:15.023942+0000","last_unstale":"2024-04-17T10:26:17.020129+0000","last_undegraded":"2024-04-17T10:26:17.020129+0000","last_fullsized":"2024-04-17T10:26:17.020129+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:33.378584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020013+0000","last_change":"2024-04-17T10:26:16.037957+0000","last_active":"2024-04-17T10:26:17.020013+0000","last_peered":"2024-04-17T10:26:17.020013+0000","last_clean":"2024-04-17T10:26:17.020013+0000","last_became_active":"2024-04-17T10:26:16.037635+0000","last_became_peered":"2024-04-17T10:26:16.037635+0000","last_unstale":"2024-04-17T10:26:17.020013+0000","last_undegraded":"2024-04-17T10:26:17.020013+0000","last_fullsized":"2024-04-17T10:26:17.020013+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130130+0000","last_change":"2024-04-17T10:26:15.024041+0000","last_active":"2024-04-17T10:26:17.130130+0000","last_peered":"2024-04-17T10:26:17.130130+0000","last_clean":"2024-04-17T10:26:17.130130+0000","last_became_active":"2024-04-17T10:26:15.023764+0000","last_became_peered":"2024-04-17T10:26:15.023764+0000","last_unstale":"2024-04-17T10:26:17.130130+0000","last_undegraded":"2024-04-17T10:26:17.130130+0000","last_fullsized":"2024-04-17T10:26:17.130130+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:12:10.234000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130327+0000","last_change":"2024-04-17T10:26:16.031381+0000","last_active":"2024-04-17T10:26:17.130327+0000","last_peered":"2024-04-17T10:26:17.130327+0000","last_clean":"2024-04-17T10:26:17.130327+0000","last_became_active":"2024-04-17T10:26:16.030975+0000","last_became_peered":"2024-04-17T10:26:16.030975+0000","last_unstale":"2024-04-17T10:26:17.130327+0000","last_undegraded":"2024-04-17T10:26:17.130327+0000","last_fullsized":"2024-04-17T10:26:17.130327+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:41.722543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021882+0000","last_change":"2024-04-17T10:26:15.031919+0000","last_active":"2024-04-17T10:26:17.021882+0000","last_peered":"2024-04-17T10:26:17.021882+0000","last_clean":"2024-04-17T10:26:17.021882+0000","last_became_active":"2024-04-17T10:26:15.031610+0000","last_became_peered":"2024-04-17T10:26:15.031610+0000","last_unstale":"2024-04-17T10:26:17.021882+0000","last_undegraded":"2024-04-17T10:26:17.021882+0000","last_fullsized":"2024-04-17T10:26:17.021882+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:44:16.094451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127368+0000","last_change":"2024-04-17T10:26:16.022774+0000","last_active":"2024-04-17T10:26:17.127368+0000","last_peered":"2024-04-17T10:26:17.127368+0000","last_clean":"2024-04-17T10:26:17.127368+0000","last_became_active":"2024-04-17T10:26:16.022523+0000","last_became_peered":"2024-04-17T10:26:16.022523+0000","last_unstale":"2024-04-17T10:26:17.127368+0000","last_undegraded":"2024-04-17T10:26:17.127368+0000","last_fullsized":"2024-04-17T10:26:17.127368+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:34.289100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130001+0000","last_change":"2024-04-17T10:26:15.022779+0000","last_active":"2024-04-17T10:26:17.130001+0000","last_peered":"2024-04-17T10:26:17.130001+0000","last_clean":"2024-04-17T10:26:17.130001+0000","last_became_active":"2024-04-17T10:26:15.022559+0000","last_became_peered":"2024-04-17T10:26:15.022559+0000","last_unstale":"2024-04-17T10:26:17.130001+0000","last_undegraded":"2024-04-17T10:26:17.130001+0000","last_fullsized":"2024-04-17T10:26:17.130001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:03:15.073761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129812+0000","last_change":"2024-04-17T10:26:16.024405+0000","last_active":"2024-04-17T10:26:17.129812+0000","last_peered":"2024-04-17T10:26:17.129812+0000","last_clean":"2024-04-17T10:26:17.129812+0000","last_became_active":"2024-04-17T10:26:16.024157+0000","last_became_peered":"2024-04-17T10:26:16.024157+0000","last_unstale":"2024-04-17T10:26:17.129812+0000","last_undegraded":"2024-04-17T10:26:17.129812+0000","last_fullsized":"2024-04-17T10:26:17.129812+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:29.545780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129834+0000","last_change":"2024-04-17T10:26:15.016970+0000","last_active":"2024-04-17T10:26:17.129834+0000","last_peered":"2024-04-17T10:26:17.129834+0000","last_clean":"2024-04-17T10:26:17.129834+0000","last_became_active":"2024-04-17T10:26:15.016647+0000","last_became_peered":"2024-04-17T10:26:15.016647+0000","last_unstale":"2024-04-17T10:26:17.129834+0000","last_undegraded":"2024-04-17T10:26:17.129834+0000","last_fullsized":"2024-04-17T10:26:17.129834+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:53:00.093534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128359+0000","last_change":"2024-04-17T10:26:16.029523+0000","last_active":"2024-04-17T10:26:17.128359+0000","last_peered":"2024-04-17T10:26:17.128359+0000","last_clean":"2024-04-17T10:26:17.128359+0000","last_became_active":"2024-04-17T10:26:16.029274+0000","last_became_peered":"2024-04-17T10:26:16.029274+0000","last_unstale":"2024-04-17T10:26:17.128359+0000","last_undegraded":"2024-04-17T10:26:17.128359+0000","last_fullsized":"2024-04-17T10:26:17.128359+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:03:44.321256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127827+0000","last_change":"2024-04-17T10:26:15.024263+0000","last_active":"2024-04-17T10:26:17.127827+0000","last_peered":"2024-04-17T10:26:17.127827+0000","last_clean":"2024-04-17T10:26:17.127827+0000","last_became_active":"2024-04-17T10:26:15.023970+0000","last_became_peered":"2024-04-17T10:26:15.023970+0000","last_unstale":"2024-04-17T10:26:17.127827+0000","last_undegraded":"2024-04-17T10:26:17.127827+0000","last_fullsized":"2024-04-17T10:26:17.127827+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:22.083842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023491+0000","last_change":"2024-04-17T10:26:16.035718+0000","last_active":"2024-04-17T10:26:17.023491+0000","last_peered":"2024-04-17T10:26:17.023491+0000","last_clean":"2024-04-17T10:26:17.023491+0000","last_became_active":"2024-04-17T10:26:16.035440+0000","last_became_peered":"2024-04-17T10:26:16.035440+0000","last_unstale":"2024-04-17T10:26:17.023491+0000","last_undegraded":"2024-04-17T10:26:17.023491+0000","last_fullsized":"2024-04-17T10:26:17.023491+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:07:53.620668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129620+0000","last_change":"2024-04-17T10:26:15.024351+0000","last_active":"2024-04-17T10:26:17.129620+0000","last_peered":"2024-04-17T10:26:17.129620+0000","last_clean":"2024-04-17T10:26:17.129620+0000","last_became_active":"2024-04-17T10:26:15.023926+0000","last_became_peered":"2024-04-17T10:26:15.023926+0000","last_unstale":"2024-04-17T10:26:17.129620+0000","last_undegraded":"2024-04-17T10:26:17.129620+0000","last_fullsized":"2024-04-17T10:26:17.129620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:39:14.994872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021703+0000","last_change":"2024-04-17T10:26:16.035712+0000","last_active":"2024-04-17T10:26:17.021703+0000","last_peered":"2024-04-17T10:26:17.021703+0000","last_clean":"2024-04-17T10:26:17.021703+0000","last_became_active":"2024-04-17T10:26:16.035401+0000","last_became_peered":"2024-04-17T10:26:16.035401+0000","last_unstale":"2024-04-17T10:26:17.021703+0000","last_undegraded":"2024-04-17T10:26:17.021703+0000","last_fullsized":"2024-04-17T10:26:17.021703+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:29.545780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128808+0000","last_change":"2024-04-17T10:26:15.024262+0000","last_active":"2024-04-17T10:26:17.128808+0000","last_peered":"2024-04-17T10:26:17.128808+0000","last_clean":"2024-04-17T10:26:17.128808+0000","last_became_active":"2024-04-17T10:26:15.023961+0000","last_became_peered":"2024-04-17T10:26:15.023961+0000","last_unstale":"2024-04-17T10:26:17.128808+0000","last_undegraded":"2024-04-17T10:26:17.128808+0000","last_fullsized":"2024-04-17T10:26:17.128808+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:33.378584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127918+0000","last_change":"2024-04-17T10:26:16.034983+0000","last_active":"2024-04-17T10:26:17.127918+0000","last_peered":"2024-04-17T10:26:17.127918+0000","last_clean":"2024-04-17T10:26:17.127918+0000","last_became_active":"2024-04-17T10:26:16.034774+0000","last_became_peered":"2024-04-17T10:26:16.034774+0000","last_unstale":"2024-04-17T10:26:17.127918+0000","last_undegraded":"2024-04-17T10:26:17.127918+0000","last_fullsized":"2024-04-17T10:26:17.127918+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:07:18.673781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.130255+0000","last_change":"2024-04-17T10:26:15.027630+0000","last_active":"2024-04-17T10:26:17.130255+0000","last_peered":"2024-04-17T10:26:17.130255+0000","last_clean":"2024-04-17T10:26:17.130255+0000","last_became_active":"2024-04-17T10:26:15.027316+0000","last_became_peered":"2024-04-17T10:26:15.027316+0000","last_unstale":"2024-04-17T10:26:17.130255+0000","last_undegraded":"2024-04-17T10:26:17.130255+0000","last_fullsized":"2024-04-17T10:26:17.130255+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:59:23.199598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023500+0000","last_change":"2024-04-17T10:26:16.037855+0000","last_active":"2024-04-17T10:26:17.023500+0000","last_peered":"2024-04-17T10:26:17.023500+0000","last_clean":"2024-04-17T10:26:17.023500+0000","last_became_active":"2024-04-17T10:26:16.037406+0000","last_became_peered":"2024-04-17T10:26:16.037406+0000","last_unstale":"2024-04-17T10:26:17.023500+0000","last_undegraded":"2024-04-17T10:26:17.023500+0000","last_fullsized":"2024-04-17T10:26:17.023500+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:32.966599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129900+0000","last_change":"2024-04-17T10:26:15.030014+0000","last_active":"2024-04-17T10:26:17.129900+0000","last_peered":"2024-04-17T10:26:17.129900+0000","last_clean":"2024-04-17T10:26:17.129900+0000","last_became_active":"2024-04-17T10:26:15.029696+0000","last_became_peered":"2024-04-17T10:26:15.029696+0000","last_unstale":"2024-04-17T10:26:17.129900+0000","last_undegraded":"2024-04-17T10:26:17.129900+0000","last_fullsized":"2024-04-17T10:26:17.129900+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:01.265968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129782+0000","last_change":"2024-04-17T10:26:16.036209+0000","last_active":"2024-04-17T10:26:17.129782+0000","last_peered":"2024-04-17T10:26:17.129782+0000","last_clean":"2024-04-17T10:26:17.129782+0000","last_became_active":"2024-04-17T10:26:16.035931+0000","last_became_peered":"2024-04-17T10:26:16.035931+0000","last_unstale":"2024-04-17T10:26:17.129782+0000","last_undegraded":"2024-04-17T10:26:17.129782+0000","last_fullsized":"2024-04-17T10:26:17.129782+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:30:39.610854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.129632+0000","last_change":"2024-04-17T10:26:15.023114+0000","last_active":"2024-04-17T10:26:17.129632+0000","last_peered":"2024-04-17T10:26:17.129632+0000","last_clean":"2024-04-17T10:26:17.129632+0000","last_became_active":"2024-04-17T10:26:15.022829+0000","last_became_peered":"2024-04-17T10:26:15.022829+0000","last_unstale":"2024-04-17T10:26:17.129632+0000","last_undegraded":"2024-04-17T10:26:17.129632+0000","last_fullsized":"2024-04-17T10:26:17.129632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:34.613035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.021802+0000","last_change":"2024-04-17T10:26:16.043888+0000","last_active":"2024-04-17T10:26:17.021802+0000","last_peered":"2024-04-17T10:26:17.021802+0000","last_clean":"2024-04-17T10:26:17.021802+0000","last_became_active":"2024-04-17T10:26:16.043600+0000","last_became_peered":"2024-04-17T10:26:16.043600+0000","last_unstale":"2024-04-17T10:26:17.021802+0000","last_undegraded":"2024-04-17T10:26:17.021802+0000","last_fullsized":"2024-04-17T10:26:17.021802+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:25.714523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128056+0000","last_change":"2024-04-17T10:26:15.020460+0000","last_active":"2024-04-17T10:26:17.128056+0000","last_peered":"2024-04-17T10:26:17.128056+0000","last_clean":"2024-04-17T10:26:17.128056+0000","last_became_active":"2024-04-17T10:26:15.020183+0000","last_became_peered":"2024-04-17T10:26:15.020183+0000","last_unstale":"2024-04-17T10:26:17.128056+0000","last_undegraded":"2024-04-17T10:26:17.128056+0000","last_fullsized":"2024-04-17T10:26:17.128056+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:34.613035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.020007+0000","last_change":"2024-04-17T10:26:16.029100+0000","last_active":"2024-04-17T10:26:17.020007+0000","last_peered":"2024-04-17T10:26:17.020007+0000","last_clean":"2024-04-17T10:26:17.020007+0000","last_became_active":"2024-04-17T10:26:16.028828+0000","last_became_peered":"2024-04-17T10:26:16.028828+0000","last_unstale":"2024-04-17T10:26:17.020007+0000","last_undegraded":"2024-04-17T10:26:17.020007+0000","last_fullsized":"2024-04-17T10:26:17.020007+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:29.545780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.019428+0000","last_change":"2024-04-17T10:26:15.019637+0000","last_active":"2024-04-17T10:26:17.019428+0000","last_peered":"2024-04-17T10:26:17.019428+0000","last_clean":"2024-04-17T10:26:17.019428+0000","last_became_active":"2024-04-17T10:26:15.019383+0000","last_became_peered":"2024-04-17T10:26:15.019383+0000","last_unstale":"2024-04-17T10:26:17.019428+0000","last_undegraded":"2024-04-17T10:26:17.019428+0000","last_fullsized":"2024-04-17T10:26:17.019428+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:44:16.094451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128395+0000","last_change":"2024-04-17T10:26:16.027190+0000","last_active":"2024-04-17T10:26:17.128395+0000","last_peered":"2024-04-17T10:26:17.128395+0000","last_clean":"2024-04-17T10:26:17.128395+0000","last_became_active":"2024-04-17T10:26:16.026812+0000","last_became_peered":"2024-04-17T10:26:16.026812+0000","last_unstale":"2024-04-17T10:26:17.128395+0000","last_undegraded":"2024-04-17T10:26:17.128395+0000","last_fullsized":"2024-04-17T10:26:17.128395+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:28.752753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.023056+0000","last_change":"2024-04-17T10:26:15.018797+0000","last_active":"2024-04-17T10:26:17.023056+0000","last_peered":"2024-04-17T10:26:17.023056+0000","last_clean":"2024-04-17T10:26:17.023056+0000","last_became_active":"2024-04-17T10:26:15.018534+0000","last_became_peered":"2024-04-17T10:26:15.018534+0000","last_unstale":"2024-04-17T10:26:17.023056+0000","last_undegraded":"2024-04-17T10:26:17.023056+0000","last_fullsized":"2024-04-17T10:26:17.023056+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:25:28.539242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128645+0000","last_change":"2024-04-17T10:26:16.038162+0000","last_active":"2024-04-17T10:26:17.128645+0000","last_peered":"2024-04-17T10:26:17.128645+0000","last_clean":"2024-04-17T10:26:17.128645+0000","last_became_active":"2024-04-17T10:26:16.037772+0000","last_became_peered":"2024-04-17T10:26:16.037772+0000","last_unstale":"2024-04-17T10:26:17.128645+0000","last_undegraded":"2024-04-17T10:26:17.128645+0000","last_fullsized":"2024-04-17T10:26:17.128645+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:14.996422+0000","last_clean_scrub_stamp":"2024-04-17T10:26:14.996422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:07:18.673781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.128152+0000","last_change":"2024-04-17T10:26:15.027300+0000","last_active":"2024-04-17T10:26:17.128152+0000","last_peered":"2024-04-17T10:26:17.128152+0000","last_clean":"2024-04-17T10:26:17.128152+0000","last_became_active":"2024-04-17T10:26:15.026973+0000","last_became_peered":"2024-04-17T10:26:15.026973+0000","last_unstale":"2024-04-17T10:26:17.128152+0000","last_undegraded":"2024-04-17T10:26:17.128152+0000","last_fullsized":"2024-04-17T10:26:17.128152+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:26:13.989884+0000","last_clean_scrub_stamp":"2024-04-17T10:26:13.989884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:46.243463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":204,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-17T10:26:17.127311+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:26:17.127311+0000","last_peered":"2024-04-17T10:26:17.127311+0000","last_clean":"2024-04-17T10:26:17.127311+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:26:17.127311+0000","last_undegraded":"2024-04-17T10:26:17.127311+0000","last_fullsized":"2024-04-17T10:26:17.127311+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343796,"statfs":{"total":96636764160,"available":96608047104,"internally_reserved":0,"allocated":1048576,"data_stored":782523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":191055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":458752,"data_stored":197277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":191077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":466944,"data_stored":198811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27468,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344372,"statfs":{"total":96636764160,"available":96608636928,"internally_reserved":0,"allocated":454656,"data_stored":197255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28040,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343800,"statfs":{"total":96636764160,"available":96608051200,"internally_reserved":0,"allocated":1036288,"data_stored":780875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:26:20.508 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:26:20.509 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:26:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 Processor -- start 2024-04-17T10:26:20.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 -- start start 2024-04-17T10:26:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6b640b0cd0 0x7f6b640cc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b640ce3b0 0x7f6b6406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b64161cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b640b1430 con 0x7f6b640ce3b0 2024-04-17T10:26:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b640b1680 con 0x7f6b640b0cd0 2024-04-17T10:26:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b640b1960 con 0x7f6b6406b1d0 2024-04-17T10:26:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b640ce3b0 0x7f6b6406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b640ce3b0 0x7f6b6406ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32814/0 (socket says 172.21.15.67:32814) 2024-04-17T10:26:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 -- 172.21.15.67:0/1046818931 learned_addr learned my addr 172.21.15.67:0/1046818931 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b6a000640 1 --2- 172.21.15.67:0/1046818931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b64161cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b69600640 1 --2- 172.21.15.67:0/1046818931 >> v2:172.21.15.86:3300/0 conn(0x7f6b640b0cd0 0x7f6b640cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 -- 172.21.15.67:0/1046818931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 msgr2=0x7f6b64161cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 --2- 172.21.15.67:0/1046818931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b64161cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 -- 172.21.15.67:0/1046818931 >> v2:172.21.15.86:3300/0 conn(0x7f6b640b0cd0 msgr2=0x7f6b640cc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 --2- 172.21.15.67:0/1046818931 >> v2:172.21.15.86:3300/0 conn(0x7f6b640b0cd0 0x7f6b640cc930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.707+0000 7f6b68c00640 1 -- 172.21.15.67:0/1046818931 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b640b1c40 con 0x7f6b640ce3b0 2024-04-17T10:26:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b68c00640 1 --2- 172.21.15.67:0/1046818931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b6406ac90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6b64068030 tx=0x7f6b5001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9fb8611588a588 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b63e00640 1 -- 172.21.15.67:0/1046818931 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b500b2070 con 0x7f6b640ce3b0 2024-04-17T10:26:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b63e00640 1 -- 172.21.15.67:0/1046818931 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b500a9070 con 0x7f6b640ce3b0 2024-04-17T10:26:20.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b63e00640 1 -- 172.21.15.67:0/1046818931 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b500ad880 con 0x7f6b640ce3b0 2024-04-17T10:26:20.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 -- 172.21.15.67:0/1046818931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 msgr2=0x7f6b6406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1046818931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b6406ac90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f6b64068030 tx=0x7f6b5001c490 comp rx=0 tx=0).stop 2024-04-17T10:26:20.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 -- 172.21.15.67:0/1046818931 shutdown_connections 2024-04-17T10:26:20.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1046818931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b64161cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1046818931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b6406ac90 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1046818931 >> v2:172.21.15.86:3300/0 conn(0x7f6b640b0cd0 0x7f6b640cc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 -- 172.21.15.67:0/1046818931 >> 172.21.15.67:0/1046818931 conn(0x7f6b640b22a0 msgr2=0x7f6b640b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 -- 172.21.15.67:0/1046818931 shutdown_connections 2024-04-17T10:26:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 -- 172.21.15.67:0/1046818931 wait complete. 2024-04-17T10:26:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.711+0000 7f6b6c000640 1 Processor -- start 2024-04-17T10:26:20.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 -- start start 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b640b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b640b0cd0 0x7f6b640aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b64156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b640cd970 con 0x7f6b640ce3b0 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6406b5e0 con 0x7f6b640b0cd0 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b640cdaf0 con 0x7f6b6406b1d0 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b64156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b64156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32824/0 (socket says 172.21.15.67:32824) 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 -- 172.21.15.67:0/1895175666 learned_addr learned my addr 172.21.15.67:0/1895175666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b68c00640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b640b0cd0 0x7f6b640aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b69600640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b640b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 -- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 msgr2=0x7f6b640b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b640b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 -- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b640b0cd0 msgr2=0x7f6b640aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b640b0cd0 0x7f6b640aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 -- 172.21.15.67:0/1895175666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b64157f60 con 0x7f6b640ce3b0 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6a000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b64156ff0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c007e60 tx=0x7f6b5c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b61600640 1 -- 172.21.15.67:0/1895175666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c014620 con 0x7f6b640ce3b0 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b61600640 1 -- 172.21.15.67:0/1895175666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c019070 con 0x7f6b640ce3b0 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b61600640 1 -- 172.21.15.67:0/1895175666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c014a20 con 0x7f6b640ce3b0 2024-04-17T10:26:20.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b64157c80 con 0x7f6b640ce3b0 2024-04-17T10:26:20.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b640b3710 con 0x7f6b640ce3b0 2024-04-17T10:26:20.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.715+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6404ef30 con 0x7f6b640ce3b0 2024-04-17T10:26:20.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.719+0000 7f6b61600640 1 -- 172.21.15.67:0/1895175666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6b5c014bc0 con 0x7f6b640ce3b0 2024-04-17T10:26:20.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.719+0000 7f6b61600640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b44066f60 0x7f6b44069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:20.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.719+0000 7f6b61600640 1 -- 172.21.15.67:0/1895175666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6b5c085990 con 0x7f6b640ce3b0 2024-04-17T10:26:20.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.719+0000 7f6b69600640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b44066f60 0x7f6b44069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:20.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.719+0000 7f6b69600640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b44066f60 0x7f6b44069420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6b58007a60 tx=0x7f6b58008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:20.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.723+0000 7f6b61600640 1 -- 172.21.15.67:0/1895175666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6b5c056120 con 0x7f6b640ce3b0 2024-04-17T10:26:20.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.943+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6b640b66d0 con 0x7f6b640ce3b0 2024-04-17T10:26:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b61600640 1 -- 172.21.15.67:0/1895175666 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f6b5c05a1c0 con 0x7f6b640ce3b0 2024-04-17T10:26:20.941 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:26:20.941 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:26:20.941 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:26:20.941 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:26:20.941 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:26:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b44066f60 msgr2=0x7f6b44069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b44066f60 0x7f6b44069420 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6b58007a60 tx=0x7f6b58008040 comp rx=0 tx=0).stop 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 msgr2=0x7f6b64156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b64156ff0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c007e60 tx=0x7f6b5c00e4b0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 shutdown_connections 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b44066f60 0x7f6b44069420 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b640ce3b0 0x7f6b64156ff0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b640b0cd0 0x7f6b640aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 --2- 172.21.15.67:0/1895175666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b6406b1d0 0x7f6b640b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 >> 172.21.15.67:0/1895175666 conn(0x7f6b640b22a0 msgr2=0x7f6b640ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 shutdown_connections 2024-04-17T10:26:20.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:20.947+0000 7f6b6c000640 1 -- 172.21.15.67:0/1895175666 wait complete. 2024-04-17T10:26:20.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:26:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.111+0000 7f7ac6c00640 1 Processor -- start 2024-04-17T10:26:21.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.111+0000 7f7ac6c00640 1 -- start start 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac6c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ac015dbc0 0x7f7ac006f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac6c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ac006db90 0x7f7ac0071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac6c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac00721a0 0x7f7ac006b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac6c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac015dfa0 con 0x7f7ac015dbc0 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac6c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac006faa0 con 0x7f7ac006db90 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac6c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac006ac90 con 0x7f7ac00721a0 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ac015dbc0 0x7f7ac006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ac015dbc0 0x7f7ac006f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32832/0 (socket says 172.21.15.67:32832) 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 -- 172.21.15.67:0/4070540473 learned_addr learned my addr 172.21.15.67:0/4070540473 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac4c00640 1 --2- 172.21.15.67:0/4070540473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac00721a0 0x7f7ac006b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ab7e00640 1 --2- 172.21.15.67:0/4070540473 >> v2:172.21.15.86:3300/0 conn(0x7f7ac006db90 0x7f7ac0071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 -- 172.21.15.67:0/4070540473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac00721a0 msgr2=0x7f7ac006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 --2- 172.21.15.67:0/4070540473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac00721a0 0x7f7ac006b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 -- 172.21.15.67:0/4070540473 >> v2:172.21.15.86:3300/0 conn(0x7f7ac006db90 msgr2=0x7f7ac0071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 --2- 172.21.15.67:0/4070540473 >> v2:172.21.15.86:3300/0 conn(0x7f7ac006db90 0x7f7ac0071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7abfe00640 1 -- 172.21.15.67:0/4070540473 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac00ad8f0 con 0x7f7ac015dbc0 2024-04-17T10:26:21.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.115+0000 7f7ac4c00640 1 --2- 172.21.15.67:0/4070540473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac00721a0 0x7f7ac006b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:21.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7abfe00640 1 --2- 172.21.15.67:0/4070540473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac006f560 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0071c10 tx=0x7f7ab001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f802e797301750 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7abf400640 1 -- 172.21.15.67:0/4070540473 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00b0070 con 0x7f7ac015dbc0 2024-04-17T10:26:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7abf400640 1 -- 172.21.15.67:0/4070540473 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab00ab600 con 0x7f7ac015dbc0 2024-04-17T10:26:21.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7abf400640 1 -- 172.21.15.67:0/4070540473 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab00ab9d0 con 0x7f7ac015dbc0 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- 172.21.15.67:0/4070540473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 msgr2=0x7f7ac006f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/4070540473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac006f560 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7ac0071c10 tx=0x7f7ab001c490 comp rx=0 tx=0).stop 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- 172.21.15.67:0/4070540473 shutdown_connections 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/4070540473 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac00721a0 0x7f7ac006b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/4070540473 >> v2:172.21.15.86:3300/0 conn(0x7f7ac006db90 0x7f7ac0071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/4070540473 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac006f560 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- 172.21.15.67:0/4070540473 >> 172.21.15.67:0/4070540473 conn(0x7f7ac00af670 msgr2=0x7f7ac00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- 172.21.15.67:0/4070540473 shutdown_connections 2024-04-17T10:26:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- 172.21.15.67:0/4070540473 wait complete. 2024-04-17T10:26:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 Processor -- start 2024-04-17T10:26:21.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- start start 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac006db90 0x7f7ac01449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ac00721a0 0x7f7ac0158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac0158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac006c2d0 con 0x7f7ac015dbc0 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac006c150 con 0x7f7ac00721a0 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.119+0000 7f7ac6c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac006c450 con 0x7f7ac006db90 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac0158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac006db90 0x7f7ac01449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac006db90 0x7f7ac01449c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:37670/0 (socket says 172.21.15.67:37670) 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abfe00640 1 -- 172.21.15.67:0/3962674133 learned_addr learned my addr 172.21.15.67:0/3962674133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:21.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac0158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32848/0 (socket says 172.21.15.67:32848) 2024-04-17T10:26:21.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ab7e00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ac00721a0 0x7f7ac0158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 -- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac006db90 msgr2=0x7f7ac01449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac006db90 0x7f7ac01449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 -- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ac00721a0 msgr2=0x7f7ac0158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ac00721a0 0x7f7ac0158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 -- 172.21.15.67:0/3962674133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac01595a0 con 0x7f7ac015dbc0 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abfe00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac006db90 0x7f7ac01449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac4c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac0158f70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8007e90 tx=0x7f7ab800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abcc00640 1 -- 172.21.15.67:0/3962674133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab801a070 con 0x7f7ac015dbc0 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abcc00640 1 -- 172.21.15.67:0/3962674133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab80039a0 con 0x7f7ac015dbc0 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abcc00640 1 -- 172.21.15.67:0/3962674133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8003d90 con 0x7f7ac015dbc0 2024-04-17T10:26:21.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac0159880 con 0x7f7ac015dbc0 2024-04-17T10:26:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac0159e40 con 0x7f7ac015dbc0 2024-04-17T10:26:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac009e260 con 0x7f7ac015dbc0 2024-04-17T10:26:21.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.123+0000 7f7abcc00640 1 -- 172.21.15.67:0/3962674133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7ab801d070 con 0x7f7ac015dbc0 2024-04-17T10:26:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.127+0000 7f7abcc00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a9c066f60 0x7f7a9c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.127+0000 7f7abcc00640 1 -- 172.21.15.67:0/3962674133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ab8085450 con 0x7f7ac015dbc0 2024-04-17T10:26:21.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.127+0000 7f7abfe00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a9c066f60 0x7f7a9c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.127+0000 7f7abfe00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a9c066f60 0x7f7a9c069420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ab001c4d0 tx=0x7f7ab00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.131+0000 7f7abcc00640 1 -- 172.21.15.67:0/3962674133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7ab8010070 con 0x7f7ac015dbc0 2024-04-17T10:26:21.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.327+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ac00b0660 con 0x7f7ac015dbc0 2024-04-17T10:26:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.327+0000 7f7abcc00640 1 -- 172.21.15.67:0/3962674133 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f7ab8055bf0 con 0x7f7ac015dbc0 2024-04-17T10:26:21.323 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:26:21.323 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1835906721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1835906721},{"type":"v1","addr":"172.21.15.86:6836","nonce":1835906721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2085506860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2085506860},{"type":"v1","addr":"172.21.15.86:6837","nonce":2085506860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2728147711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2728147711},{"type":"v1","addr":"172.21.15.67:6834","nonce":2728147711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:26:15.408252+0000","modified":"2024-04-17T10:26:18.413434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1409597842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1409597842},{"type":"v1","addr":"172.21.15.67:6835","nonce":1409597842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-17T10:26:21.324 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 18 2024-04-17T10:26:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a9c066f60 msgr2=0x7f7a9c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a9c066f60 0x7f7a9c069420 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7ab001c4d0 tx=0x7f7ab00a6000 comp rx=0 tx=0).stop 2024-04-17T10:26:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 msgr2=0x7f7ac0158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac0158f70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8007e90 tx=0x7f7ab800d4a0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 shutdown_connections 2024-04-17T10:26:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a9c066f60 0x7f7a9c069420 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ac015dbc0 0x7f7ac0158f70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ac00721a0 0x7f7ac0158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 --2- 172.21.15.67:0/3962674133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ac006db90 0x7f7ac01449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 >> 172.21.15.67:0/3962674133 conn(0x7f7ac00af670 msgr2=0x7f7ac00b45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 shutdown_connections 2024-04-17T10:26:21.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.331+0000 7f7ac6c00640 1 -- 172.21.15.67:0/3962674133 wait complete. 2024-04-17T10:26:21.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:26:15.408252+0000', 'modified': '2024-04-17T10:26:18.413434+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5283}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5283': {'gid': 5283, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/1409597842', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 1409597842}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 1409597842}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:26:21.747 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:26:21.747 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T10:26:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.895+0000 7f24c2c00640 1 Processor -- start 2024-04-17T10:26:21.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.895+0000 7f24c2c00640 1 -- start start 2024-04-17T10:26:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24c2c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc145040 0x7f24bc06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24c2c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24bc06e2f0 con 0x7f24bc06ef10 2024-04-17T10:26:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24c2c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24bc06d6e0 con 0x7f24bc071a20 2024-04-17T10:26:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24c2c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0a86c0 con 0x7f24bc145040 2024-04-17T10:26:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24b3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24bc06ef10 0x7f24bc071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35458/0 (socket says 172.21.15.67:35458) 2024-04-17T10:26:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24b3e00640 1 -- 172.21.15.67:0/279510604 learned_addr learned my addr 172.21.15.67:0/279510604 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24bbe00640 1 --2- 172.21.15.67:0/279510604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc145040 0x7f24bc06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24bb400640 1 -- 172.21.15.67:0/279510604 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093057513 0 0) 0x7f24bc06e2f0 con 0x7f24bc06ef10 2024-04-17T10:26:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24bb400640 1 -- 172.21.15.67:0/279510604 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24a0003530 con 0x7f24bc06ef10 2024-04-17T10:26:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24bbe00640 1 -- 172.21.15.67:0/279510604 >> v1:172.21.15.86:6789/0 conn(0x7f24bc071a20 legacy=0x7f24bc0cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24bbe00640 1 -- 172.21.15.67:0/279510604 >> v1:172.21.15.67:6789/0 conn(0x7f24bc06ef10 legacy=0x7f24bc071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.899+0000 7f24bbe00640 1 -- 172.21.15.67:0/279510604 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc06d9c0 con 0x7f24bc145040 2024-04-17T10:26:21.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24bbe00640 1 --2- 172.21.15.67:0/279510604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc145040 0x7f24bc06e9d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0097f0 tx=0x7f24ac01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=886fe18930c2a418 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24bb400640 1 -- 172.21.15.67:0/279510604 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0b5070 con 0x7f24bc145040 2024-04-17T10:26:21.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24bb400640 1 -- 172.21.15.67:0/279510604 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac0a89d0 con 0x7f24bc145040 2024-04-17T10:26:21.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24bb400640 1 -- 172.21.15.67:0/279510604 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0a8da0 con 0x7f24bc145040 2024-04-17T10:26:21.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 -- 172.21.15.67:0/279510604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc145040 msgr2=0x7f24bc06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 --2- 172.21.15.67:0/279510604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc145040 0x7f24bc06e9d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0097f0 tx=0x7f24ac01ecd0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 -- 172.21.15.67:0/279510604 shutdown_connections 2024-04-17T10:26:21.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 --2- 172.21.15.67:0/279510604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc145040 0x7f24bc06e9d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 -- 172.21.15.67:0/279510604 >> 172.21.15.67:0/279510604 conn(0x7f24bc06ac90 msgr2=0x7f24bc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 -- 172.21.15.67:0/279510604 shutdown_connections 2024-04-17T10:26:21.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 -- 172.21.15.67:0/279510604 wait complete. 2024-04-17T10:26:21.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 Processor -- start 2024-04-17T10:26:21.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.903+0000 7f24c2c00640 1 -- start start 2024-04-17T10:26:21.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24bc06ef10 0x7f24bc0b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc071a20 0x7f24bc15aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24bc145040 0x7f24bc1e5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc06df80 con 0x7f24bc145040 2024-04-17T10:26:21.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc15bf10 con 0x7f24bc06ef10 2024-04-17T10:26:21.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc06d370 con 0x7f24bc071a20 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24bc145040 0x7f24bc1e5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24bc145040 0x7f24bc1e5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32856/0 (socket says 172.21.15.67:32856) 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 -- 172.21.15.67:0/412103066 learned_addr learned my addr 172.21.15.67:0/412103066 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24bbe00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24bc06ef10 0x7f24bc0b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24b3e00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc071a20 0x7f24bc15aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 -- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc071a20 msgr2=0x7f24bc15aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc071a20 0x7f24bc15aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 -- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24bc06ef10 msgr2=0x7f24bc0b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24bc06ef10 0x7f24bc0b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 -- 172.21.15.67:0/412103066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc0b3420 con 0x7f24bc145040 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24bbe00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24bc06ef10 0x7f24bc0b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c0c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24bc145040 0x7f24bc1e5d60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f24b400c7a0 tx=0x7f24b400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24b8c00640 1 -- 172.21.15.67:0/412103066 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b4003b80 con 0x7f24bc145040 2024-04-17T10:26:21.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24b8c00640 1 -- 172.21.15.67:0/412103066 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b4003d20 con 0x7f24bc145040 2024-04-17T10:26:21.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24b8c00640 1 -- 172.21.15.67:0/412103066 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b400aa30 con 0x7f24bc145040 2024-04-17T10:26:21.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24bc0b3700 con 0x7f24bc145040 2024-04-17T10:26:21.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24bc0b3cc0 con 0x7f24bc145040 2024-04-17T10:26:21.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.907+0000 7f24b8c00640 1 -- 172.21.15.67:0/412103066 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f24b400abd0 con 0x7f24bc145040 2024-04-17T10:26:21.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24b8c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2498066f90 0x7f2498069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24b8c00640 1 -- 172.21.15.67:0/412103066 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24b4085160 con 0x7f24bc145040 2024-04-17T10:26:21.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2498066f90 0x7f2498069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 Processor -- start 2024-04-17T10:26:21.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 -- start start 2024-04-17T10:26:21.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2498066f90 0x7f2498069450 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0b2000 tx=0x7f24ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8b8ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8b449f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8b45310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb8a2c5f0 con 0x55ecb8b8e920 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb8993da0 con 0x55ecb8b8f240 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecb891c150 con 0x55ecb8b44f30 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8b8ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8b8ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32872/0 (socket says 172.21.15.67:32872) 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 -- 172.21.15.67:0/326102909 learned_addr learned my addr 172.21.15.67:0/326102909 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 -- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 msgr2=0x55ecb8b45310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24b3e00640 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8b449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c0c00640 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8b45310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8b45310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 -- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 msgr2=0x55ecb8b449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8b449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 -- 172.21.15.67:0/326102909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecb8935ca0 con 0x55ecb8b8e920 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24b3e00640 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8b449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24c0c00640 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8b45310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24bbe00640 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8b8ed00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0b3040 tx=0x7f24ac004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a4a28c42c5474d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24b0c00640 1 -- 172.21.15.67:0/326102909 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0cc070 con 0x55ecb8b8e920 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24b0c00640 1 -- 172.21.15.67:0/326102909 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac0b6040 con 0x55ecb8b8e920 2024-04-17T10:26:21.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.911+0000 7f24b0c00640 1 -- 172.21.15.67:0/326102909 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0b0780 con 0x55ecb8b8e920 2024-04-17T10:26:21.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- 172.21.15.67:0/326102909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 msgr2=0x55ecb8b8ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8b8ed00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0b3040 tx=0x7f24ac004740 comp rx=0 tx=0).stop 2024-04-17T10:26:21.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- 172.21.15.67:0/326102909 shutdown_connections 2024-04-17T10:26:21.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8b45310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8b449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 --2- 172.21.15.67:0/326102909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8b8ed00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- 172.21.15.67:0/326102909 >> 172.21.15.67:0/326102909 conn(0x55ecb8b4b2a0 msgr2=0x55ecb8b432d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- 172.21.15.67:0/326102909 shutdown_connections 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- 172.21.15.67:0/326102909 wait complete. 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- start start 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 Processor -- start 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b44f30 0x55ecb8b95c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b961c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8f240 0x55ecb8b48aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb894e990 con 0x55ecb8b8f240 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb898fe40 con 0x55ecb8b44f30 2024-04-17T10:26:21.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecb891f900 con 0x55ecb8b8e920 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 10 client.0 _ll_register_callbacks cb 0x55ecb8a471c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8f240 0x55ecb8b48aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8f240 0x55ecb8b48aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32886/0 (socket says 172.21.15.67:32886) 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 -- 172.21.15.67:0/2389015346 learned_addr learned my addr 172.21.15.67:0/2389015346 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24b3e00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24bbe00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b44f30 0x55ecb8b95c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 -- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 msgr2=0x55ecb8b961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 -- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b44f30 msgr2=0x55ecb8b95c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b44f30 0x55ecb8b95c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 -- 172.21.15.67:0/2389015346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecb894c630 con 0x55ecb8b8f240 2024-04-17T10:26:21.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24bbe00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b44f30 0x55ecb8b95c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c0c00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8f240 0x55ecb8b48aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f24b40557d0 tx=0x7f24b4055290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a298d6fe3634d365 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24a9600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24a9600640 1 -- 172.21.15.67:0/2389015346 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b406d070 con 0x55ecb8b8f240 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24a9600640 1 -- 172.21.15.67:0/2389015346 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24b4051eb0 con 0x55ecb8b8f240 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24ac0b6040 con 0x55ecb8b8f240 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24a9600640 1 -- 172.21.15.67:0/2389015346 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24b4083170 con 0x55ecb8b8f240 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 10 client.5448 fetch_fsmap learned FSMap version 18 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.915+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x55ecb891c3a0 con 0x55ecb8b8f240 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24a9600640 1 -- 172.21.15.67:0/2389015346 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac0b6040 con 0x55ecb8b8f240 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24a9600640 1 -- 172.21.15.67:0/2389015346 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f24b406d210 con 0x55ecb8b8f240 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 10 client.5448 fetch_fsmap finished waiting for FSMap version 18 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 10 client.5448 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:26:21.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 20 client.5448 populate_metadata read hostname 'smithi067' 2024-04-17T10:26:21.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8b4a160 0x55ecb8b9fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 4 client.5448 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:26:21.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ecb892fa50 con 0x55ecb8b4a160 2024-04-17T10:26:21.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24bbe00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8b4a160 0x55ecb8b9fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24bbe00640 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8b4a160 0x55ecb8b9fce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7bcb5920d78eb73 server_cookie=56bdbca12a608ac2 in_seq=0 out_seq=0 2024-04-17T10:26:21.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24a9600640 10 client.5448 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:26:21.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24a9600640 1 -- 172.21.15.67:0/2389015346 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ecb892fa50 con 0x55ecb8b4a160 2024-04-17T10:26:21.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24a9600640 10 client.5448 handle_command_reply: tid=0 2024-04-17T10:26:21.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 client.5448 shutdown 2024-04-17T10:26:21.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8f240 msgr2=0x55ecb8b48aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8f240 0x55ecb8b48aa0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f24b40557d0 tx=0x7f24b4055290 comp rx=0 tx=0).stop 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 shutdown_connections 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8b4a160 0x55ecb8b9fce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8f240 0x55ecb8b48aa0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 --2- 172.21.15.67:0/2389015346 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b44f30 0x55ecb8b95c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 >> 172.21.15.67:0/2389015346 conn(0x55ecb8b4b2a0 msgr2=0x55ecb8b439e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24a9600640 0 client.5448 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 shutdown_connections 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 1 -- 172.21.15.67:0/2389015346 wait complete. 2024-04-17T10:26:21.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.919+0000 7f24c6117000 20 client.5448 trim_cache size 0 max 16384 2024-04-17T10:26:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 Processor -- start 2024-04-17T10:26:21.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 -- start start 2024-04-17T10:26:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8be99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8be9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bea440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc1ff520 con 0x55ecb8b8e920 2024-04-17T10:26:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0adf00 con 0x55ecb8b8f240 2024-04-17T10:26:21.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc0ae500 con 0x55ecb8b44f30 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24b3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8be9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24b3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8be9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32892/0 (socket says 172.21.15.67:32892) 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24b3e00640 1 -- 172.21.15.67:0/2525322493 learned_addr learned my addr 172.21.15.67:0/2525322493 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24bbe00640 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8be99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24c0c00640 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bea440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24b3e00640 1 -- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 msgr2=0x55ecb8be99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.935+0000 7f24b3e00640 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8be99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24b3e00640 1 -- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 msgr2=0x55ecb8bea440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24b3e00640 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bea440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24b3e00640 1 -- 172.21.15.67:0/2525322493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ecb8b8f070 con 0x55ecb8b8e920 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c0c00640 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24b3e00640 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8be9f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f24a4005db0 tx=0x7f24a4007420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b248d7aacf6d9c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24abe00640 1 -- 172.21.15.67:0/2525322493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4007760 con 0x55ecb8b8e920 2024-04-17T10:26:21.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24abe00640 1 -- 172.21.15.67:0/2525322493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a401a070 con 0x55ecb8b8e920 2024-04-17T10:26:21.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24abe00640 1 -- 172.21.15.67:0/2525322493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4001ac0 con 0x55ecb8b8e920 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- 172.21.15.67:0/2525322493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 msgr2=0x55ecb8be9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8be9f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f24a4005db0 tx=0x7f24a4007420 comp rx=0 tx=0).stop 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- 172.21.15.67:0/2525322493 shutdown_connections 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bea440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b8e920 0x55ecb8be9f00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 --2- 172.21.15.67:0/2525322493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b44f30 0x55ecb8be99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- 172.21.15.67:0/2525322493 >> 172.21.15.67:0/2525322493 conn(0x55ecb8b4b2a0 msgr2=0x55ecb8b486a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- 172.21.15.67:0/2525322493 shutdown_connections 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- 172.21.15.67:0/2525322493 wait complete. 2024-04-17T10:26:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- start start 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 Processor -- start 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b44f30 0x55ecb8b9ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b9bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bd3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb89983d0 con 0x55ecb8b44f30 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ecb8a4fd50 con 0x55ecb8b8f240 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ecb892e980 con 0x55ecb8b8e920 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c6117000 10 client.0 _ll_register_callbacks cb 0x7f24ac0b3070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b44f30 0x55ecb8b9ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b44f30 0x55ecb8b9ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32902/0 (socket says 172.21.15.67:32902) 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 -- 172.21.15.67:0/3560465620 learned_addr learned my addr 172.21.15.67:0/3560465620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24b3e00640 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b9bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24c0c00640 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bd3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 -- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 msgr2=0x55ecb8b9bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b9bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 -- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 msgr2=0x55ecb8bd3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bd3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.939+0000 7f24bbe00640 1 -- 172.21.15.67:0/3560465620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ecb8a15600 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24bbe00640 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b44f30 0x55ecb8b9ba90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f24ac01eb00 tx=0x7f24ac0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0d886fa6bc8af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 1 -- 172.21.15.67:0/3560465620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ac0cc040 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24a401a070 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 1 -- 172.21.15.67:0/3560465620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24ac0b08d0 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 1 -- 172.21.15.67:0/3560465620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=18) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24ac0b0ad0 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 10 client.5454 fetch_fsmap learned FSMap version 18 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=18,monmap=2+}) v3 -- 0x55ecb89289b0 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 1 -- 172.21.15.67:0/3560465620 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a401a070 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 1 -- 172.21.15.67:0/3560465620 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 18) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f24ac0b0c70 con 0x55ecb8b44f30 2024-04-17T10:26:21.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 10 client.5454 fetch_fsmap finished waiting for FSMap version 18 2024-04-17T10:26:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 10 client.5454 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:26:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 20 client.5454 populate_metadata read hostname 'smithi067' 2024-04-17T10:26:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8bd3770 0x55ecb8bd3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 4 client.5454 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:26:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ecb89a12e0 con 0x55ecb8bd3770 2024-04-17T10:26:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24b3e00640 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8bd3770 0x55ecb8bd3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:21.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 10 client.5454 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:26:21.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24b3e00640 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8bd3770 0x55ecb8bd3b50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e66e1bc1f40c0a94 server_cookie=6bfe067534bd73d6 in_seq=0 out_seq=0 2024-04-17T10:26:21.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 1 -- 172.21.15.67:0/3560465620 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ecb89a12e0 con 0x55ecb8bd3770 2024-04-17T10:26:21.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 10 client.5454 handle_command_reply: tid=0 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 client.5454 shutdown 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b44f30 msgr2=0x55ecb8b9ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b44f30 0x55ecb8b9ba90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f24ac01eb00 tx=0x7f24ac0046d0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 shutdown_connections 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x55ecb8bd3770 0x55ecb8bd3b50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ecb8b8f240 0x55ecb8bd3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ecb8b8e920 0x55ecb8b9bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 --2- 172.21.15.67:0/3560465620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ecb8b44f30 0x55ecb8b9ba90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 >> 172.21.15.67:0/3560465620 conn(0x55ecb8b4b2a0 msgr2=0x55ecb8b967a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24a9600640 0 client.5454 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 shutdown_connections 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 1 -- 172.21.15.67:0/3560465620 wait complete. 2024-04-17T10:26:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c6117000 20 client.5454 trim_cache size 0 max 16384 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5283, 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:26:21.943 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 5.9410697280000004, 2024-04-17T10:26:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 18, 2024-04-17T10:26:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 39, 2024-04-17T10:26:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 39, 2024-04-17T10:26:21.944 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 13.801842085000001 2024-04-17T10:26:21.944 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:26:21.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2498066f90 msgr2=0x7f2498069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2498066f90 0x7f2498069450 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f24ac0b2000 tx=0x7f24ac008040 comp rx=0 tx=0).stop 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24bc145040 msgr2=0x7f24bc1e5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24bc145040 0x7f24bc1e5d60 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f24b400c7a0 tx=0x7f24b400cc60 comp rx=0 tx=0).stop 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 shutdown_connections 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2498066f90 0x7f2498069450 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24bc145040 0x7f24bc1e5d60 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24bc071a20 0x7f24bc15aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 --2- 172.21.15.67:0/412103066 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24bc06ef10 0x7f24bc0b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 >> 172.21.15.67:0/412103066 conn(0x7f24bc06ac90 msgr2=0x7f24bc071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 shutdown_connections 2024-04-17T10:26:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:21.943+0000 7f24c2c00640 1 -- 172.21.15.67:0/412103066 wait complete. 2024-04-17T10:26:21.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c15000640 1 Processor -- start 2024-04-17T10:26:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c15000640 1 -- start start 2024-04-17T10:26:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c15000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c15000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100cde20 con 0x7f4c10099a50 2024-04-17T10:26:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c15000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c100cf300 con 0x7f4c1009a740 2024-04-17T10:26:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c15000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c100cf5e0 con 0x7f4c100d03a0 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c0f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4c10099a50 0x7f4c10099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35468/0 (socket says 172.21.15.67:35468) 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c0f400640 1 -- 172.21.15.67:0/3232954642 learned_addr learned my addr 172.21.15.67:0/3232954642 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c07e00640 1 --2- 172.21.15.67:0/3232954642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c0ea00640 1 -- 172.21.15.67:0/3232954642 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131534309 0 0) 0x7f4c100cde20 con 0x7f4c10099a50 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c0ea00640 1 -- 172.21.15.67:0/3232954642 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf4003530 con 0x7f4c10099a50 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c0ea00640 1 -- 172.21.15.67:0/3232954642 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306290783 0 0) 0x7f4c100cf300 con 0x7f4c1009a740 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c0ea00640 1 -- 172.21.15.67:0/3232954642 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c100cde20 con 0x7f4c1009a740 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c07e00640 1 -- 172.21.15.67:0/3232954642 >> v1:172.21.15.86:6789/0 conn(0x7f4c1009a740 legacy=0x7f4c100cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c07e00640 1 -- 172.21.15.67:0/3232954642 >> v1:172.21.15.67:6789/0 conn(0x7f4c10099a50 legacy=0x7f4c10099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:22.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.107+0000 7f4c07e00640 1 -- 172.21.15.67:0/3232954642 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100cf8c0 con 0x7f4c100d03a0 2024-04-17T10:26:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c07e00640 1 --2- 172.21.15.67:0/3232954642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1015c7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4bf80097f0 tx=0x7f4bf801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe78efa7719bd414 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c0ea00640 1 -- 172.21.15.67:0/3232954642 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf80b0070 con 0x7f4c100d03a0 2024-04-17T10:26:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c0ea00640 1 -- 172.21.15.67:0/3232954642 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bf80ab600 con 0x7f4c100d03a0 2024-04-17T10:26:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c0ea00640 1 -- 172.21.15.67:0/3232954642 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bf80ab9d0 con 0x7f4c100d03a0 2024-04-17T10:26:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 -- 172.21.15.67:0/3232954642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 msgr2=0x7f4c1015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 --2- 172.21.15.67:0/3232954642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1015c7a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4bf80097f0 tx=0x7f4bf801c4e0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 -- 172.21.15.67:0/3232954642 shutdown_connections 2024-04-17T10:26:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 --2- 172.21.15.67:0/3232954642 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1015c7a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 -- 172.21.15.67:0/3232954642 >> 172.21.15.67:0/3232954642 conn(0x7f4c100d1690 msgr2=0x7f4c100d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:22.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 -- 172.21.15.67:0/3232954642 shutdown_connections 2024-04-17T10:26:22.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 -- 172.21.15.67:0/3232954642 wait complete. 2024-04-17T10:26:22.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 Processor -- start 2024-04-17T10:26:22.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.111+0000 7f4c15000640 1 -- start start 2024-04-17T10:26:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c10099a50 0x7f4c1014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c1009a740 0x7f4c1006cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10169a50 con 0x7f4c10099a50 2024-04-17T10:26:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c101698d0 con 0x7f4c1009a740 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10169bd0 con 0x7f4c100d03a0 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c10099a50 0x7f4c1014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c10099a50 0x7f4c1014de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32910/0 (socket says 172.21.15.67:32910) 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 -- 172.21.15.67:0/1762339212 learned_addr learned my addr 172.21.15.67:0/1762339212 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c07e00640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c1009a740 0x7f4c1006cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0fe00640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 -- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 msgr2=0x7f4c1006d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1006d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 -- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c1009a740 msgr2=0x7f4c1006cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c1009a740 0x7f4c1006cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 -- 172.21.15.67:0/1762339212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c100b2a70 con 0x7f4c10099a50 2024-04-17T10:26:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c0f400640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c10099a50 0x7f4c1014de70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000c420 tx=0x7f4c0000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:22.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c07400640 1 -- 172.21.15.67:0/1762339212 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00017070 con 0x7f4c10099a50 2024-04-17T10:26:22.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c07400640 1 -- 172.21.15.67:0/1762339212 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c00005cf0 con 0x7f4c10099a50 2024-04-17T10:26:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c07400640 1 -- 172.21.15.67:0/1762339212 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c00007310 con 0x7f4c10099a50 2024-04-17T10:26:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c10154310 con 0x7f4c10099a50 2024-04-17T10:26:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1014e3b0 con 0x7f4c10099a50 2024-04-17T10:26:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.115+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c10154c40 con 0x7f4c10099a50 2024-04-17T10:26:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.119+0000 7f4c07400640 1 -- 172.21.15.67:0/1762339212 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4c0001a070 con 0x7f4c10099a50 2024-04-17T10:26:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.119+0000 7f4c07400640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bec066f60 0x7f4bec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.119+0000 7f4c07400640 1 -- 172.21.15.67:0/1762339212 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c00084470 con 0x7f4c10099a50 2024-04-17T10:26:22.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.119+0000 7f4c07e00640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bec066f60 0x7f4bec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:22.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.119+0000 7f4c07e00640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bec066f60 0x7f4bec069420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4bf80ad000 tx=0x7f4bf8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:22.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.123+0000 7f4c07400640 1 -- 172.21.15.67:0/1762339212 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4c00054d30 con 0x7f4c10099a50 2024-04-17T10:26:22.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4c10074dc0 con 0x7f4c10099a50 2024-04-17T10:26:22.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c07400640 1 -- 172.21.15.67:0/1762339212 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v18) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4c00058dd0 con 0x7f4c10099a50 2024-04-17T10:26:22.314 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:26:22.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bec066f60 msgr2=0x7f4bec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bec066f60 0x7f4bec069420 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4bf80ad000 tx=0x7f4bf8008040 comp rx=0 tx=0).stop 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c10099a50 msgr2=0x7f4c1014de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c10099a50 0x7f4c1014de70 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f4c0000c420 tx=0x7f4c0000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 shutdown_connections 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bec066f60 0x7f4bec069420 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c100d03a0 0x7f4c1006d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c1009a740 0x7f4c1006cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 --2- 172.21.15.67:0/1762339212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c10099a50 0x7f4c1014de70 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 >> 172.21.15.67:0/1762339212 conn(0x7f4c100d1690 msgr2=0x7f4c10166e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 shutdown_connections 2024-04-17T10:26:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:22.319+0000 7f4c15000640 1 -- 172.21.15.67:0/1762339212 wait complete. 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:26:22.335 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:26:22.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.336 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:26:22.345 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:26:22.346 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42364sec preferred_lft 42364sec 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:26:22.347 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:26:22.348 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:26:22.349 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:26:22.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.349 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:26:22.405 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:26:22.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.407 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:26:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:26:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:26:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:26:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:26:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:26:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:26:22.458 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:26:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:26:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:26:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:26:22.459 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:26:22.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.459 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:26:22.459 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:26:22.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:26:22.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:26:22.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:26:22.460 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:26:22.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.554 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:26:22.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:26:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:26:22.563 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:26:22.564 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:26:22.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.564 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:26:22.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:26:22.629 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:26:22.629 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:26:22.707 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:26:22.707 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:26:22.707 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:26:22.755 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:26:22.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:26:22.848 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:26:22.848 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:26:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:26:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:26:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:26:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:26:22.863 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:26:22.867 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:26:22.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:26:22.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:22.888 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:26:22.948 INFO:teuthology.orchestra.run.smithi067.stdout:client5090 2024-04-17T10:26:22.948 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:26:22.949 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:26:23.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2024-04-17T10:26:23.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f447ac00640 1 Processor -- start 2024-04-17T10:26:23.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f447ac00640 1 -- start start 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f447ac00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f447406d3c0 0x7f4474071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f447ac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44740718a0 0x7f44740cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f447ac00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4474071cb0 con 0x7f44741447e0 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f447ac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44740cc310 con 0x7f44740718a0 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f447ac00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f447406d9a0 con 0x7f447406d3c0 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f4478c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44740718a0 0x7f44740cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f4478c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44740718a0 0x7f44740cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32914/0 (socket says 172.21.15.67:32914) 2024-04-17T10:26:23.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f4478c00640 1 -- 172.21.15.67:0/2089307545 learned_addr learned my addr 172.21.15.67:0/2089307545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f446be00640 1 --2- 172.21.15.67:0/2089307545 >> v2:172.21.15.86:3300/0 conn(0x7f447406d3c0 0x7f4474071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f4478c00640 1 -- 172.21.15.67:0/2089307545 >> v2:172.21.15.86:3300/0 conn(0x7f447406d3c0 msgr2=0x7f4474071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:26:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f4478c00640 1 --2- 172.21.15.67:0/2089307545 >> v2:172.21.15.86:3300/0 conn(0x7f447406d3c0 0x7f4474071360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f4478c00640 1 -- 172.21.15.67:0/2089307545 >> v1:172.21.15.86:6789/0 conn(0x7f44741447e0 legacy=0x7f4474144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:23.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.199+0000 7f4478c00640 1 -- 172.21.15.67:0/2089307545 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447406dc80 con 0x7f44740718a0 2024-04-17T10:26:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f4478c00640 1 --2- 172.21.15.67:0/2089307545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f44740cc7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f447406bb20 tx=0x7f446c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a60bd13a0ba4dae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f4473400640 1 -- 172.21.15.67:0/2089307545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c0a5700 con 0x7f44740718a0 2024-04-17T10:26:23.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f4473400640 1 -- 172.21.15.67:0/2089307545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c0a58a0 con 0x7f44740718a0 2024-04-17T10:26:23.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f4473400640 1 -- 172.21.15.67:0/2089307545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c0a5c70 con 0x7f44740718a0 2024-04-17T10:26:23.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 -- 172.21.15.67:0/2089307545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 msgr2=0x7f44740cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:23.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 --2- 172.21.15.67:0/2089307545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f44740cc7a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f447406bb20 tx=0x7f446c01c930 comp rx=0 tx=0).stop 2024-04-17T10:26:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 -- 172.21.15.67:0/2089307545 shutdown_connections 2024-04-17T10:26:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 --2- 172.21.15.67:0/2089307545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f44740cc7a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 --2- 172.21.15.67:0/2089307545 >> v2:172.21.15.86:3300/0 conn(0x7f447406d3c0 0x7f4474071360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 -- 172.21.15.67:0/2089307545 >> 172.21.15.67:0/2089307545 conn(0x7f447406ac90 msgr2=0x7f447406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 -- 172.21.15.67:0/2089307545 shutdown_connections 2024-04-17T10:26:23.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.203+0000 7f447ac00640 1 -- 172.21.15.67:0/2089307545 wait complete. 2024-04-17T10:26:23.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 Processor -- start 2024-04-17T10:26:23.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 -- start start 2024-04-17T10:26:23.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f447406d3c0 0x7f447413eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:23.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f4474155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:23.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44741447e0 0x7f44741596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740cd880 con 0x7f44740718a0 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44740cd700 con 0x7f447406d3c0 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44740cda00 con 0x7f44741447e0 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f4474155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f4474155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32922/0 (socket says 172.21.15.67:32922) 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 -- 172.21.15.67:0/1693327276 learned_addr learned my addr 172.21.15.67:0/1693327276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f4473e00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f447406d3c0 0x7f447413eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f4478c00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44741447e0 0x7f44741596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 -- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44741447e0 msgr2=0x7f44741596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44741447e0 0x7f44741596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 -- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f447406d3c0 msgr2=0x7f447413eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f447406d3c0 0x7f447413eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:23.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 -- 172.21.15.67:0/1693327276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4474155db0 con 0x7f44740718a0 2024-04-17T10:26:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f446be00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f4474155870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f445c00e930 tx=0x7f445c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f4470c00640 1 -- 172.21.15.67:0/1693327276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c017070 con 0x7f44740718a0 2024-04-17T10:26:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f4470c00640 1 -- 172.21.15.67:0/1693327276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445c00ca00 con 0x7f44740718a0 2024-04-17T10:26:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4474157f10 con 0x7f44740718a0 2024-04-17T10:26:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f4470c00640 1 -- 172.21.15.67:0/1693327276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445c00cdb0 con 0x7f44740718a0 2024-04-17T10:26:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447413e6a0 con 0x7f44740718a0 2024-04-17T10:26:23.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.207+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447413e490 con 0x7f44740718a0 2024-04-17T10:26:23.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.211+0000 7f4470c00640 1 -- 172.21.15.67:0/1693327276 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f445c01a070 con 0x7f44740718a0 2024-04-17T10:26:23.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.211+0000 7f4470c00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4450066f60 0x7f4450069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:26:23.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.211+0000 7f4470c00640 1 -- 172.21.15.67:0/1693327276 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f445c0851e0 con 0x7f44740718a0 2024-04-17T10:26:23.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.211+0000 7f4473e00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4450066f60 0x7f4450069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:26:23.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.211+0000 7f4473e00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4450066f60 0x7f4450069420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f447413bba0 tx=0x7f4464009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:26:23.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.215+0000 7f4470c00640 1 -- 172.21.15.67:0/1693327276 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f445c055a30 con 0x7f44740718a0 2024-04-17T10:26:23.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:23.411+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4474072da0 con 0x7f44740718a0 2024-04-17T10:26:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f4470c00640 1 -- 172.21.15.67:0/1693327276 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v19) v1 ==== 167+0+0 (secure 0 0 0) 0x7f445c059ad0 con 0x7f44740718a0 2024-04-17T10:26:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:enabled new snapshots 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4450066f60 msgr2=0x7f4450069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4450066f60 0x7f4450069420 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f447413bba0 tx=0x7f4464009040 comp rx=0 tx=0).stop 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 msgr2=0x7f4474155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f4474155870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f445c00e930 tx=0x7f445c00edf0 comp rx=0 tx=0).stop 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 shutdown_connections 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4450066f60 0x7f4450069420 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f44741447e0 0x7f44741596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44740718a0 0x7f4474155870 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 --2- 172.21.15.67:0/1693327276 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f447406d3c0 0x7f447413eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 >> 172.21.15.67:0/1693327276 conn(0x7f447406ac90 msgr2=0x7f44740ae080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 shutdown_connections 2024-04-17T10:26:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:24.051+0000 7f447ac00640 1 -- 172.21.15.67:0/1693327276 wait complete. 2024-04-17T10:26:24.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:24.070 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir stray_eval_dir) 2024-04-17T10:26:24.085 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:26:24.085 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> import uuid 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> n = 2000 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> create_hard_links = False 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> if 3 > 0: 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> create_hard_links = True 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> path = "/home/ubuntu/cephtest/mnt.0/stray_eval_dir/file" 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:26:24.138 DEBUG:teuthology.orchestra.run.smithi067:> dpath = os.path.dirname(path) 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> fnameprefix = os.path.basename(path) 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> os.makedirs(dpath, exist_ok=True) 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> try: 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> for i in range(n): 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> with open(fpath, '"'"'w'"'"') as f: 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> f.write(f"{i}") 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> f.flush() 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> os.fsync(f.fileno()) 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> os.unlink(fpath) 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:26:24.139 DEBUG:teuthology.orchestra.run.smithi067:> os.fsync(dirfd) 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> if create_hard_links: 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> for j in range(3): 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> os.fsync(dirfd) 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> finally: 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> os.close(dirfd) 2024-04-17T10:26:24.140 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:26:45.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:26:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T10:26:45.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:45.752 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-04-17T10:26:45.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:26:45.822 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec find /home/ubuntu/cephtest/mnt.0/stray_eval_dir/ -type f -delete) 2024-04-17T10:26:52.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:26:52.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:27:12.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:27:12.867 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-04-17T10:27:12.942 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:27:12.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 perf dump 2024-04-17T10:27:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6fa400640 1 Processor -- start 2024-04-17T10:27:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6fa400640 1 -- start start 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6fa400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6f4071af0 0x7fa6f40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6fa400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f415be70 0x7fa6f415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6fa400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6f415cb30 con 0x7fa6f4159440 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6fa400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6f40a8650 con 0x7fa6f4071af0 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6fa400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415ce10 con 0x7fa6f415be70 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6f3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa6f4159440 0x7fa6f40713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36274/0 (socket says 172.21.15.67:36274) 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6f3400640 1 -- 172.21.15.67:0/1286058609 learned_addr learned my addr 172.21.15.67:0/1286058609 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/1286058609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f415be70 0x7fa6f415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/1286058609 >> v2:172.21.15.86:3300/0 conn(0x7fa6f4071af0 0x7fa6f40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6f2a00640 1 -- 172.21.15.67:0/1286058609 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507200703 0 0) 0x7fa6f415cb30 con 0x7fa6f4159440 2024-04-17T10:27:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.090+0000 7fa6f2a00640 1 -- 172.21.15.67:0/1286058609 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6d8003530 con 0x7fa6f4159440 2024-04-17T10:27:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f3e00640 1 -- 172.21.15.67:0/1286058609 >> v2:172.21.15.86:3300/0 conn(0x7fa6f4071af0 msgr2=0x7fa6f40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/1286058609 >> v2:172.21.15.86:3300/0 conn(0x7fa6f4071af0 0x7fa6f40ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f3e00640 1 -- 172.21.15.67:0/1286058609 >> v1:172.21.15.67:6789/0 conn(0x7fa6f4159440 legacy=0x7fa6f40713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f3e00640 1 -- 172.21.15.67:0/1286058609 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40cd610 con 0x7fa6f415be70 2024-04-17T10:27:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/1286058609 >> v2:172.21.15.86:3300/0 conn(0x7fa6f4071af0 0x7fa6f40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/1286058609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f415be70 0x7fa6f415c2c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009d90 tx=0x7fa6ec01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da03a6a0e355fda8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f2a00640 1 -- 172.21.15.67:0/1286058609 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0b2070 con 0x7fa6f415be70 2024-04-17T10:27:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f2a00640 1 -- 172.21.15.67:0/1286058609 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0a5a30 con 0x7fa6f415be70 2024-04-17T10:27:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6f2a00640 1 -- 172.21.15.67:0/1286058609 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0a5e00 con 0x7fa6f415be70 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 -- 172.21.15.67:0/1286058609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f415be70 msgr2=0x7fa6f415c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 --2- 172.21.15.67:0/1286058609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f415be70 0x7fa6f415c2c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009d90 tx=0x7fa6ec01c970 comp rx=0 tx=0).stop 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 -- 172.21.15.67:0/1286058609 shutdown_connections 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 --2- 172.21.15.67:0/1286058609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f415be70 0x7fa6f415c2c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 --2- 172.21.15.67:0/1286058609 >> v2:172.21.15.86:3300/0 conn(0x7fa6f4071af0 0x7fa6f40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 -- 172.21.15.67:0/1286058609 >> 172.21.15.67:0/1286058609 conn(0x7fa6f40b1740 msgr2=0x7fa6f40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 -- 172.21.15.67:0/1286058609 shutdown_connections 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.094+0000 7fa6fa400640 1 -- 172.21.15.67:0/1286058609 wait complete. 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 Processor -- start 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 -- start start 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f4071af0 0x7fa6f414ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6f4159440 0x7fa6f4153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f415be70 0x7fa6f40b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415d840 con 0x7fa6f4159440 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415c800 con 0x7fa6f415be70 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f415d9c0 con 0x7fa6f4071af0 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6f4159440 0x7fa6f4153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6f4159440 0x7fa6f4153890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44200/0 (socket says 172.21.15.67:44200) 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 -- 172.21.15.67:0/651724181 learned_addr learned my addr 172.21.15.67:0/651724181 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f415be70 0x7fa6f40b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6f3400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f4071af0 0x7fa6f414ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 -- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f4071af0 msgr2=0x7fa6f414ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f4071af0 0x7fa6f414ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 -- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f415be70 msgr2=0x7fa6f40b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f415be70 0x7fa6f40b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 -- 172.21.15.67:0/651724181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f40b07f0 con 0x7fa6f4159440 2024-04-17T10:27:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6f4159440 0x7fa6f4153890 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00e9d0 tx=0x7fa6dc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6eb400640 1 -- 172.21.15.67:0/651724181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc017070 con 0x7fa6f4159440 2024-04-17T10:27:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6eb400640 1 -- 172.21.15.67:0/651724181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6dc00ca00 con 0x7fa6f4159440 2024-04-17T10:27:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6f40b0ad0 con 0x7fa6f4159440 2024-04-17T10:27:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6eb400640 1 -- 172.21.15.67:0/651724181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc00cdf0 con 0x7fa6f4159440 2024-04-17T10:27:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.098+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6f4153370 con 0x7fa6f4159440 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6eb400640 1 -- 172.21.15.67:0/651724181 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa6dc01a070 con 0x7fa6f4159440 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6eb400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6d0066f60 0x7fa6d0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6eb400640 1 -- 172.21.15.67:0/651724181 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7fa6dc084d00 con 0x7fa6f4159440 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 Processor -- start 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6f3400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6d0066f60 0x7fa6d0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 -- start start 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6f3400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6d0066f60 0x7fa6d0069420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4007fd0 tx=0x7fa6e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151efea40 0x562151efee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eff360 0x562151eb9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151eba170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562151d9c1f0 con 0x562151eff360 2024-04-17T10:27:13.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x562151d02e90 con 0x562151eb9d90 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.102+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x562151c8af50 con 0x562151efea40 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eff360 0x562151eb9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151eba170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151efea40 0x562151efee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151eba170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39630/0 (socket says 172.21.15.67:39630) 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 -- 172.21.15.67:0/129793506 learned_addr learned my addr 172.21.15.67:0/129793506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 -- 172.21.15.67:0/129793506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151efea40 msgr2=0x562151efee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/129793506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151efea40 0x562151efee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 -- 172.21.15.67:0/129793506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eff360 msgr2=0x562151eb9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/129793506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eff360 0x562151eb9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 -- 172.21.15.67:0/129793506 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562151cda8b0 con 0x562151eb9d90 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/129793506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151eba170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec01c9b0 tx=0x7fa6ec004ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8237bed42ac5f665 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6e3e00640 1 -- 172.21.15.67:0/129793506 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0b2040 con 0x562151eb9d90 2024-04-17T10:27:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6e3e00640 1 -- 172.21.15.67:0/129793506 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0b3040 con 0x562151eb9d90 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6e3e00640 1 -- 172.21.15.67:0/129793506 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0a5a50 con 0x562151eb9d90 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- 172.21.15.67:0/129793506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 msgr2=0x562151eba170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 --2- 172.21.15.67:0/129793506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151eba170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec01c9b0 tx=0x7fa6ec004ac0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- 172.21.15.67:0/129793506 shutdown_connections 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 --2- 172.21.15.67:0/129793506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151eba170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 --2- 172.21.15.67:0/129793506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eff360 0x562151eb9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 --2- 172.21.15.67:0/129793506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151efea40 0x562151efee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- 172.21.15.67:0/129793506 >> 172.21.15.67:0/129793506 conn(0x562151eb5d40 msgr2=0x562151eb8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- 172.21.15.67:0/129793506 shutdown_connections 2024-04-17T10:27:13.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- 172.21.15.67:0/129793506 wait complete. 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- start start 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 Processor -- start 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eb9d90 0x562151f061c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f06700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eff360 0x562151ebfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562151cbd540 con 0x562151efea40 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x562151cfef30 con 0x562151eff360 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x562151cae260 con 0x562151eb9d90 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6fd933000 10 client.0 _ll_register_callbacks cb 0x562151db7180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f06700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f06700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44214/0 (socket says 172.21.15.67:44214) 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6ebe00640 1 -- 172.21.15.67:0/2452268207 learned_addr learned my addr 172.21.15.67:0/2452268207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3400640 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eb9d90 0x562151f061c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.106+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eff360 0x562151ebfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6ebe00640 1 -- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eb9d90 msgr2=0x562151f061c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eb9d90 0x562151f061c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6ebe00640 1 -- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eff360 msgr2=0x562151ebfe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eff360 0x562151ebfe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6ebe00640 1 -- 172.21.15.67:0/2452268207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562151c8b1a0 con 0x562151efea40 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f06700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c520 tx=0x7fa6dc0595d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba58859b2a2e9cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 1 -- 172.21.15.67:0/2452268207 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6dc081060 con 0x562151efea40 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 1 -- 172.21.15.67:0/2452268207 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7fa6dc082a60 con 0x562151efea40 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6ec0b3040 con 0x562151efea40 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 1 -- 172.21.15.67:0/2452268207 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6dc072050 con 0x562151efea40 2024-04-17T10:27:13.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 10 client.5478 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x562151d85840 con 0x562151efea40 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 1 -- 172.21.15.67:0/2452268207 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec0b3040 con 0x562151efea40 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 1 -- 172.21.15.67:0/2452268207 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa6dc073240 con 0x562151efea40 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 10 client.5478 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 10 client.5478 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 20 client.5478 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151ec0bc0 0x562151ec0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 4 client.5478 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562151c9ebe0 con 0x562151ec0bc0 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6f3400640 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151ec0bc0 0x562151ec0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 10 client.5478 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6f3400640 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151ec0bc0 0x562151ec0fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ecd253de19301a server_cookie=6dd36ed2f0d5bae4 in_seq=0 out_seq=0 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 1 -- 172.21.15.67:0/2452268207 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562151c9ebe0 con 0x562151ec0bc0 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 10 client.5478 handle_command_reply: tid=0 2024-04-17T10:27:13.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 client.5478 shutdown 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 msgr2=0x562151f06700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f06700 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00c520 tx=0x7fa6dc0595d0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 shutdown_connections 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151ec0bc0 0x562151ec0fd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eff360 0x562151ebfe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6e0c00640 0 client.5478 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f06700 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 --2- 172.21.15.67:0/2452268207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eb9d90 0x562151f061c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 >> 172.21.15.67:0/2452268207 conn(0x562151eb5d40 msgr2=0x562151eb89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 shutdown_connections 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 1 -- 172.21.15.67:0/2452268207 wait complete. 2024-04-17T10:27:13.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.110+0000 7fa6fd933000 20 client.5478 trim_cache size 0 max 16384 2024-04-17T10:27:13.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 Processor -- start 2024-04-17T10:27:13.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 -- start start 2024-04-17T10:27:13.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eb9d90 0x562151f5b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151efea40 0x562151f5b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f5bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4203070 con 0x562151eb9d90 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4202270 con 0x562151efea40 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4152ad0 con 0x562151eff360 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eb9d90 0x562151f5b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eb9d90 0x562151f5b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44216/0 (socket says 172.21.15.67:44216) 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 -- 172.21.15.67:0/793783360 learned_addr learned my addr 172.21.15.67:0/793783360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151efea40 0x562151f5b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f5bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 -- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 msgr2=0x562151f5bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f5bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 -- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151efea40 msgr2=0x562151f5b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151efea40 0x562151f5b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 -- 172.21.15.67:0/793783360 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562151d5ca10 con 0x562151eb9d90 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151efea40 0x562151f5b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f5bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6f3400640 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eb9d90 0x562151f5b4a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40079d0 tx=0x7fa6e4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bbfd2ceee539d6c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6e3400640 1 -- 172.21.15.67:0/793783360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4027e60 con 0x562151eb9d90 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6e3400640 1 -- 172.21.15.67:0/793783360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e402d070 con 0x562151eb9d90 2024-04-17T10:27:13.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.126+0000 7fa6e3400640 1 -- 172.21.15.67:0/793783360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e40158e0 con 0x562151eb9d90 2024-04-17T10:27:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/793783360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eb9d90 msgr2=0x562151f5b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eb9d90 0x562151f5b4a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa6e40079d0 tx=0x7fa6e4000ee0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/793783360 shutdown_connections 2024-04-17T10:27:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f5bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151efea40 0x562151f5b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- 172.21.15.67:0/793783360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151eb9d90 0x562151f5b4a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/793783360 >> 172.21.15.67:0/793783360 conn(0x562151eb5d40 msgr2=0x562151efd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/793783360 shutdown_connections 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/793783360 wait complete. 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- start start 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 Processor -- start 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151f5f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f5fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f03ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562151cc6720 con 0x562151efea40 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6f4201c70 con 0x562151eb9d90 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x562151d074c0 con 0x562151eff360 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 10 client.0 _ll_register_callbacks cb 0x562151cc5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:13.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f5fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f5fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44222/0 (socket says 172.21.15.67:44222) 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6ebe00640 1 -- 172.21.15.67:0/3715685148 learned_addr learned my addr 172.21.15.67:0/3715685148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3400640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151f5f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3e00640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f03ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3400640 1 -- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 msgr2=0x562151f03ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3400640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f03ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3400640 1 -- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 msgr2=0x562151f5fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3400640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f5fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3400640 1 -- 172.21.15.67:0/3715685148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562151c97b00 con 0x562151eb9d90 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f5fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6f3400640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151f5f6c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400f790 tx=0x7fa6e4000bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=635c45da706c6d2d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6e0c00640 1 -- 172.21.15.67:0/3715685148 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e4038060 con 0x562151eb9d90 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6e0c00640 1 -- 172.21.15.67:0/3715685148 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7fa6e40158e0 con 0x562151eb9d90 2024-04-17T10:27:13.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6e402d070 con 0x562151eb9d90 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6e0c00640 1 -- 172.21.15.67:0/3715685148 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6e4027e60 con 0x562151eb9d90 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 10 client.5431 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x562151ca9f80 con 0x562151eb9d90 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6e0c00640 1 -- 172.21.15.67:0/3715685148 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e402d070 con 0x562151eb9d90 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6e0c00640 1 -- 172.21.15.67:0/3715685148 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa6e4015ae0 con 0x562151eb9d90 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 10 client.5431 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 10 client.5431 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 20 client.5431 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151f04be0 0x562151f04ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 4 client.5431 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.130+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x562151d103d0 con 0x562151f04be0 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151f04be0 0x562151f04ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6ebe00640 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151f04be0 0x562151f04ff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5c56935cfea7760 server_cookie=444b10aea73d6a97 in_seq=0 out_seq=0 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6e0c00640 10 client.5431 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6e0c00640 1 -- 172.21.15.67:0/3715685148 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17920 (crc 0 0 0) 0x562151d103d0 con 0x562151f04be0 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6e0c00640 10 client.5431 handle_command_reply: tid=0 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 client.5431 shutdown 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 msgr2=0x562151f5f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151f5f6c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa6e400f790 tx=0x7fa6e4000bd0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 shutdown_connections 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x562151f04be0 0x562151f04ff0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562151eff360 0x562151f03ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562151efea40 0x562151f5fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 --2- 172.21.15.67:0/3715685148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562151eb9d90 0x562151f5f6c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6e0c00640 0 client.5431 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 >> 172.21.15.67:0/3715685148 conn(0x562151eb5d40 msgr2=0x562151efda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 shutdown_connections 2024-04-17T10:27:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 1 -- 172.21.15.67:0/3715685148 wait complete. 2024-04-17T10:27:13.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fd933000 20 client.5431 trim_cache size 0 max 16384 2024-04-17T10:27:13.133 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:27:13.133 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-04-17T10:27:13.133 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 2663, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 2664, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 663940, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 37067627, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 9, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 3, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.331460382, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.166263763, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.065573776, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.076697580, 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:27:13.134 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2663, 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.511261204, 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000191986 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2, 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000200, 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000100 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 0 2024-04-17T10:27:13.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 1502, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 1517, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 375170, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 17409625, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 6, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.182531311, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.088680644, 2024-04-17T10:27:13.136 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.037279961, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.043486565, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1516, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.303411172, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000200139 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.137 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 75504 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 30121, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 50991, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 11138928, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 55731731, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 7, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 4.716282382, 2024-04-17T10:27:13.138 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 2.808769625, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 2.816779358, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.050287098, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 50991, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.477150646, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000028968 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 71749, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.016059994, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000223 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 116608, 2024-04-17T10:27:13.139 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 19296 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 1, 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-MDSRank": { 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4390, 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 13.585274389, 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.003094595 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.140 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-PurgeQueue": { 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1827, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.719087532, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000393589 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "request": 26268, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "reply": 26268, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "reply_latency": { 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 26268, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 238.888339963, 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.009094272 2024-04-17T10:27:13.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_reply": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "forward": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_complete": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_keys": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_commit": 12, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_split": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_merge": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes": 19831, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_top": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_bottom": 19831, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pin_tail": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pinned": 19831, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_expired": 0, 2024-04-17T10:27:13.142 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_with_caps": 114, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": 114, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "subtrees": 2, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse": 49497, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_hit": 48922, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_forward": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_discover": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_dir_fetch": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_remote_ino": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_lock": 162, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "load_cent": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "q": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "exported": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "exported_inodes": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "imported": 0, 2024-04-17T10:27:13.143 INFO:teuthology.orchestra.run.smithi067.stdout: "imported_inodes": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_dir_fetch": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_backtrace_fetch": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_peer_discover": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rfiles": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rbytes": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rsnaps": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_fetch": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_set_tag": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_repaired": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_inotable_repaired": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_inodes": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_base_inodes": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dirfrag_rstats": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_file_inodes": 0, 2024-04-17T10:27:13.144 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_inode_file_caps": 0, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_revoke": 942, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_grant": 4023, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_trunc": 0, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps": 3981, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps_dirty": 2000, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_cap_release": 62, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "process_request_cap_release": 14099 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache": { 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update": 0, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update_receipt": 0, 2024-04-17T10:27:13.145 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_try_discover": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_send_discover": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_handle_discover": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays": 2000, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_delayed": 162, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_enqueuing": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_created": 4138, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_enqueued": 1728, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_reintegrated": 2248, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_migrated": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_processing": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_enqueued": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_prioritized": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_started": 0, 2024-04-17T10:27:13.146 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_completed": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_enqueue_scrub": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_exportdir": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_flush": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragmentdir": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragstats": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_inodestats": 0 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log": { 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "evlrg": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "evadd": 24153, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "evex": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "evtrm": 0, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "ev": 24153, 2024-04-17T10:27:13.147 INFO:teuthology.orchestra.run.smithi067.stdout: "evexg": 0, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "evexd": 0, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "segadd": 33, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "segex": 0, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "segtrm": 0, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "seg": 33, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "segmjr": 3, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "segexg": 0, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "segexd": 0, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "expos": 4194304, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "wrpos": 88869760, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "rdpos": 0, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "jlat": { 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4067, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 71.368319803, 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.017548148 2024-04-17T10:27:13.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "replayed": 0 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mem": { 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 2014, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "ino+": 2014, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "ino-": 0, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "dir": 13, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "dir+": 13, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "dir-": 0, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "dn": 19831, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "dn+": 19832, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "dn-": 1, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "cap": 114, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "cap+": 8390, 2024-04-17T10:27:13.149 INFO:teuthology.orchestra.run.smithi067.stdout: "cap-": 8276, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "rss": 112704, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "heap": 215340 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server": { 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_client_request": 30058, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_server_request": 0, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_request": 26268, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_session": 3, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_peer_request": 0, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "req_create_latency": { 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2000, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.590244077, 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000795122 2024-04-17T10:27:13.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getattr_latency": { 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 23, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.003463817, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000150600 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getfilelock_latency": { 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "req_link_latency": { 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6000, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 7.047901284, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001174650 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.151 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookup_latency": { 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7984, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 6.460720613, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000809208 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookuphash_latency": { 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupino_latency": { 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupname_latency": { 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupparent_latency": { 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupsnap_latency": { 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000077286, 2024-04-17T10:27:13.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000077286 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lssnap_latency": { 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mkdir_latency": { 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.002982848, 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002982848 2024-04-17T10:27:13.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mknod_latency": { 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mksnap_latency": { 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:13.155 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.007953929, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.007953929 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "req_open_latency": { 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "req_readdir_latency": { 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.093474816, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.011684352 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rename_latency": { 2024-04-17T10:27:13.156 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2248, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 207.157699302, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.092152001 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "req_renamesnap_latency": { 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "req_snapdiff_latency": { 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmdir_latency": { 2024-04-17T10:27:13.157 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmsnap_latency": { 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.063101977, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.063101977 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmxattr_latency": { 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.158 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setattr_latency": { 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000251475, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000251475 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setdirlayout_latency": { 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setfilelock_latency": { 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.159 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setlayout_latency": { 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setxattr_latency": { 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "req_symlink_latency": { 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.160 INFO:teuthology.orchestra.run.smithi067.stdout: "req_unlink_latency": { 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8000, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 16.460468539, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002057558 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_revoke_eviction": 0, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_acquisition_throttle": 0, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getvxattr_latency": { 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessions": { 2024-04-17T10:27:13.161 INFO:teuthology.orchestra.run.smithi067.stdout: "session_count": 1, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "session_add": 1, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "session_remove": 0, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_open": 1, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_stale": 0, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "total_load": 18913, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "average_load": 18913, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "avg_session_uptime": 45, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "mdthresh_evicted": 0 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 0, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 0, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-04-17T10:27:13.162 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-04-17T10:27:13.163 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-04-17T10:27:13.164 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 16975669, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 4289, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 5520, 2024-04-17T10:27:13.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 78, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 19593640, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 782864, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter": { 2024-04-17T10:27:13.166 INFO:teuthology.orchestra.run.smithi067.stdout: "op_active": 0, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "op_laggy": 0, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send": 6034, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send_bytes": 85087279, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "op_resend": 37, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "op_reply": 6034, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "op_latency": { 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6034, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 28.787762805, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.004770925 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "op_inflight": 0, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "oplen_avg": { 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6034, 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 6126 2024-04-17T10:27:13.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "op": 6034, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "op_r": 51, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "op_w": 5983, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "op_rmw": 0, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "op_pg": 0, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_stat": 0, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_create": 11, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_read": 51, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_write": 4118, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writefull": 32, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writesame": 0, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_append": 0, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_zero": 1728, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_truncate": 0, 2024-04-17T10:27:13.168 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_delete": 32, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_mapext": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_sparse_read": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_clonerange": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_getxattr": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_setxattr": 22, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_cmpxattr": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_rmxattr": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_resetxattrs": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_call": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_watch": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_notify": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_src_cmpxattr": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls_filter": 0, 2024-04-17T10:27:13.169 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_other": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_active": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_send": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_resend": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_ping": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_active": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_send": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_resend": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_active": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_send": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_resend": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_active": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_send": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_resend": 0, 2024-04-17T10:27:13.170 INFO:teuthology.orchestra.run.smithi067.stdout: "command_active": 0, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "command_send": 0, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "command_resend": 0, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "map_epoch": 41, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "map_full": 0, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "map_inc": 3, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_sessions": 8, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_open": 8, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_close": 0, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_laggy": 0, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_wr": 110, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_rd": 0, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_del": 22 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.171 INFO:teuthology.orchestra.run.smithi067.stdout: "oft": { 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_objs": 1, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_kv_pairs": 0, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_updates": 0, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_removes": 0 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "osd-slow-ops": { 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_ops_count": 0 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "purge_queue": { 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops": 0, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops_high_water": 32, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing": 0, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_high_water": 32, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed_ops": 1728, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed": 1728, 2024-04-17T10:27:13.172 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_item_in_journal": 0 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 34282, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 9087669, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 34282, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 34282, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 9087669, 2024-04-17T10:27:13.173 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_bytes": { 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 6034, 2024-04-17T10:27:13.174 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 85275631, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 6034, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 85275631, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_ops": { 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 1024, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:27:13.175 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 6034, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 6034, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 6034, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 6034, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle": { 2024-04-17T10:27:13.176 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 1728, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 188352, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 1728, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 51, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 188352, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.177 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle-0x559782f64a00": { 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 84, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 24153, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 84675456, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 24153, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 4067, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 84675372, 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:13.178 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:13.179 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:13.179 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:13.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.179 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.179 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:27:13.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6d0066f60 msgr2=0x7fa6d0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6d0066f60 0x7fa6d0069420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa6e4007fd0 tx=0x7fa6e4009040 comp rx=0 tx=0).stop 2024-04-17T10:27:13.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6f4159440 msgr2=0x7fa6f4153890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6f4159440 0x7fa6f4153890 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc00e9d0 tx=0x7fa6dc00ee90 comp rx=0 tx=0).stop 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 shutdown_connections 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6d0066f60 0x7fa6d0069420 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6f415be70 0x7fa6f40b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6f4159440 0x7fa6f4153890 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 --2- 172.21.15.67:0/651724181 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6f4071af0 0x7fa6f414ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 >> 172.21.15.67:0/651724181 conn(0x7fa6f40b1740 msgr2=0x7fa6f40713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 shutdown_connections 2024-04-17T10:27:13.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.134+0000 7fa6fa400640 1 -- 172.21.15.67:0/651724181 wait complete. 2024-04-17T10:27:13.182 INFO:tasks.cephfs.test_scrub_checks:num of strays: 2000 2024-04-17T10:27:13.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:27:13.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub start / recursive,scrub_mdsdir 2024-04-17T10:27:13.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27da000640 1 Processor -- start 2024-04-17T10:27:13.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27da000640 1 -- start start 2024-04-17T10:27:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27da000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d4168a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27da000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d40b4fb0 con 0x7f27d415df80 2024-04-17T10:27:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27da000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d40b5290 con 0x7f27d4070d30 2024-04-17T10:27:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27da000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40b1240 con 0x7f27d41608b0 2024-04-17T10:27:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27cbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27d415df80 0x7f27d40b4760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36290/0 (socket says 172.21.15.67:36290) 2024-04-17T10:27:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27cbe00640 1 -- 172.21.15.67:0/3387130030 learned_addr learned my addr 172.21.15.67:0/3387130030 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:13.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d3e00640 1 --2- 172.21.15.67:0/3387130030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d4168a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d2a00640 1 -- 172.21.15.67:0/3387130030 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2372182839 0 0) 0x7f27d40b4fb0 con 0x7f27d415df80 2024-04-17T10:27:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d2a00640 1 -- 172.21.15.67:0/3387130030 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b8003530 con 0x7f27d415df80 2024-04-17T10:27:13.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d2a00640 1 -- 172.21.15.67:0/3387130030 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390403645 0 0) 0x7f27d40b5290 con 0x7f27d4070d30 2024-04-17T10:27:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d2a00640 1 -- 172.21.15.67:0/3387130030 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d40b4fb0 con 0x7f27d4070d30 2024-04-17T10:27:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d3e00640 1 -- 172.21.15.67:0/3387130030 >> v1:172.21.15.86:6789/0 conn(0x7f27d4070d30 legacy=0x7f27d4071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d3e00640 1 -- 172.21.15.67:0/3387130030 >> v1:172.21.15.67:6789/0 conn(0x7f27d415df80 legacy=0x7f27d40b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.354+0000 7f27d3e00640 1 -- 172.21.15.67:0/3387130030 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d40b5570 con 0x7f27d41608b0 2024-04-17T10:27:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27d3e00640 1 --2- 172.21.15.67:0/3387130030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d4168a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009dc0 tx=0x7f27cc01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e2f355ee21ca75 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27d2a00640 1 -- 172.21.15.67:0/3387130030 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0b2070 con 0x7f27d41608b0 2024-04-17T10:27:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27d2a00640 1 -- 172.21.15.67:0/3387130030 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc0a57c0 con 0x7f27d41608b0 2024-04-17T10:27:13.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27d2a00640 1 -- 172.21.15.67:0/3387130030 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0a5b90 con 0x7f27d41608b0 2024-04-17T10:27:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27da000640 1 -- 172.21.15.67:0/3387130030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 msgr2=0x7f27d4168a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27da000640 1 --2- 172.21.15.67:0/3387130030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d4168a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009dc0 tx=0x7f27cc01c900 comp rx=0 tx=0).stop 2024-04-17T10:27:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27da000640 1 -- 172.21.15.67:0/3387130030 shutdown_connections 2024-04-17T10:27:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27da000640 1 --2- 172.21.15.67:0/3387130030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d4168a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27da000640 1 -- 172.21.15.67:0/3387130030 >> 172.21.15.67:0/3387130030 conn(0x7f27d406f5b0 msgr2=0x7f27d406f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27da000640 1 -- 172.21.15.67:0/3387130030 shutdown_connections 2024-04-17T10:27:13.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.358+0000 7f27da000640 1 -- 172.21.15.67:0/3387130030 wait complete. 2024-04-17T10:27:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 Processor -- start 2024-04-17T10:27:13.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 -- start start 2024-04-17T10:27:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d4070d30 0x7f27d414a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27d415df80 0x7f27d4149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d41546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d416bb60 con 0x7f27d4070d30 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d416b9e0 con 0x7f27d415df80 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d416bce0 con 0x7f27d41608b0 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d4070d30 0x7f27d414a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d4070d30 0x7f27d414a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44226/0 (socket says 172.21.15.67:44226) 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 -- 172.21.15.67:0/3967238308 learned_addr learned my addr 172.21.15.67:0/3967238308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27cbe00640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27d415df80 0x7f27d4149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3e00640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d41546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 -- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 msgr2=0x7f27d41546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d41546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 -- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27d415df80 msgr2=0x7f27d4149be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27d415df80 0x7f27d4149be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 -- 172.21.15.67:0/3967238308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d406c0e0 con 0x7f27d4070d30 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3e00640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d41546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27d3400640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d4070d30 0x7f27d414a600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c460 tx=0x7f27c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27cb400640 1 -- 172.21.15.67:0/3967238308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c40052d0 con 0x7f27d4070d30 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27cb400640 1 -- 172.21.15.67:0/3967238308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4005490 con 0x7f27d4070d30 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27cb400640 1 -- 172.21.15.67:0/3967238308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4012850 con 0x7f27d4070d30 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d406c3c0 con 0x7f27d4070d30 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.362+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d414a120 con 0x7f27d4070d30 2024-04-17T10:27:13.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.366+0000 7f27cb400640 1 -- 172.21.15.67:0/3967238308 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f27c40129f0 con 0x7f27d4070d30 2024-04-17T10:27:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.366+0000 7f27cb400640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27b0066f60 0x7f27b0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.366+0000 7f27cb400640 1 -- 172.21.15.67:0/3967238308 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7f27c4084ea0 con 0x7f27d4070d30 2024-04-17T10:27:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.366+0000 7f27cbe00640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27b0066f60 0x7f27b0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.366+0000 7f27dd59a000 1 Processor -- start 2024-04-17T10:27:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.366+0000 7f27dd59a000 1 -- start start 2024-04-17T10:27:13.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.366+0000 7f27cbe00640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27b0066f60 0x7f27b0069420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f27d41fa110 tx=0x7f27bc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72520 0x56232ba72900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232ba29430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba29d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232b876d00 con 0x56232ba72e40 2024-04-17T10:27:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232b7fedf0 con 0x56232ba72520 2024-04-17T10:27:13.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56232b872da0 con 0x56232ba29970 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72520 0x56232ba72900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232ba29430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72520 0x56232ba72900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39676/0 (socket says 172.21.15.67:39676) 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 -- 172.21.15.67:0/4075322955 learned_addr learned my addr 172.21.15.67:0/4075322955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3e00640 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba29d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 -- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 msgr2=0x56232ba29d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba29d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 -- 172.21.15.67:0/4075322955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 msgr2=0x56232ba29430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232ba29430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 -- 172.21.15.67:0/4075322955 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56232b831580 con 0x56232ba72520 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27d3400640 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72520 0x56232ba72900 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f27c40554f0 tx=0x7f27c4055620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=741b3f7c815eced4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27c3e00640 1 -- 172.21.15.67:0/4075322955 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4053860 con 0x56232ba72520 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27c3e00640 1 -- 172.21.15.67:0/4075322955 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4012ce0 con 0x56232ba72520 2024-04-17T10:27:13.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27c3e00640 1 -- 172.21.15.67:0/4075322955 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c405d7e0 con 0x56232ba72520 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72520 msgr2=0x56232ba72900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72520 0x56232ba72900 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f27c40554f0 tx=0x7f27c4055620 comp rx=0 tx=0).stop 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- 172.21.15.67:0/4075322955 shutdown_connections 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba29d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232ba29430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- 172.21.15.67:0/4075322955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72520 0x56232ba72900 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- 172.21.15.67:0/4075322955 >> 172.21.15.67:0/4075322955 conn(0x56232b910c10 msgr2=0x56232ba27d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- 172.21.15.67:0/4075322955 shutdown_connections 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- 172.21.15.67:0/4075322955 wait complete. 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 -- start start 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 Processor -- start 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba788e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.370+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72520 0x56232ba78e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232ba2d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232b80f6d0 con 0x56232ba72520 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232b831950 con 0x56232ba72e40 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56232b876010 con 0x56232ba29970 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 10 client.0 _ll_register_callbacks cb 0x56232ba15030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72520 0x56232ba78e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72520 0x56232ba78e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44240/0 (socket says 172.21.15.67:44240) 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 -- 172.21.15.67:0/2517608132 learned_addr learned my addr 172.21.15.67:0/2517608132 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27d3400640 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27d3e00640 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232ba2d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 -- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 msgr2=0x56232ba788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba788e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 -- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 msgr2=0x56232ba2d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232ba2d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 -- 172.21.15.67:0/2517608132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56232b813d40 con 0x56232ba72520 2024-04-17T10:27:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27cbe00640 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72520 0x56232ba78e20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f27bc000f70 tx=0x7f27bc001160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcd18e131071381 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 1 -- 172.21.15.67:0/2517608132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc028060 con 0x56232ba72520 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 1 -- 172.21.15.67:0/2517608132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7f27bc010040 con 0x56232ba72520 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27c4012ce0 con 0x56232ba72520 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 1 -- 172.21.15.67:0/2517608132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27bc007560 con 0x56232ba72520 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 10 client.5496 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x56232b871cf0 con 0x56232ba72520 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 1 -- 172.21.15.67:0/2517608132 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4012ce0 con 0x56232ba72520 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 1 -- 172.21.15.67:0/2517608132 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f27bc013420 con 0x56232ba72520 2024-04-17T10:27:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 10 client.5496 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 10 client.5496 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 20 client.5496 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232ba2e760 0x56232ba2eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 4 client.5496 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56232b8129d0 con 0x56232ba2e760 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27d3400640 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232ba2e760 0x56232ba2eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27d3400640 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232ba2e760 0x56232ba2eb70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63c1eed39346f235 server_cookie=649d87a8b1f61fa8 in_seq=0 out_seq=0 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 10 client.5496 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 1 -- 172.21.15.67:0/2517608132 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56232b8129d0 con 0x56232ba2e760 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 10 client.5496 handle_command_reply: tid=0 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 client.5496 shutdown 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72520 msgr2=0x56232ba78e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72520 0x56232ba78e20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f27bc000f70 tx=0x7f27bc001160 comp rx=0 tx=0).stop 2024-04-17T10:27:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 shutdown_connections 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232ba2e760 0x56232ba2eb70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232ba2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72520 0x56232ba78e20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 --2- 172.21.15.67:0/2517608132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba29970 0x56232ba788e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27c0c00640 0 client.5496 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 >> 172.21.15.67:0/2517608132 conn(0x56232b910c10 msgr2=0x56232ba28420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 shutdown_connections 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.374+0000 7f27dd59a000 1 -- 172.21.15.67:0/2517608132 wait complete. 2024-04-17T10:27:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.378+0000 7f27dd59a000 20 client.5496 trim_cache size 0 max 16384 2024-04-17T10:27:13.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 Processor -- start 2024-04-17T10:27:13.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- start start 2024-04-17T10:27:13.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba29970 0x56232bace060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bace5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232baceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4203310 con 0x56232ba29970 2024-04-17T10:27:13.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4202520 con 0x56232ba72e40 2024-04-17T10:27:13.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d406dae0 con 0x56232ba72520 2024-04-17T10:27:13.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba29970 0x56232bace060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba29970 0x56232bace060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44250/0 (socket says 172.21.15.67:44250) 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 -- 172.21.15.67:0/3689966305 learned_addr learned my addr 172.21.15.67:0/3689966305 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27cbe00640 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bace5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3e00640 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232baceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 -- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 msgr2=0x56232bace5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bace5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 -- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 msgr2=0x56232baceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232baceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 -- 172.21.15.67:0/3689966305 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56232b8f9d30 con 0x56232ba29970 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27cbe00640 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bace5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27d3400640 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba29970 0x56232bace060 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c430 tx=0x7f27c4058000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc0970532f4ed37 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27c3400640 1 -- 172.21.15.67:0/3689966305 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4053860 con 0x56232ba29970 2024-04-17T10:27:13.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27c3400640 1 -- 172.21.15.67:0/3689966305 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4051bd0 con 0x56232ba29970 2024-04-17T10:27:13.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27c3400640 1 -- 172.21.15.67:0/3689966305 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c405da20 con 0x56232ba29970 2024-04-17T10:27:13.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- 172.21.15.67:0/3689966305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba29970 msgr2=0x56232bace060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba29970 0x56232bace060 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c430 tx=0x7f27c4058000 comp rx=0 tx=0).stop 2024-04-17T10:27:13.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- 172.21.15.67:0/3689966305 shutdown_connections 2024-04-17T10:27:13.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba72e40 0x56232baceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bace5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 --2- 172.21.15.67:0/3689966305 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba29970 0x56232bace060 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- 172.21.15.67:0/3689966305 >> 172.21.15.67:0/3689966305 conn(0x56232b910c10 msgr2=0x56232ba71ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- 172.21.15.67:0/3689966305 shutdown_connections 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- 172.21.15.67:0/3689966305 wait complete. 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 -- start start 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.394+0000 7f27dd59a000 1 Processor -- start 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba29970 0x56232bab6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bab7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232bab7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56232b7ff7e0 con 0x56232ba72e40 2024-04-17T10:27:13.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40cf7e0 con 0x56232ba29970 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56232b80f6d0 con 0x56232ba72520 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 10 client.0 _ll_register_callbacks cb 0x7f27c4055330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232bab7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232bab7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44264/0 (socket says 172.21.15.67:44264) 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 -- 172.21.15.67:0/429210561 learned_addr learned my addr 172.21.15.67:0/429210561 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3400640 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba29970 0x56232bab6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27cbe00640 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bab7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 -- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 msgr2=0x56232bab7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bab7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 -- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba29970 msgr2=0x56232bab6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba29970 0x56232bab6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 -- 172.21.15.67:0/429210561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56232b8435a0 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3e00640 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232bab7b40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f27cc01c940 tx=0x7f27cc01b140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d3ec5495355e45 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27c0c00640 1 -- 172.21.15.67:0/429210561 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0b2040 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27c0c00640 1 -- 172.21.15.67:0/429210561 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7f27cc0c4c80 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f27c4051bd0 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27c0c00640 1 -- 172.21.15.67:0/429210561 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27cc0a5b10 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 10 client.5502 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x56232b81df90 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27c0c00640 1 -- 172.21.15.67:0/429210561 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4051bd0 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27c0c00640 1 -- 172.21.15.67:0/429210561 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f27cc0cb050 con 0x56232ba72e40 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 10 client.5502 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 10 client.5502 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:13.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 20 client.5502 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:13.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232bab8080 0x56232bab8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 4 client.5502 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "scrub_mdsdir"]}] 2024-04-17T10:27:13.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "scrub_mdsdir"]}) v1 -- 0x56232b853610 con 0x56232bab8080 2024-04-17T10:27:13.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3400640 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232bab8080 0x56232bab8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27d3400640 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232bab8080 0x56232bab8490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22d747183b9c7bf9 server_cookie=47f26ac7d8b84490 in_seq=0 out_seq=0 2024-04-17T10:27:13.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.398+0000 7f27c0c00640 10 client.5502 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27c0c00640 1 -- 172.21.15.67:0/429210561 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56232b853610 con 0x56232bab8080 2024-04-17T10:27:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27c0c00640 10 client.5502 handle_command_reply: tid=0 2024-04-17T10:27:13.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 client.5502 shutdown 2024-04-17T10:27:13.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 msgr2=0x56232bab7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232bab7b40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f27cc01c940 tx=0x7f27cc01b140 comp rx=0 tx=0).stop 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 shutdown_connections 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x56232bab8080 0x56232bab8490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56232ba72e40 0x56232bab7b40 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56232ba72520 0x56232bab7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 --2- 172.21.15.67:0/429210561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56232ba29970 0x56232bab6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 >> 172.21.15.67:0/429210561 conn(0x56232b910c10 msgr2=0x56232ba703a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27c0c00640 0 client.5502 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 shutdown_connections 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 1 -- 172.21.15.67:0/429210561 wait complete. 2024-04-17T10:27:13.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27dd59a000 20 client.5502 trim_cache size 0 max 16384 2024-04-17T10:27:13.402 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:27:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:27:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "367b75c5-8a09-4165-8015-f3fa989476c6", 2024-04-17T10:27:13.402 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:27:13.402 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27b0066f60 msgr2=0x7f27b0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27b0066f60 0x7f27b0069420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f27d41fa110 tx=0x7f27bc009040 comp rx=0 tx=0).stop 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d4070d30 msgr2=0x7f27d414a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d4070d30 0x7f27d414a600 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f27c400c460 tx=0x7f27c400c920 comp rx=0 tx=0).stop 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 shutdown_connections 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f27b0066f60 0x7f27b0069420 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27d41608b0 0x7f27d41546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27d415df80 0x7f27d4149be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 --2- 172.21.15.67:0/3967238308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27d4070d30 0x7f27d414a600 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 >> 172.21.15.67:0/3967238308 conn(0x7f27d406f5b0 msgr2=0x7f27d4073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 shutdown_connections 2024-04-17T10:27:13.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.406+0000 7f27da000640 1 -- 172.21.15.67:0/3967238308 wait complete. 2024-04-17T10:27:13.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:27:13.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-04-17T10:27:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facaec00640 1 Processor -- start 2024-04-17T10:27:13.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facaec00640 1 -- start start 2024-04-17T10:27:13.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facaec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80ad890 0x7faca80aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facaec00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faca806c3b0 0x7faca815d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facaec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca815c060 con 0x7faca80af520 2024-04-17T10:27:13.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facaec00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca815c340 con 0x7faca806c3b0 2024-04-17T10:27:13.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facaec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca815c620 con 0x7faca80ad890 2024-04-17T10:27:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facace00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faca80af520 0x7faca806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36300/0 (socket says 172.21.15.67:36300) 2024-04-17T10:27:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facace00640 1 -- 172.21.15.67:0/3694126831 learned_addr learned my addr 172.21.15.67:0/3694126831 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facae200640 1 --2- 172.21.15.67:0/3694126831 >> v2:172.21.15.86:3300/0 conn(0x7faca806c3b0 0x7faca815d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facad800640 1 --2- 172.21.15.67:0/3694126831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80ad890 0x7faca80aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7faca7e00640 1 -- 172.21.15.67:0/3694126831 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293068705 0 0) 0x7faca815c060 con 0x7faca80af520 2024-04-17T10:27:13.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7faca7e00640 1 -- 172.21.15.67:0/3694126831 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac90003530 con 0x7faca80af520 2024-04-17T10:27:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facad800640 1 -- 172.21.15.67:0/3694126831 >> v2:172.21.15.86:3300/0 conn(0x7faca806c3b0 msgr2=0x7faca815d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facad800640 1 --2- 172.21.15.67:0/3694126831 >> v2:172.21.15.86:3300/0 conn(0x7faca806c3b0 0x7faca815d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facad800640 1 -- 172.21.15.67:0/3694126831 >> v1:172.21.15.67:6789/0 conn(0x7faca80af520 legacy=0x7faca806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facad800640 1 -- 172.21.15.67:0/3694126831 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca815c900 con 0x7faca80ad890 2024-04-17T10:27:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.622+0000 7facae200640 1 --2- 172.21.15.67:0/3694126831 >> v2:172.21.15.86:3300/0 conn(0x7faca806c3b0 0x7faca815d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facad800640 1 --2- 172.21.15.67:0/3694126831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80ad890 0x7faca80aefe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0097c0 tx=0x7fac9c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ec9b4c0ddcecd16 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7faca7e00640 1 -- 172.21.15.67:0/3694126831 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0b1070 con 0x7faca80ad890 2024-04-17T10:27:13.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7faca7e00640 1 -- 172.21.15.67:0/3694126831 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c0ac410 con 0x7faca80ad890 2024-04-17T10:27:13.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7faca7e00640 1 -- 172.21.15.67:0/3694126831 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0ac5e0 con 0x7faca80ad890 2024-04-17T10:27:13.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 -- 172.21.15.67:0/3694126831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80ad890 msgr2=0x7faca80aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 --2- 172.21.15.67:0/3694126831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80ad890 0x7faca80aefe0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0097c0 tx=0x7fac9c01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 -- 172.21.15.67:0/3694126831 shutdown_connections 2024-04-17T10:27:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 --2- 172.21.15.67:0/3694126831 >> v2:172.21.15.86:3300/0 conn(0x7faca806c3b0 0x7faca815d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 --2- 172.21.15.67:0/3694126831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80ad890 0x7faca80aefe0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 -- 172.21.15.67:0/3694126831 >> 172.21.15.67:0/3694126831 conn(0x7faca8071af0 msgr2=0x7faca8071f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 -- 172.21.15.67:0/3694126831 shutdown_connections 2024-04-17T10:27:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.630+0000 7facaec00640 1 -- 172.21.15.67:0/3694126831 wait complete. 2024-04-17T10:27:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 Processor -- start 2024-04-17T10:27:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 -- start start 2024-04-17T10:27:13.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faca806c3b0 0x7faca814df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca80ad890 0x7faca8099b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80af520 0x7faca80b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca815e230 con 0x7faca80ad890 2024-04-17T10:27:13.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca815e0b0 con 0x7faca806c3b0 2024-04-17T10:27:13.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facaec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca815e3b0 con 0x7faca80af520 2024-04-17T10:27:13.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facad800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faca806c3b0 0x7faca814df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facad800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faca806c3b0 0x7faca814df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39698/0 (socket says 172.21.15.67:39698) 2024-04-17T10:27:13.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facad800640 1 -- 172.21.15.67:0/581908744 learned_addr learned my addr 172.21.15.67:0/581908744 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facae200640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80af520 0x7faca80b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facace00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca80ad890 0x7faca8099b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facad800640 1 -- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80af520 msgr2=0x7faca80b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facad800640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80af520 0x7faca80b0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.634+0000 7facad800640 1 -- 172.21.15.67:0/581908744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca80ad890 msgr2=0x7faca8099b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7facad800640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca80ad890 0x7faca8099b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7facad800640 1 -- 172.21.15.67:0/581908744 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca81fabf0 con 0x7faca806c3b0 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7facae200640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80af520 0x7faca80b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:13.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7facad800640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faca806c3b0 0x7faca814df20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fac9c01a0d0 tx=0x7fac9c019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7faca5600640 1 -- 172.21.15.67:0/581908744 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0b9020 con 0x7faca806c3b0 2024-04-17T10:27:13.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7faca5600640 1 -- 172.21.15.67:0/581908744 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c0acce0 con 0x7faca806c3b0 2024-04-17T10:27:13.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca81faed0 con 0x7faca806c3b0 2024-04-17T10:27:13.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7faca5600640 1 -- 172.21.15.67:0/581908744 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0b1040 con 0x7faca806c3b0 2024-04-17T10:27:13.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.638+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca814e460 con 0x7faca806c3b0 2024-04-17T10:27:13.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.642+0000 7faca5600640 1 -- 172.21.15.67:0/581908744 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fac9c0bc020 con 0x7faca806c3b0 2024-04-17T10:27:13.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.642+0000 7faca5600640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac88066f60 0x7fac88069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.642+0000 7faca5600640 1 -- 172.21.15.67:0/581908744 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7fac9c11e900 con 0x7faca806c3b0 2024-04-17T10:27:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.642+0000 7facace00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac88066f60 0x7fac88069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.642+0000 7facb3621000 1 Processor -- start 2024-04-17T10:27:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.642+0000 7facb3621000 1 -- start start 2024-04-17T10:27:13.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facace00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac88066f60 0x7fac88069420 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faca81f4ee0 tx=0x7fac94009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a8488db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a843eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a843f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632a828bff0 con 0x5632a84892f0 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632a8214150 con 0x5632a843f0a0 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5632a8288090 con 0x5632a84889d0 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a843f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a843f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39700/0 (socket says 172.21.15.67:39700) 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 -- 172.21.15.67:0/241378591 learned_addr learned my addr 172.21.15.67:0/241378591 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facace00640 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a843eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facad800640 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a8488db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 -- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 msgr2=0x5632a8488db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a8488db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 -- 172.21.15.67:0/241378591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 msgr2=0x5632a843eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a843eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 -- 172.21.15.67:0/241378591 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632a84220d0 con 0x5632a843f0a0 2024-04-17T10:27:13.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facace00640 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a843eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:13.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facae200640 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a843f480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7faca0003cf0 tx=0x7faca0009750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=434d40a17c689ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7fac9a000640 1 -- 172.21.15.67:0/241378591 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0019070 con 0x5632a843f0a0 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7fac9a000640 1 -- 172.21.15.67:0/241378591 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca000e6f0 con 0x5632a843f0a0 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7fac9a000640 1 -- 172.21.15.67:0/241378591 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca000eae0 con 0x5632a843f0a0 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 msgr2=0x5632a843f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a843f480 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7faca0003cf0 tx=0x7faca0009750 comp rx=0 tx=0).stop 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- 172.21.15.67:0/241378591 shutdown_connections 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a843f480 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a843eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 --2- 172.21.15.67:0/241378591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a8488db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- 172.21.15.67:0/241378591 >> 172.21.15.67:0/241378591 conn(0x5632a843b020 msgr2=0x5632a843d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- 172.21.15.67:0/241378591 shutdown_connections 2024-04-17T10:27:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.646+0000 7facb3621000 1 -- 172.21.15.67:0/241378591 wait complete. 2024-04-17T10:27:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 -- start start 2024-04-17T10:27:13.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 Processor -- start 2024-04-17T10:27:13.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a843f0a0 0x5632a846d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a846dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 0x5632a846e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632a8250250 con 0x5632a843f0a0 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632a8246690 con 0x5632a84892f0 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5632a8214a40 con 0x5632a84889d0 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facb3621000 10 client.0 _ll_register_callbacks cb 0x5632a8429e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facace00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a846dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facace00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a846dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:42458/0 (socket says 172.21.15.67:42458) 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facace00640 1 -- 172.21.15.67:0/3247698589 learned_addr learned my addr 172.21.15.67:0/3247698589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facad800640 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a843f0a0 0x5632a846d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facad800640 1 -- 172.21.15.67:0/3247698589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 msgr2=0x5632a846dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facad800640 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a846dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facad800640 1 -- 172.21.15.67:0/3247698589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 msgr2=0x5632a846e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facad800640 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 0x5632a846e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facad800640 1 -- 172.21.15.67:0/3247698589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632a83391c0 con 0x5632a843f0a0 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7facad800640 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a843f0a0 0x5632a846d7f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fac9c11cd60 tx=0x7fac9c0eea50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fad1ad99cc867b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.650+0000 7fac8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 1 -- 172.21.15.67:0/3247698589 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c0c42b0 con 0x5632a843f0a0 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 1 -- 172.21.15.67:0/3247698589 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7fac9c0ee540 con 0x5632a843f0a0 2024-04-17T10:27:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faca000e6f0 con 0x5632a843f0a0 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 1 -- 172.21.15.67:0/3247698589 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac9c0ef7f0 con 0x5632a843f0a0 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 10 client.5514 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5632a8335170 con 0x5632a843f0a0 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 1 -- 172.21.15.67:0/3247698589 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca000e6f0 con 0x5632a843f0a0 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 1 -- 172.21.15.67:0/3247698589 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fac9c0b4050 con 0x5632a843f0a0 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 10 client.5514 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 10 client.5514 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 20 client.5514 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a8498c80 0x5632a849b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 4 client.5514 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:27:13.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632a8227c60 con 0x5632a8498c80 2024-04-17T10:27:13.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facace00640 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a8498c80 0x5632a849b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facace00640 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a8498c80 0x5632a849b070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a657f52317e727d server_cookie=78a3749e03ee53a9 in_seq=0 out_seq=0 2024-04-17T10:27:13.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 10 client.5514 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 1 -- 172.21.15.67:0/3247698589 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5632a8227c60 con 0x5632a8498c80 2024-04-17T10:27:13.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7fac8ea00640 10 client.5514 handle_command_reply: tid=0 2024-04-17T10:27:13.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.654+0000 7facb3621000 1 client.5514 shutdown 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a843f0a0 msgr2=0x5632a846d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a843f0a0 0x5632a846d7f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fac9c11cd60 tx=0x7fac9c0eea50 comp rx=0 tx=0).stop 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 shutdown_connections 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a8498c80 0x5632a849b070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 0x5632a846e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a846dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7fac8ea00640 0 client.5514 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 --2- 172.21.15.67:0/3247698589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a843f0a0 0x5632a846d7f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 >> 172.21.15.67:0/3247698589 conn(0x5632a843b020 msgr2=0x5632a843dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 shutdown_connections 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 1 -- 172.21.15.67:0/3247698589 wait complete. 2024-04-17T10:27:13.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.658+0000 7facb3621000 20 client.5514 trim_cache size 0 max 16384 2024-04-17T10:27:13.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 Processor -- start 2024-04-17T10:27:13.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 -- start start 2024-04-17T10:27:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a84e50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a84e5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a84e5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca82073d0 con 0x5632a84892f0 2024-04-17T10:27:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca8206530 con 0x5632a843f0a0 2024-04-17T10:27:13.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca809d3f0 con 0x5632a84889d0 2024-04-17T10:27:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facad800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a84e50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facace00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a84e5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facace00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a84e5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:42470/0 (socket says 172.21.15.67:42470) 2024-04-17T10:27:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facae200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a84e5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facace00640 1 -- 172.21.15.67:0/3101959697 learned_addr learned my addr 172.21.15.67:0/3101959697 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facae200640 1 -- 172.21.15.67:0/3101959697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 msgr2=0x5632a84e5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facae200640 1 --2- 172.21.15.67:0/3101959697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a84e5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facae200640 1 -- 172.21.15.67:0/3101959697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 msgr2=0x5632a84e50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facae200640 1 --2- 172.21.15.67:0/3101959697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a84e50f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facae200640 1 -- 172.21.15.67:0/3101959697 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632a82d2de0 con 0x5632a84892f0 2024-04-17T10:27:13.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.682+0000 7facae200640 1 --2- 172.21.15.67:0/3101959697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a84e5b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7faca001b000 tx=0x7faca0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f73b8534ed9e1e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7fac99600640 1 -- 172.21.15.67:0/3101959697 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0019040 con 0x5632a84892f0 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7fac99600640 1 -- 172.21.15.67:0/3101959697 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca000ea70 con 0x5632a84892f0 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7fac99600640 1 -- 172.21.15.67:0/3101959697 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca000ee60 con 0x5632a84892f0 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- 172.21.15.67:0/3101959697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 msgr2=0x5632a84e5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 --2- 172.21.15.67:0/3101959697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a84e5b70 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7faca001b000 tx=0x7faca0015000 comp rx=0 tx=0).stop 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- 172.21.15.67:0/3101959697 shutdown_connections 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 --2- 172.21.15.67:0/3101959697 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84892f0 0x5632a84e5b70 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 --2- 172.21.15.67:0/3101959697 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a84889d0 0x5632a84e5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 --2- 172.21.15.67:0/3101959697 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a843f0a0 0x5632a84e50f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- 172.21.15.67:0/3101959697 >> 172.21.15.67:0/3101959697 conn(0x5632a843b020 msgr2=0x5632a8495fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- 172.21.15.67:0/3101959697 shutdown_connections 2024-04-17T10:27:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- 172.21.15.67:0/3101959697 wait complete. 2024-04-17T10:27:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- start start 2024-04-17T10:27:13.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 Processor -- start 2024-04-17T10:27:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a843f0a0 0x5632a849b380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84889d0 0x5632a849b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 0x5632a849be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632a8226e00 con 0x5632a84889d0 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faca8206090 con 0x5632a84892f0 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5632a828b300 con 0x5632a843f0a0 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.686+0000 7facb3621000 10 client.0 _ll_register_callbacks cb 0x5632a840a040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a843f0a0 0x5632a849b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a843f0a0 0x5632a849b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:42482/0 (socket says 172.21.15.67:42482) 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 -- 172.21.15.67:0/2047867928 learned_addr learned my addr 172.21.15.67:0/2047867928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facae200640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 0x5632a849be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facace00640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84889d0 0x5632a849b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 -- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 msgr2=0x5632a849be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 0x5632a849be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 -- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84889d0 msgr2=0x5632a849b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84889d0 0x5632a849b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 -- 172.21.15.67:0/2047867928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632a822d8e0 con 0x5632a843f0a0 2024-04-17T10:27:13.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facace00640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84889d0 0x5632a849b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facad800640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a843f0a0 0x5632a849b380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0eb550 tx=0x7fac9c0c0070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b68206174d0acb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3301/0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 1 -- 172.21.15.67:0/2047867928 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9c10b080 con 0x5632a843f0a0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 1 -- 172.21.15.67:0/2047867928 <== mon.2 v2:172.21.15.86:3301/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6130+0+0 (secure 0 0 0) 0x7fac9c10ed90 con 0x5632a843f0a0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faca000ea70 con 0x5632a843f0a0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 1 -- 172.21.15.67:0/2047867928 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7fac9c0eeec0 con 0x5632a843f0a0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 10 client.5147 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5632a82e58f0 con 0x5632a843f0a0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 1 -- 172.21.15.67:0/2047867928 <== mon.2 v2:172.21.15.86:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca000ea70 con 0x5632a843f0a0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 1 -- 172.21.15.67:0/2047867928 <== mon.2 v2:172.21.15.86:3301/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fac9c0b4040 con 0x5632a843f0a0 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 10 client.5147 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:13.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 10 client.5147 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 20 client.5147 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a849c700 0x5632a84f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 4 client.5147 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5632a8299530 con 0x5632a849c700 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facace00640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a849c700 0x5632a84f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facace00640 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a849c700 0x5632a84f0f60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66f29fe7e9f7a2e8 server_cookie=fecce61700fc1f25 in_seq=0 out_seq=0 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 10 client.5147 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 1 -- 172.21.15.67:0/2047867928 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x5632a8299530 con 0x5632a849c700 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 10 client.5147 handle_command_reply: tid=0 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 client.5147 shutdown 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a843f0a0 msgr2=0x5632a849b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a843f0a0 0x5632a849b380 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fac9c0eb550 tx=0x7fac9c0c0070 comp rx=0 tx=0).stop 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 shutdown_connections 2024-04-17T10:27:13.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5632a849c700 0x5632a84f0f60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5632a84892f0 0x5632a849be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5632a84889d0 0x5632a849b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 --2- 172.21.15.67:0/2047867928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5632a843f0a0 0x5632a849b380 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7fac8ea00640 0 client.5147 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 >> 172.21.15.67:0/2047867928 conn(0x5632a843b020 msgr2=0x5632a84870a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 shutdown_connections 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 1 -- 172.21.15.67:0/2047867928 wait complete. 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.690+0000 7facb3621000 20 client.5147 trim_cache size 0 max 16384 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:27:13.690 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout: "367b75c5-8a09-4165-8015-f3fa989476c6": { 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/", 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "367b75c5-8a09-4165-8015-f3fa989476c6", 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive,scrub_mdsdir" 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:13.691 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac88066f60 msgr2=0x7fac88069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac88066f60 0x7fac88069420 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7faca81f4ee0 tx=0x7fac94009040 comp rx=0 tx=0).stop 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faca806c3b0 msgr2=0x7faca814df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faca806c3b0 0x7faca814df20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fac9c01a0d0 tx=0x7fac9c019240 comp rx=0 tx=0).stop 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 shutdown_connections 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac88066f60 0x7fac88069420 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faca80af520 0x7faca80b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faca80ad890 0x7faca8099b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 --2- 172.21.15.67:0/581908744 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faca806c3b0 0x7faca814df20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 >> 172.21.15.67:0/581908744 conn(0x7faca8071af0 msgr2=0x7faca8158150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 shutdown_connections 2024-04-17T10:27:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:13.694+0000 7facaec00640 1 -- 172.21.15.67:0/581908744 wait complete. 2024-04-17T10:27:13.705 INFO:tasks.cephfs.filesystem:scrub status for tag:367b75c5-8a09-4165-8015-f3fa989476c6 - {'path': '/', 'tag': '367b75c5-8a09-4165-8015-f3fa989476c6', 'options': 'recursive,scrub_mdsdir'} 2024-04-17T10:27:22.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:27:22.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:27:43.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:27:43.707 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.2:0 scrub status 2024-04-17T10:27:43.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.866+0000 7f6877e00640 1 Processor -- start 2024-04-17T10:27:43.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.866+0000 7f6877e00640 1 -- start start 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6877e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6870159440 0x7f68700ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6877e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f687015c580 0x7f6870071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6877e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687015cb30 con 0x7f68700713e0 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6877e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700a8650 con 0x7f6870159440 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6877e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687015ce10 con 0x7f687015c580 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6870159440 0x7f68700ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6870159440 0x7f68700ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40392/0 (socket says 172.21.15.67:40392) 2024-04-17T10:27:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875400640 1 -- 172.21.15.67:0/333596167 learned_addr learned my addr 172.21.15.67:0/333596167 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875400640 1 -- 172.21.15.67:0/333596167 >> v2:172.21.15.86:3300/0 conn(0x7f687015c580 msgr2=0x7f6870071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875e00640 1 --2- 172.21.15.67:0/333596167 >> v2:172.21.15.86:3300/0 conn(0x7f687015c580 0x7f6870071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875400640 1 --2- 172.21.15.67:0/333596167 >> v2:172.21.15.86:3300/0 conn(0x7f687015c580 0x7f6870071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875400640 1 -- 172.21.15.67:0/333596167 >> v1:172.21.15.86:6789/0 conn(0x7f68700713e0 legacy=0x7f687015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875400640 1 -- 172.21.15.67:0/333596167 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700cd610 con 0x7f6870159440 2024-04-17T10:27:43.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.870+0000 7f6875e00640 1 --2- 172.21.15.67:0/333596167 >> v2:172.21.15.86:3300/0 conn(0x7f687015c580 0x7f6870071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:43.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6875400640 1 --2- 172.21.15.67:0/333596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6870159440 0x7f68700ce0f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f68700ce660 tx=0x7f686401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c08fcfdb40f79d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f686fe00640 1 -- 172.21.15.67:0/333596167 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68640ab5f0 con 0x7f6870159440 2024-04-17T10:27:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f686fe00640 1 -- 172.21.15.67:0/333596167 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68640b0070 con 0x7f6870159440 2024-04-17T10:27:43.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f686fe00640 1 -- 172.21.15.67:0/333596167 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68640ab9b0 con 0x7f6870159440 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 -- 172.21.15.67:0/333596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6870159440 msgr2=0x7f68700ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 --2- 172.21.15.67:0/333596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6870159440 0x7f68700ce0f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f68700ce660 tx=0x7f686401c4e0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 -- 172.21.15.67:0/333596167 shutdown_connections 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 --2- 172.21.15.67:0/333596167 >> v2:172.21.15.86:3300/0 conn(0x7f687015c580 0x7f6870071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 --2- 172.21.15.67:0/333596167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6870159440 0x7f68700ce0f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 -- 172.21.15.67:0/333596167 >> 172.21.15.67:0/333596167 conn(0x7f68700b1740 msgr2=0x7f68700b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 -- 172.21.15.67:0/333596167 shutdown_connections 2024-04-17T10:27:43.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 -- 172.21.15.67:0/333596167 wait complete. 2024-04-17T10:27:43.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 Processor -- start 2024-04-17T10:27:43.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 -- start start 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68700713e0 0x7f68701401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6870159440 0x7f687015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.874+0000 7f6877e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f687015c580 0x7f6870140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6877e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687015d9c0 con 0x7f68700713e0 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6877e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687015d840 con 0x7f6870159440 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6877e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687015db40 con 0x7f687015c580 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68700713e0 0x7f68701401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68700713e0 0x7f68701401a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40402/0 (socket says 172.21.15.67:40402) 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 -- 172.21.15.67:0/511840222 learned_addr learned my addr 172.21.15.67:0/511840222 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6874a00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6870159440 0x7f687015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875e00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f687015c580 0x7f6870140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 -- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f687015c580 msgr2=0x7f6870140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f687015c580 0x7f6870140b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 -- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6870159440 msgr2=0x7f687015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6870159440 0x7f687015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 -- 172.21.15.67:0/511840222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6870152340 con 0x7f68700713e0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6874a00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6870159440 0x7f687015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6875400640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68700713e0 0x7f68701401a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f68640ad040 tx=0x7f686401ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f686d600640 1 -- 172.21.15.67:0/511840222 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68640be5f0 con 0x7f68700713e0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f686d600640 1 -- 172.21.15.67:0/511840222 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68640be7b0 con 0x7f68700713e0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f686d600640 1 -- 172.21.15.67:0/511840222 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68640beba0 con 0x7f68700713e0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6870152620 con 0x7f68700713e0 2024-04-17T10:27:43.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6870152be0 con 0x7f68700713e0 2024-04-17T10:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.878+0000 7f686d600640 1 -- 172.21.15.67:0/511840222 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f68640b43f0 con 0x7f68700713e0 2024-04-17T10:27:43.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f686d600640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6850066f90 0x7f6850069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f686d600640 1 -- 172.21.15.67:0/511840222 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f686411fb50 con 0x7f68700713e0 2024-04-17T10:27:43.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6874a00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6850066f90 0x7f6850069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 Processor -- start 2024-04-17T10:27:43.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- start start 2024-04-17T10:27:43.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6874a00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6850066f90 0x7f6850069450 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f68700b4630 tx=0x7f685c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25eb1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25eb22f0 0x558d25e67b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25e680a0 0x558d25e68480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d25cb4ff0 con 0x558d25eb19d0 2024-04-17T10:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d25c3d150 con 0x558d25e680a0 2024-04-17T10:27:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d25cb1090 con 0x558d25eb22f0 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25eb1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25eb1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40406/0 (socket says 172.21.15.67:40406) 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 -- 172.21.15.67:0/2708855350 learned_addr learned my addr 172.21.15.67:0/2708855350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6874a00640 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25eb22f0 0x558d25e67b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875e00640 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25e680a0 0x558d25e68480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 -- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25eb22f0 msgr2=0x558d25e67b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25eb22f0 0x558d25e67b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 -- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25e680a0 msgr2=0x558d25e68480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25e680a0 0x558d25e68480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 -- 172.21.15.67:0/2708855350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d25e4b0d0 con 0x558d25eb19d0 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6875400640 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25eb1db0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f68640f0340 tx=0x7f6864019280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aacb6e957882213 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6862000640 1 -- 172.21.15.67:0/2708855350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68640f4500 con 0x558d25eb19d0 2024-04-17T10:27:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6862000640 1 -- 172.21.15.67:0/2708855350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68640b0050 con 0x558d25eb19d0 2024-04-17T10:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f6862000640 1 -- 172.21.15.67:0/2708855350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68640f8ac0 con 0x558d25eb19d0 2024-04-17T10:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- 172.21.15.67:0/2708855350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 msgr2=0x558d25eb1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25eb1db0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f68640f0340 tx=0x7f6864019280 comp rx=0 tx=0).stop 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- 172.21.15.67:0/2708855350 shutdown_connections 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25e680a0 0x558d25e68480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25eb22f0 0x558d25e67b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 --2- 172.21.15.67:0/2708855350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25eb1db0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- 172.21.15.67:0/2708855350 >> 172.21.15.67:0/2708855350 conn(0x558d25e64020 msgr2=0x558d25e66440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- 172.21.15.67:0/2708855350 shutdown_connections 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.882+0000 7f687b269000 1 -- 172.21.15.67:0/2708855350 wait complete. 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 -- start start 2024-04-17T10:27:43.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 Processor -- start 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25e967f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb19d0 0x558d25e96d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25e97270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d25c79250 con 0x558d25eb22f0 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d25c6f690 con 0x558d25eb19d0 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d25c3da40 con 0x558d25e680a0 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 10 client.0 _ll_register_callbacks cb 0x558d25e52e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25e97270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25e97270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40422/0 (socket says 172.21.15.67:40422) 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 -- 172.21.15.67:0/2683653495 learned_addr learned my addr 172.21.15.67:0/2683653495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6874a00640 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb19d0 0x558d25e96d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875400640 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25e967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 -- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 msgr2=0x558d25e967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25e967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 -- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb19d0 msgr2=0x558d25e96d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb19d0 0x558d25e96d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 -- 172.21.15.67:0/2683653495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d25d621c0 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875e00640 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25e97270 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6868010b70 tx=0x7f686800e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5e5d19f0ee9bb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6856a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6856a00640 1 -- 172.21.15.67:0/2683653495 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f686801d060 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f68640b0050 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6856a00640 1 -- 172.21.15.67:0/2683653495 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f686800a780 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6856a00640 1 -- 172.21.15.67:0/2683653495 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6868023050 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 10 client.5529 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x558d25d5e170 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6856a00640 1 -- 172.21.15.67:0/2683653495 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68640b0050 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6856a00640 1 -- 172.21.15.67:0/2683653495 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6868026020 con 0x558d25eb22f0 2024-04-17T10:27:43.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 10 client.5529 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 10 client.5529 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 20 client.5529 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25ec1c80 0x558d25ec4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 4 client.5529 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558d25c50c60 con 0x558d25ec1c80 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.886+0000 7f6875400640 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25ec1c80 0x558d25ec4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f6875400640 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25ec1c80 0x558d25ec4070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c62f579b8387413a server_cookie=553101c0b411a2c6 in_seq=0 out_seq=0 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f6856a00640 10 client.5529 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f6856a00640 1 -- 172.21.15.67:0/2683653495 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558d25c50c60 con 0x558d25ec1c80 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f6856a00640 10 client.5529 handle_command_reply: tid=0 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 client.5529 shutdown 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 msgr2=0x558d25e97270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25e97270 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6868010b70 tx=0x7f686800e750 comp rx=0 tx=0).stop 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 shutdown_connections 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25ec1c80 0x558d25ec4070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25e97270 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb19d0 0x558d25e96d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f6856a00640 0 client.5529 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 --2- 172.21.15.67:0/2683653495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25e967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 >> 172.21.15.67:0/2683653495 conn(0x558d25e64020 msgr2=0x558d25e66ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 shutdown_connections 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 1 -- 172.21.15.67:0/2683653495 wait complete. 2024-04-17T10:27:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.890+0000 7f687b269000 20 client.5529 trim_cache size 0 max 16384 2024-04-17T10:27:43.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 Processor -- start 2024-04-17T10:27:43.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- start start 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25f16900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25f16e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb22f0 0x558d25f17380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68702033d0 con 0x558d25eb19d0 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68702025d0 con 0x558d25eb22f0 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700affc0 con 0x558d25e680a0 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25f16e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25f16e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40436/0 (socket says 172.21.15.67:40436) 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 -- 172.21.15.67:0/3564000032 learned_addr learned my addr 172.21.15.67:0/3564000032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6875400640 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25f16900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6875e00640 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb22f0 0x558d25f17380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 -- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 msgr2=0x558d25f16900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25f16900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 -- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb22f0 msgr2=0x558d25f17380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb22f0 0x558d25f17380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 -- 172.21.15.67:0/3564000032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558d25cfbde0 con 0x558d25eb19d0 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6875400640 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25f16900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6875e00640 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb22f0 0x558d25f17380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6874a00640 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25f16e40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f685c0051d0 tx=0x7f685c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf5044e360a382e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6861600640 1 -- 172.21.15.67:0/3564000032 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c028070 con 0x558d25eb19d0 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6861600640 1 -- 172.21.15.67:0/3564000032 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c003810 con 0x558d25eb19d0 2024-04-17T10:27:43.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f6861600640 1 -- 172.21.15.67:0/3564000032 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c0039c0 con 0x558d25eb19d0 2024-04-17T10:27:43.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- 172.21.15.67:0/3564000032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 msgr2=0x558d25f16e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25f16e40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f685c0051d0 tx=0x7f685c012000 comp rx=0 tx=0).stop 2024-04-17T10:27:43.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- 172.21.15.67:0/3564000032 shutdown_connections 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25eb22f0 0x558d25f17380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb19d0 0x558d25f16e40 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 --2- 172.21.15.67:0/3564000032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25f16900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- 172.21.15.67:0/3564000032 >> 172.21.15.67:0/3564000032 conn(0x558d25e64020 msgr2=0x558d25eb0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- 172.21.15.67:0/3564000032 shutdown_connections 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- 172.21.15.67:0/3564000032 wait complete. 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.906+0000 7f687b269000 1 -- start start 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 Processor -- start 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25ec4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25ec4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25ec51f0 0x558d25ec55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x558d25c4fe00 con 0x558d25eb22f0 2024-04-17T10:27:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68700b4090 con 0x558d25ec51f0 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x558d25cb4300 con 0x558d25e680a0 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 10 client.0 _ll_register_callbacks cb 0x558d25e33040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25ec4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25ec4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40442/0 (socket says 172.21.15.67:40442) 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 -- 172.21.15.67:0/2466146396 learned_addr learned my addr 172.21.15.67:0/2466146396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6875400640 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25ec4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6875e00640 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25ec51f0 0x558d25ec55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 -- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 msgr2=0x558d25ec4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25ec4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 -- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25ec51f0 msgr2=0x558d25ec55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25ec51f0 0x558d25ec55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 -- 172.21.15.67:0/2466146396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558d25c568e0 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6874a00640 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25ec4cb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f685c026040 tx=0x7f685c000fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea6a248413673f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 1 -- 172.21.15.67:0/2466146396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f685c028070 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 1 -- 172.21.15.67:0/2466146396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f685c02dc70 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f685c003810 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 1 -- 172.21.15.67:0/2466146396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f685c02deb0 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 10 client.5535 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x558d25d0e8f0 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 1 -- 172.21.15.67:0/2466146396 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c003810 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 1 -- 172.21.15.67:0/2466146396 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f685c023470 con 0x558d25eb22f0 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 10 client.5535 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 10 client.5535 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 20 client.5535 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:43.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25f20760 0x558d25f22b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 4 client.5535 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:27:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558d25cc2530 con 0x558d25f20760 2024-04-17T10:27:43.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6875400640 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25f20760 0x558d25f22b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6875400640 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25f20760 0x558d25f22b50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cc31a82c6f82907 server_cookie=67d82b81a2c397f in_seq=0 out_seq=0 2024-04-17T10:27:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 10 client.5535 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 1 -- 172.21.15.67:0/2466146396 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558d25cc2530 con 0x558d25f20760 2024-04-17T10:27:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f6856a00640 10 client.5535 handle_command_reply: tid=0 2024-04-17T10:27:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 client.5535 shutdown 2024-04-17T10:27:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 msgr2=0x558d25ec4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.910+0000 7f687b269000 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25ec4cb0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f685c026040 tx=0x7f685c000fc0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 shutdown_connections 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x558d25f20760 0x558d25f22b50 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x558d25ec51f0 0x558d25ec55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x558d25eb22f0 0x558d25ec4cb0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 --2- 172.21.15.67:0/2466146396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x558d25e680a0 0x558d25ec4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6856a00640 0 client.5535 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 >> 172.21.15.67:0/2466146396 conn(0x558d25e64020 msgr2=0x558d25eb0900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 shutdown_connections 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 1 -- 172.21.15.67:0/2466146396 wait complete. 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f687b269000 20 client.5535 trim_cache size 0 max 16384 2024-04-17T10:27:43.911 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:27:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-04-17T10:27:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:27:43.912 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:27:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6850066f90 msgr2=0x7f6850069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6850066f90 0x7f6850069450 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f68700b4630 tx=0x7f685c009040 comp rx=0 tx=0).stop 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68700713e0 msgr2=0x7f68701401a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68700713e0 0x7f68701401a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f68640ad040 tx=0x7f686401ad20 comp rx=0 tx=0).stop 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 shutdown_connections 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6850066f90 0x7f6850069450 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f687015c580 0x7f6870140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6870159440 0x7f687015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 --2- 172.21.15.67:0/511840222 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f68700713e0 0x7f68701401a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 >> 172.21.15.67:0/511840222 conn(0x7f68700b1740 msgr2=0x7f687006d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 shutdown_connections 2024-04-17T10:27:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:43.914+0000 7f6877e00640 1 -- 172.21.15.67:0/511840222 wait complete. 2024-04-17T10:27:43.926 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-17T10:27:43.926 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:27:43.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 perf dump 2024-04-17T10:27:44.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f86a5600640 1 Processor -- start 2024-04-17T10:27:44.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f86a5600640 1 -- start start 2024-04-17T10:27:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f86a5600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f86a01583f0 0x7f86a0072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f86a5600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a015f360 con 0x7f86a00b5300 2024-04-17T10:27:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f86a5600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a00b3160 con 0x7f86a00728c0 2024-04-17T10:27:44.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f86a5600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86a00b3440 con 0x7f86a01583f0 2024-04-17T10:27:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f86a00b5300 0x7f86a006ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43126/0 (socket says 172.21.15.67:43126) 2024-04-17T10:27:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869f400640 1 -- 172.21.15.67:0/1214375113 learned_addr learned my addr 172.21.15.67:0/1214375113 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869ea00640 1 --2- 172.21.15.67:0/1214375113 >> v2:172.21.15.86:3300/0 conn(0x7f86a01583f0 0x7f86a0072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869d600640 1 -- 172.21.15.67:0/1214375113 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283109926 0 0) 0x7f86a015f360 con 0x7f86a00b5300 2024-04-17T10:27:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869d600640 1 -- 172.21.15.67:0/1214375113 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8684003530 con 0x7f86a00b5300 2024-04-17T10:27:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869ea00640 1 -- 172.21.15.67:0/1214375113 >> v1:172.21.15.86:6789/0 conn(0x7f86a00728c0 legacy=0x7f86a00b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869d600640 1 -- 172.21.15.67:0/1214375113 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817118016 0 0) 0x7f86a00b3160 con 0x7f86a00728c0 2024-04-17T10:27:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869ea00640 1 -- 172.21.15.67:0/1214375113 >> v1:172.21.15.67:6789/0 conn(0x7f86a00b5300 legacy=0x7f86a006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.078+0000 7f869ea00640 1 -- 172.21.15.67:0/1214375113 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a00b3720 con 0x7f86a01583f0 2024-04-17T10:27:44.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f869ea00640 1 --2- 172.21.15.67:0/1214375113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01583f0 0x7f86a0072380 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f86900097c0 tx=0x7f869001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b9bf8eeeb78f21e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f869d600640 1 -- 172.21.15.67:0/1214375113 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86900b1070 con 0x7f86a01583f0 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f869d600640 1 -- 172.21.15.67:0/1214375113 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86900ac5f0 con 0x7f86a01583f0 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f869d600640 1 -- 172.21.15.67:0/1214375113 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86900ac9c0 con 0x7f86a01583f0 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f86a5600640 1 -- 172.21.15.67:0/1214375113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01583f0 msgr2=0x7f86a0072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f86a5600640 1 --2- 172.21.15.67:0/1214375113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01583f0 0x7f86a0072380 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f86900097c0 tx=0x7f869001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f86a5600640 1 -- 172.21.15.67:0/1214375113 shutdown_connections 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f86a5600640 1 --2- 172.21.15.67:0/1214375113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a01583f0 0x7f86a0072380 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f86a5600640 1 -- 172.21.15.67:0/1214375113 >> 172.21.15.67:0/1214375113 conn(0x7f86a00b2570 msgr2=0x7f86a0071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f86a5600640 1 -- 172.21.15.67:0/1214375113 shutdown_connections 2024-04-17T10:27:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.082+0000 7f86a5600640 1 -- 172.21.15.67:0/1214375113 wait complete. 2024-04-17T10:27:44.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 Processor -- start 2024-04-17T10:27:44.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 -- start start 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a00728c0 0x7f86a013eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a00b5300 0x7f86a013f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86a01583f0 0x7f86a013f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a016bb00 con 0x7f86a01583f0 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a00b4200 con 0x7f86a00728c0 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a016bc80 con 0x7f86a00b5300 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86a01583f0 0x7f86a013f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86a01583f0 0x7f86a013f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40448/0 (socket says 172.21.15.67:40448) 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 -- 172.21.15.67:0/552287524 learned_addr learned my addr 172.21.15.67:0/552287524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869ea00640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a00728c0 0x7f86a013eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869e000640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a00b5300 0x7f86a013f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 -- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a00b5300 msgr2=0x7f86a013f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a00b5300 0x7f86a013f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 -- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a00728c0 msgr2=0x7f86a013eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a00728c0 0x7f86a013eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 -- 172.21.15.67:0/552287524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a0140670 con 0x7f86a01583f0 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f869f400640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86a01583f0 0x7f86a013f910 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f869800cb40 tx=0x7f869800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f8696a00640 1 -- 172.21.15.67:0/552287524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8698017070 con 0x7f86a01583f0 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f8696a00640 1 -- 172.21.15.67:0/552287524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869800aca0 con 0x7f86a01583f0 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f8696a00640 1 -- 172.21.15.67:0/552287524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8698012620 con 0x7f86a01583f0 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a01577a0 con 0x7f86a01583f0 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.086+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a0157d60 con 0x7f86a01583f0 2024-04-17T10:27:44.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f8696a00640 1 -- 172.21.15.67:0/552287524 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f869801a070 con 0x7f86a01583f0 2024-04-17T10:27:44.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f8696a00640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8680066f60 0x7f8680069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f8696a00640 1 -- 172.21.15.67:0/552287524 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f8698084f60 con 0x7f86a01583f0 2024-04-17T10:27:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f869ea00640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8680066f60 0x7f8680069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 Processor -- start 2024-04-17T10:27:44.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 -- start start 2024-04-17T10:27:44.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f869ea00640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8680066f60 0x7f8680069420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f86900ae000 tx=0x7f86900a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26aa40 0x5616bc26ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26b360 0x5616bc225850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc225d90 0x5616bc226170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bc1081f0 con 0x5616bc225d90 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bc06ee90 con 0x5616bc26b360 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.090+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616bbff6f50 con 0x5616bc26aa40 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc225d90 0x5616bc226170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26b360 0x5616bc225850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc225d90 0x5616bc226170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40464/0 (socket says 172.21.15.67:40464) 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26aa40 0x5616bc26ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26b360 0x5616bc225850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:58474/0 (socket says 172.21.15.67:58474) 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 -- 172.21.15.67:0/223616493 learned_addr learned my addr 172.21.15.67:0/223616493 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 -- 172.21.15.67:0/223616493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26aa40 msgr2=0x5616bc26ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 --2- 172.21.15.67:0/223616493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26aa40 0x5616bc26ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 -- 172.21.15.67:0/223616493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26b360 msgr2=0x5616bc225850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 --2- 172.21.15.67:0/223616493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26b360 0x5616bc225850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 -- 172.21.15.67:0/223616493 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616bc0468b0 con 0x5616bc225d90 2024-04-17T10:27:44.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 --2- 172.21.15.67:0/223616493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc225d90 0x5616bc226170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f869805d880 tx=0x7f86980597f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d66115efa64d521e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f868b400640 1 -- 172.21.15.67:0/223616493 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8698053a50 con 0x5616bc225d90 2024-04-17T10:27:44.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f868b400640 1 -- 172.21.15.67:0/223616493 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8698051dc0 con 0x5616bc225d90 2024-04-17T10:27:44.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f868b400640 1 -- 172.21.15.67:0/223616493 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869805db20 con 0x5616bc225d90 2024-04-17T10:27:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- 172.21.15.67:0/223616493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc225d90 msgr2=0x5616bc226170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 --2- 172.21.15.67:0/223616493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc225d90 0x5616bc226170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f869805d880 tx=0x7f86980597f0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- 172.21.15.67:0/223616493 shutdown_connections 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 --2- 172.21.15.67:0/223616493 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc225d90 0x5616bc226170 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 --2- 172.21.15.67:0/223616493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26b360 0x5616bc225850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 --2- 172.21.15.67:0/223616493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26aa40 0x5616bc26ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- 172.21.15.67:0/223616493 >> 172.21.15.67:0/223616493 conn(0x5616bc221d40 msgr2=0x5616bc224130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- 172.21.15.67:0/223616493 shutdown_connections 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- 172.21.15.67:0/223616493 wait complete. 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- start start 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 Processor -- start 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc225d90 0x5616bc2721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26aa40 0x5616bc272700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26b360 0x5616bc22bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bc029540 con 0x5616bc26b360 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bc06af30 con 0x5616bc26aa40 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616bc04d010 con 0x5616bc225d90 2024-04-17T10:27:44.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f86a89d9000 10 client.0 _ll_register_callbacks cb 0x5616bc123180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.094+0000 7f869f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26b360 0x5616bc22bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26b360 0x5616bc22bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40480/0 (socket says 172.21.15.67:40480) 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 -- 172.21.15.67:0/3619229725 learned_addr learned my addr 172.21.15.67:0/3619229725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869ea00640 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc225d90 0x5616bc2721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869e000640 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26aa40 0x5616bc272700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 -- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc225d90 msgr2=0x5616bc2721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc225d90 0x5616bc2721c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 -- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26aa40 msgr2=0x5616bc272700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26aa40 0x5616bc272700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 -- 172.21.15.67:0/3619229725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616bbff71a0 con 0x5616bc26b360 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869f400640 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26b360 0x5616bc22bdc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f869805d880 tx=0x7f8698001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993713c6790cb395 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 1 -- 172.21.15.67:0/3619229725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8698004d70 con 0x5616bc26b360 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 1 -- 172.21.15.67:0/3619229725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f8698054ba0 con 0x5616bc26b360 2024-04-17T10:27:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8698051dc0 con 0x5616bc26b360 2024-04-17T10:27:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 1 -- 172.21.15.67:0/3619229725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8698073050 con 0x5616bc26b360 2024-04-17T10:27:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 10 client.5547 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5616bc0f1840 con 0x5616bc26b360 2024-04-17T10:27:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 1 -- 172.21.15.67:0/3619229725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8698051dc0 con 0x5616bc26b360 2024-04-17T10:27:44.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 1 -- 172.21.15.67:0/3619229725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f86980514a0 con 0x5616bc26b360 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 10 client.5547 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 10 client.5547 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 20 client.5547 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc22cb60 0x5616bc22cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 4 client.5547 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5616bc00abe0 con 0x5616bc22cb60 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869ea00640 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc22cb60 0x5616bc22cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f869ea00640 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc22cb60 0x5616bc22cf40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9408a7ae38512755 server_cookie=bb02b69df23e70b0 in_seq=0 out_seq=0 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 10 client.5547 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 1 -- 172.21.15.67:0/3619229725 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5616bc00abe0 con 0x5616bc22cb60 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 10 client.5547 handle_command_reply: tid=0 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 client.5547 shutdown 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26b360 msgr2=0x5616bc22bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26b360 0x5616bc22bdc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f869805d880 tx=0x7f8698001260 comp rx=0 tx=0).stop 2024-04-17T10:27:44.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 shutdown_connections 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc22cb60 0x5616bc22cf40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26b360 0x5616bc22bdc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc26aa40 0x5616bc272700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 --2- 172.21.15.67:0/3619229725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc225d90 0x5616bc2721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f867fe00640 0 client.5547 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 >> 172.21.15.67:0/3619229725 conn(0x5616bc221d40 msgr2=0x5616bc2249c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 shutdown_connections 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 1 -- 172.21.15.67:0/3619229725 wait complete. 2024-04-17T10:27:44.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.098+0000 7f86a89d9000 20 client.5547 trim_cache size 0 max 16384 2024-04-17T10:27:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 Processor -- start 2024-04-17T10:27:44.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 -- start start 2024-04-17T10:27:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0203330 con 0x5616bc26aa40 2024-04-17T10:27:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a02024d0 con 0x5616bc225d90 2024-04-17T10:27:44.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0141930 con 0x5616bc26b360 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40482/0 (socket says 172.21.15.67:40482) 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 -- 172.21.15.67:0/359122088 learned_addr learned my addr 172.21.15.67:0/359122088 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869ea00640 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869f400640 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 -- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 msgr2=0x5616bc26e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 -- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 msgr2=0x5616bc26d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 -- 172.21.15.67:0/359122088 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616bc0c8a10 con 0x5616bc26aa40 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869f400640 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869ea00640 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f869e000640 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26dc90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f868c001800 tx=0x7f868c007d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f07686bc2bb944 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f868aa00640 1 -- 172.21.15.67:0/359122088 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c0052e0 con 0x5616bc26aa40 2024-04-17T10:27:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f868aa00640 1 -- 172.21.15.67:0/359122088 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c01a030 con 0x5616bc26aa40 2024-04-17T10:27:44.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.114+0000 7f868aa00640 1 -- 172.21.15.67:0/359122088 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c015b10 con 0x5616bc26aa40 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- 172.21.15.67:0/359122088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 msgr2=0x5616bc26dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26dc90 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f868c001800 tx=0x7f868c007d40 comp rx=0 tx=0).stop 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- 172.21.15.67:0/359122088 shutdown_connections 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26dc90 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 --2- 172.21.15.67:0/359122088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- 172.21.15.67:0/359122088 >> 172.21.15.67:0/359122088 conn(0x5616bc221d40 msgr2=0x5616bc2788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- 172.21.15.67:0/359122088 shutdown_connections 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- 172.21.15.67:0/359122088 wait complete. 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- start start 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 Processor -- start 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616bc032720 con 0x5616bc26aa40 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a0141f30 con 0x5616bc225d90 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5616bc06af30 con 0x5616bc26b360 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 10 client.0 _ll_register_callbacks cb 0x5616bc031a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40494/0 (socket says 172.21.15.67:40494) 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 -- 172.21.15.67:0/2560927737 learned_addr learned my addr 172.21.15.67:0/2560927737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869ea00640 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869f400640 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 -- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 msgr2=0x5616bc26fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 -- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 msgr2=0x5616bc26f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 -- 172.21.15.67:0/2560927737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5616bc003b00 con 0x5616bc26aa40 2024-04-17T10:27:44.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f869e000640 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26f790 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f868c007da0 tx=0x7f868c000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1601f7d8de6607f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f867fe00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:27:44.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f867fe00640 1 -- 172.21.15.67:0/2560927737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c015b20 con 0x5616bc26aa40 2024-04-17T10:27:44.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f867fe00640 1 -- 172.21.15.67:0/2560927737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f868c015cc0 con 0x5616bc26aa40 2024-04-17T10:27:44.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f868c01a030 con 0x5616bc26aa40 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f867fe00640 1 -- 172.21.15.67:0/2560927737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=19) v2 ==== 24+0+0 (secure 0 0 0) 0x7f868c032050 con 0x5616bc26aa40 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 10 client.5553 fetch_fsmap learned FSMap version 19 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.118+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=19,monmap=2+}) v3 -- 0x5616bc015f80 con 0x5616bc26aa40 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f867fe00640 1 -- 172.21.15.67:0/2560927737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c01a030 con 0x5616bc26aa40 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f867fe00640 1 -- 172.21.15.67:0/2560927737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 19) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f868c035020 con 0x5616bc26aa40 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 10 client.5553 fetch_fsmap finished waiting for FSMap version 19 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 10 client.5553 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-17T10:27:44.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 20 client.5553 populate_metadata read hostname 'smithi067' 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc2709a0 0x5616bc2d2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 4 client.5553 mds_command: new command op to 5283 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 --> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x5616bc07c3d0 con 0x5616bc2709a0 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f869ea00640 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc2709a0 0x5616bc2d2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f867fe00640 10 client.5553 ms_handle_connect on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f869ea00640 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc2709a0 0x5616bc2d2ce0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e134584be253e176 server_cookie=731e0ab3262a6f11 in_seq=0 out_seq=0 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f867fe00640 1 -- 172.21.15.67:0/2560927737 <== mds.0 v2:172.21.15.67:6833/1409597842 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17930 (crc 0 0 0) 0x5616bc07c3d0 con 0x5616bc2709a0 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f867fe00640 10 client.5553 handle_command_reply: tid=0 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 client.5553 shutdown 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 msgr2=0x5616bc26f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26f790 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f868c007da0 tx=0x7f868c000ee0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 shutdown_connections 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:6833/1409597842,v1:172.21.15.67:6835/1409597842] conn(0x5616bc2709a0 0x5616bc2d2ce0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5616bc26b360 0x5616bc26fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5616bc26aa40 0x5616bc26f790 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 --2- 172.21.15.67:0/2560927737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5616bc225d90 0x5616bc26f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f867fe00640 0 client.5553 ms_handle_reset on v2:172.21.15.67:6833/1409597842 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 >> 172.21.15.67:0/2560927737 conn(0x5616bc221d40 msgr2=0x5616bc269120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 shutdown_connections 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 1 -- 172.21.15.67:0/2560927737 wait complete. 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a89d9000 20 client.5553 trim_cache size 0 max 16384 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-04-17T10:27:44.120 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 4226, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 4227, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 1061068, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 37675001, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 17, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 6, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.468886033, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.220245274, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.099810136, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.115584858, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4226, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.736036006, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000174168 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.121 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 10, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000001027, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000102 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 0 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 3055, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 3076, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 769730, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 19146797, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 6, 2024-04-17T10:27:44.122 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.321040181, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.147426654, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.067490750, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.081283408, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3075, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.511510670, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000166344 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.123 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 86352 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 30942, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 51780, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 11558346, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 56194892, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 7, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 4.790245771, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 2.834903761, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 2.835189825, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.070845140, 2024-04-17T10:27:44.124 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 51780, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.571630090, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000030352 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 71785, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.016066080, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000223 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 120752, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 26144 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 1, 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-04-17T10:27:44.125 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-MDSRank": { 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5744, 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 13.951631298, 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002428905 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-PurgeQueue": { 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3718, 2024-04-17T10:27:44.126 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.933719741, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000251134 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "request": 26268, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "reply": 26268, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "reply_latency": { 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 26268, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 238.888339963, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.009094272 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_reply": 0, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "forward": 0, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_complete": 0, 2024-04-17T10:27:44.127 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_keys": 0, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_commit": 12, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_split": 0, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_merge": 0, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes": 17831, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_top": 0, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_bottom": 17831, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pin_tail": 0, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pinned": 17831, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_expired": 0, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_with_caps": 2, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": 2, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "subtrees": 2, 2024-04-17T10:27:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse": 49498, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_hit": 48923, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_forward": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_discover": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_dir_fetch": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_remote_ino": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_lock": 162, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "load_cent": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "q": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "exported": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "exported_inodes": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "imported": 0, 2024-04-17T10:27:44.129 INFO:teuthology.orchestra.run.smithi067.stdout: "imported_inodes": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_dir_fetch": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_backtrace_fetch": 1853, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_peer_discover": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rfiles": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rbytes": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rsnaps": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_fetch": 1853, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_set_tag": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_repaired": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_inotable_repaired": 0, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_inodes": 13, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_base_inodes": 2, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dirfrag_rstats": 11, 2024-04-17T10:27:44.130 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_file_inodes": 1840, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_inode_file_caps": 0, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_revoke": 942, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_grant": 4023, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_trunc": 0, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps": 3981, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps_dirty": 2000, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_cap_release": 63, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "process_request_cap_release": 14099 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache": { 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update": 0, 2024-04-17T10:27:44.131 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update_receipt": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_try_discover": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_send_discover": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_handle_discover": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_delayed": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_enqueuing": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_created": 4138, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_enqueued": 3728, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_reintegrated": 2248, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_migrated": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_processing": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_enqueued": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_prioritized": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_started": 0, 2024-04-17T10:27:44.132 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_completed": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_enqueue_scrub": 1, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_exportdir": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_flush": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragmentdir": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragstats": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_inodestats": 0 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log": { 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "evlrg": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "evadd": 26160, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "evex": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "evtrm": 0, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "ev": 26160, 2024-04-17T10:27:44.133 INFO:teuthology.orchestra.run.smithi067.stdout: "evexg": 0, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "evexd": 0, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "segadd": 35, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "segex": 0, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "segtrm": 0, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "seg": 35, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "segmjr": 4, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "segexg": 0, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "segexd": 0, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "expos": 4194304, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "wrpos": 90085094, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "rdpos": 0, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "jlat": { 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4072, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 71.395864507, 2024-04-17T10:27:44.134 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.017533365 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "replayed": 0 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mem": { 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 14, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "ino+": 2016, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "ino-": 2002, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir": 13, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir+": 13, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir-": 0, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dn": 17831, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dn+": 19832, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dn-": 2001, 2024-04-17T10:27:44.135 INFO:teuthology.orchestra.run.smithi067.stdout: "cap": 2, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "cap+": 8390, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "cap-": 8388, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "rss": 101404, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "heap": 215340 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server": { 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_client_request": 30058, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_server_request": 0, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_request": 26268, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_session": 5, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_peer_request": 0, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "req_create_latency": { 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2000, 2024-04-17T10:27:44.136 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.590244077, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000795122 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getattr_latency": { 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 23, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.003463817, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000150600 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getfilelock_latency": { 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "req_link_latency": { 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6000, 2024-04-17T10:27:44.137 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 7.047901284, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001174650 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookup_latency": { 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7984, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 6.460720613, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000809208 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookuphash_latency": { 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupino_latency": { 2024-04-17T10:27:44.138 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupname_latency": { 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupparent_latency": { 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupsnap_latency": { 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:44.139 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000077286, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000077286 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lssnap_latency": { 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mkdir_latency": { 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.002982848, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002982848 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mknod_latency": { 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mksnap_latency": { 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.007953929, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.007953929 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "req_open_latency": { 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "req_readdir_latency": { 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8, 2024-04-17T10:27:44.141 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.093474816, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.011684352 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rename_latency": { 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2248, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 207.157699302, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.092152001 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "req_renamesnap_latency": { 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "req_snapdiff_latency": { 2024-04-17T10:27:44.142 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmdir_latency": { 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmsnap_latency": { 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.063101977, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.063101977 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmxattr_latency": { 2024-04-17T10:27:44.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setattr_latency": { 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000251475, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000251475 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setdirlayout_latency": { 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setfilelock_latency": { 2024-04-17T10:27:44.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setlayout_latency": { 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setxattr_latency": { 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "req_symlink_latency": { 2024-04-17T10:27:44.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "req_unlink_latency": { 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8000, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 16.460468539, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002057558 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_revoke_eviction": 0, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_acquisition_throttle": 0, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getvxattr_latency": { 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessions": { 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "session_count": 1, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "session_add": 1, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "session_remove": 0, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_open": 1, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_stale": 0, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "total_load": 18913, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "average_load": 18913, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "avg_session_uptime": 80, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "mdthresh_evicted": 0 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-04-17T10:27:44.147 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-04-17T10:27:44.148 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-04-17T10:27:44.149 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 250381, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 86, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-04-17T10:27:44.150 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 5624, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 80, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 13427860, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 660980, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-04-17T10:27:44.151 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter": { 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_active": 0, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_laggy": 0, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send": 9915, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send_bytes": 86516601, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_resend": 37, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_reply": 9915, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_latency": { 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 9915, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 42.291651034, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.004265421 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "op_inflight": 0, 2024-04-17T10:27:44.152 INFO:teuthology.orchestra.run.smithi067.stdout: "oplen_avg": { 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 9915, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 10010 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "op": 9915, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "op_r": 1914, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "op_w": 8001, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "op_rmw": 0, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "op_pg": 0, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_stat": 0, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_create": 11, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_read": 59, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_write": 4129, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writefull": 36, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writesame": 0, 2024-04-17T10:27:44.153 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_append": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_zero": 1728, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_truncate": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_delete": 2032, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_mapext": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_sparse_read": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_clonerange": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_getxattr": 1855, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_setxattr": 22, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_cmpxattr": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_rmxattr": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_resetxattrs": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_call": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_watch": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_notify": 0, 2024-04-17T10:27:44.154 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_src_cmpxattr": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls_filter": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_other": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_active": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_send": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_resend": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_ping": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_active": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_send": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_resend": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_active": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_send": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_resend": 0, 2024-04-17T10:27:44.155 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_active": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_send": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_resend": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "command_active": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "command_send": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "command_resend": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "map_epoch": 43, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "map_full": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "map_inc": 5, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_sessions": 8, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_open": 8, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_close": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_laggy": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_wr": 114, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_rd": 0, 2024-04-17T10:27:44.156 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_del": 24 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "oft": { 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_objs": 1, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_kv_pairs": 0, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_updates": 0, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_removes": 0 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "osd-slow-ops": { 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_ops_count": 0 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "purge_queue": { 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops": 0, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops_high_water": 32, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing": 0, 2024-04-17T10:27:44.157 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_high_water": 32, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed_ops": 3728, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed": 3728, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_item_in_journal": 0 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 38219, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 9943772, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 38219, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:27:44.158 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 38219, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 9943772, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_bytes": { 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:27:44.159 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 9915, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 86906953, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 9915, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 86906953, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_ops": { 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:44.160 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 1024, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 9915, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 9915, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 9915, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 9915, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.161 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle": { 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 3728, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 390352, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 3728, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 57, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 390352, 2024-04-17T10:27:44.162 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle-0x559782f64a00": { 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 26160, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 85890790, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 26160, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:27:44.163 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 4072, 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 85890790, 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:44.164 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:27:44.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8680066f60 msgr2=0x7f8680069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8680066f60 0x7f8680069420 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f86900ae000 tx=0x7f86900a7000 comp rx=0 tx=0).stop 2024-04-17T10:27:44.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86a01583f0 msgr2=0x7f86a013f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86a01583f0 0x7f86a013f910 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f869800cb40 tx=0x7f869800a060 comp rx=0 tx=0).stop 2024-04-17T10:27:44.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 shutdown_connections 2024-04-17T10:27:44.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8680066f60 0x7f8680069420 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f86a01583f0 0x7f86a013f910 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f86a00b5300 0x7f86a013f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 --2- 172.21.15.67:0/552287524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f86a00728c0 0x7f86a013eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 >> 172.21.15.67:0/552287524 conn(0x7f86a00b2570 msgr2=0x7f86a00b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 shutdown_connections 2024-04-17T10:27:44.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.122+0000 7f86a5600640 1 -- 172.21.15.67:0/552287524 wait complete. 2024-04-17T10:27:44.168 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:27:44.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:27:44.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:27:44.339 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:27:44.339 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:27:44.339 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:44.348 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:27:44.349 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:44.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:27:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc9c00640 1 Processor -- start 2024-04-17T10:27:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc9c00640 1 -- start start 2024-04-17T10:27:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc9c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1dc40b07b0 0x7f1dc4070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc9c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1dc4071300 0x7f1dc406c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc9c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc40b5980 con 0x7f1dc415dd10 2024-04-17T10:27:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc9c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc4071770 con 0x7f1dc4071300 2024-04-17T10:27:44.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc9c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc40b1280 con 0x7f1dc40b07b0 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1dc415dd10 0x7f1dc40b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43128/0 (socket says 172.21.15.67:43128) 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc2a00640 1 -- 172.21.15.67:0/3366883371 learned_addr learned my addr 172.21.15.67:0/3366883371 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc3400640 1 --2- 172.21.15.67:0/3366883371 >> v2:172.21.15.67:3300/0 conn(0x7f1dc4071300 0x7f1dc406c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dba000640 1 --2- 172.21.15.67:0/3366883371 >> v2:172.21.15.86:3300/0 conn(0x7f1dc40b07b0 0x7f1dc4070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc3400640 1 -- 172.21.15.67:0/3366883371 >> v2:172.21.15.86:3300/0 conn(0x7f1dc40b07b0 msgr2=0x7f1dc4070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc3400640 1 --2- 172.21.15.67:0/3366883371 >> v2:172.21.15.86:3300/0 conn(0x7f1dc40b07b0 0x7f1dc4070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc3400640 1 -- 172.21.15.67:0/3366883371 >> v1:172.21.15.67:6789/0 conn(0x7f1dc415dd10 legacy=0x7f1dc40b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.590+0000 7f1dc3400640 1 -- 172.21.15.67:0/3366883371 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc40b0cc0 con 0x7f1dc4071300 2024-04-17T10:27:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc3400640 1 --2- 172.21.15.67:0/3366883371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc4071300 0x7f1dc406c6d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4068850 tx=0x7f1dbc01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a467d22caa2702 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc2000640 1 -- 172.21.15.67:0/3366883371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc0b2070 con 0x7f1dc4071300 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc2000640 1 -- 172.21.15.67:0/3366883371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dbc0a5930 con 0x7f1dc4071300 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc2000640 1 -- 172.21.15.67:0/3366883371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc0a5d00 con 0x7f1dc4071300 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 -- 172.21.15.67:0/3366883371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc4071300 msgr2=0x7f1dc406c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/3366883371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc4071300 0x7f1dc406c6d0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4068850 tx=0x7f1dbc01c870 comp rx=0 tx=0).stop 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 -- 172.21.15.67:0/3366883371 shutdown_connections 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/3366883371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc4071300 0x7f1dc406c6d0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/3366883371 >> v2:172.21.15.86:3300/0 conn(0x7f1dc40b07b0 0x7f1dc4070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 -- 172.21.15.67:0/3366883371 >> 172.21.15.67:0/3366883371 conn(0x7f1dc40b34d0 msgr2=0x7f1dc40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 -- 172.21.15.67:0/3366883371 shutdown_connections 2024-04-17T10:27:44.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.594+0000 7f1dc9c00640 1 -- 172.21.15.67:0/3366883371 wait complete. 2024-04-17T10:27:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 Processor -- start 2024-04-17T10:27:44.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 -- start start 2024-04-17T10:27:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dc4071300 0x7f1dc40cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dc40b07b0 0x7f1dc4158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc415dd10 0x7f1dc4158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc406d420 con 0x7f1dc415dd10 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc415e0f0 con 0x7f1dc4071300 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc406d5a0 con 0x7f1dc40b07b0 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc415dd10 0x7f1dc4158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc415dd10 0x7f1dc4158870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40512/0 (socket says 172.21.15.67:40512) 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 -- 172.21.15.67:0/4283164132 learned_addr learned my addr 172.21.15.67:0/4283164132 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc2a00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dc4071300 0x7f1dc40cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dba000640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dc40b07b0 0x7f1dc4158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 -- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dc40b07b0 msgr2=0x7f1dc4158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dc40b07b0 0x7f1dc4158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 -- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dc4071300 msgr2=0x7f1dc40cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dc4071300 0x7f1dc40cf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 -- 172.21.15.67:0/4283164132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc40cf690 con 0x7f1dc415dd10 2024-04-17T10:27:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dba000640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dc40b07b0 0x7f1dc4158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc2a00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dc4071300 0x7f1dc40cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc3400640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc415dd10 0x7f1dc4158870 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc0b0040 tx=0x7f1dbc019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dbb400640 1 -- 172.21.15.67:0/4283164132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc0b2050 con 0x7f1dc415dd10 2024-04-17T10:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dbb400640 1 -- 172.21.15.67:0/4283164132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dbc0be070 con 0x7f1dc415dd10 2024-04-17T10:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dc40ce860 con 0x7f1dc415dd10 2024-04-17T10:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dbb400640 1 -- 172.21.15.67:0/4283164132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dbc0adb10 con 0x7f1dc415dd10 2024-04-17T10:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.598+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc4155dd0 con 0x7f1dc415dd10 2024-04-17T10:27:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.602+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc4070bb0 con 0x7f1dc415dd10 2024-04-17T10:27:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.602+0000 7f1dbb400640 1 -- 172.21.15.67:0/4283164132 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1dbc0adcb0 con 0x7f1dc415dd10 2024-04-17T10:27:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.602+0000 7f1dbb400640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1da0066f60 0x7f1da0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.602+0000 7f1dbb400640 1 -- 172.21.15.67:0/4283164132 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f1dbc120540 con 0x7f1dc415dd10 2024-04-17T10:27:44.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.602+0000 7f1dc2a00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1da0066f60 0x7f1da0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.602+0000 7f1dc2a00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1da0066f60 0x7f1da0069420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4153cb0 tx=0x7f1db4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.606+0000 7f1dbb400640 1 -- 172.21.15.67:0/4283164132 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1dbc016520 con 0x7f1dc415dd10 2024-04-17T10:27:44.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.770+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1dc4155fa0 con 0x7f1dc415dd10 2024-04-17T10:27:44.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.770+0000 7f1dbb400640 1 -- 172.21.15.67:0/4283164132 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17008 (secure 0 0 0) 0x7f1dbc0ab340 con 0x7f1dc415dd10 2024-04-17T10:27:44.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:44.765 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:27:28.880541+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:26:13.215866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-17T10:26:14.398708+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":42,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:27:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1da0066f60 msgr2=0x7f1da0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1da0066f60 0x7f1da0069420 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1dc4153cb0 tx=0x7f1db4009040 comp rx=0 tx=0).stop 2024-04-17T10:27:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc415dd10 msgr2=0x7f1dc4158870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc415dd10 0x7f1dc4158870 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1dbc0b0040 tx=0x7f1dbc019610 comp rx=0 tx=0).stop 2024-04-17T10:27:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 shutdown_connections 2024-04-17T10:27:44.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1da0066f60 0x7f1da0069420 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dc415dd10 0x7f1dc4158870 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dc40b07b0 0x7f1dc4158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 --2- 172.21.15.67:0/4283164132 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dc4071300 0x7f1dc40cf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 >> 172.21.15.67:0/4283164132 conn(0x7f1dc40b34d0 msgr2=0x7f1dc4150b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 shutdown_connections 2024-04-17T10:27:44.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.774+0000 7f1dc9c00640 1 -- 172.21.15.67:0/4283164132 wait complete. 2024-04-17T10:27:44.787 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:26:13.215866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:26:14.398708+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 42, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:27:44.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.934+0000 7f6f01a00640 1 Processor -- start 2024-04-17T10:27:44.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.934+0000 7f6f01a00640 1 -- start start 2024-04-17T10:27:44.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc06bf30 0x7f6efc15c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6f01a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0b36c0 con 0x7f6efc0ad080 2024-04-17T10:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6f01a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc15ce20 con 0x7f6efc06c310 2024-04-17T10:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6f01a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc0b4d80 con 0x7f6efc06bf30 2024-04-17T10:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6efc0ad080 0x7f6efc06ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43140/0 (socket says 172.21.15.67:43140) 2024-04-17T10:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efb400640 1 -- 172.21.15.67:0/2828632177 learned_addr learned my addr 172.21.15.67:0/2828632177 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:44.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efaa00640 1 --2- 172.21.15.67:0/2828632177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc06bf30 0x7f6efc15c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efa000640 1 -- 172.21.15.67:0/2828632177 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089387759 0 0) 0x7f6efc0b36c0 con 0x7f6efc0ad080 2024-04-17T10:27:44.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efa000640 1 -- 172.21.15.67:0/2828632177 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ee0003530 con 0x7f6efc0ad080 2024-04-17T10:27:44.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efa000640 1 -- 172.21.15.67:0/2828632177 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075325960 0 0) 0x7f6ee4001120 con 0x7f6efc06c310 2024-04-17T10:27:44.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efa000640 1 -- 172.21.15.67:0/2828632177 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6efc0b36c0 con 0x7f6efc06c310 2024-04-17T10:27:44.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efaa00640 1 -- 172.21.15.67:0/2828632177 >> v1:172.21.15.86:6789/0 conn(0x7f6efc06c310 legacy=0x7f6efc0b4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efaa00640 1 -- 172.21.15.67:0/2828632177 >> v1:172.21.15.67:6789/0 conn(0x7f6efc0ad080 legacy=0x7f6efc06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.938+0000 7f6efaa00640 1 -- 172.21.15.67:0/2828632177 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc0b5020 con 0x7f6efc06bf30 2024-04-17T10:27:44.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6efaa00640 1 --2- 172.21.15.67:0/2828632177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc06bf30 0x7f6efc15c8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0097f0 tx=0x7f6eec01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=102e8c5fff4a0600 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6efa000640 1 -- 172.21.15.67:0/2828632177 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0aa610 con 0x7f6efc06bf30 2024-04-17T10:27:44.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6efa000640 1 -- 172.21.15.67:0/2828632177 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec0af070 con 0x7f6efc06bf30 2024-04-17T10:27:44.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6efa000640 1 -- 172.21.15.67:0/2828632177 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0aa9d0 con 0x7f6efc06bf30 2024-04-17T10:27:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 -- 172.21.15.67:0/2828632177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc06bf30 msgr2=0x7f6efc15c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 --2- 172.21.15.67:0/2828632177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc06bf30 0x7f6efc15c8e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0097f0 tx=0x7f6eec01c4b0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 -- 172.21.15.67:0/2828632177 shutdown_connections 2024-04-17T10:27:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 --2- 172.21.15.67:0/2828632177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc06bf30 0x7f6efc15c8e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 -- 172.21.15.67:0/2828632177 >> 172.21.15.67:0/2828632177 conn(0x7f6efc0b2710 msgr2=0x7f6efc0cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 -- 172.21.15.67:0/2828632177 shutdown_connections 2024-04-17T10:27:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 -- 172.21.15.67:0/2828632177 wait complete. 2024-04-17T10:27:44.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 Processor -- start 2024-04-17T10:27:44.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.942+0000 7f6f01a00640 1 -- start start 2024-04-17T10:27:44.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc06bf30 0x7f6efc151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06c310 0x7f6efc14f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc0ad080 0x7f6efc154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc0ad8a0 con 0x7f6efc06c310 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06b7b0 con 0x7f6efc06bf30 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc152960 con 0x7f6efc0ad080 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06c310 0x7f6efc14f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06c310 0x7f6efc14f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40520/0 (socket says 172.21.15.67:40520) 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 -- 172.21.15.67:0/968304606 learned_addr learned my addr 172.21.15.67:0/968304606 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6efb400640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc0ad080 0x7f6efc154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6efaa00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc06bf30 0x7f6efc151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 -- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc0ad080 msgr2=0x7f6efc154df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc0ad080 0x7f6efc154df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 -- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc06bf30 msgr2=0x7f6efc151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc06bf30 0x7f6efc151ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 -- 172.21.15.67:0/968304606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc152bb0 con 0x7f6efc06c310 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6efb400640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc0ad080 0x7f6efc154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef3e00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06c310 0x7f6efc14f770 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400c700 tx=0x7f6ee400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef2a00640 1 -- 172.21.15.67:0/968304606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee40052d0 con 0x7f6efc06c310 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef2a00640 1 -- 172.21.15.67:0/968304606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee4005470 con 0x7f6efc06c310 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc1545f0 con 0x7f6efc06c310 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6ef2a00640 1 -- 172.21.15.67:0/968304606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee400a860 con 0x7f6efc06c310 2024-04-17T10:27:44.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc152e90 con 0x7f6efc06c310 2024-04-17T10:27:44.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.946+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6efc067f10 con 0x7f6efc06c310 2024-04-17T10:27:44.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.950+0000 7f6ef2a00640 1 -- 172.21.15.67:0/968304606 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6ee400aa00 con 0x7f6efc06c310 2024-04-17T10:27:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.950+0000 7f6ef2a00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ed8066f60 0x7f6ed8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.950+0000 7f6ef2a00640 1 -- 172.21.15.67:0/968304606 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f6ee4084e30 con 0x7f6efc06c310 2024-04-17T10:27:44.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.950+0000 7f6efaa00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ed8066f60 0x7f6ed8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:44.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.950+0000 7f6efaa00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ed8066f60 0x7f6ed8069420 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0ae000 tx=0x7f6eec0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:44.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:44.954+0000 7f6ef2a00640 1 -- 172.21.15.67:0/968304606 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6efc15ce20 con 0x7f6efc06c310 2024-04-17T10:27:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6efc04ef30 con 0x7f6efc06c310 2024-04-17T10:27:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6ef2a00640 1 -- 172.21.15.67:0/968304606 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6ee4055c70 con 0x7f6efc06c310 2024-04-17T10:27:45.112 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ed8066f60 msgr2=0x7f6ed8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ed8066f60 0x7f6ed8069420 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0ae000 tx=0x7f6eec0a7000 comp rx=0 tx=0).stop 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06c310 msgr2=0x7f6efc14f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06c310 0x7f6efc14f770 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6ee400c700 tx=0x7f6ee400cbc0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 shutdown_connections 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ed8066f60 0x7f6ed8069420 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc0ad080 0x7f6efc154df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06c310 0x7f6efc14f770 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 --2- 172.21.15.67:0/968304606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc06bf30 0x7f6efc151ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 >> 172.21.15.67:0/968304606 conn(0x7f6efc0b2710 msgr2=0x7f6efc15aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 shutdown_connections 2024-04-17T10:27:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.118+0000 7f6f01a00640 1 -- 172.21.15.67:0/968304606 wait complete. 2024-04-17T10:27:45.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:27:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.282+0000 7f5ed2000640 1 Processor -- start 2024-04-17T10:27:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.282+0000 7f5ed2000640 1 -- start start 2024-04-17T10:27:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.282+0000 7f5ed2000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc071af0 0x7f5ecc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.282+0000 7f5ed2000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ecc15be70 0x7f5ecc15c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.282+0000 7f5ed2000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc15cb30 con 0x7f5ecc159440 2024-04-17T10:27:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.282+0000 7f5ed2000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ecc0a8650 con 0x7f5ecc15be70 2024-04-17T10:27:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.282+0000 7f5ed2000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc15ce10 con 0x7f5ecc071af0 2024-04-17T10:27:45.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ecc15be70 0x7f5ecc15c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc071af0 0x7f5ecc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ecc15be70 0x7f5ecc15c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:58524/0 (socket says 172.21.15.67:58524) 2024-04-17T10:27:45.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 -- 172.21.15.67:0/3098353140 learned_addr learned my addr 172.21.15.67:0/3098353140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:45.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecb400640 1 --1- 172.21.15.67:0/3098353140 >> v1:172.21.15.86:6789/0 conn(0x7f5ecc159440 0x7f5ecc0713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54792/0 (socket says 172.21.15.67:54792) 2024-04-17T10:27:45.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 -- 172.21.15.67:0/3098353140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc071af0 msgr2=0x7f5ecc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:45.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 --2- 172.21.15.67:0/3098353140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc071af0 0x7f5ecc0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 -- 172.21.15.67:0/3098353140 >> v1:172.21.15.86:6789/0 conn(0x7f5ecc159440 legacy=0x7f5ecc0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 -- 172.21.15.67:0/3098353140 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0cd610 con 0x7f5ecc15be70 2024-04-17T10:27:45.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.286+0000 7f5ecbe00640 1 --2- 172.21.15.67:0/3098353140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 0x7f5ecc15c280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009e50 tx=0x7f5ec401ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a283d9f1f2a64832 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5eca000640 1 -- 172.21.15.67:0/3098353140 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40a5900 con 0x7f5ecc15be70 2024-04-17T10:27:45.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5eca000640 1 -- 172.21.15.67:0/3098353140 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec40a5aa0 con 0x7f5ecc15be70 2024-04-17T10:27:45.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5eca000640 1 -- 172.21.15.67:0/3098353140 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec40ad3f0 con 0x7f5ecc15be70 2024-04-17T10:27:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- 172.21.15.67:0/3098353140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 msgr2=0x7f5ecc15c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3098353140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 0x7f5ecc15c280 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009e50 tx=0x7f5ec401ca30 comp rx=0 tx=0).stop 2024-04-17T10:27:45.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- 172.21.15.67:0/3098353140 shutdown_connections 2024-04-17T10:27:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3098353140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 0x7f5ecc15c280 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3098353140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc071af0 0x7f5ecc0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- 172.21.15.67:0/3098353140 >> 172.21.15.67:0/3098353140 conn(0x7f5ecc0b1740 msgr2=0x7f5ecc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- 172.21.15.67:0/3098353140 shutdown_connections 2024-04-17T10:27:45.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- 172.21.15.67:0/3098353140 wait complete. 2024-04-17T10:27:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 Processor -- start 2024-04-17T10:27:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- start start 2024-04-17T10:27:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ecc071af0 0x7f5ecc1571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc159440 0x7f5ecc0ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 0x7f5ecc15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc15d840 con 0x7f5ecc071af0 2024-04-17T10:27:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc15c7c0 con 0x7f5ecc15be70 2024-04-17T10:27:45.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ed2000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc15d9c0 con 0x7f5ecc159440 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ecb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ecc071af0 0x7f5ecc1571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ecb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ecc071af0 0x7f5ecc1571b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40526/0 (socket says 172.21.15.67:40526) 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ecb400640 1 -- 172.21.15.67:0/3993429476 learned_addr learned my addr 172.21.15.67:0/3993429476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ecaa00640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc159440 0x7f5ecc0ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.290+0000 7f5ecbe00640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 0x7f5ecc15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ecb400640 1 -- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc159440 msgr2=0x7f5ecc0ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ecb400640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc159440 0x7f5ecc0ae270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ecb400640 1 -- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 msgr2=0x7f5ecc15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ecb400640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 0x7f5ecc15a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ecb400640 1 -- 172.21.15.67:0/3993429476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc1533c0 con 0x7f5ecc071af0 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ecb400640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ecc071af0 0x7f5ecc1571b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc00c420 tx=0x7f5ebc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ec3400640 1 -- 172.21.15.67:0/3993429476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc0074e0 con 0x7f5ecc071af0 2024-04-17T10:27:45.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ec3400640 1 -- 172.21.15.67:0/3993429476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ebc007680 con 0x7f5ecc071af0 2024-04-17T10:27:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ec3400640 1 -- 172.21.15.67:0/3993429476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ebc004a30 con 0x7f5ecc071af0 2024-04-17T10:27:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc1536a0 con 0x7f5ecc071af0 2024-04-17T10:27:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc153c60 con 0x7f5ecc071af0 2024-04-17T10:27:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc04ef30 con 0x7f5ecc071af0 2024-04-17T10:27:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ec3400640 1 -- 172.21.15.67:0/3993429476 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5ebc01a020 con 0x7f5ecc071af0 2024-04-17T10:27:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ec3400640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5eac066fe0 0x7f5eac0694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.294+0000 7f5ec3400640 1 -- 172.21.15.67:0/3993429476 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f5ebc084820 con 0x7f5ecc071af0 2024-04-17T10:27:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.298+0000 7f5ecaa00640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5eac066fe0 0x7f5eac0694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.298+0000 7f5ecaa00640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5eac066fe0 0x7f5eac0694a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc158640 tx=0x7f5eb8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.298+0000 7f5ec3400640 1 -- 172.21.15.67:0/3993429476 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5ebc055090 con 0x7f5ecc071af0 2024-04-17T10:27:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ecc0b1130 con 0x7f5ecc071af0 2024-04-17T10:27:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ec3400640 1 -- 172.21.15.67:0/3993429476 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ebc059130 con 0x7f5ecc071af0 2024-04-17T10:27:45.456 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:27:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5eac066fe0 msgr2=0x7f5eac0694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5eac066fe0 0x7f5eac0694a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc158640 tx=0x7f5eb8009190 comp rx=0 tx=0).stop 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ecc071af0 msgr2=0x7f5ecc1571b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ecc071af0 0x7f5ecc1571b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc00c420 tx=0x7f5ebc00c8e0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 shutdown_connections 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5eac066fe0 0x7f5eac0694a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ecc15be70 0x7f5ecc15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ecc159440 0x7f5ecc0ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 --2- 172.21.15.67:0/3993429476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ecc071af0 0x7f5ecc1571b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 >> 172.21.15.67:0/3993429476 conn(0x7f5ecc0b1740 msgr2=0x7f5ecc0713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 shutdown_connections 2024-04-17T10:27:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.462+0000 7f5ed2000640 1 -- 172.21.15.67:0/3993429476 wait complete. 2024-04-17T10:27:45.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:27:45.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.626+0000 7f7b03e00640 1 Processor -- start 2024-04-17T10:27:45.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.626+0000 7f7b03e00640 1 -- start start 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.626+0000 7f7b03e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7afc0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.626+0000 7f7b03e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7afc0b3920 0x7f7afc0d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.626+0000 7f7b03e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc1616a0 con 0x7f7afc1474c0 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.626+0000 7f7b03e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc161940 con 0x7f7afc0b3920 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.626+0000 7f7b03e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc161c20 con 0x7f7afc06b3a0 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7afc1474c0 0x7f7afc06ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43152/0 (socket says 172.21.15.67:43152) 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 -- 172.21.15.67:0/2282153401 learned_addr learned my addr 172.21.15.67:0/2282153401 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01e00640 1 --2- 172.21.15.67:0/2282153401 >> v2:172.21.15.86:3300/0 conn(0x7f7afc0b3920 0x7f7afc0d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b00a00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7afc0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543045547 0 0) 0x7f7afc1616a0 con 0x7f7afc1474c0 2024-04-17T10:27:45.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae4003530 con 0x7f7afc1474c0 2024-04-17T10:27:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 594450103 0 0) 0x7f7ae4003530 con 0x7f7afc1474c0 2024-04-17T10:27:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 msgr2=0x7f7afc0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7afc0b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 >> v2:172.21.15.86:3300/0 conn(0x7f7afc0b3920 msgr2=0x7f7afc0d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 --2- 172.21.15.67:0/2282153401 >> v2:172.21.15.86:3300/0 conn(0x7f7afc0b3920 0x7f7afc0d0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc161f00 con 0x7f7afc1474c0 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f7af00016a0 con 0x7f7afc1474c0 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b00a00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7afc0b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 >> v1:172.21.15.67:6789/0 conn(0x7f7afc1474c0 legacy=0x7f7afc06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ae40039c0 0x7f7ae4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae4004810 0x7f7ae4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7ae40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc161940 con 0x7f7ae40039c0 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc161c20 con 0x7f7ae4004810 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae400b9d0 con 0x7f7afc06b3a0 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01e00640 1 --2- 172.21.15.67:0/2282153401 >> v2:172.21.15.86:3300/0 conn(0x7f7afc0b3920 0x7f7afc0d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ae40039c0 0x7f7ae4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b00a00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae4004810 0x7f7ae4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01e00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7ae40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 -- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 msgr2=0x7f7ae40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7ae40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 -- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae4004810 msgr2=0x7f7ae4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae4004810 0x7f7ae4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01400640 1 -- 172.21.15.67:0/2282153401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae4003530 con 0x7f7ae40039c0 2024-04-17T10:27:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b00a00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ae4004810 0x7f7ae4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:45.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.630+0000 7f7b01e00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc06b3a0 0x7f7ae40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b01400640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ae40039c0 0x7f7ae4003de0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c210 tx=0x7f7af001f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04f83a3875c6ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007d70 con 0x7f7ae40039c0 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0005ae0 con 0x7f7ae40039c0 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7afbe00640 1 -- 172.21.15.67:0/2282153401 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007070 con 0x7f7ae40039c0 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 -- 172.21.15.67:0/2282153401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ae40039c0 msgr2=0x7f7ae4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 --2- 172.21.15.67:0/2282153401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ae40039c0 0x7f7ae4003de0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c210 tx=0x7f7af001f390 comp rx=0 tx=0).stop 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b01e00640 1 -- 172.21.15.67:0/2282153401 reap_dead start 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 -- 172.21.15.67:0/2282153401 shutdown_connections 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 -- 172.21.15.67:0/2282153401 >> 172.21.15.67:0/2282153401 conn(0x7f7afc0b5ef0 msgr2=0x7f7afc099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 -- 172.21.15.67:0/2282153401 shutdown_connections 2024-04-17T10:27:45.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 -- 172.21.15.67:0/2282153401 wait complete. 2024-04-17T10:27:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 Processor -- start 2024-04-17T10:27:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.634+0000 7f7b03e00640 1 -- start start 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc15d1d0 0x7f7afc1443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7afc15d5b0 0x7f7afc15e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7afc15f570 0x7f7afc15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc162420 con 0x7f7afc15f570 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b2250 con 0x7f7afc15d5b0 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc0b1b90 con 0x7f7afc15d1d0 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7afc15f570 0x7f7afc15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7afc15f570 0x7f7afc15f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40544/0 (socket says 172.21.15.67:40544) 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc15d1d0 0x7f7afc1443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 -- 172.21.15.67:0/1704741237 learned_addr learned my addr 172.21.15.67:0/1704741237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b00a00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7afc15d5b0 0x7f7afc15e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 -- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc15d1d0 msgr2=0x7f7afc1443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc15d1d0 0x7f7afc1443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 -- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7afc15d5b0 msgr2=0x7f7afc15e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7afc15d5b0 0x7f7afc15e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 -- 172.21.15.67:0/1704741237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc145bb0 con 0x7f7afc15f570 2024-04-17T10:27:45.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b01e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7afc15f570 0x7f7afc15f9f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7af400f380 tx=0x7f7af400f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7af9600640 1 -- 172.21.15.67:0/1704741237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af4016620 con 0x7f7afc15f570 2024-04-17T10:27:45.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7af9600640 1 -- 172.21.15.67:0/1704741237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af401b070 con 0x7f7afc15f570 2024-04-17T10:27:45.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7af9600640 1 -- 172.21.15.67:0/1704741237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af40169e0 con 0x7f7afc15f570 2024-04-17T10:27:45.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc145e90 con 0x7f7afc15f570 2024-04-17T10:27:45.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc15ef80 con 0x7f7afc15f570 2024-04-17T10:27:45.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc04ef30 con 0x7f7afc15f570 2024-04-17T10:27:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.638+0000 7f7af9600640 1 -- 172.21.15.67:0/1704741237 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7af4016b80 con 0x7f7afc15f570 2024-04-17T10:27:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.642+0000 7f7af9600640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7adc066f60 0x7f7adc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.642+0000 7f7af9600640 1 -- 172.21.15.67:0/1704741237 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f7af4085e70 con 0x7f7afc15f570 2024-04-17T10:27:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.642+0000 7f7b01400640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7adc066f60 0x7f7adc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.642+0000 7f7b01400640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7adc066f60 0x7f7adc069420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ae000 tx=0x7f7af00b3040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.646+0000 7f7af9600640 1 -- 172.21.15.67:0/1704741237 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7af4025020 con 0x7f7afc15f570 2024-04-17T10:27:45.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7afc0d0050 con 0x7f7afc15f570 2024-04-17T10:27:45.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7af9600640 1 -- 172.21.15.67:0/1704741237 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7af4056700 con 0x7f7afc15f570 2024-04-17T10:27:45.800 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:27:45.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7adc066f60 msgr2=0x7f7adc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7adc066f60 0x7f7adc069420 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7af00ae000 tx=0x7f7af00b3040 comp rx=0 tx=0).stop 2024-04-17T10:27:45.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7afc15f570 msgr2=0x7f7afc15f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7afc15f570 0x7f7afc15f9f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7af400f380 tx=0x7f7af400f840 comp rx=0 tx=0).stop 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 shutdown_connections 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7adc066f60 0x7f7adc069420 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7afc15f570 0x7f7afc15f9f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7afc15d5b0 0x7f7afc15e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 --2- 172.21.15.67:0/1704741237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7afc15d1d0 0x7f7afc1443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 >> 172.21.15.67:0/1704741237 conn(0x7f7afc0b5ef0 msgr2=0x7f7afc0d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 shutdown_connections 2024-04-17T10:27:45.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.806+0000 7f7b03e00640 1 -- 172.21.15.67:0/1704741237 wait complete. 2024-04-17T10:27:45.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:27:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c24000640 1 Processor -- start 2024-04-17T10:27:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c24000640 1 -- start start 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c24000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c1c06e6e0 0x7f2c1c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c24000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2c1c1441f0 0x7f2c1c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c24000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c15ae40 con 0x7f2c1c06eac0 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c24000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c0cf430 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c24000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c1c071440 con 0x7f2c1c1441f0 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c21600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c1c06e6e0 0x7f2c1c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c21600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c1c06e6e0 0x7f2c1c0ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40558/0 (socket says 172.21.15.67:40558) 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c21600640 1 -- 172.21.15.67:0/4217845681 learned_addr learned my addr 172.21.15.67:0/4217845681 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.970+0000 7f2c20c00640 1 --1- 172.21.15.67:0/4217845681 >> v1:172.21.15.67:6789/0 conn(0x7f2c1c06eac0 0x7f2c1c143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43168/0 (socket says 172.21.15.67:43168) 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c22000640 1 --2- 172.21.15.67:0/4217845681 >> v2:172.21.15.86:3300/0 conn(0x7f2c1c1441f0 0x7f2c1c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c21600640 1 -- 172.21.15.67:0/4217845681 >> v2:172.21.15.86:3300/0 conn(0x7f2c1c1441f0 msgr2=0x7f2c1c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c21600640 1 --2- 172.21.15.67:0/4217845681 >> v2:172.21.15.86:3300/0 conn(0x7f2c1c1441f0 0x7f2c1c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c21600640 1 -- 172.21.15.67:0/4217845681 >> v1:172.21.15.67:6789/0 conn(0x7f2c1c06eac0 legacy=0x7f2c1c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c21600640 1 -- 172.21.15.67:0/4217845681 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c071720 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c22000640 1 --2- 172.21.15.67:0/4217845681 >> v2:172.21.15.86:3300/0 conn(0x7f2c1c1441f0 0x7f2c1c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:45.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c21600640 1 --2- 172.21.15.67:0/4217845681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c0ce9b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0ce3b0 tx=0x7f2c1001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c36b6832744442 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c1be00640 1 -- 172.21.15.67:0/4217845681 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100b0070 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c1be00640 1 -- 172.21.15.67:0/4217845681 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100ab600 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c1be00640 1 -- 172.21.15.67:0/4217845681 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100ab9d0 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c24000640 1 -- 172.21.15.67:0/4217845681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 msgr2=0x7f2c1c0ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.974+0000 7f2c24000640 1 --2- 172.21.15.67:0/4217845681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c0ce9b0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0ce3b0 tx=0x7f2c1001c460 comp rx=0 tx=0).stop 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- 172.21.15.67:0/4217845681 shutdown_connections 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 --2- 172.21.15.67:0/4217845681 >> v2:172.21.15.86:3300/0 conn(0x7f2c1c1441f0 0x7f2c1c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 --2- 172.21.15.67:0/4217845681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c0ce9b0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- 172.21.15.67:0/4217845681 >> 172.21.15.67:0/4217845681 conn(0x7f2c1c06ac90 msgr2=0x7f2c1c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- 172.21.15.67:0/4217845681 shutdown_connections 2024-04-17T10:27:45.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- 172.21.15.67:0/4217845681 wait complete. 2024-04-17T10:27:45.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 Processor -- start 2024-04-17T10:27:45.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- start start 2024-04-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c1c06eac0 0x7f2c1c142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c1c1441f0 0x7f2c1c0c97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0b3a40 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0b38c0 con 0x7f2c1c1441f0 2024-04-17T10:27:45.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0b3bc0 con 0x7f2c1c06eac0 2024-04-17T10:27:45.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c06dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40574/0 (socket says 172.21.15.67:40574) 2024-04-17T10:27:45.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 -- 172.21.15.67:0/1489646006 learned_addr learned my addr 172.21.15.67:0/1489646006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 -- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c1c06eac0 msgr2=0x7f2c1c142350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c1c06eac0 0x7f2c1c142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 -- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c1c1441f0 msgr2=0x7f2c1c0c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c22000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c1c1441f0 0x7f2c1c0c97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c1c1441f0 0x7f2c1c0c97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 -- 172.21.15.67:0/1489646006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0aa9a0 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c22000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c1c1441f0 0x7f2c1c0c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c21600640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c06dc50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2c100ae040 tx=0x7f2c1001aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c19600640 1 -- 172.21.15.67:0/1489646006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100ba020 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c19600640 1 -- 172.21.15.67:0/1489646006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c100b6640 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c19600640 1 -- 172.21.15.67:0/1489646006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c100b0040 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.978+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c0aac80 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.982+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c0ab240 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.982+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c072d50 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.982+0000 7f2c19600640 1 -- 172.21.15.67:0/1489646006 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2c100bd020 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.982+0000 7f2c19600640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2bfc066f90 0x7f2bfc069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.982+0000 7f2c19600640 1 -- 172.21.15.67:0/1489646006 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f2c1011f600 con 0x7f2c1c06e6e0 2024-04-17T10:27:45.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.982+0000 7f2c20c00640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2bfc066f90 0x7f2bfc069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:45.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.982+0000 7f2c20c00640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2bfc066f90 0x7f2bfc069450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2c08000f70 tx=0x7f2c08007a10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:45.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:45.986+0000 7f2c19600640 1 -- 172.21.15.67:0/1489646006 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2c100c4030 con 0x7f2c1c06e6e0 2024-04-17T10:27:46.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2c1c143f60 con 0x7f2c1c06e6e0 2024-04-17T10:27:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c19600640 1 -- 172.21.15.67:0/1489646006 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v43) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2c10016110 con 0x7f2c1c06e6e0 2024-04-17T10:27:46.156 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2bfc066f90 msgr2=0x7f2bfc069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2bfc066f90 0x7f2bfc069450 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2c08000f70 tx=0x7f2c08007a10 comp rx=0 tx=0).stop 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 msgr2=0x7f2c1c06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c06dc50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2c100ae040 tx=0x7f2c1001aca0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 shutdown_connections 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2bfc066f90 0x7f2bfc069450 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c1c1441f0 0x7f2c1c0c97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c1c06eac0 0x7f2c1c142350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 --2- 172.21.15.67:0/1489646006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c1c06e6e0 0x7f2c1c06dc50 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 >> 172.21.15.67:0/1489646006 conn(0x7f2c1c06ac90 msgr2=0x7f2c1c13ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:46.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 shutdown_connections 2024-04-17T10:27:46.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.162+0000 7f2c24000640 1 -- 172.21.15.67:0/1489646006 wait complete. 2024-04-17T10:27:46.176 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:27:46.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:27:46.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:27:46.219 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:27:46.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:27:46.219 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:27:46.275 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:27:46.275 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:46.284 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:27:46.284 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:27:46.284 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:46.326 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:27:46.327 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:27:46.327 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:46.335 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:27:46.336 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:27:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6dd400640 1 Processor -- start 2024-04-17T10:27:46.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6dd400640 1 -- start start 2024-04-17T10:27:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6dd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa6d806eac0 0x7fa6d8143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6dd400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d815ae40 con 0x7fa6d806e6e0 2024-04-17T10:27:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6dd400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d8071440 con 0x7fa6d80cf0c0 2024-04-17T10:27:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6dd400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d8071720 con 0x7fa6d806eac0 2024-04-17T10:27:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d6800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa6d806e6e0 0x7fa6d80ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43176/0 (socket says 172.21.15.67:43176) 2024-04-17T10:27:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d6800640 1 -- 172.21.15.67:0/2421207947 learned_addr learned my addr 172.21.15.67:0/2421207947 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d5e00640 1 --2- 172.21.15.67:0/2421207947 >> v2:172.21.15.67:3300/0 conn(0x7fa6d806eac0 0x7fa6d8143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d5400640 1 -- 172.21.15.67:0/2421207947 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412290699 0 0) 0x7fa6d815ae40 con 0x7fa6d806e6e0 2024-04-17T10:27:46.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d5400640 1 -- 172.21.15.67:0/2421207947 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6bc003530 con 0x7fa6d806e6e0 2024-04-17T10:27:46.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d5e00640 1 -- 172.21.15.67:0/2421207947 >> v1:172.21.15.86:6789/0 conn(0x7fa6d80cf0c0 legacy=0x7fa6d8144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:46.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d5e00640 1 -- 172.21.15.67:0/2421207947 >> v1:172.21.15.67:6789/0 conn(0x7fa6d806e6e0 legacy=0x7fa6d80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.518+0000 7fa6d5e00640 1 -- 172.21.15.67:0/2421207947 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d8071a00 con 0x7fa6d806eac0 2024-04-17T10:27:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6d5e00640 1 --2- 172.21.15.67:0/2421207947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806eac0 0x7fa6d8143ae0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40097c0 tx=0x7fa6c401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31bfae7177ad37a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6d5400640 1 -- 172.21.15.67:0/2421207947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40b1070 con 0x7fa6d806eac0 2024-04-17T10:27:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6d5400640 1 -- 172.21.15.67:0/2421207947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c40ac600 con 0x7fa6d806eac0 2024-04-17T10:27:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6d5400640 1 -- 172.21.15.67:0/2421207947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40ac9d0 con 0x7fa6d806eac0 2024-04-17T10:27:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 -- 172.21.15.67:0/2421207947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806eac0 msgr2=0x7fa6d8143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 --2- 172.21.15.67:0/2421207947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806eac0 0x7fa6d8143ae0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40097c0 tx=0x7fa6c401c3a0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 -- 172.21.15.67:0/2421207947 shutdown_connections 2024-04-17T10:27:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 --2- 172.21.15.67:0/2421207947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806eac0 0x7fa6d8143ae0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 -- 172.21.15.67:0/2421207947 >> 172.21.15.67:0/2421207947 conn(0x7fa6d806ac90 msgr2=0x7fa6d806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 -- 172.21.15.67:0/2421207947 shutdown_connections 2024-04-17T10:27:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 -- 172.21.15.67:0/2421207947 wait complete. 2024-04-17T10:27:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.522+0000 7fa6dd400640 1 Processor -- start 2024-04-17T10:27:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 -- start start 2024-04-17T10:27:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806e6e0 0x7fa6d812ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d806eac0 0x7fa6d8142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6d80cf0c0 0x7fa6d80b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d81680b0 con 0x7fa6d806e6e0 2024-04-17T10:27:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8167f30 con 0x7fa6d806eac0 2024-04-17T10:27:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8168230 con 0x7fa6d80cf0c0 2024-04-17T10:27:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806e6e0 0x7fa6d812ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806e6e0 0x7fa6d812ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40596/0 (socket says 172.21.15.67:40596) 2024-04-17T10:27:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 -- 172.21.15.67:0/677821923 learned_addr learned my addr 172.21.15.67:0/677821923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d5e00640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d806eac0 0x7fa6d8142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d7200640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6d80cf0c0 0x7fa6d80b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 -- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6d80cf0c0 msgr2=0x7fa6d80b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6d80cf0c0 0x7fa6d80b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 -- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d806eac0 msgr2=0x7fa6d8142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d806eac0 0x7fa6d8142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 -- 172.21.15.67:0/677821923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d81f68a0 con 0x7fa6d806e6e0 2024-04-17T10:27:46.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6d6800640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806e6e0 0x7fa6d812ef20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c480 tx=0x7fa6c800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:46.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6cea00640 1 -- 172.21.15.67:0/677821923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8017070 con 0x7fa6d806e6e0 2024-04-17T10:27:46.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6cea00640 1 -- 172.21.15.67:0/677821923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c8005cf0 con 0x7fa6d806e6e0 2024-04-17T10:27:46.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6cea00640 1 -- 172.21.15.67:0/677821923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c8007290 con 0x7fa6d806e6e0 2024-04-17T10:27:46.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d81f6b80 con 0x7fa6d806e6e0 2024-04-17T10:27:46.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d81f7140 con 0x7fa6d806e6e0 2024-04-17T10:27:46.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.526+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d8072d50 con 0x7fa6d806e6e0 2024-04-17T10:27:46.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.530+0000 7fa6cea00640 1 -- 172.21.15.67:0/677821923 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa6c801a070 con 0x7fa6d806e6e0 2024-04-17T10:27:46.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.530+0000 7fa6cea00640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6b8066f60 0x7fa6b8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.530+0000 7fa6cea00640 1 -- 172.21.15.67:0/677821923 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7fa6c8084590 con 0x7fa6d806e6e0 2024-04-17T10:27:46.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.530+0000 7fa6d5e00640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6b8066f60 0x7fa6b8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.530+0000 7fa6d5e00640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6b8066f60 0x7fa6b8069420 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40ae000 tx=0x7fa6c40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:46.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.534+0000 7fa6cea00640 1 -- 172.21.15.67:0/677821923 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa6c8054d80 con 0x7fa6d806e6e0 2024-04-17T10:27:46.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6d8074e30 con 0x7fa6d806e6e0 2024-04-17T10:27:46.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6cea00640 1 -- 172.21.15.67:0/677821923 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fa6c8058e20 con 0x7fa6d806e6e0 2024-04-17T10:27:46.724 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:46.724 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1835906721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1835906721},{"type":"v1","addr":"172.21.15.86:6836","nonce":1835906721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2085506860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2085506860},{"type":"v1","addr":"172.21.15.86:6837","nonce":2085506860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2728147711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2728147711},{"type":"v1","addr":"172.21.15.67:6834","nonce":2728147711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:26:15.408252+0000","modified":"2024-04-17T10:26:23.414956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1409597842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1409597842},{"type":"v1","addr":"172.21.15.67:6835","nonce":1409597842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-17T10:27:46.724 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-04-17T10:27:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6b8066f60 msgr2=0x7fa6b8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6b8066f60 0x7fa6b8069420 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40ae000 tx=0x7fa6c40a7000 comp rx=0 tx=0).stop 2024-04-17T10:27:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806e6e0 msgr2=0x7fa6d812ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806e6e0 0x7fa6d812ef20 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800c480 tx=0x7fa6c800c940 comp rx=0 tx=0).stop 2024-04-17T10:27:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 shutdown_connections 2024-04-17T10:27:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6b8066f60 0x7fa6b8069420 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6d80cf0c0 0x7fa6d80b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa6d806eac0 0x7fa6d8142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 --2- 172.21.15.67:0/677821923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6d806e6e0 0x7fa6d812ef20 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 >> 172.21.15.67:0/677821923 conn(0x7fa6d806ac90 msgr2=0x7fa6d80b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:46.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 shutdown_connections 2024-04-17T10:27:46.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.730+0000 7fa6dd400640 1 -- 172.21.15.67:0/677821923 wait complete. 2024-04-17T10:27:46.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f3056800640 1 Processor -- start 2024-04-17T10:27:46.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f3056800640 1 -- start start 2024-04-17T10:27:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f3056800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f305015b970 0x7f30500cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f3056800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f305015cf10 con 0x7f305006b6a0 2024-04-17T10:27:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f3056800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30500cf0c0 con 0x7f30500cd690 2024-04-17T10:27:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f3056800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f305015c300 con 0x7f305015b970 2024-04-17T10:27:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f305006b6a0 0x7f30500ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43192/0 (socket says 172.21.15.67:43192) 2024-04-17T10:27:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304ea00640 1 -- 172.21.15.67:0/2555199273 learned_addr learned my addr 172.21.15.67:0/2555199273 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304f400640 1 --2- 172.21.15.67:0/2555199273 >> v2:172.21.15.86:3300/0 conn(0x7f305015b970 0x7f30500cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304e000640 1 -- 172.21.15.67:0/2555199273 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195196997 0 0) 0x7f305015cf10 con 0x7f305006b6a0 2024-04-17T10:27:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304e000640 1 -- 172.21.15.67:0/2555199273 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3034003530 con 0x7f305006b6a0 2024-04-17T10:27:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304f400640 1 -- 172.21.15.67:0/2555199273 >> v1:172.21.15.86:6789/0 conn(0x7f30500cd690 legacy=0x7f30500ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304f400640 1 -- 172.21.15.67:0/2555199273 >> v1:172.21.15.67:6789/0 conn(0x7f305006b6a0 legacy=0x7f30500ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.894+0000 7f304f400640 1 -- 172.21.15.67:0/2555199273 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f305015c5e0 con 0x7f305015b970 2024-04-17T10:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f304f400640 1 --2- 172.21.15.67:0/2555199273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 0x7f30500cf880 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30400097c0 tx=0x7f304001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11c6bdd956998142 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f304e000640 1 -- 172.21.15.67:0/2555199273 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400b0070 con 0x7f305015b970 2024-04-17T10:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f304e000640 1 -- 172.21.15.67:0/2555199273 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30400ab600 con 0x7f305015b970 2024-04-17T10:27:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f304e000640 1 -- 172.21.15.67:0/2555199273 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30400ab9b0 con 0x7f305015b970 2024-04-17T10:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f3056800640 1 -- 172.21.15.67:0/2555199273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 msgr2=0x7f30500cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f3056800640 1 --2- 172.21.15.67:0/2555199273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 0x7f30500cf880 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f30400097c0 tx=0x7f304001c410 comp rx=0 tx=0).stop 2024-04-17T10:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f3056800640 1 -- 172.21.15.67:0/2555199273 shutdown_connections 2024-04-17T10:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f3056800640 1 --2- 172.21.15.67:0/2555199273 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 0x7f30500cf880 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f3056800640 1 -- 172.21.15.67:0/2555199273 >> 172.21.15.67:0/2555199273 conn(0x7f30500b22a0 msgr2=0x7f30500b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f3056800640 1 -- 172.21.15.67:0/2555199273 shutdown_connections 2024-04-17T10:27:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.898+0000 7f3056800640 1 -- 172.21.15.67:0/2555199273 wait complete. 2024-04-17T10:27:46.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 Processor -- start 2024-04-17T10:27:46.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 -- start start 2024-04-17T10:27:46.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f305006b6a0 0x7f3050152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30500cd690 0x7f30500ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 0x7f30500c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500d0190 con 0x7f30500cd690 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500d0010 con 0x7f305015b970 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3050169da0 con 0x7f305006b6a0 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30500cd690 0x7f30500ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30500cd690 0x7f30500ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40610/0 (socket says 172.21.15.67:40610) 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 -- 172.21.15.67:0/1365844966 learned_addr learned my addr 172.21.15.67:0/1365844966 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304fe00640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 0x7f30500c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304f400640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f305006b6a0 0x7f3050152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 -- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f305006b6a0 msgr2=0x7f3050152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f305006b6a0 0x7f3050152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 -- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 msgr2=0x7f30500c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 0x7f30500c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 -- 172.21.15.67:0/1365844966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3050152a40 con 0x7f30500cd690 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f304ea00640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30500cd690 0x7f30500ca1e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f303c00c500 tx=0x7f303c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3047400640 1 -- 172.21.15.67:0/1365844966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c017070 con 0x7f30500cd690 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3047400640 1 -- 172.21.15.67:0/1365844966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303c005cf0 con 0x7f30500cd690 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3047400640 1 -- 172.21.15.67:0/1365844966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303c0072f0 con 0x7f30500cd690 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305006ea30 con 0x7f30500cd690 2024-04-17T10:27:46.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305006eff0 con 0x7f30500cd690 2024-04-17T10:27:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.902+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f305009e900 con 0x7f30500cd690 2024-04-17T10:27:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.906+0000 7f3047400640 1 -- 172.21.15.67:0/1365844966 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f303c01a070 con 0x7f30500cd690 2024-04-17T10:27:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.906+0000 7f3047400640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3030066f60 0x7f3030069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.906+0000 7f3047400640 1 -- 172.21.15.67:0/1365844966 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f303c084560 con 0x7f30500cd690 2024-04-17T10:27:46.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.906+0000 7f304f400640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3030066f60 0x7f3030069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:46.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.906+0000 7f304f400640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3030066f60 0x7f3030069420 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f30400ad000 tx=0x7f3040008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:46.910+0000 7f3047400640 1 -- 172.21.15.67:0/1365844966 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f303c054d50 con 0x7f30500cd690 2024-04-17T10:27:47.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.070+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30500caa90 con 0x7f30500cd690 2024-04-17T10:27:47.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3047400640 1 -- 172.21.15.67:0/1365844966 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17008 (secure 0 0 0) 0x7f303c058df0 con 0x7f30500cd690 2024-04-17T10:27:47.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:47.067 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:27:28.880541+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:26:13.215866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-17T10:26:14.398708+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":42,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:27:47.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3030066f60 msgr2=0x7f3030069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3030066f60 0x7f3030069420 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f30400ad000 tx=0x7f3040008040 comp rx=0 tx=0).stop 2024-04-17T10:27:47.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30500cd690 msgr2=0x7f30500ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30500cd690 0x7f30500ca1e0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f303c00c500 tx=0x7f303c00c9c0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 shutdown_connections 2024-04-17T10:27:47.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3030066f60 0x7f3030069420 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f305015b970 0x7f30500c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f30500cd690 0x7f30500ca1e0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 --2- 172.21.15.67:0/1365844966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f305006b6a0 0x7f3050152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 >> 172.21.15.67:0/1365844966 conn(0x7f30500b22a0 msgr2=0x7f30500ad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 shutdown_connections 2024-04-17T10:27:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.074+0000 7f3056800640 1 -- 172.21.15.67:0/1365844966 wait complete. 2024-04-17T10:27:47.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:26:13.215866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:26:14.398708+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 42, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:27:47.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:27:47.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.238+0000 7f2e00e00640 1 Processor -- start 2024-04-17T10:27:47.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.238+0000 7f2e00e00640 1 -- start start 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2e00e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0aef60 0x7f2dfc0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2e00e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0cce60 0x7f2dfc06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2e00e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc0b4990 con 0x7f2dfc06e7b0 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2e00e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc0cd2c0 con 0x7f2dfc0cce60 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2e00e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc0af340 con 0x7f2dfc0aef60 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2dfa800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2dfc06e7b0 0x7f2dfc0713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43208/0 (socket says 172.21.15.67:43208) 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2dfa800640 1 -- 172.21.15.67:0/179690943 learned_addr learned my addr 172.21.15.67:0/179690943 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2df3e00640 1 --2- 172.21.15.67:0/179690943 >> v2:172.21.15.86:3300/0 conn(0x7f2dfc0cce60 0x7f2dfc06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2df9e00640 1 --2- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0aef60 0x7f2dfc0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2df3e00640 1 -- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0aef60 msgr2=0x7f2dfc0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:47.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2df3e00640 1 --2- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0aef60 0x7f2dfc0cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2df3e00640 1 -- 172.21.15.67:0/179690943 >> v1:172.21.15.67:6789/0 conn(0x7f2dfc06e7b0 legacy=0x7f2dfc0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.242+0000 7f2df3e00640 1 -- 172.21.15.67:0/179690943 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc15ae90 con 0x7f2dfc0cce60 2024-04-17T10:27:47.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2df3e00640 1 --2- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc06e270 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2de40097c0 tx=0x7f2de401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aec99f55b2e8554 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2df9400640 1 -- 172.21.15.67:0/179690943 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de40b0070 con 0x7f2dfc0cce60 2024-04-17T10:27:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2df9400640 1 -- 172.21.15.67:0/179690943 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de40ab600 con 0x7f2dfc0cce60 2024-04-17T10:27:47.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2df9400640 1 -- 172.21.15.67:0/179690943 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de40ab9d0 con 0x7f2dfc0cce60 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 -- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0cce60 msgr2=0x7f2dfc06e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 --2- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc06e270 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2de40097c0 tx=0x7f2de401c410 comp rx=0 tx=0).stop 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 -- 172.21.15.67:0/179690943 shutdown_connections 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 --2- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc06e270 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 --2- 172.21.15.67:0/179690943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc0aef60 0x7f2dfc0cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 -- 172.21.15.67:0/179690943 >> 172.21.15.67:0/179690943 conn(0x7f2dfc0b1740 msgr2=0x7f2dfc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 -- 172.21.15.67:0/179690943 shutdown_connections 2024-04-17T10:27:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 -- 172.21.15.67:0/179690943 wait complete. 2024-04-17T10:27:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 Processor -- start 2024-04-17T10:27:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.246+0000 7f2e00e00640 1 -- start start 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc06e7b0 0x7f2dfc159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0aef60 0x7f2dfc133930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc15bc10 con 0x7f2dfc0cce60 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc15b410 con 0x7f2dfc0aef60 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc14e280 con 0x7f2dfc06e7b0 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2df9e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc06e7b0 0x7f2dfc159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40612/0 (socket says 172.21.15.67:40612) 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 -- 172.21.15.67:0/2893546148 learned_addr learned my addr 172.21.15.67:0/2893546148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2df3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0aef60 0x7f2dfc133930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 -- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc06e7b0 msgr2=0x7f2dfc159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc06e7b0 0x7f2dfc159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 -- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0aef60 msgr2=0x7f2dfc133930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0aef60 0x7f2dfc133930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 -- 172.21.15.67:0/2893546148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc151cf0 con 0x7f2dfc0cce60 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2dfa800640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc133e70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2df400ca80 tx=0x7f2df400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2df2000640 1 -- 172.21.15.67:0/2893546148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4017070 con 0x7f2dfc0cce60 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2df2000640 1 -- 172.21.15.67:0/2893546148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df400ac30 con 0x7f2dfc0cce60 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc1362a0 con 0x7f2dfc0cce60 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2df2000640 1 -- 172.21.15.67:0/2893546148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df4012600 con 0x7f2dfc0cce60 2024-04-17T10:27:47.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc152290 con 0x7f2dfc0cce60 2024-04-17T10:27:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc04ef30 con 0x7f2dfc0cce60 2024-04-17T10:27:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.250+0000 7f2df2000640 1 -- 172.21.15.67:0/2893546148 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2df401a070 con 0x7f2dfc0cce60 2024-04-17T10:27:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.254+0000 7f2df2000640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2dd8066f90 0x7f2dd8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.254+0000 7f2df2000640 1 -- 172.21.15.67:0/2893546148 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f2df4084d50 con 0x7f2dfc0cce60 2024-04-17T10:27:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.254+0000 7f2df9e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2dd8066f90 0x7f2dd8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.254+0000 7f2df9e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2dd8066f90 0x7f2dd8069450 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0045b0 tx=0x7f2dec00b3a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.258+0000 7f2df2000640 1 -- 172.21.15.67:0/2893546148 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2df4088030 con 0x7f2dfc0cce60 2024-04-17T10:27:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.418+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2dfc072690 con 0x7f2dfc0cce60 2024-04-17T10:27:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.418+0000 7f2df2000640 1 -- 172.21.15.67:0/2893546148 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2dfc0b4990 con 0x7f2dfc0cce60 2024-04-17T10:27:47.413 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2dd8066f90 msgr2=0x7f2dd8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2dd8066f90 0x7f2dd8069450 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2dec0045b0 tx=0x7f2dec00b3a0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc0cce60 msgr2=0x7f2dfc133e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc133e70 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f2df400ca80 tx=0x7f2df400cf40 comp rx=0 tx=0).stop 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 shutdown_connections 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2dd8066f90 0x7f2dd8069450 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2dfc0cce60 0x7f2dfc133e70 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2dfc0aef60 0x7f2dfc133930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 --2- 172.21.15.67:0/2893546148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2dfc06e7b0 0x7f2dfc159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 >> 172.21.15.67:0/2893546148 conn(0x7f2dfc0b1740 msgr2=0x7f2dfc15d5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:47.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 shutdown_connections 2024-04-17T10:27:47.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.422+0000 7f2e00e00640 1 -- 172.21.15.67:0/2893546148 wait complete. 2024-04-17T10:27:47.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:27:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.582+0000 7fa869600640 1 Processor -- start 2024-04-17T10:27:47.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.582+0000 7fa869600640 1 -- start start 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa869600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa86415f230 0x7fa86409a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa869600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa86409a3c0 0x7fa8640b17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa869600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8640b3290 con 0x7fa8640cc2e0 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa869600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa86415f610 con 0x7fa86409a3c0 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa869600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8640cf6b0 con 0x7fa86415f230 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa86409a3c0 0x7fa8640b17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa86409a3c0 0x7fa8640b17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40618/0 (socket says 172.21.15.67:40618) 2024-04-17T10:27:47.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 -- 172.21.15.67:0/3075692814 learned_addr learned my addr 172.21.15.67:0/3075692814 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa862a00640 1 --2- 172.21.15.67:0/3075692814 >> v2:172.21.15.86:3300/0 conn(0x7fa86415f230 0x7fa86409a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 -- 172.21.15.67:0/3075692814 >> v2:172.21.15.86:3300/0 conn(0x7fa86415f230 msgr2=0x7fa86409a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 --2- 172.21.15.67:0/3075692814 >> v2:172.21.15.86:3300/0 conn(0x7fa86415f230 0x7fa86409a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 -- 172.21.15.67:0/3075692814 >> v1:172.21.15.86:6789/0 conn(0x7fa8640cc2e0 legacy=0x7fa864099cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 -- 172.21.15.67:0/3075692814 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8640b0fc0 con 0x7fa86409a3c0 2024-04-17T10:27:47.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa862a00640 1 --2- 172.21.15.67:0/3075692814 >> v2:172.21.15.86:3300/0 conn(0x7fa86415f230 0x7fa86409a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.586+0000 7fa863400640 1 --2- 172.21.15.67:0/3075692814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa86409a3c0 0x7fa8640b17c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa86409fa50 tx=0x7fa85c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d279d5bdaf9b29a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa862000640 1 -- 172.21.15.67:0/3075692814 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c0b1070 con 0x7fa86409a3c0 2024-04-17T10:27:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa862000640 1 -- 172.21.15.67:0/3075692814 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85c0ac620 con 0x7fa86409a3c0 2024-04-17T10:27:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa862000640 1 -- 172.21.15.67:0/3075692814 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c0ac9f0 con 0x7fa86409a3c0 2024-04-17T10:27:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- 172.21.15.67:0/3075692814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa86409a3c0 msgr2=0x7fa8640b17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 --2- 172.21.15.67:0/3075692814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa86409a3c0 0x7fa8640b17c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa86409fa50 tx=0x7fa85c01c420 comp rx=0 tx=0).stop 2024-04-17T10:27:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- 172.21.15.67:0/3075692814 shutdown_connections 2024-04-17T10:27:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 --2- 172.21.15.67:0/3075692814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa86409a3c0 0x7fa8640b17c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 --2- 172.21.15.67:0/3075692814 >> v2:172.21.15.86:3300/0 conn(0x7fa86415f230 0x7fa86409a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- 172.21.15.67:0/3075692814 >> 172.21.15.67:0/3075692814 conn(0x7fa8640b34a0 msgr2=0x7fa8640b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- 172.21.15.67:0/3075692814 shutdown_connections 2024-04-17T10:27:47.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- 172.21.15.67:0/3075692814 wait complete. 2024-04-17T10:27:47.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 Processor -- start 2024-04-17T10:27:47.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- start start 2024-04-17T10:27:47.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86409a3c0 0x7fa864158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8640cc2e0 0x7fa8640c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86415f230 0x7fa8640c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640b3050 con 0x7fa8640cc2e0 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640b2ed0 con 0x7fa86415f230 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa869600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640b3fc0 con 0x7fa86409a3c0 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa85a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8640cc2e0 0x7fa8640c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa85a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8640cc2e0 0x7fa8640c8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40634/0 (socket says 172.21.15.67:40634) 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa85a000640 1 -- 172.21.15.67:0/3547068047 learned_addr learned my addr 172.21.15.67:0/3547068047 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa862a00640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86409a3c0 0x7fa864158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.590+0000 7fa863400640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86415f230 0x7fa8640c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85a000640 1 -- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86409a3c0 msgr2=0x7fa864158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85a000640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86409a3c0 0x7fa864158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85a000640 1 -- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86415f230 msgr2=0x7fa8640c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85a000640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86415f230 0x7fa8640c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85a000640 1 -- 172.21.15.67:0/3547068047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8640c9d90 con 0x7fa8640cc2e0 2024-04-17T10:27:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa862a00640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86409a3c0 0x7fa864158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:47.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85a000640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8640cc2e0 0x7fa8640c8a80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c450 tx=0x7fa84c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85b400640 1 -- 172.21.15.67:0/3547068047 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c012830 con 0x7fa8640cc2e0 2024-04-17T10:27:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85b400640 1 -- 172.21.15.67:0/3547068047 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c017070 con 0x7fa8640cc2e0 2024-04-17T10:27:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85b400640 1 -- 172.21.15.67:0/3547068047 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c012c30 con 0x7fa8640cc2e0 2024-04-17T10:27:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8640b0bf0 con 0x7fa8640cc2e0 2024-04-17T10:27:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa864165cb0 con 0x7fa8640cc2e0 2024-04-17T10:27:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86404ef30 con 0x7fa8640cc2e0 2024-04-17T10:27:47.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85b400640 1 -- 172.21.15.67:0/3547068047 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa84c01c460 con 0x7fa8640cc2e0 2024-04-17T10:27:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85b400640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa840066f60 0x7fa840069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa85b400640 1 -- 172.21.15.67:0/3547068047 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7fa84c084ec0 con 0x7fa8640cc2e0 2024-04-17T10:27:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.594+0000 7fa862a00640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa840066f60 0x7fa840069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.598+0000 7fa862a00640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa840066f60 0x7fa840069420 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa85400e3d0 tx=0x7fa854007a10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.598+0000 7fa85b400640 1 -- 172.21.15.67:0/3547068047 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa84c055d10 con 0x7fa8640cc2e0 2024-04-17T10:27:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.762+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8640b43b0 con 0x7fa8640cc2e0 2024-04-17T10:27:47.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.762+0000 7fa85b400640 1 -- 172.21.15.67:0/3547068047 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa84c055610 con 0x7fa8640cc2e0 2024-04-17T10:27:47.758 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa840066f60 msgr2=0x7fa840069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa840066f60 0x7fa840069420 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa85400e3d0 tx=0x7fa854007a10 comp rx=0 tx=0).stop 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8640cc2e0 msgr2=0x7fa8640c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8640cc2e0 0x7fa8640c8a80 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c450 tx=0x7fa84c00c910 comp rx=0 tx=0).stop 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 shutdown_connections 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa840066f60 0x7fa840069420 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa86415f230 0x7fa8640c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8640cc2e0 0x7fa8640c8a80 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 --2- 172.21.15.67:0/3547068047 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa86409a3c0 0x7fa864158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 >> 172.21.15.67:0/3547068047 conn(0x7fa8640b34a0 msgr2=0x7fa86406f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 shutdown_connections 2024-04-17T10:27:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.766+0000 7fa869600640 1 -- 172.21.15.67:0/3547068047 wait complete. 2024-04-17T10:27:47.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:27:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.926+0000 7fe20c400640 1 Processor -- start 2024-04-17T10:27:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.926+0000 7fe20c400640 1 -- start start 2024-04-17T10:27:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe2040b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20c400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe2040b3920 0x7fe2040d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20c400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2041616a0 con 0x7fe2041474c0 2024-04-17T10:27:47.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20c400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe204161940 con 0x7fe2040b3920 2024-04-17T10:27:47.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe204161c20 con 0x7fe20406b3a0 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20a400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe2040b3920 0x7fe2040d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe209000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe2040b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe209a00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe2041474c0 0x7fe20406ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54826/0 (socket says 172.21.15.67:54826) 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe209a00640 1 -- 172.21.15.67:0/2628589493 learned_addr learned my addr 172.21.15.67:0/2628589493 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20a400640 1 -- 172.21.15.67:0/2628589493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 msgr2=0x7fe2040b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20a400640 1 --2- 172.21.15.67:0/2628589493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe2040b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20a400640 1 -- 172.21.15.67:0/2628589493 >> v1:172.21.15.86:6789/0 conn(0x7fe2041474c0 legacy=0x7fe20406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe20a400640 1 -- 172.21.15.67:0/2628589493 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe204161f00 con 0x7fe2040b3920 2024-04-17T10:27:47.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.930+0000 7fe203e00640 1 -- 172.21.15.67:0/2628589493 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003472490 0 0) 0x7fe2041616a0 con 0x7fe2041474c0 2024-04-17T10:27:47.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20a400640 1 --2- 172.21.15.67:0/2628589493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040b3920 0x7fe2040d0480 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009d90 tx=0x7fe1fc01f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba9bdca2b2f7fcb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe203e00640 1 -- 172.21.15.67:0/2628589493 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc0b4070 con 0x7fe2040b3920 2024-04-17T10:27:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe203e00640 1 -- 172.21.15.67:0/2628589493 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1fc0af600 con 0x7fe2040b3920 2024-04-17T10:27:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe203e00640 1 -- 172.21.15.67:0/2628589493 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1fc0af9d0 con 0x7fe2040b3920 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 -- 172.21.15.67:0/2628589493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040b3920 msgr2=0x7fe2040d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 --2- 172.21.15.67:0/2628589493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040b3920 0x7fe2040d0480 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe1fc009d90 tx=0x7fe1fc01f150 comp rx=0 tx=0).stop 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 -- 172.21.15.67:0/2628589493 shutdown_connections 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 --2- 172.21.15.67:0/2628589493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2040b3920 0x7fe2040d0480 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 --2- 172.21.15.67:0/2628589493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe2040b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 -- 172.21.15.67:0/2628589493 >> 172.21.15.67:0/2628589493 conn(0x7fe2040b5ef0 msgr2=0x7fe204099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 -- 172.21.15.67:0/2628589493 shutdown_connections 2024-04-17T10:27:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 -- 172.21.15.67:0/2628589493 wait complete. 2024-04-17T10:27:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.934+0000 7fe20c400640 1 Processor -- start 2024-04-17T10:27:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 -- start start 2024-04-17T10:27:47.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe20413e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2040b3920 0x7fe2040ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041474c0 0x7fe2040adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe204162420 con 0x7fe2040b3920 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040b2250 con 0x7fe2041474c0 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2040d09c0 con 0x7fe20406b3a0 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2040b3920 0x7fe2040ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe20413e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2040b3920 0x7fe2040ad720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40636/0 (socket says 172.21.15.67:40636) 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 -- 172.21.15.67:0/2717712271 learned_addr learned my addr 172.21.15.67:0/2717712271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20a400640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041474c0 0x7fe2040adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 -- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 msgr2=0x7fe20413e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe20413e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 -- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041474c0 msgr2=0x7fe2040adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041474c0 0x7fe2040adc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:47.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 -- 172.21.15.67:0/2717712271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe20415d8b0 con 0x7fe2040b3920 2024-04-17T10:27:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe209000640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2040b3920 0x7fe2040ad720 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000e960 tx=0x7fe1f000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe201600640 1 -- 172.21.15.67:0/2717712271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f0017070 con 0x7fe2040b3920 2024-04-17T10:27:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe201600640 1 -- 172.21.15.67:0/2717712271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1f000ca00 con 0x7fe2040b3920 2024-04-17T10:27:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe20415d2f0 con 0x7fe2040b3920 2024-04-17T10:27:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe201600640 1 -- 172.21.15.67:0/2717712271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1f000cdb0 con 0x7fe2040b3920 2024-04-17T10:27:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe204145480 con 0x7fe2040b3920 2024-04-17T10:27:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.938+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe20404ef30 con 0x7fe2040b3920 2024-04-17T10:27:47.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.942+0000 7fe201600640 1 -- 172.21.15.67:0/2717712271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe1f001a070 con 0x7fe2040b3920 2024-04-17T10:27:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.942+0000 7fe201600640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe1e4066f60 0x7fe1e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.942+0000 7fe201600640 1 -- 172.21.15.67:0/2717712271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7fe1f0085230 con 0x7fe2040b3920 2024-04-17T10:27:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.942+0000 7fe209a00640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe1e4066f60 0x7fe1e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:47.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.942+0000 7fe209a00640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe1e4066f60 0x7fe1e4069420 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8000f70 tx=0x7fe1f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:47.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:47.946+0000 7fe201600640 1 -- 172.21.15.67:0/2717712271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe1f00559a0 con 0x7fe2040b3920 2024-04-17T10:27:48.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2040727e0 con 0x7fe2040b3920 2024-04-17T10:27:48.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe201600640 1 -- 172.21.15.67:0/2717712271 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe1f0059a40 con 0x7fe2040b3920 2024-04-17T10:27:48.099 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:27:48.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe1e4066f60 msgr2=0x7fe1e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe1e4066f60 0x7fe1e4069420 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe1f8000f70 tx=0x7fe1f8009040 comp rx=0 tx=0).stop 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2040b3920 msgr2=0x7fe2040ad720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2040b3920 0x7fe2040ad720 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe1f000e960 tx=0x7fe1f000ee20 comp rx=0 tx=0).stop 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 shutdown_connections 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe1e4066f60 0x7fe1e4069420 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe2041474c0 0x7fe2040adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe2040b3920 0x7fe2040ad720 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 --2- 172.21.15.67:0/2717712271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe20406b3a0 0x7fe20413e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 >> 172.21.15.67:0/2717712271 conn(0x7fe2040b5ef0 msgr2=0x7fe2040b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 shutdown_connections 2024-04-17T10:27:48.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.106+0000 7fe20c400640 1 -- 172.21.15.67:0/2717712271 wait complete. 2024-04-17T10:27:48.120 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:27:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.266+0000 7f00b1c00640 1 Processor -- start 2024-04-17T10:27:48.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.266+0000 7f00b1c00640 1 -- start start 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00b1c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00ac0ce1b0 0x7f00ac0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00b1c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f00ac15cc00 0x7f00ac15f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00b1c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0cead0 con 0x7f00ac15fa00 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00b1c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac15be60 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00b1c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac160230 con 0x7f00ac15cc00 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00aaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00ac0ce1b0 0x7f00ac0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00ab400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f00ac15fa00 0x7f00ac070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43218/0 (socket says 172.21.15.67:43218) 2024-04-17T10:27:48.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00ab400640 1 -- 172.21.15.67:0/3991445063 learned_addr learned my addr 172.21.15.67:0/3991445063 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:48.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00a2000640 1 --2- 172.21.15.67:0/3991445063 >> v2:172.21.15.86:3300/0 conn(0x7f00ac15cc00 0x7f00ac15f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00aaa00640 1 -- 172.21.15.67:0/3991445063 >> v2:172.21.15.86:3300/0 conn(0x7f00ac15cc00 msgr2=0x7f00ac15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:48.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00aaa00640 1 --2- 172.21.15.67:0/3991445063 >> v2:172.21.15.86:3300/0 conn(0x7f00ac15cc00 0x7f00ac15f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00aaa00640 1 -- 172.21.15.67:0/3991445063 >> v1:172.21.15.67:6789/0 conn(0x7f00ac15fa00 legacy=0x7f00ac070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00aaa00640 1 -- 172.21.15.67:0/3991445063 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac15c120 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00a2000640 1 --2- 172.21.15.67:0/3991445063 >> v2:172.21.15.86:3300/0 conn(0x7f00ac15cc00 0x7f00ac15f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:48.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.270+0000 7f00aa000640 1 -- 172.21.15.67:0/3991445063 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849770209 0 0) 0x7f00ac0cead0 con 0x7f00ac15fa00 2024-04-17T10:27:48.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00aaa00640 1 --2- 172.21.15.67:0/3991445063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ce590 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f00ac072de0 tx=0x7f009c01f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9fa5b5b7f3d0b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00aa000640 1 -- 172.21.15.67:0/3991445063 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0b4070 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00aa000640 1 -- 172.21.15.67:0/3991445063 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c0af600 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00aa000640 1 -- 172.21.15.67:0/3991445063 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0af9d0 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- 172.21.15.67:0/3991445063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 msgr2=0x7f00ac0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 --2- 172.21.15.67:0/3991445063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ce590 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f00ac072de0 tx=0x7f009c01f110 comp rx=0 tx=0).stop 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- 172.21.15.67:0/3991445063 shutdown_connections 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 --2- 172.21.15.67:0/3991445063 >> v2:172.21.15.86:3300/0 conn(0x7f00ac15cc00 0x7f00ac15f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 --2- 172.21.15.67:0/3991445063 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ce590 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- 172.21.15.67:0/3991445063 >> 172.21.15.67:0/3991445063 conn(0x7f00ac0b33f0 msgr2=0x7f00ac0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- 172.21.15.67:0/3991445063 shutdown_connections 2024-04-17T10:27:48.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- 172.21.15.67:0/3991445063 wait complete. 2024-04-17T10:27:48.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 Processor -- start 2024-04-17T10:27:48.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- start start 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ac15cc00 0x7f00ac0c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ac15fa00 0x7f00ac155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac159730 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac15c6e0 con 0x7f00ac15cc00 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00b1c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1598b0 con 0x7f00ac15fa00 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.274+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40656/0 (socket says 172.21.15.67:40656) 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 -- 172.21.15.67:0/1684971153 learned_addr learned my addr 172.21.15.67:0/1684971153 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a2000640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ac15cc00 0x7f00ac0c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00ab400640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ac15fa00 0x7f00ac155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 -- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ac15fa00 msgr2=0x7f00ac155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ac15fa00 0x7f00ac155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 -- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ac15cc00 msgr2=0x7f00ac0c9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ac15cc00 0x7f00ac0c9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 -- 172.21.15.67:0/1684971153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac0cd190 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a2000640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ac15cc00 0x7f00ac0c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00ab400640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ac15fa00 0x7f00ac155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00aaa00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ccc50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f009c0b2040 tx=0x7f009c01beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a3400640 1 -- 172.21.15.67:0/1684971153 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0b4040 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a3400640 1 -- 172.21.15.67:0/1684971153 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c0c1070 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a3400640 1 -- 172.21.15.67:0/1684971153 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0ba8d0 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac0c8e30 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0c96d0 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac070ab0 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a3400640 1 -- 172.21.15.67:0/1684971153 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f009c0af480 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a3400640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0088066f60 0x7f0088069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.278+0000 7f00a3400640 1 -- 172.21.15.67:0/1684971153 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f009c123fe0 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.282+0000 7f00a2000640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0088066f60 0x7f0088069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.282+0000 7f00a2000640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0088066f60 0x7f0088069420 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f00ac1559c0 tx=0x7f0094008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.282+0000 7f00a3400640 1 -- 172.21.15.67:0/1684971153 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f009c018c70 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.478+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00ac068120 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00a3400640 1 -- 172.21.15.67:0/1684971153 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f009c0f4fb0 con 0x7f00ac0ce1b0 2024-04-17T10:27:48.475 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:48.475 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1835906721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1835906721},{"type":"v1","addr":"172.21.15.86:6836","nonce":1835906721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2085506860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2085506860},{"type":"v1","addr":"172.21.15.86:6837","nonce":2085506860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2728147711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2728147711},{"type":"v1","addr":"172.21.15.67:6834","nonce":2728147711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:26:15.408252+0000","modified":"2024-04-17T10:26:23.414956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1409597842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1409597842},{"type":"v1","addr":"172.21.15.67:6835","nonce":1409597842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0088066f60 msgr2=0x7f0088069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0088066f60 0x7f0088069420 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f00ac1559c0 tx=0x7f0094008040 comp rx=0 tx=0).stop 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 msgr2=0x7f00ac0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ccc50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f009c0b2040 tx=0x7f009c01beb0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 shutdown_connections 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0088066f60 0x7f0088069420 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00ac15fa00 0x7f00ac155bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00ac15cc00 0x7f00ac0c9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 --2- 172.21.15.67:0/1684971153 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00ac0ce1b0 0x7f00ac0ccc50 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 >> 172.21.15.67:0/1684971153 conn(0x7f00ac0b33f0 msgr2=0x7f00ac0b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 shutdown_connections 2024-04-17T10:27:48.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.482+0000 7f00b1c00640 1 -- 172.21.15.67:0/1684971153 wait complete. 2024-04-17T10:27:48.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:27:48.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82b5200640 1 Processor -- start 2024-04-17T10:27:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82b5200640 1 -- start start 2024-04-17T10:27:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82b5200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82b0071310 0x7f82b0071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82b5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b00cc300 0x7f82b00cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82b5200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b006e2f0 con 0x7f82b0145040 2024-04-17T10:27:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82b5200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b00a86c0 con 0x7f82b0071310 2024-04-17T10:27:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82b5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b0071cb0 con 0x7f82b00cc300 2024-04-17T10:27:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82ade00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82b0145040 0x7f82b006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43228/0 (socket says 172.21.15.67:43228) 2024-04-17T10:27:48.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82ade00640 1 -- 172.21.15.67:0/3720206209 learned_addr learned my addr 172.21.15.67:0/3720206209 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82ae800640 1 --2- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b00cc300 0x7f82b00cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82a7e00640 1 --2- 172.21.15.67:0/3720206209 >> v2:172.21.15.86:3300/0 conn(0x7f82b0071310 0x7f82b0071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.642+0000 7f82ad400640 1 -- 172.21.15.67:0/3720206209 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863394667 0 0) 0x7f82b006e2f0 con 0x7f82b0145040 2024-04-17T10:27:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82ad400640 1 -- 172.21.15.67:0/3720206209 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8294003530 con 0x7f82b0145040 2024-04-17T10:27:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82a7e00640 1 -- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b00cc300 msgr2=0x7f82b00cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82a7e00640 1 --2- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b00cc300 0x7f82b00cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82a7e00640 1 -- 172.21.15.67:0/3720206209 >> v1:172.21.15.67:6789/0 conn(0x7f82b0145040 legacy=0x7f82b006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82a7e00640 1 -- 172.21.15.67:0/3720206209 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b006d6e0 con 0x7f82b0071310 2024-04-17T10:27:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82ae800640 1 --2- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b00cc300 0x7f82b00cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82a7e00640 1 --2- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b0071770 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f82980097c0 tx=0x7f829801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0736c6411d849af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82ad400640 1 -- 172.21.15.67:0/3720206209 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980b5070 con 0x7f82b0071310 2024-04-17T10:27:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82ad400640 1 -- 172.21.15.67:0/3720206209 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82980a8990 con 0x7f82b0071310 2024-04-17T10:27:48.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.646+0000 7f82ad400640 1 -- 172.21.15.67:0/3720206209 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980a8d60 con 0x7f82b0071310 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 msgr2=0x7f82b0071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 --2- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b0071770 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f82980097c0 tx=0x7f829801ec90 comp rx=0 tx=0).stop 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- 172.21.15.67:0/3720206209 shutdown_connections 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 --2- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b00cc300 0x7f82b00cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 --2- 172.21.15.67:0/3720206209 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b0071770 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- 172.21.15.67:0/3720206209 >> 172.21.15.67:0/3720206209 conn(0x7f82b006ac90 msgr2=0x7f82b006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- 172.21.15.67:0/3720206209 shutdown_connections 2024-04-17T10:27:48.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- 172.21.15.67:0/3720206209 wait complete. 2024-04-17T10:27:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 Processor -- start 2024-04-17T10:27:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- start start 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b00c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b00cc300 0x7f82b0141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b0145040 0x7f82b013f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b006df80 con 0x7f82b00cc300 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b006dca0 con 0x7f82b0071310 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82b5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b006d370 con 0x7f82b0145040 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b00cc300 0x7f82b0141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b00cc300 0x7f82b0141850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40664/0 (socket says 172.21.15.67:40664) 2024-04-17T10:27:48.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 -- 172.21.15.67:0/2532575105 learned_addr learned my addr 172.21.15.67:0/2532575105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82ade00640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b00c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82ae800640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b0145040 0x7f82b013f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 -- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b0145040 msgr2=0x7f82b013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b0145040 0x7f82b013f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 -- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 msgr2=0x7f82b00c50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b00c50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 -- 172.21.15.67:0/2532575105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00c5790 con 0x7f82b00cc300 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82ade00640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b00c50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.650+0000 7f82a7e00640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b00cc300 0x7f82b0141850 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f82980b3040 tx=0x7f829801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82a6000640 1 -- 172.21.15.67:0/2532575105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980bf070 con 0x7f82b00cc300 2024-04-17T10:27:48.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82a6000640 1 -- 172.21.15.67:0/2532575105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82980b0700 con 0x7f82b00cc300 2024-04-17T10:27:48.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82a6000640 1 -- 172.21.15.67:0/2532575105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980b5040 con 0x7f82b00cc300 2024-04-17T10:27:48.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b01427a0 con 0x7f82b00cc300 2024-04-17T10:27:48.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b0142d60 con 0x7f82b00cc300 2024-04-17T10:27:48.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b0072d50 con 0x7f82b00cc300 2024-04-17T10:27:48.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82a6000640 1 -- 172.21.15.67:0/2532575105 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f82980b0af0 con 0x7f82b00cc300 2024-04-17T10:27:48.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82a6000640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82a6000640 1 -- 172.21.15.67:0/2532575105 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f8298121a50 con 0x7f82b00cc300 2024-04-17T10:27:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.654+0000 7f82ade00640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.658+0000 7f82ade00640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005890 tx=0x7f82a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.658+0000 7f82a6000640 1 -- 172.21.15.67:0/2532575105 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f82980187f0 con 0x7f82b00cc300 2024-04-17T10:27:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.822+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82b0074e30 con 0x7f82b00cc300 2024-04-17T10:27:48.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82a6000640 1 -- 172.21.15.67:0/2532575105 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17008 (secure 0 0 0) 0x7f82980f2890 con 0x7f82b00cc300 2024-04-17T10:27:48.819 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:48.819 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:27:28.880541+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:26:13.215866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-17T10:26:14.398708+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":42,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 msgr2=0x7f828c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f82a0005890 tx=0x7f82a0009040 comp rx=0 tx=0).stop 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b00cc300 msgr2=0x7f82b0141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b00cc300 0x7f82b0141850 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f82980b3040 tx=0x7f829801ba30 comp rx=0 tx=0).stop 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 shutdown_connections 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b0145040 0x7f82b013f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b00cc300 0x7f82b0141850 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 --2- 172.21.15.67:0/2532575105 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b0071310 0x7f82b00c50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 >> 172.21.15.67:0/2532575105 conn(0x7f82b006ac90 msgr2=0x7f82b00acc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 shutdown_connections 2024-04-17T10:27:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.826+0000 7f82b5200640 1 -- 172.21.15.67:0/2532575105 wait complete. 2024-04-17T10:27:48.841 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:27:48.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:27:48.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.986+0000 7fce86a00640 1 Processor -- start 2024-04-17T10:27:48.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.986+0000 7fce86a00640 1 -- start start 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce86a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fce80145040 0x7fce8006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce86a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce80071a20 0x7fce800cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce86a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8006e2f0 con 0x7fce8006ef10 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce86a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce800a86c0 con 0x7fce80071a20 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce86a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8015bf10 con 0x7fce80145040 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce84a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce80071a20 0x7fce800cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce84a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce80071a20 0x7fce800cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40674/0 (socket says 172.21.15.67:40674) 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce84a00640 1 -- 172.21.15.67:0/1493012172 learned_addr learned my addr 172.21.15.67:0/1493012172 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce7fe00640 1 --2- 172.21.15.67:0/1493012172 >> v2:172.21.15.86:3300/0 conn(0x7fce80145040 0x7fce8006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce84a00640 1 -- 172.21.15.67:0/1493012172 >> v2:172.21.15.86:3300/0 conn(0x7fce80145040 msgr2=0x7fce8006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce84a00640 1 --2- 172.21.15.67:0/1493012172 >> v2:172.21.15.86:3300/0 conn(0x7fce80145040 0x7fce8006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce84a00640 1 -- 172.21.15.67:0/1493012172 >> v1:172.21.15.86:6789/0 conn(0x7fce8006ef10 legacy=0x7fce80071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.990+0000 7fce84a00640 1 -- 172.21.15.67:0/1493012172 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8006d6e0 con 0x7fce80071a20 2024-04-17T10:27:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce84a00640 1 --2- 172.21.15.67:0/1493012172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce80071a20 0x7fce800cc300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fce80072410 tx=0x7fce7801f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eeac15e70d19ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce7ea00640 1 -- 172.21.15.67:0/1493012172 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780b4070 con 0x7fce80071a20 2024-04-17T10:27:48.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce7ea00640 1 -- 172.21.15.67:0/1493012172 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce780af400 con 0x7fce80071a20 2024-04-17T10:27:48.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce7ea00640 1 -- 172.21.15.67:0/1493012172 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce780af7d0 con 0x7fce80071a20 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 -- 172.21.15.67:0/1493012172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce80071a20 msgr2=0x7fce800cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 --2- 172.21.15.67:0/1493012172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce80071a20 0x7fce800cc300 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fce80072410 tx=0x7fce7801f1a0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 -- 172.21.15.67:0/1493012172 shutdown_connections 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 --2- 172.21.15.67:0/1493012172 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce80071a20 0x7fce800cc300 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 --2- 172.21.15.67:0/1493012172 >> v2:172.21.15.86:3300/0 conn(0x7fce80145040 0x7fce8006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 -- 172.21.15.67:0/1493012172 >> 172.21.15.67:0/1493012172 conn(0x7fce8006ac90 msgr2=0x7fce8006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 -- 172.21.15.67:0/1493012172 shutdown_connections 2024-04-17T10:27:48.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 -- 172.21.15.67:0/1493012172 wait complete. 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 Processor -- start 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.994+0000 7fce86a00640 1 -- start start 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce8006ef10 0x7fce800b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80071a20 0x7fce8015a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce80145040 0x7fce8015e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8006df80 con 0x7fce8006ef10 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8006dce0 con 0x7fce80145040 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8006d370 con 0x7fce80071a20 2024-04-17T10:27:48.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce8006ef10 0x7fce800b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce8006ef10 0x7fce800b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40678/0 (socket says 172.21.15.67:40678) 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 -- 172.21.15.67:0/1804833533 learned_addr learned my addr 172.21.15.67:0/1804833533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce84a00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce80145040 0x7fce8015e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7f400640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80071a20 0x7fce8015a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 -- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80071a20 msgr2=0x7fce8015a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80071a20 0x7fce8015a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 -- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce80145040 msgr2=0x7fce8015e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce80145040 0x7fce8015e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 -- 172.21.15.67:0/1804833533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce800b2e00 con 0x7fce8006ef10 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce7fe00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce8006ef10 0x7fce800b28c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fce7000d620 tx=0x7fce7000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce77e00640 1 -- 172.21.15.67:0/1804833533 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7000b620 con 0x7fce8006ef10 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce77e00640 1 -- 172.21.15.67:0/1804833533 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce70019070 con 0x7fce8006ef10 2024-04-17T10:27:48.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce77e00640 1 -- 172.21.15.67:0/1804833533 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7000b9e0 con 0x7fce8006ef10 2024-04-17T10:27:48.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce801391a0 con 0x7fce8006ef10 2024-04-17T10:27:48.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce80139760 con 0x7fce8006ef10 2024-04-17T10:27:48.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce80072d50 con 0x7fce8006ef10 2024-04-17T10:27:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:48.998+0000 7fce77e00640 1 -- 172.21.15.67:0/1804833533 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fce7000bb80 con 0x7fce8006ef10 2024-04-17T10:27:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.002+0000 7fce77e00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce60066f60 0x7fce60069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.002+0000 7fce77e00640 1 -- 172.21.15.67:0/1804833533 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7fce700855a0 con 0x7fce8006ef10 2024-04-17T10:27:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.002+0000 7fce7f400640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce60066f60 0x7fce60069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.002+0000 7fce7f400640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce60066f60 0x7fce60069420 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fce800aec90 tx=0x7fce6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.006+0000 7fce77e00640 1 -- 172.21.15.67:0/1804833533 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fce700563e0 con 0x7fce8006ef10 2024-04-17T10:27:49.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce80074e30 con 0x7fce8006ef10 2024-04-17T10:27:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce77e00640 1 -- 172.21.15.67:0/1804833533 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4938 (secure 0 0 0) 0x7fce70055ce0 con 0x7fce8006ef10 2024-04-17T10:27:49.196 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:49.196 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1835906721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1835906721},{"type":"v1","addr":"172.21.15.86:6836","nonce":1835906721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2085506860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2085506860},{"type":"v1","addr":"172.21.15.86:6837","nonce":2085506860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2728147711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2728147711},{"type":"v1","addr":"172.21.15.67:6834","nonce":2728147711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:26:15.408252+0000","modified":"2024-04-17T10:26:23.414956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1409597842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1409597842},{"type":"v1","addr":"172.21.15.67:6835","nonce":1409597842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-17T10:27:49.196 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-04-17T10:27:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce60066f60 msgr2=0x7fce60069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce60066f60 0x7fce60069420 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fce800aec90 tx=0x7fce6c009040 comp rx=0 tx=0).stop 2024-04-17T10:27:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce8006ef10 msgr2=0x7fce800b28c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce8006ef10 0x7fce800b28c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fce7000d620 tx=0x7fce7000dae0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 shutdown_connections 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce60066f60 0x7fce60069420 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce80145040 0x7fce8015e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce80071a20 0x7fce8015a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 --2- 172.21.15.67:0/1804833533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce8006ef10 0x7fce800b28c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 >> 172.21.15.67:0/1804833533 conn(0x7fce8006ac90 msgr2=0x7fce8006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 shutdown_connections 2024-04-17T10:27:49.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.202+0000 7fce86a00640 1 -- 172.21.15.67:0/1804833533 wait complete. 2024-04-17T10:27:49.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:27:49.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.362+0000 7f5580a00640 1 Processor -- start 2024-04-17T10:27:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f5580a00640 1 -- start start 2024-04-17T10:27:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f5580a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f557c071d60 0x7f557c06df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f5580a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c0ce1b0 con 0x7f557c0aefa0 2024-04-17T10:27:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f5580a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f557c0ce490 con 0x7f557c15c9d0 2024-04-17T10:27:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f5580a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f557c06dac0 con 0x7f557c071d60 2024-04-17T10:27:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f557a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f557c0aefa0 0x7f557c0cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43240/0 (socket says 172.21.15.67:43240) 2024-04-17T10:27:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f557a800640 1 -- 172.21.15.67:0/2263048374 learned_addr learned my addr 172.21.15.67:0/2263048374 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f557b200640 1 --2- 172.21.15.67:0/2263048374 >> v2:172.21.15.86:3300/0 conn(0x7f557c071d60 0x7f557c06df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f5579400640 1 -- 172.21.15.67:0/2263048374 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187136244 0 0) 0x7f557c0ce1b0 con 0x7f557c0aefa0 2024-04-17T10:27:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f5579400640 1 -- 172.21.15.67:0/2263048374 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5560003530 con 0x7f557c0aefa0 2024-04-17T10:27:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f557b200640 1 -- 172.21.15.67:0/2263048374 >> v1:172.21.15.86:6789/0 conn(0x7f557c15c9d0 legacy=0x7f557c071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f557b200640 1 -- 172.21.15.67:0/2263048374 >> v1:172.21.15.67:6789/0 conn(0x7f557c0aefa0 legacy=0x7f557c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.366+0000 7f557b200640 1 -- 172.21.15.67:0/2263048374 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c06d470 con 0x7f557c071d60 2024-04-17T10:27:49.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f557b200640 1 --2- 172.21.15.67:0/2263048374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c071d60 0x7f557c06df50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f557c06dc40 tx=0x7f557401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6301dd7612df13e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5579400640 1 -- 172.21.15.67:0/2263048374 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740b4070 con 0x7f557c071d60 2024-04-17T10:27:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5579400640 1 -- 172.21.15.67:0/2263048374 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55740af600 con 0x7f557c071d60 2024-04-17T10:27:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5579400640 1 -- 172.21.15.67:0/2263048374 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55740af9d0 con 0x7f557c071d60 2024-04-17T10:27:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 -- 172.21.15.67:0/2263048374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c071d60 msgr2=0x7f557c06df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 --2- 172.21.15.67:0/2263048374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c071d60 0x7f557c06df50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f557c06dc40 tx=0x7f557401f130 comp rx=0 tx=0).stop 2024-04-17T10:27:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 -- 172.21.15.67:0/2263048374 shutdown_connections 2024-04-17T10:27:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 --2- 172.21.15.67:0/2263048374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c071d60 0x7f557c06df50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 -- 172.21.15.67:0/2263048374 >> 172.21.15.67:0/2263048374 conn(0x7f557c06ac90 msgr2=0x7f557c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 -- 172.21.15.67:0/2263048374 shutdown_connections 2024-04-17T10:27:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 -- 172.21.15.67:0/2263048374 wait complete. 2024-04-17T10:27:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 Processor -- start 2024-04-17T10:27:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.370+0000 7f5580a00640 1 -- start start 2024-04-17T10:27:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f557c071d60 0x7f557c0b4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557c0aefa0 0x7f557c1462a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c15c9d0 0x7f557c158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c15d020 con 0x7f557c0aefa0 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557c15cea0 con 0x7f557c15c9d0 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f557c15d1a0 con 0x7f557c071d60 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557c0aefa0 0x7f557c1462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557c0aefa0 0x7f557c1462a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40680/0 (socket says 172.21.15.67:40680) 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 -- 172.21.15.67:0/3247854467 learned_addr learned my addr 172.21.15.67:0/3247854467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f557b200640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c15c9d0 0x7f557c158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f557a800640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f557c071d60 0x7f557c0b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 -- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f557c071d60 msgr2=0x7f557c0b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f557c071d60 0x7f557c0b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 -- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c15c9d0 msgr2=0x7f557c158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c15c9d0 0x7f557c158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 -- 172.21.15.67:0/3247854467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c1f42c0 con 0x7f557c0aefa0 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f557b200640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c15c9d0 0x7f557c158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f557a800640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f557c071d60 0x7f557c0b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5579e00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557c0aefa0 0x7f557c1462a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f556800c370 tx=0x7f556800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5572a00640 1 -- 172.21.15.67:0/3247854467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680052d0 con 0x7f557c0aefa0 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5572a00640 1 -- 172.21.15.67:0/3247854467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5568005470 con 0x7f557c0aefa0 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5572a00640 1 -- 172.21.15.67:0/3247854467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568007260 con 0x7f557c0aefa0 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557c0b3b90 con 0x7f557c0aefa0 2024-04-17T10:27:49.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557c0b4150 con 0x7f557c0aefa0 2024-04-17T10:27:49.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.374+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f557c0b4bb0 con 0x7f557c0aefa0 2024-04-17T10:27:49.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.378+0000 7f5572a00640 1 -- 172.21.15.67:0/3247854467 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5568007400 con 0x7f557c0aefa0 2024-04-17T10:27:49.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.378+0000 7f5572a00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f555c066f60 0x7f555c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.378+0000 7f5572a00640 1 -- 172.21.15.67:0/3247854467 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f5568084660 con 0x7f557c0aefa0 2024-04-17T10:27:49.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.378+0000 7f557a800640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f555c066f60 0x7f555c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.378+0000 7f557a800640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f555c066f60 0x7f555c069420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f556c0011f0 tx=0x7f556c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:49.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.382+0000 7f5572a00640 1 -- 172.21.15.67:0/3247854467 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f55680554a0 con 0x7f557c0aefa0 2024-04-17T10:27:49.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.578+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f557c0ae630 con 0x7f557c0aefa0 2024-04-17T10:27:49.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5572a00640 1 -- 172.21.15.67:0/3247854467 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+4938 (secure 0 0 0) 0x7f5568054da0 con 0x7f557c0aefa0 2024-04-17T10:27:49.575 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:49.575 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1835906721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1835906721},{"type":"v1","addr":"172.21.15.86:6836","nonce":1835906721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2085506860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2085506860},{"type":"v1","addr":"172.21.15.86:6837","nonce":2085506860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2728147711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2728147711},{"type":"v1","addr":"172.21.15.67:6834","nonce":2728147711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:26:15.408252+0000","modified":"2024-04-17T10:26:23.414956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5283},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5283":{"gid":5283,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/1409597842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1409597842},{"type":"v1","addr":"172.21.15.67:6835","nonce":1409597842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-04-17T10:27:49.575 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 19 2024-04-17T10:27:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f555c066f60 msgr2=0x7f555c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f555c066f60 0x7f555c069420 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f556c0011f0 tx=0x7f556c009040 comp rx=0 tx=0).stop 2024-04-17T10:27:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557c0aefa0 msgr2=0x7f557c1462a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557c0aefa0 0x7f557c1462a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f556800c370 tx=0x7f556800c830 comp rx=0 tx=0).stop 2024-04-17T10:27:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 shutdown_connections 2024-04-17T10:27:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f555c066f60 0x7f555c069420 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f557c15c9d0 0x7f557c158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f557c0aefa0 0x7f557c1462a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 --2- 172.21.15.67:0/3247854467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f557c071d60 0x7f557c0b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 >> 172.21.15.67:0/3247854467 conn(0x7f557c06ac90 msgr2=0x7f557c15bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:49.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 shutdown_connections 2024-04-17T10:27:49.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.582+0000 7f5580a00640 1 -- 172.21.15.67:0/3247854467 wait complete. 2024-04-17T10:27:49.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:27:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.742+0000 7f0101a00640 1 Processor -- start 2024-04-17T10:27:49.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.742+0000 7f0101a00640 1 -- start start 2024-04-17T10:27:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f0101a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f0101a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00fc15e550 0x7f00fc0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f0101a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00fc15eb50 con 0x7f00fc0ce020 2024-04-17T10:27:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f0101a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc161650 con 0x7f00fc15e550 2024-04-17T10:27:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f0101a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc15ee30 con 0x7f00fc1620a0 2024-04-17T10:27:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00fb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f00fc15e550 0x7f00fc0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00faa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f00fc0ce020 0x7f00fc0ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43242/0 (socket says 172.21.15.67:43242) 2024-04-17T10:27:49.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00faa00640 1 -- 172.21.15.67:0/681814785 learned_addr learned my addr 172.21.15.67:0/681814785 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:49.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00fb400640 1 --2- 172.21.15.67:0/681814785 >> v2:172.21.15.67:3300/0 conn(0x7f00fc15e550 0x7f00fc0cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40686/0 (socket says 172.21.15.67:40686) 2024-04-17T10:27:49.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00fb400640 1 -- 172.21.15.67:0/681814785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 msgr2=0x7f00fc15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:49.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00f3e00640 1 --2- 172.21.15.67:0/681814785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00fb400640 1 --2- 172.21.15.67:0/681814785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00fb400640 1 -- 172.21.15.67:0/681814785 >> v1:172.21.15.67:6789/0 conn(0x7f00fc0ce020 legacy=0x7f00fc0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00fb400640 1 -- 172.21.15.67:0/681814785 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc1618a0 con 0x7f00fc15e550 2024-04-17T10:27:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00fa000640 1 -- 172.21.15.67:0/681814785 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404681776 0 0) 0x7f00fc15eb50 con 0x7f00fc0ce020 2024-04-17T10:27:49.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.746+0000 7f00f3e00640 1 --2- 172.21.15.67:0/681814785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:49.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f00fb400640 1 --2- 172.21.15.67:0/681814785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc0cf510 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f00f4009d90 tx=0x7f00f401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f3fea0bbd71b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:49.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f00fa000640 1 -- 172.21.15.67:0/681814785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f40b2070 con 0x7f00fc15e550 2024-04-17T10:27:49.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f00fa000640 1 -- 172.21.15.67:0/681814785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f40a5910 con 0x7f00fc15e550 2024-04-17T10:27:49.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f00fa000640 1 -- 172.21.15.67:0/681814785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f40a5ce0 con 0x7f00fc15e550 2024-04-17T10:27:49.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- 172.21.15.67:0/681814785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 msgr2=0x7f00fc0cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 --2- 172.21.15.67:0/681814785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc0cf510 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f00f4009d90 tx=0x7f00f401c940 comp rx=0 tx=0).stop 2024-04-17T10:27:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- 172.21.15.67:0/681814785 shutdown_connections 2024-04-17T10:27:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 --2- 172.21.15.67:0/681814785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc0cf510 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 --2- 172.21.15.67:0/681814785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- 172.21.15.67:0/681814785 >> 172.21.15.67:0/681814785 conn(0x7f00fc0b33f0 msgr2=0x7f00fc0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- 172.21.15.67:0/681814785 shutdown_connections 2024-04-17T10:27:49.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- 172.21.15.67:0/681814785 wait complete. 2024-04-17T10:27:49.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 Processor -- start 2024-04-17T10:27:49.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- start start 2024-04-17T10:27:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00fc0ce020 0x7f00fc1546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc15a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc14f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c8630 con 0x7f00fc15e550 2024-04-17T10:27:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc0c84b0 con 0x7f00fc0ce020 2024-04-17T10:27:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.750+0000 7f0101a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc16a9f0 con 0x7f00fc1620a0 2024-04-17T10:27:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc15a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc15a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40694/0 (socket says 172.21.15.67:40694) 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 -- 172.21.15.67:0/1066900827 learned_addr learned my addr 172.21.15.67:0/1066900827 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00faa00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00fc0ce020 0x7f00fc1546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00fb400640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc14f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 -- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 msgr2=0x7f00fc14f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc14f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 -- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00fc0ce020 msgr2=0x7f00fc1546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00fc0ce020 0x7f00fc1546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 -- 172.21.15.67:0/1066900827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc15a6d0 con 0x7f00fc15e550 2024-04-17T10:27:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f3e00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc15a190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f00e400c7f0 tx=0x7f00e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f2a00640 1 -- 172.21.15.67:0/1066900827 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4015070 con 0x7f00fc15e550 2024-04-17T10:27:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f2a00640 1 -- 172.21.15.67:0/1066900827 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e4007820 con 0x7f00fc15e550 2024-04-17T10:27:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc154ca0 con 0x7f00fc15e550 2024-04-17T10:27:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f2a00640 1 -- 172.21.15.67:0/1066900827 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e4007c10 con 0x7f00fc15e550 2024-04-17T10:27:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc155260 con 0x7f00fc15e550 2024-04-17T10:27:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc1554a0 con 0x7f00fc15e550 2024-04-17T10:27:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f2a00640 1 -- 172.21.15.67:0/1066900827 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f00e4018070 con 0x7f00fc15e550 2024-04-17T10:27:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.754+0000 7f00f2a00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00d8066f60 0x7f00d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.758+0000 7f00f2a00640 1 -- 172.21.15.67:0/1066900827 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f00e4085380 con 0x7f00fc15e550 2024-04-17T10:27:49.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.758+0000 7f00faa00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00d8066f60 0x7f00d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:49.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.758+0000 7f00faa00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00d8066f60 0x7f00d8069420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f00ec007fd0 tx=0x7f00ec007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:49.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.762+0000 7f00f2a00640 1 -- 172.21.15.67:0/1066900827 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f00fc1554a0 con 0x7f00fc15e550 2024-04-17T10:27:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.922+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00fc074e30 con 0x7f00fc15e550 2024-04-17T10:27:49.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.922+0000 7f00f2a00640 1 -- 172.21.15.67:0/1066900827 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+17008 (secure 0 0 0) 0x7f00e4055a50 con 0x7f00fc15e550 2024-04-17T10:27:49.919 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:49.919 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":43,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:27:28.880541+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:26:13.215866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-17T10:26:14.398708+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":42,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.37","mappings":[{"from":5,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":6,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00d8066f60 msgr2=0x7f00d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00d8066f60 0x7f00d8069420 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f00ec007fd0 tx=0x7f00ec007d60 comp rx=0 tx=0).stop 2024-04-17T10:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 msgr2=0x7f00fc15a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc15a190 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f00e400c7f0 tx=0x7f00e400ccb0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 shutdown_connections 2024-04-17T10:27:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00d8066f60 0x7f00d8069420 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f00fc1620a0 0x7f00fc14f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00fc15e550 0x7f00fc15a190 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 --2- 172.21.15.67:0/1066900827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f00fc0ce020 0x7f00fc1546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 >> 172.21.15.67:0/1066900827 conn(0x7f00fc0b33f0 msgr2=0x7f00fc160500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 shutdown_connections 2024-04-17T10:27:49.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:49.926+0000 7f0101a00640 1 -- 172.21.15.67:0/1066900827 wait complete. 2024-04-17T10:27:49.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:27:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.086+0000 7f0b60800640 1 Processor -- start 2024-04-17T10:27:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.086+0000 7f0b60800640 1 -- start start 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b60800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b58145040 0x7f0b5806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b60800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b580cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b60800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5806e2f0 con 0x7f0b5806ef10 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b60800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b580a86c0 con 0x7f0b58145040 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b60800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5815bf10 con 0x7f0b58071a20 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b58145040 0x7f0b5806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0b58145040 0x7f0b5806e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40706/0 (socket says 172.21.15.67:40706) 2024-04-17T10:27:50.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 -- 172.21.15.67:0/1283506096 learned_addr learned my addr 172.21.15.67:0/1283506096 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:50.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5e800640 1 --2- 172.21.15.67:0/1283506096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b580cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 -- 172.21.15.67:0/1283506096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 msgr2=0x7f0b580cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:50.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 --2- 172.21.15.67:0/1283506096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b580cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 -- 172.21.15.67:0/1283506096 >> v1:172.21.15.86:6789/0 conn(0x7f0b5806ef10 legacy=0x7f0b58071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:50.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 -- 172.21.15.67:0/1283506096 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5806d6e0 con 0x7f0b58145040 2024-04-17T10:27:50.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.090+0000 7f0b5de00640 1 --2- 172.21.15.67:0/1283506096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5806e9d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009870 tx=0x7f0b4c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb212f5472d3af3f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:50.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5ca00640 1 -- 172.21.15.67:0/1283506096 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c0b5070 con 0x7f0b58145040 2024-04-17T10:27:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5ca00640 1 -- 172.21.15.67:0/1283506096 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c0a89d0 con 0x7f0b58145040 2024-04-17T10:27:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5ca00640 1 -- 172.21.15.67:0/1283506096 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c0a8da0 con 0x7f0b58145040 2024-04-17T10:27:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- 172.21.15.67:0/1283506096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 msgr2=0x7f0b5806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 --2- 172.21.15.67:0/1283506096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5806e9d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009870 tx=0x7f0b4c01ed10 comp rx=0 tx=0).stop 2024-04-17T10:27:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- 172.21.15.67:0/1283506096 shutdown_connections 2024-04-17T10:27:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 --2- 172.21.15.67:0/1283506096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b580cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 --2- 172.21.15.67:0/1283506096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5806e9d0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- 172.21.15.67:0/1283506096 >> 172.21.15.67:0/1283506096 conn(0x7f0b5806ac90 msgr2=0x7f0b5806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- 172.21.15.67:0/1283506096 shutdown_connections 2024-04-17T10:27:50.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- 172.21.15.67:0/1283506096 wait complete. 2024-04-17T10:27:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 Processor -- start 2024-04-17T10:27:50.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- start start 2024-04-17T10:27:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5806ef10 0x7f0b5813bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b5813e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5813edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5806df80 con 0x7f0b58145040 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5806dca0 con 0x7f0b5806ef10 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b60800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5806d370 con 0x7f0b58071a20 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5813edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5813edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40718/0 (socket says 172.21.15.67:40718) 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5e800640 1 -- 172.21.15.67:0/1350541175 learned_addr learned my addr 172.21.15.67:0/1350541175 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5d400640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b5813e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.094+0000 7f0b5de00640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5806ef10 0x7f0b5813bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b5e800640 1 -- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 msgr2=0x7f0b5813e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b5e800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b5813e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b5e800640 1 -- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5806ef10 msgr2=0x7f0b5813bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b5e800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5806ef10 0x7f0b5813bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b5e800640 1 -- 172.21.15.67:0/1350541175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5813c430 con 0x7f0b58145040 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b5e800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5813edc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007e30 tx=0x7f0b5400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:50.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b52000640 1 -- 172.21.15.67:0/1350541175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b54014620 con 0x7f0b58145040 2024-04-17T10:27:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b52000640 1 -- 172.21.15.67:0/1350541175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b54019070 con 0x7f0b58145040 2024-04-17T10:27:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b52000640 1 -- 172.21.15.67:0/1350541175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b54014a00 con 0x7f0b58145040 2024-04-17T10:27:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5813c710 con 0x7f0b58145040 2024-04-17T10:27:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5813ccd0 con 0x7f0b58145040 2024-04-17T10:27:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b58072d50 con 0x7f0b58145040 2024-04-17T10:27:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b52000640 1 -- 172.21.15.67:0/1350541175 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0b54014ba0 con 0x7f0b58145040 2024-04-17T10:27:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b52000640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b3c066f60 0x7f0b3c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.098+0000 7f0b52000640 1 -- 172.21.15.67:0/1350541175 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6158+0+0 (secure 0 0 0) 0x7f0b540858c0 con 0x7f0b58145040 2024-04-17T10:27:50.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.102+0000 7f0b5de00640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b3c066f60 0x7f0b3c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.102+0000 7f0b5de00640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b3c066f60 0x7f0b3c069420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0b2000 tx=0x7f0b4c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:50.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.102+0000 7f0b52000640 1 -- 172.21.15.67:0/1350541175 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0b54056150 con 0x7f0b58145040 2024-04-17T10:27:50.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.302+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0b58145500 con 0x7f0b58145040 2024-04-17T10:27:50.297 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:27:50.302+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:27:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.306+0000 7f0b52000640 1 -- 172.21.15.67:0/1350541175 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v20) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0b5405a1f0 con 0x7f0b58145040 2024-04-17T10:27:50.300 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b3c066f60 msgr2=0x7f0b3c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b3c066f60 0x7f0b3c069420 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0b2000 tx=0x7f0b4c009040 comp rx=0 tx=0).stop 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 msgr2=0x7f0b5813edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5813edc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0b54007e30 tx=0x7f0b5400e4b0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 shutdown_connections 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b3c066f60 0x7f0b3c069420 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b58145040 0x7f0b5813edc0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b58071a20 0x7f0b5813e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 --2- 172.21.15.67:0/1350541175 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b5806ef10 0x7f0b5813bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 >> 172.21.15.67:0/1350541175 conn(0x7f0b5806ac90 msgr2=0x7f0b5815f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 shutdown_connections 2024-04-17T10:27:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.310+0000 7f0b60800640 1 -- 172.21.15.67:0/1350541175 wait complete. 2024-04-17T10:27:50.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:27:50.377 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:27:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.478+0000 7f8e9fe00640 1 Processor -- start 2024-04-17T10:27:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.478+0000 7f8e9fe00640 1 -- start start 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980aef60 0x7f8e980cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e980cce60 0x7f8e9806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9fe00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e980b4990 con 0x7f8e9806e7b0 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e980cd2c0 con 0x7f8e980cce60 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e980af340 con 0x7f8e980aef60 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e980cce60 0x7f8e9806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e980cce60 0x7f8e9806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37128/0 (socket says 172.21.15.67:37128) 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9ca00640 1 -- 172.21.15.67:0/3014751755 learned_addr learned my addr 172.21.15.67:0/3014751755 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:50.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9d400640 1 --2- 172.21.15.67:0/3014751755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980aef60 0x7f8e980cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9ca00640 1 -- 172.21.15.67:0/3014751755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980aef60 msgr2=0x7f8e980cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9ca00640 1 --2- 172.21.15.67:0/3014751755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980aef60 0x7f8e980cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9ca00640 1 -- 172.21.15.67:0/3014751755 >> v1:172.21.15.86:6789/0 conn(0x7f8e9806e7b0 legacy=0x7f8e980713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.482+0000 7f8e9ca00640 1 -- 172.21.15.67:0/3014751755 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e9815ae90 con 0x7f8e980cce60 2024-04-17T10:27:50.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9ca00640 1 --2- 172.21.15.67:0/3014751755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980cce60 0x7f8e9806e270 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400bb10 tx=0x7f8e8401eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b486c437a55fe4e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e97e00640 1 -- 172.21.15.67:0/3014751755 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e840b4070 con 0x7f8e980cce60 2024-04-17T10:27:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e97e00640 1 -- 172.21.15.67:0/3014751755 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e840a7990 con 0x7f8e980cce60 2024-04-17T10:27:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e97e00640 1 -- 172.21.15.67:0/3014751755 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e840a7d60 con 0x7f8e980cce60 2024-04-17T10:27:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 -- 172.21.15.67:0/3014751755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980cce60 msgr2=0x7f8e9806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/3014751755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980cce60 0x7f8e9806e270 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400bb10 tx=0x7f8e8401eba0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 -- 172.21.15.67:0/3014751755 shutdown_connections 2024-04-17T10:27:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/3014751755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980cce60 0x7f8e9806e270 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/3014751755 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980aef60 0x7f8e980cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 -- 172.21.15.67:0/3014751755 >> 172.21.15.67:0/3014751755 conn(0x7f8e980b1740 msgr2=0x7f8e980b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 -- 172.21.15.67:0/3014751755 shutdown_connections 2024-04-17T10:27:50.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 -- 172.21.15.67:0/3014751755 wait complete. 2024-04-17T10:27:50.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 Processor -- start 2024-04-17T10:27:50.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.486+0000 7f8e9fe00640 1 -- start start 2024-04-17T10:27:50.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9806e7b0 0x7f8e98153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980aef60 0x7f8e9806e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980cce60 0x7f8e9813cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9815bc10 con 0x7f8e980aef60 2024-04-17T10:27:50.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9815b410 con 0x7f8e9806e7b0 2024-04-17T10:27:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e9806e5a0 con 0x7f8e980cce60 2024-04-17T10:27:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980aef60 0x7f8e9806e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980aef60 0x7f8e9806e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37136/0 (socket says 172.21.15.67:37136) 2024-04-17T10:27:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 -- 172.21.15.67:0/182818976 learned_addr learned my addr 172.21.15.67:0/182818976 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9d400640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9806e7b0 0x7f8e98153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9de00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980cce60 0x7f8e9813cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 -- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980cce60 msgr2=0x7f8e9813cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980cce60 0x7f8e9813cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 -- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9806e7b0 msgr2=0x7f8e98153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9806e7b0 0x7f8e98153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 -- 172.21.15.67:0/182818976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e981542a0 con 0x7f8e980aef60 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9d400640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9806e7b0 0x7f8e98153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9ca00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980aef60 0x7f8e9806e060 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400b2c0 tx=0x7f8e8401b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e95600640 1 -- 172.21.15.67:0/182818976 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e840b4070 con 0x7f8e980aef60 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e95600640 1 -- 172.21.15.67:0/182818976 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e840b6070 con 0x7f8e980aef60 2024-04-17T10:27:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e95600640 1 -- 172.21.15.67:0/182818976 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e840bb470 con 0x7f8e980aef60 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e98154580 con 0x7f8e980aef60 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e98154b40 con 0x7f8e980aef60 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e9804ef30 con 0x7f8e980aef60 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.490+0000 7f8e95600640 1 -- 172.21.15.67:0/182818976 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8e840bb610 con 0x7f8e980aef60 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.494+0000 7f8e95600640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e78066f90 0x7f8e78069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.494+0000 7f8e95600640 1 -- 172.21.15.67:0/182818976 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6216+0+0 (secure 0 0 0) 0x7f8e8411f7f0 con 0x7f8e980aef60 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.494+0000 7f8e9d400640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e78066f90 0x7f8e78069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:50.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.494+0000 7f8e9d400640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e78066f90 0x7f8e78069450 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009eb0 tx=0x7f8e8c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:50.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.498+0000 7f8e95600640 1 -- 172.21.15.67:0/182818976 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8e84018820 con 0x7f8e980aef60 2024-04-17T10:27:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:50.694+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8e9814edf0 con 0x7f8e980aef60 2024-04-17T10:27:51.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.310+0000 7f8e95600640 1 -- 172.21.15.67:0/182818976 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8e840eff10 con 0x7f8e980aef60 2024-04-17T10:27:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e78066f90 msgr2=0x7f8e78069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e78066f90 0x7f8e78069450 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8e8c009eb0 tx=0x7f8e8c009190 comp rx=0 tx=0).stop 2024-04-17T10:27:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980aef60 msgr2=0x7f8e9806e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980aef60 0x7f8e9806e060 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f8e8400b2c0 tx=0x7f8e8401b940 comp rx=0 tx=0).stop 2024-04-17T10:27:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 shutdown_connections 2024-04-17T10:27:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e78066f90 0x7f8e78069450 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e980cce60 0x7f8e9813cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e980aef60 0x7f8e9806e060 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 --2- 172.21.15.67:0/182818976 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e9806e7b0 0x7f8e98153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 >> 172.21.15.67:0/182818976 conn(0x7f8e980b1740 msgr2=0x7f8e980713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 shutdown_connections 2024-04-17T10:27:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.314+0000 7f8e9fe00640 1 -- 172.21.15.67:0/182818976 wait complete. 2024-04-17T10:27:51.331 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:27:51.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f4000640 1 Processor -- start 2024-04-17T10:27:51.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f4000640 1 -- start start 2024-04-17T10:27:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f4000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0ec071a70 0x7fe0ec1665d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f4000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec06d9a0 con 0x7fe0ec1447e0 2024-04-17T10:27:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f4000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec06dc80 con 0x7fe0ec06d3c0 2024-04-17T10:27:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f4000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec0cc310 con 0x7fe0ec071a70 2024-04-17T10:27:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f1600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe0ec1447e0 0x7fe0ec144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43870/0 (socket says 172.21.15.67:43870) 2024-04-17T10:27:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f1600640 1 -- 172.21.15.67:0/3514134610 learned_addr learned my addr 172.21.15.67:0/3514134610 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:51.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.534+0000 7fe0f2000640 1 --2- 172.21.15.67:0/3514134610 >> v2:172.21.15.86:3300/0 conn(0x7fe0ec071a70 0x7fe0ec1665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3514134610 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189197503 0 0) 0x7fe0ec06d9a0 con 0x7fe0ec1447e0 2024-04-17T10:27:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3514134610 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4003530 con 0x7fe0ec1447e0 2024-04-17T10:27:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0f2000640 1 -- 172.21.15.67:0/3514134610 >> v1:172.21.15.86:6789/0 conn(0x7fe0ec06d3c0 legacy=0x7fe0ec071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0f2000640 1 -- 172.21.15.67:0/3514134610 >> v1:172.21.15.67:6789/0 conn(0x7fe0ec1447e0 legacy=0x7fe0ec144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3514134610 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 888594574 0 0) 0x7fe0d4003530 con 0x7fe0ec1447e0 2024-04-17T10:27:51.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0f2000640 1 -- 172.21.15.67:0/3514134610 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec06df60 con 0x7fe0ec071a70 2024-04-17T10:27:51.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0f2000640 1 --2- 172.21.15.67:0/3514134610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 0x7fe0ec1665d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009d90 tx=0x7fe0e401c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29280b0e633c4b35 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:51.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3514134610 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40b2070 con 0x7fe0ec071a70 2024-04-17T10:27:51.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3514134610 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e40a5520 con 0x7fe0ec071a70 2024-04-17T10:27:51.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.538+0000 7fe0ebe00640 1 -- 172.21.15.67:0/3514134610 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0001680 con 0x7fe0ec071a70 2024-04-17T10:27:51.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- 172.21.15.67:0/3514134610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 msgr2=0x7fe0ec1665d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:51.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 --2- 172.21.15.67:0/3514134610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 0x7fe0ec1665d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009d90 tx=0x7fe0e401c7d0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- 172.21.15.67:0/3514134610 shutdown_connections 2024-04-17T10:27:51.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 --2- 172.21.15.67:0/3514134610 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 0x7fe0ec1665d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- 172.21.15.67:0/3514134610 >> 172.21.15.67:0/3514134610 conn(0x7fe0ec06ac90 msgr2=0x7fe0ec06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:51.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- 172.21.15.67:0/3514134610 shutdown_connections 2024-04-17T10:27:51.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- 172.21.15.67:0/3514134610 wait complete. 2024-04-17T10:27:51.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 Processor -- start 2024-04-17T10:27:51.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- start start 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0ec06d3c0 0x7fe0ec13af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 0x7fe0ec139f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec1447e0 0x7fe0ec14cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec169ad0 con 0x7fe0ec1447e0 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec169950 con 0x7fe0ec071a70 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f4000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec169c50 con 0x7fe0ec06d3c0 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec1447e0 0x7fe0ec14cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:51.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec1447e0 0x7fe0ec14cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37152/0 (socket says 172.21.15.67:37152) 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f2000640 1 -- 172.21.15.67:0/2011183249 learned_addr learned my addr 172.21.15.67:0/2011183249 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f1600640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0ec06d3c0 0x7fe0ec13af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.542+0000 7fe0f0c00640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 0x7fe0ec139f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f2000640 1 -- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0ec06d3c0 msgr2=0x7fe0ec13af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f2000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0ec06d3c0 0x7fe0ec13af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f2000640 1 -- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 msgr2=0x7fe0ec139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f2000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 0x7fe0ec139f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f2000640 1 -- 172.21.15.67:0/2011183249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec140080 con 0x7fe0ec1447e0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f2000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec1447e0 0x7fe0ec14cad0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40193e0 tx=0x7fe0e4019570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0e9600640 1 -- 172.21.15.67:0/2011183249 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40bd020 con 0x7fe0ec1447e0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0e9600640 1 -- 172.21.15.67:0/2011183249 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e40a6d10 con 0x7fe0ec1447e0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0e9600640 1 -- 172.21.15.67:0/2011183249 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40b2040 con 0x7fe0ec1447e0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec140360 con 0x7fe0ec1447e0 2024-04-17T10:27:51.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec0b2730 con 0x7fe0ec1447e0 2024-04-17T10:27:51.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ec072da0 con 0x7fe0ec1447e0 2024-04-17T10:27:51.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0e9600640 1 -- 172.21.15.67:0/2011183249 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe0e40c0020 con 0x7fe0ec1447e0 2024-04-17T10:27:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0e9600640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0cc066f60 0x7fe0cc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.546+0000 7fe0e9600640 1 -- 172.21.15.67:0/2011183249 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6216+0+0 (secure 0 0 0) 0x7fe0e4120240 con 0x7fe0ec1447e0 2024-04-17T10:27:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.550+0000 7fe0f1600640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0cc066f60 0x7fe0cc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:51.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.550+0000 7fe0f1600640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0cc066f60 0x7fe0cc069420 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005890 tx=0x7fe0e0005800 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.550+0000 7fe0e9600640 1 -- 172.21.15.67:0/2011183249 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe0e40c7030 con 0x7fe0ec1447e0 2024-04-17T10:27:51.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:51.714+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe0ec074e30 con 0x7fe0ec1447e0 2024-04-17T10:27:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.318+0000 7fe0e9600640 1 -- 172.21.15.67:0/2011183249 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe0e4016480 con 0x7fe0ec1447e0 2024-04-17T10:27:52.311 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0cc066f60 msgr2=0x7fe0cc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0cc066f60 0x7fe0cc069420 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0005890 tx=0x7fe0e0005800 comp rx=0 tx=0).stop 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec1447e0 msgr2=0x7fe0ec14cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec1447e0 0x7fe0ec14cad0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40193e0 tx=0x7fe0e4019570 comp rx=0 tx=0).stop 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 shutdown_connections 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0cc066f60 0x7fe0cc069420 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0ec1447e0 0x7fe0ec14cad0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0ec071a70 0x7fe0ec139f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 --2- 172.21.15.67:0/2011183249 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0ec06d3c0 0x7fe0ec13af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 >> 172.21.15.67:0/2011183249 conn(0x7fe0ec06ac90 msgr2=0x7fe0ec144d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 shutdown_connections 2024-04-17T10:27:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.322+0000 7fe0f4000640 1 -- 172.21.15.67:0/2011183249 wait complete. 2024-04-17T10:27:52.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:27:52.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.494+0000 7fa8d1a00640 1 Processor -- start 2024-04-17T10:27:52.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.494+0000 7fa8d1a00640 1 -- start start 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.494+0000 7fa8d1a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8cc15bab0 0x7fa8cc072760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.494+0000 7fa8d1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8cc06bca0 0x7fa8cc06c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.494+0000 7fa8d1a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8cc06f450 con 0x7fa8cc15be90 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.494+0000 7fa8d1a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc06c630 con 0x7fa8cc06bca0 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.494+0000 7fa8d1a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8cc06f010 con 0x7fa8cc15bab0 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8cc06bca0 0x7fa8cc06c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8cc06bca0 0x7fa8cc06c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37160/0 (socket says 172.21.15.67:37160) 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 -- 172.21.15.67:0/2865088456 learned_addr learned my addr 172.21.15.67:0/2865088456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8caa00640 1 --2- 172.21.15.67:0/2865088456 >> v2:172.21.15.86:3300/0 conn(0x7fa8cc15bab0 0x7fa8cc072760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 -- 172.21.15.67:0/2865088456 >> v2:172.21.15.86:3300/0 conn(0x7fa8cc15bab0 msgr2=0x7fa8cc072760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 --2- 172.21.15.67:0/2865088456 >> v2:172.21.15.86:3300/0 conn(0x7fa8cc15bab0 0x7fa8cc072760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 -- 172.21.15.67:0/2865088456 >> v1:172.21.15.86:6789/0 conn(0x7fa8cc15be90 legacy=0x7fa8cc0af230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 -- 172.21.15.67:0/2865088456 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc06f730 con 0x7fa8cc06bca0 2024-04-17T10:27:52.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8cb400640 1 --2- 172.21.15.67:0/2865088456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc06bca0 0x7fa8cc06c0f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0676b0 tx=0x7fa8c401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233a6460103ecb2f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8ca000640 1 -- 172.21.15.67:0/2865088456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40a57d0 con 0x7fa8cc06bca0 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8ca000640 1 -- 172.21.15.67:0/2865088456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c40a5970 con 0x7fa8cc06bca0 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.498+0000 7fa8ca000640 1 -- 172.21.15.67:0/2865088456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c40a5d40 con 0x7fa8cc06bca0 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- 172.21.15.67:0/2865088456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc06bca0 msgr2=0x7fa8cc06c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/2865088456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc06bca0 0x7fa8cc06c0f0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0676b0 tx=0x7fa8c401c9f0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- 172.21.15.67:0/2865088456 shutdown_connections 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/2865088456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc06bca0 0x7fa8cc06c0f0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/2865088456 >> v2:172.21.15.86:3300/0 conn(0x7fa8cc15bab0 0x7fa8cc072760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- 172.21.15.67:0/2865088456 >> 172.21.15.67:0/2865088456 conn(0x7fa8cc071c40 msgr2=0x7fa8cc072050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:52.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- 172.21.15.67:0/2865088456 shutdown_connections 2024-04-17T10:27:52.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- 172.21.15.67:0/2865088456 wait complete. 2024-04-17T10:27:52.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 Processor -- start 2024-04-17T10:27:52.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- start start 2024-04-17T10:27:52.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8cc06bca0 0x7fa8cc14dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:52.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8cc15bab0 0x7fa8cc15e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:52.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc15be90 0x7fa8cc0b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:52.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc06ebe0 con 0x7fa8cc15be90 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc06ea60 con 0x7fa8cc15bab0 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8d1a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8cc06ed60 con 0x7fa8cc06bca0 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8cb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc15be90 0x7fa8cc0b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8cc15bab0 0x7fa8cc15e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8cc15bab0 0x7fa8cc15e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:50340/0 (socket says 172.21.15.67:50340) 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 -- 172.21.15.67:0/1387992369 learned_addr learned my addr 172.21.15.67:0/1387992369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8caa00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8cc06bca0 0x7fa8cc14dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 -- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8cc06bca0 msgr2=0x7fa8cc14dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8cc06bca0 0x7fa8cc14dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 -- 172.21.15.67:0/1387992369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc15be90 msgr2=0x7fa8cc0b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc15be90 0x7fa8cc0b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.502+0000 7fa8c3e00640 1 -- 172.21.15.67:0/1387992369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc15d670 con 0x7fa8cc15bab0 2024-04-17T10:27:52.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8c3e00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8cc15bab0 0x7fa8cc15e3f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400c370 tx=0x7fa8b400c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8c2a00640 1 -- 172.21.15.67:0/1387992369 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b40052d0 con 0x7fa8cc15bab0 2024-04-17T10:27:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8c2a00640 1 -- 172.21.15.67:0/1387992369 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8b4005470 con 0x7fa8cc15bab0 2024-04-17T10:27:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8cc14e320 con 0x7fa8cc15bab0 2024-04-17T10:27:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8c2a00640 1 -- 172.21.15.67:0/1387992369 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4007260 con 0x7fa8cc15bab0 2024-04-17T10:27:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8cc15dbf0 con 0x7fa8cc15bab0 2024-04-17T10:27:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8cc09e1f0 con 0x7fa8cc15bab0 2024-04-17T10:27:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8c2a00640 1 -- 172.21.15.67:0/1387992369 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa8b4007400 con 0x7fa8cc15bab0 2024-04-17T10:27:52.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8c2a00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8a8066f60 0x7fa8a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:52.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.506+0000 7fa8c2a00640 1 -- 172.21.15.67:0/1387992369 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5778+0+0 (secure 0 0 0) 0x7fa8b4084d30 con 0x7fa8cc15bab0 2024-04-17T10:27:52.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.510+0000 7fa8caa00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8a8066f60 0x7fa8a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:52.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.510+0000 7fa8caa00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8a8066f60 0x7fa8a8069420 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009b60 tx=0x7fa8bc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.510+0000 7fa8c2a00640 1 -- 172.21.15.67:0/1387992369 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa8b4020070 con 0x7fa8cc15bab0 2024-04-17T10:27:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:52.674+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8cc0af950 con 0x7fa8cc15bab0 2024-04-17T10:27:53.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8c2a00640 1 -- 172.21.15.67:0/1387992369 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa8b4055600 con 0x7fa8cc15bab0 2024-04-17T10:27:53.324 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8a8066f60 msgr2=0x7fa8a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8a8066f60 0x7fa8a8069420 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc009b60 tx=0x7fa8bc009190 comp rx=0 tx=0).stop 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8cc15bab0 msgr2=0x7fa8cc15e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8cc15bab0 0x7fa8cc15e3f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400c370 tx=0x7fa8b400c830 comp rx=0 tx=0).stop 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 shutdown_connections 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8a8066f60 0x7fa8a8069420 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8cc15be90 0x7fa8cc0b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8cc15bab0 0x7fa8cc15e3f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 --2- 172.21.15.67:0/1387992369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8cc06bca0 0x7fa8cc14dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.330+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 >> 172.21.15.67:0/1387992369 conn(0x7fa8cc071c40 msgr2=0x7fa8cc0afc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.334+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 shutdown_connections 2024-04-17T10:27:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.334+0000 7fa8d1a00640 1 -- 172.21.15.67:0/1387992369 wait complete. 2024-04-17T10:27:53.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:27:53.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740c800640 1 Processor -- start 2024-04-17T10:27:53.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740c800640 1 -- start start 2024-04-17T10:27:53.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740c800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f740415dd60 0x7f7404161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74040d06a0 0x7f7404161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740c800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740415e140 con 0x7f740409adf0 2024-04-17T10:27:53.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740c800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74041626d0 con 0x7f740415dd60 2024-04-17T10:27:53.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7404099d50 con 0x7f74040d06a0 2024-04-17T10:27:53.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7409400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f740409adf0 0x7f74040cff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43876/0 (socket says 172.21.15.67:43876) 2024-04-17T10:27:53.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7409400640 1 -- 172.21.15.67:0/1426799716 learned_addr learned my addr 172.21.15.67:0/1426799716 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:53.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7409e00640 1 --2- 172.21.15.67:0/1426799716 >> v2:172.21.15.67:3300/0 conn(0x7f740415dd60 0x7f7404161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7409e00640 1 -- 172.21.15.67:0/1426799716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74040d06a0 msgr2=0x7f7404161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740a800640 1 --2- 172.21.15.67:0/1426799716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74040d06a0 0x7f7404161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7408a00640 1 -- 172.21.15.67:0/1426799716 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687612034 0 0) 0x7f740415e140 con 0x7f740409adf0 2024-04-17T10:27:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7409e00640 1 --2- 172.21.15.67:0/1426799716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74040d06a0 0x7f7404161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7409e00640 1 -- 172.21.15.67:0/1426799716 >> v1:172.21.15.67:6789/0 conn(0x7f740409adf0 legacy=0x7f74040cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f7409e00640 1 -- 172.21.15.67:0/1426799716 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740409a030 con 0x7f740415dd60 2024-04-17T10:27:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.502+0000 7f740a800640 1 --2- 172.21.15.67:0/1426799716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74040d06a0 0x7f7404161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f7409e00640 1 --2- 172.21.15.67:0/1426799716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f740415dd60 0x7f7404161880 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f73f80097c0 tx=0x7f73f801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b5bd8f7c8b9e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f7408a00640 1 -- 172.21.15.67:0/1426799716 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80a7070 con 0x7f740415dd60 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f7408a00640 1 -- 172.21.15.67:0/1426799716 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f80b0070 con 0x7f740415dd60 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f7408a00640 1 -- 172.21.15.67:0/1426799716 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f80ab890 con 0x7f740415dd60 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 -- 172.21.15.67:0/1426799716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f740415dd60 msgr2=0x7f7404161880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 --2- 172.21.15.67:0/1426799716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f740415dd60 0x7f7404161880 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f73f80097c0 tx=0x7f73f801c470 comp rx=0 tx=0).stop 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 -- 172.21.15.67:0/1426799716 shutdown_connections 2024-04-17T10:27:53.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 --2- 172.21.15.67:0/1426799716 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74040d06a0 0x7f7404161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 --2- 172.21.15.67:0/1426799716 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f740415dd60 0x7f7404161880 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 -- 172.21.15.67:0/1426799716 >> 172.21.15.67:0/1426799716 conn(0x7f74040b34a0 msgr2=0x7f74040b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:53.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 -- 172.21.15.67:0/1426799716 shutdown_connections 2024-04-17T10:27:53.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 -- 172.21.15.67:0/1426799716 wait complete. 2024-04-17T10:27:53.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.506+0000 7f740c800640 1 Processor -- start 2024-04-17T10:27:53.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 -- start start 2024-04-17T10:27:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f740409adf0 0x7f7404153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74040d06a0 0x7f74040cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f740415dd60 0x7f74040cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740406c730 con 0x7f74040d06a0 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f740406c5b0 con 0x7f740415dd60 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740406c8b0 con 0x7f740409adf0 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74040d06a0 0x7f74040cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74040d06a0 0x7f74040cc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37166/0 (socket says 172.21.15.67:37166) 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 -- 172.21.15.67:0/1679078967 learned_addr learned my addr 172.21.15.67:0/1679078967 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740a800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f740415dd60 0x7f74040cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409e00640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f740409adf0 0x7f7404153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 -- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f740409adf0 msgr2=0x7f7404153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f740409adf0 0x7f7404153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 -- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f740415dd60 msgr2=0x7f74040cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f740415dd60 0x7f74040cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 -- 172.21.15.67:0/1679078967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404161180 con 0x7f74040d06a0 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740a800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f740415dd60 0x7f74040cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409e00640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f740409adf0 0x7f7404153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f7409400640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74040d06a0 0x7f74040cc480 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f73f40011f0 tx=0x7f73f400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f73fe000640 1 -- 172.21.15.67:0/1679078967 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4018070 con 0x7f74040d06a0 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f73fe000640 1 -- 172.21.15.67:0/1679078967 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f40074e0 con 0x7f74040d06a0 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f73fe000640 1 -- 172.21.15.67:0/1679078967 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4004a10 con 0x7f74040d06a0 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74040c8df0 con 0x7f74040d06a0 2024-04-17T10:27:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7404160ea0 con 0x7f74040d06a0 2024-04-17T10:27:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.510+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740404ef30 con 0x7f74040d06a0 2024-04-17T10:27:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.514+0000 7f73fe000640 1 -- 172.21.15.67:0/1679078967 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f73f401b070 con 0x7f74040d06a0 2024-04-17T10:27:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.514+0000 7f73fe000640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73e8066f60 0x7f73e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.514+0000 7f73fe000640 1 -- 172.21.15.67:0/1679078967 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f73f40842a0 con 0x7f74040d06a0 2024-04-17T10:27:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.514+0000 7f7409e00640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73e8066f60 0x7f73e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.514+0000 7f7409e00640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73e8066f60 0x7f73e8069420 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f73f80af000 tx=0x7f73f8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.518+0000 7f73fe000640 1 -- 172.21.15.67:0/1679078967 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f73f4055380 con 0x7f74040d06a0 2024-04-17T10:27:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.690+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f74040d0360 con 0x7f74040d06a0 2024-04-17T10:27:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.694+0000 7f73fe000640 1 -- 172.21.15.67:0/1679078967 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f73f4054c80 con 0x7f74040d06a0 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73e8066f60 msgr2=0x7f73e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73e8066f60 0x7f73e8069420 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f73f80af000 tx=0x7f73f8008040 comp rx=0 tx=0).stop 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74040d06a0 msgr2=0x7f74040cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74040d06a0 0x7f74040cc480 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f73f40011f0 tx=0x7f73f400c9b0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 shutdown_connections 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73e8066f60 0x7f73e8069420 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f740415dd60 0x7f74040cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74040d06a0 0x7f74040cc480 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 --2- 172.21.15.67:0/1679078967 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f740409adf0 0x7f7404153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 >> 172.21.15.67:0/1679078967 conn(0x7f74040b34a0 msgr2=0x7f7404070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:53.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 shutdown_connections 2024-04-17T10:27:53.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.698+0000 7f740c800640 1 -- 172.21.15.67:0/1679078967 wait complete. 2024-04-17T10:27:53.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir' 2024-04-17T10:27:53.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46b200640 1 Processor -- start 2024-04-17T10:27:53.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46b200640 1 -- start start 2024-04-17T10:27:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa46c15b860 0x7fa46c15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06c3b0 0x7fa46c15fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46c0b4f60 con 0x7fa46c06fe10 2024-04-17T10:27:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa46c0703d0 con 0x7fa46c15b860 2024-04-17T10:27:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c0b2e50 con 0x7fa46c06c3b0 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa463e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa46c06fe10 0x7fa46c06bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43888/0 (socket says 172.21.15.67:43888) 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa463e00640 1 -- 172.21.15.67:0/377177132 learned_addr learned my addr 172.21.15.67:0/377177132 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa469e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa46c15b860 0x7fa46c15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa46a800640 1 --2- 172.21.15.67:0/377177132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06c3b0 0x7fa46c15fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa469e00640 1 -- 172.21.15.67:0/377177132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06c3b0 msgr2=0x7fa46c15fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa469e00640 1 --2- 172.21.15.67:0/377177132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06c3b0 0x7fa46c15fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa469e00640 1 -- 172.21.15.67:0/377177132 >> v1:172.21.15.67:6789/0 conn(0x7fa46c06fe10 legacy=0x7fa46c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.858+0000 7fa469e00640 1 -- 172.21.15.67:0/377177132 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c0b3130 con 0x7fa46c15b860 2024-04-17T10:27:53.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa469e00640 1 --2- 172.21.15.67:0/377177132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15b860 0x7fa46c15bc40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0097c0 tx=0x7fa45c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d4e22d1f48bcd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:53.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa469400640 1 -- 172.21.15.67:0/377177132 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0b0070 con 0x7fa46c15b860 2024-04-17T10:27:53.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa469400640 1 -- 172.21.15.67:0/377177132 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c0ab600 con 0x7fa46c15b860 2024-04-17T10:27:53.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa469400640 1 -- 172.21.15.67:0/377177132 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0ab9d0 con 0x7fa46c15b860 2024-04-17T10:27:53.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 -- 172.21.15.67:0/377177132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15b860 msgr2=0x7fa46c15bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 --2- 172.21.15.67:0/377177132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15b860 0x7fa46c15bc40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa45c0097c0 tx=0x7fa45c01c380 comp rx=0 tx=0).stop 2024-04-17T10:27:53.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 -- 172.21.15.67:0/377177132 shutdown_connections 2024-04-17T10:27:53.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 --2- 172.21.15.67:0/377177132 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06c3b0 0x7fa46c15fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 --2- 172.21.15.67:0/377177132 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c15b860 0x7fa46c15bc40 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 -- 172.21.15.67:0/377177132 >> 172.21.15.67:0/377177132 conn(0x7fa46c071a80 msgr2=0x7fa46c071e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:53.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 -- 172.21.15.67:0/377177132 shutdown_connections 2024-04-17T10:27:53.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.862+0000 7fa46b200640 1 -- 172.21.15.67:0/377177132 wait complete. 2024-04-17T10:27:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 Processor -- start 2024-04-17T10:27:53.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 -- start start 2024-04-17T10:27:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c06c3b0 0x7fa46c146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06fe10 0x7fa46c0ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa46c15b860 0x7fa46c14b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c0b47b0 con 0x7fa46c06c3b0 2024-04-17T10:27:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c0b4630 con 0x7fa46c15b860 2024-04-17T10:27:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c0b4930 con 0x7fa46c06fe10 2024-04-17T10:27:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c06c3b0 0x7fa46c146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c06c3b0 0x7fa46c146000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37182/0 (socket says 172.21.15.67:37182) 2024-04-17T10:27:53.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 -- 172.21.15.67:0/1703825041 learned_addr learned my addr 172.21.15.67:0/1703825041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 -- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06fe10 msgr2=0x7fa46c0ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06fe10 0x7fa46c0ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 -- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa46c15b860 msgr2=0x7fa46c14b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa46c15b860 0x7fa46c14b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 -- 172.21.15.67:0/1703825041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c0add60 con 0x7fa46c06c3b0 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa469e00640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c06c3b0 0x7fa46c146000 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa45c019fb0 tx=0x7fa45c01abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa462000640 1 -- 172.21.15.67:0/1703825041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0b0050 con 0x7fa46c06c3b0 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa462000640 1 -- 172.21.15.67:0/1703825041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45c0bc070 con 0x7fa46c06c3b0 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa462000640 1 -- 172.21.15.67:0/1703825041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c0b5850 con 0x7fa46c06c3b0 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c0ae040 con 0x7fa46c06c3b0 2024-04-17T10:27:53.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c0ae600 con 0x7fa46c06c3b0 2024-04-17T10:27:53.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.866+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46c074dc0 con 0x7fa46c06c3b0 2024-04-17T10:27:53.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.870+0000 7fa462000640 1 -- 172.21.15.67:0/1703825041 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa45c0b59f0 con 0x7fa46c06c3b0 2024-04-17T10:27:53.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.870+0000 7fa462000640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa448066f60 0x7fa448069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:53.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.870+0000 7fa462000640 1 -- 172.21.15.67:0/1703825041 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa45c11f7c0 con 0x7fa46c06c3b0 2024-04-17T10:27:53.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.870+0000 7fa463e00640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa448066f60 0x7fa448069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:53.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.870+0000 7fa463e00640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa448066f60 0x7fa448069420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa454005890 tx=0x7fa454007a30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:53.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:53.874+0000 7fa462000640 1 -- 172.21.15.67:0/1703825041 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa45c016090 con 0x7fa46c06c3b0 2024-04-17T10:27:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.090+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]} v 0) v1 -- 0x7fa46c06beb0 con 0x7fa46c06c3b0 2024-04-17T10:27:54.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa462000640 1 -- 172.21.15.67:0/1703825041 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]}]=0 v157) v1 ==== 148+0+0 (secure 0 0 0) 0x7fa45c0f0930 con 0x7fa46c06c3b0 2024-04-17T10:27:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa448066f60 msgr2=0x7fa448069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa448066f60 0x7fa448069420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa454005890 tx=0x7fa454007a30 comp rx=0 tx=0).stop 2024-04-17T10:27:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c06c3b0 msgr2=0x7fa46c146000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c06c3b0 0x7fa46c146000 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa45c019fb0 tx=0x7fa45c01abc0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 shutdown_connections 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa448066f60 0x7fa448069420 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa46c15b860 0x7fa46c14b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa46c06fe10 0x7fa46c0ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 --2- 172.21.15.67:0/1703825041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46c06c3b0 0x7fa46c146000 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.326+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 >> 172.21.15.67:0/1703825041 conn(0x7fa46c071a80 msgr2=0x7fa46c158e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.330+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 shutdown_connections 2024-04-17T10:27:54.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.330+0000 7fa46b200640 1 -- 172.21.15.67:0/1703825041 wait complete. 2024-04-17T10:27:54.342 INFO:tasks.cephfs_test_runner:test flag scrub_mdsdir ... ok 2024-04-17T10:27:54.343 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-04-17T10:27:54.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:27:54.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f7466000640 1 Processor -- start 2024-04-17T10:27:54.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f7466000640 1 -- start start 2024-04-17T10:27:54.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f7466000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74600cf0c0 0x7f74601441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f7466000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746015ae40 con 0x7f746006e6e0 2024-04-17T10:27:54.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f7466000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7460071440 con 0x7f746006eac0 2024-04-17T10:27:54.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f7466000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7460071720 con 0x7f74600cf0c0 2024-04-17T10:27:54.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f745f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f746006e6e0 0x7f74600ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43898/0 (socket says 172.21.15.67:43898) 2024-04-17T10:27:54.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f745f400640 1 -- 172.21.15.67:0/4104872963 learned_addr learned my addr 172.21.15.67:0/4104872963 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:54.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.490+0000 7f745fe00640 1 --2- 172.21.15.67:0/4104872963 >> v2:172.21.15.67:3300/0 conn(0x7f74600cf0c0 0x7f74601441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745ea00640 1 -- 172.21.15.67:0/4104872963 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914126276 0 0) 0x7f746015ae40 con 0x7f746006e6e0 2024-04-17T10:27:54.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745fe00640 1 -- 172.21.15.67:0/4104872963 >> v1:172.21.15.86:6789/0 conn(0x7f746006eac0 legacy=0x7f7460143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745fe00640 1 -- 172.21.15.67:0/4104872963 >> v1:172.21.15.67:6789/0 conn(0x7f746006e6e0 legacy=0x7f74600ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745fe00640 1 -- 172.21.15.67:0/4104872963 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7460071a00 con 0x7f74600cf0c0 2024-04-17T10:27:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745fe00640 1 --2- 172.21.15.67:0/4104872963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f74601441f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7458009d90 tx=0x7f745801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d59cebd50e846e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745ea00640 1 -- 172.21.15.67:0/4104872963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580b2070 con 0x7f74600cf0c0 2024-04-17T10:27:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745ea00640 1 -- 172.21.15.67:0/4104872963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74580a5970 con 0x7f74600cf0c0 2024-04-17T10:27:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f745ea00640 1 -- 172.21.15.67:0/4104872963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580a5d40 con 0x7f74600cf0c0 2024-04-17T10:27:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f7466000640 1 -- 172.21.15.67:0/4104872963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 msgr2=0x7f74601441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f7466000640 1 --2- 172.21.15.67:0/4104872963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f74601441f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7458009d90 tx=0x7f745801c8b0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f7466000640 1 -- 172.21.15.67:0/4104872963 shutdown_connections 2024-04-17T10:27:54.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f7466000640 1 --2- 172.21.15.67:0/4104872963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f74601441f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f7466000640 1 -- 172.21.15.67:0/4104872963 >> 172.21.15.67:0/4104872963 conn(0x7f746006ac90 msgr2=0x7f746006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f7466000640 1 -- 172.21.15.67:0/4104872963 shutdown_connections 2024-04-17T10:27:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.494+0000 7f7466000640 1 -- 172.21.15.67:0/4104872963 wait complete. 2024-04-17T10:27:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 Processor -- start 2024-04-17T10:27:54.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 -- start start 2024-04-17T10:27:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f746006e6e0 0x7f746015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f746006eac0 0x7f74600af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f746014d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74601680b0 con 0x7f74600cf0c0 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7460167f30 con 0x7f746006eac0 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7460168230 con 0x7f746006e6e0 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f746014d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745f400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f746006e6e0 0x7f746015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f746014d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37210/0 (socket says 172.21.15.67:37210) 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 -- 172.21.15.67:0/951798844 learned_addr learned my addr 172.21.15.67:0/951798844 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7456a00640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f746006eac0 0x7f74600af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 -- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f746006e6e0 msgr2=0x7f746015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f746006e6e0 0x7f746015a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 -- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f746006eac0 msgr2=0x7f74600af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f746006eac0 0x7f74600af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 -- 172.21.15.67:0/951798844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74600b00c0 con 0x7f74600cf0c0 2024-04-17T10:27:54.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f745fe00640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f746014d0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f745801a4e0 tx=0x7f7458019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7457e00640 1 -- 172.21.15.67:0/951798844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580bd020 con 0x7f74600cf0c0 2024-04-17T10:27:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7457e00640 1 -- 172.21.15.67:0/951798844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74580ad740 con 0x7f74600cf0c0 2024-04-17T10:27:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7457e00640 1 -- 172.21.15.67:0/951798844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580b2040 con 0x7f74600cf0c0 2024-04-17T10:27:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.498+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74600b03a0 con 0x7f74600cf0c0 2024-04-17T10:27:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.502+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74600b0960 con 0x7f74600cf0c0 2024-04-17T10:27:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.502+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7460072d50 con 0x7f74600cf0c0 2024-04-17T10:27:54.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.502+0000 7f7457e00640 1 -- 172.21.15.67:0/951798844 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f74580c0020 con 0x7f74600cf0c0 2024-04-17T10:27:54.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.502+0000 7f7457e00640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.502+0000 7f7457e00640 1 -- 172.21.15.67:0/951798844 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f745811ff40 con 0x7f74600cf0c0 2024-04-17T10:27:54.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.502+0000 7f745f400640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.502+0000 7f745f400640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7450007bc0 tx=0x7f7450007b10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:54.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.506+0000 7f7457e00640 1 -- 172.21.15.67:0/951798844 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f74580c7030 con 0x7f74600cf0c0 2024-04-17T10:27:54.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.670+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7460074e30 con 0x7f74600cf0c0 2024-04-17T10:27:54.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.670+0000 7f7457e00640 1 -- 172.21.15.67:0/951798844 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+13245 (secure 0 0 0) 0x7f7458016560 con 0x7f74600cf0c0 2024-04-17T10:27:54.666 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:54.666 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":46,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:27:53.320372+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/1409597842":"2024-04-18T10:27:50.305900+0000","172.21.15.67:6833/1409597842":"2024-04-18T10:27:50.305900+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:27:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 msgr2=0x7f743c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7450007bc0 tx=0x7f7450007b10 comp rx=0 tx=0).stop 2024-04-17T10:27:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 msgr2=0x7f746014d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f746014d0e0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f745801a4e0 tx=0x7f7458019650 comp rx=0 tx=0).stop 2024-04-17T10:27:54.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 shutdown_connections 2024-04-17T10:27:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600cf0c0 0x7f746014d0e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f746006eac0 0x7f74600af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 --2- 172.21.15.67:0/951798844 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f746006e6e0 0x7f746015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 >> 172.21.15.67:0/951798844 conn(0x7f746006ac90 msgr2=0x7f746006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 shutdown_connections 2024-04-17T10:27:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.674+0000 7f7466000640 1 -- 172.21.15.67:0/951798844 wait complete. 2024-04-17T10:27:54.688 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:27:54.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:27:54.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46a00640 1 Processor -- start 2024-04-17T10:27:54.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46a00640 1 -- start start 2024-04-17T10:27:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff400726c0 0x7eff4015abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4006d840 con 0x7eff400cbdb0 2024-04-17T10:27:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4006db20 con 0x7eff4015a7e0 2024-04-17T10:27:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4006de00 con 0x7eff400726c0 2024-04-17T10:27:54.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7eff400cbdb0 0x7eff400b35b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43910/0 (socket says 172.21.15.67:43910) 2024-04-17T10:27:54.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff46000640 1 -- 172.21.15.67:0/1130638902 learned_addr learned my addr 172.21.15.67:0/1130638902 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff44c00640 1 --2- 172.21.15.67:0/1130638902 >> v2:172.21.15.67:3300/0 conn(0x7eff400726c0 0x7eff4015abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff3fe00640 1 -- 172.21.15.67:0/1130638902 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348188553 0 0) 0x7eff4006d840 con 0x7eff400cbdb0 2024-04-17T10:27:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff3fe00640 1 -- 172.21.15.67:0/1130638902 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff28005740 con 0x7eff400cbdb0 2024-04-17T10:27:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff3fe00640 1 -- 172.21.15.67:0/1130638902 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285813960 0 0) 0x7eff4006db20 con 0x7eff4015a7e0 2024-04-17T10:27:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff44c00640 1 -- 172.21.15.67:0/1130638902 >> v1:172.21.15.86:6789/0 conn(0x7eff4015a7e0 legacy=0x7eff400ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff44c00640 1 -- 172.21.15.67:0/1130638902 >> v1:172.21.15.67:6789/0 conn(0x7eff400cbdb0 legacy=0x7eff400b35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.838+0000 7eff44c00640 1 -- 172.21.15.67:0/1130638902 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff400b4a90 con 0x7eff400726c0 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff44c00640 1 --2- 172.21.15.67:0/1130638902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400726c0 0x7eff4015abc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0097c0 tx=0x7eff2c01c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bf95db3b126914 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff3fe00640 1 -- 172.21.15.67:0/1130638902 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0b0070 con 0x7eff400726c0 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff3fe00640 1 -- 172.21.15.67:0/1130638902 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c0ab600 con 0x7eff400726c0 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff3fe00640 1 -- 172.21.15.67:0/1130638902 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0ab9d0 con 0x7eff400726c0 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff46a00640 1 -- 172.21.15.67:0/1130638902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400726c0 msgr2=0x7eff4015abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff46a00640 1 --2- 172.21.15.67:0/1130638902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400726c0 0x7eff4015abc0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0097c0 tx=0x7eff2c01c350 comp rx=0 tx=0).stop 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff46a00640 1 -- 172.21.15.67:0/1130638902 shutdown_connections 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff46a00640 1 --2- 172.21.15.67:0/1130638902 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400726c0 0x7eff4015abc0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff46a00640 1 -- 172.21.15.67:0/1130638902 >> 172.21.15.67:0/1130638902 conn(0x7eff400b17a0 msgr2=0x7eff400b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff46a00640 1 -- 172.21.15.67:0/1130638902 shutdown_connections 2024-04-17T10:27:54.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.842+0000 7eff46a00640 1 -- 172.21.15.67:0/1130638902 wait complete. 2024-04-17T10:27:54.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 Processor -- start 2024-04-17T10:27:54.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 -- start start 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff400726c0 0x7eff401f6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400cbdb0 0x7eff400b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff4015a7e0 0x7eff401522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4006e310 con 0x7eff400cbdb0 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4006e190 con 0x7eff4015a7e0 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4006e490 con 0x7eff400726c0 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400cbdb0 0x7eff400b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400cbdb0 0x7eff400b2ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37226/0 (socket says 172.21.15.67:37226) 2024-04-17T10:27:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 -- 172.21.15.67:0/4063996154 learned_addr learned my addr 172.21.15.67:0/4063996154 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 -- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff400726c0 msgr2=0x7eff401f6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff400726c0 0x7eff401f6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 -- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff4015a7e0 msgr2=0x7eff401522f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff4015a7e0 0x7eff401522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 -- 172.21.15.67:0/4063996154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff40152d40 con 0x7eff400cbdb0 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff44c00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400cbdb0 0x7eff400b2ad0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0097c0 tx=0x7eff2c01ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff3d600640 1 -- 172.21.15.67:0/4063996154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0bb020 con 0x7eff400cbdb0 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff3d600640 1 -- 172.21.15.67:0/4063996154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c0b1040 con 0x7eff400cbdb0 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff3d600640 1 -- 172.21.15.67:0/4063996154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0b0050 con 0x7eff400cbdb0 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff400acb10 con 0x7eff400cbdb0 2024-04-17T10:27:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff400b3010 con 0x7eff400cbdb0 2024-04-17T10:27:54.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.846+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4004f780 con 0x7eff400cbdb0 2024-04-17T10:27:54.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.850+0000 7eff3d600640 1 -- 172.21.15.67:0/4063996154 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7eff2c0be070 con 0x7eff400cbdb0 2024-04-17T10:27:54.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.850+0000 7eff3d600640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:54.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.850+0000 7eff3d600640 1 -- 172.21.15.67:0/4063996154 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7eff2c11e390 con 0x7eff400cbdb0 2024-04-17T10:27:54.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.850+0000 7eff45600640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:54.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.850+0000 7eff45600640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7eff34000f70 tx=0x7eff340079c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:54.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:54.854+0000 7eff3d600640 1 -- 172.21.15.67:0/4063996154 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7eff2c016090 con 0x7eff400cbdb0 2024-04-17T10:27:55.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.014+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff40145e10 con 0x7eff400cbdb0 2024-04-17T10:27:55.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.014+0000 7eff3d600640 1 -- 172.21.15.67:0/4063996154 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff2c0ef500 con 0x7eff400cbdb0 2024-04-17T10:27:55.011 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:27:55.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 msgr2=0x7eff20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:55.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7eff34000f70 tx=0x7eff340079c0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400cbdb0 msgr2=0x7eff400b2ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400cbdb0 0x7eff400b2ad0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0097c0 tx=0x7eff2c01ab90 comp rx=0 tx=0).stop 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 shutdown_connections 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff4015a7e0 0x7eff401522f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff400cbdb0 0x7eff400b2ad0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 --2- 172.21.15.67:0/4063996154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff400726c0 0x7eff401f6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 >> 172.21.15.67:0/4063996154 conn(0x7eff400b17a0 msgr2=0x7eff401498b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 shutdown_connections 2024-04-17T10:27:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.018+0000 7eff46a00640 1 -- 172.21.15.67:0/4063996154 wait complete. 2024-04-17T10:27:55.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks' 2024-04-17T10:27:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dd1200640 1 Processor -- start 2024-04-17T10:27:55.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dd1200640 1 -- start start 2024-04-17T10:27:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dd1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dcc145040 0x7f0dcc06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dd1200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dd1200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dcc06e2f0 con 0x7f0dcc06ef10 2024-04-17T10:27:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dd1200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dcc0a86c0 con 0x7f0dcc145040 2024-04-17T10:27:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dd1200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc15bf10 con 0x7f0dcc071a20 2024-04-17T10:27:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0dcc145040 0x7f0dcc06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0dcc06ef10 0x7f0dcc071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43922/0 (socket says 172.21.15.67:43922) 2024-04-17T10:27:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc3e00640 1 -- 172.21.15.67:0/2957864692 learned_addr learned my addr 172.21.15.67:0/2957864692 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:55.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dca800640 1 --2- 172.21.15.67:0/2957864692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc9e00640 1 -- 172.21.15.67:0/2957864692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 msgr2=0x7f0dcc0cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc9e00640 1 --2- 172.21.15.67:0/2957864692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc9e00640 1 -- 172.21.15.67:0/2957864692 >> v1:172.21.15.67:6789/0 conn(0x7f0dcc06ef10 legacy=0x7f0dcc071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc9e00640 1 -- 172.21.15.67:0/2957864692 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc06d6e0 con 0x7f0dcc145040 2024-04-17T10:27:55.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.178+0000 7f0dc9400640 1 -- 172.21.15.67:0/2957864692 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022251008 0 0) 0x7f0dcc06e2f0 con 0x7f0dcc06ef10 2024-04-17T10:27:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dc9e00640 1 --2- 172.21.15.67:0/2957864692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc06e9d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0097c0 tx=0x7f0dbc01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc9b2b506c2023b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dc9400640 1 -- 172.21.15.67:0/2957864692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0b5070 con 0x7f0dcc145040 2024-04-17T10:27:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dc9400640 1 -- 172.21.15.67:0/2957864692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc0a8990 con 0x7f0dcc145040 2024-04-17T10:27:55.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dc9400640 1 -- 172.21.15.67:0/2957864692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0a8d60 con 0x7f0dcc145040 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 -- 172.21.15.67:0/2957864692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 msgr2=0x7f0dcc06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 --2- 172.21.15.67:0/2957864692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc06e9d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0097c0 tx=0x7f0dbc01ec90 comp rx=0 tx=0).stop 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 -- 172.21.15.67:0/2957864692 shutdown_connections 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 --2- 172.21.15.67:0/2957864692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 --2- 172.21.15.67:0/2957864692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc06e9d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 -- 172.21.15.67:0/2957864692 >> 172.21.15.67:0/2957864692 conn(0x7f0dcc06ac90 msgr2=0x7f0dcc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 -- 172.21.15.67:0/2957864692 shutdown_connections 2024-04-17T10:27:55.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.182+0000 7f0dd1200640 1 -- 172.21.15.67:0/2957864692 wait complete. 2024-04-17T10:27:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 Processor -- start 2024-04-17T10:27:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 -- start start 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dcc06ef10 0x7f0dcc15aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc0b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc06df80 con 0x7f0dcc145040 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc06dca0 con 0x7f0dcc06ef10 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dcc06d370 con 0x7f0dcc071a20 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc0b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc9e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dcc06ef10 0x7f0dcc15aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:55.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc0b4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37240/0 (socket says 172.21.15.67:37240) 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc9e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dcc06ef10 0x7f0dcc15aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:50406/0 (socket says 172.21.15.67:50406) 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc9e00640 1 -- 172.21.15.67:0/684479213 learned_addr learned my addr 172.21.15.67:0/684479213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc3e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 -- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 msgr2=0x7f0dcc0b3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0b3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 -- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dcc06ef10 msgr2=0x7f0dcc15aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dcc06ef10 0x7f0dcc15aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 -- 172.21.15.67:0/684479213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dcc1f7860 con 0x7f0dcc145040 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc9e00640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dcc06ef10 0x7f0dcc15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:27:55.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dca800640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc0b4420 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4007d60 tx=0x7f0dc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc2000640 1 -- 172.21.15.67:0/684479213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc400edd0 con 0x7f0dcc145040 2024-04-17T10:27:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc2000640 1 -- 172.21.15.67:0/684479213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc40039a0 con 0x7f0dcc145040 2024-04-17T10:27:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dc2000640 1 -- 172.21.15.67:0/684479213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc4003d90 con 0x7f0dcc145040 2024-04-17T10:27:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dcc0adba0 con 0x7f0dcc145040 2024-04-17T10:27:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.186+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dcc0ae160 con 0x7f0dcc145040 2024-04-17T10:27:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.190+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dcc0ae360 con 0x7f0dcc145040 2024-04-17T10:27:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.190+0000 7f0dc2000640 1 -- 172.21.15.67:0/684479213 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0dc401b020 con 0x7f0dcc145040 2024-04-17T10:27:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.190+0000 7f0dc2000640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da8066f90 0x7f0da8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.190+0000 7f0dc2000640 1 -- 172.21.15.67:0/684479213 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0dc4084f30 con 0x7f0dcc145040 2024-04-17T10:27:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.190+0000 7f0dc9e00640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da8066f90 0x7f0da8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.190+0000 7f0dc9e00640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da8066f90 0x7f0da8069450 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc01c8c0 tx=0x7f0dbc0aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.194+0000 7f0dc2000640 1 -- 172.21.15.67:0/684479213 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0dcc0ae360 con 0x7f0dcc145040 2024-04-17T10:27:55.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:55.414+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]} v 0) v1 -- 0x7f0dcc074e30 con 0x7f0dcc145040 2024-04-17T10:27:55.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:27:55.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:27:56.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.334+0000 7f0dc2000640 1 -- 172.21.15.67:0/684479213 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]}]=0 v159) v1 ==== 146+0+0 (secure 0 0 0) 0x7f0dc40559c0 con 0x7f0dcc145040 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da8066f90 msgr2=0x7f0da8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da8066f90 0x7f0da8069450 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc01c8c0 tx=0x7f0dbc0aa070 comp rx=0 tx=0).stop 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 msgr2=0x7f0dcc0b4420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc0b4420 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4007d60 tx=0x7f0dc400e4b0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 shutdown_connections 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da8066f90 0x7f0da8069450 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dcc145040 0x7f0dcc0b4420 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dcc071a20 0x7f0dcc0b3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 --2- 172.21.15.67:0/684479213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dcc06ef10 0x7f0dcc15aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 >> 172.21.15.67:0/684479213 conn(0x7f0dcc06ac90 msgr2=0x7f0dcc0ccd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:56.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 shutdown_connections 2024-04-17T10:27:56.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.338+0000 7f0dd1200640 1 -- 172.21.15.67:0/684479213 wait complete. 2024-04-17T10:27:56.357 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:27:56.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.510+0000 7f2fbbe00640 1 Processor -- start 2024-04-17T10:27:56.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.510+0000 7f2fbbe00640 1 -- start start 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb40aef60 0x7f2fb40cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fbbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fb406e980 0x7f2fb40b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fbbe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fb40b4990 con 0x7f2fb40cce60 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fbbe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fb40cd2c0 con 0x7f2fb406e980 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40af340 con 0x7f2fb40aef60 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fb406e980 0x7f2fb40b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb8a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2fb40cce60 0x7f2fb406e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43936/0 (socket says 172.21.15.67:43936) 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb8a00640 1 -- 172.21.15.67:0/2450466198 learned_addr learned my addr 172.21.15.67:0/2450466198 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb9400640 1 --2- 172.21.15.67:0/2450466198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb40aef60 0x7f2fb40cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb9e00640 1 -- 172.21.15.67:0/2450466198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb40aef60 msgr2=0x7f2fb40cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb9e00640 1 --2- 172.21.15.67:0/2450466198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb40aef60 0x7f2fb40cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb9e00640 1 -- 172.21.15.67:0/2450466198 >> v1:172.21.15.67:6789/0 conn(0x7f2fb40cce60 legacy=0x7f2fb406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb9e00640 1 -- 172.21.15.67:0/2450466198 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb415ae90 con 0x7f2fb406e980 2024-04-17T10:27:56.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb9400640 1 --2- 172.21.15.67:0/2450466198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb40aef60 0x7f2fb40cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:56.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.514+0000 7f2fb3e00640 1 -- 172.21.15.67:0/2450466198 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637964637 0 0) 0x7f2fa0001120 con 0x7f2fb40cce60 2024-04-17T10:27:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fb9e00640 1 --2- 172.21.15.67:0/2450466198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb406e980 0x7f2fb40b3120 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009d90 tx=0x7f2fac01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae94d974a2503971 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fb3e00640 1 -- 172.21.15.67:0/2450466198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0b2070 con 0x7f2fb406e980 2024-04-17T10:27:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fb3e00640 1 -- 172.21.15.67:0/2450466198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac0a5890 con 0x7f2fb406e980 2024-04-17T10:27:56.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fb3e00640 1 -- 172.21.15.67:0/2450466198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0a5c60 con 0x7f2fb406e980 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 -- 172.21.15.67:0/2450466198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb406e980 msgr2=0x7f2fb40b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/2450466198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb406e980 0x7f2fb40b3120 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f2fac009d90 tx=0x7f2fac01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 -- 172.21.15.67:0/2450466198 shutdown_connections 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/2450466198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb406e980 0x7f2fb40b3120 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/2450466198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb40aef60 0x7f2fb40cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 -- 172.21.15.67:0/2450466198 >> 172.21.15.67:0/2450466198 conn(0x7f2fb40b1740 msgr2=0x7f2fb40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 -- 172.21.15.67:0/2450466198 shutdown_connections 2024-04-17T10:27:56.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 -- 172.21.15.67:0/2450466198 wait complete. 2024-04-17T10:27:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 Processor -- start 2024-04-17T10:27:56.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.518+0000 7f2fbbe00640 1 -- start start 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb406e980 0x7f2fb414d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb40aef60 0x7f2fb4159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb40cce60 0x7f2fb4148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb415bc10 con 0x7f2fb40cce60 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb40b3e70 con 0x7f2fb40aef60 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb4152da0 con 0x7f2fb406e980 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb40cce60 0x7f2fb4148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb40cce60 0x7f2fb4148ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37248/0 (socket says 172.21.15.67:37248) 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 -- 172.21.15.67:0/3867644084 learned_addr learned my addr 172.21.15.67:0/3867644084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:56.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9400640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb406e980 0x7f2fb414d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb8a00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb40aef60 0x7f2fb4159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 -- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb406e980 msgr2=0x7f2fb414d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb406e980 0x7f2fb414d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 -- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb40aef60 msgr2=0x7f2fb4159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb40aef60 0x7f2fb4159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 -- 172.21.15.67:0/3867644084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fb40ae7e0 con 0x7f2fb40cce60 2024-04-17T10:27:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb9e00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb40cce60 0x7f2fb4148ea0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2fac01af90 tx=0x7f2fac019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb1600640 1 -- 172.21.15.67:0/3867644084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0b2050 con 0x7f2fb40cce60 2024-04-17T10:27:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb1600640 1 -- 172.21.15.67:0/3867644084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fac0be070 con 0x7f2fb40cce60 2024-04-17T10:27:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb1600640 1 -- 172.21.15.67:0/3867644084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fac0adc00 con 0x7f2fb40cce60 2024-04-17T10:27:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb4153080 con 0x7f2fb40cce60 2024-04-17T10:27:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb4153640 con 0x7f2fb40cce60 2024-04-17T10:27:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb404ef30 con 0x7f2fb40cce60 2024-04-17T10:27:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.522+0000 7f2fb1600640 1 -- 172.21.15.67:0/3867644084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2fac0c3460 con 0x7f2fb40cce60 2024-04-17T10:27:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.526+0000 7f2fb1600640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f94066f60 0x7f2f94069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:56.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.526+0000 7f2fb1600640 1 -- 172.21.15.67:0/3867644084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f2fac1202c0 con 0x7f2fb40cce60 2024-04-17T10:27:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.526+0000 7f2fb9400640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f94066f60 0x7f2f94069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.526+0000 7f2fb9400640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f94066f60 0x7f2f94069420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009950 tx=0x7f2fa8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:56.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.530+0000 7f2fb1600640 1 -- 172.21.15.67:0/3867644084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2fac016560 con 0x7f2fb40cce60 2024-04-17T10:27:56.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.686+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2fb4067fc0 con 0x7f2fb40cce60 2024-04-17T10:27:56.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fb1600640 1 -- 172.21.15.67:0/3867644084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2fac0f0de0 con 0x7f2fb40cce60 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f94066f60 msgr2=0x7f2f94069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f94066f60 0x7f2f94069420 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2fa8009950 tx=0x7f2fa8008040 comp rx=0 tx=0).stop 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb40cce60 msgr2=0x7f2fb4148ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb40cce60 0x7f2fb4148ea0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f2fac01af90 tx=0x7f2fac019680 comp rx=0 tx=0).stop 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 shutdown_connections 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f94066f60 0x7f2f94069420 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fb40cce60 0x7f2fb4148ea0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fb40aef60 0x7f2fb4159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 --2- 172.21.15.67:0/3867644084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fb406e980 0x7f2fb414d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:56.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 >> 172.21.15.67:0/3867644084 conn(0x7f2fb40b1740 msgr2=0x7f2fb40cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:56.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 shutdown_connections 2024-04-17T10:27:56.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:56.694+0000 7f2fbbe00640 1 -- 172.21.15.67:0/3867644084 wait complete. 2024-04-17T10:27:56.709 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:27:56.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:27:56.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:27:56.818 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:27:56.819 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:27:56.819 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:56.827 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:27:56.828 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:27:56.828 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:56.837 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:27:56.837 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:27:56.837 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:56.882 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:27:56.882 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:27:56.882 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:27:56.891 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:27:56.891 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:27:56.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:27:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.074+0000 7fd107200640 1 Processor -- start 2024-04-17T10:27:57.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.074+0000 7fd107200640 1 -- start start 2024-04-17T10:27:57.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd107200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd10015a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd107200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1000cdf10 0x7fd1000ce390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd107200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1000af1c0 con 0x7fd1000cef30 2024-04-17T10:27:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd107200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd10015a9d0 con 0x7fd1000cdf10 2024-04-17T10:27:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd107200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10015ac30 con 0x7fd1000aed60 2024-04-17T10:27:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd0ffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd1000cef30 0x7fd10006daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43948/0 (socket says 172.21.15.67:43948) 2024-04-17T10:27:57.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd0ffe00640 1 -- 172.21.15.67:0/2528629866 learned_addr learned my addr 172.21.15.67:0/2528629866 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd105200640 1 --2- 172.21.15.67:0/2528629866 >> v2:172.21.15.86:3300/0 conn(0x7fd1000cdf10 0x7fd1000ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd0f7400640 1 --2- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd10015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd0ff400640 1 -- 172.21.15.67:0/2528629866 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780994845 0 0) 0x7fd1000af1c0 con 0x7fd1000cef30 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd0ff400640 1 -- 172.21.15.67:0/2528629866 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e4003530 con 0x7fd1000cef30 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd0ff400640 1 -- 172.21.15.67:0/2528629866 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 53729895 0 0) 0x7fd0e4003530 con 0x7fd1000cef30 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd105200640 1 -- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 msgr2=0x7fd10015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd105200640 1 --2- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd10015a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd105200640 1 -- 172.21.15.67:0/2528629866 >> v1:172.21.15.67:6789/0 conn(0x7fd1000cef30 legacy=0x7fd10006daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.078+0000 7fd105200640 1 -- 172.21.15.67:0/2528629866 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10015af10 con 0x7fd1000cdf10 2024-04-17T10:27:57.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd105200640 1 --2- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cdf10 0x7fd1000ce390 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009d90 tx=0x7fd0f801c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17d320ae6cc11808 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd0ff400640 1 -- 172.21.15.67:0/2528629866 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f80ac070 con 0x7fd1000cdf10 2024-04-17T10:27:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd0ff400640 1 -- 172.21.15.67:0/2528629866 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f80a64a0 con 0x7fd1000cdf10 2024-04-17T10:27:57.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd0ff400640 1 -- 172.21.15.67:0/2528629866 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0001680 con 0x7fd1000cdf10 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 -- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cdf10 msgr2=0x7fd1000ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 --2- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cdf10 0x7fd1000ce390 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8009d90 tx=0x7fd0f801c890 comp rx=0 tx=0).stop 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 -- 172.21.15.67:0/2528629866 shutdown_connections 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 --2- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cdf10 0x7fd1000ce390 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 --2- 172.21.15.67:0/2528629866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd10015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 -- 172.21.15.67:0/2528629866 >> 172.21.15.67:0/2528629866 conn(0x7fd10006ac90 msgr2=0x7fd10006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 -- 172.21.15.67:0/2528629866 shutdown_connections 2024-04-17T10:27:57.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.082+0000 7fd107200640 1 -- 172.21.15.67:0/2528629866 wait complete. 2024-04-17T10:27:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 Processor -- start 2024-04-17T10:27:57.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 -- start start 2024-04-17T10:27:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd100151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1000cdf10 0x7fd1000b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cef30 0x7fd10013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1000cf490 con 0x7fd1000cdf10 2024-04-17T10:27:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1000cf310 con 0x7fd1000cef30 2024-04-17T10:27:57.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd100150fe0 con 0x7fd1000aed60 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1000cdf10 0x7fd1000b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1000cdf10 0x7fd1000b3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37252/0 (socket says 172.21.15.67:37252) 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 -- 172.21.15.67:0/738944247 learned_addr learned my addr 172.21.15.67:0/738944247 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0ffe00640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd100151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd105200640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cef30 0x7fd10013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 -- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 msgr2=0x7fd100151cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd100151cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 -- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cef30 msgr2=0x7fd10013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cef30 0x7fd10013ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 -- 172.21.15.67:0/738944247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd10014cc50 con 0x7fd1000cdf10 2024-04-17T10:27:57.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0f7400640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1000cdf10 0x7fd1000b3810 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800e930 tx=0x7fd0e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0fcc00640 1 -- 172.21.15.67:0/738944247 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8017070 con 0x7fd1000cdf10 2024-04-17T10:27:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0fcc00640 1 -- 172.21.15.67:0/738944247 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0e800c9a0 con 0x7fd1000cdf10 2024-04-17T10:27:57.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd0fcc00640 1 -- 172.21.15.67:0/738944247 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e800cd70 con 0x7fd1000cdf10 2024-04-17T10:27:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10014cef0 con 0x7fd1000cdf10 2024-04-17T10:27:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd100155530 con 0x7fd1000cdf10 2024-04-17T10:27:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.086+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd100072d50 con 0x7fd1000cdf10 2024-04-17T10:27:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.090+0000 7fd0fcc00640 1 -- 172.21.15.67:0/738944247 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd0e801a070 con 0x7fd1000cdf10 2024-04-17T10:27:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.090+0000 7fd0fcc00640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0dc066f60 0x7fd0dc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.090+0000 7fd0fcc00640 1 -- 172.21.15.67:0/738944247 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fd0e8084dc0 con 0x7fd1000cdf10 2024-04-17T10:27:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.090+0000 7fd0ffe00640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0dc066f60 0x7fd0dc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.090+0000 7fd0ffe00640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0dc066f60 0x7fd0dc069420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0001330 tx=0x7fd0f00012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:57.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.094+0000 7fd0fcc00640 1 -- 172.21.15.67:0/738944247 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd0e80559a0 con 0x7fd1000cdf10 2024-04-17T10:27:57.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd10006eec0 con 0x7fd1000cdf10 2024-04-17T10:27:57.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd0fcc00640 1 -- 172.21.15.67:0/738944247 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fd0e8059a40 con 0x7fd1000cdf10 2024-04-17T10:27:57.299 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:57.299 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":21,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4946,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1835906721","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1835906721},{"type":"v1","addr":"172.21.15.86:6836","nonce":1835906721}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":4949,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2085506860","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2085506860},{"type":"v1","addr":"172.21.15.86:6837","nonce":2085506860}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13},{"gid":5221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2728147711","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2728147711},{"type":"v1","addr":"172.21.15.67:6834","nonce":2728147711}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":5661,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2684232126","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2684232126},{"type":"v1","addr":"172.21.15.67:6835","nonce":2684232126}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[]} 2024-04-17T10:27:57.300 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 21 2024-04-17T10:27:57.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0dc066f60 msgr2=0x7fd0dc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0dc066f60 0x7fd0dc069420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0001330 tx=0x7fd0f00012c0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1000cdf10 msgr2=0x7fd1000b3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1000cdf10 0x7fd1000b3810 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800e930 tx=0x7fd0e800edf0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 shutdown_connections 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0dc066f60 0x7fd0dc069420 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1000cef30 0x7fd10013ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1000cdf10 0x7fd1000b3810 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 --2- 172.21.15.67:0/738944247 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1000aed60 0x7fd100151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 >> 172.21.15.67:0/738944247 conn(0x7fd10006ac90 msgr2=0x7fd100159780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 shutdown_connections 2024-04-17T10:27:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.306+0000 7fd107200640 1 -- 172.21.15.67:0/738944247 wait complete. 2024-04-17T10:27:57.321 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:27:57.321 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:27:57.321 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:27:57.322 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:27:57.322 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:27:57.322 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:27:57.322 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:27:57.322 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:27:57.322 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:27:57.323 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:27:57.323 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:27:57.323 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:27:57.323 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:27:57.323 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:27:57.323 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:27:57.323 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:27:57.336 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:27:57.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:27:57.338 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:27:57.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:27:57.340 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:27:57.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:27:57.342 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:27:57.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:27:57.384 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:27:57.385 DEBUG:teuthology.parallel:result is None 2024-04-17T10:27:57.386 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:27:57.387 DEBUG:teuthology.parallel:result is None 2024-04-17T10:27:57.389 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:27:57.389 DEBUG:teuthology.parallel:result is None 2024-04-17T10:27:57.391 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:27:57.391 DEBUG:teuthology.parallel:result is None 2024-04-17T10:27:57.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:27:57.470 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:27:57.473 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:27:57.480 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:27:57.481 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:27:57.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f726b600640 1 Processor -- start 2024-04-17T10:27:57.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f726b600640 1 -- start start 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f726b600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f726415dd10 0x7f72640b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f726b600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 0x7f726406c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f726b600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72640b0cc0 con 0x7f72640b57b0 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f726b600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72640b1280 con 0x7f726415dd10 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f726b600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72640b0fa0 con 0x7f72640714d0 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7263e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f72640b57b0 0x7f7264070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43974/0 (socket says 172.21.15.67:43974) 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7263e00640 1 -- 172.21.15.67:0/181840943 learned_addr learned my addr 172.21.15.67:0/181840943 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7269600640 1 --2- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 0x7f726406c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7268c00640 1 --2- 172.21.15.67:0/181840943 >> v2:172.21.15.86:3300/0 conn(0x7f726415dd10 0x7f72640b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7263400640 1 -- 172.21.15.67:0/181840943 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764055632 0 0) 0x7f72640b0cc0 con 0x7f72640b57b0 2024-04-17T10:27:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7263400640 1 -- 172.21.15.67:0/181840943 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f724c003530 con 0x7f72640b57b0 2024-04-17T10:27:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7268c00640 1 -- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 msgr2=0x7f726406c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7268c00640 1 --2- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 0x7f726406c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7263400640 1 -- 172.21.15.67:0/181840943 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782558138 0 0) 0x7f724c003530 con 0x7f72640b57b0 2024-04-17T10:27:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7268c00640 1 -- 172.21.15.67:0/181840943 >> v1:172.21.15.67:6789/0 conn(0x7f72640b57b0 legacy=0x7f7264070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.546+0000 7f7268c00640 1 -- 172.21.15.67:0/181840943 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726406bed0 con 0x7f726415dd10 2024-04-17T10:27:57.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f7268c00640 1 --2- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 0x7f72640b5270 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f72580097c0 tx=0x7f725801c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d30e0e00333d74fa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f7263400640 1 -- 172.21.15.67:0/181840943 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72580b0070 con 0x7f726415dd10 2024-04-17T10:27:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f7263400640 1 -- 172.21.15.67:0/181840943 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72580ab400 con 0x7f726415dd10 2024-04-17T10:27:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f7263400640 1 -- 172.21.15.67:0/181840943 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250001680 con 0x7f726415dd10 2024-04-17T10:27:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 -- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 msgr2=0x7f72640b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 --2- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 0x7f72640b5270 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f72580097c0 tx=0x7f725801c300 comp rx=0 tx=0).stop 2024-04-17T10:27:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 -- 172.21.15.67:0/181840943 shutdown_connections 2024-04-17T10:27:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 --2- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 0x7f726406c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 --2- 172.21.15.67:0/181840943 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 0x7f72640b5270 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 -- 172.21.15.67:0/181840943 >> 172.21.15.67:0/181840943 conn(0x7f72640b34d0 msgr2=0x7f72640b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 -- 172.21.15.67:0/181840943 shutdown_connections 2024-04-17T10:27:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.550+0000 7f726b600640 1 -- 172.21.15.67:0/181840943 wait complete. 2024-04-17T10:27:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 Processor -- start 2024-04-17T10:27:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 -- start start 2024-04-17T10:27:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 0x7f7264149de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 0x7f726415e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72641495a0 0x7f726415e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726406d7b0 con 0x7f72641495a0 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726406d630 con 0x7f726415dd10 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726406d930 con 0x7f72640714d0 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72641495a0 0x7f726415e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72641495a0 0x7f726415e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37298/0 (socket says 172.21.15.67:37298) 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 -- 172.21.15.67:0/1075662718 learned_addr learned my addr 172.21.15.67:0/1075662718 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 -- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 msgr2=0x7f7264149de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 0x7f7264149de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 -- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 msgr2=0x7f726415e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 0x7f726415e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:57.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 -- 172.21.15.67:0/1075662718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72640cce50 con 0x7f72641495a0 2024-04-17T10:27:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7269600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72641495a0 0x7f726415e9e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f725c00ef90 tx=0x7f725c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7260c00640 1 -- 172.21.15.67:0/1075662718 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c018020 con 0x7f72641495a0 2024-04-17T10:27:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7260c00640 1 -- 172.21.15.67:0/1075662718 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c014480 con 0x7f72641495a0 2024-04-17T10:27:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72640cd130 con 0x7f72641495a0 2024-04-17T10:27:57.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f7260c00640 1 -- 172.21.15.67:0/1075662718 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c014870 con 0x7f72641495a0 2024-04-17T10:27:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72640cd6f0 con 0x7f72641495a0 2024-04-17T10:27:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.554+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7264070bb0 con 0x7f72641495a0 2024-04-17T10:27:57.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.558+0000 7f7260c00640 1 -- 172.21.15.67:0/1075662718 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f725c01b020 con 0x7f72641495a0 2024-04-17T10:27:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.558+0000 7f7260c00640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7244066f90 0x7f7244069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.558+0000 7f7260c00640 1 -- 172.21.15.67:0/1075662718 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f725c085dd0 con 0x7f72641495a0 2024-04-17T10:27:57.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.558+0000 7f7268c00640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7244066f90 0x7f7244069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:57.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.558+0000 7f7268c00640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7244066f90 0x7f7244069450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f72580ad000 tx=0x7f7258004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:57.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.562+0000 7f7260c00640 1 -- 172.21.15.67:0/1075662718 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f725c022030 con 0x7f72641495a0 2024-04-17T10:27:57.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:57.734+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7264074e30 con 0x7f72641495a0 2024-04-17T10:27:58.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f7260c00640 1 -- 172.21.15.67:0/1075662718 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v48) v1 ==== 97+0+0 (secure 0 0 0) 0x7f725c010040 con 0x7f72641495a0 2024-04-17T10:27:58.347 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:27:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7244066f90 msgr2=0x7f7244069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7244066f90 0x7f7244069450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f72580ad000 tx=0x7f7258004740 comp rx=0 tx=0).stop 2024-04-17T10:27:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72641495a0 msgr2=0x7f726415e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72641495a0 0x7f726415e9e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f725c00ef90 tx=0x7f725c00c490 comp rx=0 tx=0).stop 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 shutdown_connections 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7244066f90 0x7f7244069450 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72641495a0 0x7f726415e9e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726415dd10 0x7f726415e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 --2- 172.21.15.67:0/1075662718 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72640714d0 0x7f7264149de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.354+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 >> 172.21.15.67:0/1075662718 conn(0x7f72640b34d0 msgr2=0x7f726415cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.358+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 shutdown_connections 2024-04-17T10:27:58.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.358+0000 7f726b600640 1 -- 172.21.15.67:0/1075662718 wait complete. 2024-04-17T10:27:58.375 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:27:58.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 Processor -- start 2024-04-17T10:27:58.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 -- start start 2024-04-17T10:27:58.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f89b40cc250 0x7f89b40cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b40cb260 0x7f89b40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 0x7f89b40cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b40cd150 con 0x7f89b40cb260 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89b406b100 con 0x7f89b40cc250 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89bc200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b406b3e0 con 0x7f89b40ce2a0 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b40cb260 0x7f89b40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89b40cb260 0x7f89b40cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37312/0 (socket says 172.21.15.67:37312) 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 -- 172.21.15.67:0/1799007308 learned_addr learned my addr 172.21.15.67:0/1799007308 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89ba200640 1 --2- 172.21.15.67:0/1799007308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 0x7f89b40cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b9800640 1 --2- 172.21.15.67:0/1799007308 >> v2:172.21.15.86:3300/0 conn(0x7f89b40cc250 0x7f89b40cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 -- 172.21.15.67:0/1799007308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 msgr2=0x7f89b40cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 --2- 172.21.15.67:0/1799007308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 0x7f89b40cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 -- 172.21.15.67:0/1799007308 >> v2:172.21.15.86:3300/0 conn(0x7f89b40cc250 msgr2=0x7f89b40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 --2- 172.21.15.67:0/1799007308 >> v2:172.21.15.86:3300/0 conn(0x7f89b40cc250 0x7f89b40cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.526+0000 7f89b8e00640 1 -- 172.21.15.67:0/1799007308 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b406b6c0 con 0x7f89b40cb260 2024-04-17T10:27:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89b8e00640 1 --2- 172.21.15.67:0/1799007308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b40cd5e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f89b4068850 tx=0x7f89a001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf8f6833c1d4d54 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89b3e00640 1 -- 172.21.15.67:0/1799007308 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00b0070 con 0x7f89b40cb260 2024-04-17T10:27:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89b3e00640 1 -- 172.21.15.67:0/1799007308 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a00ab600 con 0x7f89b40cb260 2024-04-17T10:27:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89b3e00640 1 -- 172.21.15.67:0/1799007308 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a00ab9d0 con 0x7f89b40cb260 2024-04-17T10:27:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 -- 172.21.15.67:0/1799007308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 msgr2=0x7f89b40cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 --2- 172.21.15.67:0/1799007308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b40cd5e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f89b4068850 tx=0x7f89a001c520 comp rx=0 tx=0).stop 2024-04-17T10:27:58.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 -- 172.21.15.67:0/1799007308 shutdown_connections 2024-04-17T10:27:58.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 --2- 172.21.15.67:0/1799007308 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 0x7f89b40cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 --2- 172.21.15.67:0/1799007308 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b40cd5e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 --2- 172.21.15.67:0/1799007308 >> v2:172.21.15.86:3300/0 conn(0x7f89b40cc250 0x7f89b40cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 -- 172.21.15.67:0/1799007308 >> 172.21.15.67:0/1799007308 conn(0x7f89b40b22a0 msgr2=0x7f89b40b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:58.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 -- 172.21.15.67:0/1799007308 shutdown_connections 2024-04-17T10:27:58.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.530+0000 7f89bc200640 1 -- 172.21.15.67:0/1799007308 wait complete. 2024-04-17T10:27:58.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 Processor -- start 2024-04-17T10:27:58.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 -- start start 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b41574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89b40cc250 0x7f89b415dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 0x7f89b415e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b406e850 con 0x7f89b40cb260 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89b406e6d0 con 0x7f89b40cc250 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89b406e9d0 con 0x7f89b40ce2a0 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b41574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b41574a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37314/0 (socket says 172.21.15.67:37314) 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 -- 172.21.15.67:0/699528648 learned_addr learned my addr 172.21.15.67:0/699528648 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:58.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 -- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 msgr2=0x7f89b415e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 0x7f89b415e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 -- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89b40cc250 msgr2=0x7f89b415dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b8e00640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89b40cc250 0x7f89b415dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89b40cc250 0x7f89b415dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 -- 172.21.15.67:0/699528648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89b40c6eb0 con 0x7f89b40cb260 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b8e00640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89b40cc250 0x7f89b415dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b9800640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b41574a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f89a800e860 tx=0x7f89a800ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b1600640 1 -- 172.21.15.67:0/699528648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a8017070 con 0x7f89b40cb260 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b1600640 1 -- 172.21.15.67:0/699528648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a800c940 con 0x7f89b40cb260 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89b40c4e60 con 0x7f89b40cb260 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89b1600640 1 -- 172.21.15.67:0/699528648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a800cd10 con 0x7f89b40cb260 2024-04-17T10:27:58.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89b41579e0 con 0x7f89b40cb260 2024-04-17T10:27:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.534+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89b406cd30 con 0x7f89b40cb260 2024-04-17T10:27:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.538+0000 7f89b1600640 1 -- 172.21.15.67:0/699528648 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f89a801a070 con 0x7f89b40cb260 2024-04-17T10:27:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.538+0000 7f89b1600640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8994066f90 0x7f8994069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.538+0000 7f89b1600640 1 -- 172.21.15.67:0/699528648 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f89a8085210 con 0x7f89b40cb260 2024-04-17T10:27:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.538+0000 7f89b8e00640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8994066f90 0x7f8994069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.538+0000 7f89b8e00640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8994066f90 0x7f8994069450 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f89a00ad000 tx=0x7f89a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:58.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.542+0000 7f89b1600640 1 -- 172.21.15.67:0/699528648 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f89a80563d0 con 0x7f89b40cb260 2024-04-17T10:27:58.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.758+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f89b4073de0 con 0x7f89b40cb260 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.758+0000 7f89b1600640 1 -- 172.21.15.67:0/699528648 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f89a8055cd0 con 0x7f89b40cb260 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.755 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:27:58.756 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.757 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:27:58.758 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:27:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.760 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.761 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.762 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:27:58.763 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.764 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:27:58.765 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.766 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:27:58.767 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:27:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8994066f90 msgr2=0x7f8994069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8994066f90 0x7f8994069450 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f89a00ad000 tx=0x7f89a0008040 comp rx=0 tx=0).stop 2024-04-17T10:27:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 msgr2=0x7f89b41574a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b41574a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f89a800e860 tx=0x7f89a800ed20 comp rx=0 tx=0).stop 2024-04-17T10:27:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 shutdown_connections 2024-04-17T10:27:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8994066f90 0x7f8994069450 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89b40ce2a0 0x7f89b415e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89b40cc250 0x7f89b415dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 --2- 172.21.15.67:0/699528648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89b40cb260 0x7f89b41574a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 >> 172.21.15.67:0/699528648 conn(0x7f89b40b22a0 msgr2=0x7f89b40cd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 shutdown_connections 2024-04-17T10:27:58.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.762+0000 7f89bc200640 1 -- 172.21.15.67:0/699528648 wait complete. 2024-04-17T10:27:58.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:27:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.922+0000 7fbefb400640 1 Processor -- start 2024-04-17T10:27:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.922+0000 7fbefb400640 1 -- start start 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbefb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbef415dbc0 0x7fbef40cf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbefb400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbef409a240 0x7fbef40d0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbefb400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbef40cff30 con 0x7fbef409a740 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbefb400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef40d0b40 con 0x7fbef415dbc0 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbefb400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbef40d0210 con 0x7fbef409a240 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbef415dbc0 0x7fbef40cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbef415dbc0 0x7fbef40cf930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37320/0 (socket says 172.21.15.67:37320) 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef8a00640 1 -- 172.21.15.67:0/2657611389 learned_addr learned my addr 172.21.15.67:0/2657611389 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef9400640 1 --2- 172.21.15.67:0/2657611389 >> v2:172.21.15.86:3300/0 conn(0x7fbef409a240 0x7fbef40d0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef8a00640 1 -- 172.21.15.67:0/2657611389 >> v2:172.21.15.86:3300/0 conn(0x7fbef409a240 msgr2=0x7fbef40d0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef8a00640 1 --2- 172.21.15.67:0/2657611389 >> v2:172.21.15.86:3300/0 conn(0x7fbef409a240 0x7fbef40d0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef8a00640 1 -- 172.21.15.67:0/2657611389 >> v1:172.21.15.86:6789/0 conn(0x7fbef409a740 legacy=0x7fbef4099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef8a00640 1 -- 172.21.15.67:0/2657611389 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef40d04f0 con 0x7fbef415dbc0 2024-04-17T10:27:58.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.926+0000 7fbef9400640 1 --2- 172.21.15.67:0/2657611389 >> v2:172.21.15.86:3300/0 conn(0x7fbef409a240 0x7fbef40d0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:27:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbef8a00640 1 --2- 172.21.15.67:0/2657611389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef40cf930 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbef4072e80 tx=0x7fbee801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8f28e3ed513d20 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbef3400640 1 -- 172.21.15.67:0/2657611389 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee80aa430 con 0x7fbef415dbc0 2024-04-17T10:27:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbef3400640 1 -- 172.21.15.67:0/2657611389 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee80af070 con 0x7fbef415dbc0 2024-04-17T10:27:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbef3400640 1 -- 172.21.15.67:0/2657611389 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee80aa7f0 con 0x7fbef415dbc0 2024-04-17T10:27:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 -- 172.21.15.67:0/2657611389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 msgr2=0x7fbef40cf930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 --2- 172.21.15.67:0/2657611389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef40cf930 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbef4072e80 tx=0x7fbee801c4b0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 -- 172.21.15.67:0/2657611389 shutdown_connections 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 --2- 172.21.15.67:0/2657611389 >> v2:172.21.15.86:3300/0 conn(0x7fbef409a240 0x7fbef40d0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 --2- 172.21.15.67:0/2657611389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef40cf930 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 -- 172.21.15.67:0/2657611389 >> 172.21.15.67:0/2657611389 conn(0x7fbef40b3290 msgr2=0x7fbef40b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 -- 172.21.15.67:0/2657611389 shutdown_connections 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 -- 172.21.15.67:0/2657611389 wait complete. 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 Processor -- start 2024-04-17T10:27:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.930+0000 7fbefb400640 1 -- start start 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef409a240 0x7fbef40b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbef409a740 0x7fbef40b5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef4155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40cc9d0 con 0x7fbef415dbc0 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbef40cc850 con 0x7fbef409a240 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef416abb0 con 0x7fbef409a740 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef4155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef4155ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37330/0 (socket says 172.21.15.67:37330) 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 -- 172.21.15.67:0/592834936 learned_addr learned my addr 172.21.15.67:0/592834936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:58.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef8a00640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef409a240 0x7fbef40b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef3e00640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbef409a740 0x7fbef40b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 -- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbef409a740 msgr2=0x7fbef40b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbef409a740 0x7fbef40b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 -- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef409a240 msgr2=0x7fbef40b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef409a240 0x7fbef40b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 -- 172.21.15.67:0/592834936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef4156220 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef9400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef4155ce0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00cf20 tx=0x7fbeec00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef0c00640 1 -- 172.21.15.67:0/592834936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec01a070 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef0c00640 1 -- 172.21.15.67:0/592834936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeec00d840 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef0c00640 1 -- 172.21.15.67:0/592834936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec00dc30 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbef40b61d0 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbef40b6790 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbef409f890 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.934+0000 7fbef0c00640 1 -- 172.21.15.67:0/592834936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbeec01d070 con 0x7fbef415dbc0 2024-04-17T10:27:58.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.938+0000 7fbef0c00640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbed4066f90 0x7fbed4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:58.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.938+0000 7fbef0c00640 1 -- 172.21.15.67:0/592834936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fbeec085700 con 0x7fbef415dbc0 2024-04-17T10:27:58.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.938+0000 7fbef8a00640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbed4066f90 0x7fbed4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:58.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.938+0000 7fbef8a00640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbed4066f90 0x7fbed4069450 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbee80ae000 tx=0x7fbee80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:58.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:58.942+0000 7fbef0c00640 1 -- 172.21.15.67:0/592834936 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbeec056150 con 0x7fbef415dbc0 2024-04-17T10:27:59.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbef4074e30 con 0x7fbef415dbc0 2024-04-17T10:27:59.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbef0c00640 1 -- 172.21.15.67:0/592834936 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13116 (secure 0 0 0) 0x7fbeec05a1f0 con 0x7fbef415dbc0 2024-04-17T10:27:59.100 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:59.100 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":50,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:27:58.365457+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:27:59.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbed4066f90 msgr2=0x7fbed4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbed4066f90 0x7fbed4069450 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbee80ae000 tx=0x7fbee80a7000 comp rx=0 tx=0).stop 2024-04-17T10:27:59.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 msgr2=0x7fbef4155ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef4155ce0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00cf20 tx=0x7fbeec00a7d0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 shutdown_connections 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbed4066f90 0x7fbed4069450 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbef415dbc0 0x7fbef4155ce0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbef409a740 0x7fbef40b5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 --2- 172.21.15.67:0/592834936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbef409a240 0x7fbef40b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 >> 172.21.15.67:0/592834936 conn(0x7fbef40b3290 msgr2=0x7fbef4070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 shutdown_connections 2024-04-17T10:27:59.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.106+0000 7fbefb400640 1 -- 172.21.15.67:0/592834936 wait complete. 2024-04-17T10:27:59.122 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:27:59.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:27:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.270+0000 7fce9d200640 1 Processor -- start 2024-04-17T10:27:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce9d200640 1 -- start start 2024-04-17T10:27:59.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce9d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce980cd030 0x7fce9806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce9d200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fce9806e7b0 0x7fce980b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce9d200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce980b4990 con 0x7fce980aef60 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce9d200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce980af340 con 0x7fce980cd030 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce9d200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce9806ec00 con 0x7fce9806e7b0 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce95e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fce980aef60 0x7fce980cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43984/0 (socket says 172.21.15.67:43984) 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce95e00640 1 -- 172.21.15.67:0/1278183422 learned_addr learned my addr 172.21.15.67:0/1278183422 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce8fe00640 1 --2- 172.21.15.67:0/1278183422 >> v2:172.21.15.67:3300/0 conn(0x7fce980cd030 0x7fce9806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce96800640 1 --2- 172.21.15.67:0/1278183422 >> v2:172.21.15.86:3300/0 conn(0x7fce9806e7b0 0x7fce980b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce95400640 1 -- 172.21.15.67:0/1278183422 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746509499 0 0) 0x7fce88001120 con 0x7fce980aef60 2024-04-17T10:27:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce95400640 1 -- 172.21.15.67:0/1278183422 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce7c003530 con 0x7fce980aef60 2024-04-17T10:27:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce8fe00640 1 -- 172.21.15.67:0/1278183422 >> v2:172.21.15.86:3300/0 conn(0x7fce9806e7b0 msgr2=0x7fce980b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce8fe00640 1 --2- 172.21.15.67:0/1278183422 >> v2:172.21.15.86:3300/0 conn(0x7fce9806e7b0 0x7fce980b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce8fe00640 1 -- 172.21.15.67:0/1278183422 >> v1:172.21.15.67:6789/0 conn(0x7fce980aef60 legacy=0x7fce980cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.274+0000 7fce8fe00640 1 -- 172.21.15.67:0/1278183422 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9815ae90 con 0x7fce980cd030 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce8fe00640 1 --2- 172.21.15.67:0/1278183422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce9806e270 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fce980b2e60 tx=0x7fce8001e600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee650a2a9d7ce00 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce95400640 1 -- 172.21.15.67:0/1278183422 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800ac660 con 0x7fce980cd030 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce95400640 1 -- 172.21.15.67:0/1278183422 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce800b1070 con 0x7fce980cd030 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce95400640 1 -- 172.21.15.67:0/1278183422 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800aca20 con 0x7fce980cd030 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 -- 172.21.15.67:0/1278183422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 msgr2=0x7fce9806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 --2- 172.21.15.67:0/1278183422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce9806e270 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fce980b2e60 tx=0x7fce8001e600 comp rx=0 tx=0).stop 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 -- 172.21.15.67:0/1278183422 shutdown_connections 2024-04-17T10:27:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 --2- 172.21.15.67:0/1278183422 >> v2:172.21.15.86:3300/0 conn(0x7fce9806e7b0 0x7fce980b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 --2- 172.21.15.67:0/1278183422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce9806e270 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 -- 172.21.15.67:0/1278183422 >> 172.21.15.67:0/1278183422 conn(0x7fce980b1740 msgr2=0x7fce980b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 -- 172.21.15.67:0/1278183422 shutdown_connections 2024-04-17T10:27:59.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 -- 172.21.15.67:0/1278183422 wait complete. 2024-04-17T10:27:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 Processor -- start 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.278+0000 7fce9d200640 1 -- start start 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9806e7b0 0x7fce98147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce980aef60 0x7fce98131e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce98133e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9815bc10 con 0x7fce980cd030 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980b3e70 con 0x7fce9806e7b0 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce981482e0 con 0x7fce980aef60 2024-04-17T10:27:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce98133e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce98133e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37354/0 (socket says 172.21.15.67:37354) 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 -- 172.21.15.67:0/4018002454 learned_addr learned my addr 172.21.15.67:0/4018002454 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce95e00640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9806e7b0 0x7fce98147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce8fe00640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce980aef60 0x7fce98131e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 -- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce980aef60 msgr2=0x7fce98131e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce980aef60 0x7fce98131e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 -- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9806e7b0 msgr2=0x7fce98147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9806e7b0 0x7fce98147da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 -- 172.21.15.67:0/4018002454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce981347d0 con 0x7fce980cd030 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce96800640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce98133e70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fce90007e70 tx=0x7fce9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce8e000640 1 -- 172.21.15.67:0/4018002454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90014620 con 0x7fce980cd030 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce8e000640 1 -- 172.21.15.67:0/4018002454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce90019070 con 0x7fce980cd030 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce98134ab0 con 0x7fce980cd030 2024-04-17T10:27:59.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce8e000640 1 -- 172.21.15.67:0/4018002454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce90014a20 con 0x7fce980cd030 2024-04-17T10:27:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce98153ec0 con 0x7fce980cd030 2024-04-17T10:27:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.282+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce9804ef30 con 0x7fce980cd030 2024-04-17T10:27:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.286+0000 7fce8e000640 1 -- 172.21.15.67:0/4018002454 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fce90014bc0 con 0x7fce980cd030 2024-04-17T10:27:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.286+0000 7fce8e000640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce74066f90 0x7fce74069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.286+0000 7fce8e000640 1 -- 172.21.15.67:0/4018002454 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fce900855e0 con 0x7fce980cd030 2024-04-17T10:27:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.286+0000 7fce95e00640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce74066f90 0x7fce74069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.286+0000 7fce95e00640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce74066f90 0x7fce74069450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fce88005890 tx=0x7fce88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.290+0000 7fce8e000640 1 -- 172.21.15.67:0/4018002454 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fce90056150 con 0x7fce980cd030 2024-04-17T10:27:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce9815ac70 con 0x7fce980cd030 2024-04-17T10:27:59.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce8e000640 1 -- 172.21.15.67:0/4018002454 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce9005a1f0 con 0x7fce980cd030 2024-04-17T10:27:59.447 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:27:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce74066f90 msgr2=0x7fce74069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce74066f90 0x7fce74069450 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fce88005890 tx=0x7fce88009040 comp rx=0 tx=0).stop 2024-04-17T10:27:59.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 msgr2=0x7fce98133e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce98133e70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fce90007e70 tx=0x7fce9000e4b0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 shutdown_connections 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce74066f90 0x7fce74069450 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce980cd030 0x7fce98133e70 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce980aef60 0x7fce98131e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 --2- 172.21.15.67:0/4018002454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9806e7b0 0x7fce98147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 >> 172.21.15.67:0/4018002454 conn(0x7fce980b1740 msgr2=0x7fce980b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 shutdown_connections 2024-04-17T10:27:59.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.454+0000 7fce9d200640 1 -- 172.21.15.67:0/4018002454 wait complete. 2024-04-17T10:27:59.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:27:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8fe00640 1 Processor -- start 2024-04-17T10:27:59.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8fe00640 1 -- start start 2024-04-17T10:27:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6c9006f770 0x7f6c90070470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c9006fb50 0x7f6c9015f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900cf330 con 0x7f6c9006ffe0 2024-04-17T10:27:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c9009b190 con 0x7f6c9006fb50 2024-04-17T10:27:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8fe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c900709b0 con 0x7f6c9006f770 2024-04-17T10:27:59.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c9006fb50 0x7f6c9015f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6c9006fb50 0x7f6c9015f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37370/0 (socket says 172.21.15.67:37370) 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8e000640 1 -- 172.21.15.67:0/1166747209 learned_addr learned my addr 172.21.15.67:0/1166747209 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8f400640 1 --1- 172.21.15.67:0/1166747209 >> v1:172.21.15.67:6789/0 conn(0x7f6c9006ffe0 0x7f6c9015fdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43986/0 (socket says 172.21.15.67:43986) 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8ea00640 1 --2- 172.21.15.67:0/1166747209 >> v2:172.21.15.86:3300/0 conn(0x7f6c9006f770 0x7f6c90070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8e000640 1 -- 172.21.15.67:0/1166747209 >> v2:172.21.15.86:3300/0 conn(0x7f6c9006f770 msgr2=0x7f6c90070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8e000640 1 --2- 172.21.15.67:0/1166747209 >> v2:172.21.15.86:3300/0 conn(0x7f6c9006f770 0x7f6c90070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8e000640 1 -- 172.21.15.67:0/1166747209 >> v1:172.21.15.67:6789/0 conn(0x7f6c9006ffe0 legacy=0x7f6c9015fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8e000640 1 -- 172.21.15.67:0/1166747209 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900cf610 con 0x7f6c9006fb50 2024-04-17T10:27:59.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.618+0000 7f6c8ea00640 1 --2- 172.21.15.67:0/1166747209 >> v2:172.21.15.86:3300/0 conn(0x7f6c9006f770 0x7f6c90070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:27:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8e000640 1 --2- 172.21.15.67:0/1166747209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006fb50 0x7f6c9015f870 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6c900d2310 tx=0x7f6c7c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=695955bdaab47595 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8d600640 1 -- 172.21.15.67:0/1166747209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c0b0070 con 0x7f6c9006fb50 2024-04-17T10:27:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8d600640 1 -- 172.21.15.67:0/1166747209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7c0ab600 con 0x7f6c9006fb50 2024-04-17T10:27:59.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8d600640 1 -- 172.21.15.67:0/1166747209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c0ab9d0 con 0x7f6c9006fb50 2024-04-17T10:27:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 -- 172.21.15.67:0/1166747209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006fb50 msgr2=0x7f6c9015f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/1166747209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006fb50 0x7f6c9015f870 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f6c900d2310 tx=0x7f6c7c01c490 comp rx=0 tx=0).stop 2024-04-17T10:27:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 -- 172.21.15.67:0/1166747209 shutdown_connections 2024-04-17T10:27:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/1166747209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006fb50 0x7f6c9015f870 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/1166747209 >> v2:172.21.15.86:3300/0 conn(0x7f6c9006f770 0x7f6c90070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 -- 172.21.15.67:0/1166747209 >> 172.21.15.67:0/1166747209 conn(0x7f6c900d1630 msgr2=0x7f6c900d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 -- 172.21.15.67:0/1166747209 shutdown_connections 2024-04-17T10:27:59.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 -- 172.21.15.67:0/1166747209 wait complete. 2024-04-17T10:27:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 Processor -- start 2024-04-17T10:27:59.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.622+0000 7f6c8fe00640 1 -- start start 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9006f770 0x7f6c90152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9006fb50 0x7f6c90159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006cc10 0x7f6c90159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901606d0 con 0x7f6c9006cc10 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900715d0 con 0x7f6c9006fb50 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900cc890 con 0x7f6c9006f770 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006cc10 0x7f6c90159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006cc10 0x7f6c90159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37386/0 (socket says 172.21.15.67:37386) 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 -- 172.21.15.67:0/4291905331 learned_addr learned my addr 172.21.15.67:0/4291905331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8e000640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9006fb50 0x7f6c90159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8ea00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9006f770 0x7f6c90152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 -- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9006f770 msgr2=0x7f6c90152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9006f770 0x7f6c90152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 -- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9006fb50 msgr2=0x7f6c90159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9006fb50 0x7f6c90159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 -- 172.21.15.67:0/4291905331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9006d0b0 con 0x7f6c9006cc10 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8f400640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006cc10 0x7f6c90159970 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c730 tx=0x7f6c8800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c86a00640 1 -- 172.21.15.67:0/4291905331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88018070 con 0x7f6c9006cc10 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c86a00640 1 -- 172.21.15.67:0/4291905331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c88004d70 con 0x7f6c9006cc10 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9006d390 con 0x7f6c9006cc10 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c86a00640 1 -- 172.21.15.67:0/4291905331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8800a860 con 0x7f6c9006cc10 2024-04-17T10:27:59.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9006d950 con 0x7f6c9006cc10 2024-04-17T10:27:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.626+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c90074dc0 con 0x7f6c9006cc10 2024-04-17T10:27:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.630+0000 7f6c86a00640 1 -- 172.21.15.67:0/4291905331 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6c8801b070 con 0x7f6c9006cc10 2024-04-17T10:27:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.630+0000 7f6c86a00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c70066f60 0x7f6c70069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.630+0000 7f6c86a00640 1 -- 172.21.15.67:0/4291905331 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6c88084340 con 0x7f6c9006cc10 2024-04-17T10:27:59.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.630+0000 7f6c8ea00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c70066f60 0x7f6c70069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.630+0000 7f6c8ea00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c70066f60 0x7f6c70069420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009950 tx=0x7f6c80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.634+0000 7f6c86a00640 1 -- 172.21.15.67:0/4291905331 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6c88054f50 con 0x7f6c9006cc10 2024-04-17T10:27:59.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.794+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6c90166410 con 0x7f6c9006cc10 2024-04-17T10:27:59.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.794+0000 7f6c86a00640 1 -- 172.21.15.67:0/4291905331 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v50) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6c88058ff0 con 0x7f6c9006cc10 2024-04-17T10:27:59.790 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:27:59.790 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:27:59.790 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:27:59.790 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:27:59.790 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:27:59.790 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:27:59.790 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:27:59.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c70066f60 msgr2=0x7f6c70069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c70066f60 0x7f6c70069420 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009950 tx=0x7f6c80008040 comp rx=0 tx=0).stop 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006cc10 msgr2=0x7f6c90159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006cc10 0x7f6c90159970 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c730 tx=0x7f6c8800cbf0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 shutdown_connections 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c70066f60 0x7f6c70069420 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c9006cc10 0x7f6c90159970 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c9006fb50 0x7f6c90159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 --2- 172.21.15.67:0/4291905331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c9006f770 0x7f6c90152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 >> 172.21.15.67:0/4291905331 conn(0x7f6c900d1630 msgr2=0x7f6c900b1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 shutdown_connections 2024-04-17T10:27:59.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.798+0000 7f6c8fe00640 1 -- 172.21.15.67:0/4291905331 wait complete. 2024-04-17T10:27:59.810 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:27:59.811 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:27:59.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83afe00640 1 Processor -- start 2024-04-17T10:27:59.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83afe00640 1 -- start start 2024-04-17T10:27:59.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83afe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83b00714d0 0x7f83b0071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83afe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f83b013f8e0 0x7f83b013fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83afe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b015c870 con 0x7f83b00ce030 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83afe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b015cb50 con 0x7f83b00714d0 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83afe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83b013f0e0 con 0x7f83b013f8e0 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83a7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83b00714d0 0x7f83b0071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83aea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83b00ce030 0x7f83b015bcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44002/0 (socket says 172.21.15.67:44002) 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83aea00640 1 -- 172.21.15.67:0/2943374511 learned_addr learned my addr 172.21.15.67:0/2943374511 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83a7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f83b00714d0 0x7f83b0071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37390/0 (socket says 172.21.15.67:37390) 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83af400640 1 --2- 172.21.15.67:0/2943374511 >> v2:172.21.15.86:3300/0 conn(0x7f83b013f8e0 0x7f83b013fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83a7e00640 1 -- 172.21.15.67:0/2943374511 >> v2:172.21.15.86:3300/0 conn(0x7f83b013f8e0 msgr2=0x7f83b013fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83a7e00640 1 --2- 172.21.15.67:0/2943374511 >> v2:172.21.15.86:3300/0 conn(0x7f83b013f8e0 0x7f83b013fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83a7e00640 1 -- 172.21.15.67:0/2943374511 >> v1:172.21.15.67:6789/0 conn(0x7f83b00ce030 legacy=0x7f83b015bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83a7e00640 1 -- 172.21.15.67:0/2943374511 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b013f3c0 con 0x7f83b00714d0 2024-04-17T10:27:59.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.962+0000 7f83ae000640 1 -- 172.21.15.67:0/2943374511 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128414038 0 0) 0x7f83b015c870 con 0x7f83b00ce030 2024-04-17T10:27:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83a7e00640 1 --2- 172.21.15.67:0/2943374511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b00714d0 0x7f83b0071980 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f83b00af590 tx=0x7f839801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ee3444ef9b1734 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83ae000640 1 -- 172.21.15.67:0/2943374511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980b0070 con 0x7f83b00714d0 2024-04-17T10:27:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83ae000640 1 -- 172.21.15.67:0/2943374511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980ab600 con 0x7f83b00714d0 2024-04-17T10:27:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83ae000640 1 -- 172.21.15.67:0/2943374511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980ab9d0 con 0x7f83b00714d0 2024-04-17T10:27:59.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 -- 172.21.15.67:0/2943374511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b00714d0 msgr2=0x7f83b0071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 --2- 172.21.15.67:0/2943374511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b00714d0 0x7f83b0071980 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f83b00af590 tx=0x7f839801c480 comp rx=0 tx=0).stop 2024-04-17T10:27:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 -- 172.21.15.67:0/2943374511 shutdown_connections 2024-04-17T10:27:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 --2- 172.21.15.67:0/2943374511 >> v2:172.21.15.86:3300/0 conn(0x7f83b013f8e0 0x7f83b013fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 --2- 172.21.15.67:0/2943374511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b00714d0 0x7f83b0071980 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 -- 172.21.15.67:0/2943374511 >> 172.21.15.67:0/2943374511 conn(0x7f83b00b1650 msgr2=0x7f83b00b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:27:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 -- 172.21.15.67:0/2943374511 shutdown_connections 2024-04-17T10:27:59.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.966+0000 7f83afe00640 1 -- 172.21.15.67:0/2943374511 wait complete. 2024-04-17T10:27:59.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 Processor -- start 2024-04-17T10:27:59.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 -- start start 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00714d0 0x7f83b013e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00ce030 0x7f83b013a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b013f8e0 0x7f83b013ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b0168500 con 0x7f83b013f8e0 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b0168380 con 0x7f83b00714d0 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b0168680 con 0x7f83b00ce030 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b013f8e0 0x7f83b013ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b013f8e0 0x7f83b013ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37394/0 (socket says 172.21.15.67:37394) 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 -- 172.21.15.67:0/3559538993 learned_addr learned my addr 172.21.15.67:0/3559538993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83aea00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00714d0 0x7f83b013e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83a7e00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00ce030 0x7f83b013a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 -- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00ce030 msgr2=0x7f83b013a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00ce030 0x7f83b013a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 -- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00714d0 msgr2=0x7f83b013e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00714d0 0x7f83b013e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 -- 172.21.15.67:0/3559538993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b0154d00 con 0x7f83b013f8e0 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83af400640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b013f8e0 0x7f83b013ad20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f83a800efc0 tx=0x7f83a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83a6a00640 1 -- 172.21.15.67:0/3559538993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8018020 con 0x7f83b013f8e0 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83a6a00640 1 -- 172.21.15.67:0/3559538993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a8014480 con 0x7f83b013f8e0 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83a6a00640 1 -- 172.21.15.67:0/3559538993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8014870 con 0x7f83b013f8e0 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b0133c70 con 0x7f83b013f8e0 2024-04-17T10:27:59.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.970+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b0134230 con 0x7f83b013f8e0 2024-04-17T10:27:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.974+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b0074dc0 con 0x7f83b013f8e0 2024-04-17T10:27:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.974+0000 7f83a6a00640 1 -- 172.21.15.67:0/3559538993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f83a801b020 con 0x7f83b013f8e0 2024-04-17T10:27:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.974+0000 7f83a6a00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f838c066f60 0x7f838c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:27:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.974+0000 7f83a6a00640 1 -- 172.21.15.67:0/3559538993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f83a8085840 con 0x7f83b013f8e0 2024-04-17T10:27:59.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.974+0000 7f83aea00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f838c066f60 0x7f838c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:27:59.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.974+0000 7f83aea00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f838c066f60 0x7f838c069420 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f83a0007fd0 tx=0x7f83a0007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:27:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:27:59.978+0000 7f83a6a00640 1 -- 172.21.15.67:0/3559538993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f83a8022070 con 0x7f83b013f8e0 2024-04-17T10:28:00.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.142+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f83b0134400 con 0x7f83b013f8e0 2024-04-17T10:28:00.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.358+0000 7f83a6a00640 1 -- 172.21.15.67:0/3559538993 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v51) v1 ==== 138+0+0 (secure 0 0 0) 0x7f83a8010040 con 0x7f83b013f8e0 2024-04-17T10:28:00.354 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f838c066f60 msgr2=0x7f838c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f838c066f60 0x7f838c069420 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f83a0007fd0 tx=0x7f83a0007d60 comp rx=0 tx=0).stop 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b013f8e0 msgr2=0x7f83b013ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b013f8e0 0x7f83b013ad20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f83a800efc0 tx=0x7f83a800c490 comp rx=0 tx=0).stop 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 shutdown_connections 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f838c066f60 0x7f838c069420 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83b013f8e0 0x7f83b013ad20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83b00ce030 0x7f83b013a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 --2- 172.21.15.67:0/3559538993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f83b00714d0 0x7f83b013e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 >> 172.21.15.67:0/3559538993 conn(0x7f83b00b1650 msgr2=0x7f83b015a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:00.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 shutdown_connections 2024-04-17T10:28:00.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.362+0000 7f83afe00640 1 -- 172.21.15.67:0/3559538993 wait complete. 2024-04-17T10:28:00.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:28:00.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.534+0000 7f91fd600640 1 Processor -- start 2024-04-17T10:28:00.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.534+0000 7f91fd600640 1 -- start start 2024-04-17T10:28:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91fd600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91f80c4d60 0x7f91f80d0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91fd600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f91f8162dd0 0x7f91f8161330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91fd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f80cd870 con 0x7f91f80cd490 2024-04-17T10:28:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91fd600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f80cf9c0 con 0x7f91f80c4d60 2024-04-17T10:28:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91fd600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f80cf250 con 0x7f91f8162dd0 2024-04-17T10:28:00.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91f80c4d60 0x7f91f80d0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91efe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f91f80c4d60 0x7f91f80d0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52346/0 (socket says 172.21.15.67:52346) 2024-04-17T10:28:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91efe00640 1 -- 172.21.15.67:0/2009170464 learned_addr learned my addr 172.21.15.67:0/2009170464 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91f7e00640 1 --1- 172.21.15.67:0/2009170464 >> v1:172.21.15.67:6789/0 conn(0x7f91f80cd490 0x7f91f80c4650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53746/0 (socket says 172.21.15.67:53746) 2024-04-17T10:28:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91fcc00640 1 --2- 172.21.15.67:0/2009170464 >> v2:172.21.15.86:3300/0 conn(0x7f91f8162dd0 0x7f91f8161330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:00.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91efe00640 1 -- 172.21.15.67:0/2009170464 >> v2:172.21.15.86:3300/0 conn(0x7f91f8162dd0 msgr2=0x7f91f8161330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:00.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91efe00640 1 --2- 172.21.15.67:0/2009170464 >> v2:172.21.15.86:3300/0 conn(0x7f91f8162dd0 0x7f91f8161330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91efe00640 1 -- 172.21.15.67:0/2009170464 >> v1:172.21.15.67:6789/0 conn(0x7f91f80cd490 legacy=0x7f91f80c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:00.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91efe00640 1 -- 172.21.15.67:0/2009170464 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f80b24a0 con 0x7f91f80c4d60 2024-04-17T10:28:00.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.538+0000 7f91f7400640 1 -- 172.21.15.67:0/2009170464 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4156575803 0 0) 0x7f91f80cd870 con 0x7f91f80cd490 2024-04-17T10:28:00.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91efe00640 1 --2- 172.21.15.67:0/2009170464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80c4d60 0x7f91f80d0de0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f91f80d24a0 tx=0x7f91e001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e6d5c42c206e0a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91f7400640 1 -- 172.21.15.67:0/2009170464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00b0070 con 0x7f91f80c4d60 2024-04-17T10:28:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91f7400640 1 -- 172.21.15.67:0/2009170464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e00ab600 con 0x7f91f80c4d60 2024-04-17T10:28:00.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91f7400640 1 -- 172.21.15.67:0/2009170464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00ab9d0 con 0x7f91f80c4d60 2024-04-17T10:28:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 -- 172.21.15.67:0/2009170464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80c4d60 msgr2=0x7f91f80d0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 --2- 172.21.15.67:0/2009170464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80c4d60 0x7f91f80d0de0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f91f80d24a0 tx=0x7f91e001c490 comp rx=0 tx=0).stop 2024-04-17T10:28:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 -- 172.21.15.67:0/2009170464 shutdown_connections 2024-04-17T10:28:00.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 --2- 172.21.15.67:0/2009170464 >> v2:172.21.15.86:3300/0 conn(0x7f91f8162dd0 0x7f91f8161330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 --2- 172.21.15.67:0/2009170464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80c4d60 0x7f91f80d0de0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 -- 172.21.15.67:0/2009170464 >> 172.21.15.67:0/2009170464 conn(0x7f91f80d1680 msgr2=0x7f91f80d1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 -- 172.21.15.67:0/2009170464 shutdown_connections 2024-04-17T10:28:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 -- 172.21.15.67:0/2009170464 wait complete. 2024-04-17T10:28:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 Processor -- start 2024-04-17T10:28:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 -- start start 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.542+0000 7f91fd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f80c4d60 0x7f91f81612f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80cd490 0x7f91f815ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f8162dd0 0x7f91f8157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f80b0f90 con 0x7f91f80cd490 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f81631e0 con 0x7f91f8162dd0 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f80b1110 con 0x7f91f80c4d60 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80cd490 0x7f91f815ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80cd490 0x7f91f815ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52354/0 (socket says 172.21.15.67:52354) 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 -- 172.21.15.67:0/2709764606 learned_addr learned my addr 172.21.15.67:0/2709764606 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91f7e00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f80c4d60 0x7f91f81612f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fcc00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f8162dd0 0x7f91f8157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 -- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f80c4d60 msgr2=0x7f91f81612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f80c4d60 0x7f91f81612f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 -- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f8162dd0 msgr2=0x7f91f8157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f8162dd0 0x7f91f8157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 -- 172.21.15.67:0/2709764606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f815b370 con 0x7f91f80cd490 2024-04-17T10:28:00.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91efe00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80cd490 0x7f91f815ff30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f91e00ae040 tx=0x7f91e0019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:00.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91f4c00640 1 -- 172.21.15.67:0/2709764606 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00b0040 con 0x7f91f80cd490 2024-04-17T10:28:00.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91f4c00640 1 -- 172.21.15.67:0/2709764606 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e00b9070 con 0x7f91f80cd490 2024-04-17T10:28:00.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f815b650 con 0x7f91f80cd490 2024-04-17T10:28:00.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91f4c00640 1 -- 172.21.15.67:0/2709764606 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e00b4680 con 0x7f91f80cd490 2024-04-17T10:28:00.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f815c130 con 0x7f91f80cd490 2024-04-17T10:28:00.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f8074dc0 con 0x7f91f80cd490 2024-04-17T10:28:00.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.546+0000 7f91f4c00640 1 -- 172.21.15.67:0/2709764606 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f91e00b4820 con 0x7f91f80cd490 2024-04-17T10:28:00.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.550+0000 7f91f4c00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f91d4066f90 0x7f91d4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:00.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.550+0000 7f91f4c00640 1 -- 172.21.15.67:0/2709764606 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f91e011ec30 con 0x7f91f80cd490 2024-04-17T10:28:00.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.550+0000 7f91f7e00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f91d4066f90 0x7f91d4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:00.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.550+0000 7f91f7e00640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f91d4066f90 0x7f91d4069450 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f91e8007fd0 tx=0x7f91e8007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:00.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.554+0000 7f91f4c00640 1 -- 172.21.15.67:0/2709764606 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f91e0016110 con 0x7f91f80cd490 2024-04-17T10:28:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:00.714+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f91f809a660 con 0x7f91f80cd490 2024-04-17T10:28:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.362+0000 7f91f4c00640 1 -- 172.21.15.67:0/2709764606 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v52) v1 ==== 170+0+0 (secure 0 0 0) 0x7f91e00ef5a0 con 0x7f91f80cd490 2024-04-17T10:28:01.358 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:28:01.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.370+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f91d4066f90 msgr2=0x7f91d4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:01.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.370+0000 7f91fd600640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f91d4066f90 0x7f91d4069450 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f91e8007fd0 tx=0x7f91e8007d60 comp rx=0 tx=0).stop 2024-04-17T10:28:01.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.370+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80cd490 msgr2=0x7f91f815ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:01.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.370+0000 7f91fd600640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80cd490 0x7f91f815ff30 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f91e00ae040 tx=0x7f91e0019230 comp rx=0 tx=0).stop 2024-04-17T10:28:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.374+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 shutdown_connections 2024-04-17T10:28:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.374+0000 7f91fd600640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f91d4066f90 0x7f91d4069450 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.374+0000 7f91fd600640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f91f8162dd0 0x7f91f8157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.374+0000 7f91fd600640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91f80cd490 0x7f91f815ff30 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.374+0000 7f91fd600640 1 --2- 172.21.15.67:0/2709764606 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91f80c4d60 0x7f91f81612f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.374+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 >> 172.21.15.67:0/2709764606 conn(0x7f91f80d1680 msgr2=0x7f91f8163920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:01.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.382+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 shutdown_connections 2024-04-17T10:28:01.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.382+0000 7f91fd600640 1 -- 172.21.15.67:0/2709764606 wait complete. 2024-04-17T10:28:01.407 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:28:01.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1dae800640 1 Processor -- start 2024-04-17T10:28:01.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1dae800640 1 -- start start 2024-04-17T10:28:01.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1dae800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da80cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1dae800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da80b4990 con 0x7f1da806e980 2024-04-17T10:28:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1dae800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da815ae90 con 0x7f1da80cce60 2024-04-17T10:28:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1dae800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80cd2c0 con 0x7f1da80aef60 2024-04-17T10:28:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1da806e980 0x7f1da80b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53754/0 (socket says 172.21.15.67:53754) 2024-04-17T10:28:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da7e00640 1 -- 172.21.15.67:0/400143583 learned_addr learned my addr 172.21.15.67:0/400143583 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da7400640 1 --2- 172.21.15.67:0/400143583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da80cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213527340 0 0) 0x7f1da0001120 con 0x7f1da806e980 2024-04-17T10:28:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003530 con 0x7f1da806e980 2024-04-17T10:28:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080902221 0 0) 0x7f1da815ae90 con 0x7f1da80cce60 2024-04-17T10:28:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1da0001120 con 0x7f1da80cce60 2024-04-17T10:28:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da7400640 1 -- 172.21.15.67:0/400143583 >> v1:172.21.15.86:6789/0 conn(0x7f1da80cce60 legacy=0x7f1da806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da7400640 1 -- 172.21.15.67:0/400143583 >> v1:172.21.15.67:6789/0 conn(0x7f1da806e980 legacy=0x7f1da80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 911958014 0 0) 0x7f1d8c003530 con 0x7f1da806e980 2024-04-17T10:28:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.562+0000 7f1da7400640 1 -- 172.21.15.67:0/400143583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da806ee80 con 0x7f1da80aef60 2024-04-17T10:28:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1da7400640 1 --2- 172.21.15.67:0/400143583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da80cc920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1da815b060 tx=0x7f1d9801c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ebde526ef29903c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d980ac070 con 0x7f1da80aef60 2024-04-17T10:28:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d980a6470 con 0x7f1da80aef60 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1da6a00640 1 -- 172.21.15.67:0/400143583 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0001850 con 0x7f1da80aef60 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 -- 172.21.15.67:0/400143583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 msgr2=0x7f1da80cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 --2- 172.21.15.67:0/400143583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da80cc920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1da815b060 tx=0x7f1d9801c820 comp rx=0 tx=0).stop 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 -- 172.21.15.67:0/400143583 shutdown_connections 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 --2- 172.21.15.67:0/400143583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da80cc920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 -- 172.21.15.67:0/400143583 >> 172.21.15.67:0/400143583 conn(0x7f1da80b1740 msgr2=0x7f1da80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 -- 172.21.15.67:0/400143583 shutdown_connections 2024-04-17T10:28:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 -- 172.21.15.67:0/400143583 wait complete. 2024-04-17T10:28:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.566+0000 7f1dae800640 1 Processor -- start 2024-04-17T10:28:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 -- start start 2024-04-17T10:28:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da806e980 0x7f1da81412d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da8143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80cce60 0x7f1da8143ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da8168550 con 0x7f1da80cce60 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da81683d0 con 0x7f1da806e980 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da81686d0 con 0x7f1da80aef60 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80cce60 0x7f1da8143ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80cce60 0x7f1da8143ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52366/0 (socket says 172.21.15.67:52366) 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 -- 172.21.15.67:0/2960339521 learned_addr learned my addr 172.21.15.67:0/2960339521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7400640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da806e980 0x7f1da81412d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1d9ea00640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da8143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:01.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 -- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 msgr2=0x7f1da8143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da8143570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 -- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da806e980 msgr2=0x7f1da81412d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da806e980 0x7f1da81412d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 -- 172.21.15.67:0/2960339521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da81484a0 con 0x7f1da80cce60 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1da7e00640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80cce60 0x7f1da8143ab0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1da00013c0 tx=0x7f1da000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1d9fe00640 1 -- 172.21.15.67:0/2960339521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0005cf0 con 0x7f1da80cce60 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1d9fe00640 1 -- 172.21.15.67:0/2960339521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da000ce30 con 0x7f1da80cce60 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1d9fe00640 1 -- 172.21.15.67:0/2960339521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da0007270 con 0x7f1da80cce60 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da8144240 con 0x7f1da80cce60 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da8144800 con 0x7f1da80cce60 2024-04-17T10:28:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.570+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da804ef30 con 0x7f1da80cce60 2024-04-17T10:28:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.574+0000 7f1d9fe00640 1 -- 172.21.15.67:0/2960339521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1da0007410 con 0x7f1da80cce60 2024-04-17T10:28:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.574+0000 7f1d9fe00640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d84066f60 0x7f1d84069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.574+0000 7f1d9fe00640 1 -- 172.21.15.67:0/2960339521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f1da0085a70 con 0x7f1da80cce60 2024-04-17T10:28:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.574+0000 7f1da7400640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d84066f60 0x7f1d84069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.574+0000 7f1da7400640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d84066f60 0x7f1d84069420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1d980ae000 tx=0x7f1d980b4040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:01.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.578+0000 7f1d9fe00640 1 -- 172.21.15.67:0/2960339521 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1da80b4990 con 0x7f1da80cce60 2024-04-17T10:28:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:01.774+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1da80b07c0 con 0x7f1da80cce60 2024-04-17T10:28:02.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.386+0000 7f1d9fe00640 1 -- 172.21.15.67:0/2960339521 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v24) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1da00562e0 con 0x7f1da80cce60 2024-04-17T10:28:02.382 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 7 and data pool 8 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d84066f60 msgr2=0x7f1d84069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d84066f60 0x7f1d84069420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1d980ae000 tx=0x7f1d980b4040 comp rx=0 tx=0).stop 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80cce60 msgr2=0x7f1da8143ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80cce60 0x7f1da8143ab0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1da00013c0 tx=0x7f1da000cbe0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 shutdown_connections 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d84066f60 0x7f1d84069420 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1da80cce60 0x7f1da8143ab0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80aef60 0x7f1da8143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 --2- 172.21.15.67:0/2960339521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da806e980 0x7f1da81412d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.394+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 >> 172.21.15.67:0/2960339521 conn(0x7f1da80b1740 msgr2=0x7f1da80b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:02.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.398+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 shutdown_connections 2024-04-17T10:28:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.398+0000 7f1dae800640 1 -- 172.21.15.67:0/2960339521 wait complete. 2024-04-17T10:28:02.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:02.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0417000640 1 Processor -- start 2024-04-17T10:28:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0417000640 1 -- start start 2024-04-17T10:28:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0417000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006f770 0x7f0410070470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0417000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f041006ffb0 0x7f041015ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0417000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04100cf330 con 0x7f041006fb50 2024-04-17T10:28:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0417000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f041009b190 con 0x7f041006ffb0 2024-04-17T10:28:02.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0417000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04100709b0 con 0x7f041006f770 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006f770 0x7f0410070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0416600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f041006ffb0 0x7f041015ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006f770 0x7f0410070470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:52936/0 (socket says 172.21.15.67:52936) 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0416600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f041006ffb0 0x7f041015ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39666/0 (socket says 172.21.15.67:39666) 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415200640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f041006fb50 0x7f041015f870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:32812/0 (socket says 172.21.15.67:32812) 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415c00640 1 -- 172.21.15.67:0/1372846375 learned_addr learned my addr 172.21.15.67:0/1372846375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415c00640 1 -- 172.21.15.67:0/1372846375 >> v2:172.21.15.86:3300/0 conn(0x7f041006ffb0 msgr2=0x7f041015ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415c00640 1 --2- 172.21.15.67:0/1372846375 >> v2:172.21.15.86:3300/0 conn(0x7f041006ffb0 0x7f041015ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415c00640 1 -- 172.21.15.67:0/1372846375 >> v1:172.21.15.86:6789/0 conn(0x7f041006fb50 legacy=0x7f041015f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0415c00640 1 -- 172.21.15.67:0/1372846375 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04100cf610 con 0x7f041006f770 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f040fe00640 1 -- 172.21.15.67:0/1372846375 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567438929 0 0) 0x7f04100cf330 con 0x7f041006fb50 2024-04-17T10:28:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.578+0000 7f0416600640 1 --2- 172.21.15.67:0/1372846375 >> v2:172.21.15.86:3300/0 conn(0x7f041006ffb0 0x7f041015ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:02.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0415c00640 1 --2- 172.21.15.67:0/1372846375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006f770 0x7f0410070470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0404009870 tx=0x7f040401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ea15d20ad4e98ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f040fe00640 1 -- 172.21.15.67:0/1372846375 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040b0070 con 0x7f041006f770 2024-04-17T10:28:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f040fe00640 1 -- 172.21.15.67:0/1372846375 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04040ab600 con 0x7f041006f770 2024-04-17T10:28:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f040fe00640 1 -- 172.21.15.67:0/1372846375 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040ab9d0 con 0x7f041006f770 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 -- 172.21.15.67:0/1372846375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006f770 msgr2=0x7f0410070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 --2- 172.21.15.67:0/1372846375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006f770 0x7f0410070470 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0404009870 tx=0x7f040401c4d0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 -- 172.21.15.67:0/1372846375 shutdown_connections 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 --2- 172.21.15.67:0/1372846375 >> v2:172.21.15.86:3300/0 conn(0x7f041006ffb0 0x7f041015ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 --2- 172.21.15.67:0/1372846375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006f770 0x7f0410070470 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 -- 172.21.15.67:0/1372846375 >> 172.21.15.67:0/1372846375 conn(0x7f04100d1630 msgr2=0x7f04100d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 -- 172.21.15.67:0/1372846375 shutdown_connections 2024-04-17T10:28:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.582+0000 7f0417000640 1 -- 172.21.15.67:0/1372846375 wait complete. 2024-04-17T10:28:02.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 Processor -- start 2024-04-17T10:28:02.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 -- start start 2024-04-17T10:28:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041006f770 0x7f04101f9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006fb50 0x7f04101fa290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041006ffb0 0x7f04101548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04101606d0 con 0x7f041006f770 2024-04-17T10:28:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0410071610 con 0x7f041006ffb0 2024-04-17T10:28:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04100cc890 con 0x7f041006fb50 2024-04-17T10:28:02.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041006f770 0x7f04101f9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041006f770 0x7f04101f9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52380/0 (socket says 172.21.15.67:52380) 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 -- 172.21.15.67:0/2040335359 learned_addr learned my addr 172.21.15.67:0/2040335359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415200640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006fb50 0x7f04101fa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0416600640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041006ffb0 0x7f04101548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 -- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006fb50 msgr2=0x7f04101fa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006fb50 0x7f04101fa290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 -- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041006ffb0 msgr2=0x7f04101548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041006ffb0 0x7f04101548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 -- 172.21.15.67:0/2040335359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041015b4a0 con 0x7f041006f770 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0415c00640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041006f770 0x7f04101f9d50 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0404009870 tx=0x7f040401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f040d600640 1 -- 172.21.15.67:0/2040335359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040b0040 con 0x7f041006f770 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f040d600640 1 -- 172.21.15.67:0/2040335359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0404004020 con 0x7f041006f770 2024-04-17T10:28:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f040d600640 1 -- 172.21.15.67:0/2040335359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404004410 con 0x7f041006f770 2024-04-17T10:28:02.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04101556c0 con 0x7f041006f770 2024-04-17T10:28:02.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0410155c80 con 0x7f041006f770 2024-04-17T10:28:02.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.586+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0410074dc0 con 0x7f041006f770 2024-04-17T10:28:02.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.590+0000 7f040d600640 1 -- 172.21.15.67:0/2040335359 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f04040be060 con 0x7f041006f770 2024-04-17T10:28:02.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.590+0000 7f040d600640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03f0066f60 0x7f03f0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.590+0000 7f040d600640 1 -- 172.21.15.67:0/2040335359 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f040411e6f0 con 0x7f041006f770 2024-04-17T10:28:02.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.590+0000 7f0415200640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03f0066f60 0x7f03f0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.590+0000 7f0415200640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03f0066f60 0x7f03f0069420 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f04100ca890 tx=0x7f03fc007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:02.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.594+0000 7f040d600640 1 -- 172.21.15.67:0/2040335359 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0404016150 con 0x7f041006f770 2024-04-17T10:28:02.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0410073670 con 0x7f041006f770 2024-04-17T10:28:02.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f040d600640 1 -- 172.21.15.67:0/2040335359 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16907 (secure 0 0 0) 0x7f04040ef570 con 0x7f041006f770 2024-04-17T10:28:02.752 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:02.752 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:01.776529+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:00.190460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:00.765109+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03f0066f60 msgr2=0x7f03f0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03f0066f60 0x7f03f0069420 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f04100ca890 tx=0x7f03fc007d60 comp rx=0 tx=0).stop 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041006f770 msgr2=0x7f04101f9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041006f770 0x7f04101f9d50 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0404009870 tx=0x7f040401ad10 comp rx=0 tx=0).stop 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 shutdown_connections 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03f0066f60 0x7f03f0069420 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f041006ffb0 0x7f04101548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f041006fb50 0x7f04101fa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 --2- 172.21.15.67:0/2040335359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f041006f770 0x7f04101f9d50 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 >> 172.21.15.67:0/2040335359 conn(0x7f04100d1630 msgr2=0x7f04100cfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:02.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 shutdown_connections 2024-04-17T10:28:02.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.758+0000 7f0417000640 1 -- 172.21.15.67:0/2040335359 wait complete. 2024-04-17T10:28:02.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.918+0000 7f3725800640 1 Processor -- start 2024-04-17T10:28:02.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.918+0000 7f3725800640 1 -- start start 2024-04-17T10:28:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3725800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37200cef30 0x7f37200aed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3725800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f372015a490 0x7f372006daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3725800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f372006dfe0 0x7f37200cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3725800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372015ac30 con 0x7f372015a490 2024-04-17T10:28:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3725800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f372015af10 con 0x7f372006dfe0 2024-04-17T10:28:02.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3725800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372015b1f0 con 0x7f37200cef30 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f372015a490 0x7f372006daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f372015a490 0x7f372006daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52386/0 (socket says 172.21.15.67:52386) 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 -- 172.21.15.67:0/3725660873 learned_addr learned my addr 172.21.15.67:0/3725660873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f371f400640 1 --2- 172.21.15.67:0/3725660873 >> v2:172.21.15.86:3300/0 conn(0x7f372006dfe0 0x7f37200cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f371ea00640 1 --2- 172.21.15.67:0/3725660873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37200cef30 0x7f37200aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 -- 172.21.15.67:0/3725660873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37200cef30 msgr2=0x7f37200aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 --2- 172.21.15.67:0/3725660873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37200cef30 0x7f37200aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 -- 172.21.15.67:0/3725660873 >> v2:172.21.15.86:3300/0 conn(0x7f372006dfe0 msgr2=0x7f37200cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 --2- 172.21.15.67:0/3725660873 >> v2:172.21.15.86:3300/0 conn(0x7f372006dfe0 0x7f37200cdf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.922+0000 7f3716000640 1 -- 172.21.15.67:0/3725660873 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372006d580 con 0x7f372015a490 2024-04-17T10:28:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3716000640 1 --2- 172.21.15.67:0/3725660873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f372006daa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f37200cd100 tx=0x7f370801e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb48ca5ee2ab0c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f371e000640 1 -- 172.21.15.67:0/3725660873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37080ad070 con 0x7f372015a490 2024-04-17T10:28:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f371e000640 1 -- 172.21.15.67:0/3725660873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3708004230 con 0x7f372015a490 2024-04-17T10:28:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f371e000640 1 -- 172.21.15.67:0/3725660873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37080a7660 con 0x7f372015a490 2024-04-17T10:28:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 -- 172.21.15.67:0/3725660873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 msgr2=0x7f372006daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 --2- 172.21.15.67:0/3725660873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f372006daa0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f37200cd100 tx=0x7f370801e9d0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 -- 172.21.15.67:0/3725660873 shutdown_connections 2024-04-17T10:28:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 --2- 172.21.15.67:0/3725660873 >> v2:172.21.15.86:3300/0 conn(0x7f372006dfe0 0x7f37200cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 --2- 172.21.15.67:0/3725660873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f372006daa0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 --2- 172.21.15.67:0/3725660873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37200cef30 0x7f37200aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 -- 172.21.15.67:0/3725660873 >> 172.21.15.67:0/3725660873 conn(0x7f372006ac90 msgr2=0x7f372006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 -- 172.21.15.67:0/3725660873 shutdown_connections 2024-04-17T10:28:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 -- 172.21.15.67:0/3725660873 wait complete. 2024-04-17T10:28:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 Processor -- start 2024-04-17T10:28:02.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.926+0000 7f3725800640 1 -- start start 2024-04-17T10:28:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372006dfe0 0x7f37200ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200cef30 0x7f37201535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f3720153af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37201513b0 con 0x7f372015a490 2024-04-17T10:28:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3720151230 con 0x7f37200cef30 2024-04-17T10:28:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37200cf520 con 0x7f372006dfe0 2024-04-17T10:28:02.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f3720153af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f3720153af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52392/0 (socket says 172.21.15.67:52392) 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 -- 172.21.15.67:0/2240591058 learned_addr learned my addr 172.21.15.67:0/2240591058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371ea00640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372006dfe0 0x7f37200ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3716000640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200cef30 0x7f37201535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 -- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372006dfe0 msgr2=0x7f37200ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372006dfe0 0x7f37200ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 -- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200cef30 msgr2=0x7f37201535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200cef30 0x7f37201535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 -- 172.21.15.67:0/2240591058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372014fa70 con 0x7f372015a490 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f371f400640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f3720153af0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f371800e880 tx=0x7f371800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3717400640 1 -- 172.21.15.67:0/2240591058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718017070 con 0x7f372015a490 2024-04-17T10:28:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3717400640 1 -- 172.21.15.67:0/2240591058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f371800c940 con 0x7f372015a490 2024-04-17T10:28:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3717400640 1 -- 172.21.15.67:0/2240591058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371800cd10 con 0x7f372015a490 2024-04-17T10:28:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372014fd50 con 0x7f372015a490 2024-04-17T10:28:02.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3720154fb0 con 0x7f372015a490 2024-04-17T10:28:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3720072d50 con 0x7f372015a490 2024-04-17T10:28:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.930+0000 7f3717400640 1 -- 172.21.15.67:0/2240591058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f371801a070 con 0x7f372015a490 2024-04-17T10:28:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.934+0000 7f3717400640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36fc066f60 0x7f36fc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.934+0000 7f3717400640 1 -- 172.21.15.67:0/2240591058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f37180850d0 con 0x7f372015a490 2024-04-17T10:28:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.934+0000 7f371ea00640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36fc066f60 0x7f36fc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:02.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.934+0000 7f371ea00640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36fc066f60 0x7f36fc069420 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f37100113e0 tx=0x7f371000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:02.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:02.938+0000 7f3717400640 1 -- 172.21.15.67:0/2240591058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f37180558f0 con 0x7f372015a490 2024-04-17T10:28:03.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.098+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3720165c00 con 0x7f372015a490 2024-04-17T10:28:03.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3717400640 1 -- 172.21.15.67:0/2240591058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16907 (secure 0 0 0) 0x7f3718059990 con 0x7f372015a490 2024-04-17T10:28:03.097 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:03.097 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":53,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:01.776529+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:00.190460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:00.765109+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36fc066f60 msgr2=0x7f36fc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36fc066f60 0x7f36fc069420 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f37100113e0 tx=0x7f371000b040 comp rx=0 tx=0).stop 2024-04-17T10:28:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 msgr2=0x7f3720153af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f3720153af0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f371800e880 tx=0x7f371800ed40 comp rx=0 tx=0).stop 2024-04-17T10:28:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 shutdown_connections 2024-04-17T10:28:03.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36fc066f60 0x7f36fc069420 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f372015a490 0x7f3720153af0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37200cef30 0x7f37201535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 --2- 172.21.15.67:0/2240591058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f372006dfe0 0x7f37200ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 >> 172.21.15.67:0/2240591058 conn(0x7f372006ac90 msgr2=0x7f37200b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:03.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 shutdown_connections 2024-04-17T10:28:03.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.102+0000 7f3725800640 1 -- 172.21.15.67:0/2240591058 wait complete. 2024-04-17T10:28:03.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:28:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49c1400640 1 Processor -- start 2024-04-17T10:28:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49c1400640 1 -- start start 2024-04-17T10:28:03.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49c1400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f49bc159440 0x7f49bc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:03.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49c1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49bc0713e0 0x7f49bc071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:03.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49c1400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49bc15cb30 con 0x7f49bc15be70 2024-04-17T10:28:03.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49c1400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49bc0a8650 con 0x7f49bc0713e0 2024-04-17T10:28:03.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49c1400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49bc15ce10 con 0x7f49bc159440 2024-04-17T10:28:03.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.266+0000 7f49b1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f49bc0713e0 0x7f49bc071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:03.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49bb200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f49bc15be70 0x7f49bc15c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53764/0 (socket says 172.21.15.67:53764) 2024-04-17T10:28:03.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49bb200640 1 -- 172.21.15.67:0/2335603042 learned_addr learned my addr 172.21.15.67:0/2335603042 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:03.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49ba800640 1 --2- 172.21.15.67:0/2335603042 >> v2:172.21.15.86:3300/0 conn(0x7f49bc159440 0x7f49bc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:03.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b1e00640 1 -- 172.21.15.67:0/2335603042 >> v2:172.21.15.86:3300/0 conn(0x7f49bc159440 msgr2=0x7f49bc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:03.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b1e00640 1 --2- 172.21.15.67:0/2335603042 >> v2:172.21.15.86:3300/0 conn(0x7f49bc159440 0x7f49bc0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b1e00640 1 -- 172.21.15.67:0/2335603042 >> v1:172.21.15.67:6789/0 conn(0x7f49bc15be70 legacy=0x7f49bc15c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:03.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b1e00640 1 -- 172.21.15.67:0/2335603042 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0cd610 con 0x7f49bc0713e0 2024-04-17T10:28:03.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b9e00640 1 -- 172.21.15.67:0/2335603042 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832484494 0 0) 0x7f49bc15cb30 con 0x7f49bc15be70 2024-04-17T10:28:03.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b1e00640 1 --2- 172.21.15.67:0/2335603042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc0713e0 0x7f49bc071870 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f49bc06e290 tx=0x7f49a401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51c223c3fadbc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:03.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b9e00640 1 -- 172.21.15.67:0/2335603042 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40b1070 con 0x7f49bc0713e0 2024-04-17T10:28:03.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b9e00640 1 -- 172.21.15.67:0/2335603042 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a40ac5f0 con 0x7f49bc0713e0 2024-04-17T10:28:03.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.270+0000 7f49b9e00640 1 -- 172.21.15.67:0/2335603042 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40ac9c0 con 0x7f49bc0713e0 2024-04-17T10:28:03.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- 172.21.15.67:0/2335603042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc0713e0 msgr2=0x7f49bc071870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:03.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 --2- 172.21.15.67:0/2335603042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc0713e0 0x7f49bc071870 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f49bc06e290 tx=0x7f49a401c480 comp rx=0 tx=0).stop 2024-04-17T10:28:03.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- 172.21.15.67:0/2335603042 shutdown_connections 2024-04-17T10:28:03.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 --2- 172.21.15.67:0/2335603042 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc0713e0 0x7f49bc071870 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 --2- 172.21.15.67:0/2335603042 >> v2:172.21.15.86:3300/0 conn(0x7f49bc159440 0x7f49bc0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- 172.21.15.67:0/2335603042 >> 172.21.15.67:0/2335603042 conn(0x7f49bc0b1740 msgr2=0x7f49bc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:03.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- 172.21.15.67:0/2335603042 shutdown_connections 2024-04-17T10:28:03.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- 172.21.15.67:0/2335603042 wait complete. 2024-04-17T10:28:03.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 Processor -- start 2024-04-17T10:28:03.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- start start 2024-04-17T10:28:03.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49bc0713e0 0x7f49bc0b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc159440 0x7f49bc15e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49bc15be70 0x7f49bc143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc15d9c0 con 0x7f49bc159440 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc15d840 con 0x7f49bc0713e0 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49c1400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49bc15db40 con 0x7f49bc15be70 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc159440 0x7f49bc15e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc159440 0x7f49bc15e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52416/0 (socket says 172.21.15.67:52416) 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 -- 172.21.15.67:0/3728095802 learned_addr learned my addr 172.21.15.67:0/3728095802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49ba800640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49bc0713e0 0x7f49bc0b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49bb200640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49bc15be70 0x7f49bc143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 -- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49bc15be70 msgr2=0x7f49bc143c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:03.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49bc15be70 0x7f49bc143c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 -- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49bc0713e0 msgr2=0x7f49bc0b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49bc0713e0 0x7f49bc0b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49b1e00640 1 -- 172.21.15.67:0/3728095802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49bc0b3490 con 0x7f49bc159440 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.274+0000 7f49bb200640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49bc15be70 0x7f49bc143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49b1e00640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc159440 0x7f49bc15e5f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f49a401ab30 tx=0x7f49a4019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49b3400640 1 -- 172.21.15.67:0/3728095802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a40b1040 con 0x7f49bc159440 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49b3400640 1 -- 172.21.15.67:0/3728095802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a4004030 con 0x7f49bc159440 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49b3400640 1 -- 172.21.15.67:0/3728095802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4006e20 con 0x7f49bc159440 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49bc0b3770 con 0x7f49bc159440 2024-04-17T10:28:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49bc152040 con 0x7f49bc159440 2024-04-17T10:28:03.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49bc04ef30 con 0x7f49bc159440 2024-04-17T10:28:03.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49b3400640 1 -- 172.21.15.67:0/3728095802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f49a4003620 con 0x7f49bc159440 2024-04-17T10:28:03.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49b3400640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4998066f60 0x7f4998069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49b3400640 1 -- 172.21.15.67:0/3728095802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f49a411ee30 con 0x7f49bc159440 2024-04-17T10:28:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.278+0000 7f49ba800640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4998066f60 0x7f4998069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.282+0000 7f49ba800640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4998066f60 0x7f4998069420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0ada40 tx=0x7f49ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:03.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.282+0000 7f49b3400640 1 -- 172.21.15.67:0/3728095802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f49a40b9020 con 0x7f49bc159440 2024-04-17T10:28:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:03.486+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f49bc15de70 con 0x7f49bc159440 2024-04-17T10:28:04.352 INFO:tasks.ceph.mds.d.smithi086.stderr:2024-04-17T10:28:04.355+0000 7f6885000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:28:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49b3400640 1 -- 172.21.15.67:0/3728095802 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v27) v1 ==== 118+0+0 (secure 0 0 0) 0x7f49a40a9070 con 0x7f49bc159440 2024-04-17T10:28:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4998066f60 msgr2=0x7f4998069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4998066f60 0x7f4998069420 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f49bc0ada40 tx=0x7f49ac008040 comp rx=0 tx=0).stop 2024-04-17T10:28:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc159440 msgr2=0x7f49bc15e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc159440 0x7f49bc15e5f0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f49a401ab30 tx=0x7f49a4019220 comp rx=0 tx=0).stop 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 shutdown_connections 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4998066f60 0x7f4998069420 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f49bc15be70 0x7f49bc143c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49bc159440 0x7f49bc15e5f0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 --2- 172.21.15.67:0/3728095802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f49bc0713e0 0x7f49bc0b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 >> 172.21.15.67:0/3728095802 conn(0x7f49bc0b1740 msgr2=0x7f49bc0acbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 shutdown_connections 2024-04-17T10:28:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.402+0000 7f49c1400640 1 -- 172.21.15.67:0/3728095802 wait complete. 2024-04-17T10:28:04.421 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:28:04.421 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:28:04.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.574+0000 7f8aace00640 1 Processor -- start 2024-04-17T10:28:04.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.574+0000 7f8aace00640 1 -- start start 2024-04-17T10:28:04.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aace00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:04.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aace00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa815cb30 con 0x7f8aa815be70 2024-04-17T10:28:04.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aace00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa80cd610 con 0x7f8aa8159440 2024-04-17T10:28:04.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aace00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa80a8650 con 0x7f8aa8071af0 2024-04-17T10:28:04.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aa6800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8aa815be70 0x7f8aa815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53770/0 (socket says 172.21.15.67:53770) 2024-04-17T10:28:04.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aa6800640 1 -- 172.21.15.67:0/1449038066 learned_addr learned my addr 172.21.15.67:0/1449038066 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8a9fe00640 1 --2- 172.21.15.67:0/1449038066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aa5400640 1 -- 172.21.15.67:0/1449038066 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1842653193 0 0) 0x7f8aa815cb30 con 0x7f8aa815be70 2024-04-17T10:28:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aa5400640 1 -- 172.21.15.67:0/1449038066 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a8c003530 con 0x7f8aa815be70 2024-04-17T10:28:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8a9fe00640 1 -- 172.21.15.67:0/1449038066 >> v1:172.21.15.86:6789/0 conn(0x7f8aa8159440 legacy=0x7f8aa80713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8aa5400640 1 -- 172.21.15.67:0/1449038066 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743134462 0 0) 0x7f8aa80cd610 con 0x7f8aa8159440 2024-04-17T10:28:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8a9fe00640 1 -- 172.21.15.67:0/1449038066 >> v1:172.21.15.67:6789/0 conn(0x7f8aa815be70 legacy=0x7f8aa815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.578+0000 7f8a9fe00640 1 -- 172.21.15.67:0/1449038066 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa80cd8f0 con 0x7f8aa8071af0 2024-04-17T10:28:04.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8a9fe00640 1 --2- 172.21.15.67:0/1449038066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa80ce0f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8a900097f0 tx=0x7f8a9001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ae96ad5f9252396 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:04.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aa5400640 1 -- 172.21.15.67:0/1449038066 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a900ab5f0 con 0x7f8aa8071af0 2024-04-17T10:28:04.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aa5400640 1 -- 172.21.15.67:0/1449038066 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a900b0070 con 0x7f8aa8071af0 2024-04-17T10:28:04.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aa5400640 1 -- 172.21.15.67:0/1449038066 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a900ab9b0 con 0x7f8aa8071af0 2024-04-17T10:28:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 -- 172.21.15.67:0/1449038066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 msgr2=0x7f8aa80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 --2- 172.21.15.67:0/1449038066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa80ce0f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8a900097f0 tx=0x7f8a9001c4e0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 -- 172.21.15.67:0/1449038066 shutdown_connections 2024-04-17T10:28:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 --2- 172.21.15.67:0/1449038066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa80ce0f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 -- 172.21.15.67:0/1449038066 >> 172.21.15.67:0/1449038066 conn(0x7f8aa80b1740 msgr2=0x7f8aa80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 -- 172.21.15.67:0/1449038066 shutdown_connections 2024-04-17T10:28:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 -- 172.21.15.67:0/1449038066 wait complete. 2024-04-17T10:28:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 Processor -- start 2024-04-17T10:28:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.582+0000 7f8aace00640 1 -- start start 2024-04-17T10:28:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa814da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8159440 0x7f8aa80adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa815be70 0x7f8aa8150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa815dc50 con 0x7f8aa8159440 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa815dad0 con 0x7f8aa815be70 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa815ddd0 con 0x7f8aa8071af0 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8159440 0x7f8aa80adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8159440 0x7f8aa80adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52426/0 (socket says 172.21.15.67:52426) 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 -- 172.21.15.67:0/2527039793 learned_addr learned my addr 172.21.15.67:0/2527039793 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aa5e00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa814da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:04.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aa6800640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa815be70 0x7f8aa8150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 -- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 msgr2=0x7f8aa814da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa814da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 -- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa815be70 msgr2=0x7f8aa8150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa815be70 0x7f8aa8150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 -- 172.21.15.67:0/2527039793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8156c70 con 0x7f8aa8159440 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9fe00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8159440 0x7f8aa80adf90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009740 tx=0x7f8a9001ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9e000640 1 -- 172.21.15.67:0/2527039793 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90004020 con 0x7f8aa8159440 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9e000640 1 -- 172.21.15.67:0/2527039793 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a900041c0 con 0x7f8aa8159440 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa8156f50 con 0x7f8aa8159440 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9e000640 1 -- 172.21.15.67:0/2527039793 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90003620 con 0x7f8aa8159440 2024-04-17T10:28:04.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa8157510 con 0x7f8aa8159440 2024-04-17T10:28:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa804ef30 con 0x7f8aa8159440 2024-04-17T10:28:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.586+0000 7f8a9e000640 1 -- 172.21.15.67:0/2527039793 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8a900037c0 con 0x7f8aa8159440 2024-04-17T10:28:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.590+0000 7f8a9e000640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8a84066f60 0x7f8a84069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.590+0000 7f8a9e000640 1 -- 172.21.15.67:0/2527039793 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a9011eba0 con 0x7f8aa8159440 2024-04-17T10:28:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.590+0000 7f8aa5e00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8a84066f60 0x7f8a84069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.590+0000 7f8aa5e00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8a84066f60 0x7f8a84069420 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a98000f70 tx=0x7f8a98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:04.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.594+0000 7f8a9e000640 1 -- 172.21.15.67:0/2527039793 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8a900c3070 con 0x7f8aa8159440 2024-04-17T10:28:04.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:04.794+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8aa80ae700 con 0x7f8aa8159440 2024-04-17T10:28:05.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.402+0000 7f8a9e000640 1 -- 172.21.15.67:0/2527039793 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8a900160d0 con 0x7f8aa8159440 2024-04-17T10:28:05.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8a84066f60 msgr2=0x7f8a84069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8a84066f60 0x7f8a84069420 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a98000f70 tx=0x7f8a98009040 comp rx=0 tx=0).stop 2024-04-17T10:28:05.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8159440 msgr2=0x7f8aa80adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8159440 0x7f8aa80adf90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8a90009740 tx=0x7f8a9001ad20 comp rx=0 tx=0).stop 2024-04-17T10:28:05.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 shutdown_connections 2024-04-17T10:28:05.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8a84066f60 0x7f8a84069420 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8aa815be70 0x7f8aa8150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8aa8159440 0x7f8aa80adf90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 --2- 172.21.15.67:0/2527039793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8aa8071af0 0x7f8aa814da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 >> 172.21.15.67:0/2527039793 conn(0x7f8aa80b1740 msgr2=0x7f8aa80af7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 shutdown_connections 2024-04-17T10:28:05.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.406+0000 7f8aace00640 1 -- 172.21.15.67:0/2527039793 wait complete. 2024-04-17T10:28:05.422 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:28:05.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.570+0000 7f5b2b000640 1 Processor -- start 2024-04-17T10:28:05.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.570+0000 7f5b2b000640 1 -- start start 2024-04-17T10:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b2b000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b24144980 0x7f5b2406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b2b000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5b2406eed0 0x7f5b2406b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b2b000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2406fbd0 con 0x7f5b24145080 2024-04-17T10:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b2b000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240707a0 con 0x7f5b24144980 2024-04-17T10:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b2b000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240ce020 con 0x7f5b2406eed0 2024-04-17T10:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b24144980 0x7f5b2406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5b24144980 0x7f5b2406ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52436/0 (socket says 172.21.15.67:52436) 2024-04-17T10:28:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b2a600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5b24145080 0x7f5b2415f390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53786/0 (socket says 172.21.15.67:53786) 2024-04-17T10:28:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29c00640 1 -- 172.21.15.67:0/519160532 learned_addr learned my addr 172.21.15.67:0/519160532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29200640 1 --2- 172.21.15.67:0/519160532 >> v2:172.21.15.86:3300/0 conn(0x7f5b2406eed0 0x7f5b2406b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29c00640 1 -- 172.21.15.67:0/519160532 >> v2:172.21.15.86:3300/0 conn(0x7f5b2406eed0 msgr2=0x7f5b2406b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29c00640 1 --2- 172.21.15.67:0/519160532 >> v2:172.21.15.86:3300/0 conn(0x7f5b2406eed0 0x7f5b2406b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29c00640 1 -- 172.21.15.67:0/519160532 >> v1:172.21.15.67:6789/0 conn(0x7f5b24145080 legacy=0x7f5b2415f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b29c00640 1 -- 172.21.15.67:0/519160532 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2406fe70 con 0x7f5b24144980 2024-04-17T10:28:05.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.574+0000 7f5b23e00640 1 -- 172.21.15.67:0/519160532 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393383765 0 0) 0x7f5b2406fbd0 con 0x7f5b24145080 2024-04-17T10:28:05.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b29c00640 1 --2- 172.21.15.67:0/519160532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24144980 0x7f5b2406ac90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5b240681e0 tx=0x7f5b1801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea28713461a25900 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b23e00640 1 -- 172.21.15.67:0/519160532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180b5070 con 0x7f5b24144980 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b23e00640 1 -- 172.21.15.67:0/519160532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b180a8a10 con 0x7f5b24144980 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b23e00640 1 -- 172.21.15.67:0/519160532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180a8de0 con 0x7f5b24144980 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 -- 172.21.15.67:0/519160532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24144980 msgr2=0x7f5b2406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 --2- 172.21.15.67:0/519160532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24144980 0x7f5b2406ac90 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5b240681e0 tx=0x7f5b1801ed10 comp rx=0 tx=0).stop 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 -- 172.21.15.67:0/519160532 shutdown_connections 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 --2- 172.21.15.67:0/519160532 >> v2:172.21.15.86:3300/0 conn(0x7f5b2406eed0 0x7f5b2406b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 --2- 172.21.15.67:0/519160532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b24144980 0x7f5b2406ac90 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 -- 172.21.15.67:0/519160532 >> 172.21.15.67:0/519160532 conn(0x7f5b2406e3b0 msgr2=0x7f5b2406e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:05.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 -- 172.21.15.67:0/519160532 shutdown_connections 2024-04-17T10:28:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 -- 172.21.15.67:0/519160532 wait complete. 2024-04-17T10:28:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 Processor -- start 2024-04-17T10:28:05.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.578+0000 7f5b2b000640 1 -- start start 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b2406eed0 0x7f5b240a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b24144980 0x7f5b2413afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b24145080 0x7f5b241436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24070c30 con 0x7f5b2406eed0 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24070430 con 0x7f5b24145080 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2406ba40 con 0x7f5b24144980 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b2406eed0 0x7f5b240a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b2406eed0 0x7f5b240a8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52450/0 (socket says 172.21.15.67:52450) 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 -- 172.21.15.67:0/2737565454 learned_addr learned my addr 172.21.15.67:0/2737565454 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29200640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b24144980 0x7f5b2413afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2a600640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b24145080 0x7f5b241436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 -- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b24144980 msgr2=0x7f5b2413afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b24144980 0x7f5b2413afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 -- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b24145080 msgr2=0x7f5b241436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b24145080 0x7f5b241436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 -- 172.21.15.67:0/2737565454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240a95a0 con 0x7f5b2406eed0 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2a600640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b24145080 0x7f5b241436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b29c00640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b2406eed0 0x7f5b240a8ec0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5b1801b920 tx=0x7f5b1801bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b21600640 1 -- 172.21.15.67:0/2737565454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b180b5040 con 0x7f5b2406eed0 2024-04-17T10:28:05.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b21600640 1 -- 172.21.15.67:0/2737565454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18004220 con 0x7f5b2406eed0 2024-04-17T10:28:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b240a9880 con 0x7f5b2406eed0 2024-04-17T10:28:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b21600640 1 -- 172.21.15.67:0/2737565454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18003620 con 0x7f5b2406eed0 2024-04-17T10:28:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240a9e40 con 0x7f5b2406eed0 2024-04-17T10:28:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24074dc0 con 0x7f5b2406eed0 2024-04-17T10:28:05.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.582+0000 7f5b21600640 1 -- 172.21.15.67:0/2737565454 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5b180c2060 con 0x7f5b2406eed0 2024-04-17T10:28:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.586+0000 7f5b21600640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5b04066f90 0x7f5b04069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.586+0000 7f5b21600640 1 -- 172.21.15.67:0/2737565454 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5b181219a0 con 0x7f5b2406eed0 2024-04-17T10:28:05.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.586+0000 7f5b29200640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5b04066f90 0x7f5b04069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.586+0000 7f5b29200640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5b04066f90 0x7f5b04069450 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5b24153840 tx=0x7f5b10009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:05.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.590+0000 7f5b21600640 1 -- 172.21.15.67:0/2737565454 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5b18018870 con 0x7f5b2406eed0 2024-04-17T10:28:05.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.782+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b2413f620 con 0x7f5b2406eed0 2024-04-17T10:28:05.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b21600640 1 -- 172.21.15.67:0/2737565454 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f5b180f26f0 con 0x7f5b2406eed0 2024-04-17T10:28:05.779 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:05.779 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:05.779 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5b04066f90 msgr2=0x7f5b04069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5b04066f90 0x7f5b04069450 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f5b24153840 tx=0x7f5b10009190 comp rx=0 tx=0).stop 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b2406eed0 msgr2=0x7f5b240a8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b2406eed0 0x7f5b240a8ec0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f5b1801b920 tx=0x7f5b1801bab0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 shutdown_connections 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5b04066f90 0x7f5b04069450 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5b24145080 0x7f5b241436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5b24144980 0x7f5b2413afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 --2- 172.21.15.67:0/2737565454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5b2406eed0 0x7f5b240a8ec0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 >> 172.21.15.67:0/2737565454 conn(0x7f5b2406e3b0 msgr2=0x7f5b240b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 shutdown_connections 2024-04-17T10:28:05.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.786+0000 7f5b2b000640 1 -- 172.21.15.67:0/2737565454 wait complete. 2024-04-17T10:28:05.801 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.946+0000 7fd10b200640 1 Processor -- start 2024-04-17T10:28:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.946+0000 7fd10b200640 1 -- start start 2024-04-17T10:28:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd10b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd10406e6e0 0x7fd1040ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd10b200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd1041441f0 0x7fd1040ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd10b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd10415ae40 con 0x7fd10406eac0 2024-04-17T10:28:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd10b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1040cf430 con 0x7fd10406e6e0 2024-04-17T10:28:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd10b200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd104071440 con 0x7fd1041441f0 2024-04-17T10:28:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd10406e6e0 0x7fd1040ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd10406e6e0 0x7fd1040ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52466/0 (socket says 172.21.15.67:52466) 2024-04-17T10:28:05.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 -- 172.21.15.67:0/2951453532 learned_addr learned my addr 172.21.15.67:0/2951453532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:05.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd109200640 1 --2- 172.21.15.67:0/2951453532 >> v2:172.21.15.86:3300/0 conn(0x7fd1041441f0 0x7fd1040ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 -- 172.21.15.67:0/2951453532 >> v2:172.21.15.86:3300/0 conn(0x7fd1041441f0 msgr2=0x7fd1040ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:05.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 --2- 172.21.15.67:0/2951453532 >> v2:172.21.15.86:3300/0 conn(0x7fd1041441f0 0x7fd1040ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 -- 172.21.15.67:0/2951453532 >> v1:172.21.15.67:6789/0 conn(0x7fd10406eac0 legacy=0x7fd104143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 -- 172.21.15.67:0/2951453532 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd104071720 con 0x7fd10406e6e0 2024-04-17T10:28:05.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd103e00640 1 --2- 172.21.15.67:0/2951453532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd1040ce9b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd1040ce3b0 tx=0x7fd0f401e670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6263a2d970964 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:05.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.950+0000 7fd102a00640 1 -- 172.21.15.67:0/2951453532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40ac5f0 con 0x7fd10406e6e0 2024-04-17T10:28:05.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd102a00640 1 -- 172.21.15.67:0/2951453532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f40b1070 con 0x7fd10406e6e0 2024-04-17T10:28:05.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd102a00640 1 -- 172.21.15.67:0/2951453532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40ac9b0 con 0x7fd10406e6e0 2024-04-17T10:28:05.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- 172.21.15.67:0/2951453532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 msgr2=0x7fd1040ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 --2- 172.21.15.67:0/2951453532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd1040ce9b0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd1040ce3b0 tx=0x7fd0f401e670 comp rx=0 tx=0).stop 2024-04-17T10:28:05.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- 172.21.15.67:0/2951453532 shutdown_connections 2024-04-17T10:28:05.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 --2- 172.21.15.67:0/2951453532 >> v2:172.21.15.86:3300/0 conn(0x7fd1041441f0 0x7fd1040ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 --2- 172.21.15.67:0/2951453532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd1040ce9b0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- 172.21.15.67:0/2951453532 >> 172.21.15.67:0/2951453532 conn(0x7fd10406ac90 msgr2=0x7fd10406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:05.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- 172.21.15.67:0/2951453532 shutdown_connections 2024-04-17T10:28:05.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- 172.21.15.67:0/2951453532 wait complete. 2024-04-17T10:28:05.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 Processor -- start 2024-04-17T10:28:05.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- start start 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd104138cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd10406eac0 0x7fd104134170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1041441f0 0x7fd104159ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10406dd00 con 0x7fd10406e6e0 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10406db80 con 0x7fd10406eac0 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd10b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10406de80 con 0x7fd1041441f0 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd103e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd104138cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd103e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd104138cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52472/0 (socket says 172.21.15.67:52472) 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.954+0000 7fd103e00640 1 -- 172.21.15.67:0/3653495811 learned_addr learned my addr 172.21.15.67:0/3653495811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:05.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd103e00640 1 -- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1041441f0 msgr2=0x7fd104159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd103e00640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1041441f0 0x7fd104159ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd103e00640 1 -- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd10406eac0 msgr2=0x7fd104134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:05.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd103e00640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd10406eac0 0x7fd104134170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:05.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd103e00640 1 -- 172.21.15.67:0/3653495811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1041349e0 con 0x7fd10406e6e0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd103e00640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd104138cf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd0f40ae040 tx=0x7fd0f401ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd0fbe00640 1 -- 172.21.15.67:0/3653495811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40bb070 con 0x7fd10406e6e0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd0fbe00640 1 -- 172.21.15.67:0/3653495811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f40b7660 con 0x7fd10406e6e0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd104134cc0 con 0x7fd10406e6e0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd0fbe00640 1 -- 172.21.15.67:0/3653495811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40b7a50 con 0x7fd10406e6e0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd104135280 con 0x7fd10406e6e0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd104072d50 con 0x7fd10406e6e0 2024-04-17T10:28:05.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd0fbe00640 1 -- 172.21.15.67:0/3653495811 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd0f40be070 con 0x7fd10406e6e0 2024-04-17T10:28:05.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd0fbe00640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0e4066f60 0x7fd0e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:05.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd0fbe00640 1 -- 172.21.15.67:0/3653495811 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd0f411f7f0 con 0x7fd10406e6e0 2024-04-17T10:28:05.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.958+0000 7fd103400640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0e4066f60 0x7fd0e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:05.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.962+0000 7fd103400640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0e4066f60 0x7fd0e4069420 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000f70 tx=0x7fd0f00079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:05.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:05.962+0000 7fd0fbe00640 1 -- 172.21.15.67:0/3653495811 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd0f4018320 con 0x7fd10406e6e0 2024-04-17T10:28:06.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.126+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd104074e30 con 0x7fd10406e6e0 2024-04-17T10:28:06.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.126+0000 7fd0fbe00640 1 -- 172.21.15.67:0/3653495811 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16894 (secure 0 0 0) 0x7fd0f40f0660 con 0x7fd10406e6e0 2024-04-17T10:28:06.121 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:06.121 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":54,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:03.390201+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:00.190460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:00.765109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0e4066f60 msgr2=0x7fd0e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0e4066f60 0x7fd0e4069420 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000f70 tx=0x7fd0f00079e0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 msgr2=0x7fd104138cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd104138cf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd0f40ae040 tx=0x7fd0f401ceb0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 shutdown_connections 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0e4066f60 0x7fd0e4069420 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1041441f0 0x7fd104159ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd10406eac0 0x7fd104134170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 --2- 172.21.15.67:0/3653495811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd10406e6e0 0x7fd104138cf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 >> 172.21.15.67:0/3653495811 conn(0x7fd10406ac90 msgr2=0x7fd10413f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 shutdown_connections 2024-04-17T10:28:06.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.130+0000 7fd10b200640 1 -- 172.21.15.67:0/3653495811 wait complete. 2024-04-17T10:28:06.142 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:28:06.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:28:06.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dd600640 1 Processor -- start 2024-04-17T10:28:06.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dd600640 1 -- start start 2024-04-17T10:28:06.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dd600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb4d80ce520 0x7fb4d8168830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d815dd30 con 0x7fb4d815d950 2024-04-17T10:28:06.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dd600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d8161640 con 0x7fb4d806b3a0 2024-04-17T10:28:06.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dd600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d8161920 con 0x7fb4d80ce520 2024-04-17T10:28:06.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4d7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4d815d950 0x7fb4d806ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53808/0 (socket says 172.21.15.67:53808) 2024-04-17T10:28:06.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4d7e00640 1 -- 172.21.15.67:0/166730893 learned_addr learned my addr 172.21.15.67:0/166730893 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:06.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dcc00640 1 --2- 172.21.15.67:0/166730893 >> v2:172.21.15.86:3300/0 conn(0x7fb4d80ce520 0x7fb4d8168830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750363435 0 0) 0x7fb4d815dd30 con 0x7fb4d815d950 2024-04-17T10:28:06.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4bc003530 con 0x7fb4d815d950 2024-04-17T10:28:06.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045460956 0 0) 0x7fb4d8161640 con 0x7fb4d806b3a0 2024-04-17T10:28:06.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d815dd30 con 0x7fb4d806b3a0 2024-04-17T10:28:06.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 550041326 0 0) 0x7fb4bc003530 con 0x7fb4d815d950 2024-04-17T10:28:06.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dcc00640 1 -- 172.21.15.67:0/166730893 >> v1:172.21.15.86:6789/0 conn(0x7fb4d806b3a0 legacy=0x7fb4d8070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dcc00640 1 -- 172.21.15.67:0/166730893 >> v1:172.21.15.67:6789/0 conn(0x7fb4d815d950 legacy=0x7fb4d806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.290+0000 7fb4dcc00640 1 -- 172.21.15.67:0/166730893 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d8161c00 con 0x7fb4d80ce520 2024-04-17T10:28:06.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4dcc00640 1 --2- 172.21.15.67:0/166730893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 0x7fb4d8168830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009d90 tx=0x7fb4d001c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e01164142b40e1c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:06.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00b2070 con 0x7fb4d80ce520 2024-04-17T10:28:06.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d00a55e0 con 0x7fb4d80ce520 2024-04-17T10:28:06.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4d7400640 1 -- 172.21.15.67:0/166730893 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c8001680 con 0x7fb4d80ce520 2024-04-17T10:28:06.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4dd600640 1 -- 172.21.15.67:0/166730893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 msgr2=0x7fb4d8168830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4dd600640 1 --2- 172.21.15.67:0/166730893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 0x7fb4d8168830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009d90 tx=0x7fb4d001c890 comp rx=0 tx=0).stop 2024-04-17T10:28:06.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4dd600640 1 -- 172.21.15.67:0/166730893 shutdown_connections 2024-04-17T10:28:06.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4dd600640 1 --2- 172.21.15.67:0/166730893 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 0x7fb4d8168830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.294+0000 7fb4dd600640 1 -- 172.21.15.67:0/166730893 >> 172.21.15.67:0/166730893 conn(0x7fb4d80b23f0 msgr2=0x7fb4d80b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:06.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- 172.21.15.67:0/166730893 shutdown_connections 2024-04-17T10:28:06.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- 172.21.15.67:0/166730893 wait complete. 2024-04-17T10:28:06.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 Processor -- start 2024-04-17T10:28:06.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- start start 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d806b3a0 0x7fb4d81532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 0x7fb4d80ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d815d950 0x7fb4d80ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d816b920 con 0x7fb4d815d950 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d816b7a0 con 0x7fb4d80ce520 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d816baa0 con 0x7fb4d806b3a0 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d815d950 0x7fb4d80ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d815d950 0x7fb4d80ca660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52486/0 (socket says 172.21.15.67:52486) 2024-04-17T10:28:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 -- 172.21.15.67:0/3369568043 learned_addr learned my addr 172.21.15.67:0/3369568043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:06.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 -- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d806b3a0 msgr2=0x7fb4d81532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d806b3a0 0x7fb4d81532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 -- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 msgr2=0x7fb4d80ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 0x7fb4d80ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 -- 172.21.15.67:0/3369568043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d806b910 con 0x7fb4d815d950 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dcc00640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d815d950 0x7fb4d80ca660 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00b0040 tx=0x7fb4d0019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4d4c00640 1 -- 172.21.15.67:0/3369568043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00c0070 con 0x7fb4d815d950 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4d4c00640 1 -- 172.21.15.67:0/3369568043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d0003bc0 con 0x7fb4d815d950 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4d4c00640 1 -- 172.21.15.67:0/3369568043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d00b2040 con 0x7fb4d815d950 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d8153800 con 0x7fb4d815d950 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.298+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d80cb8a0 con 0x7fb4d815d950 2024-04-17T10:28:06.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.302+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d814cba0 con 0x7fb4d815d950 2024-04-17T10:28:06.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.302+0000 7fb4d4c00640 1 -- 172.21.15.67:0/3369568043 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb4d0004030 con 0x7fb4d815d950 2024-04-17T10:28:06.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.302+0000 7fb4d4c00640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4b4066f50 0x7fb4b4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.302+0000 7fb4d4c00640 1 -- 172.21.15.67:0/3369568043 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4d011f550 con 0x7fb4d815d950 2024-04-17T10:28:06.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.302+0000 7fb4d7e00640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4b4066f50 0x7fb4b4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.302+0000 7fb4d7e00640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4b4066f50 0x7fb4b4069410 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8001390 tx=0x7fb4c8001090 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:06.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.306+0000 7fb4d4c00640 1 -- 172.21.15.67:0/3369568043 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb4d0016480 con 0x7fb4d815d950 2024-04-17T10:28:06.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.466+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4d8074dc0 con 0x7fb4b4066f50 2024-04-17T10:28:06.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.474+0000 7fb4d4c00640 1 -- 172.21.15.67:0/3369568043 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7fb4d8074dc0 con 0x7fb4b4066f50 2024-04-17T10:28:06.472 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:06.473 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:28:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4b4066f50 msgr2=0x7fb4b4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4b4066f50 0x7fb4b4069410 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8001390 tx=0x7fb4c8001090 comp rx=0 tx=0).stop 2024-04-17T10:28:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d815d950 msgr2=0x7fb4d80ca660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d815d950 0x7fb4d80ca660 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00b0040 tx=0x7fb4d0019630 comp rx=0 tx=0).stop 2024-04-17T10:28:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 shutdown_connections 2024-04-17T10:28:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4b4066f50 0x7fb4b4069410 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4d815d950 0x7fb4d80ca660 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4d80ce520 0x7fb4d80ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 --2- 172.21.15.67:0/3369568043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4d806b3a0 0x7fb4d81532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 >> 172.21.15.67:0/3369568043 conn(0x7fb4d80b23f0 msgr2=0x7fb4d80b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 shutdown_connections 2024-04-17T10:28:06.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.478+0000 7fb4dd600640 1 -- 172.21.15.67:0/3369568043 wait complete. 2024-04-17T10:28:06.492 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-17T10:28:06.177959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315112,"kb_used_data":5980,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754659608,"statfs":{"total":773094113280,"available":772771438592,"internally_reserved":0,"allocated":6123520,"data_stored":3388317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.817202"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380959+0000","last_change":"2024-04-17T10:28:01.380959+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380959+0000","last_undegraded":"2024-04-17T10:28:01.380959+0000","last_fullsized":"2024-04-17T10:28:01.380959+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376444+0000","last_change":"2024-04-17T10:28:01.376444+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376444+0000","last_undegraded":"2024-04-17T10:28:01.376444+0000","last_fullsized":"2024-04-17T10:28:01.376444+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378464+0000","last_change":"2024-04-17T10:28:01.378464+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378464+0000","last_undegraded":"2024-04-17T10:28:01.378464+0000","last_fullsized":"2024-04-17T10:28:01.378464+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377387+0000","last_change":"2024-04-17T10:28:01.377387+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377387+0000","last_undegraded":"2024-04-17T10:28:01.377387+0000","last_fullsized":"2024-04-17T10:28:01.377387+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374044+0000","last_change":"2024-04-17T10:28:01.374044+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374044+0000","last_undegraded":"2024-04-17T10:28:01.374044+0000","last_fullsized":"2024-04-17T10:28:01.374044+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378669+0000","last_change":"2024-04-17T10:28:01.378669+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378669+0000","last_undegraded":"2024-04-17T10:28:01.378669+0000","last_fullsized":"2024-04-17T10:28:01.378669+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377863+0000","last_change":"2024-04-17T10:28:01.377863+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377863+0000","last_undegraded":"2024-04-17T10:28:01.377863+0000","last_fullsized":"2024-04-17T10:28:01.377863+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370398+0000","last_change":"2024-04-17T10:28:01.370398+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370398+0000","last_undegraded":"2024-04-17T10:28:01.370398+0000","last_fullsized":"2024-04-17T10:28:01.370398+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371867+0000","last_change":"2024-04-17T10:28:01.371867+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371867+0000","last_undegraded":"2024-04-17T10:28:01.371867+0000","last_fullsized":"2024-04-17T10:28:01.371867+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379871+0000","last_change":"2024-04-17T10:28:01.379871+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379871+0000","last_undegraded":"2024-04-17T10:28:01.379871+0000","last_fullsized":"2024-04-17T10:28:01.379871+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379507+0000","last_change":"2024-04-17T10:28:01.379507+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379507+0000","last_undegraded":"2024-04-17T10:28:01.379507+0000","last_fullsized":"2024-04-17T10:28:01.379507+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371184+0000","last_change":"2024-04-17T10:28:01.371184+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371184+0000","last_undegraded":"2024-04-17T10:28:01.371184+0000","last_fullsized":"2024-04-17T10:28:01.371184+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381984+0000","last_change":"2024-04-17T10:28:01.381984+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381984+0000","last_undegraded":"2024-04-17T10:28:01.381984+0000","last_fullsized":"2024-04-17T10:28:01.381984+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377452+0000","last_change":"2024-04-17T10:28:01.377452+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377452+0000","last_undegraded":"2024-04-17T10:28:01.377452+0000","last_fullsized":"2024-04-17T10:28:01.377452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376914+0000","last_change":"2024-04-17T10:28:01.376914+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376914+0000","last_undegraded":"2024-04-17T10:28:01.376914+0000","last_fullsized":"2024-04-17T10:28:01.376914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376339+0000","last_change":"2024-04-17T10:28:01.376339+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376339+0000","last_undegraded":"2024-04-17T10:28:01.376339+0000","last_fullsized":"2024-04-17T10:28:01.376339+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379944+0000","last_change":"2024-04-17T10:28:01.379944+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379944+0000","last_undegraded":"2024-04-17T10:28:01.379944+0000","last_fullsized":"2024-04-17T10:28:01.379944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376011+0000","last_change":"2024-04-17T10:28:01.376011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376011+0000","last_undegraded":"2024-04-17T10:28:01.376011+0000","last_fullsized":"2024-04-17T10:28:01.376011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377320+0000","last_change":"2024-04-17T10:28:01.377320+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377320+0000","last_undegraded":"2024-04-17T10:28:01.377320+0000","last_fullsized":"2024-04-17T10:28:01.377320+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369232+0000","last_change":"2024-04-17T10:28:01.369232+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369232+0000","last_undegraded":"2024-04-17T10:28:01.369232+0000","last_fullsized":"2024-04-17T10:28:01.369232+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367428+0000","last_change":"2024-04-17T10:28:01.367428+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367428+0000","last_undegraded":"2024-04-17T10:28:01.367428+0000","last_fullsized":"2024-04-17T10:28:01.367428+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370644+0000","last_change":"2024-04-17T10:28:01.370644+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370644+0000","last_undegraded":"2024-04-17T10:28:01.370644+0000","last_fullsized":"2024-04-17T10:28:01.370644+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379078+0000","last_change":"2024-04-17T10:28:01.379078+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379078+0000","last_undegraded":"2024-04-17T10:28:01.379078+0000","last_fullsized":"2024-04-17T10:28:01.379078+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380365+0000","last_change":"2024-04-17T10:28:01.380365+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380365+0000","last_undegraded":"2024-04-17T10:28:01.380365+0000","last_fullsized":"2024-04-17T10:28:01.380365+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374358+0000","last_change":"2024-04-17T10:28:01.374358+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374358+0000","last_undegraded":"2024-04-17T10:28:01.374358+0000","last_fullsized":"2024-04-17T10:28:01.374358+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.383292+0000","last_change":"2024-04-17T10:28:01.383292+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.383292+0000","last_undegraded":"2024-04-17T10:28:01.383292+0000","last_fullsized":"2024-04-17T10:28:01.383292+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367828+0000","last_change":"2024-04-17T10:28:01.367828+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367828+0000","last_undegraded":"2024-04-17T10:28:01.367828+0000","last_fullsized":"2024-04-17T10:28:01.367828+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377527+0000","last_change":"2024-04-17T10:28:01.377527+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377527+0000","last_undegraded":"2024-04-17T10:28:01.377527+0000","last_fullsized":"2024-04-17T10:28:01.377527+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379750+0000","last_change":"2024-04-17T10:28:01.379750+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379750+0000","last_undegraded":"2024-04-17T10:28:01.379750+0000","last_fullsized":"2024-04-17T10:28:01.379750+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376427+0000","last_change":"2024-04-17T10:28:01.376427+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376427+0000","last_undegraded":"2024-04-17T10:28:01.376427+0000","last_fullsized":"2024-04-17T10:28:01.376427+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372163+0000","last_change":"2024-04-17T10:28:01.372163+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372163+0000","last_undegraded":"2024-04-17T10:28:01.372163+0000","last_fullsized":"2024-04-17T10:28:01.372163+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377077+0000","last_change":"2024-04-17T10:28:01.377077+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377077+0000","last_undegraded":"2024-04-17T10:28:01.377077+0000","last_fullsized":"2024-04-17T10:28:01.377077+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384066+0000","last_change":"2024-04-17T10:28:01.380916+0000","last_active":"2024-04-17T10:28:01.384066+0000","last_peered":"2024-04-17T10:28:01.384066+0000","last_clean":"2024-04-17T10:28:01.384066+0000","last_became_active":"2024-04-17T10:28:01.380635+0000","last_became_peered":"2024-04-17T10:28:01.380635+0000","last_unstale":"2024-04-17T10:28:01.384066+0000","last_undegraded":"2024-04-17T10:28:01.384066+0000","last_fullsized":"2024-04-17T10:28:01.384066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:47:03.828693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371180+0000","last_change":"2024-04-17T10:28:01.371180+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371180+0000","last_undegraded":"2024-04-17T10:28:01.371180+0000","last_fullsized":"2024-04-17T10:28:01.371180+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394963+0000","last_change":"2024-04-17T10:28:01.391871+0000","last_active":"2024-04-17T10:28:01.394963+0000","last_peered":"2024-04-17T10:28:01.394963+0000","last_clean":"2024-04-17T10:28:01.394963+0000","last_became_active":"2024-04-17T10:28:01.391540+0000","last_became_peered":"2024-04-17T10:28:01.391540+0000","last_unstale":"2024-04-17T10:28:01.394963+0000","last_undegraded":"2024-04-17T10:28:01.394963+0000","last_fullsized":"2024-04-17T10:28:01.394963+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:03:16.384430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367514+0000","last_change":"2024-04-17T10:28:01.367514+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367514+0000","last_undegraded":"2024-04-17T10:28:01.367514+0000","last_fullsized":"2024-04-17T10:28:01.367514+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394726+0000","last_change":"2024-04-17T10:28:01.392616+0000","last_active":"2024-04-17T10:28:01.394726+0000","last_peered":"2024-04-17T10:28:01.394726+0000","last_clean":"2024-04-17T10:28:01.394726+0000","last_became_active":"2024-04-17T10:28:01.391986+0000","last_became_peered":"2024-04-17T10:28:01.391986+0000","last_unstale":"2024-04-17T10:28:01.394726+0000","last_undegraded":"2024-04-17T10:28:01.394726+0000","last_fullsized":"2024-04-17T10:28:01.394726+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371610+0000","last_change":"2024-04-17T10:28:01.371610+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371610+0000","last_undegraded":"2024-04-17T10:28:01.371610+0000","last_fullsized":"2024-04-17T10:28:01.371610+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383990+0000","last_change":"2024-04-17T10:28:01.380404+0000","last_active":"2024-04-17T10:28:01.383990+0000","last_peered":"2024-04-17T10:28:01.383990+0000","last_clean":"2024-04-17T10:28:01.383990+0000","last_became_active":"2024-04-17T10:28:01.380065+0000","last_became_peered":"2024-04-17T10:28:01.380065+0000","last_unstale":"2024-04-17T10:28:01.383990+0000","last_undegraded":"2024-04-17T10:28:01.383990+0000","last_fullsized":"2024-04-17T10:28:01.383990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:12.355937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375461+0000","last_change":"2024-04-17T10:28:01.375461+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375461+0000","last_undegraded":"2024-04-17T10:28:01.375461+0000","last_fullsized":"2024-04-17T10:28:01.375461+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399062+0000","last_change":"2024-04-17T10:28:01.395609+0000","last_active":"2024-04-17T10:28:01.399062+0000","last_peered":"2024-04-17T10:28:01.399062+0000","last_clean":"2024-04-17T10:28:01.399062+0000","last_became_active":"2024-04-17T10:28:01.395267+0000","last_became_peered":"2024-04-17T10:28:01.395267+0000","last_unstale":"2024-04-17T10:28:01.399062+0000","last_undegraded":"2024-04-17T10:28:01.399062+0000","last_fullsized":"2024-04-17T10:28:01.399062+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:42.740201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377612+0000","last_change":"2024-04-17T10:28:01.377612+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377612+0000","last_undegraded":"2024-04-17T10:28:01.377612+0000","last_fullsized":"2024-04-17T10:28:01.377612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383876+0000","last_change":"2024-04-17T10:28:01.378483+0000","last_active":"2024-04-17T10:28:01.383876+0000","last_peered":"2024-04-17T10:28:01.383876+0000","last_clean":"2024-04-17T10:28:01.383876+0000","last_became_active":"2024-04-17T10:28:01.378140+0000","last_became_peered":"2024-04-17T10:28:01.378140+0000","last_unstale":"2024-04-17T10:28:01.383876+0000","last_undegraded":"2024-04-17T10:28:01.383876+0000","last_fullsized":"2024-04-17T10:28:01.383876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371666+0000","last_change":"2024-04-17T10:28:01.371666+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371666+0000","last_undegraded":"2024-04-17T10:28:01.371666+0000","last_fullsized":"2024-04-17T10:28:01.371666+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383084+0000","last_change":"2024-04-17T10:28:01.378144+0000","last_active":"2024-04-17T10:28:01.383084+0000","last_peered":"2024-04-17T10:28:01.383084+0000","last_clean":"2024-04-17T10:28:01.383084+0000","last_became_active":"2024-04-17T10:28:01.377879+0000","last_became_peered":"2024-04-17T10:28:01.377879+0000","last_unstale":"2024-04-17T10:28:01.383084+0000","last_undegraded":"2024-04-17T10:28:01.383084+0000","last_fullsized":"2024-04-17T10:28:01.383084+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:02.479603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378313+0000","last_change":"2024-04-17T10:28:01.378313+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378313+0000","last_undegraded":"2024-04-17T10:28:01.378313+0000","last_fullsized":"2024-04-17T10:28:01.378313+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381525+0000","last_change":"2024-04-17T10:28:01.376007+0000","last_active":"2024-04-17T10:28:01.381525+0000","last_peered":"2024-04-17T10:28:01.381525+0000","last_clean":"2024-04-17T10:28:01.381525+0000","last_became_active":"2024-04-17T10:28:01.375098+0000","last_became_peered":"2024-04-17T10:28:01.375098+0000","last_unstale":"2024-04-17T10:28:01.381525+0000","last_undegraded":"2024-04-17T10:28:01.381525+0000","last_fullsized":"2024-04-17T10:28:01.381525+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372934+0000","last_change":"2024-04-17T10:28:01.372934+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372934+0000","last_undegraded":"2024-04-17T10:28:01.372934+0000","last_fullsized":"2024-04-17T10:28:01.372934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394069+0000","last_change":"2024-04-17T10:28:01.390159+0000","last_active":"2024-04-17T10:28:01.394069+0000","last_peered":"2024-04-17T10:28:01.394069+0000","last_clean":"2024-04-17T10:28:01.394069+0000","last_became_active":"2024-04-17T10:28:01.389874+0000","last_became_peered":"2024-04-17T10:28:01.389874+0000","last_unstale":"2024-04-17T10:28:01.394069+0000","last_undegraded":"2024-04-17T10:28:01.394069+0000","last_fullsized":"2024-04-17T10:28:01.394069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:59.077053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372104+0000","last_change":"2024-04-17T10:28:01.372104+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372104+0000","last_undegraded":"2024-04-17T10:28:01.372104+0000","last_fullsized":"2024-04-17T10:28:01.372104+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396819+0000","last_change":"2024-04-17T10:28:01.396322+0000","last_active":"2024-04-17T10:28:01.396819+0000","last_peered":"2024-04-17T10:28:01.396819+0000","last_clean":"2024-04-17T10:28:01.396819+0000","last_became_active":"2024-04-17T10:28:01.396090+0000","last_became_peered":"2024-04-17T10:28:01.396090+0000","last_unstale":"2024-04-17T10:28:01.396819+0000","last_undegraded":"2024-04-17T10:28:01.396819+0000","last_fullsized":"2024-04-17T10:28:01.396819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394086+0000","last_change":"2024-04-17T10:28:01.390653+0000","last_active":"2024-04-17T10:28:01.394086+0000","last_peered":"2024-04-17T10:28:01.394086+0000","last_clean":"2024-04-17T10:28:01.394086+0000","last_became_active":"2024-04-17T10:28:01.390327+0000","last_became_peered":"2024-04-17T10:28:01.390327+0000","last_unstale":"2024-04-17T10:28:01.394086+0000","last_undegraded":"2024-04-17T10:28:01.394086+0000","last_fullsized":"2024-04-17T10:28:01.394086+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:12.074317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373570+0000","last_change":"2024-04-17T10:28:01.373570+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373570+0000","last_undegraded":"2024-04-17T10:28:01.373570+0000","last_fullsized":"2024-04-17T10:28:01.373570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.393035+0000","last_change":"2024-04-17T10:28:01.390744+0000","last_active":"2024-04-17T10:28:01.393035+0000","last_peered":"2024-04-17T10:28:01.393035+0000","last_clean":"2024-04-17T10:28:01.393035+0000","last_became_active":"2024-04-17T10:28:01.390487+0000","last_became_peered":"2024-04-17T10:28:01.390487+0000","last_unstale":"2024-04-17T10:28:01.393035+0000","last_undegraded":"2024-04-17T10:28:01.393035+0000","last_fullsized":"2024-04-17T10:28:01.393035+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:21.812637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371710+0000","last_change":"2024-04-17T10:28:01.371710+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371710+0000","last_undegraded":"2024-04-17T10:28:01.371710+0000","last_fullsized":"2024-04-17T10:28:01.371710+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399042+0000","last_change":"2024-04-17T10:28:01.388726+0000","last_active":"2024-04-17T10:28:01.399042+0000","last_peered":"2024-04-17T10:28:01.399042+0000","last_clean":"2024-04-17T10:28:01.399042+0000","last_became_active":"2024-04-17T10:28:01.388462+0000","last_became_peered":"2024-04-17T10:28:01.388462+0000","last_unstale":"2024-04-17T10:28:01.399042+0000","last_undegraded":"2024-04-17T10:28:01.399042+0000","last_fullsized":"2024-04-17T10:28:01.399042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376893+0000","last_change":"2024-04-17T10:28:01.376893+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376893+0000","last_undegraded":"2024-04-17T10:28:01.376893+0000","last_fullsized":"2024-04-17T10:28:01.376893+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.390536+0000","last_change":"2024-04-17T10:28:01.388333+0000","last_active":"2024-04-17T10:28:01.390536+0000","last_peered":"2024-04-17T10:28:01.390536+0000","last_clean":"2024-04-17T10:28:01.390536+0000","last_became_active":"2024-04-17T10:28:01.387931+0000","last_became_peered":"2024-04-17T10:28:01.387931+0000","last_unstale":"2024-04-17T10:28:01.390536+0000","last_undegraded":"2024-04-17T10:28:01.390536+0000","last_fullsized":"2024-04-17T10:28:01.390536+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:28:46.889727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372115+0000","last_change":"2024-04-17T10:28:01.372115+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372115+0000","last_undegraded":"2024-04-17T10:28:01.372115+0000","last_fullsized":"2024-04-17T10:28:01.372115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388876+0000","last_change":"2024-04-17T10:28:01.387372+0000","last_active":"2024-04-17T10:28:01.388876+0000","last_peered":"2024-04-17T10:28:01.388876+0000","last_clean":"2024-04-17T10:28:01.388876+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.388876+0000","last_undegraded":"2024-04-17T10:28:01.388876+0000","last_fullsized":"2024-04-17T10:28:01.388876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:31.085374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376533+0000","last_change":"2024-04-17T10:28:01.376533+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376533+0000","last_undegraded":"2024-04-17T10:28:01.376533+0000","last_fullsized":"2024-04-17T10:28:01.376533+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391514+0000","last_change":"2024-04-17T10:28:01.390499+0000","last_active":"2024-04-17T10:28:01.391514+0000","last_peered":"2024-04-17T10:28:01.391514+0000","last_clean":"2024-04-17T10:28:01.391514+0000","last_became_active":"2024-04-17T10:28:01.390208+0000","last_became_peered":"2024-04-17T10:28:01.390208+0000","last_unstale":"2024-04-17T10:28:01.391514+0000","last_undegraded":"2024-04-17T10:28:01.391514+0000","last_fullsized":"2024-04-17T10:28:01.391514+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:44.289510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369897+0000","last_change":"2024-04-17T10:28:01.369897+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369897+0000","last_undegraded":"2024-04-17T10:28:01.369897+0000","last_fullsized":"2024-04-17T10:28:01.369897+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398772+0000","last_change":"2024-04-17T10:28:01.385970+0000","last_active":"2024-04-17T10:28:01.398772+0000","last_peered":"2024-04-17T10:28:01.398772+0000","last_clean":"2024-04-17T10:28:01.398772+0000","last_became_active":"2024-04-17T10:28:01.385447+0000","last_became_peered":"2024-04-17T10:28:01.385447+0000","last_unstale":"2024-04-17T10:28:01.398772+0000","last_undegraded":"2024-04-17T10:28:01.398772+0000","last_fullsized":"2024-04-17T10:28:01.398772+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378500+0000","last_change":"2024-04-17T10:28:01.378500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378500+0000","last_undegraded":"2024-04-17T10:28:01.378500+0000","last_fullsized":"2024-04-17T10:28:01.378500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396318+0000","last_change":"2024-04-17T10:28:01.393600+0000","last_active":"2024-04-17T10:28:01.396318+0000","last_peered":"2024-04-17T10:28:01.396318+0000","last_clean":"2024-04-17T10:28:01.396318+0000","last_became_active":"2024-04-17T10:28:01.393277+0000","last_became_peered":"2024-04-17T10:28:01.393277+0000","last_unstale":"2024-04-17T10:28:01.396318+0000","last_undegraded":"2024-04-17T10:28:01.396318+0000","last_fullsized":"2024-04-17T10:28:01.396318+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:55:03.162699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373787+0000","last_change":"2024-04-17T10:28:01.373787+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373787+0000","last_undegraded":"2024-04-17T10:28:01.373787+0000","last_fullsized":"2024-04-17T10:28:01.373787+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391197+0000","last_change":"2024-04-17T10:28:01.390060+0000","last_active":"2024-04-17T10:28:01.391197+0000","last_peered":"2024-04-17T10:28:01.391197+0000","last_clean":"2024-04-17T10:28:01.391197+0000","last_became_active":"2024-04-17T10:28:01.389732+0000","last_became_peered":"2024-04-17T10:28:01.389732+0000","last_unstale":"2024-04-17T10:28:01.391197+0000","last_undegraded":"2024-04-17T10:28:01.391197+0000","last_fullsized":"2024-04-17T10:28:01.391197+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:07.506032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381577+0000","last_change":"2024-04-17T10:28:01.381577+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381577+0000","last_undegraded":"2024-04-17T10:28:01.381577+0000","last_fullsized":"2024-04-17T10:28:01.381577+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384958+0000","last_change":"2024-04-17T10:28:01.380617+0000","last_active":"2024-04-17T10:28:01.384958+0000","last_peered":"2024-04-17T10:28:01.384958+0000","last_clean":"2024-04-17T10:28:01.384958+0000","last_became_active":"2024-04-17T10:28:01.380340+0000","last_became_peered":"2024-04-17T10:28:01.380340+0000","last_unstale":"2024-04-17T10:28:01.384958+0000","last_undegraded":"2024-04-17T10:28:01.384958+0000","last_fullsized":"2024-04-17T10:28:01.384958+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:58.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371208+0000","last_change":"2024-04-17T10:28:01.371208+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371208+0000","last_undegraded":"2024-04-17T10:28:01.371208+0000","last_fullsized":"2024-04-17T10:28:01.371208+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.395078+0000","last_change":"2024-04-17T10:28:01.393021+0000","last_active":"2024-04-17T10:28:01.395078+0000","last_peered":"2024-04-17T10:28:01.395078+0000","last_clean":"2024-04-17T10:28:01.395078+0000","last_became_active":"2024-04-17T10:28:01.392144+0000","last_became_peered":"2024-04-17T10:28:01.392144+0000","last_unstale":"2024-04-17T10:28:01.395078+0000","last_undegraded":"2024-04-17T10:28:01.395078+0000","last_fullsized":"2024-04-17T10:28:01.395078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.368273+0000","last_change":"2024-04-17T10:28:01.368273+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.368273+0000","last_undegraded":"2024-04-17T10:28:01.368273+0000","last_fullsized":"2024-04-17T10:28:01.368273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.389296+0000","last_change":"2024-04-17T10:28:01.384888+0000","last_active":"2024-04-17T10:28:01.389296+0000","last_peered":"2024-04-17T10:28:01.389296+0000","last_clean":"2024-04-17T10:28:01.389296+0000","last_became_active":"2024-04-17T10:28:01.384242+0000","last_became_peered":"2024-04-17T10:28:01.384242+0000","last_unstale":"2024-04-17T10:28:01.389296+0000","last_undegraded":"2024-04-17T10:28:01.389296+0000","last_fullsized":"2024-04-17T10:28:01.389296+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:15.827189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375458+0000","last_change":"2024-04-17T10:28:01.375458+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375458+0000","last_undegraded":"2024-04-17T10:28:01.375458+0000","last_fullsized":"2024-04-17T10:28:01.375458+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399097+0000","last_change":"2024-04-17T10:28:01.395507+0000","last_active":"2024-04-17T10:28:01.399097+0000","last_peered":"2024-04-17T10:28:01.399097+0000","last_clean":"2024-04-17T10:28:01.399097+0000","last_became_active":"2024-04-17T10:28:01.392288+0000","last_became_peered":"2024-04-17T10:28:01.392288+0000","last_unstale":"2024-04-17T10:28:01.399097+0000","last_undegraded":"2024-04-17T10:28:01.399097+0000","last_fullsized":"2024-04-17T10:28:01.399097+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:04:14.700196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380500+0000","last_change":"2024-04-17T10:28:01.380500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380500+0000","last_undegraded":"2024-04-17T10:28:01.380500+0000","last_fullsized":"2024-04-17T10:28:01.380500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394290+0000","last_change":"2024-04-17T10:28:01.391397+0000","last_active":"2024-04-17T10:28:01.394290+0000","last_peered":"2024-04-17T10:28:01.394290+0000","last_clean":"2024-04-17T10:28:01.394290+0000","last_became_active":"2024-04-17T10:28:01.391114+0000","last_became_peered":"2024-04-17T10:28:01.391114+0000","last_unstale":"2024-04-17T10:28:01.394290+0000","last_undegraded":"2024-04-17T10:28:01.394290+0000","last_fullsized":"2024-04-17T10:28:01.394290+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:23.592401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371189+0000","last_change":"2024-04-17T10:28:01.371189+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371189+0000","last_undegraded":"2024-04-17T10:28:01.371189+0000","last_fullsized":"2024-04-17T10:28:01.371189+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388236+0000","last_change":"2024-04-17T10:28:01.386653+0000","last_active":"2024-04-17T10:28:01.388236+0000","last_peered":"2024-04-17T10:28:01.388236+0000","last_clean":"2024-04-17T10:28:01.388236+0000","last_became_active":"2024-04-17T10:28:01.386406+0000","last_became_peered":"2024-04-17T10:28:01.386406+0000","last_unstale":"2024-04-17T10:28:01.388236+0000","last_undegraded":"2024-04-17T10:28:01.388236+0000","last_fullsized":"2024-04-17T10:28:01.388236+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:31.804436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378166+0000","last_change":"2024-04-17T10:28:01.378166+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378166+0000","last_undegraded":"2024-04-17T10:28:01.378166+0000","last_fullsized":"2024-04-17T10:28:01.378166+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391538+0000","last_change":"2024-04-17T10:28:01.389788+0000","last_active":"2024-04-17T10:28:01.391538+0000","last_peered":"2024-04-17T10:28:01.391538+0000","last_clean":"2024-04-17T10:28:01.391538+0000","last_became_active":"2024-04-17T10:28:01.389488+0000","last_became_peered":"2024-04-17T10:28:01.389488+0000","last_unstale":"2024-04-17T10:28:01.391538+0000","last_undegraded":"2024-04-17T10:28:01.391538+0000","last_fullsized":"2024-04-17T10:28:01.391538+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:19.391683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370896+0000","last_change":"2024-04-17T10:28:01.370896+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370896+0000","last_undegraded":"2024-04-17T10:28:01.370896+0000","last_fullsized":"2024-04-17T10:28:01.370896+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398827+0000","last_change":"2024-04-17T10:28:01.386384+0000","last_active":"2024-04-17T10:28:01.398827+0000","last_peered":"2024-04-17T10:28:01.398827+0000","last_clean":"2024-04-17T10:28:01.398827+0000","last_became_active":"2024-04-17T10:28:01.386102+0000","last_became_peered":"2024-04-17T10:28:01.386102+0000","last_unstale":"2024-04-17T10:28:01.398827+0000","last_undegraded":"2024-04-17T10:28:01.398827+0000","last_fullsized":"2024-04-17T10:28:01.398827+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375829+0000","last_change":"2024-04-17T10:28:01.375829+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375829+0000","last_undegraded":"2024-04-17T10:28:01.375829+0000","last_fullsized":"2024-04-17T10:28:01.375829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398627+0000","last_change":"2024-04-17T10:28:01.394707+0000","last_active":"2024-04-17T10:28:01.398627+0000","last_peered":"2024-04-17T10:28:01.398627+0000","last_clean":"2024-04-17T10:28:01.398627+0000","last_became_active":"2024-04-17T10:28:01.394419+0000","last_became_peered":"2024-04-17T10:28:01.394419+0000","last_unstale":"2024-04-17T10:28:01.398627+0000","last_undegraded":"2024-04-17T10:28:01.398627+0000","last_fullsized":"2024-04-17T10:28:01.398627+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:05:22.975781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371207+0000","last_change":"2024-04-17T10:28:01.371207+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371207+0000","last_undegraded":"2024-04-17T10:28:01.371207+0000","last_fullsized":"2024-04-17T10:28:01.371207+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.397867+0000","last_change":"2024-04-17T10:28:01.396212+0000","last_active":"2024-04-17T10:28:01.397867+0000","last_peered":"2024-04-17T10:28:01.397867+0000","last_clean":"2024-04-17T10:28:01.397867+0000","last_became_active":"2024-04-17T10:28:01.395939+0000","last_became_peered":"2024-04-17T10:28:01.395939+0000","last_unstale":"2024-04-17T10:28:01.397867+0000","last_undegraded":"2024-04-17T10:28:01.397867+0000","last_fullsized":"2024-04-17T10:28:01.397867+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:00.316480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376341+0000","last_change":"2024-04-17T10:28:01.376341+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376341+0000","last_undegraded":"2024-04-17T10:28:01.376341+0000","last_fullsized":"2024-04-17T10:28:01.376341+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.386497+0000","last_change":"2024-04-17T10:28:01.384533+0000","last_active":"2024-04-17T10:28:01.386497+0000","last_peered":"2024-04-17T10:28:01.386497+0000","last_clean":"2024-04-17T10:28:01.386497+0000","last_became_active":"2024-04-17T10:28:01.384284+0000","last_became_peered":"2024-04-17T10:28:01.384284+0000","last_unstale":"2024-04-17T10:28:01.386497+0000","last_undegraded":"2024-04-17T10:28:01.386497+0000","last_fullsized":"2024-04-17T10:28:01.386497+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371347+0000","last_change":"2024-04-17T10:28:01.371347+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371347+0000","last_undegraded":"2024-04-17T10:28:01.371347+0000","last_fullsized":"2024-04-17T10:28:01.371347+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391583+0000","last_change":"2024-04-17T10:28:01.388193+0000","last_active":"2024-04-17T10:28:01.391583+0000","last_peered":"2024-04-17T10:28:01.391583+0000","last_clean":"2024-04-17T10:28:01.391583+0000","last_became_active":"2024-04-17T10:28:01.387887+0000","last_became_peered":"2024-04-17T10:28:01.387887+0000","last_unstale":"2024-04-17T10:28:01.391583+0000","last_undegraded":"2024-04-17T10:28:01.391583+0000","last_fullsized":"2024-04-17T10:28:01.391583+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:12.842514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367353+0000","last_change":"2024-04-17T10:28:01.367353+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367353+0000","last_undegraded":"2024-04-17T10:28:01.367353+0000","last_fullsized":"2024-04-17T10:28:01.367353+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398900+0000","last_change":"2024-04-17T10:28:01.387103+0000","last_active":"2024-04-17T10:28:01.398900+0000","last_peered":"2024-04-17T10:28:01.398900+0000","last_clean":"2024-04-17T10:28:01.398900+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.398900+0000","last_undegraded":"2024-04-17T10:28:01.398900+0000","last_fullsized":"2024-04-17T10:28:01.398900+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381319+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:01.381319+0000","last_peered":"2024-04-17T10:28:01.381319+0000","last_clean":"2024-04-17T10:28:01.381319+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:01.381319+0000","last_undegraded":"2024-04-17T10:28:01.381319+0000","last_fullsized":"2024-04-17T10:28:01.381319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36528,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335312,"statfs":{"total":96636764160,"available":96599359488,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35952,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335888,"statfs":{"total":96636764160,"available":96599949312,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40104,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331736,"statfs":{"total":96636764160,"available":96595697664,"internally_reserved":0,"allocated":606208,"data_stored":270114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35956,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335884,"statfs":{"total":96636764160,"available":96599945216,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35944,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335896,"statfs":{"total":96636764160,"available":96599957504,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58732,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313108,"statfs":{"total":96636764160,"available":96576622592,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:28:06.496 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:28:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.678+0000 7f7429400640 1 Processor -- start 2024-04-17T10:28:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.678+0000 7f7429400640 1 -- start start 2024-04-17T10:28:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7429400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f742406eac0 0x7f7424143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7429400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74240cf0c0 0x7f7424144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7429400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7424144560 con 0x7f742406e6e0 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7429400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742415ae40 con 0x7f74240cf0c0 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7429400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7424071440 con 0x7f742406eac0 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7423200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74240cf0c0 0x7f7424144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7422800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f742406e6e0 0x7f74240ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53816/0 (socket says 172.21.15.67:53816) 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7422800640 1 -- 172.21.15.67:0/92642173 learned_addr learned my addr 172.21.15.67:0/92642173 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7419e00640 1 --2- 172.21.15.67:0/92642173 >> v2:172.21.15.86:3300/0 conn(0x7f742406eac0 0x7f7424143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7421e00640 1 -- 172.21.15.67:0/92642173 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648087673 0 0) 0x7f7424144560 con 0x7f742406e6e0 2024-04-17T10:28:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7423200640 1 -- 172.21.15.67:0/92642173 >> v2:172.21.15.86:3300/0 conn(0x7f742406eac0 msgr2=0x7f7424143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:06.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7423200640 1 --2- 172.21.15.67:0/92642173 >> v2:172.21.15.86:3300/0 conn(0x7f742406eac0 0x7f7424143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7423200640 1 -- 172.21.15.67:0/92642173 >> v1:172.21.15.67:6789/0 conn(0x7f742406e6e0 legacy=0x7f74240ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.682+0000 7f7423200640 1 -- 172.21.15.67:0/92642173 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424071720 con 0x7f74240cf0c0 2024-04-17T10:28:06.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7423200640 1 --2- 172.21.15.67:0/92642173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74240cf0c0 0x7f7424144020 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f742406ef80 tx=0x7f741c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77bbb641aabc46d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:06.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7421e00640 1 -- 172.21.15.67:0/92642173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0b1070 con 0x7f74240cf0c0 2024-04-17T10:28:06.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7421e00640 1 -- 172.21.15.67:0/92642173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c0ac620 con 0x7f74240cf0c0 2024-04-17T10:28:06.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7421e00640 1 -- 172.21.15.67:0/92642173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0ac9f0 con 0x7f74240cf0c0 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 -- 172.21.15.67:0/92642173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74240cf0c0 msgr2=0x7f7424144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 --2- 172.21.15.67:0/92642173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74240cf0c0 0x7f7424144020 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f742406ef80 tx=0x7f741c01c3c0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 -- 172.21.15.67:0/92642173 shutdown_connections 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 --2- 172.21.15.67:0/92642173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74240cf0c0 0x7f7424144020 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 --2- 172.21.15.67:0/92642173 >> v2:172.21.15.86:3300/0 conn(0x7f742406eac0 0x7f7424143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 -- 172.21.15.67:0/92642173 >> 172.21.15.67:0/92642173 conn(0x7f742406ac90 msgr2=0x7f742406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 -- 172.21.15.67:0/92642173 shutdown_connections 2024-04-17T10:28:06.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 -- 172.21.15.67:0/92642173 wait complete. 2024-04-17T10:28:06.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 Processor -- start 2024-04-17T10:28:06.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.686+0000 7f7429400640 1 -- start start 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f742406e6e0 0x7f7424139510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f742406eac0 0x7f7424137b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74240cf0c0 0x7f74240acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74240b3a40 con 0x7f742406eac0 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74240b38c0 con 0x7f74240cf0c0 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74240b3bc0 con 0x7f742406e6e0 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f742406eac0 0x7f7424137b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f742406eac0 0x7f7424137b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52508/0 (socket says 172.21.15.67:52508) 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 -- 172.21.15.67:0/2578024912 learned_addr learned my addr 172.21.15.67:0/2578024912 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7422800640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f742406e6e0 0x7f7424139510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7423200640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74240cf0c0 0x7f74240acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 -- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f742406e6e0 msgr2=0x7f7424139510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f742406e6e0 0x7f7424139510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 -- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74240cf0c0 msgr2=0x7f74240acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74240cf0c0 0x7f74240acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 -- 172.21.15.67:0/2578024912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424138470 con 0x7f742406eac0 2024-04-17T10:28:06.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7419e00640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f742406eac0 0x7f7424137b00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f740c00e930 tx=0x7f740c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:06.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f741b400640 1 -- 172.21.15.67:0/2578024912 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c017070 con 0x7f742406eac0 2024-04-17T10:28:06.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f741b400640 1 -- 172.21.15.67:0/2578024912 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740c00ca00 con 0x7f742406eac0 2024-04-17T10:28:06.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f741b400640 1 -- 172.21.15.67:0/2578024912 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740c00cdb0 con 0x7f742406eac0 2024-04-17T10:28:06.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7424138750 con 0x7f742406eac0 2024-04-17T10:28:06.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7424138ff0 con 0x7f742406eac0 2024-04-17T10:28:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7424138920 con 0x7f742406eac0 2024-04-17T10:28:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.690+0000 7f741b400640 1 -- 172.21.15.67:0/2578024912 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f740c01a070 con 0x7f742406eac0 2024-04-17T10:28:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.694+0000 7f741b400640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7400066f60 0x7f7400069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.694+0000 7f741b400640 1 -- 172.21.15.67:0/2578024912 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f740c0855a0 con 0x7f742406eac0 2024-04-17T10:28:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.694+0000 7f7422800640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7400066f60 0x7f7400069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.694+0000 7f7422800640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7400066f60 0x7f7400069420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7414007fd0 tx=0x7f7414007ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:06.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.698+0000 7f741b400640 1 -- 172.21.15.67:0/2578024912 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f740c0564d0 con 0x7f742406eac0 2024-04-17T10:28:06.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.858+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7424074e30 con 0x7f7400066f60 2024-04-17T10:28:06.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.866+0000 7f741b400640 1 -- 172.21.15.67:0/2578024912 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7f7424074e30 con 0x7f7400066f60 2024-04-17T10:28:06.861 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:06.863 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:28:06.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.866+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7400066f60 msgr2=0x7f7400069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.866+0000 7f7429400640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7400066f60 0x7f7400069420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7414007fd0 tx=0x7f7414007ee0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.866+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f742406eac0 msgr2=0x7f7424137b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:06.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.866+0000 7f7429400640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f742406eac0 0x7f7424137b00 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f740c00e930 tx=0x7f740c00edf0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 shutdown_connections 2024-04-17T10:28:06.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7400066f60 0x7f7400069420 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74240cf0c0 0x7f74240acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f742406eac0 0x7f7424137b00 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 --2- 172.21.15.67:0/2578024912 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f742406e6e0 0x7f7424139510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 >> 172.21.15.67:0/2578024912 conn(0x7f742406ac90 msgr2=0x7f7424144000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 shutdown_connections 2024-04-17T10:28:06.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:06.870+0000 7f7429400640 1 -- 172.21.15.67:0/2578024912 wait complete. 2024-04-17T10:28:06.881 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-17T10:28:06.177959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315112,"kb_used_data":5980,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754659608,"statfs":{"total":773094113280,"available":772771438592,"internally_reserved":0,"allocated":6123520,"data_stored":3388317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.817202"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380959+0000","last_change":"2024-04-17T10:28:01.380959+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380959+0000","last_undegraded":"2024-04-17T10:28:01.380959+0000","last_fullsized":"2024-04-17T10:28:01.380959+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376444+0000","last_change":"2024-04-17T10:28:01.376444+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376444+0000","last_undegraded":"2024-04-17T10:28:01.376444+0000","last_fullsized":"2024-04-17T10:28:01.376444+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378464+0000","last_change":"2024-04-17T10:28:01.378464+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378464+0000","last_undegraded":"2024-04-17T10:28:01.378464+0000","last_fullsized":"2024-04-17T10:28:01.378464+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377387+0000","last_change":"2024-04-17T10:28:01.377387+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377387+0000","last_undegraded":"2024-04-17T10:28:01.377387+0000","last_fullsized":"2024-04-17T10:28:01.377387+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374044+0000","last_change":"2024-04-17T10:28:01.374044+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374044+0000","last_undegraded":"2024-04-17T10:28:01.374044+0000","last_fullsized":"2024-04-17T10:28:01.374044+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378669+0000","last_change":"2024-04-17T10:28:01.378669+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378669+0000","last_undegraded":"2024-04-17T10:28:01.378669+0000","last_fullsized":"2024-04-17T10:28:01.378669+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377863+0000","last_change":"2024-04-17T10:28:01.377863+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377863+0000","last_undegraded":"2024-04-17T10:28:01.377863+0000","last_fullsized":"2024-04-17T10:28:01.377863+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370398+0000","last_change":"2024-04-17T10:28:01.370398+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370398+0000","last_undegraded":"2024-04-17T10:28:01.370398+0000","last_fullsized":"2024-04-17T10:28:01.370398+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371867+0000","last_change":"2024-04-17T10:28:01.371867+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371867+0000","last_undegraded":"2024-04-17T10:28:01.371867+0000","last_fullsized":"2024-04-17T10:28:01.371867+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379871+0000","last_change":"2024-04-17T10:28:01.379871+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379871+0000","last_undegraded":"2024-04-17T10:28:01.379871+0000","last_fullsized":"2024-04-17T10:28:01.379871+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379507+0000","last_change":"2024-04-17T10:28:01.379507+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379507+0000","last_undegraded":"2024-04-17T10:28:01.379507+0000","last_fullsized":"2024-04-17T10:28:01.379507+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371184+0000","last_change":"2024-04-17T10:28:01.371184+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371184+0000","last_undegraded":"2024-04-17T10:28:01.371184+0000","last_fullsized":"2024-04-17T10:28:01.371184+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381984+0000","last_change":"2024-04-17T10:28:01.381984+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381984+0000","last_undegraded":"2024-04-17T10:28:01.381984+0000","last_fullsized":"2024-04-17T10:28:01.381984+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377452+0000","last_change":"2024-04-17T10:28:01.377452+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377452+0000","last_undegraded":"2024-04-17T10:28:01.377452+0000","last_fullsized":"2024-04-17T10:28:01.377452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376914+0000","last_change":"2024-04-17T10:28:01.376914+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376914+0000","last_undegraded":"2024-04-17T10:28:01.376914+0000","last_fullsized":"2024-04-17T10:28:01.376914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376339+0000","last_change":"2024-04-17T10:28:01.376339+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376339+0000","last_undegraded":"2024-04-17T10:28:01.376339+0000","last_fullsized":"2024-04-17T10:28:01.376339+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379944+0000","last_change":"2024-04-17T10:28:01.379944+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379944+0000","last_undegraded":"2024-04-17T10:28:01.379944+0000","last_fullsized":"2024-04-17T10:28:01.379944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376011+0000","last_change":"2024-04-17T10:28:01.376011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376011+0000","last_undegraded":"2024-04-17T10:28:01.376011+0000","last_fullsized":"2024-04-17T10:28:01.376011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377320+0000","last_change":"2024-04-17T10:28:01.377320+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377320+0000","last_undegraded":"2024-04-17T10:28:01.377320+0000","last_fullsized":"2024-04-17T10:28:01.377320+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369232+0000","last_change":"2024-04-17T10:28:01.369232+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369232+0000","last_undegraded":"2024-04-17T10:28:01.369232+0000","last_fullsized":"2024-04-17T10:28:01.369232+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367428+0000","last_change":"2024-04-17T10:28:01.367428+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367428+0000","last_undegraded":"2024-04-17T10:28:01.367428+0000","last_fullsized":"2024-04-17T10:28:01.367428+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370644+0000","last_change":"2024-04-17T10:28:01.370644+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370644+0000","last_undegraded":"2024-04-17T10:28:01.370644+0000","last_fullsized":"2024-04-17T10:28:01.370644+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379078+0000","last_change":"2024-04-17T10:28:01.379078+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379078+0000","last_undegraded":"2024-04-17T10:28:01.379078+0000","last_fullsized":"2024-04-17T10:28:01.379078+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380365+0000","last_change":"2024-04-17T10:28:01.380365+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380365+0000","last_undegraded":"2024-04-17T10:28:01.380365+0000","last_fullsized":"2024-04-17T10:28:01.380365+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374358+0000","last_change":"2024-04-17T10:28:01.374358+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374358+0000","last_undegraded":"2024-04-17T10:28:01.374358+0000","last_fullsized":"2024-04-17T10:28:01.374358+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.383292+0000","last_change":"2024-04-17T10:28:01.383292+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.383292+0000","last_undegraded":"2024-04-17T10:28:01.383292+0000","last_fullsized":"2024-04-17T10:28:01.383292+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367828+0000","last_change":"2024-04-17T10:28:01.367828+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367828+0000","last_undegraded":"2024-04-17T10:28:01.367828+0000","last_fullsized":"2024-04-17T10:28:01.367828+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377527+0000","last_change":"2024-04-17T10:28:01.377527+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377527+0000","last_undegraded":"2024-04-17T10:28:01.377527+0000","last_fullsized":"2024-04-17T10:28:01.377527+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379750+0000","last_change":"2024-04-17T10:28:01.379750+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379750+0000","last_undegraded":"2024-04-17T10:28:01.379750+0000","last_fullsized":"2024-04-17T10:28:01.379750+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376427+0000","last_change":"2024-04-17T10:28:01.376427+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376427+0000","last_undegraded":"2024-04-17T10:28:01.376427+0000","last_fullsized":"2024-04-17T10:28:01.376427+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372163+0000","last_change":"2024-04-17T10:28:01.372163+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372163+0000","last_undegraded":"2024-04-17T10:28:01.372163+0000","last_fullsized":"2024-04-17T10:28:01.372163+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377077+0000","last_change":"2024-04-17T10:28:01.377077+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377077+0000","last_undegraded":"2024-04-17T10:28:01.377077+0000","last_fullsized":"2024-04-17T10:28:01.377077+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384066+0000","last_change":"2024-04-17T10:28:01.380916+0000","last_active":"2024-04-17T10:28:01.384066+0000","last_peered":"2024-04-17T10:28:01.384066+0000","last_clean":"2024-04-17T10:28:01.384066+0000","last_became_active":"2024-04-17T10:28:01.380635+0000","last_became_peered":"2024-04-17T10:28:01.380635+0000","last_unstale":"2024-04-17T10:28:01.384066+0000","last_undegraded":"2024-04-17T10:28:01.384066+0000","last_fullsized":"2024-04-17T10:28:01.384066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:47:03.828693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371180+0000","last_change":"2024-04-17T10:28:01.371180+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371180+0000","last_undegraded":"2024-04-17T10:28:01.371180+0000","last_fullsized":"2024-04-17T10:28:01.371180+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394963+0000","last_change":"2024-04-17T10:28:01.391871+0000","last_active":"2024-04-17T10:28:01.394963+0000","last_peered":"2024-04-17T10:28:01.394963+0000","last_clean":"2024-04-17T10:28:01.394963+0000","last_became_active":"2024-04-17T10:28:01.391540+0000","last_became_peered":"2024-04-17T10:28:01.391540+0000","last_unstale":"2024-04-17T10:28:01.394963+0000","last_undegraded":"2024-04-17T10:28:01.394963+0000","last_fullsized":"2024-04-17T10:28:01.394963+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:03:16.384430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367514+0000","last_change":"2024-04-17T10:28:01.367514+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367514+0000","last_undegraded":"2024-04-17T10:28:01.367514+0000","last_fullsized":"2024-04-17T10:28:01.367514+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394726+0000","last_change":"2024-04-17T10:28:01.392616+0000","last_active":"2024-04-17T10:28:01.394726+0000","last_peered":"2024-04-17T10:28:01.394726+0000","last_clean":"2024-04-17T10:28:01.394726+0000","last_became_active":"2024-04-17T10:28:01.391986+0000","last_became_peered":"2024-04-17T10:28:01.391986+0000","last_unstale":"2024-04-17T10:28:01.394726+0000","last_undegraded":"2024-04-17T10:28:01.394726+0000","last_fullsized":"2024-04-17T10:28:01.394726+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371610+0000","last_change":"2024-04-17T10:28:01.371610+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371610+0000","last_undegraded":"2024-04-17T10:28:01.371610+0000","last_fullsized":"2024-04-17T10:28:01.371610+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383990+0000","last_change":"2024-04-17T10:28:01.380404+0000","last_active":"2024-04-17T10:28:01.383990+0000","last_peered":"2024-04-17T10:28:01.383990+0000","last_clean":"2024-04-17T10:28:01.383990+0000","last_became_active":"2024-04-17T10:28:01.380065+0000","last_became_peered":"2024-04-17T10:28:01.380065+0000","last_unstale":"2024-04-17T10:28:01.383990+0000","last_undegraded":"2024-04-17T10:28:01.383990+0000","last_fullsized":"2024-04-17T10:28:01.383990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:12.355937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375461+0000","last_change":"2024-04-17T10:28:01.375461+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375461+0000","last_undegraded":"2024-04-17T10:28:01.375461+0000","last_fullsized":"2024-04-17T10:28:01.375461+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399062+0000","last_change":"2024-04-17T10:28:01.395609+0000","last_active":"2024-04-17T10:28:01.399062+0000","last_peered":"2024-04-17T10:28:01.399062+0000","last_clean":"2024-04-17T10:28:01.399062+0000","last_became_active":"2024-04-17T10:28:01.395267+0000","last_became_peered":"2024-04-17T10:28:01.395267+0000","last_unstale":"2024-04-17T10:28:01.399062+0000","last_undegraded":"2024-04-17T10:28:01.399062+0000","last_fullsized":"2024-04-17T10:28:01.399062+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:42.740201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377612+0000","last_change":"2024-04-17T10:28:01.377612+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377612+0000","last_undegraded":"2024-04-17T10:28:01.377612+0000","last_fullsized":"2024-04-17T10:28:01.377612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383876+0000","last_change":"2024-04-17T10:28:01.378483+0000","last_active":"2024-04-17T10:28:01.383876+0000","last_peered":"2024-04-17T10:28:01.383876+0000","last_clean":"2024-04-17T10:28:01.383876+0000","last_became_active":"2024-04-17T10:28:01.378140+0000","last_became_peered":"2024-04-17T10:28:01.378140+0000","last_unstale":"2024-04-17T10:28:01.383876+0000","last_undegraded":"2024-04-17T10:28:01.383876+0000","last_fullsized":"2024-04-17T10:28:01.383876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371666+0000","last_change":"2024-04-17T10:28:01.371666+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371666+0000","last_undegraded":"2024-04-17T10:28:01.371666+0000","last_fullsized":"2024-04-17T10:28:01.371666+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383084+0000","last_change":"2024-04-17T10:28:01.378144+0000","last_active":"2024-04-17T10:28:01.383084+0000","last_peered":"2024-04-17T10:28:01.383084+0000","last_clean":"2024-04-17T10:28:01.383084+0000","last_became_active":"2024-04-17T10:28:01.377879+0000","last_became_peered":"2024-04-17T10:28:01.377879+0000","last_unstale":"2024-04-17T10:28:01.383084+0000","last_undegraded":"2024-04-17T10:28:01.383084+0000","last_fullsized":"2024-04-17T10:28:01.383084+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:02.479603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378313+0000","last_change":"2024-04-17T10:28:01.378313+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378313+0000","last_undegraded":"2024-04-17T10:28:01.378313+0000","last_fullsized":"2024-04-17T10:28:01.378313+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381525+0000","last_change":"2024-04-17T10:28:01.376007+0000","last_active":"2024-04-17T10:28:01.381525+0000","last_peered":"2024-04-17T10:28:01.381525+0000","last_clean":"2024-04-17T10:28:01.381525+0000","last_became_active":"2024-04-17T10:28:01.375098+0000","last_became_peered":"2024-04-17T10:28:01.375098+0000","last_unstale":"2024-04-17T10:28:01.381525+0000","last_undegraded":"2024-04-17T10:28:01.381525+0000","last_fullsized":"2024-04-17T10:28:01.381525+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372934+0000","last_change":"2024-04-17T10:28:01.372934+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372934+0000","last_undegraded":"2024-04-17T10:28:01.372934+0000","last_fullsized":"2024-04-17T10:28:01.372934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394069+0000","last_change":"2024-04-17T10:28:01.390159+0000","last_active":"2024-04-17T10:28:01.394069+0000","last_peered":"2024-04-17T10:28:01.394069+0000","last_clean":"2024-04-17T10:28:01.394069+0000","last_became_active":"2024-04-17T10:28:01.389874+0000","last_became_peered":"2024-04-17T10:28:01.389874+0000","last_unstale":"2024-04-17T10:28:01.394069+0000","last_undegraded":"2024-04-17T10:28:01.394069+0000","last_fullsized":"2024-04-17T10:28:01.394069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:59.077053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372104+0000","last_change":"2024-04-17T10:28:01.372104+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372104+0000","last_undegraded":"2024-04-17T10:28:01.372104+0000","last_fullsized":"2024-04-17T10:28:01.372104+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396819+0000","last_change":"2024-04-17T10:28:01.396322+0000","last_active":"2024-04-17T10:28:01.396819+0000","last_peered":"2024-04-17T10:28:01.396819+0000","last_clean":"2024-04-17T10:28:01.396819+0000","last_became_active":"2024-04-17T10:28:01.396090+0000","last_became_peered":"2024-04-17T10:28:01.396090+0000","last_unstale":"2024-04-17T10:28:01.396819+0000","last_undegraded":"2024-04-17T10:28:01.396819+0000","last_fullsized":"2024-04-17T10:28:01.396819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394086+0000","last_change":"2024-04-17T10:28:01.390653+0000","last_active":"2024-04-17T10:28:01.394086+0000","last_peered":"2024-04-17T10:28:01.394086+0000","last_clean":"2024-04-17T10:28:01.394086+0000","last_became_active":"2024-04-17T10:28:01.390327+0000","last_became_peered":"2024-04-17T10:28:01.390327+0000","last_unstale":"2024-04-17T10:28:01.394086+0000","last_undegraded":"2024-04-17T10:28:01.394086+0000","last_fullsized":"2024-04-17T10:28:01.394086+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:12.074317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373570+0000","last_change":"2024-04-17T10:28:01.373570+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373570+0000","last_undegraded":"2024-04-17T10:28:01.373570+0000","last_fullsized":"2024-04-17T10:28:01.373570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.393035+0000","last_change":"2024-04-17T10:28:01.390744+0000","last_active":"2024-04-17T10:28:01.393035+0000","last_peered":"2024-04-17T10:28:01.393035+0000","last_clean":"2024-04-17T10:28:01.393035+0000","last_became_active":"2024-04-17T10:28:01.390487+0000","last_became_peered":"2024-04-17T10:28:01.390487+0000","last_unstale":"2024-04-17T10:28:01.393035+0000","last_undegraded":"2024-04-17T10:28:01.393035+0000","last_fullsized":"2024-04-17T10:28:01.393035+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:21.812637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371710+0000","last_change":"2024-04-17T10:28:01.371710+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371710+0000","last_undegraded":"2024-04-17T10:28:01.371710+0000","last_fullsized":"2024-04-17T10:28:01.371710+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399042+0000","last_change":"2024-04-17T10:28:01.388726+0000","last_active":"2024-04-17T10:28:01.399042+0000","last_peered":"2024-04-17T10:28:01.399042+0000","last_clean":"2024-04-17T10:28:01.399042+0000","last_became_active":"2024-04-17T10:28:01.388462+0000","last_became_peered":"2024-04-17T10:28:01.388462+0000","last_unstale":"2024-04-17T10:28:01.399042+0000","last_undegraded":"2024-04-17T10:28:01.399042+0000","last_fullsized":"2024-04-17T10:28:01.399042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376893+0000","last_change":"2024-04-17T10:28:01.376893+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376893+0000","last_undegraded":"2024-04-17T10:28:01.376893+0000","last_fullsized":"2024-04-17T10:28:01.376893+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.390536+0000","last_change":"2024-04-17T10:28:01.388333+0000","last_active":"2024-04-17T10:28:01.390536+0000","last_peered":"2024-04-17T10:28:01.390536+0000","last_clean":"2024-04-17T10:28:01.390536+0000","last_became_active":"2024-04-17T10:28:01.387931+0000","last_became_peered":"2024-04-17T10:28:01.387931+0000","last_unstale":"2024-04-17T10:28:01.390536+0000","last_undegraded":"2024-04-17T10:28:01.390536+0000","last_fullsized":"2024-04-17T10:28:01.390536+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:28:46.889727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372115+0000","last_change":"2024-04-17T10:28:01.372115+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372115+0000","last_undegraded":"2024-04-17T10:28:01.372115+0000","last_fullsized":"2024-04-17T10:28:01.372115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388876+0000","last_change":"2024-04-17T10:28:01.387372+0000","last_active":"2024-04-17T10:28:01.388876+0000","last_peered":"2024-04-17T10:28:01.388876+0000","last_clean":"2024-04-17T10:28:01.388876+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.388876+0000","last_undegraded":"2024-04-17T10:28:01.388876+0000","last_fullsized":"2024-04-17T10:28:01.388876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:31.085374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376533+0000","last_change":"2024-04-17T10:28:01.376533+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376533+0000","last_undegraded":"2024-04-17T10:28:01.376533+0000","last_fullsized":"2024-04-17T10:28:01.376533+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391514+0000","last_change":"2024-04-17T10:28:01.390499+0000","last_active":"2024-04-17T10:28:01.391514+0000","last_peered":"2024-04-17T10:28:01.391514+0000","last_clean":"2024-04-17T10:28:01.391514+0000","last_became_active":"2024-04-17T10:28:01.390208+0000","last_became_peered":"2024-04-17T10:28:01.390208+0000","last_unstale":"2024-04-17T10:28:01.391514+0000","last_undegraded":"2024-04-17T10:28:01.391514+0000","last_fullsized":"2024-04-17T10:28:01.391514+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:44.289510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369897+0000","last_change":"2024-04-17T10:28:01.369897+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369897+0000","last_undegraded":"2024-04-17T10:28:01.369897+0000","last_fullsized":"2024-04-17T10:28:01.369897+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398772+0000","last_change":"2024-04-17T10:28:01.385970+0000","last_active":"2024-04-17T10:28:01.398772+0000","last_peered":"2024-04-17T10:28:01.398772+0000","last_clean":"2024-04-17T10:28:01.398772+0000","last_became_active":"2024-04-17T10:28:01.385447+0000","last_became_peered":"2024-04-17T10:28:01.385447+0000","last_unstale":"2024-04-17T10:28:01.398772+0000","last_undegraded":"2024-04-17T10:28:01.398772+0000","last_fullsized":"2024-04-17T10:28:01.398772+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378500+0000","last_change":"2024-04-17T10:28:01.378500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378500+0000","last_undegraded":"2024-04-17T10:28:01.378500+0000","last_fullsized":"2024-04-17T10:28:01.378500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396318+0000","last_change":"2024-04-17T10:28:01.393600+0000","last_active":"2024-04-17T10:28:01.396318+0000","last_peered":"2024-04-17T10:28:01.396318+0000","last_clean":"2024-04-17T10:28:01.396318+0000","last_became_active":"2024-04-17T10:28:01.393277+0000","last_became_peered":"2024-04-17T10:28:01.393277+0000","last_unstale":"2024-04-17T10:28:01.396318+0000","last_undegraded":"2024-04-17T10:28:01.396318+0000","last_fullsized":"2024-04-17T10:28:01.396318+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:55:03.162699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373787+0000","last_change":"2024-04-17T10:28:01.373787+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373787+0000","last_undegraded":"2024-04-17T10:28:01.373787+0000","last_fullsized":"2024-04-17T10:28:01.373787+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391197+0000","last_change":"2024-04-17T10:28:01.390060+0000","last_active":"2024-04-17T10:28:01.391197+0000","last_peered":"2024-04-17T10:28:01.391197+0000","last_clean":"2024-04-17T10:28:01.391197+0000","last_became_active":"2024-04-17T10:28:01.389732+0000","last_became_peered":"2024-04-17T10:28:01.389732+0000","last_unstale":"2024-04-17T10:28:01.391197+0000","last_undegraded":"2024-04-17T10:28:01.391197+0000","last_fullsized":"2024-04-17T10:28:01.391197+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:07.506032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381577+0000","last_change":"2024-04-17T10:28:01.381577+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381577+0000","last_undegraded":"2024-04-17T10:28:01.381577+0000","last_fullsized":"2024-04-17T10:28:01.381577+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384958+0000","last_change":"2024-04-17T10:28:01.380617+0000","last_active":"2024-04-17T10:28:01.384958+0000","last_peered":"2024-04-17T10:28:01.384958+0000","last_clean":"2024-04-17T10:28:01.384958+0000","last_became_active":"2024-04-17T10:28:01.380340+0000","last_became_peered":"2024-04-17T10:28:01.380340+0000","last_unstale":"2024-04-17T10:28:01.384958+0000","last_undegraded":"2024-04-17T10:28:01.384958+0000","last_fullsized":"2024-04-17T10:28:01.384958+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:58.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371208+0000","last_change":"2024-04-17T10:28:01.371208+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371208+0000","last_undegraded":"2024-04-17T10:28:01.371208+0000","last_fullsized":"2024-04-17T10:28:01.371208+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.395078+0000","last_change":"2024-04-17T10:28:01.393021+0000","last_active":"2024-04-17T10:28:01.395078+0000","last_peered":"2024-04-17T10:28:01.395078+0000","last_clean":"2024-04-17T10:28:01.395078+0000","last_became_active":"2024-04-17T10:28:01.392144+0000","last_became_peered":"2024-04-17T10:28:01.392144+0000","last_unstale":"2024-04-17T10:28:01.395078+0000","last_undegraded":"2024-04-17T10:28:01.395078+0000","last_fullsized":"2024-04-17T10:28:01.395078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.368273+0000","last_change":"2024-04-17T10:28:01.368273+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.368273+0000","last_undegraded":"2024-04-17T10:28:01.368273+0000","last_fullsized":"2024-04-17T10:28:01.368273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.389296+0000","last_change":"2024-04-17T10:28:01.384888+0000","last_active":"2024-04-17T10:28:01.389296+0000","last_peered":"2024-04-17T10:28:01.389296+0000","last_clean":"2024-04-17T10:28:01.389296+0000","last_became_active":"2024-04-17T10:28:01.384242+0000","last_became_peered":"2024-04-17T10:28:01.384242+0000","last_unstale":"2024-04-17T10:28:01.389296+0000","last_undegraded":"2024-04-17T10:28:01.389296+0000","last_fullsized":"2024-04-17T10:28:01.389296+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:15.827189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375458+0000","last_change":"2024-04-17T10:28:01.375458+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375458+0000","last_undegraded":"2024-04-17T10:28:01.375458+0000","last_fullsized":"2024-04-17T10:28:01.375458+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399097+0000","last_change":"2024-04-17T10:28:01.395507+0000","last_active":"2024-04-17T10:28:01.399097+0000","last_peered":"2024-04-17T10:28:01.399097+0000","last_clean":"2024-04-17T10:28:01.399097+0000","last_became_active":"2024-04-17T10:28:01.392288+0000","last_became_peered":"2024-04-17T10:28:01.392288+0000","last_unstale":"2024-04-17T10:28:01.399097+0000","last_undegraded":"2024-04-17T10:28:01.399097+0000","last_fullsized":"2024-04-17T10:28:01.399097+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:04:14.700196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380500+0000","last_change":"2024-04-17T10:28:01.380500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380500+0000","last_undegraded":"2024-04-17T10:28:01.380500+0000","last_fullsized":"2024-04-17T10:28:01.380500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394290+0000","last_change":"2024-04-17T10:28:01.391397+0000","last_active":"2024-04-17T10:28:01.394290+0000","last_peered":"2024-04-17T10:28:01.394290+0000","last_clean":"2024-04-17T10:28:01.394290+0000","last_became_active":"2024-04-17T10:28:01.391114+0000","last_became_peered":"2024-04-17T10:28:01.391114+0000","last_unstale":"2024-04-17T10:28:01.394290+0000","last_undegraded":"2024-04-17T10:28:01.394290+0000","last_fullsized":"2024-04-17T10:28:01.394290+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:23.592401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371189+0000","last_change":"2024-04-17T10:28:01.371189+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371189+0000","last_undegraded":"2024-04-17T10:28:01.371189+0000","last_fullsized":"2024-04-17T10:28:01.371189+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388236+0000","last_change":"2024-04-17T10:28:01.386653+0000","last_active":"2024-04-17T10:28:01.388236+0000","last_peered":"2024-04-17T10:28:01.388236+0000","last_clean":"2024-04-17T10:28:01.388236+0000","last_became_active":"2024-04-17T10:28:01.386406+0000","last_became_peered":"2024-04-17T10:28:01.386406+0000","last_unstale":"2024-04-17T10:28:01.388236+0000","last_undegraded":"2024-04-17T10:28:01.388236+0000","last_fullsized":"2024-04-17T10:28:01.388236+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:31.804436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378166+0000","last_change":"2024-04-17T10:28:01.378166+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378166+0000","last_undegraded":"2024-04-17T10:28:01.378166+0000","last_fullsized":"2024-04-17T10:28:01.378166+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391538+0000","last_change":"2024-04-17T10:28:01.389788+0000","last_active":"2024-04-17T10:28:01.391538+0000","last_peered":"2024-04-17T10:28:01.391538+0000","last_clean":"2024-04-17T10:28:01.391538+0000","last_became_active":"2024-04-17T10:28:01.389488+0000","last_became_peered":"2024-04-17T10:28:01.389488+0000","last_unstale":"2024-04-17T10:28:01.391538+0000","last_undegraded":"2024-04-17T10:28:01.391538+0000","last_fullsized":"2024-04-17T10:28:01.391538+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:19.391683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370896+0000","last_change":"2024-04-17T10:28:01.370896+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370896+0000","last_undegraded":"2024-04-17T10:28:01.370896+0000","last_fullsized":"2024-04-17T10:28:01.370896+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398827+0000","last_change":"2024-04-17T10:28:01.386384+0000","last_active":"2024-04-17T10:28:01.398827+0000","last_peered":"2024-04-17T10:28:01.398827+0000","last_clean":"2024-04-17T10:28:01.398827+0000","last_became_active":"2024-04-17T10:28:01.386102+0000","last_became_peered":"2024-04-17T10:28:01.386102+0000","last_unstale":"2024-04-17T10:28:01.398827+0000","last_undegraded":"2024-04-17T10:28:01.398827+0000","last_fullsized":"2024-04-17T10:28:01.398827+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375829+0000","last_change":"2024-04-17T10:28:01.375829+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375829+0000","last_undegraded":"2024-04-17T10:28:01.375829+0000","last_fullsized":"2024-04-17T10:28:01.375829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398627+0000","last_change":"2024-04-17T10:28:01.394707+0000","last_active":"2024-04-17T10:28:01.398627+0000","last_peered":"2024-04-17T10:28:01.398627+0000","last_clean":"2024-04-17T10:28:01.398627+0000","last_became_active":"2024-04-17T10:28:01.394419+0000","last_became_peered":"2024-04-17T10:28:01.394419+0000","last_unstale":"2024-04-17T10:28:01.398627+0000","last_undegraded":"2024-04-17T10:28:01.398627+0000","last_fullsized":"2024-04-17T10:28:01.398627+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:05:22.975781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371207+0000","last_change":"2024-04-17T10:28:01.371207+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371207+0000","last_undegraded":"2024-04-17T10:28:01.371207+0000","last_fullsized":"2024-04-17T10:28:01.371207+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.397867+0000","last_change":"2024-04-17T10:28:01.396212+0000","last_active":"2024-04-17T10:28:01.397867+0000","last_peered":"2024-04-17T10:28:01.397867+0000","last_clean":"2024-04-17T10:28:01.397867+0000","last_became_active":"2024-04-17T10:28:01.395939+0000","last_became_peered":"2024-04-17T10:28:01.395939+0000","last_unstale":"2024-04-17T10:28:01.397867+0000","last_undegraded":"2024-04-17T10:28:01.397867+0000","last_fullsized":"2024-04-17T10:28:01.397867+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:00.316480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376341+0000","last_change":"2024-04-17T10:28:01.376341+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376341+0000","last_undegraded":"2024-04-17T10:28:01.376341+0000","last_fullsized":"2024-04-17T10:28:01.376341+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.386497+0000","last_change":"2024-04-17T10:28:01.384533+0000","last_active":"2024-04-17T10:28:01.386497+0000","last_peered":"2024-04-17T10:28:01.386497+0000","last_clean":"2024-04-17T10:28:01.386497+0000","last_became_active":"2024-04-17T10:28:01.384284+0000","last_became_peered":"2024-04-17T10:28:01.384284+0000","last_unstale":"2024-04-17T10:28:01.386497+0000","last_undegraded":"2024-04-17T10:28:01.386497+0000","last_fullsized":"2024-04-17T10:28:01.386497+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371347+0000","last_change":"2024-04-17T10:28:01.371347+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371347+0000","last_undegraded":"2024-04-17T10:28:01.371347+0000","last_fullsized":"2024-04-17T10:28:01.371347+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391583+0000","last_change":"2024-04-17T10:28:01.388193+0000","last_active":"2024-04-17T10:28:01.391583+0000","last_peered":"2024-04-17T10:28:01.391583+0000","last_clean":"2024-04-17T10:28:01.391583+0000","last_became_active":"2024-04-17T10:28:01.387887+0000","last_became_peered":"2024-04-17T10:28:01.387887+0000","last_unstale":"2024-04-17T10:28:01.391583+0000","last_undegraded":"2024-04-17T10:28:01.391583+0000","last_fullsized":"2024-04-17T10:28:01.391583+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:12.842514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367353+0000","last_change":"2024-04-17T10:28:01.367353+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367353+0000","last_undegraded":"2024-04-17T10:28:01.367353+0000","last_fullsized":"2024-04-17T10:28:01.367353+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398900+0000","last_change":"2024-04-17T10:28:01.387103+0000","last_active":"2024-04-17T10:28:01.398900+0000","last_peered":"2024-04-17T10:28:01.398900+0000","last_clean":"2024-04-17T10:28:01.398900+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.398900+0000","last_undegraded":"2024-04-17T10:28:01.398900+0000","last_fullsized":"2024-04-17T10:28:01.398900+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381319+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:01.381319+0000","last_peered":"2024-04-17T10:28:01.381319+0000","last_clean":"2024-04-17T10:28:01.381319+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:01.381319+0000","last_undegraded":"2024-04-17T10:28:01.381319+0000","last_fullsized":"2024-04-17T10:28:01.381319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36528,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335312,"statfs":{"total":96636764160,"available":96599359488,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35952,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335888,"statfs":{"total":96636764160,"available":96599949312,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40104,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331736,"statfs":{"total":96636764160,"available":96595697664,"internally_reserved":0,"allocated":606208,"data_stored":270114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35956,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335884,"statfs":{"total":96636764160,"available":96599945216,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35944,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335896,"statfs":{"total":96636764160,"available":96599957504,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58732,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313108,"statfs":{"total":96636764160,"available":96576622592,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:28:06.887 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:28:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea20a00640 1 Processor -- start 2024-04-17T10:28:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea20a00640 1 -- start start 2024-04-17T10:28:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea20a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fea1c070470 0x7fea1c099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea20a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c0cf330 con 0x7fea1c06f770 2024-04-17T10:28:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea20a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1c0cf610 con 0x7fea1c099f90 2024-04-17T10:28:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea20a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1c09b190 con 0x7fea1c070470 2024-04-17T10:28:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea11e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fea1c070470 0x7fea1c099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea11e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fea1c070470 0x7fea1c099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52516/0 (socket says 172.21.15.67:52516) 2024-04-17T10:28:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea11e00640 1 -- 172.21.15.67:0/3215495517 learned_addr learned my addr 172.21.15.67:0/3215495517 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea1a800640 1 --1- 172.21.15.67:0/3215495517 >> v1:172.21.15.67:6789/0 conn(0x7fea1c06f770 0x7fea1c06fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53822/0 (socket says 172.21.15.67:53822) 2024-04-17T10:28:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea11e00640 1 -- 172.21.15.67:0/3215495517 >> v1:172.21.15.86:6789/0 conn(0x7fea1c099f90 legacy=0x7fea1c09a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea11e00640 1 -- 172.21.15.67:0/3215495517 >> v1:172.21.15.67:6789/0 conn(0x7fea1c06f770 legacy=0x7fea1c06fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.070+0000 7fea11e00640 1 -- 172.21.15.67:0/3215495517 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c09ac70 con 0x7fea1c070470 2024-04-17T10:28:07.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea11e00640 1 --2- 172.21.15.67:0/3215495517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c099a50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fea04009870 tx=0x7fea0401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccd0e499195fc1ce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea19e00640 1 -- 172.21.15.67:0/3215495517 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea040b0070 con 0x7fea1c070470 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea19e00640 1 -- 172.21.15.67:0/3215495517 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea040ab600 con 0x7fea1c070470 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea19e00640 1 -- 172.21.15.67:0/3215495517 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea040ab9d0 con 0x7fea1c070470 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 -- 172.21.15.67:0/3215495517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 msgr2=0x7fea1c099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 --2- 172.21.15.67:0/3215495517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c099a50 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fea04009870 tx=0x7fea0401c490 comp rx=0 tx=0).stop 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 -- 172.21.15.67:0/3215495517 shutdown_connections 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 --2- 172.21.15.67:0/3215495517 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c099a50 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 -- 172.21.15.67:0/3215495517 >> 172.21.15.67:0/3215495517 conn(0x7fea1c0d1630 msgr2=0x7fea1c0d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 -- 172.21.15.67:0/3215495517 shutdown_connections 2024-04-17T10:28:07.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 -- 172.21.15.67:0/3215495517 wait complete. 2024-04-17T10:28:07.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 Processor -- start 2024-04-17T10:28:07.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.074+0000 7fea20a00640 1 -- start start 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1c06f770 0x7fea1c157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c0c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1c099f90 0x7fea1c145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c169760 con 0x7fea1c070470 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c071610 con 0x7fea1c06f770 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1c1698e0 con 0x7fea1c099f90 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c0c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c0c8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52520/0 (socket says 172.21.15.67:52520) 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 -- 172.21.15.67:0/1874010151 learned_addr learned my addr 172.21.15.67:0/1874010151 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea1a800640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1c06f770 0x7fea1c157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea1b200640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1c099f90 0x7fea1c145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 -- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1c099f90 msgr2=0x7fea1c145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1c099f90 0x7fea1c145870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 -- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1c06f770 msgr2=0x7fea1c157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1c06f770 0x7fea1c157220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 -- 172.21.15.67:0/1874010151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c0c8c40 con 0x7fea1c070470 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea1b200640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1c099f90 0x7fea1c145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea11e00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c0c8700 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fea040190a0 tx=0x7fea04019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea13400640 1 -- 172.21.15.67:0/1874010151 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea040b9070 con 0x7fea1c070470 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea13400640 1 -- 172.21.15.67:0/1874010151 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea040b4400 con 0x7fea1c070470 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea13400640 1 -- 172.21.15.67:0/1874010151 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea040b0040 con 0x7fea1c070470 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1c0c8ec0 con 0x7fea1c070470 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1c0c9480 con 0x7fea1c070470 2024-04-17T10:28:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1c074dc0 con 0x7fea1c070470 2024-04-17T10:28:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.078+0000 7fea13400640 1 -- 172.21.15.67:0/1874010151 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fea040b33a0 con 0x7fea1c070470 2024-04-17T10:28:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.082+0000 7fea13400640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9f8066f60 0x7fe9f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.082+0000 7fea13400640 1 -- 172.21.15.67:0/1874010151 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fea0411e490 con 0x7fea1c070470 2024-04-17T10:28:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.082+0000 7fea1a800640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9f8066f60 0x7fe9f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.082+0000 7fea1a800640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9f8066f60 0x7fe9f8069420 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fea0c007fd0 tx=0x7fea0c005b80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.086+0000 7fea13400640 1 -- 172.21.15.67:0/1874010151 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fea1c0c8c40 con 0x7fea1c070470 2024-04-17T10:28:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.242+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea1c0ce360 con 0x7fe9f8066f60 2024-04-17T10:28:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.250+0000 7fea13400640 1 -- 172.21.15.67:0/1874010151 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7fea1c0ce360 con 0x7fe9f8066f60 2024-04-17T10:28:07.246 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9f8066f60 msgr2=0x7fe9f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9f8066f60 0x7fe9f8069420 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fea0c007fd0 tx=0x7fea0c005b80 comp rx=0 tx=0).stop 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 msgr2=0x7fea1c0c8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c0c8700 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fea040190a0 tx=0x7fea04019230 comp rx=0 tx=0).stop 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 shutdown_connections 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9f8066f60 0x7fe9f8069420 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fea1c099f90 0x7fea1c145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea1c070470 0x7fea1c0c8700 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 --2- 172.21.15.67:0/1874010151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fea1c06f770 0x7fea1c157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 >> 172.21.15.67:0/1874010151 conn(0x7fea1c0d1630 msgr2=0x7fea1c0ce880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 shutdown_connections 2024-04-17T10:28:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.254+0000 7fea20a00640 1 -- 172.21.15.67:0/1874010151 wait complete. 2024-04-17T10:28:07.267 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-17T10:28:06.177959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315112,"kb_used_data":5980,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754659608,"statfs":{"total":773094113280,"available":772771438592,"internally_reserved":0,"allocated":6123520,"data_stored":3388317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.817202"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380959+0000","last_change":"2024-04-17T10:28:01.380959+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380959+0000","last_undegraded":"2024-04-17T10:28:01.380959+0000","last_fullsized":"2024-04-17T10:28:01.380959+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376444+0000","last_change":"2024-04-17T10:28:01.376444+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376444+0000","last_undegraded":"2024-04-17T10:28:01.376444+0000","last_fullsized":"2024-04-17T10:28:01.376444+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378464+0000","last_change":"2024-04-17T10:28:01.378464+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378464+0000","last_undegraded":"2024-04-17T10:28:01.378464+0000","last_fullsized":"2024-04-17T10:28:01.378464+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377387+0000","last_change":"2024-04-17T10:28:01.377387+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377387+0000","last_undegraded":"2024-04-17T10:28:01.377387+0000","last_fullsized":"2024-04-17T10:28:01.377387+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374044+0000","last_change":"2024-04-17T10:28:01.374044+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374044+0000","last_undegraded":"2024-04-17T10:28:01.374044+0000","last_fullsized":"2024-04-17T10:28:01.374044+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378669+0000","last_change":"2024-04-17T10:28:01.378669+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378669+0000","last_undegraded":"2024-04-17T10:28:01.378669+0000","last_fullsized":"2024-04-17T10:28:01.378669+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377863+0000","last_change":"2024-04-17T10:28:01.377863+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377863+0000","last_undegraded":"2024-04-17T10:28:01.377863+0000","last_fullsized":"2024-04-17T10:28:01.377863+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370398+0000","last_change":"2024-04-17T10:28:01.370398+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370398+0000","last_undegraded":"2024-04-17T10:28:01.370398+0000","last_fullsized":"2024-04-17T10:28:01.370398+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371867+0000","last_change":"2024-04-17T10:28:01.371867+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371867+0000","last_undegraded":"2024-04-17T10:28:01.371867+0000","last_fullsized":"2024-04-17T10:28:01.371867+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379871+0000","last_change":"2024-04-17T10:28:01.379871+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379871+0000","last_undegraded":"2024-04-17T10:28:01.379871+0000","last_fullsized":"2024-04-17T10:28:01.379871+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379507+0000","last_change":"2024-04-17T10:28:01.379507+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379507+0000","last_undegraded":"2024-04-17T10:28:01.379507+0000","last_fullsized":"2024-04-17T10:28:01.379507+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371184+0000","last_change":"2024-04-17T10:28:01.371184+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371184+0000","last_undegraded":"2024-04-17T10:28:01.371184+0000","last_fullsized":"2024-04-17T10:28:01.371184+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381984+0000","last_change":"2024-04-17T10:28:01.381984+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381984+0000","last_undegraded":"2024-04-17T10:28:01.381984+0000","last_fullsized":"2024-04-17T10:28:01.381984+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377452+0000","last_change":"2024-04-17T10:28:01.377452+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377452+0000","last_undegraded":"2024-04-17T10:28:01.377452+0000","last_fullsized":"2024-04-17T10:28:01.377452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376914+0000","last_change":"2024-04-17T10:28:01.376914+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376914+0000","last_undegraded":"2024-04-17T10:28:01.376914+0000","last_fullsized":"2024-04-17T10:28:01.376914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376339+0000","last_change":"2024-04-17T10:28:01.376339+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376339+0000","last_undegraded":"2024-04-17T10:28:01.376339+0000","last_fullsized":"2024-04-17T10:28:01.376339+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379944+0000","last_change":"2024-04-17T10:28:01.379944+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379944+0000","last_undegraded":"2024-04-17T10:28:01.379944+0000","last_fullsized":"2024-04-17T10:28:01.379944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376011+0000","last_change":"2024-04-17T10:28:01.376011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376011+0000","last_undegraded":"2024-04-17T10:28:01.376011+0000","last_fullsized":"2024-04-17T10:28:01.376011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377320+0000","last_change":"2024-04-17T10:28:01.377320+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377320+0000","last_undegraded":"2024-04-17T10:28:01.377320+0000","last_fullsized":"2024-04-17T10:28:01.377320+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369232+0000","last_change":"2024-04-17T10:28:01.369232+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369232+0000","last_undegraded":"2024-04-17T10:28:01.369232+0000","last_fullsized":"2024-04-17T10:28:01.369232+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367428+0000","last_change":"2024-04-17T10:28:01.367428+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367428+0000","last_undegraded":"2024-04-17T10:28:01.367428+0000","last_fullsized":"2024-04-17T10:28:01.367428+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370644+0000","last_change":"2024-04-17T10:28:01.370644+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370644+0000","last_undegraded":"2024-04-17T10:28:01.370644+0000","last_fullsized":"2024-04-17T10:28:01.370644+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379078+0000","last_change":"2024-04-17T10:28:01.379078+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379078+0000","last_undegraded":"2024-04-17T10:28:01.379078+0000","last_fullsized":"2024-04-17T10:28:01.379078+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380365+0000","last_change":"2024-04-17T10:28:01.380365+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380365+0000","last_undegraded":"2024-04-17T10:28:01.380365+0000","last_fullsized":"2024-04-17T10:28:01.380365+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374358+0000","last_change":"2024-04-17T10:28:01.374358+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374358+0000","last_undegraded":"2024-04-17T10:28:01.374358+0000","last_fullsized":"2024-04-17T10:28:01.374358+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.383292+0000","last_change":"2024-04-17T10:28:01.383292+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.383292+0000","last_undegraded":"2024-04-17T10:28:01.383292+0000","last_fullsized":"2024-04-17T10:28:01.383292+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367828+0000","last_change":"2024-04-17T10:28:01.367828+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367828+0000","last_undegraded":"2024-04-17T10:28:01.367828+0000","last_fullsized":"2024-04-17T10:28:01.367828+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377527+0000","last_change":"2024-04-17T10:28:01.377527+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377527+0000","last_undegraded":"2024-04-17T10:28:01.377527+0000","last_fullsized":"2024-04-17T10:28:01.377527+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379750+0000","last_change":"2024-04-17T10:28:01.379750+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379750+0000","last_undegraded":"2024-04-17T10:28:01.379750+0000","last_fullsized":"2024-04-17T10:28:01.379750+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376427+0000","last_change":"2024-04-17T10:28:01.376427+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376427+0000","last_undegraded":"2024-04-17T10:28:01.376427+0000","last_fullsized":"2024-04-17T10:28:01.376427+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372163+0000","last_change":"2024-04-17T10:28:01.372163+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372163+0000","last_undegraded":"2024-04-17T10:28:01.372163+0000","last_fullsized":"2024-04-17T10:28:01.372163+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377077+0000","last_change":"2024-04-17T10:28:01.377077+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377077+0000","last_undegraded":"2024-04-17T10:28:01.377077+0000","last_fullsized":"2024-04-17T10:28:01.377077+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384066+0000","last_change":"2024-04-17T10:28:01.380916+0000","last_active":"2024-04-17T10:28:01.384066+0000","last_peered":"2024-04-17T10:28:01.384066+0000","last_clean":"2024-04-17T10:28:01.384066+0000","last_became_active":"2024-04-17T10:28:01.380635+0000","last_became_peered":"2024-04-17T10:28:01.380635+0000","last_unstale":"2024-04-17T10:28:01.384066+0000","last_undegraded":"2024-04-17T10:28:01.384066+0000","last_fullsized":"2024-04-17T10:28:01.384066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:47:03.828693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371180+0000","last_change":"2024-04-17T10:28:01.371180+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371180+0000","last_undegraded":"2024-04-17T10:28:01.371180+0000","last_fullsized":"2024-04-17T10:28:01.371180+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394963+0000","last_change":"2024-04-17T10:28:01.391871+0000","last_active":"2024-04-17T10:28:01.394963+0000","last_peered":"2024-04-17T10:28:01.394963+0000","last_clean":"2024-04-17T10:28:01.394963+0000","last_became_active":"2024-04-17T10:28:01.391540+0000","last_became_peered":"2024-04-17T10:28:01.391540+0000","last_unstale":"2024-04-17T10:28:01.394963+0000","last_undegraded":"2024-04-17T10:28:01.394963+0000","last_fullsized":"2024-04-17T10:28:01.394963+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:03:16.384430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367514+0000","last_change":"2024-04-17T10:28:01.367514+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367514+0000","last_undegraded":"2024-04-17T10:28:01.367514+0000","last_fullsized":"2024-04-17T10:28:01.367514+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394726+0000","last_change":"2024-04-17T10:28:01.392616+0000","last_active":"2024-04-17T10:28:01.394726+0000","last_peered":"2024-04-17T10:28:01.394726+0000","last_clean":"2024-04-17T10:28:01.394726+0000","last_became_active":"2024-04-17T10:28:01.391986+0000","last_became_peered":"2024-04-17T10:28:01.391986+0000","last_unstale":"2024-04-17T10:28:01.394726+0000","last_undegraded":"2024-04-17T10:28:01.394726+0000","last_fullsized":"2024-04-17T10:28:01.394726+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371610+0000","last_change":"2024-04-17T10:28:01.371610+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371610+0000","last_undegraded":"2024-04-17T10:28:01.371610+0000","last_fullsized":"2024-04-17T10:28:01.371610+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383990+0000","last_change":"2024-04-17T10:28:01.380404+0000","last_active":"2024-04-17T10:28:01.383990+0000","last_peered":"2024-04-17T10:28:01.383990+0000","last_clean":"2024-04-17T10:28:01.383990+0000","last_became_active":"2024-04-17T10:28:01.380065+0000","last_became_peered":"2024-04-17T10:28:01.380065+0000","last_unstale":"2024-04-17T10:28:01.383990+0000","last_undegraded":"2024-04-17T10:28:01.383990+0000","last_fullsized":"2024-04-17T10:28:01.383990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:12.355937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375461+0000","last_change":"2024-04-17T10:28:01.375461+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375461+0000","last_undegraded":"2024-04-17T10:28:01.375461+0000","last_fullsized":"2024-04-17T10:28:01.375461+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399062+0000","last_change":"2024-04-17T10:28:01.395609+0000","last_active":"2024-04-17T10:28:01.399062+0000","last_peered":"2024-04-17T10:28:01.399062+0000","last_clean":"2024-04-17T10:28:01.399062+0000","last_became_active":"2024-04-17T10:28:01.395267+0000","last_became_peered":"2024-04-17T10:28:01.395267+0000","last_unstale":"2024-04-17T10:28:01.399062+0000","last_undegraded":"2024-04-17T10:28:01.399062+0000","last_fullsized":"2024-04-17T10:28:01.399062+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:42.740201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377612+0000","last_change":"2024-04-17T10:28:01.377612+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377612+0000","last_undegraded":"2024-04-17T10:28:01.377612+0000","last_fullsized":"2024-04-17T10:28:01.377612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383876+0000","last_change":"2024-04-17T10:28:01.378483+0000","last_active":"2024-04-17T10:28:01.383876+0000","last_peered":"2024-04-17T10:28:01.383876+0000","last_clean":"2024-04-17T10:28:01.383876+0000","last_became_active":"2024-04-17T10:28:01.378140+0000","last_became_peered":"2024-04-17T10:28:01.378140+0000","last_unstale":"2024-04-17T10:28:01.383876+0000","last_undegraded":"2024-04-17T10:28:01.383876+0000","last_fullsized":"2024-04-17T10:28:01.383876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371666+0000","last_change":"2024-04-17T10:28:01.371666+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371666+0000","last_undegraded":"2024-04-17T10:28:01.371666+0000","last_fullsized":"2024-04-17T10:28:01.371666+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383084+0000","last_change":"2024-04-17T10:28:01.378144+0000","last_active":"2024-04-17T10:28:01.383084+0000","last_peered":"2024-04-17T10:28:01.383084+0000","last_clean":"2024-04-17T10:28:01.383084+0000","last_became_active":"2024-04-17T10:28:01.377879+0000","last_became_peered":"2024-04-17T10:28:01.377879+0000","last_unstale":"2024-04-17T10:28:01.383084+0000","last_undegraded":"2024-04-17T10:28:01.383084+0000","last_fullsized":"2024-04-17T10:28:01.383084+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:02.479603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378313+0000","last_change":"2024-04-17T10:28:01.378313+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378313+0000","last_undegraded":"2024-04-17T10:28:01.378313+0000","last_fullsized":"2024-04-17T10:28:01.378313+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381525+0000","last_change":"2024-04-17T10:28:01.376007+0000","last_active":"2024-04-17T10:28:01.381525+0000","last_peered":"2024-04-17T10:28:01.381525+0000","last_clean":"2024-04-17T10:28:01.381525+0000","last_became_active":"2024-04-17T10:28:01.375098+0000","last_became_peered":"2024-04-17T10:28:01.375098+0000","last_unstale":"2024-04-17T10:28:01.381525+0000","last_undegraded":"2024-04-17T10:28:01.381525+0000","last_fullsized":"2024-04-17T10:28:01.381525+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372934+0000","last_change":"2024-04-17T10:28:01.372934+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372934+0000","last_undegraded":"2024-04-17T10:28:01.372934+0000","last_fullsized":"2024-04-17T10:28:01.372934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394069+0000","last_change":"2024-04-17T10:28:01.390159+0000","last_active":"2024-04-17T10:28:01.394069+0000","last_peered":"2024-04-17T10:28:01.394069+0000","last_clean":"2024-04-17T10:28:01.394069+0000","last_became_active":"2024-04-17T10:28:01.389874+0000","last_became_peered":"2024-04-17T10:28:01.389874+0000","last_unstale":"2024-04-17T10:28:01.394069+0000","last_undegraded":"2024-04-17T10:28:01.394069+0000","last_fullsized":"2024-04-17T10:28:01.394069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:59.077053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372104+0000","last_change":"2024-04-17T10:28:01.372104+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372104+0000","last_undegraded":"2024-04-17T10:28:01.372104+0000","last_fullsized":"2024-04-17T10:28:01.372104+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396819+0000","last_change":"2024-04-17T10:28:01.396322+0000","last_active":"2024-04-17T10:28:01.396819+0000","last_peered":"2024-04-17T10:28:01.396819+0000","last_clean":"2024-04-17T10:28:01.396819+0000","last_became_active":"2024-04-17T10:28:01.396090+0000","last_became_peered":"2024-04-17T10:28:01.396090+0000","last_unstale":"2024-04-17T10:28:01.396819+0000","last_undegraded":"2024-04-17T10:28:01.396819+0000","last_fullsized":"2024-04-17T10:28:01.396819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394086+0000","last_change":"2024-04-17T10:28:01.390653+0000","last_active":"2024-04-17T10:28:01.394086+0000","last_peered":"2024-04-17T10:28:01.394086+0000","last_clean":"2024-04-17T10:28:01.394086+0000","last_became_active":"2024-04-17T10:28:01.390327+0000","last_became_peered":"2024-04-17T10:28:01.390327+0000","last_unstale":"2024-04-17T10:28:01.394086+0000","last_undegraded":"2024-04-17T10:28:01.394086+0000","last_fullsized":"2024-04-17T10:28:01.394086+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:12.074317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373570+0000","last_change":"2024-04-17T10:28:01.373570+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373570+0000","last_undegraded":"2024-04-17T10:28:01.373570+0000","last_fullsized":"2024-04-17T10:28:01.373570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.393035+0000","last_change":"2024-04-17T10:28:01.390744+0000","last_active":"2024-04-17T10:28:01.393035+0000","last_peered":"2024-04-17T10:28:01.393035+0000","last_clean":"2024-04-17T10:28:01.393035+0000","last_became_active":"2024-04-17T10:28:01.390487+0000","last_became_peered":"2024-04-17T10:28:01.390487+0000","last_unstale":"2024-04-17T10:28:01.393035+0000","last_undegraded":"2024-04-17T10:28:01.393035+0000","last_fullsized":"2024-04-17T10:28:01.393035+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:21.812637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371710+0000","last_change":"2024-04-17T10:28:01.371710+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371710+0000","last_undegraded":"2024-04-17T10:28:01.371710+0000","last_fullsized":"2024-04-17T10:28:01.371710+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399042+0000","last_change":"2024-04-17T10:28:01.388726+0000","last_active":"2024-04-17T10:28:01.399042+0000","last_peered":"2024-04-17T10:28:01.399042+0000","last_clean":"2024-04-17T10:28:01.399042+0000","last_became_active":"2024-04-17T10:28:01.388462+0000","last_became_peered":"2024-04-17T10:28:01.388462+0000","last_unstale":"2024-04-17T10:28:01.399042+0000","last_undegraded":"2024-04-17T10:28:01.399042+0000","last_fullsized":"2024-04-17T10:28:01.399042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376893+0000","last_change":"2024-04-17T10:28:01.376893+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376893+0000","last_undegraded":"2024-04-17T10:28:01.376893+0000","last_fullsized":"2024-04-17T10:28:01.376893+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.390536+0000","last_change":"2024-04-17T10:28:01.388333+0000","last_active":"2024-04-17T10:28:01.390536+0000","last_peered":"2024-04-17T10:28:01.390536+0000","last_clean":"2024-04-17T10:28:01.390536+0000","last_became_active":"2024-04-17T10:28:01.387931+0000","last_became_peered":"2024-04-17T10:28:01.387931+0000","last_unstale":"2024-04-17T10:28:01.390536+0000","last_undegraded":"2024-04-17T10:28:01.390536+0000","last_fullsized":"2024-04-17T10:28:01.390536+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:28:46.889727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372115+0000","last_change":"2024-04-17T10:28:01.372115+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372115+0000","last_undegraded":"2024-04-17T10:28:01.372115+0000","last_fullsized":"2024-04-17T10:28:01.372115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388876+0000","last_change":"2024-04-17T10:28:01.387372+0000","last_active":"2024-04-17T10:28:01.388876+0000","last_peered":"2024-04-17T10:28:01.388876+0000","last_clean":"2024-04-17T10:28:01.388876+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.388876+0000","last_undegraded":"2024-04-17T10:28:01.388876+0000","last_fullsized":"2024-04-17T10:28:01.388876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:31.085374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376533+0000","last_change":"2024-04-17T10:28:01.376533+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376533+0000","last_undegraded":"2024-04-17T10:28:01.376533+0000","last_fullsized":"2024-04-17T10:28:01.376533+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391514+0000","last_change":"2024-04-17T10:28:01.390499+0000","last_active":"2024-04-17T10:28:01.391514+0000","last_peered":"2024-04-17T10:28:01.391514+0000","last_clean":"2024-04-17T10:28:01.391514+0000","last_became_active":"2024-04-17T10:28:01.390208+0000","last_became_peered":"2024-04-17T10:28:01.390208+0000","last_unstale":"2024-04-17T10:28:01.391514+0000","last_undegraded":"2024-04-17T10:28:01.391514+0000","last_fullsized":"2024-04-17T10:28:01.391514+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:44.289510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369897+0000","last_change":"2024-04-17T10:28:01.369897+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369897+0000","last_undegraded":"2024-04-17T10:28:01.369897+0000","last_fullsized":"2024-04-17T10:28:01.369897+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398772+0000","last_change":"2024-04-17T10:28:01.385970+0000","last_active":"2024-04-17T10:28:01.398772+0000","last_peered":"2024-04-17T10:28:01.398772+0000","last_clean":"2024-04-17T10:28:01.398772+0000","last_became_active":"2024-04-17T10:28:01.385447+0000","last_became_peered":"2024-04-17T10:28:01.385447+0000","last_unstale":"2024-04-17T10:28:01.398772+0000","last_undegraded":"2024-04-17T10:28:01.398772+0000","last_fullsized":"2024-04-17T10:28:01.398772+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378500+0000","last_change":"2024-04-17T10:28:01.378500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378500+0000","last_undegraded":"2024-04-17T10:28:01.378500+0000","last_fullsized":"2024-04-17T10:28:01.378500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396318+0000","last_change":"2024-04-17T10:28:01.393600+0000","last_active":"2024-04-17T10:28:01.396318+0000","last_peered":"2024-04-17T10:28:01.396318+0000","last_clean":"2024-04-17T10:28:01.396318+0000","last_became_active":"2024-04-17T10:28:01.393277+0000","last_became_peered":"2024-04-17T10:28:01.393277+0000","last_unstale":"2024-04-17T10:28:01.396318+0000","last_undegraded":"2024-04-17T10:28:01.396318+0000","last_fullsized":"2024-04-17T10:28:01.396318+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:55:03.162699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373787+0000","last_change":"2024-04-17T10:28:01.373787+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373787+0000","last_undegraded":"2024-04-17T10:28:01.373787+0000","last_fullsized":"2024-04-17T10:28:01.373787+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391197+0000","last_change":"2024-04-17T10:28:01.390060+0000","last_active":"2024-04-17T10:28:01.391197+0000","last_peered":"2024-04-17T10:28:01.391197+0000","last_clean":"2024-04-17T10:28:01.391197+0000","last_became_active":"2024-04-17T10:28:01.389732+0000","last_became_peered":"2024-04-17T10:28:01.389732+0000","last_unstale":"2024-04-17T10:28:01.391197+0000","last_undegraded":"2024-04-17T10:28:01.391197+0000","last_fullsized":"2024-04-17T10:28:01.391197+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:07.506032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381577+0000","last_change":"2024-04-17T10:28:01.381577+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381577+0000","last_undegraded":"2024-04-17T10:28:01.381577+0000","last_fullsized":"2024-04-17T10:28:01.381577+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384958+0000","last_change":"2024-04-17T10:28:01.380617+0000","last_active":"2024-04-17T10:28:01.384958+0000","last_peered":"2024-04-17T10:28:01.384958+0000","last_clean":"2024-04-17T10:28:01.384958+0000","last_became_active":"2024-04-17T10:28:01.380340+0000","last_became_peered":"2024-04-17T10:28:01.380340+0000","last_unstale":"2024-04-17T10:28:01.384958+0000","last_undegraded":"2024-04-17T10:28:01.384958+0000","last_fullsized":"2024-04-17T10:28:01.384958+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:58.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371208+0000","last_change":"2024-04-17T10:28:01.371208+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371208+0000","last_undegraded":"2024-04-17T10:28:01.371208+0000","last_fullsized":"2024-04-17T10:28:01.371208+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.395078+0000","last_change":"2024-04-17T10:28:01.393021+0000","last_active":"2024-04-17T10:28:01.395078+0000","last_peered":"2024-04-17T10:28:01.395078+0000","last_clean":"2024-04-17T10:28:01.395078+0000","last_became_active":"2024-04-17T10:28:01.392144+0000","last_became_peered":"2024-04-17T10:28:01.392144+0000","last_unstale":"2024-04-17T10:28:01.395078+0000","last_undegraded":"2024-04-17T10:28:01.395078+0000","last_fullsized":"2024-04-17T10:28:01.395078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.368273+0000","last_change":"2024-04-17T10:28:01.368273+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.368273+0000","last_undegraded":"2024-04-17T10:28:01.368273+0000","last_fullsized":"2024-04-17T10:28:01.368273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.389296+0000","last_change":"2024-04-17T10:28:01.384888+0000","last_active":"2024-04-17T10:28:01.389296+0000","last_peered":"2024-04-17T10:28:01.389296+0000","last_clean":"2024-04-17T10:28:01.389296+0000","last_became_active":"2024-04-17T10:28:01.384242+0000","last_became_peered":"2024-04-17T10:28:01.384242+0000","last_unstale":"2024-04-17T10:28:01.389296+0000","last_undegraded":"2024-04-17T10:28:01.389296+0000","last_fullsized":"2024-04-17T10:28:01.389296+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:15.827189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375458+0000","last_change":"2024-04-17T10:28:01.375458+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375458+0000","last_undegraded":"2024-04-17T10:28:01.375458+0000","last_fullsized":"2024-04-17T10:28:01.375458+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399097+0000","last_change":"2024-04-17T10:28:01.395507+0000","last_active":"2024-04-17T10:28:01.399097+0000","last_peered":"2024-04-17T10:28:01.399097+0000","last_clean":"2024-04-17T10:28:01.399097+0000","last_became_active":"2024-04-17T10:28:01.392288+0000","last_became_peered":"2024-04-17T10:28:01.392288+0000","last_unstale":"2024-04-17T10:28:01.399097+0000","last_undegraded":"2024-04-17T10:28:01.399097+0000","last_fullsized":"2024-04-17T10:28:01.399097+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:04:14.700196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380500+0000","last_change":"2024-04-17T10:28:01.380500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380500+0000","last_undegraded":"2024-04-17T10:28:01.380500+0000","last_fullsized":"2024-04-17T10:28:01.380500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394290+0000","last_change":"2024-04-17T10:28:01.391397+0000","last_active":"2024-04-17T10:28:01.394290+0000","last_peered":"2024-04-17T10:28:01.394290+0000","last_clean":"2024-04-17T10:28:01.394290+0000","last_became_active":"2024-04-17T10:28:01.391114+0000","last_became_peered":"2024-04-17T10:28:01.391114+0000","last_unstale":"2024-04-17T10:28:01.394290+0000","last_undegraded":"2024-04-17T10:28:01.394290+0000","last_fullsized":"2024-04-17T10:28:01.394290+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:23.592401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371189+0000","last_change":"2024-04-17T10:28:01.371189+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371189+0000","last_undegraded":"2024-04-17T10:28:01.371189+0000","last_fullsized":"2024-04-17T10:28:01.371189+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388236+0000","last_change":"2024-04-17T10:28:01.386653+0000","last_active":"2024-04-17T10:28:01.388236+0000","last_peered":"2024-04-17T10:28:01.388236+0000","last_clean":"2024-04-17T10:28:01.388236+0000","last_became_active":"2024-04-17T10:28:01.386406+0000","last_became_peered":"2024-04-17T10:28:01.386406+0000","last_unstale":"2024-04-17T10:28:01.388236+0000","last_undegraded":"2024-04-17T10:28:01.388236+0000","last_fullsized":"2024-04-17T10:28:01.388236+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:31.804436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378166+0000","last_change":"2024-04-17T10:28:01.378166+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378166+0000","last_undegraded":"2024-04-17T10:28:01.378166+0000","last_fullsized":"2024-04-17T10:28:01.378166+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391538+0000","last_change":"2024-04-17T10:28:01.389788+0000","last_active":"2024-04-17T10:28:01.391538+0000","last_peered":"2024-04-17T10:28:01.391538+0000","last_clean":"2024-04-17T10:28:01.391538+0000","last_became_active":"2024-04-17T10:28:01.389488+0000","last_became_peered":"2024-04-17T10:28:01.389488+0000","last_unstale":"2024-04-17T10:28:01.391538+0000","last_undegraded":"2024-04-17T10:28:01.391538+0000","last_fullsized":"2024-04-17T10:28:01.391538+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:19.391683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370896+0000","last_change":"2024-04-17T10:28:01.370896+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370896+0000","last_undegraded":"2024-04-17T10:28:01.370896+0000","last_fullsized":"2024-04-17T10:28:01.370896+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398827+0000","last_change":"2024-04-17T10:28:01.386384+0000","last_active":"2024-04-17T10:28:01.398827+0000","last_peered":"2024-04-17T10:28:01.398827+0000","last_clean":"2024-04-17T10:28:01.398827+0000","last_became_active":"2024-04-17T10:28:01.386102+0000","last_became_peered":"2024-04-17T10:28:01.386102+0000","last_unstale":"2024-04-17T10:28:01.398827+0000","last_undegraded":"2024-04-17T10:28:01.398827+0000","last_fullsized":"2024-04-17T10:28:01.398827+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375829+0000","last_change":"2024-04-17T10:28:01.375829+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375829+0000","last_undegraded":"2024-04-17T10:28:01.375829+0000","last_fullsized":"2024-04-17T10:28:01.375829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398627+0000","last_change":"2024-04-17T10:28:01.394707+0000","last_active":"2024-04-17T10:28:01.398627+0000","last_peered":"2024-04-17T10:28:01.398627+0000","last_clean":"2024-04-17T10:28:01.398627+0000","last_became_active":"2024-04-17T10:28:01.394419+0000","last_became_peered":"2024-04-17T10:28:01.394419+0000","last_unstale":"2024-04-17T10:28:01.398627+0000","last_undegraded":"2024-04-17T10:28:01.398627+0000","last_fullsized":"2024-04-17T10:28:01.398627+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:05:22.975781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371207+0000","last_change":"2024-04-17T10:28:01.371207+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371207+0000","last_undegraded":"2024-04-17T10:28:01.371207+0000","last_fullsized":"2024-04-17T10:28:01.371207+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.397867+0000","last_change":"2024-04-17T10:28:01.396212+0000","last_active":"2024-04-17T10:28:01.397867+0000","last_peered":"2024-04-17T10:28:01.397867+0000","last_clean":"2024-04-17T10:28:01.397867+0000","last_became_active":"2024-04-17T10:28:01.395939+0000","last_became_peered":"2024-04-17T10:28:01.395939+0000","last_unstale":"2024-04-17T10:28:01.397867+0000","last_undegraded":"2024-04-17T10:28:01.397867+0000","last_fullsized":"2024-04-17T10:28:01.397867+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:00.316480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376341+0000","last_change":"2024-04-17T10:28:01.376341+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376341+0000","last_undegraded":"2024-04-17T10:28:01.376341+0000","last_fullsized":"2024-04-17T10:28:01.376341+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.386497+0000","last_change":"2024-04-17T10:28:01.384533+0000","last_active":"2024-04-17T10:28:01.386497+0000","last_peered":"2024-04-17T10:28:01.386497+0000","last_clean":"2024-04-17T10:28:01.386497+0000","last_became_active":"2024-04-17T10:28:01.384284+0000","last_became_peered":"2024-04-17T10:28:01.384284+0000","last_unstale":"2024-04-17T10:28:01.386497+0000","last_undegraded":"2024-04-17T10:28:01.386497+0000","last_fullsized":"2024-04-17T10:28:01.386497+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371347+0000","last_change":"2024-04-17T10:28:01.371347+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371347+0000","last_undegraded":"2024-04-17T10:28:01.371347+0000","last_fullsized":"2024-04-17T10:28:01.371347+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391583+0000","last_change":"2024-04-17T10:28:01.388193+0000","last_active":"2024-04-17T10:28:01.391583+0000","last_peered":"2024-04-17T10:28:01.391583+0000","last_clean":"2024-04-17T10:28:01.391583+0000","last_became_active":"2024-04-17T10:28:01.387887+0000","last_became_peered":"2024-04-17T10:28:01.387887+0000","last_unstale":"2024-04-17T10:28:01.391583+0000","last_undegraded":"2024-04-17T10:28:01.391583+0000","last_fullsized":"2024-04-17T10:28:01.391583+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:12.842514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367353+0000","last_change":"2024-04-17T10:28:01.367353+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367353+0000","last_undegraded":"2024-04-17T10:28:01.367353+0000","last_fullsized":"2024-04-17T10:28:01.367353+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398900+0000","last_change":"2024-04-17T10:28:01.387103+0000","last_active":"2024-04-17T10:28:01.398900+0000","last_peered":"2024-04-17T10:28:01.398900+0000","last_clean":"2024-04-17T10:28:01.398900+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.398900+0000","last_undegraded":"2024-04-17T10:28:01.398900+0000","last_fullsized":"2024-04-17T10:28:01.398900+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381319+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:01.381319+0000","last_peered":"2024-04-17T10:28:01.381319+0000","last_clean":"2024-04-17T10:28:01.381319+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:01.381319+0000","last_undegraded":"2024-04-17T10:28:01.381319+0000","last_fullsized":"2024-04-17T10:28:01.381319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36528,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335312,"statfs":{"total":96636764160,"available":96599359488,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35952,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335888,"statfs":{"total":96636764160,"available":96599949312,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40104,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331736,"statfs":{"total":96636764160,"available":96595697664,"internally_reserved":0,"allocated":606208,"data_stored":270114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35956,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335884,"statfs":{"total":96636764160,"available":96599945216,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35944,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335896,"statfs":{"total":96636764160,"available":96599957504,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58732,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313108,"statfs":{"total":96636764160,"available":96576622592,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:28:07.271 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3f is not active+clean 2024-04-17T10:28:07.271 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.380959+0000', 'last_change': '2024-04-17T10:28:01.380959+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.380959+0000', 'last_undegraded': '2024-04-17T10:28:01.380959+0000', 'last_fullsized': '2024-04-17T10:28:01.380959+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3e is not active+clean 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376444+0000', 'last_change': '2024-04-17T10:28:01.376444+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376444+0000', 'last_undegraded': '2024-04-17T10:28:01.376444+0000', 'last_fullsized': '2024-04-17T10:28:01.376444+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3d is not active+clean 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.378464+0000', 'last_change': '2024-04-17T10:28:01.378464+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.378464+0000', 'last_undegraded': '2024-04-17T10:28:01.378464+0000', 'last_fullsized': '2024-04-17T10:28:01.378464+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3c is not active+clean 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377387+0000', 'last_change': '2024-04-17T10:28:01.377387+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377387+0000', 'last_undegraded': '2024-04-17T10:28:01.377387+0000', 'last_fullsized': '2024-04-17T10:28:01.377387+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3b is not active+clean 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.374044+0000', 'last_change': '2024-04-17T10:28:01.374044+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.374044+0000', 'last_undegraded': '2024-04-17T10:28:01.374044+0000', 'last_fullsized': '2024-04-17T10:28:01.374044+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3a is not active+clean 2024-04-17T10:28:07.272 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.378669+0000', 'last_change': '2024-04-17T10:28:01.378669+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.378669+0000', 'last_undegraded': '2024-04-17T10:28:01.378669+0000', 'last_fullsized': '2024-04-17T10:28:01.378669+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.39 is not active+clean 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377863+0000', 'last_change': '2024-04-17T10:28:01.377863+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377863+0000', 'last_undegraded': '2024-04-17T10:28:01.377863+0000', 'last_fullsized': '2024-04-17T10:28:01.377863+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.38 is not active+clean 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.370398+0000', 'last_change': '2024-04-17T10:28:01.370398+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.370398+0000', 'last_undegraded': '2024-04-17T10:28:01.370398+0000', 'last_fullsized': '2024-04-17T10:28:01.370398+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.37 is not active+clean 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371867+0000', 'last_change': '2024-04-17T10:28:01.371867+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371867+0000', 'last_undegraded': '2024-04-17T10:28:01.371867+0000', 'last_fullsized': '2024-04-17T10:28:01.371867+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.36 is not active+clean 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.379871+0000', 'last_change': '2024-04-17T10:28:01.379871+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.379871+0000', 'last_undegraded': '2024-04-17T10:28:01.379871+0000', 'last_fullsized': '2024-04-17T10:28:01.379871+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.35 is not active+clean 2024-04-17T10:28:07.273 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.379507+0000', 'last_change': '2024-04-17T10:28:01.379507+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.379507+0000', 'last_undegraded': '2024-04-17T10:28:01.379507+0000', 'last_fullsized': '2024-04-17T10:28:01.379507+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.34 is not active+clean 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371184+0000', 'last_change': '2024-04-17T10:28:01.371184+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371184+0000', 'last_undegraded': '2024-04-17T10:28:01.371184+0000', 'last_fullsized': '2024-04-17T10:28:01.371184+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.33 is not active+clean 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.381984+0000', 'last_change': '2024-04-17T10:28:01.381984+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.381984+0000', 'last_undegraded': '2024-04-17T10:28:01.381984+0000', 'last_fullsized': '2024-04-17T10:28:01.381984+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.32 is not active+clean 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377452+0000', 'last_change': '2024-04-17T10:28:01.377452+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377452+0000', 'last_undegraded': '2024-04-17T10:28:01.377452+0000', 'last_fullsized': '2024-04-17T10:28:01.377452+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.31 is not active+clean 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376914+0000', 'last_change': '2024-04-17T10:28:01.376914+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376914+0000', 'last_undegraded': '2024-04-17T10:28:01.376914+0000', 'last_fullsized': '2024-04-17T10:28:01.376914+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.30 is not active+clean 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376339+0000', 'last_change': '2024-04-17T10:28:01.376339+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376339+0000', 'last_undegraded': '2024-04-17T10:28:01.376339+0000', 'last_fullsized': '2024-04-17T10:28:01.376339+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.274 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2f is not active+clean 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.379944+0000', 'last_change': '2024-04-17T10:28:01.379944+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.379944+0000', 'last_undegraded': '2024-04-17T10:28:01.379944+0000', 'last_fullsized': '2024-04-17T10:28:01.379944+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2e is not active+clean 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376011+0000', 'last_change': '2024-04-17T10:28:01.376011+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376011+0000', 'last_undegraded': '2024-04-17T10:28:01.376011+0000', 'last_fullsized': '2024-04-17T10:28:01.376011+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2d is not active+clean 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377320+0000', 'last_change': '2024-04-17T10:28:01.377320+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377320+0000', 'last_undegraded': '2024-04-17T10:28:01.377320+0000', 'last_fullsized': '2024-04-17T10:28:01.377320+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2c is not active+clean 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.369232+0000', 'last_change': '2024-04-17T10:28:01.369232+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.369232+0000', 'last_undegraded': '2024-04-17T10:28:01.369232+0000', 'last_fullsized': '2024-04-17T10:28:01.369232+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2b is not active+clean 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.367428+0000', 'last_change': '2024-04-17T10:28:01.367428+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.367428+0000', 'last_undegraded': '2024-04-17T10:28:01.367428+0000', 'last_fullsized': '2024-04-17T10:28:01.367428+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2a is not active+clean 2024-04-17T10:28:07.275 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.370644+0000', 'last_change': '2024-04-17T10:28:01.370644+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.370644+0000', 'last_undegraded': '2024-04-17T10:28:01.370644+0000', 'last_fullsized': '2024-04-17T10:28:01.370644+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.29 is not active+clean 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.379078+0000', 'last_change': '2024-04-17T10:28:01.379078+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.379078+0000', 'last_undegraded': '2024-04-17T10:28:01.379078+0000', 'last_fullsized': '2024-04-17T10:28:01.379078+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.28 is not active+clean 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.380365+0000', 'last_change': '2024-04-17T10:28:01.380365+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.380365+0000', 'last_undegraded': '2024-04-17T10:28:01.380365+0000', 'last_fullsized': '2024-04-17T10:28:01.380365+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.27 is not active+clean 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377011+0000', 'last_change': '2024-04-17T10:28:01.377011+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377011+0000', 'last_undegraded': '2024-04-17T10:28:01.377011+0000', 'last_fullsized': '2024-04-17T10:28:01.377011+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.26 is not active+clean 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.374358+0000', 'last_change': '2024-04-17T10:28:01.374358+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.374358+0000', 'last_undegraded': '2024-04-17T10:28:01.374358+0000', 'last_fullsized': '2024-04-17T10:28:01.374358+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.25 is not active+clean 2024-04-17T10:28:07.276 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.383292+0000', 'last_change': '2024-04-17T10:28:01.383292+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.383292+0000', 'last_undegraded': '2024-04-17T10:28:01.383292+0000', 'last_fullsized': '2024-04-17T10:28:01.383292+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.24 is not active+clean 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.367828+0000', 'last_change': '2024-04-17T10:28:01.367828+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.367828+0000', 'last_undegraded': '2024-04-17T10:28:01.367828+0000', 'last_fullsized': '2024-04-17T10:28:01.367828+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.23 is not active+clean 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377527+0000', 'last_change': '2024-04-17T10:28:01.377527+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377527+0000', 'last_undegraded': '2024-04-17T10:28:01.377527+0000', 'last_fullsized': '2024-04-17T10:28:01.377527+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.22 is not active+clean 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.379750+0000', 'last_change': '2024-04-17T10:28:01.379750+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.379750+0000', 'last_undegraded': '2024-04-17T10:28:01.379750+0000', 'last_fullsized': '2024-04-17T10:28:01.379750+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.21 is not active+clean 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376427+0000', 'last_change': '2024-04-17T10:28:01.376427+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376427+0000', 'last_undegraded': '2024-04-17T10:28:01.376427+0000', 'last_fullsized': '2024-04-17T10:28:01.376427+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.20 is not active+clean 2024-04-17T10:28:07.277 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.372163+0000', 'last_change': '2024-04-17T10:28:01.372163+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.372163+0000', 'last_undegraded': '2024-04-17T10:28:01.372163+0000', 'last_fullsized': '2024-04-17T10:28:01.372163+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.10 is not active+clean 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377077+0000', 'last_change': '2024-04-17T10:28:01.377077+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377077+0000', 'last_undegraded': '2024-04-17T10:28:01.377077+0000', 'last_fullsized': '2024-04-17T10:28:01.377077+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.11 is not active+clean 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371180+0000', 'last_change': '2024-04-17T10:28:01.371180+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371180+0000', 'last_undegraded': '2024-04-17T10:28:01.371180+0000', 'last_fullsized': '2024-04-17T10:28:01.371180+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.12 is not active+clean 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.367514+0000', 'last_change': '2024-04-17T10:28:01.367514+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.367514+0000', 'last_undegraded': '2024-04-17T10:28:01.367514+0000', 'last_fullsized': '2024-04-17T10:28:01.367514+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.13 is not active+clean 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371610+0000', 'last_change': '2024-04-17T10:28:01.371610+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371610+0000', 'last_undegraded': '2024-04-17T10:28:01.371610+0000', 'last_fullsized': '2024-04-17T10:28:01.371610+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.14 is not active+clean 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.375461+0000', 'last_change': '2024-04-17T10:28:01.375461+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.375461+0000', 'last_undegraded': '2024-04-17T10:28:01.375461+0000', 'last_fullsized': '2024-04-17T10:28:01.375461+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:28:07.278 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.15 is not active+clean 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377612+0000', 'last_change': '2024-04-17T10:28:01.377612+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377612+0000', 'last_undegraded': '2024-04-17T10:28:01.377612+0000', 'last_fullsized': '2024-04-17T10:28:01.377612+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.16 is not active+clean 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371666+0000', 'last_change': '2024-04-17T10:28:01.371666+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371666+0000', 'last_undegraded': '2024-04-17T10:28:01.371666+0000', 'last_fullsized': '2024-04-17T10:28:01.371666+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.17 is not active+clean 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.378313+0000', 'last_change': '2024-04-17T10:28:01.378313+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.378313+0000', 'last_undegraded': '2024-04-17T10:28:01.378313+0000', 'last_fullsized': '2024-04-17T10:28:01.378313+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.18 is not active+clean 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.372934+0000', 'last_change': '2024-04-17T10:28:01.372934+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.372934+0000', 'last_undegraded': '2024-04-17T10:28:01.372934+0000', 'last_fullsized': '2024-04-17T10:28:01.372934+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.19 is not active+clean 2024-04-17T10:28:07.279 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.372104+0000', 'last_change': '2024-04-17T10:28:01.372104+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.372104+0000', 'last_undegraded': '2024-04-17T10:28:01.372104+0000', 'last_fullsized': '2024-04-17T10:28:01.372104+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1a is not active+clean 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.377011+0000', 'last_change': '2024-04-17T10:28:01.377011+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.377011+0000', 'last_undegraded': '2024-04-17T10:28:01.377011+0000', 'last_fullsized': '2024-04-17T10:28:01.377011+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1b is not active+clean 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.373570+0000', 'last_change': '2024-04-17T10:28:01.373570+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.373570+0000', 'last_undegraded': '2024-04-17T10:28:01.373570+0000', 'last_fullsized': '2024-04-17T10:28:01.373570+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1c is not active+clean 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371710+0000', 'last_change': '2024-04-17T10:28:01.371710+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371710+0000', 'last_undegraded': '2024-04-17T10:28:01.371710+0000', 'last_fullsized': '2024-04-17T10:28:01.371710+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1d is not active+clean 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376893+0000', 'last_change': '2024-04-17T10:28:01.376893+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376893+0000', 'last_undegraded': '2024-04-17T10:28:01.376893+0000', 'last_fullsized': '2024-04-17T10:28:01.376893+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1e is not active+clean 2024-04-17T10:28:07.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.372115+0000', 'last_change': '2024-04-17T10:28:01.372115+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.372115+0000', 'last_undegraded': '2024-04-17T10:28:01.372115+0000', 'last_fullsized': '2024-04-17T10:28:01.372115+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1f is not active+clean 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376533+0000', 'last_change': '2024-04-17T10:28:01.376533+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376533+0000', 'last_undegraded': '2024-04-17T10:28:01.376533+0000', 'last_fullsized': '2024-04-17T10:28:01.376533+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.0 is not active+clean 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.369897+0000', 'last_change': '2024-04-17T10:28:01.369897+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.369897+0000', 'last_undegraded': '2024-04-17T10:28:01.369897+0000', 'last_fullsized': '2024-04-17T10:28:01.369897+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.1 is not active+clean 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.378500+0000', 'last_change': '2024-04-17T10:28:01.378500+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.378500+0000', 'last_undegraded': '2024-04-17T10:28:01.378500+0000', 'last_fullsized': '2024-04-17T10:28:01.378500+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.2 is not active+clean 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.373787+0000', 'last_change': '2024-04-17T10:28:01.373787+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.373787+0000', 'last_undegraded': '2024-04-17T10:28:01.373787+0000', 'last_fullsized': '2024-04-17T10:28:01.373787+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.3 is not active+clean 2024-04-17T10:28:07.281 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.381577+0000', 'last_change': '2024-04-17T10:28:01.381577+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.381577+0000', 'last_undegraded': '2024-04-17T10:28:01.381577+0000', 'last_fullsized': '2024-04-17T10:28:01.381577+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.4 is not active+clean 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371208+0000', 'last_change': '2024-04-17T10:28:01.371208+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371208+0000', 'last_undegraded': '2024-04-17T10:28:01.371208+0000', 'last_fullsized': '2024-04-17T10:28:01.371208+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.5 is not active+clean 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.368273+0000', 'last_change': '2024-04-17T10:28:01.368273+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.368273+0000', 'last_undegraded': '2024-04-17T10:28:01.368273+0000', 'last_fullsized': '2024-04-17T10:28:01.368273+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.6 is not active+clean 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.375458+0000', 'last_change': '2024-04-17T10:28:01.375458+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.375458+0000', 'last_undegraded': '2024-04-17T10:28:01.375458+0000', 'last_fullsized': '2024-04-17T10:28:01.375458+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.7 is not active+clean 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.380500+0000', 'last_change': '2024-04-17T10:28:01.380500+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.380500+0000', 'last_undegraded': '2024-04-17T10:28:01.380500+0000', 'last_fullsized': '2024-04-17T10:28:01.380500+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.8 is not active+clean 2024-04-17T10:28:07.282 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371189+0000', 'last_change': '2024-04-17T10:28:01.371189+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371189+0000', 'last_undegraded': '2024-04-17T10:28:01.371189+0000', 'last_fullsized': '2024-04-17T10:28:01.371189+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.a is not active+clean 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.378166+0000', 'last_change': '2024-04-17T10:28:01.378166+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.378166+0000', 'last_undegraded': '2024-04-17T10:28:01.378166+0000', 'last_fullsized': '2024-04-17T10:28:01.378166+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.b is not active+clean 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.370896+0000', 'last_change': '2024-04-17T10:28:01.370896+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.370896+0000', 'last_undegraded': '2024-04-17T10:28:01.370896+0000', 'last_fullsized': '2024-04-17T10:28:01.370896+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.c is not active+clean 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.375829+0000', 'last_change': '2024-04-17T10:28:01.375829+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.375829+0000', 'last_undegraded': '2024-04-17T10:28:01.375829+0000', 'last_fullsized': '2024-04-17T10:28:01.375829+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.d is not active+clean 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371207+0000', 'last_change': '2024-04-17T10:28:01.371207+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371207+0000', 'last_undegraded': '2024-04-17T10:28:01.371207+0000', 'last_fullsized': '2024-04-17T10:28:01.371207+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.e is not active+clean 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.376341+0000', 'last_change': '2024-04-17T10:28:01.376341+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.376341+0000', 'last_undegraded': '2024-04-17T10:28:01.376341+0000', 'last_fullsized': '2024-04-17T10:28:01.376341+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:28:07.283 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.f is not active+clean 2024-04-17T10:28:07.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.371347+0000', 'last_change': '2024-04-17T10:28:01.371347+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.371347+0000', 'last_undegraded': '2024-04-17T10:28:01.371347+0000', 'last_fullsized': '2024-04-17T10:28:01.371347+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:28:07.284 INFO:tasks.cephfs.filesystem.ceph_manager:PG 8.9 is not active+clean 2024-04-17T10:28:07.284 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '8.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 52, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:01.367353+0000', 'last_change': '2024-04-17T10:28:01.367353+0000', 'last_active': '2024-04-17T10:28:01.363374+0000', 'last_peered': '2024-04-17T10:28:01.363374+0000', 'last_clean': '2024-04-17T10:28:01.363374+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:01.367353+0000', 'last_undegraded': '2024-04-17T10:28:01.367353+0000', 'last_fullsized': '2024-04-17T10:28:01.367353+0000', 'mapping_epoch': 52, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 52, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:01.363374+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:28:07.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:28:07.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.454+0000 7f0cec600640 1 Processor -- start 2024-04-17T10:28:07.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.454+0000 7f0cec600640 1 -- start start 2024-04-17T10:28:07.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0cec600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0ce406c410 0x7f0ce406f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0cec600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40ad8f0 con 0x7f0ce415dbc0 2024-04-17T10:28:07.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0cec600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40adbd0 con 0x7f0ce4071c60 2024-04-17T10:28:07.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0cec600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce415dfa0 con 0x7f0ce406c410 2024-04-17T10:28:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0ce415dbc0 0x7f0ce406db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53830/0 (socket says 172.21.15.67:53830) 2024-04-17T10:28:07.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9c00640 1 -- 172.21.15.67:0/4078266552 learned_addr learned my addr 172.21.15.67:0/4078266552 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9200640 1 --2- 172.21.15.67:0/4078266552 >> v2:172.21.15.86:3300/0 conn(0x7f0ce406c410 0x7f0ce406f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1649545601 0 0) 0x7f0ce40ad8f0 con 0x7f0ce415dbc0 2024-04-17T10:28:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ccc003530 con 0x7f0ce415dbc0 2024-04-17T10:28:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420753328 0 0) 0x7f0ccc003530 con 0x7f0ce415dbc0 2024-04-17T10:28:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 >> v2:172.21.15.86:3300/0 conn(0x7f0ce406c410 msgr2=0x7f0ce406f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 --2- 172.21.15.67:0/4078266552 >> v2:172.21.15.86:3300/0 conn(0x7f0ce406c410 0x7f0ce406f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 >> v1:172.21.15.86:6789/0 conn(0x7f0ce4071c60 legacy=0x7f0ce41676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40adeb0 con 0x7f0ce415dbc0 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f0cd8001680 con 0x7f0ce415dbc0 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 >> v1:172.21.15.67:6789/0 conn(0x7f0ce415dbc0 legacy=0x7f0ce406db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc003a10 0x7f0ccc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ccc004810 0x7f0ccc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ccc005220 0x7f0ccc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce415dfa0 con 0x7f0ccc004810 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc00c170 con 0x7f0ccc003a10 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ccc00c370 con 0x7f0ccc005220 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9200640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ccc004810 0x7f0ccc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9c00640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc003a10 0x7f0ccc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0cea600640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ccc005220 0x7f0ccc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9200640 1 -- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ccc005220 msgr2=0x7f0ccc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:07.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9200640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ccc005220 0x7f0ccc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9200640 1 -- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc003a10 msgr2=0x7f0ccc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:07.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9200640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc003a10 0x7f0ccc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0ce9200640 1 -- 172.21.15.67:0/4078266552 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc003530 con 0x7f0ccc004810 2024-04-17T10:28:07.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0cea600640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ccc005220 0x7f0ccc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:07.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.458+0000 7f0cea600640 1 -- 172.21.15.67:0/4078266552 reap_dead start 2024-04-17T10:28:07.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0ce9200640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ccc004810 0x7f0ccc004c90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009ab0 tx=0x7f0cd001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6ce5b65106e769c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00b2070 con 0x7f0ccc004810 2024-04-17T10:28:07.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd00a55a0 con 0x7f0ccc004810 2024-04-17T10:28:07.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0ce3e00640 1 -- 172.21.15.67:0/4078266552 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00096b0 con 0x7f0ccc004810 2024-04-17T10:28:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0cec600640 1 -- 172.21.15.67:0/4078266552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ccc004810 msgr2=0x7f0ccc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0cec600640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ccc004810 0x7f0ccc004c90 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009ab0 tx=0x7f0cd001ca30 comp rx=0 tx=0).stop 2024-04-17T10:28:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0cec600640 1 -- 172.21.15.67:0/4078266552 shutdown_connections 2024-04-17T10:28:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0cec600640 1 --2- 172.21.15.67:0/4078266552 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ccc004810 0x7f0ccc004c90 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0cec600640 1 -- 172.21.15.67:0/4078266552 >> 172.21.15.67:0/4078266552 conn(0x7f0ce40af670 msgr2=0x7f0ce40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0cec600640 1 -- 172.21.15.67:0/4078266552 shutdown_connections 2024-04-17T10:28:07.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.462+0000 7f0cec600640 1 -- 172.21.15.67:0/4078266552 wait complete. 2024-04-17T10:28:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 Processor -- start 2024-04-17T10:28:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 -- start start 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc004810 0x7f0ce41f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ce415bda0 0x7f0ce40b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415c220 0x7f0ce40b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce406df20 con 0x7f0ce415c220 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce406dda0 con 0x7f0ccc004810 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce406e0a0 con 0x7f0ce415bda0 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415c220 0x7f0ce40b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415c220 0x7f0ce40b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52544/0 (socket says 172.21.15.67:52544) 2024-04-17T10:28:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 -- 172.21.15.67:0/4292317191 learned_addr learned my addr 172.21.15.67:0/4292317191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0ce9c00640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc004810 0x7f0ce41f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0ce9200640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ce415bda0 0x7f0ce40b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 -- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ce415bda0 msgr2=0x7f0ce40b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ce415bda0 0x7f0ce40b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 -- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc004810 msgr2=0x7f0ce41f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc004810 0x7f0ce41f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 -- 172.21.15.67:0/4292317191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce415c7b0 con 0x7f0ce415c220 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0ce9c00640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc004810 0x7f0ce41f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cea600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415c220 0x7f0ce40b3bd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0ce409e060 tx=0x7f0cdc00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0ce1600640 1 -- 172.21.15.67:0/4292317191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc003980 con 0x7f0ce415c220 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0ce1600640 1 -- 172.21.15.67:0/4292317191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc018070 con 0x7f0ce415c220 2024-04-17T10:28:07.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce415ca90 con 0x7f0ce415c220 2024-04-17T10:28:07.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0ce1600640 1 -- 172.21.15.67:0/4292317191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc003d60 con 0x7f0ce415c220 2024-04-17T10:28:07.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce415d050 con 0x7f0ce415c220 2024-04-17T10:28:07.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.466+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce409e260 con 0x7f0ce415c220 2024-04-17T10:28:07.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.470+0000 7f0ce1600640 1 -- 172.21.15.67:0/4292317191 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0cdc0050d0 con 0x7f0ce415c220 2024-04-17T10:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.470+0000 7f0ce1600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0cc4066f60 0x7f0cc4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.470+0000 7f0ce1600640 1 -- 172.21.15.67:0/4292317191 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0cdc084ba0 con 0x7f0ce415c220 2024-04-17T10:28:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.470+0000 7f0ce9c00640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0cc4066f60 0x7f0cc4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.470+0000 7f0ce9c00640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0cc4066f60 0x7f0cc4069420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4155330 tx=0x7f0cd8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.474+0000 7f0ce1600640 1 -- 172.21.15.67:0/4292317191 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0cdc022020 con 0x7f0ce415c220 2024-04-17T10:28:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0ce4074e30 con 0x7f0ce415c220 2024-04-17T10:28:07.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0ce1600640 1 -- 172.21.15.67:0/4292317191 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1195 (secure 0 0 0) 0x7f0cdc055440 con 0x7f0ce415c220 2024-04-17T10:28:07.687 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:07.687 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":218,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":322674688,"bytes_avail":772771438592,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":28,"id":3,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":3,"rank":0,"name":"d","status":"up:active","gid":5772}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-04-17T10:27:26.167976+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0cc4066f60 msgr2=0x7f0cc4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0cc4066f60 0x7f0cc4069420 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4155330 tx=0x7f0cd8009040 comp rx=0 tx=0).stop 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415c220 msgr2=0x7f0ce40b3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415c220 0x7f0ce40b3bd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f0ce409e060 tx=0x7f0cdc00eb90 comp rx=0 tx=0).stop 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 shutdown_connections 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0cc4066f60 0x7f0cc4069420 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ce415c220 0x7f0ce40b3bd0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0ce415bda0 0x7f0ce40b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 --2- 172.21.15.67:0/4292317191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0ccc004810 0x7f0ce41f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 >> 172.21.15.67:0/4292317191 conn(0x7f0ce40af670 msgr2=0x7f0ce415ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 shutdown_connections 2024-04-17T10:28:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.694+0000 7f0cec600640 1 -- 172.21.15.67:0/4292317191 wait complete. 2024-04-17T10:28:07.708 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-17T10:28:07.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:28:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc453c00640 1 Processor -- start 2024-04-17T10:28:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc453c00640 1 -- start start 2024-04-17T10:28:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc453c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc44c06eac0 0x7fc44c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc453c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c0cf0c0 0x7fc44c144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc453c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44c144560 con 0x7fc44c06e6e0 2024-04-17T10:28:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc453c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44c15ae40 con 0x7fc44c06eac0 2024-04-17T10:28:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc453c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c071440 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc44be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc44c06eac0 0x7fc44c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc451c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c0cf0c0 0x7fc44c144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc44be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc44c06eac0 0x7fc44c143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39826/0 (socket says 172.21.15.67:39826) 2024-04-17T10:28:07.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.854+0000 7fc44be00640 1 -- 172.21.15.67:0/1810108374 learned_addr learned my addr 172.21.15.67:0/1810108374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:07.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.858+0000 7fc44be00640 1 -- 172.21.15.67:0/1810108374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c0cf0c0 msgr2=0x7fc44c144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:07.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.858+0000 7fc44be00640 1 --2- 172.21.15.67:0/1810108374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c0cf0c0 0x7fc44c144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.858+0000 7fc44be00640 1 -- 172.21.15.67:0/1810108374 >> v1:172.21.15.86:6789/0 conn(0x7fc44c06e6e0 legacy=0x7fc44c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.858+0000 7fc44be00640 1 -- 172.21.15.67:0/1810108374 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c071720 con 0x7fc44c06eac0 2024-04-17T10:28:07.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.858+0000 7fc44be00640 1 --2- 172.21.15.67:0/1810108374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06eac0 0x7fc44c143ae0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc438009870 tx=0x7fc43801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7802e9302e2cfcde server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc44b400640 1 -- 172.21.15.67:0/1810108374 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380b1070 con 0x7fc44c06eac0 2024-04-17T10:28:07.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc44b400640 1 -- 172.21.15.67:0/1810108374 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4380ac620 con 0x7fc44c06eac0 2024-04-17T10:28:07.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc44b400640 1 -- 172.21.15.67:0/1810108374 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4380ac9f0 con 0x7fc44c06eac0 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- 172.21.15.67:0/1810108374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06eac0 msgr2=0x7fc44c143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 --2- 172.21.15.67:0/1810108374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06eac0 0x7fc44c143ae0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc438009870 tx=0x7fc43801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- 172.21.15.67:0/1810108374 shutdown_connections 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 --2- 172.21.15.67:0/1810108374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c0cf0c0 0x7fc44c144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 --2- 172.21.15.67:0/1810108374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06eac0 0x7fc44c143ae0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- 172.21.15.67:0/1810108374 >> 172.21.15.67:0/1810108374 conn(0x7fc44c06ac90 msgr2=0x7fc44c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- 172.21.15.67:0/1810108374 shutdown_connections 2024-04-17T10:28:07.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- 172.21.15.67:0/1810108374 wait complete. 2024-04-17T10:28:07.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 Processor -- start 2024-04-17T10:28:07.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- start start 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06e6e0 0x7fc44c13afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c06eac0 0x7fc44c0acba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c0cf0c0 0x7fc44c0ad0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c06dd00 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c06db80 con 0x7fc44c06e6e0 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc453c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c06de80 con 0x7fc44c06eac0 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc451c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c0cf0c0 0x7fc44c0ad0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc451c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c0cf0c0 0x7fc44c0ad0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52554/0 (socket says 172.21.15.67:52554) 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc451c00640 1 -- 172.21.15.67:0/3176535951 learned_addr learned my addr 172.21.15.67:0/3176535951 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc44be00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c06eac0 0x7fc44c0acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.862+0000 7fc451200640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06e6e0 0x7fc44c13afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc451c00640 1 -- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c06eac0 msgr2=0x7fc44c0acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc451c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c06eac0 0x7fc44c0acba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc451c00640 1 -- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06e6e0 msgr2=0x7fc44c13afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc451c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06e6e0 0x7fc44c13afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc451c00640 1 -- 172.21.15.67:0/3176535951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c0c4880 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc451c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c0cf0c0 0x7fc44c0ad0e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc44400bde0 tx=0x7fc44400e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc448c00640 1 -- 172.21.15.67:0/3176535951 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44401a070 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc448c00640 1 -- 172.21.15.67:0/3176535951 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44400ed20 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc448c00640 1 -- 172.21.15.67:0/3176535951 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44400b3e0 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c0c4b60 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c0c5120 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c072d50 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc448c00640 1 -- 172.21.15.67:0/3176535951 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc44401d070 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc448c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc42c066f60 0x7fc42c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:07.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.866+0000 7fc448c00640 1 -- 172.21.15.67:0/3176535951 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc444085330 con 0x7fc44c0cf0c0 2024-04-17T10:28:07.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.870+0000 7fc451200640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc42c066f60 0x7fc42c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:07.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.870+0000 7fc451200640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc42c066f60 0x7fc42c069420 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc440001810 tx=0x7fc440009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:07.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:07.870+0000 7fc448c00640 1 -- 172.21.15.67:0/3176535951 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc444055ac0 con 0x7fc44c0cf0c0 2024-04-17T10:28:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.030+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc44c074e30 con 0x7fc42c066f60 2024-04-17T10:28:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.034+0000 7fc448c00640 1 -- 172.21.15.67:0/3176535951 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246639 (secure 0 0 0) 0x7fc44c074e30 con 0x7fc42c066f60 2024-04-17T10:28:08.032 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:08.033 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:28:08.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc42c066f60 msgr2=0x7fc42c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:08.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc42c066f60 0x7fc42c069420 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc440001810 tx=0x7fc440009040 comp rx=0 tx=0).stop 2024-04-17T10:28:08.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c0cf0c0 msgr2=0x7fc44c0ad0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c0cf0c0 0x7fc44c0ad0e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fc44400bde0 tx=0x7fc44400e490 comp rx=0 tx=0).stop 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 shutdown_connections 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc42c066f60 0x7fc42c069420 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc44c0cf0c0 0x7fc44c0ad0e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc44c06eac0 0x7fc44c0acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 --2- 172.21.15.67:0/3176535951 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc44c06e6e0 0x7fc44c13afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 >> 172.21.15.67:0/3176535951 conn(0x7fc44c06ac90 msgr2=0x7fc44c13be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 shutdown_connections 2024-04-17T10:28:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:08.038+0000 7fc453c00640 1 -- 172.21.15.67:0/3176535951 wait complete. 2024-04-17T10:28:08.052 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-04-17T10:28:06.177959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315112,"kb_used_data":5980,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754659608,"statfs":{"total":773094113280,"available":772771438592,"internally_reserved":0,"allocated":6123520,"data_stored":3388317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.817202"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380959+0000","last_change":"2024-04-17T10:28:01.380959+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380959+0000","last_undegraded":"2024-04-17T10:28:01.380959+0000","last_fullsized":"2024-04-17T10:28:01.380959+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376444+0000","last_change":"2024-04-17T10:28:01.376444+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376444+0000","last_undegraded":"2024-04-17T10:28:01.376444+0000","last_fullsized":"2024-04-17T10:28:01.376444+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378464+0000","last_change":"2024-04-17T10:28:01.378464+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378464+0000","last_undegraded":"2024-04-17T10:28:01.378464+0000","last_fullsized":"2024-04-17T10:28:01.378464+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377387+0000","last_change":"2024-04-17T10:28:01.377387+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377387+0000","last_undegraded":"2024-04-17T10:28:01.377387+0000","last_fullsized":"2024-04-17T10:28:01.377387+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374044+0000","last_change":"2024-04-17T10:28:01.374044+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374044+0000","last_undegraded":"2024-04-17T10:28:01.374044+0000","last_fullsized":"2024-04-17T10:28:01.374044+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378669+0000","last_change":"2024-04-17T10:28:01.378669+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378669+0000","last_undegraded":"2024-04-17T10:28:01.378669+0000","last_fullsized":"2024-04-17T10:28:01.378669+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377863+0000","last_change":"2024-04-17T10:28:01.377863+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377863+0000","last_undegraded":"2024-04-17T10:28:01.377863+0000","last_fullsized":"2024-04-17T10:28:01.377863+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370398+0000","last_change":"2024-04-17T10:28:01.370398+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370398+0000","last_undegraded":"2024-04-17T10:28:01.370398+0000","last_fullsized":"2024-04-17T10:28:01.370398+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371867+0000","last_change":"2024-04-17T10:28:01.371867+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371867+0000","last_undegraded":"2024-04-17T10:28:01.371867+0000","last_fullsized":"2024-04-17T10:28:01.371867+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379871+0000","last_change":"2024-04-17T10:28:01.379871+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379871+0000","last_undegraded":"2024-04-17T10:28:01.379871+0000","last_fullsized":"2024-04-17T10:28:01.379871+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379507+0000","last_change":"2024-04-17T10:28:01.379507+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379507+0000","last_undegraded":"2024-04-17T10:28:01.379507+0000","last_fullsized":"2024-04-17T10:28:01.379507+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371184+0000","last_change":"2024-04-17T10:28:01.371184+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371184+0000","last_undegraded":"2024-04-17T10:28:01.371184+0000","last_fullsized":"2024-04-17T10:28:01.371184+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381984+0000","last_change":"2024-04-17T10:28:01.381984+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381984+0000","last_undegraded":"2024-04-17T10:28:01.381984+0000","last_fullsized":"2024-04-17T10:28:01.381984+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377452+0000","last_change":"2024-04-17T10:28:01.377452+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377452+0000","last_undegraded":"2024-04-17T10:28:01.377452+0000","last_fullsized":"2024-04-17T10:28:01.377452+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376914+0000","last_change":"2024-04-17T10:28:01.376914+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376914+0000","last_undegraded":"2024-04-17T10:28:01.376914+0000","last_fullsized":"2024-04-17T10:28:01.376914+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376339+0000","last_change":"2024-04-17T10:28:01.376339+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376339+0000","last_undegraded":"2024-04-17T10:28:01.376339+0000","last_fullsized":"2024-04-17T10:28:01.376339+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379944+0000","last_change":"2024-04-17T10:28:01.379944+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379944+0000","last_undegraded":"2024-04-17T10:28:01.379944+0000","last_fullsized":"2024-04-17T10:28:01.379944+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376011+0000","last_change":"2024-04-17T10:28:01.376011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376011+0000","last_undegraded":"2024-04-17T10:28:01.376011+0000","last_fullsized":"2024-04-17T10:28:01.376011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377320+0000","last_change":"2024-04-17T10:28:01.377320+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377320+0000","last_undegraded":"2024-04-17T10:28:01.377320+0000","last_fullsized":"2024-04-17T10:28:01.377320+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369232+0000","last_change":"2024-04-17T10:28:01.369232+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369232+0000","last_undegraded":"2024-04-17T10:28:01.369232+0000","last_fullsized":"2024-04-17T10:28:01.369232+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367428+0000","last_change":"2024-04-17T10:28:01.367428+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367428+0000","last_undegraded":"2024-04-17T10:28:01.367428+0000","last_fullsized":"2024-04-17T10:28:01.367428+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370644+0000","last_change":"2024-04-17T10:28:01.370644+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370644+0000","last_undegraded":"2024-04-17T10:28:01.370644+0000","last_fullsized":"2024-04-17T10:28:01.370644+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379078+0000","last_change":"2024-04-17T10:28:01.379078+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379078+0000","last_undegraded":"2024-04-17T10:28:01.379078+0000","last_fullsized":"2024-04-17T10:28:01.379078+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380365+0000","last_change":"2024-04-17T10:28:01.380365+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380365+0000","last_undegraded":"2024-04-17T10:28:01.380365+0000","last_fullsized":"2024-04-17T10:28:01.380365+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.374358+0000","last_change":"2024-04-17T10:28:01.374358+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.374358+0000","last_undegraded":"2024-04-17T10:28:01.374358+0000","last_fullsized":"2024-04-17T10:28:01.374358+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.383292+0000","last_change":"2024-04-17T10:28:01.383292+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.383292+0000","last_undegraded":"2024-04-17T10:28:01.383292+0000","last_fullsized":"2024-04-17T10:28:01.383292+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367828+0000","last_change":"2024-04-17T10:28:01.367828+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367828+0000","last_undegraded":"2024-04-17T10:28:01.367828+0000","last_fullsized":"2024-04-17T10:28:01.367828+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377527+0000","last_change":"2024-04-17T10:28:01.377527+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377527+0000","last_undegraded":"2024-04-17T10:28:01.377527+0000","last_fullsized":"2024-04-17T10:28:01.377527+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.379750+0000","last_change":"2024-04-17T10:28:01.379750+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.379750+0000","last_undegraded":"2024-04-17T10:28:01.379750+0000","last_fullsized":"2024-04-17T10:28:01.379750+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376427+0000","last_change":"2024-04-17T10:28:01.376427+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376427+0000","last_undegraded":"2024-04-17T10:28:01.376427+0000","last_fullsized":"2024-04-17T10:28:01.376427+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372163+0000","last_change":"2024-04-17T10:28:01.372163+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372163+0000","last_undegraded":"2024-04-17T10:28:01.372163+0000","last_fullsized":"2024-04-17T10:28:01.372163+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377077+0000","last_change":"2024-04-17T10:28:01.377077+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377077+0000","last_undegraded":"2024-04-17T10:28:01.377077+0000","last_fullsized":"2024-04-17T10:28:01.377077+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384066+0000","last_change":"2024-04-17T10:28:01.380916+0000","last_active":"2024-04-17T10:28:01.384066+0000","last_peered":"2024-04-17T10:28:01.384066+0000","last_clean":"2024-04-17T10:28:01.384066+0000","last_became_active":"2024-04-17T10:28:01.380635+0000","last_became_peered":"2024-04-17T10:28:01.380635+0000","last_unstale":"2024-04-17T10:28:01.384066+0000","last_undegraded":"2024-04-17T10:28:01.384066+0000","last_fullsized":"2024-04-17T10:28:01.384066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:47:03.828693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371180+0000","last_change":"2024-04-17T10:28:01.371180+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371180+0000","last_undegraded":"2024-04-17T10:28:01.371180+0000","last_fullsized":"2024-04-17T10:28:01.371180+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394963+0000","last_change":"2024-04-17T10:28:01.391871+0000","last_active":"2024-04-17T10:28:01.394963+0000","last_peered":"2024-04-17T10:28:01.394963+0000","last_clean":"2024-04-17T10:28:01.394963+0000","last_became_active":"2024-04-17T10:28:01.391540+0000","last_became_peered":"2024-04-17T10:28:01.391540+0000","last_unstale":"2024-04-17T10:28:01.394963+0000","last_undegraded":"2024-04-17T10:28:01.394963+0000","last_fullsized":"2024-04-17T10:28:01.394963+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:03:16.384430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367514+0000","last_change":"2024-04-17T10:28:01.367514+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367514+0000","last_undegraded":"2024-04-17T10:28:01.367514+0000","last_fullsized":"2024-04-17T10:28:01.367514+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394726+0000","last_change":"2024-04-17T10:28:01.392616+0000","last_active":"2024-04-17T10:28:01.394726+0000","last_peered":"2024-04-17T10:28:01.394726+0000","last_clean":"2024-04-17T10:28:01.394726+0000","last_became_active":"2024-04-17T10:28:01.391986+0000","last_became_peered":"2024-04-17T10:28:01.391986+0000","last_unstale":"2024-04-17T10:28:01.394726+0000","last_undegraded":"2024-04-17T10:28:01.394726+0000","last_fullsized":"2024-04-17T10:28:01.394726+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371610+0000","last_change":"2024-04-17T10:28:01.371610+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371610+0000","last_undegraded":"2024-04-17T10:28:01.371610+0000","last_fullsized":"2024-04-17T10:28:01.371610+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383990+0000","last_change":"2024-04-17T10:28:01.380404+0000","last_active":"2024-04-17T10:28:01.383990+0000","last_peered":"2024-04-17T10:28:01.383990+0000","last_clean":"2024-04-17T10:28:01.383990+0000","last_became_active":"2024-04-17T10:28:01.380065+0000","last_became_peered":"2024-04-17T10:28:01.380065+0000","last_unstale":"2024-04-17T10:28:01.383990+0000","last_undegraded":"2024-04-17T10:28:01.383990+0000","last_fullsized":"2024-04-17T10:28:01.383990+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:12.355937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375461+0000","last_change":"2024-04-17T10:28:01.375461+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375461+0000","last_undegraded":"2024-04-17T10:28:01.375461+0000","last_fullsized":"2024-04-17T10:28:01.375461+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399062+0000","last_change":"2024-04-17T10:28:01.395609+0000","last_active":"2024-04-17T10:28:01.399062+0000","last_peered":"2024-04-17T10:28:01.399062+0000","last_clean":"2024-04-17T10:28:01.399062+0000","last_became_active":"2024-04-17T10:28:01.395267+0000","last_became_peered":"2024-04-17T10:28:01.395267+0000","last_unstale":"2024-04-17T10:28:01.399062+0000","last_undegraded":"2024-04-17T10:28:01.399062+0000","last_fullsized":"2024-04-17T10:28:01.399062+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:42.740201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377612+0000","last_change":"2024-04-17T10:28:01.377612+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377612+0000","last_undegraded":"2024-04-17T10:28:01.377612+0000","last_fullsized":"2024-04-17T10:28:01.377612+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383876+0000","last_change":"2024-04-17T10:28:01.378483+0000","last_active":"2024-04-17T10:28:01.383876+0000","last_peered":"2024-04-17T10:28:01.383876+0000","last_clean":"2024-04-17T10:28:01.383876+0000","last_became_active":"2024-04-17T10:28:01.378140+0000","last_became_peered":"2024-04-17T10:28:01.378140+0000","last_unstale":"2024-04-17T10:28:01.383876+0000","last_undegraded":"2024-04-17T10:28:01.383876+0000","last_fullsized":"2024-04-17T10:28:01.383876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371666+0000","last_change":"2024-04-17T10:28:01.371666+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371666+0000","last_undegraded":"2024-04-17T10:28:01.371666+0000","last_fullsized":"2024-04-17T10:28:01.371666+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.383084+0000","last_change":"2024-04-17T10:28:01.378144+0000","last_active":"2024-04-17T10:28:01.383084+0000","last_peered":"2024-04-17T10:28:01.383084+0000","last_clean":"2024-04-17T10:28:01.383084+0000","last_became_active":"2024-04-17T10:28:01.377879+0000","last_became_peered":"2024-04-17T10:28:01.377879+0000","last_unstale":"2024-04-17T10:28:01.383084+0000","last_undegraded":"2024-04-17T10:28:01.383084+0000","last_fullsized":"2024-04-17T10:28:01.383084+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:02.479603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378313+0000","last_change":"2024-04-17T10:28:01.378313+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378313+0000","last_undegraded":"2024-04-17T10:28:01.378313+0000","last_fullsized":"2024-04-17T10:28:01.378313+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381525+0000","last_change":"2024-04-17T10:28:01.376007+0000","last_active":"2024-04-17T10:28:01.381525+0000","last_peered":"2024-04-17T10:28:01.381525+0000","last_clean":"2024-04-17T10:28:01.381525+0000","last_became_active":"2024-04-17T10:28:01.375098+0000","last_became_peered":"2024-04-17T10:28:01.375098+0000","last_unstale":"2024-04-17T10:28:01.381525+0000","last_undegraded":"2024-04-17T10:28:01.381525+0000","last_fullsized":"2024-04-17T10:28:01.381525+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372934+0000","last_change":"2024-04-17T10:28:01.372934+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372934+0000","last_undegraded":"2024-04-17T10:28:01.372934+0000","last_fullsized":"2024-04-17T10:28:01.372934+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394069+0000","last_change":"2024-04-17T10:28:01.390159+0000","last_active":"2024-04-17T10:28:01.394069+0000","last_peered":"2024-04-17T10:28:01.394069+0000","last_clean":"2024-04-17T10:28:01.394069+0000","last_became_active":"2024-04-17T10:28:01.389874+0000","last_became_peered":"2024-04-17T10:28:01.389874+0000","last_unstale":"2024-04-17T10:28:01.394069+0000","last_undegraded":"2024-04-17T10:28:01.394069+0000","last_fullsized":"2024-04-17T10:28:01.394069+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:59.077053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372104+0000","last_change":"2024-04-17T10:28:01.372104+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372104+0000","last_undegraded":"2024-04-17T10:28:01.372104+0000","last_fullsized":"2024-04-17T10:28:01.372104+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396819+0000","last_change":"2024-04-17T10:28:01.396322+0000","last_active":"2024-04-17T10:28:01.396819+0000","last_peered":"2024-04-17T10:28:01.396819+0000","last_clean":"2024-04-17T10:28:01.396819+0000","last_became_active":"2024-04-17T10:28:01.396090+0000","last_became_peered":"2024-04-17T10:28:01.396090+0000","last_unstale":"2024-04-17T10:28:01.396819+0000","last_undegraded":"2024-04-17T10:28:01.396819+0000","last_fullsized":"2024-04-17T10:28:01.396819+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.377011+0000","last_change":"2024-04-17T10:28:01.377011+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.377011+0000","last_undegraded":"2024-04-17T10:28:01.377011+0000","last_fullsized":"2024-04-17T10:28:01.377011+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394086+0000","last_change":"2024-04-17T10:28:01.390653+0000","last_active":"2024-04-17T10:28:01.394086+0000","last_peered":"2024-04-17T10:28:01.394086+0000","last_clean":"2024-04-17T10:28:01.394086+0000","last_became_active":"2024-04-17T10:28:01.390327+0000","last_became_peered":"2024-04-17T10:28:01.390327+0000","last_unstale":"2024-04-17T10:28:01.394086+0000","last_undegraded":"2024-04-17T10:28:01.394086+0000","last_fullsized":"2024-04-17T10:28:01.394086+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:12.074317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373570+0000","last_change":"2024-04-17T10:28:01.373570+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373570+0000","last_undegraded":"2024-04-17T10:28:01.373570+0000","last_fullsized":"2024-04-17T10:28:01.373570+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.393035+0000","last_change":"2024-04-17T10:28:01.390744+0000","last_active":"2024-04-17T10:28:01.393035+0000","last_peered":"2024-04-17T10:28:01.393035+0000","last_clean":"2024-04-17T10:28:01.393035+0000","last_became_active":"2024-04-17T10:28:01.390487+0000","last_became_peered":"2024-04-17T10:28:01.390487+0000","last_unstale":"2024-04-17T10:28:01.393035+0000","last_undegraded":"2024-04-17T10:28:01.393035+0000","last_fullsized":"2024-04-17T10:28:01.393035+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:01:21.812637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371710+0000","last_change":"2024-04-17T10:28:01.371710+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371710+0000","last_undegraded":"2024-04-17T10:28:01.371710+0000","last_fullsized":"2024-04-17T10:28:01.371710+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399042+0000","last_change":"2024-04-17T10:28:01.388726+0000","last_active":"2024-04-17T10:28:01.399042+0000","last_peered":"2024-04-17T10:28:01.399042+0000","last_clean":"2024-04-17T10:28:01.399042+0000","last_became_active":"2024-04-17T10:28:01.388462+0000","last_became_peered":"2024-04-17T10:28:01.388462+0000","last_unstale":"2024-04-17T10:28:01.399042+0000","last_undegraded":"2024-04-17T10:28:01.399042+0000","last_fullsized":"2024-04-17T10:28:01.399042+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376893+0000","last_change":"2024-04-17T10:28:01.376893+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376893+0000","last_undegraded":"2024-04-17T10:28:01.376893+0000","last_fullsized":"2024-04-17T10:28:01.376893+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.390536+0000","last_change":"2024-04-17T10:28:01.388333+0000","last_active":"2024-04-17T10:28:01.390536+0000","last_peered":"2024-04-17T10:28:01.390536+0000","last_clean":"2024-04-17T10:28:01.390536+0000","last_became_active":"2024-04-17T10:28:01.387931+0000","last_became_peered":"2024-04-17T10:28:01.387931+0000","last_unstale":"2024-04-17T10:28:01.390536+0000","last_undegraded":"2024-04-17T10:28:01.390536+0000","last_fullsized":"2024-04-17T10:28:01.390536+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:28:46.889727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.372115+0000","last_change":"2024-04-17T10:28:01.372115+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.372115+0000","last_undegraded":"2024-04-17T10:28:01.372115+0000","last_fullsized":"2024-04-17T10:28:01.372115+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388876+0000","last_change":"2024-04-17T10:28:01.387372+0000","last_active":"2024-04-17T10:28:01.388876+0000","last_peered":"2024-04-17T10:28:01.388876+0000","last_clean":"2024-04-17T10:28:01.388876+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.388876+0000","last_undegraded":"2024-04-17T10:28:01.388876+0000","last_fullsized":"2024-04-17T10:28:01.388876+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:31.085374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376533+0000","last_change":"2024-04-17T10:28:01.376533+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376533+0000","last_undegraded":"2024-04-17T10:28:01.376533+0000","last_fullsized":"2024-04-17T10:28:01.376533+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391514+0000","last_change":"2024-04-17T10:28:01.390499+0000","last_active":"2024-04-17T10:28:01.391514+0000","last_peered":"2024-04-17T10:28:01.391514+0000","last_clean":"2024-04-17T10:28:01.391514+0000","last_became_active":"2024-04-17T10:28:01.390208+0000","last_became_peered":"2024-04-17T10:28:01.390208+0000","last_unstale":"2024-04-17T10:28:01.391514+0000","last_undegraded":"2024-04-17T10:28:01.391514+0000","last_fullsized":"2024-04-17T10:28:01.391514+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:01:44.289510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.369897+0000","last_change":"2024-04-17T10:28:01.369897+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.369897+0000","last_undegraded":"2024-04-17T10:28:01.369897+0000","last_fullsized":"2024-04-17T10:28:01.369897+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398772+0000","last_change":"2024-04-17T10:28:01.385970+0000","last_active":"2024-04-17T10:28:01.398772+0000","last_peered":"2024-04-17T10:28:01.398772+0000","last_clean":"2024-04-17T10:28:01.398772+0000","last_became_active":"2024-04-17T10:28:01.385447+0000","last_became_peered":"2024-04-17T10:28:01.385447+0000","last_unstale":"2024-04-17T10:28:01.398772+0000","last_undegraded":"2024-04-17T10:28:01.398772+0000","last_fullsized":"2024-04-17T10:28:01.398772+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378500+0000","last_change":"2024-04-17T10:28:01.378500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378500+0000","last_undegraded":"2024-04-17T10:28:01.378500+0000","last_fullsized":"2024-04-17T10:28:01.378500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.396318+0000","last_change":"2024-04-17T10:28:01.393600+0000","last_active":"2024-04-17T10:28:01.396318+0000","last_peered":"2024-04-17T10:28:01.396318+0000","last_clean":"2024-04-17T10:28:01.396318+0000","last_became_active":"2024-04-17T10:28:01.393277+0000","last_became_peered":"2024-04-17T10:28:01.393277+0000","last_unstale":"2024-04-17T10:28:01.396318+0000","last_undegraded":"2024-04-17T10:28:01.396318+0000","last_fullsized":"2024-04-17T10:28:01.396318+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:55:03.162699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.373787+0000","last_change":"2024-04-17T10:28:01.373787+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.373787+0000","last_undegraded":"2024-04-17T10:28:01.373787+0000","last_fullsized":"2024-04-17T10:28:01.373787+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391197+0000","last_change":"2024-04-17T10:28:01.390060+0000","last_active":"2024-04-17T10:28:01.391197+0000","last_peered":"2024-04-17T10:28:01.391197+0000","last_clean":"2024-04-17T10:28:01.391197+0000","last_became_active":"2024-04-17T10:28:01.389732+0000","last_became_peered":"2024-04-17T10:28:01.389732+0000","last_unstale":"2024-04-17T10:28:01.391197+0000","last_undegraded":"2024-04-17T10:28:01.391197+0000","last_fullsized":"2024-04-17T10:28:01.391197+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:28:07.506032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.381577+0000","last_change":"2024-04-17T10:28:01.381577+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.381577+0000","last_undegraded":"2024-04-17T10:28:01.381577+0000","last_fullsized":"2024-04-17T10:28:01.381577+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.384958+0000","last_change":"2024-04-17T10:28:01.380617+0000","last_active":"2024-04-17T10:28:01.384958+0000","last_peered":"2024-04-17T10:28:01.384958+0000","last_clean":"2024-04-17T10:28:01.384958+0000","last_became_active":"2024-04-17T10:28:01.380340+0000","last_became_peered":"2024-04-17T10:28:01.380340+0000","last_unstale":"2024-04-17T10:28:01.384958+0000","last_undegraded":"2024-04-17T10:28:01.384958+0000","last_fullsized":"2024-04-17T10:28:01.384958+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:58.422966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371208+0000","last_change":"2024-04-17T10:28:01.371208+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371208+0000","last_undegraded":"2024-04-17T10:28:01.371208+0000","last_fullsized":"2024-04-17T10:28:01.371208+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.395078+0000","last_change":"2024-04-17T10:28:01.393021+0000","last_active":"2024-04-17T10:28:01.395078+0000","last_peered":"2024-04-17T10:28:01.395078+0000","last_clean":"2024-04-17T10:28:01.395078+0000","last_became_active":"2024-04-17T10:28:01.392144+0000","last_became_peered":"2024-04-17T10:28:01.392144+0000","last_unstale":"2024-04-17T10:28:01.395078+0000","last_undegraded":"2024-04-17T10:28:01.395078+0000","last_fullsized":"2024-04-17T10:28:01.395078+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.368273+0000","last_change":"2024-04-17T10:28:01.368273+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.368273+0000","last_undegraded":"2024-04-17T10:28:01.368273+0000","last_fullsized":"2024-04-17T10:28:01.368273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.389296+0000","last_change":"2024-04-17T10:28:01.384888+0000","last_active":"2024-04-17T10:28:01.389296+0000","last_peered":"2024-04-17T10:28:01.389296+0000","last_clean":"2024-04-17T10:28:01.389296+0000","last_became_active":"2024-04-17T10:28:01.384242+0000","last_became_peered":"2024-04-17T10:28:01.384242+0000","last_unstale":"2024-04-17T10:28:01.389296+0000","last_undegraded":"2024-04-17T10:28:01.389296+0000","last_fullsized":"2024-04-17T10:28:01.389296+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:15.827189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375458+0000","last_change":"2024-04-17T10:28:01.375458+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375458+0000","last_undegraded":"2024-04-17T10:28:01.375458+0000","last_fullsized":"2024-04-17T10:28:01.375458+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.399097+0000","last_change":"2024-04-17T10:28:01.395507+0000","last_active":"2024-04-17T10:28:01.399097+0000","last_peered":"2024-04-17T10:28:01.399097+0000","last_clean":"2024-04-17T10:28:01.399097+0000","last_became_active":"2024-04-17T10:28:01.392288+0000","last_became_peered":"2024-04-17T10:28:01.392288+0000","last_unstale":"2024-04-17T10:28:01.399097+0000","last_undegraded":"2024-04-17T10:28:01.399097+0000","last_fullsized":"2024-04-17T10:28:01.399097+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:04:14.700196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.380500+0000","last_change":"2024-04-17T10:28:01.380500+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.380500+0000","last_undegraded":"2024-04-17T10:28:01.380500+0000","last_fullsized":"2024-04-17T10:28:01.380500+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.394290+0000","last_change":"2024-04-17T10:28:01.391397+0000","last_active":"2024-04-17T10:28:01.394290+0000","last_peered":"2024-04-17T10:28:01.394290+0000","last_clean":"2024-04-17T10:28:01.394290+0000","last_became_active":"2024-04-17T10:28:01.391114+0000","last_became_peered":"2024-04-17T10:28:01.391114+0000","last_unstale":"2024-04-17T10:28:01.394290+0000","last_undegraded":"2024-04-17T10:28:01.394290+0000","last_fullsized":"2024-04-17T10:28:01.394290+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:48:23.592401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371189+0000","last_change":"2024-04-17T10:28:01.371189+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371189+0000","last_undegraded":"2024-04-17T10:28:01.371189+0000","last_fullsized":"2024-04-17T10:28:01.371189+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.388236+0000","last_change":"2024-04-17T10:28:01.386653+0000","last_active":"2024-04-17T10:28:01.388236+0000","last_peered":"2024-04-17T10:28:01.388236+0000","last_clean":"2024-04-17T10:28:01.388236+0000","last_became_active":"2024-04-17T10:28:01.386406+0000","last_became_peered":"2024-04-17T10:28:01.386406+0000","last_unstale":"2024-04-17T10:28:01.388236+0000","last_undegraded":"2024-04-17T10:28:01.388236+0000","last_fullsized":"2024-04-17T10:28:01.388236+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:31.804436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.378166+0000","last_change":"2024-04-17T10:28:01.378166+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.378166+0000","last_undegraded":"2024-04-17T10:28:01.378166+0000","last_fullsized":"2024-04-17T10:28:01.378166+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391538+0000","last_change":"2024-04-17T10:28:01.389788+0000","last_active":"2024-04-17T10:28:01.391538+0000","last_peered":"2024-04-17T10:28:01.391538+0000","last_clean":"2024-04-17T10:28:01.391538+0000","last_became_active":"2024-04-17T10:28:01.389488+0000","last_became_peered":"2024-04-17T10:28:01.389488+0000","last_unstale":"2024-04-17T10:28:01.391538+0000","last_undegraded":"2024-04-17T10:28:01.391538+0000","last_fullsized":"2024-04-17T10:28:01.391538+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:19.391683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.370896+0000","last_change":"2024-04-17T10:28:01.370896+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.370896+0000","last_undegraded":"2024-04-17T10:28:01.370896+0000","last_fullsized":"2024-04-17T10:28:01.370896+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398827+0000","last_change":"2024-04-17T10:28:01.386384+0000","last_active":"2024-04-17T10:28:01.398827+0000","last_peered":"2024-04-17T10:28:01.398827+0000","last_clean":"2024-04-17T10:28:01.398827+0000","last_became_active":"2024-04-17T10:28:01.386102+0000","last_became_peered":"2024-04-17T10:28:01.386102+0000","last_unstale":"2024-04-17T10:28:01.398827+0000","last_undegraded":"2024-04-17T10:28:01.398827+0000","last_fullsized":"2024-04-17T10:28:01.398827+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.375829+0000","last_change":"2024-04-17T10:28:01.375829+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.375829+0000","last_undegraded":"2024-04-17T10:28:01.375829+0000","last_fullsized":"2024-04-17T10:28:01.375829+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398627+0000","last_change":"2024-04-17T10:28:01.394707+0000","last_active":"2024-04-17T10:28:01.398627+0000","last_peered":"2024-04-17T10:28:01.398627+0000","last_clean":"2024-04-17T10:28:01.398627+0000","last_became_active":"2024-04-17T10:28:01.394419+0000","last_became_peered":"2024-04-17T10:28:01.394419+0000","last_unstale":"2024-04-17T10:28:01.398627+0000","last_undegraded":"2024-04-17T10:28:01.398627+0000","last_fullsized":"2024-04-17T10:28:01.398627+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:05:22.975781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371207+0000","last_change":"2024-04-17T10:28:01.371207+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371207+0000","last_undegraded":"2024-04-17T10:28:01.371207+0000","last_fullsized":"2024-04-17T10:28:01.371207+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.397867+0000","last_change":"2024-04-17T10:28:01.396212+0000","last_active":"2024-04-17T10:28:01.397867+0000","last_peered":"2024-04-17T10:28:01.397867+0000","last_clean":"2024-04-17T10:28:01.397867+0000","last_became_active":"2024-04-17T10:28:01.395939+0000","last_became_peered":"2024-04-17T10:28:01.395939+0000","last_unstale":"2024-04-17T10:28:01.397867+0000","last_undegraded":"2024-04-17T10:28:01.397867+0000","last_fullsized":"2024-04-17T10:28:01.397867+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:00.316480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.376341+0000","last_change":"2024-04-17T10:28:01.376341+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.376341+0000","last_undegraded":"2024-04-17T10:28:01.376341+0000","last_fullsized":"2024-04-17T10:28:01.376341+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.386497+0000","last_change":"2024-04-17T10:28:01.384533+0000","last_active":"2024-04-17T10:28:01.386497+0000","last_peered":"2024-04-17T10:28:01.386497+0000","last_clean":"2024-04-17T10:28:01.386497+0000","last_became_active":"2024-04-17T10:28:01.384284+0000","last_became_peered":"2024-04-17T10:28:01.384284+0000","last_unstale":"2024-04-17T10:28:01.386497+0000","last_undegraded":"2024-04-17T10:28:01.386497+0000","last_fullsized":"2024-04-17T10:28:01.386497+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.371347+0000","last_change":"2024-04-17T10:28:01.371347+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.371347+0000","last_undegraded":"2024-04-17T10:28:01.371347+0000","last_fullsized":"2024-04-17T10:28:01.371347+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.391583+0000","last_change":"2024-04-17T10:28:01.388193+0000","last_active":"2024-04-17T10:28:01.391583+0000","last_peered":"2024-04-17T10:28:01.391583+0000","last_clean":"2024-04-17T10:28:01.391583+0000","last_became_active":"2024-04-17T10:28:01.387887+0000","last_became_peered":"2024-04-17T10:28:01.387887+0000","last_unstale":"2024-04-17T10:28:01.391583+0000","last_undegraded":"2024-04-17T10:28:01.391583+0000","last_fullsized":"2024-04-17T10:28:01.391583+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:12.842514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":1,"reported_epoch":52,"state":"creating+peering","last_fresh":"2024-04-17T10:28:01.367353+0000","last_change":"2024-04-17T10:28:01.367353+0000","last_active":"2024-04-17T10:28:01.363374+0000","last_peered":"2024-04-17T10:28:01.363374+0000","last_clean":"2024-04-17T10:28:01.363374+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:01.367353+0000","last_undegraded":"2024-04-17T10:28:01.367353+0000","last_fullsized":"2024-04-17T10:28:01.367353+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.398900+0000","last_change":"2024-04-17T10:28:01.387103+0000","last_active":"2024-04-17T10:28:01.398900+0000","last_peered":"2024-04-17T10:28:01.398900+0000","last_clean":"2024-04-17T10:28:01.398900+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:01.398900+0000","last_undegraded":"2024-04-17T10:28:01.398900+0000","last_fullsized":"2024-04-17T10:28:01.398900+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":230,"reported_epoch":52,"state":"active+clean","last_fresh":"2024-04-17T10:28:01.381319+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:01.381319+0000","last_peered":"2024-04-17T10:28:01.381319+0000","last_clean":"2024-04-17T10:28:01.381319+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:01.381319+0000","last_undegraded":"2024-04-17T10:28:01.381319+0000","last_fullsized":"2024-04-17T10:28:01.381319+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36528,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335312,"statfs":{"total":96636764160,"available":96599359488,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35952,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335888,"statfs":{"total":96636764160,"available":96599949312,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40104,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331736,"statfs":{"total":96636764160,"available":96595697664,"internally_reserved":0,"allocated":606208,"data_stored":270114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35948,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335892,"statfs":{"total":96636764160,"available":96599953408,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35956,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335884,"statfs":{"total":96636764160,"available":96599945216,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35944,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335896,"statfs":{"total":96636764160,"available":96599957504,"internally_reserved":0,"allocated":618496,"data_stored":276781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58732,"kb_used_data":1184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313108,"statfs":{"total":96636764160,"available":96576622592,"internally_reserved":0,"allocated":1212416,"data_stored":867149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:28:11.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:28:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.214+0000 7f9a38600640 1 Processor -- start 2024-04-17T10:28:11.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.214+0000 7f9a38600640 1 -- start start 2024-04-17T10:28:11.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a38600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a30145040 0x7f9a3006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a38600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a30071a20 0x7f9a300cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a38600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a3006e2f0 con 0x7f9a3006ef10 2024-04-17T10:28:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a38600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a300a86c0 con 0x7f9a30145040 2024-04-17T10:28:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a38600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a3015bf10 con 0x7f9a30071a20 2024-04-17T10:28:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a35c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a30145040 0x7f9a3006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a35c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a30145040 0x7f9a3006e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60282/0 (socket says 172.21.15.67:60282) 2024-04-17T10:28:11.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a35c00640 1 -- 172.21.15.67:0/150861243 learned_addr learned my addr 172.21.15.67:0/150861243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a36600640 1 --2- 172.21.15.67:0/150861243 >> v2:172.21.15.86:3300/0 conn(0x7f9a30071a20 0x7f9a300cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a35c00640 1 -- 172.21.15.67:0/150861243 >> v2:172.21.15.86:3300/0 conn(0x7f9a30071a20 msgr2=0x7f9a300cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a35c00640 1 --2- 172.21.15.67:0/150861243 >> v2:172.21.15.86:3300/0 conn(0x7f9a30071a20 0x7f9a300cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a35c00640 1 -- 172.21.15.67:0/150861243 >> v1:172.21.15.86:6789/0 conn(0x7f9a3006ef10 legacy=0x7f9a30071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a35c00640 1 -- 172.21.15.67:0/150861243 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a3006d6e0 con 0x7f9a30145040 2024-04-17T10:28:11.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.218+0000 7f9a36600640 1 --2- 172.21.15.67:0/150861243 >> v2:172.21.15.86:3300/0 conn(0x7f9a30071a20 0x7f9a300cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:11.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a35c00640 1 --2- 172.21.15.67:0/150861243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30145040 0x7f9a3006e9d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9a300af390 tx=0x7f9a2401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616b80a72bea5172 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:11.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a2fe00640 1 -- 172.21.15.67:0/150861243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a240b5070 con 0x7f9a30145040 2024-04-17T10:28:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a2fe00640 1 -- 172.21.15.67:0/150861243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a240a89d0 con 0x7f9a30145040 2024-04-17T10:28:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a2fe00640 1 -- 172.21.15.67:0/150861243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a240a8da0 con 0x7f9a30145040 2024-04-17T10:28:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 -- 172.21.15.67:0/150861243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30145040 msgr2=0x7f9a3006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 --2- 172.21.15.67:0/150861243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30145040 0x7f9a3006e9d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9a300af390 tx=0x7f9a2401ecd0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 -- 172.21.15.67:0/150861243 shutdown_connections 2024-04-17T10:28:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 --2- 172.21.15.67:0/150861243 >> v2:172.21.15.86:3300/0 conn(0x7f9a30071a20 0x7f9a300cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 --2- 172.21.15.67:0/150861243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30145040 0x7f9a3006e9d0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 -- 172.21.15.67:0/150861243 >> 172.21.15.67:0/150861243 conn(0x7f9a3006ac90 msgr2=0x7f9a3006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 -- 172.21.15.67:0/150861243 shutdown_connections 2024-04-17T10:28:11.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 -- 172.21.15.67:0/150861243 wait complete. 2024-04-17T10:28:11.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.222+0000 7f9a38600640 1 Processor -- start 2024-04-17T10:28:11.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 -- start start 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a3006ef10 0x7f9a301383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30071a20 0x7f9a3013cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a30145040 0x7f9a30150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3006df80 con 0x7f9a30071a20 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3006dce0 con 0x7f9a30145040 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3006d370 con 0x7f9a3006ef10 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30071a20 0x7f9a3013cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30071a20 0x7f9a3013cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60290/0 (socket says 172.21.15.67:60290) 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 -- 172.21.15.67:0/1007325584 learned_addr learned my addr 172.21.15.67:0/1007325584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a36600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a30145040 0x7f9a30150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35c00640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a3006ef10 0x7f9a301383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 -- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a3006ef10 msgr2=0x7f9a301383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a3006ef10 0x7f9a301383a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 -- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a30145040 msgr2=0x7f9a30150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a30145040 0x7f9a30150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 -- 172.21.15.67:0/1007325584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a30138a10 con 0x7f9a30071a20 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a35200640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30071a20 0x7f9a3013cd70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00c450 tx=0x7f9a1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a2d600640 1 -- 172.21.15.67:0/1007325584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c012830 con 0x7f9a30071a20 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a2d600640 1 -- 172.21.15.67:0/1007325584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1c017070 con 0x7f9a30071a20 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a2d600640 1 -- 172.21.15.67:0/1007325584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c012c30 con 0x7f9a30071a20 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a30138cf0 con 0x7f9a30071a20 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a301392b0 con 0x7f9a30071a20 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.226+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a30072d50 con 0x7f9a30071a20 2024-04-17T10:28:11.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.230+0000 7f9a2d600640 1 -- 172.21.15.67:0/1007325584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9a1c01c460 con 0x7f9a30071a20 2024-04-17T10:28:11.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.230+0000 7f9a2d600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a10066f60 0x7f9a10069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.230+0000 7f9a2d600640 1 -- 172.21.15.67:0/1007325584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9a1c084e20 con 0x7f9a30071a20 2024-04-17T10:28:11.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.230+0000 7f9a35c00640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a10066f60 0x7f9a10069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.230+0000 7f9a35c00640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a10066f60 0x7f9a10069420 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9a240b2000 tx=0x7f9a24009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:11.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.234+0000 7f9a2d600640 1 -- 172.21.15.67:0/1007325584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9a1c055640 con 0x7f9a30071a20 2024-04-17T10:28:11.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.390+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a30074e30 con 0x7f9a10066f60 2024-04-17T10:28:11.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.398+0000 7f9a2d600640 1 -- 172.21.15.67:0/1007325584 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+251957 (secure 0 0 0) 0x7f9a30074e30 con 0x7f9a10066f60 2024-04-17T10:28:11.396 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:28:11.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a10066f60 msgr2=0x7f9a10069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a10066f60 0x7f9a10069420 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9a240b2000 tx=0x7f9a24009040 comp rx=0 tx=0).stop 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30071a20 msgr2=0x7f9a3013cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30071a20 0x7f9a3013cd70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c00c450 tx=0x7f9a1c00c910 comp rx=0 tx=0).stop 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 shutdown_connections 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a10066f60 0x7f9a10069420 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a30145040 0x7f9a30150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a30071a20 0x7f9a3013cd70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 --2- 172.21.15.67:0/1007325584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a3006ef10 0x7f9a301383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 >> 172.21.15.67:0/1007325584 conn(0x7f9a3006ac90 msgr2=0x7f9a300af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 shutdown_connections 2024-04-17T10:28:11.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.402+0000 7f9a38600640 1 -- 172.21.15.67:0/1007325584 wait complete. 2024-04-17T10:28:11.415 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-04-17T10:28:10.178922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":315372,"kb_used_data":6248,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754659348,"statfs":{"total":773094113280,"available":772771172352,"internally_reserved":0,"allocated":6397952,"data_stored":3510676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001952"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507527+0000","last_change":"2024-04-17T10:28:02.412794+0000","last_active":"2024-04-17T10:28:03.507527+0000","last_peered":"2024-04-17T10:28:03.507527+0000","last_clean":"2024-04-17T10:28:03.507527+0000","last_became_active":"2024-04-17T10:28:02.411140+0000","last_became_peered":"2024-04-17T10:28:02.411140+0000","last_unstale":"2024-04-17T10:28:03.507527+0000","last_undegraded":"2024-04-17T10:28:03.507527+0000","last_fullsized":"2024-04-17T10:28:03.507527+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:36.286284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403583+0000","last_change":"2024-04-17T10:28:02.407331+0000","last_active":"2024-04-17T10:28:03.403583+0000","last_peered":"2024-04-17T10:28:03.403583+0000","last_clean":"2024-04-17T10:28:03.403583+0000","last_became_active":"2024-04-17T10:28:02.406772+0000","last_became_peered":"2024-04-17T10:28:02.406772+0000","last_unstale":"2024-04-17T10:28:03.403583+0000","last_undegraded":"2024-04-17T10:28:03.403583+0000","last_fullsized":"2024-04-17T10:28:03.403583+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:42.908712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403658+0000","last_change":"2024-04-17T10:28:02.408638+0000","last_active":"2024-04-17T10:28:03.403658+0000","last_peered":"2024-04-17T10:28:03.403658+0000","last_clean":"2024-04-17T10:28:03.403658+0000","last_became_active":"2024-04-17T10:28:02.406234+0000","last_became_peered":"2024-04-17T10:28:02.406234+0000","last_unstale":"2024-04-17T10:28:03.403658+0000","last_undegraded":"2024-04-17T10:28:03.403658+0000","last_fullsized":"2024-04-17T10:28:03.403658+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:22.869254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403439+0000","last_change":"2024-04-17T10:28:02.407330+0000","last_active":"2024-04-17T10:28:03.403439+0000","last_peered":"2024-04-17T10:28:03.403439+0000","last_clean":"2024-04-17T10:28:03.403439+0000","last_became_active":"2024-04-17T10:28:02.406906+0000","last_became_peered":"2024-04-17T10:28:02.406906+0000","last_unstale":"2024-04-17T10:28:03.403439+0000","last_undegraded":"2024-04-17T10:28:03.403439+0000","last_fullsized":"2024-04-17T10:28:03.403439+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:37.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.857167+0000","last_change":"2024-04-17T10:28:02.395190+0000","last_active":"2024-04-17T10:28:03.857167+0000","last_peered":"2024-04-17T10:28:03.857167+0000","last_clean":"2024-04-17T10:28:03.857167+0000","last_became_active":"2024-04-17T10:28:02.394910+0000","last_became_peered":"2024-04-17T10:28:02.394910+0000","last_unstale":"2024-04-17T10:28:03.857167+0000","last_undegraded":"2024-04-17T10:28:03.857167+0000","last_fullsized":"2024-04-17T10:28:03.857167+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:32.091308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507180+0000","last_change":"2024-04-17T10:28:02.409751+0000","last_active":"2024-04-17T10:28:03.507180+0000","last_peered":"2024-04-17T10:28:03.507180+0000","last_clean":"2024-04-17T10:28:03.507180+0000","last_became_active":"2024-04-17T10:28:02.409278+0000","last_became_peered":"2024-04-17T10:28:02.409278+0000","last_unstale":"2024-04-17T10:28:03.507180+0000","last_undegraded":"2024-04-17T10:28:03.507180+0000","last_fullsized":"2024-04-17T10:28:03.507180+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:20.397617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403274+0000","last_change":"2024-04-17T10:28:02.406944+0000","last_active":"2024-04-17T10:28:03.403274+0000","last_peered":"2024-04-17T10:28:03.403274+0000","last_clean":"2024-04-17T10:28:03.403274+0000","last_became_active":"2024-04-17T10:28:02.406630+0000","last_became_peered":"2024-04-17T10:28:02.406630+0000","last_unstale":"2024-04-17T10:28:03.403274+0000","last_undegraded":"2024-04-17T10:28:03.403274+0000","last_fullsized":"2024-04-17T10:28:03.403274+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:01.322413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401389+0000","last_change":"2024-04-17T10:28:02.408878+0000","last_active":"2024-04-17T10:28:03.401389+0000","last_peered":"2024-04-17T10:28:03.401389+0000","last_clean":"2024-04-17T10:28:03.401389+0000","last_became_active":"2024-04-17T10:28:02.408100+0000","last_became_peered":"2024-04-17T10:28:02.408100+0000","last_unstale":"2024-04-17T10:28:03.401389+0000","last_undegraded":"2024-04-17T10:28:03.401389+0000","last_fullsized":"2024-04-17T10:28:03.401389+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:51:35.141501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401676+0000","last_change":"2024-04-17T10:28:02.407953+0000","last_active":"2024-04-17T10:28:03.401676+0000","last_peered":"2024-04-17T10:28:03.401676+0000","last_clean":"2024-04-17T10:28:03.401676+0000","last_became_active":"2024-04-17T10:28:02.407415+0000","last_became_peered":"2024-04-17T10:28:02.407415+0000","last_unstale":"2024-04-17T10:28:03.401676+0000","last_undegraded":"2024-04-17T10:28:03.401676+0000","last_fullsized":"2024-04-17T10:28:03.401676+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:02:36.869056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507474+0000","last_change":"2024-04-17T10:28:02.405137+0000","last_active":"2024-04-17T10:28:03.507474+0000","last_peered":"2024-04-17T10:28:03.507474+0000","last_clean":"2024-04-17T10:28:03.507474+0000","last_became_active":"2024-04-17T10:28:02.404266+0000","last_became_peered":"2024-04-17T10:28:02.404266+0000","last_unstale":"2024-04-17T10:28:03.507474+0000","last_undegraded":"2024-04-17T10:28:03.507474+0000","last_fullsized":"2024-04-17T10:28:03.507474+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:13.361871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.361645+0000","last_change":"2024-04-17T10:28:02.402258+0000","last_active":"2024-04-17T10:28:04.361645+0000","last_peered":"2024-04-17T10:28:04.361645+0000","last_clean":"2024-04-17T10:28:04.361645+0000","last_became_active":"2024-04-17T10:28:02.398421+0000","last_became_peered":"2024-04-17T10:28:02.398421+0000","last_unstale":"2024-04-17T10:28:04.361645+0000","last_undegraded":"2024-04-17T10:28:04.361645+0000","last_fullsized":"2024-04-17T10:28:04.361645+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:46:01.645014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401363+0000","last_change":"2024-04-17T10:28:02.395602+0000","last_active":"2024-04-17T10:28:03.401363+0000","last_peered":"2024-04-17T10:28:03.401363+0000","last_clean":"2024-04-17T10:28:03.401363+0000","last_became_active":"2024-04-17T10:28:02.394718+0000","last_became_peered":"2024-04-17T10:28:02.394718+0000","last_unstale":"2024-04-17T10:28:03.401363+0000","last_undegraded":"2024-04-17T10:28:03.401363+0000","last_fullsized":"2024-04-17T10:28:03.401363+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:01.064245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507257+0000","last_change":"2024-04-17T10:28:02.405168+0000","last_active":"2024-04-17T10:28:03.507257+0000","last_peered":"2024-04-17T10:28:03.507257+0000","last_clean":"2024-04-17T10:28:03.507257+0000","last_became_active":"2024-04-17T10:28:02.404896+0000","last_became_peered":"2024-04-17T10:28:02.404896+0000","last_unstale":"2024-04-17T10:28:03.507257+0000","last_undegraded":"2024-04-17T10:28:03.507257+0000","last_fullsized":"2024-04-17T10:28:03.507257+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:00.082987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403344+0000","last_change":"2024-04-17T10:28:02.408508+0000","last_active":"2024-04-17T10:28:03.403344+0000","last_peered":"2024-04-17T10:28:03.403344+0000","last_clean":"2024-04-17T10:28:03.403344+0000","last_became_active":"2024-04-17T10:28:02.406731+0000","last_became_peered":"2024-04-17T10:28:02.406731+0000","last_unstale":"2024-04-17T10:28:03.403344+0000","last_undegraded":"2024-04-17T10:28:03.403344+0000","last_fullsized":"2024-04-17T10:28:03.403344+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:10:23.107168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403316+0000","last_change":"2024-04-17T10:28:02.394873+0000","last_active":"2024-04-17T10:28:03.403316+0000","last_peered":"2024-04-17T10:28:03.403316+0000","last_clean":"2024-04-17T10:28:03.403316+0000","last_became_active":"2024-04-17T10:28:02.393875+0000","last_became_peered":"2024-04-17T10:28:02.393875+0000","last_unstale":"2024-04-17T10:28:03.403316+0000","last_undegraded":"2024-04-17T10:28:03.403316+0000","last_fullsized":"2024-04-17T10:28:03.403316+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:17.213038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507554+0000","last_change":"2024-04-17T10:28:02.403612+0000","last_active":"2024-04-17T10:28:03.507554+0000","last_peered":"2024-04-17T10:28:03.507554+0000","last_clean":"2024-04-17T10:28:03.507554+0000","last_became_active":"2024-04-17T10:28:02.403355+0000","last_became_peered":"2024-04-17T10:28:02.403355+0000","last_unstale":"2024-04-17T10:28:03.507554+0000","last_undegraded":"2024-04-17T10:28:03.507554+0000","last_fullsized":"2024-04-17T10:28:03.507554+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:16.833123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.404059+0000","last_change":"2024-04-17T10:28:02.417476+0000","last_active":"2024-04-17T10:28:03.404059+0000","last_peered":"2024-04-17T10:28:03.404059+0000","last_clean":"2024-04-17T10:28:03.404059+0000","last_became_active":"2024-04-17T10:28:02.416855+0000","last_became_peered":"2024-04-17T10:28:02.416855+0000","last_unstale":"2024-04-17T10:28:03.404059+0000","last_undegraded":"2024-04-17T10:28:03.404059+0000","last_fullsized":"2024-04-17T10:28:03.404059+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:25:26.574522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403766+0000","last_change":"2024-04-17T10:28:02.407044+0000","last_active":"2024-04-17T10:28:03.403766+0000","last_peered":"2024-04-17T10:28:03.403766+0000","last_clean":"2024-04-17T10:28:03.403766+0000","last_became_active":"2024-04-17T10:28:02.406284+0000","last_became_peered":"2024-04-17T10:28:02.406284+0000","last_unstale":"2024-04-17T10:28:03.403766+0000","last_undegraded":"2024-04-17T10:28:03.403766+0000","last_fullsized":"2024-04-17T10:28:03.403766+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:25:26.574522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403863+0000","last_change":"2024-04-17T10:28:02.400511+0000","last_active":"2024-04-17T10:28:03.403863+0000","last_peered":"2024-04-17T10:28:03.403863+0000","last_clean":"2024-04-17T10:28:03.403863+0000","last_became_active":"2024-04-17T10:28:02.399934+0000","last_became_peered":"2024-04-17T10:28:02.399934+0000","last_unstale":"2024-04-17T10:28:03.403863+0000","last_undegraded":"2024-04-17T10:28:03.403863+0000","last_fullsized":"2024-04-17T10:28:03.403863+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:43.462557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398283+0000","last_change":"2024-04-17T10:28:02.407264+0000","last_active":"2024-04-17T10:28:03.398283+0000","last_peered":"2024-04-17T10:28:03.398283+0000","last_clean":"2024-04-17T10:28:03.398283+0000","last_became_active":"2024-04-17T10:28:02.406823+0000","last_became_peered":"2024-04-17T10:28:02.406823+0000","last_unstale":"2024-04-17T10:28:03.398283+0000","last_undegraded":"2024-04-17T10:28:03.398283+0000","last_fullsized":"2024-04-17T10:28:03.398283+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:43.618266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398192+0000","last_change":"2024-04-17T10:28:02.392161+0000","last_active":"2024-04-17T10:28:03.398192+0000","last_peered":"2024-04-17T10:28:03.398192+0000","last_clean":"2024-04-17T10:28:03.398192+0000","last_became_active":"2024-04-17T10:28:02.391765+0000","last_became_peered":"2024-04-17T10:28:02.391765+0000","last_unstale":"2024-04-17T10:28:03.398192+0000","last_undegraded":"2024-04-17T10:28:03.398192+0000","last_fullsized":"2024-04-17T10:28:03.398192+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:29.858078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.400907+0000","last_change":"2024-04-17T10:28:02.405864+0000","last_active":"2024-04-17T10:28:03.400907+0000","last_peered":"2024-04-17T10:28:03.400907+0000","last_clean":"2024-04-17T10:28:03.400907+0000","last_became_active":"2024-04-17T10:28:02.405604+0000","last_became_peered":"2024-04-17T10:28:02.405604+0000","last_unstale":"2024-04-17T10:28:03.400907+0000","last_undegraded":"2024-04-17T10:28:03.400907+0000","last_fullsized":"2024-04-17T10:28:03.400907+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:41:26.369169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.361451+0000","last_change":"2024-04-17T10:28:02.397783+0000","last_active":"2024-04-17T10:28:04.361451+0000","last_peered":"2024-04-17T10:28:04.361451+0000","last_clean":"2024-04-17T10:28:04.361451+0000","last_became_active":"2024-04-17T10:28:02.397456+0000","last_became_peered":"2024-04-17T10:28:02.397456+0000","last_unstale":"2024-04-17T10:28:04.361451+0000","last_undegraded":"2024-04-17T10:28:04.361451+0000","last_fullsized":"2024-04-17T10:28:04.361451+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:30:39.501125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403204+0000","last_change":"2024-04-17T10:28:02.405519+0000","last_active":"2024-04-17T10:28:03.403204+0000","last_peered":"2024-04-17T10:28:03.403204+0000","last_clean":"2024-04-17T10:28:03.403204+0000","last_became_active":"2024-04-17T10:28:02.403524+0000","last_became_peered":"2024-04-17T10:28:02.403524+0000","last_unstale":"2024-04-17T10:28:03.403204+0000","last_undegraded":"2024-04-17T10:28:03.403204+0000","last_fullsized":"2024-04-17T10:28:03.403204+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:05:23.981715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.404196+0000","last_change":"2024-04-17T10:28:02.411187+0000","last_active":"2024-04-17T10:28:03.404196+0000","last_peered":"2024-04-17T10:28:03.404196+0000","last_clean":"2024-04-17T10:28:03.404196+0000","last_became_active":"2024-04-17T10:28:02.410234+0000","last_became_peered":"2024-04-17T10:28:02.410234+0000","last_unstale":"2024-04-17T10:28:03.404196+0000","last_undegraded":"2024-04-17T10:28:03.404196+0000","last_fullsized":"2024-04-17T10:28:03.404196+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:23.489475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401245+0000","last_change":"2024-04-17T10:28:02.407208+0000","last_active":"2024-04-17T10:28:03.401245+0000","last_peered":"2024-04-17T10:28:03.401245+0000","last_clean":"2024-04-17T10:28:03.401245+0000","last_became_active":"2024-04-17T10:28:02.400612+0000","last_became_peered":"2024-04-17T10:28:02.400612+0000","last_unstale":"2024-04-17T10:28:03.401245+0000","last_undegraded":"2024-04-17T10:28:03.401245+0000","last_fullsized":"2024-04-17T10:28:03.401245+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:37:24.778589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507419+0000","last_change":"2024-04-17T10:28:02.403607+0000","last_active":"2024-04-17T10:28:03.507419+0000","last_peered":"2024-04-17T10:28:03.507419+0000","last_clean":"2024-04-17T10:28:03.507419+0000","last_became_active":"2024-04-17T10:28:02.402875+0000","last_became_peered":"2024-04-17T10:28:02.402875+0000","last_unstale":"2024-04-17T10:28:03.507419+0000","last_undegraded":"2024-04-17T10:28:03.507419+0000","last_fullsized":"2024-04-17T10:28:03.507419+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:53:38.220758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398401+0000","last_change":"2024-04-17T10:28:02.401386+0000","last_active":"2024-04-17T10:28:03.398401+0000","last_peered":"2024-04-17T10:28:03.398401+0000","last_clean":"2024-04-17T10:28:03.398401+0000","last_became_active":"2024-04-17T10:28:02.400994+0000","last_became_peered":"2024-04-17T10:28:02.400994+0000","last_unstale":"2024-04-17T10:28:03.398401+0000","last_undegraded":"2024-04-17T10:28:03.398401+0000","last_fullsized":"2024-04-17T10:28:03.398401+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:32.663337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403818+0000","last_change":"2024-04-17T10:28:02.395222+0000","last_active":"2024-04-17T10:28:03.403818+0000","last_peered":"2024-04-17T10:28:03.403818+0000","last_clean":"2024-04-17T10:28:03.403818+0000","last_became_active":"2024-04-17T10:28:02.392345+0000","last_became_peered":"2024-04-17T10:28:02.392345+0000","last_unstale":"2024-04-17T10:28:03.403818+0000","last_undegraded":"2024-04-17T10:28:03.403818+0000","last_fullsized":"2024-04-17T10:28:03.403818+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:37.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403677+0000","last_change":"2024-04-17T10:28:02.396685+0000","last_active":"2024-04-17T10:28:03.403677+0000","last_peered":"2024-04-17T10:28:03.403677+0000","last_clean":"2024-04-17T10:28:03.403677+0000","last_became_active":"2024-04-17T10:28:02.396433+0000","last_became_peered":"2024-04-17T10:28:02.396433+0000","last_unstale":"2024-04-17T10:28:03.403677+0000","last_undegraded":"2024-04-17T10:28:03.403677+0000","last_fullsized":"2024-04-17T10:28:03.403677+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:04.469543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403355+0000","last_change":"2024-04-17T10:28:02.407030+0000","last_active":"2024-04-17T10:28:03.403355+0000","last_peered":"2024-04-17T10:28:03.403355+0000","last_clean":"2024-04-17T10:28:03.403355+0000","last_became_active":"2024-04-17T10:28:02.403910+0000","last_became_peered":"2024-04-17T10:28:02.403910+0000","last_unstale":"2024-04-17T10:28:03.403355+0000","last_undegraded":"2024-04-17T10:28:03.403355+0000","last_fullsized":"2024-04-17T10:28:03.403355+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:37.233983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.357994+0000","last_change":"2024-04-17T10:28:02.411154+0000","last_active":"2024-04-17T10:28:04.357994+0000","last_peered":"2024-04-17T10:28:04.357994+0000","last_clean":"2024-04-17T10:28:04.357994+0000","last_became_active":"2024-04-17T10:28:02.410831+0000","last_became_peered":"2024-04-17T10:28:02.410831+0000","last_unstale":"2024-04-17T10:28:04.357994+0000","last_undegraded":"2024-04-17T10:28:04.357994+0000","last_fullsized":"2024-04-17T10:28:04.357994+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:37.211990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403166+0000","last_change":"2024-04-17T10:28:02.405033+0000","last_active":"2024-04-17T10:28:03.403166+0000","last_peered":"2024-04-17T10:28:03.403166+0000","last_clean":"2024-04-17T10:28:03.403166+0000","last_became_active":"2024-04-17T10:28:02.404763+0000","last_became_peered":"2024-04-17T10:28:02.404763+0000","last_unstale":"2024-04-17T10:28:03.403166+0000","last_undegraded":"2024-04-17T10:28:03.403166+0000","last_fullsized":"2024-04-17T10:28:03.403166+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:01.079796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"53'4","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398537+0000","last_change":"2024-04-17T10:28:01.380916+0000","last_active":"2024-04-17T10:28:03.398537+0000","last_peered":"2024-04-17T10:28:03.398537+0000","last_clean":"2024-04-17T10:28:03.398537+0000","last_became_active":"2024-04-17T10:28:01.380635+0000","last_became_peered":"2024-04-17T10:28:01.380635+0000","last_unstale":"2024-04-17T10:28:03.398537+0000","last_undegraded":"2024-04-17T10:28:03.398537+0000","last_fullsized":"2024-04-17T10:28:03.398537+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:03:16.384430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.358181+0000","last_change":"2024-04-17T10:28:02.411280+0000","last_active":"2024-04-17T10:28:04.358181+0000","last_peered":"2024-04-17T10:28:04.358181+0000","last_clean":"2024-04-17T10:28:04.358181+0000","last_became_active":"2024-04-17T10:28:02.410401+0000","last_became_peered":"2024-04-17T10:28:02.410401+0000","last_unstale":"2024-04-17T10:28:04.358181+0000","last_undegraded":"2024-04-17T10:28:04.358181+0000","last_fullsized":"2024-04-17T10:28:04.358181+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:43.462557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507436+0000","last_change":"2024-04-17T10:28:01.391871+0000","last_active":"2024-04-17T10:28:03.507436+0000","last_peered":"2024-04-17T10:28:03.507436+0000","last_clean":"2024-04-17T10:28:03.507436+0000","last_became_active":"2024-04-17T10:28:01.391540+0000","last_became_peered":"2024-04-17T10:28:01.391540+0000","last_unstale":"2024-04-17T10:28:03.507436+0000","last_undegraded":"2024-04-17T10:28:03.507436+0000","last_fullsized":"2024-04-17T10:28:03.507436+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398087+0000","last_change":"2024-04-17T10:28:02.390006+0000","last_active":"2024-04-17T10:28:03.398087+0000","last_peered":"2024-04-17T10:28:03.398087+0000","last_clean":"2024-04-17T10:28:03.398087+0000","last_became_active":"2024-04-17T10:28:02.389701+0000","last_became_peered":"2024-04-17T10:28:02.389701+0000","last_unstale":"2024-04-17T10:28:03.398087+0000","last_undegraded":"2024-04-17T10:28:03.398087+0000","last_fullsized":"2024-04-17T10:28:03.398087+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:20.383306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507405+0000","last_change":"2024-04-17T10:28:01.392616+0000","last_active":"2024-04-17T10:28:03.507405+0000","last_peered":"2024-04-17T10:28:03.507405+0000","last_clean":"2024-04-17T10:28:03.507405+0000","last_became_active":"2024-04-17T10:28:01.391986+0000","last_became_peered":"2024-04-17T10:28:01.391986+0000","last_unstale":"2024-04-17T10:28:03.507405+0000","last_undegraded":"2024-04-17T10:28:03.507405+0000","last_fullsized":"2024-04-17T10:28:03.507405+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401120+0000","last_change":"2024-04-17T10:28:02.393718+0000","last_active":"2024-04-17T10:28:03.401120+0000","last_peered":"2024-04-17T10:28:03.401120+0000","last_clean":"2024-04-17T10:28:03.401120+0000","last_became_active":"2024-04-17T10:28:02.393443+0000","last_became_peered":"2024-04-17T10:28:02.393443+0000","last_unstale":"2024-04-17T10:28:03.401120+0000","last_undegraded":"2024-04-17T10:28:03.401120+0000","last_fullsized":"2024-04-17T10:28:03.401120+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:49.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.358397+0000","last_change":"2024-04-17T10:28:01.380404+0000","last_active":"2024-04-17T10:28:04.358397+0000","last_peered":"2024-04-17T10:28:04.358397+0000","last_clean":"2024-04-17T10:28:04.358397+0000","last_became_active":"2024-04-17T10:28:01.380065+0000","last_became_peered":"2024-04-17T10:28:01.380065+0000","last_unstale":"2024-04-17T10:28:04.358397+0000","last_undegraded":"2024-04-17T10:28:04.358397+0000","last_fullsized":"2024-04-17T10:28:04.358397+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:05:15.827189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.857324+0000","last_change":"2024-04-17T10:28:02.400747+0000","last_active":"2024-04-17T10:28:03.857324+0000","last_peered":"2024-04-17T10:28:03.857324+0000","last_clean":"2024-04-17T10:28:03.857324+0000","last_became_active":"2024-04-17T10:28:02.400465+0000","last_became_peered":"2024-04-17T10:28:02.400465+0000","last_unstale":"2024-04-17T10:28:03.857324+0000","last_undegraded":"2024-04-17T10:28:03.857324+0000","last_fullsized":"2024-04-17T10:28:03.857324+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:36:20.397617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403537+0000","last_change":"2024-04-17T10:28:01.395609+0000","last_active":"2024-04-17T10:28:03.403537+0000","last_peered":"2024-04-17T10:28:03.403537+0000","last_clean":"2024-04-17T10:28:03.403537+0000","last_became_active":"2024-04-17T10:28:01.395267+0000","last_became_peered":"2024-04-17T10:28:01.395267+0000","last_unstale":"2024-04-17T10:28:03.403537+0000","last_undegraded":"2024-04-17T10:28:03.403537+0000","last_fullsized":"2024-04-17T10:28:03.403537+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:43:15.590068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403517+0000","last_change":"2024-04-17T10:28:02.397450+0000","last_active":"2024-04-17T10:28:03.403517+0000","last_peered":"2024-04-17T10:28:03.403517+0000","last_clean":"2024-04-17T10:28:03.403517+0000","last_became_active":"2024-04-17T10:28:02.397178+0000","last_became_peered":"2024-04-17T10:28:02.397178+0000","last_unstale":"2024-04-17T10:28:03.403517+0000","last_undegraded":"2024-04-17T10:28:03.403517+0000","last_fullsized":"2024-04-17T10:28:03.403517+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:43.462557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.856891+0000","last_change":"2024-04-17T10:28:01.378483+0000","last_active":"2024-04-17T10:28:03.856891+0000","last_peered":"2024-04-17T10:28:03.856891+0000","last_clean":"2024-04-17T10:28:03.856891+0000","last_became_active":"2024-04-17T10:28:01.378140+0000","last_became_peered":"2024-04-17T10:28:01.378140+0000","last_unstale":"2024-04-17T10:28:03.856891+0000","last_undegraded":"2024-04-17T10:28:03.856891+0000","last_fullsized":"2024-04-17T10:28:03.856891+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:04:14.700196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401418+0000","last_change":"2024-04-17T10:28:02.400318+0000","last_active":"2024-04-17T10:28:03.401418+0000","last_peered":"2024-04-17T10:28:03.401418+0000","last_clean":"2024-04-17T10:28:03.401418+0000","last_became_active":"2024-04-17T10:28:02.399821+0000","last_became_peered":"2024-04-17T10:28:02.399821+0000","last_unstale":"2024-04-17T10:28:03.401418+0000","last_undegraded":"2024-04-17T10:28:03.401418+0000","last_fullsized":"2024-04-17T10:28:03.401418+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:01.946166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.397835+0000","last_change":"2024-04-17T10:28:01.378144+0000","last_active":"2024-04-17T10:28:03.397835+0000","last_peered":"2024-04-17T10:28:03.397835+0000","last_clean":"2024-04-17T10:28:03.397835+0000","last_became_active":"2024-04-17T10:28:01.377879+0000","last_became_peered":"2024-04-17T10:28:01.377879+0000","last_unstale":"2024-04-17T10:28:03.397835+0000","last_undegraded":"2024-04-17T10:28:03.397835+0000","last_fullsized":"2024-04-17T10:28:03.397835+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:21.904261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403607+0000","last_change":"2024-04-17T10:28:02.405805+0000","last_active":"2024-04-17T10:28:03.403607+0000","last_peered":"2024-04-17T10:28:03.403607+0000","last_clean":"2024-04-17T10:28:03.403607+0000","last_became_active":"2024-04-17T10:28:02.405518+0000","last_became_peered":"2024-04-17T10:28:02.405518+0000","last_unstale":"2024-04-17T10:28:03.403607+0000","last_undegraded":"2024-04-17T10:28:03.403607+0000","last_fullsized":"2024-04-17T10:28:03.403607+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:52.444597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403304+0000","last_change":"2024-04-17T10:28:01.376007+0000","last_active":"2024-04-17T10:28:03.403304+0000","last_peered":"2024-04-17T10:28:03.403304+0000","last_clean":"2024-04-17T10:28:03.403304+0000","last_became_active":"2024-04-17T10:28:01.375098+0000","last_became_peered":"2024-04-17T10:28:01.375098+0000","last_unstale":"2024-04-17T10:28:03.403304+0000","last_undegraded":"2024-04-17T10:28:03.403304+0000","last_fullsized":"2024-04-17T10:28:03.403304+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:08:26.053114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.856937+0000","last_change":"2024-04-17T10:28:02.397631+0000","last_active":"2024-04-17T10:28:03.856937+0000","last_peered":"2024-04-17T10:28:03.856937+0000","last_clean":"2024-04-17T10:28:03.856937+0000","last_became_active":"2024-04-17T10:28:02.397123+0000","last_became_peered":"2024-04-17T10:28:02.397123+0000","last_unstale":"2024-04-17T10:28:03.856937+0000","last_undegraded":"2024-04-17T10:28:03.856937+0000","last_fullsized":"2024-04-17T10:28:03.856937+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:16:09.311166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403694+0000","last_change":"2024-04-17T10:28:01.390159+0000","last_active":"2024-04-17T10:28:03.403694+0000","last_peered":"2024-04-17T10:28:03.403694+0000","last_clean":"2024-04-17T10:28:03.403694+0000","last_became_active":"2024-04-17T10:28:01.389874+0000","last_became_peered":"2024-04-17T10:28:01.389874+0000","last_unstale":"2024-04-17T10:28:03.403694+0000","last_undegraded":"2024-04-17T10:28:03.403694+0000","last_fullsized":"2024-04-17T10:28:03.403694+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:35.280350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.357983+0000","last_change":"2024-04-17T10:28:02.407855+0000","last_active":"2024-04-17T10:28:04.357983+0000","last_peered":"2024-04-17T10:28:04.357983+0000","last_clean":"2024-04-17T10:28:04.357983+0000","last_became_active":"2024-04-17T10:28:02.407583+0000","last_became_peered":"2024-04-17T10:28:02.407583+0000","last_unstale":"2024-04-17T10:28:04.357983+0000","last_undegraded":"2024-04-17T10:28:04.357983+0000","last_fullsized":"2024-04-17T10:28:04.357983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:17.213038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.857345+0000","last_change":"2024-04-17T10:28:01.396322+0000","last_active":"2024-04-17T10:28:03.857345+0000","last_peered":"2024-04-17T10:28:03.857345+0000","last_clean":"2024-04-17T10:28:03.857345+0000","last_became_active":"2024-04-17T10:28:01.396090+0000","last_became_peered":"2024-04-17T10:28:01.396090+0000","last_unstale":"2024-04-17T10:28:03.857345+0000","last_undegraded":"2024-04-17T10:28:03.857345+0000","last_fullsized":"2024-04-17T10:28:03.857345+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:08:26.053114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403701+0000","last_change":"2024-04-17T10:28:02.399194+0000","last_active":"2024-04-17T10:28:03.403701+0000","last_peered":"2024-04-17T10:28:03.403701+0000","last_clean":"2024-04-17T10:28:03.403701+0000","last_became_active":"2024-04-17T10:28:02.398922+0000","last_became_peered":"2024-04-17T10:28:02.398922+0000","last_unstale":"2024-04-17T10:28:03.403701+0000","last_undegraded":"2024-04-17T10:28:03.403701+0000","last_fullsized":"2024-04-17T10:28:03.403701+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:17.213038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"53'3","reported_seq":17,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403634+0000","last_change":"2024-04-17T10:28:01.390653+0000","last_active":"2024-04-17T10:28:03.403634+0000","last_peered":"2024-04-17T10:28:03.403634+0000","last_clean":"2024-04-17T10:28:03.403634+0000","last_became_active":"2024-04-17T10:28:01.390327+0000","last_became_peered":"2024-04-17T10:28:01.390327+0000","last_unstale":"2024-04-17T10:28:03.403634+0000","last_undegraded":"2024-04-17T10:28:03.403634+0000","last_fullsized":"2024-04-17T10:28:03.403634+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:12.355937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.857150+0000","last_change":"2024-04-17T10:28:02.403687+0000","last_active":"2024-04-17T10:28:03.857150+0000","last_peered":"2024-04-17T10:28:03.857150+0000","last_clean":"2024-04-17T10:28:03.857150+0000","last_became_active":"2024-04-17T10:28:02.403411+0000","last_became_peered":"2024-04-17T10:28:02.403411+0000","last_unstale":"2024-04-17T10:28:03.857150+0000","last_undegraded":"2024-04-17T10:28:03.857150+0000","last_fullsized":"2024-04-17T10:28:03.857150+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:32.663337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"53'6","reported_seq":20,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401404+0000","last_change":"2024-04-17T10:28:01.390744+0000","last_active":"2024-04-17T10:28:03.401404+0000","last_peered":"2024-04-17T10:28:03.401404+0000","last_clean":"2024-04-17T10:28:03.401404+0000","last_became_active":"2024-04-17T10:28:01.390487+0000","last_became_peered":"2024-04-17T10:28:01.390487+0000","last_unstale":"2024-04-17T10:28:03.401404+0000","last_undegraded":"2024-04-17T10:28:03.401404+0000","last_fullsized":"2024-04-17T10:28:03.401404+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:36.228049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.357882+0000","last_change":"2024-04-17T10:28:02.411304+0000","last_active":"2024-04-17T10:28:04.357882+0000","last_peered":"2024-04-17T10:28:04.357882+0000","last_clean":"2024-04-17T10:28:04.357882+0000","last_became_active":"2024-04-17T10:28:02.410960+0000","last_became_peered":"2024-04-17T10:28:02.410960+0000","last_unstale":"2024-04-17T10:28:04.357882+0000","last_undegraded":"2024-04-17T10:28:04.357882+0000","last_fullsized":"2024-04-17T10:28:04.357882+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:52.444597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403638+0000","last_change":"2024-04-17T10:28:01.388726+0000","last_active":"2024-04-17T10:28:03.403638+0000","last_peered":"2024-04-17T10:28:03.403638+0000","last_clean":"2024-04-17T10:28:03.403638+0000","last_became_active":"2024-04-17T10:28:01.388462+0000","last_became_peered":"2024-04-17T10:28:01.388462+0000","last_unstale":"2024-04-17T10:28:03.403638+0000","last_undegraded":"2024-04-17T10:28:03.403638+0000","last_fullsized":"2024-04-17T10:28:03.403638+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:52:28.852145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403869+0000","last_change":"2024-04-17T10:28:02.400725+0000","last_active":"2024-04-17T10:28:03.403869+0000","last_peered":"2024-04-17T10:28:03.403869+0000","last_clean":"2024-04-17T10:28:03.403869+0000","last_became_active":"2024-04-17T10:28:02.400249+0000","last_became_peered":"2024-04-17T10:28:02.400249+0000","last_unstale":"2024-04-17T10:28:03.403869+0000","last_undegraded":"2024-04-17T10:28:03.403869+0000","last_fullsized":"2024-04-17T10:28:03.403869+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:42.908712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398225+0000","last_change":"2024-04-17T10:28:01.388333+0000","last_active":"2024-04-17T10:28:03.398225+0000","last_peered":"2024-04-17T10:28:03.398225+0000","last_clean":"2024-04-17T10:28:03.398225+0000","last_became_active":"2024-04-17T10:28:01.387931+0000","last_became_peered":"2024-04-17T10:28:01.387931+0000","last_unstale":"2024-04-17T10:28:03.398225+0000","last_undegraded":"2024-04-17T10:28:03.398225+0000","last_fullsized":"2024-04-17T10:28:03.398225+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:13.609709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401276+0000","last_change":"2024-04-17T10:28:02.409929+0000","last_active":"2024-04-17T10:28:03.401276+0000","last_peered":"2024-04-17T10:28:03.401276+0000","last_clean":"2024-04-17T10:28:03.401276+0000","last_became_active":"2024-04-17T10:28:02.407640+0000","last_became_peered":"2024-04-17T10:28:02.407640+0000","last_unstale":"2024-04-17T10:28:03.401276+0000","last_undegraded":"2024-04-17T10:28:03.401276+0000","last_fullsized":"2024-04-17T10:28:03.401276+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:04:25.088802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403419+0000","last_change":"2024-04-17T10:28:01.387372+0000","last_active":"2024-04-17T10:28:03.403419+0000","last_peered":"2024-04-17T10:28:03.403419+0000","last_clean":"2024-04-17T10:28:03.403419+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:03.403419+0000","last_undegraded":"2024-04-17T10:28:03.403419+0000","last_fullsized":"2024-04-17T10:28:03.403419+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:53:37.214825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.404024+0000","last_change":"2024-04-17T10:28:02.417264+0000","last_active":"2024-04-17T10:28:03.404024+0000","last_peered":"2024-04-17T10:28:03.404024+0000","last_clean":"2024-04-17T10:28:03.404024+0000","last_became_active":"2024-04-17T10:28:02.416200+0000","last_became_peered":"2024-04-17T10:28:02.416200+0000","last_unstale":"2024-04-17T10:28:03.404024+0000","last_undegraded":"2024-04-17T10:28:03.404024+0000","last_fullsized":"2024-04-17T10:28:03.404024+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:04.469543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"53'8","reported_seq":22,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401480+0000","last_change":"2024-04-17T10:28:01.390499+0000","last_active":"2024-04-17T10:28:03.401480+0000","last_peered":"2024-04-17T10:28:03.401480+0000","last_clean":"2024-04-17T10:28:03.401480+0000","last_became_active":"2024-04-17T10:28:01.390208+0000","last_became_peered":"2024-04-17T10:28:01.390208+0000","last_unstale":"2024-04-17T10:28:03.401480+0000","last_undegraded":"2024-04-17T10:28:03.401480+0000","last_fullsized":"2024-04-17T10:28:03.401480+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:16.207104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401469+0000","last_change":"2024-04-17T10:28:02.412141+0000","last_active":"2024-04-17T10:28:03.401469+0000","last_peered":"2024-04-17T10:28:03.401469+0000","last_clean":"2024-04-17T10:28:03.401469+0000","last_became_active":"2024-04-17T10:28:02.408563+0000","last_became_peered":"2024-04-17T10:28:02.408563+0000","last_unstale":"2024-04-17T10:28:03.401469+0000","last_undegraded":"2024-04-17T10:28:03.401469+0000","last_fullsized":"2024-04-17T10:28:03.401469+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:07:13.062762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.404182+0000","last_change":"2024-04-17T10:28:01.385970+0000","last_active":"2024-04-17T10:28:03.404182+0000","last_peered":"2024-04-17T10:28:03.404182+0000","last_clean":"2024-04-17T10:28:03.404182+0000","last_became_active":"2024-04-17T10:28:01.385447+0000","last_became_peered":"2024-04-17T10:28:01.385447+0000","last_unstale":"2024-04-17T10:28:03.404182+0000","last_undegraded":"2024-04-17T10:28:03.404182+0000","last_fullsized":"2024-04-17T10:28:03.404182+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:09:19.377372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507134+0000","last_change":"2024-04-17T10:28:02.403125+0000","last_active":"2024-04-17T10:28:03.507134+0000","last_peered":"2024-04-17T10:28:03.507134+0000","last_clean":"2024-04-17T10:28:03.507134+0000","last_became_active":"2024-04-17T10:28:02.402785+0000","last_became_peered":"2024-04-17T10:28:02.402785+0000","last_unstale":"2024-04-17T10:28:03.507134+0000","last_undegraded":"2024-04-17T10:28:03.507134+0000","last_fullsized":"2024-04-17T10:28:03.507134+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:32.091308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.361745+0000","last_change":"2024-04-17T10:28:01.393600+0000","last_active":"2024-04-17T10:28:04.361745+0000","last_peered":"2024-04-17T10:28:04.361745+0000","last_clean":"2024-04-17T10:28:04.361745+0000","last_became_active":"2024-04-17T10:28:01.393277+0000","last_became_peered":"2024-04-17T10:28:01.393277+0000","last_unstale":"2024-04-17T10:28:04.361745+0000","last_undegraded":"2024-04-17T10:28:04.361745+0000","last_fullsized":"2024-04-17T10:28:04.361745+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:27.777111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.856945+0000","last_change":"2024-04-17T10:28:02.400946+0000","last_active":"2024-04-17T10:28:03.856945+0000","last_peered":"2024-04-17T10:28:03.856945+0000","last_clean":"2024-04-17T10:28:03.856945+0000","last_became_active":"2024-04-17T10:28:02.400442+0000","last_became_peered":"2024-04-17T10:28:02.400442+0000","last_unstale":"2024-04-17T10:28:03.856945+0000","last_undegraded":"2024-04-17T10:28:03.856945+0000","last_fullsized":"2024-04-17T10:28:03.856945+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:43.618266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"53'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398280+0000","last_change":"2024-04-17T10:28:01.390060+0000","last_active":"2024-04-17T10:28:03.398280+0000","last_peered":"2024-04-17T10:28:03.398280+0000","last_clean":"2024-04-17T10:28:03.398280+0000","last_became_active":"2024-04-17T10:28:01.389732+0000","last_became_peered":"2024-04-17T10:28:01.389732+0000","last_unstale":"2024-04-17T10:28:03.398280+0000","last_undegraded":"2024-04-17T10:28:03.398280+0000","last_fullsized":"2024-04-17T10:28:03.398280+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507298+0000","last_change":"2024-04-17T10:28:02.413779+0000","last_active":"2024-04-17T10:28:03.507298+0000","last_peered":"2024-04-17T10:28:03.507298+0000","last_clean":"2024-04-17T10:28:03.507298+0000","last_became_active":"2024-04-17T10:28:02.412034+0000","last_became_peered":"2024-04-17T10:28:02.412034+0000","last_unstale":"2024-04-17T10:28:03.507298+0000","last_undegraded":"2024-04-17T10:28:03.507298+0000","last_fullsized":"2024-04-17T10:28:03.507298+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:42:11.726306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398509+0000","last_change":"2024-04-17T10:28:01.380617+0000","last_active":"2024-04-17T10:28:03.398509+0000","last_peered":"2024-04-17T10:28:03.398509+0000","last_clean":"2024-04-17T10:28:03.398509+0000","last_became_active":"2024-04-17T10:28:01.380340+0000","last_became_peered":"2024-04-17T10:28:01.380340+0000","last_unstale":"2024-04-17T10:28:03.398509+0000","last_undegraded":"2024-04-17T10:28:03.398509+0000","last_fullsized":"2024-04-17T10:28:03.398509+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:08:40.336516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.358327+0000","last_change":"2024-04-17T10:28:02.403899+0000","last_active":"2024-04-17T10:28:04.358327+0000","last_peered":"2024-04-17T10:28:04.358327+0000","last_clean":"2024-04-17T10:28:04.358327+0000","last_became_active":"2024-04-17T10:28:02.403590+0000","last_became_peered":"2024-04-17T10:28:02.403590+0000","last_unstale":"2024-04-17T10:28:04.358327+0000","last_undegraded":"2024-04-17T10:28:04.358327+0000","last_fullsized":"2024-04-17T10:28:04.358327+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:46:04.469543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507344+0000","last_change":"2024-04-17T10:28:01.393021+0000","last_active":"2024-04-17T10:28:03.507344+0000","last_peered":"2024-04-17T10:28:03.507344+0000","last_clean":"2024-04-17T10:28:03.507344+0000","last_became_active":"2024-04-17T10:28:01.392144+0000","last_became_peered":"2024-04-17T10:28:01.392144+0000","last_unstale":"2024-04-17T10:28:03.507344+0000","last_undegraded":"2024-04-17T10:28:03.507344+0000","last_fullsized":"2024-04-17T10:28:03.507344+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:34:38.330069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398303+0000","last_change":"2024-04-17T10:28:02.392545+0000","last_active":"2024-04-17T10:28:03.398303+0000","last_peered":"2024-04-17T10:28:03.398303+0000","last_clean":"2024-04-17T10:28:03.398303+0000","last_became_active":"2024-04-17T10:28:02.392276+0000","last_became_peered":"2024-04-17T10:28:02.392276+0000","last_unstale":"2024-04-17T10:28:03.398303+0000","last_undegraded":"2024-04-17T10:28:03.398303+0000","last_fullsized":"2024-04-17T10:28:03.398303+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:16:09.311166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.400894+0000","last_change":"2024-04-17T10:28:01.384888+0000","last_active":"2024-04-17T10:28:03.400894+0000","last_peered":"2024-04-17T10:28:03.400894+0000","last_clean":"2024-04-17T10:28:03.400894+0000","last_became_active":"2024-04-17T10:28:01.384242+0000","last_became_peered":"2024-04-17T10:28:01.384242+0000","last_unstale":"2024-04-17T10:28:03.400894+0000","last_undegraded":"2024-04-17T10:28:03.400894+0000","last_fullsized":"2024-04-17T10:28:03.400894+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:21.863321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.404106+0000","last_change":"2024-04-17T10:28:02.393708+0000","last_active":"2024-04-17T10:28:03.404106+0000","last_peered":"2024-04-17T10:28:03.404106+0000","last_clean":"2024-04-17T10:28:03.404106+0000","last_became_active":"2024-04-17T10:28:02.393452+0000","last_became_peered":"2024-04-17T10:28:02.393452+0000","last_unstale":"2024-04-17T10:28:03.404106+0000","last_undegraded":"2024-04-17T10:28:03.404106+0000","last_fullsized":"2024-04-17T10:28:03.404106+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:38:10.671663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403420+0000","last_change":"2024-04-17T10:28:01.395507+0000","last_active":"2024-04-17T10:28:03.403420+0000","last_peered":"2024-04-17T10:28:03.403420+0000","last_clean":"2024-04-17T10:28:03.403420+0000","last_became_active":"2024-04-17T10:28:01.392288+0000","last_became_peered":"2024-04-17T10:28:01.392288+0000","last_unstale":"2024-04-17T10:28:03.403420+0000","last_undegraded":"2024-04-17T10:28:03.403420+0000","last_fullsized":"2024-04-17T10:28:03.403420+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:59:44.023271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507512+0000","last_change":"2024-04-17T10:28:02.406304+0000","last_active":"2024-04-17T10:28:03.507512+0000","last_peered":"2024-04-17T10:28:03.507512+0000","last_clean":"2024-04-17T10:28:03.507512+0000","last_became_active":"2024-04-17T10:28:02.405496+0000","last_became_peered":"2024-04-17T10:28:02.405496+0000","last_unstale":"2024-04-17T10:28:03.507512+0000","last_undegraded":"2024-04-17T10:28:03.507512+0000","last_fullsized":"2024-04-17T10:28:03.507512+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:13.848448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.361334+0000","last_change":"2024-04-17T10:28:01.391397+0000","last_active":"2024-04-17T10:28:04.361334+0000","last_peered":"2024-04-17T10:28:04.361334+0000","last_clean":"2024-04-17T10:28:04.361334+0000","last_became_active":"2024-04-17T10:28:01.391114+0000","last_became_peered":"2024-04-17T10:28:01.391114+0000","last_unstale":"2024-04-17T10:28:04.361334+0000","last_undegraded":"2024-04-17T10:28:04.361334+0000","last_fullsized":"2024-04-17T10:28:04.361334+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:34:37.136791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.358275+0000","last_change":"2024-04-17T10:28:02.403213+0000","last_active":"2024-04-17T10:28:04.358275+0000","last_peered":"2024-04-17T10:28:04.358275+0000","last_clean":"2024-04-17T10:28:04.358275+0000","last_became_active":"2024-04-17T10:28:02.402870+0000","last_became_peered":"2024-04-17T10:28:02.402870+0000","last_unstale":"2024-04-17T10:28:04.358275+0000","last_undegraded":"2024-04-17T10:28:04.358275+0000","last_fullsized":"2024-04-17T10:28:04.358275+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:42.908712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"53'2","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.398628+0000","last_change":"2024-04-17T10:28:01.386653+0000","last_active":"2024-04-17T10:28:03.398628+0000","last_peered":"2024-04-17T10:28:03.398628+0000","last_clean":"2024-04-17T10:28:03.398628+0000","last_became_active":"2024-04-17T10:28:01.386406+0000","last_became_peered":"2024-04-17T10:28:01.386406+0000","last_unstale":"2024-04-17T10:28:03.398628+0000","last_undegraded":"2024-04-17T10:28:03.398628+0000","last_fullsized":"2024-04-17T10:28:03.398628+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:00.646038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.507279+0000","last_change":"2024-04-17T10:28:02.408163+0000","last_active":"2024-04-17T10:28:03.507279+0000","last_peered":"2024-04-17T10:28:03.507279+0000","last_clean":"2024-04-17T10:28:03.507279+0000","last_became_active":"2024-04-17T10:28:02.407868+0000","last_became_peered":"2024-04-17T10:28:02.407868+0000","last_unstale":"2024-04-17T10:28:03.507279+0000","last_undegraded":"2024-04-17T10:28:03.507279+0000","last_fullsized":"2024-04-17T10:28:03.507279+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:13.080251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403697+0000","last_change":"2024-04-17T10:28:01.389788+0000","last_active":"2024-04-17T10:28:03.403697+0000","last_peered":"2024-04-17T10:28:03.403697+0000","last_clean":"2024-04-17T10:28:03.403697+0000","last_became_active":"2024-04-17T10:28:01.389488+0000","last_became_peered":"2024-04-17T10:28:01.389488+0000","last_unstale":"2024-04-17T10:28:03.403697+0000","last_undegraded":"2024-04-17T10:28:03.403697+0000","last_fullsized":"2024-04-17T10:28:03.403697+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:12.842514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401501+0000","last_change":"2024-04-17T10:28:02.403686+0000","last_active":"2024-04-17T10:28:03.401501+0000","last_peered":"2024-04-17T10:28:03.401501+0000","last_clean":"2024-04-17T10:28:03.401501+0000","last_became_active":"2024-04-17T10:28:02.403434+0000","last_became_peered":"2024-04-17T10:28:02.403434+0000","last_unstale":"2024-04-17T10:28:03.401501+0000","last_undegraded":"2024-04-17T10:28:03.401501+0000","last_fullsized":"2024-04-17T10:28:03.401501+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:47.626594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403638+0000","last_change":"2024-04-17T10:28:01.386384+0000","last_active":"2024-04-17T10:28:03.403638+0000","last_peered":"2024-04-17T10:28:03.403638+0000","last_clean":"2024-04-17T10:28:03.403638+0000","last_became_active":"2024-04-17T10:28:01.386102+0000","last_became_peered":"2024-04-17T10:28:01.386102+0000","last_unstale":"2024-04-17T10:28:03.403638+0000","last_undegraded":"2024-04-17T10:28:03.403638+0000","last_fullsized":"2024-04-17T10:28:03.403638+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:33:21.364427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403664+0000","last_change":"2024-04-17T10:28:02.399759+0000","last_active":"2024-04-17T10:28:03.403664+0000","last_peered":"2024-04-17T10:28:03.403664+0000","last_clean":"2024-04-17T10:28:03.403664+0000","last_became_active":"2024-04-17T10:28:02.399511+0000","last_became_peered":"2024-04-17T10:28:02.399511+0000","last_unstale":"2024-04-17T10:28:03.403664+0000","last_undegraded":"2024-04-17T10:28:03.403664+0000","last_fullsized":"2024-04-17T10:28:03.403664+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:01.079796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401207+0000","last_change":"2024-04-17T10:28:01.394707+0000","last_active":"2024-04-17T10:28:03.401207+0000","last_peered":"2024-04-17T10:28:03.401207+0000","last_clean":"2024-04-17T10:28:03.401207+0000","last_became_active":"2024-04-17T10:28:01.394419+0000","last_became_peered":"2024-04-17T10:28:01.394419+0000","last_unstale":"2024-04-17T10:28:03.401207+0000","last_undegraded":"2024-04-17T10:28:03.401207+0000","last_fullsized":"2024-04-17T10:28:03.401207+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:10:22.101234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.357931+0000","last_change":"2024-04-17T10:28:02.411603+0000","last_active":"2024-04-17T10:28:04.357931+0000","last_peered":"2024-04-17T10:28:04.357931+0000","last_clean":"2024-04-17T10:28:04.357931+0000","last_became_active":"2024-04-17T10:28:02.410973+0000","last_became_peered":"2024-04-17T10:28:02.410973+0000","last_unstale":"2024-04-17T10:28:04.357931+0000","last_undegraded":"2024-04-17T10:28:04.357931+0000","last_fullsized":"2024-04-17T10:28:04.357931+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:01.079796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.400971+0000","last_change":"2024-04-17T10:28:01.396212+0000","last_active":"2024-04-17T10:28:03.400971+0000","last_peered":"2024-04-17T10:28:03.400971+0000","last_clean":"2024-04-17T10:28:03.400971+0000","last_became_active":"2024-04-17T10:28:01.395939+0000","last_became_peered":"2024-04-17T10:28:01.395939+0000","last_unstale":"2024-04-17T10:28:03.400971+0000","last_undegraded":"2024-04-17T10:28:03.400971+0000","last_fullsized":"2024-04-17T10:28:03.400971+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:17:00.073863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403965+0000","last_change":"2024-04-17T10:28:02.407494+0000","last_active":"2024-04-17T10:28:03.403965+0000","last_peered":"2024-04-17T10:28:03.403965+0000","last_clean":"2024-04-17T10:28:03.403965+0000","last_became_active":"2024-04-17T10:28:02.403954+0000","last_became_peered":"2024-04-17T10:28:02.403954+0000","last_unstale":"2024-04-17T10:28:03.403965+0000","last_undegraded":"2024-04-17T10:28:03.403965+0000","last_fullsized":"2024-04-17T10:28:03.403965+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:25:52.444597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.856882+0000","last_change":"2024-04-17T10:28:01.384533+0000","last_active":"2024-04-17T10:28:03.856882+0000","last_peered":"2024-04-17T10:28:03.856882+0000","last_clean":"2024-04-17T10:28:03.856882+0000","last_became_active":"2024-04-17T10:28:01.384284+0000","last_became_peered":"2024-04-17T10:28:01.384284+0000","last_unstale":"2024-04-17T10:28:03.856882+0000","last_undegraded":"2024-04-17T10:28:03.856882+0000","last_fullsized":"2024-04-17T10:28:03.856882+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:42.740201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.401741+0000","last_change":"2024-04-17T10:28:02.411868+0000","last_active":"2024-04-17T10:28:03.401741+0000","last_peered":"2024-04-17T10:28:03.401741+0000","last_clean":"2024-04-17T10:28:03.401741+0000","last_became_active":"2024-04-17T10:28:02.407753+0000","last_became_peered":"2024-04-17T10:28:02.407753+0000","last_unstale":"2024-04-17T10:28:03.401741+0000","last_undegraded":"2024-04-17T10:28:03.401741+0000","last_fullsized":"2024-04-17T10:28:03.401741+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:44:28.532490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:04.358141+0000","last_change":"2024-04-17T10:28:01.388193+0000","last_active":"2024-04-17T10:28:04.358141+0000","last_peered":"2024-04-17T10:28:04.358141+0000","last_clean":"2024-04-17T10:28:04.358141+0000","last_became_active":"2024-04-17T10:28:01.387887+0000","last_became_peered":"2024-04-17T10:28:01.387887+0000","last_unstale":"2024-04-17T10:28:04.358141+0000","last_undegraded":"2024-04-17T10:28:04.358141+0000","last_fullsized":"2024-04-17T10:28:04.358141+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:35.280350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.397852+0000","last_change":"2024-04-17T10:28:02.406528+0000","last_active":"2024-04-17T10:28:03.397852+0000","last_peered":"2024-04-17T10:28:03.397852+0000","last_clean":"2024-04-17T10:28:03.397852+0000","last_became_active":"2024-04-17T10:28:02.406250+0000","last_became_peered":"2024-04-17T10:28:02.406250+0000","last_unstale":"2024-04-17T10:28:03.397852+0000","last_undegraded":"2024-04-17T10:28:03.397852+0000","last_fullsized":"2024-04-17T10:28:03.397852+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:01.363374+0000","last_clean_scrub_stamp":"2024-04-17T10:28:01.363374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:40:19.590441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"54'1","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.404240+0000","last_change":"2024-04-17T10:28:01.387103+0000","last_active":"2024-04-17T10:28:03.404240+0000","last_peered":"2024-04-17T10:28:03.404240+0000","last_clean":"2024-04-17T10:28:03.404240+0000","last_became_active":"2024-04-17T10:28:01.386847+0000","last_became_peered":"2024-04-17T10:28:01.386847+0000","last_unstale":"2024-04-17T10:28:03.404240+0000","last_undegraded":"2024-04-17T10:28:03.404240+0000","last_fullsized":"2024-04-17T10:28:03.404240+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:00.357440+0000","last_clean_scrub_stamp":"2024-04-17T10:28:00.357440+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:40:18.584507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":234,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-04-17T10:28:03.403137+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:03.403137+0000","last_peered":"2024-04-17T10:28:03.403137+0000","last_clean":"2024-04-17T10:28:03.403137+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:03.403137+0000","last_undegraded":"2024-04-17T10:28:03.403137+0000","last_fullsized":"2024-04-17T10:28:03.403137+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574893,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36560,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335280,"statfs":{"total":96636764160,"available":96599326720,"internally_reserved":0,"allocated":1245184,"data_stored":881132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35976,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335864,"statfs":{"total":96636764160,"available":96599924736,"internally_reserved":0,"allocated":651264,"data_stored":291292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574892,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35972,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335868,"statfs":{"total":96636764160,"available":96599928832,"internally_reserved":0,"allocated":643072,"data_stored":290640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40140,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331700,"statfs":{"total":96636764160,"available":96595660800,"internally_reserved":0,"allocated":647168,"data_stored":291222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574892,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35984,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335856,"statfs":{"total":96636764160,"available":96599916544,"internally_reserved":0,"allocated":659456,"data_stored":292274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574893,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35984,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335856,"statfs":{"total":96636764160,"available":96599916544,"internally_reserved":0,"allocated":655360,"data_stored":292252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35984,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335856,"statfs":{"total":96636764160,"available":96599916544,"internally_reserved":0,"allocated":651264,"data_stored":290800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574892,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58772,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313068,"statfs":{"total":96636764160,"available":96576581632,"internally_reserved":0,"allocated":1245184,"data_stored":881064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:28:11.419 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:28:11.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:28:11.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.610+0000 7fae62200640 1 Processor -- start 2024-04-17T10:28:11.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.610+0000 7fae62200640 1 -- start start 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae62200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae5c0b3130 0x7fae5c066c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae62200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae5c06eb80 0x7fae5c0b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae62200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae5c0b3a50 con 0x7fae5c15e550 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae62200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c06dae0 con 0x7fae5c06eb80 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae62200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae5c06ddc0 con 0x7fae5c0b3130 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae53e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae5c06eb80 0x7fae5c0b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae61800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fae5c15e550 0x7fae5c161490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45154/0 (socket says 172.21.15.67:45154) 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae61800640 1 -- 172.21.15.67:0/3610568135 learned_addr learned my addr 172.21.15.67:0/3610568135 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae60e00640 1 --2- 172.21.15.67:0/3610568135 >> v2:172.21.15.86:3300/0 conn(0x7fae5c0b3130 0x7fae5c066c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae53e00640 1 -- 172.21.15.67:0/3610568135 >> v2:172.21.15.86:3300/0 conn(0x7fae5c0b3130 msgr2=0x7fae5c066c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:11.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae53e00640 1 --2- 172.21.15.67:0/3610568135 >> v2:172.21.15.86:3300/0 conn(0x7fae5c0b3130 0x7fae5c066c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae53e00640 1 -- 172.21.15.67:0/3610568135 >> v1:172.21.15.67:6789/0 conn(0x7fae5c15e550 legacy=0x7fae5c161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae5be00640 1 -- 172.21.15.67:0/3610568135 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900505627 0 0) 0x7fae5c0b3a50 con 0x7fae5c15e550 2024-04-17T10:28:11.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae53e00640 1 -- 172.21.15.67:0/3610568135 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c06e0a0 con 0x7fae5c06eb80 2024-04-17T10:28:11.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.614+0000 7fae60e00640 1 --2- 172.21.15.67:0/3610568135 >> v2:172.21.15.86:3300/0 conn(0x7fae5c0b3130 0x7fae5c066c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:11.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae53e00640 1 --2- 172.21.15.67:0/3610568135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c0b3510 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fae5c06c830 tx=0x7fae4401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b51e0e68bd2a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:11.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae5be00640 1 -- 172.21.15.67:0/3610568135 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440b0070 con 0x7fae5c06eb80 2024-04-17T10:28:11.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae5be00640 1 -- 172.21.15.67:0/3610568135 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae440ab600 con 0x7fae5c06eb80 2024-04-17T10:28:11.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae5be00640 1 -- 172.21.15.67:0/3610568135 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae440ab9d0 con 0x7fae5c06eb80 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- 172.21.15.67:0/3610568135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 msgr2=0x7fae5c0b3510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 --2- 172.21.15.67:0/3610568135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c0b3510 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fae5c06c830 tx=0x7fae4401c3f0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- 172.21.15.67:0/3610568135 shutdown_connections 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 --2- 172.21.15.67:0/3610568135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c0b3510 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 --2- 172.21.15.67:0/3610568135 >> v2:172.21.15.86:3300/0 conn(0x7fae5c0b3130 0x7fae5c066c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- 172.21.15.67:0/3610568135 >> 172.21.15.67:0/3610568135 conn(0x7fae5c0cfdb0 msgr2=0x7fae5c0d01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- 172.21.15.67:0/3610568135 shutdown_connections 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- 172.21.15.67:0/3610568135 wait complete. 2024-04-17T10:28:11.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 Processor -- start 2024-04-17T10:28:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- start start 2024-04-17T10:28:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae5c0b3130 0x7fae5c14d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae5c15e550 0x7fae5c156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c16a810 con 0x7fae5c06eb80 2024-04-17T10:28:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c15ed70 con 0x7fae5c15e550 2024-04-17T10:28:11.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.618+0000 7fae62200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5c16a990 con 0x7fae5c0b3130 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60310/0 (socket says 172.21.15.67:60310) 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 -- 172.21.15.67:0/3524641513 learned_addr learned my addr 172.21.15.67:0/3524641513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae61800640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae5c15e550 0x7fae5c156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae53e00640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae5c0b3130 0x7fae5c14d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 -- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae5c0b3130 msgr2=0x7fae5c14d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae5c0b3130 0x7fae5c14d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 -- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae5c15e550 msgr2=0x7fae5c156c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae5c15e550 0x7fae5c156c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 -- 172.21.15.67:0/3524641513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c155e70 con 0x7fae5c06eb80 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae53e00640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae5c0b3130 0x7fae5c14d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae60e00640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c152a10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c820 tx=0x7fae4c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae59600640 1 -- 172.21.15.67:0/3524641513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c015070 con 0x7fae5c06eb80 2024-04-17T10:28:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae59600640 1 -- 172.21.15.67:0/3524641513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae4c007a30 con 0x7fae5c06eb80 2024-04-17T10:28:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae59600640 1 -- 172.21.15.67:0/3524641513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c007e40 con 0x7fae5c06eb80 2024-04-17T10:28:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae5c156150 con 0x7fae5c06eb80 2024-04-17T10:28:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5c156710 con 0x7fae5c06eb80 2024-04-17T10:28:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae5c074dc0 con 0x7fae5c06eb80 2024-04-17T10:28:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.622+0000 7fae59600640 1 -- 172.21.15.67:0/3524641513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fae4c018070 con 0x7fae5c06eb80 2024-04-17T10:28:11.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.626+0000 7fae59600640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae38066f60 0x7fae38069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.626+0000 7fae59600640 1 -- 172.21.15.67:0/3524641513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fae4c084ff0 con 0x7fae5c06eb80 2024-04-17T10:28:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.626+0000 7fae53e00640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae38066f60 0x7fae38069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.626+0000 7fae53e00640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae38066f60 0x7fae38069420 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fae440ad000 tx=0x7fae44008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:11.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.630+0000 7fae59600640 1 -- 172.21.15.67:0/3524641513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fae4c055870 con 0x7fae5c06eb80 2024-04-17T10:28:11.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.846+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fae5c0cc260 con 0x7fae5c06eb80 2024-04-17T10:28:11.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae59600640 1 -- 172.21.15.67:0/3524641513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7fae4c059910 con 0x7fae5c06eb80 2024-04-17T10:28:11.842 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:28:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:28:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:28:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:28:11.843 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:28:11.843 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:28:11.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae38066f60 msgr2=0x7fae38069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae38066f60 0x7fae38069420 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fae440ad000 tx=0x7fae44008040 comp rx=0 tx=0).stop 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 msgr2=0x7fae5c152a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c152a10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fae4c00c820 tx=0x7fae4c00cce0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 shutdown_connections 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae38066f60 0x7fae38069420 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae5c15e550 0x7fae5c156c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae5c0b3130 0x7fae5c14d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 --2- 172.21.15.67:0/3524641513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae5c06eb80 0x7fae5c152a10 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 >> 172.21.15.67:0/3524641513 conn(0x7fae5c0cfdb0 msgr2=0x7fae5c0d0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 shutdown_connections 2024-04-17T10:28:11.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:11.850+0000 7fae62200640 1 -- 172.21.15.67:0/3524641513 wait complete. 2024-04-17T10:28:11.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.010+0000 7f64adc00640 1 Processor -- start 2024-04-17T10:28:12.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.010+0000 7f64adc00640 1 -- start start 2024-04-17T10:28:12.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64adc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a80b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64adc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a815a0f0 con 0x7f64a806f400 2024-04-17T10:28:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64adc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64a815a3b0 con 0x7f64a80b3bb0 2024-04-17T10:28:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64adc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a80b4cf0 con 0x7f64a8071a20 2024-04-17T10:28:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f649e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f64a806f400 0x7f64a8071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45164/0 (socket says 172.21.15.67:45164) 2024-04-17T10:28:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f649e000640 1 -- 172.21.15.67:0/1213071811 learned_addr learned my addr 172.21.15.67:0/1213071811 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:12.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a7400640 1 --2- 172.21.15.67:0/1213071811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a80b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a6000640 1 -- 172.21.15.67:0/1213071811 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387677750 0 0) 0x7f64a815a0f0 con 0x7f64a806f400 2024-04-17T10:28:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a6000640 1 -- 172.21.15.67:0/1213071811 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f648c003530 con 0x7f64a806f400 2024-04-17T10:28:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a6000640 1 -- 172.21.15.67:0/1213071811 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408262483 0 0) 0x7f64a815a3b0 con 0x7f64a80b3bb0 2024-04-17T10:28:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a6000640 1 -- 172.21.15.67:0/1213071811 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a815a0f0 con 0x7f64a80b3bb0 2024-04-17T10:28:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a7400640 1 -- 172.21.15.67:0/1213071811 >> v1:172.21.15.86:6789/0 conn(0x7f64a80b3bb0 legacy=0x7f64a815e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a7400640 1 -- 172.21.15.67:0/1213071811 >> v1:172.21.15.67:6789/0 conn(0x7f64a806f400 legacy=0x7f64a8071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.014+0000 7f64a7400640 1 -- 172.21.15.67:0/1213071811 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a815a690 con 0x7f64a8071a20 2024-04-17T10:28:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64a7400640 1 --2- 172.21.15.67:0/1213071811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a80b47b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009dc0 tx=0x7f64a001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef399a7fa0bb38d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64a6000640 1 -- 172.21.15.67:0/1213071811 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a00b2070 con 0x7f64a8071a20 2024-04-17T10:28:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64a6000640 1 -- 172.21.15.67:0/1213071811 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a00a5940 con 0x7f64a8071a20 2024-04-17T10:28:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64a6000640 1 -- 172.21.15.67:0/1213071811 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a00a5cf0 con 0x7f64a8071a20 2024-04-17T10:28:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 -- 172.21.15.67:0/1213071811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 msgr2=0x7f64a80b47b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 --2- 172.21.15.67:0/1213071811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a80b47b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009dc0 tx=0x7f64a001c990 comp rx=0 tx=0).stop 2024-04-17T10:28:12.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 -- 172.21.15.67:0/1213071811 shutdown_connections 2024-04-17T10:28:12.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 --2- 172.21.15.67:0/1213071811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a80b47b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 -- 172.21.15.67:0/1213071811 >> 172.21.15.67:0/1213071811 conn(0x7f64a806ac90 msgr2=0x7f64a806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 -- 172.21.15.67:0/1213071811 shutdown_connections 2024-04-17T10:28:12.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 -- 172.21.15.67:0/1213071811 wait complete. 2024-04-17T10:28:12.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 Processor -- start 2024-04-17T10:28:12.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.018+0000 7f64adc00640 1 -- start start 2024-04-17T10:28:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64a806f400 0x7f64a814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a8156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a80b3bb0 0x7f64a81f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a81543a0 con 0x7f64a80b3bb0 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8154220 con 0x7f64a806f400 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a8154520 con 0x7f64a8071a20 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a80b3bb0 0x7f64a81f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a80b3bb0 0x7f64a81f69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60312/0 (socket says 172.21.15.67:60312) 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 -- 172.21.15.67:0/2758891374 learned_addr learned my addr 172.21.15.67:0/2758891374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f649e000640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a8156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a6a00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64a806f400 0x7f64a814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 -- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 msgr2=0x7f64a8156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a8156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 -- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64a806f400 msgr2=0x7f64a814a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64a806f400 0x7f64a814a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 -- 172.21.15.67:0/2758891374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a81569a0 con 0x7f64a80b3bb0 2024-04-17T10:28:12.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f649e000640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a8156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a6a00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64a806f400 0x7f64a814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64a7400640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a80b3bb0 0x7f64a81f69b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f64a00b0040 tx=0x7f64a0019730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f649f400640 1 -- 172.21.15.67:0/2758891374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a00b2050 con 0x7f64a80b3bb0 2024-04-17T10:28:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f649f400640 1 -- 172.21.15.67:0/2758891374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64a00be070 con 0x7f64a80b3bb0 2024-04-17T10:28:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a80c5290 con 0x7f64a80b3bb0 2024-04-17T10:28:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f649f400640 1 -- 172.21.15.67:0/2758891374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a00ad850 con 0x7f64a80b3bb0 2024-04-17T10:28:12.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a80c5850 con 0x7f64a80b3bb0 2024-04-17T10:28:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.022+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64a8072d50 con 0x7f64a80b3bb0 2024-04-17T10:28:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.026+0000 7f649f400640 1 -- 172.21.15.67:0/2758891374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f64a00ad9f0 con 0x7f64a80b3bb0 2024-04-17T10:28:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.026+0000 7f649f400640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6484066f60 0x7f6484069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.026+0000 7f649f400640 1 -- 172.21.15.67:0/2758891374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f64a0120db0 con 0x7f64a80b3bb0 2024-04-17T10:28:12.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.026+0000 7f64a6a00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6484066f60 0x7f6484069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.026+0000 7f64a6a00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6484066f60 0x7f6484069420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f64a80b23e0 tx=0x7f6498009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.030+0000 7f649f400640 1 -- 172.21.15.67:0/2758891374 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f64a0016580 con 0x7f64a80b3bb0 2024-04-17T10:28:12.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.222+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64a8074e30 con 0x7f64a80b3bb0 2024-04-17T10:28:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.222+0000 7f649f400640 1 -- 172.21.15.67:0/2758891374 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f64a00f1d10 con 0x7f64a80b3bb0 2024-04-17T10:28:12.217 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:12.217 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:12.217 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6484066f60 msgr2=0x7f6484069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6484066f60 0x7f6484069420 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f64a80b23e0 tx=0x7f6498009040 comp rx=0 tx=0).stop 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a80b3bb0 msgr2=0x7f64a81f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a80b3bb0 0x7f64a81f69b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f64a00b0040 tx=0x7f64a0019730 comp rx=0 tx=0).stop 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 shutdown_connections 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6484066f60 0x7f6484069420 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64a80b3bb0 0x7f64a81f69b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64a8071a20 0x7f64a8156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 --2- 172.21.15.67:0/2758891374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64a806f400 0x7f64a814a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 >> 172.21.15.67:0/2758891374 conn(0x7f64a806ac90 msgr2=0x7f64a806dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 shutdown_connections 2024-04-17T10:28:12.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.226+0000 7f64adc00640 1 -- 172.21.15.67:0/2758891374 wait complete. 2024-04-17T10:28:12.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:28:01.776873+0000', 'modified': '2024-04-17T10:28:04.796960+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5772}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5772': {'gid': 5772, 'name': 'd', 'rank': 0, 'incarnation': 24, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6837/2048306700', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6835', 'nonce': 2048306700}, {'type': 'v1', 'addr': '172.21.15.86:6837', 'nonce': 2048306700}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:28:12.641 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:28:12.641 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-17T10:28:12.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.790+0000 7f8684000640 1 Processor -- start 2024-04-17T10:28:12.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.790+0000 7f8684000640 1 -- start start 2024-04-17T10:28:12.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8684000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8684000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f867c0cf0c0 0x7f867c144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8684000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c144560 con 0x7f867c06e6e0 2024-04-17T10:28:12.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8684000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867c15ae40 con 0x7f867c0cf0c0 2024-04-17T10:28:12.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8684000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c071440 con 0x7f867c06eac0 2024-04-17T10:28:12.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8681600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f867c06e6e0 0x7f867c0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45172/0 (socket says 172.21.15.67:45172) 2024-04-17T10:28:12.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8681600640 1 -- 172.21.15.67:0/205540313 learned_addr learned my addr 172.21.15.67:0/205540313 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:12.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8682000640 1 --2- 172.21.15.67:0/205540313 >> v2:172.21.15.67:3300/0 conn(0x7f867c0cf0c0 0x7f867c144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8680c00640 1 --2- 172.21.15.67:0/205540313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8682000640 1 -- 172.21.15.67:0/205540313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 msgr2=0x7f867c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8682000640 1 --2- 172.21.15.67:0/205540313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8682000640 1 -- 172.21.15.67:0/205540313 >> v1:172.21.15.67:6789/0 conn(0x7f867c06e6e0 legacy=0x7f867c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.794+0000 7f8682000640 1 -- 172.21.15.67:0/205540313 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c071720 con 0x7f867c0cf0c0 2024-04-17T10:28:12.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8682000640 1 --2- 172.21.15.67:0/205540313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c144020 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8674009d90 tx=0x7f867401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9489a4e35f2c782b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f867be00640 1 -- 172.21.15.67:0/205540313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740b2070 con 0x7f867c0cf0c0 2024-04-17T10:28:12.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f867be00640 1 -- 172.21.15.67:0/205540313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86740a5970 con 0x7f867c0cf0c0 2024-04-17T10:28:12.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f867be00640 1 -- 172.21.15.67:0/205540313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740a5d40 con 0x7f867c0cf0c0 2024-04-17T10:28:12.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 -- 172.21.15.67:0/205540313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 msgr2=0x7f867c144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 --2- 172.21.15.67:0/205540313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c144020 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8674009d90 tx=0x7f867401c8b0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 -- 172.21.15.67:0/205540313 shutdown_connections 2024-04-17T10:28:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 --2- 172.21.15.67:0/205540313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c144020 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 --2- 172.21.15.67:0/205540313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 -- 172.21.15.67:0/205540313 >> 172.21.15.67:0/205540313 conn(0x7f867c06ac90 msgr2=0x7f867c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 -- 172.21.15.67:0/205540313 shutdown_connections 2024-04-17T10:28:12.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 -- 172.21.15.67:0/205540313 wait complete. 2024-04-17T10:28:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 Processor -- start 2024-04-17T10:28:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.798+0000 7f8684000640 1 -- start start 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867c06e6e0 0x7f867c159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c142110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b3a40 con 0x7f867c0cf0c0 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b38c0 con 0x7f867c06e6e0 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b3bc0 con 0x7f867c06eac0 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c142110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c142110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60324/0 (socket says 172.21.15.67:60324) 2024-04-17T10:28:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 -- 172.21.15.67:0/3582664363 learned_addr learned my addr 172.21.15.67:0/3582664363 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8680c00640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8681600640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867c06e6e0 0x7f867c159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 -- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 msgr2=0x7f867c0c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c0c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 -- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867c06e6e0 msgr2=0x7f867c159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867c06e6e0 0x7f867c159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 -- 172.21.15.67:0/3582664363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c13d680 con 0x7f867c0cf0c0 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8681600640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867c06e6e0 0x7f867c159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8682000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c142110 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f867401af60 tx=0x7f8674019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8679600640 1 -- 172.21.15.67:0/3582664363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740b2050 con 0x7f867c0cf0c0 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8679600640 1 -- 172.21.15.67:0/3582664363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86740be070 con 0x7f867c0cf0c0 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8679600640 1 -- 172.21.15.67:0/3582664363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740ada50 con 0x7f867c0cf0c0 2024-04-17T10:28:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c0abda0 con 0x7f867c0cf0c0 2024-04-17T10:28:12.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c0ac360 con 0x7f867c0cf0c0 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.802+0000 7f8679600640 1 -- 172.21.15.67:0/3582664363 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f86740adbf0 con 0x7f867c0cf0c0 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8679600640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f865c066f90 0x7f865c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8679600640 1 -- 172.21.15.67:0/3582664363 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f867411fdb0 con 0x7f867c0cf0c0 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 Processor -- start 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8681600640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f865c066f90 0x7f865c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- start start 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8681600640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f865c066f90 0x7f865c069450 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8670000f70 tx=0x7f8670009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da638d920 0x557da638dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63439f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da6343f30 0x557da6344310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da622b5f0 con 0x557da6343f30 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da6192da0 con 0x557da638d920 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x557da611b150 con 0x557da638e240 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da6343f30 0x557da6344310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da6343f30 0x557da6344310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60328/0 (socket says 172.21.15.67:60328) 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 -- 172.21.15.67:0/2079020319 learned_addr learned my addr 172.21.15.67:0/2079020319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8681600640 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da638d920 0x557da638dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8680c00640 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 -- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 msgr2=0x557da63439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63439f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 -- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da638d920 msgr2=0x557da638dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da638d920 0x557da638dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 -- 172.21.15.67:0/2079020319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557da6134ca0 con 0x557da6343f30 2024-04-17T10:28:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f8682000640 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da6343f30 0x557da6344310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f86740ec8f0 tx=0x7f86740f0390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8773fb0080fa2621 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f866e000640 1 -- 172.21.15.67:0/2079020319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740f45e0 con 0x557da6343f30 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f866e000640 1 -- 172.21.15.67:0/2079020319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86740b3040 con 0x557da6343f30 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f866e000640 1 -- 172.21.15.67:0/2079020319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740f8ba0 con 0x557da6343f30 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- 172.21.15.67:0/2079020319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da6343f30 msgr2=0x557da6344310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da6343f30 0x557da6344310 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f86740ec8f0 tx=0x7f86740f0390 comp rx=0 tx=0).stop 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- 172.21.15.67:0/2079020319 shutdown_connections 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da6343f30 0x557da6344310 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 --2- 172.21.15.67:0/2079020319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da638d920 0x557da638dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- 172.21.15.67:0/2079020319 >> 172.21.15.67:0/2079020319 conn(0x557da634a2a0 msgr2=0x557da63422d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- 172.21.15.67:0/2079020319 shutdown_connections 2024-04-17T10:28:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.806+0000 7f86874e8000 1 -- 172.21.15.67:0/2079020319 wait complete. 2024-04-17T10:28:12.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 -- start start 2024-04-17T10:28:12.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 Processor -- start 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da6394c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63951c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da6347aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da614d990 con 0x557da638d920 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da618ee40 con 0x557da6343f30 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x557da611e900 con 0x557da638e240 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 10 client.0 _ll_register_callbacks cb 0x557da62461c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60332/0 (socket says 172.21.15.67:60332) 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 -- 172.21.15.67:0/975781853 learned_addr learned my addr 172.21.15.67:0/975781853 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8681600640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da6394c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8682000640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da6347aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 -- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 msgr2=0x557da6347aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da6347aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 -- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 msgr2=0x557da6394c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da6394c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 -- 172.21.15.67:0/975781853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557da614b630 con 0x557da638d920 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8682000640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da6347aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8680c00640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63951c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8668012670 tx=0x7f8668012b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42af99ffa2918d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8662a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8662a00640 1 -- 172.21.15.67:0/975781853 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866801b070 con 0x557da638d920 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8662a00640 1 -- 172.21.15.67:0/975781853 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8668010830 con 0x557da638d920 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86740b3040 con 0x557da638d920 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8662a00640 1 -- 172.21.15.67:0/975781853 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8668026050 con 0x557da638d920 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 10 client.5943 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x557da611b3a0 con 0x557da638d920 2024-04-17T10:28:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8662a00640 1 -- 172.21.15.67:0/975781853 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86740b3040 con 0x557da638d920 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f8662a00640 1 -- 172.21.15.67:0/975781853 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8668029020 con 0x557da638d920 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 10 client.5943 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 10 client.5943 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 20 client.5943 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da634eaf0 0x557da63484b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 4 client.5943 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.810+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557da612ea50 con 0x557da634eaf0 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f8681600640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da634eaf0 0x557da63484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f8681600640 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da634eaf0 0x557da63484b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=866a3607345b8871 server_cookie=cfb319912fddfc73 in_seq=0 out_seq=0 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f8662a00640 10 client.5943 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f8662a00640 1 -- 172.21.15.67:0/975781853 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557da612ea50 con 0x557da634eaf0 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f8662a00640 10 client.5943 handle_command_reply: tid=0 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 client.5943 shutdown 2024-04-17T10:28:12.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 msgr2=0x557da63951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63951c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8668012670 tx=0x7f8668012b30 comp rx=0 tx=0).stop 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 shutdown_connections 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da634eaf0 0x557da63484b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da6347aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63951c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 --2- 172.21.15.67:0/975781853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da6394c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f8662a00640 0 client.5943 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 >> 172.21.15.67:0/975781853 conn(0x557da634a2a0 msgr2=0x557da63429e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 shutdown_connections 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 1 -- 172.21.15.67:0/975781853 wait complete. 2024-04-17T10:28:12.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.814+0000 7f86874e8000 20 client.5943 trim_cache size 0 max 16384 2024-04-17T10:28:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 Processor -- start 2024-04-17T10:28:12.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 -- start start 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63e9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638d920 0x557da63e9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638e240 0x557da63ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c2024e0 con 0x557da638e240 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0b1bf0 con 0x557da6343f30 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c201fa0 con 0x557da638d920 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638e240 0x557da63ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638e240 0x557da63ea1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60344/0 (socket says 172.21.15.67:60344) 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 -- 172.21.15.67:0/2465802676 learned_addr learned my addr 172.21.15.67:0/2465802676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8680c00640 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638d920 0x557da63e9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8681600640 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63e9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 -- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638d920 msgr2=0x557da63e9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638d920 0x557da63e9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 -- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 msgr2=0x557da63e9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63e9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 -- 172.21.15.67:0/2465802676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557da638e070 con 0x557da638e240 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8680c00640 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638d920 0x557da63e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8681600640 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63e9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f8682000640 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638e240 0x557da63ea1b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f867401cd90 tx=0x7f8674105000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461e22fbb09e1db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f866d600640 1 -- 172.21.15.67:0/2465802676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740f45e0 con 0x557da638e240 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f866d600640 1 -- 172.21.15.67:0/2465802676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867411d070 con 0x557da638e240 2024-04-17T10:28:12.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.830+0000 7f866d600640 1 -- 172.21.15.67:0/2465802676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740a5870 con 0x557da638e240 2024-04-17T10:28:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- 172.21.15.67:0/2465802676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638e240 msgr2=0x557da63ea1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638e240 0x557da63ea1b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f867401cd90 tx=0x7f8674105000 comp rx=0 tx=0).stop 2024-04-17T10:28:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- 172.21.15.67:0/2465802676 shutdown_connections 2024-04-17T10:28:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638e240 0x557da63ea1b0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638d920 0x557da63e9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 --2- 172.21.15.67:0/2465802676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63e9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- 172.21.15.67:0/2465802676 >> 172.21.15.67:0/2465802676 conn(0x557da634a2a0 msgr2=0x557da6379900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- 172.21.15.67:0/2465802676 shutdown_connections 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- 172.21.15.67:0/2465802676 wait complete. 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- start start 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 Processor -- start 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63ee730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63eec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63a9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da61973d0 con 0x557da638d920 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557da624ed50 con 0x557da6343f30 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x557da612d980 con 0x557da638e240 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 10 client.0 _ll_register_callbacks cb 0x7f8674009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:12.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63eec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60358/0 (socket says 172.21.15.67:60358) 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 -- 172.21.15.67:0/384652706 learned_addr learned my addr 172.21.15.67:0/384652706 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8682000640 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8681600640 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63ee730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 -- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 msgr2=0x557da63a9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63a9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 -- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 msgr2=0x557da63ee730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63ee730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 -- 172.21.15.67:0/384652706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557da6214600 con 0x557da638d920 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8680c00640 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63eec70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8668012b70 tx=0x7f86680046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f8940b97a07d4a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8662a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8662a00640 1 -- 172.21.15.67:0/384652706 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f866801b040 con 0x557da638d920 2024-04-17T10:28:12.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8662a00640 1 -- 172.21.15.67:0/384652706 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8668010dd0 con 0x557da638d920 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f867411d070 con 0x557da638d920 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8662a00640 1 -- 172.21.15.67:0/384652706 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f866803b050 con 0x557da638d920 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 10 client.5949 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x557da61279b0 con 0x557da638d920 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8662a00640 1 -- 172.21.15.67:0/384652706 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867411d070 con 0x557da638d920 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f8662a00640 1 -- 172.21.15.67:0/384652706 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f866803e020 con 0x557da638d920 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 10 client.5949 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 10 client.5949 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.834+0000 7f86874e8000 20 client.5949 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da63a9870 0x557da63a9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 4 client.5949 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:28:12.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557da61a02e0 con 0x557da63a9870 2024-04-17T10:28:12.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8681600640 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da63a9870 0x557da63a9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8681600640 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da63a9870 0x557da63a9c50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=368e6caaa6c9b2f server_cookie=ec9b0a123afbe409 in_seq=0 out_seq=0 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8662a00640 10 client.5949 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8662a00640 1 -- 172.21.15.67:0/384652706 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557da61a02e0 con 0x557da63a9870 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8662a00640 10 client.5949 handle_command_reply: tid=0 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 client.5949 shutdown 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 msgr2=0x557da63eec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63eec70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8668012b70 tx=0x7f86680046d0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 shutdown_connections 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x557da63a9870 0x557da63a9c50 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557da638e240 0x557da63a9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557da638d920 0x557da63eec70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 --2- 172.21.15.67:0/384652706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557da6343f30 0x557da63ee730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8662a00640 0 client.5949 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 >> 172.21.15.67:0/384652706 conn(0x557da634a2a0 msgr2=0x557da637a010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 shutdown_connections 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 1 -- 172.21.15.67:0/384652706 wait complete. 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f86874e8000 20 client.5949 trim_cache size 0 max 16384 2024-04-17T10:28:12.836 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 5772, 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.447447447, 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 28, 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 54, 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 54, 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 15.346517774000001 2024-04-17T10:28:12.837 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:12.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f865c066f90 msgr2=0x7f865c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f865c066f90 0x7f865c069450 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8670000f70 tx=0x7f8670009040 comp rx=0 tx=0).stop 2024-04-17T10:28:12.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 msgr2=0x7f867c142110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:12.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c142110 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f867401af60 tx=0x7f8674019650 comp rx=0 tx=0).stop 2024-04-17T10:28:12.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 shutdown_connections 2024-04-17T10:28:12.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f865c066f90 0x7f865c069450 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f867c0cf0c0 0x7f867c142110 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f867c06eac0 0x7f867c0c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 --2- 172.21.15.67:0/3582664363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f867c06e6e0 0x7f867c159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 >> 172.21.15.67:0/3582664363 conn(0x7f867c06ac90 msgr2=0x7f867c06b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:12.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 shutdown_connections 2024-04-17T10:28:12.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.838+0000 7f8684000640 1 -- 172.21.15.67:0/3582664363 wait complete. 2024-04-17T10:28:12.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:28:12.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.998+0000 7f9a8fc00640 1 Processor -- start 2024-04-17T10:28:12.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:12.998+0000 7f9a8fc00640 1 -- start start 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8fc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a880ce020 0x7f9a881620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8fc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a880cf510 0x7f9a880cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8fc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8815eb50 con 0x7f9a880ce400 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8fc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a88161650 con 0x7f9a880ce020 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8fc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a881625e0 con 0x7f9a880cf510 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a880ce020 0x7f9a881620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a880ce020 0x7f9a881620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60366/0 (socket says 172.21.15.67:60366) 2024-04-17T10:28:12.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8d200640 1 -- 172.21.15.67:0/3993775288 learned_addr learned my addr 172.21.15.67:0/3993775288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8dc00640 1 --2- 172.21.15.67:0/3993775288 >> v2:172.21.15.86:3300/0 conn(0x7f9a880cf510 0x7f9a880cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8d200640 1 -- 172.21.15.67:0/3993775288 >> v2:172.21.15.86:3300/0 conn(0x7f9a880cf510 msgr2=0x7f9a880cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8d200640 1 --2- 172.21.15.67:0/3993775288 >> v2:172.21.15.86:3300/0 conn(0x7f9a880cf510 0x7f9a880cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8d200640 1 -- 172.21.15.67:0/3993775288 >> v1:172.21.15.86:6789/0 conn(0x7f9a880ce400 legacy=0x7f9a8815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8d200640 1 -- 172.21.15.67:0/3993775288 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8815ee30 con 0x7f9a880ce020 2024-04-17T10:28:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.002+0000 7f9a8dc00640 1 --2- 172.21.15.67:0/3993775288 >> v2:172.21.15.86:3300/0 conn(0x7f9a880cf510 0x7f9a880cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:12.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8d200640 1 --2- 172.21.15.67:0/3993775288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce020 0x7f9a881620a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009870 tx=0x7f9a7801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2989e69b968312 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:12.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a87e00640 1 -- 172.21.15.67:0/3993775288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a780b1070 con 0x7f9a880ce020 2024-04-17T10:28:12.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a87e00640 1 -- 172.21.15.67:0/3993775288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a780ac620 con 0x7f9a880ce020 2024-04-17T10:28:12.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a87e00640 1 -- 172.21.15.67:0/3993775288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a780ac9f0 con 0x7f9a880ce020 2024-04-17T10:28:12.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- 172.21.15.67:0/3993775288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce020 msgr2=0x7f9a881620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:13.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/3993775288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce020 0x7f9a881620a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009870 tx=0x7f9a7801c4b0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- 172.21.15.67:0/3993775288 shutdown_connections 2024-04-17T10:28:13.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/3993775288 >> v2:172.21.15.86:3300/0 conn(0x7f9a880cf510 0x7f9a880cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/3993775288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce020 0x7f9a881620a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- 172.21.15.67:0/3993775288 >> 172.21.15.67:0/3993775288 conn(0x7f9a880b33f0 msgr2=0x7f9a880b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:13.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- 172.21.15.67:0/3993775288 shutdown_connections 2024-04-17T10:28:13.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- 172.21.15.67:0/3993775288 wait complete. 2024-04-17T10:28:13.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 Processor -- start 2024-04-17T10:28:13.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- start start 2024-04-17T10:28:13.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a880ce020 0x7f9a881587c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:13.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce400 0x7f9a8815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:13.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a880cf510 0x7f9a8815aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:13.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8806c0f0 con 0x7f9a880ce400 2024-04-17T10:28:13.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8806bf70 con 0x7f9a880ce020 2024-04-17T10:28:13.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.006+0000 7f9a8fc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8815e720 con 0x7f9a880cf510 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce400 0x7f9a8815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce400 0x7f9a8815a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60368/0 (socket says 172.21.15.67:60368) 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 -- 172.21.15.67:0/2521395908 learned_addr learned my addr 172.21.15.67:0/2521395908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a8d200640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a880ce020 0x7f9a881587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a8dc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a880cf510 0x7f9a8815aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 -- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a880cf510 msgr2=0x7f9a8815aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a880cf510 0x7f9a8815aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 -- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a880ce020 msgr2=0x7f9a881587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a880ce020 0x7f9a881587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 -- 172.21.15.67:0/2521395908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a880ca170 con 0x7f9a880ce400 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a8dc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a880cf510 0x7f9a8815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a7fe00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce400 0x7f9a8815a590 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000c480 tx=0x7f9a7000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a85600640 1 -- 172.21.15.67:0/2521395908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70012830 con 0x7f9a880ce400 2024-04-17T10:28:13.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a85600640 1 -- 172.21.15.67:0/2521395908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a70017070 con 0x7f9a880ce400 2024-04-17T10:28:13.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a85600640 1 -- 172.21.15.67:0/2521395908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70012c30 con 0x7f9a880ce400 2024-04-17T10:28:13.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a880ca450 con 0x7f9a880ce400 2024-04-17T10:28:13.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a880b4cd0 con 0x7f9a880ce400 2024-04-17T10:28:13.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a880ca970 con 0x7f9a880ce400 2024-04-17T10:28:13.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a85600640 1 -- 172.21.15.67:0/2521395908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9a7001c460 con 0x7f9a880ce400 2024-04-17T10:28:13.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.010+0000 7f9a85600640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a64066f60 0x7f9a64069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:13.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.014+0000 7f9a85600640 1 -- 172.21.15.67:0/2521395908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9a70084fa0 con 0x7f9a880ce400 2024-04-17T10:28:13.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.014+0000 7f9a8d200640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a64066f60 0x7f9a64069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:13.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.014+0000 7f9a8d200640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a64066f60 0x7f9a64069420 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9a780ae000 tx=0x7f9a78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:13.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.014+0000 7f9a85600640 1 -- 172.21.15.67:0/2521395908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9a880ca970 con 0x7f9a880ce400 2024-04-17T10:28:13.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.210+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9a88070ab0 con 0x7f9a880ce400 2024-04-17T10:28:13.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.210+0000 7f9a85600640 1 -- 172.21.15.67:0/2521395908 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9a70055e10 con 0x7f9a880ce400 2024-04-17T10:28:13.206 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a64066f60 msgr2=0x7f9a64069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a64066f60 0x7f9a64069420 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f9a780ae000 tx=0x7f9a78008040 comp rx=0 tx=0).stop 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce400 msgr2=0x7f9a8815a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce400 0x7f9a8815a590 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000c480 tx=0x7f9a7000c940 comp rx=0 tx=0).stop 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 shutdown_connections 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a64066f60 0x7f9a64069420 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a880cf510 0x7f9a8815aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a880ce400 0x7f9a8815a590 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 --2- 172.21.15.67:0/2521395908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a880ce020 0x7f9a881587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 >> 172.21.15.67:0/2521395908 conn(0x7f9a880b33f0 msgr2=0x7f9a880cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:13.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 shutdown_connections 2024-04-17T10:28:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:13.214+0000 7f9a8fc00640 1 -- 172.21.15.67:0/2521395908 wait complete. 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:28:13.227 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:28:13.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.228 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:28:13.237 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42253sec preferred_lft 42253sec 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.238 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:28:13.239 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:28:13.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.241 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:28:13.299 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:28:13.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.302 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:28:13.361 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:28:13.361 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:28:13.361 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:28:13.361 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:28:13.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.362 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:28:13.363 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:28:13.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:28:13.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:28:13.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:28:13.363 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:28:13.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.452 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:28:13.461 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:28:13.461 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:28:13.461 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:28:13.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:28:13.526 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:28:13.526 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:28:13.582 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:28:13.582 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:28:13.582 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:28:13.633 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:28:13.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:28:13.723 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:28:13.723 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:28:13.736 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:28:13.736 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:28:13.736 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:28:13.736 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:28:13.736 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:28:13.740 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:28:13.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:28:13.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.761 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:28:13.829 INFO:teuthology.orchestra.run.smithi067.stdout:client5958 2024-04-17T10:28:13.829 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:13.829 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:13.882 INFO:tasks.cephfs.test_scrub_checks:mountpoint: /home/ubuntu/cephtest/mnt.0 2024-04-17T10:28:13.882 INFO:tasks.cephfs.test_scrub_checks:client_path: /home/ubuntu/cephtest/mnt.0/scrub_test_path 2024-04-17T10:28:13.883 INFO:tasks.cephfs.test_scrub_checks:Cloning repo into place 2024-04-17T10:28:13.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:13.883 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p /home/ubuntu/cephtest/mnt.0/scrub_test_path) 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> import json 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> import sys 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> try: 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite'"'"') 2024-04-17T10:28:13.948 DEBUG:teuthology.orchestra.run.smithi067:> except OSError as e: 2024-04-17T10:28:13.949 DEBUG:teuthology.orchestra.run.smithi067:> sys.exit(e.errno) 2024-04-17T10:28:13.949 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:28:13.949 DEBUG:teuthology.orchestra.run.smithi067:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-17T10:28:13.949 DEBUG:teuthology.orchestra.run.smithi067:> print(json.dumps( 2024-04-17T10:28:13.949 DEBUG:teuthology.orchestra.run.smithi067:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-17T10:28:13.949 DEBUG:teuthology.orchestra.run.smithi067:> indent=2)) 2024-04-17T10:28:13.949 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:28:14.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:14 stdin-killer INFO: command exited with status 2: exiting normally with same code! 2024-04-17T10:28:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-04-17T10:28:14.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:14.070 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec git clone --branch giant http://github.com/ceph/ceph-qa-suite /home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite) 2024-04-17T10:28:14.126 INFO:teuthology.orchestra.run.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite'... 2024-04-17T10:28:14.357 INFO:teuthology.orchestra.run.smithi067.stderr:warning: redirecting to https://github.com/ceph/ceph-qa-suite/ 2024-04-17T10:28:17.627 INFO:teuthology.orchestra.run.smithi067.stderr:Updating files: 51% (721/1398) Updating files: 52% (727/1398) Updating files: 53% (741/1398) Updating files: 54% (755/1398) Updating files: 55% (769/1398) Updating files: 56% (783/1398) Updating files: 57% (797/1398) Updating files: 58% (811/1398) Updating files: 59% (825/1398) Updating files: 60% (839/1398) Updating files: 61% (853/1398) Updating files: 62% (867/1398) Updating files: 63% (881/1398) Updating files: 64% (895/1398) Updating files: 65% (909/1398) Updating files: 66% (923/1398) Updating files: 67% (937/1398) Updating files: 68% (951/1398) Updating files: 69% (965/1398) Updating files: 70% (979/1398) Updating files: 71% (993/1398) Updating files: 72% (1007/1398) Updating files: 73% (1021/1398) Updating files: 74% (1035/1398) Updating files: 75% (1049/1398) Updating files: 76% (1063/1398) Updating files: 77% (1077/1398) Updating files: 78% (1091/1398) Updating files: 79% (1105/1398) Updating files: 80% (1119/1398) Updating files: 81% (1133/1398) Updating files: 82% (1147/1398) Updating files: 83% (1161/1398) Updating files: 84% (1175/1398) Updating files: 85% (1189/1398) Updating files: 86% (1203/1398) Updating files: 87% (1217/1398) Updating files: 88% (1231/1398) Updating files: 89% (1245/1398) Updating files: 90% (1259/1398) Updating files: 91% (1273/1398) Updating files: 92% (1287/1398) Updating files: 93% (1301/1398) Updating files: 94% (1315/1398) Updating files: 95% (1329/1398) Updating files: 96% (1343/1398) Updating files: 97% (1357/1398) Updating files: 98% (1371/1398) Updating files: 99% (1385/1398) Updating files: 100% (1398/1398) Updating files: 100% (1398/1398), done. 2024-04-17T10:28:17.636 INFO:tasks.cephfs.test_scrub_checks:Initiating mds_scrub_checks on mds.0 test_path /scrub_test_path, run_seq 0 2024-04-17T10:28:17.636 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/i/dont/exist' 2024-04-17T10:28:17.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:17.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bca000640 1 Processor -- start 2024-04-17T10:28:17.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bca000640 1 -- start start 2024-04-17T10:28:17.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bca000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4bc41446c0 0x7f4bc4144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bca000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bc4144aa0 0x7f4bc40d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bca000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc40b6640 con 0x7f4bc4071840 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bca000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc40b51a0 con 0x7f4bc4144aa0 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bca000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc41456d0 con 0x7f4bc41446c0 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bbaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bc4144aa0 0x7f4bc40d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bbaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bc4144aa0 0x7f4bc40d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60378/0 (socket says 172.21.15.67:60378) 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bbaa00640 1 -- 172.21.15.67:0/2117963678 learned_addr learned my addr 172.21.15.67:0/2117963678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bc3400640 1 --2- 172.21.15.67:0/2117963678 >> v2:172.21.15.86:3300/0 conn(0x7f4bc41446c0 0x7f4bc4144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:17.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bbaa00640 1 -- 172.21.15.67:0/2117963678 >> v2:172.21.15.86:3300/0 conn(0x7f4bc41446c0 msgr2=0x7f4bc4144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:17.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bbaa00640 1 --2- 172.21.15.67:0/2117963678 >> v2:172.21.15.86:3300/0 conn(0x7f4bc41446c0 0x7f4bc4144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:17.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bbaa00640 1 -- 172.21.15.67:0/2117963678 >> v1:172.21.15.86:6789/0 conn(0x7f4bc4071840 legacy=0x7f4bc415f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:17.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bbaa00640 1 -- 172.21.15.67:0/2117963678 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc40b4870 con 0x7f4bc4144aa0 2024-04-17T10:28:17.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.830+0000 7f4bc3400640 1 --2- 172.21.15.67:0/2117963678 >> v2:172.21.15.86:3300/0 conn(0x7f4bc41446c0 0x7f4bc4144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:17.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bbaa00640 1 --2- 172.21.15.67:0/2117963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc4144aa0 0x7f4bc40d0910 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4bc415f620 tx=0x7f4bac01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85c45240211394c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:17.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bc2a00640 1 -- 172.21.15.67:0/2117963678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0aa5f0 con 0x7f4bc4144aa0 2024-04-17T10:28:17.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bc2a00640 1 -- 172.21.15.67:0/2117963678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac0af070 con 0x7f4bc4144aa0 2024-04-17T10:28:17.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bc2a00640 1 -- 172.21.15.67:0/2117963678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0aa9b0 con 0x7f4bc4144aa0 2024-04-17T10:28:17.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 -- 172.21.15.67:0/2117963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc4144aa0 msgr2=0x7f4bc40d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:17.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 --2- 172.21.15.67:0/2117963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc4144aa0 0x7f4bc40d0910 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f4bc415f620 tx=0x7f4bac01c490 comp rx=0 tx=0).stop 2024-04-17T10:28:17.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 -- 172.21.15.67:0/2117963678 shutdown_connections 2024-04-17T10:28:17.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 --2- 172.21.15.67:0/2117963678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc4144aa0 0x7f4bc40d0910 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:17.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 --2- 172.21.15.67:0/2117963678 >> v2:172.21.15.86:3300/0 conn(0x7f4bc41446c0 0x7f4bc4144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:17.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 -- 172.21.15.67:0/2117963678 >> 172.21.15.67:0/2117963678 conn(0x7f4bc40b34d0 msgr2=0x7f4bc40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:17.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 -- 172.21.15.67:0/2117963678 shutdown_connections 2024-04-17T10:28:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.834+0000 7f4bca000640 1 -- 172.21.15.67:0/2117963678 wait complete. 2024-04-17T10:28:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 Processor -- start 2024-04-17T10:28:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 -- start start 2024-04-17T10:28:17.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bc4071840 0x7f4bc4158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:17.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc41446c0 0x7f4bc415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:17.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bc4144aa0 0x7f4bc40aa570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:17.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc40d1260 con 0x7f4bc41446c0 2024-04-17T10:28:17.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc415c0d0 con 0x7f4bc4071840 2024-04-17T10:28:17.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc4168a40 con 0x7f4bc4144aa0 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc41446c0 0x7f4bc415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bc3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bc4071840 0x7f4bc4158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc41446c0 0x7f4bc415acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60390/0 (socket says 172.21.15.67:60390) 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 -- 172.21.15.67:0/1576452923 learned_addr learned my addr 172.21.15.67:0/1576452923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bc3e00640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bc4144aa0 0x7f4bc40aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 -- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bc4144aa0 msgr2=0x7f4bc40aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bc4144aa0 0x7f4bc40aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 -- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bc4071840 msgr2=0x7f4bc4158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bc4071840 0x7f4bc4158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:17.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 -- 172.21.15.67:0/1576452923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc41594d0 con 0x7f4bc41446c0 2024-04-17T10:28:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbaa00640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc41446c0 0x7f4bc415acf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4bac01a0c0 tx=0x7f4bac01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbbe00640 1 -- 172.21.15.67:0/1576452923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0bd5f0 con 0x7f4bc41446c0 2024-04-17T10:28:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbbe00640 1 -- 172.21.15.67:0/1576452923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac0bd790 con 0x7f4bc41446c0 2024-04-17T10:28:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bbbe00640 1 -- 172.21.15.67:0/1576452923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac0bdb80 con 0x7f4bc41446c0 2024-04-17T10:28:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc40aaf50 con 0x7f4bc41446c0 2024-04-17T10:28:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc40a89a0 con 0x7f4bc41446c0 2024-04-17T10:28:17.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.838+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc4070bb0 con 0x7f4bc41446c0 2024-04-17T10:28:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.842+0000 7f4bbbe00640 1 -- 172.21.15.67:0/1576452923 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4bac0b33f0 con 0x7f4bc41446c0 2024-04-17T10:28:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.842+0000 7f4bbbe00640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ba0066f60 0x7f4ba0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.842+0000 7f4bbbe00640 1 -- 172.21.15.67:0/1576452923 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4bac11fe20 con 0x7f4bc41446c0 2024-04-17T10:28:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.842+0000 7f4bc3400640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ba0066f60 0x7f4ba0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:17.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.842+0000 7f4bc3400640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ba0066f60 0x7f4ba0069420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009950 tx=0x7f4bb4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:17.846+0000 7f4bbbe00640 1 -- 172.21.15.67:0/1576452923 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4bac0c3060 con 0x7f4bc41446c0 2024-04-17T10:28:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.042+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bc4074e30 con 0x7f4bc41446c0 2024-04-17T10:28:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.042+0000 7f4bbbe00640 1 -- 172.21.15.67:0/1576452923 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f4bac0160b0 con 0x7f4bc41446c0 2024-04-17T10:28:18.037 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:18.037 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:18.037 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ba0066f60 msgr2=0x7f4ba0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ba0066f60 0x7f4ba0069420 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4bb4009950 tx=0x7f4bb4008040 comp rx=0 tx=0).stop 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc41446c0 msgr2=0x7f4bc415acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc41446c0 0x7f4bc415acf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4bac01a0c0 tx=0x7f4bac01acd0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 shutdown_connections 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ba0066f60 0x7f4ba0069420 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bc4144aa0 0x7f4bc40aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bc41446c0 0x7f4bc415acf0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 --2- 172.21.15.67:0/1576452923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bc4071840 0x7f4bc4158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 >> 172.21.15.67:0/1576452923 conn(0x7f4bc40b34d0 msgr2=0x7f4bc40b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 shutdown_connections 2024-04-17T10:28:18.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.046+0000 7f4bca000640 1 -- 172.21.15.67:0/1576452923 wait complete. 2024-04-17T10:28:18.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/i/dont/exist 2024-04-17T10:28:18.200 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:18.200 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": -2 2024-04-17T10:28:18.200 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:18.215 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/i/dont/exist' got response code '0' and stdout '{ "return_code": -2 } ' 2024-04-17T10:28:18.215 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/i/dont/exist' 2024-04-17T10:28:18.215 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:18.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/i/dont/exist 2024-04-17T10:28:18.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703e00640 1 Processor -- start 2024-04-17T10:28:18.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703e00640 1 -- start start 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17040c4730 0x7f17040c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f170406e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170406d830 con 0x7f17041452d0 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f170415ce20 con 0x7f17040c4730 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17040ceea0 con 0x7f170406dfc0 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1702a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f17041452d0 0x7f17040ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45182/0 (socket says 172.21.15.67:45182) 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1702a00640 1 -- 172.21.15.67:0/2577855813 learned_addr learned my addr 172.21.15.67:0/2577855813 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f16fbe00640 1 --2- 172.21.15.67:0/2577855813 >> v2:172.21.15.67:3300/0 conn(0x7f17040c4730 0x7f17040c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1703400640 1 --2- 172.21.15.67:0/2577855813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f170406e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.366+0000 7f1702000640 1 -- 172.21.15.67:0/2577855813 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728828498 0 0) 0x7f170406d830 con 0x7f17041452d0 2024-04-17T10:28:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1702000640 1 -- 172.21.15.67:0/2577855813 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16e8003530 con 0x7f17041452d0 2024-04-17T10:28:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f16fbe00640 1 -- 172.21.15.67:0/2577855813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 msgr2=0x7f170406e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f16fbe00640 1 --2- 172.21.15.67:0/2577855813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f170406e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f16fbe00640 1 -- 172.21.15.67:0/2577855813 >> v1:172.21.15.67:6789/0 conn(0x7f17041452d0 legacy=0x7f17040ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f16fbe00640 1 -- 172.21.15.67:0/2577855813 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040cf180 con 0x7f17040c4730 2024-04-17T10:28:18.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f16fbe00640 1 --2- 172.21.15.67:0/2577855813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17040c4730 0x7f17040c4bc0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0097c0 tx=0x7f16ec01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f431360a44e4cb27 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1702000640 1 -- 172.21.15.67:0/2577855813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0b1070 con 0x7f17040c4730 2024-04-17T10:28:18.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1702000640 1 -- 172.21.15.67:0/2577855813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec0ac410 con 0x7f17040c4730 2024-04-17T10:28:18.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1702000640 1 -- 172.21.15.67:0/2577855813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0ac7e0 con 0x7f17040c4730 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1703e00640 1 -- 172.21.15.67:0/2577855813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17040c4730 msgr2=0x7f17040c4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1703e00640 1 --2- 172.21.15.67:0/2577855813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17040c4730 0x7f17040c4bc0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0097c0 tx=0x7f16ec01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1703e00640 1 -- 172.21.15.67:0/2577855813 shutdown_connections 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1703e00640 1 --2- 172.21.15.67:0/2577855813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f170406e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1703e00640 1 --2- 172.21.15.67:0/2577855813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17040c4730 0x7f17040c4bc0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1703e00640 1 -- 172.21.15.67:0/2577855813 >> 172.21.15.67:0/2577855813 conn(0x7f170406ac90 msgr2=0x7f170406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.370+0000 7f1703e00640 1 -- 172.21.15.67:0/2577855813 shutdown_connections 2024-04-17T10:28:18.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 -- 172.21.15.67:0/2577855813 wait complete. 2024-04-17T10:28:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 Processor -- start 2024-04-17T10:28:18.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 -- start start 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f17041554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17040c4730 0x7f1704143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17041452d0 0x7f170413f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17041682a0 con 0x7f17041452d0 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170415bef0 con 0x7f17040c4730 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1704168420 con 0x7f170406dfc0 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17041452d0 0x7f170413f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17041452d0 0x7f170413f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60400/0 (socket says 172.21.15.67:60400) 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 -- 172.21.15.67:0/1380818415 learned_addr learned my addr 172.21.15.67:0/1380818415 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1702a00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f17041554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f16fbe00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17040c4730 0x7f1704143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 -- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 msgr2=0x7f17041554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f17041554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 -- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17040c4730 msgr2=0x7f1704143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17040c4730 0x7f1704143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 -- 172.21.15.67:0/1380818415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170413f580 con 0x7f17041452d0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f1703400640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17041452d0 0x7f170413f040 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00efc0 tx=0x7f16fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f16faa00640 1 -- 172.21.15.67:0/1380818415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc018020 con 0x7f17041452d0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.374+0000 7f16faa00640 1 -- 172.21.15.67:0/1380818415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16fc014480 con 0x7f17041452d0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170413e2a0 con 0x7f17041452d0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f16faa00640 1 -- 172.21.15.67:0/1380818415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16fc014870 con 0x7f17041452d0 2024-04-17T10:28:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f170413e860 con 0x7f17041452d0 2024-04-17T10:28:18.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f16faa00640 1 -- 172.21.15.67:0/1380818415 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f16fc01b020 con 0x7f17041452d0 2024-04-17T10:28:18.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f16faa00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e0066f60 0x7f16e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f16faa00640 1 -- 172.21.15.67:0/1380818415 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f16fc085f00 con 0x7f17041452d0 2024-04-17T10:28:18.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f1702a00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e0066f60 0x7f16e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f170cdf0000 1 Processor -- start 2024-04-17T10:28:18.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f170cdf0000 1 -- start start 2024-04-17T10:28:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.378+0000 7f1702a00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e0066f60 0x7f16e0069420 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f16f4007fd0 tx=0x7f16f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb150d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb106cd0 0x55abeb1070b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abeaf1c8a0 con 0x55abeb150970 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abeaf54d10 con 0x55abeb106cd0 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55abeaedd2d0 con 0x55abeb151290 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb150d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60410/0 (socket says 172.21.15.67:60410) 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 -- 172.21.15.67:0/3038740324 learned_addr learned my addr 172.21.15.67:0/3038740324 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f16fbe00640 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1703400640 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb106cd0 0x55abeb1070b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 -- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 msgr2=0x55abeb106790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb106790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 -- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb106cd0 msgr2=0x55abeb1070b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb106cd0 0x55abeb1070b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 -- 172.21.15.67:0/3038740324 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abeaf9b870 con 0x55abeb150970 2024-04-17T10:28:18.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f1702a00640 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb150d50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f16f400fab0 tx=0x7f16f4000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e0130ce488fb34 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f16f3400640 1 -- 172.21.15.67:0/3038740324 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4025070 con 0x55abeb150970 2024-04-17T10:28:18.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f16f3400640 1 -- 172.21.15.67:0/3038740324 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4015bf0 con 0x55abeb150970 2024-04-17T10:28:18.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f16f3400640 1 -- 172.21.15.67:0/3038740324 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4020400 con 0x55abeb150970 2024-04-17T10:28:18.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- 172.21.15.67:0/3038740324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 msgr2=0x55abeb150d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb150d50 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f16f400fab0 tx=0x7f16f4000ee0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- 172.21.15.67:0/3038740324 shutdown_connections 2024-04-17T10:28:18.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb106cd0 0x55abeb1070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb106790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 --2- 172.21.15.67:0/3038740324 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb150d50 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- 172.21.15.67:0/3038740324 >> 172.21.15.67:0/3038740324 conn(0x55abeb10d540 msgr2=0x55abeb10d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- 172.21.15.67:0/3038740324 shutdown_connections 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- 172.21.15.67:0/3038740324 wait complete. 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 -- start start 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.382+0000 7f170cdf0000 1 Processor -- start 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb158210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb10a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abeaf0fb50 con 0x55abeb150970 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abeaf50db0 con 0x55abeb151290 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55abeaf30a80 con 0x55abeb106cd0 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 10 client.0 _ll_register_callbacks cb 0x55abeb008fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:18.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb158210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb158210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60416/0 (socket says 172.21.15.67:60416) 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 -- 172.21.15.67:0/2253628897 learned_addr learned my addr 172.21.15.67:0/2253628897 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f1702a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f1703400640 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb10a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 -- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 msgr2=0x55abeb157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 -- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 msgr2=0x55abeb10a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb10a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 -- 172.21.15.67:0/2253628897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55abeaedd520 con 0x55abeb150970 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16fbe00640 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb158210 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0af040 tx=0x7f16ec01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b9915f5707da97c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16e7e00640 1 -- 172.21.15.67:0/2253628897 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0b1040 con 0x55abeb150970 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16f4015bf0 con 0x55abeb150970 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16e7e00640 1 -- 172.21.15.67:0/2253628897 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f16ec0a9040 con 0x55abeb150970 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16e7e00640 1 -- 172.21.15.67:0/2253628897 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16ec0c7050 con 0x55abeb150970 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 10 client.5979 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55abeaf4fd00 con 0x55abeb150970 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16e7e00640 1 -- 172.21.15.67:0/2253628897 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f4015bf0 con 0x55abeb150970 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16e7e00640 1 -- 172.21.15.67:0/2253628897 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f16ec0ca020 con 0x55abeb150970 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 10 client.5979 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 10 client.5979 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 20 client.5979 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb10adf0 0x55abeb10b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 4 client.5979 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55abeaef0b10 con 0x55abeb10adf0 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f1702a00640 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb10adf0 0x55abeb10b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f1702a00640 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb10adf0 0x55abeb10b1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=239a87ca5e284635 server_cookie=ad282bb41bfc7cd6 in_seq=0 out_seq=0 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.386+0000 7f16e7e00640 10 client.5979 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f16e7e00640 1 -- 172.21.15.67:0/2253628897 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55abeaef0b10 con 0x55abeb10adf0 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f16e7e00640 10 client.5979 handle_command_reply: tid=0 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 client.5979 shutdown 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 msgr2=0x55abeb158210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb158210 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0af040 tx=0x7f16ec01ace0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 shutdown_connections 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb10adf0 0x55abeb10b1d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb10a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb158210 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 --2- 172.21.15.67:0/2253628897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f16e7e00640 0 client.5979 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 >> 172.21.15.67:0/2253628897 conn(0x55abeb10d540 msgr2=0x55abeb103770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 shutdown_connections 2024-04-17T10:28:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 1 -- 172.21.15.67:0/2253628897 wait complete. 2024-04-17T10:28:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.390+0000 7f170cdf0000 20 client.5979 trim_cache size 0 max 16384 2024-04-17T10:28:18.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.406+0000 7f170cdf0000 1 Processor -- start 2024-04-17T10:28:18.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.406+0000 7f170cdf0000 1 -- start start 2024-04-17T10:28:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb1acda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb1ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb1ad820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17042030c0 con 0x55abeb150970 2024-04-17T10:28:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17042022a0 con 0x55abeb151290 2024-04-17T10:28:18.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1704138a70 con 0x55abeb106cd0 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb1ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb1ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60430/0 (socket says 172.21.15.67:60430) 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 -- 172.21.15.67:0/1055794282 learned_addr learned my addr 172.21.15.67:0/1055794282 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f1702a00640 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb1acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f1703400640 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb1ad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 -- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 msgr2=0x55abeb1acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb1acda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 -- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 msgr2=0x55abeb1ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb1ad820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 -- 172.21.15.67:0/1055794282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55abeafd8920 con 0x55abeb150970 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16fbe00640 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb1ad2e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f16ec01a0d0 tx=0x7f16ec0b8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f9fe41095ee1903 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16f2a00640 1 -- 172.21.15.67:0/1055794282 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0b1040 con 0x55abeb150970 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16f2a00640 1 -- 172.21.15.67:0/1055794282 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec0b8aa0 con 0x55abeb150970 2024-04-17T10:28:18.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f16f2a00640 1 -- 172.21.15.67:0/1055794282 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ec006e20 con 0x55abeb150970 2024-04-17T10:28:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- 172.21.15.67:0/1055794282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 msgr2=0x55abeb1ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb1ad2e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f16ec01a0d0 tx=0x7f16ec0b8000 comp rx=0 tx=0).stop 2024-04-17T10:28:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- 172.21.15.67:0/1055794282 shutdown_connections 2024-04-17T10:28:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb151290 0x55abeb1ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb150970 0x55abeb1ad2e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- 172.21.15.67:0/1055794282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb106cd0 0x55abeb1acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- 172.21.15.67:0/1055794282 >> 172.21.15.67:0/1055794282 conn(0x55abeb10d540 msgr2=0x55abeb10b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- 172.21.15.67:0/1055794282 shutdown_connections 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- 172.21.15.67:0/1055794282 wait complete. 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 -- start start 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 Processor -- start 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb106cd0 0x55abeb1b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb150970 0x55abeb1559a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.410+0000 7f170cdf0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55abeaf16700 con 0x55abeb106cd0 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1704201cd0 con 0x55abeb150970 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55abeaf59340 con 0x55abeb151290 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 10 client.0 _ll_register_callbacks cb 0x7f16f40015a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb106cd0 0x55abeb1b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb106cd0 0x55abeb1b0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60438/0 (socket says 172.21.15.67:60438) 2024-04-17T10:28:18.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 -- 172.21.15.67:0/618913255 learned_addr learned my addr 172.21.15.67:0/618913255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:18.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16fbe00640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb150970 0x55abeb1559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1703400640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 -- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 msgr2=0x55abeb155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb155ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 -- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb150970 msgr2=0x55abeb1559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb150970 0x55abeb1559a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 -- 172.21.15.67:0/618913255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55abeafaea30 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1703400640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f1702a00640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb106cd0 0x55abeb1b0560 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f16f4023040 tx=0x7f16f4000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87a997f8fff67c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 1 -- 172.21.15.67:0/618913255 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f4025040 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16ec0b8aa0 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 1 -- 172.21.15.67:0/618913255 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f16f4032d00 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 1 -- 172.21.15.67:0/618913255 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f403f050 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 10 client.5985 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55abeaefc150 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 1 -- 172.21.15.67:0/618913255 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ec0b8aa0 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 1 -- 172.21.15.67:0/618913255 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f16f4042020 con 0x55abeb106cd0 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 10 client.5985 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 10 client.5985 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:18.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 20 client.5985 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb156870 0x55abeb156c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 4 client.5985 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}] 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}) v1 -- 0x55abeaf62250 con 0x55abeb156870 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16fbe00640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb156870 0x55abeb156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16fbe00640 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb156870 0x55abeb156c50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=126f1fdbb16cdc82 server_cookie=c01f544411307d37 in_seq=0 out_seq=0 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 10 client.5985 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 1 -- 172.21.15.67:0/618913255 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+26 (crc 0 0 0) 0x55abeaf62250 con 0x55abeb156870 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 10 client.5985 handle_command_reply: tid=0 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 client.5985 shutdown 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb106cd0 msgr2=0x55abeb1b0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb106cd0 0x55abeb1b0560 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f16f4023040 tx=0x7f16f4000bd0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 shutdown_connections 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55abeb156870 0x55abeb156c50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55abeb151290 0x55abeb155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55abeb150970 0x55abeb1559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 --2- 172.21.15.67:0/618913255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55abeb106cd0 0x55abeb1b0560 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f16e7e00640 0 client.5985 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 >> 172.21.15.67:0/618913255 conn(0x55abeb10d540 msgr2=0x55abeb10ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 shutdown_connections 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 1 -- 172.21.15.67:0/618913255 wait complete. 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.414+0000 7f170cdf0000 20 client.5985 trim_cache size 0 max 16384 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": -2 2024-04-17T10:28:18.413 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:18.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e0066f60 msgr2=0x7f16e0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e0066f60 0x7f16e0069420 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f16f4007fd0 tx=0x7f16f4009040 comp rx=0 tx=0).stop 2024-04-17T10:28:18.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17041452d0 msgr2=0x7f170413f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17041452d0 0x7f170413f040 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f16fc00efc0 tx=0x7f16fc00c490 comp rx=0 tx=0).stop 2024-04-17T10:28:18.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 shutdown_connections 2024-04-17T10:28:18.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e0066f60 0x7f16e0069420 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17041452d0 0x7f170413f040 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17040c4730 0x7f1704143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 --2- 172.21.15.67:0/1380818415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170406dfc0 0x7f17041554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 >> 172.21.15.67:0/1380818415 conn(0x7f170406ac90 msgr2=0x7f170415a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 shutdown_connections 2024-04-17T10:28:18.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.418+0000 7f1703e00640 1 -- 172.21.15.67:0/1380818415 wait complete. 2024-04-17T10:28:18.430 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/i/dont/exist' returned '{'return_code': -2}' 2024-04-17T10:28:18.430 INFO:tasks.cephfs.test_scrub_checks:First run: flushing /scrub_test_path/ceph-qa-suite/suites 2024-04-17T10:28:18.430 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/suites' 2024-04-17T10:28:18.431 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6976200640 1 Processor -- start 2024-04-17T10:28:18.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6976200640 1 -- start start 2024-04-17T10:28:18.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6976200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970071310 0x7f6970071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6976200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69700cc300 0x7f69700cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6976200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697006e2f0 con 0x7f6970145040 2024-04-17T10:28:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6976200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69700a86c0 con 0x7f69700cc300 2024-04-17T10:28:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6976200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6970071cb0 con 0x7f6970071310 2024-04-17T10:28:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6970145040 0x7f697006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45184/0 (socket says 172.21.15.67:45184) 2024-04-17T10:28:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696f400640 1 -- 172.21.15.67:0/1517974479 learned_addr learned my addr 172.21.15.67:0/1517974479 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696fe00640 1 --2- 172.21.15.67:0/1517974479 >> v2:172.21.15.67:3300/0 conn(0x7f69700cc300 0x7f69700cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f6967e00640 1 --2- 172.21.15.67:0/1517974479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970071310 0x7f6970071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696ea00640 1 -- 172.21.15.67:0/1517974479 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156373087 0 0) 0x7f697006e2f0 con 0x7f6970145040 2024-04-17T10:28:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696fe00640 1 -- 172.21.15.67:0/1517974479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970071310 msgr2=0x7f6970071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696fe00640 1 --2- 172.21.15.67:0/1517974479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970071310 0x7f6970071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696fe00640 1 -- 172.21.15.67:0/1517974479 >> v1:172.21.15.67:6789/0 conn(0x7f6970145040 legacy=0x7f697006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.578+0000 7f696fe00640 1 -- 172.21.15.67:0/1517974479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697006d6e0 con 0x7f69700cc300 2024-04-17T10:28:18.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f696fe00640 1 --2- 172.21.15.67:0/1517974479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69700cc300 0x7f69700cc750 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d90 tx=0x7f696801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0442b7295f38cb7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f696ea00640 1 -- 172.21.15.67:0/1517974479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680b4070 con 0x7f69700cc300 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f696ea00640 1 -- 172.21.15.67:0/1517974479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69680af600 con 0x7f69700cc300 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f696ea00640 1 -- 172.21.15.67:0/1517974479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69680af9d0 con 0x7f69700cc300 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 -- 172.21.15.67:0/1517974479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69700cc300 msgr2=0x7f69700cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 --2- 172.21.15.67:0/1517974479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69700cc300 0x7f69700cc750 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d90 tx=0x7f696801f150 comp rx=0 tx=0).stop 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 -- 172.21.15.67:0/1517974479 shutdown_connections 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 --2- 172.21.15.67:0/1517974479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69700cc300 0x7f69700cc750 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 --2- 172.21.15.67:0/1517974479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970071310 0x7f6970071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 -- 172.21.15.67:0/1517974479 >> 172.21.15.67:0/1517974479 conn(0x7f697006ac90 msgr2=0x7f697006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 -- 172.21.15.67:0/1517974479 shutdown_connections 2024-04-17T10:28:18.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.582+0000 7f6976200640 1 -- 172.21.15.67:0/1517974479 wait complete. 2024-04-17T10:28:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 Processor -- start 2024-04-17T10:28:18.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 -- start start 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6970071310 0x7f697015b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69700cc300 0x7f6970141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970145040 0x7f69700b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697006dca0 con 0x7f6970071310 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697015bf10 con 0x7f69700cc300 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697006df80 con 0x7f6970145040 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6970071310 0x7f697015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6970071310 0x7f697015b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60448/0 (socket says 172.21.15.67:60448) 2024-04-17T10:28:18.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 -- 172.21.15.67:0/1573202458 learned_addr learned my addr 172.21.15.67:0/1573202458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 -- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970145040 msgr2=0x7f69700b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970145040 0x7f69700b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 -- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69700cc300 msgr2=0x7f6970141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6967e00640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69700cc300 0x7f6970141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69700cc300 0x7f6970141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 -- 172.21.15.67:0/1573202458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f697015b5b0 con 0x7f6970071310 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6967e00640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69700cc300 0x7f6970141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f696f400640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6970071310 0x7f697015b070 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f696000c480 tx=0x7f696000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6967400640 1 -- 172.21.15.67:0/1573202458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960018070 con 0x7f6970071310 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6967400640 1 -- 172.21.15.67:0/1573202458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6960007500 con 0x7f6970071310 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69701435c0 con 0x7f6970071310 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6967400640 1 -- 172.21.15.67:0/1573202458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6960004a30 con 0x7f6970071310 2024-04-17T10:28:18.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6970142800 con 0x7f6970071310 2024-04-17T10:28:18.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.586+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6970072d50 con 0x7f6970071310 2024-04-17T10:28:18.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.590+0000 7f6967400640 1 -- 172.21.15.67:0/1573202458 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f696001b070 con 0x7f6970071310 2024-04-17T10:28:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.590+0000 7f6967400640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f694c066f60 0x7f694c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.590+0000 7f6967400640 1 -- 172.21.15.67:0/1573202458 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6960084560 con 0x7f6970071310 2024-04-17T10:28:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.590+0000 7f6967e00640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f694c066f60 0x7f694c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:18.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.590+0000 7f6967e00640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f694c066f60 0x7f694c069420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f697013f690 tx=0x7f6958008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:18.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.594+0000 7f6967400640 1 -- 172.21.15.67:0/1573202458 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f697015b5b0 con 0x7f6970071310 2024-04-17T10:28:18.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6970074e30 con 0x7f6970071310 2024-04-17T10:28:18.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6967400640 1 -- 172.21.15.67:0/1573202458 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f69600553d0 con 0x7f6970071310 2024-04-17T10:28:18.783 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:18.783 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:18.784 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f694c066f60 msgr2=0x7f694c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f694c066f60 0x7f694c069420 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f697013f690 tx=0x7f6958008040 comp rx=0 tx=0).stop 2024-04-17T10:28:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6970071310 msgr2=0x7f697015b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6970071310 0x7f697015b070 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f696000c480 tx=0x7f696000c940 comp rx=0 tx=0).stop 2024-04-17T10:28:18.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 shutdown_connections 2024-04-17T10:28:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f694c066f60 0x7f694c069420 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6970145040 0x7f69700b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69700cc300 0x7f6970141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 --2- 172.21.15.67:0/1573202458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6970071310 0x7f697015b070 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 >> 172.21.15.67:0/1573202458 conn(0x7f697006ac90 msgr2=0x7f69700af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 shutdown_connections 2024-04-17T10:28:18.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:18.790+0000 7f6976200640 1 -- 172.21.15.67:0/1573202458 wait complete. 2024-04-17T10:28:18.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/ceph-qa-suite/suites 2024-04-17T10:28:18.996 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:18.996 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:18.996 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:19.011 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/suites' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:19.011 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites' 2024-04-17T10:28:19.011 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:19.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites 2024-04-17T10:28:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ec00640 1 Processor -- start 2024-04-17T10:28:19.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ec00640 1 -- start start 2024-04-17T10:28:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ec00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8980cd480 0x7fc8980b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8980cc3e0 con 0x7fc8980b2a30 2024-04-17T10:28:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ec00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8980cc6c0 con 0x7fc898099a50 2024-04-17T10:28:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ec00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8980cf910 con 0x7fc8980cd480 2024-04-17T10:28:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8980cd480 0x7fc8980b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ce00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc8980b2a30 0x7fc8980cf200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45192/0 (socket says 172.21.15.67:45192) 2024-04-17T10:28:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89ce00640 1 -- 172.21.15.67:0/2871312727 learned_addr learned my addr 172.21.15.67:0/2871312727 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89e200640 1 -- 172.21.15.67:0/2871312727 >> v1:172.21.15.86:6789/0 conn(0x7fc898099a50 legacy=0x7fc898099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89e200640 1 -- 172.21.15.67:0/2871312727 >> v1:172.21.15.67:6789/0 conn(0x7fc8980b2a30 legacy=0x7fc8980cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc89e200640 1 -- 172.21.15.67:0/2871312727 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8980cc9a0 con 0x7fc8980cd480 2024-04-17T10:28:19.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.162+0000 7fc897e00640 1 -- 172.21.15.67:0/2871312727 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210642640 0 0) 0x7fc8980cc3e0 con 0x7fc8980b2a30 2024-04-17T10:28:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89e200640 1 --2- 172.21.15.67:0/2871312727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980b1010 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc890009d90 tx=0x7fc89001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=266f38e40643c8a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc897e00640 1 -- 172.21.15.67:0/2871312727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8900b2070 con 0x7fc8980cd480 2024-04-17T10:28:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc897e00640 1 -- 172.21.15.67:0/2871312727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8900a5970 con 0x7fc8980cd480 2024-04-17T10:28:19.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc897e00640 1 -- 172.21.15.67:0/2871312727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8900a5d40 con 0x7fc8980cd480 2024-04-17T10:28:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89ec00640 1 -- 172.21.15.67:0/2871312727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 msgr2=0x7fc8980b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2871312727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980b1010 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc890009d90 tx=0x7fc89001c8b0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89ec00640 1 -- 172.21.15.67:0/2871312727 shutdown_connections 2024-04-17T10:28:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2871312727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980b1010 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89ec00640 1 -- 172.21.15.67:0/2871312727 >> 172.21.15.67:0/2871312727 conn(0x7fc8980d1630 msgr2=0x7fc8980d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89ec00640 1 -- 172.21.15.67:0/2871312727 shutdown_connections 2024-04-17T10:28:19.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.166+0000 7fc89ec00640 1 -- 172.21.15.67:0/2871312727 wait complete. 2024-04-17T10:28:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 Processor -- start 2024-04-17T10:28:19.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 -- start start 2024-04-17T10:28:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc898099a50 0x7fc898156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980c8700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc89815ccd0 0x7fc8980c8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8980d24f0 con 0x7fc8980cd480 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8980d2370 con 0x7fc898099a50 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89816b630 con 0x7fc89815ccd0 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980c8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980c8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60474/0 (socket says 172.21.15.67:60474) 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 -- 172.21.15.67:0/2532732227 learned_addr learned my addr 172.21.15.67:0/2532732227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89d800640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc898099a50 0x7fc898156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89e200640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc89815ccd0 0x7fc8980c8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 -- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc89815ccd0 msgr2=0x7fc8980c8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc89815ccd0 0x7fc8980c8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 -- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc898099a50 msgr2=0x7fc898156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc898099a50 0x7fc898156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 -- 172.21.15.67:0/2532732227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89815ae20 con 0x7fc8980cd480 2024-04-17T10:28:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ce00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980c8700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc88400c4d0 tx=0x7fc88400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc895600640 1 -- 172.21.15.67:0/2532732227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884018070 con 0x7fc8980cd480 2024-04-17T10:28:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc895600640 1 -- 172.21.15.67:0/2532732227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8840074e0 con 0x7fc8980cd480 2024-04-17T10:28:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc895600640 1 -- 172.21.15.67:0/2532732227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884004a30 con 0x7fc8980cd480 2024-04-17T10:28:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc89815b100 con 0x7fc8980cd480 2024-04-17T10:28:19.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.170+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89815bbe0 con 0x7fc8980cd480 2024-04-17T10:28:19.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc895600640 1 -- 172.21.15.67:0/2532732227 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc88401b070 con 0x7fc8980cd480 2024-04-17T10:28:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc895600640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc878066f60 0x7fc878069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc895600640 1 -- 172.21.15.67:0/2532732227 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc8840844d0 con 0x7fc8980cd480 2024-04-17T10:28:19.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc89d800640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc878066f60 0x7fc878069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 Processor -- start 2024-04-17T10:28:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 -- start start 2024-04-17T10:28:19.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc89d800640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc878066f60 0x7fc878069420 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc88c007fd0 tx=0x7fc88c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615deea0 0x55e1615df280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615df7c0 0x55e1615951d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e161595710 0x55e161595af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1613e3db0 con 0x55e1615deea0 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16136c230 con 0x55e1615df7c0 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.174+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e1613dfe50 con 0x55e161595710 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615df7c0 0x55e1615951d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615deea0 0x55e1615df280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615df7c0 0x55e1615951d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:52950/0 (socket says 172.21.15.67:52950) 2024-04-17T10:28:19.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 -- 172.21.15.67:0/3683325372 learned_addr learned my addr 172.21.15.67:0/3683325372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e161595710 0x55e161595af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 -- 172.21.15.67:0/3683325372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e161595710 msgr2=0x55e161595af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 --2- 172.21.15.67:0/3683325372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e161595710 0x55e161595af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 -- 172.21.15.67:0/3683325372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615deea0 msgr2=0x55e1615df280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 --2- 172.21.15.67:0/3683325372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615deea0 0x55e1615df280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 -- 172.21.15.67:0/3683325372 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1613a5750 con 0x55e1615df7c0 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 --2- 172.21.15.67:0/3683325372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615df7c0 0x55e1615951d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc884082740 tx=0x7fc8840546f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8eed5614c3a9bda server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc88a000640 1 -- 172.21.15.67:0/3683325372 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884052f60 con 0x55e1615df7c0 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc88a000640 1 -- 172.21.15.67:0/3683325372 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8840512d0 con 0x55e1615df7c0 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc88a000640 1 -- 172.21.15.67:0/3683325372 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884004df0 con 0x55e1615df7c0 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- 172.21.15.67:0/3683325372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615df7c0 msgr2=0x55e1615951d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/3683325372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615df7c0 0x55e1615951d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc884082740 tx=0x7fc8840546f0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- 172.21.15.67:0/3683325372 shutdown_connections 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/3683325372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e161595710 0x55e161595af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/3683325372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615df7c0 0x55e1615951d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/3683325372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615deea0 0x55e1615df280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- 172.21.15.67:0/3683325372 >> 172.21.15.67:0/3683325372 conn(0x55e1615916c0 msgr2=0x55e161593ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- 172.21.15.67:0/3683325372 shutdown_connections 2024-04-17T10:28:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- 172.21.15.67:0/3683325372 wait complete. 2024-04-17T10:28:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- start start 2024-04-17T10:28:19.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 Processor -- start 2024-04-17T10:28:19.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1615c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 0x55e1615c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e1615c4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16139ebf0 con 0x55e161595710 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1613a8770 con 0x55e1615deea0 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e161368990 con 0x55e1615df7c0 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc8a36a1000 10 client.0 _ll_register_callbacks cb 0x55e161580790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1615c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1615c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60494/0 (socket says 172.21.15.67:60494) 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89d800640 1 -- 172.21.15.67:0/4196792833 learned_addr learned my addr 172.21.15.67:0/4196792833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89e200640 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e1615c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.178+0000 7fc89ce00640 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 0x55e1615c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89d800640 1 -- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 msgr2=0x55e1615c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89d800640 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e1615c4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89d800640 1 -- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 msgr2=0x55e1615c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89d800640 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 0x55e1615c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89d800640 1 -- 172.21.15.67:0/4196792833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e161386640 con 0x55e161595710 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89d800640 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1615c3ed0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc88c014700 tx=0x7fc88c001130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b62d604de17c05f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 1 -- 172.21.15.67:0/4196792833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c025020 con 0x55e161595710 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 1 -- 172.21.15.67:0/4196792833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc88c013680 con 0x55e161595710 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8840512d0 con 0x55e161595710 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 1 -- 172.21.15.67:0/4196792833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc88c02f050 con 0x55e161595710 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 10 client.6009 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:19.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e16136c480 con 0x55e161595710 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 1 -- 172.21.15.67:0/4196792833 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8840512d0 con 0x55e161595710 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 1 -- 172.21.15.67:0/4196792833 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc88c032020 con 0x55e161595710 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 10 client.6009 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 10 client.6009 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 20 client.6009 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e1615eef40 0x55e1615f1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 4 client.6009 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e16137fbb0 con 0x55e1615eef40 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89ce00640 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e1615eef40 0x55e1615f1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc89ce00640 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e1615eef40 0x55e1615f1330 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=387b3100ddb06a86 server_cookie=5889ad508b7dac2f in_seq=0 out_seq=0 2024-04-17T10:28:19.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 10 client.6009 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 1 -- 172.21.15.67:0/4196792833 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e16137fbb0 con 0x55e1615eef40 2024-04-17T10:28:19.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 10 client.6009 handle_command_reply: tid=0 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 client.6009 shutdown 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 msgr2=0x55e1615c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1615c3ed0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fc88c014700 tx=0x7fc88c001130 comp rx=0 tx=0).stop 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 shutdown_connections 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e1615eef40 0x55e1615f1330 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e1615c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 0x55e1615c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc87ea00640 0 client.6009 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/4196792833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1615c3ed0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 >> 172.21.15.67:0/4196792833 conn(0x55e1615916c0 msgr2=0x55e161594340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 shutdown_connections 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 1 -- 172.21.15.67:0/4196792833 wait complete. 2024-04-17T10:28:19.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.182+0000 7fc8a36a1000 20 client.6009 trim_cache size 0 max 16384 2024-04-17T10:28:19.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 Processor -- start 2024-04-17T10:28:19.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 -- start start 2024-04-17T10:28:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e161595710 0x55e161643dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615deea0 0x55e161644310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615df7c0 0x55e161644850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc898203000 con 0x55e1615df7c0 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc898202200 con 0x55e161595710 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc898157fa0 con 0x55e1615deea0 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc89e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615df7c0 0x55e161644850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc89e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615df7c0 0x55e161644850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60508/0 (socket says 172.21.15.67:60508) 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.198+0000 7fc89e200640 1 -- 172.21.15.67:0/924902833 learned_addr learned my addr 172.21.15.67:0/924902833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e161595710 0x55e161643dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89ce00640 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615deea0 0x55e161644310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89e200640 1 -- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615deea0 msgr2=0x55e161644310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89e200640 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615deea0 0x55e161644310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89e200640 1 -- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e161595710 msgr2=0x55e161643dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89e200640 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e161595710 0x55e161643dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89e200640 1 -- 172.21.15.67:0/924902833 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e161465ef0 con 0x55e1615df7c0 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89ce00640 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615deea0 0x55e161644310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89e200640 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615df7c0 0x55e161644850 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc89001a4e0 tx=0x7fc890004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271bb9fbfe306242 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc889600640 1 -- 172.21.15.67:0/924902833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8900b2040 con 0x55e1615df7c0 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc889600640 1 -- 172.21.15.67:0/924902833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8900b3040 con 0x55e1615df7c0 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc889600640 1 -- 172.21.15.67:0/924902833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8900a5e20 con 0x55e1615df7c0 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- 172.21.15.67:0/924902833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615df7c0 msgr2=0x55e161644850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615df7c0 0x55e161644850 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fc89001a4e0 tx=0x7fc890004ac0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- 172.21.15.67:0/924902833 shutdown_connections 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e1615df7c0 0x55e161644850 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615deea0 0x55e161644310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/924902833 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e161595710 0x55e161643dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- 172.21.15.67:0/924902833 >> 172.21.15.67:0/924902833 conn(0x55e1615916c0 msgr2=0x55e1615ec490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- 172.21.15.67:0/924902833 shutdown_connections 2024-04-17T10:28:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- 172.21.15.67:0/924902833 wait complete. 2024-04-17T10:28:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- start start 2024-04-17T10:28:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 Processor -- start 2024-04-17T10:28:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1616459d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 0x55e161646300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e161646840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e161368990 con 0x55e161595710 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc898201d50 con 0x55e1615deea0 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e16139de40 con 0x55e1615df7c0 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc8a36a1000 10 client.0 _ll_register_callbacks cb 0x7fc8900b0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1616459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1616459d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60516/0 (socket says 172.21.15.67:60516) 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 -- 172.21.15.67:0/2462228361 learned_addr learned my addr 172.21.15.67:0/2462228361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89e200640 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e161646840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 -- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 msgr2=0x55e161646840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e161646840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 -- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 msgr2=0x55e161646300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 0x55e161646300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.202+0000 7fc89d800640 1 -- 172.21.15.67:0/2462228361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1613a39a0 con 0x55e161595710 2024-04-17T10:28:19.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89d800640 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1616459d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc88c014700 tx=0x7fc88c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1603ee195ffa381b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 1 -- 172.21.15.67:0/2462228361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c025040 con 0x55e161595710 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 1 -- 172.21.15.67:0/2462228361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc88c021600 con 0x55e161595710 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8900b3040 con 0x55e161595710 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 1 -- 172.21.15.67:0/2462228361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc88c044050 con 0x55e161595710 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 10 client.6015 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e1610a05a0 con 0x55e161595710 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 1 -- 172.21.15.67:0/2462228361 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8900b3040 con 0x55e161595710 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 1 -- 172.21.15.67:0/2462228361 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc88c047020 con 0x55e161595710 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 10 client.6015 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 10 client.6015 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 20 client.6015 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e161646d80 0x55e161647190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 4 client.6015 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}] 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}) v1 -- 0x55e161393080 con 0x55e161646d80 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ce00640 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e161646d80 0x55e161647190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ce00640 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e161646d80 0x55e161647190 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0d2afcd8ce0502e server_cookie=a607274ca730391b in_seq=0 out_seq=0 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 10 client.6015 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 1 -- 172.21.15.67:0/2462228361 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e161393080 con 0x55e161646d80 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 10 client.6015 handle_command_reply: tid=0 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 client.6015 shutdown 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 msgr2=0x55e1616459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1616459d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fc88c014700 tx=0x7fc88c014000 comp rx=0 tx=0).stop 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 shutdown_connections 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e161646d80 0x55e161647190 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e1615df7c0 0x55e161646840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e1615deea0 0x55e161646300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 --2- 172.21.15.67:0/2462228361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e161595710 0x55e1616459d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc87ea00640 0 client.6015 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 >> 172.21.15.67:0/2462228361 conn(0x55e1615916c0 msgr2=0x55e1615dd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 shutdown_connections 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 1 -- 172.21.15.67:0/2462228361 wait complete. 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc8a36a1000 20 client.6015 trim_cache size 0 max 16384 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "c4327ced-7616-436f-8021-d7d9dd671f98", 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:19.205 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc878066f60 msgr2=0x7fc878069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc878066f60 0x7fc878069420 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc88c007fd0 tx=0x7fc88c009040 comp rx=0 tx=0).stop 2024-04-17T10:28:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 msgr2=0x7fc8980c8700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980c8700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc88400c4d0 tx=0x7fc88400c990 comp rx=0 tx=0).stop 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 shutdown_connections 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc878066f60 0x7fc878069420 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc89815ccd0 0x7fc8980c8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8980cd480 0x7fc8980c8700 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 --2- 172.21.15.67:0/2532732227 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc898099a50 0x7fc898156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 >> 172.21.15.67:0/2532732227 conn(0x7fc8980d1630 msgr2=0x7fc8980cef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 shutdown_connections 2024-04-17T10:28:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.206+0000 7fc89ec00640 1 -- 172.21.15.67:0/2532732227 wait complete. 2024-04-17T10:28:19.220 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites' returned '{'return_code': 0, 'scrub_tag': 'c4327ced-7616-436f-8021-d7d9dd671f98', 'mode': 'asynchronous'}' 2024-04-17T10:28:19.220 INFO:tasks.cephfs.test_scrub_checks:First run: flushing /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-04-17T10:28:19.220 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' 2024-04-17T10:28:19.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:19.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.366+0000 7efd15a00640 1 Processor -- start 2024-04-17T10:28:19.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.366+0000 7efd15a00640 1 -- start start 2024-04-17T10:28:19.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.366+0000 7efd15a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd1006e6e0 0x7efd100ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd15a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd101441f0 0x7efd100ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd15a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1015ae40 con 0x7efd1006eac0 2024-04-17T10:28:19.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd15a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd100cf430 con 0x7efd1006e6e0 2024-04-17T10:28:19.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd15a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd10071440 con 0x7efd101441f0 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd1006e6e0 0x7efd100ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efd1006e6e0 0x7efd100ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60522/0 (socket says 172.21.15.67:60522) 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 -- 172.21.15.67:0/1550611736 learned_addr learned my addr 172.21.15.67:0/1550611736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0f400640 1 --2- 172.21.15.67:0/1550611736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd101441f0 0x7efd100ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 -- 172.21.15.67:0/1550611736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd101441f0 msgr2=0x7efd100ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0e000640 1 -- 172.21.15.67:0/1550611736 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078122133 0 0) 0x7efd1015ae40 con 0x7efd1006eac0 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 --2- 172.21.15.67:0/1550611736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd101441f0 0x7efd100ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 -- 172.21.15.67:0/1550611736 >> v1:172.21.15.86:6789/0 conn(0x7efd1006eac0 legacy=0x7efd10143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 -- 172.21.15.67:0/1550611736 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd10071720 con 0x7efd1006e6e0 2024-04-17T10:28:19.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0ea00640 1 --2- 172.21.15.67:0/1550611736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006e6e0 0x7efd100ce9b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7efd0000bb10 tx=0x7efd0001eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8261237f654090fd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0e000640 1 -- 172.21.15.67:0/1550611736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000b4070 con 0x7efd1006e6e0 2024-04-17T10:28:19.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0e000640 1 -- 172.21.15.67:0/1550611736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd000a7a50 con 0x7efd1006e6e0 2024-04-17T10:28:19.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.370+0000 7efd0e000640 1 -- 172.21.15.67:0/1550611736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000a7e20 con 0x7efd1006e6e0 2024-04-17T10:28:19.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- 172.21.15.67:0/1550611736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006e6e0 msgr2=0x7efd100ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 --2- 172.21.15.67:0/1550611736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006e6e0 0x7efd100ce9b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7efd0000bb10 tx=0x7efd0001eb70 comp rx=0 tx=0).stop 2024-04-17T10:28:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- 172.21.15.67:0/1550611736 shutdown_connections 2024-04-17T10:28:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 --2- 172.21.15.67:0/1550611736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd101441f0 0x7efd100ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 --2- 172.21.15.67:0/1550611736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006e6e0 0x7efd100ce9b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- 172.21.15.67:0/1550611736 >> 172.21.15.67:0/1550611736 conn(0x7efd1006ac90 msgr2=0x7efd1006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- 172.21.15.67:0/1550611736 shutdown_connections 2024-04-17T10:28:19.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- 172.21.15.67:0/1550611736 wait complete. 2024-04-17T10:28:19.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 Processor -- start 2024-04-17T10:28:19.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- start start 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd1006e6e0 0x7efd1013c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006eac0 0x7efd10136200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd101441f0 0x7efd100b1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1006dd00 con 0x7efd1006eac0 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1006db80 con 0x7efd101441f0 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd15a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1006de80 con 0x7efd1006e6e0 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006eac0 0x7efd10136200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd0ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd1006e6e0 0x7efd1013c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006eac0 0x7efd10136200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60524/0 (socket says 172.21.15.67:60524) 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd07e00640 1 -- 172.21.15.67:0/3068467533 learned_addr learned my addr 172.21.15.67:0/3068467533 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.374+0000 7efd0f400640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd101441f0 0x7efd100b1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd0ea00640 1 -- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd101441f0 msgr2=0x7efd100b1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd0ea00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd101441f0 0x7efd100b1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd0ea00640 1 -- 172.21.15.67:0/3068467533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006eac0 msgr2=0x7efd10136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd0ea00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006eac0 0x7efd10136200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd0ea00640 1 -- 172.21.15.67:0/3068467533 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1013cd30 con 0x7efd1006e6e0 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd07e00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006eac0 0x7efd10136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd0ea00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd1006e6e0 0x7efd1013c6c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd1006f0f0 tx=0x7efd0001d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd06a00640 1 -- 172.21.15.67:0/3068467533 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000bf070 con 0x7efd1006e6e0 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd06a00640 1 -- 172.21.15.67:0/3068467533 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd000ba440 con 0x7efd1006e6e0 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd10138e30 con 0x7efd1006e6e0 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd06a00640 1 -- 172.21.15.67:0/3068467533 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd000b4070 con 0x7efd1006e6e0 2024-04-17T10:28:19.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd101393f0 con 0x7efd1006e6e0 2024-04-17T10:28:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd10138c20 con 0x7efd1006e6e0 2024-04-17T10:28:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd06a00640 1 -- 172.21.15.67:0/3068467533 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efd000c2070 con 0x7efd1006e6e0 2024-04-17T10:28:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd06a00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efcec066f60 0x7efcec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd06a00640 1 -- 172.21.15.67:0/3068467533 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efd0011f6e0 con 0x7efd1006e6e0 2024-04-17T10:28:19.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.378+0000 7efd07e00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efcec066f60 0x7efcec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.382+0000 7efd07e00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efcec066f60 0x7efcec069420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efd1013c670 tx=0x7efcf8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.386+0000 7efd06a00640 1 -- 172.21.15.67:0/3068467533 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efd00018820 con 0x7efd1006e6e0 2024-04-17T10:28:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.582+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd10074e30 con 0x7efd1006e6e0 2024-04-17T10:28:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.582+0000 7efd06a00640 1 -- 172.21.15.67:0/3068467533 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7efd000eff00 con 0x7efd1006e6e0 2024-04-17T10:28:19.577 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:19.577 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:19.577 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efcec066f60 msgr2=0x7efcec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efcec066f60 0x7efcec069420 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7efd1013c670 tx=0x7efcf8009040 comp rx=0 tx=0).stop 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd1006e6e0 msgr2=0x7efd1013c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd1006e6e0 0x7efd1013c6c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd1006f0f0 tx=0x7efd0001d3b0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 shutdown_connections 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efcec066f60 0x7efcec069420 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efd101441f0 0x7efd100b1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efd1006eac0 0x7efd10136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 --2- 172.21.15.67:0/3068467533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efd1006e6e0 0x7efd1013c6c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 >> 172.21.15.67:0/3068467533 conn(0x7efd1006ac90 msgr2=0x7efd100aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 shutdown_connections 2024-04-17T10:28:19.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.586+0000 7efd15a00640 1 -- 172.21.15.67:0/3068467533 wait complete. 2024-04-17T10:28:19.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-04-17T10:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:19.741 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:19.757 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:19.757 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' 2024-04-17T10:28:19.757 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:19.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-04-17T10:28:19.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.906+0000 7f2206a00640 1 Processor -- start 2024-04-17T10:28:19.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f2206a00640 1 -- start start 2024-04-17T10:28:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f2206a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 0x7f22000ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f2206a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f220015cb30 con 0x7f2200159440 2024-04-17T10:28:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f2206a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22000cd610 con 0x7f220015be70 2024-04-17T10:28:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f2206a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22000a8650 con 0x7f2200071af0 2024-04-17T10:28:19.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2200159440 0x7f22000713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45202/0 (socket says 172.21.15.67:45202) 2024-04-17T10:28:19.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 -- 172.21.15.67:0/2980949358 learned_addr learned my addr 172.21.15.67:0/2980949358 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379866999 0 0) 0x7f220015cb30 con 0x7f2200159440 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ff400640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 0x7f22000ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21e4003530 con 0x7f2200159440 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 200144222 0 0) 0x7f21e4003530 con 0x7f2200159440 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 msgr2=0x7f22000ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 0x7f22000ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 >> v1:172.21.15.86:6789/0 conn(0x7f220015be70 legacy=0x7f220015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22000cd8f0 con 0x7f2200159440 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f21f0001680 con 0x7f2200159440 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 >> v1:172.21.15.67:6789/0 conn(0x7f2200159440 legacy=0x7f22000713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e40039f0 0x7f21e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 0x7f21e4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e4004d50 0x7f21e40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e400ba80 con 0x7f21e40039f0 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e400bc00 con 0x7f21e4004d50 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e400bdb0 con 0x7f2200071af0 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e40039f0 0x7f21e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ff400640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 0x7f21e4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f2204a00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e4004d50 0x7f21e40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 -- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 msgr2=0x7f21e4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 0x7f21e4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 -- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e4004d50 msgr2=0x7f21e40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e4004d50 0x7f21e40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ffe00640 1 -- 172.21.15.67:0/2980949358 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e4003530 con 0x7f21e40039f0 2024-04-17T10:28:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.910+0000 7f21ff400640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200071af0 0x7f21e4004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e40039f0 0x7f21e4003dd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f21f000c110 tx=0x7f21f001cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4526fccc70a30625 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0007c50 con 0x7f21e40039f0 2024-04-17T10:28:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f00a9020 con 0x7f21e40039f0 2024-04-17T10:28:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f21fea00640 1 -- 172.21.15.67:0/2980949358 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0005b20 con 0x7f21e40039f0 2024-04-17T10:28:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f2206a00640 1 -- 172.21.15.67:0/2980949358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e40039f0 msgr2=0x7f21e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f2206a00640 1 --2- 172.21.15.67:0/2980949358 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21e40039f0 0x7f21e4003dd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f21f000c110 tx=0x7f21f001cbc0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f2204a00640 1 -- 172.21.15.67:0/2980949358 reap_dead start 2024-04-17T10:28:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f2206a00640 1 -- 172.21.15.67:0/2980949358 shutdown_connections 2024-04-17T10:28:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f2206a00640 1 -- 172.21.15.67:0/2980949358 >> 172.21.15.67:0/2980949358 conn(0x7f22000b1740 msgr2=0x7f22000b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f2206a00640 1 -- 172.21.15.67:0/2980949358 shutdown_connections 2024-04-17T10:28:19.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.914+0000 7f2206a00640 1 -- 172.21.15.67:0/2980949358 wait complete. 2024-04-17T10:28:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 Processor -- start 2024-04-17T10:28:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 -- start start 2024-04-17T10:28:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e40039f0 0x7f22000ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200151d70 0x7f22001528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2200152e00 0x7f22001f98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220015dbd0 con 0x7f2200152e00 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220015da50 con 0x7f21e40039f0 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220015dd50 con 0x7f2200151d70 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2200152e00 0x7f22001f98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2200152e00 0x7f22001f98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60546/0 (socket says 172.21.15.67:60546) 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 -- 172.21.15.67:0/4012569113 learned_addr learned my addr 172.21.15.67:0/4012569113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f21ffe00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e40039f0 0x7f22000ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f21ff400640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200151d70 0x7f22001528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 -- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200151d70 msgr2=0x7f22001528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200151d70 0x7f22001528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 -- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e40039f0 msgr2=0x7f22000ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e40039f0 0x7f22000ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 -- 172.21.15.67:0/4012569113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22001fa4e0 con 0x7f2200152e00 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f21ffe00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e40039f0 0x7f22000ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2204a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2200152e00 0x7f22001f98b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f22000b4d50 tx=0x7f21f800feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f21f7e00640 1 -- 172.21.15.67:0/4012569113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8019070 con 0x7f2200152e00 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f21f7e00640 1 -- 172.21.15.67:0/4012569113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f8006d70 con 0x7f2200152e00 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f21f7e00640 1 -- 172.21.15.67:0/4012569113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec0096b0 con 0x7f2200152e00 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22000acb20 con 0x7f2200152e00 2024-04-17T10:28:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.918+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22000ad0e0 con 0x7f2200152e00 2024-04-17T10:28:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.922+0000 7f21f7e00640 1 -- 172.21.15.67:0/4012569113 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f21f8003bc0 con 0x7f2200152e00 2024-04-17T10:28:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.922+0000 7f21f7e00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f21e0066f60 0x7f21e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.922+0000 7f21f7e00640 1 -- 172.21.15.67:0/4012569113 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f21f8084280 con 0x7f2200152e00 2024-04-17T10:28:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.922+0000 7f2209e7d000 1 Processor -- start 2024-04-17T10:28:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.922+0000 7f21ffe00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f21e0066f60 0x7f21e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.922+0000 7f2209e7d000 1 -- start start 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.922+0000 7f21ffe00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f21e0066f60 0x7f21e0069420 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f21f00aa000 tx=0x7f22000cd7e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b55d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b541a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b541f60 0x55e76b542340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76b3fa4a0 con 0x55e76b55db20 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76b362210 con 0x55e76b55d200 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e76b2ea980 con 0x55e76b541f60 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b541a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b55d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2204a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b541f60 0x55e76b542340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b541a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60548/0 (socket says 172.21.15.67:60548) 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b55d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:53012/0 (socket says 172.21.15.67:53012) 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 -- 172.21.15.67:0/1037559667 learned_addr learned my addr 172.21.15.67:0/1037559667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 -- 172.21.15.67:0/1037559667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b541f60 msgr2=0x55e76b542340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 --2- 172.21.15.67:0/1037559667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b541f60 0x55e76b542340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 -- 172.21.15.67:0/1037559667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 msgr2=0x55e76b55d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 --2- 172.21.15.67:0/1037559667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b55d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 -- 172.21.15.67:0/1037559667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e76b35d200 con 0x55e76b55db20 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21ff400640 1 --2- 172.21.15.67:0/1037559667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b541a20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009d70 tx=0x7f21ec014780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b1a584170f1c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21f4c00640 1 -- 172.21.15.67:0/1037559667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec022070 con 0x55e76b55db20 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21f4c00640 1 -- 172.21.15.67:0/1037559667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec015630 con 0x55e76b55db20 2024-04-17T10:28:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f21f4c00640 1 -- 172.21.15.67:0/1037559667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec009e60 con 0x55e76b55db20 2024-04-17T10:28:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- 172.21.15.67:0/1037559667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 msgr2=0x55e76b541a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- 172.21.15.67:0/1037559667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b541a20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009d70 tx=0x7f21ec014780 comp rx=0 tx=0).stop 2024-04-17T10:28:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- 172.21.15.67:0/1037559667 shutdown_connections 2024-04-17T10:28:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- 172.21.15.67:0/1037559667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b541f60 0x55e76b542340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- 172.21.15.67:0/1037559667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b541a20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- 172.21.15.67:0/1037559667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b55d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- 172.21.15.67:0/1037559667 >> 172.21.15.67:0/1037559667 conn(0x55e76b50a4f0 msgr2=0x55e76b519a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- 172.21.15.67:0/1037559667 shutdown_connections 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- 172.21.15.67:0/1037559667 wait complete. 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- start start 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 Processor -- start 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b541f60 0x55e76b564980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b564ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b542340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76b31cd80 con 0x55e76b541f60 2024-04-17T10:28:19.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76b35e2b0 con 0x55e76b55d200 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.926+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e76b323830 con 0x55e76b55db20 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 10 client.0 _ll_register_callbacks cb 0x55e76b415070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b541f60 0x55e76b564980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2204a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b542340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ff400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b564ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b541f60 0x55e76b564980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60556/0 (socket says 172.21.15.67:60556) 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 -- 172.21.15.67:0/3703337836 learned_addr learned my addr 172.21.15.67:0/3703337836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 -- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 msgr2=0x55e76b542340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b542340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 -- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 msgr2=0x55e76b564ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b564ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 -- 172.21.15.67:0/3703337836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e76b32e130 con 0x55e76b541f60 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ffe00640 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b541f60 0x55e76b564980 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f21f001cc00 tx=0x7f21f001b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9004d8829c33bc7e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 1 -- 172.21.15.67:0/3703337836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f00ca060 con 0x55e76b541f60 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 1 -- 172.21.15.67:0/3703337836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f21f0004d80 con 0x55e76b541f60 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21ec015630 con 0x55e76b541f60 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 1 -- 172.21.15.67:0/3703337836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21f00d0050 con 0x55e76b541f60 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 10 client.6033 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e76b3e3490 con 0x55e76b541f60 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 1 -- 172.21.15.67:0/3703337836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec015630 con 0x55e76b541f60 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 1 -- 172.21.15.67:0/3703337836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f21f00d3020 con 0x55e76b541f60 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 10 client.6033 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 10 client.6033 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:19.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 20 client.6033 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:19.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5430b0 0x55e76b5434c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 4 client.6033 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:19.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e76b2fe020 con 0x55e76b5430b0 2024-04-17T10:28:19.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ff400640 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5430b0 0x55e76b5434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21ff400640 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5430b0 0x55e76b5434c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=276eebb3a8ada822 server_cookie=d279852086438547 in_seq=0 out_seq=0 2024-04-17T10:28:19.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 10 client.6033 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 1 -- 172.21.15.67:0/3703337836 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e76b2fe020 con 0x55e76b5430b0 2024-04-17T10:28:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f21e9600640 10 client.6033 handle_command_reply: tid=0 2024-04-17T10:28:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 1 client.6033 shutdown 2024-04-17T10:28:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b541f60 msgr2=0x55e76b564980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.930+0000 7f2209e7d000 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b541f60 0x55e76b564980 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f21f001cc00 tx=0x7f21f001b400 comp rx=0 tx=0).stop 2024-04-17T10:28:19.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 shutdown_connections 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5430b0 0x55e76b5434c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b542340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b55d200 0x55e76b564ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f21e9600640 0 client.6033 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 --2- 172.21.15.67:0/3703337836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b541f60 0x55e76b564980 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 >> 172.21.15.67:0/3703337836 conn(0x55e76b50a4f0 msgr2=0x55e76b511aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 shutdown_connections 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 1 -- 172.21.15.67:0/3703337836 wait complete. 2024-04-17T10:28:19.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.934+0000 7f2209e7d000 20 client.6033 trim_cache size 0 max 16384 2024-04-17T10:28:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.946+0000 7f2209e7d000 1 Processor -- start 2024-04-17T10:28:19.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.946+0000 7f2209e7d000 1 -- start start 2024-04-17T10:28:19.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b517e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55d200 0x55e76b518370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b5188b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22002039d0 con 0x55e76b55d200 2024-04-17T10:28:19.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2200202c00 con 0x55e76b541f60 2024-04-17T10:28:19.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22001f6e10 con 0x55e76b55db20 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55d200 0x55e76b518370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55d200 0x55e76b518370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60560/0 (socket says 172.21.15.67:60560) 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 -- 172.21.15.67:0/2650435466 learned_addr learned my addr 172.21.15.67:0/2650435466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b517e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2204a00640 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b5188b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 -- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 msgr2=0x55e76b5188b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b5188b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 -- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 msgr2=0x55e76b517e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b517e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 -- 172.21.15.67:0/2650435466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e76b321a30 con 0x55e76b55d200 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ff400640 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55d200 0x55e76b518370 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f21ec000c10 tx=0x7f21ec017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c92cb75608e5c9d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ebe00640 1 -- 172.21.15.67:0/2650435466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec022040 con 0x55e76b55d200 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ebe00640 1 -- 172.21.15.67:0/2650435466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec023040 con 0x55e76b55d200 2024-04-17T10:28:19.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ebe00640 1 -- 172.21.15.67:0/2650435466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ec02e110 con 0x55e76b55d200 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- 172.21.15.67:0/2650435466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55d200 msgr2=0x55e76b518370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55d200 0x55e76b518370 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f21ec000c10 tx=0x7f21ec017000 comp rx=0 tx=0).stop 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- 172.21.15.67:0/2650435466 shutdown_connections 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55db20 0x55e76b5188b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55d200 0x55e76b518370 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2650435466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b517e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- 172.21.15.67:0/2650435466 >> 172.21.15.67:0/2650435466 conn(0x55e76b50a4f0 msgr2=0x55e76b5406f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- 172.21.15.67:0/2650435466 shutdown_connections 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- 172.21.15.67:0/2650435466 wait complete. 2024-04-17T10:28:19.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- start start 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 Processor -- start 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b57b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55d200 0x55e76b57b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b512260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76b2ee190 con 0x55e76b55db20 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2200202600 con 0x55e76b541f60 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e76b308c30 con 0x55e76b55d200 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f2209e7d000 10 client.0 _ll_register_callbacks cb 0x7f21ec009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b57b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.950+0000 7f21ffe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b57b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:53034/0 (socket says 172.21.15.67:53034) 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21ffe00640 1 -- 172.21.15.67:0/2050494232 learned_addr learned my addr 172.21.15.67:0/2050494232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21ff400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55d200 0x55e76b57b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2204a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b512260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2204a00640 1 -- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55d200 msgr2=0x55e76b57b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2204a00640 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55d200 0x55e76b57b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2204a00640 1 -- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 msgr2=0x55e76b57b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2204a00640 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b57b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2204a00640 1 -- 172.21.15.67:0/2050494232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e76b32a040 con 0x55e76b55db20 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b57b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2204a00640 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b512260 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f21f8082670 tx=0x7f21f8055080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44cebc0f2dff5683 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 1 -- 172.21.15.67:0/2050494232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f806c420 con 0x55e76b55db20 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 1 -- 172.21.15.67:0/2050494232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f21f8014c30 con 0x55e76b55db20 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21ec023040 con 0x55e76b55db20 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 1 -- 172.21.15.67:0/2050494232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21f805d090 con 0x55e76b55db20 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 10 client.6039 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:19.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e76b01e900 con 0x55e76b55db20 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 1 -- 172.21.15.67:0/2050494232 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec023040 con 0x55e76b55db20 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 1 -- 172.21.15.67:0/2050494232 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f21f8071200 con 0x55e76b55db20 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 10 client.6039 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 10 client.6039 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 20 client.6039 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5130e0 0x55e76b5134f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 4 client.6039 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}] 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}) v1 -- 0x55e76b36f750 con 0x55e76b5130e0 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5130e0 0x55e76b5134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21ffe00640 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5130e0 0x55e76b5134f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fed80a6937cad68a server_cookie=eb4eb8944d38da48 in_seq=0 out_seq=0 2024-04-17T10:28:19.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 10 client.6039 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 1 -- 172.21.15.67:0/2050494232 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e76b36f750 con 0x55e76b5130e0 2024-04-17T10:28:19.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 10 client.6039 handle_command_reply: tid=0 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 client.6039 shutdown 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 msgr2=0x55e76b512260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b512260 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f21f8082670 tx=0x7f21f8055080 comp rx=0 tx=0).stop 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 shutdown_connections 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76b5130e0 0x55e76b5134f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76b55db20 0x55e76b512260 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76b55d200 0x55e76b57b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 --2- 172.21.15.67:0/2050494232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76b541f60 0x55e76b57b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 >> 172.21.15.67:0/2050494232 conn(0x55e76b50a4f0 msgr2=0x55e76b514c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f21e9600640 0 client.6039 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 shutdown_connections 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 1 -- 172.21.15.67:0/2050494232 wait complete. 2024-04-17T10:28:19.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2209e7d000 20 client.6039 trim_cache size 0 max 16384 2024-04-17T10:28:19.954 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:19.954 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:19.954 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "7d425419-3fe6-46d1-a3c5-f2fe461982fc", 2024-04-17T10:28:19.954 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:19.954 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f21e0066f60 msgr2=0x7f21e0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2206a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f21e0066f60 0x7f21e0069420 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f21f00aa000 tx=0x7f22000cd7e0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2200152e00 msgr2=0x7f22001f98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.954+0000 7f2206a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2200152e00 0x7f22001f98b0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f22000b4d50 tx=0x7f21f800feb0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 shutdown_connections 2024-04-17T10:28:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f21e0066f60 0x7f21e0069420 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2200152e00 0x7f22001f98b0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2200151d70 0x7f22001528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 --2- 172.21.15.67:0/4012569113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f21e40039f0 0x7f22000ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:19.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 >> 172.21.15.67:0/4012569113 conn(0x7f22000b1740 msgr2=0x7f22000b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:19.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 shutdown_connections 2024-04-17T10:28:19.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:19.958+0000 7f2206a00640 1 -- 172.21.15.67:0/4012569113 wait complete. 2024-04-17T10:28:19.968 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' returned '{'return_code': 0, 'scrub_tag': '7d425419-3fe6-46d1-a3c5-f2fe461982fc', 'mode': 'asynchronous'}' 2024-04-17T10:28:19.969 INFO:tasks.cephfs.test_scrub_checks:First run: flushing base dir / 2024-04-17T10:28:19.969 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /' 2024-04-17T10:28:19.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:20.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26e5000640 1 Processor -- start 2024-04-17T10:28:20.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26e5000640 1 -- start start 2024-04-17T10:28:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26e5000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26e01446c0 0x7f26e0144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26e5000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e00b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26e5000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e01456d0 con 0x7f26e0144aa0 2024-04-17T10:28:20.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26e5000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e00b51a0 con 0x7f26e01446c0 2024-04-17T10:28:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26e5000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00b4870 con 0x7f26e0071840 2024-04-17T10:28:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26dde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26e01446c0 0x7f26e0144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26d5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f26e0144aa0 0x7f26e00d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:45212/0 (socket says 172.21.15.67:45212) 2024-04-17T10:28:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26d5400640 1 -- 172.21.15.67:0/764338687 learned_addr learned my addr 172.21.15.67:0/764338687 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26de800640 1 --2- 172.21.15.67:0/764338687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e00b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26dde00640 1 -- 172.21.15.67:0/764338687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 msgr2=0x7f26e00b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26dde00640 1 --2- 172.21.15.67:0/764338687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e00b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26dde00640 1 -- 172.21.15.67:0/764338687 >> v1:172.21.15.67:6789/0 conn(0x7f26e0144aa0 legacy=0x7f26e00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26dde00640 1 -- 172.21.15.67:0/764338687 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e00b4b50 con 0x7f26e01446c0 2024-04-17T10:28:20.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.118+0000 7f26dd400640 1 -- 172.21.15.67:0/764338687 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140509631 0 0) 0x7f26e01456d0 con 0x7f26e0144aa0 2024-04-17T10:28:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26dde00640 1 --2- 172.21.15.67:0/764338687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e0144ea0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f26d00097c0 tx=0x7f26d001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c66811349b7f4b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26dd400640 1 -- 172.21.15.67:0/764338687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d00b2070 con 0x7f26e01446c0 2024-04-17T10:28:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26dd400640 1 -- 172.21.15.67:0/764338687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26d00ad680 con 0x7f26e01446c0 2024-04-17T10:28:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26dd400640 1 -- 172.21.15.67:0/764338687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d00ada50 con 0x7f26e01446c0 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 -- 172.21.15.67:0/764338687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 msgr2=0x7f26e0144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 --2- 172.21.15.67:0/764338687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e0144ea0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f26d00097c0 tx=0x7f26d001c470 comp rx=0 tx=0).stop 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 -- 172.21.15.67:0/764338687 shutdown_connections 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 --2- 172.21.15.67:0/764338687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e00b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 --2- 172.21.15.67:0/764338687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e0144ea0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 -- 172.21.15.67:0/764338687 >> 172.21.15.67:0/764338687 conn(0x7f26e00b34d0 msgr2=0x7f26e00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 -- 172.21.15.67:0/764338687 shutdown_connections 2024-04-17T10:28:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.122+0000 7f26e5000640 1 -- 172.21.15.67:0/764338687 wait complete. 2024-04-17T10:28:20.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 Processor -- start 2024-04-17T10:28:20.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 -- start start 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e015ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e013ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26e0144aa0 0x7f26e013ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00b6500 con 0x7f26e01446c0 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00b6380 con 0x7f26e0144aa0 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00b6680 con 0x7f26e0071840 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e013ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e013ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60588/0 (socket says 172.21.15.67:60588) 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 -- 172.21.15.67:0/3269938615 learned_addr learned my addr 172.21.15.67:0/3269938615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26de800640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26e0144aa0 0x7f26e013ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26dde00640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e015ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 -- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 msgr2=0x7f26e015ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e015ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 -- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26e0144aa0 msgr2=0x7f26e013ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26e0144aa0 0x7f26e013ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 -- 172.21.15.67:0/3269938615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e015f1a0 con 0x7f26e01446c0 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d5400640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e013ced0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f26c8005890 tx=0x7f26c800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d6a00640 1 -- 172.21.15.67:0/3269938615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8017070 con 0x7f26e01446c0 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d6a00640 1 -- 172.21.15.67:0/3269938615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c8005d10 con 0x7f26e01446c0 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e013d740 con 0x7f26e01446c0 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26d6a00640 1 -- 172.21.15.67:0/3269938615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8007310 con 0x7f26e01446c0 2024-04-17T10:28:20.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e013dd00 con 0x7f26e01446c0 2024-04-17T10:28:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.126+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e0070bb0 con 0x7f26e01446c0 2024-04-17T10:28:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.130+0000 7f26d6a00640 1 -- 172.21.15.67:0/3269938615 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f26c801a070 con 0x7f26e01446c0 2024-04-17T10:28:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.130+0000 7f26d6a00640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26bc066f90 0x7f26bc069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.130+0000 7f26d6a00640 1 -- 172.21.15.67:0/3269938615 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f26c80844c0 con 0x7f26e01446c0 2024-04-17T10:28:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.130+0000 7f26dde00640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26bc066f90 0x7f26bc069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.130+0000 7f26dde00640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26bc066f90 0x7f26bc069450 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f26d00af000 tx=0x7f26d00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.134+0000 7f26d6a00640 1 -- 172.21.15.67:0/3269938615 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f26c8054cf0 con 0x7f26e01446c0 2024-04-17T10:28:20.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.330+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e0073920 con 0x7f26e01446c0 2024-04-17T10:28:20.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.330+0000 7f26d6a00640 1 -- 172.21.15.67:0/3269938615 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f26c8058d90 con 0x7f26e01446c0 2024-04-17T10:28:20.325 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:20.325 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:20.325 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26bc066f90 msgr2=0x7f26bc069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26bc066f90 0x7f26bc069450 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f26d00af000 tx=0x7f26d00a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 msgr2=0x7f26e013ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e013ced0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f26c8005890 tx=0x7f26c800c8e0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 shutdown_connections 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26bc066f90 0x7f26bc069450 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26e0144aa0 0x7f26e013ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26e01446c0 0x7f26e013ced0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 --2- 172.21.15.67:0/3269938615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26e0071840 0x7f26e015ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 >> 172.21.15.67:0/3269938615 conn(0x7f26e00b34d0 msgr2=0x7f26e0163a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 shutdown_connections 2024-04-17T10:28:20.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.334+0000 7f26e5000640 1 -- 172.21.15.67:0/3269938615 wait complete. 2024-04-17T10:28:20.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path / 2024-04-17T10:28:20.489 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:20.489 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:20.489 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:20.504 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:20.504 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /' 2024-04-17T10:28:20.504 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:20.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / 2024-04-17T10:28:20.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.654+0000 7fb4fca00640 1 Processor -- start 2024-04-17T10:28:20.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.654+0000 7fb4fca00640 1 -- start start 2024-04-17T10:28:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4fca00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb4f8071af0 0x7fb4f80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4fca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4fca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f815cb30 con 0x7fb4f8159440 2024-04-17T10:28:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4fca00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f80a8650 con 0x7fb4f8071af0 2024-04-17T10:28:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4fca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815ce10 con 0x7fb4f815be70 2024-04-17T10:28:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4f5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb4f8159440 0x7fb4f80713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59604/0 (socket says 172.21.15.67:59604) 2024-04-17T10:28:20.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4f5e00640 1 -- 172.21.15.67:0/1758900465 learned_addr learned my addr 172.21.15.67:0/1758900465 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:20.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1758900465 >> v2:172.21.15.86:3300/0 conn(0x7fb4f8071af0 0x7fb4f80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4f6800640 1 --2- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4f5400640 1 -- 172.21.15.67:0/1758900465 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810194337 0 0) 0x7fb4f815cb30 con 0x7fb4f8159440 2024-04-17T10:28:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4f5400640 1 -- 172.21.15.67:0/1758900465 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4dc003530 con 0x7fb4f8159440 2024-04-17T10:28:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4ed400640 1 -- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 msgr2=0x7fb4f815c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f815c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4ed400640 1 -- 172.21.15.67:0/1758900465 >> v1:172.21.15.67:6789/0 conn(0x7fb4f8159440 legacy=0x7fb4f80713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.658+0000 7fb4ed400640 1 -- 172.21.15.67:0/1758900465 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f80cd610 con 0x7fb4f8071af0 2024-04-17T10:28:20.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 0x7fb4f80ce0f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00097c0 tx=0x7fb4e001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754822efd0bca9ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4f5400640 1 -- 172.21.15.67:0/1758900465 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00aa600 con 0x7fb4f8071af0 2024-04-17T10:28:20.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4f5400640 1 -- 172.21.15.67:0/1758900465 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e00af070 con 0x7fb4f8071af0 2024-04-17T10:28:20.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4f5400640 1 -- 172.21.15.67:0/1758900465 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00aa9c0 con 0x7fb4f8071af0 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 -- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 msgr2=0x7fb4f80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 --2- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 0x7fb4f80ce0f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00097c0 tx=0x7fb4e001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 -- 172.21.15.67:0/1758900465 shutdown_connections 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 --2- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f815c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 --2- 172.21.15.67:0/1758900465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 0x7fb4f80ce0f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 -- 172.21.15.67:0/1758900465 >> 172.21.15.67:0/1758900465 conn(0x7fb4f80b1740 msgr2=0x7fb4f80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 -- 172.21.15.67:0/1758900465 shutdown_connections 2024-04-17T10:28:20.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 -- 172.21.15.67:0/1758900465 wait complete. 2024-04-17T10:28:20.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 Processor -- start 2024-04-17T10:28:20.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.662+0000 7fb4fca00640 1 -- start start 2024-04-17T10:28:20.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 0x7fb4f80af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4f8159440 0x7fb4f814bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f8156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815d840 con 0x7fb4f8159440 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815c800 con 0x7fb4f8071af0 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f815d9c0 con 0x7fb4f815be70 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4f8159440 0x7fb4f814bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4f8159440 0x7fb4f814bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59864/0 (socket says 172.21.15.67:59864) 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 -- 172.21.15.67:0/496581828 learned_addr learned my addr 172.21.15.67:0/496581828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 0x7fb4f80af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4f6800640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f8156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 -- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 msgr2=0x7fb4f8156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f8156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 -- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 msgr2=0x7fb4f80af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 0x7fb4f80af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 -- 172.21.15.67:0/496581828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f814ce30 con 0x7fb4f8159440 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4ed400640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4f8159440 0x7fb4f814bb10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00ac040 tx=0x7fb4e001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4eea00640 1 -- 172.21.15.67:0/496581828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00be410 con 0x7fb4f8159440 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4eea00640 1 -- 172.21.15.67:0/496581828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e00be5b0 con 0x7fb4f8159440 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4eea00640 1 -- 172.21.15.67:0/496581828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00be9a0 con 0x7fb4f8159440 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f814c590 con 0x7fb4f8159440 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f814cb50 con 0x7fb4f8159440 2024-04-17T10:28:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.666+0000 7fb4eea00640 1 -- 172.21.15.67:0/496581828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb4e00bc020 con 0x7fb4f8159440 2024-04-17T10:28:20.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4eea00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4d4066f60 0x7fb4d4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4eea00640 1 -- 172.21.15.67:0/496581828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4e011f630 con 0x7fb4f8159440 2024-04-17T10:28:20.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4d4066f60 0x7fb4d4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 Processor -- start 2024-04-17T10:28:20.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 -- start start 2024-04-17T10:28:20.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4d4066f60 0x7fb4d4069420 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007fd0 tx=0x7fb4e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d1a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1cd02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1cd0820 0x55eee1cd0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee1b19f40 con 0x55eee1cd0820 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee1aa1fc0 con 0x55eee1d1ad80 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eee1b15fe0 con 0x55eee1d1a460 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1cd0820 0x55eee1cd0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1cd0820 0x55eee1cd0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59874/0 (socket says 172.21.15.67:59874) 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f5e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d1a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 -- 172.21.15.67:0/1228739971 learned_addr learned my addr 172.21.15.67:0/1228739971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1cd02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 -- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 msgr2=0x55eee1d1a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d1a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 -- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 msgr2=0x55eee1cd02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1cd02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 -- 172.21.15.67:0/1228739971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eee1b61a50 con 0x55eee1cd0820 2024-04-17T10:28:20.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d1a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:20.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4f6800640 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1cd0820 0x55eee1cd0c00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0011550 tx=0x7fb4f0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1148e3bd4b9d275 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4e6a00640 1 -- 172.21.15.67:0/1228739971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0015840 con 0x55eee1cd0820 2024-04-17T10:28:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4e6a00640 1 -- 172.21.15.67:0/1228739971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f00159e0 con 0x55eee1cd0820 2024-04-17T10:28:20.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4e6a00640 1 -- 172.21.15.67:0/1228739971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0015d90 con 0x55eee1cd0820 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1228739971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1cd0820 msgr2=0x55eee1cd0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.670+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1cd0820 0x55eee1cd0c00 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0011550 tx=0x7fb4f0009750 comp rx=0 tx=0).stop 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1228739971 shutdown_connections 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1cd0820 0x55eee1cd0c00 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1cd02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1228739971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d1a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1228739971 >> 172.21.15.67:0/1228739971 conn(0x55eee1cc83c0 msgr2=0x55eee1cca7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1228739971 shutdown_connections 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1228739971 wait complete. 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- start start 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 Processor -- start 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d08a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d08f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1ad80 0x55eee1d094b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee1b1e570 con 0x55eee1d1a460 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee1ad45f0 con 0x55eee1cd0820 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eee1ac5310 con 0x55eee1d1ad80 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 10 client.0 _ll_register_callbacks cb 0x55eee1bce7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d08f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d08f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59888/0 (socket says 172.21.15.67:59888) 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 -- 172.21.15.67:0/2512899778 learned_addr learned my addr 172.21.15.67:0/2512899778 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4f6800640 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1ad80 0x55eee1d094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d08a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 -- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1ad80 msgr2=0x55eee1d094b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1ad80 0x55eee1d094b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 -- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 msgr2=0x55eee1d08a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d08a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 -- 172.21.15.67:0/2512899778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eee1aa2210 con 0x55eee1d1a460 2024-04-17T10:28:20.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4ed400640 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d08f70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00efb50 tx=0x7fb4e00ef990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf04cfbeade9686 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4db400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4db400640 1 -- 172.21.15.67:0/2512899778 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00f3ef0 con 0x55eee1d1a460 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4db400640 1 -- 172.21.15.67:0/2512899778 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4e00beb40 con 0x55eee1d1a460 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4f00159e0 con 0x55eee1d1a460 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4db400640 1 -- 172.21.15.67:0/2512899778 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4e00f84b0 con 0x55eee1d1a460 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 10 client.6060 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55eee1b9e400 con 0x55eee1d1a460 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4db400640 1 -- 172.21.15.67:0/2512899778 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f00159e0 con 0x55eee1d1a460 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4db400640 1 -- 172.21.15.67:0/2512899778 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb4e00c6030 con 0x55eee1d1a460 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 10 client.6060 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:20.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 10 client.6060 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 20 client.6060 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d2a710 0x55eee1d2cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 4 client.6060 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eee1ab5c90 con 0x55eee1d2a710 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.674+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d2a710 0x55eee1d2cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d2a710 0x55eee1d2cb00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c6d0a4a23c98842 server_cookie=9148ced9c0e7b996 in_seq=0 out_seq=0 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4db400640 10 client.6060 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4db400640 1 -- 172.21.15.67:0/2512899778 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55eee1ab5c90 con 0x55eee1d2a710 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4db400640 10 client.6060 handle_command_reply: tid=0 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 client.6060 shutdown 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 msgr2=0x55eee1d08f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d08f70 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00efb50 tx=0x7fb4e00ef990 comp rx=0 tx=0).stop 2024-04-17T10:28:20.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 shutdown_connections 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d2a710 0x55eee1d2cb00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1ad80 0x55eee1d094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d08f70 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/2512899778 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d08a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4db400640 0 client.6060 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 >> 172.21.15.67:0/2512899778 conn(0x55eee1cc83c0 msgr2=0x55eee1cd1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 shutdown_connections 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 1 -- 172.21.15.67:0/2512899778 wait complete. 2024-04-17T10:28:20.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.678+0000 7fb4fff0a000 20 client.6060 trim_cache size 0 max 16384 2024-04-17T10:28:20.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 Processor -- start 2024-04-17T10:28:20.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- start start 2024-04-17T10:28:20.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1cd0820 0x55eee1d2eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d2f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1d06b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f81fecb0 con 0x55eee1d1a460 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f80b3de0 con 0x55eee1d1ad80 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f8153280 con 0x55eee1cd0820 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d2f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d2f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59890/0 (socket says 172.21.15.67:59890) 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 -- 172.21.15.67:0/1763176328 learned_addr learned my addr 172.21.15.67:0/1763176328 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4f6800640 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1d06b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1cd0820 0x55eee1d2eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 -- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1cd0820 msgr2=0x55eee1d2eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1cd0820 0x55eee1d2eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 -- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 msgr2=0x55eee1d06b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1d06b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 -- 172.21.15.67:0/1763176328 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eee1b74c10 con 0x55eee1d1a460 2024-04-17T10:28:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d2f060 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00efde0 tx=0x7fb4e0104000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd78cf8ab72340c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4e6000640 1 -- 172.21.15.67:0/1763176328 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00ef7b0 con 0x55eee1d1a460 2024-04-17T10:28:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4e6000640 1 -- 172.21.15.67:0/1763176328 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e00af040 con 0x55eee1d1a460 2024-04-17T10:28:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4e6000640 1 -- 172.21.15.67:0/1763176328 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00f3ef0 con 0x55eee1d1a460 2024-04-17T10:28:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1763176328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 msgr2=0x55eee1d2f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d2f060 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00efde0 tx=0x7fb4e0104000 comp rx=0 tx=0).stop 2024-04-17T10:28:20.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1763176328 shutdown_connections 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1d1ad80 0x55eee1d06b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1a460 0x55eee1d2f060 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1763176328 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1cd0820 0x55eee1d2eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1763176328 >> 172.21.15.67:0/1763176328 conn(0x55eee1cc83c0 msgr2=0x55eee1cd7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1763176328 shutdown_connections 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.694+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1763176328 wait complete. 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 -- start start 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 Processor -- start 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d2df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d2e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1ad80 0x55eee1d2edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eee1ad2ea0 con 0x55eee1d1ad80 2024-04-17T10:28:20.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f8153880 con 0x55eee1cd0820 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eee1b1e570 con 0x55eee1d1a460 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 10 client.0 _ll_register_callbacks cb 0x55eee1b6c2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4f6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1ad80 0x55eee1d2edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d2e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4f5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d2df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d2e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:60366/0 (socket says 172.21.15.67:60366) 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 -- 172.21.15.67:0/1731570864 learned_addr learned my addr 172.21.15.67:0/1731570864 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4f5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d2df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:59438/0 (socket says 172.21.15.67:59438) 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 -- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 msgr2=0x55eee1d2df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d2df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 -- 172.21.15.67:0/1731570864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1ad80 msgr2=0x55eee1d2edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1ad80 0x55eee1d2edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 -- 172.21.15.67:0/1731570864 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eee1aaebb0 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d2df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4f6800640 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1ad80 0x55eee1d2edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4ed400640 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d2e890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0000c10 tx=0x7fb4e00c0070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49045181d0f6256d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4db400640 10 client.0 ms_handle_connect on v2:172.21.15.86:3301/0 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4db400640 1 -- 172.21.15.67:0/1731570864 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e00ef7b0 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4db400640 1 -- 172.21.15.67:0/1731570864 <== mon.2 v2:172.21.15.86:3301/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb4e0107b30 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4e00af040 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4db400640 1 -- 172.21.15.67:0/1731570864 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4e00f3ef0 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 10 client.5474 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55eee1ac1030 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4db400640 1 -- 172.21.15.67:0/1731570864 <== mon.2 v2:172.21.15.86:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e00af040 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4db400640 1 -- 172.21.15.67:0/1731570864 <== mon.2 v2:172.21.15.86:3301/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb4e00c6040 con 0x55eee1d1a460 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 10 client.5474 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:20.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 10 client.5474 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 20 client.5474 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d38590 0x55eee1d38970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 4 client.5474 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/"}] 2024-04-17T10:28:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/"}) v1 -- 0x55eee1b27480 con 0x55eee1d38590 2024-04-17T10:28:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d38590 0x55eee1d38970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4f5e00640 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d38590 0x55eee1d38970 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed563a25bab1c4ec server_cookie=afd1c12b287d98f7 in_seq=0 out_seq=0 2024-04-17T10:28:20.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.698+0000 7fb4db400640 10 client.5474 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4db400640 1 -- 172.21.15.67:0/1731570864 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55eee1b27480 con 0x55eee1d38590 2024-04-17T10:28:20.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4db400640 10 client.5474 handle_command_reply: tid=0 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 client.5474 shutdown 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 msgr2=0x55eee1d2e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d2e890 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0000c10 tx=0x7fb4e00c0070 comp rx=0 tx=0).stop 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 shutdown_connections 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55eee1d38590 0x55eee1d38970 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55eee1d1ad80 0x55eee1d2edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55eee1d1a460 0x55eee1d2e890 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 --2- 172.21.15.67:0/1731570864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55eee1cd0820 0x55eee1d2df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4db400640 0 client.5474 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 >> 172.21.15.67:0/1731570864 conn(0x55eee1cc83c0 msgr2=0x55eee1cd0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 shutdown_connections 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 1 -- 172.21.15.67:0/1731570864 wait complete. 2024-04-17T10:28:20.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fff0a000 20 client.5474 trim_cache size 0 max 16384 2024-04-17T10:28:20.700 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:20.700 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:20.700 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "29ede0af-daf1-493e-a1e1-cf6674c35e2f", 2024-04-17T10:28:20.700 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:20.700 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4d4066f60 msgr2=0x7fb4d4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4d4066f60 0x7fb4d4069420 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8007fd0 tx=0x7fb4e8009040 comp rx=0 tx=0).stop 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4f8159440 msgr2=0x7fb4f814bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4f8159440 0x7fb4f814bb10 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fb4e00ac040 tx=0x7fb4e001ace0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 shutdown_connections 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4d4066f60 0x7fb4d4069420 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb4f815be70 0x7fb4f8156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb4f8159440 0x7fb4f814bb10 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 --2- 172.21.15.67:0/496581828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb4f8071af0 0x7fb4f80af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 >> 172.21.15.67:0/496581828 conn(0x7fb4f80b1740 msgr2=0x7fb4f80713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 shutdown_connections 2024-04-17T10:28:20.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.702+0000 7fb4fca00640 1 -- 172.21.15.67:0/496581828 wait complete. 2024-04-17T10:28:20.715 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /' returned '{'return_code': 0, 'scrub_tag': '29ede0af-daf1-493e-a1e1-cf6674c35e2f', 'mode': 'asynchronous'}' 2024-04-17T10:28:20.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:20.715 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_dir_0) 2024-04-17T10:28:20.727 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_0' 2024-04-17T10:28:20.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:20.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96fe00640 1 Processor -- start 2024-04-17T10:28:20.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96fe00640 1 -- start start 2024-04-17T10:28:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa97006eeb0 0x7fa970070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9700cd080 con 0x7fa970144af0 2024-04-17T10:28:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96fe00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa970145340 con 0x7fa97006b3a0 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa970144ed0 con 0x7fa97006eeb0 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa970144af0 0x7fa97006ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59612/0 (socket says 172.21.15.67:59612) 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96ea00640 1 -- 172.21.15.67:0/717012276 learned_addr learned my addr 172.21.15.67:0/717012276 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96f400640 1 --2- 172.21.15.67:0/717012276 >> v2:172.21.15.67:3300/0 conn(0x7fa97006eeb0 0x7fa970070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96e000640 1 -- 172.21.15.67:0/717012276 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042255909 0 0) 0x7fa9700cd080 con 0x7fa970144af0 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96e000640 1 -- 172.21.15.67:0/717012276 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa954003530 con 0x7fa970144af0 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96f400640 1 -- 172.21.15.67:0/717012276 >> v1:172.21.15.86:6789/0 conn(0x7fa97006b3a0 legacy=0x7fa9700cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96f400640 1 -- 172.21.15.67:0/717012276 >> v1:172.21.15.67:6789/0 conn(0x7fa970144af0 legacy=0x7fa97006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.918+0000 7fa96f400640 1 -- 172.21.15.67:0/717012276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa970070b60 con 0x7fa97006eeb0 2024-04-17T10:28:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96f400640 1 --2- 172.21.15.67:0/717012276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa97006eeb0 0x7fa970070540 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa970145050 tx=0x7fa96801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa05a53a89316e00 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96e000640 1 -- 172.21.15.67:0/717012276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680b2070 con 0x7fa97006eeb0 2024-04-17T10:28:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96e000640 1 -- 172.21.15.67:0/717012276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9680a5900 con 0x7fa97006eeb0 2024-04-17T10:28:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96e000640 1 -- 172.21.15.67:0/717012276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680a5cb0 con 0x7fa97006eeb0 2024-04-17T10:28:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96fe00640 1 -- 172.21.15.67:0/717012276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa97006eeb0 msgr2=0x7fa970070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96fe00640 1 --2- 172.21.15.67:0/717012276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa97006eeb0 0x7fa970070540 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fa970145050 tx=0x7fa96801c950 comp rx=0 tx=0).stop 2024-04-17T10:28:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96fe00640 1 -- 172.21.15.67:0/717012276 shutdown_connections 2024-04-17T10:28:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96fe00640 1 --2- 172.21.15.67:0/717012276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa97006eeb0 0x7fa970070540 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96fe00640 1 -- 172.21.15.67:0/717012276 >> 172.21.15.67:0/717012276 conn(0x7fa97006e390 msgr2=0x7fa97006e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96fe00640 1 -- 172.21.15.67:0/717012276 shutdown_connections 2024-04-17T10:28:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.922+0000 7fa96fe00640 1 -- 172.21.15.67:0/717012276 wait complete. 2024-04-17T10:28:20.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 Processor -- start 2024-04-17T10:28:20.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 -- start start 2024-04-17T10:28:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa97006b3a0 0x7fa9700c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa97006eeb0 0x7fa9700cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa970144af0 0x7fa9700cb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700cf150 con 0x7fa970144af0 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9700cefd0 con 0x7fa97006eeb0 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9701694d0 con 0x7fa97006b3a0 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa970144af0 0x7fa9700cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa970144af0 0x7fa9700cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59916/0 (socket says 172.21.15.67:59916) 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 -- 172.21.15.67:0/879603570 learned_addr learned my addr 172.21.15.67:0/879603570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96ea00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa97006b3a0 0x7fa9700c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa967e00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa97006eeb0 0x7fa9700cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 -- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa97006b3a0 msgr2=0x7fa9700c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa97006b3a0 0x7fa9700c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 -- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa97006eeb0 msgr2=0x7fa9700cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa97006eeb0 0x7fa9700cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 -- 172.21.15.67:0/879603570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9700cc920 con 0x7fa970144af0 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa967e00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa97006eeb0 0x7fa9700cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96ea00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa97006b3a0 0x7fa9700c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96f400640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa970144af0 0x7fa9700cb220 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa968019560 tx=0x7fa9680196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa966a00640 1 -- 172.21.15.67:0/879603570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680bd020 con 0x7fa970144af0 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa966a00640 1 -- 172.21.15.67:0/879603570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9680b3040 con 0x7fa970144af0 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa966a00640 1 -- 172.21.15.67:0/879603570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9680b2050 con 0x7fa970144af0 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9700cb880 con 0x7fa970144af0 2024-04-17T10:28:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.926+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9700cbe40 con 0x7fa970144af0 2024-04-17T10:28:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.930+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa970074dc0 con 0x7fa970144af0 2024-04-17T10:28:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.930+0000 7fa966a00640 1 -- 172.21.15.67:0/879603570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa9680a5780 con 0x7fa970144af0 2024-04-17T10:28:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.930+0000 7fa966a00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa94c066f60 0x7fa94c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.930+0000 7fa966a00640 1 -- 172.21.15.67:0/879603570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa96811fe90 con 0x7fa970144af0 2024-04-17T10:28:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.930+0000 7fa96ea00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa94c066f60 0x7fa94c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:20.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.930+0000 7fa96ea00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa94c066f60 0x7fa94c069420 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa960005890 tx=0x7fa960009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:20.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:20.934+0000 7fa966a00640 1 -- 172.21.15.67:0/879603570 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa9680c7020 con 0x7fa970144af0 2024-04-17T10:28:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9700b3e20 con 0x7fa970144af0 2024-04-17T10:28:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa966a00640 1 -- 172.21.15.67:0/879603570 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7fa968016540 con 0x7fa970144af0 2024-04-17T10:28:21.127 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:21.127 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:21.127 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa94c066f60 msgr2=0x7fa94c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa94c066f60 0x7fa94c069420 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa960005890 tx=0x7fa960009040 comp rx=0 tx=0).stop 2024-04-17T10:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa970144af0 msgr2=0x7fa9700cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa970144af0 0x7fa9700cb220 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fa968019560 tx=0x7fa9680196f0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 shutdown_connections 2024-04-17T10:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa94c066f60 0x7fa94c069420 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa970144af0 0x7fa9700cb220 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa97006eeb0 0x7fa9700cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 --2- 172.21.15.67:0/879603570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa97006b3a0 0x7fa9700c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 >> 172.21.15.67:0/879603570 conn(0x7fa97006e390 msgr2=0x7fa97015eaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 shutdown_connections 2024-04-17T10:28:21.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.134+0000 7fa96fe00640 1 -- 172.21.15.67:0/879603570 wait complete. 2024-04-17T10:28:21.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/ceph-qa-suite/new_dir_0 2024-04-17T10:28:21.293 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:21.294 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:21.294 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:21.309 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_0' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:21.309 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:21.309 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:21.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:21.318 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_0 bs=1M conv=fdatasync count=1 seek=0) 2024-04-17T10:28:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:28:21.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:28:21.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161281 s, 65.0 MB/s 2024-04-17T10:28:21.387 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_0' 2024-04-17T10:28:21.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:21.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f617dc00640 1 Processor -- start 2024-04-17T10:28:21.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f617dc00640 1 -- start start 2024-04-17T10:28:21.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f617dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780cdaf0 0x7f61780b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:21.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f617dc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61780ce090 con 0x7f617815dfe0 2024-04-17T10:28:21.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f617dc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61780ce3d0 con 0x7f61780cf3c0 2024-04-17T10:28:21.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f617dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61780ce950 con 0x7f61780cdaf0 2024-04-17T10:28:21.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6176a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f617815dfe0 0x7f61780cecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59624/0 (socket says 172.21.15.67:59624) 2024-04-17T10:28:21.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6176a00640 1 -- 172.21.15.67:0/2566627072 learned_addr learned my addr 172.21.15.67:0/2566627072 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:21.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6177400640 1 --2- 172.21.15.67:0/2566627072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780cdaf0 0x7f61780b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:21.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941459065 0 0) 0x7f61780ce090 con 0x7f617815dfe0 2024-04-17T10:28:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f615c003530 con 0x7f617815dfe0 2024-04-17T10:28:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505432146 0 0) 0x7f61780ce3d0 con 0x7f61780cf3c0 2024-04-17T10:28:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61780ce090 con 0x7f61780cf3c0 2024-04-17T10:28:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1731161790 0 0) 0x7f615c003530 con 0x7f617815dfe0 2024-04-17T10:28:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6177400640 1 -- 172.21.15.67:0/2566627072 >> v1:172.21.15.86:6789/0 conn(0x7f61780cf3c0 legacy=0x7f61780cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6177400640 1 -- 172.21.15.67:0/2566627072 >> v1:172.21.15.67:6789/0 conn(0x7f617815dfe0 legacy=0x7f61780cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.586+0000 7f6177400640 1 -- 172.21.15.67:0/2566627072 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61780afc90 con 0x7f61780cdaf0 2024-04-17T10:28:21.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f6177400640 1 --2- 172.21.15.67:0/2566627072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780cdaf0 0x7f61780b2dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6170009dc0 tx=0x7f617001c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cb35542907a5aa6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:21.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168001680 con 0x7f61780cdaf0 2024-04-17T10:28:21.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61700a64c0 con 0x7f61780cdaf0 2024-04-17T10:28:21.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f6176000640 1 -- 172.21.15.67:0/2566627072 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61700a6890 con 0x7f61780cdaf0 2024-04-17T10:28:21.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f617dc00640 1 -- 172.21.15.67:0/2566627072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780cdaf0 msgr2=0x7f61780b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f617dc00640 1 --2- 172.21.15.67:0/2566627072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780cdaf0 0x7f61780b2dc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6170009dc0 tx=0x7f617001c760 comp rx=0 tx=0).stop 2024-04-17T10:28:21.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f617dc00640 1 -- 172.21.15.67:0/2566627072 shutdown_connections 2024-04-17T10:28:21.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f617dc00640 1 --2- 172.21.15.67:0/2566627072 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780cdaf0 0x7f61780b2dc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f617dc00640 1 -- 172.21.15.67:0/2566627072 >> 172.21.15.67:0/2566627072 conn(0x7f61780b22a0 msgr2=0x7f61780b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:21.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f617dc00640 1 -- 172.21.15.67:0/2566627072 shutdown_connections 2024-04-17T10:28:21.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.590+0000 7f617dc00640 1 -- 172.21.15.67:0/2566627072 wait complete. 2024-04-17T10:28:21.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 Processor -- start 2024-04-17T10:28:21.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 -- start start 2024-04-17T10:28:21.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cdaf0 0x7f61780b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:21.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61780cf3c0 0x7f61780b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:21.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780b14f0 0x7f6178151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:21.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6178168fa0 con 0x7f61780cdaf0 2024-04-17T10:28:21.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61780cc610 con 0x7f61780cf3c0 2024-04-17T10:28:21.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6178169120 con 0x7f61780b14f0 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cdaf0 0x7f61780b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cdaf0 0x7f61780b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59932/0 (socket says 172.21.15.67:59932) 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 -- 172.21.15.67:0/819042107 learned_addr learned my addr 172.21.15.67:0/819042107 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6177400640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780b14f0 0x7f6178151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f616e000640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61780cf3c0 0x7f61780b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 -- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780b14f0 msgr2=0x7f6178151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780b14f0 0x7f6178151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 -- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61780cf3c0 msgr2=0x7f61780b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61780cf3c0 0x7f61780b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 -- 172.21.15.67:0/819042107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617813e970 con 0x7f61780cdaf0 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f6176a00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cdaf0 0x7f61780b38a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f616800c750 tx=0x7f616800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:21.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f616f400640 1 -- 172.21.15.67:0/819042107 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168005ae0 con 0x7f61780cdaf0 2024-04-17T10:28:21.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f616f400640 1 -- 172.21.15.67:0/819042107 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6168005c80 con 0x7f61780cdaf0 2024-04-17T10:28:21.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f616f400640 1 -- 172.21.15.67:0/819042107 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61680072b0 con 0x7f61780cdaf0 2024-04-17T10:28:21.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6178156f40 con 0x7f61780cdaf0 2024-04-17T10:28:21.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.594+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617813eef0 con 0x7f61780cdaf0 2024-04-17T10:28:21.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.598+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617809e900 con 0x7f61780cdaf0 2024-04-17T10:28:21.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.598+0000 7f616f400640 1 -- 172.21.15.67:0/819042107 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6168007450 con 0x7f61780cdaf0 2024-04-17T10:28:21.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.598+0000 7f616f400640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6154066f60 0x7f6154069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:21.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.598+0000 7f616f400640 1 -- 172.21.15.67:0/819042107 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f61680850a0 con 0x7f61780cdaf0 2024-04-17T10:28:21.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.598+0000 7f616e000640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6154066f60 0x7f6154069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:21.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.598+0000 7f616e000640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6154066f60 0x7f6154069420 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f617806e2a0 tx=0x7f6160009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:21.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.602+0000 7f616f400640 1 -- 172.21.15.67:0/819042107 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6168055eb0 con 0x7f61780cdaf0 2024-04-17T10:28:21.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6178070c80 con 0x7f61780cdaf0 2024-04-17T10:28:21.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f616f400640 1 -- 172.21.15.67:0/819042107 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f61680557b0 con 0x7f61780cdaf0 2024-04-17T10:28:21.792 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:21.792 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:21.792 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:21.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6154066f60 msgr2=0x7f6154069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6154066f60 0x7f6154069420 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f617806e2a0 tx=0x7f6160009040 comp rx=0 tx=0).stop 2024-04-17T10:28:21.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cdaf0 msgr2=0x7f61780b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cdaf0 0x7f61780b38a0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f616800c750 tx=0x7f616800cc10 comp rx=0 tx=0).stop 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 shutdown_connections 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6154066f60 0x7f6154069420 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f61780b14f0 0x7f6178151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f61780cf3c0 0x7f61780b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 --2- 172.21.15.67:0/819042107 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f61780cdaf0 0x7f61780b38a0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 >> 172.21.15.67:0/819042107 conn(0x7f61780b22a0 msgr2=0x7f61780cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 shutdown_connections 2024-04-17T10:28:21.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:21.798+0000 7f617dc00640 1 -- 172.21.15.67:0/819042107 wait complete. 2024-04-17T10:28:21.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/ceph-qa-suite/new_file_0 2024-04-17T10:28:21.960 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:21.960 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:21.960 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:21.975 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_0' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:21.976 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-04-17T10:28:21.976 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:28:21.976 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-04-17T10:28:21.976 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:28:21.976 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_0").st_ino) 2024-04-17T10:28:21.976 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:28:22.043 INFO:teuthology.orchestra.run.smithi067.stdout:1099511630187 2024-04-17T10:28:22.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T10:28:22.053 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:22.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-04-17T10:28:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.242+0000 7f79be800640 1 Processor -- start 2024-04-17T10:28:22.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.242+0000 7f79be800640 1 -- start start 2024-04-17T10:28:22.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79be800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79b80cd030 0x7f79b806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79be800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b80b4990 con 0x7f79b806e7b0 2024-04-17T10:28:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79be800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b806ec00 con 0x7f79b80aef60 2024-04-17T10:28:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79be800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b80af340 con 0x7f79b80cd030 2024-04-17T10:28:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79b6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79b80cd030 0x7f79b806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79b7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f79b806e7b0 0x7f79b80b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59632/0 (socket says 172.21.15.67:59632) 2024-04-17T10:28:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79b7e00640 1 -- 172.21.15.67:0/2375170201 learned_addr learned my addr 172.21.15.67:0/2375170201 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:22.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79b6a00640 1 -- 172.21.15.67:0/2375170201 >> v1:172.21.15.86:6789/0 conn(0x7f79b80aef60 legacy=0x7f79b80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79b6a00640 1 -- 172.21.15.67:0/2375170201 >> v1:172.21.15.67:6789/0 conn(0x7f79b806e7b0 legacy=0x7f79b80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79b6a00640 1 -- 172.21.15.67:0/2375170201 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b815ae90 con 0x7f79b80cd030 2024-04-17T10:28:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.246+0000 7f79b6a00640 1 --2- 172.21.15.67:0/2375170201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80cd030 0x7f79b806e270 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f79a40097c0 tx=0x7f79a401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595d4d8293a3917c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79b6000640 1 -- 172.21.15.67:0/2375170201 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40b0070 con 0x7f79b80cd030 2024-04-17T10:28:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79b6000640 1 -- 172.21.15.67:0/2375170201 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a40ab600 con 0x7f79b80cd030 2024-04-17T10:28:22.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79b6000640 1 -- 172.21.15.67:0/2375170201 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40ab9d0 con 0x7f79b80cd030 2024-04-17T10:28:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- 172.21.15.67:0/2375170201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80cd030 msgr2=0x7f79b806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 --2- 172.21.15.67:0/2375170201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80cd030 0x7f79b806e270 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f79a40097c0 tx=0x7f79a401c410 comp rx=0 tx=0).stop 2024-04-17T10:28:22.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- 172.21.15.67:0/2375170201 shutdown_connections 2024-04-17T10:28:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 --2- 172.21.15.67:0/2375170201 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80cd030 0x7f79b806e270 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- 172.21.15.67:0/2375170201 >> 172.21.15.67:0/2375170201 conn(0x7f79b80b1740 msgr2=0x7f79b80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- 172.21.15.67:0/2375170201 shutdown_connections 2024-04-17T10:28:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- 172.21.15.67:0/2375170201 wait complete. 2024-04-17T10:28:22.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 Processor -- start 2024-04-17T10:28:22.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- start start 2024-04-17T10:28:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b806e7b0 0x7f79b813a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80aef60 0x7f79b8149550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b80cd030 0x7f79b81531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b815bc10 con 0x7f79b80aef60 2024-04-17T10:28:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b81683d0 con 0x7f79b80cd030 2024-04-17T10:28:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79be800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b813a710 con 0x7f79b806e7b0 2024-04-17T10:28:22.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80aef60 0x7f79b8149550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80aef60 0x7f79b8149550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59956/0 (socket says 172.21.15.67:59956) 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79b7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b806e7b0 0x7f79b813a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.250+0000 7f79b6a00640 1 -- 172.21.15.67:0/2239405213 learned_addr learned my addr 172.21.15.67:0/2239405213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79b7e00640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b80cd030 0x7f79b81531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79b6a00640 1 -- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b806e7b0 msgr2=0x7f79b813a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79b6a00640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b806e7b0 0x7f79b813a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79b6a00640 1 -- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b80cd030 msgr2=0x7f79b81531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79b6a00640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b80cd030 0x7f79b81531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79b6a00640 1 -- 172.21.15.67:0/2239405213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b8151e60 con 0x7f79b80aef60 2024-04-17T10:28:22.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79b6a00640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80aef60 0x7f79b8149550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79a40ae040 tx=0x7f79a40191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79af400640 1 -- 172.21.15.67:0/2239405213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40b0040 con 0x7f79b80aef60 2024-04-17T10:28:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79af400640 1 -- 172.21.15.67:0/2239405213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a40b9070 con 0x7f79b80aef60 2024-04-17T10:28:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79af400640 1 -- 172.21.15.67:0/2239405213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40aaa30 con 0x7f79b80aef60 2024-04-17T10:28:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b8152140 con 0x7f79b80aef60 2024-04-17T10:28:22.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b8152700 con 0x7f79b80aef60 2024-04-17T10:28:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79af400640 1 -- 172.21.15.67:0/2239405213 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f79a40aabd0 con 0x7f79b80aef60 2024-04-17T10:28:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79af400640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7998066f60 0x7f7998069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.254+0000 7f79af400640 1 -- 172.21.15.67:0/2239405213 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f79a411f220 con 0x7f79b80aef60 2024-04-17T10:28:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b7400640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7998066f60 0x7f7998069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 Processor -- start 2024-04-17T10:28:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- start start 2024-04-17T10:28:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b7400640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7998066f60 0x7f7998069420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f79b815e060 tx=0x7f79a8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e6b5180 0x55e76e6b5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5aa0 0x55e76e66b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e66b600 0x55e76e66b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76e4b8f20 con 0x55e76e6b5aa0 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76e441000 con 0x55e76e6b5180 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e76e4b4fc0 con 0x55e76e66b600 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5aa0 0x55e76e66b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5aa0 0x55e76e66b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59966/0 (socket says 172.21.15.67:59966) 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 -- 172.21.15.67:0/592581978 learned_addr learned my addr 172.21.15.67:0/592581978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b7400640 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e6b5180 0x55e76e6b5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b7e00640 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e66b600 0x55e76e66b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 -- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e66b600 msgr2=0x55e76e66b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e66b600 0x55e76e66b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 -- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e6b5180 msgr2=0x55e76e6b5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e6b5180 0x55e76e6b5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 -- 172.21.15.67:0/592581978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e76e494ba0 con 0x55e76e6b5aa0 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b7e00640 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e66b600 0x55e76e66b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79b6a00640 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5aa0 0x55e76e66b0c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f79a40ef740 tx=0x7f79a40ebe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192095626fdcf166 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79a3e00640 1 -- 172.21.15.67:0/592581978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4107020 con 0x55e76e6b5aa0 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79a3e00640 1 -- 172.21.15.67:0/592581978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a40ab600 con 0x55e76e6b5aa0 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79a3e00640 1 -- 172.21.15.67:0/592581978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40efa40 con 0x55e76e6b5aa0 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- 172.21.15.67:0/592581978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5aa0 msgr2=0x55e76e66b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5aa0 0x55e76e66b0c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f79a40ef740 tx=0x7f79a40ebe10 comp rx=0 tx=0).stop 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- 172.21.15.67:0/592581978 shutdown_connections 2024-04-17T10:28:22.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e66b600 0x55e76e66b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5aa0 0x55e76e66b0c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/592581978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e6b5180 0x55e76e6b5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- 172.21.15.67:0/592581978 >> 172.21.15.67:0/592581978 conn(0x55e76e671760 msgr2=0x55e76e6699a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- 172.21.15.67:0/592581978 shutdown_connections 2024-04-17T10:28:22.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.258+0000 7f79c1cc4000 1 -- 172.21.15.67:0/592581978 wait complete. 2024-04-17T10:28:22.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 -- start start 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 Processor -- start 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e670be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5aa0 0x55e76e6a3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76e5769c0 con 0x55e76e6b5180 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76e4735d0 con 0x55e76e66b600 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e76e56e9d0 con 0x55e76e6b5aa0 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 10 client.0 _ll_register_callbacks cb 0x55e76e656a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e670be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e670be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59978/0 (socket says 172.21.15.67:59978) 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 -- 172.21.15.67:0/2496869390 learned_addr learned my addr 172.21.15.67:0/2496869390 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b7400640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b7e00640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5aa0 0x55e76e6a3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 -- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5aa0 msgr2=0x55e76e6a3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5aa0 0x55e76e6a3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 -- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 msgr2=0x55e76e6706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 -- 172.21.15.67:0/2496869390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e76e4ff5a0 con 0x55e76e6b5180 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b7e00640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5aa0 0x55e76e6a3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b6a00640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e670be0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f79a401c8f0 tx=0x7f79a40c3f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8328dd9bd33976e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79a0c00640 1 -- 172.21.15.67:0/2496869390 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40f80a0 con 0x55e76e6b5180 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79a0c00640 1 -- 172.21.15.67:0/2496869390 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f79a40edd40 con 0x55e76e6b5180 2024-04-17T10:28:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79a40ab600 con 0x55e76e6b5180 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79a0c00640 1 -- 172.21.15.67:0/2496869390 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79a4115050 con 0x55e76e6b5180 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 10 client.6096 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e76e4902b0 con 0x55e76e6b5180 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79a0c00640 1 -- 172.21.15.67:0/2496869390 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a40ab600 con 0x55e76e6b5180 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79a0c00640 1 -- 172.21.15.67:0/2496869390 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f79a40f9070 con 0x55e76e6b5180 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 10 client.6096 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 10 client.6096 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 20 client.6096 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6a2e60 0x55e76e6a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 4 client.6096 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e76e454c70 con 0x55e76e6a2e60 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b7400640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6a2e60 0x55e76e6a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79b7400640 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6a2e60 0x55e76e6a3e60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa7226874eba4cb3 server_cookie=740af10624817d02 in_seq=0 out_seq=0 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.262+0000 7f79a0c00640 10 client.6096 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79a0c00640 1 -- 172.21.15.67:0/2496869390 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e76e454c70 con 0x55e76e6a2e60 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79a0c00640 10 client.6096 handle_command_reply: tid=0 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 client.6096 shutdown 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 msgr2=0x55e76e670be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e670be0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f79a401c8f0 tx=0x7f79a40c3f90 comp rx=0 tx=0).stop 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 shutdown_connections 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6a2e60 0x55e76e6a3e60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5aa0 0x55e76e6a3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e670be0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79a0c00640 0 client.6096 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/2496869390 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 >> 172.21.15.67:0/2496869390 conn(0x55e76e671760 msgr2=0x55e76e66a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 shutdown_connections 2024-04-17T10:28:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 1 -- 172.21.15.67:0/2496869390 wait complete. 2024-04-17T10:28:22.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.266+0000 7f79c1cc4000 20 client.6096 trim_cache size 0 max 16384 2024-04-17T10:28:22.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 Processor -- start 2024-04-17T10:28:22.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 -- start start 2024-04-17T10:28:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e671760 0x55e76e6cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e66b600 0x55e76e66a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5180 0x55e76e66a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b8203680 con 0x55e76e66b600 2024-04-17T10:28:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b8202860 con 0x55e76e671760 2024-04-17T10:28:22.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b815dd50 con 0x55e76e6b5180 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e66b600 0x55e76e66a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e66b600 0x55e76e66a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59988/0 (socket says 172.21.15.67:59988) 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 -- 172.21.15.67:0/3528423242 learned_addr learned my addr 172.21.15.67:0/3528423242 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b7e00640 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5180 0x55e76e66a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b7400640 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e671760 0x55e76e6cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 -- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5180 msgr2=0x55e76e66a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5180 0x55e76e66a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 -- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e671760 msgr2=0x55e76e6cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e671760 0x55e76e6cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 -- 172.21.15.67:0/3528423242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e76e6a41d0 con 0x55e76e66b600 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79b6a00640 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e66b600 0x55e76e66a120 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f79a4097dd0 tx=0x7f79a40c1000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a131f5b4292ed491 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79a3400640 1 -- 172.21.15.67:0/3528423242 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4107040 con 0x55e76e66b600 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79a3400640 1 -- 172.21.15.67:0/3528423242 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a410d070 con 0x55e76e66b600 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.278+0000 7f79a3400640 1 -- 172.21.15.67:0/3528423242 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a40f80a0 con 0x55e76e66b600 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/3528423242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e66b600 msgr2=0x55e76e66a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e66b600 0x55e76e66a120 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f79a4097dd0 tx=0x7f79a40c1000 comp rx=0 tx=0).stop 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/3528423242 shutdown_connections 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e6b5180 0x55e76e66a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e66b600 0x55e76e66a120 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/3528423242 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e671760 0x55e76e6cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/3528423242 >> 172.21.15.67:0/3528423242 conn(0x55e76e6a35b0 msgr2=0x55e76e6c2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/3528423242 shutdown_connections 2024-04-17T10:28:22.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/3528423242 wait complete. 2024-04-17T10:28:22.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- start start 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 Processor -- start 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e671760 0x55e76e6ca630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e6cab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e76e46fdc0 con 0x55e76e6b5180 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b8202260 con 0x55e76e66b600 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e76e471e80 con 0x55e76e671760 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 10 client.0 _ll_register_callbacks cb 0x55e76e637600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e6cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e6cab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59994/0 (socket says 172.21.15.67:59994) 2024-04-17T10:28:22.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 -- 172.21.15.67:0/740962895 learned_addr learned my addr 172.21.15.67:0/740962895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7400640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b6a00640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e671760 0x55e76e6ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 -- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e671760 msgr2=0x55e76e6ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e671760 0x55e76e6ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 -- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 msgr2=0x55e76e6ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 -- 172.21.15.67:0/740962895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e76e441250 con 0x55e76e6b5180 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7400640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79b7e00640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e6cab70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f79b0007820 tx=0x7f79b0002120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a682cc0d105caa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79a0c00640 1 -- 172.21.15.67:0/740962895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79b0015b80 con 0x55e76e6b5180 2024-04-17T10:28:22.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79a0c00640 1 -- 172.21.15.67:0/740962895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f79b0015d20 con 0x55e76e6b5180 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79a410d070 con 0x55e76e6b5180 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79a0c00640 1 -- 172.21.15.67:0/740962895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79b0017430 con 0x55e76e6b5180 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 10 client.6102 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e76e44db90 con 0x55e76e6b5180 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79a0c00640 1 -- 172.21.15.67:0/740962895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a410d070 con 0x55e76e6b5180 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79a0c00640 1 -- 172.21.15.67:0/740962895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f79b00175d0 con 0x55e76e6b5180 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 10 client.6102 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 10 client.6102 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 20 client.6102 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6b5aa0 0x55e76e71cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 4 client.6102 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-17T10:28:22.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.282+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e76e4c6460 con 0x55e76e6b5aa0 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79b7400640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6b5aa0 0x55e76e71cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79b7400640 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6b5aa0 0x55e76e71cdf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20eb7cfc0507b9e9 server_cookie=4bfa3f5104be14f2 in_seq=0 out_seq=0 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79a0c00640 10 client.6102 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79a0c00640 1 -- 172.21.15.67:0/740962895 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e76e4c6460 con 0x55e76e6b5aa0 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79a0c00640 10 client.6102 handle_command_reply: tid=0 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 client.6102 shutdown 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 msgr2=0x55e76e6cab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e6cab70 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f79b0007820 tx=0x7f79b0002120 comp rx=0 tx=0).stop 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 shutdown_connections 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e76e6b5aa0 0x55e76e71cdf0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e76e6b5180 0x55e76e6cab70 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e76e671760 0x55e76e6ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 --2- 172.21.15.67:0/740962895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e76e66b600 0x55e76e6ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79a0c00640 0 client.6102 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 >> 172.21.15.67:0/740962895 conn(0x55e76e6a35b0 msgr2=0x55e76e669110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 shutdown_connections 2024-04-17T10:28:22.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 1 -- 172.21.15.67:0/740962895 wait complete. 2024-04-17T10:28:22.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79c1cc4000 20 client.6102 trim_cache size 0 max 16384 2024-04-17T10:28:22.285 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7998066f60 msgr2=0x7f7998069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7998066f60 0x7f7998069420 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f79b815e060 tx=0x7f79a8009040 comp rx=0 tx=0).stop 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80aef60 msgr2=0x7f79b8149550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80aef60 0x7f79b8149550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f79a40ae040 tx=0x7f79a40191b0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 shutdown_connections 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7998066f60 0x7f7998069420 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79b80cd030 0x7f79b81531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79b80aef60 0x7f79b8149550 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 --2- 172.21.15.67:0/2239405213 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79b806e7b0 0x7f79b813a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 >> 172.21.15.67:0/2239405213 conn(0x7f79b80b1740 msgr2=0x7f79b80b0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 shutdown_connections 2024-04-17T10:28:22.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.286+0000 7f79be800640 1 -- 172.21.15.67:0/2239405213 wait complete. 2024-04-17T10:28:22.299 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data rmxattr 1000000096b.00000000 parent 2024-04-17T10:28:22.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.350+0000 7f71aabb4080 1 Processor -- start 2024-04-17T10:28:22.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.350+0000 7f71aabb4080 1 -- start start 2024-04-17T10:28:22.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a6b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac29d68e0 0x561ac298c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac2a4b210 0x561ac2af4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ac29cdbf0 con 0x561ac29d6500 2024-04-17T10:28:22.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ac29339f0 con 0x561ac29d68e0 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ac29cdd70 con 0x561ac2a4b210 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a6b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac29d68e0 0x561ac298c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a6b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60010/0 (socket says 172.21.15.67:60010) 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 -- 172.21.15.67:0/1699536908 learned_addr learned my addr 172.21.15.67:0/1699536908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a8800640 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac2a4b210 0x561ac2af4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 -- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac2a4b210 msgr2=0x561ac2af4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac2a4b210 0x561ac2af4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 -- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac29d68e0 msgr2=0x561ac298c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac29d68e0 0x561ac298c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 -- 172.21.15.67:0/1699536908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ac2a65b70 con 0x561ac29d6500 2024-04-17T10:28:22.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7400640 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac29d68e0 0x561ac298c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:22.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a7e00640 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a6b6d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7198009950 tx=0x7f7198005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29f3df7bec0e82a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a6000640 1 -- 172.21.15.67:0/1699536908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198008da0 con 0x561ac29d6500 2024-04-17T10:28:22.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a6000640 1 -- 172.21.15.67:0/1699536908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198018070 con 0x561ac29d6500 2024-04-17T10:28:22.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71a6000640 1 -- 172.21.15.67:0/1699536908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198013640 con 0x561ac29d6500 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- 172.21.15.67:0/1699536908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 msgr2=0x561ac2a6b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a6b6d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7198009950 tx=0x7f7198005a10 comp rx=0 tx=0).stop 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- 172.21.15.67:0/1699536908 shutdown_connections 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac2a4b210 0x561ac2af4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac29d68e0 0x561ac298c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- 172.21.15.67:0/1699536908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a6b6d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- 172.21.15.67:0/1699536908 >> 172.21.15.67:0/1699536908 conn(0x561ac29b90b0 msgr2=0x561ac29bc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- 172.21.15.67:0/1699536908 shutdown_connections 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- 172.21.15.67:0/1699536908 wait complete. 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 Processor -- start 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- start start 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a32620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac29d68e0 0x561ac2a65460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac2a4b210 0x561ac2a61540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ac2a660b0 con 0x561ac29d6500 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ac2a65e10 con 0x561ac2a4b210 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.354+0000 7f71aabb4080 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x561ac2a66350 con 0x561ac29d68e0 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a32620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a32620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60018/0 (socket says 172.21.15.67:60018) 2024-04-17T10:28:22.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 -- 172.21.15.67:0/2948958987 learned_addr learned my addr 172.21.15.67:0/2948958987 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7400640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac29d68e0 0x561ac2a65460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a8800640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac2a4b210 0x561ac2a61540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 -- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac29d68e0 msgr2=0x561ac2a65460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac29d68e0 0x561ac2a65460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 -- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac2a4b210 msgr2=0x561ac2a61540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x561ac2a4b210 0x561ac2a61540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 -- 172.21.15.67:0/2948958987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ac2a659a0 con 0x561ac29d6500 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7400640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x561ac29d68e0 0x561ac2a65460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a7e00640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a32620 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7198013700 tx=0x7f7198003030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a4c00640 1 -- 172.21.15.67:0/2948958987 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198008ea0 con 0x561ac29d6500 2024-04-17T10:28:22.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a4c00640 1 -- 172.21.15.67:0/2948958987 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7198021030 con 0x561ac29d6500 2024-04-17T10:28:22.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561ac2a32c50 con 0x561ac29d6500 2024-04-17T10:28:22.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a4c00640 1 -- 172.21.15.67:0/2948958987 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7198026420 con 0x561ac29d6500 2024-04-17T10:28:22.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ac2a33190 con 0x561ac29d6500 2024-04-17T10:28:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a4c00640 1 -- 172.21.15.67:0/2948958987 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f719801c420 con 0x561ac29d6500 2024-04-17T10:28:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.358+0000 7f71a4c00640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7188065970 0x7f7188067e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71a4c00640 1 -- 172.21.15.67:0/2948958987 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7198085f10 con 0x561ac29d6500 2024-04-17T10:28:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71a7400640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7188065970 0x7f7188067e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x561ac2a33b70 0x561ac2a33f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 --> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] -- osd_op(unknown.0.0:1 8.a 8:51f47cfd:::1000000096b.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) v9 -- 0x561ac2a6a290 con 0x561ac2a33b70 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71a8800640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x561ac2a33b70 0x561ac2a33f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71a7400640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7188065970 0x7f7188067e30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f719400e830 tx=0x7f71940017a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71a8800640 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x561ac2a33b70 0x561ac2a33f80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71a8800640 1 -- 172.21.15.67:0/2948958987 <== osd.1 v2:172.21.15.67:6808/2092989796 1 ==== osd_op_reply(1 1000000096b.00000000 [rmxattr] v54'3 uv3 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f71a00a7a40 con 0x561ac2a33b70 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x561ac2a33b70 msgr2=0x561ac2a33f80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:6808/2092989796,v1:172.21.15.67:6810/2092989796] conn(0x561ac2a33b70 0x561ac2a33f80 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7188065970 msgr2=0x7f7188067e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7188065970 0x7f7188067e30 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f719400e830 tx=0x7f71940017a0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 msgr2=0x561ac2a32620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 --2- 172.21.15.67:0/2948958987 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561ac29d6500 0x561ac2a32620 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7198013700 tx=0x7f7198003030 comp rx=0 tx=0).stop 2024-04-17T10:28:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71a8800640 1 -- 172.21.15.67:0/2948958987 reap_dead start 2024-04-17T10:28:22.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 shutdown_connections 2024-04-17T10:28:22.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 >> 172.21.15.67:0/2948958987 conn(0x561ac29b90b0 msgr2=0x561ac29bc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 shutdown_connections 2024-04-17T10:28:22.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.362+0000 7f71aabb4080 1 -- 172.21.15.67:0/2948958987 wait complete. 2024-04-17T10:28:22.362 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_0' 2024-04-17T10:28:22.362 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:22.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/new_file_0 2024-04-17T10:28:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43b1200640 1 Processor -- start 2024-04-17T10:28:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43b1200640 1 -- start start 2024-04-17T10:28:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43b1200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac15db90 0x7f43ac15ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43b1200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f43ac1615e0 0x7f43ac071510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43b1200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ac161dd0 con 0x7f43ac071a50 2024-04-17T10:28:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43b1200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ac070da0 con 0x7f43ac1615e0 2024-04-17T10:28:22.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43b1200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac071080 con 0x7f43ac15db90 2024-04-17T10:28:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43aa800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f43ac071a50 0x7f43ac0cbec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59636/0 (socket says 172.21.15.67:59636) 2024-04-17T10:28:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43aa800640 1 -- 172.21.15.67:0/2147347131 learned_addr learned my addr 172.21.15.67:0/2147347131 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.586+0000 7f43a9400640 1 --2- 172.21.15.67:0/2147347131 >> v2:172.21.15.86:3300/0 conn(0x7f43ac1615e0 0x7f43ac071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.590+0000 7f43a9e00640 1 --2- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac15db90 0x7f43ac15ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.590+0000 7f43a9400640 1 -- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac15db90 msgr2=0x7f43ac15ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.590+0000 7f43a9400640 1 --2- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac15db90 0x7f43ac15ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.590+0000 7f43a9400640 1 -- 172.21.15.67:0/2147347131 >> v1:172.21.15.67:6789/0 conn(0x7f43ac071a50 legacy=0x7f43ac0cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.590+0000 7f43a9400640 1 -- 172.21.15.67:0/2147347131 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac1620b0 con 0x7f43ac1615e0 2024-04-17T10:28:22.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43a9400640 1 --2- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac1615e0 0x7f43ac071510 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f43980097c0 tx=0x7f439801c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e926dbe647ce7a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43a8a00640 1 -- 172.21.15.67:0/2147347131 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980b0070 con 0x7f43ac1615e0 2024-04-17T10:28:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43a8a00640 1 -- 172.21.15.67:0/2147347131 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43980ab600 con 0x7f43ac1615e0 2024-04-17T10:28:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43a8a00640 1 -- 172.21.15.67:0/2147347131 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980ab9d0 con 0x7f43ac1615e0 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 -- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac1615e0 msgr2=0x7f43ac071510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 --2- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac1615e0 0x7f43ac071510 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f43980097c0 tx=0x7f439801c350 comp rx=0 tx=0).stop 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 -- 172.21.15.67:0/2147347131 shutdown_connections 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 --2- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac1615e0 0x7f43ac071510 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 --2- 172.21.15.67:0/2147347131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac15db90 0x7f43ac15ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 -- 172.21.15.67:0/2147347131 >> 172.21.15.67:0/2147347131 conn(0x7f43ac0b3460 msgr2=0x7f43ac0b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 -- 172.21.15.67:0/2147347131 shutdown_connections 2024-04-17T10:28:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.594+0000 7f43b1200640 1 -- 172.21.15.67:0/2147347131 wait complete. 2024-04-17T10:28:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 Processor -- start 2024-04-17T10:28:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 -- start start 2024-04-17T10:28:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac071a50 0x7f43ac0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ac15db90 0x7f43ac15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac1615e0 0x7f43ac14d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac16a980 con 0x7f43ac15db90 2024-04-17T10:28:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac0cc7e0 con 0x7f43ac071a50 2024-04-17T10:28:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43b1200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac16ab00 con 0x7f43ac1615e0 2024-04-17T10:28:22.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac071a50 0x7f43ac0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac071a50 0x7f43ac0b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:59534/0 (socket says 172.21.15.67:59534) 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9e00640 1 -- 172.21.15.67:0/321082591 learned_addr learned my addr 172.21.15.67:0/321082591 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9400640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ac15db90 0x7f43ac15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9400640 1 -- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac1615e0 msgr2=0x7f43ac14d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9400640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac1615e0 0x7f43ac14d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9400640 1 -- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac071a50 msgr2=0x7f43ac0b4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9400640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac071a50 0x7f43ac0b4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.598+0000 7f43a9400640 1 -- 172.21.15.67:0/321082591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac1fe5f0 con 0x7f43ac15db90 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.602+0000 7f43a9400640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ac15db90 0x7f43ac15bc90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f43980ab000 tx=0x7f43980190f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.602+0000 7f43a2000640 1 -- 172.21.15.67:0/321082591 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980b8020 con 0x7f43ac15db90 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.602+0000 7f43a2000640 1 -- 172.21.15.67:0/321082591 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43980b4410 con 0x7f43ac15db90 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.602+0000 7f43a2000640 1 -- 172.21.15.67:0/321082591 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980b0040 con 0x7f43ac15db90 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.602+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ac1feb70 con 0x7f43ac15db90 2024-04-17T10:28:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.602+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ac1ff370 con 0x7f43ac15db90 2024-04-17T10:28:22.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a2000640 1 -- 172.21.15.67:0/321082591 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f43980bb020 con 0x7f43ac15db90 2024-04-17T10:28:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a2000640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f438c066f90 0x7f438c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a2000640 1 -- 172.21.15.67:0/321082591 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f439811ebb0 con 0x7f43ac15db90 2024-04-17T10:28:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 Processor -- start 2024-04-17T10:28:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f438c066f90 0x7f438c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 -- start start 2024-04-17T10:28:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f438c066f90 0x7f438c069450 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f439c009c20 tx=0x7f439c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e8332475d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e833247b10 0x55e8331fa3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8331fa920 0x55e8331fad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e83304ace0 con 0x55e8332471f0 2024-04-17T10:28:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e832fd3530 con 0x55e833247b10 2024-04-17T10:28:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e833046d80 con 0x55e8331fa920 2024-04-17T10:28:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e8332475d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e8332475d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60038/0 (socket says 172.21.15.67:60038) 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 -- 172.21.15.67:0/3000174491 learned_addr learned my addr 172.21.15.67:0/3000174491 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43aa800640 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8331fa920 0x55e8331fad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9400640 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e833247b10 0x55e8331fa3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 -- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8331fa920 msgr2=0x55e8331fad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8331fa920 0x55e8331fad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 -- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e833247b10 msgr2=0x55e8331fa3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e833247b10 0x55e8331fa3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 -- 172.21.15.67:0/3000174491 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e833045cd0 con 0x55e8332471f0 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9400640 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e833247b10 0x55e8331fa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.606+0000 7f43a9e00640 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e8332475d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f439c009ae0 tx=0x7f439c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2db4ce52bd555e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f4396a00640 1 -- 172.21.15.67:0/3000174491 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c01d050 con 0x55e8332471f0 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f4396a00640 1 -- 172.21.15.67:0/3000174491 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c027070 con 0x55e8332471f0 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f4396a00640 1 -- 172.21.15.67:0/3000174491 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c003af0 con 0x55e8332471f0 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- 172.21.15.67:0/3000174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 msgr2=0x55e8332475d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e8332475d0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f439c009ae0 tx=0x7f439c014000 comp rx=0 tx=0).stop 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- 172.21.15.67:0/3000174491 shutdown_connections 2024-04-17T10:28:22.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8331fa920 0x55e8331fad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e833247b10 0x55e8331fa3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 --2- 172.21.15.67:0/3000174491 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e8332475d0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- 172.21.15.67:0/3000174491 >> 172.21.15.67:0/3000174491 conn(0x55e833227f90 msgr2=0x55e83322a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- 172.21.15.67:0/3000174491 shutdown_connections 2024-04-17T10:28:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- 172.21.15.67:0/3000174491 wait complete. 2024-04-17T10:28:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- start start 2024-04-17T10:28:22.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 Processor -- start 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e83324e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e83324eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e833200a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e833005bc0 con 0x55e8332471f0 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e833206130 con 0x55e8331fa920 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e833047d50 con 0x55e833247b10 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43b4715000 10 client.0 _ll_register_callbacks cb 0x55e8331e84a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e83324eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e83324eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60050/0 (socket says 172.21.15.67:60050) 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 -- 172.21.15.67:0/4269402551 learned_addr learned my addr 172.21.15.67:0/4269402551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43aa800640 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e833200a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9e00640 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e83324e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 -- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 msgr2=0x55e833200a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e833200a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 -- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 msgr2=0x55e83324e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e83324e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 -- 172.21.15.67:0/4269402551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e833058950 con 0x55e8332471f0 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f43a9400640 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e83324eca0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f43980ef030 tx=0x7f43980eed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fec2465d0e37c5a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.610+0000 7f438b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:22.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 1 -- 172.21.15.67:0/4269402551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43980f3480 con 0x55e8332471f0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 1 -- 172.21.15.67:0/4269402551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f43980ef590 con 0x55e8332471f0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f439c027070 con 0x55e8332471f0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 1 -- 172.21.15.67:0/4269402551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43980f7a40 con 0x55e8332471f0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 10 client.6120 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e8330c7790 con 0x55e8332471f0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 1 -- 172.21.15.67:0/4269402551 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c027070 con 0x55e8332471f0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 1 -- 172.21.15.67:0/4269402551 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4398104070 con 0x55e8332471f0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 10 client.6120 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 10 client.6120 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 20 client.6120 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332017c0 0x55e833201ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 4 client.6120 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e832fe6b80 con 0x55e8332017c0 2024-04-17T10:28:22.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43a9e00640 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332017c0 0x55e833201ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43a9e00640 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332017c0 0x55e833201ba0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6f8ccd2f4be9afd server_cookie=a76927343377a9f1 in_seq=0 out_seq=0 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 10 client.6120 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 1 -- 172.21.15.67:0/4269402551 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e832fe6b80 con 0x55e8332017c0 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 10 client.6120 handle_command_reply: tid=0 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 client.6120 shutdown 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 msgr2=0x55e83324eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e83324eca0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f43980ef030 tx=0x7f43980eed50 comp rx=0 tx=0).stop 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 shutdown_connections 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332017c0 0x55e833201ba0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e833200a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8332471f0 0x55e83324eca0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f438b400640 0 client.6120 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 --2- 172.21.15.67:0/4269402551 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e83324e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 >> 172.21.15.67:0/4269402551 conn(0x55e833227f90 msgr2=0x55e8331f9ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 shutdown_connections 2024-04-17T10:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 1 -- 172.21.15.67:0/4269402551 wait complete. 2024-04-17T10:28:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.614+0000 7f43b4715000 20 client.6120 trim_cache size 0 max 16384 2024-04-17T10:28:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 Processor -- start 2024-04-17T10:28:22.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 -- start start 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e8331f8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8332471f0 0x55e8331f9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e833247b10 0x55e8331f95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac207820 con 0x55e833247b10 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac206a30 con 0x55e8331fa920 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac1fc980 con 0x55e8332471f0 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e833247b10 0x55e8331f95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e833247b10 0x55e8331f95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60064/0 (socket says 172.21.15.67:60064) 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 -- 172.21.15.67:0/83680190 learned_addr learned my addr 172.21.15.67:0/83680190 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43a9e00640 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e8331f8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43a9400640 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8332471f0 0x55e8331f9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 -- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8332471f0 msgr2=0x55e8331f9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8332471f0 0x55e8331f9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 -- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 msgr2=0x55e8331f8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e8331f8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 -- 172.21.15.67:0/83680190 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e8330106f0 con 0x55e833247b10 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43a9e00640 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e8331f8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f43aa800640 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e833247b10 0x55e8331f95c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f43a4002160 tx=0x7f43a40019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b124c8790aa203 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f4396000640 1 -- 172.21.15.67:0/83680190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a4019020 con 0x55e833247b10 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.630+0000 7f4396000640 1 -- 172.21.15.67:0/83680190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a4012e90 con 0x55e833247b10 2024-04-17T10:28:22.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f4396000640 1 -- 172.21.15.67:0/83680190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a40074e0 con 0x55e833247b10 2024-04-17T10:28:22.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- 172.21.15.67:0/83680190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e833247b10 msgr2=0x55e8331f95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e833247b10 0x55e8331f95c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f43a4002160 tx=0x7f43a40019f0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- 172.21.15.67:0/83680190 shutdown_connections 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e833247b10 0x55e8331f95c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e8332471f0 0x55e8331f9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 --2- 172.21.15.67:0/83680190 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8331fa920 0x55e8331f8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- 172.21.15.67:0/83680190 >> 172.21.15.67:0/83680190 conn(0x55e833227f90 msgr2=0x55e833201ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- 172.21.15.67:0/83680190 shutdown_connections 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- 172.21.15.67:0/83680190 wait complete. 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- start start 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 Processor -- start 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8331fa920 0x55e8331f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8332471f0 0x55e83324af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e83324b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e833255440 con 0x55e8331fa920 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ac1fcf80 con 0x55e8332471f0 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e832fcfa80 con 0x55e833247b10 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 10 client.0 _ll_register_callbacks cb 0x7f439c009950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8331fa920 0x55e8331f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8331fa920 0x55e8331f9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60076/0 (socket says 172.21.15.67:60076) 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 -- 172.21.15.67:0/2514029647 learned_addr learned my addr 172.21.15.67:0/2514029647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9400640 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8332471f0 0x55e83324af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43aa800640 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e83324b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 -- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 msgr2=0x55e83324b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e83324b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 -- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8332471f0 msgr2=0x55e83324af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8332471f0 0x55e83324af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 -- 172.21.15.67:0/2514029647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e833006c80 con 0x55e8331fa920 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43a9e00640 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8331fa920 0x55e8331f9f40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f439c024040 tx=0x7f439c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c8d9d8df8c1b65 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f438b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f438b400640 1 -- 172.21.15.67:0/2514029647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c01d040 con 0x55e8331fa920 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f438b400640 1 -- 172.21.15.67:0/2514029647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f439c003d30 con 0x55e8331fa920 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.634+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43a4012e90 con 0x55e8331fa920 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f438b400640 1 -- 172.21.15.67:0/2514029647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f439c048050 con 0x55e8331fa920 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 10 client.6126 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55e832d07600 con 0x55e8331fa920 2024-04-17T10:28:22.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f438b400640 1 -- 172.21.15.67:0/2514029647 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a4012e90 con 0x55e8331fa920 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f438b400640 1 -- 172.21.15.67:0/2514029647 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f439c04b020 con 0x55e8331fa920 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 10 client.6126 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 10 client.6126 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 20 client.6126 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332a7a70 0x55e8332a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 4 client.6126 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_0"}] 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_0"}) v1 -- 0x55e833058220 con 0x55e8332a7a70 2024-04-17T10:28:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43a9400640 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332a7a70 0x55e8332a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.635 INFO:tasks.ceph.mds.d.smithi086.stderr:2024-04-17T10:28:22.635+0000 7f6880000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000096b [2,head] /scrub_test_path/ceph-qa-suite/new_file_0 auth v624 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5958=0-4194304@1} caps={5958=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@3},l=5958 | request=0 lock=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x561ab84c7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x1000000096b:[<0x10000000001/new_file_0 v624>,<0x10000000000/ceph-qa-suite v53>,<0x1/scrub_test_path v15>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":-61} 2024-04-17T10:28:22.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43a9400640 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332a7a70 0x55e8332a7e80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a3073df5d4e4037 server_cookie=1602f64960e325fa in_seq=0 out_seq=0 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f438b400640 10 client.6126 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f438b400640 1 -- 172.21.15.67:0/2514029647 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e833058220 con 0x55e8332a7a70 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f438b400640 10 client.6126 handle_command_reply: tid=0 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 client.6126 shutdown 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8331fa920 msgr2=0x55e8331f9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8331fa920 0x55e8331f9f40 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f439c024040 tx=0x7f439c004740 comp rx=0 tx=0).stop 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 shutdown_connections 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55e8332a7a70 0x55e8332a7e80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e833247b10 0x55e83324b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e8332471f0 0x55e83324af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 --2- 172.21.15.67:0/2514029647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e8331fa920 0x55e8331f9f40 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f438b400640 0 client.6126 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 >> 172.21.15.67:0/2514029647 conn(0x55e833227f90 msgr2=0x55e833202430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 shutdown_connections 2024-04-17T10:28:22.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 1 -- 172.21.15.67:0/2514029647 wait complete. 2024-04-17T10:28:22.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b4715000 20 client.6126 trim_cache size 0 max 16384 2024-04-17T10:28:22.638 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:22.638 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:22.638 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "592accb4-e553-428f-ab07-a88dfb6539cb", 2024-04-17T10:28:22.638 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:22.638 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f438c066f90 msgr2=0x7f438c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f438c066f90 0x7f438c069450 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f439c009c20 tx=0x7f439c009190 comp rx=0 tx=0).stop 2024-04-17T10:28:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ac15db90 msgr2=0x7f43ac15bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ac15db90 0x7f43ac15bc90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f43980ab000 tx=0x7f43980190f0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 shutdown_connections 2024-04-17T10:28:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f438c066f90 0x7f438c069450 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f43ac1615e0 0x7f43ac14d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ac15db90 0x7f43ac15bc90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 --2- 172.21.15.67:0/321082591 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f43ac071a50 0x7f43ac0b4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 >> 172.21.15.67:0/321082591 conn(0x7f43ac0b3460 msgr2=0x7f43ac15a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 shutdown_connections 2024-04-17T10:28:22.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.638+0000 7f43b1200640 1 -- 172.21.15.67:0/321082591 wait complete. 2024-04-17T10:28:22.652 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_0' returned '{'return_code': 0, 'scrub_tag': '592accb4-e553-428f-ab07-a88dfb6539cb', 'mode': 'asynchronous'}' 2024-04-17T10:28:22.653 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-17T10:28:22.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-17T10:28:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.802+0000 7f34f0800640 1 Processor -- start 2024-04-17T10:28:22.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.802+0000 7f34f0800640 1 -- start start 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.802+0000 7f34f0800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e815b780 0x7f34e80b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.802+0000 7f34f0800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e80d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.802+0000 7f34f0800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34e815e0d0 con 0x7f34e80b1fc0 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.802+0000 7f34f0800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34e80b3cf0 con 0x7f34e815b780 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.802+0000 7f34f0800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e815e390 con 0x7f34e806b3a0 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ede00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34e815b780 0x7f34e80b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ed400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f34e80b1fc0 0x7f34e806ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59650/0 (socket says 172.21.15.67:59650) 2024-04-17T10:28:22.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ed400640 1 -- 172.21.15.67:0/2129370256 learned_addr learned my addr 172.21.15.67:0/2129370256 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:22.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ee800640 1 --2- 172.21.15.67:0/2129370256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e80d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ede00640 1 -- 172.21.15.67:0/2129370256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 msgr2=0x7f34e80d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ede00640 1 --2- 172.21.15.67:0/2129370256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e80d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ede00640 1 -- 172.21.15.67:0/2129370256 >> v1:172.21.15.67:6789/0 conn(0x7f34e80b1fc0 legacy=0x7f34e806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ede00640 1 -- 172.21.15.67:0/2129370256 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e815e670 con 0x7f34e815b780 2024-04-17T10:28:22.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34eca00640 1 -- 172.21.15.67:0/2129370256 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808773721 0 0) 0x7f34e815e0d0 con 0x7f34e80b1fc0 2024-04-17T10:28:22.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ee800640 1 --2- 172.21.15.67:0/2129370256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e80d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:22.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34ede00640 1 --2- 172.21.15.67:0/2129370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e815b780 0x7f34e80b37b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0097c0 tx=0x7f34dc01c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e69484d8a6eceebc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34eca00640 1 -- 172.21.15.67:0/2129370256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0b0070 con 0x7f34e815b780 2024-04-17T10:28:22.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34eca00640 1 -- 172.21.15.67:0/2129370256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc004230 con 0x7f34e815b780 2024-04-17T10:28:22.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.806+0000 7f34eca00640 1 -- 172.21.15.67:0/2129370256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc003620 con 0x7f34e815b780 2024-04-17T10:28:22.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- 172.21.15.67:0/2129370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e815b780 msgr2=0x7f34e80b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 --2- 172.21.15.67:0/2129370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e815b780 0x7f34e80b37b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0097c0 tx=0x7f34dc01c350 comp rx=0 tx=0).stop 2024-04-17T10:28:22.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- 172.21.15.67:0/2129370256 shutdown_connections 2024-04-17T10:28:22.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 --2- 172.21.15.67:0/2129370256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e80d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 --2- 172.21.15.67:0/2129370256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e815b780 0x7f34e80b37b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- 172.21.15.67:0/2129370256 >> 172.21.15.67:0/2129370256 conn(0x7f34e80b5ef0 msgr2=0x7f34e8099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- 172.21.15.67:0/2129370256 shutdown_connections 2024-04-17T10:28:22.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- 172.21.15.67:0/2129370256 wait complete. 2024-04-17T10:28:22.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 Processor -- start 2024-04-17T10:28:22.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- start start 2024-04-17T10:28:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e8159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80b1fc0 0x7f34e813cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e815b780 0x7f34e806eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80d13e0 con 0x7f34e80b1fc0 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80d1260 con 0x7f34e815b780 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34f0800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e80d1560 con 0x7f34e806b3a0 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80b1fc0 0x7f34e813cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80b1fc0 0x7f34e813cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60090/0 (socket says 172.21.15.67:60090) 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 -- 172.21.15.67:0/2245713512 learned_addr learned my addr 172.21.15.67:0/2245713512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ede00640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e8159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ee800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e815b780 0x7f34e806eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 -- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 msgr2=0x7f34e8159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e8159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 -- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e815b780 msgr2=0x7f34e806eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e815b780 0x7f34e806eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 -- 172.21.15.67:0/2245713512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e81434f0 con 0x7f34e80b1fc0 2024-04-17T10:28:22.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ee800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e815b780 0x7f34e806eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.810+0000 7f34ed400640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80b1fc0 0x7f34e813cd60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f34d800c480 tx=0x7f34d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34e2000640 1 -- 172.21.15.67:0/2245713512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8018070 con 0x7f34e80b1fc0 2024-04-17T10:28:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34e2000640 1 -- 172.21.15.67:0/2245713512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d8007500 con 0x7f34e80b1fc0 2024-04-17T10:28:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34e81437d0 con 0x7f34e80b1fc0 2024-04-17T10:28:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34e2000640 1 -- 172.21.15.67:0/2245713512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8004a30 con 0x7f34e80b1fc0 2024-04-17T10:28:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34e8143d90 con 0x7f34e80b1fc0 2024-04-17T10:28:22.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34e2000640 1 -- 172.21.15.67:0/2245713512 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f34d801b070 con 0x7f34e80b1fc0 2024-04-17T10:28:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34e2000640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f34cc066f60 0x7f34cc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34e2000640 1 -- 172.21.15.67:0/2245713512 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f34d8084460 con 0x7f34e80b1fc0 2024-04-17T10:28:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34ede00640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f34cc066f60 0x7f34cc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34f3d28000 1 Processor -- start 2024-04-17T10:28:22.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.814+0000 7f34f3d28000 1 -- start start 2024-04-17T10:28:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ede00640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f34cc066f60 0x7f34cc069420 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0ad000 tx=0x7f34dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ecdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6a2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c9a4efff0 con 0x556c9a6a30a0 2024-04-17T10:28:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c9a478150 con 0x556c9a6ec9d0 2024-04-17T10:28:22.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556c9a4ec090 con 0x556c9a6ed2f0 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60104/0 (socket says 172.21.15.67:60104) 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 -- 172.21.15.67:0/2258482460 learned_addr learned my addr 172.21.15.67:0/2258482460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ed400640 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6a2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ede00640 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ecdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 -- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 msgr2=0x556c9a6a2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6a2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 -- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 msgr2=0x556c9a6ecdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ecdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 -- 172.21.15.67:0/2258482460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c9a6860d0 con 0x556c9a6a30a0 2024-04-17T10:28:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34ee800640 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a3480 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f34e4004510 tx=0x7f34e40095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a9a349859a2672 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34d6a00640 1 -- 172.21.15.67:0/2258482460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4013780 con 0x556c9a6a30a0 2024-04-17T10:28:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34d6a00640 1 -- 172.21.15.67:0/2258482460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4019070 con 0x556c9a6a30a0 2024-04-17T10:28:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34d6a00640 1 -- 172.21.15.67:0/2258482460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4013b20 con 0x556c9a6a30a0 2024-04-17T10:28:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- 172.21.15.67:0/2258482460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 msgr2=0x556c9a6a3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a3480 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f34e4004510 tx=0x7f34e40095f0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- 172.21.15.67:0/2258482460 shutdown_connections 2024-04-17T10:28:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a3480 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6a2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 --2- 172.21.15.67:0/2258482460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ecdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- 172.21.15.67:0/2258482460 >> 172.21.15.67:0/2258482460 conn(0x556c9a69f020 msgr2=0x556c9a6a1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- 172.21.15.67:0/2258482460 shutdown_connections 2024-04-17T10:28:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- 172.21.15.67:0/2258482460 wait complete. 2024-04-17T10:28:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 -- start start 2024-04-17T10:28:22.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.818+0000 7f34f3d28000 1 Processor -- start 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6a30a0 0x556c9a6d17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6d1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6ed2f0 0x556c9a6d2270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c9a4b4250 con 0x556c9a6ed2f0 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c9a4aa690 con 0x556c9a6ec9d0 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556c9a478a40 con 0x556c9a6a30a0 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 10 client.0 _ll_register_callbacks cb 0x556c9a68de90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:22.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6ed2f0 0x556c9a6d2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ed400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6d1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6ed2f0 0x556c9a6d2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60116/0 (socket says 172.21.15.67:60116) 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 -- 172.21.15.67:0/653002583 learned_addr learned my addr 172.21.15.67:0/653002583 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ede00640 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6a30a0 0x556c9a6d17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 -- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6a30a0 msgr2=0x556c9a6d17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6a30a0 0x556c9a6d17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 -- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 msgr2=0x556c9a6d1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6d1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 -- 172.21.15.67:0/653002583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c9a59d1c0 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ed400640 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ee800640 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6ed2f0 0x556c9a6d2270 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f34e4016040 tx=0x7f34e4004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1985b0eccc967e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 1 -- 172.21.15.67:0/653002583 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e4013e80 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 1 -- 172.21.15.67:0/653002583 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f34e4029c70 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34e4019070 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 1 -- 172.21.15.67:0/653002583 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34e4030050 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 10 client.6141 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x556c9a599170 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 1 -- 172.21.15.67:0/653002583 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e4019070 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 1 -- 172.21.15.67:0/653002583 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f34e4033020 con 0x556c9a6ed2f0 2024-04-17T10:28:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 10 client.6141 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 10 client.6141 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 20 client.6141 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a6d2f50 0x556c9a6fec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 4 client.6141 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556c9a48bc60 con 0x556c9a6d2f50 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ede00640 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a6d2f50 0x556c9a6fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34ede00640 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a6d2f50 0x556c9a6fec90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dc086c5be9fbe69 server_cookie=c5844b60cad3867c in_seq=0 out_seq=0 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 10 client.6141 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 1 -- 172.21.15.67:0/653002583 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556c9a48bc60 con 0x556c9a6d2f50 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.822+0000 7f34cb400640 10 client.6141 handle_command_reply: tid=0 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 client.6141 shutdown 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6ed2f0 msgr2=0x556c9a6d2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6ed2f0 0x556c9a6d2270 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f34e4016040 tx=0x7f34e4004ac0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 shutdown_connections 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a6d2f50 0x556c9a6fec90 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6ed2f0 0x556c9a6d2270 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6d1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 --2- 172.21.15.67:0/653002583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6a30a0 0x556c9a6d17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34cb400640 0 client.6141 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 >> 172.21.15.67:0/653002583 conn(0x556c9a69f020 msgr2=0x556c9a6a1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 shutdown_connections 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 1 -- 172.21.15.67:0/653002583 wait complete. 2024-04-17T10:28:22.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.826+0000 7f34f3d28000 20 client.6141 trim_cache size 0 max 16384 2024-04-17T10:28:22.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.838+0000 7f34f3d28000 1 Processor -- start 2024-04-17T10:28:22.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.838+0000 7f34f3d28000 1 -- start start 2024-04-17T10:28:22.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ebb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ec040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8203020 con 0x556c9a6a30a0 2024-04-17T10:28:22.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8202220 con 0x556c9a6ec9d0 2024-04-17T10:28:22.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8140a70 con 0x556c9a6ed2f0 2024-04-17T10:28:22.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60132/0 (socket says 172.21.15.67:60132) 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 -- 172.21.15.67:0/3410788146 learned_addr learned my addr 172.21.15.67:0/3410788146 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ee800640 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ed400640 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ebb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 -- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 msgr2=0x556c9a6ec040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ec040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 -- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 msgr2=0x556c9a6ebb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ebb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 -- 172.21.15.67:0/3410788146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556c9a536de0 con 0x556c9a6a30a0 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ee800640 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ec040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:22.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34ede00640 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a2630 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f34dc008010 tx=0x7f34dc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11e79a5930ee7cbe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34d6000640 1 -- 172.21.15.67:0/3410788146 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0d1030 con 0x556c9a6a30a0 2024-04-17T10:28:22.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34d6000640 1 -- 172.21.15.67:0/3410788146 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc0bc040 con 0x556c9a6a30a0 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34d6000640 1 -- 172.21.15.67:0/3410788146 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0bb040 con 0x556c9a6a30a0 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- 172.21.15.67:0/3410788146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 msgr2=0x556c9a6a2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a2630 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f34dc008010 tx=0x7f34dc004740 comp rx=0 tx=0).stop 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- 172.21.15.67:0/3410788146 shutdown_connections 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ec040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a6ebb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 --2- 172.21.15.67:0/3410788146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a6a2630 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- 172.21.15.67:0/3410788146 >> 172.21.15.67:0/3410788146 conn(0x556c9a69f020 msgr2=0x556c9a6eb3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- 172.21.15.67:0/3410788146 shutdown_connections 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- 172.21.15.67:0/3410788146 wait complete. 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 -- start start 2024-04-17T10:28:22.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.842+0000 7f34f3d28000 1 Processor -- start 2024-04-17T10:28:22.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a74cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a74d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ff910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556c9a48ae00 con 0x556c9a6a30a0 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34e8141070 con 0x556c9a6ec9d0 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556c9a4ef300 con 0x556c9a6ed2f0 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 10 client.0 _ll_register_callbacks cb 0x556c9a66e040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a74cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a74cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60146/0 (socket says 172.21.15.67:60146) 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 -- 172.21.15.67:0/4144785534 learned_addr learned my addr 172.21.15.67:0/4144785534 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ee800640 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ff910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ed400640 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a74d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 -- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 msgr2=0x556c9a6ff910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ff910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 -- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 msgr2=0x556c9a74d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a74d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 -- 172.21.15.67:0/4144785534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556c9a4918e0 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ede00640 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a74cf30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f34dc01aa00 tx=0x7f34dc01ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da515bfebdadfbe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 1 -- 172.21.15.67:0/4144785534 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34dc0bb070 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 1 -- 172.21.15.67:0/4144785534 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f34dc003d30 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34dc0bc040 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 1 -- 172.21.15.67:0/4144785534 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34dc0b0040 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 10 client.6147 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x556c9a5498f0 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 1 -- 172.21.15.67:0/4144785534 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34dc0bc040 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 1 -- 172.21.15.67:0/4144785534 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f34dc0059a0 con 0x556c9a6a30a0 2024-04-17T10:28:22.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 10 client.6147 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 10 client.6147 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 20 client.6147 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a7002d0 0x556c9a7006b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 4 client.6147 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556c9a4fd530 con 0x556c9a7002d0 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ed400640 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a7002d0 0x556c9a7006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34ed400640 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a7002d0 0x556c9a7006b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bae3a925055fa096 server_cookie=9a99416e1ccadf91 in_seq=0 out_seq=0 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 10 client.6147 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 1 -- 172.21.15.67:0/4144785534 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+284 (crc 0 0 0) 0x556c9a4fd530 con 0x556c9a7002d0 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 10 client.6147 handle_command_reply: tid=0 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 client.6147 shutdown 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 msgr2=0x556c9a74cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a74cf30 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f34dc01aa00 tx=0x7f34dc01ab90 comp rx=0 tx=0).stop 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 shutdown_connections 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556c9a7002d0 0x556c9a7006b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556c9a6ed2f0 0x556c9a6ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556c9a6ec9d0 0x556c9a74d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 --2- 172.21.15.67:0/4144785534 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556c9a6a30a0 0x556c9a74cf30 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34cb400640 0 client.6147 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 >> 172.21.15.67:0/4144785534 conn(0x556c9a69f020 msgr2=0x556c9a6ebc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 shutdown_connections 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 1 -- 172.21.15.67:0/4144785534 wait complete. 2024-04-17T10:28:22.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.846+0000 7f34f3d28000 20 client.6147 trim_cache size 0 max 16384 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "592accb4-e553-428f-ab07-a88dfb6539cb": { 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_0", 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "592accb4-e553-428f-ab07-a88dfb6539cb", 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "" 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:22.847 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f34cc066f60 msgr2=0x7f34cc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f34cc066f60 0x7f34cc069420 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f34dc0ad000 tx=0x7f34dc008040 comp rx=0 tx=0).stop 2024-04-17T10:28:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80b1fc0 msgr2=0x7f34e813cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80b1fc0 0x7f34e813cd60 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f34d800c480 tx=0x7f34d800c940 comp rx=0 tx=0).stop 2024-04-17T10:28:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 shutdown_connections 2024-04-17T10:28:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f34cc066f60 0x7f34cc069420 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34e815b780 0x7f34e806eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34e80b1fc0 0x7f34e813cd60 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 --2- 172.21.15.67:0/2245713512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f34e806b3a0 0x7f34e8159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 >> 172.21.15.67:0/2245713512 conn(0x7f34e80b5ef0 msgr2=0x7f34e8099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 shutdown_connections 2024-04-17T10:28:22.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:22.850+0000 7f34f0800640 1 -- 172.21.15.67:0/2245713512 wait complete. 2024-04-17T10:28:25.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:28:25.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:28:27.861 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-17T10:28:27.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-17T10:28:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.022+0000 7f24ad200640 1 Processor -- start 2024-04-17T10:28:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.022+0000 7f24ad200640 1 -- start start 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24ad200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f24a8071af0 0x7f24a80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24ad200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24a815be70 0x7f24a815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24ad200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a815cb30 con 0x7f24a8159440 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24ad200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a80a8650 con 0x7f24a815be70 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24ad200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a815ce10 con 0x7f24a8071af0 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f24a815be70 0x7f24a815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f24a8159440 0x7f24a80713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59660/0 (socket says 172.21.15.67:59660) 2024-04-17T10:28:28.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a5e00640 1 -- 172.21.15.67:0/3594558204 learned_addr learned my addr 172.21.15.67:0/3594558204 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f249fe00640 1 --2- 172.21.15.67:0/3594558204 >> v2:172.21.15.86:3300/0 conn(0x7f24a8071af0 0x7f24a80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a6800640 1 -- 172.21.15.67:0/3594558204 >> v2:172.21.15.86:3300/0 conn(0x7f24a8071af0 msgr2=0x7f24a80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a6800640 1 --2- 172.21.15.67:0/3594558204 >> v2:172.21.15.86:3300/0 conn(0x7f24a8071af0 0x7f24a80ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a6800640 1 -- 172.21.15.67:0/3594558204 >> v1:172.21.15.67:6789/0 conn(0x7f24a8159440 legacy=0x7f24a80713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a6800640 1 -- 172.21.15.67:0/3594558204 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a80cd610 con 0x7f24a815be70 2024-04-17T10:28:28.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.026+0000 7f24a5400640 1 -- 172.21.15.67:0/3594558204 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896150671 0 0) 0x7f24a815cb30 con 0x7f24a8159440 2024-04-17T10:28:28.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24a6800640 1 --2- 172.21.15.67:0/3594558204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a815c2c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f24a80cf770 tx=0x7f24a001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6504cafea7796c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24a5400640 1 -- 172.21.15.67:0/3594558204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a00b2070 con 0x7f24a815be70 2024-04-17T10:28:28.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24a5400640 1 -- 172.21.15.67:0/3594558204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a00a59f0 con 0x7f24a815be70 2024-04-17T10:28:28.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24a5400640 1 -- 172.21.15.67:0/3594558204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a00a5dc0 con 0x7f24a815be70 2024-04-17T10:28:28.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 -- 172.21.15.67:0/3594558204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 msgr2=0x7f24a815c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 --2- 172.21.15.67:0/3594558204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a815c2c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f24a80cf770 tx=0x7f24a001c930 comp rx=0 tx=0).stop 2024-04-17T10:28:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 -- 172.21.15.67:0/3594558204 shutdown_connections 2024-04-17T10:28:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 --2- 172.21.15.67:0/3594558204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a815c2c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 --2- 172.21.15.67:0/3594558204 >> v2:172.21.15.86:3300/0 conn(0x7f24a8071af0 0x7f24a80ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 -- 172.21.15.67:0/3594558204 >> 172.21.15.67:0/3594558204 conn(0x7f24a80b1740 msgr2=0x7f24a80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 -- 172.21.15.67:0/3594558204 shutdown_connections 2024-04-17T10:28:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 -- 172.21.15.67:0/3594558204 wait complete. 2024-04-17T10:28:28.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 Processor -- start 2024-04-17T10:28:28.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.030+0000 7f24ad200640 1 -- start start 2024-04-17T10:28:28.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24a8071af0 0x7f24a80b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24a8159440 0x7f24a80b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a8153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a815d840 con 0x7f24a815be70 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a81598e0 con 0x7f24a8159440 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a815d9c0 con 0x7f24a8071af0 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a8153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a8153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60154/0 (socket says 172.21.15.67:60154) 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 -- 172.21.15.67:0/1177121996 learned_addr learned my addr 172.21.15.67:0/1177121996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f249fe00640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24a8159440 0x7f24a80b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a5e00640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24a8071af0 0x7f24a80b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 -- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24a8071af0 msgr2=0x7f24a80b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24a8071af0 0x7f24a80b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 -- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24a8159440 msgr2=0x7f24a80b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24a8159440 0x7f24a80b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 -- 172.21.15.67:0/1177121996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a806e4d0 con 0x7f24a815be70 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a5e00640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24a8071af0 0x7f24a80b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24a6800640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a8153820 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f24a00b0040 tx=0x7f24a00196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f249e000640 1 -- 172.21.15.67:0/1177121996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a00b2040 con 0x7f24a815be70 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f249e000640 1 -- 172.21.15.67:0/1177121996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a00bf070 con 0x7f24a815be70 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a80b3040 con 0x7f24a815be70 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f249e000640 1 -- 172.21.15.67:0/1177121996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a00b8890 con 0x7f24a815be70 2024-04-17T10:28:28.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a80b35a0 con 0x7f24a815be70 2024-04-17T10:28:28.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.034+0000 7f249e000640 1 -- 172.21.15.67:0/1177121996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f24a00b8a30 con 0x7f24a815be70 2024-04-17T10:28:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249e000640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2484066f60 0x7f2484069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249e000640 1 -- 172.21.15.67:0/1177121996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24a0120320 con 0x7f24a815be70 2024-04-17T10:28:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24a5e00640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2484066f60 0x7f2484069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 Processor -- start 2024-04-17T10:28:28.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 -- start start 2024-04-17T10:28:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24a5e00640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2484066f60 0x7f2484069420 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2498000f70 tx=0x7f2498009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46dfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a4695b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46960a0 0x55b9a4696480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a44e2ff0 con 0x55b9a46e02f0 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a446b150 con 0x55b9a46960a0 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9a44df090 con 0x55b9a46df9d0 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a4695b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a4695b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60158/0 (socket says 172.21.15.67:60158) 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 -- 172.21.15.67:0/1012662031 learned_addr learned my addr 172.21.15.67:0/1012662031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24a5e00640 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46dfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24a6800640 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46960a0 0x55b9a4696480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 -- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 msgr2=0x55b9a46dfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46dfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 -- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46960a0 msgr2=0x55b9a4696480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46960a0 0x55b9a4696480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 -- 172.21.15.67:0/1012662031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9a46790d0 con 0x55b9a46e02f0 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f249fe00640 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a4695b60 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f249000b8b0 tx=0x7f249000bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b5b673cce4d4f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f2496a00640 1 -- 172.21.15.67:0/1012662031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490004230 con 0x55b9a46e02f0 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f2496a00640 1 -- 172.21.15.67:0/1012662031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24900043d0 con 0x55b9a46e02f0 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f2496a00640 1 -- 172.21.15.67:0/1012662031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249000f9c0 con 0x55b9a46e02f0 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 -- 172.21.15.67:0/1012662031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 msgr2=0x55b9a4695b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.038+0000 7f24b076c000 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a4695b60 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f249000b8b0 tx=0x7f249000bd70 comp rx=0 tx=0).stop 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- 172.21.15.67:0/1012662031 shutdown_connections 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46960a0 0x55b9a4696480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a4695b60 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 --2- 172.21.15.67:0/1012662031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46dfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- 172.21.15.67:0/1012662031 >> 172.21.15.67:0/1012662031 conn(0x55b9a4692020 msgr2=0x55b9a4694440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- 172.21.15.67:0/1012662031 shutdown_connections 2024-04-17T10:28:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- 172.21.15.67:0/1012662031 wait complete. 2024-04-17T10:28:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- start start 2024-04-17T10:28:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 Processor -- start 2024-04-17T10:28:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a46c47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a44a7250 con 0x55b9a46960a0 2024-04-17T10:28:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a449d690 con 0x55b9a46e02f0 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9a446ba40 con 0x55b9a46df9d0 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 10 client.0 _ll_register_callbacks cb 0x55b9a4680e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a46c47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a46c47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60174/0 (socket says 172.21.15.67:60174) 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 -- 172.21.15.67:0/3500306476 learned_addr learned my addr 172.21.15.67:0/3500306476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f249fe00640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a6800640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 -- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 msgr2=0x55b9a46c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 -- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 msgr2=0x55b9a46c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 -- 172.21.15.67:0/3500306476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9a45901c0 con 0x55b9a46960a0 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f249fe00640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:28.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24a5e00640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a46c47f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2498005b30 tx=0x7f2498001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba678e50dfd7564 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f248b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f248b400640 1 -- 172.21.15.67:0/3500306476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498027060 con 0x55b9a46960a0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f248b400640 1 -- 172.21.15.67:0/3500306476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2498022c90 con 0x55b9a46960a0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24900043d0 con 0x55b9a46960a0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f248b400640 1 -- 172.21.15.67:0/3500306476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2498012410 con 0x55b9a46960a0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 10 client.6162 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55b9a458c170 con 0x55b9a46960a0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f248b400640 1 -- 172.21.15.67:0/3500306476 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24900043d0 con 0x55b9a46960a0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f248b400640 1 -- 172.21.15.67:0/3500306476 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f24980125b0 con 0x55b9a46960a0 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 10 client.6162 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 10 client.6162 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.042+0000 7f24b076c000 20 client.6162 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46efc80 0x55b9a46f2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 4 client.6162 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9a447ec60 con 0x55b9a46efc80 2024-04-17T10:28:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f249fe00640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46efc80 0x55b9a46f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f249fe00640 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46efc80 0x55b9a46f2070 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d980faf7dfc094c server_cookie=c12fda91d4b0f935 in_seq=0 out_seq=0 2024-04-17T10:28:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f248b400640 10 client.6162 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f248b400640 1 -- 172.21.15.67:0/3500306476 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b9a447ec60 con 0x55b9a46efc80 2024-04-17T10:28:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f248b400640 10 client.6162 handle_command_reply: tid=0 2024-04-17T10:28:28.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 client.6162 shutdown 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 msgr2=0x55b9a46c47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a46c47f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f2498005b30 tx=0x7f2498001090 comp rx=0 tx=0).stop 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 shutdown_connections 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46efc80 0x55b9a46f2070 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a46c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 --2- 172.21.15.67:0/3500306476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a46c47f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 >> 172.21.15.67:0/3500306476 conn(0x55b9a4692020 msgr2=0x55b9a4694ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f248b400640 0 client.6162 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 shutdown_connections 2024-04-17T10:28:28.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 1 -- 172.21.15.67:0/3500306476 wait complete. 2024-04-17T10:28:28.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.046+0000 7f24b076c000 20 client.6162 trim_cache size 0 max 16384 2024-04-17T10:28:28.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 Processor -- start 2024-04-17T10:28:28.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 -- start start 2024-04-17T10:28:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46960a0 0x55b9a4695a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46df9d0 0x55b9a46ece40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a46ed380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a8202f60 con 0x55b9a46e02f0 2024-04-17T10:28:28.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a8202100 con 0x55b9a46df9d0 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a814c5b0 con 0x55b9a46960a0 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a46ed380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a46ed380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60184/0 (socket says 172.21.15.67:60184) 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 -- 172.21.15.67:0/574844441 learned_addr learned my addr 172.21.15.67:0/574844441 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 -- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46960a0 msgr2=0x55b9a4695a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46960a0 0x55b9a4695a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 -- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46df9d0 msgr2=0x55b9a46ece40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f249fe00640 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46df9d0 0x55b9a46ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46df9d0 0x55b9a46ece40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 -- 172.21.15.67:0/574844441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9a4529de0 con 0x55b9a46e02f0 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f249fe00640 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46df9d0 0x55b9a46ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24a6800640 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a46ed380 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f24a0098510 tx=0x7f24a0120040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb9b9984daa76bd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f2496000640 1 -- 172.21.15.67:0/574844441 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a00f4b50 con 0x55b9a46e02f0 2024-04-17T10:28:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f2496000640 1 -- 172.21.15.67:0/574844441 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a00f9110 con 0x55b9a46e02f0 2024-04-17T10:28:28.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f2496000640 1 -- 172.21.15.67:0/574844441 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a00eed00 con 0x55b9a46e02f0 2024-04-17T10:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 -- 172.21.15.67:0/574844441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 msgr2=0x55b9a46ed380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.062+0000 7f24b076c000 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a46ed380 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f24a0098510 tx=0x7f24a0120040 comp rx=0 tx=0).stop 2024-04-17T10:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- 172.21.15.67:0/574844441 shutdown_connections 2024-04-17T10:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46e02f0 0x55b9a46ed380 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46df9d0 0x55b9a46ece40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 --2- 172.21.15.67:0/574844441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46960a0 0x55b9a4695a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- 172.21.15.67:0/574844441 >> 172.21.15.67:0/574844441 conn(0x55b9a4692020 msgr2=0x55b9a46ec730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- 172.21.15.67:0/574844441 shutdown_connections 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- 172.21.15.67:0/574844441 wait complete. 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- start start 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 Processor -- start 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a473f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a473f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46f1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9a447de00 con 0x55b9a46960a0 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a8157190 con 0x55b9a46e02f0 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9a44e2300 con 0x55b9a46df9d0 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 10 client.0 _ll_register_callbacks cb 0x55b9a4661040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:28.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a473f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a473f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60200/0 (socket says 172.21.15.67:60200) 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 -- 172.21.15.67:0/3168299148 learned_addr learned my addr 172.21.15.67:0/3168299148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f249fe00640 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a473f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a6800640 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 -- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 msgr2=0x55b9a473f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a473f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 -- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 msgr2=0x55b9a46f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46f1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 -- 172.21.15.67:0/3168299148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9a44848e0 con 0x55b9a46960a0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24a5e00640 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a473f300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f24980071e0 tx=0x7f2498007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a7adb09a5d55a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f248b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f248b400640 1 -- 172.21.15.67:0/3168299148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498013e60 con 0x55b9a46960a0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f248b400640 1 -- 172.21.15.67:0/3168299148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2498021660 con 0x55b9a46960a0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f24a00f9110 con 0x55b9a46960a0 2024-04-17T10:28:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f248b400640 1 -- 172.21.15.67:0/3168299148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24980013e0 con 0x55b9a46960a0 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 10 client.6168 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55b9a453c8f0 con 0x55b9a46960a0 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f248b400640 1 -- 172.21.15.67:0/3168299148 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a00f9110 con 0x55b9a46960a0 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f248b400640 1 -- 172.21.15.67:0/3168299148 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f24980218a0 con 0x55b9a46960a0 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 10 client.6168 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 10 client.6168 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 20 client.6168 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46f23b0 0x55b9a46f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 4 client.6168 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.066+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b9a44f0530 con 0x55b9a46f23b0 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f249fe00640 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46f23b0 0x55b9a46f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f249fe00640 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46f23b0 0x55b9a46f27c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5247435ec0147175 server_cookie=ebfc02c7ac48a85d in_seq=0 out_seq=0 2024-04-17T10:28:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f248b400640 10 client.6168 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f248b400640 1 -- 172.21.15.67:0/3168299148 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b9a44f0530 con 0x55b9a46f23b0 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f248b400640 10 client.6168 handle_command_reply: tid=0 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 client.6168 shutdown 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 msgr2=0x55b9a473f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a473f300 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f24980071e0 tx=0x7f2498007000 comp rx=0 tx=0).stop 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 shutdown_connections 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55b9a46f23b0 0x55b9a46f27c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55b9a46e02f0 0x55b9a46f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55b9a46df9d0 0x55b9a473f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 --2- 172.21.15.67:0/3168299148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b9a46960a0 0x55b9a473f300 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f248b400640 0 client.6168 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 >> 172.21.15.67:0/3168299148 conn(0x55b9a4692020 msgr2=0x55b9a46ecfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 shutdown_connections 2024-04-17T10:28:28.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 1 -- 172.21.15.67:0/3168299148 wait complete. 2024-04-17T10:28:28.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24b076c000 20 client.6168 trim_cache size 0 max 16384 2024-04-17T10:28:28.068 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:28.068 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-04-17T10:28:28.068 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:28:28.068 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2484066f60 msgr2=0x7f2484069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2484066f60 0x7f2484069420 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2498000f70 tx=0x7f2498009040 comp rx=0 tx=0).stop 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 msgr2=0x7f24a8153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a8153820 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f24a00b0040 tx=0x7f24a00196d0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 shutdown_connections 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2484066f60 0x7f2484069420 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f24a815be70 0x7f24a8153820 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f24a8159440 0x7f24a80b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 --2- 172.21.15.67:0/1177121996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24a8071af0 0x7f24a80b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 >> 172.21.15.67:0/1177121996 conn(0x7f24a80b1740 msgr2=0x7f24a806ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 shutdown_connections 2024-04-17T10:28:28.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.070+0000 7f24ad200640 1 -- 172.21.15.67:0/1177121996 wait complete. 2024-04-17T10:28:28.083 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-17T10:28:28.083 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:28.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-04-17T10:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0513200640 1 Processor -- start 2024-04-17T10:28:28.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0513200640 1 -- start start 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0513200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0513200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f050c0cce60 0x7f050c06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0513200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050c0b4990 con 0x7f050c06e7b0 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0513200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f050c0cd2c0 con 0x7f050c0cce60 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0513200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c0af340 con 0x7f050c0aef60 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0511200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f050c06e7b0 0x7f050c0713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59664/0 (socket says 172.21.15.67:59664) 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f0511200640 1 -- 172.21.15.67:0/3980997497 learned_addr learned my addr 172.21.15.67:0/3980997497 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f050b400640 1 --2- 172.21.15.67:0/3980997497 >> v2:172.21.15.67:3300/0 conn(0x7f050c0cce60 0x7f050c06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f050be00640 1 --2- 172.21.15.67:0/3980997497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f050aa00640 1 -- 172.21.15.67:0/3980997497 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209957918 0 0) 0x7f05040014a0 con 0x7f050c06e7b0 2024-04-17T10:28:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f050b400640 1 -- 172.21.15.67:0/3980997497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 msgr2=0x7f050c0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f050b400640 1 --2- 172.21.15.67:0/3980997497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c0cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f050b400640 1 -- 172.21.15.67:0/3980997497 >> v1:172.21.15.67:6789/0 conn(0x7f050c06e7b0 legacy=0x7f050c0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.234+0000 7f050b400640 1 -- 172.21.15.67:0/3980997497 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c15ae90 con 0x7f050c0cce60 2024-04-17T10:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f050b400640 1 --2- 172.21.15.67:0/3980997497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c06e270 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f04f80097c0 tx=0x7f04f801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685a1e0c476e86c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f050aa00640 1 -- 172.21.15.67:0/3980997497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f80b0070 con 0x7f050c0cce60 2024-04-17T10:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f050aa00640 1 -- 172.21.15.67:0/3980997497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f80ab600 con 0x7f050c0cce60 2024-04-17T10:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f050aa00640 1 -- 172.21.15.67:0/3980997497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f80ab9d0 con 0x7f050c0cce60 2024-04-17T10:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 -- 172.21.15.67:0/3980997497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 msgr2=0x7f050c06e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 --2- 172.21.15.67:0/3980997497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c06e270 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f04f80097c0 tx=0x7f04f801c410 comp rx=0 tx=0).stop 2024-04-17T10:28:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 -- 172.21.15.67:0/3980997497 shutdown_connections 2024-04-17T10:28:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 --2- 172.21.15.67:0/3980997497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c06e270 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 --2- 172.21.15.67:0/3980997497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c0cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 -- 172.21.15.67:0/3980997497 >> 172.21.15.67:0/3980997497 conn(0x7f050c0b1740 msgr2=0x7f050c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 -- 172.21.15.67:0/3980997497 shutdown_connections 2024-04-17T10:28:28.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 -- 172.21.15.67:0/3980997497 wait complete. 2024-04-17T10:28:28.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.238+0000 7f0513200640 1 Processor -- start 2024-04-17T10:28:28.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 -- start start 2024-04-17T10:28:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f050c06e7b0 0x7f050c137110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c137950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c15bc10 con 0x7f050c0cce60 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c15b410 con 0x7f050c06e7b0 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c151fb0 con 0x7f050c0aef60 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c137950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f050be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f050c06e7b0 0x7f050c137110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c137950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60218/0 (socket says 172.21.15.67:60218) 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 -- 172.21.15.67:0/191273302 learned_addr learned my addr 172.21.15.67:0/191273302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f050b400640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 -- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 msgr2=0x7f050c14b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c14b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 -- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f050c06e7b0 msgr2=0x7f050c137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f050c06e7b0 0x7f050c137110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 -- 172.21.15.67:0/191273302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050c152290 con 0x7f050c0cce60 2024-04-17T10:28:28.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0511200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c137950 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f050400c730 tx=0x7f050400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0503e00640 1 -- 172.21.15.67:0/191273302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504018070 con 0x7f050c0cce60 2024-04-17T10:28:28.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0503e00640 1 -- 172.21.15.67:0/191273302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050400ce40 con 0x7f050c0cce60 2024-04-17T10:28:28.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0503e00640 1 -- 172.21.15.67:0/191273302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050400a860 con 0x7f050c0cce60 2024-04-17T10:28:28.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f050c152570 con 0x7f050c0cce60 2024-04-17T10:28:28.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.242+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f050c152b30 con 0x7f050c0cce60 2024-04-17T10:28:28.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0503e00640 1 -- 172.21.15.67:0/191273302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f050401b070 con 0x7f050c0cce60 2024-04-17T10:28:28.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0503e00640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0503e00640 1 -- 172.21.15.67:0/191273302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0504084750 con 0x7f050c0cce60 2024-04-17T10:28:28.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f050be00640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 Processor -- start 2024-04-17T10:28:28.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 -- start start 2024-04-17T10:28:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f050be00640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009ac0 tx=0x7f04fc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04ff07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff0cf0 0x55ac04fa64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04fa6a10 0x55ac04fa6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac04deff20 con 0x55ac04fa6a10 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac04d78000 con 0x55ac04ff03d0 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac04debfc0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04fa6a10 0x55ac04fa6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04fa6a10 0x55ac04fa6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60234/0 (socket says 172.21.15.67:60234) 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 -- 172.21.15.67:0/3594472672 learned_addr learned my addr 172.21.15.67:0/3594472672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f050b400640 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff0cf0 0x55ac04fa64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f050be00640 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04ff07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 -- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff0cf0 msgr2=0x55ac04fa64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff0cf0 0x55ac04fa64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 -- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 msgr2=0x55ac04ff07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04ff07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 -- 172.21.15.67:0/3594472672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac04dcbba0 con 0x55ac04fa6a10 2024-04-17T10:28:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0511200640 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04fa6a10 0x55ac04fa6df0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f05040829c0 tx=0x7f0504054ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8577a04acb9dd02 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0500c00640 1 -- 172.21.15.67:0/3594472672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050405d540 con 0x55ac04fa6a10 2024-04-17T10:28:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0500c00640 1 -- 172.21.15.67:0/3594472672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504052f80 con 0x55ac04fa6a10 2024-04-17T10:28:28.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.246+0000 7f0500c00640 1 -- 172.21.15.67:0/3594472672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504051550 con 0x55ac04fa6a10 2024-04-17T10:28:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- 172.21.15.67:0/3594472672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04fa6a10 msgr2=0x55ac04fa6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04fa6a10 0x55ac04fa6df0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f05040829c0 tx=0x7f0504054ee0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- 172.21.15.67:0/3594472672 shutdown_connections 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04fa6a10 0x55ac04fa6df0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff0cf0 0x55ac04fa64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 --2- 172.21.15.67:0/3594472672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04ff07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- 172.21.15.67:0/3594472672 >> 172.21.15.67:0/3594472672 conn(0x55ac04fa8760 msgr2=0x55ac04fa09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- 172.21.15.67:0/3594472672 shutdown_connections 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- 172.21.15.67:0/3594472672 wait complete. 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- start start 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 Processor -- start 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04fa6a10 0x55ac04fb1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff03d0 0x55ac04fb1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fb2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac04daa5d0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac04ea59d0 con 0x55ac04fa6a10 2024-04-17T10:28:28.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac04da6dc0 con 0x55ac04ff03d0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 10 client.0 _ll_register_callbacks cb 0x55ac04f8da60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fb2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fb2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60248/0 (socket says 172.21.15.67:60248) 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 -- 172.21.15.67:0/4060252637 learned_addr learned my addr 172.21.15.67:0/4060252637 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f050be00640 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04fa6a10 0x55ac04fb1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f050b400640 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff03d0 0x55ac04fb1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 -- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff03d0 msgr2=0x55ac04fb1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff03d0 0x55ac04fb1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 -- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04fa6a10 msgr2=0x55ac04fb1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04fa6a10 0x55ac04fb1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 -- 172.21.15.67:0/4060252637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac04e365a0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f0511200640 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fb2080 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0504053190 tx=0x7f0504013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7437f114b88b2e87 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f04f5600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:28.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f04f5600640 1 -- 172.21.15.67:0/4060252637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05040532a0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f04f5600640 1 -- 172.21.15.67:0/4060252637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f050406cbc0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0504052f80 con 0x55ac04ff0cf0 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f04f5600640 1 -- 172.21.15.67:0/4060252637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f050406ce00 con 0x55ac04ff0cf0 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 10 client.6183 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55ac04dc72b0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.250+0000 7f04f5600640 1 -- 172.21.15.67:0/4060252637 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504052f80 con 0x55ac04ff0cf0 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f04f5600640 1 -- 172.21.15.67:0/4060252637 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0504075030 con 0x55ac04ff0cf0 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 10 client.6183 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 10 client.6183 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 20 client.6183 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac04fb2950 0x55ac05001b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 4 client.6183 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac04d8bc70 con 0x55ac04fb2950 2024-04-17T10:28:28.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f050be00640 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac04fb2950 0x55ac05001b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f050be00640 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac04fb2950 0x55ac05001b50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13515328331018ad server_cookie=2d5abed58be31c09 in_seq=0 out_seq=0 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f04f5600640 10 client.6183 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f04f5600640 1 -- 172.21.15.67:0/4060252637 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ac04d8bc70 con 0x55ac04fb2950 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f04f5600640 10 client.6183 handle_command_reply: tid=0 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 client.6183 shutdown 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 msgr2=0x55ac04fb2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fb2080 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f0504053190 tx=0x7f0504013000 comp rx=0 tx=0).stop 2024-04-17T10:28:28.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 shutdown_connections 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac04fb2950 0x55ac05001b50 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fb2080 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04ff03d0 0x55ac04fb1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 --2- 172.21.15.67:0/4060252637 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04fa6a10 0x55ac04fb1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 >> 172.21.15.67:0/4060252637 conn(0x55ac04fa8760 msgr2=0x55ac04fa7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f04f5600640 0 client.6183 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 shutdown_connections 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 1 -- 172.21.15.67:0/4060252637 wait complete. 2024-04-17T10:28:28.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.254+0000 7f05166f0000 20 client.6183 trim_cache size 0 max 16384 2024-04-17T10:28:28.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 Processor -- start 2024-04-17T10:28:28.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 -- start start 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac05003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04fad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fadb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c1ff220 con 0x55ac04ff0cf0 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c15d5a0 con 0x55ac04ff03d0 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f050c14f8a0 con 0x55ac04fa6a10 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fadb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fadb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60260/0 (socket says 172.21.15.67:60260) 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 -- 172.21.15.67:0/1946381316 learned_addr learned my addr 172.21.15.67:0/1946381316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f050b400640 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04fad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f050be00640 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac05003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 -- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 msgr2=0x55ac05003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac05003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 -- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 msgr2=0x55ac04fad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04fad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.266+0000 7f0511200640 1 -- 172.21.15.67:0/1946381316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac04d78250 con 0x55ac04ff0cf0 2024-04-17T10:28:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050be00640 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac05003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:28.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f0511200640 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fadb80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f05040546f0 tx=0x7f0504009070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8101d35939aff299 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f04f7e00640 1 -- 172.21.15.67:0/1946381316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05040532a0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f04f7e00640 1 -- 172.21.15.67:0/1946381316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050406cbc0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f04f7e00640 1 -- 172.21.15.67:0/1946381316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504050c30 con 0x55ac04ff0cf0 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- 172.21.15.67:0/1946381316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 msgr2=0x55ac04fadb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fadb80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f05040546f0 tx=0x7f0504009070 comp rx=0 tx=0).stop 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- 172.21.15.67:0/1946381316 shutdown_connections 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff0cf0 0x55ac04fadb80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff03d0 0x55ac04fad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 --2- 172.21.15.67:0/1946381316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac05003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- 172.21.15.67:0/1946381316 >> 172.21.15.67:0/1946381316 conn(0x55ac04fa8760 msgr2=0x55ac04facf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- 172.21.15.67:0/1946381316 shutdown_connections 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- 172.21.15.67:0/1946381316 wait complete. 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- start start 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 Processor -- start 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac04fef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff03d0 0x55ac05002ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff0cf0 0x55ac050030e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f050c15cfa0 con 0x55ac04ff03d0 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac04da6dc0 con 0x55ac04ff0cf0 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ac04deff20 con 0x55ac04fa6a10 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f05166f0000 10 client.0 _ll_register_callbacks cb 0x55ac04f6e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff03d0 0x55ac05002ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff03d0 0x55ac05002ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60268/0 (socket says 172.21.15.67:60268) 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 -- 172.21.15.67:0/121317685 learned_addr learned my addr 172.21.15.67:0/121317685 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050be00640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac04fef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f0511200640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff0cf0 0x55ac050030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 -- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 msgr2=0x55ac04fef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac04fef9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 -- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff0cf0 msgr2=0x55ac050030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff0cf0 0x55ac050030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f050b400640 1 -- 172.21.15.67:0/121317685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ac04d84b90 con 0x55ac04ff03d0 2024-04-17T10:28:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.270+0000 7f0511200640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff0cf0 0x55ac050030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f050b400640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff03d0 0x55ac05002ba0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f04f801c450 tx=0x7f04f801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d66c9f01947cc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 1 -- 172.21.15.67:0/121317685 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f80b0040 con 0x55ac04ff03d0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 1 -- 172.21.15.67:0/121317685 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f04f80c2480 con 0x55ac04ff03d0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f050406cbc0 con 0x55ac04ff03d0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 1 -- 172.21.15.67:0/121317685 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04f80ca050 con 0x55ac04ff03d0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 10 client.6189 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55ac04d97010 con 0x55ac04ff03d0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 1 -- 172.21.15.67:0/121317685 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050406cbc0 con 0x55ac04ff03d0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 1 -- 172.21.15.67:0/121317685 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f04f80cd020 con 0x55ac04ff03d0 2024-04-17T10:28:28.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 10 client.6189 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 10 client.6189 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 20 client.6189 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac05050040 0x55ac05050450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 4 client.6189 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ac04dfd460 con 0x55ac05050040 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f050be00640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac05050040 0x55ac05050450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f050be00640 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac05050040 0x55ac05050450 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d62e7609afde4d6 server_cookie=7fb8918e5dd7c9f2 in_seq=0 out_seq=0 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 10 client.6189 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 1 -- 172.21.15.67:0/121317685 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+168 (crc 0 0 0) 0x55ac04dfd460 con 0x55ac05050040 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 10 client.6189 handle_command_reply: tid=0 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 client.6189 shutdown 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff03d0 msgr2=0x55ac05002ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff03d0 0x55ac05002ba0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f04f801c450 tx=0x7f04f801ac50 comp rx=0 tx=0).stop 2024-04-17T10:28:28.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 shutdown_connections 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ac05050040 0x55ac05050450 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ac04ff0cf0 0x55ac050030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ac04ff03d0 0x55ac05002ba0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 --2- 172.21.15.67:0/121317685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ac04fa6a10 0x55ac04fef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f04f5600640 0 client.6189 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 >> 172.21.15.67:0/121317685 conn(0x55ac04fa8760 msgr2=0x55ac04fad7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 shutdown_connections 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 1 -- 172.21.15.67:0/121317685 wait complete. 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f05166f0000 20 client.6189 trim_cache size 0 max 16384 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:28:28.272 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:28.273 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "backtrace", 2024-04-17T10:28:28.273 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 4224541580, 2024-04-17T10:28:28.273 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511630187, 2024-04-17T10:28:28.273 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_0" 2024-04-17T10:28:28.273 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:28.273 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 msgr2=0x7f04ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009ac0 tx=0x7f04fc009190 comp rx=0 tx=0).stop 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 msgr2=0x7f050c137950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c137950 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f050400c730 tx=0x7f050400cbf0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 shutdown_connections 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f050c0cce60 0x7f050c137950 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f050c0aef60 0x7f050c14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 --2- 172.21.15.67:0/191273302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f050c06e7b0 0x7f050c137110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 >> 172.21.15.67:0/191273302 conn(0x7f050c0b1740 msgr2=0x7f050c0afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 shutdown_connections 2024-04-17T10:28:28.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.274+0000 7f0513200640 1 -- 172.21.15.67:0/191273302 wait complete. 2024-04-17T10:28:28.288 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:28.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.434+0000 7f92a2800640 1 Processor -- start 2024-04-17T10:28:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.434+0000 7f92a2800640 1 -- start start 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f92a2800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f929c145040 0x7f929c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f92a2800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c06ef10 0x7f929c071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f92a2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f929c071850 0x7f929c15f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f92a2800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c0a86c0 con 0x7f929c071850 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f92a2800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c15bf10 con 0x7f929c145040 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f92a2800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c071ca0 con 0x7f929c06ef10 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f929c071850 0x7f929c15f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f929c071850 0x7f929c15f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60270/0 (socket says 172.21.15.67:60270) 2024-04-17T10:28:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 -- 172.21.15.67:0/3819132946 learned_addr learned my addr 172.21.15.67:0/3819132946 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929b400640 1 --2- 172.21.15.67:0/3819132946 >> v2:172.21.15.86:3300/0 conn(0x7f929c145040 0x7f929c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f9293e00640 1 --2- 172.21.15.67:0/3819132946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c06ef10 0x7f929c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 -- 172.21.15.67:0/3819132946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c06ef10 msgr2=0x7f929c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 --2- 172.21.15.67:0/3819132946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c06ef10 0x7f929c071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 -- 172.21.15.67:0/3819132946 >> v2:172.21.15.86:3300/0 conn(0x7f929c145040 msgr2=0x7f929c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 --2- 172.21.15.67:0/3819132946 >> v2:172.21.15.86:3300/0 conn(0x7f929c145040 0x7f929c06e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.438+0000 7f929be00640 1 -- 172.21.15.67:0/3819132946 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c06e2f0 con 0x7f929c071850 2024-04-17T10:28:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f929be00640 1 --2- 172.21.15.67:0/3819132946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15f510 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f929c072410 tx=0x7f929401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ae6b81f170fc9b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f929aa00640 1 -- 172.21.15.67:0/3819132946 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92940b4070 con 0x7f929c071850 2024-04-17T10:28:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f929aa00640 1 -- 172.21.15.67:0/3819132946 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92940af400 con 0x7f929c071850 2024-04-17T10:28:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f929aa00640 1 -- 172.21.15.67:0/3819132946 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92940af7d0 con 0x7f929c071850 2024-04-17T10:28:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 -- 172.21.15.67:0/3819132946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 msgr2=0x7f929c15f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 --2- 172.21.15.67:0/3819132946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15f510 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f929c072410 tx=0x7f929401f1e0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 -- 172.21.15.67:0/3819132946 shutdown_connections 2024-04-17T10:28:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 --2- 172.21.15.67:0/3819132946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15f510 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 --2- 172.21.15.67:0/3819132946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c06ef10 0x7f929c071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 --2- 172.21.15.67:0/3819132946 >> v2:172.21.15.86:3300/0 conn(0x7f929c145040 0x7f929c06e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 -- 172.21.15.67:0/3819132946 >> 172.21.15.67:0/3819132946 conn(0x7f929c06ac90 msgr2=0x7f929c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 -- 172.21.15.67:0/3819132946 shutdown_connections 2024-04-17T10:28:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 -- 172.21.15.67:0/3819132946 wait complete. 2024-04-17T10:28:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 Processor -- start 2024-04-17T10:28:28.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.442+0000 7f92a2800640 1 -- start start 2024-04-17T10:28:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929c06ef10 0x7f929c143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c145040 0x7f929c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c06dca0 con 0x7f929c071850 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c06d9c0 con 0x7f929c06ef10 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c06df80 con 0x7f929c145040 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60280/0 (socket says 172.21.15.67:60280) 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 -- 172.21.15.67:0/1887120974 learned_addr learned my addr 172.21.15.67:0/1887120974 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f929b400640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929c06ef10 0x7f929c143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f929be00640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c145040 0x7f929c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 -- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c145040 msgr2=0x7f929c15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c145040 0x7f929c15b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 -- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929c06ef10 msgr2=0x7f929c143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929c06ef10 0x7f929c143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 -- 172.21.15.67:0/1887120974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c141fd0 con 0x7f929c071850 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293e00640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15acd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f928400e8b0 tx=0x7f928400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293400640 1 -- 172.21.15.67:0/1887120974 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9284017070 con 0x7f929c071850 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293400640 1 -- 172.21.15.67:0/1887120974 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928400c940 con 0x7f929c071850 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c1422b0 con 0x7f929c071850 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293400640 1 -- 172.21.15.67:0/1887120974 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928400cd10 con 0x7f929c071850 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c142870 con 0x7f929c071850 2024-04-17T10:28:28.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c072d50 con 0x7f929c071850 2024-04-17T10:28:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.446+0000 7f9293400640 1 -- 172.21.15.67:0/1887120974 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f928401a070 con 0x7f929c071850 2024-04-17T10:28:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.450+0000 7f9293400640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066f60 0x7f9278069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.450+0000 7f9293400640 1 -- 172.21.15.67:0/1887120974 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92840854d0 con 0x7f929c071850 2024-04-17T10:28:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.450+0000 7f929b400640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066f60 0x7f9278069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.450+0000 7f929b400640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066f60 0x7f9278069420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f928c009ac0 tx=0x7f928c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.454+0000 7f9293400640 1 -- 172.21.15.67:0/1887120974 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9284055cf0 con 0x7f929c071850 2024-04-17T10:28:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f929c074e30 con 0x7f929c071850 2024-04-17T10:28:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f9293400640 1 -- 172.21.15.67:0/1887120974 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7f9284059d90 con 0x7f929c071850 2024-04-17T10:28:28.644 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:28.644 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066f60 msgr2=0x7f9278069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066f60 0x7f9278069420 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f928c009ac0 tx=0x7f928c009190 comp rx=0 tx=0).stop 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 msgr2=0x7f929c15acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15acd0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f928400e8b0 tx=0x7f928400ed70 comp rx=0 tx=0).stop 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 shutdown_connections 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066f60 0x7f9278069420 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f929c145040 0x7f929c15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f929c071850 0x7f929c15acd0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 --2- 172.21.15.67:0/1887120974 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929c06ef10 0x7f929c143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 >> 172.21.15.67:0/1887120974 conn(0x7f929c06ac90 msgr2=0x7f929c06b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.650+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 shutdown_connections 2024-04-17T10:28:28.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.654+0000 7f92a2800640 1 -- 172.21.15.67:0/1887120974 wait complete. 2024-04-17T10:28:28.667 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d damage rm 4224541580 2024-04-17T10:28:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6309800640 1 Processor -- start 2024-04-17T10:28:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6309800640 1 -- start start 2024-04-17T10:28:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6309800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f630415f200 0x7f630406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6309800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630415f690 con 0x7f6304162160 2024-04-17T10:28:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6309800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630406d8d0 con 0x7f63040713e0 2024-04-17T10:28:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6309800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630406d6a0 con 0x7f630415f200 2024-04-17T10:28:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6302000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6304162160 0x7f63040ae4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59666/0 (socket says 172.21.15.67:59666) 2024-04-17T10:28:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6302000640 1 -- 172.21.15.67:0/4062116845 learned_addr learned my addr 172.21.15.67:0/4062116845 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:28.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6303400640 1 --2- 172.21.15.67:0/4062116845 >> v2:172.21.15.67:3300/0 conn(0x7f630415f200 0x7f630406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6303400640 1 -- 172.21.15.67:0/4062116845 >> v1:172.21.15.86:6789/0 conn(0x7f63040713e0 legacy=0x7f63040717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6303400640 1 -- 172.21.15.67:0/4062116845 >> v1:172.21.15.67:6789/0 conn(0x7f6304162160 legacy=0x7f63040ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6301600640 1 -- 172.21.15.67:0/4062116845 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963411900 0 0) 0x7f630415f690 con 0x7f6304162160 2024-04-17T10:28:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.818+0000 7f6303400640 1 -- 172.21.15.67:0/4062116845 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630406dbb0 con 0x7f630415f200 2024-04-17T10:28:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6303400640 1 --2- 172.21.15.67:0/4062116845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f200 0x7f630406d160 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009d90 tx=0x7f62fc01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db1bf5a0a4cffc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6301600640 1 -- 172.21.15.67:0/4062116845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0b2070 con 0x7f630415f200 2024-04-17T10:28:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6301600640 1 -- 172.21.15.67:0/4062116845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc0a5920 con 0x7f630415f200 2024-04-17T10:28:28.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6301600640 1 -- 172.21.15.67:0/4062116845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0a5cf0 con 0x7f630415f200 2024-04-17T10:28:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 -- 172.21.15.67:0/4062116845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f200 msgr2=0x7f630406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 --2- 172.21.15.67:0/4062116845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f200 0x7f630406d160 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009d90 tx=0x7f62fc01c970 comp rx=0 tx=0).stop 2024-04-17T10:28:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 -- 172.21.15.67:0/4062116845 shutdown_connections 2024-04-17T10:28:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 --2- 172.21.15.67:0/4062116845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f630415f200 0x7f630406d160 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 -- 172.21.15.67:0/4062116845 >> 172.21.15.67:0/4062116845 conn(0x7f63040b1740 msgr2=0x7f63040b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 -- 172.21.15.67:0/4062116845 shutdown_connections 2024-04-17T10:28:28.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 -- 172.21.15.67:0/4062116845 wait complete. 2024-04-17T10:28:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 Processor -- start 2024-04-17T10:28:28.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.822+0000 7f6309800640 1 -- start start 2024-04-17T10:28:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63040713e0 0x7f6304145650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415f200 0x7f63040b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6304162160 0x7f6304139410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63041477c0 con 0x7f6304162160 2024-04-17T10:28:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304147640 con 0x7f63040713e0 2024-04-17T10:28:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040aefe0 con 0x7f630415f200 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6304162160 0x7f6304139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6302a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63040713e0 0x7f6304145650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6304162160 0x7f6304139410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60286/0 (socket says 172.21.15.67:60286) 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 -- 172.21.15.67:0/939901115 learned_addr learned my addr 172.21.15.67:0/939901115 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6302000640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415f200 0x7f63040b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 -- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415f200 msgr2=0x7f63040b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415f200 0x7f63040b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 -- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63040713e0 msgr2=0x7f6304145650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63040713e0 0x7f6304145650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 -- 172.21.15.67:0/939901115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6304145b90 con 0x7f6304162160 2024-04-17T10:28:28.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6303400640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6304162160 0x7f6304139410 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0b0040 tx=0x7f62fc019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f62faa00640 1 -- 172.21.15.67:0/939901115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0b2050 con 0x7f6304162160 2024-04-17T10:28:28.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f62faa00640 1 -- 172.21.15.67:0/939901115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc0be070 con 0x7f6304162160 2024-04-17T10:28:28.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f62faa00640 1 -- 172.21.15.67:0/939901115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0adad0 con 0x7f6304162160 2024-04-17T10:28:28.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6304146030 con 0x7f6304162160 2024-04-17T10:28:28.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.826+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63041465f0 con 0x7f6304162160 2024-04-17T10:28:28.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f62faa00640 1 -- 172.21.15.67:0/939901115 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f62fc0adc70 con 0x7f6304162160 2024-04-17T10:28:28.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f62faa00640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e4066f60 0x7f62e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f62faa00640 1 -- 172.21.15.67:0/939901115 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f62fc120470 con 0x7f6304162160 2024-04-17T10:28:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6302a00640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e4066f60 0x7f62e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 Processor -- start 2024-04-17T10:28:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 -- start start 2024-04-17T10:28:28.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6302a00640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e4066f60 0x7f62e4069420 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f62f4007fd0 tx=0x7f62f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13cb2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13c656a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c65fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e13ab5f00 con 0x563e13c65be0 2024-04-17T10:28:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e13a3e4a0 con 0x563e13cb2070 2024-04-17T10:28:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e13ab1fa0 con 0x563e13cb2990 2024-04-17T10:28:28.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6303400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c65fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6303400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c65fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60288/0 (socket says 172.21.15.67:60288) 2024-04-17T10:28:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6303400640 1 -- 172.21.15.67:0/953552384 learned_addr learned my addr 172.21.15.67:0/953552384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6302a00640 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13cb2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6302000640 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13c656a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6303400640 1 -- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 msgr2=0x563e13c656a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6303400640 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13c656a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6303400640 1 -- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 msgr2=0x563e13cb2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.830+0000 7f6303400640 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13cb2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6303400640 1 -- 172.21.15.67:0/953552384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e13a3bd00 con 0x563e13c65be0 2024-04-17T10:28:28.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13cb2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:28.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6303400640 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c65fc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f62fc01cee0 tx=0x7f62fc0f0570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3133a0c924340ade server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f62ef400640 1 -- 172.21.15.67:0/953552384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc0f9270 con 0x563e13c65be0 2024-04-17T10:28:28.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f62ef400640 1 -- 172.21.15.67:0/953552384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc0b3040 con 0x563e13c65be0 2024-04-17T10:28:28.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f62ef400640 1 -- 172.21.15.67:0/953552384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62fc1200e0 con 0x563e13c65be0 2024-04-17T10:28:28.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- 172.21.15.67:0/953552384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 msgr2=0x563e13c65fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c65fc0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f62fc01cee0 tx=0x7f62fc0f0570 comp rx=0 tx=0).stop 2024-04-17T10:28:28.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- 172.21.15.67:0/953552384 shutdown_connections 2024-04-17T10:28:28.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c65fc0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13c656a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 --2- 172.21.15.67:0/953552384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13cb2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- 172.21.15.67:0/953552384 >> 172.21.15.67:0/953552384 conn(0x563e13c92e10 msgr2=0x563e13c95200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- 172.21.15.67:0/953552384 shutdown_connections 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- 172.21.15.67:0/953552384 wait complete. 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- start start 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 Processor -- start 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c6b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2070 0x563e13c6bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13c6c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e13aba530 con 0x563e13c65be0 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e13a70ae0 con 0x563e13cb2990 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e13a74e10 con 0x563e13cb2070 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f630cc74000 10 client.0 _ll_register_callbacks cb 0x563e13c53750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c6b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c6b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60292/0 (socket says 172.21.15.67:60292) 2024-04-17T10:28:28.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 -- 172.21.15.67:0/1917115848 learned_addr learned my addr 172.21.15.67:0/1917115848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302000640 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2070 0x563e13c6bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6303400640 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13c6c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 -- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2070 msgr2=0x563e13c6bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2070 0x563e13c6bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 -- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 msgr2=0x563e13c6c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13c6c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.834+0000 7f6302a00640 1 -- 172.21.15.67:0/1917115848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e13a58a10 con 0x563e13c65be0 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f6302a00640 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c6b6a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f62f4014780 tx=0x7f62f4001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b070d30b3c9e645a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 1 -- 172.21.15.67:0/1917115848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4014e50 con 0x563e13c65be0 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 1 -- 172.21.15.67:0/1917115848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f62f4013930 con 0x563e13c65be0 2024-04-17T10:28:28.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62fc0b3040 con 0x563e13c65be0 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 1 -- 172.21.15.67:0/1917115848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f402e050 con 0x563e13c65be0 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 10 client.6210 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x563e13ab0ef0 con 0x563e13c65be0 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 1 -- 172.21.15.67:0/1917115848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62fc0b3040 con 0x563e13c65be0 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 1 -- 172.21.15.67:0/1917115848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f62f4031020 con 0x563e13c65be0 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 10 client.6210 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 10 client.6210 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 20 client.6210 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13cc1100 0x563e13cc34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 4 client.6210 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563e13a51b20 con 0x563e13cc1100 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f6302000640 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13cc1100 0x563e13cc34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f6302000640 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13cc1100 0x563e13cc34f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f56ae1ec2284e017 server_cookie=9469836a64c1e010 in_seq=0 out_seq=0 2024-04-17T10:28:28.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 10 client.6210 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 1 -- 172.21.15.67:0/1917115848 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563e13a51b20 con 0x563e13cc1100 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 10 client.6210 handle_command_reply: tid=0 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 client.6210 shutdown 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 msgr2=0x563e13c6b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c6b6a0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f62f4014780 tx=0x7f62f4001090 comp rx=0 tx=0).stop 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 shutdown_connections 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13cc1100 0x563e13cc34f0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13c6c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2070 0x563e13c6bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 --2- 172.21.15.67:0/1917115848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13c6b6a0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f62e3e00640 0 client.6210 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 >> 172.21.15.67:0/1917115848 conn(0x563e13c92e10 msgr2=0x563e13c64da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 shutdown_connections 2024-04-17T10:28:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 1 -- 172.21.15.67:0/1917115848 wait complete. 2024-04-17T10:28:28.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.838+0000 7f630cc74000 20 client.6210 trim_cache size 0 max 16384 2024-04-17T10:28:28.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.850+0000 7f630cc74000 1 Processor -- start 2024-04-17T10:28:28.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.850+0000 7f630cc74000 1 -- start start 2024-04-17T10:28:28.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13c65be0 0x563e13d0e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13cb2070 0x563e13d0e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13d0ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6304202d40 con 0x563e13cb2070 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63041fb6c0 con 0x563e13cb2990 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040ce110 con 0x563e13c65be0 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13cb2070 0x563e13d0e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13cb2070 0x563e13d0e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60296/0 (socket says 172.21.15.67:60296) 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 -- 172.21.15.67:0/1458494694 learned_addr learned my addr 172.21.15.67:0/1458494694 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302a00640 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13c65be0 0x563e13d0e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6303400640 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13d0ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 -- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13c65be0 msgr2=0x563e13d0e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13c65be0 0x563e13d0e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 -- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 msgr2=0x563e13d0ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13d0ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 -- 172.21.15.67:0/1458494694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563e13a75790 con 0x563e13cb2070 2024-04-17T10:28:28.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6303400640 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13d0ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:28.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f6302000640 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13cb2070 0x563e13d0e8c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f62f0007fd0 tx=0x7f62f0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb4db18e4654c08b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f62eea00640 1 -- 172.21.15.67:0/1458494694 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0016950 con 0x563e13cb2070 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f62eea00640 1 -- 172.21.15.67:0/1458494694 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f0016af0 con 0x563e13cb2070 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f62eea00640 1 -- 172.21.15.67:0/1458494694 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0024410 con 0x563e13cb2070 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- 172.21.15.67:0/1458494694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13cb2070 msgr2=0x563e13d0e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13cb2070 0x563e13d0e8c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f62f0007fd0 tx=0x7f62f0009040 comp rx=0 tx=0).stop 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- 172.21.15.67:0/1458494694 shutdown_connections 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2990 0x563e13d0ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13cb2070 0x563e13d0e8c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- 172.21.15.67:0/1458494694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13c65be0 0x563e13d0e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- 172.21.15.67:0/1458494694 >> 172.21.15.67:0/1458494694 conn(0x563e13c92e10 msgr2=0x563e13c6cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- 172.21.15.67:0/1458494694 shutdown_connections 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- 172.21.15.67:0/1458494694 wait complete. 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- start start 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 Processor -- start 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13cc4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13d11570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13d11ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e13ab5210 con 0x563e13c65be0 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x563e13a82b90 con 0x563e13cb2070 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x563e13a73700 con 0x563e13cb2990 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.854+0000 7f630cc74000 10 client.0 _ll_register_callbacks cb 0x7f62fc009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:28.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13cc4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13cc4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60314/0 (socket says 172.21.15.67:60314) 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 -- 172.21.15.67:0/4059558300 learned_addr learned my addr 172.21.15.67:0/4059558300 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6303400640 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13d11ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302000640 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13d11570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 -- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 msgr2=0x563e13d11ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13d11ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 -- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 msgr2=0x563e13d11570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13d11570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 -- 172.21.15.67:0/4059558300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563e13a7dda0 con 0x563e13c65be0 2024-04-17T10:28:28.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302a00640 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13cc4bb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f62f4013990 tx=0x7f62f4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5f030edc09582e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:28.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:28.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 1 -- 172.21.15.67:0/4059558300 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4021840 con 0x563e13c65be0 2024-04-17T10:28:28.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 1 -- 172.21.15.67:0/4059558300 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f62f403dc70 con 0x563e13c65be0 2024-04-17T10:28:28.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62f0016af0 con 0x563e13c65be0 2024-04-17T10:28:28.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 1 -- 172.21.15.67:0/4059558300 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62f4014e50 con 0x563e13c65be0 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 10 client.6216 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x563e13772360 con 0x563e13c65be0 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 1 -- 172.21.15.67:0/4059558300 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f0016af0 con 0x563e13c65be0 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 1 -- 172.21.15.67:0/4059558300 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f62f4044020 con 0x563e13c65be0 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 10 client.6216 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 10 client.6216 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 20 client.6216 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13d19260 0x563e13d1b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 4 client.6216 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 4224541580}] 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "damage rm", "damage_id": 4224541580}) v1 -- 0x563e13ac3440 con 0x563e13d19260 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302000640 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13d19260 0x563e13d1b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6302000640 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13d19260 0x563e13d1b650 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fce4eee1406e473a server_cookie=d5846a0dfa8f3d8d in_seq=0 out_seq=0 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 10 client.6216 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 1 -- 172.21.15.67:0/4059558300 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x563e13ac3440 con 0x563e13d19260 2024-04-17T10:28:28.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 10 client.6216 handle_command_reply: tid=0 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 client.6216 shutdown 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 msgr2=0x563e13cc4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13cc4bb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f62f4013990 tx=0x7f62f4015000 comp rx=0 tx=0).stop 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 shutdown_connections 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x563e13d19260 0x563e13d1b650 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563e13cb2990 0x563e13d11ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x563e13cb2070 0x563e13d11570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f62e3e00640 0 client.6216 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 --2- 172.21.15.67:0/4059558300 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563e13c65be0 0x563e13cc4bb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 >> 172.21.15.67:0/4059558300 conn(0x563e13c92e10 msgr2=0x563e13c6d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 shutdown_connections 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 1 -- 172.21.15.67:0/4059558300 wait complete. 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f630cc74000 20 client.6216 trim_cache size 0 max 16384 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e4066f60 msgr2=0x7f62e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e4066f60 0x7f62e4069420 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f62f4007fd0 tx=0x7f62f4009040 comp rx=0 tx=0).stop 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6304162160 msgr2=0x7f6304139410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6304162160 0x7f6304139410 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f62fc0b0040 tx=0x7f62fc019710 comp rx=0 tx=0).stop 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 shutdown_connections 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f62e4066f60 0x7f62e4069420 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6304162160 0x7f6304139410 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f630415f200 0x7f63040b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.858+0000 7f6309800640 1 --2- 172.21.15.67:0/939901115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63040713e0 0x7f6304145650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.862+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 >> 172.21.15.67:0/939901115 conn(0x7f63040b1740 msgr2=0x7f630413bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.862+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 shutdown_connections 2024-04-17T10:28:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:28.862+0000 7f6309800640 1 -- 172.21.15.67:0/939901115 wait complete. 2024-04-17T10:28:28.873 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /' 2024-04-17T10:28:28.873 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4f4a00640 1 Processor -- start 2024-04-17T10:28:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4f4a00640 1 -- start start 2024-04-17T10:28:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4f4a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd4f0144aa0 0x7fd4f00d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4f4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4f0071840 0x7fd4f00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4f4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f01456d0 con 0x7fd4f01446c0 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4f4a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f00b51a0 con 0x7fd4f0071840 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4f4a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f00d1390 con 0x7fd4f0144aa0 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ee800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4f0071840 0x7fd4f00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ede00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd4f01446c0 0x7fd4f0144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59674/0 (socket says 172.21.15.67:59674) 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ede00640 1 -- 172.21.15.67:0/1696967620 learned_addr learned my addr 172.21.15.67:0/1696967620 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4e5400640 1 --2- 172.21.15.67:0/1696967620 >> v2:172.21.15.86:3300/0 conn(0x7fd4f0144aa0 0x7fd4f00d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ee800640 1 -- 172.21.15.67:0/1696967620 >> v2:172.21.15.86:3300/0 conn(0x7fd4f0144aa0 msgr2=0x7fd4f00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ee800640 1 --2- 172.21.15.67:0/1696967620 >> v2:172.21.15.86:3300/0 conn(0x7fd4f0144aa0 0x7fd4f00d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ee800640 1 -- 172.21.15.67:0/1696967620 >> v1:172.21.15.67:6789/0 conn(0x7fd4f01446c0 legacy=0x7fd4f0144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ee800640 1 -- 172.21.15.67:0/1696967620 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f00b6640 con 0x7fd4f0071840 2024-04-17T10:28:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.022+0000 7fd4ed400640 1 -- 172.21.15.67:0/1696967620 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3833334682 0 0) 0x7fd4f01456d0 con 0x7fd4f01446c0 2024-04-17T10:28:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4ee800640 1 --2- 172.21.15.67:0/1696967620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f0071840 0x7fd4f00d0e50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd4f015f620 tx=0x7fd4e801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee02cd883bffc796 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4ed400640 1 -- 172.21.15.67:0/1696967620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e80b2070 con 0x7fd4f0071840 2024-04-17T10:28:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4ed400640 1 -- 172.21.15.67:0/1696967620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e80a59c0 con 0x7fd4f0071840 2024-04-17T10:28:29.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4ed400640 1 -- 172.21.15.67:0/1696967620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e80a5d90 con 0x7fd4f0071840 2024-04-17T10:28:29.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1696967620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f0071840 msgr2=0x7fd4f00d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1696967620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f0071840 0x7fd4f00d0e50 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd4f015f620 tx=0x7fd4e801c900 comp rx=0 tx=0).stop 2024-04-17T10:28:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1696967620 shutdown_connections 2024-04-17T10:28:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1696967620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f0071840 0x7fd4f00d0e50 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1696967620 >> v2:172.21.15.86:3300/0 conn(0x7fd4f0144aa0 0x7fd4f00d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1696967620 >> 172.21.15.67:0/1696967620 conn(0x7fd4f00b34d0 msgr2=0x7fd4f00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1696967620 shutdown_connections 2024-04-17T10:28:29.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.026+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1696967620 wait complete. 2024-04-17T10:28:29.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 Processor -- start 2024-04-17T10:28:29.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 -- start start 2024-04-17T10:28:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4f0071840 0x7fd4f0143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f01446c0 0x7fd4f013c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4f0144aa0 0x7fd4f0154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00b5e40 con 0x7fd4f01446c0 2024-04-17T10:28:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f015c0d0 con 0x7fd4f0144aa0 2024-04-17T10:28:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00b5fc0 con 0x7fd4f0071840 2024-04-17T10:28:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f01446c0 0x7fd4f013c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f01446c0 0x7fd4f013c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60334/0 (socket says 172.21.15.67:60334) 2024-04-17T10:28:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 -- 172.21.15.67:0/1521675753 learned_addr learned my addr 172.21.15.67:0/1521675753 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4ee800640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4f0144aa0 0x7fd4f0154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4ede00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4f0071840 0x7fd4f0143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 -- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4f0071840 msgr2=0x7fd4f0143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4f0071840 0x7fd4f0143170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 -- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4f0144aa0 msgr2=0x7fd4f0154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4f0144aa0 0x7fd4f0154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 -- 172.21.15.67:0/1521675753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f015ae60 con 0x7fd4f01446c0 2024-04-17T10:28:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4ede00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4f0071840 0x7fd4f0143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4ee800640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4f0144aa0 0x7fd4f0154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:29.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e5400640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f01446c0 0x7fd4f013c9e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800d620 tx=0x7fd4d800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:29.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e6a00640 1 -- 172.21.15.67:0/1521675753 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800b620 con 0x7fd4f01446c0 2024-04-17T10:28:29.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e6a00640 1 -- 172.21.15.67:0/1521675753 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d8019070 con 0x7fd4f01446c0 2024-04-17T10:28:29.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4e6a00640 1 -- 172.21.15.67:0/1521675753 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d800b9e0 con 0x7fd4f01446c0 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f015b140 con 0x7fd4f01446c0 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f0142730 con 0x7fd4f01446c0 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.030+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f0070bb0 con 0x7fd4f01446c0 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.034+0000 7fd4e6a00640 1 -- 172.21.15.67:0/1521675753 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd4d800bb80 con 0x7fd4f01446c0 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.034+0000 7fd4e6a00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4cc066f90 0x7fd4cc069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.034+0000 7fd4e6a00640 1 -- 172.21.15.67:0/1521675753 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd4d8085540 con 0x7fd4f01446c0 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.034+0000 7fd4ede00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4cc066f90 0x7fd4cc069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.034+0000 7fd4ede00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4cc066f90 0x7fd4cc069450 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0007fd0 tx=0x7fd4e0007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:29.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.038+0000 7fd4e6a00640 1 -- 172.21.15.67:0/1521675753 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd4d80563b0 con 0x7fd4f01446c0 2024-04-17T10:28:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.234+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4f0074e30 con 0x7fd4f01446c0 2024-04-17T10:28:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.234+0000 7fd4e6a00640 1 -- 172.21.15.67:0/1521675753 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7fd4d8055cb0 con 0x7fd4f01446c0 2024-04-17T10:28:29.228 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:29.229 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.234+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4cc066f90 msgr2=0x7fd4cc069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4cc066f90 0x7fd4cc069450 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0007fd0 tx=0x7fd4e0007c20 comp rx=0 tx=0).stop 2024-04-17T10:28:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f01446c0 msgr2=0x7fd4f013c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f01446c0 0x7fd4f013c9e0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800d620 tx=0x7fd4d800dae0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 shutdown_connections 2024-04-17T10:28:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4cc066f90 0x7fd4cc069450 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4f0144aa0 0x7fd4f0154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4f01446c0 0x7fd4f013c9e0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 --2- 172.21.15.67:0/1521675753 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4f0071840 0x7fd4f0143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 >> 172.21.15.67:0/1521675753 conn(0x7fd4f00b34d0 msgr2=0x7fd4f006c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 shutdown_connections 2024-04-17T10:28:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.238+0000 7fd4f4a00640 1 -- 172.21.15.67:0/1521675753 wait complete. 2024-04-17T10:28:29.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path / 2024-04-17T10:28:29.400 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:29.400 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:29.400 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:29.415 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:29.415 INFO:tasks.cephfs.test_scrub_checks:mountpoint: /home/ubuntu/cephtest/mnt.0 2024-04-17T10:28:29.415 INFO:tasks.cephfs.test_scrub_checks:client_path: /home/ubuntu/cephtest/mnt.0/scrub_test_path 2024-04-17T10:28:29.416 INFO:tasks.cephfs.test_scrub_checks:Cloning repo into place 2024-04-17T10:28:29.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:29.416 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p /home/ubuntu/cephtest/mnt.0/scrub_test_path) 2024-04-17T10:28:29.426 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-04-17T10:28:29.426 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:28:29.426 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-04-17T10:28:29.426 DEBUG:teuthology.orchestra.run.smithi067:> import json 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> import sys 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> try: 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite'"'"') 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> except OSError as e: 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> sys.exit(e.errno) 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> print(json.dumps( 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> dict([(a, getattr(s, a)) for a in attrs]), 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> indent=2)) 2024-04-17T10:28:29.427 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:28:29.544 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "st_mode": 16893, 2024-04-17T10:28:29.544 INFO:teuthology.orchestra.run.smithi067.stdout: "st_ino": 1099511627777, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_dev": 44, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_nlink": 16, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_uid": 1000, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_gid": 1271, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_size": 18, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_atime": 1713349694.1303868, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_mtime": 1713349701.374326, 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout: "st_ctime": 1713349701.374326 2024-04-17T10:28:29.545 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:29.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T10:28:29.553 INFO:tasks.cephfs.test_scrub_checks:Initiating mds_scrub_checks on mds.0 test_path /scrub_test_path, run_seq 1 2024-04-17T10:28:29.553 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/i/dont/exist' 2024-04-17T10:28:29.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:29.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.742+0000 7fdefaa00640 1 Processor -- start 2024-04-17T10:28:29.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.742+0000 7fdefaa00640 1 -- start start 2024-04-17T10:28:29.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdefaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdef40ce020 0x7fdef41620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdefaa00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdef40cf510 0x7fdef40cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdefaa00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdef415eb50 con 0x7fdef40ce400 2024-04-17T10:28:29.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdefaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef4161650 con 0x7fdef40ce020 2024-04-17T10:28:29.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdefaa00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef41625e0 con 0x7fdef40cf510 2024-04-17T10:28:29.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdef40ce020 0x7fdef41620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdef40ce020 0x7fdef41620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60336/0 (socket says 172.21.15.67:60336) 2024-04-17T10:28:29.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef3e00640 1 -- 172.21.15.67:0/4244187431 learned_addr learned my addr 172.21.15.67:0/4244187431 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:29.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef8a00640 1 --2- 172.21.15.67:0/4244187431 >> v2:172.21.15.86:3300/0 conn(0x7fdef40cf510 0x7fdef40cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef3e00640 1 -- 172.21.15.67:0/4244187431 >> v2:172.21.15.86:3300/0 conn(0x7fdef40cf510 msgr2=0x7fdef40cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:29.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef3e00640 1 --2- 172.21.15.67:0/4244187431 >> v2:172.21.15.86:3300/0 conn(0x7fdef40cf510 0x7fdef40cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef3e00640 1 -- 172.21.15.67:0/4244187431 >> v1:172.21.15.86:6789/0 conn(0x7fdef40ce400 legacy=0x7fdef415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:29.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef3e00640 1 -- 172.21.15.67:0/4244187431 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef415ee30 con 0x7fdef40ce020 2024-04-17T10:28:29.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.746+0000 7fdef8a00640 1 --2- 172.21.15.67:0/4244187431 >> v2:172.21.15.86:3300/0 conn(0x7fdef40cf510 0x7fdef40cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:29.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdef3e00640 1 --2- 172.21.15.67:0/4244187431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40ce020 0x7fdef41620a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009870 tx=0x7fdee401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d570a6cf57efd1e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:29.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdef3400640 1 -- 172.21.15.67:0/4244187431 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee40b1070 con 0x7fdef40ce020 2024-04-17T10:28:29.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdef3400640 1 -- 172.21.15.67:0/4244187431 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee40ac620 con 0x7fdef40ce020 2024-04-17T10:28:29.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdef3400640 1 -- 172.21.15.67:0/4244187431 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee40ac9f0 con 0x7fdef40ce020 2024-04-17T10:28:29.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- 172.21.15.67:0/4244187431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40ce020 msgr2=0x7fdef41620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 --2- 172.21.15.67:0/4244187431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40ce020 0x7fdef41620a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009870 tx=0x7fdee401c4b0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- 172.21.15.67:0/4244187431 shutdown_connections 2024-04-17T10:28:29.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 --2- 172.21.15.67:0/4244187431 >> v2:172.21.15.86:3300/0 conn(0x7fdef40cf510 0x7fdef40cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 --2- 172.21.15.67:0/4244187431 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40ce020 0x7fdef41620a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- 172.21.15.67:0/4244187431 >> 172.21.15.67:0/4244187431 conn(0x7fdef40b33f0 msgr2=0x7fdef40b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:29.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- 172.21.15.67:0/4244187431 shutdown_connections 2024-04-17T10:28:29.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- 172.21.15.67:0/4244187431 wait complete. 2024-04-17T10:28:29.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 Processor -- start 2024-04-17T10:28:29.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- start start 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef40ce020 0x7fdef4153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef40ce400 0x7fdef415d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40cf510 0x7fdef4159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef406c0f0 con 0x7fdef40cf510 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef406bf70 con 0x7fdef40ce020 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.750+0000 7fdefaa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef415e720 con 0x7fdef40ce400 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef40ce020 0x7fdef4153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40cf510 0x7fdef4159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef40ce020 0x7fdef4153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:59834/0 (socket says 172.21.15.67:59834) 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef3e00640 1 -- 172.21.15.67:0/2077965532 learned_addr learned my addr 172.21.15.67:0/2077965532 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdeebe00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef40ce400 0x7fdef415d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef8a00640 1 -- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef40ce400 msgr2=0x7fdef415d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef8a00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef40ce400 0x7fdef415d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef8a00640 1 -- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef40ce020 msgr2=0x7fdef4153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef8a00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef40ce020 0x7fdef4153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef8a00640 1 -- 172.21.15.67:0/2077965532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef40cb680 con 0x7fdef40cf510 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef8a00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40cf510 0x7fdef4159940 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00cc90 tx=0x7fdeec00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef0c00640 1 -- 172.21.15.67:0/2077965532 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00f680 con 0x7fdef40cf510 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef0c00640 1 -- 172.21.15.67:0/2077965532 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec00f840 con 0x7fdef40cf510 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef0c00640 1 -- 172.21.15.67:0/2077965532 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00fc30 con 0x7fdef40cf510 2024-04-17T10:28:29.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef40cb3a0 con 0x7fdef40cf510 2024-04-17T10:28:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef415c9a0 con 0x7fdef40cf510 2024-04-17T10:28:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef4070ab0 con 0x7fdef40cf510 2024-04-17T10:28:29.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef0c00640 1 -- 172.21.15.67:0/2077965532 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdeec0213f0 con 0x7fdef40cf510 2024-04-17T10:28:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.754+0000 7fdef0c00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fded0066f60 0x7fded0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.758+0000 7fdef0c00640 1 -- 172.21.15.67:0/2077965532 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdeec085e60 con 0x7fdef40cf510 2024-04-17T10:28:29.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.758+0000 7fdef3e00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fded0066f60 0x7fded0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:29.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.758+0000 7fdef3e00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fded0066f60 0x7fded0069420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009870 tx=0x7fdee40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:29.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.762+0000 7fdef0c00640 1 -- 172.21.15.67:0/2077965532 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdeec014030 con 0x7fdef40cf510 2024-04-17T10:28:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdef4074e30 con 0x7fdef40cf510 2024-04-17T10:28:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdef0c00640 1 -- 172.21.15.67:0/2077965532 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+4935 (secure 0 0 0) 0x7fdeec056c40 con 0x7fdef40cf510 2024-04-17T10:28:29.952 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:29.952 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":28,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:04.796960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:29.952 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 28 2024-04-17T10:28:29.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fded0066f60 msgr2=0x7fded0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fded0066f60 0x7fded0069420 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdee4009870 tx=0x7fdee40a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:29.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40cf510 msgr2=0x7fdef4159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40cf510 0x7fdef4159940 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00cc90 tx=0x7fdeec00a650 comp rx=0 tx=0).stop 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 shutdown_connections 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fded0066f60 0x7fded0069420 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdef40cf510 0x7fdef4159940 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdef40ce400 0x7fdef415d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 --2- 172.21.15.67:0/2077965532 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdef40ce020 0x7fdef4153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 >> 172.21.15.67:0/2077965532 conn(0x7fdef40b33f0 msgr2=0x7fdef40cd120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 shutdown_connections 2024-04-17T10:28:29.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:29.958+0000 7fdefaa00640 1 -- 172.21.15.67:0/2077965532 wait complete. 2024-04-17T10:28:29.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/i/dont/exist 2024-04-17T10:28:30.120 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:30.120 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": -2 2024-04-17T10:28:30.120 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:30.136 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/i/dont/exist' got response code '0' and stdout '{ "return_code": -2 } ' 2024-04-17T10:28:30.136 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/i/dont/exist' 2024-04-17T10:28:30.136 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:30.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/i/dont/exist 2024-04-17T10:28:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.286+0000 7fafdca00640 1 Processor -- start 2024-04-17T10:28:30.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.286+0000 7fafdca00640 1 -- start start 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafdca00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fafd815da90 0x7fafd8162130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafdca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fafd80cf880 0x7fafd80d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafdca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80d1380 con 0x7fafd8099a50 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafdca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd8162670 con 0x7fafd80cf880 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafdca00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafd80cfda0 con 0x7fafd815da90 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafd7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fafd8099a50 0x7fafd80cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59684/0 (socket says 172.21.15.67:59684) 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafcea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fafd80cf880 0x7fafd80d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafd7e00640 1 -- 172.21.15.67:0/1711967339 learned_addr learned my addr 172.21.15.67:0/1711967339 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafd7400640 1 --2- 172.21.15.67:0/1711967339 >> v2:172.21.15.86:3300/0 conn(0x7fafd815da90 0x7fafd8162130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafcea00640 1 -- 172.21.15.67:0/1711967339 >> v2:172.21.15.86:3300/0 conn(0x7fafd815da90 msgr2=0x7fafd8162130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafcea00640 1 --2- 172.21.15.67:0/1711967339 >> v2:172.21.15.86:3300/0 conn(0x7fafd815da90 0x7fafd8162130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafcea00640 1 -- 172.21.15.67:0/1711967339 >> v1:172.21.15.67:6789/0 conn(0x7fafd8099a50 legacy=0x7fafd80cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafcea00640 1 -- 172.21.15.67:0/1711967339 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80d0080 con 0x7fafd80cf880 2024-04-17T10:28:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.290+0000 7fafd6a00640 1 -- 172.21.15.67:0/1711967339 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615686312 0 0) 0x7fafd80d1380 con 0x7fafd8099a50 2024-04-17T10:28:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafcea00640 1 --2- 172.21.15.67:0/1711967339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80d0e40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fafc00097c0 tx=0x7fafc001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4dd70d198c1e01 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafd6a00640 1 -- 172.21.15.67:0/1711967339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00b0070 con 0x7fafd80cf880 2024-04-17T10:28:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafd6a00640 1 -- 172.21.15.67:0/1711967339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0004230 con 0x7fafd80cf880 2024-04-17T10:28:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafd6a00640 1 -- 172.21.15.67:0/1711967339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0003620 con 0x7fafd80cf880 2024-04-17T10:28:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 -- 172.21.15.67:0/1711967339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 msgr2=0x7fafd80d0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 --2- 172.21.15.67:0/1711967339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80d0e40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fafc00097c0 tx=0x7fafc001c460 comp rx=0 tx=0).stop 2024-04-17T10:28:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 -- 172.21.15.67:0/1711967339 shutdown_connections 2024-04-17T10:28:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 --2- 172.21.15.67:0/1711967339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80d0e40 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 --2- 172.21.15.67:0/1711967339 >> v2:172.21.15.86:3300/0 conn(0x7fafd815da90 0x7fafd8162130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 -- 172.21.15.67:0/1711967339 >> 172.21.15.67:0/1711967339 conn(0x7fafd80d15b0 msgr2=0x7fafd80d19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 -- 172.21.15.67:0/1711967339 shutdown_connections 2024-04-17T10:28:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 -- 172.21.15.67:0/1711967339 wait complete. 2024-04-17T10:28:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 Processor -- start 2024-04-17T10:28:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.294+0000 7fafdca00640 1 -- start start 2024-04-17T10:28:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd8099a50 0x7fafd8151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd815da90 0x7fafd81f7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd809a960 con 0x7fafd80cf880 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80cc720 con 0x7fafd8099a50 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd809a290 con 0x7fafd815da90 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35780/0 (socket says 172.21.15.67:35780) 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafd7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd8099a50 0x7fafd8151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 -- 172.21.15.67:0/601620266 learned_addr learned my addr 172.21.15.67:0/601620266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafd7e00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd815da90 0x7fafd81f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 -- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd815da90 msgr2=0x7fafd81f7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd815da90 0x7fafd81f7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 -- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd8099a50 msgr2=0x7fafd8151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd8099a50 0x7fafd8151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 -- 172.21.15.67:0/601620266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80b47d0 con 0x7fafd80cf880 2024-04-17T10:28:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafd7400640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd8099a50 0x7fafd8151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcea00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80b4290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fafc00ae040 tx=0x7fafc0019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcfe00640 1 -- 172.21.15.67:0/601620266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00b0040 con 0x7fafd80cf880 2024-04-17T10:28:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcfe00640 1 -- 172.21.15.67:0/601620266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc00bb070 con 0x7fafd80cf880 2024-04-17T10:28:30.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcfe00640 1 -- 172.21.15.67:0/601620266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00043e0 con 0x7fafd80cf880 2024-04-17T10:28:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd80b4a70 con 0x7fafd80cf880 2024-04-17T10:28:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd80b5030 con 0x7fafd80cf880 2024-04-17T10:28:30.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.298+0000 7fafcfe00640 1 -- 172.21.15.67:0/601620266 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fafc00b3070 con 0x7fafd80cf880 2024-04-17T10:28:30.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafcfe00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fafb4066f90 0x7fafb4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafcfe00640 1 -- 172.21.15.67:0/601620266 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fafc011ee90 con 0x7fafd80cf880 2024-04-17T10:28:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fafb4066f90 0x7fafb4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 Processor -- start 2024-04-17T10:28:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- start start 2024-04-17T10:28:30.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fafb4066f90 0x7fafb4069450 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009fd0 tx=0x7fafc800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f2290 0x55cc179a7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179a80b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc177bd8a0 con 0x55cc179f1970 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc177f5d10 con 0x55cc179f2290 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cc1777e2d0 con 0x55cc179a7cd0 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35790/0 (socket says 172.21.15.67:35790) 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 -- 172.21.15.67:0/2980534749 learned_addr learned my addr 172.21.15.67:0/2980534749 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafcea00640 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f2290 0x55cc179a7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7e00640 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179a80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 -- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 msgr2=0x55cc179a80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179a80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 -- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f2290 msgr2=0x55cc179a7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f2290 0x55cc179a7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 -- 172.21.15.67:0/2980534749 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc1783c870 con 0x55cc179f1970 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafcea00640 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f2290 0x55cc179a7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafd7400640 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f1d50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fafc8007220 tx=0x7fafc8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e75fb246d9f096b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafc7e00640 1 -- 172.21.15.67:0/2980534749 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8017dd0 con 0x55cc179f1970 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafc7e00640 1 -- 172.21.15.67:0/2980534749 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc801b070 con 0x55cc179f1970 2024-04-17T10:28:30.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafc7e00640 1 -- 172.21.15.67:0/2980534749 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc8015610 con 0x55cc179f1970 2024-04-17T10:28:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- 172.21.15.67:0/2980534749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 msgr2=0x55cc179f1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f1d50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fafc8007220 tx=0x7fafc8014000 comp rx=0 tx=0).stop 2024-04-17T10:28:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- 172.21.15.67:0/2980534749 shutdown_connections 2024-04-17T10:28:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179a80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f2290 0x55cc179a7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 --2- 172.21.15.67:0/2980534749 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f1d50 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- 172.21.15.67:0/2980534749 >> 172.21.15.67:0/2980534749 conn(0x55cc179ae540 msgr2=0x55cc179ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.302+0000 7fafe1403000 1 -- 172.21.15.67:0/2980534749 shutdown_connections 2024-04-17T10:28:30.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- 172.21.15.67:0/2980534749 wait complete. 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- start start 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 Processor -- start 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f1970 0x55cc179f8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc177b0b50 con 0x55cc179f2290 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc177f1db0 con 0x55cc179a7cd0 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cc1779c870 con 0x55cc179f1970 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 10 client.0 _ll_register_callbacks cb 0x55cc178a9fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35802/0 (socket says 172.21.15.67:35802) 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 -- 172.21.15.67:0/1713703896 learned_addr learned my addr 172.21.15.67:0/1713703896 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7400640 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafcea00640 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f1970 0x55cc179f8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 -- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f1970 msgr2=0x55cc179f8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f1970 0x55cc179f8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 -- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 msgr2=0x55cc179f7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f7cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 -- 172.21.15.67:0/1713703896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc1777e520 con 0x55cc179f2290 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafd7e00640 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179ac7e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fafd0004ec0 tx=0x7fafd0001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ed84080d3b0810 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafc4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafc4c00640 1 -- 172.21.15.67:0/1713703896 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0015650 con 0x55cc179f2290 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafc4c00640 1 -- 172.21.15.67:0/1713703896 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fafd0010040 con 0x55cc179f2290 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafc801b070 con 0x55cc179f2290 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafc4c00640 1 -- 172.21.15.67:0/1713703896 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafd00157f0 con 0x55cc179f2290 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 10 client.6246 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55cc177f0d00 con 0x55cc179f2290 2024-04-17T10:28:30.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafc4c00640 1 -- 172.21.15.67:0/1713703896 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc801b070 con 0x55cc179f2290 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafc4c00640 1 -- 172.21.15.67:0/1713703896 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fafd0015990 con 0x55cc179f2290 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 10 client.6246 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 10 client.6246 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 20 client.6246 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179ad550 0x55cc179ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 4 client.6246 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.306+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc17791b10 con 0x55cc179ad550 2024-04-17T10:28:30.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafd7400640 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179ad550 0x55cc179ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafd7400640 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179ad550 0x55cc179ad960 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cad6095fa8adf4a7 server_cookie=e075b9c6a01bff41 in_seq=0 out_seq=0 2024-04-17T10:28:30.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafc4c00640 10 client.6246 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafc4c00640 1 -- 172.21.15.67:0/1713703896 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc17791b10 con 0x55cc179ad550 2024-04-17T10:28:30.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafc4c00640 10 client.6246 handle_command_reply: tid=0 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 client.6246 shutdown 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 msgr2=0x55cc179ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179ac7e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fafd0004ec0 tx=0x7fafd0001690 comp rx=0 tx=0).stop 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 shutdown_connections 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179ad550 0x55cc179ad960 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179ac7e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f1970 0x55cc179f8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 --2- 172.21.15.67:0/1713703896 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafc4c00640 0 client.6246 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 >> 172.21.15.67:0/1713703896 conn(0x55cc179ae540 msgr2=0x55cc179a6780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 shutdown_connections 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 1 -- 172.21.15.67:0/1713703896 wait complete. 2024-04-17T10:28:30.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.310+0000 7fafe1403000 20 client.6246 trim_cache size 0 max 16384 2024-04-17T10:28:30.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 Processor -- start 2024-04-17T10:28:30.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 -- start start 2024-04-17T10:28:30.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f1970 0x55cc179f12a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8202830 con 0x55cc179f2290 2024-04-17T10:28:30.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8201d40 con 0x55cc179f1970 2024-04-17T10:28:30.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8202230 con 0x55cc179a7cd0 2024-04-17T10:28:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179a5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35814/0 (socket says 172.21.15.67:35814) 2024-04-17T10:28:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 -- 172.21.15.67:0/321873183 learned_addr learned my addr 172.21.15.67:0/321873183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7400640 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafcea00640 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f1970 0x55cc179f12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 -- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 msgr2=0x55cc179f0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179f0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 -- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f1970 msgr2=0x55cc179f12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f1970 0x55cc179f12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 -- 172.21.15.67:0/321873183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc17879920 con 0x55cc179f2290 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafd7e00640 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179a5ef0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fafd00011f0 tx=0x7fafd000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285e46bbe973530d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafc7400640 1 -- 172.21.15.67:0/321873183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd002e070 con 0x55cc179f2290 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafc7400640 1 -- 172.21.15.67:0/321873183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd001a020 con 0x55cc179f2290 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.326+0000 7fafc7400640 1 -- 172.21.15.67:0/321873183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd0019020 con 0x55cc179f2290 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/321873183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 msgr2=0x55cc179a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179a5ef0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fafd00011f0 tx=0x7fafd000f000 comp rx=0 tx=0).stop 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/321873183 shutdown_connections 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f2290 0x55cc179a5ef0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179f1970 0x55cc179f12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- 172.21.15.67:0/321873183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179a7cd0 0x55cc179f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/321873183 >> 172.21.15.67:0/321873183 conn(0x55cc179ae540 msgr2=0x55cc179f0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/321873183 shutdown_connections 2024-04-17T10:28:30.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/321873183 wait complete. 2024-04-17T10:28:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- start start 2024-04-17T10:28:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 Processor -- start 2024-04-17T10:28:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f2290 0x55cc179f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc177b7700 con 0x55cc179f1970 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc177fa340 con 0x55cc179a7cd0 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cc177acd20 con 0x55cc179f2290 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 10 client.0 _ll_register_callbacks cb 0x7fafc800f3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafd7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f2290 0x55cc179f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafd7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f6070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35822/0 (socket says 172.21.15.67:35822) 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 -- 172.21.15.67:0/2987258361 learned_addr learned my addr 172.21.15.67:0/2987258361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 -- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f2290 msgr2=0x55cc179f65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f2290 0x55cc179f65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 -- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 msgr2=0x55cc179f5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 -- 172.21.15.67:0/2987258361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc1784fa30 con 0x55cc179f1970 2024-04-17T10:28:30.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafd7e00640 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f2290 0x55cc179f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafd7400640 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafcea00640 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f6070 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fafc001c940 tx=0x7fafc00edb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3525fc624d2215ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafc4c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafc4c00640 1 -- 172.21.15.67:0/2987258361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc00f79f0 con 0x55cc179f1970 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafc4c00640 1 -- 172.21.15.67:0/2987258361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fafc00eea70 con 0x55cc179f1970 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafd001a020 con 0x55cc179f1970 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafc4c00640 1 -- 172.21.15.67:0/2987258361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafc00f7c80 con 0x55cc179f1970 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 10 client.6252 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x55cc1779d150 con 0x55cc179f1970 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafc4c00640 1 -- 172.21.15.67:0/2987258361 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd001a020 con 0x55cc179f1970 2024-04-17T10:28:30.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafc4c00640 1 -- 172.21.15.67:0/2987258361 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fafc0116af0 con 0x55cc179f1970 2024-04-17T10:28:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 10 client.6252 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 10 client.6252 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 20 client.6252 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179f6af0 0x55cc179f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 4 client.6252 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}] 2024-04-17T10:28:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.330+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}) v1 -- 0x55cc179fe9e0 con 0x55cc179f6af0 2024-04-17T10:28:30.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafd7400640 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179f6af0 0x55cc179f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafd7400640 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179f6af0 0x55cc179f6f00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d95fddace43d8e6 server_cookie=d70df0a9cc31fc39 in_seq=0 out_seq=0 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafc4c00640 10 client.6252 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafc4c00640 1 -- 172.21.15.67:0/2987258361 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+26 (crc 0 0 0) 0x55cc179fe9e0 con 0x55cc179f6af0 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafc4c00640 10 client.6252 handle_command_reply: tid=0 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 client.6252 shutdown 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 msgr2=0x55cc179f6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f6070 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fafc001c940 tx=0x7fafc00edb30 comp rx=0 tx=0).stop 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 shutdown_connections 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55cc179f6af0 0x55cc179f6f00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55cc179f2290 0x55cc179f65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55cc179f1970 0x55cc179f6070 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafc4c00640 0 client.6252 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 --2- 172.21.15.67:0/2987258361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55cc179a7cd0 0x55cc179f5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 >> 172.21.15.67:0/2987258361 conn(0x55cc179ae540 msgr2=0x55cc179f0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 shutdown_connections 2024-04-17T10:28:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 1 -- 172.21.15.67:0/2987258361 wait complete. 2024-04-17T10:28:30.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafe1403000 20 client.6252 trim_cache size 0 max 16384 2024-04-17T10:28:30.333 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:30.333 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": -2 2024-04-17T10:28:30.333 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fafb4066f90 msgr2=0x7fafb4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fafb4066f90 0x7fafb4069450 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fafc8009fd0 tx=0x7fafc800b040 comp rx=0 tx=0).stop 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 msgr2=0x7fafd80b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80b4290 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fafc00ae040 tx=0x7fafc0019200 comp rx=0 tx=0).stop 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 shutdown_connections 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fafb4066f90 0x7fafb4069450 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fafd815da90 0x7fafd81f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fafd80cf880 0x7fafd80b4290 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 --2- 172.21.15.67:0/601620266 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fafd8099a50 0x7fafd8151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 >> 172.21.15.67:0/601620266 conn(0x7fafd80d15b0 msgr2=0x7fafd80707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 shutdown_connections 2024-04-17T10:28:30.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.334+0000 7fafdca00640 1 -- 172.21.15.67:0/601620266 wait complete. 2024-04-17T10:28:30.347 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/i/dont/exist' returned '{'return_code': -2}' 2024-04-17T10:28:30.347 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites' 2024-04-17T10:28:30.347 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:30.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites 2024-04-17T10:28:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf94600640 1 Processor -- start 2024-04-17T10:28:30.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf94600640 1 -- start start 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf94600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf8c159440 0x7fcf8c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf94600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c0713e0 0x7fcf8c071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf94600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf8c15cb30 con 0x7fcf8c15be70 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf94600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8c0a8650 con 0x7fcf8c159440 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf94600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c071d80 con 0x7fcf8c0713e0 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf91c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf8c159440 0x7fcf8c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf91c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf8c159440 0x7fcf8c0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35826/0 (socket says 172.21.15.67:35826) 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf91c00640 1 -- 172.21.15.67:0/3077750874 learned_addr learned my addr 172.21.15.67:0/3077750874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf91200640 1 --2- 172.21.15.67:0/3077750874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c0713e0 0x7fcf8c071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf91c00640 1 -- 172.21.15.67:0/3077750874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c0713e0 msgr2=0x7fcf8c071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf91c00640 1 --2- 172.21.15.67:0/3077750874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c0713e0 0x7fcf8c071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.498+0000 7fcf91c00640 1 -- 172.21.15.67:0/3077750874 >> v1:172.21.15.86:6789/0 conn(0x7fcf8c15be70 legacy=0x7fcf8c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf91c00640 1 -- 172.21.15.67:0/3077750874 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c0cd610 con 0x7fcf8c159440 2024-04-17T10:28:30.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf91200640 1 --2- 172.21.15.67:0/3077750874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c0713e0 0x7fcf8c071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:30.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf91c00640 1 --2- 172.21.15.67:0/3077750874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c159440 0x7fcf8c0ce0f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009870 tx=0x7fcf8001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=528abedf5bc8e65f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf8be00640 1 -- 172.21.15.67:0/3077750874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800a7070 con 0x7fcf8c159440 2024-04-17T10:28:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf8be00640 1 -- 172.21.15.67:0/3077750874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf800b0070 con 0x7fcf8c159440 2024-04-17T10:28:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf8be00640 1 -- 172.21.15.67:0/3077750874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800ab890 con 0x7fcf8c159440 2024-04-17T10:28:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 -- 172.21.15.67:0/3077750874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c159440 msgr2=0x7fcf8c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 --2- 172.21.15.67:0/3077750874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c159440 0x7fcf8c0ce0f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009870 tx=0x7fcf8001c560 comp rx=0 tx=0).stop 2024-04-17T10:28:30.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 -- 172.21.15.67:0/3077750874 shutdown_connections 2024-04-17T10:28:30.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 --2- 172.21.15.67:0/3077750874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c0713e0 0x7fcf8c071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 --2- 172.21.15.67:0/3077750874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c159440 0x7fcf8c0ce0f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 -- 172.21.15.67:0/3077750874 >> 172.21.15.67:0/3077750874 conn(0x7fcf8c0b1740 msgr2=0x7fcf8c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 -- 172.21.15.67:0/3077750874 shutdown_connections 2024-04-17T10:28:30.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.502+0000 7fcf94600640 1 -- 172.21.15.67:0/3077750874 wait complete. 2024-04-17T10:28:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 Processor -- start 2024-04-17T10:28:30.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 -- start start 2024-04-17T10:28:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c0713e0 0x7fcf8c150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8c159440 0x7fcf8c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c15be70 0x7fcf8c14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c15d9c0 con 0x7fcf8c0713e0 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c15d840 con 0x7fcf8c159440 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c15db40 con 0x7fcf8c15be70 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c0713e0 0x7fcf8c150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c0713e0 0x7fcf8c150010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35830/0 (socket says 172.21.15.67:35830) 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 -- 172.21.15.67:0/1064245351 learned_addr learned my addr 172.21.15.67:0/1064245351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91200640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8c159440 0x7fcf8c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf92600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c15be70 0x7fcf8c14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 -- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c15be70 msgr2=0x7fcf8c14f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c15be70 0x7fcf8c14f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 -- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8c159440 msgr2=0x7fcf8c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8c159440 0x7fcf8c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 -- 172.21.15.67:0/1064245351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c154f50 con 0x7fcf8c0713e0 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf91c00640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c0713e0 0x7fcf8c150010 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcf800ad040 tx=0x7fcf8001ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf89600640 1 -- 172.21.15.67:0/1064245351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800abe60 con 0x7fcf8c0713e0 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf89600640 1 -- 172.21.15.67:0/1064245351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf800b4820 con 0x7fcf8c0713e0 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf89600640 1 -- 172.21.15.67:0/1064245351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800b4c10 con 0x7fcf8c0713e0 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8c155230 con 0x7fcf8c0713e0 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.506+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8c14fd40 con 0x7fcf8c0713e0 2024-04-17T10:28:30.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.510+0000 7fcf89600640 1 -- 172.21.15.67:0/1064245351 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcf800b3380 con 0x7fcf8c0713e0 2024-04-17T10:28:30.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.510+0000 7fcf89600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf6c066f90 0x7fcf6c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.510+0000 7fcf89600640 1 -- 172.21.15.67:0/1064245351 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf8011f5d0 con 0x7fcf8c0713e0 2024-04-17T10:28:30.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.510+0000 7fcf91200640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf6c066f90 0x7fcf6c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.510+0000 7fcf97afc000 1 Processor -- start 2024-04-17T10:28:30.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.510+0000 7fcf91200640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf6c066f90 0x7fcf6c069450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c1f9930 tx=0x7fcf78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.510+0000 7fcf97afc000 1 -- start start 2024-04-17T10:28:30.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d62ea0 0x5572c1d63280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d637c0 0x5572c1d191d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d19af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c1b67db0 con 0x5572c1d62ea0 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c1af0230 con 0x5572c1d637c0 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572c1b63e50 con 0x5572c1d19710 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d62ea0 0x5572c1d63280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d62ea0 0x5572c1d63280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35844/0 (socket says 172.21.15.67:35844) 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 -- 172.21.15.67:0/438632662 learned_addr learned my addr 172.21.15.67:0/438632662 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf92600640 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d19af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91200640 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d637c0 0x5572c1d191d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 -- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 msgr2=0x5572c1d19af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d19af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 -- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d637c0 msgr2=0x5572c1d191d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d637c0 0x5572c1d191d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 -- 172.21.15.67:0/438632662 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572c1b29750 con 0x5572c1d62ea0 2024-04-17T10:28:30.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf91c00640 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d62ea0 0x5572c1d63280 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcf8001a190 tx=0x7fcf800efe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43fe92f8ed0206f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf7e000640 1 -- 172.21.15.67:0/438632662 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800c4030 con 0x5572c1d62ea0 2024-04-17T10:28:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf7e000640 1 -- 172.21.15.67:0/438632662 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf800b0070 con 0x5572c1d62ea0 2024-04-17T10:28:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf7e000640 1 -- 172.21.15.67:0/438632662 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800f8570 con 0x5572c1d62ea0 2024-04-17T10:28:30.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- 172.21.15.67:0/438632662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d62ea0 msgr2=0x5572c1d63280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d62ea0 0x5572c1d63280 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcf8001a190 tx=0x7fcf800efe10 comp rx=0 tx=0).stop 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- 172.21.15.67:0/438632662 shutdown_connections 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d19af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d637c0 0x5572c1d191d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 --2- 172.21.15.67:0/438632662 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d62ea0 0x5572c1d63280 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- 172.21.15.67:0/438632662 >> 172.21.15.67:0/438632662 conn(0x5572c1d156c0 msgr2=0x5572c1d17ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- 172.21.15.67:0/438632662 shutdown_connections 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- 172.21.15.67:0/438632662 wait complete. 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 -- start start 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.514+0000 7fcf97afc000 1 Processor -- start 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d47ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1d48410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1d48950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c1b22bf0 con 0x5572c1d637c0 2024-04-17T10:28:30.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c1b2c770 con 0x5572c1d62ea0 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572c1b65440 con 0x5572c1d19710 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 10 client.0 _ll_register_callbacks cb 0x5572c1d04790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d47ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d47ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:55874/0 (socket says 172.21.15.67:55874) 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1d48410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 -- 172.21.15.67:0/252862980 learned_addr learned my addr 172.21.15.67:0/252862980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf92600640 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1d48950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 -- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 msgr2=0x5572c1d48410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1d48410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 -- 172.21.15.67:0/252862980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 msgr2=0x5572c1d48950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1d48950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 -- 172.21.15.67:0/252862980 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572c1b0a640 con 0x5572c1d19710 2024-04-17T10:28:30.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91c00640 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d47ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009870 tx=0x7fcf8011d830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5a9762b1a76e08c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3301/0 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 1 -- 172.21.15.67:0/252862980 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800c4040 con 0x5572c1d19710 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 1 -- 172.21.15.67:0/252862980 <== mon.2 v2:172.21.15.86:3301/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf800ef360 con 0x5572c1d19710 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf800b0070 con 0x5572c1d19710 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 1 -- 172.21.15.67:0/252862980 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf800ef560 con 0x5572c1d19710 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 10 client.5609 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x5572c1af0480 con 0x5572c1d19710 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 1 -- 172.21.15.67:0/252862980 <== mon.2 v2:172.21.15.86:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf800b0070 con 0x5572c1d19710 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 1 -- 172.21.15.67:0/252862980 <== mon.2 v2:172.21.15.86:3301/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcf800ebc60 con 0x5572c1d19710 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 10 client.5609 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 10 client.5609 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 20 client.5609 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d72f40 0x5572c1d75330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 4 client.5609 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:30.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572c1b03bb0 con 0x5572c1d72f40 2024-04-17T10:28:30.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91200640 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d72f40 0x5572c1d75330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf91200640 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d72f40 0x5572c1d75330 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2928211f6eb8327 server_cookie=b81ea52b6763cb20 in_seq=0 out_seq=0 2024-04-17T10:28:30.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 10 client.5609 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 1 -- 172.21.15.67:0/252862980 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5572c1b03bb0 con 0x5572c1d72f40 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 10 client.5609 handle_command_reply: tid=0 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 client.5609 shutdown 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 msgr2=0x5572c1d47ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d47ed0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcf80009870 tx=0x7fcf8011d830 comp rx=0 tx=0).stop 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 shutdown_connections 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d72f40 0x5572c1d75330 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1d48950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1d48410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf72a00640 0 client.5609 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 --2- 172.21.15.67:0/252862980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1d47ed0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 >> 172.21.15.67:0/252862980 conn(0x5572c1d156c0 msgr2=0x5572c1d18340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.518+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 shutdown_connections 2024-04-17T10:28:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.522+0000 7fcf97afc000 1 -- 172.21.15.67:0/252862980 wait complete. 2024-04-17T10:28:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.522+0000 7fcf97afc000 20 client.5609 trim_cache size 0 max 16384 2024-04-17T10:28:30.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.534+0000 7fcf97afc000 1 Processor -- start 2024-04-17T10:28:30.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.534+0000 7fcf97afc000 1 -- start start 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1dbfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1dc0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c203170 con 0x5572c1d637c0 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c202330 con 0x5572c1d62ea0 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c151b40 con 0x5572c1d19710 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf92600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1dc0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf92600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1dc0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35864/0 (socket says 172.21.15.67:35864) 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf92600640 1 -- 172.21.15.67:0/1446648815 learned_addr learned my addr 172.21.15.67:0/1446648815 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1dbfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91200640 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 -- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 msgr2=0x5572c1dc0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc0130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 -- 172.21.15.67:0/1446648815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 msgr2=0x5572c1dc0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1dc0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 -- 172.21.15.67:0/1446648815 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572c1be9ef0 con 0x5572c1d19710 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1dbfbf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf800ec370 tx=0x7fcf80105040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3c51aaf33a3f8c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf7d600640 1 -- 172.21.15.67:0/1446648815 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800c4020 con 0x5572c1d19710 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf7d600640 1 -- 172.21.15.67:0/1446648815 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf800b0070 con 0x5572c1d19710 2024-04-17T10:28:30.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf7d600640 1 -- 172.21.15.67:0/1446648815 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800eff10 con 0x5572c1d19710 2024-04-17T10:28:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 msgr2=0x5572c1dbfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1dbfbf0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fcf800ec370 tx=0x7fcf80105040 comp rx=0 tx=0).stop 2024-04-17T10:28:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- 172.21.15.67:0/1446648815 shutdown_connections 2024-04-17T10:28:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d637c0 0x5572c1dc0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc0130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- 172.21.15.67:0/1446648815 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d19710 0x5572c1dbfbf0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- 172.21.15.67:0/1446648815 >> 172.21.15.67:0/1446648815 conn(0x5572c1d156c0 msgr2=0x5572c1d70490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- 172.21.15.67:0/1446648815 shutdown_connections 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- 172.21.15.67:0/1446648815 wait complete. 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- start start 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 Processor -- start 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d19710 0x5572c1dc3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d637c0 0x5572c1d75640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572c1b22bf0 con 0x5572c1d19710 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c201d90 con 0x5572c1d62ea0 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572c1b670c0 con 0x5572c1d637c0 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf97afc000 10 client.0 _ll_register_callbacks cb 0x7fcf7800a1d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d19710 0x5572c1dc3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf92600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d637c0 0x5572c1d75640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d19710 0x5572c1dc3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35870/0 (socket says 172.21.15.67:35870) 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91c00640 1 -- 172.21.15.67:0/2969247280 learned_addr learned my addr 172.21.15.67:0/2969247280 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.538+0000 7fcf91200640 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91c00640 1 -- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d637c0 msgr2=0x5572c1d75640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91c00640 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d637c0 0x5572c1d75640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91c00640 1 -- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 msgr2=0x5572c1dc4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91c00640 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91c00640 1 -- 172.21.15.67:0/2969247280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572c1b279a0 con 0x5572c1d19710 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf92600640 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d637c0 0x5572c1d75640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91c00640 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d19710 0x5572c1dc3d20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fcf800f8730 tx=0x7fcf80019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c72cd4f985a223 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 1 -- 172.21.15.67:0/2969247280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf800c4050 con 0x5572c1d19710 2024-04-17T10:28:30.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 1 -- 172.21.15.67:0/2969247280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf8011f250 con 0x5572c1d19710 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf800b0070 con 0x5572c1d19710 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 1 -- 172.21.15.67:0/2969247280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf800eff10 con 0x5572c1d19710 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 10 client.6270 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x5572c18245a0 con 0x5572c1d19710 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 1 -- 172.21.15.67:0/2969247280 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf800b0070 con 0x5572c1d19710 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 1 -- 172.21.15.67:0/2969247280 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcf800eedd0 con 0x5572c1d19710 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 10 client.6270 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 10 client.6270 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 20 client.6270 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d76000 0x5572c1d76410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 4 client.6270 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}] 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}) v1 -- 0x5572c1b17080 con 0x5572c1d76000 2024-04-17T10:28:30.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91200640 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d76000 0x5572c1d76410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf91200640 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d76000 0x5572c1d76410 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf8b634db21e7037 server_cookie=d9099f92a973c89c in_seq=0 out_seq=0 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 10 client.6270 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 1 -- 172.21.15.67:0/2969247280 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5572c1b17080 con 0x5572c1d76000 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 10 client.6270 handle_command_reply: tid=0 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 client.6270 shutdown 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d19710 msgr2=0x5572c1dc3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d19710 0x5572c1dc3d20 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fcf800f8730 tx=0x7fcf80019300 comp rx=0 tx=0).stop 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 shutdown_connections 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5572c1d76000 0x5572c1d76410 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5572c1d637c0 0x5572c1d75640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5572c1d62ea0 0x5572c1dc4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 --2- 172.21.15.67:0/2969247280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5572c1d19710 0x5572c1dc3d20 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 >> 172.21.15.67:0/2969247280 conn(0x5572c1d156c0 msgr2=0x5572c1d617b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf72a00640 0 client.6270 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 shutdown_connections 2024-04-17T10:28:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 1 -- 172.21.15.67:0/2969247280 wait complete. 2024-04-17T10:28:30.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.542+0000 7fcf97afc000 20 client.6270 trim_cache size 0 max 16384 2024-04-17T10:28:30.542 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "9097c79d-4223-4eab-a635-a753e3a8a673", 2024-04-17T10:28:30.542 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:30.542 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf6c066f90 msgr2=0x7fcf6c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf6c066f90 0x7fcf6c069450 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c1f9930 tx=0x7fcf78008040 comp rx=0 tx=0).stop 2024-04-17T10:28:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c0713e0 msgr2=0x7fcf8c150010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c0713e0 0x7fcf8c150010 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fcf800ad040 tx=0x7fcf8001ada0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 shutdown_connections 2024-04-17T10:28:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf6c066f90 0x7fcf6c069450 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf8c15be70 0x7fcf8c14f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf8c159440 0x7fcf8c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 --2- 172.21.15.67:0/1064245351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf8c0713e0 0x7fcf8c150010 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 >> 172.21.15.67:0/1064245351 conn(0x7fcf8c0b1740 msgr2=0x7fcf8c0acbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 shutdown_connections 2024-04-17T10:28:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.546+0000 7fcf94600640 1 -- 172.21.15.67:0/1064245351 wait complete. 2024-04-17T10:28:30.556 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites' returned '{'return_code': 0, 'scrub_tag': '9097c79d-4223-4eab-a635-a753e3a8a673', 'mode': 'asynchronous'}' 2024-04-17T10:28:30.556 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' 2024-04-17T10:28:30.557 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:30.557 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-04-17T10:28:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.706+0000 7f6abda00640 1 Processor -- start 2024-04-17T10:28:30.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.706+0000 7f6abda00640 1 -- start start 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.706+0000 7f6abda00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ab81474c0 0x7f6ab80b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.706+0000 7f6abda00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6abda00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab81616a0 con 0x7f6ab80b3920 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6abda00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab8161940 con 0x7f6ab81474c0 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6abda00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8161c20 con 0x7f6ab806b3a0 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6aae000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ab80b3920 0x7f6ab806ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56138/0 (socket says 172.21.15.67:56138) 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6aae000640 1 -- 172.21.15.67:0/2892133807 learned_addr learned my addr 172.21.15.67:0/2892133807 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab7400640 1 --2- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/2892133807 >> v2:172.21.15.86:3300/0 conn(0x7f6ab81474c0 0x7f6ab80b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6000640 1 -- 172.21.15.67:0/2892133807 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940078506 0 0) 0x7f6ab81616a0 con 0x7f6ab80b3920 2024-04-17T10:28:30.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6000640 1 -- 172.21.15.67:0/2892133807 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a9c003530 con 0x7f6ab80b3920 2024-04-17T10:28:30.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6a00640 1 -- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 msgr2=0x7f6ab80d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80d0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6a00640 1 -- 172.21.15.67:0/2892133807 >> v1:172.21.15.67:6789/0 conn(0x7f6ab80b3920 legacy=0x7f6ab806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6a00640 1 -- 172.21.15.67:0/2892133807 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab8161f00 con 0x7f6ab81474c0 2024-04-17T10:28:30.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab7400640 1 --2- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.710+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81474c0 0x7f6ab80b1b90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80097c0 tx=0x7f6aa801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=615e17ba4b58b5e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6ab6000640 1 -- 172.21.15.67:0/2892133807 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80b5070 con 0x7f6ab81474c0 2024-04-17T10:28:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6ab6000640 1 -- 172.21.15.67:0/2892133807 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa80a8990 con 0x7f6ab81474c0 2024-04-17T10:28:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6ab6000640 1 -- 172.21.15.67:0/2892133807 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80a8d60 con 0x7f6ab81474c0 2024-04-17T10:28:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81474c0 msgr2=0x7f6ab80b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 --2- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81474c0 0x7f6ab80b1b90 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80097c0 tx=0x7f6aa801ec90 comp rx=0 tx=0).stop 2024-04-17T10:28:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- 172.21.15.67:0/2892133807 shutdown_connections 2024-04-17T10:28:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 --2- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80d0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 --2- 172.21.15.67:0/2892133807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab81474c0 0x7f6ab80b1b90 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- 172.21.15.67:0/2892133807 >> 172.21.15.67:0/2892133807 conn(0x7f6ab80b5ef0 msgr2=0x7f6ab8099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- 172.21.15.67:0/2892133807 shutdown_connections 2024-04-17T10:28:30.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- 172.21.15.67:0/2892133807 wait complete. 2024-04-17T10:28:30.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 Processor -- start 2024-04-17T10:28:30.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- start start 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80ce090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab80b3920 0x7f6ab8142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ab81474c0 0x7f6ab8142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab8162420 con 0x7f6ab81474c0 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80b3d70 con 0x7f6ab80b3920 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6abda00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80d09c0 con 0x7f6ab806b3a0 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6ab7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ab81474c0 0x7f6ab8142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6ab7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ab81474c0 0x7f6ab8142690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35884/0 (socket says 172.21.15.67:35884) 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6ab7400640 1 -- 172.21.15.67:0/1723499687 learned_addr learned my addr 172.21.15.67:0/1723499687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.714+0000 7f6aae000640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab80b3920 0x7f6ab8142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6ab7400640 1 -- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 msgr2=0x7f6ab80ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6ab7400640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6ab7400640 1 -- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab80b3920 msgr2=0x7f6ab8142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6ab7400640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab80b3920 0x7f6ab8142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6ab7400640 1 -- 172.21.15.67:0/1723499687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab81f74f0 con 0x7f6ab81474c0 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6ab7400640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ab81474c0 0x7f6ab8142690 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e950 tx=0x7f6ab000ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6aaf400640 1 -- 172.21.15.67:0/1723499687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0017070 con 0x7f6ab81474c0 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6aaf400640 1 -- 172.21.15.67:0/1723499687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab000ca00 con 0x7f6ab81474c0 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6aaf400640 1 -- 172.21.15.67:0/1723499687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab000cdd0 con 0x7f6ab81474c0 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab81f77d0 con 0x7f6ab81474c0 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab81f7d90 con 0x7f6ab81474c0 2024-04-17T10:28:30.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6aaf400640 1 -- 172.21.15.67:0/1723499687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6ab0013470 con 0x7f6ab81474c0 2024-04-17T10:28:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6aaf400640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a94066f60 0x7f6a94069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.718+0000 7f6aaf400640 1 -- 172.21.15.67:0/1723499687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6ab00850a0 con 0x7f6ab81474c0 2024-04-17T10:28:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a94066f60 0x7f6a94069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 Processor -- start 2024-04-17T10:28:30.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- start start 2024-04-17T10:28:30.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a94066f60 0x7f6a94069420 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80b2000 tx=0x7f6aa8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc22575e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257b20 0x564cc223ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc223bf60 0x564cc223c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc20f44a0 con 0x564cc2257b20 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc205c210 con 0x564cc223bf60 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x564cc1fe4980 con 0x564cc2257200 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257b20 0x564cc223ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc22575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257b20 0x564cc223ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35900/0 (socket says 172.21.15.67:35900) 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 -- 172.21.15.67:0/2822880439 learned_addr learned my addr 172.21.15.67:0/2822880439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ab7400640 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc223bf60 0x564cc223c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 -- 172.21.15.67:0/2822880439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 msgr2=0x564cc22575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc22575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 -- 172.21.15.67:0/2822880439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc223bf60 msgr2=0x564cc223c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc223bf60 0x564cc223c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 -- 172.21.15.67:0/2822880439 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cc2057200 con 0x564cc2257b20 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aae000640 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257b20 0x564cc223ba20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000f300 tx=0x7f6aa0000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=538226e23d86813b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aa7400640 1 -- 172.21.15.67:0/2822880439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00072e0 con 0x564cc2257b20 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aa7400640 1 -- 172.21.15.67:0/2822880439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0018070 con 0x564cc2257b20 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6aa7400640 1 -- 172.21.15.67:0/2822880439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00076c0 con 0x564cc2257b20 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2822880439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257b20 msgr2=0x564cc223ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257b20 0x564cc223ba20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f6aa000f300 tx=0x7f6aa0000f50 comp rx=0 tx=0).stop 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2822880439 shutdown_connections 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc223bf60 0x564cc223c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257b20 0x564cc223ba20 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2822880439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc22575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2822880439 >> 172.21.15.67:0/2822880439 conn(0x564cc22044f0 msgr2=0x564cc2213a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2822880439 shutdown_connections 2024-04-17T10:28:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2822880439 wait complete. 2024-04-17T10:28:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 -- start start 2024-04-17T10:28:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.722+0000 7f6ac0f78000 1 Processor -- start 2024-04-17T10:28:30.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc223bf60 0x564cc225e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc225eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc223c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc2016d80 con 0x564cc223bf60 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc20582b0 con 0x564cc2257b20 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x564cc201d830 con 0x564cc2257200 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 10 client.0 _ll_register_callbacks cb 0x564cc210f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc223bf60 0x564cc225e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6aae000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc225eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc223c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc223bf60 0x564cc225e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35914/0 (socket says 172.21.15.67:35914) 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 -- 172.21.15.67:0/2924520282 learned_addr learned my addr 172.21.15.67:0/2924520282 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 -- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 msgr2=0x564cc225eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc225eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 -- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 msgr2=0x564cc223c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc223c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 -- 172.21.15.67:0/2924520282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564cc2028130 con 0x564cc223bf60 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab7400640 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc223c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc223bf60 0x564cc225e980 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80b3040 tx=0x7f6aa80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac42f5488ee1be58 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6a9be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6a9be00640 1 -- 172.21.15.67:0/2924520282 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80b5040 con 0x564cc223bf60 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6a9be00640 1 -- 172.21.15.67:0/2924520282 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6aa80bbb40 con 0x564cc223bf60 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6aa0018070 con 0x564cc223bf60 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6a9be00640 1 -- 172.21.15.67:0/2924520282 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6aa80d5050 con 0x564cc223bf60 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 10 client.6282 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x564cc20dd490 con 0x564cc223bf60 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6a9be00640 1 -- 172.21.15.67:0/2924520282 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0018070 con 0x564cc223bf60 2024-04-17T10:28:30.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6a9be00640 1 -- 172.21.15.67:0/2924520282 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6aa80d8020 con 0x564cc223bf60 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 10 client.6282 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 10 client.6282 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 20 client.6282 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc223d080 0x564cc223d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 4 client.6282 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564cc1ff8020 con 0x564cc223d080 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6aae000640 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc223d080 0x564cc223d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6aae000640 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc223d080 0x564cc223d460 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c70e8db5fcbf05d server_cookie=2a0910614a010d74 in_seq=0 out_seq=0 2024-04-17T10:28:30.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.726+0000 7f6a9be00640 10 client.6282 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6a9be00640 1 -- 172.21.15.67:0/2924520282 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564cc1ff8020 con 0x564cc223d080 2024-04-17T10:28:30.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6a9be00640 10 client.6282 handle_command_reply: tid=0 2024-04-17T10:28:30.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 client.6282 shutdown 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc223bf60 msgr2=0x564cc225e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc223bf60 0x564cc225e980 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80b3040 tx=0x7f6aa80046d0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 shutdown_connections 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc223d080 0x564cc223d460 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc223c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc2257200 0x564cc225eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/2924520282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc223bf60 0x564cc225e980 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6a9be00640 0 client.6282 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 >> 172.21.15.67:0/2924520282 conn(0x564cc22044f0 msgr2=0x564cc220baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 shutdown_connections 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 1 -- 172.21.15.67:0/2924520282 wait complete. 2024-04-17T10:28:30.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.730+0000 7f6ac0f78000 20 client.6282 trim_cache size 0 max 16384 2024-04-17T10:28:30.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 Processor -- start 2024-04-17T10:28:30.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- start start 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2256950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2255520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc2255a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab82002f0 con 0x564cc2257200 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81f4d10 con 0x564cc2257b20 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81f4110 con 0x564cc223bf60 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2255520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2255520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35924/0 (socket says 172.21.15.67:35924) 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 -- 172.21.15.67:0/790370465 learned_addr learned my addr 172.21.15.67:0/790370465 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ab7400640 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc2255a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2256950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 -- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 msgr2=0x564cc2256950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2256950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 -- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 msgr2=0x564cc2255a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc2255a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 -- 172.21.15.67:0/790370465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564cc201ba30 con 0x564cc2257200 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2256950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ab7400640 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc2255a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aae000640 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2255520 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0014000 tx=0x7f6aa0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d74e5b2b5561325 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aa6a00640 1 -- 172.21.15.67:0/790370465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa0004d70 con 0x564cc2257200 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aa6a00640 1 -- 172.21.15.67:0/790370465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0018070 con 0x564cc2257200 2024-04-17T10:28:30.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6aa6a00640 1 -- 172.21.15.67:0/790370465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00077e0 con 0x564cc2257200 2024-04-17T10:28:30.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- 172.21.15.67:0/790370465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 msgr2=0x564cc2255520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2255520 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0014000 tx=0x7f6aa0012000 comp rx=0 tx=0).stop 2024-04-17T10:28:30.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- 172.21.15.67:0/790370465 shutdown_connections 2024-04-17T10:28:30.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc2255a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2255520 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/790370465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2256950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- 172.21.15.67:0/790370465 >> 172.21.15.67:0/790370465 conn(0x564cc22044f0 msgr2=0x564cc223a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- 172.21.15.67:0/790370465 shutdown_connections 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- 172.21.15.67:0/790370465 wait complete. 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 -- start start 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.746+0000 7f6ac0f78000 1 Processor -- start 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2275430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2275970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc220c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564cc1fe8190 con 0x564cc2257200 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81f4710 con 0x564cc2257b20 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x564cc205b520 con 0x564cc223bf60 2024-04-17T10:28:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 10 client.0 _ll_register_callbacks cb 0x7f6aa0001d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2275970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ab6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2275430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2275970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35928/0 (socket says 172.21.15.67:35928) 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 -- 172.21.15.67:0/604062940 learned_addr learned my addr 172.21.15.67:0/604062940 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ab7400640 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc220c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 -- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 msgr2=0x564cc2275430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2275430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 -- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 msgr2=0x564cc220c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc220c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 -- 172.21.15.67:0/604062940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564cc2024040 con 0x564cc2257200 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2275430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6aae000640 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2275970 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0011010 tx=0x7f6aa0001430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81ae5151b0354d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 1 -- 172.21.15.67:0/604062940 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa00079d0 con 0x564cc2257200 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 1 -- 172.21.15.67:0/604062940 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6aa002a030 con 0x564cc2257200 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6aa0018070 con 0x564cc2257200 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 1 -- 172.21.15.67:0/604062940 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6aa003a050 con 0x564cc2257200 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 10 client.6288 fetch_fsmap learned FSMap version 28 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=2+}) v3 -- 0x564cc1d18900 con 0x564cc2257200 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 1 -- 172.21.15.67:0/604062940 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa0018070 con 0x564cc2257200 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 1 -- 172.21.15.67:0/604062940 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 28) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6aa003d020 con 0x564cc2257200 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 10 client.6288 fetch_fsmap finished waiting for FSMap version 28 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 10 client.6288 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 20 client.6288 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc220cc20 0x564cc220d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 4 client.6288 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}] 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}) v1 -- 0x564cc2069750 con 0x564cc220cc20 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc220cc20 0x564cc220d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ab6a00640 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc220cc20 0x564cc220d030 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a6ecc6fd6a6d254 server_cookie=d477842d9332e814 in_seq=0 out_seq=0 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 10 client.6288 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 1 -- 172.21.15.67:0/604062940 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564cc2069750 con 0x564cc220cc20 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6a9be00640 10 client.6288 handle_command_reply: tid=0 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 client.6288 shutdown 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 msgr2=0x564cc2275970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2275970 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0011010 tx=0x7f6aa0001430 comp rx=0 tx=0).stop 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.750+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 shutdown_connections 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x564cc220cc20 0x564cc220d030 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564cc2257b20 0x564cc220c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564cc2257200 0x564cc2275970 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 1 --2- 172.21.15.67:0/604062940 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564cc223bf60 0x564cc2275430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6a9be00640 0 client.6288 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 >> 172.21.15.67:0/604062940 conn(0x564cc22044f0 msgr2=0x564cc220ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 shutdown_connections 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 1 -- 172.21.15.67:0/604062940 wait complete. 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6ac0f78000 20 client.6288 trim_cache size 0 max 16384 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "20b9dada-16c7-4e7d-918e-2784a7b4f77e", 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:30.751 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a94066f60 msgr2=0x7f6a94069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a94066f60 0x7f6a94069420 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80b2000 tx=0x7f6aa8008040 comp rx=0 tx=0).stop 2024-04-17T10:28:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ab81474c0 msgr2=0x7f6ab8142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ab81474c0 0x7f6ab8142690 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000e950 tx=0x7f6ab000ee10 comp rx=0 tx=0).stop 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 shutdown_connections 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a94066f60 0x7f6a94069420 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ab81474c0 0x7f6ab8142690 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ab80b3920 0x7f6ab8142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 --2- 172.21.15.67:0/1723499687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ab806b3a0 0x7f6ab80ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 >> 172.21.15.67:0/1723499687 conn(0x7f6ab80b5ef0 msgr2=0x7f6ab806fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 shutdown_connections 2024-04-17T10:28:30.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.754+0000 7f6abda00640 1 -- 172.21.15.67:0/1723499687 wait complete. 2024-04-17T10:28:30.765 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' returned '{'return_code': 0, 'scrub_tag': '20b9dada-16c7-4e7d-918e-2784a7b4f77e', 'mode': 'asynchronous'}' 2024-04-17T10:28:30.765 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /' 2024-04-17T10:28:30.765 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:30.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / 2024-04-17T10:28:30.886 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:28:30.894+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-17T10:28:30.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf8c00640 1 Processor -- start 2024-04-17T10:28:30.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf8c00640 1 -- start start 2024-04-17T10:28:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf8c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcf4159440 0x7fbcf40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf8c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf415cb30 con 0x7fbcf40713e0 2024-04-17T10:28:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf8c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf40cd610 con 0x7fbcf415c580 2024-04-17T10:28:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf8c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf40a8650 con 0x7fbcf4159440 2024-04-17T10:28:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf1400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcf40713e0 0x7fbcf415be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56150/0 (socket says 172.21.15.67:56150) 2024-04-17T10:28:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf1400640 1 -- 172.21.15.67:0/4206000435 learned_addr learned my addr 172.21.15.67:0/4206000435 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcf4159440 0x7fbcf40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf1e00640 1 -- 172.21.15.67:0/4206000435 >> v1:172.21.15.86:6789/0 conn(0x7fbcf415c580 legacy=0x7fbcf415d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf1e00640 1 -- 172.21.15.67:0/4206000435 >> v1:172.21.15.67:6789/0 conn(0x7fbcf40713e0 legacy=0x7fbcf415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf0a00640 1 -- 172.21.15.67:0/4206000435 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814569058 0 0) 0x7fbcf415cb30 con 0x7fbcf40713e0 2024-04-17T10:28:30.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.918+0000 7fbcf1e00640 1 -- 172.21.15.67:0/4206000435 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf40cd8f0 con 0x7fbcf4159440 2024-04-17T10:28:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/4206000435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4159440 0x7fbcf40ce0f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097c0 tx=0x7fbcf415df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69950be2ca2c0b84 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf0a00640 1 -- 172.21.15.67:0/4206000435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40b1070 con 0x7fbcf4159440 2024-04-17T10:28:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf0a00640 1 -- 172.21.15.67:0/4206000435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40ac410 con 0x7fbcf4159440 2024-04-17T10:28:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf0a00640 1 -- 172.21.15.67:0/4206000435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40ac7e0 con 0x7fbcf4159440 2024-04-17T10:28:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4206000435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4159440 msgr2=0x7fbcf40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4206000435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4159440 0x7fbcf40ce0f0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097c0 tx=0x7fbcf415df50 comp rx=0 tx=0).stop 2024-04-17T10:28:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4206000435 shutdown_connections 2024-04-17T10:28:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4206000435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf4159440 0x7fbcf40ce0f0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4206000435 >> 172.21.15.67:0/4206000435 conn(0x7fbcf40b1740 msgr2=0x7fbcf40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4206000435 shutdown_connections 2024-04-17T10:28:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4206000435 wait complete. 2024-04-17T10:28:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.922+0000 7fbcf8c00640 1 Processor -- start 2024-04-17T10:28:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 -- start start 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf40713e0 0x7fbcf40b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf4159440 0x7fbcf41525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf415c580 0x7fbcf4140a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf4169640 con 0x7fbcf40713e0 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf41694c0 con 0x7fbcf4159440 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf41697c0 con 0x7fbcf415c580 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf40713e0 0x7fbcf40b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf40713e0 0x7fbcf40b0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35954/0 (socket says 172.21.15.67:35954) 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 -- 172.21.15.67:0/4151567770 learned_addr learned my addr 172.21.15.67:0/4151567770 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1400640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf4159440 0x7fbcf41525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf2800640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf415c580 0x7fbcf4140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 -- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf415c580 msgr2=0x7fbcf4140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf415c580 0x7fbcf4140a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 -- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf4159440 msgr2=0x7fbcf41525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf4159440 0x7fbcf41525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 -- 172.21.15.67:0/4151567770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf40b2630 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf40713e0 0x7fbcf40b0ad0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbce40190b0 tx=0x7fbce4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcea000640 1 -- 172.21.15.67:0/4151567770 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40b1040 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcea000640 1 -- 172.21.15.67:0/4151567770 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40bc070 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf40b2910 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcea000640 1 -- 172.21.15.67:0/4151567770 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40b5410 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.926+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf40b2ed0 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcea000640 1 -- 172.21.15.67:0/4151567770 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbce40b55b0 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcea000640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd4066f90 0x7fbcd4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcea000640 1 -- 172.21.15.67:0/4151567770 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbce411eb60 con 0x7fbcf40713e0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1400640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd4066f90 0x7fbcd4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 Processor -- start 2024-04-17T10:28:30.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 -- start start 2024-04-17T10:28:30.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1400640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd4066f90 0x7fbcd4069450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4156630 tx=0x7fbce0007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820edd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820e92ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820e93410 0x555820e937f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555820ce0f40 con 0x555820edd000 2024-04-17T10:28:30.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555820c68fc0 con 0x555820e93410 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555820cdcfe0 con 0x555820edd920 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820edd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820edd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35958/0 (socket says 172.21.15.67:35958) 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 -- 172.21.15.67:0/3940278129 learned_addr learned my addr 172.21.15.67:0/3940278129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf2800640 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820e93410 0x555820e937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1400640 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820e92ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 -- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 msgr2=0x555820e92ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820e92ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 -- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820e93410 msgr2=0x555820e937f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820e93410 0x555820e937f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 -- 172.21.15.67:0/3940278129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555820d28a50 con 0x555820edd000 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1400640 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820e92ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820edd3e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbce40eeff0 tx=0x7fbce40eed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32baba00b728fea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcdea00640 1 -- 172.21.15.67:0/3940278129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40ed5f0 con 0x555820edd000 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcdea00640 1 -- 172.21.15.67:0/3940278129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40f3030 con 0x555820edd000 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.930+0000 7fbcdea00640 1 -- 172.21.15.67:0/3940278129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40ee740 con 0x555820edd000 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/3940278129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 msgr2=0x555820edd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820edd3e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbce40eeff0 tx=0x7fbce40eed10 comp rx=0 tx=0).stop 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/3940278129 shutdown_connections 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820e93410 0x555820e937f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820e92ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/3940278129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820edd3e0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/3940278129 >> 172.21.15.67:0/3940278129 conn(0x555820e8f3c0 msgr2=0x555820e917b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/3940278129 shutdown_connections 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/3940278129 wait complete. 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- start start 2024-04-17T10:28:30.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 Processor -- start 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820e9ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820e9f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820e9f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555820ce5570 con 0x555820edd000 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555820c9b5f0 con 0x555820edd920 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555820c9d0a0 con 0x555820e93410 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 10 client.0 _ll_register_callbacks cb 0x555820d957a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820e9f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820e9f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35972/0 (socket says 172.21.15.67:35972) 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 -- 172.21.15.67:0/1910034577 learned_addr learned my addr 172.21.15.67:0/1910034577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820e9ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf2800640 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820e9f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 -- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 msgr2=0x555820e9ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820e9ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 -- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 msgr2=0x555820e9f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820e9f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 -- 172.21.15.67:0/1910034577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555820c69210 con 0x555820edd000 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820e9f340 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fbce0013470 tx=0x7fbce0007e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c2f3d94d269799 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcd3400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcd3400640 1 -- 172.21.15.67:0/1910034577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0027060 con 0x555820edd000 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcd3400640 1 -- 172.21.15.67:0/1910034577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbce0022c40 con 0x555820edd000 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbce40f3030 con 0x555820edd000 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcd3400640 1 -- 172.21.15.67:0/1910034577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbce0022e40 con 0x555820edd000 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 10 client.6303 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.934+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x555820d65400 con 0x555820edd000 2024-04-17T10:28:30.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcd3400640 1 -- 172.21.15.67:0/1910034577 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40f3030 con 0x555820edd000 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcd3400640 1 -- 172.21.15.67:0/1910034577 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbce0012560 con 0x555820edd000 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 10 client.6303 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 10 client.6303 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 20 client.6303 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ea0290 0x555820eef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 4 client.6303 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555820c7cc90 con 0x555820ea0290 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ea0290 0x555820eef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ea0290 0x555820eef2c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5808daf97a9e5985 server_cookie=b6c501b824d66fda in_seq=0 out_seq=0 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcd3400640 10 client.6303 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcd3400640 1 -- 172.21.15.67:0/1910034577 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555820c7cc90 con 0x555820ea0290 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcd3400640 10 client.6303 handle_command_reply: tid=0 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 client.6303 shutdown 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 msgr2=0x555820e9f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820e9f340 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fbce0013470 tx=0x7fbce0007e50 comp rx=0 tx=0).stop 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 shutdown_connections 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ea0290 0x555820eef2c0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820e9f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820e9f340 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1910034577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820e9ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcd3400640 0 client.6303 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 >> 172.21.15.67:0/1910034577 conn(0x555820e8f3c0 msgr2=0x555820e92040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 shutdown_connections 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1910034577 wait complete. 2024-04-17T10:28:30.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.938+0000 7fbcfbfcb000 20 client.6303 trim_cache size 0 max 16384 2024-04-17T10:28:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 Processor -- start 2024-04-17T10:28:30.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 -- start start 2024-04-17T10:28:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820e93410 0x555820e927c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd000 0x555820e92d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820eef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf4203610 con 0x555820e93410 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf4202810 con 0x555820edd000 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf40affe0 con 0x555820edd920 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820e93410 0x555820e927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820e93410 0x555820e927c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35988/0 (socket says 172.21.15.67:35988) 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1339297876 learned_addr learned my addr 172.21.15.67:0/1339297876 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf2800640 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820eef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd000 0x555820e92d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 msgr2=0x555820eef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820eef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd000 msgr2=0x555820e92d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd000 0x555820e92d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 -- 172.21.15.67:0/1339297876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555820d3bc10 con 0x555820e93410 2024-04-17T10:28:30.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820e93410 0x555820e927c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fbce4097e80 tx=0x7fbce40f2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=843916dcef36060e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcde000640 1 -- 172.21.15.67:0/1339297876 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce411b040 con 0x555820e93410 2024-04-17T10:28:30.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcde000640 1 -- 172.21.15.67:0/1339297876 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40ed5f0 con 0x555820e93410 2024-04-17T10:28:30.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.954+0000 7fbcde000640 1 -- 172.21.15.67:0/1339297876 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40eb590 con 0x555820e93410 2024-04-17T10:28:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1339297876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820e93410 msgr2=0x555820e927c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820e93410 0x555820e927c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fbce4097e80 tx=0x7fbce40f2000 comp rx=0 tx=0).stop 2024-04-17T10:28:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1339297876 shutdown_connections 2024-04-17T10:28:30.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820edd920 0x555820eef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd000 0x555820e92d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1339297876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820e93410 0x555820e927c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1339297876 >> 172.21.15.67:0/1339297876 conn(0x555820e8f3c0 msgr2=0x555820e9eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1339297876 shutdown_connections 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1339297876 wait complete. 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- start start 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 Processor -- start 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820f3df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820f3e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820ef8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555820d1ee80 con 0x555820edd000 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf4202330 con 0x555820edd920 2024-04-17T10:28:30.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555820ca47d0 con 0x555820e93410 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 10 client.0 _ll_register_callbacks cb 0x555820d332c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820f3e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820f3e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35994/0 (socket says 172.21.15.67:35994) 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820f3df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 -- 172.21.15.67:0/1189814254 learned_addr learned my addr 172.21.15.67:0/1189814254 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf2800640 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820ef8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 -- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 msgr2=0x555820f3df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820f3df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 -- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 msgr2=0x555820ef8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820ef8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 -- 172.21.15.67:0/1189814254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555820c75bb0 con 0x555820edd000 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf2800640 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820ef8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcf1400640 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820f3e440 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbce0001e70 tx=0x7fbce000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef73840153fb5f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcd3400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcd3400640 1 -- 172.21.15.67:0/1189814254 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0012e10 con 0x555820edd000 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcd3400640 1 -- 172.21.15.67:0/1189814254 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbce0032040 con 0x555820edd000 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbce40ed5f0 con 0x555820edd000 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcd3400640 1 -- 172.21.15.67:0/1189814254 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbce0012a60 con 0x555820edd000 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 10 client.6309 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.958+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x555820c88030 con 0x555820edd000 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcd3400640 1 -- 172.21.15.67:0/1189814254 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40ed5f0 con 0x555820edd000 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcd3400640 1 -- 172.21.15.67:0/1189814254 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbce002d430 con 0x555820edd000 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 10 client.6309 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 10 client.6309 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 20 client.6309 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ef9be0 0x555820ef9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 4 client.6309 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/"}] 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/"}) v1 -- 0x555820cee480 con 0x555820ef9be0 2024-04-17T10:28:30.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ef9be0 0x555820ef9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:30.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf1e00640 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ef9be0 0x555820ef9ff0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=686fe45f7236957a server_cookie=8aa3ebd1f510fa75 in_seq=0 out_seq=0 2024-04-17T10:28:30.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcd3400640 10 client.6309 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcd3400640 1 -- 172.21.15.67:0/1189814254 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555820cee480 con 0x555820ef9be0 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcd3400640 10 client.6309 handle_command_reply: tid=0 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 client.6309 shutdown 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 msgr2=0x555820f3e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820f3e440 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbce0001e70 tx=0x7fbce000a000 comp rx=0 tx=0).stop 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 shutdown_connections 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555820ef9be0 0x555820ef9ff0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555820edd920 0x555820ef8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555820edd000 0x555820f3e440 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 --2- 172.21.15.67:0/1189814254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555820e93410 0x555820f3df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcd3400640 0 client.6309 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 >> 172.21.15.67:0/1189814254 conn(0x555820e8f3c0 msgr2=0x555820e9f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 shutdown_connections 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 1 -- 172.21.15.67:0/1189814254 wait complete. 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcfbfcb000 20 client.6309 trim_cache size 0 max 16384 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "7154ce75-b751-43dd-b741-7cc0ec2fb57e", 2024-04-17T10:28:30.960 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:30.961 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:30.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd4066f90 msgr2=0x7fbcd4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd4066f90 0x7fbcd4069450 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4156630 tx=0x7fbce0007240 comp rx=0 tx=0).stop 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf40713e0 msgr2=0x7fbcf40b0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf40713e0 0x7fbcf40b0ad0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbce40190b0 tx=0x7fbce4019240 comp rx=0 tx=0).stop 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 shutdown_connections 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd4066f90 0x7fbcd4069450 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf415c580 0x7fbcf4140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf4159440 0x7fbcf41525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 --2- 172.21.15.67:0/4151567770 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf40713e0 0x7fbcf40b0ad0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 >> 172.21.15.67:0/4151567770 conn(0x7fbcf40b1740 msgr2=0x7fbcf40ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 shutdown_connections 2024-04-17T10:28:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:30.962+0000 7fbcf8c00640 1 -- 172.21.15.67:0/4151567770 wait complete. 2024-04-17T10:28:30.975 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /' returned '{'return_code': 0, 'scrub_tag': '7154ce75-b751-43dd-b741-7cc0ec2fb57e', 'mode': 'asynchronous'}' 2024-04-17T10:28:30.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:30.976 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_dir_1) 2024-04-17T10:28:30.987 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_1' 2024-04-17T10:28:30.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:31.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.174+0000 7f66dc400640 1 Processor -- start 2024-04-17T10:28:31.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.174+0000 7f66dc400640 1 -- start start 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66dc400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66d40cc250 0x7f66d40cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66dc400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40cb260 0x7f66d40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66dc400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d406b100 con 0x7f66d40ce2a0 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66dc400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d40cd150 con 0x7f66d40cc250 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66dc400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d406b3e0 con 0x7f66d40cb260 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66d40cc250 0x7f66d40cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40cb260 0x7f66d40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f66d40cc250 0x7f66d40cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:54454/0 (socket says 172.21.15.67:54454) 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40cb260 0x7f66d40cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:55982/0 (socket says 172.21.15.67:55982) 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9a00640 1 -- 172.21.15.67:0/2609309996 learned_addr learned my addr 172.21.15.67:0/2609309996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9a00640 1 -- 172.21.15.67:0/2609309996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40cb260 msgr2=0x7f66d40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:31.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9a00640 1 --2- 172.21.15.67:0/2609309996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40cb260 0x7f66d40cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9a00640 1 -- 172.21.15.67:0/2609309996 >> v1:172.21.15.86:6789/0 conn(0x7f66d40ce2a0 legacy=0x7f66d40cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d9a00640 1 -- 172.21.15.67:0/2609309996 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d406b6c0 con 0x7f66d40cc250 2024-04-17T10:28:31.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.178+0000 7f66d3e00640 1 -- 172.21.15.67:0/2609309996 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852290651 0 0) 0x7f66d406b100 con 0x7f66d40ce2a0 2024-04-17T10:28:31.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66d9a00640 1 --2- 172.21.15.67:0/2609309996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 0x7f66d40cad20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009870 tx=0x7f66c801c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b67c8ee67d26dcb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:31.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66d3e00640 1 -- 172.21.15.67:0/2609309996 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c80b2070 con 0x7f66d40cc250 2024-04-17T10:28:31.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66d3e00640 1 -- 172.21.15.67:0/2609309996 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c80a9070 con 0x7f66d40cc250 2024-04-17T10:28:31.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66d3e00640 1 -- 172.21.15.67:0/2609309996 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c80ad880 con 0x7f66d40cc250 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- 172.21.15.67:0/2609309996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 msgr2=0x7f66d40cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 --2- 172.21.15.67:0/2609309996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 0x7f66d40cad20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f66c8009870 tx=0x7f66c801c560 comp rx=0 tx=0).stop 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- 172.21.15.67:0/2609309996 shutdown_connections 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 --2- 172.21.15.67:0/2609309996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40cb260 0x7f66d40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 --2- 172.21.15.67:0/2609309996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 0x7f66d40cad20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- 172.21.15.67:0/2609309996 >> 172.21.15.67:0/2609309996 conn(0x7f66d40b22a0 msgr2=0x7f66d40b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- 172.21.15.67:0/2609309996 shutdown_connections 2024-04-17T10:28:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- 172.21.15.67:0/2609309996 wait complete. 2024-04-17T10:28:31.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 Processor -- start 2024-04-17T10:28:31.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- start start 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66d40cb260 0x7f66d40c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 0x7f66d4150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40ce2a0 0x7f66d41603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40cdeb0 con 0x7f66d40cb260 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40cdd30 con 0x7f66d40cc250 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66dc400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d416b100 con 0x7f66d40ce2a0 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66d40cb260 0x7f66d40c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66d40cb260 0x7f66d40c4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36000/0 (socket says 172.21.15.67:36000) 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.182+0000 7f66d9a00640 1 -- 172.21.15.67:0/2077657992 learned_addr learned my addr 172.21.15.67:0/2077657992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d9000640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 0x7f66d4150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66da400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40ce2a0 0x7f66d41603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d9a00640 1 -- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40ce2a0 msgr2=0x7f66d41603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d9a00640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40ce2a0 0x7f66d41603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d9a00640 1 -- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 msgr2=0x7f66d4150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d9a00640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 0x7f66d4150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d9a00640 1 -- 172.21.15.67:0/2077657992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d4160900 con 0x7f66d40cb260 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66da400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40ce2a0 0x7f66d41603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d9a00640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66d40cb260 0x7f66d40c4e00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f66c80b0040 tx=0x7f66c801ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d1600640 1 -- 172.21.15.67:0/2077657992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c80b2040 con 0x7f66d40cb260 2024-04-17T10:28:31.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d1600640 1 -- 172.21.15.67:0/2077657992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c80be070 con 0x7f66d40cb260 2024-04-17T10:28:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d1600640 1 -- 172.21.15.67:0/2077657992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c80b8660 con 0x7f66d40cb260 2024-04-17T10:28:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d4160bd0 con 0x7f66d40cb260 2024-04-17T10:28:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d415f010 con 0x7f66d40cb260 2024-04-17T10:28:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d406cd30 con 0x7f66d40cb260 2024-04-17T10:28:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.186+0000 7f66d1600640 1 -- 172.21.15.67:0/2077657992 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f66c80ab070 con 0x7f66d40cb260 2024-04-17T10:28:31.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.190+0000 7f66d1600640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f66b4066f60 0x7f66b4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.190+0000 7f66d1600640 1 -- 172.21.15.67:0/2077657992 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f66c811f3c0 con 0x7f66d40cb260 2024-04-17T10:28:31.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.190+0000 7f66d9000640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f66b4066f60 0x7f66b4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.190+0000 7f66d9000640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f66b4066f60 0x7f66b4069420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f66d415b5c0 tx=0x7f66c0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:31.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.194+0000 7f66d1600640 1 -- 172.21.15.67:0/2077657992 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f66c8016150 con 0x7f66d40cb260 2024-04-17T10:28:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66d409e900 con 0x7f66d40cb260 2024-04-17T10:28:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66d1600640 1 -- 172.21.15.67:0/2077657992 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+4936 (secure 0 0 0) 0x7f66c80f01a0 con 0x7f66d40cb260 2024-04-17T10:28:31.384 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:31.384 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:30.427162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 29 2024-04-17T10:28:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f66b4066f60 msgr2=0x7f66b4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f66b4066f60 0x7f66b4069420 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f66d415b5c0 tx=0x7f66c0009190 comp rx=0 tx=0).stop 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66d40cb260 msgr2=0x7f66d40c4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66d40cb260 0x7f66d40c4e00 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f66c80b0040 tx=0x7f66c801ada0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 shutdown_connections 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f66b4066f60 0x7f66b4069420 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66d40ce2a0 0x7f66d41603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66d40cc250 0x7f66d4150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 --2- 172.21.15.67:0/2077657992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66d40cb260 0x7f66d40c4e00 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 >> 172.21.15.67:0/2077657992 conn(0x7f66d40b22a0 msgr2=0x7f66d40ad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 shutdown_connections 2024-04-17T10:28:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.390+0000 7f66dc400640 1 -- 172.21.15.67:0/2077657992 wait complete. 2024-04-17T10:28:31.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/ceph-qa-suite/new_dir_1 2024-04-17T10:28:31.548 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:31.548 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:31.548 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:31.564 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_1' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:31.564 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:31.564 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:31.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:31.573 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_1 bs=1M conv=fdatasync count=1 seek=0) 2024-04-17T10:28:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:28:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:28:31.641 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182862 s, 57.3 MB/s 2024-04-17T10:28:31.642 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_1' 2024-04-17T10:28:31.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdba0400640 1 Processor -- start 2024-04-17T10:28:31.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdba0400640 1 -- start start 2024-04-17T10:28:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdba0400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb981446c0 0x7fdb98144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdba0400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdba0400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb981456d0 con 0x7fdb98144aa0 2024-04-17T10:28:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdba0400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb980b51a0 con 0x7fdb981446c0 2024-04-17T10:28:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdba0400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980b4870 con 0x7fdb98071840 2024-04-17T10:28:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdb981446c0 0x7fdb98144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9d000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdb98144aa0 0x7fdb980d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56154/0 (socket says 172.21.15.67:56154) 2024-04-17T10:28:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9d000640 1 -- 172.21.15.67:0/1382439310 learned_addr learned my addr 172.21.15.67:0/1382439310 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:31.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9e400640 1 --2- 172.21.15.67:0/1382439310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9da00640 1 -- 172.21.15.67:0/1382439310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 msgr2=0x7fdb980b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9da00640 1 --2- 172.21.15.67:0/1382439310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9da00640 1 -- 172.21.15.67:0/1382439310 >> v1:172.21.15.67:6789/0 conn(0x7fdb98144aa0 legacy=0x7fdb980d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb9da00640 1 -- 172.21.15.67:0/1382439310 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb980b4b50 con 0x7fdb981446c0 2024-04-17T10:28:31.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.834+0000 7fdb97e00640 1 -- 172.21.15.67:0/1382439310 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850693487 0 0) 0x7fdb981456d0 con 0x7fdb98144aa0 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdb9da00640 1 --2- 172.21.15.67:0/1382439310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb98144ea0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0097c0 tx=0x7fdb8c01c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf501a1d4ab7ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdb97e00640 1 -- 172.21.15.67:0/1382439310 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c0b2070 con 0x7fdb981446c0 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdb97e00640 1 -- 172.21.15.67:0/1382439310 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8c0ad680 con 0x7fdb981446c0 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdb97e00640 1 -- 172.21.15.67:0/1382439310 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8c0ada50 con 0x7fdb981446c0 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 -- 172.21.15.67:0/1382439310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 msgr2=0x7fdb98144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 --2- 172.21.15.67:0/1382439310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb98144ea0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0097c0 tx=0x7fdb8c01c470 comp rx=0 tx=0).stop 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 -- 172.21.15.67:0/1382439310 shutdown_connections 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 --2- 172.21.15.67:0/1382439310 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 --2- 172.21.15.67:0/1382439310 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb98144ea0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 -- 172.21.15.67:0/1382439310 >> 172.21.15.67:0/1382439310 conn(0x7fdb980b34d0 msgr2=0x7fdb980b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 -- 172.21.15.67:0/1382439310 shutdown_connections 2024-04-17T10:28:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 -- 172.21.15.67:0/1382439310 wait complete. 2024-04-17T10:28:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 Processor -- start 2024-04-17T10:28:31.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.838+0000 7fdba0400640 1 -- start start 2024-04-17T10:28:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb980c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb98144aa0 0x7fdb980c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980b6500 con 0x7fdb981446c0 2024-04-17T10:28:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980b6380 con 0x7fdb98144aa0 2024-04-17T10:28:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb980b6680 con 0x7fdb98071840 2024-04-17T10:28:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb980c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb980c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36026/0 (socket says 172.21.15.67:36026) 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9da00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 -- 172.21.15.67:0/1031616135 learned_addr learned my addr 172.21.15.67:0/1031616135 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9e400640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb98144aa0 0x7fdb980c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 -- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 msgr2=0x7fdb980cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 -- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb98144aa0 msgr2=0x7fdb980c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb98144aa0 0x7fdb980c6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 -- 172.21.15.67:0/1031616135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9814bd80 con 0x7fdb981446c0 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9da00640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb9d000640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb980c6740 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fdb84007fd0 tx=0x7fdb8400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb95600640 1 -- 172.21.15.67:0/1031616135 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84017070 con 0x7fdb981446c0 2024-04-17T10:28:31.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb95600640 1 -- 172.21.15.67:0/1031616135 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb84005d10 con 0x7fdb981446c0 2024-04-17T10:28:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdb95600640 1 -- 172.21.15.67:0/1031616135 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb84007310 con 0x7fdb981446c0 2024-04-17T10:28:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb9814c060 con 0x7fdb981446c0 2024-04-17T10:28:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb9814c620 con 0x7fdb981446c0 2024-04-17T10:28:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.842+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb98070bb0 con 0x7fdb981446c0 2024-04-17T10:28:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.846+0000 7fdb95600640 1 -- 172.21.15.67:0/1031616135 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdb8401a070 con 0x7fdb981446c0 2024-04-17T10:28:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.846+0000 7fdb95600640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdb78066f60 0x7fdb78069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.846+0000 7fdb95600640 1 -- 172.21.15.67:0/1031616135 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdb84084540 con 0x7fdb981446c0 2024-04-17T10:28:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.846+0000 7fdb9da00640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdb78066f60 0x7fdb78069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.846+0000 7fdb9da00640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdb78066f60 0x7fdb78069420 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0af000 tx=0x7fdb8c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:31.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:31.850+0000 7fdb95600640 1 -- 172.21.15.67:0/1031616135 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdb84054d60 con 0x7fdb981446c0 2024-04-17T10:28:32.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.046+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb981451c0 con 0x7fdb981446c0 2024-04-17T10:28:32.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.046+0000 7fdb95600640 1 -- 172.21.15.67:0/1031616135 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+4936 (secure 0 0 0) 0x7fdb84058e00 con 0x7fdb981446c0 2024-04-17T10:28:32.041 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:32.041 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":29,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:30.427162+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":10,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:32.041 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 29 2024-04-17T10:28:32.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdb78066f60 msgr2=0x7fdb78069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdb78066f60 0x7fdb78069420 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdb8c0af000 tx=0x7fdb8c0a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:32.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 msgr2=0x7fdb980c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb980c6740 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fdb84007fd0 tx=0x7fdb8400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 shutdown_connections 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdb78066f60 0x7fdb78069420 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdb98144aa0 0x7fdb980c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdb981446c0 0x7fdb980c6740 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 --2- 172.21.15.67:0/1031616135 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdb98071840 0x7fdb980cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 >> 172.21.15.67:0/1031616135 conn(0x7fdb980b34d0 msgr2=0x7fdb9806d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 shutdown_connections 2024-04-17T10:28:32.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.050+0000 7fdba0400640 1 -- 172.21.15.67:0/1031616135 wait complete. 2024-04-17T10:28:32.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path /scrub_test_path/ceph-qa-suite/new_file_1 2024-04-17T10:28:32.208 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:32.208 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:32.209 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:32.223 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_1' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:32.224 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-04-17T10:28:32.224 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:28:32.224 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-04-17T10:28:32.224 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:28:32.224 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_1").st_ino) 2024-04-17T10:28:32.224 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:28:32.290 INFO:teuthology.orchestra.run.smithi067.stdout:1099511630189 2024-04-17T10:28:32.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T10:28:32.299 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:32.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-04-17T10:28:32.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.490+0000 7f0499e00640 1 Processor -- start 2024-04-17T10:28:32.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.490+0000 7f0499e00640 1 -- start start 2024-04-17T10:28:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0499e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0499e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f049406e980 0x7f04940b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0499e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04940b4990 con 0x7f04940cce60 2024-04-17T10:28:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0499e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04940cd2c0 con 0x7f049406e980 2024-04-17T10:28:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0499e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04940af340 con 0x7f04940aef60 2024-04-17T10:28:32.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f048a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f04940cce60 0x7f049406e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56160/0 (socket says 172.21.15.67:56160) 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f048a000640 1 -- 172.21.15.67:0/1298156515 learned_addr learned my addr 172.21.15.67:0/1298156515 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0493400640 1 --2- 172.21.15.67:0/1298156515 >> v2:172.21.15.86:3300/0 conn(0x7f049406e980 0x7f04940b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0492a00640 1 --2- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0492000640 1 -- 172.21.15.67:0/1298156515 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 432772720 0 0) 0x7f047c001120 con 0x7f04940cce60 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0492000640 1 -- 172.21.15.67:0/1298156515 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0478003530 con 0x7f04940cce60 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0493400640 1 -- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 msgr2=0x7f04940cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0493400640 1 --2- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0493400640 1 -- 172.21.15.67:0/1298156515 >> v1:172.21.15.67:6789/0 conn(0x7f04940cce60 legacy=0x7f049406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0493400640 1 -- 172.21.15.67:0/1298156515 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049415ae90 con 0x7f049406e980 2024-04-17T10:28:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.494+0000 7f0492a00640 1 --2- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0493400640 1 --2- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f04940b3120 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f048c0097c0 tx=0x7f048c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb393678657428b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0492000640 1 -- 172.21.15.67:0/1298156515 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c0b0070 con 0x7f049406e980 2024-04-17T10:28:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0492000640 1 -- 172.21.15.67:0/1298156515 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c0ab600 con 0x7f049406e980 2024-04-17T10:28:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0492000640 1 -- 172.21.15.67:0/1298156515 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c0ab9d0 con 0x7f049406e980 2024-04-17T10:28:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 -- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 msgr2=0x7f04940b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 --2- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f04940b3120 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f048c0097c0 tx=0x7f048c01c410 comp rx=0 tx=0).stop 2024-04-17T10:28:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 -- 172.21.15.67:0/1298156515 shutdown_connections 2024-04-17T10:28:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 --2- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f04940b3120 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 --2- 172.21.15.67:0/1298156515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 -- 172.21.15.67:0/1298156515 >> 172.21.15.67:0/1298156515 conn(0x7f04940b1740 msgr2=0x7f04940b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 -- 172.21.15.67:0/1298156515 shutdown_connections 2024-04-17T10:28:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.498+0000 7f0499e00640 1 -- 172.21.15.67:0/1298156515 wait complete. 2024-04-17T10:28:32.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 Processor -- start 2024-04-17T10:28:32.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 -- start start 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f049415bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04940cce60 0x7f049413fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049415bc10 con 0x7f04940cce60 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04940b3e70 con 0x7f049406e980 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0499e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049415d4a0 con 0x7f04940aef60 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0493400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04940cce60 0x7f049413fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0493400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04940cce60 0x7f049413fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36042/0 (socket says 172.21.15.67:36042) 2024-04-17T10:28:32.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0493400640 1 -- 172.21.15.67:0/520817519 learned_addr learned my addr 172.21.15.67:0/520817519 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0492a00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f049415bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f048a000640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0492a00640 1 -- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 msgr2=0x7f04940b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0492a00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0492a00640 1 -- 172.21.15.67:0/520817519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04940cce60 msgr2=0x7f049413fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0492a00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04940cce60 0x7f049413fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0492a00640 1 -- 172.21.15.67:0/520817519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049415c890 con 0x7f049406e980 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0493400640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04940cce60 0x7f049413fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f0492a00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f049415bf20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f048400efc0 tx=0x7f048400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.502+0000 7f048be00640 1 -- 172.21.15.67:0/520817519 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484018020 con 0x7f049406e980 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.506+0000 7f048be00640 1 -- 172.21.15.67:0/520817519 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484014680 con 0x7f049406e980 2024-04-17T10:28:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.506+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049415cb70 con 0x7f049406e980 2024-04-17T10:28:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.506+0000 7f048be00640 1 -- 172.21.15.67:0/520817519 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484014a70 con 0x7f049406e980 2024-04-17T10:28:32.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.506+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049415d130 con 0x7f049406e980 2024-04-17T10:28:32.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.506+0000 7f048be00640 1 -- 172.21.15.67:0/520817519 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f048401b020 con 0x7f049406e980 2024-04-17T10:28:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.506+0000 7f048be00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0470066fb0 0x7f0470069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.506+0000 7f048be00640 1 -- 172.21.15.67:0/520817519 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0484085ef0 con 0x7f049406e980 2024-04-17T10:28:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0470066fb0 0x7f0470069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 Processor -- start 2024-04-17T10:28:32.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- start start 2024-04-17T10:28:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0470066fb0 0x7f0470069470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f049414c860 tx=0x7f047c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33c7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f337d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f31caf20 con 0x5615f33c7aa0 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f3153000 con 0x5615f33c7180 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615f31c6fc0 con 0x5615f337d600 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36050/0 (socket says 172.21.15.67:36050) 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 -- 172.21.15.67:0/2552576758 learned_addr learned my addr 172.21.15.67:0/2552576758 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f0492a00640 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f0493400640 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f337d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 -- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 msgr2=0x5615f337d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f337d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 -- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 msgr2=0x5615f33c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 -- 172.21.15.67:0/2552576758 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615f31a6ba0 con 0x5615f33c7aa0 2024-04-17T10:28:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f0493400640 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f337d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f048a000640 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337d0c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f047c00f960 tx=0x7f047c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d809f4858b2918f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f0483e00640 1 -- 172.21.15.67:0/2552576758 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c025070 con 0x5615f33c7aa0 2024-04-17T10:28:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f0483e00640 1 -- 172.21.15.67:0/2552576758 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c01b050 con 0x5615f33c7aa0 2024-04-17T10:28:32.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f0483e00640 1 -- 172.21.15.67:0/2552576758 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c014de0 con 0x5615f33c7aa0 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- 172.21.15.67:0/2552576758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 msgr2=0x5615f337d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337d0c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f047c00f960 tx=0x7f047c012000 comp rx=0 tx=0).stop 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- 172.21.15.67:0/2552576758 shutdown_connections 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f337d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337d0c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 --2- 172.21.15.67:0/2552576758 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33c7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- 172.21.15.67:0/2552576758 >> 172.21.15.67:0/2552576758 conn(0x5615f3383760 msgr2=0x5615f337b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- 172.21.15.67:0/2552576758 shutdown_connections 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.510+0000 7f049d370000 1 -- 172.21.15.67:0/2552576758 wait complete. 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 -- start start 2024-04-17T10:28:32.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 Processor -- start 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f337d600 0x5615f33826d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3382c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f33c7aa0 0x5615f33b5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f32889c0 con 0x5615f337d600 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f31855d0 con 0x5615f33c7180 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615f31c7f90 con 0x5615f33c7aa0 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 10 client.0 _ll_register_callbacks cb 0x5615f3368a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f337d600 0x5615f33826d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f337d600 0x5615f33826d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36058/0 (socket says 172.21.15.67:36058) 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 -- 172.21.15.67:0/311240636 learned_addr learned my addr 172.21.15.67:0/311240636 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f048a000640 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3382c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0493400640 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f33c7aa0 0x5615f33b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 -- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f33c7aa0 msgr2=0x5615f33b5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f33c7aa0 0x5615f33b5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 -- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 msgr2=0x5615f3382c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3382c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 -- 172.21.15.67:0/311240636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5615f32115a0 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0492a00640 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f337d600 0x5615f33826d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0484084270 tx=0x7f0484056e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdc1bf84ff1b491 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0480c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0480c00640 1 -- 172.21.15.67:0/311240636 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0484082060 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0480c00640 1 -- 172.21.15.67:0/311240636 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f048407dbd0 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f047c01b050 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0480c00640 1 -- 172.21.15.67:0/311240636 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f048406e070 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 10 client.6333 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x5615f31a22b0 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0480c00640 1 -- 172.21.15.67:0/311240636 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c01b050 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0480c00640 1 -- 172.21.15.67:0/311240636 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0484055aa0 con 0x5615f337d600 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 10 client.6333 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:32.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 10 client.6333 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 20 client.6333 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33b5e60 0x5615f33d9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 4 client.6333 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5615f3166c70 con 0x5615f33b5e60 2024-04-17T10:28:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f048a000640 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33b5e60 0x5615f33d9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f048a000640 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33b5e60 0x5615f33d9440 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c267bd6f47efe6fc server_cookie=388dcfb6ea496d35 in_seq=0 out_seq=0 2024-04-17T10:28:32.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.514+0000 7f0480c00640 10 client.6333 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f0480c00640 1 -- 172.21.15.67:0/311240636 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5615f3166c70 con 0x5615f33b5e60 2024-04-17T10:28:32.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f0480c00640 10 client.6333 handle_command_reply: tid=0 2024-04-17T10:28:32.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 client.6333 shutdown 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f337d600 msgr2=0x5615f33826d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f337d600 0x5615f33826d0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f0484084270 tx=0x7f0484056e90 comp rx=0 tx=0).stop 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 shutdown_connections 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33b5e60 0x5615f33d9440 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f33c7aa0 0x5615f33b5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3382c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f0480c00640 0 client.6333 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 --2- 172.21.15.67:0/311240636 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f337d600 0x5615f33826d0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 >> 172.21.15.67:0/311240636 conn(0x5615f3383760 msgr2=0x5615f337c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 shutdown_connections 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 1 -- 172.21.15.67:0/311240636 wait complete. 2024-04-17T10:28:32.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.518+0000 7f049d370000 20 client.6333 trim_cache size 0 max 16384 2024-04-17T10:28:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 Processor -- start 2024-04-17T10:28:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 -- start start 2024-04-17T10:28:32.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f33d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33d4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494203290 con 0x5615f33c7aa0 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494202470 con 0x5615f33c7180 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04941f7190 con 0x5615f337d600 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36066/0 (socket says 172.21.15.67:36066) 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 -- 172.21.15.67:0/3293716830 learned_addr learned my addr 172.21.15.67:0/3293716830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0492a00640 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f33d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f048a000640 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33d4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 -- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 msgr2=0x5615f33d45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f33d45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 -- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 msgr2=0x5615f33d4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33d4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 -- 172.21.15.67:0/3293716830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615f3153250 con 0x5615f33c7aa0 2024-04-17T10:28:32.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f048a000640 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33d4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:32.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0492a00640 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f33d45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:32.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0493400640 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337c120 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f048c097dd0 tx=0x7f048c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbd6c5fa336737c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0483400640 1 -- 172.21.15.67:0/3293716830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c0b0040 con 0x5615f33c7aa0 2024-04-17T10:28:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0483400640 1 -- 172.21.15.67:0/3293716830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c0ab740 con 0x5615f33c7aa0 2024-04-17T10:28:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f0483400640 1 -- 172.21.15.67:0/3293716830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c0abaf0 con 0x5615f33c7aa0 2024-04-17T10:28:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 -- 172.21.15.67:0/3293716830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 msgr2=0x5615f337c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.530+0000 7f049d370000 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337c120 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f048c097dd0 tx=0x7f048c004740 comp rx=0 tx=0).stop 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- 172.21.15.67:0/3293716830 shutdown_connections 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f337c120 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f33d4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 --2- 172.21.15.67:0/3293716830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f33d45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- 172.21.15.67:0/3293716830 >> 172.21.15.67:0/3293716830 conn(0x5615f3383760 msgr2=0x5615f33d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- 172.21.15.67:0/3293716830 shutdown_connections 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- 172.21.15.67:0/3293716830 wait complete. 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- start start 2024-04-17T10:28:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 Processor -- start 2024-04-17T10:28:32.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f34277c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3427d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f33d9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615f3181dc0 con 0x5615f33c7aa0 2024-04-17T10:28:32.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494201ea0 con 0x5615f33c7180 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615f31762f0 con 0x5615f337d600 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 10 client.0 _ll_register_callbacks cb 0x5615f3349600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f33d9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f33d9750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36076/0 (socket says 172.21.15.67:36076) 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0492a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f34277c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 -- 172.21.15.67:0/4236725334 learned_addr learned my addr 172.21.15.67:0/4236725334 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f048a000640 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3427d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 -- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 msgr2=0x5615f34277c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f34277c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 -- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 msgr2=0x5615f3427d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3427d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 -- 172.21.15.67:0/4236725334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5615f315fb90 con 0x5615f33c7aa0 2024-04-17T10:28:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0493400640 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f33d9750 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f048c097dd0 tx=0x7f048c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=477d72425bbc1d01 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0480c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0480c00640 1 -- 172.21.15.67:0/4236725334 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c0ca060 con 0x5615f33c7aa0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0480c00640 1 -- 172.21.15.67:0/4236725334 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f048c0ab540 con 0x5615f33c7aa0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f048c0ab740 con 0x5615f33c7aa0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0480c00640 1 -- 172.21.15.67:0/4236725334 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f048c0d0050 con 0x5615f33c7aa0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 10 client.6339 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x5615f3172010 con 0x5615f33c7aa0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0480c00640 1 -- 172.21.15.67:0/4236725334 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c0ab740 con 0x5615f33c7aa0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f0480c00640 1 -- 172.21.15.67:0/4236725334 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f048c0d3020 con 0x5615f33c7aa0 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 10 client.6339 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 10 client.6339 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 20 client.6339 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33da4b0 0x5615f33da8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 4 client.6339 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.534+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5615f31d8460 con 0x5615f33da4b0 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0492a00640 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33da4b0 0x5615f33da8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0492a00640 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33da4b0 0x5615f33da8c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38acf40be2e16ce7 server_cookie=1f0835deabf110f2 in_seq=0 out_seq=0 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0480c00640 10 client.6339 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0480c00640 1 -- 172.21.15.67:0/4236725334 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5615f31d8460 con 0x5615f33da4b0 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0480c00640 10 client.6339 handle_command_reply: tid=0 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 client.6339 shutdown 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 msgr2=0x5615f33d9750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f33d9750 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f048c097dd0 tx=0x7f048c01ac50 comp rx=0 tx=0).stop 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 shutdown_connections 2024-04-17T10:28:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x5615f33da4b0 0x5615f33da8c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5615f33c7aa0 0x5615f33d9750 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5615f33c7180 0x5615f3427d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 --2- 172.21.15.67:0/4236725334 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5615f337d600 0x5615f34277c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0480c00640 0 client.6339 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 >> 172.21.15.67:0/4236725334 conn(0x5615f3383760 msgr2=0x5615f33c69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 shutdown_connections 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 1 -- 172.21.15.67:0/4236725334 wait complete. 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f049d370000 20 client.6339 trim_cache size 0 max 16384 2024-04-17T10:28:32.535 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2024-04-17T10:28:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0470066fb0 msgr2=0x7f0470069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0470066fb0 0x7f0470069470 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f049414c860 tx=0x7f047c009040 comp rx=0 tx=0).stop 2024-04-17T10:28:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 msgr2=0x7f049415bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f049415bf20 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f048400efc0 tx=0x7f048400c490 comp rx=0 tx=0).stop 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 shutdown_connections 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0470066fb0 0x7f0470069470 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04940cce60 0x7f049413fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04940aef60 0x7f04940b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 --2- 172.21.15.67:0/520817519 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f049406e980 0x7f049415bf20 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 >> 172.21.15.67:0/520817519 conn(0x7f04940b1740 msgr2=0x7f0494150940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 shutdown_connections 2024-04-17T10:28:32.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.538+0000 7f0499e00640 1 -- 172.21.15.67:0/520817519 wait complete. 2024-04-17T10:28:32.550 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data rmxattr 1000000096d.00000000 parent 2024-04-17T10:28:32.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 Processor -- start 2024-04-17T10:28:32.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 -- start start 2024-04-17T10:28:32.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9630 0x562069aeb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9a10 0x562069b9adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069af4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562069af8c50 con 0x562069af9a10 2024-04-17T10:28:32.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x562069a6c9f0 con 0x562069af9630 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.602+0000 7f4d0aa2e080 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x562069af8ef0 con 0x562069b9d5b0 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9a10 0x562069b9adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9a10 0x562069b9adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36096/0 (socket says 172.21.15.67:36096) 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 -- 172.21.15.67:0/3518505166 learned_addr learned my addr 172.21.15.67:0/3518505166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06c00640 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9630 0x562069aeb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d07600640 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069af4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 -- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 msgr2=0x562069af4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069af4ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 -- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9630 msgr2=0x562069aeb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9630 0x562069aeb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 -- 172.21.15.67:0/3518505166 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562069b0c040 con 0x562069af9a10 2024-04-17T10:28:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d07600640 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069af4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9a10 0x562069b9adb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4009a40 tx=0x7f4cf4005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3629d6fc111bd3b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d04e00640 1 -- 172.21.15.67:0/3518505166 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4008e60 con 0x562069af9a10 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d04e00640 1 -- 172.21.15.67:0/3518505166 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf4018070 con 0x562069af9a10 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d04e00640 1 -- 172.21.15.67:0/3518505166 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf4013620 con 0x562069af9a10 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3518505166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9a10 msgr2=0x562069b9adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9a10 0x562069b9adb0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4009a40 tx=0x7f4cf4005a10 comp rx=0 tx=0).stop 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3518505166 shutdown_connections 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069af4ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9a10 0x562069b9adb0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3518505166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9630 0x562069aeb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3518505166 >> 172.21.15.67:0/3518505166 conn(0x562069af3e60 msgr2=0x562069ac7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3518505166 shutdown_connections 2024-04-17T10:28:32.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3518505166 wait complete. 2024-04-17T10:28:32.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 Processor -- start 2024-04-17T10:28:32.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- start start 2024-04-17T10:28:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9630 0x562069b99730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9a10 0x562069b9c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069b9c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x562069b0a410 con 0x562069af9630 2024-04-17T10:28:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x562069b9b2f0 con 0x562069af9a10 2024-04-17T10:28:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d0aa2e080 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x562069b9bfd0 con 0x562069b9d5b0 2024-04-17T10:28:32.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9630 0x562069b99730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9630 0x562069b99730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36108/0 (socket says 172.21.15.67:36108) 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06c00640 1 -- 172.21.15.67:0/3244061208 learned_addr learned my addr 172.21.15.67:0/3244061208 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d06200640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9a10 0x562069b9c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.606+0000 7f4d07600640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069b9c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d06c00640 1 -- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 msgr2=0x562069b9c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d06c00640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x562069b9d5b0 0x562069b9c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d06c00640 1 -- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9a10 msgr2=0x562069b9c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d06c00640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x562069af9a10 0x562069b9c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d06c00640 1 -- 172.21.15.67:0/3244061208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562069b98130 con 0x562069af9630 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d06c00640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9630 0x562069b99730 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8000c10 tx=0x7f4cf80180c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4cff400640 1 -- 172.21.15.67:0/3244061208 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80b9070 con 0x562069af9630 2024-04-17T10:28:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4cff400640 1 -- 172.21.15.67:0/3244061208 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf80aabc0 con 0x562069af9630 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4cff400640 1 -- 172.21.15.67:0/3244061208 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80af040 con 0x562069af9630 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x562069b983d0 con 0x562069af9630 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562069b98910 con 0x562069af9630 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4cff400640 1 -- 172.21.15.67:0/3244061208 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4cf80bc070 con 0x562069af9630 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4cff400640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ce8065940 0x7f4ce8067e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4cff400640 1 -- 172.21.15.67:0/3244061208 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4cf811ee20 con 0x562069af9630 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x562069c37ea0 0x562069b970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d06200640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ce8065940 0x7f4ce8067e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.610+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 --> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] -- osd_op(unknown.0.0:1 8.21 8:844f785e:::1000000096d.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) v9 -- 0x562069b975f0 con 0x562069c37ea0 2024-04-17T10:28:32.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d07600640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x562069c37ea0 0x562069b970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d06200640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ce8065940 0x7f4ce8067e00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4015040 tx=0x7f4cf4012000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d07600640 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x562069c37ea0 0x562069b970b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d07600640 1 -- 172.21.15.67:0/3244061208 <== osd.0 v2:172.21.15.67:6800/3376065543 1 ==== osd_op_reply(1 1000000096d.00000000 [rmxattr] v54'3 uv3 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4d00001930 con 0x562069c37ea0 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x562069c37ea0 msgr2=0x562069b970b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:6800/3376065543,v1:172.21.15.67:6801/3376065543] conn(0x562069c37ea0 0x562069b970b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ce8065940 msgr2=0x7f4ce8067e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4ce8065940 0x7f4ce8067e00 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4015040 tx=0x7f4cf4012000 comp rx=0 tx=0).stop 2024-04-17T10:28:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9630 msgr2=0x562069b99730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 --2- 172.21.15.67:0/3244061208 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x562069af9630 0x562069b99730 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8000c10 tx=0x7f4cf80180c0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d07600640 1 -- 172.21.15.67:0/3244061208 reap_dead start 2024-04-17T10:28:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 shutdown_connections 2024-04-17T10:28:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 >> 172.21.15.67:0/3244061208 conn(0x562069af3e60 msgr2=0x562069b9b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 shutdown_connections 2024-04-17T10:28:32.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.614+0000 7f4d0aa2e080 1 -- 172.21.15.67:0/3244061208 wait complete. 2024-04-17T10:28:32.613 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_1' 2024-04-17T10:28:32.613 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:32.613 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/new_file_1 2024-04-17T10:28:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.806+0000 7fc3ba200640 1 Processor -- start 2024-04-17T10:28:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.806+0000 7fc3ba200640 1 -- start start 2024-04-17T10:28:32.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3ba200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b40b1b90 0x7fc3b40b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3ba200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b40b1fd0 con 0x7fc3b406b3a0 2024-04-17T10:28:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3ba200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b41616a0 con 0x7fc3b41474c0 2024-04-17T10:28:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3ba200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b40b3e60 con 0x7fc3b40b1b90 2024-04-17T10:28:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3aaa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3b406b3a0 0x7fc3b40d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56182/0 (socket says 172.21.15.67:56182) 2024-04-17T10:28:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3aaa00640 1 -- 172.21.15.67:0/840565435 learned_addr learned my addr 172.21.15.67:0/840565435 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b3e00640 1 --2- 172.21.15.67:0/840565435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b40b1b90 0x7fc3b40b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b2a00640 1 -- 172.21.15.67:0/840565435 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634473003 0 0) 0x7fc39c001120 con 0x7fc3b406b3a0 2024-04-17T10:28:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b2a00640 1 -- 172.21.15.67:0/840565435 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc398003530 con 0x7fc3b406b3a0 2024-04-17T10:28:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b2a00640 1 -- 172.21.15.67:0/840565435 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471298629 0 0) 0x7fc3b41616a0 con 0x7fc3b41474c0 2024-04-17T10:28:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b2a00640 1 -- 172.21.15.67:0/840565435 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39c001120 con 0x7fc3b41474c0 2024-04-17T10:28:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b3e00640 1 -- 172.21.15.67:0/840565435 >> v1:172.21.15.86:6789/0 conn(0x7fc3b41474c0 legacy=0x7fc3b406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b3e00640 1 -- 172.21.15.67:0/840565435 >> v1:172.21.15.67:6789/0 conn(0x7fc3b406b3a0 legacy=0x7fc3b40d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.810+0000 7fc3b3e00640 1 -- 172.21.15.67:0/840565435 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b4161940 con 0x7fc3b40b1b90 2024-04-17T10:28:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3b3e00640 1 --2- 172.21.15.67:0/840565435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b40b1b90 0x7fc3b40b3920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009dc0 tx=0x7fc3ac01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f74fc94f512aa95 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3b2a00640 1 -- 172.21.15.67:0/840565435 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0b4070 con 0x7fc3b40b1b90 2024-04-17T10:28:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3b2a00640 1 -- 172.21.15.67:0/840565435 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac0af600 con 0x7fc3b40b1b90 2024-04-17T10:28:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3b2a00640 1 -- 172.21.15.67:0/840565435 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0af9d0 con 0x7fc3b40b1b90 2024-04-17T10:28:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- 172.21.15.67:0/840565435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b40b1b90 msgr2=0x7fc3b40b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 --2- 172.21.15.67:0/840565435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b40b1b90 0x7fc3b40b3920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009dc0 tx=0x7fc3ac01f170 comp rx=0 tx=0).stop 2024-04-17T10:28:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- 172.21.15.67:0/840565435 shutdown_connections 2024-04-17T10:28:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 --2- 172.21.15.67:0/840565435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b40b1b90 0x7fc3b40b3920 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- 172.21.15.67:0/840565435 >> 172.21.15.67:0/840565435 conn(0x7fc3b40b5ef0 msgr2=0x7fc3b4099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- 172.21.15.67:0/840565435 shutdown_connections 2024-04-17T10:28:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- 172.21.15.67:0/840565435 wait complete. 2024-04-17T10:28:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 Processor -- start 2024-04-17T10:28:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- start start 2024-04-17T10:28:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b406b3a0 0x7fc3b40cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b40b1b90 0x7fc3b415d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b41474c0 0x7fc3b40cf930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b40b2250 con 0x7fc3b40b1b90 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b4161f00 con 0x7fc3b41474c0 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.814+0000 7fc3ba200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b4162420 con 0x7fc3b406b3a0 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b40b1b90 0x7fc3b415d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b40b1b90 0x7fc3b415d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36110/0 (socket says 172.21.15.67:36110) 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 -- 172.21.15.67:0/2302891231 learned_addr learned my addr 172.21.15.67:0/2302891231 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3b3e00640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b41474c0 0x7fc3b40cf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3b3400640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b406b3a0 0x7fc3b40cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 -- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b406b3a0 msgr2=0x7fc3b40cab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b406b3a0 0x7fc3b40cab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 -- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b41474c0 msgr2=0x7fc3b40cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b41474c0 0x7fc3b40cf930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 -- 172.21.15.67:0/2302891231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b40d13e0 con 0x7fc3b40b1b90 2024-04-17T10:28:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3b3e00640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b41474c0 0x7fc3b40cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b40b1b90 0x7fc3b415d4c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00c700 tx=0x7fc39c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3abe00640 1 -- 172.21.15.67:0/2302891231 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c017070 con 0x7fc3b40b1b90 2024-04-17T10:28:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3abe00640 1 -- 172.21.15.67:0/2302891231 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c005cf0 con 0x7fc3b40b1b90 2024-04-17T10:28:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3abe00640 1 -- 172.21.15.67:0/2302891231 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c007290 con 0x7fc3b40b1b90 2024-04-17T10:28:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b40cec60 con 0x7fc3b40b1b90 2024-04-17T10:28:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b40cef00 con 0x7fc3b40b1b90 2024-04-17T10:28:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.818+0000 7fc3abe00640 1 -- 172.21.15.67:0/2302891231 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc39c01a070 con 0x7fc3b40b1b90 2024-04-17T10:28:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3abe00640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc390066f90 0x7fc390069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3abe00640 1 -- 172.21.15.67:0/2302891231 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc39c084820 con 0x7fc3b40b1b90 2024-04-17T10:28:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3b3400640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc390066f90 0x7fc390069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 Processor -- start 2024-04-17T10:28:32.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- start start 2024-04-17T10:28:32.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3b3400640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc390066f90 0x7fc390069450 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40011f0 tx=0x7fc3a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408a1f0 0x555a3408a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408ab10 0x555a3403d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 0x555a3403dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a33e8dce0 con 0x555a3408ab10 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a33e16530 con 0x555a3408a1f0 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a33e89d80 con 0x555a3403d920 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408ab10 0x555a3403d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408ab10 0x555a3403d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36118/0 (socket says 172.21.15.67:36118) 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 -- 172.21.15.67:0/4294604877 learned_addr learned my addr 172.21.15.67:0/4294604877 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3b3400640 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408a1f0 0x555a3408a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3b3e00640 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 0x555a3403dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 -- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 msgr2=0x555a3403dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 0x555a3403dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 -- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408a1f0 msgr2=0x555a3408a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408a1f0 0x555a3408a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 -- 172.21.15.67:0/4294604877 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a33e88cd0 con 0x555a3408ab10 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3b3400640 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408a1f0 0x555a3408a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408ab10 0x555a3403d3e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc39c051360 tx=0x7fc39c054f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9815e09fa84735cc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3a3e00640 1 -- 172.21.15.67:0/4294604877 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c00ce10 con 0x555a3408ab10 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3a3e00640 1 -- 172.21.15.67:0/4294604877 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c053200 con 0x555a3408ab10 2024-04-17T10:28:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3a3e00640 1 -- 172.21.15.67:0/4294604877 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c05d180 con 0x555a3408ab10 2024-04-17T10:28:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/4294604877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408ab10 msgr2=0x555a3403d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408ab10 0x555a3403d3e0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fc39c051360 tx=0x7fc39c054f00 comp rx=0 tx=0).stop 2024-04-17T10:28:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/4294604877 shutdown_connections 2024-04-17T10:28:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 0x555a3403dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408ab10 0x555a3403d3e0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/4294604877 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408a1f0 0x555a3408a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/4294604877 >> 172.21.15.67:0/4294604877 conn(0x555a3406af90 msgr2=0x555a3406d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/4294604877 shutdown_connections 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.822+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/4294604877 wait complete. 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 -- start start 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 Processor -- start 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a34091760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a34091ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a34043a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a33e48bc0 con 0x555a3408a1f0 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a33e348e0 con 0x555a3403d920 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a33e5a0b0 con 0x555a3408ab10 2024-04-17T10:28:32.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 10 client.0 _ll_register_callbacks cb 0x555a3402b4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a34091ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a34091ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36126/0 (socket says 172.21.15.67:36126) 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3b3e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a34043a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3b3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a34091760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 -- 172.21.15.67:0/1591507587 learned_addr learned my addr 172.21.15.67:0/1591507587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 -- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 msgr2=0x555a34043a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a34043a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 -- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 msgr2=0x555a34091760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a34091760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 -- 172.21.15.67:0/1591507587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a33e9b950 con 0x555a3408a1f0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a34091ca0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc39c005fd0 tx=0x7fc39c000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7c5726068bffa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3a0c00640 1 -- 172.21.15.67:0/1591507587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c071060 con 0x555a3408a1f0 2024-04-17T10:28:32.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3a0c00640 1 -- 172.21.15.67:0/1591507587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc39c05b070 con 0x555a3408a1f0 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc39c053200 con 0x555a3408a1f0 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3a0c00640 1 -- 172.21.15.67:0/1591507587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c05b270 con 0x555a3408a1f0 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 10 client.6363 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x555a33f0a790 con 0x555a3408a1f0 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3a0c00640 1 -- 172.21.15.67:0/1591507587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c053200 con 0x555a3408a1f0 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3a0c00640 1 -- 172.21.15.67:0/1591507587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc39c054350 con 0x555a3408a1f0 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 10 client.6363 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 10 client.6363 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 20 client.6363 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:32.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a340447d0 0x555a34044bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 4 client.6363 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:32.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a33e29b80 con 0x555a340447d0 2024-04-17T10:28:32.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.826+0000 7fc3b3400640 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a340447d0 0x555a34044bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3b3400640 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a340447d0 0x555a34044bb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9118ba9318a187de server_cookie=7c5f047c12c4c94f in_seq=0 out_seq=0 2024-04-17T10:28:32.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3a0c00640 10 client.6363 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3a0c00640 1 -- 172.21.15.67:0/1591507587 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555a33e29b80 con 0x555a340447d0 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3a0c00640 10 client.6363 handle_command_reply: tid=0 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 client.6363 shutdown 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 msgr2=0x555a34091ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a34091ca0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc39c005fd0 tx=0x7fc39c000ee0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 shutdown_connections 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a340447d0 0x555a34044bb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a34043a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a34091ca0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/1591507587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a34091760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3a0c00640 0 client.6363 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 >> 172.21.15.67:0/1591507587 conn(0x555a3406af90 msgr2=0x555a3403cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 shutdown_connections 2024-04-17T10:28:32.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/1591507587 wait complete. 2024-04-17T10:28:32.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.830+0000 7fc3bd5d4000 20 client.6363 trim_cache size 0 max 16384 2024-04-17T10:28:32.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 Processor -- start 2024-04-17T10:28:32.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- start start 2024-04-17T10:28:32.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 0x555a340898e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a3403bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408ab10 0x555a3403c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b42003e0 con 0x555a3408a1f0 2024-04-17T10:28:32.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b41ff5c0 con 0x555a3408ab10 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b40aeb60 con 0x555a3403d920 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a3403bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a3403bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36138/0 (socket says 172.21.15.67:36138) 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 -- 172.21.15.67:0/3687004028 learned_addr learned my addr 172.21.15.67:0/3687004028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 -- 172.21.15.67:0/3687004028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 msgr2=0x555a340898e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/3687004028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 0x555a340898e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 -- 172.21.15.67:0/3687004028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408ab10 msgr2=0x555a3403c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/3687004028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408ab10 0x555a3403c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 -- 172.21.15.67:0/3687004028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a33e536f0 con 0x555a3408a1f0 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/3687004028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a3403bb40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc39c054c00 tx=0x7fc39c07e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b26bcababf37e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3a3400640 1 -- 172.21.15.67:0/3687004028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c05d390 con 0x555a3408a1f0 2024-04-17T10:28:32.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3a3400640 1 -- 172.21.15.67:0/3687004028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c080070 con 0x555a3408a1f0 2024-04-17T10:28:32.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3a3400640 1 -- 172.21.15.67:0/3687004028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c054350 con 0x555a3408a1f0 2024-04-17T10:28:32.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/3687004028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 msgr2=0x555a3403bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/3687004028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a3403bb40 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc39c054c00 tx=0x7fc39c07e070 comp rx=0 tx=0).stop 2024-04-17T10:28:32.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/3687004028 shutdown_connections 2024-04-17T10:28:32.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/3687004028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3408ab10 0x555a3403c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/3687004028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a3403bb40 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/3687004028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3403d920 0x555a340898e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/3687004028 >> 172.21.15.67:0/3687004028 conn(0x555a3406af90 msgr2=0x555a34097790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/3687004028 shutdown_connections 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/3687004028 wait complete. 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 -- start start 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.846+0000 7fc3bd5d4000 1 Processor -- start 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a340f3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a340f3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a3408cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a33e92310 con 0x555a3408a1f0 2024-04-17T10:28:32.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b41feff0 con 0x555a3403d920 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a33e348e0 con 0x555a3408ab10 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 10 client.0 _ll_register_callbacks cb 0x7fc39c005f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a340f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a340f3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36148/0 (socket says 172.21.15.67:36148) 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 -- 172.21.15.67:0/246252 learned_addr learned my addr 172.21.15.67:0/246252 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3b3400640 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a340f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3b3e00640 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a3408cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 -- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 msgr2=0x555a3408cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a3408cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 -- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 msgr2=0x555a340f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a340f3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 -- 172.21.15.67:0/246252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a33e49c80 con 0x555a3408a1f0 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3aaa00640 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a340f3790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc39c000f70 tx=0x7fc39c051220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6bba9cada25c4e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:32.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 1 -- 172.21.15.67:0/246252 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c054350 con 0x555a3408a1f0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 1 -- 172.21.15.67:0/246252 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc39c007430 con 0x555a3408a1f0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc39c080070 con 0x555a3408a1f0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 1 -- 172.21.15.67:0/246252 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c007630 con 0x555a3408a1f0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 10 client.6369 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x555a33b4a600 con 0x555a3408a1f0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 1 -- 172.21.15.67:0/246252 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c080070 con 0x555a3408a1f0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 1 -- 172.21.15.67:0/246252 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc39c050c50 con 0x555a3408a1f0 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 10 client.6369 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 10 client.6369 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 20 client.6369 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a3408d8f0 0x555a3408dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 4 client.6369 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_1"}] 2024-04-17T10:28:32.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_1"}) v1 -- 0x555a3406ade0 con 0x555a3408d8f0 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3b3400640 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a3408d8f0 0x555a3408dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3b3400640 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a3408d8f0 0x555a3408dcd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f821c89b0394d0d server_cookie=b852decc2ff9d6e0 in_seq=0 out_seq=0 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 10 client.6369 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 1 -- 172.21.15.67:0/246252 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555a3406ade0 con 0x555a3408d8f0 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3a0c00640 10 client.6369 handle_command_reply: tid=0 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.850+0000 7fc3bd5d4000 1 client.6369 shutdown 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 msgr2=0x555a340f3790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a340f3790 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fc39c000f70 tx=0x7fc39c051220 comp rx=0 tx=0).stop 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 shutdown_connections 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x555a3408d8f0 0x555a3408dcd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555a3408ab10 0x555a3408cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555a3408a1f0 0x555a340f3790 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 --2- 172.21.15.67:0/246252 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555a3403d920 0x555a340f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3a0c00640 0 client.6369 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 >> 172.21.15.67:0/246252 conn(0x555a3406af90 msgr2=0x555a34044bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 shutdown_connections 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 1 -- 172.21.15.67:0/246252 wait complete. 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3bd5d4000 20 client.6369 trim_cache size 0 max 16384 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "71bb7d24-e443-42d0-b6a0-ecfc71d44328", 2024-04-17T10:28:32.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:28:32.852 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc390066f90 msgr2=0x7fc390069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc390066f90 0x7fc390069450 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc3a40011f0 tx=0x7fc3a4009040 comp rx=0 tx=0).stop 2024-04-17T10:28:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b40b1b90 msgr2=0x7fc3b415d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:32.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b40b1b90 0x7fc3b415d4c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc39c00c700 tx=0x7fc39c00cbc0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 shutdown_connections 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc390066f90 0x7fc390069450 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b41474c0 0x7fc3b40cf930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b40b1b90 0x7fc3b415d4c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 --2- 172.21.15.67:0/2302891231 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b406b3a0 0x7fc3b40cab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 >> 172.21.15.67:0/2302891231 conn(0x7fc3b40b5ef0 msgr2=0x7fc3b4142610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 shutdown_connections 2024-04-17T10:28:32.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:32.854+0000 7fc3ba200640 1 -- 172.21.15.67:0/2302891231 wait complete. 2024-04-17T10:28:32.854 INFO:tasks.ceph.mds.d.smithi086.stderr:2024-04-17T10:28:32.847+0000 7f6880000640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000096d [2,head] /scrub_test_path/ceph-qa-suite/new_file_1 auth v649 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5958=0-4194304@1} caps={5958=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@3},l=5958 | request=0 lock=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x561ab6816b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x1000000096d:[<0x10000000001/new_file_1 v649>,<0x10000000000/ceph-qa-suite v70>,<0x1/scrub_test_path v23>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":-61} 2024-04-17T10:28:32.866 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_1' returned '{'return_code': 0, 'scrub_tag': '71bb7d24-e443-42d0-b6a0-ecfc71d44328', 'mode': 'asynchronous'}' 2024-04-17T10:28:32.866 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-17T10:28:32.867 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-17T10:28:33.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 Processor -- start 2024-04-17T10:28:33.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 -- start start 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37e4159440 0x7f37e40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e40713e0 0x7f37e4071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37e415be70 0x7f37e415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e40a8650 con 0x7f37e4159440 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37e4071d80 con 0x7f37e415be70 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ec800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e415ce10 con 0x7f37e40713e0 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37e4159440 0x7f37e40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37e4159440 0x7f37e40ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36152/0 (socket says 172.21.15.67:36152) 2024-04-17T10:28:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 -- 172.21.15.67:0/557881562 learned_addr learned my addr 172.21.15.67:0/557881562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37ea800640 1 --2- 172.21.15.67:0/557881562 >> v2:172.21.15.86:3300/0 conn(0x7f37e415be70 0x7f37e415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9400640 1 --2- 172.21.15.67:0/557881562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e40713e0 0x7f37e4071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 -- 172.21.15.67:0/557881562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e40713e0 msgr2=0x7f37e4071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 --2- 172.21.15.67:0/557881562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e40713e0 0x7f37e4071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 -- 172.21.15.67:0/557881562 >> v2:172.21.15.86:3300/0 conn(0x7f37e415be70 msgr2=0x7f37e415c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 --2- 172.21.15.67:0/557881562 >> v2:172.21.15.86:3300/0 conn(0x7f37e415be70 0x7f37e415c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9e00640 1 -- 172.21.15.67:0/557881562 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e415cb30 con 0x7f37e4159440 2024-04-17T10:28:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.018+0000 7f37e9400640 1 --2- 172.21.15.67:0/557881562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e40713e0 0x7f37e4071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:33.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37e9e00640 1 --2- 172.21.15.67:0/557881562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e4159440 0x7f37e40ce0f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f37e40726f0 tx=0x7f37d801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dda65f6834f4f16 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37e8a00640 1 -- 172.21.15.67:0/557881562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d80a6040 con 0x7f37e4159440 2024-04-17T10:28:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37e8a00640 1 -- 172.21.15.67:0/557881562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d80af070 con 0x7f37e4159440 2024-04-17T10:28:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37e8a00640 1 -- 172.21.15.67:0/557881562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d80aa860 con 0x7f37e4159440 2024-04-17T10:28:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 -- 172.21.15.67:0/557881562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e4159440 msgr2=0x7f37e40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 --2- 172.21.15.67:0/557881562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e4159440 0x7f37e40ce0f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f37e40726f0 tx=0x7f37d801c520 comp rx=0 tx=0).stop 2024-04-17T10:28:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 -- 172.21.15.67:0/557881562 shutdown_connections 2024-04-17T10:28:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 --2- 172.21.15.67:0/557881562 >> v2:172.21.15.86:3300/0 conn(0x7f37e415be70 0x7f37e415c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 --2- 172.21.15.67:0/557881562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e40713e0 0x7f37e4071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 --2- 172.21.15.67:0/557881562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e4159440 0x7f37e40ce0f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 -- 172.21.15.67:0/557881562 >> 172.21.15.67:0/557881562 conn(0x7f37e40b1740 msgr2=0x7f37e40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:33.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 -- 172.21.15.67:0/557881562 shutdown_connections 2024-04-17T10:28:33.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.022+0000 7f37ec800640 1 -- 172.21.15.67:0/557881562 wait complete. 2024-04-17T10:28:33.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 Processor -- start 2024-04-17T10:28:33.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 -- start start 2024-04-17T10:28:33.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e40713e0 0x7f37e4151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e4159440 0x7f37e4152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e415be70 0x7f37e415a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e415c7e0 con 0x7f37e40713e0 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4159820 con 0x7f37e415be70 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e415d840 con 0x7f37e4159440 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e40713e0 0x7f37e4151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e40713e0 0x7f37e4151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36154/0 (socket says 172.21.15.67:36154) 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 -- 172.21.15.67:0/122747097 learned_addr learned my addr 172.21.15.67:0/122747097 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ea800640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e415be70 0x7f37e415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9400640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e4159440 0x7f37e4152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 -- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e4159440 msgr2=0x7f37e4152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e4159440 0x7f37e4152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 -- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e415be70 msgr2=0x7f37e415a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e415be70 0x7f37e415a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 -- 172.21.15.67:0/122747097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40b04c0 con 0x7f37e40713e0 2024-04-17T10:28:33.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9400640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e4159440 0x7f37e4152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37e9e00640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e40713e0 0x7f37e4151220 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f37d801abd0 tx=0x7f37d801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37de000640 1 -- 172.21.15.67:0/122747097 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d80bd3f0 con 0x7f37e40713e0 2024-04-17T10:28:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37de000640 1 -- 172.21.15.67:0/122747097 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d80bd590 con 0x7f37e40713e0 2024-04-17T10:28:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37de000640 1 -- 172.21.15.67:0/122747097 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d80bd940 con 0x7f37e40713e0 2024-04-17T10:28:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e40b07a0 con 0x7f37e40713e0 2024-04-17T10:28:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.026+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e40b0d60 con 0x7f37e40713e0 2024-04-17T10:28:33.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37de000640 1 -- 172.21.15.67:0/122747097 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f37d80b33f0 con 0x7f37e40713e0 2024-04-17T10:28:33.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37de000640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f37c8066f60 0x7f37c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37de000640 1 -- 172.21.15.67:0/122747097 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f37d811f140 con 0x7f37e40713e0 2024-04-17T10:28:33.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 Processor -- start 2024-04-17T10:28:33.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37e9400640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f37c8066f60 0x7f37c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 -- start start 2024-04-17T10:28:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37e9400640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f37c8066f60 0x7f37c8069420 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f37e41f9aa0 tx=0x7f37d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692499d0 0x560669249db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x5606691ffb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669200480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56066904cff0 con 0x5606692000a0 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560668fd5150 con 0x5606692499d0 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x560669049090 con 0x56066924a2f0 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669200480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669200480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36166/0 (socket says 172.21.15.67:36166) 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 -- 172.21.15.67:0/4002778121 learned_addr learned my addr 172.21.15.67:0/4002778121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37e9400640 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x5606691ffb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37e9e00640 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692499d0 0x560669249db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 -- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 msgr2=0x5606691ffb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x5606691ffb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 -- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692499d0 msgr2=0x560669249db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692499d0 0x560669249db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.030+0000 7f37ea800640 1 -- 172.21.15.67:0/4002778121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606691e30d0 con 0x5606692000a0 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37ea800640 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669200480 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009f70 tx=0x7f37e00077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceefda928e318a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37d2a00640 1 -- 172.21.15.67:0/4002778121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e00147e0 con 0x5606692000a0 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37d2a00640 1 -- 172.21.15.67:0/4002778121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e001a070 con 0x5606692000a0 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37d2a00640 1 -- 172.21.15.67:0/4002778121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0014bc0 con 0x5606692000a0 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- 172.21.15.67:0/4002778121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 msgr2=0x560669200480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669200480 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f37e0009f70 tx=0x7f37e00077c0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- 172.21.15.67:0/4002778121 shutdown_connections 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669200480 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x5606691ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 --2- 172.21.15.67:0/4002778121 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692499d0 0x560669249db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- 172.21.15.67:0/4002778121 >> 172.21.15.67:0/4002778121 conn(0x5606691fc020 msgr2=0x5606691fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- 172.21.15.67:0/4002778121 shutdown_connections 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- 172.21.15.67:0/4002778121 wait complete. 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- start start 2024-04-17T10:28:33.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 Processor -- start 2024-04-17T10:28:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x56066922e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x56066922ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066922f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x560669011250 con 0x5606692499d0 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560669007690 con 0x5606692000a0 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x560668fd5a40 con 0x56066924a2f0 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37efd8f000 10 client.0 _ll_register_callbacks cb 0x5606691eae90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x56066922ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x56066922ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36168/0 (socket says 172.21.15.67:36168) 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 -- 172.21.15.67:0/1655738692 learned_addr learned my addr 172.21.15.67:0/1655738692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9e00640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x56066922e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37ea800640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066922f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 -- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 msgr2=0x56066922f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066922f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 -- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 msgr2=0x56066922e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x56066922e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 -- 172.21.15.67:0/1655738692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606690fa1c0 con 0x5606692499d0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9e00640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x56066922e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37e9400640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x56066922ed30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f37d40126c0 tx=0x7f37d40046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52be6df95d53d218 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.034+0000 7f37c7400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 1 -- 172.21.15.67:0/1655738692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4023070 con 0x5606692499d0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 1 -- 172.21.15.67:0/1655738692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f37d401f5f0 con 0x5606692499d0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37e001a070 con 0x5606692499d0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 1 -- 172.21.15.67:0/1655738692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37d4019040 con 0x5606692499d0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 10 client.6381 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x5606690f6170 con 0x5606692499d0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 1 -- 172.21.15.67:0/1655738692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e001a070 con 0x5606692499d0 2024-04-17T10:28:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 1 -- 172.21.15.67:0/1655738692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f37d4031080 con 0x5606692499d0 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 10 client.6381 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 10 client.6381 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 20 client.6381 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x560669259c80 0x56066925c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 4 client.6381 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560668fe8c60 con 0x560669259c80 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37e9e00640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x560669259c80 0x56066925c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37e9e00640 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x560669259c80 0x56066925c070 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6f57a4de08ec1f1 server_cookie=ee451b353ce9c2dc in_seq=0 out_seq=0 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 10 client.6381 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 1 -- 172.21.15.67:0/1655738692 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560668fe8c60 con 0x560669259c80 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 10 client.6381 handle_command_reply: tid=0 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 client.6381 shutdown 2024-04-17T10:28:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 msgr2=0x56066922ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x56066922ed30 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f37d40126c0 tx=0x7f37d40046d0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 shutdown_connections 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x560669259c80 0x56066925c070 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066922f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x56066922ed30 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37c7400640 0 client.6381 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 --2- 172.21.15.67:0/1655738692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x56066922e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 >> 172.21.15.67:0/1655738692 conn(0x5606691fc020 msgr2=0x5606691feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 shutdown_connections 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 1 -- 172.21.15.67:0/1655738692 wait complete. 2024-04-17T10:28:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.038+0000 7f37efd8f000 20 client.6381 trim_cache size 0 max 16384 2024-04-17T10:28:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 Processor -- start 2024-04-17T10:28:33.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 -- start start 2024-04-17T10:28:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669248c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5606692499d0 0x560669249140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56066924a2f0 0x5606691ff420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e42030d0 con 0x5606692000a0 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4202290 con 0x56066924a2f0 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37e41f76e0 con 0x5606692499d0 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669248c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669248c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36170/0 (socket says 172.21.15.67:36170) 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 -- 172.21.15.67:0/2327674437 learned_addr learned my addr 172.21.15.67:0/2327674437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37ea800640 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56066924a2f0 0x5606691ff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9400640 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5606692499d0 0x560669249140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 -- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5606692499d0 msgr2=0x560669249140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5606692499d0 0x560669249140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 -- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56066924a2f0 msgr2=0x5606691ff420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56066924a2f0 0x5606691ff420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 -- 172.21.15.67:0/2327674437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560669093de0 con 0x5606692000a0 2024-04-17T10:28:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37ea800640 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56066924a2f0 0x5606691ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37e9e00640 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669248c00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f37d801ca00 tx=0x7f37d811f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e57cb7ef2c0180 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.054+0000 7f37d2000640 1 -- 172.21.15.67:0/2327674437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d80c3040 con 0x5606692000a0 2024-04-17T10:28:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37d2000640 1 -- 172.21.15.67:0/2327674437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d80af040 con 0x5606692000a0 2024-04-17T10:28:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37d2000640 1 -- 172.21.15.67:0/2327674437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d80ec060 con 0x5606692000a0 2024-04-17T10:28:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- 172.21.15.67:0/2327674437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 msgr2=0x560669248c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669248c00 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f37d801ca00 tx=0x7f37d811f040 comp rx=0 tx=0).stop 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- 172.21.15.67:0/2327674437 shutdown_connections 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56066924a2f0 0x5606691ff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5606692499d0 0x560669249140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 --2- 172.21.15.67:0/2327674437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692000a0 0x560669248c00 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- 172.21.15.67:0/2327674437 >> 172.21.15.67:0/2327674437 conn(0x5606691fc020 msgr2=0x5606692480d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- 172.21.15.67:0/2327674437 shutdown_connections 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- 172.21.15.67:0/2327674437 wait complete. 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- start start 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 Processor -- start 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x5606692aa310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x5606692aa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066925c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x560668fe7e00 con 0x5606692499d0 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e4201d90 con 0x5606692000a0 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56066904c300 con 0x56066924a2f0 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 10 client.0 _ll_register_callbacks cb 0x5606691cb040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x5606692aa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x5606692aa850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36178/0 (socket says 172.21.15.67:36178) 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 -- 172.21.15.67:0/3436935349 learned_addr learned my addr 172.21.15.67:0/3436935349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9e00640 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x5606692aa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37ea800640 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066925c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 -- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 msgr2=0x56066925c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066925c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 -- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 msgr2=0x5606692aa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x5606692aa310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 -- 172.21.15.67:0/3436935349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560668fee8e0 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37e9400640 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x5606692aa850 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f37d400d690 tx=0x7f37d4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f47813ea1d5643 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37c7400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37c7400640 1 -- 172.21.15.67:0/3436935349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4038060 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37c7400640 1 -- 172.21.15.67:0/3436935349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f37d401f3f0 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37d80af040 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37c7400640 1 -- 172.21.15.67:0/3436935349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37d4023070 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 10 client.6387 fetch_fsmap learned FSMap version 29 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.058+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x5606690a68f0 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37c7400640 1 -- 172.21.15.67:0/3436935349 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d80af040 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37c7400640 1 -- 172.21.15.67:0/3436935349 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 29) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f37d4041020 con 0x5606692499d0 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 10 client.6387 fetch_fsmap finished waiting for FSMap version 29 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 10 client.6387 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:33.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 20 client.6387 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x56066925d160 0x56066925d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 4 client.6387 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56066905a530 con 0x56066925d160 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37e9e00640 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x56066925d160 0x56066925d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37e9e00640 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x56066925d160 0x56066925d570 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=935b15f2b49213f0 server_cookie=1ba9f90ff999657 in_seq=0 out_seq=0 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37c7400640 10 client.6387 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37c7400640 1 -- 172.21.15.67:0/3436935349 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+284 (crc 0 0 0) 0x56066905a530 con 0x56066925d160 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37c7400640 10 client.6387 handle_command_reply: tid=0 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 client.6387 shutdown 2024-04-17T10:28:33.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 msgr2=0x5606692aa850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x5606692aa850 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f37d400d690 tx=0x7f37d4014000 comp rx=0 tx=0).stop 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 shutdown_connections 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x56066925d160 0x56066925d570 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56066924a2f0 0x56066925c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5606692499d0 0x5606692aa850 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 --2- 172.21.15.67:0/3436935349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5606692000a0 0x5606692aa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37c7400640 0 client.6387 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 >> 172.21.15.67:0/3436935349 conn(0x5606691fc020 msgr2=0x560669248960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 shutdown_connections 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 1 -- 172.21.15.67:0/3436935349 wait complete. 2024-04-17T10:28:33.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37efd8f000 20 client.6387 trim_cache size 0 max 16384 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: "71bb7d24-e443-42d0-b6a0-ecfc71d44328": { 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_1", 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "71bb7d24-e443-42d0-b6a0-ecfc71d44328", 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "" 2024-04-17T10:28:33.061 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:33.062 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:33.062 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:33.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f37c8066f60 msgr2=0x7f37c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f37c8066f60 0x7f37c8069420 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f37e41f9aa0 tx=0x7f37d4008040 comp rx=0 tx=0).stop 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e40713e0 msgr2=0x7f37e4151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e40713e0 0x7f37e4151220 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f37d801abd0 tx=0x7f37d801ad60 comp rx=0 tx=0).stop 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 shutdown_connections 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f37c8066f60 0x7f37c8069420 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f37e415be70 0x7f37e415a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37e4159440 0x7f37e4152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 --2- 172.21.15.67:0/122747097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37e40713e0 0x7f37e4151220 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 >> 172.21.15.67:0/122747097 conn(0x7f37e40b1740 msgr2=0x7f37e40adc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 shutdown_connections 2024-04-17T10:28:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:33.062+0000 7f37ec800640 1 -- 172.21.15.67:0/122747097 wait complete. 2024-04-17T10:28:38.076 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-17T10:28:38.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.3:0 scrub status 2024-04-17T10:28:38.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402ec00640 1 Processor -- start 2024-04-17T10:28:38.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402ec00640 1 -- start start 2024-04-17T10:28:38.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402ec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 0x7f40281653b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402ec00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40280b4990 con 0x7f40280cd030 2024-04-17T10:28:38.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402ec00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f402815ae90 con 0x7f40280aef60 2024-04-17T10:28:38.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402ec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40280af340 con 0x7f402806e980 2024-04-17T10:28:38.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f40280cd030 0x7f402806e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56192/0 (socket says 172.21.15.67:56192) 2024-04-17T10:28:38.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 -- 172.21.15.67:0/4124704945 learned_addr learned my addr 172.21.15.67:0/4124704945 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:38.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402cc00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 0x7f40281653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384764130 0 0) 0x7f4014001120 con 0x7f40280cd030 2024-04-17T10:28:38.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c005740 con 0x7f40280cd030 2024-04-17T10:28:38.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285200632 0 0) 0x7f400c005740 con 0x7f40280cd030 2024-04-17T10:28:38.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 msgr2=0x7f40281653b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 0x7f40281653b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 >> v1:172.21.15.86:6789/0 conn(0x7f40280aef60 legacy=0x7f40280cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402806ee80 con 0x7f40280cd030 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f4014001870 con 0x7f40280cd030 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 >> v1:172.21.15.67:6789/0 conn(0x7f40280cd030 legacy=0x7f402806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 0x7f400c005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f400c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f400c006f60 0x7f400c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40280af340 con 0x7f400c006140 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f400c00dd20 con 0x7f400c006f60 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f400c00df20 con 0x7f402806e980 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f400c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027e00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 0x7f400c005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f402cc00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f400c006f60 0x7f400c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 -- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 msgr2=0x7f400c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f402806e980 0x7f400c005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 -- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f400c006f60 msgr2=0x7f400c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f400c006f60 0x7f400c0073e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.242+0000 7f4027400640 1 -- 172.21.15.67:0/4124704945 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c005740 con 0x7f400c006140 2024-04-17T10:28:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f4027400640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f400c006a20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f40280b4f60 tx=0x7f401401cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1400d0e634e478 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401400c0e0 con 0x7f400c006140 2024-04-17T10:28:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40140a9070 con 0x7f400c006140 2024-04-17T10:28:38.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f4026a00640 1 -- 172.21.15.67:0/4124704945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4014005d40 con 0x7f400c006140 2024-04-17T10:28:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f402ec00640 1 -- 172.21.15.67:0/4124704945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 msgr2=0x7f400c006a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f402ec00640 1 --2- 172.21.15.67:0/4124704945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f400c006a20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f40280b4f60 tx=0x7f401401cb60 comp rx=0 tx=0).stop 2024-04-17T10:28:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f402cc00640 1 -- 172.21.15.67:0/4124704945 reap_dead start 2024-04-17T10:28:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f402ec00640 1 -- 172.21.15.67:0/4124704945 shutdown_connections 2024-04-17T10:28:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.246+0000 7f402ec00640 1 -- 172.21.15.67:0/4124704945 >> 172.21.15.67:0/4124704945 conn(0x7f40280b1740 msgr2=0x7f40280b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 -- 172.21.15.67:0/4124704945 shutdown_connections 2024-04-17T10:28:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 -- 172.21.15.67:0/4124704945 wait complete. 2024-04-17T10:28:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 Processor -- start 2024-04-17T10:28:38.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 -- start start 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f4028162d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402815eec0 0x7f40280b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4028149fc0 0x7f4028161a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402806daf0 con 0x7f400c006140 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f402806d970 con 0x7f402815eec0 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f402806dc70 con 0x7f4028149fc0 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f4028162d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f4028162d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36186/0 (socket says 172.21.15.67:36186) 2024-04-17T10:28:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 -- 172.21.15.67:0/746674262 learned_addr learned my addr 172.21.15.67:0/746674262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 -- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4028149fc0 msgr2=0x7f4028161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4028149fc0 0x7f4028161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 -- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402815eec0 msgr2=0x7f40280b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402815eec0 0x7f40280b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 -- 172.21.15.67:0/746674262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f402815f340 con 0x7f400c006140 2024-04-17T10:28:38.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f4027e00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f4028162d60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f401800f740 tx=0x7f401800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f401fe00640 1 -- 172.21.15.67:0/746674262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401800fe50 con 0x7f400c006140 2024-04-17T10:28:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f401fe00640 1 -- 172.21.15.67:0/746674262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4018018070 con 0x7f400c006140 2024-04-17T10:28:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f401fe00640 1 -- 172.21.15.67:0/746674262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401800d6b0 con 0x7f400c006140 2024-04-17T10:28:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.250+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4028160fa0 con 0x7f400c006140 2024-04-17T10:28:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.254+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f402815f8a0 con 0x7f400c006140 2024-04-17T10:28:38.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.254+0000 7f401fe00640 1 -- 172.21.15.67:0/746674262 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f401800d850 con 0x7f400c006140 2024-04-17T10:28:38.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.254+0000 7f401fe00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4008066f90 0x7f4008069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.254+0000 7f401fe00640 1 -- 172.21.15.67:0/746674262 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f40180850d0 con 0x7f400c006140 2024-04-17T10:28:38.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.254+0000 7f403201d000 1 Processor -- start 2024-04-17T10:28:38.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.254+0000 7f4027400640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4008066f90 0x7f4008069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.254+0000 7f403201d000 1 -- start start 2024-04-17T10:28:38.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f4027400640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4008066f90 0x7f4008069450 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f40140ab000 tx=0x7f4014001b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7a2480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5a5eeff0 con 0x556e5a7a20a0 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5a577150 con 0x556e5a7ec2f0 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e5a5eb090 con 0x556e5a7eb9d0 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7a2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7a2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36196/0 (socket says 172.21.15.67:36196) 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 -- 172.21.15.67:0/3466710098 learned_addr learned my addr 172.21.15.67:0/3466710098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f4027400640 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f4027e00640 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 -- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 msgr2=0x556e5a7ebdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7ebdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 -- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 msgr2=0x556e5a7a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 -- 172.21.15.67:0/3466710098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e5a7850d0 con 0x556e5a7a20a0 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f4027e00640 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7ebdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f402cc00640 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7a2480 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f40280cf720 tx=0x7f4020011600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4e2e583a537e74 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f401cc00640 1 -- 172.21.15.67:0/3466710098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020015630 con 0x556e5a7a20a0 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f401cc00640 1 -- 172.21.15.67:0/3466710098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40200157d0 con 0x556e5a7a20a0 2024-04-17T10:28:38.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f401cc00640 1 -- 172.21.15.67:0/3466710098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020015bc0 con 0x556e5a7a20a0 2024-04-17T10:28:38.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- 172.21.15.67:0/3466710098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 msgr2=0x556e5a7a2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7a2480 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f40280cf720 tx=0x7f4020011600 comp rx=0 tx=0).stop 2024-04-17T10:28:38.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- 172.21.15.67:0/3466710098 shutdown_connections 2024-04-17T10:28:38.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7a2480 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 --2- 172.21.15.67:0/3466710098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7ebdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- 172.21.15.67:0/3466710098 >> 172.21.15.67:0/3466710098 conn(0x556e5a79e020 msgr2=0x556e5a7a0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- 172.21.15.67:0/3466710098 shutdown_connections 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- 172.21.15.67:0/3466710098 wait complete. 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 -- start start 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.258+0000 7f403201d000 1 Processor -- start 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7d07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5a5b3250 con 0x556e5a7a20a0 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5a5a9690 con 0x556e5a7ec2f0 2024-04-17T10:28:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e5a577a40 con 0x556e5a7eb9d0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 10 client.0 _ll_register_callbacks cb 0x556e5a78ce90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7d07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7d07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36206/0 (socket says 172.21.15.67:36206) 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 -- 172.21.15.67:0/2768641687 learned_addr learned my addr 172.21.15.67:0/2768641687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027400640 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f402cc00640 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 -- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 msgr2=0x556e5a7d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7d0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 -- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 msgr2=0x556e5a7d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 -- 172.21.15.67:0/2768641687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e5a69c1c0 con 0x556e5a7a20a0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027e00640 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7d07f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4018055510 tx=0x7f4018056050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8a8b0c6335f5e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4011600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4011600640 1 -- 172.21.15.67:0/2768641687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018081060 con 0x556e5a7a20a0 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4011600640 1 -- 172.21.15.67:0/2768641687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f401806d100 con 0x556e5a7a20a0 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40200157d0 con 0x556e5a7a20a0 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4011600640 1 -- 172.21.15.67:0/2768641687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4018072050 con 0x556e5a7a20a0 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 10 client.6399 fetch_fsmap learned FSMap version 30 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x556e5a698170 con 0x556e5a7a20a0 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4011600640 1 -- 172.21.15.67:0/2768641687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40200157d0 con 0x556e5a7a20a0 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4011600640 1 -- 172.21.15.67:0/2768641687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4018056b40 con 0x556e5a7a20a0 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 10 client.6399 fetch_fsmap finished waiting for FSMap version 30 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 10 client.6399 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 20 client.6399 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a7fbc80 0x556e5a7fe070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 4 client.6399 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e5a58ac60 con 0x556e5a7fbc80 2024-04-17T10:28:38.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027400640 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a7fbc80 0x556e5a7fe070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4027400640 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a7fbc80 0x556e5a7fe070 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=461501f68215033d server_cookie=bcb7f722b0a727e3 in_seq=0 out_seq=0 2024-04-17T10:28:38.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.262+0000 7f4011600640 10 client.6399 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f4011600640 1 -- 172.21.15.67:0/2768641687 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556e5a58ac60 con 0x556e5a7fbc80 2024-04-17T10:28:38.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f4011600640 10 client.6399 handle_command_reply: tid=0 2024-04-17T10:28:38.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 client.6399 shutdown 2024-04-17T10:28:38.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 msgr2=0x556e5a7d07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7d07f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f4018055510 tx=0x7f4018056050 comp rx=0 tx=0).stop 2024-04-17T10:28:38.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 shutdown_connections 2024-04-17T10:28:38.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a7fbc80 0x556e5a7fe070 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7ec2f0 0x556e5a7d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7eb9d0 0x556e5a7d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 --2- 172.21.15.67:0/2768641687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7d07f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f4011600640 0 client.6399 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 >> 172.21.15.67:0/2768641687 conn(0x556e5a79e020 msgr2=0x556e5a7a0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 shutdown_connections 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 1 -- 172.21.15.67:0/2768641687 wait complete. 2024-04-17T10:28:38.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.266+0000 7f403201d000 20 client.6399 trim_cache size 0 max 16384 2024-04-17T10:28:38.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.278+0000 7f403201d000 1 Processor -- start 2024-04-17T10:28:38.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- start start 2024-04-17T10:28:38.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7a20a0 0x556e5a848300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7eb9d0 0x556e5a848840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a848d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40281ff7d0 con 0x556e5a7eb9d0 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40281fe8f0 con 0x556e5a7a20a0 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40281fdcd0 con 0x556e5a7ec2f0 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7eb9d0 0x556e5a848840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7eb9d0 0x556e5a848840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36220/0 (socket says 172.21.15.67:36220) 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 -- 172.21.15.67:0/3514521908 learned_addr learned my addr 172.21.15.67:0/3514521908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027e00640 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7a20a0 0x556e5a848300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f402cc00640 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a848d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 -- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 msgr2=0x556e5a848d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a848d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 -- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7a20a0 msgr2=0x556e5a848300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7a20a0 0x556e5a848300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 -- 172.21.15.67:0/3514521908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e5a635de0 con 0x556e5a7eb9d0 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027e00640 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7a20a0 0x556e5a848300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4027400640 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7eb9d0 0x556e5a848840 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f401401c5d0 tx=0x7f40140a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4280529ce60e42 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4013e00640 1 -- 172.21.15.67:0/3514521908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40140cc070 con 0x556e5a7eb9d0 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4013e00640 1 -- 172.21.15.67:0/3514521908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40140a9030 con 0x556e5a7eb9d0 2024-04-17T10:28:38.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f4013e00640 1 -- 172.21.15.67:0/3514521908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40140c7680 con 0x556e5a7eb9d0 2024-04-17T10:28:38.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- 172.21.15.67:0/3514521908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7eb9d0 msgr2=0x556e5a848840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7eb9d0 0x556e5a848840 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f401401c5d0 tx=0x7f40140a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:38.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- 172.21.15.67:0/3514521908 shutdown_connections 2024-04-17T10:28:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a848d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7eb9d0 0x556e5a848840 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 --2- 172.21.15.67:0/3514521908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7a20a0 0x556e5a848300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- 172.21.15.67:0/3514521908 >> 172.21.15.67:0/3514521908 conn(0x556e5a79e020 msgr2=0x556e5a7f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- 172.21.15.67:0/3514521908 shutdown_connections 2024-04-17T10:28:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- 172.21.15.67:0/3514521908 wait complete. 2024-04-17T10:28:38.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 -- start start 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.282+0000 7f403201d000 1 Processor -- start 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7fe380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7eb9d0 0x556e5a7fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a7fee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5a589e00 con 0x556e5a7a20a0 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40281fe2d0 con 0x556e5a7eb9d0 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e5a5ee300 con 0x556e5a7ec2f0 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 10 client.0 _ll_register_callbacks cb 0x556e5a76d040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36226/0 (socket says 172.21.15.67:36226) 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 -- 172.21.15.67:0/3194528076 learned_addr learned my addr 172.21.15.67:0/3194528076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027400640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7eb9d0 0x556e5a7fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f402cc00640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a7fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 -- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 msgr2=0x556e5a7fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a7fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 -- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7eb9d0 msgr2=0x556e5a7fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7eb9d0 0x556e5a7fe8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 -- 172.21.15.67:0/3194528076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e5a5908e0 con 0x556e5a7a20a0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f402cc00640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a7fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027e00640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7fe380 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4018052190 tx=0x7f4018083300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407642c02658d438 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 1 -- 172.21.15.67:0/3194528076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4018077020 con 0x556e5a7a20a0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40140a9030 con 0x556e5a7a20a0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 1 -- 172.21.15.67:0/3194528076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f401800dd40 con 0x556e5a7a20a0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 1 -- 172.21.15.67:0/3194528076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4018055170 con 0x556e5a7a20a0 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 10 client.6405 fetch_fsmap learned FSMap version 30 2024-04-17T10:28:38.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x556e5a6488f0 con 0x556e5a7a20a0 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 1 -- 172.21.15.67:0/3194528076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40140a9030 con 0x556e5a7a20a0 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 1 -- 172.21.15.67:0/3194528076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4018054c60 con 0x556e5a7a20a0 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 10 client.6405 fetch_fsmap finished waiting for FSMap version 30 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 10 client.6405 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 20 client.6405 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a851f50 0x556e5a854340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 4 client.6405 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556e5a5fc530 con 0x556e5a851f50 2024-04-17T10:28:38.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027400640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a851f50 0x556e5a854340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4027400640 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a851f50 0x556e5a854340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=907d950987935100 server_cookie=261a6b15886f9f19 in_seq=0 out_seq=0 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 10 client.6405 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 1 -- 172.21.15.67:0/3194528076 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556e5a5fc530 con 0x556e5a851f50 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 10 client.6405 handle_command_reply: tid=0 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 client.6405 shutdown 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 msgr2=0x556e5a7fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7fe380 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f4018052190 tx=0x7f4018083300 comp rx=0 tx=0).stop 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 shutdown_connections 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x556e5a851f50 0x556e5a854340 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556e5a7ec2f0 0x556e5a7fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556e5a7eb9d0 0x556e5a7fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 --2- 172.21.15.67:0/3194528076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556e5a7a20a0 0x556e5a7fe380 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f4011600640 0 client.6405 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 >> 172.21.15.67:0/3194528076 conn(0x556e5a79e020 msgr2=0x556e5a7ea0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 shutdown_connections 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.286+0000 7f403201d000 1 -- 172.21.15.67:0/3194528076 wait complete. 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f403201d000 20 client.6405 trim_cache size 0 max 16384 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:28:38.287 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:38.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4008066f90 msgr2=0x7f4008069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4008066f90 0x7f4008069450 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f40140ab000 tx=0x7f4014001b70 comp rx=0 tx=0).stop 2024-04-17T10:28:38.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 msgr2=0x7f4028162d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f4028162d60 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f401800f740 tx=0x7f401800fc00 comp rx=0 tx=0).stop 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 shutdown_connections 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4008066f90 0x7f4008069450 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4028149fc0 0x7f4028161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f402815eec0 0x7f40280b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 --2- 172.21.15.67:0/746674262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f400c006140 0x7f4028162d60 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 >> 172.21.15.67:0/746674262 conn(0x7f40280b1740 msgr2=0x7f402806e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 shutdown_connections 2024-04-17T10:28:38.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.290+0000 7f402ec00640 1 -- 172.21.15.67:0/746674262 wait complete. 2024-04-17T10:28:38.303 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-17T10:28:38.303 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:28:38.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-04-17T10:28:38.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd356e00640 1 Processor -- start 2024-04-17T10:28:38.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd356e00640 1 -- start start 2024-04-17T10:28:38.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd356e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd35006d810 0x7fd3500cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd356e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3500707c0 0x7fd35015d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd356e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3500d03c0 con 0x7fd35006f8b0 2024-04-17T10:28:38.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd356e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3500d06a0 con 0x7fd35006d810 2024-04-17T10:28:38.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd356e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500d0980 con 0x7fd3500707c0 2024-04-17T10:28:38.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd35006d810 0x7fd3500cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd354e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3500707c0 0x7fd35015d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd35006d810 0x7fd3500cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:54754/0 (socket says 172.21.15.67:54754) 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34fe00640 1 -- 172.21.15.67:0/3128945853 learned_addr learned my addr 172.21.15.67:0/3128945853 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd347400640 1 --1- 172.21.15.67:0/3128945853 >> v1:172.21.15.86:6789/0 conn(0x7fd35006f8b0 0x7fd3500700b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:35402/0 (socket says 172.21.15.67:35402) 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34fe00640 1 -- 172.21.15.67:0/3128945853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3500707c0 msgr2=0x7fd35015d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34fe00640 1 --2- 172.21.15.67:0/3128945853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3500707c0 0x7fd35015d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34fe00640 1 -- 172.21.15.67:0/3128945853 >> v1:172.21.15.86:6789/0 conn(0x7fd35006f8b0 legacy=0x7fd3500700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34fe00640 1 -- 172.21.15.67:0/3128945853 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500d0c60 con 0x7fd35006d810 2024-04-17T10:28:38.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.454+0000 7fd34f400640 1 -- 172.21.15.67:0/3128945853 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501750177 0 0) 0x7fd3500d03c0 con 0x7fd35006f8b0 2024-04-17T10:28:38.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd34fe00640 1 --2- 172.21.15.67:0/3128945853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 0x7fd3500cfad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd340009870 tx=0x7fd34001c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f50feb08fbe3591c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd34f400640 1 -- 172.21.15.67:0/3128945853 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400ab450 con 0x7fd35006d810 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd34f400640 1 -- 172.21.15.67:0/3128945853 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3400b0070 con 0x7fd35006d810 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd34f400640 1 -- 172.21.15.67:0/3128945853 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400ab810 con 0x7fd35006d810 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 -- 172.21.15.67:0/3128945853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 msgr2=0x7fd3500cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 --2- 172.21.15.67:0/3128945853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 0x7fd3500cfad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd340009870 tx=0x7fd34001c530 comp rx=0 tx=0).stop 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 -- 172.21.15.67:0/3128945853 shutdown_connections 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 --2- 172.21.15.67:0/3128945853 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd3500707c0 0x7fd35015d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 --2- 172.21.15.67:0/3128945853 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 0x7fd3500cfad0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 -- 172.21.15.67:0/3128945853 >> 172.21.15.67:0/3128945853 conn(0x7fd3500d1800 msgr2=0x7fd3500d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 -- 172.21.15.67:0/3128945853 shutdown_connections 2024-04-17T10:28:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.458+0000 7fd356e00640 1 -- 172.21.15.67:0/3128945853 wait complete. 2024-04-17T10:28:38.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 Processor -- start 2024-04-17T10:28:38.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 -- start start 2024-04-17T10:28:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 0x7fd350150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd35006f8b0 0x7fd3501515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3500707c0 0x7fd35015ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35006bda0 con 0x7fd3500707c0 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35006de00 con 0x7fd35006d810 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35006bf20 con 0x7fd35006f8b0 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3500707c0 0x7fd35015ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3500707c0 0x7fd35015ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36240/0 (socket says 172.21.15.67:36240) 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 -- 172.21.15.67:0/1784182797 learned_addr learned my addr 172.21.15.67:0/1784182797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd34fe00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 0x7fd350150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd347400640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd35006f8b0 0x7fd3501515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 -- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd35006f8b0 msgr2=0x7fd3501515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd35006f8b0 0x7fd3501515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 -- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 msgr2=0x7fd350150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 0x7fd350150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 -- 172.21.15.67:0/1784182797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35015bae0 con 0x7fd3500707c0 2024-04-17T10:28:38.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd354e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3500707c0 0x7fd35015ce60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd34800efc0 tx=0x7fd34800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd34cc00640 1 -- 172.21.15.67:0/1784182797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348018020 con 0x7fd3500707c0 2024-04-17T10:28:38.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd34cc00640 1 -- 172.21.15.67:0/1784182797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348014480 con 0x7fd3500707c0 2024-04-17T10:28:38.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35015bdc0 con 0x7fd3500707c0 2024-04-17T10:28:38.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd34cc00640 1 -- 172.21.15.67:0/1784182797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348009e60 con 0x7fd3500707c0 2024-04-17T10:28:38.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.462+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35015c380 con 0x7fd3500707c0 2024-04-17T10:28:38.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd34cc00640 1 -- 172.21.15.67:0/1784182797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd34801b020 con 0x7fd3500707c0 2024-04-17T10:28:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd34cc00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd32c066f60 0x7fd32c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd34cc00640 1 -- 172.21.15.67:0/1784182797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd348085f20 con 0x7fd3500707c0 2024-04-17T10:28:38.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd34fe00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd32c066f60 0x7fd32c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 Processor -- start 2024-04-17T10:28:38.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 -- start start 2024-04-17T10:28:38.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd34fe00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd32c066f60 0x7fd32c069420 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd3400af000 tx=0x7fd3400a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5180 0x55ed69fe5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9b600 0x55ed69f9b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed69de8f20 con 0x55ed69fe5aa0 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed69d71000 con 0x55ed69fe5180 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.466+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed69de4fc0 con 0x55ed69f9b600 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36248/0 (socket says 172.21.15.67:36248) 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 -- 172.21.15.67:0/2875353318 learned_addr learned my addr 172.21.15.67:0/2875353318 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd34fe00640 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5180 0x55ed69fe5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd354e00640 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9b600 0x55ed69f9b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 -- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9b600 msgr2=0x55ed69f9b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9b600 0x55ed69f9b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 -- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5180 msgr2=0x55ed69fe5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5180 0x55ed69fe5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 -- 172.21.15.67:0/2875353318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed69dc4ba0 con 0x55ed69fe5aa0 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd347400640 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9b0c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f70 tx=0x7fd338004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76f8d2c9b7f5db6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd344c00640 1 -- 172.21.15.67:0/2875353318 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33801a070 con 0x55ed69fe5aa0 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd344c00640 1 -- 172.21.15.67:0/2875353318 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3380018b0 con 0x55ed69fe5aa0 2024-04-17T10:28:38.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd344c00640 1 -- 172.21.15.67:0/2875353318 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338001ca0 con 0x55ed69fe5aa0 2024-04-17T10:28:38.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2875353318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5aa0 msgr2=0x55ed69f9b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9b0c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd338000f70 tx=0x7fd338004bc0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2875353318 shutdown_connections 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9b600 0x55ed69f9b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9b0c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2875353318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5180 0x55ed69fe5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2875353318 >> 172.21.15.67:0/2875353318 conn(0x55ed69fa1760 msgr2=0x55ed69f999a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2875353318 shutdown_connections 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2875353318 wait complete. 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- start start 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 Processor -- start 2024-04-17T10:28:38.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9b600 0x55ed69fa06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69fa0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69fd3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed69ea69c0 con 0x55ed69f9b600 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed69da35d0 con 0x55ed69fe5aa0 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed69d942f0 con 0x55ed69fe5180 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd35a2d2000 10 client.0 _ll_register_callbacks cb 0x55ed69f86a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd34fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9b600 0x55ed69fa06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.470+0000 7fd34fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9b600 0x55ed69fa06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36258/0 (socket says 172.21.15.67:36258) 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd34fe00640 1 -- 172.21.15.67:0/11020608 learned_addr learned my addr 172.21.15.67:0/11020608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd354e00640 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69fd3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd347400640 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69fa0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd34fe00640 1 -- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 msgr2=0x55ed69fa0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd34fe00640 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69fa0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd34fe00640 1 -- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 msgr2=0x55ed69fd3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd34fe00640 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69fd3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd34fe00640 1 -- 172.21.15.67:0/11020608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed69e2f5a0 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd34fe00640 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9b600 0x55ed69fa06e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd340098020 tx=0x7fd34001ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b9d40aa13a580c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 1 -- 172.21.15.67:0/11020608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400cf060 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 1 -- 172.21.15.67:0/11020608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd340003d80 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3380018b0 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 1 -- 172.21.15.67:0/11020608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3400ab450 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 10 client.6414 fetch_fsmap learned FSMap version 30 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x55ed69dc02b0 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 1 -- 172.21.15.67:0/11020608 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3380018b0 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 1 -- 172.21.15.67:0/11020608 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd3400ab5f0 con 0x55ed69f9b600 2024-04-17T10:28:38.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 10 client.6414 fetch_fsmap finished waiting for FSMap version 30 2024-04-17T10:28:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 10 client.6414 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 20 client.6414 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69fd3e60 0x55ed69ff7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 4 client.6414 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed69d84c70 con 0x55ed69fd3e60 2024-04-17T10:28:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd347400640 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69fd3e60 0x55ed69ff7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd347400640 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69fd3e60 0x55ed69ff7440 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2aed4835d4c211c server_cookie=6637e199a49a73f7 in_seq=0 out_seq=0 2024-04-17T10:28:38.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 10 client.6414 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 1 -- 172.21.15.67:0/11020608 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ed69d84c70 con 0x55ed69fd3e60 2024-04-17T10:28:38.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 10 client.6414 handle_command_reply: tid=0 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 client.6414 shutdown 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9b600 msgr2=0x55ed69fa06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9b600 0x55ed69fa06e0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd340098020 tx=0x7fd34001ad70 comp rx=0 tx=0).stop 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 shutdown_connections 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69fd3e60 0x55ed69ff7440 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69fd3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69fa0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/11020608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9b600 0x55ed69fa06e0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 >> 172.21.15.67:0/11020608 conn(0x55ed69fa1760 msgr2=0x55ed69f9a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd33d600640 0 client.6414 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 shutdown_connections 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.474+0000 7fd35a2d2000 1 -- 172.21.15.67:0/11020608 wait complete. 2024-04-17T10:28:38.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.478+0000 7fd35a2d2000 20 client.6414 trim_cache size 0 max 16384 2024-04-17T10:28:38.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 Processor -- start 2024-04-17T10:28:38.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- start start 2024-04-17T10:28:38.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69ff9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9a660 0x55ed69ff5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350203190 con 0x55ed69f9a660 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350202390 con 0x55ed69fe5aa0 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500b5560 con 0x55ed69fe5180 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9a660 0x55ed69ff5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9a660 0x55ed69ff5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36262/0 (socket says 172.21.15.67:36262) 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 -- 172.21.15.67:0/554431451 learned_addr learned my addr 172.21.15.67:0/554431451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd34fe00640 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69ff9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd347400640 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 -- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 msgr2=0x55ed69ff9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69ff9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 -- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 msgr2=0x55ed69f9a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 -- 172.21.15.67:0/554431451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed69d71250 con 0x55ed69f9a660 2024-04-17T10:28:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd354e00640 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9a660 0x55ed69ff5430 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd348056380 tx=0x7fd3480840c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369fcdd5108639e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd33fe00640 1 -- 172.21.15.67:0/554431451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348082070 con 0x55ed69f9a660 2024-04-17T10:28:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd33fe00640 1 -- 172.21.15.67:0/554431451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348052cf0 con 0x55ed69f9a660 2024-04-17T10:28:38.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd33fe00640 1 -- 172.21.15.67:0/554431451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34807dd90 con 0x55ed69f9a660 2024-04-17T10:28:38.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- 172.21.15.67:0/554431451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9a660 msgr2=0x55ed69ff5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9a660 0x55ed69ff5430 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd348056380 tx=0x7fd3480840c0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- 172.21.15.67:0/554431451 shutdown_connections 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69f9a660 0x55ed69ff5430 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69f9a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/554431451 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69fe5180 0x55ed69ff9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- 172.21.15.67:0/554431451 >> 172.21.15.67:0/554431451 conn(0x55ed69fa1760 msgr2=0x55ed69fe4930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- 172.21.15.67:0/554431451 shutdown_connections 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- 172.21.15.67:0/554431451 wait complete. 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 -- start start 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.490+0000 7fd35a2d2000 1 Processor -- start 2024-04-17T10:28:38.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9a660 0x55ed6a045070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5180 0x55ed69ff82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69ff87e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed69dac7b0 con 0x55ed69fe5180 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350201dc0 con 0x55ed69fe5aa0 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed69daaee0 con 0x55ed69f9a660 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 10 client.0 _ll_register_callbacks cb 0x55ed69f67600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5180 0x55ed69ff82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5180 0x55ed69ff82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36270/0 (socket says 172.21.15.67:36270) 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 -- 172.21.15.67:0/2984762123 learned_addr learned my addr 172.21.15.67:0/2984762123 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd354e00640 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69ff87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd34fe00640 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9a660 0x55ed6a045070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 -- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9a660 msgr2=0x55ed6a045070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9a660 0x55ed6a045070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 -- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 msgr2=0x55ed69ff87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69ff87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 -- 172.21.15.67:0/2984762123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed69d7db90 con 0x55ed69fe5180 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd347400640 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5180 0x55ed69ff82a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd338006010 tx=0x7fd338000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e6ce6c7e5a8a72 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 1 -- 172.21.15.67:0/2984762123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338029020 con 0x55ed69fe5180 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 1 -- 172.21.15.67:0/2984762123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd3380018b0 con 0x55ed69fe5180 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd348052cf0 con 0x55ed69fe5180 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 1 -- 172.21.15.67:0/2984762123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd338033050 con 0x55ed69fe5180 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 10 client.6420 fetch_fsmap learned FSMap version 30 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) v3 -- 0x55ed69d90010 con 0x55ed69fe5180 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 1 -- 172.21.15.67:0/2984762123 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348052cf0 con 0x55ed69fe5180 2024-04-17T10:28:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 1 -- 172.21.15.67:0/2984762123 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 30) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd338036020 con 0x55ed69fe5180 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 10 client.6420 fetch_fsmap finished waiting for FSMap version 30 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 10 client.6420 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.d 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 20 client.6420 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69ff9410 0x55ed69ff97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 4 client.6420 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ed69df6460 con 0x55ed69ff9410 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd34fe00640 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69ff9410 0x55ed69ff97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd34fe00640 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69ff9410 0x55ed69ff97f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9e1850742466897 server_cookie=4b3a45e4909e3a5 in_seq=0 out_seq=0 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 10 client.6420 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 1 -- 172.21.15.67:0/2984762123 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+168 (crc 0 0 0) 0x55ed69df6460 con 0x55ed69ff9410 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 10 client.6420 handle_command_reply: tid=0 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 client.6420 shutdown 2024-04-17T10:28:38.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5180 msgr2=0x55ed69ff82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5180 0x55ed69ff82a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd338006010 tx=0x7fd338000ee0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 shutdown_connections 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55ed69ff9410 0x55ed69ff97f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ed69fe5aa0 0x55ed69ff87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ed69fe5180 0x55ed69ff82a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd33d600640 0 client.6420 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 --2- 172.21.15.67:0/2984762123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ed69f9a660 0x55ed6a045070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 >> 172.21.15.67:0/2984762123 conn(0x55ed69fa1760 msgr2=0x55ed69ff2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 shutdown_connections 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 1 -- 172.21.15.67:0/2984762123 wait complete. 2024-04-17T10:28:38.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.494+0000 7fd35a2d2000 20 client.6420 trim_cache size 0 max 16384 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout: "damage_type": "backtrace", 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 1933588164, 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 1099511630189, 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_1" 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:38.495 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:28:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd32c066f60 msgr2=0x7fd32c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd32c066f60 0x7fd32c069420 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd3400af000 tx=0x7fd3400a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3500707c0 msgr2=0x7fd35015ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3500707c0 0x7fd35015ce60 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd34800efc0 tx=0x7fd34800c490 comp rx=0 tx=0).stop 2024-04-17T10:28:38.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 shutdown_connections 2024-04-17T10:28:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd32c066f60 0x7fd32c069420 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd3500707c0 0x7fd35015ce60 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd35006f8b0 0x7fd3501515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 --2- 172.21.15.67:0/1784182797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd35006d810 0x7fd350150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 >> 172.21.15.67:0/1784182797 conn(0x7fd3500d1800 msgr2=0x7fd3500d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 shutdown_connections 2024-04-17T10:28:38.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.498+0000 7fd356e00640 1 -- 172.21.15.67:0/1784182797 wait complete. 2024-04-17T10:28:38.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:38.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 Processor -- start 2024-04-17T10:28:38.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 -- start start 2024-04-17T10:28:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c0b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd86c0b57b0 0x7fd86c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd86c0b07b0 0x7fd86c071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c0b1280 con 0x7fd86c0b07b0 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd86c0b0cc0 con 0x7fd86c0b57b0 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd873400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c0b0fa0 con 0x7fd86c15dd10 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd86c0b07b0 0x7fd86c071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd86c0b07b0 0x7fd86c071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36278/0 (socket says 172.21.15.67:36278) 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 -- 172.21.15.67:0/3805027271 learned_addr learned my addr 172.21.15.67:0/3805027271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd870a00640 1 --2- 172.21.15.67:0/3805027271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c0b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd86be00640 1 --2- 172.21.15.67:0/3805027271 >> v2:172.21.15.86:3300/0 conn(0x7fd86c0b57b0 0x7fd86c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 -- 172.21.15.67:0/3805027271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 msgr2=0x7fd86c0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 --2- 172.21.15.67:0/3805027271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c0b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 -- 172.21.15.67:0/3805027271 >> v2:172.21.15.86:3300/0 conn(0x7fd86c0b57b0 msgr2=0x7fd86c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 --2- 172.21.15.67:0/3805027271 >> v2:172.21.15.86:3300/0 conn(0x7fd86c0b57b0 0x7fd86c070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd871400640 1 -- 172.21.15.67:0/3805027271 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c06bed0 con 0x7fd86c0b07b0 2024-04-17T10:28:38.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.658+0000 7fd870a00640 1 --2- 172.21.15.67:0/3805027271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c0b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:38.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd871400640 1 --2- 172.21.15.67:0/3805027271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c0b07b0 0x7fd86c071300 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd86c068530 tx=0x7fd86401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e65cbfa00cf9dc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd86b400640 1 -- 172.21.15.67:0/3805027271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640a5790 con 0x7fd86c0b07b0 2024-04-17T10:28:38.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd86b400640 1 -- 172.21.15.67:0/3805027271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8640a5930 con 0x7fd86c0b07b0 2024-04-17T10:28:38.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd86b400640 1 -- 172.21.15.67:0/3805027271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640a5d00 con 0x7fd86c0b07b0 2024-04-17T10:28:38.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 -- 172.21.15.67:0/3805027271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c0b07b0 msgr2=0x7fd86c071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 --2- 172.21.15.67:0/3805027271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c0b07b0 0x7fd86c071300 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fd86c068530 tx=0x7fd86401c930 comp rx=0 tx=0).stop 2024-04-17T10:28:38.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 -- 172.21.15.67:0/3805027271 shutdown_connections 2024-04-17T10:28:38.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 --2- 172.21.15.67:0/3805027271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c0b07b0 0x7fd86c071300 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 --2- 172.21.15.67:0/3805027271 >> v2:172.21.15.86:3300/0 conn(0x7fd86c0b57b0 0x7fd86c070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 --2- 172.21.15.67:0/3805027271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c0b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 -- 172.21.15.67:0/3805027271 >> 172.21.15.67:0/3805027271 conn(0x7fd86c0b34d0 msgr2=0x7fd86c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 -- 172.21.15.67:0/3805027271 shutdown_connections 2024-04-17T10:28:38.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.662+0000 7fd873400640 1 -- 172.21.15.67:0/3805027271 wait complete. 2024-04-17T10:28:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 Processor -- start 2024-04-17T10:28:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 -- start start 2024-04-17T10:28:38.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd86c0b07b0 0x7fd86c15e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c152180 0x7fd86c148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c162460 con 0x7fd86c152180 2024-04-17T10:28:38.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c1622e0 con 0x7fd86c0b07b0 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd86c1625e0 con 0x7fd86c15dd10 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c152180 0x7fd86c148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c152180 0x7fd86c148d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36290/0 (socket says 172.21.15.67:36290) 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 -- 172.21.15.67:0/1665600684 learned_addr learned my addr 172.21.15.67:0/1665600684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd870a00640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd86c0b07b0 0x7fd86c15e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd86be00640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 -- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 msgr2=0x7fd86c151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 -- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd86c0b07b0 msgr2=0x7fd86c15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd86c0b07b0 0x7fd86c15e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 -- 172.21.15.67:0/1665600684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0d2540 con 0x7fd86c152180 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd86be00640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd871400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c152180 0x7fd86c148d10 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd8640af040 tx=0x7fd8640196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd868c00640 1 -- 172.21.15.67:0/1665600684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640b8670 con 0x7fd86c152180 2024-04-17T10:28:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd868c00640 1 -- 172.21.15.67:0/1665600684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8640be070 con 0x7fd86c152180 2024-04-17T10:28:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd868c00640 1 -- 172.21.15.67:0/1665600684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640b8a70 con 0x7fd86c152180 2024-04-17T10:28:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86c150ba0 con 0x7fd86c152180 2024-04-17T10:28:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd86c151160 con 0x7fd86c152180 2024-04-17T10:28:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.666+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86c070bb0 con 0x7fd86c152180 2024-04-17T10:28:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.670+0000 7fd868c00640 1 -- 172.21.15.67:0/1665600684 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd8640b8c10 con 0x7fd86c152180 2024-04-17T10:28:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.670+0000 7fd868c00640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd84c066f60 0x7fd84c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.670+0000 7fd868c00640 1 -- 172.21.15.67:0/1665600684 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd8641204b0 con 0x7fd86c152180 2024-04-17T10:28:38.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.670+0000 7fd870a00640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd84c066f60 0x7fd84c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:38.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.670+0000 7fd870a00640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd84c066f60 0x7fd84c069420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd8600045f0 tx=0x7fd860008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.674+0000 7fd868c00640 1 -- 172.21.15.67:0/1665600684 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd8640165e0 con 0x7fd86c152180 2024-04-17T10:28:38.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.874+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd86c074e30 con 0x7fd86c152180 2024-04-17T10:28:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.874+0000 7fd868c00640 1 -- 172.21.15.67:0/1665600684 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+4936 (secure 0 0 0) 0x7fd8640f11d0 con 0x7fd86c152180 2024-04-17T10:28:38.868 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:38.868 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:34.432126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":11,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:38.868 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 30 2024-04-17T10:28:38.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd84c066f60 msgr2=0x7fd84c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd84c066f60 0x7fd84c069420 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd8600045f0 tx=0x7fd860008040 comp rx=0 tx=0).stop 2024-04-17T10:28:38.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c152180 msgr2=0x7fd86c148d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c152180 0x7fd86c148d10 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fd8640af040 tx=0x7fd8640196d0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 shutdown_connections 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd84c066f60 0x7fd84c069420 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd86c152180 0x7fd86c148d10 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd86c15dd10 0x7fd86c151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 --2- 172.21.15.67:0/1665600684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd86c0b07b0 0x7fd86c15e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 >> 172.21.15.67:0/1665600684 conn(0x7fd86c0b34d0 msgr2=0x7fd86c071010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 shutdown_connections 2024-04-17T10:28:38.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:38.878+0000 7fd873400640 1 -- 172.21.15.67:0/1665600684 wait complete. 2024-04-17T10:28:38.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d damage rm 1933588164 2024-04-17T10:28:38.926 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:28:38.934+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-04-17T10:28:39.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 Processor -- start 2024-04-17T10:28:39.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 -- start start 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad781446c0 0x7fad7815be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fad78072a40 0x7fad7806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad7815f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad7806b640 con 0x7fad781446c0 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad7815d060 con 0x7fad78072a40 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7f800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7806ba60 con 0x7fad7806b1d0 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad781446c0 0x7fad7815be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad6fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fad78072a40 0x7fad7806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad781446c0 0x7fad7815be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36294/0 (socket says 172.21.15.67:36294) 2024-04-17T10:28:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad7815f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 -- 172.21.15.67:0/410840148 learned_addr learned my addr 172.21.15.67:0/410840148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 -- 172.21.15.67:0/410840148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 msgr2=0x7fad7815f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 --2- 172.21.15.67:0/410840148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad7815f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 -- 172.21.15.67:0/410840148 >> v2:172.21.15.86:3300/0 conn(0x7fad78072a40 msgr2=0x7fad7806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 --2- 172.21.15.67:0/410840148 >> v2:172.21.15.86:3300/0 conn(0x7fad78072a40 0x7fad7806ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.042+0000 7fad7ce00640 1 -- 172.21.15.67:0/410840148 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7815c450 con 0x7fad781446c0 2024-04-17T10:28:39.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7ce00640 1 --2- 172.21.15.67:0/410840148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad781446c0 0x7fad7815be70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fad780b63d0 tx=0x7fad6801c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee3ecea7021dffe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad77e00640 1 -- 172.21.15.67:0/410840148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680b0070 con 0x7fad781446c0 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad77e00640 1 -- 172.21.15.67:0/410840148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad680ab600 con 0x7fad781446c0 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad77e00640 1 -- 172.21.15.67:0/410840148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680ab9d0 con 0x7fad781446c0 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 -- 172.21.15.67:0/410840148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad781446c0 msgr2=0x7fad7815be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 --2- 172.21.15.67:0/410840148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad781446c0 0x7fad7815be70 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fad780b63d0 tx=0x7fad6801c1d0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 -- 172.21.15.67:0/410840148 shutdown_connections 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 --2- 172.21.15.67:0/410840148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad7815f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 --2- 172.21.15.67:0/410840148 >> v2:172.21.15.86:3300/0 conn(0x7fad78072a40 0x7fad7806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 --2- 172.21.15.67:0/410840148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad781446c0 0x7fad7815be70 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 -- 172.21.15.67:0/410840148 >> 172.21.15.67:0/410840148 conn(0x7fad780b5ef0 msgr2=0x7fad78099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 -- 172.21.15.67:0/410840148 shutdown_connections 2024-04-17T10:28:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.046+0000 7fad7f800640 1 -- 172.21.15.67:0/410840148 wait complete. 2024-04-17T10:28:39.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 Processor -- start 2024-04-17T10:28:39.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 -- start start 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad78155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad78072a40 0x7fad780d0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad781446c0 0x7fad7813d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78144c20 con 0x7fad78072a40 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78144aa0 con 0x7fad781446c0 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad7815d4f0 con 0x7fad7806b1d0 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad78072a40 0x7fad780d0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad78155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad78155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:56298/0 (socket says 172.21.15.67:56298) 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7ce00640 1 -- 172.21.15.67:0/1239798044 learned_addr learned my addr 172.21.15.67:0/1239798044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:39.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7d800640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad781446c0 0x7fad7813d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad6fe00640 1 -- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 msgr2=0x7fad78155cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad78155cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad6fe00640 1 -- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad781446c0 msgr2=0x7fad7813d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad781446c0 0x7fad7813d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad6fe00640 1 -- 172.21.15.67:0/1239798044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad78141a90 con 0x7fad78072a40 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad78072a40 0x7fad780d0750 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e8a0 tx=0x7fad6000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad75600640 1 -- 172.21.15.67:0/1239798044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad60017070 con 0x7fad78072a40 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad75600640 1 -- 172.21.15.67:0/1239798044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6000c8e0 con 0x7fad78072a40 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad75600640 1 -- 172.21.15.67:0/1239798044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6000cc90 con 0x7fad78072a40 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad78140f10 con 0x7fad78072a40 2024-04-17T10:28:39.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.050+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad781414d0 con 0x7fad78072a40 2024-04-17T10:28:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad75600640 1 -- 172.21.15.67:0/1239798044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fad6001a070 con 0x7fad78072a40 2024-04-17T10:28:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad75600640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad54066f60 0x7fad54069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad75600640 1 -- 172.21.15.67:0/1239798044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fad600850a0 con 0x7fad78072a40 2024-04-17T10:28:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 Processor -- start 2024-04-17T10:28:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad54066f60 0x7fad54069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 -- start start 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad54066f60 0x7fad54069420 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fad6801c210 tx=0x7fad68008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b80a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7bd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b7bdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55731b60df00 con 0x55731b80a070 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55731b5964a0 con 0x55731b80a990 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55731b609fa0 con 0x55731b7bdbe0 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b80a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b80a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36318/0 (socket says 172.21.15.67:36318) 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 -- 172.21.15.67:0/317174119 learned_addr learned my addr 172.21.15.67:0/317174119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad6fe00640 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7bd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7d800640 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b7bdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 -- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 msgr2=0x55731b7bdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b7bdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 -- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 msgr2=0x55731b7bd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7bd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad7ce00640 1 -- 172.21.15.67:0/317174119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55731b593d00 con 0x55731b80a070 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.054+0000 7fad6fe00640 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7bd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad7ce00640 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b80a450 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fad68097b50 tx=0x7fad68004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef6fa8b1d99da131 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6d600640 1 -- 172.21.15.67:0/317174119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680c7070 con 0x55731b80a070 2024-04-17T10:28:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6d600640 1 -- 172.21.15.67:0/317174119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad680abcf0 con 0x55731b80a070 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6d600640 1 -- 172.21.15.67:0/317174119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680b0040 con 0x55731b80a070 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- 172.21.15.67:0/317174119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 msgr2=0x55731b80a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b80a450 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fad68097b50 tx=0x7fad68004740 comp rx=0 tx=0).stop 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- 172.21.15.67:0/317174119 shutdown_connections 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b7bdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7bd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 --2- 172.21.15.67:0/317174119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b80a450 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- 172.21.15.67:0/317174119 >> 172.21.15.67:0/317174119 conn(0x55731b7eae10 msgr2=0x55731b7ed200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- 172.21.15.67:0/317174119 shutdown_connections 2024-04-17T10:28:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- 172.21.15.67:0/317174119 wait complete. 2024-04-17T10:28:39.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- start start 2024-04-17T10:28:39.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 Processor -- start 2024-04-17T10:28:39.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b815410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b815950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7ef0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55731b612530 con 0x55731b80a070 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55731b5c8ae0 con 0x55731b80a990 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55731b5cce10 con 0x55731b7bdbe0 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad82d77000 10 client.0 _ll_register_callbacks cb 0x55731b7ab750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b815950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b815950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36322/0 (socket says 172.21.15.67:36322) 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 -- 172.21.15.67:0/1001370924 learned_addr learned my addr 172.21.15.67:0/1001370924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b815410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad7d800640 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7ef0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 -- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 msgr2=0x55731b815410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b815410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 -- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 msgr2=0x55731b7ef0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7ef0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.058+0000 7fad6fe00640 1 -- 172.21.15.67:0/1001370924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55731b5b0a10 con 0x55731b80a070 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b815950 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fad6000f010 tx=0x7fad6005df30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20852ddd8cf9043 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 1 -- 172.21.15.67:0/1001370924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6006d020 con 0x55731b80a070 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 1 -- 172.21.15.67:0/1001370924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fad6006ed60 con 0x55731b80a070 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad680abcf0 con 0x55731b80a070 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 1 -- 172.21.15.67:0/1001370924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad60021040 con 0x55731b80a070 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 10 client.6438 fetch_fsmap learned FSMap version 31 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x55731b608ef0 con 0x55731b80a070 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 1 -- 172.21.15.67:0/1001370924 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad680abcf0 con 0x55731b80a070 2024-04-17T10:28:39.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 1 -- 172.21.15.67:0/1001370924 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fad60024060 con 0x55731b80a070 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 10 client.6438 fetch_fsmap finished waiting for FSMap version 31 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 10 client.6438 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 20 client.6438 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b7efe40 0x55731b7f0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 4 client.6438 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55731b5a9b20 con 0x55731b7efe40 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b7efe40 0x55731b7f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 10 client.6438 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b7efe40 0x55731b7f0220 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfa251b9b4c143a6 server_cookie=e1d0cc889f08b2b9 in_seq=0 out_seq=0 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 1 -- 172.21.15.67:0/1001370924 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55731b5a9b20 con 0x55731b7efe40 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 10 client.6438 handle_command_reply: tid=0 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 client.6438 shutdown 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 msgr2=0x55731b815950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b815950 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fad6000f010 tx=0x7fad6005df30 comp rx=0 tx=0).stop 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 shutdown_connections 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b7efe40 0x55731b7f0220 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b7ef0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b80a070 0x55731b815950 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad66000640 0 client.6438 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 --2- 172.21.15.67:0/1001370924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b7bdbe0 0x55731b815410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 >> 172.21.15.67:0/1001370924 conn(0x55731b7eae10 msgr2=0x55731b7bcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 shutdown_connections 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 1 -- 172.21.15.67:0/1001370924 wait complete. 2024-04-17T10:28:39.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.062+0000 7fad82d77000 20 client.6438 trim_cache size 0 max 16384 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.074+0000 7fad82d77000 1 Processor -- start 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.074+0000 7fad82d77000 1 -- start start 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a070 0x55731b86ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b86b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad78157ce0 con 0x55731b7bdbe0 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55731b5dab90 con 0x55731b80a990 2024-04-17T10:28:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55731b7bd750 con 0x55731b80a070 2024-04-17T10:28:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36326/0 (socket says 172.21.15.67:36326) 2024-04-17T10:28:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 -- 172.21.15.67:0/1128841229 learned_addr learned my addr 172.21.15.67:0/1128841229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7d800640 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b86b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a070 0x55731b86ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 -- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a070 msgr2=0x55731b86ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a070 0x55731b86ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 -- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 msgr2=0x55731b86b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b86b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 -- 172.21.15.67:0/1128841229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55731b80a7c0 con 0x55731b7bdbe0 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad6fe00640 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a070 0x55731b86ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad7ce00640 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86a6f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fad680c5040 tx=0x7fad68009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efddfeec6fa27e0b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad6cc00640 1 -- 172.21.15.67:0/1128841229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680c7040 con 0x55731b7bdbe0 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad6cc00640 1 -- 172.21.15.67:0/1128841229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad680d0170 con 0x55731b7bdbe0 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad6cc00640 1 -- 172.21.15.67:0/1128841229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680b0040 con 0x55731b7bdbe0 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- 172.21.15.67:0/1128841229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 msgr2=0x55731b86a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86a6f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fad680c5040 tx=0x7fad68009240 comp rx=0 tx=0).stop 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- 172.21.15.67:0/1128841229 shutdown_connections 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a990 0x55731b86b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a070 0x55731b86ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 --2- 172.21.15.67:0/1128841229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86a6f0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- 172.21.15.67:0/1128841229 >> 172.21.15.67:0/1128841229 conn(0x55731b7eae10 msgr2=0x55731b7eee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- 172.21.15.67:0/1128841229 shutdown_connections 2024-04-17T10:28:39.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- 172.21.15.67:0/1128841229 wait complete. 2024-04-17T10:28:39.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 -- start start 2024-04-17T10:28:39.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.078+0000 7fad82d77000 1 Processor -- start 2024-04-17T10:28:39.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a070 0x55731b86f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a990 0x55731b81a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55731b64dca0 con 0x55731b7bdbe0 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55731b7c91c0 con 0x55731b80a070 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55731b5cb700 con 0x55731b80a990 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 10 client.0 _ll_register_callbacks cb 0x7fad60009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36332/0 (socket says 172.21.15.67:36332) 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 -- 172.21.15.67:0/550496771 learned_addr learned my addr 172.21.15.67:0/550496771 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad6fe00640 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a070 0x55731b86f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7d800640 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a990 0x55731b81a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 -- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a990 msgr2=0x55731b81a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a990 0x55731b81a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 -- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a070 msgr2=0x55731b86f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a070 0x55731b86f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 -- 172.21.15.67:0/550496771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55731b5cd790 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad7ce00640 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86f070 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fad6801a880 tx=0x7fad6801aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a45a5eb9c0acb5b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 1 -- 172.21.15.67:0/550496771 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad680c7070 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 1 -- 172.21.15.67:0/550496771 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fad680d0320 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fad680d0170 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 1 -- 172.21.15.67:0/550496771 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fad680b0040 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 10 client.6444 fetch_fsmap learned FSMap version 31 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=2+}) v3 -- 0x55731b5d5da0 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 1 -- 172.21.15.67:0/550496771 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad680d0170 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 1 -- 172.21.15.67:0/550496771 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 31) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fad680d0520 con 0x55731b7bdbe0 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 10 client.6444 fetch_fsmap finished waiting for FSMap version 31 2024-04-17T10:28:39.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 10 client.6444 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 20 client.6444 populate_metadata read hostname 'smithi067' 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b81ac40 0x55731b81b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 4 client.6444 mds_command: new command op to 5772 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 1933588164}] 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 --> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] -- command(tid 0: {"prefix": "damage rm", "damage_id": 1933588164}) v1 -- 0x55731b61b440 con 0x55731b81ac40 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad6fe00640 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b81ac40 0x55731b81b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad6fe00640 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b81ac40 0x55731b81b020 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66b7b76ab3f835e4 server_cookie=978f6f38b562b87c in_seq=0 out_seq=0 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 10 client.6444 ms_handle_connect on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 1 -- 172.21.15.67:0/550496771 <== mds.0 v2:172.21.15.86:6835/2048306700 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x55731b61b440 con 0x55731b81ac40 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 10 client.6444 handle_command_reply: tid=0 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 client.6444 shutdown 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 msgr2=0x55731b86f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86f070 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fad6801a880 tx=0x7fad6801aa10 comp rx=0 tx=0).stop 2024-04-17T10:28:39.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 shutdown_connections 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:6835/2048306700,v1:172.21.15.86:6837/2048306700] conn(0x55731b81ac40 0x55731b81b020 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55731b80a990 0x55731b81a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55731b80a070 0x55731b86f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad66000640 0 client.6444 ms_handle_reset on v2:172.21.15.86:6835/2048306700 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 --2- 172.21.15.67:0/550496771 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55731b7bdbe0 0x55731b86f070 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 >> 172.21.15.67:0/550496771 conn(0x55731b7eae10 msgr2=0x55731b7bbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 shutdown_connections 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 1 -- 172.21.15.67:0/550496771 wait complete. 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.082+0000 7fad82d77000 20 client.6444 trim_cache size 0 max 16384 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad54066f60 msgr2=0x7fad54069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad54066f60 0x7fad54069420 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fad6801c210 tx=0x7fad68008040 comp rx=0 tx=0).stop 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad78072a40 msgr2=0x7fad780d0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad78072a40 0x7fad780d0750 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fad6000e8a0 tx=0x7fad6000ed60 comp rx=0 tx=0).stop 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 shutdown_connections 2024-04-17T10:28:39.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad54066f60 0x7fad54069420 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad781446c0 0x7fad7813d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad78072a40 0x7fad780d0750 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 --2- 172.21.15.67:0/1239798044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad7806b1d0 0x7fad78155cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 >> 172.21.15.67:0/1239798044 conn(0x7fad780b5ef0 msgr2=0x7fad781433b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 shutdown_connections 2024-04-17T10:28:39.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.086+0000 7fad7f800640 1 -- 172.21.15.67:0/1239798044 wait complete. 2024-04-17T10:28:39.096 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /' 2024-04-17T10:28:39.097 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe89bc00640 1 Processor -- start 2024-04-17T10:28:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe89bc00640 1 -- start start 2024-04-17T10:28:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe89bc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe89406eac0 0x7fe894143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe89bc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89415ae40 con 0x7fe8940cf0c0 2024-04-17T10:28:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe89bc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe894071440 con 0x7fe89406e6e0 2024-04-17T10:28:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe89bc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe894071720 con 0x7fe89406eac0 2024-04-17T10:28:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe899c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe8940cf0c0 0x7fe894144020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56204/0 (socket says 172.21.15.67:56204) 2024-04-17T10:28:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe899c00640 1 -- 172.21.15.67:0/2010757464 learned_addr learned my addr 172.21.15.67:0/2010757464 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:39.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe893e00640 1 --2- 172.21.15.67:0/2010757464 >> v2:172.21.15.86:3300/0 conn(0x7fe89406eac0 0x7fe894143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383897313 0 0) 0x7fe89415ae40 con 0x7fe8940cf0c0 2024-04-17T10:28:39.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.286+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c003530 con 0x7fe8940cf0c0 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 86629981 0 0) 0x7fe87c003530 con 0x7fe8940cf0c0 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 >> v2:172.21.15.86:3300/0 conn(0x7fe89406eac0 msgr2=0x7fe894143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 --2- 172.21.15.67:0/2010757464 >> v2:172.21.15.86:3300/0 conn(0x7fe89406eac0 0x7fe894143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 >> v1:172.21.15.86:6789/0 conn(0x7fe89406e6e0 legacy=0x7fe8940ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894071a00 con 0x7fe8940cf0c0 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fe88c001a00 con 0x7fe8940cf0c0 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 >> v1:172.21.15.67:6789/0 conn(0x7fe8940cf0c0 legacy=0x7fe894144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c0039c0 0x7fe87c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c004810 0x7fe87c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe87c0051d0 0x7fe87c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe894071720 con 0x7fe87c0039c0 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c00c170 con 0x7fe87c004810 2024-04-17T10:28:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe87c00c370 con 0x7fe87c0051d0 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899200640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c0039c0 0x7fe87c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe893e00640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c004810 0x7fe87c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899c00640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe87c0051d0 0x7fe87c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899200640 1 -- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe87c0051d0 msgr2=0x7fe87c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899200640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe87c0051d0 0x7fe87c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899200640 1 -- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c004810 msgr2=0x7fe87c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899200640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c004810 0x7fe87c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899200640 1 -- 172.21.15.67:0/2010757464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c003530 con 0x7fe87c0039c0 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899c00640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe87c0051d0 0x7fe87c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899c00640 1 -- 172.21.15.67:0/2010757464 reap_dead start 2024-04-17T10:28:39.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.290+0000 7fe899200640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c0039c0 0x7fe87c003de0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe888001780 tx=0x7fe88801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2ed7248ba2128d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880013a0 con 0x7fe87c0039c0 2024-04-17T10:28:39.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8880052f0 con 0x7fe87c0039c0 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe893400640 1 -- 172.21.15.67:0/2010757464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8880a4e60 con 0x7fe87c0039c0 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- 172.21.15.67:0/2010757464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c0039c0 msgr2=0x7fe87c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c0039c0 0x7fe87c003de0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe888001780 tx=0x7fe88801c3d0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- 172.21.15.67:0/2010757464 shutdown_connections 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 --2- 172.21.15.67:0/2010757464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe87c0039c0 0x7fe87c003de0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- 172.21.15.67:0/2010757464 >> 172.21.15.67:0/2010757464 conn(0x7fe89406ac90 msgr2=0x7fe89406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- 172.21.15.67:0/2010757464 shutdown_connections 2024-04-17T10:28:39.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- 172.21.15.67:0/2010757464 wait complete. 2024-04-17T10:28:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 Processor -- start 2024-04-17T10:28:39.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- start start 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c0039c0 0x7fe8941eba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8941ed110 0x7fe8940b1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89415a2b0 0x7fe8940aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe894144470 con 0x7fe89415a2b0 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8941442f0 con 0x7fe87c0039c0 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe89bc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89415e3e0 con 0x7fe8941ed110 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe899c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89415a2b0 0x7fe8940aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe899c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89415a2b0 0x7fe8940aeaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36344/0 (socket says 172.21.15.67:36344) 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.294+0000 7fe899c00640 1 -- 172.21.15.67:0/4023325512 learned_addr learned my addr 172.21.15.67:0/4023325512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899200640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c0039c0 0x7fe8941eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe893e00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8941ed110 0x7fe8940b1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899c00640 1 -- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8941ed110 msgr2=0x7fe8940b1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899c00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8941ed110 0x7fe8940b1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899c00640 1 -- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c0039c0 msgr2=0x7fe8941eba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899c00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c0039c0 0x7fe8941eba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899c00640 1 -- 172.21.15.67:0/4023325512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8941ec070 con 0x7fe89415a2b0 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899200640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c0039c0 0x7fe8941eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe899c00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89415a2b0 0x7fe8940aeaf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe8940ce0d0 tx=0x7fe88c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe890c00640 1 -- 172.21.15.67:0/4023325512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c00eb90 con 0x7fe89415a2b0 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe890c00640 1 -- 172.21.15.67:0/4023325512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c018070 con 0x7fe89415a2b0 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8941ec350 con 0x7fe89415a2b0 2024-04-17T10:28:39.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe890c00640 1 -- 172.21.15.67:0/4023325512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c013420 con 0x7fe89415a2b0 2024-04-17T10:28:39.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8941ec910 con 0x7fe89415a2b0 2024-04-17T10:28:39.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe894072d50 con 0x7fe89415a2b0 2024-04-17T10:28:39.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe890c00640 1 -- 172.21.15.67:0/4023325512 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe88c0135c0 con 0x7fe89415a2b0 2024-04-17T10:28:39.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe890c00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe874066f90 0x7fe874069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:39.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.298+0000 7fe890c00640 1 -- 172.21.15.67:0/4023325512 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe88c0855d0 con 0x7fe89415a2b0 2024-04-17T10:28:39.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.302+0000 7fe899200640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe874066f90 0x7fe874069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:39.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.302+0000 7fe899200640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe874066f90 0x7fe874069450 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe888001750 tx=0x7fe8880011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:39.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.306+0000 7fe890c00640 1 -- 172.21.15.67:0/4023325512 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe88c056380 con 0x7fe89415a2b0 2024-04-17T10:28:39.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.498+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe8941434e0 con 0x7fe89415a2b0 2024-04-17T10:28:39.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe890c00640 1 -- 172.21.15.67:0/4023325512 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4936 (secure 0 0 0) 0x7fe88c055c80 con 0x7fe89415a2b0 2024-04-17T10:28:39.494 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:39.494 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:38.431605+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:39.495 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe874066f90 msgr2=0x7fe874069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe874066f90 0x7fe874069450 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe888001750 tx=0x7fe8880011a0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89415a2b0 msgr2=0x7fe8940aeaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89415a2b0 0x7fe8940aeaf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe8940ce0d0 tx=0x7fe88c00e530 comp rx=0 tx=0).stop 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 shutdown_connections 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe874066f90 0x7fe874069450 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe89415a2b0 0x7fe8940aeaf0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe8941ed110 0x7fe8940b1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 --2- 172.21.15.67:0/4023325512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe87c0039c0 0x7fe8941eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:39.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 >> 172.21.15.67:0/4023325512 conn(0x7fe89406ac90 msgr2=0x7fe8940ad9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 shutdown_connections 2024-04-17T10:28:39.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:39.502+0000 7fe89bc00640 1 -- 172.21.15.67:0/4023325512 wait complete. 2024-04-17T10:28:39.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok flush_path / 2024-04-17T10:28:39.667 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-04-17T10:28:39.667 INFO:teuthology.orchestra.run.smithi086.stdout: "return_code": 0 2024-04-17T10:28:39.667 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-04-17T10:28:39.683 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /' got response code '0' and stdout '{ "return_code": 0 } ' 2024-04-17T10:28:39.683 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:28:39.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:28:39.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:28:39.817 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:28:39.818 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:39.818 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:39.829 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:39.829 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:39.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f71a0a00640 1 Processor -- start 2024-04-17T10:28:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f71a0a00640 1 -- start start 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f71a0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c06c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f71a0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f719c0b3520 0x7f719c0b39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f71a0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f719c06c660 con 0x7f719c158260 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f71a0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f719c0ae590 con 0x7f719c0b3520 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f71a0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719c0adcf0 con 0x7f719c06bca0 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f7199e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f719c158260 0x7f719c0ad2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56216/0 (socket says 172.21.15.67:56216) 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f7199e00640 1 -- 172.21.15.67:0/3351494993 learned_addr learned my addr 172.21.15.67:0/3351494993 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f719a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f719c0b3520 0x7f719c0b39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f7191400640 1 --2- 172.21.15.67:0/3351494993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c06c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f719a800640 1 -- 172.21.15.67:0/3351494993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 msgr2=0x7f719c06c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f719a800640 1 --2- 172.21.15.67:0/3351494993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c06c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f719a800640 1 -- 172.21.15.67:0/3351494993 >> v1:172.21.15.67:6789/0 conn(0x7f719c158260 legacy=0x7f719c0ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f719a800640 1 -- 172.21.15.67:0/3351494993 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c0adfd0 con 0x7f719c0b3520 2024-04-17T10:28:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.070+0000 7f7199400640 1 -- 172.21.15.67:0/3351494993 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056483104 0 0) 0x7f719c06c660 con 0x7f719c158260 2024-04-17T10:28:40.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f719a800640 1 --2- 172.21.15.67:0/3351494993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0b39b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7194009d90 tx=0x7f719401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d0d0f251488fdfb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f7199400640 1 -- 172.21.15.67:0/3351494993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940b4070 con 0x7f719c0b3520 2024-04-17T10:28:40.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f7199400640 1 -- 172.21.15.67:0/3351494993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71940af600 con 0x7f719c0b3520 2024-04-17T10:28:40.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f7199400640 1 -- 172.21.15.67:0/3351494993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71940af9d0 con 0x7f719c0b3520 2024-04-17T10:28:40.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 -- 172.21.15.67:0/3351494993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 msgr2=0x7f719c0b39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 --2- 172.21.15.67:0/3351494993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0b39b0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f7194009d90 tx=0x7f719401f150 comp rx=0 tx=0).stop 2024-04-17T10:28:40.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 -- 172.21.15.67:0/3351494993 shutdown_connections 2024-04-17T10:28:40.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 --2- 172.21.15.67:0/3351494993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0b39b0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 --2- 172.21.15.67:0/3351494993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c06c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 -- 172.21.15.67:0/3351494993 >> 172.21.15.67:0/3351494993 conn(0x7f719c071c20 msgr2=0x7f719c072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:40.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 -- 172.21.15.67:0/3351494993 shutdown_connections 2024-04-17T10:28:40.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.074+0000 7f71a0a00640 1 -- 172.21.15.67:0/3351494993 wait complete. 2024-04-17T10:28:40.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 Processor -- start 2024-04-17T10:28:40.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 -- start start 2024-04-17T10:28:40.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f719c158260 0x7f719c14f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c0ad9b0 con 0x7f719c0b3520 2024-04-17T10:28:40.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719c0b3000 con 0x7f719c158260 2024-04-17T10:28:40.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f719c072bf0 con 0x7f719c06bca0 2024-04-17T10:28:40.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36366/0 (socket says 172.21.15.67:36366) 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 -- 172.21.15.67:0/1094346913 learned_addr learned my addr 172.21.15.67:0/1094346913 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7199e00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f719a800640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f719c158260 0x7f719c14f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 -- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 msgr2=0x7f719c151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 -- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f719c158260 msgr2=0x7f719c14f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f719c158260 0x7f719c14f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 -- 172.21.15.67:0/1094346913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f719c0cd250 con 0x7f719c0b3520 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f719a800640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f719c158260 0x7f719c14f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7191400640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0cdfd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f718400e930 tx=0x7f718400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7192a00640 1 -- 172.21.15.67:0/1094346913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7184017070 con 0x7f719c0b3520 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7192a00640 1 -- 172.21.15.67:0/1094346913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718400ca00 con 0x7f719c0b3520 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f7192a00640 1 -- 172.21.15.67:0/1094346913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718400cdb0 con 0x7f719c0b3520 2024-04-17T10:28:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f719c0cd4f0 con 0x7f719c0b3520 2024-04-17T10:28:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719c0cdab0 con 0x7f719c0b3520 2024-04-17T10:28:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.078+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f719c09e180 con 0x7f719c0b3520 2024-04-17T10:28:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.082+0000 7f7192a00640 1 -- 172.21.15.67:0/1094346913 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f718401a070 con 0x7f719c0b3520 2024-04-17T10:28:40.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.082+0000 7f7192a00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7178066f60 0x7f7178069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.082+0000 7f7192a00640 1 -- 172.21.15.67:0/1094346913 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7184085230 con 0x7f719c0b3520 2024-04-17T10:28:40.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.082+0000 7f7199e00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7178066f60 0x7f7178069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.082+0000 7f7199e00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7178066f60 0x7f7178069420 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f718c000f70 tx=0x7f718c007c10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.086+0000 7f7192a00640 1 -- 172.21.15.67:0/1094346913 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7184056130 con 0x7f719c0b3520 2024-04-17T10:28:40.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f719c074e30 con 0x7f719c0b3520 2024-04-17T10:28:40.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f7192a00640 1 -- 172.21.15.67:0/1094346913 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16934 (secure 0 0 0) 0x7f7184055a30 con 0x7f719c0b3520 2024-04-17T10:28:40.243 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:40.243 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:37.929771+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:00.190460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:00.765109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.27","mappings":[{"from":0,"to":6}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7178066f60 msgr2=0x7f7178069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7178066f60 0x7f7178069420 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f718c000f70 tx=0x7f718c007c10 comp rx=0 tx=0).stop 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 msgr2=0x7f719c0cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0cdfd0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f718400e930 tx=0x7f718400edf0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 shutdown_connections 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7178066f60 0x7f7178069420 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f719c158260 0x7f719c14f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f719c0b3520 0x7f719c0cdfd0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 --2- 172.21.15.67:0/1094346913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f719c06bca0 0x7f719c151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 >> 172.21.15.67:0/1094346913 conn(0x7f719c071c20 msgr2=0x7f719c1528d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 shutdown_connections 2024-04-17T10:28:40.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.250+0000 7f71a0a00640 1 -- 172.21.15.67:0/1094346913 wait complete. 2024-04-17T10:28:40.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:28:00.190460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:28:00.765109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:28:40.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:28:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ef1600640 1 Processor -- start 2024-04-17T10:28:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ef1600640 1 -- start start 2024-04-17T10:28:40.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ef1600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9eec0cd030 0x7f9eec06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ef1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9eec06e7b0 0x7f9eec0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ef1600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eec0b4990 con 0x7f9eec0aef60 2024-04-17T10:28:40.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ef1600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec0af340 con 0x7f9eec06e7b0 2024-04-17T10:28:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ef1600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9eec06ec00 con 0x7f9eec0cd030 2024-04-17T10:28:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eeaa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9eec0aef60 0x7f9eec0cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46086/0 (socket says 172.21.15.67:46086) 2024-04-17T10:28:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eeaa00640 1 -- 172.21.15.67:0/2381766432 learned_addr learned my addr 172.21.15.67:0/2381766432 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eeb400640 1 --2- 172.21.15.67:0/2381766432 >> v2:172.21.15.67:3300/0 conn(0x7f9eec06e7b0 0x7f9eec0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9ee3e00640 1 --2- 172.21.15.67:0/2381766432 >> v2:172.21.15.86:3300/0 conn(0x7f9eec0cd030 0x7f9eec06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eea000640 1 -- 172.21.15.67:0/2381766432 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620222255 0 0) 0x7f9edc001120 con 0x7f9eec0aef60 2024-04-17T10:28:40.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eea000640 1 -- 172.21.15.67:0/2381766432 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed0003530 con 0x7f9eec0aef60 2024-04-17T10:28:40.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eeb400640 1 -- 172.21.15.67:0/2381766432 >> v2:172.21.15.86:3300/0 conn(0x7f9eec0cd030 msgr2=0x7f9eec06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:40.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eeb400640 1 --2- 172.21.15.67:0/2381766432 >> v2:172.21.15.86:3300/0 conn(0x7f9eec0cd030 0x7f9eec06e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eeb400640 1 -- 172.21.15.67:0/2381766432 >> v1:172.21.15.67:6789/0 conn(0x7f9eec0aef60 legacy=0x7f9eec0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.418+0000 7f9eeb400640 1 -- 172.21.15.67:0/2381766432 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec15ae90 con 0x7f9eec06e7b0 2024-04-17T10:28:40.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9eeb400640 1 --2- 172.21.15.67:0/2381766432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec06e7b0 0x7f9eec0b3120 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9eec15e2b0 tx=0x7f9ee401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9351d7f3f72cab30 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9eea000640 1 -- 172.21.15.67:0/2381766432 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee40b2070 con 0x7f9eec06e7b0 2024-04-17T10:28:40.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9eea000640 1 -- 172.21.15.67:0/2381766432 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ee40a5850 con 0x7f9eec06e7b0 2024-04-17T10:28:40.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9eea000640 1 -- 172.21.15.67:0/2381766432 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee40a5c00 con 0x7f9eec06e7b0 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 -- 172.21.15.67:0/2381766432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec06e7b0 msgr2=0x7f9eec0b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 --2- 172.21.15.67:0/2381766432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec06e7b0 0x7f9eec0b3120 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f9eec15e2b0 tx=0x7f9ee401c8a0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 -- 172.21.15.67:0/2381766432 shutdown_connections 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 --2- 172.21.15.67:0/2381766432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec06e7b0 0x7f9eec0b3120 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 --2- 172.21.15.67:0/2381766432 >> v2:172.21.15.86:3300/0 conn(0x7f9eec0cd030 0x7f9eec06e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 -- 172.21.15.67:0/2381766432 >> 172.21.15.67:0/2381766432 conn(0x7f9eec0b1740 msgr2=0x7f9eec0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 -- 172.21.15.67:0/2381766432 shutdown_connections 2024-04-17T10:28:40.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 -- 172.21.15.67:0/2381766432 wait complete. 2024-04-17T10:28:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 Processor -- start 2024-04-17T10:28:40.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.422+0000 7f9ef1600640 1 -- start start 2024-04-17T10:28:40.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eec06e7b0 0x7f9eec148d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec0aef60 0x7f9eec151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eec0cd030 0x7f9eec159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec15bc10 con 0x7f9eec0aef60 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec0b3e70 con 0x7f9eec06e7b0 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec1526d0 con 0x7f9eec0cd030 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec0aef60 0x7f9eec151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec0aef60 0x7f9eec151d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44052/0 (socket says 172.21.15.67:44052) 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 -- 172.21.15.67:0/3626287008 learned_addr learned my addr 172.21.15.67:0/3626287008 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9eeaa00640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eec06e7b0 0x7f9eec148d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9eeb400640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eec0cd030 0x7f9eec159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 -- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eec0cd030 msgr2=0x7f9eec159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eec0cd030 0x7f9eec159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 -- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eec06e7b0 msgr2=0x7f9eec148d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eec06e7b0 0x7f9eec148d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 -- 172.21.15.67:0/3626287008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec1529b0 con 0x7f9eec0aef60 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9eeb400640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eec0cd030 0x7f9eec159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee3e00640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec0aef60 0x7f9eec151d60 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e840 tx=0x7f9ed400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee2a00640 1 -- 172.21.15.67:0/3626287008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4017070 con 0x7f9eec0aef60 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee2a00640 1 -- 172.21.15.67:0/3626287008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed400ee90 con 0x7f9eec0aef60 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ee2a00640 1 -- 172.21.15.67:0/3626287008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400cb80 con 0x7f9eec0aef60 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec152c90 con 0x7f9eec0aef60 2024-04-17T10:28:40.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec153250 con 0x7f9eec0aef60 2024-04-17T10:28:40.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.426+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec04ef30 con 0x7f9eec0aef60 2024-04-17T10:28:40.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.430+0000 7f9ee2a00640 1 -- 172.21.15.67:0/3626287008 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9ed401a070 con 0x7f9eec0aef60 2024-04-17T10:28:40.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.430+0000 7f9ee2a00640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ec8066f60 0x7f9ec8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.430+0000 7f9ee2a00640 1 -- 172.21.15.67:0/3626287008 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9ed4085390 con 0x7f9eec0aef60 2024-04-17T10:28:40.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.430+0000 7f9eeaa00640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ec8066f60 0x7f9ec8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.430+0000 7f9eeaa00640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ec8066f60 0x7f9ec8069420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9edc005890 tx=0x7f9edc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.434+0000 7f9ee2a00640 1 -- 172.21.15.67:0/3626287008 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9ed4055b80 con 0x7f9eec0aef60 2024-04-17T10:28:40.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.594+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9eec0b1c30 con 0x7f9eec0aef60 2024-04-17T10:28:40.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.594+0000 7f9ee2a00640 1 -- 172.21.15.67:0/3626287008 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ed4059c20 con 0x7f9eec0aef60 2024-04-17T10:28:40.589 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ec8066f60 msgr2=0x7f9ec8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ec8066f60 0x7f9ec8069420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9edc005890 tx=0x7f9edc009040 comp rx=0 tx=0).stop 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec0aef60 msgr2=0x7f9eec151d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec0aef60 0x7f9eec151d60 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e840 tx=0x7f9ed400ec40 comp rx=0 tx=0).stop 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 shutdown_connections 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ec8066f60 0x7f9ec8069420 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9eec0cd030 0x7f9eec159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9eec0aef60 0x7f9eec151d60 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 --2- 172.21.15.67:0/3626287008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9eec06e7b0 0x7f9eec148d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 >> 172.21.15.67:0/3626287008 conn(0x7f9eec0b1740 msgr2=0x7f9eec0af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:40.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 shutdown_connections 2024-04-17T10:28:40.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.598+0000 7f9ef1600640 1 -- 172.21.15.67:0/3626287008 wait complete. 2024-04-17T10:28:40.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:28:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.758+0000 7f9911c00640 1 Processor -- start 2024-04-17T10:28:40.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f9911c00640 1 -- start start 2024-04-17T10:28:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f9911c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c1474c0 0x7f990c0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f9911c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990c1616a0 con 0x7f990c0b3920 2024-04-17T10:28:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f9911c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f990c161940 con 0x7f990c06b3a0 2024-04-17T10:28:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f9911c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f990c161c20 con 0x7f990c1474c0 2024-04-17T10:28:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f990aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c1474c0 0x7f990c0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f9903e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f990c0b3920 0x7f990c06ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46102/0 (socket says 172.21.15.67:46102) 2024-04-17T10:28:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f990aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f990c1474c0 0x7f990c0b1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44062/0 (socket says 172.21.15.67:44062) 2024-04-17T10:28:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f9903e00640 1 -- 172.21.15.67:0/3274546052 learned_addr learned my addr 172.21.15.67:0/3274546052 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f990aa00640 1 -- 172.21.15.67:0/3274546052 >> v1:172.21.15.86:6789/0 conn(0x7f990c06b3a0 legacy=0x7f990c0d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f990aa00640 1 -- 172.21.15.67:0/3274546052 >> v1:172.21.15.67:6789/0 conn(0x7f990c0b3920 legacy=0x7f990c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.762+0000 7f990aa00640 1 -- 172.21.15.67:0/3274546052 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c161f00 con 0x7f990c1474c0 2024-04-17T10:28:40.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f990aa00640 1 --2- 172.21.15.67:0/3274546052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c1474c0 0x7f990c0b1b90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0097c0 tx=0x7f98fc01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd381b44882f524b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f990a000640 1 -- 172.21.15.67:0/3274546052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0b5070 con 0x7f990c1474c0 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f990a000640 1 -- 172.21.15.67:0/3274546052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98fc0a8990 con 0x7f990c1474c0 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f990a000640 1 -- 172.21.15.67:0/3274546052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98fc0a8d60 con 0x7f990c1474c0 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 -- 172.21.15.67:0/3274546052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c1474c0 msgr2=0x7f990c0b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 --2- 172.21.15.67:0/3274546052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c1474c0 0x7f990c0b1b90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0097c0 tx=0x7f98fc01ec90 comp rx=0 tx=0).stop 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 -- 172.21.15.67:0/3274546052 shutdown_connections 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 --2- 172.21.15.67:0/3274546052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c1474c0 0x7f990c0b1b90 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 -- 172.21.15.67:0/3274546052 >> 172.21.15.67:0/3274546052 conn(0x7f990c0b5ef0 msgr2=0x7f990c099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 -- 172.21.15.67:0/3274546052 shutdown_connections 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 -- 172.21.15.67:0/3274546052 wait complete. 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 Processor -- start 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.766+0000 7f9911c00640 1 -- start start 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f990c06b3a0 0x7f990c0c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0b3920 0x7f990c0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f990c1474c0 0x7f990c09e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c162420 con 0x7f990c0b3920 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c0b3d70 con 0x7f990c06b3a0 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c0d0b90 con 0x7f990c1474c0 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0b3920 0x7f990c0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0b3920 0x7f990c0ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44072/0 (socket says 172.21.15.67:44072) 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 -- 172.21.15.67:0/3428823068 learned_addr learned my addr 172.21.15.67:0/3428823068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f990aa00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f990c06b3a0 0x7f990c0c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f990b400640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f990c1474c0 0x7f990c09e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 -- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f990c1474c0 msgr2=0x7f990c09e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f990c1474c0 0x7f990c09e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 -- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f990c06b3a0 msgr2=0x7f990c0c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f990c06b3a0 0x7f990c0c9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 -- 172.21.15.67:0/3428823068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c09cf60 con 0x7f990c0b3920 2024-04-17T10:28:40.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9903e00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0b3920 0x7f990c0ca7a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f98f400c530 tx=0x7f98f400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9902a00640 1 -- 172.21.15.67:0/3428823068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4017070 con 0x7f990c0b3920 2024-04-17T10:28:40.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9902a00640 1 -- 172.21.15.67:0/3428823068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f4005cf0 con 0x7f990c0b3920 2024-04-17T10:28:40.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9902a00640 1 -- 172.21.15.67:0/3428823068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f4007270 con 0x7f990c0b3920 2024-04-17T10:28:40.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c09d240 con 0x7f990c0b3920 2024-04-17T10:28:40.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c09d800 con 0x7f990c0b3920 2024-04-17T10:28:40.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f990c04ef30 con 0x7f990c0b3920 2024-04-17T10:28:40.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.770+0000 7f9902a00640 1 -- 172.21.15.67:0/3428823068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f98f401a070 con 0x7f990c0b3920 2024-04-17T10:28:40.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.774+0000 7f9902a00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f98e8066f60 0x7f98e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:40.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.774+0000 7f9902a00640 1 -- 172.21.15.67:0/3428823068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f98f40845b0 con 0x7f990c0b3920 2024-04-17T10:28:40.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.774+0000 7f990aa00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f98e8066f60 0x7f98e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:40.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.774+0000 7f990aa00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f98e8066f60 0x7f98e8069420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0b2000 tx=0x7f98fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:40.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.778+0000 7f9902a00640 1 -- 172.21.15.67:0/3428823068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f98f4054c80 con 0x7f990c0b3920 2024-04-17T10:28:40.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f990c0727e0 con 0x7f990c0b3920 2024-04-17T10:28:40.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9902a00640 1 -- 172.21.15.67:0/3428823068 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7f98f4058d20 con 0x7f990c0b3920 2024-04-17T10:28:40.930 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:28:40.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f98e8066f60 msgr2=0x7f98e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f98e8066f60 0x7f98e8069420 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f98fc0b2000 tx=0x7f98fc008040 comp rx=0 tx=0).stop 2024-04-17T10:28:40.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0b3920 msgr2=0x7f990c0ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:40.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0b3920 0x7f990c0ca7a0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f98f400c530 tx=0x7f98f400c9f0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 shutdown_connections 2024-04-17T10:28:40.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f98e8066f60 0x7f98e8069420 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f990c1474c0 0x7f990c09e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f990c0b3920 0x7f990c0ca7a0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 --2- 172.21.15.67:0/3428823068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f990c06b3a0 0x7f990c0c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:40.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 >> 172.21.15.67:0/3428823068 conn(0x7f990c0b5ef0 msgr2=0x7f990c1466c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:40.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 shutdown_connections 2024-04-17T10:28:40.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:40.938+0000 7f9911c00640 1 -- 172.21.15.67:0/3428823068 wait complete. 2024-04-17T10:28:40.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:28:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.102+0000 7f7b3fe00640 1 Processor -- start 2024-04-17T10:28:41.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.102+0000 7f7b3fe00640 1 -- start start 2024-04-17T10:28:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.102+0000 7f7b3fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7b380cd030 0x7f7b3806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b3806e7b0 0x7f7b380b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b380b4990 con 0x7f7b380aef60 2024-04-17T10:28:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3fe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b380af340 con 0x7f7b380cd030 2024-04-17T10:28:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3806ec00 con 0x7f7b3806e7b0 2024-04-17T10:28:41.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7b380aef60 0x7f7b380cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46110/0 (socket says 172.21.15.67:46110) 2024-04-17T10:28:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3d400640 1 -- 172.21.15.67:0/2628903720 learned_addr learned my addr 172.21.15.67:0/2628903720 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b3806e7b0 0x7f7b380b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3ca00640 1 --2- 172.21.15.67:0/2628903720 >> v2:172.21.15.86:3300/0 conn(0x7f7b380cd030 0x7f7b3806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849472041 0 0) 0x7f7b2c001120 con 0x7f7b380aef60 2024-04-17T10:28:41.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b20005740 con 0x7f7b380aef60 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1267678062 0 0) 0x7f7b20005740 con 0x7f7b380aef60 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b3806e7b0 msgr2=0x7f7b380b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b3806e7b0 0x7f7b380b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 >> v2:172.21.15.86:3300/0 conn(0x7f7b380cd030 msgr2=0x7f7b3806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 --2- 172.21.15.67:0/2628903720 >> v2:172.21.15.86:3300/0 conn(0x7f7b380cd030 0x7f7b3806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3815ae90 con 0x7f7b380aef60 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f7b2c001870 con 0x7f7b380aef60 2024-04-17T10:28:41.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 >> v1:172.21.15.67:6789/0 conn(0x7f7b380aef60 legacy=0x7f7b380cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b20005bd0 0x7f7b20005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b20006a20 0x7f7b20006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b2000dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b380af340 con 0x7f7b200073e0 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3806ec00 con 0x7f7b20005bd0 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2000e170 con 0x7f7b20006a20 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b3806e7b0 0x7f7b380b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3ca00640 1 --2- 172.21.15.67:0/2628903720 >> v2:172.21.15.86:3300/0 conn(0x7f7b380cd030 0x7f7b3806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:41.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3d400640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b20005bd0 0x7f7b20005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b2000dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3ca00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b20006a20 0x7f7b20006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 -- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b20006a20 msgr2=0x7f7b20006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b20006a20 0x7f7b20006ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 -- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b20005bd0 msgr2=0x7f7b20005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b20005bd0 0x7f7b20005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3de00640 1 -- 172.21.15.67:0/2628903720 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b20005740 con 0x7f7b200073e0 2024-04-17T10:28:41.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3ca00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b20006a20 0x7f7b20006ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:41.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.106+0000 7f7b3d400640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b20005bd0 0x7f7b20005ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3de00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b2000dbe0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000e220 tx=0x7f7b3001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719bcaca8fcffad0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b30009d40 con 0x7f7b200073e0 2024-04-17T10:28:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b30003c60 con 0x7f7b200073e0 2024-04-17T10:28:41.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b37e00640 1 -- 172.21.15.67:0/2628903720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b300a5ba0 con 0x7f7b200073e0 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 -- 172.21.15.67:0/2628903720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 msgr2=0x7f7b2000dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 --2- 172.21.15.67:0/2628903720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b2000dbe0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000e220 tx=0x7f7b3001cd00 comp rx=0 tx=0).stop 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3de00640 1 -- 172.21.15.67:0/2628903720 reap_dead start 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 -- 172.21.15.67:0/2628903720 shutdown_connections 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 -- 172.21.15.67:0/2628903720 >> 172.21.15.67:0/2628903720 conn(0x7f7b380b1740 msgr2=0x7f7b380b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 -- 172.21.15.67:0/2628903720 shutdown_connections 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 -- 172.21.15.67:0/2628903720 wait complete. 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 Processor -- start 2024-04-17T10:28:41.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.110+0000 7f7b3fe00640 1 -- start start 2024-04-17T10:28:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b38149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b38149e80 0x7f7b381f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b381f6fa0 0x7f7b38147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3815b410 con 0x7f7b200073e0 2024-04-17T10:28:41.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3815b130 con 0x7f7b38149e80 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3815bc10 con 0x7f7b381f6fa0 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b38149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b38149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44094/0 (socket says 172.21.15.67:44094) 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 -- 172.21.15.67:0/1844926349 learned_addr learned my addr 172.21.15.67:0/1844926349 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 -- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b381f6fa0 msgr2=0x7f7b38147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b381f6fa0 0x7f7b38147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 -- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b38149e80 msgr2=0x7f7b381f6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b38149e80 0x7f7b381f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 -- 172.21.15.67:0/1844926349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b3814e3b0 con 0x7f7b200073e0 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3d400640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b38149940 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001020 tx=0x7f7b2c0117f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b35600640 1 -- 172.21.15.67:0/1844926349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c019020 con 0x7f7b200073e0 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b35600640 1 -- 172.21.15.67:0/1844926349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c015820 con 0x7f7b200073e0 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b35600640 1 -- 172.21.15.67:0/1844926349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c010040 con 0x7f7b200073e0 2024-04-17T10:28:41.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b3814e690 con 0x7f7b200073e0 2024-04-17T10:28:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3814ec50 con 0x7f7b200073e0 2024-04-17T10:28:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b3814ba10 con 0x7f7b200073e0 2024-04-17T10:28:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.114+0000 7f7b35600640 1 -- 172.21.15.67:0/1844926349 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7b2c01c020 con 0x7f7b200073e0 2024-04-17T10:28:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.118+0000 7f7b35600640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7b18066f60 0x7f7b18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.118+0000 7f7b35600640 1 -- 172.21.15.67:0/1844926349 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f7b2c088100 con 0x7f7b200073e0 2024-04-17T10:28:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.118+0000 7f7b3ca00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7b18066f60 0x7f7b18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.118+0000 7f7b3ca00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7b18066f60 0x7f7b18069420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b380b2a20 tx=0x7f7b24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:41.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.122+0000 7f7b35600640 1 -- 172.21.15.67:0/1844926349 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7b2c023020 con 0x7f7b200073e0 2024-04-17T10:28:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7b3804ef30 con 0x7f7b200073e0 2024-04-17T10:28:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b35600640 1 -- 172.21.15.67:0/1844926349 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7b2c00f070 con 0x7f7b200073e0 2024-04-17T10:28:41.276 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7b18066f60 msgr2=0x7f7b18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7b18066f60 0x7f7b18069420 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7b380b2a20 tx=0x7f7b24008040 comp rx=0 tx=0).stop 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 msgr2=0x7f7b38149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b38149940 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001020 tx=0x7f7b2c0117f0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 shutdown_connections 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7b18066f60 0x7f7b18069420 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7b381f6fa0 0x7f7b38147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7b38149e80 0x7f7b381f6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 --2- 172.21.15.67:0/1844926349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7b200073e0 0x7f7b38149940 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 >> 172.21.15.67:0/1844926349 conn(0x7f7b380b1740 msgr2=0x7f7b380cc920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:41.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 shutdown_connections 2024-04-17T10:28:41.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.282+0000 7f7b3fe00640 1 -- 172.21.15.67:0/1844926349 wait complete. 2024-04-17T10:28:41.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:28:41.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03ff400640 1 Processor -- start 2024-04-17T10:28:41.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03ff400640 1 -- start start 2024-04-17T10:28:41.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03ff400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f03f815cc00 0x7f03f815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03ff400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f815fa00 0x7f03f80c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03ff400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f815be60 con 0x7f03f80ce1b0 2024-04-17T10:28:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03ff400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f8160230 con 0x7f03f815cc00 2024-04-17T10:28:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03ff400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f815c120 con 0x7f03f815fa00 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03fd400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f815fa00 0x7f03f80c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03efe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f03f815cc00 0x7f03f815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03efe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f03f815cc00 0x7f03f815f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:44556/0 (socket says 172.21.15.67:44556) 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03efe00640 1 -- 172.21.15.67:0/4033858818 learned_addr learned my addr 172.21.15.67:0/4033858818 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03fca00640 1 --1- 172.21.15.67:0/4033858818 >> v1:172.21.15.86:6789/0 conn(0x7f03f80ce1b0 0x7f03f80ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:49178/0 (socket says 172.21.15.67:49178) 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03fd400640 1 -- 172.21.15.67:0/4033858818 >> v2:172.21.15.86:3300/0 conn(0x7f03f815cc00 msgr2=0x7f03f815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03fd400640 1 --2- 172.21.15.67:0/4033858818 >> v2:172.21.15.86:3300/0 conn(0x7f03f815cc00 0x7f03f815f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03fd400640 1 -- 172.21.15.67:0/4033858818 >> v1:172.21.15.86:6789/0 conn(0x7f03f80ce1b0 legacy=0x7f03f80ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03fd400640 1 -- 172.21.15.67:0/4033858818 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f815c400 con 0x7f03f815fa00 2024-04-17T10:28:41.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.450+0000 7f03efe00640 1 --2- 172.21.15.67:0/4033858818 >> v2:172.21.15.86:3300/0 conn(0x7f03f815cc00 0x7f03f815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:41.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03fd400640 1 --2- 172.21.15.67:0/4033858818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f815fa00 0x7f03f80c6b30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f03f0009d90 tx=0x7f03f001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5a917ef6a2330d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03f7e00640 1 -- 172.21.15.67:0/4033858818 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f00b4070 con 0x7f03f815fa00 2024-04-17T10:28:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03f7e00640 1 -- 172.21.15.67:0/4033858818 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f00af600 con 0x7f03f815fa00 2024-04-17T10:28:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03f7e00640 1 -- 172.21.15.67:0/4033858818 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f00af9d0 con 0x7f03f815fa00 2024-04-17T10:28:41.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 -- 172.21.15.67:0/4033858818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f815fa00 msgr2=0x7f03f80c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 --2- 172.21.15.67:0/4033858818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f815fa00 0x7f03f80c6b30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f03f0009d90 tx=0x7f03f001f150 comp rx=0 tx=0).stop 2024-04-17T10:28:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 -- 172.21.15.67:0/4033858818 shutdown_connections 2024-04-17T10:28:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 --2- 172.21.15.67:0/4033858818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f815fa00 0x7f03f80c6b30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 --2- 172.21.15.67:0/4033858818 >> v2:172.21.15.86:3300/0 conn(0x7f03f815cc00 0x7f03f815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 -- 172.21.15.67:0/4033858818 >> 172.21.15.67:0/4033858818 conn(0x7f03f80b33f0 msgr2=0x7f03f80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 -- 172.21.15.67:0/4033858818 shutdown_connections 2024-04-17T10:28:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.454+0000 7f03ff400640 1 -- 172.21.15.67:0/4033858818 wait complete. 2024-04-17T10:28:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 Processor -- start 2024-04-17T10:28:41.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 -- start start 2024-04-17T10:28:41.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f80ce1b0 0x7f03f806bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f815cc00 0x7f03f806d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f815fa00 0x7f03f8144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f815fff0 con 0x7f03f815fa00 2024-04-17T10:28:41.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f815fe70 con 0x7f03f815cc00 2024-04-17T10:28:41.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8159730 con 0x7f03f80ce1b0 2024-04-17T10:28:41.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f815fa00 0x7f03f8144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f80ce1b0 0x7f03f806bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03efe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f815cc00 0x7f03f806d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f815fa00 0x7f03f8144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44096/0 (socket says 172.21.15.67:44096) 2024-04-17T10:28:41.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 -- 172.21.15.67:0/3392752968 learned_addr learned my addr 172.21.15.67:0/3392752968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:41.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 -- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f80ce1b0 msgr2=0x7f03f806bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f80ce1b0 0x7f03f806bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 -- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f815cc00 msgr2=0x7f03f806d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f815cc00 0x7f03f806d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 -- 172.21.15.67:0/3392752968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f81f3ac0 con 0x7f03f815fa00 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03fd400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f815fa00 0x7f03f8144150 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f03f00b2040 tx=0x7f03f001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03f5600640 1 -- 172.21.15.67:0/3392752968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f00b4040 con 0x7f03f815fa00 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03f5600640 1 -- 172.21.15.67:0/3392752968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f00c1070 con 0x7f03f815fa00 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03f5600640 1 -- 172.21.15.67:0/3392752968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f0009450 con 0x7f03f815fa00 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f80b5db0 con 0x7f03f815fa00 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.458+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f806c710 con 0x7f03f815fa00 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.462+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f806df30 con 0x7f03f815fa00 2024-04-17T10:28:41.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.462+0000 7f03f5600640 1 -- 172.21.15.67:0/3392752968 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f03f00ba6b0 con 0x7f03f815fa00 2024-04-17T10:28:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.462+0000 7f03f5600640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03d4066f60 0x7f03d4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.462+0000 7f03f5600640 1 -- 172.21.15.67:0/3392752968 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f03f0123e30 con 0x7f03f815fa00 2024-04-17T10:28:41.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.462+0000 7f03fca00640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03d4066f60 0x7f03d4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:41.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.462+0000 7f03fca00640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03d4066f60 0x7f03d4069420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f03e8000f70 tx=0x7f03e8007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:41.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.466+0000 7f03f5600640 1 -- 172.21.15.67:0/3392752968 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f03f0018cb0 con 0x7f03f815fa00 2024-04-17T10:28:41.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.638+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f03f8070ab0 con 0x7f03f815fa00 2024-04-17T10:28:41.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03f5600640 1 -- 172.21.15.67:0/3392752968 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v56) v1 ==== 80+0+0 (secure 0 0 0) 0x7f03f00f46b0 con 0x7f03f815fa00 2024-04-17T10:28:41.634 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03d4066f60 msgr2=0x7f03d4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03d4066f60 0x7f03d4069420 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f03e8000f70 tx=0x7f03e8007e90 comp rx=0 tx=0).stop 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f815fa00 msgr2=0x7f03f8144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f815fa00 0x7f03f8144150 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f03f00b2040 tx=0x7f03f001bef0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 shutdown_connections 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03d4066f60 0x7f03d4069420 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03f815fa00 0x7f03f8144150 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03f815cc00 0x7f03f806d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 --2- 172.21.15.67:0/3392752968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03f80ce1b0 0x7f03f806bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:41.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 >> 172.21.15.67:0/3392752968 conn(0x7f03f80b33f0 msgr2=0x7f03f80b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:41.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 shutdown_connections 2024-04-17T10:28:41.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:41.642+0000 7f03ff400640 1 -- 172.21.15.67:0/3392752968 wait complete. 2024-04-17T10:28:41.655 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:28:41.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:41.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:28:42.588 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:28:42.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:28:42.588 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:28:42.597 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:28:42.597 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:42.606 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:28:42.606 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:42.606 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:42.650 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:28:42.651 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:28:42.651 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:42.660 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:28:42.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:42.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.838+0000 7f6b7c000640 1 Processor -- start 2024-04-17T10:28:42.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.838+0000 7f6b7c000640 1 -- start start 2024-04-17T10:28:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7c000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6b74071a20 0x7f6b740aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7c000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b74161740 con 0x7f6b741466e0 2024-04-17T10:28:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7c000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b74161a20 con 0x7f6b74146da0 2024-04-17T10:28:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7c000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b74161d00 con 0x7f6b74071a20 2024-04-17T10:28:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b79600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6b741466e0 0x7f6b7406eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46112/0 (socket says 172.21.15.67:46112) 2024-04-17T10:28:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b79600640 1 -- 172.21.15.67:0/663178508 learned_addr learned my addr 172.21.15.67:0/663178508 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7a000640 1 --2- 172.21.15.67:0/663178508 >> v2:172.21.15.67:3300/0 conn(0x7f6b74071a20 0x7f6b740aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:42.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7a000640 1 -- 172.21.15.67:0/663178508 >> v1:172.21.15.86:6789/0 conn(0x7f6b74146da0 legacy=0x7f6b74071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:42.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b73e00640 1 -- 172.21.15.67:0/663178508 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632798359 0 0) 0x7f6b74161740 con 0x7f6b741466e0 2024-04-17T10:28:42.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7a000640 1 -- 172.21.15.67:0/663178508 >> v1:172.21.15.67:6789/0 conn(0x7f6b741466e0 legacy=0x7f6b7406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:42.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.842+0000 7f6b7a000640 1 -- 172.21.15.67:0/663178508 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b74161fe0 con 0x7f6b74071a20 2024-04-17T10:28:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7a000640 1 --2- 172.21.15.67:0/663178508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74071a20 0x7f6b740aedf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d90 tx=0x7f6b6c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c713b1eca1335f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:42.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b73e00640 1 -- 172.21.15.67:0/663178508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0b2070 con 0x7f6b74071a20 2024-04-17T10:28:42.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b73e00640 1 -- 172.21.15.67:0/663178508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c0a5830 con 0x7f6b74071a20 2024-04-17T10:28:42.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b73e00640 1 -- 172.21.15.67:0/663178508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0a5c00 con 0x7f6b74071a20 2024-04-17T10:28:42.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 -- 172.21.15.67:0/663178508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74071a20 msgr2=0x7f6b740aedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:42.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 --2- 172.21.15.67:0/663178508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74071a20 0x7f6b740aedf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c009d90 tx=0x7f6b6c01c970 comp rx=0 tx=0).stop 2024-04-17T10:28:42.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 -- 172.21.15.67:0/663178508 shutdown_connections 2024-04-17T10:28:42.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 --2- 172.21.15.67:0/663178508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74071a20 0x7f6b740aedf0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:42.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 -- 172.21.15.67:0/663178508 >> 172.21.15.67:0/663178508 conn(0x7f6b7406ac90 msgr2=0x7f6b7406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:42.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 -- 172.21.15.67:0/663178508 shutdown_connections 2024-04-17T10:28:42.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 -- 172.21.15.67:0/663178508 wait complete. 2024-04-17T10:28:42.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 Processor -- start 2024-04-17T10:28:42.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.846+0000 7f6b7c000640 1 -- start start 2024-04-17T10:28:42.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74071a20 0x7f6b7415d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:42.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b741466e0 0x7f6b74141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:42.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74146da0 0x7f6b74142460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7416b7e0 con 0x7f6b74146da0 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7416b660 con 0x7f6b74071a20 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7416b960 con 0x7f6b741466e0 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74146da0 0x7f6b74142460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74146da0 0x7f6b74142460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44114/0 (socket says 172.21.15.67:44114) 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 -- 172.21.15.67:0/3648469021 learned_addr learned my addr 172.21.15.67:0/3648469021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b79600640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74071a20 0x7f6b7415d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b78c00640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b741466e0 0x7f6b74141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 -- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b741466e0 msgr2=0x7f6b74141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b741466e0 0x7f6b74141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 -- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74071a20 msgr2=0x7f6b7415d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74071a20 0x7f6b7415d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:42.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 -- 172.21.15.67:0/3648469021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7415dbe0 con 0x7f6b74146da0 2024-04-17T10:28:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7a000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74146da0 0x7f6b74142460 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c01a5a0 tx=0x7f6b6c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b71600640 1 -- 172.21.15.67:0/3648469021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0b2050 con 0x7f6b74146da0 2024-04-17T10:28:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b71600640 1 -- 172.21.15.67:0/3648469021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b6c0be070 con 0x7f6b74146da0 2024-04-17T10:28:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b71600640 1 -- 172.21.15.67:0/3648469021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b6c0ade20 con 0x7f6b74146da0 2024-04-17T10:28:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b741414e0 con 0x7f6b74146da0 2024-04-17T10:28:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b740b0600 con 0x7f6b74146da0 2024-04-17T10:28:42.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b74072d50 con 0x7f6b74146da0 2024-04-17T10:28:42.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.850+0000 7f6b71600640 1 -- 172.21.15.67:0/3648469021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6b6c0c3460 con 0x7f6b74146da0 2024-04-17T10:28:42.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.854+0000 7f6b71600640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b54066f60 0x7f6b54069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:42.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.854+0000 7f6b71600640 1 -- 172.21.15.67:0/3648469021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6b6c1205a0 con 0x7f6b74146da0 2024-04-17T10:28:42.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.854+0000 7f6b79600640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b54066f60 0x7f6b54069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:42.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.854+0000 7f6b79600640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b54066f60 0x7f6b54069420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6b68000f70 tx=0x7f6b68007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:42.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:42.858+0000 7f6b71600640 1 -- 172.21.15.67:0/3648469021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6b6c016560 con 0x7f6b74146da0 2024-04-17T10:28:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.054+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b74074e30 con 0x7f6b74146da0 2024-04-17T10:28:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b71600640 1 -- 172.21.15.67:0/3648469021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4936 (secure 0 0 0) 0x7f6b6c0f0dc0 con 0x7f6b74146da0 2024-04-17T10:28:43.050 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:43.050 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:38.431605+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-17T10:28:43.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b54066f60 msgr2=0x7f6b54069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b54066f60 0x7f6b54069420 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6b68000f70 tx=0x7f6b68007240 comp rx=0 tx=0).stop 2024-04-17T10:28:43.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74146da0 msgr2=0x7f6b74142460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74146da0 0x7f6b74142460 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c01a5a0 tx=0x7f6b6c019710 comp rx=0 tx=0).stop 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 shutdown_connections 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b54066f60 0x7f6b54069420 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74146da0 0x7f6b74142460 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b741466e0 0x7f6b74141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 --2- 172.21.15.67:0/3648469021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74071a20 0x7f6b7415d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 >> 172.21.15.67:0/3648469021 conn(0x7f6b7406ac90 msgr2=0x7f6b7406d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 shutdown_connections 2024-04-17T10:28:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.058+0000 7f6b7c000640 1 -- 172.21.15.67:0/3648469021 wait complete. 2024-04-17T10:28:43.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:43.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.218+0000 7f2e97000640 1 Processor -- start 2024-04-17T10:28:43.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e97000640 1 -- start start 2024-04-17T10:28:43.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e97000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2e9015dd10 0x7f2e900b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e97000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900b0cc0 con 0x7f2e900b57b0 2024-04-17T10:28:43.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e97000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900b0fa0 con 0x7f2e900714d0 2024-04-17T10:28:43.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e97000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e900b1280 con 0x7f2e9015dd10 2024-04-17T10:28:43.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e87400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2e900b57b0 0x7f2e90070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46126/0 (socket says 172.21.15.67:46126) 2024-04-17T10:28:43.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e87400640 1 -- 172.21.15.67:0/1904577332 learned_addr learned my addr 172.21.15.67:0/1904577332 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8fe00640 1 --2- 172.21.15.67:0/1904577332 >> v2:172.21.15.86:3300/0 conn(0x7f2e9015dd10 0x7f2e900b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608309654 0 0) 0x7f2e900b0cc0 con 0x7f2e900b57b0 2024-04-17T10:28:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e74003530 con 0x7f2e900b57b0 2024-04-17T10:28:43.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381679845 0 0) 0x7f2e900b0fa0 con 0x7f2e900714d0 2024-04-17T10:28:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e900b0cc0 con 0x7f2e900714d0 2024-04-17T10:28:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8fe00640 1 -- 172.21.15.67:0/1904577332 >> v1:172.21.15.86:6789/0 conn(0x7f2e900714d0 legacy=0x7f2e9006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8fe00640 1 -- 172.21.15.67:0/1904577332 >> v1:172.21.15.67:6789/0 conn(0x7f2e900b57b0 legacy=0x7f2e90070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4290681152 0 0) 0x7f2e74003530 con 0x7f2e900b57b0 2024-04-17T10:28:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.222+0000 7f2e8fe00640 1 -- 172.21.15.67:0/1904577332 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9006bed0 con 0x7f2e9015dd10 2024-04-17T10:28:43.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e8fe00640 1 --2- 172.21.15.67:0/1904577332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9015dd10 0x7f2e900b5270 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2e800097c0 tx=0x7f2e8001c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b77edcff75c569c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800b0070 con 0x7f2e9015dd10 2024-04-17T10:28:43.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e800ab400 con 0x7f2e9015dd10 2024-04-17T10:28:43.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e8f400640 1 -- 172.21.15.67:0/1904577332 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78001a00 con 0x7f2e9015dd10 2024-04-17T10:28:43.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 -- 172.21.15.67:0/1904577332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9015dd10 msgr2=0x7f2e900b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 --2- 172.21.15.67:0/1904577332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9015dd10 0x7f2e900b5270 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2e800097c0 tx=0x7f2e8001c300 comp rx=0 tx=0).stop 2024-04-17T10:28:43.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 -- 172.21.15.67:0/1904577332 shutdown_connections 2024-04-17T10:28:43.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 --2- 172.21.15.67:0/1904577332 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e9015dd10 0x7f2e900b5270 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 -- 172.21.15.67:0/1904577332 >> 172.21.15.67:0/1904577332 conn(0x7f2e900b34d0 msgr2=0x7f2e900b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:43.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 -- 172.21.15.67:0/1904577332 shutdown_connections 2024-04-17T10:28:43.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 -- 172.21.15.67:0/1904577332 wait complete. 2024-04-17T10:28:43.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.226+0000 7f2e97000640 1 Processor -- start 2024-04-17T10:28:43.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 -- start start 2024-04-17T10:28:43.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e900714d0 0x7f2e900cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9015dd10 0x7f2e9015fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e90160300 0x7f2e90160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9006da00 con 0x7f2e90160300 2024-04-17T10:28:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9006d880 con 0x7f2e900714d0 2024-04-17T10:28:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9016bc70 con 0x7f2e9015dd10 2024-04-17T10:28:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e90160300 0x7f2e90160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e90160300 0x7f2e90160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44124/0 (socket says 172.21.15.67:44124) 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 -- 172.21.15.67:0/4142669235 learned_addr learned my addr 172.21.15.67:0/4142669235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e87400640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9015dd10 0x7f2e9015fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e8fe00640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e900714d0 0x7f2e900cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 -- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9015dd10 msgr2=0x7f2e9015fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9015dd10 0x7f2e9015fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 -- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e900714d0 msgr2=0x7f2e900cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e900714d0 0x7f2e900cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 -- 172.21.15.67:0/4142669235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e901468c0 con 0x7f2e90160300 2024-04-17T10:28:43.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e95000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e90160300 0x7f2e90160a60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800c3e0 tx=0x7f2e8800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e8cc00640 1 -- 172.21.15.67:0/4142669235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e88017070 con 0x7f2e90160300 2024-04-17T10:28:43.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e8cc00640 1 -- 172.21.15.67:0/4142669235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e880052d0 con 0x7f2e90160300 2024-04-17T10:28:43.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e8cc00640 1 -- 172.21.15.67:0/4142669235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e880054c0 con 0x7f2e90160300 2024-04-17T10:28:43.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e90146b60 con 0x7f2e90160300 2024-04-17T10:28:43.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e90147120 con 0x7f2e90160300 2024-04-17T10:28:43.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.230+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e90070bb0 con 0x7f2e90160300 2024-04-17T10:28:43.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.234+0000 7f2e8cc00640 1 -- 172.21.15.67:0/4142669235 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2e8801a070 con 0x7f2e90160300 2024-04-17T10:28:43.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.234+0000 7f2e8cc00640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2e6c066f90 0x7f2e6c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.234+0000 7f2e8cc00640 1 -- 172.21.15.67:0/4142669235 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2e88084470 con 0x7f2e90160300 2024-04-17T10:28:43.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.234+0000 7f2e8fe00640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2e6c066f90 0x7f2e6c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.234+0000 7f2e8fe00640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2e6c066f90 0x7f2e6c069450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2e800ad000 tx=0x7f2e80008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.238+0000 7f2e8cc00640 1 -- 172.21.15.67:0/4142669235 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2e88054bd0 con 0x7f2e90160300 2024-04-17T10:28:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.398+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e901607b0 con 0x7f2e90160300 2024-04-17T10:28:43.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e8cc00640 1 -- 172.21.15.67:0/4142669235 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16934 (secure 0 0 0) 0x7f2e88058c70 con 0x7f2e90160300 2024-04-17T10:28:43.394 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:43.394 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:37.929771+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:00.190460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:00.765109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.27","mappings":[{"from":0,"to":6}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2e6c066f90 msgr2=0x7f2e6c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2e6c066f90 0x7f2e6c069450 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2e800ad000 tx=0x7f2e80008040 comp rx=0 tx=0).stop 2024-04-17T10:28:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e90160300 msgr2=0x7f2e90160a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e90160300 0x7f2e90160a60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f2e8800c3e0 tx=0x7f2e8800c8a0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 shutdown_connections 2024-04-17T10:28:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2e6c066f90 0x7f2e6c069450 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e90160300 0x7f2e90160a60 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e9015dd10 0x7f2e9015fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 --2- 172.21.15.67:0/4142669235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e900714d0 0x7f2e900cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 >> 172.21.15.67:0/4142669235 conn(0x7f2e900b34d0 msgr2=0x7f2e900cea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 shutdown_connections 2024-04-17T10:28:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.402+0000 7f2e97000640 1 -- 172.21.15.67:0/4142669235 wait complete. 2024-04-17T10:28:43.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:28:00.190460+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:28:00.765109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:28:43.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:28:43.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfaae00640 1 Processor -- start 2024-04-17T10:28:43.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfaae00640 1 -- start start 2024-04-17T10:28:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfaae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa406b3a0 0x7fcfa40b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfaae00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa40b3920 0x7fcfa40d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfaae00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa41616a0 con 0x7fcfa41474c0 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfaae00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa4161940 con 0x7fcfa40b3920 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfaae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4161c20 con 0x7fcfa406b3a0 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfa8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa40b3920 0x7fcfa40d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfa8e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfa40b3920 0x7fcfa40d0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44128/0 (socket says 172.21.15.67:44128) 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfa8e00640 1 -- 172.21.15.67:0/888624216 learned_addr learned my addr 172.21.15.67:0/888624216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcf9b400640 1 --2- 172.21.15.67:0/888624216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa406b3a0 0x7fcfa40b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfa8e00640 1 -- 172.21.15.67:0/888624216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa406b3a0 msgr2=0x7fcfa40b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfa8e00640 1 --2- 172.21.15.67:0/888624216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa406b3a0 0x7fcfa40b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfa8e00640 1 -- 172.21.15.67:0/888624216 >> v1:172.21.15.86:6789/0 conn(0x7fcfa41474c0 legacy=0x7fcfa406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.574+0000 7fcfa8e00640 1 -- 172.21.15.67:0/888624216 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa4161f00 con 0x7fcfa40b3920 2024-04-17T10:28:43.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfa8e00640 1 --2- 172.21.15.67:0/888624216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa40d0480 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009e50 tx=0x7fcf9c01f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92782a69d472e26d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfa3400640 1 -- 172.21.15.67:0/888624216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0b4070 con 0x7fcfa40b3920 2024-04-17T10:28:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfa3400640 1 -- 172.21.15.67:0/888624216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c0af400 con 0x7fcfa40b3920 2024-04-17T10:28:43.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfa3400640 1 -- 172.21.15.67:0/888624216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c0af7d0 con 0x7fcfa40b3920 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 -- 172.21.15.67:0/888624216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 msgr2=0x7fcfa40d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 --2- 172.21.15.67:0/888624216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa40d0480 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009e50 tx=0x7fcf9c01f210 comp rx=0 tx=0).stop 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 -- 172.21.15.67:0/888624216 shutdown_connections 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 --2- 172.21.15.67:0/888624216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa40d0480 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 --2- 172.21.15.67:0/888624216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa406b3a0 0x7fcfa40b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 -- 172.21.15.67:0/888624216 >> 172.21.15.67:0/888624216 conn(0x7fcfa40b5ef0 msgr2=0x7fcfa4099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 -- 172.21.15.67:0/888624216 shutdown_connections 2024-04-17T10:28:43.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.578+0000 7fcfaae00640 1 -- 172.21.15.67:0/888624216 wait complete. 2024-04-17T10:28:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 Processor -- start 2024-04-17T10:28:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 -- start start 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa406b3a0 0x7fcfa415d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa41e9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa41474c0 0x7fcfa41e9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4162420 con 0x7fcfa40b3920 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40b2250 con 0x7fcfa406b3a0 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa40d09c0 con 0x7fcfa41474c0 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa41e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa41e9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44136/0 (socket says 172.21.15.67:44136) 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 -- 172.21.15.67:0/3810007155 learned_addr learned my addr 172.21.15.67:0/3810007155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfa3e00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa406b3a0 0x7fcfa415d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfa8e00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa41474c0 0x7fcfa41e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 -- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa41474c0 msgr2=0x7fcfa41e9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa41474c0 0x7fcfa41e9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 -- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa406b3a0 msgr2=0x7fcfa415d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa406b3a0 0x7fcfa415d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 -- 172.21.15.67:0/3810007155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa41e7fa0 con 0x7fcfa40b3920 2024-04-17T10:28:43.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcf9b400640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa41e9420 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00d800 tx=0x7fcf8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfa0c00640 1 -- 172.21.15.67:0/3810007155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00b6c0 con 0x7fcfa40b3920 2024-04-17T10:28:43.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfa0c00640 1 -- 172.21.15.67:0/3810007155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf8c019070 con 0x7fcfa40b3920 2024-04-17T10:28:43.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfa0c00640 1 -- 172.21.15.67:0/3810007155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c00ba60 con 0x7fcfa40b3920 2024-04-17T10:28:43.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa41e8280 con 0x7fcfa40b3920 2024-04-17T10:28:43.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa41e8840 con 0x7fcfa40b3920 2024-04-17T10:28:43.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.582+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa404ef30 con 0x7fcfa40b3920 2024-04-17T10:28:43.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.586+0000 7fcfa0c00640 1 -- 172.21.15.67:0/3810007155 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcf8c00bc00 con 0x7fcfa40b3920 2024-04-17T10:28:43.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.586+0000 7fcfa0c00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf80066f90 0x7fcf80069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.586+0000 7fcfa0c00640 1 -- 172.21.15.67:0/3810007155 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fcf8c085600 con 0x7fcfa40b3920 2024-04-17T10:28:43.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.586+0000 7fcfa3e00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf80066f90 0x7fcf80069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.586+0000 7fcfa3e00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf80066f90 0x7fcf80069450 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007fd0 tx=0x7fcf94009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.590+0000 7fcfa0c00640 1 -- 172.21.15.67:0/3810007155 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcf8c055d60 con 0x7fcfa40b3920 2024-04-17T10:28:43.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.750+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfa409a310 con 0x7fcfa40b3920 2024-04-17T10:28:43.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.750+0000 7fcfa0c00640 1 -- 172.21.15.67:0/3810007155 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf8c059e00 con 0x7fcfa40b3920 2024-04-17T10:28:43.746 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf80066f90 msgr2=0x7fcf80069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf80066f90 0x7fcf80069450 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007fd0 tx=0x7fcf94009040 comp rx=0 tx=0).stop 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 msgr2=0x7fcfa41e9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa41e9420 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00d800 tx=0x7fcf8c00dcc0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 shutdown_connections 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf80066f90 0x7fcf80069450 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcfa41474c0 0x7fcfa41e9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfa40b3920 0x7fcfa41e9420 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 --2- 172.21.15.67:0/3810007155 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcfa406b3a0 0x7fcfa415d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 >> 172.21.15.67:0/3810007155 conn(0x7fcfa40b5ef0 msgr2=0x7fcfa40b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 shutdown_connections 2024-04-17T10:28:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.754+0000 7fcfaae00640 1 -- 172.21.15.67:0/3810007155 wait complete. 2024-04-17T10:28:43.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:28:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7fecd1a00640 1 Processor -- start 2024-04-17T10:28:43.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7fecd1a00640 1 -- start start 2024-04-17T10:28:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7fecd1a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feccc15dd60 0x7feccc161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7fecd1a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc0d06a0 0x7feccc161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7fecd1a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feccc15e140 con 0x7feccc09adf0 2024-04-17T10:28:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7fecd1a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feccc1626d0 con 0x7feccc15dd60 2024-04-17T10:28:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7fecd1a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc099d50 con 0x7feccc0d06a0 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccaa00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feccc15dd60 0x7feccc161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccb400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc0d06a0 0x7feccc161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccaa00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feccc15dd60 0x7feccc161880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:44606/0 (socket says 172.21.15.67:44606) 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccaa00640 1 -- 172.21.15.67:0/307260317 learned_addr learned my addr 172.21.15.67:0/307260317 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccb400640 1 -- 172.21.15.67:0/307260317 >> v2:172.21.15.86:3300/0 conn(0x7feccc15dd60 msgr2=0x7feccc161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccb400640 1 --2- 172.21.15.67:0/307260317 >> v2:172.21.15.86:3300/0 conn(0x7feccc15dd60 0x7feccc161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccb400640 1 -- 172.21.15.67:0/307260317 >> v1:172.21.15.86:6789/0 conn(0x7feccc09adf0 legacy=0x7feccc0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.918+0000 7feccb400640 1 -- 172.21.15.67:0/307260317 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc09a030 con 0x7feccc0d06a0 2024-04-17T10:28:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7feccb400640 1 --2- 172.21.15.67:0/307260317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc0d06a0 0x7feccc161dc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009d90 tx=0x7fecc401c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a51a53f932976785 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecca000640 1 -- 172.21.15.67:0/307260317 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc40b2070 con 0x7feccc0d06a0 2024-04-17T10:28:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecca000640 1 -- 172.21.15.67:0/307260317 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc40a5a00 con 0x7feccc0d06a0 2024-04-17T10:28:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecca000640 1 -- 172.21.15.67:0/307260317 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc40a5dd0 con 0x7feccc0d06a0 2024-04-17T10:28:43.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 -- 172.21.15.67:0/307260317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc0d06a0 msgr2=0x7feccc161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 --2- 172.21.15.67:0/307260317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc0d06a0 0x7feccc161dc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009d90 tx=0x7fecc401c940 comp rx=0 tx=0).stop 2024-04-17T10:28:43.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 -- 172.21.15.67:0/307260317 shutdown_connections 2024-04-17T10:28:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 --2- 172.21.15.67:0/307260317 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc0d06a0 0x7feccc161dc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 --2- 172.21.15.67:0/307260317 >> v2:172.21.15.86:3300/0 conn(0x7feccc15dd60 0x7feccc161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 -- 172.21.15.67:0/307260317 >> 172.21.15.67:0/307260317 conn(0x7feccc0b34a0 msgr2=0x7feccc0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 -- 172.21.15.67:0/307260317 shutdown_connections 2024-04-17T10:28:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.922+0000 7fecd1a00640 1 -- 172.21.15.67:0/307260317 wait complete. 2024-04-17T10:28:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 Processor -- start 2024-04-17T10:28:43.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 -- start start 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc09adf0 0x7feccc0b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feccc0d06a0 0x7feccc0b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc15dd60 0x7feccc15e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc06c730 con 0x7feccc0d06a0 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feccc06c5b0 con 0x7feccc15dd60 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feccc06c8b0 con 0x7feccc09adf0 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feccc0d06a0 0x7feccc0b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feccc0d06a0 0x7feccc0b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44146/0 (socket says 172.21.15.67:44146) 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 -- 172.21.15.67:0/2247355456 learned_addr learned my addr 172.21.15.67:0/2247355456 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:43.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7feccb400640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc15dd60 0x7feccc15e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7feccaa00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc09adf0 0x7feccc0b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 -- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc09adf0 msgr2=0x7feccc0b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc09adf0 0x7feccc0b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 -- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc15dd60 msgr2=0x7feccc15e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc15dd60 0x7feccc15e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 -- 172.21.15.67:0/2247355456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feccc0b3130 con 0x7feccc0d06a0 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7feccaa00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc09adf0 0x7feccc0b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc3e00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feccc0d06a0 0x7feccc0b1e90 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fecb400c420 tx=0x7fecb400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc2a00640 1 -- 172.21.15.67:0/2247355456 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb40052d0 con 0x7feccc0d06a0 2024-04-17T10:28:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc2a00640 1 -- 172.21.15.67:0/2247355456 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecb4005470 con 0x7feccc0d06a0 2024-04-17T10:28:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecc2a00640 1 -- 172.21.15.67:0/2247355456 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecb4007260 con 0x7feccc0d06a0 2024-04-17T10:28:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feccc0c78d0 con 0x7feccc0d06a0 2024-04-17T10:28:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feccc0c7e90 con 0x7feccc0d06a0 2024-04-17T10:28:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.926+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc04ef30 con 0x7feccc0d06a0 2024-04-17T10:28:43.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.930+0000 7fecc2a00640 1 -- 172.21.15.67:0/2247355456 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fecb4007400 con 0x7feccc0d06a0 2024-04-17T10:28:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.930+0000 7fecc2a00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feca8066f90 0x7feca8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.930+0000 7fecc2a00640 1 -- 172.21.15.67:0/2247355456 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fecb40845a0 con 0x7feccc0d06a0 2024-04-17T10:28:43.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.930+0000 7feccaa00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feca8066f90 0x7feca8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:43.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.930+0000 7feccaa00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feca8066f90 0x7feca8069450 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009b00 tx=0x7fecbc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:43.934+0000 7fecc2a00640 1 -- 172.21.15.67:0/2247355456 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fecb4054d90 con 0x7feccc0d06a0 2024-04-17T10:28:44.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feccc0b42d0 con 0x7feccc0d06a0 2024-04-17T10:28:44.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecc2a00640 1 -- 172.21.15.67:0/2247355456 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v56) v1 ==== 104+0+11 (secure 0 0 0) 0x7fecb4058e30 con 0x7feccc0d06a0 2024-04-17T10:28:44.091 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:28:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feca8066f90 msgr2=0x7feca8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feca8066f90 0x7feca8069450 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009b00 tx=0x7fecbc008040 comp rx=0 tx=0).stop 2024-04-17T10:28:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feccc0d06a0 msgr2=0x7feccc0b1e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feccc0d06a0 0x7feccc0b1e90 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fecb400c420 tx=0x7fecb400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 shutdown_connections 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feca8066f90 0x7feca8069450 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feccc15dd60 0x7feccc15e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feccc0d06a0 0x7feccc0b1e90 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 --2- 172.21.15.67:0/2247355456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feccc09adf0 0x7feccc0b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 >> 172.21.15.67:0/2247355456 conn(0x7feccc0b34a0 msgr2=0x7feccc0cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 shutdown_connections 2024-04-17T10:28:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.098+0000 7fecd1a00640 1 -- 172.21.15.67:0/2247355456 wait complete. 2024-04-17T10:28:44.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:28:44.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f598a000640 1 Processor -- start 2024-04-17T10:28:44.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f598a000640 1 -- start start 2024-04-17T10:28:44.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f598a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f598415b8e0 0x7f59840cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f598a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840cf570 con 0x7f598415bcc0 2024-04-17T10:28:44.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f598a000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59840cf850 con 0x7f598415fcf0 2024-04-17T10:28:44.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f598a000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5984160440 con 0x7f598415b8e0 2024-04-17T10:28:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f5983400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f598415b8e0 0x7f59840cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f597aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f598415bcc0 0x7f598415f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46142/0 (socket says 172.21.15.67:46142) 2024-04-17T10:28:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f597aa00640 1 -- 172.21.15.67:0/2491481743 learned_addr learned my addr 172.21.15.67:0/2491481743 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f5983400640 1 -- 172.21.15.67:0/2491481743 >> v1:172.21.15.86:6789/0 conn(0x7f598415fcf0 legacy=0x7f598406f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f5983400640 1 -- 172.21.15.67:0/2491481743 >> v1:172.21.15.67:6789/0 conn(0x7f598415bcc0 legacy=0x7f598415f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.266+0000 7f5983400640 1 -- 172.21.15.67:0/2491481743 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840cfb30 con 0x7f598415b8e0 2024-04-17T10:28:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f5983400640 1 --2- 172.21.15.67:0/2491481743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415b8e0 0x7f59840cce20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f59740097c0 tx=0x7f597401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0dd0bdd9d1565d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f5982a00640 1 -- 172.21.15.67:0/2491481743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740b0070 con 0x7f598415b8e0 2024-04-17T10:28:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f5982a00640 1 -- 172.21.15.67:0/2491481743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59740ab600 con 0x7f598415b8e0 2024-04-17T10:28:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f5982a00640 1 -- 172.21.15.67:0/2491481743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740ab9d0 con 0x7f598415b8e0 2024-04-17T10:28:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f598a000640 1 -- 172.21.15.67:0/2491481743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415b8e0 msgr2=0x7f59840cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f598a000640 1 --2- 172.21.15.67:0/2491481743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415b8e0 0x7f59840cce20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f59740097c0 tx=0x7f597401c460 comp rx=0 tx=0).stop 2024-04-17T10:28:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f598a000640 1 -- 172.21.15.67:0/2491481743 shutdown_connections 2024-04-17T10:28:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f598a000640 1 --2- 172.21.15.67:0/2491481743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415b8e0 0x7f59840cce20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f598a000640 1 -- 172.21.15.67:0/2491481743 >> 172.21.15.67:0/2491481743 conn(0x7f59840d1800 msgr2=0x7f59840d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:44.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f598a000640 1 -- 172.21.15.67:0/2491481743 shutdown_connections 2024-04-17T10:28:44.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.270+0000 7f598a000640 1 -- 172.21.15.67:0/2491481743 wait complete. 2024-04-17T10:28:44.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 Processor -- start 2024-04-17T10:28:44.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 -- start start 2024-04-17T10:28:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598415b8e0 0x7f598415a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598415bcc0 0x7f59840ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415fcf0 0x7f5984165a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984169cc0 con 0x7f598415fcf0 2024-04-17T10:28:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5984169b40 con 0x7f598415b8e0 2024-04-17T10:28:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5984169e40 con 0x7f598415bcc0 2024-04-17T10:28:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415fcf0 0x7f5984165a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415fcf0 0x7f5984165a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44160/0 (socket says 172.21.15.67:44160) 2024-04-17T10:28:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 -- 172.21.15.67:0/3546241383 learned_addr learned my addr 172.21.15.67:0/3546241383 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983400640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598415b8e0 0x7f598415a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f597aa00640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598415bcc0 0x7f59840ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 -- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598415bcc0 msgr2=0x7f59840ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598415bcc0 0x7f59840ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 -- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598415b8e0 msgr2=0x7f598415a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598415b8e0 0x7f598415a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 -- 172.21.15.67:0/3546241383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840cb250 con 0x7f598415fcf0 2024-04-17T10:28:44.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f597aa00640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598415bcc0 0x7f59840ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f5983e00640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415fcf0 0x7f5984165a20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f597c00c780 tx=0x7f597c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f597be00640 1 -- 172.21.15.67:0/3546241383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c017070 con 0x7f598415fcf0 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f597be00640 1 -- 172.21.15.67:0/3546241383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c003db0 con 0x7f598415fcf0 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59840cb530 con 0x7f598415fcf0 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f597be00640 1 -- 172.21.15.67:0/3546241383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c0052b0 con 0x7f598415fcf0 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59840b4420 con 0x7f598415fcf0 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.274+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59840cb700 con 0x7f598415fcf0 2024-04-17T10:28:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.278+0000 7f597be00640 1 -- 172.21.15.67:0/3546241383 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f597c01a070 con 0x7f598415fcf0 2024-04-17T10:28:44.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.278+0000 7f597be00640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5960066f60 0x7f5960069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.278+0000 7f597be00640 1 -- 172.21.15.67:0/3546241383 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f597c084770 con 0x7f598415fcf0 2024-04-17T10:28:44.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.278+0000 7f5983400640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5960066f60 0x7f5960069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.278+0000 7f5983400640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5960066f60 0x7f5960069420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f59740ad000 tx=0x7f59740a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.282+0000 7f597be00640 1 -- 172.21.15.67:0/3546241383 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f597c054f60 con 0x7f598415fcf0 2024-04-17T10:28:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f598404ef30 con 0x7f598415fcf0 2024-04-17T10:28:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f597be00640 1 -- 172.21.15.67:0/3546241383 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v56) v1 ==== 100+0+11 (secure 0 0 0) 0x7f597c059000 con 0x7f598415fcf0 2024-04-17T10:28:44.442 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:28:44.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5960066f60 msgr2=0x7f5960069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5960066f60 0x7f5960069420 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f59740ad000 tx=0x7f59740a6000 comp rx=0 tx=0).stop 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415fcf0 msgr2=0x7f5984165a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415fcf0 0x7f5984165a20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f597c00c780 tx=0x7f597c00cc40 comp rx=0 tx=0).stop 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 shutdown_connections 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5960066f60 0x7f5960069420 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598415fcf0 0x7f5984165a20 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598415bcc0 0x7f59840ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 --2- 172.21.15.67:0/3546241383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598415b8e0 0x7f598415a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 >> 172.21.15.67:0/3546241383 conn(0x7f59840d1800 msgr2=0x7f59840d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 shutdown_connections 2024-04-17T10:28:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.446+0000 7f598a000640 1 -- 172.21.15.67:0/3546241383 wait complete. 2024-04-17T10:28:44.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:44.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6f01200640 1 Processor -- start 2024-04-17T10:28:44.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6f01200640 1 -- start start 2024-04-17T10:28:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6f01200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6efc09a160 0x7f6efc09a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6f01200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0cf330 con 0x7f6efc06f770 2024-04-17T10:28:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6f01200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc0cf610 con 0x7f6efc070470 2024-04-17T10:28:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6f01200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6efc09b190 con 0x7f6efc09a160 2024-04-17T10:28:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6efbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6efc09a160 0x7f6efc09a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6efb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6efc06f770 0x7f6efc06fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46148/0 (socket says 172.21.15.67:46148) 2024-04-17T10:28:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6efb400640 1 -- 172.21.15.67:0/2118252794 learned_addr learned my addr 172.21.15.67:0/2118252794 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6efbe00640 1 -- 172.21.15.67:0/2118252794 >> v1:172.21.15.86:6789/0 conn(0x7f6efc070470 legacy=0x7f6efc099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6efbe00640 1 -- 172.21.15.67:0/2118252794 >> v1:172.21.15.67:6789/0 conn(0x7f6efc06f770 legacy=0x7f6efc06fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6efa000640 1 -- 172.21.15.67:0/2118252794 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955124959 0 0) 0x7f6efc0cf330 con 0x7f6efc06f770 2024-04-17T10:28:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.610+0000 7f6efbe00640 1 -- 172.21.15.67:0/2118252794 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc09ac70 con 0x7f6efc09a160 2024-04-17T10:28:44.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6efbe00640 1 --2- 172.21.15.67:0/2118252794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc09a160 0x7f6efc09a5b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009d90 tx=0x7f6ef401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4fbed503b45919b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6efa000640 1 -- 172.21.15.67:0/2118252794 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40b2070 con 0x7f6efc09a160 2024-04-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6efa000640 1 -- 172.21.15.67:0/2118252794 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef40a57a0 con 0x7f6efc09a160 2024-04-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6efa000640 1 -- 172.21.15.67:0/2118252794 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef40a5b70 con 0x7f6efc09a160 2024-04-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 -- 172.21.15.67:0/2118252794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc09a160 msgr2=0x7f6efc09a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 --2- 172.21.15.67:0/2118252794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc09a160 0x7f6efc09a5b0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4009d90 tx=0x7f6ef401c8e0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 -- 172.21.15.67:0/2118252794 shutdown_connections 2024-04-17T10:28:44.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 --2- 172.21.15.67:0/2118252794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc09a160 0x7f6efc09a5b0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 -- 172.21.15.67:0/2118252794 >> 172.21.15.67:0/2118252794 conn(0x7f6efc0d1630 msgr2=0x7f6efc0d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 -- 172.21.15.67:0/2118252794 shutdown_connections 2024-04-17T10:28:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 -- 172.21.15.67:0/2118252794 wait complete. 2024-04-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 Processor -- start 2024-04-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.614+0000 7f6f01200640 1 -- start start 2024-04-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06f770 0x7f6efc155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc070470 0x7f6efc154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc09a160 0x7f6efc154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc169760 con 0x7f6efc06f770 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc071610 con 0x7f6efc070470 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc1698e0 con 0x7f6efc09a160 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06f770 0x7f6efc155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06f770 0x7f6efc155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44164/0 (socket says 172.21.15.67:44164) 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 -- 172.21.15.67:0/3461018001 learned_addr learned my addr 172.21.15.67:0/3461018001 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efbe00640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc09a160 0x7f6efc154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efaa00640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc070470 0x7f6efc154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 -- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc09a160 msgr2=0x7f6efc154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc09a160 0x7f6efc154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 -- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc070470 msgr2=0x7f6efc154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc070470 0x7f6efc154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 -- 172.21.15.67:0/3461018001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc155130 con 0x7f6efc06f770 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efbe00640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc09a160 0x7f6efc154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efaa00640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc070470 0x7f6efc154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6efb400640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06f770 0x7f6efc155630 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00c410 tx=0x7f6eec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6ef3400640 1 -- 172.21.15.67:0/3461018001 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec0074e0 con 0x7f6efc06f770 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6ef3400640 1 -- 172.21.15.67:0/3461018001 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec0047f0 con 0x7f6efc06f770 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc155cc0 con 0x7f6efc06f770 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6ef3400640 1 -- 172.21.15.67:0/3461018001 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec007680 con 0x7f6efc06f770 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc156280 con 0x7f6efc06f770 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6efc074dc0 con 0x7f6efc06f770 2024-04-17T10:28:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.618+0000 7f6ef3400640 1 -- 172.21.15.67:0/3461018001 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6eec011070 con 0x7f6efc06f770 2024-04-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.622+0000 7f6ef3400640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6edc066f60 0x7f6edc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.622+0000 7f6ef3400640 1 -- 172.21.15.67:0/3461018001 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6eec084dc0 con 0x7f6efc06f770 2024-04-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.622+0000 7f6efaa00640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6edc066f60 0x7f6edc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.622+0000 7f6efaa00640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6edc066f60 0x7f6edc069420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6efc15eb10 tx=0x7f6ee8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.626+0000 7f6ef3400640 1 -- 172.21.15.67:0/3461018001 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6eec055c80 con 0x7f6efc06f770 2024-04-17T10:28:44.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6efc099a50 con 0x7f6efc06f770 2024-04-17T10:28:44.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6ef3400640 1 -- 172.21.15.67:0/3461018001 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4936 (secure 0 0 0) 0x7f6eec055580 con 0x7f6efc06f770 2024-04-17T10:28:44.816 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:44.816 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:38.431605+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:44.816 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-17T10:28:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6edc066f60 msgr2=0x7f6edc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6edc066f60 0x7f6edc069420 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6efc15eb10 tx=0x7f6ee8009040 comp rx=0 tx=0).stop 2024-04-17T10:28:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06f770 msgr2=0x7f6efc155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06f770 0x7f6efc155630 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f6eec00c410 tx=0x7f6eec00c8d0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 shutdown_connections 2024-04-17T10:28:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6edc066f60 0x7f6edc069420 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6efc09a160 0x7f6efc154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6efc070470 0x7f6efc154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 --2- 172.21.15.67:0/3461018001 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6efc06f770 0x7f6efc155630 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 >> 172.21.15.67:0/3461018001 conn(0x7f6efc0d1630 msgr2=0x7f6efc0aedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:44.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 shutdown_connections 2024-04-17T10:28:44.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.822+0000 7f6f01200640 1 -- 172.21.15.67:0/3461018001 wait complete. 2024-04-17T10:28:44.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:44.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2721200640 1 Processor -- start 2024-04-17T10:28:44.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2721200640 1 -- start start 2024-04-17T10:28:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2721200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f271c15dd10 0x7f271c0b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2721200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c0b57b0 0x7f271c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2721200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c0b0cc0 con 0x7f271c0b07b0 2024-04-17T10:28:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2721200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271c0b1280 con 0x7f271c15dd10 2024-04-17T10:28:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2721200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c0b0fa0 con 0x7f271c0b57b0 2024-04-17T10:28:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f271a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f271c0b07b0 0x7f271c1618c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46158/0 (socket says 172.21.15.67:46158) 2024-04-17T10:28:44.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f271a800640 1 -- 172.21.15.67:0/3005250745 learned_addr learned my addr 172.21.15.67:0/3005250745 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2711400640 1 --2- 172.21.15.67:0/3005250745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c0b57b0 0x7f271c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2719e00640 1 --2- 172.21.15.67:0/3005250745 >> v2:172.21.15.86:3300/0 conn(0x7f271c15dd10 0x7f271c0b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2719400640 1 -- 172.21.15.67:0/3005250745 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033945737 0 0) 0x7f271c0b0cc0 con 0x7f271c0b07b0 2024-04-17T10:28:44.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2719400640 1 -- 172.21.15.67:0/3005250745 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2700003530 con 0x7f271c0b07b0 2024-04-17T10:28:44.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2711400640 1 -- 172.21.15.67:0/3005250745 >> v2:172.21.15.86:3300/0 conn(0x7f271c15dd10 msgr2=0x7f271c0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:44.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2711400640 1 --2- 172.21.15.67:0/3005250745 >> v2:172.21.15.86:3300/0 conn(0x7f271c15dd10 0x7f271c0b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2711400640 1 -- 172.21.15.67:0/3005250745 >> v1:172.21.15.67:6789/0 conn(0x7f271c0b07b0 legacy=0x7f271c1618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2711400640 1 -- 172.21.15.67:0/3005250745 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c06bed0 con 0x7f271c0b57b0 2024-04-17T10:28:44.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.990+0000 7f2719e00640 1 --2- 172.21.15.67:0/3005250745 >> v2:172.21.15.86:3300/0 conn(0x7f271c15dd10 0x7f271c0b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2711400640 1 --2- 172.21.15.67:0/3005250745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c0b57b0 0x7f271c070dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f27040097c0 tx=0x7f270401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a2406747df000b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2719400640 1 -- 172.21.15.67:0/3005250745 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27040b1070 con 0x7f271c0b57b0 2024-04-17T10:28:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2719400640 1 -- 172.21.15.67:0/3005250745 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27040ac620 con 0x7f271c0b57b0 2024-04-17T10:28:44.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2719400640 1 -- 172.21.15.67:0/3005250745 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27040ac9f0 con 0x7f271c0b57b0 2024-04-17T10:28:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 -- 172.21.15.67:0/3005250745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c0b57b0 msgr2=0x7f271c070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 --2- 172.21.15.67:0/3005250745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c0b57b0 0x7f271c070dc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f27040097c0 tx=0x7f270401c3e0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 -- 172.21.15.67:0/3005250745 shutdown_connections 2024-04-17T10:28:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 --2- 172.21.15.67:0/3005250745 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c0b57b0 0x7f271c070dc0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 --2- 172.21.15.67:0/3005250745 >> v2:172.21.15.86:3300/0 conn(0x7f271c15dd10 0x7f271c0b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 -- 172.21.15.67:0/3005250745 >> 172.21.15.67:0/3005250745 conn(0x7f271c0b34d0 msgr2=0x7f271c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 -- 172.21.15.67:0/3005250745 shutdown_connections 2024-04-17T10:28:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.994+0000 7f2721200640 1 -- 172.21.15.67:0/3005250745 wait complete. 2024-04-17T10:28:44.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 Processor -- start 2024-04-17T10:28:44.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 -- start start 2024-04-17T10:28:44.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f271c0b07b0 0x7f271c14f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c15dd10 0x7f271c0cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f271c141140 0x7f271c151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c16ae70 con 0x7f271c141140 2024-04-17T10:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c071710 con 0x7f271c0b07b0 2024-04-17T10:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c16aff0 con 0x7f271c15dd10 2024-04-17T10:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f271c141140 0x7f271c151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2711400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c15dd10 0x7f271c0cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2719e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f271c0b07b0 0x7f271c14f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f271c141140 0x7f271c151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44172/0 (socket says 172.21.15.67:44172) 2024-04-17T10:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 -- 172.21.15.67:0/875628014 learned_addr learned my addr 172.21.15.67:0/875628014 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 -- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c15dd10 msgr2=0x7f271c0cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c15dd10 0x7f271c0cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 -- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f271c0b07b0 msgr2=0x7f271c14f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f271c0b07b0 0x7f271c14f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 -- 172.21.15.67:0/875628014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c14f820 con 0x7f271c141140 2024-04-17T10:28:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2711400640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c15dd10 0x7f271c0cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f271a800640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f271c141140 0x7f271c151360 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f271400cbe0 tx=0x7f271400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2712a00640 1 -- 172.21.15.67:0/875628014 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714017070 con 0x7f271c141140 2024-04-17T10:28:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2712a00640 1 -- 172.21.15.67:0/875628014 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271400ad40 con 0x7f271c141140 2024-04-17T10:28:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2712a00640 1 -- 172.21.15.67:0/875628014 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2714012600 con 0x7f271c141140 2024-04-17T10:28:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c14fb00 con 0x7f271c141140 2024-04-17T10:28:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:44.998+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c1500c0 con 0x7f271c141140 2024-04-17T10:28:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.002+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c070bb0 con 0x7f271c141140 2024-04-17T10:28:44.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.002+0000 7f2712a00640 1 -- 172.21.15.67:0/875628014 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f271401a070 con 0x7f271c141140 2024-04-17T10:28:45.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.002+0000 7f2712a00640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26f8066f90 0x7f26f8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.002+0000 7f2712a00640 1 -- 172.21.15.67:0/875628014 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f2714084f20 con 0x7f271c141140 2024-04-17T10:28:45.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.002+0000 7f2719e00640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26f8066f90 0x7f26f8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.002+0000 7f2719e00640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26f8066f90 0x7f26f8069450 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f270c009580 tx=0x7f270c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:45.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.006+0000 7f2712a00640 1 -- 172.21.15.67:0/875628014 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2714055690 con 0x7f271c141140 2024-04-17T10:28:45.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.170+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f271c074e30 con 0x7f271c141140 2024-04-17T10:28:45.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.170+0000 7f2712a00640 1 -- 172.21.15.67:0/875628014 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16934 (secure 0 0 0) 0x7f2714059730 con 0x7f271c141140 2024-04-17T10:28:45.165 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:45.165 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:37.929771+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:00.190460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:00.765109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.27","mappings":[{"from":0,"to":6}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26f8066f90 msgr2=0x7f26f8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26f8066f90 0x7f26f8069450 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f270c009580 tx=0x7f270c009190 comp rx=0 tx=0).stop 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f271c141140 msgr2=0x7f271c151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f271c141140 0x7f271c151360 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f271400cbe0 tx=0x7f271400a430 comp rx=0 tx=0).stop 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 shutdown_connections 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26f8066f90 0x7f26f8069450 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f271c141140 0x7f271c151360 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f271c15dd10 0x7f271c0cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 --2- 172.21.15.67:0/875628014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f271c0b07b0 0x7f271c14f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 >> 172.21.15.67:0/875628014 conn(0x7f271c0b34d0 msgr2=0x7f271c15b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:45.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 shutdown_connections 2024-04-17T10:28:45.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.174+0000 7f2721200640 1 -- 172.21.15.67:0/875628014 wait complete. 2024-04-17T10:28:45.186 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:28:45.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee38000640 1 Processor -- start 2024-04-17T10:28:45.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee38000640 1 -- start start 2024-04-17T10:28:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee38000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fee3015b970 0x7fee300cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee38000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee3006b6a0 0x7fee300cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee38000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee3015cf10 con 0x7fee300b2dc0 2024-04-17T10:28:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee38000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3015c300 con 0x7fee3006b6a0 2024-04-17T10:28:45.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee38000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee3015c5e0 con 0x7fee3015b970 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee34c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee3006b6a0 0x7fee300cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee36000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fee300b2dc0 0x7fee300ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46170/0 (socket says 172.21.15.67:46170) 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee34c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fee3006b6a0 0x7fee300cfdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44184/0 (socket says 172.21.15.67:44184) 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee36000640 1 -- 172.21.15.67:0/2631504215 learned_addr learned my addr 172.21.15.67:0/2631504215 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee35600640 1 --2- 172.21.15.67:0/2631504215 >> v2:172.21.15.86:3300/0 conn(0x7fee3015b970 0x7fee300cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee34c00640 1 -- 172.21.15.67:0/2631504215 >> v2:172.21.15.86:3300/0 conn(0x7fee3015b970 msgr2=0x7fee300cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee34c00640 1 --2- 172.21.15.67:0/2631504215 >> v2:172.21.15.86:3300/0 conn(0x7fee3015b970 0x7fee300cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee34c00640 1 -- 172.21.15.67:0/2631504215 >> v1:172.21.15.67:6789/0 conn(0x7fee300b2dc0 legacy=0x7fee300ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.334+0000 7fee34c00640 1 -- 172.21.15.67:0/2631504215 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee3015c8c0 con 0x7fee3006b6a0 2024-04-17T10:28:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee34c00640 1 --2- 172.21.15.67:0/2631504215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3006b6a0 0x7fee300cfdc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fee300716f0 tx=0x7fee1c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82b3836ba426512f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee2fe00640 1 -- 172.21.15.67:0/2631504215 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0b0070 con 0x7fee3006b6a0 2024-04-17T10:28:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee2fe00640 1 -- 172.21.15.67:0/2631504215 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1c0ab600 con 0x7fee3006b6a0 2024-04-17T10:28:45.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee2fe00640 1 -- 172.21.15.67:0/2631504215 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0ab9d0 con 0x7fee3006b6a0 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 -- 172.21.15.67:0/2631504215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3006b6a0 msgr2=0x7fee300cfdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 --2- 172.21.15.67:0/2631504215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3006b6a0 0x7fee300cfdc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fee300716f0 tx=0x7fee1c01c3f0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 -- 172.21.15.67:0/2631504215 shutdown_connections 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 --2- 172.21.15.67:0/2631504215 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3006b6a0 0x7fee300cfdc0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 --2- 172.21.15.67:0/2631504215 >> v2:172.21.15.86:3300/0 conn(0x7fee3015b970 0x7fee300cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 -- 172.21.15.67:0/2631504215 >> 172.21.15.67:0/2631504215 conn(0x7fee300b22a0 msgr2=0x7fee300b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 -- 172.21.15.67:0/2631504215 shutdown_connections 2024-04-17T10:28:45.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.338+0000 7fee38000640 1 -- 172.21.15.67:0/2631504215 wait complete. 2024-04-17T10:28:45.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 Processor -- start 2024-04-17T10:28:45.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 -- start start 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3006b6a0 0x7fee300c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee300b2dc0 0x7fee30142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3015b970 0x7fee30150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee30168910 con 0x7fee3015b970 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee30168790 con 0x7fee3006b6a0 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee30168a90 con 0x7fee300b2dc0 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3015b970 0x7fee30150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3015b970 0x7fee30150910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44192/0 (socket says 172.21.15.67:44192) 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 -- 172.21.15.67:0/2074140690 learned_addr learned my addr 172.21.15.67:0/2074140690 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee35600640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3006b6a0 0x7fee300c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee34c00640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee300b2dc0 0x7fee30142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 -- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee300b2dc0 msgr2=0x7fee30142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee300b2dc0 0x7fee30142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 -- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3006b6a0 msgr2=0x7fee300c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3006b6a0 0x7fee300c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 -- 172.21.15.67:0/2074140690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee300c70a0 con 0x7fee3015b970 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee36000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3015b970 0x7fee30150910 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fee2800caf0 tx=0x7fee2800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee2d600640 1 -- 172.21.15.67:0/2074140690 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28017070 con 0x7fee3015b970 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee2d600640 1 -- 172.21.15.67:0/2074140690 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2800aca0 con 0x7fee3015b970 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee2d600640 1 -- 172.21.15.67:0/2074140690 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28012600 con 0x7fee3015b970 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee300c6ae0 con 0x7fee3015b970 2024-04-17T10:28:45.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee300c6700 con 0x7fee3015b970 2024-04-17T10:28:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.342+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee3009e900 con 0x7fee3015b970 2024-04-17T10:28:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.346+0000 7fee2d600640 1 -- 172.21.15.67:0/2074140690 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fee2801a070 con 0x7fee3015b970 2024-04-17T10:28:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.346+0000 7fee2d600640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fee10066f60 0x7fee10069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.346+0000 7fee2d600640 1 -- 172.21.15.67:0/2074140690 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fee28084e00 con 0x7fee3015b970 2024-04-17T10:28:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.346+0000 7fee35600640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fee10066f60 0x7fee10069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.346+0000 7fee35600640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fee10066f60 0x7fee10069420 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fee24009ac0 tx=0x7fee24009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:45.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.350+0000 7fee2d600640 1 -- 172.21.15.67:0/2074140690 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fee280555f0 con 0x7fee3015b970 2024-04-17T10:28:45.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.542+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee30074e30 con 0x7fee3015b970 2024-04-17T10:28:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.542+0000 7fee2d600640 1 -- 172.21.15.67:0/2074140690 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4936 (secure 0 0 0) 0x7fee28059690 con 0x7fee3015b970 2024-04-17T10:28:45.538 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:45.538 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:38.431605+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:45.538 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fee10066f60 msgr2=0x7fee10069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fee10066f60 0x7fee10069420 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fee24009ac0 tx=0x7fee24009190 comp rx=0 tx=0).stop 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3015b970 msgr2=0x7fee30150910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3015b970 0x7fee30150910 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fee2800caf0 tx=0x7fee2800a060 comp rx=0 tx=0).stop 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 shutdown_connections 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fee10066f60 0x7fee10069420 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fee3015b970 0x7fee30150910 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fee300b2dc0 0x7fee30142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 --2- 172.21.15.67:0/2074140690 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fee3006b6a0 0x7fee300c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 >> 172.21.15.67:0/2074140690 conn(0x7fee300b22a0 msgr2=0x7fee300d0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 shutdown_connections 2024-04-17T10:28:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.546+0000 7fee38000640 1 -- 172.21.15.67:0/2074140690 wait complete. 2024-04-17T10:28:45.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:45.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.706+0000 7fe9dce00640 1 Processor -- start 2024-04-17T10:28:45.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9dce00640 1 -- start start 2024-04-17T10:28:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9dce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d8158170 0x7fe9d815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9dce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d806d9f0 con 0x7fe9d80ced40 2024-04-17T10:28:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9dce00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d806dcd0 con 0x7fe9d806d210 2024-04-17T10:28:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9dce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d80cf3c0 con 0x7fe9d8158170 2024-04-17T10:28:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe9d80ced40 0x7fe9d80cc6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46186/0 (socket says 172.21.15.67:46186) 2024-04-17T10:28:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d7e00640 1 -- 172.21.15.67:0/3699107226 learned_addr learned my addr 172.21.15.67:0/3699107226 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:45.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d7400640 1 --2- 172.21.15.67:0/3699107226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d8158170 0x7fe9d815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d6a00640 1 -- 172.21.15.67:0/3699107226 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600526608 0 0) 0x7fe9d806d9f0 con 0x7fe9d80ced40 2024-04-17T10:28:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d6a00640 1 -- 172.21.15.67:0/3699107226 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9bc005740 con 0x7fe9d80ced40 2024-04-17T10:28:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d6a00640 1 -- 172.21.15.67:0/3699107226 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551494651 0 0) 0x7fe9d806dcd0 con 0x7fe9d806d210 2024-04-17T10:28:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d6a00640 1 -- 172.21.15.67:0/3699107226 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d806d9f0 con 0x7fe9d806d210 2024-04-17T10:28:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d7400640 1 -- 172.21.15.67:0/3699107226 >> v1:172.21.15.86:6789/0 conn(0x7fe9d806d210 legacy=0x7fe9d80ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d7400640 1 -- 172.21.15.67:0/3699107226 >> v1:172.21.15.67:6789/0 conn(0x7fe9d80ced40 legacy=0x7fe9d80cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.710+0000 7fe9d7400640 1 -- 172.21.15.67:0/3699107226 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d806dfb0 con 0x7fe9d8158170 2024-04-17T10:28:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9d7400640 1 --2- 172.21.15.67:0/3699107226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d8158170 0x7fe9d815ace0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80097f0 tx=0x7fe9c801c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19fbf2006faa91f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9d6a00640 1 -- 172.21.15.67:0/3699107226 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80aa410 con 0x7fe9d8158170 2024-04-17T10:28:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9d6a00640 1 -- 172.21.15.67:0/3699107226 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c80af070 con 0x7fe9d8158170 2024-04-17T10:28:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9d6a00640 1 -- 172.21.15.67:0/3699107226 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80aa7d0 con 0x7fe9d8158170 2024-04-17T10:28:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 -- 172.21.15.67:0/3699107226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d8158170 msgr2=0x7fe9d815ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 --2- 172.21.15.67:0/3699107226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d8158170 0x7fe9d815ace0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80097f0 tx=0x7fe9c801c4b0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 -- 172.21.15.67:0/3699107226 shutdown_connections 2024-04-17T10:28:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 --2- 172.21.15.67:0/3699107226 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d8158170 0x7fe9d815ace0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 -- 172.21.15.67:0/3699107226 >> 172.21.15.67:0/3699107226 conn(0x7fe9d8067070 msgr2=0x7fe9d806ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 -- 172.21.15.67:0/3699107226 shutdown_connections 2024-04-17T10:28:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 -- 172.21.15.67:0/3699107226 wait complete. 2024-04-17T10:28:45.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 Processor -- start 2024-04-17T10:28:45.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.714+0000 7fe9dce00640 1 -- start start 2024-04-17T10:28:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d806d210 0x7fe9d80ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d80ced40 0x7fe9d8165f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8158170 0x7fe9d8149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d81586d0 con 0x7fe9d8158170 2024-04-17T10:28:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8158550 con 0x7fe9d806d210 2024-04-17T10:28:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d806ea90 con 0x7fe9d80ced40 2024-04-17T10:28:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8158170 0x7fe9d8149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9cfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d80ced40 0x7fe9d8165f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d806d210 0x7fe9d80ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8158170 0x7fe9d8149180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44196/0 (socket says 172.21.15.67:44196) 2024-04-17T10:28:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9cfe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d80ced40 0x7fe9d8165f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:50674/0 (socket says 172.21.15.67:50674) 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7e00640 1 -- 172.21.15.67:0/2247273651 learned_addr learned my addr 172.21.15.67:0/2247273651 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7400640 1 -- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d80ced40 msgr2=0x7fe9d8165f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7400640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d80ced40 0x7fe9d8165f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7400640 1 -- 172.21.15.67:0/2247273651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8158170 msgr2=0x7fe9d8149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7400640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8158170 0x7fe9d8149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7400640 1 -- 172.21.15.67:0/2247273651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d81664a0 con 0x7fe9d806d210 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7e00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8158170 0x7fe9d8149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9d7400640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d806d210 0x7fe9d80ca980 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80ab000 tx=0x7fe9c801acf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:45.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9cf400640 1 -- 172.21.15.67:0/2247273651 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80be610 con 0x7fe9d806d210 2024-04-17T10:28:45.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9cf400640 1 -- 172.21.15.67:0/2247273651 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c80be7b0 con 0x7fe9d806d210 2024-04-17T10:28:45.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9cf400640 1 -- 172.21.15.67:0/2247273651 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80beba0 con 0x7fe9d806d210 2024-04-17T10:28:45.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d80caff0 con 0x7fe9d806d210 2024-04-17T10:28:45.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d80cb5b0 con 0x7fe9d806d210 2024-04-17T10:28:45.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.718+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d8074dc0 con 0x7fe9d806d210 2024-04-17T10:28:45.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.722+0000 7fe9cf400640 1 -- 172.21.15.67:0/2247273651 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe9c80bc020 con 0x7fe9d806d210 2024-04-17T10:28:45.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.722+0000 7fe9cf400640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9b4066f60 0x7fe9b4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:45.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.722+0000 7fe9cf400640 1 -- 172.21.15.67:0/2247273651 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe9c811fb10 con 0x7fe9d806d210 2024-04-17T10:28:45.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.722+0000 7fe9cfe00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9b4066f60 0x7fe9b4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:45.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.722+0000 7fe9cfe00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9b4066f60 0x7fe9b4069420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8162350 tx=0x7fe9c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:45.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.726+0000 7fe9cf400640 1 -- 172.21.15.67:0/2247273651 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe9c80c3060 con 0x7fe9d806d210 2024-04-17T10:28:45.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.926+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9d80b4210 con 0x7fe9d806d210 2024-04-17T10:28:45.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.926+0000 7fe9cf400640 1 -- 172.21.15.67:0/2247273651 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+4936 (secure 0 0 0) 0x7fe9c80a7070 con 0x7fe9d806d210 2024-04-17T10:28:45.921 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:45.922 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":31,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23}],"filesystems":[{"mdsmap":{"epoch":31,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:01.776873+0000","modified":"2024-04-17T10:28:38.431605+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5772":{"gid":5772,"name":"d","rank":0,"incarnation":24,"state":"up:active","state_seq":12,"addr":"172.21.15.86:6837/2048306700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2048306700},{"type":"v1","addr":"172.21.15.86:6837","nonce":2048306700}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-04-17T10:28:45.922 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 31 2024-04-17T10:28:45.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9b4066f60 msgr2=0x7fe9b4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9b4066f60 0x7fe9b4069420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8162350 tx=0x7fe9c0009040 comp rx=0 tx=0).stop 2024-04-17T10:28:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d806d210 msgr2=0x7fe9d80ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d806d210 0x7fe9d80ca980 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe9c80ab000 tx=0x7fe9c801acf0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 shutdown_connections 2024-04-17T10:28:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe9b4066f60 0x7fe9b4069420 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9d8158170 0x7fe9d8149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9d80ced40 0x7fe9d8165f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 --2- 172.21.15.67:0/2247273651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe9d806d210 0x7fe9d80ca980 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 >> 172.21.15.67:0/2247273651 conn(0x7fe9d8067070 msgr2=0x7fe9d80ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 shutdown_connections 2024-04-17T10:28:45.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:45.930+0000 7fe9dce00640 1 -- 172.21.15.67:0/2247273651 wait complete. 2024-04-17T10:28:45.943 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 Processor -- start 2024-04-17T10:28:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 -- start start 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac64145040 0x7fac6406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac6406ef10 0x7fac64071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac64071850 0x7fac6415f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac640a86c0 con 0x7fac64071850 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6415bf10 con 0x7fac6406ef10 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.090+0000 7fac6a600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac64071ca0 con 0x7fac64145040 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac64071850 0x7fac6415f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac64071850 0x7fac6415f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44212/0 (socket says 172.21.15.67:44212) 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 -- 172.21.15.67:0/2708198271 learned_addr learned my addr 172.21.15.67:0/2708198271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac5aa00640 1 --2- 172.21.15.67:0/2708198271 >> v2:172.21.15.86:3300/0 conn(0x7fac6406ef10 0x7fac64071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63400640 1 --2- 172.21.15.67:0/2708198271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac64145040 0x7fac6406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 -- 172.21.15.67:0/2708198271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac64145040 msgr2=0x7fac6406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:46.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 --2- 172.21.15.67:0/2708198271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac64145040 0x7fac6406e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 -- 172.21.15.67:0/2708198271 >> v2:172.21.15.86:3300/0 conn(0x7fac6406ef10 msgr2=0x7fac64071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:46.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 --2- 172.21.15.67:0/2708198271 >> v2:172.21.15.86:3300/0 conn(0x7fac6406ef10 0x7fac64071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 -- 172.21.15.67:0/2708198271 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6406e2f0 con 0x7fac64071850 2024-04-17T10:28:46.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac63e00640 1 --2- 172.21.15.67:0/2708198271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415f510 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fac6406bac0 tx=0x7fac5c01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862d9edb217c517b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac62a00640 1 -- 172.21.15.67:0/2708198271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c0b4070 con 0x7fac64071850 2024-04-17T10:28:46.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac62a00640 1 -- 172.21.15.67:0/2708198271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c0af400 con 0x7fac64071850 2024-04-17T10:28:46.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.094+0000 7fac62a00640 1 -- 172.21.15.67:0/2708198271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c0af7d0 con 0x7fac64071850 2024-04-17T10:28:46.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- 172.21.15.67:0/2708198271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 msgr2=0x7fac6415f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 --2- 172.21.15.67:0/2708198271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415f510 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fac6406bac0 tx=0x7fac5c01f1e0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- 172.21.15.67:0/2708198271 shutdown_connections 2024-04-17T10:28:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 --2- 172.21.15.67:0/2708198271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415f510 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 --2- 172.21.15.67:0/2708198271 >> v2:172.21.15.86:3300/0 conn(0x7fac6406ef10 0x7fac64071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 --2- 172.21.15.67:0/2708198271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac64145040 0x7fac6406e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- 172.21.15.67:0/2708198271 >> 172.21.15.67:0/2708198271 conn(0x7fac6406ac90 msgr2=0x7fac6406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- 172.21.15.67:0/2708198271 shutdown_connections 2024-04-17T10:28:46.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- 172.21.15.67:0/2708198271 wait complete. 2024-04-17T10:28:46.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 Processor -- start 2024-04-17T10:28:46.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- start start 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6406ef10 0x7fac6413cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac64145040 0x7fac6415dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6406dca0 con 0x7fac64071850 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6406d9c0 con 0x7fac64145040 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac6a600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6406df80 con 0x7fac6406ef10 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac5aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac5aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44220/0 (socket says 172.21.15.67:44220) 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac5aa00640 1 -- 172.21.15.67:0/3169068545 learned_addr learned my addr 172.21.15.67:0/3169068545 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac63400640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6406ef10 0x7fac6413cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.098+0000 7fac63e00640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac64145040 0x7fac6415dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5aa00640 1 -- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6406ef10 msgr2=0x7fac6413cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5aa00640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6406ef10 0x7fac6413cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5aa00640 1 -- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac64145040 msgr2=0x7fac6415dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5aa00640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac64145040 0x7fac6415dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5aa00640 1 -- 172.21.15.67:0/3169068545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6415a5a0 con 0x7fac64071850 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5aa00640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415a060 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00e8b0 tx=0x7fac4c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5be00640 1 -- 172.21.15.67:0/3169068545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c017070 con 0x7fac64071850 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5be00640 1 -- 172.21.15.67:0/3169068545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c00c940 con 0x7fac64071850 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5be00640 1 -- 172.21.15.67:0/3169068545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c00cd10 con 0x7fac64071850 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6413d300 con 0x7fac64071850 2024-04-17T10:28:46.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6413efd0 con 0x7fac64071850 2024-04-17T10:28:46.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac64072d50 con 0x7fac64071850 2024-04-17T10:28:46.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5be00640 1 -- 172.21.15.67:0/3169068545 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fac4c01a070 con 0x7fac64071850 2024-04-17T10:28:46.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5be00640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac40066f60 0x7fac40069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.102+0000 7fac5be00640 1 -- 172.21.15.67:0/3169068545 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fac4c0851b0 con 0x7fac64071850 2024-04-17T10:28:46.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.106+0000 7fac63400640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac40066f60 0x7fac40069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.106+0000 7fac63400640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac40066f60 0x7fac40069420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fac54009ac0 tx=0x7fac54009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.106+0000 7fac5be00640 1 -- 172.21.15.67:0/3169068545 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fac4c055910 con 0x7fac64071850 2024-04-17T10:28:46.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.270+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac64074e30 con 0x7fac64071850 2024-04-17T10:28:46.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.270+0000 7fac5be00640 1 -- 172.21.15.67:0/3169068545 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16934 (secure 0 0 0) 0x7fac4c0599b0 con 0x7fac64071850 2024-04-17T10:28:46.265 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:46.265 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:37.929771+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:00.190460+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:00.765109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.27","mappings":[{"from":0,"to":6}]},{"pgid":"8.37","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac40066f60 msgr2=0x7fac40069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac40066f60 0x7fac40069420 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fac54009ac0 tx=0x7fac54009190 comp rx=0 tx=0).stop 2024-04-17T10:28:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 msgr2=0x7fac6415a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415a060 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00e8b0 tx=0x7fac4c00ed70 comp rx=0 tx=0).stop 2024-04-17T10:28:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 shutdown_connections 2024-04-17T10:28:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac40066f60 0x7fac40069420 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac64145040 0x7fac6415dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac64071850 0x7fac6415a060 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 --2- 172.21.15.67:0/3169068545 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac6406ef10 0x7fac6413cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 >> 172.21.15.67:0/3169068545 conn(0x7fac6406ac90 msgr2=0x7fac6415f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:46.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 shutdown_connections 2024-04-17T10:28:46.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.274+0000 7fac6a600640 1 -- 172.21.15.67:0/3169068545 wait complete. 2024-04-17T10:28:46.286 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:28:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.434+0000 7f6f66a00640 1 Processor -- start 2024-04-17T10:28:46.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.434+0000 7f6f66a00640 1 -- start start 2024-04-17T10:28:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.434+0000 7f6f66a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f60145040 0x7f6f6006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.434+0000 7f6f66a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f60071a20 0x7f6f600cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.434+0000 7f6f66a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6006e2f0 con 0x7f6f6006ef10 2024-04-17T10:28:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.434+0000 7f6f66a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f600a86c0 con 0x7f6f60145040 2024-04-17T10:28:46.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.434+0000 7f6f66a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f6015bf10 con 0x7f6f60071a20 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f60145040 0x7f6f6006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f60145040 0x7f6f6006e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44222/0 (socket says 172.21.15.67:44222) 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 -- 172.21.15.67:0/1747990861 learned_addr learned my addr 172.21.15.67:0/1747990861 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f64a00640 1 --2- 172.21.15.67:0/1747990861 >> v2:172.21.15.86:3300/0 conn(0x7f6f60071a20 0x7f6f600cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 -- 172.21.15.67:0/1747990861 >> v2:172.21.15.86:3300/0 conn(0x7f6f60071a20 msgr2=0x7f6f600cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 --2- 172.21.15.67:0/1747990861 >> v2:172.21.15.86:3300/0 conn(0x7f6f60071a20 0x7f6f600cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 -- 172.21.15.67:0/1747990861 >> v1:172.21.15.86:6789/0 conn(0x7f6f6006ef10 legacy=0x7f6f60071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:46.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 -- 172.21.15.67:0/1747990861 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6006d6e0 con 0x7f6f60145040 2024-04-17T10:28:46.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5fe00640 1 --2- 172.21.15.67:0/1747990861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f60145040 0x7f6f6006e9d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6f600af390 tx=0x7f6f5001ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa40a8d7cd19d09e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5f400640 1 -- 172.21.15.67:0/1747990861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500b5070 con 0x7f6f60145040 2024-04-17T10:28:46.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5f400640 1 -- 172.21.15.67:0/1747990861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f500a89d0 con 0x7f6f60145040 2024-04-17T10:28:46.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.438+0000 7f6f5f400640 1 -- 172.21.15.67:0/1747990861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500a8da0 con 0x7f6f60145040 2024-04-17T10:28:46.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- 172.21.15.67:0/1747990861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f60145040 msgr2=0x7f6f6006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 --2- 172.21.15.67:0/1747990861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f60145040 0x7f6f6006e9d0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f6f600af390 tx=0x7f6f5001ecd0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- 172.21.15.67:0/1747990861 shutdown_connections 2024-04-17T10:28:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 --2- 172.21.15.67:0/1747990861 >> v2:172.21.15.86:3300/0 conn(0x7f6f60071a20 0x7f6f600cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 --2- 172.21.15.67:0/1747990861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f60145040 0x7f6f6006e9d0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- 172.21.15.67:0/1747990861 >> 172.21.15.67:0/1747990861 conn(0x7f6f6006ac90 msgr2=0x7f6f6006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- 172.21.15.67:0/1747990861 shutdown_connections 2024-04-17T10:28:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- 172.21.15.67:0/1747990861 wait complete. 2024-04-17T10:28:46.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 Processor -- start 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- start start 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f6006ef10 0x7f6f6014d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f60071a20 0x7f6f6013b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f60145040 0x7f6f6013ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6006df80 con 0x7f6f6006ef10 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6006dce0 con 0x7f6f60145040 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f66a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6006d370 con 0x7f6f60071a20 2024-04-17T10:28:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f6006ef10 0x7f6f6014d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f6006ef10 0x7f6f6014d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44238/0 (socket says 172.21.15.67:44238) 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 -- 172.21.15.67:0/2595594557 learned_addr learned my addr 172.21.15.67:0/2595594557 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f64a00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f60145040 0x7f6f6013ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f57e00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f60071a20 0x7f6f6013b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 -- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f60071a20 msgr2=0x7f6f6013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f60071a20 0x7f6f6013b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 -- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f60145040 msgr2=0x7f6f6013ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f60145040 0x7f6f6013ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f5fe00640 1 -- 172.21.15.67:0/2595594557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f60138190 con 0x7f6f6006ef10 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.442+0000 7f6f57e00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f60071a20 0x7f6f6013b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f5fe00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f6006ef10 0x7f6f6014d000 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6f5001d380 tx=0x7f6f5001ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f5cc00640 1 -- 172.21.15.67:0/2595594557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500b5070 con 0x7f6f6006ef10 2024-04-17T10:28:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f5cc00640 1 -- 172.21.15.67:0/2595594557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f500b0a50 con 0x7f6f6006ef10 2024-04-17T10:28:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f5cc00640 1 -- 172.21.15.67:0/2595594557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f500b0e40 con 0x7f6f6006ef10 2024-04-17T10:28:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6014ad10 con 0x7f6f6006ef10 2024-04-17T10:28:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6014b2d0 con 0x7f6f6006ef10 2024-04-17T10:28:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f60072d50 con 0x7f6f6006ef10 2024-04-17T10:28:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f5cc00640 1 -- 172.21.15.67:0/2595594557 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6f500c2060 con 0x7f6f6006ef10 2024-04-17T10:28:46.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f5cc00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f3c066f60 0x7f6f3c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f5cc00640 1 -- 172.21.15.67:0/2595594557 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6f50122360 con 0x7f6f6006ef10 2024-04-17T10:28:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.446+0000 7f6f57e00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f3c066f60 0x7f6f3c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.450+0000 7f6f57e00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f3c066f60 0x7f6f3c069420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6f60138840 tx=0x7f6f48009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.450+0000 7f6f5cc00640 1 -- 172.21.15.67:0/2595594557 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6f50018830 con 0x7f6f6006ef10 2024-04-17T10:28:46.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.646+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6f60074e30 con 0x7f6f6006ef10 2024-04-17T10:28:46.640 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:28:46.646+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:28:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f5cc00640 1 -- 172.21.15.67:0/2595594557 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6f500f3110 con 0x7f6f6006ef10 2024-04-17T10:28:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:28:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f3c066f60 msgr2=0x7f6f3c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f3c066f60 0x7f6f3c069420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6f60138840 tx=0x7f6f48009040 comp rx=0 tx=0).stop 2024-04-17T10:28:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f6006ef10 msgr2=0x7f6f6014d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f6006ef10 0x7f6f6014d000 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f6f5001d380 tx=0x7f6f5001ba70 comp rx=0 tx=0).stop 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 shutdown_connections 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f3c066f60 0x7f6f3c069420 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f60145040 0x7f6f6013ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f60071a20 0x7f6f6013b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.650+0000 7f6f66a00640 1 --2- 172.21.15.67:0/2595594557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f6006ef10 0x7f6f6014d000 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.654+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 >> 172.21.15.67:0/2595594557 conn(0x7f6f6006ac90 msgr2=0x7f6f600af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.654+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 shutdown_connections 2024-04-17T10:28:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.654+0000 7f6f66a00640 1 -- 172.21.15.67:0/2595594557 wait complete. 2024-04-17T10:28:46.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:28:46.717 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:28:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbd01e00640 1 Processor -- start 2024-04-17T10:28:46.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbd01e00640 1 -- start start 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbd01e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcfc15b980 0x7fbcfc0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbd01e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc09a240 0x7fbcfc0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbd01e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcfc0cca60 con 0x7fbcfc15bd60 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbd01e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcfc0ccd40 con 0x7fbcfc15b980 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbd01e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0cd020 con 0x7fbcfc09a240 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcfc15b980 0x7fbcfc0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfa000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcfc15bd60 0x7fbcfc099b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46196/0 (socket says 172.21.15.67:46196) 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfa000640 1 -- 172.21.15.67:0/2845094039 learned_addr learned my addr 172.21.15.67:0/2845094039 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:46.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfb400640 1 --2- 172.21.15.67:0/2845094039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc09a240 0x7fbcfc0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfaa00640 1 -- 172.21.15.67:0/2845094039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc09a240 msgr2=0x7fbcfc0cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfaa00640 1 --2- 172.21.15.67:0/2845094039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc09a240 0x7fbcfc0cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfaa00640 1 -- 172.21.15.67:0/2845094039 >> v1:172.21.15.67:6789/0 conn(0x7fbcfc15bd60 legacy=0x7fbcfc099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcf9600640 1 -- 172.21.15.67:0/2845094039 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233773293 0 0) 0x7fbcfc0cca60 con 0x7fbcfc15bd60 2024-04-17T10:28:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.818+0000 7fbcfaa00640 1 -- 172.21.15.67:0/2845094039 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc0cd300 con 0x7fbcfc15b980 2024-04-17T10:28:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbcfaa00640 1 --2- 172.21.15.67:0/2845094039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15b980 0x7fbcfc0cc1f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0097c0 tx=0x7fbcec01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3268976997b77800 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbcf9600640 1 -- 172.21.15.67:0/2845094039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0b5070 con 0x7fbcfc15b980 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbcf9600640 1 -- 172.21.15.67:0/2845094039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcec0a8990 con 0x7fbcfc15b980 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbcf9600640 1 -- 172.21.15.67:0/2845094039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec0a8d60 con 0x7fbcfc15b980 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 -- 172.21.15.67:0/2845094039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15b980 msgr2=0x7fbcfc0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 --2- 172.21.15.67:0/2845094039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15b980 0x7fbcfc0cc1f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0097c0 tx=0x7fbcec01ec90 comp rx=0 tx=0).stop 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 -- 172.21.15.67:0/2845094039 shutdown_connections 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 --2- 172.21.15.67:0/2845094039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc09a240 0x7fbcfc0cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 --2- 172.21.15.67:0/2845094039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15b980 0x7fbcfc0cc1f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 -- 172.21.15.67:0/2845094039 >> 172.21.15.67:0/2845094039 conn(0x7fbcfc0b33a0 msgr2=0x7fbcfc0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:46.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 -- 172.21.15.67:0/2845094039 shutdown_connections 2024-04-17T10:28:46.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.822+0000 7fbd01e00640 1 -- 172.21.15.67:0/2845094039 wait complete. 2024-04-17T10:28:46.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 Processor -- start 2024-04-17T10:28:46.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 -- start start 2024-04-17T10:28:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcfc09a240 0x7fbcfc0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc15b980 0x7fbcfc0c44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15bd60 0x7fbcfc0c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0cfdf0 con 0x7fbcfc15bd60 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0cc730 con 0x7fbcfc09a240 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc0cff70 con 0x7fbcfc15b980 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15bd60 0x7fbcfc0c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15bd60 0x7fbcfc0c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44258/0 (socket says 172.21.15.67:44258) 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 -- 172.21.15.67:0/3751541114 learned_addr learned my addr 172.21.15.67:0/3751541114 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfaa00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcfc09a240 0x7fbcfc0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfa000640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc15b980 0x7fbcfc0c44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 -- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc15b980 msgr2=0x7fbcfc0c44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc15b980 0x7fbcfc0c44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 -- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcfc09a240 msgr2=0x7fbcfc0c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcfc09a240 0x7fbcfc0c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:46.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 -- 172.21.15.67:0/3751541114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc0c5f10 con 0x7fbcfc15bd60 2024-04-17T10:28:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcfb400640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15bd60 0x7fbcfc0c76c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400efc0 tx=0x7fbcf400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcf3400640 1 -- 172.21.15.67:0/3751541114 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4018020 con 0x7fbcfc15bd60 2024-04-17T10:28:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcf3400640 1 -- 172.21.15.67:0/3751541114 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf4014480 con 0x7fbcfc15bd60 2024-04-17T10:28:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbcf3400640 1 -- 172.21.15.67:0/3751541114 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4014870 con 0x7fbcfc15bd60 2024-04-17T10:28:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc06d190 con 0x7fbcfc15bd60 2024-04-17T10:28:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.826+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc15cd40 con 0x7fbcfc15bd60 2024-04-17T10:28:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.830+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc04ef30 con 0x7fbcfc15bd60 2024-04-17T10:28:46.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.830+0000 7fbcf3400640 1 -- 172.21.15.67:0/3751541114 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbcf401b020 con 0x7fbcfc15bd60 2024-04-17T10:28:46.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.830+0000 7fbcf3400640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcdc066f60 0x7fbcdc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:46.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.830+0000 7fbcf3400640 1 -- 172.21.15.67:0/3751541114 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbcf4085fe0 con 0x7fbcfc15bd60 2024-04-17T10:28:46.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.830+0000 7fbcfaa00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcdc066f60 0x7fbcdc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.830+0000 7fbcfaa00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcdc066f60 0x7fbcdc069420 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0b2000 tx=0x7fbcec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:46.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:46.834+0000 7fbcf3400640 1 -- 172.21.15.67:0/3751541114 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbcf4022030 con 0x7fbcfc15bd60 2024-04-17T10:28:47.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.034+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbcfc0cbdd0 con 0x7fbcfc15bd60 2024-04-17T10:28:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.650+0000 7fbcf3400640 1 -- 172.21.15.67:0/3751541114 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbcfc0c5f10 con 0x7fbcfc15bd60 2024-04-17T10:28:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcdc066f60 msgr2=0x7fbcdc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcdc066f60 0x7fbcdc069420 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0b2000 tx=0x7fbcec008040 comp rx=0 tx=0).stop 2024-04-17T10:28:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15bd60 msgr2=0x7fbcfc0c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:47.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15bd60 0x7fbcfc0c76c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fbcf400efc0 tx=0x7fbcf400c490 comp rx=0 tx=0).stop 2024-04-17T10:28:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 shutdown_connections 2024-04-17T10:28:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcdc066f60 0x7fbcdc069420 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcfc15bd60 0x7fbcfc0c76c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcfc15b980 0x7fbcfc0c44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 --2- 172.21.15.67:0/3751541114 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcfc09a240 0x7fbcfc0c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 >> 172.21.15.67:0/3751541114 conn(0x7fbcfc0b33a0 msgr2=0x7fbcfc0b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 shutdown_connections 2024-04-17T10:28:47.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.654+0000 7fbd01e00640 1 -- 172.21.15.67:0/3751541114 wait complete. 2024-04-17T10:28:47.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:28:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bcd600640 1 Processor -- start 2024-04-17T10:28:47.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bcd600640 1 -- start start 2024-04-17T10:28:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bcd600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bc81447e0 0x7f9bc806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bcd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc8144f00 0x7f9bc8071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bcd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc806d9a0 con 0x7f9bc80718a0 2024-04-17T10:28:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bcd600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc8071d10 con 0x7f9bc81447e0 2024-04-17T10:28:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bcd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80cc310 con 0x7f9bc8144f00 2024-04-17T10:28:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bc7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9bc80718a0 0x7f9bc815f490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46204/0 (socket says 172.21.15.67:46204) 2024-04-17T10:28:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bc6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9bc81447e0 0x7f9bc806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bc7400640 1 -- 172.21.15.67:0/1479595602 learned_addr learned my addr 172.21.15.67:0/1479595602 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bbfe00640 1 --2- 172.21.15.67:0/1479595602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc8144f00 0x7f9bc8071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bc6a00640 1 -- 172.21.15.67:0/1479595602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc8144f00 msgr2=0x7f9bc8071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bc6a00640 1 --2- 172.21.15.67:0/1479595602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc8144f00 0x7f9bc8071360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bc6a00640 1 -- 172.21.15.67:0/1479595602 >> v1:172.21.15.67:6789/0 conn(0x7f9bc80718a0 legacy=0x7f9bc815f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bc6a00640 1 -- 172.21.15.67:0/1479595602 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc806dc80 con 0x7f9bc81447e0 2024-04-17T10:28:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.826+0000 7f9bbfe00640 1 --2- 172.21.15.67:0/1479595602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc8144f00 0x7f9bc8071360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bc6a00640 1 --2- 172.21.15.67:0/1479595602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc81447e0 0x7f9bc806d3c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80097c0 tx=0x7f9bb801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626f37395d69ceb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bc6000640 1 -- 172.21.15.67:0/1479595602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb80b1070 con 0x7f9bc81447e0 2024-04-17T10:28:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bc6000640 1 -- 172.21.15.67:0/1479595602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb80a8050 con 0x7f9bc81447e0 2024-04-17T10:28:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bc6000640 1 -- 172.21.15.67:0/1479595602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb80ac860 con 0x7f9bc81447e0 2024-04-17T10:28:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 -- 172.21.15.67:0/1479595602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc81447e0 msgr2=0x7f9bc806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:47.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 --2- 172.21.15.67:0/1479595602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc81447e0 0x7f9bc806d3c0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80097c0 tx=0x7f9bb801c3e0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 -- 172.21.15.67:0/1479595602 shutdown_connections 2024-04-17T10:28:47.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 --2- 172.21.15.67:0/1479595602 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc8144f00 0x7f9bc8071360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 --2- 172.21.15.67:0/1479595602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc81447e0 0x7f9bc806d3c0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 -- 172.21.15.67:0/1479595602 >> 172.21.15.67:0/1479595602 conn(0x7f9bc806ac90 msgr2=0x7f9bc806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:47.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 -- 172.21.15.67:0/1479595602 shutdown_connections 2024-04-17T10:28:47.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.830+0000 7f9bcd600640 1 -- 172.21.15.67:0/1479595602 wait complete. 2024-04-17T10:28:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 Processor -- start 2024-04-17T10:28:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 -- start start 2024-04-17T10:28:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bc80718a0 0x7f9bc813f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc81447e0 0x7f9bc813e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc8144f00 0x7f9bc813b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8168970 con 0x7f9bc8144f00 2024-04-17T10:28:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81687f0 con 0x7f9bc80718a0 2024-04-17T10:28:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8168af0 con 0x7f9bc81447e0 2024-04-17T10:28:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc8144f00 0x7f9bc813b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc8144f00 0x7f9bc813b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44278/0 (socket says 172.21.15.67:44278) 2024-04-17T10:28:47.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 -- 172.21.15.67:0/645851814 learned_addr learned my addr 172.21.15.67:0/645851814 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bbfe00640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc81447e0 0x7f9bc813e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc6a00640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bc80718a0 0x7f9bc813f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 -- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc81447e0 msgr2=0x7f9bc813e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc81447e0 0x7f9bc813e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 -- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bc80718a0 msgr2=0x7f9bc813f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:47.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bc80718a0 0x7f9bc813f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 -- 172.21.15.67:0/645851814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc813a130 con 0x7f9bc8144f00 2024-04-17T10:28:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc6a00640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bc80718a0 0x7f9bc813f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bc7400640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc8144f00 0x7f9bc813b0a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000cbc0 tx=0x7f9bc000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bbea00640 1 -- 172.21.15.67:0/645851814 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0017070 con 0x7f9bc8144f00 2024-04-17T10:28:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bbea00640 1 -- 172.21.15.67:0/645851814 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc000ad40 con 0x7f9bc8144f00 2024-04-17T10:28:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bbea00640 1 -- 172.21.15.67:0/645851814 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0012600 con 0x7f9bc8144f00 2024-04-17T10:28:47.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc813a410 con 0x7f9bc8144f00 2024-04-17T10:28:47.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.834+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc813a9d0 con 0x7f9bc8144f00 2024-04-17T10:28:47.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.838+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc8072da0 con 0x7f9bc8144f00 2024-04-17T10:28:47.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.838+0000 7f9bbea00640 1 -- 172.21.15.67:0/645851814 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9bc001a070 con 0x7f9bc8144f00 2024-04-17T10:28:47.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.838+0000 7f9bbea00640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ba4066f60 0x7f9ba4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:47.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.838+0000 7f9bbea00640 1 -- 172.21.15.67:0/645851814 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f9bc0084f80 con 0x7f9bc8144f00 2024-04-17T10:28:47.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.838+0000 7f9bc6a00640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ba4066f60 0x7f9ba4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:47.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.838+0000 7f9bc6a00640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ba4066f60 0x7f9ba4069420 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80ae000 tx=0x7f9bb80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:47.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:47.842+0000 7f9bbea00640 1 -- 172.21.15.67:0/645851814 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9bc0055680 con 0x7f9bc8144f00 2024-04-17T10:28:48.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.006+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9bc806b070 con 0x7f9bc8144f00 2024-04-17T10:28:48.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.974+0000 7f9bbea00640 1 -- 172.21.15.67:0/645851814 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9bc0059720 con 0x7f9bc8144f00 2024-04-17T10:28:48.968 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:28:48.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ba4066f60 msgr2=0x7f9ba4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ba4066f60 0x7f9ba4069420 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80ae000 tx=0x7f9bb80a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc8144f00 msgr2=0x7f9bc813b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:48.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc8144f00 0x7f9bc813b0a0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f9bc000cbc0 tx=0x7f9bc000a430 comp rx=0 tx=0).stop 2024-04-17T10:28:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 shutdown_connections 2024-04-17T10:28:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9ba4066f60 0x7f9ba4069420 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:48.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9bc8144f00 0x7f9bc813b0a0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9bc81447e0 0x7f9bc813e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 --2- 172.21.15.67:0/645851814 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9bc80718a0 0x7f9bc813f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.978+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 >> 172.21.15.67:0/645851814 conn(0x7f9bc806ac90 msgr2=0x7f9bc80cc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.982+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 shutdown_connections 2024-04-17T10:28:48.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:48.982+0000 7f9bcd600640 1 -- 172.21.15.67:0/645851814 wait complete. 2024-04-17T10:28:49.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:28:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.158+0000 7ff08b400640 1 Processor -- start 2024-04-17T10:28:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff08b400640 1 -- start start 2024-04-17T10:28:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff08b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff08406cd70 0x7ff08415de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff08b400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08415f150 con 0x7ff08415e3d0 2024-04-17T10:28:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff08b400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08415e8b0 con 0x7ff084099b30 2024-04-17T10:28:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff08b400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08406c0f0 con 0x7ff08406cd70 2024-04-17T10:28:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff07be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff08406cd70 0x7ff08415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff089400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff08415e3d0 0x7ff0840d00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46210/0 (socket says 172.21.15.67:46210) 2024-04-17T10:28:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff089400640 1 -- 172.21.15.67:0/446624413 learned_addr learned my addr 172.21.15.67:0/446624413 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff07be00640 1 -- 172.21.15.67:0/446624413 >> v1:172.21.15.86:6789/0 conn(0x7ff084099b30 legacy=0x7ff084099f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff07be00640 1 -- 172.21.15.67:0/446624413 >> v1:172.21.15.67:6789/0 conn(0x7ff08415e3d0 legacy=0x7ff0840d00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff07be00640 1 -- 172.21.15.67:0/446624413 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08415eb90 con 0x7ff08406cd70 2024-04-17T10:28:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.162+0000 7ff083e00640 1 -- 172.21.15.67:0/446624413 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150467477 0 0) 0x7ff08415f150 con 0x7ff08415e3d0 2024-04-17T10:28:49.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff07be00640 1 --2- 172.21.15.67:0/446624413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff08406cd70 0x7ff08415de90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0097c0 tx=0x7ff06c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cabc44d2f434a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:49.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff083e00640 1 -- 172.21.15.67:0/446624413 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0b0070 con 0x7ff08406cd70 2024-04-17T10:28:49.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff083e00640 1 -- 172.21.15.67:0/446624413 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c0ab600 con 0x7ff08406cd70 2024-04-17T10:28:49.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff083e00640 1 -- 172.21.15.67:0/446624413 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0ab9d0 con 0x7ff08406cd70 2024-04-17T10:28:49.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 -- 172.21.15.67:0/446624413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff08406cd70 msgr2=0x7ff08415de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:49.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 --2- 172.21.15.67:0/446624413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff08406cd70 0x7ff08415de90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0097c0 tx=0x7ff06c01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 -- 172.21.15.67:0/446624413 shutdown_connections 2024-04-17T10:28:49.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 --2- 172.21.15.67:0/446624413 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff08406cd70 0x7ff08415de90 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 -- 172.21.15.67:0/446624413 >> 172.21.15.67:0/446624413 conn(0x7ff0840b32e0 msgr2=0x7ff0840b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:49.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 -- 172.21.15.67:0/446624413 shutdown_connections 2024-04-17T10:28:49.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 -- 172.21.15.67:0/446624413 wait complete. 2024-04-17T10:28:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 Processor -- start 2024-04-17T10:28:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.166+0000 7ff08b400640 1 -- start start 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff08406cd70 0x7ff08414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff084099b30 0x7ff0840b6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff08415e3d0 0x7ff0840c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08416bb40 con 0x7ff084099b30 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0840d1220 con 0x7ff08406cd70 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08416bcc0 con 0x7ff08415e3d0 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff084099b30 0x7ff0840b6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff084099b30 0x7ff0840b6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44306/0 (socket says 172.21.15.67:44306) 2024-04-17T10:28:49.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 -- 172.21.15.67:0/1421985079 learned_addr learned my addr 172.21.15.67:0/1421985079 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff088a00640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff08406cd70 0x7ff08414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff089400640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff08415e3d0 0x7ff0840c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 -- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff08415e3d0 msgr2=0x7ff0840c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff08415e3d0 0x7ff0840c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 -- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff08406cd70 msgr2=0x7ff08414ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff08406cd70 0x7ff08414ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 -- 172.21.15.67:0/1421985079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08414bfc0 con 0x7ff084099b30 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff088a00640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff08406cd70 0x7ff08414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff07be00640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff084099b30 0x7ff0840b6460 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0ae040 tx=0x7ff06c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff081600640 1 -- 172.21.15.67:0/1421985079 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0b0040 con 0x7ff084099b30 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff081600640 1 -- 172.21.15.67:0/1421985079 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06c0b4410 con 0x7ff084099b30 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff081600640 1 -- 172.21.15.67:0/1421985079 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c0b4800 con 0x7ff084099b30 2024-04-17T10:28:49.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0840c5ed0 con 0x7ff084099b30 2024-04-17T10:28:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0840c6490 con 0x7ff084099b30 2024-04-17T10:28:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08409f900 con 0x7ff084099b30 2024-04-17T10:28:49.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.170+0000 7ff081600640 1 -- 172.21.15.67:0/1421985079 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff06c0bc020 con 0x7ff084099b30 2024-04-17T10:28:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.174+0000 7ff081600640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff060066f60 0x7ff060069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.174+0000 7ff081600640 1 -- 172.21.15.67:0/1421985079 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5807+0+0 (secure 0 0 0) 0x7ff06c11e6b0 con 0x7ff084099b30 2024-04-17T10:28:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.174+0000 7ff088a00640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff060066f60 0x7ff060069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:49.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.174+0000 7ff088a00640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff060066f60 0x7ff060069420 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff0840b39a0 tx=0x7ff074009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.178+0000 7ff081600640 1 -- 172.21.15.67:0/1421985079 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff06c0c3020 con 0x7ff084099b30 2024-04-17T10:28:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.338+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff084074e30 con 0x7ff084099b30 2024-04-17T10:28:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.978+0000 7ff081600640 1 -- 172.21.15.67:0/1421985079 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff06c016090 con 0x7ff084099b30 2024-04-17T10:28:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:28:49.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.978+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff060066f60 msgr2=0x7ff060069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:49.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.978+0000 7ff08b400640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff060066f60 0x7ff060069420 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff0840b39a0 tx=0x7ff074009040 comp rx=0 tx=0).stop 2024-04-17T10:28:49.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.978+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff084099b30 msgr2=0x7ff0840b6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:49.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.978+0000 7ff08b400640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff084099b30 0x7ff0840b6460 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ff06c0ae040 tx=0x7ff06c019240 comp rx=0 tx=0).stop 2024-04-17T10:28:49.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 shutdown_connections 2024-04-17T10:28:49.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff060066f60 0x7ff060069420 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff08415e3d0 0x7ff0840c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff084099b30 0x7ff0840b6460 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 --2- 172.21.15.67:0/1421985079 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff08406cd70 0x7ff08414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:49.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 >> 172.21.15.67:0/1421985079 conn(0x7ff0840b32e0 msgr2=0x7ff084155aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:49.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 shutdown_connections 2024-04-17T10:28:49.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:49.982+0000 7ff08b400640 1 -- 172.21.15.67:0/1421985079 wait complete. 2024-04-17T10:28:50.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:28:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.170+0000 7f065a200640 1 Processor -- start 2024-04-17T10:28:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.170+0000 7f065a200640 1 -- start start 2024-04-17T10:28:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f065a200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0654161f90 0x7f06540cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f065a200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065415e140 con 0x7f065409adf0 2024-04-17T10:28:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f065a200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0654099d50 con 0x7f065415dd60 2024-04-17T10:28:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f065a200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06541626d0 con 0x7f0654161f90 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f065409adf0 0x7f065406bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46222/0 (socket says 172.21.15.67:46222) 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653e00640 1 -- 172.21.15.67:0/2914082791 learned_addr learned my addr 172.21.15.67:0/2914082791 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f064be00640 1 --2- 172.21.15.67:0/2914082791 >> v2:172.21.15.86:3300/0 conn(0x7f0654161f90 0x7f06540cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188261182 0 0) 0x7f065415e140 con 0x7f065409adf0 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0638003530 con 0x7f065409adf0 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024961667 0 0) 0x7f0654099d50 con 0x7f065415dd60 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f065415e140 con 0x7f065415dd60 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1256224718 0 0) 0x7f0638003530 con 0x7f065409adf0 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 >> v2:172.21.15.86:3300/0 conn(0x7f0654161f90 msgr2=0x7f06540cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 --2- 172.21.15.67:0/2914082791 >> v2:172.21.15.86:3300/0 conn(0x7f0654161f90 0x7f06540cff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 >> v1:172.21.15.86:6789/0 conn(0x7f065415dd60 legacy=0x7f0654161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065409a030 con 0x7f065409adf0 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f064c001a00 con 0x7f065409adf0 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 >> v1:172.21.15.67:6789/0 conn(0x7f065409adf0 legacy=0x7f065406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f0638003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0638004810 0x7f0638004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06380051d0 0x7f063800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06541626d0 con 0x7f0638003ba0 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063800c0a0 con 0x7f0638004810 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f063800c2a0 con 0x7f06380051d0 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653400640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f0638003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f064be00640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0638004810 0x7f0638004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653e00640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06380051d0 0x7f063800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653400640 1 -- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06380051d0 msgr2=0x7f063800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653400640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06380051d0 0x7f063800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653400640 1 -- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0638004810 msgr2=0x7f0638004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:50.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653400640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0638004810 0x7f0638004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653400640 1 -- 172.21.15.67:0/2914082791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0638003530 con 0x7f0638003ba0 2024-04-17T10:28:50.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.174+0000 7f0653e00640 1 -- 172.21.15.67:0/2914082791 reap_dead start 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f0653400640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f0638003fa0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f06440018d0 tx=0x7f064401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6caa9a11b6293544 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06440af070 con 0x7f0638003ba0 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06440052f0 con 0x7f0638003ba0 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f0652a00640 1 -- 172.21.15.67:0/2914082791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644001520 con 0x7f0638003ba0 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f065a200640 1 -- 172.21.15.67:0/2914082791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 msgr2=0x7f0638003fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f065a200640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f0638003fa0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f06440018d0 tx=0x7f064401c4f0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f065a200640 1 -- 172.21.15.67:0/2914082791 shutdown_connections 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f065a200640 1 --2- 172.21.15.67:0/2914082791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f0638003fa0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f065a200640 1 -- 172.21.15.67:0/2914082791 >> 172.21.15.67:0/2914082791 conn(0x7f06540b34a0 msgr2=0x7f06540b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:50.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f065a200640 1 -- 172.21.15.67:0/2914082791 shutdown_connections 2024-04-17T10:28:50.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.178+0000 7f065a200640 1 -- 172.21.15.67:0/2914082791 wait complete. 2024-04-17T10:28:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 Processor -- start 2024-04-17T10:28:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 -- start start 2024-04-17T10:28:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f06541f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540b15f0 0x7f06540af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06540afc90 0x7f06541f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0654161c90 con 0x7f0638003ba0 2024-04-17T10:28:50.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065409a8d0 con 0x7f06540b15f0 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0654161e10 con 0x7f06540afc90 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f06541f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f06541f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44330/0 (socket says 172.21.15.67:44330) 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 -- 172.21.15.67:0/906330529 learned_addr learned my addr 172.21.15.67:0/906330529 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f064be00640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540b15f0 0x7f06540af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653e00640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06540afc90 0x7f06541f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 -- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06540afc90 msgr2=0x7f06541f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06540afc90 0x7f06541f9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 -- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540b15f0 msgr2=0x7f06540af750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540b15f0 0x7f06540af750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 -- 172.21.15.67:0/906330529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06540b24a0 con 0x7f0638003ba0 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f0653400640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f06541f6270 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f06440ad040 tx=0x7f0638005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f064b400640 1 -- 172.21.15.67:0/906330529 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06440af040 con 0x7f0638003ba0 2024-04-17T10:28:50.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f064b400640 1 -- 172.21.15.67:0/906330529 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06440067e0 con 0x7f0638003ba0 2024-04-17T10:28:50.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06540b2780 con 0x7f0638003ba0 2024-04-17T10:28:50.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f064b400640 1 -- 172.21.15.67:0/906330529 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644006bd0 con 0x7f0638003ba0 2024-04-17T10:28:50.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06540c54a0 con 0x7f0638003ba0 2024-04-17T10:28:50.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.182+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065404ef30 con 0x7f0638003ba0 2024-04-17T10:28:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.186+0000 7f064b400640 1 -- 172.21.15.67:0/906330529 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f06440b2070 con 0x7f0638003ba0 2024-04-17T10:28:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.186+0000 7f064b400640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0630066f90 0x7f0630069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.186+0000 7f064b400640 1 -- 172.21.15.67:0/906330529 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f064411e0d0 con 0x7f0638003ba0 2024-04-17T10:28:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.186+0000 7f064be00640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0630066f90 0x7f0630069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.186+0000 7f064be00640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0630066f90 0x7f0630069450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f06540b03e0 tx=0x7f063c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.190+0000 7f064b400640 1 -- 172.21.15.67:0/906330529 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0644016230 con 0x7f0638003ba0 2024-04-17T10:28:50.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.354+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f06540b42d0 con 0x7f0638003ba0 2024-04-17T10:28:50.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f064b400640 1 -- 172.21.15.67:0/906330529 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f06440ef1b0 con 0x7f0638003ba0 2024-04-17T10:28:50.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0630066f90 msgr2=0x7f0630069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0630066f90 0x7f0630069450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f06540b03e0 tx=0x7f063c008040 comp rx=0 tx=0).stop 2024-04-17T10:28:50.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 msgr2=0x7f06541f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f06541f6270 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f06440ad040 tx=0x7f0638005840 comp rx=0 tx=0).stop 2024-04-17T10:28:50.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 shutdown_connections 2024-04-17T10:28:50.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0630066f90 0x7f0630069450 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06540afc90 0x7f06541f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540b15f0 0x7f06540af750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 --2- 172.21.15.67:0/906330529 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0638003ba0 0x7f06541f6270 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 >> 172.21.15.67:0/906330529 conn(0x7f06540b34a0 msgr2=0x7f0654070330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 shutdown_connections 2024-04-17T10:28:50.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.358+0000 7f065a200640 1 -- 172.21.15.67:0/906330529 wait complete. 2024-04-17T10:28:50.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks' 2024-04-17T10:28:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046be00640 1 Processor -- start 2024-04-17T10:28:50.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046be00640 1 -- start start 2024-04-17T10:28:50.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 0x7f046c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f046c09a740 0x7f046c0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046be00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046c0cde20 con 0x7f046c0d03a0 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046be00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046c0cf300 con 0x7f046c09a740 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046be00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0cf5e0 con 0x7f046c099a50 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f046c0d03a0 0x7f046c099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46280/0 (socket says 172.21.15.67:46280) 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046a000640 1 -- 172.21.15.67:0/4242190259 learned_addr learned my addr 172.21.15.67:0/4242190259 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046b400640 1 --2- 172.21.15.67:0/4242190259 >> v2:172.21.15.86:3300/0 conn(0x7f046c09a740 0x7f046c0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046aa00640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 0x7f046c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636530190 0 0) 0x7f046c0cde20 con 0x7f046c0d03a0 2024-04-17T10:28:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450003530 con 0x7f046c0d03a0 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517513903 0 0) 0x7f0450003530 con 0x7f046c0d03a0 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 msgr2=0x7f046c15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 0x7f046c15c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 >> v2:172.21.15.86:3300/0 conn(0x7f046c09a740 msgr2=0x7f046c0cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 --2- 172.21.15.67:0/4242190259 >> v2:172.21.15.86:3300/0 conn(0x7f046c09a740 0x7f046c0cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c0cf8c0 con 0x7f046c0d03a0 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f0458001680 con 0x7f046c0d03a0 2024-04-17T10:28:50.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 >> v1:172.21.15.67:6789/0 conn(0x7f046c0d03a0 legacy=0x7f046c099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 0x7f04500039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0450003f40 0x7f0450004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0450004d50 0x7f04500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0cf300 con 0x7f0450004d50 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0cf5e0 con 0x7f0450003f40 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045000b9d0 con 0x7f046c099a50 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046b400640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0450004d50 0x7f04500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046a000640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0450003f40 0x7f0450004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046aa00640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 0x7f04500039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046b400640 1 -- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 msgr2=0x7f04500039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:50.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046b400640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 0x7f04500039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046b400640 1 -- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0450003f40 msgr2=0x7f0450004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:50.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046b400640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0450003f40 0x7f0450004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046b400640 1 -- 172.21.15.67:0/4242190259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450003530 con 0x7f0450004d50 2024-04-17T10:28:50.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.522+0000 7f046aa00640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c099a50 0x7f04500039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046b400640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0450004d50 0x7f04500051d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0464009fc0 tx=0x7f046401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c3e9b393387a9bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04640b2070 con 0x7f0450004d50 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04640a5cd0 con 0x7f0450004d50 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f0469600640 1 -- 172.21.15.67:0/4242190259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464009c10 con 0x7f0450004d50 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046be00640 1 -- 172.21.15.67:0/4242190259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0450004d50 msgr2=0x7f04500051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046be00640 1 --2- 172.21.15.67:0/4242190259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0450004d50 0x7f04500051d0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f0464009fc0 tx=0x7f046401cc60 comp rx=0 tx=0).stop 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046b400640 1 -- 172.21.15.67:0/4242190259 reap_dead start 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046be00640 1 -- 172.21.15.67:0/4242190259 shutdown_connections 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046be00640 1 -- 172.21.15.67:0/4242190259 >> 172.21.15.67:0/4242190259 conn(0x7f046c0d1690 msgr2=0x7f046c0d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046be00640 1 -- 172.21.15.67:0/4242190259 shutdown_connections 2024-04-17T10:28:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.526+0000 7f046be00640 1 -- 172.21.15.67:0/4242190259 wait complete. 2024-04-17T10:28:50.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 Processor -- start 2024-04-17T10:28:50.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 -- start start 2024-04-17T10:28:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046c14bc50 0x7f046c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c0b6810 0x7f046c14c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f046c14c570 0x7f046c0ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c15c7a0 con 0x7f046c14bc50 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046c0cfe80 con 0x7f046c14c570 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046c15c920 con 0x7f046c0b6810 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046c14bc50 0x7f046c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046c14bc50 0x7f046c0b62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59618/0 (socket says 172.21.15.67:59618) 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 -- 172.21.15.67:0/900834776 learned_addr learned my addr 172.21.15.67:0/900834776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 -- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c0b6810 msgr2=0x7f046c14c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c0b6810 0x7f046c14c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 -- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f046c14c570 msgr2=0x7f046c0ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046b400640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f046c14c570 0x7f046c0ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f046c14c570 0x7f046c0ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 -- 172.21.15.67:0/900834776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046c06c410 con 0x7f046c14bc50 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046b400640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f046c14c570 0x7f046c0ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046aa00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046c14bc50 0x7f046c0b62d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f045c0115f0 tx=0x7f045c011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f0463400640 1 -- 172.21.15.67:0/900834776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c004220 con 0x7f046c14bc50 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f0463400640 1 -- 172.21.15.67:0/900834776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f045c0043c0 con 0x7f046c14bc50 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046c14c9f0 con 0x7f046c14bc50 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f0463400640 1 -- 172.21.15.67:0/900834776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f045c00f8f0 con 0x7f046c14bc50 2024-04-17T10:28:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.530+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046c14cf90 con 0x7f046c14bc50 2024-04-17T10:28:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.534+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046c074dc0 con 0x7f046c14bc50 2024-04-17T10:28:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.534+0000 7f0463400640 1 -- 172.21.15.67:0/900834776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f045c00fa90 con 0x7f046c14bc50 2024-04-17T10:28:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.534+0000 7f0463400640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f044c066f90 0x7f044c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.534+0000 7f0463400640 1 -- 172.21.15.67:0/900834776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f045c085240 con 0x7f046c14bc50 2024-04-17T10:28:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.534+0000 7f046a000640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f044c066f90 0x7f044c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:50.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.534+0000 7f046a000640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f044c066f90 0x7f044c069450 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f046c0c5d50 tx=0x7f0458007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:50.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.538+0000 7f0463400640 1 -- 172.21.15.67:0/900834776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f045c019030 con 0x7f046c14bc50 2024-04-17T10:28:50.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.762+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]} v 0) v1 -- 0x7f046c0b6c50 con 0x7f046c14bc50 2024-04-17T10:28:50.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f0463400640 1 -- 172.21.15.67:0/900834776 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]}]=0 v207) v1 ==== 143+0+0 (secure 0 0 0) 0x7f045c056320 con 0x7f046c14bc50 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f044c066f90 msgr2=0x7f044c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f044c066f90 0x7f044c069450 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f046c0c5d50 tx=0x7f0458007c20 comp rx=0 tx=0).stop 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046c14bc50 msgr2=0x7f046c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046c14bc50 0x7f046c0b62d0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f045c0115f0 tx=0x7f045c011ab0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 shutdown_connections 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f044c066f90 0x7f044c069450 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f046c14c570 0x7f046c0ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f046c0b6810 0x7f046c14c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 --2- 172.21.15.67:0/900834776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f046c14bc50 0x7f046c0b62d0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 >> 172.21.15.67:0/900834776 conn(0x7f046c0d1690 msgr2=0x7f046c15a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:50.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 shutdown_connections 2024-04-17T10:28:50.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:50.990+0000 7f046be00640 1 -- 172.21.15.67:0/900834776 wait complete. 2024-04-17T10:28:51.003 INFO:tasks.cephfs_test_runner:test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ok 2024-04-17T10:28:51.004 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-04-17T10:28:51.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.150+0000 7f940e000640 1 Processor -- start 2024-04-17T10:28:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.150+0000 7f940e000640 1 -- start start 2024-04-17T10:28:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f940e000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94080cf0c0 0x7f94081441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f940e000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940815ae40 con 0x7f940806eac0 2024-04-17T10:28:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f940e000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9408071440 con 0x7f940806e6e0 2024-04-17T10:28:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f940e000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9408071720 con 0x7f94080cf0c0 2024-04-17T10:28:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f93fea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f940806eac0 0x7f9408143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46296/0 (socket says 172.21.15.67:46296) 2024-04-17T10:28:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f93fea00640 1 -- 172.21.15.67:0/218005847 learned_addr learned my addr 172.21.15.67:0/218005847 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f9407e00640 1 --2- 172.21.15.67:0/218005847 >> v2:172.21.15.86:3300/0 conn(0x7f94080cf0c0 0x7f94081441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f9406a00640 1 -- 172.21.15.67:0/218005847 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304052205 0 0) 0x7f940815ae40 con 0x7f940806eac0 2024-04-17T10:28:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f9406a00640 1 -- 172.21.15.67:0/218005847 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ec003530 con 0x7f940806eac0 2024-04-17T10:28:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f9407e00640 1 -- 172.21.15.67:0/218005847 >> v1:172.21.15.86:6789/0 conn(0x7f940806e6e0 legacy=0x7f94080ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f9407e00640 1 -- 172.21.15.67:0/218005847 >> v1:172.21.15.67:6789/0 conn(0x7f940806eac0 legacy=0x7f9408143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.154+0000 7f9407e00640 1 -- 172.21.15.67:0/218005847 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408071a00 con 0x7f94080cf0c0 2024-04-17T10:28:51.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f9407e00640 1 --2- 172.21.15.67:0/218005847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94080cf0c0 0x7f94081441f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9400009d90 tx=0x7f940001c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0cda01d3461d184 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f9406a00640 1 -- 172.21.15.67:0/218005847 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000b2070 con 0x7f94080cf0c0 2024-04-17T10:28:51.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f9406a00640 1 -- 172.21.15.67:0/218005847 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94000a5970 con 0x7f94080cf0c0 2024-04-17T10:28:51.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f9406a00640 1 -- 172.21.15.67:0/218005847 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000a5d40 con 0x7f94080cf0c0 2024-04-17T10:28:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 -- 172.21.15.67:0/218005847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94080cf0c0 msgr2=0x7f94081441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 --2- 172.21.15.67:0/218005847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94080cf0c0 0x7f94081441f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9400009d90 tx=0x7f940001c8b0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 -- 172.21.15.67:0/218005847 shutdown_connections 2024-04-17T10:28:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 --2- 172.21.15.67:0/218005847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f94080cf0c0 0x7f94081441f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 -- 172.21.15.67:0/218005847 >> 172.21.15.67:0/218005847 conn(0x7f940806ac90 msgr2=0x7f940806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:51.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 -- 172.21.15.67:0/218005847 shutdown_connections 2024-04-17T10:28:51.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 -- 172.21.15.67:0/218005847 wait complete. 2024-04-17T10:28:51.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 Processor -- start 2024-04-17T10:28:51.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.158+0000 7f940e000640 1 -- start start 2024-04-17T10:28:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f940806e6e0 0x7f94080af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f940806eac0 0x7f94080c5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080cf0c0 0x7f940815a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94081680b0 con 0x7f94080cf0c0 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9408167f30 con 0x7f940806e6e0 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9408168230 con 0x7f940806eac0 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080cf0c0 0x7f940815a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f940806e6e0 0x7f94080af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080cf0c0 0x7f940815a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59632/0 (socket says 172.21.15.67:59632) 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 -- 172.21.15.67:0/1614103426 learned_addr learned my addr 172.21.15.67:0/1614103426 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f93fea00640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f940806eac0 0x7f94080c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 -- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f940806eac0 msgr2=0x7f94080c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f940806eac0 0x7f94080c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 -- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f940806e6e0 msgr2=0x7f94080af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f940806e6e0 0x7f94080af040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 -- 172.21.15.67:0/1614103426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080a87f0 con 0x7f94080cf0c0 2024-04-17T10:28:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f93fea00640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f940806eac0 0x7f94080c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407400640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f940806e6e0 0x7f94080af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:51.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f9407e00640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080cf0c0 0x7f940815a110 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f940001a4e0 tx=0x7f9400019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f93ffe00640 1 -- 172.21.15.67:0/1614103426 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000b2050 con 0x7f94080cf0c0 2024-04-17T10:28:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f93ffe00640 1 -- 172.21.15.67:0/1614103426 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94000be070 con 0x7f94080cf0c0 2024-04-17T10:28:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f93ffe00640 1 -- 172.21.15.67:0/1614103426 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94000ada50 con 0x7f94080cf0c0 2024-04-17T10:28:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94080af620 con 0x7f94080cf0c0 2024-04-17T10:28:51.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94080afbe0 con 0x7f94080cf0c0 2024-04-17T10:28:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9408072d50 con 0x7f94080cf0c0 2024-04-17T10:28:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.162+0000 7f93ffe00640 1 -- 172.21.15.67:0/1614103426 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f94000adbf0 con 0x7f94080cf0c0 2024-04-17T10:28:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.166+0000 7f93ffe00640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e4066f60 0x7f93e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.166+0000 7f93ffe00640 1 -- 172.21.15.67:0/1614103426 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f9400120100 con 0x7f94080cf0c0 2024-04-17T10:28:51.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.166+0000 7f9407400640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e4066f60 0x7f93e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.166+0000 7f9407400640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e4066f60 0x7f93e4069420 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f93f8001b60 tx=0x7f93f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.170+0000 7f93ffe00640 1 -- 172.21.15.67:0/1614103426 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9400016560 con 0x7f94080cf0c0 2024-04-17T10:28:51.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94080b2c60 con 0x7f94080cf0c0 2024-04-17T10:28:51.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f93ffe00640 1 -- 172.21.15.67:0/1614103426 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13245 (secure 0 0 0) 0x7f94000f1310 con 0x7f94080cf0c0 2024-04-17T10:28:51.327 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:51.327 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":59,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:49.977783+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6837/2048306700":"2024-04-18T10:28:46.650254+0000","172.21.15.86:6835/2048306700":"2024-04-18T10:28:46.650254+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e4066f60 msgr2=0x7f93e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e4066f60 0x7f93e4069420 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f93f8001b60 tx=0x7f93f8009040 comp rx=0 tx=0).stop 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080cf0c0 msgr2=0x7f940815a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080cf0c0 0x7f940815a110 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f940001a4e0 tx=0x7f9400019650 comp rx=0 tx=0).stop 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 shutdown_connections 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e4066f60 0x7f93e4069420 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080cf0c0 0x7f940815a110 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f940806eac0 0x7f94080c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 --2- 172.21.15.67:0/1614103426 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f940806e6e0 0x7f94080af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 >> 172.21.15.67:0/1614103426 conn(0x7f940806ac90 msgr2=0x7f94080adce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 shutdown_connections 2024-04-17T10:28:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.334+0000 7f940e000640 1 -- 172.21.15.67:0/1614103426 wait complete. 2024-04-17T10:28:51.348 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:28:51.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:28:51.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c72200640 1 Processor -- start 2024-04-17T10:28:51.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c72200640 1 -- start start 2024-04-17T10:28:51.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c72200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8c6c099a50 0x7f8c6c0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c72200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c6c0cf200 0x7f8c6c0b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c72200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c0cc3e0 con 0x7f8c6c0b2a30 2024-04-17T10:28:51.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c72200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c0cd9c0 con 0x7f8c6c0cf200 2024-04-17T10:28:51.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c72200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c0cc6c0 con 0x7f8c6c099a50 2024-04-17T10:28:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.498+0000 7f8c71800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c6c0cf200 0x7f8c6c0b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c71800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c6c0cf200 0x7f8c6c0b1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59636/0 (socket says 172.21.15.67:59636) 2024-04-17T10:28:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c71800640 1 -- 172.21.15.67:0/1635915702 learned_addr learned my addr 172.21.15.67:0/1635915702 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c70e00640 1 --2- 172.21.15.67:0/1635915702 >> v2:172.21.15.86:3300/0 conn(0x7f8c6c099a50 0x7f8c6c0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c71800640 1 -- 172.21.15.67:0/1635915702 >> v2:172.21.15.86:3300/0 conn(0x7f8c6c099a50 msgr2=0x7f8c6c0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c71800640 1 --2- 172.21.15.67:0/1635915702 >> v2:172.21.15.86:3300/0 conn(0x7f8c6c099a50 0x7f8c6c0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c71800640 1 -- 172.21.15.67:0/1635915702 >> v1:172.21.15.86:6789/0 conn(0x7f8c6c0b2a30 legacy=0x7f8c6c099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c71800640 1 -- 172.21.15.67:0/1635915702 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c0cc9a0 con 0x7f8c6c0cf200 2024-04-17T10:28:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c71800640 1 --2- 172.21.15.67:0/1635915702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c0cf200 0x7f8c6c0b1010 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c072450 tx=0x7f8c6401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db759186197755fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c6b400640 1 -- 172.21.15.67:0/1635915702 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640a57f0 con 0x7f8c6c0cf200 2024-04-17T10:28:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c6b400640 1 -- 172.21.15.67:0/1635915702 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c640a5990 con 0x7f8c6c0cf200 2024-04-17T10:28:51.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.502+0000 7f8c6b400640 1 -- 172.21.15.67:0/1635915702 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640a5d60 con 0x7f8c6c0cf200 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- 172.21.15.67:0/1635915702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c0cf200 msgr2=0x7f8c6c0b1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 --2- 172.21.15.67:0/1635915702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c0cf200 0x7f8c6c0b1010 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c072450 tx=0x7f8c6401c930 comp rx=0 tx=0).stop 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- 172.21.15.67:0/1635915702 shutdown_connections 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 --2- 172.21.15.67:0/1635915702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c0cf200 0x7f8c6c0b1010 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 --2- 172.21.15.67:0/1635915702 >> v2:172.21.15.86:3300/0 conn(0x7f8c6c099a50 0x7f8c6c0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- 172.21.15.67:0/1635915702 >> 172.21.15.67:0/1635915702 conn(0x7f8c6c0d1630 msgr2=0x7f8c6c0d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- 172.21.15.67:0/1635915702 shutdown_connections 2024-04-17T10:28:51.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- 172.21.15.67:0/1635915702 wait complete. 2024-04-17T10:28:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 Processor -- start 2024-04-17T10:28:51.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- start start 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6c099a50 0x7f8c6c156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c6c0cf200 0x7f8c6c0c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c1fa5d0 0x7f8c6c06ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c16a620 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c0d2580 con 0x7f8c6c0cf200 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c72200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c16a7a0 con 0x7f8c6c099a50 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c1fa5d0 0x7f8c6c06ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c1fa5d0 0x7f8c6c06ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59646/0 (socket says 172.21.15.67:59646) 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c70e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6c099a50 0x7f8c6c156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 -- 172.21.15.67:0/166588812 learned_addr learned my addr 172.21.15.67:0/166588812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c6be00640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c6c0cf200 0x7f8c6c0c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 -- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6c099a50 msgr2=0x7f8c6c156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6c099a50 0x7f8c6c156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 -- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c6c0cf200 msgr2=0x7f8c6c0c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c6c0cf200 0x7f8c6c0c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 -- 172.21.15.67:0/166588812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c0c7ee0 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.506+0000 7f8c71800640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c1fa5d0 0x7f8c6c06ca40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8c640af040 tx=0x7f8c640196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c68c00640 1 -- 172.21.15.67:0/166588812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640b8670 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c68c00640 1 -- 172.21.15.67:0/166588812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c640be070 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c15edf0 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c68c00640 1 -- 172.21.15.67:0/166588812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c640b8a90 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c1fad70 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c04f780 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c68c00640 1 -- 172.21.15.67:0/166588812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8c640b8c30 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c68c00640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c4c066f60 0x7f8c4c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c68c00640 1 -- 172.21.15.67:0/166588812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f8c64120090 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.510+0000 7f8c70e00640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c4c066f60 0x7f8c4c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.514+0000 7f8c70e00640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c4c066f60 0x7f8c4c069420 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c1685e0 tx=0x7f8c60009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.514+0000 7f8c68c00640 1 -- 172.21.15.67:0/166588812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8c6c0c7ee0 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c6c0cd480 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c68c00640 1 -- 172.21.15.67:0/166588812 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c640165e0 con 0x7f8c6c1fa5d0 2024-04-17T10:28:51.671 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c4c066f60 msgr2=0x7f8c4c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c4c066f60 0x7f8c4c069420 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c1685e0 tx=0x7f8c60009190 comp rx=0 tx=0).stop 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c1fa5d0 msgr2=0x7f8c6c06ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c1fa5d0 0x7f8c6c06ca40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8c640af040 tx=0x7f8c640196d0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 shutdown_connections 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c4c066f60 0x7f8c4c069420 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c6c1fa5d0 0x7f8c6c06ca40 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c6c0cf200 0x7f8c6c0c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 --2- 172.21.15.67:0/166588812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c6c099a50 0x7f8c6c156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 >> 172.21.15.67:0/166588812 conn(0x7f8c6c0d1630 msgr2=0x7f8c6c073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 shutdown_connections 2024-04-17T10:28:51.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.678+0000 7f8c72200640 1 -- 172.21.15.67:0/166588812 wait complete. 2024-04-17T10:28:51.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair' 2024-04-17T10:28:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.838+0000 7f7a4dc00640 1 Processor -- start 2024-04-17T10:28:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.838+0000 7f7a4dc00640 1 -- start start 2024-04-17T10:28:51.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a4dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a481620a0 0x7f7a4815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a4dc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4815eb50 con 0x7f7a480ce020 2024-04-17T10:28:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a4dc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4815ee30 con 0x7f7a4815e550 2024-04-17T10:28:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a4dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a48161650 con 0x7f7a481620a0 2024-04-17T10:28:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a46a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7a480ce020 0x7f7a480ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46300/0 (socket says 172.21.15.67:46300) 2024-04-17T10:28:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a46a00640 1 -- 172.21.15.67:0/3971039161 learned_addr learned my addr 172.21.15.67:0/3971039161 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a3e000640 1 --2- 172.21.15.67:0/3971039161 >> v2:172.21.15.67:3300/0 conn(0x7f7a481620a0 0x7f7a4815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a3e000640 1 -- 172.21.15.67:0/3971039161 >> v1:172.21.15.86:6789/0 conn(0x7f7a4815e550 legacy=0x7f7a480cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:51.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a3e000640 1 -- 172.21.15.67:0/3971039161 >> v1:172.21.15.67:6789/0 conn(0x7f7a480ce020 legacy=0x7f7a480ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a46000640 1 -- 172.21.15.67:0/3971039161 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559454343 0 0) 0x7f7a4815eb50 con 0x7f7a480ce020 2024-04-17T10:28:51.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.842+0000 7f7a3e000640 1 -- 172.21.15.67:0/3971039161 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a481618a0 con 0x7f7a481620a0 2024-04-17T10:28:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a3e000640 1 --2- 172.21.15.67:0/3971039161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a481620a0 0x7f7a4815e010 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7a300097f0 tx=0x7f7a3001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ae42660a59728b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a46000640 1 -- 172.21.15.67:0/3971039161 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300b1070 con 0x7f7a481620a0 2024-04-17T10:28:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a46000640 1 -- 172.21.15.67:0/3971039161 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a300ac410 con 0x7f7a481620a0 2024-04-17T10:28:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a46000640 1 -- 172.21.15.67:0/3971039161 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300ac7e0 con 0x7f7a481620a0 2024-04-17T10:28:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3971039161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a481620a0 msgr2=0x7f7a4815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3971039161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a481620a0 0x7f7a4815e010 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7a300097f0 tx=0x7f7a3001c3f0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3971039161 shutdown_connections 2024-04-17T10:28:51.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3971039161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a481620a0 0x7f7a4815e010 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3971039161 >> 172.21.15.67:0/3971039161 conn(0x7f7a480b33f0 msgr2=0x7f7a480b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:51.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3971039161 shutdown_connections 2024-04-17T10:28:51.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3971039161 wait complete. 2024-04-17T10:28:51.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 Processor -- start 2024-04-17T10:28:51.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 -- start start 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.846+0000 7f7a4dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a480ce020 0x7f7a480b5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a4815e550 0x7f7a480b56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a481620a0 0x7f7a48158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a480c8630 con 0x7f7a480ce020 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a480c84b0 con 0x7f7a4815e550 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4816ba00 con 0x7f7a481620a0 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a480ce020 0x7f7a480b5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a480ce020 0x7f7a480b5160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59670/0 (socket says 172.21.15.67:59670) 2024-04-17T10:28:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 -- 172.21.15.67:0/3022072506 learned_addr learned my addr 172.21.15.67:0/3022072506 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a3e000640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a4815e550 0x7f7a480b56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a47400640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a481620a0 0x7f7a48158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 -- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a481620a0 msgr2=0x7f7a48158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a481620a0 0x7f7a48158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 -- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a4815e550 msgr2=0x7f7a480b56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a4815e550 0x7f7a480b56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 -- 172.21.15.67:0/3022072506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48153610 con 0x7f7a480ce020 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a46a00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a480ce020 0x7f7a480b5160 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f7a380011f0 tx=0x7f7a3800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a3f400640 1 -- 172.21.15.67:0/3022072506 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38018070 con 0x7f7a480ce020 2024-04-17T10:28:51.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a3f400640 1 -- 172.21.15.67:0/3022072506 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a380074e0 con 0x7f7a480ce020 2024-04-17T10:28:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a3f400640 1 -- 172.21.15.67:0/3022072506 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38004a10 con 0x7f7a480ce020 2024-04-17T10:28:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a481538f0 con 0x7f7a480ce020 2024-04-17T10:28:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a480b5be0 con 0x7f7a480ce020 2024-04-17T10:28:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a48070ab0 con 0x7f7a480ce020 2024-04-17T10:28:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.850+0000 7f7a3f400640 1 -- 172.21.15.67:0/3022072506 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7a3801b070 con 0x7f7a480ce020 2024-04-17T10:28:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.854+0000 7f7a3f400640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a24066f60 0x7f7a24069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:51.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.854+0000 7f7a3f400640 1 -- 172.21.15.67:0/3022072506 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f7a38084160 con 0x7f7a480ce020 2024-04-17T10:28:51.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.854+0000 7f7a3e000640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a24066f60 0x7f7a24069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:51.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.854+0000 7f7a3e000640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a24066f60 0x7f7a24069420 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7a300ae000 tx=0x7f7a300a6040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:51.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:51.858+0000 7f7a3f400640 1 -- 172.21.15.67:0/3022072506 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7a38054d00 con 0x7f7a480ce020 2024-04-17T10:28:52.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.078+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]} v 0) v1 -- 0x7f7a48158020 con 0x7f7a480ce020 2024-04-17T10:28:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a3f400640 1 -- 172.21.15.67:0/3022072506 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]}]=0 v209) v1 ==== 146+0+0 (secure 0 0 0) 0x7f7a38058da0 con 0x7f7a480ce020 2024-04-17T10:28:52.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a24066f60 msgr2=0x7f7a24069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a24066f60 0x7f7a24069420 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7a300ae000 tx=0x7f7a300a6040 comp rx=0 tx=0).stop 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a480ce020 msgr2=0x7f7a480b5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a480ce020 0x7f7a480b5160 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f7a380011f0 tx=0x7f7a3800c9b0 comp rx=0 tx=0).stop 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 shutdown_connections 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a24066f60 0x7f7a24069420 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a481620a0 0x7f7a48158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a4815e550 0x7f7a480b56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 --2- 172.21.15.67:0/3022072506 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a480ce020 0x7f7a480b5160 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 >> 172.21.15.67:0/3022072506 conn(0x7f7a480b33f0 msgr2=0x7f7a480cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:52.998+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 shutdown_connections 2024-04-17T10:28:52.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.002+0000 7f7a4dc00640 1 -- 172.21.15.67:0/3022072506 wait complete. 2024-04-17T10:28:53.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:28:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff46b800640 1 Processor -- start 2024-04-17T10:28:53.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff46b800640 1 -- start start 2024-04-17T10:28:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff46b800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff4640b2e00 0x7ff4640b5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff46b800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4640b3290 con 0x7ff46415dc80 2024-04-17T10:28:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff46b800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4640b5de0 con 0x7ff4640b59d0 2024-04-17T10:28:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff46b800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4640b9280 con 0x7ff4640b2e00 2024-04-17T10:28:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff468e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff46415dc80 0x7ff4640b8b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46314/0 (socket says 172.21.15.67:46314) 2024-04-17T10:28:53.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff468e00640 1 -- 172.21.15.67:0/3727154684 learned_addr learned my addr 172.21.15.67:0/3727154684 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908509707 0 0) 0x7ff454001120 con 0x7ff46415dc80 2024-04-17T10:28:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff448003530 con 0x7ff46415dc80 2024-04-17T10:28:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff45be00640 1 --2- 172.21.15.67:0/3727154684 >> v2:172.21.15.86:3300/0 conn(0x7ff4640b2e00 0x7ff4640b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281411251 0 0) 0x7ff448003530 con 0x7ff46415dc80 2024-04-17T10:28:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 >> v2:172.21.15.86:3300/0 conn(0x7ff4640b2e00 msgr2=0x7ff4640b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 --2- 172.21.15.67:0/3727154684 >> v2:172.21.15.86:3300/0 conn(0x7ff4640b2e00 0x7ff4640b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 >> v1:172.21.15.86:6789/0 conn(0x7ff4640b59d0 legacy=0x7ff4640d1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640b7db0 con 0x7ff46415dc80 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7ff454001870 con 0x7ff46415dc80 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 >> v1:172.21.15.67:6789/0 conn(0x7ff46415dc80 legacy=0x7ff4640b8b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff448003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff448004810 0x7ff448004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4480051d0 0x7ff44800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44800c050 con 0x7ff4480039c0 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44800c200 con 0x7ff4480051d0 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.174+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44800c400 con 0x7ff448004810 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff468e00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff448003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff469800640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4480051d0 0x7ff44800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff45be00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff448004810 0x7ff448004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff468e00640 1 -- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff448004810 msgr2=0x7ff448004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:53.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff468e00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff448004810 0x7ff448004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff468e00640 1 -- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4480051d0 msgr2=0x7ff44800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:53.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff468e00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4480051d0 0x7ff44800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff468e00640 1 -- 172.21.15.67:0/3727154684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff448003530 con 0x7ff4480039c0 2024-04-17T10:28:53.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff469800640 1 -- 172.21.15.67:0/3727154684 reap_dead start 2024-04-17T10:28:53.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff468e00640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff448003de0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff45400c490 tx=0x7ff45401f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae36cab265c9d17 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:53.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45400c0e0 con 0x7ff4480039c0 2024-04-17T10:28:53.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454005ae0 con 0x7ff4480039c0 2024-04-17T10:28:53.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff463e00640 1 -- 172.21.15.67:0/3727154684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454007070 con 0x7ff4480039c0 2024-04-17T10:28:53.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff46b800640 1 -- 172.21.15.67:0/3727154684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 msgr2=0x7ff448003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.178+0000 7ff46b800640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff448003de0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ff45400c490 tx=0x7ff45401f3c0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- 172.21.15.67:0/3727154684 shutdown_connections 2024-04-17T10:28:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 --2- 172.21.15.67:0/3727154684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff448003de0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- 172.21.15.67:0/3727154684 >> 172.21.15.67:0/3727154684 conn(0x7ff4640727c0 msgr2=0x7ff464072bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- 172.21.15.67:0/3727154684 shutdown_connections 2024-04-17T10:28:53.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- 172.21.15.67:0/3727154684 wait complete. 2024-04-17T10:28:53.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 Processor -- start 2024-04-17T10:28:53.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- start start 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff4640c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4641f6bf0 0x7ff4641f7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4641f3b40 0x7ff4641f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640b8650 con 0x7ff4480039c0 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640b8370 con 0x7ff4641f3b40 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff46b800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4640ce6a0 con 0x7ff4641f6bf0 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff4640c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff4640c6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59688/0 (socket says 172.21.15.67:59688) 2024-04-17T10:28:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 -- 172.21.15.67:0/683960729 learned_addr learned my addr 172.21.15.67:0/683960729 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 -- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4641f6bf0 msgr2=0x7ff4641f7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff469800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4641f3b40 0x7ff4641f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff45be00640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4641f6bf0 0x7ff4641f7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4641f6bf0 0x7ff4641f7070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 -- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4641f3b40 msgr2=0x7ff4641f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4641f3b40 0x7ff4641f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 -- 172.21.15.67:0/683960729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4640cb9d0 con 0x7ff4480039c0 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff469800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4641f3b40 0x7ff4641f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff45be00640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4641f6bf0 0x7ff4641f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.182+0000 7ff468e00640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff4640c6ea0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff454026030 tx=0x7ff448005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff461600640 1 -- 172.21.15.67:0/683960729 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454001120 con 0x7ff4480039c0 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff461600640 1 -- 172.21.15.67:0/683960729 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4540076d0 con 0x7ff4480039c0 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff461600640 1 -- 172.21.15.67:0/683960729 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4540047f0 con 0x7ff4480039c0 2024-04-17T10:28:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4640cbcb0 con 0x7ff4480039c0 2024-04-17T10:28:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4640cc270 con 0x7ff4480039c0 2024-04-17T10:28:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46404ef30 con 0x7ff4480039c0 2024-04-17T10:28:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff461600640 1 -- 172.21.15.67:0/683960729 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff4540b8030 con 0x7ff4480039c0 2024-04-17T10:28:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff461600640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff440066f60 0x7ff440069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff461600640 1 -- 172.21.15.67:0/683960729 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7ff454121920 con 0x7ff4480039c0 2024-04-17T10:28:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.186+0000 7ff45be00640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff440066f60 0x7ff440069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.190+0000 7ff45be00640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff440066f60 0x7ff440069420 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff4640cdfe0 tx=0x7ff44c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:53.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.190+0000 7ff461600640 1 -- 172.21.15.67:0/683960729 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff4640b3290 con 0x7ff4480039c0 2024-04-17T10:28:53.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.354+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff4640d3be0 con 0x7ff4480039c0 2024-04-17T10:28:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff461600640 1 -- 172.21.15.67:0/683960729 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff4540f2a00 con 0x7ff4480039c0 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff440066f60 msgr2=0x7ff440069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff440066f60 0x7ff440069420 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff4640cdfe0 tx=0x7ff44c008040 comp rx=0 tx=0).stop 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 msgr2=0x7ff4640c6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff4640c6ea0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff454026030 tx=0x7ff448005840 comp rx=0 tx=0).stop 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 shutdown_connections 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff440066f60 0x7ff440069420 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4641f3b40 0x7ff4641f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4641f6bf0 0x7ff4641f7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 --2- 172.21.15.67:0/683960729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4480039c0 0x7ff4640c6ea0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 >> 172.21.15.67:0/683960729 conn(0x7ff4640727c0 msgr2=0x7ff46415cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:53.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 shutdown_connections 2024-04-17T10:28:53.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.362+0000 7ff46b800640 1 -- 172.21.15.67:0/683960729 wait complete. 2024-04-17T10:28:53.376 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:28:53.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:28:53.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:28:53.561 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:28:53.562 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:53.562 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:53.571 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:28:53.571 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:28:53.571 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:53.580 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:28:53.581 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:28:53.581 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:53.621 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:28:53.621 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:28:53.621 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:28:53.630 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:28:53.631 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:28:53.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:28:53.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.810+0000 7fa392000640 1 Processor -- start 2024-04-17T10:28:53.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.810+0000 7fa392000640 1 -- start start 2024-04-17T10:28:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa392000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0cf510 0x7fa38c0cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa392000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c15eb50 con 0x7fa38c0ce020 2024-04-17T10:28:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa392000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38c15ee30 con 0x7fa38c1620a0 2024-04-17T10:28:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa392000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c161650 con 0x7fa38c0cf510 2024-04-17T10:28:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa38c0ce020 0x7fa38c0ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46328/0 (socket says 172.21.15.67:46328) 2024-04-17T10:28:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38b400640 1 -- 172.21.15.67:0/758976938 learned_addr learned my addr 172.21.15.67:0/758976938 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38be00640 1 --2- 172.21.15.67:0/758976938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0cf510 0x7fa38c0cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38aa00640 1 -- 172.21.15.67:0/758976938 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764905464 0 0) 0x7fa38c15eb50 con 0x7fa38c0ce020 2024-04-17T10:28:53.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38aa00640 1 -- 172.21.15.67:0/758976938 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa370003530 con 0x7fa38c0ce020 2024-04-17T10:28:53.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38aa00640 1 -- 172.21.15.67:0/758976938 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810761961 0 0) 0x7fa38c15ee30 con 0x7fa38c1620a0 2024-04-17T10:28:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38aa00640 1 -- 172.21.15.67:0/758976938 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa38c15eb50 con 0x7fa38c1620a0 2024-04-17T10:28:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38be00640 1 -- 172.21.15.67:0/758976938 >> v1:172.21.15.86:6789/0 conn(0x7fa38c1620a0 legacy=0x7fa38c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38be00640 1 -- 172.21.15.67:0/758976938 >> v1:172.21.15.67:6789/0 conn(0x7fa38c0ce020 legacy=0x7fa38c0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.814+0000 7fa38be00640 1 -- 172.21.15.67:0/758976938 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c1618a0 con 0x7fa38c0cf510 2024-04-17T10:28:53.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa38be00640 1 --2- 172.21.15.67:0/758976938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0cf510 0x7fa38c0cf9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa384009df0 tx=0x7fa38401c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=187f71914bf4a357 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:53.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa38aa00640 1 -- 172.21.15.67:0/758976938 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840b2070 con 0x7fa38c0cf510 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa38aa00640 1 -- 172.21.15.67:0/758976938 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3840a5930 con 0x7fa38c0cf510 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa38aa00640 1 -- 172.21.15.67:0/758976938 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840a5d00 con 0x7fa38c0cf510 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- 172.21.15.67:0/758976938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0cf510 msgr2=0x7fa38c0cf9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 --2- 172.21.15.67:0/758976938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0cf510 0x7fa38c0cf9a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa384009df0 tx=0x7fa38401c960 comp rx=0 tx=0).stop 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- 172.21.15.67:0/758976938 shutdown_connections 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 --2- 172.21.15.67:0/758976938 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0cf510 0x7fa38c0cf9a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- 172.21.15.67:0/758976938 >> 172.21.15.67:0/758976938 conn(0x7fa38c0b33f0 msgr2=0x7fa38c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- 172.21.15.67:0/758976938 shutdown_connections 2024-04-17T10:28:53.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- 172.21.15.67:0/758976938 wait complete. 2024-04-17T10:28:53.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 Processor -- start 2024-04-17T10:28:53.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- start start 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0ce020 0x7fa38c0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38c0cf510 0x7fa38c147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38c1620a0 0x7fa38c1f7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c15e8a0 con 0x7fa38c1620a0 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c15e720 con 0x7fa38c0cf510 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.818+0000 7fa392000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa38c0c84b0 con 0x7fa38c0ce020 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38c1620a0 0x7fa38c1f7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38c1620a0 0x7fa38c1f7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59700/0 (socket says 172.21.15.67:59700) 2024-04-17T10:28:53.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 -- 172.21.15.67:0/2837263598 learned_addr learned my addr 172.21.15.67:0/2837263598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38b400640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0ce020 0x7fa38c0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa383e00640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38c0cf510 0x7fa38c147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 -- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0ce020 msgr2=0x7fa38c0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0ce020 0x7fa38c0b4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 -- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38c0cf510 msgr2=0x7fa38c147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38c0cf510 0x7fa38c147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 -- 172.21.15.67:0/2837263598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0b55f0 con 0x7fa38c1620a0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa383e00640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38c0cf510 0x7fa38c147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa38be00640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38c1620a0 0x7fa38c1f7a60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa38401a590 tx=0x7fa384019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa383400640 1 -- 172.21.15.67:0/2837263598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840b2050 con 0x7fa38c1620a0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa383400640 1 -- 172.21.15.67:0/2837263598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3840be070 con 0x7fa38c1620a0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa383400640 1 -- 172.21.15.67:0/2837263598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840ad870 con 0x7fa38c1620a0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38c0b58d0 con 0x7fa38c1620a0 2024-04-17T10:28:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa38c0b5e90 con 0x7fa38c1620a0 2024-04-17T10:28:53.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c070ab0 con 0x7fa38c1620a0 2024-04-17T10:28:53.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.822+0000 7fa383400640 1 -- 172.21.15.67:0/2837263598 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa3840ada10 con 0x7fa38c1620a0 2024-04-17T10:28:53.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.826+0000 7fa383400640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa368066f90 0x7fa368069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:53.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.826+0000 7fa383400640 1 -- 172.21.15.67:0/2837263598 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fa38411ff60 con 0x7fa38c1620a0 2024-04-17T10:28:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.826+0000 7fa38b400640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa368066f90 0x7fa368069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.826+0000 7fa38b400640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa368066f90 0x7fa368069450 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa37c000f70 tx=0x7fa37c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:53.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:53.830+0000 7fa383400640 1 -- 172.21.15.67:0/2837263598 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa384016580 con 0x7fa38c1620a0 2024-04-17T10:28:54.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.022+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa38c161020 con 0x7fa38c1620a0 2024-04-17T10:28:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.022+0000 7fa383400640 1 -- 172.21.15.67:0/2837263598 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fa3840add00 con 0x7fa38c1620a0 2024-04-17T10:28:54.017 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:54.017 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":33,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/2433614600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":2433614600},{"type":"v1","addr":"172.21.15.86:6836","nonce":2433614600}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5760,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6833/160837053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":160837053},{"type":"v1","addr":"172.21.15.67:6833","nonce":160837053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":22},{"gid":5766,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2644540190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":2644540190},{"type":"v1","addr":"172.21.15.67:6835","nonce":2644540190}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":23},{"gid":5849,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3029192346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3029192346},{"type":"v1","addr":"172.21.15.86:6837","nonce":3029192346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":33}],"filesystems":[]} 2024-04-17T10:28:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 33 2024-04-17T10:28:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa368066f90 msgr2=0x7fa368069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa368066f90 0x7fa368069450 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa37c000f70 tx=0x7fa37c009040 comp rx=0 tx=0).stop 2024-04-17T10:28:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38c1620a0 msgr2=0x7fa38c1f7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38c1620a0 0x7fa38c1f7a60 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fa38401a590 tx=0x7fa384019700 comp rx=0 tx=0).stop 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 shutdown_connections 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa368066f90 0x7fa368069450 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa38c1620a0 0x7fa38c1f7a60 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa38c0cf510 0x7fa38c147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 --2- 172.21.15.67:0/2837263598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa38c0ce020 0x7fa38c0b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 >> 172.21.15.67:0/2837263598 conn(0x7fa38c0b33f0 msgr2=0x7fa38c0d0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 shutdown_connections 2024-04-17T10:28:54.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.026+0000 7fa392000640 1 -- 172.21.15.67:0/2837263598 wait complete. 2024-04-17T10:28:54.039 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:28:54.039 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:28:54.040 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:28:54.040 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:28:54.040 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:28:54.040 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:28:54.040 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:28:54.040 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:28:54.041 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:28:54.041 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:28:54.041 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:28:54.041 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:28:54.041 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:28:54.041 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:28:54.041 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:28:54.041 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:28:54.053 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:28:54.054 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:28:54.054 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:28:54.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:28:54.062 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:28:54.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:28:54.063 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:28:54.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:28:54.100 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:28:54.100 DEBUG:teuthology.parallel:result is None 2024-04-17T10:28:54.101 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:28:54.102 DEBUG:teuthology.parallel:result is None 2024-04-17T10:28:54.105 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:28:54.105 DEBUG:teuthology.parallel:result is None 2024-04-17T10:28:54.107 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:28:54.107 DEBUG:teuthology.parallel:result is None 2024-04-17T10:28:54.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:28:54.189 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:28:54.189 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:28:54.199 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:28:54.200 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:28:54.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.258+0000 7f337a000640 1 Processor -- start 2024-04-17T10:28:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.258+0000 7f337a000640 1 -- start start 2024-04-17T10:28:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f337a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f337415dbc0 0x7f337406f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f337a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337406db90 0x7f3374071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f337a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740ad8f0 con 0x7f33740721a0 2024-04-17T10:28:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f337a000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337415dfa0 con 0x7f337415dbc0 2024-04-17T10:28:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f337a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337406faa0 con 0x7f337406db90 2024-04-17T10:28:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f3373400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f337415dbc0 0x7f337406f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f3373e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f33740721a0 0x7f337406b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46362/0 (socket says 172.21.15.67:46362) 2024-04-17T10:28:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f3373e00640 1 -- 172.21.15.67:0/3420400710 learned_addr learned my addr 172.21.15.67:0/3420400710 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f336aa00640 1 --2- 172.21.15.67:0/3420400710 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337406db90 0x7f3374071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f3373400640 1 -- 172.21.15.67:0/3420400710 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337406db90 msgr2=0x7f3374071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f3373400640 1 --2- 172.21.15.67:0/3420400710 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337406db90 0x7f3374071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f3373400640 1 -- 172.21.15.67:0/3420400710 >> v1:172.21.15.67:6789/0 conn(0x7f33740721a0 legacy=0x7f337406b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.262+0000 7f3373400640 1 -- 172.21.15.67:0/3420400710 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740adbd0 con 0x7f337415dbc0 2024-04-17T10:28:54.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f3373400640 1 --2- 172.21.15.67:0/3420400710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337415dbc0 0x7f337406f560 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f33640097c0 tx=0x7f336401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=487af5525f15724b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:54.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f3372a00640 1 -- 172.21.15.67:0/3420400710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640b0070 con 0x7f337415dbc0 2024-04-17T10:28:54.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f3372a00640 1 -- 172.21.15.67:0/3420400710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33640ab600 con 0x7f337415dbc0 2024-04-17T10:28:54.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f3372a00640 1 -- 172.21.15.67:0/3420400710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640ab9d0 con 0x7f337415dbc0 2024-04-17T10:28:54.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 -- 172.21.15.67:0/3420400710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337415dbc0 msgr2=0x7f337406f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:54.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 --2- 172.21.15.67:0/3420400710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337415dbc0 0x7f337406f560 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f33640097c0 tx=0x7f336401c410 comp rx=0 tx=0).stop 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 -- 172.21.15.67:0/3420400710 shutdown_connections 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 --2- 172.21.15.67:0/3420400710 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337406db90 0x7f3374071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 --2- 172.21.15.67:0/3420400710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337415dbc0 0x7f337406f560 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 -- 172.21.15.67:0/3420400710 >> 172.21.15.67:0/3420400710 conn(0x7f33740af670 msgr2=0x7f33740b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 -- 172.21.15.67:0/3420400710 shutdown_connections 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 -- 172.21.15.67:0/3420400710 wait complete. 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 Processor -- start 2024-04-17T10:28:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.266+0000 7f337a000640 1 -- start start 2024-04-17T10:28:54.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337406db90 0x7f337413aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:54.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33740721a0 0x7f33740cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337415dbc0 0x7f33741516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337406c730 con 0x7f337406db90 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337406c5b0 con 0x7f33740721a0 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337406af70 con 0x7f337415dbc0 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337406db90 0x7f337413aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337406db90 0x7f337413aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59752/0 (socket says 172.21.15.67:59752) 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 -- 172.21.15.67:0/2585595793 learned_addr learned my addr 172.21.15.67:0/2585595793 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373e00640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337415dbc0 0x7f33741516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f336aa00640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33740721a0 0x7f33740cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 -- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337415dbc0 msgr2=0x7f33741516c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337415dbc0 0x7f33741516c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 -- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33740721a0 msgr2=0x7f33740cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:54.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33740721a0 0x7f33740cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 -- 172.21.15.67:0/2585595793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337413d520 con 0x7f337406db90 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f336aa00640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33740721a0 0x7f33740cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f3373400640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337406db90 0x7f337413aa90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f33640097c0 tx=0x7f33640191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f336be00640 1 -- 172.21.15.67:0/2585595793 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640b0040 con 0x7f337406db90 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f336be00640 1 -- 172.21.15.67:0/2585595793 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33640b43f0 con 0x7f337406db90 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337413d800 con 0x7f337406db90 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f336be00640 1 -- 172.21.15.67:0/2585595793 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640b47c0 con 0x7f337406db90 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337413ddc0 con 0x7f337406db90 2024-04-17T10:28:54.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337409e260 con 0x7f337406db90 2024-04-17T10:28:54.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.270+0000 7f336be00640 1 -- 172.21.15.67:0/2585595793 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f33640b4960 con 0x7f337406db90 2024-04-17T10:28:54.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.274+0000 7f336be00640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3350066f60 0x7f3350069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:54.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.274+0000 7f336be00640 1 -- 172.21.15.67:0/2585595793 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f336411ea40 con 0x7f337406db90 2024-04-17T10:28:54.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.274+0000 7f336aa00640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3350066f60 0x7f3350069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:54.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.274+0000 7f336aa00640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3350066f60 0x7f3350069420 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3374150e80 tx=0x7f335c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.278+0000 7f336be00640 1 -- 172.21.15.67:0/2585595793 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3364016090 con 0x7f337406db90 2024-04-17T10:28:54.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:54.450+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f337413afd0 con 0x7f337406db90 2024-04-17T10:28:55.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.018+0000 7f336be00640 1 -- 172.21.15.67:0/2585595793 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v61) v1 ==== 97+0+0 (secure 0 0 0) 0x7f33640efbb0 con 0x7f337406db90 2024-04-17T10:28:55.012 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:28:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3350066f60 msgr2=0x7f3350069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3350066f60 0x7f3350069420 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3374150e80 tx=0x7f335c008040 comp rx=0 tx=0).stop 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337406db90 msgr2=0x7f337413aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337406db90 0x7f337413aa90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f33640097c0 tx=0x7f33640191b0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 shutdown_connections 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3350066f60 0x7f3350069420 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337415dbc0 0x7f33741516c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33740721a0 0x7f33740cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 --2- 172.21.15.67:0/2585595793 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f337406db90 0x7f337413aa90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 >> 172.21.15.67:0/2585595793 conn(0x7f33740af670 msgr2=0x7f33740b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 shutdown_connections 2024-04-17T10:28:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.022+0000 7f337a000640 1 -- 172.21.15.67:0/2585595793 wait complete. 2024-04-17T10:28:55.040 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:28:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.190+0000 7f5eaaa00640 1 Processor -- start 2024-04-17T10:28:55.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.190+0000 7f5eaaa00640 1 -- start start 2024-04-17T10:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5eaaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ea41446c0 0x7f5ea4144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5eaaa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea41456d0 con 0x7f5ea4071840 2024-04-17T10:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5eaaa00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea40b4870 con 0x7f5ea4144aa0 2024-04-17T10:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5eaaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea40b51a0 con 0x7f5ea41446c0 2024-04-17T10:28:55.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ea41446c0 0x7f5ea4144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ea41446c0 0x7f5ea4144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59756/0 (socket says 172.21.15.67:59756) 2024-04-17T10:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3e00640 1 -- 172.21.15.67:0/2140723035 learned_addr learned my addr 172.21.15.67:0/2140723035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea8a00640 1 --1- 172.21.15.67:0/2140723035 >> v1:172.21.15.67:6789/0 conn(0x7f5ea4071840 0x7f5ea40b5630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46374/0 (socket says 172.21.15.67:46374) 2024-04-17T10:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3e00640 1 -- 172.21.15.67:0/2140723035 >> v1:172.21.15.86:6789/0 conn(0x7f5ea4144aa0 legacy=0x7f5ea40d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3e00640 1 -- 172.21.15.67:0/2140723035 >> v1:172.21.15.67:6789/0 conn(0x7f5ea4071840 legacy=0x7f5ea40b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3e00640 1 -- 172.21.15.67:0/2140723035 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea40b4b50 con 0x7f5ea41446c0 2024-04-17T10:28:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3e00640 1 --2- 172.21.15.67:0/2140723035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4144ea0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009870 tx=0x7f5e9401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1176a6e0c58ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.194+0000 7f5ea3400640 1 -- 172.21.15.67:0/2140723035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e940ab640 con 0x7f5ea41446c0 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5ea3400640 1 -- 172.21.15.67:0/2140723035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e940b0070 con 0x7f5ea41446c0 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5ea3400640 1 -- 172.21.15.67:0/2140723035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e940aba00 con 0x7f5ea41446c0 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- 172.21.15.67:0/2140723035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 msgr2=0x7f5ea4144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/2140723035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4144ea0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009870 tx=0x7f5e9401c530 comp rx=0 tx=0).stop 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- 172.21.15.67:0/2140723035 shutdown_connections 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/2140723035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4144ea0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- 172.21.15.67:0/2140723035 >> 172.21.15.67:0/2140723035 conn(0x7f5ea40b34d0 msgr2=0x7f5ea40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- 172.21.15.67:0/2140723035 shutdown_connections 2024-04-17T10:28:55.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- 172.21.15.67:0/2140723035 wait complete. 2024-04-17T10:28:55.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 Processor -- start 2024-04-17T10:28:55.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- start start 2024-04-17T10:28:55.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ea4071840 0x7f5ea40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ea4144aa0 0x7f5ea4157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4169bd0 con 0x7f5ea41446c0 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4169a50 con 0x7f5ea4144aa0 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5eaaa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4169d50 con 0x7f5ea4071840 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4141120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59762/0 (socket says 172.21.15.67:59762) 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 -- 172.21.15.67:0/552815045 learned_addr learned my addr 172.21.15.67:0/552815045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5ea3e00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ea4071840 0x7f5ea40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5ea8a00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ea4144aa0 0x7f5ea4157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 -- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ea4071840 msgr2=0x7f5ea40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ea4071840 0x7f5ea40c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 -- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ea4144aa0 msgr2=0x7f5ea4157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ea4144aa0 0x7f5ea4157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5e9be00640 1 -- 172.21.15.67:0/552815045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea4158380 con 0x7f5ea41446c0 2024-04-17T10:28:55.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.198+0000 7f5ea8a00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ea4144aa0 0x7f5ea4157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:55.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5e9be00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4141120 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c370 tx=0x7f5e8c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5ea0c00640 1 -- 172.21.15.67:0/552815045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c0052d0 con 0x7f5ea41446c0 2024-04-17T10:28:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5ea0c00640 1 -- 172.21.15.67:0/552815045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c005470 con 0x7f5ea41446c0 2024-04-17T10:28:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea40c6990 con 0x7f5ea41446c0 2024-04-17T10:28:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5ea0c00640 1 -- 172.21.15.67:0/552815045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c007260 con 0x7f5ea41446c0 2024-04-17T10:28:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea40c6f50 con 0x7f5ea41446c0 2024-04-17T10:28:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4070bb0 con 0x7f5ea41446c0 2024-04-17T10:28:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5ea0c00640 1 -- 172.21.15.67:0/552815045 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5e8c007400 con 0x7f5ea41446c0 2024-04-17T10:28:55.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5ea0c00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e80066f60 0x7f5e80069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5ea0c00640 1 -- 172.21.15.67:0/552815045 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5e8c084230 con 0x7f5ea41446c0 2024-04-17T10:28:55.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.202+0000 7f5ea3e00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e80066f60 0x7f5e80069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.206+0000 7f5ea3e00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e80066f60 0x7f5e80069420 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5e940af000 tx=0x7f5e940a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.206+0000 7f5ea0c00640 1 -- 172.21.15.67:0/552815045 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5e8c054da0 con 0x7f5ea41446c0 2024-04-17T10:28:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.430+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5ea4074e30 con 0x7f5ea41446c0 2024-04-17T10:28:55.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.430+0000 7f5ea0c00640 1 -- 172.21.15.67:0/552815045 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5e8c058e40 con 0x7f5ea41446c0 2024-04-17T10:28:55.424 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.425 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.426 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.427 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.428 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.430 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:28:55.431 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.432 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.433 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:28:55.434 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:28:55.435 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.436 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:28:55.437 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:28:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e80066f60 msgr2=0x7f5e80069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e80066f60 0x7f5e80069420 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5e940af000 tx=0x7f5e940a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 msgr2=0x7f5ea4141120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4141120 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c00c370 tx=0x7f5e8c00c830 comp rx=0 tx=0).stop 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 shutdown_connections 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e80066f60 0x7f5e80069420 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ea4144aa0 0x7f5ea4157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ea41446c0 0x7f5ea4141120 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 --2- 172.21.15.67:0/552815045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ea4071840 0x7f5ea40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 >> 172.21.15.67:0/552815045 conn(0x7f5ea40b34d0 msgr2=0x7f5ea41425a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 shutdown_connections 2024-04-17T10:28:55.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.434+0000 7f5eaaa00640 1 -- 172.21.15.67:0/552815045 wait complete. 2024-04-17T10:28:55.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:55.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce19600640 1 Processor -- start 2024-04-17T10:28:55.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce19600640 1 -- start start 2024-04-17T10:28:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce19600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce140ce2a0 0x7fce140cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce19600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fce140cc840 0x7fce140cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce19600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce1406b100 con 0x7fce140cc250 2024-04-17T10:28:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce19600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce140cd150 con 0x7fce140ce2a0 2024-04-17T10:28:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce19600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce1406b3e0 con 0x7fce140cc840 2024-04-17T10:28:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce0a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce140ce2a0 0x7fce140cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce0a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce140ce2a0 0x7fce140cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59774/0 (socket says 172.21.15.67:59774) 2024-04-17T10:28:55.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce0a000640 1 -- 172.21.15.67:0/3015709667 learned_addr learned my addr 172.21.15.67:0/3015709667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:55.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce13400640 1 --2- 172.21.15.67:0/3015709667 >> v2:172.21.15.86:3300/0 conn(0x7fce140cc840 0x7fce140cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce0a000640 1 -- 172.21.15.67:0/3015709667 >> v2:172.21.15.86:3300/0 conn(0x7fce140cc840 msgr2=0x7fce140cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:55.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce0a000640 1 --2- 172.21.15.67:0/3015709667 >> v2:172.21.15.86:3300/0 conn(0x7fce140cc840 0x7fce140cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce0a000640 1 -- 172.21.15.67:0/3015709667 >> v1:172.21.15.86:6789/0 conn(0x7fce140cc250 legacy=0x7fce140cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.594+0000 7fce0a000640 1 -- 172.21.15.67:0/3015709667 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1406b6c0 con 0x7fce140ce2a0 2024-04-17T10:28:55.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce0a000640 1 --2- 172.21.15.67:0/3015709667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140ce2a0 0x7fce140cd5e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fce14071b20 tx=0x7fcdfc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37608b20e91ad6a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce12000640 1 -- 172.21.15.67:0/3015709667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0ab430 con 0x7fce140ce2a0 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce12000640 1 -- 172.21.15.67:0/3015709667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc0b0070 con 0x7fce140ce2a0 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce12000640 1 -- 172.21.15.67:0/3015709667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0ab7f0 con 0x7fce140ce2a0 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 -- 172.21.15.67:0/3015709667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140ce2a0 msgr2=0x7fce140cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 --2- 172.21.15.67:0/3015709667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140ce2a0 0x7fce140cd5e0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fce14071b20 tx=0x7fcdfc01c520 comp rx=0 tx=0).stop 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 -- 172.21.15.67:0/3015709667 shutdown_connections 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 --2- 172.21.15.67:0/3015709667 >> v2:172.21.15.86:3300/0 conn(0x7fce140cc840 0x7fce140cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 --2- 172.21.15.67:0/3015709667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140ce2a0 0x7fce140cd5e0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 -- 172.21.15.67:0/3015709667 >> 172.21.15.67:0/3015709667 conn(0x7fce140b22a0 msgr2=0x7fce140b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 -- 172.21.15.67:0/3015709667 shutdown_connections 2024-04-17T10:28:55.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.598+0000 7fce19600640 1 -- 172.21.15.67:0/3015709667 wait complete. 2024-04-17T10:28:55.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 Processor -- start 2024-04-17T10:28:55.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 -- start start 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140cc250 0x7fce14149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce140cc840 0x7fce1406e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce140ce2a0 0x7fce14160160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1416ac80 con 0x7fce140cc250 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1416ab00 con 0x7fce140cc840 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1416ae00 con 0x7fce140ce2a0 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140cc250 0x7fce14149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140cc250 0x7fce14149a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59776/0 (socket says 172.21.15.67:59776) 2024-04-17T10:28:55.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 -- 172.21.15.67:0/2676047092 learned_addr learned my addr 172.21.15.67:0/2676047092 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce0a000640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce140cc840 0x7fce1406e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce13400640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce140ce2a0 0x7fce14160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 -- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce140ce2a0 msgr2=0x7fce14160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce140ce2a0 0x7fce14160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 -- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce140cc840 msgr2=0x7fce1406e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce140cc840 0x7fce1406e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 -- 172.21.15.67:0/2676047092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce14158180 con 0x7fce140cc250 2024-04-17T10:28:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce12a00640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140cc250 0x7fce14149a60 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c420 tx=0x7fce0400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce0b400640 1 -- 172.21.15.67:0/2676047092 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce040052d0 con 0x7fce140cc250 2024-04-17T10:28:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce0b400640 1 -- 172.21.15.67:0/2676047092 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce04005470 con 0x7fce140cc250 2024-04-17T10:28:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce0b400640 1 -- 172.21.15.67:0/2676047092 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce04007260 con 0x7fce140cc250 2024-04-17T10:28:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce14158460 con 0x7fce140cc250 2024-04-17T10:28:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.602+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce14158a20 con 0x7fce140cc250 2024-04-17T10:28:55.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.606+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce1406cd30 con 0x7fce140cc250 2024-04-17T10:28:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.606+0000 7fce0b400640 1 -- 172.21.15.67:0/2676047092 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fce04007400 con 0x7fce140cc250 2024-04-17T10:28:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.606+0000 7fce0b400640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdf0066f60 0x7fcdf0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.606+0000 7fce0b400640 1 -- 172.21.15.67:0/2676047092 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fce04084420 con 0x7fce140cc250 2024-04-17T10:28:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.606+0000 7fce0a000640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdf0066f60 0x7fcdf0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.606+0000 7fce0a000640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdf0066f60 0x7fcdf0069420 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0af000 tx=0x7fcdfc0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.610+0000 7fce0b400640 1 -- 172.21.15.67:0/2676047092 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fce04054e70 con 0x7fce140cc250 2024-04-17T10:28:55.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:28:55.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:28:55.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce1409e900 con 0x7fce140cc250 2024-04-17T10:28:55.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce0b400640 1 -- 172.21.15.67:0/2676047092 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+13116 (secure 0 0 0) 0x7fce04058f10 con 0x7fce140cc250 2024-04-17T10:28:55.767 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:55.767 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":63,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:55.035111+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:55.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdf0066f60 msgr2=0x7fcdf0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdf0066f60 0x7fcdf0069420 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0af000 tx=0x7fcdfc0a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140cc250 msgr2=0x7fce14149a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140cc250 0x7fce14149a60 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c420 tx=0x7fce0400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 shutdown_connections 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdf0066f60 0x7fcdf0069420 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce140ce2a0 0x7fce14160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce140cc840 0x7fce1406e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 --2- 172.21.15.67:0/2676047092 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce140cc250 0x7fce14149a60 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 >> 172.21.15.67:0/2676047092 conn(0x7fce140b22a0 msgr2=0x7fce1415e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 shutdown_connections 2024-04-17T10:28:55.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.774+0000 7fce19600640 1 -- 172.21.15.67:0/2676047092 wait complete. 2024-04-17T10:28:55.789 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:28:55.789 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:28:55.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd330e00640 1 Processor -- start 2024-04-17T10:28:55.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd330e00640 1 -- start start 2024-04-17T10:28:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd330e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd330e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd32c06b3a0 0x7fd32c0d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd330e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32c1616a0 con 0x7fd32c0b3920 2024-04-17T10:28:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd330e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32c161940 con 0x7fd32c06b3a0 2024-04-17T10:28:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd330e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c161c20 con 0x7fd32c1474c0 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd32a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd32c06b3a0 0x7fd32c0d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd32a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd32c06b3a0 0x7fd32c0d0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59782/0 (socket says 172.21.15.67:59782) 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd32a800640 1 -- 172.21.15.67:0/149878703 learned_addr learned my addr 172.21.15.67:0/149878703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd329e00640 1 --2- 172.21.15.67:0/149878703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd32a800640 1 -- 172.21.15.67:0/149878703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 msgr2=0x7fd32c0b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd32a800640 1 --2- 172.21.15.67:0/149878703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd32a800640 1 -- 172.21.15.67:0/149878703 >> v1:172.21.15.86:6789/0 conn(0x7fd32c0b3920 legacy=0x7fd32c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.942+0000 7fd32a800640 1 -- 172.21.15.67:0/149878703 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c161f00 con 0x7fd32c06b3a0 2024-04-17T10:28:55.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd32a800640 1 --2- 172.21.15.67:0/149878703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0d0480 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd324009e50 tx=0x7fd32401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e1d8332cec4c98 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd329400640 1 -- 172.21.15.67:0/149878703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3240b4070 con 0x7fd32c06b3a0 2024-04-17T10:28:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd329400640 1 -- 172.21.15.67:0/149878703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3240af400 con 0x7fd32c06b3a0 2024-04-17T10:28:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd329400640 1 -- 172.21.15.67:0/149878703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3240af7d0 con 0x7fd32c06b3a0 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 -- 172.21.15.67:0/149878703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 msgr2=0x7fd32c0d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 --2- 172.21.15.67:0/149878703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0d0480 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fd324009e50 tx=0x7fd32401f210 comp rx=0 tx=0).stop 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 -- 172.21.15.67:0/149878703 shutdown_connections 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 --2- 172.21.15.67:0/149878703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0d0480 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 --2- 172.21.15.67:0/149878703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 -- 172.21.15.67:0/149878703 >> 172.21.15.67:0/149878703 conn(0x7fd32c0b5ef0 msgr2=0x7fd32c099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 -- 172.21.15.67:0/149878703 shutdown_connections 2024-04-17T10:28:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 -- 172.21.15.67:0/149878703 wait complete. 2024-04-17T10:28:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 Processor -- start 2024-04-17T10:28:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.946+0000 7fd330e00640 1 -- start start 2024-04-17T10:28:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c0b3920 0x7fd32c0cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0b3d40 con 0x7fd32c06b3a0 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c162420 con 0x7fd32c0b3920 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32c0d09c0 con 0x7fd32c1474c0 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0cf690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59796/0 (socket says 172.21.15.67:59796) 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 -- 172.21.15.67:0/1820249032 learned_addr learned my addr 172.21.15.67:0/1820249032 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd32a800640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd321400640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c0b3920 0x7fd32c0cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 -- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 msgr2=0x7fd32c0aec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0aec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 -- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c0b3920 msgr2=0x7fd32c0cfbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c0b3920 0x7fd32c0cfbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 -- 172.21.15.67:0/1820249032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32c0af9d0 con 0x7fd32c06b3a0 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd321400640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c0b3920 0x7fd32c0cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd329e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0cf690 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00e8a0 tx=0x7fd31c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd322a00640 1 -- 172.21.15.67:0/1820249032 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c017070 con 0x7fd32c06b3a0 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd322a00640 1 -- 172.21.15.67:0/1820249032 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31c00c940 con 0x7fd32c06b3a0 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd322a00640 1 -- 172.21.15.67:0/1820249032 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c00ccf0 con 0x7fd32c06b3a0 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32c0afcb0 con 0x7fd32c06b3a0 2024-04-17T10:28:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32c12cd70 con 0x7fd32c06b3a0 2024-04-17T10:28:55.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.950+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32c04ef30 con 0x7fd32c06b3a0 2024-04-17T10:28:55.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.954+0000 7fd322a00640 1 -- 172.21.15.67:0/1820249032 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd31c01a070 con 0x7fd32c06b3a0 2024-04-17T10:28:55.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.954+0000 7fd322a00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd308066f60 0x7fd308069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.954+0000 7fd322a00640 1 -- 172.21.15.67:0/1820249032 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd31c084ff0 con 0x7fd32c06b3a0 2024-04-17T10:28:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.954+0000 7fd321400640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd308066f60 0x7fd308069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.954+0000 7fd321400640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd308066f60 0x7fd308069420 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd32c1e73d0 tx=0x7fd314009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:55.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:55.958+0000 7fd322a00640 1 -- 172.21.15.67:0/1820249032 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd31c055b70 con 0x7fd32c06b3a0 2024-04-17T10:28:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.122+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd32c09a310 con 0x7fd32c06b3a0 2024-04-17T10:28:56.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd322a00640 1 -- 172.21.15.67:0/1820249032 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd31c059c10 con 0x7fd32c06b3a0 2024-04-17T10:28:56.118 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:28:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd308066f60 msgr2=0x7fd308069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd308066f60 0x7fd308069420 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd32c1e73d0 tx=0x7fd314009040 comp rx=0 tx=0).stop 2024-04-17T10:28:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 msgr2=0x7fd32c0cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0cf690 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00e8a0 tx=0x7fd31c00eca0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 shutdown_connections 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd308066f60 0x7fd308069420 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd32c1474c0 0x7fd32c0aec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd32c0b3920 0x7fd32c0cfbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 --2- 172.21.15.67:0/1820249032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd32c06b3a0 0x7fd32c0cf690 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 >> 172.21.15.67:0/1820249032 conn(0x7fd32c0b5ef0 msgr2=0x7fd32c0b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 shutdown_connections 2024-04-17T10:28:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.126+0000 7fd330e00640 1 -- 172.21.15.67:0/1820249032 wait complete. 2024-04-17T10:28:56.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:28:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c62e00640 1 Processor -- start 2024-04-17T10:28:56.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c62e00640 1 -- start start 2024-04-17T10:28:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c62e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c5c0cff90 0x7f3c5c0cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c62e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0cc8b0 con 0x7f3c5c1597a0 2024-04-17T10:28:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c62e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0ccb90 con 0x7f3c5c06b6a0 2024-04-17T10:28:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c62e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5c0cd4f0 con 0x7f3c5c0cff90 2024-04-17T10:28:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c60e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c5c0cff90 0x7f3c5c0cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c60e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c5c0cff90 0x7f3c5c0cc260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59798/0 (socket says 172.21.15.67:59798) 2024-04-17T10:28:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c60e00640 1 -- 172.21.15.67:0/3495524945 learned_addr learned my addr 172.21.15.67:0/3495524945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c5be00640 1 --1- 172.21.15.67:0/3495524945 >> v1:172.21.15.67:6789/0 conn(0x7f3c5c1597a0 0x7f3c5c0cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46382/0 (socket says 172.21.15.67:46382) 2024-04-17T10:28:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c60e00640 1 -- 172.21.15.67:0/3495524945 >> v1:172.21.15.86:6789/0 conn(0x7f3c5c06b6a0 legacy=0x7f3c5c0cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c60e00640 1 -- 172.21.15.67:0/3495524945 >> v1:172.21.15.67:6789/0 conn(0x7f3c5c1597a0 legacy=0x7f3c5c0cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.286+0000 7f3c60e00640 1 -- 172.21.15.67:0/3495524945 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0cd180 con 0x7f3c5c0cff90 2024-04-17T10:28:56.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c60e00640 1 --2- 172.21.15.67:0/3495524945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0cc260 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009e50 tx=0x7f3c5401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7931420d3d484a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c5b400640 1 -- 172.21.15.67:0/3495524945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c540a58b0 con 0x7f3c5c0cff90 2024-04-17T10:28:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c5b400640 1 -- 172.21.15.67:0/3495524945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c540a5a50 con 0x7f3c5c0cff90 2024-04-17T10:28:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c5b400640 1 -- 172.21.15.67:0/3495524945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c540a5e20 con 0x7f3c5c0cff90 2024-04-17T10:28:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c62e00640 1 -- 172.21.15.67:0/3495524945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 msgr2=0x7f3c5c0cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c62e00640 1 --2- 172.21.15.67:0/3495524945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0cc260 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009e50 tx=0x7f3c5401c9d0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c62e00640 1 -- 172.21.15.67:0/3495524945 shutdown_connections 2024-04-17T10:28:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c62e00640 1 --2- 172.21.15.67:0/3495524945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0cc260 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c62e00640 1 -- 172.21.15.67:0/3495524945 >> 172.21.15.67:0/3495524945 conn(0x7f3c5c0b22a0 msgr2=0x7f3c5c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c62e00640 1 -- 172.21.15.67:0/3495524945 shutdown_connections 2024-04-17T10:28:56.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.290+0000 7f3c62e00640 1 -- 172.21.15.67:0/3495524945 wait complete. 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 Processor -- start 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 -- start start 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c5c06b6a0 0x7f3c5c1514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0ca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c5c1597a0 0x7f3c5c1503c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c1695d0 con 0x7f3c5c0cff90 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c15ca50 con 0x7f3c5c06b6a0 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c169750 con 0x7f3c5c1597a0 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0ca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0ca0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59804/0 (socket says 172.21.15.67:59804) 2024-04-17T10:28:56.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 -- 172.21.15.67:0/2234853666 learned_addr learned my addr 172.21.15.67:0/2234853666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c60e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c5c1597a0 0x7f3c5c1503c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c5be00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c5c06b6a0 0x7f3c5c1514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 -- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c5c1597a0 msgr2=0x7f3c5c1503c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c5c1597a0 0x7f3c5c1503c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 -- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c5c06b6a0 msgr2=0x7f3c5c1514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c5c06b6a0 0x7f3c5c1514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 -- 172.21.15.67:0/2234853666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c151b20 con 0x7f3c5c0cff90 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c53e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0ca0e0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400c420 tx=0x7f3c4400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c58c00640 1 -- 172.21.15.67:0/2234853666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c440052d0 con 0x7f3c5c0cff90 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c58c00640 1 -- 172.21.15.67:0/2234853666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c44005470 con 0x7f3c5c0cff90 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c151e00 con 0x7f3c5c0cff90 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c58c00640 1 -- 172.21.15.67:0/2234853666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c44007260 con 0x7f3c5c0cff90 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.294+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c1523c0 con 0x7f3c5c0cff90 2024-04-17T10:28:56.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.298+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c09e900 con 0x7f3c5c0cff90 2024-04-17T10:28:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.298+0000 7f3c58c00640 1 -- 172.21.15.67:0/2234853666 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3c44007400 con 0x7f3c5c0cff90 2024-04-17T10:28:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.298+0000 7f3c58c00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c38066f60 0x7f3c38069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.298+0000 7f3c58c00640 1 -- 172.21.15.67:0/2234853666 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f3c44084360 con 0x7f3c5c0cff90 2024-04-17T10:28:56.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.298+0000 7f3c5be00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c38066f60 0x7f3c38069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.298+0000 7f3c5be00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c38066f60 0x7f3c38069420 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c000f70 tx=0x7f3c4c007e50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:56.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.302+0000 7f3c58c00640 1 -- 172.21.15.67:0/2234853666 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3c44054e40 con 0x7f3c5c0cff90 2024-04-17T10:28:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.466+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3c5c074e30 con 0x7f3c5c0cff90 2024-04-17T10:28:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.466+0000 7f3c58c00640 1 -- 172.21.15.67:0/2234853666 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3c44058ee0 con 0x7f3c5c0cff90 2024-04-17T10:28:56.460 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:56.460 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:28:56.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:28:56.460 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:28:56.460 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:28:56.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:28:56.461 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:28:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c38066f60 msgr2=0x7f3c38069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c38066f60 0x7f3c38069420 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c000f70 tx=0x7f3c4c007e50 comp rx=0 tx=0).stop 2024-04-17T10:28:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 msgr2=0x7f3c5c0ca0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0ca0e0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3c4400c420 tx=0x7f3c4400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 shutdown_connections 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c38066f60 0x7f3c38069420 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c5c1597a0 0x7f3c5c1503c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c5c0cff90 0x7f3c5c0ca0e0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 --2- 172.21.15.67:0/2234853666 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c5c06b6a0 0x7f3c5c1514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 >> 172.21.15.67:0/2234853666 conn(0x7f3c5c0b22a0 msgr2=0x7f3c5c06e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 shutdown_connections 2024-04-17T10:28:56.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.470+0000 7f3c62e00640 1 -- 172.21.15.67:0/2234853666 wait complete. 2024-04-17T10:28:56.481 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:28:56.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:28:56.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 Processor -- start 2024-04-17T10:28:56.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 -- start start 2024-04-17T10:28:56.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2478099b30 0x7f247806cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2478099f10 0x7f247815de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247815e3d0 0x7f2478161790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247806c0f0 con 0x7f2478099b30 2024-04-17T10:28:56.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24780cf980 con 0x7f2478099f10 2024-04-17T10:28:56.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f247d600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247815f150 con 0x7f247815e3d0 2024-04-17T10:28:56.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2478099b30 0x7f247806cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2478099b30 0x7f247806cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59820/0 (socket says 172.21.15.67:59820) 2024-04-17T10:28:56.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 -- 172.21.15.67:0/689795878 learned_addr learned my addr 172.21.15.67:0/689795878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f246e000640 1 --2- 172.21.15.67:0/689795878 >> v2:172.21.15.86:3300/0 conn(0x7f2478099f10 0x7f247815de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2477400640 1 --2- 172.21.15.67:0/689795878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247815e3d0 0x7f2478161790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 -- 172.21.15.67:0/689795878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247815e3d0 msgr2=0x7f2478161790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 --2- 172.21.15.67:0/689795878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247815e3d0 0x7f2478161790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 -- 172.21.15.67:0/689795878 >> v2:172.21.15.86:3300/0 conn(0x7f2478099f10 msgr2=0x7f247815de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 --2- 172.21.15.67:0/689795878 >> v2:172.21.15.86:3300/0 conn(0x7f2478099f10 0x7f247815de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2476a00640 1 -- 172.21.15.67:0/689795878 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247815e8b0 con 0x7f2478099b30 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f246e000640 1 --2- 172.21.15.67:0/689795878 >> v2:172.21.15.86:3300/0 conn(0x7f2478099f10 0x7f247815de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.634+0000 7f2477400640 1 --2- 172.21.15.67:0/689795878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247815e3d0 0x7f2478161790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:56.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f2476a00640 1 --2- 172.21.15.67:0/689795878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2478099b30 0x7f247806cd70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2478068970 tx=0x7f246801c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265bffb72a733a8b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:56.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f2476000640 1 -- 172.21.15.67:0/689795878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24680b0070 con 0x7f2478099b30 2024-04-17T10:28:56.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f2476000640 1 -- 172.21.15.67:0/689795878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24680ab600 con 0x7f2478099b30 2024-04-17T10:28:56.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f2476000640 1 -- 172.21.15.67:0/689795878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24680ab9d0 con 0x7f2478099b30 2024-04-17T10:28:56.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 -- 172.21.15.67:0/689795878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2478099b30 msgr2=0x7f247806cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 --2- 172.21.15.67:0/689795878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2478099b30 0x7f247806cd70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2478068970 tx=0x7f246801c230 comp rx=0 tx=0).stop 2024-04-17T10:28:56.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 -- 172.21.15.67:0/689795878 shutdown_connections 2024-04-17T10:28:56.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 --2- 172.21.15.67:0/689795878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247815e3d0 0x7f2478161790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 --2- 172.21.15.67:0/689795878 >> v2:172.21.15.86:3300/0 conn(0x7f2478099f10 0x7f247815de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 --2- 172.21.15.67:0/689795878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2478099b30 0x7f247806cd70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 -- 172.21.15.67:0/689795878 >> 172.21.15.67:0/689795878 conn(0x7f24780b32e0 msgr2=0x7f24780b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 -- 172.21.15.67:0/689795878 shutdown_connections 2024-04-17T10:28:56.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.638+0000 7f247d600640 1 -- 172.21.15.67:0/689795878 wait complete. 2024-04-17T10:28:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 Processor -- start 2024-04-17T10:28:56.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 -- start start 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2478099b30 0x7f247814f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2478099f10 0x7f2478160190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247815e3d0 0x7f24781606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247809a4a0 con 0x7f247815e3d0 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247809a320 con 0x7f2478099b30 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24780d00b0 con 0x7f2478099f10 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247815e3d0 0x7f24781606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2476a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2478099b30 0x7f247814f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247815e3d0 0x7f24781606d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59834/0 (socket says 172.21.15.67:59834) 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2476a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2478099b30 0x7f247814f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39382/0 (socket says 172.21.15.67:39382) 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 -- 172.21.15.67:0/3092209174 learned_addr learned my addr 172.21.15.67:0/3092209174 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f246e000640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2478099f10 0x7f2478160190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 -- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2478099f10 msgr2=0x7f2478160190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2478099f10 0x7f2478160190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 -- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2478099b30 msgr2=0x7f247814f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2478099b30 0x7f247814f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 -- 172.21.15.67:0/3092209174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24781586a0 con 0x7f247815e3d0 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f2477400640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247815e3d0 0x7f24781606d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f247000cd80 tx=0x7f247000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f246f400640 1 -- 172.21.15.67:0/3092209174 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470003ba0 con 0x7f247815e3d0 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f246f400640 1 -- 172.21.15.67:0/3092209174 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2470003d60 con 0x7f247815e3d0 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2478155870 con 0x7f247815e3d0 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f246f400640 1 -- 172.21.15.67:0/3092209174 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2470004260 con 0x7f247815e3d0 2024-04-17T10:28:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2478158c20 con 0x7f247815e3d0 2024-04-17T10:28:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.642+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247809f900 con 0x7f247815e3d0 2024-04-17T10:28:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.646+0000 7f246f400640 1 -- 172.21.15.67:0/3092209174 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2470004400 con 0x7f247815e3d0 2024-04-17T10:28:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.646+0000 7f246f400640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2454066f60 0x7f2454069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.646+0000 7f246f400640 1 -- 172.21.15.67:0/3092209174 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f24700857d0 con 0x7f247815e3d0 2024-04-17T10:28:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.646+0000 7f2476a00640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2454066f60 0x7f2454069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:56.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.646+0000 7f2476a00640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2454066f60 0x7f2454069420 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f24680ad000 tx=0x7f2468008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:56.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.650+0000 7f246f400640 1 -- 172.21.15.67:0/3092209174 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2470014030 con 0x7f247815e3d0 2024-04-17T10:28:56.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:56.810+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2478074e30 con 0x7f247815e3d0 2024-04-17T10:28:57.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f246f400640 1 -- 172.21.15.67:0/3092209174 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2470056340 con 0x7f247815e3d0 2024-04-17T10:28:57.022 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2454066f60 msgr2=0x7f2454069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2454066f60 0x7f2454069420 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f24680ad000 tx=0x7f2468008040 comp rx=0 tx=0).stop 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247815e3d0 msgr2=0x7f24781606d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247815e3d0 0x7f24781606d0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f247000cd80 tx=0x7f247000a650 comp rx=0 tx=0).stop 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 shutdown_connections 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2454066f60 0x7f2454069420 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f247815e3d0 0x7f24781606d0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2478099f10 0x7f2478160190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 --2- 172.21.15.67:0/3092209174 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2478099b30 0x7f247814f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 >> 172.21.15.67:0/3092209174 conn(0x7f24780b32e0 msgr2=0x7f24780b6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 shutdown_connections 2024-04-17T10:28:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.030+0000 7f247d600640 1 -- 172.21.15.67:0/3092209174 wait complete. 2024-04-17T10:28:57.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:28:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 Processor -- start 2024-04-17T10:28:57.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 -- start start 2024-04-17T10:28:57.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1300aef60 0x7fd1300cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 0x7fd13006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd13006e7b0 0x7fd1300713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1300cd2c0 con 0x7fd1300aef60 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1300af340 con 0x7fd13006e7b0 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd135e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1300b4990 con 0x7fd1300cce60 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1300aef60 0x7fd1300cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd1300aef60 0x7fd1300cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59846/0 (socket says 172.21.15.67:59846) 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 -- 172.21.15.67:0/318727673 learned_addr learned my addr 172.21.15.67:0/318727673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 -- 172.21.15.67:0/318727673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 msgr2=0x7fd13006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 --2- 172.21.15.67:0/318727673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 0x7fd13006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 -- 172.21.15.67:0/318727673 >> v2:172.21.15.86:3300/0 conn(0x7fd13006e7b0 msgr2=0x7fd1300713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 --2- 172.21.15.67:0/318727673 >> v2:172.21.15.86:3300/0 conn(0x7fd13006e7b0 0x7fd1300713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.206+0000 7fd12ea00640 1 -- 172.21.15.67:0/318727673 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13015ae90 con 0x7fd1300aef60 2024-04-17T10:28:57.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd12ea00640 1 --2- 172.21.15.67:0/318727673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300aef60 0x7fd1300cc920 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b2900 tx=0x7fd12001c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77776b3ca4673b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:57.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd12d600640 1 -- 172.21.15.67:0/318727673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1200b0070 con 0x7fd1300aef60 2024-04-17T10:28:57.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd12d600640 1 -- 172.21.15.67:0/318727673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1200ab600 con 0x7fd1300aef60 2024-04-17T10:28:57.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd12d600640 1 -- 172.21.15.67:0/318727673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1200ab9d0 con 0x7fd1300aef60 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 -- 172.21.15.67:0/318727673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300aef60 msgr2=0x7fd1300cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 --2- 172.21.15.67:0/318727673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300aef60 0x7fd1300cc920 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd1300b2900 tx=0x7fd12001c1d0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 -- 172.21.15.67:0/318727673 shutdown_connections 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 --2- 172.21.15.67:0/318727673 >> v2:172.21.15.86:3300/0 conn(0x7fd13006e7b0 0x7fd1300713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 --2- 172.21.15.67:0/318727673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 0x7fd13006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 --2- 172.21.15.67:0/318727673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd1300aef60 0x7fd1300cc920 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 -- 172.21.15.67:0/318727673 >> 172.21.15.67:0/318727673 conn(0x7fd1300b1740 msgr2=0x7fd1300b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 -- 172.21.15.67:0/318727673 shutdown_connections 2024-04-17T10:28:57.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 -- 172.21.15.67:0/318727673 wait complete. 2024-04-17T10:28:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.210+0000 7fd135e00640 1 Processor -- start 2024-04-17T10:28:57.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 -- start start 2024-04-17T10:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd13006e7b0 0x7fd130137c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1300aef60 0x7fd1301f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 0x7fd1300b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13015bc10 con 0x7fd13006e7b0 2024-04-17T10:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd13015b410 con 0x7fd1300aef60 2024-04-17T10:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1301f7e30 con 0x7fd1300cce60 2024-04-17T10:28:57.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd13006e7b0 0x7fd130137c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd13006e7b0 0x7fd130137c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59856/0 (socket says 172.21.15.67:59856) 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 -- 172.21.15.67:0/631430792 learned_addr learned my addr 172.21.15.67:0/631430792 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12f400640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 0x7fd1300b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12e000640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1300aef60 0x7fd1301f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 -- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 msgr2=0x7fd1300b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 0x7fd1300b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 -- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1300aef60 msgr2=0x7fd1301f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1300aef60 0x7fd1301f6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 -- 172.21.15.67:0/631430792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1301f73a0 con 0x7fd13006e7b0 2024-04-17T10:28:57.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd12ea00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd13006e7b0 0x7fd130137c60 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd1200ae040 tx=0x7fd12001aa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd127400640 1 -- 172.21.15.67:0/631430792 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1200ba020 con 0x7fd13006e7b0 2024-04-17T10:28:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd127400640 1 -- 172.21.15.67:0/631430792 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1200aa9d0 con 0x7fd13006e7b0 2024-04-17T10:28:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd127400640 1 -- 172.21.15.67:0/631430792 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1200b0040 con 0x7fd13006e7b0 2024-04-17T10:28:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1301f7680 con 0x7fd13006e7b0 2024-04-17T10:28:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1301383b0 con 0x7fd13006e7b0 2024-04-17T10:28:57.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.214+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd13004ef30 con 0x7fd13006e7b0 2024-04-17T10:28:57.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.218+0000 7fd127400640 1 -- 172.21.15.67:0/631430792 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd1200bd020 con 0x7fd13006e7b0 2024-04-17T10:28:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.218+0000 7fd127400640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd110066f60 0x7fd110069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.218+0000 7fd127400640 1 -- 172.21.15.67:0/631430792 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fd12011fbd0 con 0x7fd13006e7b0 2024-04-17T10:28:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.218+0000 7fd12e000640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd110066f60 0x7fd110069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:57.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.218+0000 7fd12e000640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd110066f60 0x7fd110069420 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd11c006fd0 tx=0x7fd11c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:57.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.222+0000 7fd127400640 1 -- 172.21.15.67:0/631430792 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd1200c4030 con 0x7fd13006e7b0 2024-04-17T10:28:57.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:57.386+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd1300af900 con 0x7fd13006e7b0 2024-04-17T10:28:58.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.034+0000 7fd127400640 1 -- 172.21.15.67:0/631430792 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd120016110 con 0x7fd13006e7b0 2024-04-17T10:28:58.030 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:28:58.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.038+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd110066f60 msgr2=0x7fd110069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:58.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.038+0000 7fd135e00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd110066f60 0x7fd110069420 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd11c006fd0 tx=0x7fd11c008040 comp rx=0 tx=0).stop 2024-04-17T10:28:58.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.038+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd13006e7b0 msgr2=0x7fd130137c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:58.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.038+0000 7fd135e00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd13006e7b0 0x7fd130137c60 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd1200ae040 tx=0x7fd12001aa10 comp rx=0 tx=0).stop 2024-04-17T10:28:58.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 shutdown_connections 2024-04-17T10:28:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd110066f60 0x7fd110069420 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd1300cce60 0x7fd1300b09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd1300aef60 0x7fd1301f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 --2- 172.21.15.67:0/631430792 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd13006e7b0 0x7fd130137c60 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 >> 172.21.15.67:0/631430792 conn(0x7fd1300b1740 msgr2=0x7fd13014a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 shutdown_connections 2024-04-17T10:28:58.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.042+0000 7fd135e00640 1 -- 172.21.15.67:0/631430792 wait complete. 2024-04-17T10:28:58.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:28:58.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa53c00640 1 Processor -- start 2024-04-17T10:28:58.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa53c00640 1 -- start start 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa53c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa4c159440 0x7faa4c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa53c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c15c580 0x7faa4c071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa53c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa4c15cb30 con 0x7faa4c0713e0 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa53c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa4c0a8650 con 0x7faa4c159440 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa53c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c15ce10 con 0x7faa4c15c580 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa51200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa4c159440 0x7faa4c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa51200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa4c159440 0x7faa4c0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59868/0 (socket says 172.21.15.67:59868) 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa51200640 1 -- 172.21.15.67:0/2917648093 learned_addr learned my addr 172.21.15.67:0/2917648093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.226+0000 7faa51c00640 1 --2- 172.21.15.67:0/2917648093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c15c580 0x7faa4c071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa51200640 1 -- 172.21.15.67:0/2917648093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c15c580 msgr2=0x7faa4c071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa51200640 1 --2- 172.21.15.67:0/2917648093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c15c580 0x7faa4c071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa51200640 1 -- 172.21.15.67:0/2917648093 >> v1:172.21.15.86:6789/0 conn(0x7faa4c0713e0 legacy=0x7faa4c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa51200640 1 -- 172.21.15.67:0/2917648093 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c0cd610 con 0x7faa4c159440 2024-04-17T10:28:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa51200640 1 --2- 172.21.15.67:0/2917648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c159440 0x7faa4c0ce0f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009870 tx=0x7faa3c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fd07888201ccc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa4be00640 1 -- 172.21.15.67:0/2917648093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c0a7070 con 0x7faa4c159440 2024-04-17T10:28:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa4be00640 1 -- 172.21.15.67:0/2917648093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c0b0070 con 0x7faa4c159440 2024-04-17T10:28:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa4be00640 1 -- 172.21.15.67:0/2917648093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c0ab890 con 0x7faa4c159440 2024-04-17T10:28:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 -- 172.21.15.67:0/2917648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c159440 msgr2=0x7faa4c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 --2- 172.21.15.67:0/2917648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c159440 0x7faa4c0ce0f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009870 tx=0x7faa3c01c560 comp rx=0 tx=0).stop 2024-04-17T10:28:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 -- 172.21.15.67:0/2917648093 shutdown_connections 2024-04-17T10:28:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 --2- 172.21.15.67:0/2917648093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c15c580 0x7faa4c071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 --2- 172.21.15.67:0/2917648093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c159440 0x7faa4c0ce0f0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 -- 172.21.15.67:0/2917648093 >> 172.21.15.67:0/2917648093 conn(0x7faa4c0b1740 msgr2=0x7faa4c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 -- 172.21.15.67:0/2917648093 shutdown_connections 2024-04-17T10:28:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.230+0000 7faa53c00640 1 -- 172.21.15.67:0/2917648093 wait complete. 2024-04-17T10:28:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 Processor -- start 2024-04-17T10:28:58.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 -- start start 2024-04-17T10:28:58.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c0713e0 0x7faa4c153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa4c159440 0x7faa4c14bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c15c580 0x7faa4c14c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c15d9c0 con 0x7faa4c15c580 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c15d840 con 0x7faa4c159440 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa4c15db40 con 0x7faa4c0713e0 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c15c580 0x7faa4c14c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c15c580 0x7faa4c14c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59872/0 (socket says 172.21.15.67:59872) 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 -- 172.21.15.67:0/3976664321 learned_addr learned my addr 172.21.15.67:0/3976664321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51200640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c0713e0 0x7faa4c153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 -- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c0713e0 msgr2=0x7faa4c153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c0713e0 0x7faa4c153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa43e00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa4c159440 0x7faa4c14bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 -- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa4c159440 msgr2=0x7faa4c14bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:58.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa4c159440 0x7faa4c14bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 -- 172.21.15.67:0/3976664321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c154440 con 0x7faa4c15c580 2024-04-17T10:28:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51200640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c0713e0 0x7faa4c153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa43e00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa4c159440 0x7faa4c14bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa51c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c15c580 0x7faa4c14c290 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7faa44009d30 tx=0x7faa4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa49600640 1 -- 172.21.15.67:0/3976664321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44018020 con 0x7faa4c15c580 2024-04-17T10:28:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa49600640 1 -- 172.21.15.67:0/3976664321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa44014480 con 0x7faa4c15c580 2024-04-17T10:28:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa49600640 1 -- 172.21.15.67:0/3976664321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44014870 con 0x7faa4c15c580 2024-04-17T10:28:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4c154720 con 0x7faa4c15c580 2024-04-17T10:28:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.234+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4c152600 con 0x7faa4c15c580 2024-04-17T10:28:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.238+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa4c152800 con 0x7faa4c15c580 2024-04-17T10:28:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.238+0000 7faa49600640 1 -- 172.21.15.67:0/3976664321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7faa4401b020 con 0x7faa4c15c580 2024-04-17T10:28:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.238+0000 7faa49600640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa28066f60 0x7faa28069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.238+0000 7faa49600640 1 -- 172.21.15.67:0/3976664321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6001+0+0 (secure 0 0 0) 0x7faa44085b90 con 0x7faa4c15c580 2024-04-17T10:28:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.238+0000 7faa51200640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa28066f60 0x7faa28069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.238+0000 7faa51200640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa28066f60 0x7faa28069420 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0af000 tx=0x7faa3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.242+0000 7faa49600640 1 -- 172.21.15.67:0/3976664321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7faa4c152800 con 0x7faa4c15c580 2024-04-17T10:28:58.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:58.438+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7faa4c15a790 con 0x7faa4c15c580 2024-04-17T10:28:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.038+0000 7faa49600640 1 -- 172.21.15.67:0/3976664321 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v36) v1 ==== 173+0+0 (secure 0 0 0) 0x7faa44022070 con 0x7faa4c15c580 2024-04-17T10:28:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 9 and data pool 10 2024-04-17T10:28:59.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa28066f60 msgr2=0x7faa28069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa28066f60 0x7faa28069420 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0af000 tx=0x7faa3c008040 comp rx=0 tx=0).stop 2024-04-17T10:28:59.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c15c580 msgr2=0x7faa4c14c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c15c580 0x7faa4c14c290 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7faa44009d30 tx=0x7faa4400c490 comp rx=0 tx=0).stop 2024-04-17T10:28:59.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 shutdown_connections 2024-04-17T10:28:59.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa28066f60 0x7faa28069420 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa4c15c580 0x7faa4c14c290 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa4c159440 0x7faa4c14bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 --2- 172.21.15.67:0/3976664321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa4c0713e0 0x7faa4c153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 >> 172.21.15.67:0/3976664321 conn(0x7faa4c0b1740 msgr2=0x7faa4c1560d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 shutdown_connections 2024-04-17T10:28:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.042+0000 7faa53c00640 1 -- 172.21.15.67:0/3976664321 wait complete. 2024-04-17T10:28:59.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.230+0000 7f69ec400640 1 Processor -- start 2024-04-17T10:28:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.230+0000 7f69ec400640 1 -- start start 2024-04-17T10:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ec400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69e415dd10 0x7f69e40b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ec400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69e40b57b0 0x7f69e4070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ec400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e40b07b0 0x7f69e4071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ec400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e40b1280 con 0x7f69e40b57b0 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ec400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e40b0cc0 con 0x7f69e415dd10 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ec400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e40b0fa0 con 0x7f69e40b07b0 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69e40b57b0 0x7f69e4070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f69e40b57b0 0x7f69e4070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59886/0 (socket says 172.21.15.67:59886) 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 -- 172.21.15.67:0/362496271 learned_addr learned my addr 172.21.15.67:0/362496271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9a00640 1 --2- 172.21.15.67:0/362496271 >> v2:172.21.15.86:3300/0 conn(0x7f69e415dd10 0x7f69e40b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ea400640 1 --2- 172.21.15.67:0/362496271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e40b07b0 0x7f69e4071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 -- 172.21.15.67:0/362496271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e40b07b0 msgr2=0x7f69e4071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 --2- 172.21.15.67:0/362496271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e40b07b0 0x7f69e4071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 -- 172.21.15.67:0/362496271 >> v2:172.21.15.86:3300/0 conn(0x7f69e415dd10 msgr2=0x7f69e40b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 --2- 172.21.15.67:0/362496271 >> v2:172.21.15.86:3300/0 conn(0x7f69e415dd10 0x7f69e40b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9000640 1 -- 172.21.15.67:0/362496271 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e406bed0 con 0x7f69e40b57b0 2024-04-17T10:28:59.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69e9a00640 1 --2- 172.21.15.67:0/362496271 >> v2:172.21.15.86:3300/0 conn(0x7f69e415dd10 0x7f69e40b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:59.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.234+0000 7f69ea400640 1 --2- 172.21.15.67:0/362496271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e40b07b0 0x7f69e4071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:28:59.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69e9000640 1 --2- 172.21.15.67:0/362496271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b57b0 0x7f69e4070dc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f69e40d2900 tx=0x7f69d001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f24d03037370f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69e3e00640 1 -- 172.21.15.67:0/362496271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00ab600 con 0x7f69e40b57b0 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69e3e00640 1 -- 172.21.15.67:0/362496271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d00b0070 con 0x7f69e40b57b0 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69e3e00640 1 -- 172.21.15.67:0/362496271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00ab9c0 con 0x7f69e40b57b0 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 -- 172.21.15.67:0/362496271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b57b0 msgr2=0x7f69e4070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 --2- 172.21.15.67:0/362496271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b57b0 0x7f69e4070dc0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f69e40d2900 tx=0x7f69d001c460 comp rx=0 tx=0).stop 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 -- 172.21.15.67:0/362496271 shutdown_connections 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 --2- 172.21.15.67:0/362496271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e40b07b0 0x7f69e4071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 --2- 172.21.15.67:0/362496271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b57b0 0x7f69e4070dc0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 --2- 172.21.15.67:0/362496271 >> v2:172.21.15.86:3300/0 conn(0x7f69e415dd10 0x7f69e40b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 -- 172.21.15.67:0/362496271 >> 172.21.15.67:0/362496271 conn(0x7f69e40b34d0 msgr2=0x7f69e40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 -- 172.21.15.67:0/362496271 shutdown_connections 2024-04-17T10:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 -- 172.21.15.67:0/362496271 wait complete. 2024-04-17T10:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 Processor -- start 2024-04-17T10:28:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.238+0000 7f69ec400640 1 -- start start 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b07b0 0x7f69e415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e415dd10 0x7f69e414e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69e41504c0 0x7f69e4158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4162460 con 0x7f69e40b07b0 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e41622e0 con 0x7f69e41504c0 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e41625e0 con 0x7f69e415dd10 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b07b0 0x7f69e415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b07b0 0x7f69e415a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59900/0 (socket says 172.21.15.67:59900) 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 -- 172.21.15.67:0/1906638543 learned_addr learned my addr 172.21.15.67:0/1906638543 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ea400640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69e41504c0 0x7f69e4158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9000640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e415dd10 0x7f69e414e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 -- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e415dd10 msgr2=0x7f69e414e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e415dd10 0x7f69e414e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 -- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69e41504c0 msgr2=0x7f69e4158610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69e41504c0 0x7f69e4158610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 -- 172.21.15.67:0/1906638543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4155e00 con 0x7f69e40b07b0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e9a00640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b07b0 0x7f69e415a750 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f69d800c820 tx=0x7f69d800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e1600640 1 -- 172.21.15.67:0/1906638543 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8015070 con 0x7f69e40b07b0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e1600640 1 -- 172.21.15.67:0/1906638543 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d8007a30 con 0x7f69e40b07b0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e1600640 1 -- 172.21.15.67:0/1906638543 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d8007e40 con 0x7f69e40b07b0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e41560e0 con 0x7f69e40b07b0 2024-04-17T10:28:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e4157390 con 0x7f69e40b07b0 2024-04-17T10:28:59.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e4070bb0 con 0x7f69e40b07b0 2024-04-17T10:28:59.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.242+0000 7f69e1600640 1 -- 172.21.15.67:0/1906638543 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f69d8018070 con 0x7f69e40b07b0 2024-04-17T10:28:59.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.246+0000 7f69e1600640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69c4066f60 0x7f69c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.246+0000 7f69e1600640 1 -- 172.21.15.67:0/1906638543 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f69d8085190 con 0x7f69e40b07b0 2024-04-17T10:28:59.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.246+0000 7f69e9000640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69c4066f60 0x7f69c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.246+0000 7f69e9000640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69c4066f60 0x7f69c4069420 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f69d00af000 tx=0x7f69d00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.250+0000 7f69e1600640 1 -- 172.21.15.67:0/1906638543 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f69d8055f70 con 0x7f69e40b07b0 2024-04-17T10:28:59.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.414+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69e41562b0 con 0x7f69e40b07b0 2024-04-17T10:28:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.414+0000 7f69e1600640 1 -- 172.21.15.67:0/1906638543 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16849 (secure 0 0 0) 0x7f69d8055870 con 0x7f69e40b07b0 2024-04-17T10:28:59.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:59.409 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:58.443750+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69c4066f60 msgr2=0x7f69c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69c4066f60 0x7f69c4069420 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f69d00af000 tx=0x7f69d00a7000 comp rx=0 tx=0).stop 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b07b0 msgr2=0x7f69e415a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b07b0 0x7f69e415a750 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f69d800c820 tx=0x7f69d800cce0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 shutdown_connections 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69c4066f60 0x7f69c4069420 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69e41504c0 0x7f69e4158610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69e415dd10 0x7f69e414e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 --2- 172.21.15.67:0/1906638543 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69e40b07b0 0x7f69e415a750 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 >> 172.21.15.67:0/1906638543 conn(0x7f69e40b34d0 msgr2=0x7f69e406cdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 shutdown_connections 2024-04-17T10:28:59.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.418+0000 7f69ec400640 1 -- 172.21.15.67:0/1906638543 wait complete. 2024-04-17T10:28:59.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:28:59.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48bdc00640 1 Processor -- start 2024-04-17T10:28:59.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48bdc00640 1 -- start start 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48bdc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48b806f770 0x7f48b8070470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48bdc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48b806fb50 0x7f48b815f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48bdc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b80cf330 con 0x7f48b806ffe0 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48bdc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b809b190 con 0x7f48b806fb50 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48bdc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b80709b0 con 0x7f48b806f770 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48b806fb50 0x7f48b815f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48b806fb50 0x7f48b815f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59910/0 (socket says 172.21.15.67:59910) 2024-04-17T10:28:59.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7400640 1 -- 172.21.15.67:0/1147727610 learned_addr learned my addr 172.21.15.67:0/1147727610 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:59.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7e00640 1 --2- 172.21.15.67:0/1147727610 >> v2:172.21.15.86:3300/0 conn(0x7f48b806f770 0x7f48b8070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7400640 1 -- 172.21.15.67:0/1147727610 >> v2:172.21.15.86:3300/0 conn(0x7f48b806f770 msgr2=0x7f48b8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:59.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7400640 1 --2- 172.21.15.67:0/1147727610 >> v2:172.21.15.86:3300/0 conn(0x7f48b806f770 0x7f48b8070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7400640 1 -- 172.21.15.67:0/1147727610 >> v1:172.21.15.86:6789/0 conn(0x7f48b806ffe0 legacy=0x7f48b815fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.578+0000 7f48b7400640 1 -- 172.21.15.67:0/1147727610 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b80cf610 con 0x7f48b806fb50 2024-04-17T10:28:59.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48b7400640 1 --2- 172.21.15.67:0/1147727610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b806fb50 0x7f48b815f870 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f48b80d2310 tx=0x7f48a401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9577ef65945a20f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48b6a00640 1 -- 172.21.15.67:0/1147727610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40b0070 con 0x7f48b806fb50 2024-04-17T10:28:59.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48b6a00640 1 -- 172.21.15.67:0/1147727610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a40ab600 con 0x7f48b806fb50 2024-04-17T10:28:59.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48b6a00640 1 -- 172.21.15.67:0/1147727610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a40ab9d0 con 0x7f48b806fb50 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 -- 172.21.15.67:0/1147727610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b806fb50 msgr2=0x7f48b815f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 --2- 172.21.15.67:0/1147727610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b806fb50 0x7f48b815f870 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f48b80d2310 tx=0x7f48a401c490 comp rx=0 tx=0).stop 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 -- 172.21.15.67:0/1147727610 shutdown_connections 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 --2- 172.21.15.67:0/1147727610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b806fb50 0x7f48b815f870 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 --2- 172.21.15.67:0/1147727610 >> v2:172.21.15.86:3300/0 conn(0x7f48b806f770 0x7f48b8070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 -- 172.21.15.67:0/1147727610 >> 172.21.15.67:0/1147727610 conn(0x7f48b80d1630 msgr2=0x7f48b80d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 -- 172.21.15.67:0/1147727610 shutdown_connections 2024-04-17T10:28:59.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 -- 172.21.15.67:0/1147727610 wait complete. 2024-04-17T10:28:59.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 Processor -- start 2024-04-17T10:28:59.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.582+0000 7f48bdc00640 1 -- start start 2024-04-17T10:28:59.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b806f770 0x7f48b806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b806fb50 0x7f48b8150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b81655c0 0x7f48b806d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b81606d0 con 0x7f48b81655c0 2024-04-17T10:28:59.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80715d0 con 0x7f48b806fb50 2024-04-17T10:28:59.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b80cc890 con 0x7f48b806f770 2024-04-17T10:28:59.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b81655c0 0x7f48b806d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48b7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b806fb50 0x7f48b8150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48b7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b806f770 0x7f48b806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48b7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b806fb50 0x7f48b8150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39448/0 (socket says 172.21.15.67:39448) 2024-04-17T10:28:59.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48b7400640 1 -- 172.21.15.67:0/2174442112 learned_addr learned my addr 172.21.15.67:0/2174442112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:59.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bd200640 1 -- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b806f770 msgr2=0x7f48b806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bd200640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b806f770 0x7f48b806cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bd200640 1 -- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b806fb50 msgr2=0x7f48b8150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bd200640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b806fb50 0x7f48b8150660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bd200640 1 -- 172.21.15.67:0/2174442112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b8165a70 con 0x7f48b81655c0 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48b7400640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b806fb50 0x7f48b8150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bd200640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b81655c0 0x7f48b806d420 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f48b80cbc30 tx=0x7f48b000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48afe00640 1 -- 172.21.15.67:0/2174442112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0003b80 con 0x7f48b81655c0 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48afe00640 1 -- 172.21.15.67:0/2174442112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b0003d20 con 0x7f48b81655c0 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48afe00640 1 -- 172.21.15.67:0/2174442112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b0005310 con 0x7f48b81655c0 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b8165d30 con 0x7f48b81655c0 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b81662f0 con 0x7f48b81655c0 2024-04-17T10:28:59.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.586+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b8074dc0 con 0x7f48b81655c0 2024-04-17T10:28:59.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.590+0000 7f48afe00640 1 -- 172.21.15.67:0/2174442112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f48b00054b0 con 0x7f48b81655c0 2024-04-17T10:28:59.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.590+0000 7f48afe00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4898066f60 0x7f4898069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.590+0000 7f48afe00640 1 -- 172.21.15.67:0/2174442112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f48b0084d90 con 0x7f48b81655c0 2024-04-17T10:28:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.590+0000 7f48b7e00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4898066f60 0x7f4898069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.590+0000 7f48b7e00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4898066f60 0x7f4898069420 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009b60 tx=0x7f48a8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.594+0000 7f48afe00640 1 -- 172.21.15.67:0/2174442112 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f48b0055630 con 0x7f48b81655c0 2024-04-17T10:28:59.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.758+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48b80ae820 con 0x7f48b81655c0 2024-04-17T10:28:59.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.758+0000 7f48afe00640 1 -- 172.21.15.67:0/2174442112 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16849 (secure 0 0 0) 0x7f48b00596d0 con 0x7f48b81655c0 2024-04-17T10:28:59.752 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:28:59.752 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:28:58.443750+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:28:59.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4898066f60 msgr2=0x7f4898069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4898066f60 0x7f4898069420 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009b60 tx=0x7f48a8009190 comp rx=0 tx=0).stop 2024-04-17T10:28:59.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b81655c0 msgr2=0x7f48b806d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b81655c0 0x7f48b806d420 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f48b80cbc30 tx=0x7f48b000cb90 comp rx=0 tx=0).stop 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 shutdown_connections 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4898066f60 0x7f4898069420 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b81655c0 0x7f48b806d420 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b806fb50 0x7f48b8150660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 --2- 172.21.15.67:0/2174442112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b806f770 0x7f48b806cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 >> 172.21.15.67:0/2174442112 conn(0x7f48b80d1630 msgr2=0x7f48b80b1110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 shutdown_connections 2024-04-17T10:28:59.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.762+0000 7f48bdc00640 1 -- 172.21.15.67:0/2174442112 wait complete. 2024-04-17T10:28:59.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:28:59.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a7200640 1 Processor -- start 2024-04-17T10:28:59.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a7200640 1 -- start start 2024-04-17T10:28:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a7200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff9a0159440 0x7ff9a00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a7200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a015c580 0x7ff9a0071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a7200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a015cb30 con 0x7ff9a00713e0 2024-04-17T10:28:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a7200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a00a8650 con 0x7ff9a0159440 2024-04-17T10:28:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a7200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a015ce10 con 0x7ff9a015c580 2024-04-17T10:28:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff997e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff9a00713e0 0x7ff9a015be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46398/0 (socket says 172.21.15.67:46398) 2024-04-17T10:28:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff997e00640 1 -- 172.21.15.67:0/647225146 learned_addr learned my addr 172.21.15.67:0/647225146 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:28:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a5200640 1 --2- 172.21.15.67:0/647225146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a015c580 0x7ff9a0071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff99fe00640 1 --2- 172.21.15.67:0/647225146 >> v2:172.21.15.86:3300/0 conn(0x7ff9a0159440 0x7ff9a00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff99f400640 1 -- 172.21.15.67:0/647225146 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023499888 0 0) 0x7ff9a015cb30 con 0x7ff9a00713e0 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff99f400640 1 -- 172.21.15.67:0/647225146 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff984003530 con 0x7ff9a00713e0 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff99f400640 1 -- 172.21.15.67:0/647225146 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2336950887 0 0) 0x7ff984003530 con 0x7ff9a00713e0 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a5200640 1 -- 172.21.15.67:0/647225146 >> v2:172.21.15.86:3300/0 conn(0x7ff9a0159440 msgr2=0x7ff9a00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a5200640 1 --2- 172.21.15.67:0/647225146 >> v2:172.21.15.86:3300/0 conn(0x7ff9a0159440 0x7ff9a00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a5200640 1 -- 172.21.15.67:0/647225146 >> v1:172.21.15.67:6789/0 conn(0x7ff9a00713e0 legacy=0x7ff9a015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff9a5200640 1 -- 172.21.15.67:0/647225146 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a00cd610 con 0x7ff9a015c580 2024-04-17T10:28:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.926+0000 7ff99fe00640 1 --2- 172.21.15.67:0/647225146 >> v2:172.21.15.86:3300/0 conn(0x7ff9a0159440 0x7ff9a00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:28:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a5200640 1 --2- 172.21.15.67:0/647225146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a015c580 0x7ff9a0071870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff998009d90 tx=0x7ff99801c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fef1e9627c5eb59e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff99f400640 1 -- 172.21.15.67:0/647225146 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980b2070 con 0x7ff9a015c580 2024-04-17T10:28:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff99f400640 1 -- 172.21.15.67:0/647225146 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980a55a0 con 0x7ff9a015c580 2024-04-17T10:28:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff99f400640 1 -- 172.21.15.67:0/647225146 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff988001680 con 0x7ff9a015c580 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 -- 172.21.15.67:0/647225146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a015c580 msgr2=0x7ff9a0071870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 --2- 172.21.15.67:0/647225146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a015c580 0x7ff9a0071870 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff998009d90 tx=0x7ff99801c890 comp rx=0 tx=0).stop 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 -- 172.21.15.67:0/647225146 shutdown_connections 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 --2- 172.21.15.67:0/647225146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a015c580 0x7ff9a0071870 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 --2- 172.21.15.67:0/647225146 >> v2:172.21.15.86:3300/0 conn(0x7ff9a0159440 0x7ff9a00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 -- 172.21.15.67:0/647225146 >> 172.21.15.67:0/647225146 conn(0x7ff9a00b1740 msgr2=0x7ff9a00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 -- 172.21.15.67:0/647225146 shutdown_connections 2024-04-17T10:28:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.930+0000 7ff9a7200640 1 -- 172.21.15.67:0/647225146 wait complete. 2024-04-17T10:28:59.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 Processor -- start 2024-04-17T10:28:59.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 -- start start 2024-04-17T10:28:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a00713e0 0x7ff9a00adfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a0159440 0x7ff9a015bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a015c580 0x7ff9a00b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a015d9c0 con 0x7ff9a015c580 2024-04-17T10:28:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a015d840 con 0x7ff9a0159440 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a015db40 con 0x7ff9a00713e0 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a015c580 0x7ff9a00b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a015c580 0x7ff9a00b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59926/0 (socket says 172.21.15.67:59926) 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 -- 172.21.15.67:0/3049435810 learned_addr learned my addr 172.21.15.67:0/3049435810 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff997e00640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a0159440 0x7ff9a015bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff99fe00640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a00713e0 0x7ff9a00adfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 -- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a00713e0 msgr2=0x7ff9a00adfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a00713e0 0x7ff9a00adfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 -- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a0159440 msgr2=0x7ff9a015bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a0159440 0x7ff9a015bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:28:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 -- 172.21.15.67:0/3049435810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a01432b0 con 0x7ff9a015c580 2024-04-17T10:28:59.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a5200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a015c580 0x7ff9a00b42c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff9980b0040 tx=0x7ff998019630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff99cc00640 1 -- 172.21.15.67:0/3049435810 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980b2070 con 0x7ff9a015c580 2024-04-17T10:28:59.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff99cc00640 1 -- 172.21.15.67:0/3049435810 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980bc070 con 0x7ff9a015c580 2024-04-17T10:28:59.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff99cc00640 1 -- 172.21.15.67:0/3049435810 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff998009450 con 0x7ff9a015c580 2024-04-17T10:28:59.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a00ae500 con 0x7ff9a015c580 2024-04-17T10:28:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a00ad4e0 con 0x7ff9a015c580 2024-04-17T10:28:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.934+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a004ef30 con 0x7ff9a015c580 2024-04-17T10:28:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.938+0000 7ff99cc00640 1 -- 172.21.15.67:0/3049435810 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff998004040 con 0x7ff9a015c580 2024-04-17T10:28:59.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.938+0000 7ff99cc00640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff97c066f60 0x7ff97c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:28:59.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.938+0000 7ff99cc00640 1 -- 172.21.15.67:0/3049435810 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff99811f8d0 con 0x7ff9a015c580 2024-04-17T10:28:59.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.938+0000 7ff99fe00640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff97c066f60 0x7ff97c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:28:59.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.938+0000 7ff99fe00640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff97c066f60 0x7ff97c069420 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff990009580 tx=0x7ff990009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:28:59.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:28:59.942+0000 7ff99cc00640 1 -- 172.21.15.67:0/3049435810 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff998016480 con 0x7ff9a015c580 2024-04-17T10:29:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:00.141+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff9a0151c90 con 0x7ff9a015c580 2024-04-17T10:29:01.021 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T10:29:01.025+0000 7f08d2200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:29:01.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.069+0000 7ff99cc00640 1 -- 172.21.15.67:0/3049435810 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff9980f0060 con 0x7ff9a015c580 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff97c066f60 msgr2=0x7ff97c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff97c066f60 0x7ff97c069420 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ff990009580 tx=0x7ff990009190 comp rx=0 tx=0).stop 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a015c580 msgr2=0x7ff9a00b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a015c580 0x7ff9a00b42c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff9980b0040 tx=0x7ff998019630 comp rx=0 tx=0).stop 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 shutdown_connections 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff97c066f60 0x7ff97c069420 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a015c580 0x7ff9a00b42c0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a0159440 0x7ff9a015bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 --2- 172.21.15.67:0/3049435810 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a00713e0 0x7ff9a00adfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 >> 172.21.15.67:0/3049435810 conn(0x7ff9a00b1740 msgr2=0x7ff9a00af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:01.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 shutdown_connections 2024-04-17T10:29:01.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.073+0000 7ff9a7200640 1 -- 172.21.15.67:0/3049435810 wait complete. 2024-04-17T10:29:01.089 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:29:01.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:29:01.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa900200640 1 Processor -- start 2024-04-17T10:29:01.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa900200640 1 -- start start 2024-04-17T10:29:01.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa900200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8f80cd030 0x7fa8f806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:01.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa900200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa8f806e7b0 0x7fa8f80b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:01.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa900200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f80b4990 con 0x7fa8f80aef60 2024-04-17T10:29:01.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa900200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f80af340 con 0x7fa8f80cd030 2024-04-17T10:29:01.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa900200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f806ec00 con 0x7fa8f806e7b0 2024-04-17T10:29:01.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8f80cd030 0x7fa8f806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:01.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa8f80cd030 0x7fa8f806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55654/0 (socket says 172.21.15.67:55654) 2024-04-17T10:29:01.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fce00640 1 -- 172.21.15.67:0/1200072181 learned_addr learned my addr 172.21.15.67:0/1200072181 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fce00640 1 -- 172.21.15.67:0/1200072181 >> v2:172.21.15.86:3300/0 conn(0x7fa8f806e7b0 msgr2=0x7fa8f80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fe200640 1 --2- 172.21.15.67:0/1200072181 >> v2:172.21.15.86:3300/0 conn(0x7fa8f806e7b0 0x7fa8f80b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fce00640 1 --2- 172.21.15.67:0/1200072181 >> v2:172.21.15.86:3300/0 conn(0x7fa8f806e7b0 0x7fa8f80b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fce00640 1 -- 172.21.15.67:0/1200072181 >> v1:172.21.15.67:6789/0 conn(0x7fa8f80aef60 legacy=0x7fa8f80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fce00640 1 -- 172.21.15.67:0/1200072181 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f815ae90 con 0x7fa8f80cd030 2024-04-17T10:29:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8f7e00640 1 -- 172.21.15.67:0/1200072181 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105814042 0 0) 0x7fa8ec001120 con 0x7fa8f80aef60 2024-04-17T10:29:01.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.241+0000 7fa8fe200640 1 --2- 172.21.15.67:0/1200072181 >> v2:172.21.15.86:3300/0 conn(0x7fa8f806e7b0 0x7fa8f80b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:01.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa8fce00640 1 --2- 172.21.15.67:0/1200072181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f806e270 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80b2e60 tx=0x7fa8e401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54c08c3f07a2f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:01.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa8f7e00640 1 -- 172.21.15.67:0/1200072181 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e40b2070 con 0x7fa8f80cd030 2024-04-17T10:29:01.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa8f7e00640 1 -- 172.21.15.67:0/1200072181 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e40ad680 con 0x7fa8f80cd030 2024-04-17T10:29:01.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa8f7e00640 1 -- 172.21.15.67:0/1200072181 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e40ada50 con 0x7fa8f80cd030 2024-04-17T10:29:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 -- 172.21.15.67:0/1200072181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 msgr2=0x7fa8f806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 --2- 172.21.15.67:0/1200072181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f806e270 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80b2e60 tx=0x7fa8e401c490 comp rx=0 tx=0).stop 2024-04-17T10:29:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 -- 172.21.15.67:0/1200072181 shutdown_connections 2024-04-17T10:29:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 --2- 172.21.15.67:0/1200072181 >> v2:172.21.15.86:3300/0 conn(0x7fa8f806e7b0 0x7fa8f80b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 --2- 172.21.15.67:0/1200072181 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f806e270 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 -- 172.21.15.67:0/1200072181 >> 172.21.15.67:0/1200072181 conn(0x7fa8f80b1740 msgr2=0x7fa8f80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 -- 172.21.15.67:0/1200072181 shutdown_connections 2024-04-17T10:29:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.245+0000 7fa900200640 1 -- 172.21.15.67:0/1200072181 wait complete. 2024-04-17T10:29:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 Processor -- start 2024-04-17T10:29:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 -- start start 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8f806e7b0 0x7fa8f8136990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8f80aef60 0x7fa8f8137040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f8139070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f815bc10 con 0x7fa8f80cd030 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f80b3e70 con 0x7fa8f80aef60 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f8154050 con 0x7fa8f806e7b0 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f8139070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f8139070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55662/0 (socket says 172.21.15.67:55662) 2024-04-17T10:29:01.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 -- 172.21.15.67:0/3682902828 learned_addr learned my addr 172.21.15.67:0/3682902828 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fce00640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8f80aef60 0x7fa8f8137040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fd800640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8f806e7b0 0x7fa8f8136990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 -- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8f806e7b0 msgr2=0x7fa8f8136990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8f806e7b0 0x7fa8f8136990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 -- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8f80aef60 msgr2=0x7fa8f8137040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8f80aef60 0x7fa8f8137040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 -- 172.21.15.67:0/3682902828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8139b90 con 0x7fa8f80cd030 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fd800640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8f806e7b0 0x7fa8f8136990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8fe200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f8139070 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000cef0 tx=0x7fa8f000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8f5600640 1 -- 172.21.15.67:0/3682902828 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f001a070 con 0x7fa8f80cd030 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8f5600640 1 -- 172.21.15.67:0/3682902828 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8f000d860 con 0x7fa8f80cd030 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa8f5600640 1 -- 172.21.15.67:0/3682902828 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f000dc50 con 0x7fa8f80cd030 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f8139e70 con 0x7fa8f80cd030 2024-04-17T10:29:01.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.249+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f81539d0 con 0x7fa8f80cd030 2024-04-17T10:29:01.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.253+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f804ef30 con 0x7fa8f80cd030 2024-04-17T10:29:01.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.253+0000 7fa8f5600640 1 -- 172.21.15.67:0/3682902828 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa8f001d070 con 0x7fa8f80cd030 2024-04-17T10:29:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.253+0000 7fa8f5600640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8d8066fe0 0x7fa8d80694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.253+0000 7fa8f5600640 1 -- 172.21.15.67:0/3682902828 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa8f00858a0 con 0x7fa8f80cd030 2024-04-17T10:29:01.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.253+0000 7fa8fd800640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8d8066fe0 0x7fa8d80694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:01.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.253+0000 7fa8fd800640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8d8066fe0 0x7fa8d80694a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec007fd0 tx=0x7fa8ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.257+0000 7fa8f5600640 1 -- 172.21.15.67:0/3682902828 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa8f00561b0 con 0x7fa8f80cd030 2024-04-17T10:29:01.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:01.457+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa8f80b1d30 con 0x7fa8f80cd030 2024-04-17T10:29:02.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.073+0000 7fa8f5600640 1 -- 172.21.15.67:0/3682902828 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa8f005a250 con 0x7fa8f80cd030 2024-04-17T10:29:02.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8d8066fe0 msgr2=0x7fa8d80694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8d8066fe0 0x7fa8d80694a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec007fd0 tx=0x7fa8ec009040 comp rx=0 tx=0).stop 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 msgr2=0x7fa8f8139070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f8139070 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000cef0 tx=0x7fa8f000a710 comp rx=0 tx=0).stop 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 shutdown_connections 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8d8066fe0 0x7fa8d80694a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa8f80cd030 0x7fa8f8139070 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa8f80aef60 0x7fa8f8137040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 --2- 172.21.15.67:0/3682902828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa8f806e7b0 0x7fa8f8136990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 >> 172.21.15.67:0/3682902828 conn(0x7fa8f80b1740 msgr2=0x7fa8f806de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 shutdown_connections 2024-04-17T10:29:02.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.077+0000 7fa900200640 1 -- 172.21.15.67:0/3682902828 wait complete. 2024-04-17T10:29:02.094 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:29:02.094 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:02.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a97200640 1 Processor -- start 2024-04-17T10:29:02.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a97200640 1 -- start start 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a97200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a98099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a97200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a98099f90 0x7f4a98161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a97200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980d0760 con 0x7f4a980b2570 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a97200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9809a400 con 0x7f4a98099f90 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a97200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980cfc70 con 0x7f4a98161160 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a96800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4a98099f90 0x7f4a98161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a95400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4a980b2570 0x7f4a981616c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42732/0 (socket says 172.21.15.67:42732) 2024-04-17T10:29:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a95400640 1 -- 172.21.15.67:0/2212552610 learned_addr learned my addr 172.21.15.67:0/2212552610 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:02.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a95e00640 1 --2- 172.21.15.67:0/2212552610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a98099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a96800640 1 -- 172.21.15.67:0/2212552610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 msgr2=0x7f4a98099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:02.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a96800640 1 --2- 172.21.15.67:0/2212552610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a98099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a96800640 1 -- 172.21.15.67:0/2212552610 >> v1:172.21.15.67:6789/0 conn(0x7f4a980b2570 legacy=0x7f4a981616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a96800640 1 -- 172.21.15.67:0/2212552610 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980cc2a0 con 0x7f4a98099f90 2024-04-17T10:29:02.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.249+0000 7f4a95e00640 1 --2- 172.21.15.67:0/2212552610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a98099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:02.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a96800640 1 --2- 172.21.15.67:0/2212552610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a98161dd0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009d90 tx=0x7f4a9001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce9eef630e74400c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a94a00640 1 -- 172.21.15.67:0/2212552610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900b2070 con 0x7f4a98099f90 2024-04-17T10:29:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a94a00640 1 -- 172.21.15.67:0/2212552610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a900a5940 con 0x7f4a98099f90 2024-04-17T10:29:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a94a00640 1 -- 172.21.15.67:0/2212552610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a900a5d10 con 0x7f4a98099f90 2024-04-17T10:29:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 -- 172.21.15.67:0/2212552610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 msgr2=0x7f4a98161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 --2- 172.21.15.67:0/2212552610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a98161dd0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f4a90009d90 tx=0x7f4a9001c970 comp rx=0 tx=0).stop 2024-04-17T10:29:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 -- 172.21.15.67:0/2212552610 shutdown_connections 2024-04-17T10:29:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 --2- 172.21.15.67:0/2212552610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a98161dd0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 --2- 172.21.15.67:0/2212552610 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a98099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 -- 172.21.15.67:0/2212552610 >> 172.21.15.67:0/2212552610 conn(0x7f4a980d1690 msgr2=0x7f4a980d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 -- 172.21.15.67:0/2212552610 shutdown_connections 2024-04-17T10:29:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 -- 172.21.15.67:0/2212552610 wait complete. 2024-04-17T10:29:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 Processor -- start 2024-04-17T10:29:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.253+0000 7f4a97200640 1 -- start start 2024-04-17T10:29:02.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a9814e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a980b2570 0x7f4a9814e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a980b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980b2bc0 con 0x7f4a98099f90 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a980cc5d0 con 0x7f4a980b2570 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9806bda0 con 0x7f4a98161160 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a9814e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a9814e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55686/0 (socket says 172.21.15.67:55686) 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 -- 172.21.15.67:0/2569375021 learned_addr learned my addr 172.21.15.67:0/2569375021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a96800640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a980b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95400640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a980b2570 0x7f4a9814e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 -- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 msgr2=0x7f4a980b5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a980b5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 -- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a980b2570 msgr2=0x7f4a9814e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a980b2570 0x7f4a9814e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 -- 172.21.15.67:0/2569375021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9814ee20 con 0x7f4a98099f90 2024-04-17T10:29:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a95e00640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a9814e1c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c7c0 tx=0x7f4a8800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a8e000640 1 -- 172.21.15.67:0/2569375021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88007a30 con 0x7f4a98099f90 2024-04-17T10:29:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a8e000640 1 -- 172.21.15.67:0/2569375021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88007bd0 con 0x7f4a98099f90 2024-04-17T10:29:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a8e000640 1 -- 172.21.15.67:0/2569375021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a880113f0 con 0x7f4a98099f90 2024-04-17T10:29:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9814f0c0 con 0x7f4a98099f90 2024-04-17T10:29:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9814f680 con 0x7f4a98099f90 2024-04-17T10:29:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.257+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a98074dc0 con 0x7f4a98099f90 2024-04-17T10:29:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.261+0000 7f4a8e000640 1 -- 172.21.15.67:0/2569375021 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4a88011590 con 0x7f4a98099f90 2024-04-17T10:29:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.261+0000 7f4a8e000640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4a78066f60 0x7f4a78069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.261+0000 7f4a8e000640 1 -- 172.21.15.67:0/2569375021 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4a88085310 con 0x7f4a98099f90 2024-04-17T10:29:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.261+0000 7f4a95400640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4a78066f60 0x7f4a78069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.261+0000 7f4a95400640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4a78066f60 0x7f4a78069420 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4a981689c0 tx=0x7f4a84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.265+0000 7f4a8e000640 1 -- 172.21.15.67:0/2569375021 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4a88015020 con 0x7f4a98099f90 2024-04-17T10:29:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a9809ac60 con 0x7f4a98099f90 2024-04-17T10:29:02.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a8e000640 1 -- 172.21.15.67:0/2569375021 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4936 (secure 0 0 0) 0x7f4a88056220 con 0x7f4a98099f90 2024-04-17T10:29:02.455 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:02.455 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1277872832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1277872832},{"type":"v1","addr":"172.21.15.86:6836","nonce":1277872832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5900,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3787337522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3787337522},{"type":"v1","addr":"172.21.15.86:6837","nonce":3787337522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3103235412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3103235412},{"type":"v1","addr":"172.21.15.67:6834","nonce":3103235412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:01.462609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6651},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6651":{"gid":6651,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/392417778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":392417778},{"type":"v1","addr":"172.21.15.67:6835","nonce":392417778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:02.456 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 40 2024-04-17T10:29:02.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4a78066f60 msgr2=0x7f4a78069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4a78066f60 0x7f4a78069420 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4a981689c0 tx=0x7f4a84009040 comp rx=0 tx=0).stop 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 msgr2=0x7f4a9814e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a9814e1c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800c7c0 tx=0x7f4a8800cc80 comp rx=0 tx=0).stop 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 shutdown_connections 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4a78066f60 0x7f4a78069420 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4a98161160 0x7f4a980b5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4a980b2570 0x7f4a9814e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 --2- 172.21.15.67:0/2569375021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a98099f90 0x7f4a9814e1c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 >> 172.21.15.67:0/2569375021 conn(0x7f4a980d1690 msgr2=0x7f4a9806d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 shutdown_connections 2024-04-17T10:29:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.461+0000 7f4a97200640 1 -- 172.21.15.67:0/2569375021 wait complete. 2024-04-17T10:29:02.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5eb200640 1 Processor -- start 2024-04-17T10:29:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5eb200640 1 -- start start 2024-04-17T10:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5eb200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5e41446c0 0x7fb5e4144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5eb200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e4071840 0x7fb5e40b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5eb200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e41456d0 con 0x7fb5e4144aa0 2024-04-17T10:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5eb200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e40b51a0 con 0x7fb5e41446c0 2024-04-17T10:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5eb200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40b4870 con 0x7fb5e4071840 2024-04-17T10:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5e41446c0 0x7fb5e4144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb5e4144aa0 0x7fb5e40d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42746/0 (socket says 172.21.15.67:42746) 2024-04-17T10:29:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e3400640 1 -- 172.21.15.67:0/1467301708 learned_addr learned my addr 172.21.15.67:0/1467301708 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e9200640 1 --2- 172.21.15.67:0/1467301708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e4071840 0x7fb5e40b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e3e00640 1 -- 172.21.15.67:0/1467301708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e4071840 msgr2=0x7fb5e40b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e3e00640 1 --2- 172.21.15.67:0/1467301708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e4071840 0x7fb5e40b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e3e00640 1 -- 172.21.15.67:0/1467301708 >> v1:172.21.15.67:6789/0 conn(0x7fb5e4144aa0 legacy=0x7fb5e40d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e2a00640 1 -- 172.21.15.67:0/1467301708 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190710219 0 0) 0x7fb5e41456d0 con 0x7fb5e4144aa0 2024-04-17T10:29:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e3e00640 1 -- 172.21.15.67:0/1467301708 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e40b4b50 con 0x7fb5e41446c0 2024-04-17T10:29:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.625+0000 7fb5e9200640 1 --2- 172.21.15.67:0/1467301708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e4071840 0x7fb5e40b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5e3e00640 1 --2- 172.21.15.67:0/1467301708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e41446c0 0x7fb5e4144ea0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40097c0 tx=0x7fb5d401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a19edc3f0420b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5e2a00640 1 -- 172.21.15.67:0/1467301708 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40b2070 con 0x7fb5e41446c0 2024-04-17T10:29:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5e2a00640 1 -- 172.21.15.67:0/1467301708 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d40ad680 con 0x7fb5e41446c0 2024-04-17T10:29:02.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5e2a00640 1 -- 172.21.15.67:0/1467301708 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d40ada50 con 0x7fb5e41446c0 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 -- 172.21.15.67:0/1467301708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e41446c0 msgr2=0x7fb5e4144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1467301708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e41446c0 0x7fb5e4144ea0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40097c0 tx=0x7fb5d401c470 comp rx=0 tx=0).stop 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 -- 172.21.15.67:0/1467301708 shutdown_connections 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1467301708 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e4071840 0x7fb5e40b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1467301708 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e41446c0 0x7fb5e4144ea0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 -- 172.21.15.67:0/1467301708 >> 172.21.15.67:0/1467301708 conn(0x7fb5e40b34d0 msgr2=0x7fb5e40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 -- 172.21.15.67:0/1467301708 shutdown_connections 2024-04-17T10:29:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 -- 172.21.15.67:0/1467301708 wait complete. 2024-04-17T10:29:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 Processor -- start 2024-04-17T10:29:02.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.629+0000 7fb5eb200640 1 -- start start 2024-04-17T10:29:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e4071840 0x7fb5e40a8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e41446c0 0x7fb5e415a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e4144aa0 0x7fb5e41594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40b6500 con 0x7fb5e4144aa0 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40b6380 con 0x7fb5e4071840 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e40b6680 con 0x7fb5e41446c0 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e4144aa0 0x7fb5e41594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e4144aa0 0x7fb5e41594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55706/0 (socket says 172.21.15.67:55706) 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 -- 172.21.15.67:0/1210851359 learned_addr learned my addr 172.21.15.67:0/1210851359 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e3400640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e41446c0 0x7fb5e415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e3e00640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e4071840 0x7fb5e40a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 -- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e41446c0 msgr2=0x7fb5e415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e41446c0 0x7fb5e415a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 -- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e4071840 msgr2=0x7fb5e40a8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e4071840 0x7fb5e40a8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 -- 172.21.15.67:0/1210851359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e4159a10 con 0x7fb5e4144aa0 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5e9200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e4144aa0 0x7fb5e41594d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00cef0 tx=0x7fb5dc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5dbe00640 1 -- 172.21.15.67:0/1210851359 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc01a070 con 0x7fb5e4144aa0 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5dbe00640 1 -- 172.21.15.67:0/1210851359 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5dc00d860 con 0x7fb5e4144aa0 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5dbe00640 1 -- 172.21.15.67:0/1210851359 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc00dc50 con 0x7fb5e4144aa0 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e415acd0 con 0x7fb5e4144aa0 2024-04-17T10:29:02.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e415b290 con 0x7fb5e4144aa0 2024-04-17T10:29:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.633+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e4070bb0 con 0x7fb5e4144aa0 2024-04-17T10:29:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.637+0000 7fb5dbe00640 1 -- 172.21.15.67:0/1210851359 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb5dc01d070 con 0x7fb5e4144aa0 2024-04-17T10:29:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.637+0000 7fb5dbe00640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb5c4066f60 0x7fb5c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.637+0000 7fb5dbe00640 1 -- 172.21.15.67:0/1210851359 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb5dc085a20 con 0x7fb5e4144aa0 2024-04-17T10:29:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.637+0000 7fb5e3e00640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb5c4066f60 0x7fb5c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.637+0000 7fb5e3e00640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb5c4066f60 0x7fb5c4069420 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40af000 tx=0x7fb5d40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.641+0000 7fb5dbe00640 1 -- 172.21.15.67:0/1210851359 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb5dc056270 con 0x7fb5e4144aa0 2024-04-17T10:29:02.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.805+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5e4074e30 con 0x7fb5e4144aa0 2024-04-17T10:29:02.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.805+0000 7fb5dbe00640 1 -- 172.21.15.67:0/1210851359 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16836 (secure 0 0 0) 0x7fb5dc05a310 con 0x7fb5e4144aa0 2024-04-17T10:29:02.801 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:02.802 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:00.041735+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb5c4066f60 msgr2=0x7fb5c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb5c4066f60 0x7fb5c4069420 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb5d40af000 tx=0x7fb5d40a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e4144aa0 msgr2=0x7fb5e41594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e4144aa0 0x7fb5e41594d0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00cef0 tx=0x7fb5dc00a710 comp rx=0 tx=0).stop 2024-04-17T10:29:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 shutdown_connections 2024-04-17T10:29:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb5c4066f60 0x7fb5c4069420 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e4144aa0 0x7fb5e41594d0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5e41446c0 0x7fb5e415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 --2- 172.21.15.67:0/1210851359 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5e4071840 0x7fb5e40a8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 >> 172.21.15.67:0/1210851359 conn(0x7fb5e40b34d0 msgr2=0x7fb5e40b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:02.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 shutdown_connections 2024-04-17T10:29:02.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.809+0000 7fb5eb200640 1 -- 172.21.15.67:0/1210851359 wait complete. 2024-04-17T10:29:02.822 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:29:02.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:02.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 Processor -- start 2024-04-17T10:29:02.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 -- start start 2024-04-17T10:29:02.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c1597a0 0x7f6a6c0cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6a6c06b6a0 0x7f6a6c0cd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a6c0cf880 0x7f6a6c0cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c0d0230 con 0x7f6a6c0cf880 2024-04-17T10:29:02.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c0cd4f0 con 0x7f6a6c06b6a0 2024-04-17T10:29:02.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a72000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0cc8b0 con 0x7f6a6c1597a0 2024-04-17T10:29:02.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a6c0cf880 0x7f6a6c0cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a6c0cf880 0x7f6a6c0cfcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55718/0 (socket says 172.21.15.67:55718) 2024-04-17T10:29:02.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 -- 172.21.15.67:0/3686825029 learned_addr learned my addr 172.21.15.67:0/3686825029 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6b400640 1 --2- 172.21.15.67:0/3686825029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c1597a0 0x7f6a6c0cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a62a00640 1 --2- 172.21.15.67:0/3686825029 >> v2:172.21.15.86:3300/0 conn(0x7f6a6c06b6a0 0x7f6a6c0cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 -- 172.21.15.67:0/3686825029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c1597a0 msgr2=0x7f6a6c0cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 --2- 172.21.15.67:0/3686825029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c1597a0 0x7f6a6c0cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 -- 172.21.15.67:0/3686825029 >> v2:172.21.15.86:3300/0 conn(0x7f6a6c06b6a0 msgr2=0x7f6a6c0cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 --2- 172.21.15.67:0/3686825029 >> v2:172.21.15.86:3300/0 conn(0x7f6a6c06b6a0 0x7f6a6c0cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6be00640 1 -- 172.21.15.67:0/3686825029 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c0ccb90 con 0x7f6a6c0cf880 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a6b400640 1 --2- 172.21.15.67:0/3686825029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c1597a0 0x7f6a6c0cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.969+0000 7f6a62a00640 1 --2- 172.21.15.67:0/3686825029 >> v2:172.21.15.86:3300/0 conn(0x7f6a6c06b6a0 0x7f6a6c0cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a6be00640 1 --2- 172.21.15.67:0/3686825029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0cf880 0x7f6a6c0cfcf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c06cd80 tx=0x7f6a6401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b8fcdfd2e176ac server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a6aa00640 1 -- 172.21.15.67:0/3686825029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640a5800 con 0x7f6a6c0cf880 2024-04-17T10:29:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a6aa00640 1 -- 172.21.15.67:0/3686825029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a640a59a0 con 0x7f6a6c0cf880 2024-04-17T10:29:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a6aa00640 1 -- 172.21.15.67:0/3686825029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640a5d70 con 0x7f6a6c0cf880 2024-04-17T10:29:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 -- 172.21.15.67:0/3686825029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0cf880 msgr2=0x7f6a6c0cfcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 --2- 172.21.15.67:0/3686825029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0cf880 0x7f6a6c0cfcf0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c06cd80 tx=0x7f6a6401c9c0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 -- 172.21.15.67:0/3686825029 shutdown_connections 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 --2- 172.21.15.67:0/3686825029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c0cf880 0x7f6a6c0cfcf0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 --2- 172.21.15.67:0/3686825029 >> v2:172.21.15.86:3300/0 conn(0x7f6a6c06b6a0 0x7f6a6c0cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 --2- 172.21.15.67:0/3686825029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c1597a0 0x7f6a6c0cc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 -- 172.21.15.67:0/3686825029 >> 172.21.15.67:0/3686825029 conn(0x7f6a6c0b22a0 msgr2=0x7f6a6c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 -- 172.21.15.67:0/3686825029 shutdown_connections 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.973+0000 7f6a72000640 1 -- 172.21.15.67:0/3686825029 wait complete. 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 Processor -- start 2024-04-17T10:29:02.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 -- start start 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c06b6a0 0x7f6a6c158ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c0cf880 0x7f6a6c151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c1597a0 0x7f6a6c156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0b3020 con 0x7f6a6c1597a0 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0b2ea0 con 0x7f6a6c06b6a0 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c15c800 con 0x7f6a6c0cf880 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c1597a0 0x7f6a6c156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c1597a0 0x7f6a6c156bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55732/0 (socket says 172.21.15.67:55732) 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 -- 172.21.15.67:0/1984422990 learned_addr learned my addr 172.21.15.67:0/1984422990 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a62a00640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c0cf880 0x7f6a6c151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6b400640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c06b6a0 0x7f6a6c158ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 -- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c0cf880 msgr2=0x7f6a6c151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c0cf880 0x7f6a6c151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 -- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c06b6a0 msgr2=0x7f6a6c158ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c06b6a0 0x7f6a6c158ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 -- 172.21.15.67:0/1984422990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c157960 con 0x7f6a6c1597a0 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6b400640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c06b6a0 0x7f6a6c158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a6be00640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c1597a0 0x7f6a6c156bf0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6a640af040 tx=0x7f6a64019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a63e00640 1 -- 172.21.15.67:0/1984422990 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640b8670 con 0x7f6a6c1597a0 2024-04-17T10:29:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a63e00640 1 -- 172.21.15.67:0/1984422990 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a640be070 con 0x7f6a6c1597a0 2024-04-17T10:29:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a63e00640 1 -- 172.21.15.67:0/1984422990 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a640b8a90 con 0x7f6a6c1597a0 2024-04-17T10:29:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c158110 con 0x7f6a6c1597a0 2024-04-17T10:29:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.977+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c1586d0 con 0x7f6a6c1597a0 2024-04-17T10:29:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.981+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c09e900 con 0x7f6a6c1597a0 2024-04-17T10:29:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.981+0000 7f6a63e00640 1 -- 172.21.15.67:0/1984422990 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6a640b8c30 con 0x7f6a6c1597a0 2024-04-17T10:29:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.981+0000 7f6a63e00640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a48066f60 0x7f6a48069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.981+0000 7f6a63e00640 1 -- 172.21.15.67:0/1984422990 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6a64120520 con 0x7f6a6c1597a0 2024-04-17T10:29:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.981+0000 7f6a6b400640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a48066f60 0x7f6a48069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.981+0000 7f6a6b400640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a48066f60 0x7f6a48069420 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00a9b0 tx=0x7f6a5c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:02.985+0000 7f6a63e00640 1 -- 172.21.15.67:0/1984422990 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6a640165e0 con 0x7f6a6c1597a0 2024-04-17T10:29:03.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.149+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6a6c0ce910 con 0x7f6a48066f60 2024-04-17T10:29:03.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.153+0000 7f6a63e00640 1 -- 172.21.15.67:0/1984422990 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239920 (secure 0 0 0) 0x7f6a6c0ce910 con 0x7f6a48066f60 2024-04-17T10:29:03.152 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:03.154 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a48066f60 msgr2=0x7f6a48069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a48066f60 0x7f6a48069420 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6a5c00a9b0 tx=0x7f6a5c008040 comp rx=0 tx=0).stop 2024-04-17T10:29:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c1597a0 msgr2=0x7f6a6c156bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c1597a0 0x7f6a6c156bf0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f6a640af040 tx=0x7f6a64019760 comp rx=0 tx=0).stop 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 shutdown_connections 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6a48066f60 0x7f6a48069420 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6a6c1597a0 0x7f6a6c156bf0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6a6c0cf880 0x7f6a6c151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 --2- 172.21.15.67:0/1984422990 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6a6c06b6a0 0x7f6a6c158ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 >> 172.21.15.67:0/1984422990 conn(0x7f6a6c0b22a0 msgr2=0x7f6a6c15b310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 shutdown_connections 2024-04-17T10:29:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.157+0000 7f6a72000640 1 -- 172.21.15.67:0/1984422990 wait complete. 2024-04-17T10:29:03.172 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-04-17T10:29:02.191603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":316236,"kb_used_data":7112,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754658484,"statfs":{"total":773094113280,"available":772770287616,"internally_reserved":0,"allocated":7282688,"data_stored":3936368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.160535"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039864+0000","last_change":"2024-04-17T10:28:57.039864+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039864+0000","last_undegraded":"2024-04-17T10:28:57.039864+0000","last_fullsized":"2024-04-17T10:28:57.039864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044352+0000","last_change":"2024-04-17T10:28:57.044352+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044352+0000","last_undegraded":"2024-04-17T10:28:57.044352+0000","last_fullsized":"2024-04-17T10:28:57.044352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044169+0000","last_change":"2024-04-17T10:28:57.044169+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044169+0000","last_undegraded":"2024-04-17T10:28:57.044169+0000","last_fullsized":"2024-04-17T10:28:57.044169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044080+0000","last_change":"2024-04-17T10:28:57.044080+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044080+0000","last_undegraded":"2024-04-17T10:28:57.044080+0000","last_fullsized":"2024-04-17T10:28:57.044080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040312+0000","last_change":"2024-04-17T10:28:57.040312+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040312+0000","last_undegraded":"2024-04-17T10:28:57.040312+0000","last_fullsized":"2024-04-17T10:28:57.040312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.041387+0000","last_change":"2024-04-17T10:28:57.041387+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.041387+0000","last_undegraded":"2024-04-17T10:28:57.041387+0000","last_fullsized":"2024-04-17T10:28:57.041387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040850+0000","last_change":"2024-04-17T10:28:57.040850+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040850+0000","last_undegraded":"2024-04-17T10:28:57.040850+0000","last_fullsized":"2024-04-17T10:28:57.040850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039600+0000","last_change":"2024-04-17T10:28:57.039600+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039600+0000","last_undegraded":"2024-04-17T10:28:57.039600+0000","last_fullsized":"2024-04-17T10:28:57.039600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038703+0000","last_change":"2024-04-17T10:28:57.038703+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038703+0000","last_undegraded":"2024-04-17T10:28:57.038703+0000","last_fullsized":"2024-04-17T10:28:57.038703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.035308+0000","last_change":"2024-04-17T10:28:57.035308+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.035308+0000","last_undegraded":"2024-04-17T10:28:57.035308+0000","last_fullsized":"2024-04-17T10:28:57.035308+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039802+0000","last_change":"2024-04-17T10:28:57.039802+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039802+0000","last_undegraded":"2024-04-17T10:28:57.039802+0000","last_fullsized":"2024-04-17T10:28:57.039802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040341+0000","last_change":"2024-04-17T10:28:57.040341+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040341+0000","last_undegraded":"2024-04-17T10:28:57.040341+0000","last_fullsized":"2024-04-17T10:28:57.040341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040059+0000","last_change":"2024-04-17T10:28:57.040059+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040059+0000","last_undegraded":"2024-04-17T10:28:57.040059+0000","last_fullsized":"2024-04-17T10:28:57.040059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043675+0000","last_change":"2024-04-17T10:28:57.043675+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043675+0000","last_undegraded":"2024-04-17T10:28:57.043675+0000","last_fullsized":"2024-04-17T10:28:57.043675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.037846+0000","last_change":"2024-04-17T10:28:57.037846+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.037846+0000","last_undegraded":"2024-04-17T10:28:57.037846+0000","last_fullsized":"2024-04-17T10:28:57.037846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.035420+0000","last_change":"2024-04-17T10:28:57.035420+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.035420+0000","last_undegraded":"2024-04-17T10:28:57.035420+0000","last_fullsized":"2024-04-17T10:28:57.035420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043743+0000","last_change":"2024-04-17T10:28:57.043743+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043743+0000","last_undegraded":"2024-04-17T10:28:57.043743+0000","last_fullsized":"2024-04-17T10:28:57.043743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038266+0000","last_change":"2024-04-17T10:28:57.038266+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038266+0000","last_undegraded":"2024-04-17T10:28:57.038266+0000","last_fullsized":"2024-04-17T10:28:57.038266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038033+0000","last_change":"2024-04-17T10:28:57.038033+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038033+0000","last_undegraded":"2024-04-17T10:28:57.038033+0000","last_fullsized":"2024-04-17T10:28:57.038033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040730+0000","last_change":"2024-04-17T10:28:57.040730+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040730+0000","last_undegraded":"2024-04-17T10:28:57.040730+0000","last_fullsized":"2024-04-17T10:28:57.040730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043346+0000","last_change":"2024-04-17T10:28:57.043346+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043346+0000","last_undegraded":"2024-04-17T10:28:57.043346+0000","last_fullsized":"2024-04-17T10:28:57.043346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043128+0000","last_change":"2024-04-17T10:28:57.043128+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043128+0000","last_undegraded":"2024-04-17T10:28:57.043128+0000","last_fullsized":"2024-04-17T10:28:57.043128+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043252+0000","last_change":"2024-04-17T10:28:57.043252+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043252+0000","last_undegraded":"2024-04-17T10:28:57.043252+0000","last_fullsized":"2024-04-17T10:28:57.043252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043155+0000","last_change":"2024-04-17T10:28:57.043155+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043155+0000","last_undegraded":"2024-04-17T10:28:57.043155+0000","last_fullsized":"2024-04-17T10:28:57.043155+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043132+0000","last_change":"2024-04-17T10:28:57.043132+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043132+0000","last_undegraded":"2024-04-17T10:28:57.043132+0000","last_fullsized":"2024-04-17T10:28:57.043132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043420+0000","last_change":"2024-04-17T10:28:57.043420+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043420+0000","last_undegraded":"2024-04-17T10:28:57.043420+0000","last_fullsized":"2024-04-17T10:28:57.043420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039818+0000","last_change":"2024-04-17T10:28:57.039818+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039818+0000","last_undegraded":"2024-04-17T10:28:57.039818+0000","last_fullsized":"2024-04-17T10:28:57.039818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039061+0000","last_change":"2024-04-17T10:28:57.039061+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039061+0000","last_undegraded":"2024-04-17T10:28:57.039061+0000","last_fullsized":"2024-04-17T10:28:57.039061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040059+0000","last_change":"2024-04-17T10:28:57.040059+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040059+0000","last_undegraded":"2024-04-17T10:28:57.040059+0000","last_fullsized":"2024-04-17T10:28:57.040059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040010+0000","last_change":"2024-04-17T10:28:57.040010+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040010+0000","last_undegraded":"2024-04-17T10:28:57.040010+0000","last_fullsized":"2024-04-17T10:28:57.040010+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039826+0000","last_change":"2024-04-17T10:28:57.039826+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039826+0000","last_undegraded":"2024-04-17T10:28:57.039826+0000","last_fullsized":"2024-04-17T10:28:57.039826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040004+0000","last_change":"2024-04-17T10:28:57.040004+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040004+0000","last_undegraded":"2024-04-17T10:28:57.040004+0000","last_fullsized":"2024-04-17T10:28:57.040004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":253,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-17T10:28:57.040437+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:57.040437+0000","last_peered":"2024-04-17T10:28:57.040437+0000","last_clean":"2024-04-17T10:28:57.040437+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:57.040437+0000","last_undegraded":"2024-04-17T10:28:57.040437+0000","last_fullsized":"2024-04-17T10:28:57.040437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":1355776,"data_stored":934727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36088,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335752,"statfs":{"total":96636764160,"available":96599810048,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40232,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331608,"statfs":{"total":96636764160,"available":96595566592,"internally_reserved":0,"allocated":749568,"data_stored":338910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36084,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335756,"statfs":{"total":96636764160,"available":96599814144,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58888,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312952,"statfs":{"total":96636764160,"available":96576462848,"internally_reserved":0,"allocated":1368064,"data_stored":940936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:03.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:03.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc544a00640 1 Processor -- start 2024-04-17T10:29:03.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc544a00640 1 -- start start 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc544a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc54006f770 0x7fc540070470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc544a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc54006ffe0 0x7fc54015ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc544a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5400cf330 con 0x7fc54006fb50 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc544a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54009b190 con 0x7fc54006f770 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc544a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5400709b0 con 0x7fc54006ffe0 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc54006f770 0x7fc540070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc54006fb50 0x7fc54015f870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42748/0 (socket says 172.21.15.67:42748) 2024-04-17T10:29:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53ea00640 1 -- 172.21.15.67:0/2316788040 learned_addr learned my addr 172.21.15.67:0/2316788040 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53fe00640 1 --2- 172.21.15.67:0/2316788040 >> v2:172.21.15.86:3300/0 conn(0x7fc54006ffe0 0x7fc54015ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53f400640 1 -- 172.21.15.67:0/2316788040 >> v2:172.21.15.86:3300/0 conn(0x7fc54006ffe0 msgr2=0x7fc54015ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53e000640 1 -- 172.21.15.67:0/2316788040 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192230022 0 0) 0x7fc5400cf330 con 0x7fc54006fb50 2024-04-17T10:29:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53f400640 1 --2- 172.21.15.67:0/2316788040 >> v2:172.21.15.86:3300/0 conn(0x7fc54006ffe0 0x7fc54015ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53f400640 1 -- 172.21.15.67:0/2316788040 >> v1:172.21.15.67:6789/0 conn(0x7fc54006fb50 legacy=0x7fc54015f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53f400640 1 -- 172.21.15.67:0/2316788040 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5400cf610 con 0x7fc54006f770 2024-04-17T10:29:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.361+0000 7fc53fe00640 1 --2- 172.21.15.67:0/2316788040 >> v2:172.21.15.86:3300/0 conn(0x7fc54006ffe0 0x7fc54015ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:03.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc53f400640 1 --2- 172.21.15.67:0/2316788040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540070470 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc5400cd540 tx=0x7fc53001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75dcacebfe48815c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc53e000640 1 -- 172.21.15.67:0/2316788040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5300b0070 con 0x7fc54006f770 2024-04-17T10:29:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc53e000640 1 -- 172.21.15.67:0/2316788040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5300ab600 con 0x7fc54006f770 2024-04-17T10:29:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc53e000640 1 -- 172.21.15.67:0/2316788040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5300ab9d0 con 0x7fc54006f770 2024-04-17T10:29:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 -- 172.21.15.67:0/2316788040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 msgr2=0x7fc540070470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 --2- 172.21.15.67:0/2316788040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540070470 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc5400cd540 tx=0x7fc53001c3f0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 -- 172.21.15.67:0/2316788040 shutdown_connections 2024-04-17T10:29:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 --2- 172.21.15.67:0/2316788040 >> v2:172.21.15.86:3300/0 conn(0x7fc54006ffe0 0x7fc54015ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 --2- 172.21.15.67:0/2316788040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540070470 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 -- 172.21.15.67:0/2316788040 >> 172.21.15.67:0/2316788040 conn(0x7fc5400d1630 msgr2=0x7fc5400d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 -- 172.21.15.67:0/2316788040 shutdown_connections 2024-04-17T10:29:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 -- 172.21.15.67:0/2316788040 wait complete. 2024-04-17T10:29:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 Processor -- start 2024-04-17T10:29:03.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.365+0000 7fc544a00640 1 -- start start 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc54006fb50 0x7fc5400b1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc540150f30 0x7fc5400c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5401606d0 con 0x7fc54006f770 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5400715d0 con 0x7fc54006fb50 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5400cc890 con 0x7fc540150f30 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55758/0 (socket says 172.21.15.67:55758) 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 -- 172.21.15.67:0/2357984933 learned_addr learned my addr 172.21.15.67:0/2357984933 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 -- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc540150f30 msgr2=0x7fc5400c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc540150f30 0x7fc5400c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 -- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc54006fb50 msgr2=0x7fc5400b1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc54006fb50 0x7fc5400b1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 -- 172.21.15.67:0/2357984933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540151430 con 0x7fc54006f770 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc53f400640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540154320 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc530019000 tx=0x7fc530019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc537400640 1 -- 172.21.15.67:0/2357984933 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5300b0040 con 0x7fc54006f770 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc537400640 1 -- 172.21.15.67:0/2357984933 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5300b9070 con 0x7fc54006f770 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc540151710 con 0x7fc54006f770 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc537400640 1 -- 172.21.15.67:0/2357984933 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5300b4810 con 0x7fc54006f770 2024-04-17T10:29:03.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc540151cd0 con 0x7fc54006f770 2024-04-17T10:29:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc540074dc0 con 0x7fc54006f770 2024-04-17T10:29:03.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.369+0000 7fc537400640 1 -- 172.21.15.67:0/2357984933 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc5300ab4a0 con 0x7fc54006f770 2024-04-17T10:29:03.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.373+0000 7fc537400640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc520066f60 0x7fc520069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.373+0000 7fc537400640 1 -- 172.21.15.67:0/2357984933 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc53011f140 con 0x7fc54006f770 2024-04-17T10:29:03.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.373+0000 7fc53ea00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc520066f60 0x7fc520069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.373+0000 7fc53ea00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc520066f60 0x7fc520069420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0011f0 tx=0x7fc52c007930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:03.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.377+0000 7fc537400640 1 -- 172.21.15.67:0/2357984933 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc530016070 con 0x7fc54006f770 2024-04-17T10:29:03.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.533+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc5400b05a0 con 0x7fc520066f60 2024-04-17T10:29:03.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.541+0000 7fc537400640 1 -- 172.21.15.67:0/2357984933 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239920 (secure 0 0 0) 0x7fc5400b05a0 con 0x7fc520066f60 2024-04-17T10:29:03.536 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:03.538 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc520066f60 msgr2=0x7fc520069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc520066f60 0x7fc520069420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0011f0 tx=0x7fc52c007930 comp rx=0 tx=0).stop 2024-04-17T10:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 msgr2=0x7fc540154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540154320 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc530019000 tx=0x7fc530019190 comp rx=0 tx=0).stop 2024-04-17T10:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 shutdown_connections 2024-04-17T10:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc520066f60 0x7fc520069420 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc540150f30 0x7fc5400c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc54006fb50 0x7fc5400b1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 --2- 172.21.15.67:0/2357984933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc54006f770 0x7fc540154320 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 >> 172.21.15.67:0/2357984933 conn(0x7fc5400d1630 msgr2=0x7fc540155250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 shutdown_connections 2024-04-17T10:29:03.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.545+0000 7fc544a00640 1 -- 172.21.15.67:0/2357984933 wait complete. 2024-04-17T10:29:03.557 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-04-17T10:29:02.191603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":316236,"kb_used_data":7112,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754658484,"statfs":{"total":773094113280,"available":772770287616,"internally_reserved":0,"allocated":7282688,"data_stored":3936368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.160535"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039864+0000","last_change":"2024-04-17T10:28:57.039864+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039864+0000","last_undegraded":"2024-04-17T10:28:57.039864+0000","last_fullsized":"2024-04-17T10:28:57.039864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044352+0000","last_change":"2024-04-17T10:28:57.044352+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044352+0000","last_undegraded":"2024-04-17T10:28:57.044352+0000","last_fullsized":"2024-04-17T10:28:57.044352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044169+0000","last_change":"2024-04-17T10:28:57.044169+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044169+0000","last_undegraded":"2024-04-17T10:28:57.044169+0000","last_fullsized":"2024-04-17T10:28:57.044169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044080+0000","last_change":"2024-04-17T10:28:57.044080+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044080+0000","last_undegraded":"2024-04-17T10:28:57.044080+0000","last_fullsized":"2024-04-17T10:28:57.044080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040312+0000","last_change":"2024-04-17T10:28:57.040312+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040312+0000","last_undegraded":"2024-04-17T10:28:57.040312+0000","last_fullsized":"2024-04-17T10:28:57.040312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.041387+0000","last_change":"2024-04-17T10:28:57.041387+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.041387+0000","last_undegraded":"2024-04-17T10:28:57.041387+0000","last_fullsized":"2024-04-17T10:28:57.041387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040850+0000","last_change":"2024-04-17T10:28:57.040850+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040850+0000","last_undegraded":"2024-04-17T10:28:57.040850+0000","last_fullsized":"2024-04-17T10:28:57.040850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039600+0000","last_change":"2024-04-17T10:28:57.039600+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039600+0000","last_undegraded":"2024-04-17T10:28:57.039600+0000","last_fullsized":"2024-04-17T10:28:57.039600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038703+0000","last_change":"2024-04-17T10:28:57.038703+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038703+0000","last_undegraded":"2024-04-17T10:28:57.038703+0000","last_fullsized":"2024-04-17T10:28:57.038703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.035308+0000","last_change":"2024-04-17T10:28:57.035308+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.035308+0000","last_undegraded":"2024-04-17T10:28:57.035308+0000","last_fullsized":"2024-04-17T10:28:57.035308+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039802+0000","last_change":"2024-04-17T10:28:57.039802+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039802+0000","last_undegraded":"2024-04-17T10:28:57.039802+0000","last_fullsized":"2024-04-17T10:28:57.039802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040341+0000","last_change":"2024-04-17T10:28:57.040341+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040341+0000","last_undegraded":"2024-04-17T10:28:57.040341+0000","last_fullsized":"2024-04-17T10:28:57.040341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040059+0000","last_change":"2024-04-17T10:28:57.040059+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040059+0000","last_undegraded":"2024-04-17T10:28:57.040059+0000","last_fullsized":"2024-04-17T10:28:57.040059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043675+0000","last_change":"2024-04-17T10:28:57.043675+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043675+0000","last_undegraded":"2024-04-17T10:28:57.043675+0000","last_fullsized":"2024-04-17T10:28:57.043675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.037846+0000","last_change":"2024-04-17T10:28:57.037846+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.037846+0000","last_undegraded":"2024-04-17T10:28:57.037846+0000","last_fullsized":"2024-04-17T10:28:57.037846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.035420+0000","last_change":"2024-04-17T10:28:57.035420+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.035420+0000","last_undegraded":"2024-04-17T10:28:57.035420+0000","last_fullsized":"2024-04-17T10:28:57.035420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043743+0000","last_change":"2024-04-17T10:28:57.043743+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043743+0000","last_undegraded":"2024-04-17T10:28:57.043743+0000","last_fullsized":"2024-04-17T10:28:57.043743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038266+0000","last_change":"2024-04-17T10:28:57.038266+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038266+0000","last_undegraded":"2024-04-17T10:28:57.038266+0000","last_fullsized":"2024-04-17T10:28:57.038266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038033+0000","last_change":"2024-04-17T10:28:57.038033+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038033+0000","last_undegraded":"2024-04-17T10:28:57.038033+0000","last_fullsized":"2024-04-17T10:28:57.038033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040730+0000","last_change":"2024-04-17T10:28:57.040730+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040730+0000","last_undegraded":"2024-04-17T10:28:57.040730+0000","last_fullsized":"2024-04-17T10:28:57.040730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043346+0000","last_change":"2024-04-17T10:28:57.043346+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043346+0000","last_undegraded":"2024-04-17T10:28:57.043346+0000","last_fullsized":"2024-04-17T10:28:57.043346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043128+0000","last_change":"2024-04-17T10:28:57.043128+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043128+0000","last_undegraded":"2024-04-17T10:28:57.043128+0000","last_fullsized":"2024-04-17T10:28:57.043128+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043252+0000","last_change":"2024-04-17T10:28:57.043252+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043252+0000","last_undegraded":"2024-04-17T10:28:57.043252+0000","last_fullsized":"2024-04-17T10:28:57.043252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043155+0000","last_change":"2024-04-17T10:28:57.043155+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043155+0000","last_undegraded":"2024-04-17T10:28:57.043155+0000","last_fullsized":"2024-04-17T10:28:57.043155+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043132+0000","last_change":"2024-04-17T10:28:57.043132+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043132+0000","last_undegraded":"2024-04-17T10:28:57.043132+0000","last_fullsized":"2024-04-17T10:28:57.043132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043420+0000","last_change":"2024-04-17T10:28:57.043420+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043420+0000","last_undegraded":"2024-04-17T10:28:57.043420+0000","last_fullsized":"2024-04-17T10:28:57.043420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039818+0000","last_change":"2024-04-17T10:28:57.039818+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039818+0000","last_undegraded":"2024-04-17T10:28:57.039818+0000","last_fullsized":"2024-04-17T10:28:57.039818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039061+0000","last_change":"2024-04-17T10:28:57.039061+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039061+0000","last_undegraded":"2024-04-17T10:28:57.039061+0000","last_fullsized":"2024-04-17T10:28:57.039061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040059+0000","last_change":"2024-04-17T10:28:57.040059+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040059+0000","last_undegraded":"2024-04-17T10:28:57.040059+0000","last_fullsized":"2024-04-17T10:28:57.040059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040010+0000","last_change":"2024-04-17T10:28:57.040010+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040010+0000","last_undegraded":"2024-04-17T10:28:57.040010+0000","last_fullsized":"2024-04-17T10:28:57.040010+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039826+0000","last_change":"2024-04-17T10:28:57.039826+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039826+0000","last_undegraded":"2024-04-17T10:28:57.039826+0000","last_fullsized":"2024-04-17T10:28:57.039826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040004+0000","last_change":"2024-04-17T10:28:57.040004+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040004+0000","last_undegraded":"2024-04-17T10:28:57.040004+0000","last_fullsized":"2024-04-17T10:28:57.040004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":253,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-17T10:28:57.040437+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:57.040437+0000","last_peered":"2024-04-17T10:28:57.040437+0000","last_clean":"2024-04-17T10:28:57.040437+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:57.040437+0000","last_undegraded":"2024-04-17T10:28:57.040437+0000","last_fullsized":"2024-04-17T10:28:57.040437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":1355776,"data_stored":934727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36088,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335752,"statfs":{"total":96636764160,"available":96599810048,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40232,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331608,"statfs":{"total":96636764160,"available":96595566592,"internally_reserved":0,"allocated":749568,"data_stored":338910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36084,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335756,"statfs":{"total":96636764160,"available":96599814144,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58888,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312952,"statfs":{"total":96636764160,"available":96576462848,"internally_reserved":0,"allocated":1368064,"data_stored":940936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:03.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.745+0000 7f57c6000640 1 Processor -- start 2024-04-17T10:29:03.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.745+0000 7f57c6000640 1 -- start start 2024-04-17T10:29:03.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57c6000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f57c01620a0 0x7f57c015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57c6000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 0x7f57c00cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57c6000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c015eb50 con 0x7f57c00ce020 2024-04-17T10:29:03.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57c6000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c0161650 con 0x7f57c01620a0 2024-04-17T10:29:03.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57c6000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c015ee30 con 0x7f57c015e550 2024-04-17T10:29:03.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57c00ce020 0x7f57c00ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42762/0 (socket says 172.21.15.67:42762) 2024-04-17T10:29:03.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bf400640 1 -- 172.21.15.67:0/943432651 learned_addr learned my addr 172.21.15.67:0/943432651 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bfe00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 0x7f57c00cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57b6a00640 1 --2- 172.21.15.67:0/943432651 >> v2:172.21.15.86:3300/0 conn(0x7f57c01620a0 0x7f57c015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3864020632 0 0) 0x7f57c015eb50 con 0x7f57c00ce020 2024-04-17T10:29:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a4003530 con 0x7f57c00ce020 2024-04-17T10:29:03.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1606616103 0 0) 0x7f57a4003530 con 0x7f57c00ce020 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 msgr2=0x7f57c00cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 0x7f57c00cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 >> v2:172.21.15.86:3300/0 conn(0x7f57c01620a0 msgr2=0x7f57c015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 --2- 172.21.15.67:0/943432651 >> v2:172.21.15.86:3300/0 conn(0x7f57c01620a0 0x7f57c015e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c01618a0 con 0x7f57c00ce020 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f57b0001680 con 0x7f57c00ce020 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 >> v1:172.21.15.67:6789/0 conn(0x7f57c00ce020 legacy=0x7f57c00ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57a40039c0 0x7f57a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 0x7f57a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a4004d50 0x7f57a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0161650 con 0x7f57a4004d50 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c015ee30 con 0x7f57a40039c0 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a400b9d0 con 0x7f57c015e550 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bfe00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a4004d50 0x7f57a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57b6a00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 0x7f57a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bf400640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57a40039c0 0x7f57a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bfe00640 1 -- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 msgr2=0x7f57a4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:03.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bfe00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c015e550 0x7f57a4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bfe00640 1 -- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57a40039c0 msgr2=0x7f57a4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bfe00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57a40039c0 0x7f57a4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bfe00640 1 -- 172.21.15.67:0/943432651 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4003530 con 0x7f57a4004d50 2024-04-17T10:29:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.749+0000 7f57bf400640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57a40039c0 0x7f57a4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57bfe00640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a4004d50 0x7f57a40051d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009fc0 tx=0x7f57b801cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e817c0ed416a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80b2070 con 0x7f57a4004d50 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b80a5c90 con 0x7f57a4004d50 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57bea00640 1 -- 172.21.15.67:0/943432651 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8009c10 con 0x7f57a4004d50 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57c6000640 1 -- 172.21.15.67:0/943432651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a4004d50 msgr2=0x7f57a40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57c6000640 1 --2- 172.21.15.67:0/943432651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a4004d50 0x7f57a40051d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009fc0 tx=0x7f57b801cc80 comp rx=0 tx=0).stop 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57bfe00640 1 -- 172.21.15.67:0/943432651 reap_dead start 2024-04-17T10:29:03.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57c6000640 1 -- 172.21.15.67:0/943432651 shutdown_connections 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57c6000640 1 -- 172.21.15.67:0/943432651 >> 172.21.15.67:0/943432651 conn(0x7f57c00b33f0 msgr2=0x7f57c00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57c6000640 1 -- 172.21.15.67:0/943432651 shutdown_connections 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.753+0000 7f57c6000640 1 -- 172.21.15.67:0/943432651 wait complete. 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 Processor -- start 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 -- start start 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c0157280 0x7f57c01e8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01e93c0 0x7f57c0160560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01f3c40 0x7f57c0160aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c006bf70 con 0x7f57c0157280 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0162530 con 0x7f57c01e93c0 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c006c0f0 con 0x7f57c01f3c40 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c0157280 0x7f57c01e8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c0157280 0x7f57c01e8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55764/0 (socket says 172.21.15.67:55764) 2024-04-17T10:29:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 -- 172.21.15.67:0/870375449 learned_addr learned my addr 172.21.15.67:0/870375449 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bfe00640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01f3c40 0x7f57c0160aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57b6a00640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01e93c0 0x7f57c0160560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 -- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01f3c40 msgr2=0x7f57c0160aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01f3c40 0x7f57c0160aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 -- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01e93c0 msgr2=0x7f57c0160560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01e93c0 0x7f57c0160560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 -- 172.21.15.67:0/870375449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c01e81b0 con 0x7f57c0157280 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57bf400640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c0157280 0x7f57c01e8e80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f57b000e830 tx=0x7f57b000ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57b7e00640 1 -- 172.21.15.67:0/870375449 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b000c8d0 con 0x7f57c0157280 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57b7e00640 1 -- 172.21.15.67:0/870375449 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b0018070 con 0x7f57c0157280 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c01e8490 con 0x7f57c0157280 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57b7e00640 1 -- 172.21.15.67:0/870375449 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b000cc70 con 0x7f57c0157280 2024-04-17T10:29:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c00b5910 con 0x7f57c0157280 2024-04-17T10:29:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.757+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c0070ab0 con 0x7f57c0157280 2024-04-17T10:29:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.761+0000 7f57b7e00640 1 -- 172.21.15.67:0/870375449 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f57b001d460 con 0x7f57c0157280 2024-04-17T10:29:03.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.761+0000 7f57b7e00640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f579c066f90 0x7f579c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.761+0000 7f57b7e00640 1 -- 172.21.15.67:0/870375449 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f57b0085110 con 0x7f57c0157280 2024-04-17T10:29:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.761+0000 7f57b6a00640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f579c066f90 0x7f579c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:03.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.761+0000 7f57b6a00640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f579c066f90 0x7f579c069450 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f57c00b11b0 tx=0x7f57a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:03.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.765+0000 7f57b7e00640 1 -- 172.21.15.67:0/870375449 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f57b0055f80 con 0x7f57c0157280 2024-04-17T10:29:03.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.925+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57c01e9860 con 0x7f579c066f90 2024-04-17T10:29:03.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.929+0000 7f57b7e00640 1 -- 172.21.15.67:0/870375449 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+239920 (secure 0 0 0) 0x7f57c01e9860 con 0x7f579c066f90 2024-04-17T10:29:03.928 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:03.929 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:03.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f579c066f90 msgr2=0x7f579c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f579c066f90 0x7f579c069450 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f57c00b11b0 tx=0x7f57a8008040 comp rx=0 tx=0).stop 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c0157280 msgr2=0x7f57c01e8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c0157280 0x7f57c01e8e80 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f57b000e830 tx=0x7f57b000ecf0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 shutdown_connections 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f579c066f90 0x7f579c069450 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f57c01f3c40 0x7f57c0160aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f57c01e93c0 0x7f57c0160560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 --2- 172.21.15.67:0/870375449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57c0157280 0x7f57c01e8e80 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 >> 172.21.15.67:0/870375449 conn(0x7f57c00b33f0 msgr2=0x7f57c00cff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 shutdown_connections 2024-04-17T10:29:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:03.933+0000 7f57c6000640 1 -- 172.21.15.67:0/870375449 wait complete. 2024-04-17T10:29:03.947 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-04-17T10:29:02.191603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":316236,"kb_used_data":7112,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754658484,"statfs":{"total":773094113280,"available":772770287616,"internally_reserved":0,"allocated":7282688,"data_stored":3936368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.160535"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039864+0000","last_change":"2024-04-17T10:28:57.039864+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039864+0000","last_undegraded":"2024-04-17T10:28:57.039864+0000","last_fullsized":"2024-04-17T10:28:57.039864+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044352+0000","last_change":"2024-04-17T10:28:57.044352+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044352+0000","last_undegraded":"2024-04-17T10:28:57.044352+0000","last_fullsized":"2024-04-17T10:28:57.044352+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044169+0000","last_change":"2024-04-17T10:28:57.044169+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044169+0000","last_undegraded":"2024-04-17T10:28:57.044169+0000","last_fullsized":"2024-04-17T10:28:57.044169+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.044080+0000","last_change":"2024-04-17T10:28:57.044080+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.044080+0000","last_undegraded":"2024-04-17T10:28:57.044080+0000","last_fullsized":"2024-04-17T10:28:57.044080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040312+0000","last_change":"2024-04-17T10:28:57.040312+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040312+0000","last_undegraded":"2024-04-17T10:28:57.040312+0000","last_fullsized":"2024-04-17T10:28:57.040312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.041387+0000","last_change":"2024-04-17T10:28:57.041387+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.041387+0000","last_undegraded":"2024-04-17T10:28:57.041387+0000","last_fullsized":"2024-04-17T10:28:57.041387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040850+0000","last_change":"2024-04-17T10:28:57.040850+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040850+0000","last_undegraded":"2024-04-17T10:28:57.040850+0000","last_fullsized":"2024-04-17T10:28:57.040850+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039600+0000","last_change":"2024-04-17T10:28:57.039600+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039600+0000","last_undegraded":"2024-04-17T10:28:57.039600+0000","last_fullsized":"2024-04-17T10:28:57.039600+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038703+0000","last_change":"2024-04-17T10:28:57.038703+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038703+0000","last_undegraded":"2024-04-17T10:28:57.038703+0000","last_fullsized":"2024-04-17T10:28:57.038703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.035308+0000","last_change":"2024-04-17T10:28:57.035308+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.035308+0000","last_undegraded":"2024-04-17T10:28:57.035308+0000","last_fullsized":"2024-04-17T10:28:57.035308+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039802+0000","last_change":"2024-04-17T10:28:57.039802+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039802+0000","last_undegraded":"2024-04-17T10:28:57.039802+0000","last_fullsized":"2024-04-17T10:28:57.039802+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040341+0000","last_change":"2024-04-17T10:28:57.040341+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040341+0000","last_undegraded":"2024-04-17T10:28:57.040341+0000","last_fullsized":"2024-04-17T10:28:57.040341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040059+0000","last_change":"2024-04-17T10:28:57.040059+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040059+0000","last_undegraded":"2024-04-17T10:28:57.040059+0000","last_fullsized":"2024-04-17T10:28:57.040059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043675+0000","last_change":"2024-04-17T10:28:57.043675+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043675+0000","last_undegraded":"2024-04-17T10:28:57.043675+0000","last_fullsized":"2024-04-17T10:28:57.043675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.037846+0000","last_change":"2024-04-17T10:28:57.037846+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.037846+0000","last_undegraded":"2024-04-17T10:28:57.037846+0000","last_fullsized":"2024-04-17T10:28:57.037846+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.035420+0000","last_change":"2024-04-17T10:28:57.035420+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.035420+0000","last_undegraded":"2024-04-17T10:28:57.035420+0000","last_fullsized":"2024-04-17T10:28:57.035420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043743+0000","last_change":"2024-04-17T10:28:57.043743+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043743+0000","last_undegraded":"2024-04-17T10:28:57.043743+0000","last_fullsized":"2024-04-17T10:28:57.043743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038266+0000","last_change":"2024-04-17T10:28:57.038266+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038266+0000","last_undegraded":"2024-04-17T10:28:57.038266+0000","last_fullsized":"2024-04-17T10:28:57.038266+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.038033+0000","last_change":"2024-04-17T10:28:57.038033+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.038033+0000","last_undegraded":"2024-04-17T10:28:57.038033+0000","last_fullsized":"2024-04-17T10:28:57.038033+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040730+0000","last_change":"2024-04-17T10:28:57.040730+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040730+0000","last_undegraded":"2024-04-17T10:28:57.040730+0000","last_fullsized":"2024-04-17T10:28:57.040730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043346+0000","last_change":"2024-04-17T10:28:57.043346+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043346+0000","last_undegraded":"2024-04-17T10:28:57.043346+0000","last_fullsized":"2024-04-17T10:28:57.043346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043128+0000","last_change":"2024-04-17T10:28:57.043128+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043128+0000","last_undegraded":"2024-04-17T10:28:57.043128+0000","last_fullsized":"2024-04-17T10:28:57.043128+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043252+0000","last_change":"2024-04-17T10:28:57.043252+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043252+0000","last_undegraded":"2024-04-17T10:28:57.043252+0000","last_fullsized":"2024-04-17T10:28:57.043252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043155+0000","last_change":"2024-04-17T10:28:57.043155+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043155+0000","last_undegraded":"2024-04-17T10:28:57.043155+0000","last_fullsized":"2024-04-17T10:28:57.043155+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043132+0000","last_change":"2024-04-17T10:28:57.043132+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043132+0000","last_undegraded":"2024-04-17T10:28:57.043132+0000","last_fullsized":"2024-04-17T10:28:57.043132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.043420+0000","last_change":"2024-04-17T10:28:57.043420+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.043420+0000","last_undegraded":"2024-04-17T10:28:57.043420+0000","last_fullsized":"2024-04-17T10:28:57.043420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039818+0000","last_change":"2024-04-17T10:28:57.039818+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039818+0000","last_undegraded":"2024-04-17T10:28:57.039818+0000","last_fullsized":"2024-04-17T10:28:57.039818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039061+0000","last_change":"2024-04-17T10:28:57.039061+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039061+0000","last_undegraded":"2024-04-17T10:28:57.039061+0000","last_fullsized":"2024-04-17T10:28:57.039061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040059+0000","last_change":"2024-04-17T10:28:57.040059+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040059+0000","last_undegraded":"2024-04-17T10:28:57.040059+0000","last_fullsized":"2024-04-17T10:28:57.040059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040010+0000","last_change":"2024-04-17T10:28:57.040010+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040010+0000","last_undegraded":"2024-04-17T10:28:57.040010+0000","last_fullsized":"2024-04-17T10:28:57.040010+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.039826+0000","last_change":"2024-04-17T10:28:57.039826+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.039826+0000","last_undegraded":"2024-04-17T10:28:57.039826+0000","last_fullsized":"2024-04-17T10:28:57.039826+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:28:58.032381+0000","last_change":"2024-04-17T10:28:58.032381+0000","last_active":"2024-04-17T10:28:58.032381+0000","last_peered":"2024-04-17T10:28:58.032381+0000","last_clean":"2024-04-17T10:28:58.032381+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:58.032381+0000","last_undegraded":"2024-04-17T10:28:58.032381+0000","last_fullsized":"2024-04-17T10:28:58.032381+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":64,"state":"creating+peering","last_fresh":"2024-04-17T10:28:57.040004+0000","last_change":"2024-04-17T10:28:57.040004+0000","last_active":"2024-04-17T10:28:57.024219+0000","last_peered":"2024-04-17T10:28:57.024219+0000","last_clean":"2024-04-17T10:28:57.024219+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:28:57.040004+0000","last_undegraded":"2024-04-17T10:28:57.040004+0000","last_fullsized":"2024-04-17T10:28:57.040004+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":253,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-17T10:28:57.040437+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:28:57.040437+0000","last_peered":"2024-04-17T10:28:57.040437+0000","last_clean":"2024-04-17T10:28:57.040437+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:28:57.040437+0000","last_undegraded":"2024-04-17T10:28:57.040437+0000","last_fullsized":"2024-04-17T10:28:57.040437+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36668,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335172,"statfs":{"total":96636764160,"available":96599216128,"internally_reserved":0,"allocated":1355776,"data_stored":934727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36088,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335752,"statfs":{"total":96636764160,"available":96599810048,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40232,"kb_used_data":732,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331608,"statfs":{"total":96636764160,"available":96595566592,"internally_reserved":0,"allocated":749568,"data_stored":338910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574902,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36092,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335748,"statfs":{"total":96636764160,"available":96599805952,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36084,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335756,"statfs":{"total":96636764160,"available":96599814144,"internally_reserved":0,"allocated":761856,"data_stored":344359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58888,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312952,"statfs":{"total":96636764160,"available":96576462848,"internally_reserved":0,"allocated":1368064,"data_stored":940936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:03.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3f is not active+clean 2024-04-17T10:29:03.951 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.951 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3e is not active+clean 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3d is not active+clean 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3c is not active+clean 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3b is not active+clean 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3a is not active+clean 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.39 is not active+clean 2024-04-17T10:29:03.952 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.38 is not active+clean 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.37 is not active+clean 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.36 is not active+clean 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.35 is not active+clean 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.34 is not active+clean 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.953 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.33 is not active+clean 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.32 is not active+clean 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.31 is not active+clean 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.30 is not active+clean 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2f is not active+clean 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.954 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2e is not active+clean 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2d is not active+clean 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2c is not active+clean 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2b is not active+clean 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2a is not active+clean 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.29 is not active+clean 2024-04-17T10:29:03.955 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.28 is not active+clean 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.27 is not active+clean 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.26 is not active+clean 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.25 is not active+clean 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.24 is not active+clean 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.23 is not active+clean 2024-04-17T10:29:03.956 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.22 is not active+clean 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.21 is not active+clean 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.20 is not active+clean 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1c is not active+clean 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1f is not active+clean 2024-04-17T10:29:03.957 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.039864+0000', 'last_change': '2024-04-17T10:28:57.039864+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.039864+0000', 'last_undegraded': '2024-04-17T10:28:57.039864+0000', 'last_fullsized': '2024-04-17T10:28:57.039864+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1d is not active+clean 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.044352+0000', 'last_change': '2024-04-17T10:28:57.044352+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.044352+0000', 'last_undegraded': '2024-04-17T10:28:57.044352+0000', 'last_fullsized': '2024-04-17T10:28:57.044352+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1e is not active+clean 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.044169+0000', 'last_change': '2024-04-17T10:28:57.044169+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.044169+0000', 'last_undegraded': '2024-04-17T10:28:57.044169+0000', 'last_fullsized': '2024-04-17T10:28:57.044169+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.958 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.044080+0000', 'last_change': '2024-04-17T10:28:57.044080+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.044080+0000', 'last_undegraded': '2024-04-17T10:28:57.044080+0000', 'last_fullsized': '2024-04-17T10:28:57.044080+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.18 is not active+clean 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040312+0000', 'last_change': '2024-04-17T10:28:57.040312+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040312+0000', 'last_undegraded': '2024-04-17T10:28:57.040312+0000', 'last_fullsized': '2024-04-17T10:28:57.040312+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.19 is not active+clean 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.041387+0000', 'last_change': '2024-04-17T10:28:57.041387+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.041387+0000', 'last_undegraded': '2024-04-17T10:28:57.041387+0000', 'last_fullsized': '2024-04-17T10:28:57.041387+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1a is not active+clean 2024-04-17T10:29:03.959 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040850+0000', 'last_change': '2024-04-17T10:28:57.040850+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040850+0000', 'last_undegraded': '2024-04-17T10:28:57.040850+0000', 'last_fullsized': '2024-04-17T10:28:57.040850+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1b is not active+clean 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.039600+0000', 'last_change': '2024-04-17T10:28:57.039600+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.039600+0000', 'last_undegraded': '2024-04-17T10:28:57.039600+0000', 'last_fullsized': '2024-04-17T10:28:57.039600+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.14 is not active+clean 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-04-17T10:29:03.960 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.038703+0000', 'last_change': '2024-04-17T10:28:57.038703+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.038703+0000', 'last_undegraded': '2024-04-17T10:28:57.038703+0000', 'last_fullsized': '2024-04-17T10:28:57.038703+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.15 is not active+clean 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.035308+0000', 'last_change': '2024-04-17T10:28:57.035308+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.035308+0000', 'last_undegraded': '2024-04-17T10:28:57.035308+0000', 'last_fullsized': '2024-04-17T10:28:57.035308+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.16 is not active+clean 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.039802+0000', 'last_change': '2024-04-17T10:28:57.039802+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.039802+0000', 'last_undegraded': '2024-04-17T10:28:57.039802+0000', 'last_fullsized': '2024-04-17T10:28:57.039802+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.17 is not active+clean 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.14 is not active+clean 2024-04-17T10:29:03.961 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040341+0000', 'last_change': '2024-04-17T10:28:57.040341+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040341+0000', 'last_undegraded': '2024-04-17T10:28:57.040341+0000', 'last_fullsized': '2024-04-17T10:28:57.040341+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.10 is not active+clean 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040059+0000', 'last_change': '2024-04-17T10:28:57.040059+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040059+0000', 'last_undegraded': '2024-04-17T10:28:57.040059+0000', 'last_fullsized': '2024-04-17T10:28:57.040059+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.11 is not active+clean 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043675+0000', 'last_change': '2024-04-17T10:28:57.043675+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043675+0000', 'last_undegraded': '2024-04-17T10:28:57.043675+0000', 'last_fullsized': '2024-04-17T10:28:57.043675+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.12 is not active+clean 2024-04-17T10:29:03.962 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.037846+0000', 'last_change': '2024-04-17T10:28:57.037846+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.037846+0000', 'last_undegraded': '2024-04-17T10:28:57.037846+0000', 'last_fullsized': '2024-04-17T10:28:57.037846+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.13 is not active+clean 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.035420+0000', 'last_change': '2024-04-17T10:28:57.035420+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.035420+0000', 'last_undegraded': '2024-04-17T10:28:57.035420+0000', 'last_fullsized': '2024-04-17T10:28:57.035420+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043743+0000', 'last_change': '2024-04-17T10:28:57.043743+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043743+0000', 'last_undegraded': '2024-04-17T10:28:57.043743+0000', 'last_fullsized': '2024-04-17T10:28:57.043743+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:03.963 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.d is not active+clean 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.038266+0000', 'last_change': '2024-04-17T10:28:57.038266+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.038266+0000', 'last_undegraded': '2024-04-17T10:28:57.038266+0000', 'last_fullsized': '2024-04-17T10:28:57.038266+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.e is not active+clean 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.038033+0000', 'last_change': '2024-04-17T10:28:57.038033+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.038033+0000', 'last_undegraded': '2024-04-17T10:28:57.038033+0000', 'last_fullsized': '2024-04-17T10:28:57.038033+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.f is not active+clean 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-04-17T10:29:03.964 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040730+0000', 'last_change': '2024-04-17T10:28:57.040730+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040730+0000', 'last_undegraded': '2024-04-17T10:28:57.040730+0000', 'last_fullsized': '2024-04-17T10:28:57.040730+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.8 is not active+clean 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043346+0000', 'last_change': '2024-04-17T10:28:57.043346+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043346+0000', 'last_undegraded': '2024-04-17T10:28:57.043346+0000', 'last_fullsized': '2024-04-17T10:28:57.043346+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.9 is not active+clean 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043128+0000', 'last_change': '2024-04-17T10:28:57.043128+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043128+0000', 'last_undegraded': '2024-04-17T10:28:57.043128+0000', 'last_fullsized': '2024-04-17T10:28:57.043128+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.a is not active+clean 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.965 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043252+0000', 'last_change': '2024-04-17T10:28:57.043252+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043252+0000', 'last_undegraded': '2024-04-17T10:28:57.043252+0000', 'last_fullsized': '2024-04-17T10:28:57.043252+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.4 is not active+clean 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043155+0000', 'last_change': '2024-04-17T10:28:57.043155+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043155+0000', 'last_undegraded': '2024-04-17T10:28:57.043155+0000', 'last_fullsized': '2024-04-17T10:28:57.043155+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.5 is not active+clean 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043132+0000', 'last_change': '2024-04-17T10:28:57.043132+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043132+0000', 'last_undegraded': '2024-04-17T10:28:57.043132+0000', 'last_fullsized': '2024-04-17T10:28:57.043132+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.6 is not active+clean 2024-04-17T10:29:03.966 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.043420+0000', 'last_change': '2024-04-17T10:28:57.043420+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.043420+0000', 'last_undegraded': '2024-04-17T10:28:57.043420+0000', 'last_fullsized': '2024-04-17T10:28:57.043420+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.7 is not active+clean 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.4 is not active+clean 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.039818+0000', 'last_change': '2024-04-17T10:28:57.039818+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.039818+0000', 'last_undegraded': '2024-04-17T10:28:57.039818+0000', 'last_fullsized': '2024-04-17T10:28:57.039818+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.0 is not active+clean 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.039061+0000', 'last_change': '2024-04-17T10:28:57.039061+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.039061+0000', 'last_undegraded': '2024-04-17T10:28:57.039061+0000', 'last_fullsized': '2024-04-17T10:28:57.039061+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:03.967 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1 is not active+clean 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040059+0000', 'last_change': '2024-04-17T10:28:57.040059+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040059+0000', 'last_undegraded': '2024-04-17T10:28:57.040059+0000', 'last_fullsized': '2024-04-17T10:28:57.040059+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2 is not active+clean 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040010+0000', 'last_change': '2024-04-17T10:28:57.040010+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040010+0000', 'last_undegraded': '2024-04-17T10:28:57.040010+0000', 'last_fullsized': '2024-04-17T10:28:57.040010+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3 is not active+clean 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.0 is not active+clean 2024-04-17T10:29:03.968 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.039826+0000', 'last_change': '2024-04-17T10:28:57.039826+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.039826+0000', 'last_undegraded': '2024-04-17T10:28:57.039826+0000', 'last_fullsized': '2024-04-17T10:28:57.039826+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:29:03.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.b is not active+clean 2024-04-17T10:29:03.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:28:58.032381+0000', 'last_change': '2024-04-17T10:28:58.032381+0000', 'last_active': '2024-04-17T10:28:58.032381+0000', 'last_peered': '2024-04-17T10:28:58.032381+0000', 'last_clean': '2024-04-17T10:28:58.032381+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:58.032381+0000', 'last_undegraded': '2024-04-17T10:28:58.032381+0000', 'last_fullsized': '2024-04-17T10:28:58.032381+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:58.032381+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:03.969 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-04-17T10:29:03.969 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 64, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:28:57.040004+0000', 'last_change': '2024-04-17T10:28:57.040004+0000', 'last_active': '2024-04-17T10:28:57.024219+0000', 'last_peered': '2024-04-17T10:28:57.024219+0000', 'last_clean': '2024-04-17T10:28:57.024219+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:28:57.040004+0000', 'last_undegraded': '2024-04-17T10:28:57.040004+0000', 'last_fullsized': '2024-04-17T10:28:57.040004+0000', 'mapping_epoch': 64, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 64, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'last_clean_scrub_stamp': '2024-04-17T10:28:57.024219+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:03.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:29:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5450c00640 1 Processor -- start 2024-04-17T10:29:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5450c00640 1 -- start start 2024-04-17T10:29:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5450c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f544c1446c0 0x7f544c144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5450c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c0b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5450c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f544c1456d0 con 0x7f544c144aa0 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5450c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544c0b51a0 con 0x7f544c1446c0 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5450c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544c0b4870 con 0x7f544c071840 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5449e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f544c1446c0 0x7f544c144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5449e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f544c1446c0 0x7f544c144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55774/0 (socket says 172.21.15.67:55774) 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5449e00640 1 -- 172.21.15.67:0/780867886 learned_addr learned my addr 172.21.15.67:0/780867886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f544a800640 1 --2- 172.21.15.67:0/780867886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c0b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5449e00640 1 -- 172.21.15.67:0/780867886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 msgr2=0x7f544c0b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5449e00640 1 --2- 172.21.15.67:0/780867886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c0b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5449e00640 1 -- 172.21.15.67:0/780867886 >> v1:172.21.15.86:6789/0 conn(0x7f544c144aa0 legacy=0x7f544c0d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.137+0000 7f5449e00640 1 -- 172.21.15.67:0/780867886 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c0b4b50 con 0x7f544c1446c0 2024-04-17T10:29:04.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5449e00640 1 --2- 172.21.15.67:0/780867886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c1446c0 0x7f544c144ea0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f543c009870 tx=0x7f543c01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=894231fc0cb8bf2e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:04.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5448a00640 1 -- 172.21.15.67:0/780867886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c0ab640 con 0x7f544c1446c0 2024-04-17T10:29:04.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5448a00640 1 -- 172.21.15.67:0/780867886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c0b0070 con 0x7f544c1446c0 2024-04-17T10:29:04.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5448a00640 1 -- 172.21.15.67:0/780867886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c0aba00 con 0x7f544c1446c0 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 -- 172.21.15.67:0/780867886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c1446c0 msgr2=0x7f544c144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 --2- 172.21.15.67:0/780867886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c1446c0 0x7f544c144ea0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f543c009870 tx=0x7f543c01c530 comp rx=0 tx=0).stop 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 -- 172.21.15.67:0/780867886 shutdown_connections 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 --2- 172.21.15.67:0/780867886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c0b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 --2- 172.21.15.67:0/780867886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c1446c0 0x7f544c144ea0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 -- 172.21.15.67:0/780867886 >> 172.21.15.67:0/780867886 conn(0x7f544c0b34d0 msgr2=0x7f544c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 -- 172.21.15.67:0/780867886 shutdown_connections 2024-04-17T10:29:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 -- 172.21.15.67:0/780867886 wait complete. 2024-04-17T10:29:04.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 Processor -- start 2024-04-17T10:29:04.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.141+0000 7f5450c00640 1 -- start start 2024-04-17T10:29:04.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c13bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f544c1446c0 0x7f544c0cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c144aa0 0x7f544c1579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c0b6500 con 0x7f544c144aa0 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c0b6380 con 0x7f544c1446c0 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544c0b6680 con 0x7f544c071840 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c144aa0 0x7f544c1579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c144aa0 0x7f544c1579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55780/0 (socket says 172.21.15.67:55780) 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 -- 172.21.15.67:0/1749679198 learned_addr learned my addr 172.21.15.67:0/1749679198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5449400640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f544c1446c0 0x7f544c0cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5449e00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c13bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 -- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 msgr2=0x7f544c13bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c13bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 -- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f544c1446c0 msgr2=0x7f544c0cec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f544c1446c0 0x7f544c0cec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 -- 172.21.15.67:0/1749679198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c158980 con 0x7f544c144aa0 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f544a800640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c144aa0 0x7f544c1579e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5444007e40 tx=0x7f544400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5442000640 1 -- 172.21.15.67:0/1749679198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444014620 con 0x7f544c144aa0 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5442000640 1 -- 172.21.15.67:0/1749679198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5444019070 con 0x7f544c144aa0 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5442000640 1 -- 172.21.15.67:0/1749679198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5444014a20 con 0x7f544c144aa0 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544c158c60 con 0x7f544c144aa0 2024-04-17T10:29:04.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544c159220 con 0x7f544c144aa0 2024-04-17T10:29:04.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.145+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544c074e30 con 0x7f544c144aa0 2024-04-17T10:29:04.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.149+0000 7f5442000640 1 -- 172.21.15.67:0/1749679198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5444014bc0 con 0x7f544c144aa0 2024-04-17T10:29:04.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.149+0000 7f5442000640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f542c06b370 0x7f542c06d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.149+0000 7f5442000640 1 -- 172.21.15.67:0/1749679198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5444085860 con 0x7f544c144aa0 2024-04-17T10:29:04.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.149+0000 7f5449e00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f542c06b370 0x7f542c06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.149+0000 7f5449e00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f542c06b370 0x7f542c06d830 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f543c0af000 tx=0x7f543c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:04.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.153+0000 7f5442000640 1 -- 172.21.15.67:0/1749679198 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5444056120 con 0x7f544c144aa0 2024-04-17T10:29:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f544c073810 con 0x7f544c144aa0 2024-04-17T10:29:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5442000640 1 -- 172.21.15.67:0/1749679198 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1658 (secure 0 0 0) 0x7f544405a1c0 con 0x7f544c144aa0 2024-04-17T10:29:04.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:04.368 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":275,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":67,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":323825664,"bytes_avail":772770287616,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":40,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"c","status":"up:active","gid":6651}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-04-17T10:28:40.186415+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:29:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f542c06b370 msgr2=0x7f542c06d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f542c06b370 0x7f542c06d830 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f543c0af000 tx=0x7f543c0a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c144aa0 msgr2=0x7f544c1579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c144aa0 0x7f544c1579e0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5444007e40 tx=0x7f544400e4b0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 shutdown_connections 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f542c06b370 0x7f542c06d830 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f544c144aa0 0x7f544c1579e0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f544c1446c0 0x7f544c0cec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.373+0000 7f5450c00640 1 --2- 172.21.15.67:0/1749679198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f544c071840 0x7f544c13bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.377+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 >> 172.21.15.67:0/1749679198 conn(0x7f544c0b34d0 msgr2=0x7f544c0b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.377+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 shutdown_connections 2024-04-17T10:29:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.377+0000 7f5450c00640 1 -- 172.21.15.67:0/1749679198 wait complete. 2024-04-17T10:29:04.390 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-17T10:29:04.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:04.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc8c00640 1 Processor -- start 2024-04-17T10:29:04.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc8c00640 1 -- start start 2024-04-17T10:29:04.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc8c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc406c4f0 0x7fdbc40acee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc8c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdbc4071c60 0x7fdbc40720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc8c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbc415c2d0 con 0x7fdbc406c110 2024-04-17T10:29:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc8c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc4072610 con 0x7fdbc4071c60 2024-04-17T10:29:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc8c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc40ad420 con 0x7fdbc406c4f0 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc2800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdbc4071c60 0x7fdbc40720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbb9400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc406c4f0 0x7fdbc40acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc1e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fdbc406c110 0x7fdbc406f560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:57888/0 (socket says 172.21.15.67:57888) 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc2800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdbc4071c60 0x7fdbc40720d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:56362/0 (socket says 172.21.15.67:56362) 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc1e00640 1 -- 172.21.15.67:0/2493820881 learned_addr learned my addr 172.21.15.67:0/2493820881 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbb9400640 1 -- 172.21.15.67:0/2493820881 >> v2:172.21.15.86:3300/0 conn(0x7fdbc4071c60 msgr2=0x7fdbc40720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbb9400640 1 --2- 172.21.15.67:0/2493820881 >> v2:172.21.15.86:3300/0 conn(0x7fdbc4071c60 0x7fdbc40720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbb9400640 1 -- 172.21.15.67:0/2493820881 >> v1:172.21.15.86:6789/0 conn(0x7fdbc406c110 legacy=0x7fdbc406f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbb9400640 1 -- 172.21.15.67:0/2493820881 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc415c5b0 con 0x7fdbc406c4f0 2024-04-17T10:29:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc1400640 1 -- 172.21.15.67:0/2493820881 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507479876 0 0) 0x7fdbc415c2d0 con 0x7fdbc406c110 2024-04-17T10:29:04.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.537+0000 7fdbc2800640 1 --2- 172.21.15.67:0/2493820881 >> v2:172.21.15.86:3300/0 conn(0x7fdbc4071c60 0x7fdbc40720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbb9400640 1 --2- 172.21.15.67:0/2493820881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc406c4f0 0x7fdbc40acee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40097c0 tx=0x7fdbb401c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7af3ac863c03d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:04.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc1400640 1 -- 172.21.15.67:0/2493820881 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40a7070 con 0x7fdbc406c4f0 2024-04-17T10:29:04.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc1400640 1 -- 172.21.15.67:0/2493820881 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb40b0070 con 0x7fdbc406c4f0 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc1400640 1 -- 172.21.15.67:0/2493820881 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb40ab890 con 0x7fdbc406c4f0 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 -- 172.21.15.67:0/2493820881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc406c4f0 msgr2=0x7fdbc40acee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/2493820881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc406c4f0 0x7fdbc40acee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40097c0 tx=0x7fdbb401c470 comp rx=0 tx=0).stop 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 -- 172.21.15.67:0/2493820881 shutdown_connections 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/2493820881 >> v2:172.21.15.86:3300/0 conn(0x7fdbc4071c60 0x7fdbc40720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/2493820881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc406c4f0 0x7fdbc40acee0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 -- 172.21.15.67:0/2493820881 >> 172.21.15.67:0/2493820881 conn(0x7fdbc40af670 msgr2=0x7fdbc40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 -- 172.21.15.67:0/2493820881 shutdown_connections 2024-04-17T10:29:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.541+0000 7fdbc8c00640 1 -- 172.21.15.67:0/2493820881 wait complete. 2024-04-17T10:29:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 Processor -- start 2024-04-17T10:29:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 -- start start 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc406c110 0x7fdbc414c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbc406c4f0 0x7fdbc41f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc4071c60 0x7fdbc41f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc406dcb0 con 0x7fdbc406c110 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc415bf80 con 0x7fdbc406c4f0 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc406de30 con 0x7fdbc4071c60 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc406c110 0x7fdbc414c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc406c110 0x7fdbc414c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55796/0 (socket says 172.21.15.67:55796) 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 -- 172.21.15.67:0/1564771254 learned_addr learned my addr 172.21.15.67:0/1564771254 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbb9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbc406c4f0 0x7fdbc41f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc2800640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc4071c60 0x7fdbc41f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 -- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc4071c60 msgr2=0x7fdbc41f6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc4071c60 0x7fdbc41f6bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 -- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbc406c4f0 msgr2=0x7fdbc41f66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbc406c4f0 0x7fdbc41f66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 -- 172.21.15.67:0/1564771254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc414ece0 con 0x7fdbc406c110 2024-04-17T10:29:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc2800640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc4071c60 0x7fdbc41f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc1e00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc406c110 0x7fdbc414c9b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c520 tx=0x7fdbac00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbbaa00640 1 -- 172.21.15.67:0/1564771254 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac018070 con 0x7fdbc406c110 2024-04-17T10:29:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbbaa00640 1 -- 172.21.15.67:0/1564771254 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac0074e0 con 0x7fdbc406c110 2024-04-17T10:29:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc4150c30 con 0x7fdbc406c110 2024-04-17T10:29:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbbaa00640 1 -- 172.21.15.67:0/1564771254 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0049f0 con 0x7fdbc406c110 2024-04-17T10:29:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.545+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc414ef80 con 0x7fdbc406c110 2024-04-17T10:29:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.549+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc409e260 con 0x7fdbc406c110 2024-04-17T10:29:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.549+0000 7fdbbaa00640 1 -- 172.21.15.67:0/1564771254 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdbac01b070 con 0x7fdbc406c110 2024-04-17T10:29:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.549+0000 7fdbbaa00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdba0066f60 0x7fdba0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.549+0000 7fdbbaa00640 1 -- 172.21.15.67:0/1564771254 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdbac084650 con 0x7fdbc406c110 2024-04-17T10:29:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.549+0000 7fdbb9400640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdba0066f60 0x7fdba0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.549+0000 7fdbb9400640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdba0066f60 0x7fdba0069420 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40af000 tx=0x7fdbb40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.553+0000 7fdbbaa00640 1 -- 172.21.15.67:0/1564771254 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdbac054de0 con 0x7fdbc406c110 2024-04-17T10:29:04.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.709+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbc4151150 con 0x7fdba0066f60 2024-04-17T10:29:04.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.717+0000 7fdbbaa00640 1 -- 172.21.15.67:0/1564771254 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252020 (secure 0 0 0) 0x7fdbc4151150 con 0x7fdba0066f60 2024-04-17T10:29:04.715 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:04.717 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:04.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdba0066f60 msgr2=0x7fdba0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdba0066f60 0x7fdba0069420 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdbb40af000 tx=0x7fdbb40a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:04.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc406c110 msgr2=0x7fdbc414c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:04.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc406c110 0x7fdbc414c9b0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00c520 tx=0x7fdbac00c9e0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 shutdown_connections 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdba0066f60 0x7fdba0069420 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbc4071c60 0x7fdbc41f6bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbc406c4f0 0x7fdbc41f66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 --2- 172.21.15.67:0/1564771254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbc406c110 0x7fdbc414c9b0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 >> 172.21.15.67:0/1564771254 conn(0x7fdbc40af670 msgr2=0x7fdbc406f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 shutdown_connections 2024-04-17T10:29:04.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:04.721+0000 7fdbc8c00640 1 -- 172.21.15.67:0/1564771254 wait complete. 2024-04-17T10:29:04.735 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-04-17T10:29:04.192019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":316668,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754658052,"statfs":{"total":773094113280,"available":772769845248,"internally_reserved":0,"allocated":7741440,"data_stored":4154244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.154903"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469935+0000","last_change":"2024-04-17T10:28:59.073293+0000","last_active":"2024-04-17T10:29:00.469935+0000","last_peered":"2024-04-17T10:29:00.469935+0000","last_clean":"2024-04-17T10:29:00.469935+0000","last_became_active":"2024-04-17T10:28:59.072979+0000","last_became_peered":"2024-04-17T10:28:59.072979+0000","last_unstale":"2024-04-17T10:29:00.469935+0000","last_undegraded":"2024-04-17T10:29:00.469935+0000","last_fullsized":"2024-04-17T10:29:00.469935+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:39.540090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469483+0000","last_change":"2024-04-17T10:28:59.069368+0000","last_active":"2024-04-17T10:29:00.469483+0000","last_peered":"2024-04-17T10:29:00.469483+0000","last_clean":"2024-04-17T10:29:00.469483+0000","last_became_active":"2024-04-17T10:28:59.069079+0000","last_became_peered":"2024-04-17T10:28:59.069079+0000","last_unstale":"2024-04-17T10:29:00.469483+0000","last_undegraded":"2024-04-17T10:29:00.469483+0000","last_fullsized":"2024-04-17T10:29:00.469483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:27:23.690447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466483+0000","last_change":"2024-04-17T10:28:59.075137+0000","last_active":"2024-04-17T10:29:00.466483+0000","last_peered":"2024-04-17T10:29:00.466483+0000","last_clean":"2024-04-17T10:29:00.466483+0000","last_became_active":"2024-04-17T10:28:59.072221+0000","last_became_peered":"2024-04-17T10:28:59.072221+0000","last_unstale":"2024-04-17T10:29:00.466483+0000","last_undegraded":"2024-04-17T10:29:00.466483+0000","last_fullsized":"2024-04-17T10:29:00.466483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:41.817731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044415+0000","last_change":"2024-04-17T10:28:59.061148+0000","last_active":"2024-04-17T10:29:00.044415+0000","last_peered":"2024-04-17T10:29:00.044415+0000","last_clean":"2024-04-17T10:29:00.044415+0000","last_became_active":"2024-04-17T10:28:59.057038+0000","last_became_peered":"2024-04-17T10:28:59.057038+0000","last_unstale":"2024-04-17T10:29:00.044415+0000","last_undegraded":"2024-04-17T10:29:00.044415+0000","last_fullsized":"2024-04-17T10:29:00.044415+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:52:31.810508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465803+0000","last_change":"2024-04-17T10:28:59.052912+0000","last_active":"2024-04-17T10:29:00.465803+0000","last_peered":"2024-04-17T10:29:00.465803+0000","last_clean":"2024-04-17T10:29:00.465803+0000","last_became_active":"2024-04-17T10:28:59.052642+0000","last_became_peered":"2024-04-17T10:28:59.052642+0000","last_unstale":"2024-04-17T10:29:00.465803+0000","last_undegraded":"2024-04-17T10:29:00.465803+0000","last_fullsized":"2024-04-17T10:29:00.465803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:21.757809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046249+0000","last_change":"2024-04-17T10:28:59.065923+0000","last_active":"2024-04-17T10:29:00.046249+0000","last_peered":"2024-04-17T10:29:00.046249+0000","last_clean":"2024-04-17T10:29:00.046249+0000","last_became_active":"2024-04-17T10:28:59.065640+0000","last_became_peered":"2024-04-17T10:28:59.065640+0000","last_unstale":"2024-04-17T10:29:00.046249+0000","last_undegraded":"2024-04-17T10:29:00.046249+0000","last_fullsized":"2024-04-17T10:29:00.046249+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:51.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.468938+0000","last_change":"2024-04-17T10:28:59.062726+0000","last_active":"2024-04-17T10:29:00.468938+0000","last_peered":"2024-04-17T10:29:00.468938+0000","last_clean":"2024-04-17T10:29:00.468938+0000","last_became_active":"2024-04-17T10:28:59.062418+0000","last_became_peered":"2024-04-17T10:28:59.062418+0000","last_unstale":"2024-04-17T10:29:00.468938+0000","last_undegraded":"2024-04-17T10:29:00.468938+0000","last_fullsized":"2024-04-17T10:29:00.468938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:37:17.066624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051805+0000","last_change":"2024-04-17T10:28:59.070732+0000","last_active":"2024-04-17T10:29:00.051805+0000","last_peered":"2024-04-17T10:29:00.051805+0000","last_clean":"2024-04-17T10:29:00.051805+0000","last_became_active":"2024-04-17T10:28:59.070464+0000","last_became_peered":"2024-04-17T10:28:59.070464+0000","last_unstale":"2024-04-17T10:29:00.051805+0000","last_undegraded":"2024-04-17T10:29:00.051805+0000","last_fullsized":"2024-04-17T10:29:00.051805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:47:28.733650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466787+0000","last_change":"2024-04-17T10:28:59.064898+0000","last_active":"2024-04-17T10:29:00.466787+0000","last_peered":"2024-04-17T10:29:00.466787+0000","last_clean":"2024-04-17T10:29:00.466787+0000","last_became_active":"2024-04-17T10:28:59.064563+0000","last_became_peered":"2024-04-17T10:28:59.064563+0000","last_unstale":"2024-04-17T10:29:00.466787+0000","last_undegraded":"2024-04-17T10:29:00.466787+0000","last_fullsized":"2024-04-17T10:29:00.466787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:59.527524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466316+0000","last_change":"2024-04-17T10:28:59.070664+0000","last_active":"2024-04-17T10:29:00.466316+0000","last_peered":"2024-04-17T10:29:00.466316+0000","last_clean":"2024-04-17T10:29:00.466316+0000","last_became_active":"2024-04-17T10:28:59.066601+0000","last_became_peered":"2024-04-17T10:28:59.066601+0000","last_unstale":"2024-04-17T10:29:00.466316+0000","last_undegraded":"2024-04-17T10:29:00.466316+0000","last_fullsized":"2024-04-17T10:29:00.466316+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:42.570951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044287+0000","last_change":"2024-04-17T10:28:59.052440+0000","last_active":"2024-04-17T10:29:00.044287+0000","last_peered":"2024-04-17T10:29:00.044287+0000","last_clean":"2024-04-17T10:29:00.044287+0000","last_became_active":"2024-04-17T10:28:59.052113+0000","last_became_peered":"2024-04-17T10:28:59.052113+0000","last_unstale":"2024-04-17T10:29:00.044287+0000","last_undegraded":"2024-04-17T10:29:00.044287+0000","last_fullsized":"2024-04-17T10:29:00.044287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:57.733252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052126+0000","last_change":"2024-04-17T10:28:59.078195+0000","last_active":"2024-04-17T10:29:00.052126+0000","last_peered":"2024-04-17T10:29:00.052126+0000","last_clean":"2024-04-17T10:29:00.052126+0000","last_became_active":"2024-04-17T10:28:59.077897+0000","last_became_peered":"2024-04-17T10:28:59.077897+0000","last_unstale":"2024-04-17T10:29:00.052126+0000","last_undegraded":"2024-04-17T10:29:00.052126+0000","last_fullsized":"2024-04-17T10:29:00.052126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:04.822143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052059+0000","last_change":"2024-04-17T10:28:59.072616+0000","last_active":"2024-04-17T10:29:00.052059+0000","last_peered":"2024-04-17T10:29:00.052059+0000","last_clean":"2024-04-17T10:29:00.052059+0000","last_became_active":"2024-04-17T10:28:59.072141+0000","last_became_peered":"2024-04-17T10:28:59.072141+0000","last_unstale":"2024-04-17T10:29:00.052059+0000","last_undegraded":"2024-04-17T10:29:00.052059+0000","last_fullsized":"2024-04-17T10:29:00.052059+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:42.570951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466463+0000","last_change":"2024-04-17T10:28:59.072969+0000","last_active":"2024-04-17T10:29:00.466463+0000","last_peered":"2024-04-17T10:29:00.466463+0000","last_clean":"2024-04-17T10:29:00.466463+0000","last_became_active":"2024-04-17T10:28:59.072408+0000","last_became_peered":"2024-04-17T10:28:59.072408+0000","last_unstale":"2024-04-17T10:29:00.466463+0000","last_undegraded":"2024-04-17T10:29:00.466463+0000","last_fullsized":"2024-04-17T10:29:00.466463+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:09.740067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465996+0000","last_change":"2024-04-17T10:28:59.074659+0000","last_active":"2024-04-17T10:29:00.465996+0000","last_peered":"2024-04-17T10:29:00.465996+0000","last_clean":"2024-04-17T10:29:00.465996+0000","last_became_active":"2024-04-17T10:28:59.074389+0000","last_became_peered":"2024-04-17T10:28:59.074389+0000","last_unstale":"2024-04-17T10:29:00.465996+0000","last_undegraded":"2024-04-17T10:29:00.465996+0000","last_fullsized":"2024-04-17T10:29:00.465996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:49:29.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.468881+0000","last_change":"2024-04-17T10:28:59.074052+0000","last_active":"2024-04-17T10:29:00.468881+0000","last_peered":"2024-04-17T10:29:00.468881+0000","last_clean":"2024-04-17T10:29:00.468881+0000","last_became_active":"2024-04-17T10:28:59.073683+0000","last_became_peered":"2024-04-17T10:28:59.073683+0000","last_unstale":"2024-04-17T10:29:00.468881+0000","last_undegraded":"2024-04-17T10:29:00.468881+0000","last_fullsized":"2024-04-17T10:29:00.468881+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:51:28.760315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469998+0000","last_change":"2024-04-17T10:28:59.065676+0000","last_active":"2024-04-17T10:29:00.469998+0000","last_peered":"2024-04-17T10:29:00.469998+0000","last_clean":"2024-04-17T10:29:00.469998+0000","last_became_active":"2024-04-17T10:28:59.065399+0000","last_became_peered":"2024-04-17T10:28:59.065399+0000","last_unstale":"2024-04-17T10:29:00.469998+0000","last_undegraded":"2024-04-17T10:29:00.469998+0000","last_fullsized":"2024-04-17T10:29:00.469998+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:25.541166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044699+0000","last_change":"2024-04-17T10:28:59.063981+0000","last_active":"2024-04-17T10:29:00.044699+0000","last_peered":"2024-04-17T10:29:00.044699+0000","last_clean":"2024-04-17T10:29:00.044699+0000","last_became_active":"2024-04-17T10:28:59.063472+0000","last_became_peered":"2024-04-17T10:28:59.063472+0000","last_unstale":"2024-04-17T10:29:00.044699+0000","last_undegraded":"2024-04-17T10:29:00.044699+0000","last_fullsized":"2024-04-17T10:29:00.044699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:44.295601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466964+0000","last_change":"2024-04-17T10:28:59.052390+0000","last_active":"2024-04-17T10:29:00.466964+0000","last_peered":"2024-04-17T10:29:00.466964+0000","last_clean":"2024-04-17T10:29:00.466964+0000","last_became_active":"2024-04-17T10:28:59.052061+0000","last_became_peered":"2024-04-17T10:28:59.052061+0000","last_unstale":"2024-04-17T10:29:00.466964+0000","last_undegraded":"2024-04-17T10:29:00.466964+0000","last_fullsized":"2024-04-17T10:29:00.466964+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:44.295601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047896+0000","last_change":"2024-04-17T10:28:59.058514+0000","last_active":"2024-04-17T10:29:00.047896+0000","last_peered":"2024-04-17T10:29:00.047896+0000","last_clean":"2024-04-17T10:29:00.047896+0000","last_became_active":"2024-04-17T10:28:59.058208+0000","last_became_peered":"2024-04-17T10:28:59.058208+0000","last_unstale":"2024-04-17T10:29:00.047896+0000","last_undegraded":"2024-04-17T10:29:00.047896+0000","last_fullsized":"2024-04-17T10:29:00.047896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:38.908806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052169+0000","last_change":"2024-04-17T10:28:59.075880+0000","last_active":"2024-04-17T10:29:00.052169+0000","last_peered":"2024-04-17T10:29:00.052169+0000","last_clean":"2024-04-17T10:29:00.052169+0000","last_became_active":"2024-04-17T10:28:59.075443+0000","last_became_peered":"2024-04-17T10:28:59.075443+0000","last_unstale":"2024-04-17T10:29:00.052169+0000","last_undegraded":"2024-04-17T10:29:00.052169+0000","last_fullsized":"2024-04-17T10:29:00.052169+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:09.740067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047636+0000","last_change":"2024-04-17T10:28:59.066076+0000","last_active":"2024-04-17T10:29:00.047636+0000","last_peered":"2024-04-17T10:29:00.047636+0000","last_clean":"2024-04-17T10:29:00.047636+0000","last_became_active":"2024-04-17T10:28:59.065790+0000","last_became_peered":"2024-04-17T10:28:59.065790+0000","last_unstale":"2024-04-17T10:29:00.047636+0000","last_undegraded":"2024-04-17T10:29:00.047636+0000","last_fullsized":"2024-04-17T10:29:00.047636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:03.632248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047545+0000","last_change":"2024-04-17T10:28:59.064041+0000","last_active":"2024-04-17T10:29:00.047545+0000","last_peered":"2024-04-17T10:29:00.047545+0000","last_clean":"2024-04-17T10:29:00.047545+0000","last_became_active":"2024-04-17T10:28:59.063072+0000","last_became_peered":"2024-04-17T10:28:59.063072+0000","last_unstale":"2024-04-17T10:29:00.047545+0000","last_undegraded":"2024-04-17T10:29:00.047545+0000","last_fullsized":"2024-04-17T10:29:00.047545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:25.185570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465728+0000","last_change":"2024-04-17T10:28:59.062782+0000","last_active":"2024-04-17T10:29:00.465728+0000","last_peered":"2024-04-17T10:29:00.465728+0000","last_clean":"2024-04-17T10:29:00.465728+0000","last_became_active":"2024-04-17T10:28:59.062479+0000","last_became_peered":"2024-04-17T10:28:59.062479+0000","last_unstale":"2024-04-17T10:29:00.465728+0000","last_undegraded":"2024-04-17T10:29:00.465728+0000","last_fullsized":"2024-04-17T10:29:00.465728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:25.359261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052356+0000","last_change":"2024-04-17T10:28:59.079637+0000","last_active":"2024-04-17T10:29:00.052356+0000","last_peered":"2024-04-17T10:29:00.052356+0000","last_clean":"2024-04-17T10:29:00.052356+0000","last_became_active":"2024-04-17T10:28:59.079190+0000","last_became_peered":"2024-04-17T10:28:59.079190+0000","last_unstale":"2024-04-17T10:29:00.052356+0000","last_undegraded":"2024-04-17T10:29:00.052356+0000","last_fullsized":"2024-04-17T10:29:00.052356+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:41.817731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052050+0000","last_change":"2024-04-17T10:28:59.078978+0000","last_active":"2024-04-17T10:29:00.052050+0000","last_peered":"2024-04-17T10:29:00.052050+0000","last_clean":"2024-04-17T10:29:00.052050+0000","last_became_active":"2024-04-17T10:28:59.078689+0000","last_became_peered":"2024-04-17T10:28:59.078689+0000","last_unstale":"2024-04-17T10:29:00.052050+0000","last_undegraded":"2024-04-17T10:29:00.052050+0000","last_fullsized":"2024-04-17T10:29:00.052050+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:49:29.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044435+0000","last_change":"2024-04-17T10:28:59.048317+0000","last_active":"2024-04-17T10:29:00.044435+0000","last_peered":"2024-04-17T10:29:00.044435+0000","last_clean":"2024-04-17T10:29:00.044435+0000","last_became_active":"2024-04-17T10:28:59.047390+0000","last_became_peered":"2024-04-17T10:28:59.047390+0000","last_unstale":"2024-04-17T10:29:00.044435+0000","last_undegraded":"2024-04-17T10:29:00.044435+0000","last_fullsized":"2024-04-17T10:29:00.044435+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:58.615173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469595+0000","last_change":"2024-04-17T10:28:59.057401+0000","last_active":"2024-04-17T10:29:00.469595+0000","last_peered":"2024-04-17T10:29:00.469595+0000","last_clean":"2024-04-17T10:29:00.469595+0000","last_became_active":"2024-04-17T10:28:59.056779+0000","last_became_peered":"2024-04-17T10:28:59.056779+0000","last_unstale":"2024-04-17T10:29:00.469595+0000","last_undegraded":"2024-04-17T10:29:00.469595+0000","last_fullsized":"2024-04-17T10:29:00.469595+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:16.453475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466347+0000","last_change":"2024-04-17T10:28:59.066155+0000","last_active":"2024-04-17T10:29:00.466347+0000","last_peered":"2024-04-17T10:29:00.466347+0000","last_clean":"2024-04-17T10:29:00.466347+0000","last_became_active":"2024-04-17T10:28:59.065844+0000","last_became_peered":"2024-04-17T10:28:59.065844+0000","last_unstale":"2024-04-17T10:29:00.466347+0000","last_undegraded":"2024-04-17T10:29:00.466347+0000","last_fullsized":"2024-04-17T10:29:00.466347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:11.622461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051812+0000","last_change":"2024-04-17T10:28:59.071567+0000","last_active":"2024-04-17T10:29:00.051812+0000","last_peered":"2024-04-17T10:29:00.051812+0000","last_clean":"2024-04-17T10:29:00.051812+0000","last_became_active":"2024-04-17T10:28:59.071314+0000","last_became_peered":"2024-04-17T10:28:59.071314+0000","last_unstale":"2024-04-17T10:29:00.051812+0000","last_undegraded":"2024-04-17T10:29:00.051812+0000","last_fullsized":"2024-04-17T10:29:00.051812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:30.871584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.043960+0000","last_change":"2024-04-17T10:28:59.047872+0000","last_active":"2024-04-17T10:29:00.043960+0000","last_peered":"2024-04-17T10:29:00.043960+0000","last_clean":"2024-04-17T10:29:00.043960+0000","last_became_active":"2024-04-17T10:28:59.046338+0000","last_became_peered":"2024-04-17T10:28:59.046338+0000","last_unstale":"2024-04-17T10:29:00.043960+0000","last_undegraded":"2024-04-17T10:29:00.043960+0000","last_fullsized":"2024-04-17T10:29:00.043960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:21.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.045956+0000","last_change":"2024-04-17T10:28:59.079665+0000","last_active":"2024-04-17T10:29:00.045956+0000","last_peered":"2024-04-17T10:29:00.045956+0000","last_clean":"2024-04-17T10:29:00.045956+0000","last_became_active":"2024-04-17T10:28:59.079348+0000","last_became_peered":"2024-04-17T10:28:59.079348+0000","last_unstale":"2024-04-17T10:29:00.045956+0000","last_undegraded":"2024-04-17T10:29:00.045956+0000","last_fullsized":"2024-04-17T10:29:00.045956+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:05.008242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466472+0000","last_change":"2024-04-17T10:28:59.054599+0000","last_active":"2024-04-17T10:29:00.466472+0000","last_peered":"2024-04-17T10:29:00.466472+0000","last_clean":"2024-04-17T10:29:00.466472+0000","last_became_active":"2024-04-17T10:28:59.054286+0000","last_became_peered":"2024-04-17T10:28:59.054286+0000","last_unstale":"2024-04-17T10:29:00.466472+0000","last_undegraded":"2024-04-17T10:29:00.466472+0000","last_fullsized":"2024-04-17T10:29:00.466472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:58.615173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"66'4","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469986+0000","last_change":"2024-04-17T10:28:58.060849+0000","last_active":"2024-04-17T10:29:00.469986+0000","last_peered":"2024-04-17T10:29:00.469986+0000","last_clean":"2024-04-17T10:29:00.469986+0000","last_became_active":"2024-04-17T10:28:58.060557+0000","last_became_peered":"2024-04-17T10:28:58.060557+0000","last_unstale":"2024-04-17T10:29:00.469986+0000","last_undegraded":"2024-04-17T10:29:00.469986+0000","last_fullsized":"2024-04-17T10:29:00.469986+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:20.749646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044486+0000","last_change":"2024-04-17T10:28:59.051480+0000","last_active":"2024-04-17T10:29:00.044486+0000","last_peered":"2024-04-17T10:29:00.044486+0000","last_clean":"2024-04-17T10:29:00.044486+0000","last_became_active":"2024-04-17T10:28:59.050953+0000","last_became_peered":"2024-04-17T10:28:59.050953+0000","last_unstale":"2024-04-17T10:29:00.044486+0000","last_undegraded":"2024-04-17T10:29:00.044486+0000","last_fullsized":"2024-04-17T10:29:00.044486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:08:09.731769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046726+0000","last_change":"2024-04-17T10:28:58.063442+0000","last_active":"2024-04-17T10:29:00.046726+0000","last_peered":"2024-04-17T10:29:00.046726+0000","last_clean":"2024-04-17T10:29:00.046726+0000","last_became_active":"2024-04-17T10:28:58.063171+0000","last_became_peered":"2024-04-17T10:28:58.063171+0000","last_unstale":"2024-04-17T10:29:00.046726+0000","last_undegraded":"2024-04-17T10:29:00.046726+0000","last_fullsized":"2024-04-17T10:29:00.046726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:35:45.288433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046689+0000","last_change":"2024-04-17T10:28:59.076390+0000","last_active":"2024-04-17T10:29:00.046689+0000","last_peered":"2024-04-17T10:29:00.046689+0000","last_clean":"2024-04-17T10:29:00.046689+0000","last_became_active":"2024-04-17T10:28:59.076125+0000","last_became_peered":"2024-04-17T10:28:59.076125+0000","last_unstale":"2024-04-17T10:29:00.046689+0000","last_undegraded":"2024-04-17T10:29:00.046689+0000","last_fullsized":"2024-04-17T10:29:00.046689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:30.871584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046449+0000","last_change":"2024-04-17T10:28:58.058277+0000","last_active":"2024-04-17T10:29:00.046449+0000","last_peered":"2024-04-17T10:29:00.046449+0000","last_clean":"2024-04-17T10:29:00.046449+0000","last_became_active":"2024-04-17T10:28:58.058026+0000","last_became_peered":"2024-04-17T10:28:58.058026+0000","last_unstale":"2024-04-17T10:29:00.046449+0000","last_undegraded":"2024-04-17T10:29:00.046449+0000","last_fullsized":"2024-04-17T10:29:00.046449+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:24.533003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466681+0000","last_change":"2024-04-17T10:28:59.075194+0000","last_active":"2024-04-17T10:29:00.466681+0000","last_peered":"2024-04-17T10:29:00.466681+0000","last_clean":"2024-04-17T10:29:00.466681+0000","last_became_active":"2024-04-17T10:28:59.074590+0000","last_became_peered":"2024-04-17T10:28:59.074590+0000","last_unstale":"2024-04-17T10:29:00.466681+0000","last_undegraded":"2024-04-17T10:29:00.466681+0000","last_fullsized":"2024-04-17T10:29:00.466681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:01.783001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046566+0000","last_change":"2024-04-17T10:28:58.056309+0000","last_active":"2024-04-17T10:29:00.046566+0000","last_peered":"2024-04-17T10:29:00.046566+0000","last_clean":"2024-04-17T10:29:00.046566+0000","last_became_active":"2024-04-17T10:28:58.056021+0000","last_became_peered":"2024-04-17T10:28:58.056021+0000","last_unstale":"2024-04-17T10:29:00.046566+0000","last_undegraded":"2024-04-17T10:29:00.046566+0000","last_fullsized":"2024-04-17T10:29:00.046566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:15.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469023+0000","last_change":"2024-04-17T10:28:59.052426+0000","last_active":"2024-04-17T10:29:00.469023+0000","last_peered":"2024-04-17T10:29:00.469023+0000","last_clean":"2024-04-17T10:29:00.469023+0000","last_became_active":"2024-04-17T10:28:59.052137+0000","last_became_peered":"2024-04-17T10:28:59.052137+0000","last_unstale":"2024-04-17T10:29:00.469023+0000","last_undegraded":"2024-04-17T10:29:00.469023+0000","last_fullsized":"2024-04-17T10:29:00.469023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:56.751994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047818+0000","last_change":"2024-04-17T10:28:58.058572+0000","last_active":"2024-04-17T10:29:00.047818+0000","last_peered":"2024-04-17T10:29:00.047818+0000","last_clean":"2024-04-17T10:29:00.047818+0000","last_became_active":"2024-04-17T10:28:58.058251+0000","last_became_peered":"2024-04-17T10:28:58.058251+0000","last_unstale":"2024-04-17T10:29:00.047818+0000","last_undegraded":"2024-04-17T10:29:00.047818+0000","last_fullsized":"2024-04-17T10:29:00.047818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:10.614299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469514+0000","last_change":"2024-04-17T10:28:59.069257+0000","last_active":"2024-04-17T10:29:00.469514+0000","last_peered":"2024-04-17T10:29:00.469514+0000","last_clean":"2024-04-17T10:29:00.469514+0000","last_became_active":"2024-04-17T10:28:59.068920+0000","last_became_peered":"2024-04-17T10:28:59.068920+0000","last_unstale":"2024-04-17T10:29:00.469514+0000","last_undegraded":"2024-04-17T10:29:00.469514+0000","last_fullsized":"2024-04-17T10:29:00.469514+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:22:47.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047723+0000","last_change":"2024-04-17T10:28:58.052750+0000","last_active":"2024-04-17T10:29:00.047723+0000","last_peered":"2024-04-17T10:29:00.047723+0000","last_clean":"2024-04-17T10:29:00.047723+0000","last_became_active":"2024-04-17T10:28:58.052450+0000","last_became_peered":"2024-04-17T10:28:58.052450+0000","last_unstale":"2024-04-17T10:29:00.047723+0000","last_undegraded":"2024-04-17T10:29:00.047723+0000","last_fullsized":"2024-04-17T10:29:00.047723+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:03.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465734+0000","last_change":"2024-04-17T10:28:59.064446+0000","last_active":"2024-04-17T10:29:00.465734+0000","last_peered":"2024-04-17T10:29:00.465734+0000","last_clean":"2024-04-17T10:29:00.465734+0000","last_became_active":"2024-04-17T10:28:59.064171+0000","last_became_peered":"2024-04-17T10:28:59.064171+0000","last_unstale":"2024-04-17T10:29:00.465734+0000","last_undegraded":"2024-04-17T10:29:00.465734+0000","last_fullsized":"2024-04-17T10:29:00.465734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:21.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047532+0000","last_change":"2024-04-17T10:28:58.052813+0000","last_active":"2024-04-17T10:29:00.047532+0000","last_peered":"2024-04-17T10:29:00.047532+0000","last_clean":"2024-04-17T10:29:00.047532+0000","last_became_active":"2024-04-17T10:28:58.052521+0000","last_became_peered":"2024-04-17T10:28:58.052521+0000","last_unstale":"2024-04-17T10:29:00.047532+0000","last_undegraded":"2024-04-17T10:29:00.047532+0000","last_fullsized":"2024-04-17T10:29:00.047532+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:24.351099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047909+0000","last_change":"2024-04-17T10:28:59.068891+0000","last_active":"2024-04-17T10:29:00.047909+0000","last_peered":"2024-04-17T10:29:00.047909+0000","last_clean":"2024-04-17T10:29:00.047909+0000","last_became_active":"2024-04-17T10:28:59.068602+0000","last_became_peered":"2024-04-17T10:28:59.068602+0000","last_unstale":"2024-04-17T10:29:00.047909+0000","last_undegraded":"2024-04-17T10:29:00.047909+0000","last_fullsized":"2024-04-17T10:29:00.047909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:06.693279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465498+0000","last_change":"2024-04-17T10:28:58.059010+0000","last_active":"2024-04-17T10:29:00.465498+0000","last_peered":"2024-04-17T10:29:00.465498+0000","last_clean":"2024-04-17T10:29:00.465498+0000","last_became_active":"2024-04-17T10:28:58.058730+0000","last_became_peered":"2024-04-17T10:28:58.058730+0000","last_unstale":"2024-04-17T10:29:00.465498+0000","last_undegraded":"2024-04-17T10:29:00.465498+0000","last_fullsized":"2024-04-17T10:29:00.465498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:14:12.873882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469572+0000","last_change":"2024-04-17T10:28:59.057563+0000","last_active":"2024-04-17T10:29:00.469572+0000","last_peered":"2024-04-17T10:29:00.469572+0000","last_clean":"2024-04-17T10:29:00.469572+0000","last_became_active":"2024-04-17T10:28:59.057153+0000","last_became_peered":"2024-04-17T10:28:59.057153+0000","last_unstale":"2024-04-17T10:29:00.469572+0000","last_undegraded":"2024-04-17T10:29:00.469572+0000","last_fullsized":"2024-04-17T10:29:00.469572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:49.692570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466726+0000","last_change":"2024-04-17T10:28:58.062886+0000","last_active":"2024-04-17T10:29:00.466726+0000","last_peered":"2024-04-17T10:29:00.466726+0000","last_clean":"2024-04-17T10:29:00.466726+0000","last_became_active":"2024-04-17T10:28:58.062591+0000","last_became_peered":"2024-04-17T10:28:58.062591+0000","last_unstale":"2024-04-17T10:29:00.466726+0000","last_undegraded":"2024-04-17T10:29:00.466726+0000","last_fullsized":"2024-04-17T10:29:00.466726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:37:24.922495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.048120+0000","last_change":"2024-04-17T10:28:59.068148+0000","last_active":"2024-04-17T10:29:00.048120+0000","last_peered":"2024-04-17T10:29:00.048120+0000","last_clean":"2024-04-17T10:29:00.048120+0000","last_became_active":"2024-04-17T10:28:59.067773+0000","last_became_peered":"2024-04-17T10:28:59.067773+0000","last_unstale":"2024-04-17T10:29:00.048120+0000","last_undegraded":"2024-04-17T10:29:00.048120+0000","last_fullsized":"2024-04-17T10:29:00.048120+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:13:31.141053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044748+0000","last_change":"2024-04-17T10:28:58.055268+0000","last_active":"2024-04-17T10:29:00.044748+0000","last_peered":"2024-04-17T10:29:00.044748+0000","last_clean":"2024-04-17T10:29:00.044748+0000","last_became_active":"2024-04-17T10:28:58.052483+0000","last_became_peered":"2024-04-17T10:28:58.052483+0000","last_unstale":"2024-04-17T10:29:00.044748+0000","last_undegraded":"2024-04-17T10:29:00.044748+0000","last_fullsized":"2024-04-17T10:29:00.044748+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:39.123401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.467018+0000","last_change":"2024-04-17T10:28:59.069930+0000","last_active":"2024-04-17T10:29:00.467018+0000","last_peered":"2024-04-17T10:29:00.467018+0000","last_clean":"2024-04-17T10:29:00.467018+0000","last_became_active":"2024-04-17T10:28:59.069613+0000","last_became_peered":"2024-04-17T10:28:59.069613+0000","last_unstale":"2024-04-17T10:29:00.467018+0000","last_undegraded":"2024-04-17T10:29:00.467018+0000","last_fullsized":"2024-04-17T10:29:00.467018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:31:28.227671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"66'3","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466048+0000","last_change":"2024-04-17T10:28:58.062458+0000","last_active":"2024-04-17T10:29:00.466048+0000","last_peered":"2024-04-17T10:29:00.466048+0000","last_clean":"2024-04-17T10:29:00.466048+0000","last_became_active":"2024-04-17T10:28:58.062162+0000","last_became_peered":"2024-04-17T10:28:58.062162+0000","last_unstale":"2024-04-17T10:29:00.466048+0000","last_undegraded":"2024-04-17T10:29:00.466048+0000","last_fullsized":"2024-04-17T10:29:00.466048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:24.533003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044899+0000","last_change":"2024-04-17T10:28:59.055801+0000","last_active":"2024-04-17T10:29:00.044899+0000","last_peered":"2024-04-17T10:29:00.044899+0000","last_clean":"2024-04-17T10:29:00.044899+0000","last_became_active":"2024-04-17T10:28:59.055525+0000","last_became_peered":"2024-04-17T10:28:59.055525+0000","last_unstale":"2024-04-17T10:29:00.044899+0000","last_undegraded":"2024-04-17T10:29:00.044899+0000","last_fullsized":"2024-04-17T10:29:00.044899+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:25.201497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"66'6","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469628+0000","last_change":"2024-04-17T10:28:58.066220+0000","last_active":"2024-04-17T10:29:00.469628+0000","last_peered":"2024-04-17T10:29:00.469628+0000","last_clean":"2024-04-17T10:29:00.469628+0000","last_became_active":"2024-04-17T10:28:58.065842+0000","last_became_peered":"2024-04-17T10:28:58.065842+0000","last_unstale":"2024-04-17T10:29:00.469628+0000","last_undegraded":"2024-04-17T10:29:00.469628+0000","last_fullsized":"2024-04-17T10:29:00.469628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:20.439434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.043810+0000","last_change":"2024-04-17T10:28:59.059955+0000","last_active":"2024-04-17T10:29:00.043810+0000","last_peered":"2024-04-17T10:29:00.043810+0000","last_clean":"2024-04-17T10:29:00.043810+0000","last_became_active":"2024-04-17T10:28:59.059681+0000","last_became_peered":"2024-04-17T10:28:59.059681+0000","last_unstale":"2024-04-17T10:29:00.043810+0000","last_undegraded":"2024-04-17T10:29:00.043810+0000","last_fullsized":"2024-04-17T10:29:00.043810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:46.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466189+0000","last_change":"2024-04-17T10:28:58.060610+0000","last_active":"2024-04-17T10:29:00.466189+0000","last_peered":"2024-04-17T10:29:00.466189+0000","last_clean":"2024-04-17T10:29:00.466189+0000","last_became_active":"2024-04-17T10:28:58.060319+0000","last_became_peered":"2024-04-17T10:28:58.060319+0000","last_unstale":"2024-04-17T10:29:00.466189+0000","last_undegraded":"2024-04-17T10:29:00.466189+0000","last_fullsized":"2024-04-17T10:29:00.466189+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:57.544495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051899+0000","last_change":"2024-04-17T10:28:59.074660+0000","last_active":"2024-04-17T10:29:00.051899+0000","last_peered":"2024-04-17T10:29:00.051899+0000","last_clean":"2024-04-17T10:29:00.051899+0000","last_became_active":"2024-04-17T10:28:59.074302+0000","last_became_peered":"2024-04-17T10:28:59.074302+0000","last_unstale":"2024-04-17T10:29:00.051899+0000","last_undegraded":"2024-04-17T10:29:00.051899+0000","last_fullsized":"2024-04-17T10:29:00.051899+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:25.359261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051689+0000","last_change":"2024-04-17T10:28:58.065249+0000","last_active":"2024-04-17T10:29:00.051689+0000","last_peered":"2024-04-17T10:29:00.051689+0000","last_clean":"2024-04-17T10:29:00.051689+0000","last_became_active":"2024-04-17T10:28:58.064871+0000","last_became_peered":"2024-04-17T10:28:58.064871+0000","last_unstale":"2024-04-17T10:29:00.051689+0000","last_undegraded":"2024-04-17T10:29:00.051689+0000","last_fullsized":"2024-04-17T10:29:00.051689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:24.533003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465883+0000","last_change":"2024-04-17T10:28:59.052255+0000","last_active":"2024-04-17T10:29:00.465883+0000","last_peered":"2024-04-17T10:29:00.465883+0000","last_clean":"2024-04-17T10:29:00.465883+0000","last_became_active":"2024-04-17T10:28:59.052004+0000","last_became_peered":"2024-04-17T10:28:59.052004+0000","last_unstale":"2024-04-17T10:29:00.465883+0000","last_undegraded":"2024-04-17T10:29:00.465883+0000","last_fullsized":"2024-04-17T10:29:00.465883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:52:31.810508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466050+0000","last_change":"2024-04-17T10:28:58.059092+0000","last_active":"2024-04-17T10:29:00.466050+0000","last_peered":"2024-04-17T10:29:00.466050+0000","last_clean":"2024-04-17T10:29:00.466050+0000","last_became_active":"2024-04-17T10:28:58.058756+0000","last_became_peered":"2024-04-17T10:28:58.058756+0000","last_unstale":"2024-04-17T10:29:00.466050+0000","last_undegraded":"2024-04-17T10:29:00.466050+0000","last_fullsized":"2024-04-17T10:29:00.466050+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:35:45.288433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469444+0000","last_change":"2024-04-17T10:28:59.062191+0000","last_active":"2024-04-17T10:29:00.469444+0000","last_peered":"2024-04-17T10:29:00.469444+0000","last_clean":"2024-04-17T10:29:00.469444+0000","last_became_active":"2024-04-17T10:28:59.061799+0000","last_became_peered":"2024-04-17T10:28:59.061799+0000","last_unstale":"2024-04-17T10:29:00.469444+0000","last_undegraded":"2024-04-17T10:29:00.469444+0000","last_fullsized":"2024-04-17T10:29:00.469444+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:19:16.737073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.043669+0000","last_change":"2024-04-17T10:28:58.054318+0000","last_active":"2024-04-17T10:29:00.043669+0000","last_peered":"2024-04-17T10:29:00.043669+0000","last_clean":"2024-04-17T10:29:00.043669+0000","last_became_active":"2024-04-17T10:28:58.053813+0000","last_became_peered":"2024-04-17T10:28:58.053813+0000","last_unstale":"2024-04-17T10:29:00.043669+0000","last_undegraded":"2024-04-17T10:29:00.043669+0000","last_fullsized":"2024-04-17T10:29:00.043669+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:32.872834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466361+0000","last_change":"2024-04-17T10:28:59.066167+0000","last_active":"2024-04-17T10:29:00.466361+0000","last_peered":"2024-04-17T10:29:00.466361+0000","last_clean":"2024-04-17T10:29:00.466361+0000","last_became_active":"2024-04-17T10:28:59.065920+0000","last_became_peered":"2024-04-17T10:28:59.065920+0000","last_unstale":"2024-04-17T10:29:00.466361+0000","last_undegraded":"2024-04-17T10:29:00.466361+0000","last_fullsized":"2024-04-17T10:29:00.466361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:04.822143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052326+0000","last_change":"2024-04-17T10:28:58.064493+0000","last_active":"2024-04-17T10:29:00.052326+0000","last_peered":"2024-04-17T10:29:00.052326+0000","last_clean":"2024-04-17T10:29:00.052326+0000","last_became_active":"2024-04-17T10:28:58.064214+0000","last_became_peered":"2024-04-17T10:28:58.064214+0000","last_unstale":"2024-04-17T10:29:00.052326+0000","last_undegraded":"2024-04-17T10:29:00.052326+0000","last_fullsized":"2024-04-17T10:29:00.052326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:15.445312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466964+0000","last_change":"2024-04-17T10:28:59.051445+0000","last_active":"2024-04-17T10:29:00.466964+0000","last_peered":"2024-04-17T10:29:00.466964+0000","last_clean":"2024-04-17T10:29:00.466964+0000","last_became_active":"2024-04-17T10:28:59.051050+0000","last_became_peered":"2024-04-17T10:28:59.051050+0000","last_unstale":"2024-04-17T10:29:00.466964+0000","last_undegraded":"2024-04-17T10:29:00.466964+0000","last_fullsized":"2024-04-17T10:29:00.466964+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:33.538063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466374+0000","last_change":"2024-04-17T10:28:58.061859+0000","last_active":"2024-04-17T10:29:00.466374+0000","last_peered":"2024-04-17T10:29:00.466374+0000","last_clean":"2024-04-17T10:29:00.466374+0000","last_became_active":"2024-04-17T10:28:58.060099+0000","last_became_peered":"2024-04-17T10:28:58.060099+0000","last_unstale":"2024-04-17T10:29:00.466374+0000","last_undegraded":"2024-04-17T10:29:00.466374+0000","last_fullsized":"2024-04-17T10:29:00.466374+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:15.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044814+0000","last_change":"2024-04-17T10:28:59.051101+0000","last_active":"2024-04-17T10:29:00.044814+0000","last_peered":"2024-04-17T10:29:00.044814+0000","last_clean":"2024-04-17T10:29:00.044814+0000","last_became_active":"2024-04-17T10:28:59.050825+0000","last_became_peered":"2024-04-17T10:28:59.050825+0000","last_unstale":"2024-04-17T10:29:00.044814+0000","last_undegraded":"2024-04-17T10:29:00.044814+0000","last_fullsized":"2024-04-17T10:29:00.044814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:33.538063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466258+0000","last_change":"2024-04-17T10:28:58.062681+0000","last_active":"2024-04-17T10:29:00.466258+0000","last_peered":"2024-04-17T10:29:00.466258+0000","last_clean":"2024-04-17T10:29:00.466258+0000","last_became_active":"2024-04-17T10:28:58.062352+0000","last_became_peered":"2024-04-17T10:28:58.062352+0000","last_unstale":"2024-04-17T10:29:00.466258+0000","last_undegraded":"2024-04-17T10:29:00.466258+0000","last_fullsized":"2024-04-17T10:29:00.466258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:38:20.077285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466999+0000","last_change":"2024-04-17T10:28:59.066288+0000","last_active":"2024-04-17T10:29:00.466999+0000","last_peered":"2024-04-17T10:29:00.466999+0000","last_clean":"2024-04-17T10:29:00.466999+0000","last_became_active":"2024-04-17T10:28:59.065655+0000","last_became_peered":"2024-04-17T10:28:59.065655+0000","last_unstale":"2024-04-17T10:29:00.466999+0000","last_undegraded":"2024-04-17T10:29:00.466999+0000","last_fullsized":"2024-04-17T10:29:00.466999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:25.201497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047978+0000","last_change":"2024-04-17T10:28:58.057257+0000","last_active":"2024-04-17T10:29:00.047978+0000","last_peered":"2024-04-17T10:29:00.047978+0000","last_clean":"2024-04-17T10:29:00.047978+0000","last_became_active":"2024-04-17T10:28:58.056929+0000","last_became_peered":"2024-04-17T10:28:58.056929+0000","last_unstale":"2024-04-17T10:29:00.047978+0000","last_undegraded":"2024-04-17T10:29:00.047978+0000","last_fullsized":"2024-04-17T10:29:00.047978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:41.562789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.045972+0000","last_change":"2024-04-17T10:28:59.079670+0000","last_active":"2024-04-17T10:29:00.045972+0000","last_peered":"2024-04-17T10:29:00.045972+0000","last_clean":"2024-04-17T10:29:00.045972+0000","last_became_active":"2024-04-17T10:28:59.079358+0000","last_became_peered":"2024-04-17T10:28:59.079358+0000","last_unstale":"2024-04-17T10:29:00.045972+0000","last_undegraded":"2024-04-17T10:29:00.045972+0000","last_fullsized":"2024-04-17T10:29:00.045972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:56:31.170733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046490+0000","last_change":"2024-04-17T10:28:58.058005+0000","last_active":"2024-04-17T10:29:00.046490+0000","last_peered":"2024-04-17T10:29:00.046490+0000","last_clean":"2024-04-17T10:29:00.046490+0000","last_became_active":"2024-04-17T10:28:58.057713+0000","last_became_peered":"2024-04-17T10:28:58.057713+0000","last_unstale":"2024-04-17T10:29:00.046490+0000","last_undegraded":"2024-04-17T10:29:00.046490+0000","last_fullsized":"2024-04-17T10:29:00.046490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:00.650684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051784+0000","last_change":"2024-04-17T10:28:59.079556+0000","last_active":"2024-04-17T10:29:00.051784+0000","last_peered":"2024-04-17T10:29:00.051784+0000","last_clean":"2024-04-17T10:29:00.051784+0000","last_became_active":"2024-04-17T10:28:59.078785+0000","last_became_peered":"2024-04-17T10:28:59.078785+0000","last_unstale":"2024-04-17T10:29:00.051784+0000","last_undegraded":"2024-04-17T10:29:00.051784+0000","last_fullsized":"2024-04-17T10:29:00.051784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:51.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051817+0000","last_change":"2024-04-17T10:28:58.064996+0000","last_active":"2024-04-17T10:29:00.051817+0000","last_peered":"2024-04-17T10:29:00.051817+0000","last_clean":"2024-04-17T10:29:00.051817+0000","last_became_active":"2024-04-17T10:28:58.063329+0000","last_became_peered":"2024-04-17T10:28:58.063329+0000","last_unstale":"2024-04-17T10:29:00.051817+0000","last_undegraded":"2024-04-17T10:29:00.051817+0000","last_fullsized":"2024-04-17T10:29:00.051817+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:57.544495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466533+0000","last_change":"2024-04-17T10:28:59.063852+0000","last_active":"2024-04-17T10:29:00.466533+0000","last_peered":"2024-04-17T10:29:00.466533+0000","last_clean":"2024-04-17T10:29:00.466533+0000","last_became_active":"2024-04-17T10:28:59.062569+0000","last_became_peered":"2024-04-17T10:28:59.062569+0000","last_unstale":"2024-04-17T10:29:00.466533+0000","last_undegraded":"2024-04-17T10:29:00.466533+0000","last_fullsized":"2024-04-17T10:29:00.466533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:08:09.731769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046106+0000","last_change":"2024-04-17T10:28:58.064794+0000","last_active":"2024-04-17T10:29:00.046106+0000","last_peered":"2024-04-17T10:29:00.046106+0000","last_clean":"2024-04-17T10:29:00.046106+0000","last_became_active":"2024-04-17T10:28:58.063301+0000","last_became_peered":"2024-04-17T10:28:58.063301+0000","last_unstale":"2024-04-17T10:29:00.046106+0000","last_undegraded":"2024-04-17T10:29:00.046106+0000","last_fullsized":"2024-04-17T10:29:00.046106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:57.544495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046436+0000","last_change":"2024-04-17T10:28:59.067788+0000","last_active":"2024-04-17T10:29:00.046436+0000","last_peered":"2024-04-17T10:29:00.046436+0000","last_clean":"2024-04-17T10:29:00.046436+0000","last_became_active":"2024-04-17T10:28:59.067496+0000","last_became_peered":"2024-04-17T10:28:59.067496+0000","last_unstale":"2024-04-17T10:29:00.046436+0000","last_undegraded":"2024-04-17T10:29:00.046436+0000","last_fullsized":"2024-04-17T10:29:00.046436+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:47:28.733650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052430+0000","last_change":"2024-04-17T10:28:58.063903+0000","last_active":"2024-04-17T10:29:00.052430+0000","last_peered":"2024-04-17T10:29:00.052430+0000","last_clean":"2024-04-17T10:29:00.052430+0000","last_became_active":"2024-04-17T10:28:58.063611+0000","last_became_peered":"2024-04-17T10:28:58.063611+0000","last_unstale":"2024-04-17T10:29:00.052430+0000","last_undegraded":"2024-04-17T10:29:00.052430+0000","last_fullsized":"2024-04-17T10:29:00.052430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:38.531928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469268+0000","last_change":"2024-04-17T10:28:59.061664+0000","last_active":"2024-04-17T10:29:00.469268+0000","last_peered":"2024-04-17T10:29:00.469268+0000","last_clean":"2024-04-17T10:29:00.469268+0000","last_became_active":"2024-04-17T10:28:59.061414+0000","last_became_peered":"2024-04-17T10:28:59.061414+0000","last_unstale":"2024-04-17T10:29:00.469268+0000","last_undegraded":"2024-04-17T10:29:00.469268+0000","last_fullsized":"2024-04-17T10:29:00.469268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:43:08.395313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052291+0000","last_change":"2024-04-17T10:28:58.060159+0000","last_active":"2024-04-17T10:29:00.052291+0000","last_peered":"2024-04-17T10:29:00.052291+0000","last_clean":"2024-04-17T10:29:00.052291+0000","last_became_active":"2024-04-17T10:28:58.059726+0000","last_became_peered":"2024-04-17T10:28:58.059726+0000","last_unstale":"2024-04-17T10:29:00.052291+0000","last_undegraded":"2024-04-17T10:29:00.052291+0000","last_fullsized":"2024-04-17T10:29:00.052291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:48.684408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052124+0000","last_change":"2024-04-17T10:28:59.073149+0000","last_active":"2024-04-17T10:29:00.052124+0000","last_peered":"2024-04-17T10:29:00.052124+0000","last_clean":"2024-04-17T10:29:00.052124+0000","last_became_active":"2024-04-17T10:28:59.072818+0000","last_became_peered":"2024-04-17T10:28:59.072818+0000","last_unstale":"2024-04-17T10:29:00.052124+0000","last_undegraded":"2024-04-17T10:29:00.052124+0000","last_fullsized":"2024-04-17T10:29:00.052124+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:11.622461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046546+0000","last_change":"2024-04-17T10:28:58.056208+0000","last_active":"2024-04-17T10:29:00.046546+0000","last_peered":"2024-04-17T10:29:00.046546+0000","last_clean":"2024-04-17T10:29:00.046546+0000","last_became_active":"2024-04-17T10:28:58.055861+0000","last_became_peered":"2024-04-17T10:28:58.055861+0000","last_unstale":"2024-04-17T10:29:00.046546+0000","last_undegraded":"2024-04-17T10:29:00.046546+0000","last_fullsized":"2024-04-17T10:29:00.046546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:38:20.077285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.467071+0000","last_change":"2024-04-17T10:28:59.066588+0000","last_active":"2024-04-17T10:29:00.467071+0000","last_peered":"2024-04-17T10:29:00.467071+0000","last_clean":"2024-04-17T10:29:00.467071+0000","last_became_active":"2024-04-17T10:28:59.065998+0000","last_became_peered":"2024-04-17T10:28:59.065998+0000","last_unstale":"2024-04-17T10:29:00.467071+0000","last_undegraded":"2024-04-17T10:29:00.467071+0000","last_fullsized":"2024-04-17T10:29:00.467071+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:54.993259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469648+0000","last_change":"2024-04-17T10:28:58.066633+0000","last_active":"2024-04-17T10:29:00.469648+0000","last_peered":"2024-04-17T10:29:00.469648+0000","last_clean":"2024-04-17T10:29:00.469648+0000","last_became_active":"2024-04-17T10:28:58.066242+0000","last_became_peered":"2024-04-17T10:28:58.066242+0000","last_unstale":"2024-04-17T10:29:00.469648+0000","last_undegraded":"2024-04-17T10:29:00.469648+0000","last_fullsized":"2024-04-17T10:29:00.469648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:45.008213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465559+0000","last_change":"2024-04-17T10:28:59.055789+0000","last_active":"2024-04-17T10:29:00.465559+0000","last_peered":"2024-04-17T10:29:00.465559+0000","last_clean":"2024-04-17T10:29:00.465559+0000","last_became_active":"2024-04-17T10:28:59.055434+0000","last_became_peered":"2024-04-17T10:28:59.055434+0000","last_unstale":"2024-04-17T10:29:00.465559+0000","last_undegraded":"2024-04-17T10:29:00.465559+0000","last_fullsized":"2024-04-17T10:29:00.465559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:46.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466315+0000","last_change":"2024-04-17T10:28:58.063062+0000","last_active":"2024-04-17T10:29:00.466315+0000","last_peered":"2024-04-17T10:29:00.466315+0000","last_clean":"2024-04-17T10:29:00.466315+0000","last_became_active":"2024-04-17T10:28:58.062785+0000","last_became_peered":"2024-04-17T10:28:58.062785+0000","last_unstale":"2024-04-17T10:29:00.466315+0000","last_undegraded":"2024-04-17T10:29:00.466315+0000","last_fullsized":"2024-04-17T10:29:00.466315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:00.650684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044051+0000","last_change":"2024-04-17T10:28:59.054328+0000","last_active":"2024-04-17T10:29:00.044051+0000","last_peered":"2024-04-17T10:29:00.044051+0000","last_clean":"2024-04-17T10:29:00.044051+0000","last_became_active":"2024-04-17T10:28:59.054059+0000","last_became_peered":"2024-04-17T10:28:59.054059+0000","last_unstale":"2024-04-17T10:29:00.044051+0000","last_undegraded":"2024-04-17T10:29:00.044051+0000","last_fullsized":"2024-04-17T10:29:00.044051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:21.757809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047747+0000","last_change":"2024-04-17T10:28:58.053566+0000","last_active":"2024-04-17T10:29:00.047747+0000","last_peered":"2024-04-17T10:29:00.047747+0000","last_clean":"2024-04-17T10:29:00.047747+0000","last_became_active":"2024-04-17T10:28:58.053306+0000","last_became_peered":"2024-04-17T10:28:58.053306+0000","last_unstale":"2024-04-17T10:29:00.047747+0000","last_undegraded":"2024-04-17T10:29:00.047747+0000","last_fullsized":"2024-04-17T10:29:00.047747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:08.731905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469369+0000","last_change":"2024-04-17T10:28:59.059115+0000","last_active":"2024-04-17T10:29:00.469369+0000","last_peered":"2024-04-17T10:29:00.469369+0000","last_clean":"2024-04-17T10:29:00.469369+0000","last_became_active":"2024-04-17T10:28:59.058828+0000","last_became_peered":"2024-04-17T10:28:59.058828+0000","last_unstale":"2024-04-17T10:29:00.469369+0000","last_undegraded":"2024-04-17T10:29:00.469369+0000","last_fullsized":"2024-04-17T10:29:00.469369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:40:28.186710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047578+0000","last_change":"2024-04-17T10:28:58.052819+0000","last_active":"2024-04-17T10:29:00.047578+0000","last_peered":"2024-04-17T10:29:00.047578+0000","last_clean":"2024-04-17T10:29:00.047578+0000","last_became_active":"2024-04-17T10:28:58.052385+0000","last_became_peered":"2024-04-17T10:28:58.052385+0000","last_unstale":"2024-04-17T10:29:00.047578+0000","last_undegraded":"2024-04-17T10:29:00.047578+0000","last_fullsized":"2024-04-17T10:29:00.047578+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:49:28.308279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469188+0000","last_change":"2024-04-17T10:28:59.074122+0000","last_active":"2024-04-17T10:29:00.469188+0000","last_peered":"2024-04-17T10:29:00.469188+0000","last_clean":"2024-04-17T10:29:00.469188+0000","last_became_active":"2024-04-17T10:28:59.073829+0000","last_became_peered":"2024-04-17T10:28:59.073829+0000","last_unstale":"2024-04-17T10:29:00.469188+0000","last_undegraded":"2024-04-17T10:29:00.469188+0000","last_fullsized":"2024-04-17T10:29:00.469188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:09.749258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.468899+0000","last_change":"2024-04-17T10:28:58.062346+0000","last_active":"2024-04-17T10:29:00.468899+0000","last_peered":"2024-04-17T10:29:00.468899+0000","last_clean":"2024-04-17T10:29:00.468899+0000","last_became_active":"2024-04-17T10:28:58.062085+0000","last_became_peered":"2024-04-17T10:28:58.062085+0000","last_unstale":"2024-04-17T10:29:00.468899+0000","last_undegraded":"2024-04-17T10:29:00.468899+0000","last_fullsized":"2024-04-17T10:29:00.468899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:42:22.030013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465878+0000","last_change":"2024-04-17T10:28:59.056041+0000","last_active":"2024-04-17T10:29:00.465878+0000","last_peered":"2024-04-17T10:29:00.465878+0000","last_clean":"2024-04-17T10:29:00.465878+0000","last_became_active":"2024-04-17T10:28:59.055314+0000","last_became_peered":"2024-04-17T10:28:59.055314+0000","last_unstale":"2024-04-17T10:29:00.465878+0000","last_undegraded":"2024-04-17T10:29:00.465878+0000","last_fullsized":"2024-04-17T10:29:00.465878+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:57.733252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047406+0000","last_change":"2024-04-17T10:28:58.057183+0000","last_active":"2024-04-17T10:29:00.047406+0000","last_peered":"2024-04-17T10:29:00.047406+0000","last_clean":"2024-04-17T10:29:00.047406+0000","last_became_active":"2024-04-17T10:28:58.056893+0000","last_became_peered":"2024-04-17T10:28:58.056893+0000","last_unstale":"2024-04-17T10:29:00.047406+0000","last_undegraded":"2024-04-17T10:29:00.047406+0000","last_fullsized":"2024-04-17T10:29:00.047406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:29.863421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":259,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465685+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:29:00.465685+0000","last_peered":"2024-04-17T10:29:00.465685+0000","last_clean":"2024-04-17T10:29:00.465685+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:29:00.465685+0000","last_undegraded":"2024-04-17T10:29:00.465685+0000","last_fullsized":"2024-04-17T10:29:00.465685+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1421312,"data_stored":962230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":815104,"data_stored":371120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":815104,"data_stored":371108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40308,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331532,"statfs":{"total":96636764160,"available":96595488768,"internally_reserved":0,"allocated":823296,"data_stored":372698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36140,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335700,"statfs":{"total":96636764160,"available":96599756800,"internally_reserved":0,"allocated":811008,"data_stored":371086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36132,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335708,"statfs":{"total":96636764160,"available":96599764992,"internally_reserved":0,"allocated":811008,"data_stored":371086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36164,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335676,"statfs":{"total":96636764160,"available":96599732224,"internally_reserved":0,"allocated":839680,"data_stored":373462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58924,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312916,"statfs":{"total":96636764160,"available":96576425984,"internally_reserved":0,"allocated":1404928,"data_stored":961454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:07.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:07.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eee00640 1 Processor -- start 2024-04-17T10:29:07.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eee00640 1 -- start start 2024-04-17T10:29:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02e815dd70 0x7f02e80d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e80d23b0 con 0x7f02e815d990 2024-04-17T10:29:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eee00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e80d2690 con 0x7f02e80b91d0 2024-04-17T10:29:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e80d2fc0 con 0x7f02e815dd70 2024-04-17T10:29:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02ed000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02e815dd70 0x7f02e80d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eda00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f02e815d990 0x7f02e80d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42768/0 (socket says 172.21.15.67:42768) 2024-04-17T10:29:07.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02eda00640 1 -- 172.21.15.67:0/194319483 learned_addr learned my addr 172.21.15.67:0/194319483 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:07.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02ed000640 1 -- 172.21.15.67:0/194319483 >> v1:172.21.15.86:6789/0 conn(0x7f02e80b91d0 legacy=0x7f02e80b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:07.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02ed000640 1 -- 172.21.15.67:0/194319483 >> v1:172.21.15.67:6789/0 conn(0x7f02e815d990 legacy=0x7f02e80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:07.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02ed000640 1 -- 172.21.15.67:0/194319483 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e80d2970 con 0x7f02e815dd70 2024-04-17T10:29:07.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.901+0000 7f02e7e00640 1 -- 172.21.15.67:0/194319483 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829114050 0 0) 0x7f02e80d23b0 con 0x7f02e815d990 2024-04-17T10:29:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02ed000640 1 --2- 172.21.15.67:0/194319483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80d00c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f02d40097c0 tx=0x7f02d401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a125c140eaff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02e7e00640 1 -- 172.21.15.67:0/194319483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40b1070 con 0x7f02e815dd70 2024-04-17T10:29:07.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02e7e00640 1 -- 172.21.15.67:0/194319483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d40ac410 con 0x7f02e815dd70 2024-04-17T10:29:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02e7e00640 1 -- 172.21.15.67:0/194319483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40ac7e0 con 0x7f02e815dd70 2024-04-17T10:29:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 -- 172.21.15.67:0/194319483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 msgr2=0x7f02e80d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 --2- 172.21.15.67:0/194319483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80d00c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f02d40097c0 tx=0x7f02d401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:29:07.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 -- 172.21.15.67:0/194319483 shutdown_connections 2024-04-17T10:29:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 --2- 172.21.15.67:0/194319483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80d00c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 -- 172.21.15.67:0/194319483 >> 172.21.15.67:0/194319483 conn(0x7f02e80b53c0 msgr2=0x7f02e80b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 -- 172.21.15.67:0/194319483 shutdown_connections 2024-04-17T10:29:07.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 -- 172.21.15.67:0/194319483 wait complete. 2024-04-17T10:29:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 Processor -- start 2024-04-17T10:29:07.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.905+0000 7f02eee00640 1 -- start start 2024-04-17T10:29:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e80b91d0 0x7f02e80c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:07.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e815d990 0x7f02e80c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e816ba80 con 0x7f02e815dd70 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e816b900 con 0x7f02e815d990 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e816bc00 con 0x7f02e80b91d0 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80ce040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55812/0 (socket says 172.21.15.67:55812) 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 -- 172.21.15.67:0/2206228821 learned_addr learned my addr 172.21.15.67:0/2206228821 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ed000640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e815d990 0x7f02e80c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eda00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e80b91d0 0x7f02e80c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 -- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e80b91d0 msgr2=0x7f02e80c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e80b91d0 0x7f02e80c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 -- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e815d990 msgr2=0x7f02e80c9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e815d990 0x7f02e80c9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:07.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 -- 172.21.15.67:0/2206228821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e815e190 con 0x7f02e815dd70 2024-04-17T10:29:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02ee400640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80ce040 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c7a0 tx=0x7f02e000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:07.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02e5600640 1 -- 172.21.15.67:0/2206228821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0003b80 con 0x7f02e815dd70 2024-04-17T10:29:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02e5600640 1 -- 172.21.15.67:0/2206228821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02e0003d20 con 0x7f02e815dd70 2024-04-17T10:29:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02e5600640 1 -- 172.21.15.67:0/2206228821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e000aa30 con 0x7f02e815dd70 2024-04-17T10:29:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e80ca030 con 0x7f02e815dd70 2024-04-17T10:29:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e80c8250 con 0x7f02e815dd70 2024-04-17T10:29:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.909+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8074dc0 con 0x7f02e815dd70 2024-04-17T10:29:07.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.913+0000 7f02e5600640 1 -- 172.21.15.67:0/2206228821 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f02e000abd0 con 0x7f02e815dd70 2024-04-17T10:29:07.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.913+0000 7f02e5600640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:07.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.913+0000 7f02e5600640 1 -- 172.21.15.67:0/2206228821 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f02e0084fe0 con 0x7f02e815dd70 2024-04-17T10:29:07.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.913+0000 7f02eda00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:07.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.913+0000 7f02eda00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f02dc007a90 tx=0x7f02dc0079e0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:07.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:07.917+0000 7f02e5600640 1 -- 172.21.15.67:0/2206228821 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f02e815e190 con 0x7f02e815dd70 2024-04-17T10:29:08.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.073+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f02e80b3210 con 0x7f02c8066f60 2024-04-17T10:29:08.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.081+0000 7f02e5600640 1 -- 172.21.15.67:0/2206228821 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252020 (secure 0 0 0) 0x7f02e80b3210 con 0x7f02c8066f60 2024-04-17T10:29:08.077 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:08.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 msgr2=0x7f02c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f02dc007a90 tx=0x7f02dc0079e0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 msgr2=0x7f02e80ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80ce040 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c7a0 tx=0x7f02e000cc60 comp rx=0 tx=0).stop 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 shutdown_connections 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e815dd70 0x7f02e80ce040 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e815d990 0x7f02e80c9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 --2- 172.21.15.67:0/2206228821 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e80b91d0 0x7f02e80c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 >> 172.21.15.67:0/2206228821 conn(0x7f02e80b53c0 msgr2=0x7f02e80b8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:08.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 shutdown_connections 2024-04-17T10:29:08.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.085+0000 7f02eee00640 1 -- 172.21.15.67:0/2206228821 wait complete. 2024-04-17T10:29:08.097 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":192,"stamp":"2024-04-17T10:29:06.192609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":316668,"kb_used_data":7560,"kb_used_omap":12,"kb_used_meta":308979,"kb_avail":754658052,"statfs":{"total":773094113280,"available":772769845248,"internally_reserved":0,"allocated":7741440,"data_stored":4154244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":316395057},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001701"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469935+0000","last_change":"2024-04-17T10:28:59.073293+0000","last_active":"2024-04-17T10:29:00.469935+0000","last_peered":"2024-04-17T10:29:00.469935+0000","last_clean":"2024-04-17T10:29:00.469935+0000","last_became_active":"2024-04-17T10:28:59.072979+0000","last_became_peered":"2024-04-17T10:28:59.072979+0000","last_unstale":"2024-04-17T10:29:00.469935+0000","last_undegraded":"2024-04-17T10:29:00.469935+0000","last_fullsized":"2024-04-17T10:29:00.469935+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:39.540090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469483+0000","last_change":"2024-04-17T10:28:59.069368+0000","last_active":"2024-04-17T10:29:00.469483+0000","last_peered":"2024-04-17T10:29:00.469483+0000","last_clean":"2024-04-17T10:29:00.469483+0000","last_became_active":"2024-04-17T10:28:59.069079+0000","last_became_peered":"2024-04-17T10:28:59.069079+0000","last_unstale":"2024-04-17T10:29:00.469483+0000","last_undegraded":"2024-04-17T10:29:00.469483+0000","last_fullsized":"2024-04-17T10:29:00.469483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:27:23.690447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466483+0000","last_change":"2024-04-17T10:28:59.075137+0000","last_active":"2024-04-17T10:29:00.466483+0000","last_peered":"2024-04-17T10:29:00.466483+0000","last_clean":"2024-04-17T10:29:00.466483+0000","last_became_active":"2024-04-17T10:28:59.072221+0000","last_became_peered":"2024-04-17T10:28:59.072221+0000","last_unstale":"2024-04-17T10:29:00.466483+0000","last_undegraded":"2024-04-17T10:29:00.466483+0000","last_fullsized":"2024-04-17T10:29:00.466483+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:41.817731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044415+0000","last_change":"2024-04-17T10:28:59.061148+0000","last_active":"2024-04-17T10:29:00.044415+0000","last_peered":"2024-04-17T10:29:00.044415+0000","last_clean":"2024-04-17T10:29:00.044415+0000","last_became_active":"2024-04-17T10:28:59.057038+0000","last_became_peered":"2024-04-17T10:28:59.057038+0000","last_unstale":"2024-04-17T10:29:00.044415+0000","last_undegraded":"2024-04-17T10:29:00.044415+0000","last_fullsized":"2024-04-17T10:29:00.044415+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:52:31.810508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465803+0000","last_change":"2024-04-17T10:28:59.052912+0000","last_active":"2024-04-17T10:29:00.465803+0000","last_peered":"2024-04-17T10:29:00.465803+0000","last_clean":"2024-04-17T10:29:00.465803+0000","last_became_active":"2024-04-17T10:28:59.052642+0000","last_became_peered":"2024-04-17T10:28:59.052642+0000","last_unstale":"2024-04-17T10:29:00.465803+0000","last_undegraded":"2024-04-17T10:29:00.465803+0000","last_fullsized":"2024-04-17T10:29:00.465803+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:21.757809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046249+0000","last_change":"2024-04-17T10:28:59.065923+0000","last_active":"2024-04-17T10:29:00.046249+0000","last_peered":"2024-04-17T10:29:00.046249+0000","last_clean":"2024-04-17T10:29:00.046249+0000","last_became_active":"2024-04-17T10:28:59.065640+0000","last_became_peered":"2024-04-17T10:28:59.065640+0000","last_unstale":"2024-04-17T10:29:00.046249+0000","last_undegraded":"2024-04-17T10:29:00.046249+0000","last_fullsized":"2024-04-17T10:29:00.046249+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:51.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.468938+0000","last_change":"2024-04-17T10:28:59.062726+0000","last_active":"2024-04-17T10:29:00.468938+0000","last_peered":"2024-04-17T10:29:00.468938+0000","last_clean":"2024-04-17T10:29:00.468938+0000","last_became_active":"2024-04-17T10:28:59.062418+0000","last_became_peered":"2024-04-17T10:28:59.062418+0000","last_unstale":"2024-04-17T10:29:00.468938+0000","last_undegraded":"2024-04-17T10:29:00.468938+0000","last_fullsized":"2024-04-17T10:29:00.468938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:37:17.066624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051805+0000","last_change":"2024-04-17T10:28:59.070732+0000","last_active":"2024-04-17T10:29:00.051805+0000","last_peered":"2024-04-17T10:29:00.051805+0000","last_clean":"2024-04-17T10:29:00.051805+0000","last_became_active":"2024-04-17T10:28:59.070464+0000","last_became_peered":"2024-04-17T10:28:59.070464+0000","last_unstale":"2024-04-17T10:29:00.051805+0000","last_undegraded":"2024-04-17T10:29:00.051805+0000","last_fullsized":"2024-04-17T10:29:00.051805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:47:28.733650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466787+0000","last_change":"2024-04-17T10:28:59.064898+0000","last_active":"2024-04-17T10:29:00.466787+0000","last_peered":"2024-04-17T10:29:00.466787+0000","last_clean":"2024-04-17T10:29:00.466787+0000","last_became_active":"2024-04-17T10:28:59.064563+0000","last_became_peered":"2024-04-17T10:28:59.064563+0000","last_unstale":"2024-04-17T10:29:00.466787+0000","last_undegraded":"2024-04-17T10:29:00.466787+0000","last_fullsized":"2024-04-17T10:29:00.466787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:59.527524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466316+0000","last_change":"2024-04-17T10:28:59.070664+0000","last_active":"2024-04-17T10:29:00.466316+0000","last_peered":"2024-04-17T10:29:00.466316+0000","last_clean":"2024-04-17T10:29:00.466316+0000","last_became_active":"2024-04-17T10:28:59.066601+0000","last_became_peered":"2024-04-17T10:28:59.066601+0000","last_unstale":"2024-04-17T10:29:00.466316+0000","last_undegraded":"2024-04-17T10:29:00.466316+0000","last_fullsized":"2024-04-17T10:29:00.466316+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:42.570951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044287+0000","last_change":"2024-04-17T10:28:59.052440+0000","last_active":"2024-04-17T10:29:00.044287+0000","last_peered":"2024-04-17T10:29:00.044287+0000","last_clean":"2024-04-17T10:29:00.044287+0000","last_became_active":"2024-04-17T10:28:59.052113+0000","last_became_peered":"2024-04-17T10:28:59.052113+0000","last_unstale":"2024-04-17T10:29:00.044287+0000","last_undegraded":"2024-04-17T10:29:00.044287+0000","last_fullsized":"2024-04-17T10:29:00.044287+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:57.733252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052126+0000","last_change":"2024-04-17T10:28:59.078195+0000","last_active":"2024-04-17T10:29:00.052126+0000","last_peered":"2024-04-17T10:29:00.052126+0000","last_clean":"2024-04-17T10:29:00.052126+0000","last_became_active":"2024-04-17T10:28:59.077897+0000","last_became_peered":"2024-04-17T10:28:59.077897+0000","last_unstale":"2024-04-17T10:29:00.052126+0000","last_undegraded":"2024-04-17T10:29:00.052126+0000","last_fullsized":"2024-04-17T10:29:00.052126+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:04.822143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052059+0000","last_change":"2024-04-17T10:28:59.072616+0000","last_active":"2024-04-17T10:29:00.052059+0000","last_peered":"2024-04-17T10:29:00.052059+0000","last_clean":"2024-04-17T10:29:00.052059+0000","last_became_active":"2024-04-17T10:28:59.072141+0000","last_became_peered":"2024-04-17T10:28:59.072141+0000","last_unstale":"2024-04-17T10:29:00.052059+0000","last_undegraded":"2024-04-17T10:29:00.052059+0000","last_fullsized":"2024-04-17T10:29:00.052059+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:42.570951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466463+0000","last_change":"2024-04-17T10:28:59.072969+0000","last_active":"2024-04-17T10:29:00.466463+0000","last_peered":"2024-04-17T10:29:00.466463+0000","last_clean":"2024-04-17T10:29:00.466463+0000","last_became_active":"2024-04-17T10:28:59.072408+0000","last_became_peered":"2024-04-17T10:28:59.072408+0000","last_unstale":"2024-04-17T10:29:00.466463+0000","last_undegraded":"2024-04-17T10:29:00.466463+0000","last_fullsized":"2024-04-17T10:29:00.466463+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:09.740067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465996+0000","last_change":"2024-04-17T10:28:59.074659+0000","last_active":"2024-04-17T10:29:00.465996+0000","last_peered":"2024-04-17T10:29:00.465996+0000","last_clean":"2024-04-17T10:29:00.465996+0000","last_became_active":"2024-04-17T10:28:59.074389+0000","last_became_peered":"2024-04-17T10:28:59.074389+0000","last_unstale":"2024-04-17T10:29:00.465996+0000","last_undegraded":"2024-04-17T10:29:00.465996+0000","last_fullsized":"2024-04-17T10:29:00.465996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:49:29.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.468881+0000","last_change":"2024-04-17T10:28:59.074052+0000","last_active":"2024-04-17T10:29:00.468881+0000","last_peered":"2024-04-17T10:29:00.468881+0000","last_clean":"2024-04-17T10:29:00.468881+0000","last_became_active":"2024-04-17T10:28:59.073683+0000","last_became_peered":"2024-04-17T10:28:59.073683+0000","last_unstale":"2024-04-17T10:29:00.468881+0000","last_undegraded":"2024-04-17T10:29:00.468881+0000","last_fullsized":"2024-04-17T10:29:00.468881+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:51:28.760315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469998+0000","last_change":"2024-04-17T10:28:59.065676+0000","last_active":"2024-04-17T10:29:00.469998+0000","last_peered":"2024-04-17T10:29:00.469998+0000","last_clean":"2024-04-17T10:29:00.469998+0000","last_became_active":"2024-04-17T10:28:59.065399+0000","last_became_peered":"2024-04-17T10:28:59.065399+0000","last_unstale":"2024-04-17T10:29:00.469998+0000","last_undegraded":"2024-04-17T10:29:00.469998+0000","last_fullsized":"2024-04-17T10:29:00.469998+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:25.541166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044699+0000","last_change":"2024-04-17T10:28:59.063981+0000","last_active":"2024-04-17T10:29:00.044699+0000","last_peered":"2024-04-17T10:29:00.044699+0000","last_clean":"2024-04-17T10:29:00.044699+0000","last_became_active":"2024-04-17T10:28:59.063472+0000","last_became_peered":"2024-04-17T10:28:59.063472+0000","last_unstale":"2024-04-17T10:29:00.044699+0000","last_undegraded":"2024-04-17T10:29:00.044699+0000","last_fullsized":"2024-04-17T10:29:00.044699+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:44.295601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466964+0000","last_change":"2024-04-17T10:28:59.052390+0000","last_active":"2024-04-17T10:29:00.466964+0000","last_peered":"2024-04-17T10:29:00.466964+0000","last_clean":"2024-04-17T10:29:00.466964+0000","last_became_active":"2024-04-17T10:28:59.052061+0000","last_became_peered":"2024-04-17T10:28:59.052061+0000","last_unstale":"2024-04-17T10:29:00.466964+0000","last_undegraded":"2024-04-17T10:29:00.466964+0000","last_fullsized":"2024-04-17T10:29:00.466964+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:44.295601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047896+0000","last_change":"2024-04-17T10:28:59.058514+0000","last_active":"2024-04-17T10:29:00.047896+0000","last_peered":"2024-04-17T10:29:00.047896+0000","last_clean":"2024-04-17T10:29:00.047896+0000","last_became_active":"2024-04-17T10:28:59.058208+0000","last_became_peered":"2024-04-17T10:28:59.058208+0000","last_unstale":"2024-04-17T10:29:00.047896+0000","last_undegraded":"2024-04-17T10:29:00.047896+0000","last_fullsized":"2024-04-17T10:29:00.047896+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:38.908806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052169+0000","last_change":"2024-04-17T10:28:59.075880+0000","last_active":"2024-04-17T10:29:00.052169+0000","last_peered":"2024-04-17T10:29:00.052169+0000","last_clean":"2024-04-17T10:29:00.052169+0000","last_became_active":"2024-04-17T10:28:59.075443+0000","last_became_peered":"2024-04-17T10:28:59.075443+0000","last_unstale":"2024-04-17T10:29:00.052169+0000","last_undegraded":"2024-04-17T10:29:00.052169+0000","last_fullsized":"2024-04-17T10:29:00.052169+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:09.740067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047636+0000","last_change":"2024-04-17T10:28:59.066076+0000","last_active":"2024-04-17T10:29:00.047636+0000","last_peered":"2024-04-17T10:29:00.047636+0000","last_clean":"2024-04-17T10:29:00.047636+0000","last_became_active":"2024-04-17T10:28:59.065790+0000","last_became_peered":"2024-04-17T10:28:59.065790+0000","last_unstale":"2024-04-17T10:29:00.047636+0000","last_undegraded":"2024-04-17T10:29:00.047636+0000","last_fullsized":"2024-04-17T10:29:00.047636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:03.632248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047545+0000","last_change":"2024-04-17T10:28:59.064041+0000","last_active":"2024-04-17T10:29:00.047545+0000","last_peered":"2024-04-17T10:29:00.047545+0000","last_clean":"2024-04-17T10:29:00.047545+0000","last_became_active":"2024-04-17T10:28:59.063072+0000","last_became_peered":"2024-04-17T10:28:59.063072+0000","last_unstale":"2024-04-17T10:29:00.047545+0000","last_undegraded":"2024-04-17T10:29:00.047545+0000","last_fullsized":"2024-04-17T10:29:00.047545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:25.185570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465728+0000","last_change":"2024-04-17T10:28:59.062782+0000","last_active":"2024-04-17T10:29:00.465728+0000","last_peered":"2024-04-17T10:29:00.465728+0000","last_clean":"2024-04-17T10:29:00.465728+0000","last_became_active":"2024-04-17T10:28:59.062479+0000","last_became_peered":"2024-04-17T10:28:59.062479+0000","last_unstale":"2024-04-17T10:29:00.465728+0000","last_undegraded":"2024-04-17T10:29:00.465728+0000","last_fullsized":"2024-04-17T10:29:00.465728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:25.359261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052356+0000","last_change":"2024-04-17T10:28:59.079637+0000","last_active":"2024-04-17T10:29:00.052356+0000","last_peered":"2024-04-17T10:29:00.052356+0000","last_clean":"2024-04-17T10:29:00.052356+0000","last_became_active":"2024-04-17T10:28:59.079190+0000","last_became_peered":"2024-04-17T10:28:59.079190+0000","last_unstale":"2024-04-17T10:29:00.052356+0000","last_undegraded":"2024-04-17T10:29:00.052356+0000","last_fullsized":"2024-04-17T10:29:00.052356+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:41.817731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052050+0000","last_change":"2024-04-17T10:28:59.078978+0000","last_active":"2024-04-17T10:29:00.052050+0000","last_peered":"2024-04-17T10:29:00.052050+0000","last_clean":"2024-04-17T10:29:00.052050+0000","last_became_active":"2024-04-17T10:28:59.078689+0000","last_became_peered":"2024-04-17T10:28:59.078689+0000","last_unstale":"2024-04-17T10:29:00.052050+0000","last_undegraded":"2024-04-17T10:29:00.052050+0000","last_fullsized":"2024-04-17T10:29:00.052050+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:49:29.316441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044435+0000","last_change":"2024-04-17T10:28:59.048317+0000","last_active":"2024-04-17T10:29:00.044435+0000","last_peered":"2024-04-17T10:29:00.044435+0000","last_clean":"2024-04-17T10:29:00.044435+0000","last_became_active":"2024-04-17T10:28:59.047390+0000","last_became_peered":"2024-04-17T10:28:59.047390+0000","last_unstale":"2024-04-17T10:29:00.044435+0000","last_undegraded":"2024-04-17T10:29:00.044435+0000","last_fullsized":"2024-04-17T10:29:00.044435+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:58.615173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469595+0000","last_change":"2024-04-17T10:28:59.057401+0000","last_active":"2024-04-17T10:29:00.469595+0000","last_peered":"2024-04-17T10:29:00.469595+0000","last_clean":"2024-04-17T10:29:00.469595+0000","last_became_active":"2024-04-17T10:28:59.056779+0000","last_became_peered":"2024-04-17T10:28:59.056779+0000","last_unstale":"2024-04-17T10:29:00.469595+0000","last_undegraded":"2024-04-17T10:29:00.469595+0000","last_fullsized":"2024-04-17T10:29:00.469595+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:16.453475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466347+0000","last_change":"2024-04-17T10:28:59.066155+0000","last_active":"2024-04-17T10:29:00.466347+0000","last_peered":"2024-04-17T10:29:00.466347+0000","last_clean":"2024-04-17T10:29:00.466347+0000","last_became_active":"2024-04-17T10:28:59.065844+0000","last_became_peered":"2024-04-17T10:28:59.065844+0000","last_unstale":"2024-04-17T10:29:00.466347+0000","last_undegraded":"2024-04-17T10:29:00.466347+0000","last_fullsized":"2024-04-17T10:29:00.466347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:11.622461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051812+0000","last_change":"2024-04-17T10:28:59.071567+0000","last_active":"2024-04-17T10:29:00.051812+0000","last_peered":"2024-04-17T10:29:00.051812+0000","last_clean":"2024-04-17T10:29:00.051812+0000","last_became_active":"2024-04-17T10:28:59.071314+0000","last_became_peered":"2024-04-17T10:28:59.071314+0000","last_unstale":"2024-04-17T10:29:00.051812+0000","last_undegraded":"2024-04-17T10:29:00.051812+0000","last_fullsized":"2024-04-17T10:29:00.051812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:30.871584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.043960+0000","last_change":"2024-04-17T10:28:59.047872+0000","last_active":"2024-04-17T10:29:00.043960+0000","last_peered":"2024-04-17T10:29:00.043960+0000","last_clean":"2024-04-17T10:29:00.043960+0000","last_became_active":"2024-04-17T10:28:59.046338+0000","last_became_peered":"2024-04-17T10:28:59.046338+0000","last_unstale":"2024-04-17T10:29:00.043960+0000","last_undegraded":"2024-04-17T10:29:00.043960+0000","last_fullsized":"2024-04-17T10:29:00.043960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:21.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.045956+0000","last_change":"2024-04-17T10:28:59.079665+0000","last_active":"2024-04-17T10:29:00.045956+0000","last_peered":"2024-04-17T10:29:00.045956+0000","last_clean":"2024-04-17T10:29:00.045956+0000","last_became_active":"2024-04-17T10:28:59.079348+0000","last_became_peered":"2024-04-17T10:28:59.079348+0000","last_unstale":"2024-04-17T10:29:00.045956+0000","last_undegraded":"2024-04-17T10:29:00.045956+0000","last_fullsized":"2024-04-17T10:29:00.045956+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:05.008242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466472+0000","last_change":"2024-04-17T10:28:59.054599+0000","last_active":"2024-04-17T10:29:00.466472+0000","last_peered":"2024-04-17T10:29:00.466472+0000","last_clean":"2024-04-17T10:29:00.466472+0000","last_became_active":"2024-04-17T10:28:59.054286+0000","last_became_peered":"2024-04-17T10:28:59.054286+0000","last_unstale":"2024-04-17T10:29:00.466472+0000","last_undegraded":"2024-04-17T10:29:00.466472+0000","last_fullsized":"2024-04-17T10:29:00.466472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:58.615173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"66'4","reported_seq":18,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469986+0000","last_change":"2024-04-17T10:28:58.060849+0000","last_active":"2024-04-17T10:29:00.469986+0000","last_peered":"2024-04-17T10:29:00.469986+0000","last_clean":"2024-04-17T10:29:00.469986+0000","last_became_active":"2024-04-17T10:28:58.060557+0000","last_became_peered":"2024-04-17T10:28:58.060557+0000","last_unstale":"2024-04-17T10:29:00.469986+0000","last_undegraded":"2024-04-17T10:29:00.469986+0000","last_fullsized":"2024-04-17T10:29:00.469986+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:20.749646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044486+0000","last_change":"2024-04-17T10:28:59.051480+0000","last_active":"2024-04-17T10:29:00.044486+0000","last_peered":"2024-04-17T10:29:00.044486+0000","last_clean":"2024-04-17T10:29:00.044486+0000","last_became_active":"2024-04-17T10:28:59.050953+0000","last_became_peered":"2024-04-17T10:28:59.050953+0000","last_unstale":"2024-04-17T10:29:00.044486+0000","last_undegraded":"2024-04-17T10:29:00.044486+0000","last_fullsized":"2024-04-17T10:29:00.044486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:08:09.731769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046726+0000","last_change":"2024-04-17T10:28:58.063442+0000","last_active":"2024-04-17T10:29:00.046726+0000","last_peered":"2024-04-17T10:29:00.046726+0000","last_clean":"2024-04-17T10:29:00.046726+0000","last_became_active":"2024-04-17T10:28:58.063171+0000","last_became_peered":"2024-04-17T10:28:58.063171+0000","last_unstale":"2024-04-17T10:29:00.046726+0000","last_undegraded":"2024-04-17T10:29:00.046726+0000","last_fullsized":"2024-04-17T10:29:00.046726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:35:45.288433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046689+0000","last_change":"2024-04-17T10:28:59.076390+0000","last_active":"2024-04-17T10:29:00.046689+0000","last_peered":"2024-04-17T10:29:00.046689+0000","last_clean":"2024-04-17T10:29:00.046689+0000","last_became_active":"2024-04-17T10:28:59.076125+0000","last_became_peered":"2024-04-17T10:28:59.076125+0000","last_unstale":"2024-04-17T10:29:00.046689+0000","last_undegraded":"2024-04-17T10:29:00.046689+0000","last_fullsized":"2024-04-17T10:29:00.046689+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:30.871584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046449+0000","last_change":"2024-04-17T10:28:58.058277+0000","last_active":"2024-04-17T10:29:00.046449+0000","last_peered":"2024-04-17T10:29:00.046449+0000","last_clean":"2024-04-17T10:29:00.046449+0000","last_became_active":"2024-04-17T10:28:58.058026+0000","last_became_peered":"2024-04-17T10:28:58.058026+0000","last_unstale":"2024-04-17T10:29:00.046449+0000","last_undegraded":"2024-04-17T10:29:00.046449+0000","last_fullsized":"2024-04-17T10:29:00.046449+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:24.533003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466681+0000","last_change":"2024-04-17T10:28:59.075194+0000","last_active":"2024-04-17T10:29:00.466681+0000","last_peered":"2024-04-17T10:29:00.466681+0000","last_clean":"2024-04-17T10:29:00.466681+0000","last_became_active":"2024-04-17T10:28:59.074590+0000","last_became_peered":"2024-04-17T10:28:59.074590+0000","last_unstale":"2024-04-17T10:29:00.466681+0000","last_undegraded":"2024-04-17T10:29:00.466681+0000","last_fullsized":"2024-04-17T10:29:00.466681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:01.783001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046566+0000","last_change":"2024-04-17T10:28:58.056309+0000","last_active":"2024-04-17T10:29:00.046566+0000","last_peered":"2024-04-17T10:29:00.046566+0000","last_clean":"2024-04-17T10:29:00.046566+0000","last_became_active":"2024-04-17T10:28:58.056021+0000","last_became_peered":"2024-04-17T10:28:58.056021+0000","last_unstale":"2024-04-17T10:29:00.046566+0000","last_undegraded":"2024-04-17T10:29:00.046566+0000","last_fullsized":"2024-04-17T10:29:00.046566+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:15.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469023+0000","last_change":"2024-04-17T10:28:59.052426+0000","last_active":"2024-04-17T10:29:00.469023+0000","last_peered":"2024-04-17T10:29:00.469023+0000","last_clean":"2024-04-17T10:29:00.469023+0000","last_became_active":"2024-04-17T10:28:59.052137+0000","last_became_peered":"2024-04-17T10:28:59.052137+0000","last_unstale":"2024-04-17T10:29:00.469023+0000","last_undegraded":"2024-04-17T10:29:00.469023+0000","last_fullsized":"2024-04-17T10:29:00.469023+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:56.751994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047818+0000","last_change":"2024-04-17T10:28:58.058572+0000","last_active":"2024-04-17T10:29:00.047818+0000","last_peered":"2024-04-17T10:29:00.047818+0000","last_clean":"2024-04-17T10:29:00.047818+0000","last_became_active":"2024-04-17T10:28:58.058251+0000","last_became_peered":"2024-04-17T10:28:58.058251+0000","last_unstale":"2024-04-17T10:29:00.047818+0000","last_undegraded":"2024-04-17T10:29:00.047818+0000","last_fullsized":"2024-04-17T10:29:00.047818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:10.614299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469514+0000","last_change":"2024-04-17T10:28:59.069257+0000","last_active":"2024-04-17T10:29:00.469514+0000","last_peered":"2024-04-17T10:29:00.469514+0000","last_clean":"2024-04-17T10:29:00.469514+0000","last_became_active":"2024-04-17T10:28:59.068920+0000","last_became_peered":"2024-04-17T10:28:59.068920+0000","last_unstale":"2024-04-17T10:29:00.469514+0000","last_undegraded":"2024-04-17T10:29:00.469514+0000","last_fullsized":"2024-04-17T10:29:00.469514+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:22:47.819838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047723+0000","last_change":"2024-04-17T10:28:58.052750+0000","last_active":"2024-04-17T10:29:00.047723+0000","last_peered":"2024-04-17T10:29:00.047723+0000","last_clean":"2024-04-17T10:29:00.047723+0000","last_became_active":"2024-04-17T10:28:58.052450+0000","last_became_peered":"2024-04-17T10:28:58.052450+0000","last_unstale":"2024-04-17T10:29:00.047723+0000","last_undegraded":"2024-04-17T10:29:00.047723+0000","last_fullsized":"2024-04-17T10:29:00.047723+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:03.813980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465734+0000","last_change":"2024-04-17T10:28:59.064446+0000","last_active":"2024-04-17T10:29:00.465734+0000","last_peered":"2024-04-17T10:29:00.465734+0000","last_clean":"2024-04-17T10:29:00.465734+0000","last_became_active":"2024-04-17T10:28:59.064171+0000","last_became_peered":"2024-04-17T10:28:59.064171+0000","last_unstale":"2024-04-17T10:29:00.465734+0000","last_undegraded":"2024-04-17T10:29:00.465734+0000","last_fullsized":"2024-04-17T10:29:00.465734+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:21.447596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047532+0000","last_change":"2024-04-17T10:28:58.052813+0000","last_active":"2024-04-17T10:29:00.047532+0000","last_peered":"2024-04-17T10:29:00.047532+0000","last_clean":"2024-04-17T10:29:00.047532+0000","last_became_active":"2024-04-17T10:28:58.052521+0000","last_became_peered":"2024-04-17T10:28:58.052521+0000","last_unstale":"2024-04-17T10:29:00.047532+0000","last_undegraded":"2024-04-17T10:29:00.047532+0000","last_fullsized":"2024-04-17T10:29:00.047532+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:24.351099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047909+0000","last_change":"2024-04-17T10:28:59.068891+0000","last_active":"2024-04-17T10:29:00.047909+0000","last_peered":"2024-04-17T10:29:00.047909+0000","last_clean":"2024-04-17T10:29:00.047909+0000","last_became_active":"2024-04-17T10:28:59.068602+0000","last_became_peered":"2024-04-17T10:28:59.068602+0000","last_unstale":"2024-04-17T10:29:00.047909+0000","last_undegraded":"2024-04-17T10:29:00.047909+0000","last_fullsized":"2024-04-17T10:29:00.047909+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:06.693279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465498+0000","last_change":"2024-04-17T10:28:58.059010+0000","last_active":"2024-04-17T10:29:00.465498+0000","last_peered":"2024-04-17T10:29:00.465498+0000","last_clean":"2024-04-17T10:29:00.465498+0000","last_became_active":"2024-04-17T10:28:58.058730+0000","last_became_peered":"2024-04-17T10:28:58.058730+0000","last_unstale":"2024-04-17T10:29:00.465498+0000","last_undegraded":"2024-04-17T10:29:00.465498+0000","last_fullsized":"2024-04-17T10:29:00.465498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:14:12.873882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469572+0000","last_change":"2024-04-17T10:28:59.057563+0000","last_active":"2024-04-17T10:29:00.469572+0000","last_peered":"2024-04-17T10:29:00.469572+0000","last_clean":"2024-04-17T10:29:00.469572+0000","last_became_active":"2024-04-17T10:28:59.057153+0000","last_became_peered":"2024-04-17T10:28:59.057153+0000","last_unstale":"2024-04-17T10:29:00.469572+0000","last_undegraded":"2024-04-17T10:29:00.469572+0000","last_fullsized":"2024-04-17T10:29:00.469572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:49.692570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466726+0000","last_change":"2024-04-17T10:28:58.062886+0000","last_active":"2024-04-17T10:29:00.466726+0000","last_peered":"2024-04-17T10:29:00.466726+0000","last_clean":"2024-04-17T10:29:00.466726+0000","last_became_active":"2024-04-17T10:28:58.062591+0000","last_became_peered":"2024-04-17T10:28:58.062591+0000","last_unstale":"2024-04-17T10:29:00.466726+0000","last_undegraded":"2024-04-17T10:29:00.466726+0000","last_fullsized":"2024-04-17T10:29:00.466726+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:37:24.922495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.048120+0000","last_change":"2024-04-17T10:28:59.068148+0000","last_active":"2024-04-17T10:29:00.048120+0000","last_peered":"2024-04-17T10:29:00.048120+0000","last_clean":"2024-04-17T10:29:00.048120+0000","last_became_active":"2024-04-17T10:28:59.067773+0000","last_became_peered":"2024-04-17T10:28:59.067773+0000","last_unstale":"2024-04-17T10:29:00.048120+0000","last_undegraded":"2024-04-17T10:29:00.048120+0000","last_fullsized":"2024-04-17T10:29:00.048120+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:13:31.141053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044748+0000","last_change":"2024-04-17T10:28:58.055268+0000","last_active":"2024-04-17T10:29:00.044748+0000","last_peered":"2024-04-17T10:29:00.044748+0000","last_clean":"2024-04-17T10:29:00.044748+0000","last_became_active":"2024-04-17T10:28:58.052483+0000","last_became_peered":"2024-04-17T10:28:58.052483+0000","last_unstale":"2024-04-17T10:29:00.044748+0000","last_undegraded":"2024-04-17T10:29:00.044748+0000","last_fullsized":"2024-04-17T10:29:00.044748+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:41:39.123401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.467018+0000","last_change":"2024-04-17T10:28:59.069930+0000","last_active":"2024-04-17T10:29:00.467018+0000","last_peered":"2024-04-17T10:29:00.467018+0000","last_clean":"2024-04-17T10:29:00.467018+0000","last_became_active":"2024-04-17T10:28:59.069613+0000","last_became_peered":"2024-04-17T10:28:59.069613+0000","last_unstale":"2024-04-17T10:29:00.467018+0000","last_undegraded":"2024-04-17T10:29:00.467018+0000","last_fullsized":"2024-04-17T10:29:00.467018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:31:28.227671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"66'3","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466048+0000","last_change":"2024-04-17T10:28:58.062458+0000","last_active":"2024-04-17T10:29:00.466048+0000","last_peered":"2024-04-17T10:29:00.466048+0000","last_clean":"2024-04-17T10:29:00.466048+0000","last_became_active":"2024-04-17T10:28:58.062162+0000","last_became_peered":"2024-04-17T10:28:58.062162+0000","last_unstale":"2024-04-17T10:29:00.466048+0000","last_undegraded":"2024-04-17T10:29:00.466048+0000","last_fullsized":"2024-04-17T10:29:00.466048+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:24.533003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044899+0000","last_change":"2024-04-17T10:28:59.055801+0000","last_active":"2024-04-17T10:29:00.044899+0000","last_peered":"2024-04-17T10:29:00.044899+0000","last_clean":"2024-04-17T10:29:00.044899+0000","last_became_active":"2024-04-17T10:28:59.055525+0000","last_became_peered":"2024-04-17T10:28:59.055525+0000","last_unstale":"2024-04-17T10:29:00.044899+0000","last_undegraded":"2024-04-17T10:29:00.044899+0000","last_fullsized":"2024-04-17T10:29:00.044899+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:25.201497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"66'6","reported_seq":20,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469628+0000","last_change":"2024-04-17T10:28:58.066220+0000","last_active":"2024-04-17T10:29:00.469628+0000","last_peered":"2024-04-17T10:29:00.469628+0000","last_clean":"2024-04-17T10:29:00.469628+0000","last_became_active":"2024-04-17T10:28:58.065842+0000","last_became_peered":"2024-04-17T10:28:58.065842+0000","last_unstale":"2024-04-17T10:29:00.469628+0000","last_undegraded":"2024-04-17T10:29:00.469628+0000","last_fullsized":"2024-04-17T10:29:00.469628+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:38:20.439434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.043810+0000","last_change":"2024-04-17T10:28:59.059955+0000","last_active":"2024-04-17T10:29:00.043810+0000","last_peered":"2024-04-17T10:29:00.043810+0000","last_clean":"2024-04-17T10:29:00.043810+0000","last_became_active":"2024-04-17T10:28:59.059681+0000","last_became_peered":"2024-04-17T10:28:59.059681+0000","last_unstale":"2024-04-17T10:29:00.043810+0000","last_undegraded":"2024-04-17T10:29:00.043810+0000","last_fullsized":"2024-04-17T10:29:00.043810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:46.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466189+0000","last_change":"2024-04-17T10:28:58.060610+0000","last_active":"2024-04-17T10:29:00.466189+0000","last_peered":"2024-04-17T10:29:00.466189+0000","last_clean":"2024-04-17T10:29:00.466189+0000","last_became_active":"2024-04-17T10:28:58.060319+0000","last_became_peered":"2024-04-17T10:28:58.060319+0000","last_unstale":"2024-04-17T10:29:00.466189+0000","last_undegraded":"2024-04-17T10:29:00.466189+0000","last_fullsized":"2024-04-17T10:29:00.466189+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:57.544495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051899+0000","last_change":"2024-04-17T10:28:59.074660+0000","last_active":"2024-04-17T10:29:00.051899+0000","last_peered":"2024-04-17T10:29:00.051899+0000","last_clean":"2024-04-17T10:29:00.051899+0000","last_became_active":"2024-04-17T10:28:59.074302+0000","last_became_peered":"2024-04-17T10:28:59.074302+0000","last_unstale":"2024-04-17T10:29:00.051899+0000","last_undegraded":"2024-04-17T10:29:00.051899+0000","last_fullsized":"2024-04-17T10:29:00.051899+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:12:25.359261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051689+0000","last_change":"2024-04-17T10:28:58.065249+0000","last_active":"2024-04-17T10:29:00.051689+0000","last_peered":"2024-04-17T10:29:00.051689+0000","last_clean":"2024-04-17T10:29:00.051689+0000","last_became_active":"2024-04-17T10:28:58.064871+0000","last_became_peered":"2024-04-17T10:28:58.064871+0000","last_unstale":"2024-04-17T10:29:00.051689+0000","last_undegraded":"2024-04-17T10:29:00.051689+0000","last_fullsized":"2024-04-17T10:29:00.051689+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:24.533003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465883+0000","last_change":"2024-04-17T10:28:59.052255+0000","last_active":"2024-04-17T10:29:00.465883+0000","last_peered":"2024-04-17T10:29:00.465883+0000","last_clean":"2024-04-17T10:29:00.465883+0000","last_became_active":"2024-04-17T10:28:59.052004+0000","last_became_peered":"2024-04-17T10:28:59.052004+0000","last_unstale":"2024-04-17T10:29:00.465883+0000","last_undegraded":"2024-04-17T10:29:00.465883+0000","last_fullsized":"2024-04-17T10:29:00.465883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:52:31.810508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466050+0000","last_change":"2024-04-17T10:28:58.059092+0000","last_active":"2024-04-17T10:29:00.466050+0000","last_peered":"2024-04-17T10:29:00.466050+0000","last_clean":"2024-04-17T10:29:00.466050+0000","last_became_active":"2024-04-17T10:28:58.058756+0000","last_became_peered":"2024-04-17T10:28:58.058756+0000","last_unstale":"2024-04-17T10:29:00.466050+0000","last_undegraded":"2024-04-17T10:29:00.466050+0000","last_fullsized":"2024-04-17T10:29:00.466050+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:35:45.288433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469444+0000","last_change":"2024-04-17T10:28:59.062191+0000","last_active":"2024-04-17T10:29:00.469444+0000","last_peered":"2024-04-17T10:29:00.469444+0000","last_clean":"2024-04-17T10:29:00.469444+0000","last_became_active":"2024-04-17T10:28:59.061799+0000","last_became_peered":"2024-04-17T10:28:59.061799+0000","last_unstale":"2024-04-17T10:29:00.469444+0000","last_undegraded":"2024-04-17T10:29:00.469444+0000","last_fullsized":"2024-04-17T10:29:00.469444+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:19:16.737073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"66'8","reported_seq":22,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.043669+0000","last_change":"2024-04-17T10:28:58.054318+0000","last_active":"2024-04-17T10:29:00.043669+0000","last_peered":"2024-04-17T10:29:00.043669+0000","last_clean":"2024-04-17T10:29:00.043669+0000","last_became_active":"2024-04-17T10:28:58.053813+0000","last_became_peered":"2024-04-17T10:28:58.053813+0000","last_unstale":"2024-04-17T10:29:00.043669+0000","last_undegraded":"2024-04-17T10:29:00.043669+0000","last_fullsized":"2024-04-17T10:29:00.043669+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:11:32.872834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466361+0000","last_change":"2024-04-17T10:28:59.066167+0000","last_active":"2024-04-17T10:29:00.466361+0000","last_peered":"2024-04-17T10:29:00.466361+0000","last_clean":"2024-04-17T10:29:00.466361+0000","last_became_active":"2024-04-17T10:28:59.065920+0000","last_became_peered":"2024-04-17T10:28:59.065920+0000","last_unstale":"2024-04-17T10:29:00.466361+0000","last_undegraded":"2024-04-17T10:29:00.466361+0000","last_fullsized":"2024-04-17T10:29:00.466361+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:04.822143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052326+0000","last_change":"2024-04-17T10:28:58.064493+0000","last_active":"2024-04-17T10:29:00.052326+0000","last_peered":"2024-04-17T10:29:00.052326+0000","last_clean":"2024-04-17T10:29:00.052326+0000","last_became_active":"2024-04-17T10:28:58.064214+0000","last_became_peered":"2024-04-17T10:28:58.064214+0000","last_unstale":"2024-04-17T10:29:00.052326+0000","last_undegraded":"2024-04-17T10:29:00.052326+0000","last_fullsized":"2024-04-17T10:29:00.052326+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:08:15.445312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466964+0000","last_change":"2024-04-17T10:28:59.051445+0000","last_active":"2024-04-17T10:29:00.466964+0000","last_peered":"2024-04-17T10:29:00.466964+0000","last_clean":"2024-04-17T10:29:00.466964+0000","last_became_active":"2024-04-17T10:28:59.051050+0000","last_became_peered":"2024-04-17T10:28:59.051050+0000","last_unstale":"2024-04-17T10:29:00.466964+0000","last_undegraded":"2024-04-17T10:29:00.466964+0000","last_fullsized":"2024-04-17T10:29:00.466964+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:33.538063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466374+0000","last_change":"2024-04-17T10:28:58.061859+0000","last_active":"2024-04-17T10:29:00.466374+0000","last_peered":"2024-04-17T10:29:00.466374+0000","last_clean":"2024-04-17T10:29:00.466374+0000","last_became_active":"2024-04-17T10:28:58.060099+0000","last_became_peered":"2024-04-17T10:28:58.060099+0000","last_unstale":"2024-04-17T10:29:00.466374+0000","last_undegraded":"2024-04-17T10:29:00.466374+0000","last_fullsized":"2024-04-17T10:29:00.466374+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:27:15.614226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044814+0000","last_change":"2024-04-17T10:28:59.051101+0000","last_active":"2024-04-17T10:29:00.044814+0000","last_peered":"2024-04-17T10:29:00.044814+0000","last_clean":"2024-04-17T10:29:00.044814+0000","last_became_active":"2024-04-17T10:28:59.050825+0000","last_became_peered":"2024-04-17T10:28:59.050825+0000","last_unstale":"2024-04-17T10:29:00.044814+0000","last_undegraded":"2024-04-17T10:29:00.044814+0000","last_fullsized":"2024-04-17T10:29:00.044814+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:03:33.538063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"66'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466258+0000","last_change":"2024-04-17T10:28:58.062681+0000","last_active":"2024-04-17T10:29:00.466258+0000","last_peered":"2024-04-17T10:29:00.466258+0000","last_clean":"2024-04-17T10:29:00.466258+0000","last_became_active":"2024-04-17T10:28:58.062352+0000","last_became_peered":"2024-04-17T10:28:58.062352+0000","last_unstale":"2024-04-17T10:29:00.466258+0000","last_undegraded":"2024-04-17T10:29:00.466258+0000","last_fullsized":"2024-04-17T10:29:00.466258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:38:20.077285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466999+0000","last_change":"2024-04-17T10:28:59.066288+0000","last_active":"2024-04-17T10:29:00.466999+0000","last_peered":"2024-04-17T10:29:00.466999+0000","last_clean":"2024-04-17T10:29:00.466999+0000","last_became_active":"2024-04-17T10:28:59.065655+0000","last_became_peered":"2024-04-17T10:28:59.065655+0000","last_unstale":"2024-04-17T10:29:00.466999+0000","last_undegraded":"2024-04-17T10:29:00.466999+0000","last_fullsized":"2024-04-17T10:29:00.466999+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:25.201497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047978+0000","last_change":"2024-04-17T10:28:58.057257+0000","last_active":"2024-04-17T10:29:00.047978+0000","last_peered":"2024-04-17T10:29:00.047978+0000","last_clean":"2024-04-17T10:29:00.047978+0000","last_became_active":"2024-04-17T10:28:58.056929+0000","last_became_peered":"2024-04-17T10:28:58.056929+0000","last_unstale":"2024-04-17T10:29:00.047978+0000","last_undegraded":"2024-04-17T10:29:00.047978+0000","last_fullsized":"2024-04-17T10:29:00.047978+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:56:41.562789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.045972+0000","last_change":"2024-04-17T10:28:59.079670+0000","last_active":"2024-04-17T10:29:00.045972+0000","last_peered":"2024-04-17T10:29:00.045972+0000","last_clean":"2024-04-17T10:29:00.045972+0000","last_became_active":"2024-04-17T10:28:59.079358+0000","last_became_peered":"2024-04-17T10:28:59.079358+0000","last_unstale":"2024-04-17T10:29:00.045972+0000","last_undegraded":"2024-04-17T10:29:00.045972+0000","last_fullsized":"2024-04-17T10:29:00.045972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:56:31.170733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046490+0000","last_change":"2024-04-17T10:28:58.058005+0000","last_active":"2024-04-17T10:29:00.046490+0000","last_peered":"2024-04-17T10:29:00.046490+0000","last_clean":"2024-04-17T10:29:00.046490+0000","last_became_active":"2024-04-17T10:28:58.057713+0000","last_became_peered":"2024-04-17T10:28:58.057713+0000","last_unstale":"2024-04-17T10:29:00.046490+0000","last_undegraded":"2024-04-17T10:29:00.046490+0000","last_fullsized":"2024-04-17T10:29:00.046490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:00.650684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051784+0000","last_change":"2024-04-17T10:28:59.079556+0000","last_active":"2024-04-17T10:29:00.051784+0000","last_peered":"2024-04-17T10:29:00.051784+0000","last_clean":"2024-04-17T10:29:00.051784+0000","last_became_active":"2024-04-17T10:28:59.078785+0000","last_became_peered":"2024-04-17T10:28:59.078785+0000","last_unstale":"2024-04-17T10:29:00.051784+0000","last_undegraded":"2024-04-17T10:29:00.051784+0000","last_fullsized":"2024-04-17T10:29:00.051784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:51.581135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.051817+0000","last_change":"2024-04-17T10:28:58.064996+0000","last_active":"2024-04-17T10:29:00.051817+0000","last_peered":"2024-04-17T10:29:00.051817+0000","last_clean":"2024-04-17T10:29:00.051817+0000","last_became_active":"2024-04-17T10:28:58.063329+0000","last_became_peered":"2024-04-17T10:28:58.063329+0000","last_unstale":"2024-04-17T10:29:00.051817+0000","last_undegraded":"2024-04-17T10:29:00.051817+0000","last_fullsized":"2024-04-17T10:29:00.051817+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:57.544495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466533+0000","last_change":"2024-04-17T10:28:59.063852+0000","last_active":"2024-04-17T10:29:00.466533+0000","last_peered":"2024-04-17T10:29:00.466533+0000","last_clean":"2024-04-17T10:29:00.466533+0000","last_became_active":"2024-04-17T10:28:59.062569+0000","last_became_peered":"2024-04-17T10:28:59.062569+0000","last_unstale":"2024-04-17T10:29:00.466533+0000","last_undegraded":"2024-04-17T10:29:00.466533+0000","last_fullsized":"2024-04-17T10:29:00.466533+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:08:09.731769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046106+0000","last_change":"2024-04-17T10:28:58.064794+0000","last_active":"2024-04-17T10:29:00.046106+0000","last_peered":"2024-04-17T10:29:00.046106+0000","last_clean":"2024-04-17T10:29:00.046106+0000","last_became_active":"2024-04-17T10:28:58.063301+0000","last_became_peered":"2024-04-17T10:28:58.063301+0000","last_unstale":"2024-04-17T10:29:00.046106+0000","last_undegraded":"2024-04-17T10:29:00.046106+0000","last_fullsized":"2024-04-17T10:29:00.046106+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:57.544495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046436+0000","last_change":"2024-04-17T10:28:59.067788+0000","last_active":"2024-04-17T10:29:00.046436+0000","last_peered":"2024-04-17T10:29:00.046436+0000","last_clean":"2024-04-17T10:29:00.046436+0000","last_became_active":"2024-04-17T10:28:59.067496+0000","last_became_peered":"2024-04-17T10:28:59.067496+0000","last_unstale":"2024-04-17T10:29:00.046436+0000","last_undegraded":"2024-04-17T10:29:00.046436+0000","last_fullsized":"2024-04-17T10:29:00.046436+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:47:28.733650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"66'2","reported_seq":16,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052430+0000","last_change":"2024-04-17T10:28:58.063903+0000","last_active":"2024-04-17T10:29:00.052430+0000","last_peered":"2024-04-17T10:29:00.052430+0000","last_clean":"2024-04-17T10:29:00.052430+0000","last_became_active":"2024-04-17T10:28:58.063611+0000","last_became_peered":"2024-04-17T10:28:58.063611+0000","last_unstale":"2024-04-17T10:29:00.052430+0000","last_undegraded":"2024-04-17T10:29:00.052430+0000","last_fullsized":"2024-04-17T10:29:00.052430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:12:38.531928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469268+0000","last_change":"2024-04-17T10:28:59.061664+0000","last_active":"2024-04-17T10:29:00.469268+0000","last_peered":"2024-04-17T10:29:00.469268+0000","last_clean":"2024-04-17T10:29:00.469268+0000","last_became_active":"2024-04-17T10:28:59.061414+0000","last_became_peered":"2024-04-17T10:28:59.061414+0000","last_unstale":"2024-04-17T10:29:00.469268+0000","last_undegraded":"2024-04-17T10:29:00.469268+0000","last_fullsized":"2024-04-17T10:29:00.469268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:43:08.395313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"67'1","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052291+0000","last_change":"2024-04-17T10:28:58.060159+0000","last_active":"2024-04-17T10:29:00.052291+0000","last_peered":"2024-04-17T10:29:00.052291+0000","last_clean":"2024-04-17T10:29:00.052291+0000","last_became_active":"2024-04-17T10:28:58.059726+0000","last_became_peered":"2024-04-17T10:28:58.059726+0000","last_unstale":"2024-04-17T10:29:00.052291+0000","last_undegraded":"2024-04-17T10:29:00.052291+0000","last_fullsized":"2024-04-17T10:29:00.052291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:48.684408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.052124+0000","last_change":"2024-04-17T10:28:59.073149+0000","last_active":"2024-04-17T10:29:00.052124+0000","last_peered":"2024-04-17T10:29:00.052124+0000","last_clean":"2024-04-17T10:29:00.052124+0000","last_became_active":"2024-04-17T10:28:59.072818+0000","last_became_peered":"2024-04-17T10:28:59.072818+0000","last_unstale":"2024-04-17T10:29:00.052124+0000","last_undegraded":"2024-04-17T10:29:00.052124+0000","last_fullsized":"2024-04-17T10:29:00.052124+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:11.622461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.046546+0000","last_change":"2024-04-17T10:28:58.056208+0000","last_active":"2024-04-17T10:29:00.046546+0000","last_peered":"2024-04-17T10:29:00.046546+0000","last_clean":"2024-04-17T10:29:00.046546+0000","last_became_active":"2024-04-17T10:28:58.055861+0000","last_became_peered":"2024-04-17T10:28:58.055861+0000","last_unstale":"2024-04-17T10:29:00.046546+0000","last_undegraded":"2024-04-17T10:29:00.046546+0000","last_fullsized":"2024-04-17T10:29:00.046546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:38:20.077285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.467071+0000","last_change":"2024-04-17T10:28:59.066588+0000","last_active":"2024-04-17T10:29:00.467071+0000","last_peered":"2024-04-17T10:29:00.467071+0000","last_clean":"2024-04-17T10:29:00.467071+0000","last_became_active":"2024-04-17T10:28:59.065998+0000","last_became_peered":"2024-04-17T10:28:59.065998+0000","last_unstale":"2024-04-17T10:29:00.467071+0000","last_undegraded":"2024-04-17T10:29:00.467071+0000","last_fullsized":"2024-04-17T10:29:00.467071+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:54.993259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469648+0000","last_change":"2024-04-17T10:28:58.066633+0000","last_active":"2024-04-17T10:29:00.469648+0000","last_peered":"2024-04-17T10:29:00.469648+0000","last_clean":"2024-04-17T10:29:00.469648+0000","last_became_active":"2024-04-17T10:28:58.066242+0000","last_became_peered":"2024-04-17T10:28:58.066242+0000","last_unstale":"2024-04-17T10:29:00.469648+0000","last_undegraded":"2024-04-17T10:29:00.469648+0000","last_fullsized":"2024-04-17T10:29:00.469648+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:45.008213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465559+0000","last_change":"2024-04-17T10:28:59.055789+0000","last_active":"2024-04-17T10:29:00.465559+0000","last_peered":"2024-04-17T10:29:00.465559+0000","last_clean":"2024-04-17T10:29:00.465559+0000","last_became_active":"2024-04-17T10:28:59.055434+0000","last_became_peered":"2024-04-17T10:28:59.055434+0000","last_unstale":"2024-04-17T10:29:00.465559+0000","last_undegraded":"2024-04-17T10:29:00.465559+0000","last_fullsized":"2024-04-17T10:29:00.465559+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:49:46.016376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.466315+0000","last_change":"2024-04-17T10:28:58.063062+0000","last_active":"2024-04-17T10:29:00.466315+0000","last_peered":"2024-04-17T10:29:00.466315+0000","last_clean":"2024-04-17T10:29:00.466315+0000","last_became_active":"2024-04-17T10:28:58.062785+0000","last_became_peered":"2024-04-17T10:28:58.062785+0000","last_unstale":"2024-04-17T10:29:00.466315+0000","last_undegraded":"2024-04-17T10:29:00.466315+0000","last_fullsized":"2024-04-17T10:29:00.466315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:00.650684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.044051+0000","last_change":"2024-04-17T10:28:59.054328+0000","last_active":"2024-04-17T10:29:00.044051+0000","last_peered":"2024-04-17T10:29:00.044051+0000","last_clean":"2024-04-17T10:29:00.044051+0000","last_became_active":"2024-04-17T10:28:59.054059+0000","last_became_peered":"2024-04-17T10:28:59.054059+0000","last_unstale":"2024-04-17T10:29:00.044051+0000","last_undegraded":"2024-04-17T10:29:00.044051+0000","last_fullsized":"2024-04-17T10:29:00.044051+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:21.757809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047747+0000","last_change":"2024-04-17T10:28:58.053566+0000","last_active":"2024-04-17T10:29:00.047747+0000","last_peered":"2024-04-17T10:29:00.047747+0000","last_clean":"2024-04-17T10:29:00.047747+0000","last_became_active":"2024-04-17T10:28:58.053306+0000","last_became_peered":"2024-04-17T10:28:58.053306+0000","last_unstale":"2024-04-17T10:29:00.047747+0000","last_undegraded":"2024-04-17T10:29:00.047747+0000","last_fullsized":"2024-04-17T10:29:00.047747+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:08.731905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469369+0000","last_change":"2024-04-17T10:28:59.059115+0000","last_active":"2024-04-17T10:29:00.469369+0000","last_peered":"2024-04-17T10:29:00.469369+0000","last_clean":"2024-04-17T10:29:00.469369+0000","last_became_active":"2024-04-17T10:28:59.058828+0000","last_became_peered":"2024-04-17T10:28:59.058828+0000","last_unstale":"2024-04-17T10:29:00.469369+0000","last_undegraded":"2024-04-17T10:29:00.469369+0000","last_fullsized":"2024-04-17T10:29:00.469369+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:40:28.186710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047578+0000","last_change":"2024-04-17T10:28:58.052819+0000","last_active":"2024-04-17T10:29:00.047578+0000","last_peered":"2024-04-17T10:29:00.047578+0000","last_clean":"2024-04-17T10:29:00.047578+0000","last_became_active":"2024-04-17T10:28:58.052385+0000","last_became_peered":"2024-04-17T10:28:58.052385+0000","last_unstale":"2024-04-17T10:29:00.047578+0000","last_undegraded":"2024-04-17T10:29:00.047578+0000","last_fullsized":"2024-04-17T10:29:00.047578+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:49:28.308279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.469188+0000","last_change":"2024-04-17T10:28:59.074122+0000","last_active":"2024-04-17T10:29:00.469188+0000","last_peered":"2024-04-17T10:29:00.469188+0000","last_clean":"2024-04-17T10:29:00.469188+0000","last_became_active":"2024-04-17T10:28:59.073829+0000","last_became_peered":"2024-04-17T10:28:59.073829+0000","last_unstale":"2024-04-17T10:29:00.469188+0000","last_undegraded":"2024-04-17T10:29:00.469188+0000","last_fullsized":"2024-04-17T10:29:00.469188+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:29:09.749258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.468899+0000","last_change":"2024-04-17T10:28:58.062346+0000","last_active":"2024-04-17T10:29:00.468899+0000","last_peered":"2024-04-17T10:29:00.468899+0000","last_clean":"2024-04-17T10:29:00.468899+0000","last_became_active":"2024-04-17T10:28:58.062085+0000","last_became_peered":"2024-04-17T10:28:58.062085+0000","last_unstale":"2024-04-17T10:29:00.468899+0000","last_undegraded":"2024-04-17T10:29:00.468899+0000","last_fullsized":"2024-04-17T10:29:00.468899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:42:22.030013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465878+0000","last_change":"2024-04-17T10:28:59.056041+0000","last_active":"2024-04-17T10:29:00.465878+0000","last_peered":"2024-04-17T10:29:00.465878+0000","last_clean":"2024-04-17T10:29:00.465878+0000","last_became_active":"2024-04-17T10:28:59.055314+0000","last_became_peered":"2024-04-17T10:28:59.055314+0000","last_unstale":"2024-04-17T10:29:00.465878+0000","last_undegraded":"2024-04-17T10:29:00.465878+0000","last_fullsized":"2024-04-17T10:29:00.465878+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:58.032381+0000","last_clean_scrub_stamp":"2024-04-17T10:28:58.032381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:57.733252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.047406+0000","last_change":"2024-04-17T10:28:58.057183+0000","last_active":"2024-04-17T10:29:00.047406+0000","last_peered":"2024-04-17T10:29:00.047406+0000","last_clean":"2024-04-17T10:29:00.047406+0000","last_became_active":"2024-04-17T10:28:58.056893+0000","last_became_peered":"2024-04-17T10:28:58.056893+0000","last_unstale":"2024-04-17T10:29:00.047406+0000","last_undegraded":"2024-04-17T10:29:00.047406+0000","last_fullsized":"2024-04-17T10:29:00.047406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:28:57.024219+0000","last_clean_scrub_stamp":"2024-04-17T10:28:57.024219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:17:29.863421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":259,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-04-17T10:29:00.465685+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:29:00.465685+0000","last_peered":"2024-04-17T10:29:00.465685+0000","last_clean":"2024-04-17T10:29:00.465685+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:29:00.465685+0000","last_undegraded":"2024-04-17T10:29:00.465685+0000","last_fullsized":"2024-04-17T10:29:00.465685+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574904,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1421312,"data_stored":962230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":815104,"data_stored":371120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36136,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335704,"statfs":{"total":96636764160,"available":96599760896,"internally_reserved":0,"allocated":815104,"data_stored":371108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574903,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40308,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331532,"statfs":{"total":96636764160,"available":96595488768,"internally_reserved":0,"allocated":823296,"data_stored":372698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574903,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36140,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335700,"statfs":{"total":96636764160,"available":96599756800,"internally_reserved":0,"allocated":811008,"data_stored":371086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574904,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36132,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335708,"statfs":{"total":96636764160,"available":96599764992,"internally_reserved":0,"allocated":811008,"data_stored":371086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36164,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335676,"statfs":{"total":96636764160,"available":96599732224,"internally_reserved":0,"allocated":839680,"data_stored":373462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574903,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58924,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312916,"statfs":{"total":96636764160,"available":96576425984,"internally_reserved":0,"allocated":1404928,"data_stored":961454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:08.103 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:29:08.103 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:29:08.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.297+0000 7ff418600640 1 Processor -- start 2024-04-17T10:29:08.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff418600640 1 -- start start 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff418600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff4101583f0 0x7ff410072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff418600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4100b5300 0x7ff41006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff418600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41015f360 con 0x7ff4100728c0 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff418600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4100b3160 con 0x7ff4100b5300 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff418600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4100b3440 con 0x7ff4101583f0 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff416600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4100b5300 0x7ff41006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff416600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff4100b5300 0x7ff41006ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55816/0 (socket says 172.21.15.67:55816) 2024-04-17T10:29:08.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff416600640 1 -- 172.21.15.67:0/3975356683 learned_addr learned my addr 172.21.15.67:0/3975356683 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:08.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff415c00640 1 --2- 172.21.15.67:0/3975356683 >> v2:172.21.15.86:3300/0 conn(0x7ff4101583f0 0x7ff410072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff416600640 1 -- 172.21.15.67:0/3975356683 >> v2:172.21.15.86:3300/0 conn(0x7ff4101583f0 msgr2=0x7ff410072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:08.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff416600640 1 --2- 172.21.15.67:0/3975356683 >> v2:172.21.15.86:3300/0 conn(0x7ff4101583f0 0x7ff410072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff416600640 1 -- 172.21.15.67:0/3975356683 >> v1:172.21.15.86:6789/0 conn(0x7ff4100728c0 legacy=0x7ff4100b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:08.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff416600640 1 -- 172.21.15.67:0/3975356683 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4100b3720 con 0x7ff4100b5300 2024-04-17T10:29:08.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.301+0000 7ff415c00640 1 --2- 172.21.15.67:0/3975356683 >> v2:172.21.15.86:3300/0 conn(0x7ff4101583f0 0x7ff410072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:08.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff416600640 1 --2- 172.21.15.67:0/3975356683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff41006ac90 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff410066fe0 tx=0x7ff40801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d650247275d48ceb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:08.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff40fe00640 1 -- 172.21.15.67:0/3975356683 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4080a5760 con 0x7ff4100b5300 2024-04-17T10:29:08.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff40fe00640 1 -- 172.21.15.67:0/3975356683 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4080a5900 con 0x7ff4100b5300 2024-04-17T10:29:08.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff40fe00640 1 -- 172.21.15.67:0/3975356683 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4080a5cd0 con 0x7ff4100b5300 2024-04-17T10:29:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 -- 172.21.15.67:0/3975356683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 msgr2=0x7ff41006ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 --2- 172.21.15.67:0/3975356683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff41006ac90 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff410066fe0 tx=0x7ff40801c9b0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 -- 172.21.15.67:0/3975356683 shutdown_connections 2024-04-17T10:29:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 --2- 172.21.15.67:0/3975356683 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff41006ac90 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 --2- 172.21.15.67:0/3975356683 >> v2:172.21.15.86:3300/0 conn(0x7ff4101583f0 0x7ff410072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 -- 172.21.15.67:0/3975356683 >> 172.21.15.67:0/3975356683 conn(0x7ff4100b2570 msgr2=0x7ff410071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:08.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 -- 172.21.15.67:0/3975356683 shutdown_connections 2024-04-17T10:29:08.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 -- 172.21.15.67:0/3975356683 wait complete. 2024-04-17T10:29:08.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 Processor -- start 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.305+0000 7ff418600640 1 -- start start 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4100728c0 0x7ff4101521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff410149c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4101583f0 0x7ff4101574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41006b9e0 con 0x7ff4100b5300 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100b4200 con 0x7ff4100728c0 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41006bb60 con 0x7ff4101583f0 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff410149c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff410149c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55820/0 (socket says 172.21.15.67:55820) 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 -- 172.21.15.67:0/2595681568 learned_addr learned my addr 172.21.15.67:0/2595681568 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:08.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415c00640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4100728c0 0x7ff4101521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff416600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4101583f0 0x7ff4101574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 -- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4101583f0 msgr2=0x7ff4101574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4101583f0 0x7ff4101574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 -- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4100728c0 msgr2=0x7ff4101521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4100728c0 0x7ff4101521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 -- 172.21.15.67:0/2595681568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41014a150 con 0x7ff4100b5300 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415c00640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4100728c0 0x7ff4101521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff416600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4101583f0 0x7ff4101574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff415200640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff410149c10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00c450 tx=0x7ff3fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff40d600640 1 -- 172.21.15.67:0/2595681568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc012830 con 0x7ff4100b5300 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff40d600640 1 -- 172.21.15.67:0/2595681568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc017070 con 0x7ff4100b5300 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff40d600640 1 -- 172.21.15.67:0/2595681568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc012c30 con 0x7ff4100b5300 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff410161f00 con 0x7ff4100b5300 2024-04-17T10:29:08.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff410153e10 con 0x7ff4100b5300 2024-04-17T10:29:08.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff410154010 con 0x7ff4100b5300 2024-04-17T10:29:08.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.309+0000 7ff40d600640 1 -- 172.21.15.67:0/2595681568 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff3fc01c460 con 0x7ff4100b5300 2024-04-17T10:29:08.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.313+0000 7ff40d600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff3f0066f60 0x7ff3f0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.313+0000 7ff40d600640 1 -- 172.21.15.67:0/2595681568 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff3fc084f40 con 0x7ff4100b5300 2024-04-17T10:29:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.313+0000 7ff415c00640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff3f0066f60 0x7ff3f0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.313+0000 7ff415c00640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff3f0066f60 0x7ff3f0069420 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff41015e0f0 tx=0x7ff404008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:08.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.317+0000 7ff40d600640 1 -- 172.21.15.67:0/2595681568 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff410154010 con 0x7ff4100b5300 2024-04-17T10:29:08.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.533+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff41006adc0 con 0x7ff4100b5300 2024-04-17T10:29:08.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.537+0000 7ff40d600640 1 -- 172.21.15.67:0/2595681568 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff3fc055db0 con 0x7ff4100b5300 2024-04-17T10:29:08.532 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:29:08.532 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:29:08.532 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:29:08.532 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:29:08.532 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:29:08.533 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff3f0066f60 msgr2=0x7ff3f0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff3f0066f60 0x7ff3f0069420 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ff41015e0f0 tx=0x7ff404008040 comp rx=0 tx=0).stop 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 msgr2=0x7ff410149c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff410149c10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc00c450 tx=0x7ff3fc00c910 comp rx=0 tx=0).stop 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 shutdown_connections 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff3f0066f60 0x7ff3f0069420 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff4101583f0 0x7ff4101574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff4100b5300 0x7ff410149c10 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 --2- 172.21.15.67:0/2595681568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff4100728c0 0x7ff4101521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 >> 172.21.15.67:0/2595681568 conn(0x7ff4100b2570 msgr2=0x7ff410071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:08.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 shutdown_connections 2024-04-17T10:29:08.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.541+0000 7ff418600640 1 -- 172.21.15.67:0/2595681568 wait complete. 2024-04-17T10:29:08.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ee000640 1 Processor -- start 2024-04-17T10:29:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ee000640 1 -- start start 2024-04-17T10:29:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ee000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f02e8161160 0x7f02e8099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ee000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e80d0760 con 0x7f02e8099f90 2024-04-17T10:29:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ee000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e81624e0 con 0x7f02e80b2570 2024-04-17T10:29:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ee000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02e80cfc70 con 0x7f02e8161160 2024-04-17T10:29:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ed600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f02e8099f90 0x7f02e8161dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42778/0 (socket says 172.21.15.67:42778) 2024-04-17T10:29:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ed600640 1 -- 172.21.15.67:0/4000080608 learned_addr learned my addr 172.21.15.67:0/4000080608 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ecc00640 1 --2- 172.21.15.67:0/4000080608 >> v2:172.21.15.86:3300/0 conn(0x7f02e8161160 0x7f02e8099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479767564 0 0) 0x7f02e80d0760 con 0x7f02e8099f90 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d0003530 con 0x7f02e8099f90 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136460411 0 0) 0x7f02e81624e0 con 0x7f02e80b2570 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e80d0760 con 0x7f02e80b2570 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244149027 0 0) 0x7f02d0003530 con 0x7f02e8099f90 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ecc00640 1 -- 172.21.15.67:0/4000080608 >> v1:172.21.15.86:6789/0 conn(0x7f02e80b2570 legacy=0x7f02e81616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ecc00640 1 -- 172.21.15.67:0/4000080608 >> v1:172.21.15.67:6789/0 conn(0x7f02e8099f90 legacy=0x7f02e8161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.701+0000 7f02ecc00640 1 -- 172.21.15.67:0/4000080608 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e80cc2a0 con 0x7f02e8161160 2024-04-17T10:29:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ecc00640 1 --2- 172.21.15.67:0/4000080608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 0x7f02e8099a50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0097c0 tx=0x7f02dc01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6217be120121d411 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0b0070 con 0x7f02e8161160 2024-04-17T10:29:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc0ab400 con 0x7f02e8161160 2024-04-17T10:29:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02e7400640 1 -- 172.21.15.67:0/4000080608 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02e0001a00 con 0x7f02e8161160 2024-04-17T10:29:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ee000640 1 -- 172.21.15.67:0/4000080608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 msgr2=0x7f02e8099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ee000640 1 --2- 172.21.15.67:0/4000080608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 0x7f02e8099a50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0097c0 tx=0x7f02dc01c3c0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ee000640 1 -- 172.21.15.67:0/4000080608 shutdown_connections 2024-04-17T10:29:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ee000640 1 --2- 172.21.15.67:0/4000080608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 0x7f02e8099a50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ee000640 1 -- 172.21.15.67:0/4000080608 >> 172.21.15.67:0/4000080608 conn(0x7f02e80d1690 msgr2=0x7f02e80d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ee000640 1 -- 172.21.15.67:0/4000080608 shutdown_connections 2024-04-17T10:29:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.705+0000 7f02ee000640 1 -- 172.21.15.67:0/4000080608 wait complete. 2024-04-17T10:29:08.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 Processor -- start 2024-04-17T10:29:08.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 -- start start 2024-04-17T10:29:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e8099f90 0x7f02e80c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e80b2570 0x7f02e80c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 0x7f02e814c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e809a3c0 con 0x7f02e80b2570 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e80cc5d0 con 0x7f02e8161160 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e80b2bc0 con 0x7f02e8099f90 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e80b2570 0x7f02e80c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e80b2570 0x7f02e80c51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55834/0 (socket says 172.21.15.67:55834) 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 -- 172.21.15.67:0/172469724 learned_addr learned my addr 172.21.15.67:0/172469724 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ed600640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 0x7f02e814c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ecc00640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e8099f90 0x7f02e80c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 -- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e8099f90 msgr2=0x7f02e80c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e8099f90 0x7f02e80c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 -- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 msgr2=0x7f02e814c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 0x7f02e814c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 -- 172.21.15.67:0/172469724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e80ca3b0 con 0x7f02e80b2570 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e7e00640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e80b2570 0x7f02e80c51c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c460 tx=0x7f02d400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e4c00640 1 -- 172.21.15.67:0/172469724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40052d0 con 0x7f02e80b2570 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e4c00640 1 -- 172.21.15.67:0/172469724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d4005490 con 0x7f02e80b2570 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02e4c00640 1 -- 172.21.15.67:0/172469724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4012850 con 0x7f02e80b2570 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e80ca690 con 0x7f02e80b2570 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e80cac50 con 0x7f02e80b2570 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.709+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8074dc0 con 0x7f02e80b2570 2024-04-17T10:29:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.713+0000 7f02e4c00640 1 -- 172.21.15.67:0/172469724 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f02d40129f0 con 0x7f02e80b2570 2024-04-17T10:29:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.713+0000 7f02e4c00640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.713+0000 7f02e4c00640 1 -- 172.21.15.67:0/172469724 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f02d4084da0 con 0x7f02e80b2570 2024-04-17T10:29:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.713+0000 7f02ecc00640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.713+0000 7f02ecc00640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0ad000 tx=0x7f02dc0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.717+0000 7f02e4c00640 1 -- 172.21.15.67:0/172469724 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f02d40554a0 con 0x7f02e80b2570 2024-04-17T10:29:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.909+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02e8099b50 con 0x7f02e80b2570 2024-04-17T10:29:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02e4c00640 1 -- 172.21.15.67:0/172469724 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4936 (secure 0 0 0) 0x7f02d4059540 con 0x7f02e80b2570 2024-04-17T10:29:08.906 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:08.906 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1277872832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1277872832},{"type":"v1","addr":"172.21.15.86:6836","nonce":1277872832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5900,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3787337522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3787337522},{"type":"v1","addr":"172.21.15.86:6837","nonce":3787337522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3103235412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3103235412},{"type":"v1","addr":"172.21.15.67:6834","nonce":3103235412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:01.462609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6651},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6651":{"gid":6651,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/392417778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":392417778},{"type":"v1","addr":"172.21.15.67:6835","nonce":392417778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:08.906 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 40 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 msgr2=0x7f02c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f02dc0ad000 tx=0x7f02dc0a6000 comp rx=0 tx=0).stop 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e80b2570 msgr2=0x7f02e80c51c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e80b2570 0x7f02e80c51c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f02d400c460 tx=0x7f02d400c920 comp rx=0 tx=0).stop 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 shutdown_connections 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02c8066f60 0x7f02c8069420 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02e8161160 0x7f02e814c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02e80b2570 0x7f02e80c51c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 --2- 172.21.15.67:0/172469724 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02e8099f90 0x7f02e80c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 >> 172.21.15.67:0/172469724 conn(0x7f02e80d1690 msgr2=0x7f02e80cf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 shutdown_connections 2024-04-17T10:29:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:08.913+0000 7f02ee000640 1 -- 172.21.15.67:0/172469724 wait complete. 2024-04-17T10:29:09.329 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:28:58.444091+0000', 'modified': '2024-04-17T10:29:01.462609+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6651}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6651': {'gid': 6651, 'name': 'c', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/392417778', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 392417778}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 392417778}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:29:09.330 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:29:09.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T10:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbfc00640 1 Processor -- start 2024-04-17T10:29:09.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbfc00640 1 -- start start 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbfc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8fb8071310 0x7f8fb8071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbfc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb80cc300 0x7f8fb80cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbfc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb806e2f0 con 0x7f8fb8145040 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbfc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb80a86c0 con 0x7f8fb8071310 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbfc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb8071cb0 con 0x7f8fb80cc300 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fafe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8fb8071310 0x7f8fb8071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fafe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8fb8071310 0x7f8fb8071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55850/0 (socket says 172.21.15.67:55850) 2024-04-17T10:29:09.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fafe00640 1 -- 172.21.15.67:0/3218111432 learned_addr learned my addr 172.21.15.67:0/3218111432 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/3218111432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb80cc300 0x7f8fb80cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fafe00640 1 -- 172.21.15.67:0/3218111432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb80cc300 msgr2=0x7f8fb80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fafe00640 1 --2- 172.21.15.67:0/3218111432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb80cc300 0x7f8fb80cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fafe00640 1 -- 172.21.15.67:0/3218111432 >> v1:172.21.15.86:6789/0 conn(0x7f8fb8145040 legacy=0x7f8fb806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.481+0000 7f8fafe00640 1 -- 172.21.15.67:0/3218111432 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb806d6e0 con 0x7f8fb8071310 2024-04-17T10:29:09.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fafe00640 1 --2- 172.21.15.67:0/3218111432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb8071770 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009870 tx=0x7f8fa001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d60e25591d3cea6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fb7e00640 1 -- 172.21.15.67:0/3218111432 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00b5070 con 0x7f8fb8071310 2024-04-17T10:29:09.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fb7e00640 1 -- 172.21.15.67:0/3218111432 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa00a8a50 con 0x7f8fb8071310 2024-04-17T10:29:09.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fb7e00640 1 -- 172.21.15.67:0/3218111432 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa00a8e20 con 0x7f8fb8071310 2024-04-17T10:29:09.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 -- 172.21.15.67:0/3218111432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 msgr2=0x7f8fb8071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/3218111432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb8071770 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009870 tx=0x7f8fa001ed50 comp rx=0 tx=0).stop 2024-04-17T10:29:09.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 -- 172.21.15.67:0/3218111432 shutdown_connections 2024-04-17T10:29:09.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/3218111432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb80cc300 0x7f8fb80cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/3218111432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb8071770 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 -- 172.21.15.67:0/3218111432 >> 172.21.15.67:0/3218111432 conn(0x7f8fb806ac90 msgr2=0x7f8fb806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 -- 172.21.15.67:0/3218111432 shutdown_connections 2024-04-17T10:29:09.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.485+0000 7f8fbfc00640 1 -- 172.21.15.67:0/3218111432 wait complete. 2024-04-17T10:29:09.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 Processor -- start 2024-04-17T10:29:09.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 -- start start 2024-04-17T10:29:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb80b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb80cc300 0x7f8fb815ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb8145040 0x7f8fb815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb806df80 con 0x7f8fb8071310 2024-04-17T10:29:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb806dca0 con 0x7f8fb80cc300 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb806d370 con 0x7f8fb8145040 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb80b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb80b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55862/0 (socket says 172.21.15.67:55862) 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 -- 172.21.15.67:0/791596614 learned_addr learned my addr 172.21.15.67:0/791596614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fafe00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb80cc300 0x7f8fb815ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb8145040 0x7f8fb815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 -- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb8145040 msgr2=0x7f8fb815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb8145040 0x7f8fb815e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 -- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb80cc300 msgr2=0x7f8fb815ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb80cc300 0x7f8fb815ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 -- 172.21.15.67:0/791596614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb81f3e70 con 0x7f8fb8071310 2024-04-17T10:29:09.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbd200640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb80b4050 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c370 tx=0x7f8fa800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fb5600640 1 -- 172.21.15.67:0/791596614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa80052d0 con 0x7f8fb8071310 2024-04-17T10:29:09.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fb5600640 1 -- 172.21.15.67:0/791596614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa8005470 con 0x7f8fb8071310 2024-04-17T10:29:09.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb81f4110 con 0x7f8fb8071310 2024-04-17T10:29:09.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fb5600640 1 -- 172.21.15.67:0/791596614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8007260 con 0x7f8fb8071310 2024-04-17T10:29:09.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.489+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb81f4bf0 con 0x7f8fb8071310 2024-04-17T10:29:09.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.493+0000 7f8fb5600640 1 -- 172.21.15.67:0/791596614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8fa8007400 con 0x7f8fb8071310 2024-04-17T10:29:09.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.493+0000 7f8fb5600640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f94066f60 0x7f8f94069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.493+0000 7f8fb5600640 1 -- 172.21.15.67:0/791596614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fa8084580 con 0x7f8fb8071310 2024-04-17T10:29:09.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.493+0000 7f8fafe00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f94066f60 0x7f8f94069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.493+0000 7f8fc31a9000 1 Processor -- start 2024-04-17T10:29:09.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.493+0000 7f8fc31a9000 1 -- start start 2024-04-17T10:29:09.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.493+0000 7f8fafe00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f94066f60 0x7f8f94069420 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00b2000 tx=0x7f8fa0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x5636789769f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x563678977310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56367885e5f0 con 0x563678976f30 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636787c5da0 con 0x5636789c0920 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56367874e150 con 0x5636789c1240 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x563678977310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x563678977310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55864/0 (socket says 172.21.15.67:55864) 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 -- 172.21.15.67:0/990395101 learned_addr learned my addr 172.21.15.67:0/990395101 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbd200640 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fafe00640 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x5636789769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 -- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 msgr2=0x5636789769f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x5636789769f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 -- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 msgr2=0x5636789c0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 -- 172.21.15.67:0/990395101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563678767ca0 con 0x563678976f30 2024-04-17T10:29:09.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x563678977310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0009f80 tx=0x7f8fb0003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ce0c9cec045faf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fad600640 1 -- 172.21.15.67:0/990395101 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0015420 con 0x563678976f30 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fad600640 1 -- 172.21.15.67:0/990395101 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb001a070 con 0x563678976f30 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fad600640 1 -- 172.21.15.67:0/990395101 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb00157c0 con 0x563678976f30 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- 172.21.15.67:0/990395101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 msgr2=0x563678977310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x563678977310 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0009f80 tx=0x7f8fb0003e30 comp rx=0 tx=0).stop 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- 172.21.15.67:0/990395101 shutdown_connections 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x563678977310 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x5636789769f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/990395101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- 172.21.15.67:0/990395101 >> 172.21.15.67:0/990395101 conn(0x56367897d2a0 msgr2=0x5636789752d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- 172.21.15.67:0/990395101 shutdown_connections 2024-04-17T10:29:09.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- 172.21.15.67:0/990395101 wait complete. 2024-04-17T10:29:09.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 -- start start 2024-04-17T10:29:09.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.497+0000 7f8fc31a9000 1 Processor -- start 2024-04-17T10:29:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x56367897aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x563678780990 con 0x563678976f30 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636787c1e40 con 0x5636789c0920 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x563678751900 con 0x5636789c1240 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 10 client.0 _ll_register_callbacks cb 0x5636788791c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789c7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55874/0 (socket says 172.21.15.67:55874) 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 -- 172.21.15.67:0/1794081410 learned_addr learned my addr 172.21.15.67:0/1794081410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fafe00640 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x56367897aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 -- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 msgr2=0x56367897aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x56367897aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 -- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 msgr2=0x5636789c8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 -- 172.21.15.67:0/1794081410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56367877e630 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fbd200640 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789c7cf0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8000fc0 tx=0x7f8fa8054e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc5217b6f1f389f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 1 -- 172.21.15.67:0/1794081410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa805d400 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 1 -- 172.21.15.67:0/1794081410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fa8051410 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fb001a070 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 1 -- 172.21.15.67:0/1794081410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fa80840d0 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 10 client.6825 fetch_fsmap learned FSMap version 40 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x56367874e3a0 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 1 -- 172.21.15.67:0/1794081410 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb001a070 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 1 -- 172.21.15.67:0/1794081410 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 40) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8fa8082ce0 con 0x563678976f30 2024-04-17T10:29:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 10 client.6825 fetch_fsmap finished waiting for FSMap version 40 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 10 client.6825 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 20 client.6825 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x56367897c160 0x5636789d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 4 client.6825 mds_command: new command op to 6651 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 --> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563678761a50 con 0x56367897c160 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fafe00640 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x56367897c160 0x5636789d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fafe00640 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x56367897c160 0x5636789d1d50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a46e3e65f1f3dec server_cookie=adff4db270918cc4 in_seq=0 out_seq=0 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 10 client.6825 ms_handle_connect on v2:172.21.15.67:6833/392417778 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 1 -- 172.21.15.67:0/1794081410 <== mds.0 v2:172.21.15.67:6833/392417778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563678761a50 con 0x56367897c160 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 10 client.6825 handle_command_reply: tid=0 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 client.6825 shutdown 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 msgr2=0x5636789c7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789c7cf0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8000fc0 tx=0x7f8fa8054e60 comp rx=0 tx=0).stop 2024-04-17T10:29:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 shutdown_connections 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x56367897c160 0x5636789d1d50 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x56367897aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789c8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/1794081410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789c7cf0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fa6000640 0 client.6825 ms_handle_reset on v2:172.21.15.67:6833/392417778 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 >> 172.21.15.67:0/1794081410 conn(0x56367897d2a0 msgr2=0x5636789759e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.501+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 shutdown_connections 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.505+0000 7f8fc31a9000 1 -- 172.21.15.67:0/1794081410 wait complete. 2024-04-17T10:29:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.505+0000 7f8fc31a9000 20 client.6825 trim_cache size 0 max 16384 2024-04-17T10:29:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.517+0000 7f8fc31a9000 1 Processor -- start 2024-04-17T10:29:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.517+0000 7f8fc31a9000 1 -- start start 2024-04-17T10:29:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563678976f30 0x56367897a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x56367897a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636789c1240 0x5636789c9920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb81ff620 con 0x5636789c1240 2024-04-17T10:29:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80ada10 con 0x5636789c0920 2024-04-17T10:29:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80b3650 con 0x563678976f30 2024-04-17T10:29:09.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636789c1240 0x5636789c9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636789c1240 0x5636789c9920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55878/0 (socket says 172.21.15.67:55878) 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 -- 172.21.15.67:0/2724501762 learned_addr learned my addr 172.21.15.67:0/2724501762 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 -- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563678976f30 msgr2=0x56367897a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbd200640 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563678976f30 0x56367897a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fafe00640 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x56367897a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563678976f30 0x56367897a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 -- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 msgr2=0x56367897a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x56367897a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 -- 172.21.15.67:0/2724501762 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636789c1070 con 0x5636789c1240 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbd200640 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563678976f30 0x56367897a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fafe00640 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x56367897a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636789c1240 0x5636789c9920 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0009f20 tx=0x7f8fb000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d8cb9530d83ac9a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8facc00640 1 -- 172.21.15.67:0/2724501762 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb00156b0 con 0x5636789c1240 2024-04-17T10:29:09.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8facc00640 1 -- 172.21.15.67:0/2724501762 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb001a040 con 0x5636789c1240 2024-04-17T10:29:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8facc00640 1 -- 172.21.15.67:0/2724501762 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb0015ab0 con 0x5636789c1240 2024-04-17T10:29:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- 172.21.15.67:0/2724501762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636789c1240 msgr2=0x5636789c9920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636789c1240 0x5636789c9920 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f8fb0009f20 tx=0x7f8fb000e000 comp rx=0 tx=0).stop 2024-04-17T10:29:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- 172.21.15.67:0/2724501762 shutdown_connections 2024-04-17T10:29:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5636789c1240 0x5636789c9920 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x56367897a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/2724501762 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x563678976f30 0x56367897a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- 172.21.15.67:0/2724501762 >> 172.21.15.67:0/2724501762 conn(0x56367897d2a0 msgr2=0x563678979bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- 172.21.15.67:0/2724501762 shutdown_connections 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- 172.21.15.67:0/2724501762 wait complete. 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 -- start start 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.521+0000 7f8fc31a9000 1 Processor -- start 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789cca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x563678a052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636787ca3d0 con 0x563678976f30 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x563678881d50 con 0x5636789c0920 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x563678760980 con 0x5636789c1240 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 10 client.0 _ll_register_callbacks cb 0x7f8fb0009bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55886/0 (socket says 172.21.15.67:55886) 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 -- 172.21.15.67:0/3596013872 learned_addr learned my addr 172.21.15.67:0/3596013872 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbdc00640 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x563678a052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fafe00640 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 -- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 msgr2=0x563678a052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x563678a052a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 -- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 msgr2=0x5636789ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 -- 172.21.15.67:0/3596013872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563678847600 con 0x563678976f30 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fbd200640 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789cca80 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8000bf0 tx=0x7f8fa80827e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ef6df2f27a6e8d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 1 -- 172.21.15.67:0/3596013872 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8084360 con 0x563678976f30 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 1 -- 172.21.15.67:0/3596013872 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8fa8051410 con 0x563678976f30 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8fb001a040 con 0x563678976f30 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 1 -- 172.21.15.67:0/3596013872 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8fa805d400 con 0x563678976f30 2024-04-17T10:29:09.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 10 client.6831 fetch_fsmap learned FSMap version 40 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x56367875a9b0 con 0x563678976f30 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 1 -- 172.21.15.67:0/3596013872 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb001a040 con 0x563678976f30 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 1 -- 172.21.15.67:0/3596013872 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 40) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8fa80541f0 con 0x563678976f30 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 10 client.6831 fetch_fsmap finished waiting for FSMap version 40 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 10 client.6831 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 20 client.6831 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x563678a057e0 0x563678a05bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 4 client.6831 mds_command: new command op to 6651 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 --> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5636787d32e0 con 0x563678a057e0 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fafe00640 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x563678a057e0 0x563678a05bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fafe00640 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x563678a057e0 0x563678a05bc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6bb17aa8f33df69 server_cookie=868e25ef228299a6 in_seq=0 out_seq=0 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 10 client.6831 ms_handle_connect on v2:172.21.15.67:6833/392417778 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 1 -- 172.21.15.67:0/3596013872 <== mds.0 v2:172.21.15.67:6833/392417778 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5636787d32e0 con 0x563678a057e0 2024-04-17T10:29:09.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 10 client.6831 handle_command_reply: tid=0 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 client.6831 shutdown 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 msgr2=0x5636789cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789cca80 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8000bf0 tx=0x7f8fa80827e0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 shutdown_connections 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:6833/392417778,v1:172.21.15.67:6835/392417778] conn(0x563678a057e0 0x563678a05bc0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5636789c1240 0x563678a052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5636789c0920 0x5636789ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fa6000640 0 client.6831 ms_handle_reset on v2:172.21.15.67:6833/392417778 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 --2- 172.21.15.67:0/3596013872 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x563678976f30 0x5636789cca80 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 >> 172.21.15.67:0/3596013872 conn(0x56367897d2a0 msgr2=0x56367897a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 shutdown_connections 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 1 -- 172.21.15.67:0/3596013872 wait complete. 2024-04-17T10:29:09.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.525+0000 7f8fc31a9000 20 client.6831 trim_cache size 0 max 16384 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6651, 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.487816898, 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 40, 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 67, 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 67, 2024-04-17T10:29:09.526 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 15.328231841999999 2024-04-17T10:29:09.527 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:29:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f94066f60 msgr2=0x7f8f94069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f94066f60 0x7f8f94069420 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f8fa00b2000 tx=0x7f8fa0009040 comp rx=0 tx=0).stop 2024-04-17T10:29:09.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 msgr2=0x7f8fb80b4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb80b4050 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f8fa800c370 tx=0x7f8fa800c830 comp rx=0 tx=0).stop 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 shutdown_connections 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f94066f60 0x7f8f94069420 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8fb8145040 0x7f8fb815e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8fb80cc300 0x7f8fb815ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 --2- 172.21.15.67:0/791596614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8fb8071310 0x7f8fb80b4050 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 >> 172.21.15.67:0/791596614 conn(0x7f8fb806ac90 msgr2=0x7f8fb80af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 shutdown_connections 2024-04-17T10:29:09.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.529+0000 7f8fbfc00640 1 -- 172.21.15.67:0/791596614 wait complete. 2024-04-17T10:29:09.541 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:29:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.685+0000 7f47cae00640 1 Processor -- start 2024-04-17T10:29:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.685+0000 7f47cae00640 1 -- start start 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47cae00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47c406e6e0 0x7f47c40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47cae00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47c406eac0 0x7f47c40ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47cae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4143ae0 0x7f47c4143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47cae00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c40cf430 con 0x7f47c406eac0 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47cae00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c415ae40 con 0x7f47c406e6e0 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47cae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c4071440 con 0x7f47c4143ae0 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47c406eac0 0x7f47c40ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47c406eac0 0x7f47c40ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55896/0 (socket says 172.21.15.67:55896) 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 -- 172.21.15.67:0/2322794554 learned_addr learned my addr 172.21.15.67:0/2322794554 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47c8e00640 1 --2- 172.21.15.67:0/2322794554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4143ae0 0x7f47c4143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47c3e00640 1 --2- 172.21.15.67:0/2322794554 >> v2:172.21.15.86:3300/0 conn(0x7f47c406e6e0 0x7f47c40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 -- 172.21.15.67:0/2322794554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4143ae0 msgr2=0x7f47c4143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 --2- 172.21.15.67:0/2322794554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4143ae0 0x7f47c4143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 -- 172.21.15.67:0/2322794554 >> v2:172.21.15.86:3300/0 conn(0x7f47c406e6e0 msgr2=0x7f47c40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 --2- 172.21.15.67:0/2322794554 >> v2:172.21.15.86:3300/0 conn(0x7f47c406e6e0 0x7f47c40ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.689+0000 7f47bb400640 1 -- 172.21.15.67:0/2322794554 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c4071720 con 0x7f47c406eac0 2024-04-17T10:29:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47bb400640 1 --2- 172.21.15.67:0/2322794554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c40ceef0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f47c40cf620 tx=0x7f47ac01eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa0e320a6be051b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47c3400640 1 -- 172.21.15.67:0/2322794554 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0b4070 con 0x7f47c406eac0 2024-04-17T10:29:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47c3400640 1 -- 172.21.15.67:0/2322794554 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac0a7bc0 con 0x7f47c406eac0 2024-04-17T10:29:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47c3400640 1 -- 172.21.15.67:0/2322794554 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0af440 con 0x7f47c406eac0 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 -- 172.21.15.67:0/2322794554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 msgr2=0x7f47c40ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 --2- 172.21.15.67:0/2322794554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c40ceef0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f47c40cf620 tx=0x7f47ac01eb30 comp rx=0 tx=0).stop 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 -- 172.21.15.67:0/2322794554 shutdown_connections 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 --2- 172.21.15.67:0/2322794554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4143ae0 0x7f47c4143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 --2- 172.21.15.67:0/2322794554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c40ceef0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 --2- 172.21.15.67:0/2322794554 >> v2:172.21.15.86:3300/0 conn(0x7f47c406e6e0 0x7f47c40ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 -- 172.21.15.67:0/2322794554 >> 172.21.15.67:0/2322794554 conn(0x7f47c406ac90 msgr2=0x7f47c406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 -- 172.21.15.67:0/2322794554 shutdown_connections 2024-04-17T10:29:09.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 -- 172.21.15.67:0/2322794554 wait complete. 2024-04-17T10:29:09.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 Processor -- start 2024-04-17T10:29:09.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.693+0000 7f47cae00640 1 -- start start 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c406e6e0 0x7f47c4159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c4141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4143ae0 0x7f47c413d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c415ebf0 con 0x7f47c406eac0 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c41444b0 con 0x7f47c4143ae0 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c415ed70 con 0x7f47c406e6e0 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c4141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c4141410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55898/0 (socket says 172.21.15.67:55898) 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 -- 172.21.15.67:0/1840023831 learned_addr learned my addr 172.21.15.67:0/1840023831 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:09.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47c3e00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c406e6e0 0x7f47c4159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47c8e00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4143ae0 0x7f47c413d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 -- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c406e6e0 msgr2=0x7f47c4159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c406e6e0 0x7f47c4159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 -- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4143ae0 msgr2=0x7f47c413d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4143ae0 0x7f47c413d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 -- 172.21.15.67:0/1840023831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c413e440 con 0x7f47c406eac0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47bb400640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c4141410 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f47ac01d1e0 tx=0x7f47ac01b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47c0c00640 1 -- 172.21.15.67:0/1840023831 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0b4070 con 0x7f47c406eac0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47c0c00640 1 -- 172.21.15.67:0/1840023831 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac0b6070 con 0x7f47c406eac0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c4140930 con 0x7f47c406eac0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47c0c00640 1 -- 172.21.15.67:0/1840023831 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0bb470 con 0x7f47c406eac0 2024-04-17T10:29:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c4140ef0 con 0x7f47c406eac0 2024-04-17T10:29:09.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c4072d50 con 0x7f47c406eac0 2024-04-17T10:29:09.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.697+0000 7f47c0c00640 1 -- 172.21.15.67:0/1840023831 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f47ac0bb610 con 0x7f47c406eac0 2024-04-17T10:29:09.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.701+0000 7f47c0c00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:09.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.701+0000 7f47c0c00640 1 -- 172.21.15.67:0/1840023831 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f47ac11f850 con 0x7f47c406eac0 2024-04-17T10:29:09.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.701+0000 7f47c3e00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:09.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.701+0000 7f47c3e00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009b60 tx=0x7f47b4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:09.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.705+0000 7f47c0c00640 1 -- 172.21.15.67:0/1840023831 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f47ac0187e0 con 0x7f47c406eac0 2024-04-17T10:29:09.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.901+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f47c4074e30 con 0x7f47c406eac0 2024-04-17T10:29:09.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47c0c00640 1 -- 172.21.15.67:0/1840023831 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7f47ac0effe0 con 0x7f47c406eac0 2024-04-17T10:29:09.898 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:29:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 msgr2=0x7f47a0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f47b4009b60 tx=0x7f47b4009190 comp rx=0 tx=0).stop 2024-04-17T10:29:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 msgr2=0x7f47c4141410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c4141410 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f47ac01d1e0 tx=0x7f47ac01b8d0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 shutdown_connections 2024-04-17T10:29:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4143ae0 0x7f47c413d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c406eac0 0x7f47c4141410 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 --2- 172.21.15.67:0/1840023831 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c406e6e0 0x7f47c4159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 >> 172.21.15.67:0/1840023831 conn(0x7f47c406ac90 msgr2=0x7f47c40ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 shutdown_connections 2024-04-17T10:29:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:09.905+0000 7f47cae00640 1 -- 172.21.15.67:0/1840023831 wait complete. 2024-04-17T10:29:09.919 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:29:09.920 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:09.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:09.922 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:29:09.930 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:09.931 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42197sec preferred_lft 42197sec 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.932 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:29:09.933 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:09.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:09.934 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:29:09.999 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:29:10.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.001 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:29:10.057 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:10.058 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:10.058 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:10.058 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:29:10.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.058 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:29:10.058 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:29:10.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:29:10.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:29:10.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:29:10.059 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:29:10.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.156 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:29:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:29:10.165 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:29:10.166 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:10.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:29:10.229 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:29:10.229 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:10.277 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:10.278 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:29:10.278 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:10.328 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:29:10.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:10.419 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:10.419 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:29:10.432 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:10.432 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:29:10.432 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:29:10.432 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:29:10.432 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:29:10.436 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:29:10.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:10.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.457 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:29:10.518 INFO:teuthology.orchestra.run.smithi067.stdout:client6005 2024-04-17T10:29:10.519 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:10.519 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:10.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.570 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir scrub_repair_path) 2024-04-17T10:29:10.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:10.628 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch scrub_repair_path/file) 2024-04-17T10:29:10.679 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-04-17T10:29:10.680 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:29:10.680 DEBUG:teuthology.orchestra.run.smithi067:> import stat 2024-04-17T10:29:10.680 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:29:10.680 DEBUG:teuthology.orchestra.run.smithi067:> print(os.stat("/home/ubuntu/cephtest/mnt.0/scrub_repair_path").st_ino) 2024-04-17T10:29:10.680 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:29:10.790 INFO:teuthology.orchestra.run.smithi067.stdout:1099511627776 2024-04-17T10:29:10.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T10:29:10.800 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:10.800 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:10.849 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:10.849 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:10.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:11.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e4b400640 1 Processor -- start 2024-04-17T10:29:11.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e4b400640 1 -- start start 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e4b400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6e440ce020 0x7f6e441620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e4b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e440ce400 0x7f6e4415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e4b400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4415eb50 con 0x7f6e4415e550 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e4b400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e44161650 con 0x7f6e440ce400 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e4b400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e441625e0 con 0x7f6e440ce020 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e43e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e440ce400 0x7f6e4415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e43e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6e440ce400 0x7f6e4415e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48626/0 (socket says 172.21.15.67:48626) 2024-04-17T10:29:11.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e43e00640 1 -- 172.21.15.67:0/381758360 learned_addr learned my addr 172.21.15.67:0/381758360 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e48a00640 1 --2- 172.21.15.67:0/381758360 >> v2:172.21.15.86:3300/0 conn(0x7f6e440ce020 0x7f6e441620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e43e00640 1 -- 172.21.15.67:0/381758360 >> v2:172.21.15.86:3300/0 conn(0x7f6e440ce020 msgr2=0x7f6e441620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e43e00640 1 --2- 172.21.15.67:0/381758360 >> v2:172.21.15.86:3300/0 conn(0x7f6e440ce020 0x7f6e441620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e43e00640 1 -- 172.21.15.67:0/381758360 >> v1:172.21.15.86:6789/0 conn(0x7f6e4415e550 legacy=0x7f6e440cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e43e00640 1 -- 172.21.15.67:0/381758360 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4415ee30 con 0x7f6e440ce400 2024-04-17T10:29:11.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.097+0000 7f6e48a00640 1 --2- 172.21.15.67:0/381758360 >> v2:172.21.15.86:3300/0 conn(0x7f6e440ce020 0x7f6e441620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:11.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e43e00640 1 --2- 172.21.15.67:0/381758360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e4415e010 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009870 tx=0x7f6e3001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471c59180fcd6c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e43400640 1 -- 172.21.15.67:0/381758360 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300b2070 con 0x7f6e440ce400 2024-04-17T10:29:11.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e43400640 1 -- 172.21.15.67:0/381758360 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e300ad640 con 0x7f6e440ce400 2024-04-17T10:29:11.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e43400640 1 -- 172.21.15.67:0/381758360 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300ada10 con 0x7f6e440ce400 2024-04-17T10:29:11.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 -- 172.21.15.67:0/381758360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 msgr2=0x7f6e4415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 --2- 172.21.15.67:0/381758360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e4415e010 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009870 tx=0x7f6e3001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 -- 172.21.15.67:0/381758360 shutdown_connections 2024-04-17T10:29:11.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 --2- 172.21.15.67:0/381758360 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e4415e010 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 --2- 172.21.15.67:0/381758360 >> v2:172.21.15.86:3300/0 conn(0x7f6e440ce020 0x7f6e441620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 -- 172.21.15.67:0/381758360 >> 172.21.15.67:0/381758360 conn(0x7f6e440b33f0 msgr2=0x7f6e440b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:11.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 -- 172.21.15.67:0/381758360 shutdown_connections 2024-04-17T10:29:11.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.101+0000 7f6e4b400640 1 -- 172.21.15.67:0/381758360 wait complete. 2024-04-17T10:29:11.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 Processor -- start 2024-04-17T10:29:11.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 -- start start 2024-04-17T10:29:11.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e440ce020 0x7f6e44143160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e440b1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4415e550 0x7f6e44153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4406c0f0 con 0x7f6e440ce400 2024-04-17T10:29:11.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4406bf70 con 0x7f6e4415e550 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e440c84b0 con 0x7f6e440ce020 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e440b1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e440b1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48642/0 (socket says 172.21.15.67:48642) 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 -- 172.21.15.67:0/112515834 learned_addr learned my addr 172.21.15.67:0/112515834 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e49400640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4415e550 0x7f6e44153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e48a00640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e440ce020 0x7f6e44143160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 -- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e440ce020 msgr2=0x7f6e44143160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e440ce020 0x7f6e44143160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 -- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4415e550 msgr2=0x7f6e44153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4415e550 0x7f6e44153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 -- 172.21.15.67:0/112515834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e441436a0 con 0x7f6e440ce400 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e43e00640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e440b1630 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6e300b0040 tx=0x7f6e30019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e40c00640 1 -- 172.21.15.67:0/112515834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300bd070 con 0x7f6e440ce400 2024-04-17T10:29:11.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e40c00640 1 -- 172.21.15.67:0/112515834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e300b6790 con 0x7f6e440ce400 2024-04-17T10:29:11.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e40c00640 1 -- 172.21.15.67:0/112515834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e300b2070 con 0x7f6e440ce400 2024-04-17T10:29:11.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e440b2010 con 0x7f6e440ce400 2024-04-17T10:29:11.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e440b25d0 con 0x7f6e440ce400 2024-04-17T10:29:11.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.105+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e44070ab0 con 0x7f6e440ce400 2024-04-17T10:29:11.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.109+0000 7f6e40c00640 1 -- 172.21.15.67:0/112515834 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6e300b6b80 con 0x7f6e440ce400 2024-04-17T10:29:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.109+0000 7f6e40c00640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e24066f60 0x7f6e24069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.109+0000 7f6e40c00640 1 -- 172.21.15.67:0/112515834 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6e3011eff0 con 0x7f6e440ce400 2024-04-17T10:29:11.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.109+0000 7f6e48a00640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e24066f60 0x7f6e24069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.109+0000 7f6e48a00640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e24066f60 0x7f6e24069420 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800a9a0 tx=0x7f6e38008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.113+0000 7f6e40c00640 1 -- 172.21.15.67:0/112515834 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6e30016090 con 0x7f6e440ce400 2024-04-17T10:29:11.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.277+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e440b0970 con 0x7f6e440ce400 2024-04-17T10:29:11.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.277+0000 7f6e40c00640 1 -- 172.21.15.67:0/112515834 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16836 (secure 0 0 0) 0x7f6e300efe10 con 0x7f6e440ce400 2024-04-17T10:29:11.272 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:11.272 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":67,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:00.041735+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e24066f60 msgr2=0x7f6e24069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e24066f60 0x7f6e24069420 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800a9a0 tx=0x7f6e38008040 comp rx=0 tx=0).stop 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 msgr2=0x7f6e440b1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e440b1630 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f6e300b0040 tx=0x7f6e30019240 comp rx=0 tx=0).stop 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 shutdown_connections 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e24066f60 0x7f6e24069420 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6e4415e550 0x7f6e44153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6e440ce400 0x7f6e440b1630 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 --2- 172.21.15.67:0/112515834 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6e440ce020 0x7f6e44143160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 >> 172.21.15.67:0/112515834 conn(0x7f6e440b33f0 msgr2=0x7f6e440cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 shutdown_connections 2024-04-17T10:29:11.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.281+0000 7f6e4b400640 1 -- 172.21.15.67:0/112515834 wait complete. 2024-04-17T10:29:11.294 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:28:56.861344+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:28:57.434569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:29:11.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:29:11.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9992600640 1 Processor -- start 2024-04-17T10:29:11.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9992600640 1 -- start start 2024-04-17T10:29:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9992600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f998c161110 0x7f998c0d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9992600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f998c0b05c0 0x7f998c0b10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9992600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c0d1290 con 0x7f998c0b1630 2024-04-17T10:29:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9992600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998c1614f0 con 0x7f998c161110 2024-04-17T10:29:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9992600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998c0d1040 con 0x7f998c0b05c0 2024-04-17T10:29:11.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9991200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f998c161110 0x7f998c0d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9991c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f998c0b1630 0x7f998c099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40076/0 (socket says 172.21.15.67:40076) 2024-04-17T10:29:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9991c00640 1 -- 172.21.15.67:0/3383587088 learned_addr learned my addr 172.21.15.67:0/3383587088 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9983e00640 1 --2- 172.21.15.67:0/3383587088 >> v2:172.21.15.86:3300/0 conn(0x7f998c0b05c0 0x7f998c0b10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9991200640 1 -- 172.21.15.67:0/3383587088 >> v2:172.21.15.86:3300/0 conn(0x7f998c0b05c0 msgr2=0x7f998c0b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9991200640 1 --2- 172.21.15.67:0/3383587088 >> v2:172.21.15.86:3300/0 conn(0x7f998c0b05c0 0x7f998c0b10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9991200640 1 -- 172.21.15.67:0/3383587088 >> v1:172.21.15.67:6789/0 conn(0x7f998c0b1630 legacy=0x7f998c099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.445+0000 7f9991200640 1 -- 172.21.15.67:0/3383587088 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c0b2a50 con 0x7f998c161110 2024-04-17T10:29:11.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9991200640 1 --2- 172.21.15.67:0/3383587088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c161110 0x7f998c0d05c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f998c071df0 tx=0x7f997c01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dc59e65fb50148 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f998be00640 1 -- 172.21.15.67:0/3383587088 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c0b0070 con 0x7f998c161110 2024-04-17T10:29:11.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f998be00640 1 -- 172.21.15.67:0/3383587088 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997c0ab600 con 0x7f998c161110 2024-04-17T10:29:11.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f998be00640 1 -- 172.21.15.67:0/3383587088 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c0ab9d0 con 0x7f998c161110 2024-04-17T10:29:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 -- 172.21.15.67:0/3383587088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c161110 msgr2=0x7f998c0d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 --2- 172.21.15.67:0/3383587088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c161110 0x7f998c0d05c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f998c071df0 tx=0x7f997c01c390 comp rx=0 tx=0).stop 2024-04-17T10:29:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 -- 172.21.15.67:0/3383587088 shutdown_connections 2024-04-17T10:29:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 --2- 172.21.15.67:0/3383587088 >> v2:172.21.15.86:3300/0 conn(0x7f998c0b05c0 0x7f998c0b10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 --2- 172.21.15.67:0/3383587088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c161110 0x7f998c0d05c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 -- 172.21.15.67:0/3383587088 >> 172.21.15.67:0/3383587088 conn(0x7f998c0d16d0 msgr2=0x7f998c0d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:11.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 -- 172.21.15.67:0/3383587088 shutdown_connections 2024-04-17T10:29:11.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.449+0000 7f9992600640 1 -- 172.21.15.67:0/3383587088 wait complete. 2024-04-17T10:29:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 Processor -- start 2024-04-17T10:29:11.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 -- start start 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f998c0b05c0 0x7f998c0c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c0b1630 0x7f998c0af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f998c161110 0x7f998c152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0b1bc0 con 0x7f998c0b1630 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c0b1a40 con 0x7f998c161110 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c16a750 con 0x7f998c0b05c0 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c0b1630 0x7f998c0af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c0b1630 0x7f998c0af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48662/0 (socket says 172.21.15.67:48662) 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 -- 172.21.15.67:0/627030022 learned_addr learned my addr 172.21.15.67:0/627030022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9991c00640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f998c161110 0x7f998c152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9991200640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f998c0b05c0 0x7f998c0c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 -- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f998c0b05c0 msgr2=0x7f998c0c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f998c0b05c0 0x7f998c0c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 -- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f998c161110 msgr2=0x7f998c152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f998c161110 0x7f998c152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 -- 172.21.15.67:0/627030022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c154ac0 con 0x7f998c0b1630 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9991200640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f998c0b05c0 0x7f998c0c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:11.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9983e00640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c0b1630 0x7f998c0af000 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f997400d5f0 tx=0x7f997400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9989600640 1 -- 172.21.15.67:0/627030022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740125f0 con 0x7f998c0b1630 2024-04-17T10:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9989600640 1 -- 172.21.15.67:0/627030022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974019070 con 0x7f998c0b1630 2024-04-17T10:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c153740 con 0x7f998c0b1630 2024-04-17T10:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9989600640 1 -- 172.21.15.67:0/627030022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740129b0 con 0x7f998c0b1630 2024-04-17T10:29:11.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c153d00 con 0x7f998c0b1630 2024-04-17T10:29:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.453+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c1608d0 con 0x7f998c0b1630 2024-04-17T10:29:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.457+0000 7f9989600640 1 -- 172.21.15.67:0/627030022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9974012b50 con 0x7f998c0b1630 2024-04-17T10:29:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.457+0000 7f9989600640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9968066f60 0x7f9968069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.457+0000 7f9989600640 1 -- 172.21.15.67:0/627030022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f99740855e0 con 0x7f998c0b1630 2024-04-17T10:29:11.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.457+0000 7f9991200640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9968066f60 0x7f9968069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.457+0000 7f9991200640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9968066f60 0x7f9968069420 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f997c0ad000 tx=0x7f997c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.461+0000 7f9989600640 1 -- 172.21.15.67:0/627030022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f998c154ac0 con 0x7f998c0b1630 2024-04-17T10:29:11.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.621+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f998c0ae9f0 con 0x7f998c0b1630 2024-04-17T10:29:11.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.621+0000 7f9989600640 1 -- 172.21.15.67:0/627030022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9974056450 con 0x7f998c0b1630 2024-04-17T10:29:11.618 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:29:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9968066f60 msgr2=0x7f9968069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9968066f60 0x7f9968069420 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f997c0ad000 tx=0x7f997c008040 comp rx=0 tx=0).stop 2024-04-17T10:29:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c0b1630 msgr2=0x7f998c0af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c0b1630 0x7f998c0af000 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f997400d5f0 tx=0x7f997400dab0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 shutdown_connections 2024-04-17T10:29:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9968066f60 0x7f9968069420 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f998c161110 0x7f998c152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f998c0b1630 0x7f998c0af000 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 --2- 172.21.15.67:0/627030022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f998c0b05c0 0x7f998c0c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 >> 172.21.15.67:0/627030022 conn(0x7f998c0d16d0 msgr2=0x7f998c15b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 shutdown_connections 2024-04-17T10:29:11.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.625+0000 7f9992600640 1 -- 172.21.15.67:0/627030022 wait complete. 2024-04-17T10:29:11.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:29:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff858000640 1 Processor -- start 2024-04-17T10:29:11.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff858000640 1 -- start start 2024-04-17T10:29:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff858000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff85006b3a0 0x7ff8500b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff858000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8501616a0 con 0x7ff8501474c0 2024-04-17T10:29:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff858000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff850161940 con 0x7ff8500b3920 2024-04-17T10:29:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff858000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff850161c20 con 0x7ff85006b3a0 2024-04-17T10:29:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff855600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8501474c0 0x7ff85006ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40090/0 (socket says 172.21.15.67:40090) 2024-04-17T10:29:11.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff855600640 1 -- 172.21.15.67:0/714312898 learned_addr learned my addr 172.21.15.67:0/714312898 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202068215 0 0) 0x7ff8501616a0 con 0x7ff8501474c0 2024-04-17T10:29:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff838003530 con 0x7ff8501474c0 2024-04-17T10:29:11.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.789+0000 7ff854c00640 1 --2- 172.21.15.67:0/714312898 >> v2:172.21.15.86:3300/0 conn(0x7ff85006b3a0 0x7ff8500b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843531442 0 0) 0x7ff838003530 con 0x7ff8501474c0 2024-04-17T10:29:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 >> v2:172.21.15.86:3300/0 conn(0x7ff85006b3a0 msgr2=0x7ff8500b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 --2- 172.21.15.67:0/714312898 >> v2:172.21.15.86:3300/0 conn(0x7ff85006b3a0 0x7ff8500b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 >> v1:172.21.15.86:6789/0 conn(0x7ff8500b3920 legacy=0x7ff8500d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850161f00 con 0x7ff8501474c0 2024-04-17T10:29:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7ff844001680 con 0x7ff8501474c0 2024-04-17T10:29:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 >> v1:172.21.15.67:6789/0 conn(0x7ff8501474c0 legacy=0x7ff85006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff838003a10 0x7ff838003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff838004810 0x7ff838004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff83800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850161c20 con 0x7ff8380051d0 2024-04-17T10:29:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff83800c0b0 con 0x7ff838003a10 2024-04-17T10:29:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83800c2b0 con 0x7ff838004810 2024-04-17T10:29:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff855600640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff838003a10 0x7ff838003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff83800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff854c00640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff838004810 0x7ff838004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 -- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff838004810 msgr2=0x7ff838004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff838004810 0x7ff838004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 -- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff838003a10 msgr2=0x7ff838003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff838003a10 0x7ff838003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 -- 172.21.15.67:0/714312898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff838003530 con 0x7ff8380051d0 2024-04-17T10:29:11.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff855600640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff838003a10 0x7ff838003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 -- 172.21.15.67:0/714312898 reap_dead start 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.793+0000 7ff856000640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff83800b9d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff848001e70 tx=0x7ff84801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c690d90586addb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff844007c50 con 0x7ff8380051d0 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff848003bc0 con 0x7ff8380051d0 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff84fe00640 1 -- 172.21.15.67:0/714312898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848001a90 con 0x7ff8380051d0 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- 172.21.15.67:0/714312898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 msgr2=0x7ff83800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff83800b9d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff848001e70 tx=0x7ff84801c780 comp rx=0 tx=0).stop 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- 172.21.15.67:0/714312898 shutdown_connections 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 --2- 172.21.15.67:0/714312898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff83800b9d0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- 172.21.15.67:0/714312898 >> 172.21.15.67:0/714312898 conn(0x7ff8500b5ef0 msgr2=0x7ff850099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- 172.21.15.67:0/714312898 shutdown_connections 2024-04-17T10:29:11.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- 172.21.15.67:0/714312898 wait complete. 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 Processor -- start 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- start start 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff8501f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8500b27e0 0x7ff85015cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8500b2c60 0x7ff85015d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85006ac90 con 0x7ff8380051d0 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850162420 con 0x7ff8500b27e0 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff858000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85006ae10 con 0x7ff8500b2c60 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff855600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff8501f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff855600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff8501f45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48680/0 (socket says 172.21.15.67:48680) 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.797+0000 7ff855600640 1 -- 172.21.15.67:0/1472642469 learned_addr learned my addr 172.21.15.67:0/1472642469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff854c00640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8500b27e0 0x7ff85015cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff856000640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8500b2c60 0x7ff85015d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff855600640 1 -- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8500b2c60 msgr2=0x7ff85015d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff855600640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8500b2c60 0x7ff85015d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff855600640 1 -- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8500b27e0 msgr2=0x7ff85015cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff855600640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8500b27e0 0x7ff85015cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff855600640 1 -- 172.21.15.67:0/1472642469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff850160810 con 0x7ff8380051d0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff855600640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff8501f45d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff844007290 tx=0x7ff844007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff84d600640 1 -- 172.21.15.67:0/1472642469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400f9c0 con 0x7ff8380051d0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff84d600640 1 -- 172.21.15.67:0/1472642469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff844018070 con 0x7ff8380051d0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff84d600640 1 -- 172.21.15.67:0/1472642469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400fd60 con 0x7ff8380051d0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85015f3a0 con 0x7ff8380051d0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85015f960 con 0x7ff8380051d0 2024-04-17T10:29:11.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85004ef30 con 0x7ff8380051d0 2024-04-17T10:29:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff84d600640 1 -- 172.21.15.67:0/1472642469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff84401d460 con 0x7ff8380051d0 2024-04-17T10:29:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.801+0000 7ff84d600640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff830066f90 0x7ff830069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.805+0000 7ff84d600640 1 -- 172.21.15.67:0/1472642469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff844085300 con 0x7ff8380051d0 2024-04-17T10:29:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.805+0000 7ff854c00640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff830066f90 0x7ff830069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:11.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.805+0000 7ff854c00640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff830066f90 0x7ff830069450 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff850138fe0 tx=0x7ff83c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:11.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.809+0000 7ff84d600640 1 -- 172.21.15.67:0/1472642469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff8440560e0 con 0x7ff8380051d0 2024-04-17T10:29:11.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff85009a310 con 0x7ff8380051d0 2024-04-17T10:29:11.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff84d600640 1 -- 172.21.15.67:0/1472642469 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff8440559e0 con 0x7ff8380051d0 2024-04-17T10:29:11.963 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff830066f90 msgr2=0x7ff830069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff830066f90 0x7ff830069450 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff850138fe0 tx=0x7ff83c008040 comp rx=0 tx=0).stop 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 msgr2=0x7ff8501f45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff8501f45d0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7ff844007290 tx=0x7ff844007750 comp rx=0 tx=0).stop 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 shutdown_connections 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff830066f90 0x7ff830069450 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8500b2c60 0x7ff85015d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8500b27e0 0x7ff85015cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 --2- 172.21.15.67:0/1472642469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8380051d0 0x7ff8501f45d0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 >> 172.21.15.67:0/1472642469 conn(0x7ff8500b5ef0 msgr2=0x7ff85006f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 shutdown_connections 2024-04-17T10:29:11.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:11.969+0000 7ff858000640 1 -- 172.21.15.67:0/1472642469 wait complete. 2024-04-17T10:29:11.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:29:12.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.133+0000 7f195e400640 1 Processor -- start 2024-04-17T10:29:12.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f195e400640 1 -- start start 2024-04-17T10:29:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f195e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1958159440 0x7f19580ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f195e400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19580713e0 0x7f1958071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f195e400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195815cb30 con 0x7f195815be70 2024-04-17T10:29:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f195e400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19580a8650 con 0x7f1958159440 2024-04-17T10:29:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f195e400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f195815ce10 con 0x7f19580713e0 2024-04-17T10:29:12.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1958159440 0x7f19580ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1958159440 0x7f19580ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48684/0 (socket says 172.21.15.67:48684) 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957400640 1 -- 172.21.15.67:0/3495350112 learned_addr learned my addr 172.21.15.67:0/3495350112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957e00640 1 --1- 172.21.15.67:0/3495350112 >> v1:172.21.15.67:6789/0 conn(0x7f195815be70 0x7f195815c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40102/0 (socket says 172.21.15.67:40102) 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f194ea00640 1 --2- 172.21.15.67:0/3495350112 >> v2:172.21.15.86:3300/0 conn(0x7f19580713e0 0x7f1958071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957400640 1 -- 172.21.15.67:0/3495350112 >> v2:172.21.15.86:3300/0 conn(0x7f19580713e0 msgr2=0x7f1958071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957400640 1 --2- 172.21.15.67:0/3495350112 >> v2:172.21.15.86:3300/0 conn(0x7f19580713e0 0x7f1958071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957400640 1 -- 172.21.15.67:0/3495350112 >> v1:172.21.15.67:6789/0 conn(0x7f195815be70 legacy=0x7f195815c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f1957400640 1 -- 172.21.15.67:0/3495350112 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19580cd610 con 0x7f1958159440 2024-04-17T10:29:12.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.137+0000 7f194ea00640 1 --2- 172.21.15.67:0/3495350112 >> v2:172.21.15.86:3300/0 conn(0x7f19580713e0 0x7f1958071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:12.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f1957400640 1 --2- 172.21.15.67:0/3495350112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f19580ce0f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f195806f840 tx=0x7f194801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a04f2a64d9b1d09 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f1956a00640 1 -- 172.21.15.67:0/3495350112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480ab430 con 0x7f1958159440 2024-04-17T10:29:12.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f1956a00640 1 -- 172.21.15.67:0/3495350112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19480b0070 con 0x7f1958159440 2024-04-17T10:29:12.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f1956a00640 1 -- 172.21.15.67:0/3495350112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480ab7f0 con 0x7f1958159440 2024-04-17T10:29:12.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 -- 172.21.15.67:0/3495350112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 msgr2=0x7f19580ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 --2- 172.21.15.67:0/3495350112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f19580ce0f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f195806f840 tx=0x7f194801c520 comp rx=0 tx=0).stop 2024-04-17T10:29:12.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 -- 172.21.15.67:0/3495350112 shutdown_connections 2024-04-17T10:29:12.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 --2- 172.21.15.67:0/3495350112 >> v2:172.21.15.86:3300/0 conn(0x7f19580713e0 0x7f1958071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 --2- 172.21.15.67:0/3495350112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f19580ce0f0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 -- 172.21.15.67:0/3495350112 >> 172.21.15.67:0/3495350112 conn(0x7f19580b1740 msgr2=0x7f19580b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:12.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 -- 172.21.15.67:0/3495350112 shutdown_connections 2024-04-17T10:29:12.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 -- 172.21.15.67:0/3495350112 wait complete. 2024-04-17T10:29:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 Processor -- start 2024-04-17T10:29:12.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.141+0000 7f195e400640 1 -- start start 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19580713e0 0x7f1958131e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f1958149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f195815be70 0x7f195815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195815d9c0 con 0x7f1958159440 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195815d840 con 0x7f19580713e0 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f195815db40 con 0x7f195815be70 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f1958149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f1958149340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48690/0 (socket says 172.21.15.67:48690) 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 -- 172.21.15.67:0/489896102 learned_addr learned my addr 172.21.15.67:0/489896102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f1957400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19580713e0 0x7f1958131e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f1957e00640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f195815be70 0x7f195815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 -- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f195815be70 msgr2=0x7f195815ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f195815be70 0x7f195815ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 -- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19580713e0 msgr2=0x7f1958131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19580713e0 0x7f1958131e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 -- 172.21.15.67:0/489896102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1958149d90 con 0x7f1958159440 2024-04-17T10:29:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f1957e00640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f195815be70 0x7f195815ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194ea00640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f1958149340 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f194000e840 tx=0x7f194000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194fe00640 1 -- 172.21.15.67:0/489896102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940017070 con 0x7f1958159440 2024-04-17T10:29:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194fe00640 1 -- 172.21.15.67:0/489896102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194000c8d0 con 0x7f1958159440 2024-04-17T10:29:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194fe00640 1 -- 172.21.15.67:0/489896102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194000f040 con 0x7f1958159440 2024-04-17T10:29:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f195815a0a0 con 0x7f1958159440 2024-04-17T10:29:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f195815a660 con 0x7f1958159440 2024-04-17T10:29:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195804ef30 con 0x7f1958159440 2024-04-17T10:29:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.145+0000 7f194fe00640 1 -- 172.21.15.67:0/489896102 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f194001a070 con 0x7f1958159440 2024-04-17T10:29:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.149+0000 7f194fe00640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.149+0000 7f194fe00640 1 -- 172.21.15.67:0/489896102 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1940084d60 con 0x7f1958159440 2024-04-17T10:29:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.149+0000 7f1957400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.149+0000 7f1957400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f19480af000 tx=0x7f19480a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.153+0000 7f194fe00640 1 -- 172.21.15.67:0/489896102 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f19400510c0 con 0x7f1958159440 2024-04-17T10:29:12.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.313+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f195815e7e0 con 0x7f1958159440 2024-04-17T10:29:12.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.313+0000 7f194fe00640 1 -- 172.21.15.67:0/489896102 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v67) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1940055590 con 0x7f1958159440 2024-04-17T10:29:12.308 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 msgr2=0x7f1934069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f19480af000 tx=0x7f19480a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 msgr2=0x7f1958149340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f1958149340 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f194000e840 tx=0x7f194000ed00 comp rx=0 tx=0).stop 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 shutdown_connections 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f195815be70 0x7f195815ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1958159440 0x7f1958149340 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 --2- 172.21.15.67:0/489896102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19580713e0 0x7f1958131e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 >> 172.21.15.67:0/489896102 conn(0x7f19580b1740 msgr2=0x7f19580af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 shutdown_connections 2024-04-17T10:29:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.317+0000 7f195e400640 1 -- 172.21.15.67:0/489896102 wait complete. 2024-04-17T10:29:12.330 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:29:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.477+0000 7f3701800640 1 Processor -- start 2024-04-17T10:29:12.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.477+0000 7f3701800640 1 -- start start 2024-04-17T10:29:12.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f3701800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc15dbc0 0x7f36fc06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f3701800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f36fc06db90 0x7f36fc071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f3701800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36fc0ad8f0 con 0x7f36fc0721a0 2024-04-17T10:29:12.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f3701800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36fc15dfa0 con 0x7f36fc06db90 2024-04-17T10:29:12.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f3701800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc06faa0 con 0x7f36fc15dbc0 2024-04-17T10:29:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36f3e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f36fc06db90 0x7f36fc071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36faa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc15dbc0 0x7f36fc06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36f3e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f36fc06db90 0x7f36fc071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:53802/0 (socket says 172.21.15.67:53802) 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36f3e00640 1 -- 172.21.15.67:0/3472950131 learned_addr learned my addr 172.21.15.67:0/3472950131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36fb400640 1 --1- 172.21.15.67:0/3472950131 >> v1:172.21.15.86:6789/0 conn(0x7f36fc0721a0 0x7f36fc06b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:40908/0 (socket says 172.21.15.67:40908) 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36faa00640 1 -- 172.21.15.67:0/3472950131 >> v2:172.21.15.86:3300/0 conn(0x7f36fc06db90 msgr2=0x7f36fc071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36faa00640 1 --2- 172.21.15.67:0/3472950131 >> v2:172.21.15.86:3300/0 conn(0x7f36fc06db90 0x7f36fc071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36faa00640 1 -- 172.21.15.67:0/3472950131 >> v1:172.21.15.86:6789/0 conn(0x7f36fc0721a0 legacy=0x7f36fc06b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36faa00640 1 -- 172.21.15.67:0/3472950131 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc0adbd0 con 0x7f36fc15dbc0 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36fa000640 1 -- 172.21.15.67:0/3472950131 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326471544 0 0) 0x7f36fc0ad8f0 con 0x7f36fc0721a0 2024-04-17T10:29:12.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.481+0000 7f36f3e00640 1 --2- 172.21.15.67:0/3472950131 >> v2:172.21.15.86:3300/0 conn(0x7f36fc06db90 0x7f36fc071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:12.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f36faa00640 1 --2- 172.21.15.67:0/3472950131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc15dbc0 0x7f36fc06f560 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0097c0 tx=0x7f36ec01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489869d6fa38ee36 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f36fa000640 1 -- 172.21.15.67:0/3472950131 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0b0070 con 0x7f36fc15dbc0 2024-04-17T10:29:12.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f36fa000640 1 -- 172.21.15.67:0/3472950131 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36ec0ab600 con 0x7f36fc15dbc0 2024-04-17T10:29:12.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f36fa000640 1 -- 172.21.15.67:0/3472950131 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36ec0ab9d0 con 0x7f36fc15dbc0 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- 172.21.15.67:0/3472950131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc15dbc0 msgr2=0x7f36fc06f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 --2- 172.21.15.67:0/3472950131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc15dbc0 0x7f36fc06f560 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0097c0 tx=0x7f36ec01c410 comp rx=0 tx=0).stop 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- 172.21.15.67:0/3472950131 shutdown_connections 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 --2- 172.21.15.67:0/3472950131 >> v2:172.21.15.86:3300/0 conn(0x7f36fc06db90 0x7f36fc071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 --2- 172.21.15.67:0/3472950131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc15dbc0 0x7f36fc06f560 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- 172.21.15.67:0/3472950131 >> 172.21.15.67:0/3472950131 conn(0x7f36fc0af670 msgr2=0x7f36fc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- 172.21.15.67:0/3472950131 shutdown_connections 2024-04-17T10:29:12.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- 172.21.15.67:0/3472950131 wait complete. 2024-04-17T10:29:12.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 Processor -- start 2024-04-17T10:29:12.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- start start 2024-04-17T10:29:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc06db90 0x7f36fc0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36fc0721a0 0x7f36fc149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36fc15dbc0 0x7f36fc13a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc06c6f0 con 0x7f36fc15dbc0 2024-04-17T10:29:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc06c570 con 0x7f36fc0721a0 2024-04-17T10:29:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f3701800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36fc06af70 con 0x7f36fc06db90 2024-04-17T10:29:12.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.485+0000 7f36fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36fc15dbc0 0x7f36fc13a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36fc15dbc0 0x7f36fc13a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48692/0 (socket says 172.21.15.67:48692) 2024-04-17T10:29:12.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 -- 172.21.15.67:0/885408806 learned_addr learned my addr 172.21.15.67:0/885408806 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:12.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36f3e00640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36fc0721a0 0x7f36fc149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36faa00640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc06db90 0x7f36fc0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 -- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc06db90 msgr2=0x7f36fc0cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc06db90 0x7f36fc0cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 -- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36fc0721a0 msgr2=0x7f36fc149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36fc0721a0 0x7f36fc149650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 -- 172.21.15.67:0/885408806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36fc0ceb20 con 0x7f36fc15dbc0 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36fb400640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36fc15dbc0 0x7f36fc13a6f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f36f400c7a0 tx=0x7f36f400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36f2a00640 1 -- 172.21.15.67:0/885408806 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f4018070 con 0x7f36fc15dbc0 2024-04-17T10:29:12.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36f2a00640 1 -- 172.21.15.67:0/885408806 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f400a640 con 0x7f36fc15dbc0 2024-04-17T10:29:12.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36fc0ce560 con 0x7f36fc15dbc0 2024-04-17T10:29:12.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36f2a00640 1 -- 172.21.15.67:0/885408806 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f400aab0 con 0x7f36fc15dbc0 2024-04-17T10:29:12.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36fc15e1c0 con 0x7f36fc15dbc0 2024-04-17T10:29:12.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36fc09e260 con 0x7f36fc15dbc0 2024-04-17T10:29:12.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36f2a00640 1 -- 172.21.15.67:0/885408806 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f36f401b070 con 0x7f36fc15dbc0 2024-04-17T10:29:12.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36f2a00640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36d8066f60 0x7f36d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.489+0000 7f36f2a00640 1 -- 172.21.15.67:0/885408806 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f36f4084850 con 0x7f36fc15dbc0 2024-04-17T10:29:12.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.493+0000 7f36faa00640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36d8066f60 0x7f36d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.493+0000 7f36faa00640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36d8066f60 0x7f36d8069420 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0ad000 tx=0x7f36ec0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.493+0000 7f36f2a00640 1 -- 172.21.15.67:0/885408806 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f36f4055630 con 0x7f36fc15dbc0 2024-04-17T10:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f36fc06b590 con 0x7f36fc15dbc0 2024-04-17T10:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f36f2a00640 1 -- 172.21.15.67:0/885408806 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v67) v1 ==== 80+0+0 (secure 0 0 0) 0x7f36f4054f30 con 0x7f36fc15dbc0 2024-04-17T10:29:12.667 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36d8066f60 msgr2=0x7f36d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36d8066f60 0x7f36d8069420 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f36ec0ad000 tx=0x7f36ec0a6000 comp rx=0 tx=0).stop 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36fc15dbc0 msgr2=0x7f36fc13a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36fc15dbc0 0x7f36fc13a6f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f36f400c7a0 tx=0x7f36f400cc60 comp rx=0 tx=0).stop 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 shutdown_connections 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f36d8066f60 0x7f36d8069420 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36fc15dbc0 0x7f36fc13a6f0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f36fc0721a0 0x7f36fc149650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 --2- 172.21.15.67:0/885408806 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f36fc06db90 0x7f36fc0cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 >> 172.21.15.67:0/885408806 conn(0x7f36fc0af670 msgr2=0x7f36fc0b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 shutdown_connections 2024-04-17T10:29:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.673+0000 7f3701800640 1 -- 172.21.15.67:0/885408806 wait complete. 2024-04-17T10:29:12.688 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:29:12.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:12.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:12.730 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:29:12.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:12.730 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:29:12.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:12.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.969+0000 7ff26aa00640 1 Processor -- start 2024-04-17T10:29:12.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26aa00640 1 -- start start 2024-04-17T10:29:12.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26aa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264160f60 0x7ff264099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff26409a3b0 0x7ff26406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26aa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff26409a840 con 0x7ff264099f90 2024-04-17T10:29:12.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26aa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff26409b170 con 0x7ff26409a3b0 2024-04-17T10:29:12.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26aa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff26406ddf0 con 0x7ff264160f60 2024-04-17T10:29:12.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff26409a3b0 0x7ff26406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff268c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff264099f90 0x7ff2640cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40104/0 (socket says 172.21.15.67:40104) 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff26409a3b0 0x7ff26406cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48708/0 (socket says 172.21.15.67:48708) 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff268c00640 1 -- 172.21.15.67:0/1487457395 learned_addr learned my addr 172.21.15.67:0/1487457395 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff269600640 1 --2- 172.21.15.67:0/1487457395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264160f60 0x7ff264099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26a000640 1 -- 172.21.15.67:0/1487457395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264160f60 msgr2=0x7ff264099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26a000640 1 --2- 172.21.15.67:0/1487457395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264160f60 0x7ff264099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26a000640 1 -- 172.21.15.67:0/1487457395 >> v1:172.21.15.67:6789/0 conn(0x7ff264099f90 legacy=0x7ff2640cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.973+0000 7ff26a000640 1 -- 172.21.15.67:0/1487457395 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26409ab20 con 0x7ff26409a3b0 2024-04-17T10:29:12.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26a000640 1 --2- 172.21.15.67:0/1487457395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff26409a3b0 0x7ff26406cdb0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009d90 tx=0x7ff25c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33bd315ec5560e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff263e00640 1 -- 172.21.15.67:0/1487457395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0b2070 con 0x7ff26409a3b0 2024-04-17T10:29:12.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff263e00640 1 -- 172.21.15.67:0/1487457395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c0a5a20 con 0x7ff26409a3b0 2024-04-17T10:29:12.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff263e00640 1 -- 172.21.15.67:0/1487457395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0a5df0 con 0x7ff26409a3b0 2024-04-17T10:29:12.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 -- 172.21.15.67:0/1487457395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff26409a3b0 msgr2=0x7ff26406cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 --2- 172.21.15.67:0/1487457395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff26409a3b0 0x7ff26406cdb0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7ff25c009d90 tx=0x7ff25c01c940 comp rx=0 tx=0).stop 2024-04-17T10:29:12.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 -- 172.21.15.67:0/1487457395 shutdown_connections 2024-04-17T10:29:12.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 --2- 172.21.15.67:0/1487457395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff26409a3b0 0x7ff26406cdb0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 --2- 172.21.15.67:0/1487457395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264160f60 0x7ff264099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 -- 172.21.15.67:0/1487457395 >> 172.21.15.67:0/1487457395 conn(0x7ff2640d1630 msgr2=0x7ff2640d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:12.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 -- 172.21.15.67:0/1487457395 shutdown_connections 2024-04-17T10:29:12.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 -- 172.21.15.67:0/1487457395 wait complete. 2024-04-17T10:29:12.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 Processor -- start 2024-04-17T10:29:12.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.977+0000 7ff26aa00640 1 -- start start 2024-04-17T10:29:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264099f90 0x7ff2640b7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff264160f60 0x7ff2640c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff264154730 0x7ff26409ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640d0bd0 con 0x7ff264154730 2024-04-17T10:29:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640d0a50 con 0x7ff264160f60 2024-04-17T10:29:12.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2640d0d50 con 0x7ff264099f90 2024-04-17T10:29:12.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff264154730 0x7ff26409ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff264154730 0x7ff26409ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48712/0 (socket says 172.21.15.67:48712) 2024-04-17T10:29:12.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 -- 172.21.15.67:0/2958138577 learned_addr learned my addr 172.21.15.67:0/2958138577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff269600640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264099f90 0x7ff2640b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff268c00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff264160f60 0x7ff2640c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 -- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264099f90 msgr2=0x7ff2640b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264099f90 0x7ff2640b7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 -- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff264160f60 msgr2=0x7ff2640c5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff264160f60 0x7ff2640c5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 -- 172.21.15.67:0/2958138577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2640c6660 con 0x7ff264154730 2024-04-17T10:29:12.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff269600640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264099f90 0x7ff2640b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff268c00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff264160f60 0x7ff2640c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26a000640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff264154730 0x7ff26409ea90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0b0040 tx=0x7ff25c0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff261600640 1 -- 172.21.15.67:0/2958138577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0b2050 con 0x7ff264154730 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff261600640 1 -- 172.21.15.67:0/2958138577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25c0be070 con 0x7ff264154730 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff261600640 1 -- 172.21.15.67:0/2958138577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25c0addc0 con 0x7ff264154730 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2640c6940 con 0x7ff264154730 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2640c6f00 con 0x7ff264154730 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff264074dc0 con 0x7ff264154730 2024-04-17T10:29:12.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.981+0000 7ff261600640 1 -- 172.21.15.67:0/2958138577 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff25c0c3460 con 0x7ff264154730 2024-04-17T10:29:12.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.985+0000 7ff261600640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff244066f60 0x7ff244069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:12.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.985+0000 7ff261600640 1 -- 172.21.15.67:0/2958138577 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff25c120700 con 0x7ff264154730 2024-04-17T10:29:12.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.985+0000 7ff269600640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff244066f60 0x7ff244069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:12.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.985+0000 7ff269600640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff244066f60 0x7ff244069420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff258009ae0 tx=0x7ff258008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:12.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:12.989+0000 7ff261600640 1 -- 172.21.15.67:0/2958138577 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff25c016560 con 0x7ff264154730 2024-04-17T10:29:13.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.189+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2640b1be0 con 0x7ff264154730 2024-04-17T10:29:13.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.189+0000 7ff261600640 1 -- 172.21.15.67:0/2958138577 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4936 (secure 0 0 0) 0x7ff25c11ee20 con 0x7ff264154730 2024-04-17T10:29:13.185 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:13.185 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":40,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5897,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1277872832","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1277872832},{"type":"v1","addr":"172.21.15.86:6836","nonce":1277872832}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5900,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3787337522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":3787337522},{"type":"v1","addr":"172.21.15.86:6837","nonce":3787337522}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6648,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3103235412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3103235412},{"type":"v1","addr":"172.21.15.67:6834","nonce":3103235412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:01.462609+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6651},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6651":{"gid":6651,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/392417778","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":392417778},{"type":"v1","addr":"172.21.15.67:6835","nonce":392417778}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:13.185 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 40 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff244066f60 msgr2=0x7ff244069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff244066f60 0x7ff244069420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7ff258009ae0 tx=0x7ff258008040 comp rx=0 tx=0).stop 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff264154730 msgr2=0x7ff26409ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff264154730 0x7ff26409ea90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff25c0b0040 tx=0x7ff25c0196e0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 shutdown_connections 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff244066f60 0x7ff244069420 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff264154730 0x7ff26409ea90 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff264160f60 0x7ff2640c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 --2- 172.21.15.67:0/2958138577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff264099f90 0x7ff2640b7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 >> 172.21.15.67:0/2958138577 conn(0x7ff2640d1630 msgr2=0x7ff26415ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:13.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 shutdown_connections 2024-04-17T10:29:13.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.193+0000 7ff26aa00640 1 -- 172.21.15.67:0/2958138577 wait complete. 2024-04-17T10:29:13.206 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-04-17T10:29:13.378 INFO:teuthology.orchestra.run.smithi067.stdout:{"message":"","return_code":0} 2024-04-17T10:29:13.378 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-04-17T10:29:13.379 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:29:13.379 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:13.379 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:29:13.380 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:13.380 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:29:13.380 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:13.380 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:29:13.380 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:13.393 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:29:13.393 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:13.395 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:29:13.395 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:13.396 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:29:13.397 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:13.397 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:29:13.397 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:13.398 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000000.00000000 file_head 2024-04-17T10:29:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 Processor -- start 2024-04-17T10:29:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 -- start start 2024-04-17T10:29:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a919500 0x55916a8f1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 0x55916a9ac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a98e040 0x55916a9acce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916a8d0920 con 0x55916a98e040 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916a8d0bc0 con 0x55916a919500 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55916a8d0e60 con 0x55916a9198e0 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a98e040 0x55916a9acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a98e040 0x55916a9acce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48730/0 (socket says 172.21.15.67:48730) 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 -- 172.21.15.67:0/1899217084 learned_addr learned my addr 172.21.15.67:0/1899217084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe767e00640 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 0x55916a9ac7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76cc00640 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a919500 0x55916a8f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 -- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 msgr2=0x55916a9ac7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 0x55916a9ac7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 -- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a919500 msgr2=0x55916a8f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a919500 0x55916a8f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 -- 172.21.15.67:0/1899217084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55916a9abdc0 con 0x55916a98e040 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76cc00640 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a919500 0x55916a8f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76d600640 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a98e040 0x55916a9acce0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe76800a510 tx=0x7fe76800a9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d583a269142c18da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe766a00640 1 -- 172.21.15.67:0/1899217084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768015070 con 0x55916a98e040 2024-04-17T10:29:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe766a00640 1 -- 172.21.15.67:0/1899217084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76800e730 con 0x55916a98e040 2024-04-17T10:29:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe766a00640 1 -- 172.21.15.67:0/1899217084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76800eb20 con 0x55916a98e040 2024-04-17T10:29:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 -- 172.21.15.67:0/1899217084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a98e040 msgr2=0x55916a9acce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a98e040 0x55916a9acce0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe76800a510 tx=0x7fe76800a9d0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 -- 172.21.15.67:0/1899217084 shutdown_connections 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a98e040 0x55916a9acce0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 0x55916a9ac7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 --2- 172.21.15.67:0/1899217084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a919500 0x55916a8f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.493+0000 7fe76fa65080 1 -- 172.21.15.67:0/1899217084 >> 172.21.15.67:0/1899217084 conn(0x55916a8fc0b0 msgr2=0x55916a918160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- 172.21.15.67:0/1899217084 shutdown_connections 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- 172.21.15.67:0/1899217084 wait complete. 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 Processor -- start 2024-04-17T10:29:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- start start 2024-04-17T10:29:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a919500 0x55916a98cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 0x55916a912e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a98e040 0x55916a9133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916a916c10 con 0x55916a919500 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55916a9ac060 con 0x55916a98e040 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55916a995bb0 con 0x55916a9198e0 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a919500 0x55916a98cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a919500 0x55916a98cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48738/0 (socket says 172.21.15.67:48738) 2024-04-17T10:29:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 -- 172.21.15.67:0/2877739015 learned_addr learned my addr 172.21.15.67:0/2877739015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76d600640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a98e040 0x55916a9133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe767e00640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 0x55916a912e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 -- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 msgr2=0x55916a912e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55916a9198e0 0x55916a912e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 -- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a98e040 msgr2=0x55916a9133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55916a98e040 0x55916a9133c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 -- 172.21.15.67:0/2877739015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55916a995e50 con 0x55916a919500 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76cc00640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a919500 0x55916a98cef0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe76000a590 tx=0x7fe7600180d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe765600640 1 -- 172.21.15.67:0/2877739015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7600af040 con 0x55916a919500 2024-04-17T10:29:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe765600640 1 -- 172.21.15.67:0/2877739015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7600b5420 con 0x55916a919500 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe765600640 1 -- 172.21.15.67:0/2877739015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7600b57d0 con 0x55916a919500 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55916a9960f0 con 0x55916a919500 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.497+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55916a996630 con 0x55916a919500 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe765600640 1 -- 172.21.15.67:0/2877739015 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe7600bd020 con 0x55916a919500 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe765600640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe74c065970 0x7fe74c067e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe765600640 1 -- 172.21.15.67:0/2877739015 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe76011f0f0 con 0x55916a919500 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe76fa65080 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x55916a8f6c00 0x55916a8f6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 --> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] -- osd_op(unknown.0.0:1 9.10 9:0cf6ad0f:::10000000000.00000000:head [omap-rm-keys in=17b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e67) v9 -- 0x55916a8f7520 con 0x55916a8f6c00 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe767e00640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe74c065970 0x7fe74c067e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe76d600640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x55916a8f6c00 0x55916a8f6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe767e00640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe74c065970 0x7fe74c067e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe75800a570 tx=0x7fe7580058a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.501+0000 7fe76d600640 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x55916a8f6c00 0x55916a8f6fe0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76d600640 1 -- 172.21.15.67:0/2877739015 <== osd.4 v2:172.21.15.86:6800/710871084 1 ==== osd_op_reply(1 10000000000.00000000 [omap-rm-keys] v67'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fe76800e070 con 0x55916a8f6c00 2024-04-17T10:29:13.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x55916a8f6c00 msgr2=0x55916a8f6fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6800/710871084,v1:172.21.15.86:6801/710871084] conn(0x55916a8f6c00 0x55916a8f6fe0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe74c065970 msgr2=0x7fe74c067e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe74c065970 0x7fe74c067e30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe75800a570 tx=0x7fe7580058a0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a919500 msgr2=0x55916a98cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 --2- 172.21.15.67:0/2877739015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55916a919500 0x55916a98cef0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe76000a590 tx=0x7fe7600180d0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76d600640 1 -- 172.21.15.67:0/2877739015 reap_dead start 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 shutdown_connections 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 >> 172.21.15.67:0/2877739015 conn(0x55916a8fc0b0 msgr2=0x55916a9ac730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 shutdown_connections 2024-04-17T10:29:13.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.505+0000 7fe76fa65080 1 -- 172.21.15.67:0/2877739015 wait complete. 2024-04-17T10:29:13.505 ERROR:tasks.ceph.mds.a:tried to stop a non-running daemon 2024-04-17T10:29:13.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-04-17T10:29:13.505 ERROR:tasks.ceph.mds.c:tried to stop a non-running daemon 2024-04-17T10:29:13.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-04-17T10:29:13.506 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-04-17T10:29:13.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-04-17T10:29:13.506 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-04-17T10:29:13.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-04-17T10:29:13.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2babe00640 1 Processor -- start 2024-04-17T10:29:13.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2babe00640 1 -- start start 2024-04-17T10:29:13.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f7c04c00640 1 Processor -- start 2024-04-17T10:29:13.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f7c04c00640 1 -- start start 2024-04-17T10:29:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f929d000640 1 Processor -- start 2024-04-17T10:29:13.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f929d000640 1 -- start start 2024-04-17T10:29:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2babe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2ba415dd10 0x7f2ba40b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2babe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ba40b57b0 0x7f2ba4070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2babe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba40b0cc0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2babe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40b1280 con 0x7f2ba40b57b0 2024-04-17T10:29:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2babe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba40b0fa0 con 0x7f2ba415dd10 2024-04-17T10:29:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ba40b57b0 0x7f2ba4070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ba40b57b0 0x7f2ba4070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48744/0 (socket says 172.21.15.67:48744) 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba8a00640 1 -- 172.21.15.67:0/3465285084 learned_addr learned my addr 172.21.15.67:0/3465285084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba9400640 1 --2- 172.21.15.67:0/3465285084 >> v2:172.21.15.86:3300/0 conn(0x7f2ba415dd10 0x7f2ba40b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba8a00640 1 -- 172.21.15.67:0/3465285084 >> v2:172.21.15.86:3300/0 conn(0x7f2ba415dd10 msgr2=0x7f2ba40b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba8a00640 1 --2- 172.21.15.67:0/3465285084 >> v2:172.21.15.86:3300/0 conn(0x7f2ba415dd10 0x7f2ba40b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba8a00640 1 -- 172.21.15.67:0/3465285084 >> v1:172.21.15.67:6789/0 conn(0x7f2ba40b07b0 legacy=0x7f2ba41618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f2ba8a00640 1 -- 172.21.15.67:0/3465285084 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba406bed0 con 0x7f2ba40b57b0 2024-04-17T10:29:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f70ca000640 1 Processor -- start 2024-04-17T10:29:13.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.697+0000 7f7c04c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c0015db40 0x7f7c0006f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7c04c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c000723b0 0x7f7c0006af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7c04c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c0015df20 con 0x7f7c0006dbd0 2024-04-17T10:29:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7c04c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c0006fb10 con 0x7f7c0015db40 2024-04-17T10:29:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7c04c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c000ad990 con 0x7f7c000723b0 2024-04-17T10:29:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bf7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7c0006dbd0 0x7f7c00071ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40120/0 (socket says 172.21.15.67:40120) 2024-04-17T10:29:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bf7e00640 1 -- 172.21.15.67:0/2670876504 learned_addr learned my addr 172.21.15.67:0/2670876504 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:13.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70ca000640 1 -- start start 2024-04-17T10:29:13.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bfe800640 1 --2- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c000723b0 0x7f7c0006af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bfde00640 1 --2- 172.21.15.67:0/2670876504 >> v2:172.21.15.86:3300/0 conn(0x7f7c0015db40 0x7f7c0006f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f929d000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f92980d0290 0x7f9298168a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bfde00640 1 -- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c000723b0 msgr2=0x7f7c0006af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bfde00640 1 --2- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c000723b0 0x7f7c0006af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bfd400640 1 -- 172.21.15.67:0/2670876504 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3524640127 0 0) 0x7f7c0015df20 con 0x7f7c0006dbd0 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bfde00640 1 -- 172.21.15.67:0/2670876504 >> v1:172.21.15.67:6789/0 conn(0x7f7c0006dbd0 legacy=0x7f7c00071ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f7bfde00640 1 -- 172.21.15.67:0/2670876504 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c000adc70 con 0x7f7c0015db40 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f929d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929806ff10 con 0x7f929806e5f0 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f929d000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92981625e0 con 0x7f92980cf7f0 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f929d000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92980cfbd0 con 0x7f92980d0290 2024-04-17T10:29:13.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9295e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f92980cf7f0 0x7f9298161ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:40918/0 (socket says 172.21.15.67:40918) 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9295e00640 1 -- 172.21.15.67:0/696469013 learned_addr learned my addr 172.21.15.67:0/696469013 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9295400640 1 --1- 172.21.15.67:0/696469013 >> v1:172.21.15.67:6789/0 conn(0x7f929806e5f0 0x7f92980c55b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40132/0 (socket says 172.21.15.67:40132) 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9296800640 1 --2- 172.21.15.67:0/696469013 >> v2:172.21.15.86:3300/0 conn(0x7f92980d0290 0x7f9298168a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70ca000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70c4161ff0 0x7f70c406fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70ca000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70c40cc870 0x7f70c4070060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70ca000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c406e680 0x7f70c406eb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70ca000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c40ccdc0 con 0x7f70c40cc870 2024-04-17T10:29:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70ca000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c41623d0 con 0x7f70c4161ff0 2024-04-17T10:29:13.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70ca000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c406ae30 con 0x7f70c406e680 2024-04-17T10:29:13.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9294a00640 1 -- 172.21.15.67:0/696469013 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988962721 0 0) 0x7f92981625e0 con 0x7f92980cf7f0 2024-04-17T10:29:13.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9294a00640 1 -- 172.21.15.67:0/696469013 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f927c003530 con 0x7f92980cf7f0 2024-04-17T10:29:13.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70c4161ff0 0x7f70c406fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9294a00640 1 -- 172.21.15.67:0/696469013 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9208455 0 0) 0x7f929806ff10 con 0x7f929806e5f0 2024-04-17T10:29:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9296800640 1 -- 172.21.15.67:0/696469013 >> v1:172.21.15.86:6789/0 conn(0x7f92980cf7f0 legacy=0x7f9298161ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9296800640 1 -- 172.21.15.67:0/696469013 >> v1:172.21.15.67:6789/0 conn(0x7f929806e5f0 legacy=0x7f92980c55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f9296800640 1 -- 172.21.15.67:0/696469013 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92980701b0 con 0x7f92980d0290 2024-04-17T10:29:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c406e680 0x7f70c406eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70c4161ff0 0x7f70c406fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:53872/0 (socket says 172.21.15.67:53872) 2024-04-17T10:29:13.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3400640 1 -- 172.21.15.67:0/2483141471 learned_addr learned my addr 172.21.15.67:0/2483141471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70baa00640 1 --2- 172.21.15.67:0/2483141471 >> v2:172.21.15.67:3300/0 conn(0x7f70c40cc870 0x7f70c4070060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3e00640 1 -- 172.21.15.67:0/2483141471 >> v2:172.21.15.86:3300/0 conn(0x7f70c4161ff0 msgr2=0x7f70c406fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3e00640 1 --2- 172.21.15.67:0/2483141471 >> v2:172.21.15.86:3300/0 conn(0x7f70c4161ff0 0x7f70c406fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3e00640 1 -- 172.21.15.67:0/2483141471 >> v2:172.21.15.67:3300/0 conn(0x7f70c40cc870 msgr2=0x7f70c4070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3e00640 1 --2- 172.21.15.67:0/2483141471 >> v2:172.21.15.67:3300/0 conn(0x7f70c40cc870 0x7f70c4070060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.701+0000 7f70c3e00640 1 -- 172.21.15.67:0/2483141471 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c406b110 con 0x7f70c406e680 2024-04-17T10:29:13.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2ba8a00640 1 --2- 172.21.15.67:0/3465285084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b57b0 0x7f2ba4070dc0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40d2900 tx=0x7f2b9001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ed6101b74202253 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7bfde00640 1 --2- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0015db40 0x7f7c0006f4c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00097c0 tx=0x7f7bf001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10984b393a843c26 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2ba3e00640 1 -- 172.21.15.67:0/3465285084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900ab640 con 0x7f2ba40b57b0 2024-04-17T10:29:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2ba3e00640 1 -- 172.21.15.67:0/3465285084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b900b0070 con 0x7f2ba40b57b0 2024-04-17T10:29:13.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2ba3e00640 1 -- 172.21.15.67:0/3465285084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b900aba00 con 0x7f2ba40b57b0 2024-04-17T10:29:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7bfd400640 1 -- 172.21.15.67:0/2670876504 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00a7070 con 0x7f7c0015db40 2024-04-17T10:29:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7bfd400640 1 -- 172.21.15.67:0/2670876504 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf00b0070 con 0x7f7c0015db40 2024-04-17T10:29:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7bfd400640 1 -- 172.21.15.67:0/2670876504 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00ab890 con 0x7f7c0015db40 2024-04-17T10:29:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 -- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0015db40 msgr2=0x7f7c0006f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 --2- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0015db40 0x7f7c0006f4c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00097c0 tx=0x7f7bf001c470 comp rx=0 tx=0).stop 2024-04-17T10:29:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 -- 172.21.15.67:0/3465285084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b57b0 msgr2=0x7f2ba4070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 --2- 172.21.15.67:0/3465285084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b57b0 0x7f2ba4070dc0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40d2900 tx=0x7f2b9001c460 comp rx=0 tx=0).stop 2024-04-17T10:29:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 -- 172.21.15.67:0/2670876504 shutdown_connections 2024-04-17T10:29:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 --2- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c000723b0 0x7f7c0006af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 --2- 172.21.15.67:0/2670876504 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c0015db40 0x7f7c0006f4c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 -- 172.21.15.67:0/2670876504 >> 172.21.15.67:0/2670876504 conn(0x7f7c000b1c00 msgr2=0x7f7c000b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 -- 172.21.15.67:0/3465285084 shutdown_connections 2024-04-17T10:29:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 --2- 172.21.15.67:0/3465285084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b57b0 0x7f2ba4070dc0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 --2- 172.21.15.67:0/3465285084 >> v2:172.21.15.86:3300/0 conn(0x7f2ba415dd10 0x7f2ba40b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 -- 172.21.15.67:0/3465285084 >> 172.21.15.67:0/3465285084 conn(0x7f2ba40b34d0 msgr2=0x7f2ba40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:13.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 -- 172.21.15.67:0/2670876504 shutdown_connections 2024-04-17T10:29:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70c3e00640 1 --2- 172.21.15.67:0/2483141471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c406e680 0x7f70c406eb10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f70c4071c00 tx=0x7f70bc01c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1ded186efa369aa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 -- 172.21.15.67:0/3465285084 shutdown_connections 2024-04-17T10:29:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f2babe00640 1 -- 172.21.15.67:0/3465285084 wait complete. 2024-04-17T10:29:13.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f7c04c00640 1 -- 172.21.15.67:0/2670876504 wait complete. 2024-04-17T10:29:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70c2a00640 1 -- 172.21.15.67:0/2483141471 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0b2070 con 0x7f70c406e680 2024-04-17T10:29:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70c2a00640 1 -- 172.21.15.67:0/2483141471 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc0a5720 con 0x7f70c406e680 2024-04-17T10:29:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70c2a00640 1 -- 172.21.15.67:0/2483141471 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc0a5af0 con 0x7f70c406e680 2024-04-17T10:29:13.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f9296800640 1 --2- 172.21.15.67:0/696469013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92980d0290 0x7f9298168a90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9290009d90 tx=0x7f929001f120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66fb12e13f3e49ea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f9294a00640 1 -- 172.21.15.67:0/696469013 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92900b4070 con 0x7f92980d0290 2024-04-17T10:29:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f9294a00640 1 -- 172.21.15.67:0/696469013 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92900af600 con 0x7f92980d0290 2024-04-17T10:29:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f9294a00640 1 -- 172.21.15.67:0/696469013 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92900af9d0 con 0x7f92980d0290 2024-04-17T10:29:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 -- 172.21.15.67:0/2483141471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c406e680 msgr2=0x7f70c406eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 --2- 172.21.15.67:0/2483141471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c406e680 0x7f70c406eb10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f70c4071c00 tx=0x7f70bc01c8a0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 -- 172.21.15.67:0/2483141471 shutdown_connections 2024-04-17T10:29:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 --2- 172.21.15.67:0/2483141471 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c406e680 0x7f70c406eb10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 --2- 172.21.15.67:0/2483141471 >> v2:172.21.15.67:3300/0 conn(0x7f70c40cc870 0x7f70c4070060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 --2- 172.21.15.67:0/2483141471 >> v2:172.21.15.86:3300/0 conn(0x7f70c4161ff0 0x7f70c406fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 -- 172.21.15.67:0/2483141471 >> 172.21.15.67:0/2483141471 conn(0x7f70c40b23b0 msgr2=0x7f70c40b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 -- 172.21.15.67:0/2483141471 shutdown_connections 2024-04-17T10:29:13.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.705+0000 7f70ca000640 1 -- 172.21.15.67:0/2483141471 wait complete. 2024-04-17T10:29:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 -- 172.21.15.67:0/696469013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92980d0290 msgr2=0x7f9298168a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 --2- 172.21.15.67:0/696469013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92980d0290 0x7f9298168a90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9290009d90 tx=0x7f929001f120 comp rx=0 tx=0).stop 2024-04-17T10:29:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 Processor -- start 2024-04-17T10:29:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 -- 172.21.15.67:0/696469013 shutdown_connections 2024-04-17T10:29:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 --2- 172.21.15.67:0/696469013 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92980d0290 0x7f9298168a90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 -- 172.21.15.67:0/696469013 >> 172.21.15.67:0/696469013 conn(0x7f92980b2340 msgr2=0x7f92980b2750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 Processor -- start 2024-04-17T10:29:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 -- start start 2024-04-17T10:29:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 -- 172.21.15.67:0/696469013 shutdown_connections 2024-04-17T10:29:13.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 -- 172.21.15.67:0/696469013 wait complete. 2024-04-17T10:29:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 -- start start 2024-04-17T10:29:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b07b0 0x7f2ba41fb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ba415dd10 0x7f2ba41516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ba415e930 0x7f2ba40cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba416ae70 con 0x7f2ba40b07b0 2024-04-17T10:29:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba4071710 con 0x7f2ba415dd10 2024-04-17T10:29:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba416aff0 con 0x7f2ba415e930 2024-04-17T10:29:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 Processor -- start 2024-04-17T10:29:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b07b0 0x7f2ba41fb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b07b0 0x7f2ba41fb360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48772/0 (socket says 172.21.15.67:48772) 2024-04-17T10:29:13.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 -- 172.21.15.67:0/3945806989 learned_addr learned my addr 172.21.15.67:0/3945806989 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 -- start start 2024-04-17T10:29:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0006dbd0 0x7f7c0014b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c000723b0 0x7f7c001384d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0015db40 0x7f7c000b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba8a00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ba415dd10 0x7f2ba41516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9e00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ba415e930 0x7f2ba40cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0006be40 con 0x7f7c0015db40 2024-04-17T10:29:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0006bcc0 con 0x7f7c000723b0 2024-04-17T10:29:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7c04c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0006bfc0 con 0x7f7c0006dbd0 2024-04-17T10:29:13.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 -- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ba415e930 msgr2=0x7f2ba40cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ba415e930 0x7f2ba40cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 -- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ba415dd10 msgr2=0x7f2ba41516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ba415dd10 0x7f2ba41516b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 -- 172.21.15.67:0/3945806989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba415edc0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba8a00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ba415dd10 0x7f2ba41516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0006dbd0 0x7f7c0014b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0006dbd0 0x7f7c0014b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:46272/0 (socket says 172.21.15.67:46272) 2024-04-17T10:29:13.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 -- 172.21.15.67:0/30030668 learned_addr learned my addr 172.21.15.67:0/30030668 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfe800640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0015db40 0x7f7c000b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba9400640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b07b0 0x7f2ba41fb360 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800d5f0 tx=0x7f2b9800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bf7e00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c000723b0 0x7f7c001384d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba1600640 1 -- 172.21.15.67:0/3945806989 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980125f0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba1600640 1 -- 172.21.15.67:0/3945806989 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b98019070 con 0x7f2ba40b07b0 2024-04-17T10:29:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2ba1600640 1 -- 172.21.15.67:0/3945806989 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980129b0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba415f0a0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 Processor -- start 2024-04-17T10:29:13.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba4156790 con 0x7f2ba40b07b0 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 -- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c000723b0 msgr2=0x7f7c001384d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c000723b0 0x7f7c001384d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 -- 172.21.15.67:0/30030668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0015db40 msgr2=0x7f7c000b0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0015db40 0x7f7c000b0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f7bfde00640 1 -- 172.21.15.67:0/30030668 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00136480 con 0x7f7c0006dbd0 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c406e680 0x7f70c41467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40cc870 0x7f70c4158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4161ff0 0x7f70c415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40aff90 con 0x7f70c4161ff0 2024-04-17T10:29:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40afe10 con 0x7f70c406e680 2024-04-17T10:29:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70ca000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c416db20 con 0x7f70c40cc870 2024-04-17T10:29:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f929d000640 1 -- start start 2024-04-17T10:29:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.709+0000 7f70c3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c406e680 0x7f70c41467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c406e680 0x7f70c41467c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:53882/0 (socket says 172.21.15.67:53882) 2024-04-17T10:29:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 -- 172.21.15.67:0/1588264124 learned_addr learned my addr 172.21.15.67:0/1588264124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70baa00640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40cc870 0x7f70c4158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3e00640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4161ff0 0x7f70c415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f7bfde00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0006dbd0 0x7f7c0014b340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00ad040 tx=0x7f7bf001acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 -- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40cc870 msgr2=0x7f70c4158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40cc870 0x7f70c4158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 -- 172.21.15.67:0/1588264124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4161ff0 msgr2=0x7f70c415a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4161ff0 0x7f70c415a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 -- 172.21.15.67:0/1588264124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4159610 con 0x7f70c406e680 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70baa00640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40cc870 0x7f70c4158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3e00640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4161ff0 0x7f70c415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f7bf6000640 1 -- 172.21.15.67:0/30030668 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00bf610 con 0x7f7c0006dbd0 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f7bf6000640 1 -- 172.21.15.67:0/30030668 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf00bf7f0 con 0x7f7c0006dbd0 2024-04-17T10:29:13.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f7bf6000640 1 -- 172.21.15.67:0/30030668 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf00bfbe0 con 0x7f7c0006dbd0 2024-04-17T10:29:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929806e5f0 0x7f92980ad620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba4070bb0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92980cf7f0 0x7f929809ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70c3400640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c406e680 0x7f70c41467c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e7f0 tx=0x7f70b400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92980d0290 0x7f92980ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298161a40 con 0x7f92980cf7f0 2024-04-17T10:29:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298161760 con 0x7f929806e5f0 2024-04-17T10:29:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92980c5cc0 con 0x7f92980d0290 2024-04-17T10:29:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929806e5f0 0x7f92980ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929806e5f0 0x7f92980ad620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:53896/0 (socket says 172.21.15.67:53896) 2024-04-17T10:29:13.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 -- 172.21.15.67:0/308921112 learned_addr learned my addr 172.21.15.67:0/308921112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295400640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92980cf7f0 0x7f929809ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9296800640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92980d0290 0x7f92980ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70bbe00640 1 -- 172.21.15.67:0/1588264124 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b4017070 con 0x7f70c406e680 2024-04-17T10:29:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70bbe00640 1 -- 172.21.15.67:0/1588264124 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b400c940 con 0x7f70c406e680 2024-04-17T10:29:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70bbe00640 1 -- 172.21.15.67:0/1588264124 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b400cd30 con 0x7f70c406e680 2024-04-17T10:29:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c00138d60 con 0x7f7c0006dbd0 2024-04-17T10:29:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c00139320 con 0x7f7c0006dbd0 2024-04-17T10:29:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c41598f0 con 0x7f70c406e680 2024-04-17T10:29:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c4159eb0 con 0x7f70c406e680 2024-04-17T10:29:13.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0009e260 con 0x7f7c0006dbd0 2024-04-17T10:29:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 -- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92980d0290 msgr2=0x7f92980ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92980d0290 0x7f92980ae150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 -- 172.21.15.67:0/308921112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92980cf7f0 msgr2=0x7f929809ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92980cf7f0 0x7f929809ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 -- 172.21.15.67:0/308921112 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92980addc0 con 0x7f929806e5f0 2024-04-17T10:29:13.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f2ba1600640 1 -- 172.21.15.67:0/3945806989 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2b98012b50 con 0x7f2ba40b07b0 2024-04-17T10:29:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f9295e00640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929806e5f0 0x7f92980ad620 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f928800c550 tx=0x7f928800ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70c409ea10 con 0x7f70c406e680 2024-04-17T10:29:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f928e000640 1 -- 172.21.15.67:0/308921112 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288018070 con 0x7f929806e5f0 2024-04-17T10:29:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f928e000640 1 -- 172.21.15.67:0/308921112 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92880074e0 con 0x7f929806e5f0 2024-04-17T10:29:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f928e000640 1 -- 172.21.15.67:0/308921112 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288004a30 con 0x7f929806e5f0 2024-04-17T10:29:13.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929809b1e0 con 0x7f929806e5f0 2024-04-17T10:29:13.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929809b7a0 con 0x7f929806e5f0 2024-04-17T10:29:13.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.713+0000 7f2ba1600640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2b84066fb0 0x7f2b84069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f2ba1600640 1 -- 172.21.15.67:0/3945806989 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2b980855c0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f2ba8a00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2b84066fb0 0x7f2b84069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929809e9a0 con 0x7f929806e5f0 2024-04-17T10:29:13.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f70bbe00640 1 -- 172.21.15.67:0/1588264124 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f70b401a070 con 0x7f70c406e680 2024-04-17T10:29:13.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f7bf6000640 1 -- 172.21.15.67:0/30030668 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7bf00bd020 con 0x7f7c0006dbd0 2024-04-17T10:29:13.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f2ba8a00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2b84066fb0 0x7f2b84069470 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2b900af000 tx=0x7f2b900a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f7bf6000640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7bdc067030 0x7f7bdc0694f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f7bf6000640 1 -- 172.21.15.67:0/30030668 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7bf01200c0 con 0x7f7c0006dbd0 2024-04-17T10:29:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f70bbe00640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70a0066fe0 0x7f70a00694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f70bbe00640 1 -- 172.21.15.67:0/1588264124 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f70b4085770 con 0x7f70c406e680 2024-04-17T10:29:13.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f7bf7e00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7bdc067030 0x7f7bdc0694f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f70baa00640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70a0066fe0 0x7f70a00694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f7bf7e00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7bdc067030 0x7f7bdc0694f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c000cc820 tx=0x7f7be8007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.717+0000 7f70baa00640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70a0066fe0 0x7f70a00694a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f70c41fe2f0 tx=0x7f70ac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.721+0000 7f7bf6000640 1 -- 172.21.15.67:0/30030668 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7bf00c4060 con 0x7f7c0006dbd0 2024-04-17T10:29:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.721+0000 7f2ba1600640 1 -- 172.21.15.67:0/3945806989 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2b98011340 con 0x7f2ba40b07b0 2024-04-17T10:29:13.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.721+0000 7f70bbe00640 1 -- 172.21.15.67:0/1588264124 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f70b4056010 con 0x7f70c406e680 2024-04-17T10:29:13.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.721+0000 7f928e000640 1 -- 172.21.15.67:0/308921112 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f928801b070 con 0x7f929806e5f0 2024-04-17T10:29:13.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.725+0000 7f928e000640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066fb0 0x7f9278069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:13.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.725+0000 7f928e000640 1 -- 172.21.15.67:0/308921112 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f92880849c0 con 0x7f929806e5f0 2024-04-17T10:29:13.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.725+0000 7f9295400640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066fb0 0x7f9278069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:13.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.725+0000 7f9295400640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066fb0 0x7f9278069470 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f92981578b0 tx=0x7f9284007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:13.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.725+0000 7f928e000640 1 -- 172.21.15.67:0/308921112 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9288022020 con 0x7f929806e5f0 2024-04-17T10:29:13.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.957+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f70c4074e30 con 0x7f70c406e680 2024-04-17T10:29:13.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.961+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f9298074e30 con 0x7f929806e5f0 2024-04-17T10:29:13.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.985+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f2ba40b41a0 con 0x7f2ba40b07b0 2024-04-17T10:29:13.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:13.985+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f7c00071fb0 con 0x7f7c0006dbd0 2024-04-17T10:29:14.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.277+0000 7f70bbe00640 1 -- 172.21.15.67:0/1588264124 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 5900 v41) v1 ==== 95+0+0 (secure 0 0 0) 0x7f70b405a0b0 con 0x7f70c406e680 2024-04-17T10:29:14.272 INFO:teuthology.orchestra.run.smithi067.stderr:failed mds gid 5900 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70a0066fe0 msgr2=0x7f70a00694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70a0066fe0 0x7f70a00694a0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f70c41fe2f0 tx=0x7f70ac008040 comp rx=0 tx=0).stop 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c406e680 msgr2=0x7f70c41467c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c406e680 0x7f70c41467c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f70b400e7f0 tx=0x7f70b400ecb0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 shutdown_connections 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f70a0066fe0 0x7f70a00694a0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70c4161ff0 0x7f70c415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70c40cc870 0x7f70c4158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 --2- 172.21.15.67:0/1588264124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70c406e680 0x7f70c41467c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 >> 172.21.15.67:0/1588264124 conn(0x7f70c40b23b0 msgr2=0x7f70c40b1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:14.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 shutdown_connections 2024-04-17T10:29:14.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.281+0000 7f70ca000640 1 -- 172.21.15.67:0/1588264124 wait complete. 2024-04-17T10:29:14.283 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:29:14.289+0000 7ff63c000640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:29:14.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.293+0000 7f2ba1600640 1 -- 172.21.15.67:0/3945806989 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6648 v42) v1 ==== 95+0+0 (secure 0 0 0) 0x7f2b98056420 con 0x7f2ba40b07b0 2024-04-17T10:29:14.288 INFO:teuthology.orchestra.run.smithi067.stderr:failed mds gid 6648 2024-04-17T10:29:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f928e000640 1 -- 172.21.15.67:0/308921112 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 5897 v42) v1 ==== 95+0+0 (secure 0 0 0) 0x7f9288055260 con 0x7f929806e5f0 2024-04-17T10:29:14.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2b84066fb0 msgr2=0x7f2b84069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2b84066fb0 0x7f2b84069470 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2b900af000 tx=0x7f2b900a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:14.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b07b0 msgr2=0x7f2ba41fb360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b07b0 0x7f2ba41fb360 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800d5f0 tx=0x7f2b9800dab0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f7bf6000640 1 -- 172.21.15.67:0/30030668 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6651 v42) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7bf0016090 con 0x7f7c0006dbd0 2024-04-17T10:29:14.292 INFO:teuthology.orchestra.run.smithi067.stderr:failed mds gid 6651 2024-04-17T10:29:14.293 INFO:teuthology.orchestra.run.smithi067.stderr:failed mds gid 5897 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 shutdown_connections 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2b84066fb0 0x7f2b84069470 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ba415e930 0x7f2ba40cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ba415dd10 0x7f2ba41516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 --2- 172.21.15.67:0/3945806989 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ba40b07b0 0x7f2ba41fb360 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 >> 172.21.15.67:0/3945806989 conn(0x7f2ba40b34d0 msgr2=0x7f2ba41583b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 shutdown_connections 2024-04-17T10:29:14.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.297+0000 7f2babe00640 1 -- 172.21.15.67:0/3945806989 wait complete. 2024-04-17T10:29:14.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7bdc067030 msgr2=0x7f7bdc0694f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7bdc067030 0x7f7bdc0694f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c000cc820 tx=0x7f7be8007d60 comp rx=0 tx=0).stop 2024-04-17T10:29:14.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0006dbd0 msgr2=0x7f7c0014b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0006dbd0 0x7f7c0014b340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7bf00ad040 tx=0x7f7bf001acb0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066fb0 msgr2=0x7f9278069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066fb0 0x7f9278069470 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f92981578b0 tx=0x7f9284007d60 comp rx=0 tx=0).stop 2024-04-17T10:29:14.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 shutdown_connections 2024-04-17T10:29:14.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7bdc067030 0x7f7bdc0694f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c0015db40 0x7f7c000b0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c000723b0 0x7f7c001384d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 --2- 172.21.15.67:0/30030668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c0006dbd0 0x7f7c0014b340 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 >> 172.21.15.67:0/30030668 conn(0x7f7c000b1c00 msgr2=0x7f7c000b2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929806e5f0 msgr2=0x7f92980ad620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929806e5f0 0x7f92980ad620 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f928800c550 tx=0x7f928800ca10 comp rx=0 tx=0).stop 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 shutdown_connections 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9278066fb0 0x7f9278069470 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92980d0290 0x7f92980ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92980cf7f0 0x7f929809ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 --2- 172.21.15.67:0/308921112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f929806e5f0 0x7f92980ad620 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 >> 172.21.15.67:0/308921112 conn(0x7f92980b2340 msgr2=0x7f92980ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 shutdown_connections 2024-04-17T10:29:14.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f929d000640 1 -- 172.21.15.67:0/308921112 wait complete. 2024-04-17T10:29:14.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 shutdown_connections 2024-04-17T10:29:14.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.301+0000 7f7c04c00640 1 -- 172.21.15.67:0/30030668 wait complete. 2024-04-17T10:29:14.302 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:29:14.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:29:14.305 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:29:14.306 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:14.321 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:29:14.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:29:14.323 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:29:14.323 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:29:14.324 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:29:14.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:29:14.327 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:29:14.327 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:14.372 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:29:14.372 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:14.373 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:29:14.374 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:14.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:14.394 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:29:14.417 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:29:14.459 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:29:14.460 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:29:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.525+0000 7f8355800640 1 Processor -- start 2024-04-17T10:29:14.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f8355800640 1 -- start start 2024-04-17T10:29:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f8355800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8350071840 0x7f8350166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:14.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f8355800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83500b6640 con 0x7f83501446c0 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f8355800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83501456d0 con 0x7f8350144aa0 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f8355800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83500b51a0 con 0x7f8350071840 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f834f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8350071840 0x7f8350166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f834ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f83501446c0 0x7f8350144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40146/0 (socket says 172.21.15.67:40146) 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f834f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8350071840 0x7f8350166a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48822/0 (socket says 172.21.15.67:48822) 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f834ea00640 1 -- 172.21.15.67:0/1763752020 learned_addr learned my addr 172.21.15.67:0/1763752020 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f834f400640 1 -- 172.21.15.67:0/1763752020 >> v1:172.21.15.86:6789/0 conn(0x7f8350144aa0 legacy=0x7f83500d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f834f400640 1 -- 172.21.15.67:0/1763752020 >> v1:172.21.15.67:6789/0 conn(0x7f83501446c0 legacy=0x7f8350144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.529+0000 7f834f400640 1 -- 172.21.15.67:0/1763752020 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83500b4870 con 0x7f8350071840 2024-04-17T10:29:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f834f400640 1 --2- 172.21.15.67:0/1763752020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8350071840 0x7f8350166a30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8348009d90 tx=0x7f834801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67659dc8e3476d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f834e000640 1 -- 172.21.15.67:0/1763752020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83480b2070 con 0x7f8350071840 2024-04-17T10:29:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f834e000640 1 -- 172.21.15.67:0/1763752020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83480a5910 con 0x7f8350071840 2024-04-17T10:29:14.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f834e000640 1 -- 172.21.15.67:0/1763752020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83480a5ce0 con 0x7f8350071840 2024-04-17T10:29:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 -- 172.21.15.67:0/1763752020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8350071840 msgr2=0x7f8350166a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 --2- 172.21.15.67:0/1763752020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8350071840 0x7f8350166a30 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8348009d90 tx=0x7f834801c940 comp rx=0 tx=0).stop 2024-04-17T10:29:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 -- 172.21.15.67:0/1763752020 shutdown_connections 2024-04-17T10:29:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 --2- 172.21.15.67:0/1763752020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8350071840 0x7f8350166a30 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 -- 172.21.15.67:0/1763752020 >> 172.21.15.67:0/1763752020 conn(0x7f83500b34d0 msgr2=0x7f83500b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:14.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 -- 172.21.15.67:0/1763752020 shutdown_connections 2024-04-17T10:29:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 -- 172.21.15.67:0/1763752020 wait complete. 2024-04-17T10:29:14.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 Processor -- start 2024-04-17T10:29:14.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.533+0000 7f8355800640 1 -- start start 2024-04-17T10:29:14.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8350071840 0x7f835013f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:14.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83501446c0 0x7f835014fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8350144aa0 0x7f8350155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8350169910 con 0x7f83501446c0 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8350169790 con 0x7f8350144aa0 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8350169a90 con 0x7f8350071840 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83501446c0 0x7f835014fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83501446c0 0x7f835014fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48832/0 (socket says 172.21.15.67:48832) 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 -- 172.21.15.67:0/3791953993 learned_addr learned my addr 172.21.15.67:0/3791953993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f834ea00640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8350071840 0x7f835013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f834f400640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8350144aa0 0x7f8350155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 -- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8350071840 msgr2=0x7f835013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8350071840 0x7f835013f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 -- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8350144aa0 msgr2=0x7f8350155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8350144aa0 0x7f8350155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 -- 172.21.15.67:0/3791953993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8350140fd0 con 0x7f83501446c0 2024-04-17T10:29:14.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8347e00640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83501446c0 0x7f835014fe30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f833800c3f0 tx=0x7f833800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8346a00640 1 -- 172.21.15.67:0/3791953993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338017070 con 0x7f83501446c0 2024-04-17T10:29:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8346a00640 1 -- 172.21.15.67:0/3791953993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83380052d0 con 0x7f83501446c0 2024-04-17T10:29:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83501514d0 con 0x7f83501446c0 2024-04-17T10:29:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8346a00640 1 -- 172.21.15.67:0/3791953993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338003840 con 0x7f83501446c0 2024-04-17T10:29:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f835014f630 con 0x7f83501446c0 2024-04-17T10:29:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83501519f0 con 0x7f83501446c0 2024-04-17T10:29:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.537+0000 7f8346a00640 1 -- 172.21.15.67:0/3791953993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f833801a070 con 0x7f83501446c0 2024-04-17T10:29:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.541+0000 7f8346a00640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f832c066f60 0x7f832c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.541+0000 7f8346a00640 1 -- 172.21.15.67:0/3791953993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8338084510 con 0x7f83501446c0 2024-04-17T10:29:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.541+0000 7f834ea00640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f832c066f60 0x7f832c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.541+0000 7f834ea00640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f832c066f60 0x7f832c069420 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8340000f70 tx=0x7f8340007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:14.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.545+0000 7f8346a00640 1 -- 172.21.15.67:0/3791953993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f83501519f0 con 0x7f83501446c0 2024-04-17T10:29:14.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.741+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8350070bb0 con 0x7f83501446c0 2024-04-17T10:29:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.741+0000 7f8346a00640 1 -- 172.21.15.67:0/3791953993 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f8338005480 con 0x7f83501446c0 2024-04-17T10:29:14.737 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:14.737 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":42,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:14.294062+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:14.737 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 42 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f832c066f60 msgr2=0x7f832c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f832c066f60 0x7f832c069420 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8340000f70 tx=0x7f8340007e90 comp rx=0 tx=0).stop 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83501446c0 msgr2=0x7f835014fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83501446c0 0x7f835014fe30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f833800c3f0 tx=0x7f833800c8b0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 shutdown_connections 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f832c066f60 0x7f832c069420 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8350144aa0 0x7f8350155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f83501446c0 0x7f835014fe30 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 --2- 172.21.15.67:0/3791953993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8350071840 0x7f835013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 >> 172.21.15.67:0/3791953993 conn(0x7f83500b34d0 msgr2=0x7f8350154450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 shutdown_connections 2024-04-17T10:29:14.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:14.745+0000 7f8355800640 1 -- 172.21.15.67:0/3791953993 wait complete. 2024-04-17T10:29:15.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:28:58.444091+0000', 'modified': '2024-04-17T10:29:14.294062+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:29:15.161 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:29:16.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:16.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9d600640 1 Processor -- start 2024-04-17T10:29:16.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9d600640 1 -- start start 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d9806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9d600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1d9806c3b0 0x7f1d980b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9d600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d980700a0 con 0x7f1d98066c20 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9d600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d980b3390 con 0x7f1d9806c3b0 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9d600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d98070380 con 0x7f1d9815b5d0 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d97400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1d98066c20 0x7f1d9806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40152/0 (socket says 172.21.15.67:40152) 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d97400640 1 -- 172.21.15.67:0/3967698872 learned_addr learned my addr 172.21.15.67:0/3967698872 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d97e00640 1 --2- 172.21.15.67:0/3967698872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d9806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:16.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d9cc00640 1 --2- 172.21.15.67:0/3967698872 >> v2:172.21.15.86:3300/0 conn(0x7f1d9806c3b0 0x7f1d980b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d96a00640 1 -- 172.21.15.67:0/3967698872 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715775143 0 0) 0x7f1d980700a0 con 0x7f1d98066c20 2024-04-17T10:29:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d96a00640 1 -- 172.21.15.67:0/3967698872 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d7c003530 con 0x7f1d98066c20 2024-04-17T10:29:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d97e00640 1 -- 172.21.15.67:0/3967698872 >> v2:172.21.15.86:3300/0 conn(0x7f1d9806c3b0 msgr2=0x7f1d980b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d97e00640 1 --2- 172.21.15.67:0/3967698872 >> v2:172.21.15.86:3300/0 conn(0x7f1d9806c3b0 0x7f1d980b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d97e00640 1 -- 172.21.15.67:0/3967698872 >> v1:172.21.15.67:6789/0 conn(0x7f1d98066c20 legacy=0x7f1d9806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.325+0000 7f1d97e00640 1 -- 172.21.15.67:0/3967698872 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d980b2d40 con 0x7f1d9815b5d0 2024-04-17T10:29:16.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d97e00640 1 --2- 172.21.15.67:0/3967698872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d9806e470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d880097c0 tx=0x7f1d8801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dc7cb2d18bacceb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:16.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d96a00640 1 -- 172.21.15.67:0/3967698872 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880b2070 con 0x7f1d9815b5d0 2024-04-17T10:29:16.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d96a00640 1 -- 172.21.15.67:0/3967698872 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d880ad680 con 0x7f1d9815b5d0 2024-04-17T10:29:16.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d96a00640 1 -- 172.21.15.67:0/3967698872 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880ada30 con 0x7f1d9815b5d0 2024-04-17T10:29:16.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 -- 172.21.15.67:0/3967698872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 msgr2=0x7f1d9806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3967698872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d9806e470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1d880097c0 tx=0x7f1d8801c470 comp rx=0 tx=0).stop 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 -- 172.21.15.67:0/3967698872 shutdown_connections 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3967698872 >> v2:172.21.15.86:3300/0 conn(0x7f1d9806c3b0 0x7f1d980b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3967698872 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d9806e470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 -- 172.21.15.67:0/3967698872 >> 172.21.15.67:0/3967698872 conn(0x7f1d98071ad0 msgr2=0x7f1d98071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 -- 172.21.15.67:0/3967698872 shutdown_connections 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.329+0000 7f1d9d600640 1 -- 172.21.15.67:0/3967698872 wait complete. 2024-04-17T10:29:16.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 Processor -- start 2024-04-17T10:29:16.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 -- start start 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d98066c20 0x7f1d98146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d9806c3b0 0x7f1d980ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d980ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980b46f0 con 0x7f1d98066c20 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980b4570 con 0x7f1d9806c3b0 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980b4870 con 0x7f1d9815b5d0 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d98066c20 0x7f1d98146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d98066c20 0x7f1d98146460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48842/0 (socket says 172.21.15.67:48842) 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 -- 172.21.15.67:0/3185074817 learned_addr learned my addr 172.21.15.67:0/3185074817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9cc00640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d980ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97400640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d9806c3b0 0x7f1d980ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:16.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 -- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 msgr2=0x7f1d980ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d980ade80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 -- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d9806c3b0 msgr2=0x7f1d980ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d9806c3b0 0x7f1d980ad940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 -- 172.21.15.67:0/3185074817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9814b6e0 con 0x7f1d98066c20 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d97e00640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d98066c20 0x7f1d98146460 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1d88000c10 tx=0x7f1d88019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d8fe00640 1 -- 172.21.15.67:0/3185074817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880bb070 con 0x7f1d98066c20 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d8fe00640 1 -- 172.21.15.67:0/3185074817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88004300 con 0x7f1d98066c20 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9814b9c0 con 0x7f1d98066c20 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d8fe00640 1 -- 172.21.15.67:0/3185074817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880b2040 con 0x7f1d98066c20 2024-04-17T10:29:16.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9814bf80 con 0x7f1d98066c20 2024-04-17T10:29:16.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.333+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d98074dc0 con 0x7f1d98066c20 2024-04-17T10:29:16.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.337+0000 7f1d8fe00640 1 -- 172.21.15.67:0/3185074817 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1d880be070 con 0x7f1d98066c20 2024-04-17T10:29:16.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.337+0000 7f1d8fe00640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d78066f60 0x7f1d78069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:16.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.337+0000 7f1d8fe00640 1 -- 172.21.15.67:0/3185074817 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1d8811dfe0 con 0x7f1d98066c20 2024-04-17T10:29:16.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.337+0000 7f1d97400640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d78066f60 0x7f1d78069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:16.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.337+0000 7f1d97400640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d78066f60 0x7f1d78069420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1d981664f0 tx=0x7f1d84009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:16.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.341+0000 7f1d8fe00640 1 -- 172.21.15.67:0/3185074817 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1d88016090 con 0x7f1d98066c20 2024-04-17T10:29:16.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.537+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d981469a0 con 0x7f1d98066c20 2024-04-17T10:29:16.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d8fe00640 1 -- 172.21.15.67:0/3185074817 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4940 (secure 0 0 0) 0x7f1d880044b0 con 0x7f1d98066c20 2024-04-17T10:29:16.533 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:16.534 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":45,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3163523544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3163523544},{"type":"v1","addr":"172.21.15.86:6835","nonce":3163523544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3682484259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3682484259},{"type":"v1","addr":"172.21.15.86:6837","nonce":3682484259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6945,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/252088612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":252088612},{"type":"v1","addr":"172.21.15.67:6834","nonce":252088612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:15.316955+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"c","rank":0,"incarnation":44,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.67:6835/2976334498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2976334498},{"type":"v1","addr":"172.21.15.67:6835","nonce":2976334498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:16.534 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 45 2024-04-17T10:29:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d78066f60 msgr2=0x7f1d78069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:16.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d78066f60 0x7f1d78069420 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f1d981664f0 tx=0x7f1d84009040 comp rx=0 tx=0).stop 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d98066c20 msgr2=0x7f1d98146460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d98066c20 0x7f1d98146460 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1d88000c10 tx=0x7f1d88019210 comp rx=0 tx=0).stop 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 shutdown_connections 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d78066f60 0x7f1d78069420 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d9815b5d0 0x7f1d980ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d9806c3b0 0x7f1d980ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 --2- 172.21.15.67:0/3185074817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d98066c20 0x7f1d98146460 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 >> 172.21.15.67:0/3185074817 conn(0x7f1d98071ad0 msgr2=0x7f1d9806b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 shutdown_connections 2024-04-17T10:29:16.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:16.541+0000 7f1d9d600640 1 -- 172.21.15.67:0/3185074817 wait complete. 2024-04-17T10:29:16.958 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:28:58.444091+0000', 'modified': '2024-04-17T10:29:15.316955+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6948}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6948': {'gid': 6948, 'name': 'c', 'rank': 0, 'incarnation': 44, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.67:6835/2976334498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 2976334498}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 2976334498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:29:16.958 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6948:up:reconnect 2024-04-17T10:29:17.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:18.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.117+0000 7f3b01e00640 1 Processor -- start 2024-04-17T10:29:18.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3b01e00640 1 -- start start 2024-04-17T10:29:18.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3b01e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3afc15dbc0 0x7f3afc06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3b01e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3afc072370 0x7f3afc1676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3b01e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3afc0ad8f0 con 0x7f3afc06db90 2024-04-17T10:29:18.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3b01e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3afc15dfa0 con 0x7f3afc072370 2024-04-17T10:29:18.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3b01e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3afc06faa0 con 0x7f3afc15dbc0 2024-04-17T10:29:18.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3afb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3afc072370 0x7f3afc1676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3af2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3afc06db90 0x7f3afc071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40162/0 (socket says 172.21.15.67:40162) 2024-04-17T10:29:18.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3af2000640 1 -- 172.21.15.67:0/1637713796 learned_addr learned my addr 172.21.15.67:0/1637713796 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:18.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3afaa00640 1 --2- 172.21.15.67:0/1637713796 >> v2:172.21.15.86:3300/0 conn(0x7f3afc15dbc0 0x7f3afc06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3afb400640 1 -- 172.21.15.67:0/1637713796 >> v2:172.21.15.86:3300/0 conn(0x7f3afc15dbc0 msgr2=0x7f3afc06f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3afb400640 1 --2- 172.21.15.67:0/1637713796 >> v2:172.21.15.86:3300/0 conn(0x7f3afc15dbc0 0x7f3afc06f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3afb400640 1 -- 172.21.15.67:0/1637713796 >> v1:172.21.15.67:6789/0 conn(0x7f3afc06db90 legacy=0x7f3afc071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.121+0000 7f3afb400640 1 -- 172.21.15.67:0/1637713796 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0adbd0 con 0x7f3afc072370 2024-04-17T10:29:18.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3afb400640 1 --2- 172.21.15.67:0/1637713796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc1676d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009d90 tx=0x7f3af401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db03f99657fa7a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3afa000640 1 -- 172.21.15.67:0/1637713796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af40b2070 con 0x7f3afc072370 2024-04-17T10:29:18.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3afa000640 1 -- 172.21.15.67:0/1637713796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af40a5760 con 0x7f3afc072370 2024-04-17T10:29:18.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3afa000640 1 -- 172.21.15.67:0/1637713796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af40a5b30 con 0x7f3afc072370 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 -- 172.21.15.67:0/1637713796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 msgr2=0x7f3afc1676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1637713796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc1676d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3af4009d90 tx=0x7f3af401c8a0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 -- 172.21.15.67:0/1637713796 shutdown_connections 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1637713796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc1676d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1637713796 >> v2:172.21.15.86:3300/0 conn(0x7f3afc15dbc0 0x7f3afc06f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 -- 172.21.15.67:0/1637713796 >> 172.21.15.67:0/1637713796 conn(0x7f3afc0af670 msgr2=0x7f3afc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 -- 172.21.15.67:0/1637713796 shutdown_connections 2024-04-17T10:29:18.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 -- 172.21.15.67:0/1637713796 wait complete. 2024-04-17T10:29:18.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 Processor -- start 2024-04-17T10:29:18.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.125+0000 7f3b01e00640 1 -- start start 2024-04-17T10:29:18.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc06db90 0x7f3afc1f9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc15f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc15dbc0 0x7f3afc156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc16a7c0 con 0x7f3afc072370 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc16a640 con 0x7f3afc15dbc0 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc16a940 con 0x7f3afc06db90 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc15f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc15f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48856/0 (socket says 172.21.15.67:48856) 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 -- 172.21.15.67:0/1946215600 learned_addr learned my addr 172.21.15.67:0/1946215600 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3afb400640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc15dbc0 0x7f3afc156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3afaa00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc06db90 0x7f3afc1f9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 -- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc06db90 msgr2=0x7f3afc1f9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc06db90 0x7f3afc1f9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 -- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc15dbc0 msgr2=0x7f3afc156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc15dbc0 0x7f3afc156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 -- 172.21.15.67:0/1946215600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0cc0d0 con 0x7f3afc072370 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3afb400640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc15dbc0 0x7f3afc156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3afaa00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc06db90 0x7f3afc1f9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:18.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af2000640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc15f490 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c450 tx=0x7f3ae400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af3e00640 1 -- 172.21.15.67:0/1946215600 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4017070 con 0x7f3afc072370 2024-04-17T10:29:18.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af3e00640 1 -- 172.21.15.67:0/1946215600 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4005cf0 con 0x7f3afc072370 2024-04-17T10:29:18.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af3e00640 1 -- 172.21.15.67:0/1946215600 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4007290 con 0x7f3afc072370 2024-04-17T10:29:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc15fe00 con 0x7f3afc072370 2024-04-17T10:29:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc1606a0 con 0x7f3afc072370 2024-04-17T10:29:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3afc09e260 con 0x7f3afc072370 2024-04-17T10:29:18.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.129+0000 7f3af3e00640 1 -- 172.21.15.67:0/1946215600 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3ae401a070 con 0x7f3afc072370 2024-04-17T10:29:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.133+0000 7f3af3e00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ad8066f60 0x7f3ad8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.133+0000 7f3af3e00640 1 -- 172.21.15.67:0/1946215600 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3ae40845a0 con 0x7f3afc072370 2024-04-17T10:29:18.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.133+0000 7f3afaa00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ad8066f60 0x7f3ad8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.133+0000 7f3afaa00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ad8066f60 0x7f3ad8069420 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00b9f0 tx=0x7f3aec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.137+0000 7f3af3e00640 1 -- 172.21.15.67:0/1946215600 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3ae4055330 con 0x7f3afc072370 2024-04-17T10:29:18.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.333+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3afc074e30 con 0x7f3afc072370 2024-04-17T10:29:18.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3af3e00640 1 -- 172.21.15.67:0/1946215600 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f3ae4054c30 con 0x7f3afc072370 2024-04-17T10:29:18.330 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:18.330 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3163523544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3163523544},{"type":"v1","addr":"172.21.15.86:6835","nonce":3163523544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3682484259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3682484259},{"type":"v1","addr":"172.21.15.86:6837","nonce":3682484259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6945,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/252088612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":252088612},{"type":"v1","addr":"172.21.15.67:6834","nonce":252088612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:17.323326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/2976334498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2976334498},{"type":"v1","addr":"172.21.15.67:6835","nonce":2976334498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:18.330 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 47 2024-04-17T10:29:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ad8066f60 msgr2=0x7f3ad8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ad8066f60 0x7f3ad8069420 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00b9f0 tx=0x7f3aec008040 comp rx=0 tx=0).stop 2024-04-17T10:29:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 msgr2=0x7f3afc15f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc15f490 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c450 tx=0x7f3ae400c910 comp rx=0 tx=0).stop 2024-04-17T10:29:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 shutdown_connections 2024-04-17T10:29:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ad8066f60 0x7f3ad8069420 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3afc15dbc0 0x7f3afc156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3afc072370 0x7f3afc15f490 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 --2- 172.21.15.67:0/1946215600 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3afc06db90 0x7f3afc1f9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 >> 172.21.15.67:0/1946215600 conn(0x7f3afc0af670 msgr2=0x7f3afc153450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 shutdown_connections 2024-04-17T10:29:18.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.337+0000 7f3b01e00640 1 -- 172.21.15.67:0/1946215600 wait complete. 2024-04-17T10:29:18.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:28:58.444091+0000', 'modified': '2024-04-17T10:29:17.323326+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 71, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6948}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6948': {'gid': 6948, 'name': 'c', 'rank': 0, 'incarnation': 44, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.67:6835/2976334498', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 2976334498}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 2976334498}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:29:18.755 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:29:18.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T10:29:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.905+0000 7f943d200640 1 Processor -- start 2024-04-17T10:29:18.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.905+0000 7f943d200640 1 -- start start 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.905+0000 7f943d200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f943806e6e0 0x7f94380ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.905+0000 7f943d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f943806eac0 0x7f94380ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.905+0000 7f943d200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943815ae40 con 0x7f9438143ae0 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.905+0000 7f943d200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94380cf430 con 0x7f943806eac0 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.905+0000 7f943d200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9438071440 con 0x7f943806e6e0 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f943806eac0 0x7f94380ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f943806eac0 0x7f94380ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48872/0 (socket says 172.21.15.67:48872) 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 -- 172.21.15.67:0/1537071725 learned_addr learned my addr 172.21.15.67:0/1537071725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:18.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f9435e00640 1 --2- 172.21.15.67:0/1537071725 >> v2:172.21.15.86:3300/0 conn(0x7f943806e6e0 0x7f94380ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 -- 172.21.15.67:0/1537071725 >> v2:172.21.15.86:3300/0 conn(0x7f943806e6e0 msgr2=0x7f94380ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 --2- 172.21.15.67:0/1537071725 >> v2:172.21.15.86:3300/0 conn(0x7f943806e6e0 0x7f94380ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 -- 172.21.15.67:0/1537071725 >> v1:172.21.15.86:6789/0 conn(0x7f9438143ae0 legacy=0x7f9438143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 -- 172.21.15.67:0/1537071725 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9438071720 con 0x7f943806eac0 2024-04-17T10:29:18.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f942d400640 1 --2- 172.21.15.67:0/1537071725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943806eac0 0x7f94380ceef0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f94380cf620 tx=0x7f942001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826b6b5b2afeeaa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f9435400640 1 -- 172.21.15.67:0/1537071725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94200ab660 con 0x7f943806eac0 2024-04-17T10:29:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f9435400640 1 -- 172.21.15.67:0/1537071725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94200b0070 con 0x7f943806eac0 2024-04-17T10:29:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.909+0000 7f9435400640 1 -- 172.21.15.67:0/1537071725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94200aba20 con 0x7f943806eac0 2024-04-17T10:29:18.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- 172.21.15.67:0/1537071725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943806eac0 msgr2=0x7f94380ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 --2- 172.21.15.67:0/1537071725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943806eac0 0x7f94380ceef0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f94380cf620 tx=0x7f942001c460 comp rx=0 tx=0).stop 2024-04-17T10:29:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- 172.21.15.67:0/1537071725 shutdown_connections 2024-04-17T10:29:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 --2- 172.21.15.67:0/1537071725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f943806eac0 0x7f94380ceef0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 --2- 172.21.15.67:0/1537071725 >> v2:172.21.15.86:3300/0 conn(0x7f943806e6e0 0x7f94380ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- 172.21.15.67:0/1537071725 >> 172.21.15.67:0/1537071725 conn(0x7f943806ac90 msgr2=0x7f943806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- 172.21.15.67:0/1537071725 shutdown_connections 2024-04-17T10:29:18.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- 172.21.15.67:0/1537071725 wait complete. 2024-04-17T10:29:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 Processor -- start 2024-04-17T10:29:18.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- start start 2024-04-17T10:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943806e6e0 0x7f94381370b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f943806eac0 0x7f9438159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9438143ae0 0x7f9438138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94380b3a40 con 0x7f9438143ae0 2024-04-17T10:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94380b38c0 con 0x7f943806e6e0 2024-04-17T10:29:18.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f943d200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94380b3bc0 con 0x7f943806eac0 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9438143ae0 0x7f9438138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9438143ae0 0x7f9438138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48886/0 (socket says 172.21.15.67:48886) 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 -- 172.21.15.67:0/3365263659 learned_addr learned my addr 172.21.15.67:0/3365263659 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9435e00640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943806e6e0 0x7f94381370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f942d400640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f943806eac0 0x7f9438159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 -- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f943806eac0 msgr2=0x7f9438159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f943806eac0 0x7f9438159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 -- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943806e6e0 msgr2=0x7f94381370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943806e6e0 0x7f94381370b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.913+0000 7f9436800640 1 -- 172.21.15.67:0/3365263659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94380b1050 con 0x7f9438143ae0 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f9436800640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9438143ae0 0x7f9438138ac0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f943000c730 tx=0x7f943000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f942ea00640 1 -- 172.21.15.67:0/3365263659 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430016020 con 0x7f9438143ae0 2024-04-17T10:29:18.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f942ea00640 1 -- 172.21.15.67:0/3365263659 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9430006d60 con 0x7f9438143ae0 2024-04-17T10:29:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f942ea00640 1 -- 172.21.15.67:0/3365263659 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430003bc0 con 0x7f9438143ae0 2024-04-17T10:29:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94380b0d70 con 0x7f9438143ae0 2024-04-17T10:29:18.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9438139a70 con 0x7f9438143ae0 2024-04-17T10:29:18.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f942ea00640 1 -- 172.21.15.67:0/3365263659 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f94300050d0 con 0x7f9438143ae0 2024-04-17T10:29:18.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f942ea00640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9414066f60 0x7f9414069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f942ea00640 1 -- 172.21.15.67:0/3365263659 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9430084ce0 con 0x7f9438143ae0 2024-04-17T10:29:18.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f9435e00640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9414066f60 0x7f9414069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f94406e2000 1 Processor -- start 2024-04-17T10:29:18.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.917+0000 7f94406e2000 1 -- start start 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9435e00640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9414066f60 0x7f9414069420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f943815aac0 tx=0x7f9428009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c8945f0 con 0x55ba1c9acf30 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c7fbda0 con 0x55ba1c9f6920 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c784150 con 0x55ba1c9f7240 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48890/0 (socket says 172.21.15.67:48890) 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 -- 172.21.15.67:0/1031397361 learned_addr learned my addr 172.21.15.67:0/1031397361 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9435e00640 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f942d400640 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 -- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 msgr2=0x55ba1c9ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 -- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 msgr2=0x55ba1c9f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9f6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 -- 172.21.15.67:0/1031397361 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba1c79dca0 con 0x55ba1c9acf30 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9436800640 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9ad310 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f94300553f0 tx=0x7f9430000f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79a47868536507e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9426a00640 1 -- 172.21.15.67:0/1031397361 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430051b20 con 0x55ba1c9acf30 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9426a00640 1 -- 172.21.15.67:0/1031397361 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9430059150 con 0x55ba1c9acf30 2024-04-17T10:29:18.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f9426a00640 1 -- 172.21.15.67:0/1031397361 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430051cc0 con 0x55ba1c9acf30 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- 172.21.15.67:0/1031397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 msgr2=0x55ba1c9ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9ad310 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f94300553f0 tx=0x7f9430000f50 comp rx=0 tx=0).stop 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- 172.21.15.67:0/1031397361 shutdown_connections 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9ad310 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 --2- 172.21.15.67:0/1031397361 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- 172.21.15.67:0/1031397361 >> 172.21.15.67:0/1031397361 conn(0x55ba1c9b32a0 msgr2=0x55ba1c9ab2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- 172.21.15.67:0/1031397361 shutdown_connections 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- 172.21.15.67:0/1031397361 wait complete. 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 -- start start 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.921+0000 7f94406e2000 1 Processor -- start 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9acf30 0x55ba1c9b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f7240 0x55ba1c9b1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c7b6990 con 0x55ba1c9f6920 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c7f7e40 con 0x55ba1c9f7240 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c787900 con 0x55ba1c9acf30 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 10 client.0 _ll_register_callbacks cb 0x55ba1c8af1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48894/0 (socket says 172.21.15.67:48894) 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 -- 172.21.15.67:0/3727774064 learned_addr learned my addr 172.21.15.67:0/3727774064 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f9435e00640 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9acf30 0x55ba1c9b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f9436800640 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f7240 0x55ba1c9b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 -- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9acf30 msgr2=0x55ba1c9b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9acf30 0x55ba1c9b0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 -- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f7240 msgr2=0x55ba1c9b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f7240 0x55ba1c9b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 -- 172.21.15.67:0/3727774064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba1c7b4630 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f942d400640 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b13d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9420019070 tx=0x7f94200046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20ee6edd87df2c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f941b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f941b400640 1 -- 172.21.15.67:0/3727774064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94200abc00 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f941b400640 1 -- 172.21.15.67:0/3727774064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f94200abda0 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9430059150 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f941b400640 1 -- 172.21.15.67:0/3727774064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94200be400 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 10 client.6981 fetch_fsmap learned FSMap version 47 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55ba1c7843a0 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f941b400640 1 -- 172.21.15.67:0/3727774064 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9430059150 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f941b400640 1 -- 172.21.15.67:0/3727774064 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f94200be5a0 con 0x55ba1c9f6920 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 10 client.6981 fetch_fsmap finished waiting for FSMap version 47 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 10 client.6981 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:29:18.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 20 client.6981 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:18.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2320 0x55ba1ca0c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 4 client.6981 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:29:18.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 --> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba1c797a50 con 0x55ba1c9b2320 2024-04-17T10:29:18.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f9435e00640 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2320 0x55ba1ca0c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f9435e00640 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2320 0x55ba1ca0c2c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d31e083264e39f24 server_cookie=f337f4a5706133f5 in_seq=0 out_seq=0 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.925+0000 7f941b400640 10 client.6981 ms_handle_connect on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f941b400640 1 -- 172.21.15.67:0/3727774064 <== mds.0 v2:172.21.15.67:6833/2976334498 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ba1c797a50 con 0x55ba1c9b2320 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f941b400640 10 client.6981 handle_command_reply: tid=0 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 client.6981 shutdown 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9f6920 msgr2=0x55ba1c9b13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b13d0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f9420019070 tx=0x7f94200046d0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 shutdown_connections 2024-04-17T10:29:18.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2320 0x55ba1ca0c2c0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f7240 0x55ba1c9b1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b13d0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f941b400640 0 client.6981 ms_handle_reset on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 --2- 172.21.15.67:0/3727774064 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9acf30 0x55ba1c9b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 >> 172.21.15.67:0/3727774064 conn(0x55ba1c9b32a0 msgr2=0x55ba1c9ab9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 shutdown_connections 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 1 -- 172.21.15.67:0/3727774064 wait complete. 2024-04-17T10:29:18.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.929+0000 7f94406e2000 20 client.6981 trim_cache size 0 max 16384 2024-04-17T10:29:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 Processor -- start 2024-04-17T10:29:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 -- start start 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ad310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943813f090 con 0x55ba1c9acf30 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94380b0500 con 0x55ba1c9f6920 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94380b0b00 con 0x55ba1c9f7240 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f9435e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f9435e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48900/0 (socket says 172.21.15.67:48900) 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.945+0000 7f9435e00640 1 -- 172.21.15.67:0/817779060 learned_addr learned my addr 172.21.15.67:0/817779060 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f942d400640 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9436800640 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 -- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 msgr2=0x55ba1c9ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 -- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 msgr2=0x55ba1c9b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 -- 172.21.15.67:0/817779060 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba1c9f7070 con 0x55ba1c9acf30 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9436800640 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f942d400640 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9b1290 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55ba1c7bb880 tx=0x7f9428018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140d8c4c3ab609d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9426000640 1 -- 172.21.15.67:0/817779060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942802ad70 con 0x55ba1c9acf30 2024-04-17T10:29:18.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9426000640 1 -- 172.21.15.67:0/817779060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9428030070 con 0x55ba1c9acf30 2024-04-17T10:29:18.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9426000640 1 -- 172.21.15.67:0/817779060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942802a550 con 0x55ba1c9acf30 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- 172.21.15.67:0/817779060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 msgr2=0x55ba1c9b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9b1290 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55ba1c7bb880 tx=0x7f9428018000 comp rx=0 tx=0).stop 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- 172.21.15.67:0/817779060 shutdown_connections 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1c9ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1c9b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 --2- 172.21.15.67:0/817779060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1c9b1290 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- 172.21.15.67:0/817779060 >> 172.21.15.67:0/817779060 conn(0x55ba1c9b32a0 msgr2=0x55ba1c9f4bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- 172.21.15.67:0/817779060 shutdown_connections 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- 172.21.15.67:0/817779060 wait complete. 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- start start 2024-04-17T10:29:18.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 Processor -- start 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1ca3f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1ca59bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1ca5a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c8003d0 con 0x55ba1c9acf30 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c8b7d50 con 0x55ba1c9f6920 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba1c796980 con 0x55ba1c9f7240 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f94406e2000 10 client.0 _ll_register_callbacks cb 0x7f9430055280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1ca3f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1ca3f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48902/0 (socket says 172.21.15.67:48902) 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9435e00640 1 -- 172.21.15.67:0/2920676737 learned_addr learned my addr 172.21.15.67:0/2920676737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f942d400640 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1ca59bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.949+0000 7f9436800640 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1ca5a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f9435e00640 1 -- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 msgr2=0x55ba1ca5a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f9435e00640 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1ca5a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f9435e00640 1 -- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 msgr2=0x55ba1ca59bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f9435e00640 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1ca59bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f9435e00640 1 -- 172.21.15.67:0/2920676737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba1c87d600 con 0x55ba1c9acf30 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f9435e00640 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1ca3f250 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9428011690 tx=0x7f9428004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d56b19d17f4b8d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 1 -- 172.21.15.67:0/2920676737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428018040 con 0x55ba1c9acf30 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 1 -- 172.21.15.67:0/2920676737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9428003830 con 0x55ba1c9acf30 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9428030070 con 0x55ba1c9acf30 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 1 -- 172.21.15.67:0/2920676737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9428003a30 con 0x55ba1c9acf30 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 10 client.6987 fetch_fsmap learned FSMap version 47 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x55ba1c7909b0 con 0x55ba1c9acf30 2024-04-17T10:29:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 1 -- 172.21.15.67:0/2920676737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9428030070 con 0x55ba1c9acf30 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 1 -- 172.21.15.67:0/2920676737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9428020040 con 0x55ba1c9acf30 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 10 client.6987 fetch_fsmap finished waiting for FSMap version 47 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 10 client.6987 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 20 client.6987 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2db0 0x55ba1ca5a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 4 client.6987 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 --> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ba1c8092e0 con 0x55ba1c9b2db0 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f942d400640 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2db0 0x55ba1ca5a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f942d400640 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2db0 0x55ba1ca5a630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83157deee3a9c471 server_cookie=c49fade80c29573d in_seq=0 out_seq=0 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 10 client.6987 ms_handle_connect on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 1 -- 172.21.15.67:0/2920676737 <== mds.0 v2:172.21.15.67:6833/2976334498 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ba1c8092e0 con 0x55ba1c9b2db0 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 10 client.6987 handle_command_reply: tid=0 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 client.6987 shutdown 2024-04-17T10:29:18.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 msgr2=0x55ba1ca3f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1ca3f250 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9428011690 tx=0x7f9428004740 comp rx=0 tx=0).stop 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 shutdown_connections 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x55ba1c9b2db0 0x55ba1ca5a630 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba1c9f7240 0x55ba1ca5a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba1c9f6920 0x55ba1ca59bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 --2- 172.21.15.67:0/2920676737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba1c9acf30 0x55ba1ca3f250 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f941b400640 0 client.6987 ms_handle_reset on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 >> 172.21.15.67:0/2920676737 conn(0x55ba1c9b32a0 msgr2=0x55ba1c9af320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 shutdown_connections 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 1 -- 172.21.15.67:0/2920676737 wait complete. 2024-04-17T10:29:18.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f94406e2000 20 client.6987 trim_cache size 0 max 16384 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 6948, 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 3.6487257529999999, 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 47, 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 71, 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 71, 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 4.4840805079999999 2024-04-17T10:29:18.953 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:29:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9414066f60 msgr2=0x7f9414069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9414066f60 0x7f9414069420 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f943815aac0 tx=0x7f9428009190 comp rx=0 tx=0).stop 2024-04-17T10:29:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9438143ae0 msgr2=0x7f9438138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9438143ae0 0x7f9438138ac0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f943000c730 tx=0x7f943000cb30 comp rx=0 tx=0).stop 2024-04-17T10:29:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 shutdown_connections 2024-04-17T10:29:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9414066f60 0x7f9414069420 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9438143ae0 0x7f9438138ac0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f943806eac0 0x7f9438159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 --2- 172.21.15.67:0/3365263659 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f943806e6e0 0x7f94381370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 >> 172.21.15.67:0/3365263659 conn(0x7f943806ac90 msgr2=0x7f943813c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 shutdown_connections 2024-04-17T10:29:18.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:18.953+0000 7f943d200640 1 -- 172.21.15.67:0/3365263659 wait complete. 2024-04-17T10:29:18.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:29:19.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0c6c00640 1 Processor -- start 2024-04-17T10:29:19.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0c6c00640 1 -- start start 2024-04-17T10:29:19.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0c6c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:19.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0c6c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0c006ef10 0x7fa0c0071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:19.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0c6c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c006e2f0 con 0x7fa0c0071850 2024-04-17T10:29:19.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0c6c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c00a86c0 con 0x7fa0c006ef10 2024-04-17T10:29:19.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0c6c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c015bf10 con 0x7fa0c0145040 2024-04-17T10:29:19.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.113+0000 7fa0b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0c006ef10 0x7fa0c0071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:19.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0c006ef10 0x7fa0c0071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48916/0 (socket says 172.21.15.67:48916) 2024-04-17T10:29:19.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0b7e00640 1 -- 172.21.15.67:0/2847293501 learned_addr learned my addr 172.21.15.67:0/2847293501 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:19.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0bfe00640 1 --2- 172.21.15.67:0/2847293501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:19.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0b7e00640 1 -- 172.21.15.67:0/2847293501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 msgr2=0x7fa0c006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:19.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0b7e00640 1 --2- 172.21.15.67:0/2847293501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0b7e00640 1 -- 172.21.15.67:0/2847293501 >> v1:172.21.15.86:6789/0 conn(0x7fa0c0071850 legacy=0x7fa0c015f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:19.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0b7e00640 1 -- 172.21.15.67:0/2847293501 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c006d6e0 con 0x7fa0c006ef10 2024-04-17T10:29:19.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0b7e00640 1 --2- 172.21.15.67:0/2847293501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ef10 0x7fa0c0071310 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009870 tx=0x7fa0a801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3563d6c105c190da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:19.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0bf400640 1 -- 172.21.15.67:0/2847293501 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a80b5070 con 0x7fa0c006ef10 2024-04-17T10:29:19.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0bf400640 1 -- 172.21.15.67:0/2847293501 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a80a8a50 con 0x7fa0c006ef10 2024-04-17T10:29:19.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.117+0000 7fa0bf400640 1 -- 172.21.15.67:0/2847293501 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a80a8e20 con 0x7fa0c006ef10 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- 172.21.15.67:0/2847293501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ef10 msgr2=0x7fa0c0071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/2847293501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ef10 0x7fa0c0071310 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009870 tx=0x7fa0a801ed50 comp rx=0 tx=0).stop 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- 172.21.15.67:0/2847293501 shutdown_connections 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/2847293501 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c006ef10 0x7fa0c0071310 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/2847293501 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- 172.21.15.67:0/2847293501 >> 172.21.15.67:0/2847293501 conn(0x7fa0c006ac90 msgr2=0x7fa0c006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- 172.21.15.67:0/2847293501 shutdown_connections 2024-04-17T10:29:19.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- 172.21.15.67:0/2847293501 wait complete. 2024-04-17T10:29:19.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 Processor -- start 2024-04-17T10:29:19.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- start start 2024-04-17T10:29:19.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c006ef10 0x7fa0c0143fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:19.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0071850 0x7fa0c014ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:19.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c014f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:19.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c006dca0 con 0x7fa0c0071850 2024-04-17T10:29:19.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0071ca0 con 0x7fa0c006ef10 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c6c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c006df80 con 0x7fa0c0145040 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0071850 0x7fa0c014ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0071850 0x7fa0c014ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48930/0 (socket says 172.21.15.67:48930) 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 -- 172.21.15.67:0/1874429667 learned_addr learned my addr 172.21.15.67:0/1874429667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0bfe00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c006ef10 0x7fa0c0143fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c4c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c014f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 -- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 msgr2=0x7fa0c014f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c014f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 -- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c006ef10 msgr2=0x7fa0c0143fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c006ef10 0x7fa0c0143fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0b7e00640 1 -- 172.21.15.67:0/1874429667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c015d650 con 0x7fa0c0071850 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0bfe00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c006ef10 0x7fa0c0143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:19.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.121+0000 7fa0c4c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c014f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0b7e00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0071850 0x7fa0c014ecd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa0a801d400 tx=0x7fa0a801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0bcc00640 1 -- 172.21.15.67:0/1874429667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a80b5070 con 0x7fa0c0071850 2024-04-17T10:29:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0bcc00640 1 -- 172.21.15.67:0/1874429667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a80b0de0 con 0x7fa0c0071850 2024-04-17T10:29:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0bcc00640 1 -- 172.21.15.67:0/1874429667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a80bb670 con 0x7fa0c0071850 2024-04-17T10:29:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c015d930 con 0x7fa0c0071850 2024-04-17T10:29:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c015def0 con 0x7fa0c0071850 2024-04-17T10:29:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c0072d50 con 0x7fa0c0071850 2024-04-17T10:29:19.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0bcc00640 1 -- 172.21.15.67:0/1874429667 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa0a80c2060 con 0x7fa0c0071850 2024-04-17T10:29:19.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0bcc00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa09c066f60 0x7fa09c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:19.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.125+0000 7fa0bcc00640 1 -- 172.21.15.67:0/1874429667 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa0a8122850 con 0x7fa0c0071850 2024-04-17T10:29:19.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.129+0000 7fa0bfe00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa09c066f60 0x7fa09c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:19.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.129+0000 7fa0bfe00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa09c066f60 0x7fa09c069420 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009a40 tx=0x7fa0b0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:19.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.133+0000 7fa0bcc00640 1 -- 172.21.15.67:0/1874429667 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa0a80188b0 con 0x7fa0c0071850 2024-04-17T10:29:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.329+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa0c00cce80 con 0x7fa0c0071850 2024-04-17T10:29:19.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.329+0000 7fa0bcc00640 1 -- 172.21.15.67:0/1874429667 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa0a80f3550 con 0x7fa0c0071850 2024-04-17T10:29:19.325 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa09c066f60 msgr2=0x7fa09c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa09c066f60 0x7fa09c069420 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009a40 tx=0x7fa0b0008040 comp rx=0 tx=0).stop 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0071850 msgr2=0x7fa0c014ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0071850 0x7fa0c014ecd0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa0a801d400 tx=0x7fa0a801baf0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 shutdown_connections 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa09c066f60 0x7fa09c069420 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0c0145040 0x7fa0c014f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0c0071850 0x7fa0c014ecd0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 --2- 172.21.15.67:0/1874429667 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0c006ef10 0x7fa0c0143fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 >> 172.21.15.67:0/1874429667 conn(0x7fa0c006ac90 msgr2=0x7fa0c00cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 shutdown_connections 2024-04-17T10:29:19.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:19.333+0000 7fa0c6c00640 1 -- 172.21.15.67:0/1874429667 wait complete. 2024-04-17T10:29:19.345 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:29:19.345 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:29:19.346 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:29:19.346 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:29:19.346 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:19.346 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:29:19.346 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:29:19.346 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:29:19.346 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:19.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.346 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:29:19.355 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:19.356 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42187sec preferred_lft 42187sec 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:19.357 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:29:19.358 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:19.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.359 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:29:19.419 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:29:19.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.421 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:29:19.477 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:29:19.477 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:29:19.477 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:29:19.477 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:29:19.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.478 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:29:19.479 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:29:19.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:29:19.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:29:19.479 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:29:19.479 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:29:19.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.576 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:29:19.585 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:29:19.585 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:29:19.586 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:19.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:29:19.649 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:29:19.649 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:19.705 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:19.705 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:29:19.705 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:19.756 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:29:19.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:19.849 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:19.849 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:29:19.862 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:19.862 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:29:19.862 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:29:19.862 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:29:19.863 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:29:19.870 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:29:19.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:19.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.890 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:29:19.943 INFO:teuthology.orchestra.run.smithi067.stdout:client6996 2024-04-17T10:29:19.944 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:19.944 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:19.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:19.997 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir scrub_repair_path) 2024-04-17T10:29:20.050 INFO:teuthology.orchestra.run.smithi067.stderr:rmdir: failed to remove 'scrub_repair_path': Directory not empty 2024-04-17T10:29:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-17T10:29:20.051 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_repair_path repair' 2024-04-17T10:29:20.051 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:29:20.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start /scrub_repair_path repair 2024-04-17T10:29:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45c800640 1 Processor -- start 2024-04-17T10:29:20.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45c800640 1 -- start start 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161af0 0x7fb45415f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45c800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb45415fa30 0x7fb45415fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45c800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4540b53a0 con 0x7fb454161710 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45c800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4540b5990 con 0x7fb45415fa30 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4540b3920 con 0x7fb454161af0 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb459e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb454161710 0x7fb4540b1cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40178/0 (socket says 172.21.15.67:40178) 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb459e00640 1 -- 172.21.15.67:0/2988011572 learned_addr learned my addr 172.21.15.67:0/2988011572 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:20.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45a800640 1 --2- 172.21.15.67:0/2988011572 >> v2:172.21.15.67:3300/0 conn(0x7fb45415fa30 0x7fb45415fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb459400640 1 --2- 172.21.15.67:0/2988011572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161af0 0x7fb45415f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45a800640 1 -- 172.21.15.67:0/2988011572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161af0 msgr2=0x7fb45415f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45a800640 1 --2- 172.21.15.67:0/2988011572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161af0 0x7fb45415f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45a800640 1 -- 172.21.15.67:0/2988011572 >> v1:172.21.15.67:6789/0 conn(0x7fb454161710 legacy=0x7fb4540b1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.245+0000 7fb45a800640 1 -- 172.21.15.67:0/2988011572 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4540b3c00 con 0x7fb45415fa30 2024-04-17T10:29:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45a800640 1 --2- 172.21.15.67:0/2988011572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb45415fa30 0x7fb45415fec0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb450009d90 tx=0x7fb45001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcff7c129e82044 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb458a00640 1 -- 172.21.15.67:0/2988011572 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500b2070 con 0x7fb45415fa30 2024-04-17T10:29:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb458a00640 1 -- 172.21.15.67:0/2988011572 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4500a57a0 con 0x7fb45415fa30 2024-04-17T10:29:20.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb458a00640 1 -- 172.21.15.67:0/2988011572 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500a5b70 con 0x7fb45415fa30 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 -- 172.21.15.67:0/2988011572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb45415fa30 msgr2=0x7fb45415fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 --2- 172.21.15.67:0/2988011572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb45415fa30 0x7fb45415fec0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb450009d90 tx=0x7fb45001c8e0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 -- 172.21.15.67:0/2988011572 shutdown_connections 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 --2- 172.21.15.67:0/2988011572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb45415fa30 0x7fb45415fec0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 --2- 172.21.15.67:0/2988011572 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161af0 0x7fb45415f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 -- 172.21.15.67:0/2988011572 >> 172.21.15.67:0/2988011572 conn(0x7fb4540b5e50 msgr2=0x7fb454099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 -- 172.21.15.67:0/2988011572 shutdown_connections 2024-04-17T10:29:20.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.249+0000 7fb45c800640 1 -- 172.21.15.67:0/2988011572 wait complete. 2024-04-17T10:29:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 Processor -- start 2024-04-17T10:29:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 -- start start 2024-04-17T10:29:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb45415fa30 0x7fb45413f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161710 0x7fb454156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb454161af0 0x7fb454156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4540b13c0 con 0x7fb454161af0 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4540b1240 con 0x7fb45415fa30 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb454162390 con 0x7fb454161710 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb454161af0 0x7fb454156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb454161af0 0x7fb454156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48956/0 (socket says 172.21.15.67:48956) 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 -- 172.21.15.67:0/583877569 learned_addr learned my addr 172.21.15.67:0/583877569 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb459e00640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb45415fa30 0x7fb45413f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb459400640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161710 0x7fb454156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 -- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161710 msgr2=0x7fb454156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161710 0x7fb454156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 -- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb45415fa30 msgr2=0x7fb45413f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb45415fa30 0x7fb45413f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 -- 172.21.15.67:0/583877569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4540aeb40 con 0x7fb454161af0 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb459e00640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb45415fa30 0x7fb45413f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45a800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb454161af0 0x7fb454156fd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb4500b0040 tx=0x7fb450019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb44e000640 1 -- 172.21.15.67:0/583877569 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500b2050 con 0x7fb454161af0 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb44e000640 1 -- 172.21.15.67:0/583877569 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4500be070 con 0x7fb454161af0 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb44e000640 1 -- 172.21.15.67:0/583877569 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500ad870 con 0x7fb454161af0 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4540aee20 con 0x7fb454161af0 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.253+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4540af6c0 con 0x7fb454161af0 2024-04-17T10:29:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb44e000640 1 -- 172.21.15.67:0/583877569 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb4500ada10 con 0x7fb454161af0 2024-04-17T10:29:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb44e000640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb438066f90 0x7fb438069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb44e000640 1 -- 172.21.15.67:0/583877569 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb450120380 con 0x7fb454161af0 2024-04-17T10:29:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb459e00640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb438066f90 0x7fb438069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 Processor -- start 2024-04-17T10:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 -- start start 2024-04-17T10:29:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb459e00640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb438066f90 0x7fb438069450 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb448000f70 tx=0x7fb448009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf925240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf925780 0x556fcf8dd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf8de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fcf7c2c70 con 0x556fcf924e60 2024-04-17T10:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fcf729040 con 0x556fcf8ddd80 2024-04-17T10:29:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556fcf6b1040 con 0x556fcf925780 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb459e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf925240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb459e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf925240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48968/0 (socket says 172.21.15.67:48968) 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.257+0000 7fb459e00640 1 -- 172.21.15.67:0/2020736365 learned_addr learned my addr 172.21.15.67:0/2020736365 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459400640 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf925780 0x556fcf8dd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45a800640 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf8de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459e00640 1 -- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf925780 msgr2=0x556fcf8dd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459e00640 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf925780 0x556fcf8dd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459e00640 1 -- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 msgr2=0x556fcf8de160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459e00640 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf8de160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459e00640 1 -- 172.21.15.67:0/2020736365 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fcf6e0240 con 0x556fcf924e60 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45a800640 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf8de160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459e00640 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf925240 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb4480058f0 tx=0x7fb448011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6baf1cf7bc9dd1b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb442a00640 1 -- 172.21.15.67:0/2020736365 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44800e7e0 con 0x556fcf924e60 2024-04-17T10:29:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb442a00640 1 -- 172.21.15.67:0/2020736365 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44800e980 con 0x556fcf924e60 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb442a00640 1 -- 172.21.15.67:0/2020736365 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44800ed30 con 0x556fcf924e60 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- 172.21.15.67:0/2020736365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 msgr2=0x556fcf925240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf925240 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb4480058f0 tx=0x7fb448011000 comp rx=0 tx=0).stop 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- 172.21.15.67:0/2020736365 shutdown_connections 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf8de160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf925780 0x556fcf8dd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/2020736365 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf925240 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- 172.21.15.67:0/2020736365 >> 172.21.15.67:0/2020736365 conn(0x556fcf8d5100 msgr2=0x556fcf8dc120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- 172.21.15.67:0/2020736365 shutdown_connections 2024-04-17T10:29:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- 172.21.15.67:0/2020736365 wait complete. 2024-04-17T10:29:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- start start 2024-04-17T10:29:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 Processor -- start 2024-04-17T10:29:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf8ddd80 0x556fcf8e6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf8e6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf925780 0x556fcf8e6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fcf6e3540 con 0x556fcf924e60 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fcf7250e0 con 0x556fcf925780 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556fcf6e1df0 con 0x556fcf8ddd80 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45fc1d000 10 client.0 _ll_register_callbacks cb 0x556fcf7dd3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf8e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf8e6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48972/0 (socket says 172.21.15.67:48972) 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb459400640 1 -- 172.21.15.67:0/1793979336 learned_addr learned my addr 172.21.15.67:0/1793979336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.261+0000 7fb45a800640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf925780 0x556fcf8e6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459e00640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf8ddd80 0x556fcf8e6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459400640 1 -- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf8ddd80 msgr2=0x556fcf8e6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459400640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf8ddd80 0x556fcf8e6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459400640 1 -- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf925780 msgr2=0x556fcf8e6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459400640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf925780 0x556fcf8e6c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459400640 1 -- 172.21.15.67:0/1793979336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fcf6f1410 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459e00640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf8ddd80 0x556fcf8e6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459400640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf8e6750 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb444013660 tx=0x7fb444013b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443fc9b9deab88b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 1 -- 172.21.15.67:0/1793979336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb444019070 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 1 -- 172.21.15.67:0/1793979336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb444004030 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb44800e980 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 1 -- 172.21.15.67:0/1793979336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb444004230 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 10 client.7011 fetch_fsmap learned FSMap version 47 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x556fcf6b1290 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 1 -- 172.21.15.67:0/1793979336 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44800e980 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 1 -- 172.21.15.67:0/1793979336 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb444028020 con 0x556fcf924e60 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 10 client.7011 fetch_fsmap finished waiting for FSMap version 47 2024-04-17T10:29:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 10 client.7011 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 20 client.7011 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf8e7500 0x556fcf9365e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 4 client.7011 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 --> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556fcf6c4d70 con 0x556fcf8e7500 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459e00640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf8e7500 0x556fcf9365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 10 client.7011 ms_handle_connect on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb459e00640 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf8e7500 0x556fcf9365e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=773401aa91e47d4d server_cookie=e6d98d840e513316 in_seq=0 out_seq=0 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 1 -- 172.21.15.67:0/1793979336 <== mds.0 v2:172.21.15.67:6833/2976334498 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556fcf6c4d70 con 0x556fcf8e7500 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 10 client.7011 handle_command_reply: tid=0 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 client.7011 shutdown 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 msgr2=0x556fcf8e6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf8e6750 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb444013660 tx=0x7fb444013b20 comp rx=0 tx=0).stop 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 shutdown_connections 2024-04-17T10:29:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf8e7500 0x556fcf9365e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf925780 0x556fcf8e6c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf924e60 0x556fcf8e6750 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1793979336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf8ddd80 0x556fcf8e6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb437400640 0 client.7011 ms_handle_reset on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 >> 172.21.15.67:0/1793979336 conn(0x556fcf8d5100 msgr2=0x556fcf8dc830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 shutdown_connections 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1793979336 wait complete. 2024-04-17T10:29:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.265+0000 7fb45fc1d000 20 client.7011 trim_cache size 0 max 16384 2024-04-17T10:29:20.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.281+0000 7fb45fc1d000 1 Processor -- start 2024-04-17T10:29:20.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.281+0000 7fb45fc1d000 1 -- start start 2024-04-17T10:29:20.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.281+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf981890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf981dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf982310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb454203850 con 0x556fcf925780 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4542029d0 con 0x556fcf8ddd80 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb454201dd0 con 0x556fcf924e60 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf982310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf982310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48986/0 (socket says 172.21.15.67:48986) 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 -- 172.21.15.67:0/1837026113 learned_addr learned my addr 172.21.15.67:0/1837026113 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb459400640 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf981dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb459e00640 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf981890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 -- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 msgr2=0x556fcf981dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf981dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 -- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 msgr2=0x556fcf981890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf981890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 -- 172.21.15.67:0/1837026113 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556fcf7ab890 con 0x556fcf925780 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45a800640 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf982310 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fb4500f0ac0 tx=0x7fb450120040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd35f8935006c39 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb442000640 1 -- 172.21.15.67:0/1837026113 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45011c070 con 0x556fcf925780 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb442000640 1 -- 172.21.15.67:0/1837026113 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4500b3040 con 0x556fcf925780 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb442000640 1 -- 172.21.15.67:0/1837026113 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500f1e60 con 0x556fcf925780 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1837026113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 msgr2=0x556fcf982310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf982310 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fb4500f0ac0 tx=0x7fb450120040 comp rx=0 tx=0).stop 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1837026113 shutdown_connections 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf982310 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf981dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/1837026113 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf981890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1837026113 >> 172.21.15.67:0/1837026113 conn(0x556fcf8d5100 msgr2=0x556fcf9345d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:20.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1837026113 shutdown_connections 2024-04-17T10:29:20.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- 172.21.15.67:0/1837026113 wait complete. 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- start start 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 Processor -- start 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf985a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf985f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf9410b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556fcf7ab400 con 0x556fcf925780 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4542023d0 con 0x556fcf8ddd80 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556fcf7250e0 con 0x556fcf924e60 2024-04-17T10:29:20.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.285+0000 7fb45fc1d000 10 client.0 _ll_register_callbacks cb 0x7fb448001e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf9410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb459e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf985a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb459e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf985a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:54084/0 (socket says 172.21.15.67:54084) 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb459400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf985f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb459e00640 1 -- 172.21.15.67:0/3039934929 learned_addr learned my addr 172.21.15.67:0/3039934929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45a800640 1 -- 172.21.15.67:0/3039934929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 msgr2=0x556fcf985f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45a800640 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf985f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45a800640 1 -- 172.21.15.67:0/3039934929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 msgr2=0x556fcf985a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45a800640 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf985a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45a800640 1 -- 172.21.15.67:0/3039934929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556fcf6bde50 con 0x556fcf925780 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb459e00640 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf985a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45a800640 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf9410b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fb4500982c0 tx=0x7fb45011e500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3510d240727481e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:20.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 1 -- 172.21.15.67:0/3039934929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500fe060 con 0x556fcf925780 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 1 -- 172.21.15.67:0/3039934929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb4500f8db0 con 0x556fcf925780 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4500b3040 con 0x556fcf925780 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 1 -- 172.21.15.67:0/3039934929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb45011c070 con 0x556fcf925780 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 10 client.7017 fetch_fsmap learned FSMap version 47 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x556fcf6cff80 con 0x556fcf925780 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 1 -- 172.21.15.67:0/3039934929 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4500b3040 con 0x556fcf925780 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 1 -- 172.21.15.67:0/3039934929 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 47) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb450101050 con 0x556fcf925780 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 10 client.7017 fetch_fsmap finished waiting for FSMap version 47 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 10 client.7017 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 20 client.7017 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf941aa0 0x556fcf941eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 4 client.7017 mds_command: new command op to 6948 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_repair_path", "scrubops": ["repair"]}] 2024-04-17T10:29:20.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 --> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_repair_path", "scrubops": ["repair"]}) v1 -- 0x556fcf736580 con 0x556fcf941aa0 2024-04-17T10:29:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb459e00640 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf941aa0 0x556fcf941eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 10 client.7017 ms_handle_connect on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb459e00640 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf941aa0 0x556fcf941eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=855fc1f68df49cf server_cookie=853a4875d5ef66aa in_seq=0 out_seq=0 2024-04-17T10:29:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 1 -- 172.21.15.67:0/3039934929 <== mds.0 v2:172.21.15.67:6833/2976334498 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556fcf736580 con 0x556fcf941aa0 2024-04-17T10:29:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 10 client.7017 handle_command_reply: tid=0 2024-04-17T10:29:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 client.7017 shutdown 2024-04-17T10:29:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 msgr2=0x556fcf9410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf9410b0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fb4500982c0 tx=0x7fb45011e500 comp rx=0 tx=0).stop 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 shutdown_connections 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:6833/2976334498,v1:172.21.15.67:6835/2976334498] conn(0x556fcf941aa0 0x556fcf941eb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556fcf925780 0x556fcf9410b0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556fcf924e60 0x556fcf985f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 --2- 172.21.15.67:0/3039934929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556fcf8ddd80 0x556fcf985a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb437400640 0 client.7017 ms_handle_reset on v2:172.21.15.67:6833/2976334498 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 >> 172.21.15.67:0/3039934929 conn(0x556fcf8d5100 msgr2=0x556fcf923140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 shutdown_connections 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 1 -- 172.21.15.67:0/3039934929 wait complete. 2024-04-17T10:29:20.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45fc1d000 20 client.7017 trim_cache size 0 max 16384 2024-04-17T10:29:20.293 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:29:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:29:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "e07eec05-6a9e-45cb-a177-4ec23394b313", 2024-04-17T10:29:20.293 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:29:20.293 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb438066f90 msgr2=0x7fb438069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb438066f90 0x7fb438069450 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb448000f70 tx=0x7fb448009040 comp rx=0 tx=0).stop 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb454161af0 msgr2=0x7fb454156fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb454161af0 0x7fb454156fd0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb4500b0040 tx=0x7fb450019680 comp rx=0 tx=0).stop 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 shutdown_connections 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb438066f90 0x7fb438069450 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb454161af0 0x7fb454156fd0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb454161710 0x7fb454156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.289+0000 7fb45c800640 1 --2- 172.21.15.67:0/583877569 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb45415fa30 0x7fb45413f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.293+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 >> 172.21.15.67:0/583877569 conn(0x7fb4540b5e50 msgr2=0x7fb45406f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:20.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.293+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 shutdown_connections 2024-04-17T10:29:20.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:20.293+0000 7fb45c800640 1 -- 172.21.15.67:0/583877569 wait complete. 2024-04-17T10:29:20.300 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T10:29:20.305+0000 7f71f3a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:29:20.304 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_repair_path repair' returned '{'return_code': 0, 'scrub_tag': 'e07eec05-6a9e-45cb-a177-4ec23394b313', 'mode': 'asynchronous'}' 2024-04-17T10:29:25.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:29:25.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:29:30.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:30.306 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir scrub_repair_path) 2024-04-17T10:29:30.319 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:29:30.320 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:29:30.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:29:30.537 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:29:30.537 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:30.537 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:30.547 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:30.547 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:30.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:30.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.797+0000 7fdbfcc00640 1 Processor -- start 2024-04-17T10:29:30.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.797+0000 7fdbfcc00640 1 -- start start 2024-04-17T10:29:30.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbfcc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdbf806e6e0 0x7fdbf80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbfcc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf815ae40 con 0x7fdbf806eac0 2024-04-17T10:29:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbfcc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf8071440 con 0x7fdbf81441f0 2024-04-17T10:29:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbfcc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf8071720 con 0x7fdbf806e6e0 2024-04-17T10:29:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbed400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdbf806eac0 0x7fdbf8143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35362/0 (socket says 172.21.15.67:35362) 2024-04-17T10:29:30.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbed400640 1 -- 172.21.15.67:0/1290499086 learned_addr learned my addr 172.21.15.67:0/1290499086 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/1290499086 >> v2:172.21.15.86:3300/0 conn(0x7fdbf806e6e0 0x7fdbf80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975759533 0 0) 0x7fdbf815ae40 con 0x7fdbf806eac0 2024-04-17T10:29:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbdc003530 con 0x7fdbf806eac0 2024-04-17T10:29:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600303308 0 0) 0x7fdbf8071440 con 0x7fdbf81441f0 2024-04-17T10:29:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf815ae40 con 0x7fdbf81441f0 2024-04-17T10:29:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223669188 0 0) 0x7fdbdc003530 con 0x7fdbf806eac0 2024-04-17T10:29:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 >> v2:172.21.15.86:3300/0 conn(0x7fdbf806e6e0 msgr2=0x7fdbf80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 --2- 172.21.15.67:0/1290499086 >> v2:172.21.15.86:3300/0 conn(0x7fdbf806e6e0 0x7fdbf80ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 >> v1:172.21.15.86:6789/0 conn(0x7fdbf81441f0 legacy=0x7fdbf80b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf8071a00 con 0x7fdbf806eac0 2024-04-17T10:29:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fdbe0001680 con 0x7fdbf806eac0 2024-04-17T10:29:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/1290499086 >> v2:172.21.15.86:3300/0 conn(0x7fdbf806e6e0 0x7fdbf80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 >> v1:172.21.15.67:6789/0 conn(0x7fdbf806eac0 legacy=0x7fdbf8143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbdc003b50 0x7fdbdc003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbdc004810 0x7fdbdc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc0051d0 0x7fdbdc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8071720 con 0x7fdbdc003b50 2024-04-17T10:29:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc00c100 con 0x7fdbdc0051d0 2024-04-17T10:29:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc00c300 con 0x7fdbdc004810 2024-04-17T10:29:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbdc003b50 0x7fdbdc003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbed400640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbdc004810 0x7fdbdc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf6800640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc0051d0 0x7fdbdc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 -- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbdc004810 msgr2=0x7fdbdc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbdc004810 0x7fdbdc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 -- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc0051d0 msgr2=0x7fdbdc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc0051d0 0x7fdbdc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf5e00640 1 -- 172.21.15.67:0/1290499086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc003530 con 0x7fdbdc003b50 2024-04-17T10:29:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.801+0000 7fdbf6800640 1 -- 172.21.15.67:0/1290499086 reap_dead start 2024-04-17T10:29:30.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbdc003b50 0x7fdbdc003f50 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009cb0 tx=0x7fdbe801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c637e517e7b8da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe80b2070 con 0x7fdbdc003b50 2024-04-17T10:29:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe80a5790 con 0x7fdbdc003b50 2024-04-17T10:29:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbf5400640 1 -- 172.21.15.67:0/1290499086 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8009880 con 0x7fdbdc003b50 2024-04-17T10:29:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbfcc00640 1 -- 172.21.15.67:0/1290499086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbdc003b50 msgr2=0x7fdbdc003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbdc003b50 0x7fdbdc003f50 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8009cb0 tx=0x7fdbe801cbe0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbfcc00640 1 -- 172.21.15.67:0/1290499086 shutdown_connections 2024-04-17T10:29:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/1290499086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbdc003b50 0x7fdbdc003f50 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbfcc00640 1 -- 172.21.15.67:0/1290499086 >> 172.21.15.67:0/1290499086 conn(0x7fdbf806ac90 msgr2=0x7fdbf806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbfcc00640 1 -- 172.21.15.67:0/1290499086 shutdown_connections 2024-04-17T10:29:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.805+0000 7fdbfcc00640 1 -- 172.21.15.67:0/1290499086 wait complete. 2024-04-17T10:29:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 Processor -- start 2024-04-17T10:29:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 -- start start 2024-04-17T10:29:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc003b50 0x7fdbf815cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf80b13d0 0x7fdbf80b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf80acaf0 0x7fdbf80acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80cf340 con 0x7fdbf80acaf0 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf80cf1c0 con 0x7fdbdc003b50 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf815e3e0 con 0x7fdbf80b13d0 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf80acaf0 0x7fdbf80acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf80acaf0 0x7fdbf80acfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60950/0 (socket says 172.21.15.67:60950) 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 -- 172.21.15.67:0/2439098971 learned_addr learned my addr 172.21.15.67:0/2439098971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc003b50 0x7fdbf815cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbed400640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf80b13d0 0x7fdbf80b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 -- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf80b13d0 msgr2=0x7fdbf80b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf80b13d0 0x7fdbf80b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 -- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc003b50 msgr2=0x7fdbf815cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc003b50 0x7fdbf815cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 -- 172.21.15.67:0/2439098971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80ad860 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbf6800640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf80acaf0 0x7fdbf80acfa0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fdbf806ef50 tx=0x7fdbf000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbeea00640 1 -- 172.21.15.67:0/2439098971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0006b60 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbeea00640 1 -- 172.21.15.67:0/2439098971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf0006d20 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbeea00640 1 -- 172.21.15.67:0/2439098971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0003bc0 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf812dd20 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf812e2e0 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.809+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf80b1d90 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.813+0000 7fdbeea00640 1 -- 172.21.15.67:0/2439098971 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdbf0005070 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.813+0000 7fdbeea00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.813+0000 7fdbeea00640 1 -- 172.21.15.67:0/2439098971 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdbf0084a80 con 0x7fdbf80acaf0 2024-04-17T10:29:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.813+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:30.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.813+0000 7fdbf5e00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80af000 tx=0x7fdbe80046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:30.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.817+0000 7fdbeea00640 1 -- 172.21.15.67:0/2439098971 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdbf00558b0 con 0x7fdbf80acaf0 2024-04-17T10:29:30.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbf8074e30 con 0x7fdbf80acaf0 2024-04-17T10:29:30.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbeea00640 1 -- 172.21.15.67:0/2439098971 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16963 (secure 0 0 0) 0x7fdbf00551b0 con 0x7fdbf80acaf0 2024-04-17T10:29:30.978 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:30.979 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":71,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:14.294073+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/392417778":"2024-04-18T10:29:14.294053+0000","172.21.15.67:6833/392417778":"2024-04-18T10:29:14.294053+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:30.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 msgr2=0x7fdbd4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fdbe80af000 tx=0x7fdbe80046d0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf80acaf0 msgr2=0x7fdbf80acfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf80acaf0 0x7fdbf80acfa0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fdbf806ef50 tx=0x7fdbf000eb90 comp rx=0 tx=0).stop 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 shutdown_connections 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf80acaf0 0x7fdbf80acfa0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf80b13d0 0x7fdbf80b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 --2- 172.21.15.67:0/2439098971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbdc003b50 0x7fdbf815cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 >> 172.21.15.67:0/2439098971 conn(0x7fdbf806ac90 msgr2=0x7fdbf80b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 shutdown_connections 2024-04-17T10:29:30.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:30.985+0000 7fdbfcc00640 1 -- 172.21.15.67:0/2439098971 wait complete. 2024-04-17T10:29:31.000 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:28:56.861344+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:28:57.434569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:29:31.001 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1c600640 1 Processor -- start 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1c600640 1 -- start start 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1c600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fad140aef60 0x7fad140cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1c600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad1406e980 0x7fad140b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1c600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad140b4990 con 0x7fad140cce60 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1c600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad140af340 con 0x7fad1406e980 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1c600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1415ae90 con 0x7fad140aef60 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1a600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad1406e980 0x7fad140b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1a600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad1406e980 0x7fad140b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60954/0 (socket says 172.21.15.67:60954) 2024-04-17T10:29:31.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1a600640 1 -- 172.21.15.67:0/422595480 learned_addr learned my addr 172.21.15.67:0/422595480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:31.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad19c00640 1 --2- 172.21.15.67:0/422595480 >> v2:172.21.15.86:3300/0 conn(0x7fad140aef60 0x7fad140cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1a600640 1 -- 172.21.15.67:0/422595480 >> v2:172.21.15.86:3300/0 conn(0x7fad140aef60 msgr2=0x7fad140cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:31.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1a600640 1 --2- 172.21.15.67:0/422595480 >> v2:172.21.15.86:3300/0 conn(0x7fad140aef60 0x7fad140cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1a600640 1 -- 172.21.15.67:0/422595480 >> v1:172.21.15.86:6789/0 conn(0x7fad140cce60 legacy=0x7fad1406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:31.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad1a600640 1 -- 172.21.15.67:0/422595480 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1406ee80 con 0x7fad1406e980 2024-04-17T10:29:31.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.153+0000 7fad19c00640 1 --2- 172.21.15.67:0/422595480 >> v2:172.21.15.86:3300/0 conn(0x7fad140aef60 0x7fad140cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1a600640 1 --2- 172.21.15.67:0/422595480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad1406e980 0x7fad140b3120 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fad1406bdc0 tx=0x7fad0c01c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b328badfa1692811 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad13e00640 1 -- 172.21.15.67:0/422595480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0ac020 con 0x7fad1406e980 2024-04-17T10:29:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad13e00640 1 -- 172.21.15.67:0/422595480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad0c0a6410 con 0x7fad1406e980 2024-04-17T10:29:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad13e00640 1 -- 172.21.15.67:0/422595480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0c0a67e0 con 0x7fad1406e980 2024-04-17T10:29:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 -- 172.21.15.67:0/422595480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad1406e980 msgr2=0x7fad140b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 --2- 172.21.15.67:0/422595480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad1406e980 0x7fad140b3120 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fad1406bdc0 tx=0x7fad0c01c740 comp rx=0 tx=0).stop 2024-04-17T10:29:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 -- 172.21.15.67:0/422595480 shutdown_connections 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 --2- 172.21.15.67:0/422595480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad1406e980 0x7fad140b3120 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 --2- 172.21.15.67:0/422595480 >> v2:172.21.15.86:3300/0 conn(0x7fad140aef60 0x7fad140cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 -- 172.21.15.67:0/422595480 >> 172.21.15.67:0/422595480 conn(0x7fad140b1740 msgr2=0x7fad140b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 -- 172.21.15.67:0/422595480 shutdown_connections 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.157+0000 7fad1c600640 1 -- 172.21.15.67:0/422595480 wait complete. 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 Processor -- start 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 -- start start 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad1406e980 0x7fad14149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140aef60 0x7fad1414fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140cce60 0x7fad14150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad140b3ff0 con 0x7fad140aef60 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad140b3e70 con 0x7fad140cce60 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad1406d160 con 0x7fad1406e980 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140aef60 0x7fad1414fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140aef60 0x7fad1414fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60968/0 (socket says 172.21.15.67:60968) 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 -- 172.21.15.67:0/574476582 learned_addr learned my addr 172.21.15.67:0/574476582 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1a600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140cce60 0x7fad14150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19c00640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad1406e980 0x7fad14149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 -- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad1406e980 msgr2=0x7fad14149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad1406e980 0x7fad14149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 -- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140cce60 msgr2=0x7fad14150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140cce60 0x7fad14150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 -- 172.21.15.67:0/574476582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1414f3a0 con 0x7fad140aef60 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad19200640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140aef60 0x7fad1414fb40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fad0000b410 tx=0x7fad0000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad11600640 1 -- 172.21.15.67:0/574476582 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00015030 con 0x7fad140aef60 2024-04-17T10:29:31.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad11600640 1 -- 172.21.15.67:0/574476582 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad00004020 con 0x7fad140aef60 2024-04-17T10:29:31.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad11600640 1 -- 172.21.15.67:0/574476582 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00004450 con 0x7fad140aef60 2024-04-17T10:29:31.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad1414f680 con 0x7fad140aef60 2024-04-17T10:29:31.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.161+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad14136a60 con 0x7fad140aef60 2024-04-17T10:29:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.165+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad1404ef30 con 0x7fad140aef60 2024-04-17T10:29:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.165+0000 7fad11600640 1 -- 172.21.15.67:0/574476582 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fad00018070 con 0x7fad140aef60 2024-04-17T10:29:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.165+0000 7fad11600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facf4066f60 0x7facf4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.165+0000 7fad11600640 1 -- 172.21.15.67:0/574476582 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fad00084d40 con 0x7fad140aef60 2024-04-17T10:29:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.165+0000 7fad19c00640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facf4066f60 0x7facf4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.165+0000 7fad19c00640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facf4066f60 0x7facf4069420 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fad0800ab60 tx=0x7fad08008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.169+0000 7fad11600640 1 -- 172.21.15.67:0/574476582 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fad00055480 con 0x7fad140aef60 2024-04-17T10:29:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.329+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad1415da50 con 0x7fad140aef60 2024-04-17T10:29:31.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.329+0000 7fad11600640 1 -- 172.21.15.67:0/574476582 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad00059520 con 0x7fad140aef60 2024-04-17T10:29:31.324 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:29:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facf4066f60 msgr2=0x7facf4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facf4066f60 0x7facf4069420 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fad0800ab60 tx=0x7fad08008040 comp rx=0 tx=0).stop 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140aef60 msgr2=0x7fad1414fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140aef60 0x7fad1414fb40 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fad0000b410 tx=0x7fad0000b8d0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 shutdown_connections 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7facf4066f60 0x7facf4069420 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad140cce60 0x7fad14150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad140aef60 0x7fad1414fb40 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 --2- 172.21.15.67:0/574476582 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad1406e980 0x7fad14149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 >> 172.21.15.67:0/574476582 conn(0x7fad140b1740 msgr2=0x7fad140b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 shutdown_connections 2024-04-17T10:29:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.333+0000 7fad1c600640 1 -- 172.21.15.67:0/574476582 wait complete. 2024-04-17T10:29:31.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:29:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e6de00640 1 Processor -- start 2024-04-17T10:29:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e6de00640 1 -- start start 2024-04-17T10:29:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e6de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806d810 0x7f8e680cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e6de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8e680707c0 0x7f8e6815d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e6de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e680d03c0 con 0x7f8e6806f8b0 2024-04-17T10:29:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e6de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e680d06a0 con 0x7f8e680707c0 2024-04-17T10:29:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e6de00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e680d0980 con 0x7f8e6806d810 2024-04-17T10:29:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e5e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8e6806f8b0 0x7f8e680700b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35374/0 (socket says 172.21.15.67:35374) 2024-04-17T10:29:31.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e5e000640 1 -- 172.21.15.67:0/63551559 learned_addr learned my addr 172.21.15.67:0/63551559 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e67400640 1 --2- 172.21.15.67:0/63551559 >> v2:172.21.15.67:3300/0 conn(0x7f8e680707c0 0x7f8e6815d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e66a00640 1 --2- 172.21.15.67:0/63551559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806d810 0x7f8e680cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e66000640 1 -- 172.21.15.67:0/63551559 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496939031 0 0) 0x7f8e680d03c0 con 0x7f8e6806f8b0 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e66000640 1 -- 172.21.15.67:0/63551559 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c003530 con 0x7f8e6806f8b0 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e67400640 1 -- 172.21.15.67:0/63551559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806d810 msgr2=0x7f8e680cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e67400640 1 --2- 172.21.15.67:0/63551559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806d810 0x7f8e680cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e67400640 1 -- 172.21.15.67:0/63551559 >> v1:172.21.15.67:6789/0 conn(0x7f8e6806f8b0 legacy=0x7f8e680700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e67400640 1 -- 172.21.15.67:0/63551559 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e680d0c60 con 0x7f8e680707c0 2024-04-17T10:29:31.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.501+0000 7f8e66a00640 1 --2- 172.21.15.67:0/63551559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806d810 0x7f8e680cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:31.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e67400640 1 --2- 172.21.15.67:0/63551559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e680707c0 0x7f8e6815d470 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009d90 tx=0x7f8e6001c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42aaffb5c64915f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e66000640 1 -- 172.21.15.67:0/63551559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e600b2070 con 0x7f8e680707c0 2024-04-17T10:29:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e66000640 1 -- 172.21.15.67:0/63551559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e600a5720 con 0x7f8e680707c0 2024-04-17T10:29:31.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e66000640 1 -- 172.21.15.67:0/63551559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e600a5af0 con 0x7f8e680707c0 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 -- 172.21.15.67:0/63551559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e680707c0 msgr2=0x7f8e6815d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 --2- 172.21.15.67:0/63551559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e680707c0 0x7f8e6815d470 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009d90 tx=0x7f8e6001c940 comp rx=0 tx=0).stop 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 -- 172.21.15.67:0/63551559 shutdown_connections 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 --2- 172.21.15.67:0/63551559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e680707c0 0x7f8e6815d470 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 --2- 172.21.15.67:0/63551559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806d810 0x7f8e680cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 -- 172.21.15.67:0/63551559 >> 172.21.15.67:0/63551559 conn(0x7f8e680d1800 msgr2=0x7f8e680d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 -- 172.21.15.67:0/63551559 shutdown_connections 2024-04-17T10:29:31.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 -- 172.21.15.67:0/63551559 wait complete. 2024-04-17T10:29:31.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 Processor -- start 2024-04-17T10:29:31.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.505+0000 7f8e6de00640 1 -- start start 2024-04-17T10:29:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6806d810 0x7f8e68153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806f8b0 0x7f8e6815c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e680707c0 0x7f8e681f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6806bda0 con 0x7f8e6806d810 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6806de00 con 0x7f8e680707c0 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6806bf20 con 0x7f8e6806f8b0 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6806d810 0x7f8e68153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6806d810 0x7f8e68153f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60994/0 (socket says 172.21.15.67:60994) 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 -- 172.21.15.67:0/3512675851 learned_addr learned my addr 172.21.15.67:0/3512675851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e5e000640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806f8b0 0x7f8e6815c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e67400640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e680707c0 0x7f8e681f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 -- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806f8b0 msgr2=0x7f8e6815c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806f8b0 0x7f8e6815c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 -- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e680707c0 msgr2=0x7f8e681f9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e680707c0 0x7f8e681f9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 -- 172.21.15.67:0/3512675851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e680685d0 con 0x7f8e6806d810 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e5e000640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806f8b0 0x7f8e6815c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:31.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e67400640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e680707c0 0x7f8e681f9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:31.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e66a00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6806d810 0x7f8e68153f70 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800e840 tx=0x7f8e5800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e5fe00640 1 -- 172.21.15.67:0/3512675851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58017070 con 0x7f8e6806d810 2024-04-17T10:29:31.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e5fe00640 1 -- 172.21.15.67:0/3512675851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5800c8d0 con 0x7f8e6806d810 2024-04-17T10:29:31.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e5fe00640 1 -- 172.21.15.67:0/3512675851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5800f040 con 0x7f8e6806d810 2024-04-17T10:29:31.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e680b4b30 con 0x7f8e6806d810 2024-04-17T10:29:31.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e680b50f0 con 0x7f8e6806d810 2024-04-17T10:29:31.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.509+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e6804ef30 con 0x7f8e6806d810 2024-04-17T10:29:31.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.513+0000 7f8e5fe00640 1 -- 172.21.15.67:0/3512675851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8e5801a070 con 0x7f8e6806d810 2024-04-17T10:29:31.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.513+0000 7f8e5fe00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e44066f60 0x7f8e44069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.513+0000 7f8e5fe00640 1 -- 172.21.15.67:0/3512675851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8e58084db0 con 0x7f8e6806d810 2024-04-17T10:29:31.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.513+0000 7f8e5e000640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e44066f60 0x7f8e44069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.513+0000 7f8e5e000640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e44066f60 0x7f8e44069420 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8e6809f780 tx=0x7f8e50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.517+0000 7f8e5fe00640 1 -- 172.21.15.67:0/3512675851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8e58089070 con 0x7f8e6806d810 2024-04-17T10:29:31.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e680b7780 con 0x7f8e6806d810 2024-04-17T10:29:31.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e5fe00640 1 -- 172.21.15.67:0/3512675851 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e58055bd0 con 0x7f8e6806d810 2024-04-17T10:29:31.671 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:29:31.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e44066f60 msgr2=0x7f8e44069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e44066f60 0x7f8e44069420 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8e6809f780 tx=0x7f8e50009040 comp rx=0 tx=0).stop 2024-04-17T10:29:31.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6806d810 msgr2=0x7f8e68153f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6806d810 0x7f8e68153f70 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800e840 tx=0x7f8e5800ed00 comp rx=0 tx=0).stop 2024-04-17T10:29:31.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 shutdown_connections 2024-04-17T10:29:31.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e44066f60 0x7f8e44069420 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8e680707c0 0x7f8e681f9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8e6806f8b0 0x7f8e6815c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 --2- 172.21.15.67:0/3512675851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e6806d810 0x7f8e68153f70 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 >> 172.21.15.67:0/3512675851 conn(0x7f8e680d1800 msgr2=0x7f8e680b6f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:31.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 shutdown_connections 2024-04-17T10:29:31.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.677+0000 7f8e6de00640 1 -- 172.21.15.67:0/3512675851 wait complete. 2024-04-17T10:29:31.692 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:29:31.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a13c00640 1 Processor -- start 2024-04-17T10:29:31.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a13c00640 1 -- start start 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a13c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1a0c0cd030 0x7f1a0c06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a13c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 0x7f1a0c0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a13c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0c0b4990 con 0x7f1a0c0aef60 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a13c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a0c0af340 con 0x7f1a0c0cd030 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a13c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c06ec00 con 0x7f1a0c06e7b0 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a11200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1a0c0aef60 0x7f1a0c0cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35384/0 (socket says 172.21.15.67:35384) 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a11200640 1 -- 172.21.15.67:0/2062889864 learned_addr learned my addr 172.21.15.67:0/2062889864 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a03e00640 1 --2- 172.21.15.67:0/2062889864 >> v2:172.21.15.86:3300/0 conn(0x7f1a0c0cd030 0x7f1a0c06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a11c00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 0x7f1a0c0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.841+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654350088 0 0) 0x7f19fc001120 con 0x7f1a0c0aef60 2024-04-17T10:29:31.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19f0003530 con 0x7f1a0c0aef60 2024-04-17T10:29:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 143380618 0 0) 0x7f19f0003530 con 0x7f1a0c0aef60 2024-04-17T10:29:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 msgr2=0x7f1a0c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 0x7f1a0c0b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 >> v2:172.21.15.86:3300/0 conn(0x7f1a0c0cd030 msgr2=0x7f1a0c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 --2- 172.21.15.67:0/2062889864 >> v2:172.21.15.86:3300/0 conn(0x7f1a0c0cd030 0x7f1a0c06e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c15ae90 con 0x7f1a0c0aef60 2024-04-17T10:29:31.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f19fc001870 con 0x7f1a0c0aef60 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 >> v1:172.21.15.67:6789/0 conn(0x7f1a0c0aef60 legacy=0x7f1a0c0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f00039c0 0x7f19f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 0x7f19f0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f0004d50 0x7f19f00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c0af340 con 0x7f19f0004d50 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c06ec00 con 0x7f19f00039c0 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f000ba00 con 0x7f1a0c06e7b0 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a11200640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f00039c0 0x7f19f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a03e00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 0x7f19f0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a11c00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f0004d50 0x7f19f00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a11c00640 1 -- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 msgr2=0x7f19f0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a11c00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c06e7b0 0x7f19f0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a11c00640 1 -- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f00039c0 msgr2=0x7f19f0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:31.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a11c00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f00039c0 0x7f19f0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.845+0000 7f1a11c00640 1 -- 172.21.15.67:0/2062889864 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f0003530 con 0x7f19f0004d50 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a11c00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f0004d50 0x7f19f00051d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009fc0 tx=0x7f1a0401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51bcabf02c86ddd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a040b2070 con 0x7f19f0004d50 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a040a5b80 con 0x7f19f0004d50 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a0be00640 1 -- 172.21.15.67:0/2062889864 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a04009c10 con 0x7f19f0004d50 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- 172.21.15.67:0/2062889864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f0004d50 msgr2=0x7f19f00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 --2- 172.21.15.67:0/2062889864 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19f0004d50 0x7f19f00051d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f1a04009fc0 tx=0x7f1a0401cc80 comp rx=0 tx=0).stop 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a11c00640 1 -- 172.21.15.67:0/2062889864 reap_dead start 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- 172.21.15.67:0/2062889864 shutdown_connections 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- 172.21.15.67:0/2062889864 >> 172.21.15.67:0/2062889864 conn(0x7f1a0c0b1740 msgr2=0x7f1a0c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- 172.21.15.67:0/2062889864 shutdown_connections 2024-04-17T10:29:31.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- 172.21.15.67:0/2062889864 wait complete. 2024-04-17T10:29:31.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 Processor -- start 2024-04-17T10:29:31.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- start start 2024-04-17T10:29:31.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f0004d50 0x7f1a0c1430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c14b170 0x7f1a0c0ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0aa730 0x7f1a0c0acde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c15b410 con 0x7f1a0c0aa730 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c15b130 con 0x7f19f0004d50 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.849+0000 7f1a13c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0c15bc10 con 0x7f1a0c14b170 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0aa730 0x7f1a0c0acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0aa730 0x7f1a0c0acde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32774/0 (socket says 172.21.15.67:32774) 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 -- 172.21.15.67:0/3928558553 learned_addr learned my addr 172.21.15.67:0/3928558553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11200640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f0004d50 0x7f1a0c1430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a03e00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c14b170 0x7f1a0c0ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 -- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c14b170 msgr2=0x7f1a0c0ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c14b170 0x7f1a0c0ac0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 -- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f0004d50 msgr2=0x7f1a0c1430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:31.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f0004d50 0x7f1a0c1430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 -- 172.21.15.67:0/3928558553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c0ac610 con 0x7f1a0c0aa730 2024-04-17T10:29:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a11c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0aa730 0x7f1a0c0acde0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1a040b0040 tx=0x7f1a04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a09600640 1 -- 172.21.15.67:0/3928558553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a040bd070 con 0x7f1a0c0aa730 2024-04-17T10:29:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a09600640 1 -- 172.21.15.67:0/3928558553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a040a5980 con 0x7f1a0c0aa730 2024-04-17T10:29:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0c0ab030 con 0x7f1a0c0aa730 2024-04-17T10:29:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a09600640 1 -- 172.21.15.67:0/3928558553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a040b2040 con 0x7f1a0c0aa730 2024-04-17T10:29:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0c0ab5f0 con 0x7f1a0c0aa730 2024-04-17T10:29:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0c04ef30 con 0x7f1a0c0aa730 2024-04-17T10:29:31.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.853+0000 7f1a09600640 1 -- 172.21.15.67:0/3928558553 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1a040aecf0 con 0x7f1a0c0aa730 2024-04-17T10:29:31.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.857+0000 7f1a09600640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19e8066f60 0x7f19e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:31.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.857+0000 7f1a09600640 1 -- 172.21.15.67:0/3928558553 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1a0411f3d0 con 0x7f1a0c0aa730 2024-04-17T10:29:31.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.857+0000 7f1a11200640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19e8066f60 0x7f19e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:31.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.857+0000 7f1a11200640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19e8066f60 0x7f19e8069420 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f19fc000c10 tx=0x7f19fc007af0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:31.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:31.861+0000 7f1a09600640 1 -- 172.21.15.67:0/3928558553 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1a04016990 con 0x7f1a0c0aa730 2024-04-17T10:29:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a0c151830 con 0x7f1a0c0aa730 2024-04-17T10:29:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a09600640 1 -- 172.21.15.67:0/3928558553 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a040efb10 con 0x7f1a0c0aa730 2024-04-17T10:29:32.014 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:29:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19e8066f60 msgr2=0x7f19e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19e8066f60 0x7f19e8069420 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f19fc000c10 tx=0x7f19fc007af0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0aa730 msgr2=0x7f1a0c0acde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0aa730 0x7f1a0c0acde0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f1a040b0040 tx=0x7f1a04009750 comp rx=0 tx=0).stop 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 shutdown_connections 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f19e8066f60 0x7f19e8069420 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a0c0aa730 0x7f1a0c0acde0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a0c14b170 0x7f1a0c0ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 --2- 172.21.15.67:0/3928558553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19f0004d50 0x7f1a0c1430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 >> 172.21.15.67:0/3928558553 conn(0x7f1a0c0b1740 msgr2=0x7f1a0c14fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 shutdown_connections 2024-04-17T10:29:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.021+0000 7f1a13c00640 1 -- 172.21.15.67:0/3928558553 wait complete. 2024-04-17T10:29:32.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:29:32.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8489a00640 1 Processor -- start 2024-04-17T10:29:32.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8489a00640 1 -- start start 2024-04-17T10:29:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8489a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 0x7f8484144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8489a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8484144aa0 0x7f84840d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8489a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84840b6640 con 0x7f8484071840 2024-04-17T10:29:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8489a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84840b51a0 con 0x7f8484144aa0 2024-04-17T10:29:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8489a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84841456d0 con 0x7f84841446c0 2024-04-17T10:29:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8483400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8484071840 0x7f848415f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35386/0 (socket says 172.21.15.67:35386) 2024-04-17T10:29:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8483400640 1 -- 172.21.15.67:0/152729998 learned_addr learned my addr 172.21.15.67:0/152729998 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482a00640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 0x7f8484144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f847a000640 1 --2- 172.21.15.67:0/152729998 >> v2:172.21.15.86:3300/0 conn(0x7f8484144aa0 0x7f84840d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543744141 0 0) 0x7f84840b6640 con 0x7f8484071840 2024-04-17T10:29:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8468003530 con 0x7f8484071840 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535980541 0 0) 0x7f8468003530 con 0x7f8484071840 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 msgr2=0x7f8484144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 0x7f8484144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 >> v2:172.21.15.86:3300/0 conn(0x7f8484144aa0 msgr2=0x7f84840d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 --2- 172.21.15.67:0/152729998 >> v2:172.21.15.86:3300/0 conn(0x7f8484144aa0 0x7f84840d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84840b4870 con 0x7f8484071840 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f847c001680 con 0x7f8484071840 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 >> v1:172.21.15.67:6789/0 conn(0x7f8484071840 legacy=0x7f848415f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84680039c0 0x7f8468003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004810 0x7f8468004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 0x7f84680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84840b51a0 con 0x7f8468004810 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84841456d0 con 0x7f84680039c0 2024-04-17T10:29:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846800b9d0 con 0x7f84841446c0 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f847a000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004810 0x7f8468004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8482a00640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84680039c0 0x7f8468003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f8483400640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 0x7f84680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f847a000640 1 -- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 msgr2=0x7f84680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f847a000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f84841446c0 0x7f84680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f847a000640 1 -- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84680039c0 msgr2=0x7f8468003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f847a000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f84680039c0 0x7f8468003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.185+0000 7f847a000640 1 -- 172.21.15.67:0/152729998 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8468003530 con 0x7f8468004810 2024-04-17T10:29:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f847a000640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004810 0x7f8468004c90 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f846c009b10 tx=0x7f846c01ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3424bb6c5e876ba6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0b2070 con 0x7f8468004810 2024-04-17T10:29:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c0a5570 con 0x7f8468004810 2024-04-17T10:29:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8482000640 1 -- 172.21.15.67:0/152729998 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c0096b0 con 0x7f8468004810 2024-04-17T10:29:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8489a00640 1 -- 172.21.15.67:0/152729998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004810 msgr2=0x7f8468004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8489a00640 1 --2- 172.21.15.67:0/152729998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8468004810 0x7f8468004c90 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f846c009b10 tx=0x7f846c01ca80 comp rx=0 tx=0).stop 2024-04-17T10:29:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8483400640 1 -- 172.21.15.67:0/152729998 reap_dead start 2024-04-17T10:29:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8489a00640 1 -- 172.21.15.67:0/152729998 shutdown_connections 2024-04-17T10:29:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8489a00640 1 -- 172.21.15.67:0/152729998 >> 172.21.15.67:0/152729998 conn(0x7f84840b34d0 msgr2=0x7f84840b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8489a00640 1 -- 172.21.15.67:0/152729998 shutdown_connections 2024-04-17T10:29:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.189+0000 7f8489a00640 1 -- 172.21.15.67:0/152729998 wait complete. 2024-04-17T10:29:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 Processor -- start 2024-04-17T10:29:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 -- start start 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8468004810 0x7f84841eacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8484133d80 0x7f84841e9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84841341e0 0x7f84841ea400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84841452b0 con 0x7f84841341e0 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84840d1260 con 0x7f8484133d80 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8484145430 con 0x7f8468004810 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84841341e0 0x7f84841ea400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84841341e0 0x7f84841ea400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32790/0 (socket says 172.21.15.67:32790) 2024-04-17T10:29:32.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 -- 172.21.15.67:0/1651763411 learned_addr learned my addr 172.21.15.67:0/1651763411 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8482a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8468004810 0x7f84841eacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f847a000640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8484133d80 0x7f84841e9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 -- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8468004810 msgr2=0x7f84841eacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8468004810 0x7f84841eacc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 -- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8484133d80 msgr2=0x7f84841e9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8484133d80 0x7f84841e9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 -- 172.21.15.67:0/1651763411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84841e8e20 con 0x7f84841341e0 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f847a000640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8484133d80 0x7f84841e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8483400640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84841341e0 0x7f84841ea400 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f848406f0e0 tx=0x7f847c00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f847b400640 1 -- 172.21.15.67:0/1651763411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c019070 con 0x7f84841341e0 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f847b400640 1 -- 172.21.15.67:0/1651763411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c00c690 con 0x7f84841341e0 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f847b400640 1 -- 172.21.15.67:0/1651763411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f847c00ca80 con 0x7f84841341e0 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84841e9100 con 0x7f84841341e0 2024-04-17T10:29:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.193+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84841e96c0 con 0x7f84841341e0 2024-04-17T10:29:32.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.197+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8484070bb0 con 0x7f84841341e0 2024-04-17T10:29:32.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.197+0000 7f847b400640 1 -- 172.21.15.67:0/1651763411 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f847c00cc20 con 0x7f84841341e0 2024-04-17T10:29:32.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.197+0000 7f847b400640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8460066f90 0x7f8460069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.197+0000 7f847b400640 1 -- 172.21.15.67:0/1651763411 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f847c084710 con 0x7f84841341e0 2024-04-17T10:29:32.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.197+0000 7f8482a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8460066f90 0x7f8460069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.197+0000 7f8482a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8460066f90 0x7f8460069450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8474009f80 tx=0x7f847400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:32.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.201+0000 7f847b400640 1 -- 172.21.15.67:0/1651763411 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f847c055410 con 0x7f84841341e0 2024-04-17T10:29:32.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.377+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8484074e30 con 0x7f84841341e0 2024-04-17T10:29:32.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.377+0000 7f847b400640 1 -- 172.21.15.67:0/1651763411 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v71) v1 ==== 80+0+120 (secure 0 0 0) 0x7f847c054d10 con 0x7f84841341e0 2024-04-17T10:29:32.373 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6835/392417778 2024-04-18T10:29:14.294053+0000 2024-04-17T10:29:32.373 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6833/392417778 2024-04-18T10:29:14.294053+0000 2024-04-17T10:29:32.373 INFO:teuthology.orchestra.run.smithi067.stderr:listed 2 entries 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8460066f90 msgr2=0x7f8460069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8460066f90 0x7f8460069450 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8474009f80 tx=0x7f847400b040 comp rx=0 tx=0).stop 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84841341e0 msgr2=0x7f84841ea400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84841341e0 0x7f84841ea400 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f848406f0e0 tx=0x7f847c00eb90 comp rx=0 tx=0).stop 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 shutdown_connections 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8460066f90 0x7f8460069450 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84841341e0 0x7f84841ea400 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8484133d80 0x7f84841e9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 --2- 172.21.15.67:0/1651763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8468004810 0x7f84841eacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 >> 172.21.15.67:0/1651763411 conn(0x7f84840b34d0 msgr2=0x7f84840b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:32.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 shutdown_connections 2024-04-17T10:29:32.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.381+0000 7f8489a00640 1 -- 172.21.15.67:0/1651763411 wait complete. 2024-04-17T10:29:32.394 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:29:32.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:32.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:32.423 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:29:32.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:32.423 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:29:32.477 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:29:32.478 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:32.487 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:29:32.488 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:32.488 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:32.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:29:32.530 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:29:32.530 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:32.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:29:32.543 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:32.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.721+0000 7fcf0c200640 1 Processor -- start 2024-04-17T10:29:32.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.721+0000 7fcf0c200640 1 -- start start 2024-04-17T10:29:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.721+0000 7fcf0c200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf040ce020 0x7fcf041620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.721+0000 7fcf0c200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0415eb50 con 0x7fcf040ce400 2024-04-17T10:29:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.721+0000 7fcf0c200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0415ee30 con 0x7fcf040cf510 2024-04-17T10:29:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.721+0000 7fcf0c200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf04161650 con 0x7fcf040ce020 2024-04-17T10:29:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf09800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf040ce020 0x7fcf041620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf08e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcf040ce400 0x7fcf0415e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35394/0 (socket says 172.21.15.67:35394) 2024-04-17T10:29:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf08e00640 1 -- 172.21.15.67:0/1476890705 learned_addr learned my addr 172.21.15.67:0/1476890705 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:32.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf09800640 1 -- 172.21.15.67:0/1476890705 >> v1:172.21.15.86:6789/0 conn(0x7fcf040cf510 legacy=0x7fcf040cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:32.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf09800640 1 -- 172.21.15.67:0/1476890705 >> v1:172.21.15.67:6789/0 conn(0x7fcf040ce400 legacy=0x7fcf0415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf09800640 1 -- 172.21.15.67:0/1476890705 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf041618a0 con 0x7fcf040ce020 2024-04-17T10:29:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf09800640 1 --2- 172.21.15.67:0/1476890705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce020 0x7fcf041620a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcef80097f0 tx=0x7fcef801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e39ccbab0b25ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf03e00640 1 -- 172.21.15.67:0/1476890705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef80b1070 con 0x7fcf040ce020 2024-04-17T10:29:32.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf03e00640 1 -- 172.21.15.67:0/1476890705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef80ac410 con 0x7fcf040ce020 2024-04-17T10:29:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.725+0000 7fcf03e00640 1 -- 172.21.15.67:0/1476890705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef80ac7e0 con 0x7fcf040ce020 2024-04-17T10:29:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- 172.21.15.67:0/1476890705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce020 msgr2=0x7fcf041620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1476890705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce020 0x7fcf041620a0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcef80097f0 tx=0x7fcef801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- 172.21.15.67:0/1476890705 shutdown_connections 2024-04-17T10:29:32.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1476890705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce020 0x7fcf041620a0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- 172.21.15.67:0/1476890705 >> 172.21.15.67:0/1476890705 conn(0x7fcf040b33f0 msgr2=0x7fcf040b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- 172.21.15.67:0/1476890705 shutdown_connections 2024-04-17T10:29:32.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- 172.21.15.67:0/1476890705 wait complete. 2024-04-17T10:29:32.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 Processor -- start 2024-04-17T10:29:32.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- start start 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040ce020 0x7fcf04158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce400 0x7fcf04153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf040cf510 0x7fcf0415a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0415e8a0 con 0x7fcf040ce400 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0415e720 con 0x7fcf040ce020 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0c200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040c84b0 con 0x7fcf040cf510 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce400 0x7fcf04153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce400 0x7fcf04153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32820/0 (socket says 172.21.15.67:32820) 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf09800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040ce020 0x7fcf04158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 -- 172.21.15.67:0/1571010570 learned_addr learned my addr 172.21.15.67:0/1571010570 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf0a200640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf040cf510 0x7fcf0415a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 -- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf040cf510 msgr2=0x7fcf0415a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf040cf510 0x7fcf0415a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 -- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040ce020 msgr2=0x7fcf04158f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040ce020 0x7fcf04158f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.729+0000 7fcf08e00640 1 -- 172.21.15.67:0/1571010570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf04157bd0 con 0x7fcf040ce400 2024-04-17T10:29:32.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf08e00640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce400 0x7fcf04153460 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c500 tx=0x7fcef000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf01600640 1 -- 172.21.15.67:0/1571010570 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0017070 con 0x7fcf040ce400 2024-04-17T10:29:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf01600640 1 -- 172.21.15.67:0/1571010570 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0005cf0 con 0x7fcf040ce400 2024-04-17T10:29:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf01600640 1 -- 172.21.15.67:0/1571010570 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef00072f0 con 0x7fcf040ce400 2024-04-17T10:29:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf04157eb0 con 0x7fcf040ce400 2024-04-17T10:29:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf04158470 con 0x7fcf040ce400 2024-04-17T10:29:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf04070ab0 con 0x7fcf040ce400 2024-04-17T10:29:32.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf01600640 1 -- 172.21.15.67:0/1571010570 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcef001a070 con 0x7fcf040ce400 2024-04-17T10:29:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf01600640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:32.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.733+0000 7fcf01600640 1 -- 172.21.15.67:0/1571010570 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcef0084530 con 0x7fcf040ce400 2024-04-17T10:29:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.737+0000 7fcf09800640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:32.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.737+0000 7fcf09800640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcef80ae000 tx=0x7fcef8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:32.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.737+0000 7fcf01600640 1 -- 172.21.15.67:0/1571010570 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcef0054d00 con 0x7fcf040ce400 2024-04-17T10:29:32.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.937+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf040cf040 con 0x7fcf040ce400 2024-04-17T10:29:32.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.937+0000 7fcf01600640 1 -- 172.21.15.67:0/1571010570 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4937 (secure 0 0 0) 0x7fcef0058da0 con 0x7fcf040ce400 2024-04-17T10:29:32.932 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:32.932 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3163523544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3163523544},{"type":"v1","addr":"172.21.15.86:6835","nonce":3163523544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3682484259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3682484259},{"type":"v1","addr":"172.21.15.86:6837","nonce":3682484259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6945,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/252088612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":252088612},{"type":"v1","addr":"172.21.15.67:6834","nonce":252088612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:17.323326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/2976334498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2976334498},{"type":"v1","addr":"172.21.15.67:6835","nonce":2976334498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:32.932 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 47 2024-04-17T10:29:32.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 msgr2=0x7fcee4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcef80ae000 tx=0x7fcef8008040 comp rx=0 tx=0).stop 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce400 msgr2=0x7fcf04153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce400 0x7fcf04153460 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c500 tx=0x7fcef000c9c0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 shutdown_connections 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf040cf510 0x7fcf0415a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040ce400 0x7fcf04153460 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 --2- 172.21.15.67:0/1571010570 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040ce020 0x7fcf04158f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 >> 172.21.15.67:0/1571010570 conn(0x7fcf040b33f0 msgr2=0x7fcf040cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 shutdown_connections 2024-04-17T10:29:32.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:32.941+0000 7fcf0c200640 1 -- 172.21.15.67:0/1571010570 wait complete. 2024-04-17T10:29:32.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:33.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741f000640 1 Processor -- start 2024-04-17T10:29:33.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741f000640 1 -- start start 2024-04-17T10:29:33.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741f000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7418144aa0 0x7f74180d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741f000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418071840 0x7f74180d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741f000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74181456d0 con 0x7f74181446c0 2024-04-17T10:29:33.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741f000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74180b51a0 con 0x7f7418144aa0 2024-04-17T10:29:33.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741f000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74180d1390 con 0x7f7418071840 2024-04-17T10:29:33.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f7417e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f74181446c0 0x7f7418144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35396/0 (socket says 172.21.15.67:35396) 2024-04-17T10:29:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f7417e00640 1 -- 172.21.15.67:0/3844695678 learned_addr learned my addr 172.21.15.67:0/3844695678 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f740fe00640 1 --2- 172.21.15.67:0/3844695678 >> v2:172.21.15.86:3300/0 conn(0x7f7418144aa0 0x7f74180d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f741d000640 1 --2- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418071840 0x7f74180d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f7417400640 1 -- 172.21.15.67:0/3844695678 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693798590 0 0) 0x7f74181456d0 con 0x7f74181446c0 2024-04-17T10:29:33.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f7417400640 1 -- 172.21.15.67:0/3844695678 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73fc003530 con 0x7f74181446c0 2024-04-17T10:29:33.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f740fe00640 1 -- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418071840 msgr2=0x7f74180d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:33.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f740fe00640 1 --2- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418071840 0x7f74180d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f740fe00640 1 -- 172.21.15.67:0/3844695678 >> v1:172.21.15.67:6789/0 conn(0x7f74181446c0 legacy=0x7f7418144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.101+0000 7f740fe00640 1 -- 172.21.15.67:0/3844695678 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74180b6640 con 0x7f7418144aa0 2024-04-17T10:29:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f740fe00640 1 --2- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418144aa0 0x7f74180d0910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f74000097c0 tx=0x7f740001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26b5a57a24327ed server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f7417400640 1 -- 172.21.15.67:0/3844695678 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74000ab650 con 0x7f7418144aa0 2024-04-17T10:29:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f7417400640 1 -- 172.21.15.67:0/3844695678 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000b0070 con 0x7f7418144aa0 2024-04-17T10:29:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f7417400640 1 -- 172.21.15.67:0/3844695678 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74000ab9f0 con 0x7f7418144aa0 2024-04-17T10:29:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 -- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418144aa0 msgr2=0x7f74180d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 --2- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418144aa0 0x7f74180d0910 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f74000097c0 tx=0x7f740001c450 comp rx=0 tx=0).stop 2024-04-17T10:29:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 -- 172.21.15.67:0/3844695678 shutdown_connections 2024-04-17T10:29:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 --2- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418071840 0x7f74180d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 --2- 172.21.15.67:0/3844695678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418144aa0 0x7f74180d0910 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 -- 172.21.15.67:0/3844695678 >> 172.21.15.67:0/3844695678 conn(0x7f74180b34d0 msgr2=0x7f74180b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 -- 172.21.15.67:0/3844695678 shutdown_connections 2024-04-17T10:29:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.105+0000 7f741f000640 1 -- 172.21.15.67:0/3844695678 wait complete. 2024-04-17T10:29:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 Processor -- start 2024-04-17T10:29:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 -- start start 2024-04-17T10:29:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418071840 0x7f741815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74181446c0 0x7f74181f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418144aa0 0x7f741806bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74180b5e20 con 0x7f74181446c0 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741815c0d0 con 0x7f7418071840 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74180b5fa0 con 0x7f7418144aa0 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74181446c0 0x7f74181f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74181446c0 0x7f74181f5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32832/0 (socket says 172.21.15.67:32832) 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 -- 172.21.15.67:0/1991435230 learned_addr learned my addr 172.21.15.67:0/1991435230 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f7417e00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418071840 0x7f741815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741d000640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418144aa0 0x7f741806bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 -- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418144aa0 msgr2=0x7f741806bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418144aa0 0x7f741806bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 -- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418071840 msgr2=0x7f741815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418071840 0x7f741815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 -- 172.21.15.67:0/1991435230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741806dd40 con 0x7f74181446c0 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f7417e00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418071840 0x7f741815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f740fe00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74181446c0 0x7f74181f5c60 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f74000ad040 tx=0x7f74000191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f7414c00640 1 -- 172.21.15.67:0/1991435230 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400003820 con 0x7f74181446c0 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f7414c00640 1 -- 172.21.15.67:0/1991435230 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74000039c0 con 0x7f74181446c0 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741806d4a0 con 0x7f74181446c0 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f7414c00640 1 -- 172.21.15.67:0/1991435230 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7400003d90 con 0x7f74181446c0 2024-04-17T10:29:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741806da60 con 0x7f74181446c0 2024-04-17T10:29:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.109+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7418070bb0 con 0x7f74181446c0 2024-04-17T10:29:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.113+0000 7f7414c00640 1 -- 172.21.15.67:0/1991435230 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f74000c3060 con 0x7f74181446c0 2024-04-17T10:29:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.113+0000 7f7414c00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f4066f60 0x7f73f4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.113+0000 7f7414c00640 1 -- 172.21.15.67:0/1991435230 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f740011ed60 con 0x7f74181446c0 2024-04-17T10:29:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.113+0000 7f7417e00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f4066f60 0x7f73f4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.113+0000 7f7417e00640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f4066f60 0x7f73f4069420 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f741813ea90 tx=0x7f7408009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.117+0000 7f7414c00640 1 -- 172.21.15.67:0/1991435230 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7400016070 con 0x7f74181446c0 2024-04-17T10:29:33.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74180b41a0 con 0x7f74181446c0 2024-04-17T10:29:33.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f7414c00640 1 -- 172.21.15.67:0/1991435230 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f74000efb80 con 0x7f74181446c0 2024-04-17T10:29:33.275 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:33.276 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":71,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:14.294073+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/392417778":"2024-04-18T10:29:14.294053+0000","172.21.15.67:6833/392417778":"2024-04-18T10:29:14.294053+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f4066f60 msgr2=0x7f73f4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f4066f60 0x7f73f4069420 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f741813ea90 tx=0x7f7408009040 comp rx=0 tx=0).stop 2024-04-17T10:29:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74181446c0 msgr2=0x7f74181f5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74181446c0 0x7f74181f5c60 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f74000ad040 tx=0x7f74000191f0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 shutdown_connections 2024-04-17T10:29:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f4066f60 0x7f73f4069420 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7418144aa0 0x7f741806bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74181446c0 0x7f74181f5c60 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 --2- 172.21.15.67:0/1991435230 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7418071840 0x7f741815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.281+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 >> 172.21.15.67:0/1991435230 conn(0x7f74180b34d0 msgr2=0x7f74181415a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.285+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 shutdown_connections 2024-04-17T10:29:33.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.285+0000 7f741f000640 1 -- 172.21.15.67:0/1991435230 wait complete. 2024-04-17T10:29:33.296 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:28:56.861344+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:28:57.434569+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:29:33.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:29:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e8a600640 1 Processor -- start 2024-04-17T10:29:33.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e8a600640 1 -- start start 2024-04-17T10:29:33.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e8a600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e8406e980 0x7f4e841653b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e8a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e840b4990 con 0x7f4e840aef60 2024-04-17T10:29:33.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e8a600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8415ae90 con 0x7f4e840cd030 2024-04-17T10:29:33.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e8a600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840af340 con 0x7f4e8406e980 2024-04-17T10:29:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e83e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e8406e980 0x7f4e841653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e83400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4e840aef60 0x7f4e840cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35402/0 (socket says 172.21.15.67:35402) 2024-04-17T10:29:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.449+0000 7f4e83400640 1 -- 172.21.15.67:0/2156569834 learned_addr learned my addr 172.21.15.67:0/2156569834 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e82a00640 1 -- 172.21.15.67:0/2156569834 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612797194 0 0) 0x7f4e74001120 con 0x7f4e840aef60 2024-04-17T10:29:33.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e82a00640 1 -- 172.21.15.67:0/2156569834 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e68003530 con 0x7f4e840aef60 2024-04-17T10:29:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e83e00640 1 -- 172.21.15.67:0/2156569834 >> v1:172.21.15.86:6789/0 conn(0x7f4e840cd030 legacy=0x7f4e8406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e83e00640 1 -- 172.21.15.67:0/2156569834 >> v1:172.21.15.67:6789/0 conn(0x7f4e840aef60 legacy=0x7f4e840cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e83e00640 1 -- 172.21.15.67:0/2156569834 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8406ee80 con 0x7f4e8406e980 2024-04-17T10:29:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e83e00640 1 --2- 172.21.15.67:0/2156569834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841653b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009d90 tx=0x7f4e7c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ad44587a12c6c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e82a00640 1 -- 172.21.15.67:0/2156569834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0b2070 con 0x7f4e8406e980 2024-04-17T10:29:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e82a00640 1 -- 172.21.15.67:0/2156569834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c0a57a0 con 0x7f4e8406e980 2024-04-17T10:29:33.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e82a00640 1 -- 172.21.15.67:0/2156569834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c0a5b50 con 0x7f4e8406e980 2024-04-17T10:29:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e8a600640 1 -- 172.21.15.67:0/2156569834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 msgr2=0x7f4e841653b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e8a600640 1 --2- 172.21.15.67:0/2156569834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841653b0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009d90 tx=0x7f4e7c01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e8a600640 1 -- 172.21.15.67:0/2156569834 shutdown_connections 2024-04-17T10:29:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e8a600640 1 --2- 172.21.15.67:0/2156569834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841653b0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.453+0000 7f4e8a600640 1 -- 172.21.15.67:0/2156569834 >> 172.21.15.67:0/2156569834 conn(0x7f4e840b1740 msgr2=0x7f4e840b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 -- 172.21.15.67:0/2156569834 shutdown_connections 2024-04-17T10:29:33.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 -- 172.21.15.67:0/2156569834 wait complete. 2024-04-17T10:29:33.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 Processor -- start 2024-04-17T10:29:33.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 -- start start 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841f6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e840aef60 0x7f4e841f9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e840cd030 0x7f4e84153490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e841686a0 con 0x7f4e8406e980 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84168520 con 0x7f4e840aef60 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e84168820 con 0x7f4e840cd030 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841f6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841f6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32840/0 (socket says 172.21.15.67:32840) 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 -- 172.21.15.67:0/4174709897 learned_addr learned my addr 172.21.15.67:0/4174709897 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83e00640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e840cd030 0x7f4e84153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 -- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e840cd030 msgr2=0x7f4e84153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e840cd030 0x7f4e84153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e7be00640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e840aef60 0x7f4e841f9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 -- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e840aef60 msgr2=0x7f4e841f9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e840aef60 0x7f4e841f9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 -- 172.21.15.67:0/4174709897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8414adc0 con 0x7f4e8406e980 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83e00640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e840cd030 0x7f4e84153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e7be00640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e840aef60 0x7f4e841f9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e83400640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841f6e10 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c730 tx=0x7f4e7400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e7b400640 1 -- 172.21.15.67:0/4174709897 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74017070 con 0x7f4e8406e980 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e7b400640 1 -- 172.21.15.67:0/4174709897 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e74005ac0 con 0x7f4e8406e980 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e7b400640 1 -- 172.21.15.67:0/4174709897 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74007070 con 0x7f4e8406e980 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.457+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8414b0a0 con 0x7f4e8406e980 2024-04-17T10:29:33.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.461+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8414b660 con 0x7f4e8406e980 2024-04-17T10:29:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.461+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8404ef30 con 0x7f4e8406e980 2024-04-17T10:29:33.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.461+0000 7f4e7b400640 1 -- 172.21.15.67:0/4174709897 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4e7401a070 con 0x7f4e8406e980 2024-04-17T10:29:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.461+0000 7f4e7b400640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e60066f60 0x7f4e60069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.461+0000 7f4e7b400640 1 -- 172.21.15.67:0/4174709897 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4e74084b20 con 0x7f4e8406e980 2024-04-17T10:29:33.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.461+0000 7f4e7be00640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e60066f60 0x7f4e60069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.465+0000 7f4e7be00640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e60066f60 0x7f4e60069420 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4e84153f40 tx=0x7f4e6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.465+0000 7f4e7b400640 1 -- 172.21.15.67:0/4174709897 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4e74021020 con 0x7f4e8406e980 2024-04-17T10:29:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.625+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4e8414edf0 con 0x7f4e8406e980 2024-04-17T10:29:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.625+0000 7f4e7b400640 1 -- 172.21.15.67:0/4174709897 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4e840b4990 con 0x7f4e8406e980 2024-04-17T10:29:33.620 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:29:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e60066f60 msgr2=0x7f4e60069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e60066f60 0x7f4e60069420 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f4e84153f40 tx=0x7f4e6c009040 comp rx=0 tx=0).stop 2024-04-17T10:29:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 msgr2=0x7f4e841f6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841f6e10 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c730 tx=0x7f4e7400cbf0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 shutdown_connections 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e60066f60 0x7f4e60069420 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e840cd030 0x7f4e84153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e840aef60 0x7f4e841f9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 --2- 172.21.15.67:0/4174709897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8406e980 0x7f4e841f6e10 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 >> 172.21.15.67:0/4174709897 conn(0x7f4e840b1740 msgr2=0x7f4e84153f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 shutdown_connections 2024-04-17T10:29:33.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.629+0000 7f4e8a600640 1 -- 172.21.15.67:0/4174709897 wait complete. 2024-04-17T10:29:33.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:29:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4209400640 1 Processor -- start 2024-04-17T10:29:33.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4209400640 1 -- start start 2024-04-17T10:29:33.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4209400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f420415dcf0 0x7f42040cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4209400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42040ce7a0 0x7f42040b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4209400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42040d0040 con 0x7f420406cac0 2024-04-17T10:29:33.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4209400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42040cfde0 con 0x7f420415dcf0 2024-04-17T10:29:33.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4209400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42040d0320 con 0x7f42040ce7a0 2024-04-17T10:29:33.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4202800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f420415dcf0 0x7f42040cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4202800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f420415dcf0 0x7f42040cf8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32856/0 (socket says 172.21.15.67:32856) 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4202800640 1 -- 172.21.15.67:0/1925806761 learned_addr learned my addr 172.21.15.67:0/1925806761 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4201e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f420406cac0 0x7f42040ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35404/0 (socket says 172.21.15.67:35404) 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4203200640 1 --2- 172.21.15.67:0/1925806761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42040ce7a0 0x7f42040b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4202800640 1 -- 172.21.15.67:0/1925806761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42040ce7a0 msgr2=0x7f42040b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4202800640 1 --2- 172.21.15.67:0/1925806761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42040ce7a0 0x7f42040b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4202800640 1 -- 172.21.15.67:0/1925806761 >> v1:172.21.15.67:6789/0 conn(0x7f420406cac0 legacy=0x7f42040ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4202800640 1 -- 172.21.15.67:0/1925806761 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42040d0600 con 0x7f420415dcf0 2024-04-17T10:29:33.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.793+0000 7f4201400640 1 -- 172.21.15.67:0/1925806761 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2023524078 0 0) 0x7f42040d0040 con 0x7f420406cac0 2024-04-17T10:29:33.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4202800640 1 --2- 172.21.15.67:0/1925806761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f420415dcf0 0x7f42040cf8a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009870 tx=0x7f41f401c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51131be3b6ccd405 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4201400640 1 -- 172.21.15.67:0/1925806761 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f40a95f0 con 0x7f420415dcf0 2024-04-17T10:29:33.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4201400640 1 -- 172.21.15.67:0/1925806761 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f40a9790 con 0x7f420415dcf0 2024-04-17T10:29:33.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4201400640 1 -- 172.21.15.67:0/1925806761 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f40a9b60 con 0x7f420415dcf0 2024-04-17T10:29:33.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 -- 172.21.15.67:0/1925806761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f420415dcf0 msgr2=0x7f42040cf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 --2- 172.21.15.67:0/1925806761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f420415dcf0 0x7f42040cf8a0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f41f4009870 tx=0x7f41f401c530 comp rx=0 tx=0).stop 2024-04-17T10:29:33.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 -- 172.21.15.67:0/1925806761 shutdown_connections 2024-04-17T10:29:33.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 --2- 172.21.15.67:0/1925806761 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42040ce7a0 0x7f42040b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 --2- 172.21.15.67:0/1925806761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f420415dcf0 0x7f42040cf8a0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 -- 172.21.15.67:0/1925806761 >> 172.21.15.67:0/1925806761 conn(0x7f42040b33a0 msgr2=0x7f42040b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:33.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 -- 172.21.15.67:0/1925806761 shutdown_connections 2024-04-17T10:29:33.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 -- 172.21.15.67:0/1925806761 wait complete. 2024-04-17T10:29:33.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 Processor -- start 2024-04-17T10:29:33.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.797+0000 7f4209400640 1 -- start start 2024-04-17T10:29:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f420406cac0 0x7f420414cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040ce7a0 0x7f420415e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f420415dcf0 0x7f42040b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42040b2350 con 0x7f42040ce7a0 2024-04-17T10:29:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42040b21d0 con 0x7f420406cac0 2024-04-17T10:29:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42040b24d0 con 0x7f420415dcf0 2024-04-17T10:29:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040ce7a0 0x7f420415e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040ce7a0 0x7f420415e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32866/0 (socket says 172.21.15.67:32866) 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 -- 172.21.15.67:0/1855345099 learned_addr learned my addr 172.21.15.67:0/1855345099 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4202800640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f420406cac0 0x7f420414cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4203200640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f420415dcf0 0x7f42040b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 -- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f420415dcf0 msgr2=0x7f42040b6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f420415dcf0 0x7f42040b6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 -- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f420406cac0 msgr2=0x7f420414cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f420406cac0 0x7f420414cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 -- 172.21.15.67:0/1855345099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42040b5b70 con 0x7f42040ce7a0 2024-04-17T10:29:33.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4201e00640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040ce7a0 0x7f420415e5b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f41f00011f0 tx=0x7f41f000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f41faa00640 1 -- 172.21.15.67:0/1855345099 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0018070 con 0x7f42040ce7a0 2024-04-17T10:29:33.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f41faa00640 1 -- 172.21.15.67:0/1855345099 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f00074e0 con 0x7f42040ce7a0 2024-04-17T10:29:33.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f41faa00640 1 -- 172.21.15.67:0/1855345099 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f0004a10 con 0x7f42040ce7a0 2024-04-17T10:29:33.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42040b5e50 con 0x7f42040ce7a0 2024-04-17T10:29:33.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42040b6410 con 0x7f42040ce7a0 2024-04-17T10:29:33.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.801+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420404ef30 con 0x7f42040ce7a0 2024-04-17T10:29:33.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.805+0000 7f41faa00640 1 -- 172.21.15.67:0/1855345099 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f41f0004bb0 con 0x7f42040ce7a0 2024-04-17T10:29:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.805+0000 7f41faa00640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41e4066f60 0x7f41e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.805+0000 7f41faa00640 1 -- 172.21.15.67:0/1855345099 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f41f0084540 con 0x7f42040ce7a0 2024-04-17T10:29:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.805+0000 7f4202800640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41e4066f60 0x7f41e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:33.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.805+0000 7f4202800640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41e4066f60 0x7f41e4069420 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f41f40ae000 tx=0x7f41f40b4040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:33.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.809+0000 7f41faa00640 1 -- 172.21.15.67:0/1855345099 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f41f0054d20 con 0x7f42040ce7a0 2024-04-17T10:29:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.969+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f420406f960 con 0x7f42040ce7a0 2024-04-17T10:29:33.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f41faa00640 1 -- 172.21.15.67:0/1855345099 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+11 (secure 0 0 0) 0x7f41f0058dc0 con 0x7f42040ce7a0 2024-04-17T10:29:33.965 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41e4066f60 msgr2=0x7f41e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41e4066f60 0x7f41e4069420 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f41f40ae000 tx=0x7f41f40b4040 comp rx=0 tx=0).stop 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040ce7a0 msgr2=0x7f420415e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040ce7a0 0x7f420415e5b0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f41f00011f0 tx=0x7f41f000c9b0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 shutdown_connections 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41e4066f60 0x7f41e4069420 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f420415dcf0 0x7f42040b6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42040ce7a0 0x7f420415e5b0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 --2- 172.21.15.67:0/1855345099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f420406cac0 0x7f420414cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 >> 172.21.15.67:0/1855345099 conn(0x7f42040b33a0 msgr2=0x7f4204073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 shutdown_connections 2024-04-17T10:29:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:33.973+0000 7f4209400640 1 -- 172.21.15.67:0/1855345099 wait complete. 2024-04-17T10:29:33.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:29:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.137+0000 7fb672000640 1 Processor -- start 2024-04-17T10:29:34.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.137+0000 7fb672000640 1 -- start start 2024-04-17T10:29:34.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.137+0000 7fb672000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.137+0000 7fb672000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb66c0728c0 0x7fb66c0b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.137+0000 7fb672000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c15f360 con 0x7fb66c0b5130 2024-04-17T10:29:34.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.137+0000 7fb672000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb66c06bca0 con 0x7fb66c0728c0 2024-04-17T10:29:34.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.137+0000 7fb672000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c0b3160 con 0x7fb66c1583f0 2024-04-17T10:29:34.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb662a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb66c0728c0 0x7fb66c0b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:45236/0 (socket says 172.21.15.67:45236) 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb662a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb66c0728c0 0x7fb66c0b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:46924/0 (socket says 172.21.15.67:46924) 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 -- 172.21.15.67:0/1112425481 learned_addr learned my addr 172.21.15.67:0/1112425481 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66be00640 1 --1- 172.21.15.67:0/1112425481 >> v1:172.21.15.86:6789/0 conn(0x7fb66c0b5130 0x7fb66c161980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:48918/0 (socket says 172.21.15.67:48918) 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 -- 172.21.15.67:0/1112425481 >> v2:172.21.15.86:3300/0 conn(0x7fb66c0728c0 msgr2=0x7fb66c0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 --2- 172.21.15.67:0/1112425481 >> v2:172.21.15.86:3300/0 conn(0x7fb66c0728c0 0x7fb66c0b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 -- 172.21.15.67:0/1112425481 >> v1:172.21.15.86:6789/0 conn(0x7fb66c0b5130 legacy=0x7fb66c161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 -- 172.21.15.67:0/1112425481 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0b3440 con 0x7fb66c1583f0 2024-04-17T10:29:34.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb662a00640 1 --2- 172.21.15.67:0/1112425481 >> v2:172.21.15.86:3300/0 conn(0x7fb66c0728c0 0x7fb66c0b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:34.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66b400640 1 --2- 172.21.15.67:0/1112425481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c072380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009e50 tx=0x7fb65c01ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298c0d1e2d32d296 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66aa00640 1 -- 172.21.15.67:0/1112425481 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0a5700 con 0x7fb66c1583f0 2024-04-17T10:29:34.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66aa00640 1 -- 172.21.15.67:0/1112425481 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c0a58a0 con 0x7fb66c1583f0 2024-04-17T10:29:34.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.141+0000 7fb66aa00640 1 -- 172.21.15.67:0/1112425481 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0a5c70 con 0x7fb66c1583f0 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- 172.21.15.67:0/1112425481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 msgr2=0x7fb66c072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 --2- 172.21.15.67:0/1112425481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c072380 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb65c009e50 tx=0x7fb65c01ca30 comp rx=0 tx=0).stop 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- 172.21.15.67:0/1112425481 shutdown_connections 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 --2- 172.21.15.67:0/1112425481 >> v2:172.21.15.86:3300/0 conn(0x7fb66c0728c0 0x7fb66c0b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 --2- 172.21.15.67:0/1112425481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c072380 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- 172.21.15.67:0/1112425481 >> 172.21.15.67:0/1112425481 conn(0x7fb66c0b2570 msgr2=0x7fb66c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- 172.21.15.67:0/1112425481 shutdown_connections 2024-04-17T10:29:34.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- 172.21.15.67:0/1112425481 wait complete. 2024-04-17T10:29:34.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 Processor -- start 2024-04-17T10:29:34.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- start start 2024-04-17T10:29:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0728c0 0x7fb66c157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb66c0b5130 0x7fb66c15cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c15d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c0b4200 con 0x7fb66c0728c0 2024-04-17T10:29:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1587d0 con 0x7fb66c0b5130 2024-04-17T10:29:34.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb672000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c0b5540 con 0x7fb66c1583f0 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb66b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0728c0 0x7fb66c157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb66b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0728c0 0x7fb66c157b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32870/0 (socket says 172.21.15.67:32870) 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb66b400640 1 -- 172.21.15.67:0/3001019454 learned_addr learned my addr 172.21.15.67:0/3001019454 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb66be00640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c15d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.145+0000 7fb662a00640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb66c0b5130 0x7fb66c15cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb66b400640 1 -- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 msgr2=0x7fb66c15d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb66b400640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c15d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb66b400640 1 -- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb66c0b5130 msgr2=0x7fb66c15cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb66b400640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb66c0b5130 0x7fb66c15cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb66b400640 1 -- 172.21.15.67:0/3001019454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c150b10 con 0x7fb66c0728c0 2024-04-17T10:29:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb662a00640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb66c0b5130 0x7fb66c15cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb66b400640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0728c0 0x7fb66c157b30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0af040 tx=0x7fb65c0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb663e00640 1 -- 172.21.15.67:0/3001019454 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0bc070 con 0x7fb66c0728c0 2024-04-17T10:29:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb663e00640 1 -- 172.21.15.67:0/3001019454 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c0b8660 con 0x7fb66c0728c0 2024-04-17T10:29:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb663e00640 1 -- 172.21.15.67:0/3001019454 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0b8a50 con 0x7fb66c0728c0 2024-04-17T10:29:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c150df0 con 0x7fb66c0728c0 2024-04-17T10:29:34.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c1513b0 con 0x7fb66c0728c0 2024-04-17T10:29:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66c04ef30 con 0x7fb66c0728c0 2024-04-17T10:29:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb663e00640 1 -- 172.21.15.67:0/3001019454 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb65c0b8bf0 con 0x7fb66c0728c0 2024-04-17T10:29:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb663e00640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb648066f60 0x7fb648069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.149+0000 7fb663e00640 1 -- 172.21.15.67:0/3001019454 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb65c120150 con 0x7fb66c0728c0 2024-04-17T10:29:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.153+0000 7fb662a00640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb648066f60 0x7fb648069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.153+0000 7fb662a00640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb648066f60 0x7fb648069420 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb66c152170 tx=0x7fb654008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.153+0000 7fb663e00640 1 -- 172.21.15.67:0/3001019454 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb65c016620 con 0x7fb66c0728c0 2024-04-17T10:29:34.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.317+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb66c0aed10 con 0x7fb66c0728c0 2024-04-17T10:29:34.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb663e00640 1 -- 172.21.15.67:0/3001019454 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb65c0f0a40 con 0x7fb66c0728c0 2024-04-17T10:29:34.313 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:29:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb648066f60 msgr2=0x7fb648069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb648066f60 0x7fb648069420 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb66c152170 tx=0x7fb654008040 comp rx=0 tx=0).stop 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0728c0 msgr2=0x7fb66c157b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0728c0 0x7fb66c157b30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0af040 tx=0x7fb65c0197d0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 shutdown_connections 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb648066f60 0x7fb648069420 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb66c1583f0 0x7fb66c15d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb66c0b5130 0x7fb66c15cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 --2- 172.21.15.67:0/3001019454 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb66c0728c0 0x7fb66c157b30 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 >> 172.21.15.67:0/3001019454 conn(0x7fb66c0b2570 msgr2=0x7fb66c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 shutdown_connections 2024-04-17T10:29:34.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.321+0000 7fb672000640 1 -- 172.21.15.67:0/3001019454 wait complete. 2024-04-17T10:29:34.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.481+0000 7f96d6400640 1 Processor -- start 2024-04-17T10:29:34.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.481+0000 7f96d6400640 1 -- start start 2024-04-17T10:29:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96d6400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f96d006e6e0 0x7f96d00ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96d6400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d015ae40 con 0x7f96d01441f0 2024-04-17T10:29:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96d6400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96d0071440 con 0x7f96d006eac0 2024-04-17T10:29:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96d6400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d0071720 con 0x7f96d006e6e0 2024-04-17T10:29:34.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f96d01441f0 0x7f96d00b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35406/0 (socket says 172.21.15.67:35406) 2024-04-17T10:29:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cfe00640 1 -- 172.21.15.67:0/1082678595 learned_addr learned my addr 172.21.15.67:0/1082678595 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cf400640 1 --2- 172.21.15.67:0/1082678595 >> v2:172.21.15.86:3300/0 conn(0x7f96d006e6e0 0x7f96d00ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cea00640 1 -- 172.21.15.67:0/1082678595 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237677856 0 0) 0x7f96d015ae40 con 0x7f96d01441f0 2024-04-17T10:29:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cea00640 1 -- 172.21.15.67:0/1082678595 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b4003530 con 0x7f96d01441f0 2024-04-17T10:29:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cf400640 1 -- 172.21.15.67:0/1082678595 >> v1:172.21.15.86:6789/0 conn(0x7f96d006eac0 legacy=0x7f96d0143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cf400640 1 -- 172.21.15.67:0/1082678595 >> v1:172.21.15.67:6789/0 conn(0x7f96d01441f0 legacy=0x7f96d00b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.485+0000 7f96cf400640 1 -- 172.21.15.67:0/1082678595 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d0071a00 con 0x7f96d006e6e0 2024-04-17T10:29:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96cf400640 1 --2- 172.21.15.67:0/1082678595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d006e6e0 0x7f96d00ce9b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f96c00097c0 tx=0x7f96c001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddd91b43b9b921b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96cea00640 1 -- 172.21.15.67:0/1082678595 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c00b0070 con 0x7f96d006e6e0 2024-04-17T10:29:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96cea00640 1 -- 172.21.15.67:0/1082678595 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c00ab600 con 0x7f96d006e6e0 2024-04-17T10:29:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96cea00640 1 -- 172.21.15.67:0/1082678595 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c00ab9d0 con 0x7f96d006e6e0 2024-04-17T10:29:34.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 -- 172.21.15.67:0/1082678595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d006e6e0 msgr2=0x7f96d00ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 --2- 172.21.15.67:0/1082678595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d006e6e0 0x7f96d00ce9b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f96c00097c0 tx=0x7f96c001c3e0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 -- 172.21.15.67:0/1082678595 shutdown_connections 2024-04-17T10:29:34.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 --2- 172.21.15.67:0/1082678595 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d006e6e0 0x7f96d00ce9b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 -- 172.21.15.67:0/1082678595 >> 172.21.15.67:0/1082678595 conn(0x7f96d006ac90 msgr2=0x7f96d006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:34.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 -- 172.21.15.67:0/1082678595 shutdown_connections 2024-04-17T10:29:34.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 -- 172.21.15.67:0/1082678595 wait complete. 2024-04-17T10:29:34.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 Processor -- start 2024-04-17T10:29:34.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.489+0000 7f96d6400640 1 -- start start 2024-04-17T10:29:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96d006e6e0 0x7f96d012ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d006eac0 0x7f96d00afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d01441f0 0x7f96d00aa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d01688b0 con 0x7f96d006eac0 2024-04-17T10:29:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d0168730 con 0x7f96d01441f0 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d0168a30 con 0x7f96d006e6e0 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d006eac0 0x7f96d00afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96cf400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96d006e6e0 0x7f96d012ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d006eac0 0x7f96d00afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32886/0 (socket says 172.21.15.67:32886) 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 -- 172.21.15.67:0/112931165 learned_addr learned my addr 172.21.15.67:0/112931165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96cf400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96d006e6e0 0x7f96d012ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:45260/0 (socket says 172.21.15.67:45260) 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96cfe00640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d01441f0 0x7f96d00aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 -- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96d006e6e0 msgr2=0x7f96d012ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96d006e6e0 0x7f96d012ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 -- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d01441f0 msgr2=0x7f96d00aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d01441f0 0x7f96d00aa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 -- 172.21.15.67:0/112931165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d00b0800 con 0x7f96d006eac0 2024-04-17T10:29:34.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c6a00640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d006eac0 0x7f96d00afdb0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f96b800c7c0 tx=0x7f96b800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c7e00640 1 -- 172.21.15.67:0/112931165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8016020 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c7e00640 1 -- 172.21.15.67:0/112931165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b8006d60 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d00b0ae0 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96c7e00640 1 -- 172.21.15.67:0/112931165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96b8003bc0 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d00b10a0 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.493+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d0072d50 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.497+0000 7f96c7e00640 1 -- 172.21.15.67:0/112931165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f96b800a850 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.497+0000 7f96c7e00640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96ac066f60 0x7f96ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.497+0000 7f96c7e00640 1 -- 172.21.15.67:0/112931165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f96b8084db0 con 0x7f96d006eac0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.497+0000 7f96cf400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96ac066f60 0x7f96ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.497+0000 7f96cf400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96ac066f60 0x7f96ac069420 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f96c00ad000 tx=0x7f96c00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.501+0000 7f96c7e00640 1 -- 172.21.15.67:0/112931165 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f96b8020020 con 0x7f96d006eac0 2024-04-17T10:29:34.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96d00b2c60 con 0x7f96d006eac0 2024-04-17T10:29:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96c7e00640 1 -- 172.21.15.67:0/112931165 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f96b8055620 con 0x7f96d006eac0 2024-04-17T10:29:34.690 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:34.690 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3163523544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3163523544},{"type":"v1","addr":"172.21.15.86:6835","nonce":3163523544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3682484259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3682484259},{"type":"v1","addr":"172.21.15.86:6837","nonce":3682484259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6945,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/252088612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":252088612},{"type":"v1","addr":"172.21.15.67:6834","nonce":252088612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:17.323326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/2976334498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2976334498},{"type":"v1","addr":"172.21.15.67:6835","nonce":2976334498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:34.690 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 47 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96ac066f60 msgr2=0x7f96ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96ac066f60 0x7f96ac069420 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f96c00ad000 tx=0x7f96c00a6000 comp rx=0 tx=0).stop 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d006eac0 msgr2=0x7f96d00afdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d006eac0 0x7f96d00afdb0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f96b800c7c0 tx=0x7f96b800cc80 comp rx=0 tx=0).stop 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 shutdown_connections 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96ac066f60 0x7f96ac069420 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f96d01441f0 0x7f96d00aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96d006eac0 0x7f96d00afdb0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 --2- 172.21.15.67:0/112931165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96d006e6e0 0x7f96d012ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 >> 172.21.15.67:0/112931165 conn(0x7f96d006ac90 msgr2=0x7f96d00a9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 shutdown_connections 2024-04-17T10:29:34.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.697+0000 7f96d6400640 1 -- 172.21.15.67:0/112931165 wait complete. 2024-04-17T10:29:34.712 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.857+0000 7fc9d0200640 1 Processor -- start 2024-04-17T10:29:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.857+0000 7fc9d0200640 1 -- start start 2024-04-17T10:29:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9d0200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc9c80cdcf0 0x7fc9c80cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9d0200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c806b100 con 0x7fc9c80cc250 2024-04-17T10:29:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9d0200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c806b3e0 con 0x7fc9c80ce2a0 2024-04-17T10:29:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9d0200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c80cd150 con 0x7fc9c80cdcf0 2024-04-17T10:29:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9cd800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc9c80cc250 0x7fc9c80cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35414/0 (socket says 172.21.15.67:35414) 2024-04-17T10:29:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9cd800640 1 -- 172.21.15.67:0/1369206942 learned_addr learned my addr 172.21.15.67:0/1369206942 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:34.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9ce200640 1 --2- 172.21.15.67:0/1369206942 >> v2:172.21.15.86:3300/0 conn(0x7fc9c80cdcf0 0x7fc9c80cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9c7e00640 1 -- 172.21.15.67:0/1369206942 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288203131 0 0) 0x7fc9c806b100 con 0x7fc9c80cc250 2024-04-17T10:29:34.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9c7e00640 1 -- 172.21.15.67:0/1369206942 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b0003530 con 0x7fc9c80cc250 2024-04-17T10:29:34.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9ce200640 1 -- 172.21.15.67:0/1369206942 >> v1:172.21.15.86:6789/0 conn(0x7fc9c80ce2a0 legacy=0x7fc9c80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9ce200640 1 -- 172.21.15.67:0/1369206942 >> v1:172.21.15.67:6789/0 conn(0x7fc9c80cc250 legacy=0x7fc9c80cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.861+0000 7fc9ce200640 1 -- 172.21.15.67:0/1369206942 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c806b6c0 con 0x7fc9c80cdcf0 2024-04-17T10:29:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9ce200640 1 --2- 172.21.15.67:0/1369206942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80cf9d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0009d90 tx=0x7fc9c001c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de47a62ceac28480 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9c7e00640 1 -- 172.21.15.67:0/1369206942 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00b2070 con 0x7fc9c80cdcf0 2024-04-17T10:29:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9c7e00640 1 -- 172.21.15.67:0/1369206942 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c00a5940 con 0x7fc9c80cdcf0 2024-04-17T10:29:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9c7e00640 1 -- 172.21.15.67:0/1369206942 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c00a5d10 con 0x7fc9c80cdcf0 2024-04-17T10:29:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 -- 172.21.15.67:0/1369206942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 msgr2=0x7fc9c80cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 --2- 172.21.15.67:0/1369206942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80cf9d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0009d90 tx=0x7fc9c001c970 comp rx=0 tx=0).stop 2024-04-17T10:29:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 -- 172.21.15.67:0/1369206942 shutdown_connections 2024-04-17T10:29:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 --2- 172.21.15.67:0/1369206942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80cf9d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 -- 172.21.15.67:0/1369206942 >> 172.21.15.67:0/1369206942 conn(0x7fc9c80b22a0 msgr2=0x7fc9c80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 -- 172.21.15.67:0/1369206942 shutdown_connections 2024-04-17T10:29:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 -- 172.21.15.67:0/1369206942 wait complete. 2024-04-17T10:29:34.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 Processor -- start 2024-04-17T10:29:34.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.865+0000 7fc9d0200640 1 -- start start 2024-04-17T10:29:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c80cc250 0x7fc9c815dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80c5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9c80ce2a0 0x7fc9c80c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c816c290 con 0x7fc9c80cc250 2024-04-17T10:29:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c816c110 con 0x7fc9c80cdcf0 2024-04-17T10:29:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c816c410 con 0x7fc9c80ce2a0 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c80cc250 0x7fc9c815dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c80cc250 0x7fc9c815dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32902/0 (socket says 172.21.15.67:32902) 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 -- 172.21.15.67:0/108180629 learned_addr learned my addr 172.21.15.67:0/108180629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cce00640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80c5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9ce200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9c80ce2a0 0x7fc9c80c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 -- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9c80ce2a0 msgr2=0x7fc9c80c6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9c80ce2a0 0x7fc9c80c6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 -- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 msgr2=0x7fc9c80c5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80c5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 -- 172.21.15.67:0/108180629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c80c6750 con 0x7fc9c80cc250 2024-04-17T10:29:34.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9ce200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9c80ce2a0 0x7fc9c80c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:34.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cce00640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:34.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9cd800640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c80cc250 0x7fc9c815dd70 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00c500 tx=0x7fc9bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9c5600640 1 -- 172.21.15.67:0/108180629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc017070 con 0x7fc9c80cc250 2024-04-17T10:29:34.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9c5600640 1 -- 172.21.15.67:0/108180629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc005cf0 con 0x7fc9c80cc250 2024-04-17T10:29:34.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9c5600640 1 -- 172.21.15.67:0/108180629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0072f0 con 0x7fc9c80cc250 2024-04-17T10:29:34.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c80c6a20 con 0x7fc9c80cc250 2024-04-17T10:29:34.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c814df50 con 0x7fc9c80cc250 2024-04-17T10:29:34.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c806cd30 con 0x7fc9c80cc250 2024-04-17T10:29:34.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.869+0000 7fc9c5600640 1 -- 172.21.15.67:0/108180629 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc9bc01a070 con 0x7fc9c80cc250 2024-04-17T10:29:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.873+0000 7fc9c5600640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc9a8066f60 0x7fc9a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.873+0000 7fc9c5600640 1 -- 172.21.15.67:0/108180629 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fc9bc084580 con 0x7fc9c80cc250 2024-04-17T10:29:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.873+0000 7fc9cce00640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc9a8066f60 0x7fc9a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.873+0000 7fc9cce00640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc9a8066f60 0x7fc9a8069420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80b66c0 tx=0x7fc9b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:34.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:34.877+0000 7fc9c5600640 1 -- 172.21.15.67:0/108180629 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc9bc054d50 con 0x7fc9c80cc250 2024-04-17T10:29:35.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.037+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9c809e900 con 0x7fc9c80cc250 2024-04-17T10:29:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9c5600640 1 -- 172.21.15.67:0/108180629 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16963 (secure 0 0 0) 0x7fc9bc058df0 con 0x7fc9c80cc250 2024-04-17T10:29:35.034 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:35.034 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":71,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:14.294073+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/392417778":"2024-04-18T10:29:14.294053+0000","172.21.15.67:6833/392417778":"2024-04-18T10:29:14.294053+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc9a8066f60 msgr2=0x7fc9a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc9a8066f60 0x7fc9a8069420 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fc9c80b66c0 tx=0x7fc9b4009040 comp rx=0 tx=0).stop 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c80cc250 msgr2=0x7fc9c815dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c80cc250 0x7fc9c815dd70 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc00c500 tx=0x7fc9bc00c9c0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 shutdown_connections 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc9a8066f60 0x7fc9a8069420 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9c80ce2a0 0x7fc9c80c6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9c80cdcf0 0x7fc9c80c5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 --2- 172.21.15.67:0/108180629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9c80cc250 0x7fc9c815dd70 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 >> 172.21.15.67:0/108180629 conn(0x7fc9c80b22a0 msgr2=0x7fc9c80ad760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 shutdown_connections 2024-04-17T10:29:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.041+0000 7fc9d0200640 1 -- 172.21.15.67:0/108180629 wait complete. 2024-04-17T10:29:35.055 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:29:35.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:35.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.201+0000 7f0844000640 1 Processor -- start 2024-04-17T10:29:35.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.201+0000 7f0844000640 1 -- start start 2024-04-17T10:29:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0844000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c06e6e0 0x7f083c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0844000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c15ae40 con 0x7f083c1441f0 2024-04-17T10:29:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0844000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083c071440 con 0x7f083c06eac0 2024-04-17T10:29:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0844000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c071720 con 0x7f083c06e6e0 2024-04-17T10:29:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0842000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f083c1441f0 0x7f083c0b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35424/0 (socket says 172.21.15.67:35424) 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0842000640 1 -- 172.21.15.67:0/183430168 learned_addr learned my addr 172.21.15.67:0/183430168 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0841600640 1 --2- 172.21.15.67:0/183430168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c06e6e0 0x7f083c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f083be00640 1 -- 172.21.15.67:0/183430168 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637991784 0 0) 0x7f083c071440 con 0x7f083c06eac0 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f083be00640 1 -- 172.21.15.67:0/183430168 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0824005740 con 0x7f083c06eac0 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f083be00640 1 -- 172.21.15.67:0/183430168 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308173691 0 0) 0x7f083c15ae40 con 0x7f083c1441f0 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0841600640 1 -- 172.21.15.67:0/183430168 >> v1:172.21.15.86:6789/0 conn(0x7f083c06eac0 legacy=0x7f083c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0841600640 1 -- 172.21.15.67:0/183430168 >> v1:172.21.15.67:6789/0 conn(0x7f083c1441f0 legacy=0x7f083c0b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.205+0000 7f0841600640 1 -- 172.21.15.67:0/183430168 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c071a00 con 0x7f083c06e6e0 2024-04-17T10:29:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0841600640 1 --2- 172.21.15.67:0/183430168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c06e6e0 0x7f083c0ce9b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08300097f0 tx=0x7f083001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebb6466268e9c07 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f083be00640 1 -- 172.21.15.67:0/183430168 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08300b0070 con 0x7f083c06e6e0 2024-04-17T10:29:35.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f083be00640 1 -- 172.21.15.67:0/183430168 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08300ab600 con 0x7f083c06e6e0 2024-04-17T10:29:35.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f083be00640 1 -- 172.21.15.67:0/183430168 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08300ab9d0 con 0x7f083c06e6e0 2024-04-17T10:29:35.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 -- 172.21.15.67:0/183430168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c06e6e0 msgr2=0x7f083c0ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 --2- 172.21.15.67:0/183430168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c06e6e0 0x7f083c0ce9b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f08300097f0 tx=0x7f083001c420 comp rx=0 tx=0).stop 2024-04-17T10:29:35.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 -- 172.21.15.67:0/183430168 shutdown_connections 2024-04-17T10:29:35.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 --2- 172.21.15.67:0/183430168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c06e6e0 0x7f083c0ce9b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 -- 172.21.15.67:0/183430168 >> 172.21.15.67:0/183430168 conn(0x7f083c06ac90 msgr2=0x7f083c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:35.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 -- 172.21.15.67:0/183430168 shutdown_connections 2024-04-17T10:29:35.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 -- 172.21.15.67:0/183430168 wait complete. 2024-04-17T10:29:35.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 Processor -- start 2024-04-17T10:29:35.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 -- start start 2024-04-17T10:29:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f083c06e6e0 0x7f083c0c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.209+0000 7f0844000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c06eac0 0x7f083c1fa740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0844000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1441f0 0x7f083c142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0844000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c06d4f0 con 0x7f083c06e6e0 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0844000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c06d370 con 0x7f083c06eac0 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0844000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083c06d670 con 0x7f083c1441f0 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f083c06e6e0 0x7f083c0c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f083c06e6e0 0x7f083c0c7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32910/0 (socket says 172.21.15.67:32910) 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 -- 172.21.15.67:0/1365533003 learned_addr learned my addr 172.21.15.67:0/1365533003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0840c00640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c06eac0 0x7f083c1fa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0842000640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1441f0 0x7f083c142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 -- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1441f0 msgr2=0x7f083c142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1441f0 0x7f083c142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 -- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c06eac0 msgr2=0x7f083c1fa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c06eac0 0x7f083c1fa740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 -- 172.21.15.67:0/1365533003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c0c4d30 con 0x7f083c06e6e0 2024-04-17T10:29:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0841600640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f083c06e6e0 0x7f083c0c7620 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f08300ae040 tx=0x7f083001ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0839600640 1 -- 172.21.15.67:0/1365533003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08300b0050 con 0x7f083c06e6e0 2024-04-17T10:29:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0839600640 1 -- 172.21.15.67:0/1365533003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08300b1040 con 0x7f083c06e6e0 2024-04-17T10:29:35.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0839600640 1 -- 172.21.15.67:0/1365533003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830006e20 con 0x7f083c06e6e0 2024-04-17T10:29:35.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c0c5010 con 0x7f083c06e6e0 2024-04-17T10:29:35.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c0c58b0 con 0x7f083c06e6e0 2024-04-17T10:29:35.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c0c51e0 con 0x7f083c06e6e0 2024-04-17T10:29:35.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.213+0000 7f0839600640 1 -- 172.21.15.67:0/1365533003 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f08300be030 con 0x7f083c06e6e0 2024-04-17T10:29:35.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.217+0000 7f0839600640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f081c066f60 0x7f081c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.217+0000 7f0839600640 1 -- 172.21.15.67:0/1365533003 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f083011edf0 con 0x7f083c06e6e0 2024-04-17T10:29:35.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.217+0000 7f0840c00640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f081c066f60 0x7f081c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.217+0000 7f0840c00640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f081c066f60 0x7f081c069420 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f083c073920 tx=0x7f0828009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.221+0000 7f0839600640 1 -- 172.21.15.67:0/1365533003 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f08300c5060 con 0x7f083c06e6e0 2024-04-17T10:29:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f083c074e30 con 0x7f083c06e6e0 2024-04-17T10:29:35.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0839600640 1 -- 172.21.15.67:0/1365533003 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f08300160d0 con 0x7f083c06e6e0 2024-04-17T10:29:35.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:35.411 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3163523544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3163523544},{"type":"v1","addr":"172.21.15.86:6835","nonce":3163523544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3682484259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3682484259},{"type":"v1","addr":"172.21.15.86:6837","nonce":3682484259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6945,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/252088612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":252088612},{"type":"v1","addr":"172.21.15.67:6834","nonce":252088612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:17.323326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/2976334498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2976334498},{"type":"v1","addr":"172.21.15.67:6835","nonce":2976334498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:35.411 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 47 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f081c066f60 msgr2=0x7f081c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f081c066f60 0x7f081c069420 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f083c073920 tx=0x7f0828009040 comp rx=0 tx=0).stop 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f083c06e6e0 msgr2=0x7f083c0c7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f083c06e6e0 0x7f083c0c7620 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f08300ae040 tx=0x7f083001ac60 comp rx=0 tx=0).stop 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 shutdown_connections 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f081c066f60 0x7f081c069420 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f083c1441f0 0x7f083c142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f083c06eac0 0x7f083c1fa740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 --2- 172.21.15.67:0/1365533003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f083c06e6e0 0x7f083c0c7620 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 >> 172.21.15.67:0/1365533003 conn(0x7f083c06ac90 msgr2=0x7f083c159d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 shutdown_connections 2024-04-17T10:29:35.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.417+0000 7f0844000640 1 -- 172.21.15.67:0/1365533003 wait complete. 2024-04-17T10:29:35.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b61e00640 1 Processor -- start 2024-04-17T10:29:35.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b61e00640 1 -- start start 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b61e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1b5c0b07b0 0x7f1b5c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b61e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5c0b5980 con 0x7f1b5c15dd10 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b61e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5c0b0cc0 con 0x7f1b5c071300 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b61e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b5c071770 con 0x7f1b5c0b07b0 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b5c15dd10 0x7f1b5c0b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35430/0 (socket says 172.21.15.67:35430) 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5aa00640 1 -- 172.21.15.67:0/2129617567 learned_addr learned my addr 172.21.15.67:0/2129617567 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2129617567 >> v2:172.21.15.86:3300/0 conn(0x7f1b5c0b07b0 0x7f1b5c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163570352 0 0) 0x7f1b5c0b5980 con 0x7f1b5c15dd10 2024-04-17T10:29:35.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b40003530 con 0x7f1b5c15dd10 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2048072253 0 0) 0x7f1b40003530 con 0x7f1b5c15dd10 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 >> v2:172.21.15.86:3300/0 conn(0x7f1b5c0b07b0 msgr2=0x7f1b5c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 --2- 172.21.15.67:0/2129617567 >> v2:172.21.15.86:3300/0 conn(0x7f1b5c0b07b0 0x7f1b5c070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 >> v1:172.21.15.86:6789/0 conn(0x7f1b5c071300 legacy=0x7f1b5c06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c0b0fa0 con 0x7f1b5c15dd10 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f1b4c001680 con 0x7f1b5c15dd10 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2129617567 >> v2:172.21.15.86:3300/0 conn(0x7f1b5c0b07b0 0x7f1b5c070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 >> v1:172.21.15.67:6789/0 conn(0x7f1b5c15dd10 legacy=0x7f1b5c0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b400039c0 0x7f1b40003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b40004810 0x7f1b40004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b400051d0 0x7f1b4000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c071770 con 0x7f1b40004810 2024-04-17T10:29:35.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4000c0d0 con 0x7f1b400039c0 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4000c2d0 con 0x7f1b400051d0 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b40004810 0x7f1b40004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5aa00640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b400039c0 0x7f1b40003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5b400640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b400051d0 0x7f1b4000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 -- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b400051d0 msgr2=0x7f1b4000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b400051d0 0x7f1b4000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 -- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b400039c0 msgr2=0x7f1b40003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b400039c0 0x7f1b40003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5a000640 1 -- 172.21.15.67:0/2129617567 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b40003530 con 0x7f1b40004810 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5b400640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b400051d0 0x7f1b4000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.581+0000 7f1b5b400640 1 -- 172.21.15.67:0/2129617567 reap_dead start 2024-04-17T10:29:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b40004810 0x7f1b40004c90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009c80 tx=0x7f1b4801cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e13ec4543bc202 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b480b2070 con 0x7f1b40004810 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b480a5890 con 0x7f1b40004810 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b59600640 1 -- 172.21.15.67:0/2129617567 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48009880 con 0x7f1b40004810 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b61e00640 1 -- 172.21.15.67:0/2129617567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b40004810 msgr2=0x7f1b40004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b40004810 0x7f1b40004c90 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009c80 tx=0x7f1b4801cbd0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b61e00640 1 -- 172.21.15.67:0/2129617567 shutdown_connections 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2129617567 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b40004810 0x7f1b40004c90 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b61e00640 1 -- 172.21.15.67:0/2129617567 >> 172.21.15.67:0/2129617567 conn(0x7f1b5c0b34d0 msgr2=0x7f1b5c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b61e00640 1 -- 172.21.15.67:0/2129617567 shutdown_connections 2024-04-17T10:29:35.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.585+0000 7f1b61e00640 1 -- 172.21.15.67:0/2129617567 wait complete. 2024-04-17T10:29:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 Processor -- start 2024-04-17T10:29:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 -- start start 2024-04-17T10:29:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b40004810 0x7f1b5c15f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5c160fc0 0x7f1b5c0ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5c0cec00 0x7f1b5c0b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c06c1b0 con 0x7f1b5c160fc0 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c0b1280 con 0x7f1b5c0cec00 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5c15e0f0 con 0x7f1b40004810 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5c160fc0 0x7f1b5c0ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5c160fc0 0x7f1b5c0ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32928/0 (socket says 172.21.15.67:32928) 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 -- 172.21.15.67:0/2763303040 learned_addr learned my addr 172.21.15.67:0/2763303040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 -- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b40004810 msgr2=0x7f1b5c15f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b40004810 0x7f1b5c15f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 -- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5c0cec00 msgr2=0x7f1b5c0b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5c0cec00 0x7f1b5c0b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 -- 172.21.15.67:0/2763303040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5c0cf280 con 0x7f1b5c160fc0 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b5a000640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5c160fc0 0x7f1b5c0ce6c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009c80 tx=0x7f1b48019970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b53400640 1 -- 172.21.15.67:0/2763303040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b480b2040 con 0x7f1b5c160fc0 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b53400640 1 -- 172.21.15.67:0/2763303040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b480bd070 con 0x7f1b5c160fc0 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b53400640 1 -- 172.21.15.67:0/2763303040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b480b88b0 con 0x7f1b5c160fc0 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5c0cabf0 con 0x7f1b5c160fc0 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.589+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5c0cf560 con 0x7f1b5c160fc0 2024-04-17T10:29:35.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.593+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5c070bb0 con 0x7f1b5c160fc0 2024-04-17T10:29:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.593+0000 7f1b53400640 1 -- 172.21.15.67:0/2763303040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1b480b8a50 con 0x7f1b5c160fc0 2024-04-17T10:29:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.593+0000 7f1b53400640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b3c066f60 0x7f1b3c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.593+0000 7f1b53400640 1 -- 172.21.15.67:0/2763303040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1b4811f6b0 con 0x7f1b5c160fc0 2024-04-17T10:29:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.593+0000 7f1b5aa00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b3c066f60 0x7f1b3c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.593+0000 7f1b5aa00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b3c066f60 0x7f1b3c069420 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c1f7140 tx=0x7f1b4c001b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.597+0000 7f1b53400640 1 -- 172.21.15.67:0/2763303040 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1b480c6020 con 0x7f1b5c160fc0 2024-04-17T10:29:35.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.793+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b5c0b5cb0 con 0x7f1b5c160fc0 2024-04-17T10:29:35.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.793+0000 7f1b53400640 1 -- 172.21.15.67:0/2763303040 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f1b480168e0 con 0x7f1b5c160fc0 2024-04-17T10:29:35.788 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":47,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3163523544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3163523544},{"type":"v1","addr":"172.21.15.86:6835","nonce":3163523544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3682484259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3682484259},{"type":"v1","addr":"172.21.15.86:6837","nonce":3682484259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6945,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/252088612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":252088612},{"type":"v1","addr":"172.21.15.67:6834","nonce":252088612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:28:58.444091+0000","modified":"2024-04-17T10:29:17.323326+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6948},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6948":{"gid":6948,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.67:6835/2976334498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2976334498},{"type":"v1","addr":"172.21.15.67:6835","nonce":2976334498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-04-17T10:29:35.788 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 47 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b3c066f60 msgr2=0x7f1b3c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b3c066f60 0x7f1b3c069420 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c1f7140 tx=0x7f1b4c001b70 comp rx=0 tx=0).stop 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5c160fc0 msgr2=0x7f1b5c0ce6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5c160fc0 0x7f1b5c0ce6c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1b48009c80 tx=0x7f1b48019970 comp rx=0 tx=0).stop 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 shutdown_connections 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b3c066f60 0x7f1b3c069420 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b5c0cec00 0x7f1b5c0b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b5c160fc0 0x7f1b5c0ce6c0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 --2- 172.21.15.67:0/2763303040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b40004810 0x7f1b5c15f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 >> 172.21.15.67:0/2763303040 conn(0x7f1b5c0b34d0 msgr2=0x7f1b5c06c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:35.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 shutdown_connections 2024-04-17T10:29:35.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.797+0000 7f1b61e00640 1 -- 172.21.15.67:0/2763303040 wait complete. 2024-04-17T10:29:35.809 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:35.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2312000640 1 Processor -- start 2024-04-17T10:29:35.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2312000640 1 -- start start 2024-04-17T10:29:35.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2312000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2312000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f230c071a20 0x7f230c0cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2312000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c06e2f0 con 0x7f230c06ef10 2024-04-17T10:29:35.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2312000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230c0a86c0 con 0x7f230c071a20 2024-04-17T10:29:35.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2312000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c15bf10 con 0x7f230c145040 2024-04-17T10:29:35.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f230c071a20 0x7f230c0cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f230c071a20 0x7f230c0cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32934/0 (socket says 172.21.15.67:32934) 2024-04-17T10:29:35.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f2303e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f230c06ef10 0x7f230c071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35432/0 (socket says 172.21.15.67:35432) 2024-04-17T10:29:35.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230be00640 1 -- 172.21.15.67:0/3670472650 learned_addr learned my addr 172.21.15.67:0/3670472650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:35.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230b400640 1 --2- 172.21.15.67:0/3670472650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230be00640 1 -- 172.21.15.67:0/3670472650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 msgr2=0x7f230c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:35.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230be00640 1 --2- 172.21.15.67:0/3670472650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c06e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230be00640 1 -- 172.21.15.67:0/3670472650 >> v1:172.21.15.67:6789/0 conn(0x7f230c06ef10 legacy=0x7f230c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230be00640 1 -- 172.21.15.67:0/3670472650 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c06d6e0 con 0x7f230c071a20 2024-04-17T10:29:35.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.957+0000 7f230b400640 1 --2- 172.21.15.67:0/3670472650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c06e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f230be00640 1 --2- 172.21.15.67:0/3670472650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c071a20 0x7f230c0cc300 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f2304009e50 tx=0x7f230401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5012f5bed97920b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f230aa00640 1 -- 172.21.15.67:0/3670472650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23040b4070 con 0x7f230c071a20 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f230aa00640 1 -- 172.21.15.67:0/3670472650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23040af400 con 0x7f230c071a20 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f230aa00640 1 -- 172.21.15.67:0/3670472650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23040af7d0 con 0x7f230c071a20 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 -- 172.21.15.67:0/3670472650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c071a20 msgr2=0x7f230c0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 --2- 172.21.15.67:0/3670472650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c071a20 0x7f230c0cc300 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f2304009e50 tx=0x7f230401f210 comp rx=0 tx=0).stop 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 -- 172.21.15.67:0/3670472650 shutdown_connections 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 --2- 172.21.15.67:0/3670472650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c071a20 0x7f230c0cc300 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 --2- 172.21.15.67:0/3670472650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c06e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 -- 172.21.15.67:0/3670472650 >> 172.21.15.67:0/3670472650 conn(0x7f230c06ac90 msgr2=0x7f230c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 -- 172.21.15.67:0/3670472650 shutdown_connections 2024-04-17T10:29:35.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.961+0000 7f2312000640 1 -- 172.21.15.67:0/3670472650 wait complete. 2024-04-17T10:29:35.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 Processor -- start 2024-04-17T10:29:35.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 -- start start 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c06ef10 0x7f230c13b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f230c071a20 0x7f230c15ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c15e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c06df80 con 0x7f230c06ef10 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c06dca0 con 0x7f230c071a20 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c06d370 con 0x7f230c145040 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c06ef10 0x7f230c13b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c06ef10 0x7f230c13b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32942/0 (socket says 172.21.15.67:32942) 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 -- 172.21.15.67:0/572228907 learned_addr learned my addr 172.21.15.67:0/572228907 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 -- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 msgr2=0x7f230c15e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c15e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 -- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f230c071a20 msgr2=0x7f230c15ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2303e00640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f230c071a20 0x7f230c15ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f230c071a20 0x7f230c15ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 -- 172.21.15.67:0/572228907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c1414a0 con 0x7f230c06ef10 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2303e00640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f230c071a20 0x7f230c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f230b400640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c06ef10 0x7f230c13b770 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c7f0 tx=0x7f22fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2303400640 1 -- 172.21.15.67:0/572228907 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc015070 con 0x7f230c06ef10 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2303400640 1 -- 172.21.15.67:0/572228907 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc007820 con 0x7f230c06ef10 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c141710 con 0x7f230c06ef10 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2303400640 1 -- 172.21.15.67:0/572228907 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc007c10 con 0x7f230c06ef10 2024-04-17T10:29:35.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c13af70 con 0x7f230c06ef10 2024-04-17T10:29:35.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.965+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c072d50 con 0x7f230c06ef10 2024-04-17T10:29:35.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.969+0000 7f2303400640 1 -- 172.21.15.67:0/572228907 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f22fc018070 con 0x7f230c06ef10 2024-04-17T10:29:35.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.969+0000 7f2303400640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22e8066f60 0x7f22e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:35.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.969+0000 7f2303400640 1 -- 172.21.15.67:0/572228907 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f22fc085330 con 0x7f230c06ef10 2024-04-17T10:29:35.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.969+0000 7f2303e00640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22e8066f60 0x7f22e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:35.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.969+0000 7f2303e00640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22e8066f60 0x7f22e8069420 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f230c0715f0 tx=0x7f22f4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:35.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:35.973+0000 7f2303400640 1 -- 172.21.15.67:0/572228907 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f22fc056150 con 0x7f230c06ef10 2024-04-17T10:29:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.141+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f230c0b11e0 con 0x7f230c06ef10 2024-04-17T10:29:36.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.141+0000 7f2303400640 1 -- 172.21.15.67:0/572228907 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f22fc055a50 con 0x7f230c06ef10 2024-04-17T10:29:36.136 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:36.136 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":71,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:14.294073+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:28:56.861344+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-04-17T10:28:57.434569+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/392417778":"2024-04-18T10:29:14.294053+0000","172.21.15.67:6833/392417778":"2024-04-18T10:29:14.294053+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22e8066f60 msgr2=0x7f22e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22e8066f60 0x7f22e8069420 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f230c0715f0 tx=0x7f22f4009040 comp rx=0 tx=0).stop 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c06ef10 msgr2=0x7f230c13b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c06ef10 0x7f230c13b770 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f22fc00c7f0 tx=0x7f22fc00ccb0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 shutdown_connections 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22e8066f60 0x7f22e8069420 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f230c145040 0x7f230c15e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f230c071a20 0x7f230c15ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 --2- 172.21.15.67:0/572228907 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f230c06ef10 0x7f230c13b770 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 >> 172.21.15.67:0/572228907 conn(0x7f230c06ac90 msgr2=0x7f230c0cc950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 shutdown_connections 2024-04-17T10:29:36.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.145+0000 7f2312000640 1 -- 172.21.15.67:0/572228907 wait complete. 2024-04-17T10:29:36.158 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:29:36.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.305+0000 7fd8a5c00640 1 Processor -- start 2024-04-17T10:29:36.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.305+0000 7fd8a5c00640 1 -- start start 2024-04-17T10:29:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd8a5c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006eac0 0x7fd8a0143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd8a5c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8a00cf0c0 0x7fd8a0144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd8a5c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8a0144560 con 0x7fd8a006e6e0 2024-04-17T10:29:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd8a5c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8a015ae40 con 0x7fd8a00cf0c0 2024-04-17T10:29:36.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd8a5c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a0071440 con 0x7fd8a006eac0 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd89f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8a00cf0c0 0x7fd8a0144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd89f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8a00cf0c0 0x7fd8a0144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32958/0 (socket says 172.21.15.67:32958) 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd89f400640 1 -- 172.21.15.67:0/623798620 learned_addr learned my addr 172.21.15.67:0/623798620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd897e00640 1 --2- 172.21.15.67:0/623798620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006eac0 0x7fd8a0143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd89f400640 1 -- 172.21.15.67:0/623798620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006eac0 msgr2=0x7fd8a0143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd89f400640 1 --2- 172.21.15.67:0/623798620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006eac0 0x7fd8a0143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd89f400640 1 -- 172.21.15.67:0/623798620 >> v1:172.21.15.86:6789/0 conn(0x7fd8a006e6e0 legacy=0x7fd8a00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.309+0000 7fd89f400640 1 -- 172.21.15.67:0/623798620 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0071720 con 0x7fd8a00cf0c0 2024-04-17T10:29:36.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd89f400640 1 --2- 172.21.15.67:0/623798620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a00cf0c0 0x7fd8a0144020 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd898009e50 tx=0x7fd89801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca8a716b646a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd89e000640 1 -- 172.21.15.67:0/623798620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980a5830 con 0x7fd8a00cf0c0 2024-04-17T10:29:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd89e000640 1 -- 172.21.15.67:0/623798620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8980a59d0 con 0x7fd8a00cf0c0 2024-04-17T10:29:36.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd89e000640 1 -- 172.21.15.67:0/623798620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8980a5da0 con 0x7fd8a00cf0c0 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 -- 172.21.15.67:0/623798620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a00cf0c0 msgr2=0x7fd8a0144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/623798620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a00cf0c0 0x7fd8a0144020 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd898009e50 tx=0x7fd89801c970 comp rx=0 tx=0).stop 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 -- 172.21.15.67:0/623798620 shutdown_connections 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/623798620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a00cf0c0 0x7fd8a0144020 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/623798620 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006eac0 0x7fd8a0143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 -- 172.21.15.67:0/623798620 >> 172.21.15.67:0/623798620 conn(0x7fd8a006ac90 msgr2=0x7fd8a006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 -- 172.21.15.67:0/623798620 shutdown_connections 2024-04-17T10:29:36.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 -- 172.21.15.67:0/623798620 wait complete. 2024-04-17T10:29:36.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 Processor -- start 2024-04-17T10:29:36.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 -- start start 2024-04-17T10:29:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.313+0000 7fd8a5c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006e6e0 0x7fd8a0159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a006eac0 0x7fd8a00b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a00cf0c0 0x7fd8a0159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00b3a40 con 0x7fd8a006eac0 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00b38c0 con 0x7fd8a00cf0c0 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a00b3bc0 con 0x7fd8a006e6e0 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a006eac0 0x7fd8a00b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a006eac0 0x7fd8a00b1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32970/0 (socket says 172.21.15.67:32970) 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 -- 172.21.15.67:0/1490061337 learned_addr learned my addr 172.21.15.67:0/1490061337 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd89f400640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a00cf0c0 0x7fd8a0159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd89ea00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006e6e0 0x7fd8a0159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 -- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006e6e0 msgr2=0x7fd8a0159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006e6e0 0x7fd8a0159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 -- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a00cf0c0 msgr2=0x7fd8a0159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a00cf0c0 0x7fd8a0159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 -- 172.21.15.67:0/1490061337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a00b0240 con 0x7fd8a006eac0 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd897e00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a006eac0 0x7fd8a00b1680 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd88800d800 tx=0x7fd88800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd896a00640 1 -- 172.21.15.67:0/1490061337 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800b6c0 con 0x7fd8a006eac0 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd896a00640 1 -- 172.21.15.67:0/1490061337 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd888019070 con 0x7fd8a006eac0 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd896a00640 1 -- 172.21.15.67:0/1490061337 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800ba60 con 0x7fd8a006eac0 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a00b0520 con 0x7fd8a006eac0 2024-04-17T10:29:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a00b0ae0 con 0x7fd8a006eac0 2024-04-17T10:29:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a0072d50 con 0x7fd8a006eac0 2024-04-17T10:29:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.317+0000 7fd896a00640 1 -- 172.21.15.67:0/1490061337 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd88800bc00 con 0x7fd8a006eac0 2024-04-17T10:29:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.321+0000 7fd896a00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd87c066f60 0x7fd87c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.321+0000 7fd896a00640 1 -- 172.21.15.67:0/1490061337 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd8880855a0 con 0x7fd8a006eac0 2024-04-17T10:29:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.321+0000 7fd89ea00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd87c066f60 0x7fd87c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.321+0000 7fd89ea00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd87c066f60 0x7fd87c069420 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd890001810 tx=0x7fd890009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:36.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.325+0000 7fd896a00640 1 -- 172.21.15.67:0/1490061337 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd888055d80 con 0x7fd8a006eac0 2024-04-17T10:29:36.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.525+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd8a00b2c60 con 0x7fd8a006eac0 2024-04-17T10:29:36.519 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:29:36.525+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:29:36.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd896a00640 1 -- 172.21.15.67:0/1490061337 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v48) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd888059e20 con 0x7fd8a006eac0 2024-04-17T10:29:36.521 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:29:36.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd87c066f60 msgr2=0x7fd87c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd87c066f60 0x7fd87c069420 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd890001810 tx=0x7fd890009040 comp rx=0 tx=0).stop 2024-04-17T10:29:36.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a006eac0 msgr2=0x7fd8a00b1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a006eac0 0x7fd8a00b1680 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fd88800d800 tx=0x7fd88800dcc0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 shutdown_connections 2024-04-17T10:29:36.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd87c066f60 0x7fd87c069420 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd8a00cf0c0 0x7fd8a0159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8a006eac0 0x7fd8a00b1680 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 --2- 172.21.15.67:0/1490061337 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd8a006e6e0 0x7fd8a0159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 >> 172.21.15.67:0/1490061337 conn(0x7fd8a006ac90 msgr2=0x7fd8a00ce990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:36.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.529+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 shutdown_connections 2024-04-17T10:29:36.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.533+0000 7fd8a5c00640 1 -- 172.21.15.67:0/1490061337 wait complete. 2024-04-17T10:29:36.537 INFO:tasks.ceph.mds.c.smithi067.stderr: -1716> 2024-04-17T10:29:20.305+0000 7f71f3a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:29:36.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:29:36.582 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:29:36.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1ac00640 1 Processor -- start 2024-04-17T10:29:36.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1ac00640 1 -- start start 2024-04-17T10:29:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1ac00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1ac00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed140cf200 0x7fed140cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1ac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1415f050 con 0x7fed1415da50 2024-04-17T10:29:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1ac00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1409a3f0 con 0x7fed140cf200 2024-04-17T10:29:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1ac00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed140ccfd0 con 0x7fed14099a50 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed1a200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed140cf200 0x7fed140cf690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed19800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fed1415da50 0x7fed1415e940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35440/0 (socket says 172.21.15.67:35440) 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.701+0000 7fed19800640 1 -- 172.21.15.67:0/1396546676 learned_addr learned my addr 172.21.15.67:0/1396546676 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed18e00640 1 --2- 172.21.15.67:0/1396546676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed1a200640 1 -- 172.21.15.67:0/1396546676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 msgr2=0x7fed14099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed1a200640 1 --2- 172.21.15.67:0/1396546676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14099eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed1a200640 1 -- 172.21.15.67:0/1396546676 >> v1:172.21.15.67:6789/0 conn(0x7fed1415da50 legacy=0x7fed1415e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed1a200640 1 -- 172.21.15.67:0/1396546676 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed140cc140 con 0x7fed140cf200 2024-04-17T10:29:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed13e00640 1 -- 172.21.15.67:0/1396546676 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261119942 0 0) 0x7fed1415f050 con 0x7fed1415da50 2024-04-17T10:29:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed1a200640 1 --2- 172.21.15.67:0/1396546676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140cf690 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0097c0 tx=0x7fed0c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff320c9660ec7ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed13e00640 1 -- 172.21.15.67:0/1396546676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0ab640 con 0x7fed140cf200 2024-04-17T10:29:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed13e00640 1 -- 172.21.15.67:0/1396546676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c0b0070 con 0x7fed140cf200 2024-04-17T10:29:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.705+0000 7fed13e00640 1 -- 172.21.15.67:0/1396546676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c0aba00 con 0x7fed140cf200 2024-04-17T10:29:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- 172.21.15.67:0/1396546676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 msgr2=0x7fed140cf690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 --2- 172.21.15.67:0/1396546676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140cf690 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fed0c0097c0 tx=0x7fed0c01c480 comp rx=0 tx=0).stop 2024-04-17T10:29:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- 172.21.15.67:0/1396546676 shutdown_connections 2024-04-17T10:29:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 --2- 172.21.15.67:0/1396546676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140cf690 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 --2- 172.21.15.67:0/1396546676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14099eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- 172.21.15.67:0/1396546676 >> 172.21.15.67:0/1396546676 conn(0x7fed140d1630 msgr2=0x7fed140d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- 172.21.15.67:0/1396546676 shutdown_connections 2024-04-17T10:29:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- 172.21.15.67:0/1396546676 wait complete. 2024-04-17T10:29:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 Processor -- start 2024-04-17T10:29:36.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- start start 2024-04-17T10:29:36.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140c7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1415da50 0x7fed14152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed140ccc60 con 0x7fed140cf200 2024-04-17T10:29:36.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed140cc980 con 0x7fed1415da50 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1ac00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1409ad40 con 0x7fed14099a50 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140c7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140c7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32996/0 (socket says 172.21.15.67:32996) 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 -- 172.21.15.67:0/3271325100 learned_addr learned my addr 172.21.15.67:0/3271325100 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed19800640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 -- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 msgr2=0x7fed14151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1a200640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1415da50 0x7fed14152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 -- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1415da50 msgr2=0x7fed14152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1415da50 0x7fed14152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 -- 172.21.15.67:0/3271325100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed140c5680 con 0x7fed140cf200 2024-04-17T10:29:36.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed1a200640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1415da50 0x7fed14152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.709+0000 7fed18e00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140c7c00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e870 tx=0x7fed0000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed11600640 1 -- 172.21.15.67:0/3271325100 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00017070 con 0x7fed140cf200 2024-04-17T10:29:36.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed11600640 1 -- 172.21.15.67:0/3271325100 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0000c8e0 con 0x7fed140cf200 2024-04-17T10:29:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed140cb720 con 0x7fed140cf200 2024-04-17T10:29:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed11600640 1 -- 172.21.15.67:0/3271325100 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000cc90 con 0x7fed140cf200 2024-04-17T10:29:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed140c5c00 con 0x7fed140cf200 2024-04-17T10:29:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1404f780 con 0x7fed140cf200 2024-04-17T10:29:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed11600640 1 -- 172.21.15.67:0/3271325100 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fed0001a070 con 0x7fed140cf200 2024-04-17T10:29:36.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed11600640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066f60 0x7fecf4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed11600640 1 -- 172.21.15.67:0/3271325100 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fed000850d0 con 0x7fed140cf200 2024-04-17T10:29:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed19800640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066f60 0x7fecf4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:36.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.713+0000 7fed19800640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066f60 0x7fecf4069420 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fed080012f0 tx=0x7fed08001770 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:36.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.717+0000 7fed11600640 1 -- 172.21.15.67:0/3271325100 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fed000557c0 con 0x7fed140cf200 2024-04-17T10:29:36.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:36.917+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fed14152420 con 0x7fed140cf200 2024-04-17T10:29:37.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.529+0000 7fed11600640 1 -- 172.21.15.67:0/3271325100 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7fed00059860 con 0x7fed140cf200 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066f60 msgr2=0x7fecf4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066f60 0x7fecf4069420 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fed080012f0 tx=0x7fed08001770 comp rx=0 tx=0).stop 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 msgr2=0x7fed140c7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140c7c00 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fed0000e870 tx=0x7fed0000ec70 comp rx=0 tx=0).stop 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 shutdown_connections 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066f60 0x7fecf4069420 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed1415da50 0x7fed14152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed140cf200 0x7fed140c7c00 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 --2- 172.21.15.67:0/3271325100 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed14099a50 0x7fed14151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 >> 172.21.15.67:0/3271325100 conn(0x7fed140d1630 msgr2=0x7fed1415e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 shutdown_connections 2024-04-17T10:29:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.533+0000 7fed1ac00640 1 -- 172.21.15.67:0/3271325100 wait complete. 2024-04-17T10:29:37.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:29:37.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.745+0000 7fec62800640 1 Processor -- start 2024-04-17T10:29:37.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.745+0000 7fec62800640 1 -- start start 2024-04-17T10:29:37.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec62800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec5c06d3c0 0x7fec5c071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:37.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec62800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5c071cb0 con 0x7fec5c0718a0 2024-04-17T10:29:37.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec62800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5c06d9a0 con 0x7fec5c1447e0 2024-04-17T10:29:37.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec62800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec5c0cc310 con 0x7fec5c06d3c0 2024-04-17T10:29:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec5be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fec5c0718a0 0x7fec5c0cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35444/0 (socket says 172.21.15.67:35444) 2024-04-17T10:29:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec5be00640 1 -- 172.21.15.67:0/2893855978 learned_addr learned my addr 172.21.15.67:0/2893855978 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec53e00640 1 --2- 172.21.15.67:0/2893855978 >> v2:172.21.15.86:3300/0 conn(0x7fec5c06d3c0 0x7fec5c071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972961897 0 0) 0x7fec540014a0 con 0x7fec5c0718a0 2024-04-17T10:29:37.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec40003530 con 0x7fec5c0718a0 2024-04-17T10:29:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555334566 0 0) 0x7fec5c06d9a0 con 0x7fec5c1447e0 2024-04-17T10:29:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec540014a0 con 0x7fec5c1447e0 2024-04-17T10:29:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec53e00640 1 -- 172.21.15.67:0/2893855978 >> v1:172.21.15.86:6789/0 conn(0x7fec5c1447e0 legacy=0x7fec5c144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec53e00640 1 -- 172.21.15.67:0/2893855978 >> v1:172.21.15.67:6789/0 conn(0x7fec5c0718a0 legacy=0x7fec5c0cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4157641607 0 0) 0x7fec40003530 con 0x7fec5c0718a0 2024-04-17T10:29:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.749+0000 7fec53e00640 1 -- 172.21.15.67:0/2893855978 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c06dc80 con 0x7fec5c06d3c0 2024-04-17T10:29:37.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec53e00640 1 --2- 172.21.15.67:0/2893855978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c06d3c0 0x7fec5c071360 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec440097c0 tx=0x7fec4401c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73d6fe7d00614633 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:37.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec440b0070 con 0x7fec5c06d3c0 2024-04-17T10:29:37.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec440ab400 con 0x7fec5c06d3c0 2024-04-17T10:29:37.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec5aa00640 1 -- 172.21.15.67:0/2893855978 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec54001bd0 con 0x7fec5c06d3c0 2024-04-17T10:29:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 -- 172.21.15.67:0/2893855978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c06d3c0 msgr2=0x7fec5c071360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 --2- 172.21.15.67:0/2893855978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c06d3c0 0x7fec5c071360 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec440097c0 tx=0x7fec4401c300 comp rx=0 tx=0).stop 2024-04-17T10:29:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 -- 172.21.15.67:0/2893855978 shutdown_connections 2024-04-17T10:29:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 --2- 172.21.15.67:0/2893855978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c06d3c0 0x7fec5c071360 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 -- 172.21.15.67:0/2893855978 >> 172.21.15.67:0/2893855978 conn(0x7fec5c06ac90 msgr2=0x7fec5c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 -- 172.21.15.67:0/2893855978 shutdown_connections 2024-04-17T10:29:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 -- 172.21.15.67:0/2893855978 wait complete. 2024-04-17T10:29:37.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 Processor -- start 2024-04-17T10:29:37.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.753+0000 7fec62800640 1 -- start start 2024-04-17T10:29:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec5c06d3c0 0x7fec5c0b4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c0718a0 0x7fec5c15abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5c1447e0 0x7fec5c135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0cda90 con 0x7fec5c06d3c0 2024-04-17T10:29:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c0cd910 con 0x7fec5c0718a0 2024-04-17T10:29:37.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec5c1695f0 con 0x7fec5c1447e0 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec5c06d3c0 0x7fec5c0b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec5c06d3c0 0x7fec5c0b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33008/0 (socket says 172.21.15.67:33008) 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 -- 172.21.15.67:0/3431400589 learned_addr learned my addr 172.21.15.67:0/3431400589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec53e00640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c0718a0 0x7fec5c15abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5be00640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5c1447e0 0x7fec5c135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 -- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5c1447e0 msgr2=0x7fec5c135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5c1447e0 0x7fec5c135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 -- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c0718a0 msgr2=0x7fec5c15abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c0718a0 0x7fec5c15abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 -- 172.21.15.67:0/3431400589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c15b5b0 con 0x7fec5c06d3c0 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5be00640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5c1447e0 0x7fec5c135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec5b400640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec5c06d3c0 0x7fec5c0b4210 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c430 tx=0x7fec4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:37.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec53400640 1 -- 172.21.15.67:0/3431400589 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c017070 con 0x7fec5c06d3c0 2024-04-17T10:29:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec53400640 1 -- 172.21.15.67:0/3431400589 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec4c0052f0 con 0x7fec5c06d3c0 2024-04-17T10:29:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec53400640 1 -- 172.21.15.67:0/3431400589 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4c003840 con 0x7fec5c06d3c0 2024-04-17T10:29:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5c0b4820 con 0x7fec5c06d3c0 2024-04-17T10:29:37.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec5c1331d0 con 0x7fec5c06d3c0 2024-04-17T10:29:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5c072da0 con 0x7fec5c06d3c0 2024-04-17T10:29:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.757+0000 7fec53400640 1 -- 172.21.15.67:0/3431400589 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fec4c01a070 con 0x7fec5c06d3c0 2024-04-17T10:29:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.761+0000 7fec53400640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec38066f90 0x7fec38069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.761+0000 7fec53400640 1 -- 172.21.15.67:0/3431400589 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6418+0+0 (secure 0 0 0) 0x7fec4c084660 con 0x7fec5c06d3c0 2024-04-17T10:29:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.761+0000 7fec53e00640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec38066f90 0x7fec38069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.761+0000 7fec53e00640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec38066f90 0x7fec38069450 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fec440ad000 tx=0x7fec440a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:37.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.765+0000 7fec53400640 1 -- 172.21.15.67:0/3431400589 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fec5c15b5b0 con 0x7fec5c06d3c0 2024-04-17T10:29:37.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:37.925+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fec5c067e20 con 0x7fec5c06d3c0 2024-04-17T10:29:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.533+0000 7fec53400640 1 -- 172.21.15.67:0/3431400589 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 181+0+0 (secure 0 0 0) 0x7fec4c054cb0 con 0x7fec5c06d3c0 2024-04-17T10:29:38.528 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec38066f90 msgr2=0x7fec38069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec38066f90 0x7fec38069450 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fec440ad000 tx=0x7fec440a6000 comp rx=0 tx=0).stop 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec5c06d3c0 msgr2=0x7fec5c0b4210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec5c06d3c0 0x7fec5c0b4210 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fec4c00c430 tx=0x7fec4c00c8f0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 shutdown_connections 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec38066f90 0x7fec38069450 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec5c1447e0 0x7fec5c135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec5c0718a0 0x7fec5c15abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 --2- 172.21.15.67:0/3431400589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec5c06d3c0 0x7fec5c0b4210 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 >> 172.21.15.67:0/3431400589 conn(0x7fec5c06ac90 msgr2=0x7fec5c1419a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:38.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 shutdown_connections 2024-04-17T10:29:38.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.537+0000 7fec62800640 1 -- 172.21.15.67:0/3431400589 wait complete. 2024-04-17T10:29:38.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:29:38.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 Processor -- start 2024-04-17T10:29:38.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 -- start start 2024-04-17T10:29:38.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb2415db90 0x7fbb2415eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:38.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb240b5270 0x7fbb240b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:38.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb24070d10 0x7fbb240711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb240b0b30 con 0x7fbb240b5270 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb240b0e10 con 0x7fbb24070d10 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb28e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb240b10f0 con 0x7fbb2415db90 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb240b5270 0x7fbb240b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb240b5270 0x7fbb240b56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33018/0 (socket says 172.21.15.67:33018) 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 -- 172.21.15.67:0/315176123 learned_addr learned my addr 172.21.15.67:0/315176123 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb21e00640 1 --2- 172.21.15.67:0/315176123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb2415db90 0x7fbb2415eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb22800640 1 --2- 172.21.15.67:0/315176123 >> v2:172.21.15.86:3300/0 conn(0x7fbb24070d10 0x7fbb240711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 -- 172.21.15.67:0/315176123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb2415db90 msgr2=0x7fbb2415eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 --2- 172.21.15.67:0/315176123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb2415db90 0x7fbb2415eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 -- 172.21.15.67:0/315176123 >> v2:172.21.15.86:3300/0 conn(0x7fbb24070d10 msgr2=0x7fbb240711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 --2- 172.21.15.67:0/315176123 >> v2:172.21.15.86:3300/0 conn(0x7fbb24070d10 0x7fbb240711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.709+0000 7fbb1be00640 1 -- 172.21.15.67:0/315176123 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2415e2a0 con 0x7fbb240b5270 2024-04-17T10:29:38.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb1be00640 1 --2- 172.21.15.67:0/315176123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb240b5270 0x7fbb240b56b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fbb240b5120 tx=0x7fbb0c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d251300c58cc0d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:38.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb21400640 1 -- 172.21.15.67:0/315176123 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c0b0070 con 0x7fbb240b5270 2024-04-17T10:29:38.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb21400640 1 -- 172.21.15.67:0/315176123 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0c0ab600 con 0x7fbb240b5270 2024-04-17T10:29:38.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb21400640 1 -- 172.21.15.67:0/315176123 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c0ab9d0 con 0x7fbb240b5270 2024-04-17T10:29:38.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 -- 172.21.15.67:0/315176123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb240b5270 msgr2=0x7fbb240b56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:38.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 --2- 172.21.15.67:0/315176123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb240b5270 0x7fbb240b56b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fbb240b5120 tx=0x7fbb0c01c490 comp rx=0 tx=0).stop 2024-04-17T10:29:38.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 -- 172.21.15.67:0/315176123 shutdown_connections 2024-04-17T10:29:38.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 --2- 172.21.15.67:0/315176123 >> v2:172.21.15.86:3300/0 conn(0x7fbb24070d10 0x7fbb240711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 --2- 172.21.15.67:0/315176123 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb240b5270 0x7fbb240b56b0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 --2- 172.21.15.67:0/315176123 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb2415db90 0x7fbb2415eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 -- 172.21.15.67:0/315176123 >> 172.21.15.67:0/315176123 conn(0x7fbb2406f590 msgr2=0x7fbb2406f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:38.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 -- 172.21.15.67:0/315176123 shutdown_connections 2024-04-17T10:29:38.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.713+0000 7fbb28e00640 1 -- 172.21.15.67:0/315176123 wait complete. 2024-04-17T10:29:38.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 Processor -- start 2024-04-17T10:29:38.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 -- start start 2024-04-17T10:29:38.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24070d10 0x7fbb241552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb240b5270 0x7fbb240cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2415db90 0x7fbb2415ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2415f160 con 0x7fbb2415db90 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2415efe0 con 0x7fbb24070d10 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2406c510 con 0x7fbb240b5270 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2415db90 0x7fbb2415ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2415db90 0x7fbb2415ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33020/0 (socket says 172.21.15.67:33020) 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 -- 172.21.15.67:0/883782306 learned_addr learned my addr 172.21.15.67:0/883782306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb1be00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb240b5270 0x7fbb240cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb21e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24070d10 0x7fbb241552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 -- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb240b5270 msgr2=0x7fbb240cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb240b5270 0x7fbb240cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 -- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24070d10 msgr2=0x7fbb241552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24070d10 0x7fbb241552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 -- 172.21.15.67:0/883782306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24158b80 con 0x7fbb2415db90 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb22800640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2415db90 0x7fbb2415ba30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c007e60 tx=0x7fbb1c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb1a000640 1 -- 172.21.15.67:0/883782306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c014620 con 0x7fbb2415db90 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb1a000640 1 -- 172.21.15.67:0/883782306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1c019070 con 0x7fbb2415db90 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb1a000640 1 -- 172.21.15.67:0/883782306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c014a20 con 0x7fbb2415db90 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb240cbe20 con 0x7fbb2415db90 2024-04-17T10:29:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.717+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb240cbb40 con 0x7fbb2415db90 2024-04-17T10:29:38.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.721+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb24074e30 con 0x7fbb2415db90 2024-04-17T10:29:38.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.721+0000 7fbb1a000640 1 -- 172.21.15.67:0/883782306 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbb1c014bc0 con 0x7fbb2415db90 2024-04-17T10:29:38.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.721+0000 7fbb1a000640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb00066f60 0x7fbb00069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:38.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.721+0000 7fbb1a000640 1 -- 172.21.15.67:0/883782306 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5951+0+0 (secure 0 0 0) 0x7fbb1c0858a0 con 0x7fbb2415db90 2024-04-17T10:29:38.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.721+0000 7fbb21e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb00066f60 0x7fbb00069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:38.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.721+0000 7fbb21e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb00066f60 0x7fbb00069420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbb241f9e70 tx=0x7fbb14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:38.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.725+0000 7fbb1a000640 1 -- 172.21.15.67:0/883782306 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbb1c0560d0 con 0x7fbb2415db90 2024-04-17T10:29:38.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:38.889+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbb240b3040 con 0x7fbb2415db90 2024-04-17T10:29:39.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.537+0000 7fbb1a000640 1 -- 172.21.15.67:0/883782306 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbb1c05a170 con 0x7fbb2415db90 2024-04-17T10:29:39.530 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.537+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb00066f60 msgr2=0x7fbb00069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.537+0000 7fbb28e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb00066f60 0x7fbb00069420 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbb241f9e70 tx=0x7fbb14008040 comp rx=0 tx=0).stop 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.537+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2415db90 msgr2=0x7fbb2415ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.537+0000 7fbb28e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2415db90 0x7fbb2415ba30 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c007e60 tx=0x7fbb1c00e4b0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 shutdown_connections 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb00066f60 0x7fbb00069420 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb2415db90 0x7fbb2415ba30 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb240b5270 0x7fbb240cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 --2- 172.21.15.67:0/883782306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb24070d10 0x7fbb241552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 >> 172.21.15.67:0/883782306 conn(0x7fbb2406f590 msgr2=0x7fbb240d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:39.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 shutdown_connections 2024-04-17T10:29:39.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.541+0000 7fbb28e00640 1 -- 172.21.15.67:0/883782306 wait complete. 2024-04-17T10:29:39.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:29:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.717+0000 7f09c0600640 1 Processor -- start 2024-04-17T10:29:39.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.717+0000 7f09c0600640 1 -- start start 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.717+0000 7f09c0600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09b8159440 0x7f09b80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.717+0000 7f09c0600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b815c580 0x7f09b8071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.717+0000 7f09c0600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b815cb30 con 0x7f09b80713e0 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.717+0000 7f09c0600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b80a8650 con 0x7f09b8159440 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.717+0000 7f09c0600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815ce10 con 0x7f09b815c580 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09bdc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09b8159440 0x7f09b80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09be600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b815c580 0x7f09b8071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09bdc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09b8159440 0x7f09b80ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:47150/0 (socket says 172.21.15.67:47150) 2024-04-17T10:29:39.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09bdc00640 1 -- 172.21.15.67:0/1395626717 learned_addr learned my addr 172.21.15.67:0/1395626717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09be600640 1 -- 172.21.15.67:0/1395626717 >> v2:172.21.15.86:3300/0 conn(0x7f09b8159440 msgr2=0x7f09b80ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09be600640 1 --2- 172.21.15.67:0/1395626717 >> v2:172.21.15.86:3300/0 conn(0x7f09b8159440 0x7f09b80ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09be600640 1 -- 172.21.15.67:0/1395626717 >> v1:172.21.15.86:6789/0 conn(0x7f09b80713e0 legacy=0x7f09b815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09be600640 1 -- 172.21.15.67:0/1395626717 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b80cd610 con 0x7f09b815c580 2024-04-17T10:29:39.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09be600640 1 --2- 172.21.15.67:0/1395626717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b815c580 0x7f09b8071870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009d90 tx=0x7f09b001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=faf9e13c6a80e490 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:39.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.721+0000 7f09b7e00640 1 -- 172.21.15.67:0/1395626717 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b00b2070 con 0x7f09b815c580 2024-04-17T10:29:39.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09b7e00640 1 -- 172.21.15.67:0/1395626717 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b00a5940 con 0x7f09b815c580 2024-04-17T10:29:39.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09b7e00640 1 -- 172.21.15.67:0/1395626717 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b00a5d10 con 0x7f09b815c580 2024-04-17T10:29:39.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- 172.21.15.67:0/1395626717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b815c580 msgr2=0x7f09b8071870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 --2- 172.21.15.67:0/1395626717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b815c580 0x7f09b8071870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009d90 tx=0x7f09b001c970 comp rx=0 tx=0).stop 2024-04-17T10:29:39.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- 172.21.15.67:0/1395626717 shutdown_connections 2024-04-17T10:29:39.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 --2- 172.21.15.67:0/1395626717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b815c580 0x7f09b8071870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 --2- 172.21.15.67:0/1395626717 >> v2:172.21.15.86:3300/0 conn(0x7f09b8159440 0x7f09b80ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- 172.21.15.67:0/1395626717 >> 172.21.15.67:0/1395626717 conn(0x7f09b80b1740 msgr2=0x7f09b80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:39.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- 172.21.15.67:0/1395626717 shutdown_connections 2024-04-17T10:29:39.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- 172.21.15.67:0/1395626717 wait complete. 2024-04-17T10:29:39.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 Processor -- start 2024-04-17T10:29:39.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- start start 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b80713e0 0x7f09b814e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09b8159440 0x7f09b80b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b815c580 0x7f09b8159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815d9c0 con 0x7f09b815c580 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815d840 con 0x7f09b8159440 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09c0600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b815db40 con 0x7f09b80713e0 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b815c580 0x7f09b8159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b815c580 0x7f09b8159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33032/0 (socket says 172.21.15.67:33032) 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 -- 172.21.15.67:0/3498571972 learned_addr learned my addr 172.21.15.67:0/3498571972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09bdc00640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b80713e0 0x7f09b814e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09bd200640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09b8159440 0x7f09b80b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 -- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b80713e0 msgr2=0x7f09b814e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b80713e0 0x7f09b814e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 -- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09b8159440 msgr2=0x7f09b80b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09b8159440 0x7f09b80b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.725+0000 7f09be600640 1 -- 172.21.15.67:0/3498571972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b815ab10 con 0x7f09b815c580 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09bd200640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09b8159440 0x7f09b80b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09be600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b815c580 0x7f09b8159a90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f09b00b0040 tx=0x7f09b0019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09b5600640 1 -- 172.21.15.67:0/3498571972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b00b2050 con 0x7f09b815c580 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09b5600640 1 -- 172.21.15.67:0/3498571972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b00be070 con 0x7f09b815c580 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b815adf0 con 0x7f09b815c580 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09b5600640 1 -- 172.21.15.67:0/3498571972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b00ada90 con 0x7f09b815c580 2024-04-17T10:29:39.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b815b3b0 con 0x7f09b815c580 2024-04-17T10:29:39.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b804ef30 con 0x7f09b815c580 2024-04-17T10:29:39.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09b5600640 1 -- 172.21.15.67:0/3498571972 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f09b00adc30 con 0x7f09b815c580 2024-04-17T10:29:39.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09b5600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0998066f60 0x7f0998069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:39.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.729+0000 7f09b5600640 1 -- 172.21.15.67:0/3498571972 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f09b011fe40 con 0x7f09b815c580 2024-04-17T10:29:39.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.733+0000 7f09bdc00640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0998066f60 0x7f0998069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:39.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.733+0000 7f09bdc00640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0998066f60 0x7f0998069420 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f09b80ce140 tx=0x7f09ac009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:39.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.737+0000 7f09b5600640 1 -- 172.21.15.67:0/3498571972 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f09b0016560 con 0x7f09b815c580 2024-04-17T10:29:39.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.897+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f09b815edb0 con 0x7f09b815c580 2024-04-17T10:29:39.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.901+0000 7f09b5600640 1 -- 172.21.15.67:0/3498571972 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f09b00f1000 con 0x7f09b815c580 2024-04-17T10:29:39.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0998066f60 msgr2=0x7f0998069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0998066f60 0x7f0998069420 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f09b80ce140 tx=0x7f09ac009240 comp rx=0 tx=0).stop 2024-04-17T10:29:39.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b815c580 msgr2=0x7f09b8159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b815c580 0x7f09b8159a90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f09b00b0040 tx=0x7f09b0019710 comp rx=0 tx=0).stop 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 shutdown_connections 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0998066f60 0x7f0998069420 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09b815c580 0x7f09b8159a90 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09b8159440 0x7f09b80b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 --2- 172.21.15.67:0/3498571972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09b80713e0 0x7f09b814e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 >> 172.21.15.67:0/3498571972 conn(0x7f09b80b1740 msgr2=0x7f09b8158440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 shutdown_connections 2024-04-17T10:29:39.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:39.905+0000 7f09c0600640 1 -- 172.21.15.67:0/3498571972 wait complete. 2024-04-17T10:29:39.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair' 2024-04-17T10:29:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd3b400640 1 Processor -- start 2024-04-17T10:29:40.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd3b400640 1 -- start start 2024-04-17T10:29:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd3b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 0x7fcd3406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd3b400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3406e2f0 con 0x7fcd3406ef10 2024-04-17T10:29:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd3b400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3406d6e0 con 0x7fcd34071a20 2024-04-17T10:29:40.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd3b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340a86c0 con 0x7fcd34145040 2024-04-17T10:29:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd2be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcd3406ef10 0x7fcd34071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35456/0 (socket says 172.21.15.67:35456) 2024-04-17T10:29:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd2be00640 1 -- 172.21.15.67:0/1610376259 learned_addr learned my addr 172.21.15.67:0/1610376259 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd38a00640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 0x7fcd3406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964150991 0 0) 0x7fcd3406e2f0 con 0x7fcd3406ef10 2024-04-17T10:29:40.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd18003530 con 0x7fcd3406ef10 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1257135845 0 0) 0x7fcd18003530 con 0x7fcd3406ef10 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 msgr2=0x7fcd3406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 0x7fcd3406e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 >> v1:172.21.15.86:6789/0 conn(0x7fcd34071a20 legacy=0x7fcd340cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3406d9c0 con 0x7fcd3406ef10 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fcd1c001680 con 0x7fcd3406ef10 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 >> v1:172.21.15.67:6789/0 conn(0x7fcd3406ef10 legacy=0x7fcd34071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 0x7fcd180039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd18003f70 0x7fcd18004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd18004d50 0x7fcd180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1800bb10 con 0x7fcd18004d50 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1800bcc0 con 0x7fcd18003f70 2024-04-17T10:29:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.065+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1800bec0 con 0x7fcd34145040 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd39400640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd18004d50 0x7fcd180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd38a00640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 0x7fcd180039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd2be00640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd18003f70 0x7fcd18004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd39400640 1 -- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 msgr2=0x7fcd180039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd39400640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd34145040 0x7fcd180039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd39400640 1 -- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd18003f70 msgr2=0x7fcd18004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd39400640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd18003f70 0x7fcd18004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd39400640 1 -- 172.21.15.67:0/1610376259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18003530 con 0x7fcd18004d50 2024-04-17T10:29:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd39400640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd18004d50 0x7fcd180051d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c001e40 tx=0x7fcd2c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b54af06dc37f85 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c001a90 con 0x7fcd18004d50 2024-04-17T10:29:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c003b80 con 0x7fcd18004d50 2024-04-17T10:29:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.069+0000 7fcd33e00640 1 -- 172.21.15.67:0/1610376259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c005080 con 0x7fcd18004d50 2024-04-17T10:29:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- 172.21.15.67:0/1610376259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd18004d50 msgr2=0x7fcd180051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 --2- 172.21.15.67:0/1610376259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd18004d50 0x7fcd180051d0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c001e40 tx=0x7fcd2c01c760 comp rx=0 tx=0).stop 2024-04-17T10:29:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd39400640 1 -- 172.21.15.67:0/1610376259 reap_dead start 2024-04-17T10:29:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- 172.21.15.67:0/1610376259 shutdown_connections 2024-04-17T10:29:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- 172.21.15.67:0/1610376259 >> 172.21.15.67:0/1610376259 conn(0x7fcd3406ac90 msgr2=0x7fcd3406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- 172.21.15.67:0/1610376259 shutdown_connections 2024-04-17T10:29:40.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- 172.21.15.67:0/1610376259 wait complete. 2024-04-17T10:29:40.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 Processor -- start 2024-04-17T10:29:40.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- start start 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3415b280 0x7fcd341f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3413cfd0 0x7fcd3413d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd341427e0 0x7fcd34142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3406df80 con 0x7fcd341427e0 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3415bf10 con 0x7fcd3415b280 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.073+0000 7fcd3b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3406d370 con 0x7fcd3413cfd0 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3413cfd0 0x7fcd3413d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3413cfd0 0x7fcd3413d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:45476/0 (socket says 172.21.15.67:45476) 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 -- 172.21.15.67:0/555984948 learned_addr learned my addr 172.21.15.67:0/555984948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:40.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd38a00640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3415b280 0x7fcd341f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 -- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3415b280 msgr2=0x7fcd341f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3415b280 0x7fcd341f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd39400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd341427e0 0x7fcd34142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 -- 172.21.15.67:0/555984948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd341427e0 msgr2=0x7fcd34142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd341427e0 0x7fcd34142c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 -- 172.21.15.67:0/555984948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3413a8a0 con 0x7fcd3413cfd0 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd39400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd341427e0 0x7fcd34142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd2be00640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3413cfd0 0x7fcd3413d450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c007110 tx=0x7fcd1c004f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd31600640 1 -- 172.21.15.67:0/555984948 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c01a070 con 0x7fcd3413cfd0 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd31600640 1 -- 172.21.15.67:0/555984948 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c00f750 con 0x7fcd3413cfd0 2024-04-17T10:29:40.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.077+0000 7fcd31600640 1 -- 172.21.15.67:0/555984948 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c00fb40 con 0x7fcd3413cfd0 2024-04-17T10:29:40.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.081+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3413ab10 con 0x7fcd3413cfd0 2024-04-17T10:29:40.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.081+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3413b0d0 con 0x7fcd3413cfd0 2024-04-17T10:29:40.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.081+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3413b2d0 con 0x7fcd3413cfd0 2024-04-17T10:29:40.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.085+0000 7fcd31600640 1 -- 172.21.15.67:0/555984948 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcd1c01d070 con 0x7fcd3413cfd0 2024-04-17T10:29:40.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.085+0000 7fcd31600640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd10066fb0 0x7fcd10069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.085+0000 7fcd31600640 1 -- 172.21.15.67:0/555984948 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fcd1c084940 con 0x7fcd3413cfd0 2024-04-17T10:29:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.085+0000 7fcd38a00640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd10066fb0 0x7fcd10069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.085+0000 7fcd38a00640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd10066fb0 0x7fcd10069470 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009be0 tx=0x7fcd24009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:40.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.089+0000 7fcd31600640 1 -- 172.21.15.67:0/555984948 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcd3413b2d0 con 0x7fcd3413cfd0 2024-04-17T10:29:40.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.309+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]} v 0) v1 -- 0x7fcd34074e30 con 0x7fcd3413cfd0 2024-04-17T10:29:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.549+0000 7fcd31600640 1 -- 172.21.15.67:0/555984948 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]}]=0 v249) v1 ==== 143+0+0 (secure 0 0 0) 0x7fcd1c055ae0 con 0x7fcd3413cfd0 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd10066fb0 msgr2=0x7fcd10069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd10066fb0 0x7fcd10069470 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009be0 tx=0x7fcd24009190 comp rx=0 tx=0).stop 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3413cfd0 msgr2=0x7fcd3413d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3413cfd0 0x7fcd3413d450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c007110 tx=0x7fcd1c004f90 comp rx=0 tx=0).stop 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 shutdown_connections 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd10066fb0 0x7fcd10069470 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd341427e0 0x7fcd34142c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd3413cfd0 0x7fcd3413d450 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 --2- 172.21.15.67:0/555984948 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd3415b280 0x7fcd341f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 >> 172.21.15.67:0/555984948 conn(0x7fcd3406ac90 msgr2=0x7fcd340cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 shutdown_connections 2024-04-17T10:29:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.553+0000 7fcd3b400640 1 -- 172.21.15.67:0/555984948 wait complete. 2024-04-17T10:29:40.569 INFO:tasks.cephfs_test_runner:test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ok 2024-04-17T10:29:40.570 INFO:tasks.cephfs_test_runner:Starting test: test_stray_evaluation_with_scrub (tasks.cephfs.test_scrub_checks.TestScrubChecks) test that scrub can iterate over ~mdsdir and evaluate strays 2024-04-17T10:29:40.570 INFO:tasks.cephfs_test_runner:test_stray_evaluation_with_scrub (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-04-17T10:29:40.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.717+0000 7fb524e00640 1 Processor -- start 2024-04-17T10:29:40.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb524e00640 1 -- start start 2024-04-17T10:29:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb524e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb52015dd10 0x7fb5200b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb524e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5200b57b0 0x7fb520070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb524e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5200b0cc0 con 0x7fb5200b07b0 2024-04-17T10:29:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb524e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5200b1280 con 0x7fb5200b57b0 2024-04-17T10:29:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb524e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5200b0fa0 con 0x7fb52015dd10 2024-04-17T10:29:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb515400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5200b57b0 0x7fb520070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb515400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb5200b57b0 0x7fb520070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36060/0 (socket says 172.21.15.67:36060) 2024-04-17T10:29:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb515400640 1 -- 172.21.15.67:0/909897885 learned_addr learned my addr 172.21.15.67:0/909897885 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb51de00640 1 --2- 172.21.15.67:0/909897885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb52015dd10 0x7fb5200b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb515400640 1 -- 172.21.15.67:0/909897885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb52015dd10 msgr2=0x7fb5200b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:40.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb515400640 1 --2- 172.21.15.67:0/909897885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb52015dd10 0x7fb5200b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb515400640 1 -- 172.21.15.67:0/909897885 >> v1:172.21.15.86:6789/0 conn(0x7fb5200b07b0 legacy=0x7fb5201618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:40.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb515400640 1 -- 172.21.15.67:0/909897885 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52006bed0 con 0x7fb5200b57b0 2024-04-17T10:29:40.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.721+0000 7fb51de00640 1 --2- 172.21.15.67:0/909897885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb52015dd10 0x7fb5200b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:40.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb515400640 1 --2- 172.21.15.67:0/909897885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5200b57b0 0x7fb520070dc0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb508009870 tx=0x7fb50801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c6f9ee276c131e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb51d400640 1 -- 172.21.15.67:0/909897885 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5080ab600 con 0x7fb5200b57b0 2024-04-17T10:29:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb51d400640 1 -- 172.21.15.67:0/909897885 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5080b0070 con 0x7fb5200b57b0 2024-04-17T10:29:40.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb51d400640 1 -- 172.21.15.67:0/909897885 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5080ab9c0 con 0x7fb5200b57b0 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 -- 172.21.15.67:0/909897885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5200b57b0 msgr2=0x7fb520070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 --2- 172.21.15.67:0/909897885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5200b57b0 0x7fb520070dc0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fb508009870 tx=0x7fb50801c460 comp rx=0 tx=0).stop 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 -- 172.21.15.67:0/909897885 shutdown_connections 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 --2- 172.21.15.67:0/909897885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5200b57b0 0x7fb520070dc0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 --2- 172.21.15.67:0/909897885 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb52015dd10 0x7fb5200b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 -- 172.21.15.67:0/909897885 >> 172.21.15.67:0/909897885 conn(0x7fb5200b34d0 msgr2=0x7fb5200b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 -- 172.21.15.67:0/909897885 shutdown_connections 2024-04-17T10:29:40.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 -- 172.21.15.67:0/909897885 wait complete. 2024-04-17T10:29:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 Processor -- start 2024-04-17T10:29:40.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.725+0000 7fb524e00640 1 -- start start 2024-04-17T10:29:40.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5200b07b0 0x7fb52014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52015dd10 0x7fb520156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5201505f0 0x7fb5200d16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52016ae70 con 0x7fb52015dd10 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb520071710 con 0x7fb5201505f0 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52016aff0 con 0x7fb5200b07b0 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52015dd10 0x7fb520156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52015dd10 0x7fb520156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36066/0 (socket says 172.21.15.67:36066) 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 -- 172.21.15.67:0/2557552689 learned_addr learned my addr 172.21.15.67:0/2557552689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb51de00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5200b07b0 0x7fb52014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb51e800640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5201505f0 0x7fb5200d16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 -- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5200b07b0 msgr2=0x7fb52014e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5200b07b0 0x7fb52014e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 -- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5201505f0 msgr2=0x7fb5200d16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5201505f0 0x7fb5200d16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 -- 172.21.15.67:0/2557552689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb520159600 con 0x7fb52015dd10 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb515400640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52015dd10 0x7fb520156110 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb5080ad040 tx=0x7fb50801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb516a00640 1 -- 172.21.15.67:0/2557552689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5080b4800 con 0x7fb52015dd10 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb516a00640 1 -- 172.21.15.67:0/2557552689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5080abe50 con 0x7fb52015dd10 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb520161940 con 0x7fb52015dd10 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb516a00640 1 -- 172.21.15.67:0/2557552689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5080b4c00 con 0x7fb52015dd10 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52014e860 con 0x7fb52015dd10 2024-04-17T10:29:40.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb520070bb0 con 0x7fb52015dd10 2024-04-17T10:29:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.729+0000 7fb516a00640 1 -- 172.21.15.67:0/2557552689 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb5080aa070 con 0x7fb52015dd10 2024-04-17T10:29:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.733+0000 7fb516a00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4fc066f60 0x7fb4fc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.733+0000 7fb516a00640 1 -- 172.21.15.67:0/2557552689 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb50811f570 con 0x7fb52015dd10 2024-04-17T10:29:40.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.733+0000 7fb51de00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4fc066f60 0x7fb4fc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:40.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.733+0000 7fb51de00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4fc066f60 0x7fb4fc069420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb510007510 tx=0x7fb5100074a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:40.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.737+0000 7fb516a00640 1 -- 172.21.15.67:0/2557552689 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb508016110 con 0x7fb52015dd10 2024-04-17T10:29:40.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.897+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb520070bb0 con 0x7fb52015dd10 2024-04-17T10:29:40.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.897+0000 7fb516a00640 1 -- 172.21.15.67:0/2557552689 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) v1 ==== 74+0+13374 (secure 0 0 0) 0x7fb5080f0690 con 0x7fb52015dd10 2024-04-17T10:29:40.892 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:40.892 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":75,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:39.538401+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/2976334498":"2024-04-18T10:29:36.529550+0000","172.21.15.67:6833/2976334498":"2024-04-18T10:29:36.529550+0000","172.21.15.67:6835/392417778":"2024-04-18T10:29:14.294053+0000","172.21.15.67:6833/392417778":"2024-04-18T10:29:14.294053+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:40.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4fc066f60 msgr2=0x7fb4fc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4fc066f60 0x7fb4fc069420 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fb510007510 tx=0x7fb5100074a0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52015dd10 msgr2=0x7fb520156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52015dd10 0x7fb520156110 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb5080ad040 tx=0x7fb50801aca0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 shutdown_connections 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb4fc066f60 0x7fb4fc069420 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb5201505f0 0x7fb5200d16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb52015dd10 0x7fb520156110 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 --2- 172.21.15.67:0/2557552689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb5200b07b0 0x7fb52014e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 >> 172.21.15.67:0/2557552689 conn(0x7fb5200b34d0 msgr2=0x7fb5200b1d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 shutdown_connections 2024-04-17T10:29:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:40.901+0000 7fb524e00640 1 -- 172.21.15.67:0/2557552689 wait complete. 2024-04-17T10:29:40.914 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:29:40.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:29:41.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7482200640 1 Processor -- start 2024-04-17T10:29:41.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7482200640 1 -- start start 2024-04-17T10:29:41.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7482200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f747c15bd60 0x7f747c099b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7482200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f747c0cfdf0 0x7f747c09a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7482200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c0cca60 con 0x7f747c15b980 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7482200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c0ccd40 con 0x7f747c15bd60 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7482200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c0cd020 con 0x7f747c0cfdf0 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f747b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f747c15b980 0x7f747c0cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46310/0 (socket says 172.21.15.67:46310) 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f747b400640 1 -- 172.21.15.67:0/3613481821 learned_addr learned my addr 172.21.15.67:0/3613481821 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7472a00640 1 --2- 172.21.15.67:0/3613481821 >> v2:172.21.15.67:3300/0 conn(0x7f747c15bd60 0x7f747c099b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f747be00640 1 --2- 172.21.15.67:0/3613481821 >> v2:172.21.15.86:3300/0 conn(0x7f747c0cfdf0 0x7f747c09a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f747aa00640 1 -- 172.21.15.67:0/3613481821 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975844585 0 0) 0x7f747c0cca60 con 0x7f747c15b980 2024-04-17T10:29:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f747aa00640 1 -- 172.21.15.67:0/3613481821 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460003530 con 0x7f747c15b980 2024-04-17T10:29:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7472a00640 1 -- 172.21.15.67:0/3613481821 >> v2:172.21.15.86:3300/0 conn(0x7f747c0cfdf0 msgr2=0x7f747c09a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7472a00640 1 --2- 172.21.15.67:0/3613481821 >> v2:172.21.15.86:3300/0 conn(0x7f747c0cfdf0 0x7f747c09a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7472a00640 1 -- 172.21.15.67:0/3613481821 >> v1:172.21.15.67:6789/0 conn(0x7f747c15b980 legacy=0x7f747c0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.065+0000 7f7472a00640 1 -- 172.21.15.67:0/3613481821 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0cd300 con 0x7f747c15bd60 2024-04-17T10:29:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7472a00640 1 --2- 172.21.15.67:0/3613481821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c099b30 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f747c0b42a0 tx=0x7f746401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=870e6b876b33c1bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f747aa00640 1 -- 172.21.15.67:0/3613481821 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74640b5070 con 0x7f747c15bd60 2024-04-17T10:29:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f747aa00640 1 -- 172.21.15.67:0/3613481821 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74640a8970 con 0x7f747c15bd60 2024-04-17T10:29:41.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f747aa00640 1 -- 172.21.15.67:0/3613481821 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74640a8d40 con 0x7f747c15bd60 2024-04-17T10:29:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 -- 172.21.15.67:0/3613481821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 msgr2=0x7f747c099b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 --2- 172.21.15.67:0/3613481821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c099b30 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f747c0b42a0 tx=0x7f746401ec70 comp rx=0 tx=0).stop 2024-04-17T10:29:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 -- 172.21.15.67:0/3613481821 shutdown_connections 2024-04-17T10:29:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 --2- 172.21.15.67:0/3613481821 >> v2:172.21.15.86:3300/0 conn(0x7f747c0cfdf0 0x7f747c09a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 --2- 172.21.15.67:0/3613481821 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c099b30 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 -- 172.21.15.67:0/3613481821 >> 172.21.15.67:0/3613481821 conn(0x7f747c0b33a0 msgr2=0x7f747c0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 -- 172.21.15.67:0/3613481821 shutdown_connections 2024-04-17T10:29:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.069+0000 7f7482200640 1 -- 172.21.15.67:0/3613481821 wait complete. 2024-04-17T10:29:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 Processor -- start 2024-04-17T10:29:41.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 -- start start 2024-04-17T10:29:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f747c0cfdf0 0x7f747c15a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747c15b980 0x7f747c0caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0cdc80 con 0x7f747c15bd60 2024-04-17T10:29:41.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0cdb00 con 0x7f747c15b980 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0cde00 con 0x7f747c0cfdf0 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c0caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36086/0 (socket says 172.21.15.67:36086) 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 -- 172.21.15.67:0/2623701041 learned_addr learned my addr 172.21.15.67:0/2623701041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747b400640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f747c0cfdf0 0x7f747c15a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7472a00640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747c15b980 0x7f747c0caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 -- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f747c0cfdf0 msgr2=0x7f747c15a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f747c0cfdf0 0x7f747c15a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 -- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747c15b980 msgr2=0x7f747c0caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747c15b980 0x7f747c0caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 -- 172.21.15.67:0/2623701041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c0c8320 con 0x7f747c15bd60 2024-04-17T10:29:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f747be00640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c0caff0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f747400efc0 tx=0x7f747400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7473e00640 1 -- 172.21.15.67:0/2623701041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474018020 con 0x7f747c15bd60 2024-04-17T10:29:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7473e00640 1 -- 172.21.15.67:0/2623701041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7474014480 con 0x7f747c15bd60 2024-04-17T10:29:41.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7473e00640 1 -- 172.21.15.67:0/2623701041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474014870 con 0x7f747c15bd60 2024-04-17T10:29:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c15caf0 con 0x7f747c15bd60 2024-04-17T10:29:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.073+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c15a8d0 con 0x7f747c15bd60 2024-04-17T10:29:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.077+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c04ef30 con 0x7f747c15bd60 2024-04-17T10:29:41.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.077+0000 7f7473e00640 1 -- 172.21.15.67:0/2623701041 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f747401b020 con 0x7f747c15bd60 2024-04-17T10:29:41.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.077+0000 7f7473e00640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7458066f60 0x7f7458069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.077+0000 7f7473e00640 1 -- 172.21.15.67:0/2623701041 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f7474085980 con 0x7f747c15bd60 2024-04-17T10:29:41.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.077+0000 7f747b400640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7458066f60 0x7f7458069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.077+0000 7f747b400640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7458066f60 0x7f7458069420 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f746c007fd0 tx=0x7f746c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:41.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.081+0000 7f7473e00640 1 -- 172.21.15.67:0/2623701041 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7474022070 con 0x7f747c15bd60 2024-04-17T10:29:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.241+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f747c0d0700 con 0x7f747c15bd60 2024-04-17T10:29:41.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.241+0000 7f7473e00640 1 -- 172.21.15.67:0/2623701041 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v75) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7474010040 con 0x7f747c15bd60 2024-04-17T10:29:41.236 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:29:41.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7458066f60 msgr2=0x7f7458069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7458066f60 0x7f7458069420 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f746c007fd0 tx=0x7f746c009040 comp rx=0 tx=0).stop 2024-04-17T10:29:41.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 msgr2=0x7f747c0caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c0caff0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f747400efc0 tx=0x7f747400c490 comp rx=0 tx=0).stop 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 shutdown_connections 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7458066f60 0x7f7458069420 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f747c15bd60 0x7f747c0caff0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f747c15b980 0x7f747c0caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 --2- 172.21.15.67:0/2623701041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f747c0cfdf0 0x7f747c15a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 >> 172.21.15.67:0/2623701041 conn(0x7f747c0b33a0 msgr2=0x7f747c06bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 shutdown_connections 2024-04-17T10:29:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.245+0000 7f7482200640 1 -- 172.21.15.67:0/2623701041 wait complete. 2024-04-17T10:29:41.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub' 2024-04-17T10:29:41.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7ee200640 1 Processor -- start 2024-04-17T10:29:41.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7ee200640 1 -- start start 2024-04-17T10:29:41.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7ee200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb7e815ba60 0x7fb7e80cecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7ee200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e80cd1c0 0x7fb7e80cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7ee200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e80b5b40 con 0x7fb7e80cbf40 2024-04-17T10:29:41.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7ee200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e815be40 con 0x7fb7e815ba60 2024-04-17T10:29:41.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7ee200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80b2bd0 con 0x7fb7e80cd1c0 2024-04-17T10:29:41.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7e7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb7e80cbf40 0x7fb7e80b3990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46312/0 (socket says 172.21.15.67:46312) 2024-04-17T10:29:41.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7e7e00640 1 -- 172.21.15.67:0/3897009170 learned_addr learned my addr 172.21.15.67:0/3897009170 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7dea00640 1 --2- 172.21.15.67:0/3897009170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e80cd1c0 0x7fb7e80cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7e7400640 1 --2- 172.21.15.67:0/3897009170 >> v2:172.21.15.86:3300/0 conn(0x7fb7e815ba60 0x7fb7e80cecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7e6a00640 1 -- 172.21.15.67:0/3897009170 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157702056 0 0) 0x7fb7e80b5b40 con 0x7fb7e80cbf40 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7e6a00640 1 -- 172.21.15.67:0/3897009170 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7cc003530 con 0x7fb7e80cbf40 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7dea00640 1 -- 172.21.15.67:0/3897009170 >> v2:172.21.15.86:3300/0 conn(0x7fb7e815ba60 msgr2=0x7fb7e80cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7dea00640 1 --2- 172.21.15.67:0/3897009170 >> v2:172.21.15.86:3300/0 conn(0x7fb7e815ba60 0x7fb7e80cecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7dea00640 1 -- 172.21.15.67:0/3897009170 >> v1:172.21.15.67:6789/0 conn(0x7fb7e80cbf40 legacy=0x7fb7e80b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7dea00640 1 -- 172.21.15.67:0/3897009170 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e80b2eb0 con 0x7fb7e80cd1c0 2024-04-17T10:29:41.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.405+0000 7fb7e7400640 1 --2- 172.21.15.67:0/3897009170 >> v2:172.21.15.86:3300/0 conn(0x7fb7e815ba60 0x7fb7e80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:41.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7dea00640 1 --2- 172.21.15.67:0/3897009170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e80cd1c0 0x7fb7e80cf230 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00097c0 tx=0x7fb7d001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cdbf63c469736c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7e6a00640 1 -- 172.21.15.67:0/3897009170 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d00a7070 con 0x7fb7e80cd1c0 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7e6a00640 1 -- 172.21.15.67:0/3897009170 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d00b0070 con 0x7fb7e80cd1c0 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7e6a00640 1 -- 172.21.15.67:0/3897009170 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d00ab890 con 0x7fb7e80cd1c0 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 -- 172.21.15.67:0/3897009170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e80cd1c0 msgr2=0x7fb7e80cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 --2- 172.21.15.67:0/3897009170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e80cd1c0 0x7fb7e80cf230 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00097c0 tx=0x7fb7d001c470 comp rx=0 tx=0).stop 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 -- 172.21.15.67:0/3897009170 shutdown_connections 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 --2- 172.21.15.67:0/3897009170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e80cd1c0 0x7fb7e80cf230 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 --2- 172.21.15.67:0/3897009170 >> v2:172.21.15.86:3300/0 conn(0x7fb7e815ba60 0x7fb7e80cecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 -- 172.21.15.67:0/3897009170 >> 172.21.15.67:0/3897009170 conn(0x7fb7e806ac90 msgr2=0x7fb7e806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:41.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 -- 172.21.15.67:0/3897009170 shutdown_connections 2024-04-17T10:29:41.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.409+0000 7fb7ee200640 1 -- 172.21.15.67:0/3897009170 wait complete. 2024-04-17T10:29:41.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 Processor -- start 2024-04-17T10:29:41.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 -- start start 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e80cbf40 0x7fb7e81443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7e80cd1c0 0x7fb7e815a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e815ba60 0x7fb7e8157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80b4a30 con 0x7fb7e80cbf40 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80b48b0 con 0x7fb7e80cd1c0 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e80b4bb0 con 0x7fb7e815ba60 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e80cbf40 0x7fb7e81443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e80cbf40 0x7fb7e81443d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36100/0 (socket says 172.21.15.67:36100) 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 -- 172.21.15.67:0/918232128 learned_addr learned my addr 172.21.15.67:0/918232128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:41.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7e00640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e815ba60 0x7fb7e8157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7dea00640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7e80cd1c0 0x7fb7e815a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 -- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e815ba60 msgr2=0x7fb7e8157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e815ba60 0x7fb7e8157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 -- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7e80cd1c0 msgr2=0x7fb7e815a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7e80cd1c0 0x7fb7e815a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 -- 172.21.15.67:0/918232128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e81540f0 con 0x7fb7e80cbf40 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7e7400640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e80cbf40 0x7fb7e81443d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800ebe0 tx=0x7fb7d800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7dfe00640 1 -- 172.21.15.67:0/918232128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8017070 con 0x7fb7e80cbf40 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7dfe00640 1 -- 172.21.15.67:0/918232128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d800cc30 con 0x7fb7e80cbf40 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e81543d0 con 0x7fb7e80cbf40 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7dfe00640 1 -- 172.21.15.67:0/918232128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8013470 con 0x7fb7e80cbf40 2024-04-17T10:29:41.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.413+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e8154990 con 0x7fb7e80cbf40 2024-04-17T10:29:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.417+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e80730f0 con 0x7fb7e80cbf40 2024-04-17T10:29:41.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.417+0000 7fb7dfe00640 1 -- 172.21.15.67:0/918232128 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb7d801a070 con 0x7fb7e80cbf40 2024-04-17T10:29:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.417+0000 7fb7dfe00640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7c4066f60 0x7fb7c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.417+0000 7fb7dfe00640 1 -- 172.21.15.67:0/918232128 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fb7d8085180 con 0x7fb7e80cbf40 2024-04-17T10:29:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.417+0000 7fb7dea00640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7c4066f60 0x7fb7c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:41.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.417+0000 7fb7dea00640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7c4066f60 0x7fb7c4069420 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00af000 tx=0x7fb7d00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:41.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.421+0000 7fb7dfe00640 1 -- 172.21.15.67:0/918232128 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb7d8056340 con 0x7fb7e80cbf40 2024-04-17T10:29:41.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:41.641+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]} v 0) v1 -- 0x7fb7e8074e30 con 0x7fb7e80cbf40 2024-04-17T10:29:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7dfe00640 1 -- 172.21.15.67:0/918232128 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]}]=0 v251) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb7d8055c40 con 0x7fb7e80cbf40 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7c4066f60 msgr2=0x7fb7c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7c4066f60 0x7fb7c4069420 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fb7d00af000 tx=0x7fb7d00a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e80cbf40 msgr2=0x7fb7e81443d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e80cbf40 0x7fb7e81443d0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800ebe0 tx=0x7fb7d800c520 comp rx=0 tx=0).stop 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 shutdown_connections 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7c4066f60 0x7fb7c4069420 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb7e815ba60 0x7fb7e8157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7e80cd1c0 0x7fb7e815a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 --2- 172.21.15.67:0/918232128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7e80cbf40 0x7fb7e81443d0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 >> 172.21.15.67:0/918232128 conn(0x7fb7e806ac90 msgr2=0x7fb7e80cc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 shutdown_connections 2024-04-17T10:29:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.557+0000 7fb7ee200640 1 -- 172.21.15.67:0/918232128 wait complete. 2024-04-17T10:29:42.576 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:29:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c3a00640 1 Processor -- start 2024-04-17T10:29:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c3a00640 1 -- start start 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c3a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc0700b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c3a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09bc0705f0 0x7f09bc15d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c3a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09bc0d03c0 con 0x7f09bc06d810 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c3a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09bc070a60 con 0x7f09bc0705f0 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c3a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc0d06a0 con 0x7f09bc06f8b0 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09bc0705f0 0x7f09bc15d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f09bc0705f0 0x7f09bc15d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36102/0 (socket says 172.21.15.67:36102) 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c1a00640 1 -- 172.21.15.67:0/408872617 learned_addr learned my addr 172.21.15.67:0/408872617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09bbe00640 1 --2- 172.21.15.67:0/408872617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc0700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c1a00640 1 -- 172.21.15.67:0/408872617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 msgr2=0x7f09bc0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c1a00640 1 --2- 172.21.15.67:0/408872617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc0700b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c1a00640 1 -- 172.21.15.67:0/408872617 >> v1:172.21.15.67:6789/0 conn(0x7f09bc06d810 legacy=0x7f09bc0cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09c1a00640 1 -- 172.21.15.67:0/408872617 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc0d0980 con 0x7f09bc0705f0 2024-04-17T10:29:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09bb400640 1 -- 172.21.15.67:0/408872617 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309606588 0 0) 0x7f09bc0d03c0 con 0x7f09bc06d810 2024-04-17T10:29:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.733+0000 7f09bbe00640 1 --2- 172.21.15.67:0/408872617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc0700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:42.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c1a00640 1 --2- 172.21.15.67:0/408872617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc0705f0 0x7f09bc15d470 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009e50 tx=0x7f09b401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1522696237e0354e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:42.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09bb400640 1 -- 172.21.15.67:0/408872617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b40090d0 con 0x7f09bc0705f0 2024-04-17T10:29:42.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09bb400640 1 -- 172.21.15.67:0/408872617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b40a57f0 con 0x7f09bc0705f0 2024-04-17T10:29:42.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09bb400640 1 -- 172.21.15.67:0/408872617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4009270 con 0x7f09bc0705f0 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 -- 172.21.15.67:0/408872617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc0705f0 msgr2=0x7f09bc15d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 --2- 172.21.15.67:0/408872617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc0705f0 0x7f09bc15d470 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f09b4009e50 tx=0x7f09b401ca30 comp rx=0 tx=0).stop 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 -- 172.21.15.67:0/408872617 shutdown_connections 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 --2- 172.21.15.67:0/408872617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc0705f0 0x7f09bc15d470 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 --2- 172.21.15.67:0/408872617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc0700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 -- 172.21.15.67:0/408872617 >> 172.21.15.67:0/408872617 conn(0x7f09bc0d1800 msgr2=0x7f09bc0d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 -- 172.21.15.67:0/408872617 shutdown_connections 2024-04-17T10:29:42.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 -- 172.21.15.67:0/408872617 wait complete. 2024-04-17T10:29:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 Processor -- start 2024-04-17T10:29:42.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.737+0000 7f09c3a00640 1 -- start start 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc06d810 0x7f09bc154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09bc0705f0 0x7f09bc157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc06bda0 con 0x7f09bc06d810 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc06de00 con 0x7f09bc0705f0 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09bc06bf20 con 0x7f09bc06f8b0 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc06d810 0x7f09bc154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc06d810 0x7f09bc154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36108/0 (socket says 172.21.15.67:36108) 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 -- 172.21.15.67:0/2759305097 learned_addr learned my addr 172.21.15.67:0/2759305097 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1a00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09bc0705f0 0x7f09bc157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09bbe00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 -- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 msgr2=0x7f09bc159510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc159510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 -- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09bc0705f0 msgr2=0x7f09bc157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09bc0705f0 0x7f09bc157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 -- 172.21.15.67:0/2759305097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc158340 con 0x7f09bc06d810 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c1000640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc06d810 0x7f09bc154190 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c4d0 tx=0x7f09b000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09b8c00640 1 -- 172.21.15.67:0/2759305097 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0017070 con 0x7f09bc06d810 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09b8c00640 1 -- 172.21.15.67:0/2759305097 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b0005480 con 0x7f09bc06d810 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09bc1590a0 con 0x7f09bc06d810 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09b8c00640 1 -- 172.21.15.67:0/2759305097 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b0003840 con 0x7f09bc06d810 2024-04-17T10:29:42.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09bc1578e0 con 0x7f09bc06d810 2024-04-17T10:29:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09bc04ef30 con 0x7f09bc06d810 2024-04-17T10:29:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.741+0000 7f09b8c00640 1 -- 172.21.15.67:0/2759305097 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f09b001a070 con 0x7f09bc06d810 2024-04-17T10:29:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.745+0000 7f09b8c00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f099c066f90 0x7f099c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:42.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.745+0000 7f09b8c00640 1 -- 172.21.15.67:0/2759305097 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f09b0084300 con 0x7f09bc06d810 2024-04-17T10:29:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.745+0000 7f09bbe00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f099c066f90 0x7f099c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:42.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.745+0000 7f09bbe00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f099c066f90 0x7f099c069450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f09bc1550a0 tx=0x7f09a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:42.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.749+0000 7f09b8c00640 1 -- 172.21.15.67:0/2759305097 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f09b00554e0 con 0x7f09bc06d810 2024-04-17T10:29:42.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.909+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f09bc071050 con 0x7f09bc06d810 2024-04-17T10:29:42.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.913+0000 7f09b8c00640 1 -- 172.21.15.67:0/2759305097 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f09b00840c0 con 0x7f09bc06d810 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f099c066f90 msgr2=0x7f099c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f099c066f90 0x7f099c069450 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f09bc1550a0 tx=0x7f09a8008040 comp rx=0 tx=0).stop 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc06d810 msgr2=0x7f09bc154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc06d810 0x7f09bc154190 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f09b000c4d0 tx=0x7f09b000c990 comp rx=0 tx=0).stop 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 shutdown_connections 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f099c066f90 0x7f099c069450 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09bc0705f0 0x7f09bc157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09bc06f8b0 0x7f09bc159510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 --2- 172.21.15.67:0/2759305097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09bc06d810 0x7f09bc154190 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 >> 172.21.15.67:0/2759305097 conn(0x7f09bc0d1800 msgr2=0x7f09bc06ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 shutdown_connections 2024-04-17T10:29:42.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:42.917+0000 7f09c3a00640 1 -- 172.21.15.67:0/2759305097 wait complete. 2024-04-17T10:29:42.931 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:29:42.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:29:43.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:29:43.085 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:29:43.085 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:43.085 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:43.094 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:29:43.095 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:29:43.095 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:43.103 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:29:43.104 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:29:43.104 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:43.159 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:29:43.159 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:29:43.159 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:29:43.168 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:29:43.168 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:29:43.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:43.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.345+0000 7fe99ae00640 1 Processor -- start 2024-04-17T10:29:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.345+0000 7fe99ae00640 1 -- start start 2024-04-17T10:29:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe99ae00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9940cf0c0 0x7fe9941441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe99ae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99415ae40 con 0x7fe99406eac0 2024-04-17T10:29:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe99ae00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe994071440 con 0x7fe99406e6e0 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe99ae00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe994071720 con 0x7fe9940cf0c0 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe998e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9940cf0c0 0x7fe9941441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe998e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe9940cf0c0 0x7fe9941441f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36114/0 (socket says 172.21.15.67:36114) 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe998e00640 1 -- 172.21.15.67:0/2070220732 learned_addr learned my addr 172.21.15.67:0/2070220732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe98be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe99406eac0 0x7fe994143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46336/0 (socket says 172.21.15.67:46336) 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe998e00640 1 -- 172.21.15.67:0/2070220732 >> v1:172.21.15.86:6789/0 conn(0x7fe99406e6e0 legacy=0x7fe9940ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe998e00640 1 -- 172.21.15.67:0/2070220732 >> v1:172.21.15.67:6789/0 conn(0x7fe99406eac0 legacy=0x7fe994143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe998e00640 1 -- 172.21.15.67:0/2070220732 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe994071a00 con 0x7fe9940cf0c0 2024-04-17T10:29:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.349+0000 7fe993400640 1 -- 172.21.15.67:0/2070220732 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340125374 0 0) 0x7fe99415ae40 con 0x7fe99406eac0 2024-04-17T10:29:43.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe998e00640 1 --2- 172.21.15.67:0/2070220732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9940cf0c0 0x7fe9941441f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009e50 tx=0x7fe98c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=914dabf73f75bca2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:43.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe993400640 1 -- 172.21.15.67:0/2070220732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c0a57f0 con 0x7fe9940cf0c0 2024-04-17T10:29:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe993400640 1 -- 172.21.15.67:0/2070220732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c0a5990 con 0x7fe9940cf0c0 2024-04-17T10:29:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe993400640 1 -- 172.21.15.67:0/2070220732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c0a5d60 con 0x7fe9940cf0c0 2024-04-17T10:29:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 -- 172.21.15.67:0/2070220732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9940cf0c0 msgr2=0x7fe9941441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 --2- 172.21.15.67:0/2070220732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9940cf0c0 0x7fe9941441f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fe98c009e50 tx=0x7fe98c01c930 comp rx=0 tx=0).stop 2024-04-17T10:29:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 -- 172.21.15.67:0/2070220732 shutdown_connections 2024-04-17T10:29:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 --2- 172.21.15.67:0/2070220732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe9940cf0c0 0x7fe9941441f0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 -- 172.21.15.67:0/2070220732 >> 172.21.15.67:0/2070220732 conn(0x7fe99406ac90 msgr2=0x7fe99406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 -- 172.21.15.67:0/2070220732 shutdown_connections 2024-04-17T10:29:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 -- 172.21.15.67:0/2070220732 wait complete. 2024-04-17T10:29:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 Processor -- start 2024-04-17T10:29:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.353+0000 7fe99ae00640 1 -- start start 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe99406e6e0 0x7fe99413c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe99406eac0 0x7fe9940b0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9940cf0c0 0x7fe9940b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9941680b0 con 0x7fe99406e6e0 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe994167f30 con 0x7fe99406eac0 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe994168230 con 0x7fe9940cf0c0 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe99406e6e0 0x7fe99413c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe99406e6e0 0x7fe99413c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36118/0 (socket says 172.21.15.67:36118) 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 -- 172.21.15.67:0/1897554574 learned_addr learned my addr 172.21.15.67:0/1897554574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe998e00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9940cf0c0 0x7fe9940b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe98be00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe99406eac0 0x7fe9940b0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 -- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9940cf0c0 msgr2=0x7fe9940b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9940cf0c0 0x7fe9940b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 -- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe99406eac0 msgr2=0x7fe9940b0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe99406eac0 0x7fe9940b0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 -- 172.21.15.67:0/1897554574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe994141640 con 0x7fe99406e6e0 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe98be00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe99406eac0 0x7fe9940b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe998e00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9940cf0c0 0x7fe9940b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe993e00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe99406e6e0 0x7fe99413c210 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe98400c450 tx=0x7fe98400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe990c00640 1 -- 172.21.15.67:0/1897554574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984012830 con 0x7fe99406e6e0 2024-04-17T10:29:43.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe990c00640 1 -- 172.21.15.67:0/1897554574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe984017070 con 0x7fe99406e6e0 2024-04-17T10:29:43.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe990c00640 1 -- 172.21.15.67:0/1897554574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984012c30 con 0x7fe99406e6e0 2024-04-17T10:29:43.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe994141920 con 0x7fe99406e6e0 2024-04-17T10:29:43.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe994141ee0 con 0x7fe99406e6e0 2024-04-17T10:29:43.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe994072d50 con 0x7fe99406e6e0 2024-04-17T10:29:43.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.357+0000 7fe990c00640 1 -- 172.21.15.67:0/1897554574 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe98401c460 con 0x7fe99406e6e0 2024-04-17T10:29:43.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.361+0000 7fe990c00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe970066f60 0x7fe970069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.361+0000 7fe990c00640 1 -- 172.21.15.67:0/1897554574 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fe984084c60 con 0x7fe99406e6e0 2024-04-17T10:29:43.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.361+0000 7fe98be00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe970066f60 0x7fe970069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.361+0000 7fe98be00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe970066f60 0x7fe970069420 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe97c007fd0 tx=0x7fe97c007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:43.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.365+0000 7fe990c00640 1 -- 172.21.15.67:0/1897554574 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe984055e20 con 0x7fe99406e6e0 2024-04-17T10:29:43.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.561+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe994142530 con 0x7fe99406e6e0 2024-04-17T10:29:43.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe990c00640 1 -- 172.21.15.67:0/1897554574 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fe984055720 con 0x7fe99406e6e0 2024-04-17T10:29:43.557 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:43.557 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6589,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3163523544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3163523544},{"type":"v1","addr":"172.21.15.86:6835","nonce":3163523544}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6595,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/3682484259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":3682484259},{"type":"v1","addr":"172.21.15.86:6837","nonce":3682484259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6945,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/252088612","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":252088612},{"type":"v1","addr":"172.21.15.67:6834","nonce":252088612}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":7131,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/2028577745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2028577745},{"type":"v1","addr":"172.21.15.67:6835","nonce":2028577745}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-04-17T10:29:43.557 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 49 2024-04-17T10:29:43.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe970066f60 msgr2=0x7fe970069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe970066f60 0x7fe970069420 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe97c007fd0 tx=0x7fe97c007c20 comp rx=0 tx=0).stop 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe99406e6e0 msgr2=0x7fe99413c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe99406e6e0 0x7fe99413c210 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fe98400c450 tx=0x7fe98400c910 comp rx=0 tx=0).stop 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 shutdown_connections 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe970066f60 0x7fe970069420 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe9940cf0c0 0x7fe9940b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe99406eac0 0x7fe9940b0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 --2- 172.21.15.67:0/1897554574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe99406e6e0 0x7fe99413c210 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 >> 172.21.15.67:0/1897554574 conn(0x7fe99406ac90 msgr2=0x7fe99406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 shutdown_connections 2024-04-17T10:29:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.565+0000 7fe99ae00640 1 -- 172.21.15.67:0/1897554574 wait complete. 2024-04-17T10:29:43.579 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:29:43.579 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:29:43.579 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:29:43.580 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:43.580 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:29:43.580 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:29:43.580 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:29:43.580 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:43.580 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:29:43.580 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:29:43.581 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:29:43.581 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:43.581 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:29:43.581 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:29:43.581 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:29:43.581 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:29:43.593 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:29:43.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:29:43.594 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:29:43.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:29:43.597 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:29:43.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:29:43.598 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:29:43.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:29:43.640 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:29:43.641 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:43.643 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:29:43.643 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:43.645 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:29:43.645 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:43.646 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:29:43.647 DEBUG:teuthology.parallel:result is None 2024-04-17T10:29:43.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:29:43.727 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:29:43.728 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:29:43.738 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:29:43.739 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:29:43.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2752a00640 1 Processor -- start 2024-04-17T10:29:43.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2752a00640 1 -- start start 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2752a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f274c06eac0 0x7f274c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2752a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f274c0cf0c0 0x7f274c144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2752a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c144560 con 0x7f274c06e6e0 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2752a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c15ae40 con 0x7f274c06eac0 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2752a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c071440 con 0x7f274c0cf0c0 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2743e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f274c06eac0 0x7f274c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2743e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f274c06eac0 0x7f274c143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36154/0 (socket says 172.21.15.67:36154) 2024-04-17T10:29:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2743e00640 1 -- 172.21.15.67:0/4087763796 learned_addr learned my addr 172.21.15.67:0/4087763796 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:43.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2750a00640 1 --2- 172.21.15.67:0/4087763796 >> v2:172.21.15.86:3300/0 conn(0x7f274c0cf0c0 0x7f274c144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2743e00640 1 -- 172.21.15.67:0/4087763796 >> v2:172.21.15.86:3300/0 conn(0x7f274c0cf0c0 msgr2=0x7f274c144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:43.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2743e00640 1 --2- 172.21.15.67:0/4087763796 >> v2:172.21.15.86:3300/0 conn(0x7f274c0cf0c0 0x7f274c144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2743e00640 1 -- 172.21.15.67:0/4087763796 >> v1:172.21.15.86:6789/0 conn(0x7f274c06e6e0 legacy=0x7f274c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.801+0000 7f2743e00640 1 -- 172.21.15.67:0/4087763796 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c071720 con 0x7f274c06eac0 2024-04-17T10:29:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2743e00640 1 --2- 172.21.15.67:0/4087763796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06eac0 0x7f274c143ae0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f274c0ce3b0 tx=0x7f273401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da7e58afd00e870 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:43.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f274b400640 1 -- 172.21.15.67:0/4087763796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27340ab680 con 0x7f274c06eac0 2024-04-17T10:29:43.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f274b400640 1 -- 172.21.15.67:0/4087763796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27340b0070 con 0x7f274c06eac0 2024-04-17T10:29:43.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f274b400640 1 -- 172.21.15.67:0/4087763796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27340aba40 con 0x7f274c06eac0 2024-04-17T10:29:43.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 -- 172.21.15.67:0/4087763796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06eac0 msgr2=0x7f274c143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 --2- 172.21.15.67:0/4087763796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06eac0 0x7f274c143ae0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f274c0ce3b0 tx=0x7f273401c460 comp rx=0 tx=0).stop 2024-04-17T10:29:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 -- 172.21.15.67:0/4087763796 shutdown_connections 2024-04-17T10:29:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 --2- 172.21.15.67:0/4087763796 >> v2:172.21.15.86:3300/0 conn(0x7f274c0cf0c0 0x7f274c144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 --2- 172.21.15.67:0/4087763796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06eac0 0x7f274c143ae0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 -- 172.21.15.67:0/4087763796 >> 172.21.15.67:0/4087763796 conn(0x7f274c06ac90 msgr2=0x7f274c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 -- 172.21.15.67:0/4087763796 shutdown_connections 2024-04-17T10:29:43.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.805+0000 7f2752a00640 1 -- 172.21.15.67:0/4087763796 wait complete. 2024-04-17T10:29:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 Processor -- start 2024-04-17T10:29:43.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 -- start start 2024-04-17T10:29:43.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06e6e0 0x7f274c159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274c06eac0 0x7f274c13a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f274c0cf0c0 0x7f274c1426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0b3a40 con 0x7f274c06e6e0 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0b38c0 con 0x7f274c06eac0 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0b3bc0 con 0x7f274c0cf0c0 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06e6e0 0x7f274c159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06e6e0 0x7f274c159b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36160/0 (socket says 172.21.15.67:36160) 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 -- 172.21.15.67:0/3559095781 learned_addr learned my addr 172.21.15.67:0/3559095781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2750a00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f274c0cf0c0 0x7f274c1426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2743e00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274c06eac0 0x7f274c13a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 -- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f274c0cf0c0 msgr2=0x7f274c1426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f274c0cf0c0 0x7f274c1426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 -- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274c06eac0 msgr2=0x7f274c13a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:43.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274c06eac0 0x7f274c13a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 -- 172.21.15.67:0/3559095781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c1fa320 con 0x7f274c06e6e0 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f274be00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06e6e0 0x7f274c159b00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f273c00c420 tx=0x7f273c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2748c00640 1 -- 172.21.15.67:0/3559095781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c0052d0 con 0x7f274c06e6e0 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2748c00640 1 -- 172.21.15.67:0/3559095781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c005470 con 0x7f274c06e6e0 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2748c00640 1 -- 172.21.15.67:0/3559095781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c007260 con 0x7f274c06e6e0 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274c1fa5c0 con 0x7f274c06e6e0 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c1fab80 con 0x7f274c06e6e0 2024-04-17T10:29:43.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.809+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274c072d50 con 0x7f274c06e6e0 2024-04-17T10:29:43.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.813+0000 7f2748c00640 1 -- 172.21.15.67:0/3559095781 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f273c007400 con 0x7f274c06e6e0 2024-04-17T10:29:43.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.813+0000 7f2748c00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2728066f60 0x7f2728069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:43.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.813+0000 7f2748c00640 1 -- 172.21.15.67:0/3559095781 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f273c0843b0 con 0x7f274c06e6e0 2024-04-17T10:29:43.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.813+0000 7f2743e00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2728066f60 0x7f2728069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:43.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.813+0000 7f2743e00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2728066f60 0x7f2728069420 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f27340af000 tx=0x7f27340a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:43.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.817+0000 7f2748c00640 1 -- 172.21.15.67:0/3559095781 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f273c054e70 con 0x7f274c06e6e0 2024-04-17T10:29:43.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:43.989+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f274c074e30 con 0x7f274c06e6e0 2024-04-17T10:29:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.569+0000 7f2748c00640 1 -- 172.21.15.67:0/3559095781 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v77) v1 ==== 97+0+0 (secure 0 0 0) 0x7f273c058f10 con 0x7f274c06e6e0 2024-04-17T10:29:44.563 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2728066f60 msgr2=0x7f2728069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2728066f60 0x7f2728069420 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f27340af000 tx=0x7f27340a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06e6e0 msgr2=0x7f274c159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06e6e0 0x7f274c159b00 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f273c00c420 tx=0x7f273c00c8e0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 shutdown_connections 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2728066f60 0x7f2728069420 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f274c0cf0c0 0x7f274c1426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f274c06eac0 0x7f274c13a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 --2- 172.21.15.67:0/3559095781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f274c06e6e0 0x7f274c159b00 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 >> 172.21.15.67:0/3559095781 conn(0x7f274c06ac90 msgr2=0x7f274c0afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 shutdown_connections 2024-04-17T10:29:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.573+0000 7f2752a00640 1 -- 172.21.15.67:0/3559095781 wait complete. 2024-04-17T10:29:44.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:29:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ae5600640 1 Processor -- start 2024-04-17T10:29:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ae5600640 1 -- start start 2024-04-17T10:29:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ae5600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ae00cf0c0 0x7f5ae01441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ae5600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae015ae40 con 0x7f5ae006e6e0 2024-04-17T10:29:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ae5600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae0071440 con 0x7f5ae006eac0 2024-04-17T10:29:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ae5600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae0071720 con 0x7f5ae00cf0c0 2024-04-17T10:29:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5adea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ae006e6e0 0x7f5ae00ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46354/0 (socket says 172.21.15.67:46354) 2024-04-17T10:29:44.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5adea00640 1 -- 172.21.15.67:0/2172413290 learned_addr learned my addr 172.21.15.67:0/2172413290 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5adf400640 1 --2- 172.21.15.67:0/2172413290 >> v2:172.21.15.86:3300/0 conn(0x7f5ae00cf0c0 0x7f5ae01441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362112875 0 0) 0x7f5ae015ae40 con 0x7f5ae006e6e0 2024-04-17T10:29:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac4003530 con 0x7f5ae006e6e0 2024-04-17T10:29:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555898846 0 0) 0x7f5ac4003530 con 0x7f5ae006e6e0 2024-04-17T10:29:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 >> v2:172.21.15.86:3300/0 conn(0x7f5ae00cf0c0 msgr2=0x7f5ae01441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:44.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 --2- 172.21.15.67:0/2172413290 >> v2:172.21.15.86:3300/0 conn(0x7f5ae00cf0c0 0x7f5ae01441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 >> v1:172.21.15.86:6789/0 conn(0x7f5ae006eac0 legacy=0x7f5ae0143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae0071a00 con 0x7f5ae006e6e0 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f5ad0001680 con 0x7f5ae006e6e0 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 >> v1:172.21.15.67:6789/0 conn(0x7f5ae006e6e0 legacy=0x7f5ae00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40039c0 0x7f5ac4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac4004810 0x7f5ac4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40051d0 0x7f5ac400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0071720 con 0x7f5ac40051d0 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac400c170 con 0x7f5ac4004810 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac400c370 con 0x7f5ac40039c0 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5adf400640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40051d0 0x7f5ac400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5adea00640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40039c0 0x7f5ac4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.745+0000 7f5ad7e00640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac4004810 0x7f5ac4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:44.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5adf400640 1 -- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40039c0 msgr2=0x7f5ac4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5adf400640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40039c0 0x7f5ac4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5adf400640 1 -- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac4004810 msgr2=0x7f5ac4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5adf400640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ac4004810 0x7f5ac4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5adf400640 1 -- 172.21.15.67:0/2172413290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac4003530 con 0x7f5ac40051d0 2024-04-17T10:29:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5adf400640 1 -- 172.21.15.67:0/2172413290 reap_dead start 2024-04-17T10:29:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5adf400640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40051d0 0x7f5ac400b9d0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009fc0 tx=0x7f5ad801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533957d0e684b161 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8009c10 con 0x7f5ac40051d0 2024-04-17T10:29:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad80a5bc0 con 0x7f5ac40051d0 2024-04-17T10:29:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ade000640 1 -- 172.21.15.67:0/2172413290 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad80039a0 con 0x7f5ac40051d0 2024-04-17T10:29:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ae5600640 1 -- 172.21.15.67:0/2172413290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40051d0 msgr2=0x7f5ac400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ae5600640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40051d0 0x7f5ac400b9d0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5ad8009fc0 tx=0x7f5ad801ccc0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ae5600640 1 -- 172.21.15.67:0/2172413290 shutdown_connections 2024-04-17T10:29:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ae5600640 1 --2- 172.21.15.67:0/2172413290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ac40051d0 0x7f5ac400b9d0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ae5600640 1 -- 172.21.15.67:0/2172413290 >> 172.21.15.67:0/2172413290 conn(0x7f5ae006ac90 msgr2=0x7f5ae006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ae5600640 1 -- 172.21.15.67:0/2172413290 shutdown_connections 2024-04-17T10:29:44.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.749+0000 7f5ae5600640 1 -- 172.21.15.67:0/2172413290 wait complete. 2024-04-17T10:29:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 Processor -- start 2024-04-17T10:29:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 -- start start 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40051d0 0x7f5ae00c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ae01398c0 0x7f5ae0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae00c7e60 0x7f5ae00c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae00ce9b0 con 0x7f5ae00c7e60 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae0071fc0 con 0x7f5ae01398c0 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae00ceb30 con 0x7f5ac40051d0 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae00c7e60 0x7f5ae00c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae00c7e60 0x7f5ae00c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36174/0 (socket says 172.21.15.67:36174) 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 -- 172.21.15.67:0/3429837026 learned_addr learned my addr 172.21.15.67:0/3429837026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 -- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40051d0 msgr2=0x7f5ae00c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40051d0 0x7f5ae00c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 -- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ae01398c0 msgr2=0x7f5ae0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ae01398c0 0x7f5ae0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 -- 172.21.15.67:0/3429837026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae00ca640 con 0x7f5ae00c7e60 2024-04-17T10:29:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5adf400640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae00c7e60 0x7f5ae00c8310 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80986e0 tx=0x7f5ad8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ad6a00640 1 -- 172.21.15.67:0/3429837026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8003e60 con 0x7f5ae00c7e60 2024-04-17T10:29:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ad6a00640 1 -- 172.21.15.67:0/3429837026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad80be070 con 0x7f5ae00c7e60 2024-04-17T10:29:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ad6a00640 1 -- 172.21.15.67:0/3429837026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad8004680 con 0x7f5ae00c7e60 2024-04-17T10:29:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae00ca920 con 0x7f5ae00c7e60 2024-04-17T10:29:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.753+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae00caee0 con 0x7f5ae00c7e60 2024-04-17T10:29:44.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.757+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae0072d50 con 0x7f5ae00c7e60 2024-04-17T10:29:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.757+0000 7f5ad6a00640 1 -- 172.21.15.67:0/3429837026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5ad80b8470 con 0x7f5ae00c7e60 2024-04-17T10:29:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.757+0000 7f5ad6a00640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5abc066fe0 0x7f5abc0694a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:44.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.757+0000 7f5ad6a00640 1 -- 172.21.15.67:0/3429837026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5ad8120670 con 0x7f5ae00c7e60 2024-04-17T10:29:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.757+0000 7f5adea00640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5abc066fe0 0x7f5abc0694a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:44.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.761+0000 7f5adea00640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5abc066fe0 0x7f5abc0694a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00050c0 tx=0x7f5ad0000ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:44.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.761+0000 7f5ad6a00640 1 -- 172.21.15.67:0/3429837026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5ad80169d0 con 0x7f5ae00c7e60 2024-04-17T10:29:44.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5ae0074e30 con 0x7f5ae00c7e60 2024-04-17T10:29:44.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ad6a00640 1 -- 172.21.15.67:0/3429837026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f5ad80f1920 con 0x7f5ae00c7e60 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.978 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:29:44.979 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:29:44.980 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.981 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.982 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.982 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.982 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.983 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.984 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:29:44.984 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:29:44.984 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.984 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.985 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.986 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.987 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:29:44.988 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:29:44.988 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.988 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.988 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.988 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.989 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.990 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.991 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.992 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.993 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:29:44.994 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.995 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:29:44.996 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:29:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5abc066fe0 msgr2=0x7f5abc0694a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5abc066fe0 0x7f5abc0694a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5ad00050c0 tx=0x7f5ad0000ee0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae00c7e60 msgr2=0x7f5ae00c8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae00c7e60 0x7f5ae00c8310 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f5ad80986e0 tx=0x7f5ad8009750 comp rx=0 tx=0).stop 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 shutdown_connections 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5abc066fe0 0x7f5abc0694a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ae00c7e60 0x7f5ae00c8310 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ae01398c0 0x7f5ae0158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 --2- 172.21.15.67:0/3429837026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ac40051d0 0x7f5ae00c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 >> 172.21.15.67:0/3429837026 conn(0x7f5ae006ac90 msgr2=0x7f5ae00b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 shutdown_connections 2024-04-17T10:29:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:44.985+0000 7f5ae5600640 1 -- 172.21.15.67:0/3429837026 wait complete. 2024-04-17T10:29:45.000 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.145+0000 7fa4b8c00640 1 Processor -- start 2024-04-17T10:29:45.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b8c00640 1 -- start start 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b8c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4b4145040 0x7fa4b406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b8c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b4071a20 0x7fa4b40cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b8c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b406e2f0 con 0x7fa4b406ef10 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b8c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4b40a86c0 con 0x7fa4b4145040 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b8c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b415bf10 con 0x7fa4b4071a20 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b1e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4b4145040 0x7fa4b406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b2800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b4071a20 0x7fa4b40cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4abe00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa4b406ef10 0x7fa4b4071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:35060/0 (socket says 172.21.15.67:35060) 2024-04-17T10:29:45.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b2800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b4071a20 0x7fa4b40cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:45472/0 (socket says 172.21.15.67:45472) 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b1e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa4b4145040 0x7fa4b406e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:40764/0 (socket says 172.21.15.67:40764) 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4abe00640 1 -- 172.21.15.67:0/1754965887 learned_addr learned my addr 172.21.15.67:0/1754965887 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b1400640 1 -- 172.21.15.67:0/1754965887 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728635117 0 0) 0x7fa4b406e2f0 con 0x7fa4b406ef10 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b2800640 1 -- 172.21.15.67:0/1754965887 >> v2:172.21.15.86:3300/0 conn(0x7fa4b4145040 msgr2=0x7fa4b406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b2800640 1 --2- 172.21.15.67:0/1754965887 >> v2:172.21.15.86:3300/0 conn(0x7fa4b4145040 0x7fa4b406e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b2800640 1 -- 172.21.15.67:0/1754965887 >> v1:172.21.15.86:6789/0 conn(0x7fa4b406ef10 legacy=0x7fa4b4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b2800640 1 -- 172.21.15.67:0/1754965887 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b406d6e0 con 0x7fa4b4071a20 2024-04-17T10:29:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.149+0000 7fa4b1e00640 1 --2- 172.21.15.67:0/1754965887 >> v2:172.21.15.86:3300/0 conn(0x7fa4b4145040 0x7fa4b406e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b2800640 1 --2- 172.21.15.67:0/1754965887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b4071a20 0x7fa4b40cc300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009d90 tx=0x7fa4ac01f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f12f1c67532af5f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b1400640 1 -- 172.21.15.67:0/1754965887 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0b4070 con 0x7fa4b4071a20 2024-04-17T10:29:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b1400640 1 -- 172.21.15.67:0/1754965887 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac0af600 con 0x7fa4b4071a20 2024-04-17T10:29:45.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b1400640 1 -- 172.21.15.67:0/1754965887 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0af9d0 con 0x7fa4b4071a20 2024-04-17T10:29:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 -- 172.21.15.67:0/1754965887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b4071a20 msgr2=0x7fa4b40cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/1754965887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b4071a20 0x7fa4b40cc300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac009d90 tx=0x7fa4ac01f150 comp rx=0 tx=0).stop 2024-04-17T10:29:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 -- 172.21.15.67:0/1754965887 shutdown_connections 2024-04-17T10:29:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/1754965887 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b4071a20 0x7fa4b40cc300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/1754965887 >> v2:172.21.15.86:3300/0 conn(0x7fa4b4145040 0x7fa4b406e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 -- 172.21.15.67:0/1754965887 >> 172.21.15.67:0/1754965887 conn(0x7fa4b406ac90 msgr2=0x7fa4b406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:45.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 -- 172.21.15.67:0/1754965887 shutdown_connections 2024-04-17T10:29:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 -- 172.21.15.67:0/1754965887 wait complete. 2024-04-17T10:29:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.153+0000 7fa4b8c00640 1 Processor -- start 2024-04-17T10:29:45.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 -- start start 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b406ef10 0x7fa4b40acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b4071a20 0x7fa4b415b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b4145040 0x7fa4b40b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b406df80 con 0x7fa4b4145040 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b406dca0 con 0x7fa4b4071a20 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b406d370 con 0x7fa4b406ef10 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b4145040 0x7fa4b40b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b4145040 0x7fa4b40b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36176/0 (socket says 172.21.15.67:36176) 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 -- 172.21.15.67:0/4148667939 learned_addr learned my addr 172.21.15.67:0/4148667939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:45.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4abe00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b4071a20 0x7fa4b415b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b1e00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b406ef10 0x7fa4b40acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 -- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b406ef10 msgr2=0x7fa4b40acc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b406ef10 0x7fa4b40acc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 -- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b4071a20 msgr2=0x7fa4b415b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b4071a20 0x7fa4b415b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 -- 172.21.15.67:0/4148667939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b4142710 con 0x7fa4b4145040 2024-04-17T10:29:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b1e00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b406ef10 0x7fa4b40acc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b2800640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b4145040 0x7fa4b40b0850 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0b2040 tx=0x7fa4ac01bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4aa000640 1 -- 172.21.15.67:0/4148667939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0b4040 con 0x7fa4b4145040 2024-04-17T10:29:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4aa000640 1 -- 172.21.15.67:0/4148667939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac0c1070 con 0x7fa4b4145040 2024-04-17T10:29:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4aa000640 1 -- 172.21.15.67:0/4148667939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac009450 con 0x7fa4b4145040 2024-04-17T10:29:45.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b40ad240 con 0x7fa4b4145040 2024-04-17T10:29:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b40ad800 con 0x7fa4b4145040 2024-04-17T10:29:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.157+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b41428e0 con 0x7fa4b4145040 2024-04-17T10:29:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.161+0000 7fa4aa000640 1 -- 172.21.15.67:0/4148667939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa4ac0ba6b0 con 0x7fa4b4145040 2024-04-17T10:29:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.161+0000 7fa4aa000640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa490066f60 0x7fa490069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.161+0000 7fa4aa000640 1 -- 172.21.15.67:0/4148667939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa4ac1237d0 con 0x7fa4b4145040 2024-04-17T10:29:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.161+0000 7fa4b1e00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa490066f60 0x7fa490069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.161+0000 7fa4b1e00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa490066f60 0x7fa490069420 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009b10 tx=0x7fa4a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.165+0000 7fa4aa000640 1 -- 172.21.15.67:0/4148667939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa4ac018cb0 con 0x7fa4b4145040 2024-04-17T10:29:45.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4b4074e30 con 0x7fa4b4145040 2024-04-17T10:29:45.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4aa000640 1 -- 172.21.15.67:0/4148667939 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+13117 (secure 0 0 0) 0x7fa4ac0f4900 con 0x7fa4b4145040 2024-04-17T10:29:45.322 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:45.323 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":79,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:44.581046+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa490066f60 msgr2=0x7fa490069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa490066f60 0x7fa490069420 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009b10 tx=0x7fa4a4008040 comp rx=0 tx=0).stop 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b4145040 msgr2=0x7fa4b40b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b4145040 0x7fa4b40b0850 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0b2040 tx=0x7fa4ac01bef0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 shutdown_connections 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa490066f60 0x7fa490069420 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4b4145040 0x7fa4b40b0850 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4b4071a20 0x7fa4b415b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 --2- 172.21.15.67:0/4148667939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4b406ef10 0x7fa4b40acc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.329+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 >> 172.21.15.67:0/4148667939 conn(0x7fa4b406ac90 msgr2=0x7fa4b40ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:45.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.333+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 shutdown_connections 2024-04-17T10:29:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.333+0000 7fa4b8c00640 1 -- 172.21.15.67:0/4148667939 wait complete. 2024-04-17T10:29:45.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:29:45.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:29:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.493+0000 7ff9bb600640 1 Processor -- start 2024-04-17T10:29:45.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.493+0000 7ff9bb600640 1 -- start start 2024-04-17T10:29:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9bb600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b40cf020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9bb600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b415a9c0 con 0x7ff9b4071ed0 2024-04-17T10:29:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9bb600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b415ac40 con 0x7ff9b40722b0 2024-04-17T10:29:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9bb600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b40ccbf0 con 0x7ff9b415a560 2024-04-17T10:29:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b8c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff9b4071ed0 0x7ff9b40aeda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46366/0 (socket says 172.21.15.67:46366) 2024-04-17T10:29:45.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b8c00640 1 -- 172.21.15.67:0/3416594379 learned_addr learned my addr 172.21.15.67:0/3416594379 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:45.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b9600640 1 --2- 172.21.15.67:0/3416594379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b40cf020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b3400640 1 -- 172.21.15.67:0/3416594379 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077622701 0 0) 0x7ff9a8001120 con 0x7ff9b4071ed0 2024-04-17T10:29:45.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b3400640 1 -- 172.21.15.67:0/3416594379 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff99c003530 con 0x7ff9b4071ed0 2024-04-17T10:29:45.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b3400640 1 -- 172.21.15.67:0/3416594379 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168936458 0 0) 0x7ff9b415ac40 con 0x7ff9b40722b0 2024-04-17T10:29:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b3400640 1 -- 172.21.15.67:0/3416594379 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a8001120 con 0x7ff9b40722b0 2024-04-17T10:29:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b9600640 1 -- 172.21.15.67:0/3416594379 >> v1:172.21.15.86:6789/0 conn(0x7ff9b40722b0 legacy=0x7ff9b415e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b9600640 1 -- 172.21.15.67:0/3416594379 >> v1:172.21.15.67:6789/0 conn(0x7ff9b4071ed0 legacy=0x7ff9b40aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.497+0000 7ff9b9600640 1 -- 172.21.15.67:0/3416594379 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b415af20 con 0x7ff9b415a560 2024-04-17T10:29:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9b9600640 1 --2- 172.21.15.67:0/3416594379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b40cf020 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00aed0 tx=0x7ff9ac01dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7419f315d6e645b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9b3400640 1 -- 172.21.15.67:0/3416594379 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0a66f0 con 0x7ff9b415a560 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9b3400640 1 -- 172.21.15.67:0/3416594379 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ac0a6890 con 0x7ff9b415a560 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9b3400640 1 -- 172.21.15.67:0/3416594379 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0a6c60 con 0x7ff9b415a560 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9bb600640 1 -- 172.21.15.67:0/3416594379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 msgr2=0x7ff9b40cf020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9bb600640 1 --2- 172.21.15.67:0/3416594379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b40cf020 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00aed0 tx=0x7ff9ac01dab0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9bb600640 1 -- 172.21.15.67:0/3416594379 shutdown_connections 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9bb600640 1 --2- 172.21.15.67:0/3416594379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b40cf020 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9bb600640 1 -- 172.21.15.67:0/3416594379 >> 172.21.15.67:0/3416594379 conn(0x7ff9b40b1740 msgr2=0x7ff9b40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9bb600640 1 -- 172.21.15.67:0/3416594379 shutdown_connections 2024-04-17T10:29:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.501+0000 7ff9bb600640 1 -- 172.21.15.67:0/3416594379 wait complete. 2024-04-17T10:29:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 Processor -- start 2024-04-17T10:29:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 -- start start 2024-04-17T10:29:45.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9b4071ed0 0x7ff9b41f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9b40722b0 0x7ff9b41f95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b41508d0 con 0x7ff9b4071ed0 2024-04-17T10:29:45.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b415e920 con 0x7ff9b40722b0 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b4150a50 con 0x7ff9b415a560 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9b4071ed0 0x7ff9b41f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9b4071ed0 0x7ff9b41f79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36186/0 (socket says 172.21.15.67:36186) 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 -- 172.21.15.67:0/1271828704 learned_addr learned my addr 172.21.15.67:0/1271828704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b9600640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b3e00640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9b40722b0 0x7ff9b41f95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 -- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 msgr2=0x7ff9b41598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b41598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 -- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9b40722b0 msgr2=0x7ff9b41f95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9b40722b0 0x7ff9b41f95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 -- 172.21.15.67:0/1271828704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b40cd030 con 0x7ff9b4071ed0 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b8c00640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9b4071ed0 0x7ff9b41f79a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c6c0 tx=0x7ff9a800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b0c00640 1 -- 172.21.15.67:0/1271828704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8017070 con 0x7ff9b4071ed0 2024-04-17T10:29:45.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b0c00640 1 -- 172.21.15.67:0/1271828704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a8005ac0 con 0x7ff9b4071ed0 2024-04-17T10:29:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9b0c00640 1 -- 172.21.15.67:0/1271828704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8007070 con 0x7ff9b4071ed0 2024-04-17T10:29:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b4153ca0 con 0x7ff9b4071ed0 2024-04-17T10:29:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b4151070 con 0x7ff9b4071ed0 2024-04-17T10:29:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.505+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b404ef30 con 0x7ff9b4071ed0 2024-04-17T10:29:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.509+0000 7ff9b0c00640 1 -- 172.21.15.67:0/1271828704 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff9a801a070 con 0x7ff9b4071ed0 2024-04-17T10:29:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.509+0000 7ff9b0c00640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff994066f60 0x7ff994069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.509+0000 7ff9b0c00640 1 -- 172.21.15.67:0/1271828704 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff9a8084370 con 0x7ff9b4071ed0 2024-04-17T10:29:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.509+0000 7ff9b3e00640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff994066f60 0x7ff994069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.509+0000 7ff9b3e00640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff994066f60 0x7ff994069420 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff9b406b210 tx=0x7ff9a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.513+0000 7ff9b0c00640 1 -- 172.21.15.67:0/1271828704 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff9b415a9c0 con 0x7ff9b4071ed0 2024-04-17T10:29:45.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.677+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9b406ef80 con 0x7ff9b4071ed0 2024-04-17T10:29:45.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.677+0000 7ff9b0c00640 1 -- 172.21.15.67:0/1271828704 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff9a8054ee0 con 0x7ff9b4071ed0 2024-04-17T10:29:45.672 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff994066f60 msgr2=0x7ff994069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff994066f60 0x7ff994069420 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff9b406b210 tx=0x7ff9a0009040 comp rx=0 tx=0).stop 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9b4071ed0 msgr2=0x7ff9b41f79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9b4071ed0 0x7ff9b41f79a0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800c6c0 tx=0x7ff9a800cb80 comp rx=0 tx=0).stop 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 shutdown_connections 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff994066f60 0x7ff994069420 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9b415a560 0x7ff9b41598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9b40722b0 0x7ff9b41f95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 --2- 172.21.15.67:0/1271828704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9b4071ed0 0x7ff9b41f79a0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 >> 172.21.15.67:0/1271828704 conn(0x7ff9b40b1740 msgr2=0x7ff9b40b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 shutdown_connections 2024-04-17T10:29:45.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.681+0000 7ff9bb600640 1 -- 172.21.15.67:0/1271828704 wait complete. 2024-04-17T10:29:45.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:29:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f051b400640 1 Processor -- start 2024-04-17T10:29:45.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f051b400640 1 -- start start 2024-04-17T10:29:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f051b400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0514071840 0x7f0514166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f051b400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05140b6640 con 0x7f05141446c0 2024-04-17T10:29:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f051b400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05141456d0 con 0x7f0514144aa0 2024-04-17T10:29:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f051b400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05140b51a0 con 0x7f0514071840 2024-04-17T10:29:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0518a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f05141446c0 0x7f0514144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46370/0 (socket says 172.21.15.67:46370) 2024-04-17T10:29:45.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0518a00640 1 -- 172.21.15.67:0/316733548 learned_addr learned my addr 172.21.15.67:0/316733548 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0519400640 1 --2- 172.21.15.67:0/316733548 >> v2:172.21.15.86:3300/0 conn(0x7f0514071840 0x7f0514166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258553531 0 0) 0x7f05140b6640 con 0x7f05141446c0 2024-04-17T10:29:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc003530 con 0x7f05141446c0 2024-04-17T10:29:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1779630015 0 0) 0x7f04fc003530 con 0x7f05141446c0 2024-04-17T10:29:45.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 >> v2:172.21.15.86:3300/0 conn(0x7f0514071840 msgr2=0x7f0514166a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 --2- 172.21.15.67:0/316733548 >> v2:172.21.15.86:3300/0 conn(0x7f0514071840 0x7f0514166a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 >> v1:172.21.15.86:6789/0 conn(0x7f0514144aa0 legacy=0x7f05140d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05140b4870 con 0x7f05141446c0 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f05080016a0 con 0x7f05141446c0 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 >> v1:172.21.15.67:6789/0 conn(0x7f05141446c0 legacy=0x7f0514144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04fc0039c0 0x7f04fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04fc004810 0x7f04fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f04fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140b51a0 con 0x7f04fc0051d0 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc00c070 con 0x7f04fc004810 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc00c270 con 0x7f04fc0039c0 2024-04-17T10:29:45.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0519400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f04fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0513e00640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04fc004810 0x7f04fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0518a00640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04fc0039c0 0x7f04fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.841+0000 7f0519400640 1 -- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04fc0039c0 msgr2=0x7f04fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0519400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f04fc0039c0 0x7f04fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0519400640 1 -- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04fc004810 msgr2=0x7f04fc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0519400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f04fc004810 0x7f04fc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0519400640 1 -- 172.21.15.67:0/316733548 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc003530 con 0x7f04fc0051d0 2024-04-17T10:29:45.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0519400640 1 -- 172.21.15.67:0/316733548 reap_dead start 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0519400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f04fc00b9d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f050c009fc0 tx=0x7f050c01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1bff54a3c051c9e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c009c10 con 0x7f04fc0051d0 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f050c0a5d10 con 0x7f04fc0051d0 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f0513400640 1 -- 172.21.15.67:0/316733548 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050c003be0 con 0x7f04fc0051d0 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f051b400640 1 -- 172.21.15.67:0/316733548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 msgr2=0x7f04fc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f051b400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f04fc00b9d0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f050c009fc0 tx=0x7f050c01ccc0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f051b400640 1 -- 172.21.15.67:0/316733548 shutdown_connections 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f051b400640 1 --2- 172.21.15.67:0/316733548 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f04fc00b9d0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f051b400640 1 -- 172.21.15.67:0/316733548 >> 172.21.15.67:0/316733548 conn(0x7f05140b34d0 msgr2=0x7f05140b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f051b400640 1 -- 172.21.15.67:0/316733548 shutdown_connections 2024-04-17T10:29:45.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.845+0000 7f051b400640 1 -- 172.21.15.67:0/316733548 wait complete. 2024-04-17T10:29:45.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 Processor -- start 2024-04-17T10:29:45.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 -- start start 2024-04-17T10:29:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f05141f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f051406c7c0 0x7f05141f4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f051413d5e0 0x7f05141417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051415c0d0 con 0x7f04fc0051d0 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05140b4e30 con 0x7f051413d5e0 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0514144ea0 con 0x7f051406c7c0 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f05141f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f05141f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36194/0 (socket says 172.21.15.67:36194) 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 -- 172.21.15.67:0/1784367150 learned_addr learned my addr 172.21.15.67:0/1784367150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0519400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f051413d5e0 0x7f05141417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0513e00640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f051406c7c0 0x7f05141f4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 -- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f051406c7c0 msgr2=0x7f05141f4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f051406c7c0 0x7f05141f4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 -- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f051413d5e0 msgr2=0x7f05141417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f051413d5e0 0x7f05141417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 -- 172.21.15.67:0/1784367150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05141f4c50 con 0x7f04fc0051d0 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0519400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f051413d5e0 0x7f05141417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0518a00640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f05141f51b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f050800e780 tx=0x7f050800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0510c00640 1 -- 172.21.15.67:0/1784367150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050800c660 con 0x7f04fc0051d0 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0510c00640 1 -- 172.21.15.67:0/1784367150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0508018070 con 0x7f04fc0051d0 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0514155a20 con 0x7f04fc0051d0 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f0510c00640 1 -- 172.21.15.67:0/1784367150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f050800ca40 con 0x7f04fc0051d0 2024-04-17T10:29:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.849+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0514155fe0 con 0x7f04fc0051d0 2024-04-17T10:29:45.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.853+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514070bb0 con 0x7f04fc0051d0 2024-04-17T10:29:45.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.853+0000 7f0510c00640 1 -- 172.21.15.67:0/1784367150 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f050800cbe0 con 0x7f04fc0051d0 2024-04-17T10:29:45.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.853+0000 7f0510c00640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04f4066f60 0x7f04f4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:45.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.853+0000 7f0510c00640 1 -- 172.21.15.67:0/1784367150 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0508084c20 con 0x7f04fc0051d0 2024-04-17T10:29:45.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.853+0000 7f0513e00640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04f4066f60 0x7f04f4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:45.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.853+0000 7f0513e00640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04f4066f60 0x7f04f4069420 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f051415b210 tx=0x7f0500009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:45.857+0000 7f0510c00640 1 -- 172.21.15.67:0/1784367150 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0508055de0 con 0x7f04fc0051d0 2024-04-17T10:29:46.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0514074e30 con 0x7f04fc0051d0 2024-04-17T10:29:46.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f0510c00640 1 -- 172.21.15.67:0/1784367150 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7f05080556e0 con 0x7f04fc0051d0 2024-04-17T10:29:46.019 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:46.019 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:29:46.019 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:29:46.019 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:29:46.019 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:29:46.019 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:29:46.019 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:29:46.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04f4066f60 msgr2=0x7f04f4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04f4066f60 0x7f04f4069420 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f051415b210 tx=0x7f0500009040 comp rx=0 tx=0).stop 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 msgr2=0x7f05141f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f05141f51b0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f050800e780 tx=0x7f050800ec40 comp rx=0 tx=0).stop 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 shutdown_connections 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04f4066f60 0x7f04f4069420 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f051413d5e0 0x7f05141417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f051406c7c0 0x7f05141f4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 --2- 172.21.15.67:0/1784367150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04fc0051d0 0x7f05141f51b0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 >> 172.21.15.67:0/1784367150 conn(0x7f05140b34d0 msgr2=0x7f05140d08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 shutdown_connections 2024-04-17T10:29:46.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.025+0000 7f051b400640 1 -- 172.21.15.67:0/1784367150 wait complete. 2024-04-17T10:29:46.041 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:29:46.041 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:29:46.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d3200640 1 Processor -- start 2024-04-17T10:29:46.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d3200640 1 -- start start 2024-04-17T10:29:46.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d3200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd4d4099d20 0x7fd4d409a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d3200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d4071500 con 0x7fd4d4070f90 2024-04-17T10:29:46.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d3200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d40cfdd0 con 0x7fd4d4099d20 2024-04-17T10:29:46.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d3200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d40cf8f0 con 0x7fd4d409a680 2024-04-17T10:29:46.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d1400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd4d4070f90 0x7fd4d406f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46378/0 (socket says 172.21.15.67:46378) 2024-04-17T10:29:46.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d1400640 1 -- 172.21.15.67:0/3396659343 learned_addr learned my addr 172.21.15.67:0/3396659343 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d1e00640 1 --2- 172.21.15.67:0/3396659343 >> v2:172.21.15.86:3300/0 conn(0x7fd4d4099d20 0x7fd4d409a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d2800640 1 --2- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d0a00640 1 -- 172.21.15.67:0/3396659343 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201308574 0 0) 0x7fd4d4071500 con 0x7fd4d4070f90 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d0a00640 1 -- 172.21.15.67:0/3396659343 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4b8003530 con 0x7fd4d4070f90 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d1e00640 1 -- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 msgr2=0x7fd4d406fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d1e00640 1 --2- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d406fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d1e00640 1 -- 172.21.15.67:0/3396659343 >> v1:172.21.15.67:6789/0 conn(0x7fd4d4070f90 legacy=0x7fd4d406f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d1e00640 1 -- 172.21.15.67:0/3396659343 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d40cf330 con 0x7fd4d4099d20 2024-04-17T10:29:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.193+0000 7fd4d2800640 1 --2- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d406fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d1e00640 1 --2- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d409a100 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40097c0 tx=0x7fd4c401c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a46b504bf11ecfd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d0a00640 1 -- 172.21.15.67:0/3396659343 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c40aa5f0 con 0x7fd4d4099d20 2024-04-17T10:29:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d0a00640 1 -- 172.21.15.67:0/3396659343 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c40af070 con 0x7fd4d4099d20 2024-04-17T10:29:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d0a00640 1 -- 172.21.15.67:0/3396659343 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c40aa9b0 con 0x7fd4d4099d20 2024-04-17T10:29:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 -- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 msgr2=0x7fd4d409a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 --2- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d409a100 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40097c0 tx=0x7fd4c401c470 comp rx=0 tx=0).stop 2024-04-17T10:29:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 -- 172.21.15.67:0/3396659343 shutdown_connections 2024-04-17T10:29:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 --2- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d406fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 --2- 172.21.15.67:0/3396659343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d409a100 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 -- 172.21.15.67:0/3396659343 >> 172.21.15.67:0/3396659343 conn(0x7fd4d40d16c0 msgr2=0x7fd4d40d1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 -- 172.21.15.67:0/3396659343 shutdown_connections 2024-04-17T10:29:46.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.197+0000 7fd4d3200640 1 -- 172.21.15.67:0/3396659343 wait complete. 2024-04-17T10:29:46.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 Processor -- start 2024-04-17T10:29:46.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 -- start start 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4070f90 0x7fd4d40b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d40b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d41542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d4070340 con 0x7fd4d4070f90 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d40701c0 con 0x7fd4d4099d20 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d40aed20 con 0x7fd4d409a680 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4070f90 0x7fd4d40b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4070f90 0x7fd4d40b4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36208/0 (socket says 172.21.15.67:36208) 2024-04-17T10:29:46.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 -- 172.21.15.67:0/2733040924 learned_addr learned my addr 172.21.15.67:0/2733040924 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d2800640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d41542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1400640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d40b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 -- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 msgr2=0x7fd4d41542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d41542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 -- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 msgr2=0x7fd4d40b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d40b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 -- 172.21.15.67:0/2733040924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d40b4820 con 0x7fd4d4070f90 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1400640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d40b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d1e00640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4070f90 0x7fd4d40b4cc0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40ac040 tx=0x7fd4c401acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4ca000640 1 -- 172.21.15.67:0/2733040924 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4006e20 con 0x7fd4d4070f90 2024-04-17T10:29:46.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4ca000640 1 -- 172.21.15.67:0/2733040924 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d40af520 con 0x7fd4d4070f90 2024-04-17T10:29:46.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d406c150 con 0x7fd4d4070f90 2024-04-17T10:29:46.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4ca000640 1 -- 172.21.15.67:0/2733040924 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c40be670 con 0x7fd4d4070f90 2024-04-17T10:29:46.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.201+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d406c710 con 0x7fd4d4070f90 2024-04-17T10:29:46.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.205+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d4074dc0 con 0x7fd4d4070f90 2024-04-17T10:29:46.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.205+0000 7fd4ca000640 1 -- 172.21.15.67:0/2733040924 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd4c40bc020 con 0x7fd4d4070f90 2024-04-17T10:29:46.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.205+0000 7fd4ca000640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4b4066f90 0x7fd4b4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.205+0000 7fd4ca000640 1 -- 172.21.15.67:0/2733040924 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd4c411ebb0 con 0x7fd4d4070f90 2024-04-17T10:29:46.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.205+0000 7fd4d1400640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4b4066f90 0x7fd4b4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.205+0000 7fd4d1400640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4b4066f90 0x7fd4b4069450 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0005890 tx=0x7fd4c0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:46.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.209+0000 7fd4ca000640 1 -- 172.21.15.67:0/2733040924 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd4c40c5070 con 0x7fd4d4070f90 2024-04-17T10:29:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.369+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd4d40cc270 con 0x7fd4d4070f90 2024-04-17T10:29:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.581+0000 7fd4ca000640 1 -- 172.21.15.67:0/2733040924 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd4c4016090 con 0x7fd4d4070f90 2024-04-17T10:29:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:29:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4b4066f90 msgr2=0x7fd4b4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4b4066f90 0x7fd4b4069450 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0005890 tx=0x7fd4c0009040 comp rx=0 tx=0).stop 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4070f90 msgr2=0x7fd4d40b4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4070f90 0x7fd4d40b4cc0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fd4c40ac040 tx=0x7fd4c401acb0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 shutdown_connections 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4b4066f90 0x7fd4b4069450 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d409a680 0x7fd4d41542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d4099d20 0x7fd4d40b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 --2- 172.21.15.67:0/2733040924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d4070f90 0x7fd4d40b4cc0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 >> 172.21.15.67:0/2733040924 conn(0x7fd4d40d16c0 msgr2=0x7fd4d406f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 shutdown_connections 2024-04-17T10:29:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.585+0000 7fd4d3200640 1 -- 172.21.15.67:0/2733040924 wait complete. 2024-04-17T10:29:46.602 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:29:46.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc139c00640 1 Processor -- start 2024-04-17T10:29:46.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc139c00640 1 -- start start 2024-04-17T10:29:46.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc139c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1340cd030 0x7fc13406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc139c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc13406e7b0 0x7fc1340b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc139c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1340b4990 con 0x7fc1340aef60 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc139c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1340af340 con 0x7fc1340cd030 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc139c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13406ec00 con 0x7fc13406e7b0 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc12be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc1340cd030 0x7fc13406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc132a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc1340aef60 0x7fc1340cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46392/0 (socket says 172.21.15.67:46392) 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.757+0000 7fc132a00640 1 -- 172.21.15.67:0/1850651917 learned_addr learned my addr 172.21.15.67:0/1850651917 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc133400640 1 --2- 172.21.15.67:0/1850651917 >> v2:172.21.15.86:3300/0 conn(0x7fc13406e7b0 0x7fc1340b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc12be00640 1 -- 172.21.15.67:0/1850651917 >> v2:172.21.15.86:3300/0 conn(0x7fc13406e7b0 msgr2=0x7fc1340b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:46.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc12be00640 1 --2- 172.21.15.67:0/1850651917 >> v2:172.21.15.86:3300/0 conn(0x7fc13406e7b0 0x7fc1340b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc12be00640 1 -- 172.21.15.67:0/1850651917 >> v1:172.21.15.67:6789/0 conn(0x7fc1340aef60 legacy=0x7fc1340cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc132000640 1 -- 172.21.15.67:0/1850651917 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618035790 0 0) 0x7fc124001120 con 0x7fc1340aef60 2024-04-17T10:29:46.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc12be00640 1 -- 172.21.15.67:0/1850651917 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13415ae90 con 0x7fc1340cd030 2024-04-17T10:29:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc12be00640 1 --2- 172.21.15.67:0/1850651917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1340cd030 0x7fc13406e270 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc1340b2e60 tx=0x7fc11c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5811126c33c360 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc132000640 1 -- 172.21.15.67:0/1850651917 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0b0070 con 0x7fc1340cd030 2024-04-17T10:29:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc132000640 1 -- 172.21.15.67:0/1850651917 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c0ab600 con 0x7fc1340cd030 2024-04-17T10:29:46.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.761+0000 7fc132000640 1 -- 172.21.15.67:0/1850651917 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0ab9d0 con 0x7fc1340cd030 2024-04-17T10:29:46.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- 172.21.15.67:0/1850651917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1340cd030 msgr2=0x7fc13406e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 --2- 172.21.15.67:0/1850651917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1340cd030 0x7fc13406e270 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fc1340b2e60 tx=0x7fc11c01c3f0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- 172.21.15.67:0/1850651917 shutdown_connections 2024-04-17T10:29:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 --2- 172.21.15.67:0/1850651917 >> v2:172.21.15.86:3300/0 conn(0x7fc13406e7b0 0x7fc1340b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 --2- 172.21.15.67:0/1850651917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc1340cd030 0x7fc13406e270 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- 172.21.15.67:0/1850651917 >> 172.21.15.67:0/1850651917 conn(0x7fc1340b1740 msgr2=0x7fc1340b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- 172.21.15.67:0/1850651917 shutdown_connections 2024-04-17T10:29:46.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- 172.21.15.67:0/1850651917 wait complete. 2024-04-17T10:29:46.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 Processor -- start 2024-04-17T10:29:46.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- start start 2024-04-17T10:29:46.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13406e7b0 0x7fc134154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1340aef60 0x7fc13414bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1340cd030 0x7fc134149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13415bc10 con 0x7fc13406e7b0 2024-04-17T10:29:46.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1340b3e70 con 0x7fc1340cd030 2024-04-17T10:29:46.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc139c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13414c1a0 con 0x7fc1340aef60 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13406e7b0 0x7fc134154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13406e7b0 0x7fc134154e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36238/0 (socket says 172.21.15.67:36238) 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 -- 172.21.15.67:0/2944598393 learned_addr learned my addr 172.21.15.67:0/2944598393 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc12be00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1340aef60 0x7fc13414bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc133400640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1340cd030 0x7fc134149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 -- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1340aef60 msgr2=0x7fc13414bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1340aef60 0x7fc13414bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 -- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1340cd030 msgr2=0x7fc134149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1340cd030 0x7fc134149ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 -- 172.21.15.67:0/2944598393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13406d160 con 0x7fc13406e7b0 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc133400640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1340cd030 0x7fc134149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.765+0000 7fc132a00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13406e7b0 0x7fc134154e50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c7b0 tx=0x7fc12400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc12aa00640 1 -- 172.21.15.67:0/2944598393 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124018070 con 0x7fc13406e7b0 2024-04-17T10:29:46.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc12aa00640 1 -- 172.21.15.67:0/2944598393 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1240072f0 con 0x7fc13406e7b0 2024-04-17T10:29:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc12aa00640 1 -- 172.21.15.67:0/2944598393 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1240047f0 con 0x7fc13406e7b0 2024-04-17T10:29:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1340af560 con 0x7fc13406e7b0 2024-04-17T10:29:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13414c6b0 con 0x7fc13406e7b0 2024-04-17T10:29:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13404ef30 con 0x7fc13406e7b0 2024-04-17T10:29:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc12aa00640 1 -- 172.21.15.67:0/2944598393 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc12400a640 con 0x7fc13406e7b0 2024-04-17T10:29:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc12aa00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc110066f60 0x7fc110069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc12aa00640 1 -- 172.21.15.67:0/2944598393 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fc124084550 con 0x7fc13406e7b0 2024-04-17T10:29:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.769+0000 7fc12be00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc110066f60 0x7fc110069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:46.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.773+0000 7fc12be00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc110066f60 0x7fc110069420 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0ad000 tx=0x7fc11c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:46.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.773+0000 7fc12aa00640 1 -- 172.21.15.67:0/2944598393 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc1340b4990 con 0x7fc13406e7b0 2024-04-17T10:29:46.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:46.937+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc13406c3e0 con 0x7fc13406e7b0 2024-04-17T10:29:47.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc12aa00640 1 -- 172.21.15.67:0/2944598393 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v81) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc124055720 con 0x7fc13406e7b0 2024-04-17T10:29:47.578 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc110066f60 msgr2=0x7fc110069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc110066f60 0x7fc110069420 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fc11c0ad000 tx=0x7fc11c0a6000 comp rx=0 tx=0).stop 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13406e7b0 msgr2=0x7fc134154e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13406e7b0 0x7fc134154e50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c7b0 tx=0x7fc12400cc70 comp rx=0 tx=0).stop 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 shutdown_connections 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc110066f60 0x7fc110069420 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc1340cd030 0x7fc134149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc1340aef60 0x7fc13414bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 --2- 172.21.15.67:0/2944598393 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc13406e7b0 0x7fc134154e50 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.585+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 >> 172.21.15.67:0/2944598393 conn(0x7fc1340b1740 msgr2=0x7fc13415a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.589+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 shutdown_connections 2024-04-17T10:29:47.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.589+0000 7fc139c00640 1 -- 172.21.15.67:0/2944598393 wait complete. 2024-04-17T10:29:47.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:29:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.769+0000 7fa3d0c00640 1 Processor -- start 2024-04-17T10:29:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.769+0000 7fa3d0c00640 1 -- start start 2024-04-17T10:29:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.769+0000 7fa3d0c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3cc159440 0x7fa3cc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.769+0000 7fa3d0c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc0713e0 0x7fa3cc071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:47.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.769+0000 7fa3d0c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3cc15cb30 con 0x7fa3cc15be70 2024-04-17T10:29:47.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.769+0000 7fa3d0c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3cc0a8650 con 0x7fa3cc159440 2024-04-17T10:29:47.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.769+0000 7fa3d0c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc071d80 con 0x7fa3cc0713e0 2024-04-17T10:29:47.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa3cc159440 0x7fa3cc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3ca800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa3cc15be70 0x7fa3cc15c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46400/0 (socket says 172.21.15.67:46400) 2024-04-17T10:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3ca800640 1 -- 172.21.15.67:0/699652899 learned_addr learned my addr 172.21.15.67:0/699652899 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c3e00640 1 --2- 172.21.15.67:0/699652899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc0713e0 0x7fa3cc071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9e00640 1 -- 172.21.15.67:0/699652899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc0713e0 msgr2=0x7fa3cc071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9e00640 1 --2- 172.21.15.67:0/699652899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc0713e0 0x7fa3cc071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9e00640 1 -- 172.21.15.67:0/699652899 >> v1:172.21.15.67:6789/0 conn(0x7fa3cc15be70 legacy=0x7fa3cc15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:47.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9e00640 1 -- 172.21.15.67:0/699652899 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc0cd610 con 0x7fa3cc159440 2024-04-17T10:29:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9e00640 1 --2- 172.21.15.67:0/699652899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc159440 0x7fa3cc0ce0f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0097c0 tx=0x7fa3bc01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b88831c108f010d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9400640 1 -- 172.21.15.67:0/699652899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc0b0070 con 0x7fa3cc159440 2024-04-17T10:29:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9400640 1 -- 172.21.15.67:0/699652899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc0ab600 con 0x7fa3cc159440 2024-04-17T10:29:47.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.773+0000 7fa3c9400640 1 -- 172.21.15.67:0/699652899 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc0ab9d0 con 0x7fa3cc159440 2024-04-17T10:29:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- 172.21.15.67:0/699652899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc159440 msgr2=0x7fa3cc0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/699652899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc159440 0x7fa3cc0ce0f0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0097c0 tx=0x7fa3bc01c330 comp rx=0 tx=0).stop 2024-04-17T10:29:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- 172.21.15.67:0/699652899 shutdown_connections 2024-04-17T10:29:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/699652899 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc0713e0 0x7fa3cc071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/699652899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc159440 0x7fa3cc0ce0f0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- 172.21.15.67:0/699652899 >> 172.21.15.67:0/699652899 conn(0x7fa3cc0b1740 msgr2=0x7fa3cc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:47.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- 172.21.15.67:0/699652899 shutdown_connections 2024-04-17T10:29:47.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- 172.21.15.67:0/699652899 wait complete. 2024-04-17T10:29:47.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 Processor -- start 2024-04-17T10:29:47.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- start start 2024-04-17T10:29:47.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc0713e0 0x7fa3cc1e5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:47.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc159440 0x7fa3cc1e6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3cc15be70 0x7fa3cc1e8db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0cdbd0 con 0x7fa3cc0713e0 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc15ce10 con 0x7fa3cc15be70 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3d0c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc159820 con 0x7fa3cc159440 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc0713e0 0x7fa3cc1e5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc0713e0 0x7fa3cc1e5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36252/0 (socket says 172.21.15.67:36252) 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 -- 172.21.15.67:0/2042347214 learned_addr learned my addr 172.21.15.67:0/2042347214 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 -- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc159440 msgr2=0x7fa3cc1e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc159440 0x7fa3cc1e6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 -- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3cc15be70 msgr2=0x7fa3cc1e8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3cc15be70 0x7fa3cc1e8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.777+0000 7fa3c9e00640 1 -- 172.21.15.67:0/2042347214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc1e7700 con 0x7fa3cc0713e0 2024-04-17T10:29:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c9e00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc0713e0 0x7fa3cc1e5820 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0ae040 tx=0x7fa3bc0190d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:47.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c2000640 1 -- 172.21.15.67:0/2042347214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc0b0040 con 0x7fa3cc0713e0 2024-04-17T10:29:47.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c2000640 1 -- 172.21.15.67:0/2042347214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3bc0b9070 con 0x7fa3cc0713e0 2024-04-17T10:29:47.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c2000640 1 -- 172.21.15.67:0/2042347214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc0aaa30 con 0x7fa3cc0713e0 2024-04-17T10:29:47.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc1e79e0 con 0x7fa3cc0713e0 2024-04-17T10:29:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc1e7fa0 con 0x7fa3cc0713e0 2024-04-17T10:29:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc04ef30 con 0x7fa3cc0713e0 2024-04-17T10:29:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c2000640 1 -- 172.21.15.67:0/2042347214 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa3bc0aabd0 con 0x7fa3cc0713e0 2024-04-17T10:29:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c2000640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3a8066f60 0x7fa3a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c2000640 1 -- 172.21.15.67:0/2042347214 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fa3bc11fde0 con 0x7fa3cc0713e0 2024-04-17T10:29:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.781+0000 7fa3c3e00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3a8066f60 0x7fa3a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.785+0000 7fa3c3e00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3a8066f60 0x7fa3a8069420 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4006aa0 tx=0x7fa3b4006a30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:47.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.785+0000 7fa3c2000640 1 -- 172.21.15.67:0/2042347214 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa3bc015fb0 con 0x7fa3cc0713e0 2024-04-17T10:29:47.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:47.981+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa3cc15bae0 con 0x7fa3cc0713e0 2024-04-17T10:29:48.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.613+0000 7fa3c2000640 1 -- 172.21.15.67:0/2042347214 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v52) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa3bc0bc360 con 0x7fa3cc0713e0 2024-04-17T10:29:48.618 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 11 and data pool 12 2024-04-17T10:29:48.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.629+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3a8066f60 msgr2=0x7fa3a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:48.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.629+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3a8066f60 0x7fa3a8069420 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa3b4006aa0 tx=0x7fa3b4006a30 comp rx=0 tx=0).stop 2024-04-17T10:29:48.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.629+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc0713e0 msgr2=0x7fa3cc1e5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:48.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.629+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc0713e0 0x7fa3cc1e5820 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc0ae040 tx=0x7fa3bc0190d0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.633+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 shutdown_connections 2024-04-17T10:29:48.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.633+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa3a8066f60 0x7fa3a8069420 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.633+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa3cc15be70 0x7fa3cc1e8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.633+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa3cc159440 0x7fa3cc1e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.633+0000 7fa3d0c00640 1 --2- 172.21.15.67:0/2042347214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3cc0713e0 0x7fa3cc1e5820 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.633+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 >> 172.21.15.67:0/2042347214 conn(0x7fa3cc0b1740 msgr2=0x7fa3cc0b02f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:48.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.637+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 shutdown_connections 2024-04-17T10:29:48.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.637+0000 7fa3d0c00640 1 -- 172.21.15.67:0/2042347214 wait complete. 2024-04-17T10:29:48.659 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa920600640 1 Processor -- start 2024-04-17T10:29:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa920600640 1 -- start start 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa920600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa91815dd10 0x7fa9180b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa920600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa9180b57b0 0x7fa918070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa920600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9180b0cc0 con 0x7fa9180b07b0 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa920600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180b1280 con 0x7fa91815dd10 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa920600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9180b0fa0 con 0x7fa9180b57b0 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa91dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa91815dd10 0x7fa9180b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa91dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa91815dd10 0x7fa9180b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36258/0 (socket says 172.21.15.67:36258) 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa91dc00640 1 -- 172.21.15.67:0/2266028524 learned_addr learned my addr 172.21.15.67:0/2266028524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa91dc00640 1 -- 172.21.15.67:0/2266028524 >> v2:172.21.15.86:3300/0 conn(0x7fa9180b57b0 msgr2=0x7fa918070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa91dc00640 1 --2- 172.21.15.67:0/2266028524 >> v2:172.21.15.86:3300/0 conn(0x7fa9180b57b0 0x7fa918070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa91dc00640 1 -- 172.21.15.67:0/2266028524 >> v1:172.21.15.86:6789/0 conn(0x7fa9180b07b0 legacy=0x7fa9181618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:48.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.809+0000 7fa91dc00640 1 -- 172.21.15.67:0/2266028524 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91806bed0 con 0x7fa91815dd10 2024-04-17T10:29:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa91dc00640 1 --2- 172.21.15.67:0/2266028524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa91815dd10 0x7fa9180b5270 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fa918068850 tx=0x7fa90c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19eff473499e2e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa917e00640 1 -- 172.21.15.67:0/2266028524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0ab600 con 0x7fa91815dd10 2024-04-17T10:29:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa917e00640 1 -- 172.21.15.67:0/2266028524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c0b0070 con 0x7fa91815dd10 2024-04-17T10:29:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa917e00640 1 -- 172.21.15.67:0/2266028524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0ab9c0 con 0x7fa91815dd10 2024-04-17T10:29:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 -- 172.21.15.67:0/2266028524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa91815dd10 msgr2=0x7fa9180b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 --2- 172.21.15.67:0/2266028524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa91815dd10 0x7fa9180b5270 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fa918068850 tx=0x7fa90c01c420 comp rx=0 tx=0).stop 2024-04-17T10:29:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 -- 172.21.15.67:0/2266028524 shutdown_connections 2024-04-17T10:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 --2- 172.21.15.67:0/2266028524 >> v2:172.21.15.86:3300/0 conn(0x7fa9180b57b0 0x7fa918070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 --2- 172.21.15.67:0/2266028524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa91815dd10 0x7fa9180b5270 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 -- 172.21.15.67:0/2266028524 >> 172.21.15.67:0/2266028524 conn(0x7fa9180b34d0 msgr2=0x7fa9180b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 -- 172.21.15.67:0/2266028524 shutdown_connections 2024-04-17T10:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.813+0000 7fa920600640 1 -- 172.21.15.67:0/2266028524 wait complete. 2024-04-17T10:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 Processor -- start 2024-04-17T10:29:48.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 -- start start 2024-04-17T10:29:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9180b07b0 0x7fa91815e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa91815dd10 0x7fa9180cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9181489e0 0x7fa918148e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91816ae70 con 0x7fa9180b07b0 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa918071710 con 0x7fa91815dd10 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91816aff0 con 0x7fa9181489e0 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9180b07b0 0x7fa91815e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9180b07b0 0x7fa91815e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36260/0 (socket says 172.21.15.67:36260) 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 -- 172.21.15.67:0/3010087195 learned_addr learned my addr 172.21.15.67:0/3010087195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 -- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9181489e0 msgr2=0x7fa918148e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9181489e0 0x7fa918148e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 -- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa91815dd10 msgr2=0x7fa9180cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa91815dd10 0x7fa9180cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 -- 172.21.15.67:0/3010087195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918160850 con 0x7fa9180b07b0 2024-04-17T10:29:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa91dc00640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9180b07b0 0x7fa91815e340 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0ad040 tx=0x7fa90c01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa915600640 1 -- 172.21.15.67:0/3010087195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0b4800 con 0x7fa9180b07b0 2024-04-17T10:29:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa915600640 1 -- 172.21.15.67:0/3010087195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c0b49c0 con 0x7fa9180b07b0 2024-04-17T10:29:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa915600640 1 -- 172.21.15.67:0/3010087195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c0b4db0 con 0x7fa9180b07b0 2024-04-17T10:29:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa918160af0 con 0x7fa9180b07b0 2024-04-17T10:29:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.817+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9181610b0 con 0x7fa9180b07b0 2024-04-17T10:29:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.821+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa918070bb0 con 0x7fa9180b07b0 2024-04-17T10:29:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.821+0000 7fa915600640 1 -- 172.21.15.67:0/3010087195 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa90c0aa070 con 0x7fa9180b07b0 2024-04-17T10:29:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.821+0000 7fa915600640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8f8066f90 0x7fa8f8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.821+0000 7fa915600640 1 -- 172.21.15.67:0/3010087195 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa90c11ede0 con 0x7fa9180b07b0 2024-04-17T10:29:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.821+0000 7fa91d200640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8f8066f90 0x7fa8f8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.821+0000 7fa91d200640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8f8066f90 0x7fa8f8069450 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa91814a470 tx=0x7fa904006ca0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.825+0000 7fa915600640 1 -- 172.21.15.67:0/3010087195 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa90c0160d0 con 0x7fa9180b07b0 2024-04-17T10:29:48.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa918070bb0 con 0x7fa9180b07b0 2024-04-17T10:29:48.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa915600640 1 -- 172.21.15.67:0/3010087195 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16910 (secure 0 0 0) 0x7fa90c0ef4e0 con 0x7fa9180b07b0 2024-04-17T10:29:48.982 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:48.983 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:47.987037+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:29:46.421958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-17T10:29:46.986879+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8f8066f90 msgr2=0x7fa8f8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8f8066f90 0x7fa8f8069450 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa91814a470 tx=0x7fa904006ca0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9180b07b0 msgr2=0x7fa91815e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9180b07b0 0x7fa91815e340 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0ad040 tx=0x7fa90c01ac60 comp rx=0 tx=0).stop 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 shutdown_connections 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa8f8066f90 0x7fa8f8069450 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9181489e0 0x7fa918148e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa91815dd10 0x7fa9180cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 --2- 172.21.15.67:0/3010087195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa9180b07b0 0x7fa91815e340 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 >> 172.21.15.67:0/3010087195 conn(0x7fa9180b34d0 msgr2=0x7fa918156770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 shutdown_connections 2024-04-17T10:29:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:48.989+0000 7fa920600640 1 -- 172.21.15.67:0/3010087195 wait complete. 2024-04-17T10:29:49.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.149+0000 7fe45ca00640 1 Processor -- start 2024-04-17T10:29:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.149+0000 7fe45ca00640 1 -- start start 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe45ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe458144980 0x7fe45806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe45ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe458145080 0x7fe45806b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe45ca00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe45806fbd0 con 0x7fe45806eed0 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe45ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4580707a0 con 0x7fe458145080 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe45ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4580ce020 con 0x7fe458144980 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe458145080 0x7fe45806b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe458145080 0x7fe45806b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36264/0 (socket says 172.21.15.67:36264) 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457e00640 1 -- 172.21.15.67:0/3330224776 learned_addr learned my addr 172.21.15.67:0/3330224776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457400640 1 --2- 172.21.15.67:0/3330224776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe458144980 0x7fe45806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457e00640 1 -- 172.21.15.67:0/3330224776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe458144980 msgr2=0x7fe45806ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457e00640 1 --2- 172.21.15.67:0/3330224776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe458144980 0x7fe45806ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457e00640 1 -- 172.21.15.67:0/3330224776 >> v1:172.21.15.86:6789/0 conn(0x7fe45806eed0 legacy=0x7fe45815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.153+0000 7fe457e00640 1 -- 172.21.15.67:0/3330224776 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45806fe70 con 0x7fe458145080 2024-04-17T10:29:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe457e00640 1 --2- 172.21.15.67:0/3330224776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458145080 0x7fe45806b1d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fe450009870 tx=0x7fe45001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ca0827d6932fab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe456a00640 1 -- 172.21.15.67:0/3330224776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4500b5070 con 0x7fe458145080 2024-04-17T10:29:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe456a00640 1 -- 172.21.15.67:0/3330224776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4500a8a50 con 0x7fe458145080 2024-04-17T10:29:49.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe456a00640 1 -- 172.21.15.67:0/3330224776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4500a8e20 con 0x7fe458145080 2024-04-17T10:29:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- 172.21.15.67:0/3330224776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458145080 msgr2=0x7fe45806b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 --2- 172.21.15.67:0/3330224776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458145080 0x7fe45806b1d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fe450009870 tx=0x7fe45001ed50 comp rx=0 tx=0).stop 2024-04-17T10:29:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- 172.21.15.67:0/3330224776 shutdown_connections 2024-04-17T10:29:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 --2- 172.21.15.67:0/3330224776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458145080 0x7fe45806b1d0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 --2- 172.21.15.67:0/3330224776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe458144980 0x7fe45806ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- 172.21.15.67:0/3330224776 >> 172.21.15.67:0/3330224776 conn(0x7fe45806e3b0 msgr2=0x7fe45806e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- 172.21.15.67:0/3330224776 shutdown_connections 2024-04-17T10:29:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- 172.21.15.67:0/3330224776 wait complete. 2024-04-17T10:29:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 Processor -- start 2024-04-17T10:29:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- start start 2024-04-17T10:29:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe45806eed0 0x7fe4581e9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458144980 0x7fe4581e7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe458145080 0x7fe45815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe458070c30 con 0x7fe458144980 2024-04-17T10:29:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe458070430 con 0x7fe458145080 2024-04-17T10:29:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe45ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45806ba40 con 0x7fe45806eed0 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.157+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458144980 0x7fe4581e7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458144980 0x7fe4581e7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36276/0 (socket says 172.21.15.67:36276) 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 -- 172.21.15.67:0/1255303818 learned_addr learned my addr 172.21.15.67:0/1255303818 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe457400640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe45806eed0 0x7fe4581e9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe457e00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe458145080 0x7fe45815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 -- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe45806eed0 msgr2=0x7fe4581e9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe45806eed0 0x7fe4581e9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 -- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe458145080 msgr2=0x7fe45815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe458145080 0x7fe45815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 -- 172.21.15.67:0/1255303818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4581e99e0 con 0x7fe458144980 2024-04-17T10:29:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44ea00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458144980 0x7fe4581e7560 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fe44000c790 tx=0x7fe44000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44fe00640 1 -- 172.21.15.67:0/1255303818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440016020 con 0x7fe458144980 2024-04-17T10:29:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44fe00640 1 -- 172.21.15.67:0/1255303818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe440006d60 con 0x7fe458144980 2024-04-17T10:29:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4581e9cc0 con 0x7fe458144980 2024-04-17T10:29:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44fe00640 1 -- 172.21.15.67:0/1255303818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe440003bc0 con 0x7fe458144980 2024-04-17T10:29:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4581ea560 con 0x7fe458144980 2024-04-17T10:29:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4581e9e90 con 0x7fe458144980 2024-04-17T10:29:49.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.161+0000 7fe44fe00640 1 -- 172.21.15.67:0/1255303818 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe44000a850 con 0x7fe458144980 2024-04-17T10:29:49.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.165+0000 7fe44fe00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe434066f90 0x7fe434069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.165+0000 7fe44fe00640 1 -- 172.21.15.67:0/1255303818 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe440084e00 con 0x7fe458144980 2024-04-17T10:29:49.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.165+0000 7fe457400640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe434066f90 0x7fe434069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.165+0000 7fe457400640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe434066f90 0x7fe434069450 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe448009b60 tx=0x7fe448009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:49.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.169+0000 7fe44fe00640 1 -- 172.21.15.67:0/1255303818 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe440020020 con 0x7fe458144980 2024-04-17T10:29:49.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe458074dc0 con 0x7fe458144980 2024-04-17T10:29:49.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe44fe00640 1 -- 172.21.15.67:0/1255303818 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16910 (secure 0 0 0) 0x7fe440055590 con 0x7fe458144980 2024-04-17T10:29:49.326 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:49.327 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:47.987037+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:29:46.421958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-17T10:29:46.986879+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:49.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe434066f90 msgr2=0x7fe434069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe434066f90 0x7fe434069450 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe448009b60 tx=0x7fe448009190 comp rx=0 tx=0).stop 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458144980 msgr2=0x7fe4581e7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458144980 0x7fe4581e7560 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fe44000c790 tx=0x7fe44000cc50 comp rx=0 tx=0).stop 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 shutdown_connections 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe434066f90 0x7fe434069450 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe458145080 0x7fe45815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe458144980 0x7fe4581e7560 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 --2- 172.21.15.67:0/1255303818 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe45806eed0 0x7fe4581e9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 >> 172.21.15.67:0/1255303818 conn(0x7fe45806e3b0 msgr2=0x7fe4580b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 shutdown_connections 2024-04-17T10:29:49.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.333+0000 7fe45ca00640 1 -- 172.21.15.67:0/1255303818 wait complete. 2024-04-17T10:29:49.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:29:49.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.497+0000 7f1d74000640 1 Processor -- start 2024-04-17T10:29:49.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.497+0000 7f1d74000640 1 -- start start 2024-04-17T10:29:49.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.497+0000 7f1d74000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c15c580 0x7f1d6c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d74000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6c15cb30 con 0x7f1d6c071af0 2024-04-17T10:29:49.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d74000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d6c0cd610 con 0x7f1d6c159440 2024-04-17T10:29:49.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d74000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0a8650 con 0x7f1d6c15c580 2024-04-17T10:29:49.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d70c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1d6c071af0 0x7f1d6c15be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46404/0 (socket says 172.21.15.67:46404) 2024-04-17T10:29:49.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d70c00640 1 -- 172.21.15.67:0/1202800682 learned_addr learned my addr 172.21.15.67:0/1202800682 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:49.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d72000640 1 --2- 172.21.15.67:0/1202800682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c15c580 0x7f1d6c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d6be00640 1 -- 172.21.15.67:0/1202800682 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36749761 0 0) 0x7f1d6c15cb30 con 0x7f1d6c071af0 2024-04-17T10:29:49.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d6be00640 1 -- 172.21.15.67:0/1202800682 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d54003530 con 0x7f1d6c071af0 2024-04-17T10:29:49.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d6be00640 1 -- 172.21.15.67:0/1202800682 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896322048 0 0) 0x7f1d6c0cd610 con 0x7f1d6c159440 2024-04-17T10:29:49.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d6be00640 1 -- 172.21.15.67:0/1202800682 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d6c15cb30 con 0x7f1d6c159440 2024-04-17T10:29:49.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d72000640 1 -- 172.21.15.67:0/1202800682 >> v1:172.21.15.86:6789/0 conn(0x7f1d6c159440 legacy=0x7f1d6c0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d72000640 1 -- 172.21.15.67:0/1202800682 >> v1:172.21.15.67:6789/0 conn(0x7f1d6c071af0 legacy=0x7f1d6c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.501+0000 7f1d72000640 1 -- 172.21.15.67:0/1202800682 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c0cd8f0 con 0x7f1d6c15c580 2024-04-17T10:29:49.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d72000640 1 --2- 172.21.15.67:0/1202800682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c15c580 0x7f1d6c0ce0f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009dc0 tx=0x7f1d6401c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bda6a35917e1ce6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d6be00640 1 -- 172.21.15.67:0/1202800682 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d640b2070 con 0x7f1d6c15c580 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d6be00640 1 -- 172.21.15.67:0/1202800682 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d640a5960 con 0x7f1d6c15c580 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d6be00640 1 -- 172.21.15.67:0/1202800682 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d640a5d30 con 0x7f1d6c15c580 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- 172.21.15.67:0/1202800682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c15c580 msgr2=0x7f1d6c0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 --2- 172.21.15.67:0/1202800682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c15c580 0x7f1d6c0ce0f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1d64009dc0 tx=0x7f1d6401c990 comp rx=0 tx=0).stop 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- 172.21.15.67:0/1202800682 shutdown_connections 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 --2- 172.21.15.67:0/1202800682 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c15c580 0x7f1d6c0ce0f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- 172.21.15.67:0/1202800682 >> 172.21.15.67:0/1202800682 conn(0x7f1d6c0b1740 msgr2=0x7f1d6c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:49.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- 172.21.15.67:0/1202800682 shutdown_connections 2024-04-17T10:29:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- 172.21.15.67:0/1202800682 wait complete. 2024-04-17T10:29:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 Processor -- start 2024-04-17T10:29:49.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- start start 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c071af0 0x7f1d6c14db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6c159440 0x7f1d6c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d6c15c580 0x7f1d6c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c15dc10 con 0x7f1d6c159440 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c15da90 con 0x7f1d6c15c580 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d74000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c15dd90 con 0x7f1d6c071af0 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.505+0000 7f1d70c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6c159440 0x7f1d6c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6c159440 0x7f1d6c159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36284/0 (socket says 172.21.15.67:36284) 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 -- 172.21.15.67:0/1246912700 learned_addr learned my addr 172.21.15.67:0/1246912700 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d72000640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d6c15c580 0x7f1d6c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d71600640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c071af0 0x7f1d6c14db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 -- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c071af0 msgr2=0x7f1d6c14db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c071af0 0x7f1d6c14db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 -- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d6c15c580 msgr2=0x7f1d6c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d6c15c580 0x7f1d6c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 -- 172.21.15.67:0/1246912700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c15a470 con 0x7f1d6c159440 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d70c00640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6c159440 0x7f1d6c159a90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005890 tx=0x7f1d5800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d69600640 1 -- 172.21.15.67:0/1246912700 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58017070 con 0x7f1d6c159440 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d69600640 1 -- 172.21.15.67:0/1246912700 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d58005cf0 con 0x7f1d6c159440 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c15a750 con 0x7f1d6c159440 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d69600640 1 -- 172.21.15.67:0/1246912700 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d58007310 con 0x7f1d6c159440 2024-04-17T10:29:49.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c15ad10 con 0x7f1d6c159440 2024-04-17T10:29:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c04ef30 con 0x7f1d6c159440 2024-04-17T10:29:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d69600640 1 -- 172.21.15.67:0/1246912700 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1d5801a070 con 0x7f1d6c159440 2024-04-17T10:29:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d69600640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d4c066f60 0x7f1d4c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.509+0000 7f1d69600640 1 -- 172.21.15.67:0/1246912700 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1d58084560 con 0x7f1d6c159440 2024-04-17T10:29:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.513+0000 7f1d71600640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d4c066f60 0x7f1d4c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.513+0000 7f1d71600640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d4c066f60 0x7f1d4c069420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1d600011f0 tx=0x7f1d60009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:49.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.513+0000 7f1d69600640 1 -- 172.21.15.67:0/1246912700 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1d58054d80 con 0x7f1d6c159440 2024-04-17T10:29:49.569 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T10:29:49.573+0000 7f5e81c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:29:49.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:49.717+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1d6c154800 con 0x7f1d6c159440 2024-04-17T10:29:50.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.621+0000 7f1d69600640 1 -- 172.21.15.67:0/1246912700 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1d58058e20 con 0x7f1d6c159440 2024-04-17T10:29:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d4c066f60 msgr2=0x7f1d4c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d4c066f60 0x7f1d4c069420 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f1d600011f0 tx=0x7f1d60009040 comp rx=0 tx=0).stop 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6c159440 msgr2=0x7f1d6c159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6c159440 0x7f1d6c159a90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005890 tx=0x7f1d5800c8b0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 shutdown_connections 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d4c066f60 0x7f1d4c069420 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d6c15c580 0x7f1d6c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d6c159440 0x7f1d6c159a90 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 --2- 172.21.15.67:0/1246912700 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d6c071af0 0x7f1d6c14db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 >> 172.21.15.67:0/1246912700 conn(0x7f1d6c0b1740 msgr2=0x7f1d6c06de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 shutdown_connections 2024-04-17T10:29:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.625+0000 7f1d74000640 1 -- 172.21.15.67:0/1246912700 wait complete. 2024-04-17T10:29:50.642 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:29:50.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:29:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 Processor -- start 2024-04-17T10:29:50.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 -- start start 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac980aef60 0x7fac980cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac980cce60 0x7fac9806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac9806e7b0 0x7fac980713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac980af340 con 0x7fac980cce60 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac980b4990 con 0x7fac980aef60 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7faca0600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9815ae90 con 0x7fac9806e7b0 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.797+0000 7fac9d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac980cce60 0x7fac9806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:50.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac980cce60 0x7fac9806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57192/0 (socket says 172.21.15.67:57192) 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 -- 172.21.15.67:0/2720788833 learned_addr learned my addr 172.21.15.67:0/2720788833 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9dc00640 1 --2- 172.21.15.67:0/2720788833 >> v2:172.21.15.86:3300/0 conn(0x7fac980aef60 0x7fac980cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9e600640 1 --2- 172.21.15.67:0/2720788833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac9806e7b0 0x7fac980713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 -- 172.21.15.67:0/2720788833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac9806e7b0 msgr2=0x7fac980713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 --2- 172.21.15.67:0/2720788833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac9806e7b0 0x7fac980713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 -- 172.21.15.67:0/2720788833 >> v2:172.21.15.86:3300/0 conn(0x7fac980aef60 msgr2=0x7fac980cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 --2- 172.21.15.67:0/2720788833 >> v2:172.21.15.86:3300/0 conn(0x7fac980aef60 0x7fac980cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 -- 172.21.15.67:0/2720788833 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9806ee80 con 0x7fac980cce60 2024-04-17T10:29:50.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac9d200640 1 --2- 172.21.15.67:0/2720788833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980cce60 0x7fac9806e270 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fac9806bdc0 tx=0x7fac8401c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4907057fb3d926dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac97e00640 1 -- 172.21.15.67:0/2720788833 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840b0070 con 0x7fac980cce60 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac97e00640 1 -- 172.21.15.67:0/2720788833 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac840ab600 con 0x7fac980cce60 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7fac97e00640 1 -- 172.21.15.67:0/2720788833 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840ab9d0 con 0x7fac980cce60 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7faca0600640 1 -- 172.21.15.67:0/2720788833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980cce60 msgr2=0x7fac9806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.801+0000 7faca0600640 1 --2- 172.21.15.67:0/2720788833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980cce60 0x7fac9806e270 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fac9806bdc0 tx=0x7fac8401c1d0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- 172.21.15.67:0/2720788833 shutdown_connections 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 --2- 172.21.15.67:0/2720788833 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac9806e7b0 0x7fac980713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 --2- 172.21.15.67:0/2720788833 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980cce60 0x7fac9806e270 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 --2- 172.21.15.67:0/2720788833 >> v2:172.21.15.86:3300/0 conn(0x7fac980aef60 0x7fac980cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- 172.21.15.67:0/2720788833 >> 172.21.15.67:0/2720788833 conn(0x7fac980b1740 msgr2=0x7fac980b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- 172.21.15.67:0/2720788833 shutdown_connections 2024-04-17T10:29:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- 172.21.15.67:0/2720788833 wait complete. 2024-04-17T10:29:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 Processor -- start 2024-04-17T10:29:50.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- start start 2024-04-17T10:29:50.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9806e7b0 0x7fac9814d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:50.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980aef60 0x7fac98147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac980cce60 0x7fac981560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9815bc10 con 0x7fac980aef60 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980b3120 con 0x7fac9806e7b0 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9814d080 con 0x7fac980cce60 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980aef60 0x7fac98147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980aef60 0x7fac98147210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57194/0 (socket says 172.21.15.67:57194) 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 -- 172.21.15.67:0/536831239 learned_addr learned my addr 172.21.15.67:0/536831239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9dc00640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9806e7b0 0x7fac9814d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9e600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac980cce60 0x7fac981560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 -- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac980cce60 msgr2=0x7fac981560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac980cce60 0x7fac981560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 -- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9806e7b0 msgr2=0x7fac9814d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9806e7b0 0x7fac9814d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 -- 172.21.15.67:0/536831239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9814c470 con 0x7fac980aef60 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac9d200640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980aef60 0x7fac98147210 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fac8401a880 tx=0x7fac84018f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac95600640 1 -- 172.21.15.67:0/536831239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840b0040 con 0x7fac980aef60 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac95600640 1 -- 172.21.15.67:0/536831239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac840b9070 con 0x7fac980aef60 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7fac95600640 1 -- 172.21.15.67:0/536831239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840b38a0 con 0x7fac980aef60 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.805+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9814c750 con 0x7fac980aef60 2024-04-17T10:29:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.809+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9814cd10 con 0x7fac980aef60 2024-04-17T10:29:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.809+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9814d200 con 0x7fac980aef60 2024-04-17T10:29:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.809+0000 7fac95600640 1 -- 172.21.15.67:0/536831239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fac840b3a40 con 0x7fac980aef60 2024-04-17T10:29:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.809+0000 7fac95600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac78066f60 0x7fac78069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.809+0000 7fac95600640 1 -- 172.21.15.67:0/536831239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fac8411f140 con 0x7fac980aef60 2024-04-17T10:29:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.809+0000 7fac9dc00640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac78066f60 0x7fac78069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.813+0000 7fac9dc00640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac78066f60 0x7fac78069420 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fac981505f0 tx=0x7fac8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:50.813+0000 7fac95600640 1 -- 172.21.15.67:0/536831239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fac84016110 con 0x7fac980aef60 2024-04-17T10:29:51.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.009+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fac9804ef30 con 0x7fac980aef60 2024-04-17T10:29:51.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7fac95600640 1 -- 172.21.15.67:0/536831239 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7fac840eff20 con 0x7fac980aef60 2024-04-17T10:29:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac78066f60 msgr2=0x7fac78069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac78066f60 0x7fac78069420 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fac981505f0 tx=0x7fac8c008040 comp rx=0 tx=0).stop 2024-04-17T10:29:51.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980aef60 msgr2=0x7fac98147210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980aef60 0x7fac98147210 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fac8401a880 tx=0x7fac84018f70 comp rx=0 tx=0).stop 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 shutdown_connections 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac78066f60 0x7fac78069420 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac980cce60 0x7fac981560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac980aef60 0x7fac98147210 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 --2- 172.21.15.67:0/536831239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9806e7b0 0x7fac9814d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 >> 172.21.15.67:0/536831239 conn(0x7fac980b1740 msgr2=0x7fac980af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 shutdown_connections 2024-04-17T10:29:51.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.625+0000 7faca0600640 1 -- 172.21.15.67:0/536831239 wait complete. 2024-04-17T10:29:51.642 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:29:51.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.789+0000 7fc7b4400640 1 Processor -- start 2024-04-17T10:29:51.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.789+0000 7fc7b4400640 1 -- start start 2024-04-17T10:29:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b4400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac06eac0 0x7fc7ac143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b4400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7ac0cf0c0 0x7fc7ac144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b4400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac144560 con 0x7fc7ac06e6e0 2024-04-17T10:29:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b4400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac15ae40 con 0x7fc7ac0cf0c0 2024-04-17T10:29:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b4400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac071440 con 0x7fc7ac06eac0 2024-04-17T10:29:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b2400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc7ac0cf0c0 0x7fc7ac144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:51.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b1a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc7ac06e6e0 0x7fc7ac0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50902/0 (socket says 172.21.15.67:50902) 2024-04-17T10:29:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b1a00640 1 -- 172.21.15.67:0/4285665551 learned_addr learned my addr 172.21.15.67:0/4285665551 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b1000640 1 --2- 172.21.15.67:0/4285665551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac06eac0 0x7fc7ac143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b2400640 1 -- 172.21.15.67:0/4285665551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac06eac0 msgr2=0x7fc7ac143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b2400640 1 --2- 172.21.15.67:0/4285665551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac06eac0 0x7fc7ac143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b2400640 1 -- 172.21.15.67:0/4285665551 >> v1:172.21.15.67:6789/0 conn(0x7fc7ac06e6e0 legacy=0x7fc7ac0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:51.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b2400640 1 -- 172.21.15.67:0/4285665551 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac071720 con 0x7fc7ac0cf0c0 2024-04-17T10:29:51.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7abe00640 1 -- 172.21.15.67:0/4285665551 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025028202 0 0) 0x7fc7ac144560 con 0x7fc7ac06e6e0 2024-04-17T10:29:51.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.793+0000 7fc7b2400640 1 --2- 172.21.15.67:0/4285665551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac0cf0c0 0x7fc7ac144020 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009d90 tx=0x7fc7a401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb007c51cdc4b2b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7abe00640 1 -- 172.21.15.67:0/4285665551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40b2070 con 0x7fc7ac0cf0c0 2024-04-17T10:29:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7abe00640 1 -- 172.21.15.67:0/4285665551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a40a5970 con 0x7fc7ac0cf0c0 2024-04-17T10:29:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7abe00640 1 -- 172.21.15.67:0/4285665551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a40a5d40 con 0x7fc7ac0cf0c0 2024-04-17T10:29:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- 172.21.15.67:0/4285665551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac0cf0c0 msgr2=0x7fc7ac144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:51.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 --2- 172.21.15.67:0/4285665551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac0cf0c0 0x7fc7ac144020 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4009d90 tx=0x7fc7a401c8b0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- 172.21.15.67:0/4285665551 shutdown_connections 2024-04-17T10:29:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 --2- 172.21.15.67:0/4285665551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac0cf0c0 0x7fc7ac144020 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 --2- 172.21.15.67:0/4285665551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac06eac0 0x7fc7ac143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- 172.21.15.67:0/4285665551 >> 172.21.15.67:0/4285665551 conn(0x7fc7ac06ac90 msgr2=0x7fc7ac06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- 172.21.15.67:0/4285665551 shutdown_connections 2024-04-17T10:29:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- 172.21.15.67:0/4285665551 wait complete. 2024-04-17T10:29:51.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 Processor -- start 2024-04-17T10:29:51.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- start start 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac06e6e0 0x7fc7ac0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac06eac0 0x7fc7ac13bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0cf0c0 0x7fc7ac0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b3a40 con 0x7fc7ac06eac0 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b38c0 con 0x7fc7ac06e6e0 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b4400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b3bc0 con 0x7fc7ac0cf0c0 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac06eac0 0x7fc7ac13bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac06eac0 0x7fc7ac13bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57222/0 (socket says 172.21.15.67:57222) 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b1000640 1 -- 172.21.15.67:0/3089513925 learned_addr learned my addr 172.21.15.67:0/3089513925 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b1a00640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac06e6e0 0x7fc7ac0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.797+0000 7fc7b2400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0cf0c0 0x7fc7ac0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1000640 1 -- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0cf0c0 msgr2=0x7fc7ac0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1000640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0cf0c0 0x7fc7ac0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1000640 1 -- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac06e6e0 msgr2=0x7fc7ac0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1000640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac06e6e0 0x7fc7ac0c46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1000640 1 -- 172.21.15.67:0/3089513925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac13c510 con 0x7fc7ac06eac0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b2400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0cf0c0 0x7fc7ac0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1a00640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac06e6e0 0x7fc7ac0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1000640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac06eac0 0x7fc7ac13bfd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc79800e930 tx=0x7fc79800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7a9600640 1 -- 172.21.15.67:0/3089513925 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc798017070 con 0x7fc7ac06eac0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7a9600640 1 -- 172.21.15.67:0/3089513925 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79800ca00 con 0x7fc7ac06eac0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7a9600640 1 -- 172.21.15.67:0/3089513925 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79800cdb0 con 0x7fc7ac06eac0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac0c4d70 con 0x7fc7ac06eac0 2024-04-17T10:29:51.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0c5330 con 0x7fc7ac06eac0 2024-04-17T10:29:51.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac072d50 con 0x7fc7ac06eac0 2024-04-17T10:29:51.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7a9600640 1 -- 172.21.15.67:0/3089513925 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc79801a070 con 0x7fc7ac06eac0 2024-04-17T10:29:51.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7a9600640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc78c066f90 0x7fc78c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7a9600640 1 -- 172.21.15.67:0/3089513925 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc798085310 con 0x7fc7ac06eac0 2024-04-17T10:29:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.801+0000 7fc7b1a00640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc78c066f90 0x7fc78c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:51.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.805+0000 7fc7b1a00640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc78c066f90 0x7fc78c069450 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0007fd0 tx=0x7fc7a0007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:51.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:51.805+0000 7fc7a9600640 1 -- 172.21.15.67:0/3089513925 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc798056120 con 0x7fc7ac06eac0 2024-04-17T10:29:51.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7ac074e30 con 0x7fc7ac06eac0 2024-04-17T10:29:51.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7a9600640 1 -- 172.21.15.67:0/3089513925 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fc798055a20 con 0x7fc7ac06eac0 2024-04-17T10:29:51.998 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:51.998 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/3654887462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3654887462},{"type":"v1","addr":"172.21.15.86:6836","nonce":3654887462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6814,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1627300843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1627300843},{"type":"v1","addr":"172.21.15.67:6835","nonce":1627300843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/39011498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":39011498},{"type":"v1","addr":"172.21.15.86:6837","nonce":39011498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:29:47.987388+0000","modified":"2024-04-17T10:29:51.015058+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/3374829114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3374829114},{"type":"v1","addr":"172.21.15.67:6834","nonce":3374829114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-17T10:29:51.998 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 55 2024-04-17T10:29:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc78c066f90 msgr2=0x7fc78c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc78c066f90 0x7fc78c069450 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0007fd0 tx=0x7fc7a0007c20 comp rx=0 tx=0).stop 2024-04-17T10:29:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac06eac0 msgr2=0x7fc7ac13bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac06eac0 0x7fc7ac13bfd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fc79800e930 tx=0x7fc79800edf0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 shutdown_connections 2024-04-17T10:29:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc78c066f90 0x7fc78c069450 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc7ac0cf0c0 0x7fc7ac0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc7ac06eac0 0x7fc7ac13bfd0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 --2- 172.21.15.67:0/3089513925 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc7ac06e6e0 0x7fc7ac0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 >> 172.21.15.67:0/3089513925 conn(0x7fc7ac06ac90 msgr2=0x7fc7ac0af040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 shutdown_connections 2024-04-17T10:29:52.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.005+0000 7fc7b4400640 1 -- 172.21.15.67:0/3089513925 wait complete. 2024-04-17T10:29:52.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:29:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.165+0000 7f229b600640 1 Processor -- start 2024-04-17T10:29:52.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.165+0000 7f229b600640 1 -- start start 2024-04-17T10:29:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f229b600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22941466e0 0x7f229406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f229b600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294146da0 0x7f2294071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f229b600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2294071850 0x7f229406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f229b600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2294161740 con 0x7f22941466e0 2024-04-17T10:29:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f229b600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2294161a20 con 0x7f2294071850 2024-04-17T10:29:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f229b600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2294161d00 con 0x7f2294146da0 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22941466e0 0x7f229406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f22941466e0 0x7f229406eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57226/0 (socket says 172.21.15.67:57226) 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 -- 172.21.15.67:0/782932421 learned_addr learned my addr 172.21.15.67:0/782932421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2293e00640 1 --2- 172.21.15.67:0/782932421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294146da0 0x7f2294071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2299600640 1 --2- 172.21.15.67:0/782932421 >> v2:172.21.15.86:3300/0 conn(0x7f2294071850 0x7f229406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 -- 172.21.15.67:0/782932421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294146da0 msgr2=0x7f2294071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 --2- 172.21.15.67:0/782932421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294146da0 0x7f2294071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 -- 172.21.15.67:0/782932421 >> v2:172.21.15.86:3300/0 conn(0x7f2294071850 msgr2=0x7f229406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 --2- 172.21.15.67:0/782932421 >> v2:172.21.15.86:3300/0 conn(0x7f2294071850 0x7f229406d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.169+0000 7f2298c00640 1 -- 172.21.15.67:0/782932421 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2294161fe0 con 0x7f22941466e0 2024-04-17T10:29:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f2298c00640 1 --2- 172.21.15.67:0/782932421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22941466e0 0x7f229406eb70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f229415e0b0 tx=0x7f228801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4bd70e6cf8bf06c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f2293400640 1 -- 172.21.15.67:0/782932421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22880b0070 con 0x7f22941466e0 2024-04-17T10:29:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f2293400640 1 -- 172.21.15.67:0/782932421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22880ab600 con 0x7f22941466e0 2024-04-17T10:29:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f2293400640 1 -- 172.21.15.67:0/782932421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22880ab9d0 con 0x7f22941466e0 2024-04-17T10:29:52.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 -- 172.21.15.67:0/782932421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22941466e0 msgr2=0x7f229406eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 --2- 172.21.15.67:0/782932421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22941466e0 0x7f229406eb70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f229415e0b0 tx=0x7f228801c260 comp rx=0 tx=0).stop 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 -- 172.21.15.67:0/782932421 shutdown_connections 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 --2- 172.21.15.67:0/782932421 >> v2:172.21.15.86:3300/0 conn(0x7f2294071850 0x7f229406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 --2- 172.21.15.67:0/782932421 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294146da0 0x7f2294071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 --2- 172.21.15.67:0/782932421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f22941466e0 0x7f229406eb70 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 -- 172.21.15.67:0/782932421 >> 172.21.15.67:0/782932421 conn(0x7f229406ac90 msgr2=0x7f229406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 -- 172.21.15.67:0/782932421 shutdown_connections 2024-04-17T10:29:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 -- 172.21.15.67:0/782932421 wait complete. 2024-04-17T10:29:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 Processor -- start 2024-04-17T10:29:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 -- start start 2024-04-17T10:29:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294071850 0x7f229413dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22941466e0 0x7f22940affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2294146da0 0x7f229415f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.173+0000 7f229b600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22940ae170 con 0x7f2294146da0 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f229b600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22940adff0 con 0x7f22941466e0 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f229b600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22940ae2f0 con 0x7f2294071850 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2294146da0 0x7f229415f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2294146da0 0x7f229415f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57240/0 (socket says 172.21.15.67:57240) 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 -- 172.21.15.67:0/1536378003 learned_addr learned my addr 172.21.15.67:0/1536378003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2298c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294071850 0x7f229413dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2293e00640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22941466e0 0x7f22940affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 -- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294071850 msgr2=0x7f229413dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294071850 0x7f229413dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 -- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22941466e0 msgr2=0x7f22940affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22941466e0 0x7f22940affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 -- 172.21.15.67:0/1536378003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f229415f9c0 con 0x7f2294146da0 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2298c00640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294071850 0x7f229413dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2299600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2294146da0 0x7f229415f100 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f228c007e90 tx=0x7f228c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2290c00640 1 -- 172.21.15.67:0/1536378003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c014420 con 0x7f2294146da0 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2290c00640 1 -- 172.21.15.67:0/1536378003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228c019070 con 0x7f2294146da0 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2290c00640 1 -- 172.21.15.67:0/1536378003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228c014820 con 0x7f2294146da0 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229415fca0 con 0x7f2294146da0 2024-04-17T10:29:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229415f640 con 0x7f2294146da0 2024-04-17T10:29:52.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22941601c0 con 0x7f2294146da0 2024-04-17T10:29:52.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.177+0000 7f2290c00640 1 -- 172.21.15.67:0/1536378003 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f228c0149c0 con 0x7f2294146da0 2024-04-17T10:29:52.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.181+0000 7f2290c00640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2274066f60 0x7f2274069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.181+0000 7f2290c00640 1 -- 172.21.15.67:0/1536378003 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f228c085850 con 0x7f2294146da0 2024-04-17T10:29:52.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.181+0000 7f2298c00640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2274066f60 0x7f2274069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.181+0000 7f2298c00640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2274066f60 0x7f2274069420 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f22880ad000 tx=0x7f22880a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.185+0000 7f2290c00640 1 -- 172.21.15.67:0/1536378003 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f228c056110 con 0x7f2294146da0 2024-04-17T10:29:52.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.345+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22940ae9a0 con 0x7f2294146da0 2024-04-17T10:29:52.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f2290c00640 1 -- 172.21.15.67:0/1536378003 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16897 (secure 0 0 0) 0x7f228c05a1b0 con 0x7f2294146da0 2024-04-17T10:29:52.341 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:52.342 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":83,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:29:49.614759+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:29:46.421958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-17T10:29:46.986879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:29:52.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2274066f60 msgr2=0x7f2274069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2274066f60 0x7f2274069420 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f22880ad000 tx=0x7f22880a6000 comp rx=0 tx=0).stop 2024-04-17T10:29:52.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2294146da0 msgr2=0x7f229415f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2294146da0 0x7f229415f100 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f228c007e90 tx=0x7f228c00e4b0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 shutdown_connections 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2274066f60 0x7f2274069420 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2294146da0 0x7f229415f100 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f22941466e0 0x7f22940affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 --2- 172.21.15.67:0/1536378003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2294071850 0x7f229413dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 >> 172.21.15.67:0/1536378003 conn(0x7f229406ac90 msgr2=0x7f2294143c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 shutdown_connections 2024-04-17T10:29:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.349+0000 7f229b600640 1 -- 172.21.15.67:0/1536378003 wait complete. 2024-04-17T10:29:52.363 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:29:52.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.509+0000 7f48b7c00640 1 Processor -- start 2024-04-17T10:29:52.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.509+0000 7f48b7c00640 1 -- start start 2024-04-17T10:29:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b7c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b01620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b7c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f48b00ce400 0x7f48b015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b7c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48b015e550 0x7f48b00cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b7c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b0161650 con 0x7f48b015e550 2024-04-17T10:29:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b7c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48b01625e0 con 0x7f48b00ce400 2024-04-17T10:29:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b7c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b015eb50 con 0x7f48b00ce020 2024-04-17T10:29:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48b015e550 0x7f48b00cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f48b015e550 0x7f48b00cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57244/0 (socket says 172.21.15.67:57244) 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 -- 172.21.15.67:0/3305171560 learned_addr learned my addr 172.21.15.67:0/3305171560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48a7e00640 1 --2- 172.21.15.67:0/3305171560 >> v2:172.21.15.86:3300/0 conn(0x7f48b00ce400 0x7f48b015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5200640 1 --2- 172.21.15.67:0/3305171560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b01620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 -- 172.21.15.67:0/3305171560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 msgr2=0x7f48b01620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 --2- 172.21.15.67:0/3305171560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b01620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 -- 172.21.15.67:0/3305171560 >> v2:172.21.15.86:3300/0 conn(0x7f48b00ce400 msgr2=0x7f48b015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 --2- 172.21.15.67:0/3305171560 >> v2:172.21.15.86:3300/0 conn(0x7f48b00ce400 0x7f48b015e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.513+0000 7f48b5c00640 1 -- 172.21.15.67:0/3305171560 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b015ee30 con 0x7f48b015e550 2024-04-17T10:29:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b5c00640 1 --2- 172.21.15.67:0/3305171560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b00cf510 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f48b006e1b0 tx=0x7f48a801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2772a499119a097 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48afe00640 1 -- 172.21.15.67:0/3305171560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a80a5840 con 0x7f48b015e550 2024-04-17T10:29:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48afe00640 1 -- 172.21.15.67:0/3305171560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a80a59e0 con 0x7f48b015e550 2024-04-17T10:29:52.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48afe00640 1 -- 172.21.15.67:0/3305171560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a80a5db0 con 0x7f48b015e550 2024-04-17T10:29:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- 172.21.15.67:0/3305171560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 msgr2=0x7f48b00cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3305171560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b00cf510 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f48b006e1b0 tx=0x7f48a801c9c0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- 172.21.15.67:0/3305171560 shutdown_connections 2024-04-17T10:29:52.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3305171560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b00cf510 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3305171560 >> v2:172.21.15.86:3300/0 conn(0x7f48b00ce400 0x7f48b015e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3305171560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b01620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- 172.21.15.67:0/3305171560 >> 172.21.15.67:0/3305171560 conn(0x7f48b00b33f0 msgr2=0x7f48b00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- 172.21.15.67:0/3305171560 shutdown_connections 2024-04-17T10:29:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- 172.21.15.67:0/3305171560 wait complete. 2024-04-17T10:29:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 Processor -- start 2024-04-17T10:29:52.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- start start 2024-04-17T10:29:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b015a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00ce400 0x7f48b00b2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b015d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00ce960 con 0x7f48b015e550 2024-04-17T10:29:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00ce7e0 con 0x7f48b00ce400 2024-04-17T10:29:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.517+0000 7f48b7c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48b00c84b0 con 0x7f48b00ce020 2024-04-17T10:29:52.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b015d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b015d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57246/0 (socket says 172.21.15.67:57246) 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 -- 172.21.15.67:0/3180937204 learned_addr learned my addr 172.21.15.67:0/3180937204 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48a7e00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00ce400 0x7f48b00b2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5200640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b015a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 -- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 msgr2=0x7f48b015a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b015a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 -- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00ce400 msgr2=0x7f48b00b2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00ce400 0x7f48b00b2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 -- 172.21.15.67:0/3180937204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b00b18d0 con 0x7f48b015e550 2024-04-17T10:29:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b5c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b015d3e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f48a80af040 tx=0x7f48a8019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48ad600640 1 -- 172.21.15.67:0/3180937204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a80bc020 con 0x7f48b015e550 2024-04-17T10:29:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48ad600640 1 -- 172.21.15.67:0/3180937204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a80b8470 con 0x7f48b015e550 2024-04-17T10:29:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48ad600640 1 -- 172.21.15.67:0/3180937204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a80b8840 con 0x7f48b015e550 2024-04-17T10:29:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b00b1bb0 con 0x7f48b015e550 2024-04-17T10:29:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b00b2170 con 0x7f48b015e550 2024-04-17T10:29:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b0070ab0 con 0x7f48b015e550 2024-04-17T10:29:52.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.521+0000 7f48ad600640 1 -- 172.21.15.67:0/3180937204 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f48a80bf020 con 0x7f48b015e550 2024-04-17T10:29:52.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.525+0000 7f48ad600640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f488c066f60 0x7f488c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.525+0000 7f48ad600640 1 -- 172.21.15.67:0/3180937204 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f48a81200d0 con 0x7f48b015e550 2024-04-17T10:29:52.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.525+0000 7f48b5200640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f488c066f60 0x7f488c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.525+0000 7f48b5200640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f488c066f60 0x7f488c069420 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f48a00045f0 tx=0x7f48a0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.529+0000 7f48ad600640 1 -- 172.21.15.67:0/3180937204 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f48a80c6030 con 0x7f48b015e550 2024-04-17T10:29:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.685+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48b00b1680 con 0x7f488c066f60 2024-04-17T10:29:52.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.693+0000 7f48ad600640 1 -- 172.21.15.67:0/3180937204 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240186 (secure 0 0 0) 0x7f48b00b1680 con 0x7f488c066f60 2024-04-17T10:29:52.689 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:52.691 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:52.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f488c066f60 msgr2=0x7f488c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f488c066f60 0x7f488c069420 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f48a00045f0 tx=0x7f48a0008040 comp rx=0 tx=0).stop 2024-04-17T10:29:52.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 msgr2=0x7f48b015d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b015d3e0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f48a80af040 tx=0x7f48a8019760 comp rx=0 tx=0).stop 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 shutdown_connections 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f488c066f60 0x7f488c069420 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f48b015e550 0x7f48b015d3e0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f48b00ce400 0x7f48b00b2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 --2- 172.21.15.67:0/3180937204 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f48b00ce020 0x7f48b015a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 >> 172.21.15.67:0/3180937204 conn(0x7f48b00b33f0 msgr2=0x7f48b0160500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 shutdown_connections 2024-04-17T10:29:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.697+0000 7f48b7c00640 1 -- 172.21.15.67:0/3180937204 wait complete. 2024-04-17T10:29:52.709 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-04-17T10:29:52.204814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":322060,"kb_used_data":8772,"kb_used_omap":12,"kb_used_meta":313139,"kb_avail":754652660,"statfs":{"total":773094113280,"available":772764323840,"internally_reserved":0,"allocated":8982528,"data_stored":4769445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":320654897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.619489"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596550+0000","last_change":"2024-04-17T10:29:46.596550+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596550+0000","last_undegraded":"2024-04-17T10:29:46.596550+0000","last_fullsized":"2024-04-17T10:29:46.596550+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590391+0000","last_change":"2024-04-17T10:29:46.590391+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590391+0000","last_undegraded":"2024-04-17T10:29:46.590391+0000","last_fullsized":"2024-04-17T10:29:46.590391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.591131+0000","last_change":"2024-04-17T10:29:46.591131+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.591131+0000","last_undegraded":"2024-04-17T10:29:46.591131+0000","last_fullsized":"2024-04-17T10:29:46.591131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589295+0000","last_change":"2024-04-17T10:29:46.589295+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589295+0000","last_undegraded":"2024-04-17T10:29:46.589295+0000","last_fullsized":"2024-04-17T10:29:46.589295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.586939+0000","last_change":"2024-04-17T10:29:46.586939+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.586939+0000","last_undegraded":"2024-04-17T10:29:46.586939+0000","last_fullsized":"2024-04-17T10:29:46.586939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593868+0000","last_change":"2024-04-17T10:29:46.593868+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593868+0000","last_undegraded":"2024-04-17T10:29:46.593868+0000","last_fullsized":"2024-04-17T10:29:46.593868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597311+0000","last_change":"2024-04-17T10:29:46.597311+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597311+0000","last_undegraded":"2024-04-17T10:29:46.597311+0000","last_fullsized":"2024-04-17T10:29:46.597311+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596099+0000","last_change":"2024-04-17T10:29:46.596099+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596099+0000","last_undegraded":"2024-04-17T10:29:46.596099+0000","last_fullsized":"2024-04-17T10:29:46.596099+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.591196+0000","last_change":"2024-04-17T10:29:46.591196+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.591196+0000","last_undegraded":"2024-04-17T10:29:46.591196+0000","last_fullsized":"2024-04-17T10:29:46.591196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593895+0000","last_change":"2024-04-17T10:29:46.593895+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593895+0000","last_undegraded":"2024-04-17T10:29:46.593895+0000","last_fullsized":"2024-04-17T10:29:46.593895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597137+0000","last_change":"2024-04-17T10:29:46.597137+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597137+0000","last_undegraded":"2024-04-17T10:29:46.597137+0000","last_fullsized":"2024-04-17T10:29:46.597137+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588803+0000","last_change":"2024-04-17T10:29:46.588803+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588803+0000","last_undegraded":"2024-04-17T10:29:46.588803+0000","last_fullsized":"2024-04-17T10:29:46.588803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593961+0000","last_change":"2024-04-17T10:29:46.593961+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593961+0000","last_undegraded":"2024-04-17T10:29:46.593961+0000","last_fullsized":"2024-04-17T10:29:46.593961+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590295+0000","last_change":"2024-04-17T10:29:46.590295+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590295+0000","last_undegraded":"2024-04-17T10:29:46.590295+0000","last_fullsized":"2024-04-17T10:29:46.590295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596053+0000","last_change":"2024-04-17T10:29:46.596053+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596053+0000","last_undegraded":"2024-04-17T10:29:46.596053+0000","last_fullsized":"2024-04-17T10:29:46.596053+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597382+0000","last_change":"2024-04-17T10:29:46.597382+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597382+0000","last_undegraded":"2024-04-17T10:29:46.597382+0000","last_fullsized":"2024-04-17T10:29:46.597382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590406+0000","last_change":"2024-04-17T10:29:46.590406+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590406+0000","last_undegraded":"2024-04-17T10:29:46.590406+0000","last_fullsized":"2024-04-17T10:29:46.590406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.586934+0000","last_change":"2024-04-17T10:29:46.586934+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.586934+0000","last_undegraded":"2024-04-17T10:29:46.586934+0000","last_fullsized":"2024-04-17T10:29:46.586934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596551+0000","last_change":"2024-04-17T10:29:46.596551+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596551+0000","last_undegraded":"2024-04-17T10:29:46.596551+0000","last_fullsized":"2024-04-17T10:29:46.596551+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.594831+0000","last_change":"2024-04-17T10:29:46.594831+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.594831+0000","last_undegraded":"2024-04-17T10:29:46.594831+0000","last_fullsized":"2024-04-17T10:29:46.594831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589777+0000","last_change":"2024-04-17T10:29:46.589777+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589777+0000","last_undegraded":"2024-04-17T10:29:46.589777+0000","last_fullsized":"2024-04-17T10:29:46.589777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589706+0000","last_change":"2024-04-17T10:29:46.589706+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589706+0000","last_undegraded":"2024-04-17T10:29:46.589706+0000","last_fullsized":"2024-04-17T10:29:46.589706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589740+0000","last_change":"2024-04-17T10:29:46.589740+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589740+0000","last_undegraded":"2024-04-17T10:29:46.589740+0000","last_fullsized":"2024-04-17T10:29:46.589740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593995+0000","last_change":"2024-04-17T10:29:46.593995+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593995+0000","last_undegraded":"2024-04-17T10:29:46.593995+0000","last_fullsized":"2024-04-17T10:29:46.593995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597315+0000","last_change":"2024-04-17T10:29:46.597315+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597315+0000","last_undegraded":"2024-04-17T10:29:46.597315+0000","last_fullsized":"2024-04-17T10:29:46.597315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590636+0000","last_change":"2024-04-17T10:29:46.590636+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590636+0000","last_undegraded":"2024-04-17T10:29:46.590636+0000","last_fullsized":"2024-04-17T10:29:46.590636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590408+0000","last_change":"2024-04-17T10:29:46.590408+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590408+0000","last_undegraded":"2024-04-17T10:29:46.590408+0000","last_fullsized":"2024-04-17T10:29:46.590408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588620+0000","last_change":"2024-04-17T10:29:46.588620+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588620+0000","last_undegraded":"2024-04-17T10:29:46.588620+0000","last_fullsized":"2024-04-17T10:29:46.588620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596313+0000","last_change":"2024-04-17T10:29:46.596313+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596313+0000","last_undegraded":"2024-04-17T10:29:46.596313+0000","last_fullsized":"2024-04-17T10:29:46.596313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588432+0000","last_change":"2024-04-17T10:29:46.588432+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588432+0000","last_undegraded":"2024-04-17T10:29:46.588432+0000","last_fullsized":"2024-04-17T10:29:46.588432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.594395+0000","last_change":"2024-04-17T10:29:46.594395+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.594395+0000","last_undegraded":"2024-04-17T10:29:46.594395+0000","last_fullsized":"2024-04-17T10:29:46.594395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589777+0000","last_change":"2024-04-17T10:29:46.589777+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589777+0000","last_undegraded":"2024-04-17T10:29:46.589777+0000","last_fullsized":"2024-04-17T10:29:46.589777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":280,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-17T10:29:46.591453+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:29:46.591453+0000","last_peered":"2024-04-17T10:29:46.591453+0000","last_clean":"2024-04-17T10:29:46.591453+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:29:46.591453+0000","last_undegraded":"2024-04-17T10:29:46.591453+0000","last_fullsized":"2024-04-17T10:29:46.591453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1560576,"data_stored":1035076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40452,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331388,"statfs":{"total":96636764160,"available":96595341312,"internally_reserved":0,"allocated":966656,"data_stored":444708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335552,"statfs":{"total":96636764160,"available":96599605248,"internally_reserved":0,"allocated":966656,"data_stored":444708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63248,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308592,"statfs":{"total":96636764160,"available":96571998208,"internally_reserved":0,"allocated":1572864,"data_stored":1041285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:52.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 Processor -- start 2024-04-17T10:29:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 -- start start 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815dd10 0x7f8df80b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8df80b57b0 0x7f8df8070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8df80b07b0 0x7f8df8071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df80b1280 con 0x7f8df80b57b0 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df80b0cc0 con 0x7f8df80b07b0 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8dfdc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df80b0fa0 con 0x7f8df815dd10 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8df80b57b0 0x7f8df8070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8df80b57b0 0x7f8df8070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57252/0 (socket says 172.21.15.67:57252) 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 -- 172.21.15.67:0/3248001649 learned_addr learned my addr 172.21.15.67:0/3248001649 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:52.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df7400640 1 --2- 172.21.15.67:0/3248001649 >> v2:172.21.15.86:3300/0 conn(0x7f8df80b07b0 0x7f8df8071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6a00640 1 --2- 172.21.15.67:0/3248001649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815dd10 0x7f8df80b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 -- 172.21.15.67:0/3248001649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815dd10 msgr2=0x7f8df80b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 --2- 172.21.15.67:0/3248001649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815dd10 0x7f8df80b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 -- 172.21.15.67:0/3248001649 >> v2:172.21.15.86:3300/0 conn(0x7f8df80b07b0 msgr2=0x7f8df8071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 --2- 172.21.15.67:0/3248001649 >> v2:172.21.15.86:3300/0 conn(0x7f8df80b07b0 0x7f8df8071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.905+0000 7f8df6000640 1 -- 172.21.15.67:0/3248001649 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df806bed0 con 0x7f8df80b57b0 2024-04-17T10:29:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8df6000640 1 --2- 172.21.15.67:0/3248001649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b57b0 0x7f8df8070dc0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8df8071b90 tx=0x7f8de401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4c6f606716c2e2f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8df5600640 1 -- 172.21.15.67:0/3248001649 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40ab600 con 0x7f8df80b57b0 2024-04-17T10:29:52.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8df5600640 1 -- 172.21.15.67:0/3248001649 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de40b0070 con 0x7f8df80b57b0 2024-04-17T10:29:52.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8df5600640 1 -- 172.21.15.67:0/3248001649 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de40ab9c0 con 0x7f8df80b57b0 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 -- 172.21.15.67:0/3248001649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b57b0 msgr2=0x7f8df8070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/3248001649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b57b0 0x7f8df8070dc0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f8df8071b90 tx=0x7f8de401c460 comp rx=0 tx=0).stop 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 -- 172.21.15.67:0/3248001649 shutdown_connections 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/3248001649 >> v2:172.21.15.86:3300/0 conn(0x7f8df80b07b0 0x7f8df8071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/3248001649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b57b0 0x7f8df8070dc0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/3248001649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815dd10 0x7f8df80b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 -- 172.21.15.67:0/3248001649 >> 172.21.15.67:0/3248001649 conn(0x7f8df80b34d0 msgr2=0x7f8df80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 -- 172.21.15.67:0/3248001649 shutdown_connections 2024-04-17T10:29:52.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 -- 172.21.15.67:0/3248001649 wait complete. 2024-04-17T10:29:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 Processor -- start 2024-04-17T10:29:52.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.909+0000 7f8dfdc00640 1 -- start start 2024-04-17T10:29:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b07b0 0x7f8df80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df815dd10 0x7f8df80cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815e340 0x7f8df8149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df8162460 con 0x7f8df80b07b0 2024-04-17T10:29:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df81622e0 con 0x7f8df815dd10 2024-04-17T10:29:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df81625e0 con 0x7f8df815e340 2024-04-17T10:29:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b07b0 0x7f8df80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b07b0 0x7f8df80b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57266/0 (socket says 172.21.15.67:57266) 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 -- 172.21.15.67:0/463487650 learned_addr learned my addr 172.21.15.67:0/463487650 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6000640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df815dd10 0x7f8df80cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df7400640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815e340 0x7f8df8149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 -- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815e340 msgr2=0x7f8df8149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815e340 0x7f8df8149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 -- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df815dd10 msgr2=0x7f8df80cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df815dd10 0x7f8df80cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 -- 172.21.15.67:0/463487650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df81540d0 con 0x7f8df80b07b0 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df7400640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815e340 0x7f8df8149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8df6a00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b07b0 0x7f8df80b1660 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8de800e870 tx=0x7f8de800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8deea00640 1 -- 172.21.15.67:0/463487650 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de8017070 con 0x7f8df80b07b0 2024-04-17T10:29:52.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8deea00640 1 -- 172.21.15.67:0/463487650 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de800c940 con 0x7f8df80b07b0 2024-04-17T10:29:52.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8deea00640 1 -- 172.21.15.67:0/463487650 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de800ccf0 con 0x7f8df80b07b0 2024-04-17T10:29:52.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df81543b0 con 0x7f8df80b07b0 2024-04-17T10:29:52.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df8154970 con 0x7f8df80b07b0 2024-04-17T10:29:52.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.913+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df8070bb0 con 0x7f8df80b07b0 2024-04-17T10:29:52.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.917+0000 7f8deea00640 1 -- 172.21.15.67:0/463487650 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8de801a070 con 0x7f8df80b07b0 2024-04-17T10:29:52.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.917+0000 7f8deea00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8dd8066f60 0x7f8dd8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:52.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.917+0000 7f8deea00640 1 -- 172.21.15.67:0/463487650 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8de80853b0 con 0x7f8df80b07b0 2024-04-17T10:29:52.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.917+0000 7f8df6000640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8dd8066f60 0x7f8dd8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:52.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.917+0000 7f8df6000640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8dd8066f60 0x7f8dd8069420 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8de40af000 tx=0x7f8de40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:52.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:52.921+0000 7f8deea00640 1 -- 172.21.15.67:0/463487650 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8de8055bd0 con 0x7f8df80b07b0 2024-04-17T10:29:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.077+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8df8153a90 con 0x7f8dd8066f60 2024-04-17T10:29:53.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.085+0000 7f8deea00640 1 -- 172.21.15.67:0/463487650 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240186 (secure 0 0 0) 0x7f8df8153a90 con 0x7f8dd8066f60 2024-04-17T10:29:53.083 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:53.084 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8dd8066f60 msgr2=0x7f8dd8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8dd8066f60 0x7f8dd8069420 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8de40af000 tx=0x7f8de40a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b07b0 msgr2=0x7f8df80b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b07b0 0x7f8df80b1660 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f8de800e870 tx=0x7f8de800ec70 comp rx=0 tx=0).stop 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 shutdown_connections 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8dd8066f60 0x7f8dd8069420 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8df815e340 0x7f8df8149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8df815dd10 0x7f8df80cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 --2- 172.21.15.67:0/463487650 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8df80b07b0 0x7f8df80b1660 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 >> 172.21.15.67:0/463487650 conn(0x7f8df80b34d0 msgr2=0x7f8df81563c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 shutdown_connections 2024-04-17T10:29:53.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.089+0000 7f8dfdc00640 1 -- 172.21.15.67:0/463487650 wait complete. 2024-04-17T10:29:53.101 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-04-17T10:29:52.204814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":322060,"kb_used_data":8772,"kb_used_omap":12,"kb_used_meta":313139,"kb_avail":754652660,"statfs":{"total":773094113280,"available":772764323840,"internally_reserved":0,"allocated":8982528,"data_stored":4769445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":320654897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.619489"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596550+0000","last_change":"2024-04-17T10:29:46.596550+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596550+0000","last_undegraded":"2024-04-17T10:29:46.596550+0000","last_fullsized":"2024-04-17T10:29:46.596550+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590391+0000","last_change":"2024-04-17T10:29:46.590391+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590391+0000","last_undegraded":"2024-04-17T10:29:46.590391+0000","last_fullsized":"2024-04-17T10:29:46.590391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.591131+0000","last_change":"2024-04-17T10:29:46.591131+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.591131+0000","last_undegraded":"2024-04-17T10:29:46.591131+0000","last_fullsized":"2024-04-17T10:29:46.591131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589295+0000","last_change":"2024-04-17T10:29:46.589295+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589295+0000","last_undegraded":"2024-04-17T10:29:46.589295+0000","last_fullsized":"2024-04-17T10:29:46.589295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.586939+0000","last_change":"2024-04-17T10:29:46.586939+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.586939+0000","last_undegraded":"2024-04-17T10:29:46.586939+0000","last_fullsized":"2024-04-17T10:29:46.586939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593868+0000","last_change":"2024-04-17T10:29:46.593868+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593868+0000","last_undegraded":"2024-04-17T10:29:46.593868+0000","last_fullsized":"2024-04-17T10:29:46.593868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597311+0000","last_change":"2024-04-17T10:29:46.597311+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597311+0000","last_undegraded":"2024-04-17T10:29:46.597311+0000","last_fullsized":"2024-04-17T10:29:46.597311+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596099+0000","last_change":"2024-04-17T10:29:46.596099+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596099+0000","last_undegraded":"2024-04-17T10:29:46.596099+0000","last_fullsized":"2024-04-17T10:29:46.596099+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.591196+0000","last_change":"2024-04-17T10:29:46.591196+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.591196+0000","last_undegraded":"2024-04-17T10:29:46.591196+0000","last_fullsized":"2024-04-17T10:29:46.591196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593895+0000","last_change":"2024-04-17T10:29:46.593895+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593895+0000","last_undegraded":"2024-04-17T10:29:46.593895+0000","last_fullsized":"2024-04-17T10:29:46.593895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597137+0000","last_change":"2024-04-17T10:29:46.597137+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597137+0000","last_undegraded":"2024-04-17T10:29:46.597137+0000","last_fullsized":"2024-04-17T10:29:46.597137+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588803+0000","last_change":"2024-04-17T10:29:46.588803+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588803+0000","last_undegraded":"2024-04-17T10:29:46.588803+0000","last_fullsized":"2024-04-17T10:29:46.588803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593961+0000","last_change":"2024-04-17T10:29:46.593961+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593961+0000","last_undegraded":"2024-04-17T10:29:46.593961+0000","last_fullsized":"2024-04-17T10:29:46.593961+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590295+0000","last_change":"2024-04-17T10:29:46.590295+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590295+0000","last_undegraded":"2024-04-17T10:29:46.590295+0000","last_fullsized":"2024-04-17T10:29:46.590295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596053+0000","last_change":"2024-04-17T10:29:46.596053+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596053+0000","last_undegraded":"2024-04-17T10:29:46.596053+0000","last_fullsized":"2024-04-17T10:29:46.596053+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597382+0000","last_change":"2024-04-17T10:29:46.597382+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597382+0000","last_undegraded":"2024-04-17T10:29:46.597382+0000","last_fullsized":"2024-04-17T10:29:46.597382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590406+0000","last_change":"2024-04-17T10:29:46.590406+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590406+0000","last_undegraded":"2024-04-17T10:29:46.590406+0000","last_fullsized":"2024-04-17T10:29:46.590406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.586934+0000","last_change":"2024-04-17T10:29:46.586934+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.586934+0000","last_undegraded":"2024-04-17T10:29:46.586934+0000","last_fullsized":"2024-04-17T10:29:46.586934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596551+0000","last_change":"2024-04-17T10:29:46.596551+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596551+0000","last_undegraded":"2024-04-17T10:29:46.596551+0000","last_fullsized":"2024-04-17T10:29:46.596551+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.594831+0000","last_change":"2024-04-17T10:29:46.594831+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.594831+0000","last_undegraded":"2024-04-17T10:29:46.594831+0000","last_fullsized":"2024-04-17T10:29:46.594831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589777+0000","last_change":"2024-04-17T10:29:46.589777+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589777+0000","last_undegraded":"2024-04-17T10:29:46.589777+0000","last_fullsized":"2024-04-17T10:29:46.589777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589706+0000","last_change":"2024-04-17T10:29:46.589706+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589706+0000","last_undegraded":"2024-04-17T10:29:46.589706+0000","last_fullsized":"2024-04-17T10:29:46.589706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589740+0000","last_change":"2024-04-17T10:29:46.589740+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589740+0000","last_undegraded":"2024-04-17T10:29:46.589740+0000","last_fullsized":"2024-04-17T10:29:46.589740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593995+0000","last_change":"2024-04-17T10:29:46.593995+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593995+0000","last_undegraded":"2024-04-17T10:29:46.593995+0000","last_fullsized":"2024-04-17T10:29:46.593995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597315+0000","last_change":"2024-04-17T10:29:46.597315+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597315+0000","last_undegraded":"2024-04-17T10:29:46.597315+0000","last_fullsized":"2024-04-17T10:29:46.597315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590636+0000","last_change":"2024-04-17T10:29:46.590636+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590636+0000","last_undegraded":"2024-04-17T10:29:46.590636+0000","last_fullsized":"2024-04-17T10:29:46.590636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590408+0000","last_change":"2024-04-17T10:29:46.590408+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590408+0000","last_undegraded":"2024-04-17T10:29:46.590408+0000","last_fullsized":"2024-04-17T10:29:46.590408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588620+0000","last_change":"2024-04-17T10:29:46.588620+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588620+0000","last_undegraded":"2024-04-17T10:29:46.588620+0000","last_fullsized":"2024-04-17T10:29:46.588620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596313+0000","last_change":"2024-04-17T10:29:46.596313+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596313+0000","last_undegraded":"2024-04-17T10:29:46.596313+0000","last_fullsized":"2024-04-17T10:29:46.596313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588432+0000","last_change":"2024-04-17T10:29:46.588432+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588432+0000","last_undegraded":"2024-04-17T10:29:46.588432+0000","last_fullsized":"2024-04-17T10:29:46.588432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.594395+0000","last_change":"2024-04-17T10:29:46.594395+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.594395+0000","last_undegraded":"2024-04-17T10:29:46.594395+0000","last_fullsized":"2024-04-17T10:29:46.594395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589777+0000","last_change":"2024-04-17T10:29:46.589777+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589777+0000","last_undegraded":"2024-04-17T10:29:46.589777+0000","last_fullsized":"2024-04-17T10:29:46.589777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":280,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-17T10:29:46.591453+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:29:46.591453+0000","last_peered":"2024-04-17T10:29:46.591453+0000","last_clean":"2024-04-17T10:29:46.591453+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:29:46.591453+0000","last_undegraded":"2024-04-17T10:29:46.591453+0000","last_fullsized":"2024-04-17T10:29:46.591453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1560576,"data_stored":1035076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40452,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331388,"statfs":{"total":96636764160,"available":96595341312,"internally_reserved":0,"allocated":966656,"data_stored":444708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335552,"statfs":{"total":96636764160,"available":96599605248,"internally_reserved":0,"allocated":966656,"data_stored":444708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63248,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308592,"statfs":{"total":96636764160,"available":96571998208,"internally_reserved":0,"allocated":1572864,"data_stored":1041285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:53.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.289+0000 7f9fc3400640 1 Processor -- start 2024-04-17T10:29:53.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.289+0000 7f9fc3400640 1 -- start start 2024-04-17T10:29:53.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc3400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9fbc144aa0 0x7f9fbc0d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fbc071840 0x7f9fbc0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc3400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0b51a0 con 0x7f9fbc071840 2024-04-17T10:29:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc3400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0d1390 con 0x7f9fbc144aa0 2024-04-17T10:29:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc3400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc1456d0 con 0x7f9fbc1446c0 2024-04-17T10:29:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fbc071840 0x7f9fbc0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9fbc071840 0x7f9fbc0d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57282/0 (socket says 172.21.15.67:57282) 2024-04-17T10:29:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 -- 172.21.15.67:0/2455779056 learned_addr learned my addr 172.21.15.67:0/2455779056 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc0a00640 1 --2- 172.21.15.67:0/2455779056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fb3e00640 1 --2- 172.21.15.67:0/2455779056 >> v2:172.21.15.86:3300/0 conn(0x7f9fbc144aa0 0x7f9fbc0d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 -- 172.21.15.67:0/2455779056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 msgr2=0x7f9fbc144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 --2- 172.21.15.67:0/2455779056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 -- 172.21.15.67:0/2455779056 >> v2:172.21.15.86:3300/0 conn(0x7f9fbc144aa0 msgr2=0x7f9fbc0d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 --2- 172.21.15.67:0/2455779056 >> v2:172.21.15.86:3300/0 conn(0x7f9fbc144aa0 0x7f9fbc0d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.293+0000 7f9fc1400640 1 -- 172.21.15.67:0/2455779056 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0b4870 con 0x7f9fbc071840 2024-04-17T10:29:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc1400640 1 --2- 172.21.15.67:0/2455779056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc071840 0x7f9fbc0d0e50 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc068530 tx=0x7f9fb401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6085fedcf8282c7d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fbbe00640 1 -- 172.21.15.67:0/2455779056 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40a57b0 con 0x7f9fbc071840 2024-04-17T10:29:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fbbe00640 1 -- 172.21.15.67:0/2455779056 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb40a5950 con 0x7f9fbc071840 2024-04-17T10:29:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fbbe00640 1 -- 172.21.15.67:0/2455779056 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40a5d20 con 0x7f9fbc071840 2024-04-17T10:29:53.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 -- 172.21.15.67:0/2455779056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc071840 msgr2=0x7f9fbc0d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 --2- 172.21.15.67:0/2455779056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc071840 0x7f9fbc0d0e50 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc068530 tx=0x7f9fb401c930 comp rx=0 tx=0).stop 2024-04-17T10:29:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 -- 172.21.15.67:0/2455779056 shutdown_connections 2024-04-17T10:29:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 --2- 172.21.15.67:0/2455779056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc071840 0x7f9fbc0d0e50 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 --2- 172.21.15.67:0/2455779056 >> v2:172.21.15.86:3300/0 conn(0x7f9fbc144aa0 0x7f9fbc0d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 --2- 172.21.15.67:0/2455779056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 -- 172.21.15.67:0/2455779056 >> 172.21.15.67:0/2455779056 conn(0x7f9fbc0b34d0 msgr2=0x7f9fbc0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:53.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 -- 172.21.15.67:0/2455779056 shutdown_connections 2024-04-17T10:29:53.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 -- 172.21.15.67:0/2455779056 wait complete. 2024-04-17T10:29:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 Processor -- start 2024-04-17T10:29:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.297+0000 7f9fc3400640 1 -- start start 2024-04-17T10:29:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fbc071840 0x7f9fbc1580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc14b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc144aa0 0x7f9fbc153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc15fed0 con 0x7f9fbc144aa0 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc15c0d0 con 0x7f9fbc071840 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc160050 con 0x7f9fbc1446c0 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc144aa0 0x7f9fbc153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc144aa0 0x7f9fbc153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57286/0 (socket says 172.21.15.67:57286) 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 -- 172.21.15.67:0/3397058384 learned_addr learned my addr 172.21.15.67:0/3397058384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fb3e00640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc14b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc0a00640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fbc071840 0x7f9fbc1580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 -- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 msgr2=0x7f9fbc14b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc14b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 -- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fbc071840 msgr2=0x7f9fbc1580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fbc071840 0x7f9fbc1580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 -- 172.21.15.67:0/3397058384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc1f7140 con 0x7f9fbc144aa0 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc0a00640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fbc071840 0x7f9fbc1580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fb3e00640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc14b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc1400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc144aa0 0x7f9fbc153ca0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40af040 tx=0x7f9fb40196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fb9600640 1 -- 172.21.15.67:0/3397058384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40bc020 con 0x7f9fbc144aa0 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fb9600640 1 -- 172.21.15.67:0/3397058384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb40b8690 con 0x7f9fbc144aa0 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc1586c0 con 0x7f9fbc144aa0 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fb9600640 1 -- 172.21.15.67:0/3397058384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb40b8a80 con 0x7f9fbc144aa0 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc158c80 con 0x7f9fbc144aa0 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc070bb0 con 0x7f9fbc144aa0 2024-04-17T10:29:53.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.301+0000 7f9fb9600640 1 -- 172.21.15.67:0/3397058384 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9fb40bf020 con 0x7f9fbc144aa0 2024-04-17T10:29:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.305+0000 7f9fb9600640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f98066f60 0x7f9f98069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.305+0000 7f9fb9600640 1 -- 172.21.15.67:0/3397058384 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9fb41202e0 con 0x7f9fbc144aa0 2024-04-17T10:29:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.305+0000 7f9fc0a00640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f98066f60 0x7f9f98069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.305+0000 7f9fc0a00640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f98066f60 0x7f9f98069420 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9fac004660 tx=0x7f9fac008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:53.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.309+0000 7f9fb9600640 1 -- 172.21.15.67:0/3397058384 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9fb40c6030 con 0x7f9fbc144aa0 2024-04-17T10:29:53.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.465+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9fbc071070 con 0x7f9f98066f60 2024-04-17T10:29:53.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.473+0000 7f9fb9600640 1 -- 172.21.15.67:0/3397058384 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240186 (secure 0 0 0) 0x7f9fbc071070 con 0x7f9f98066f60 2024-04-17T10:29:53.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:53.469 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:53.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f98066f60 msgr2=0x7f9f98069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f98066f60 0x7f9f98069420 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f9fac004660 tx=0x7f9fac008040 comp rx=0 tx=0).stop 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc144aa0 msgr2=0x7f9fbc153ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc144aa0 0x7f9fbc153ca0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40af040 tx=0x7f9fb40196d0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 shutdown_connections 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f98066f60 0x7f9f98069420 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9fbc144aa0 0x7f9fbc153ca0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9fbc1446c0 0x7f9fbc14b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 --2- 172.21.15.67:0/3397058384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9fbc071840 0x7f9fbc1580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 >> 172.21.15.67:0/3397058384 conn(0x7f9fbc0b34d0 msgr2=0x7f9fbc15d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 shutdown_connections 2024-04-17T10:29:53.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.477+0000 7f9fc3400640 1 -- 172.21.15.67:0/3397058384 wait complete. 2024-04-17T10:29:53.488 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-04-17T10:29:52.204814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":322060,"kb_used_data":8772,"kb_used_omap":12,"kb_used_meta":313139,"kb_avail":754652660,"statfs":{"total":773094113280,"available":772764323840,"internally_reserved":0,"allocated":8982528,"data_stored":4769445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":320654897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.619489"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596550+0000","last_change":"2024-04-17T10:29:46.596550+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596550+0000","last_undegraded":"2024-04-17T10:29:46.596550+0000","last_fullsized":"2024-04-17T10:29:46.596550+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590391+0000","last_change":"2024-04-17T10:29:46.590391+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590391+0000","last_undegraded":"2024-04-17T10:29:46.590391+0000","last_fullsized":"2024-04-17T10:29:46.590391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.591131+0000","last_change":"2024-04-17T10:29:46.591131+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.591131+0000","last_undegraded":"2024-04-17T10:29:46.591131+0000","last_fullsized":"2024-04-17T10:29:46.591131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589295+0000","last_change":"2024-04-17T10:29:46.589295+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589295+0000","last_undegraded":"2024-04-17T10:29:46.589295+0000","last_fullsized":"2024-04-17T10:29:46.589295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.586939+0000","last_change":"2024-04-17T10:29:46.586939+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.586939+0000","last_undegraded":"2024-04-17T10:29:46.586939+0000","last_fullsized":"2024-04-17T10:29:46.586939+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593868+0000","last_change":"2024-04-17T10:29:46.593868+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593868+0000","last_undegraded":"2024-04-17T10:29:46.593868+0000","last_fullsized":"2024-04-17T10:29:46.593868+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597311+0000","last_change":"2024-04-17T10:29:46.597311+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597311+0000","last_undegraded":"2024-04-17T10:29:46.597311+0000","last_fullsized":"2024-04-17T10:29:46.597311+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596099+0000","last_change":"2024-04-17T10:29:46.596099+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596099+0000","last_undegraded":"2024-04-17T10:29:46.596099+0000","last_fullsized":"2024-04-17T10:29:46.596099+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.591196+0000","last_change":"2024-04-17T10:29:46.591196+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.591196+0000","last_undegraded":"2024-04-17T10:29:46.591196+0000","last_fullsized":"2024-04-17T10:29:46.591196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593895+0000","last_change":"2024-04-17T10:29:46.593895+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593895+0000","last_undegraded":"2024-04-17T10:29:46.593895+0000","last_fullsized":"2024-04-17T10:29:46.593895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597137+0000","last_change":"2024-04-17T10:29:46.597137+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597137+0000","last_undegraded":"2024-04-17T10:29:46.597137+0000","last_fullsized":"2024-04-17T10:29:46.597137+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588803+0000","last_change":"2024-04-17T10:29:46.588803+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588803+0000","last_undegraded":"2024-04-17T10:29:46.588803+0000","last_fullsized":"2024-04-17T10:29:46.588803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593961+0000","last_change":"2024-04-17T10:29:46.593961+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593961+0000","last_undegraded":"2024-04-17T10:29:46.593961+0000","last_fullsized":"2024-04-17T10:29:46.593961+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590295+0000","last_change":"2024-04-17T10:29:46.590295+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590295+0000","last_undegraded":"2024-04-17T10:29:46.590295+0000","last_fullsized":"2024-04-17T10:29:46.590295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596053+0000","last_change":"2024-04-17T10:29:46.596053+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596053+0000","last_undegraded":"2024-04-17T10:29:46.596053+0000","last_fullsized":"2024-04-17T10:29:46.596053+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597382+0000","last_change":"2024-04-17T10:29:46.597382+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597382+0000","last_undegraded":"2024-04-17T10:29:46.597382+0000","last_fullsized":"2024-04-17T10:29:46.597382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590406+0000","last_change":"2024-04-17T10:29:46.590406+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590406+0000","last_undegraded":"2024-04-17T10:29:46.590406+0000","last_fullsized":"2024-04-17T10:29:46.590406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.586934+0000","last_change":"2024-04-17T10:29:46.586934+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.586934+0000","last_undegraded":"2024-04-17T10:29:46.586934+0000","last_fullsized":"2024-04-17T10:29:46.586934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596551+0000","last_change":"2024-04-17T10:29:46.596551+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596551+0000","last_undegraded":"2024-04-17T10:29:46.596551+0000","last_fullsized":"2024-04-17T10:29:46.596551+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.594831+0000","last_change":"2024-04-17T10:29:46.594831+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.594831+0000","last_undegraded":"2024-04-17T10:29:46.594831+0000","last_fullsized":"2024-04-17T10:29:46.594831+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589777+0000","last_change":"2024-04-17T10:29:46.589777+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589777+0000","last_undegraded":"2024-04-17T10:29:46.589777+0000","last_fullsized":"2024-04-17T10:29:46.589777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589706+0000","last_change":"2024-04-17T10:29:46.589706+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589706+0000","last_undegraded":"2024-04-17T10:29:46.589706+0000","last_fullsized":"2024-04-17T10:29:46.589706+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589740+0000","last_change":"2024-04-17T10:29:46.589740+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589740+0000","last_undegraded":"2024-04-17T10:29:46.589740+0000","last_fullsized":"2024-04-17T10:29:46.589740+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.593995+0000","last_change":"2024-04-17T10:29:46.593995+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.593995+0000","last_undegraded":"2024-04-17T10:29:46.593995+0000","last_fullsized":"2024-04-17T10:29:46.593995+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.597315+0000","last_change":"2024-04-17T10:29:46.597315+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.597315+0000","last_undegraded":"2024-04-17T10:29:46.597315+0000","last_fullsized":"2024-04-17T10:29:46.597315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590636+0000","last_change":"2024-04-17T10:29:46.590636+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590636+0000","last_undegraded":"2024-04-17T10:29:46.590636+0000","last_fullsized":"2024-04-17T10:29:46.590636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.590408+0000","last_change":"2024-04-17T10:29:46.590408+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.590408+0000","last_undegraded":"2024-04-17T10:29:46.590408+0000","last_fullsized":"2024-04-17T10:29:46.590408+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588620+0000","last_change":"2024-04-17T10:29:46.588620+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588620+0000","last_undegraded":"2024-04-17T10:29:46.588620+0000","last_fullsized":"2024-04-17T10:29:46.588620+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.596313+0000","last_change":"2024-04-17T10:29:46.596313+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.596313+0000","last_undegraded":"2024-04-17T10:29:46.596313+0000","last_fullsized":"2024-04-17T10:29:46.596313+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.588432+0000","last_change":"2024-04-17T10:29:46.588432+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.588432+0000","last_undegraded":"2024-04-17T10:29:46.588432+0000","last_fullsized":"2024-04-17T10:29:46.588432+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.594395+0000","last_change":"2024-04-17T10:29:46.594395+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.594395+0000","last_undegraded":"2024-04-17T10:29:46.594395+0000","last_fullsized":"2024-04-17T10:29:46.594395+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:29:47.585936+0000","last_change":"2024-04-17T10:29:47.585936+0000","last_active":"2024-04-17T10:29:47.585936+0000","last_peered":"2024-04-17T10:29:47.585936+0000","last_clean":"2024-04-17T10:29:47.585936+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:47.585936+0000","last_undegraded":"2024-04-17T10:29:47.585936+0000","last_fullsized":"2024-04-17T10:29:47.585936+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":80,"state":"creating+peering","last_fresh":"2024-04-17T10:29:46.589777+0000","last_change":"2024-04-17T10:29:46.589777+0000","last_active":"2024-04-17T10:29:46.578850+0000","last_peered":"2024-04-17T10:29:46.578850+0000","last_clean":"2024-04-17T10:29:46.578850+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:29:46.589777+0000","last_undegraded":"2024-04-17T10:29:46.589777+0000","last_fullsized":"2024-04-17T10:29:46.589777+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":280,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-04-17T10:29:46.591453+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:29:46.591453+0000","last_peered":"2024-04-17T10:29:46.591453+0000","last_clean":"2024-04-17T10:29:46.591453+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:29:46.591453+0000","last_undegraded":"2024-04-17T10:29:46.591453+0000","last_fullsized":"2024-04-17T10:29:46.591453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36864,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334976,"statfs":{"total":96636764160,"available":96599015424,"internally_reserved":0,"allocated":1560576,"data_stored":1035076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40452,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331388,"statfs":{"total":96636764160,"available":96595341312,"internally_reserved":0,"allocated":966656,"data_stored":444708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36296,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335544,"statfs":{"total":96636764160,"available":96599597056,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36308,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335532,"statfs":{"total":96636764160,"available":96599584768,"internally_reserved":0,"allocated":978944,"data_stored":450917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36288,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335552,"statfs":{"total":96636764160,"available":96599605248,"internally_reserved":0,"allocated":966656,"data_stored":444708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63248,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308592,"statfs":{"total":96636764160,"available":96571998208,"internally_reserved":0,"allocated":1572864,"data_stored":1041285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:53.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3f is not active+clean 2024-04-17T10:29:53.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3e is not active+clean 2024-04-17T10:29:53.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.496 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3d is not active+clean 2024-04-17T10:29:53.496 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3c is not active+clean 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3b is not active+clean 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3a is not active+clean 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.39 is not active+clean 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.497 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.38 is not active+clean 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.37 is not active+clean 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.36 is not active+clean 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.35 is not active+clean 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.34 is not active+clean 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.498 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.33 is not active+clean 2024-04-17T10:29:53.499 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.499 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.32 is not active+clean 2024-04-17T10:29:53.499 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.499 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.31 is not active+clean 2024-04-17T10:29:53.499 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.499 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.30 is not active+clean 2024-04-17T10:29:53.499 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.500 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2f is not active+clean 2024-04-17T10:29:53.500 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.500 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2e is not active+clean 2024-04-17T10:29:53.500 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.500 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2d is not active+clean 2024-04-17T10:29:53.500 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.500 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2c is not active+clean 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2b is not active+clean 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2a is not active+clean 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.29 is not active+clean 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.28 is not active+clean 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.27 is not active+clean 2024-04-17T10:29:53.501 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.26 is not active+clean 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.25 is not active+clean 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.24 is not active+clean 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.23 is not active+clean 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.22 is not active+clean 2024-04-17T10:29:53.502 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.21 is not active+clean 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.20 is not active+clean 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.18 is not active+clean 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.596550+0000', 'last_change': '2024-04-17T10:29:46.596550+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.596550+0000', 'last_undegraded': '2024-04-17T10:29:46.596550+0000', 'last_fullsized': '2024-04-17T10:29:46.596550+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.19 is not active+clean 2024-04-17T10:29:53.503 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.590391+0000', 'last_change': '2024-04-17T10:29:46.590391+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.590391+0000', 'last_undegraded': '2024-04-17T10:29:46.590391+0000', 'last_fullsized': '2024-04-17T10:29:46.590391+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1a is not active+clean 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.591131+0000', 'last_change': '2024-04-17T10:29:46.591131+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.591131+0000', 'last_undegraded': '2024-04-17T10:29:46.591131+0000', 'last_fullsized': '2024-04-17T10:29:46.591131+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1b is not active+clean 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.589295+0000', 'last_change': '2024-04-17T10:29:46.589295+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.589295+0000', 'last_undegraded': '2024-04-17T10:29:46.589295+0000', 'last_fullsized': '2024-04-17T10:29:46.589295+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:29:53.504 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1c is not active+clean 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.586939+0000', 'last_change': '2024-04-17T10:29:46.586939+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.586939+0000', 'last_undegraded': '2024-04-17T10:29:46.586939+0000', 'last_fullsized': '2024-04-17T10:29:46.586939+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1d is not active+clean 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.593868+0000', 'last_change': '2024-04-17T10:29:46.593868+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.593868+0000', 'last_undegraded': '2024-04-17T10:29:46.593868+0000', 'last_fullsized': '2024-04-17T10:29:46.593868+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1e is not active+clean 2024-04-17T10:29:53.505 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.597311+0000', 'last_change': '2024-04-17T10:29:46.597311+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.597311+0000', 'last_undegraded': '2024-04-17T10:29:46.597311+0000', 'last_fullsized': '2024-04-17T10:29:46.597311+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1f is not active+clean 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.596099+0000', 'last_change': '2024-04-17T10:29:46.596099+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.596099+0000', 'last_undegraded': '2024-04-17T10:29:46.596099+0000', 'last_fullsized': '2024-04-17T10:29:46.596099+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.10 is not active+clean 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.591196+0000', 'last_change': '2024-04-17T10:29:46.591196+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.591196+0000', 'last_undegraded': '2024-04-17T10:29:46.591196+0000', 'last_fullsized': '2024-04-17T10:29:46.591196+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:53.506 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.11 is not active+clean 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.593895+0000', 'last_change': '2024-04-17T10:29:46.593895+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.593895+0000', 'last_undegraded': '2024-04-17T10:29:46.593895+0000', 'last_fullsized': '2024-04-17T10:29:46.593895+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.12 is not active+clean 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.597137+0000', 'last_change': '2024-04-17T10:29:46.597137+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.597137+0000', 'last_undegraded': '2024-04-17T10:29:46.597137+0000', 'last_fullsized': '2024-04-17T10:29:46.597137+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.13 is not active+clean 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-04-17T10:29:53.507 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.588803+0000', 'last_change': '2024-04-17T10:29:46.588803+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.588803+0000', 'last_undegraded': '2024-04-17T10:29:46.588803+0000', 'last_fullsized': '2024-04-17T10:29:46.588803+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.14 is not active+clean 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.593961+0000', 'last_change': '2024-04-17T10:29:46.593961+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.593961+0000', 'last_undegraded': '2024-04-17T10:29:46.593961+0000', 'last_fullsized': '2024-04-17T10:29:46.593961+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.15 is not active+clean 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.590295+0000', 'last_change': '2024-04-17T10:29:46.590295+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.590295+0000', 'last_undegraded': '2024-04-17T10:29:46.590295+0000', 'last_fullsized': '2024-04-17T10:29:46.590295+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.16 is not active+clean 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.508 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.596053+0000', 'last_change': '2024-04-17T10:29:46.596053+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.596053+0000', 'last_undegraded': '2024-04-17T10:29:46.596053+0000', 'last_fullsized': '2024-04-17T10:29:46.596053+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.17 is not active+clean 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.597382+0000', 'last_change': '2024-04-17T10:29:46.597382+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.597382+0000', 'last_undegraded': '2024-04-17T10:29:46.597382+0000', 'last_fullsized': '2024-04-17T10:29:46.597382+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.8 is not active+clean 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.590406+0000', 'last_change': '2024-04-17T10:29:46.590406+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.590406+0000', 'last_undegraded': '2024-04-17T10:29:46.590406+0000', 'last_fullsized': '2024-04-17T10:29:46.590406+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.9 is not active+clean 2024-04-17T10:29:53.509 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.586934+0000', 'last_change': '2024-04-17T10:29:46.586934+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.586934+0000', 'last_undegraded': '2024-04-17T10:29:46.586934+0000', 'last_fullsized': '2024-04-17T10:29:46.586934+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.a is not active+clean 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.596551+0000', 'last_change': '2024-04-17T10:29:46.596551+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.596551+0000', 'last_undegraded': '2024-04-17T10:29:46.596551+0000', 'last_fullsized': '2024-04-17T10:29:46.596551+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.b is not active+clean 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-04-17T10:29:53.510 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.594831+0000', 'last_change': '2024-04-17T10:29:46.594831+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.594831+0000', 'last_undegraded': '2024-04-17T10:29:46.594831+0000', 'last_fullsized': '2024-04-17T10:29:46.594831+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.c is not active+clean 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.589777+0000', 'last_change': '2024-04-17T10:29:46.589777+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.589777+0000', 'last_undegraded': '2024-04-17T10:29:46.589777+0000', 'last_fullsized': '2024-04-17T10:29:46.589777+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.e is not active+clean 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.589706+0000', 'last_change': '2024-04-17T10:29:46.589706+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.589706+0000', 'last_undegraded': '2024-04-17T10:29:46.589706+0000', 'last_fullsized': '2024-04-17T10:29:46.589706+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.f is not active+clean 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.511 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-04-17T10:29:53.512 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.589740+0000', 'last_change': '2024-04-17T10:29:46.589740+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.589740+0000', 'last_undegraded': '2024-04-17T10:29:46.589740+0000', 'last_fullsized': '2024-04-17T10:29:46.589740+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:53.512 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.0 is not active+clean 2024-04-17T10:29:53.512 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.512 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-04-17T10:29:53.512 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.593995+0000', 'last_change': '2024-04-17T10:29:46.593995+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.593995+0000', 'last_undegraded': '2024-04-17T10:29:46.593995+0000', 'last_fullsized': '2024-04-17T10:29:46.593995+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:29:53.512 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1 is not active+clean 2024-04-17T10:29:53.512 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.597315+0000', 'last_change': '2024-04-17T10:29:46.597315+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.597315+0000', 'last_undegraded': '2024-04-17T10:29:46.597315+0000', 'last_fullsized': '2024-04-17T10:29:46.597315+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2 is not active+clean 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.590636+0000', 'last_change': '2024-04-17T10:29:46.590636+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.590636+0000', 'last_undegraded': '2024-04-17T10:29:46.590636+0000', 'last_fullsized': '2024-04-17T10:29:46.590636+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3 is not active+clean 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.590408+0000', 'last_change': '2024-04-17T10:29:46.590408+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.590408+0000', 'last_undegraded': '2024-04-17T10:29:46.590408+0000', 'last_fullsized': '2024-04-17T10:29:46.590408+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:29:53.513 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.4 is not active+clean 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.588620+0000', 'last_change': '2024-04-17T10:29:46.588620+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.588620+0000', 'last_undegraded': '2024-04-17T10:29:46.588620+0000', 'last_fullsized': '2024-04-17T10:29:46.588620+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.5 is not active+clean 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.596313+0000', 'last_change': '2024-04-17T10:29:46.596313+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.596313+0000', 'last_undegraded': '2024-04-17T10:29:46.596313+0000', 'last_fullsized': '2024-04-17T10:29:46.596313+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.6 is not active+clean 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.514 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.588432+0000', 'last_change': '2024-04-17T10:29:46.588432+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.588432+0000', 'last_undegraded': '2024-04-17T10:29:46.588432+0000', 'last_fullsized': '2024-04-17T10:29:46.588432+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.7 is not active+clean 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.594395+0000', 'last_change': '2024-04-17T10:29:46.594395+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.594395+0000', 'last_undegraded': '2024-04-17T10:29:46.594395+0000', 'last_fullsized': '2024-04-17T10:29:46.594395+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.d is not active+clean 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:29:47.585936+0000', 'last_change': '2024-04-17T10:29:47.585936+0000', 'last_active': '2024-04-17T10:29:47.585936+0000', 'last_peered': '2024-04-17T10:29:47.585936+0000', 'last_clean': '2024-04-17T10:29:47.585936+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:47.585936+0000', 'last_undegraded': '2024-04-17T10:29:47.585936+0000', 'last_fullsized': '2024-04-17T10:29:47.585936+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:47.585936+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-04-17T10:29:53.515 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 80, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:29:46.589777+0000', 'last_change': '2024-04-17T10:29:46.589777+0000', 'last_active': '2024-04-17T10:29:46.578850+0000', 'last_peered': '2024-04-17T10:29:46.578850+0000', 'last_clean': '2024-04-17T10:29:46.578850+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:29:46.589777+0000', 'last_undegraded': '2024-04-17T10:29:46.589777+0000', 'last_fullsized': '2024-04-17T10:29:46.589777+0000', 'mapping_epoch': 80, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 80, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'last_clean_scrub_stamp': '2024-04-17T10:29:46.578850+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-17T10:29:53.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:29:53.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef76600640 1 Processor -- start 2024-04-17T10:29:53.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef76600640 1 -- start start 2024-04-17T10:29:53.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef76600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fef7015b970 0x7fef700cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef76600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7015cf10 con 0x7fef7006b6a0 2024-04-17T10:29:53.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef76600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef700cf0c0 con 0x7fef700cd690 2024-04-17T10:29:53.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef76600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7015c300 con 0x7fef7015b970 2024-04-17T10:29:53.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef67e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef7006b6a0 0x7fef700ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50918/0 (socket says 172.21.15.67:50918) 2024-04-17T10:29:53.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef67e00640 1 -- 172.21.15.67:0/431454854 learned_addr learned my addr 172.21.15.67:0/431454854 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:53.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6f400640 1 --2- 172.21.15.67:0/431454854 >> v2:172.21.15.86:3300/0 conn(0x7fef7015b970 0x7fef700cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72239292 0 0) 0x7fef7015cf10 con 0x7fef7006b6a0 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef54003530 con 0x7fef7006b6a0 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1712073675 0 0) 0x7fef700cf0c0 con 0x7fef700cd690 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef7015cf10 con 0x7fef700cd690 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628250196 0 0) 0x7fef54003530 con 0x7fef7006b6a0 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 >> v2:172.21.15.86:3300/0 conn(0x7fef7015b970 msgr2=0x7fef700cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 --2- 172.21.15.67:0/431454854 >> v2:172.21.15.86:3300/0 conn(0x7fef7015b970 0x7fef700cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 >> v1:172.21.15.86:6789/0 conn(0x7fef700cd690 legacy=0x7fef700ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7015c5e0 con 0x7fef7006b6a0 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fef58001680 con 0x7fef7006b6a0 2024-04-17T10:29:53.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 >> v1:172.21.15.67:6789/0 conn(0x7fef7006b6a0 legacy=0x7fef700ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef54003b50 0x7fef54003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef54004810 0x7fef54004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef540051d0 0x7fef5400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7015c300 con 0x7fef54003b50 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef5400c0d0 con 0x7fef54004810 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5400c2d0 con 0x7fef540051d0 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef6f400640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef54003b50 0x7fef54003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.681+0000 7fef67e00640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef54004810 0x7fef54004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6fe00640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef540051d0 0x7fef5400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6f400640 1 -- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef540051d0 msgr2=0x7fef5400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6f400640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef540051d0 0x7fef5400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6f400640 1 -- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef54004810 msgr2=0x7fef54004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:53.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6f400640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef54004810 0x7fef54004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6f400640 1 -- 172.21.15.67:0/431454854 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef54003530 con 0x7fef54003b50 2024-04-17T10:29:53.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6fe00640 1 -- 172.21.15.67:0/431454854 reap_dead start 2024-04-17T10:29:53.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6f400640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef54003b50 0x7fef54003f30 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fef60009ae0 tx=0x7fef6001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146835dd39b0411d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef600096b0 con 0x7fef54003b50 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef600a6440 con 0x7fef54003b50 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef6ea00640 1 -- 172.21.15.67:0/431454854 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef600a6830 con 0x7fef54003b50 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef76600640 1 -- 172.21.15.67:0/431454854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef54003b50 msgr2=0x7fef54003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef76600640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef54003b50 0x7fef54003f30 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fef60009ae0 tx=0x7fef6001ca90 comp rx=0 tx=0).stop 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef76600640 1 -- 172.21.15.67:0/431454854 shutdown_connections 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef76600640 1 --2- 172.21.15.67:0/431454854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef54003b50 0x7fef54003f30 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.685+0000 7fef76600640 1 -- 172.21.15.67:0/431454854 >> 172.21.15.67:0/431454854 conn(0x7fef700b22a0 msgr2=0x7fef700b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 -- 172.21.15.67:0/431454854 shutdown_connections 2024-04-17T10:29:53.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 -- 172.21.15.67:0/431454854 wait complete. 2024-04-17T10:29:53.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 Processor -- start 2024-04-17T10:29:53.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 -- start start 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef54003b50 0x7fef70140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef700b52b0 0x7fef7014ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef70158020 0x7fef700b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef700ce2a0 con 0x7fef70158020 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef700704d0 con 0x7fef700b52b0 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef700ce420 con 0x7fef54003b50 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef70158020 0x7fef700b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef70158020 0x7fef700b43e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57308/0 (socket says 172.21.15.67:57308) 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 -- 172.21.15.67:0/3034883599 learned_addr learned my addr 172.21.15.67:0/3034883599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6f400640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef54003b50 0x7fef70140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef67e00640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef700b52b0 0x7fef7014ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 -- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef54003b50 msgr2=0x7fef70140bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef54003b50 0x7fef70140bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 -- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef700b52b0 msgr2=0x7fef7014ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef700b52b0 0x7fef7014ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 -- 172.21.15.67:0/3034883599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef70159680 con 0x7fef70158020 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef6fe00640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef70158020 0x7fef700b43e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fef700cdb20 tx=0x7fef6800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef67400640 1 -- 172.21.15.67:0/3034883599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6800ee50 con 0x7fef70158020 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef67400640 1 -- 172.21.15.67:0/3034883599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef68006d70 con 0x7fef70158020 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef70158920 con 0x7fef70158020 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.689+0000 7fef67400640 1 -- 172.21.15.67:0/3034883599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68003bc0 con 0x7fef70158020 2024-04-17T10:29:53.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.693+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef70158ee0 con 0x7fef70158020 2024-04-17T10:29:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.693+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7009e900 con 0x7fef70158020 2024-04-17T10:29:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.693+0000 7fef67400640 1 -- 172.21.15.67:0/3034883599 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fef68005070 con 0x7fef70158020 2024-04-17T10:29:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.693+0000 7fef67400640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef4c066f60 0x7fef4c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.693+0000 7fef67400640 1 -- 172.21.15.67:0/3034883599 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fef68084940 con 0x7fef70158020 2024-04-17T10:29:53.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.693+0000 7fef6f400640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef4c066f60 0x7fef4c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:53.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.693+0000 7fef6f400640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef4c066f60 0x7fef4c069420 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fef600ad000 tx=0x7fef600b2040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:53.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.697+0000 7fef67400640 1 -- 172.21.15.67:0/3034883599 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fef68055170 con 0x7fef70158020 2024-04-17T10:29:53.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.913+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fef70074e30 con 0x7fef70158020 2024-04-17T10:29:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef67400640 1 -- 172.21.15.67:0/3034883599 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1658 (secure 0 0 0) 0x7fef68059210 con 0x7fef70158020 2024-04-17T10:29:53.909 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:53.909 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":324,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":83,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":329789440,"bytes_avail":772764323840,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":55,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":7206}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-04-17T10:29:36.200874+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef4c066f60 msgr2=0x7fef4c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef4c066f60 0x7fef4c069420 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fef600ad000 tx=0x7fef600b2040 comp rx=0 tx=0).stop 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef70158020 msgr2=0x7fef700b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef70158020 0x7fef700b43e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fef700cdb20 tx=0x7fef6800ec00 comp rx=0 tx=0).stop 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 shutdown_connections 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef4c066f60 0x7fef4c069420 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef70158020 0x7fef700b43e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef700b52b0 0x7fef7014ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 --2- 172.21.15.67:0/3034883599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef54003b50 0x7fef70140bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 >> 172.21.15.67:0/3034883599 conn(0x7fef700b22a0 msgr2=0x7fef700b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 shutdown_connections 2024-04-17T10:29:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:53.917+0000 7fef76600640 1 -- 172.21.15.67:0/3034883599 wait complete. 2024-04-17T10:29:53.930 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-17T10:29:53.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:54.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7faaa00640 1 Processor -- start 2024-04-17T10:29:54.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7faaa00640 1 -- start start 2024-04-17T10:29:54.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7faaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fa415d9e0 0x7f7fa40afe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:54.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7faaa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa415ed90 con 0x7f7fa4162260 2024-04-17T10:29:54.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7faaa00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa4161780 con 0x7f7fa406b3a0 2024-04-17T10:29:54.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7faaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa4161dd0 con 0x7f7fa415d9e0 2024-04-17T10:29:54.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7fa9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fa415d9e0 0x7f7fa40afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:54.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7fa8c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7fa4162260 0x7f7fa406ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50934/0 (socket says 172.21.15.67:50934) 2024-04-17T10:29:54.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7fa8c00640 1 -- 172.21.15.67:0/933807818 learned_addr learned my addr 172.21.15.67:0/933807818 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:54.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7fa9600640 1 -- 172.21.15.67:0/933807818 >> v1:172.21.15.86:6789/0 conn(0x7f7fa406b3a0 legacy=0x7f7fa40ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:54.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7fa9600640 1 -- 172.21.15.67:0/933807818 >> v1:172.21.15.67:6789/0 conn(0x7f7fa4162260 legacy=0x7f7fa406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:54.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7fa9600640 1 -- 172.21.15.67:0/933807818 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa4161a60 con 0x7f7fa415d9e0 2024-04-17T10:29:54.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.077+0000 7f7fa3e00640 1 -- 172.21.15.67:0/933807818 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15611140 0 0) 0x7f7fa415ed90 con 0x7f7fa4162260 2024-04-17T10:29:54.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7fa9600640 1 --2- 172.21.15.67:0/933807818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa415d9e0 0x7f7fa40afe10 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7f980097c0 tx=0x7f7f9801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca823bd32e560dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:54.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7fa3e00640 1 -- 172.21.15.67:0/933807818 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f980b0070 con 0x7f7fa415d9e0 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7fa3e00640 1 -- 172.21.15.67:0/933807818 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f980ab600 con 0x7f7fa415d9e0 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7fa3e00640 1 -- 172.21.15.67:0/933807818 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f980ab9d0 con 0x7f7fa415d9e0 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7faaa00640 1 -- 172.21.15.67:0/933807818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa415d9e0 msgr2=0x7f7fa40afe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7faaa00640 1 --2- 172.21.15.67:0/933807818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa415d9e0 0x7f7fa40afe10 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f7f980097c0 tx=0x7f7f9801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7faaa00640 1 -- 172.21.15.67:0/933807818 shutdown_connections 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7faaa00640 1 --2- 172.21.15.67:0/933807818 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa415d9e0 0x7f7fa40afe10 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7faaa00640 1 -- 172.21.15.67:0/933807818 >> 172.21.15.67:0/933807818 conn(0x7f7fa406e3b0 msgr2=0x7f7fa406e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7faaa00640 1 -- 172.21.15.67:0/933807818 shutdown_connections 2024-04-17T10:29:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.081+0000 7f7faaa00640 1 -- 172.21.15.67:0/933807818 wait complete. 2024-04-17T10:29:54.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 Processor -- start 2024-04-17T10:29:54.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 -- start start 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fa406b3a0 0x7f7fa4150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fa415d9e0 0x7f7fa40cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa4162260 0x7f7fa4137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa416baa0 con 0x7f7fa4162260 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa416b920 con 0x7f7fa406b3a0 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa416bc20 con 0x7f7fa415d9e0 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa4162260 0x7f7fa4137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa4162260 0x7f7fa4137e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57324/0 (socket says 172.21.15.67:57324) 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 -- 172.21.15.67:0/3553009590 learned_addr learned my addr 172.21.15.67:0/3553009590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7fa9600640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fa406b3a0 0x7f7fa4150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:54.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7fa8c00640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fa415d9e0 0x7f7fa40cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 -- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fa415d9e0 msgr2=0x7f7fa40cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fa415d9e0 0x7f7fa40cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 -- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fa406b3a0 msgr2=0x7f7fa4150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fa406b3a0 0x7f7fa4150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 -- 172.21.15.67:0/3553009590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa414fdc0 con 0x7f7fa4162260 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faa000640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa4162260 0x7f7fa4137e20 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00c770 tx=0x7f7f9c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7fa1600640 1 -- 172.21.15.67:0/3553009590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c003b80 con 0x7f7fa4162260 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7fa1600640 1 -- 172.21.15.67:0/3553009590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c003d20 con 0x7f7fa4162260 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa41500a0 con 0x7f7fa4162260 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7fa1600640 1 -- 172.21.15.67:0/3553009590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c00aa90 con 0x7f7fa4162260 2024-04-17T10:29:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa4150660 con 0x7f7fa4162260 2024-04-17T10:29:54.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.085+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa4074dc0 con 0x7f7fa4162260 2024-04-17T10:29:54.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.089+0000 7f7fa1600640 1 -- 172.21.15.67:0/3553009590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7f9c00ac30 con 0x7f7fa4162260 2024-04-17T10:29:54.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.089+0000 7f7fa1600640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f84066f60 0x7f7f84069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:54.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.089+0000 7f7fa1600640 1 -- 172.21.15.67:0/3553009590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7f9c084fb0 con 0x7f7fa4162260 2024-04-17T10:29:54.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.089+0000 7f7fa9600640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f84066f60 0x7f7f84069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:54.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.089+0000 7f7fa9600640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f84066f60 0x7f7f84069420 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7f980ad000 tx=0x7f7f980a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:54.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.093+0000 7f7fa1600640 1 -- 172.21.15.67:0/3553009590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7f9c0557e0 con 0x7f7fa4162260 2024-04-17T10:29:54.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.249+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fa40cf9c0 con 0x7f7f84066f60 2024-04-17T10:29:54.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.257+0000 7f7fa1600640 1 -- 172.21.15.67:0/3553009590 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252083 (secure 0 0 0) 0x7f7fa40cf9c0 con 0x7f7f84066f60 2024-04-17T10:29:54.254 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:54.255 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f84066f60 msgr2=0x7f7f84069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f84066f60 0x7f7f84069420 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f7f980ad000 tx=0x7f7f980a6000 comp rx=0 tx=0).stop 2024-04-17T10:29:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa4162260 msgr2=0x7f7fa4137e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa4162260 0x7f7fa4137e20 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00c770 tx=0x7f7f9c00cc30 comp rx=0 tx=0).stop 2024-04-17T10:29:54.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 shutdown_connections 2024-04-17T10:29:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f84066f60 0x7f7f84069420 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fa4162260 0x7f7fa4137e20 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fa415d9e0 0x7f7fa40cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 --2- 172.21.15.67:0/3553009590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fa406b3a0 0x7f7fa4150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 >> 172.21.15.67:0/3553009590 conn(0x7f7fa406e3b0 msgr2=0x7f7fa4155b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 shutdown_connections 2024-04-17T10:29:54.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:54.261+0000 7f7faaa00640 1 -- 172.21.15.67:0/3553009590 wait complete. 2024-04-17T10:29:54.273 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-04-17T10:29:54.205230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326600,"kb_used_data":9160,"kb_used_omap":12,"kb_used_meta":317299,"kb_avail":754648120,"statfs":{"total":773094113280,"available":772759674880,"internally_reserved":0,"allocated":9379840,"data_stored":4957100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":324914737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.616930"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624220+0000","last_change":"2024-04-17T10:29:48.624547+0000","last_active":"2024-04-17T10:29:49.624220+0000","last_peered":"2024-04-17T10:29:49.624220+0000","last_clean":"2024-04-17T10:29:49.624220+0000","last_became_active":"2024-04-17T10:29:48.623620+0000","last_became_peered":"2024-04-17T10:29:48.623620+0000","last_unstale":"2024-04-17T10:29:49.624220+0000","last_undegraded":"2024-04-17T10:29:49.624220+0000","last_fullsized":"2024-04-17T10:29:49.624220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:03.584192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622732+0000","last_change":"2024-04-17T10:29:48.628788+0000","last_active":"2024-04-17T10:29:49.622732+0000","last_peered":"2024-04-17T10:29:49.622732+0000","last_clean":"2024-04-17T10:29:49.622732+0000","last_became_active":"2024-04-17T10:29:48.628522+0000","last_became_peered":"2024-04-17T10:29:48.628522+0000","last_unstale":"2024-04-17T10:29:49.622732+0000","last_undegraded":"2024-04-17T10:29:49.622732+0000","last_fullsized":"2024-04-17T10:29:49.622732+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:10:23.209350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473306+0000","last_change":"2024-04-17T10:29:48.638655+0000","last_active":"2024-04-17T10:29:50.473306+0000","last_peered":"2024-04-17T10:29:50.473306+0000","last_clean":"2024-04-17T10:29:50.473306+0000","last_became_active":"2024-04-17T10:29:48.638384+0000","last_became_peered":"2024-04-17T10:29:48.638384+0000","last_unstale":"2024-04-17T10:29:50.473306+0000","last_undegraded":"2024-04-17T10:29:50.473306+0000","last_fullsized":"2024-04-17T10:29:50.473306+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:10.041247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263588+0000","last_change":"2024-04-17T10:29:48.622391+0000","last_active":"2024-04-17T10:29:50.263588+0000","last_peered":"2024-04-17T10:29:50.263588+0000","last_clean":"2024-04-17T10:29:50.263588+0000","last_became_active":"2024-04-17T10:29:48.621932+0000","last_became_peered":"2024-04-17T10:29:48.621932+0000","last_unstale":"2024-04-17T10:29:50.263588+0000","last_undegraded":"2024-04-17T10:29:50.263588+0000","last_fullsized":"2024-04-17T10:29:50.263588+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:46.746210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631721+0000","last_change":"2024-04-17T10:29:48.628687+0000","last_active":"2024-04-17T10:29:49.631721+0000","last_peered":"2024-04-17T10:29:49.631721+0000","last_clean":"2024-04-17T10:29:49.631721+0000","last_became_active":"2024-04-17T10:29:48.628367+0000","last_became_peered":"2024-04-17T10:29:48.628367+0000","last_unstale":"2024-04-17T10:29:49.631721+0000","last_undegraded":"2024-04-17T10:29:49.631721+0000","last_fullsized":"2024-04-17T10:29:49.631721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:17.740265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627412+0000","last_change":"2024-04-17T10:29:48.620231+0000","last_active":"2024-04-17T10:29:49.627412+0000","last_peered":"2024-04-17T10:29:49.627412+0000","last_clean":"2024-04-17T10:29:49.627412+0000","last_became_active":"2024-04-17T10:29:48.619957+0000","last_became_peered":"2024-04-17T10:29:48.619957+0000","last_unstale":"2024-04-17T10:29:49.627412+0000","last_undegraded":"2024-04-17T10:29:49.627412+0000","last_fullsized":"2024-04-17T10:29:49.627412+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:12.367407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469291+0000","last_change":"2024-04-17T10:29:48.623499+0000","last_active":"2024-04-17T10:29:50.469291+0000","last_peered":"2024-04-17T10:29:50.469291+0000","last_clean":"2024-04-17T10:29:50.469291+0000","last_became_active":"2024-04-17T10:29:48.623214+0000","last_became_peered":"2024-04-17T10:29:48.623214+0000","last_unstale":"2024-04-17T10:29:50.469291+0000","last_undegraded":"2024-04-17T10:29:50.469291+0000","last_fullsized":"2024-04-17T10:29:50.469291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:56:01.525634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631402+0000","last_change":"2024-04-17T10:29:48.629131+0000","last_active":"2024-04-17T10:29:49.631402+0000","last_peered":"2024-04-17T10:29:49.631402+0000","last_clean":"2024-04-17T10:29:49.631402+0000","last_became_active":"2024-04-17T10:29:48.628818+0000","last_became_peered":"2024-04-17T10:29:48.628818+0000","last_unstale":"2024-04-17T10:29:49.631402+0000","last_undegraded":"2024-04-17T10:29:49.631402+0000","last_fullsized":"2024-04-17T10:29:49.631402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:14.755052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469775+0000","last_change":"2024-04-17T10:29:48.613703+0000","last_active":"2024-04-17T10:29:50.469775+0000","last_peered":"2024-04-17T10:29:50.469775+0000","last_clean":"2024-04-17T10:29:50.469775+0000","last_became_active":"2024-04-17T10:29:48.613435+0000","last_became_peered":"2024-04-17T10:29:48.613435+0000","last_unstale":"2024-04-17T10:29:50.469775+0000","last_undegraded":"2024-04-17T10:29:50.469775+0000","last_fullsized":"2024-04-17T10:29:50.469775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:49.081079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263601+0000","last_change":"2024-04-17T10:29:48.623135+0000","last_active":"2024-04-17T10:29:50.263601+0000","last_peered":"2024-04-17T10:29:50.263601+0000","last_clean":"2024-04-17T10:29:50.263601+0000","last_became_active":"2024-04-17T10:29:48.622795+0000","last_became_peered":"2024-04-17T10:29:48.622795+0000","last_unstale":"2024-04-17T10:29:50.263601+0000","last_undegraded":"2024-04-17T10:29:50.263601+0000","last_fullsized":"2024-04-17T10:29:50.263601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:09.035711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622714+0000","last_change":"2024-04-17T10:29:48.631279+0000","last_active":"2024-04-17T10:29:49.622714+0000","last_peered":"2024-04-17T10:29:49.622714+0000","last_clean":"2024-04-17T10:29:49.622714+0000","last_became_active":"2024-04-17T10:29:48.627396+0000","last_became_peered":"2024-04-17T10:29:48.627396+0000","last_unstale":"2024-04-17T10:29:49.622714+0000","last_undegraded":"2024-04-17T10:29:49.622714+0000","last_fullsized":"2024-04-17T10:29:49.622714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:31.491002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263606+0000","last_change":"2024-04-17T10:29:48.624630+0000","last_active":"2024-04-17T10:29:50.263606+0000","last_peered":"2024-04-17T10:29:50.263606+0000","last_clean":"2024-04-17T10:29:50.263606+0000","last_became_active":"2024-04-17T10:29:48.624287+0000","last_became_peered":"2024-04-17T10:29:48.624287+0000","last_unstale":"2024-04-17T10:29:50.263606+0000","last_undegraded":"2024-04-17T10:29:50.263606+0000","last_fullsized":"2024-04-17T10:29:50.263606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:54.561019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631758+0000","last_change":"2024-04-17T10:29:48.628214+0000","last_active":"2024-04-17T10:29:49.631758+0000","last_peered":"2024-04-17T10:29:49.631758+0000","last_clean":"2024-04-17T10:29:49.631758+0000","last_became_active":"2024-04-17T10:29:48.627533+0000","last_became_peered":"2024-04-17T10:29:48.627533+0000","last_unstale":"2024-04-17T10:29:49.631758+0000","last_undegraded":"2024-04-17T10:29:49.631758+0000","last_fullsized":"2024-04-17T10:29:49.631758+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:06.290628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627426+0000","last_change":"2024-04-17T10:29:48.620922+0000","last_active":"2024-04-17T10:29:49.627426+0000","last_peered":"2024-04-17T10:29:49.627426+0000","last_clean":"2024-04-17T10:29:49.627426+0000","last_became_active":"2024-04-17T10:29:48.620539+0000","last_became_peered":"2024-04-17T10:29:48.620539+0000","last_unstale":"2024-04-17T10:29:49.627426+0000","last_undegraded":"2024-04-17T10:29:49.627426+0000","last_fullsized":"2024-04-17T10:29:49.627426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:31.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622389+0000","last_change":"2024-04-17T10:29:48.641288+0000","last_active":"2024-04-17T10:29:49.622389+0000","last_peered":"2024-04-17T10:29:49.622389+0000","last_clean":"2024-04-17T10:29:49.622389+0000","last_became_active":"2024-04-17T10:29:48.641041+0000","last_became_peered":"2024-04-17T10:29:48.641041+0000","last_unstale":"2024-04-17T10:29:49.622389+0000","last_undegraded":"2024-04-17T10:29:49.622389+0000","last_fullsized":"2024-04-17T10:29:49.622389+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:46.746210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624298+0000","last_change":"2024-04-17T10:29:48.618685+0000","last_active":"2024-04-17T10:29:49.624298+0000","last_peered":"2024-04-17T10:29:49.624298+0000","last_clean":"2024-04-17T10:29:49.624298+0000","last_became_active":"2024-04-17T10:29:48.618355+0000","last_became_peered":"2024-04-17T10:29:48.618355+0000","last_unstale":"2024-04-17T10:29:49.624298+0000","last_undegraded":"2024-04-17T10:29:49.624298+0000","last_fullsized":"2024-04-17T10:29:49.624298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:31.093086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.632040+0000","last_change":"2024-04-17T10:29:48.632455+0000","last_active":"2024-04-17T10:29:49.632040+0000","last_peered":"2024-04-17T10:29:49.632040+0000","last_clean":"2024-04-17T10:29:49.632040+0000","last_became_active":"2024-04-17T10:29:48.632105+0000","last_became_peered":"2024-04-17T10:29:48.632105+0000","last_unstale":"2024-04-17T10:29:49.632040+0000","last_undegraded":"2024-04-17T10:29:49.632040+0000","last_fullsized":"2024-04-17T10:29:49.632040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:06.007030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473376+0000","last_change":"2024-04-17T10:29:48.633707+0000","last_active":"2024-04-17T10:29:50.473376+0000","last_peered":"2024-04-17T10:29:50.473376+0000","last_clean":"2024-04-17T10:29:50.473376+0000","last_became_active":"2024-04-17T10:29:48.633424+0000","last_became_peered":"2024-04-17T10:29:48.633424+0000","last_unstale":"2024-04-17T10:29:50.473376+0000","last_undegraded":"2024-04-17T10:29:50.473376+0000","last_fullsized":"2024-04-17T10:29:50.473376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:14.739124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627741+0000","last_change":"2024-04-17T10:29:48.628426+0000","last_active":"2024-04-17T10:29:49.627741+0000","last_peered":"2024-04-17T10:29:49.627741+0000","last_clean":"2024-04-17T10:29:49.627741+0000","last_became_active":"2024-04-17T10:29:48.628021+0000","last_became_peered":"2024-04-17T10:29:48.628021+0000","last_unstale":"2024-04-17T10:29:49.627741+0000","last_undegraded":"2024-04-17T10:29:49.627741+0000","last_fullsized":"2024-04-17T10:29:49.627741+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:03.584192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631859+0000","last_change":"2024-04-17T10:29:48.636282+0000","last_active":"2024-04-17T10:29:49.631859+0000","last_peered":"2024-04-17T10:29:49.631859+0000","last_clean":"2024-04-17T10:29:49.631859+0000","last_became_active":"2024-04-17T10:29:48.635825+0000","last_became_peered":"2024-04-17T10:29:48.635825+0000","last_unstale":"2024-04-17T10:29:49.631859+0000","last_undegraded":"2024-04-17T10:29:49.631859+0000","last_fullsized":"2024-04-17T10:29:49.631859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:28:13.244001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469198+0000","last_change":"2024-04-17T10:29:48.635189+0000","last_active":"2024-04-17T10:29:50.469198+0000","last_peered":"2024-04-17T10:29:50.469198+0000","last_clean":"2024-04-17T10:29:50.469198+0000","last_became_active":"2024-04-17T10:29:48.634921+0000","last_became_peered":"2024-04-17T10:29:48.634921+0000","last_unstale":"2024-04-17T10:29:50.469198+0000","last_undegraded":"2024-04-17T10:29:50.469198+0000","last_fullsized":"2024-04-17T10:29:50.469198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:59.293622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631455+0000","last_change":"2024-04-17T10:29:48.621026+0000","last_active":"2024-04-17T10:29:49.631455+0000","last_peered":"2024-04-17T10:29:49.631455+0000","last_clean":"2024-04-17T10:29:49.631455+0000","last_became_active":"2024-04-17T10:29:48.620743+0000","last_became_peered":"2024-04-17T10:29:48.620743+0000","last_unstale":"2024-04-17T10:29:49.631455+0000","last_undegraded":"2024-04-17T10:29:49.631455+0000","last_fullsized":"2024-04-17T10:29:49.631455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:17.781225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.468837+0000","last_change":"2024-04-17T10:29:48.616474+0000","last_active":"2024-04-17T10:29:50.468837+0000","last_peered":"2024-04-17T10:29:50.468837+0000","last_clean":"2024-04-17T10:29:50.468837+0000","last_became_active":"2024-04-17T10:29:48.616213+0000","last_became_peered":"2024-04-17T10:29:48.616213+0000","last_unstale":"2024-04-17T10:29:50.468837+0000","last_undegraded":"2024-04-17T10:29:50.468837+0000","last_fullsized":"2024-04-17T10:29:50.468837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:32.124506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622319+0000","last_change":"2024-04-17T10:29:48.615060+0000","last_active":"2024-04-17T10:29:49.622319+0000","last_peered":"2024-04-17T10:29:49.622319+0000","last_clean":"2024-04-17T10:29:49.622319+0000","last_became_active":"2024-04-17T10:29:48.614732+0000","last_became_peered":"2024-04-17T10:29:48.614732+0000","last_unstale":"2024-04-17T10:29:49.622319+0000","last_undegraded":"2024-04-17T10:29:49.622319+0000","last_fullsized":"2024-04-17T10:29:49.622319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:05.285071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622815+0000","last_change":"2024-04-17T10:29:48.614690+0000","last_active":"2024-04-17T10:29:49.622815+0000","last_peered":"2024-04-17T10:29:49.622815+0000","last_clean":"2024-04-17T10:29:49.622815+0000","last_became_active":"2024-04-17T10:29:48.614408+0000","last_became_peered":"2024-04-17T10:29:48.614408+0000","last_unstale":"2024-04-17T10:29:49.622815+0000","last_undegraded":"2024-04-17T10:29:49.622815+0000","last_fullsized":"2024-04-17T10:29:49.622815+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:53.764108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263616+0000","last_change":"2024-04-17T10:29:48.626210+0000","last_active":"2024-04-17T10:29:50.263616+0000","last_peered":"2024-04-17T10:29:50.263616+0000","last_clean":"2024-04-17T10:29:50.263616+0000","last_became_active":"2024-04-17T10:29:48.625916+0000","last_became_peered":"2024-04-17T10:29:48.625916+0000","last_unstale":"2024-04-17T10:29:50.263616+0000","last_undegraded":"2024-04-17T10:29:50.263616+0000","last_fullsized":"2024-04-17T10:29:50.263616+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:56.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624132+0000","last_change":"2024-04-17T10:29:48.618251+0000","last_active":"2024-04-17T10:29:49.624132+0000","last_peered":"2024-04-17T10:29:49.624132+0000","last_clean":"2024-04-17T10:29:49.624132+0000","last_became_active":"2024-04-17T10:29:48.617966+0000","last_became_peered":"2024-04-17T10:29:48.617966+0000","last_unstale":"2024-04-17T10:29:49.624132+0000","last_undegraded":"2024-04-17T10:29:49.624132+0000","last_fullsized":"2024-04-17T10:29:49.624132+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:53.764108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631885+0000","last_change":"2024-04-17T10:29:48.625511+0000","last_active":"2024-04-17T10:29:49.631885+0000","last_peered":"2024-04-17T10:29:49.631885+0000","last_clean":"2024-04-17T10:29:49.631885+0000","last_became_active":"2024-04-17T10:29:48.625144+0000","last_became_peered":"2024-04-17T10:29:48.625144+0000","last_unstale":"2024-04-17T10:29:49.631885+0000","last_undegraded":"2024-04-17T10:29:49.631885+0000","last_fullsized":"2024-04-17T10:29:49.631885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:39.246125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622671+0000","last_change":"2024-04-17T10:29:48.626108+0000","last_active":"2024-04-17T10:29:49.622671+0000","last_peered":"2024-04-17T10:29:49.622671+0000","last_clean":"2024-04-17T10:29:49.622671+0000","last_became_active":"2024-04-17T10:29:48.625797+0000","last_became_peered":"2024-04-17T10:29:48.625797+0000","last_unstale":"2024-04-17T10:29:49.622671+0000","last_undegraded":"2024-04-17T10:29:49.622671+0000","last_fullsized":"2024-04-17T10:29:49.622671+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:21:00.685909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623768+0000","last_change":"2024-04-17T10:29:48.631564+0000","last_active":"2024-04-17T10:29:49.623768+0000","last_peered":"2024-04-17T10:29:49.623768+0000","last_clean":"2024-04-17T10:29:49.623768+0000","last_became_active":"2024-04-17T10:29:48.631103+0000","last_became_peered":"2024-04-17T10:29:48.631103+0000","last_unstale":"2024-04-17T10:29:49.623768+0000","last_undegraded":"2024-04-17T10:29:49.623768+0000","last_fullsized":"2024-04-17T10:29:49.623768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:12.367407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473094+0000","last_change":"2024-04-17T10:29:48.634885+0000","last_active":"2024-04-17T10:29:50.473094+0000","last_peered":"2024-04-17T10:29:50.473094+0000","last_clean":"2024-04-17T10:29:50.473094+0000","last_became_active":"2024-04-17T10:29:48.634566+0000","last_became_peered":"2024-04-17T10:29:48.634566+0000","last_unstale":"2024-04-17T10:29:50.473094+0000","last_undegraded":"2024-04-17T10:29:50.473094+0000","last_fullsized":"2024-04-17T10:29:50.473094+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:56:01.525634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624099+0000","last_change":"2024-04-17T10:29:48.619966+0000","last_active":"2024-04-17T10:29:49.624099+0000","last_peered":"2024-04-17T10:29:49.624099+0000","last_clean":"2024-04-17T10:29:49.624099+0000","last_became_active":"2024-04-17T10:29:48.617677+0000","last_became_peered":"2024-04-17T10:29:48.617677+0000","last_unstale":"2024-04-17T10:29:49.624099+0000","last_undegraded":"2024-04-17T10:29:49.624099+0000","last_fullsized":"2024-04-17T10:29:49.624099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:39.394407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473490+0000","last_change":"2024-04-17T10:29:48.633916+0000","last_active":"2024-04-17T10:29:50.473490+0000","last_peered":"2024-04-17T10:29:50.473490+0000","last_clean":"2024-04-17T10:29:50.473490+0000","last_became_active":"2024-04-17T10:29:48.633644+0000","last_became_peered":"2024-04-17T10:29:48.633644+0000","last_unstale":"2024-04-17T10:29:50.473490+0000","last_undegraded":"2024-04-17T10:29:50.473490+0000","last_fullsized":"2024-04-17T10:29:50.473490+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:28.462361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"82'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631957+0000","last_change":"2024-04-17T10:29:47.622473+0000","last_active":"2024-04-17T10:29:49.631957+0000","last_peered":"2024-04-17T10:29:49.631957+0000","last_clean":"2024-04-17T10:29:49.631957+0000","last_became_active":"2024-04-17T10:29:47.622183+0000","last_became_peered":"2024-04-17T10:29:47.622183+0000","last_unstale":"2024-04-17T10:29:49.631957+0000","last_undegraded":"2024-04-17T10:29:49.631957+0000","last_fullsized":"2024-04-17T10:29:49.631957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:09.994065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622660+0000","last_change":"2024-04-17T10:29:48.621313+0000","last_active":"2024-04-17T10:29:49.622660+0000","last_peered":"2024-04-17T10:29:49.622660+0000","last_clean":"2024-04-17T10:29:49.622660+0000","last_became_active":"2024-04-17T10:29:48.621040+0000","last_became_peered":"2024-04-17T10:29:48.621040+0000","last_unstale":"2024-04-17T10:29:49.622660+0000","last_undegraded":"2024-04-17T10:29:49.622660+0000","last_fullsized":"2024-04-17T10:29:49.622660+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:56.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624394+0000","last_change":"2024-04-17T10:29:47.614585+0000","last_active":"2024-04-17T10:29:49.624394+0000","last_peered":"2024-04-17T10:29:49.624394+0000","last_clean":"2024-04-17T10:29:49.624394+0000","last_became_active":"2024-04-17T10:29:47.614295+0000","last_became_peered":"2024-04-17T10:29:47.614295+0000","last_unstale":"2024-04-17T10:29:49.624394+0000","last_undegraded":"2024-04-17T10:29:49.624394+0000","last_fullsized":"2024-04-17T10:29:49.624394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:53.553933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.468967+0000","last_change":"2024-04-17T10:29:48.613157+0000","last_active":"2024-04-17T10:29:50.468967+0000","last_peered":"2024-04-17T10:29:50.468967+0000","last_clean":"2024-04-17T10:29:50.468967+0000","last_became_active":"2024-04-17T10:29:48.612865+0000","last_became_peered":"2024-04-17T10:29:48.612865+0000","last_unstale":"2024-04-17T10:29:50.468967+0000","last_undegraded":"2024-04-17T10:29:50.468967+0000","last_fullsized":"2024-04-17T10:29:50.468967+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:01.176016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623755+0000","last_change":"2024-04-17T10:29:47.609395+0000","last_active":"2024-04-17T10:29:49.623755+0000","last_peered":"2024-04-17T10:29:49.623755+0000","last_clean":"2024-04-17T10:29:49.623755+0000","last_became_active":"2024-04-17T10:29:47.609097+0000","last_became_peered":"2024-04-17T10:29:47.609097+0000","last_unstale":"2024-04-17T10:29:49.623755+0000","last_undegraded":"2024-04-17T10:29:49.623755+0000","last_fullsized":"2024-04-17T10:29:49.623755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:41.846247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469414+0000","last_change":"2024-04-17T10:29:48.635119+0000","last_active":"2024-04-17T10:29:50.469414+0000","last_peered":"2024-04-17T10:29:50.469414+0000","last_clean":"2024-04-17T10:29:50.469414+0000","last_became_active":"2024-04-17T10:29:48.634841+0000","last_became_peered":"2024-04-17T10:29:48.634841+0000","last_unstale":"2024-04-17T10:29:50.469414+0000","last_undegraded":"2024-04-17T10:29:50.469414+0000","last_fullsized":"2024-04-17T10:29:50.469414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:44:31.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469676+0000","last_change":"2024-04-17T10:29:47.609681+0000","last_active":"2024-04-17T10:29:50.469676+0000","last_peered":"2024-04-17T10:29:50.469676+0000","last_clean":"2024-04-17T10:29:50.469676+0000","last_became_active":"2024-04-17T10:29:47.609421+0000","last_became_peered":"2024-04-17T10:29:47.609421+0000","last_unstale":"2024-04-17T10:29:50.469676+0000","last_undegraded":"2024-04-17T10:29:50.469676+0000","last_fullsized":"2024-04-17T10:29:50.469676+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:19.717202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622740+0000","last_change":"2024-04-17T10:29:48.629710+0000","last_active":"2024-04-17T10:29:49.622740+0000","last_peered":"2024-04-17T10:29:49.622740+0000","last_clean":"2024-04-17T10:29:49.622740+0000","last_became_active":"2024-04-17T10:29:48.629147+0000","last_became_peered":"2024-04-17T10:29:48.629147+0000","last_unstale":"2024-04-17T10:29:49.622740+0000","last_undegraded":"2024-04-17T10:29:49.622740+0000","last_fullsized":"2024-04-17T10:29:49.622740+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:12.367407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469479+0000","last_change":"2024-04-17T10:29:47.608006+0000","last_active":"2024-04-17T10:29:50.469479+0000","last_peered":"2024-04-17T10:29:50.469479+0000","last_clean":"2024-04-17T10:29:50.469479+0000","last_became_active":"2024-04-17T10:29:47.607640+0000","last_became_peered":"2024-04-17T10:29:47.607640+0000","last_unstale":"2024-04-17T10:29:50.469479+0000","last_undegraded":"2024-04-17T10:29:50.469479+0000","last_fullsized":"2024-04-17T10:29:50.469479+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:39:12.331416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622786+0000","last_change":"2024-04-17T10:29:48.627703+0000","last_active":"2024-04-17T10:29:49.622786+0000","last_peered":"2024-04-17T10:29:49.622786+0000","last_clean":"2024-04-17T10:29:49.622786+0000","last_became_active":"2024-04-17T10:29:48.627071+0000","last_became_peered":"2024-04-17T10:29:48.627071+0000","last_unstale":"2024-04-17T10:29:49.622786+0000","last_undegraded":"2024-04-17T10:29:49.622786+0000","last_fullsized":"2024-04-17T10:29:49.622786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:31.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263175+0000","last_change":"2024-04-17T10:29:47.615219+0000","last_active":"2024-04-17T10:29:50.263175+0000","last_peered":"2024-04-17T10:29:50.263175+0000","last_clean":"2024-04-17T10:29:50.263175+0000","last_became_active":"2024-04-17T10:29:47.614925+0000","last_became_peered":"2024-04-17T10:29:47.614925+0000","last_unstale":"2024-04-17T10:29:50.263175+0000","last_undegraded":"2024-04-17T10:29:50.263175+0000","last_fullsized":"2024-04-17T10:29:50.263175+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:24.468474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469550+0000","last_change":"2024-04-17T10:29:48.612686+0000","last_active":"2024-04-17T10:29:50.469550+0000","last_peered":"2024-04-17T10:29:50.469550+0000","last_clean":"2024-04-17T10:29:50.469550+0000","last_became_active":"2024-04-17T10:29:48.612403+0000","last_became_peered":"2024-04-17T10:29:48.612403+0000","last_unstale":"2024-04-17T10:29:50.469550+0000","last_undegraded":"2024-04-17T10:29:50.469550+0000","last_fullsized":"2024-04-17T10:29:50.469550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:51.336555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622482+0000","last_change":"2024-04-17T10:29:47.610640+0000","last_active":"2024-04-17T10:29:49.622482+0000","last_peered":"2024-04-17T10:29:49.622482+0000","last_clean":"2024-04-17T10:29:49.622482+0000","last_became_active":"2024-04-17T10:29:47.610355+0000","last_became_peered":"2024-04-17T10:29:47.610355+0000","last_unstale":"2024-04-17T10:29:49.622482+0000","last_undegraded":"2024-04-17T10:29:49.622482+0000","last_fullsized":"2024-04-17T10:29:49.622482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:06.557009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469894+0000","last_change":"2024-04-17T10:29:48.627887+0000","last_active":"2024-04-17T10:29:50.469894+0000","last_peered":"2024-04-17T10:29:50.469894+0000","last_clean":"2024-04-17T10:29:50.469894+0000","last_became_active":"2024-04-17T10:29:48.627570+0000","last_became_peered":"2024-04-17T10:29:48.627570+0000","last_unstale":"2024-04-17T10:29:50.469894+0000","last_undegraded":"2024-04-17T10:29:50.469894+0000","last_fullsized":"2024-04-17T10:29:50.469894+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:56.246834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631389+0000","last_change":"2024-04-17T10:29:47.609754+0000","last_active":"2024-04-17T10:29:49.631389+0000","last_peered":"2024-04-17T10:29:49.631389+0000","last_clean":"2024-04-17T10:29:49.631389+0000","last_became_active":"2024-04-17T10:29:47.609526+0000","last_became_peered":"2024-04-17T10:29:47.609526+0000","last_unstale":"2024-04-17T10:29:49.631389+0000","last_undegraded":"2024-04-17T10:29:49.631389+0000","last_fullsized":"2024-04-17T10:29:49.631389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:55.887140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622339+0000","last_change":"2024-04-17T10:29:48.626523+0000","last_active":"2024-04-17T10:29:49.622339+0000","last_peered":"2024-04-17T10:29:49.622339+0000","last_clean":"2024-04-17T10:29:49.622339+0000","last_became_active":"2024-04-17T10:29:48.626189+0000","last_became_peered":"2024-04-17T10:29:48.626189+0000","last_unstale":"2024-04-17T10:29:49.622339+0000","last_undegraded":"2024-04-17T10:29:49.622339+0000","last_fullsized":"2024-04-17T10:29:49.622339+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:54.561019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624227+0000","last_change":"2024-04-17T10:29:47.614092+0000","last_active":"2024-04-17T10:29:49.624227+0000","last_peered":"2024-04-17T10:29:49.624227+0000","last_clean":"2024-04-17T10:29:49.624227+0000","last_became_active":"2024-04-17T10:29:47.613772+0000","last_became_peered":"2024-04-17T10:29:47.613772+0000","last_unstale":"2024-04-17T10:29:49.624227+0000","last_undegraded":"2024-04-17T10:29:49.624227+0000","last_fullsized":"2024-04-17T10:29:49.624227+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:43:57.307677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263373+0000","last_change":"2024-04-17T10:29:48.624273+0000","last_active":"2024-04-17T10:29:50.263373+0000","last_peered":"2024-04-17T10:29:50.263373+0000","last_clean":"2024-04-17T10:29:50.263373+0000","last_became_active":"2024-04-17T10:29:48.624014+0000","last_became_peered":"2024-04-17T10:29:48.624014+0000","last_unstale":"2024-04-17T10:29:50.263373+0000","last_undegraded":"2024-04-17T10:29:50.263373+0000","last_fullsized":"2024-04-17T10:29:50.263373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:05.285071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263686+0000","last_change":"2024-04-17T10:29:47.607460+0000","last_active":"2024-04-17T10:29:50.263686+0000","last_peered":"2024-04-17T10:29:50.263686+0000","last_clean":"2024-04-17T10:29:50.263686+0000","last_became_active":"2024-04-17T10:29:47.607008+0000","last_became_peered":"2024-04-17T10:29:47.607008+0000","last_unstale":"2024-04-17T10:29:50.263686+0000","last_undegraded":"2024-04-17T10:29:50.263686+0000","last_fullsized":"2024-04-17T10:29:50.263686+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:55.502916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627523+0000","last_change":"2024-04-17T10:29:48.634420+0000","last_active":"2024-04-17T10:29:49.627523+0000","last_peered":"2024-04-17T10:29:49.627523+0000","last_clean":"2024-04-17T10:29:49.627523+0000","last_became_active":"2024-04-17T10:29:48.634018+0000","last_became_peered":"2024-04-17T10:29:48.634018+0000","last_unstale":"2024-04-17T10:29:49.627523+0000","last_undegraded":"2024-04-17T10:29:49.627523+0000","last_fullsized":"2024-04-17T10:29:49.627523+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:39.394407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"82'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631942+0000","last_change":"2024-04-17T10:29:47.608107+0000","last_active":"2024-04-17T10:29:49.631942+0000","last_peered":"2024-04-17T10:29:49.631942+0000","last_clean":"2024-04-17T10:29:49.631942+0000","last_became_active":"2024-04-17T10:29:47.607667+0000","last_became_peered":"2024-04-17T10:29:47.607667+0000","last_unstale":"2024-04-17T10:29:49.631942+0000","last_undegraded":"2024-04-17T10:29:49.631942+0000","last_fullsized":"2024-04-17T10:29:49.631942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:43:11.584645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473418+0000","last_change":"2024-04-17T10:29:48.650004+0000","last_active":"2024-04-17T10:29:50.473418+0000","last_peered":"2024-04-17T10:29:50.473418+0000","last_clean":"2024-04-17T10:29:50.473418+0000","last_became_active":"2024-04-17T10:29:48.649710+0000","last_became_peered":"2024-04-17T10:29:48.649710+0000","last_unstale":"2024-04-17T10:29:50.473418+0000","last_undegraded":"2024-04-17T10:29:50.473418+0000","last_fullsized":"2024-04-17T10:29:50.473418+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:53.185803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"82'6","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469755+0000","last_change":"2024-04-17T10:29:47.612926+0000","last_active":"2024-04-17T10:29:50.469755+0000","last_peered":"2024-04-17T10:29:50.469755+0000","last_clean":"2024-04-17T10:29:50.469755+0000","last_became_active":"2024-04-17T10:29:47.612270+0000","last_became_peered":"2024-04-17T10:29:47.612270+0000","last_unstale":"2024-04-17T10:29:50.469755+0000","last_undegraded":"2024-04-17T10:29:50.469755+0000","last_fullsized":"2024-04-17T10:29:50.469755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:40.127604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263737+0000","last_change":"2024-04-17T10:29:48.615481+0000","last_active":"2024-04-17T10:29:50.263737+0000","last_peered":"2024-04-17T10:29:50.263737+0000","last_clean":"2024-04-17T10:29:50.263737+0000","last_became_active":"2024-04-17T10:29:48.615031+0000","last_became_peered":"2024-04-17T10:29:48.615031+0000","last_unstale":"2024-04-17T10:29:50.263737+0000","last_undegraded":"2024-04-17T10:29:50.263737+0000","last_fullsized":"2024-04-17T10:29:50.263737+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:21:00.685909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263205+0000","last_change":"2024-04-17T10:29:47.606535+0000","last_active":"2024-04-17T10:29:50.263205+0000","last_peered":"2024-04-17T10:29:50.263205+0000","last_clean":"2024-04-17T10:29:50.263205+0000","last_became_active":"2024-04-17T10:29:47.605972+0000","last_became_peered":"2024-04-17T10:29:47.605972+0000","last_unstale":"2024-04-17T10:29:50.263205+0000","last_undegraded":"2024-04-17T10:29:50.263205+0000","last_fullsized":"2024-04-17T10:29:50.263205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:24:27.488061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469800+0000","last_change":"2024-04-17T10:29:48.631017+0000","last_active":"2024-04-17T10:29:50.469800+0000","last_peered":"2024-04-17T10:29:50.469800+0000","last_clean":"2024-04-17T10:29:50.469800+0000","last_became_active":"2024-04-17T10:29:48.630723+0000","last_became_peered":"2024-04-17T10:29:48.630723+0000","last_unstale":"2024-04-17T10:29:50.469800+0000","last_undegraded":"2024-04-17T10:29:50.469800+0000","last_fullsized":"2024-04-17T10:29:50.469800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:53.185803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623723+0000","last_change":"2024-04-17T10:29:47.613565+0000","last_active":"2024-04-17T10:29:49.623723+0000","last_peered":"2024-04-17T10:29:49.623723+0000","last_clean":"2024-04-17T10:29:49.623723+0000","last_became_active":"2024-04-17T10:29:47.612951+0000","last_became_peered":"2024-04-17T10:29:47.612951+0000","last_unstale":"2024-04-17T10:29:49.623723+0000","last_undegraded":"2024-04-17T10:29:49.623723+0000","last_fullsized":"2024-04-17T10:29:49.623723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:48.363695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469834+0000","last_change":"2024-04-17T10:29:48.618167+0000","last_active":"2024-04-17T10:29:50.469834+0000","last_peered":"2024-04-17T10:29:50.469834+0000","last_clean":"2024-04-17T10:29:50.469834+0000","last_became_active":"2024-04-17T10:29:48.617891+0000","last_became_peered":"2024-04-17T10:29:48.617891+0000","last_unstale":"2024-04-17T10:29:50.469834+0000","last_undegraded":"2024-04-17T10:29:50.469834+0000","last_fullsized":"2024-04-17T10:29:50.469834+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:28.462361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631415+0000","last_change":"2024-04-17T10:29:47.622528+0000","last_active":"2024-04-17T10:29:49.631415+0000","last_peered":"2024-04-17T10:29:49.631415+0000","last_clean":"2024-04-17T10:29:49.631415+0000","last_became_active":"2024-04-17T10:29:47.622269+0000","last_became_peered":"2024-04-17T10:29:47.622269+0000","last_unstale":"2024-04-17T10:29:49.631415+0000","last_undegraded":"2024-04-17T10:29:49.631415+0000","last_fullsized":"2024-04-17T10:29:49.631415+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:40.284702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.470000+0000","last_change":"2024-04-17T10:29:48.623100+0000","last_active":"2024-04-17T10:29:50.470000+0000","last_peered":"2024-04-17T10:29:50.470000+0000","last_clean":"2024-04-17T10:29:50.470000+0000","last_became_active":"2024-04-17T10:29:48.621739+0000","last_became_peered":"2024-04-17T10:29:48.621739+0000","last_unstale":"2024-04-17T10:29:50.470000+0000","last_undegraded":"2024-04-17T10:29:50.470000+0000","last_fullsized":"2024-04-17T10:29:50.470000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:28.982617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"82'8","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622377+0000","last_change":"2024-04-17T10:29:47.617718+0000","last_active":"2024-04-17T10:29:49.622377+0000","last_peered":"2024-04-17T10:29:49.622377+0000","last_clean":"2024-04-17T10:29:49.622377+0000","last_became_active":"2024-04-17T10:29:47.616671+0000","last_became_peered":"2024-04-17T10:29:47.616671+0000","last_unstale":"2024-04-17T10:29:49.622377+0000","last_undegraded":"2024-04-17T10:29:49.622377+0000","last_fullsized":"2024-04-17T10:29:49.622377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:38.277509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627301+0000","last_change":"2024-04-17T10:29:48.621840+0000","last_active":"2024-04-17T10:29:49.627301+0000","last_peered":"2024-04-17T10:29:49.627301+0000","last_clean":"2024-04-17T10:29:49.627301+0000","last_became_active":"2024-04-17T10:29:48.621577+0000","last_became_peered":"2024-04-17T10:29:48.621577+0000","last_unstale":"2024-04-17T10:29:49.627301+0000","last_undegraded":"2024-04-17T10:29:49.627301+0000","last_fullsized":"2024-04-17T10:29:49.627301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:06.285756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624274+0000","last_change":"2024-04-17T10:29:47.608319+0000","last_active":"2024-04-17T10:29:49.624274+0000","last_peered":"2024-04-17T10:29:49.624274+0000","last_clean":"2024-04-17T10:29:49.624274+0000","last_became_active":"2024-04-17T10:29:47.608039+0000","last_became_peered":"2024-04-17T10:29:47.608039+0000","last_unstale":"2024-04-17T10:29:49.624274+0000","last_undegraded":"2024-04-17T10:29:49.624274+0000","last_fullsized":"2024-04-17T10:29:49.624274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:04.277986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631477+0000","last_change":"2024-04-17T10:29:48.623008+0000","last_active":"2024-04-17T10:29:49.631477+0000","last_peered":"2024-04-17T10:29:49.631477+0000","last_clean":"2024-04-17T10:29:49.631477+0000","last_became_active":"2024-04-17T10:29:48.622707+0000","last_became_peered":"2024-04-17T10:29:48.622707+0000","last_unstale":"2024-04-17T10:29:49.631477+0000","last_undegraded":"2024-04-17T10:29:49.631477+0000","last_fullsized":"2024-04-17T10:29:49.631477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:44.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469612+0000","last_change":"2024-04-17T10:29:47.608095+0000","last_active":"2024-04-17T10:29:50.469612+0000","last_peered":"2024-04-17T10:29:50.469612+0000","last_clean":"2024-04-17T10:29:50.469612+0000","last_became_active":"2024-04-17T10:29:47.607646+0000","last_became_peered":"2024-04-17T10:29:47.607646+0000","last_unstale":"2024-04-17T10:29:50.469612+0000","last_undegraded":"2024-04-17T10:29:50.469612+0000","last_fullsized":"2024-04-17T10:29:50.469612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:28.548917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631579+0000","last_change":"2024-04-17T10:29:48.631862+0000","last_active":"2024-04-17T10:29:49.631579+0000","last_peered":"2024-04-17T10:29:49.631579+0000","last_clean":"2024-04-17T10:29:49.631579+0000","last_became_active":"2024-04-17T10:29:48.630560+0000","last_became_peered":"2024-04-17T10:29:48.630560+0000","last_unstale":"2024-04-17T10:29:49.631579+0000","last_undegraded":"2024-04-17T10:29:49.631579+0000","last_fullsized":"2024-04-17T10:29:49.631579+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:23:37.373393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631994+0000","last_change":"2024-04-17T10:29:47.611758+0000","last_active":"2024-04-17T10:29:49.631994+0000","last_peered":"2024-04-17T10:29:49.631994+0000","last_clean":"2024-04-17T10:29:49.631994+0000","last_became_active":"2024-04-17T10:29:47.611490+0000","last_became_peered":"2024-04-17T10:29:47.611490+0000","last_unstale":"2024-04-17T10:29:49.631994+0000","last_undegraded":"2024-04-17T10:29:49.631994+0000","last_fullsized":"2024-04-17T10:29:49.631994+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:50:34.562845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627650+0000","last_change":"2024-04-17T10:29:48.632172+0000","last_active":"2024-04-17T10:29:49.627650+0000","last_peered":"2024-04-17T10:29:49.627650+0000","last_clean":"2024-04-17T10:29:49.627650+0000","last_became_active":"2024-04-17T10:29:48.631705+0000","last_became_peered":"2024-04-17T10:29:48.631705+0000","last_unstale":"2024-04-17T10:29:49.627650+0000","last_undegraded":"2024-04-17T10:29:49.627650+0000","last_fullsized":"2024-04-17T10:29:49.627650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:53.764108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473121+0000","last_change":"2024-04-17T10:29:47.614754+0000","last_active":"2024-04-17T10:29:50.473121+0000","last_peered":"2024-04-17T10:29:50.473121+0000","last_clean":"2024-04-17T10:29:50.473121+0000","last_became_active":"2024-04-17T10:29:47.614120+0000","last_became_peered":"2024-04-17T10:29:47.614120+0000","last_unstale":"2024-04-17T10:29:50.473121+0000","last_undegraded":"2024-04-17T10:29:50.473121+0000","last_fullsized":"2024-04-17T10:29:50.473121+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:17.862910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622643+0000","last_change":"2024-04-17T10:29:48.620326+0000","last_active":"2024-04-17T10:29:49.622643+0000","last_peered":"2024-04-17T10:29:49.622643+0000","last_clean":"2024-04-17T10:29:49.622643+0000","last_became_active":"2024-04-17T10:29:48.620066+0000","last_became_peered":"2024-04-17T10:29:48.620066+0000","last_unstale":"2024-04-17T10:29:49.622643+0000","last_undegraded":"2024-04-17T10:29:49.622643+0000","last_fullsized":"2024-04-17T10:29:49.622643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:09.035711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627622+0000","last_change":"2024-04-17T10:29:47.621663+0000","last_active":"2024-04-17T10:29:49.627622+0000","last_peered":"2024-04-17T10:29:49.627622+0000","last_clean":"2024-04-17T10:29:49.627622+0000","last_became_active":"2024-04-17T10:29:47.621342+0000","last_became_peered":"2024-04-17T10:29:47.621342+0000","last_unstale":"2024-04-17T10:29:49.627622+0000","last_undegraded":"2024-04-17T10:29:49.627622+0000","last_fullsized":"2024-04-17T10:29:49.627622+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:40:01.953755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622786+0000","last_change":"2024-04-17T10:29:48.615967+0000","last_active":"2024-04-17T10:29:49.622786+0000","last_peered":"2024-04-17T10:29:49.622786+0000","last_clean":"2024-04-17T10:29:49.622786+0000","last_became_active":"2024-04-17T10:29:48.615692+0000","last_became_peered":"2024-04-17T10:29:48.615692+0000","last_unstale":"2024-04-17T10:29:49.622786+0000","last_undegraded":"2024-04-17T10:29:49.622786+0000","last_fullsized":"2024-04-17T10:29:49.622786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:39.394407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627344+0000","last_change":"2024-04-17T10:29:47.620608+0000","last_active":"2024-04-17T10:29:49.627344+0000","last_peered":"2024-04-17T10:29:49.627344+0000","last_clean":"2024-04-17T10:29:49.627344+0000","last_became_active":"2024-04-17T10:29:47.620350+0000","last_became_peered":"2024-04-17T10:29:47.620350+0000","last_unstale":"2024-04-17T10:29:49.627344+0000","last_undegraded":"2024-04-17T10:29:49.627344+0000","last_fullsized":"2024-04-17T10:29:49.627344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:04.277986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.470084+0000","last_change":"2024-04-17T10:29:48.618486+0000","last_active":"2024-04-17T10:29:50.470084+0000","last_peered":"2024-04-17T10:29:50.470084+0000","last_clean":"2024-04-17T10:29:50.470084+0000","last_became_active":"2024-04-17T10:29:48.618217+0000","last_became_peered":"2024-04-17T10:29:48.618217+0000","last_unstale":"2024-04-17T10:29:50.470084+0000","last_undegraded":"2024-04-17T10:29:50.470084+0000","last_fullsized":"2024-04-17T10:29:50.470084+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:59:19.299880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627306+0000","last_change":"2024-04-17T10:29:47.619479+0000","last_active":"2024-04-17T10:29:49.627306+0000","last_peered":"2024-04-17T10:29:49.627306+0000","last_clean":"2024-04-17T10:29:49.627306+0000","last_became_active":"2024-04-17T10:29:47.619168+0000","last_became_peered":"2024-04-17T10:29:47.619168+0000","last_unstale":"2024-04-17T10:29:49.627306+0000","last_undegraded":"2024-04-17T10:29:49.627306+0000","last_fullsized":"2024-04-17T10:29:49.627306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:41.846247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.468823+0000","last_change":"2024-04-17T10:29:48.615973+0000","last_active":"2024-04-17T10:29:50.468823+0000","last_peered":"2024-04-17T10:29:50.468823+0000","last_clean":"2024-04-17T10:29:50.468823+0000","last_became_active":"2024-04-17T10:29:48.615604+0000","last_became_peered":"2024-04-17T10:29:48.615604+0000","last_unstale":"2024-04-17T10:29:50.468823+0000","last_undegraded":"2024-04-17T10:29:50.468823+0000","last_fullsized":"2024-04-17T10:29:50.468823+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:18.869996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263588+0000","last_change":"2024-04-17T10:29:47.603453+0000","last_active":"2024-04-17T10:29:50.263588+0000","last_peered":"2024-04-17T10:29:50.263588+0000","last_clean":"2024-04-17T10:29:50.263588+0000","last_became_active":"2024-04-17T10:29:47.603169+0000","last_became_peered":"2024-04-17T10:29:47.603169+0000","last_unstale":"2024-04-17T10:29:50.263588+0000","last_undegraded":"2024-04-17T10:29:50.263588+0000","last_fullsized":"2024-04-17T10:29:50.263588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:51.330142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469445+0000","last_change":"2024-04-17T10:29:48.623902+0000","last_active":"2024-04-17T10:29:50.469445+0000","last_peered":"2024-04-17T10:29:50.469445+0000","last_clean":"2024-04-17T10:29:50.469445+0000","last_became_active":"2024-04-17T10:29:48.623546+0000","last_became_peered":"2024-04-17T10:29:48.623546+0000","last_unstale":"2024-04-17T10:29:50.469445+0000","last_undegraded":"2024-04-17T10:29:50.469445+0000","last_fullsized":"2024-04-17T10:29:50.469445+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:10.041247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622829+0000","last_change":"2024-04-17T10:29:47.619750+0000","last_active":"2024-04-17T10:29:49.622829+0000","last_peered":"2024-04-17T10:29:49.622829+0000","last_clean":"2024-04-17T10:29:49.622829+0000","last_became_active":"2024-04-17T10:29:47.618886+0000","last_became_peered":"2024-04-17T10:29:47.618886+0000","last_unstale":"2024-04-17T10:29:49.622829+0000","last_undegraded":"2024-04-17T10:29:49.622829+0000","last_fullsized":"2024-04-17T10:29:49.622829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:04:31.601997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469005+0000","last_change":"2024-04-17T10:29:48.613263+0000","last_active":"2024-04-17T10:29:50.469005+0000","last_peered":"2024-04-17T10:29:50.469005+0000","last_clean":"2024-04-17T10:29:50.469005+0000","last_became_active":"2024-04-17T10:29:48.612973+0000","last_became_peered":"2024-04-17T10:29:48.612973+0000","last_unstale":"2024-04-17T10:29:50.469005+0000","last_undegraded":"2024-04-17T10:29:50.469005+0000","last_fullsized":"2024-04-17T10:29:50.469005+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:54.375698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624016+0000","last_change":"2024-04-17T10:29:47.615534+0000","last_active":"2024-04-17T10:29:49.624016+0000","last_peered":"2024-04-17T10:29:49.624016+0000","last_clean":"2024-04-17T10:29:49.624016+0000","last_became_active":"2024-04-17T10:29:47.614432+0000","last_became_peered":"2024-04-17T10:29:47.614432+0000","last_unstale":"2024-04-17T10:29:49.624016+0000","last_undegraded":"2024-04-17T10:29:49.624016+0000","last_fullsized":"2024-04-17T10:29:49.624016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:40:01.953755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622733+0000","last_change":"2024-04-17T10:29:48.630697+0000","last_active":"2024-04-17T10:29:49.622733+0000","last_peered":"2024-04-17T10:29:49.622733+0000","last_clean":"2024-04-17T10:29:49.622733+0000","last_became_active":"2024-04-17T10:29:48.626157+0000","last_became_peered":"2024-04-17T10:29:48.626157+0000","last_unstale":"2024-04-17T10:29:49.622733+0000","last_undegraded":"2024-04-17T10:29:49.622733+0000","last_fullsized":"2024-04-17T10:29:49.622733+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:06.285756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623827+0000","last_change":"2024-04-17T10:29:47.614945+0000","last_active":"2024-04-17T10:29:49.623827+0000","last_peered":"2024-04-17T10:29:49.623827+0000","last_clean":"2024-04-17T10:29:49.623827+0000","last_became_active":"2024-04-17T10:29:47.614276+0000","last_became_peered":"2024-04-17T10:29:47.614276+0000","last_unstale":"2024-04-17T10:29:49.623827+0000","last_undegraded":"2024-04-17T10:29:49.623827+0000","last_fullsized":"2024-04-17T10:29:49.623827+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:57.488244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469695+0000","last_change":"2024-04-17T10:29:48.622459+0000","last_active":"2024-04-17T10:29:50.469695+0000","last_peered":"2024-04-17T10:29:50.469695+0000","last_clean":"2024-04-17T10:29:50.469695+0000","last_became_active":"2024-04-17T10:29:48.622169+0000","last_became_peered":"2024-04-17T10:29:48.622169+0000","last_unstale":"2024-04-17T10:29:50.469695+0000","last_undegraded":"2024-04-17T10:29:50.469695+0000","last_fullsized":"2024-04-17T10:29:50.469695+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:14.739124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469513+0000","last_change":"2024-04-17T10:29:47.609861+0000","last_active":"2024-04-17T10:29:50.469513+0000","last_peered":"2024-04-17T10:29:50.469513+0000","last_clean":"2024-04-17T10:29:50.469513+0000","last_became_active":"2024-04-17T10:29:47.609241+0000","last_became_peered":"2024-04-17T10:29:47.609241+0000","last_unstale":"2024-04-17T10:29:50.469513+0000","last_undegraded":"2024-04-17T10:29:50.469513+0000","last_fullsized":"2024-04-17T10:29:50.469513+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:48:17.280119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469950+0000","last_change":"2024-04-17T10:29:48.624773+0000","last_active":"2024-04-17T10:29:50.469950+0000","last_peered":"2024-04-17T10:29:50.469950+0000","last_clean":"2024-04-17T10:29:50.469950+0000","last_became_active":"2024-04-17T10:29:48.623489+0000","last_became_peered":"2024-04-17T10:29:48.623489+0000","last_unstale":"2024-04-17T10:29:50.469950+0000","last_undegraded":"2024-04-17T10:29:50.469950+0000","last_fullsized":"2024-04-17T10:29:50.469950+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:20.694608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631520+0000","last_change":"2024-04-17T10:29:47.619501+0000","last_active":"2024-04-17T10:29:49.631520+0000","last_peered":"2024-04-17T10:29:49.631520+0000","last_clean":"2024-04-17T10:29:49.631520+0000","last_became_active":"2024-04-17T10:29:47.619199+0000","last_became_peered":"2024-04-17T10:29:47.619199+0000","last_unstale":"2024-04-17T10:29:49.631520+0000","last_undegraded":"2024-04-17T10:29:49.631520+0000","last_fullsized":"2024-04-17T10:29:49.631520+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:08.704951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473540+0000","last_change":"2024-04-17T10:29:48.645174+0000","last_active":"2024-04-17T10:29:50.473540+0000","last_peered":"2024-04-17T10:29:50.473540+0000","last_clean":"2024-04-17T10:29:50.473540+0000","last_became_active":"2024-04-17T10:29:48.644891+0000","last_became_peered":"2024-04-17T10:29:48.644891+0000","last_unstale":"2024-04-17T10:29:50.473540+0000","last_undegraded":"2024-04-17T10:29:50.473540+0000","last_fullsized":"2024-04-17T10:29:50.473540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:20.694608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469356+0000","last_change":"2024-04-17T10:29:47.616872+0000","last_active":"2024-04-17T10:29:50.469356+0000","last_peered":"2024-04-17T10:29:50.469356+0000","last_clean":"2024-04-17T10:29:50.469356+0000","last_became_active":"2024-04-17T10:29:47.616590+0000","last_became_peered":"2024-04-17T10:29:47.616590+0000","last_unstale":"2024-04-17T10:29:50.469356+0000","last_undegraded":"2024-04-17T10:29:50.469356+0000","last_fullsized":"2024-04-17T10:29:50.469356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:19.418053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473501+0000","last_change":"2024-04-17T10:29:48.634278+0000","last_active":"2024-04-17T10:29:50.473501+0000","last_peered":"2024-04-17T10:29:50.473501+0000","last_clean":"2024-04-17T10:29:50.473501+0000","last_became_active":"2024-04-17T10:29:48.633977+0000","last_became_peered":"2024-04-17T10:29:48.633977+0000","last_unstale":"2024-04-17T10:29:50.473501+0000","last_undegraded":"2024-04-17T10:29:50.473501+0000","last_fullsized":"2024-04-17T10:29:50.473501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:56.246834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.472928+0000","last_change":"2024-04-17T10:29:47.620589+0000","last_active":"2024-04-17T10:29:50.472928+0000","last_peered":"2024-04-17T10:29:50.472928+0000","last_clean":"2024-04-17T10:29:50.472928+0000","last_became_active":"2024-04-17T10:29:47.620292+0000","last_became_peered":"2024-04-17T10:29:47.620292+0000","last_unstale":"2024-04-17T10:29:50.472928+0000","last_undegraded":"2024-04-17T10:29:50.472928+0000","last_fullsized":"2024-04-17T10:29:50.472928+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:13:13.905730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623962+0000","last_change":"2024-04-17T10:29:48.632954+0000","last_active":"2024-04-17T10:29:49.623962+0000","last_peered":"2024-04-17T10:29:49.623962+0000","last_clean":"2024-04-17T10:29:49.623962+0000","last_became_active":"2024-04-17T10:29:48.632778+0000","last_became_peered":"2024-04-17T10:29:48.632778+0000","last_unstale":"2024-04-17T10:29:49.623962+0000","last_undegraded":"2024-04-17T10:29:49.623962+0000","last_fullsized":"2024-04-17T10:29:49.623962+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:31.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627712+0000","last_change":"2024-04-17T10:29:47.620546+0000","last_active":"2024-04-17T10:29:49.627712+0000","last_peered":"2024-04-17T10:29:49.627712+0000","last_clean":"2024-04-17T10:29:49.627712+0000","last_became_active":"2024-04-17T10:29:47.620261+0000","last_became_peered":"2024-04-17T10:29:47.620261+0000","last_unstale":"2024-04-17T10:29:49.627712+0000","last_undegraded":"2024-04-17T10:29:49.627712+0000","last_fullsized":"2024-04-17T10:29:49.627712+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:53.553933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":286,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624157+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:29:49.624157+0000","last_peered":"2024-04-17T10:29:49.624157+0000","last_clean":"2024-04-17T10:29:49.624157+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:29:49.624157+0000","last_undegraded":"2024-04-17T10:29:49.624157+0000","last_fullsized":"2024-04-17T10:29:49.624157+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41084,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330756,"statfs":{"total":96636764160,"available":96594694144,"internally_reserved":0,"allocated":1613824,"data_stored":1061833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335492,"statfs":{"total":96636764160,"available":96599543808,"internally_reserved":0,"allocated":1028096,"data_stored":473055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":471477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40512,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331328,"statfs":{"total":96636764160,"available":96595279872,"internally_reserved":0,"allocated":1036288,"data_stored":473159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1019904,"data_stored":472025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36352,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335488,"statfs":{"total":96636764160,"available":96599539712,"internally_reserved":0,"allocated":1028096,"data_stored":472185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1024000,"data_stored":471555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63288,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308552,"statfs":{"total":96636764160,"available":96571957248,"internally_reserved":0,"allocated":1609728,"data_stored":1061811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:55.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:29:55.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:29:57.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:29:57.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae92a00640 1 Processor -- start 2024-04-17T10:29:57.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae92a00640 1 -- start start 2024-04-17T10:29:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae92a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae8c146da0 0x7fae8c071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae92a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c161740 con 0x7fae8c1466e0 2024-04-17T10:29:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae92a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c161a20 con 0x7fae8c071850 2024-04-17T10:29:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae92a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c161d00 con 0x7fae8c146da0 2024-04-17T10:29:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae8be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fae8c1466e0 0x7fae8c06eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50944/0 (socket says 172.21.15.67:50944) 2024-04-17T10:29:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae8be00640 1 -- 172.21.15.67:0/2377139321 learned_addr learned my addr 172.21.15.67:0/2377139321 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:57.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae83400640 1 --2- 172.21.15.67:0/2377139321 >> v2:172.21.15.67:3300/0 conn(0x7fae8c146da0 0x7fae8c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae8b400640 1 -- 172.21.15.67:0/2377139321 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526986160 0 0) 0x7fae8c161740 con 0x7fae8c1466e0 2024-04-17T10:29:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae8b400640 1 -- 172.21.15.67:0/2377139321 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae70003530 con 0x7fae8c1466e0 2024-04-17T10:29:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae83400640 1 -- 172.21.15.67:0/2377139321 >> v1:172.21.15.86:6789/0 conn(0x7fae8c071850 legacy=0x7fae8c0adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae83400640 1 -- 172.21.15.67:0/2377139321 >> v1:172.21.15.67:6789/0 conn(0x7fae8c1466e0 legacy=0x7fae8c06eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.437+0000 7fae83400640 1 -- 172.21.15.67:0/2377139321 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c161fe0 con 0x7fae8c146da0 2024-04-17T10:29:57.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae83400640 1 --2- 172.21.15.67:0/2377139321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c071310 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fae740097c0 tx=0x7fae7401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d18ca00b88cacb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:57.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae8b400640 1 -- 172.21.15.67:0/2377139321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae740b0070 con 0x7fae8c146da0 2024-04-17T10:29:57.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae8b400640 1 -- 172.21.15.67:0/2377139321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae740ab600 con 0x7fae8c146da0 2024-04-17T10:29:57.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae8b400640 1 -- 172.21.15.67:0/2377139321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae740ab9d0 con 0x7fae8c146da0 2024-04-17T10:29:57.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae92a00640 1 -- 172.21.15.67:0/2377139321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 msgr2=0x7fae8c071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae92a00640 1 --2- 172.21.15.67:0/2377139321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c071310 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fae740097c0 tx=0x7fae7401c460 comp rx=0 tx=0).stop 2024-04-17T10:29:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae92a00640 1 -- 172.21.15.67:0/2377139321 shutdown_connections 2024-04-17T10:29:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae92a00640 1 --2- 172.21.15.67:0/2377139321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c071310 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae92a00640 1 -- 172.21.15.67:0/2377139321 >> 172.21.15.67:0/2377139321 conn(0x7fae8c06ac90 msgr2=0x7fae8c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae92a00640 1 -- 172.21.15.67:0/2377139321 shutdown_connections 2024-04-17T10:29:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.441+0000 7fae92a00640 1 -- 172.21.15.67:0/2377139321 wait complete. 2024-04-17T10:29:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 Processor -- start 2024-04-17T10:29:57.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 -- start start 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8c071850 0x7fae8c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8c1466e0 0x7fae8c13bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c13c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0af090 con 0x7fae8c146da0 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0aef10 con 0x7fae8c071850 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c0af210 con 0x7fae8c1466e0 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c13c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c13c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57346/0 (socket says 172.21.15.67:57346) 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 -- 172.21.15.67:0/1125853159 learned_addr learned my addr 172.21.15.67:0/1125853159 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae8be00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8c071850 0x7fae8c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae83400640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8c1466e0 0x7fae8c13bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 -- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8c1466e0 msgr2=0x7fae8c13bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8c1466e0 0x7fae8c13bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 -- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8c071850 msgr2=0x7fae8c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8c071850 0x7fae8c13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 -- 172.21.15.67:0/1125853159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0b0a20 con 0x7fae8c146da0 2024-04-17T10:29:57.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae8be00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8c071850 0x7fae8c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:57.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae90a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c13c430 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c780 tx=0x7fae8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:57.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae88c00640 1 -- 172.21.15.67:0/1125853159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84004b70 con 0x7fae8c146da0 2024-04-17T10:29:57.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae88c00640 1 -- 172.21.15.67:0/1125853159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae84004d10 con 0x7fae8c146da0 2024-04-17T10:29:57.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae88c00640 1 -- 172.21.15.67:0/1125853159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400aa90 con 0x7fae8c146da0 2024-04-17T10:29:57.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c13fbf0 con 0x7fae8c146da0 2024-04-17T10:29:57.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c1401b0 con 0x7fae8c146da0 2024-04-17T10:29:57.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.445+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c072d50 con 0x7fae8c146da0 2024-04-17T10:29:57.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.449+0000 7fae88c00640 1 -- 172.21.15.67:0/1125853159 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fae8400ac30 con 0x7fae8c146da0 2024-04-17T10:29:57.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.449+0000 7fae88c00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae68066f90 0x7fae68069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.449+0000 7fae88c00640 1 -- 172.21.15.67:0/1125853159 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fae840854b0 con 0x7fae8c146da0 2024-04-17T10:29:57.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.449+0000 7fae8be00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae68066f90 0x7fae68069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.449+0000 7fae8be00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae68066f90 0x7fae68069450 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fae7c005890 tx=0x7fae7c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:57.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.453+0000 7fae88c00640 1 -- 172.21.15.67:0/1125853159 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fae84020070 con 0x7fae8c146da0 2024-04-17T10:29:57.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.613+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae8c06b170 con 0x7fae68066f90 2024-04-17T10:29:57.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.621+0000 7fae88c00640 1 -- 172.21.15.67:0/1125853159 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252083 (secure 0 0 0) 0x7fae8c06b170 con 0x7fae68066f90 2024-04-17T10:29:57.616 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:57.618 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae68066f90 msgr2=0x7fae68069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae68066f90 0x7fae68069450 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fae7c005890 tx=0x7fae7c009040 comp rx=0 tx=0).stop 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 msgr2=0x7fae8c13c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c13c430 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fae8400c780 tx=0x7fae8400cc40 comp rx=0 tx=0).stop 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 shutdown_connections 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae68066f90 0x7fae68069450 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae8c146da0 0x7fae8c13c430 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae8c1466e0 0x7fae8c13bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 --2- 172.21.15.67:0/1125853159 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae8c071850 0x7fae8c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 >> 172.21.15.67:0/1125853159 conn(0x7fae8c06ac90 msgr2=0x7fae8c144d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 shutdown_connections 2024-04-17T10:29:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.625+0000 7fae92a00640 1 -- 172.21.15.67:0/1125853159 wait complete. 2024-04-17T10:29:57.638 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-04-17T10:29:56.205812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":326600,"kb_used_data":9160,"kb_used_omap":12,"kb_used_meta":317299,"kb_avail":754648120,"statfs":{"total":773094113280,"available":772759674880,"internally_reserved":0,"allocated":9379840,"data_stored":4957100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":324914737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002071"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624220+0000","last_change":"2024-04-17T10:29:48.624547+0000","last_active":"2024-04-17T10:29:49.624220+0000","last_peered":"2024-04-17T10:29:49.624220+0000","last_clean":"2024-04-17T10:29:49.624220+0000","last_became_active":"2024-04-17T10:29:48.623620+0000","last_became_peered":"2024-04-17T10:29:48.623620+0000","last_unstale":"2024-04-17T10:29:49.624220+0000","last_undegraded":"2024-04-17T10:29:49.624220+0000","last_fullsized":"2024-04-17T10:29:49.624220+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:03.584192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622732+0000","last_change":"2024-04-17T10:29:48.628788+0000","last_active":"2024-04-17T10:29:49.622732+0000","last_peered":"2024-04-17T10:29:49.622732+0000","last_clean":"2024-04-17T10:29:49.622732+0000","last_became_active":"2024-04-17T10:29:48.628522+0000","last_became_peered":"2024-04-17T10:29:48.628522+0000","last_unstale":"2024-04-17T10:29:49.622732+0000","last_undegraded":"2024-04-17T10:29:49.622732+0000","last_fullsized":"2024-04-17T10:29:49.622732+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:10:23.209350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473306+0000","last_change":"2024-04-17T10:29:48.638655+0000","last_active":"2024-04-17T10:29:50.473306+0000","last_peered":"2024-04-17T10:29:50.473306+0000","last_clean":"2024-04-17T10:29:50.473306+0000","last_became_active":"2024-04-17T10:29:48.638384+0000","last_became_peered":"2024-04-17T10:29:48.638384+0000","last_unstale":"2024-04-17T10:29:50.473306+0000","last_undegraded":"2024-04-17T10:29:50.473306+0000","last_fullsized":"2024-04-17T10:29:50.473306+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:10.041247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263588+0000","last_change":"2024-04-17T10:29:48.622391+0000","last_active":"2024-04-17T10:29:50.263588+0000","last_peered":"2024-04-17T10:29:50.263588+0000","last_clean":"2024-04-17T10:29:50.263588+0000","last_became_active":"2024-04-17T10:29:48.621932+0000","last_became_peered":"2024-04-17T10:29:48.621932+0000","last_unstale":"2024-04-17T10:29:50.263588+0000","last_undegraded":"2024-04-17T10:29:50.263588+0000","last_fullsized":"2024-04-17T10:29:50.263588+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:46.746210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631721+0000","last_change":"2024-04-17T10:29:48.628687+0000","last_active":"2024-04-17T10:29:49.631721+0000","last_peered":"2024-04-17T10:29:49.631721+0000","last_clean":"2024-04-17T10:29:49.631721+0000","last_became_active":"2024-04-17T10:29:48.628367+0000","last_became_peered":"2024-04-17T10:29:48.628367+0000","last_unstale":"2024-04-17T10:29:49.631721+0000","last_undegraded":"2024-04-17T10:29:49.631721+0000","last_fullsized":"2024-04-17T10:29:49.631721+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:41:17.740265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627412+0000","last_change":"2024-04-17T10:29:48.620231+0000","last_active":"2024-04-17T10:29:49.627412+0000","last_peered":"2024-04-17T10:29:49.627412+0000","last_clean":"2024-04-17T10:29:49.627412+0000","last_became_active":"2024-04-17T10:29:48.619957+0000","last_became_peered":"2024-04-17T10:29:48.619957+0000","last_unstale":"2024-04-17T10:29:49.627412+0000","last_undegraded":"2024-04-17T10:29:49.627412+0000","last_fullsized":"2024-04-17T10:29:49.627412+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:12.367407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469291+0000","last_change":"2024-04-17T10:29:48.623499+0000","last_active":"2024-04-17T10:29:50.469291+0000","last_peered":"2024-04-17T10:29:50.469291+0000","last_clean":"2024-04-17T10:29:50.469291+0000","last_became_active":"2024-04-17T10:29:48.623214+0000","last_became_peered":"2024-04-17T10:29:48.623214+0000","last_unstale":"2024-04-17T10:29:50.469291+0000","last_undegraded":"2024-04-17T10:29:50.469291+0000","last_fullsized":"2024-04-17T10:29:50.469291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:56:01.525634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631402+0000","last_change":"2024-04-17T10:29:48.629131+0000","last_active":"2024-04-17T10:29:49.631402+0000","last_peered":"2024-04-17T10:29:49.631402+0000","last_clean":"2024-04-17T10:29:49.631402+0000","last_became_active":"2024-04-17T10:29:48.628818+0000","last_became_peered":"2024-04-17T10:29:48.628818+0000","last_unstale":"2024-04-17T10:29:49.631402+0000","last_undegraded":"2024-04-17T10:29:49.631402+0000","last_fullsized":"2024-04-17T10:29:49.631402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:14.755052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469775+0000","last_change":"2024-04-17T10:29:48.613703+0000","last_active":"2024-04-17T10:29:50.469775+0000","last_peered":"2024-04-17T10:29:50.469775+0000","last_clean":"2024-04-17T10:29:50.469775+0000","last_became_active":"2024-04-17T10:29:48.613435+0000","last_became_peered":"2024-04-17T10:29:48.613435+0000","last_unstale":"2024-04-17T10:29:50.469775+0000","last_undegraded":"2024-04-17T10:29:50.469775+0000","last_fullsized":"2024-04-17T10:29:50.469775+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:07:49.081079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263601+0000","last_change":"2024-04-17T10:29:48.623135+0000","last_active":"2024-04-17T10:29:50.263601+0000","last_peered":"2024-04-17T10:29:50.263601+0000","last_clean":"2024-04-17T10:29:50.263601+0000","last_became_active":"2024-04-17T10:29:48.622795+0000","last_became_peered":"2024-04-17T10:29:48.622795+0000","last_unstale":"2024-04-17T10:29:50.263601+0000","last_undegraded":"2024-04-17T10:29:50.263601+0000","last_fullsized":"2024-04-17T10:29:50.263601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:09.035711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622714+0000","last_change":"2024-04-17T10:29:48.631279+0000","last_active":"2024-04-17T10:29:49.622714+0000","last_peered":"2024-04-17T10:29:49.622714+0000","last_clean":"2024-04-17T10:29:49.622714+0000","last_became_active":"2024-04-17T10:29:48.627396+0000","last_became_peered":"2024-04-17T10:29:48.627396+0000","last_unstale":"2024-04-17T10:29:49.622714+0000","last_undegraded":"2024-04-17T10:29:49.622714+0000","last_fullsized":"2024-04-17T10:29:49.622714+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:31.491002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263606+0000","last_change":"2024-04-17T10:29:48.624630+0000","last_active":"2024-04-17T10:29:50.263606+0000","last_peered":"2024-04-17T10:29:50.263606+0000","last_clean":"2024-04-17T10:29:50.263606+0000","last_became_active":"2024-04-17T10:29:48.624287+0000","last_became_peered":"2024-04-17T10:29:48.624287+0000","last_unstale":"2024-04-17T10:29:50.263606+0000","last_undegraded":"2024-04-17T10:29:50.263606+0000","last_fullsized":"2024-04-17T10:29:50.263606+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:54.561019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631758+0000","last_change":"2024-04-17T10:29:48.628214+0000","last_active":"2024-04-17T10:29:49.631758+0000","last_peered":"2024-04-17T10:29:49.631758+0000","last_clean":"2024-04-17T10:29:49.631758+0000","last_became_active":"2024-04-17T10:29:48.627533+0000","last_became_peered":"2024-04-17T10:29:48.627533+0000","last_unstale":"2024-04-17T10:29:49.631758+0000","last_undegraded":"2024-04-17T10:29:49.631758+0000","last_fullsized":"2024-04-17T10:29:49.631758+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:06.290628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627426+0000","last_change":"2024-04-17T10:29:48.620922+0000","last_active":"2024-04-17T10:29:49.627426+0000","last_peered":"2024-04-17T10:29:49.627426+0000","last_clean":"2024-04-17T10:29:49.627426+0000","last_became_active":"2024-04-17T10:29:48.620539+0000","last_became_peered":"2024-04-17T10:29:48.620539+0000","last_unstale":"2024-04-17T10:29:49.627426+0000","last_undegraded":"2024-04-17T10:29:49.627426+0000","last_fullsized":"2024-04-17T10:29:49.627426+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:31.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622389+0000","last_change":"2024-04-17T10:29:48.641288+0000","last_active":"2024-04-17T10:29:49.622389+0000","last_peered":"2024-04-17T10:29:49.622389+0000","last_clean":"2024-04-17T10:29:49.622389+0000","last_became_active":"2024-04-17T10:29:48.641041+0000","last_became_peered":"2024-04-17T10:29:48.641041+0000","last_unstale":"2024-04-17T10:29:49.622389+0000","last_undegraded":"2024-04-17T10:29:49.622389+0000","last_fullsized":"2024-04-17T10:29:49.622389+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:46.746210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624298+0000","last_change":"2024-04-17T10:29:48.618685+0000","last_active":"2024-04-17T10:29:49.624298+0000","last_peered":"2024-04-17T10:29:49.624298+0000","last_clean":"2024-04-17T10:29:49.624298+0000","last_became_active":"2024-04-17T10:29:48.618355+0000","last_became_peered":"2024-04-17T10:29:48.618355+0000","last_unstale":"2024-04-17T10:29:49.624298+0000","last_undegraded":"2024-04-17T10:29:49.624298+0000","last_fullsized":"2024-04-17T10:29:49.624298+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:32:31.093086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.632040+0000","last_change":"2024-04-17T10:29:48.632455+0000","last_active":"2024-04-17T10:29:49.632040+0000","last_peered":"2024-04-17T10:29:49.632040+0000","last_clean":"2024-04-17T10:29:49.632040+0000","last_became_active":"2024-04-17T10:29:48.632105+0000","last_became_peered":"2024-04-17T10:29:48.632105+0000","last_unstale":"2024-04-17T10:29:49.632040+0000","last_undegraded":"2024-04-17T10:29:49.632040+0000","last_fullsized":"2024-04-17T10:29:49.632040+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:09:06.007030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473376+0000","last_change":"2024-04-17T10:29:48.633707+0000","last_active":"2024-04-17T10:29:50.473376+0000","last_peered":"2024-04-17T10:29:50.473376+0000","last_clean":"2024-04-17T10:29:50.473376+0000","last_became_active":"2024-04-17T10:29:48.633424+0000","last_became_peered":"2024-04-17T10:29:48.633424+0000","last_unstale":"2024-04-17T10:29:50.473376+0000","last_undegraded":"2024-04-17T10:29:50.473376+0000","last_fullsized":"2024-04-17T10:29:50.473376+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:14.739124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627741+0000","last_change":"2024-04-17T10:29:48.628426+0000","last_active":"2024-04-17T10:29:49.627741+0000","last_peered":"2024-04-17T10:29:49.627741+0000","last_clean":"2024-04-17T10:29:49.627741+0000","last_became_active":"2024-04-17T10:29:48.628021+0000","last_became_peered":"2024-04-17T10:29:48.628021+0000","last_unstale":"2024-04-17T10:29:49.627741+0000","last_undegraded":"2024-04-17T10:29:49.627741+0000","last_fullsized":"2024-04-17T10:29:49.627741+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:00:03.584192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631859+0000","last_change":"2024-04-17T10:29:48.636282+0000","last_active":"2024-04-17T10:29:49.631859+0000","last_peered":"2024-04-17T10:29:49.631859+0000","last_clean":"2024-04-17T10:29:49.631859+0000","last_became_active":"2024-04-17T10:29:48.635825+0000","last_became_peered":"2024-04-17T10:29:48.635825+0000","last_unstale":"2024-04-17T10:29:49.631859+0000","last_undegraded":"2024-04-17T10:29:49.631859+0000","last_fullsized":"2024-04-17T10:29:49.631859+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:28:13.244001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469198+0000","last_change":"2024-04-17T10:29:48.635189+0000","last_active":"2024-04-17T10:29:50.469198+0000","last_peered":"2024-04-17T10:29:50.469198+0000","last_clean":"2024-04-17T10:29:50.469198+0000","last_became_active":"2024-04-17T10:29:48.634921+0000","last_became_peered":"2024-04-17T10:29:48.634921+0000","last_unstale":"2024-04-17T10:29:50.469198+0000","last_undegraded":"2024-04-17T10:29:50.469198+0000","last_fullsized":"2024-04-17T10:29:50.469198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:13:59.293622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631455+0000","last_change":"2024-04-17T10:29:48.621026+0000","last_active":"2024-04-17T10:29:49.631455+0000","last_peered":"2024-04-17T10:29:49.631455+0000","last_clean":"2024-04-17T10:29:49.631455+0000","last_became_active":"2024-04-17T10:29:48.620743+0000","last_became_peered":"2024-04-17T10:29:48.620743+0000","last_unstale":"2024-04-17T10:29:49.631455+0000","last_undegraded":"2024-04-17T10:29:49.631455+0000","last_fullsized":"2024-04-17T10:29:49.631455+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:32:17.781225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.468837+0000","last_change":"2024-04-17T10:29:48.616474+0000","last_active":"2024-04-17T10:29:50.468837+0000","last_peered":"2024-04-17T10:29:50.468837+0000","last_clean":"2024-04-17T10:29:50.468837+0000","last_became_active":"2024-04-17T10:29:48.616213+0000","last_became_peered":"2024-04-17T10:29:48.616213+0000","last_unstale":"2024-04-17T10:29:50.468837+0000","last_undegraded":"2024-04-17T10:29:50.468837+0000","last_fullsized":"2024-04-17T10:29:50.468837+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:57:32.124506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622319+0000","last_change":"2024-04-17T10:29:48.615060+0000","last_active":"2024-04-17T10:29:49.622319+0000","last_peered":"2024-04-17T10:29:49.622319+0000","last_clean":"2024-04-17T10:29:49.622319+0000","last_became_active":"2024-04-17T10:29:48.614732+0000","last_became_peered":"2024-04-17T10:29:48.614732+0000","last_unstale":"2024-04-17T10:29:49.622319+0000","last_undegraded":"2024-04-17T10:29:49.622319+0000","last_fullsized":"2024-04-17T10:29:49.622319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:05.285071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622815+0000","last_change":"2024-04-17T10:29:48.614690+0000","last_active":"2024-04-17T10:29:49.622815+0000","last_peered":"2024-04-17T10:29:49.622815+0000","last_clean":"2024-04-17T10:29:49.622815+0000","last_became_active":"2024-04-17T10:29:48.614408+0000","last_became_peered":"2024-04-17T10:29:48.614408+0000","last_unstale":"2024-04-17T10:29:49.622815+0000","last_undegraded":"2024-04-17T10:29:49.622815+0000","last_fullsized":"2024-04-17T10:29:49.622815+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:53.764108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263616+0000","last_change":"2024-04-17T10:29:48.626210+0000","last_active":"2024-04-17T10:29:50.263616+0000","last_peered":"2024-04-17T10:29:50.263616+0000","last_clean":"2024-04-17T10:29:50.263616+0000","last_became_active":"2024-04-17T10:29:48.625916+0000","last_became_peered":"2024-04-17T10:29:48.625916+0000","last_unstale":"2024-04-17T10:29:50.263616+0000","last_undegraded":"2024-04-17T10:29:50.263616+0000","last_fullsized":"2024-04-17T10:29:50.263616+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:56.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624132+0000","last_change":"2024-04-17T10:29:48.618251+0000","last_active":"2024-04-17T10:29:49.624132+0000","last_peered":"2024-04-17T10:29:49.624132+0000","last_clean":"2024-04-17T10:29:49.624132+0000","last_became_active":"2024-04-17T10:29:48.617966+0000","last_became_peered":"2024-04-17T10:29:48.617966+0000","last_unstale":"2024-04-17T10:29:49.624132+0000","last_undegraded":"2024-04-17T10:29:49.624132+0000","last_fullsized":"2024-04-17T10:29:49.624132+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:53.764108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631885+0000","last_change":"2024-04-17T10:29:48.625511+0000","last_active":"2024-04-17T10:29:49.631885+0000","last_peered":"2024-04-17T10:29:49.631885+0000","last_clean":"2024-04-17T10:29:49.631885+0000","last_became_active":"2024-04-17T10:29:48.625144+0000","last_became_peered":"2024-04-17T10:29:48.625144+0000","last_unstale":"2024-04-17T10:29:49.631885+0000","last_undegraded":"2024-04-17T10:29:49.631885+0000","last_fullsized":"2024-04-17T10:29:49.631885+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:39.246125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622671+0000","last_change":"2024-04-17T10:29:48.626108+0000","last_active":"2024-04-17T10:29:49.622671+0000","last_peered":"2024-04-17T10:29:49.622671+0000","last_clean":"2024-04-17T10:29:49.622671+0000","last_became_active":"2024-04-17T10:29:48.625797+0000","last_became_peered":"2024-04-17T10:29:48.625797+0000","last_unstale":"2024-04-17T10:29:49.622671+0000","last_undegraded":"2024-04-17T10:29:49.622671+0000","last_fullsized":"2024-04-17T10:29:49.622671+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:21:00.685909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623768+0000","last_change":"2024-04-17T10:29:48.631564+0000","last_active":"2024-04-17T10:29:49.623768+0000","last_peered":"2024-04-17T10:29:49.623768+0000","last_clean":"2024-04-17T10:29:49.623768+0000","last_became_active":"2024-04-17T10:29:48.631103+0000","last_became_peered":"2024-04-17T10:29:48.631103+0000","last_unstale":"2024-04-17T10:29:49.623768+0000","last_undegraded":"2024-04-17T10:29:49.623768+0000","last_fullsized":"2024-04-17T10:29:49.623768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:12.367407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473094+0000","last_change":"2024-04-17T10:29:48.634885+0000","last_active":"2024-04-17T10:29:50.473094+0000","last_peered":"2024-04-17T10:29:50.473094+0000","last_clean":"2024-04-17T10:29:50.473094+0000","last_became_active":"2024-04-17T10:29:48.634566+0000","last_became_peered":"2024-04-17T10:29:48.634566+0000","last_unstale":"2024-04-17T10:29:50.473094+0000","last_undegraded":"2024-04-17T10:29:50.473094+0000","last_fullsized":"2024-04-17T10:29:50.473094+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:56:01.525634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624099+0000","last_change":"2024-04-17T10:29:48.619966+0000","last_active":"2024-04-17T10:29:49.624099+0000","last_peered":"2024-04-17T10:29:49.624099+0000","last_clean":"2024-04-17T10:29:49.624099+0000","last_became_active":"2024-04-17T10:29:48.617677+0000","last_became_peered":"2024-04-17T10:29:48.617677+0000","last_unstale":"2024-04-17T10:29:49.624099+0000","last_undegraded":"2024-04-17T10:29:49.624099+0000","last_fullsized":"2024-04-17T10:29:49.624099+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:39.394407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473490+0000","last_change":"2024-04-17T10:29:48.633916+0000","last_active":"2024-04-17T10:29:50.473490+0000","last_peered":"2024-04-17T10:29:50.473490+0000","last_clean":"2024-04-17T10:29:50.473490+0000","last_became_active":"2024-04-17T10:29:48.633644+0000","last_became_peered":"2024-04-17T10:29:48.633644+0000","last_unstale":"2024-04-17T10:29:50.473490+0000","last_undegraded":"2024-04-17T10:29:50.473490+0000","last_fullsized":"2024-04-17T10:29:50.473490+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:28.462361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"82'4","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631957+0000","last_change":"2024-04-17T10:29:47.622473+0000","last_active":"2024-04-17T10:29:49.631957+0000","last_peered":"2024-04-17T10:29:49.631957+0000","last_clean":"2024-04-17T10:29:49.631957+0000","last_became_active":"2024-04-17T10:29:47.622183+0000","last_became_peered":"2024-04-17T10:29:47.622183+0000","last_unstale":"2024-04-17T10:29:49.631957+0000","last_undegraded":"2024-04-17T10:29:49.631957+0000","last_fullsized":"2024-04-17T10:29:49.631957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:09.994065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622660+0000","last_change":"2024-04-17T10:29:48.621313+0000","last_active":"2024-04-17T10:29:49.622660+0000","last_peered":"2024-04-17T10:29:49.622660+0000","last_clean":"2024-04-17T10:29:49.622660+0000","last_became_active":"2024-04-17T10:29:48.621040+0000","last_became_peered":"2024-04-17T10:29:48.621040+0000","last_unstale":"2024-04-17T10:29:49.622660+0000","last_undegraded":"2024-04-17T10:29:49.622660+0000","last_fullsized":"2024-04-17T10:29:49.622660+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:56.056162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624394+0000","last_change":"2024-04-17T10:29:47.614585+0000","last_active":"2024-04-17T10:29:49.624394+0000","last_peered":"2024-04-17T10:29:49.624394+0000","last_clean":"2024-04-17T10:29:49.624394+0000","last_became_active":"2024-04-17T10:29:47.614295+0000","last_became_peered":"2024-04-17T10:29:47.614295+0000","last_unstale":"2024-04-17T10:29:49.624394+0000","last_undegraded":"2024-04-17T10:29:49.624394+0000","last_fullsized":"2024-04-17T10:29:49.624394+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:53.553933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.468967+0000","last_change":"2024-04-17T10:29:48.613157+0000","last_active":"2024-04-17T10:29:50.468967+0000","last_peered":"2024-04-17T10:29:50.468967+0000","last_clean":"2024-04-17T10:29:50.468967+0000","last_became_active":"2024-04-17T10:29:48.612865+0000","last_became_peered":"2024-04-17T10:29:48.612865+0000","last_unstale":"2024-04-17T10:29:50.468967+0000","last_undegraded":"2024-04-17T10:29:50.468967+0000","last_fullsized":"2024-04-17T10:29:50.468967+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:01.176016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623755+0000","last_change":"2024-04-17T10:29:47.609395+0000","last_active":"2024-04-17T10:29:49.623755+0000","last_peered":"2024-04-17T10:29:49.623755+0000","last_clean":"2024-04-17T10:29:49.623755+0000","last_became_active":"2024-04-17T10:29:47.609097+0000","last_became_peered":"2024-04-17T10:29:47.609097+0000","last_unstale":"2024-04-17T10:29:49.623755+0000","last_undegraded":"2024-04-17T10:29:49.623755+0000","last_fullsized":"2024-04-17T10:29:49.623755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:41.846247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469414+0000","last_change":"2024-04-17T10:29:48.635119+0000","last_active":"2024-04-17T10:29:50.469414+0000","last_peered":"2024-04-17T10:29:50.469414+0000","last_clean":"2024-04-17T10:29:50.469414+0000","last_became_active":"2024-04-17T10:29:48.634841+0000","last_became_peered":"2024-04-17T10:29:48.634841+0000","last_unstale":"2024-04-17T10:29:50.469414+0000","last_undegraded":"2024-04-17T10:29:50.469414+0000","last_fullsized":"2024-04-17T10:29:50.469414+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:44:31.371286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469676+0000","last_change":"2024-04-17T10:29:47.609681+0000","last_active":"2024-04-17T10:29:50.469676+0000","last_peered":"2024-04-17T10:29:50.469676+0000","last_clean":"2024-04-17T10:29:50.469676+0000","last_became_active":"2024-04-17T10:29:47.609421+0000","last_became_peered":"2024-04-17T10:29:47.609421+0000","last_unstale":"2024-04-17T10:29:50.469676+0000","last_undegraded":"2024-04-17T10:29:50.469676+0000","last_fullsized":"2024-04-17T10:29:50.469676+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:57:19.717202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622740+0000","last_change":"2024-04-17T10:29:48.629710+0000","last_active":"2024-04-17T10:29:49.622740+0000","last_peered":"2024-04-17T10:29:49.622740+0000","last_clean":"2024-04-17T10:29:49.622740+0000","last_became_active":"2024-04-17T10:29:48.629147+0000","last_became_peered":"2024-04-17T10:29:48.629147+0000","last_unstale":"2024-04-17T10:29:49.622740+0000","last_undegraded":"2024-04-17T10:29:49.622740+0000","last_fullsized":"2024-04-17T10:29:49.622740+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:20:12.367407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469479+0000","last_change":"2024-04-17T10:29:47.608006+0000","last_active":"2024-04-17T10:29:50.469479+0000","last_peered":"2024-04-17T10:29:50.469479+0000","last_clean":"2024-04-17T10:29:50.469479+0000","last_became_active":"2024-04-17T10:29:47.607640+0000","last_became_peered":"2024-04-17T10:29:47.607640+0000","last_unstale":"2024-04-17T10:29:50.469479+0000","last_undegraded":"2024-04-17T10:29:50.469479+0000","last_fullsized":"2024-04-17T10:29:50.469479+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:39:12.331416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622786+0000","last_change":"2024-04-17T10:29:48.627703+0000","last_active":"2024-04-17T10:29:49.622786+0000","last_peered":"2024-04-17T10:29:49.622786+0000","last_clean":"2024-04-17T10:29:49.622786+0000","last_became_active":"2024-04-17T10:29:48.627071+0000","last_became_peered":"2024-04-17T10:29:48.627071+0000","last_unstale":"2024-04-17T10:29:49.622786+0000","last_undegraded":"2024-04-17T10:29:49.622786+0000","last_fullsized":"2024-04-17T10:29:49.622786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:31.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263175+0000","last_change":"2024-04-17T10:29:47.615219+0000","last_active":"2024-04-17T10:29:50.263175+0000","last_peered":"2024-04-17T10:29:50.263175+0000","last_clean":"2024-04-17T10:29:50.263175+0000","last_became_active":"2024-04-17T10:29:47.614925+0000","last_became_peered":"2024-04-17T10:29:47.614925+0000","last_unstale":"2024-04-17T10:29:50.263175+0000","last_undegraded":"2024-04-17T10:29:50.263175+0000","last_fullsized":"2024-04-17T10:29:50.263175+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:24.468474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469550+0000","last_change":"2024-04-17T10:29:48.612686+0000","last_active":"2024-04-17T10:29:50.469550+0000","last_peered":"2024-04-17T10:29:50.469550+0000","last_clean":"2024-04-17T10:29:50.469550+0000","last_became_active":"2024-04-17T10:29:48.612403+0000","last_became_peered":"2024-04-17T10:29:48.612403+0000","last_unstale":"2024-04-17T10:29:50.469550+0000","last_undegraded":"2024-04-17T10:29:50.469550+0000","last_fullsized":"2024-04-17T10:29:50.469550+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:32:51.336555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622482+0000","last_change":"2024-04-17T10:29:47.610640+0000","last_active":"2024-04-17T10:29:49.622482+0000","last_peered":"2024-04-17T10:29:49.622482+0000","last_clean":"2024-04-17T10:29:49.622482+0000","last_became_active":"2024-04-17T10:29:47.610355+0000","last_became_peered":"2024-04-17T10:29:47.610355+0000","last_unstale":"2024-04-17T10:29:49.622482+0000","last_undegraded":"2024-04-17T10:29:49.622482+0000","last_fullsized":"2024-04-17T10:29:49.622482+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:06.557009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469894+0000","last_change":"2024-04-17T10:29:48.627887+0000","last_active":"2024-04-17T10:29:50.469894+0000","last_peered":"2024-04-17T10:29:50.469894+0000","last_clean":"2024-04-17T10:29:50.469894+0000","last_became_active":"2024-04-17T10:29:48.627570+0000","last_became_peered":"2024-04-17T10:29:48.627570+0000","last_unstale":"2024-04-17T10:29:50.469894+0000","last_undegraded":"2024-04-17T10:29:50.469894+0000","last_fullsized":"2024-04-17T10:29:50.469894+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:56.246834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631389+0000","last_change":"2024-04-17T10:29:47.609754+0000","last_active":"2024-04-17T10:29:49.631389+0000","last_peered":"2024-04-17T10:29:49.631389+0000","last_clean":"2024-04-17T10:29:49.631389+0000","last_became_active":"2024-04-17T10:29:47.609526+0000","last_became_peered":"2024-04-17T10:29:47.609526+0000","last_unstale":"2024-04-17T10:29:49.631389+0000","last_undegraded":"2024-04-17T10:29:49.631389+0000","last_fullsized":"2024-04-17T10:29:49.631389+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:55.887140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622339+0000","last_change":"2024-04-17T10:29:48.626523+0000","last_active":"2024-04-17T10:29:49.622339+0000","last_peered":"2024-04-17T10:29:49.622339+0000","last_clean":"2024-04-17T10:29:49.622339+0000","last_became_active":"2024-04-17T10:29:48.626189+0000","last_became_peered":"2024-04-17T10:29:48.626189+0000","last_unstale":"2024-04-17T10:29:49.622339+0000","last_undegraded":"2024-04-17T10:29:49.622339+0000","last_fullsized":"2024-04-17T10:29:49.622339+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:54.561019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624227+0000","last_change":"2024-04-17T10:29:47.614092+0000","last_active":"2024-04-17T10:29:49.624227+0000","last_peered":"2024-04-17T10:29:49.624227+0000","last_clean":"2024-04-17T10:29:49.624227+0000","last_became_active":"2024-04-17T10:29:47.613772+0000","last_became_peered":"2024-04-17T10:29:47.613772+0000","last_unstale":"2024-04-17T10:29:49.624227+0000","last_undegraded":"2024-04-17T10:29:49.624227+0000","last_fullsized":"2024-04-17T10:29:49.624227+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:43:57.307677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263373+0000","last_change":"2024-04-17T10:29:48.624273+0000","last_active":"2024-04-17T10:29:50.263373+0000","last_peered":"2024-04-17T10:29:50.263373+0000","last_clean":"2024-04-17T10:29:50.263373+0000","last_became_active":"2024-04-17T10:29:48.624014+0000","last_became_peered":"2024-04-17T10:29:48.624014+0000","last_unstale":"2024-04-17T10:29:50.263373+0000","last_undegraded":"2024-04-17T10:29:50.263373+0000","last_fullsized":"2024-04-17T10:29:50.263373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:05.285071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263686+0000","last_change":"2024-04-17T10:29:47.607460+0000","last_active":"2024-04-17T10:29:50.263686+0000","last_peered":"2024-04-17T10:29:50.263686+0000","last_clean":"2024-04-17T10:29:50.263686+0000","last_became_active":"2024-04-17T10:29:47.607008+0000","last_became_peered":"2024-04-17T10:29:47.607008+0000","last_unstale":"2024-04-17T10:29:50.263686+0000","last_undegraded":"2024-04-17T10:29:50.263686+0000","last_fullsized":"2024-04-17T10:29:50.263686+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:55.502916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627523+0000","last_change":"2024-04-17T10:29:48.634420+0000","last_active":"2024-04-17T10:29:49.627523+0000","last_peered":"2024-04-17T10:29:49.627523+0000","last_clean":"2024-04-17T10:29:49.627523+0000","last_became_active":"2024-04-17T10:29:48.634018+0000","last_became_peered":"2024-04-17T10:29:48.634018+0000","last_unstale":"2024-04-17T10:29:49.627523+0000","last_undegraded":"2024-04-17T10:29:49.627523+0000","last_fullsized":"2024-04-17T10:29:49.627523+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:39.394407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"82'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631942+0000","last_change":"2024-04-17T10:29:47.608107+0000","last_active":"2024-04-17T10:29:49.631942+0000","last_peered":"2024-04-17T10:29:49.631942+0000","last_clean":"2024-04-17T10:29:49.631942+0000","last_became_active":"2024-04-17T10:29:47.607667+0000","last_became_peered":"2024-04-17T10:29:47.607667+0000","last_unstale":"2024-04-17T10:29:49.631942+0000","last_undegraded":"2024-04-17T10:29:49.631942+0000","last_fullsized":"2024-04-17T10:29:49.631942+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:43:11.584645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473418+0000","last_change":"2024-04-17T10:29:48.650004+0000","last_active":"2024-04-17T10:29:50.473418+0000","last_peered":"2024-04-17T10:29:50.473418+0000","last_clean":"2024-04-17T10:29:50.473418+0000","last_became_active":"2024-04-17T10:29:48.649710+0000","last_became_peered":"2024-04-17T10:29:48.649710+0000","last_unstale":"2024-04-17T10:29:50.473418+0000","last_undegraded":"2024-04-17T10:29:50.473418+0000","last_fullsized":"2024-04-17T10:29:50.473418+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:53.185803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"82'6","reported_seq":20,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469755+0000","last_change":"2024-04-17T10:29:47.612926+0000","last_active":"2024-04-17T10:29:50.469755+0000","last_peered":"2024-04-17T10:29:50.469755+0000","last_clean":"2024-04-17T10:29:50.469755+0000","last_became_active":"2024-04-17T10:29:47.612270+0000","last_became_peered":"2024-04-17T10:29:47.612270+0000","last_unstale":"2024-04-17T10:29:50.469755+0000","last_undegraded":"2024-04-17T10:29:50.469755+0000","last_fullsized":"2024-04-17T10:29:50.469755+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:40.127604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263737+0000","last_change":"2024-04-17T10:29:48.615481+0000","last_active":"2024-04-17T10:29:50.263737+0000","last_peered":"2024-04-17T10:29:50.263737+0000","last_clean":"2024-04-17T10:29:50.263737+0000","last_became_active":"2024-04-17T10:29:48.615031+0000","last_became_peered":"2024-04-17T10:29:48.615031+0000","last_unstale":"2024-04-17T10:29:50.263737+0000","last_undegraded":"2024-04-17T10:29:50.263737+0000","last_fullsized":"2024-04-17T10:29:50.263737+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:21:00.685909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263205+0000","last_change":"2024-04-17T10:29:47.606535+0000","last_active":"2024-04-17T10:29:50.263205+0000","last_peered":"2024-04-17T10:29:50.263205+0000","last_clean":"2024-04-17T10:29:50.263205+0000","last_became_active":"2024-04-17T10:29:47.605972+0000","last_became_peered":"2024-04-17T10:29:47.605972+0000","last_unstale":"2024-04-17T10:29:50.263205+0000","last_undegraded":"2024-04-17T10:29:50.263205+0000","last_fullsized":"2024-04-17T10:29:50.263205+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:24:27.488061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469800+0000","last_change":"2024-04-17T10:29:48.631017+0000","last_active":"2024-04-17T10:29:50.469800+0000","last_peered":"2024-04-17T10:29:50.469800+0000","last_clean":"2024-04-17T10:29:50.469800+0000","last_became_active":"2024-04-17T10:29:48.630723+0000","last_became_peered":"2024-04-17T10:29:48.630723+0000","last_unstale":"2024-04-17T10:29:50.469800+0000","last_undegraded":"2024-04-17T10:29:50.469800+0000","last_fullsized":"2024-04-17T10:29:50.469800+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:13:53.185803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623723+0000","last_change":"2024-04-17T10:29:47.613565+0000","last_active":"2024-04-17T10:29:49.623723+0000","last_peered":"2024-04-17T10:29:49.623723+0000","last_clean":"2024-04-17T10:29:49.623723+0000","last_became_active":"2024-04-17T10:29:47.612951+0000","last_became_peered":"2024-04-17T10:29:47.612951+0000","last_unstale":"2024-04-17T10:29:49.623723+0000","last_undegraded":"2024-04-17T10:29:49.623723+0000","last_fullsized":"2024-04-17T10:29:49.623723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:27:48.363695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469834+0000","last_change":"2024-04-17T10:29:48.618167+0000","last_active":"2024-04-17T10:29:50.469834+0000","last_peered":"2024-04-17T10:29:50.469834+0000","last_clean":"2024-04-17T10:29:50.469834+0000","last_became_active":"2024-04-17T10:29:48.617891+0000","last_became_peered":"2024-04-17T10:29:48.617891+0000","last_unstale":"2024-04-17T10:29:50.469834+0000","last_undegraded":"2024-04-17T10:29:50.469834+0000","last_fullsized":"2024-04-17T10:29:50.469834+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:28.462361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631415+0000","last_change":"2024-04-17T10:29:47.622528+0000","last_active":"2024-04-17T10:29:49.631415+0000","last_peered":"2024-04-17T10:29:49.631415+0000","last_clean":"2024-04-17T10:29:49.631415+0000","last_became_active":"2024-04-17T10:29:47.622269+0000","last_became_peered":"2024-04-17T10:29:47.622269+0000","last_unstale":"2024-04-17T10:29:49.631415+0000","last_undegraded":"2024-04-17T10:29:49.631415+0000","last_fullsized":"2024-04-17T10:29:49.631415+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:40.284702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.470000+0000","last_change":"2024-04-17T10:29:48.623100+0000","last_active":"2024-04-17T10:29:50.470000+0000","last_peered":"2024-04-17T10:29:50.470000+0000","last_clean":"2024-04-17T10:29:50.470000+0000","last_became_active":"2024-04-17T10:29:48.621739+0000","last_became_peered":"2024-04-17T10:29:48.621739+0000","last_unstale":"2024-04-17T10:29:50.470000+0000","last_undegraded":"2024-04-17T10:29:50.470000+0000","last_fullsized":"2024-04-17T10:29:50.470000+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:54:28.982617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"82'8","reported_seq":22,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622377+0000","last_change":"2024-04-17T10:29:47.617718+0000","last_active":"2024-04-17T10:29:49.622377+0000","last_peered":"2024-04-17T10:29:49.622377+0000","last_clean":"2024-04-17T10:29:49.622377+0000","last_became_active":"2024-04-17T10:29:47.616671+0000","last_became_peered":"2024-04-17T10:29:47.616671+0000","last_unstale":"2024-04-17T10:29:49.622377+0000","last_undegraded":"2024-04-17T10:29:49.622377+0000","last_fullsized":"2024-04-17T10:29:49.622377+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:12:38.277509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627301+0000","last_change":"2024-04-17T10:29:48.621840+0000","last_active":"2024-04-17T10:29:49.627301+0000","last_peered":"2024-04-17T10:29:49.627301+0000","last_clean":"2024-04-17T10:29:49.627301+0000","last_became_active":"2024-04-17T10:29:48.621577+0000","last_became_peered":"2024-04-17T10:29:48.621577+0000","last_unstale":"2024-04-17T10:29:49.627301+0000","last_undegraded":"2024-04-17T10:29:49.627301+0000","last_fullsized":"2024-04-17T10:29:49.627301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:06.285756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624274+0000","last_change":"2024-04-17T10:29:47.608319+0000","last_active":"2024-04-17T10:29:49.624274+0000","last_peered":"2024-04-17T10:29:49.624274+0000","last_clean":"2024-04-17T10:29:49.624274+0000","last_became_active":"2024-04-17T10:29:47.608039+0000","last_became_peered":"2024-04-17T10:29:47.608039+0000","last_unstale":"2024-04-17T10:29:49.624274+0000","last_undegraded":"2024-04-17T10:29:49.624274+0000","last_fullsized":"2024-04-17T10:29:49.624274+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:04.277986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631477+0000","last_change":"2024-04-17T10:29:48.623008+0000","last_active":"2024-04-17T10:29:49.631477+0000","last_peered":"2024-04-17T10:29:49.631477+0000","last_clean":"2024-04-17T10:29:49.631477+0000","last_became_active":"2024-04-17T10:29:48.622707+0000","last_became_peered":"2024-04-17T10:29:48.622707+0000","last_unstale":"2024-04-17T10:29:49.631477+0000","last_undegraded":"2024-04-17T10:29:49.631477+0000","last_fullsized":"2024-04-17T10:29:49.631477+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:44.546814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469612+0000","last_change":"2024-04-17T10:29:47.608095+0000","last_active":"2024-04-17T10:29:50.469612+0000","last_peered":"2024-04-17T10:29:50.469612+0000","last_clean":"2024-04-17T10:29:50.469612+0000","last_became_active":"2024-04-17T10:29:47.607646+0000","last_became_peered":"2024-04-17T10:29:47.607646+0000","last_unstale":"2024-04-17T10:29:50.469612+0000","last_undegraded":"2024-04-17T10:29:50.469612+0000","last_fullsized":"2024-04-17T10:29:50.469612+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:58:28.548917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631579+0000","last_change":"2024-04-17T10:29:48.631862+0000","last_active":"2024-04-17T10:29:49.631579+0000","last_peered":"2024-04-17T10:29:49.631579+0000","last_clean":"2024-04-17T10:29:49.631579+0000","last_became_active":"2024-04-17T10:29:48.630560+0000","last_became_peered":"2024-04-17T10:29:48.630560+0000","last_unstale":"2024-04-17T10:29:49.631579+0000","last_undegraded":"2024-04-17T10:29:49.631579+0000","last_fullsized":"2024-04-17T10:29:49.631579+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:23:37.373393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"82'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631994+0000","last_change":"2024-04-17T10:29:47.611758+0000","last_active":"2024-04-17T10:29:49.631994+0000","last_peered":"2024-04-17T10:29:49.631994+0000","last_clean":"2024-04-17T10:29:49.631994+0000","last_became_active":"2024-04-17T10:29:47.611490+0000","last_became_peered":"2024-04-17T10:29:47.611490+0000","last_unstale":"2024-04-17T10:29:49.631994+0000","last_undegraded":"2024-04-17T10:29:49.631994+0000","last_fullsized":"2024-04-17T10:29:49.631994+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:50:34.562845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627650+0000","last_change":"2024-04-17T10:29:48.632172+0000","last_active":"2024-04-17T10:29:49.627650+0000","last_peered":"2024-04-17T10:29:49.627650+0000","last_clean":"2024-04-17T10:29:49.627650+0000","last_became_active":"2024-04-17T10:29:48.631705+0000","last_became_peered":"2024-04-17T10:29:48.631705+0000","last_unstale":"2024-04-17T10:29:49.627650+0000","last_undegraded":"2024-04-17T10:29:49.627650+0000","last_fullsized":"2024-04-17T10:29:49.627650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:53.764108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473121+0000","last_change":"2024-04-17T10:29:47.614754+0000","last_active":"2024-04-17T10:29:50.473121+0000","last_peered":"2024-04-17T10:29:50.473121+0000","last_clean":"2024-04-17T10:29:50.473121+0000","last_became_active":"2024-04-17T10:29:47.614120+0000","last_became_peered":"2024-04-17T10:29:47.614120+0000","last_unstale":"2024-04-17T10:29:50.473121+0000","last_undegraded":"2024-04-17T10:29:50.473121+0000","last_fullsized":"2024-04-17T10:29:50.473121+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:17.862910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622643+0000","last_change":"2024-04-17T10:29:48.620326+0000","last_active":"2024-04-17T10:29:49.622643+0000","last_peered":"2024-04-17T10:29:49.622643+0000","last_clean":"2024-04-17T10:29:49.622643+0000","last_became_active":"2024-04-17T10:29:48.620066+0000","last_became_peered":"2024-04-17T10:29:48.620066+0000","last_unstale":"2024-04-17T10:29:49.622643+0000","last_undegraded":"2024-04-17T10:29:49.622643+0000","last_fullsized":"2024-04-17T10:29:49.622643+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:54:09.035711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627622+0000","last_change":"2024-04-17T10:29:47.621663+0000","last_active":"2024-04-17T10:29:49.627622+0000","last_peered":"2024-04-17T10:29:49.627622+0000","last_clean":"2024-04-17T10:29:49.627622+0000","last_became_active":"2024-04-17T10:29:47.621342+0000","last_became_peered":"2024-04-17T10:29:47.621342+0000","last_unstale":"2024-04-17T10:29:49.627622+0000","last_undegraded":"2024-04-17T10:29:49.627622+0000","last_fullsized":"2024-04-17T10:29:49.627622+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:40:01.953755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622786+0000","last_change":"2024-04-17T10:29:48.615967+0000","last_active":"2024-04-17T10:29:49.622786+0000","last_peered":"2024-04-17T10:29:49.622786+0000","last_clean":"2024-04-17T10:29:49.622786+0000","last_became_active":"2024-04-17T10:29:48.615692+0000","last_became_peered":"2024-04-17T10:29:48.615692+0000","last_unstale":"2024-04-17T10:29:49.622786+0000","last_undegraded":"2024-04-17T10:29:49.622786+0000","last_fullsized":"2024-04-17T10:29:49.622786+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:49:39.394407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627344+0000","last_change":"2024-04-17T10:29:47.620608+0000","last_active":"2024-04-17T10:29:49.627344+0000","last_peered":"2024-04-17T10:29:49.627344+0000","last_clean":"2024-04-17T10:29:49.627344+0000","last_became_active":"2024-04-17T10:29:47.620350+0000","last_became_peered":"2024-04-17T10:29:47.620350+0000","last_unstale":"2024-04-17T10:29:49.627344+0000","last_undegraded":"2024-04-17T10:29:49.627344+0000","last_fullsized":"2024-04-17T10:29:49.627344+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:51:04.277986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.470084+0000","last_change":"2024-04-17T10:29:48.618486+0000","last_active":"2024-04-17T10:29:50.470084+0000","last_peered":"2024-04-17T10:29:50.470084+0000","last_clean":"2024-04-17T10:29:50.470084+0000","last_became_active":"2024-04-17T10:29:48.618217+0000","last_became_peered":"2024-04-17T10:29:48.618217+0000","last_unstale":"2024-04-17T10:29:50.470084+0000","last_undegraded":"2024-04-17T10:29:50.470084+0000","last_fullsized":"2024-04-17T10:29:50.470084+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:59:19.299880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627306+0000","last_change":"2024-04-17T10:29:47.619479+0000","last_active":"2024-04-17T10:29:49.627306+0000","last_peered":"2024-04-17T10:29:49.627306+0000","last_clean":"2024-04-17T10:29:49.627306+0000","last_became_active":"2024-04-17T10:29:47.619168+0000","last_became_peered":"2024-04-17T10:29:47.619168+0000","last_unstale":"2024-04-17T10:29:49.627306+0000","last_undegraded":"2024-04-17T10:29:49.627306+0000","last_fullsized":"2024-04-17T10:29:49.627306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:11:41.846247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.468823+0000","last_change":"2024-04-17T10:29:48.615973+0000","last_active":"2024-04-17T10:29:50.468823+0000","last_peered":"2024-04-17T10:29:50.468823+0000","last_clean":"2024-04-17T10:29:50.468823+0000","last_became_active":"2024-04-17T10:29:48.615604+0000","last_became_peered":"2024-04-17T10:29:48.615604+0000","last_unstale":"2024-04-17T10:29:50.468823+0000","last_undegraded":"2024-04-17T10:29:50.468823+0000","last_fullsized":"2024-04-17T10:29:50.468823+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:50:18.869996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"82'2","reported_seq":16,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.263588+0000","last_change":"2024-04-17T10:29:47.603453+0000","last_active":"2024-04-17T10:29:50.263588+0000","last_peered":"2024-04-17T10:29:50.263588+0000","last_clean":"2024-04-17T10:29:50.263588+0000","last_became_active":"2024-04-17T10:29:47.603169+0000","last_became_peered":"2024-04-17T10:29:47.603169+0000","last_unstale":"2024-04-17T10:29:50.263588+0000","last_undegraded":"2024-04-17T10:29:50.263588+0000","last_fullsized":"2024-04-17T10:29:50.263588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:51.330142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469445+0000","last_change":"2024-04-17T10:29:48.623902+0000","last_active":"2024-04-17T10:29:50.469445+0000","last_peered":"2024-04-17T10:29:50.469445+0000","last_clean":"2024-04-17T10:29:50.469445+0000","last_became_active":"2024-04-17T10:29:48.623546+0000","last_became_peered":"2024-04-17T10:29:48.623546+0000","last_unstale":"2024-04-17T10:29:50.469445+0000","last_undegraded":"2024-04-17T10:29:50.469445+0000","last_fullsized":"2024-04-17T10:29:50.469445+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:10.041247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"83'1","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622829+0000","last_change":"2024-04-17T10:29:47.619750+0000","last_active":"2024-04-17T10:29:49.622829+0000","last_peered":"2024-04-17T10:29:49.622829+0000","last_clean":"2024-04-17T10:29:49.622829+0000","last_became_active":"2024-04-17T10:29:47.618886+0000","last_became_peered":"2024-04-17T10:29:47.618886+0000","last_unstale":"2024-04-17T10:29:49.622829+0000","last_undegraded":"2024-04-17T10:29:49.622829+0000","last_fullsized":"2024-04-17T10:29:49.622829+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:04:31.601997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469005+0000","last_change":"2024-04-17T10:29:48.613263+0000","last_active":"2024-04-17T10:29:50.469005+0000","last_peered":"2024-04-17T10:29:50.469005+0000","last_clean":"2024-04-17T10:29:50.469005+0000","last_became_active":"2024-04-17T10:29:48.612973+0000","last_became_peered":"2024-04-17T10:29:48.612973+0000","last_unstale":"2024-04-17T10:29:50.469005+0000","last_undegraded":"2024-04-17T10:29:50.469005+0000","last_fullsized":"2024-04-17T10:29:50.469005+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:24:54.375698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624016+0000","last_change":"2024-04-17T10:29:47.615534+0000","last_active":"2024-04-17T10:29:49.624016+0000","last_peered":"2024-04-17T10:29:49.624016+0000","last_clean":"2024-04-17T10:29:49.624016+0000","last_became_active":"2024-04-17T10:29:47.614432+0000","last_became_peered":"2024-04-17T10:29:47.614432+0000","last_unstale":"2024-04-17T10:29:49.624016+0000","last_undegraded":"2024-04-17T10:29:49.624016+0000","last_fullsized":"2024-04-17T10:29:49.624016+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:40:01.953755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.622733+0000","last_change":"2024-04-17T10:29:48.630697+0000","last_active":"2024-04-17T10:29:49.622733+0000","last_peered":"2024-04-17T10:29:49.622733+0000","last_clean":"2024-04-17T10:29:49.622733+0000","last_became_active":"2024-04-17T10:29:48.626157+0000","last_became_peered":"2024-04-17T10:29:48.626157+0000","last_unstale":"2024-04-17T10:29:49.622733+0000","last_undegraded":"2024-04-17T10:29:49.622733+0000","last_fullsized":"2024-04-17T10:29:49.622733+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:06.285756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623827+0000","last_change":"2024-04-17T10:29:47.614945+0000","last_active":"2024-04-17T10:29:49.623827+0000","last_peered":"2024-04-17T10:29:49.623827+0000","last_clean":"2024-04-17T10:29:49.623827+0000","last_became_active":"2024-04-17T10:29:47.614276+0000","last_became_peered":"2024-04-17T10:29:47.614276+0000","last_unstale":"2024-04-17T10:29:49.623827+0000","last_undegraded":"2024-04-17T10:29:49.623827+0000","last_fullsized":"2024-04-17T10:29:49.623827+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:55:57.488244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469695+0000","last_change":"2024-04-17T10:29:48.622459+0000","last_active":"2024-04-17T10:29:50.469695+0000","last_peered":"2024-04-17T10:29:50.469695+0000","last_clean":"2024-04-17T10:29:50.469695+0000","last_became_active":"2024-04-17T10:29:48.622169+0000","last_became_peered":"2024-04-17T10:29:48.622169+0000","last_unstale":"2024-04-17T10:29:50.469695+0000","last_undegraded":"2024-04-17T10:29:50.469695+0000","last_fullsized":"2024-04-17T10:29:50.469695+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:06:14.739124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469513+0000","last_change":"2024-04-17T10:29:47.609861+0000","last_active":"2024-04-17T10:29:50.469513+0000","last_peered":"2024-04-17T10:29:50.469513+0000","last_clean":"2024-04-17T10:29:50.469513+0000","last_became_active":"2024-04-17T10:29:47.609241+0000","last_became_peered":"2024-04-17T10:29:47.609241+0000","last_unstale":"2024-04-17T10:29:50.469513+0000","last_undegraded":"2024-04-17T10:29:50.469513+0000","last_fullsized":"2024-04-17T10:29:50.469513+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:48:17.280119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469950+0000","last_change":"2024-04-17T10:29:48.624773+0000","last_active":"2024-04-17T10:29:50.469950+0000","last_peered":"2024-04-17T10:29:50.469950+0000","last_clean":"2024-04-17T10:29:50.469950+0000","last_became_active":"2024-04-17T10:29:48.623489+0000","last_became_peered":"2024-04-17T10:29:48.623489+0000","last_unstale":"2024-04-17T10:29:50.469950+0000","last_undegraded":"2024-04-17T10:29:50.469950+0000","last_fullsized":"2024-04-17T10:29:50.469950+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:20.694608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.631520+0000","last_change":"2024-04-17T10:29:47.619501+0000","last_active":"2024-04-17T10:29:49.631520+0000","last_peered":"2024-04-17T10:29:49.631520+0000","last_clean":"2024-04-17T10:29:49.631520+0000","last_became_active":"2024-04-17T10:29:47.619199+0000","last_became_peered":"2024-04-17T10:29:47.619199+0000","last_unstale":"2024-04-17T10:29:49.631520+0000","last_undegraded":"2024-04-17T10:29:49.631520+0000","last_fullsized":"2024-04-17T10:29:49.631520+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:08.704951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473540+0000","last_change":"2024-04-17T10:29:48.645174+0000","last_active":"2024-04-17T10:29:50.473540+0000","last_peered":"2024-04-17T10:29:50.473540+0000","last_clean":"2024-04-17T10:29:50.473540+0000","last_became_active":"2024-04-17T10:29:48.644891+0000","last_became_peered":"2024-04-17T10:29:48.644891+0000","last_unstale":"2024-04-17T10:29:50.473540+0000","last_undegraded":"2024-04-17T10:29:50.473540+0000","last_fullsized":"2024-04-17T10:29:50.473540+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:20.694608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.469356+0000","last_change":"2024-04-17T10:29:47.616872+0000","last_active":"2024-04-17T10:29:50.469356+0000","last_peered":"2024-04-17T10:29:50.469356+0000","last_clean":"2024-04-17T10:29:50.469356+0000","last_became_active":"2024-04-17T10:29:47.616590+0000","last_became_peered":"2024-04-17T10:29:47.616590+0000","last_unstale":"2024-04-17T10:29:50.469356+0000","last_undegraded":"2024-04-17T10:29:50.469356+0000","last_fullsized":"2024-04-17T10:29:50.469356+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:18:19.418053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.473501+0000","last_change":"2024-04-17T10:29:48.634278+0000","last_active":"2024-04-17T10:29:50.473501+0000","last_peered":"2024-04-17T10:29:50.473501+0000","last_clean":"2024-04-17T10:29:50.473501+0000","last_became_active":"2024-04-17T10:29:48.633977+0000","last_became_peered":"2024-04-17T10:29:48.633977+0000","last_unstale":"2024-04-17T10:29:50.473501+0000","last_undegraded":"2024-04-17T10:29:50.473501+0000","last_fullsized":"2024-04-17T10:29:50.473501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:49:56.246834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:50.472928+0000","last_change":"2024-04-17T10:29:47.620589+0000","last_active":"2024-04-17T10:29:50.472928+0000","last_peered":"2024-04-17T10:29:50.472928+0000","last_clean":"2024-04-17T10:29:50.472928+0000","last_became_active":"2024-04-17T10:29:47.620292+0000","last_became_peered":"2024-04-17T10:29:47.620292+0000","last_unstale":"2024-04-17T10:29:50.472928+0000","last_undegraded":"2024-04-17T10:29:50.472928+0000","last_fullsized":"2024-04-17T10:29:50.472928+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:13:13.905730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.623962+0000","last_change":"2024-04-17T10:29:48.632954+0000","last_active":"2024-04-17T10:29:49.623962+0000","last_peered":"2024-04-17T10:29:49.623962+0000","last_clean":"2024-04-17T10:29:49.623962+0000","last_became_active":"2024-04-17T10:29:48.632778+0000","last_became_peered":"2024-04-17T10:29:48.632778+0000","last_unstale":"2024-04-17T10:29:49.623962+0000","last_undegraded":"2024-04-17T10:29:49.623962+0000","last_fullsized":"2024-04-17T10:29:49.623962+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:47.585936+0000","last_clean_scrub_stamp":"2024-04-17T10:29:47.585936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:30:31.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":14,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.627712+0000","last_change":"2024-04-17T10:29:47.620546+0000","last_active":"2024-04-17T10:29:49.627712+0000","last_peered":"2024-04-17T10:29:49.627712+0000","last_clean":"2024-04-17T10:29:49.627712+0000","last_became_active":"2024-04-17T10:29:47.620261+0000","last_became_peered":"2024-04-17T10:29:47.620261+0000","last_unstale":"2024-04-17T10:29:49.627712+0000","last_undegraded":"2024-04-17T10:29:49.627712+0000","last_fullsized":"2024-04-17T10:29:49.627712+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:29:46.578850+0000","last_clean_scrub_stamp":"2024-04-17T10:29:46.578850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:53.553933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":286,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-17T10:29:49.624157+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:29:49.624157+0000","last_peered":"2024-04-17T10:29:49.624157+0000","last_clean":"2024-04-17T10:29:49.624157+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:29:49.624157+0000","last_undegraded":"2024-04-17T10:29:49.624157+0000","last_fullsized":"2024-04-17T10:29:49.624157+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41084,"kb_used_data":1576,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330756,"statfs":{"total":96636764160,"available":96594694144,"internally_reserved":0,"allocated":1613824,"data_stored":1061833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335492,"statfs":{"total":96636764160,"available":96599543808,"internally_reserved":0,"allocated":1028096,"data_stored":473055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574913,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335504,"statfs":{"total":96636764160,"available":96599556096,"internally_reserved":0,"allocated":1019904,"data_stored":471477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40512,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331328,"statfs":{"total":96636764160,"available":96595279872,"internally_reserved":0,"allocated":1036288,"data_stored":473159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574913,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1019904,"data_stored":472025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36352,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335488,"statfs":{"total":96636764160,"available":96599539712,"internally_reserved":0,"allocated":1028096,"data_stored":472185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36340,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335500,"statfs":{"total":96636764160,"available":96599552000,"internally_reserved":0,"allocated":1024000,"data_stored":471555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63288,"kb_used_data":1572,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94308552,"statfs":{"total":96636764160,"available":96571957248,"internally_reserved":0,"allocated":1609728,"data_stored":1061811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:29:57.642 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:29:57.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:29:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.837+0000 7fac8fe00640 1 Processor -- start 2024-04-17T10:29:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.837+0000 7fac8fe00640 1 -- start start 2024-04-17T10:29:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac90160f40 0x7fac900b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac9009a160 0x7fac900cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac900cf9d0 con 0x7fac9006c7b0 2024-04-17T10:29:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9006cce0 con 0x7fac9009a160 2024-04-17T10:29:57.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8fe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac900cfc40 con 0x7fac90160f40 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac9009a160 0x7fac900cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac86000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac9006c7b0 0x7fac90099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50958/0 (socket says 172.21.15.67:50958) 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac86000640 1 -- 172.21.15.67:0/3506809327 learned_addr learned my addr 172.21.15.67:0/3506809327 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8ea00640 1 --2- 172.21.15.67:0/3506809327 >> v2:172.21.15.86:3300/0 conn(0x7fac90160f40 0x7fac900b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8f400640 1 -- 172.21.15.67:0/3506809327 >> v2:172.21.15.86:3300/0 conn(0x7fac90160f40 msgr2=0x7fac900b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8f400640 1 --2- 172.21.15.67:0/3506809327 >> v2:172.21.15.86:3300/0 conn(0x7fac90160f40 0x7fac900b2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8f400640 1 -- 172.21.15.67:0/3506809327 >> v1:172.21.15.67:6789/0 conn(0x7fac9006c7b0 legacy=0x7fac90099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8f400640 1 -- 172.21.15.67:0/3506809327 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900cff20 con 0x7fac9009a160 2024-04-17T10:29:57.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8e000640 1 -- 172.21.15.67:0/3506809327 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642501773 0 0) 0x7fac900cf9d0 con 0x7fac9006c7b0 2024-04-17T10:29:57.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8f400640 1 --2- 172.21.15.67:0/3506809327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac900cbcf0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fac9006ecf0 tx=0x7fac8801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1821c20d400f5acf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.841+0000 7fac8e000640 1 -- 172.21.15.67:0/3506809327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac880b1070 con 0x7fac9009a160 2024-04-17T10:29:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8e000640 1 -- 172.21.15.67:0/3506809327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac880ac5f0 con 0x7fac9009a160 2024-04-17T10:29:57.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8e000640 1 -- 172.21.15.67:0/3506809327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac880ac9c0 con 0x7fac9009a160 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- 172.21.15.67:0/3506809327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 msgr2=0x7fac900cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 --2- 172.21.15.67:0/3506809327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac900cbcf0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fac9006ecf0 tx=0x7fac8801c480 comp rx=0 tx=0).stop 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- 172.21.15.67:0/3506809327 shutdown_connections 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 --2- 172.21.15.67:0/3506809327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac900cbcf0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 --2- 172.21.15.67:0/3506809327 >> v2:172.21.15.86:3300/0 conn(0x7fac90160f40 0x7fac900b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- 172.21.15.67:0/3506809327 >> 172.21.15.67:0/3506809327 conn(0x7fac900b3190 msgr2=0x7fac900b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- 172.21.15.67:0/3506809327 shutdown_connections 2024-04-17T10:29:57.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- 172.21.15.67:0/3506809327 wait complete. 2024-04-17T10:29:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 Processor -- start 2024-04-17T10:29:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- start start 2024-04-17T10:29:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9006c7b0 0x7fac90154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac90155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac90160f40 0x7fac90155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac900d0c50 con 0x7fac9009a160 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac90161530 con 0x7fac9006c7b0 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac900d0dd0 con 0x7fac90160f40 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac90155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac90155830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57360/0 (socket says 172.21.15.67:57360) 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 -- 172.21.15.67:0/2967739401 learned_addr learned my addr 172.21.15.67:0/2967739401 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8f400640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac90160f40 0x7fac90155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac8ea00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9006c7b0 0x7fac90154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 -- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac90160f40 msgr2=0x7fac90155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac90160f40 0x7fac90155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 -- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9006c7b0 msgr2=0x7fac90154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9006c7b0 0x7fac90154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.845+0000 7fac86000640 1 -- 172.21.15.67:0/2967739401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900c9610 con 0x7fac9009a160 2024-04-17T10:29:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac86000640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac90155830 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fac780011f0 tx=0x7fac7800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac87e00640 1 -- 172.21.15.67:0/2967739401 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78018070 con 0x7fac9009a160 2024-04-17T10:29:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac87e00640 1 -- 172.21.15.67:0/2967739401 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac780074e0 con 0x7fac9009a160 2024-04-17T10:29:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac87e00640 1 -- 172.21.15.67:0/2967739401 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78004a10 con 0x7fac9009a160 2024-04-17T10:29:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac900c98f0 con 0x7fac9009a160 2024-04-17T10:29:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac900c9eb0 con 0x7fac9009a160 2024-04-17T10:29:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9015a850 con 0x7fac9009a160 2024-04-17T10:29:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac87e00640 1 -- 172.21.15.67:0/2967739401 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fac7801b070 con 0x7fac9009a160 2024-04-17T10:29:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac87e00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac6c066f60 0x7fac6c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac87e00640 1 -- 172.21.15.67:0/2967739401 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fac78084520 con 0x7fac9009a160 2024-04-17T10:29:57.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.849+0000 7fac8ea00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac6c066f60 0x7fac6c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:57.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.853+0000 7fac8ea00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac6c066f60 0x7fac6c069420 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fac80003fd0 tx=0x7fac800039a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:57.853+0000 7fac87e00640 1 -- 172.21.15.67:0/2967739401 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fac9015a850 con 0x7fac9009a160 2024-04-17T10:29:58.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.073+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fac90074dc0 con 0x7fac9009a160 2024-04-17T10:29:58.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.073+0000 7fac87e00640 1 -- 172.21.15.67:0/2967739401 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7fac780553a0 con 0x7fac9009a160 2024-04-17T10:29:58.069 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:29:58.069 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:29:58.069 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:29:58.069 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:29:58.069 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:29:58.069 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:29:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac6c066f60 msgr2=0x7fac6c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac6c066f60 0x7fac6c069420 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fac80003fd0 tx=0x7fac800039a0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 msgr2=0x7fac90155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac90155830 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fac780011f0 tx=0x7fac7800c9b0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 shutdown_connections 2024-04-17T10:29:58.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac6c066f60 0x7fac6c069420 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac90160f40 0x7fac90155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac9009a160 0x7fac90155830 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 --2- 172.21.15.67:0/2967739401 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac9006c7b0 0x7fac90154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 >> 172.21.15.67:0/2967739401 conn(0x7fac900b3190 msgr2=0x7fac90099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 shutdown_connections 2024-04-17T10:29:58.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.077+0000 7fac8fe00640 1 -- 172.21.15.67:0/2967739401 wait complete. 2024-04-17T10:29:58.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:29:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.237+0000 7f181fa00640 1 Processor -- start 2024-04-17T10:29:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.237+0000 7f181fa00640 1 -- start start 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181fa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806eac0 0x7f1818143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181fa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18180cf0c0 0x7f1818144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181fa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1818144560 con 0x7f181806e6e0 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181fa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181815ae40 con 0x7f18180cf0c0 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181fa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1818071440 con 0x7f181806eac0 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18180cf0c0 0x7f1818144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181d000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f181806e6e0 0x7f18180ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50964/0 (socket says 172.21.15.67:50964) 2024-04-17T10:29:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181d000640 1 -- 172.21.15.67:0/1836168881 learned_addr learned my addr 172.21.15.67:0/1836168881 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:58.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f180fe00640 1 --2- 172.21.15.67:0/1836168881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806eac0 0x7f1818143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:58.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181da00640 1 -- 172.21.15.67:0/1836168881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806eac0 msgr2=0x7f1818143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:58.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181da00640 1 --2- 172.21.15.67:0/1836168881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806eac0 0x7f1818143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181da00640 1 -- 172.21.15.67:0/1836168881 >> v1:172.21.15.67:6789/0 conn(0x7f181806e6e0 legacy=0x7f18180ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181da00640 1 -- 172.21.15.67:0/1836168881 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818071720 con 0x7f18180cf0c0 2024-04-17T10:29:58.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f1817e00640 1 -- 172.21.15.67:0/1836168881 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510266700 0 0) 0x7f1818144560 con 0x7f181806e6e0 2024-04-17T10:29:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f181da00640 1 --2- 172.21.15.67:0/1836168881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f1818144020 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1810009d90 tx=0x7f181001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae9f5614035a64 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.241+0000 7f1817e00640 1 -- 172.21.15.67:0/1836168881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100b2070 con 0x7f18180cf0c0 2024-04-17T10:29:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f1817e00640 1 -- 172.21.15.67:0/1836168881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18100a5970 con 0x7f18180cf0c0 2024-04-17T10:29:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f1817e00640 1 -- 172.21.15.67:0/1836168881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100a5d40 con 0x7f18180cf0c0 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- 172.21.15.67:0/1836168881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 msgr2=0x7f1818144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 --2- 172.21.15.67:0/1836168881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f1818144020 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f1810009d90 tx=0x7f181001c8b0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- 172.21.15.67:0/1836168881 shutdown_connections 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 --2- 172.21.15.67:0/1836168881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f1818144020 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 --2- 172.21.15.67:0/1836168881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806eac0 0x7f1818143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- 172.21.15.67:0/1836168881 >> 172.21.15.67:0/1836168881 conn(0x7f181806ac90 msgr2=0x7f181806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- 172.21.15.67:0/1836168881 shutdown_connections 2024-04-17T10:29:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- 172.21.15.67:0/1836168881 wait complete. 2024-04-17T10:29:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 Processor -- start 2024-04-17T10:29:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- start start 2024-04-17T10:29:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806e6e0 0x7f1818138370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f181806eac0 0x7f18181f99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f181813c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180b3a40 con 0x7f18180cf0c0 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18180b38c0 con 0x7f181806eac0 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181fa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18180b3bc0 con 0x7f181806e6e0 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f181813c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f181813c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57374/0 (socket says 172.21.15.67:57374) 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181da00640 1 -- 172.21.15.67:0/760261010 learned_addr learned my addr 172.21.15.67:0/760261010 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f180fe00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f181806eac0 0x7f18181f99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.245+0000 7f181d000640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806e6e0 0x7f1818138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181da00640 1 -- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806e6e0 msgr2=0x7f1818138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181da00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806e6e0 0x7f1818138370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181da00640 1 -- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f181806eac0 msgr2=0x7f18181f99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181da00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f181806eac0 0x7f18181f99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181da00640 1 -- 172.21.15.67:0/760261010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1818138950 con 0x7f18180cf0c0 2024-04-17T10:29:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f180fe00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f181806eac0 0x7f18181f99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181da00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f181813c310 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f181001af60 tx=0x7f1810019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f1815600640 1 -- 172.21.15.67:0/760261010 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100bd020 con 0x7f18180cf0c0 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f1815600640 1 -- 172.21.15.67:0/760261010 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18100b8670 con 0x7f18180cf0c0 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f1815600640 1 -- 172.21.15.67:0/760261010 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100b2040 con 0x7f18180cf0c0 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1818138c30 con 0x7f18180cf0c0 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18181391f0 con 0x7f18180cf0c0 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1818072d50 con 0x7f18180cf0c0 2024-04-17T10:29:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f1815600640 1 -- 172.21.15.67:0/760261010 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f18100c0020 con 0x7f18180cf0c0 2024-04-17T10:29:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f1815600640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17f4066f60 0x7f17f4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.249+0000 7f1815600640 1 -- 172.21.15.67:0/760261010 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f18101202a0 con 0x7f18180cf0c0 2024-04-17T10:29:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.253+0000 7f181d000640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17f4066f60 0x7f17f4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.253+0000 7f181d000640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17f4066f60 0x7f17f4069420 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1808000f70 tx=0x7f1808007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.253+0000 7f1815600640 1 -- 172.21.15.67:0/760261010 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f18100c7030 con 0x7f18180cf0c0 2024-04-17T10:29:58.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.453+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18180ceea0 con 0x7f18180cf0c0 2024-04-17T10:29:58.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.453+0000 7f1815600640 1 -- 172.21.15.67:0/760261010 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f1810016560 con 0x7f18180cf0c0 2024-04-17T10:29:58.447 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:29:58.447 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":55,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/3654887462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3654887462},{"type":"v1","addr":"172.21.15.86:6836","nonce":3654887462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6814,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1627300843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1627300843},{"type":"v1","addr":"172.21.15.67:6835","nonce":1627300843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/39011498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":39011498},{"type":"v1","addr":"172.21.15.86:6837","nonce":39011498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:29:47.987388+0000","modified":"2024-04-17T10:29:51.015058+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/3374829114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3374829114},{"type":"v1","addr":"172.21.15.67:6834","nonce":3374829114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-17T10:29:58.447 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 55 2024-04-17T10:29:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17f4066f60 msgr2=0x7f17f4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17f4066f60 0x7f17f4069420 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1808000f70 tx=0x7f1808007b60 comp rx=0 tx=0).stop 2024-04-17T10:29:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 msgr2=0x7f181813c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:58.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f181813c310 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f181001af60 tx=0x7f1810019650 comp rx=0 tx=0).stop 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 shutdown_connections 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f17f4066f60 0x7f17f4069420 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18180cf0c0 0x7f181813c310 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f181806eac0 0x7f18181f99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 --2- 172.21.15.67:0/760261010 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f181806e6e0 0x7f1818138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 >> 172.21.15.67:0/760261010 conn(0x7f181806ac90 msgr2=0x7f181806b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 shutdown_connections 2024-04-17T10:29:58.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:58.457+0000 7f181fa00640 1 -- 172.21.15.67:0/760261010 wait complete. 2024-04-17T10:29:58.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:29:47.987388+0000', 'modified': '2024-04-17T10:29:51.015058+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7206}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7206': {'gid': 7206, 'name': 'a', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6834/3374829114', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 3374829114}, {'type': 'v1', 'addr': '172.21.15.67:6834', 'nonce': 3374829114}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:29:58.871 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:29:58.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T10:29:59.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.021+0000 7f7ee4a00640 1 Processor -- start 2024-04-17T10:29:59.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.021+0000 7f7ee4a00640 1 -- start start 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.021+0000 7f7ee4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ee00ce020 0x7f7ee01620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.021+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00cf510 0x7f7ee00cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.021+0000 7f7ee4a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee015eb50 con 0x7f7ee00ce400 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7ee4a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee0161650 con 0x7f7ee00ce020 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee01625e0 con 0x7f7ee00cf510 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ee00ce020 0x7f7ee01620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7ee00ce020 0x7f7ee01620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57386/0 (socket says 172.21.15.67:57386) 2024-04-17T10:29:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 -- 172.21.15.67:0/1955935511 learned_addr learned my addr 172.21.15.67:0/1955935511 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:59.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 -- 172.21.15.67:0/1955935511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00cf510 msgr2=0x7f7ee00cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 --2- 172.21.15.67:0/1955935511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00cf510 0x7f7ee00cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 -- 172.21.15.67:0/1955935511 >> v1:172.21.15.86:6789/0 conn(0x7f7ee00ce400 legacy=0x7f7ee015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 -- 172.21.15.67:0/1955935511 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee015ee30 con 0x7f7ee00ce020 2024-04-17T10:29:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edde00640 1 --2- 172.21.15.67:0/1955935511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00ce020 0x7f7ee01620a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00098a0 tx=0x7f7ed001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a093c69a09efa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edca00640 1 -- 172.21.15.67:0/1955935511 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed00b1070 con 0x7f7ee00ce020 2024-04-17T10:29:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edca00640 1 -- 172.21.15.67:0/1955935511 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed00ac600 con 0x7f7ee00ce020 2024-04-17T10:29:59.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.025+0000 7f7edca00640 1 -- 172.21.15.67:0/1955935511 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed00ac9d0 con 0x7f7ee00ce020 2024-04-17T10:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1955935511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00ce020 msgr2=0x7f7ee01620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1955935511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00ce020 0x7f7ee01620a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00098a0 tx=0x7f7ed001c4d0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1955935511 shutdown_connections 2024-04-17T10:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1955935511 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00cf510 0x7f7ee00cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1955935511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00ce020 0x7f7ee01620a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1955935511 >> 172.21.15.67:0/1955935511 conn(0x7f7ee00b33f0 msgr2=0x7f7ee00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1955935511 shutdown_connections 2024-04-17T10:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1955935511 wait complete. 2024-04-17T10:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 Processor -- start 2024-04-17T10:29:59.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- start start 2024-04-17T10:29:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee00ce020 0x7f7ee01597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00ce400 0x7f7ee0154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00cf510 0x7f7ee0155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee006c0f0 con 0x7f7ee00cf510 2024-04-17T10:29:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee006bf70 con 0x7f7ee00ce020 2024-04-17T10:29:59.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ee4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee015e720 con 0x7f7ee00ce400 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00cf510 0x7f7ee0155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00cf510 0x7f7ee0155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57400/0 (socket says 172.21.15.67:57400) 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 -- 172.21.15.67:0/1192338968 learned_addr learned my addr 172.21.15.67:0/1192338968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7edd400640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00ce400 0x7f7ee0154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7edde00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee00ce020 0x7f7ee01597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 -- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00ce400 msgr2=0x7f7ee0154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00ce400 0x7f7ee0154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 -- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee00ce020 msgr2=0x7f7ee01597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee00ce020 0x7f7ee01597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.029+0000 7f7ede800640 1 -- 172.21.15.67:0/1192338968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee00b1df0 con 0x7f7ee00cf510 2024-04-17T10:29:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.033+0000 7f7ede800640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00cf510 0x7f7ee0155410 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800b970 tx=0x7f7ed800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.033+0000 7f7ed6000640 1 -- 172.21.15.67:0/1192338968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8019070 con 0x7f7ee00cf510 2024-04-17T10:29:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.033+0000 7f7ed6000640 1 -- 172.21.15.67:0/1192338968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed800cb60 con 0x7f7ee00cf510 2024-04-17T10:29:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.033+0000 7f7ed6000640 1 -- 172.21.15.67:0/1192338968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8014440 con 0x7f7ee00cf510 2024-04-17T10:29:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.033+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee00b20d0 con 0x7f7ee00cf510 2024-04-17T10:29:59.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.033+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee00b2690 con 0x7f7ee00cf510 2024-04-17T10:29:59.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.033+0000 7f7ed6000640 1 -- 172.21.15.67:0/1192338968 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7ed801c070 con 0x7f7ee00cf510 2024-04-17T10:29:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ed6000640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ec0066f90 0x7f7ec0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ed6000640 1 -- 172.21.15.67:0/1192338968 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ed8084d90 con 0x7f7ee00cf510 2024-04-17T10:29:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 Processor -- start 2024-04-17T10:29:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edde00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ec0066f90 0x7f7ec0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- start start 2024-04-17T10:29:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edde00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ec0066f90 0x7f7ec0069450 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00ae000 tx=0x7f7ed00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e820920 0x56264e820d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e7d69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e7d7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264e6be5f0 con 0x56264e821240 2024-04-17T10:29:59.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264e625da0 con 0x56264e7d6f30 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56264e5ae150 con 0x56264e820920 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e7d69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edde00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e820920 0x56264e820d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e7d69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57406/0 (socket says 172.21.15.67:57406) 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 -- 172.21.15.67:0/1092444477 learned_addr learned my addr 172.21.15.67:0/1092444477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ede800640 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e7d7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 -- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e820920 msgr2=0x56264e820d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e820920 0x56264e820d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 -- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 msgr2=0x56264e7d7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e7d7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 -- 172.21.15.67:0/1092444477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56264e5c7ca0 con 0x56264e821240 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edde00640 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e820920 0x56264e820d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7edd400640 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e7d69f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc001850 tx=0x7f7ecc007e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c87afe18e3979c5c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ecaa00640 1 -- 172.21.15.67:0/1092444477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc0052e0 con 0x56264e821240 2024-04-17T10:29:59.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ecaa00640 1 -- 172.21.15.67:0/1092444477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc018070 con 0x56264e821240 2024-04-17T10:29:59.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ecaa00640 1 -- 172.21.15.67:0/1092444477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc005480 con 0x56264e821240 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1092444477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 msgr2=0x56264e7d69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e7d69f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc001850 tx=0x7f7ecc007e20 comp rx=0 tx=0).stop 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1092444477 shutdown_connections 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e7d7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e7d69f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1092444477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e820920 0x56264e820d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1092444477 >> 172.21.15.67:0/1092444477 conn(0x56264e7dd2a0 msgr2=0x56264e7d52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1092444477 shutdown_connections 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.037+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1092444477 wait complete. 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 -- start start 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 Processor -- start 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e827dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e828310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e821240 0x56264e7daaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264e5e0990 con 0x56264e820920 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264e621e40 con 0x56264e7d6f30 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56264e5b1900 con 0x56264e821240 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 10 client.0 _ll_register_callbacks cb 0x56264e6d91c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edde00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e827dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e828310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edde00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e827dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:41186/0 (socket says 172.21.15.67:41186) 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ede800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e821240 0x56264e7daaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edde00640 1 -- 172.21.15.67:0/239440040 learned_addr learned my addr 172.21.15.67:0/239440040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edd400640 1 -- 172.21.15.67:0/239440040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e821240 msgr2=0x56264e7daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edd400640 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e821240 0x56264e7daaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edd400640 1 -- 172.21.15.67:0/239440040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 msgr2=0x56264e827dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edd400640 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e827dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edd400640 1 -- 172.21.15.67:0/239440040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56264e5de630 con 0x56264e820920 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edde00640 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e827dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edd400640 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e828310 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc017000 tx=0x7f7ecc000fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=515b77515282db0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ebf400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ebf400640 1 -- 172.21.15.67:0/239440040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc007030 con 0x56264e820920 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ebf400640 1 -- 172.21.15.67:0/239440040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ecc0071d0 con 0x56264e820920 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ecc018070 con 0x56264e820920 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ebf400640 1 -- 172.21.15.67:0/239440040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ecc02e050 con 0x56264e820920 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 10 client.7386 fetch_fsmap learned FSMap version 55 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x56264e5ae3a0 con 0x56264e820920 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ebf400640 1 -- 172.21.15.67:0/239440040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc018070 con 0x56264e820920 2024-04-17T10:29:59.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ebf400640 1 -- 172.21.15.67:0/239440040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 55) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7ecc031020 con 0x56264e820920 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 10 client.7386 fetch_fsmap finished waiting for FSMap version 55 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 10 client.7386 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 20 client.7386 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e7db4b0 0x56264e7db890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 4 client.7386 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56264e5c1a50 con 0x56264e7db4b0 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.041+0000 7f7edde00640 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e7db4b0 0x56264e7db890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7edde00640 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e7db4b0 0x56264e7db890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2378e4c51e0397a1 server_cookie=f5102d0b9dbdb555 in_seq=0 out_seq=0 2024-04-17T10:29:59.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ebf400640 10 client.7386 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:29:59.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ebf400640 1 -- 172.21.15.67:0/239440040 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56264e5c1a50 con 0x56264e7db4b0 2024-04-17T10:29:59.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ebf400640 10 client.7386 handle_command_reply: tid=0 2024-04-17T10:29:59.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 client.7386 shutdown 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 msgr2=0x56264e828310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e828310 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc017000 tx=0x7f7ecc000fc0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 shutdown_connections 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e7db4b0 0x56264e7db890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e821240 0x56264e7daaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e828310 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ebf400640 0 client.7386 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/239440040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e7d6f30 0x56264e827dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 >> 172.21.15.67:0/239440040 conn(0x56264e7dd2a0 msgr2=0x56264e7d59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 shutdown_connections 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 1 -- 172.21.15.67:0/239440040 wait complete. 2024-04-17T10:29:59.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.045+0000 7f7ee7df6000 20 client.7386 trim_cache size 0 max 16384 2024-04-17T10:29:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 Processor -- start 2024-04-17T10:29:59.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 -- start start 2024-04-17T10:29:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e87c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e820920 0x56264e87c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e87cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee0202790 con 0x56264e821240 2024-04-17T10:29:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee015cac0 con 0x56264e820920 2024-04-17T10:29:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee015d0c0 con 0x56264e7d6f30 2024-04-17T10:29:59.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e87cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e87cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57432/0 (socket says 172.21.15.67:57432) 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 -- 172.21.15.67:0/1605721750 learned_addr learned my addr 172.21.15.67:0/1605721750 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7edde00640 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e87c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7edd400640 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e820920 0x56264e87c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 -- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 msgr2=0x56264e87c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e87c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 -- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e820920 msgr2=0x56264e87c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e820920 0x56264e87c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 -- 172.21.15.67:0/1605721750 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56264e6a7600 con 0x56264e821240 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7edd400640 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e820920 0x56264e87c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7ede800640 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e87cba0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800b940 tx=0x7f7ed8083090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fa58d72dd60404 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7eca000640 1 -- 172.21.15.67:0/1605721750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8081070 con 0x56264e821240 2024-04-17T10:29:59.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7eca000640 1 -- 172.21.15.67:0/1605721750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed8051cc0 con 0x56264e821240 2024-04-17T10:29:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.061+0000 7f7eca000640 1 -- 172.21.15.67:0/1605721750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed805da40 con 0x56264e821240 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1605721750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 msgr2=0x56264e87cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e87cba0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800b940 tx=0x7f7ed8083090 comp rx=0 tx=0).stop 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1605721750 shutdown_connections 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e821240 0x56264e87cba0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e820920 0x56264e87c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1605721750 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e87c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1605721750 >> 172.21.15.67:0/1605721750 conn(0x56264e7dd2a0 msgr2=0x56264e7da220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1605721750 shutdown_connections 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1605721750 wait complete. 2024-04-17T10:29:59.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- start start 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 Processor -- start 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e82d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e82d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e821240 0x56264e87d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264e62a3d0 con 0x56264e820920 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56264e6e1d50 con 0x56264e821240 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56264e5c0980 con 0x56264e7d6f30 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 10 client.0 _ll_register_callbacks cb 0x7f7ecc001270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e82d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e82d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57438/0 (socket says 172.21.15.67:57438) 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 -- 172.21.15.67:0/1044312686 learned_addr learned my addr 172.21.15.67:0/1044312686 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ede800640 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e821240 0x56264e87d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edde00640 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e82d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 -- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 msgr2=0x56264e82d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e82d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 -- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e821240 msgr2=0x56264e87d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e821240 0x56264e87d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 -- 172.21.15.67:0/1044312686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56264e5ba9b0 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7edd400640 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e82d820 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc007e80 tx=0x7f7ecc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df0f689c5cffc808 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ebf400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ebf400640 1 -- 172.21.15.67:0/1044312686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc037060 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ebf400640 1 -- 172.21.15.67:0/1044312686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7ecc026410 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ed8051cc0 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ebf400640 1 -- 172.21.15.67:0/1044312686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ecc03d050 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 10 client.7392 fetch_fsmap learned FSMap version 55 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x56264e5cd0b0 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ebf400640 1 -- 172.21.15.67:0/1044312686 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed8051cc0 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ebf400640 1 -- 172.21.15.67:0/1044312686 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 55) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7ecc040020 con 0x56264e820920 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 10 client.7392 fetch_fsmap finished waiting for FSMap version 55 2024-04-17T10:29:59.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 10 client.7392 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 20 client.7392 populate_metadata read hostname 'smithi067' 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e87d800 0x56264e87dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 4 client.7392 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.065+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56264e6332e0 con 0x56264e87d800 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7edde00640 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e87d800 0x56264e87dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7edde00640 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e87d800 0x56264e87dbe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5abaf01de31641e server_cookie=a93ed11611296d27 in_seq=0 out_seq=0 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ebf400640 10 client.7392 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ebf400640 1 -- 172.21.15.67:0/1044312686 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x56264e6332e0 con 0x56264e87d800 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ebf400640 10 client.7392 handle_command_reply: tid=0 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 client.7392 shutdown 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 msgr2=0x56264e82d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e82d820 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc007e80 tx=0x7f7ecc009040 comp rx=0 tx=0).stop 2024-04-17T10:29:59.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 shutdown_connections 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56264e821240 0x56264e87d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56264e820920 0x56264e82d820 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56264e87d800 0x56264e87dbe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 --2- 172.21.15.67:0/1044312686 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56264e7d6f30 0x56264e82d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ebf400640 0 client.7392 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 >> 172.21.15.67:0/1044312686 conn(0x56264e7dd2a0 msgr2=0x56264e828990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 shutdown_connections 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 1 -- 172.21.15.67:0/1044312686 wait complete. 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee7df6000 20 client.7392 trim_cache size 0 max 16384 2024-04-17T10:29:59.066 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 7206, 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.451869637, 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 55, 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 83, 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 83, 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 15.331313665 2024-04-17T10:29:59.067 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:29:59.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ec0066f90 msgr2=0x7f7ec0069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ec0066f90 0x7f7ec0069450 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f7ed00ae000 tx=0x7f7ed00a7000 comp rx=0 tx=0).stop 2024-04-17T10:29:59.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00cf510 msgr2=0x7f7ee0155410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00cf510 0x7f7ee0155410 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7ed800b970 tx=0x7f7ed800be30 comp rx=0 tx=0).stop 2024-04-17T10:29:59.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 shutdown_connections 2024-04-17T10:29:59.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7ec0066f90 0x7f7ec0069450 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ee00cf510 0x7f7ee0155410 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ee00ce400 0x7f7ee0154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 --2- 172.21.15.67:0/1192338968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ee00ce020 0x7f7ee01597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 >> 172.21.15.67:0/1192338968 conn(0x7f7ee00b33f0 msgr2=0x7f7ee00738f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 shutdown_connections 2024-04-17T10:29:59.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.069+0000 7f7ee4a00640 1 -- 172.21.15.67:0/1192338968 wait complete. 2024-04-17T10:29:59.083 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:29:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa580c00640 1 Processor -- start 2024-04-17T10:29:59.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa580c00640 1 -- start start 2024-04-17T10:29:59.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa580c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa57c071770 0x7fa57c0cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa580c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57c06e2f0 con 0x7fa57c071310 2024-04-17T10:29:59.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa580c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57c06d6e0 con 0x7fa57c145040 2024-04-17T10:29:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa580c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa57c0a86c0 con 0x7fa57c071770 2024-04-17T10:29:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa57a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa57c071770 0x7fa57c0cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa571400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa57c071310 0x7fa57c0cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:50966/0 (socket says 172.21.15.67:50966) 2024-04-17T10:29:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa571400640 1 -- 172.21.15.67:0/3556849034 learned_addr learned my addr 172.21.15.67:0/3556849034 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:29:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa57a800640 1 -- 172.21.15.67:0/3556849034 >> v1:172.21.15.86:6789/0 conn(0x7fa57c145040 legacy=0x7fa57c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa57a800640 1 -- 172.21.15.67:0/3556849034 >> v1:172.21.15.67:6789/0 conn(0x7fa57c071310 legacy=0x7fa57c0cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.229+0000 7fa57a800640 1 -- 172.21.15.67:0/3556849034 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c06d9c0 con 0x7fa57c071770 2024-04-17T10:29:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa57a800640 1 --2- 172.21.15.67:0/3556849034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c0cca10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fa574009d90 tx=0x7fa57401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e856e6cfa25f68 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa579400640 1 -- 172.21.15.67:0/3556849034 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5740b4070 con 0x7fa57c071770 2024-04-17T10:29:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa579400640 1 -- 172.21.15.67:0/3556849034 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5740af600 con 0x7fa57c071770 2024-04-17T10:29:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa579400640 1 -- 172.21.15.67:0/3556849034 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5740af9d0 con 0x7fa57c071770 2024-04-17T10:29:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa580c00640 1 -- 172.21.15.67:0/3556849034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 msgr2=0x7fa57c0cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa580c00640 1 --2- 172.21.15.67:0/3556849034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c0cca10 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fa574009d90 tx=0x7fa57401f150 comp rx=0 tx=0).stop 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa580c00640 1 -- 172.21.15.67:0/3556849034 shutdown_connections 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa580c00640 1 --2- 172.21.15.67:0/3556849034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c0cca10 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa580c00640 1 -- 172.21.15.67:0/3556849034 >> 172.21.15.67:0/3556849034 conn(0x7fa57c06ac90 msgr2=0x7fa57c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa580c00640 1 -- 172.21.15.67:0/3556849034 shutdown_connections 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.233+0000 7fa580c00640 1 -- 172.21.15.67:0/3556849034 wait complete. 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 Processor -- start 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 -- start start 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa57c071310 0x7fa57c15a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c139440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa57c145040 0x7fa57c139980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c071b80 con 0x7fa57c071770 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c06df80 con 0x7fa57c071310 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c06d370 con 0x7fa57c145040 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c139440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c139440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57466/0 (socket says 172.21.15.67:57466) 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 -- 172.21.15.67:0/2769867623 learned_addr learned my addr 172.21.15.67:0/2769867623 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa57a800640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa57c145040 0x7fa57c139980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 -- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa57c145040 msgr2=0x7fa57c139980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa57c145040 0x7fa57c139980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa579e00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa57c071310 0x7fa57c15a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 -- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa57c071310 msgr2=0x7fa57c15a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa57c071310 0x7fa57c15a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 -- 172.21.15.67:0/2769867623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c137da0 con 0x7fa57c071770 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa579e00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa57c071310 0x7fa57c15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa571400640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c139440 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c530 tx=0x7fa56400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa572a00640 1 -- 172.21.15.67:0/2769867623 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564017070 con 0x7fa57c071770 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa572a00640 1 -- 172.21.15.67:0/2769867623 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa564005cf0 con 0x7fa57c071770 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa572a00640 1 -- 172.21.15.67:0/2769867623 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564007270 con 0x7fa57c071770 2024-04-17T10:29:59.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57c15af00 con 0x7fa57c071770 2024-04-17T10:29:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.237+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57c137500 con 0x7fa57c071770 2024-04-17T10:29:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.241+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57c072d50 con 0x7fa57c071770 2024-04-17T10:29:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.241+0000 7fa572a00640 1 -- 172.21.15.67:0/2769867623 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa56401a070 con 0x7fa57c071770 2024-04-17T10:29:59.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.241+0000 7fa572a00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f60 0x7fa558069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:29:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.241+0000 7fa572a00640 1 -- 172.21.15.67:0/2769867623 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa5640843e0 con 0x7fa57c071770 2024-04-17T10:29:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.241+0000 7fa579e00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f60 0x7fa558069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:29:59.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.241+0000 7fa579e00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f60 0x7fa558069420 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa56c001650 tx=0x7fa56c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:29:59.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.245+0000 7fa572a00640 1 -- 172.21.15.67:0/2769867623 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa564054c00 con 0x7fa57c071770 2024-04-17T10:29:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.441+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa57c0ce380 con 0x7fa57c071770 2024-04-17T10:29:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.441+0000 7fa572a00640 1 -- 172.21.15.67:0/2769867623 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa564058ca0 con 0x7fa57c071770 2024-04-17T10:29:59.436 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:29:59.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f60 msgr2=0x7fa558069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f60 0x7fa558069420 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fa56c001650 tx=0x7fa56c009040 comp rx=0 tx=0).stop 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 msgr2=0x7fa57c139440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c139440 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fa56400c530 tx=0x7fa56400c9f0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 shutdown_connections 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f60 0x7fa558069420 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa57c145040 0x7fa57c139980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa57c071770 0x7fa57c139440 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 --2- 172.21.15.67:0/2769867623 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa57c071310 0x7fa57c15a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 >> 172.21.15.67:0/2769867623 conn(0x7fa57c06ac90 msgr2=0x7fa57c06b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 shutdown_connections 2024-04-17T10:29:59.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:29:59.445+0000 7fa580c00640 1 -- 172.21.15.67:0/2769867623 wait complete. 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:29:59.458 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:59.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.459 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:29:59.468 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42147sec preferred_lft 42147sec 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:29:59.469 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:29:59.470 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:29:59.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.471 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:29:59.531 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:29:59.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.533 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:59.592 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:59.593 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:29:59.593 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:29:59.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.593 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:29:59.593 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:29:59.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:29:59.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:29:59.593 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:29:59.593 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:29:59.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.683 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:29:59.692 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:29:59.692 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:29:59.692 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:59.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.693 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:29:59.760 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:29:59.760 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:59.812 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:29:59.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:29:59.813 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:59.864 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:29:59.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:59.958 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:59.958 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:29:59.971 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:29:59.971 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:29:59.971 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:29:59.972 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:29:59.972 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:29:59.975 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:29:59.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:29:59.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:29:59.996 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:30:00.061 INFO:teuthology.orchestra.run.smithi067.stdout:client6347 2024-04-17T10:30:00.061 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:30:00.062 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:30:00.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2024-04-17T10:30:00.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd718a00640 1 Processor -- start 2024-04-17T10:30:00.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd718a00640 1 -- start start 2024-04-17T10:30:00.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd718a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd71406f320 0x7fd7140b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd718a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd7140c9c70 0x7fd71406b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd718a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd71406ee40 con 0x7fd71415db10 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd718a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7140cf490 con 0x7fd71406f320 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd718a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd71406ba10 con 0x7fd7140c9c70 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd70be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd71406f320 0x7fd7140b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd70be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd71406f320 0x7fd7140b2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46166/0 (socket says 172.21.15.67:46166) 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd70be00640 1 -- 172.21.15.67:0/2026550213 learned_addr learned my addr 172.21.15.67:0/2026550213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:00.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd713e00640 1 --2- 172.21.15.67:0/2026550213 >> v2:172.21.15.86:3300/0 conn(0x7fd7140c9c70 0x7fd71406b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:00.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd70be00640 1 -- 172.21.15.67:0/2026550213 >> v2:172.21.15.86:3300/0 conn(0x7fd7140c9c70 msgr2=0x7fd71406b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:00.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd70be00640 1 --2- 172.21.15.67:0/2026550213 >> v2:172.21.15.86:3300/0 conn(0x7fd7140c9c70 0x7fd71406b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd70be00640 1 -- 172.21.15.67:0/2026550213 >> v1:172.21.15.86:6789/0 conn(0x7fd71415db10 legacy=0x7fd71406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:00.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.313+0000 7fd70be00640 1 -- 172.21.15.67:0/2026550213 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71406e6d0 con 0x7fd71406f320 2024-04-17T10:30:00.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd70be00640 1 --2- 172.21.15.67:0/2026550213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71406f320 0x7fd7140b2dc0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd71406e2b0 tx=0x7fd6fc01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97546957f08a952 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:00.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd712a00640 1 -- 172.21.15.67:0/2026550213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0b1070 con 0x7fd71406f320 2024-04-17T10:30:00.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd712a00640 1 -- 172.21.15.67:0/2026550213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6fc0ac600 con 0x7fd71406f320 2024-04-17T10:30:00.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd712a00640 1 -- 172.21.15.67:0/2026550213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6fc0ac9d0 con 0x7fd71406f320 2024-04-17T10:30:00.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 -- 172.21.15.67:0/2026550213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71406f320 msgr2=0x7fd7140b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:00.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 --2- 172.21.15.67:0/2026550213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71406f320 0x7fd7140b2dc0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fd71406e2b0 tx=0x7fd6fc01c490 comp rx=0 tx=0).stop 2024-04-17T10:30:00.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 -- 172.21.15.67:0/2026550213 shutdown_connections 2024-04-17T10:30:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 --2- 172.21.15.67:0/2026550213 >> v2:172.21.15.86:3300/0 conn(0x7fd7140c9c70 0x7fd71406b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 --2- 172.21.15.67:0/2026550213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71406f320 0x7fd7140b2dc0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 -- 172.21.15.67:0/2026550213 >> 172.21.15.67:0/2026550213 conn(0x7fd7140b22a0 msgr2=0x7fd7140b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 -- 172.21.15.67:0/2026550213 shutdown_connections 2024-04-17T10:30:00.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.317+0000 7fd718a00640 1 -- 172.21.15.67:0/2026550213 wait complete. 2024-04-17T10:30:00.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 Processor -- start 2024-04-17T10:30:00.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 -- start start 2024-04-17T10:30:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71406f320 0x7fd7140b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7140c9c70 0x7fd714151a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:00.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71415db10 0x7fd71414b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7140cf0a0 con 0x7fd71415db10 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7140cef20 con 0x7fd71406f320 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7140cf220 con 0x7fd7140c9c70 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71415db10 0x7fd71414b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71406f320 0x7fd7140b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71415db10 0x7fd71414b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46172/0 (socket says 172.21.15.67:46172) 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 -- 172.21.15.67:0/2945192764 learned_addr learned my addr 172.21.15.67:0/2945192764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd70be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7140c9c70 0x7fd714151a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 -- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7140c9c70 msgr2=0x7fd714151a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7140c9c70 0x7fd714151a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 -- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71406f320 msgr2=0x7fd7140b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71406f320 0x7fd7140b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 -- 172.21.15.67:0/2945192764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7141f9e90 con 0x7fd71415db10 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd713e00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71415db10 0x7fd71414b390 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd70c009d30 tx=0x7fd70c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd70b400640 1 -- 172.21.15.67:0/2945192764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c018020 con 0x7fd71415db10 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd70b400640 1 -- 172.21.15.67:0/2945192764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd70c014480 con 0x7fd71415db10 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7141fa150 con 0x7fd71415db10 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd70b400640 1 -- 172.21.15.67:0/2945192764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70c014870 con 0x7fd71415db10 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7141fa710 con 0x7fd71415db10 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.321+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd714074dc0 con 0x7fd71415db10 2024-04-17T10:30:00.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.325+0000 7fd70b400640 1 -- 172.21.15.67:0/2945192764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd70c01b020 con 0x7fd71415db10 2024-04-17T10:30:00.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.325+0000 7fd70b400640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6f0066f60 0x7fd6f0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:00.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.325+0000 7fd70b400640 1 -- 172.21.15.67:0/2945192764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd70c085b30 con 0x7fd71415db10 2024-04-17T10:30:00.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.325+0000 7fd713400640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6f0066f60 0x7fd6f0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:00.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.325+0000 7fd713400640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6f0066f60 0x7fd6f0069420 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd704001770 tx=0x7fd704009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:00.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.329+0000 7fd70b400640 1 -- 172.21.15.67:0/2945192764 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd70c022070 con 0x7fd71415db10 2024-04-17T10:30:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.529+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd7140684a0 con 0x7fd71415db10 2024-04-17T10:30:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.653+0000 7fd70b400640 1 -- 172.21.15.67:0/2945192764 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v56) v1 ==== 167+0+0 (secure 0 0 0) 0x7fd70c010040 con 0x7fd71415db10 2024-04-17T10:30:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:enabled new snapshots 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6f0066f60 msgr2=0x7fd6f0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6f0066f60 0x7fd6f0069420 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd704001770 tx=0x7fd704009040 comp rx=0 tx=0).stop 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71415db10 msgr2=0x7fd71414b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71415db10 0x7fd71414b390 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd70c009d30 tx=0x7fd70c00c490 comp rx=0 tx=0).stop 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 shutdown_connections 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6f0066f60 0x7fd6f0069420 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd71415db10 0x7fd71414b390 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd7140c9c70 0x7fd714151a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 --2- 172.21.15.67:0/2945192764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd71406f320 0x7fd7140b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 >> 172.21.15.67:0/2945192764 conn(0x7fd7140b22a0 msgr2=0x7fd7140ca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 shutdown_connections 2024-04-17T10:30:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:00.657+0000 7fd718a00640 1 -- 172.21.15.67:0/2945192764 wait complete. 2024-04-17T10:30:00.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:30:00.672 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir stray_eval_dir) 2024-04-17T10:30:00.688 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:30:00.688 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:30:00.737 DEBUG:teuthology.orchestra.run.smithi067:> stdin-killer -- python3 -c ' 2024-04-17T10:30:00.737 DEBUG:teuthology.orchestra.run.smithi067:> import os 2024-04-17T10:30:00.737 DEBUG:teuthology.orchestra.run.smithi067:> import uuid 2024-04-17T10:30:00.737 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:30:00.737 DEBUG:teuthology.orchestra.run.smithi067:> n = 2000 2024-04-17T10:30:00.737 DEBUG:teuthology.orchestra.run.smithi067:> create_hard_links = False 2024-04-17T10:30:00.737 DEBUG:teuthology.orchestra.run.smithi067:> if 3 > 0: 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> create_hard_links = True 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> path = "/home/ubuntu/cephtest/mnt.0/stray_eval_dir/file" 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> dpath = os.path.dirname(path) 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> fnameprefix = os.path.basename(path) 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> os.makedirs(dpath, exist_ok=True) 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> try: 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> for i in range(n): 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> with open(fpath, '"'"'w'"'"') as f: 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> f.write(f"{i}") 2024-04-17T10:30:00.738 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> f.flush() 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> os.fsync(f.fileno()) 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> os.unlink(fpath) 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> os.fsync(dirfd) 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> if create_hard_links: 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> for j in range(3): 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> if False: 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> os.fsync(dirfd) 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> finally: 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> os.close(dirfd) 2024-04-17T10:30:00.739 DEBUG:teuthology.orchestra.run.smithi067:> ' 2024-04-17T10:30:22.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-17T10:30:22.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:30:22.389 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-04-17T10:30:22.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:30:22.457 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec find /home/ubuntu/cephtest/mnt.0/stray_eval_dir/ -type f -delete) 2024-04-17T10:30:25.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:30:25.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:30:48.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:30:48.862 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-04-17T10:30:48.910 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:30:48.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 perf dump 2024-04-17T10:30:49.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.104+0000 7f9d57000640 1 Processor -- start 2024-04-17T10:30:49.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.104+0000 7f9d57000640 1 -- start start 2024-04-17T10:30:49.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d57000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d5015d970 0x7f9d50161df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d57000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9d5015dd50 0x7f9d5006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d57000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d500722d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d57000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d50162330 con 0x7f9d5015d970 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d57000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d5006b620 con 0x7f9d5015dd50 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d57000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5015ede0 con 0x7f9d5006b1d0 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d5015d970 0x7f9d50161df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d5015d970 0x7f9d50161df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53490/0 (socket says 172.21.15.67:53490) 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 -- 172.21.15.67:0/2579437085 learned_addr learned my addr 172.21.15.67:0/2579437085 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d56600640 1 --2- 172.21.15.67:0/2579437085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d500722d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55200640 1 --2- 172.21.15.67:0/2579437085 >> v2:172.21.15.86:3300/0 conn(0x7f9d5015dd50 0x7f9d5006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 -- 172.21.15.67:0/2579437085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 msgr2=0x7f9d500722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 --2- 172.21.15.67:0/2579437085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d500722d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 -- 172.21.15.67:0/2579437085 >> v2:172.21.15.86:3300/0 conn(0x7f9d5015dd50 msgr2=0x7f9d5006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 --2- 172.21.15.67:0/2579437085 >> v2:172.21.15.86:3300/0 conn(0x7f9d5015dd50 0x7f9d5006ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55c00640 1 -- 172.21.15.67:0/2579437085 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5015e1d0 con 0x7f9d5015d970 2024-04-17T10:30:49.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.108+0000 7f9d55200640 1 --2- 172.21.15.67:0/2579437085 >> v2:172.21.15.86:3300/0 conn(0x7f9d5015dd50 0x7f9d5006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:30:49.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d55c00640 1 --2- 172.21.15.67:0/2579437085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d50161df0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f9d50073440 tx=0x7f9d4401c230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6526d8939e0fc523 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d4fe00640 1 -- 172.21.15.67:0/2579437085 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440b0070 con 0x7f9d5015d970 2024-04-17T10:30:49.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d4fe00640 1 -- 172.21.15.67:0/2579437085 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d440ab600 con 0x7f9d5015d970 2024-04-17T10:30:49.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d4fe00640 1 -- 172.21.15.67:0/2579437085 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440ab9d0 con 0x7f9d5015d970 2024-04-17T10:30:49.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- 172.21.15.67:0/2579437085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 msgr2=0x7f9d50161df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 --2- 172.21.15.67:0/2579437085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d50161df0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f9d50073440 tx=0x7f9d4401c230 comp rx=0 tx=0).stop 2024-04-17T10:30:49.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- 172.21.15.67:0/2579437085 shutdown_connections 2024-04-17T10:30:49.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 --2- 172.21.15.67:0/2579437085 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d500722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 --2- 172.21.15.67:0/2579437085 >> v2:172.21.15.86:3300/0 conn(0x7f9d5015dd50 0x7f9d5006ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 --2- 172.21.15.67:0/2579437085 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d50161df0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- 172.21.15.67:0/2579437085 >> 172.21.15.67:0/2579437085 conn(0x7f9d500b2310 msgr2=0x7f9d500b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- 172.21.15.67:0/2579437085 shutdown_connections 2024-04-17T10:30:49.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- 172.21.15.67:0/2579437085 wait complete. 2024-04-17T10:30:49.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 Processor -- start 2024-04-17T10:30:49.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- start start 2024-04-17T10:30:49.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d50155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d5013ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d5015dd50 0x7f9d50156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d500709a0 con 0x7f9d5015d970 2024-04-17T10:30:49.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50070820 con 0x7f9d5015dd50 2024-04-17T10:30:49.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.112+0000 7f9d57000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d500ce4d0 con 0x7f9d5006b1d0 2024-04-17T10:30:49.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d5013ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d50155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d56600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d5015dd50 0x7f9d50156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d50155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:56200/0 (socket says 172.21.15.67:56200) 2024-04-17T10:30:49.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d5013ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53504/0 (socket says 172.21.15.67:53504) 2024-04-17T10:30:49.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 -- 172.21.15.67:0/1296129131 learned_addr learned my addr 172.21.15.67:0/1296129131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 -- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d5015dd50 msgr2=0x7f9d50156940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d5015dd50 0x7f9d50156940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 -- 172.21.15.67:0/1296129131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 msgr2=0x7f9d5013ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d5013ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 -- 172.21.15.67:0/1296129131 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d50158aa0 con 0x7f9d5006b1d0 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d56600640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d5015dd50 0x7f9d50156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d50155fb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d440ae040 tx=0x7f9d4401aa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d4d600640 1 -- 172.21.15.67:0/1296129131 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440b0040 con 0x7f9d5006b1d0 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d4d600640 1 -- 172.21.15.67:0/1296129131 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d440bc070 con 0x7f9d5006b1d0 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d4d600640 1 -- 172.21.15.67:0/1296129131 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440b6680 con 0x7f9d5006b1d0 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5014da50 con 0x7f9d5006b1d0 2024-04-17T10:30:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5014e010 con 0x7f9d5006b1d0 2024-04-17T10:30:49.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.116+0000 7f9d4d600640 1 -- 172.21.15.67:0/1296129131 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9d440b6820 con 0x7f9d5006b1d0 2024-04-17T10:30:49.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d4d600640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d30066f60 0x7f9d30069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d4d600640 1 -- 172.21.15.67:0/1296129131 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9d4411f3f0 con 0x7f9d5006b1d0 2024-04-17T10:30:49.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 Processor -- start 2024-04-17T10:30:49.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55200640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d30066f60 0x7f9d30069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- start start 2024-04-17T10:30:49.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea69e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea24850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55200640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d30066f60 0x7f9d30069420 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9d501f9c50 tx=0x7f9d3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea25170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558de9071f0 con 0x5558dea69a40 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558de86de90 con 0x5558dea24d90 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558de7f5f50 con 0x5558dea6a360 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea69e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea69e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53520/0 (socket says 172.21.15.67:53520) 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 -- 172.21.15.67:0/1164174916 learned_addr learned my addr 172.21.15.67:0/1164174916 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55200640 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea24850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d56600640 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea25170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 -- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 msgr2=0x5558dea24850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea24850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 -- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 msgr2=0x5558dea25170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea25170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 -- 172.21.15.67:0/1164174916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558de8458b0 con 0x5558dea69a40 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55200640 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea24850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea69e20 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f9d4401a8e0 tx=0x7f9d44009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a067fdad9506fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d42000640 1 -- 172.21.15.67:0/1164174916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440f82a0 con 0x5558dea69a40 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d42000640 1 -- 172.21.15.67:0/1164174916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d440edf40 con 0x5558dea69a40 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d42000640 1 -- 172.21.15.67:0/1164174916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440ec2b0 con 0x5558dea69a40 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1164174916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 msgr2=0x5558dea69e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea69e20 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f9d4401a8e0 tx=0x7f9d44009240 comp rx=0 tx=0).stop 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1164174916 shutdown_connections 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea25170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea24850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1164174916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea69e20 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1164174916 >> 172.21.15.67:0/1164174916 conn(0x5558dea20d40 msgr2=0x5558dea23130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1164174916 shutdown_connections 2024-04-17T10:30:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.120+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1164174916 wait complete. 2024-04-17T10:30:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 -- start start 2024-04-17T10:30:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 Processor -- start 2024-04-17T10:30:49.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea24d90 0x5558dea2a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea2a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea6a360 0x5558dea2ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558de828540 con 0x5558dea69a40 2024-04-17T10:30:49.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558de869f30 con 0x5558dea6a360 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558de86de90 con 0x5558dea24d90 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 10 client.0 _ll_register_callbacks cb 0x5558de922180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea2a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea2a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53526/0 (socket says 172.21.15.67:53526) 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 -- 172.21.15.67:0/1969416962 learned_addr learned my addr 172.21.15.67:0/1969416962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d56600640 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea6a360 0x5558dea2ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea24d90 0x5558dea2a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 -- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea24d90 msgr2=0x5558dea2a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea24d90 0x5558dea2a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 -- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea6a360 msgr2=0x5558dea2ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea6a360 0x5558dea2ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 -- 172.21.15.67:0/1969416962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558de7f61a0 con 0x5558dea69a40 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55200640 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea2a930 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c016790 tx=0x7f9d3c004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336f5881d87ad3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d36a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:30:49.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d36a00640 1 -- 172.21.15.67:0/1969416962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c026020 con 0x5558dea69a40 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d36a00640 1 -- 172.21.15.67:0/1969416962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9d3c0149b0 con 0x5558dea69a40 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d440edf40 con 0x5558dea69a40 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d36a00640 1 -- 172.21.15.67:0/1969416962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d3c01c070 con 0x5558dea69a40 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 10 client.7419 fetch_fsmap learned FSMap version 56 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x5558de8f0840 con 0x5558dea69a40 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d36a00640 1 -- 172.21.15.67:0/1969416962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d440edf40 con 0x5558dea69a40 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d36a00640 1 -- 172.21.15.67:0/1969416962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9d3c033080 con 0x5558dea69a40 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 10 client.7419 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 10 client.7419 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 20 client.7419 populate_metadata read hostname 'smithi067' 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea78ad0 0x5558dea7aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 4 client.7419 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558de809be0 con 0x5558dea78ad0 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea78ad0 0x5558dea7aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea78ad0 0x5558dea7aec0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14e80a5dc8617045 server_cookie=9b65ea09923aee30 in_seq=0 out_seq=0 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.124+0000 7f9d36a00640 10 client.7419 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d36a00640 1 -- 172.21.15.67:0/1969416962 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5558de809be0 con 0x5558dea78ad0 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d36a00640 10 client.7419 handle_command_reply: tid=0 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 client.7419 shutdown 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 msgr2=0x5558dea2a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea2a930 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c016790 tx=0x7f9d3c004740 comp rx=0 tx=0).stop 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 shutdown_connections 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea6a360 0x5558dea2ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea2a930 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea78ad0 0x5558dea7aec0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1969416962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea24d90 0x5558dea2a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 >> 172.21.15.67:0/1969416962 conn(0x5558dea20d40 msgr2=0x5558dea239c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d36a00640 0 client.7419 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 shutdown_connections 2024-04-17T10:30:49.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1969416962 wait complete. 2024-04-17T10:30:49.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.128+0000 7f9d5bae9000 20 client.7419 trim_cache size 0 max 16384 2024-04-17T10:30:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.140+0000 7f9d5bae9000 1 Processor -- start 2024-04-17T10:30:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.140+0000 7f9d5bae9000 1 -- start start 2024-04-17T10:30:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea24140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea69a40 0x5558dea24680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea6a360 0x5558dea7aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d502030d0 con 0x5558dea6a360 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50202270 con 0x5558dea24d90 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50159be0 con 0x5558dea69a40 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea6a360 0x5558dea7aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea6a360 0x5558dea7aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53532/0 (socket says 172.21.15.67:53532) 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 -- 172.21.15.67:0/613115866 learned_addr learned my addr 172.21.15.67:0/613115866 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d55c00640 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea24140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d55200640 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea69a40 0x5558dea24680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 -- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea69a40 msgr2=0x5558dea24680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea69a40 0x5558dea24680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 -- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 msgr2=0x5558dea24140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea24140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 -- 172.21.15.67:0/613115866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558de8c7a10 con 0x5558dea6a360 2024-04-17T10:30:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d55c00640 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea24140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:30:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d55200640 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea69a40 0x5558dea24680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:30:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d56600640 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea6a360 0x5558dea7aec0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f9d48004040 tx=0x7f9d4800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bc9bdcac48315e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d41600640 1 -- 172.21.15.67:0/613115866 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4800e8f0 con 0x5558dea6a360 2024-04-17T10:30:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d41600640 1 -- 172.21.15.67:0/613115866 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4800ea90 con 0x5558dea6a360 2024-04-17T10:30:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d41600640 1 -- 172.21.15.67:0/613115866 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4800ee60 con 0x5558dea6a360 2024-04-17T10:30:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- 172.21.15.67:0/613115866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea6a360 msgr2=0x5558dea7aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea6a360 0x5558dea7aec0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f9d48004040 tx=0x7f9d4800b040 comp rx=0 tx=0).stop 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- 172.21.15.67:0/613115866 shutdown_connections 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea6a360 0x5558dea7aec0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea69a40 0x5558dea24680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/613115866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558dea24140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- 172.21.15.67:0/613115866 >> 172.21.15.67:0/613115866 conn(0x5558dea20d40 msgr2=0x5558dea68680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- 172.21.15.67:0/613115866 shutdown_connections 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- 172.21.15.67:0/613115866 wait complete. 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- start start 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 Processor -- start 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558deac8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea84940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea84e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558de86de90 con 0x5558dea69a40 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5015a1e0 con 0x5558dea24d90 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.144+0000 7f9d5bae9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5558de86d1a0 con 0x5558dea6a360 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 10 client.0 _ll_register_callbacks cb 0x5558de830a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea84940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea84940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53536/0 (socket says 172.21.15.67:53536) 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 -- 172.21.15.67:0/1489903134 learned_addr learned my addr 172.21.15.67:0/1489903134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d56600640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea84e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558deac8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 -- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 msgr2=0x5558dea84e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea84e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 -- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 msgr2=0x5558deac8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558deac8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 -- 172.21.15.67:0/1489903134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558de802b00 con 0x5558dea69a40 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558deac8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:30:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55200640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea84940 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c010fc0 tx=0x7f9d3c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b1d0d3a5445587 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:30:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 1 -- 172.21.15.67:0/1489903134 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c026040 con 0x5558dea69a40 2024-04-17T10:30:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 1 -- 172.21.15.67:0/1489903134 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9d3c014800 con 0x5558dea69a40 2024-04-17T10:30:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d4800ea90 con 0x5558dea69a40 2024-04-17T10:30:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 1 -- 172.21.15.67:0/1489903134 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d3c042050 con 0x5558dea69a40 2024-04-17T10:30:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 10 client.7425 fetch_fsmap learned FSMap version 56 2024-04-17T10:30:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x5558de814f80 con 0x5558dea69a40 2024-04-17T10:30:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 1 -- 172.21.15.67:0/1489903134 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4800ea90 con 0x5558dea69a40 2024-04-17T10:30:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 1 -- 172.21.15.67:0/1489903134 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9d3c045020 con 0x5558dea69a40 2024-04-17T10:30:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 10 client.7425 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:30:49.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 10 client.7425 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 20 client.7425 populate_metadata read hostname 'smithi067' 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea853c0 0x5558dea857d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 4 client.7425 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x5558de87b3d0 con 0x5558dea853c0 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea853c0 0x5558dea857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d55c00640 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea853c0 0x5558dea857d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4901cfd907e37136 server_cookie=d75a8e1616e5c1bf in_seq=0 out_seq=0 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 10 client.7425 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 1 -- 172.21.15.67:0/1489903134 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17922 (crc 0 0 0) 0x5558de87b3d0 con 0x5558dea853c0 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 10 client.7425 handle_command_reply: tid=0 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 client.7425 shutdown 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 msgr2=0x5558dea84940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea84940 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c010fc0 tx=0x7f9d3c015000 comp rx=0 tx=0).stop 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 shutdown_connections 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5558dea853c0 0x5558dea857d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5558dea6a360 0x5558dea84e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5558dea69a40 0x5558dea84940 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 --2- 172.21.15.67:0/1489903134 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5558dea24d90 0x5558deac8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d36a00640 0 client.7425 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 >> 172.21.15.67:0/1489903134 conn(0x5558dea20d40 msgr2=0x5558dea68d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 shutdown_connections 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 1 -- 172.21.15.67:0/1489903134 wait complete. 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.148+0000 7f9d5bae9000 20 client.7425 trim_cache size 0 max 16384 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 1570, 2024-04-17T10:30:49.148 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 1571, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 591579, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 14559201, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 9, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 3, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.188281307, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.087230553, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.038218970, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.048916308, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1570, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.445003413, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000283441 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.149 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000168, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000084 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 0 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 29039, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 49778, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 10534494, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 44172488, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 8, 2024-04-17T10:30:49.150 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 4.552760084, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 2.733573473, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 2.366688784, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.040843952, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 49777, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.402841567, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000028182 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 70500, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.015270605, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000216 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.151 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 108368 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 3156, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 3149, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 879412, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 46794453, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 5, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.361994730, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.182465263, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.075687084, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.084249491, 2024-04-17T10:30:49.152 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3149, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.332653211, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000105637 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 115632, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 19568 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 1, 2024-04-17T10:30:49.153 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-MDSRank": { 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4148, 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 13.534989411, 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.003263015 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-PurgeQueue": { 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1855, 2024-04-17T10:30:49.154 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.834072191, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000449634 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "request": 25520, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "reply": 25520, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "reply_latency": { 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 25520, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 280.396724229, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.010987332 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_reply": 0, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "forward": 0, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_complete": 0, 2024-04-17T10:30:49.155 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_keys": 0, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_commit": 12, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_split": 0, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_merge": 0, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes": 19785, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_top": 0, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_bottom": 19785, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pin_tail": 0, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pinned": 19785, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_expired": 0, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_with_caps": 44, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": 44, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "subtrees": 2, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse": 47205, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_hit": 46687, 2024-04-17T10:30:49.156 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_forward": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_discover": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_dir_fetch": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_remote_ino": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_lock": 152, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "load_cent": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "q": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "exported": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "exported_inodes": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "imported": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "imported_inodes": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_dir_fetch": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_backtrace_fetch": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_peer_discover": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rfiles": 0, 2024-04-17T10:30:49.157 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rbytes": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rsnaps": 1, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_fetch": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_set_tag": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_repaired": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_inotable_repaired": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_inodes": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_base_inodes": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dirfrag_rstats": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_file_inodes": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_inode_file_caps": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_revoke": 976, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_grant": 4020, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_trunc": 0, 2024-04-17T10:30:49.158 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps": 4175, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps_dirty": 2000, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_cap_release": 54, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "process_request_cap_release": 13991 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache": { 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update": 0, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update_receipt": 0, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_try_discover": 0, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_send_discover": 0, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_handle_discover": 0, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays": 2000, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_delayed": 162, 2024-04-17T10:30:49.159 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_enqueuing": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_created": 4075, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_enqueued": 1798, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_reintegrated": 2170, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_migrated": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_processing": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_enqueued": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_prioritized": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_started": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_completed": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_enqueue_scrub": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_exportdir": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_flush": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragmentdir": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragstats": 0, 2024-04-17T10:30:49.160 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_inodestats": 0 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log": { 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "evlrg": 0, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "evadd": 24457, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "evex": 0, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "evtrm": 0, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "ev": 24457, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "evexg": 0, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "evexd": 0, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "segadd": 34, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "segex": 0, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "segtrm": 0, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "seg": 34, 2024-04-17T10:30:49.161 INFO:teuthology.orchestra.run.smithi067.stdout: "segmjr": 3, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "segexg": 0, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "segexd": 0, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "expos": 4194304, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "wrpos": 88651456, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "rdpos": 0, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "jlat": { 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3980, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 91.082407108, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.022885026 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "replayed": 0 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mem": { 2024-04-17T10:30:49.162 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 2014, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "ino+": 2014, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "ino-": 0, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dir": 13, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dir+": 13, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dir-": 0, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dn": 19785, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dn+": 19786, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "dn-": 1, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "cap": 44, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "cap+": 7882, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "cap-": 7838, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "rss": 111400, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "heap": 215340 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.163 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server": { 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_client_request": 28867, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_server_request": 0, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_request": 25520, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_session": 3, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_peer_request": 0, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "req_create_latency": { 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2000, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.283897725, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000641948 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getattr_latency": { 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 28, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.005976960, 2024-04-17T10:30:49.164 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000213462 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getfilelock_latency": { 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "req_link_latency": { 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6000, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 7.109198168, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001184866 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookup_latency": { 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7309, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 6.317768506, 2024-04-17T10:30:49.165 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000864382 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookuphash_latency": { 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupino_latency": { 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupname_latency": { 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.166 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupparent_latency": { 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupsnap_latency": { 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000050182, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000050182 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lssnap_latency": { 2024-04-17T10:30:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mkdir_latency": { 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.003591210, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.003591210 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mknod_latency": { 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.168 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mksnap_latency": { 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.007287313, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.007287313 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "req_open_latency": { 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "req_readdir_latency": { 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.091240339, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.011405042 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.169 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rename_latency": { 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2170, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 249.389218119, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.114925906 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "req_renamesnap_latency": { 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "req_snapdiff_latency": { 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmdir_latency": { 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmsnap_latency": { 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.036592336, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.036592336 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmxattr_latency": { 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setattr_latency": { 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000245453, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000245453 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setdirlayout_latency": { 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setfilelock_latency": { 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setlayout_latency": { 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setxattr_latency": { 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "req_symlink_latency": { 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.173 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "req_unlink_latency": { 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8000, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 16.151657918, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002018957 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_revoke_eviction": 0, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_acquisition_throttle": 0, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getvxattr_latency": { 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.174 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessions": { 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "session_count": 1, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "session_add": 1, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "session_remove": 0, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_open": 1, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_stale": 0, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "total_load": 18459, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "average_load": 18459, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "avg_session_uptime": 48, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "mdthresh_evicted": 0 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 0, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 0, 2024-04-17T10:30:49.175 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-04-17T10:30:49.176 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-04-17T10:30:49.177 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 17930825, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 4557, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-04-17T10:30:49.178 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 5424, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 76, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 21088705, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 818462, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.179 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter": { 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_active": 0, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_laggy": 0, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send": 6016, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send_bytes": 84645448, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_resend": 37, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_reply": 6016, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_latency": { 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6016, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 25.999525500, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.004321729 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "op_inflight": 0, 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "oplen_avg": { 2024-04-17T10:30:49.180 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6016, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 6109 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "op": 6016, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "op_r": 50, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "op_w": 5966, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "op_rmw": 0, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "op_pg": 0, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_stat": 0, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_create": 11, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_read": 50, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_write": 4030, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writefull": 33, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writesame": 0, 2024-04-17T10:30:49.181 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_append": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_zero": 1798, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_truncate": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_delete": 32, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_mapext": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_sparse_read": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_clonerange": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_getxattr": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_setxattr": 22, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_cmpxattr": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_rmxattr": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_resetxattrs": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_call": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_watch": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_notify": 0, 2024-04-17T10:30:49.182 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_src_cmpxattr": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls_filter": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_other": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_active": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_send": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_resend": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_ping": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_active": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_send": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_resend": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_active": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_send": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_resend": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_active": 0, 2024-04-17T10:30:49.183 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_send": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_resend": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "command_active": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "command_send": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "command_resend": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "map_epoch": 83, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "map_full": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "map_inc": 1, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_sessions": 8, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_open": 8, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_close": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_laggy": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_wr": 109, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_rd": 0, 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_del": 24 2024-04-17T10:30:49.184 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "oft": { 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_objs": 1, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_kv_pairs": 0, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_updates": 0, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_removes": 0 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "osd-slow-ops": { 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_ops_count": 0 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "purge_queue": { 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops": 0, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops_high_water": 32, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing": 0, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_high_water": 32, 2024-04-17T10:30:49.185 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed_ops": 1798, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed": 1798, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_item_in_journal": 0 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 33759, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 8962060, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 33759, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 33759, 2024-04-17T10:30:49.186 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 8962060, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_bytes": { 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:30:49.187 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 6016, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 84841430, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 6016, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 84841430, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_ops": { 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 1024, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:30:49.188 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 6016, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 6016, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 6016, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 6016, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle": { 2024-04-17T10:30:49.189 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 1798, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 195982, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 1798, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 50, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 195982, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.190 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle-0x5593ebb6aa00": { 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 232462, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 24457, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 84457152, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 24457, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 3980, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 84224690, 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:30:49.191 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:30:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:30:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:30:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.192 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.192 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:30:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d30066f60 msgr2=0x7f9d30069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d30066f60 0x7f9d30069420 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f9d501f9c50 tx=0x7f9d3c008040 comp rx=0 tx=0).stop 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 msgr2=0x7f9d50155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d50155fb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9d440ae040 tx=0x7f9d4401aa70 comp rx=0 tx=0).stop 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 shutdown_connections 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d30066f60 0x7f9d30069420 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d5015dd50 0x7f9d50156940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d5015d970 0x7f9d5013ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 --2- 172.21.15.67:0/1296129131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d5006b1d0 0x7f9d50155fb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 >> 172.21.15.67:0/1296129131 conn(0x7f9d500b2310 msgr2=0x7f9d5015f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 shutdown_connections 2024-04-17T10:30:49.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.152+0000 7f9d57000640 1 -- 172.21.15.67:0/1296129131 wait complete. 2024-04-17T10:30:49.195 INFO:tasks.cephfs.test_scrub_checks:num of strays: 2000 2024-04-17T10:30:49.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:30:49.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub start '~mdsdir' recursive 2024-04-17T10:30:49.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f234da00640 1 Processor -- start 2024-04-17T10:30:49.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f234da00640 1 -- start start 2024-04-17T10:30:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f234da00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348161710 0x7f23480b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f234da00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2348160500 0x7f234815e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f234da00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23480b53a0 con 0x7f2348161af0 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f234da00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23480b5990 con 0x7f2348160500 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f234da00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23480b2210 con 0x7f2348161710 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f233e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2348161af0 0x7f234815e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37270/0 (socket says 172.21.15.67:37270) 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f233e000640 1 -- 172.21.15.67:0/2588242405 learned_addr learned my addr 172.21.15.67:0/2588242405 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f2347400640 1 --2- 172.21.15.67:0/2588242405 >> v2:172.21.15.86:3300/0 conn(0x7f2348160500 0x7f234815e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.348+0000 7f2346a00640 1 --2- 172.21.15.67:0/2588242405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348161710 0x7f23480b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346000640 1 -- 172.21.15.67:0/2588242405 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229520423 0 0) 0x7f23300014a0 con 0x7f2348161af0 2024-04-17T10:30:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346000640 1 -- 172.21.15.67:0/2588242405 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232c003530 con 0x7f2348161af0 2024-04-17T10:30:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346a00640 1 -- 172.21.15.67:0/2588242405 >> v2:172.21.15.86:3300/0 conn(0x7f2348160500 msgr2=0x7f234815e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346a00640 1 --2- 172.21.15.67:0/2588242405 >> v2:172.21.15.86:3300/0 conn(0x7f2348160500 0x7f234815e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346a00640 1 -- 172.21.15.67:0/2588242405 >> v1:172.21.15.67:6789/0 conn(0x7f2348161af0 legacy=0x7f234815e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346a00640 1 -- 172.21.15.67:0/2588242405 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23480b3920 con 0x7f2348161710 2024-04-17T10:30:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2347400640 1 --2- 172.21.15.67:0/2588242405 >> v2:172.21.15.86:3300/0 conn(0x7f2348160500 0x7f234815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:30:49.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346a00640 1 --2- 172.21.15.67:0/2588242405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348161710 0x7f23480b1cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f23480b5b10 tx=0x7f233801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2872b398e30fe204 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346000640 1 -- 172.21.15.67:0/2588242405 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23380b0070 con 0x7f2348161710 2024-04-17T10:30:49.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346000640 1 -- 172.21.15.67:0/2588242405 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23380ab600 con 0x7f2348161710 2024-04-17T10:30:49.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.352+0000 7f2346000640 1 -- 172.21.15.67:0/2588242405 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23380ab9d0 con 0x7f2348161710 2024-04-17T10:30:49.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- 172.21.15.67:0/2588242405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348161710 msgr2=0x7f23480b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 --2- 172.21.15.67:0/2588242405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348161710 0x7f23480b1cd0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f23480b5b10 tx=0x7f233801c410 comp rx=0 tx=0).stop 2024-04-17T10:30:49.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- 172.21.15.67:0/2588242405 shutdown_connections 2024-04-17T10:30:49.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 --2- 172.21.15.67:0/2588242405 >> v2:172.21.15.86:3300/0 conn(0x7f2348160500 0x7f234815e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 --2- 172.21.15.67:0/2588242405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348161710 0x7f23480b1cd0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- 172.21.15.67:0/2588242405 >> 172.21.15.67:0/2588242405 conn(0x7f23480b5e50 msgr2=0x7f2348099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- 172.21.15.67:0/2588242405 shutdown_connections 2024-04-17T10:30:49.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- 172.21.15.67:0/2588242405 wait complete. 2024-04-17T10:30:49.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 Processor -- start 2024-04-17T10:30:49.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- start start 2024-04-17T10:30:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348160500 0x7f234814e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2348161710 0x7f234814c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2348161af0 0x7f2348157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2348162050 con 0x7f2348161710 2024-04-17T10:30:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2348161ed0 con 0x7f2348161af0 2024-04-17T10:30:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f234da00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23480b1240 con 0x7f2348160500 2024-04-17T10:30:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f233e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2348161710 0x7f234814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f233e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2348161710 0x7f234814c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53538/0 (socket says 172.21.15.67:53538) 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f233e000640 1 -- 172.21.15.67:0/1013933813 learned_addr learned my addr 172.21.15.67:0/1013933813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f2346a00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348160500 0x7f234814e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f2347400640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2348161af0 0x7f2348157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f233e000640 1 -- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348160500 msgr2=0x7f234814e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f233e000640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348160500 0x7f234814e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.356+0000 7f233e000640 1 -- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2348161af0 msgr2=0x7f2348157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233e000640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2348161af0 0x7f2348157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233e000640 1 -- 172.21.15.67:0/1013933813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234814ce40 con 0x7f2348161710 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f2346a00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348160500 0x7f234814e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233e000640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2348161710 0x7f234814c900 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f233000cb80 tx=0x7f233000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233f400640 1 -- 172.21.15.67:0/1013933813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330017070 con 0x7f2348161710 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233f400640 1 -- 172.21.15.67:0/1013933813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f233000ad40 con 0x7f2348161710 2024-04-17T10:30:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233f400640 1 -- 172.21.15.67:0/1013933813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2330012640 con 0x7f2348161710 2024-04-17T10:30:49.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23480c9a90 con 0x7f2348161710 2024-04-17T10:30:49.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23480c9d30 con 0x7f2348161710 2024-04-17T10:30:49.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233f400640 1 -- 172.21.15.67:0/1013933813 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f233001a070 con 0x7f2348161710 2024-04-17T10:30:49.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233f400640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2324066f60 0x7f2324069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.360+0000 7f233f400640 1 -- 172.21.15.67:0/1013933813 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2330084fe0 con 0x7f2348161710 2024-04-17T10:30:49.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 Processor -- start 2024-04-17T10:30:49.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2324066f60 0x7f2324069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- start start 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2324066f60 0x7f2324069420 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f23380ad000 tx=0x7f23380a4f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b50f8c0 0x56336b50fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4c5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b4c5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56336b313dd0 con 0x56336b50f8c0 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56336b29c010 con 0x56336b5101e0 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56336b30fe70 con 0x56336b4c5650 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b50f8c0 0x56336b50fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b50f8c0 0x56336b50fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53546/0 (socket says 172.21.15.67:53546) 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 -- 172.21.15.67:0/366082327 learned_addr learned my addr 172.21.15.67:0/366082327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f233e000640 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2347400640 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b4c5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 -- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 msgr2=0x56336b4c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b4c5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 -- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 msgr2=0x56336b4c5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4c5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 -- 172.21.15.67:0/366082327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56336b2cc9c0 con 0x56336b50f8c0 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2346a00640 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b50f8c0 0x56336b50fca0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f2338097dd0 tx=0x7f2338004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a630a2a0f106a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2337400640 1 -- 172.21.15.67:0/366082327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23380c8070 con 0x56336b50f8c0 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2337400640 1 -- 172.21.15.67:0/366082327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23380b1040 con 0x56336b50f8c0 2024-04-17T10:30:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2337400640 1 -- 172.21.15.67:0/366082327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23380b0040 con 0x56336b50f8c0 2024-04-17T10:30:49.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- 172.21.15.67:0/366082327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b50f8c0 msgr2=0x56336b50fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b50f8c0 0x56336b50fca0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f2338097dd0 tx=0x7f2338004740 comp rx=0 tx=0).stop 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- 172.21.15.67:0/366082327 shutdown_connections 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b4c5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4c5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 --2- 172.21.15.67:0/366082327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b50f8c0 0x56336b50fca0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- 172.21.15.67:0/366082327 >> 172.21.15.67:0/366082327 conn(0x56336b3ad870 msgr2=0x56336b4c39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- 172.21.15.67:0/366082327 shutdown_connections 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- 172.21.15.67:0/366082327 wait complete. 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.364+0000 7f2350fa1000 1 -- start start 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 Processor -- start 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b4cac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b50f8c0 0x56336b4f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4f3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56336b2ce420 con 0x56336b4c5650 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56336b396470 con 0x56336b5101e0 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56336b318400 con 0x56336b50f8c0 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 10 client.0 _ll_register_callbacks cb 0x56336b4b0f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b4cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b4cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53556/0 (socket says 172.21.15.67:53556) 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 -- 172.21.15.67:0/2333647482 learned_addr learned my addr 172.21.15.67:0/2333647482 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f233e000640 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b50f8c0 0x56336b4f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2347400640 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4f3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 -- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b50f8c0 msgr2=0x56336b4f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b50f8c0 0x56336b4f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 -- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 msgr2=0x56336b4f3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4f3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 -- 172.21.15.67:0/2333647482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56336b29c260 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2346a00640 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b4cac90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f233801c8f0 tx=0x7f233801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=396eb0f7be8ecde6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f232be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f232be00640 1 -- 172.21.15.67:0/2333647482 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23380d1060 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f232be00640 1 -- 172.21.15.67:0/2333647482 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f23380b6870 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f23380b1040 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f232be00640 1 -- 172.21.15.67:0/2333647482 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23380d7050 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 10 client.7437 fetch_fsmap learned FSMap version 56 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x56336b2ceb30 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f232be00640 1 -- 172.21.15.67:0/2333647482 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23380b1040 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f232be00640 1 -- 172.21.15.67:0/2333647482 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f23380da020 con 0x56336b4c5650 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 10 client.7437 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 10 client.7437 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:30:49.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 20 client.7437 populate_metadata read hostname 'smithi067' 2024-04-17T10:30:49.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b516af0 0x56336b516ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 4 client.7437 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:30:49.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56336b2afc80 con 0x56336b516af0 2024-04-17T10:30:49.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f233e000640 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b516af0 0x56336b516ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f232be00640 10 client.7437 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.368+0000 7f233e000640 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b516af0 0x56336b516ed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67cbe07dd4d0cf7d server_cookie=7b4289baadb847bc in_seq=0 out_seq=0 2024-04-17T10:30:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f232be00640 1 -- 172.21.15.67:0/2333647482 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56336b2afc80 con 0x56336b516af0 2024-04-17T10:30:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f232be00640 10 client.7437 handle_command_reply: tid=0 2024-04-17T10:30:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 client.7437 shutdown 2024-04-17T10:30:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 msgr2=0x56336b4cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b4cac90 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f233801c8f0 tx=0x7f233801ac50 comp rx=0 tx=0).stop 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 shutdown_connections 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b5101e0 0x56336b4f3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b516af0 0x56336b516ed0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b50f8c0 0x56336b4f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2333647482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b4cac90 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f232be00640 0 client.7437 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 >> 172.21.15.67:0/2333647482 conn(0x56336b3ad870 msgr2=0x56336b4c39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 shutdown_connections 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 1 -- 172.21.15.67:0/2333647482 wait complete. 2024-04-17T10:30:49.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.372+0000 7f2350fa1000 20 client.7437 trim_cache size 0 max 16384 2024-04-17T10:30:49.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 Processor -- start 2024-04-17T10:30:49.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- start start 2024-04-17T10:30:49.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b3ad870 0x56336b5740e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b574620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b574b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23482035c0 con 0x56336b4c5650 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234814e0f0 con 0x56336b50f8c0 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2348156440 con 0x56336b3ad870 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b574620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b574620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53572/0 (socket says 172.21.15.67:53572) 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 -- 172.21.15.67:0/2704588243 learned_addr learned my addr 172.21.15.67:0/2704588243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2347400640 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b574b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2346a00640 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b3ad870 0x56336b5740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 -- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b3ad870 msgr2=0x56336b5740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b3ad870 0x56336b5740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 -- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 msgr2=0x56336b574b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b574b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 -- 172.21.15.67:0/2704588243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56336b3996d0 con 0x56336b4c5650 2024-04-17T10:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2347400640 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b574b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f233e000640 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b574620 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f2330051bf0 tx=0x7f23300831d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e2e8c0f6b22c85e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2336a00640 1 -- 172.21.15.67:0/2704588243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233006d190 con 0x56336b4c5650 2024-04-17T10:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2336a00640 1 -- 172.21.15.67:0/2704588243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2330051e00 con 0x56336b4c5650 2024-04-17T10:30:49.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2336a00640 1 -- 172.21.15.67:0/2704588243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233005db80 con 0x56336b4c5650 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- 172.21.15.67:0/2704588243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 msgr2=0x56336b574620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b574620 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f2330051bf0 tx=0x7f23300831d0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- 172.21.15.67:0/2704588243 shutdown_connections 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b574b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b4c5650 0x56336b574620 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2704588243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b3ad870 0x56336b5740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- 172.21.15.67:0/2704588243 >> 172.21.15.67:0/2704588243 conn(0x56336b4f3c10 msgr2=0x56336b4c9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- 172.21.15.67:0/2704588243 shutdown_connections 2024-04-17T10:30:49.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- 172.21.15.67:0/2704588243 wait complete. 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 -- start start 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.388+0000 7f2350fa1000 1 Processor -- start 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b3ad870 0x56336b513620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b513b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b5140a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f234814daf0 con 0x56336b3ad870 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56336b318400 con 0x56336b50f8c0 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56336b396470 con 0x56336b4c5650 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 10 client.0 _ll_register_callbacks cb 0x56336b2b0200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b3ad870 0x56336b513620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b3ad870 0x56336b513620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53576/0 (socket says 172.21.15.67:53576) 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 -- 172.21.15.67:0/2687622430 learned_addr learned my addr 172.21.15.67:0/2687622430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2347400640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b5140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f233e000640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b513b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 -- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 msgr2=0x56336b513b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b513b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 -- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 msgr2=0x56336b5140a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b5140a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 -- 172.21.15.67:0/2687622430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56336b396900 con 0x56336b3ad870 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f233e000640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b513b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2346a00640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b3ad870 0x56336b513620 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f233801c980 tx=0x7f23380be000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47396f12fa722426 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f232be00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:30:49.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f232be00640 1 -- 172.21.15.67:0/2687622430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23380c8040 con 0x56336b3ad870 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f232be00640 1 -- 172.21.15.67:0/2687622430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f23380c4c40 con 0x56336b3ad870 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2330051e00 con 0x56336b3ad870 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f232be00640 1 -- 172.21.15.67:0/2687622430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f23380b0040 con 0x56336b3ad870 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 10 client.7443 fetch_fsmap learned FSMap version 56 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x56336b2baf20 con 0x56336b3ad870 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f232be00640 1 -- 172.21.15.67:0/2687622430 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2330051e00 con 0x56336b3ad870 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f232be00640 1 -- 172.21.15.67:0/2687622430 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f23380da040 con 0x56336b3ad870 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 10 client.7443 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 10 client.7443 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 20 client.7443 populate_metadata read hostname 'smithi067' 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b5101e0 0x56336b5145e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 4 client.7443 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "~mdsdir", "scrubops": ["recursive"]}] 2024-04-17T10:30:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "scrub start", "path": "~mdsdir", "scrubops": ["recursive"]}) v1 -- 0x56336b321310 con 0x56336b5101e0 2024-04-17T10:30:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f233e000640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b5101e0 0x56336b5145e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f233e000640 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b5101e0 0x56336b5145e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38956fb4535c6504 server_cookie=d38b5a7cf599889d in_seq=0 out_seq=0 2024-04-17T10:30:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.392+0000 7f232be00640 10 client.7443 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f232be00640 1 -- 172.21.15.67:0/2687622430 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56336b321310 con 0x56336b5101e0 2024-04-17T10:30:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f232be00640 10 client.7443 handle_command_reply: tid=0 2024-04-17T10:30:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 client.7443 shutdown 2024-04-17T10:30:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b3ad870 msgr2=0x56336b513620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b3ad870 0x56336b513620 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f233801c980 tx=0x7f23380be000 comp rx=0 tx=0).stop 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 shutdown_connections 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56336b50f8c0 0x56336b5140a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x56336b5101e0 0x56336b5145e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56336b4c5650 0x56336b513b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 --2- 172.21.15.67:0/2687622430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56336b3ad870 0x56336b513620 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 >> 172.21.15.67:0/2687622430 conn(0x56336b4f3c10 msgr2=0x56336b50efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f232be00640 0 client.7443 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 shutdown_connections 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 1 -- 172.21.15.67:0/2687622430 wait complete. 2024-04-17T10:30:49.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f2350fa1000 20 client.7443 trim_cache size 0 max 16384 2024-04-17T10:30:49.400 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:30:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:30:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "57163d82-5993-4857-867a-a79addfdd516", 2024-04-17T10:30:49.400 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:30:49.400 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2324066f60 msgr2=0x7f2324069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2324066f60 0x7f2324069420 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f23380ad000 tx=0x7f23380a4f90 comp rx=0 tx=0).stop 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2348161710 msgr2=0x7f234814c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2348161710 0x7f234814c900 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f233000cb80 tx=0x7f233000a430 comp rx=0 tx=0).stop 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 shutdown_connections 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2324066f60 0x7f2324069420 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2348161af0 0x7f2348157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2348161710 0x7f234814c900 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 --2- 172.21.15.67:0/1013933813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2348160500 0x7f234814e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 >> 172.21.15.67:0/1013933813 conn(0x7f23480b5e50 msgr2=0x7f2348154550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 shutdown_connections 2024-04-17T10:30:49.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.404+0000 7f234da00640 1 -- 172.21.15.67:0/1013933813 wait complete. 2024-04-17T10:30:49.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:30:49.426 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-17T10:30:49.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8ccf400640 1 Processor -- start 2024-04-17T10:30:49.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8ccf400640 1 -- start start 2024-04-17T10:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8ccf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cc8159440 0x7f8cc80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8ccf400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc815cb30 con 0x7f8cc80713e0 2024-04-17T10:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8ccf400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc80cd610 con 0x7f8cc815c580 2024-04-17T10:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8ccf400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc80a8650 con 0x7f8cc8159440 2024-04-17T10:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8cbfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8cc80713e0 0x7f8cc815be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37282/0 (socket says 172.21.15.67:37282) 2024-04-17T10:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8cbfe00640 1 -- 172.21.15.67:0/1553896812 learned_addr learned my addr 172.21.15.67:0/1553896812 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:30:49.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.652+0000 7f8ccca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8cc8159440 0x7f8cc80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8ccca00640 1 -- 172.21.15.67:0/1553896812 >> v1:172.21.15.86:6789/0 conn(0x7f8cc815c580 legacy=0x7f8cc815d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8ccca00640 1 -- 172.21.15.67:0/1553896812 >> v1:172.21.15.67:6789/0 conn(0x7f8cc80713e0 legacy=0x7f8cc815be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8ccca00640 1 -- 172.21.15.67:0/1553896812 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc80cd8f0 con 0x7f8cc8159440 2024-04-17T10:30:49.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8cc7e00640 1 -- 172.21.15.67:0/1553896812 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821682504 0 0) 0x7f8cc815cb30 con 0x7f8cc80713e0 2024-04-17T10:30:49.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8ccca00640 1 --2- 172.21.15.67:0/1553896812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc80ce0f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80097c0 tx=0x7f8cc815df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140ed7026450c473 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8cc7e00640 1 -- 172.21.15.67:0/1553896812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80b1070 con 0x7f8cc8159440 2024-04-17T10:30:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8cc7e00640 1 -- 172.21.15.67:0/1553896812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb80ac410 con 0x7f8cc8159440 2024-04-17T10:30:49.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8cc7e00640 1 -- 172.21.15.67:0/1553896812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80ac7e0 con 0x7f8cc8159440 2024-04-17T10:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8ccf400640 1 -- 172.21.15.67:0/1553896812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 msgr2=0x7f8cc80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.656+0000 7f8ccf400640 1 --2- 172.21.15.67:0/1553896812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc80ce0f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80097c0 tx=0x7f8cc815df50 comp rx=0 tx=0).stop 2024-04-17T10:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- 172.21.15.67:0/1553896812 shutdown_connections 2024-04-17T10:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 --2- 172.21.15.67:0/1553896812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc80ce0f0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- 172.21.15.67:0/1553896812 >> 172.21.15.67:0/1553896812 conn(0x7f8cc80b1740 msgr2=0x7f8cc80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- 172.21.15.67:0/1553896812 shutdown_connections 2024-04-17T10:30:49.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- 172.21.15.67:0/1553896812 wait complete. 2024-04-17T10:30:49.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 Processor -- start 2024-04-17T10:30:49.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- start start 2024-04-17T10:30:49.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cc80713e0 0x7f8cc81f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc81f7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cc815c580 0x7f8cc8151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc8169640 con 0x7f8cc8159440 2024-04-17T10:30:49.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc81694c0 con 0x7f8cc80713e0 2024-04-17T10:30:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccf400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc81697c0 con 0x7f8cc815c580 2024-04-17T10:30:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc81f7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc81f7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53584/0 (socket says 172.21.15.67:53584) 2024-04-17T10:30:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2195327466 learned_addr learned my addr 172.21.15.67:0/2195327466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccca00640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cc80713e0 0x7f8cc81f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8ccd400640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cc815c580 0x7f8cc8151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cc815c580 msgr2=0x7f8cc8151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cc815c580 0x7f8cc8151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cc80713e0 msgr2=0x7f8cc81f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cc80713e0 0x7f8cc81f4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2195327466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc8152a40 con 0x7f8cc8159440 2024-04-17T10:30:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.660+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc81f7b80 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c410 tx=0x7f8cb000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8cc5600640 1 -- 172.21.15.67:0/2195327466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb00074e0 con 0x7f8cc8159440 2024-04-17T10:30:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8cc5600640 1 -- 172.21.15.67:0/2195327466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb00047f0 con 0x7f8cc8159440 2024-04-17T10:30:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8cc5600640 1 -- 172.21.15.67:0/2195327466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0007680 con 0x7f8cc8159440 2024-04-17T10:30:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc8152d20 con 0x7f8cc8159440 2024-04-17T10:30:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc8153a10 con 0x7f8cc8159440 2024-04-17T10:30:49.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8cc5600640 1 -- 172.21.15.67:0/2195327466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8cb0011070 con 0x7f8cc8159440 2024-04-17T10:30:49.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8cc5600640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ca4066f60 0x7f8ca4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.664+0000 7f8cc5600640 1 -- 172.21.15.67:0/2195327466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8cb0084d10 con 0x7f8cc8159440 2024-04-17T10:30:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ca4066f60 0x7f8ca4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 Processor -- start 2024-04-17T10:30:49.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- start start 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ca4066f60 0x7f8ca4069420 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80ae000 tx=0x7f8cb8004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621897bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218931b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x556218932480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55621877eff0 con 0x55621897b9d0 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556218707150 con 0x55621897c2f0 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55621877b090 con 0x5562189320a0 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621897bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621897bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53590/0 (socket says 172.21.15.67:53590) 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 -- 172.21.15.67:0/3700439109 learned_addr learned my addr 172.21.15.67:0/3700439109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218931b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccd400640 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x556218932480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 -- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 msgr2=0x556218932480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x556218932480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 -- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 msgr2=0x556218931b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218931b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 -- 172.21.15.67:0/3700439109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562189150d0 con 0x55621897b9d0 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218931b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8ccca00640 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621897bdb0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8cb801a0d0 tx=0x7f8cb80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d4e6d06fa8d3b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cbd600640 1 -- 172.21.15.67:0/3700439109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80b1040 con 0x55621897b9d0 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cbd600640 1 -- 172.21.15.67:0/3700439109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb80c8070 con 0x55621897b9d0 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cbd600640 1 -- 172.21.15.67:0/3700439109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80b8480 con 0x55621897b9d0 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- 172.21.15.67:0/3700439109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 msgr2=0x55621897bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621897bdb0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f8cb801a0d0 tx=0x7f8cb80046d0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- 172.21.15.67:0/3700439109 shutdown_connections 2024-04-17T10:30:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x556218932480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218931b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/3700439109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621897bdb0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- 172.21.15.67:0/3700439109 >> 172.21.15.67:0/3700439109 conn(0x55621892e020 msgr2=0x556218930440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- 172.21.15.67:0/3700439109 shutdown_connections 2024-04-17T10:30:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.668+0000 7f8cd27ea000 1 -- 172.21.15.67:0/3700439109 wait complete. 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 -- start start 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 Processor -- start 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x5562189607f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x556218960d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218961270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556218743250 con 0x55621897b9d0 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556218739690 con 0x55621897c2f0 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556218707a40 con 0x5562189320a0 2024-04-17T10:30:49.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 10 client.0 _ll_register_callbacks cb 0x55621891ce90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x556218960d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8ccca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x5562189607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x556218960d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53598/0 (socket says 172.21.15.67:53598) 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 -- 172.21.15.67:0/1375618271 learned_addr learned my addr 172.21.15.67:0/1375618271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8ccd400640 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218961270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 -- 172.21.15.67:0/1375618271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 msgr2=0x5562189607f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x5562189607f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 -- 172.21.15.67:0/1375618271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 msgr2=0x556218961270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218961270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 -- 172.21.15.67:0/1375618271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55621882c1c0 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x556218960d30 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000cdd0 tx=0x7f8cb0059720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e235dce1f39cf7da server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cb6000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cb6000640 1 -- 172.21.15.67:0/1375618271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb005d700 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cb6000640 1 -- 172.21.15.67:0/1375618271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8cb005d8a0 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cb80c8070 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cb6000640 1 -- 172.21.15.67:0/1375618271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb0054980 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 10 client.7458 fetch_fsmap learned FSMap version 56 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x556218828170 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cb6000640 1 -- 172.21.15.67:0/1375618271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb80c8070 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cb6000640 1 -- 172.21.15.67:0/1375618271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8cb0054b20 con 0x55621897b9d0 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 10 client.7458 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:30:49.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 10 client.7458 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 20 client.7458 populate_metadata read hostname 'smithi067' 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x556218961f50 0x55621898dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 4 client.7458 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55621871ac60 con 0x556218961f50 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.672+0000 7f8ccca00640 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x556218961f50 0x55621898dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cb6000640 10 client.7458 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8ccca00640 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x556218961f50 0x55621898dc90 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7065767269309f1 server_cookie=84b522f58d420f22 in_seq=0 out_seq=0 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cb6000640 1 -- 172.21.15.67:0/1375618271 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55621871ac60 con 0x556218961f50 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cb6000640 10 client.7458 handle_command_reply: tid=0 2024-04-17T10:30:49.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 client.7458 shutdown 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 msgr2=0x556218960d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x556218960d30 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000cdd0 tx=0x7f8cb0059720 comp rx=0 tx=0).stop 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 shutdown_connections 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x556218961270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x556218960d30 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x556218961f50 0x55621898dc90 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/1375618271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x5562189607f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 >> 172.21.15.67:0/1375618271 conn(0x55621892e020 msgr2=0x556218930ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cb6000640 0 client.7458 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 shutdown_connections 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 1 -- 172.21.15.67:0/1375618271 wait complete. 2024-04-17T10:30:49.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.676+0000 7f8cd27ea000 20 client.7458 trim_cache size 0 max 16384 2024-04-17T10:30:49.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 Processor -- start 2024-04-17T10:30:49.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- start start 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5562189320a0 0x5562189d8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x5562189d8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55621897c2f0 0x5562189d8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc82011b0 con 0x55621897b9d0 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc82003b0 con 0x5562189320a0 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc81ff7b0 con 0x55621897c2f0 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x5562189d8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x5562189d8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53602/0 (socket says 172.21.15.67:53602) 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2787122448 learned_addr learned my addr 172.21.15.67:0/2787122448 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8ccca00640 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5562189320a0 0x5562189d8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8ccd400640 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55621897c2f0 0x5562189d8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55621897c2f0 msgr2=0x5562189d8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55621897c2f0 0x5562189d8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5562189320a0 msgr2=0x5562189d8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5562189320a0 0x5562189d8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 -- 172.21.15.67:0/2787122448 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562187c5de0 con 0x55621897b9d0 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8ccca00640 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5562189320a0 0x5562189d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x5562189d8880 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0054eb0 tx=0x7f8cb006a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246942a3638754c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbcc00640 1 -- 172.21.15.67:0/2787122448 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb0055070 con 0x55621897b9d0 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbcc00640 1 -- 172.21.15.67:0/2787122448 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb0054b20 con 0x55621897b9d0 2024-04-17T10:30:49.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cbcc00640 1 -- 172.21.15.67:0/2787122448 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb00513c0 con 0x55621897b9d0 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- 172.21.15.67:0/2787122448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 msgr2=0x5562189d8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x5562189d8880 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0054eb0 tx=0x7f8cb006a000 comp rx=0 tx=0).stop 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- 172.21.15.67:0/2787122448 shutdown_connections 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55621897c2f0 0x5562189d8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x5562189d8880 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/2787122448 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5562189320a0 0x5562189d8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- 172.21.15.67:0/2787122448 >> 172.21.15.67:0/2787122448 conn(0x55621892e020 msgr2=0x556218988fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- 172.21.15.67:0/2787122448 shutdown_connections 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- 172.21.15.67:0/2787122448 wait complete. 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.692+0000 7f8cd27ea000 1 -- start start 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 Processor -- start 2024-04-17T10:30:49.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x55621898e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621898e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x55621898ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556218719e00 con 0x55621897b9d0 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc81ffdb0 con 0x55621897c2f0 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55621877e300 con 0x5562189320a0 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 10 client.0 _ll_register_callbacks cb 0x5562188fd040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621898e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x55621898e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x55621898e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:56338/0 (socket says 172.21.15.67:56338) 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 -- 172.21.15.67:0/538558568 learned_addr learned my addr 172.21.15.67:0/538558568 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccd400640 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x55621898ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 -- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 msgr2=0x55621898ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x55621898ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 -- 172.21.15.67:0/538558568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 msgr2=0x55621898e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621898e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 -- 172.21.15.67:0/538558568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562187208e0 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8ccca00640 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x55621898e390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8000c10 tx=0x7f8cb801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12fa7884a812f8eb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 10 client.0 ms_handle_connect on v2:172.21.15.86:3301/0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 1 -- 172.21.15.67:0/538558568 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb80b1040 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 1 -- 172.21.15.67:0/538558568 <== mon.2 v2:172.21.15.86:3301/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8cb80d5b80 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8cb0054b20 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 1 -- 172.21.15.67:0/538558568 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cb80e5050 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 10 client.6398 fetch_fsmap learned FSMap version 56 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x5562187d88f0 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 1 -- 172.21.15.67:0/538558568 <== mon.2 v2:172.21.15.86:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb0054b20 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 1 -- 172.21.15.67:0/538558568 <== mon.2 v2:172.21.15.86:3301/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8cb80e8020 con 0x5562189320a0 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 10 client.6398 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:30:49.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 10 client.6398 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 20 client.6398 populate_metadata read hostname 'smithi067' 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5562189e1b70 0x5562189e3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 4 client.6398 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55621878c530 con 0x5562189e1b70 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5562189e1b70 0x5562189e3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cbfe00640 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5562189e1b70 0x5562189e3f60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cccb1da1230a50b server_cookie=754f3975734f7d0a in_seq=0 out_seq=0 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 10 client.6398 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 1 -- 172.21.15.67:0/538558568 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+257 (crc 0 0 0) 0x55621878c530 con 0x5562189e1b70 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cb6000640 10 client.6398 handle_command_reply: tid=0 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.696+0000 7f8cd27ea000 1 client.6398 shutdown 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 msgr2=0x55621898e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x55621898e390 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8000c10 tx=0x7f8cb801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 shutdown_connections 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55621897c2f0 0x55621898ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55621897b9d0 0x55621898e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x5562189e1b70 0x5562189e3f60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 --2- 172.21.15.67:0/538558568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5562189320a0 0x55621898e390 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cb6000640 0 client.6398 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 >> 172.21.15.67:0/538558568 conn(0x55621892e020 msgr2=0x55621897a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 shutdown_connections 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 1 -- 172.21.15.67:0/538558568 wait complete. 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8cd27ea000 20 client.6398 trim_cache size 0 max 16384 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:30:49.697 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "57163d82-5993-4857-867a-a79addfdd516": { 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "~mds0", 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "57163d82-5993-4857-867a-a79addfdd516", 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:30:49.698 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ca4066f60 msgr2=0x7f8ca4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ca4066f60 0x7f8ca4069420 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80ae000 tx=0x7f8cb8004740 comp rx=0 tx=0).stop 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 msgr2=0x7f8cc81f7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc81f7b80 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000c410 tx=0x7f8cb000c8d0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 shutdown_connections 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ca4066f60 0x7f8ca4069420 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8cc815c580 0x7f8cc8151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8cc8159440 0x7f8cc81f7b80 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 --2- 172.21.15.67:0/2195327466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8cc80713e0 0x7f8cc81f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:30:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 >> 172.21.15.67:0/2195327466 conn(0x7f8cc80b1740 msgr2=0x7f8cc80ce0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:30:49.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 shutdown_connections 2024-04-17T10:30:49.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:30:49.700+0000 7f8ccf400640 1 -- 172.21.15.67:0/2195327466 wait complete. 2024-04-17T10:30:49.712 INFO:tasks.cephfs.filesystem:scrub status for tag:57163d82-5993-4857-867a-a79addfdd516 - {'path': '~mds0', 'tag': '57163d82-5993-4857-867a-a79addfdd516', 'options': 'recursive'} 2024-04-17T10:30:55.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:30:55.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:31:19.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:19.714 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.5:0 scrub status 2024-04-17T10:31:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1859600640 1 Processor -- start 2024-04-17T10:31:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1859600640 1 -- start start 2024-04-17T10:31:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1859600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 0x7f185409a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1859600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185415f390 con 0x7f18540cdcd0 2024-04-17T10:31:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1859600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18540cf5c0 con 0x7f18540cfa80 2024-04-17T10:31:19.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1859600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18540cc9b0 con 0x7f185415efb0 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f184b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f18540cdcd0 0x7f18540cce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55282/0 (socket says 172.21.15.67:55282) 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f184b400640 1 -- 172.21.15.67:0/94638074 learned_addr learned my addr 172.21.15.67:0/94638074 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853e00640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 0x7f185409a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536034184 0 0) 0x7f185415f390 con 0x7f18540cdcd0 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1838003530 con 0x7f18540cdcd0 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102228392 0 0) 0x7f18540cf5c0 con 0x7f18540cfa80 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185415f390 con 0x7f18540cfa80 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3224719132 0 0) 0x7f1838003530 con 0x7f18540cdcd0 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 msgr2=0x7f185409a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 0x7f185409a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 >> v1:172.21.15.86:6789/0 conn(0x7f18540cfa80 legacy=0x7f18540c5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18540cc360 con 0x7f18540cdcd0 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f183c001680 con 0x7f18540cdcd0 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 >> v1:172.21.15.67:6789/0 conn(0x7f18540cdcd0 legacy=0x7f18540cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1838003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 0x7f1838004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1838004d50 0x7f18380051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18540cc9b0 con 0x7f1838003b80 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183800bb60 con 0x7f1838004d50 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.875+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183800bd30 con 0x7f185415efb0 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853e00640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1838003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f184b400640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 0x7f1838004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1858c00640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1838004d50 0x7f18380051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853e00640 1 -- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 msgr2=0x7f1838004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853e00640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f185415efb0 0x7f1838004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853e00640 1 -- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1838004d50 msgr2=0x7f18380051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853e00640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1838004d50 0x7f18380051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853e00640 1 -- 172.21.15.67:0/94638074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1838003530 con 0x7f1838003b80 2024-04-17T10:31:19.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853e00640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1838003f80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f184400e170 tx=0x7f184401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5370a7e2c47b0cce server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:19.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440b2070 con 0x7f1838003b80 2024-04-17T10:31:19.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18440a5a50 con 0x7f1838003b80 2024-04-17T10:31:19.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.879+0000 7f1853400640 1 -- 172.21.15.67:0/94638074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844009c10 con 0x7f1838003b80 2024-04-17T10:31:19.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- 172.21.15.67:0/94638074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 msgr2=0x7f1838003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 --2- 172.21.15.67:0/94638074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1838003f80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f184400e170 tx=0x7f184401cc60 comp rx=0 tx=0).stop 2024-04-17T10:31:19.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1858c00640 1 -- 172.21.15.67:0/94638074 reap_dead start 2024-04-17T10:31:19.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- 172.21.15.67:0/94638074 shutdown_connections 2024-04-17T10:31:19.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- 172.21.15.67:0/94638074 >> 172.21.15.67:0/94638074 conn(0x7f18540d1630 msgr2=0x7f18540d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:19.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- 172.21.15.67:0/94638074 shutdown_connections 2024-04-17T10:31:19.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- 172.21.15.67:0/94638074 wait complete. 2024-04-17T10:31:19.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 Processor -- start 2024-04-17T10:31:19.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- start start 2024-04-17T10:31:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1854159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1854158ce0 0x7f185414ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18540ca710 0x7f18540caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18540cf380 con 0x7f1838003b80 2024-04-17T10:31:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18540cf200 con 0x7f18540ca710 2024-04-17T10:31:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1859600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18540cfe90 con 0x7f1854158ce0 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1854159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1854159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32952/0 (socket says 172.21.15.67:32952) 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 -- 172.21.15.67:0/3379748874 learned_addr learned my addr 172.21.15.67:0/3379748874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1858c00640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18540ca710 0x7f18540caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f184b400640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1854158ce0 0x7f185414ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 -- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1854158ce0 msgr2=0x7f185414ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1854158ce0 0x7f185414ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 -- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18540ca710 msgr2=0x7f18540caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18540ca710 0x7f18540caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f1853e00640 1 -- 172.21.15.67:0/3379748874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185415c790 con 0x7f1838003b80 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.883+0000 7f184b400640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1854158ce0 0x7f185414ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1853e00640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1854159210 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f18440b0040 tx=0x7f1844009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1850c00640 1 -- 172.21.15.67:0/3379748874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440bc020 con 0x7f1838003b80 2024-04-17T10:31:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1850c00640 1 -- 172.21.15.67:0/3379748874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18440ada20 con 0x7f1838003b80 2024-04-17T10:31:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1850c00640 1 -- 172.21.15.67:0/3379748874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440b2040 con 0x7f1838003b80 2024-04-17T10:31:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185415cff0 con 0x7f1838003b80 2024-04-17T10:31:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185415cd10 con 0x7f1838003b80 2024-04-17T10:31:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1850c00640 1 -- 172.21.15.67:0/3379748874 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f18440bf070 con 0x7f1838003b80 2024-04-17T10:31:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1850c00640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1830066f60 0x7f1830069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f1850c00640 1 -- 172.21.15.67:0/3379748874 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f184411ebf0 con 0x7f1838003b80 2024-04-17T10:31:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f184b400640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1830066f60 0x7f1830069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.887+0000 7f185e0e8000 1 Processor -- start 2024-04-17T10:31:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- start start 2024-04-17T10:31:19.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1830066f60 0x7f1830069420 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f183c001210 tx=0x7f183c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242bf9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242bafb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bb0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642429fcff0 con 0x564242bfa2f0 2024-04-17T10:31:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564242985150 con 0x564242bf99d0 2024-04-17T10:31:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642429f9090 con 0x564242bb00a0 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242bafb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242bafb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32962/0 (socket says 172.21.15.67:32962) 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 -- 172.21.15.67:0/402460672 learned_addr learned my addr 172.21.15.67:0/402460672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f1853e00640 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242bf9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f1858c00640 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bb0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 -- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 msgr2=0x564242bb0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bb0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 -- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 msgr2=0x564242bf9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242bf9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 -- 172.21.15.67:0/402460672 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564242b930d0 con 0x564242bfa2f0 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f1858c00640 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bb0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f184b400640 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242bafb60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f183c0109f0 tx=0x7f183c00c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9db4f64f33f4ed server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f1848c00640 1 -- 172.21.15.67:0/402460672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c027070 con 0x564242bfa2f0 2024-04-17T10:31:19.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f1848c00640 1 -- 172.21.15.67:0/402460672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c014b80 con 0x564242bfa2f0 2024-04-17T10:31:19.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f1848c00640 1 -- 172.21.15.67:0/402460672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c001620 con 0x564242bfa2f0 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- 172.21.15.67:0/402460672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 msgr2=0x564242bafb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242bafb60 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f183c0109f0 tx=0x7f183c00c000 comp rx=0 tx=0).stop 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- 172.21.15.67:0/402460672 shutdown_connections 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bb0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242bafb60 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 --2- 172.21.15.67:0/402460672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242bf9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- 172.21.15.67:0/402460672 >> 172.21.15.67:0/402460672 conn(0x564242bac020 msgr2=0x564242bae440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- 172.21.15.67:0/402460672 shutdown_connections 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- 172.21.15.67:0/402460672 wait complete. 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 -- start start 2024-04-17T10:31:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.891+0000 7f185e0e8000 1 Processor -- start 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bde7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242bded30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bfa2f0 0x564242bdf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642429c1250 con 0x564242bf99d0 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642429b7690 con 0x564242bfa2f0 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x564242985a40 con 0x564242bb00a0 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 10 client.0 _ll_register_callbacks cb 0x564242b9ae90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242bded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242bded30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32976/0 (socket says 172.21.15.67:32976) 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 -- 172.21.15.67:0/2060216318 learned_addr learned my addr 172.21.15.67:0/2060216318 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1853e00640 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bde7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1858c00640 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bfa2f0 0x564242bdf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 -- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 msgr2=0x564242bde7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bde7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 -- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bfa2f0 msgr2=0x564242bdf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bfa2f0 0x564242bdf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 -- 172.21.15.67:0/2060216318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564242aaa1c0 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f184b400640 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242bded30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f183c001000 tx=0x7f183c001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cfad2db5a7f4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 1 -- 172.21.15.67:0/2060216318 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c027070 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 1 -- 172.21.15.67:0/2060216318 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f183c022c60 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f183c014b80 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 1 -- 172.21.15.67:0/2060216318 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f183c03a050 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 10 client.7476 fetch_fsmap learned FSMap version 56 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x564242aa6170 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 1 -- 172.21.15.67:0/2060216318 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c014b80 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 1 -- 172.21.15.67:0/2060216318 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f183c03d020 con 0x564242bf99d0 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 10 client.7476 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:31:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 10 client.7476 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:31:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 20 client.7476 populate_metadata read hostname 'smithi067' 2024-04-17T10:31:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c09c80 0x564242c0c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 4 client.7476 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:31:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564242998c60 con 0x564242c09c80 2024-04-17T10:31:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1853e00640 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c09c80 0x564242c0c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1853e00640 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c09c80 0x564242c0c070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a5c3368acf6a894 server_cookie=e195dbece4df5164 in_seq=0 out_seq=0 2024-04-17T10:31:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 10 client.7476 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 1 -- 172.21.15.67:0/2060216318 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564242998c60 con 0x564242c09c80 2024-04-17T10:31:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f1841600640 10 client.7476 handle_command_reply: tid=0 2024-04-17T10:31:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.895+0000 7f185e0e8000 1 client.7476 shutdown 2024-04-17T10:31:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 msgr2=0x564242bded30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242bded30 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f183c001000 tx=0x7f183c001980 comp rx=0 tx=0).stop 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 shutdown_connections 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bfa2f0 0x564242bdf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242bded30 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c09c80 0x564242c0c070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2060216318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242bde7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f1841600640 0 client.7476 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 >> 172.21.15.67:0/2060216318 conn(0x564242bac020 msgr2=0x564242baeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 shutdown_connections 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 1 -- 172.21.15.67:0/2060216318 wait complete. 2024-04-17T10:31:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.899+0000 7f185e0e8000 20 client.7476 trim_cache size 0 max 16384 2024-04-17T10:31:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.911+0000 7f185e0e8000 1 Processor -- start 2024-04-17T10:31:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.911+0000 7f185e0e8000 1 -- start start 2024-04-17T10:31:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242baf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242baf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242c17b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854201f50 con 0x564242bfa2f0 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854201110 con 0x564242bf99d0 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1854200670 con 0x564242bb00a0 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242c17b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242c17b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:32984/0 (socket says 172.21.15.67:32984) 2024-04-17T10:31:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 -- 172.21.15.67:0/2524110453 learned_addr learned my addr 172.21.15.67:0/2524110453 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1853e00640 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242baf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f184b400640 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242baf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 -- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 msgr2=0x564242baf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242baf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 -- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 msgr2=0x564242baf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242baf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 -- 172.21.15.67:0/2524110453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564242a43de0 con 0x564242bfa2f0 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1853e00640 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242baf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1858c00640 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242c17b00 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1854072720 tx=0x7f184c0039d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a503a433901687d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1843e00640 1 -- 172.21.15.67:0/2524110453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c0067e0 con 0x564242bfa2f0 2024-04-17T10:31:19.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1843e00640 1 -- 172.21.15.67:0/2524110453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c006980 con 0x564242bfa2f0 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f1843e00640 1 -- 172.21.15.67:0/2524110453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c006b70 con 0x564242bfa2f0 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- 172.21.15.67:0/2524110453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 msgr2=0x564242c17b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242c17b00 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f1854072720 tx=0x7f184c0039d0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- 172.21.15.67:0/2524110453 shutdown_connections 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bfa2f0 0x564242c17b00 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bf99d0 0x564242baf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- 172.21.15.67:0/2524110453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bb00a0 0x564242baf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- 172.21.15.67:0/2524110453 >> 172.21.15.67:0/2524110453 conn(0x564242bac020 msgr2=0x564242bf80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- 172.21.15.67:0/2524110453 shutdown_connections 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- 172.21.15.67:0/2524110453 wait complete. 2024-04-17T10:31:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 -- start start 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 Processor -- start 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.915+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bb00a0 0x564242c5e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242c5eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bfa2f0 0x564242c19e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564242997e00 con 0x564242bf99d0 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1854200b40 con 0x564242bb00a0 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5642429fc300 con 0x564242bfa2f0 2024-04-17T10:31:19.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 10 client.0 _ll_register_callbacks cb 0x564242b7b040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f184b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242c5eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1853e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bb00a0 0x564242c5e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f184b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242c5eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33000/0 (socket says 172.21.15.67:33000) 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1858c00640 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bfa2f0 0x564242c19e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f184b400640 1 -- 172.21.15.67:0/3400981944 learned_addr learned my addr 172.21.15.67:0/3400981944 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1853e00640 1 -- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bfa2f0 msgr2=0x564242c19e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1853e00640 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bfa2f0 0x564242c19e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1853e00640 1 -- 172.21.15.67:0/3400981944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 msgr2=0x564242c5eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1853e00640 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242c5eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1853e00640 1 -- 172.21.15.67:0/3400981944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56424299e8e0 con 0x564242bb00a0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1853e00640 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bb00a0 0x564242c5e9b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f184411cea0 tx=0x7f18440eef60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd094373623468fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 1 -- 172.21.15.67:0/3400981944 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184411cfe0 con 0x564242bb00a0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 1 -- 172.21.15.67:0/3400981944 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f18440ed660 con 0x564242bb00a0 2024-04-17T10:31:19.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f184c006980 con 0x564242bb00a0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 1 -- 172.21.15.67:0/3400981944 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18440f30c0 con 0x564242bb00a0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 10 client.7036 fetch_fsmap learned FSMap version 56 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x564242a568f0 con 0x564242bb00a0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 1 -- 172.21.15.67:0/3400981944 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c006980 con 0x564242bb00a0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 1 -- 172.21.15.67:0/3400981944 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f18440b5050 con 0x564242bb00a0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 10 client.7036 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 10 client.7036 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 20 client.7036 populate_metadata read hostname 'smithi067' 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c1a7b0 0x564242c1ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 4 client.7036 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564242a0a530 con 0x564242c1a7b0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f184b400640 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c1a7b0 0x564242c1ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f184b400640 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c1a7b0 0x564242c1ab90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6cfc0aa67a4fffaa server_cookie=cc4021f25bec4999 in_seq=0 out_seq=0 2024-04-17T10:31:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 10 client.7036 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 1 -- 172.21.15.67:0/3400981944 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564242a0a530 con 0x564242c1a7b0 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 10 client.7036 handle_command_reply: tid=0 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 client.7036 shutdown 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bb00a0 msgr2=0x564242c5e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bb00a0 0x564242c5e9b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f184411cea0 tx=0x7f18440eef60 comp rx=0 tx=0).stop 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 shutdown_connections 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x564242c1a7b0 0x564242c1ab90 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x564242bfa2f0 0x564242c19e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x564242bf99d0 0x564242c5eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f1841600640 0 client.7036 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 --2- 172.21.15.67:0/3400981944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x564242bb00a0 0x564242c5e9b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 >> 172.21.15.67:0/3400981944 conn(0x564242bac020 msgr2=0x564242bf8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 shutdown_connections 2024-04-17T10:31:19.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 1 -- 172.21.15.67:0/3400981944 wait complete. 2024-04-17T10:31:19.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.919+0000 7f185e0e8000 20 client.7036 trim_cache size 0 max 16384 2024-04-17T10:31:19.920 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:31:19.920 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-04-17T10:31:19.920 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:31:19.920 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:31:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1830066f60 msgr2=0x7f1830069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1830066f60 0x7f1830069420 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f183c001210 tx=0x7f183c009040 comp rx=0 tx=0).stop 2024-04-17T10:31:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 msgr2=0x7f1854159210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1854159210 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f18440b0040 tx=0x7f1844009750 comp rx=0 tx=0).stop 2024-04-17T10:31:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 shutdown_connections 2024-04-17T10:31:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1830066f60 0x7f1830069420 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18540ca710 0x7f18540caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1854158ce0 0x7f185414ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 --2- 172.21.15.67:0/3379748874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1838003b80 0x7f1854159210 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:19.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 >> 172.21.15.67:0/3379748874 conn(0x7f18540d1630 msgr2=0x7f18540c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:19.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 shutdown_connections 2024-04-17T10:31:19.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:19.923+0000 7f1859600640 1 -- 172.21.15.67:0/3379748874 wait complete. 2024-04-17T10:31:19.935 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-17T10:31:19.936 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-04-17T10:31:19.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 perf dump 2024-04-17T10:31:20.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc18200640 1 Processor -- start 2024-04-17T10:31:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc18200640 1 -- start start 2024-04-17T10:31:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc18200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc1015b8e0 0x7fcc100cce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc18200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100cf570 con 0x7fcc1015bcc0 2024-04-17T10:31:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc18200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc100cf850 con 0x7fcc1015fcf0 2024-04-17T10:31:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc18200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc10160440 con 0x7fcc1015b8e0 2024-04-17T10:31:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc15800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcc1015b8e0 0x7fcc100cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc14e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc1015bcc0 0x7fcc1015f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55294/0 (socket says 172.21.15.67:55294) 2024-04-17T10:31:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.083+0000 7fcc14e00640 1 -- 172.21.15.67:0/2981525580 learned_addr learned my addr 172.21.15.67:0/2981525580 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:20.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc15800640 1 -- 172.21.15.67:0/2981525580 >> v1:172.21.15.86:6789/0 conn(0x7fcc1015fcf0 legacy=0x7fcc1006f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc15800640 1 -- 172.21.15.67:0/2981525580 >> v1:172.21.15.67:6789/0 conn(0x7fcc1015bcc0 legacy=0x7fcc1015f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc15800640 1 -- 172.21.15.67:0/2981525580 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc100cfb30 con 0x7fcc1015b8e0 2024-04-17T10:31:20.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc0fe00640 1 -- 172.21.15.67:0/2981525580 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685064268 0 0) 0x7fcc100cf570 con 0x7fcc1015bcc0 2024-04-17T10:31:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc15800640 1 --2- 172.21.15.67:0/2981525580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100cce20 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcc040097c0 tx=0x7fcc0401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27780bf956d48cc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc0fe00640 1 -- 172.21.15.67:0/2981525580 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040b0070 con 0x7fcc1015b8e0 2024-04-17T10:31:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc0fe00640 1 -- 172.21.15.67:0/2981525580 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc040ab600 con 0x7fcc1015b8e0 2024-04-17T10:31:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc0fe00640 1 -- 172.21.15.67:0/2981525580 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040ab9d0 con 0x7fcc1015b8e0 2024-04-17T10:31:20.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc18200640 1 -- 172.21.15.67:0/2981525580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 msgr2=0x7fcc100cce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc18200640 1 --2- 172.21.15.67:0/2981525580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100cce20 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcc040097c0 tx=0x7fcc0401c460 comp rx=0 tx=0).stop 2024-04-17T10:31:20.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc18200640 1 -- 172.21.15.67:0/2981525580 shutdown_connections 2024-04-17T10:31:20.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc18200640 1 --2- 172.21.15.67:0/2981525580 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100cce20 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.087+0000 7fcc18200640 1 -- 172.21.15.67:0/2981525580 >> 172.21.15.67:0/2981525580 conn(0x7fcc100d1800 msgr2=0x7fcc100d1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 -- 172.21.15.67:0/2981525580 shutdown_connections 2024-04-17T10:31:20.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 -- 172.21.15.67:0/2981525580 wait complete. 2024-04-17T10:31:20.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 Processor -- start 2024-04-17T10:31:20.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 -- start start 2024-04-17T10:31:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc1015bcc0 0x7fcc1015ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc1015fcf0 0x7fcc101666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10169cc0 con 0x7fcc1015b8e0 2024-04-17T10:31:20.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10169b40 con 0x7fcc1015bcc0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10169e40 con 0x7fcc1015fcf0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100ca700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33016/0 (socket says 172.21.15.67:33016) 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 -- 172.21.15.67:0/4144984481 learned_addr learned my addr 172.21.15.67:0/4144984481 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 -- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc1015fcf0 msgr2=0x7fcc101666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc1015fcf0 0x7fcc101666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 -- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc1015bcc0 msgr2=0x7fcc1015ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc14e00640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc1015bcc0 0x7fcc1015ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc1015bcc0 0x7fcc1015ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 -- 172.21.15.67:0/4144984481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc100b53a0 con 0x7fcc1015b8e0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc14e00640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc1015bcc0 0x7fcc1015ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc15800640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100ca700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcc040ae040 tx=0x7fcc0401aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc0d600640 1 -- 172.21.15.67:0/4144984481 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040b0050 con 0x7fcc1015b8e0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc0d600640 1 -- 172.21.15.67:0/4144984481 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc040bc070 con 0x7fcc1015b8e0 2024-04-17T10:31:20.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.091+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc100b5680 con 0x7fcc1015b8e0 2024-04-17T10:31:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc0d600640 1 -- 172.21.15.67:0/4144984481 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040b5a50 con 0x7fcc1015b8e0 2024-04-17T10:31:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc100b5c40 con 0x7fcc1015b8e0 2024-04-17T10:31:20.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc0d600640 1 -- 172.21.15.67:0/4144984481 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcc040ab4a0 con 0x7fcc1015b8e0 2024-04-17T10:31:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc0d600640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbf0066f60 0x7fcbf0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc0d600640 1 -- 172.21.15.67:0/4144984481 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fcc0411fc90 con 0x7fcc1015b8e0 2024-04-17T10:31:20.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc14e00640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbf0066f60 0x7fcbf0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc1b78b000 1 Processor -- start 2024-04-17T10:31:20.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.095+0000 7fcc1b78b000 1 -- start start 2024-04-17T10:31:20.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc14e00640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbf0066f60 0x7fcbf0069420 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcc100ca500 tx=0x7fcbfc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08e8ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e49850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08e4a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d08d2c1f0 con 0x557d08e8ea40 2024-04-17T10:31:20.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d08c92e90 con 0x557d08e8f360 2024-04-17T10:31:20.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x557d08c1af50 con 0x557d08e49d90 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08e8ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08e8ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33028/0 (socket says 172.21.15.67:33028) 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 -- 172.21.15.67:0/2673129115 learned_addr learned my addr 172.21.15.67:0/2673129115 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc14e00640 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e49850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc16200640 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08e4a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 -- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 msgr2=0x557d08e4a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08e4a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 -- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 msgr2=0x557d08e49850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e49850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 -- 172.21.15.67:0/2673129115 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d08c6a8b0 con 0x557d08e8ea40 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc14e00640 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e49850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc16200640 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08e4a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc15800640 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08e8ee20 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcc040f8530 tx=0x7fcc040ec720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3bbacecbdc0ad2f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc02000640 1 -- 172.21.15.67:0/2673129115 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040f0350 con 0x557d08e8ea40 2024-04-17T10:31:20.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc02000640 1 -- 172.21.15.67:0/2673129115 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc040b1040 con 0x557d08e8ea40 2024-04-17T10:31:20.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc02000640 1 -- 172.21.15.67:0/2673129115 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040f44a0 con 0x557d08e8ea40 2024-04-17T10:31:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- 172.21.15.67:0/2673129115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 msgr2=0x557d08e8ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08e8ee20 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcc040f8530 tx=0x7fcc040ec720 comp rx=0 tx=0).stop 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- 172.21.15.67:0/2673129115 shutdown_connections 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08e4a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e49850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/2673129115 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08e8ee20 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- 172.21.15.67:0/2673129115 >> 172.21.15.67:0/2673129115 conn(0x557d08e45d40 msgr2=0x557d08e48130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- 172.21.15.67:0/2673129115 shutdown_connections 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- 172.21.15.67:0/2673129115 wait complete. 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 -- start start 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.099+0000 7fcc1b78b000 1 Processor -- start 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e49d90 0x557d08e961c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08e96700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e8f360 0x557d08e4fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d08c4d540 con 0x557d08e49d90 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d08c8ef30 con 0x557d08e8ea40 2024-04-17T10:31:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x557d08c5f7c0 con 0x557d08e8f360 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 10 client.0 _ll_register_callbacks cb 0x557d08d47180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e49d90 0x557d08e961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e49d90 0x557d08e961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33044/0 (socket says 172.21.15.67:33044) 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 -- 172.21.15.67:0/4121583282 learned_addr learned my addr 172.21.15.67:0/4121583282 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc14e00640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08e96700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc16200640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e8f360 0x557d08e4fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 -- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e8f360 msgr2=0x557d08e4fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e8f360 0x557d08e4fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 -- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 msgr2=0x557d08e96700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08e96700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 -- 172.21.15.67:0/4121583282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d08c1b1a0 con 0x557d08e49d90 2024-04-17T10:31:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc16200640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e8f360 0x557d08e4fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc15800640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e49d90 0x557d08e961c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fcc0401a090 tx=0x7fcc04019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fed95966e39f577 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 1 -- 172.21.15.67:0/4121583282 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc040f0350 con 0x557d08e49d90 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 1 -- 172.21.15.67:0/4121583282 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fcc0410e380 con 0x557d08e49d90 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc040b1040 con 0x557d08e49d90 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 1 -- 172.21.15.67:0/4121583282 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc04115050 con 0x557d08e49d90 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 10 client.7497 fetch_fsmap learned FSMap version 56 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x557d08d15840 con 0x557d08e49d90 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 1 -- 172.21.15.67:0/4121583282 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc040b1040 con 0x557d08e49d90 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 1 -- 172.21.15.67:0/4121583282 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcc040ec9c0 con 0x557d08e49d90 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 10 client.7497 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 10 client.7497 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 20 client.7497 populate_metadata read hostname 'smithi067' 2024-04-17T10:31:20.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e50b70 0x557d08e50f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 4 client.7497 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:31:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d08c2ebe0 con 0x557d08e50b70 2024-04-17T10:31:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc14e00640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e50b70 0x557d08e50f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc14e00640 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e50b70 0x557d08e50f50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba85d0e76497a5ef server_cookie=2b70c5c643ce2d17 in_seq=0 out_seq=0 2024-04-17T10:31:20.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 10 client.7497 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 1 -- 172.21.15.67:0/4121583282 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557d08c2ebe0 con 0x557d08e50b70 2024-04-17T10:31:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcbf6a00640 10 client.7497 handle_command_reply: tid=0 2024-04-17T10:31:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.103+0000 7fcc1b78b000 1 client.7497 shutdown 2024-04-17T10:31:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e49d90 msgr2=0x557d08e961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e49d90 0x557d08e961c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fcc0401a090 tx=0x7fcc04019200 comp rx=0 tx=0).stop 2024-04-17T10:31:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 shutdown_connections 2024-04-17T10:31:20.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e50b70 0x557d08e50f50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e8f360 0x557d08e4fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08e96700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/4121583282 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e49d90 0x557d08e961c0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 >> 172.21.15.67:0/4121583282 conn(0x557d08e45d40 msgr2=0x557d08e489c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcbf6a00640 0 client.7497 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 shutdown_connections 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 1 -- 172.21.15.67:0/4121583282 wait complete. 2024-04-17T10:31:20.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.107+0000 7fcc1b78b000 20 client.7497 trim_cache size 0 max 16384 2024-04-17T10:31:20.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 Processor -- start 2024-04-17T10:31:20.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 -- start start 2024-04-17T10:31:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eeb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08eeb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8f360 0x557d08eebf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10203870 con 0x557d08e8f360 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10202ad0 con 0x557d08e8ea40 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1015ccb0 con 0x557d08e49d90 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc16200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8f360 0x557d08eebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc16200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8f360 0x557d08eebf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33060/0 (socket says 172.21.15.67:33060) 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc16200640 1 -- 172.21.15.67:0/3087252453 learned_addr learned my addr 172.21.15.67:0/3087252453 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc15800640 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eeb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.119+0000 7fcc14e00640 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08eeb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc16200640 1 -- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 msgr2=0x557d08eeb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc16200640 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eeb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc16200640 1 -- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 msgr2=0x557d08eeb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc16200640 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08eeb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc16200640 1 -- 172.21.15.67:0/3087252453 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d08e8f190 con 0x557d08e8f360 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08eeb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:20.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc16200640 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8f360 0x557d08eebf20 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc08007f10 tx=0x7fcc0800e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889a3cc769b1ffd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc01600640 1 -- 172.21.15.67:0/3087252453 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08015030 con 0x557d08e8f360 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc01600640 1 -- 172.21.15.67:0/3087252453 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08011480 con 0x557d08e8f360 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc01600640 1 -- 172.21.15.67:0/3087252453 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08008040 con 0x557d08e8f360 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- 172.21.15.67:0/3087252453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8f360 msgr2=0x557d08eebf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8f360 0x557d08eebf20 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcc08007f10 tx=0x7fcc0800e6d0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- 172.21.15.67:0/3087252453 shutdown_connections 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8f360 0x557d08eebf20 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8ea40 0x557d08eeb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/3087252453 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eeb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- 172.21.15.67:0/3087252453 >> 172.21.15.67:0/3087252453 conn(0x557d08e45d40 msgr2=0x557d08e8d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- 172.21.15.67:0/3087252453 shutdown_connections 2024-04-17T10:31:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- 172.21.15.67:0/3087252453 wait complete. 2024-04-17T10:31:20.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- start start 2024-04-17T10:31:20.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 Processor -- start 2024-04-17T10:31:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08eefc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e93ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1015d2b0 con 0x557d08e8ea40 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d08c92e90 con 0x557d08e8f360 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x557d08c56720 con 0x557d08e49d90 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc1b78b000 10 client.0 _ll_register_callbacks cb 0x557d08c55a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08eefc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08eefc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33070/0 (socket says 172.21.15.67:33070) 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 -- 172.21.15.67:0/862491926 learned_addr learned my addr 172.21.15.67:0/862491926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc15800640 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc16200640 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e93ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 -- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 msgr2=0x557d08eef6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eef6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 -- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 msgr2=0x557d08e93ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e93ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.123+0000 7fcc14e00640 1 -- 172.21.15.67:0/862491926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d08ceca10 con 0x557d08e8ea40 2024-04-17T10:31:20.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc14e00640 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08eefc00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc001e50 tx=0x7fcbfc000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd005c1871576b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 1 -- 172.21.15.67:0/862491926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbfc00de40 con 0x557d08e8ea40 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 1 -- 172.21.15.67:0/862491926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fcbfc02d420 con 0x557d08e8ea40 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc08011480 con 0x557d08e8ea40 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 1 -- 172.21.15.67:0/862491926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbfc034050 con 0x557d08e8ea40 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 10 client.7503 fetch_fsmap learned FSMap version 56 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x557d08c27b00 con 0x557d08e8ea40 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 1 -- 172.21.15.67:0/862491926 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08011480 con 0x557d08e8ea40 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 1 -- 172.21.15.67:0/862491926 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcbfc037020 con 0x557d08e8ea40 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 10 client.7503 fetch_fsmap finished waiting for FSMap version 56 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 10 client.7503 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.a 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 20 client.7503 populate_metadata read hostname 'smithi067' 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e94c70 0x557d08e95080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 4 client.7503 mds_command: new command op to 7206 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 --> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x557d08ca03d0 con 0x557d08e94c70 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc15800640 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e94c70 0x557d08e95080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc15800640 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e94c70 0x557d08e95080 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf26a7a228f7bfcd server_cookie=fae1788b65e00e4d in_seq=0 out_seq=0 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 10 client.7503 ms_handle_connect on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 1 -- 172.21.15.67:0/862491926 <== mds.0 v2:172.21.15.67:6832/3374829114 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17932 (crc 0 0 0) 0x557d08ca03d0 con 0x557d08e94c70 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 10 client.7503 handle_command_reply: tid=0 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 client.7503 shutdown 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 msgr2=0x557d08eefc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08eefc00 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc001e50 tx=0x7fcbfc000ee0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 shutdown_connections 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x557d08e8f360 0x557d08e93ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557d08e8ea40 0x557d08eefc00 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.67:6832/3374829114,v1:172.21.15.67:6834/3374829114] conn(0x557d08e94c70 0x557d08e95080 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 --2- 172.21.15.67:0/862491926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x557d08e49d90 0x557d08eef6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcbf6a00640 0 client.7503 ms_handle_reset on v2:172.21.15.67:6832/3374829114 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 >> 172.21.15.67:0/862491926 conn(0x557d08e45d40 msgr2=0x557d08e8d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 shutdown_connections 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 1 -- 172.21.15.67:0/862491926 wait complete. 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc1b78b000 20 client.7503 trim_cache size 0 max 16384 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-0": { 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 3249, 2024-04-17T10:31:20.127 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 3250, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 1219445, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 15406402, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 17, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 6, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.339169258, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.149044256, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.074591026, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.089641441, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3249, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.757307834, 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000233089 2024-04-17T10:31:20.128 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 10, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000842, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000084 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 0 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-1": { 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 29965, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 50679, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 10775740, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 45926285, 2024-04-17T10:31:20.129 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 8, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 4.637036916, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 2.767371063, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 2.386011889, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.064890639, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 50678, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.544431544, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000030475 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 70536, 2024-04-17T10:31:20.130 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.015276294, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000216 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 120960 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "AsyncMessenger::Worker-2": { 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_messages": 4522, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages": 4515, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_bytes": 1227586, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_bytes": 47245988, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_created_connections": 5, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_active_connections": 4, 2024-04-17T10:31:20.131 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_total_time": 0.480799459, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_send_time": 0.233437308, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_recv_time": 0.102800955, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_running_fast_dispatch_time": 0.116983605, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_messages_queue_lat": { 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 4515, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.594925610, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000131766 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_handle_ack_lat": { 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_recv_encrypted_bytes": 119504, 2024-04-17T10:31:20.132 INFO:teuthology.orchestra.run.smithi067.stdout: "msgr_send_encrypted_bytes": 25808 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "cct": { 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "total_workers": 1, 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "unhealthy_workers": 0 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-MDSRank": { 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 5558, 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 13.893269134, 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002499688 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.133 INFO:teuthology.orchestra.run.smithi067.stdout: "finisher-PurgeQueue": { 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "queue_len": 0, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "complete_latency": { 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3355, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.068942289, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000318611 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": { 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "request": 25520, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "reply": 25520, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "reply_latency": { 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 25520, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 280.396724229, 2024-04-17T10:31:20.134 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.010987332 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_reply": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "forward": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_complete": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_fetch_keys": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_commit": 12, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_split": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_merge": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes": 17785, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_top": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_bottom": 17785, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pin_tail": 0, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_pinned": 17785, 2024-04-17T10:31:20.135 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_expired": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "inodes_with_caps": 2, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": 2, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "subtrees": 2, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse": 47206, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_hit": 46688, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_forward": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_discover": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_dir_fetch": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_remote_ino": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "traverse_lock": 152, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "load_cent": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "q": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "exported": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "exported_inodes": 0, 2024-04-17T10:31:20.136 INFO:teuthology.orchestra.run.smithi067.stdout: "imported": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "imported_inodes": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_dir_fetch": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_backtrace_fetch": 1876, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "openino_peer_discover": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rfiles": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rbytes": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "root_rsnaps": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_fetch": 1876, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_set_tag": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_backtrace_repaired": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_inotable_repaired": 0, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_inodes": 11, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dir_base_inodes": 1, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_dirfrag_rstats": 10, 2024-04-17T10:31:20.137 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_file_inodes": 1865, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_inode_file_caps": 0, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_revoke": 976, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_grant": 4020, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_trunc": 0, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps": 4175, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_caps_dirty": 2000, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_cap_release": 55, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "process_request_cap_release": 13991 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_cache": { 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update": 0, 2024-04-17T10:31:20.138 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_update_receipt": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_try_discover": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_send_discover": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "dir_handle_discover": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_delayed": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "num_strays_enqueuing": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_created": 4075, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_enqueued": 3798, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_reintegrated": 2170, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "strays_migrated": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_processing": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_enqueued": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "num_recovering_prioritized": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_started": 0, 2024-04-17T10:31:20.139 INFO:teuthology.orchestra.run.smithi067.stdout: "recovery_completed": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_enqueue_scrub": 1, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_exportdir": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_flush": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragmentdir": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_fragstats": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ireq_inodestats": 0 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_log": { 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "evlrg": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "evadd": 26463, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "evex": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "evtrm": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "ev": 26463, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "evexg": 0, 2024-04-17T10:31:20.140 INFO:teuthology.orchestra.run.smithi067.stdout: "evexd": 0, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "segadd": 36, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "segex": 0, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "segtrm": 0, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "seg": 36, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "segmjr": 4, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "segexg": 0, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "segexd": 0, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "expos": 4194304, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "wrpos": 89866627, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "rdpos": 0, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "jlat": { 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 3984, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 91.107160240, 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.022868263 2024-04-17T10:31:20.141 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "replayed": 0 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_mem": { 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "ino": 14, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "ino+": 2015, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "ino-": 2001, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir": 13, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir+": 13, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dir-": 0, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dn": 17785, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dn+": 19786, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "dn-": 2001, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "cap": 2, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "cap+": 7882, 2024-04-17T10:31:20.142 INFO:teuthology.orchestra.run.smithi067.stdout: "cap-": 7880, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "rss": 101644, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "heap": 215340 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_server": { 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_client_request": 28867, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "dispatch_server_request": 0, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_request": 25520, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_client_session": 5, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "handle_peer_request": 0, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "req_create_latency": { 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2000, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 1.283897725, 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000641948 2024-04-17T10:31:20.143 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getattr_latency": { 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 28, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.005976960, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000213462 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getfilelock_latency": { 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "req_link_latency": { 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 6000, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 7.109198168, 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.001184866 2024-04-17T10:31:20.144 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookup_latency": { 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 7309, 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 6.317768506, 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000864382 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookuphash_latency": { 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.145 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupino_latency": { 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupname_latency": { 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.146 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupparent_latency": { 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lookupsnap_latency": { 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000050182, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000050182 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "req_lssnap_latency": { 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.147 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mkdir_latency": { 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.003591210, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.003591210 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mknod_latency": { 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "req_mksnap_latency": { 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.007287313, 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.007287313 2024-04-17T10:31:20.148 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "req_open_latency": { 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "req_readdir_latency": { 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.091240339, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.011405042 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rename_latency": { 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 2170, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 249.389218119, 2024-04-17T10:31:20.149 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.114925906 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "req_renamesnap_latency": { 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "req_snapdiff_latency": { 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmdir_latency": { 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.150 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmsnap_latency": { 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.036592336, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.036592336 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "req_rmxattr_latency": { 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setattr_latency": { 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 1, 2024-04-17T10:31:20.151 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000245453, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000245453 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setdirlayout_latency": { 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setfilelock_latency": { 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setlayout_latency": { 2024-04-17T10:31:20.152 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "req_setxattr_latency": { 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "req_symlink_latency": { 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "req_unlink_latency": { 2024-04-17T10:31:20.153 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 8000, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 16.151657918, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.002018957 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_revoke_eviction": 0, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "cap_acquisition_throttle": 0, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "req_getvxattr_latency": { 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_sessions": { 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "session_count": 1, 2024-04-17T10:31:20.154 INFO:teuthology.orchestra.run.smithi067.stdout: "session_add": 1, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "session_remove": 0, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_open": 1, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "sessions_stale": 0, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "total_load": 18459, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "average_load": 18459, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "avg_session_uptime": 78, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "mdthresh_evicted": 0 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "mempool": { 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_bytes": 0, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "bloom_filter_items": 0, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_bytes": 0, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_alloc_items": 0, 2024-04-17T10:31:20.155 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_data_items": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_onode_items": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_meta_items": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_other_items": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_cache_buffer_items": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_extent_items": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_blob_items": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-17T10:31:20.156 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_shared_blob_items": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_inline_bl_items": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_bytes": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_fsck_items": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_bytes": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_txc_items": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_deferred_items": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_bytes": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluestore_writing_items": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_bytes": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_items": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_bytes": 0, 2024-04-17T10:31:20.157 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_reader_items": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_bytes": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "bluefs_file_writer_items": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_bytes": 250468, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_anon_items": 86, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_bytes": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "buffer_meta_items": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_bytes": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_items": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_bytes": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_mapbl_items": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_bytes": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_pglog_items": 0, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_bytes": 5528, 2024-04-17T10:31:20.158 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_items": 78, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_bytes": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_mapping_items": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_bytes": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "pgmap_items": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_bytes": 13391564, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "mds_co_items": 660416, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_bytes": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_1_items": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_bytes": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "unittest_2_items": 0 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "objecter": { 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "op_active": 0, 2024-04-17T10:31:20.159 INFO:teuthology.orchestra.run.smithi067.stdout: "op_laggy": 0, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send": 9932, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "op_send_bytes": 86307145, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "op_resend": 37, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "op_reply": 9932, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "op_latency": { 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 9932, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 40.375269477, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.004065170 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "op_inflight": 0, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "oplen_avg": { 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 9932, 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 10028 2024-04-17T10:31:20.160 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "op": 9932, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "op_r": 1941, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "op_w": 7991, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "op_rmw": 0, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "op_pg": 0, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_stat": 0, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_create": 11, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_read": 64, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_write": 4047, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writefull": 38, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_writesame": 0, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_append": 0, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_zero": 1798, 2024-04-17T10:31:20.161 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_truncate": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_delete": 2032, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_mapext": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_sparse_read": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_clonerange": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_getxattr": 1877, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_setxattr": 22, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_cmpxattr": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_rmxattr": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_resetxattrs": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_call": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_watch": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_notify": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_src_cmpxattr": 0, 2024-04-17T10:31:20.162 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_pgls_filter": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "osdop_other": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_active": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_send": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_resend": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "linger_ping": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_active": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_send": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "poolop_resend": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_active": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_send": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "poolstat_resend": 0, 2024-04-17T10:31:20.163 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_active": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_send": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "statfs_resend": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "command_active": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "command_send": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "command_resend": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "map_epoch": 85, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "map_full": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "map_inc": 3, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_sessions": 8, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_open": 8, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_session_close": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "osd_laggy": 0, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_wr": 113, 2024-04-17T10:31:20.164 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_rd": 0, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_del": 26 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "oft": { 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_objs": 1, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_kv_pairs": 0, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_updates": 0, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "omap_total_removes": 0 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "osd-slow-ops": { 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "slow_ops_count": 0 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "purge_queue": { 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops": 0, 2024-04-17T10:31:20.165 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_ops_high_water": 32, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing": 0, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executing_high_water": 32, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed_ops": 3798, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_executed": 3798, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "pq_item_in_journal": 0 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 37730, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 9821335, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:31:20.166 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 37730, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 37730, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 9821335, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_bytes": { 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 104857600, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:31:20.167 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 9932, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 86705127, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 9932, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 86705127, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.168 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-objecter_ops": { 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 1024, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 0, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 0, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 0, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 9932, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 9932, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 9932, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 9932, 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:31:20.169 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle": { 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 3798, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 397982, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 3798, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:31:20.170 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 63, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 397982, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "throttle-write_buf_throttle-0x5593ebb6aa00": { 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "val": 0, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "max": 3758096384, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "get_started": 0, 2024-04-17T10:31:20.171 INFO:teuthology.orchestra.run.smithi067.stdout: "get": 26463, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "get_sum": 85672323, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_fail": 0, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "get_or_fail_success": 26463, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "take": 0, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "take_sum": 0, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "put": 3984, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "put_sum": 85672323, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "wait": { 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgcount": 0, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "sum": 0.000000000, 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: "avgtime": 0.000000000 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:20.172 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:31:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbf0066f60 msgr2=0x7fcbf0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbf0066f60 0x7fcbf0069420 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcc100ca500 tx=0x7fcbfc009040 comp rx=0 tx=0).stop 2024-04-17T10:31:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 msgr2=0x7fcc100ca700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100ca700 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcc040ae040 tx=0x7fcc0401aca0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 shutdown_connections 2024-04-17T10:31:20.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbf0066f60 0x7fcbf0069420 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc1015fcf0 0x7fcc101666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc1015bcc0 0x7fcc1015ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 --2- 172.21.15.67:0/4144984481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc1015b8e0 0x7fcc100ca700 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 >> 172.21.15.67:0/4144984481 conn(0x7fcc100d1800 msgr2=0x7fcc1006f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 shutdown_connections 2024-04-17T10:31:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.127+0000 7fcc18200640 1 -- 172.21.15.67:0/4144984481 wait complete. 2024-04-17T10:31:20.176 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:31:20.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:31:20.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:31:20.316 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:31:20.317 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:31:20.317 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:20.326 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:31:20.326 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:20.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:20.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.563+0000 7f8c64400640 1 Processor -- start 2024-04-17T10:31:20.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.563+0000 7f8c64400640 1 -- start start 2024-04-17T10:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c64400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06e6e0 0x7f8c5c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c64400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c5c06eac0 0x7f8c5c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c64400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5c15ae40 con 0x7f8c5c143ae0 2024-04-17T10:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c64400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5c0cf430 con 0x7f8c5c06eac0 2024-04-17T10:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c64400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c071440 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c61000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c5c06eac0 0x7f8c5c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c62400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c5c143ae0 0x7f8c5c143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54680/0 (socket says 172.21.15.67:54680) 2024-04-17T10:31:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c62400640 1 -- 172.21.15.67:0/3005650842 learned_addr learned my addr 172.21.15.67:0/3005650842 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c61a00640 1 --2- 172.21.15.67:0/3005650842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06e6e0 0x7f8c5c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c61000640 1 -- 172.21.15.67:0/3005650842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06e6e0 msgr2=0x7f8c5c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c61000640 1 --2- 172.21.15.67:0/3005650842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06e6e0 0x7f8c5c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c61000640 1 -- 172.21.15.67:0/3005650842 >> v1:172.21.15.67:6789/0 conn(0x7f8c5c143ae0 legacy=0x7f8c5c143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c61000640 1 -- 172.21.15.67:0/3005650842 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c071720 con 0x7f8c5c06eac0 2024-04-17T10:31:20.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.567+0000 7f8c5be00640 1 -- 172.21.15.67:0/3005650842 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756941881 0 0) 0x7f8c5c15ae40 con 0x7f8c5c143ae0 2024-04-17T10:31:20.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c61000640 1 --2- 172.21.15.67:0/3005650842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06eac0 0x7f8c5c0ceef0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8c480097c0 tx=0x7f8c4801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25890985f9ba04e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c5be00640 1 -- 172.21.15.67:0/3005650842 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c480b1070 con 0x7f8c5c06eac0 2024-04-17T10:31:20.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c5be00640 1 -- 172.21.15.67:0/3005650842 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c480a8050 con 0x7f8c5c06eac0 2024-04-17T10:31:20.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c5be00640 1 -- 172.21.15.67:0/3005650842 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c480ac860 con 0x7f8c5c06eac0 2024-04-17T10:31:20.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- 172.21.15.67:0/3005650842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06eac0 msgr2=0x7f8c5c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 --2- 172.21.15.67:0/3005650842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06eac0 0x7f8c5c0ceef0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f8c480097c0 tx=0x7f8c4801c3e0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- 172.21.15.67:0/3005650842 shutdown_connections 2024-04-17T10:31:20.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 --2- 172.21.15.67:0/3005650842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06eac0 0x7f8c5c0ceef0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 --2- 172.21.15.67:0/3005650842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06e6e0 0x7f8c5c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- 172.21.15.67:0/3005650842 >> 172.21.15.67:0/3005650842 conn(0x7f8c5c06ac90 msgr2=0x7f8c5c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- 172.21.15.67:0/3005650842 shutdown_connections 2024-04-17T10:31:20.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- 172.21.15.67:0/3005650842 wait complete. 2024-04-17T10:31:20.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 Processor -- start 2024-04-17T10:31:20.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- start start 2024-04-17T10:31:20.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06e6e0 0x7f8c5c13d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06eac0 0x7f8c5c0ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c143ae0 0x7f8c5c0c6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c0b3a40 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c0b38c0 con 0x7f8c5c143ae0 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.571+0000 7f8c64400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c5c0b3bc0 con 0x7f8c5c06eac0 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06e6e0 0x7f8c5c13d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06e6e0 0x7f8c5c13d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54106/0 (socket says 172.21.15.67:54106) 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 -- 172.21.15.67:0/1783142295 learned_addr learned my addr 172.21.15.67:0/1783142295 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61000640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06eac0 0x7f8c5c0ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c62400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c143ae0 0x7f8c5c0c6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 -- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06eac0 msgr2=0x7f8c5c0ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06eac0 0x7f8c5c0ac810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 -- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c143ae0 msgr2=0x7f8c5c0c6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c143ae0 0x7f8c5c0c6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 -- 172.21.15.67:0/1783142295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c13d640 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c62400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c143ae0 0x7f8c5c0c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c61a00640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06e6e0 0x7f8c5c13d100 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000e930 tx=0x7f8c5000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c59600640 1 -- 172.21.15.67:0/1783142295 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c50017070 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c59600640 1 -- 172.21.15.67:0/1783142295 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5000ca00 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c59600640 1 -- 172.21.15.67:0/1783142295 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5000cdb0 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5c0ae370 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5c0ad5b0 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5c072d50 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.575+0000 7f8c59600640 1 -- 172.21.15.67:0/1783142295 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8c5001a070 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.579+0000 7f8c59600640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c3c066f60 0x7f8c3c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.579+0000 7f8c59600640 1 -- 172.21.15.67:0/1783142295 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f8c500852b0 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.579+0000 7f8c61000640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c3c066f60 0x7f8c3c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.579+0000 7f8c61000640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c3c066f60 0x7f8c3c069420 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8c480ae000 tx=0x7f8c48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.583+0000 7f8c59600640 1 -- 172.21.15.67:0/1783142295 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8c50056130 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c5c0cd730 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c59600640 1 -- 172.21.15.67:0/1783142295 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f8c50055a30 con 0x7f8c5c06e6e0 2024-04-17T10:31:20.742 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:20.742 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:08.910824+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:29:46.421958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-17T10:29:46.986879+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":84,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":12,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c3c066f60 msgr2=0x7f8c3c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c3c066f60 0x7f8c3c069420 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8c480ae000 tx=0x7f8c48008040 comp rx=0 tx=0).stop 2024-04-17T10:31:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06e6e0 msgr2=0x7f8c5c13d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06e6e0 0x7f8c5c13d100 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f8c5000e930 tx=0x7f8c5000edf0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 shutdown_connections 2024-04-17T10:31:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c3c066f60 0x7f8c3c069420 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c5c143ae0 0x7f8c5c0c6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c5c06eac0 0x7f8c5c0ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 --2- 172.21.15.67:0/1783142295 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c5c06e6e0 0x7f8c5c13d100 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 >> 172.21.15.67:0/1783142295 conn(0x7f8c5c06ac90 msgr2=0x7f8c5c13db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 shutdown_connections 2024-04-17T10:31:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.747+0000 7f8c64400640 1 -- 172.21.15.67:0/1783142295 wait complete. 2024-04-17T10:31:20.764 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:29:46.421958+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:29:46.986879+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 84, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:31:20.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:31:20.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.911+0000 7fc99d600640 1 Processor -- start 2024-04-17T10:31:20.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.911+0000 7fc99d600640 1 -- start start 2024-04-17T10:31:20.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc99d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 0x7fc9980cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc99d600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9980b4990 con 0x7fc99806e980 2024-04-17T10:31:20.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc99d600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99815ae90 con 0x7fc9980cce60 2024-04-17T10:31:20.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc99d600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9980cd2c0 con 0x7fc9980aef60 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc997400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc99806e980 0x7fc9980b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54692/0 (socket says 172.21.15.67:54692) 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc997400640 1 -- 172.21.15.67:0/3125534016 learned_addr learned my addr 172.21.15.67:0/3125534016 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996a00640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 0x7fc9980cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025260898 0 0) 0x7fc9900014a0 con 0x7fc99806e980 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97c003530 con 0x7fc99806e980 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154373095 0 0) 0x7fc99815ae90 con 0x7fc9980cce60 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9900014a0 con 0x7fc9980cce60 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1252238935 0 0) 0x7fc97c003530 con 0x7fc99806e980 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 msgr2=0x7fc9980cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 0x7fc9980cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 >> v1:172.21.15.86:6789/0 conn(0x7fc9980cce60 legacy=0x7fc99806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99806ee80 con 0x7fc99806e980 2024-04-17T10:31:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fc990001bd0 con 0x7fc99806e980 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 >> v1:172.21.15.67:6789/0 conn(0x7fc99806e980 legacy=0x7fc9980b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 0x7fc97c003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97c004100 0x7fc97c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004d50 0x7fc97c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c00dd70 con 0x7fc97c004100 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c00def0 con 0x7fc97c004d50 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc97c00e0f0 con 0x7fc9980aef60 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc98e000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97c004100 0x7fc97c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996a00640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 0x7fc97c003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc997400640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004d50 0x7fc97c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc98e000640 1 -- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 msgr2=0x7fc97c003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc98e000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 0x7fc97c003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc98e000640 1 -- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004d50 msgr2=0x7fc97c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc98e000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004d50 0x7fc97c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc98e000640 1 -- 172.21.15.67:0/3125534016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c003530 con 0x7fc97c004100 2024-04-17T10:31:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.915+0000 7fc996a00640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9980aef60 0x7fc97c003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc98e000640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97c004100 0x7fc97c004810 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc9800018d0 tx=0x7fc98001c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd950eeb53ed414 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980001520 con 0x7fc97c004100 2024-04-17T10:31:20.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9800052f0 con 0x7fc97c004100 2024-04-17T10:31:20.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc996000640 1 -- 172.21.15.67:0/3125534016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980003840 con 0x7fc97c004100 2024-04-17T10:31:20.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc99d600640 1 -- 172.21.15.67:0/3125534016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97c004100 msgr2=0x7fc97c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc99d600640 1 --2- 172.21.15.67:0/3125534016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc97c004100 0x7fc97c004810 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc9800018d0 tx=0x7fc98001c580 comp rx=0 tx=0).stop 2024-04-17T10:31:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc997400640 1 -- 172.21.15.67:0/3125534016 reap_dead start 2024-04-17T10:31:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc99d600640 1 -- 172.21.15.67:0/3125534016 shutdown_connections 2024-04-17T10:31:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc99d600640 1 -- 172.21.15.67:0/3125534016 >> 172.21.15.67:0/3125534016 conn(0x7fc9980b1740 msgr2=0x7fc9980b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc99d600640 1 -- 172.21.15.67:0/3125534016 shutdown_connections 2024-04-17T10:31:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.919+0000 7fc99d600640 1 -- 172.21.15.67:0/3125534016 wait complete. 2024-04-17T10:31:20.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 Processor -- start 2024-04-17T10:31:20.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 -- start start 2024-04-17T10:31:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004100 0x7fc998144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99815de30 0x7fc99815e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9981fa1f0 0x7fc9981fa650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99806d160 con 0x7fc99815de30 2024-04-17T10:31:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9980af340 con 0x7fc97c004100 2024-04-17T10:31:20.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99806d2e0 con 0x7fc9981fa1f0 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99815de30 0x7fc99815e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99815de30 0x7fc99815e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54110/0 (socket says 172.21.15.67:54110) 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 -- 172.21.15.67:0/1315852631 learned_addr learned my addr 172.21.15.67:0/1315852631 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc996a00640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004100 0x7fc998144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc997400640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9981fa1f0 0x7fc9981fa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 -- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9981fa1f0 msgr2=0x7fc9981fa650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9981fa1f0 0x7fc9981fa650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 -- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004100 msgr2=0x7fc998144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004100 0x7fc998144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 -- 172.21.15.67:0/1315852631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc998145830 con 0x7fc99815de30 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98e000640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99815de30 0x7fc99815e2b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fc980005040 tx=0x7fc97c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98f400640 1 -- 172.21.15.67:0/1315852631 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9800bc070 con 0x7fc99815de30 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98f400640 1 -- 172.21.15.67:0/1315852631 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9800054a0 con 0x7fc99815de30 2024-04-17T10:31:20.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc98f400640 1 -- 172.21.15.67:0/1315852631 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980001500 con 0x7fc99815de30 2024-04-17T10:31:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc998145b10 con 0x7fc99815de30 2024-04-17T10:31:20.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.923+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9981431e0 con 0x7fc99815de30 2024-04-17T10:31:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.927+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99804ef30 con 0x7fc99815de30 2024-04-17T10:31:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.927+0000 7fc98f400640 1 -- 172.21.15.67:0/1315852631 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc9800bf070 con 0x7fc99815de30 2024-04-17T10:31:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.927+0000 7fc98f400640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc974066f60 0x7fc974069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.927+0000 7fc98f400640 1 -- 172.21.15.67:0/1315852631 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fc98011e3b0 con 0x7fc99815de30 2024-04-17T10:31:20.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.927+0000 7fc996a00640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc974066f60 0x7fc974069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:20.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.927+0000 7fc996a00640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc974066f60 0x7fc974069420 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc988009ca0 tx=0x7fc988009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:20.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:20.931+0000 7fc98f400640 1 -- 172.21.15.67:0/1315852631 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc980016290 con 0x7fc99815de30 2024-04-17T10:31:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc99806e120 con 0x7fc99815de30 2024-04-17T10:31:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc98f400640 1 -- 172.21.15.67:0/1315852631 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9800eea90 con 0x7fc99815de30 2024-04-17T10:31:21.089 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc974066f60 msgr2=0x7fc974069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc974066f60 0x7fc974069420 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc988009ca0 tx=0x7fc988009190 comp rx=0 tx=0).stop 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99815de30 msgr2=0x7fc99815e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99815de30 0x7fc99815e2b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fc980005040 tx=0x7fc97c005840 comp rx=0 tx=0).stop 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 shutdown_connections 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc974066f60 0x7fc974069420 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9981fa1f0 0x7fc9981fa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc99815de30 0x7fc99815e2b0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 --2- 172.21.15.67:0/1315852631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc97c004100 0x7fc998144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 >> 172.21.15.67:0/1315852631 conn(0x7fc9980b1740 msgr2=0x7fc99806d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 shutdown_connections 2024-04-17T10:31:21.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.095+0000 7fc99d600640 1 -- 172.21.15.67:0/1315852631 wait complete. 2024-04-17T10:31:21.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:31:21.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe081200640 1 Processor -- start 2024-04-17T10:31:21.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe081200640 1 -- start start 2024-04-17T10:31:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe081200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe07c161f90 0x7fe07c0cff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe081200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe07c09adf0 0x7fe07c0d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe081200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe07c15e140 con 0x7fe07c15dd60 2024-04-17T10:31:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe081200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07c1626d0 con 0x7fe07c161f90 2024-04-17T10:31:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe081200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe07c099d50 con 0x7fe07c09adf0 2024-04-17T10:31:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe079e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe07c15dd60 0x7fe07c161880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54702/0 (socket says 172.21.15.67:54702) 2024-04-17T10:31:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe079e00640 1 -- 172.21.15.67:0/30952913 learned_addr learned my addr 172.21.15.67:0/30952913 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe071400640 1 --2- 172.21.15.67:0/30952913 >> v2:172.21.15.67:3300/0 conn(0x7fe07c161f90 0x7fe07c0cff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe07a800640 1 --2- 172.21.15.67:0/30952913 >> v2:172.21.15.86:3300/0 conn(0x7fe07c09adf0 0x7fe07c0d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe071400640 1 -- 172.21.15.67:0/30952913 >> v2:172.21.15.86:3300/0 conn(0x7fe07c09adf0 msgr2=0x7fe07c0d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe071400640 1 --2- 172.21.15.67:0/30952913 >> v2:172.21.15.86:3300/0 conn(0x7fe07c09adf0 0x7fe07c0d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe071400640 1 -- 172.21.15.67:0/30952913 >> v1:172.21.15.67:6789/0 conn(0x7fe07c15dd60 legacy=0x7fe07c161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe071400640 1 -- 172.21.15.67:0/30952913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c09a030 con 0x7fe07c161f90 2024-04-17T10:31:21.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.259+0000 7fe079400640 1 -- 172.21.15.67:0/30952913 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489282249 0 0) 0x7fe07c15e140 con 0x7fe07c15dd60 2024-04-17T10:31:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe071400640 1 --2- 172.21.15.67:0/30952913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c161f90 0x7fe07c0cff90 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0c8a10 tx=0x7fe06401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0e89926a9c2a8b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe079400640 1 -- 172.21.15.67:0/30952913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640b0070 con 0x7fe07c161f90 2024-04-17T10:31:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe079400640 1 -- 172.21.15.67:0/30952913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0640ab600 con 0x7fe07c161f90 2024-04-17T10:31:21.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe079400640 1 -- 172.21.15.67:0/30952913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640ab9d0 con 0x7fe07c161f90 2024-04-17T10:31:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 -- 172.21.15.67:0/30952913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c161f90 msgr2=0x7fe07c0cff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 --2- 172.21.15.67:0/30952913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c161f90 0x7fe07c0cff90 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe07c0c8a10 tx=0x7fe06401c480 comp rx=0 tx=0).stop 2024-04-17T10:31:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 -- 172.21.15.67:0/30952913 shutdown_connections 2024-04-17T10:31:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 --2- 172.21.15.67:0/30952913 >> v2:172.21.15.86:3300/0 conn(0x7fe07c09adf0 0x7fe07c0d04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 --2- 172.21.15.67:0/30952913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c161f90 0x7fe07c0cff90 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 -- 172.21.15.67:0/30952913 >> 172.21.15.67:0/30952913 conn(0x7fe07c0b34a0 msgr2=0x7fe07c0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 -- 172.21.15.67:0/30952913 shutdown_connections 2024-04-17T10:31:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.263+0000 7fe081200640 1 -- 172.21.15.67:0/30952913 wait complete. 2024-04-17T10:31:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 Processor -- start 2024-04-17T10:31:21.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 -- start start 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe07c09adf0 0x7fe07c15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15dd60 0x7fe07c0b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07c161f90 0x7fe07c0b70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c06c730 con 0x7fe07c15dd60 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c06c5b0 con 0x7fe07c09adf0 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07c06c8b0 con 0x7fe07c161f90 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15dd60 0x7fe07c0b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15dd60 0x7fe07c0b6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54132/0 (socket says 172.21.15.67:54132) 2024-04-17T10:31:21.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 -- 172.21.15.67:0/620685457 learned_addr learned my addr 172.21.15.67:0/620685457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe079e00640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe07c09adf0 0x7fe07c15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe07a800640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07c161f90 0x7fe07c0b70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 -- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07c161f90 msgr2=0x7fe07c0b70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07c161f90 0x7fe07c0b70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 -- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe07c09adf0 msgr2=0x7fe07c15a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe07c09adf0 0x7fe07c15a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 -- 172.21.15.67:0/620685457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c1f7460 con 0x7fe07c15dd60 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe079e00640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe07c09adf0 0x7fe07c15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe071400640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15dd60 0x7fe07c0b6bb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe06401a0b0 tx=0x7fe064019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe072a00640 1 -- 172.21.15.67:0/620685457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640b0040 con 0x7fe07c15dd60 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe072a00640 1 -- 172.21.15.67:0/620685457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0640be070 con 0x7fe07c15dd60 2024-04-17T10:31:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe072a00640 1 -- 172.21.15.67:0/620685457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064006e20 con 0x7fe07c15dd60 2024-04-17T10:31:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07c1f7740 con 0x7fe07c15dd60 2024-04-17T10:31:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07c1f7d00 con 0x7fe07c15dd60 2024-04-17T10:31:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.267+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07c04ef30 con 0x7fe07c15dd60 2024-04-17T10:31:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.271+0000 7fe072a00640 1 -- 172.21.15.67:0/620685457 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe0640b3040 con 0x7fe07c15dd60 2024-04-17T10:31:21.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.271+0000 7fe072a00640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe058066f90 0x7fe058069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.271+0000 7fe072a00640 1 -- 172.21.15.67:0/620685457 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fe06411f2c0 con 0x7fe07c15dd60 2024-04-17T10:31:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.271+0000 7fe079e00640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe058066f90 0x7fe058069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.271+0000 7fe079e00640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe058066f90 0x7fe058069450 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000f70 tx=0x7fe06c007c40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.275+0000 7fe072a00640 1 -- 172.21.15.67:0/620685457 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe064016070 con 0x7fe07c15dd60 2024-04-17T10:31:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe07c0b42d0 con 0x7fe07c15dd60 2024-04-17T10:31:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe072a00640 1 -- 172.21.15.67:0/620685457 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0640f0080 con 0x7fe07c15dd60 2024-04-17T10:31:21.434 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe058066f90 msgr2=0x7fe058069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe058066f90 0x7fe058069450 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000f70 tx=0x7fe06c007c40 comp rx=0 tx=0).stop 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15dd60 msgr2=0x7fe07c0b6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15dd60 0x7fe07c0b6bb0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe06401a0b0 tx=0x7fe064019220 comp rx=0 tx=0).stop 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 shutdown_connections 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe058066f90 0x7fe058069450 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe07c161f90 0x7fe07c0b70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe07c15dd60 0x7fe07c0b6bb0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 --2- 172.21.15.67:0/620685457 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe07c09adf0 0x7fe07c15a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 >> 172.21.15.67:0/620685457 conn(0x7fe07c0b34a0 msgr2=0x7fe07c0cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:21.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 shutdown_connections 2024-04-17T10:31:21.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.439+0000 7fe081200640 1 -- 172.21.15.67:0/620685457 wait complete. 2024-04-17T10:31:21.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:31:21.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3c05400640 1 Processor -- start 2024-04-17T10:31:21.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3c05400640 1 -- start start 2024-04-17T10:31:21.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3c05400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c00071af0 0x7f3c000ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3c05400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0015cb30 con 0x7f3c00159440 2024-04-17T10:31:21.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3c05400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c000cd610 con 0x7f3c0015be70 2024-04-17T10:31:21.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3c05400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c000a8650 con 0x7f3c00071af0 2024-04-17T10:31:21.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3bfe800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3c00159440 0x7f3c000713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54704/0 (socket says 172.21.15.67:54704) 2024-04-17T10:31:21.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.603+0000 7f3bfe800640 1 -- 172.21.15.67:0/2116508201 learned_addr learned my addr 172.21.15.67:0/2116508201 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bf7e00640 1 --2- 172.21.15.67:0/2116508201 >> v2:172.21.15.86:3300/0 conn(0x7f3c00071af0 0x7f3c000ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bfde00640 1 -- 172.21.15.67:0/2116508201 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290717503 0 0) 0x7f3c0015cb30 con 0x7f3c00159440 2024-04-17T10:31:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bfde00640 1 -- 172.21.15.67:0/2116508201 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be4003530 con 0x7f3c00159440 2024-04-17T10:31:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bf7e00640 1 -- 172.21.15.67:0/2116508201 >> v1:172.21.15.86:6789/0 conn(0x7f3c0015be70 legacy=0x7f3c0015c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bfde00640 1 -- 172.21.15.67:0/2116508201 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52064415 0 0) 0x7f3c000cd610 con 0x7f3c0015be70 2024-04-17T10:31:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bf7e00640 1 -- 172.21.15.67:0/2116508201 >> v1:172.21.15.67:6789/0 conn(0x7f3c00159440 legacy=0x7f3c000713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bf7e00640 1 -- 172.21.15.67:0/2116508201 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c000cd8f0 con 0x7f3c00071af0 2024-04-17T10:31:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bf7e00640 1 --2- 172.21.15.67:0/2116508201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c00071af0 0x7f3c000ce0f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3be80097c0 tx=0x7f3be801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee892a7024615b30 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bfde00640 1 -- 172.21.15.67:0/2116508201 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be80b1070 con 0x7f3c00071af0 2024-04-17T10:31:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bfde00640 1 -- 172.21.15.67:0/2116508201 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be80ac410 con 0x7f3c00071af0 2024-04-17T10:31:21.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.607+0000 7f3bfde00640 1 -- 172.21.15.67:0/2116508201 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be80ac7c0 con 0x7f3c00071af0 2024-04-17T10:31:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- 172.21.15.67:0/2116508201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c00071af0 msgr2=0x7f3c000ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 --2- 172.21.15.67:0/2116508201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c00071af0 0x7f3c000ce0f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3be80097c0 tx=0x7f3be801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- 172.21.15.67:0/2116508201 shutdown_connections 2024-04-17T10:31:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 --2- 172.21.15.67:0/2116508201 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c00071af0 0x7f3c000ce0f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- 172.21.15.67:0/2116508201 >> 172.21.15.67:0/2116508201 conn(0x7f3c000b1740 msgr2=0x7f3c000b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- 172.21.15.67:0/2116508201 shutdown_connections 2024-04-17T10:31:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- 172.21.15.67:0/2116508201 wait complete. 2024-04-17T10:31:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 Processor -- start 2024-04-17T10:31:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- start start 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c00071af0 0x7f3c0014b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c00159440 0x7f3c000ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0015be70 0x7f3c0014b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0015dc10 con 0x7f3c00071af0 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0015da90 con 0x7f3c0015be70 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3c05400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0015dd90 con 0x7f3c00159440 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c00071af0 0x7f3c0014b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c00071af0 0x7f3c0014b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54138/0 (socket says 172.21.15.67:54138) 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 -- 172.21.15.67:0/1606809224 learned_addr learned my addr 172.21.15.67:0/1606809224 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:21.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bff200640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0015be70 0x7f3c0014b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bf7e00640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c00159440 0x7f3c000ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 -- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c00159440 msgr2=0x7f3c000ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c00159440 0x7f3c000ae1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 -- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0015be70 msgr2=0x7f3c0014b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0015be70 0x7f3c0014b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bfe800640 1 -- 172.21.15.67:0/1606809224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c00156330 con 0x7f3c00071af0 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bf7e00640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c00159440 0x7f3c000ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.611+0000 7f3bff200640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0015be70 0x7f3c0014b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bfe800640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c00071af0 0x7f3c0014b840 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c3c0 tx=0x7f3bf000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bf6a00640 1 -- 172.21.15.67:0/1606809224 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0017070 con 0x7f3c00071af0 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bf6a00640 1 -- 172.21.15.67:0/1606809224 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0005cf0 con 0x7f3c00071af0 2024-04-17T10:31:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bf6a00640 1 -- 172.21.15.67:0/1606809224 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00072f0 con 0x7f3c00071af0 2024-04-17T10:31:21.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c001520f0 con 0x7f3c00071af0 2024-04-17T10:31:21.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c001526b0 con 0x7f3c00071af0 2024-04-17T10:31:21.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0004ef30 con 0x7f3c00071af0 2024-04-17T10:31:21.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bf6a00640 1 -- 172.21.15.67:0/1606809224 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3bf001a070 con 0x7f3c00071af0 2024-04-17T10:31:21.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bf6a00640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3bdc066f60 0x7f3bdc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bf6a00640 1 -- 172.21.15.67:0/1606809224 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3bf0084680 con 0x7f3c00071af0 2024-04-17T10:31:21.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.615+0000 7f3bf7e00640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3bdc066f60 0x7f3bdc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.619+0000 7f3bf7e00640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3bdc066f60 0x7f3bdc069420 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3be80ae000 tx=0x7f3be80a6040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.619+0000 7f3bf6a00640 1 -- 172.21.15.67:0/1606809224 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3bf0055440 con 0x7f3c00071af0 2024-04-17T10:31:21.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.779+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c001528f0 con 0x7f3c00071af0 2024-04-17T10:31:21.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.779+0000 7f3bf6a00640 1 -- 172.21.15.67:0/1606809224 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3bf0054d40 con 0x7f3c00071af0 2024-04-17T10:31:21.776 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:31:21.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3bdc066f60 msgr2=0x7f3bdc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3bdc066f60 0x7f3bdc069420 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3be80ae000 tx=0x7f3be80a6040 comp rx=0 tx=0).stop 2024-04-17T10:31:21.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c00071af0 msgr2=0x7f3c0014b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c00071af0 0x7f3c0014b840 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c3c0 tx=0x7f3bf000c880 comp rx=0 tx=0).stop 2024-04-17T10:31:21.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 shutdown_connections 2024-04-17T10:31:21.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3bdc066f60 0x7f3bdc069420 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0015be70 0x7f3c0014b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c00159440 0x7f3c000ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 --2- 172.21.15.67:0/1606809224 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c00071af0 0x7f3c0014b840 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 >> 172.21.15.67:0/1606809224 conn(0x7f3c000b1740 msgr2=0x7f3c0015c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 shutdown_connections 2024-04-17T10:31:21.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.783+0000 7f3c05400640 1 -- 172.21.15.67:0/1606809224 wait complete. 2024-04-17T10:31:21.798 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:31:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.943+0000 7f065c000640 1 Processor -- start 2024-04-17T10:31:21.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f065c000640 1 -- start start 2024-04-17T10:31:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f065c000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f065406eac0 0x7f0654143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f065c000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065415ae40 con 0x7f065406e6e0 2024-04-17T10:31:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f065c000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0654071440 con 0x7f06540cf0c0 2024-04-17T10:31:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f065c000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0654071720 con 0x7f065406eac0 2024-04-17T10:31:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f0659600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f065406e6e0 0x7f06540ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54706/0 (socket says 172.21.15.67:54706) 2024-04-17T10:31:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f0659600640 1 -- 172.21.15.67:0/3245748111 learned_addr learned my addr 172.21.15.67:0/3245748111 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f0658c00640 1 --2- 172.21.15.67:0/3245748111 >> v2:172.21.15.67:3300/0 conn(0x7f065406eac0 0x7f0654143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f0658c00640 1 -- 172.21.15.67:0/3245748111 >> v1:172.21.15.86:6789/0 conn(0x7f06540cf0c0 legacy=0x7f0654144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f0653e00640 1 -- 172.21.15.67:0/3245748111 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575018721 0 0) 0x7f065415ae40 con 0x7f065406e6e0 2024-04-17T10:31:21.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f0658c00640 1 -- 172.21.15.67:0/3245748111 >> v1:172.21.15.67:6789/0 conn(0x7f065406e6e0 legacy=0x7f06540ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.947+0000 7f0658c00640 1 -- 172.21.15.67:0/3245748111 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0654071a00 con 0x7f065406eac0 2024-04-17T10:31:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f0658c00640 1 --2- 172.21.15.67:0/3245748111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f0654143ae0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f06400097c0 tx=0x7f064001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae30f1bd65dc9925 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f0653e00640 1 -- 172.21.15.67:0/3245748111 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06400ab600 con 0x7f065406eac0 2024-04-17T10:31:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f0653e00640 1 -- 172.21.15.67:0/3245748111 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06400b0070 con 0x7f065406eac0 2024-04-17T10:31:21.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f0653e00640 1 -- 172.21.15.67:0/3245748111 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06400ab9c0 con 0x7f065406eac0 2024-04-17T10:31:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 -- 172.21.15.67:0/3245748111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 msgr2=0x7f0654143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 --2- 172.21.15.67:0/3245748111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f0654143ae0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f06400097c0 tx=0x7f064001c3e0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 -- 172.21.15.67:0/3245748111 shutdown_connections 2024-04-17T10:31:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 --2- 172.21.15.67:0/3245748111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f0654143ae0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 -- 172.21.15.67:0/3245748111 >> 172.21.15.67:0/3245748111 conn(0x7f065406ac90 msgr2=0x7f065406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:21.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 -- 172.21.15.67:0/3245748111 shutdown_connections 2024-04-17T10:31:21.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 -- 172.21.15.67:0/3245748111 wait complete. 2024-04-17T10:31:21.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 Processor -- start 2024-04-17T10:31:21.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.951+0000 7f065c000640 1 -- start start 2024-04-17T10:31:21.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f065406e6e0 0x7f0654151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f06541524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540cf0c0 0x7f0654157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065406d4f0 con 0x7f065406eac0 2024-04-17T10:31:21.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065406d370 con 0x7f06540cf0c0 2024-04-17T10:31:21.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065406d670 con 0x7f065406e6e0 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f06541524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f06541524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54150/0 (socket says 172.21.15.67:54150) 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 -- 172.21.15.67:0/508166258 learned_addr learned my addr 172.21.15.67:0/508166258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065a000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540cf0c0 0x7f0654157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0659600640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f065406e6e0 0x7f0654151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 -- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f065406e6e0 msgr2=0x7f0654151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f065406e6e0 0x7f0654151f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 -- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540cf0c0 msgr2=0x7f0654157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540cf0c0 0x7f0654157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 -- 172.21.15.67:0/508166258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065413daa0 con 0x7f065406eac0 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0659600640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f065406e6e0 0x7f0654151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065a000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540cf0c0 0x7f0654157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0658c00640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f06541524b0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f06400ad040 tx=0x7f064001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0651600640 1 -- 172.21.15.67:0/508166258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06400bf610 con 0x7f065406eac0 2024-04-17T10:31:21.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0651600640 1 -- 172.21.15.67:0/508166258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06400bf7b0 con 0x7f065406eac0 2024-04-17T10:31:21.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0651600640 1 -- 172.21.15.67:0/508166258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06400bfba0 con 0x7f065406eac0 2024-04-17T10:31:21.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0654150090 con 0x7f065406eac0 2024-04-17T10:31:21.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0654150650 con 0x7f065406eac0 2024-04-17T10:31:21.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0654072d50 con 0x7f065406eac0 2024-04-17T10:31:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.955+0000 7f0651600640 1 -- 172.21.15.67:0/508166258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f06400bd020 con 0x7f065406eac0 2024-04-17T10:31:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.959+0000 7f0651600640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0634066f60 0x7f0634069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.959+0000 7f0651600640 1 -- 172.21.15.67:0/508166258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f064011f890 con 0x7f065406eac0 2024-04-17T10:31:21.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.959+0000 7f0659600640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0634066f60 0x7f0634069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:21.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.959+0000 7f0659600640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0634066f60 0x7f0634069420 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0648000f70 tx=0x7f0648009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:21.963+0000 7f0651600640 1 -- 172.21.15.67:0/508166258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f065413daa0 con 0x7f065406eac0 2024-04-17T10:31:22.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0654074e30 con 0x7f065406eac0 2024-04-17T10:31:22.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f0651600640 1 -- 172.21.15.67:0/508166258 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v85) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0640016090 con 0x7f065406eac0 2024-04-17T10:31:22.130 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:31:22.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0634066f60 msgr2=0x7f0634069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0634066f60 0x7f0634069420 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f0648000f70 tx=0x7f0648009040 comp rx=0 tx=0).stop 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 msgr2=0x7f06541524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f06541524b0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f06400ad040 tx=0x7f064001ac20 comp rx=0 tx=0).stop 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 shutdown_connections 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0634066f60 0x7f0634069420 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06540cf0c0 0x7f0654157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065406eac0 0x7f06541524b0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 --2- 172.21.15.67:0/508166258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f065406e6e0 0x7f0654151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 >> 172.21.15.67:0/508166258 conn(0x7f065406ac90 msgr2=0x7f06540b0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 shutdown_connections 2024-04-17T10:31:22.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.135+0000 7f065c000640 1 -- 172.21.15.67:0/508166258 wait complete. 2024-04-17T10:31:22.151 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:31:22.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:22.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:31:22.190 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:31:22.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:22.191 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:31:22.245 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:31:22.245 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:22.255 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:31:22.255 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:31:22.255 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:22.300 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:31:22.300 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:31:22.300 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:22.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:31:22.309 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:31:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.487+0000 7f9e2ca00640 1 Processor -- start 2024-04-17T10:31:22.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.487+0000 7f9e2ca00640 1 -- start start 2024-04-17T10:31:22.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e2ca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e28071770 0x7f9e280cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e2ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2806e2f0 con 0x7f9e28145040 2024-04-17T10:31:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e2ca00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2806d6e0 con 0x7f9e28071310 2024-04-17T10:31:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e2ca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280a86c0 con 0x7f9e28071770 2024-04-17T10:31:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e25e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9e28145040 0x7f9e2806e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54722/0 (socket says 172.21.15.67:54722) 2024-04-17T10:31:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e25e00640 1 -- 172.21.15.67:0/3803950503 learned_addr learned my addr 172.21.15.67:0/3803950503 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:22.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e26800640 1 --2- 172.21.15.67:0/3803950503 >> v2:172.21.15.67:3300/0 conn(0x7f9e28071770 0x7f9e280cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e25400640 1 -- 172.21.15.67:0/3803950503 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153764041 0 0) 0x7f9e2806e2f0 con 0x7f9e28145040 2024-04-17T10:31:22.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e25400640 1 -- 172.21.15.67:0/3803950503 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c003530 con 0x7f9e28145040 2024-04-17T10:31:22.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e26800640 1 -- 172.21.15.67:0/3803950503 >> v1:172.21.15.86:6789/0 conn(0x7f9e28071310 legacy=0x7f9e280cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e25400640 1 -- 172.21.15.67:0/3803950503 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800952207 0 0) 0x7f9e2806d6e0 con 0x7f9e28071310 2024-04-17T10:31:22.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e26800640 1 -- 172.21.15.67:0/3803950503 >> v1:172.21.15.67:6789/0 conn(0x7f9e28145040 legacy=0x7f9e2806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.491+0000 7f9e26800640 1 -- 172.21.15.67:0/3803950503 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2806d9c0 con 0x7f9e28071770 2024-04-17T10:31:22.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e26800640 1 --2- 172.21.15.67:0/3803950503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e280cca10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f9e200097c0 tx=0x7f9e2001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19fe421a9f7d992 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e25400640 1 -- 172.21.15.67:0/3803950503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e200b5070 con 0x7f9e28071770 2024-04-17T10:31:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e25400640 1 -- 172.21.15.67:0/3803950503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e200a8990 con 0x7f9e28071770 2024-04-17T10:31:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e25400640 1 -- 172.21.15.67:0/3803950503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e200a8d60 con 0x7f9e28071770 2024-04-17T10:31:22.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 -- 172.21.15.67:0/3803950503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 msgr2=0x7f9e280cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/3803950503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e280cca10 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f9e200097c0 tx=0x7f9e2001ec90 comp rx=0 tx=0).stop 2024-04-17T10:31:22.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 -- 172.21.15.67:0/3803950503 shutdown_connections 2024-04-17T10:31:22.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/3803950503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e280cca10 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 -- 172.21.15.67:0/3803950503 >> 172.21.15.67:0/3803950503 conn(0x7f9e2806ac90 msgr2=0x7f9e2806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:22.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 -- 172.21.15.67:0/3803950503 shutdown_connections 2024-04-17T10:31:22.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 -- 172.21.15.67:0/3803950503 wait complete. 2024-04-17T10:31:22.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 Processor -- start 2024-04-17T10:31:22.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.495+0000 7f9e2ca00640 1 -- start start 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e28071310 0x7f9e2815b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e28139db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e28145040 0x7f9e281f7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e28071b80 con 0x7f9e28071770 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2806df80 con 0x7f9e28071310 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2806d370 con 0x7f9e28145040 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e28139db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e28139db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54172/0 (socket says 172.21.15.67:54172) 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 -- 172.21.15.67:0/261166590 learned_addr learned my addr 172.21.15.67:0/261166590 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e25e00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e28071310 0x7f9e2815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e26800640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e28145040 0x7f9e281f7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 -- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e28145040 msgr2=0x7f9e281f7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e28145040 0x7f9e281f7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 -- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e28071310 msgr2=0x7f9e2815b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e28071310 0x7f9e2815b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 -- 172.21.15.67:0/261166590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2815c590 con 0x7f9e28071770 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e25e00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e28071310 0x7f9e2815b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e26800640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e28145040 0x7f9e281f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1d400640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e28139db0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000c500 tx=0x7f9e1000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1ea00640 1 -- 172.21.15.67:0/261166590 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e10017070 con 0x7f9e28071770 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1ea00640 1 -- 172.21.15.67:0/261166590 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e10003600 con 0x7f9e28071770 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e1ea00640 1 -- 172.21.15.67:0/261166590 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e100039f0 con 0x7f9e28071770 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2815c870 con 0x7f9e28071770 2024-04-17T10:31:22.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2815b5b0 con 0x7f9e28071770 2024-04-17T10:31:22.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.499+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e28072d50 con 0x7f9e28071770 2024-04-17T10:31:22.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.503+0000 7f9e1ea00640 1 -- 172.21.15.67:0/261166590 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9e1001a070 con 0x7f9e28071770 2024-04-17T10:31:22.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.503+0000 7f9e1ea00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e04066f60 0x7f9e04069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.503+0000 7f9e1ea00640 1 -- 172.21.15.67:0/261166590 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f9e100846a0 con 0x7f9e28071770 2024-04-17T10:31:22.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.503+0000 7f9e25e00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e04066f60 0x7f9e04069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.503+0000 7f9e25e00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e04066f60 0x7f9e04069420 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9e18005c10 tx=0x7f9e18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:22.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.507+0000 7f9e1ea00640 1 -- 172.21.15.67:0/261166590 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9e100555b0 con 0x7f9e28071770 2024-04-17T10:31:22.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.703+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e28074e30 con 0x7f9e28071770 2024-04-17T10:31:22.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.703+0000 7f9e1ea00640 1 -- 172.21.15.67:0/261166590 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f9e10054eb0 con 0x7f9e28071770 2024-04-17T10:31:22.699 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:22.699 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/3654887462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3654887462},{"type":"v1","addr":"172.21.15.86:6836","nonce":3654887462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6814,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1627300843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1627300843},{"type":"v1","addr":"172.21.15.67:6835","nonce":1627300843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/39011498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":39011498},{"type":"v1","addr":"172.21.15.86:6837","nonce":39011498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:29:47.987388+0000","modified":"2024-04-17T10:30:00.535875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/3374829114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3374829114},{"type":"v1","addr":"172.21.15.67:6834","nonce":3374829114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-17T10:31:22.699 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 56 2024-04-17T10:31:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.703+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e04066f60 msgr2=0x7f9e04069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.703+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e04066f60 0x7f9e04069420 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f9e18005c10 tx=0x7f9e18009040 comp rx=0 tx=0).stop 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.703+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 msgr2=0x7f9e28139db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.703+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e28139db0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f9e1000c500 tx=0x7f9e1000c9c0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 shutdown_connections 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e04066f60 0x7f9e04069420 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e28145040 0x7f9e281f7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e28071770 0x7f9e28139db0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 --2- 172.21.15.67:0/261166590 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e28071310 0x7f9e2815b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 >> 172.21.15.67:0/261166590 conn(0x7f9e2806ac90 msgr2=0x7f9e2806b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 shutdown_connections 2024-04-17T10:31:22.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.707+0000 7f9e2ca00640 1 -- 172.21.15.67:0/261166590 wait complete. 2024-04-17T10:31:22.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:22.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d7c00640 1 Processor -- start 2024-04-17T10:31:22.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d7c00640 1 -- start start 2024-04-17T10:31:22.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d7c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f03d006e6e0 0x7f03d00ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d7c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03d01441f0 0x7f03d00ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d7c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d015ae40 con 0x7f03d006eac0 2024-04-17T10:31:22.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d7c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00cf430 con 0x7f03d01441f0 2024-04-17T10:31:22.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d7c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d0071440 con 0x7f03d006e6e0 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03d01441f0 0x7f03d00ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f03d01441f0 0x7f03d00ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54188/0 (socket says 172.21.15.67:54188) 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5c00640 1 -- 172.21.15.67:0/1481370679 learned_addr learned my addr 172.21.15.67:0/1481370679 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5200640 1 --2- 172.21.15.67:0/1481370679 >> v2:172.21.15.86:3300/0 conn(0x7f03d006e6e0 0x7f03d00ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5c00640 1 -- 172.21.15.67:0/1481370679 >> v2:172.21.15.86:3300/0 conn(0x7f03d006e6e0 msgr2=0x7f03d00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5c00640 1 --2- 172.21.15.67:0/1481370679 >> v2:172.21.15.86:3300/0 conn(0x7f03d006e6e0 0x7f03d00ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5c00640 1 -- 172.21.15.67:0/1481370679 >> v1:172.21.15.86:6789/0 conn(0x7f03d006eac0 legacy=0x7f03d0143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:22.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.867+0000 7f03d5c00640 1 -- 172.21.15.67:0/1481370679 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0071720 con 0x7f03d01441f0 2024-04-17T10:31:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d5c00640 1 --2- 172.21.15.67:0/1481370679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d01441f0 0x7f03d00ceef0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f03d00cf620 tx=0x7f03c801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590a811d9d1d0e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03cfe00640 1 -- 172.21.15.67:0/1481370679 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c80a57b0 con 0x7f03d01441f0 2024-04-17T10:31:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03cfe00640 1 -- 172.21.15.67:0/1481370679 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c80a5950 con 0x7f03d01441f0 2024-04-17T10:31:22.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03cfe00640 1 -- 172.21.15.67:0/1481370679 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c80a5d20 con 0x7f03d01441f0 2024-04-17T10:31:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 -- 172.21.15.67:0/1481370679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d01441f0 msgr2=0x7f03d00ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 --2- 172.21.15.67:0/1481370679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d01441f0 0x7f03d00ceef0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f03d00cf620 tx=0x7f03c801c8f0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 -- 172.21.15.67:0/1481370679 shutdown_connections 2024-04-17T10:31:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 --2- 172.21.15.67:0/1481370679 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d01441f0 0x7f03d00ceef0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 --2- 172.21.15.67:0/1481370679 >> v2:172.21.15.86:3300/0 conn(0x7f03d006e6e0 0x7f03d00ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 -- 172.21.15.67:0/1481370679 >> 172.21.15.67:0/1481370679 conn(0x7f03d006ac90 msgr2=0x7f03d006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:22.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 -- 172.21.15.67:0/1481370679 shutdown_connections 2024-04-17T10:31:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.871+0000 7f03d7c00640 1 -- 172.21.15.67:0/1481370679 wait complete. 2024-04-17T10:31:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 Processor -- start 2024-04-17T10:31:22.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 -- start start 2024-04-17T10:31:22.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d006e6e0 0x7f03d00c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d006eac0 0x7f03d0159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d01441f0 0x7f03d00c7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00b3a40 con 0x7f03d006e6e0 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00b38c0 con 0x7f03d01441f0 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00b3bc0 con 0x7f03d006eac0 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d006e6e0 0x7f03d00c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d006e6e0 0x7f03d00c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54200/0 (socket says 172.21.15.67:54200) 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 -- 172.21.15.67:0/351065660 learned_addr learned my addr 172.21.15.67:0/351065660 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5c00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d01441f0 0x7f03d00c7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03c7e00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d006eac0 0x7f03d0159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 -- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d006eac0 msgr2=0x7f03d0159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d006eac0 0x7f03d0159c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 -- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d01441f0 msgr2=0x7f03d00c7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d01441f0 0x7f03d00c7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:22.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 -- 172.21.15.67:0/351065660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d00c4ed0 con 0x7f03d006e6e0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03c7e00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d006eac0 0x7f03d0159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d5200640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d006e6e0 0x7f03d00c5f70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f03c000d800 tx=0x7f03c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03cd600640 1 -- 172.21.15.67:0/351065660 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c000b6c0 con 0x7f03d006e6e0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03cd600640 1 -- 172.21.15.67:0/351065660 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c0019070 con 0x7f03d006e6e0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03cd600640 1 -- 172.21.15.67:0/351065660 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c000ba60 con 0x7f03d006e6e0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d00c51b0 con 0x7f03d006e6e0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.875+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d00c5770 con 0x7f03d006e6e0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.879+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d0072d50 con 0x7f03d006e6e0 2024-04-17T10:31:22.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.879+0000 7f03cd600640 1 -- 172.21.15.67:0/351065660 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f03c000bc00 con 0x7f03d006e6e0 2024-04-17T10:31:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.879+0000 7f03cd600640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03ac066f60 0x7f03ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.879+0000 7f03cd600640 1 -- 172.21.15.67:0/351065660 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f03c0085610 con 0x7f03d006e6e0 2024-04-17T10:31:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.879+0000 7f03c7e00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03ac066f60 0x7f03ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:22.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.879+0000 7f03c7e00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03ac066f60 0x7f03ac069420 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ad8a0 tx=0x7f03b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:22.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:22.883+0000 7f03cd600640 1 -- 172.21.15.67:0/351065660 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f03c0056460 con 0x7f03d006e6e0 2024-04-17T10:31:23.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.047+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d0074e30 con 0x7f03d006e6e0 2024-04-17T10:31:23.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03cd600640 1 -- 172.21.15.67:0/351065660 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f03c0055d60 con 0x7f03d006e6e0 2024-04-17T10:31:23.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:23.045 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:08.910824+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:29:46.421958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-17T10:29:46.986879+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":84,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":12,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03ac066f60 msgr2=0x7f03ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03ac066f60 0x7f03ac069420 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f03d00ad8a0 tx=0x7f03b8009040 comp rx=0 tx=0).stop 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d006e6e0 msgr2=0x7f03d00c5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d006e6e0 0x7f03d00c5f70 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f03c000d800 tx=0x7f03c000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 shutdown_connections 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f03ac066f60 0x7f03ac069420 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f03d01441f0 0x7f03d00c7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f03d006eac0 0x7f03d0159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 --2- 172.21.15.67:0/351065660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f03d006e6e0 0x7f03d00c5f70 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 >> 172.21.15.67:0/351065660 conn(0x7f03d006ac90 msgr2=0x7f03d006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 shutdown_connections 2024-04-17T10:31:23.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.051+0000 7f03d7c00640 1 -- 172.21.15.67:0/351065660 wait complete. 2024-04-17T10:31:23.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:29:46.421958+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:29:46.986879+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 84, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:31:23.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:31:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.215+0000 7f31fd600640 1 Processor -- start 2024-04-17T10:31:23.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.215+0000 7f31fd600640 1 -- start start 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.215+0000 7f31fd600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f31f80cd030 0x7f31f806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.215+0000 7f31fd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f806e7b0 0x7f31f80b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.215+0000 7f31fd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31f80b4990 con 0x7f31f80aef60 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31fd600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31f80af340 con 0x7f31f80cd030 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31fd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f806ec00 con 0x7f31f806e7b0 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f31f80aef60 0x7f31f80cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54728/0 (socket says 172.21.15.67:54728) 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f6a00640 1 -- 172.21.15.67:0/3103585395 learned_addr learned my addr 172.21.15.67:0/3103585395 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f6000640 1 --2- 172.21.15.67:0/3103585395 >> v2:172.21.15.86:3300/0 conn(0x7f31f80cd030 0x7f31f806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f7400640 1 --2- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f806e7b0 0x7f31f80b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f5600640 1 -- 172.21.15.67:0/3103585395 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784302992 0 0) 0x7f31e8001120 con 0x7f31f80aef60 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f5600640 1 -- 172.21.15.67:0/3103585395 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31dc003530 con 0x7f31f80aef60 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f6000640 1 -- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f806e7b0 msgr2=0x7f31f80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f6000640 1 --2- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f806e7b0 0x7f31f80b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f6000640 1 -- 172.21.15.67:0/3103585395 >> v1:172.21.15.67:6789/0 conn(0x7f31f80aef60 legacy=0x7f31f80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.219+0000 7f31f6000640 1 -- 172.21.15.67:0/3103585395 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f815ae90 con 0x7f31f80cd030 2024-04-17T10:31:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31f6000640 1 --2- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f80cd030 0x7f31f806e270 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f31e40097c0 tx=0x7f31e401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87bc17517c4fc8d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31f5600640 1 -- 172.21.15.67:0/3103585395 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b0070 con 0x7f31f80cd030 2024-04-17T10:31:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31f5600640 1 -- 172.21.15.67:0/3103585395 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e40ab600 con 0x7f31f80cd030 2024-04-17T10:31:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31f5600640 1 -- 172.21.15.67:0/3103585395 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e40ab9d0 con 0x7f31f80cd030 2024-04-17T10:31:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f80cd030 msgr2=0x7f31f806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 --2- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f80cd030 0x7f31f806e270 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f31e40097c0 tx=0x7f31e401c410 comp rx=0 tx=0).stop 2024-04-17T10:31:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- 172.21.15.67:0/3103585395 shutdown_connections 2024-04-17T10:31:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 --2- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f806e7b0 0x7f31f80b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 --2- 172.21.15.67:0/3103585395 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f80cd030 0x7f31f806e270 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- 172.21.15.67:0/3103585395 >> 172.21.15.67:0/3103585395 conn(0x7f31f80b1740 msgr2=0x7f31f80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- 172.21.15.67:0/3103585395 shutdown_connections 2024-04-17T10:31:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- 172.21.15.67:0/3103585395 wait complete. 2024-04-17T10:31:23.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 Processor -- start 2024-04-17T10:31:23.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- start start 2024-04-17T10:31:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f806e7b0 0x7f31f813f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f80aef60 0x7f31f80a94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80cd030 0x7f31f8159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f815bc10 con 0x7f31f80aef60 2024-04-17T10:31:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f80b3e70 con 0x7f31f806e7b0 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.223+0000 7f31fd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f80a84b0 con 0x7f31f80cd030 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f80aef60 0x7f31f80a94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f80aef60 0x7f31f80a94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54212/0 (socket says 172.21.15.67:54212) 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 -- 172.21.15.67:0/556390058 learned_addr learned my addr 172.21.15.67:0/556390058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6a00640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f806e7b0 0x7f31f813f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f7400640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80cd030 0x7f31f8159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 -- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80cd030 msgr2=0x7f31f8159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80cd030 0x7f31f8159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 -- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f806e7b0 msgr2=0x7f31f813f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f806e7b0 0x7f31f813f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 -- 172.21.15.67:0/556390058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f80a8700 con 0x7f31f80aef60 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31f6000640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f80aef60 0x7f31f80a94c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f31e40ae040 tx=0x7f31e40191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31eea00640 1 -- 172.21.15.67:0/556390058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b8020 con 0x7f31f80aef60 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31eea00640 1 -- 172.21.15.67:0/556390058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31e40b4630 con 0x7f31f80aef60 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31f80a89e0 con 0x7f31f80aef60 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31eea00640 1 -- 172.21.15.67:0/556390058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31e40b0040 con 0x7f31f80aef60 2024-04-17T10:31:23.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31f80a8fa0 con 0x7f31f80aef60 2024-04-17T10:31:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31f804ef30 con 0x7f31f80aef60 2024-04-17T10:31:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.227+0000 7f31eea00640 1 -- 172.21.15.67:0/556390058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f31e40bb020 con 0x7f31f80aef60 2024-04-17T10:31:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.231+0000 7f31eea00640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f31d8066f60 0x7f31d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.231+0000 7f31eea00640 1 -- 172.21.15.67:0/556390058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f31e411eec0 con 0x7f31f80aef60 2024-04-17T10:31:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.231+0000 7f31f6a00640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f31d8066f60 0x7f31d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.231+0000 7f31f6a00640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f31d8066f60 0x7f31d8069420 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f31f81f9c80 tx=0x7f31e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.231+0000 7f31eea00640 1 -- 172.21.15.67:0/556390058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f31e40c2030 con 0x7f31f80aef60 2024-04-17T10:31:23.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f31f8154f50 con 0x7f31f80aef60 2024-04-17T10:31:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31eea00640 1 -- 172.21.15.67:0/556390058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f31e4016090 con 0x7f31f80aef60 2024-04-17T10:31:23.390 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:31:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f31d8066f60 msgr2=0x7f31d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f31d8066f60 0x7f31d8069420 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f31f81f9c80 tx=0x7f31e8009040 comp rx=0 tx=0).stop 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f80aef60 msgr2=0x7f31f80a94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f80aef60 0x7f31f80a94c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f31e40ae040 tx=0x7f31e40191b0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 shutdown_connections 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f31d8066f60 0x7f31d8069420 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f31f80cd030 0x7f31f8159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f31f80aef60 0x7f31f80a94c0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 --2- 172.21.15.67:0/556390058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f31f806e7b0 0x7f31f813f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 >> 172.21.15.67:0/556390058 conn(0x7f31f80b1740 msgr2=0x7f31f80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 shutdown_connections 2024-04-17T10:31:23.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.395+0000 7f31fd600640 1 -- 172.21.15.67:0/556390058 wait complete. 2024-04-17T10:31:23.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:31:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.559+0000 7fccc5200640 1 Processor -- start 2024-04-17T10:31:23.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.559+0000 7fccc5200640 1 -- start start 2024-04-17T10:31:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccc5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc0071af0 0x7fccc00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccc5200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc015cb30 con 0x7fccc0159440 2024-04-17T10:31:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccc5200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00cd610 con 0x7fccc015be70 2024-04-17T10:31:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccc5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00a8650 con 0x7fccc0071af0 2024-04-17T10:31:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccbde00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fccc0159440 0x7fccc00713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54742/0 (socket says 172.21.15.67:54742) 2024-04-17T10:31:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccbde00640 1 -- 172.21.15.67:0/3670326862 learned_addr learned my addr 172.21.15.67:0/3670326862 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:23.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccb5400640 1 --2- 172.21.15.67:0/3670326862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc0071af0 0x7fccc00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccbd400640 1 -- 172.21.15.67:0/3670326862 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606987806 0 0) 0x7fccc015cb30 con 0x7fccc0159440 2024-04-17T10:31:23.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccbd400640 1 -- 172.21.15.67:0/3670326862 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca4003530 con 0x7fccc0159440 2024-04-17T10:31:23.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccbd400640 1 -- 172.21.15.67:0/3670326862 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176224337 0 0) 0x7fccc00cd610 con 0x7fccc015be70 2024-04-17T10:31:23.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccb5400640 1 -- 172.21.15.67:0/3670326862 >> v1:172.21.15.86:6789/0 conn(0x7fccc015be70 legacy=0x7fccc015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccb5400640 1 -- 172.21.15.67:0/3670326862 >> v1:172.21.15.67:6789/0 conn(0x7fccc0159440 legacy=0x7fccc00713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.563+0000 7fccb5400640 1 -- 172.21.15.67:0/3670326862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00cd8f0 con 0x7fccc0071af0 2024-04-17T10:31:23.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccb5400640 1 --2- 172.21.15.67:0/3670326862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc0071af0 0x7fccc00ce0f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcca80097f0 tx=0x7fcca801c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1a3077a85f7406f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccbd400640 1 -- 172.21.15.67:0/3670326862 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca80b0070 con 0x7fccc0071af0 2024-04-17T10:31:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccbd400640 1 -- 172.21.15.67:0/3670326862 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca80ab400 con 0x7fccc0071af0 2024-04-17T10:31:23.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccbd400640 1 -- 172.21.15.67:0/3670326862 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0001980 con 0x7fccc0071af0 2024-04-17T10:31:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 -- 172.21.15.67:0/3670326862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc0071af0 msgr2=0x7fccc00ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 --2- 172.21.15.67:0/3670326862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc0071af0 0x7fccc00ce0f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcca80097f0 tx=0x7fcca801c320 comp rx=0 tx=0).stop 2024-04-17T10:31:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 -- 172.21.15.67:0/3670326862 shutdown_connections 2024-04-17T10:31:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 --2- 172.21.15.67:0/3670326862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc0071af0 0x7fccc00ce0f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 -- 172.21.15.67:0/3670326862 >> 172.21.15.67:0/3670326862 conn(0x7fccc00b1740 msgr2=0x7fccc00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 -- 172.21.15.67:0/3670326862 shutdown_connections 2024-04-17T10:31:23.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 -- 172.21.15.67:0/3670326862 wait complete. 2024-04-17T10:31:23.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 Processor -- start 2024-04-17T10:31:23.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.567+0000 7fccc5200640 1 -- start start 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccc0071af0 0x7fccc00acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0159440 0x7fccc00ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc015be70 0x7fccc00b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc015dc50 con 0x7fccc0071af0 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc015dad0 con 0x7fccc0159440 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc015ddd0 con 0x7fccc015be70 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccc0071af0 0x7fccc00acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccc0071af0 0x7fccc00acbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54222/0 (socket says 172.21.15.67:54222) 2024-04-17T10:31:23.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 -- 172.21.15.67:0/1264789108 learned_addr learned my addr 172.21.15.67:0/1264789108 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccb5400640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0159440 0x7fccc00ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbe800640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc015be70 0x7fccc00b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 -- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc015be70 msgr2=0x7fccc00b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc015be70 0x7fccc00b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 -- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0159440 msgr2=0x7fccc00ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0159440 0x7fccc00ae150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 -- 172.21.15.67:0/1264789108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00ae690 con 0x7fccc0071af0 2024-04-17T10:31:23.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccbde00640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccc0071af0 0x7fccc00acbf0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fccb000caa0 tx=0x7fccb000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccb6a00640 1 -- 172.21.15.67:0/1264789108 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb0018070 con 0x7fccc0071af0 2024-04-17T10:31:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccb6a00640 1 -- 172.21.15.67:0/1264789108 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb000a7f0 con 0x7fccc0071af0 2024-04-17T10:31:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccb6a00640 1 -- 172.21.15.67:0/1264789108 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb000abe0 con 0x7fccc0071af0 2024-04-17T10:31:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc00ad1d0 con 0x7fccc0071af0 2024-04-17T10:31:23.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc00ada70 con 0x7fccc0071af0 2024-04-17T10:31:23.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc004ef30 con 0x7fccc0071af0 2024-04-17T10:31:23.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.571+0000 7fccb6a00640 1 -- 172.21.15.67:0/1264789108 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fccb00070d0 con 0x7fccc0071af0 2024-04-17T10:31:23.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.575+0000 7fccb6a00640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc9c066f60 0x7fcc9c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.575+0000 7fccb6a00640 1 -- 172.21.15.67:0/1264789108 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fccb0084af0 con 0x7fccc0071af0 2024-04-17T10:31:23.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.575+0000 7fccb5400640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc9c066f60 0x7fcc9c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.575+0000 7fccb5400640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc9c066f60 0x7fcc9c069420 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcca80ad000 tx=0x7fcca80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.579+0000 7fccb6a00640 1 -- 172.21.15.67:0/1264789108 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fccb0055320 con 0x7fccc0071af0 2024-04-17T10:31:23.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fccc0067ea0 con 0x7fccc0071af0 2024-04-17T10:31:23.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccb6a00640 1 -- 172.21.15.67:0/1264789108 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fccb00593c0 con 0x7fccc0071af0 2024-04-17T10:31:23.734 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:31:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc9c066f60 msgr2=0x7fcc9c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc9c066f60 0x7fcc9c069420 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fcca80ad000 tx=0x7fcca80a6000 comp rx=0 tx=0).stop 2024-04-17T10:31:23.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccc0071af0 msgr2=0x7fccc00acbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccc0071af0 0x7fccc00acbf0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fccb000caa0 tx=0x7fccb000cf60 comp rx=0 tx=0).stop 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 shutdown_connections 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc9c066f60 0x7fcc9c069420 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccc015be70 0x7fccc00b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccc0159440 0x7fccc00ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 --2- 172.21.15.67:0/1264789108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fccc0071af0 0x7fccc00acbf0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 >> 172.21.15.67:0/1264789108 conn(0x7fccc00b1740 msgr2=0x7fccc00713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 shutdown_connections 2024-04-17T10:31:23.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.739+0000 7fccc5200640 1 -- 172.21.15.67:0/1264789108 wait complete. 2024-04-17T10:31:23.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:31:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0c000640 1 Processor -- start 2024-04-17T10:31:23.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0c000640 1 -- start start 2024-04-17T10:31:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0c000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf040aef60 0x7fcf040cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf040b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0c000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf040b4990 con 0x7fcf040cce60 2024-04-17T10:31:23.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0c000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf040af340 con 0x7fcf040aef60 2024-04-17T10:31:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0415ae90 con 0x7fcf0406e980 2024-04-17T10:31:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf09600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf040aef60 0x7fcf040cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf0a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf040b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf09600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf040aef60 0x7fcf040cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37600/0 (socket says 172.21.15.67:37600) 2024-04-17T10:31:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.903+0000 7fcf09600640 1 -- 172.21.15.67:0/2540313116 learned_addr learned my addr 172.21.15.67:0/2540313116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf0a000640 1 -- 172.21.15.67:0/2540313116 >> v2:172.21.15.86:3300/0 conn(0x7fcf040aef60 msgr2=0x7fcf040cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf0a000640 1 --2- 172.21.15.67:0/2540313116 >> v2:172.21.15.86:3300/0 conn(0x7fcf040aef60 0x7fcf040cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf0a000640 1 -- 172.21.15.67:0/2540313116 >> v1:172.21.15.86:6789/0 conn(0x7fcf040cce60 legacy=0x7fcf0406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf0a000640 1 -- 172.21.15.67:0/2540313116 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0406ee80 con 0x7fcf0406e980 2024-04-17T10:31:23.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf0a000640 1 --2- 172.21.15.67:0/2540313116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf040b3120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009d90 tx=0x7fcefc01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e05e609ca6bbbb05 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf03e00640 1 -- 172.21.15.67:0/2540313116 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0b2070 con 0x7fcf0406e980 2024-04-17T10:31:23.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf03e00640 1 -- 172.21.15.67:0/2540313116 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc0a57a0 con 0x7fcf0406e980 2024-04-17T10:31:23.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf03e00640 1 -- 172.21.15.67:0/2540313116 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc0a5b70 con 0x7fcf0406e980 2024-04-17T10:31:23.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf0c000640 1 -- 172.21.15.67:0/2540313116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 msgr2=0x7fcf040b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.907+0000 7fcf0c000640 1 --2- 172.21.15.67:0/2540313116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf040b3120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcefc009d90 tx=0x7fcefc01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- 172.21.15.67:0/2540313116 shutdown_connections 2024-04-17T10:31:23.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 --2- 172.21.15.67:0/2540313116 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf040b3120 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 --2- 172.21.15.67:0/2540313116 >> v2:172.21.15.86:3300/0 conn(0x7fcf040aef60 0x7fcf040cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- 172.21.15.67:0/2540313116 >> 172.21.15.67:0/2540313116 conn(0x7fcf040b1740 msgr2=0x7fcf040b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:23.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- 172.21.15.67:0/2540313116 shutdown_connections 2024-04-17T10:31:23.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- 172.21.15.67:0/2540313116 wait complete. 2024-04-17T10:31:23.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 Processor -- start 2024-04-17T10:31:23.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- start start 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf04153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040aef60 0x7fcf04140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040cce60 0x7fcf04151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040b3ff0 con 0x7fcf040aef60 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040b3e70 con 0x7fcf040cce60 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf041673c0 con 0x7fcf0406e980 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040aef60 0x7fcf04140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040aef60 0x7fcf04140110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54224/0 (socket says 172.21.15.67:54224) 2024-04-17T10:31:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 -- 172.21.15.67:0/386096677 learned_addr learned my addr 172.21.15.67:0/386096677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0a000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040cce60 0x7fcf04151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf09600640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf04153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 -- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 msgr2=0x7fcf04153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf04153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 -- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040cce60 msgr2=0x7fcf04151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040cce60 0x7fcf04151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 -- 172.21.15.67:0/386096677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf041526d0 con 0x7fcf040aef60 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf0a000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040cce60 0x7fcf04151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf08c00640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040aef60 0x7fcf04140110 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c420 tx=0x7fcef000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf01600640 1 -- 172.21.15.67:0/386096677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef00052d0 con 0x7fcf040aef60 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.911+0000 7fcf01600640 1 -- 172.21.15.67:0/386096677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0005470 con 0x7fcf040aef60 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf01600640 1 -- 172.21.15.67:0/386096677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef0007260 con 0x7fcf040aef60 2024-04-17T10:31:23.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf041529b0 con 0x7fcf040aef60 2024-04-17T10:31:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf04152f70 con 0x7fcf040aef60 2024-04-17T10:31:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0404ef30 con 0x7fcf040aef60 2024-04-17T10:31:23.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf01600640 1 -- 172.21.15.67:0/386096677 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcef0007400 con 0x7fcf040aef60 2024-04-17T10:31:23.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf01600640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:23.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf01600640 1 -- 172.21.15.67:0/386096677 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fcef0084700 con 0x7fcf040aef60 2024-04-17T10:31:23.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.915+0000 7fcf09600640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:23.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.919+0000 7fcf09600640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009ab0 tx=0x7fcef8009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:23.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:23.919+0000 7fcf01600640 1 -- 172.21.15.67:0/386096677 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcf040b4990 con 0x7fcf040aef60 2024-04-17T10:31:24.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.083+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcf040b1c50 con 0x7fcf040aef60 2024-04-17T10:31:24.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.083+0000 7fcf01600640 1 -- 172.21.15.67:0/386096677 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcef0054e70 con 0x7fcf040aef60 2024-04-17T10:31:24.079 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 msgr2=0x7fcee4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fcef8009ab0 tx=0x7fcef8009240 comp rx=0 tx=0).stop 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040aef60 msgr2=0x7fcf04140110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040aef60 0x7fcf04140110 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fcef000c420 tx=0x7fcef000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 shutdown_connections 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcee4066f60 0x7fcee4069420 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf040cce60 0x7fcf04151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf040aef60 0x7fcf04140110 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 --2- 172.21.15.67:0/386096677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf0406e980 0x7fcf04153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 >> 172.21.15.67:0/386096677 conn(0x7fcf040b1740 msgr2=0x7fcf04159850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 shutdown_connections 2024-04-17T10:31:24.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.087+0000 7fcf0c000640 1 -- 172.21.15.67:0/386096677 wait complete. 2024-04-17T10:31:24.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:31:24.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceea00640 1 Processor -- start 2024-04-17T10:31:24.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceea00640 1 -- start start 2024-04-17T10:31:24.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce80acee0 0x7f3ce80ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce80b4e50 con 0x7f3ce80b4900 2024-04-17T10:31:24.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceea00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce806ad30 con 0x7f3ce815a990 2024-04-17T10:31:24.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce806b010 con 0x7f3ce80acee0 2024-04-17T10:31:24.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3cdf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3ce80b4900 0x7f3ce806dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54758/0 (socket says 172.21.15.67:54758) 2024-04-17T10:31:24.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3cdf400640 1 -- 172.21.15.67:0/1863668746 learned_addr learned my addr 172.21.15.67:0/1863668746 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:24.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceca00640 1 --2- 172.21.15.67:0/1863668746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce80acee0 0x7f3ce80ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ce7400640 1 -- 172.21.15.67:0/1863668746 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319011506 0 0) 0x7f3ce80b4e50 con 0x7f3ce80b4900 2024-04-17T10:31:24.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ce7400640 1 -- 172.21.15.67:0/1863668746 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc005740 con 0x7f3ce80b4900 2024-04-17T10:31:24.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceca00640 1 -- 172.21.15.67:0/1863668746 >> v1:172.21.15.86:6789/0 conn(0x7f3ce815a990 legacy=0x7f3ce815ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ce7400640 1 -- 172.21.15.67:0/1863668746 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348497260 0 0) 0x7f3ce806ad30 con 0x7f3ce815a990 2024-04-17T10:31:24.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceca00640 1 -- 172.21.15.67:0/1863668746 >> v1:172.21.15.67:6789/0 conn(0x7f3ce80b4900 legacy=0x7f3ce806dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.247+0000 7f3ceca00640 1 -- 172.21.15.67:0/1863668746 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce806b2f0 con 0x7f3ce80acee0 2024-04-17T10:31:24.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceca00640 1 --2- 172.21.15.67:0/1863668746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce80acee0 0x7f3ce80ad390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009dc0 tx=0x7f3ce001c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=971843fd97d0c64e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ce7400640 1 -- 172.21.15.67:0/1863668746 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00b2070 con 0x7f3ce80acee0 2024-04-17T10:31:24.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ce7400640 1 -- 172.21.15.67:0/1863668746 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce00a5960 con 0x7f3ce80acee0 2024-04-17T10:31:24.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ce7400640 1 -- 172.21.15.67:0/1863668746 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00a5d30 con 0x7f3ce80acee0 2024-04-17T10:31:24.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceea00640 1 -- 172.21.15.67:0/1863668746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce80acee0 msgr2=0x7f3ce80ad390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceea00640 1 --2- 172.21.15.67:0/1863668746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce80acee0 0x7f3ce80ad390 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009dc0 tx=0x7f3ce001c990 comp rx=0 tx=0).stop 2024-04-17T10:31:24.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceea00640 1 -- 172.21.15.67:0/1863668746 shutdown_connections 2024-04-17T10:31:24.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceea00640 1 --2- 172.21.15.67:0/1863668746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce80acee0 0x7f3ce80ad390 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceea00640 1 -- 172.21.15.67:0/1863668746 >> 172.21.15.67:0/1863668746 conn(0x7f3ce80af670 msgr2=0x7f3ce80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:24.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceea00640 1 -- 172.21.15.67:0/1863668746 shutdown_connections 2024-04-17T10:31:24.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.251+0000 7f3ceea00640 1 -- 172.21.15.67:0/1863668746 wait complete. 2024-04-17T10:31:24.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 Processor -- start 2024-04-17T10:31:24.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 -- start start 2024-04-17T10:31:24.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce80acee0 0x7f3ce813d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce80b4900 0x7f3ce80cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce815a990 0x7f3ce81f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce806c430 con 0x7f3ce80acee0 2024-04-17T10:31:24.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce806c2b0 con 0x7f3ce80b4900 2024-04-17T10:31:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce8071c60 con 0x7f3ce815a990 2024-04-17T10:31:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce80acee0 0x7f3ce813d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce80acee0 0x7f3ce813d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54230/0 (socket says 172.21.15.67:54230) 2024-04-17T10:31:24.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 -- 172.21.15.67:0/4192024908 learned_addr learned my addr 172.21.15.67:0/4192024908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:24.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 -- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce815a990 msgr2=0x7f3ce81f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3cdf400640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce80b4900 0x7f3ce80cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceca00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce815a990 0x7f3ce81f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce815a990 0x7f3ce81f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 -- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce80b4900 msgr2=0x7f3ce80cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce80b4900 0x7f3ce80cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 -- 172.21.15.67:0/4192024908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce813dd70 con 0x7f3ce80acee0 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3cdf400640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce80b4900 0x7f3ce80cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceca00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce815a990 0x7f3ce81f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce7e00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce80acee0 0x7f3ce813d830 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c450 tx=0x7f3cd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce4c00640 1 -- 172.21.15.67:0/4192024908 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8017070 con 0x7f3ce80acee0 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce4c00640 1 -- 172.21.15.67:0/4192024908 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd8005480 con 0x7f3ce80acee0 2024-04-17T10:31:24.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ce4c00640 1 -- 172.21.15.67:0/4192024908 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8003840 con 0x7f3ce80acee0 2024-04-17T10:31:24.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce815b920 con 0x7f3ce80acee0 2024-04-17T10:31:24.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.255+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce815c160 con 0x7f3ce80acee0 2024-04-17T10:31:24.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.259+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce809e260 con 0x7f3ce80acee0 2024-04-17T10:31:24.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.259+0000 7f3ce4c00640 1 -- 172.21.15.67:0/4192024908 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3cd801a070 con 0x7f3ce80acee0 2024-04-17T10:31:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.259+0000 7f3ce4c00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.259+0000 7f3ce4c00640 1 -- 172.21.15.67:0/4192024908 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f3cd8084580 con 0x7f3ce80acee0 2024-04-17T10:31:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.259+0000 7f3cdf400640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.259+0000 7f3cdf400640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80afba0 tx=0x7f3cd0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.263+0000 7f3ce4c00640 1 -- 172.21.15.67:0/4192024908 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3ce815b920 con 0x7f3ce80acee0 2024-04-17T10:31:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.455+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ce80b42e0 con 0x7f3ce80acee0 2024-04-17T10:31:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.455+0000 7f3ce4c00640 1 -- 172.21.15.67:0/4192024908 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f3cd8055340 con 0x7f3ce80acee0 2024-04-17T10:31:24.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:24.452 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/3654887462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3654887462},{"type":"v1","addr":"172.21.15.86:6836","nonce":3654887462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6814,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1627300843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1627300843},{"type":"v1","addr":"172.21.15.67:6835","nonce":1627300843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/39011498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":39011498},{"type":"v1","addr":"172.21.15.86:6837","nonce":39011498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:29:47.987388+0000","modified":"2024-04-17T10:30:00.535875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/3374829114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3374829114},{"type":"v1","addr":"172.21.15.67:6834","nonce":3374829114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-17T10:31:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 56 2024-04-17T10:31:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 msgr2=0x7f3cc4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3ce80afba0 tx=0x7f3cd0009040 comp rx=0 tx=0).stop 2024-04-17T10:31:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce80acee0 msgr2=0x7f3ce813d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce80acee0 0x7f3ce813d830 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800c450 tx=0x7f3cd800c910 comp rx=0 tx=0).stop 2024-04-17T10:31:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 shutdown_connections 2024-04-17T10:31:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce815a990 0x7f3ce81f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce80b4900 0x7f3ce80cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 --2- 172.21.15.67:0/4192024908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce80acee0 0x7f3ce813d830 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 >> 172.21.15.67:0/4192024908 conn(0x7f3ce80af670 msgr2=0x7f3ce80b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 shutdown_connections 2024-04-17T10:31:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.459+0000 7f3ceea00640 1 -- 172.21.15.67:0/4192024908 wait complete. 2024-04-17T10:31:24.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:24.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.619+0000 7f4547200640 1 Processor -- start 2024-04-17T10:31:24.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.619+0000 7f4547200640 1 -- start start 2024-04-17T10:31:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4547200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4548161160 0x7f4548099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4547200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548099f90 0x7f4548161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4547200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45480d0760 con 0x7f45480b2570 2024-04-17T10:31:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4547200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454809a400 con 0x7f4548161160 2024-04-17T10:31:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4547200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45480cfc70 con 0x7f4548099f90 2024-04-17T10:31:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f453fe00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f45480b2570 0x7f45481616c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:59900/0 (socket says 172.21.15.67:59900) 2024-04-17T10:31:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f453fe00640 1 -- 172.21.15.67:0/2607392368 learned_addr learned my addr 172.21.15.67:0/2607392368 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4545e00640 1 --2- 172.21.15.67:0/2607392368 >> v2:172.21.15.86:3300/0 conn(0x7f4548161160 0x7f4548099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4546800640 1 --2- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548099f90 0x7f4548161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4545e00640 1 -- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548099f90 msgr2=0x7f4548161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4545e00640 1 --2- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548099f90 0x7f4548161dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4545e00640 1 -- 172.21.15.67:0/2607392368 >> v1:172.21.15.86:6789/0 conn(0x7f45480b2570 legacy=0x7f45481616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.623+0000 7f4545e00640 1 -- 172.21.15.67:0/2607392368 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480cc2a0 con 0x7f4548161160 2024-04-17T10:31:24.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4545e00640 1 --2- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4548161160 0x7f4548099a50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f45380097c0 tx=0x7f453801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d9fd2ac39034688 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4545400640 1 -- 172.21.15.67:0/2607392368 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45380b1070 con 0x7f4548161160 2024-04-17T10:31:24.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4545400640 1 -- 172.21.15.67:0/2607392368 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45380ac410 con 0x7f4548161160 2024-04-17T10:31:24.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4545400640 1 -- 172.21.15.67:0/2607392368 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45380ac7e0 con 0x7f4548161160 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 -- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4548161160 msgr2=0x7f4548099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 --2- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4548161160 0x7f4548099a50 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f45380097c0 tx=0x7f453801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 -- 172.21.15.67:0/2607392368 shutdown_connections 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 --2- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548099f90 0x7f4548161dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 --2- 172.21.15.67:0/2607392368 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4548161160 0x7f4548099a50 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 -- 172.21.15.67:0/2607392368 >> 172.21.15.67:0/2607392368 conn(0x7f45480d1690 msgr2=0x7f45480d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 -- 172.21.15.67:0/2607392368 shutdown_connections 2024-04-17T10:31:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 -- 172.21.15.67:0/2607392368 wait complete. 2024-04-17T10:31:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.627+0000 7f4547200640 1 Processor -- start 2024-04-17T10:31:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 -- start start 2024-04-17T10:31:24.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548099f90 0x7f45480c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45480b2570 0x7f45480c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548161160 0x7f45480c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45480b2bc0 con 0x7f4548099f90 2024-04-17T10:31:24.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45480cc5d0 con 0x7f45480b2570 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454806bda0 con 0x7f4548161160 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548099f90 0x7f45480c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548099f90 0x7f45480c9640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54236/0 (socket says 172.21.15.67:54236) 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 -- 172.21.15.67:0/620814678 learned_addr learned my addr 172.21.15.67:0/620814678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4546800640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548161160 0x7f45480c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f453fe00640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45480b2570 0x7f45480c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 -- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548161160 msgr2=0x7f45480c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548161160 0x7f45480c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 -- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45480b2570 msgr2=0x7f45480c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45480b2570 0x7f45480c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 -- 172.21.15.67:0/620814678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480c9e70 con 0x7f4548099f90 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f453fe00640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45480b2570 0x7f45480c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4545e00640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548099f90 0x7f45480c9640 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f45380af040 tx=0x7f4538019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f453e000640 1 -- 172.21.15.67:0/620814678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45380b9070 con 0x7f4548099f90 2024-04-17T10:31:24.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f453e000640 1 -- 172.21.15.67:0/620814678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45380acd90 con 0x7f4548099f90 2024-04-17T10:31:24.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f453e000640 1 -- 172.21.15.67:0/620814678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45380b1040 con 0x7f4548099f90 2024-04-17T10:31:24.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4548153b70 con 0x7f4548099f90 2024-04-17T10:31:24.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4548154130 con 0x7f4548099f90 2024-04-17T10:31:24.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.631+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4548074dc0 con 0x7f4548099f90 2024-04-17T10:31:24.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.635+0000 7f453e000640 1 -- 172.21.15.67:0/620814678 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f45380bc070 con 0x7f4548099f90 2024-04-17T10:31:24.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.635+0000 7f453e000640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4524066f60 0x7f4524069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.635+0000 7f453e000640 1 -- 172.21.15.67:0/620814678 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f453811ebd0 con 0x7f4548099f90 2024-04-17T10:31:24.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.635+0000 7f453fe00640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4524066f60 0x7f4524069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.635+0000 7f453fe00640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4524066f60 0x7f4524069420 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4548162730 tx=0x7f4530009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.639+0000 7f453e000640 1 -- 172.21.15.67:0/620814678 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4538016090 con 0x7f4548099f90 2024-04-17T10:31:24.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.803+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45480c9420 con 0x7f4548099f90 2024-04-17T10:31:24.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f453e000640 1 -- 172.21.15.67:0/620814678 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f45380ef3d0 con 0x7f4548099f90 2024-04-17T10:31:24.800 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:24.801 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:08.910824+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:29:46.421958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-17T10:29:46.986879+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":84,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":12,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:24.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4524066f60 msgr2=0x7f4524069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4524066f60 0x7f4524069420 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f4548162730 tx=0x7f4530009040 comp rx=0 tx=0).stop 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548099f90 msgr2=0x7f45480c9640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548099f90 0x7f45480c9640 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f45380af040 tx=0x7f4538019240 comp rx=0 tx=0).stop 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 shutdown_connections 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4524066f60 0x7f4524069420 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4548161160 0x7f45480c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f45480b2570 0x7f45480c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 --2- 172.21.15.67:0/620814678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548099f90 0x7f45480c9640 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 >> 172.21.15.67:0/620814678 conn(0x7f45480d1690 msgr2=0x7f4548161650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 shutdown_connections 2024-04-17T10:31:24.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.807+0000 7f4547200640 1 -- 172.21.15.67:0/620814678 wait complete. 2024-04-17T10:31:24.822 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:31:24.822 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:31:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.967+0000 7eff19800640 1 Processor -- start 2024-04-17T10:31:24.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.967+0000 7eff19800640 1 -- start start 2024-04-17T10:31:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff19800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7eff140b07b0 0x7eff14070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff19800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff1406c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff19800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff140b5980 con 0x7eff1415dd10 2024-04-17T10:31:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff19800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff140b1280 con 0x7eff140b07b0 2024-04-17T10:31:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff19800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff140b0cc0 con 0x7eff14071300 2024-04-17T10:31:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff12a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7eff1415dd10 0x7eff140b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54762/0 (socket says 172.21.15.67:54762) 2024-04-17T10:31:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff12a00640 1 -- 172.21.15.67:0/3469059746 learned_addr learned my addr 172.21.15.67:0/3469059746 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff0a000640 1 --2- 172.21.15.67:0/3469059746 >> v2:172.21.15.86:3300/0 conn(0x7eff140b07b0 0x7eff14070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff13400640 1 --2- 172.21.15.67:0/3469059746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff1406c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff12000640 1 -- 172.21.15.67:0/3469059746 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226963635 0 0) 0x7eff140b5980 con 0x7eff1415dd10 2024-04-17T10:31:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff12000640 1 -- 172.21.15.67:0/3469059746 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef8003530 con 0x7eff1415dd10 2024-04-17T10:31:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff13400640 1 -- 172.21.15.67:0/3469059746 >> v2:172.21.15.86:3300/0 conn(0x7eff140b07b0 msgr2=0x7eff14070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff13400640 1 --2- 172.21.15.67:0/3469059746 >> v2:172.21.15.86:3300/0 conn(0x7eff140b07b0 0x7eff14070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff13400640 1 -- 172.21.15.67:0/3469059746 >> v1:172.21.15.67:6789/0 conn(0x7eff1415dd10 legacy=0x7eff140b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff13400640 1 -- 172.21.15.67:0/3469059746 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140b0fa0 con 0x7eff14071300 2024-04-17T10:31:24.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.971+0000 7eff0a000640 1 --2- 172.21.15.67:0/3469059746 >> v2:172.21.15.86:3300/0 conn(0x7eff140b07b0 0x7eff14070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff13400640 1 --2- 172.21.15.67:0/3469059746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff1406c6d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7eff0c009d90 tx=0x7eff0c01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9f227718a323d6a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff12000640 1 -- 172.21.15.67:0/3469059746 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c0b2070 con 0x7eff14071300 2024-04-17T10:31:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff12000640 1 -- 172.21.15.67:0/3469059746 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0c0a5890 con 0x7eff14071300 2024-04-17T10:31:24.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff12000640 1 -- 172.21.15.67:0/3469059746 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff0c0a5c40 con 0x7eff14071300 2024-04-17T10:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 -- 172.21.15.67:0/3469059746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 msgr2=0x7eff1406c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 --2- 172.21.15.67:0/3469059746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff1406c6d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7eff0c009d90 tx=0x7eff0c01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 -- 172.21.15.67:0/3469059746 shutdown_connections 2024-04-17T10:31:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 --2- 172.21.15.67:0/3469059746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff1406c6d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 --2- 172.21.15.67:0/3469059746 >> v2:172.21.15.86:3300/0 conn(0x7eff140b07b0 0x7eff14070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 -- 172.21.15.67:0/3469059746 >> 172.21.15.67:0/3469059746 conn(0x7eff140b34d0 msgr2=0x7eff140b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 -- 172.21.15.67:0/3469059746 shutdown_connections 2024-04-17T10:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 -- 172.21.15.67:0/3469059746 wait complete. 2024-04-17T10:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 Processor -- start 2024-04-17T10:31:24.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.975+0000 7eff19800640 1 -- start start 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff141532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff140b07b0 0x7eff141470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff1415dd10 0x7eff140cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1406d5a0 con 0x7eff140b07b0 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1406d420 con 0x7eff1415dd10 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1406d720 con 0x7eff14071300 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff140b07b0 0x7eff141470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff140b07b0 0x7eff141470a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54246/0 (socket says 172.21.15.67:54246) 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 -- 172.21.15.67:0/3867430958 learned_addr learned my addr 172.21.15.67:0/3867430958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff13400640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff1415dd10 0x7eff140cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff12a00640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff141532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 -- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 msgr2=0x7eff141532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff141532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 -- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff1415dd10 msgr2=0x7eff140cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff1415dd10 0x7eff140cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 -- 172.21.15.67:0/3867430958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff14158f80 con 0x7eff140b07b0 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff13400640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff1415dd10 0x7eff140cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0a000640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff140b07b0 0x7eff141470a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7efefc00ea00 tx=0x7efefc00eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0b400640 1 -- 172.21.15.67:0/3867430958 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc017070 con 0x7eff140b07b0 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0b400640 1 -- 172.21.15.67:0/3867430958 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00ca60 con 0x7eff140b07b0 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0b400640 1 -- 172.21.15.67:0/3867430958 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00ce50 con 0x7eff140b07b0 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff1415a2c0 con 0x7eff140b07b0 2024-04-17T10:31:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff14159500 con 0x7eff140b07b0 2024-04-17T10:31:24.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff14070bb0 con 0x7eff140b07b0 2024-04-17T10:31:24.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.979+0000 7eff0b400640 1 -- 172.21.15.67:0/3867430958 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efefc01a070 con 0x7eff140b07b0 2024-04-17T10:31:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.983+0000 7eff0b400640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efef0066f60 0x7efef0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.983+0000 7eff0b400640 1 -- 172.21.15.67:0/3867430958 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7efefc084e70 con 0x7eff140b07b0 2024-04-17T10:31:24.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.983+0000 7eff12a00640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efef0066f60 0x7efef0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.983+0000 7eff12a00640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efef0066f60 0x7efef0069420 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7eff04005890 tx=0x7eff04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:24.987+0000 7eff0b400640 1 -- 172.21.15.67:0/3867430958 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efefc055c30 con 0x7eff140b07b0 2024-04-17T10:31:25.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.183+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff14074e30 con 0x7eff140b07b0 2024-04-17T10:31:25.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.183+0000 7eff0b400640 1 -- 172.21.15.67:0/3867430958 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4934 (secure 0 0 0) 0x7efefc055530 con 0x7eff140b07b0 2024-04-17T10:31:25.180 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:25.180 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/3654887462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3654887462},{"type":"v1","addr":"172.21.15.86:6836","nonce":3654887462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6814,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1627300843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1627300843},{"type":"v1","addr":"172.21.15.67:6835","nonce":1627300843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/39011498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":39011498},{"type":"v1","addr":"172.21.15.86:6837","nonce":39011498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:29:47.987388+0000","modified":"2024-04-17T10:30:00.535875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/3374829114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3374829114},{"type":"v1","addr":"172.21.15.67:6834","nonce":3374829114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-17T10:31:25.180 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 56 2024-04-17T10:31:25.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efef0066f60 msgr2=0x7efef0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efef0066f60 0x7efef0069420 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7eff04005890 tx=0x7eff04009040 comp rx=0 tx=0).stop 2024-04-17T10:31:25.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff140b07b0 msgr2=0x7eff141470a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff140b07b0 0x7eff141470a0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7efefc00ea00 tx=0x7efefc00eec0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 shutdown_connections 2024-04-17T10:31:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efef0066f60 0x7efef0069420 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff1415dd10 0x7eff140cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff140b07b0 0x7eff141470a0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 --2- 172.21.15.67:0/3867430958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff14071300 0x7eff141532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 >> 172.21.15.67:0/3867430958 conn(0x7eff140b34d0 msgr2=0x7eff140d0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 shutdown_connections 2024-04-17T10:31:25.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.191+0000 7eff19800640 1 -- 172.21.15.67:0/3867430958 wait complete. 2024-04-17T10:31:25.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:31:25.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27aaa00640 1 Processor -- start 2024-04-17T10:31:25.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27aaa00640 1 -- start start 2024-04-17T10:31:25.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27aaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27a415dd10 0x7f27a40b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27aaa00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f27a40714d0 0x7f27a406c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27aaa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27a40b0cc0 con 0x7f27a40b57b0 2024-04-17T10:31:25.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27aaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a40b1280 con 0x7f27a415dd10 2024-04-17T10:31:25.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27aaa00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27a40b0fa0 con 0x7f27a40714d0 2024-04-17T10:31:25.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f27a415dd10 0x7f27a40b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f279b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f27a40b57b0 0x7f27a4070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54776/0 (socket says 172.21.15.67:54776) 2024-04-17T10:31:25.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f279b400640 1 -- 172.21.15.67:0/1906099661 learned_addr learned my addr 172.21.15.67:0/1906099661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:25.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a8a00640 1 --2- 172.21.15.67:0/1906099661 >> v2:172.21.15.86:3300/0 conn(0x7f27a40714d0 0x7f27a406c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a3400640 1 -- 172.21.15.67:0/1906099661 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303350228 0 0) 0x7f27a40b0cc0 con 0x7f27a40b57b0 2024-04-17T10:31:25.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a3e00640 1 -- 172.21.15.67:0/1906099661 >> v2:172.21.15.86:3300/0 conn(0x7f27a40714d0 msgr2=0x7f27a406c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:25.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a3e00640 1 --2- 172.21.15.67:0/1906099661 >> v2:172.21.15.86:3300/0 conn(0x7f27a40714d0 0x7f27a406c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a3e00640 1 -- 172.21.15.67:0/1906099661 >> v1:172.21.15.67:6789/0 conn(0x7f27a40b57b0 legacy=0x7f27a4070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a3e00640 1 -- 172.21.15.67:0/1906099661 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a406bed0 con 0x7f27a415dd10 2024-04-17T10:31:25.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.351+0000 7f27a8a00640 1 --2- 172.21.15.67:0/1906099661 >> v2:172.21.15.86:3300/0 conn(0x7f27a40714d0 0x7f27a406c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:25.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27a3e00640 1 --2- 172.21.15.67:0/1906099661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a415dd10 0x7f27a40b5270 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f27a40d11c0 tx=0x7f279401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0cf215c7a2bced server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:25.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27a3400640 1 -- 172.21.15.67:0/1906099661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940b1070 con 0x7f27a415dd10 2024-04-17T10:31:25.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27a3400640 1 -- 172.21.15.67:0/1906099661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27940ac600 con 0x7f27a415dd10 2024-04-17T10:31:25.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27a3400640 1 -- 172.21.15.67:0/1906099661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940ac9d0 con 0x7f27a415dd10 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 -- 172.21.15.67:0/1906099661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a415dd10 msgr2=0x7f27a40b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 --2- 172.21.15.67:0/1906099661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a415dd10 0x7f27a40b5270 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f27a40d11c0 tx=0x7f279401c3c0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 -- 172.21.15.67:0/1906099661 shutdown_connections 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 --2- 172.21.15.67:0/1906099661 >> v2:172.21.15.86:3300/0 conn(0x7f27a40714d0 0x7f27a406c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 --2- 172.21.15.67:0/1906099661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a415dd10 0x7f27a40b5270 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 -- 172.21.15.67:0/1906099661 >> 172.21.15.67:0/1906099661 conn(0x7f27a40b34d0 msgr2=0x7f27a40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 -- 172.21.15.67:0/1906099661 shutdown_connections 2024-04-17T10:31:25.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.355+0000 7f27aaa00640 1 -- 172.21.15.67:0/1906099661 wait complete. 2024-04-17T10:31:25.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 Processor -- start 2024-04-17T10:31:25.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 -- start start 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a40714d0 0x7f27a414dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a415dd10 0x7f27a40cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a414e5d0 0x7f27a40d0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a406d7f0 con 0x7f27a40714d0 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a406d670 con 0x7f27a415dd10 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27a406d970 con 0x7f27a414e5d0 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a40714d0 0x7f27a414dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a40714d0 0x7f27a414dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54262/0 (socket says 172.21.15.67:54262) 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 -- 172.21.15.67:0/2521339232 learned_addr learned my addr 172.21.15.67:0/2521339232 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f279b400640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a415dd10 0x7f27a40cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a8a00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a414e5d0 0x7f27a40d0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 -- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a414e5d0 msgr2=0x7f27a40d0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a414e5d0 0x7f27a40d0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 -- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a415dd10 msgr2=0x7f27a40cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a415dd10 0x7f27a40cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 -- 172.21.15.67:0/2521339232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a40cada0 con 0x7f27a40714d0 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a8a00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a414e5d0 0x7f27a40d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a3e00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a40714d0 0x7f27a414dcb0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2794018fd0 tx=0x7f279401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a0c00640 1 -- 172.21.15.67:0/2521339232 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940b1040 con 0x7f27a40714d0 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a0c00640 1 -- 172.21.15.67:0/2521339232 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27940bc070 con 0x7f27a40714d0 2024-04-17T10:31:25.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a40cb080 con 0x7f27a40714d0 2024-04-17T10:31:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27a0c00640 1 -- 172.21.15.67:0/2521339232 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940b7850 con 0x7f27a40714d0 2024-04-17T10:31:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a40cb640 con 0x7f27a40714d0 2024-04-17T10:31:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.359+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a4070bb0 con 0x7f27a40714d0 2024-04-17T10:31:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.363+0000 7f27a0c00640 1 -- 172.21.15.67:0/2521339232 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f27940ac480 con 0x7f27a40714d0 2024-04-17T10:31:25.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.363+0000 7f27a0c00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2780066f60 0x7f2780069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.363+0000 7f27a0c00640 1 -- 172.21.15.67:0/2521339232 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f279411efe0 con 0x7f27a40714d0 2024-04-17T10:31:25.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.363+0000 7f279b400640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2780066f60 0x7f2780069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.363+0000 7f279b400640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2780066f60 0x7f2780069420 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f27a40d11a0 tx=0x7f278c007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:25.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.367+0000 7f27a0c00640 1 -- 172.21.15.67:0/2521339232 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f27940c5020 con 0x7f27a40714d0 2024-04-17T10:31:25.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.559+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27a4074e30 con 0x7f27a40714d0 2024-04-17T10:31:25.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.559+0000 7f27a0c00640 1 -- 172.21.15.67:0/2521339232 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f2794016070 con 0x7f27a40714d0 2024-04-17T10:31:25.551 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:25.551 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":56,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/3654887462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3654887462},{"type":"v1","addr":"172.21.15.86:6836","nonce":3654887462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6814,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1627300843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1627300843},{"type":"v1","addr":"172.21.15.67:6835","nonce":1627300843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/39011498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":39011498},{"type":"v1","addr":"172.21.15.86:6837","nonce":39011498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-04-17T10:29:47.987388+0000","modified":"2024-04-17T10:30:00.535875+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7206},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7206":{"gid":7206,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/3374829114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3374829114},{"type":"v1","addr":"172.21.15.67:6834","nonce":3374829114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-04-17T10:31:25.551 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 56 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2780066f60 msgr2=0x7f2780069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2780066f60 0x7f2780069420 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f27a40d11a0 tx=0x7f278c007d60 comp rx=0 tx=0).stop 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a40714d0 msgr2=0x7f27a414dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a40714d0 0x7f27a414dcb0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2794018fd0 tx=0x7f279401ac00 comp rx=0 tx=0).stop 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 shutdown_connections 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2780066f60 0x7f2780069420 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f27a414e5d0 0x7f27a40d0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f27a415dd10 0x7f27a40cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 --2- 172.21.15.67:0/2521339232 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f27a40714d0 0x7f27a414dcb0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 >> 172.21.15.67:0/2521339232 conn(0x7f27a40b34d0 msgr2=0x7f27a40b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:25.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 shutdown_connections 2024-04-17T10:31:25.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.563+0000 7f27aaa00640 1 -- 172.21.15.67:0/2521339232 wait complete. 2024-04-17T10:31:25.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8b200640 1 Processor -- start 2024-04-17T10:31:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8b200640 1 -- start start 2024-04-17T10:31:25.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8b200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9f8c15d950 0x7f9f8c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f8c070990 0x7f9f8c06b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c15dd30 con 0x7f9f8c0ce520 2024-04-17T10:31:25.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8c161640 con 0x7f9f8c070990 2024-04-17T10:31:25.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8b200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8c161920 con 0x7f9f8c15d950 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9f8c070990 0x7f9f8c06b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f83e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9f8c0ce520 0x7f9f8c070280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54790/0 (socket says 172.21.15.67:54790) 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f83e00640 1 -- 172.21.15.67:0/924057909 learned_addr learned my addr 172.21.15.67:0/924057909 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f89e00640 1 --2- 172.21.15.67:0/924057909 >> v2:172.21.15.86:3300/0 conn(0x7f9f8c15d950 0x7f9f8c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8a800640 1 -- 172.21.15.67:0/924057909 >> v2:172.21.15.86:3300/0 conn(0x7f9f8c15d950 msgr2=0x7f9f8c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8a800640 1 --2- 172.21.15.67:0/924057909 >> v2:172.21.15.86:3300/0 conn(0x7f9f8c15d950 0x7f9f8c06ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8a800640 1 -- 172.21.15.67:0/924057909 >> v1:172.21.15.67:6789/0 conn(0x7f9f8c0ce520 legacy=0x7f9f8c070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f8a800640 1 -- 172.21.15.67:0/924057909 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c161c00 con 0x7f9f8c070990 2024-04-17T10:31:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.723+0000 7f9f89400640 1 -- 172.21.15.67:0/924057909 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605273351 0 0) 0x7f9f8c15dd30 con 0x7f9f8c0ce520 2024-04-17T10:31:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8a800640 1 --2- 172.21.15.67:0/924057909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c06b1d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0b3340 tx=0x7f9f8401c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e78e88a0e0c903 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f89400640 1 -- 172.21.15.67:0/924057909 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840b2070 con 0x7f9f8c070990 2024-04-17T10:31:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f89400640 1 -- 172.21.15.67:0/924057909 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f840a59c0 con 0x7f9f8c070990 2024-04-17T10:31:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f89400640 1 -- 172.21.15.67:0/924057909 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f840a5d90 con 0x7f9f8c070990 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 -- 172.21.15.67:0/924057909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 msgr2=0x7f9f8c06b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 --2- 172.21.15.67:0/924057909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c06b1d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c0b3340 tx=0x7f9f8401c900 comp rx=0 tx=0).stop 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 -- 172.21.15.67:0/924057909 shutdown_connections 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 --2- 172.21.15.67:0/924057909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c06b1d0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 --2- 172.21.15.67:0/924057909 >> v2:172.21.15.86:3300/0 conn(0x7f9f8c15d950 0x7f9f8c06ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 -- 172.21.15.67:0/924057909 >> 172.21.15.67:0/924057909 conn(0x7f9f8c0b23f0 msgr2=0x7f9f8c0b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 -- 172.21.15.67:0/924057909 shutdown_connections 2024-04-17T10:31:25.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 -- 172.21.15.67:0/924057909 wait complete. 2024-04-17T10:31:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 Processor -- start 2024-04-17T10:31:25.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.727+0000 7f9f8b200640 1 -- start start 2024-04-17T10:31:25.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f8c0ce520 0x7f9f8c156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f8c15d950 0x7f9f8c14f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b4390 con 0x7f9f8c070990 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b4210 con 0x7f9f8c15d950 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b4510 con 0x7f9f8c0ce520 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c15bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54282/0 (socket says 172.21.15.67:54282) 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 -- 172.21.15.67:0/646840067 learned_addr learned my addr 172.21.15.67:0/646840067 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 -- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f8c0ce520 msgr2=0x7f9f8c156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f8c0ce520 0x7f9f8c156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 -- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f8c15d950 msgr2=0x7f9f8c14f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f8c15d950 0x7f9f8c14f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 -- 172.21.15.67:0/646840067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c0ca810 con 0x7f9f8c070990 2024-04-17T10:31:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f89e00640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c15bc40 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00d5f0 tx=0x7f9f7c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f82000640 1 -- 172.21.15.67:0/646840067 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c0125f0 con 0x7f9f8c070990 2024-04-17T10:31:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f82000640 1 -- 172.21.15.67:0/646840067 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c019070 con 0x7f9f8c070990 2024-04-17T10:31:25.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c15b160 con 0x7f9f8c070990 2024-04-17T10:31:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f82000640 1 -- 172.21.15.67:0/646840067 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7c0129b0 con 0x7f9f8c070990 2024-04-17T10:31:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c15b720 con 0x7f9f8c070990 2024-04-17T10:31:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8c074dc0 con 0x7f9f8c070990 2024-04-17T10:31:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.731+0000 7f9f82000640 1 -- 172.21.15.67:0/646840067 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9f7c012b50 con 0x7f9f8c070990 2024-04-17T10:31:25.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.735+0000 7f9f82000640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f68066f60 0x7f9f68069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:25.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.735+0000 7f9f82000640 1 -- 172.21.15.67:0/646840067 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7f9f7c0856f0 con 0x7f9f8c070990 2024-04-17T10:31:25.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.735+0000 7f9f83e00640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f68066f60 0x7f9f68069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:25.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.735+0000 7f9f83e00640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f68066f60 0x7f9f68069420 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9f740011f0 tx=0x7f9f74007930 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:25.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.739+0000 7f9f82000640 1 -- 172.21.15.67:0/646840067 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9f7c0564b0 con 0x7f9f8c070990 2024-04-17T10:31:25.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.899+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f8c06fdc0 con 0x7f9f8c070990 2024-04-17T10:31:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.899+0000 7f9f82000640 1 -- 172.21.15.67:0/646840067 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16963 (secure 0 0 0) 0x7f9f7c05ae20 con 0x7f9f8c070990 2024-04-17T10:31:25.892 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:25.892 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":85,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:08.910824+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:29:46.421958+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-04-17T10:29:46.986879+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":84,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":12,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f68066f60 msgr2=0x7f9f68069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f68066f60 0x7f9f68069420 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f9f740011f0 tx=0x7f9f74007930 comp rx=0 tx=0).stop 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 msgr2=0x7f9f8c15bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c15bc40 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c00d5f0 tx=0x7f9f7c00dab0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 shutdown_connections 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f68066f60 0x7f9f68069420 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f8c15d950 0x7f9f8c14f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f8c0ce520 0x7f9f8c156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 --2- 172.21.15.67:0/646840067 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f8c070990 0x7f9f8c15bc40 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 >> 172.21.15.67:0/646840067 conn(0x7f9f8c0b23f0 msgr2=0x7f9f8c0cff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 shutdown_connections 2024-04-17T10:31:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:25.903+0000 7f9f8b200640 1 -- 172.21.15.67:0/646840067 wait complete. 2024-04-17T10:31:25.914 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:31:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.063+0000 7fb40a400640 1 Processor -- start 2024-04-17T10:31:26.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.063+0000 7fb40a400640 1 -- start start 2024-04-17T10:31:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb40a400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb404145040 0x7fb40406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb40a400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404071a20 0x7fb4040cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb40a400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40406e2f0 con 0x7fb40406ef10 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb40a400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4040a86c0 con 0x7fb404145040 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb40a400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40415bf10 con 0x7fb404071a20 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb404145040 0x7fb40406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404071a20 0x7fb4040cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb404145040 0x7fb40406e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37716/0 (socket says 172.21.15.67:37716) 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403400640 1 -- 172.21.15.67:0/759630906 learned_addr learned my addr 172.21.15.67:0/759630906 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:26.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb3fbe00640 1 --1- 172.21.15.67:0/759630906 >> v1:172.21.15.86:6789/0 conn(0x7fb40406ef10 0x7fb404071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:59910/0 (socket says 172.21.15.67:59910) 2024-04-17T10:31:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403400640 1 -- 172.21.15.67:0/759630906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404071a20 msgr2=0x7fb4040cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403400640 1 --2- 172.21.15.67:0/759630906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404071a20 0x7fb4040cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403400640 1 -- 172.21.15.67:0/759630906 >> v1:172.21.15.86:6789/0 conn(0x7fb40406ef10 legacy=0x7fb404071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.067+0000 7fb403400640 1 -- 172.21.15.67:0/759630906 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40406d6e0 con 0x7fb404145040 2024-04-17T10:31:26.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb403400640 1 --2- 172.21.15.67:0/759630906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404145040 0x7fb40406e9d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009870 tx=0x7fb3f401ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d21753deaabf3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb402a00640 1 -- 172.21.15.67:0/759630906 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40b5070 con 0x7fb404145040 2024-04-17T10:31:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb402a00640 1 -- 172.21.15.67:0/759630906 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f40a8a50 con 0x7fb404145040 2024-04-17T10:31:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb402a00640 1 -- 172.21.15.67:0/759630906 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40a8e20 con 0x7fb404145040 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 -- 172.21.15.67:0/759630906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404145040 msgr2=0x7fb40406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 --2- 172.21.15.67:0/759630906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404145040 0x7fb40406e9d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4009870 tx=0x7fb3f401ed50 comp rx=0 tx=0).stop 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 -- 172.21.15.67:0/759630906 shutdown_connections 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 --2- 172.21.15.67:0/759630906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404071a20 0x7fb4040cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 --2- 172.21.15.67:0/759630906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404145040 0x7fb40406e9d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 -- 172.21.15.67:0/759630906 >> 172.21.15.67:0/759630906 conn(0x7fb40406ac90 msgr2=0x7fb40406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 -- 172.21.15.67:0/759630906 shutdown_connections 2024-04-17T10:31:26.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 -- 172.21.15.67:0/759630906 wait complete. 2024-04-17T10:31:26.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 Processor -- start 2024-04-17T10:31:26.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.071+0000 7fb40a400640 1 -- start start 2024-04-17T10:31:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40406ef10 0x7fb40415f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404071a20 0x7fb4040aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404145040 0x7fb4040aaa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40406df80 con 0x7fb40406ef10 2024-04-17T10:31:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40406dca0 con 0x7fb404071a20 2024-04-17T10:31:26.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40406d370 con 0x7fb404145040 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40406ef10 0x7fb40415f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40406ef10 0x7fb40415f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54284/0 (socket says 172.21.15.67:54284) 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 -- 172.21.15.67:0/4077986598 learned_addr learned my addr 172.21.15.67:0/4077986598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403e00640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404145040 0x7fb4040aaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 -- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404145040 msgr2=0x7fb4040aaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb3fbe00640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404071a20 0x7fb4040aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404145040 0x7fb4040aaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 -- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404071a20 msgr2=0x7fb4040aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404071a20 0x7fb4040aa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 -- 172.21.15.67:0/4077986598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4041f51c0 con 0x7fb40406ef10 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb3fbe00640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404071a20 0x7fb4040aa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403e00640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404145040 0x7fb4040aaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:26.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb403400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40406ef10 0x7fb40415f670 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb3f401d400 tx=0x7fb3f401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb3fb400640 1 -- 172.21.15.67:0/4077986598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40b5070 con 0x7fb40406ef10 2024-04-17T10:31:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb3fb400640 1 -- 172.21.15.67:0/4077986598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f40b0de0 con 0x7fb40406ef10 2024-04-17T10:31:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb3fb400640 1 -- 172.21.15.67:0/4077986598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40bb670 con 0x7fb40406ef10 2024-04-17T10:31:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4041f5490 con 0x7fb40406ef10 2024-04-17T10:31:26.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4041f5cd0 con 0x7fb40406ef10 2024-04-17T10:31:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb404072d50 con 0x7fb40406ef10 2024-04-17T10:31:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.075+0000 7fb3fb400640 1 -- 172.21.15.67:0/4077986598 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb3f40c2060 con 0x7fb40406ef10 2024-04-17T10:31:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.079+0000 7fb3fb400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3e0066f60 0x7fb3e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.079+0000 7fb3fb400640 1 -- 172.21.15.67:0/4077986598 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6129+0+0 (secure 0 0 0) 0x7fb3f4122650 con 0x7fb40406ef10 2024-04-17T10:31:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.079+0000 7fb3fbe00640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3e0066f60 0x7fb3e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.079+0000 7fb3fbe00640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3e0066f60 0x7fb3e0069420 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec007fd0 tx=0x7fb3ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:26.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.083+0000 7fb3fb400640 1 -- 172.21.15.67:0/4077986598 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb4041f5490 con 0x7fb40406ef10 2024-04-17T10:31:26.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.279+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb404074e30 con 0x7fb40406ef10 2024-04-17T10:31:26.269 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:31:26.279+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:31:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.279+0000 7fb3fb400640 1 -- 172.21.15.67:0/4077986598 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb3f40188b0 con 0x7fb40406ef10 2024-04-17T10:31:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3e0066f60 msgr2=0x7fb3e0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3e0066f60 0x7fb3e0069420 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec007fd0 tx=0x7fb3ec009040 comp rx=0 tx=0).stop 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40406ef10 msgr2=0x7fb40415f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40406ef10 0x7fb40415f670 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fb3f401d400 tx=0x7fb3f401baf0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 shutdown_connections 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3e0066f60 0x7fb3e0069420 secure :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec007fd0 tx=0x7fb3ec009040 comp rx=0 tx=0).stop 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb404145040 0x7fb4040aaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb404071a20 0x7fb4040aa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 --2- 172.21.15.67:0/4077986598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb40406ef10 0x7fb40415f670 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.283+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 >> 172.21.15.67:0/4077986598 conn(0x7fb40406ac90 msgr2=0x7fb40406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:26.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.287+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 shutdown_connections 2024-04-17T10:31:26.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.287+0000 7fb40a400640 1 -- 172.21.15.67:0/4077986598 wait complete. 2024-04-17T10:31:26.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:31:26.348 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:31:26.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.451+0000 7ff462e00640 1 Processor -- start 2024-04-17T10:31:26.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.451+0000 7ff462e00640 1 -- start start 2024-04-17T10:31:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff462e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff45c15b780 0x7ff45c0b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff462e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c06b3a0 0x7ff45c0d0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff462e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45c15e0d0 con 0x7ff45c0b1fc0 2024-04-17T10:31:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff462e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45c0b3cf0 con 0x7ff45c15b780 2024-04-17T10:31:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff462e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c15e390 con 0x7ff45c06b3a0 2024-04-17T10:31:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff45be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff45c15b780 0x7ff45c0b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff453400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff45c0b1fc0 0x7ff45c06ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54812/0 (socket says 172.21.15.67:54812) 2024-04-17T10:31:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff453400640 1 -- 172.21.15.67:0/1612247577 learned_addr learned my addr 172.21.15.67:0/1612247577 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff460e00640 1 --2- 172.21.15.67:0/1612247577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c06b3a0 0x7ff45c0d0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff45be00640 1 -- 172.21.15.67:0/1612247577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c06b3a0 msgr2=0x7ff45c0d0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff45be00640 1 --2- 172.21.15.67:0/1612247577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c06b3a0 0x7ff45c0d0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff45be00640 1 -- 172.21.15.67:0/1612247577 >> v1:172.21.15.67:6789/0 conn(0x7ff45c0b1fc0 legacy=0x7ff45c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff45be00640 1 -- 172.21.15.67:0/1612247577 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c15e670 con 0x7ff45c15b780 2024-04-17T10:31:26.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.455+0000 7ff45b400640 1 -- 172.21.15.67:0/1612247577 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021007879 0 0) 0x7ff45c15e0d0 con 0x7ff45c0b1fc0 2024-04-17T10:31:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff45be00640 1 --2- 172.21.15.67:0/1612247577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c0b37b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097c0 tx=0x7ff44c01c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1f317061524223 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff45b400640 1 -- 172.21.15.67:0/1612247577 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c0b0070 con 0x7ff45c15b780 2024-04-17T10:31:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff45b400640 1 -- 172.21.15.67:0/1612247577 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44c004230 con 0x7ff45c15b780 2024-04-17T10:31:26.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff45b400640 1 -- 172.21.15.67:0/1612247577 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44c003620 con 0x7ff45c15b780 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 -- 172.21.15.67:0/1612247577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 msgr2=0x7ff45c0b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 --2- 172.21.15.67:0/1612247577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c0b37b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0097c0 tx=0x7ff44c01c350 comp rx=0 tx=0).stop 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 -- 172.21.15.67:0/1612247577 shutdown_connections 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 --2- 172.21.15.67:0/1612247577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c06b3a0 0x7ff45c0d0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 --2- 172.21.15.67:0/1612247577 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c0b37b0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 -- 172.21.15.67:0/1612247577 >> 172.21.15.67:0/1612247577 conn(0x7ff45c0b5ef0 msgr2=0x7ff45c099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 -- 172.21.15.67:0/1612247577 shutdown_connections 2024-04-17T10:31:26.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 -- 172.21.15.67:0/1612247577 wait complete. 2024-04-17T10:31:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 Processor -- start 2024-04-17T10:31:26.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.459+0000 7ff462e00640 1 -- start start 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff45c06b3a0 0x7ff45c13c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c0b1fc0 0x7ff45c1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c143d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0d13e0 con 0x7ff45c15b780 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0d1260 con 0x7ff45c06b3a0 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c0d1560 con 0x7ff45c0b1fc0 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c143d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c143d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54312/0 (socket says 172.21.15.67:54312) 2024-04-17T10:31:26.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 -- 172.21.15.67:0/2885390626 learned_addr learned my addr 172.21.15.67:0/2885390626 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff45be00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff45c06b3a0 0x7ff45c13c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff453400640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c0b1fc0 0x7ff45c1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 -- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c0b1fc0 msgr2=0x7ff45c1587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c0b1fc0 0x7ff45c1587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 -- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff45c06b3a0 msgr2=0x7ff45c13c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff45c06b3a0 0x7ff45c13c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:26.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 -- 172.21.15.67:0/2885390626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c144700 con 0x7ff45c15b780 2024-04-17T10:31:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff460e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c143d20 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff454007e90 tx=0x7ff45400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff458c00640 1 -- 172.21.15.67:0/2885390626 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454014420 con 0x7ff45c15b780 2024-04-17T10:31:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff458c00640 1 -- 172.21.15.67:0/2885390626 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff454019070 con 0x7ff45c15b780 2024-04-17T10:31:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff458c00640 1 -- 172.21.15.67:0/2885390626 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff454014800 con 0x7ff45c15b780 2024-04-17T10:31:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c1449e0 con 0x7ff45c15b780 2024-04-17T10:31:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c13bfc0 con 0x7ff45c15b780 2024-04-17T10:31:26.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45c04ef30 con 0x7ff45c15b780 2024-04-17T10:31:26.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.463+0000 7ff458c00640 1 -- 172.21.15.67:0/2885390626 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff4540149a0 con 0x7ff45c15b780 2024-04-17T10:31:26.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.467+0000 7ff458c00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff438066f60 0x7ff438069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.467+0000 7ff458c00640 1 -- 172.21.15.67:0/2885390626 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff454085970 con 0x7ff45c15b780 2024-04-17T10:31:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.467+0000 7ff45be00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff438066f60 0x7ff438069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:26.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.467+0000 7ff45be00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff438066f60 0x7ff438069420 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0ad000 tx=0x7ff44c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:26.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.471+0000 7ff458c00640 1 -- 172.21.15.67:0/2885390626 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff4540560b0 con 0x7ff45c15b780 2024-04-17T10:31:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:26.671+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff45c06e640 con 0x7ff45c15b780 2024-04-17T10:31:27.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.283+0000 7ff458c00640 1 -- 172.21.15.67:0/2885390626 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff45405a150 con 0x7ff45c15b780 2024-04-17T10:31:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.283+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff438066f60 msgr2=0x7ff438069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:27.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.283+0000 7ff462e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff438066f60 0x7ff438069420 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff44c0ad000 tx=0x7ff44c008040 comp rx=0 tx=0).stop 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.283+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 msgr2=0x7ff45c143d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.283+0000 7ff462e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c143d20 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff454007e90 tx=0x7ff45400e4b0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 shutdown_connections 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff438066f60 0x7ff438069420 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45c15b780 0x7ff45c143d20 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45c0b1fc0 0x7ff45c1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 --2- 172.21.15.67:0/2885390626 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff45c06b3a0 0x7ff45c13c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 >> 172.21.15.67:0/2885390626 conn(0x7ff45c0b5ef0 msgr2=0x7ff45c099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 shutdown_connections 2024-04-17T10:31:27.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.287+0000 7ff462e00640 1 -- 172.21.15.67:0/2885390626 wait complete. 2024-04-17T10:31:27.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:31:27.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a2ba00640 1 Processor -- start 2024-04-17T10:31:27.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a2ba00640 1 -- start start 2024-04-17T10:31:27.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a2ba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a241440e0 0x7f7a24161680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:27.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a2ba00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7a240ae880 0x7f7a240aed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a2ba00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a240adde0 con 0x7f7a24161bc0 2024-04-17T10:31:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a2ba00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a24162060 con 0x7f7a240ae880 2024-04-17T10:31:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a2ba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a24162340 con 0x7f7a241440e0 2024-04-17T10:31:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a29000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a241440e0 0x7f7a24161680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a29a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7a240ae880 0x7f7a240aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a23e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7a24161bc0 0x7f7a240b2800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:59928/0 (socket says 172.21.15.67:59928) 2024-04-17T10:31:27.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a23e00640 1 -- 172.21.15.67:0/3094058531 learned_addr learned my addr 172.21.15.67:0/3094058531 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:27.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a29000640 1 -- 172.21.15.67:0/3094058531 >> v2:172.21.15.86:3300/0 conn(0x7f7a240ae880 msgr2=0x7f7a240aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:27.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a29000640 1 --2- 172.21.15.67:0/3094058531 >> v2:172.21.15.86:3300/0 conn(0x7f7a240ae880 0x7f7a240aed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a29000640 1 -- 172.21.15.67:0/3094058531 >> v1:172.21.15.86:6789/0 conn(0x7f7a24161bc0 legacy=0x7f7a240b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:27.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a29000640 1 -- 172.21.15.67:0/3094058531 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a240ae080 con 0x7f7a241440e0 2024-04-17T10:31:27.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.459+0000 7f7a23400640 1 -- 172.21.15.67:0/3094058531 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713752466 0 0) 0x7f7a240adde0 con 0x7f7a24161bc0 2024-04-17T10:31:27.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a29000640 1 --2- 172.21.15.67:0/3094058531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a241440e0 0x7f7a24161680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7a180097c0 tx=0x7f7a1801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ed71cb97dec1f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:27.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a23400640 1 -- 172.21.15.67:0/3094058531 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180b1070 con 0x7f7a241440e0 2024-04-17T10:31:27.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a23400640 1 -- 172.21.15.67:0/3094058531 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a180ac410 con 0x7f7a241440e0 2024-04-17T10:31:27.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a23400640 1 -- 172.21.15.67:0/3094058531 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180ac7e0 con 0x7f7a241440e0 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 -- 172.21.15.67:0/3094058531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a241440e0 msgr2=0x7f7a24161680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/3094058531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a241440e0 0x7f7a24161680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7a180097c0 tx=0x7f7a1801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 -- 172.21.15.67:0/3094058531 shutdown_connections 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/3094058531 >> v2:172.21.15.86:3300/0 conn(0x7f7a240ae880 0x7f7a240aed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/3094058531 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a241440e0 0x7f7a24161680 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 -- 172.21.15.67:0/3094058531 >> 172.21.15.67:0/3094058531 conn(0x7f7a2406ac90 msgr2=0x7f7a2406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 -- 172.21.15.67:0/3094058531 shutdown_connections 2024-04-17T10:31:27.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.463+0000 7f7a2ba00640 1 -- 172.21.15.67:0/3094058531 wait complete. 2024-04-17T10:31:27.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 Processor -- start 2024-04-17T10:31:27.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 -- start start 2024-04-17T10:31:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a240ae880 0x7f7a2415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a241440e0 0x7f7a241f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24161bc0 0x7f7a24167f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2406e750 con 0x7f7a24161bc0 2024-04-17T10:31:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2406e5d0 con 0x7f7a241440e0 2024-04-17T10:31:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2416a7d0 con 0x7f7a240ae880 2024-04-17T10:31:27.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24161bc0 0x7f7a24167f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24161bc0 0x7f7a24167f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54316/0 (socket says 172.21.15.67:54316) 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a240ae880 0x7f7a2415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29a00640 1 -- 172.21.15.67:0/2063551362 learned_addr learned my addr 172.21.15.67:0/2063551362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a23e00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a241440e0 0x7f7a241f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29000640 1 -- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a241440e0 msgr2=0x7f7a241f6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29000640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a241440e0 0x7f7a241f6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29000640 1 -- 172.21.15.67:0/2063551362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24161bc0 msgr2=0x7f7a24167f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29000640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24161bc0 0x7f7a24167f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29000640 1 -- 172.21.15.67:0/2063551362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2415c5c0 con 0x7f7a240ae880 2024-04-17T10:31:27.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a29000640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a240ae880 0x7f7a2415bfe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7a180ac000 tx=0x7f7a1801ace0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:27.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a20c00640 1 -- 172.21.15.67:0/2063551362 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a180b1050 con 0x7f7a240ae880 2024-04-17T10:31:27.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a20c00640 1 -- 172.21.15.67:0/2063551362 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a180b2040 con 0x7f7a240ae880 2024-04-17T10:31:27.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a20c00640 1 -- 172.21.15.67:0/2063551362 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a18006e20 con 0x7f7a240ae880 2024-04-17T10:31:27.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2415c8a0 con 0x7f7a240ae880 2024-04-17T10:31:27.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.467+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2415ce60 con 0x7f7a240ae880 2024-04-17T10:31:27.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.471+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a24072da0 con 0x7f7a240ae880 2024-04-17T10:31:27.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.471+0000 7f7a20c00640 1 -- 172.21.15.67:0/2063551362 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7a180bf060 con 0x7f7a240ae880 2024-04-17T10:31:27.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.471+0000 7f7a20c00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a04066f60 0x7f7a04069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:27.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.471+0000 7f7a20c00640 1 -- 172.21.15.67:0/2063551362 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7a1811ecf0 con 0x7f7a240ae880 2024-04-17T10:31:27.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.471+0000 7f7a23e00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a04066f60 0x7f7a04069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:27.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.471+0000 7f7a23e00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a04066f60 0x7f7a04069420 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7a240acec0 tx=0x7f7a10007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:27.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.475+0000 7f7a20c00640 1 -- 172.21.15.67:0/2063551362 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7a18016090 con 0x7f7a240ae880 2024-04-17T10:31:27.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:31:27.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:31:27.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:27.639+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7a24074e30 con 0x7f7a240ae880 2024-04-17T10:31:28.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.291+0000 7f7a20c00640 1 -- 172.21.15.67:0/2063551362 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v87) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7a180ef540 con 0x7f7a240ae880 2024-04-17T10:31:28.283 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a04066f60 msgr2=0x7f7a04069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a04066f60 0x7f7a04069420 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f7a240acec0 tx=0x7f7a10007df0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a240ae880 msgr2=0x7f7a2415bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a240ae880 0x7f7a2415bfe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7a180ac000 tx=0x7f7a1801ace0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 shutdown_connections 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a04066f60 0x7f7a04069420 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a24161bc0 0x7f7a24167f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a241440e0 0x7f7a241f6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 --2- 172.21.15.67:0/2063551362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a240ae880 0x7f7a2415bfe0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 >> 172.21.15.67:0/2063551362 conn(0x7f7a2406ac90 msgr2=0x7f7a2413cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 shutdown_connections 2024-04-17T10:31:28.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.295+0000 7f7a2ba00640 1 -- 172.21.15.67:0/2063551362 wait complete. 2024-04-17T10:31:28.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:31:28.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.507+0000 7f75dde00640 1 Processor -- start 2024-04-17T10:31:28.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.507+0000 7f75dde00640 1 -- start start 2024-04-17T10:31:28.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75dde00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 0x7f75d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75dde00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d806d9a0 con 0x7f75d8144f00 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75dde00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d806dc80 con 0x7f75d8071a70 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75dde00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80cc310 con 0x7f75d81447e0 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75d8144f00 0x7f75d8071360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54820/0 (socket says 172.21.15.67:54820) 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 -- 172.21.15.67:0/1545869703 learned_addr learned my addr 172.21.15.67:0/1545869703 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6a00640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 0x7f75d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270201045 0 0) 0x7f75d806d9a0 con 0x7f75d8144f00 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75bc003530 con 0x7f75d8144f00 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851118954 0 0) 0x7f75d806dc80 con 0x7f75d8071a70 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d806d9a0 con 0x7f75d8071a70 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2113471298 0 0) 0x7f75bc003530 con 0x7f75d8144f00 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 msgr2=0x7f75d806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:28.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 0x7f75d806d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 >> v1:172.21.15.86:6789/0 conn(0x7f75d8071a70 legacy=0x7f75d80cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d806df60 con 0x7f75d8144f00 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f75c8001680 con 0x7f75d8144f00 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6a00640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 0x7f75d806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 >> v1:172.21.15.67:6789/0 conn(0x7f75d8144f00 legacy=0x7f75d8071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 0x7f75bc003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75bc004110 0x7f75bc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004d50 0x7f75bc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80cc310 con 0x7f75bc004110 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc00bb90 con 0x7f75bc004d50 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc00bd90 con 0x7f75d81447e0 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75bc004110 0x7f75bc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d6a00640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 0x7f75bc003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75d7400640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004d50 0x7f75bc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 -- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 msgr2=0x7f75bc003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:28.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81447e0 0x7f75bc003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 -- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004d50 msgr2=0x7f75bc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:28.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004d50 0x7f75bc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.511+0000 7f75ce000640 1 -- 172.21.15.67:0/1545869703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc003530 con 0x7f75bc004110 2024-04-17T10:31:28.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75ce000640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75bc004110 0x7f75bc004810 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c0e0 tx=0x7f75c801cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f859de77c6e83ff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:28.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c8007c50 con 0x7f75bc004110 2024-04-17T10:31:28.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c80a9070 con 0x7f75bc004110 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75d6000640 1 -- 172.21.15.67:0/1545869703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c8005da0 con 0x7f75bc004110 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75dde00640 1 -- 172.21.15.67:0/1545869703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75bc004110 msgr2=0x7f75bc004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75dde00640 1 --2- 172.21.15.67:0/1545869703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75bc004110 0x7f75bc004810 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f75c800c0e0 tx=0x7f75c801cb10 comp rx=0 tx=0).stop 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75d7400640 1 -- 172.21.15.67:0/1545869703 reap_dead start 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75dde00640 1 -- 172.21.15.67:0/1545869703 shutdown_connections 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75dde00640 1 -- 172.21.15.67:0/1545869703 >> 172.21.15.67:0/1545869703 conn(0x7f75d806ac90 msgr2=0x7f75d806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75dde00640 1 -- 172.21.15.67:0/1545869703 shutdown_connections 2024-04-17T10:31:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.515+0000 7f75dde00640 1 -- 172.21.15.67:0/1545869703 wait complete. 2024-04-17T10:31:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 Processor -- start 2024-04-17T10:31:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 -- start start 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004110 0x7f75d815aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81377b0 0x7f75d80b1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d813d200 0x7f75d80b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80cda90 con 0x7f75d813d200 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d80cd910 con 0x7f75bc004110 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d8145530 con 0x7f75d81377b0 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d813d200 0x7f75d80b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d813d200 0x7f75d80b1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54346/0 (socket says 172.21.15.67:54346) 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 -- 172.21.15.67:0/3115573797 learned_addr learned my addr 172.21.15.67:0/3115573797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75ce000640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81377b0 0x7f75d80b1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d6a00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004110 0x7f75d815aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 -- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81377b0 msgr2=0x7f75d80b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:28.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81377b0 0x7f75d80b1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 -- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004110 msgr2=0x7f75d815aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:28.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004110 0x7f75d815aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:28.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 -- 172.21.15.67:0/3115573797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d8137d00 con 0x7f75d813d200 2024-04-17T10:31:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75d7400640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d813d200 0x7f75d80b1b80 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f75d000fae0 tx=0x7f75d0002130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75cfe00640 1 -- 172.21.15.67:0/3115573797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d0019070 con 0x7f75d813d200 2024-04-17T10:31:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75cfe00640 1 -- 172.21.15.67:0/3115573797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d0006e30 con 0x7f75d813d200 2024-04-17T10:31:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d8137fe0 con 0x7f75d813d200 2024-04-17T10:31:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75cfe00640 1 -- 172.21.15.67:0/3115573797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d0003b80 con 0x7f75d813d200 2024-04-17T10:31:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d80b31d0 con 0x7f75d813d200 2024-04-17T10:31:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.519+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d8072da0 con 0x7f75d813d200 2024-04-17T10:31:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.523+0000 7f75cfe00640 1 -- 172.21.15.67:0/3115573797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f75d001c070 con 0x7f75d813d200 2024-04-17T10:31:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.523+0000 7f75cfe00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b4066f60 0x7f75b4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.523+0000 7f75cfe00640 1 -- 172.21.15.67:0/3115573797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f75d00844a0 con 0x7f75d813d200 2024-04-17T10:31:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.523+0000 7f75d6a00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b4066f60 0x7f75b4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.523+0000 7f75d6a00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b4066f60 0x7f75b4069420 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009580 tx=0x7f75c0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:28.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.527+0000 7f75cfe00640 1 -- 172.21.15.67:0/3115573797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f75d0055470 con 0x7f75d813d200 2024-04-17T10:31:28.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:28.691+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f75d8143e70 con 0x7f75d813d200 2024-04-17T10:31:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.295+0000 7f75cfe00640 1 -- 172.21.15.67:0/3115573797 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7f75d0054d70 con 0x7f75d813d200 2024-04-17T10:31:29.286 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b4066f60 msgr2=0x7f75b4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b4066f60 0x7f75b4069420 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009580 tx=0x7f75c0008040 comp rx=0 tx=0).stop 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d813d200 msgr2=0x7f75d80b1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d813d200 0x7f75d80b1b80 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f75d000fae0 tx=0x7f75d0002130 comp rx=0 tx=0).stop 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 shutdown_connections 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75b4066f60 0x7f75b4069420 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75d813d200 0x7f75d80b1b80 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75d81377b0 0x7f75d80b1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 --2- 172.21.15.67:0/3115573797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75bc004110 0x7f75d815aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 >> 172.21.15.67:0/3115573797 conn(0x7f75d806ac90 msgr2=0x7f75d806e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:29.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 shutdown_connections 2024-04-17T10:31:29.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.299+0000 7f75dde00640 1 -- 172.21.15.67:0/3115573797 wait complete. 2024-04-17T10:31:29.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:31:29.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc9200640 1 Processor -- start 2024-04-17T10:31:29.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc9200640 1 -- start start 2024-04-17T10:31:29.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc4072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc9200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2fc40b5300 0x7f2fc406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc9200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc415f360 con 0x7f2fc40728c0 2024-04-17T10:31:29.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc9200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc40b3160 con 0x7f2fc40b5300 2024-04-17T10:31:29.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc9200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b3440 con 0x7f2fc41583f0 2024-04-17T10:31:29.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fb9400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2fc40728c0 0x7f2fc40b4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:54830/0 (socket says 172.21.15.67:54830) 2024-04-17T10:31:29.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fb9400640 1 -- 172.21.15.67:0/1347997949 learned_addr learned my addr 172.21.15.67:0/1347997949 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:29.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc2800640 1 --2- 172.21.15.67:0/1347997949 >> v2:172.21.15.67:3300/0 conn(0x7f2fc40b5300 0x7f2fc406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc1e00640 1 --2- 172.21.15.67:0/1347997949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc4072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc1400640 1 -- 172.21.15.67:0/1347997949 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685902973 0 0) 0x7f2fc415f360 con 0x7f2fc40728c0 2024-04-17T10:31:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc1400640 1 -- 172.21.15.67:0/1347997949 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa8003530 con 0x7f2fc40728c0 2024-04-17T10:31:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc2800640 1 -- 172.21.15.67:0/1347997949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 msgr2=0x7f2fc4072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc2800640 1 --2- 172.21.15.67:0/1347997949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc4072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc2800640 1 -- 172.21.15.67:0/1347997949 >> v1:172.21.15.67:6789/0 conn(0x7f2fc40728c0 legacy=0x7f2fc40b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.479+0000 7f2fc2800640 1 -- 172.21.15.67:0/1347997949 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40b3720 con 0x7f2fc40b5300 2024-04-17T10:31:29.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc2800640 1 --2- 172.21.15.67:0/1347997949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc406ac90 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009d90 tx=0x7f2fbc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c90de9243c562f20 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc1400640 1 -- 172.21.15.67:0/1347997949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0b2070 con 0x7f2fc40b5300 2024-04-17T10:31:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc1400640 1 -- 172.21.15.67:0/1347997949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc0a5890 con 0x7f2fc40b5300 2024-04-17T10:31:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc1400640 1 -- 172.21.15.67:0/1347997949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc0a5c60 con 0x7f2fc40b5300 2024-04-17T10:31:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 -- 172.21.15.67:0/1347997949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 msgr2=0x7f2fc406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1347997949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc406ac90 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc009d90 tx=0x7f2fbc01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 -- 172.21.15.67:0/1347997949 shutdown_connections 2024-04-17T10:31:29.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1347997949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc406ac90 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1347997949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc4072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 -- 172.21.15.67:0/1347997949 >> 172.21.15.67:0/1347997949 conn(0x7f2fc40b2570 msgr2=0x7f2fc4071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:29.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 -- 172.21.15.67:0/1347997949 shutdown_connections 2024-04-17T10:31:29.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.483+0000 7f2fc9200640 1 -- 172.21.15.67:0/1347997949 wait complete. 2024-04-17T10:31:29.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 Processor -- start 2024-04-17T10:31:29.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 -- start start 2024-04-17T10:31:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40728c0 0x7f2fc4150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc4149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc414a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc406b9e0 con 0x7f2fc40b5300 2024-04-17T10:31:29.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b4200 con 0x7f2fc40728c0 2024-04-17T10:31:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc406bb60 con 0x7f2fc41583f0 2024-04-17T10:31:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc4149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc4149f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54364/0 (socket says 172.21.15.67:54364) 2024-04-17T10:31:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40728c0 0x7f2fc4150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 -- 172.21.15.67:0/1281252133 learned_addr learned my addr 172.21.15.67:0/1281252133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:29.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc2800640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc414a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 -- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 msgr2=0x7f2fc414a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc414a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 -- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40728c0 msgr2=0x7f2fc4150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40728c0 0x7f2fc4150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 -- 172.21.15.67:0/1281252133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4155880 con 0x7f2fc40b5300 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fb9400640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc4149f90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400c3c0 tx=0x7f2fb400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fbaa00640 1 -- 172.21.15.67:0/1281252133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4017070 con 0x7f2fc40b5300 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fbaa00640 1 -- 172.21.15.67:0/1281252133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4005cf0 con 0x7f2fc40b5300 2024-04-17T10:31:29.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fbaa00640 1 -- 172.21.15.67:0/1281252133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb40072f0 con 0x7f2fc40b5300 2024-04-17T10:31:29.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc4155b60 con 0x7f2fc40b5300 2024-04-17T10:31:29.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.487+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc4156120 con 0x7f2fc40b5300 2024-04-17T10:31:29.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.491+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc404ef30 con 0x7f2fc40b5300 2024-04-17T10:31:29.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.491+0000 7f2fbaa00640 1 -- 172.21.15.67:0/1281252133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2fb401a070 con 0x7f2fc40b5300 2024-04-17T10:31:29.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.491+0000 7f2fbaa00640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fa0066f60 0x7f2fa0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.491+0000 7f2fbaa00640 1 -- 172.21.15.67:0/1281252133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f2fb40842f0 con 0x7f2fc40b5300 2024-04-17T10:31:29.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.491+0000 7f2fc1e00640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fa0066f60 0x7f2fa0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.491+0000 7f2fc1e00640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fa0066f60 0x7f2fa0069420 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4150eb0 tx=0x7f2fac009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:29.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.495+0000 7f2fbaa00640 1 -- 172.21.15.67:0/1281252133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2fb4054d80 con 0x7f2fc40b5300 2024-04-17T10:31:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.655+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2fc40af3b0 con 0x7f2fc40b5300 2024-04-17T10:31:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.659+0000 7f2fbaa00640 1 -- 172.21.15.67:0/1281252133 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fb405b0d0 con 0x7f2fc40b5300 2024-04-17T10:31:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fa0066f60 msgr2=0x7f2fa0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fa0066f60 0x7f2fa0069420 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4150eb0 tx=0x7f2fac009190 comp rx=0 tx=0).stop 2024-04-17T10:31:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 msgr2=0x7f2fc4149f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc4149f90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f2fb400c3c0 tx=0x7f2fb400c880 comp rx=0 tx=0).stop 2024-04-17T10:31:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 shutdown_connections 2024-04-17T10:31:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fa0066f60 0x7f2fa0069420 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2fc41583f0 0x7f2fc414a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2fc40b5300 0x7f2fc4149f90 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 --2- 172.21.15.67:0/1281252133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2fc40728c0 0x7f2fc4150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 >> 172.21.15.67:0/1281252133 conn(0x7f2fc40b2570 msgr2=0x7f2fc40ae060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 shutdown_connections 2024-04-17T10:31:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.663+0000 7f2fc9200640 1 -- 172.21.15.67:0/1281252133 wait complete. 2024-04-17T10:31:29.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub' 2024-04-17T10:31:29.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff49600640 1 Processor -- start 2024-04-17T10:31:29.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff49600640 1 -- start start 2024-04-17T10:31:29.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff49600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff44071310 0x7eff44071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff49600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff440cc300 0x7eff440cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff49600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4406e2f0 con 0x7eff44145040 2024-04-17T10:31:29.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff49600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff440a86c0 con 0x7eff44071310 2024-04-17T10:31:29.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff49600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff44071cb0 con 0x7eff440cc300 2024-04-17T10:31:29.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff3be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff44071310 0x7eff44071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff3be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7eff44071310 0x7eff44071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54370/0 (socket says 172.21.15.67:54370) 2024-04-17T10:31:29.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.823+0000 7eff3be00640 1 -- 172.21.15.67:0/2849047316 learned_addr learned my addr 172.21.15.67:0/2849047316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:29.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff43400640 1 --2- 172.21.15.67:0/2849047316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff440cc300 0x7eff440cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff3be00640 1 -- 172.21.15.67:0/2849047316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff440cc300 msgr2=0x7eff440cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:29.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff3be00640 1 --2- 172.21.15.67:0/2849047316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff440cc300 0x7eff440cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff3be00640 1 -- 172.21.15.67:0/2849047316 >> v1:172.21.15.86:6789/0 conn(0x7eff44145040 legacy=0x7eff4406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff3be00640 1 -- 172.21.15.67:0/2849047316 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4406d6e0 con 0x7eff44071310 2024-04-17T10:31:29.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff3be00640 1 --2- 172.21.15.67:0/2849047316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff44071310 0x7eff44071770 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009870 tx=0x7eff2c01ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1871b35b3fda560 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:29.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff42000640 1 -- 172.21.15.67:0/2849047316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0b5070 con 0x7eff44071310 2024-04-17T10:31:29.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff42000640 1 -- 172.21.15.67:0/2849047316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c0a8a50 con 0x7eff44071310 2024-04-17T10:31:29.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff42000640 1 -- 172.21.15.67:0/2849047316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0a8e20 con 0x7eff44071310 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff49600640 1 -- 172.21.15.67:0/2849047316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff44071310 msgr2=0x7eff44071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.827+0000 7eff49600640 1 --2- 172.21.15.67:0/2849047316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff44071310 0x7eff44071770 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009870 tx=0x7eff2c01ed50 comp rx=0 tx=0).stop 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- 172.21.15.67:0/2849047316 shutdown_connections 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 --2- 172.21.15.67:0/2849047316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff440cc300 0x7eff440cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 --2- 172.21.15.67:0/2849047316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff44071310 0x7eff44071770 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- 172.21.15.67:0/2849047316 >> 172.21.15.67:0/2849047316 conn(0x7eff4406ac90 msgr2=0x7eff4406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- 172.21.15.67:0/2849047316 shutdown_connections 2024-04-17T10:31:29.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- 172.21.15.67:0/2849047316 wait complete. 2024-04-17T10:31:29.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 Processor -- start 2024-04-17T10:31:29.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- start start 2024-04-17T10:31:29.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff44071310 0x7eff441e5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff440cc300 0x7eff44142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff44145040 0x7eff44144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4406df80 con 0x7eff440cc300 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4406dca0 con 0x7eff44071310 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff49600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4406d370 con 0x7eff44145040 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff440cc300 0x7eff44142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff440cc300 0x7eff44142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54378/0 (socket says 172.21.15.67:54378) 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 -- 172.21.15.67:0/3530171069 learned_addr learned my addr 172.21.15.67:0/3530171069 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff42a00640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff44071310 0x7eff441e5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff43400640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff44145040 0x7eff44144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 -- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff44145040 msgr2=0x7eff44144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff44145040 0x7eff44144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 -- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff44071310 msgr2=0x7eff441e5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff44071310 0x7eff441e5350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 -- 172.21.15.67:0/3530171069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff44144870 con 0x7eff440cc300 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff43400640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff44145040 0x7eff44144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.831+0000 7eff3be00640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff440cc300 0x7eff44142af0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0b3040 tx=0x7eff2c01baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff3aa00640 1 -- 172.21.15.67:0/3530171069 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0b5070 con 0x7eff440cc300 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff3aa00640 1 -- 172.21.15.67:0/3530171069 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c0b0de0 con 0x7eff440cc300 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff44143290 con 0x7eff440cc300 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff3aa00640 1 -- 172.21.15.67:0/3530171069 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0bb670 con 0x7eff440cc300 2024-04-17T10:31:29.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff44143850 con 0x7eff440cc300 2024-04-17T10:31:29.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff44072d50 con 0x7eff440cc300 2024-04-17T10:31:29.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff3aa00640 1 -- 172.21.15.67:0/3530171069 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7eff2c0c2060 con 0x7eff440cc300 2024-04-17T10:31:29.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff3aa00640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:29.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff3aa00640 1 -- 172.21.15.67:0/3530171069 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7eff2c122360 con 0x7eff440cc300 2024-04-17T10:31:29.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.835+0000 7eff42a00640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.839+0000 7eff42a00640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7eff34001770 tx=0x7eff34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:29.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:29.839+0000 7eff3aa00640 1 -- 172.21.15.67:0/3530171069 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7eff2c0188b0 con 0x7eff440cc300 2024-04-17T10:31:30.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.059+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]} v 0) v1 -- 0x7eff44074e30 con 0x7eff440cc300 2024-04-17T10:31:30.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.295+0000 7eff3aa00640 1 -- 172.21.15.67:0/3530171069 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]}]=0 v321) v1 ==== 158+0+0 (secure 0 0 0) 0x7eff2c0f34d0 con 0x7eff440cc300 2024-04-17T10:31:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 msgr2=0x7eff20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7eff34001770 tx=0x7eff34009040 comp rx=0 tx=0).stop 2024-04-17T10:31:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff440cc300 msgr2=0x7eff44142af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff440cc300 0x7eff44142af0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7eff2c0b3040 tx=0x7eff2c01baf0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 shutdown_connections 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff20066f60 0x7eff20069420 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff44145040 0x7eff44144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff440cc300 0x7eff44142af0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 --2- 172.21.15.67:0/3530171069 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff44071310 0x7eff441e5350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 >> 172.21.15.67:0/3530171069 conn(0x7eff4406ac90 msgr2=0x7eff4406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 shutdown_connections 2024-04-17T10:31:30.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.299+0000 7eff49600640 1 -- 172.21.15.67:0/3530171069 wait complete. 2024-04-17T10:31:30.311 INFO:tasks.cephfs_test_runner:test that scrub can iterate over ~mdsdir and evaluate strays ... ok 2024-04-17T10:31:30.311 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-04-17T10:31:30.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.459+0000 7f4d17e00640 1 Processor -- start 2024-04-17T10:31:30.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d17e00640 1 -- start start 2024-04-17T10:31:30.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d17e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d18070470 0x7f4d18099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d17e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d180cf330 con 0x7f4d1806f770 2024-04-17T10:31:30.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d17e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d180cf610 con 0x7f4d18099f90 2024-04-17T10:31:30.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d17e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d1809b190 con 0x7f4d18070470 2024-04-17T10:31:30.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d16a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d1806f770 0x7f4d1806fb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60446/0 (socket says 172.21.15.67:60446) 2024-04-17T10:31:30.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d16a00640 1 -- 172.21.15.67:0/1788810312 learned_addr learned my addr 172.21.15.67:0/1788810312 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:30.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d0fe00640 1 --2- 172.21.15.67:0/1788810312 >> v2:172.21.15.67:3300/0 conn(0x7f4d18070470 0x7f4d18099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d16000640 1 -- 172.21.15.67:0/1788810312 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288546193 0 0) 0x7f4d180cf330 con 0x7f4d1806f770 2024-04-17T10:31:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d16000640 1 -- 172.21.15.67:0/1788810312 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cfc003530 con 0x7f4d1806f770 2024-04-17T10:31:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d0fe00640 1 -- 172.21.15.67:0/1788810312 >> v1:172.21.15.86:6789/0 conn(0x7f4d18099f90 legacy=0x7f4d1809a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d0fe00640 1 -- 172.21.15.67:0/1788810312 >> v1:172.21.15.67:6789/0 conn(0x7f4d1806f770 legacy=0x7f4d1806fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.463+0000 7f4d0fe00640 1 -- 172.21.15.67:0/1788810312 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1809ac70 con 0x7f4d18070470 2024-04-17T10:31:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d0fe00640 1 --2- 172.21.15.67:0/1788810312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18099a50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4d000097c0 tx=0x7f4d0001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e874639ac0a6c25 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d16000640 1 -- 172.21.15.67:0/1788810312 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d000b0070 con 0x7f4d18070470 2024-04-17T10:31:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d16000640 1 -- 172.21.15.67:0/1788810312 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d000ab600 con 0x7f4d18070470 2024-04-17T10:31:30.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d16000640 1 -- 172.21.15.67:0/1788810312 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d000a4e60 con 0x7f4d18070470 2024-04-17T10:31:30.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 -- 172.21.15.67:0/1788810312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 msgr2=0x7f4d18099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1788810312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18099a50 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4d000097c0 tx=0x7f4d0001c3d0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 -- 172.21.15.67:0/1788810312 shutdown_connections 2024-04-17T10:31:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1788810312 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18099a50 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 -- 172.21.15.67:0/1788810312 >> 172.21.15.67:0/1788810312 conn(0x7f4d180d1630 msgr2=0x7f4d180d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 -- 172.21.15.67:0/1788810312 shutdown_connections 2024-04-17T10:31:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 -- 172.21.15.67:0/1788810312 wait complete. 2024-04-17T10:31:30.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 Processor -- start 2024-04-17T10:31:30.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.467+0000 7f4d17e00640 1 -- start start 2024-04-17T10:31:30.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1806f770 0x7f4d1809f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d18099f90 0x7f4d1809b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d18169760 con 0x7f4d18070470 2024-04-17T10:31:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d18071610 con 0x7f4d18099f90 2024-04-17T10:31:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d181698e0 con 0x7f4d1806f770 2024-04-17T10:31:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55082/0 (socket says 172.21.15.67:55082) 2024-04-17T10:31:30.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 -- 172.21.15.67:0/1263804200 learned_addr learned my addr 172.21.15.67:0/1263804200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:30.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 -- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1806f770 msgr2=0x7f4d1809f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1806f770 0x7f4d1809f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 -- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d18099f90 msgr2=0x7f4d1809b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d18099f90 0x7f4d1809b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 -- 172.21.15.67:0/1263804200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1809e1c0 con 0x7f4d18070470 2024-04-17T10:31:30.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0fe00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18150af0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4d00018fe0 tx=0x7f4d00019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:30.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0ea00640 1 -- 172.21.15.67:0/1263804200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d000b8070 con 0x7f4d18070470 2024-04-17T10:31:30.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0ea00640 1 -- 172.21.15.67:0/1263804200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d000a4e60 con 0x7f4d18070470 2024-04-17T10:31:30.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0ea00640 1 -- 172.21.15.67:0/1263804200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d000b0040 con 0x7f4d18070470 2024-04-17T10:31:30.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1809e4a0 con 0x7f4d18070470 2024-04-17T10:31:30.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1809ea60 con 0x7f4d18070470 2024-04-17T10:31:30.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d18074dc0 con 0x7f4d18070470 2024-04-17T10:31:30.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.471+0000 7f4d0ea00640 1 -- 172.21.15.67:0/1263804200 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4d000bb070 con 0x7f4d18070470 2024-04-17T10:31:30.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.475+0000 7f4d0ea00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.475+0000 7f4d0ea00640 1 -- 172.21.15.67:0/1263804200 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f4d0011ee60 con 0x7f4d18070470 2024-04-17T10:31:30.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.475+0000 7f4d16a00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.475+0000 7f4d16a00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4d08007fd0 tx=0x7f4d08007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:30.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.479+0000 7f4d0ea00640 1 -- 172.21.15.67:0/1263804200 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4d00016050 con 0x7f4d18070470 2024-04-17T10:31:30.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.639+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d180b24a0 con 0x7f4d18070470 2024-04-17T10:31:30.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.639+0000 7f4d0ea00640 1 -- 172.21.15.67:0/1263804200 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+13246 (secure 0 0 0) 0x7f4d000eff40 con 0x7f4d18070470 2024-04-17T10:31:30.631 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:30.631 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":88,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:29.287004+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6834/3374829114":"2024-04-18T10:31:26.280733+0000","172.21.15.67:6832/3374829114":"2024-04-18T10:31:26.280733+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:30.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 msgr2=0x7f4cf4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4d08007fd0 tx=0x7f4d08007900 comp rx=0 tx=0).stop 2024-04-17T10:31:30.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 msgr2=0x7f4d18150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18150af0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f4d00018fe0 tx=0x7f4d00019170 comp rx=0 tx=0).stop 2024-04-17T10:31:30.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 shutdown_connections 2024-04-17T10:31:30.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d18099f90 0x7f4d1809b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18070470 0x7f4d18150af0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 --2- 172.21.15.67:0/1263804200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1806f770 0x7f4d1809f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 >> 172.21.15.67:0/1263804200 conn(0x7f4d180d1630 msgr2=0x7f4d180ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:30.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 shutdown_connections 2024-04-17T10:31:30.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.643+0000 7f4d17e00640 1 -- 172.21.15.67:0/1263804200 wait complete. 2024-04-17T10:31:30.653 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:31:30.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:31:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f9410000640 1 Processor -- start 2024-04-17T10:31:30.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f9410000640 1 -- start start 2024-04-17T10:31:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f9410000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9408159440 0x7f94080ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f9410000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94080713e0 0x7f9408071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f9410000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940815cb30 con 0x7f940815be70 2024-04-17T10:31:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f9410000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94080a8650 con 0x7f94080713e0 2024-04-17T10:31:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f9410000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9408071d80 con 0x7f9408159440 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f940d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9408159440 0x7f94080ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.807+0000 7f940cc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f94080713e0 0x7f9408071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940e000640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f940815be70 0x7f940815c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:57680/0 (socket says 172.21.15.67:57680) 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940e000640 1 -- 172.21.15.67:0/2133555848 learned_addr learned my addr 172.21.15.67:0/2133555848 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940d600640 1 -- 172.21.15.67:0/2133555848 >> v2:172.21.15.86:3300/0 conn(0x7f94080713e0 msgr2=0x7f9408071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940d600640 1 --2- 172.21.15.67:0/2133555848 >> v2:172.21.15.86:3300/0 conn(0x7f94080713e0 0x7f9408071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940d600640 1 -- 172.21.15.67:0/2133555848 >> v1:172.21.15.86:6789/0 conn(0x7f940815be70 legacy=0x7f940815c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940d600640 1 -- 172.21.15.67:0/2133555848 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94080cd610 con 0x7f9408159440 2024-04-17T10:31:30.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940cc00640 1 --2- 172.21.15.67:0/2133555848 >> v2:172.21.15.86:3300/0 conn(0x7f94080713e0 0x7f9408071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:30.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f940d600640 1 --2- 172.21.15.67:0/2133555848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9408159440 0x7f94080ce0f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0097c0 tx=0x7f93fc01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeed56d1040bd2a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f9407e00640 1 -- 172.21.15.67:0/2133555848 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0b1070 con 0x7f9408159440 2024-04-17T10:31:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f9407e00640 1 -- 172.21.15.67:0/2133555848 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc0ac410 con 0x7f9408159440 2024-04-17T10:31:30.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.811+0000 7f9407e00640 1 -- 172.21.15.67:0/2133555848 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc0ac7e0 con 0x7f9408159440 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- 172.21.15.67:0/2133555848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9408159440 msgr2=0x7f94080ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 --2- 172.21.15.67:0/2133555848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9408159440 0x7f94080ce0f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0097c0 tx=0x7f93fc01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- 172.21.15.67:0/2133555848 shutdown_connections 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 --2- 172.21.15.67:0/2133555848 >> v2:172.21.15.86:3300/0 conn(0x7f94080713e0 0x7f9408071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 --2- 172.21.15.67:0/2133555848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9408159440 0x7f94080ce0f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- 172.21.15.67:0/2133555848 >> 172.21.15.67:0/2133555848 conn(0x7f94080b1740 msgr2=0x7f94080b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- 172.21.15.67:0/2133555848 shutdown_connections 2024-04-17T10:31:30.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- 172.21.15.67:0/2133555848 wait complete. 2024-04-17T10:31:30.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 Processor -- start 2024-04-17T10:31:30.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- start start 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94080713e0 0x7f94080cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9408159440 0x7f94081f4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f940815be70 0x7f9408155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940815d9c0 con 0x7f940815be70 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940815d840 con 0x7f9408159440 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f9410000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940815db40 con 0x7f94080713e0 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f940815be70 0x7f9408155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f940815be70 0x7f9408155420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55098/0 (socket says 172.21.15.67:55098) 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 -- 172.21.15.67:0/1320825937 learned_addr learned my addr 172.21.15.67:0/1320825937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940d600640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94080713e0 0x7f94080cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940cc00640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9408159440 0x7f94081f4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 -- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94080713e0 msgr2=0x7f94080cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94080713e0 0x7f94080cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 -- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9408159440 msgr2=0x7f94081f4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9408159440 0x7f94081f4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 -- 172.21.15.67:0/1320825937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408154660 con 0x7f940815be70 2024-04-17T10:31:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.815+0000 7f940e000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f940815be70 0x7f9408155420 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f940000ca80 tx=0x7f940000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9405600640 1 -- 172.21.15.67:0/1320825937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400017070 con 0x7f940815be70 2024-04-17T10:31:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9405600640 1 -- 172.21.15.67:0/1320825937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940000ac30 con 0x7f940815be70 2024-04-17T10:31:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9405600640 1 -- 172.21.15.67:0/1320825937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9400012600 con 0x7f940815be70 2024-04-17T10:31:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9408154940 con 0x7f940815be70 2024-04-17T10:31:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9408154f00 con 0x7f940815be70 2024-04-17T10:31:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940804ef30 con 0x7f940815be70 2024-04-17T10:31:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9405600640 1 -- 172.21.15.67:0/1320825937 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f940001a070 con 0x7f940815be70 2024-04-17T10:31:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9405600640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e8066f60 0x7f93e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f9405600640 1 -- 172.21.15.67:0/1320825937 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f9400084a00 con 0x7f940815be70 2024-04-17T10:31:30.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.819+0000 7f940d600640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e8066f60 0x7f93e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:30.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.823+0000 7f940d600640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e8066f60 0x7f93e8069420 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0ae000 tx=0x7f93fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:30.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.823+0000 7f9405600640 1 -- 172.21.15.67:0/1320825937 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f940000ae20 con 0x7f940815be70 2024-04-17T10:31:30.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:30.999+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94080739b0 con 0x7f940815be70 2024-04-17T10:31:30.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9405600640 1 -- 172.21.15.67:0/1320825937 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f94000555c0 con 0x7f940815be70 2024-04-17T10:31:30.993 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:31:30.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e8066f60 msgr2=0x7f93e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e8066f60 0x7f93e8069420 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0ae000 tx=0x7f93fc008040 comp rx=0 tx=0).stop 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f940815be70 msgr2=0x7f9408155420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f940815be70 0x7f9408155420 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f940000ca80 tx=0x7f940000cf40 comp rx=0 tx=0).stop 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 shutdown_connections 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f93e8066f60 0x7f93e8069420 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f940815be70 0x7f9408155420 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9408159440 0x7f94081f4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 --2- 172.21.15.67:0/1320825937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f94080713e0 0x7f94080cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 >> 172.21.15.67:0/1320825937 conn(0x7f94080b1740 msgr2=0x7f94080b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 shutdown_connections 2024-04-17T10:31:30.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.003+0000 7f9410000640 1 -- 172.21.15.67:0/1320825937 wait complete. 2024-04-17T10:31:31.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort' 2024-04-17T10:31:31.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feadca00640 1 Processor -- start 2024-04-17T10:31:31.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feadca00640 1 -- start start 2024-04-17T10:31:31.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feadca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead815dd10 0x7fead80b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feadca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fead80b57b0 0x7fead8070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feadca00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fead80b0cc0 con 0x7fead80b07b0 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feadca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fead80b1280 con 0x7fead80b57b0 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feadca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead80b0fa0 con 0x7fead815dd10 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.163+0000 7feacd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fead80b57b0 0x7fead8070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7feacd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fead80b57b0 0x7fead8070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55106/0 (socket says 172.21.15.67:55106) 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7feacd400640 1 -- 172.21.15.67:0/3955606150 learned_addr learned my addr 172.21.15.67:0/3955606150 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7fead5e00640 1 --2- 172.21.15.67:0/3955606150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead815dd10 0x7fead80b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7feacd400640 1 -- 172.21.15.67:0/3955606150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead815dd10 msgr2=0x7fead80b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7feacd400640 1 --2- 172.21.15.67:0/3955606150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead815dd10 0x7fead80b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7feacd400640 1 -- 172.21.15.67:0/3955606150 >> v1:172.21.15.86:6789/0 conn(0x7fead80b07b0 legacy=0x7fead81618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:31.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7feacd400640 1 -- 172.21.15.67:0/3955606150 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead806bed0 con 0x7fead80b57b0 2024-04-17T10:31:31.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7feacd400640 1 --2- 172.21.15.67:0/3955606150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead80b57b0 0x7fead8070dc0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7feac0009e50 tx=0x7feac001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c41b0c32874851 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7fead5400640 1 -- 172.21.15.67:0/3955606150 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00a5790 con 0x7fead80b57b0 2024-04-17T10:31:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7fead5400640 1 -- 172.21.15.67:0/3955606150 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac00a5930 con 0x7fead80b57b0 2024-04-17T10:31:31.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.167+0000 7fead5400640 1 -- 172.21.15.67:0/3955606150 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00a5d00 con 0x7fead80b57b0 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- 172.21.15.67:0/3955606150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead80b57b0 msgr2=0x7fead8070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 --2- 172.21.15.67:0/3955606150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead80b57b0 0x7fead8070dc0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7feac0009e50 tx=0x7feac001c930 comp rx=0 tx=0).stop 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- 172.21.15.67:0/3955606150 shutdown_connections 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 --2- 172.21.15.67:0/3955606150 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead80b57b0 0x7fead8070dc0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 --2- 172.21.15.67:0/3955606150 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead815dd10 0x7fead80b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- 172.21.15.67:0/3955606150 >> 172.21.15.67:0/3955606150 conn(0x7fead80b34d0 msgr2=0x7fead80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- 172.21.15.67:0/3955606150 shutdown_connections 2024-04-17T10:31:31.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- 172.21.15.67:0/3955606150 wait complete. 2024-04-17T10:31:31.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 Processor -- start 2024-04-17T10:31:31.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- start start 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead80b07b0 0x7fead80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead815dd10 0x7fead8153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fead8154370 0x7fead80cfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead816ae70 con 0x7fead815dd10 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead8071710 con 0x7fead8154370 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feadca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead816aff0 con 0x7fead80b07b0 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead815dd10 0x7fead8153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead815dd10 0x7fead8153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55112/0 (socket says 172.21.15.67:55112) 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 -- 172.21.15.67:0/2915681026 learned_addr learned my addr 172.21.15.67:0/2915681026 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:31.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7fead6800640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fead8154370 0x7fead80cfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7fead5e00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead80b07b0 0x7fead80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 -- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead80b07b0 msgr2=0x7fead80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead80b07b0 0x7fead80b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 -- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fead8154370 msgr2=0x7fead80cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fead8154370 0x7fead80cfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 -- 172.21.15.67:0/2915681026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8152170 con 0x7fead815dd10 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7fead6800640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fead8154370 0x7fead80cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:31.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.171+0000 7feacd400640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead815dd10 0x7fead8153e30 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7feac00af040 tx=0x7feac00196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feacea00640 1 -- 172.21.15.67:0/2915681026 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00bc020 con 0x7fead815dd10 2024-04-17T10:31:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feacea00640 1 -- 172.21.15.67:0/2915681026 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac00b8690 con 0x7fead815dd10 2024-04-17T10:31:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feacea00640 1 -- 172.21.15.67:0/2915681026 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac00b8a80 con 0x7fead815dd10 2024-04-17T10:31:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fead8152450 con 0x7fead815dd10 2024-04-17T10:31:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fead8152a10 con 0x7fead815dd10 2024-04-17T10:31:31.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fead8154780 con 0x7fead815dd10 2024-04-17T10:31:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feacea00640 1 -- 172.21.15.67:0/2915681026 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7feac00bf020 con 0x7fead815dd10 2024-04-17T10:31:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feacea00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feab4066f60 0x7feab4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7feacea00640 1 -- 172.21.15.67:0/2915681026 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7feac01200b0 con 0x7fead815dd10 2024-04-17T10:31:31.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.175+0000 7fead5e00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feab4066f60 0x7feab4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:31.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.179+0000 7fead5e00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feab4066f60 0x7feab4069420 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7feac800b930 tx=0x7feac8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:31.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.179+0000 7feacea00640 1 -- 172.21.15.67:0/2915681026 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7feac00c6030 con 0x7fead815dd10 2024-04-17T10:31:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:31.399+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]} v 0) v1 -- 0x7fead8070bb0 con 0x7fead815dd10 2024-04-17T10:31:32.295 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.303+0000 7feacea00640 1 -- 172.21.15.67:0/2915681026 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]}]=0 v323) v1 ==== 147+0+0 (secure 0 0 0) 0x7feac00165e0 con 0x7fead815dd10 2024-04-17T10:31:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feab4066f60 msgr2=0x7feab4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feab4066f60 0x7feab4069420 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7feac800b930 tx=0x7feac8008040 comp rx=0 tx=0).stop 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead815dd10 msgr2=0x7fead8153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead815dd10 0x7fead8153e30 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7feac00af040 tx=0x7feac00196d0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 shutdown_connections 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feab4066f60 0x7feab4069420 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fead8154370 0x7fead80cfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fead815dd10 0x7fead8153e30 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 --2- 172.21.15.67:0/2915681026 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fead80b07b0 0x7fead80b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 >> 172.21.15.67:0/2915681026 conn(0x7fead80b34d0 msgr2=0x7fead80b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 shutdown_connections 2024-04-17T10:31:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.307+0000 7feadca00640 1 -- 172.21.15.67:0/2915681026 wait complete. 2024-04-17T10:31:32.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:31:32.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.483+0000 7f4caf800640 1 Processor -- start 2024-04-17T10:31:32.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.483+0000 7f4caf800640 1 -- start start 2024-04-17T10:31:32.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4caf800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca80cd030 0x7f4ca806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4caf800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ca806e7b0 0x7f4ca80b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4caf800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80b4990 con 0x7f4ca80aef60 2024-04-17T10:31:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4caf800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80af340 con 0x7f4ca806e7b0 2024-04-17T10:31:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4caf800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca815ae90 con 0x7f4ca80cd030 2024-04-17T10:31:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4cad800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4ca806e7b0 0x7f4ca80b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4cace00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4ca80aef60 0x7f4ca80cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60462/0 (socket says 172.21.15.67:60462) 2024-04-17T10:31:32.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4cace00640 1 -- 172.21.15.67:0/2165397160 learned_addr learned my addr 172.21.15.67:0/2165397160 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:32.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4c9fe00640 1 --2- 172.21.15.67:0/2165397160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca80cd030 0x7f4ca806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:32.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4cad800640 1 -- 172.21.15.67:0/2165397160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca80cd030 msgr2=0x7f4ca806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:32.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4cad800640 1 --2- 172.21.15.67:0/2165397160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca80cd030 0x7f4ca806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4cad800640 1 -- 172.21.15.67:0/2165397160 >> v1:172.21.15.67:6789/0 conn(0x7f4ca80aef60 legacy=0x7f4ca80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4ca7e00640 1 -- 172.21.15.67:0/2165397160 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3850119254 0 0) 0x7f4c98001120 con 0x7f4ca80aef60 2024-04-17T10:31:32.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.487+0000 7f4cad800640 1 -- 172.21.15.67:0/2165397160 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca806ee80 con 0x7f4ca806e7b0 2024-04-17T10:31:32.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4cad800640 1 --2- 172.21.15.67:0/2165397160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca806e7b0 0x7f4ca80b3120 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009d90 tx=0x7f4ca001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb3a459c30b667 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4ca7e00640 1 -- 172.21.15.67:0/2165397160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca00b2070 con 0x7f4ca806e7b0 2024-04-17T10:31:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4ca7e00640 1 -- 172.21.15.67:0/2165397160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca00a5890 con 0x7f4ca806e7b0 2024-04-17T10:31:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4ca7e00640 1 -- 172.21.15.67:0/2165397160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca00a5c60 con 0x7f4ca806e7b0 2024-04-17T10:31:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 -- 172.21.15.67:0/2165397160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca806e7b0 msgr2=0x7f4ca80b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 --2- 172.21.15.67:0/2165397160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca806e7b0 0x7f4ca80b3120 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f4ca0009d90 tx=0x7f4ca001c8e0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 -- 172.21.15.67:0/2165397160 shutdown_connections 2024-04-17T10:31:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 --2- 172.21.15.67:0/2165397160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca806e7b0 0x7f4ca80b3120 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 --2- 172.21.15.67:0/2165397160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca80cd030 0x7f4ca806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 -- 172.21.15.67:0/2165397160 >> 172.21.15.67:0/2165397160 conn(0x7f4ca80b1740 msgr2=0x7f4ca80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 -- 172.21.15.67:0/2165397160 shutdown_connections 2024-04-17T10:31:32.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 -- 172.21.15.67:0/2165397160 wait complete. 2024-04-17T10:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 Processor -- start 2024-04-17T10:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.491+0000 7f4caf800640 1 -- start start 2024-04-17T10:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca806e7b0 0x7f4ca814f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:32.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca80aef60 0x7f4ca8159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca80cd030 0x7f4ca8157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca815bc10 con 0x7f4ca80aef60 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80b3e70 con 0x7f4ca80cd030 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca81f4bc0 con 0x7f4ca806e7b0 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca80aef60 0x7f4ca8159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca80aef60 0x7f4ca8159dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55134/0 (socket says 172.21.15.67:55134) 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 -- 172.21.15.67:0/1015271559 learned_addr learned my addr 172.21.15.67:0/1015271559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4cad800640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca80cd030 0x7f4ca8157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4cace00640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca806e7b0 0x7f4ca814f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 -- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca806e7b0 msgr2=0x7f4ca814f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca806e7b0 0x7f4ca814f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 -- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca80cd030 msgr2=0x7f4ca8157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca80cd030 0x7f4ca8157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 -- 172.21.15.67:0/1015271559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca81f8600 con 0x7f4ca80aef60 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4c9fe00640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca80aef60 0x7f4ca8159dc0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000d5f0 tx=0x7f4c9000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4ca5600640 1 -- 172.21.15.67:0/1015271559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c900125f0 con 0x7f4ca80aef60 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4ca5600640 1 -- 172.21.15.67:0/1015271559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c90019070 con 0x7f4ca80aef60 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca81f8940 con 0x7f4ca80aef60 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4ca5600640 1 -- 172.21.15.67:0/1015271559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c900129b0 con 0x7f4ca80aef60 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca81f81b0 con 0x7f4ca80aef60 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca804ef30 con 0x7f4ca80aef60 2024-04-17T10:31:32.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.495+0000 7f4ca5600640 1 -- 172.21.15.67:0/1015271559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4c90012b50 con 0x7f4ca80aef60 2024-04-17T10:31:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.499+0000 7f4ca5600640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c84066f90 0x7f4c84069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.499+0000 7f4ca5600640 1 -- 172.21.15.67:0/1015271559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f4c900850e0 con 0x7f4ca80aef60 2024-04-17T10:31:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.499+0000 7f4cace00640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c84066f90 0x7f4c84069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:32.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.499+0000 7f4cace00640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c84066f90 0x7f4c84069450 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000f70 tx=0x7f4c98007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:32.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.503+0000 7f4ca5600640 1 -- 172.21.15.67:0/1015271559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4ca81f8940 con 0x7f4ca80aef60 2024-04-17T10:31:32.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.667+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4ca815ac70 con 0x7f4ca80aef60 2024-04-17T10:31:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4ca5600640 1 -- 172.21.15.67:0/1015271559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4c90055c00 con 0x7f4ca80aef60 2024-04-17T10:31:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c84066f90 msgr2=0x7f4c84069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c84066f90 0x7f4c84069450 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4c98000f70 tx=0x7f4c98007b60 comp rx=0 tx=0).stop 2024-04-17T10:31:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca80aef60 msgr2=0x7f4ca8159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca80aef60 0x7f4ca8159dc0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000d5f0 tx=0x7f4c9000dab0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 shutdown_connections 2024-04-17T10:31:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c84066f90 0x7f4c84069450 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4ca80cd030 0x7f4ca8157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ca80aef60 0x7f4ca8159dc0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 --2- 172.21.15.67:0/1015271559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4ca806e7b0 0x7f4ca814f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 >> 172.21.15.67:0/1015271559 conn(0x7f4ca80b1740 msgr2=0x7f4ca80cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 shutdown_connections 2024-04-17T10:31:32.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:32.671+0000 7f4caf800640 1 -- 172.21.15.67:0/1015271559 wait complete. 2024-04-17T10:31:32.682 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:31:32.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:31:32.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:31:32.856 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:31:32.856 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:31:32.857 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:32.867 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:31:32.867 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:31:32.867 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:32.876 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:31:32.877 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:31:32.877 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:32.916 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:31:32.916 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:31:32.916 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:32.925 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:31:32.925 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:31:32.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:31:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.107+0000 7fdc2b200640 1 Processor -- start 2024-04-17T10:31:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.107+0000 7fdc2b200640 1 -- start start 2024-04-17T10:31:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc2b200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc2406c410 0x7fdc2406f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc2b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc240720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc2b200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc240ad8f0 con 0x7fdc2415dbc0 2024-04-17T10:31:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc2b200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc2415dfa0 con 0x7fdc2406c410 2024-04-17T10:31:33.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc2b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2406faa0 con 0x7fdc24071c60 2024-04-17T10:31:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc1b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc2406c410 0x7fdc2406f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc1b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc2406c410 0x7fdc2406f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55142/0 (socket says 172.21.15.67:55142) 2024-04-17T10:31:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc1b400640 1 -- 172.21.15.67:0/2404266589 learned_addr learned my addr 172.21.15.67:0/2404266589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc29200640 1 --2- 172.21.15.67:0/2404266589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc240720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc1b400640 1 -- 172.21.15.67:0/2404266589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 msgr2=0x7fdc240720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc1b400640 1 --2- 172.21.15.67:0/2404266589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc240720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc1b400640 1 -- 172.21.15.67:0/2404266589 >> v1:172.21.15.86:6789/0 conn(0x7fdc2415dbc0 legacy=0x7fdc2406db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc1b400640 1 -- 172.21.15.67:0/2404266589 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc240adbd0 con 0x7fdc2406c410 2024-04-17T10:31:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.111+0000 7fdc29200640 1 --2- 172.21.15.67:0/2404266589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc240720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc1b400640 1 --2- 172.21.15.67:0/2404266589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2406f560 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009870 tx=0x7fdc0c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e222216054f57fd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc23400640 1 -- 172.21.15.67:0/2404266589 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0b0070 con 0x7fdc2406c410 2024-04-17T10:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc23400640 1 -- 172.21.15.67:0/2404266589 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0c0ab600 con 0x7fdc2406c410 2024-04-17T10:31:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc23400640 1 -- 172.21.15.67:0/2404266589 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c0ab9d0 con 0x7fdc2406c410 2024-04-17T10:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 -- 172.21.15.67:0/2404266589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 msgr2=0x7fdc2406f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2404266589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2406f560 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009870 tx=0x7fdc0c01c490 comp rx=0 tx=0).stop 2024-04-17T10:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 -- 172.21.15.67:0/2404266589 shutdown_connections 2024-04-17T10:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2404266589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc240720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2404266589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2406f560 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 -- 172.21.15.67:0/2404266589 >> 172.21.15.67:0/2404266589 conn(0x7fdc240af670 msgr2=0x7fdc240b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 -- 172.21.15.67:0/2404266589 shutdown_connections 2024-04-17T10:31:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 -- 172.21.15.67:0/2404266589 wait complete. 2024-04-17T10:31:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 Processor -- start 2024-04-17T10:31:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.115+0000 7fdc2b200640 1 -- start start 2024-04-17T10:31:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2415af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc24158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc2415dbc0 0x7fdc24141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2416a100 con 0x7fdc2406c410 2024-04-17T10:31:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2406af70 con 0x7fdc2415dbc0 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2416a280 con 0x7fdc24071c60 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2415af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2415af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55146/0 (socket says 172.21.15.67:55146) 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 -- 172.21.15.67:0/2818443409 learned_addr learned my addr 172.21.15.67:0/2818443409 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc29200640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc2415dbc0 0x7fdc24141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc1b400640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc24158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 -- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 msgr2=0x7fdc24158a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc24158a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 -- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc2415dbc0 msgr2=0x7fdc24141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc2415dbc0 0x7fdc24141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 -- 172.21.15.67:0/2818443409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2415b650 con 0x7fdc2406c410 2024-04-17T10:31:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc1b400640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc24158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc23e00640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2415af00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400c450 tx=0x7fdc1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc20c00640 1 -- 172.21.15.67:0/2818443409 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14012830 con 0x7fdc2406c410 2024-04-17T10:31:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc20c00640 1 -- 172.21.15.67:0/2818443409 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc14017070 con 0x7fdc2406c410 2024-04-17T10:31:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2415b930 con 0x7fdc2406c410 2024-04-17T10:31:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc20c00640 1 -- 172.21.15.67:0/2818443409 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc14012c30 con 0x7fdc2406c410 2024-04-17T10:31:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2415bef0 con 0x7fdc2406c410 2024-04-17T10:31:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc2409e260 con 0x7fdc2406c410 2024-04-17T10:31:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.119+0000 7fdc20c00640 1 -- 172.21.15.67:0/2818443409 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdc1401c460 con 0x7fdc2406c410 2024-04-17T10:31:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.123+0000 7fdc20c00640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdc00066f60 0x7fdc00069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.123+0000 7fdc20c00640 1 -- 172.21.15.67:0/2818443409 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fdc14084c30 con 0x7fdc2406c410 2024-04-17T10:31:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.123+0000 7fdc1b400640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdc00066f60 0x7fdc00069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.123+0000 7fdc1b400640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdc00066f60 0x7fdc00069420 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0ad000 tx=0x7fdc0c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:33.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.127+0000 7fdc20c00640 1 -- 172.21.15.67:0/2818443409 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdc1408b080 con 0x7fdc2406c410 2024-04-17T10:31:33.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.319+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc24074e30 con 0x7fdc2406c410 2024-04-17T10:31:33.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.319+0000 7fdc20c00640 1 -- 172.21.15.67:0/2818443409 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+3693 (secure 0 0 0) 0x7fdc14055d90 con 0x7fdc2406c410 2024-04-17T10:31:33.312 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:33.312 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/3654887462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3654887462},{"type":"v1","addr":"172.21.15.86:6836","nonce":3654887462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6814,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1627300843","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1627300843},{"type":"v1","addr":"172.21.15.67:6835","nonce":1627300843}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":6820,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/39011498","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":39011498},{"type":"v1","addr":"172.21.15.86:6837","nonce":39011498}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51},{"gid":7617,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/1358702045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1358702045},{"type":"v1","addr":"172.21.15.67:6834","nonce":1358702045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58}],"filesystems":[]} 2024-04-17T10:31:33.313 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 58 2024-04-17T10:31:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdc00066f60 msgr2=0x7fdc00069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdc00066f60 0x7fdc00069420 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c0ad000 tx=0x7fdc0c0a6000 comp rx=0 tx=0).stop 2024-04-17T10:31:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 msgr2=0x7fdc2415af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2415af00 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fdc1400c450 tx=0x7fdc1400c910 comp rx=0 tx=0).stop 2024-04-17T10:31:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 shutdown_connections 2024-04-17T10:31:33.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdc00066f60 0x7fdc00069420 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdc2415dbc0 0x7fdc24141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdc24071c60 0x7fdc24158a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 --2- 172.21.15.67:0/2818443409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc2406c410 0x7fdc2415af00 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 >> 172.21.15.67:0/2818443409 conn(0x7fdc240af670 msgr2=0x7fdc240ce340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 shutdown_connections 2024-04-17T10:31:33.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.323+0000 7fdc2b200640 1 -- 172.21.15.67:0/2818443409 wait complete. 2024-04-17T10:31:33.334 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:31:33.334 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:31:33.334 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:31:33.334 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:31:33.334 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:31:33.335 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:31:33.335 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:31:33.335 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:31:33.335 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:31:33.335 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:31:33.335 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:31:33.335 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:31:33.336 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:31:33.336 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:31:33.336 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:31:33.336 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:31:33.346 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:31:33.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:31:33.347 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:31:33.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:31:33.349 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:31:33.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:31:33.351 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:31:33.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:31:33.391 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:31:33.391 DEBUG:teuthology.parallel:result is None 2024-04-17T10:31:33.392 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:31:33.392 DEBUG:teuthology.parallel:result is None 2024-04-17T10:31:33.395 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:31:33.395 DEBUG:teuthology.parallel:result is None 2024-04-17T10:31:33.397 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:31:33.397 DEBUG:teuthology.parallel:result is None 2024-04-17T10:31:33.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:31:33.478 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:31:33.479 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:31:33.486 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:31:33.487 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:31:33.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97e00640 1 Processor -- start 2024-04-17T10:31:33.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97e00640 1 -- start start 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e980d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e980d33d0 0x7f0e980b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e980d2330 con 0x7f0e98160f30 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e980b9910 con 0x7f0e980d33d0 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980d2610 con 0x7f0e980b5a80 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e980d33d0 0x7f0e980b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e96000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e980d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0e980d33d0 0x7f0e980b93d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55194/0 (socket says 172.21.15.67:55194) 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97400640 1 -- 172.21.15.67:0/2762474746 learned_addr learned my addr 172.21.15.67:0/2762474746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e96a00640 1 --1- 172.21.15.67:0/2762474746 >> v1:172.21.15.86:6789/0 conn(0x7f0e98160f30 0x7f0e980b5370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:57710/0 (socket says 172.21.15.67:57710) 2024-04-17T10:31:33.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e95600640 1 -- 172.21.15.67:0/2762474746 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945868041 0 0) 0x7f0e980d2330 con 0x7f0e98160f30 2024-04-17T10:31:33.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97400640 1 -- 172.21.15.67:0/2762474746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 msgr2=0x7f0e980d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:33.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97400640 1 --2- 172.21.15.67:0/2762474746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e980d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97400640 1 -- 172.21.15.67:0/2762474746 >> v1:172.21.15.86:6789/0 conn(0x7f0e98160f30 legacy=0x7f0e980b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.555+0000 7f0e97400640 1 -- 172.21.15.67:0/2762474746 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980d28f0 con 0x7f0e980d33d0 2024-04-17T10:31:33.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97400640 1 --2- 172.21.15.67:0/2762474746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e980d33d0 0x7f0e980b93d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009e50 tx=0x7f0e9001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c865ae49f5aeb1d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:33.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e95600640 1 -- 172.21.15.67:0/2762474746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900a5810 con 0x7f0e980d33d0 2024-04-17T10:31:33.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e95600640 1 -- 172.21.15.67:0/2762474746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e900a59b0 con 0x7f0e980d33d0 2024-04-17T10:31:33.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e95600640 1 -- 172.21.15.67:0/2762474746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900a5d80 con 0x7f0e980d33d0 2024-04-17T10:31:33.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 -- 172.21.15.67:0/2762474746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e980d33d0 msgr2=0x7f0e980b93d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2762474746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e980d33d0 0x7f0e980b93d0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f0e90009e50 tx=0x7f0e9001ca30 comp rx=0 tx=0).stop 2024-04-17T10:31:33.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 -- 172.21.15.67:0/2762474746 shutdown_connections 2024-04-17T10:31:33.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2762474746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e980d33d0 0x7f0e980b93d0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2762474746 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e980d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 -- 172.21.15.67:0/2762474746 >> 172.21.15.67:0/2762474746 conn(0x7f0e980d0540 msgr2=0x7f0e980d0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:33.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 -- 172.21.15.67:0/2762474746 shutdown_connections 2024-04-17T10:31:33.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 -- 172.21.15.67:0/2762474746 wait complete. 2024-04-17T10:31:33.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 Processor -- start 2024-04-17T10:31:33.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.559+0000 7f0e97e00640 1 -- start start 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e98152a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e980d33d0 0x7f0e981f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e98160f30 0x7f0e981538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b7630 con 0x7f0e98160f30 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b74b0 con 0x7f0e980d33d0 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e980b77b0 con 0x7f0e980b5a80 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e98160f30 0x7f0e981538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e96000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e980d33d0 0x7f0e981f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e98160f30 0x7f0e981538c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55204/0 (socket says 172.21.15.67:55204) 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e96a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e98152a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 -- 172.21.15.67:0/2518767997 learned_addr learned my addr 172.21.15.67:0/2518767997 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 -- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 msgr2=0x7f0e98152a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e98152a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 -- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e980d33d0 msgr2=0x7f0e981f6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e980d33d0 0x7f0e981f6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 -- 172.21.15.67:0/2518767997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980c87a0 con 0x7f0e98160f30 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97400640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e98160f30 0x7f0e981538c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0e900af040 tx=0x7f0e900197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e8ea00640 1 -- 172.21.15.67:0/2518767997 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900b8670 con 0x7f0e98160f30 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e8ea00640 1 -- 172.21.15.67:0/2518767997 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e900be070 con 0x7f0e98160f30 2024-04-17T10:31:33.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e8ea00640 1 -- 172.21.15.67:0/2518767997 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e900b8a70 con 0x7f0e98160f30 2024-04-17T10:31:33.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e980c8a80 con 0x7f0e98160f30 2024-04-17T10:31:33.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e980c9040 con 0x7f0e98160f30 2024-04-17T10:31:33.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98074dc0 con 0x7f0e98160f30 2024-04-17T10:31:33.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.563+0000 7f0e8ea00640 1 -- 172.21.15.67:0/2518767997 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0e900b8c10 con 0x7f0e98160f30 2024-04-17T10:31:33.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.567+0000 7f0e8ea00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0e78066f60 0x7f0e78069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:33.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.567+0000 7f0e8ea00640 1 -- 172.21.15.67:0/2518767997 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f0e9011ff30 con 0x7f0e98160f30 2024-04-17T10:31:33.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.567+0000 7f0e96a00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0e78066f60 0x7f0e78069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:33.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.567+0000 7f0e96a00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0e78066f60 0x7f0e78069420 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0e88000f70 tx=0x7f0e88007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:33.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.571+0000 7f0e8ea00640 1 -- 172.21.15.67:0/2518767997 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0e90016620 con 0x7f0e98160f30 2024-04-17T10:31:33.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:33.743+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0e9815d430 con 0x7f0e98160f30 2024-04-17T10:31:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.327+0000 7f0e8ea00640 1 -- 172.21.15.67:0/2518767997 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v90) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0e900a5980 con 0x7f0e98160f30 2024-04-17T10:31:34.318 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:31:34.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.327+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0e78066f60 msgr2=0x7f0e78069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.327+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0e78066f60 0x7f0e78069420 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f0e88000f70 tx=0x7f0e88007240 comp rx=0 tx=0).stop 2024-04-17T10:31:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.327+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e98160f30 msgr2=0x7f0e981538c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.327+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e98160f30 0x7f0e981538c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f0e900af040 tx=0x7f0e900197d0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 shutdown_connections 2024-04-17T10:31:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0e78066f60 0x7f0e78069420 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e98160f30 0x7f0e981538c0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e980d33d0 0x7f0e981f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 --2- 172.21.15.67:0/2518767997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e980b5a80 0x7f0e98152a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 >> 172.21.15.67:0/2518767997 conn(0x7f0e980d0540 msgr2=0x7f0e9815fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 shutdown_connections 2024-04-17T10:31:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.331+0000 7f0e97e00640 1 -- 172.21.15.67:0/2518767997 wait complete. 2024-04-17T10:31:34.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:31:34.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa55000640 1 Processor -- start 2024-04-17T10:31:34.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa55000640 1 -- start start 2024-04-17T10:31:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa55000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa500cdaf0 0x7faa500b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa55000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500ce090 con 0x7faa500cf3c0 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa55000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa500ce3d0 con 0x7faa5015dfe0 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa55000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa500ce950 con 0x7faa500cdaf0 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa4e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa500cdaf0 0x7faa500b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa4e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa500cdaf0 0x7faa500b2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55208/0 (socket says 172.21.15.67:55208) 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa4e800640 1 -- 172.21.15.67:0/2117477180 learned_addr learned my addr 172.21.15.67:0/2117477180 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa47e00640 1 --1- 172.21.15.67:0/2117477180 >> v1:172.21.15.67:6789/0 conn(0x7faa500cf3c0 0x7faa500cb8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60464/0 (socket says 172.21.15.67:60464) 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa4e800640 1 -- 172.21.15.67:0/2117477180 >> v1:172.21.15.86:6789/0 conn(0x7faa5015dfe0 legacy=0x7faa500cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa4e800640 1 -- 172.21.15.67:0/2117477180 >> v1:172.21.15.67:6789/0 conn(0x7faa500cf3c0 legacy=0x7faa500cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.499+0000 7faa4e800640 1 -- 172.21.15.67:0/2117477180 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500afc90 con 0x7faa500cdaf0 2024-04-17T10:31:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa4e800640 1 --2- 172.21.15.67:0/2117477180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cdaf0 0x7faa500b2dc0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7faa48009e50 tx=0x7faa4801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27193afb367a28f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa4d400640 1 -- 172.21.15.67:0/2117477180 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa480a56b0 con 0x7faa500cdaf0 2024-04-17T10:31:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa4d400640 1 -- 172.21.15.67:0/2117477180 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa480a5850 con 0x7faa500cdaf0 2024-04-17T10:31:34.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa4d400640 1 -- 172.21.15.67:0/2117477180 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa480a5c20 con 0x7faa500cdaf0 2024-04-17T10:31:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa55000640 1 -- 172.21.15.67:0/2117477180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cdaf0 msgr2=0x7faa500b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa55000640 1 --2- 172.21.15.67:0/2117477180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cdaf0 0x7faa500b2dc0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7faa48009e50 tx=0x7faa4801c8e0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa55000640 1 -- 172.21.15.67:0/2117477180 shutdown_connections 2024-04-17T10:31:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa55000640 1 --2- 172.21.15.67:0/2117477180 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cdaf0 0x7faa500b2dc0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa55000640 1 -- 172.21.15.67:0/2117477180 >> 172.21.15.67:0/2117477180 conn(0x7faa500b22a0 msgr2=0x7faa500b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa55000640 1 -- 172.21.15.67:0/2117477180 shutdown_connections 2024-04-17T10:31:34.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.503+0000 7faa55000640 1 -- 172.21.15.67:0/2117477180 wait complete. 2024-04-17T10:31:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 Processor -- start 2024-04-17T10:31:34.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 -- start start 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa500cdaf0 0x7faa500c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cf3c0 0x7faa500b1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa500b4620 0x7faa500b5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa50168fa0 con 0x7faa500cf3c0 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa500cc610 con 0x7faa500b4620 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa50169120 con 0x7faa500cdaf0 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cf3c0 0x7faa500b1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cf3c0 0x7faa500b1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55214/0 (socket says 172.21.15.67:55214) 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 -- 172.21.15.67:0/3222192999 learned_addr learned my addr 172.21.15.67:0/3222192999 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa4e800640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa500b4620 0x7faa500b5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa4de00640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa500cdaf0 0x7faa500c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 -- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa500cdaf0 msgr2=0x7faa500c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa500cdaf0 0x7faa500c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 -- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa500b4620 msgr2=0x7faa500b5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa500b4620 0x7faa500b5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 -- 172.21.15.67:0/3222192999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa500b4b50 con 0x7faa500cf3c0 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa4e800640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa500b4620 0x7faa500b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa47e00640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cf3c0 0x7faa500b1650 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7faa3800c430 tx=0x7faa3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa46000640 1 -- 172.21.15.67:0/3222192999 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa380074e0 con 0x7faa500cf3c0 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa46000640 1 -- 172.21.15.67:0/3222192999 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa38007680 con 0x7faa500cf3c0 2024-04-17T10:31:34.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa46000640 1 -- 172.21.15.67:0/3222192999 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa38004a10 con 0x7faa500cf3c0 2024-04-17T10:31:34.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa500b5100 con 0x7faa500cf3c0 2024-04-17T10:31:34.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.507+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa500b56c0 con 0x7faa500cf3c0 2024-04-17T10:31:34.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.511+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5009e900 con 0x7faa500cf3c0 2024-04-17T10:31:34.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.511+0000 7faa46000640 1 -- 172.21.15.67:0/3222192999 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7faa3801a020 con 0x7faa500cf3c0 2024-04-17T10:31:34.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.511+0000 7faa46000640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa2c066f60 0x7faa2c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.511+0000 7faa46000640 1 -- 172.21.15.67:0/3222192999 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5249+0+0 (secure 0 0 0) 0x7faa380846d0 con 0x7faa500cf3c0 2024-04-17T10:31:34.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.511+0000 7faa4de00640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa2c066f60 0x7faa2c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.511+0000 7faa4de00640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa2c066f60 0x7faa2c069420 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7faa40000f70 tx=0x7faa40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:34.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.515+0000 7faa46000640 1 -- 172.21.15.67:0/3222192999 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7faa38055800 con 0x7faa500cf3c0 2024-04-17T10:31:34.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.731+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7faa50074e30 con 0x7faa500cf3c0 2024-04-17T10:31:34.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa46000640 1 -- 172.21.15.67:0/3222192999 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7faa38055100 con 0x7faa500cf3c0 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.725 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.726 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.727 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.728 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.729 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.730 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.731 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.732 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.733 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.734 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:31:34.735 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:31:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:34.737 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:31:34.738 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:31:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa2c066f60 msgr2=0x7faa2c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa2c066f60 0x7faa2c069420 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7faa40000f70 tx=0x7faa40009040 comp rx=0 tx=0).stop 2024-04-17T10:31:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cf3c0 msgr2=0x7faa500b1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cf3c0 0x7faa500b1650 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7faa3800c430 tx=0x7faa3800c8f0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 shutdown_connections 2024-04-17T10:31:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa2c066f60 0x7faa2c069420 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa500b4620 0x7faa500b5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa500cf3c0 0x7faa500b1650 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 --2- 172.21.15.67:0/3222192999 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa500cdaf0 0x7faa500c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 >> 172.21.15.67:0/3222192999 conn(0x7faa500b22a0 msgr2=0x7faa500cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 shutdown_connections 2024-04-17T10:31:34.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.735+0000 7faa55000640 1 -- 172.21.15.67:0/3222192999 wait complete. 2024-04-17T10:31:34.746 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:34.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bf2a00640 1 Processor -- start 2024-04-17T10:31:34.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bf2a00640 1 -- start start 2024-04-17T10:31:34.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bf2a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4bec06e6e0 0x7f4bec0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bf2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bf2a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bec15ae40 con 0x7f4bec143ae0 2024-04-17T10:31:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bf2a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bec0cf430 con 0x7f4bec06e6e0 2024-04-17T10:31:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bf2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec071440 con 0x7f4bec06eac0 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.895+0000 7f4bebe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4bec06e6e0 0x7f4bec0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4bebe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4bec06e6e0 0x7f4bec0ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:52320/0 (socket says 172.21.15.67:52320) 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4be3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4bf0a00640 1 --1- 172.21.15.67:0/4222633489 >> v1:172.21.15.86:6789/0 conn(0x7f4bec143ae0 0x7f4bec143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:57734/0 (socket says 172.21.15.67:57734) 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4bebe00640 1 -- 172.21.15.67:0/4222633489 learned_addr learned my addr 172.21.15.67:0/4222633489 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4be3400640 1 -- 172.21.15.67:0/4222633489 >> v2:172.21.15.86:3300/0 conn(0x7f4bec06e6e0 msgr2=0x7f4bec0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4be3400640 1 --2- 172.21.15.67:0/4222633489 >> v2:172.21.15.86:3300/0 conn(0x7f4bec06e6e0 0x7f4bec0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4be3400640 1 -- 172.21.15.67:0/4222633489 >> v1:172.21.15.86:6789/0 conn(0x7f4bec143ae0 legacy=0x7f4bec143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4be3400640 1 -- 172.21.15.67:0/4222633489 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec071720 con 0x7f4bec06eac0 2024-04-17T10:31:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4bebe00640 1 --2- 172.21.15.67:0/4222633489 >> v2:172.21.15.86:3300/0 conn(0x7f4bec06e6e0 0x7f4bec0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:34.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4be3400640 1 --2- 172.21.15.67:0/4222633489 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec0ceef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40097c0 tx=0x7f4bd401c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90a685c2869791a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:34.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4beb400640 1 -- 172.21.15.67:0/4222633489 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40b0070 con 0x7f4bec06eac0 2024-04-17T10:31:34.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4beb400640 1 -- 172.21.15.67:0/4222633489 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd40ab600 con 0x7f4bec06eac0 2024-04-17T10:31:34.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.899+0000 7f4beb400640 1 -- 172.21.15.67:0/4222633489 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40ab9d0 con 0x7f4bec06eac0 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- 172.21.15.67:0/4222633489 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 msgr2=0x7f4bec0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/4222633489 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec0ceef0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40097c0 tx=0x7f4bd401c3e0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- 172.21.15.67:0/4222633489 shutdown_connections 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/4222633489 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec0ceef0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/4222633489 >> v2:172.21.15.86:3300/0 conn(0x7f4bec06e6e0 0x7f4bec0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- 172.21.15.67:0/4222633489 >> 172.21.15.67:0/4222633489 conn(0x7f4bec06ac90 msgr2=0x7f4bec06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- 172.21.15.67:0/4222633489 shutdown_connections 2024-04-17T10:31:34.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- 172.21.15.67:0/4222633489 wait complete. 2024-04-17T10:31:34.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 Processor -- start 2024-04-17T10:31:34.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- start start 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bec06e6e0 0x7f4bec0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec13c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bec143ae0 0x7f4bec156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec06dd00 con 0x7f4bec06e6e0 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec06db80 con 0x7f4bec143ae0 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bec06de80 con 0x7f4bec06eac0 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bec06e6e0 0x7f4bec0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bec06e6e0 0x7f4bec0c46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55220/0 (socket says 172.21.15.67:55220) 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bebe00640 1 -- 172.21.15.67:0/3507490330 learned_addr learned my addr 172.21.15.67:0/3507490330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4be3400640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec13c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bf0a00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bec143ae0 0x7f4bec156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bebe00640 1 -- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 msgr2=0x7f4bec13c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bebe00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec13c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.903+0000 7f4bebe00640 1 -- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bec143ae0 msgr2=0x7f4bec156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4bebe00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bec143ae0 0x7f4bec156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4bebe00640 1 -- 172.21.15.67:0/3507490330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bec0b1820 con 0x7f4bec06e6e0 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4bebe00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bec06e6e0 0x7f4bec0c46e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00e9d0 tx=0x7f4bdc00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4be8c00640 1 -- 172.21.15.67:0/3507490330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc017070 con 0x7f4bec06e6e0 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4be8c00640 1 -- 172.21.15.67:0/3507490330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bdc00ca60 con 0x7f4bec06e6e0 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bec0c4c20 con 0x7f4bec06e6e0 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4be8c00640 1 -- 172.21.15.67:0/3507490330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bdc00ce10 con 0x7f4bec06e6e0 2024-04-17T10:31:34.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bec1581e0 con 0x7f4bec06e6e0 2024-04-17T10:31:34.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bec072d50 con 0x7f4bec06e6e0 2024-04-17T10:31:34.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4be8c00640 1 -- 172.21.15.67:0/3507490330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4bdc01a070 con 0x7f4bec06e6e0 2024-04-17T10:31:34.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4be8c00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bc8066f60 0x7f4bc8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:34.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4be8c00640 1 -- 172.21.15.67:0/3507490330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f4bdc084ed0 con 0x7f4bec06e6e0 2024-04-17T10:31:34.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.907+0000 7f4be3400640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bc8066f60 0x7f4bc8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:34.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.911+0000 7f4be3400640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bc8066f60 0x7f4bc8069420 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40097c0 tx=0x7f4bd4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:34.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:34.911+0000 7f4be8c00640 1 -- 172.21.15.67:0/3507490330 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4bdc056010 con 0x7f4bec06e6e0 2024-04-17T10:31:35.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bec074e30 con 0x7f4bec06e6e0 2024-04-17T10:31:35.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4be8c00640 1 -- 172.21.15.67:0/3507490330 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+13117 (secure 0 0 0) 0x7f4bdc055910 con 0x7f4bec06e6e0 2024-04-17T10:31:35.066 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:35.066 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":92,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:34.336216+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bc8066f60 msgr2=0x7f4bc8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bc8066f60 0x7f4bc8069420 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40097c0 tx=0x7f4bd4008040 comp rx=0 tx=0).stop 2024-04-17T10:31:35.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bec06e6e0 msgr2=0x7f4bec0c46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bec06e6e0 0x7f4bec0c46e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc00e9d0 tx=0x7f4bdc00ee90 comp rx=0 tx=0).stop 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 shutdown_connections 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4bc8066f60 0x7f4bc8069420 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bec143ae0 0x7f4bec156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bec06eac0 0x7f4bec13c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 --2- 172.21.15.67:0/3507490330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bec06e6e0 0x7f4bec0c46e0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 >> 172.21.15.67:0/3507490330 conn(0x7f4bec06ac90 msgr2=0x7f4bec0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.075+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 shutdown_connections 2024-04-17T10:31:35.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.079+0000 7f4bf2a00640 1 -- 172.21.15.67:0/3507490330 wait complete. 2024-04-17T10:31:35.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:31:35.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:31:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ed1800640 1 Processor -- start 2024-04-17T10:31:35.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ed1800640 1 -- start start 2024-04-17T10:31:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ed1800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc072a90 0x7f7ecc0b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ed1800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ecc0b5130 0x7f7ecc06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ed1800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ecc15f360 con 0x7f7ecc1583f0 2024-04-17T10:31:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ed1800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ecc0b3160 con 0x7f7ecc0b5130 2024-04-17T10:31:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ed1800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0b3440 con 0x7f7ecc072a90 2024-04-17T10:31:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ecb400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7ecc0b5130 0x7f7ecc06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7eca000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc072a90 0x7f7ecc0b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ecaa00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7ecc1583f0 0x7f7ecc072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:57746/0 (socket says 172.21.15.67:57746) 2024-04-17T10:31:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ecaa00640 1 -- 172.21.15.67:0/1397651224 learned_addr learned my addr 172.21.15.67:0/1397651224 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ec9600640 1 -- 172.21.15.67:0/1397651224 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229726343 0 0) 0x7f7ecc15f360 con 0x7f7ecc1583f0 2024-04-17T10:31:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ecb400640 1 -- 172.21.15.67:0/1397651224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc072a90 msgr2=0x7f7ecc0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ecb400640 1 --2- 172.21.15.67:0/1397651224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc072a90 0x7f7ecc0b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ecb400640 1 -- 172.21.15.67:0/1397651224 >> v1:172.21.15.86:6789/0 conn(0x7f7ecc1583f0 legacy=0x7f7ecc072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.243+0000 7f7ecb400640 1 -- 172.21.15.67:0/1397651224 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc0b3720 con 0x7f7ecc0b5130 2024-04-17T10:31:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ecb400640 1 --2- 172.21.15.67:0/1397651224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc06ac90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009d90 tx=0x7f7ec401c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75be93850e957477 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ec9600640 1 -- 172.21.15.67:0/1397651224 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40b2070 con 0x7f7ecc0b5130 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ec9600640 1 -- 172.21.15.67:0/1397651224 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ec40a5a00 con 0x7f7ecc0b5130 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ec9600640 1 -- 172.21.15.67:0/1397651224 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ec40a5dd0 con 0x7f7ecc0b5130 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 -- 172.21.15.67:0/1397651224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc0b5130 msgr2=0x7f7ecc06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 --2- 172.21.15.67:0/1397651224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc06ac90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009d90 tx=0x7f7ec401c940 comp rx=0 tx=0).stop 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 -- 172.21.15.67:0/1397651224 shutdown_connections 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 --2- 172.21.15.67:0/1397651224 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc06ac90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 --2- 172.21.15.67:0/1397651224 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc072a90 0x7f7ecc0b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 -- 172.21.15.67:0/1397651224 >> 172.21.15.67:0/1397651224 conn(0x7f7ecc0b2570 msgr2=0x7f7ecc071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 -- 172.21.15.67:0/1397651224 shutdown_connections 2024-04-17T10:31:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.247+0000 7f7ed1800640 1 -- 172.21.15.67:0/1397651224 wait complete. 2024-04-17T10:31:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 Processor -- start 2024-04-17T10:31:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 -- start start 2024-04-17T10:31:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc072a90 0x7f7ecc1511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc1583f0 0x7f7ecc0afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0b4200 con 0x7f7ecc0b5130 2024-04-17T10:31:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc1587d0 con 0x7f7ecc072a90 2024-04-17T10:31:35.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc06b9e0 con 0x7f7ecc1583f0 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55224/0 (socket says 172.21.15.67:55224) 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 -- 172.21.15.67:0/2491052240 learned_addr learned my addr 172.21.15.67:0/2491052240 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ecb400640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc1583f0 0x7f7ecc0afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ecaa00640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc072a90 0x7f7ecc1511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 -- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc1583f0 msgr2=0x7f7ecc0afd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc1583f0 0x7f7ecc0afd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 -- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc072a90 msgr2=0x7f7ecc1511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc072a90 0x7f7ecc1511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 -- 172.21.15.67:0/2491052240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc159520 con 0x7f7ecc0b5130 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7eca000640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc158950 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800e930 tx=0x7f7eb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ec2a00640 1 -- 172.21.15.67:0/2491052240 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8017070 con 0x7f7ecc0b5130 2024-04-17T10:31:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ec2a00640 1 -- 172.21.15.67:0/2491052240 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb800ca00 con 0x7f7ecc0b5130 2024-04-17T10:31:35.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc1597c0 con 0x7f7ecc0b5130 2024-04-17T10:31:35.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ec2a00640 1 -- 172.21.15.67:0/2491052240 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb800cdb0 con 0x7f7ecc0b5130 2024-04-17T10:31:35.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc159d80 con 0x7f7ecc0b5130 2024-04-17T10:31:35.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.251+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc159fc0 con 0x7f7ecc0b5130 2024-04-17T10:31:35.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.255+0000 7f7ec2a00640 1 -- 172.21.15.67:0/2491052240 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7eb801a070 con 0x7f7ecc0b5130 2024-04-17T10:31:35.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.255+0000 7f7ec2a00640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7eac066f60 0x7f7eac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.255+0000 7f7ec2a00640 1 -- 172.21.15.67:0/2491052240 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7eb8085210 con 0x7f7ecc0b5130 2024-04-17T10:31:35.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.255+0000 7f7ecaa00640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7eac066f60 0x7f7eac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.255+0000 7f7ecaa00640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7eac066f60 0x7f7eac069420 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000f70 tx=0x7f7ebc007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.259+0000 7f7ec2a00640 1 -- 172.21.15.67:0/2491052240 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7ecc159fc0 con 0x7f7ecc0b5130 2024-04-17T10:31:35.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.423+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ecc04ef30 con 0x7f7ecc0b5130 2024-04-17T10:31:35.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.423+0000 7f7ec2a00640 1 -- 172.21.15.67:0/2491052240 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7eb8055dc0 con 0x7f7ecc0b5130 2024-04-17T10:31:35.415 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:31:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7eac066f60 msgr2=0x7f7eac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7eac066f60 0x7f7eac069420 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc000f70 tx=0x7f7ebc007df0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ecc0b5130 msgr2=0x7f7ecc158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc158950 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800e930 tx=0x7f7eb800edf0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 shutdown_connections 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7eac066f60 0x7f7eac069420 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7ecc1583f0 0x7f7ecc0afd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7ecc0b5130 0x7f7ecc158950 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 --2- 172.21.15.67:0/2491052240 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7ecc072a90 0x7f7ecc1511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 >> 172.21.15.67:0/2491052240 conn(0x7f7ecc0b2570 msgr2=0x7f7ecc167b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 shutdown_connections 2024-04-17T10:31:35.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.427+0000 7f7ed1800640 1 -- 172.21.15.67:0/2491052240 wait complete. 2024-04-17T10:31:35.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:31:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc9a00640 1 Processor -- start 2024-04-17T10:31:35.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc9a00640 1 -- start start 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc9a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc9a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc40b6640 con 0x7f0dc4144aa0 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc9a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dc41456d0 con 0x7f0dc41446c0 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc9a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc40b51a0 con 0x7f0dc4071840 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc2000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0dc4144aa0 0x7f0dc40d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60474/0 (socket says 172.21.15.67:60474) 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc2000640 1 -- 172.21.15.67:0/541713788 learned_addr learned my addr 172.21.15.67:0/541713788 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc3400640 1 --2- 172.21.15.67:0/541713788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260487635 0 0) 0x7f0dc40b6640 con 0x7f0dc4144aa0 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da8003530 con 0x7f0dc4144aa0 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919593284 0 0) 0x7f0dc41456d0 con 0x7f0dc41446c0 2024-04-17T10:31:35.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc40b6640 con 0x7f0dc41446c0 2024-04-17T10:31:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc3400640 1 -- 172.21.15.67:0/541713788 >> v1:172.21.15.86:6789/0 conn(0x7f0dc41446c0 legacy=0x7f0dc4144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc3400640 1 -- 172.21.15.67:0/541713788 >> v1:172.21.15.67:6789/0 conn(0x7f0dc4144aa0 legacy=0x7f0dc40d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3050550931 0 0) 0x7f0da8003530 con 0x7f0dc4144aa0 2024-04-17T10:31:35.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.587+0000 7f0dc3400640 1 -- 172.21.15.67:0/541713788 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc40b4870 con 0x7f0dc4071840 2024-04-17T10:31:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc3400640 1 --2- 172.21.15.67:0/541713788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4166a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009dc0 tx=0x7f0dbc01c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b753e76cedc05c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0b2070 con 0x7f0dc4071840 2024-04-17T10:31:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc0a5930 con 0x7f0dc4071840 2024-04-17T10:31:35.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc1600640 1 -- 172.21.15.67:0/541713788 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc0a5d00 con 0x7f0dc4071840 2024-04-17T10:31:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc9a00640 1 -- 172.21.15.67:0/541713788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 msgr2=0x7f0dc4166a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/541713788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4166a30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009dc0 tx=0x7f0dbc01c960 comp rx=0 tx=0).stop 2024-04-17T10:31:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc9a00640 1 -- 172.21.15.67:0/541713788 shutdown_connections 2024-04-17T10:31:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/541713788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4166a30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc9a00640 1 -- 172.21.15.67:0/541713788 >> 172.21.15.67:0/541713788 conn(0x7f0dc40b34d0 msgr2=0x7f0dc40b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc9a00640 1 -- 172.21.15.67:0/541713788 shutdown_connections 2024-04-17T10:31:35.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.591+0000 7f0dc9a00640 1 -- 172.21.15.67:0/541713788 wait complete. 2024-04-17T10:31:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 Processor -- start 2024-04-17T10:31:35.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 -- start start 2024-04-17T10:31:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc41446c0 0x7f0dc4155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc4144aa0 0x7f0dc40b2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4169910 con 0x7f0dc41446c0 2024-04-17T10:31:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4169790 con 0x7f0dc4144aa0 2024-04-17T10:31:35.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc4169a90 con 0x7f0dc4071840 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc41446c0 0x7f0dc4155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4142580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:34910/0 (socket says 172.21.15.67:34910) 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc4144aa0 0x7f0dc40b2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 -- 172.21.15.67:0/3738248095 learned_addr learned my addr 172.21.15.67:0/3738248095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 -- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc4144aa0 msgr2=0x7f0dc40b2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc4144aa0 0x7f0dc40b2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 -- 172.21.15.67:0/3738248095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc41446c0 msgr2=0x7f0dc4155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc41446c0 0x7f0dc4155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 -- 172.21.15.67:0/3738248095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4142ac0 con 0x7f0dc4071840 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2000640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc41446c0 0x7f0dc4155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc2a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4142580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c490 tx=0x7f0db400c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dbaa00640 1 -- 172.21.15.67:0/3738248095 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db40052d0 con 0x7f0dc4071840 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dbaa00640 1 -- 172.21.15.67:0/3738248095 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db4005470 con 0x7f0dc4071840 2024-04-17T10:31:35.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dbaa00640 1 -- 172.21.15.67:0/3738248095 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4012870 con 0x7f0dc4071840 2024-04-17T10:31:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dc4164fb0 con 0x7f0dc4071840 2024-04-17T10:31:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.595+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dc4165570 con 0x7f0dc4071840 2024-04-17T10:31:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.599+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dc4070bb0 con 0x7f0dc4071840 2024-04-17T10:31:35.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.599+0000 7f0dbaa00640 1 -- 172.21.15.67:0/3738248095 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0db4012a10 con 0x7f0dc4071840 2024-04-17T10:31:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.599+0000 7f0dbaa00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da4066f60 0x7f0da4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.599+0000 7f0dbaa00640 1 -- 172.21.15.67:0/3738248095 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0db4084c90 con 0x7f0dc4071840 2024-04-17T10:31:35.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.599+0000 7f0dc2000640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da4066f60 0x7f0da4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.599+0000 7f0dc2000640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da4066f60 0x7f0da4069420 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4154090 tx=0x7f0db0007c60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.603+0000 7f0dbaa00640 1 -- 172.21.15.67:0/3738248095 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0db40557f0 con 0x7f0dc4071840 2024-04-17T10:31:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.763+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0dc40b41a0 con 0x7f0dc4071840 2024-04-17T10:31:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dbaa00640 1 -- 172.21.15.67:0/3738248095 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v92) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0db4059890 con 0x7f0dc4071840 2024-04-17T10:31:35.756 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:35.757 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:31:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:31:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:31:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:31:35.757 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:31:35.757 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da4066f60 msgr2=0x7f0da4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da4066f60 0x7f0da4069420 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4154090 tx=0x7f0db0007c60 comp rx=0 tx=0).stop 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 msgr2=0x7f0dc4142580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4142580 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0db400c490 tx=0x7f0db400c950 comp rx=0 tx=0).stop 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 shutdown_connections 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0da4066f60 0x7f0da4069420 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0dc4144aa0 0x7f0dc40b2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0dc41446c0 0x7f0dc4155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 --2- 172.21.15.67:0/3738248095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0dc4071840 0x7f0dc4142580 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 >> 172.21.15.67:0/3738248095 conn(0x7f0dc40b34d0 msgr2=0x7f0dc406bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 shutdown_connections 2024-04-17T10:31:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.767+0000 7f0dc9a00640 1 -- 172.21.15.67:0/3738248095 wait complete. 2024-04-17T10:31:35.779 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:31:35.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:31:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.931+0000 7f5ca3e00640 1 Processor -- start 2024-04-17T10:31:35.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3e00640 1 -- start start 2024-04-17T10:31:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5ca40b4be0 0x7f5ca41652a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca406d900 con 0x7f5ca415a690 2024-04-17T10:31:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca40ccec0 con 0x7f5ca40cf3f0 2024-04-17T10:31:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca40cd480 con 0x7f5ca40b4be0 2024-04-17T10:31:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ca415a690 0x7f5ca40cc690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60480/0 (socket says 172.21.15.67:60480) 2024-04-17T10:31:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2a00640 1 -- 172.21.15.67:0/595902106 learned_addr learned my addr 172.21.15.67:0/595902106 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:35.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3400640 1 --2- 172.21.15.67:0/595902106 >> v2:172.21.15.86:3300/0 conn(0x7f5ca40b4be0 0x7f5ca41652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415746801 0 0) 0x7f5ca406d900 con 0x7f5ca415a690 2024-04-17T10:31:35.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c88003530 con 0x7f5ca415a690 2024-04-17T10:31:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2278580030 0 0) 0x7f5c88003530 con 0x7f5ca415a690 2024-04-17T10:31:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 >> v2:172.21.15.86:3300/0 conn(0x7f5ca40b4be0 msgr2=0x7f5ca41652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:35.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 --2- 172.21.15.67:0/595902106 >> v2:172.21.15.86:3300/0 conn(0x7f5ca40b4be0 0x7f5ca41652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 >> v1:172.21.15.86:6789/0 conn(0x7f5ca40cf3f0 legacy=0x7f5ca40b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca40cd1a0 con 0x7f5ca415a690 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f5c94001680 con 0x7f5ca415a690 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 >> v1:172.21.15.67:6789/0 conn(0x7f5ca415a690 legacy=0x7f5ca40cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c88003a10 0x7f5c88003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c88004810 0x7f5c88004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5c8800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40cd480 con 0x7f5c880051d0 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8800c0d0 con 0x7f5c88004810 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8800c2d0 con 0x7f5c88003a10 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3400640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5c8800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5c9a000640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c88004810 0x7f5c88004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2a00640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c88003a10 0x7f5c88003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3400640 1 -- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c88003a10 msgr2=0x7f5c88003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3400640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c88003a10 0x7f5c88003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3400640 1 -- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c88004810 msgr2=0x7f5c88004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3400640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5c88004810 0x7f5c88004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca3400640 1 -- 172.21.15.67:0/595902106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c88003530 con 0x7f5c880051d0 2024-04-17T10:31:35.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.935+0000 7f5ca2a00640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5c88003a10 0x7f5c88003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3400640 1 -- 172.21.15.67:0/595902106 reap_dead start 2024-04-17T10:31:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3400640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5c8800b9d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c010320 tx=0x7f5c9c01ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d673e071495c61c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c00be20 con 0x7f5c880051d0 2024-04-17T10:31:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c00b0d0 con 0x7f5c880051d0 2024-04-17T10:31:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca2000640 1 -- 172.21.15.67:0/595902106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c009540 con 0x7f5c880051d0 2024-04-17T10:31:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3e00640 1 -- 172.21.15.67:0/595902106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 msgr2=0x7f5c8800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5c8800b9d0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c010320 tx=0x7f5c9c01ee80 comp rx=0 tx=0).stop 2024-04-17T10:31:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3e00640 1 -- 172.21.15.67:0/595902106 shutdown_connections 2024-04-17T10:31:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/595902106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5c8800b9d0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3e00640 1 -- 172.21.15.67:0/595902106 >> 172.21.15.67:0/595902106 conn(0x7f5ca406ac90 msgr2=0x7f5ca406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3e00640 1 -- 172.21.15.67:0/595902106 shutdown_connections 2024-04-17T10:31:35.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.939+0000 7f5ca3e00640 1 -- 172.21.15.67:0/595902106 wait complete. 2024-04-17T10:31:35.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 Processor -- start 2024-04-17T10:31:35.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 -- start start 2024-04-17T10:31:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5ca415b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca414e540 0x7f5ca4152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca40b0800 0x7f5ca40b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40cc810 con 0x7f5c880051d0 2024-04-17T10:31:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40cc690 con 0x7f5ca414e540 2024-04-17T10:31:35.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40cc990 con 0x7f5ca40b0800 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5ca415b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5ca415b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55262/0 (socket says 172.21.15.67:55262) 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 -- 172.21.15.67:0/818074 learned_addr learned my addr 172.21.15.67:0/818074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5c9a000640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca414e540 0x7f5ca4152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 -- 172.21.15.67:0/818074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca40b0800 msgr2=0x7f5ca40b0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca40b0800 0x7f5ca40b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 -- 172.21.15.67:0/818074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca414e540 msgr2=0x7f5ca4152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca414e540 0x7f5ca4152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 -- 172.21.15.67:0/818074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca415bc70 con 0x7f5c880051d0 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca2a00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5ca415b5e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400e780 tx=0x7f5c9400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5c9b400640 1 -- 172.21.15.67:0/818074 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9400ee30 con 0x7f5c880051d0 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5c9b400640 1 -- 172.21.15.67:0/818074 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c94018070 con 0x7f5c880051d0 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5c9b400640 1 -- 172.21.15.67:0/818074 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9400cad0 con 0x7f5c880051d0 2024-04-17T10:31:35.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca41f39f0 con 0x7f5c880051d0 2024-04-17T10:31:35.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca41f3c90 con 0x7f5c880051d0 2024-04-17T10:31:35.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.943+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca4074dc0 con 0x7f5c880051d0 2024-04-17T10:31:35.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.947+0000 7f5c9b400640 1 -- 172.21.15.67:0/818074 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5c9400cc70 con 0x7f5c880051d0 2024-04-17T10:31:35.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.947+0000 7f5c9b400640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c80066f90 0x7f5c80069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:35.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.947+0000 7f5c9b400640 1 -- 172.21.15.67:0/818074 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f5c94084c80 con 0x7f5c880051d0 2024-04-17T10:31:35.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.947+0000 7f5c9a000640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c80066f90 0x7f5c80069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:35.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.947+0000 7f5c9a000640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c80066f90 0x7f5c80069450 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c000c10 tx=0x7f5c8c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:35.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:35.951+0000 7f5c9b400640 1 -- 172.21.15.67:0/818074 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5c94055e40 con 0x7f5c880051d0 2024-04-17T10:31:36.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.115+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5ca415deb0 con 0x7f5c880051d0 2024-04-17T10:31:36.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.327+0000 7f5c9b400640 1 -- 172.21.15.67:0/818074 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v93) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5c94055740 con 0x7f5c880051d0 2024-04-17T10:31:36.320 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:31:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c80066f90 msgr2=0x7f5c80069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c80066f90 0x7f5c80069450 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c000c10 tx=0x7f5c8c009040 comp rx=0 tx=0).stop 2024-04-17T10:31:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 msgr2=0x7f5ca415b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5ca415b5e0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f5c9400e780 tx=0x7f5c9400ec40 comp rx=0 tx=0).stop 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 shutdown_connections 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c80066f90 0x7f5c80069450 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca40b0800 0x7f5ca40b0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca414e540 0x7f5ca4152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 --2- 172.21.15.67:0/818074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5c880051d0 0x7f5ca415b5e0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 >> 172.21.15.67:0/818074 conn(0x7f5ca406ac90 msgr2=0x7f5ca4156880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 shutdown_connections 2024-04-17T10:31:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.331+0000 7f5ca3e00640 1 -- 172.21.15.67:0/818074 wait complete. 2024-04-17T10:31:36.351 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:31:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.507+0000 7f2a08800640 1 Processor -- start 2024-04-17T10:31:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.507+0000 7f2a08800640 1 -- start start 2024-04-17T10:31:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a08800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 0x7f2a0006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a08800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a000b4990 con 0x7f2a0006e7b0 2024-04-17T10:31:36.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a08800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0015ae90 con 0x7f2a000aef60 2024-04-17T10:31:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a08800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000af340 con 0x7f2a000cd030 2024-04-17T10:31:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a06800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2a0006e7b0 0x7f2a000b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60484/0 (socket says 172.21.15.67:60484) 2024-04-17T10:31:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a06800640 1 -- 172.21.15.67:0/2504195625 learned_addr learned my addr 172.21.15.67:0/2504195625 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05400640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 0x7f2a0006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544424999 0 0) 0x7f29fc0014a0 con 0x7f2a0006e7b0 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e8003530 con 0x7f2a0006e7b0 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2156640009 0 0) 0x7f29e8003530 con 0x7f2a0006e7b0 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 msgr2=0x7f2a0006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 0x7f2a0006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 >> v1:172.21.15.86:6789/0 conn(0x7f2a000aef60 legacy=0x7f2a000cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0006ee80 con 0x7f2a0006e7b0 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f29fc001bd0 con 0x7f2a0006e7b0 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 >> v1:172.21.15.67:6789/0 conn(0x7f2a0006e7b0 legacy=0x7f2a000b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e80039f0 0x7f29e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29e80043a0 0x7f29e8004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 0x7f29e8005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a000af340 con 0x7f29e80039f0 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800ba80 con 0x7f29e80043a0 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29e800de90 con 0x7f2a000cd030 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05e00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e80039f0 0x7f29e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05400640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29e80043a0 0x7f29e8004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a06800640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 0x7f29e8005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05e00640 1 -- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 msgr2=0x7f29e8005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05e00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a000cd030 0x7f29e8005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05e00640 1 -- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29e80043a0 msgr2=0x7f29e8004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05e00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29e80043a0 0x7f29e8004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.511+0000 7f2a05e00640 1 -- 172.21.15.67:0/2504195625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e8003530 con 0x7f29e80039f0 2024-04-17T10:31:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a05e00640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e80039f0 0x7f29e8003dd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f29f4001860 tx=0x7f29f401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2307e9c53078e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f40b0070 con 0x7f29e80039f0 2024-04-17T10:31:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f40ab600 con 0x7f29e80039f0 2024-04-17T10:31:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a04a00640 1 -- 172.21.15.67:0/2504195625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4001460 con 0x7f29e80039f0 2024-04-17T10:31:36.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 -- 172.21.15.67:0/2504195625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e80039f0 msgr2=0x7f29e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 --2- 172.21.15.67:0/2504195625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29e80039f0 0x7f29e8003dd0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f29f4001860 tx=0x7f29f401c520 comp rx=0 tx=0).stop 2024-04-17T10:31:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a06800640 1 -- 172.21.15.67:0/2504195625 reap_dead start 2024-04-17T10:31:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 -- 172.21.15.67:0/2504195625 shutdown_connections 2024-04-17T10:31:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 -- 172.21.15.67:0/2504195625 >> 172.21.15.67:0/2504195625 conn(0x7f2a000b1740 msgr2=0x7f2a000b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 -- 172.21.15.67:0/2504195625 shutdown_connections 2024-04-17T10:31:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 -- 172.21.15.67:0/2504195625 wait complete. 2024-04-17T10:31:36.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 Processor -- start 2024-04-17T10:31:36.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.515+0000 7f2a08800640 1 -- start start 2024-04-17T10:31:36.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29e80039f0 0x7f2a0014b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a0013e270 0x7f2a001f5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a001f5a70 0x7f2a001f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0015bc10 con 0x7f2a001f5a70 2024-04-17T10:31:36.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0006d160 con 0x7f2a0013e270 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a00160000 con 0x7f29e80039f0 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a001f5a70 0x7f2a001f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a001f5a70 0x7f2a001f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55266/0 (socket says 172.21.15.67:55266) 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 -- 172.21.15.67:0/451939812 learned_addr learned my addr 172.21.15.67:0/451939812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a05e00640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29e80039f0 0x7f2a0014b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a05400640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a0013e270 0x7f2a001f5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 -- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29e80039f0 msgr2=0x7f2a0014b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29e80039f0 0x7f2a0014b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 -- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a0013e270 msgr2=0x7f2a001f5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a0013e270 0x7f2a001f5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 -- 172.21.15.67:0/451939812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0014c0a0 con 0x7f2a001f5a70 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a05e00640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29e80039f0 0x7f2a0014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a05400640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a0013e270 0x7f2a001f5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:36.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a06800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a001f5a70 0x7f2a001f1c50 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2a000b2a00 tx=0x7f29fc00af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:36.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f29fa000640 1 -- 172.21.15.67:0/451939812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc007800 con 0x7f2a001f5a70 2024-04-17T10:31:36.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f29fa000640 1 -- 172.21.15.67:0/451939812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc018070 con 0x7f2a001f5a70 2024-04-17T10:31:36.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f29fa000640 1 -- 172.21.15.67:0/451939812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0133f0 con 0x7f2a001f5a70 2024-04-17T10:31:36.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0014c380 con 0x7f2a001f5a70 2024-04-17T10:31:36.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0014c940 con 0x7f2a001f5a70 2024-04-17T10:31:36.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.519+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0004ef30 con 0x7f2a001f5a70 2024-04-17T10:31:36.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.523+0000 7f29fa000640 1 -- 172.21.15.67:0/451939812 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f29fc013590 con 0x7f2a001f5a70 2024-04-17T10:31:36.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.523+0000 7f29fa000640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29e4066f60 0x7f29e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:36.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.523+0000 7f29fa000640 1 -- 172.21.15.67:0/451939812 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f29fc085600 con 0x7f2a001f5a70 2024-04-17T10:31:36.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.523+0000 7f2a05e00640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29e4066f60 0x7f29e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:36.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.523+0000 7f2a05e00640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29e4066f60 0x7f29e4069420 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f29f40ad000 tx=0x7f29f40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:36.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.527+0000 7f29fa000640 1 -- 172.21.15.67:0/451939812 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2a000b4990 con 0x7f2a001f5a70 2024-04-17T10:31:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:36.691+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2a000b32c0 con 0x7f2a001f5a70 2024-04-17T10:31:37.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.335+0000 7f29fa000640 1 -- 172.21.15.67:0/451939812 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v94) v1 ==== 170+0+0 (secure 0 0 0) 0x7f29fc0565c0 con 0x7f2a001f5a70 2024-04-17T10:31:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:31:37.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.339+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29e4066f60 msgr2=0x7f29e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:37.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.339+0000 7f2a08800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29e4066f60 0x7f29e4069420 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f29f40ad000 tx=0x7f29f40a6000 comp rx=0 tx=0).stop 2024-04-17T10:31:37.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.339+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a001f5a70 msgr2=0x7f2a001f1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:37.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.339+0000 7f2a08800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a001f5a70 0x7f2a001f1c50 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f2a000b2a00 tx=0x7f29fc00af60 comp rx=0 tx=0).stop 2024-04-17T10:31:37.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 shutdown_connections 2024-04-17T10:31:37.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29e4066f60 0x7f29e4069420 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a001f5a70 0x7f2a001f1c50 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a0013e270 0x7f2a001f5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 --2- 172.21.15.67:0/451939812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29e80039f0 0x7f2a0014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 >> 172.21.15.67:0/451939812 conn(0x7f2a000b1740 msgr2=0x7f2a0014dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:37.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 shutdown_connections 2024-04-17T10:31:37.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.343+0000 7f2a08800640 1 -- 172.21.15.67:0/451939812 wait complete. 2024-04-17T10:31:37.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:31:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.531+0000 7ff8acc00640 1 Processor -- start 2024-04-17T10:31:37.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.531+0000 7ff8acc00640 1 -- start start 2024-04-17T10:31:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8acc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff8a81583f0 0x7ff8a8072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8acc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80728c0 0x7ff8a80b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8acc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a815f360 con 0x7ff8a80b5130 2024-04-17T10:31:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8acc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a806bca0 con 0x7ff8a81583f0 2024-04-17T10:31:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8acc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80b3160 con 0x7ff8a80728c0 2024-04-17T10:31:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a6800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff8a80b5130 0x7ff8a8161980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60500/0 (socket says 172.21.15.67:60500) 2024-04-17T10:31:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a6800640 1 -- 172.21.15.67:0/4209810088 learned_addr learned my addr 172.21.15.67:0/4209810088 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a5e00640 1 --2- 172.21.15.67:0/4209810088 >> v2:172.21.15.86:3300/0 conn(0x7ff8a81583f0 0x7ff8a8072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff89d400640 1 --2- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80728c0 0x7ff8a80b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a5400640 1 -- 172.21.15.67:0/4209810088 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544689648 0 0) 0x7ff8a815f360 con 0x7ff8a80b5130 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a5400640 1 -- 172.21.15.67:0/4209810088 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff88c003530 con 0x7ff8a80b5130 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a5e00640 1 -- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80728c0 msgr2=0x7ff8a80b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a5e00640 1 --2- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80728c0 0x7ff8a80b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a5e00640 1 -- 172.21.15.67:0/4209810088 >> v1:172.21.15.67:6789/0 conn(0x7ff8a80b5130 legacy=0x7ff8a8161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.535+0000 7ff8a5e00640 1 -- 172.21.15.67:0/4209810088 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a80b3440 con 0x7ff8a81583f0 2024-04-17T10:31:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8a5e00640 1 --2- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a81583f0 0x7ff8a8072380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff8980097c0 tx=0x7ff89801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=735e9197bdd89c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8a5400640 1 -- 172.21.15.67:0/4209810088 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8980b0070 con 0x7ff8a81583f0 2024-04-17T10:31:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8a5400640 1 -- 172.21.15.67:0/4209810088 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8980ab600 con 0x7ff8a81583f0 2024-04-17T10:31:37.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8a5400640 1 -- 172.21.15.67:0/4209810088 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8980ab9d0 con 0x7ff8a81583f0 2024-04-17T10:31:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 -- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a81583f0 msgr2=0x7ff8a8072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a81583f0 0x7ff8a8072380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ff8980097c0 tx=0x7ff89801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 -- 172.21.15.67:0/4209810088 shutdown_connections 2024-04-17T10:31:37.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80728c0 0x7ff8a80b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4209810088 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a81583f0 0x7ff8a8072380 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 -- 172.21.15.67:0/4209810088 >> 172.21.15.67:0/4209810088 conn(0x7ff8a80b2570 msgr2=0x7ff8a8071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 -- 172.21.15.67:0/4209810088 shutdown_connections 2024-04-17T10:31:37.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 -- 172.21.15.67:0/4209810088 wait complete. 2024-04-17T10:31:37.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 Processor -- start 2024-04-17T10:31:37.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.539+0000 7ff8acc00640 1 -- start start 2024-04-17T10:31:37.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a80728c0 0x7ff8a814fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:37.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80b5130 0x7ff8a81504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:37.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a81583f0 0x7ff8a80aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:37.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80b4200 con 0x7ff8a81583f0 2024-04-17T10:31:37.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a81587d0 con 0x7ff8a80728c0 2024-04-17T10:31:37.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a80b5540 con 0x7ff8a80b5130 2024-04-17T10:31:37.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a81583f0 0x7ff8a80aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:37.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a80728c0 0x7ff8a814fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:37.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a80728c0 0x7ff8a814fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:52408/0 (socket says 172.21.15.67:52408) 2024-04-17T10:31:37.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a5e00640 1 -- 172.21.15.67:0/4081764898 learned_addr learned my addr 172.21.15.67:0/4081764898 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:37.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff89d400640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80b5130 0x7ff8a81504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:37.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a6800640 1 -- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80b5130 msgr2=0x7ff8a81504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:37.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a6800640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80b5130 0x7ff8a81504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a6800640 1 -- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a80728c0 msgr2=0x7ff8a814fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:37.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a6800640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a80728c0 0x7ff8a814fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:37.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a6800640 1 -- 172.21.15.67:0/4081764898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a8151310 con 0x7ff8a81583f0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8a6800640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a81583f0 0x7ff8a80aff60 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000ca80 tx=0x7ff8a000cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff89ea00640 1 -- 172.21.15.67:0/4081764898 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0017070 con 0x7ff8a81583f0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff89ea00640 1 -- 172.21.15.67:0/4081764898 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a000ac30 con 0x7ff8a81583f0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff89ea00640 1 -- 172.21.15.67:0/4081764898 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a0012400 con 0x7ff8a81583f0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a81515f0 con 0x7ff8a81583f0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a8151bb0 con 0x7ff8a81583f0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.543+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a804ef30 con 0x7ff8a81583f0 2024-04-17T10:31:37.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.547+0000 7ff89ea00640 1 -- 172.21.15.67:0/4081764898 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff8a001a070 con 0x7ff8a81583f0 2024-04-17T10:31:37.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.547+0000 7ff89ea00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff884066f60 0x7ff884069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:37.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.547+0000 7ff89ea00640 1 -- 172.21.15.67:0/4081764898 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6001+0+0 (secure 0 0 0) 0x7ff8a0084da0 con 0x7ff8a81583f0 2024-04-17T10:31:37.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.547+0000 7ff8a5e00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff884066f60 0x7ff884069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:37.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.547+0000 7ff8a5e00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff884066f60 0x7ff884069420 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff89801ab50 tx=0x7ff8980a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:37.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.551+0000 7ff89ea00640 1 -- 172.21.15.67:0/4081764898 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff8a0055620 con 0x7ff8a81583f0 2024-04-17T10:31:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:37.747+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff8a80ae7c0 con 0x7ff8a81583f0 2024-04-17T10:31:38.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.351+0000 7ff89ea00640 1 -- 172.21.15.67:0/4081764898 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v61) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff8a00596c0 con 0x7ff8a81583f0 2024-04-17T10:31:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 13 and data pool 14 2024-04-17T10:31:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff884066f60 msgr2=0x7ff884069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff884066f60 0x7ff884069420 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7ff89801ab50 tx=0x7ff8980a6000 comp rx=0 tx=0).stop 2024-04-17T10:31:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a81583f0 msgr2=0x7ff8a80aff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a81583f0 0x7ff8a80aff60 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000ca80 tx=0x7ff8a000cf40 comp rx=0 tx=0).stop 2024-04-17T10:31:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 shutdown_connections 2024-04-17T10:31:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff884066f60 0x7ff884069420 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff8a81583f0 0x7ff8a80aff60 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff8a80b5130 0x7ff8a81504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 --2- 172.21.15.67:0/4081764898 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff8a80728c0 0x7ff8a814fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 >> 172.21.15.67:0/4081764898 conn(0x7ff8a80b2570 msgr2=0x7ff8a8160ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.367+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 shutdown_connections 2024-04-17T10:31:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.371+0000 7ff8acc00640 1 -- 172.21.15.67:0/4081764898 wait complete. 2024-04-17T10:31:38.396 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.547+0000 7febac200640 1 Processor -- start 2024-04-17T10:31:38.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.547+0000 7febac200640 1 -- start start 2024-04-17T10:31:38.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7febac200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feba4145040 0x7feba406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7febac200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba406ef10 0x7feba4071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7febac200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feba406e2f0 con 0x7feba4071850 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7febac200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feba40a86c0 con 0x7feba4145040 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7febac200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba415bf10 con 0x7feba406ef10 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feba4145040 0x7feba406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feba4145040 0x7feba406e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55284/0 (socket says 172.21.15.67:55284) 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 -- 172.21.15.67:0/1215477627 learned_addr learned my addr 172.21.15.67:0/1215477627 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba8e00640 1 --2- 172.21.15.67:0/1215477627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba406ef10 0x7feba4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 -- 172.21.15.67:0/1215477627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba406ef10 msgr2=0x7feba4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 --2- 172.21.15.67:0/1215477627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba406ef10 0x7feba4071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 -- 172.21.15.67:0/1215477627 >> v1:172.21.15.86:6789/0 conn(0x7feba4071850 legacy=0x7feba415f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 -- 172.21.15.67:0/1215477627 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba406d6e0 con 0x7feba4145040 2024-04-17T10:31:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.551+0000 7feba9800640 1 --2- 172.21.15.67:0/1215477627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba4145040 0x7feba406e9d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7feb98009870 tx=0x7feb9801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd57ba593fe1c67 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7feba3e00640 1 -- 172.21.15.67:0/1215477627 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980b5070 con 0x7feba4145040 2024-04-17T10:31:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7feba3e00640 1 -- 172.21.15.67:0/1215477627 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb980a8a50 con 0x7feba4145040 2024-04-17T10:31:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7feba3e00640 1 -- 172.21.15.67:0/1215477627 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980a8e20 con 0x7feba4145040 2024-04-17T10:31:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- 172.21.15.67:0/1215477627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba4145040 msgr2=0x7feba406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 --2- 172.21.15.67:0/1215477627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba4145040 0x7feba406e9d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7feb98009870 tx=0x7feb9801ed50 comp rx=0 tx=0).stop 2024-04-17T10:31:38.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- 172.21.15.67:0/1215477627 shutdown_connections 2024-04-17T10:31:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 --2- 172.21.15.67:0/1215477627 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba406ef10 0x7feba4071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 --2- 172.21.15.67:0/1215477627 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba4145040 0x7feba406e9d0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- 172.21.15.67:0/1215477627 >> 172.21.15.67:0/1215477627 conn(0x7feba406ac90 msgr2=0x7feba406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- 172.21.15.67:0/1215477627 shutdown_connections 2024-04-17T10:31:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- 172.21.15.67:0/1215477627 wait complete. 2024-04-17T10:31:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 Processor -- start 2024-04-17T10:31:38.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- start start 2024-04-17T10:31:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba406ef10 0x7feba415eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba4071850 0x7feba41434c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4145040 0x7feba415aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba406dca0 con 0x7feba406ef10 2024-04-17T10:31:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4071ca0 con 0x7feba4145040 2024-04-17T10:31:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7febac200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feba406df80 con 0x7feba4071850 2024-04-17T10:31:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7feba9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba406ef10 0x7feba415eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7feba9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba406ef10 0x7feba415eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55298/0 (socket says 172.21.15.67:55298) 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.555+0000 7feba9800640 1 -- 172.21.15.67:0/30811550 learned_addr learned my addr 172.21.15.67:0/30811550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7febaa200640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4145040 0x7feba415aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba8e00640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba4071850 0x7feba41434c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba9800640 1 -- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba4071850 msgr2=0x7feba41434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba9800640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba4071850 0x7feba41434c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba9800640 1 -- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4145040 msgr2=0x7feba415aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba9800640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4145040 0x7feba415aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba9800640 1 -- 172.21.15.67:0/30811550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4141af0 con 0x7feba406ef10 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba8e00640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba4071850 0x7feba41434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba9800640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba406ef10 0x7feba415eb70 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7feb980b3040 tx=0x7feb9801baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba1600640 1 -- 172.21.15.67:0/30811550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980b5070 con 0x7feba406ef10 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba1600640 1 -- 172.21.15.67:0/30811550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb980b0de0 con 0x7feba406ef10 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba1600640 1 -- 172.21.15.67:0/30811550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb980bb670 con 0x7feba406ef10 2024-04-17T10:31:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7febac200640 1 -- 172.21.15.67:0/30811550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba413be60 con 0x7feba406ef10 2024-04-17T10:31:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7febac200640 1 -- 172.21.15.67:0/30811550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba4142070 con 0x7feba406ef10 2024-04-17T10:31:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7febac200640 1 -- 172.21.15.67:0/30811550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba4072d50 con 0x7feba406ef10 2024-04-17T10:31:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba1600640 1 -- 172.21.15.67:0/30811550 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7feb980c2060 con 0x7feba406ef10 2024-04-17T10:31:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba1600640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb84066f90 0x7feb84069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.559+0000 7feba1600640 1 -- 172.21.15.67:0/30811550 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7feb98122570 con 0x7feba406ef10 2024-04-17T10:31:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.563+0000 7feba8e00640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb84066f90 0x7feb84069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.563+0000 7feba8e00640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb84066f90 0x7feb84069450 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7feba40b4970 tx=0x7feb90008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:38.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.563+0000 7feba1600640 1 -- 172.21.15.67:0/30811550 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7feb980188b0 con 0x7feba406ef10 2024-04-17T10:31:38.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 -- 172.21.15.67:0/30811550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feba4074e30 con 0x7feba406ef10 2024-04-17T10:31:38.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7feba1600640 1 -- 172.21.15.67:0/30811550 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16850 (secure 0 0 0) 0x7feb980f33f0 con 0x7feba406ef10 2024-04-17T10:31:38.722 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:38.722 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":95,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:37.748967+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:31:36.162943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-17T10:31:36.740175+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 -- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb84066f90 msgr2=0x7feb84069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb84066f90 0x7feb84069450 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7feba40b4970 tx=0x7feb90008040 comp rx=0 tx=0).stop 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 -- 172.21.15.67:0/30811550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba406ef10 msgr2=0x7feba415eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba406ef10 0x7feba415eb70 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7feb980b3040 tx=0x7feb9801baf0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 -- 172.21.15.67:0/30811550 shutdown_connections 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb84066f90 0x7feb84069450 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feba4145040 0x7feba415aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feba4071850 0x7feba41434c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 --2- 172.21.15.67:0/30811550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feba406ef10 0x7feba415eb70 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.731+0000 7febac200640 1 -- 172.21.15.67:0/30811550 >> 172.21.15.67:0/30811550 conn(0x7feba406ac90 msgr2=0x7feba406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:38.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.735+0000 7febac200640 1 -- 172.21.15.67:0/30811550 shutdown_connections 2024-04-17T10:31:38.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.735+0000 7febac200640 1 -- 172.21.15.67:0/30811550 wait complete. 2024-04-17T10:31:38.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febec000640 1 Processor -- start 2024-04-17T10:31:38.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febec000640 1 -- start start 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febec000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7febe4145040 0x7febe406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febec000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7febe406ef10 0x7febe4071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febec000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe406e2f0 con 0x7febe4071850 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febec000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe40a86c0 con 0x7febe4145040 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febec000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe415bf10 con 0x7febe406ef10 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febe9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7febe4145040 0x7febe406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febea000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7febe4071850 0x7febe415f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60512/0 (socket says 172.21.15.67:60512) 2024-04-17T10:31:38.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febea000640 1 -- 172.21.15.67:0/3406410564 learned_addr learned my addr 172.21.15.67:0/3406410564 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febe8c00640 1 --2- 172.21.15.67:0/3406410564 >> v2:172.21.15.86:3300/0 conn(0x7febe406ef10 0x7febe4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febe9600640 1 -- 172.21.15.67:0/3406410564 >> v2:172.21.15.86:3300/0 conn(0x7febe406ef10 msgr2=0x7febe4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febe9600640 1 --2- 172.21.15.67:0/3406410564 >> v2:172.21.15.86:3300/0 conn(0x7febe406ef10 0x7febe4071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febe9600640 1 -- 172.21.15.67:0/3406410564 >> v1:172.21.15.67:6789/0 conn(0x7febe4071850 legacy=0x7febe415f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febe9600640 1 -- 172.21.15.67:0/3406410564 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe4071c60 con 0x7febe4145040 2024-04-17T10:31:38.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.895+0000 7febe3e00640 1 -- 172.21.15.67:0/3406410564 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3732096331 0 0) 0x7febe406e2f0 con 0x7febe4071850 2024-04-17T10:31:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febe9600640 1 --2- 172.21.15.67:0/3406410564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4145040 0x7febe406e9d0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7febe40727b0 tx=0x7febd801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff2cfa77c72fa692 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febe3e00640 1 -- 172.21.15.67:0/3406410564 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd80b5070 con 0x7febe4145040 2024-04-17T10:31:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febe3e00640 1 -- 172.21.15.67:0/3406410564 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd80a8970 con 0x7febe4145040 2024-04-17T10:31:38.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febe3e00640 1 -- 172.21.15.67:0/3406410564 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd80a8d40 con 0x7febe4145040 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 -- 172.21.15.67:0/3406410564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4145040 msgr2=0x7febe406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 --2- 172.21.15.67:0/3406410564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4145040 0x7febe406e9d0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7febe40727b0 tx=0x7febd801ec70 comp rx=0 tx=0).stop 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 -- 172.21.15.67:0/3406410564 shutdown_connections 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 --2- 172.21.15.67:0/3406410564 >> v2:172.21.15.86:3300/0 conn(0x7febe406ef10 0x7febe4071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 --2- 172.21.15.67:0/3406410564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4145040 0x7febe406e9d0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 -- 172.21.15.67:0/3406410564 >> 172.21.15.67:0/3406410564 conn(0x7febe406ac90 msgr2=0x7febe406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 -- 172.21.15.67:0/3406410564 shutdown_connections 2024-04-17T10:31:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.899+0000 7febec000640 1 -- 172.21.15.67:0/3406410564 wait complete. 2024-04-17T10:31:38.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 Processor -- start 2024-04-17T10:31:38.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 -- start start 2024-04-17T10:31:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febe406ef10 0x7febe413fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4071850 0x7febe40b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febe4145040 0x7febe40b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe406df80 con 0x7febe4071850 2024-04-17T10:31:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe406dca0 con 0x7febe406ef10 2024-04-17T10:31:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe406d370 con 0x7febe4145040 2024-04-17T10:31:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4071850 0x7febe40b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4071850 0x7febe40b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55326/0 (socket says 172.21.15.67:55326) 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 -- 172.21.15.67:0/919394259 learned_addr learned my addr 172.21.15.67:0/919394259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe9600640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febe406ef10 0x7febe413fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febea000640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febe4145040 0x7febe40b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 -- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febe4145040 msgr2=0x7febe40b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febe4145040 0x7febe40b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 -- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febe406ef10 msgr2=0x7febe413fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febe406ef10 0x7febe413fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 -- 172.21.15.67:0/919394259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40b33e0 con 0x7febe4071850 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe8c00640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4071850 0x7febe40b2870 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7febd000d800 tx=0x7febd000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe1600640 1 -- 172.21.15.67:0/919394259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd000b6c0 con 0x7febe4071850 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe1600640 1 -- 172.21.15.67:0/919394259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd0019070 con 0x7febe4071850 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febe1600640 1 -- 172.21.15.67:0/919394259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd000ba60 con 0x7febe4071850 2024-04-17T10:31:38.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 -- 172.21.15.67:0/919394259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe40b36c0 con 0x7febe4071850 2024-04-17T10:31:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 -- 172.21.15.67:0/919394259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe4144770 con 0x7febe4071850 2024-04-17T10:31:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.903+0000 7febec000640 1 -- 172.21.15.67:0/919394259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe40b3890 con 0x7febe4071850 2024-04-17T10:31:38.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.907+0000 7febe1600640 1 -- 172.21.15.67:0/919394259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7febd000bc00 con 0x7febe4071850 2024-04-17T10:31:38.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.907+0000 7febe1600640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7febc4066f60 0x7febc4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:38.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.907+0000 7febe1600640 1 -- 172.21.15.67:0/919394259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7febd0085560 con 0x7febe4071850 2024-04-17T10:31:38.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.907+0000 7febe9600640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7febc4066f60 0x7febc4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:38.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.907+0000 7febe9600640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7febc4066f60 0x7febc4069420 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7febd80b2000 tx=0x7febd80aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:38.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:38.911+0000 7febe1600640 1 -- 172.21.15.67:0/919394259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7febd0055d80 con 0x7febe4071850 2024-04-17T10:31:39.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.075+0000 7febec000640 1 -- 172.21.15.67:0/919394259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febe4074e30 con 0x7febe4071850 2024-04-17T10:31:39.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.075+0000 7febe1600640 1 -- 172.21.15.67:0/919394259 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16850 (secure 0 0 0) 0x7febd005c0d0 con 0x7febe4071850 2024-04-17T10:31:39.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:39.067 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":95,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:37.748967+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:31:36.162943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-17T10:31:36.740175+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 -- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7febc4066f60 msgr2=0x7febc4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7febc4066f60 0x7febc4069420 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7febd80b2000 tx=0x7febd80aa070 comp rx=0 tx=0).stop 2024-04-17T10:31:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 -- 172.21.15.67:0/919394259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4071850 msgr2=0x7febe40b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4071850 0x7febe40b2870 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7febd000d800 tx=0x7febd000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 -- 172.21.15.67:0/919394259 shutdown_connections 2024-04-17T10:31:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7febc4066f60 0x7febc4069420 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7febe4145040 0x7febe40b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe4071850 0x7febe40b2870 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 --2- 172.21.15.67:0/919394259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7febe406ef10 0x7febe413fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 -- 172.21.15.67:0/919394259 >> 172.21.15.67:0/919394259 conn(0x7febe406ac90 msgr2=0x7febe40cd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 -- 172.21.15.67:0/919394259 shutdown_connections 2024-04-17T10:31:39.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.079+0000 7febec000640 1 -- 172.21.15.67:0/919394259 wait complete. 2024-04-17T10:31:39.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:31:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.243+0000 7f0ac3600640 1 Processor -- start 2024-04-17T10:31:39.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.243+0000 7f0ac3600640 1 -- start start 2024-04-17T10:31:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac3600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0abc15c580 0x7f0abc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac3600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc15cb30 con 0x7f0abc159440 2024-04-17T10:31:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac3600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0abc0cd610 con 0x7f0abc071af0 2024-04-17T10:31:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac3600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0abc0a8650 con 0x7f0abc15c580 2024-04-17T10:31:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0abc15c580 0x7f0abc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac1600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0abc15c580 0x7f0abc0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55330/0 (socket says 172.21.15.67:55330) 2024-04-17T10:31:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac1600640 1 -- 172.21.15.67:0/2820479084 learned_addr learned my addr 172.21.15.67:0/2820479084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac0c00640 1 --1- 172.21.15.67:0/2820479084 >> v1:172.21.15.67:6789/0 conn(0x7f0abc159440 0x7f0abc0713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60524/0 (socket says 172.21.15.67:60524) 2024-04-17T10:31:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac1600640 1 -- 172.21.15.67:0/2820479084 >> v1:172.21.15.86:6789/0 conn(0x7f0abc071af0 legacy=0x7f0abc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac1600640 1 -- 172.21.15.67:0/2820479084 >> v1:172.21.15.67:6789/0 conn(0x7f0abc159440 legacy=0x7f0abc0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:39.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac1600640 1 -- 172.21.15.67:0/2820479084 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc0cd8f0 con 0x7f0abc15c580 2024-04-17T10:31:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0ac1600640 1 --2- 172.21.15.67:0/2820479084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc15c580 0x7f0abc0ce0f0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009e50 tx=0x7f0ab401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b58f7228e2fcdd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.247+0000 7f0abb400640 1 -- 172.21.15.67:0/2820479084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40a57d0 con 0x7f0abc15c580 2024-04-17T10:31:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0abb400640 1 -- 172.21.15.67:0/2820479084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab40a5970 con 0x7f0abc15c580 2024-04-17T10:31:39.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0abb400640 1 -- 172.21.15.67:0/2820479084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40a5d40 con 0x7f0abc15c580 2024-04-17T10:31:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- 172.21.15.67:0/2820479084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc15c580 msgr2=0x7f0abc0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 --2- 172.21.15.67:0/2820479084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc15c580 0x7f0abc0ce0f0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009e50 tx=0x7f0ab401c9f0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- 172.21.15.67:0/2820479084 shutdown_connections 2024-04-17T10:31:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 --2- 172.21.15.67:0/2820479084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc15c580 0x7f0abc0ce0f0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- 172.21.15.67:0/2820479084 >> 172.21.15.67:0/2820479084 conn(0x7f0abc0b1740 msgr2=0x7f0abc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- 172.21.15.67:0/2820479084 shutdown_connections 2024-04-17T10:31:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- 172.21.15.67:0/2820479084 wait complete. 2024-04-17T10:31:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 Processor -- start 2024-04-17T10:31:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- start start 2024-04-17T10:31:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0abc071af0 0x7f0abc14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:39.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc159440 0x7f0abc14cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0abc15c580 0x7f0abc14e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc15dc50 con 0x7f0abc159440 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc15dad0 con 0x7f0abc15c580 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac3600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0abc15ddd0 con 0x7f0abc071af0 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0abbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc159440 0x7f0abc14cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0abbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc159440 0x7f0abc14cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55340/0 (socket says 172.21.15.67:55340) 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0abbe00640 1 -- 172.21.15.67:0/3170846053 learned_addr learned my addr 172.21.15.67:0/3170846053 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac0c00640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0abc071af0 0x7f0abc14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.251+0000 7f0ac1600640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0abc15c580 0x7f0abc14e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0abbe00640 1 -- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0abc071af0 msgr2=0x7f0abc14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0abbe00640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0abc071af0 0x7f0abc14be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0abbe00640 1 -- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0abc15c580 msgr2=0x7f0abc14e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0abbe00640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0abc15c580 0x7f0abc14e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:39.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0abbe00640 1 -- 172.21.15.67:0/3170846053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc1f6950 con 0x7f0abc159440 2024-04-17T10:31:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0abbe00640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc159440 0x7f0abc14cee0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800b410 tx=0x7f0aa800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ab8c00640 1 -- 172.21.15.67:0/3170846053 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8015030 con 0x7f0abc159440 2024-04-17T10:31:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ab8c00640 1 -- 172.21.15.67:0/3170846053 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8004020 con 0x7f0abc159440 2024-04-17T10:31:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ab8c00640 1 -- 172.21.15.67:0/3170846053 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8004410 con 0x7f0abc159440 2024-04-17T10:31:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0abc1fa300 con 0x7f0abc159440 2024-04-17T10:31:39.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0abc1fa8c0 con 0x7f0abc159440 2024-04-17T10:31:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0abc04ef30 con 0x7f0abc159440 2024-04-17T10:31:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ab8c00640 1 -- 172.21.15.67:0/3170846053 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0aa8018070 con 0x7f0abc159440 2024-04-17T10:31:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ab8c00640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a9c066f60 0x7f0a9c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ab8c00640 1 -- 172.21.15.67:0/3170846053 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0aa8084c90 con 0x7f0abc159440 2024-04-17T10:31:39.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.255+0000 7f0ac0c00640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a9c066f60 0x7f0a9c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:39.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.259+0000 7f0ac0c00640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a9c066f60 0x7f0a9c069420 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000f70 tx=0x7f0ab0007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:39.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.259+0000 7f0ab8c00640 1 -- 172.21.15.67:0/3170846053 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0aa8055420 con 0x7f0abc159440 2024-04-17T10:31:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:39.463+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0abc157450 con 0x7f0abc159440 2024-04-17T10:31:40.316 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T10:31:40.323+0000 7fdb87600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:31:40.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ab8c00640 1 -- 172.21.15.67:0/3170846053 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v63) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0aa80594c0 con 0x7f0abc159440 2024-04-17T10:31:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a9c066f60 msgr2=0x7f0a9c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a9c066f60 0x7f0a9c069420 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0000f70 tx=0x7f0ab0007e90 comp rx=0 tx=0).stop 2024-04-17T10:31:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc159440 msgr2=0x7f0abc14cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc159440 0x7f0abc14cee0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800b410 tx=0x7f0aa800b8d0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 shutdown_connections 2024-04-17T10:31:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a9c066f60 0x7f0a9c069420 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0abc15c580 0x7f0abc14e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0abc159440 0x7f0abc14cee0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 --2- 172.21.15.67:0/3170846053 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0abc071af0 0x7f0abc14be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 >> 172.21.15.67:0/3170846053 conn(0x7f0abc0b1740 msgr2=0x7f0abc0ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 shutdown_connections 2024-04-17T10:31:40.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.367+0000 7f0ac3600640 1 -- 172.21.15.67:0/3170846053 wait complete. 2024-04-17T10:31:40.385 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:31:40.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:31:40.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.543+0000 7f7e67e00640 1 Processor -- start 2024-04-17T10:31:40.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.543+0000 7f7e67e00640 1 -- start start 2024-04-17T10:31:40.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6806d820 0x7f7e680cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e6809a160 0x7f7e6815d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e680cc8b0 con 0x7f7e68161f50 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e6806bda0 con 0x7f7e6809a160 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680cc680 con 0x7f7e6806d820 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e6809a160 0x7f7e6815d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e6809a160 0x7f7e6815d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46302/0 (socket says 172.21.15.67:46302) 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67400640 1 -- 172.21.15.67:0/4009525873 learned_addr learned my addr 172.21.15.67:0/4009525873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e5fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7e68161f50 0x7f7e68099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34454/0 (socket says 172.21.15.67:34454) 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e66a00640 1 --2- 172.21.15.67:0/4009525873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6806d820 0x7f7e680cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67400640 1 -- 172.21.15.67:0/4009525873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6806d820 msgr2=0x7f7e680cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67400640 1 --2- 172.21.15.67:0/4009525873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6806d820 0x7f7e680cc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67400640 1 -- 172.21.15.67:0/4009525873 >> v1:172.21.15.67:6789/0 conn(0x7f7e68161f50 legacy=0x7f7e68099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:40.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.547+0000 7f7e67400640 1 -- 172.21.15.67:0/4009525873 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680ccb90 con 0x7f7e6809a160 2024-04-17T10:31:40.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67400640 1 --2- 172.21.15.67:0/4009525873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6809a160 0x7f7e6815d2a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009e50 tx=0x7f7e6001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4820d9a5a8dfeaec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:40.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e66000640 1 -- 172.21.15.67:0/4009525873 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600a57f0 con 0x7f7e6809a160 2024-04-17T10:31:40.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e66000640 1 -- 172.21.15.67:0/4009525873 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e600a5990 con 0x7f7e6809a160 2024-04-17T10:31:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e66000640 1 -- 172.21.15.67:0/4009525873 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600a5d60 con 0x7f7e6809a160 2024-04-17T10:31:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- 172.21.15.67:0/4009525873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6809a160 msgr2=0x7f7e6815d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 --2- 172.21.15.67:0/4009525873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6809a160 0x7f7e6815d2a0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7e60009e50 tx=0x7f7e6001ca30 comp rx=0 tx=0).stop 2024-04-17T10:31:40.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- 172.21.15.67:0/4009525873 shutdown_connections 2024-04-17T10:31:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 --2- 172.21.15.67:0/4009525873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6809a160 0x7f7e6815d2a0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 --2- 172.21.15.67:0/4009525873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e6806d820 0x7f7e680cc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- 172.21.15.67:0/4009525873 >> 172.21.15.67:0/4009525873 conn(0x7f7e680d1630 msgr2=0x7f7e680d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- 172.21.15.67:0/4009525873 shutdown_connections 2024-04-17T10:31:40.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- 172.21.15.67:0/4009525873 wait complete. 2024-04-17T10:31:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 Processor -- start 2024-04-17T10:31:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- start start 2024-04-17T10:31:40.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6806d820 0x7f7e68070b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:40.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e6809a160 0x7f7e68159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:40.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e68161f50 0x7f7e68159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:40.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e681624b0 con 0x7f7e6806d820 2024-04-17T10:31:40.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e68162330 con 0x7f7e6809a160 2024-04-17T10:31:40.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e67e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680d0450 con 0x7f7e68161f50 2024-04-17T10:31:40.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6806d820 0x7f7e68070b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.551+0000 7f7e66a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6806d820 0x7f7e68070b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46310/0 (socket says 172.21.15.67:46310) 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e66a00640 1 -- 172.21.15.67:0/3349552891 learned_addr learned my addr 172.21.15.67:0/3349552891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e67400640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e68161f50 0x7f7e68159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e5fe00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e6809a160 0x7f7e68159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e66a00640 1 -- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e68161f50 msgr2=0x7f7e68159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e66a00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e68161f50 0x7f7e68159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e66a00640 1 -- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e6809a160 msgr2=0x7f7e68159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e66a00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e6809a160 0x7f7e68159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e66a00640 1 -- 172.21.15.67:0/3349552891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680b4500 con 0x7f7e6806d820 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e5fe00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e6809a160 0x7f7e68159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e66a00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6806d820 0x7f7e68070b20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800e870 tx=0x7f7e5800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e5ea00640 1 -- 172.21.15.67:0/3349552891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58017070 con 0x7f7e6806d820 2024-04-17T10:31:40.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e680b47e0 con 0x7f7e6806d820 2024-04-17T10:31:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e5ea00640 1 -- 172.21.15.67:0/3349552891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e5800c940 con 0x7f7e6806d820 2024-04-17T10:31:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e5ea00640 1 -- 172.21.15.67:0/3349552891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5800ccf0 con 0x7f7e6806d820 2024-04-17T10:31:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e680b4da0 con 0x7f7e6806d820 2024-04-17T10:31:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6804f780 con 0x7f7e6806d820 2024-04-17T10:31:40.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e5ea00640 1 -- 172.21.15.67:0/3349552891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7e5801a070 con 0x7f7e6806d820 2024-04-17T10:31:40.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.555+0000 7f7e5ea00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e44066f60 0x7f7e44069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:40.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.559+0000 7f7e5ea00640 1 -- 172.21.15.67:0/3349552891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7e58085480 con 0x7f7e6806d820 2024-04-17T10:31:40.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.559+0000 7f7e5fe00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e44066f60 0x7f7e44069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:40.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.559+0000 7f7e5fe00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e44066f60 0x7f7e44069420 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7e50007fd0 tx=0x7f7e50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:40.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.563+0000 7f7e5ea00640 1 -- 172.21.15.67:0/3349552891 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7e580562f0 con 0x7f7e6806d820 2024-04-17T10:31:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:40.767+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f7e6815e8e0 con 0x7f7e6806d820 2024-04-17T10:31:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.367+0000 7f7e5ea00640 1 -- 172.21.15.67:0/3349552891 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7e58055bf0 con 0x7f7e6806d820 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e44066f60 msgr2=0x7f7e44069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e44066f60 0x7f7e44069420 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7e50007fd0 tx=0x7f7e50009040 comp rx=0 tx=0).stop 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6806d820 msgr2=0x7f7e68070b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6806d820 0x7f7e68070b20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f7e5800e870 tx=0x7f7e5800ec70 comp rx=0 tx=0).stop 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 shutdown_connections 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e44066f60 0x7f7e44069420 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e68161f50 0x7f7e68159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e6809a160 0x7f7e68159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 --2- 172.21.15.67:0/3349552891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e6806d820 0x7f7e68070b20 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 >> 172.21.15.67:0/3349552891 conn(0x7f7e680d1630 msgr2=0x7f7e68099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 shutdown_connections 2024-04-17T10:31:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.371+0000 7f7e67e00640 1 -- 172.21.15.67:0/3349552891 wait complete. 2024-04-17T10:31:41.387 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:31:41.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:31:41.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4249600640 1 Processor -- start 2024-04-17T10:31:41.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4249600640 1 -- start start 2024-04-17T10:31:41.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4249600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4244160f60 0x7f4244099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4249600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f424409a3b0 0x7f424406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4249600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424409a840 con 0x7f4244099f90 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4249600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424409b170 con 0x7f424409a3b0 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4249600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424406ddf0 con 0x7f4244160f60 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4248c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f424409a3b0 0x7f424406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4248c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f424409a3b0 0x7f424406cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46312/0 (socket says 172.21.15.67:46312) 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4248c00640 1 -- 172.21.15.67:0/2848907329 learned_addr learned my addr 172.21.15.67:0/2848907329 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4243400640 1 --1- 172.21.15.67:0/2848907329 >> v1:172.21.15.67:6789/0 conn(0x7f4244099f90 0x7f42440cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34464/0 (socket says 172.21.15.67:34464) 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4243e00640 1 --2- 172.21.15.67:0/2848907329 >> v2:172.21.15.86:3300/0 conn(0x7f4244160f60 0x7f4244099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4248c00640 1 -- 172.21.15.67:0/2848907329 >> v2:172.21.15.86:3300/0 conn(0x7f4244160f60 msgr2=0x7f4244099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4248c00640 1 --2- 172.21.15.67:0/2848907329 >> v2:172.21.15.86:3300/0 conn(0x7f4244160f60 0x7f4244099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4248c00640 1 -- 172.21.15.67:0/2848907329 >> v1:172.21.15.67:6789/0 conn(0x7f4244099f90 legacy=0x7f42440cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.543+0000 7f4248c00640 1 -- 172.21.15.67:0/2848907329 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424409ab20 con 0x7f424409a3b0 2024-04-17T10:31:41.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4248c00640 1 --2- 172.21.15.67:0/2848907329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424409a3b0 0x7f424406cdb0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f42440ce010 tx=0x7f423c01c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4b5d0b7a65ef1af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:41.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4242a00640 1 -- 172.21.15.67:0/2848907329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c0a58a0 con 0x7f424409a3b0 2024-04-17T10:31:41.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4242a00640 1 -- 172.21.15.67:0/2848907329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c0a5a40 con 0x7f424409a3b0 2024-04-17T10:31:41.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4242a00640 1 -- 172.21.15.67:0/2848907329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c0a5e10 con 0x7f424409a3b0 2024-04-17T10:31:41.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 -- 172.21.15.67:0/2848907329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424409a3b0 msgr2=0x7f424406cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 --2- 172.21.15.67:0/2848907329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424409a3b0 0x7f424406cdb0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f42440ce010 tx=0x7f423c01c9c0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 -- 172.21.15.67:0/2848907329 shutdown_connections 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 --2- 172.21.15.67:0/2848907329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f424409a3b0 0x7f424406cdb0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 --2- 172.21.15.67:0/2848907329 >> v2:172.21.15.86:3300/0 conn(0x7f4244160f60 0x7f4244099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 -- 172.21.15.67:0/2848907329 >> 172.21.15.67:0/2848907329 conn(0x7f42440d1630 msgr2=0x7f42440d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 -- 172.21.15.67:0/2848907329 shutdown_connections 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 -- 172.21.15.67:0/2848907329 wait complete. 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 Processor -- start 2024-04-17T10:31:41.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.547+0000 7f4249600640 1 -- start start 2024-04-17T10:31:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4244099f90 0x7f42441551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4244160f60 0x7f424415fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4244160040 0x7f4244155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42440d0bd0 con 0x7f4244160040 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42440d0a50 con 0x7f4244160f60 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42440d0d50 con 0x7f4244099f90 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4244160040 0x7f4244155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4244160040 0x7f4244155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46322/0 (socket says 172.21.15.67:46322) 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 -- 172.21.15.67:0/4221997017 learned_addr learned my addr 172.21.15.67:0/4221997017 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4243e00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4244099f90 0x7f42441551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4243400640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4244160f60 0x7f424415fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 -- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4244099f90 msgr2=0x7f42441551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4244099f90 0x7f42441551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 -- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4244160f60 msgr2=0x7f424415fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4244160f60 0x7f424415fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 -- 172.21.15.67:0/4221997017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4244156810 con 0x7f4244160040 2024-04-17T10:31:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4243e00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4244099f90 0x7f42441551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4248c00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4244160040 0x7f4244155a90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f423c01a5f0 tx=0x7f423c019760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f423be00640 1 -- 172.21.15.67:0/4221997017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c0a58a0 con 0x7f4244160040 2024-04-17T10:31:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f423be00640 1 -- 172.21.15.67:0/4221997017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c0be070 con 0x7f4244160040 2024-04-17T10:31:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f423be00640 1 -- 172.21.15.67:0/4221997017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c009510 con 0x7f4244160040 2024-04-17T10:31:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4244156af0 con 0x7f4244160040 2024-04-17T10:31:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42440c5420 con 0x7f4244160040 2024-04-17T10:31:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.551+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4244074dc0 con 0x7f4244160040 2024-04-17T10:31:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.555+0000 7f423be00640 1 -- 172.21.15.67:0/4221997017 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f423c0b88b0 con 0x7f4244160040 2024-04-17T10:31:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.555+0000 7f423be00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4224066f90 0x7f4224069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.555+0000 7f423be00640 1 -- 172.21.15.67:0/4221997017 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f423c120b70 con 0x7f4244160040 2024-04-17T10:31:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.555+0000 7f4243e00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4224066f90 0x7f4224069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.555+0000 7f4243e00640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4224066f90 0x7f4224069450 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4234007a70 tx=0x7f4234008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.559+0000 7f423be00640 1 -- 172.21.15.67:0/4221997017 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f423c0165e0 con 0x7f4244160040 2024-04-17T10:31:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.755+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4244156cc0 con 0x7f4244160040 2024-04-17T10:31:41.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.755+0000 7f423be00640 1 -- 172.21.15.67:0/4221997017 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f423c0f1980 con 0x7f4244160040 2024-04-17T10:31:41.748 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:41.748 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:31:37.749417+0000","modified":"2024-04-17T10:31:40.768850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7698":{"gid":7698,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3523329517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3523329517},{"type":"v1","addr":"172.21.15.67:6835","nonce":3523329517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-17T10:31:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 64 2024-04-17T10:31:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4224066f90 msgr2=0x7f4224069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4224066f90 0x7f4224069450 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4234007a70 tx=0x7f4234008040 comp rx=0 tx=0).stop 2024-04-17T10:31:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4244160040 msgr2=0x7f4244155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4244160040 0x7f4244155a90 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f423c01a5f0 tx=0x7f423c019760 comp rx=0 tx=0).stop 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 shutdown_connections 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4224066f90 0x7f4224069450 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4244160040 0x7f4244155a90 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4244160f60 0x7f424415fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 --2- 172.21.15.67:0/4221997017 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4244099f90 0x7f42441551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 >> 172.21.15.67:0/4221997017 conn(0x7f42440d1630 msgr2=0x7f424406bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 shutdown_connections 2024-04-17T10:31:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.759+0000 7f4249600640 1 -- 172.21.15.67:0/4221997017 wait complete. 2024-04-17T10:31:41.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:31:41.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.919+0000 7fe0c5200640 1 Processor -- start 2024-04-17T10:31:41.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.919+0000 7fe0c5200640 1 -- start start 2024-04-17T10:31:41.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0c5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c013e880 0x7fe0c006fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0c5200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0c00cecc0 0x7fe0c006ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0c5200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe0c006b1d0 0x7fe0c0141200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0c5200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c006b640 con 0x7fe0c00cecc0 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0c5200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c015c030 con 0x7fe0c006b1d0 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0c5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c015c310 con 0x7fe0c013e880 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0c00cecc0 0x7fe0c006ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe0c00cecc0 0x7fe0c006ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46338/0 (socket says 172.21.15.67:46338) 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 -- 172.21.15.67:0/565854598 learned_addr learned my addr 172.21.15.67:0/565854598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0bfe00640 1 --2- 172.21.15.67:0/565854598 >> v2:172.21.15.86:3300/0 conn(0x7fe0c006b1d0 0x7fe0c0141200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0bf400640 1 --2- 172.21.15.67:0/565854598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c013e880 0x7fe0c006fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 -- 172.21.15.67:0/565854598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c013e880 msgr2=0x7fe0c006fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/565854598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c013e880 0x7fe0c006fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 -- 172.21.15.67:0/565854598 >> v2:172.21.15.86:3300/0 conn(0x7fe0c006b1d0 msgr2=0x7fe0c0141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/565854598 >> v2:172.21.15.86:3300/0 conn(0x7fe0c006b1d0 0x7fe0c0141200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0b7e00640 1 -- 172.21.15.67:0/565854598 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c015c5f0 con 0x7fe0c00cecc0 2024-04-17T10:31:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.923+0000 7fe0bfe00640 1 --2- 172.21.15.67:0/565854598 >> v2:172.21.15.86:3300/0 conn(0x7fe0c006b1d0 0x7fe0c0141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:41.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/565854598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c00cecc0 0x7fe0c006ac90 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0072000 tx=0x7fe0a801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80bafe68e63f534b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0bea00640 1 -- 172.21.15.67:0/565854598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80b0070 con 0x7fe0c00cecc0 2024-04-17T10:31:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0bea00640 1 -- 172.21.15.67:0/565854598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a80ab600 con 0x7fe0c00cecc0 2024-04-17T10:31:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0bea00640 1 -- 172.21.15.67:0/565854598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a80ab9d0 con 0x7fe0c00cecc0 2024-04-17T10:31:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- 172.21.15.67:0/565854598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c00cecc0 msgr2=0x7fe0c006ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 --2- 172.21.15.67:0/565854598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c00cecc0 0x7fe0c006ac90 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0072000 tx=0x7fe0a801c490 comp rx=0 tx=0).stop 2024-04-17T10:31:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- 172.21.15.67:0/565854598 shutdown_connections 2024-04-17T10:31:41.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 --2- 172.21.15.67:0/565854598 >> v2:172.21.15.86:3300/0 conn(0x7fe0c006b1d0 0x7fe0c0141200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 --2- 172.21.15.67:0/565854598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c00cecc0 0x7fe0c006ac90 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 --2- 172.21.15.67:0/565854598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c013e880 0x7fe0c006fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- 172.21.15.67:0/565854598 >> 172.21.15.67:0/565854598 conn(0x7fe0c00b23f0 msgr2=0x7fe0c00b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- 172.21.15.67:0/565854598 shutdown_connections 2024-04-17T10:31:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- 172.21.15.67:0/565854598 wait complete. 2024-04-17T10:31:41.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 Processor -- start 2024-04-17T10:31:41.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- start start 2024-04-17T10:31:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c006b1d0 0x7fe0c0155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00cecc0 0x7fe0c013d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c013e880 0x7fe0c0154c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c015bdd0 con 0x7fe0c006b1d0 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c015bc50 con 0x7fe0c013e880 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0c5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c00cf140 con 0x7fe0c00cecc0 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c006b1d0 0x7fe0c0155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c006b1d0 0x7fe0c0155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46352/0 (socket says 172.21.15.67:46352) 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.927+0000 7fe0bf400640 1 -- 172.21.15.67:0/1343811955 learned_addr learned my addr 172.21.15.67:0/1343811955 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00cecc0 0x7fe0c013d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0bfe00640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c013e880 0x7fe0c0154c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0bf400640 1 -- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00cecc0 msgr2=0x7fe0c013d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0bf400640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00cecc0 0x7fe0c013d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0bf400640 1 -- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c013e880 msgr2=0x7fe0c0154c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:41.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0bf400640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c013e880 0x7fe0c0154c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0bf400640 1 -- 172.21.15.67:0/1343811955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c013da80 con 0x7fe0c006b1d0 2024-04-17T10:31:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0bf400640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c006b1d0 0x7fe0c0155510 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fe0b000d800 tx=0x7fe0b000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0b7400640 1 -- 172.21.15.67:0/1343811955 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b000b6c0 con 0x7fe0c006b1d0 2024-04-17T10:31:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0b7400640 1 -- 172.21.15.67:0/1343811955 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b0019070 con 0x7fe0c006b1d0 2024-04-17T10:31:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c013dd20 con 0x7fe0c006b1d0 2024-04-17T10:31:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0b7400640 1 -- 172.21.15.67:0/1343811955 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b000ba60 con 0x7fe0c006b1d0 2024-04-17T10:31:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c0140110 con 0x7fe0c006b1d0 2024-04-17T10:31:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c0074dc0 con 0x7fe0c006b1d0 2024-04-17T10:31:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0b7400640 1 -- 172.21.15.67:0/1343811955 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe0b000bc00 con 0x7fe0c006b1d0 2024-04-17T10:31:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0b7400640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe09c066f90 0x7fe09c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.931+0000 7fe0b7400640 1 -- 172.21.15.67:0/1343811955 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe0b00853f0 con 0x7fe0c006b1d0 2024-04-17T10:31:41.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.935+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe09c066f90 0x7fe09c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:41.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.935+0000 7fe0b7e00640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe09c066f90 0x7fe09c069450 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00733e0 tx=0x7fe0a8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:41.935+0000 7fe0b7400640 1 -- 172.21.15.67:0/1343811955 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe0b0055cb0 con 0x7fe0c006b1d0 2024-04-17T10:31:42.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0c006fbc0 con 0x7fe0c006b1d0 2024-04-17T10:31:42.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0b7400640 1 -- 172.21.15.67:0/1343811955 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fe0b0059d50 con 0x7fe0c006b1d0 2024-04-17T10:31:42.094 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:42.094 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:39.352312+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:31:36.162943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-17T10:31:36.740175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:31:42.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe09c066f90 msgr2=0x7fe09c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe09c066f90 0x7fe09c069450 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe0c00733e0 tx=0x7fe0a8008040 comp rx=0 tx=0).stop 2024-04-17T10:31:42.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c006b1d0 msgr2=0x7fe0c0155510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c006b1d0 0x7fe0c0155510 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7fe0b000d800 tx=0x7fe0b000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 shutdown_connections 2024-04-17T10:31:42.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe09c066f90 0x7fe09c069450 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe0c013e880 0x7fe0c0154c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe0c00cecc0 0x7fe0c013d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 --2- 172.21.15.67:0/1343811955 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe0c006b1d0 0x7fe0c0155510 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 >> 172.21.15.67:0/1343811955 conn(0x7fe0c00b23f0 msgr2=0x7fe0c013b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:42.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 shutdown_connections 2024-04-17T10:31:42.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.103+0000 7fe0c5200640 1 -- 172.21.15.67:0/1343811955 wait complete. 2024-04-17T10:31:42.116 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:31:42.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:31:42.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.267+0000 7fef90600640 1 Processor -- start 2024-04-17T10:31:42.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.267+0000 7fef90600640 1 -- start start 2024-04-17T10:31:42.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef90600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fef88145040 0x7fef8806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef90600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef8806ef10 0x7fef88071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef90600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef8806e2f0 con 0x7fef88071850 2024-04-17T10:31:42.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef90600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef880a86c0 con 0x7fef88145040 2024-04-17T10:31:42.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef90600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8815bf10 con 0x7fef8806ef10 2024-04-17T10:31:42.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef8806ef10 0x7fef88071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8dc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fef88145040 0x7fef8806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef8806ef10 0x7fef88071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38560/0 (socket says 172.21.15.67:38560) 2024-04-17T10:31:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8d200640 1 -- 172.21.15.67:0/3842029160 learned_addr learned my addr 172.21.15.67:0/3842029160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8d200640 1 -- 172.21.15.67:0/3842029160 >> v2:172.21.15.86:3300/0 conn(0x7fef88145040 msgr2=0x7fef8806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8d200640 1 --2- 172.21.15.67:0/3842029160 >> v2:172.21.15.86:3300/0 conn(0x7fef88145040 0x7fef8806e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8d200640 1 -- 172.21.15.67:0/3842029160 >> v1:172.21.15.86:6789/0 conn(0x7fef88071850 legacy=0x7fef8815f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.271+0000 7fef8d200640 1 -- 172.21.15.67:0/3842029160 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8806d6e0 con 0x7fef8806ef10 2024-04-17T10:31:42.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef8d200640 1 --2- 172.21.15.67:0/3842029160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef8806ef10 0x7fef88071310 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fef74009870 tx=0x7fef7401ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f234b604fc6ef75 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:42.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef87e00640 1 -- 172.21.15.67:0/3842029160 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef740b5070 con 0x7fef8806ef10 2024-04-17T10:31:42.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef87e00640 1 -- 172.21.15.67:0/3842029160 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef740a8a50 con 0x7fef8806ef10 2024-04-17T10:31:42.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef87e00640 1 -- 172.21.15.67:0/3842029160 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef740a8e20 con 0x7fef8806ef10 2024-04-17T10:31:42.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- 172.21.15.67:0/3842029160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef8806ef10 msgr2=0x7fef88071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 --2- 172.21.15.67:0/3842029160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef8806ef10 0x7fef88071310 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fef74009870 tx=0x7fef7401ed50 comp rx=0 tx=0).stop 2024-04-17T10:31:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- 172.21.15.67:0/3842029160 shutdown_connections 2024-04-17T10:31:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 --2- 172.21.15.67:0/3842029160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef8806ef10 0x7fef88071310 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 --2- 172.21.15.67:0/3842029160 >> v2:172.21.15.86:3300/0 conn(0x7fef88145040 0x7fef8806e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- 172.21.15.67:0/3842029160 >> 172.21.15.67:0/3842029160 conn(0x7fef8806ac90 msgr2=0x7fef8806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- 172.21.15.67:0/3842029160 shutdown_connections 2024-04-17T10:31:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- 172.21.15.67:0/3842029160 wait complete. 2024-04-17T10:31:42.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 Processor -- start 2024-04-17T10:31:42.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- start start 2024-04-17T10:31:42.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef8806ef10 0x7fef8815af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef88071850 0x7fef880ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef88145040 0x7fef88132b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8806dca0 con 0x7fef8806ef10 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef88071ca0 con 0x7fef88071850 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.275+0000 7fef90600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef8806df80 con 0x7fef88145040 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef8806ef10 0x7fef8815af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef8806ef10 0x7fef8815af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46368/0 (socket says 172.21.15.67:46368) 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 -- 172.21.15.67:0/75753807 learned_addr learned my addr 172.21.15.67:0/75753807 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8d200640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef88071850 0x7fef880ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8e600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef88145040 0x7fef88132b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 -- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef88145040 msgr2=0x7fef88132b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef88145040 0x7fef88132b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 -- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef88071850 msgr2=0x7fef880ad670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef88071850 0x7fef880ad670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 -- 172.21.15.67:0/75753807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8815b490 con 0x7fef8806ef10 2024-04-17T10:31:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef8dc00640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef8806ef10 0x7fef8815af50 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e870 tx=0x7fef7c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef85600640 1 -- 172.21.15.67:0/75753807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c017070 con 0x7fef8806ef10 2024-04-17T10:31:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef85600640 1 -- 172.21.15.67:0/75753807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7c00c940 con 0x7fef8806ef10 2024-04-17T10:31:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef880ae110 con 0x7fef8806ef10 2024-04-17T10:31:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef85600640 1 -- 172.21.15.67:0/75753807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7c00ccf0 con 0x7fef8806ef10 2024-04-17T10:31:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef88130cc0 con 0x7fef8806ef10 2024-04-17T10:31:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef88072d50 con 0x7fef8806ef10 2024-04-17T10:31:42.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef85600640 1 -- 172.21.15.67:0/75753807 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fef7c01a070 con 0x7fef8806ef10 2024-04-17T10:31:42.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.279+0000 7fef85600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef68066f60 0x7fef68069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.283+0000 7fef85600640 1 -- 172.21.15.67:0/75753807 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fef7c0850f0 con 0x7fef8806ef10 2024-04-17T10:31:42.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.283+0000 7fef8d200640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef68066f60 0x7fef68069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.283+0000 7fef8d200640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef68066f60 0x7fef68069420 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fef740b2000 tx=0x7fef74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:42.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.283+0000 7fef85600640 1 -- 172.21.15.67:0/75753807 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fef7c055800 con 0x7fef8806ef10 2024-04-17T10:31:42.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.447+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef88074e30 con 0x7fef68066f60 2024-04-17T10:31:42.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.451+0000 7fef85600640 1 -- 172.21.15.67:0/75753807 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246197 (secure 0 0 0) 0x7fef88074e30 con 0x7fef68066f60 2024-04-17T10:31:42.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:31:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef68066f60 msgr2=0x7fef68069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef68066f60 0x7fef68069420 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fef740b2000 tx=0x7fef74009040 comp rx=0 tx=0).stop 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef8806ef10 msgr2=0x7fef8815af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef8806ef10 0x7fef8815af50 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7fef7c00e870 tx=0x7fef7c00ec70 comp rx=0 tx=0).stop 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 shutdown_connections 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef68066f60 0x7fef68069420 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fef88145040 0x7fef88132b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fef88071850 0x7fef880ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 --2- 172.21.15.67:0/75753807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef8806ef10 0x7fef8815af50 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 >> 172.21.15.67:0/75753807 conn(0x7fef8806ac90 msgr2=0x7fef8815e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 shutdown_connections 2024-04-17T10:31:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.455+0000 7fef90600640 1 -- 172.21.15.67:0/75753807 wait complete. 2024-04-17T10:31:42.467 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-17T10:31:42.234222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591980,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":138,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412460,"kb_used_data":10292,"kb_used_omap":12,"kb_used_meta":402035,"kb_avail":754562260,"statfs":{"total":773094113280,"available":772671754240,"internally_reserved":0,"allocated":10539008,"data_stored":5517724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":411684401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.903728"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366379+0000","last_change":"2024-04-17T10:31:38.372177+0000","last_active":"2024-04-17T10:31:39.366379+0000","last_peered":"2024-04-17T10:31:39.366379+0000","last_clean":"2024-04-17T10:31:39.366379+0000","last_became_active":"2024-04-17T10:31:38.371875+0000","last_became_peered":"2024-04-17T10:31:38.371875+0000","last_unstale":"2024-04-17T10:31:39.366379+0000","last_undegraded":"2024-04-17T10:31:39.366379+0000","last_fullsized":"2024-04-17T10:31:39.366379+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:36:29.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366082+0000","last_change":"2024-04-17T10:31:38.362877+0000","last_active":"2024-04-17T10:31:39.366082+0000","last_peered":"2024-04-17T10:31:39.366082+0000","last_clean":"2024-04-17T10:31:39.366082+0000","last_became_active":"2024-04-17T10:31:38.362413+0000","last_became_peered":"2024-04-17T10:31:38.362413+0000","last_unstale":"2024-04-17T10:31:39.366082+0000","last_undegraded":"2024-04-17T10:31:39.366082+0000","last_fullsized":"2024-04-17T10:31:39.366082+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:03.031089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637416+0000","last_change":"2024-04-17T10:31:38.367300+0000","last_active":"2024-04-17T10:31:39.637416+0000","last_peered":"2024-04-17T10:31:39.637416+0000","last_clean":"2024-04-17T10:31:39.637416+0000","last_became_active":"2024-04-17T10:31:38.366715+0000","last_became_peered":"2024-04-17T10:31:38.366715+0000","last_unstale":"2024-04-17T10:31:39.637416+0000","last_undegraded":"2024-04-17T10:31:39.637416+0000","last_fullsized":"2024-04-17T10:31:39.637416+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:02.550932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193707+0000","last_change":"2024-04-17T10:31:38.382652+0000","last_active":"2024-04-17T10:31:40.193707+0000","last_peered":"2024-04-17T10:31:40.193707+0000","last_clean":"2024-04-17T10:31:40.193707+0000","last_became_active":"2024-04-17T10:31:38.382384+0000","last_became_peered":"2024-04-17T10:31:38.382384+0000","last_unstale":"2024-04-17T10:31:40.193707+0000","last_undegraded":"2024-04-17T10:31:40.193707+0000","last_fullsized":"2024-04-17T10:31:40.193707+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:20.999198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637175+0000","last_change":"2024-04-17T10:31:38.367243+0000","last_active":"2024-04-17T10:31:39.637175+0000","last_peered":"2024-04-17T10:31:39.637175+0000","last_clean":"2024-04-17T10:31:39.637175+0000","last_became_active":"2024-04-17T10:31:38.366952+0000","last_became_peered":"2024-04-17T10:31:38.366952+0000","last_unstale":"2024-04-17T10:31:39.637175+0000","last_undegraded":"2024-04-17T10:31:39.637175+0000","last_fullsized":"2024-04-17T10:31:39.637175+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:24.096194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367386+0000","last_change":"2024-04-17T10:31:38.369001+0000","last_active":"2024-04-17T10:31:39.367386+0000","last_peered":"2024-04-17T10:31:39.367386+0000","last_clean":"2024-04-17T10:31:39.367386+0000","last_became_active":"2024-04-17T10:31:38.367830+0000","last_became_peered":"2024-04-17T10:31:38.367830+0000","last_unstale":"2024-04-17T10:31:39.367386+0000","last_undegraded":"2024-04-17T10:31:39.367386+0000","last_fullsized":"2024-04-17T10:31:39.367386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:52.706957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367226+0000","last_change":"2024-04-17T10:31:38.357758+0000","last_active":"2024-04-17T10:31:39.367226+0000","last_peered":"2024-04-17T10:31:39.367226+0000","last_clean":"2024-04-17T10:31:39.367226+0000","last_became_active":"2024-04-17T10:31:38.357482+0000","last_became_peered":"2024-04-17T10:31:38.357482+0000","last_unstale":"2024-04-17T10:31:39.367226+0000","last_undegraded":"2024-04-17T10:31:39.367226+0000","last_fullsized":"2024-04-17T10:31:39.367226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:48.241446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637619+0000","last_change":"2024-04-17T10:31:38.366467+0000","last_active":"2024-04-17T10:31:39.637619+0000","last_peered":"2024-04-17T10:31:39.637619+0000","last_clean":"2024-04-17T10:31:39.637619+0000","last_became_active":"2024-04-17T10:31:38.366066+0000","last_became_peered":"2024-04-17T10:31:38.366066+0000","last_unstale":"2024-04-17T10:31:39.637619+0000","last_undegraded":"2024-04-17T10:31:39.637619+0000","last_fullsized":"2024-04-17T10:31:39.637619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:08.729085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637251+0000","last_change":"2024-04-17T10:31:38.363077+0000","last_active":"2024-04-17T10:31:39.637251+0000","last_peered":"2024-04-17T10:31:39.637251+0000","last_clean":"2024-04-17T10:31:39.637251+0000","last_became_active":"2024-04-17T10:31:38.362827+0000","last_became_peered":"2024-04-17T10:31:38.362827+0000","last_unstale":"2024-04-17T10:31:39.637251+0000","last_undegraded":"2024-04-17T10:31:39.637251+0000","last_fullsized":"2024-04-17T10:31:39.637251+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:56:20.141939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193544+0000","last_change":"2024-04-17T10:31:38.390689+0000","last_active":"2024-04-17T10:31:40.193544+0000","last_peered":"2024-04-17T10:31:40.193544+0000","last_clean":"2024-04-17T10:31:40.193544+0000","last_became_active":"2024-04-17T10:31:38.389673+0000","last_became_peered":"2024-04-17T10:31:38.389673+0000","last_unstale":"2024-04-17T10:31:40.193544+0000","last_undegraded":"2024-04-17T10:31:40.193544+0000","last_fullsized":"2024-04-17T10:31:40.193544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:05:33.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194163+0000","last_change":"2024-04-17T10:31:38.386404+0000","last_active":"2024-04-17T10:31:40.194163+0000","last_peered":"2024-04-17T10:31:40.194163+0000","last_clean":"2024-04-17T10:31:40.194163+0000","last_became_active":"2024-04-17T10:31:38.383973+0000","last_became_peered":"2024-04-17T10:31:38.383973+0000","last_unstale":"2024-04-17T10:31:40.194163+0000","last_undegraded":"2024-04-17T10:31:40.194163+0000","last_fullsized":"2024-04-17T10:31:40.194163+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:12.821691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366183+0000","last_change":"2024-04-17T10:31:38.362736+0000","last_active":"2024-04-17T10:31:39.366183+0000","last_peered":"2024-04-17T10:31:39.366183+0000","last_clean":"2024-04-17T10:31:39.366183+0000","last_became_active":"2024-04-17T10:31:38.362405+0000","last_became_peered":"2024-04-17T10:31:38.362405+0000","last_unstale":"2024-04-17T10:31:39.366183+0000","last_undegraded":"2024-04-17T10:31:39.366183+0000","last_fullsized":"2024-04-17T10:31:39.366183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:55.770687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.367062+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.366356+0000","last_became_peered":"2024-04-17T10:31:38.366356+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:34.604484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365841+0000","last_change":"2024-04-17T10:31:38.363969+0000","last_active":"2024-04-17T10:31:39.365841+0000","last_peered":"2024-04-17T10:31:39.365841+0000","last_clean":"2024-04-17T10:31:39.365841+0000","last_became_active":"2024-04-17T10:31:38.363731+0000","last_became_peered":"2024-04-17T10:31:38.363731+0000","last_unstale":"2024-04-17T10:31:39.365841+0000","last_undegraded":"2024-04-17T10:31:39.365841+0000","last_fullsized":"2024-04-17T10:31:39.365841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:25.031678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367325+0000","last_change":"2024-04-17T10:31:38.364443+0000","last_active":"2024-04-17T10:31:39.367325+0000","last_peered":"2024-04-17T10:31:39.367325+0000","last_clean":"2024-04-17T10:31:39.367325+0000","last_became_active":"2024-04-17T10:31:38.364134+0000","last_became_peered":"2024-04-17T10:31:38.364134+0000","last_unstale":"2024-04-17T10:31:39.367325+0000","last_undegraded":"2024-04-17T10:31:39.367325+0000","last_fullsized":"2024-04-17T10:31:39.367325+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:13:32.599449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637675+0000","last_change":"2024-04-17T10:31:38.367434+0000","last_active":"2024-04-17T10:31:39.637675+0000","last_peered":"2024-04-17T10:31:39.637675+0000","last_clean":"2024-04-17T10:31:39.637675+0000","last_became_active":"2024-04-17T10:31:38.366004+0000","last_became_peered":"2024-04-17T10:31:38.366004+0000","last_unstale":"2024-04-17T10:31:39.637675+0000","last_undegraded":"2024-04-17T10:31:39.637675+0000","last_fullsized":"2024-04-17T10:31:39.637675+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:20.424487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193672+0000","last_change":"2024-04-17T10:31:38.387676+0000","last_active":"2024-04-17T10:31:40.193672+0000","last_peered":"2024-04-17T10:31:40.193672+0000","last_clean":"2024-04-17T10:31:40.193672+0000","last_became_active":"2024-04-17T10:31:38.380682+0000","last_became_peered":"2024-04-17T10:31:38.380682+0000","last_unstale":"2024-04-17T10:31:40.193672+0000","last_undegraded":"2024-04-17T10:31:40.193672+0000","last_fullsized":"2024-04-17T10:31:40.193672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:59.714995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193655+0000","last_change":"2024-04-17T10:31:38.379789+0000","last_active":"2024-04-17T10:31:40.193655+0000","last_peered":"2024-04-17T10:31:40.193655+0000","last_clean":"2024-04-17T10:31:40.193655+0000","last_became_active":"2024-04-17T10:31:38.379237+0000","last_became_peered":"2024-04-17T10:31:38.379237+0000","last_unstale":"2024-04-17T10:31:40.193655+0000","last_undegraded":"2024-04-17T10:31:40.193655+0000","last_fullsized":"2024-04-17T10:31:40.193655+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:01.930369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.338128+0000","last_change":"2024-04-17T10:31:36.338128+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.338128+0000","last_undegraded":"2024-04-17T10:31:36.338128+0000","last_fullsized":"2024-04-17T10:31:36.338128+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.386544+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.380593+0000","last_became_peered":"2024-04-17T10:31:38.380593+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:53:17.992647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335826+0000","last_change":"2024-04-17T10:31:36.335826+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335826+0000","last_undegraded":"2024-04-17T10:31:36.335826+0000","last_fullsized":"2024-04-17T10:31:36.335826+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340523+0000","last_change":"2024-04-17T10:31:36.340523+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340523+0000","last_undegraded":"2024-04-17T10:31:36.340523+0000","last_fullsized":"2024-04-17T10:31:36.340523+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194225+0000","last_change":"2024-04-17T10:31:38.383915+0000","last_active":"2024-04-17T10:31:40.194225+0000","last_peered":"2024-04-17T10:31:40.194225+0000","last_clean":"2024-04-17T10:31:40.194225+0000","last_became_active":"2024-04-17T10:31:38.382791+0000","last_became_peered":"2024-04-17T10:31:38.382791+0000","last_unstale":"2024-04-17T10:31:40.194225+0000","last_undegraded":"2024-04-17T10:31:40.194225+0000","last_fullsized":"2024-04-17T10:31:40.194225+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:12.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337899+0000","last_change":"2024-04-17T10:31:36.337899+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337899+0000","last_undegraded":"2024-04-17T10:31:36.337899+0000","last_fullsized":"2024-04-17T10:31:36.337899+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637195+0000","last_change":"2024-04-17T10:31:38.366551+0000","last_active":"2024-04-17T10:31:39.637195+0000","last_peered":"2024-04-17T10:31:39.637195+0000","last_clean":"2024-04-17T10:31:39.637195+0000","last_became_active":"2024-04-17T10:31:38.366244+0000","last_became_peered":"2024-04-17T10:31:38.366244+0000","last_unstale":"2024-04-17T10:31:39.637195+0000","last_undegraded":"2024-04-17T10:31:39.637195+0000","last_fullsized":"2024-04-17T10:31:39.637195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:37.769462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.334883+0000","last_change":"2024-04-17T10:31:36.334883+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.334883+0000","last_undegraded":"2024-04-17T10:31:36.334883+0000","last_fullsized":"2024-04-17T10:31:36.334883+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637612+0000","last_change":"2024-04-17T10:31:37.366420+0000","last_active":"2024-04-17T10:31:39.637612+0000","last_peered":"2024-04-17T10:31:39.637612+0000","last_clean":"2024-04-17T10:31:39.637612+0000","last_became_active":"2024-04-17T10:31:37.365712+0000","last_became_peered":"2024-04-17T10:31:37.365712+0000","last_unstale":"2024-04-17T10:31:39.637612+0000","last_undegraded":"2024-04-17T10:31:39.637612+0000","last_fullsized":"2024-04-17T10:31:39.637612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:53.669724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193705+0000","last_change":"2024-04-17T10:31:38.370365+0000","last_active":"2024-04-17T10:31:40.193705+0000","last_peered":"2024-04-17T10:31:40.193705+0000","last_clean":"2024-04-17T10:31:40.193705+0000","last_became_active":"2024-04-17T10:31:38.370089+0000","last_became_peered":"2024-04-17T10:31:38.370089+0000","last_unstale":"2024-04-17T10:31:40.193705+0000","last_undegraded":"2024-04-17T10:31:40.193705+0000","last_fullsized":"2024-04-17T10:31:40.193705+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:56.298204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365982+0000","last_change":"2024-04-17T10:31:37.354469+0000","last_active":"2024-04-17T10:31:39.365982+0000","last_peered":"2024-04-17T10:31:39.365982+0000","last_clean":"2024-04-17T10:31:39.365982+0000","last_became_active":"2024-04-17T10:31:37.354215+0000","last_became_peered":"2024-04-17T10:31:37.354215+0000","last_unstale":"2024-04-17T10:31:39.365982+0000","last_undegraded":"2024-04-17T10:31:39.365982+0000","last_fullsized":"2024-04-17T10:31:39.365982+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:33.318986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.364135+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.363335+0000","last_became_peered":"2024-04-17T10:31:38.363335+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:04.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.341274+0000","last_change":"2024-04-17T10:31:36.341274+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.341274+0000","last_undegraded":"2024-04-17T10:31:36.341274+0000","last_fullsized":"2024-04-17T10:31:36.341274+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340070+0000","last_change":"2024-04-17T10:31:36.340070+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340070+0000","last_undegraded":"2024-04-17T10:31:36.340070+0000","last_fullsized":"2024-04-17T10:31:36.340070+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366370+0000","last_change":"2024-04-17T10:31:37.359814+0000","last_active":"2024-04-17T10:31:39.366370+0000","last_peered":"2024-04-17T10:31:39.366370+0000","last_clean":"2024-04-17T10:31:39.366370+0000","last_became_active":"2024-04-17T10:31:37.359486+0000","last_became_peered":"2024-04-17T10:31:37.359486+0000","last_unstale":"2024-04-17T10:31:39.366370+0000","last_undegraded":"2024-04-17T10:31:39.366370+0000","last_fullsized":"2024-04-17T10:31:39.366370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:23:28.064562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366322+0000","last_change":"2024-04-17T10:31:38.371429+0000","last_active":"2024-04-17T10:31:39.366322+0000","last_peered":"2024-04-17T10:31:39.366322+0000","last_clean":"2024-04-17T10:31:39.366322+0000","last_became_active":"2024-04-17T10:31:38.371073+0000","last_became_peered":"2024-04-17T10:31:38.371073+0000","last_unstale":"2024-04-17T10:31:39.366322+0000","last_undegraded":"2024-04-17T10:31:39.366322+0000","last_fullsized":"2024-04-17T10:31:39.366322+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:01.747376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.336055+0000","last_change":"2024-04-17T10:31:36.336055+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.336055+0000","last_undegraded":"2024-04-17T10:31:36.336055+0000","last_fullsized":"2024-04-17T10:31:36.336055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637829+0000","last_change":"2024-04-17T10:31:38.366147+0000","last_active":"2024-04-17T10:31:39.637829+0000","last_peered":"2024-04-17T10:31:39.637829+0000","last_clean":"2024-04-17T10:31:39.637829+0000","last_became_active":"2024-04-17T10:31:38.365889+0000","last_became_peered":"2024-04-17T10:31:38.365889+0000","last_unstale":"2024-04-17T10:31:39.637829+0000","last_undegraded":"2024-04-17T10:31:39.637829+0000","last_fullsized":"2024-04-17T10:31:39.637829+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:18.111634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"95'6","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193733+0000","last_change":"2024-04-17T10:31:37.367318+0000","last_active":"2024-04-17T10:31:40.193733+0000","last_peered":"2024-04-17T10:31:40.193733+0000","last_clean":"2024-04-17T10:31:40.193733+0000","last_became_active":"2024-04-17T10:31:37.367023+0000","last_became_peered":"2024-04-17T10:31:37.367023+0000","last_unstale":"2024-04-17T10:31:40.193733+0000","last_undegraded":"2024-04-17T10:31:40.193733+0000","last_fullsized":"2024-04-17T10:31:40.193733+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:02.024574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637403+0000","last_change":"2024-04-17T10:31:38.362727+0000","last_active":"2024-04-17T10:31:39.637403+0000","last_peered":"2024-04-17T10:31:39.637403+0000","last_clean":"2024-04-17T10:31:39.637403+0000","last_became_active":"2024-04-17T10:31:38.362083+0000","last_became_peered":"2024-04-17T10:31:38.362083+0000","last_unstale":"2024-04-17T10:31:39.637403+0000","last_undegraded":"2024-04-17T10:31:39.637403+0000","last_fullsized":"2024-04-17T10:31:39.637403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:42.796034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340608+0000","last_change":"2024-04-17T10:31:36.340608+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340608+0000","last_undegraded":"2024-04-17T10:31:36.340608+0000","last_fullsized":"2024-04-17T10:31:36.340608+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193763+0000","last_change":"2024-04-17T10:31:37.354839+0000","last_active":"2024-04-17T10:31:40.193763+0000","last_peered":"2024-04-17T10:31:40.193763+0000","last_clean":"2024-04-17T10:31:40.193763+0000","last_became_active":"2024-04-17T10:31:37.354538+0000","last_became_peered":"2024-04-17T10:31:37.354538+0000","last_unstale":"2024-04-17T10:31:40.193763+0000","last_undegraded":"2024-04-17T10:31:40.193763+0000","last_fullsized":"2024-04-17T10:31:40.193763+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:10.283370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366942+0000","last_change":"2024-04-17T10:31:38.368131+0000","last_active":"2024-04-17T10:31:39.366942+0000","last_peered":"2024-04-17T10:31:39.366942+0000","last_clean":"2024-04-17T10:31:39.366942+0000","last_became_active":"2024-04-17T10:31:38.367815+0000","last_became_peered":"2024-04-17T10:31:38.367815+0000","last_unstale":"2024-04-17T10:31:39.366942+0000","last_undegraded":"2024-04-17T10:31:39.366942+0000","last_fullsized":"2024-04-17T10:31:39.366942+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:39.116896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637313+0000","last_change":"2024-04-17T10:31:37.358749+0000","last_active":"2024-04-17T10:31:39.637313+0000","last_peered":"2024-04-17T10:31:39.637313+0000","last_clean":"2024-04-17T10:31:39.637313+0000","last_became_active":"2024-04-17T10:31:37.357922+0000","last_became_peered":"2024-04-17T10:31:37.357922+0000","last_unstale":"2024-04-17T10:31:39.637313+0000","last_undegraded":"2024-04-17T10:31:39.637313+0000","last_fullsized":"2024-04-17T10:31:39.637313+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:23.174223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367510+0000","last_change":"2024-04-17T10:31:38.356263+0000","last_active":"2024-04-17T10:31:39.367510+0000","last_peered":"2024-04-17T10:31:39.367510+0000","last_clean":"2024-04-17T10:31:39.367510+0000","last_became_active":"2024-04-17T10:31:38.355993+0000","last_became_peered":"2024-04-17T10:31:38.355993+0000","last_unstale":"2024-04-17T10:31:39.367510+0000","last_undegraded":"2024-04-17T10:31:39.367510+0000","last_fullsized":"2024-04-17T10:31:39.367510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:55.031187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"95'8","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637287+0000","last_change":"2024-04-17T10:31:37.359691+0000","last_active":"2024-04-17T10:31:39.637287+0000","last_peered":"2024-04-17T10:31:39.637287+0000","last_clean":"2024-04-17T10:31:39.637287+0000","last_became_active":"2024-04-17T10:31:37.359365+0000","last_became_peered":"2024-04-17T10:31:37.359365+0000","last_unstale":"2024-04-17T10:31:39.637287+0000","last_undegraded":"2024-04-17T10:31:39.637287+0000","last_fullsized":"2024-04-17T10:31:39.637287+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:00:19.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.379892+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.379586+0000","last_became_peered":"2024-04-17T10:31:38.379586+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:35.337686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337309+0000","last_change":"2024-04-17T10:31:36.337309+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337309+0000","last_undegraded":"2024-04-17T10:31:36.337309+0000","last_fullsized":"2024-04-17T10:31:36.337309+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340462+0000","last_change":"2024-04-17T10:31:36.340462+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340462+0000","last_undegraded":"2024-04-17T10:31:36.340462+0000","last_fullsized":"2024-04-17T10:31:36.340462+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335517+0000","last_change":"2024-04-17T10:31:36.335517+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335517+0000","last_undegraded":"2024-04-17T10:31:36.335517+0000","last_fullsized":"2024-04-17T10:31:36.335517+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194178+0000","last_change":"2024-04-17T10:31:38.383600+0000","last_active":"2024-04-17T10:31:40.194178+0000","last_peered":"2024-04-17T10:31:40.194178+0000","last_clean":"2024-04-17T10:31:40.194178+0000","last_became_active":"2024-04-17T10:31:38.383354+0000","last_became_peered":"2024-04-17T10:31:38.383354+0000","last_unstale":"2024-04-17T10:31:40.194178+0000","last_undegraded":"2024-04-17T10:31:40.194178+0000","last_fullsized":"2024-04-17T10:31:40.194178+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:25:53.140820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340140+0000","last_change":"2024-04-17T10:31:36.340140+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340140+0000","last_undegraded":"2024-04-17T10:31:36.340140+0000","last_fullsized":"2024-04-17T10:31:36.340140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366890+0000","last_change":"2024-04-17T10:31:37.365058+0000","last_active":"2024-04-17T10:31:39.366890+0000","last_peered":"2024-04-17T10:31:39.366890+0000","last_clean":"2024-04-17T10:31:39.366890+0000","last_became_active":"2024-04-17T10:31:37.364541+0000","last_became_peered":"2024-04-17T10:31:37.364541+0000","last_unstale":"2024-04-17T10:31:39.366890+0000","last_undegraded":"2024-04-17T10:31:39.366890+0000","last_fullsized":"2024-04-17T10:31:39.366890+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:38.273762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340172+0000","last_change":"2024-04-17T10:31:36.340172+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340172+0000","last_undegraded":"2024-04-17T10:31:36.340172+0000","last_fullsized":"2024-04-17T10:31:36.340172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193864+0000","last_change":"2024-04-17T10:31:38.376121+0000","last_active":"2024-04-17T10:31:40.193864+0000","last_peered":"2024-04-17T10:31:40.193864+0000","last_clean":"2024-04-17T10:31:40.193864+0000","last_became_active":"2024-04-17T10:31:38.375833+0000","last_became_peered":"2024-04-17T10:31:38.375833+0000","last_unstale":"2024-04-17T10:31:40.193864+0000","last_undegraded":"2024-04-17T10:31:40.193864+0000","last_fullsized":"2024-04-17T10:31:40.193864+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:35:43.598662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366769+0000","last_change":"2024-04-17T10:31:37.362595+0000","last_active":"2024-04-17T10:31:39.366769+0000","last_peered":"2024-04-17T10:31:39.366769+0000","last_clean":"2024-04-17T10:31:39.366769+0000","last_became_active":"2024-04-17T10:31:37.362307+0000","last_became_peered":"2024-04-17T10:31:37.362307+0000","last_unstale":"2024-04-17T10:31:39.366769+0000","last_undegraded":"2024-04-17T10:31:39.366769+0000","last_fullsized":"2024-04-17T10:31:39.366769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:15:47.101230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340408+0000","last_change":"2024-04-17T10:31:36.340408+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340408+0000","last_undegraded":"2024-04-17T10:31:36.340408+0000","last_fullsized":"2024-04-17T10:31:36.340408+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194103+0000","last_change":"2024-04-17T10:31:38.386818+0000","last_active":"2024-04-17T10:31:40.194103+0000","last_peered":"2024-04-17T10:31:40.194103+0000","last_clean":"2024-04-17T10:31:40.194103+0000","last_became_active":"2024-04-17T10:31:38.383939+0000","last_became_peered":"2024-04-17T10:31:38.383939+0000","last_unstale":"2024-04-17T10:31:40.194103+0000","last_undegraded":"2024-04-17T10:31:40.194103+0000","last_fullsized":"2024-04-17T10:31:40.194103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:01.834781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335266+0000","last_change":"2024-04-17T10:31:36.335266+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335266+0000","last_undegraded":"2024-04-17T10:31:36.335266+0000","last_fullsized":"2024-04-17T10:31:36.335266+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194023+0000","last_change":"2024-04-17T10:31:38.376661+0000","last_active":"2024-04-17T10:31:40.194023+0000","last_peered":"2024-04-17T10:31:40.194023+0000","last_clean":"2024-04-17T10:31:40.194023+0000","last_became_active":"2024-04-17T10:31:38.376364+0000","last_became_peered":"2024-04-17T10:31:38.376364+0000","last_unstale":"2024-04-17T10:31:40.194023+0000","last_undegraded":"2024-04-17T10:31:40.194023+0000","last_fullsized":"2024-04-17T10:31:40.194023+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:28.335090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335050+0000","last_change":"2024-04-17T10:31:36.335050+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335050+0000","last_undegraded":"2024-04-17T10:31:36.335050+0000","last_fullsized":"2024-04-17T10:31:36.335050+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337226+0000","last_change":"2024-04-17T10:31:36.337226+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337226+0000","last_undegraded":"2024-04-17T10:31:36.337226+0000","last_fullsized":"2024-04-17T10:31:36.337226+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366830+0000","last_change":"2024-04-17T10:31:38.384626+0000","last_active":"2024-04-17T10:31:39.366830+0000","last_peered":"2024-04-17T10:31:39.366830+0000","last_clean":"2024-04-17T10:31:39.366830+0000","last_became_active":"2024-04-17T10:31:38.384373+0000","last_became_peered":"2024-04-17T10:31:38.384373+0000","last_unstale":"2024-04-17T10:31:39.366830+0000","last_undegraded":"2024-04-17T10:31:39.366830+0000","last_fullsized":"2024-04-17T10:31:39.366830+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:48.060879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193972+0000","last_change":"2024-04-17T10:31:37.354088+0000","last_active":"2024-04-17T10:31:40.193972+0000","last_peered":"2024-04-17T10:31:40.193972+0000","last_clean":"2024-04-17T10:31:40.193972+0000","last_became_active":"2024-04-17T10:31:37.353817+0000","last_became_peered":"2024-04-17T10:31:37.353817+0000","last_unstale":"2024-04-17T10:31:40.193972+0000","last_undegraded":"2024-04-17T10:31:40.193972+0000","last_fullsized":"2024-04-17T10:31:40.193972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:20.260258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366210+0000","last_change":"2024-04-17T10:31:38.374513+0000","last_active":"2024-04-17T10:31:39.366210+0000","last_peered":"2024-04-17T10:31:39.366210+0000","last_clean":"2024-04-17T10:31:39.366210+0000","last_became_active":"2024-04-17T10:31:38.374235+0000","last_became_peered":"2024-04-17T10:31:38.374235+0000","last_unstale":"2024-04-17T10:31:39.366210+0000","last_undegraded":"2024-04-17T10:31:39.366210+0000","last_fullsized":"2024-04-17T10:31:39.366210+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:20.114227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340005+0000","last_change":"2024-04-17T10:31:36.340005+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340005+0000","last_undegraded":"2024-04-17T10:31:36.340005+0000","last_fullsized":"2024-04-17T10:31:36.340005+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337484+0000","last_change":"2024-04-17T10:31:36.337484+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337484+0000","last_undegraded":"2024-04-17T10:31:36.337484+0000","last_fullsized":"2024-04-17T10:31:36.337484+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193954+0000","last_change":"2024-04-17T10:31:38.368340+0000","last_active":"2024-04-17T10:31:40.193954+0000","last_peered":"2024-04-17T10:31:40.193954+0000","last_clean":"2024-04-17T10:31:40.193954+0000","last_became_active":"2024-04-17T10:31:38.368065+0000","last_became_peered":"2024-04-17T10:31:38.368065+0000","last_unstale":"2024-04-17T10:31:40.193954+0000","last_undegraded":"2024-04-17T10:31:40.193954+0000","last_fullsized":"2024-04-17T10:31:40.193954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:38:20.873547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340808+0000","last_change":"2024-04-17T10:31:36.340808+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340808+0000","last_undegraded":"2024-04-17T10:31:36.340808+0000","last_fullsized":"2024-04-17T10:31:36.340808+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637594+0000","last_change":"2024-04-17T10:31:37.360694+0000","last_active":"2024-04-17T10:31:39.637594+0000","last_peered":"2024-04-17T10:31:39.637594+0000","last_clean":"2024-04-17T10:31:39.637594+0000","last_became_active":"2024-04-17T10:31:37.360046+0000","last_became_peered":"2024-04-17T10:31:37.360046+0000","last_unstale":"2024-04-17T10:31:39.637594+0000","last_undegraded":"2024-04-17T10:31:39.637594+0000","last_fullsized":"2024-04-17T10:31:39.637594+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:22.070539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-17T10:31:36.342331+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:31:36.342331+0000","last_peered":"2024-04-17T10:31:36.342331+0000","last_clean":"2024-04-17T10:31:36.342331+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:31:36.342331+0000","last_undegraded":"2024-04-17T10:31:36.342331+0000","last_fullsized":"2024-04-17T10:31:36.342331+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":72,"acting":72,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46772,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325068,"statfs":{"total":96636764160,"available":96588869632,"internally_reserved":0,"allocated":1740800,"data_stored":1121930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46192,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325648,"statfs":{"total":96636764160,"available":96589463552,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62836,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94309004,"statfs":{"total":96636764160,"available":96572420096,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54508,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317332,"statfs":{"total":96636764160,"available":96580947968,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54544,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317296,"statfs":{"total":96636764160,"available":96580911104,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50392,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321448,"statfs":{"total":96636764160,"available":96585162752,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46228,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325612,"statfs":{"total":96636764160,"available":96589426688,"internally_reserved":0,"allocated":1187840,"data_stored":552158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":1802240,"data_stored":1144774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:31:42.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:31:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb359400640 1 Processor -- start 2024-04-17T10:31:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb359400640 1 -- start start 2024-04-17T10:31:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb359400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb3540ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb359400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3540d0b00 0x7fb3540b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb359400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3540d2970 con 0x7fb3540cf070 2024-04-17T10:31:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb359400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3540d3580 con 0x7fb3540d0b00 2024-04-17T10:31:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb359400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3540d2c50 con 0x7fb35415dc20 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb352800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb3540ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb353200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3540d0b00 0x7fb3540b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb352800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb3540ceb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38580/0 (socket says 172.21.15.67:38580) 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb352800640 1 -- 172.21.15.67:0/2447242601 learned_addr learned my addr 172.21.15.67:0/2447242601 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb352800640 1 -- 172.21.15.67:0/2447242601 >> v2:172.21.15.86:3300/0 conn(0x7fb3540d0b00 msgr2=0x7fb3540b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb352800640 1 --2- 172.21.15.67:0/2447242601 >> v2:172.21.15.86:3300/0 conn(0x7fb3540d0b00 0x7fb3540b6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb352800640 1 -- 172.21.15.67:0/2447242601 >> v1:172.21.15.86:6789/0 conn(0x7fb3540cf070 legacy=0x7fb3540d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb351e00640 1 -- 172.21.15.67:0/2447242601 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195389278 0 0) 0x7fb3540d2970 con 0x7fb3540cf070 2024-04-17T10:31:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.659+0000 7fb352800640 1 -- 172.21.15.67:0/2447242601 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3540d2f30 con 0x7fb35415dc20 2024-04-17T10:31:42.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb352800640 1 --2- 172.21.15.67:0/2447242601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb3540ceb30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb344009870 tx=0x7fb34401c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7571a1ed18ed756 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb351e00640 1 -- 172.21.15.67:0/2447242601 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3440ab640 con 0x7fb35415dc20 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb351e00640 1 -- 172.21.15.67:0/2447242601 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3440b0070 con 0x7fb35415dc20 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb351e00640 1 -- 172.21.15.67:0/2447242601 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3440aba00 con 0x7fb35415dc20 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 -- 172.21.15.67:0/2447242601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 msgr2=0x7fb3540ceb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 --2- 172.21.15.67:0/2447242601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb3540ceb30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb344009870 tx=0x7fb34401c460 comp rx=0 tx=0).stop 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 -- 172.21.15.67:0/2447242601 shutdown_connections 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 --2- 172.21.15.67:0/2447242601 >> v2:172.21.15.86:3300/0 conn(0x7fb3540d0b00 0x7fb3540b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 --2- 172.21.15.67:0/2447242601 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb3540ceb30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 -- 172.21.15.67:0/2447242601 >> 172.21.15.67:0/2447242601 conn(0x7fb354072a40 msgr2=0x7fb3540b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 -- 172.21.15.67:0/2447242601 shutdown_connections 2024-04-17T10:31:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.663+0000 7fb359400640 1 -- 172.21.15.67:0/2447242601 wait complete. 2024-04-17T10:31:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 Processor -- start 2024-04-17T10:31:42.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 -- start start 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3540cf070 0x7fb3540cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3540d0b00 0x7fb3540cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb354148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3540b7bd0 con 0x7fb3540cf070 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3540b7a50 con 0x7fb3540d0b00 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3540b7d50 con 0x7fb35415dc20 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3540cf070 0x7fb3540cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3540cf070 0x7fb3540cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46382/0 (socket says 172.21.15.67:46382) 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 -- 172.21.15.67:0/3712524408 learned_addr learned my addr 172.21.15.67:0/3712524408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb353200640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb354148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb349e00640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3540d0b00 0x7fb3540cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 -- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 msgr2=0x7fb354148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb354148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 -- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3540d0b00 msgr2=0x7fb3540cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3540d0b00 0x7fb3540cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 -- 172.21.15.67:0/3712524408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3540cbd00 con 0x7fb3540cf070 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb352800640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3540cf070 0x7fb3540cd640 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb344009870 tx=0x7fb344019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb34b400640 1 -- 172.21.15.67:0/3712524408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb344004030 con 0x7fb3540cf070 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb34b400640 1 -- 172.21.15.67:0/3712524408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3440041f0 con 0x7fb3540cf070 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb34b400640 1 -- 172.21.15.67:0/3712524408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb344003620 con 0x7fb3540cf070 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb354150a60 con 0x7fb3540cf070 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.667+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb354149250 con 0x7fb3540cf070 2024-04-17T10:31:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.671+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb354150850 con 0x7fb3540cf070 2024-04-17T10:31:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.671+0000 7fb34b400640 1 -- 172.21.15.67:0/3712524408 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb3440037c0 con 0x7fb3540cf070 2024-04-17T10:31:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.671+0000 7fb34b400640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb330066f60 0x7fb330069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.671+0000 7fb34b400640 1 -- 172.21.15.67:0/3712524408 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb34411fae0 con 0x7fb3540cf070 2024-04-17T10:31:42.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.671+0000 7fb349e00640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb330066f60 0x7fb330069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:42.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.671+0000 7fb349e00640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb330066f60 0x7fb330069420 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb354158710 tx=0x7fb33c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:42.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.675+0000 7fb34b400640 1 -- 172.21.15.67:0/3712524408 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb344016110 con 0x7fb3540cf070 2024-04-17T10:31:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.835+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb354074e30 con 0x7fb330066f60 2024-04-17T10:31:42.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.839+0000 7fb34b400640 1 -- 172.21.15.67:0/3712524408 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246197 (secure 0 0 0) 0x7fb354074e30 con 0x7fb330066f60 2024-04-17T10:31:42.834 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb330066f60 msgr2=0x7fb330069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb330066f60 0x7fb330069420 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fb354158710 tx=0x7fb33c009040 comp rx=0 tx=0).stop 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3540cf070 msgr2=0x7fb3540cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3540cf070 0x7fb3540cd640 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7fb344009870 tx=0x7fb344019200 comp rx=0 tx=0).stop 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 shutdown_connections 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb330066f60 0x7fb330069420 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb35415dc20 0x7fb354148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3540d0b00 0x7fb3540cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 --2- 172.21.15.67:0/3712524408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3540cf070 0x7fb3540cd640 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:42.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 >> 172.21.15.67:0/3712524408 conn(0x7fb354072a40 msgr2=0x7fb35415c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 shutdown_connections 2024-04-17T10:31:42.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:42.843+0000 7fb359400640 1 -- 172.21.15.67:0/3712524408 wait complete. 2024-04-17T10:31:42.854 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-17T10:31:42.234222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591980,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":138,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412460,"kb_used_data":10292,"kb_used_omap":12,"kb_used_meta":402035,"kb_avail":754562260,"statfs":{"total":773094113280,"available":772671754240,"internally_reserved":0,"allocated":10539008,"data_stored":5517724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":411684401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.903728"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366379+0000","last_change":"2024-04-17T10:31:38.372177+0000","last_active":"2024-04-17T10:31:39.366379+0000","last_peered":"2024-04-17T10:31:39.366379+0000","last_clean":"2024-04-17T10:31:39.366379+0000","last_became_active":"2024-04-17T10:31:38.371875+0000","last_became_peered":"2024-04-17T10:31:38.371875+0000","last_unstale":"2024-04-17T10:31:39.366379+0000","last_undegraded":"2024-04-17T10:31:39.366379+0000","last_fullsized":"2024-04-17T10:31:39.366379+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:36:29.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366082+0000","last_change":"2024-04-17T10:31:38.362877+0000","last_active":"2024-04-17T10:31:39.366082+0000","last_peered":"2024-04-17T10:31:39.366082+0000","last_clean":"2024-04-17T10:31:39.366082+0000","last_became_active":"2024-04-17T10:31:38.362413+0000","last_became_peered":"2024-04-17T10:31:38.362413+0000","last_unstale":"2024-04-17T10:31:39.366082+0000","last_undegraded":"2024-04-17T10:31:39.366082+0000","last_fullsized":"2024-04-17T10:31:39.366082+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:03.031089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637416+0000","last_change":"2024-04-17T10:31:38.367300+0000","last_active":"2024-04-17T10:31:39.637416+0000","last_peered":"2024-04-17T10:31:39.637416+0000","last_clean":"2024-04-17T10:31:39.637416+0000","last_became_active":"2024-04-17T10:31:38.366715+0000","last_became_peered":"2024-04-17T10:31:38.366715+0000","last_unstale":"2024-04-17T10:31:39.637416+0000","last_undegraded":"2024-04-17T10:31:39.637416+0000","last_fullsized":"2024-04-17T10:31:39.637416+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:02.550932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193707+0000","last_change":"2024-04-17T10:31:38.382652+0000","last_active":"2024-04-17T10:31:40.193707+0000","last_peered":"2024-04-17T10:31:40.193707+0000","last_clean":"2024-04-17T10:31:40.193707+0000","last_became_active":"2024-04-17T10:31:38.382384+0000","last_became_peered":"2024-04-17T10:31:38.382384+0000","last_unstale":"2024-04-17T10:31:40.193707+0000","last_undegraded":"2024-04-17T10:31:40.193707+0000","last_fullsized":"2024-04-17T10:31:40.193707+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:20.999198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637175+0000","last_change":"2024-04-17T10:31:38.367243+0000","last_active":"2024-04-17T10:31:39.637175+0000","last_peered":"2024-04-17T10:31:39.637175+0000","last_clean":"2024-04-17T10:31:39.637175+0000","last_became_active":"2024-04-17T10:31:38.366952+0000","last_became_peered":"2024-04-17T10:31:38.366952+0000","last_unstale":"2024-04-17T10:31:39.637175+0000","last_undegraded":"2024-04-17T10:31:39.637175+0000","last_fullsized":"2024-04-17T10:31:39.637175+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:24.096194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367386+0000","last_change":"2024-04-17T10:31:38.369001+0000","last_active":"2024-04-17T10:31:39.367386+0000","last_peered":"2024-04-17T10:31:39.367386+0000","last_clean":"2024-04-17T10:31:39.367386+0000","last_became_active":"2024-04-17T10:31:38.367830+0000","last_became_peered":"2024-04-17T10:31:38.367830+0000","last_unstale":"2024-04-17T10:31:39.367386+0000","last_undegraded":"2024-04-17T10:31:39.367386+0000","last_fullsized":"2024-04-17T10:31:39.367386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:52.706957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367226+0000","last_change":"2024-04-17T10:31:38.357758+0000","last_active":"2024-04-17T10:31:39.367226+0000","last_peered":"2024-04-17T10:31:39.367226+0000","last_clean":"2024-04-17T10:31:39.367226+0000","last_became_active":"2024-04-17T10:31:38.357482+0000","last_became_peered":"2024-04-17T10:31:38.357482+0000","last_unstale":"2024-04-17T10:31:39.367226+0000","last_undegraded":"2024-04-17T10:31:39.367226+0000","last_fullsized":"2024-04-17T10:31:39.367226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:48.241446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637619+0000","last_change":"2024-04-17T10:31:38.366467+0000","last_active":"2024-04-17T10:31:39.637619+0000","last_peered":"2024-04-17T10:31:39.637619+0000","last_clean":"2024-04-17T10:31:39.637619+0000","last_became_active":"2024-04-17T10:31:38.366066+0000","last_became_peered":"2024-04-17T10:31:38.366066+0000","last_unstale":"2024-04-17T10:31:39.637619+0000","last_undegraded":"2024-04-17T10:31:39.637619+0000","last_fullsized":"2024-04-17T10:31:39.637619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:08.729085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637251+0000","last_change":"2024-04-17T10:31:38.363077+0000","last_active":"2024-04-17T10:31:39.637251+0000","last_peered":"2024-04-17T10:31:39.637251+0000","last_clean":"2024-04-17T10:31:39.637251+0000","last_became_active":"2024-04-17T10:31:38.362827+0000","last_became_peered":"2024-04-17T10:31:38.362827+0000","last_unstale":"2024-04-17T10:31:39.637251+0000","last_undegraded":"2024-04-17T10:31:39.637251+0000","last_fullsized":"2024-04-17T10:31:39.637251+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:56:20.141939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193544+0000","last_change":"2024-04-17T10:31:38.390689+0000","last_active":"2024-04-17T10:31:40.193544+0000","last_peered":"2024-04-17T10:31:40.193544+0000","last_clean":"2024-04-17T10:31:40.193544+0000","last_became_active":"2024-04-17T10:31:38.389673+0000","last_became_peered":"2024-04-17T10:31:38.389673+0000","last_unstale":"2024-04-17T10:31:40.193544+0000","last_undegraded":"2024-04-17T10:31:40.193544+0000","last_fullsized":"2024-04-17T10:31:40.193544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:05:33.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194163+0000","last_change":"2024-04-17T10:31:38.386404+0000","last_active":"2024-04-17T10:31:40.194163+0000","last_peered":"2024-04-17T10:31:40.194163+0000","last_clean":"2024-04-17T10:31:40.194163+0000","last_became_active":"2024-04-17T10:31:38.383973+0000","last_became_peered":"2024-04-17T10:31:38.383973+0000","last_unstale":"2024-04-17T10:31:40.194163+0000","last_undegraded":"2024-04-17T10:31:40.194163+0000","last_fullsized":"2024-04-17T10:31:40.194163+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:12.821691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366183+0000","last_change":"2024-04-17T10:31:38.362736+0000","last_active":"2024-04-17T10:31:39.366183+0000","last_peered":"2024-04-17T10:31:39.366183+0000","last_clean":"2024-04-17T10:31:39.366183+0000","last_became_active":"2024-04-17T10:31:38.362405+0000","last_became_peered":"2024-04-17T10:31:38.362405+0000","last_unstale":"2024-04-17T10:31:39.366183+0000","last_undegraded":"2024-04-17T10:31:39.366183+0000","last_fullsized":"2024-04-17T10:31:39.366183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:55.770687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.367062+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.366356+0000","last_became_peered":"2024-04-17T10:31:38.366356+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:34.604484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365841+0000","last_change":"2024-04-17T10:31:38.363969+0000","last_active":"2024-04-17T10:31:39.365841+0000","last_peered":"2024-04-17T10:31:39.365841+0000","last_clean":"2024-04-17T10:31:39.365841+0000","last_became_active":"2024-04-17T10:31:38.363731+0000","last_became_peered":"2024-04-17T10:31:38.363731+0000","last_unstale":"2024-04-17T10:31:39.365841+0000","last_undegraded":"2024-04-17T10:31:39.365841+0000","last_fullsized":"2024-04-17T10:31:39.365841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:25.031678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367325+0000","last_change":"2024-04-17T10:31:38.364443+0000","last_active":"2024-04-17T10:31:39.367325+0000","last_peered":"2024-04-17T10:31:39.367325+0000","last_clean":"2024-04-17T10:31:39.367325+0000","last_became_active":"2024-04-17T10:31:38.364134+0000","last_became_peered":"2024-04-17T10:31:38.364134+0000","last_unstale":"2024-04-17T10:31:39.367325+0000","last_undegraded":"2024-04-17T10:31:39.367325+0000","last_fullsized":"2024-04-17T10:31:39.367325+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:13:32.599449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637675+0000","last_change":"2024-04-17T10:31:38.367434+0000","last_active":"2024-04-17T10:31:39.637675+0000","last_peered":"2024-04-17T10:31:39.637675+0000","last_clean":"2024-04-17T10:31:39.637675+0000","last_became_active":"2024-04-17T10:31:38.366004+0000","last_became_peered":"2024-04-17T10:31:38.366004+0000","last_unstale":"2024-04-17T10:31:39.637675+0000","last_undegraded":"2024-04-17T10:31:39.637675+0000","last_fullsized":"2024-04-17T10:31:39.637675+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:20.424487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193672+0000","last_change":"2024-04-17T10:31:38.387676+0000","last_active":"2024-04-17T10:31:40.193672+0000","last_peered":"2024-04-17T10:31:40.193672+0000","last_clean":"2024-04-17T10:31:40.193672+0000","last_became_active":"2024-04-17T10:31:38.380682+0000","last_became_peered":"2024-04-17T10:31:38.380682+0000","last_unstale":"2024-04-17T10:31:40.193672+0000","last_undegraded":"2024-04-17T10:31:40.193672+0000","last_fullsized":"2024-04-17T10:31:40.193672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:59.714995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193655+0000","last_change":"2024-04-17T10:31:38.379789+0000","last_active":"2024-04-17T10:31:40.193655+0000","last_peered":"2024-04-17T10:31:40.193655+0000","last_clean":"2024-04-17T10:31:40.193655+0000","last_became_active":"2024-04-17T10:31:38.379237+0000","last_became_peered":"2024-04-17T10:31:38.379237+0000","last_unstale":"2024-04-17T10:31:40.193655+0000","last_undegraded":"2024-04-17T10:31:40.193655+0000","last_fullsized":"2024-04-17T10:31:40.193655+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:01.930369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.338128+0000","last_change":"2024-04-17T10:31:36.338128+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.338128+0000","last_undegraded":"2024-04-17T10:31:36.338128+0000","last_fullsized":"2024-04-17T10:31:36.338128+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.386544+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.380593+0000","last_became_peered":"2024-04-17T10:31:38.380593+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:53:17.992647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335826+0000","last_change":"2024-04-17T10:31:36.335826+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335826+0000","last_undegraded":"2024-04-17T10:31:36.335826+0000","last_fullsized":"2024-04-17T10:31:36.335826+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340523+0000","last_change":"2024-04-17T10:31:36.340523+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340523+0000","last_undegraded":"2024-04-17T10:31:36.340523+0000","last_fullsized":"2024-04-17T10:31:36.340523+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194225+0000","last_change":"2024-04-17T10:31:38.383915+0000","last_active":"2024-04-17T10:31:40.194225+0000","last_peered":"2024-04-17T10:31:40.194225+0000","last_clean":"2024-04-17T10:31:40.194225+0000","last_became_active":"2024-04-17T10:31:38.382791+0000","last_became_peered":"2024-04-17T10:31:38.382791+0000","last_unstale":"2024-04-17T10:31:40.194225+0000","last_undegraded":"2024-04-17T10:31:40.194225+0000","last_fullsized":"2024-04-17T10:31:40.194225+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:12.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337899+0000","last_change":"2024-04-17T10:31:36.337899+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337899+0000","last_undegraded":"2024-04-17T10:31:36.337899+0000","last_fullsized":"2024-04-17T10:31:36.337899+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637195+0000","last_change":"2024-04-17T10:31:38.366551+0000","last_active":"2024-04-17T10:31:39.637195+0000","last_peered":"2024-04-17T10:31:39.637195+0000","last_clean":"2024-04-17T10:31:39.637195+0000","last_became_active":"2024-04-17T10:31:38.366244+0000","last_became_peered":"2024-04-17T10:31:38.366244+0000","last_unstale":"2024-04-17T10:31:39.637195+0000","last_undegraded":"2024-04-17T10:31:39.637195+0000","last_fullsized":"2024-04-17T10:31:39.637195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:37.769462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.334883+0000","last_change":"2024-04-17T10:31:36.334883+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.334883+0000","last_undegraded":"2024-04-17T10:31:36.334883+0000","last_fullsized":"2024-04-17T10:31:36.334883+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637612+0000","last_change":"2024-04-17T10:31:37.366420+0000","last_active":"2024-04-17T10:31:39.637612+0000","last_peered":"2024-04-17T10:31:39.637612+0000","last_clean":"2024-04-17T10:31:39.637612+0000","last_became_active":"2024-04-17T10:31:37.365712+0000","last_became_peered":"2024-04-17T10:31:37.365712+0000","last_unstale":"2024-04-17T10:31:39.637612+0000","last_undegraded":"2024-04-17T10:31:39.637612+0000","last_fullsized":"2024-04-17T10:31:39.637612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:53.669724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193705+0000","last_change":"2024-04-17T10:31:38.370365+0000","last_active":"2024-04-17T10:31:40.193705+0000","last_peered":"2024-04-17T10:31:40.193705+0000","last_clean":"2024-04-17T10:31:40.193705+0000","last_became_active":"2024-04-17T10:31:38.370089+0000","last_became_peered":"2024-04-17T10:31:38.370089+0000","last_unstale":"2024-04-17T10:31:40.193705+0000","last_undegraded":"2024-04-17T10:31:40.193705+0000","last_fullsized":"2024-04-17T10:31:40.193705+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:56.298204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365982+0000","last_change":"2024-04-17T10:31:37.354469+0000","last_active":"2024-04-17T10:31:39.365982+0000","last_peered":"2024-04-17T10:31:39.365982+0000","last_clean":"2024-04-17T10:31:39.365982+0000","last_became_active":"2024-04-17T10:31:37.354215+0000","last_became_peered":"2024-04-17T10:31:37.354215+0000","last_unstale":"2024-04-17T10:31:39.365982+0000","last_undegraded":"2024-04-17T10:31:39.365982+0000","last_fullsized":"2024-04-17T10:31:39.365982+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:33.318986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.364135+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.363335+0000","last_became_peered":"2024-04-17T10:31:38.363335+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:04.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.341274+0000","last_change":"2024-04-17T10:31:36.341274+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.341274+0000","last_undegraded":"2024-04-17T10:31:36.341274+0000","last_fullsized":"2024-04-17T10:31:36.341274+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340070+0000","last_change":"2024-04-17T10:31:36.340070+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340070+0000","last_undegraded":"2024-04-17T10:31:36.340070+0000","last_fullsized":"2024-04-17T10:31:36.340070+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366370+0000","last_change":"2024-04-17T10:31:37.359814+0000","last_active":"2024-04-17T10:31:39.366370+0000","last_peered":"2024-04-17T10:31:39.366370+0000","last_clean":"2024-04-17T10:31:39.366370+0000","last_became_active":"2024-04-17T10:31:37.359486+0000","last_became_peered":"2024-04-17T10:31:37.359486+0000","last_unstale":"2024-04-17T10:31:39.366370+0000","last_undegraded":"2024-04-17T10:31:39.366370+0000","last_fullsized":"2024-04-17T10:31:39.366370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:23:28.064562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366322+0000","last_change":"2024-04-17T10:31:38.371429+0000","last_active":"2024-04-17T10:31:39.366322+0000","last_peered":"2024-04-17T10:31:39.366322+0000","last_clean":"2024-04-17T10:31:39.366322+0000","last_became_active":"2024-04-17T10:31:38.371073+0000","last_became_peered":"2024-04-17T10:31:38.371073+0000","last_unstale":"2024-04-17T10:31:39.366322+0000","last_undegraded":"2024-04-17T10:31:39.366322+0000","last_fullsized":"2024-04-17T10:31:39.366322+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:01.747376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.336055+0000","last_change":"2024-04-17T10:31:36.336055+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.336055+0000","last_undegraded":"2024-04-17T10:31:36.336055+0000","last_fullsized":"2024-04-17T10:31:36.336055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637829+0000","last_change":"2024-04-17T10:31:38.366147+0000","last_active":"2024-04-17T10:31:39.637829+0000","last_peered":"2024-04-17T10:31:39.637829+0000","last_clean":"2024-04-17T10:31:39.637829+0000","last_became_active":"2024-04-17T10:31:38.365889+0000","last_became_peered":"2024-04-17T10:31:38.365889+0000","last_unstale":"2024-04-17T10:31:39.637829+0000","last_undegraded":"2024-04-17T10:31:39.637829+0000","last_fullsized":"2024-04-17T10:31:39.637829+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:18.111634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"95'6","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193733+0000","last_change":"2024-04-17T10:31:37.367318+0000","last_active":"2024-04-17T10:31:40.193733+0000","last_peered":"2024-04-17T10:31:40.193733+0000","last_clean":"2024-04-17T10:31:40.193733+0000","last_became_active":"2024-04-17T10:31:37.367023+0000","last_became_peered":"2024-04-17T10:31:37.367023+0000","last_unstale":"2024-04-17T10:31:40.193733+0000","last_undegraded":"2024-04-17T10:31:40.193733+0000","last_fullsized":"2024-04-17T10:31:40.193733+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:02.024574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637403+0000","last_change":"2024-04-17T10:31:38.362727+0000","last_active":"2024-04-17T10:31:39.637403+0000","last_peered":"2024-04-17T10:31:39.637403+0000","last_clean":"2024-04-17T10:31:39.637403+0000","last_became_active":"2024-04-17T10:31:38.362083+0000","last_became_peered":"2024-04-17T10:31:38.362083+0000","last_unstale":"2024-04-17T10:31:39.637403+0000","last_undegraded":"2024-04-17T10:31:39.637403+0000","last_fullsized":"2024-04-17T10:31:39.637403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:42.796034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340608+0000","last_change":"2024-04-17T10:31:36.340608+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340608+0000","last_undegraded":"2024-04-17T10:31:36.340608+0000","last_fullsized":"2024-04-17T10:31:36.340608+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193763+0000","last_change":"2024-04-17T10:31:37.354839+0000","last_active":"2024-04-17T10:31:40.193763+0000","last_peered":"2024-04-17T10:31:40.193763+0000","last_clean":"2024-04-17T10:31:40.193763+0000","last_became_active":"2024-04-17T10:31:37.354538+0000","last_became_peered":"2024-04-17T10:31:37.354538+0000","last_unstale":"2024-04-17T10:31:40.193763+0000","last_undegraded":"2024-04-17T10:31:40.193763+0000","last_fullsized":"2024-04-17T10:31:40.193763+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:10.283370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366942+0000","last_change":"2024-04-17T10:31:38.368131+0000","last_active":"2024-04-17T10:31:39.366942+0000","last_peered":"2024-04-17T10:31:39.366942+0000","last_clean":"2024-04-17T10:31:39.366942+0000","last_became_active":"2024-04-17T10:31:38.367815+0000","last_became_peered":"2024-04-17T10:31:38.367815+0000","last_unstale":"2024-04-17T10:31:39.366942+0000","last_undegraded":"2024-04-17T10:31:39.366942+0000","last_fullsized":"2024-04-17T10:31:39.366942+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:39.116896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637313+0000","last_change":"2024-04-17T10:31:37.358749+0000","last_active":"2024-04-17T10:31:39.637313+0000","last_peered":"2024-04-17T10:31:39.637313+0000","last_clean":"2024-04-17T10:31:39.637313+0000","last_became_active":"2024-04-17T10:31:37.357922+0000","last_became_peered":"2024-04-17T10:31:37.357922+0000","last_unstale":"2024-04-17T10:31:39.637313+0000","last_undegraded":"2024-04-17T10:31:39.637313+0000","last_fullsized":"2024-04-17T10:31:39.637313+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:23.174223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367510+0000","last_change":"2024-04-17T10:31:38.356263+0000","last_active":"2024-04-17T10:31:39.367510+0000","last_peered":"2024-04-17T10:31:39.367510+0000","last_clean":"2024-04-17T10:31:39.367510+0000","last_became_active":"2024-04-17T10:31:38.355993+0000","last_became_peered":"2024-04-17T10:31:38.355993+0000","last_unstale":"2024-04-17T10:31:39.367510+0000","last_undegraded":"2024-04-17T10:31:39.367510+0000","last_fullsized":"2024-04-17T10:31:39.367510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:55.031187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"95'8","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637287+0000","last_change":"2024-04-17T10:31:37.359691+0000","last_active":"2024-04-17T10:31:39.637287+0000","last_peered":"2024-04-17T10:31:39.637287+0000","last_clean":"2024-04-17T10:31:39.637287+0000","last_became_active":"2024-04-17T10:31:37.359365+0000","last_became_peered":"2024-04-17T10:31:37.359365+0000","last_unstale":"2024-04-17T10:31:39.637287+0000","last_undegraded":"2024-04-17T10:31:39.637287+0000","last_fullsized":"2024-04-17T10:31:39.637287+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:00:19.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.379892+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.379586+0000","last_became_peered":"2024-04-17T10:31:38.379586+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:35.337686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337309+0000","last_change":"2024-04-17T10:31:36.337309+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337309+0000","last_undegraded":"2024-04-17T10:31:36.337309+0000","last_fullsized":"2024-04-17T10:31:36.337309+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340462+0000","last_change":"2024-04-17T10:31:36.340462+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340462+0000","last_undegraded":"2024-04-17T10:31:36.340462+0000","last_fullsized":"2024-04-17T10:31:36.340462+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335517+0000","last_change":"2024-04-17T10:31:36.335517+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335517+0000","last_undegraded":"2024-04-17T10:31:36.335517+0000","last_fullsized":"2024-04-17T10:31:36.335517+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194178+0000","last_change":"2024-04-17T10:31:38.383600+0000","last_active":"2024-04-17T10:31:40.194178+0000","last_peered":"2024-04-17T10:31:40.194178+0000","last_clean":"2024-04-17T10:31:40.194178+0000","last_became_active":"2024-04-17T10:31:38.383354+0000","last_became_peered":"2024-04-17T10:31:38.383354+0000","last_unstale":"2024-04-17T10:31:40.194178+0000","last_undegraded":"2024-04-17T10:31:40.194178+0000","last_fullsized":"2024-04-17T10:31:40.194178+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:25:53.140820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340140+0000","last_change":"2024-04-17T10:31:36.340140+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340140+0000","last_undegraded":"2024-04-17T10:31:36.340140+0000","last_fullsized":"2024-04-17T10:31:36.340140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366890+0000","last_change":"2024-04-17T10:31:37.365058+0000","last_active":"2024-04-17T10:31:39.366890+0000","last_peered":"2024-04-17T10:31:39.366890+0000","last_clean":"2024-04-17T10:31:39.366890+0000","last_became_active":"2024-04-17T10:31:37.364541+0000","last_became_peered":"2024-04-17T10:31:37.364541+0000","last_unstale":"2024-04-17T10:31:39.366890+0000","last_undegraded":"2024-04-17T10:31:39.366890+0000","last_fullsized":"2024-04-17T10:31:39.366890+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:38.273762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340172+0000","last_change":"2024-04-17T10:31:36.340172+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340172+0000","last_undegraded":"2024-04-17T10:31:36.340172+0000","last_fullsized":"2024-04-17T10:31:36.340172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193864+0000","last_change":"2024-04-17T10:31:38.376121+0000","last_active":"2024-04-17T10:31:40.193864+0000","last_peered":"2024-04-17T10:31:40.193864+0000","last_clean":"2024-04-17T10:31:40.193864+0000","last_became_active":"2024-04-17T10:31:38.375833+0000","last_became_peered":"2024-04-17T10:31:38.375833+0000","last_unstale":"2024-04-17T10:31:40.193864+0000","last_undegraded":"2024-04-17T10:31:40.193864+0000","last_fullsized":"2024-04-17T10:31:40.193864+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:35:43.598662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366769+0000","last_change":"2024-04-17T10:31:37.362595+0000","last_active":"2024-04-17T10:31:39.366769+0000","last_peered":"2024-04-17T10:31:39.366769+0000","last_clean":"2024-04-17T10:31:39.366769+0000","last_became_active":"2024-04-17T10:31:37.362307+0000","last_became_peered":"2024-04-17T10:31:37.362307+0000","last_unstale":"2024-04-17T10:31:39.366769+0000","last_undegraded":"2024-04-17T10:31:39.366769+0000","last_fullsized":"2024-04-17T10:31:39.366769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:15:47.101230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340408+0000","last_change":"2024-04-17T10:31:36.340408+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340408+0000","last_undegraded":"2024-04-17T10:31:36.340408+0000","last_fullsized":"2024-04-17T10:31:36.340408+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194103+0000","last_change":"2024-04-17T10:31:38.386818+0000","last_active":"2024-04-17T10:31:40.194103+0000","last_peered":"2024-04-17T10:31:40.194103+0000","last_clean":"2024-04-17T10:31:40.194103+0000","last_became_active":"2024-04-17T10:31:38.383939+0000","last_became_peered":"2024-04-17T10:31:38.383939+0000","last_unstale":"2024-04-17T10:31:40.194103+0000","last_undegraded":"2024-04-17T10:31:40.194103+0000","last_fullsized":"2024-04-17T10:31:40.194103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:01.834781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335266+0000","last_change":"2024-04-17T10:31:36.335266+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335266+0000","last_undegraded":"2024-04-17T10:31:36.335266+0000","last_fullsized":"2024-04-17T10:31:36.335266+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194023+0000","last_change":"2024-04-17T10:31:38.376661+0000","last_active":"2024-04-17T10:31:40.194023+0000","last_peered":"2024-04-17T10:31:40.194023+0000","last_clean":"2024-04-17T10:31:40.194023+0000","last_became_active":"2024-04-17T10:31:38.376364+0000","last_became_peered":"2024-04-17T10:31:38.376364+0000","last_unstale":"2024-04-17T10:31:40.194023+0000","last_undegraded":"2024-04-17T10:31:40.194023+0000","last_fullsized":"2024-04-17T10:31:40.194023+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:28.335090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335050+0000","last_change":"2024-04-17T10:31:36.335050+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335050+0000","last_undegraded":"2024-04-17T10:31:36.335050+0000","last_fullsized":"2024-04-17T10:31:36.335050+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337226+0000","last_change":"2024-04-17T10:31:36.337226+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337226+0000","last_undegraded":"2024-04-17T10:31:36.337226+0000","last_fullsized":"2024-04-17T10:31:36.337226+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366830+0000","last_change":"2024-04-17T10:31:38.384626+0000","last_active":"2024-04-17T10:31:39.366830+0000","last_peered":"2024-04-17T10:31:39.366830+0000","last_clean":"2024-04-17T10:31:39.366830+0000","last_became_active":"2024-04-17T10:31:38.384373+0000","last_became_peered":"2024-04-17T10:31:38.384373+0000","last_unstale":"2024-04-17T10:31:39.366830+0000","last_undegraded":"2024-04-17T10:31:39.366830+0000","last_fullsized":"2024-04-17T10:31:39.366830+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:48.060879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193972+0000","last_change":"2024-04-17T10:31:37.354088+0000","last_active":"2024-04-17T10:31:40.193972+0000","last_peered":"2024-04-17T10:31:40.193972+0000","last_clean":"2024-04-17T10:31:40.193972+0000","last_became_active":"2024-04-17T10:31:37.353817+0000","last_became_peered":"2024-04-17T10:31:37.353817+0000","last_unstale":"2024-04-17T10:31:40.193972+0000","last_undegraded":"2024-04-17T10:31:40.193972+0000","last_fullsized":"2024-04-17T10:31:40.193972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:20.260258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366210+0000","last_change":"2024-04-17T10:31:38.374513+0000","last_active":"2024-04-17T10:31:39.366210+0000","last_peered":"2024-04-17T10:31:39.366210+0000","last_clean":"2024-04-17T10:31:39.366210+0000","last_became_active":"2024-04-17T10:31:38.374235+0000","last_became_peered":"2024-04-17T10:31:38.374235+0000","last_unstale":"2024-04-17T10:31:39.366210+0000","last_undegraded":"2024-04-17T10:31:39.366210+0000","last_fullsized":"2024-04-17T10:31:39.366210+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:20.114227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340005+0000","last_change":"2024-04-17T10:31:36.340005+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340005+0000","last_undegraded":"2024-04-17T10:31:36.340005+0000","last_fullsized":"2024-04-17T10:31:36.340005+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337484+0000","last_change":"2024-04-17T10:31:36.337484+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337484+0000","last_undegraded":"2024-04-17T10:31:36.337484+0000","last_fullsized":"2024-04-17T10:31:36.337484+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193954+0000","last_change":"2024-04-17T10:31:38.368340+0000","last_active":"2024-04-17T10:31:40.193954+0000","last_peered":"2024-04-17T10:31:40.193954+0000","last_clean":"2024-04-17T10:31:40.193954+0000","last_became_active":"2024-04-17T10:31:38.368065+0000","last_became_peered":"2024-04-17T10:31:38.368065+0000","last_unstale":"2024-04-17T10:31:40.193954+0000","last_undegraded":"2024-04-17T10:31:40.193954+0000","last_fullsized":"2024-04-17T10:31:40.193954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:38:20.873547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340808+0000","last_change":"2024-04-17T10:31:36.340808+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340808+0000","last_undegraded":"2024-04-17T10:31:36.340808+0000","last_fullsized":"2024-04-17T10:31:36.340808+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637594+0000","last_change":"2024-04-17T10:31:37.360694+0000","last_active":"2024-04-17T10:31:39.637594+0000","last_peered":"2024-04-17T10:31:39.637594+0000","last_clean":"2024-04-17T10:31:39.637594+0000","last_became_active":"2024-04-17T10:31:37.360046+0000","last_became_peered":"2024-04-17T10:31:37.360046+0000","last_unstale":"2024-04-17T10:31:39.637594+0000","last_undegraded":"2024-04-17T10:31:39.637594+0000","last_fullsized":"2024-04-17T10:31:39.637594+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:22.070539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-17T10:31:36.342331+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:31:36.342331+0000","last_peered":"2024-04-17T10:31:36.342331+0000","last_clean":"2024-04-17T10:31:36.342331+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:31:36.342331+0000","last_undegraded":"2024-04-17T10:31:36.342331+0000","last_fullsized":"2024-04-17T10:31:36.342331+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":72,"acting":72,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46772,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325068,"statfs":{"total":96636764160,"available":96588869632,"internally_reserved":0,"allocated":1740800,"data_stored":1121930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46192,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325648,"statfs":{"total":96636764160,"available":96589463552,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62836,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94309004,"statfs":{"total":96636764160,"available":96572420096,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54508,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317332,"statfs":{"total":96636764160,"available":96580947968,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54544,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317296,"statfs":{"total":96636764160,"available":96580911104,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50392,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321448,"statfs":{"total":96636764160,"available":96585162752,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46228,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325612,"statfs":{"total":96636764160,"available":96589426688,"internally_reserved":0,"allocated":1187840,"data_stored":552158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":1802240,"data_stored":1144774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:31:42.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:31:43.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.043+0000 7fadede00640 1 Processor -- start 2024-04-17T10:31:43.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fadede00640 1 -- start start 2024-04-17T10:31:43.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fadede00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade80ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fadede00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fade80cc0d0 0x7fade806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fadede00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80cc530 con 0x7fade815e440 2024-04-17T10:31:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fadede00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade80cbc40 con 0x7fade80cc0d0 2024-04-17T10:31:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fadede00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade80ce930 con 0x7fade80b2dc0 2024-04-17T10:31:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fade6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fade815e440 0x7fade815e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34474/0 (socket says 172.21.15.67:34474) 2024-04-17T10:31:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fade6a00640 1 -- 172.21.15.67:0/3717305598 learned_addr learned my addr 172.21.15.67:0/3717305598 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7faddfe00640 1 --2- 172.21.15.67:0/3717305598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade80ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fade7400640 1 --2- 172.21.15.67:0/3717305598 >> v2:172.21.15.86:3300/0 conn(0x7fade80cc0d0 0x7fade806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fade6000640 1 -- 172.21.15.67:0/3717305598 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841060272 0 0) 0x7fade80cc530 con 0x7fade815e440 2024-04-17T10:31:43.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fade6000640 1 -- 172.21.15.67:0/3717305598 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadcc003530 con 0x7fade815e440 2024-04-17T10:31:43.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7faddfe00640 1 -- 172.21.15.67:0/3717305598 >> v2:172.21.15.86:3300/0 conn(0x7fade80cc0d0 msgr2=0x7fade806e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:43.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7faddfe00640 1 --2- 172.21.15.67:0/3717305598 >> v2:172.21.15.86:3300/0 conn(0x7fade80cc0d0 0x7fade806e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7faddfe00640 1 -- 172.21.15.67:0/3717305598 >> v1:172.21.15.67:6789/0 conn(0x7fade815e440 legacy=0x7fade815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7faddfe00640 1 -- 172.21.15.67:0/3717305598 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade80cb310 con 0x7fade80b2dc0 2024-04-17T10:31:43.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.047+0000 7fade7400640 1 --2- 172.21.15.67:0/3717305598 >> v2:172.21.15.86:3300/0 conn(0x7fade80cc0d0 0x7fade806e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:31:43.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7faddfe00640 1 --2- 172.21.15.67:0/3717305598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade80ce2a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fadd00097c0 tx=0x7fadd001c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b232bb7b78add36 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fade6000640 1 -- 172.21.15.67:0/3717305598 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00a7070 con 0x7fade80b2dc0 2024-04-17T10:31:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fade6000640 1 -- 172.21.15.67:0/3717305598 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd00b0070 con 0x7fade80b2dc0 2024-04-17T10:31:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fade6000640 1 -- 172.21.15.67:0/3717305598 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00ab870 con 0x7fade80b2dc0 2024-04-17T10:31:43.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 -- 172.21.15.67:0/3717305598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 msgr2=0x7fade80ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 --2- 172.21.15.67:0/3717305598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade80ce2a0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fadd00097c0 tx=0x7fadd001c470 comp rx=0 tx=0).stop 2024-04-17T10:31:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 -- 172.21.15.67:0/3717305598 shutdown_connections 2024-04-17T10:31:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 --2- 172.21.15.67:0/3717305598 >> v2:172.21.15.86:3300/0 conn(0x7fade80cc0d0 0x7fade806e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 --2- 172.21.15.67:0/3717305598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade80ce2a0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 -- 172.21.15.67:0/3717305598 >> 172.21.15.67:0/3717305598 conn(0x7fade80b22a0 msgr2=0x7fade80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 -- 172.21.15.67:0/3717305598 shutdown_connections 2024-04-17T10:31:43.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 -- 172.21.15.67:0/3717305598 wait complete. 2024-04-17T10:31:43.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 Processor -- start 2024-04-17T10:31:43.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.051+0000 7fadede00640 1 -- start start 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fade80cc0d0 0x7fade815cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fade815e440 0x7fade80ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade806ac90 con 0x7fade80cc0d0 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade815ad80 con 0x7fade815e440 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade806ae10 con 0x7fade80b2dc0 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fade80cc0d0 0x7fade815cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fade80cc0d0 0x7fade815cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46386/0 (socket says 172.21.15.67:46386) 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 -- 172.21.15.67:0/3145956099 learned_addr learned my addr 172.21.15.67:0/3145956099 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fade7400640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fade815e440 0x7fade80ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fade6a00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 -- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 msgr2=0x7fade814d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade814d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 -- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fade815e440 msgr2=0x7fade80ad9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fade815e440 0x7fade80ad9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 -- 172.21.15.67:0/3145956099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade814d620 con 0x7fade80cc0d0 2024-04-17T10:31:43.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddfe00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fade80cc0d0 0x7fade815cd30 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fadd00ad040 tx=0x7fadd0019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddf400640 1 -- 172.21.15.67:0/3145956099 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0003820 con 0x7fade80cc0d0 2024-04-17T10:31:43.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddf400640 1 -- 172.21.15.67:0/3145956099 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd00039c0 con 0x7fade80cc0d0 2024-04-17T10:31:43.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7faddf400640 1 -- 172.21.15.67:0/3145956099 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0003d90 con 0x7fade80cc0d0 2024-04-17T10:31:43.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade815d270 con 0x7fade80cc0d0 2024-04-17T10:31:43.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade815c250 con 0x7fade80cc0d0 2024-04-17T10:31:43.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.055+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade809e900 con 0x7fade80cc0d0 2024-04-17T10:31:43.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.059+0000 7faddf400640 1 -- 172.21.15.67:0/3145956099 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fadd00c3060 con 0x7fade80cc0d0 2024-04-17T10:31:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.059+0000 7faddf400640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadc4066f60 0x7fadc4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.059+0000 7faddf400640 1 -- 172.21.15.67:0/3145956099 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fadd011ed80 con 0x7fade80cc0d0 2024-04-17T10:31:43.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.059+0000 7fade6a00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadc4066f60 0x7fadc4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.059+0000 7fade6a00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadc4066f60 0x7fadc4069420 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fadd8005890 tx=0x7fadd8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.063+0000 7faddf400640 1 -- 172.21.15.67:0/3145956099 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fadd0016090 con 0x7fade80cc0d0 2024-04-17T10:31:43.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.219+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fade8154880 con 0x7fadc4066f60 2024-04-17T10:31:43.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.227+0000 7faddf400640 1 -- 172.21.15.67:0/3145956099 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246197 (secure 0 0 0) 0x7fade8154880 con 0x7fadc4066f60 2024-04-17T10:31:43.220 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:43.221 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:31:43.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadc4066f60 msgr2=0x7fadc4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadc4066f60 0x7fadc4069420 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fadd8005890 tx=0x7fadd8009040 comp rx=0 tx=0).stop 2024-04-17T10:31:43.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fade80cc0d0 msgr2=0x7fade815cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fade80cc0d0 0x7fade815cd30 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fadd00ad040 tx=0x7fadd0019210 comp rx=0 tx=0).stop 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 shutdown_connections 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadc4066f60 0x7fadc4069420 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fade815e440 0x7fade80ad9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fade80cc0d0 0x7fade815cd30 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 --2- 172.21.15.67:0/3145956099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fade80b2dc0 0x7fade814d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 >> 172.21.15.67:0/3145956099 conn(0x7fade80b22a0 msgr2=0x7fade80afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 shutdown_connections 2024-04-17T10:31:43.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.231+0000 7fadede00640 1 -- 172.21.15.67:0/3145956099 wait complete. 2024-04-17T10:31:43.240 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-17T10:31:42.234222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591980,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":138,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412460,"kb_used_data":10292,"kb_used_omap":12,"kb_used_meta":402035,"kb_avail":754562260,"statfs":{"total":773094113280,"available":772671754240,"internally_reserved":0,"allocated":10539008,"data_stored":5517724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":411684401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.903728"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366379+0000","last_change":"2024-04-17T10:31:38.372177+0000","last_active":"2024-04-17T10:31:39.366379+0000","last_peered":"2024-04-17T10:31:39.366379+0000","last_clean":"2024-04-17T10:31:39.366379+0000","last_became_active":"2024-04-17T10:31:38.371875+0000","last_became_peered":"2024-04-17T10:31:38.371875+0000","last_unstale":"2024-04-17T10:31:39.366379+0000","last_undegraded":"2024-04-17T10:31:39.366379+0000","last_fullsized":"2024-04-17T10:31:39.366379+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:36:29.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366082+0000","last_change":"2024-04-17T10:31:38.362877+0000","last_active":"2024-04-17T10:31:39.366082+0000","last_peered":"2024-04-17T10:31:39.366082+0000","last_clean":"2024-04-17T10:31:39.366082+0000","last_became_active":"2024-04-17T10:31:38.362413+0000","last_became_peered":"2024-04-17T10:31:38.362413+0000","last_unstale":"2024-04-17T10:31:39.366082+0000","last_undegraded":"2024-04-17T10:31:39.366082+0000","last_fullsized":"2024-04-17T10:31:39.366082+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:03.031089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637416+0000","last_change":"2024-04-17T10:31:38.367300+0000","last_active":"2024-04-17T10:31:39.637416+0000","last_peered":"2024-04-17T10:31:39.637416+0000","last_clean":"2024-04-17T10:31:39.637416+0000","last_became_active":"2024-04-17T10:31:38.366715+0000","last_became_peered":"2024-04-17T10:31:38.366715+0000","last_unstale":"2024-04-17T10:31:39.637416+0000","last_undegraded":"2024-04-17T10:31:39.637416+0000","last_fullsized":"2024-04-17T10:31:39.637416+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:02.550932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193707+0000","last_change":"2024-04-17T10:31:38.382652+0000","last_active":"2024-04-17T10:31:40.193707+0000","last_peered":"2024-04-17T10:31:40.193707+0000","last_clean":"2024-04-17T10:31:40.193707+0000","last_became_active":"2024-04-17T10:31:38.382384+0000","last_became_peered":"2024-04-17T10:31:38.382384+0000","last_unstale":"2024-04-17T10:31:40.193707+0000","last_undegraded":"2024-04-17T10:31:40.193707+0000","last_fullsized":"2024-04-17T10:31:40.193707+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:20.999198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637175+0000","last_change":"2024-04-17T10:31:38.367243+0000","last_active":"2024-04-17T10:31:39.637175+0000","last_peered":"2024-04-17T10:31:39.637175+0000","last_clean":"2024-04-17T10:31:39.637175+0000","last_became_active":"2024-04-17T10:31:38.366952+0000","last_became_peered":"2024-04-17T10:31:38.366952+0000","last_unstale":"2024-04-17T10:31:39.637175+0000","last_undegraded":"2024-04-17T10:31:39.637175+0000","last_fullsized":"2024-04-17T10:31:39.637175+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:24.096194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367386+0000","last_change":"2024-04-17T10:31:38.369001+0000","last_active":"2024-04-17T10:31:39.367386+0000","last_peered":"2024-04-17T10:31:39.367386+0000","last_clean":"2024-04-17T10:31:39.367386+0000","last_became_active":"2024-04-17T10:31:38.367830+0000","last_became_peered":"2024-04-17T10:31:38.367830+0000","last_unstale":"2024-04-17T10:31:39.367386+0000","last_undegraded":"2024-04-17T10:31:39.367386+0000","last_fullsized":"2024-04-17T10:31:39.367386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:52.706957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367226+0000","last_change":"2024-04-17T10:31:38.357758+0000","last_active":"2024-04-17T10:31:39.367226+0000","last_peered":"2024-04-17T10:31:39.367226+0000","last_clean":"2024-04-17T10:31:39.367226+0000","last_became_active":"2024-04-17T10:31:38.357482+0000","last_became_peered":"2024-04-17T10:31:38.357482+0000","last_unstale":"2024-04-17T10:31:39.367226+0000","last_undegraded":"2024-04-17T10:31:39.367226+0000","last_fullsized":"2024-04-17T10:31:39.367226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:48.241446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637619+0000","last_change":"2024-04-17T10:31:38.366467+0000","last_active":"2024-04-17T10:31:39.637619+0000","last_peered":"2024-04-17T10:31:39.637619+0000","last_clean":"2024-04-17T10:31:39.637619+0000","last_became_active":"2024-04-17T10:31:38.366066+0000","last_became_peered":"2024-04-17T10:31:38.366066+0000","last_unstale":"2024-04-17T10:31:39.637619+0000","last_undegraded":"2024-04-17T10:31:39.637619+0000","last_fullsized":"2024-04-17T10:31:39.637619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:08.729085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637251+0000","last_change":"2024-04-17T10:31:38.363077+0000","last_active":"2024-04-17T10:31:39.637251+0000","last_peered":"2024-04-17T10:31:39.637251+0000","last_clean":"2024-04-17T10:31:39.637251+0000","last_became_active":"2024-04-17T10:31:38.362827+0000","last_became_peered":"2024-04-17T10:31:38.362827+0000","last_unstale":"2024-04-17T10:31:39.637251+0000","last_undegraded":"2024-04-17T10:31:39.637251+0000","last_fullsized":"2024-04-17T10:31:39.637251+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:56:20.141939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193544+0000","last_change":"2024-04-17T10:31:38.390689+0000","last_active":"2024-04-17T10:31:40.193544+0000","last_peered":"2024-04-17T10:31:40.193544+0000","last_clean":"2024-04-17T10:31:40.193544+0000","last_became_active":"2024-04-17T10:31:38.389673+0000","last_became_peered":"2024-04-17T10:31:38.389673+0000","last_unstale":"2024-04-17T10:31:40.193544+0000","last_undegraded":"2024-04-17T10:31:40.193544+0000","last_fullsized":"2024-04-17T10:31:40.193544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:05:33.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194163+0000","last_change":"2024-04-17T10:31:38.386404+0000","last_active":"2024-04-17T10:31:40.194163+0000","last_peered":"2024-04-17T10:31:40.194163+0000","last_clean":"2024-04-17T10:31:40.194163+0000","last_became_active":"2024-04-17T10:31:38.383973+0000","last_became_peered":"2024-04-17T10:31:38.383973+0000","last_unstale":"2024-04-17T10:31:40.194163+0000","last_undegraded":"2024-04-17T10:31:40.194163+0000","last_fullsized":"2024-04-17T10:31:40.194163+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:12.821691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366183+0000","last_change":"2024-04-17T10:31:38.362736+0000","last_active":"2024-04-17T10:31:39.366183+0000","last_peered":"2024-04-17T10:31:39.366183+0000","last_clean":"2024-04-17T10:31:39.366183+0000","last_became_active":"2024-04-17T10:31:38.362405+0000","last_became_peered":"2024-04-17T10:31:38.362405+0000","last_unstale":"2024-04-17T10:31:39.366183+0000","last_undegraded":"2024-04-17T10:31:39.366183+0000","last_fullsized":"2024-04-17T10:31:39.366183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:55.770687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.367062+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.366356+0000","last_became_peered":"2024-04-17T10:31:38.366356+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:34.604484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365841+0000","last_change":"2024-04-17T10:31:38.363969+0000","last_active":"2024-04-17T10:31:39.365841+0000","last_peered":"2024-04-17T10:31:39.365841+0000","last_clean":"2024-04-17T10:31:39.365841+0000","last_became_active":"2024-04-17T10:31:38.363731+0000","last_became_peered":"2024-04-17T10:31:38.363731+0000","last_unstale":"2024-04-17T10:31:39.365841+0000","last_undegraded":"2024-04-17T10:31:39.365841+0000","last_fullsized":"2024-04-17T10:31:39.365841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:25.031678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367325+0000","last_change":"2024-04-17T10:31:38.364443+0000","last_active":"2024-04-17T10:31:39.367325+0000","last_peered":"2024-04-17T10:31:39.367325+0000","last_clean":"2024-04-17T10:31:39.367325+0000","last_became_active":"2024-04-17T10:31:38.364134+0000","last_became_peered":"2024-04-17T10:31:38.364134+0000","last_unstale":"2024-04-17T10:31:39.367325+0000","last_undegraded":"2024-04-17T10:31:39.367325+0000","last_fullsized":"2024-04-17T10:31:39.367325+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:13:32.599449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637675+0000","last_change":"2024-04-17T10:31:38.367434+0000","last_active":"2024-04-17T10:31:39.637675+0000","last_peered":"2024-04-17T10:31:39.637675+0000","last_clean":"2024-04-17T10:31:39.637675+0000","last_became_active":"2024-04-17T10:31:38.366004+0000","last_became_peered":"2024-04-17T10:31:38.366004+0000","last_unstale":"2024-04-17T10:31:39.637675+0000","last_undegraded":"2024-04-17T10:31:39.637675+0000","last_fullsized":"2024-04-17T10:31:39.637675+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:20.424487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193672+0000","last_change":"2024-04-17T10:31:38.387676+0000","last_active":"2024-04-17T10:31:40.193672+0000","last_peered":"2024-04-17T10:31:40.193672+0000","last_clean":"2024-04-17T10:31:40.193672+0000","last_became_active":"2024-04-17T10:31:38.380682+0000","last_became_peered":"2024-04-17T10:31:38.380682+0000","last_unstale":"2024-04-17T10:31:40.193672+0000","last_undegraded":"2024-04-17T10:31:40.193672+0000","last_fullsized":"2024-04-17T10:31:40.193672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:59.714995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193655+0000","last_change":"2024-04-17T10:31:38.379789+0000","last_active":"2024-04-17T10:31:40.193655+0000","last_peered":"2024-04-17T10:31:40.193655+0000","last_clean":"2024-04-17T10:31:40.193655+0000","last_became_active":"2024-04-17T10:31:38.379237+0000","last_became_peered":"2024-04-17T10:31:38.379237+0000","last_unstale":"2024-04-17T10:31:40.193655+0000","last_undegraded":"2024-04-17T10:31:40.193655+0000","last_fullsized":"2024-04-17T10:31:40.193655+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:01.930369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.338128+0000","last_change":"2024-04-17T10:31:36.338128+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.338128+0000","last_undegraded":"2024-04-17T10:31:36.338128+0000","last_fullsized":"2024-04-17T10:31:36.338128+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.386544+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.380593+0000","last_became_peered":"2024-04-17T10:31:38.380593+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:53:17.992647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335826+0000","last_change":"2024-04-17T10:31:36.335826+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335826+0000","last_undegraded":"2024-04-17T10:31:36.335826+0000","last_fullsized":"2024-04-17T10:31:36.335826+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340523+0000","last_change":"2024-04-17T10:31:36.340523+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340523+0000","last_undegraded":"2024-04-17T10:31:36.340523+0000","last_fullsized":"2024-04-17T10:31:36.340523+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194225+0000","last_change":"2024-04-17T10:31:38.383915+0000","last_active":"2024-04-17T10:31:40.194225+0000","last_peered":"2024-04-17T10:31:40.194225+0000","last_clean":"2024-04-17T10:31:40.194225+0000","last_became_active":"2024-04-17T10:31:38.382791+0000","last_became_peered":"2024-04-17T10:31:38.382791+0000","last_unstale":"2024-04-17T10:31:40.194225+0000","last_undegraded":"2024-04-17T10:31:40.194225+0000","last_fullsized":"2024-04-17T10:31:40.194225+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:12.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337899+0000","last_change":"2024-04-17T10:31:36.337899+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337899+0000","last_undegraded":"2024-04-17T10:31:36.337899+0000","last_fullsized":"2024-04-17T10:31:36.337899+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637195+0000","last_change":"2024-04-17T10:31:38.366551+0000","last_active":"2024-04-17T10:31:39.637195+0000","last_peered":"2024-04-17T10:31:39.637195+0000","last_clean":"2024-04-17T10:31:39.637195+0000","last_became_active":"2024-04-17T10:31:38.366244+0000","last_became_peered":"2024-04-17T10:31:38.366244+0000","last_unstale":"2024-04-17T10:31:39.637195+0000","last_undegraded":"2024-04-17T10:31:39.637195+0000","last_fullsized":"2024-04-17T10:31:39.637195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:37.769462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.334883+0000","last_change":"2024-04-17T10:31:36.334883+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.334883+0000","last_undegraded":"2024-04-17T10:31:36.334883+0000","last_fullsized":"2024-04-17T10:31:36.334883+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637612+0000","last_change":"2024-04-17T10:31:37.366420+0000","last_active":"2024-04-17T10:31:39.637612+0000","last_peered":"2024-04-17T10:31:39.637612+0000","last_clean":"2024-04-17T10:31:39.637612+0000","last_became_active":"2024-04-17T10:31:37.365712+0000","last_became_peered":"2024-04-17T10:31:37.365712+0000","last_unstale":"2024-04-17T10:31:39.637612+0000","last_undegraded":"2024-04-17T10:31:39.637612+0000","last_fullsized":"2024-04-17T10:31:39.637612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:53.669724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193705+0000","last_change":"2024-04-17T10:31:38.370365+0000","last_active":"2024-04-17T10:31:40.193705+0000","last_peered":"2024-04-17T10:31:40.193705+0000","last_clean":"2024-04-17T10:31:40.193705+0000","last_became_active":"2024-04-17T10:31:38.370089+0000","last_became_peered":"2024-04-17T10:31:38.370089+0000","last_unstale":"2024-04-17T10:31:40.193705+0000","last_undegraded":"2024-04-17T10:31:40.193705+0000","last_fullsized":"2024-04-17T10:31:40.193705+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:56.298204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365982+0000","last_change":"2024-04-17T10:31:37.354469+0000","last_active":"2024-04-17T10:31:39.365982+0000","last_peered":"2024-04-17T10:31:39.365982+0000","last_clean":"2024-04-17T10:31:39.365982+0000","last_became_active":"2024-04-17T10:31:37.354215+0000","last_became_peered":"2024-04-17T10:31:37.354215+0000","last_unstale":"2024-04-17T10:31:39.365982+0000","last_undegraded":"2024-04-17T10:31:39.365982+0000","last_fullsized":"2024-04-17T10:31:39.365982+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:33.318986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.364135+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.363335+0000","last_became_peered":"2024-04-17T10:31:38.363335+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:04.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.341274+0000","last_change":"2024-04-17T10:31:36.341274+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.341274+0000","last_undegraded":"2024-04-17T10:31:36.341274+0000","last_fullsized":"2024-04-17T10:31:36.341274+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340070+0000","last_change":"2024-04-17T10:31:36.340070+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340070+0000","last_undegraded":"2024-04-17T10:31:36.340070+0000","last_fullsized":"2024-04-17T10:31:36.340070+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366370+0000","last_change":"2024-04-17T10:31:37.359814+0000","last_active":"2024-04-17T10:31:39.366370+0000","last_peered":"2024-04-17T10:31:39.366370+0000","last_clean":"2024-04-17T10:31:39.366370+0000","last_became_active":"2024-04-17T10:31:37.359486+0000","last_became_peered":"2024-04-17T10:31:37.359486+0000","last_unstale":"2024-04-17T10:31:39.366370+0000","last_undegraded":"2024-04-17T10:31:39.366370+0000","last_fullsized":"2024-04-17T10:31:39.366370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:23:28.064562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366322+0000","last_change":"2024-04-17T10:31:38.371429+0000","last_active":"2024-04-17T10:31:39.366322+0000","last_peered":"2024-04-17T10:31:39.366322+0000","last_clean":"2024-04-17T10:31:39.366322+0000","last_became_active":"2024-04-17T10:31:38.371073+0000","last_became_peered":"2024-04-17T10:31:38.371073+0000","last_unstale":"2024-04-17T10:31:39.366322+0000","last_undegraded":"2024-04-17T10:31:39.366322+0000","last_fullsized":"2024-04-17T10:31:39.366322+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:01.747376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.336055+0000","last_change":"2024-04-17T10:31:36.336055+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.336055+0000","last_undegraded":"2024-04-17T10:31:36.336055+0000","last_fullsized":"2024-04-17T10:31:36.336055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637829+0000","last_change":"2024-04-17T10:31:38.366147+0000","last_active":"2024-04-17T10:31:39.637829+0000","last_peered":"2024-04-17T10:31:39.637829+0000","last_clean":"2024-04-17T10:31:39.637829+0000","last_became_active":"2024-04-17T10:31:38.365889+0000","last_became_peered":"2024-04-17T10:31:38.365889+0000","last_unstale":"2024-04-17T10:31:39.637829+0000","last_undegraded":"2024-04-17T10:31:39.637829+0000","last_fullsized":"2024-04-17T10:31:39.637829+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:18.111634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"95'6","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193733+0000","last_change":"2024-04-17T10:31:37.367318+0000","last_active":"2024-04-17T10:31:40.193733+0000","last_peered":"2024-04-17T10:31:40.193733+0000","last_clean":"2024-04-17T10:31:40.193733+0000","last_became_active":"2024-04-17T10:31:37.367023+0000","last_became_peered":"2024-04-17T10:31:37.367023+0000","last_unstale":"2024-04-17T10:31:40.193733+0000","last_undegraded":"2024-04-17T10:31:40.193733+0000","last_fullsized":"2024-04-17T10:31:40.193733+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:02.024574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637403+0000","last_change":"2024-04-17T10:31:38.362727+0000","last_active":"2024-04-17T10:31:39.637403+0000","last_peered":"2024-04-17T10:31:39.637403+0000","last_clean":"2024-04-17T10:31:39.637403+0000","last_became_active":"2024-04-17T10:31:38.362083+0000","last_became_peered":"2024-04-17T10:31:38.362083+0000","last_unstale":"2024-04-17T10:31:39.637403+0000","last_undegraded":"2024-04-17T10:31:39.637403+0000","last_fullsized":"2024-04-17T10:31:39.637403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:42.796034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340608+0000","last_change":"2024-04-17T10:31:36.340608+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340608+0000","last_undegraded":"2024-04-17T10:31:36.340608+0000","last_fullsized":"2024-04-17T10:31:36.340608+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193763+0000","last_change":"2024-04-17T10:31:37.354839+0000","last_active":"2024-04-17T10:31:40.193763+0000","last_peered":"2024-04-17T10:31:40.193763+0000","last_clean":"2024-04-17T10:31:40.193763+0000","last_became_active":"2024-04-17T10:31:37.354538+0000","last_became_peered":"2024-04-17T10:31:37.354538+0000","last_unstale":"2024-04-17T10:31:40.193763+0000","last_undegraded":"2024-04-17T10:31:40.193763+0000","last_fullsized":"2024-04-17T10:31:40.193763+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:10.283370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366942+0000","last_change":"2024-04-17T10:31:38.368131+0000","last_active":"2024-04-17T10:31:39.366942+0000","last_peered":"2024-04-17T10:31:39.366942+0000","last_clean":"2024-04-17T10:31:39.366942+0000","last_became_active":"2024-04-17T10:31:38.367815+0000","last_became_peered":"2024-04-17T10:31:38.367815+0000","last_unstale":"2024-04-17T10:31:39.366942+0000","last_undegraded":"2024-04-17T10:31:39.366942+0000","last_fullsized":"2024-04-17T10:31:39.366942+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:39.116896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637313+0000","last_change":"2024-04-17T10:31:37.358749+0000","last_active":"2024-04-17T10:31:39.637313+0000","last_peered":"2024-04-17T10:31:39.637313+0000","last_clean":"2024-04-17T10:31:39.637313+0000","last_became_active":"2024-04-17T10:31:37.357922+0000","last_became_peered":"2024-04-17T10:31:37.357922+0000","last_unstale":"2024-04-17T10:31:39.637313+0000","last_undegraded":"2024-04-17T10:31:39.637313+0000","last_fullsized":"2024-04-17T10:31:39.637313+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:23.174223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367510+0000","last_change":"2024-04-17T10:31:38.356263+0000","last_active":"2024-04-17T10:31:39.367510+0000","last_peered":"2024-04-17T10:31:39.367510+0000","last_clean":"2024-04-17T10:31:39.367510+0000","last_became_active":"2024-04-17T10:31:38.355993+0000","last_became_peered":"2024-04-17T10:31:38.355993+0000","last_unstale":"2024-04-17T10:31:39.367510+0000","last_undegraded":"2024-04-17T10:31:39.367510+0000","last_fullsized":"2024-04-17T10:31:39.367510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:55.031187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"95'8","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637287+0000","last_change":"2024-04-17T10:31:37.359691+0000","last_active":"2024-04-17T10:31:39.637287+0000","last_peered":"2024-04-17T10:31:39.637287+0000","last_clean":"2024-04-17T10:31:39.637287+0000","last_became_active":"2024-04-17T10:31:37.359365+0000","last_became_peered":"2024-04-17T10:31:37.359365+0000","last_unstale":"2024-04-17T10:31:39.637287+0000","last_undegraded":"2024-04-17T10:31:39.637287+0000","last_fullsized":"2024-04-17T10:31:39.637287+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:00:19.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.379892+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.379586+0000","last_became_peered":"2024-04-17T10:31:38.379586+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:35.337686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337309+0000","last_change":"2024-04-17T10:31:36.337309+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337309+0000","last_undegraded":"2024-04-17T10:31:36.337309+0000","last_fullsized":"2024-04-17T10:31:36.337309+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340462+0000","last_change":"2024-04-17T10:31:36.340462+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340462+0000","last_undegraded":"2024-04-17T10:31:36.340462+0000","last_fullsized":"2024-04-17T10:31:36.340462+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335517+0000","last_change":"2024-04-17T10:31:36.335517+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335517+0000","last_undegraded":"2024-04-17T10:31:36.335517+0000","last_fullsized":"2024-04-17T10:31:36.335517+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194178+0000","last_change":"2024-04-17T10:31:38.383600+0000","last_active":"2024-04-17T10:31:40.194178+0000","last_peered":"2024-04-17T10:31:40.194178+0000","last_clean":"2024-04-17T10:31:40.194178+0000","last_became_active":"2024-04-17T10:31:38.383354+0000","last_became_peered":"2024-04-17T10:31:38.383354+0000","last_unstale":"2024-04-17T10:31:40.194178+0000","last_undegraded":"2024-04-17T10:31:40.194178+0000","last_fullsized":"2024-04-17T10:31:40.194178+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:25:53.140820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340140+0000","last_change":"2024-04-17T10:31:36.340140+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340140+0000","last_undegraded":"2024-04-17T10:31:36.340140+0000","last_fullsized":"2024-04-17T10:31:36.340140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366890+0000","last_change":"2024-04-17T10:31:37.365058+0000","last_active":"2024-04-17T10:31:39.366890+0000","last_peered":"2024-04-17T10:31:39.366890+0000","last_clean":"2024-04-17T10:31:39.366890+0000","last_became_active":"2024-04-17T10:31:37.364541+0000","last_became_peered":"2024-04-17T10:31:37.364541+0000","last_unstale":"2024-04-17T10:31:39.366890+0000","last_undegraded":"2024-04-17T10:31:39.366890+0000","last_fullsized":"2024-04-17T10:31:39.366890+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:38.273762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340172+0000","last_change":"2024-04-17T10:31:36.340172+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340172+0000","last_undegraded":"2024-04-17T10:31:36.340172+0000","last_fullsized":"2024-04-17T10:31:36.340172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193864+0000","last_change":"2024-04-17T10:31:38.376121+0000","last_active":"2024-04-17T10:31:40.193864+0000","last_peered":"2024-04-17T10:31:40.193864+0000","last_clean":"2024-04-17T10:31:40.193864+0000","last_became_active":"2024-04-17T10:31:38.375833+0000","last_became_peered":"2024-04-17T10:31:38.375833+0000","last_unstale":"2024-04-17T10:31:40.193864+0000","last_undegraded":"2024-04-17T10:31:40.193864+0000","last_fullsized":"2024-04-17T10:31:40.193864+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:35:43.598662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366769+0000","last_change":"2024-04-17T10:31:37.362595+0000","last_active":"2024-04-17T10:31:39.366769+0000","last_peered":"2024-04-17T10:31:39.366769+0000","last_clean":"2024-04-17T10:31:39.366769+0000","last_became_active":"2024-04-17T10:31:37.362307+0000","last_became_peered":"2024-04-17T10:31:37.362307+0000","last_unstale":"2024-04-17T10:31:39.366769+0000","last_undegraded":"2024-04-17T10:31:39.366769+0000","last_fullsized":"2024-04-17T10:31:39.366769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:15:47.101230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340408+0000","last_change":"2024-04-17T10:31:36.340408+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340408+0000","last_undegraded":"2024-04-17T10:31:36.340408+0000","last_fullsized":"2024-04-17T10:31:36.340408+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194103+0000","last_change":"2024-04-17T10:31:38.386818+0000","last_active":"2024-04-17T10:31:40.194103+0000","last_peered":"2024-04-17T10:31:40.194103+0000","last_clean":"2024-04-17T10:31:40.194103+0000","last_became_active":"2024-04-17T10:31:38.383939+0000","last_became_peered":"2024-04-17T10:31:38.383939+0000","last_unstale":"2024-04-17T10:31:40.194103+0000","last_undegraded":"2024-04-17T10:31:40.194103+0000","last_fullsized":"2024-04-17T10:31:40.194103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:01.834781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335266+0000","last_change":"2024-04-17T10:31:36.335266+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335266+0000","last_undegraded":"2024-04-17T10:31:36.335266+0000","last_fullsized":"2024-04-17T10:31:36.335266+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194023+0000","last_change":"2024-04-17T10:31:38.376661+0000","last_active":"2024-04-17T10:31:40.194023+0000","last_peered":"2024-04-17T10:31:40.194023+0000","last_clean":"2024-04-17T10:31:40.194023+0000","last_became_active":"2024-04-17T10:31:38.376364+0000","last_became_peered":"2024-04-17T10:31:38.376364+0000","last_unstale":"2024-04-17T10:31:40.194023+0000","last_undegraded":"2024-04-17T10:31:40.194023+0000","last_fullsized":"2024-04-17T10:31:40.194023+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:28.335090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335050+0000","last_change":"2024-04-17T10:31:36.335050+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335050+0000","last_undegraded":"2024-04-17T10:31:36.335050+0000","last_fullsized":"2024-04-17T10:31:36.335050+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337226+0000","last_change":"2024-04-17T10:31:36.337226+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337226+0000","last_undegraded":"2024-04-17T10:31:36.337226+0000","last_fullsized":"2024-04-17T10:31:36.337226+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366830+0000","last_change":"2024-04-17T10:31:38.384626+0000","last_active":"2024-04-17T10:31:39.366830+0000","last_peered":"2024-04-17T10:31:39.366830+0000","last_clean":"2024-04-17T10:31:39.366830+0000","last_became_active":"2024-04-17T10:31:38.384373+0000","last_became_peered":"2024-04-17T10:31:38.384373+0000","last_unstale":"2024-04-17T10:31:39.366830+0000","last_undegraded":"2024-04-17T10:31:39.366830+0000","last_fullsized":"2024-04-17T10:31:39.366830+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:48.060879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193972+0000","last_change":"2024-04-17T10:31:37.354088+0000","last_active":"2024-04-17T10:31:40.193972+0000","last_peered":"2024-04-17T10:31:40.193972+0000","last_clean":"2024-04-17T10:31:40.193972+0000","last_became_active":"2024-04-17T10:31:37.353817+0000","last_became_peered":"2024-04-17T10:31:37.353817+0000","last_unstale":"2024-04-17T10:31:40.193972+0000","last_undegraded":"2024-04-17T10:31:40.193972+0000","last_fullsized":"2024-04-17T10:31:40.193972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:20.260258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366210+0000","last_change":"2024-04-17T10:31:38.374513+0000","last_active":"2024-04-17T10:31:39.366210+0000","last_peered":"2024-04-17T10:31:39.366210+0000","last_clean":"2024-04-17T10:31:39.366210+0000","last_became_active":"2024-04-17T10:31:38.374235+0000","last_became_peered":"2024-04-17T10:31:38.374235+0000","last_unstale":"2024-04-17T10:31:39.366210+0000","last_undegraded":"2024-04-17T10:31:39.366210+0000","last_fullsized":"2024-04-17T10:31:39.366210+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:20.114227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340005+0000","last_change":"2024-04-17T10:31:36.340005+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340005+0000","last_undegraded":"2024-04-17T10:31:36.340005+0000","last_fullsized":"2024-04-17T10:31:36.340005+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337484+0000","last_change":"2024-04-17T10:31:36.337484+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337484+0000","last_undegraded":"2024-04-17T10:31:36.337484+0000","last_fullsized":"2024-04-17T10:31:36.337484+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193954+0000","last_change":"2024-04-17T10:31:38.368340+0000","last_active":"2024-04-17T10:31:40.193954+0000","last_peered":"2024-04-17T10:31:40.193954+0000","last_clean":"2024-04-17T10:31:40.193954+0000","last_became_active":"2024-04-17T10:31:38.368065+0000","last_became_peered":"2024-04-17T10:31:38.368065+0000","last_unstale":"2024-04-17T10:31:40.193954+0000","last_undegraded":"2024-04-17T10:31:40.193954+0000","last_fullsized":"2024-04-17T10:31:40.193954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:38:20.873547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340808+0000","last_change":"2024-04-17T10:31:36.340808+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340808+0000","last_undegraded":"2024-04-17T10:31:36.340808+0000","last_fullsized":"2024-04-17T10:31:36.340808+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637594+0000","last_change":"2024-04-17T10:31:37.360694+0000","last_active":"2024-04-17T10:31:39.637594+0000","last_peered":"2024-04-17T10:31:39.637594+0000","last_clean":"2024-04-17T10:31:39.637594+0000","last_became_active":"2024-04-17T10:31:37.360046+0000","last_became_peered":"2024-04-17T10:31:37.360046+0000","last_unstale":"2024-04-17T10:31:39.637594+0000","last_undegraded":"2024-04-17T10:31:39.637594+0000","last_fullsized":"2024-04-17T10:31:39.637594+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:22.070539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-17T10:31:36.342331+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:31:36.342331+0000","last_peered":"2024-04-17T10:31:36.342331+0000","last_clean":"2024-04-17T10:31:36.342331+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:31:36.342331+0000","last_undegraded":"2024-04-17T10:31:36.342331+0000","last_fullsized":"2024-04-17T10:31:36.342331+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":72,"acting":72,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46772,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325068,"statfs":{"total":96636764160,"available":96588869632,"internally_reserved":0,"allocated":1740800,"data_stored":1121930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46192,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325648,"statfs":{"total":96636764160,"available":96589463552,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62836,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94309004,"statfs":{"total":96636764160,"available":96572420096,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54508,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317332,"statfs":{"total":96636764160,"available":96580947968,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54544,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317296,"statfs":{"total":96636764160,"available":96580911104,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50392,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321448,"statfs":{"total":96636764160,"available":96585162752,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46228,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325612,"statfs":{"total":96636764160,"available":96589426688,"internally_reserved":0,"allocated":1187840,"data_stored":552158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":1802240,"data_stored":1144774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3e is not active+clean 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.3b is not active+clean 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.39 is not active+clean 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.37 is not active+clean 2024-04-17T10:31:43.244 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.35 is not active+clean 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.32 is not active+clean 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2d is not active+clean 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2b is not active+clean 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2a is not active+clean 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.29 is not active+clean 2024-04-17T10:31:43.245 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.27 is not active+clean 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.26 is not active+clean 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.23 is not active+clean 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.22 is not active+clean 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.21 is not active+clean 2024-04-17T10:31:43.246 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.338128+0000', 'last_change': '2024-04-17T10:31:36.338128+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.338128+0000', 'last_undegraded': '2024-04-17T10:31:36.338128+0000', 'last_fullsized': '2024-04-17T10:31:36.338128+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.335826+0000', 'last_change': '2024-04-17T10:31:36.335826+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.335826+0000', 'last_undegraded': '2024-04-17T10:31:36.335826+0000', 'last_fullsized': '2024-04-17T10:31:36.335826+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.1e is not active+clean 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1d is not active+clean 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340523+0000', 'last_change': '2024-04-17T10:31:36.340523+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340523+0000', 'last_undegraded': '2024-04-17T10:31:36.340523+0000', 'last_fullsized': '2024-04-17T10:31:36.340523+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.337899+0000', 'last_change': '2024-04-17T10:31:36.337899+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.337899+0000', 'last_undegraded': '2024-04-17T10:31:36.337899+0000', 'last_fullsized': '2024-04-17T10:31:36.337899+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:31:43.247 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-04-17T10:31:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.334883+0000', 'last_change': '2024-04-17T10:31:36.334883+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.334883+0000', 'last_undegraded': '2024-04-17T10:31:36.334883+0000', 'last_fullsized': '2024-04-17T10:31:36.334883+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:31:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.19 is not active+clean 2024-04-17T10:31:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.18 is not active+clean 2024-04-17T10:31:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.341274+0000', 'last_change': '2024-04-17T10:31:36.341274+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.341274+0000', 'last_undegraded': '2024-04-17T10:31:36.341274+0000', 'last_fullsized': '2024-04-17T10:31:36.341274+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:31:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.14 is not active+clean 2024-04-17T10:31:43.248 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340070+0000', 'last_change': '2024-04-17T10:31:36.340070+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340070+0000', 'last_undegraded': '2024-04-17T10:31:36.340070+0000', 'last_fullsized': '2024-04-17T10:31:36.340070+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.15 is not active+clean 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.336055+0000', 'last_change': '2024-04-17T10:31:36.336055+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.336055+0000', 'last_undegraded': '2024-04-17T10:31:36.336055+0000', 'last_fullsized': '2024-04-17T10:31:36.336055+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.13 is not active+clean 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340608+0000', 'last_change': '2024-04-17T10:31:36.340608+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340608+0000', 'last_undegraded': '2024-04-17T10:31:36.340608+0000', 'last_fullsized': '2024-04-17T10:31:36.340608+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.11 is not active+clean 2024-04-17T10:31:43.249 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.337309+0000', 'last_change': '2024-04-17T10:31:36.337309+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.337309+0000', 'last_undegraded': '2024-04-17T10:31:36.337309+0000', 'last_fullsized': '2024-04-17T10:31:36.337309+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.d is not active+clean 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.e is not active+clean 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340462+0000', 'last_change': '2024-04-17T10:31:36.340462+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340462+0000', 'last_undegraded': '2024-04-17T10:31:36.340462+0000', 'last_fullsized': '2024-04-17T10:31:36.340462+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.e is not active+clean 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-04-17T10:31:43.250 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.335517+0000', 'last_change': '2024-04-17T10:31:36.335517+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.335517+0000', 'last_undegraded': '2024-04-17T10:31:36.335517+0000', 'last_fullsized': '2024-04-17T10:31:36.335517+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340140+0000', 'last_change': '2024-04-17T10:31:36.340140+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340140+0000', 'last_undegraded': '2024-04-17T10:31:36.340140+0000', 'last_fullsized': '2024-04-17T10:31:36.340140+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.9 is not active+clean 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.a is not active+clean 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340172+0000', 'last_change': '2024-04-17T10:31:36.340172+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340172+0000', 'last_undegraded': '2024-04-17T10:31:36.340172+0000', 'last_fullsized': '2024-04-17T10:31:36.340172+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.4 is not active+clean 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.7 is not active+clean 2024-04-17T10:31:43.251 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340408+0000', 'last_change': '2024-04-17T10:31:36.340408+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340408+0000', 'last_undegraded': '2024-04-17T10:31:36.340408+0000', 'last_fullsized': '2024-04-17T10:31:36.340408+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.335266+0000', 'last_change': '2024-04-17T10:31:36.335266+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.335266+0000', 'last_undegraded': '2024-04-17T10:31:36.335266+0000', 'last_fullsized': '2024-04-17T10:31:36.335266+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.335050+0000', 'last_change': '2024-04-17T10:31:36.335050+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.335050+0000', 'last_undegraded': '2024-04-17T10:31:36.335050+0000', 'last_fullsized': '2024-04-17T10:31:36.335050+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.7 is not active+clean 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.337226+0000', 'last_change': '2024-04-17T10:31:36.337226+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.337226+0000', 'last_undegraded': '2024-04-17T10:31:36.337226+0000', 'last_fullsized': '2024-04-17T10:31:36.337226+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340005+0000', 'last_change': '2024-04-17T10:31:36.340005+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340005+0000', 'last_undegraded': '2024-04-17T10:31:36.340005+0000', 'last_fullsized': '2024-04-17T10:31:36.340005+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-17T10:31:43.252 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.2 is not active+clean 2024-04-17T10:31:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-04-17T10:31:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.337484+0000', 'last_change': '2024-04-17T10:31:36.337484+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.337484+0000', 'last_undegraded': '2024-04-17T10:31:36.337484+0000', 'last_fullsized': '2024-04-17T10:31:36.337484+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-17T10:31:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.0 is not active+clean 2024-04-17T10:31:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 93, 'state': 'creating+peering', 'last_fresh': '2024-04-17T10:31:36.340808+0000', 'last_change': '2024-04-17T10:31:36.340808+0000', 'last_active': '2024-04-17T10:31:36.325538+0000', 'last_peered': '2024-04-17T10:31:36.325538+0000', 'last_clean': '2024-04-17T10:31:36.325538+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:36.340808+0000', 'last_undegraded': '2024-04-17T10:31:36.340808+0000', 'last_fullsized': '2024-04-17T10:31:36.340808+0000', 'mapping_epoch': 93, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 93, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:36.325538+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-17T10:31:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:PG 14.f is not active+clean 2024-04-17T10:31:43.253 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '14.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-17T10:31:37.332052+0000', 'last_change': '2024-04-17T10:31:37.332052+0000', 'last_active': '2024-04-17T10:31:37.332052+0000', 'last_peered': '2024-04-17T10:31:37.332052+0000', 'last_clean': '2024-04-17T10:31:37.332052+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-17T10:31:37.332052+0000', 'last_undegraded': '2024-04-17T10:31:37.332052+0000', 'last_fullsized': '2024-04-17T10:31:37.332052+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'last_clean_scrub_stamp': '2024-04-17T10:31:37.332052+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-17T10:31:43.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:31:43.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.431+0000 7f79ee400640 1 Processor -- start 2024-04-17T10:31:43.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.431+0000 7f79ee400640 1 -- start start 2024-04-17T10:31:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ee400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79e8099a50 0x7f79e815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ee400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f79e80d03a0 0x7f79e8099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ee400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e80cde20 con 0x7f79e809a740 2024-04-17T10:31:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ee400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e80cd1b0 con 0x7f79e8099a50 2024-04-17T10:31:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ee400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e809a370 con 0x7f79e80d03a0 2024-04-17T10:31:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ed000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79e8099a50 0x7f79e815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ed000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f79e8099a50 0x7f79e815c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46394/0 (socket says 172.21.15.67:46394) 2024-04-17T10:31:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ed000640 1 -- 172.21.15.67:0/2221053966 learned_addr learned my addr 172.21.15.67:0/2221053966 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79dfe00640 1 --2- 172.21.15.67:0/2221053966 >> v2:172.21.15.86:3300/0 conn(0x7f79e80d03a0 0x7f79e8099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ed000640 1 -- 172.21.15.67:0/2221053966 >> v2:172.21.15.86:3300/0 conn(0x7f79e80d03a0 msgr2=0x7f79e8099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ed000640 1 --2- 172.21.15.67:0/2221053966 >> v2:172.21.15.86:3300/0 conn(0x7f79e80d03a0 0x7f79e8099e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ed000640 1 -- 172.21.15.67:0/2221053966 >> v1:172.21.15.86:6789/0 conn(0x7f79e809a740 legacy=0x7f79e815f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:43.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.435+0000 7f79ed000640 1 -- 172.21.15.67:0/2221053966 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e80cf300 con 0x7f79e8099a50 2024-04-17T10:31:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ed000640 1 --2- 172.21.15.67:0/2221053966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e8099a50 0x7f79e815c7a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f79e809acd0 tx=0x7f79d801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b676d65b8e2ea60 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79e7e00640 1 -- 172.21.15.67:0/2221053966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d80b0070 con 0x7f79e8099a50 2024-04-17T10:31:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79e7e00640 1 -- 172.21.15.67:0/2221053966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d80ab400 con 0x7f79e8099a50 2024-04-17T10:31:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79e7e00640 1 -- 172.21.15.67:0/2221053966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79d80ab7d0 con 0x7f79e8099a50 2024-04-17T10:31:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- 172.21.15.67:0/2221053966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e8099a50 msgr2=0x7f79e815c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 --2- 172.21.15.67:0/2221053966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e8099a50 0x7f79e815c7a0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f79e809acd0 tx=0x7f79d801c450 comp rx=0 tx=0).stop 2024-04-17T10:31:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- 172.21.15.67:0/2221053966 shutdown_connections 2024-04-17T10:31:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 --2- 172.21.15.67:0/2221053966 >> v2:172.21.15.86:3300/0 conn(0x7f79e80d03a0 0x7f79e8099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 --2- 172.21.15.67:0/2221053966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e8099a50 0x7f79e815c7a0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- 172.21.15.67:0/2221053966 >> 172.21.15.67:0/2221053966 conn(0x7f79e80d1690 msgr2=0x7f79e80d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- 172.21.15.67:0/2221053966 shutdown_connections 2024-04-17T10:31:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- 172.21.15.67:0/2221053966 wait complete. 2024-04-17T10:31:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 Processor -- start 2024-04-17T10:31:43.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- start start 2024-04-17T10:31:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8099a50 0x7f79e815ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e809a740 0x7f79e815ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e80d03a0 0x7f79e80b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81686b0 con 0x7f79e80d03a0 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80cfe80 con 0x7f79e8099a50 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79ee400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8168830 con 0x7f79e809a740 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.439+0000 7f79eda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e80d03a0 0x7f79e80b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79ed000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8099a50 0x7f79e815ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79ed000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8099a50 0x7f79e815ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:57982/0 (socket says 172.21.15.67:57982) 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79eda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e80d03a0 0x7f79e80b5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46404/0 (socket says 172.21.15.67:46404) 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79ed000640 1 -- 172.21.15.67:0/2701333738 learned_addr learned my addr 172.21.15.67:0/2701333738 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79dfe00640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e809a740 0x7f79e815ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79eda00640 1 -- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e809a740 msgr2=0x7f79e815ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79eda00640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e809a740 0x7f79e815ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79eda00640 1 -- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8099a50 msgr2=0x7f79e815ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79eda00640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8099a50 0x7f79e815ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79eda00640 1 -- 172.21.15.67:0/2701333738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8070050 con 0x7f79e80d03a0 2024-04-17T10:31:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79eda00640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e80d03a0 0x7f79e80b5af0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f79e000be00 tx=0x7f79e000bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79e5600640 1 -- 172.21.15.67:0/2701333738 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0003980 con 0x7f79e80d03a0 2024-04-17T10:31:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79e5600640 1 -- 172.21.15.67:0/2701333738 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e0003b20 con 0x7f79e80d03a0 2024-04-17T10:31:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79e5600640 1 -- 172.21.15.67:0/2701333738 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e0004020 con 0x7f79e80d03a0 2024-04-17T10:31:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e80b0960 con 0x7f79e80d03a0 2024-04-17T10:31:43.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e80b64e0 con 0x7f79e80d03a0 2024-04-17T10:31:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e8074dc0 con 0x7f79e80d03a0 2024-04-17T10:31:43.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79e5600640 1 -- 172.21.15.67:0/2701333738 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f79e0006d10 con 0x7f79e80d03a0 2024-04-17T10:31:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.443+0000 7f79e5600640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f79c4066f60 0x7f79c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.447+0000 7f79e5600640 1 -- 172.21.15.67:0/2701333738 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f79e0084e40 con 0x7f79e80d03a0 2024-04-17T10:31:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.447+0000 7f79ed000640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f79c4066f60 0x7f79c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.447+0000 7f79ed000640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f79c4066f60 0x7f79c4069420 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f79d801c490 tx=0x7f79d80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.447+0000 7f79e5600640 1 -- 172.21.15.67:0/2701333738 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f79e0055700 con 0x7f79e80d03a0 2024-04-17T10:31:43.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.667+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f79e80d24a0 con 0x7f79e80d03a0 2024-04-17T10:31:43.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.667+0000 7f79e5600640 1 -- 172.21.15.67:0/2701333738 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1335 (secure 0 0 0) 0x7f79e00597a0 con 0x7f79e80d03a0 2024-04-17T10:31:43.660 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:43.660 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":434,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":96,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":48},{"state_name":"unknown","count":28},{"state_name":"creating+peering","count":21}],"num_pgs":97,"num_pools":3,"num_objects":10,"data_bytes":591980,"bytes_used":422359040,"bytes_avail":772671754240,"bytes_total":773094113280,"unknown_pgs_ratio":0.28865978121757507,"inactive_pgs_ratio":0.2164948433637619,"write_bytes_sec":520,"read_op_per_sec":0,"write_op_per_sec":3},"fsmap":{"epoch":64,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"a","status":"up:active","gid":7698}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-17T10:31:00.223648+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:31:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f79c4066f60 msgr2=0x7f79c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f79c4066f60 0x7f79c4069420 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f79d801c490 tx=0x7f79d80a6000 comp rx=0 tx=0).stop 2024-04-17T10:31:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e80d03a0 msgr2=0x7f79e80b5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e80d03a0 0x7f79e80b5af0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f79e000be00 tx=0x7f79e000bef0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 shutdown_connections 2024-04-17T10:31:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f79c4066f60 0x7f79c4069420 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f79e80d03a0 0x7f79e80b5af0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f79e809a740 0x7f79e815ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 --2- 172.21.15.67:0/2701333738 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f79e8099a50 0x7f79e815ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 >> 172.21.15.67:0/2701333738 conn(0x7f79e80d1690 msgr2=0x7f79e80ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 shutdown_connections 2024-04-17T10:31:43.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.671+0000 7f79ee400640 1 -- 172.21.15.67:0/2701333738 wait complete. 2024-04-17T10:31:43.681 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-17T10:31:43.681 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:31:43.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8948000640 1 Processor -- start 2024-04-17T10:31:43.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8948000640 1 -- start start 2024-04-17T10:31:43.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8948000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f89400b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8948000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89400b0cc0 con 0x7f89400714d0 2024-04-17T10:31:43.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8948000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89400b0fa0 con 0x7f89400b57b0 2024-04-17T10:31:43.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8948000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89400b1280 con 0x7f894015dd10 2024-04-17T10:31:43.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8946000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f89400714d0 0x7f894006c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34488/0 (socket says 172.21.15.67:34488) 2024-04-17T10:31:43.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8946000640 1 -- 172.21.15.67:0/1354460435 learned_addr learned my addr 172.21.15.67:0/1354460435 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:43.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8945600640 1 --2- 172.21.15.67:0/1354460435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f89400b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f893fe00640 1 -- 172.21.15.67:0/1354460435 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823171355 0 0) 0x7f89400b0cc0 con 0x7f89400714d0 2024-04-17T10:31:43.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f893fe00640 1 -- 172.21.15.67:0/1354460435 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8928003530 con 0x7f89400714d0 2024-04-17T10:31:43.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8945600640 1 -- 172.21.15.67:0/1354460435 >> v1:172.21.15.86:6789/0 conn(0x7f89400b57b0 legacy=0x7f8940070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8945600640 1 -- 172.21.15.67:0/1354460435 >> v1:172.21.15.67:6789/0 conn(0x7f89400714d0 legacy=0x7f894006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.831+0000 7f8945600640 1 -- 172.21.15.67:0/1354460435 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894006bed0 con 0x7f894015dd10 2024-04-17T10:31:43.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8945600640 1 --2- 172.21.15.67:0/1354460435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f89400b5270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f89340097f0 tx=0x7f893401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac7c4eb8f83a2a98 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f893fe00640 1 -- 172.21.15.67:0/1354460435 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340ab600 con 0x7f894015dd10 2024-04-17T10:31:43.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f893fe00640 1 -- 172.21.15.67:0/1354460435 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89340b0070 con 0x7f894015dd10 2024-04-17T10:31:43.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f893fe00640 1 -- 172.21.15.67:0/1354460435 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340ab9c0 con 0x7f894015dd10 2024-04-17T10:31:43.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8948000640 1 -- 172.21.15.67:0/1354460435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 msgr2=0x7f89400b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8948000640 1 --2- 172.21.15.67:0/1354460435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f89400b5270 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f89340097f0 tx=0x7f893401c420 comp rx=0 tx=0).stop 2024-04-17T10:31:43.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8948000640 1 -- 172.21.15.67:0/1354460435 shutdown_connections 2024-04-17T10:31:43.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8948000640 1 --2- 172.21.15.67:0/1354460435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f89400b5270 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8948000640 1 -- 172.21.15.67:0/1354460435 >> 172.21.15.67:0/1354460435 conn(0x7f89400b34d0 msgr2=0x7f89400b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:43.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8948000640 1 -- 172.21.15.67:0/1354460435 shutdown_connections 2024-04-17T10:31:43.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.835+0000 7f8948000640 1 -- 172.21.15.67:0/1354460435 wait complete. 2024-04-17T10:31:43.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 Processor -- start 2024-04-17T10:31:43.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 -- start start 2024-04-17T10:31:43.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89400714d0 0x7f8940148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f894015f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89401609e0 0x7f8940145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894006da00 con 0x7f89400714d0 2024-04-17T10:31:43.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894006d880 con 0x7f89401609e0 2024-04-17T10:31:43.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f894016bc70 con 0x7f894015dd10 2024-04-17T10:31:43.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89400714d0 0x7f8940148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89400714d0 0x7f8940148b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46412/0 (socket says 172.21.15.67:46412) 2024-04-17T10:31:43.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 -- 172.21.15.67:0/3983513303 learned_addr learned my addr 172.21.15.67:0/3983513303 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:43.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8946000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89401609e0 0x7f8940145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 -- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 msgr2=0x7f894015f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8944c00640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f894015f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f894015f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 -- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89401609e0 msgr2=0x7f8940145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89401609e0 0x7f8940145e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 -- 172.21.15.67:0/3983513303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894015e690 con 0x7f89400714d0 2024-04-17T10:31:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8946000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89401609e0 0x7f8940145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8944c00640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f894015f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8945600640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89400714d0 0x7f8940148b10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8934019030 tx=0x7f893401ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f893d600640 1 -- 172.21.15.67:0/3983513303 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340bf610 con 0x7f89400714d0 2024-04-17T10:31:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f893d600640 1 -- 172.21.15.67:0/3983513303 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89340bf7d0 con 0x7f89400714d0 2024-04-17T10:31:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f893d600640 1 -- 172.21.15.67:0/3983513303 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89340bfbc0 con 0x7f89400714d0 2024-04-17T10:31:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894015e970 con 0x7f89400714d0 2024-04-17T10:31:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.839+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894015ef30 con 0x7f89400714d0 2024-04-17T10:31:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.843+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8940070bb0 con 0x7f89400714d0 2024-04-17T10:31:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.843+0000 7f893d600640 1 -- 172.21.15.67:0/3983513303 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f89340bd020 con 0x7f89400714d0 2024-04-17T10:31:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.843+0000 7f893d600640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8920066f60 0x7f8920069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.843+0000 7f893d600640 1 -- 172.21.15.67:0/3983513303 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f893411f700 con 0x7f89400714d0 2024-04-17T10:31:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.843+0000 7f8944c00640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8920066f60 0x7f8920069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.843+0000 7f8944c00640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8920066f60 0x7f8920069420 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8940153210 tx=0x7f892c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:43.847+0000 7f893d600640 1 -- 172.21.15.67:0/3983513303 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f89340160d0 con 0x7f89400714d0 2024-04-17T10:31:44.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.011+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8940074e30 con 0x7f8920066f60 2024-04-17T10:31:44.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.019+0000 7f893d600640 1 -- 172.21.15.67:0/3983513303 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+246197 (secure 0 0 0) 0x7f8940074e30 con 0x7f8920066f60 2024-04-17T10:31:44.013 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:44.014 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:31:44.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8920066f60 msgr2=0x7f8920069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:44.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8920066f60 0x7f8920069420 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8940153210 tx=0x7f892c009040 comp rx=0 tx=0).stop 2024-04-17T10:31:44.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89400714d0 msgr2=0x7f8940148b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:44.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89400714d0 0x7f8940148b10 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8934019030 tx=0x7f893401ac60 comp rx=0 tx=0).stop 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 shutdown_connections 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8920066f60 0x7f8920069420 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89401609e0 0x7f8940145e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f894015dd10 0x7f894015f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 --2- 172.21.15.67:0/3983513303 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89400714d0 0x7f8940148b10 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 >> 172.21.15.67:0/3983513303 conn(0x7f89400b34d0 msgr2=0x7f89400b2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 shutdown_connections 2024-04-17T10:31:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:44.023+0000 7f8948000640 1 -- 172.21.15.67:0/3983513303 wait complete. 2024-04-17T10:31:44.033 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":299,"stamp":"2024-04-17T10:31:42.234222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591980,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":177,"num_write_kb":2419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":132,"ondisk_log_size":132,"up":138,"acting":138,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":117,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":412460,"kb_used_data":10292,"kb_used_omap":12,"kb_used_meta":402035,"kb_avail":754562260,"statfs":{"total":773094113280,"available":772671754240,"internally_reserved":0,"allocated":10539008,"data_stored":5517724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":411684401},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.903728"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366379+0000","last_change":"2024-04-17T10:31:38.372177+0000","last_active":"2024-04-17T10:31:39.366379+0000","last_peered":"2024-04-17T10:31:39.366379+0000","last_clean":"2024-04-17T10:31:39.366379+0000","last_became_active":"2024-04-17T10:31:38.371875+0000","last_became_peered":"2024-04-17T10:31:38.371875+0000","last_unstale":"2024-04-17T10:31:39.366379+0000","last_undegraded":"2024-04-17T10:31:39.366379+0000","last_fullsized":"2024-04-17T10:31:39.366379+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:36:29.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366082+0000","last_change":"2024-04-17T10:31:38.362877+0000","last_active":"2024-04-17T10:31:39.366082+0000","last_peered":"2024-04-17T10:31:39.366082+0000","last_clean":"2024-04-17T10:31:39.366082+0000","last_became_active":"2024-04-17T10:31:38.362413+0000","last_became_peered":"2024-04-17T10:31:38.362413+0000","last_unstale":"2024-04-17T10:31:39.366082+0000","last_undegraded":"2024-04-17T10:31:39.366082+0000","last_fullsized":"2024-04-17T10:31:39.366082+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:03.031089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637416+0000","last_change":"2024-04-17T10:31:38.367300+0000","last_active":"2024-04-17T10:31:39.637416+0000","last_peered":"2024-04-17T10:31:39.637416+0000","last_clean":"2024-04-17T10:31:39.637416+0000","last_became_active":"2024-04-17T10:31:38.366715+0000","last_became_peered":"2024-04-17T10:31:38.366715+0000","last_unstale":"2024-04-17T10:31:39.637416+0000","last_undegraded":"2024-04-17T10:31:39.637416+0000","last_fullsized":"2024-04-17T10:31:39.637416+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:02.550932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193707+0000","last_change":"2024-04-17T10:31:38.382652+0000","last_active":"2024-04-17T10:31:40.193707+0000","last_peered":"2024-04-17T10:31:40.193707+0000","last_clean":"2024-04-17T10:31:40.193707+0000","last_became_active":"2024-04-17T10:31:38.382384+0000","last_became_peered":"2024-04-17T10:31:38.382384+0000","last_unstale":"2024-04-17T10:31:40.193707+0000","last_undegraded":"2024-04-17T10:31:40.193707+0000","last_fullsized":"2024-04-17T10:31:40.193707+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:20.999198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637175+0000","last_change":"2024-04-17T10:31:38.367243+0000","last_active":"2024-04-17T10:31:39.637175+0000","last_peered":"2024-04-17T10:31:39.637175+0000","last_clean":"2024-04-17T10:31:39.637175+0000","last_became_active":"2024-04-17T10:31:38.366952+0000","last_became_peered":"2024-04-17T10:31:38.366952+0000","last_unstale":"2024-04-17T10:31:39.637175+0000","last_undegraded":"2024-04-17T10:31:39.637175+0000","last_fullsized":"2024-04-17T10:31:39.637175+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:24.096194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367386+0000","last_change":"2024-04-17T10:31:38.369001+0000","last_active":"2024-04-17T10:31:39.367386+0000","last_peered":"2024-04-17T10:31:39.367386+0000","last_clean":"2024-04-17T10:31:39.367386+0000","last_became_active":"2024-04-17T10:31:38.367830+0000","last_became_peered":"2024-04-17T10:31:38.367830+0000","last_unstale":"2024-04-17T10:31:39.367386+0000","last_undegraded":"2024-04-17T10:31:39.367386+0000","last_fullsized":"2024-04-17T10:31:39.367386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:52.706957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367226+0000","last_change":"2024-04-17T10:31:38.357758+0000","last_active":"2024-04-17T10:31:39.367226+0000","last_peered":"2024-04-17T10:31:39.367226+0000","last_clean":"2024-04-17T10:31:39.367226+0000","last_became_active":"2024-04-17T10:31:38.357482+0000","last_became_peered":"2024-04-17T10:31:38.357482+0000","last_unstale":"2024-04-17T10:31:39.367226+0000","last_undegraded":"2024-04-17T10:31:39.367226+0000","last_fullsized":"2024-04-17T10:31:39.367226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:48.241446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637619+0000","last_change":"2024-04-17T10:31:38.366467+0000","last_active":"2024-04-17T10:31:39.637619+0000","last_peered":"2024-04-17T10:31:39.637619+0000","last_clean":"2024-04-17T10:31:39.637619+0000","last_became_active":"2024-04-17T10:31:38.366066+0000","last_became_peered":"2024-04-17T10:31:38.366066+0000","last_unstale":"2024-04-17T10:31:39.637619+0000","last_undegraded":"2024-04-17T10:31:39.637619+0000","last_fullsized":"2024-04-17T10:31:39.637619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:08.729085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637251+0000","last_change":"2024-04-17T10:31:38.363077+0000","last_active":"2024-04-17T10:31:39.637251+0000","last_peered":"2024-04-17T10:31:39.637251+0000","last_clean":"2024-04-17T10:31:39.637251+0000","last_became_active":"2024-04-17T10:31:38.362827+0000","last_became_peered":"2024-04-17T10:31:38.362827+0000","last_unstale":"2024-04-17T10:31:39.637251+0000","last_undegraded":"2024-04-17T10:31:39.637251+0000","last_fullsized":"2024-04-17T10:31:39.637251+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:56:20.141939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193544+0000","last_change":"2024-04-17T10:31:38.390689+0000","last_active":"2024-04-17T10:31:40.193544+0000","last_peered":"2024-04-17T10:31:40.193544+0000","last_clean":"2024-04-17T10:31:40.193544+0000","last_became_active":"2024-04-17T10:31:38.389673+0000","last_became_peered":"2024-04-17T10:31:38.389673+0000","last_unstale":"2024-04-17T10:31:40.193544+0000","last_undegraded":"2024-04-17T10:31:40.193544+0000","last_fullsized":"2024-04-17T10:31:40.193544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:05:33.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194163+0000","last_change":"2024-04-17T10:31:38.386404+0000","last_active":"2024-04-17T10:31:40.194163+0000","last_peered":"2024-04-17T10:31:40.194163+0000","last_clean":"2024-04-17T10:31:40.194163+0000","last_became_active":"2024-04-17T10:31:38.383973+0000","last_became_peered":"2024-04-17T10:31:38.383973+0000","last_unstale":"2024-04-17T10:31:40.194163+0000","last_undegraded":"2024-04-17T10:31:40.194163+0000","last_fullsized":"2024-04-17T10:31:40.194163+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:12.821691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366183+0000","last_change":"2024-04-17T10:31:38.362736+0000","last_active":"2024-04-17T10:31:39.366183+0000","last_peered":"2024-04-17T10:31:39.366183+0000","last_clean":"2024-04-17T10:31:39.366183+0000","last_became_active":"2024-04-17T10:31:38.362405+0000","last_became_peered":"2024-04-17T10:31:38.362405+0000","last_unstale":"2024-04-17T10:31:39.366183+0000","last_undegraded":"2024-04-17T10:31:39.366183+0000","last_fullsized":"2024-04-17T10:31:39.366183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:55.770687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.367062+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.366356+0000","last_became_peered":"2024-04-17T10:31:38.366356+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:34.604484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365841+0000","last_change":"2024-04-17T10:31:38.363969+0000","last_active":"2024-04-17T10:31:39.365841+0000","last_peered":"2024-04-17T10:31:39.365841+0000","last_clean":"2024-04-17T10:31:39.365841+0000","last_became_active":"2024-04-17T10:31:38.363731+0000","last_became_peered":"2024-04-17T10:31:38.363731+0000","last_unstale":"2024-04-17T10:31:39.365841+0000","last_undegraded":"2024-04-17T10:31:39.365841+0000","last_fullsized":"2024-04-17T10:31:39.365841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:25.031678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367325+0000","last_change":"2024-04-17T10:31:38.364443+0000","last_active":"2024-04-17T10:31:39.367325+0000","last_peered":"2024-04-17T10:31:39.367325+0000","last_clean":"2024-04-17T10:31:39.367325+0000","last_became_active":"2024-04-17T10:31:38.364134+0000","last_became_peered":"2024-04-17T10:31:38.364134+0000","last_unstale":"2024-04-17T10:31:39.367325+0000","last_undegraded":"2024-04-17T10:31:39.367325+0000","last_fullsized":"2024-04-17T10:31:39.367325+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:13:32.599449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637675+0000","last_change":"2024-04-17T10:31:38.367434+0000","last_active":"2024-04-17T10:31:39.637675+0000","last_peered":"2024-04-17T10:31:39.637675+0000","last_clean":"2024-04-17T10:31:39.637675+0000","last_became_active":"2024-04-17T10:31:38.366004+0000","last_became_peered":"2024-04-17T10:31:38.366004+0000","last_unstale":"2024-04-17T10:31:39.637675+0000","last_undegraded":"2024-04-17T10:31:39.637675+0000","last_fullsized":"2024-04-17T10:31:39.637675+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:20.424487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193672+0000","last_change":"2024-04-17T10:31:38.387676+0000","last_active":"2024-04-17T10:31:40.193672+0000","last_peered":"2024-04-17T10:31:40.193672+0000","last_clean":"2024-04-17T10:31:40.193672+0000","last_became_active":"2024-04-17T10:31:38.380682+0000","last_became_peered":"2024-04-17T10:31:38.380682+0000","last_unstale":"2024-04-17T10:31:40.193672+0000","last_undegraded":"2024-04-17T10:31:40.193672+0000","last_fullsized":"2024-04-17T10:31:40.193672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:59.714995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193655+0000","last_change":"2024-04-17T10:31:38.379789+0000","last_active":"2024-04-17T10:31:40.193655+0000","last_peered":"2024-04-17T10:31:40.193655+0000","last_clean":"2024-04-17T10:31:40.193655+0000","last_became_active":"2024-04-17T10:31:38.379237+0000","last_became_peered":"2024-04-17T10:31:38.379237+0000","last_unstale":"2024-04-17T10:31:40.193655+0000","last_undegraded":"2024-04-17T10:31:40.193655+0000","last_fullsized":"2024-04-17T10:31:40.193655+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:01.930369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.338128+0000","last_change":"2024-04-17T10:31:36.338128+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.338128+0000","last_undegraded":"2024-04-17T10:31:36.338128+0000","last_fullsized":"2024-04-17T10:31:36.338128+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.386544+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.380593+0000","last_became_peered":"2024-04-17T10:31:38.380593+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:53:17.992647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335826+0000","last_change":"2024-04-17T10:31:36.335826+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335826+0000","last_undegraded":"2024-04-17T10:31:36.335826+0000","last_fullsized":"2024-04-17T10:31:36.335826+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340523+0000","last_change":"2024-04-17T10:31:36.340523+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340523+0000","last_undegraded":"2024-04-17T10:31:36.340523+0000","last_fullsized":"2024-04-17T10:31:36.340523+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194225+0000","last_change":"2024-04-17T10:31:38.383915+0000","last_active":"2024-04-17T10:31:40.194225+0000","last_peered":"2024-04-17T10:31:40.194225+0000","last_clean":"2024-04-17T10:31:40.194225+0000","last_became_active":"2024-04-17T10:31:38.382791+0000","last_became_peered":"2024-04-17T10:31:38.382791+0000","last_unstale":"2024-04-17T10:31:40.194225+0000","last_undegraded":"2024-04-17T10:31:40.194225+0000","last_fullsized":"2024-04-17T10:31:40.194225+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:12.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337899+0000","last_change":"2024-04-17T10:31:36.337899+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337899+0000","last_undegraded":"2024-04-17T10:31:36.337899+0000","last_fullsized":"2024-04-17T10:31:36.337899+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637195+0000","last_change":"2024-04-17T10:31:38.366551+0000","last_active":"2024-04-17T10:31:39.637195+0000","last_peered":"2024-04-17T10:31:39.637195+0000","last_clean":"2024-04-17T10:31:39.637195+0000","last_became_active":"2024-04-17T10:31:38.366244+0000","last_became_peered":"2024-04-17T10:31:38.366244+0000","last_unstale":"2024-04-17T10:31:39.637195+0000","last_undegraded":"2024-04-17T10:31:39.637195+0000","last_fullsized":"2024-04-17T10:31:39.637195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:37.769462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.334883+0000","last_change":"2024-04-17T10:31:36.334883+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.334883+0000","last_undegraded":"2024-04-17T10:31:36.334883+0000","last_fullsized":"2024-04-17T10:31:36.334883+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637612+0000","last_change":"2024-04-17T10:31:37.366420+0000","last_active":"2024-04-17T10:31:39.637612+0000","last_peered":"2024-04-17T10:31:39.637612+0000","last_clean":"2024-04-17T10:31:39.637612+0000","last_became_active":"2024-04-17T10:31:37.365712+0000","last_became_peered":"2024-04-17T10:31:37.365712+0000","last_unstale":"2024-04-17T10:31:39.637612+0000","last_undegraded":"2024-04-17T10:31:39.637612+0000","last_fullsized":"2024-04-17T10:31:39.637612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:53.669724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193705+0000","last_change":"2024-04-17T10:31:38.370365+0000","last_active":"2024-04-17T10:31:40.193705+0000","last_peered":"2024-04-17T10:31:40.193705+0000","last_clean":"2024-04-17T10:31:40.193705+0000","last_became_active":"2024-04-17T10:31:38.370089+0000","last_became_peered":"2024-04-17T10:31:38.370089+0000","last_unstale":"2024-04-17T10:31:40.193705+0000","last_undegraded":"2024-04-17T10:31:40.193705+0000","last_fullsized":"2024-04-17T10:31:40.193705+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:56.298204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365982+0000","last_change":"2024-04-17T10:31:37.354469+0000","last_active":"2024-04-17T10:31:39.365982+0000","last_peered":"2024-04-17T10:31:39.365982+0000","last_clean":"2024-04-17T10:31:39.365982+0000","last_became_active":"2024-04-17T10:31:37.354215+0000","last_became_peered":"2024-04-17T10:31:37.354215+0000","last_unstale":"2024-04-17T10:31:39.365982+0000","last_undegraded":"2024-04-17T10:31:39.365982+0000","last_fullsized":"2024-04-17T10:31:39.365982+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:33.318986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.364135+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.363335+0000","last_became_peered":"2024-04-17T10:31:38.363335+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:04.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.341274+0000","last_change":"2024-04-17T10:31:36.341274+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.341274+0000","last_undegraded":"2024-04-17T10:31:36.341274+0000","last_fullsized":"2024-04-17T10:31:36.341274+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340070+0000","last_change":"2024-04-17T10:31:36.340070+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340070+0000","last_undegraded":"2024-04-17T10:31:36.340070+0000","last_fullsized":"2024-04-17T10:31:36.340070+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366370+0000","last_change":"2024-04-17T10:31:37.359814+0000","last_active":"2024-04-17T10:31:39.366370+0000","last_peered":"2024-04-17T10:31:39.366370+0000","last_clean":"2024-04-17T10:31:39.366370+0000","last_became_active":"2024-04-17T10:31:37.359486+0000","last_became_peered":"2024-04-17T10:31:37.359486+0000","last_unstale":"2024-04-17T10:31:39.366370+0000","last_undegraded":"2024-04-17T10:31:39.366370+0000","last_fullsized":"2024-04-17T10:31:39.366370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:23:28.064562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366322+0000","last_change":"2024-04-17T10:31:38.371429+0000","last_active":"2024-04-17T10:31:39.366322+0000","last_peered":"2024-04-17T10:31:39.366322+0000","last_clean":"2024-04-17T10:31:39.366322+0000","last_became_active":"2024-04-17T10:31:38.371073+0000","last_became_peered":"2024-04-17T10:31:38.371073+0000","last_unstale":"2024-04-17T10:31:39.366322+0000","last_undegraded":"2024-04-17T10:31:39.366322+0000","last_fullsized":"2024-04-17T10:31:39.366322+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:01.747376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.336055+0000","last_change":"2024-04-17T10:31:36.336055+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.336055+0000","last_undegraded":"2024-04-17T10:31:36.336055+0000","last_fullsized":"2024-04-17T10:31:36.336055+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637829+0000","last_change":"2024-04-17T10:31:38.366147+0000","last_active":"2024-04-17T10:31:39.637829+0000","last_peered":"2024-04-17T10:31:39.637829+0000","last_clean":"2024-04-17T10:31:39.637829+0000","last_became_active":"2024-04-17T10:31:38.365889+0000","last_became_peered":"2024-04-17T10:31:38.365889+0000","last_unstale":"2024-04-17T10:31:39.637829+0000","last_undegraded":"2024-04-17T10:31:39.637829+0000","last_fullsized":"2024-04-17T10:31:39.637829+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:18.111634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"95'6","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193733+0000","last_change":"2024-04-17T10:31:37.367318+0000","last_active":"2024-04-17T10:31:40.193733+0000","last_peered":"2024-04-17T10:31:40.193733+0000","last_clean":"2024-04-17T10:31:40.193733+0000","last_became_active":"2024-04-17T10:31:37.367023+0000","last_became_peered":"2024-04-17T10:31:37.367023+0000","last_unstale":"2024-04-17T10:31:40.193733+0000","last_undegraded":"2024-04-17T10:31:40.193733+0000","last_fullsized":"2024-04-17T10:31:40.193733+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:02.024574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637403+0000","last_change":"2024-04-17T10:31:38.362727+0000","last_active":"2024-04-17T10:31:39.637403+0000","last_peered":"2024-04-17T10:31:39.637403+0000","last_clean":"2024-04-17T10:31:39.637403+0000","last_became_active":"2024-04-17T10:31:38.362083+0000","last_became_peered":"2024-04-17T10:31:38.362083+0000","last_unstale":"2024-04-17T10:31:39.637403+0000","last_undegraded":"2024-04-17T10:31:39.637403+0000","last_fullsized":"2024-04-17T10:31:39.637403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:42.796034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340608+0000","last_change":"2024-04-17T10:31:36.340608+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340608+0000","last_undegraded":"2024-04-17T10:31:36.340608+0000","last_fullsized":"2024-04-17T10:31:36.340608+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193763+0000","last_change":"2024-04-17T10:31:37.354839+0000","last_active":"2024-04-17T10:31:40.193763+0000","last_peered":"2024-04-17T10:31:40.193763+0000","last_clean":"2024-04-17T10:31:40.193763+0000","last_became_active":"2024-04-17T10:31:37.354538+0000","last_became_peered":"2024-04-17T10:31:37.354538+0000","last_unstale":"2024-04-17T10:31:40.193763+0000","last_undegraded":"2024-04-17T10:31:40.193763+0000","last_fullsized":"2024-04-17T10:31:40.193763+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:10.283370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366942+0000","last_change":"2024-04-17T10:31:38.368131+0000","last_active":"2024-04-17T10:31:39.366942+0000","last_peered":"2024-04-17T10:31:39.366942+0000","last_clean":"2024-04-17T10:31:39.366942+0000","last_became_active":"2024-04-17T10:31:38.367815+0000","last_became_peered":"2024-04-17T10:31:38.367815+0000","last_unstale":"2024-04-17T10:31:39.366942+0000","last_undegraded":"2024-04-17T10:31:39.366942+0000","last_fullsized":"2024-04-17T10:31:39.366942+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:39.116896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637313+0000","last_change":"2024-04-17T10:31:37.358749+0000","last_active":"2024-04-17T10:31:39.637313+0000","last_peered":"2024-04-17T10:31:39.637313+0000","last_clean":"2024-04-17T10:31:39.637313+0000","last_became_active":"2024-04-17T10:31:37.357922+0000","last_became_peered":"2024-04-17T10:31:37.357922+0000","last_unstale":"2024-04-17T10:31:39.637313+0000","last_undegraded":"2024-04-17T10:31:39.637313+0000","last_fullsized":"2024-04-17T10:31:39.637313+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:23.174223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367510+0000","last_change":"2024-04-17T10:31:38.356263+0000","last_active":"2024-04-17T10:31:39.367510+0000","last_peered":"2024-04-17T10:31:39.367510+0000","last_clean":"2024-04-17T10:31:39.367510+0000","last_became_active":"2024-04-17T10:31:38.355993+0000","last_became_peered":"2024-04-17T10:31:38.355993+0000","last_unstale":"2024-04-17T10:31:39.367510+0000","last_undegraded":"2024-04-17T10:31:39.367510+0000","last_fullsized":"2024-04-17T10:31:39.367510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:55.031187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"95'8","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637287+0000","last_change":"2024-04-17T10:31:37.359691+0000","last_active":"2024-04-17T10:31:39.637287+0000","last_peered":"2024-04-17T10:31:39.637287+0000","last_clean":"2024-04-17T10:31:39.637287+0000","last_became_active":"2024-04-17T10:31:37.359365+0000","last_became_peered":"2024-04-17T10:31:37.359365+0000","last_unstale":"2024-04-17T10:31:39.637287+0000","last_undegraded":"2024-04-17T10:31:39.637287+0000","last_fullsized":"2024-04-17T10:31:39.637287+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:00:19.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.379892+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.379586+0000","last_became_peered":"2024-04-17T10:31:38.379586+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:35.337686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337309+0000","last_change":"2024-04-17T10:31:36.337309+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337309+0000","last_undegraded":"2024-04-17T10:31:36.337309+0000","last_fullsized":"2024-04-17T10:31:36.337309+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340462+0000","last_change":"2024-04-17T10:31:36.340462+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340462+0000","last_undegraded":"2024-04-17T10:31:36.340462+0000","last_fullsized":"2024-04-17T10:31:36.340462+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335517+0000","last_change":"2024-04-17T10:31:36.335517+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335517+0000","last_undegraded":"2024-04-17T10:31:36.335517+0000","last_fullsized":"2024-04-17T10:31:36.335517+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194178+0000","last_change":"2024-04-17T10:31:38.383600+0000","last_active":"2024-04-17T10:31:40.194178+0000","last_peered":"2024-04-17T10:31:40.194178+0000","last_clean":"2024-04-17T10:31:40.194178+0000","last_became_active":"2024-04-17T10:31:38.383354+0000","last_became_peered":"2024-04-17T10:31:38.383354+0000","last_unstale":"2024-04-17T10:31:40.194178+0000","last_undegraded":"2024-04-17T10:31:40.194178+0000","last_fullsized":"2024-04-17T10:31:40.194178+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:25:53.140820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340140+0000","last_change":"2024-04-17T10:31:36.340140+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340140+0000","last_undegraded":"2024-04-17T10:31:36.340140+0000","last_fullsized":"2024-04-17T10:31:36.340140+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366890+0000","last_change":"2024-04-17T10:31:37.365058+0000","last_active":"2024-04-17T10:31:39.366890+0000","last_peered":"2024-04-17T10:31:39.366890+0000","last_clean":"2024-04-17T10:31:39.366890+0000","last_became_active":"2024-04-17T10:31:37.364541+0000","last_became_peered":"2024-04-17T10:31:37.364541+0000","last_unstale":"2024-04-17T10:31:39.366890+0000","last_undegraded":"2024-04-17T10:31:39.366890+0000","last_fullsized":"2024-04-17T10:31:39.366890+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:38.273762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340172+0000","last_change":"2024-04-17T10:31:36.340172+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340172+0000","last_undegraded":"2024-04-17T10:31:36.340172+0000","last_fullsized":"2024-04-17T10:31:36.340172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193864+0000","last_change":"2024-04-17T10:31:38.376121+0000","last_active":"2024-04-17T10:31:40.193864+0000","last_peered":"2024-04-17T10:31:40.193864+0000","last_clean":"2024-04-17T10:31:40.193864+0000","last_became_active":"2024-04-17T10:31:38.375833+0000","last_became_peered":"2024-04-17T10:31:38.375833+0000","last_unstale":"2024-04-17T10:31:40.193864+0000","last_undegraded":"2024-04-17T10:31:40.193864+0000","last_fullsized":"2024-04-17T10:31:40.193864+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:35:43.598662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366769+0000","last_change":"2024-04-17T10:31:37.362595+0000","last_active":"2024-04-17T10:31:39.366769+0000","last_peered":"2024-04-17T10:31:39.366769+0000","last_clean":"2024-04-17T10:31:39.366769+0000","last_became_active":"2024-04-17T10:31:37.362307+0000","last_became_peered":"2024-04-17T10:31:37.362307+0000","last_unstale":"2024-04-17T10:31:39.366769+0000","last_undegraded":"2024-04-17T10:31:39.366769+0000","last_fullsized":"2024-04-17T10:31:39.366769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:15:47.101230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340408+0000","last_change":"2024-04-17T10:31:36.340408+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340408+0000","last_undegraded":"2024-04-17T10:31:36.340408+0000","last_fullsized":"2024-04-17T10:31:36.340408+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194103+0000","last_change":"2024-04-17T10:31:38.386818+0000","last_active":"2024-04-17T10:31:40.194103+0000","last_peered":"2024-04-17T10:31:40.194103+0000","last_clean":"2024-04-17T10:31:40.194103+0000","last_became_active":"2024-04-17T10:31:38.383939+0000","last_became_peered":"2024-04-17T10:31:38.383939+0000","last_unstale":"2024-04-17T10:31:40.194103+0000","last_undegraded":"2024-04-17T10:31:40.194103+0000","last_fullsized":"2024-04-17T10:31:40.194103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:01.834781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335266+0000","last_change":"2024-04-17T10:31:36.335266+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335266+0000","last_undegraded":"2024-04-17T10:31:36.335266+0000","last_fullsized":"2024-04-17T10:31:36.335266+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194023+0000","last_change":"2024-04-17T10:31:38.376661+0000","last_active":"2024-04-17T10:31:40.194023+0000","last_peered":"2024-04-17T10:31:40.194023+0000","last_clean":"2024-04-17T10:31:40.194023+0000","last_became_active":"2024-04-17T10:31:38.376364+0000","last_became_peered":"2024-04-17T10:31:38.376364+0000","last_unstale":"2024-04-17T10:31:40.194023+0000","last_undegraded":"2024-04-17T10:31:40.194023+0000","last_fullsized":"2024-04-17T10:31:40.194023+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:28.335090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.335050+0000","last_change":"2024-04-17T10:31:36.335050+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.335050+0000","last_undegraded":"2024-04-17T10:31:36.335050+0000","last_fullsized":"2024-04-17T10:31:36.335050+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337226+0000","last_change":"2024-04-17T10:31:36.337226+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337226+0000","last_undegraded":"2024-04-17T10:31:36.337226+0000","last_fullsized":"2024-04-17T10:31:36.337226+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366830+0000","last_change":"2024-04-17T10:31:38.384626+0000","last_active":"2024-04-17T10:31:39.366830+0000","last_peered":"2024-04-17T10:31:39.366830+0000","last_clean":"2024-04-17T10:31:39.366830+0000","last_became_active":"2024-04-17T10:31:38.384373+0000","last_became_peered":"2024-04-17T10:31:38.384373+0000","last_unstale":"2024-04-17T10:31:39.366830+0000","last_undegraded":"2024-04-17T10:31:39.366830+0000","last_fullsized":"2024-04-17T10:31:39.366830+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:48.060879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193972+0000","last_change":"2024-04-17T10:31:37.354088+0000","last_active":"2024-04-17T10:31:40.193972+0000","last_peered":"2024-04-17T10:31:40.193972+0000","last_clean":"2024-04-17T10:31:40.193972+0000","last_became_active":"2024-04-17T10:31:37.353817+0000","last_became_peered":"2024-04-17T10:31:37.353817+0000","last_unstale":"2024-04-17T10:31:40.193972+0000","last_undegraded":"2024-04-17T10:31:40.193972+0000","last_fullsized":"2024-04-17T10:31:40.193972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:20.260258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366210+0000","last_change":"2024-04-17T10:31:38.374513+0000","last_active":"2024-04-17T10:31:39.366210+0000","last_peered":"2024-04-17T10:31:39.366210+0000","last_clean":"2024-04-17T10:31:39.366210+0000","last_became_active":"2024-04-17T10:31:38.374235+0000","last_became_peered":"2024-04-17T10:31:38.374235+0000","last_unstale":"2024-04-17T10:31:39.366210+0000","last_undegraded":"2024-04-17T10:31:39.366210+0000","last_fullsized":"2024-04-17T10:31:39.366210+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:20.114227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340005+0000","last_change":"2024-04-17T10:31:36.340005+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340005+0000","last_undegraded":"2024-04-17T10:31:36.340005+0000","last_fullsized":"2024-04-17T10:31:36.340005+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.337484+0000","last_change":"2024-04-17T10:31:36.337484+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.337484+0000","last_undegraded":"2024-04-17T10:31:36.337484+0000","last_fullsized":"2024-04-17T10:31:36.337484+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193954+0000","last_change":"2024-04-17T10:31:38.368340+0000","last_active":"2024-04-17T10:31:40.193954+0000","last_peered":"2024-04-17T10:31:40.193954+0000","last_clean":"2024-04-17T10:31:40.193954+0000","last_became_active":"2024-04-17T10:31:38.368065+0000","last_became_peered":"2024-04-17T10:31:38.368065+0000","last_unstale":"2024-04-17T10:31:40.193954+0000","last_undegraded":"2024-04-17T10:31:40.193954+0000","last_fullsized":"2024-04-17T10:31:40.193954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:38:20.873547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":1,"reported_epoch":93,"state":"creating+peering","last_fresh":"2024-04-17T10:31:36.340808+0000","last_change":"2024-04-17T10:31:36.340808+0000","last_active":"2024-04-17T10:31:36.325538+0000","last_peered":"2024-04-17T10:31:36.325538+0000","last_clean":"2024-04-17T10:31:36.325538+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:36.340808+0000","last_undegraded":"2024-04-17T10:31:36.340808+0000","last_fullsized":"2024-04-17T10:31:36.340808+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:31:37.332052+0000","last_change":"2024-04-17T10:31:37.332052+0000","last_active":"2024-04-17T10:31:37.332052+0000","last_peered":"2024-04-17T10:31:37.332052+0000","last_clean":"2024-04-17T10:31:37.332052+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:31:37.332052+0000","last_undegraded":"2024-04-17T10:31:37.332052+0000","last_fullsized":"2024-04-17T10:31:37.332052+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637594+0000","last_change":"2024-04-17T10:31:37.360694+0000","last_active":"2024-04-17T10:31:39.637594+0000","last_peered":"2024-04-17T10:31:39.637594+0000","last_clean":"2024-04-17T10:31:39.637594+0000","last_became_active":"2024-04-17T10:31:37.360046+0000","last_became_peered":"2024-04-17T10:31:37.360046+0000","last_unstale":"2024-04-17T10:31:39.637594+0000","last_undegraded":"2024-04-17T10:31:39.637594+0000","last_fullsized":"2024-04-17T10:31:39.637594+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:22.070539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-04-17T10:31:36.342331+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:31:36.342331+0000","last_peered":"2024-04-17T10:31:36.342331+0000","last_clean":"2024-04-17T10:31:36.342331+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:31:36.342331+0000","last_undegraded":"2024-04-17T10:31:36.342331+0000","last_fullsized":"2024-04-17T10:31:36.342331+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":1612,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15,"ondisk_log_size":15,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":72,"acting":72,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46772,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325068,"statfs":{"total":96636764160,"available":96588869632,"internally_reserved":0,"allocated":1740800,"data_stored":1121930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46192,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325648,"statfs":{"total":96636764160,"available":96589463552,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574934,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62836,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94309004,"statfs":{"total":96636764160,"available":96572420096,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54508,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317332,"statfs":{"total":96636764160,"available":96580947968,"internally_reserved":0,"allocated":1146880,"data_stored":531562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54544,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317296,"statfs":{"total":96636764160,"available":96580911104,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50392,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321448,"statfs":{"total":96636764160,"available":96585162752,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46228,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325612,"statfs":{"total":96636764160,"available":96589426688,"internally_reserved":0,"allocated":1187840,"data_stored":552158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":1802240,"data_stored":1144774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:31:47.038 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:31:47.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50be000640 1 Processor -- start 2024-04-17T10:31:47.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50be000640 1 -- start start 2024-04-17T10:31:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50be000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50b815dd70 0x7f50b80d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50be000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b80b91d0 0x7f50b80d0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50be000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b80d23b0 con 0x7f50b815d990 2024-04-17T10:31:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50be000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b80d2fc0 con 0x7f50b815dd70 2024-04-17T10:31:47.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50be000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80b79f0 con 0x7f50b80b91d0 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50bd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b80b91d0 0x7f50b80d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50b7e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f50b815dd70 0x7f50b80d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50bd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b80b91d0 0x7f50b80d0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38646/0 (socket says 172.21.15.67:38646) 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50bd600640 1 -- 172.21.15.67:0/2270641319 learned_addr learned my addr 172.21.15.67:0/2270641319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50bd600640 1 -- 172.21.15.67:0/2270641319 >> v2:172.21.15.86:3300/0 conn(0x7f50b815dd70 msgr2=0x7f50b80d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50bd600640 1 --2- 172.21.15.67:0/2270641319 >> v2:172.21.15.86:3300/0 conn(0x7f50b815dd70 0x7f50b80d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50bd600640 1 -- 172.21.15.67:0/2270641319 >> v1:172.21.15.86:6789/0 conn(0x7f50b815d990 legacy=0x7f50b80d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.203+0000 7f50bd600640 1 -- 172.21.15.67:0/2270641319 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b80d2690 con 0x7f50b80b91d0 2024-04-17T10:31:47.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50bd600640 1 --2- 172.21.15.67:0/2270641319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b80b91d0 0x7f50b80d0600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009e50 tx=0x7f50b001ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4702331a26ca62f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50b7400640 1 -- 172.21.15.67:0/2270641319 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00a5900 con 0x7f50b80b91d0 2024-04-17T10:31:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50b7400640 1 -- 172.21.15.67:0/2270641319 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b00a5aa0 con 0x7f50b80b91d0 2024-04-17T10:31:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50b7400640 1 -- 172.21.15.67:0/2270641319 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00ad3f0 con 0x7f50b80b91d0 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 -- 172.21.15.67:0/2270641319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b80b91d0 msgr2=0x7f50b80d0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 --2- 172.21.15.67:0/2270641319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b80b91d0 0x7f50b80d0600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f50b0009e50 tx=0x7f50b001ca30 comp rx=0 tx=0).stop 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 -- 172.21.15.67:0/2270641319 shutdown_connections 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 --2- 172.21.15.67:0/2270641319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b80b91d0 0x7f50b80d0600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 --2- 172.21.15.67:0/2270641319 >> v2:172.21.15.86:3300/0 conn(0x7f50b815dd70 0x7f50b80d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 -- 172.21.15.67:0/2270641319 >> 172.21.15.67:0/2270641319 conn(0x7f50b80b53c0 msgr2=0x7f50b80b57a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 -- 172.21.15.67:0/2270641319 shutdown_connections 2024-04-17T10:31:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.207+0000 7f50be000640 1 -- 172.21.15.67:0/2270641319 wait complete. 2024-04-17T10:31:47.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 Processor -- start 2024-04-17T10:31:47.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 -- start start 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b80b91d0 0x7f50b8155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b815d990 0x7f50b815e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b815dd70 0x7f50b81f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80b7370 con 0x7f50b815dd70 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80b71f0 con 0x7f50b80b91d0 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b80b74f0 con 0x7f50b815d990 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b815dd70 0x7f50b81f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b815dd70 0x7f50b81f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46420/0 (socket says 172.21.15.67:46420) 2024-04-17T10:31:47.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 -- 172.21.15.67:0/1833684133 learned_addr learned my addr 172.21.15.67:0/1833684133 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bcc00640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b80b91d0 0x7f50b8155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50b7e00640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b815d990 0x7f50b815e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 -- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b815d990 msgr2=0x7f50b815e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b815d990 0x7f50b815e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 -- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b80b91d0 msgr2=0x7f50b8155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b80b91d0 0x7f50b8155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 -- 172.21.15.67:0/1833684133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b8156330 con 0x7f50b815dd70 2024-04-17T10:31:47.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50bd600640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b815dd70 0x7f50b81f6510 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f50b00af040 tx=0x7f50b00197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50b4c00640 1 -- 172.21.15.67:0/1833684133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00bc020 con 0x7f50b815dd70 2024-04-17T10:31:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50b4c00640 1 -- 172.21.15.67:0/1833684133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b00b8690 con 0x7f50b815dd70 2024-04-17T10:31:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50b4c00640 1 -- 172.21.15.67:0/1833684133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00b8a80 con 0x7f50b815dd70 2024-04-17T10:31:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b815f1b0 con 0x7f50b815dd70 2024-04-17T10:31:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b80cc0a0 con 0x7f50b815dd70 2024-04-17T10:31:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.211+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b8074dc0 con 0x7f50b815dd70 2024-04-17T10:31:47.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.215+0000 7f50b4c00640 1 -- 172.21.15.67:0/1833684133 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f50b00bf020 con 0x7f50b815dd70 2024-04-17T10:31:47.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.215+0000 7f50b4c00640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5098066f60 0x7f5098069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.215+0000 7f50b4c00640 1 -- 172.21.15.67:0/1833684133 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50b0120200 con 0x7f50b815dd70 2024-04-17T10:31:47.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.215+0000 7f50bcc00640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5098066f60 0x7f5098069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.215+0000 7f50bcc00640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5098066f60 0x7f5098069420 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f50ac001650 tx=0x7f50ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:47.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.219+0000 7f50b4c00640 1 -- 172.21.15.67:0/1833684133 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f50b00c6030 con 0x7f50b815dd70 2024-04-17T10:31:47.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.379+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50b80b7d30 con 0x7f5098066f60 2024-04-17T10:31:47.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.387+0000 7f50b4c00640 1 -- 172.21.15.67:0/1833684133 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252076 (secure 0 0 0) 0x7f50b80b7d30 con 0x7f5098066f60 2024-04-17T10:31:47.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:47.383 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:31:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5098066f60 msgr2=0x7f5098069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5098066f60 0x7f5098069420 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f50ac001650 tx=0x7f50ac009040 comp rx=0 tx=0).stop 2024-04-17T10:31:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b815dd70 msgr2=0x7f50b81f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b815dd70 0x7f50b81f6510 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f50b00af040 tx=0x7f50b00197d0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 shutdown_connections 2024-04-17T10:31:47.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5098066f60 0x7f5098069420 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50b815dd70 0x7f50b81f6510 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50b815d990 0x7f50b815e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 --2- 172.21.15.67:0/1833684133 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50b80b91d0 0x7f50b8155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 >> 172.21.15.67:0/1833684133 conn(0x7f50b80b53c0 msgr2=0x7f50b80ce6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:47.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 shutdown_connections 2024-04-17T10:31:47.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.391+0000 7f50be000640 1 -- 172.21.15.67:0/1833684133 wait complete. 2024-04-17T10:31:47.403 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":301,"stamp":"2024-04-17T10:31:46.235312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":400148,"kb_used_data":10472,"kb_used_omap":12,"kb_used_meta":389555,"kb_avail":754574572,"statfs":{"total":773094113280,"available":772684361728,"internally_reserved":0,"allocated":10723328,"data_stored":5602260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":398904881},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002007"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366379+0000","last_change":"2024-04-17T10:31:38.372177+0000","last_active":"2024-04-17T10:31:39.366379+0000","last_peered":"2024-04-17T10:31:39.366379+0000","last_clean":"2024-04-17T10:31:39.366379+0000","last_became_active":"2024-04-17T10:31:38.371875+0000","last_became_peered":"2024-04-17T10:31:38.371875+0000","last_unstale":"2024-04-17T10:31:39.366379+0000","last_undegraded":"2024-04-17T10:31:39.366379+0000","last_fullsized":"2024-04-17T10:31:39.366379+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:36:29.434877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191330+0000","last_change":"2024-04-17T10:31:38.360282+0000","last_active":"2024-04-17T10:31:40.191330+0000","last_peered":"2024-04-17T10:31:40.191330+0000","last_clean":"2024-04-17T10:31:40.191330+0000","last_became_active":"2024-04-17T10:31:38.359188+0000","last_became_peered":"2024-04-17T10:31:38.359188+0000","last_unstale":"2024-04-17T10:31:40.191330+0000","last_undegraded":"2024-04-17T10:31:40.191330+0000","last_fullsized":"2024-04-17T10:31:40.191330+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:20.114227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366082+0000","last_change":"2024-04-17T10:31:38.362877+0000","last_active":"2024-04-17T10:31:39.366082+0000","last_peered":"2024-04-17T10:31:39.366082+0000","last_clean":"2024-04-17T10:31:39.366082+0000","last_became_active":"2024-04-17T10:31:38.362413+0000","last_became_peered":"2024-04-17T10:31:38.362413+0000","last_unstale":"2024-04-17T10:31:39.366082+0000","last_undegraded":"2024-04-17T10:31:39.366082+0000","last_fullsized":"2024-04-17T10:31:39.366082+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:03.031089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637416+0000","last_change":"2024-04-17T10:31:38.367300+0000","last_active":"2024-04-17T10:31:39.637416+0000","last_peered":"2024-04-17T10:31:39.637416+0000","last_clean":"2024-04-17T10:31:39.637416+0000","last_became_active":"2024-04-17T10:31:38.366715+0000","last_became_peered":"2024-04-17T10:31:38.366715+0000","last_unstale":"2024-04-17T10:31:39.637416+0000","last_undegraded":"2024-04-17T10:31:39.637416+0000","last_fullsized":"2024-04-17T10:31:39.637416+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:02.550932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362931+0000","last_change":"2024-04-17T10:31:38.360848+0000","last_active":"2024-04-17T10:31:39.362931+0000","last_peered":"2024-04-17T10:31:39.362931+0000","last_clean":"2024-04-17T10:31:39.362931+0000","last_became_active":"2024-04-17T10:31:38.360599+0000","last_became_peered":"2024-04-17T10:31:38.360599+0000","last_unstale":"2024-04-17T10:31:39.362931+0000","last_undegraded":"2024-04-17T10:31:39.362931+0000","last_fullsized":"2024-04-17T10:31:39.362931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:21.215719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193707+0000","last_change":"2024-04-17T10:31:38.382652+0000","last_active":"2024-04-17T10:31:40.193707+0000","last_peered":"2024-04-17T10:31:40.193707+0000","last_clean":"2024-04-17T10:31:40.193707+0000","last_became_active":"2024-04-17T10:31:38.382384+0000","last_became_peered":"2024-04-17T10:31:38.382384+0000","last_unstale":"2024-04-17T10:31:40.193707+0000","last_undegraded":"2024-04-17T10:31:40.193707+0000","last_fullsized":"2024-04-17T10:31:40.193707+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:39:20.999198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190662+0000","last_change":"2024-04-17T10:31:38.370687+0000","last_active":"2024-04-17T10:31:40.190662+0000","last_peered":"2024-04-17T10:31:40.190662+0000","last_clean":"2024-04-17T10:31:40.190662+0000","last_became_active":"2024-04-17T10:31:38.370374+0000","last_became_peered":"2024-04-17T10:31:38.370374+0000","last_unstale":"2024-04-17T10:31:40.190662+0000","last_undegraded":"2024-04-17T10:31:40.190662+0000","last_fullsized":"2024-04-17T10:31:40.190662+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:27:21.662980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637175+0000","last_change":"2024-04-17T10:31:38.367243+0000","last_active":"2024-04-17T10:31:39.637175+0000","last_peered":"2024-04-17T10:31:39.637175+0000","last_clean":"2024-04-17T10:31:39.637175+0000","last_became_active":"2024-04-17T10:31:38.366952+0000","last_became_peered":"2024-04-17T10:31:38.366952+0000","last_unstale":"2024-04-17T10:31:39.637175+0000","last_undegraded":"2024-04-17T10:31:39.637175+0000","last_fullsized":"2024-04-17T10:31:39.637175+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:39:24.096194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362834+0000","last_change":"2024-04-17T10:31:38.356863+0000","last_active":"2024-04-17T10:31:39.362834+0000","last_peered":"2024-04-17T10:31:39.362834+0000","last_clean":"2024-04-17T10:31:39.362834+0000","last_became_active":"2024-04-17T10:31:38.356568+0000","last_became_peered":"2024-04-17T10:31:38.356568+0000","last_unstale":"2024-04-17T10:31:39.362834+0000","last_undegraded":"2024-04-17T10:31:39.362834+0000","last_fullsized":"2024-04-17T10:31:39.362834+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:33:56.400881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367386+0000","last_change":"2024-04-17T10:31:38.369001+0000","last_active":"2024-04-17T10:31:39.367386+0000","last_peered":"2024-04-17T10:31:39.367386+0000","last_clean":"2024-04-17T10:31:39.367386+0000","last_became_active":"2024-04-17T10:31:38.367830+0000","last_became_peered":"2024-04-17T10:31:38.367830+0000","last_unstale":"2024-04-17T10:31:39.367386+0000","last_undegraded":"2024-04-17T10:31:39.367386+0000","last_fullsized":"2024-04-17T10:31:39.367386+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:41:52.706957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.363090+0000","last_change":"2024-04-17T10:31:38.383632+0000","last_active":"2024-04-17T10:31:39.363090+0000","last_peered":"2024-04-17T10:31:39.363090+0000","last_clean":"2024-04-17T10:31:39.363090+0000","last_became_active":"2024-04-17T10:31:38.383405+0000","last_became_peered":"2024-04-17T10:31:38.383405+0000","last_unstale":"2024-04-17T10:31:39.363090+0000","last_undegraded":"2024-04-17T10:31:39.363090+0000","last_fullsized":"2024-04-17T10:31:39.363090+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:34.325501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367226+0000","last_change":"2024-04-17T10:31:38.357758+0000","last_active":"2024-04-17T10:31:39.367226+0000","last_peered":"2024-04-17T10:31:39.367226+0000","last_clean":"2024-04-17T10:31:39.367226+0000","last_became_active":"2024-04-17T10:31:38.357482+0000","last_became_peered":"2024-04-17T10:31:38.357482+0000","last_unstale":"2024-04-17T10:31:39.367226+0000","last_undegraded":"2024-04-17T10:31:39.367226+0000","last_fullsized":"2024-04-17T10:31:39.367226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:57:48.241446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637619+0000","last_change":"2024-04-17T10:31:38.366467+0000","last_active":"2024-04-17T10:31:39.637619+0000","last_peered":"2024-04-17T10:31:39.637619+0000","last_clean":"2024-04-17T10:31:39.637619+0000","last_became_active":"2024-04-17T10:31:38.366066+0000","last_became_peered":"2024-04-17T10:31:38.366066+0000","last_unstale":"2024-04-17T10:31:39.637619+0000","last_undegraded":"2024-04-17T10:31:39.637619+0000","last_fullsized":"2024-04-17T10:31:39.637619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:08.729085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362464+0000","last_change":"2024-04-17T10:31:38.368666+0000","last_active":"2024-04-17T10:31:39.362464+0000","last_peered":"2024-04-17T10:31:39.362464+0000","last_clean":"2024-04-17T10:31:39.362464+0000","last_became_active":"2024-04-17T10:31:38.368343+0000","last_became_peered":"2024-04-17T10:31:38.368343+0000","last_unstale":"2024-04-17T10:31:39.362464+0000","last_undegraded":"2024-04-17T10:31:39.362464+0000","last_fullsized":"2024-04-17T10:31:39.362464+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:04.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637251+0000","last_change":"2024-04-17T10:31:38.363077+0000","last_active":"2024-04-17T10:31:39.637251+0000","last_peered":"2024-04-17T10:31:39.637251+0000","last_clean":"2024-04-17T10:31:39.637251+0000","last_became_active":"2024-04-17T10:31:38.362827+0000","last_became_peered":"2024-04-17T10:31:38.362827+0000","last_unstale":"2024-04-17T10:31:39.637251+0000","last_undegraded":"2024-04-17T10:31:39.637251+0000","last_fullsized":"2024-04-17T10:31:39.637251+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:56:20.141939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193544+0000","last_change":"2024-04-17T10:31:38.390689+0000","last_active":"2024-04-17T10:31:40.193544+0000","last_peered":"2024-04-17T10:31:40.193544+0000","last_clean":"2024-04-17T10:31:40.193544+0000","last_became_active":"2024-04-17T10:31:38.389673+0000","last_became_peered":"2024-04-17T10:31:38.389673+0000","last_unstale":"2024-04-17T10:31:40.193544+0000","last_undegraded":"2024-04-17T10:31:40.193544+0000","last_fullsized":"2024-04-17T10:31:40.193544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:05:33.699853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194163+0000","last_change":"2024-04-17T10:31:38.386404+0000","last_active":"2024-04-17T10:31:40.194163+0000","last_peered":"2024-04-17T10:31:40.194163+0000","last_clean":"2024-04-17T10:31:40.194163+0000","last_became_active":"2024-04-17T10:31:38.383973+0000","last_became_peered":"2024-04-17T10:31:38.383973+0000","last_unstale":"2024-04-17T10:31:40.194163+0000","last_undegraded":"2024-04-17T10:31:40.194163+0000","last_fullsized":"2024-04-17T10:31:40.194163+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:37:12.821691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366183+0000","last_change":"2024-04-17T10:31:38.362736+0000","last_active":"2024-04-17T10:31:39.366183+0000","last_peered":"2024-04-17T10:31:39.366183+0000","last_clean":"2024-04-17T10:31:39.366183+0000","last_became_active":"2024-04-17T10:31:38.362405+0000","last_became_peered":"2024-04-17T10:31:38.362405+0000","last_unstale":"2024-04-17T10:31:39.366183+0000","last_undegraded":"2024-04-17T10:31:39.366183+0000","last_fullsized":"2024-04-17T10:31:39.366183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:55.770687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362752+0000","last_change":"2024-04-17T10:31:38.367019+0000","last_active":"2024-04-17T10:31:39.362752+0000","last_peered":"2024-04-17T10:31:39.362752+0000","last_clean":"2024-04-17T10:31:39.362752+0000","last_became_active":"2024-04-17T10:31:38.366739+0000","last_became_peered":"2024-04-17T10:31:38.366739+0000","last_unstale":"2024-04-17T10:31:39.362752+0000","last_undegraded":"2024-04-17T10:31:39.362752+0000","last_fullsized":"2024-04-17T10:31:39.362752+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:20.424487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.367062+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.366356+0000","last_became_peered":"2024-04-17T10:31:38.366356+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:34.604484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362771+0000","last_change":"2024-04-17T10:31:38.355788+0000","last_active":"2024-04-17T10:31:39.362771+0000","last_peered":"2024-04-17T10:31:39.362771+0000","last_clean":"2024-04-17T10:31:39.362771+0000","last_became_active":"2024-04-17T10:31:38.355253+0000","last_became_peered":"2024-04-17T10:31:38.355253+0000","last_unstale":"2024-04-17T10:31:39.362771+0000","last_undegraded":"2024-04-17T10:31:39.362771+0000","last_fullsized":"2024-04-17T10:31:39.362771+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:00:59.930403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362269+0000","last_change":"2024-04-17T10:31:38.371722+0000","last_active":"2024-04-17T10:31:39.362269+0000","last_peered":"2024-04-17T10:31:39.362269+0000","last_clean":"2024-04-17T10:31:39.362269+0000","last_became_active":"2024-04-17T10:31:38.371348+0000","last_became_peered":"2024-04-17T10:31:38.371348+0000","last_unstale":"2024-04-17T10:31:39.362269+0000","last_undegraded":"2024-04-17T10:31:39.362269+0000","last_fullsized":"2024-04-17T10:31:39.362269+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:02.550932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191715+0000","last_change":"2024-04-17T10:31:38.370541+0000","last_active":"2024-04-17T10:31:40.191715+0000","last_peered":"2024-04-17T10:31:40.191715+0000","last_clean":"2024-04-17T10:31:40.191715+0000","last_became_active":"2024-04-17T10:31:38.370228+0000","last_became_peered":"2024-04-17T10:31:38.370228+0000","last_unstale":"2024-04-17T10:31:40.191715+0000","last_undegraded":"2024-04-17T10:31:40.191715+0000","last_fullsized":"2024-04-17T10:31:40.191715+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:56:02.550932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365841+0000","last_change":"2024-04-17T10:31:38.363969+0000","last_active":"2024-04-17T10:31:39.365841+0000","last_peered":"2024-04-17T10:31:39.365841+0000","last_clean":"2024-04-17T10:31:39.365841+0000","last_became_active":"2024-04-17T10:31:38.363731+0000","last_became_peered":"2024-04-17T10:31:38.363731+0000","last_unstale":"2024-04-17T10:31:39.365841+0000","last_undegraded":"2024-04-17T10:31:39.365841+0000","last_fullsized":"2024-04-17T10:31:39.365841+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:25.031678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191273+0000","last_change":"2024-04-17T10:31:38.361393+0000","last_active":"2024-04-17T10:31:40.191273+0000","last_peered":"2024-04-17T10:31:40.191273+0000","last_clean":"2024-04-17T10:31:40.191273+0000","last_became_active":"2024-04-17T10:31:38.361054+0000","last_became_peered":"2024-04-17T10:31:38.361054+0000","last_unstale":"2024-04-17T10:31:40.191273+0000","last_undegraded":"2024-04-17T10:31:40.191273+0000","last_fullsized":"2024-04-17T10:31:40.191273+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:03.031089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362759+0000","last_change":"2024-04-17T10:31:38.371774+0000","last_active":"2024-04-17T10:31:39.362759+0000","last_peered":"2024-04-17T10:31:39.362759+0000","last_clean":"2024-04-17T10:31:39.362759+0000","last_became_active":"2024-04-17T10:31:38.371503+0000","last_became_peered":"2024-04-17T10:31:38.371503+0000","last_unstale":"2024-04-17T10:31:39.362759+0000","last_undegraded":"2024-04-17T10:31:39.362759+0000","last_fullsized":"2024-04-17T10:31:39.362759+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:18.111634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367325+0000","last_change":"2024-04-17T10:31:38.364443+0000","last_active":"2024-04-17T10:31:39.367325+0000","last_peered":"2024-04-17T10:31:39.367325+0000","last_clean":"2024-04-17T10:31:39.367325+0000","last_became_active":"2024-04-17T10:31:38.364134+0000","last_became_peered":"2024-04-17T10:31:38.364134+0000","last_unstale":"2024-04-17T10:31:39.367325+0000","last_undegraded":"2024-04-17T10:31:39.367325+0000","last_fullsized":"2024-04-17T10:31:39.367325+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:13:32.599449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637675+0000","last_change":"2024-04-17T10:31:38.367434+0000","last_active":"2024-04-17T10:31:39.637675+0000","last_peered":"2024-04-17T10:31:39.637675+0000","last_clean":"2024-04-17T10:31:39.637675+0000","last_became_active":"2024-04-17T10:31:38.366004+0000","last_became_peered":"2024-04-17T10:31:38.366004+0000","last_unstale":"2024-04-17T10:31:39.637675+0000","last_undegraded":"2024-04-17T10:31:39.637675+0000","last_fullsized":"2024-04-17T10:31:39.637675+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:27:20.424487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190799+0000","last_change":"2024-04-17T10:31:38.361008+0000","last_active":"2024-04-17T10:31:40.190799+0000","last_peered":"2024-04-17T10:31:40.190799+0000","last_clean":"2024-04-17T10:31:40.190799+0000","last_became_active":"2024-04-17T10:31:38.360751+0000","last_became_peered":"2024-04-17T10:31:38.360751+0000","last_unstale":"2024-04-17T10:31:40.190799+0000","last_undegraded":"2024-04-17T10:31:40.190799+0000","last_fullsized":"2024-04-17T10:31:40.190799+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:26:02.302821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362654+0000","last_change":"2024-04-17T10:31:38.356933+0000","last_active":"2024-04-17T10:31:39.362654+0000","last_peered":"2024-04-17T10:31:39.362654+0000","last_clean":"2024-04-17T10:31:39.362654+0000","last_became_active":"2024-04-17T10:31:38.356626+0000","last_became_peered":"2024-04-17T10:31:38.356626+0000","last_unstale":"2024-04-17T10:31:39.362654+0000","last_undegraded":"2024-04-17T10:31:39.362654+0000","last_fullsized":"2024-04-17T10:31:39.362654+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:11:08.729085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191310+0000","last_change":"2024-04-17T10:31:38.371827+0000","last_active":"2024-04-17T10:31:40.191310+0000","last_peered":"2024-04-17T10:31:40.191310+0000","last_clean":"2024-04-17T10:31:40.191310+0000","last_became_active":"2024-04-17T10:31:38.371048+0000","last_became_peered":"2024-04-17T10:31:38.371048+0000","last_unstale":"2024-04-17T10:31:40.191310+0000","last_undegraded":"2024-04-17T10:31:40.191310+0000","last_fullsized":"2024-04-17T10:31:40.191310+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:55.770687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193672+0000","last_change":"2024-04-17T10:31:38.387676+0000","last_active":"2024-04-17T10:31:40.193672+0000","last_peered":"2024-04-17T10:31:40.193672+0000","last_clean":"2024-04-17T10:31:40.193672+0000","last_became_active":"2024-04-17T10:31:38.380682+0000","last_became_peered":"2024-04-17T10:31:38.380682+0000","last_unstale":"2024-04-17T10:31:40.193672+0000","last_undegraded":"2024-04-17T10:31:40.193672+0000","last_fullsized":"2024-04-17T10:31:40.193672+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:35:59.714995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193655+0000","last_change":"2024-04-17T10:31:38.379789+0000","last_active":"2024-04-17T10:31:40.193655+0000","last_peered":"2024-04-17T10:31:40.193655+0000","last_clean":"2024-04-17T10:31:40.193655+0000","last_became_active":"2024-04-17T10:31:38.379237+0000","last_became_peered":"2024-04-17T10:31:38.379237+0000","last_unstale":"2024-04-17T10:31:40.193655+0000","last_undegraded":"2024-04-17T10:31:40.193655+0000","last_fullsized":"2024-04-17T10:31:40.193655+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:01.930369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"95'4","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191767+0000","last_change":"2024-04-17T10:31:37.366608+0000","last_active":"2024-04-17T10:31:40.191767+0000","last_peered":"2024-04-17T10:31:40.191767+0000","last_clean":"2024-04-17T10:31:40.191767+0000","last_became_active":"2024-04-17T10:31:37.366320+0000","last_became_peered":"2024-04-17T10:31:37.366320+0000","last_unstale":"2024-04-17T10:31:40.191767+0000","last_undegraded":"2024-04-17T10:31:40.191767+0000","last_fullsized":"2024-04-17T10:31:40.191767+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:55.454548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.386544+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.380593+0000","last_became_peered":"2024-04-17T10:31:38.380593+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:53:17.992647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362834+0000","last_change":"2024-04-17T10:31:37.352071+0000","last_active":"2024-04-17T10:31:39.362834+0000","last_peered":"2024-04-17T10:31:39.362834+0000","last_clean":"2024-04-17T10:31:39.362834+0000","last_became_active":"2024-04-17T10:31:37.351761+0000","last_became_peered":"2024-04-17T10:31:37.351761+0000","last_unstale":"2024-04-17T10:31:39.362834+0000","last_undegraded":"2024-04-17T10:31:39.362834+0000","last_fullsized":"2024-04-17T10:31:39.362834+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:00:19.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190458+0000","last_change":"2024-04-17T10:31:38.372081+0000","last_active":"2024-04-17T10:31:40.190458+0000","last_peered":"2024-04-17T10:31:40.190458+0000","last_clean":"2024-04-17T10:31:40.190458+0000","last_became_active":"2024-04-17T10:31:38.371754+0000","last_became_peered":"2024-04-17T10:31:38.371754+0000","last_unstale":"2024-04-17T10:31:40.190458+0000","last_undegraded":"2024-04-17T10:31:40.190458+0000","last_fullsized":"2024-04-17T10:31:40.190458+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:44:16.199018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190443+0000","last_change":"2024-04-17T10:31:37.360095+0000","last_active":"2024-04-17T10:31:40.190443+0000","last_peered":"2024-04-17T10:31:40.190443+0000","last_clean":"2024-04-17T10:31:40.190443+0000","last_became_active":"2024-04-17T10:31:37.359784+0000","last_became_peered":"2024-04-17T10:31:37.359784+0000","last_unstale":"2024-04-17T10:31:40.190443+0000","last_undegraded":"2024-04-17T10:31:40.190443+0000","last_fullsized":"2024-04-17T10:31:40.190443+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:48:46.641731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194225+0000","last_change":"2024-04-17T10:31:38.383915+0000","last_active":"2024-04-17T10:31:40.194225+0000","last_peered":"2024-04-17T10:31:40.194225+0000","last_clean":"2024-04-17T10:31:40.194225+0000","last_became_active":"2024-04-17T10:31:38.382791+0000","last_became_peered":"2024-04-17T10:31:38.382791+0000","last_unstale":"2024-04-17T10:31:40.194225+0000","last_undegraded":"2024-04-17T10:31:40.194225+0000","last_fullsized":"2024-04-17T10:31:40.194225+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:12.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191745+0000","last_change":"2024-04-17T10:31:37.361810+0000","last_active":"2024-04-17T10:31:40.191745+0000","last_peered":"2024-04-17T10:31:40.191745+0000","last_clean":"2024-04-17T10:31:40.191745+0000","last_became_active":"2024-04-17T10:31:37.361515+0000","last_became_peered":"2024-04-17T10:31:37.361515+0000","last_unstale":"2024-04-17T10:31:40.191745+0000","last_undegraded":"2024-04-17T10:31:40.191745+0000","last_fullsized":"2024-04-17T10:31:40.191745+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:13.708971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637195+0000","last_change":"2024-04-17T10:31:38.366551+0000","last_active":"2024-04-17T10:31:39.637195+0000","last_peered":"2024-04-17T10:31:39.637195+0000","last_clean":"2024-04-17T10:31:39.637195+0000","last_became_active":"2024-04-17T10:31:38.366244+0000","last_became_peered":"2024-04-17T10:31:38.366244+0000","last_unstale":"2024-04-17T10:31:39.637195+0000","last_undegraded":"2024-04-17T10:31:39.637195+0000","last_fullsized":"2024-04-17T10:31:39.637195+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:37.769462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362852+0000","last_change":"2024-04-17T10:31:37.363004+0000","last_active":"2024-04-17T10:31:39.362852+0000","last_peered":"2024-04-17T10:31:39.362852+0000","last_clean":"2024-04-17T10:31:39.362852+0000","last_became_active":"2024-04-17T10:31:37.362507+0000","last_became_peered":"2024-04-17T10:31:37.362507+0000","last_unstale":"2024-04-17T10:31:39.362852+0000","last_undegraded":"2024-04-17T10:31:39.362852+0000","last_fullsized":"2024-04-17T10:31:39.362852+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:53.669724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191846+0000","last_change":"2024-04-17T10:31:38.380054+0000","last_active":"2024-04-17T10:31:40.191846+0000","last_peered":"2024-04-17T10:31:40.191846+0000","last_clean":"2024-04-17T10:31:40.191846+0000","last_became_active":"2024-04-17T10:31:38.379785+0000","last_became_peered":"2024-04-17T10:31:38.379785+0000","last_unstale":"2024-04-17T10:31:40.191846+0000","last_undegraded":"2024-04-17T10:31:40.191846+0000","last_fullsized":"2024-04-17T10:31:40.191846+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:04.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637612+0000","last_change":"2024-04-17T10:31:37.366420+0000","last_active":"2024-04-17T10:31:39.637612+0000","last_peered":"2024-04-17T10:31:39.637612+0000","last_clean":"2024-04-17T10:31:39.637612+0000","last_became_active":"2024-04-17T10:31:37.365712+0000","last_became_peered":"2024-04-17T10:31:37.365712+0000","last_unstale":"2024-04-17T10:31:39.637612+0000","last_undegraded":"2024-04-17T10:31:39.637612+0000","last_fullsized":"2024-04-17T10:31:39.637612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:14:53.669724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193705+0000","last_change":"2024-04-17T10:31:38.370365+0000","last_active":"2024-04-17T10:31:40.193705+0000","last_peered":"2024-04-17T10:31:40.193705+0000","last_clean":"2024-04-17T10:31:40.193705+0000","last_became_active":"2024-04-17T10:31:38.370089+0000","last_became_peered":"2024-04-17T10:31:38.370089+0000","last_unstale":"2024-04-17T10:31:40.193705+0000","last_undegraded":"2024-04-17T10:31:40.193705+0000","last_fullsized":"2024-04-17T10:31:40.193705+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:56.298204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.365982+0000","last_change":"2024-04-17T10:31:37.354469+0000","last_active":"2024-04-17T10:31:39.365982+0000","last_peered":"2024-04-17T10:31:39.365982+0000","last_clean":"2024-04-17T10:31:39.365982+0000","last_became_active":"2024-04-17T10:31:37.354215+0000","last_became_peered":"2024-04-17T10:31:37.354215+0000","last_unstale":"2024-04-17T10:31:39.365982+0000","last_undegraded":"2024-04-17T10:31:39.365982+0000","last_fullsized":"2024-04-17T10:31:39.365982+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:33.318986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637473+0000","last_change":"2024-04-17T10:31:38.364135+0000","last_active":"2024-04-17T10:31:39.637473+0000","last_peered":"2024-04-17T10:31:39.637473+0000","last_clean":"2024-04-17T10:31:39.637473+0000","last_became_active":"2024-04-17T10:31:38.363335+0000","last_became_peered":"2024-04-17T10:31:38.363335+0000","last_unstale":"2024-04-17T10:31:39.637473+0000","last_undegraded":"2024-04-17T10:31:39.637473+0000","last_fullsized":"2024-04-17T10:31:39.637473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:04.426251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190431+0000","last_change":"2024-04-17T10:31:37.364093+0000","last_active":"2024-04-17T10:31:40.190431+0000","last_peered":"2024-04-17T10:31:40.190431+0000","last_clean":"2024-04-17T10:31:40.190431+0000","last_became_active":"2024-04-17T10:31:37.363819+0000","last_became_peered":"2024-04-17T10:31:37.363819+0000","last_unstale":"2024-04-17T10:31:40.190431+0000","last_undegraded":"2024-04-17T10:31:40.190431+0000","last_fullsized":"2024-04-17T10:31:40.190431+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:24.025164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362642+0000","last_change":"2024-04-17T10:31:38.354652+0000","last_active":"2024-04-17T10:31:39.362642+0000","last_peered":"2024-04-17T10:31:39.362642+0000","last_clean":"2024-04-17T10:31:39.362642+0000","last_became_active":"2024-04-17T10:31:38.354360+0000","last_became_peered":"2024-04-17T10:31:38.354360+0000","last_unstale":"2024-04-17T10:31:39.362642+0000","last_undegraded":"2024-04-17T10:31:39.362642+0000","last_fullsized":"2024-04-17T10:31:39.362642+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:34.604484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191366+0000","last_change":"2024-04-17T10:31:37.362840+0000","last_active":"2024-04-17T10:31:40.191366+0000","last_peered":"2024-04-17T10:31:40.191366+0000","last_clean":"2024-04-17T10:31:40.191366+0000","last_became_active":"2024-04-17T10:31:37.362491+0000","last_became_peered":"2024-04-17T10:31:37.362491+0000","last_unstale":"2024-04-17T10:31:40.191366+0000","last_undegraded":"2024-04-17T10:31:40.191366+0000","last_fullsized":"2024-04-17T10:31:40.191366+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:23:28.064562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190562+0000","last_change":"2024-04-17T10:31:38.360007+0000","last_active":"2024-04-17T10:31:40.190562+0000","last_peered":"2024-04-17T10:31:40.190562+0000","last_clean":"2024-04-17T10:31:40.190562+0000","last_became_active":"2024-04-17T10:31:38.359718+0000","last_became_peered":"2024-04-17T10:31:38.359718+0000","last_unstale":"2024-04-17T10:31:40.190562+0000","last_undegraded":"2024-04-17T10:31:40.190562+0000","last_fullsized":"2024-04-17T10:31:40.190562+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:01:37.083960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366370+0000","last_change":"2024-04-17T10:31:37.359814+0000","last_active":"2024-04-17T10:31:39.366370+0000","last_peered":"2024-04-17T10:31:39.366370+0000","last_clean":"2024-04-17T10:31:39.366370+0000","last_became_active":"2024-04-17T10:31:37.359486+0000","last_became_peered":"2024-04-17T10:31:37.359486+0000","last_unstale":"2024-04-17T10:31:39.366370+0000","last_undegraded":"2024-04-17T10:31:39.366370+0000","last_fullsized":"2024-04-17T10:31:39.366370+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:23:28.064562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366322+0000","last_change":"2024-04-17T10:31:38.371429+0000","last_active":"2024-04-17T10:31:39.366322+0000","last_peered":"2024-04-17T10:31:39.366322+0000","last_clean":"2024-04-17T10:31:39.366322+0000","last_became_active":"2024-04-17T10:31:38.371073+0000","last_became_peered":"2024-04-17T10:31:38.371073+0000","last_unstale":"2024-04-17T10:31:39.366322+0000","last_undegraded":"2024-04-17T10:31:39.366322+0000","last_fullsized":"2024-04-17T10:31:39.366322+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:01.747376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"95'3","reported_seq":17,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362938+0000","last_change":"2024-04-17T10:31:37.350278+0000","last_active":"2024-04-17T10:31:39.362938+0000","last_peered":"2024-04-17T10:31:39.362938+0000","last_clean":"2024-04-17T10:31:39.362938+0000","last_became_active":"2024-04-17T10:31:37.349866+0000","last_became_peered":"2024-04-17T10:31:37.349866+0000","last_unstale":"2024-04-17T10:31:39.362938+0000","last_undegraded":"2024-04-17T10:31:39.362938+0000","last_fullsized":"2024-04-17T10:31:39.362938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:23:44.180915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637829+0000","last_change":"2024-04-17T10:31:38.366147+0000","last_active":"2024-04-17T10:31:39.637829+0000","last_peered":"2024-04-17T10:31:39.637829+0000","last_clean":"2024-04-17T10:31:39.637829+0000","last_became_active":"2024-04-17T10:31:38.365889+0000","last_became_peered":"2024-04-17T10:31:38.365889+0000","last_unstale":"2024-04-17T10:31:39.637829+0000","last_undegraded":"2024-04-17T10:31:39.637829+0000","last_fullsized":"2024-04-17T10:31:39.637829+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:37:18.111634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"95'6","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193733+0000","last_change":"2024-04-17T10:31:37.367318+0000","last_active":"2024-04-17T10:31:40.193733+0000","last_peered":"2024-04-17T10:31:40.193733+0000","last_clean":"2024-04-17T10:31:40.193733+0000","last_became_active":"2024-04-17T10:31:37.367023+0000","last_became_peered":"2024-04-17T10:31:37.367023+0000","last_unstale":"2024-04-17T10:31:40.193733+0000","last_undegraded":"2024-04-17T10:31:40.193733+0000","last_fullsized":"2024-04-17T10:31:40.193733+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:02.024574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637403+0000","last_change":"2024-04-17T10:31:38.362727+0000","last_active":"2024-04-17T10:31:39.637403+0000","last_peered":"2024-04-17T10:31:39.637403+0000","last_clean":"2024-04-17T10:31:39.637403+0000","last_became_active":"2024-04-17T10:31:38.362083+0000","last_became_peered":"2024-04-17T10:31:38.362083+0000","last_unstale":"2024-04-17T10:31:39.637403+0000","last_undegraded":"2024-04-17T10:31:39.637403+0000","last_fullsized":"2024-04-17T10:31:39.637403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:42.796034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190854+0000","last_change":"2024-04-17T10:31:37.361305+0000","last_active":"2024-04-17T10:31:40.190854+0000","last_peered":"2024-04-17T10:31:40.190854+0000","last_clean":"2024-04-17T10:31:40.190854+0000","last_became_active":"2024-04-17T10:31:37.360974+0000","last_became_peered":"2024-04-17T10:31:37.360974+0000","last_unstale":"2024-04-17T10:31:40.190854+0000","last_undegraded":"2024-04-17T10:31:40.190854+0000","last_fullsized":"2024-04-17T10:31:40.190854+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:10.283370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362191+0000","last_change":"2024-04-17T10:31:38.362874+0000","last_active":"2024-04-17T10:31:39.362191+0000","last_peered":"2024-04-17T10:31:39.362191+0000","last_clean":"2024-04-17T10:31:39.362191+0000","last_became_active":"2024-04-17T10:31:38.362178+0000","last_became_peered":"2024-04-17T10:31:38.362178+0000","last_unstale":"2024-04-17T10:31:39.362191+0000","last_undegraded":"2024-04-17T10:31:39.362191+0000","last_fullsized":"2024-04-17T10:31:39.362191+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:42.796034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193763+0000","last_change":"2024-04-17T10:31:37.354839+0000","last_active":"2024-04-17T10:31:40.193763+0000","last_peered":"2024-04-17T10:31:40.193763+0000","last_clean":"2024-04-17T10:31:40.193763+0000","last_became_active":"2024-04-17T10:31:37.354538+0000","last_became_peered":"2024-04-17T10:31:37.354538+0000","last_unstale":"2024-04-17T10:31:40.193763+0000","last_undegraded":"2024-04-17T10:31:40.193763+0000","last_fullsized":"2024-04-17T10:31:40.193763+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:13:10.283370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366942+0000","last_change":"2024-04-17T10:31:38.368131+0000","last_active":"2024-04-17T10:31:39.366942+0000","last_peered":"2024-04-17T10:31:39.366942+0000","last_clean":"2024-04-17T10:31:39.366942+0000","last_became_active":"2024-04-17T10:31:38.367815+0000","last_became_peered":"2024-04-17T10:31:38.367815+0000","last_unstale":"2024-04-17T10:31:39.366942+0000","last_undegraded":"2024-04-17T10:31:39.366942+0000","last_fullsized":"2024-04-17T10:31:39.366942+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:29:39.116896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637313+0000","last_change":"2024-04-17T10:31:37.358749+0000","last_active":"2024-04-17T10:31:39.637313+0000","last_peered":"2024-04-17T10:31:39.637313+0000","last_clean":"2024-04-17T10:31:39.637313+0000","last_became_active":"2024-04-17T10:31:37.357922+0000","last_became_peered":"2024-04-17T10:31:37.357922+0000","last_unstale":"2024-04-17T10:31:39.637313+0000","last_undegraded":"2024-04-17T10:31:39.637313+0000","last_fullsized":"2024-04-17T10:31:39.637313+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:23.174223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.367510+0000","last_change":"2024-04-17T10:31:38.356263+0000","last_active":"2024-04-17T10:31:39.367510+0000","last_peered":"2024-04-17T10:31:39.367510+0000","last_clean":"2024-04-17T10:31:39.367510+0000","last_became_active":"2024-04-17T10:31:38.355993+0000","last_became_peered":"2024-04-17T10:31:38.355993+0000","last_unstale":"2024-04-17T10:31:39.367510+0000","last_undegraded":"2024-04-17T10:31:39.367510+0000","last_fullsized":"2024-04-17T10:31:39.367510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:52:55.031187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"95'8","reported_seq":22,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637287+0000","last_change":"2024-04-17T10:31:37.359691+0000","last_active":"2024-04-17T10:31:39.637287+0000","last_peered":"2024-04-17T10:31:39.637287+0000","last_clean":"2024-04-17T10:31:39.637287+0000","last_became_active":"2024-04-17T10:31:37.359365+0000","last_became_peered":"2024-04-17T10:31:37.359365+0000","last_unstale":"2024-04-17T10:31:39.637287+0000","last_undegraded":"2024-04-17T10:31:39.637287+0000","last_fullsized":"2024-04-17T10:31:39.637287+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:00:19.211759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193968+0000","last_change":"2024-04-17T10:31:38.379892+0000","last_active":"2024-04-17T10:31:40.193968+0000","last_peered":"2024-04-17T10:31:40.193968+0000","last_clean":"2024-04-17T10:31:40.193968+0000","last_became_active":"2024-04-17T10:31:38.379586+0000","last_became_peered":"2024-04-17T10:31:38.379586+0000","last_unstale":"2024-04-17T10:31:40.193968+0000","last_undegraded":"2024-04-17T10:31:40.193968+0000","last_fullsized":"2024-04-17T10:31:40.193968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:27:35.337686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191854+0000","last_change":"2024-04-17T10:31:37.354389+0000","last_active":"2024-04-17T10:31:40.191854+0000","last_peered":"2024-04-17T10:31:40.191854+0000","last_clean":"2024-04-17T10:31:40.191854+0000","last_became_active":"2024-04-17T10:31:37.354106+0000","last_became_peered":"2024-04-17T10:31:37.354106+0000","last_unstale":"2024-04-17T10:31:40.191854+0000","last_undegraded":"2024-04-17T10:31:40.191854+0000","last_fullsized":"2024-04-17T10:31:40.191854+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:28.066742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191656+0000","last_change":"2024-04-17T10:31:38.358095+0000","last_active":"2024-04-17T10:31:40.191656+0000","last_peered":"2024-04-17T10:31:40.191656+0000","last_clean":"2024-04-17T10:31:40.191656+0000","last_became_active":"2024-04-17T10:31:38.357794+0000","last_became_peered":"2024-04-17T10:31:38.357794+0000","last_unstale":"2024-04-17T10:31:40.191656+0000","last_undegraded":"2024-04-17T10:31:40.191656+0000","last_fullsized":"2024-04-17T10:31:40.191656+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:56:20.141939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190481+0000","last_change":"2024-04-17T10:31:37.356298+0000","last_active":"2024-04-17T10:31:40.190481+0000","last_peered":"2024-04-17T10:31:40.190481+0000","last_clean":"2024-04-17T10:31:40.190481+0000","last_became_active":"2024-04-17T10:31:37.356034+0000","last_became_peered":"2024-04-17T10:31:37.356034+0000","last_unstale":"2024-04-17T10:31:40.190481+0000","last_undegraded":"2024-04-17T10:31:40.190481+0000","last_fullsized":"2024-04-17T10:31:40.190481+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:20.260258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362968+0000","last_change":"2024-04-17T10:31:38.352270+0000","last_active":"2024-04-17T10:31:39.362968+0000","last_peered":"2024-04-17T10:31:39.362968+0000","last_clean":"2024-04-17T10:31:39.362968+0000","last_became_active":"2024-04-17T10:31:38.351904+0000","last_became_peered":"2024-04-17T10:31:38.351904+0000","last_unstale":"2024-04-17T10:31:39.362968+0000","last_undegraded":"2024-04-17T10:31:39.362968+0000","last_fullsized":"2024-04-17T10:31:39.362968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:35:01.907394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.363062+0000","last_change":"2024-04-17T10:31:37.357122+0000","last_active":"2024-04-17T10:31:39.363062+0000","last_peered":"2024-04-17T10:31:39.363062+0000","last_clean":"2024-04-17T10:31:39.363062+0000","last_became_active":"2024-04-17T10:31:37.356832+0000","last_became_peered":"2024-04-17T10:31:37.356832+0000","last_unstale":"2024-04-17T10:31:39.363062+0000","last_undegraded":"2024-04-17T10:31:39.363062+0000","last_fullsized":"2024-04-17T10:31:39.363062+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:30:23.174223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194178+0000","last_change":"2024-04-17T10:31:38.383600+0000","last_active":"2024-04-17T10:31:40.194178+0000","last_peered":"2024-04-17T10:31:40.194178+0000","last_clean":"2024-04-17T10:31:40.194178+0000","last_became_active":"2024-04-17T10:31:38.383354+0000","last_became_peered":"2024-04-17T10:31:38.383354+0000","last_unstale":"2024-04-17T10:31:40.194178+0000","last_undegraded":"2024-04-17T10:31:40.194178+0000","last_fullsized":"2024-04-17T10:31:40.194178+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:25:53.140820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190860+0000","last_change":"2024-04-17T10:31:37.359348+0000","last_active":"2024-04-17T10:31:40.190860+0000","last_peered":"2024-04-17T10:31:40.190860+0000","last_clean":"2024-04-17T10:31:40.190860+0000","last_became_active":"2024-04-17T10:31:37.359107+0000","last_became_peered":"2024-04-17T10:31:37.359107+0000","last_unstale":"2024-04-17T10:31:40.190860+0000","last_undegraded":"2024-04-17T10:31:40.190860+0000","last_fullsized":"2024-04-17T10:31:40.190860+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:31:20.209205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191194+0000","last_change":"2024-04-17T10:31:38.360836+0000","last_active":"2024-04-17T10:31:40.191194+0000","last_peered":"2024-04-17T10:31:40.191194+0000","last_clean":"2024-04-17T10:31:40.191194+0000","last_became_active":"2024-04-17T10:31:38.360372+0000","last_became_peered":"2024-04-17T10:31:38.360372+0000","last_unstale":"2024-04-17T10:31:40.191194+0000","last_undegraded":"2024-04-17T10:31:40.191194+0000","last_fullsized":"2024-04-17T10:31:40.191194+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:25.031678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366890+0000","last_change":"2024-04-17T10:31:37.365058+0000","last_active":"2024-04-17T10:31:39.366890+0000","last_peered":"2024-04-17T10:31:39.366890+0000","last_clean":"2024-04-17T10:31:39.366890+0000","last_became_active":"2024-04-17T10:31:37.364541+0000","last_became_peered":"2024-04-17T10:31:37.364541+0000","last_unstale":"2024-04-17T10:31:39.366890+0000","last_undegraded":"2024-04-17T10:31:39.366890+0000","last_fullsized":"2024-04-17T10:31:39.366890+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:38.273762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190925+0000","last_change":"2024-04-17T10:31:38.371373+0000","last_active":"2024-04-17T10:31:40.190925+0000","last_peered":"2024-04-17T10:31:40.190925+0000","last_clean":"2024-04-17T10:31:40.190925+0000","last_became_active":"2024-04-17T10:31:38.371072+0000","last_became_peered":"2024-04-17T10:31:38.371072+0000","last_unstale":"2024-04-17T10:31:40.190925+0000","last_undegraded":"2024-04-17T10:31:40.190925+0000","last_fullsized":"2024-04-17T10:31:40.190925+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:21.266772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191270+0000","last_change":"2024-04-17T10:31:37.360114+0000","last_active":"2024-04-17T10:31:40.191270+0000","last_peered":"2024-04-17T10:31:40.191270+0000","last_clean":"2024-04-17T10:31:40.191270+0000","last_became_active":"2024-04-17T10:31:37.359725+0000","last_became_peered":"2024-04-17T10:31:37.359725+0000","last_unstale":"2024-04-17T10:31:40.191270+0000","last_undegraded":"2024-04-17T10:31:40.191270+0000","last_fullsized":"2024-04-17T10:31:40.191270+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:35:00.900880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193864+0000","last_change":"2024-04-17T10:31:38.376121+0000","last_active":"2024-04-17T10:31:40.193864+0000","last_peered":"2024-04-17T10:31:40.193864+0000","last_clean":"2024-04-17T10:31:40.193864+0000","last_became_active":"2024-04-17T10:31:38.375833+0000","last_became_peered":"2024-04-17T10:31:38.375833+0000","last_unstale":"2024-04-17T10:31:40.193864+0000","last_undegraded":"2024-04-17T10:31:40.193864+0000","last_fullsized":"2024-04-17T10:31:40.193864+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:35:43.598662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366769+0000","last_change":"2024-04-17T10:31:37.362595+0000","last_active":"2024-04-17T10:31:39.366769+0000","last_peered":"2024-04-17T10:31:39.366769+0000","last_clean":"2024-04-17T10:31:39.366769+0000","last_became_active":"2024-04-17T10:31:37.362307+0000","last_became_peered":"2024-04-17T10:31:37.362307+0000","last_unstale":"2024-04-17T10:31:39.366769+0000","last_undegraded":"2024-04-17T10:31:39.366769+0000","last_fullsized":"2024-04-17T10:31:39.366769+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:15:47.101230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191674+0000","last_change":"2024-04-17T10:31:38.358182+0000","last_active":"2024-04-17T10:31:40.191674+0000","last_peered":"2024-04-17T10:31:40.191674+0000","last_clean":"2024-04-17T10:31:40.191674+0000","last_became_active":"2024-04-17T10:31:38.357883+0000","last_became_peered":"2024-04-17T10:31:38.357883+0000","last_unstale":"2024-04-17T10:31:40.191674+0000","last_undegraded":"2024-04-17T10:31:40.191674+0000","last_fullsized":"2024-04-17T10:31:40.191674+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:42.796034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"95'2","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190790+0000","last_change":"2024-04-17T10:31:37.354403+0000","last_active":"2024-04-17T10:31:40.190790+0000","last_peered":"2024-04-17T10:31:40.190790+0000","last_clean":"2024-04-17T10:31:40.190790+0000","last_became_active":"2024-04-17T10:31:37.353930+0000","last_became_peered":"2024-04-17T10:31:37.353930+0000","last_unstale":"2024-04-17T10:31:40.190790+0000","last_undegraded":"2024-04-17T10:31:40.190790+0000","last_fullsized":"2024-04-17T10:31:40.190790+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:54.764173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194103+0000","last_change":"2024-04-17T10:31:38.386818+0000","last_active":"2024-04-17T10:31:40.194103+0000","last_peered":"2024-04-17T10:31:40.194103+0000","last_clean":"2024-04-17T10:31:40.194103+0000","last_became_active":"2024-04-17T10:31:38.383939+0000","last_became_peered":"2024-04-17T10:31:38.383939+0000","last_unstale":"2024-04-17T10:31:40.194103+0000","last_undegraded":"2024-04-17T10:31:40.194103+0000","last_fullsized":"2024-04-17T10:31:40.194103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:01.834781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"95'1","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.362998+0000","last_change":"2024-04-17T10:31:37.354939+0000","last_active":"2024-04-17T10:31:39.362998+0000","last_peered":"2024-04-17T10:31:39.362998+0000","last_clean":"2024-04-17T10:31:39.362998+0000","last_became_active":"2024-04-17T10:31:37.354671+0000","last_became_peered":"2024-04-17T10:31:37.354671+0000","last_unstale":"2024-04-17T10:31:39.362998+0000","last_undegraded":"2024-04-17T10:31:39.362998+0000","last_fullsized":"2024-04-17T10:31:39.362998+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:36:02.799577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.194023+0000","last_change":"2024-04-17T10:31:38.376661+0000","last_active":"2024-04-17T10:31:40.194023+0000","last_peered":"2024-04-17T10:31:40.194023+0000","last_clean":"2024-04-17T10:31:40.194023+0000","last_became_active":"2024-04-17T10:31:38.376364+0000","last_became_peered":"2024-04-17T10:31:38.376364+0000","last_unstale":"2024-04-17T10:31:40.194023+0000","last_undegraded":"2024-04-17T10:31:40.194023+0000","last_fullsized":"2024-04-17T10:31:40.194023+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:10:28.335090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.363154+0000","last_change":"2024-04-17T10:31:37.349354+0000","last_active":"2024-04-17T10:31:39.363154+0000","last_peered":"2024-04-17T10:31:39.363154+0000","last_clean":"2024-04-17T10:31:39.363154+0000","last_became_active":"2024-04-17T10:31:37.348447+0000","last_became_peered":"2024-04-17T10:31:37.348447+0000","last_unstale":"2024-04-17T10:31:39.363154+0000","last_undegraded":"2024-04-17T10:31:39.363154+0000","last_fullsized":"2024-04-17T10:31:39.363154+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:22.070539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191375+0000","last_change":"2024-04-17T10:31:38.369656+0000","last_active":"2024-04-17T10:31:40.191375+0000","last_peered":"2024-04-17T10:31:40.191375+0000","last_clean":"2024-04-17T10:31:40.191375+0000","last_became_active":"2024-04-17T10:31:38.366284+0000","last_became_peered":"2024-04-17T10:31:38.366284+0000","last_unstale":"2024-04-17T10:31:40.191375+0000","last_undegraded":"2024-04-17T10:31:40.191375+0000","last_fullsized":"2024-04-17T10:31:40.191375+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:07:01.747376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191869+0000","last_change":"2024-04-17T10:31:37.355915+0000","last_active":"2024-04-17T10:31:40.191869+0000","last_peered":"2024-04-17T10:31:40.191869+0000","last_clean":"2024-04-17T10:31:40.191869+0000","last_became_active":"2024-04-17T10:31:37.355650+0000","last_became_peered":"2024-04-17T10:31:37.355650+0000","last_unstale":"2024-04-17T10:31:40.191869+0000","last_undegraded":"2024-04-17T10:31:40.191869+0000","last_fullsized":"2024-04-17T10:31:40.191869+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:31.408106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366830+0000","last_change":"2024-04-17T10:31:38.384626+0000","last_active":"2024-04-17T10:31:39.366830+0000","last_peered":"2024-04-17T10:31:39.366830+0000","last_clean":"2024-04-17T10:31:39.366830+0000","last_became_active":"2024-04-17T10:31:38.384373+0000","last_became_peered":"2024-04-17T10:31:38.384373+0000","last_unstale":"2024-04-17T10:31:39.366830+0000","last_undegraded":"2024-04-17T10:31:39.366830+0000","last_fullsized":"2024-04-17T10:31:39.366830+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:48.060879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193972+0000","last_change":"2024-04-17T10:31:37.354088+0000","last_active":"2024-04-17T10:31:40.193972+0000","last_peered":"2024-04-17T10:31:40.193972+0000","last_clean":"2024-04-17T10:31:40.193972+0000","last_became_active":"2024-04-17T10:31:37.353817+0000","last_became_peered":"2024-04-17T10:31:37.353817+0000","last_unstale":"2024-04-17T10:31:40.193972+0000","last_undegraded":"2024-04-17T10:31:40.193972+0000","last_fullsized":"2024-04-17T10:31:40.193972+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:28:20.260258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.366210+0000","last_change":"2024-04-17T10:31:38.374513+0000","last_active":"2024-04-17T10:31:39.366210+0000","last_peered":"2024-04-17T10:31:39.366210+0000","last_clean":"2024-04-17T10:31:39.366210+0000","last_became_active":"2024-04-17T10:31:38.374235+0000","last_became_peered":"2024-04-17T10:31:38.374235+0000","last_unstale":"2024-04-17T10:31:39.366210+0000","last_undegraded":"2024-04-17T10:31:39.366210+0000","last_fullsized":"2024-04-17T10:31:39.366210+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:11:20.114227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190954+0000","last_change":"2024-04-17T10:31:37.360638+0000","last_active":"2024-04-17T10:31:40.190954+0000","last_peered":"2024-04-17T10:31:40.190954+0000","last_clean":"2024-04-17T10:31:40.190954+0000","last_became_active":"2024-04-17T10:31:37.360212+0000","last_became_peered":"2024-04-17T10:31:37.360212+0000","last_unstale":"2024-04-17T10:31:40.190954+0000","last_undegraded":"2024-04-17T10:31:40.190954+0000","last_fullsized":"2024-04-17T10:31:40.190954+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:45:33.318986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191928+0000","last_change":"2024-04-17T10:31:38.372865+0000","last_active":"2024-04-17T10:31:40.191928+0000","last_peered":"2024-04-17T10:31:40.191928+0000","last_clean":"2024-04-17T10:31:40.191928+0000","last_became_active":"2024-04-17T10:31:38.372582+0000","last_became_peered":"2024-04-17T10:31:38.372582+0000","last_unstale":"2024-04-17T10:31:40.191928+0000","last_undegraded":"2024-04-17T10:31:40.191928+0000","last_fullsized":"2024-04-17T10:31:40.191928+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:34:34.604484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.191796+0000","last_change":"2024-04-17T10:31:37.362924+0000","last_active":"2024-04-17T10:31:40.191796+0000","last_peered":"2024-04-17T10:31:40.191796+0000","last_clean":"2024-04-17T10:31:40.191796+0000","last_became_active":"2024-04-17T10:31:37.362630+0000","last_became_peered":"2024-04-17T10:31:37.362630+0000","last_unstale":"2024-04-17T10:31:40.191796+0000","last_undegraded":"2024-04-17T10:31:40.191796+0000","last_fullsized":"2024-04-17T10:31:40.191796+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:23:10.829453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.193954+0000","last_change":"2024-04-17T10:31:38.368340+0000","last_active":"2024-04-17T10:31:40.193954+0000","last_peered":"2024-04-17T10:31:40.193954+0000","last_clean":"2024-04-17T10:31:40.193954+0000","last_became_active":"2024-04-17T10:31:38.368065+0000","last_became_peered":"2024-04-17T10:31:38.368065+0000","last_unstale":"2024-04-17T10:31:40.193954+0000","last_undegraded":"2024-04-17T10:31:40.193954+0000","last_fullsized":"2024-04-17T10:31:40.193954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:38:20.873547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190568+0000","last_change":"2024-04-17T10:31:37.361092+0000","last_active":"2024-04-17T10:31:40.190568+0000","last_peered":"2024-04-17T10:31:40.190568+0000","last_clean":"2024-04-17T10:31:40.190568+0000","last_became_active":"2024-04-17T10:31:37.360613+0000","last_became_peered":"2024-04-17T10:31:37.360613+0000","last_unstale":"2024-04-17T10:31:40.190568+0000","last_undegraded":"2024-04-17T10:31:40.190568+0000","last_fullsized":"2024-04-17T10:31:40.190568+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:27:02.024574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190549+0000","last_change":"2024-04-17T10:31:38.359917+0000","last_active":"2024-04-17T10:31:40.190549+0000","last_peered":"2024-04-17T10:31:40.190549+0000","last_clean":"2024-04-17T10:31:40.190549+0000","last_became_active":"2024-04-17T10:31:38.359610+0000","last_became_peered":"2024-04-17T10:31:38.359610+0000","last_unstale":"2024-04-17T10:31:40.190549+0000","last_undegraded":"2024-04-17T10:31:40.190549+0000","last_fullsized":"2024-04-17T10:31:40.190549+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:37.332052+0000","last_clean_scrub_stamp":"2024-04-17T10:31:37.332052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:15:51.743755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:39.637594+0000","last_change":"2024-04-17T10:31:37.360694+0000","last_active":"2024-04-17T10:31:39.637594+0000","last_peered":"2024-04-17T10:31:39.637594+0000","last_clean":"2024-04-17T10:31:39.637594+0000","last_became_active":"2024-04-17T10:31:37.360046+0000","last_became_peered":"2024-04-17T10:31:37.360046+0000","last_unstale":"2024-04-17T10:31:39.637594+0000","last_undegraded":"2024-04-17T10:31:39.637594+0000","last_fullsized":"2024-04-17T10:31:39.637594+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:31:36.325538+0000","last_clean_scrub_stamp":"2024-04-17T10:31:36.325538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:49:22.070539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":310,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-04-17T10:31:40.190684+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:31:40.190684+0000","last_peered":"2024-04-17T10:31:40.190684+0000","last_clean":"2024-04-17T10:31:40.190684+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:31:40.190684+0000","last_undegraded":"2024-04-17T10:31:40.190684+0000","last_fullsized":"2024-04-17T10:31:40.190684+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574936,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46812,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325028,"statfs":{"total":96636764160,"available":96588828672,"internally_reserved":0,"allocated":1785856,"data_stored":1142512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574934,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50408,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321432,"statfs":{"total":96636764160,"available":96585146368,"internally_reserved":0,"allocated":1204224,"data_stored":554304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574935,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46224,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325616,"statfs":{"total":96636764160,"available":96589430784,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54552,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317288,"statfs":{"total":96636764160,"available":96580902912,"internally_reserved":0,"allocated":1191936,"data_stored":552248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54544,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317296,"statfs":{"total":96636764160,"available":96580911104,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574936,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50392,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321448,"statfs":{"total":96636764160,"available":96585162752,"internally_reserved":0,"allocated":1183744,"data_stored":552088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46228,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325612,"statfs":{"total":96636764160,"available":96589426688,"internally_reserved":0,"allocated":1187840,"data_stored":552158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574935,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50988,"kb_used_data":1760,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320852,"statfs":{"total":96636764160,"available":96584552448,"internally_reserved":0,"allocated":1802240,"data_stored":1144774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:31:47.453 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:31:47.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:31:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 Processor -- start 2024-04-17T10:31:47.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 -- start start 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f56ec15dd60 0x7f56ec161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec09adf0 0x7f56ec161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56ec0cff90 0x7f56ec0d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec1626d0 con 0x7f56ec0cff90 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56ec15e140 con 0x7f56ec15dd60 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56f1c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec0d0960 con 0x7f56ec09adf0 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56ec0cff90 0x7f56ec0d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f56ec0cff90 0x7f56ec0d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46428/0 (socket says 172.21.15.67:46428) 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 -- 172.21.15.67:0/4222760553 learned_addr learned my addr 172.21.15.67:0/4222760553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56e3e00640 1 --2- 172.21.15.67:0/4222760553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec09adf0 0x7f56ec161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eaa00640 1 --2- 172.21.15.67:0/4222760553 >> v2:172.21.15.86:3300/0 conn(0x7f56ec15dd60 0x7f56ec161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 -- 172.21.15.67:0/4222760553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec09adf0 msgr2=0x7f56ec161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 --2- 172.21.15.67:0/4222760553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec09adf0 0x7f56ec161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 -- 172.21.15.67:0/4222760553 >> v2:172.21.15.86:3300/0 conn(0x7f56ec15dd60 msgr2=0x7f56ec161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 --2- 172.21.15.67:0/4222760553 >> v2:172.21.15.86:3300/0 conn(0x7f56ec15dd60 0x7f56ec161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.611+0000 7f56eb400640 1 -- 172.21.15.67:0/4222760553 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec099d50 con 0x7f56ec0cff90 2024-04-17T10:31:47.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56eb400640 1 --2- 172.21.15.67:0/4222760553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0d0420 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f56ec06f620 tx=0x7f56e401c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfcd1e32a7f9ade server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56ea000640 1 -- 172.21.15.67:0/4222760553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e40a5860 con 0x7f56ec0cff90 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56ea000640 1 -- 172.21.15.67:0/4222760553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e40a5a00 con 0x7f56ec0cff90 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56ea000640 1 -- 172.21.15.67:0/4222760553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e40a5dd0 con 0x7f56ec0cff90 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 -- 172.21.15.67:0/4222760553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 msgr2=0x7f56ec0d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 --2- 172.21.15.67:0/4222760553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0d0420 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f56ec06f620 tx=0x7f56e401c9c0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 -- 172.21.15.67:0/4222760553 shutdown_connections 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 --2- 172.21.15.67:0/4222760553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0d0420 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 --2- 172.21.15.67:0/4222760553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec09adf0 0x7f56ec161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 --2- 172.21.15.67:0/4222760553 >> v2:172.21.15.86:3300/0 conn(0x7f56ec15dd60 0x7f56ec161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 -- 172.21.15.67:0/4222760553 >> 172.21.15.67:0/4222760553 conn(0x7f56ec0b34a0 msgr2=0x7f56ec0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:47.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 -- 172.21.15.67:0/4222760553 shutdown_connections 2024-04-17T10:31:47.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.615+0000 7f56f1c00640 1 -- 172.21.15.67:0/4222760553 wait complete. 2024-04-17T10:31:47.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 Processor -- start 2024-04-17T10:31:47.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 -- start start 2024-04-17T10:31:47.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56ec09adf0 0x7f56ec0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec15dd60 0x7f56ec152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec06c5b0 con 0x7f56ec0cff90 2024-04-17T10:31:47.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec09a8d0 con 0x7f56ec09adf0 2024-04-17T10:31:47.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56ec06c730 con 0x7f56ec15dd60 2024-04-17T10:31:47.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46440/0 (socket says 172.21.15.67:46440) 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 -- 172.21.15.67:0/2388767462 learned_addr learned my addr 172.21.15.67:0/2388767462 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56eb400640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec15dd60 0x7f56ec152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56eaa00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56ec09adf0 0x7f56ec0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 -- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec15dd60 msgr2=0x7f56ec152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec15dd60 0x7f56ec152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 -- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56ec09adf0 msgr2=0x7f56ec0c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56ec09adf0 0x7f56ec0c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 -- 172.21.15.67:0/2388767462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56ec15b030 con 0x7f56ec0cff90 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56eaa00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56ec09adf0 0x7f56ec0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e3e00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0c7bd0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f56d400e8b0 tx=0x7f56d400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e2a00640 1 -- 172.21.15.67:0/2388767462 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4017070 con 0x7f56ec0cff90 2024-04-17T10:31:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e2a00640 1 -- 172.21.15.67:0/2388767462 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d400c940 con 0x7f56ec0cff90 2024-04-17T10:31:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56e2a00640 1 -- 172.21.15.67:0/2388767462 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d400cd10 con 0x7f56ec0cff90 2024-04-17T10:31:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56ec15b310 con 0x7f56ec0cff90 2024-04-17T10:31:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56ec15d060 con 0x7f56ec0cff90 2024-04-17T10:31:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.619+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56ec04ef30 con 0x7f56ec0cff90 2024-04-17T10:31:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.623+0000 7f56e2a00640 1 -- 172.21.15.67:0/2388767462 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f56d401a070 con 0x7f56ec0cff90 2024-04-17T10:31:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.623+0000 7f56e2a00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56c8066f60 0x7f56c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.623+0000 7f56e2a00640 1 -- 172.21.15.67:0/2388767462 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f56d40854d0 con 0x7f56ec0cff90 2024-04-17T10:31:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.623+0000 7f56eaa00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56c8066f60 0x7f56c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.623+0000 7f56eaa00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56c8066f60 0x7f56c8069420 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f56ec15a3f0 tx=0x7f56dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.627+0000 7f56e2a00640 1 -- 172.21.15.67:0/2388767462 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f56d4055cf0 con 0x7f56ec0cff90 2024-04-17T10:31:47.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.851+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f56ec0b42d0 con 0x7f56ec0cff90 2024-04-17T10:31:47.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.855+0000 7f56e2a00640 1 -- 172.21.15.67:0/2388767462 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f56d4059d90 con 0x7f56ec0cff90 2024-04-17T10:31:47.847 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:31:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:31:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:31:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:31:47.847 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:31:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:31:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56c8066f60 msgr2=0x7f56c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56c8066f60 0x7f56c8069420 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f56ec15a3f0 tx=0x7f56dc008040 comp rx=0 tx=0).stop 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 msgr2=0x7f56ec0c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0c7bd0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f56d400e8b0 tx=0x7f56d400ed70 comp rx=0 tx=0).stop 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 shutdown_connections 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f56c8066f60 0x7f56c8069420 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f56ec15dd60 0x7f56ec152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56ec0cff90 0x7f56ec0c7bd0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 --2- 172.21.15.67:0/2388767462 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f56ec09adf0 0x7f56ec0c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 >> 172.21.15.67:0/2388767462 conn(0x7f56ec0b34a0 msgr2=0x7f56ec161f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 shutdown_connections 2024-04-17T10:31:47.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:47.859+0000 7f56f1c00640 1 -- 172.21.15.67:0/2388767462 wait complete. 2024-04-17T10:31:47.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:31:48.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77c0400640 1 Processor -- start 2024-04-17T10:31:48.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77c0400640 1 -- start start 2024-04-17T10:31:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77c0400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b8163060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77c0400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f77b8071310 0x7f77b8166430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77c0400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b8071720 con 0x7f77b80cef80 2024-04-17T10:31:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77c0400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b80719a0 con 0x7f77b8071310 2024-04-17T10:31:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77c0400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b8071c80 con 0x7f77b80b4970 2024-04-17T10:31:48.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77bd000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b8163060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77be400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f77b8071310 0x7f77b8166430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77bd000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b8163060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38682/0 (socket says 172.21.15.67:38682) 2024-04-17T10:31:48.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77bd000640 1 -- 172.21.15.67:0/4079949495 learned_addr learned my addr 172.21.15.67:0/4079949495 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77bda00640 1 --1- 172.21.15.67:0/4079949495 >> v1:172.21.15.86:6789/0 conn(0x7f77b80cef80 0x7f77b80b4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:60356/0 (socket says 172.21.15.67:60356) 2024-04-17T10:31:48.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77bd000640 1 -- 172.21.15.67:0/4079949495 >> v2:172.21.15.86:3300/0 conn(0x7f77b8071310 msgr2=0x7f77b8166430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77bd000640 1 --2- 172.21.15.67:0/4079949495 >> v2:172.21.15.86:3300/0 conn(0x7f77b8071310 0x7f77b8166430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77bd000640 1 -- 172.21.15.67:0/4079949495 >> v1:172.21.15.86:6789/0 conn(0x7f77b80cef80 legacy=0x7f77b80b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.019+0000 7f77b7e00640 1 -- 172.21.15.67:0/4079949495 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948973910 0 0) 0x7f77ac001120 con 0x7f77b80cef80 2024-04-17T10:31:48.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77bd000640 1 -- 172.21.15.67:0/4079949495 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b8162580 con 0x7f77b80b4970 2024-04-17T10:31:48.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77bd000640 1 --2- 172.21.15.67:0/4079949495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b8163060 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009870 tx=0x7f77a401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64cf2f0628f55962 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77b7e00640 1 -- 172.21.15.67:0/4079949495 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a40ab650 con 0x7f77b80b4970 2024-04-17T10:31:48.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77b7e00640 1 -- 172.21.15.67:0/4079949495 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77a40b0070 con 0x7f77b80b4970 2024-04-17T10:31:48.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77b7e00640 1 -- 172.21.15.67:0/4079949495 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a40aba10 con 0x7f77b80b4970 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77c0400640 1 -- 172.21.15.67:0/4079949495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 msgr2=0x7f77b8163060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77c0400640 1 --2- 172.21.15.67:0/4079949495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b8163060 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77a4009870 tx=0x7f77a401c560 comp rx=0 tx=0).stop 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77c0400640 1 -- 172.21.15.67:0/4079949495 shutdown_connections 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77c0400640 1 --2- 172.21.15.67:0/4079949495 >> v2:172.21.15.86:3300/0 conn(0x7f77b8071310 0x7f77b8166430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77c0400640 1 --2- 172.21.15.67:0/4079949495 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b8163060 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.023+0000 7f77c0400640 1 -- 172.21.15.67:0/4079949495 >> 172.21.15.67:0/4079949495 conn(0x7f77b806ac90 msgr2=0x7f77b806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 -- 172.21.15.67:0/4079949495 shutdown_connections 2024-04-17T10:31:48.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 -- 172.21.15.67:0/4079949495 wait complete. 2024-04-17T10:31:48.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 Processor -- start 2024-04-17T10:31:48.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 -- start start 2024-04-17T10:31:48.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8071310 0x7f77b80b13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77b80cef80 0x7f77b81faa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80b3100 con 0x7f77b8071310 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80b2f80 con 0x7f77b80cef80 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b80b3280 con 0x7f77b80b4970 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8071310 0x7f77b80b13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8071310 0x7f77b80b13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46442/0 (socket says 172.21.15.67:46442) 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 -- 172.21.15.67:0/3541781867 learned_addr learned my addr 172.21.15.67:0/3541781867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bd000640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77be400640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77b80cef80 0x7f77b81faa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 -- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 msgr2=0x7f77b815ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b815ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 -- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77b80cef80 msgr2=0x7f77b81faa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77b80cef80 0x7f77b81faa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 -- 172.21.15.67:0/3541781867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b815f420 con 0x7f77b8071310 2024-04-17T10:31:48.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77bda00640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8071310 0x7f77b80b13a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00c4b0 tx=0x7f77ac00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77b5600640 1 -- 172.21.15.67:0/3541781867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac018070 con 0x7f77b8071310 2024-04-17T10:31:48.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77b5600640 1 -- 172.21.15.67:0/3541781867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac007500 con 0x7f77b8071310 2024-04-17T10:31:48.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b815f700 con 0x7f77b8071310 2024-04-17T10:31:48.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.027+0000 7f77b5600640 1 -- 172.21.15.67:0/3541781867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac004a30 con 0x7f77b8071310 2024-04-17T10:31:48.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.031+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b815fcc0 con 0x7f77b8071310 2024-04-17T10:31:48.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.031+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b8072d50 con 0x7f77b8071310 2024-04-17T10:31:48.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.031+0000 7f77b5600640 1 -- 172.21.15.67:0/3541781867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f77ac01b070 con 0x7f77b8071310 2024-04-17T10:31:48.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.031+0000 7f77b5600640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7798066f60 0x7f7798069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.031+0000 7f77b5600640 1 -- 172.21.15.67:0/3541781867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f77ac084460 con 0x7f77b8071310 2024-04-17T10:31:48.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.031+0000 7f77bd000640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7798066f60 0x7f7798069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.031+0000 7f77bd000640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7798066f60 0x7f7798069420 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f77a40af000 tx=0x7f77a40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.035+0000 7f77b5600640 1 -- 172.21.15.67:0/3541781867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f77b8071720 con 0x7f77b8071310 2024-04-17T10:31:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77b80ae360 con 0x7f77b8071310 2024-04-17T10:31:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77b5600640 1 -- 172.21.15.67:0/3541781867 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f77ac054c80 con 0x7f77b8071310 2024-04-17T10:31:48.222 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:31:48.223 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:31:37.749417+0000","modified":"2024-04-17T10:31:40.768850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7698":{"gid":7698,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3523329517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3523329517},{"type":"v1","addr":"172.21.15.67:6835","nonce":3523329517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-17T10:31:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 64 2024-04-17T10:31:48.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7798066f60 msgr2=0x7f7798069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7798066f60 0x7f7798069420 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f77a40af000 tx=0x7f77a40a7000 comp rx=0 tx=0).stop 2024-04-17T10:31:48.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8071310 msgr2=0x7f77b80b13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8071310 0x7f77b80b13a0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f77ac00c4b0 tx=0x7f77ac00c970 comp rx=0 tx=0).stop 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 shutdown_connections 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7798066f60 0x7f7798069420 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f77b80cef80 0x7f77b81faa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f77b80b4970 0x7f77b815ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 --2- 172.21.15.67:0/3541781867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77b8071310 0x7f77b80b13a0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.231+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 >> 172.21.15.67:0/3541781867 conn(0x7f77b806ac90 msgr2=0x7f77b8164450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.235+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 shutdown_connections 2024-04-17T10:31:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.235+0000 7f77c0400640 1 -- 172.21.15.67:0/3541781867 wait complete. 2024-04-17T10:31:48.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:31:37.749417+0000', 'modified': '2024-04-17T10:31:40.768850+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7698}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7698': {'gid': 7698, 'name': 'a', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6835/3523329517', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 3523329517}, {'type': 'v1', 'addr': '172.21.15.67:6835', 'nonce': 3523329517}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:31:48.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:31:48.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T10:31:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.799+0000 7ff087400640 1 Processor -- start 2024-04-17T10:31:48.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.799+0000 7ff087400640 1 -- start start 2024-04-17T10:31:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff087400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff08006eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff087400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff080146da0 0x7ff080071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff087400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff080161740 con 0x7ff080071850 2024-04-17T10:31:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff087400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff080161a20 con 0x7ff080146da0 2024-04-17T10:31:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff087400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff080161d00 con 0x7ff0801466e0 2024-04-17T10:31:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff080146da0 0x7ff080071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff080146da0 0x7ff080071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46454/0 (socket says 172.21.15.67:46454) 2024-04-17T10:31:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 -- 172.21.15.67:0/1484648266 learned_addr learned my addr 172.21.15.67:0/1484648266 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff084a00640 1 --2- 172.21.15.67:0/1484648266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff08006eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 -- 172.21.15.67:0/1484648266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 msgr2=0x7ff08006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 --2- 172.21.15.67:0/1484648266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff08006eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 -- 172.21.15.67:0/1484648266 >> v1:172.21.15.86:6789/0 conn(0x7ff080071850 legacy=0x7ff08006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 -- 172.21.15.67:0/1484648266 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff080161fe0 con 0x7ff080146da0 2024-04-17T10:31:48.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff084a00640 1 --2- 172.21.15.67:0/1484648266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff08006eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff077e00640 1 --2- 172.21.15.67:0/1484648266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080146da0 0x7ff080071310 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff068009870 tx=0x7ff06801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ded10977606bfd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff07fe00640 1 -- 172.21.15.67:0/1484648266 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680b0070 con 0x7ff080146da0 2024-04-17T10:31:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.803+0000 7ff07fe00640 1 -- 172.21.15.67:0/1484648266 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0680ab600 con 0x7ff080146da0 2024-04-17T10:31:48.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff07fe00640 1 -- 172.21.15.67:0/1484648266 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680ab9d0 con 0x7ff080146da0 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- 172.21.15.67:0/1484648266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080146da0 msgr2=0x7ff080071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 --2- 172.21.15.67:0/1484648266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080146da0 0x7ff080071310 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7ff068009870 tx=0x7ff06801c520 comp rx=0 tx=0).stop 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- 172.21.15.67:0/1484648266 shutdown_connections 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 --2- 172.21.15.67:0/1484648266 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080146da0 0x7ff080071310 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 --2- 172.21.15.67:0/1484648266 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff08006eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- 172.21.15.67:0/1484648266 >> 172.21.15.67:0/1484648266 conn(0x7ff08006ac90 msgr2=0x7ff08006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- 172.21.15.67:0/1484648266 shutdown_connections 2024-04-17T10:31:48.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- 172.21.15.67:0/1484648266 wait complete. 2024-04-17T10:31:48.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 Processor -- start 2024-04-17T10:31:48.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- start start 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080071850 0x7ff08015b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff0800ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080146da0 0x7ff08015cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08016aa40 con 0x7ff080071850 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800af200 con 0x7ff080146da0 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff087400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08016abc0 con 0x7ff0801466e0 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff084a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080071850 0x7ff08015b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff084a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080071850 0x7ff08015b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46460/0 (socket says 172.21.15.67:46460) 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff084a00640 1 -- 172.21.15.67:0/2288291692 learned_addr learned my addr 172.21.15.67:0/2288291692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff077e00640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff0800ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.807+0000 7ff085400640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080146da0 0x7ff08015cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff084a00640 1 -- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 msgr2=0x7ff0800ce8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff084a00640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff0800ce8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff084a00640 1 -- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080146da0 msgr2=0x7ff08015cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff084a00640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080146da0 0x7ff08015cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff084a00640 1 -- 172.21.15.67:0/2288291692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0800cd840 con 0x7ff080071850 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff084a00640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080071850 0x7ff08015b130 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c7f0 tx=0x7ff07000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff07d600640 1 -- 172.21.15.67:0/2288291692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070007a30 con 0x7ff080071850 2024-04-17T10:31:48.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff07d600640 1 -- 172.21.15.67:0/2288291692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff070007bd0 con 0x7ff080071850 2024-04-17T10:31:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff07d600640 1 -- 172.21.15.67:0/2288291692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700113f0 con 0x7ff080071850 2024-04-17T10:31:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0800cdb20 con 0x7ff080071850 2024-04-17T10:31:48.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0800ce0e0 con 0x7ff080071850 2024-04-17T10:31:48.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff07d600640 1 -- 172.21.15.67:0/2288291692 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff070011590 con 0x7ff080071850 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff07d600640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff05c066f60 0x7ff05c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.811+0000 7ff07d600640 1 -- 172.21.15.67:0/2288291692 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff0700853f0 con 0x7ff080071850 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff05c066f60 0x7ff05c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 Processor -- start 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- start start 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff05c066f60 0x7ff05c069420 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff0680ad000 tx=0x7ff068008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308c4bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308c019f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308c02310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e308ae95f0 con 0x55e308c4c240 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e308a50da0 con 0x55e308c4b920 2024-04-17T10:31:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3089d9150 con 0x55e308c01f30 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308c019f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308c019f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46468/0 (socket says 172.21.15.67:46468) 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 -- 172.21.15.67:0/862621250 learned_addr learned my addr 172.21.15.67:0/862621250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff084a00640 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308c4bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff085400640 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308c02310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 -- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 msgr2=0x55e308c02310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308c02310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 -- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 msgr2=0x55e308c4bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308c4bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 -- 172.21.15.67:0/862621250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3089f2ca0 con 0x55e308c4c240 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff077e00640 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308c019f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff0680ae040 tx=0x7ff0680a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b801f1282f1dde5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff075600640 1 -- 172.21.15.67:0/862621250 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680c7070 con 0x55e308c4c240 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff075600640 1 -- 172.21.15.67:0/862621250 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0680b1040 con 0x55e308c4c240 2024-04-17T10:31:48.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff075600640 1 -- 172.21.15.67:0/862621250 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680b7830 con 0x55e308c4c240 2024-04-17T10:31:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- 172.21.15.67:0/862621250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 msgr2=0x55e308c019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308c019f0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7ff0680ae040 tx=0x7ff0680a6000 comp rx=0 tx=0).stop 2024-04-17T10:31:48.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- 172.21.15.67:0/862621250 shutdown_connections 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308c02310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308c019f0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/862621250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308c4bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- 172.21.15.67:0/862621250 >> 172.21.15.67:0/862621250 conn(0x55e308c082a0 msgr2=0x55e308c002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- 172.21.15.67:0/862621250 shutdown_connections 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.815+0000 7ff08a8e4000 1 -- 172.21.15.67:0/862621250 wait complete. 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 -- start start 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 Processor -- start 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c01f30 0x55e308c52cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4b920 0x55e308c53230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4c240 0x55e308c05aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e308a0b990 con 0x55e308c01f30 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e308a4ce40 con 0x55e308c4c240 2024-04-17T10:31:48.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3089dc900 con 0x55e308c4b920 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 10 client.0 _ll_register_callbacks cb 0x55e308b041c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c01f30 0x55e308c52cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c01f30 0x55e308c52cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46472/0 (socket says 172.21.15.67:46472) 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 -- 172.21.15.67:0/1681322897 learned_addr learned my addr 172.21.15.67:0/1681322897 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff077e00640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4b920 0x55e308c53230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff085400640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4c240 0x55e308c05aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 -- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4b920 msgr2=0x55e308c53230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4b920 0x55e308c53230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 -- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4c240 msgr2=0x55e308c05aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4c240 0x55e308c05aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 -- 172.21.15.67:0/1681322897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e308a09630 con 0x55e308c01f30 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff085400640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4c240 0x55e308c05aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff077e00640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4b920 0x55e308c53230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff084a00640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c01f30 0x55e308c52cf0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c7c0 tx=0x7ff070059c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338e049a1368dab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff06e000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff06e000640 1 -- 172.21.15.67:0/1681322897 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07001e040 con 0x55e308c01f30 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff06e000640 1 -- 172.21.15.67:0/1681322897 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff07005ab40 con 0x55e308c01f30 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0680b1040 con 0x55e308c01f30 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff06e000640 1 -- 172.21.15.67:0/1681322897 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff070053f40 con 0x55e308c01f30 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 10 client.7845 fetch_fsmap learned FSMap version 64 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55e3089d93a0 con 0x55e308c01f30 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff06e000640 1 -- 172.21.15.67:0/1681322897 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff070081030 con 0x55e308c01f30 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff06e000640 1 -- 172.21.15.67:0/1681322897 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff070083120 con 0x55e308c01f30 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 10 client.7845 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 10 client.7845 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 20 client.7845 populate_metadata read hostname 'smithi067' 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c064b0 0x55e308c06890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 4 client.7845 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:31:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3089eca50 con 0x55e308c064b0 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff077e00640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c064b0 0x55e308c06890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff06e000640 10 client.7845 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.819+0000 7ff077e00640 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c064b0 0x55e308c06890 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92b65c1ab9dad2d7 server_cookie=a84261a59b2aa41f in_seq=0 out_seq=0 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff06e000640 1 -- 172.21.15.67:0/1681322897 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e3089eca50 con 0x55e308c064b0 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff06e000640 10 client.7845 handle_command_reply: tid=0 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 client.7845 shutdown 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c01f30 msgr2=0x55e308c52cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c01f30 0x55e308c52cf0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c7c0 tx=0x7ff070059c20 comp rx=0 tx=0).stop 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 shutdown_connections 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c064b0 0x55e308c06890 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4c240 0x55e308c05aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4b920 0x55e308c53230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1681322897 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c01f30 0x55e308c52cf0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff06e000640 0 client.7845 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:31:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 >> 172.21.15.67:0/1681322897 conn(0x55e308c082a0 msgr2=0x55e308c009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 shutdown_connections 2024-04-17T10:31:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1681322897 wait complete. 2024-04-17T10:31:48.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.823+0000 7ff08a8e4000 20 client.7845 trim_cache size 0 max 16384 2024-04-17T10:31:48.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 Processor -- start 2024-04-17T10:31:48.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- start start 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308ca7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308ca7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308ca7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080202950 con 0x55e308c4c240 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800cc9e0 con 0x55e308c4b920 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800ccfe0 con 0x55e308c01f30 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308ca7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308ca7cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46488/0 (socket says 172.21.15.67:46488) 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 -- 172.21.15.67:0/1697287369 learned_addr learned my addr 172.21.15.67:0/1697287369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff084a00640 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308ca7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff077e00640 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308ca7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 -- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 msgr2=0x55e308ca7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308ca7250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 -- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 msgr2=0x55e308ca7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308ca7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 -- 172.21.15.67:0/1697287369 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e308c4c070 con 0x55e308c4c240 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff085400640 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308ca7cd0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff078005400 tx=0x7ff0780058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f809916e9894d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff074c00640 1 -- 172.21.15.67:0/1697287369 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078019070 con 0x55e308c4c240 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff074c00640 1 -- 172.21.15.67:0/1697287369 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078011d60 con 0x55e308c4c240 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff074c00640 1 -- 172.21.15.67:0/1697287369 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078010040 con 0x55e308c4c240 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1697287369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 msgr2=0x55e308ca7cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308ca7cd0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff078005400 tx=0x7ff0780058c0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1697287369 shutdown_connections 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4c240 0x55e308ca7cd0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c4b920 0x55e308ca7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/1697287369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c01f30 0x55e308ca7250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1697287369 >> 172.21.15.67:0/1697287369 conn(0x55e308c082a0 msgr2=0x55e308c05470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1697287369 shutdown_connections 2024-04-17T10:31:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.839+0000 7ff08a8e4000 1 -- 172.21.15.67:0/1697287369 wait complete. 2024-04-17T10:31:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 -- start start 2024-04-17T10:31:48.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 Processor -- start 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c01f30 0x55e308c90ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4b920 0x55e308c91000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4c240 0x55e308c91540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e308a553d0 con 0x55e308c4b920 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e308b0cd50 con 0x55e308c01f30 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e3089eb980 con 0x55e308c4c240 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 10 client.0 _ll_register_callbacks cb 0x7ff0680096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4b920 0x55e308c91000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4b920 0x55e308c91000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46496/0 (socket says 172.21.15.67:46496) 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 -- 172.21.15.67:0/2241949918 learned_addr learned my addr 172.21.15.67:0/2241949918 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff085400640 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4c240 0x55e308c91540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff084a00640 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c01f30 0x55e308c90ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 -- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4c240 msgr2=0x55e308c91540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4c240 0x55e308c91540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 -- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c01f30 msgr2=0x55e308c90ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c01f30 0x55e308c90ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 -- 172.21.15.67:0/2241949918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e308ad2600 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff077e00640 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4b920 0x55e308c91000 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff06801abd0 tx=0x7ff06801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=344077be17e98fd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff06e000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff06e000640 1 -- 172.21.15.67:0/2241949918 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0680c7040 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff06e000640 1 -- 172.21.15.67:0/2241949918 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff0680d1270 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff078011d60 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff06e000640 1 -- 172.21.15.67:0/2241949918 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0680ab400 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 10 client.7851 fetch_fsmap learned FSMap version 64 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55e3089e59b0 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff06e000640 1 -- 172.21.15.67:0/2241949918 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078011d60 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff06e000640 1 -- 172.21.15.67:0/2241949918 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff0680c1070 con 0x55e308c4b920 2024-04-17T10:31:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 10 client.7851 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 10 client.7851 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 20 client.7851 populate_metadata read hostname 'smithi067' 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c58300 0x55e308c586e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 4 client.7851 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e308a5e2e0 con 0x55e308c58300 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.843+0000 7ff084a00640 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c58300 0x55e308c586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff084a00640 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c58300 0x55e308c586e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3263ba3999550d6 server_cookie=1930660b0c357bab in_seq=0 out_seq=0 2024-04-17T10:31:48.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff06e000640 10 client.7851 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff06e000640 1 -- 172.21.15.67:0/2241949918 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55e308a5e2e0 con 0x55e308c58300 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff06e000640 10 client.7851 handle_command_reply: tid=0 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 client.7851 shutdown 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4b920 msgr2=0x55e308c91000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4b920 0x55e308c91000 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7ff06801abd0 tx=0x7ff06801ad60 comp rx=0 tx=0).stop 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 shutdown_connections 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55e308c58300 0x55e308c586e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e308c4c240 0x55e308c91540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e308c4b920 0x55e308c91000 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 --2- 172.21.15.67:0/2241949918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e308c01f30 0x55e308c90ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff06e000640 0 client.7851 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 >> 172.21.15.67:0/2241949918 conn(0x55e308c082a0 msgr2=0x55e308c53860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 shutdown_connections 2024-04-17T10:31:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 1 -- 172.21.15.67:0/2241949918 wait complete. 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff08a8e4000 20 client.7851 trim_cache size 0 max 16384 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 7698, 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 10.494657138999999, 2024-04-17T10:31:48.842 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 64, 2024-04-17T10:31:48.843 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 96, 2024-04-17T10:31:48.843 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 96, 2024-04-17T10:31:48.843 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 15.354258658999999 2024-04-17T10:31:48.843 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff05c066f60 msgr2=0x7ff05c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff05c066f60 0x7ff05c069420 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff0680ad000 tx=0x7ff068008040 comp rx=0 tx=0).stop 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080071850 msgr2=0x7ff08015b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080071850 0x7ff08015b130 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7ff07000c7f0 tx=0x7ff07000ccb0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 shutdown_connections 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff05c066f60 0x7ff05c069420 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff080146da0 0x7ff08015cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0801466e0 0x7ff0800ce8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 --2- 172.21.15.67:0/2288291692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff080071850 0x7ff08015b130 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 >> 172.21.15.67:0/2288291692 conn(0x7ff08006ac90 msgr2=0x7ff0800b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 shutdown_connections 2024-04-17T10:31:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:48.847+0000 7ff087400640 1 -- 172.21.15.67:0/2288291692 wait complete. 2024-04-17T10:31:48.858 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:31:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf9800640 1 Processor -- start 2024-04-17T10:31:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf9800640 1 -- start start 2024-04-17T10:31:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf9800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ffaf4145040 0x7ffaf406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf9800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf406e2f0 con 0x7ffaf4071a20 2024-04-17T10:31:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf9800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaf406d6e0 con 0x7ffaf406ef10 2024-04-17T10:31:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf9800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaf40a86c0 con 0x7ffaf4145040 2024-04-17T10:31:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffaf4071a20 0x7ffaf40cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34490/0 (socket says 172.21.15.67:34490) 2024-04-17T10:31:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.011+0000 7ffaf3400640 1 -- 172.21.15.67:0/2077534356 learned_addr learned my addr 172.21.15.67:0/2077534356 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2a00640 1 --2- 172.21.15.67:0/2077534356 >> v2:172.21.15.86:3300/0 conn(0x7ffaf4145040 0x7ffaf406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21000121 0 0) 0x7ffaf406e2f0 con 0x7ffaf4071a20 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003530 con 0x7ffaf4071a20 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604313352 0 0) 0x7ffaf406d6e0 con 0x7ffaf406ef10 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaf406e2f0 con 0x7ffaf406ef10 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2a00640 1 -- 172.21.15.67:0/2077534356 >> v1:172.21.15.86:6789/0 conn(0x7ffaf406ef10 legacy=0x7ffaf4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2a00640 1 -- 172.21.15.67:0/2077534356 >> v1:172.21.15.67:6789/0 conn(0x7ffaf4071a20 legacy=0x7ffaf40cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3858766214 0 0) 0x7ffad8003530 con 0x7ffaf4071a20 2024-04-17T10:31:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2a00640 1 -- 172.21.15.67:0/2077534356 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf406d9c0 con 0x7ffaf4145040 2024-04-17T10:31:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.015+0000 7ffaf2a00640 1 --2- 172.21.15.67:0/2077534356 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf4145040 0x7ffaf406e9d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ffae40097c0 tx=0x7ffae401ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b3011e0950cc296 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffae40b5070 con 0x7ffaf4145040 2024-04-17T10:31:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffae40a8630 con 0x7ffaf4145040 2024-04-17T10:31:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf2000640 1 -- 172.21.15.67:0/2077534356 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec001a00 con 0x7ffaf4145040 2024-04-17T10:31:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- 172.21.15.67:0/2077534356 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf4145040 msgr2=0x7ffaf406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 --2- 172.21.15.67:0/2077534356 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf4145040 0x7ffaf406e9d0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ffae40097c0 tx=0x7ffae401ebb0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- 172.21.15.67:0/2077534356 shutdown_connections 2024-04-17T10:31:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 --2- 172.21.15.67:0/2077534356 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf4145040 0x7ffaf406e9d0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- 172.21.15.67:0/2077534356 >> 172.21.15.67:0/2077534356 conn(0x7ffaf406ac90 msgr2=0x7ffaf406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- 172.21.15.67:0/2077534356 shutdown_connections 2024-04-17T10:31:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- 172.21.15.67:0/2077534356 wait complete. 2024-04-17T10:31:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 Processor -- start 2024-04-17T10:31:49.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- start start 2024-04-17T10:31:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf406ef10 0x7ffaf413b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaf4071a20 0x7ffaf413ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaf4145040 0x7ffaf4140470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf406df80 con 0x7ffaf4145040 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf415bf10 con 0x7ffaf406ef10 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf9800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf406d370 con 0x7ffaf4071a20 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaf4145040 0x7ffaf4140470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaf4145040 0x7ffaf4140470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46504/0 (socket says 172.21.15.67:46504) 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.019+0000 7ffaf3400640 1 -- 172.21.15.67:0/1842876733 learned_addr learned my addr 172.21.15.67:0/1842876733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf2a00640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf406ef10 0x7ffaf413b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaea000640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaf4071a20 0x7ffaf413ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf3400640 1 -- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaf4071a20 msgr2=0x7ffaf413ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf3400640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaf4071a20 0x7ffaf413ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf3400640 1 -- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf406ef10 msgr2=0x7ffaf413b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf3400640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf406ef10 0x7ffaf413b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf3400640 1 -- 172.21.15.67:0/1842876733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf413c6d0 con 0x7ffaf4145040 2024-04-17T10:31:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf3400640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaf4145040 0x7ffaf4140470 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7ffaec001f40 tx=0x7ffaec00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaeb400640 1 -- 172.21.15.67:0/1842876733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec017070 con 0x7ffaf4145040 2024-04-17T10:31:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaeb400640 1 -- 172.21.15.67:0/1842876733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaec012400 con 0x7ffaf4145040 2024-04-17T10:31:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf413c9b0 con 0x7ffaf4145040 2024-04-17T10:31:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaeb400640 1 -- 172.21.15.67:0/1842876733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaec012830 con 0x7ffaf4145040 2024-04-17T10:31:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf413cf70 con 0x7ffaf4145040 2024-04-17T10:31:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf4072d50 con 0x7ffaf4145040 2024-04-17T10:31:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaeb400640 1 -- 172.21.15.67:0/1842876733 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ffaec01a070 con 0x7ffaf4145040 2024-04-17T10:31:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaeb400640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffad0066f60 0x7ffad0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:31:49.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.023+0000 7ffaeb400640 1 -- 172.21.15.67:0/1842876733 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffaec085110 con 0x7ffaf4145040 2024-04-17T10:31:49.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.027+0000 7ffaf2a00640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffad0066f60 0x7ffad0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:31:49.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.027+0000 7ffaf2a00640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffad0066f60 0x7ffad0069420 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ffae40b2000 tx=0x7ffae4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:31:49.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.027+0000 7ffaeb400640 1 -- 172.21.15.67:0/1842876733 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ffaec055920 con 0x7ffaf4145040 2024-04-17T10:31:49.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffaf40b11b0 con 0x7ffaf4145040 2024-04-17T10:31:49.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaeb400640 1 -- 172.21.15.67:0/1842876733 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffaec0599c0 con 0x7ffaf4145040 2024-04-17T10:31:49.219 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:31:49.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffad0066f60 msgr2=0x7ffad0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffad0066f60 0x7ffad0069420 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ffae40b2000 tx=0x7ffae4008040 comp rx=0 tx=0).stop 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaf4145040 msgr2=0x7ffaf4140470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaf4145040 0x7ffaf4140470 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7ffaec001f40 tx=0x7ffaec00a430 comp rx=0 tx=0).stop 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 shutdown_connections 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffad0066f60 0x7ffad0069420 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffaf4145040 0x7ffaf4140470 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffaf4071a20 0x7ffaf413ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 --2- 172.21.15.67:0/1842876733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffaf406ef10 0x7ffaf413b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:31:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.227+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 >> 172.21.15.67:0/1842876733 conn(0x7ffaf406ac90 msgr2=0x7ffaf413f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:31:49.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.231+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 shutdown_connections 2024-04-17T10:31:49.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:31:49.231+0000 7ffaf9800640 1 -- 172.21.15.67:0/1842876733 wait complete. 2024-04-17T10:31:49.239 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:31:49.239 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:31:49.240 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:31:49.240 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:31:49.240 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:31:49.240 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:31:49.240 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:31:49.240 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:31:49.240 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:31:49.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.240 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:31:49.250 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:31:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 42037sec preferred_lft 42037sec 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.252 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:31:49.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.254 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:31:49.314 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:31:49.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.316 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:31:49.371 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:31:49.372 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:31:49.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.373 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:31:49.373 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:49.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:31:49.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:31:49.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:31:49.373 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:49.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.466 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:31:49.475 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:31:49.475 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:31:49.476 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:31:49.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:31:49.536 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:31:49.536 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:31:49.592 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:31:49.592 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:31:49.592 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:31:49.647 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:31:49.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:31:49.740 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:31:49.741 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:31:49.754 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:31:49.754 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:31:49.754 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:31:49.754 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:31:49.754 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:31:49.758 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:31:49.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:31:49.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.779 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:31:49.841 INFO:teuthology.orchestra.run.smithi067.stdout:client6674 2024-04-17T10:31:49.841 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:31:49.841 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:31:49.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:49.892 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:49.893 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:49.893 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.0 2024-04-17T10:31:49.893 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:49.893 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:49.893 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:49.893 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:49.975 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:49.975 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:49.975 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175978 s, 59.6 MB/s 2024-04-17T10:31:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177147 s, 59.2 MB/s 2024-04-17T10:31:50.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.014 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170812 s, 61.4 MB/s 2024-04-17T10:31:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193436 s, 54.2 MB/s 2024-04-17T10:31:50.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.058 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212153 s, 49.4 MB/s 2024-04-17T10:31:50.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164556 s, 63.7 MB/s 2024-04-17T10:31:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.093 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147613 s, 71.0 MB/s 2024-04-17T10:31:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.109 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013144 s, 79.8 MB/s 2024-04-17T10:31:50.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197234 s, 53.2 MB/s 2024-04-17T10:31:50.148 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.149 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170137 s, 61.6 MB/s 2024-04-17T10:31:50.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.165 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144512 s, 72.6 MB/s 2024-04-17T10:31:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.182 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154455 s, 67.9 MB/s 2024-04-17T10:31:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147225 s, 71.2 MB/s 2024-04-17T10:31:50.214 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.214 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.214 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140186 s, 74.8 MB/s 2024-04-17T10:31:50.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.231 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.231 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144894 s, 72.4 MB/s 2024-04-17T10:31:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138706 s, 75.6 MB/s 2024-04-17T10:31:50.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.264 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160816 s, 65.2 MB/s 2024-04-17T10:31:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.280 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.280 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142723 s, 73.5 MB/s 2024-04-17T10:31:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157548 s, 66.6 MB/s 2024-04-17T10:31:50.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.316 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167422 s, 62.6 MB/s 2024-04-17T10:31:50.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197903 s, 53.0 MB/s 2024-04-17T10:31:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153122 s, 68.5 MB/s 2024-04-17T10:31:50.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142372 s, 73.7 MB/s 2024-04-17T10:31:50.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.388 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150585 s, 69.6 MB/s 2024-04-17T10:31:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.406 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164337 s, 63.8 MB/s 2024-04-17T10:31:50.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.422 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139368 s, 75.2 MB/s 2024-04-17T10:31:50.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.439 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153627 s, 68.3 MB/s 2024-04-17T10:31:50.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.457 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161067 s, 65.1 MB/s 2024-04-17T10:31:50.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.475 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154647 s, 67.8 MB/s 2024-04-17T10:31:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.490 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136565 s, 76.8 MB/s 2024-04-17T10:31:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133022 s, 78.8 MB/s 2024-04-17T10:31:50.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136298 s, 76.9 MB/s 2024-04-17T10:31:50.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:50.522 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:50.522 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:50.522 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.1 2024-04-17T10:31:50.522 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:50.522 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:50.522 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:50.523 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.591 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138586 s, 75.7 MB/s 2024-04-17T10:31:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143626 s, 73.0 MB/s 2024-04-17T10:31:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149563 s, 70.1 MB/s 2024-04-17T10:31:50.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.640 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143232 s, 73.2 MB/s 2024-04-17T10:31:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146659 s, 71.5 MB/s 2024-04-17T10:31:50.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138489 s, 75.7 MB/s 2024-04-17T10:31:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.689 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144556 s, 72.5 MB/s 2024-04-17T10:31:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.704 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014241 s, 73.6 MB/s 2024-04-17T10:31:50.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.720 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137037 s, 76.5 MB/s 2024-04-17T10:31:50.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.736 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144187 s, 72.7 MB/s 2024-04-17T10:31:50.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.753 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149793 s, 70.0 MB/s 2024-04-17T10:31:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.770 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154192 s, 68.0 MB/s 2024-04-17T10:31:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.787 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149087 s, 70.3 MB/s 2024-04-17T10:31:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131358 s, 79.8 MB/s 2024-04-17T10:31:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.817 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138169 s, 75.9 MB/s 2024-04-17T10:31:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.832 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013025 s, 80.5 MB/s 2024-04-17T10:31:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142538 s, 73.6 MB/s 2024-04-17T10:31:50.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.864 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133889 s, 78.3 MB/s 2024-04-17T10:31:50.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.881 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155938 s, 67.2 MB/s 2024-04-17T10:31:50.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.903 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016521 s, 63.5 MB/s 2024-04-17T10:31:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.917 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150068 s, 69.9 MB/s 2024-04-17T10:31:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146338 s, 71.7 MB/s 2024-04-17T10:31:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132442 s, 79.2 MB/s 2024-04-17T10:31:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152251 s, 68.9 MB/s 2024-04-17T10:31:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136904 s, 76.6 MB/s 2024-04-17T10:31:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149392 s, 70.2 MB/s 2024-04-17T10:31:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138378 s, 75.8 MB/s 2024-04-17T10:31:51.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.028 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134457 s, 78.0 MB/s 2024-04-17T10:31:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.043 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133195 s, 78.7 MB/s 2024-04-17T10:31:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.058 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129464 s, 81.0 MB/s 2024-04-17T10:31:51.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185166 s, 56.6 MB/s 2024-04-17T10:31:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150192 s, 69.8 MB/s 2024-04-17T10:31:51.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:51.097 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:51.097 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:51.097 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.2 2024-04-17T10:31:51.097 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:51.097 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:51.097 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:51.097 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:51.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156591 s, 67.0 MB/s 2024-04-17T10:31:51.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.189 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143821 s, 72.9 MB/s 2024-04-17T10:31:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173687 s, 60.4 MB/s 2024-04-17T10:31:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137702 s, 76.1 MB/s 2024-04-17T10:31:51.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.239 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131081 s, 80.0 MB/s 2024-04-17T10:31:51.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.254 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131578 s, 79.7 MB/s 2024-04-17T10:31:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154372 s, 67.9 MB/s 2024-04-17T10:31:51.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132588 s, 79.1 MB/s 2024-04-17T10:31:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.302 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133285 s, 78.7 MB/s 2024-04-17T10:31:51.317 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.317 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.317 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134795 s, 77.8 MB/s 2024-04-17T10:31:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.332 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132156 s, 79.3 MB/s 2024-04-17T10:31:51.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.348 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137085 s, 76.5 MB/s 2024-04-17T10:31:51.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.364 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149793 s, 70.0 MB/s 2024-04-17T10:31:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.379 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133762 s, 78.4 MB/s 2024-04-17T10:31:51.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.395 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134889 s, 77.7 MB/s 2024-04-17T10:31:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122086 s, 85.9 MB/s 2024-04-17T10:31:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180208 s, 58.2 MB/s 2024-04-17T10:31:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.445 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144191 s, 72.7 MB/s 2024-04-17T10:31:51.460 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137387 s, 76.3 MB/s 2024-04-17T10:31:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166634 s, 62.9 MB/s 2024-04-17T10:31:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013176 s, 79.6 MB/s 2024-04-17T10:31:51.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.511 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149446 s, 70.2 MB/s 2024-04-17T10:31:51.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.527 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138978 s, 75.4 MB/s 2024-04-17T10:31:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.542 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135999 s, 77.1 MB/s 2024-04-17T10:31:51.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.559 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148286 s, 70.7 MB/s 2024-04-17T10:31:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.578 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017617 s, 59.5 MB/s 2024-04-17T10:31:51.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.594 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134577 s, 77.9 MB/s 2024-04-17T10:31:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148246 s, 70.7 MB/s 2024-04-17T10:31:51.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.625 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133656 s, 78.5 MB/s 2024-04-17T10:31:51.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150035 s, 69.9 MB/s 2024-04-17T10:31:51.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.659 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155862 s, 67.3 MB/s 2024-04-17T10:31:51.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.674 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129865 s, 80.7 MB/s 2024-04-17T10:31:51.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:51.676 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:51.676 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:51.676 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.3 2024-04-17T10:31:51.676 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:51.676 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:51.676 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:51.676 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139633 s, 75.1 MB/s 2024-04-17T10:31:51.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.763 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013772 s, 76.1 MB/s 2024-04-17T10:31:51.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142332 s, 73.7 MB/s 2024-04-17T10:31:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.794 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133447 s, 78.6 MB/s 2024-04-17T10:31:51.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.811 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151791 s, 69.1 MB/s 2024-04-17T10:31:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130908 s, 80.1 MB/s 2024-04-17T10:31:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.843 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152876 s, 68.6 MB/s 2024-04-17T10:31:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.859 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138122 s, 75.9 MB/s 2024-04-17T10:31:51.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.875 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143804 s, 72.9 MB/s 2024-04-17T10:31:51.890 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.890 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.890 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136379 s, 76.9 MB/s 2024-04-17T10:31:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.906 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131263 s, 79.9 MB/s 2024-04-17T10:31:51.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134014 s, 78.2 MB/s 2024-04-17T10:31:51.936 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.936 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.936 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132989 s, 78.8 MB/s 2024-04-17T10:31:51.952 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150179 s, 69.8 MB/s 2024-04-17T10:31:51.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.969 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148126 s, 70.8 MB/s 2024-04-17T10:31:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:51.986 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154533 s, 67.9 MB/s 2024-04-17T10:31:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.003 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149497 s, 70.1 MB/s 2024-04-17T10:31:52.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.019 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137847 s, 76.1 MB/s 2024-04-17T10:31:52.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.034 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134276 s, 78.1 MB/s 2024-04-17T10:31:52.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.050 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138688 s, 75.6 MB/s 2024-04-17T10:31:52.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.066 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152473 s, 68.8 MB/s 2024-04-17T10:31:52.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132188 s, 79.3 MB/s 2024-04-17T10:31:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127024 s, 82.5 MB/s 2024-04-17T10:31:52.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.111 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013763 s, 76.2 MB/s 2024-04-17T10:31:52.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130657 s, 80.3 MB/s 2024-04-17T10:31:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.142 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013368 s, 78.4 MB/s 2024-04-17T10:31:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.158 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150634 s, 69.6 MB/s 2024-04-17T10:31:52.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.175 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015095 s, 69.5 MB/s 2024-04-17T10:31:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.192 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150808 s, 69.5 MB/s 2024-04-17T10:31:52.207 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134594 s, 77.9 MB/s 2024-04-17T10:31:52.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.225 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156344 s, 67.1 MB/s 2024-04-17T10:31:52.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.241 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136654 s, 76.7 MB/s 2024-04-17T10:31:52.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:52.242 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:52.242 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:52.242 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.4 2024-04-17T10:31:52.242 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:52.242 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:52.243 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:52.243 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.319 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175745 s, 59.7 MB/s 2024-04-17T10:31:52.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.334 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136301 s, 76.9 MB/s 2024-04-17T10:31:52.349 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.350 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131022 s, 80.0 MB/s 2024-04-17T10:31:52.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.365 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01346 s, 77.9 MB/s 2024-04-17T10:31:52.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.381 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148113 s, 70.8 MB/s 2024-04-17T10:31:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.398 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147941 s, 70.9 MB/s 2024-04-17T10:31:52.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.413 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129455 s, 81.0 MB/s 2024-04-17T10:31:52.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.428 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137817 s, 76.1 MB/s 2024-04-17T10:31:52.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.444 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140421 s, 74.7 MB/s 2024-04-17T10:31:52.459 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.459 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.460 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138224 s, 75.9 MB/s 2024-04-17T10:31:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.475 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133994 s, 78.3 MB/s 2024-04-17T10:31:52.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.490 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133573 s, 78.5 MB/s 2024-04-17T10:31:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150934 s, 69.5 MB/s 2024-04-17T10:31:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.523 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141855 s, 73.9 MB/s 2024-04-17T10:31:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.539 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139794 s, 75.0 MB/s 2024-04-17T10:31:52.556 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.556 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.556 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015216 s, 68.9 MB/s 2024-04-17T10:31:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.572 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140732 s, 74.5 MB/s 2024-04-17T10:31:52.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.589 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148045 s, 70.8 MB/s 2024-04-17T10:31:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149704 s, 70.0 MB/s 2024-04-17T10:31:52.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194038 s, 54.0 MB/s 2024-04-17T10:31:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132547 s, 79.1 MB/s 2024-04-17T10:31:52.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.659 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149231 s, 70.3 MB/s 2024-04-17T10:31:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180171 s, 58.2 MB/s 2024-04-17T10:31:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.694 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133123 s, 78.8 MB/s 2024-04-17T10:31:52.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.709 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136496 s, 76.8 MB/s 2024-04-17T10:31:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133544 s, 78.5 MB/s 2024-04-17T10:31:52.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.743 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141543 s, 74.1 MB/s 2024-04-17T10:31:52.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.757 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149545 s, 70.1 MB/s 2024-04-17T10:31:52.776 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.776 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.776 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173049 s, 60.6 MB/s 2024-04-17T10:31:52.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.796 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183609 s, 57.1 MB/s 2024-04-17T10:31:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.811 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133454 s, 78.6 MB/s 2024-04-17T10:31:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131766 s, 79.6 MB/s 2024-04-17T10:31:52.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:52.828 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:52.828 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:52.828 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.5 2024-04-17T10:31:52.828 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:52.828 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:52.828 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:52.828 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147977 s, 70.9 MB/s 2024-04-17T10:31:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187402 s, 56.0 MB/s 2024-04-17T10:31:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147117 s, 71.3 MB/s 2024-04-17T10:31:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014135 s, 74.2 MB/s 2024-04-17T10:31:52.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.969 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142171 s, 73.8 MB/s 2024-04-17T10:31:52.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:52.986 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146419 s, 71.6 MB/s 2024-04-17T10:31:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.002 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147797 s, 70.9 MB/s 2024-04-17T10:31:53.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.018 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014104 s, 74.3 MB/s 2024-04-17T10:31:53.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.033 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135153 s, 77.6 MB/s 2024-04-17T10:31:53.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.048 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128296 s, 81.7 MB/s 2024-04-17T10:31:53.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.064 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145504 s, 72.1 MB/s 2024-04-17T10:31:53.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.080 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134807 s, 77.8 MB/s 2024-04-17T10:31:53.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131831 s, 79.5 MB/s 2024-04-17T10:31:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.115 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182463 s, 57.5 MB/s 2024-04-17T10:31:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135185 s, 77.6 MB/s 2024-04-17T10:31:53.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.147 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147941 s, 70.9 MB/s 2024-04-17T10:31:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138039 s, 76.0 MB/s 2024-04-17T10:31:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.179 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141206 s, 74.3 MB/s 2024-04-17T10:31:53.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.193 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131765 s, 79.6 MB/s 2024-04-17T10:31:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.210 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150257 s, 69.8 MB/s 2024-04-17T10:31:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176828 s, 59.3 MB/s 2024-04-17T10:31:53.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.245 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013558 s, 77.3 MB/s 2024-04-17T10:31:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170392 s, 61.5 MB/s 2024-04-17T10:31:53.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130366 s, 80.4 MB/s 2024-04-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155564 s, 67.4 MB/s 2024-04-17T10:31:53.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143095 s, 73.3 MB/s 2024-04-17T10:31:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.327 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133667 s, 78.4 MB/s 2024-04-17T10:31:53.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.344 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150277 s, 69.8 MB/s 2024-04-17T10:31:53.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.362 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160838 s, 65.2 MB/s 2024-04-17T10:31:53.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142812 s, 73.4 MB/s 2024-04-17T10:31:53.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.395 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143612 s, 73.0 MB/s 2024-04-17T10:31:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133723 s, 78.4 MB/s 2024-04-17T10:31:53.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:53.411 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:53.411 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:53.411 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.6 2024-04-17T10:31:53.411 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:53.411 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:53.412 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:53.412 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:53.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.483 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133746 s, 78.4 MB/s 2024-04-17T10:31:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.504 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189395 s, 55.4 MB/s 2024-04-17T10:31:53.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.520 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143446 s, 73.1 MB/s 2024-04-17T10:31:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133855 s, 78.3 MB/s 2024-04-17T10:31:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149398 s, 70.2 MB/s 2024-04-17T10:31:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131594 s, 79.7 MB/s 2024-04-17T10:31:53.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131832 s, 79.5 MB/s 2024-04-17T10:31:53.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.600 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154255 s, 68.0 MB/s 2024-04-17T10:31:53.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.617 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143131 s, 73.3 MB/s 2024-04-17T10:31:53.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130313 s, 80.5 MB/s 2024-04-17T10:31:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.648 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144811 s, 72.4 MB/s 2024-04-17T10:31:53.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.663 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013273 s, 79.0 MB/s 2024-04-17T10:31:53.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.680 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154196 s, 68.0 MB/s 2024-04-17T10:31:53.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.697 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015494 s, 67.7 MB/s 2024-04-17T10:31:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.713 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143755 s, 72.9 MB/s 2024-04-17T10:31:53.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.728 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132315 s, 79.2 MB/s 2024-04-17T10:31:53.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.743 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126609 s, 82.8 MB/s 2024-04-17T10:31:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013838 s, 75.8 MB/s 2024-04-17T10:31:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.775 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013887 s, 75.5 MB/s 2024-04-17T10:31:53.791 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.791 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.791 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149658 s, 70.1 MB/s 2024-04-17T10:31:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.806 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132258 s, 79.3 MB/s 2024-04-17T10:31:53.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135541 s, 77.4 MB/s 2024-04-17T10:31:53.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.837 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138387 s, 75.8 MB/s 2024-04-17T10:31:53.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.852 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136447 s, 76.8 MB/s 2024-04-17T10:31:53.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.867 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132274 s, 79.3 MB/s 2024-04-17T10:31:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153142 s, 68.5 MB/s 2024-04-17T10:31:53.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.900 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140343 s, 74.7 MB/s 2024-04-17T10:31:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.915 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134166 s, 78.2 MB/s 2024-04-17T10:31:53.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.933 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154899 s, 67.7 MB/s 2024-04-17T10:31:53.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138538 s, 75.7 MB/s 2024-04-17T10:31:53.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195365 s, 53.7 MB/s 2024-04-17T10:31:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151958 s, 69.0 MB/s 2024-04-17T10:31:53.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:53.988 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:53.988 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:53.988 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.7 2024-04-17T10:31:53.988 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:53.989 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:53.989 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:53.989 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:54.059 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.060 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.060 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141179 s, 74.3 MB/s 2024-04-17T10:31:54.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.076 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140118 s, 74.8 MB/s 2024-04-17T10:31:54.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.093 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153935 s, 68.1 MB/s 2024-04-17T10:31:54.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.114 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191955 s, 54.6 MB/s 2024-04-17T10:31:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.128 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132118 s, 79.4 MB/s 2024-04-17T10:31:54.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.143 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129314 s, 81.1 MB/s 2024-04-17T10:31:54.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.159 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135344 s, 77.5 MB/s 2024-04-17T10:31:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.178 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172367 s, 60.8 MB/s 2024-04-17T10:31:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.193 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134012 s, 78.2 MB/s 2024-04-17T10:31:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.211 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160755 s, 65.2 MB/s 2024-04-17T10:31:54.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.227 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139739 s, 75.0 MB/s 2024-04-17T10:31:54.244 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.245 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153975 s, 68.1 MB/s 2024-04-17T10:31:54.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.261 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.261 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139841 s, 75.0 MB/s 2024-04-17T10:31:54.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.276 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135155 s, 77.6 MB/s 2024-04-17T10:31:54.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183601 s, 57.1 MB/s 2024-04-17T10:31:54.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153454 s, 68.3 MB/s 2024-04-17T10:31:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014128 s, 74.2 MB/s 2024-04-17T10:31:54.346 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014646 s, 71.6 MB/s 2024-04-17T10:31:54.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.363 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148223 s, 70.7 MB/s 2024-04-17T10:31:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136618 s, 76.8 MB/s 2024-04-17T10:31:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141458 s, 74.1 MB/s 2024-04-17T10:31:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142154 s, 73.8 MB/s 2024-04-17T10:31:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138266 s, 75.8 MB/s 2024-04-17T10:31:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.442 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134015 s, 78.2 MB/s 2024-04-17T10:31:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.457 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137325 s, 76.4 MB/s 2024-04-17T10:31:54.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.473 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139122 s, 75.4 MB/s 2024-04-17T10:31:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140332 s, 74.7 MB/s 2024-04-17T10:31:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133641 s, 78.5 MB/s 2024-04-17T10:31:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138019 s, 76.0 MB/s 2024-04-17T10:31:54.535 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.535 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.535 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134838 s, 77.8 MB/s 2024-04-17T10:31:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.551 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147194 s, 71.2 MB/s 2024-04-17T10:31:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013646 s, 76.8 MB/s 2024-04-17T10:31:54.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:54.568 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:54.568 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:54.568 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.8 2024-04-17T10:31:54.568 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:54.569 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:54.569 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:54.569 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.638 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128856 s, 81.4 MB/s 2024-04-17T10:31:54.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.654 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013613 s, 77.0 MB/s 2024-04-17T10:31:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.670 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138763 s, 75.6 MB/s 2024-04-17T10:31:54.687 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.687 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.687 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155768 s, 67.3 MB/s 2024-04-17T10:31:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.706 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167678 s, 62.5 MB/s 2024-04-17T10:31:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.721 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013451 s, 78.0 MB/s 2024-04-17T10:31:54.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171776 s, 61.0 MB/s 2024-04-17T10:31:54.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.754 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124833 s, 84.0 MB/s 2024-04-17T10:31:54.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.769 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132803 s, 79.0 MB/s 2024-04-17T10:31:54.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.785 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139769 s, 75.0 MB/s 2024-04-17T10:31:54.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135327 s, 77.5 MB/s 2024-04-17T10:31:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135823 s, 77.2 MB/s 2024-04-17T10:31:54.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.833 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151646 s, 69.1 MB/s 2024-04-17T10:31:54.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013564 s, 77.3 MB/s 2024-04-17T10:31:54.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137412 s, 76.3 MB/s 2024-04-17T10:31:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144741 s, 72.4 MB/s 2024-04-17T10:31:54.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.894 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124523 s, 84.2 MB/s 2024-04-17T10:31:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131589 s, 79.7 MB/s 2024-04-17T10:31:54.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.924 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.924 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124623 s, 84.1 MB/s 2024-04-17T10:31:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.940 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151855 s, 69.1 MB/s 2024-04-17T10:31:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148942 s, 70.4 MB/s 2024-04-17T10:31:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.972 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135629 s, 77.3 MB/s 2024-04-17T10:31:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151837 s, 69.1 MB/s 2024-04-17T10:31:55.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.006 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146113 s, 71.8 MB/s 2024-04-17T10:31:55.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.021 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136674 s, 76.7 MB/s 2024-04-17T10:31:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.037 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138673 s, 75.6 MB/s 2024-04-17T10:31:55.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.053 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137164 s, 76.4 MB/s 2024-04-17T10:31:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151327 s, 69.3 MB/s 2024-04-17T10:31:55.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.085 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140815 s, 74.5 MB/s 2024-04-17T10:31:55.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.100 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128223 s, 81.8 MB/s 2024-04-17T10:31:55.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.116 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136598 s, 76.8 MB/s 2024-04-17T10:31:55.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.131 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132829 s, 78.9 MB/s 2024-04-17T10:31:55.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:55.132 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:55.132 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:55.132 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.9 2024-04-17T10:31:55.132 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:55.132 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:55.132 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:55.133 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145449 s, 72.1 MB/s 2024-04-17T10:31:55.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.221 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149182 s, 70.3 MB/s 2024-04-17T10:31:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129695 s, 80.8 MB/s 2024-04-17T10:31:55.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174742 s, 60.0 MB/s 2024-04-17T10:31:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.270 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132894 s, 78.9 MB/s 2024-04-17T10:31:55.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140463 s, 74.7 MB/s 2024-04-17T10:31:55.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.303 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142535 s, 73.6 MB/s 2024-04-17T10:31:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.317 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014044 s, 74.7 MB/s 2024-04-17T10:31:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150065 s, 69.9 MB/s 2024-04-17T10:31:55.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.348 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118582 s, 88.4 MB/s 2024-04-17T10:31:55.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.363 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135086 s, 77.6 MB/s 2024-04-17T10:31:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129316 s, 81.1 MB/s 2024-04-17T10:31:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.394 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148668 s, 70.5 MB/s 2024-04-17T10:31:55.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132644 s, 79.1 MB/s 2024-04-17T10:31:55.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140998 s, 74.4 MB/s 2024-04-17T10:31:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.443 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155575 s, 67.4 MB/s 2024-04-17T10:31:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130813 s, 80.2 MB/s 2024-04-17T10:31:55.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.478 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181202 s, 57.9 MB/s 2024-04-17T10:31:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.495 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158191 s, 66.3 MB/s 2024-04-17T10:31:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.512 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148688 s, 70.5 MB/s 2024-04-17T10:31:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.528 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144506 s, 72.6 MB/s 2024-04-17T10:31:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161296 s, 65.0 MB/s 2024-04-17T10:31:55.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.562 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141422 s, 74.1 MB/s 2024-04-17T10:31:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.582 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182161 s, 57.6 MB/s 2024-04-17T10:31:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015436 s, 67.9 MB/s 2024-04-17T10:31:55.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.615 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135953 s, 77.1 MB/s 2024-04-17T10:31:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138097 s, 75.9 MB/s 2024-04-17T10:31:55.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.647 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148946 s, 70.4 MB/s 2024-04-17T10:31:55.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.663 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137094 s, 76.5 MB/s 2024-04-17T10:31:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134685 s, 77.9 MB/s 2024-04-17T10:31:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146349 s, 71.6 MB/s 2024-04-17T10:31:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140711 s, 74.5 MB/s 2024-04-17T10:31:55.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:55.712 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:55.712 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:55.712 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.10 2024-04-17T10:31:55.712 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:55.712 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:55.712 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:55.712 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:55.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.785 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147438 s, 71.1 MB/s 2024-04-17T10:31:55.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132316 s, 79.2 MB/s 2024-04-17T10:31:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.819 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174281 s, 60.2 MB/s 2024-04-17T10:31:55.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140125 s, 74.8 MB/s 2024-04-17T10:31:55.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.850 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129554 s, 80.9 MB/s 2024-04-17T10:31:55.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.870 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182659 s, 57.4 MB/s 2024-04-17T10:31:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01518 s, 69.1 MB/s 2024-04-17T10:31:55.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.907 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181638 s, 57.7 MB/s 2024-04-17T10:31:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155032 s, 67.6 MB/s 2024-04-17T10:31:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014612 s, 71.8 MB/s 2024-04-17T10:31:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137391 s, 76.3 MB/s 2024-04-17T10:31:55.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.972 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142563 s, 73.6 MB/s 2024-04-17T10:31:55.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:55.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:55.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129793 s, 80.8 MB/s 2024-04-17T10:31:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.002 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128193 s, 81.8 MB/s 2024-04-17T10:31:56.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.017 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133889 s, 78.3 MB/s 2024-04-17T10:31:56.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134075 s, 78.2 MB/s 2024-04-17T10:31:56.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.049 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151367 s, 69.3 MB/s 2024-04-17T10:31:56.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.065 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144896 s, 72.4 MB/s 2024-04-17T10:31:56.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.080 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013843 s, 75.7 MB/s 2024-04-17T10:31:56.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132714 s, 79.0 MB/s 2024-04-17T10:31:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128099 s, 81.9 MB/s 2024-04-17T10:31:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.126 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135088 s, 77.6 MB/s 2024-04-17T10:31:56.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.140 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.140 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012104 s, 86.6 MB/s 2024-04-17T10:31:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014552 s, 72.1 MB/s 2024-04-17T10:31:56.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134705 s, 77.8 MB/s 2024-04-17T10:31:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.187 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137615 s, 76.2 MB/s 2024-04-17T10:31:56.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.204 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150846 s, 69.5 MB/s 2024-04-17T10:31:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124886 s, 84.0 MB/s 2024-04-17T10:31:56.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.234 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135142 s, 77.6 MB/s 2024-04-17T10:31:56.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.249 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136949 s, 76.6 MB/s 2024-04-17T10:31:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.270 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189878 s, 55.2 MB/s 2024-04-17T10:31:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.285 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134619 s, 77.9 MB/s 2024-04-17T10:31:56.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:56.287 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:56.287 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:56.287 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.11 2024-04-17T10:31:56.287 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:56.287 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:56.287 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:56.287 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130805 s, 80.2 MB/s 2024-04-17T10:31:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142401 s, 73.6 MB/s 2024-04-17T10:31:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151425 s, 69.2 MB/s 2024-04-17T10:31:56.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.412 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182066 s, 57.6 MB/s 2024-04-17T10:31:56.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.428 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140895 s, 74.4 MB/s 2024-04-17T10:31:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015367 s, 68.2 MB/s 2024-04-17T10:31:56.465 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188391 s, 55.7 MB/s 2024-04-17T10:31:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171832 s, 61.0 MB/s 2024-04-17T10:31:56.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.502 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156495 s, 67.0 MB/s 2024-04-17T10:31:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148759 s, 70.5 MB/s 2024-04-17T10:31:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.534 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133049 s, 78.8 MB/s 2024-04-17T10:31:56.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.550 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014173 s, 74.0 MB/s 2024-04-17T10:31:56.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.566 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138095 s, 75.9 MB/s 2024-04-17T10:31:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152112 s, 68.9 MB/s 2024-04-17T10:31:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.598 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135553 s, 77.4 MB/s 2024-04-17T10:31:56.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136368 s, 76.9 MB/s 2024-04-17T10:31:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.630 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143686 s, 73.0 MB/s 2024-04-17T10:31:56.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.645 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133798 s, 78.4 MB/s 2024-04-17T10:31:56.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.663 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162087 s, 64.7 MB/s 2024-04-17T10:31:56.677 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131312 s, 79.9 MB/s 2024-04-17T10:31:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176609 s, 59.4 MB/s 2024-04-17T10:31:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145252 s, 72.2 MB/s 2024-04-17T10:31:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.729 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131994 s, 79.4 MB/s 2024-04-17T10:31:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172339 s, 60.8 MB/s 2024-04-17T10:31:56.768 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.768 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.768 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187542 s, 55.9 MB/s 2024-04-17T10:31:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147627 s, 71.0 MB/s 2024-04-17T10:31:56.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.801 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143316 s, 73.2 MB/s 2024-04-17T10:31:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.815 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127315 s, 82.4 MB/s 2024-04-17T10:31:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.830 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132405 s, 79.2 MB/s 2024-04-17T10:31:56.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.846 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140285 s, 74.7 MB/s 2024-04-17T10:31:56.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.863 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132967 s, 78.9 MB/s 2024-04-17T10:31:56.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.879 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153791 s, 68.2 MB/s 2024-04-17T10:31:56.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:56.881 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:56.881 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:56.881 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.12 2024-04-17T10:31:56.881 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:56.881 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:56.881 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:56.881 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:56.955 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.956 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137775 s, 76.1 MB/s 2024-04-17T10:31:56.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.972 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014714 s, 71.3 MB/s 2024-04-17T10:31:56.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:56.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:56.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133552 s, 78.5 MB/s 2024-04-17T10:31:57.004 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.004 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.005 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155606 s, 67.4 MB/s 2024-04-17T10:31:57.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.021 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144805 s, 72.4 MB/s 2024-04-17T10:31:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132294 s, 79.3 MB/s 2024-04-17T10:31:57.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.052 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145156 s, 72.2 MB/s 2024-04-17T10:31:57.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.068 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137862 s, 76.1 MB/s 2024-04-17T10:31:57.088 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.088 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.088 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181205 s, 57.9 MB/s 2024-04-17T10:31:57.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.103 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131098 s, 80.0 MB/s 2024-04-17T10:31:57.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.118 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134693 s, 77.8 MB/s 2024-04-17T10:31:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.135 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150207 s, 69.8 MB/s 2024-04-17T10:31:57.150 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.150 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.151 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139737 s, 75.0 MB/s 2024-04-17T10:31:57.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.165 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130795 s, 80.2 MB/s 2024-04-17T10:31:57.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.182 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145644 s, 72.0 MB/s 2024-04-17T10:31:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.195 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118773 s, 88.3 MB/s 2024-04-17T10:31:57.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.211 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136002 s, 77.1 MB/s 2024-04-17T10:31:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133969 s, 78.3 MB/s 2024-04-17T10:31:57.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.241 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133176 s, 78.7 MB/s 2024-04-17T10:31:57.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.256 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132342 s, 79.2 MB/s 2024-04-17T10:31:57.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.272 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137588 s, 76.2 MB/s 2024-04-17T10:31:57.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.287 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129245 s, 81.1 MB/s 2024-04-17T10:31:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.302 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132552 s, 79.1 MB/s 2024-04-17T10:31:57.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.318 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134857 s, 77.8 MB/s 2024-04-17T10:31:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137743 s, 76.1 MB/s 2024-04-17T10:31:57.349 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01501 s, 69.9 MB/s 2024-04-17T10:31:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133954 s, 78.3 MB/s 2024-04-17T10:31:57.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.379 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122435 s, 85.6 MB/s 2024-04-17T10:31:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.394 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134257 s, 78.1 MB/s 2024-04-17T10:31:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136844 s, 76.6 MB/s 2024-04-17T10:31:57.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.423 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118555 s, 88.4 MB/s 2024-04-17T10:31:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.438 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135884 s, 77.2 MB/s 2024-04-17T10:31:57.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:57.440 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:57.440 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:57.440 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.13 2024-04-17T10:31:57.440 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:57.440 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:57.440 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:57.440 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.511 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140711 s, 74.5 MB/s 2024-04-17T10:31:57.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.533 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197872 s, 53.0 MB/s 2024-04-17T10:31:57.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.548 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131744 s, 79.6 MB/s 2024-04-17T10:31:57.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.567 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170629 s, 61.5 MB/s 2024-04-17T10:31:57.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.582 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135094 s, 77.6 MB/s 2024-04-17T10:31:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014092 s, 74.4 MB/s 2024-04-17T10:31:57.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139032 s, 75.4 MB/s 2024-04-17T10:31:57.628 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:31:57.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.629 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137705 s, 76.1 MB/s 2024-04-17T10:31:57.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:31:57.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.645 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137995 s, 76.0 MB/s 2024-04-17T10:31:57.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.660 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013371 s, 78.4 MB/s 2024-04-17T10:31:57.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.675 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137275 s, 76.4 MB/s 2024-04-17T10:31:57.691 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.692 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136728 s, 76.7 MB/s 2024-04-17T10:31:57.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131494 s, 79.7 MB/s 2024-04-17T10:31:57.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136211 s, 77.0 MB/s 2024-04-17T10:31:57.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157 s, 66.8 MB/s 2024-04-17T10:31:57.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.755 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137882 s, 76.0 MB/s 2024-04-17T10:31:57.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.771 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142716 s, 73.5 MB/s 2024-04-17T10:31:57.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.787 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.787 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141429 s, 74.1 MB/s 2024-04-17T10:31:57.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.807 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.807 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182041 s, 57.6 MB/s 2024-04-17T10:31:57.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.824 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.824 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015094 s, 69.5 MB/s 2024-04-17T10:31:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.843 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170502 s, 61.5 MB/s 2024-04-17T10:31:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137031 s, 76.5 MB/s 2024-04-17T10:31:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155525 s, 67.4 MB/s 2024-04-17T10:31:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143671 s, 73.0 MB/s 2024-04-17T10:31:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134076 s, 78.2 MB/s 2024-04-17T10:31:57.924 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.924 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.924 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159564 s, 65.7 MB/s 2024-04-17T10:31:57.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.945 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186501 s, 56.2 MB/s 2024-04-17T10:31:57.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147173 s, 71.2 MB/s 2024-04-17T10:31:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137153 s, 76.5 MB/s 2024-04-17T10:31:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013643 s, 76.9 MB/s 2024-04-17T10:31:58.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.009 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146005 s, 71.8 MB/s 2024-04-17T10:31:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148861 s, 70.4 MB/s 2024-04-17T10:31:58.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:58.027 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:58.027 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:58.027 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.14 2024-04-17T10:31:58.027 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:58.027 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:58.028 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:58.028 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154316 s, 67.9 MB/s 2024-04-17T10:31:58.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.118 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151671 s, 69.1 MB/s 2024-04-17T10:31:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135006 s, 77.7 MB/s 2024-04-17T10:31:58.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.150 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.150 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148713 s, 70.5 MB/s 2024-04-17T10:31:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.166 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142632 s, 73.5 MB/s 2024-04-17T10:31:58.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.182 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150644 s, 69.6 MB/s 2024-04-17T10:31:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172064 s, 60.9 MB/s 2024-04-17T10:31:58.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145079 s, 72.3 MB/s 2024-04-17T10:31:58.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155141 s, 67.6 MB/s 2024-04-17T10:31:58.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158301 s, 66.2 MB/s 2024-04-17T10:31:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.268 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135014 s, 77.7 MB/s 2024-04-17T10:31:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.285 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152835 s, 68.6 MB/s 2024-04-17T10:31:58.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.302 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146573 s, 71.5 MB/s 2024-04-17T10:31:58.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.316 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128767 s, 81.4 MB/s 2024-04-17T10:31:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.333 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148313 s, 70.7 MB/s 2024-04-17T10:31:58.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.349 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.349 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139193 s, 75.3 MB/s 2024-04-17T10:31:58.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.363 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127055 s, 82.5 MB/s 2024-04-17T10:31:58.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.379 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144407 s, 72.6 MB/s 2024-04-17T10:31:58.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.395 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136986 s, 76.5 MB/s 2024-04-17T10:31:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.411 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138858 s, 75.5 MB/s 2024-04-17T10:31:58.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133972 s, 78.3 MB/s 2024-04-17T10:31:58.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131703 s, 79.6 MB/s 2024-04-17T10:31:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137158 s, 76.4 MB/s 2024-04-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137638 s, 76.2 MB/s 2024-04-17T10:31:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153962 s, 68.1 MB/s 2024-04-17T10:31:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138212 s, 75.9 MB/s 2024-04-17T10:31:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.524 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168482 s, 62.2 MB/s 2024-04-17T10:31:58.541 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157466 s, 66.6 MB/s 2024-04-17T10:31:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.562 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181038 s, 57.9 MB/s 2024-04-17T10:31:58.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.580 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154678 s, 67.8 MB/s 2024-04-17T10:31:58.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.597 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155272 s, 67.5 MB/s 2024-04-17T10:31:58.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151803 s, 69.1 MB/s 2024-04-17T10:31:58.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:58.616 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:58.616 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:58.616 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.15 2024-04-17T10:31:58.616 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:58.616 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:58.616 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:58.616 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011929 s, 87.9 MB/s 2024-04-17T10:31:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.701 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013734 s, 76.3 MB/s 2024-04-17T10:31:58.716 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.716 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.716 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138572 s, 75.7 MB/s 2024-04-17T10:31:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.731 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131802 s, 79.6 MB/s 2024-04-17T10:31:58.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138359 s, 75.8 MB/s 2024-04-17T10:31:58.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.762 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130739 s, 80.2 MB/s 2024-04-17T10:31:58.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.777 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130602 s, 80.3 MB/s 2024-04-17T10:31:58.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.797 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186646 s, 56.2 MB/s 2024-04-17T10:31:58.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.813 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138234 s, 75.9 MB/s 2024-04-17T10:31:58.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.829 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014187 s, 73.9 MB/s 2024-04-17T10:31:58.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.845 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146945 s, 71.4 MB/s 2024-04-17T10:31:58.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014137 s, 74.2 MB/s 2024-04-17T10:31:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172014 s, 61.0 MB/s 2024-04-17T10:31:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145663 s, 72.0 MB/s 2024-04-17T10:31:58.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128396 s, 81.7 MB/s 2024-04-17T10:31:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.927 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141097 s, 74.3 MB/s 2024-04-17T10:31:58.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.942 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133692 s, 78.4 MB/s 2024-04-17T10:31:58.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.962 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176994 s, 59.2 MB/s 2024-04-17T10:31:58.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.976 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129451 s, 81.0 MB/s 2024-04-17T10:31:58.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:58.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:58.993 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148377 s, 70.7 MB/s 2024-04-17T10:31:59.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.012 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017216 s, 60.9 MB/s 2024-04-17T10:31:59.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01645 s, 63.7 MB/s 2024-04-17T10:31:59.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134233 s, 78.1 MB/s 2024-04-17T10:31:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136132 s, 77.0 MB/s 2024-04-17T10:31:59.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150254 s, 69.8 MB/s 2024-04-17T10:31:59.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.098 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188209 s, 55.7 MB/s 2024-04-17T10:31:59.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.115 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144529 s, 72.6 MB/s 2024-04-17T10:31:59.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136088 s, 77.1 MB/s 2024-04-17T10:31:59.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.147 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148001 s, 70.8 MB/s 2024-04-17T10:31:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.163 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139398 s, 75.2 MB/s 2024-04-17T10:31:59.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014107 s, 74.3 MB/s 2024-04-17T10:31:59.194 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.194 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.194 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130769 s, 80.2 MB/s 2024-04-17T10:31:59.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:59.195 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:59.195 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:59.195 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.16 2024-04-17T10:31:59.195 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:59.195 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:59.195 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:59.196 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.263 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129447 s, 81.0 MB/s 2024-04-17T10:31:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.278 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137487 s, 76.3 MB/s 2024-04-17T10:31:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.295 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149504 s, 70.1 MB/s 2024-04-17T10:31:59.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.308 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115951 s, 90.4 MB/s 2024-04-17T10:31:59.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.324 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146938 s, 71.4 MB/s 2024-04-17T10:31:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.344 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182093 s, 57.6 MB/s 2024-04-17T10:31:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147612 s, 71.0 MB/s 2024-04-17T10:31:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.376 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136727 s, 76.7 MB/s 2024-04-17T10:31:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.397 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183369 s, 57.2 MB/s 2024-04-17T10:31:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.416 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175113 s, 59.9 MB/s 2024-04-17T10:31:59.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188314 s, 55.7 MB/s 2024-04-17T10:31:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178532 s, 58.7 MB/s 2024-04-17T10:31:59.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.473 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153212 s, 68.4 MB/s 2024-04-17T10:31:59.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.490 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147627 s, 71.0 MB/s 2024-04-17T10:31:59.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.506 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149006 s, 70.4 MB/s 2024-04-17T10:31:59.523 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152164 s, 68.9 MB/s 2024-04-17T10:31:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138634 s, 75.6 MB/s 2024-04-17T10:31:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156189 s, 67.1 MB/s 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138288 s, 75.8 MB/s 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137744 s, 76.1 MB/s 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139463 s, 75.2 MB/s 2024-04-17T10:31:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.619 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136955 s, 76.6 MB/s 2024-04-17T10:31:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.636 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151565 s, 69.2 MB/s 2024-04-17T10:31:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.651 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133966 s, 78.3 MB/s 2024-04-17T10:31:59.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.668 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149535 s, 70.1 MB/s 2024-04-17T10:31:59.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.684 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014326 s, 73.2 MB/s 2024-04-17T10:31:59.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.700 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138474 s, 75.7 MB/s 2024-04-17T10:31:59.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.715 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135828 s, 77.2 MB/s 2024-04-17T10:31:59.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.731 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.731 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013672 s, 76.7 MB/s 2024-04-17T10:31:59.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142694 s, 73.5 MB/s 2024-04-17T10:31:59.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.763 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146248 s, 71.7 MB/s 2024-04-17T10:31:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122492 s, 85.6 MB/s 2024-04-17T10:31:59.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:31:59.778 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:31:59.779 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:31:59.779 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.17 2024-04-17T10:31:59.779 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:31:59.779 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:31:59.779 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:31:59.779 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:31:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150176 s, 69.8 MB/s 2024-04-17T10:31:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122851 s, 85.4 MB/s 2024-04-17T10:31:59.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.882 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175864 s, 59.6 MB/s 2024-04-17T10:31:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130325 s, 80.5 MB/s 2024-04-17T10:31:59.912 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.912 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.912 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133143 s, 78.8 MB/s 2024-04-17T10:31:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013235 s, 79.2 MB/s 2024-04-17T10:31:59.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.942 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131975 s, 79.5 MB/s 2024-04-17T10:31:59.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.957 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130843 s, 80.1 MB/s 2024-04-17T10:31:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190602 s, 55.0 MB/s 2024-04-17T10:31:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:31:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:31:59.994 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014881 s, 70.5 MB/s 2024-04-17T10:32:00.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148052 s, 70.8 MB/s 2024-04-17T10:32:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.029 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162276 s, 64.6 MB/s 2024-04-17T10:32:00.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014526 s, 72.2 MB/s 2024-04-17T10:32:00.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.063 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153991 s, 68.1 MB/s 2024-04-17T10:32:00.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.080 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149371 s, 70.2 MB/s 2024-04-17T10:32:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.094 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120621 s, 86.9 MB/s 2024-04-17T10:32:00.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137705 s, 76.1 MB/s 2024-04-17T10:32:00.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.124 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133779 s, 78.4 MB/s 2024-04-17T10:32:00.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.142 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137809 s, 76.1 MB/s 2024-04-17T10:32:00.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135642 s, 77.3 MB/s 2024-04-17T10:32:00.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.175 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180775 s, 58.0 MB/s 2024-04-17T10:32:00.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138049 s, 76.0 MB/s 2024-04-17T10:32:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138447 s, 75.7 MB/s 2024-04-17T10:32:00.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178709 s, 58.7 MB/s 2024-04-17T10:32:00.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137515 s, 76.3 MB/s 2024-04-17T10:32:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133815 s, 78.4 MB/s 2024-04-17T10:32:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.274 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149782 s, 70.0 MB/s 2024-04-17T10:32:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.289 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013915 s, 75.4 MB/s 2024-04-17T10:32:00.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146743 s, 71.5 MB/s 2024-04-17T10:32:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.322 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139927 s, 74.9 MB/s 2024-04-17T10:32:00.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149466 s, 70.2 MB/s 2024-04-17T10:32:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145384 s, 72.1 MB/s 2024-04-17T10:32:00.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:00.356 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:00.356 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:00.356 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.18 2024-04-17T10:32:00.356 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:00.356 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:00.357 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:00.357 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163599 s, 64.1 MB/s 2024-04-17T10:32:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136972 s, 76.6 MB/s 2024-04-17T10:32:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140503 s, 74.6 MB/s 2024-04-17T10:32:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175532 s, 59.7 MB/s 2024-04-17T10:32:00.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.495 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131098 s, 80.0 MB/s 2024-04-17T10:32:00.509 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123539 s, 84.9 MB/s 2024-04-17T10:32:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.525 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131463 s, 79.8 MB/s 2024-04-17T10:32:00.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.543 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016387 s, 64.0 MB/s 2024-04-17T10:32:00.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.558 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137131 s, 76.5 MB/s 2024-04-17T10:32:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140086 s, 74.9 MB/s 2024-04-17T10:32:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188574 s, 55.6 MB/s 2024-04-17T10:32:00.611 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.611 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.611 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140289 s, 74.7 MB/s 2024-04-17T10:32:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014664 s, 71.5 MB/s 2024-04-17T10:32:00.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.644 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153145 s, 68.5 MB/s 2024-04-17T10:32:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.665 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185369 s, 56.6 MB/s 2024-04-17T10:32:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143512 s, 73.1 MB/s 2024-04-17T10:32:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.695 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.696 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129989 s, 80.7 MB/s 2024-04-17T10:32:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128806 s, 81.4 MB/s 2024-04-17T10:32:00.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.731 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.731 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185089 s, 56.7 MB/s 2024-04-17T10:32:00.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.746 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136613 s, 76.8 MB/s 2024-04-17T10:32:00.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.762 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138523 s, 75.7 MB/s 2024-04-17T10:32:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.780 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164653 s, 63.7 MB/s 2024-04-17T10:32:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.797 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150514 s, 69.7 MB/s 2024-04-17T10:32:00.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.812 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133275 s, 78.7 MB/s 2024-04-17T10:32:00.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.829 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148044 s, 70.8 MB/s 2024-04-17T10:32:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.844 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133761 s, 78.4 MB/s 2024-04-17T10:32:00.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.860 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144786 s, 72.4 MB/s 2024-04-17T10:32:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.876 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133399 s, 78.6 MB/s 2024-04-17T10:32:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143012 s, 73.3 MB/s 2024-04-17T10:32:00.909 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.909 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.909 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156593 s, 67.0 MB/s 2024-04-17T10:32:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.926 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149129 s, 70.3 MB/s 2024-04-17T10:32:00.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:00.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:00.942 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137198 s, 76.4 MB/s 2024-04-17T10:32:00.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:00.943 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:00.943 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:00.943 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.19 2024-04-17T10:32:00.943 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:00.943 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:00.943 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:00.943 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133299 s, 78.7 MB/s 2024-04-17T10:32:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159091 s, 65.9 MB/s 2024-04-17T10:32:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.044 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136577 s, 76.8 MB/s 2024-04-17T10:32:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151647 s, 69.1 MB/s 2024-04-17T10:32:01.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.077 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.077 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013796 s, 76.0 MB/s 2024-04-17T10:32:01.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.092 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141558 s, 74.1 MB/s 2024-04-17T10:32:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.107 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132069 s, 79.4 MB/s 2024-04-17T10:32:01.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135678 s, 77.3 MB/s 2024-04-17T10:32:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154956 s, 67.7 MB/s 2024-04-17T10:32:01.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125398 s, 83.6 MB/s 2024-04-17T10:32:01.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014755 s, 71.1 MB/s 2024-04-17T10:32:01.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.190 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168474 s, 62.2 MB/s 2024-04-17T10:32:01.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.209 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174351 s, 60.1 MB/s 2024-04-17T10:32:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133448 s, 78.6 MB/s 2024-04-17T10:32:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.239 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129113 s, 81.2 MB/s 2024-04-17T10:32:01.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.256 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148074 s, 70.8 MB/s 2024-04-17T10:32:01.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137272 s, 76.4 MB/s 2024-04-17T10:32:01.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.287 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143284 s, 73.2 MB/s 2024-04-17T10:32:01.307 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186041 s, 56.4 MB/s 2024-04-17T10:32:01.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130477 s, 80.4 MB/s 2024-04-17T10:32:01.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133609 s, 78.5 MB/s 2024-04-17T10:32:01.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.354 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143189 s, 73.2 MB/s 2024-04-17T10:32:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152216 s, 68.9 MB/s 2024-04-17T10:32:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148751 s, 70.5 MB/s 2024-04-17T10:32:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.403 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131863 s, 79.5 MB/s 2024-04-17T10:32:01.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144939 s, 72.3 MB/s 2024-04-17T10:32:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01536 s, 68.3 MB/s 2024-04-17T10:32:01.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.452 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139039 s, 75.4 MB/s 2024-04-17T10:32:01.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137957 s, 76.0 MB/s 2024-04-17T10:32:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.484 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150388 s, 69.7 MB/s 2024-04-17T10:32:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169224 s, 62.0 MB/s 2024-04-17T10:32:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150099 s, 69.9 MB/s 2024-04-17T10:32:01.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:01.521 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:01.521 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:01.522 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.20 2024-04-17T10:32:01.522 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:01.522 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:01.522 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:01.522 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:01.592 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.593 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154268 s, 68.0 MB/s 2024-04-17T10:32:01.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147127 s, 71.3 MB/s 2024-04-17T10:32:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.626 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149548 s, 70.1 MB/s 2024-04-17T10:32:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.643 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01526 s, 68.7 MB/s 2024-04-17T10:32:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.659 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142191 s, 73.7 MB/s 2024-04-17T10:32:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012476 s, 84.0 MB/s 2024-04-17T10:32:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.689 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135621 s, 77.3 MB/s 2024-04-17T10:32:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.706 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154493 s, 67.9 MB/s 2024-04-17T10:32:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.721 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133889 s, 78.3 MB/s 2024-04-17T10:32:01.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.736 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130119 s, 80.6 MB/s 2024-04-17T10:32:01.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.753 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151393 s, 69.3 MB/s 2024-04-17T10:32:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016359 s, 64.1 MB/s 2024-04-17T10:32:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.786 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132427 s, 79.2 MB/s 2024-04-17T10:32:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.802 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135467 s, 77.4 MB/s 2024-04-17T10:32:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.817 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137148 s, 76.5 MB/s 2024-04-17T10:32:01.834 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152034 s, 69.0 MB/s 2024-04-17T10:32:01.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.851 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149785 s, 70.0 MB/s 2024-04-17T10:32:01.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.865 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121719 s, 86.1 MB/s 2024-04-17T10:32:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149201 s, 70.3 MB/s 2024-04-17T10:32:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142999 s, 73.3 MB/s 2024-04-17T10:32:01.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.917 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174373 s, 60.1 MB/s 2024-04-17T10:32:01.934 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.934 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.934 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014965 s, 70.1 MB/s 2024-04-17T10:32:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132453 s, 79.2 MB/s 2024-04-17T10:32:01.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013625 s, 77.0 MB/s 2024-04-17T10:32:01.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.982 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015164 s, 69.1 MB/s 2024-04-17T10:32:01.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:01.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:01.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140243 s, 74.8 MB/s 2024-04-17T10:32:02.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148948 s, 70.4 MB/s 2024-04-17T10:32:02.031 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157491 s, 66.6 MB/s 2024-04-17T10:32:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.052 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183802 s, 57.0 MB/s 2024-04-17T10:32:02.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.067 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135372 s, 77.5 MB/s 2024-04-17T10:32:02.083 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.083 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.083 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013606 s, 77.1 MB/s 2024-04-17T10:32:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.099 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146688 s, 71.5 MB/s 2024-04-17T10:32:02.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:02.101 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:02.101 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:02.101 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.21 2024-04-17T10:32:02.101 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:02.101 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:02.101 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:02.102 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:02.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.170 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129792 s, 80.8 MB/s 2024-04-17T10:32:02.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145811 s, 71.9 MB/s 2024-04-17T10:32:02.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133418 s, 78.6 MB/s 2024-04-17T10:32:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148631 s, 70.5 MB/s 2024-04-17T10:32:02.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135089 s, 77.6 MB/s 2024-04-17T10:32:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119436 s, 87.8 MB/s 2024-04-17T10:32:02.265 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.265 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.265 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016157 s, 64.9 MB/s 2024-04-17T10:32:02.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.281 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014079 s, 74.5 MB/s 2024-04-17T10:32:02.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135998 s, 77.1 MB/s 2024-04-17T10:32:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132769 s, 79.0 MB/s 2024-04-17T10:32:02.328 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.328 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.328 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148362 s, 70.7 MB/s 2024-04-17T10:32:02.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.344 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140892 s, 74.4 MB/s 2024-04-17T10:32:02.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.361 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014812 s, 70.8 MB/s 2024-04-17T10:32:02.377 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.377 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.377 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147753 s, 71.0 MB/s 2024-04-17T10:32:02.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125069 s, 83.8 MB/s 2024-04-17T10:32:02.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.409 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015118 s, 69.4 MB/s 2024-04-17T10:32:02.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.427 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165968 s, 63.2 MB/s 2024-04-17T10:32:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133591 s, 78.5 MB/s 2024-04-17T10:32:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.457 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134459 s, 78.0 MB/s 2024-04-17T10:32:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.473 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135433 s, 77.4 MB/s 2024-04-17T10:32:02.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143926 s, 72.9 MB/s 2024-04-17T10:32:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141348 s, 74.2 MB/s 2024-04-17T10:32:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141453 s, 74.1 MB/s 2024-04-17T10:32:02.535 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.536 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132991 s, 78.8 MB/s 2024-04-17T10:32:02.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.552 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014017 s, 74.8 MB/s 2024-04-17T10:32:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140895 s, 74.4 MB/s 2024-04-17T10:32:02.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.584 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152366 s, 68.8 MB/s 2024-04-17T10:32:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.603 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167859 s, 62.5 MB/s 2024-04-17T10:32:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.618 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131227 s, 79.9 MB/s 2024-04-17T10:32:02.637 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.637 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.637 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171775 s, 61.0 MB/s 2024-04-17T10:32:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012499 s, 83.9 MB/s 2024-04-17T10:32:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.669 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155146 s, 67.6 MB/s 2024-04-17T10:32:02.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:02.670 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:02.670 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:02.670 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.22 2024-04-17T10:32:02.670 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:02.670 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:02.670 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:02.670 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:02.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.739 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131379 s, 79.8 MB/s 2024-04-17T10:32:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.758 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171399 s, 61.2 MB/s 2024-04-17T10:32:02.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.773 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134754 s, 77.8 MB/s 2024-04-17T10:32:02.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.788 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131429 s, 79.8 MB/s 2024-04-17T10:32:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.804 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138196 s, 75.9 MB/s 2024-04-17T10:32:02.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.820 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134259 s, 78.1 MB/s 2024-04-17T10:32:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180764 s, 58.0 MB/s 2024-04-17T10:32:02.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146116 s, 71.8 MB/s 2024-04-17T10:32:02.871 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.871 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.871 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139906 s, 74.9 MB/s 2024-04-17T10:32:02.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.886 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132337 s, 79.2 MB/s 2024-04-17T10:32:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.903 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145238 s, 72.2 MB/s 2024-04-17T10:32:02.919 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.920 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149615 s, 70.1 MB/s 2024-04-17T10:32:02.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.935 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139417 s, 75.2 MB/s 2024-04-17T10:32:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.951 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136113 s, 77.0 MB/s 2024-04-17T10:32:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149013 s, 70.4 MB/s 2024-04-17T10:32:02.983 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:02.983 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:02.983 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137592 s, 76.2 MB/s 2024-04-17T10:32:03.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.003 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186365 s, 56.3 MB/s 2024-04-17T10:32:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.018 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131552 s, 79.7 MB/s 2024-04-17T10:32:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149222 s, 70.3 MB/s 2024-04-17T10:32:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.050 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131132 s, 80.0 MB/s 2024-04-17T10:32:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.066 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144643 s, 72.5 MB/s 2024-04-17T10:32:03.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.081 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134579 s, 77.9 MB/s 2024-04-17T10:32:03.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.097 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.097 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134802 s, 77.8 MB/s 2024-04-17T10:32:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.115 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167958 s, 62.4 MB/s 2024-04-17T10:32:03.133 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.134 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165383 s, 63.4 MB/s 2024-04-17T10:32:03.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.149 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137969 s, 76.0 MB/s 2024-04-17T10:32:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.163 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121754 s, 86.1 MB/s 2024-04-17T10:32:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.179 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138261 s, 75.8 MB/s 2024-04-17T10:32:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.196 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149335 s, 70.2 MB/s 2024-04-17T10:32:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138064 s, 75.9 MB/s 2024-04-17T10:32:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128179 s, 81.8 MB/s 2024-04-17T10:32:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139574 s, 75.1 MB/s 2024-04-17T10:32:03.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:03.243 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:03.243 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:03.243 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.23 2024-04-17T10:32:03.243 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:03.243 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:03.243 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:03.243 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:03.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.316 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147436 s, 71.1 MB/s 2024-04-17T10:32:03.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132131 s, 79.4 MB/s 2024-04-17T10:32:03.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014607 s, 71.8 MB/s 2024-04-17T10:32:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.363 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136122 s, 77.0 MB/s 2024-04-17T10:32:03.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137793 s, 76.1 MB/s 2024-04-17T10:32:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.397 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169769 s, 61.8 MB/s 2024-04-17T10:32:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.413 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146434 s, 71.6 MB/s 2024-04-17T10:32:03.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.429 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142614 s, 73.5 MB/s 2024-04-17T10:32:03.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149875 s, 70.0 MB/s 2024-04-17T10:32:03.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132893 s, 78.9 MB/s 2024-04-17T10:32:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129398 s, 81.0 MB/s 2024-04-17T10:32:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153449 s, 68.3 MB/s 2024-04-17T10:32:03.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.515 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194016 s, 54.0 MB/s 2024-04-17T10:32:03.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.533 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015193 s, 69.0 MB/s 2024-04-17T10:32:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.551 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161708 s, 64.8 MB/s 2024-04-17T10:32:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131338 s, 79.8 MB/s 2024-04-17T10:32:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153439 s, 68.3 MB/s 2024-04-17T10:32:03.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.600 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143043 s, 73.3 MB/s 2024-04-17T10:32:03.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.615 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01398 s, 75.0 MB/s 2024-04-17T10:32:03.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132013 s, 79.4 MB/s 2024-04-17T10:32:03.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.648 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.648 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153978 s, 68.1 MB/s 2024-04-17T10:32:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140955 s, 74.4 MB/s 2024-04-17T10:32:03.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.679 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.679 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134869 s, 77.7 MB/s 2024-04-17T10:32:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.694 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137219 s, 76.4 MB/s 2024-04-17T10:32:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.711 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149453 s, 70.2 MB/s 2024-04-17T10:32:03.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.729 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157606 s, 66.5 MB/s 2024-04-17T10:32:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.749 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185234 s, 56.6 MB/s 2024-04-17T10:32:03.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.765 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136662 s, 76.7 MB/s 2024-04-17T10:32:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.780 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137947 s, 76.0 MB/s 2024-04-17T10:32:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.797 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150047 s, 69.9 MB/s 2024-04-17T10:32:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.812 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129632 s, 80.9 MB/s 2024-04-17T10:32:03.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.827 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136389 s, 76.9 MB/s 2024-04-17T10:32:03.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:03.829 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:03.829 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:03.829 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.24 2024-04-17T10:32:03.829 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:03.829 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:03.829 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:03.829 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.905 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162086 s, 64.7 MB/s 2024-04-17T10:32:03.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135657 s, 77.3 MB/s 2024-04-17T10:32:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142424 s, 73.6 MB/s 2024-04-17T10:32:03.952 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140667 s, 74.5 MB/s 2024-04-17T10:32:03.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.969 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148575 s, 70.6 MB/s 2024-04-17T10:32:03.983 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:03.984 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:03.984 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124654 s, 84.1 MB/s 2024-04-17T10:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185822 s, 56.4 MB/s 2024-04-17T10:32:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014502 s, 72.3 MB/s 2024-04-17T10:32:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139787 s, 75.0 MB/s 2024-04-17T10:32:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147463 s, 71.1 MB/s 2024-04-17T10:32:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178797 s, 58.6 MB/s 2024-04-17T10:32:04.088 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.088 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149956 s, 69.9 MB/s 2024-04-17T10:32:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137956 s, 76.0 MB/s 2024-04-17T10:32:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149721 s, 70.0 MB/s 2024-04-17T10:32:04.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.138 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.138 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015154 s, 69.2 MB/s 2024-04-17T10:32:04.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143177 s, 73.2 MB/s 2024-04-17T10:32:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.169 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134306 s, 78.1 MB/s 2024-04-17T10:32:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150629 s, 69.6 MB/s 2024-04-17T10:32:04.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.201 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132645 s, 79.1 MB/s 2024-04-17T10:32:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.216 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132207 s, 79.3 MB/s 2024-04-17T10:32:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139273 s, 75.3 MB/s 2024-04-17T10:32:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.248 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141264 s, 74.2 MB/s 2024-04-17T10:32:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.263 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138191 s, 75.9 MB/s 2024-04-17T10:32:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.278 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128304 s, 81.7 MB/s 2024-04-17T10:32:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129086 s, 81.2 MB/s 2024-04-17T10:32:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132083 s, 79.4 MB/s 2024-04-17T10:32:04.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.323 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130703 s, 80.2 MB/s 2024-04-17T10:32:04.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.340 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147602 s, 71.0 MB/s 2024-04-17T10:32:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133963 s, 78.3 MB/s 2024-04-17T10:32:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014232 s, 73.7 MB/s 2024-04-17T10:32:04.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014836 s, 70.7 MB/s 2024-04-17T10:32:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183917 s, 57.0 MB/s 2024-04-17T10:32:04.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:04.409 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:04.409 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:04.409 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.25 2024-04-17T10:32:04.409 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:04.409 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:04.409 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:04.409 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183537 s, 57.1 MB/s 2024-04-17T10:32:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135809 s, 77.2 MB/s 2024-04-17T10:32:04.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.516 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151991 s, 69.0 MB/s 2024-04-17T10:32:04.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137195 s, 76.4 MB/s 2024-04-17T10:32:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.547 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136276 s, 76.9 MB/s 2024-04-17T10:32:04.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140144 s, 74.8 MB/s 2024-04-17T10:32:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187134 s, 56.0 MB/s 2024-04-17T10:32:04.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.599 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142261 s, 73.7 MB/s 2024-04-17T10:32:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155273 s, 67.5 MB/s 2024-04-17T10:32:04.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.633 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149263 s, 70.3 MB/s 2024-04-17T10:32:04.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.650 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154143 s, 68.0 MB/s 2024-04-17T10:32:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.667 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014463 s, 72.5 MB/s 2024-04-17T10:32:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139428 s, 75.2 MB/s 2024-04-17T10:32:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.698 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138872 s, 75.5 MB/s 2024-04-17T10:32:04.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139147 s, 75.4 MB/s 2024-04-17T10:32:04.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.729 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132897 s, 78.9 MB/s 2024-04-17T10:32:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133633 s, 78.5 MB/s 2024-04-17T10:32:04.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.766 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190316 s, 55.1 MB/s 2024-04-17T10:32:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145984 s, 71.8 MB/s 2024-04-17T10:32:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.797 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013289 s, 78.9 MB/s 2024-04-17T10:32:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017904 s, 58.6 MB/s 2024-04-17T10:32:04.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.833 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149745 s, 70.0 MB/s 2024-04-17T10:32:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147642 s, 71.0 MB/s 2024-04-17T10:32:04.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.866 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146442 s, 71.6 MB/s 2024-04-17T10:32:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012999 s, 80.7 MB/s 2024-04-17T10:32:04.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.896 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140735 s, 74.5 MB/s 2024-04-17T10:32:04.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.913 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149953 s, 69.9 MB/s 2024-04-17T10:32:04.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.928 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127552 s, 82.2 MB/s 2024-04-17T10:32:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149799 s, 70.0 MB/s 2024-04-17T10:32:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.964 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179053 s, 58.6 MB/s 2024-04-17T10:32:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150586 s, 69.6 MB/s 2024-04-17T10:32:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014704 s, 71.3 MB/s 2024-04-17T10:32:04.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:04.999 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:04.999 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:04.999 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.26 2024-04-17T10:32:04.999 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:04.999 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:04.999 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:04.999 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.071 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134857 s, 77.8 MB/s 2024-04-17T10:32:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013762 s, 76.2 MB/s 2024-04-17T10:32:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131753 s, 79.6 MB/s 2024-04-17T10:32:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.119 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015362 s, 68.3 MB/s 2024-04-17T10:32:05.134 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.135 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133368 s, 78.6 MB/s 2024-04-17T10:32:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153656 s, 68.2 MB/s 2024-04-17T10:32:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177093 s, 59.2 MB/s 2024-04-17T10:32:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133825 s, 78.4 MB/s 2024-04-17T10:32:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.200 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.201 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127816 s, 82.0 MB/s 2024-04-17T10:32:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.216 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134999 s, 77.7 MB/s 2024-04-17T10:32:05.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.232 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014759 s, 71.0 MB/s 2024-04-17T10:32:05.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.248 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133896 s, 78.3 MB/s 2024-04-17T10:32:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135931 s, 77.1 MB/s 2024-04-17T10:32:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148843 s, 70.4 MB/s 2024-04-17T10:32:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.295 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135247 s, 77.5 MB/s 2024-04-17T10:32:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.310 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131435 s, 79.8 MB/s 2024-04-17T10:32:05.326 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.326 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.326 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014159 s, 74.1 MB/s 2024-04-17T10:32:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.342 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.343 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146515 s, 71.6 MB/s 2024-04-17T10:32:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130175 s, 80.6 MB/s 2024-04-17T10:32:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132966 s, 78.9 MB/s 2024-04-17T10:32:05.390 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.390 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.390 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015789 s, 66.4 MB/s 2024-04-17T10:32:05.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182811 s, 57.4 MB/s 2024-04-17T10:32:05.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.425 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013233 s, 79.2 MB/s 2024-04-17T10:32:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.445 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177159 s, 59.2 MB/s 2024-04-17T10:32:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145141 s, 72.2 MB/s 2024-04-17T10:32:05.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.482 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189448 s, 55.3 MB/s 2024-04-17T10:32:05.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128852 s, 81.4 MB/s 2024-04-17T10:32:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.512 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013842 s, 75.8 MB/s 2024-04-17T10:32:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.528 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013893 s, 75.5 MB/s 2024-04-17T10:32:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.549 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193703 s, 54.1 MB/s 2024-04-17T10:32:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176214 s, 59.5 MB/s 2024-04-17T10:32:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.584 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134131 s, 78.2 MB/s 2024-04-17T10:32:05.585 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:05.585 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:05.585 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:05.585 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.27 2024-04-17T10:32:05.585 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:05.585 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:05.585 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:05.586 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:05.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.656 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147222 s, 71.2 MB/s 2024-04-17T10:32:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138933 s, 75.5 MB/s 2024-04-17T10:32:05.687 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.687 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.687 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138412 s, 75.8 MB/s 2024-04-17T10:32:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153159 s, 68.5 MB/s 2024-04-17T10:32:05.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.721 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139841 s, 75.0 MB/s 2024-04-17T10:32:05.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.736 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137301 s, 76.4 MB/s 2024-04-17T10:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.751 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139762 s, 75.0 MB/s 2024-04-17T10:32:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133628 s, 78.5 MB/s 2024-04-17T10:32:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.782 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133844 s, 78.3 MB/s 2024-04-17T10:32:05.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.797 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133804 s, 78.4 MB/s 2024-04-17T10:32:05.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.813 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139368 s, 75.2 MB/s 2024-04-17T10:32:05.828 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.828 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.828 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134344 s, 78.1 MB/s 2024-04-17T10:32:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137663 s, 76.2 MB/s 2024-04-17T10:32:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149885 s, 70.0 MB/s 2024-04-17T10:32:05.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140725 s, 74.5 MB/s 2024-04-17T10:32:05.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134876 s, 77.7 MB/s 2024-04-17T10:32:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.907 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127694 s, 82.1 MB/s 2024-04-17T10:32:05.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141117 s, 74.3 MB/s 2024-04-17T10:32:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132223 s, 79.3 MB/s 2024-04-17T10:32:05.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137207 s, 76.4 MB/s 2024-04-17T10:32:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.968 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013476 s, 77.8 MB/s 2024-04-17T10:32:05.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:05.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:05.985 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150708 s, 69.6 MB/s 2024-04-17T10:32:06.000 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.001 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132536 s, 79.1 MB/s 2024-04-17T10:32:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136198 s, 77.0 MB/s 2024-04-17T10:32:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140233 s, 74.8 MB/s 2024-04-17T10:32:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148548 s, 70.6 MB/s 2024-04-17T10:32:06.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.067 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017323 s, 60.5 MB/s 2024-04-17T10:32:06.087 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.087 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.087 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179661 s, 58.4 MB/s 2024-04-17T10:32:06.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.102 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130146 s, 80.6 MB/s 2024-04-17T10:32:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.117 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132965 s, 78.9 MB/s 2024-04-17T10:32:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.132 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013391 s, 78.3 MB/s 2024-04-17T10:32:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131679 s, 79.6 MB/s 2024-04-17T10:32:06.149 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:06.149 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:06.149 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:06.149 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.28 2024-04-17T10:32:06.149 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:06.149 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:06.149 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:06.149 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:06.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.219 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134202 s, 78.1 MB/s 2024-04-17T10:32:06.236 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.236 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.236 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151549 s, 69.2 MB/s 2024-04-17T10:32:06.250 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.250 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.250 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125002 s, 83.9 MB/s 2024-04-17T10:32:06.265 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.265 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.265 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132045 s, 79.4 MB/s 2024-04-17T10:32:06.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.282 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146325 s, 71.7 MB/s 2024-04-17T10:32:06.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135816 s, 77.2 MB/s 2024-04-17T10:32:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151503 s, 69.2 MB/s 2024-04-17T10:32:06.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.330 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140825 s, 74.5 MB/s 2024-04-17T10:32:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178979 s, 58.6 MB/s 2024-04-17T10:32:06.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.364 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123778 s, 84.7 MB/s 2024-04-17T10:32:06.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.379 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135837 s, 77.2 MB/s 2024-04-17T10:32:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149021 s, 70.4 MB/s 2024-04-17T10:32:06.415 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.415 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.415 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175726 s, 59.7 MB/s 2024-04-17T10:32:06.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.431 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142624 s, 73.5 MB/s 2024-04-17T10:32:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134228 s, 78.1 MB/s 2024-04-17T10:32:06.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.462 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136851 s, 76.6 MB/s 2024-04-17T10:32:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.478 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141891 s, 73.9 MB/s 2024-04-17T10:32:06.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.493 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133936 s, 78.3 MB/s 2024-04-17T10:32:06.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146655 s, 71.5 MB/s 2024-04-17T10:32:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.527 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151958 s, 69.0 MB/s 2024-04-17T10:32:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.542 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131893 s, 79.5 MB/s 2024-04-17T10:32:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.557 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134641 s, 77.9 MB/s 2024-04-17T10:32:06.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.575 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155847 s, 67.3 MB/s 2024-04-17T10:32:06.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.594 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176844 s, 59.3 MB/s 2024-04-17T10:32:06.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137076 s, 76.5 MB/s 2024-04-17T10:32:06.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.626 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141618 s, 74.0 MB/s 2024-04-17T10:32:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147207 s, 71.2 MB/s 2024-04-17T10:32:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.658 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143647 s, 73.0 MB/s 2024-04-17T10:32:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.676 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161987 s, 64.7 MB/s 2024-04-17T10:32:06.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.692 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143235 s, 73.2 MB/s 2024-04-17T10:32:06.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.709 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147386 s, 71.1 MB/s 2024-04-17T10:32:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.725 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151133 s, 69.4 MB/s 2024-04-17T10:32:06.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:06.727 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:06.727 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:06.727 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.29 2024-04-17T10:32:06.727 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:06.727 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:06.727 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:06.727 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:06.794 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.794 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.794 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131227 s, 79.9 MB/s 2024-04-17T10:32:06.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130623 s, 80.3 MB/s 2024-04-17T10:32:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.825 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142193 s, 73.7 MB/s 2024-04-17T10:32:06.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.840 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132384 s, 79.2 MB/s 2024-04-17T10:32:06.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.856 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141972 s, 73.9 MB/s 2024-04-17T10:32:06.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.871 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126852 s, 82.7 MB/s 2024-04-17T10:32:06.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.888 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151869 s, 69.0 MB/s 2024-04-17T10:32:06.902 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.903 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130572 s, 80.3 MB/s 2024-04-17T10:32:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.917 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127534 s, 82.2 MB/s 2024-04-17T10:32:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.933 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137429 s, 76.3 MB/s 2024-04-17T10:32:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135982 s, 77.1 MB/s 2024-04-17T10:32:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155509 s, 67.4 MB/s 2024-04-17T10:32:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132514 s, 79.1 MB/s 2024-04-17T10:32:06.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:06.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:06.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152964 s, 68.6 MB/s 2024-04-17T10:32:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127963 s, 81.9 MB/s 2024-04-17T10:32:07.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130826 s, 80.2 MB/s 2024-04-17T10:32:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140557 s, 74.6 MB/s 2024-04-17T10:32:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133191 s, 78.7 MB/s 2024-04-17T10:32:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.074 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013538 s, 77.5 MB/s 2024-04-17T10:32:07.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148717 s, 70.5 MB/s 2024-04-17T10:32:07.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.105 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134798 s, 77.8 MB/s 2024-04-17T10:32:07.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150576 s, 69.6 MB/s 2024-04-17T10:32:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118461 s, 88.5 MB/s 2024-04-17T10:32:07.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.151 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136381 s, 76.9 MB/s 2024-04-17T10:32:07.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.170 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170222 s, 61.6 MB/s 2024-04-17T10:32:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.185 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124067 s, 84.5 MB/s 2024-04-17T10:32:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153641 s, 68.2 MB/s 2024-04-17T10:32:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013283 s, 78.9 MB/s 2024-04-17T10:32:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166907 s, 62.8 MB/s 2024-04-17T10:32:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138482 s, 75.7 MB/s 2024-04-17T10:32:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013304 s, 78.8 MB/s 2024-04-17T10:32:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134494 s, 78.0 MB/s 2024-04-17T10:32:07.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:07.283 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:07.283 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:07.284 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.30 2024-04-17T10:32:07.284 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:07.284 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:07.284 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:07.284 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.352 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145054 s, 72.3 MB/s 2024-04-17T10:32:07.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.372 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188048 s, 55.8 MB/s 2024-04-17T10:32:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130911 s, 80.1 MB/s 2024-04-17T10:32:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185216 s, 56.6 MB/s 2024-04-17T10:32:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.425 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149969 s, 69.9 MB/s 2024-04-17T10:32:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193017 s, 54.3 MB/s 2024-04-17T10:32:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013643 s, 76.9 MB/s 2024-04-17T10:32:07.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.476 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013397 s, 78.3 MB/s 2024-04-17T10:32:07.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.493 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145502 s, 72.1 MB/s 2024-04-17T10:32:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.508 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136415 s, 76.9 MB/s 2024-04-17T10:32:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.529 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189217 s, 55.4 MB/s 2024-04-17T10:32:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134719 s, 77.8 MB/s 2024-04-17T10:32:07.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.560 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138338 s, 75.8 MB/s 2024-04-17T10:32:07.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.575 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139002 s, 75.4 MB/s 2024-04-17T10:32:07.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145044 s, 72.3 MB/s 2024-04-17T10:32:07.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160657 s, 65.3 MB/s 2024-04-17T10:32:07.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.627 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016094 s, 65.2 MB/s 2024-04-17T10:32:07.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.643 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141919 s, 73.9 MB/s 2024-04-17T10:32:07.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.658 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132329 s, 79.2 MB/s 2024-04-17T10:32:07.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.674 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135186 s, 77.6 MB/s 2024-04-17T10:32:07.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.694 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182487 s, 57.5 MB/s 2024-04-17T10:32:07.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.708 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121198 s, 86.5 MB/s 2024-04-17T10:32:07.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014694 s, 71.4 MB/s 2024-04-17T10:32:07.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.742 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016043 s, 65.4 MB/s 2024-04-17T10:32:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.760 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158294 s, 66.2 MB/s 2024-04-17T10:32:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.777 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149708 s, 70.0 MB/s 2024-04-17T10:32:07.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.793 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139881 s, 75.0 MB/s 2024-04-17T10:32:07.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215433 s, 48.7 MB/s 2024-04-17T10:32:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.831 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139329 s, 75.3 MB/s 2024-04-17T10:32:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148876 s, 70.4 MB/s 2024-04-17T10:32:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016929 s, 61.9 MB/s 2024-04-17T10:32:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143546 s, 73.0 MB/s 2024-04-17T10:32:07.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:07.884 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:07.885 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:07.885 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.31 2024-04-17T10:32:07.885 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:07.885 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:07.885 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:07.885 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185554 s, 56.5 MB/s 2024-04-17T10:32:07.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:07.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138953 s, 75.5 MB/s 2024-04-17T10:32:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190824 s, 54.9 MB/s 2024-04-17T10:32:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013457 s, 77.9 MB/s 2024-04-17T10:32:08.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.034 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129369 s, 81.1 MB/s 2024-04-17T10:32:08.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.047 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141779 s, 74.0 MB/s 2024-04-17T10:32:08.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.062 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131328 s, 79.8 MB/s 2024-04-17T10:32:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134782 s, 77.8 MB/s 2024-04-17T10:32:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128386 s, 81.7 MB/s 2024-04-17T10:32:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.111 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016733 s, 62.7 MB/s 2024-04-17T10:32:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.126 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134914 s, 77.7 MB/s 2024-04-17T10:32:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.141 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129786 s, 80.8 MB/s 2024-04-17T10:32:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.159 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016028 s, 65.4 MB/s 2024-04-17T10:32:08.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.176 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146329 s, 71.7 MB/s 2024-04-17T10:32:08.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.190 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133037 s, 78.8 MB/s 2024-04-17T10:32:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137219 s, 76.4 MB/s 2024-04-17T10:32:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131216 s, 79.9 MB/s 2024-04-17T10:32:08.237 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.238 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146254 s, 71.7 MB/s 2024-04-17T10:32:08.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013643 s, 76.9 MB/s 2024-04-17T10:32:08.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.268 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133227 s, 78.7 MB/s 2024-04-17T10:32:08.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.283 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129319 s, 81.1 MB/s 2024-04-17T10:32:08.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.299 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147343 s, 71.2 MB/s 2024-04-17T10:32:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131865 s, 79.5 MB/s 2024-04-17T10:32:08.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.329 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131398 s, 79.8 MB/s 2024-04-17T10:32:08.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.345 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.345 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013644 s, 76.9 MB/s 2024-04-17T10:32:08.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.360 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135642 s, 77.3 MB/s 2024-04-17T10:32:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135865 s, 77.2 MB/s 2024-04-17T10:32:08.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.393 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151911 s, 69.0 MB/s 2024-04-17T10:32:08.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136726 s, 76.7 MB/s 2024-04-17T10:32:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116278 s, 90.2 MB/s 2024-04-17T10:32:08.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124663 s, 84.1 MB/s 2024-04-17T10:32:08.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:08.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:08.452 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138983 s, 75.4 MB/s 2024-04-17T10:32:08.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:08.453 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub start /scrub_control_test_path recursive 2024-04-17T10:32:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.655+0000 7f3d26a00640 1 Processor -- start 2024-04-17T10:32:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.655+0000 7f3d26a00640 1 -- start start 2024-04-17T10:32:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.655+0000 7f3d26a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d201447e0 0x7f3d2006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.655+0000 7f3d26a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2006d9a0 con 0x7f3d20071a70 2024-04-17T10:32:08.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.655+0000 7f3d26a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2006dc80 con 0x7f3d20144f00 2024-04-17T10:32:08.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.655+0000 7f3d26a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d200cc310 con 0x7f3d201447e0 2024-04-17T10:32:08.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.655+0000 7f3d24a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3d20071a70 0x7f3d200cc7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32860/0 (socket says 172.21.15.67:32860) 2024-04-17T10:32:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d24a00640 1 -- 172.21.15.67:0/3574863795 learned_addr learned my addr 172.21.15.67:0/3574863795 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/3574863795 >> v2:172.21.15.86:3300/0 conn(0x7f3d201447e0 0x7f3d2006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1f400640 1 -- 172.21.15.67:0/3574863795 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783512152 0 0) 0x7f3d2006d9a0 con 0x7f3d20071a70 2024-04-17T10:32:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1f400640 1 -- 172.21.15.67:0/3574863795 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d04003530 con 0x7f3d20071a70 2024-04-17T10:32:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1fe00640 1 -- 172.21.15.67:0/3574863795 >> v1:172.21.15.86:6789/0 conn(0x7f3d20144f00 legacy=0x7f3d20071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1fe00640 1 -- 172.21.15.67:0/3574863795 >> v1:172.21.15.67:6789/0 conn(0x7f3d20071a70 legacy=0x7f3d200cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1fe00640 1 -- 172.21.15.67:0/3574863795 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2006df60 con 0x7f3d201447e0 2024-04-17T10:32:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/3574863795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2006d3c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3d100097c0 tx=0x7f3d1001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd95bad0ebe3c312 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1f400640 1 -- 172.21.15.67:0/3574863795 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100ab600 con 0x7f3d201447e0 2024-04-17T10:32:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1f400640 1 -- 172.21.15.67:0/3574863795 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d100b0070 con 0x7f3d201447e0 2024-04-17T10:32:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.659+0000 7f3d1f400640 1 -- 172.21.15.67:0/3574863795 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100ab9c0 con 0x7f3d201447e0 2024-04-17T10:32:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- 172.21.15.67:0/3574863795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 msgr2=0x7f3d2006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 --2- 172.21.15.67:0/3574863795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2006d3c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f3d100097c0 tx=0x7f3d1001c3e0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- 172.21.15.67:0/3574863795 shutdown_connections 2024-04-17T10:32:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 --2- 172.21.15.67:0/3574863795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2006d3c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- 172.21.15.67:0/3574863795 >> 172.21.15.67:0/3574863795 conn(0x7f3d2006ac90 msgr2=0x7f3d2006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- 172.21.15.67:0/3574863795 shutdown_connections 2024-04-17T10:32:08.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- 172.21.15.67:0/3574863795 wait complete. 2024-04-17T10:32:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 Processor -- start 2024-04-17T10:32:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- start start 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20071a70 0x7f3d2013dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2013f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d20144f00 0x7f3d20158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d200cda90 con 0x7f3d20071a70 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d200cd910 con 0x7f3d201447e0 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d26a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d201695f0 con 0x7f3d20144f00 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20071a70 0x7f3d2013dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20071a70 0x7f3d2013dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40024/0 (socket says 172.21.15.67:40024) 2024-04-17T10:32:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 -- 172.21.15.67:0/1196299461 learned_addr learned my addr 172.21.15.67:0/1196299461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d17400640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d24a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d20144f00 0x7f3d20158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 -- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d20144f00 msgr2=0x7f3d20158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d20144f00 0x7f3d20158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 -- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 msgr2=0x7f3d2013f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2013f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 -- 172.21.15.67:0/1196299461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2013fe50 con 0x7f3d20071a70 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d17400640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d24a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d20144f00 0x7f3d20158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.663+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20071a70 0x7f3d2013dcf0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f3d100ad040 tx=0x7f3d1001ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.667+0000 7f3d1cc00640 1 -- 172.21.15.67:0/1196299461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100bf610 con 0x7f3d20071a70 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.667+0000 7f3d1cc00640 1 -- 172.21.15.67:0/1196299461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d100bf7d0 con 0x7f3d20071a70 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.667+0000 7f3d1cc00640 1 -- 172.21.15.67:0/1196299461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100bfbc0 con 0x7f3d20071a70 2024-04-17T10:32:08.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.667+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d20140130 con 0x7f3d20071a70 2024-04-17T10:32:08.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.667+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d201406f0 con 0x7f3d20071a70 2024-04-17T10:32:08.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.667+0000 7f3d1cc00640 1 -- 172.21.15.67:0/1196299461 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3d100bd020 con 0x7f3d20071a70 2024-04-17T10:32:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.667+0000 7f3d1cc00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cfc066f60 0x7f3cfc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d1cc00640 1 -- 172.21.15.67:0/1196299461 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d1011fa50 con 0x7f3d20071a70 2024-04-17T10:32:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 Processor -- start 2024-04-17T10:32:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cfc066f60 0x7f3cfc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- start start 2024-04-17T10:32:08.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cfc066f60 0x7f3cfc069420 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3d20159990 tx=0x7f3d08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb5617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb517fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb518520 0x55f2eb518900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb365d50 con 0x55f2eb561d10 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb2ede40 con 0x55f2eb5613f0 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb361df0 con 0x55f2eb518520 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb517fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb517fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40036/0 (socket says 172.21.15.67:40036) 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 -- 172.21.15.67:0/1293675440 learned_addr learned my addr 172.21.15.67:0/1293675440 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb5617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d24a00640 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb518520 0x55f2eb518900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 -- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb518520 msgr2=0x55f2eb518900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb518520 0x55f2eb518900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 -- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 msgr2=0x55f2eb5617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb5617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 -- 172.21.15.67:0/1293675440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2eb302d90 con 0x55f2eb561d10 2024-04-17T10:32:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb5617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d24a00640 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb518520 0x55f2eb518900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d17400640 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb517fe0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800d830 tx=0x7f3d08010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45ebcdcbe07b35f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d14c00640 1 -- 172.21.15.67:0/1293675440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08014ba0 con 0x55f2eb561d10 2024-04-17T10:32:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d14c00640 1 -- 172.21.15.67:0/1293675440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08014d40 con 0x55f2eb561d10 2024-04-17T10:32:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d14c00640 1 -- 172.21.15.67:0/1293675440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d08007220 con 0x55f2eb561d10 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- 172.21.15.67:0/1293675440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 msgr2=0x55f2eb517fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb517fe0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800d830 tx=0x7f3d08010000 comp rx=0 tx=0).stop 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- 172.21.15.67:0/1293675440 shutdown_connections 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb518520 0x55f2eb518900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb517fe0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/1293675440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb5617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- 172.21.15.67:0/1293675440 >> 172.21.15.67:0/1293675440 conn(0x55f2eb5144d0 msgr2=0x55f2eb5168c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- 172.21.15.67:0/1293675440 shutdown_connections 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.671+0000 7f3d29ec7000 1 -- 172.21.15.67:0/1293675440 wait complete. 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 -- start start 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 Processor -- start 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb518520 0x55f2eb568540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb568a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb522bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb30b610 con 0x55f2eb561d10 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb3209a0 con 0x55f2eb518520 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb36a380 con 0x55f2eb5613f0 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 10 client.0 _ll_register_callbacks cb 0x55f2eb41a0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb522bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d17400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb568a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d1fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb518520 0x55f2eb568540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb522bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40052/0 (socket says 172.21.15.67:40052) 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 -- 172.21.15.67:0/2412417518 learned_addr learned my addr 172.21.15.67:0/2412417518 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 -- 172.21.15.67:0/2412417518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 msgr2=0x55f2eb568a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb568a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 -- 172.21.15.67:0/2412417518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb518520 msgr2=0x55f2eb568540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb518520 0x55f2eb568540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 -- 172.21.15.67:0/2412417518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2eb360d40 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d24a00640 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb522bf0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800e7f0 tx=0x7f3d1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9010a701aa486a40 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d0d600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d0d600640 1 -- 172.21.15.67:0/2412417518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1800cb30 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d0d600640 1 -- 172.21.15.67:0/2412417518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d1800ccd0 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d08014d40 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d0d600640 1 -- 172.21.15.67:0/2412417518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d18020050 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 10 client.7869 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55f2eb3e8f00 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d0d600640 1 -- 172.21.15.67:0/2412417518 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d08014d40 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d0d600640 1 -- 172.21.15.67:0/2412417518 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3d18023020 con 0x55f2eb561d10 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 10 client.7869 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 10 client.7869 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 20 client.7869 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb523960 0x55f2eb523d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 4 client.7869 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:32:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2eb301a20 con 0x55f2eb523960 2024-04-17T10:32:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.675+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb523960 0x55f2eb523d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb523960 0x55f2eb523d70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13d82fbc5ceb97ff server_cookie=a466872a7cd0879c in_seq=0 out_seq=0 2024-04-17T10:32:08.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d0d600640 10 client.7869 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d0d600640 1 -- 172.21.15.67:0/2412417518 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f2eb301a20 con 0x55f2eb523960 2024-04-17T10:32:08.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d0d600640 10 client.7869 handle_command_reply: tid=0 2024-04-17T10:32:08.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 client.7869 shutdown 2024-04-17T10:32:08.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 msgr2=0x55f2eb522bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb522bf0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800e7f0 tx=0x7f3d1800ecb0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 shutdown_connections 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb523960 0x55f2eb523d70 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb561d10 0x55f2eb522bf0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb568a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/2412417518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb518520 0x55f2eb568540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d0d600640 0 client.7869 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 >> 172.21.15.67:0/2412417518 conn(0x55f2eb5144d0 msgr2=0x55f2eb517150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 shutdown_connections 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 1 -- 172.21.15.67:0/2412417518 wait complete. 2024-04-17T10:32:08.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.679+0000 7f3d29ec7000 20 client.7869 trim_cache size 0 max 16384 2024-04-17T10:32:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 Processor -- start 2024-04-17T10:32:08.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- start start 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb566120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb566660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb561d10 0x55f2eb560630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20200ec0 con 0x55f2eb518520 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d202000a0 con 0x55f2eb5613f0 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20137500 con 0x55f2eb561d10 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb566120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb566120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40062/0 (socket says 172.21.15.67:40062) 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 -- 172.21.15.67:0/861740409 learned_addr learned my addr 172.21.15.67:0/861740409 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d24a00640 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb561d10 0x55f2eb560630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d17400640 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb566660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 -- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb561d10 msgr2=0x55f2eb560630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb561d10 0x55f2eb560630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 -- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 msgr2=0x55f2eb566660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb566660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 -- 172.21.15.67:0/861740409 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2eb3bf330 con 0x55f2eb518520 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d24a00640 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb561d10 0x55f2eb560630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d17400640 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb566660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb566120 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f3d10018ff0 tx=0x7f3d100b9000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b6c4ead59936e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d0fe00640 1 -- 172.21.15.67:0/861740409 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100c4040 con 0x55f2eb518520 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d0fe00640 1 -- 172.21.15.67:0/861740409 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d100b0040 con 0x55f2eb518520 2024-04-17T10:32:08.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d0fe00640 1 -- 172.21.15.67:0/861740409 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100f4430 con 0x55f2eb518520 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- 172.21.15.67:0/861740409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 msgr2=0x55f2eb566120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb566120 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f3d10018ff0 tx=0x7f3d100b9000 comp rx=0 tx=0).stop 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- 172.21.15.67:0/861740409 shutdown_connections 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb561d10 0x55f2eb560630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb5613f0 0x55f2eb566660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/861740409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb566120 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- 172.21.15.67:0/861740409 >> 172.21.15.67:0/861740409 conn(0x55f2eb5144d0 msgr2=0x55f2eb55ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- 172.21.15.67:0/861740409 shutdown_connections 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.695+0000 7f3d29ec7000 1 -- 172.21.15.67:0/861740409 wait complete. 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 -- start start 2024-04-17T10:32:08.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 Processor -- start 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb565f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb566440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb561d10 0x55f2eb57ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb31cdc0 con 0x55f2eb518520 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d20137b00 con 0x55f2eb561d10 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f2eb365060 con 0x55f2eb5613f0 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 10 client.0 _ll_register_callbacks cb 0x7f3d08001350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb565f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb565f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40068/0 (socket says 172.21.15.67:40068) 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 -- 172.21.15.67:0/3802924444 learned_addr learned my addr 172.21.15.67:0/3802924444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d24a00640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb561d10 0x55f2eb57ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d17400640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb566440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 -- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 msgr2=0x55f2eb566440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb566440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 -- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb561d10 msgr2=0x55f2eb57ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb561d10 0x55f2eb57ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 -- 172.21.15.67:0/3802924444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2eb3325f0 con 0x55f2eb518520 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d17400640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb566440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d1fe00640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb565f00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f3d1001c950 tx=0x7f3d10019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef12d22f3d0295f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d0d600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d0d600640 1 -- 172.21.15.67:0/3802924444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d100fc030 con 0x55f2eb518520 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d0d600640 1 -- 172.21.15.67:0/3802924444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3d1011de60 con 0x55f2eb518520 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d100b0040 con 0x55f2eb518520 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d0d600640 1 -- 172.21.15.67:0/3802924444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d100c4070 con 0x55f2eb518520 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 10 client.7875 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55f2eb30cfe0 con 0x55f2eb518520 2024-04-17T10:32:08.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d0d600640 1 -- 172.21.15.67:0/3802924444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d100b0040 con 0x55f2eb518520 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d0d600640 1 -- 172.21.15.67:0/3802924444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3d100ec950 con 0x55f2eb518520 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 10 client.7875 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 10 client.7875 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 20 client.7875 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb565ab0 0x55f2eb57f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 4 client.7875 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x55f2eb373290 con 0x55f2eb565ab0 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d17400640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb565ab0 0x55f2eb57f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d0d600640 10 client.7875 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.699+0000 7f3d17400640 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb565ab0 0x55f2eb57f140 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70fab3ddf284544e server_cookie=d82fb5d8e7dc09f8 in_seq=0 out_seq=0 2024-04-17T10:32:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d0d600640 1 -- 172.21.15.67:0/3802924444 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f2eb373290 con 0x55f2eb565ab0 2024-04-17T10:32:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d0d600640 10 client.7875 handle_command_reply: tid=0 2024-04-17T10:32:08.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 client.7875 shutdown 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 msgr2=0x55f2eb565f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb565f00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f3d1001c950 tx=0x7f3d10019180 comp rx=0 tx=0).stop 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 shutdown_connections 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55f2eb565ab0 0x55f2eb57f140 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2eb561d10 0x55f2eb57ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2eb5613f0 0x55f2eb566440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 --2- 172.21.15.67:0/3802924444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2eb518520 0x55f2eb565f00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 >> 172.21.15.67:0/3802924444 conn(0x55f2eb5144d0 msgr2=0x55f2eb56eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d0d600640 0 client.7875 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 shutdown_connections 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 1 -- 172.21.15.67:0/3802924444 wait complete. 2024-04-17T10:32:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d29ec7000 20 client.7875 trim_cache size 0 max 16384 2024-04-17T10:32:08.698 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:32:08.698 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:32:08.698 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "2d8f24ea-0773-494c-b32e-a02e95716720", 2024-04-17T10:32:08.698 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:32:08.698 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cfc066f60 msgr2=0x7f3cfc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cfc066f60 0x7f3cfc069420 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3d20159990 tx=0x7f3d08009040 comp rx=0 tx=0).stop 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20071a70 msgr2=0x7f3d2013dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20071a70 0x7f3d2013dcf0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f3d100ad040 tx=0x7f3d1001ac20 comp rx=0 tx=0).stop 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 shutdown_connections 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cfc066f60 0x7f3cfc069420 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d20144f00 0x7f3d20158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d201447e0 0x7f3d2013f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 --2- 172.21.15.67:0/1196299461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d20071a70 0x7f3d2013dcf0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.703+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 >> 172.21.15.67:0/1196299461 conn(0x7f3d2006ac90 msgr2=0x7f3d2006e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.707+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 shutdown_connections 2024-04-17T10:32:08.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.707+0000 7f3d26a00640 1 -- 172.21.15.67:0/1196299461 wait complete. 2024-04-17T10:32:08.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:08.723 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.6:0 scrub abort 2024-04-17T10:32:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa30a00640 1 Processor -- start 2024-04-17T10:32:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa30a00640 1 -- start start 2024-04-17T10:32:08.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa30a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c06eaa0 0x7ffa2c1651e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa30a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0ce020 con 0x7ffa2c15a590 2024-04-17T10:32:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa30a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa2c0ce300 con 0x7ffa2c0cf0c0 2024-04-17T10:32:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa30a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cc8b0 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa2a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ffa2c15a590 0x7ffa2c0cd450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32874/0 (socket says 172.21.15.67:32874) 2024-04-17T10:32:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa2a800640 1 -- 172.21.15.67:0/368643533 learned_addr learned my addr 172.21.15.67:0/368643533 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:08.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.911+0000 7ffa2b200640 1 --2- 172.21.15.67:0/368643533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c06eaa0 0x7ffa2c1651e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa29e00640 1 -- 172.21.15.67:0/368643533 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100664219 0 0) 0x7ffa2c0ce020 con 0x7ffa2c15a590 2024-04-17T10:32:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa29e00640 1 -- 172.21.15.67:0/368643533 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa10005740 con 0x7ffa2c15a590 2024-04-17T10:32:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa2b200640 1 -- 172.21.15.67:0/368643533 >> v1:172.21.15.86:6789/0 conn(0x7ffa2c0cf0c0 legacy=0x7ffa2c06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa29e00640 1 -- 172.21.15.67:0/368643533 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508754185 0 0) 0x7ffa2c0ce300 con 0x7ffa2c0cf0c0 2024-04-17T10:32:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa2b200640 1 -- 172.21.15.67:0/368643533 >> v1:172.21.15.67:6789/0 conn(0x7ffa2c15a590 legacy=0x7ffa2c0cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa2b200640 1 -- 172.21.15.67:0/368643533 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c0ce5e0 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa2b200640 1 --2- 172.21.15.67:0/368643533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c06eaa0 0x7ffa2c1651e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009dc0 tx=0x7ffa2401c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7df26d8e76256020 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa29e00640 1 -- 172.21.15.67:0/368643533 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240b2070 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa29e00640 1 -- 172.21.15.67:0/368643533 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa240a5920 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.915+0000 7ffa29e00640 1 -- 172.21.15.67:0/368643533 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa240a5cd0 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- 172.21.15.67:0/368643533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c06eaa0 msgr2=0x7ffa2c1651e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 --2- 172.21.15.67:0/368643533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c06eaa0 0x7ffa2c1651e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009dc0 tx=0x7ffa2401c960 comp rx=0 tx=0).stop 2024-04-17T10:32:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- 172.21.15.67:0/368643533 shutdown_connections 2024-04-17T10:32:08.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 --2- 172.21.15.67:0/368643533 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c06eaa0 0x7ffa2c1651e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- 172.21.15.67:0/368643533 >> 172.21.15.67:0/368643533 conn(0x7ffa2c06ac90 msgr2=0x7ffa2c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- 172.21.15.67:0/368643533 shutdown_connections 2024-04-17T10:32:08.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- 172.21.15.67:0/368643533 wait complete. 2024-04-17T10:32:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 Processor -- start 2024-04-17T10:32:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- start start 2024-04-17T10:32:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa2c06eaa0 0x7ffa2c0b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c0cf0c0 0x7ffa2c148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c15a590 0x7ffa2c156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c15b980 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0cddb0 con 0x7ffa2c0cf0c0 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa30a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c15bb00 con 0x7ffa2c15a590 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa2c06eaa0 0x7ffa2c0b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa2c06eaa0 0x7ffa2c0b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40082/0 (socket says 172.21.15.67:40082) 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 -- 172.21.15.67:0/228783691 learned_addr learned my addr 172.21.15.67:0/228783691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 -- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c15a590 msgr2=0x7ffa2c156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c15a590 0x7ffa2c156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 -- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c0cf0c0 msgr2=0x7ffa2c148870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c0cf0c0 0x7ffa2c148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 -- 172.21.15.67:0/228783691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2c148db0 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.919+0000 7ffa2a800640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa2c06eaa0 0x7ffa2c0b3130 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c480 tx=0x7ffa1c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa22a00640 1 -- 172.21.15.67:0/228783691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c017070 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa22a00640 1 -- 172.21.15.67:0/228783691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c005db0 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa22a00640 1 -- 172.21.15.67:0/228783691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c007290 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2c0b2090 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2c0b2650 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa22a00640 1 -- 172.21.15.67:0/228783691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ffa1c01a070 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa22a00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa08066f60 0x7ffa08069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa22a00640 1 -- 172.21.15.67:0/228783691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa1c084660 con 0x7ffa2c06eaa0 2024-04-17T10:32:08.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa23e00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa08066f60 0x7ffa08069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa3497e000 1 Processor -- start 2024-04-17T10:32:08.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.923+0000 7ffa3497e000 1 -- start start 2024-04-17T10:32:08.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa23e00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa08066f60 0x7ffa08069420 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ffa14007fd0 tx=0x7ffa14007940 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06ca1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca2360 0x55ad06c5c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06c5d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad06b3f1f0 con 0x55ad06ca1a40 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad06aa5e90 con 0x55ad06ca2360 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad06a2df50 con 0x55ad06c5cd90 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06ca1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06ca1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40096/0 (socket says 172.21.15.67:40096) 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 -- 172.21.15.67:0/2371788191 learned_addr learned my addr 172.21.15.67:0/2371788191 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa23e00640 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca2360 0x55ad06c5c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2b200640 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06c5d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 -- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 msgr2=0x55ad06c5d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06c5d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 -- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca2360 msgr2=0x55ad06c5c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca2360 0x55ad06c5c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 -- 172.21.15.67:0/2371788191 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad06a7d8b0 con 0x55ad06ca1a40 2024-04-17T10:32:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06ca1e20 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c05cfb0 tx=0x7ffa1c054dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c58f59fbf67f51fb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa1b400640 1 -- 172.21.15.67:0/2371788191 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0530d0 con 0x55ad06ca1a40 2024-04-17T10:32:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa1b400640 1 -- 172.21.15.67:0/2371788191 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c05b0b0 con 0x55ad06ca1a40 2024-04-17T10:32:08.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa1b400640 1 -- 172.21.15.67:0/2371788191 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c05d050 con 0x55ad06ca1a40 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- 172.21.15.67:0/2371788191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 msgr2=0x55ad06ca1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06ca1e20 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c05cfb0 tx=0x7ffa1c054dd0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- 172.21.15.67:0/2371788191 shutdown_connections 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06c5d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca2360 0x55ad06c5c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2371788191 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06ca1e20 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- 172.21.15.67:0/2371788191 >> 172.21.15.67:0/2371788191 conn(0x55ad06c58d40 msgr2=0x55ad06c5b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- 172.21.15.67:0/2371788191 shutdown_connections 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- 172.21.15.67:0/2371788191 wait complete. 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 -- start start 2024-04-17T10:32:08.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.927+0000 7ffa3497e000 1 Processor -- start 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06ca91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06ca9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06c62d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad06a60540 con 0x55ad06ca2360 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad06aa1f30 con 0x55ad06ca1a40 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad06c435b0 con 0x55ad06c5cd90 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 10 client.0 _ll_register_callbacks cb 0x55ad06b5a180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa2a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06ca91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa2b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06c62d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa23e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06ca9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa2a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06ca91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:44100/0 (socket says 172.21.15.67:44100) 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa2a800640 1 -- 172.21.15.67:0/1418317381 learned_addr learned my addr 172.21.15.67:0/1418317381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa23e00640 1 -- 172.21.15.67:0/1418317381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 msgr2=0x55ad06ca91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa23e00640 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06ca91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa23e00640 1 -- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 msgr2=0x55ad06c62d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa23e00640 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06c62d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa23e00640 1 -- 172.21.15.67:0/1418317381 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad06a2e1a0 con 0x55ad06ca1a40 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa2b200640 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06c62d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa23e00640 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06ca9700 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ffa14000f70 tx=0x7ffa14001090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac9a1b66c0941d5d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa0fe00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa0fe00640 1 -- 172.21.15.67:0/1418317381 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14026060 con 0x55ad06ca1a40 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa1c05b0b0 con 0x55ad06ca1a40 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa0fe00640 1 -- 172.21.15.67:0/1418317381 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa14021660 con 0x55ad06ca1a40 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa0fe00640 1 -- 172.21.15.67:0/1418317381 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa14021860 con 0x55ad06ca1a40 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 10 client.7384 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:08.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55ad06b28840 con 0x55ad06ca1a40 2024-04-17T10:32:08.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa0fe00640 1 -- 172.21.15.67:0/1418317381 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c05b0b0 con 0x55ad06ca1a40 2024-04-17T10:32:08.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa0fe00640 1 -- 172.21.15.67:0/1418317381 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ffa14021a00 con 0x55ad06ca1a40 2024-04-17T10:32:08.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 10 client.7384 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 10 client.7384 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 20 client.7384 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06c63b20 0x55ad06c63f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 4 client.7384 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad06a41be0 con 0x55ad06c63b20 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa2a800640 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06c63b20 0x55ad06c63f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa2a800640 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06c63b20 0x55ad06c63f30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d6828596cc2283f server_cookie=2490252ed4b233be in_seq=0 out_seq=0 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.931+0000 7ffa0fe00640 10 client.7384 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa0fe00640 1 -- 172.21.15.67:0/1418317381 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ad06a41be0 con 0x55ad06c63b20 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa0fe00640 10 client.7384 handle_command_reply: tid=0 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 client.7384 shutdown 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 msgr2=0x55ad06ca9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06ca9700 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ffa14000f70 tx=0x7ffa14001090 comp rx=0 tx=0).stop 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 shutdown_connections 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06c63b20 0x55ad06c63f30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06c62d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06ca9700 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 --2- 172.21.15.67:0/1418317381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06ca91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa0fe00640 0 client.7384 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 >> 172.21.15.67:0/1418317381 conn(0x55ad06c58d40 msgr2=0x55ad06c5b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 shutdown_connections 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 1 -- 172.21.15.67:0/1418317381 wait complete. 2024-04-17T10:32:08.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.935+0000 7ffa3497e000 20 client.7384 trim_cache size 0 max 16384 2024-04-17T10:32:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.947+0000 7ffa3497e000 1 Processor -- start 2024-04-17T10:32:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.947+0000 7ffa3497e000 1 -- start start 2024-04-17T10:32:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06c5cd90 0x55ad06cfe290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06cfe7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06ca2360 0x55ad06cfed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c202d80 con 0x55ad06ca1a40 2024-04-17T10:32:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c201fb0 con 0x55ad06c5cd90 2024-04-17T10:32:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c0ca980 con 0x55ad06ca2360 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06cfe7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06cfe7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40116/0 (socket says 172.21.15.67:40116) 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 -- 172.21.15.67:0/2960316375 learned_addr learned my addr 172.21.15.67:0/2960316375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa2b200640 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06ca2360 0x55ad06cfed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06c5cd90 0x55ad06cfe290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 -- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06ca2360 msgr2=0x55ad06cfed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06ca2360 0x55ad06cfed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 -- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06c5cd90 msgr2=0x55ad06cfe290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06c5cd90 0x55ad06cfe290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 -- 172.21.15.67:0/2960316375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad06ca2190 con 0x55ad06ca1a40 2024-04-17T10:32:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06cfe7d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ffa14007b80 tx=0x7ffa1401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4ed5beba2e0b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa1aa00640 1 -- 172.21.15.67:0/2960316375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa14034070 con 0x55ad06ca1a40 2024-04-17T10:32:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa1aa00640 1 -- 172.21.15.67:0/2960316375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa140216a0 con 0x55ad06ca1a40 2024-04-17T10:32:08.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa1aa00640 1 -- 172.21.15.67:0/2960316375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa140072b0 con 0x55ad06ca1a40 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- 172.21.15.67:0/2960316375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 msgr2=0x55ad06cfe7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06cfe7d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7ffa14007b80 tx=0x7ffa1401e000 comp rx=0 tx=0).stop 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- 172.21.15.67:0/2960316375 shutdown_connections 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06ca2360 0x55ad06cfed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca1a40 0x55ad06cfe7d0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2960316375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06c5cd90 0x55ad06cfe290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- 172.21.15.67:0/2960316375 >> 172.21.15.67:0/2960316375 conn(0x55ad06c58d40 msgr2=0x55ad06ca0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- 172.21.15.67:0/2960316375 shutdown_connections 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- 172.21.15.67:0/2960316375 wait complete. 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- start start 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 Processor -- start 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06d024b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06d029f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06ca6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad06c58bc0 con 0x55ad06ca2360 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2c1531a0 con 0x55ad06ca1a40 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad06aa4e60 con 0x55ad06c5cd90 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa3497e000 10 client.0 _ll_register_callbacks cb 0x55ad06a68a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06d029f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa2b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06ca6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06d029f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:38478/0 (socket says 172.21.15.67:38478) 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa23e00640 1 -- 172.21.15.67:0/2237399980 learned_addr learned my addr 172.21.15.67:0/2237399980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.951+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06d024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa23e00640 1 -- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 msgr2=0x55ad06d024b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa23e00640 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06d024b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa23e00640 1 -- 172.21.15.67:0/2237399980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 msgr2=0x55ad06ca6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:08.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa23e00640 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06ca6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa23e00640 1 -- 172.21.15.67:0/2237399980 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad06affa10 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06d024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa23e00640 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06d029f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffa14017040 tx=0x7ffa14011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7665eae460aa953e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 1 -- 172.21.15.67:0/2237399980 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1403c020 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 1 -- 172.21.15.67:0/2237399980 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa140069c0 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad06ab33d0 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 1 -- 172.21.15.67:0/2237399980 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa14043050 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 10 client.7390 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55ad06a3ab00 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 1 -- 172.21.15.67:0/2237399980 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ad06ab33d0 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 1 -- 172.21.15.67:0/2237399980 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ffa14046020 con 0x55ad06ca1a40 2024-04-17T10:32:08.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 10 client.7390 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 10 client.7390 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 20 client.7390 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06ca7890 0x55ad06ca7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 4 client.7390 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "scrub abort"}) v1 -- 0x55ad06aad720 con 0x55ad06ca7890 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06ca7890 0x55ad06ca7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa2a800640 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06ca7890 0x55ad06ca7ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dcccf014b23f7ee server_cookie=d7c08295607bad83 in_seq=0 out_seq=0 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 10 client.7390 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 1 -- 172.21.15.67:0/2237399980 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x55ad06aad720 con 0x55ad06ca7890 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 10 client.7390 handle_command_reply: tid=0 2024-04-17T10:32:08.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 client.7390 shutdown 2024-04-17T10:32:08.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 msgr2=0x55ad06d029f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06d029f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffa14017040 tx=0x7ffa14011000 comp rx=0 tx=0).stop 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 shutdown_connections 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55ad06ca7890 0x55ad06ca7ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad06ca2360 0x55ad06ca6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad06ca1a40 0x55ad06d029f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 --2- 172.21.15.67:0/2237399980 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad06c5cd90 0x55ad06d024b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 >> 172.21.15.67:0/2237399980 conn(0x55ad06c58d40 msgr2=0x55ad06ca09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa0fe00640 0 client.7390 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 shutdown_connections 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 1 -- 172.21.15.67:0/2237399980 wait complete. 2024-04-17T10:32:08.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa3497e000 20 client.7390 trim_cache size 0 max 16384 2024-04-17T10:32:08.954 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:32:08.954 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-04-17T10:32:08.954 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:32:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa08066f60 msgr2=0x7ffa08069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa08066f60 0x7ffa08069420 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7ffa14007fd0 tx=0x7ffa14007940 comp rx=0 tx=0).stop 2024-04-17T10:32:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa2c06eaa0 msgr2=0x7ffa2c0b3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa2c06eaa0 0x7ffa2c0b3130 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c480 tx=0x7ffa1c00c940 comp rx=0 tx=0).stop 2024-04-17T10:32:08.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 shutdown_connections 2024-04-17T10:32:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa08066f60 0x7ffa08069420 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa2c15a590 0x7ffa2c156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa2c0cf0c0 0x7ffa2c148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 --2- 172.21.15.67:0/228783691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa2c06eaa0 0x7ffa2c0b3130 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 >> 172.21.15.67:0/228783691 conn(0x7ffa2c06ac90 msgr2=0x7ffa2c158ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 shutdown_connections 2024-04-17T10:32:08.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:08.955+0000 7ffa30a00640 1 -- 172.21.15.67:0/228783691 wait complete. 2024-04-17T10:32:08.966 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-17T10:32:08.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-17T10:32:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7466800640 1 Processor -- start 2024-04-17T10:32:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7466800640 1 -- start start 2024-04-17T10:32:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7466800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7466800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74600728c0 0x7f74600b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7466800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f746015f360 con 0x7f74600b5130 2024-04-17T10:32:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7466800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f746006bca0 con 0x7f74600728c0 2024-04-17T10:32:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7466800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74600b3160 con 0x7f74601583f0 2024-04-17T10:32:09.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f745f400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f745f400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460072380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:44128/0 (socket says 172.21.15.67:44128) 2024-04-17T10:32:09.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7457e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f74600728c0 0x7f74600b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f745f400640 1 -- 172.21.15.67:0/1765053166 learned_addr learned my addr 172.21.15.67:0/1765053166 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:09.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7457e00640 1 -- 172.21.15.67:0/1765053166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 msgr2=0x7f7460072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7457e00640 1 --2- 172.21.15.67:0/1765053166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7457e00640 1 -- 172.21.15.67:0/1765053166 >> v1:172.21.15.86:6789/0 conn(0x7f74600b5130 legacy=0x7f7460161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.119+0000 7f7457e00640 1 -- 172.21.15.67:0/1765053166 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74600b3440 con 0x7f74600728c0 2024-04-17T10:32:09.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7457e00640 1 --2- 172.21.15.67:0/1765053166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 0x7f74600b4bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f744801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4872122b50469256 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f745ea00640 1 -- 172.21.15.67:0/1765053166 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480b0070 con 0x7f74600728c0 2024-04-17T10:32:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f745ea00640 1 -- 172.21.15.67:0/1765053166 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74480ab600 con 0x7f74600728c0 2024-04-17T10:32:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f745ea00640 1 -- 172.21.15.67:0/1765053166 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480ab9d0 con 0x7f74600728c0 2024-04-17T10:32:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 -- 172.21.15.67:0/1765053166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 msgr2=0x7f74600b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 --2- 172.21.15.67:0/1765053166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 0x7f74600b4bf0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f74480097c0 tx=0x7f744801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 -- 172.21.15.67:0/1765053166 shutdown_connections 2024-04-17T10:32:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 --2- 172.21.15.67:0/1765053166 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 0x7f74600b4bf0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 --2- 172.21.15.67:0/1765053166 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 -- 172.21.15.67:0/1765053166 >> 172.21.15.67:0/1765053166 conn(0x7f74600b2570 msgr2=0x7f7460071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 -- 172.21.15.67:0/1765053166 shutdown_connections 2024-04-17T10:32:09.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.123+0000 7f7466800640 1 -- 172.21.15.67:0/1765053166 wait complete. 2024-04-17T10:32:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 Processor -- start 2024-04-17T10:32:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 -- start start 2024-04-17T10:32:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 0x7f7460156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600b5130 0x7f7460150250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460071c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74600b4200 con 0x7f74600b5130 2024-04-17T10:32:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74601587d0 con 0x7f74600728c0 2024-04-17T10:32:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74600b5540 con 0x7f74601583f0 2024-04-17T10:32:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600b5130 0x7f7460150250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600b5130 0x7f7460150250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40134/0 (socket says 172.21.15.67:40134) 2024-04-17T10:32:09.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 -- 172.21.15.67:0/3978672239 learned_addr learned my addr 172.21.15.67:0/3978672239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f745f400640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 0x7f7460156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f745fe00640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 -- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 msgr2=0x7f7460071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460071c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 -- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 msgr2=0x7f7460156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 0x7f7460156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 -- 172.21.15.67:0/3978672239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74600ad9e0 con 0x7f74600b5130 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457e00640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600b5130 0x7f7460150250 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f74480ae040 tx=0x7f7448019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457400640 1 -- 172.21.15.67:0/3978672239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480ba070 con 0x7f74600b5130 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457400640 1 -- 172.21.15.67:0/3978672239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74480b43f0 con 0x7f74600b5130 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746015e8f0 con 0x7f74600b5130 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7457400640 1 -- 172.21.15.67:0/3978672239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480b0040 con 0x7f74600b5130 2024-04-17T10:32:09.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.127+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746015c1b0 con 0x7f74600b5130 2024-04-17T10:32:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7457400640 1 -- 172.21.15.67:0/3978672239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f74480bd070 con 0x7f74600b5130 2024-04-17T10:32:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7457400640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7457400640 1 -- 172.21.15.67:0/3978672239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f744811e0a0 con 0x7f74600b5130 2024-04-17T10:32:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 Processor -- start 2024-04-17T10:32:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f745f400640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 -- start start 2024-04-17T10:32:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f745f400640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7460150ef0 tx=0x7f7450009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790ee7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790e9db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790e9e0a0 0x559790e9e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559790ceaff0 con 0x559790e9e0a0 2024-04-17T10:32:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x559790c73150 con 0x559790ee79d0 2024-04-17T10:32:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x559790ce7090 con 0x559790ee82f0 2024-04-17T10:32:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790e9e0a0 0x559790e9e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790e9e0a0 0x559790e9e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40138/0 (socket says 172.21.15.67:40138) 2024-04-17T10:32:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.131+0000 7f745fe00640 1 -- 172.21.15.67:0/2826268 learned_addr learned my addr 172.21.15.67:0/2826268 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745f400640 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790ee7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7457e00640 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790e9db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 -- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 msgr2=0x559790e9db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790e9db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 -- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 msgr2=0x559790ee7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790ee7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 -- 172.21.15.67:0/2826268 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559790e810d0 con 0x559790e9e0a0 2024-04-17T10:32:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790e9e0a0 0x559790e9e480 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7458001ef0 tx=0x7f74580016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11f82d7f3136f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f744fe00640 1 -- 172.21.15.67:0/2826268 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745800fe50 con 0x559790e9e0a0 2024-04-17T10:32:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f744fe00640 1 -- 172.21.15.67:0/2826268 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458018070 con 0x559790e9e0a0 2024-04-17T10:32:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f744fe00640 1 -- 172.21.15.67:0/2826268 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458003ba0 con 0x559790e9e0a0 2024-04-17T10:32:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- 172.21.15.67:0/2826268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790e9e0a0 msgr2=0x559790e9e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790e9e0a0 0x559790e9e480 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f7458001ef0 tx=0x7f74580016b0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- 172.21.15.67:0/2826268 shutdown_connections 2024-04-17T10:32:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790e9e0a0 0x559790e9e480 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790e9db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 --2- 172.21.15.67:0/2826268 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790ee7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- 172.21.15.67:0/2826268 >> 172.21.15.67:0/2826268 conn(0x559790e9a020 msgr2=0x559790e9c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- 172.21.15.67:0/2826268 shutdown_connections 2024-04-17T10:32:09.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- 172.21.15.67:0/2826268 wait complete. 2024-04-17T10:32:09.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- start start 2024-04-17T10:32:09.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 Processor -- start 2024-04-17T10:32:09.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ecc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790eccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790ecd270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559790caf250 con 0x559790ee82f0 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x559790ca5690 con 0x559790ee79d0 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x559790c73a40 con 0x559790e9e0a0 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7469d47000 10 client.0 _ll_register_callbacks cb 0x559790e88e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790ecd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790ecd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40146/0 (socket says 172.21.15.67:40146) 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 -- 172.21.15.67:0/686937769 learned_addr learned my addr 172.21.15.67:0/686937769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f7457e00640 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790eccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745f400640 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ecc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 -- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 msgr2=0x559790ecc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ecc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 -- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 msgr2=0x559790eccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790eccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.135+0000 7f745fe00640 1 -- 172.21.15.67:0/686937769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559790d981c0 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f745fe00640 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790ecd270 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f745800fa30 tx=0x7f7458001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dca90d4c4ad49f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 1 -- 172.21.15.67:0/686937769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580054e0 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 1 -- 172.21.15.67:0/686937769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7458005680 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7458018070 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 1 -- 172.21.15.67:0/686937769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f745802e050 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 10 client.7902 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x559790d94170 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 1 -- 172.21.15.67:0/686937769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458018070 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 1 -- 172.21.15.67:0/686937769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f7458031020 con 0x559790ee82f0 2024-04-17T10:32:09.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 10 client.7902 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 10 client.7902 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 20 client.7902 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790ef7c80 0x559790efa070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 4 client.7902 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559790c86c60 con 0x559790ef7c80 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f745f400640 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790ef7c80 0x559790efa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f745f400640 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790ef7c80 0x559790efa070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a17da7ebfb43010f server_cookie=e965fd77a8332c96 in_seq=0 out_seq=0 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 10 client.7902 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 1 -- 172.21.15.67:0/686937769 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559790c86c60 con 0x559790ef7c80 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 10 client.7902 handle_command_reply: tid=0 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 client.7902 shutdown 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 msgr2=0x559790ecd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790ecd270 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f745800fa30 tx=0x7f7458001260 comp rx=0 tx=0).stop 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 shutdown_connections 2024-04-17T10:32:09.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790ef7c80 0x559790efa070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790ecd270 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790eccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 --2- 172.21.15.67:0/686937769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ecc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f744cc00640 0 client.7902 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 >> 172.21.15.67:0/686937769 conn(0x559790e9a020 msgr2=0x559790e9cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 shutdown_connections 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 1 -- 172.21.15.67:0/686937769 wait complete. 2024-04-17T10:32:09.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.139+0000 7f7469d47000 20 client.7902 trim_cache size 0 max 16384 2024-04-17T10:32:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 Processor -- start 2024-04-17T10:32:09.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 -- start start 2024-04-17T10:32:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ee6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790e9d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790e9d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7460203270 con 0x559790ee82f0 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7460202450 con 0x559790ee79d0 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7460159260 con 0x559790e9e0a0 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790e9d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790e9d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40148/0 (socket says 172.21.15.67:40148) 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 -- 172.21.15.67:0/3866270880 learned_addr learned my addr 172.21.15.67:0/3866270880 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f7457e00640 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790e9d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745f400640 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ee6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 -- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 msgr2=0x559790ee6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ee6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 -- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 msgr2=0x559790e9d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790e9d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 -- 172.21.15.67:0/3866270880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559790d31de0 con 0x559790ee82f0 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f745fe00640 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790e9d960 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f74580016f0 tx=0x7f745801f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dc4552a928ff43 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.155+0000 7f744f400640 1 -- 172.21.15.67:0/3866270880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458037070 con 0x559790ee82f0 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f744f400640 1 -- 172.21.15.67:0/3866270880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458018070 con 0x559790ee82f0 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f744f400640 1 -- 172.21.15.67:0/3866270880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458006da0 con 0x559790ee82f0 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- 172.21.15.67:0/3866270880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 msgr2=0x559790e9d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790e9d960 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f74580016f0 tx=0x7f745801f070 comp rx=0 tx=0).stop 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- 172.21.15.67:0/3866270880 shutdown_connections 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee82f0 0x559790e9d960 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790ee79d0 0x559790e9d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 --2- 172.21.15.67:0/3866270880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790e9e0a0 0x559790ee6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- 172.21.15.67:0/3866270880 >> 172.21.15.67:0/3866270880 conn(0x559790e9a020 msgr2=0x559790ee60d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- 172.21.15.67:0/3866270880 shutdown_connections 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- 172.21.15.67:0/3866270880 wait complete. 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- start start 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 Processor -- start 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790e9e0a0 0x559790f48520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee79d0 0x559790f48a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790efa380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x559790c85e00 con 0x559790ee79d0 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7460159860 con 0x559790e9e0a0 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x559790cea300 con 0x559790ee82f0 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 10 client.0 _ll_register_callbacks cb 0x559790e69040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:09.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee79d0 0x559790f48a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f745f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790e9e0a0 0x559790f48520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f745f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790e9e0a0 0x559790f48520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:38538/0 (socket says 172.21.15.67:38538) 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee79d0 0x559790f48a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40158/0 (socket says 172.21.15.67:40158) 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f745f400640 1 -- 172.21.15.67:0/3573447372 learned_addr learned my addr 172.21.15.67:0/3573447372 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f745fe00640 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790efa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 -- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 msgr2=0x559790efa380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790efa380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 -- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790e9e0a0 msgr2=0x559790f48520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790e9e0a0 0x559790f48520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 -- 172.21.15.67:0/3573447372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559790c8c8e0 con 0x559790ee79d0 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f745f400640 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790e9e0a0 0x559790f48520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7457e00640 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee79d0 0x559790f48a60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f744811c400 tx=0x7f74480ee400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbdb7f8b9fe14af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:09.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f744cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f744cc00640 1 -- 172.21.15.67:0/3573447372 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480f6c90 con 0x559790ee79d0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f744cc00640 1 -- 172.21.15.67:0/3573447372 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f74480efb40 con 0x559790ee79d0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7458018070 con 0x559790ee79d0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f744cc00640 1 -- 172.21.15.67:0/3573447372 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74480ee8c0 con 0x559790ee79d0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 10 client.7908 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x559790d448f0 con 0x559790ee79d0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.159+0000 7f744cc00640 1 -- 172.21.15.67:0/3573447372 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458018070 con 0x559790ee79d0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f744cc00640 1 -- 172.21.15.67:0/3573447372 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f74480b3070 con 0x559790ee79d0 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 10 client.7908 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 10 client.7908 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 20 client.7908 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790eface0 0x559790efb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:09.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 4 client.7908 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559790cf8530 con 0x559790eface0 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f745f400640 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790eface0 0x559790efb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f745f400640 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790eface0 0x559790efb0f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ace5745a64e0b033 server_cookie=e27a440a082c8356 in_seq=0 out_seq=0 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f744cc00640 10 client.7908 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f744cc00640 1 -- 172.21.15.67:0/3573447372 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x559790cf8530 con 0x559790eface0 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f744cc00640 10 client.7908 handle_command_reply: tid=0 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 client.7908 shutdown 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee79d0 msgr2=0x559790f48a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee79d0 0x559790f48a60 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f744811c400 tx=0x7f74480ee400 comp rx=0 tx=0).stop 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 shutdown_connections 2024-04-17T10:32:09.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x559790eface0 0x559790efb0f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x559790ee82f0 0x559790efa380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f744cc00640 0 client.7908 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x559790ee79d0 0x559790f48a60 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 --2- 172.21.15.67:0/3573447372 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x559790e9e0a0 0x559790f48520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 >> 172.21.15.67:0/3573447372 conn(0x559790e9a020 msgr2=0x559790ee6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 shutdown_connections 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 1 -- 172.21.15.67:0/3573447372 wait complete. 2024-04-17T10:32:09.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7469d47000 20 client.7908 trim_cache size 0 max 16384 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: "2d8f24ea-0773-494c-b32e-a02e95716720": { 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_control_test_path", 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "2d8f24ea-0773-494c-b32e-a02e95716720", 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:09.163 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:32:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 msgr2=0x7f743c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7460150ef0 tx=0x7f7450009240 comp rx=0 tx=0).stop 2024-04-17T10:32:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600b5130 msgr2=0x7f7460150250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600b5130 0x7f7460150250 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f74480ae040 tx=0x7f7448019240 comp rx=0 tx=0).stop 2024-04-17T10:32:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 shutdown_connections 2024-04-17T10:32:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f743c066f60 0x7f743c069420 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74601583f0 0x7f7460071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74600b5130 0x7f7460150250 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 --2- 172.21.15.67:0/3978672239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74600728c0 0x7f7460156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 >> 172.21.15.67:0/3978672239 conn(0x7f74600b2570 msgr2=0x7f74600b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 shutdown_connections 2024-04-17T10:32:09.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:09.163+0000 7f7466800640 1 -- 172.21.15.67:0/3978672239 wait complete. 2024-04-17T10:32:14.173 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-17T10:32:14.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.6:0 scrub status 2024-04-17T10:32:14.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.335+0000 7f4048c00640 1 Processor -- start 2024-04-17T10:32:14.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.335+0000 7f4048c00640 1 -- start start 2024-04-17T10:32:14.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4048c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40440aef60 0x7f40440cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4048c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f404406e980 0x7f40440b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4048c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40440b4990 con 0x7f40440cce60 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4048c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40440af340 con 0x7f40440aef60 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4048c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404415ae90 con 0x7f404406e980 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4041e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f40440aef60 0x7f40440cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f403be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f40440cce60 0x7f404406e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46856/0 (socket says 172.21.15.67:46856) 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f403be00640 1 -- 172.21.15.67:0/3209704661 learned_addr learned my addr 172.21.15.67:0/3209704661 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4042800640 1 --2- 172.21.15.67:0/3209704661 >> v2:172.21.15.86:3300/0 conn(0x7f404406e980 0x7f40440b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4041e00640 1 -- 172.21.15.67:0/3209704661 >> v2:172.21.15.86:3300/0 conn(0x7f404406e980 msgr2=0x7f40440b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4041e00640 1 --2- 172.21.15.67:0/3209704661 >> v2:172.21.15.86:3300/0 conn(0x7f404406e980 0x7f40440b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4041e00640 1 -- 172.21.15.67:0/3209704661 >> v1:172.21.15.67:6789/0 conn(0x7f40440cce60 legacy=0x7f404406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4041e00640 1 -- 172.21.15.67:0/3209704661 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404406ee80 con 0x7f40440aef60 2024-04-17T10:32:14.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.339+0000 7f4041400640 1 -- 172.21.15.67:0/3209704661 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315776017 0 0) 0x7f402c001120 con 0x7f40440cce60 2024-04-17T10:32:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4041e00640 1 --2- 172.21.15.67:0/3209704661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40440cc920 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f40440b2e30 tx=0x7f403401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250ef3875cf12355 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4041400640 1 -- 172.21.15.67:0/3209704661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40340b0070 con 0x7f40440aef60 2024-04-17T10:32:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4041400640 1 -- 172.21.15.67:0/3209704661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40340ab600 con 0x7f40440aef60 2024-04-17T10:32:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4041400640 1 -- 172.21.15.67:0/3209704661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40340ab9d0 con 0x7f40440aef60 2024-04-17T10:32:14.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- 172.21.15.67:0/3209704661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 msgr2=0x7f40440cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 --2- 172.21.15.67:0/3209704661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40440cc920 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f40440b2e30 tx=0x7f403401c3f0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- 172.21.15.67:0/3209704661 shutdown_connections 2024-04-17T10:32:14.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 --2- 172.21.15.67:0/3209704661 >> v2:172.21.15.86:3300/0 conn(0x7f404406e980 0x7f40440b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 --2- 172.21.15.67:0/3209704661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40440cc920 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- 172.21.15.67:0/3209704661 >> 172.21.15.67:0/3209704661 conn(0x7f40440b1740 msgr2=0x7f40440b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- 172.21.15.67:0/3209704661 shutdown_connections 2024-04-17T10:32:14.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- 172.21.15.67:0/3209704661 wait complete. 2024-04-17T10:32:14.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 Processor -- start 2024-04-17T10:32:14.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- start start 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f404406e980 0x7f404415d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40441417e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40440cce60 0x7f40441f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40440b3ff0 con 0x7f40440aef60 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40440b3e70 con 0x7f404406e980 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f4048c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40441673c0 con 0x7f40440cce60 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f403be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40441417e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f403be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40441417e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35746/0 (socket says 172.21.15.67:35746) 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.343+0000 7f403be00640 1 -- 172.21.15.67:0/2082525881 learned_addr learned my addr 172.21.15.67:0/2082525881 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:14.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f4041e00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f404406e980 0x7f404415d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f4042800640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40440cce60 0x7f40441f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403be00640 1 -- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40440cce60 msgr2=0x7f40441f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403be00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40440cce60 0x7f40441f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403be00640 1 -- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f404406e980 msgr2=0x7f404415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403be00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f404406e980 0x7f404415d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403be00640 1 -- 172.21.15.67:0/2082525881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404414d6e0 con 0x7f40440aef60 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403be00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40441417e0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f402c00c780 tx=0x7f402c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403a000640 1 -- 172.21.15.67:0/2082525881 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c018070 con 0x7f40440aef60 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403a000640 1 -- 172.21.15.67:0/2082525881 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c0072f0 con 0x7f40440aef60 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f404414d9c0 con 0x7f40440aef60 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403a000640 1 -- 172.21.15.67:0/2082525881 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c0047f0 con 0x7f40440aef60 2024-04-17T10:32:14.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40441f6be0 con 0x7f40440aef60 2024-04-17T10:32:14.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403a000640 1 -- 172.21.15.67:0/2082525881 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f402c01b070 con 0x7f40440aef60 2024-04-17T10:32:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.347+0000 7f403a000640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4020066f60 0x7f4020069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f403a000640 1 -- 172.21.15.67:0/2082525881 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f402c084570 con 0x7f40440aef60 2024-04-17T10:32:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 Processor -- start 2024-04-17T10:32:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- start start 2024-04-17T10:32:14.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4020066f60 0x7f4020069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef8a6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef85d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef6a9ff0 con 0x55c2ef8a69d0 2024-04-17T10:32:14.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef632150 con 0x55c2ef8a72f0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef6a6090 con 0x55c2ef85d0a0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4020066f60 0x7f4020069420 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f40340ad000 tx=0x7f40340a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef8a6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef8a6db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35756/0 (socket says 172.21.15.67:35756) 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 -- 172.21.15.67:0/1892964251 learned_addr learned my addr 172.21.15.67:0/1892964251 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4042800640 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef85d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f403be00640 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 -- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 msgr2=0x55c2ef85d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef85d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 -- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 msgr2=0x55c2ef85cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 -- 172.21.15.67:0/1892964251 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ef8400d0 con 0x55c2ef8a69d0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4041e00640 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef8a6db0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4034019000 tx=0x7f4034004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e42841ea90e45a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4032a00640 1 -- 172.21.15.67:0/1892964251 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40340b0040 con 0x55c2ef8a69d0 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4032a00640 1 -- 172.21.15.67:0/1892964251 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40340b1040 con 0x55c2ef8a69d0 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f4032a00640 1 -- 172.21.15.67:0/1892964251 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4034004220 con 0x55c2ef8a69d0 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- 172.21.15.67:0/1892964251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 msgr2=0x55c2ef8a6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef8a6db0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f4034019000 tx=0x7f4034004740 comp rx=0 tx=0).stop 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- 172.21.15.67:0/1892964251 shutdown_connections 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef85d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 --2- 172.21.15.67:0/1892964251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef8a6db0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- 172.21.15.67:0/1892964251 >> 172.21.15.67:0/1892964251 conn(0x55c2ef859020 msgr2=0x55c2ef85b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- 172.21.15.67:0/1892964251 shutdown_connections 2024-04-17T10:32:14.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.351+0000 7f404c104000 1 -- 172.21.15.67:0/1892964251 wait complete. 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 -- start start 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 Processor -- start 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef85d0a0 0x55c2ef88b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef88bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a72f0 0x55c2ef88c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef66e250 con 0x55c2ef8a72f0 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef664690 con 0x55c2ef85d0a0 2024-04-17T10:32:14.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef632a40 con 0x55c2ef8a69d0 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 10 client.0 _ll_register_callbacks cb 0x55c2ef847e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a72f0 0x55c2ef88c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a72f0 0x55c2ef88c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35762/0 (socket says 172.21.15.67:35762) 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 -- 172.21.15.67:0/1055053343 learned_addr learned my addr 172.21.15.67:0/1055053343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f403be00640 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef88bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4041e00640 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef85d0a0 0x55c2ef88b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 -- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 msgr2=0x55c2ef88bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef88bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 -- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef85d0a0 msgr2=0x55c2ef88b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef85d0a0 0x55c2ef88b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 -- 172.21.15.67:0/1055053343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2ef7571c0 con 0x55c2ef8a72f0 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4042800640 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a72f0 0x55c2ef88c270 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f403c004660 tx=0x7f403c007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3ed829c8c8ba36 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4027400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4027400640 1 -- 172.21.15.67:0/1055053343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f403c01b020 con 0x55c2ef8a72f0 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4027400640 1 -- 172.21.15.67:0/1055053343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f403c004750 con 0x55c2ef8a72f0 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40340b1040 con 0x55c2ef8a72f0 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4027400640 1 -- 172.21.15.67:0/1055053343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f403c025050 con 0x55c2ef8a72f0 2024-04-17T10:32:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 10 client.7923 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55c2ef753170 con 0x55c2ef8a72f0 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4027400640 1 -- 172.21.15.67:0/1055053343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40340b1040 con 0x55c2ef8a72f0 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4027400640 1 -- 172.21.15.67:0/1055053343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f403c028020 con 0x55c2ef8a72f0 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 10 client.7923 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 10 client.7923 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 20 client.7923 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b6c80 0x55c2ef8b9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 4 client.7923 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:32:14.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c2ef645c60 con 0x55c2ef8b6c80 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.355+0000 7f4041e00640 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b6c80 0x55c2ef8b9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f4041e00640 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b6c80 0x55c2ef8b9070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8454182fc8b19f82 server_cookie=81934acdd3dd185b in_seq=0 out_seq=0 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f4027400640 10 client.7923 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f4027400640 1 -- 172.21.15.67:0/1055053343 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c2ef645c60 con 0x55c2ef8b6c80 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f4027400640 10 client.7923 handle_command_reply: tid=0 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 client.7923 shutdown 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a72f0 msgr2=0x55c2ef88c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a72f0 0x55c2ef88c270 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f403c004660 tx=0x7f403c007700 comp rx=0 tx=0).stop 2024-04-17T10:32:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 shutdown_connections 2024-04-17T10:32:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b6c80 0x55c2ef8b9070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a72f0 0x55c2ef88c270 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef88bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 --2- 172.21.15.67:0/1055053343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef85d0a0 0x55c2ef88b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f4027400640 0 client.7923 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 >> 172.21.15.67:0/1055053343 conn(0x55c2ef859020 msgr2=0x55c2ef85bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 shutdown_connections 2024-04-17T10:32:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 1 -- 172.21.15.67:0/1055053343 wait complete. 2024-04-17T10:32:14.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.359+0000 7f404c104000 20 client.7923 trim_cache size 0 max 16384 2024-04-17T10:32:14.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 Processor -- start 2024-04-17T10:32:14.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- start start 2024-04-17T10:32:14.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef8a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef85c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40442033b0 con 0x55c2ef8a69d0 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4044202570 con 0x55c2ef8a72f0 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40441fac10 con 0x55c2ef85d0a0 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef85c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef85c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35774/0 (socket says 172.21.15.67:35774) 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 -- 172.21.15.67:0/1319432612 learned_addr learned my addr 172.21.15.67:0/1319432612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f4042800640 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f4041e00640 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef8a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 -- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 msgr2=0x55c2ef8a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef8a5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 -- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 msgr2=0x55c2ef85c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 -- 172.21.15.67:0/1319432612 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c2ef6f0de0 con 0x55c2ef8a69d0 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f4042800640 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f4041e00640 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef8a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f403be00640 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef85c420 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f402c0511a0 tx=0x7f402c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72444047f4dc671 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f4032000640 1 -- 172.21.15.67:0/1319432612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c053090 con 0x55c2ef8a69d0 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f4032000640 1 -- 172.21.15.67:0/1319432612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c051400 con 0x55c2ef8a69d0 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f4032000640 1 -- 172.21.15.67:0/1319432612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f402c0842d0 con 0x55c2ef8a69d0 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- 172.21.15.67:0/1319432612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 msgr2=0x55c2ef85c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef85c420 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f402c0511a0 tx=0x7f402c009040 comp rx=0 tx=0).stop 2024-04-17T10:32:14.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- 172.21.15.67:0/1319432612 shutdown_connections 2024-04-17T10:32:14.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef85c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef8a69d0 0x55c2ef85c420 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 --2- 172.21.15.67:0/1319432612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef85d0a0 0x55c2ef8a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- 172.21.15.67:0/1319432612 >> 172.21.15.67:0/1319432612 conn(0x55c2ef859020 msgr2=0x55c2ef8a50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- 172.21.15.67:0/1319432612 shutdown_connections 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- 172.21.15.67:0/1319432612 wait complete. 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.375+0000 7f404c104000 1 -- start start 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 Processor -- start 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef85d0a0 0x55c2ef907520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef907a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef8b9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef6a9300 con 0x55c2ef85d0a0 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef644e00 con 0x55c2ef8a72f0 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c2ef6ae620 con 0x55c2ef8a69d0 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 10 client.0 _ll_register_callbacks cb 0x55c2ef828040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef85d0a0 0x55c2ef907520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef85d0a0 0x55c2ef907520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35784/0 (socket says 172.21.15.67:35784) 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 -- 172.21.15.67:0/2551256170 learned_addr learned my addr 172.21.15.67:0/2551256170 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f403be00640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef907a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4042800640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef8b9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 -- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 msgr2=0x55c2ef907a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef907a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 -- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 msgr2=0x55c2ef8b9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef8b9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 -- 172.21.15.67:0/2551256170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c2ef64b8e0 con 0x55c2ef85d0a0 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f403be00640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef907a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4042800640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef8b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4041e00640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef85d0a0 0x55c2ef907520 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f403401a020 tx=0x7f403401ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d142794b8537 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:14.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 1 -- 172.21.15.67:0/2551256170 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40340b0040 con 0x55c2ef85d0a0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f402c051400 con 0x55c2ef85d0a0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 1 -- 172.21.15.67:0/2551256170 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4034004280 con 0x55c2ef85d0a0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 1 -- 172.21.15.67:0/2551256170 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40340dd050 con 0x55c2ef85d0a0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 10 client.7929 fetch_fsmap learned FSMap version 64 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x55c2ef7038f0 con 0x55c2ef85d0a0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 1 -- 172.21.15.67:0/2551256170 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f402c051400 con 0x55c2ef85d0a0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 1 -- 172.21.15.67:0/2551256170 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f40340e0020 con 0x55c2ef85d0a0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 10 client.7929 fetch_fsmap finished waiting for FSMap version 64 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 10 client.7929 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 20 client.7929 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b9d70 0x55c2ef8ba180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 4 client.7929 mds_command: new command op to 7698 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 --> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c2ef6b7530 con 0x55c2ef8b9d70 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f403be00640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b9d70 0x55c2ef8ba180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f403be00640 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b9d70 0x55c2ef8ba180 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39583b1639be5620 server_cookie=96e3ed4a4053240 in_seq=0 out_seq=0 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 10 client.7929 ms_handle_connect on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 1 -- 172.21.15.67:0/2551256170 <== mds.0 v2:172.21.15.67:6833/3523329517 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c2ef6b7530 con 0x55c2ef8b9d70 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 10 client.7929 handle_command_reply: tid=0 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 client.7929 shutdown 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef85d0a0 msgr2=0x55c2ef907520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef85d0a0 0x55c2ef907520 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f403401a020 tx=0x7f403401ac30 comp rx=0 tx=0).stop 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 shutdown_connections 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:6833/3523329517,v1:172.21.15.67:6835/3523329517] conn(0x55c2ef8b9d70 0x55c2ef8ba180 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c2ef8a72f0 0x55c2ef8b9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c2ef8a69d0 0x55c2ef907a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 --2- 172.21.15.67:0/2551256170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c2ef85d0a0 0x55c2ef907520 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f4027400640 0 client.7929 ms_handle_reset on v2:172.21.15.67:6833/3523329517 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.379+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 >> 172.21.15.67:0/2551256170 conn(0x55c2ef859020 msgr2=0x55c2ef8a5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 shutdown_connections 2024-04-17T10:32:14.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f404c104000 1 -- 172.21.15.67:0/2551256170 wait complete. 2024-04-17T10:32:14.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f404c104000 20 client.7929 trim_cache size 0 max 16384 2024-04-17T10:32:14.377 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:32:14.377 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-04-17T10:32:14.377 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:32:14.377 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:32:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4020066f60 msgr2=0x7f4020069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4020066f60 0x7f4020069420 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f40340ad000 tx=0x7f40340a6000 comp rx=0 tx=0).stop 2024-04-17T10:32:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 msgr2=0x7f40441417e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40441417e0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f402c00c780 tx=0x7f402c00cc40 comp rx=0 tx=0).stop 2024-04-17T10:32:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 shutdown_connections 2024-04-17T10:32:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4020066f60 0x7f4020069420 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40440cce60 0x7f40441f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40440aef60 0x7f40441417e0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 --2- 172.21.15.67:0/2082525881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f404406e980 0x7f404415d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 >> 172.21.15.67:0/2082525881 conn(0x7f40440b1740 msgr2=0x7f40440b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 shutdown_connections 2024-04-17T10:32:14.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.383+0000 7f4048c00640 1 -- 172.21.15.67:0/2082525881 wait complete. 2024-04-17T10:32:14.392 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-17T10:32:14.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f3064c00640 1 Processor -- start 2024-04-17T10:32:14.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f3064c00640 1 -- start start 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f3064c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3060071310 0x7f3060071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f3064c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f30600cc300 0x7f30600cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f3064c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306006e2f0 con 0x7f3060145040 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f3064c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30600a86c0 con 0x7f3060071310 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f3064c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3060071cb0 con 0x7f30600cc300 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3060071310 0x7f3060071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3060145040 0x7f306006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46868/0 (socket says 172.21.15.67:46868) 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3060071310 0x7f3060071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35796/0 (socket says 172.21.15.67:35796) 2024-04-17T10:32:14.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305de00640 1 -- 172.21.15.67:0/4176606089 learned_addr learned my addr 172.21.15.67:0/4176606089 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:14.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305e800640 1 --2- 172.21.15.67:0/4176606089 >> v2:172.21.15.86:3300/0 conn(0x7f30600cc300 0x7f30600cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305d400640 1 -- 172.21.15.67:0/4176606089 >> v2:172.21.15.86:3300/0 conn(0x7f30600cc300 msgr2=0x7f30600cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305d400640 1 --2- 172.21.15.67:0/4176606089 >> v2:172.21.15.86:3300/0 conn(0x7f30600cc300 0x7f30600cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305d400640 1 -- 172.21.15.67:0/4176606089 >> v1:172.21.15.67:6789/0 conn(0x7f3060145040 legacy=0x7f306006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305d400640 1 -- 172.21.15.67:0/4176606089 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306006d6e0 con 0x7f3060071310 2024-04-17T10:32:14.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.543+0000 7f305e800640 1 --2- 172.21.15.67:0/4176606089 >> v2:172.21.15.86:3300/0 conn(0x7f30600cc300 0x7f30600cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f305d400640 1 --2- 172.21.15.67:0/4176606089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060071310 0x7f3060071770 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f30600af390 tx=0x7f304c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df54273c9e84a91 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f305ca00640 1 -- 172.21.15.67:0/4176606089 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c0b5070 con 0x7f3060071310 2024-04-17T10:32:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f305ca00640 1 -- 172.21.15.67:0/4176606089 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c0a8970 con 0x7f3060071310 2024-04-17T10:32:14.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f305ca00640 1 -- 172.21.15.67:0/4176606089 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c0a8d40 con 0x7f3060071310 2024-04-17T10:32:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 -- 172.21.15.67:0/4176606089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060071310 msgr2=0x7f3060071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 --2- 172.21.15.67:0/4176606089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060071310 0x7f3060071770 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f30600af390 tx=0x7f304c01ec70 comp rx=0 tx=0).stop 2024-04-17T10:32:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 -- 172.21.15.67:0/4176606089 shutdown_connections 2024-04-17T10:32:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 --2- 172.21.15.67:0/4176606089 >> v2:172.21.15.86:3300/0 conn(0x7f30600cc300 0x7f30600cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 --2- 172.21.15.67:0/4176606089 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060071310 0x7f3060071770 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 -- 172.21.15.67:0/4176606089 >> 172.21.15.67:0/4176606089 conn(0x7f306006ac90 msgr2=0x7f306006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 -- 172.21.15.67:0/4176606089 shutdown_connections 2024-04-17T10:32:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 -- 172.21.15.67:0/4176606089 wait complete. 2024-04-17T10:32:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.547+0000 7f3064c00640 1 Processor -- start 2024-04-17T10:32:14.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 -- start start 2024-04-17T10:32:14.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3060071310 0x7f30600a8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30600cc300 0x7f30600ae210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060145040 0x7f306015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306006df80 con 0x7f3060145040 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306006dca0 con 0x7f30600cc300 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306006d370 con 0x7f3060071310 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060145040 0x7f306015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060145040 0x7f306015ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35808/0 (socket says 172.21.15.67:35808) 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 -- 172.21.15.67:0/582709748 learned_addr learned my addr 172.21.15.67:0/582709748 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305de00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3060071310 0x7f30600a8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305d400640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30600cc300 0x7f30600ae210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 -- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3060071310 msgr2=0x7f30600a8850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3060071310 0x7f30600a8850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 -- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30600cc300 msgr2=0x7f30600ae210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30600cc300 0x7f30600ae210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 -- 172.21.15.67:0/582709748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30600ae750 con 0x7f3060145040 2024-04-17T10:32:14.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305d400640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30600cc300 0x7f30600ae210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:14.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f305e800640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060145040 0x7f306015ae50 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f305800cef0 tx=0x7f305800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3056000640 1 -- 172.21.15.67:0/582709748 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305801a070 con 0x7f3060145040 2024-04-17T10:32:14.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3056000640 1 -- 172.21.15.67:0/582709748 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305800d860 con 0x7f3060145040 2024-04-17T10:32:14.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30600a8e30 con 0x7f3060145040 2024-04-17T10:32:14.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3056000640 1 -- 172.21.15.67:0/582709748 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305800dc50 con 0x7f3060145040 2024-04-17T10:32:14.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30600a93f0 con 0x7f3060145040 2024-04-17T10:32:14.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.551+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3060072d50 con 0x7f3060145040 2024-04-17T10:32:14.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.555+0000 7f3056000640 1 -- 172.21.15.67:0/582709748 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f305801d070 con 0x7f3060145040 2024-04-17T10:32:14.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.555+0000 7f3056000640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3040066f60 0x7f3040069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.555+0000 7f3056000640 1 -- 172.21.15.67:0/582709748 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f30580859e0 con 0x7f3060145040 2024-04-17T10:32:14.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.555+0000 7f305de00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3040066f60 0x7f3040069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.555+0000 7f305de00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3040066f60 0x7f3040069420 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3050000f70 tx=0x7f3050009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.559+0000 7f3056000640 1 -- 172.21.15.67:0/582709748 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3058056850 con 0x7f3060145040 2024-04-17T10:32:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30600af120 con 0x7f3060145040 2024-04-17T10:32:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3056000640 1 -- 172.21.15.67:0/582709748 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f3058056150 con 0x7f3060145040 2024-04-17T10:32:14.749 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:14.750 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:31:37.749417+0000","modified":"2024-04-17T10:31:40.768850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7698":{"gid":7698,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3523329517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3523329517},{"type":"v1","addr":"172.21.15.67:6835","nonce":3523329517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-17T10:32:14.750 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 64 2024-04-17T10:32:14.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3040066f60 msgr2=0x7f3040069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3040066f60 0x7f3040069420 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3050000f70 tx=0x7f3050009040 comp rx=0 tx=0).stop 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060145040 msgr2=0x7f306015ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060145040 0x7f306015ae50 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f305800cef0 tx=0x7f305800a710 comp rx=0 tx=0).stop 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 shutdown_connections 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3040066f60 0x7f3040069420 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3060145040 0x7f306015ae50 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f30600cc300 0x7f30600ae210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 --2- 172.21.15.67:0/582709748 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3060071310 0x7f30600a8850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 >> 172.21.15.67:0/582709748 conn(0x7f306006ac90 msgr2=0x7f30600b0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 shutdown_connections 2024-04-17T10:32:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.758+0000 7f3064c00640 1 -- 172.21.15.67:0/582709748 wait complete. 2024-04-17T10:32:14.770 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:32:14.771 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:32:14.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.918+0000 7f0cc3600640 1 Processor -- start 2024-04-17T10:32:14.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.918+0000 7f0cc3600640 1 -- start start 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc3600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cbc0b07b0 0x7f0cbc070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc3600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc071300 0x7f0cbc06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc3600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0b5980 con 0x7f0cbc15dd10 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc3600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cbc0b1280 con 0x7f0cbc0b07b0 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc3600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0b0cc0 con 0x7f0cbc071300 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cb3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0cbc0b07b0 0x7f0cbc070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc0c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0cbc15dd10 0x7f0cbc0b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46872/0 (socket says 172.21.15.67:46872) 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc0c00640 1 -- 172.21.15.67:0/695842801 learned_addr learned my addr 172.21.15.67:0/695842801 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cc1600640 1 --2- 172.21.15.67:0/695842801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc071300 0x7f0cbc06c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cb3e00640 1 -- 172.21.15.67:0/695842801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc071300 msgr2=0x7f0cbc06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:14.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cb3e00640 1 --2- 172.21.15.67:0/695842801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc071300 0x7f0cbc06c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cb3e00640 1 -- 172.21.15.67:0/695842801 >> v1:172.21.15.67:6789/0 conn(0x7f0cbc15dd10 legacy=0x7f0cbc0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.922+0000 7f0cb3e00640 1 -- 172.21.15.67:0/695842801 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0b0fa0 con 0x7f0cbc0b07b0 2024-04-17T10:32:14.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cb3e00640 1 --2- 172.21.15.67:0/695842801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc0b07b0 0x7f0cbc070dc0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40097c0 tx=0x7f0ca401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2761cb33ed4104ba server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cbbe00640 1 -- 172.21.15.67:0/695842801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40b0070 con 0x7f0cbc0b07b0 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cbbe00640 1 -- 172.21.15.67:0/695842801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca40ab600 con 0x7f0cbc0b07b0 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cbbe00640 1 -- 172.21.15.67:0/695842801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca40ab9d0 con 0x7f0cbc0b07b0 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 -- 172.21.15.67:0/695842801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc0b07b0 msgr2=0x7f0cbc070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 --2- 172.21.15.67:0/695842801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc0b07b0 0x7f0cbc070dc0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40097c0 tx=0x7f0ca401c410 comp rx=0 tx=0).stop 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 -- 172.21.15.67:0/695842801 shutdown_connections 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 --2- 172.21.15.67:0/695842801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc071300 0x7f0cbc06c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 --2- 172.21.15.67:0/695842801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc0b07b0 0x7f0cbc070dc0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 -- 172.21.15.67:0/695842801 >> 172.21.15.67:0/695842801 conn(0x7f0cbc0b34d0 msgr2=0x7f0cbc0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 -- 172.21.15.67:0/695842801 shutdown_connections 2024-04-17T10:32:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 -- 172.21.15.67:0/695842801 wait complete. 2024-04-17T10:32:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 Processor -- start 2024-04-17T10:32:14.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.926+0000 7f0cc3600640 1 -- start start 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071300 0x7f0cbc1574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0b07b0 0x7f0cbc150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc15dd10 0x7f0cbc141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc06d5a0 con 0x7f0cbc071300 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc06d420 con 0x7f0cbc0b07b0 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc06d720 con 0x7f0cbc15dd10 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071300 0x7f0cbc1574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071300 0x7f0cbc1574b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35826/0 (socket says 172.21.15.67:35826) 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 -- 172.21.15.67:0/3080014352 learned_addr learned my addr 172.21.15.67:0/3080014352 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cb3e00640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0b07b0 0x7f0cbc150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc1600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc15dd10 0x7f0cbc141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 -- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc15dd10 msgr2=0x7f0cbc141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc15dd10 0x7f0cbc141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 -- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0b07b0 msgr2=0x7f0cbc150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0b07b0 0x7f0cbc150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 -- 172.21.15.67:0/3080014352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc151350 con 0x7f0cbc071300 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc0c00640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071300 0x7f0cbc1574b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c370 tx=0x7f0cac00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cb9600640 1 -- 172.21.15.67:0/3080014352 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac017070 con 0x7f0cbc071300 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cb9600640 1 -- 172.21.15.67:0/3080014352 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac005cf0 con 0x7f0cbc071300 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc151630 con 0x7f0cbc071300 2024-04-17T10:32:14.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cb9600640 1 -- 172.21.15.67:0/3080014352 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac0072f0 con 0x7f0cbc071300 2024-04-17T10:32:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc151bf0 con 0x7f0cbc071300 2024-04-17T10:32:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.930+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc074e30 con 0x7f0cbc071300 2024-04-17T10:32:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.934+0000 7f0cb9600640 1 -- 172.21.15.67:0/3080014352 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0cac01a070 con 0x7f0cbc071300 2024-04-17T10:32:14.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.934+0000 7f0cb9600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c98066f60 0x7f0c98069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.934+0000 7f0cb9600640 1 -- 172.21.15.67:0/3080014352 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0cac0844c0 con 0x7f0cbc071300 2024-04-17T10:32:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.934+0000 7f0cb3e00640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c98066f60 0x7f0c98069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.934+0000 7f0cb3e00640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c98066f60 0x7f0c98069420 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40ad000 tx=0x7f0ca4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:14.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:14.938+0000 7f0cb9600640 1 -- 172.21.15.67:0/3080014352 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0cac054ce0 con 0x7f0cbc071300 2024-04-17T10:32:15.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.154+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0cbc070b90 con 0x7f0cbc071300 2024-04-17T10:32:15.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.154+0000 7f0cb9600640 1 -- 172.21.15.67:0/3080014352 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1187 (secure 0 0 0) 0x7f0cac058d80 con 0x7f0cbc071300 2024-04-17T10:32:15.147 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:15.147 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":96,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076590372,"bytes_used":2570215424,"bytes_avail":770523897856,"bytes_total":773094113280,"write_bytes_sec":30508598,"read_op_per_sec":0,"write_op_per_sec":30},"fsmap":{"epoch":64,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"a","status":"up:active","gid":7698}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-04-17T10:31:00.223648+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:32:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c98066f60 msgr2=0x7f0c98069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c98066f60 0x7f0c98069420 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40ad000 tx=0x7f0ca4008040 comp rx=0 tx=0).stop 2024-04-17T10:32:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071300 msgr2=0x7f0cbc1574b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071300 0x7f0cbc1574b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f0cac00c370 tx=0x7f0cac00c830 comp rx=0 tx=0).stop 2024-04-17T10:32:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 shutdown_connections 2024-04-17T10:32:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c98066f60 0x7f0c98069420 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0cbc15dd10 0x7f0cbc141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0cbc0b07b0 0x7f0cbc150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 --2- 172.21.15.67:0/3080014352 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0cbc071300 0x7f0cbc1574b0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 >> 172.21.15.67:0/3080014352 conn(0x7f0cbc0b34d0 msgr2=0x7f0cbc0d0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:15.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 shutdown_connections 2024-04-17T10:32:15.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.158+0000 7f0cc3600640 1 -- 172.21.15.67:0/3080014352 wait complete. 2024-04-17T10:32:15.167 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:32:15.168 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:32:15.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:32:15.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:32:15.335 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:32:15.336 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:32:15.336 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:15.345 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:32:15.345 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:15.401 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:15.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.590+0000 7fe52e800640 1 Processor -- start 2024-04-17T10:32:15.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.590+0000 7fe52e800640 1 -- start start 2024-04-17T10:32:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe52e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 0x7fe5280ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe52e800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5281441f0 0x7fe5280ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe52e800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52815ae40 con 0x7fe52806eac0 2024-04-17T10:32:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe52e800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5280cf430 con 0x7fe5281441f0 2024-04-17T10:32:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe52e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe528071440 con 0x7fe52806e6e0 2024-04-17T10:32:15.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe51ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe52806eac0 0x7fe528143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46878/0 (socket says 172.21.15.67:46878) 2024-04-17T10:32:15.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe51ea00640 1 -- 172.21.15.67:0/1538676264 learned_addr learned my addr 172.21.15.67:0/1538676264 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527400640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 0x7fe5280ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527e00640 1 --2- 172.21.15.67:0/1538676264 >> v2:172.21.15.86:3300/0 conn(0x7fe5281441f0 0x7fe5280ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464119883 0 0) 0x7fe52815ae40 con 0x7fe52806eac0 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe50c003530 con 0x7fe52806eac0 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 21093453 0 0) 0x7fe50c003530 con 0x7fe52806eac0 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 msgr2=0x7fe5280ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 0x7fe5280ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 >> v2:172.21.15.86:3300/0 conn(0x7fe5281441f0 msgr2=0x7fe5280ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 --2- 172.21.15.67:0/1538676264 >> v2:172.21.15.86:3300/0 conn(0x7fe5281441f0 0x7fe5280ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528071720 con 0x7fe52806eac0 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fe510001680 con 0x7fe52806eac0 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 >> v1:172.21.15.67:6789/0 conn(0x7fe52806eac0 legacy=0x7fe528143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe50c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 0x7fe50c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe50c004d50 0x7fe50c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280cf430 con 0x7fe50c004d50 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528071440 con 0x7fe50c0039c0 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe50c00b9d0 con 0x7fe52806e6e0 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527400640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe50c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527e00640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe50c004d50 0x7fe50c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe51ea00640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 0x7fe50c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527400640 1 -- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 msgr2=0x7fe50c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527400640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe52806e6e0 0x7fe50c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527400640 1 -- 172.21.15.67:0/1538676264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe50c004d50 msgr2=0x7fe50c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:15.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527400640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe50c004d50 0x7fe50c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527400640 1 -- 172.21.15.67:0/1538676264 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe50c003530 con 0x7fe50c0039c0 2024-04-17T10:32:15.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.594+0000 7fe527e00640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe50c004d50 0x7fe50c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe527400640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe50c003de0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe518009ac0 tx=0x7fe51801c700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=314773d5ceeeb3b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5180b1070 con 0x7fe50c0039c0 2024-04-17T10:32:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5180a4a90 con 0x7fe50c0039c0 2024-04-17T10:32:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe526a00640 1 -- 172.21.15.67:0/1538676264 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe518009690 con 0x7fe50c0039c0 2024-04-17T10:32:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe52e800640 1 -- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 msgr2=0x7fe50c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe52e800640 1 --2- 172.21.15.67:0/1538676264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe50c003de0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe518009ac0 tx=0x7fe51801c700 comp rx=0 tx=0).stop 2024-04-17T10:32:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe527e00640 1 -- 172.21.15.67:0/1538676264 reap_dead start 2024-04-17T10:32:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe52e800640 1 -- 172.21.15.67:0/1538676264 shutdown_connections 2024-04-17T10:32:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe52e800640 1 -- 172.21.15.67:0/1538676264 >> 172.21.15.67:0/1538676264 conn(0x7fe52806ac90 msgr2=0x7fe52806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe52e800640 1 -- 172.21.15.67:0/1538676264 shutdown_connections 2024-04-17T10:32:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.598+0000 7fe52e800640 1 -- 172.21.15.67:0/1538676264 wait complete. 2024-04-17T10:32:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 Processor -- start 2024-04-17T10:32:15.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 -- start start 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe52813ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528150fc0 0x7fe5281e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5281e6d60 0x7fe5281f8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280b34c0 con 0x7fe528150fc0 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe528144600 con 0x7fe50c0039c0 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5280b3640 con 0x7fe5281e6d60 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528150fc0 0x7fe5281e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528150fc0 0x7fe5281e6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35848/0 (socket says 172.21.15.67:35848) 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 -- 172.21.15.67:0/2676525148 learned_addr learned my addr 172.21.15.67:0/2676525148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe527e00640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5281e6d60 0x7fe5281f8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe527400640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe52813ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 -- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5281e6d60 msgr2=0x7fe5281f8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5281e6d60 0x7fe5281f8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 -- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 msgr2=0x7fe52813ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe52813ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 -- 172.21.15.67:0/2676525148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52813b290 con 0x7fe528150fc0 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51ea00640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528150fc0 0x7fe5281e6820 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe510012550 tx=0x7fe510012a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51fe00640 1 -- 172.21.15.67:0/2676525148 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51001c070 con 0x7fe528150fc0 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51fe00640 1 -- 172.21.15.67:0/2676525148 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe510014830 con 0x7fe528150fc0 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe52813e520 con 0x7fe528150fc0 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe51fe00640 1 -- 172.21.15.67:0/2676525148 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe510014c20 con 0x7fe528150fc0 2024-04-17T10:32:15.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe52813eae0 con 0x7fe528150fc0 2024-04-17T10:32:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.602+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528072d50 con 0x7fe528150fc0 2024-04-17T10:32:15.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.606+0000 7fe51fe00640 1 -- 172.21.15.67:0/2676525148 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe51001f070 con 0x7fe528150fc0 2024-04-17T10:32:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.606+0000 7fe51fe00640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe504066f60 0x7fe504069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.606+0000 7fe51fe00640 1 -- 172.21.15.67:0/2676525148 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe510087b90 con 0x7fe528150fc0 2024-04-17T10:32:15.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.606+0000 7fe527400640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe504066f60 0x7fe504069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.606+0000 7fe527400640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe504066f60 0x7fe504069420 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe5180ae000 tx=0x7fe5180046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:15.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.610+0000 7fe51fe00640 1 -- 172.21.15.67:0/2676525148 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe5100583b0 con 0x7fe528150fc0 2024-04-17T10:32:15.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.770+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5281e7140 con 0x7fe528150fc0 2024-04-17T10:32:15.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe51fe00640 1 -- 172.21.15.67:0/2676525148 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fe51005c450 con 0x7fe528150fc0 2024-04-17T10:32:15.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:15.765 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:39.352312+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:31:36.162943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-17T10:31:36.740175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe504066f60 msgr2=0x7fe504069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe504066f60 0x7fe504069420 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe5180ae000 tx=0x7fe5180046d0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528150fc0 msgr2=0x7fe5281e6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528150fc0 0x7fe5281e6820 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7fe510012550 tx=0x7fe510012a10 comp rx=0 tx=0).stop 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 shutdown_connections 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe504066f60 0x7fe504069420 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5281e6d60 0x7fe5281f8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528150fc0 0x7fe5281e6820 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 --2- 172.21.15.67:0/2676525148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe50c0039c0 0x7fe52813ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 >> 172.21.15.67:0/2676525148 conn(0x7fe52806ac90 msgr2=0x7fe5280a9b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:15.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 shutdown_connections 2024-04-17T10:32:15.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.774+0000 7fe52e800640 1 -- 172.21.15.67:0/2676525148 wait complete. 2024-04-17T10:32:15.785 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:31:36.162943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:31:36.740175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:32:15.786 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:32:15.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.938+0000 7fe1a4a00640 1 Processor -- start 2024-04-17T10:32:15.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.938+0000 7fe1a4a00640 1 -- start start 2024-04-17T10:32:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.938+0000 7fe1a4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0071af0 0x7fe1a00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.938+0000 7fe1a4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a015cb30 con 0x7fe1a015be70 2024-04-17T10:32:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.938+0000 7fe1a4a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a00cd610 con 0x7fe1a0159440 2024-04-17T10:32:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.938+0000 7fe1a4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00a8650 con 0x7fe1a0071af0 2024-04-17T10:32:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe19e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe1a015be70 0x7fe1a015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46882/0 (socket says 172.21.15.67:46882) 2024-04-17T10:32:15.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe19e800640 1 -- 172.21.15.67:0/3078645549 learned_addr learned my addr 172.21.15.67:0/3078645549 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe197e00640 1 --2- 172.21.15.67:0/3078645549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0071af0 0x7fe1a00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe19d400640 1 -- 172.21.15.67:0/3078645549 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390597200 0 0) 0x7fe1a015cb30 con 0x7fe1a015be70 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe19d400640 1 -- 172.21.15.67:0/3078645549 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe184003530 con 0x7fe1a015be70 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe19d400640 1 -- 172.21.15.67:0/3078645549 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806742252 0 0) 0x7fe1a00cd610 con 0x7fe1a0159440 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe19d400640 1 -- 172.21.15.67:0/3078645549 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1a015cb30 con 0x7fe1a0159440 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe197e00640 1 -- 172.21.15.67:0/3078645549 >> v1:172.21.15.86:6789/0 conn(0x7fe1a0159440 legacy=0x7fe1a00713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe197e00640 1 -- 172.21.15.67:0/3078645549 >> v1:172.21.15.67:6789/0 conn(0x7fe1a015be70 legacy=0x7fe1a015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe197e00640 1 -- 172.21.15.67:0/3078645549 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00cd8f0 con 0x7fe1a0071af0 2024-04-17T10:32:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.942+0000 7fe197e00640 1 --2- 172.21.15.67:0/3078645549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0071af0 0x7fe1a00ce0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097f0 tx=0x7fe18801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a439b110fe4b978f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe19d400640 1 -- 172.21.15.67:0/3078645549 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1880ab5f0 con 0x7fe1a0071af0 2024-04-17T10:32:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe19d400640 1 -- 172.21.15.67:0/3078645549 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1880b0070 con 0x7fe1a0071af0 2024-04-17T10:32:15.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe19d400640 1 -- 172.21.15.67:0/3078645549 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1880ab990 con 0x7fe1a0071af0 2024-04-17T10:32:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- 172.21.15.67:0/3078645549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0071af0 msgr2=0x7fe1a00ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/3078645549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0071af0 0x7fe1a00ce0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097f0 tx=0x7fe18801c4e0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- 172.21.15.67:0/3078645549 shutdown_connections 2024-04-17T10:32:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/3078645549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0071af0 0x7fe1a00ce0f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- 172.21.15.67:0/3078645549 >> 172.21.15.67:0/3078645549 conn(0x7fe1a00b1740 msgr2=0x7fe1a00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- 172.21.15.67:0/3078645549 shutdown_connections 2024-04-17T10:32:15.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- 172.21.15.67:0/3078645549 wait complete. 2024-04-17T10:32:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 Processor -- start 2024-04-17T10:32:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- start start 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a0071af0 0x7fe1a015ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0159440 0x7fe1a014d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a015be70 0x7fe1a00acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a015dc50 con 0x7fe1a0071af0 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a015dad0 con 0x7fe1a015be70 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe1a4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a015ddd0 con 0x7fe1a0159440 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe19de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a0071af0 0x7fe1a015ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe19de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a0071af0 0x7fe1a015ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35854/0 (socket says 172.21.15.67:35854) 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe19de00640 1 -- 172.21.15.67:0/854645769 learned_addr learned my addr 172.21.15.67:0/854645769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe197e00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0159440 0x7fe1a014d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.946+0000 7fe19e800640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a015be70 0x7fe1a00acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe19de00640 1 -- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0159440 msgr2=0x7fe1a014d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe19de00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0159440 0x7fe1a014d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe19de00640 1 -- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a015be70 msgr2=0x7fe1a00acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe19de00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a015be70 0x7fe1a00acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:15.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe19de00640 1 -- 172.21.15.67:0/854645769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a014da40 con 0x7fe1a0071af0 2024-04-17T10:32:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe19de00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a0071af0 0x7fe1a015ae40 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c430 tx=0x7fe19000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe196000640 1 -- 172.21.15.67:0/854645769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190017070 con 0x7fe1a0071af0 2024-04-17T10:32:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe196000640 1 -- 172.21.15.67:0/854645769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1900052f0 con 0x7fe1a0071af0 2024-04-17T10:32:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe196000640 1 -- 172.21.15.67:0/854645769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190003840 con 0x7fe1a0071af0 2024-04-17T10:32:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a00b41a0 con 0x7fe1a0071af0 2024-04-17T10:32:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a00ad970 con 0x7fe1a0071af0 2024-04-17T10:32:15.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a004ef30 con 0x7fe1a0071af0 2024-04-17T10:32:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe196000640 1 -- 172.21.15.67:0/854645769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe19001a070 con 0x7fe1a0071af0 2024-04-17T10:32:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe196000640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe17c066f60 0x7fe17c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe196000640 1 -- 172.21.15.67:0/854645769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe190084430 con 0x7fe1a0071af0 2024-04-17T10:32:15.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.950+0000 7fe197e00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe17c066f60 0x7fe17c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:15.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.954+0000 7fe197e00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe17c066f60 0x7fe17c069420 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe1880af000 tx=0x7fe1880a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:15.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:15.954+0000 7fe196000640 1 -- 172.21.15.67:0/854645769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe190054c50 con 0x7fe1a0071af0 2024-04-17T10:32:16.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe1a006c3e0 con 0x7fe1a0071af0 2024-04-17T10:32:16.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe196000640 1 -- 172.21.15.67:0/854645769 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe190058cf0 con 0x7fe1a0071af0 2024-04-17T10:32:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:32:16.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe17c066f60 msgr2=0x7fe17c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe17c066f60 0x7fe17c069420 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe1880af000 tx=0x7fe1880a7000 comp rx=0 tx=0).stop 2024-04-17T10:32:16.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a0071af0 msgr2=0x7fe1a015ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a0071af0 0x7fe1a015ae40 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c430 tx=0x7fe19000c8f0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 shutdown_connections 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe17c066f60 0x7fe17c069420 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe1a015be70 0x7fe1a00acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe1a0159440 0x7fe1a014d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 --2- 172.21.15.67:0/854645769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1a0071af0 0x7fe1a015ae40 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 >> 172.21.15.67:0/854645769 conn(0x7fe1a00b1740 msgr2=0x7fe1a0150eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 shutdown_connections 2024-04-17T10:32:16.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.118+0000 7fe1a4a00640 1 -- 172.21.15.67:0/854645769 wait complete. 2024-04-17T10:32:16.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:32:16.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa24200640 1 Processor -- start 2024-04-17T10:32:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa24200640 1 -- start start 2024-04-17T10:32:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa24200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faa1c15dd60 0x7faa1c161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa24200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa24200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c15e140 con 0x7faa1c0cff90 2024-04-17T10:32:16.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa24200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c1626d0 con 0x7faa1c15dd60 2024-04-17T10:32:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa24200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c099d50 con 0x7faa1c09adf0 2024-04-17T10:32:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa22200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faa1c0cff90 0x7faa1c0d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46884/0 (socket says 172.21.15.67:46884) 2024-04-17T10:32:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa22200640 1 -- 172.21.15.67:0/2965659955 learned_addr learned my addr 172.21.15.67:0/2965659955 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa20e00640 1 --2- 172.21.15.67:0/2965659955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.282+0000 7faa21800640 1 --2- 172.21.15.67:0/2965659955 >> v2:172.21.15.86:3300/0 conn(0x7faa1c15dd60 0x7faa1c161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa1be00640 1 -- 172.21.15.67:0/2965659955 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759620288 0 0) 0x7faa1c15e140 con 0x7faa1c0cff90 2024-04-17T10:32:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa1be00640 1 -- 172.21.15.67:0/2965659955 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa04003530 con 0x7faa1c0cff90 2024-04-17T10:32:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa20e00640 1 -- 172.21.15.67:0/2965659955 >> v2:172.21.15.86:3300/0 conn(0x7faa1c15dd60 msgr2=0x7faa1c161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa20e00640 1 --2- 172.21.15.67:0/2965659955 >> v2:172.21.15.86:3300/0 conn(0x7faa1c15dd60 0x7faa1c161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa1be00640 1 -- 172.21.15.67:0/2965659955 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086568722 0 0) 0x7faa04003530 con 0x7faa1c0cff90 2024-04-17T10:32:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa20e00640 1 -- 172.21.15.67:0/2965659955 >> v1:172.21.15.67:6789/0 conn(0x7faa1c0cff90 legacy=0x7faa1c0d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa20e00640 1 -- 172.21.15.67:0/2965659955 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c09a030 con 0x7faa1c09adf0 2024-04-17T10:32:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa20e00640 1 --2- 172.21.15.67:0/2965659955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c161dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faa080097c0 tx=0x7faa0801c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26886b5af261516 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa1be00640 1 -- 172.21.15.67:0/2965659955 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa080b0070 con 0x7faa1c09adf0 2024-04-17T10:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa1be00640 1 -- 172.21.15.67:0/2965659955 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa080ab400 con 0x7faa1c09adf0 2024-04-17T10:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.286+0000 7faa1be00640 1 -- 172.21.15.67:0/2965659955 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14001a00 con 0x7faa1c09adf0 2024-04-17T10:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- 172.21.15.67:0/2965659955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 msgr2=0x7faa1c161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 --2- 172.21.15.67:0/2965659955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c161dc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7faa080097c0 tx=0x7faa0801c390 comp rx=0 tx=0).stop 2024-04-17T10:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- 172.21.15.67:0/2965659955 shutdown_connections 2024-04-17T10:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 --2- 172.21.15.67:0/2965659955 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c161dc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 --2- 172.21.15.67:0/2965659955 >> v2:172.21.15.86:3300/0 conn(0x7faa1c15dd60 0x7faa1c161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- 172.21.15.67:0/2965659955 >> 172.21.15.67:0/2965659955 conn(0x7faa1c0b34a0 msgr2=0x7faa1c0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- 172.21.15.67:0/2965659955 shutdown_connections 2024-04-17T10:32:16.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- 172.21.15.67:0/2965659955 wait complete. 2024-04-17T10:32:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 Processor -- start 2024-04-17T10:32:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- start start 2024-04-17T10:32:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c0af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa1c0cff90 0x7faa1c152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa1c15dd60 0x7faa1c1556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c06c730 con 0x7faa1c0cff90 2024-04-17T10:32:16.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c06c5b0 con 0x7faa1c15dd60 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa24200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c06c8b0 con 0x7faa1c09adf0 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa1c0cff90 0x7faa1c152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa1c0cff90 0x7faa1c152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35860/0 (socket says 172.21.15.67:35860) 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 -- 172.21.15.67:0/3332608530 learned_addr learned my addr 172.21.15.67:0/3332608530 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 -- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 msgr2=0x7faa1c0af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c0af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 -- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa1c15dd60 msgr2=0x7faa1c1556a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa1c15dd60 0x7faa1c1556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 -- 172.21.15.67:0/3332608530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c0b0040 con 0x7faa1c0cff90 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.290+0000 7faa20e00640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa1c0cff90 0x7faa1c152d10 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7faa080ae040 tx=0x7faa08019130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa19600640 1 -- 172.21.15.67:0/3332608530 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa080b0040 con 0x7faa1c0cff90 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa19600640 1 -- 172.21.15.67:0/3332608530 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa080b9070 con 0x7faa1c0cff90 2024-04-17T10:32:16.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa19600640 1 -- 172.21.15.67:0/3332608530 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa080b4890 con 0x7faa1c0cff90 2024-04-17T10:32:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c0af7a0 con 0x7faa1c0cff90 2024-04-17T10:32:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c0afd60 con 0x7faa1c0cff90 2024-04-17T10:32:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c04ef30 con 0x7faa1c0cff90 2024-04-17T10:32:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa19600640 1 -- 172.21.15.67:0/3332608530 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7faa080b4a30 con 0x7faa1c0cff90 2024-04-17T10:32:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa19600640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa9fc066f60 0x7fa9fc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa19600640 1 -- 172.21.15.67:0/3332608530 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faa0811f250 con 0x7faa1c0cff90 2024-04-17T10:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.294+0000 7faa21800640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa9fc066f60 0x7fa9fc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.298+0000 7faa21800640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa9fc066f60 0x7fa9fc069420 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7faa10006aa0 tx=0x7faa10006a30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.298+0000 7faa19600640 1 -- 172.21.15.67:0/3332608530 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7faa08015fb0 con 0x7faa1c0cff90 2024-04-17T10:32:16.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.458+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa1c0b42d0 con 0x7faa1c0cff90 2024-04-17T10:32:16.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.458+0000 7faa19600640 1 -- 172.21.15.67:0/3332608530 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa080f0030 con 0x7faa1c0cff90 2024-04-17T10:32:16.453 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:32:16.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa9fc066f60 msgr2=0x7fa9fc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa9fc066f60 0x7fa9fc069420 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7faa10006aa0 tx=0x7faa10006a30 comp rx=0 tx=0).stop 2024-04-17T10:32:16.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa1c0cff90 msgr2=0x7faa1c152d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa1c0cff90 0x7faa1c152d10 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7faa080ae040 tx=0x7faa08019130 comp rx=0 tx=0).stop 2024-04-17T10:32:16.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 shutdown_connections 2024-04-17T10:32:16.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa9fc066f60 0x7fa9fc069420 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa1c15dd60 0x7faa1c1556a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa1c0cff90 0x7faa1c152d10 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 --2- 172.21.15.67:0/3332608530 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa1c09adf0 0x7faa1c0af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 >> 172.21.15.67:0/3332608530 conn(0x7faa1c0b34a0 msgr2=0x7faa1c161f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:16.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 shutdown_connections 2024-04-17T10:32:16.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.462+0000 7faa24200640 1 -- 172.21.15.67:0/3332608530 wait complete. 2024-04-17T10:32:16.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:32:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7861a00640 1 Processor -- start 2024-04-17T10:32:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7861a00640 1 -- start start 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7861a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f785c15dbb0 0x7f785c0c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7861a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c0d14b0 con 0x7f785c06ac90 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7861a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c15df90 con 0x7f785c0cfd80 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7861a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785c0ce8e0 con 0x7f785c15dbb0 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7853e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f785c06ac90 0x7f785c0d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46896/0 (socket says 172.21.15.67:46896) 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f7853e00640 1 -- 172.21.15.67:0/2817701794 learned_addr learned my addr 172.21.15.67:0/2817701794 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f785aa00640 1 --2- 172.21.15.67:0/2817701794 >> v2:172.21.15.86:3300/0 conn(0x7f785c15dbb0 0x7f785c0c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f785a000640 1 -- 172.21.15.67:0/2817701794 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384875754 0 0) 0x7f785c0d14b0 con 0x7f785c06ac90 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f785a000640 1 -- 172.21.15.67:0/2817701794 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7840003530 con 0x7f785c06ac90 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f785aa00640 1 -- 172.21.15.67:0/2817701794 >> v1:172.21.15.86:6789/0 conn(0x7f785c0cfd80 legacy=0x7f785c0d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f785aa00640 1 -- 172.21.15.67:0/2817701794 >> v1:172.21.15.67:6789/0 conn(0x7f785c06ac90 legacy=0x7f785c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.626+0000 7f785aa00640 1 -- 172.21.15.67:0/2817701794 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c0ca3b0 con 0x7f785c15dbb0 2024-04-17T10:32:16.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f785aa00640 1 --2- 172.21.15.67:0/2817701794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c15dbb0 0x7f785c0c9a30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f785c0cea60 tx=0x7f784c01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6980727bb9b7fea9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f785a000640 1 -- 172.21.15.67:0/2817701794 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c0b1070 con 0x7f785c15dbb0 2024-04-17T10:32:16.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f785a000640 1 -- 172.21.15.67:0/2817701794 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c0ac5f0 con 0x7f785c15dbb0 2024-04-17T10:32:16.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f785a000640 1 -- 172.21.15.67:0/2817701794 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c0ac9c0 con 0x7f785c15dbb0 2024-04-17T10:32:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f7861a00640 1 -- 172.21.15.67:0/2817701794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c15dbb0 msgr2=0x7f785c0c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f7861a00640 1 --2- 172.21.15.67:0/2817701794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c15dbb0 0x7f785c0c9a30 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f785c0cea60 tx=0x7f784c01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f7861a00640 1 -- 172.21.15.67:0/2817701794 shutdown_connections 2024-04-17T10:32:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f7861a00640 1 --2- 172.21.15.67:0/2817701794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c15dbb0 0x7f785c0c9a30 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f7861a00640 1 -- 172.21.15.67:0/2817701794 >> 172.21.15.67:0/2817701794 conn(0x7f785c099a50 msgr2=0x7f785c099e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f7861a00640 1 -- 172.21.15.67:0/2817701794 shutdown_connections 2024-04-17T10:32:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.630+0000 7f7861a00640 1 -- 172.21.15.67:0/2817701794 wait complete. 2024-04-17T10:32:16.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 Processor -- start 2024-04-17T10:32:16.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 -- start start 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f785c06ac90 0x7f785c148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c0cfd80 0x7f785c149ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785c15dbb0 0x7f785c14c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c06b600 con 0x7f785c15dbb0 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c06b480 con 0x7f785c0cfd80 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c0cd9e0 con 0x7f785c06ac90 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785c15dbb0 0x7f785c14c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785c15dbb0 0x7f785c14c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35864/0 (socket says 172.21.15.67:35864) 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 -- 172.21.15.67:0/2819690609 learned_addr learned my addr 172.21.15.67:0/2819690609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 -- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f785c06ac90 msgr2=0x7f785c148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7853e00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c0cfd80 0x7f785c149ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f785c06ac90 0x7f785c148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 -- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c0cfd80 msgr2=0x7f785c149ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c0cfd80 0x7f785c149ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 -- 172.21.15.67:0/2819690609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c14b7b0 con 0x7f785c15dbb0 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7853e00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c0cfd80 0x7f785c149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f785b400640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785c15dbb0 0x7f785c14c400 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f785400c7a0 tx=0x7f785400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7852a00640 1 -- 172.21.15.67:0/2819690609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854003b80 con 0x7f785c15dbb0 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7852a00640 1 -- 172.21.15.67:0/2819690609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854003d20 con 0x7f785c15dbb0 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7852a00640 1 -- 172.21.15.67:0/2819690609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785400aa90 con 0x7f785c15dbb0 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c14a710 con 0x7f785c15dbb0 2024-04-17T10:32:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c14acd0 con 0x7f785c15dbb0 2024-04-17T10:32:16.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.634+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c14b3e0 con 0x7f785c15dbb0 2024-04-17T10:32:16.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.638+0000 7f7852a00640 1 -- 172.21.15.67:0/2819690609 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f785400ac30 con 0x7f785c15dbb0 2024-04-17T10:32:16.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.638+0000 7f7852a00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7838066f90 0x7f7838069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.638+0000 7f7852a00640 1 -- 172.21.15.67:0/2819690609 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78540851b0 con 0x7f785c15dbb0 2024-04-17T10:32:16.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.638+0000 7f785aa00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7838066f90 0x7f7838069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.638+0000 7f785aa00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7838066f90 0x7f7838069450 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f784c0ae000 tx=0x7f784c004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.642+0000 7f7852a00640 1 -- 172.21.15.67:0/2819690609 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7854055f90 con 0x7f785c15dbb0 2024-04-17T10:32:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.802+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f785c1569f0 con 0x7f785c15dbb0 2024-04-17T10:32:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.802+0000 7f7852a00640 1 -- 172.21.15.67:0/2819690609 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7854055890 con 0x7f785c15dbb0 2024-04-17T10:32:16.796 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:32:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7838066f90 msgr2=0x7f7838069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7838066f90 0x7f7838069450 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f784c0ae000 tx=0x7f784c004740 comp rx=0 tx=0).stop 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785c15dbb0 msgr2=0x7f785c14c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785c15dbb0 0x7f785c14c400 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f785400c7a0 tx=0x7f785400cc60 comp rx=0 tx=0).stop 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 shutdown_connections 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7838066f90 0x7f7838069450 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f785c15dbb0 0x7f785c14c400 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f785c0cfd80 0x7f785c149ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 --2- 172.21.15.67:0/2819690609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f785c06ac90 0x7f785c148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 >> 172.21.15.67:0/2819690609 conn(0x7f785c099a50 msgr2=0x7f785c0b6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 shutdown_connections 2024-04-17T10:32:16.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.806+0000 7f7861a00640 1 -- 172.21.15.67:0/2819690609 wait complete. 2024-04-17T10:32:16.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:32:16.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0098e00640 1 Processor -- start 2024-04-17T10:32:16.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0098e00640 1 -- start start 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0098e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f009406e6e0 0x7f00940ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0098e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f009406eac0 0x7f00940ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0098e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009415ae40 con 0x7f0094143ae0 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0098e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00940cf430 con 0x7f009406e6e0 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0098e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0094071440 con 0x7f009406eac0 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0091e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f009406e6e0 0x7f00940ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0091e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f009406e6e0 0x7f00940ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35872/0 (socket says 172.21.15.67:35872) 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0091e00640 1 -- 172.21.15.67:0/3620897797 learned_addr learned my addr 172.21.15.67:0/3620897797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0092800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0094143ae0 0x7f0094143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46904/0 (socket says 172.21.15.67:46904) 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.970+0000 7f0089400640 1 --2- 172.21.15.67:0/3620897797 >> v2:172.21.15.86:3300/0 conn(0x7f009406eac0 0x7f00940ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091e00640 1 -- 172.21.15.67:0/3620897797 >> v2:172.21.15.86:3300/0 conn(0x7f009406eac0 msgr2=0x7f00940ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:16.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091e00640 1 --2- 172.21.15.67:0/3620897797 >> v2:172.21.15.86:3300/0 conn(0x7f009406eac0 0x7f00940ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091e00640 1 -- 172.21.15.67:0/3620897797 >> v1:172.21.15.67:6789/0 conn(0x7f0094143ae0 legacy=0x7f0094143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091e00640 1 -- 172.21.15.67:0/3620897797 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0094071720 con 0x7f009406e6e0 2024-04-17T10:32:16.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091e00640 1 --2- 172.21.15.67:0/3620897797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009406e6e0 0x7f00940ce9b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f009406ef80 tx=0x7f008401eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8a19ebe9330a10 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091400640 1 -- 172.21.15.67:0/3620897797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008400b2d0 con 0x7f009406e6e0 2024-04-17T10:32:16.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091400640 1 -- 172.21.15.67:0/3620897797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008400b470 con 0x7f009406e6e0 2024-04-17T10:32:16.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0091400640 1 -- 172.21.15.67:0/3620897797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00840a7b20 con 0x7f009406e6e0 2024-04-17T10:32:16.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 -- 172.21.15.67:0/3620897797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009406e6e0 msgr2=0x7f00940ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 --2- 172.21.15.67:0/3620897797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009406e6e0 0x7f00940ce9b0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f009406ef80 tx=0x7f008401eb30 comp rx=0 tx=0).stop 2024-04-17T10:32:16.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 -- 172.21.15.67:0/3620897797 shutdown_connections 2024-04-17T10:32:16.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 --2- 172.21.15.67:0/3620897797 >> v2:172.21.15.86:3300/0 conn(0x7f009406eac0 0x7f00940ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 --2- 172.21.15.67:0/3620897797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f009406e6e0 0x7f00940ce9b0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 -- 172.21.15.67:0/3620897797 >> 172.21.15.67:0/3620897797 conn(0x7f009406ac90 msgr2=0x7f009406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:16.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 -- 172.21.15.67:0/3620897797 shutdown_connections 2024-04-17T10:32:16.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.974+0000 7f0098e00640 1 -- 172.21.15.67:0/3620897797 wait complete. 2024-04-17T10:32:16.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 Processor -- start 2024-04-17T10:32:16.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 -- start start 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f009406e6e0 0x7f00941381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f009406eac0 0x7f0094140500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0094143ae0 0x7f0094142060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009406dd00 con 0x7f0094143ae0 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009406db80 con 0x7f009406e6e0 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009406de80 con 0x7f009406eac0 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0094143ae0 0x7f0094142060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0094143ae0 0x7f0094142060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35876/0 (socket says 172.21.15.67:35876) 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 -- 172.21.15.67:0/2155728022 learned_addr learned my addr 172.21.15.67:0/2155728022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0091e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f009406e6e0 0x7f00941381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0089400640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f009406eac0 0x7f0094140500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 -- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f009406eac0 msgr2=0x7f0094140500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f009406eac0 0x7f0094140500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 -- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f009406e6e0 msgr2=0x7f00941381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f009406e6e0 0x7f00941381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 -- 172.21.15.67:0/2155728022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00941386f0 con 0x7f0094143ae0 2024-04-17T10:32:16.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0092800640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0094143ae0 0x7f0094142060 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c3c0 tx=0x7f008c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f008aa00640 1 -- 172.21.15.67:0/2155728022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c0074e0 con 0x7f0094143ae0 2024-04-17T10:32:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f008aa00640 1 -- 172.21.15.67:0/2155728022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008c007680 con 0x7f0094143ae0 2024-04-17T10:32:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f008aa00640 1 -- 172.21.15.67:0/2155728022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008c004a10 con 0x7f0094143ae0 2024-04-17T10:32:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0094140fc0 con 0x7f0094143ae0 2024-04-17T10:32:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.978+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0094141580 con 0x7f0094143ae0 2024-04-17T10:32:16.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.982+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0094072d50 con 0x7f0094143ae0 2024-04-17T10:32:16.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.982+0000 7f008aa00640 1 -- 172.21.15.67:0/2155728022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f008c01a020 con 0x7f0094143ae0 2024-04-17T10:32:16.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.982+0000 7f008aa00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0070066f60 0x7f0070069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:16.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.982+0000 7f008aa00640 1 -- 172.21.15.67:0/2155728022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f008c084890 con 0x7f0094143ae0 2024-04-17T10:32:16.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.982+0000 7f0091e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0070066f60 0x7f0070069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:16.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.982+0000 7f0091e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0070066f60 0x7f0070069420 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f00840b0000 tx=0x7f00840a9070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:16.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:16.986+0000 7f008aa00640 1 -- 172.21.15.67:0/2155728022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f008c055020 con 0x7f0094143ae0 2024-04-17T10:32:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.158+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0094074e30 con 0x7f0094143ae0 2024-04-17T10:32:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.158+0000 7f008aa00640 1 -- 172.21.15.67:0/2155728022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v96) v1 ==== 80+0+0 (secure 0 0 0) 0x7f008c0590c0 con 0x7f0094143ae0 2024-04-17T10:32:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:32:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0070066f60 msgr2=0x7f0070069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0070066f60 0x7f0070069420 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f00840b0000 tx=0x7f00840a9070 comp rx=0 tx=0).stop 2024-04-17T10:32:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0094143ae0 msgr2=0x7f0094142060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0094143ae0 0x7f0094142060 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c3c0 tx=0x7f008c00c880 comp rx=0 tx=0).stop 2024-04-17T10:32:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 shutdown_connections 2024-04-17T10:32:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0070066f60 0x7f0070069420 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0094143ae0 0x7f0094142060 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f009406eac0 0x7f0094140500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 --2- 172.21.15.67:0/2155728022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f009406e6e0 0x7f00941381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 >> 172.21.15.67:0/2155728022 conn(0x7f009406ac90 msgr2=0x7f00940b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 shutdown_connections 2024-04-17T10:32:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:17.162+0000 7f0098e00640 1 -- 172.21.15.67:0/2155728022 wait complete. 2024-04-17T10:32:17.172 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:32:17.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:17.172 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:32:17.879 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:32:17.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:17.879 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:32:17.889 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:32:17.889 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:17.898 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:32:17.898 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:32:17.898 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:17.939 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:32:17.940 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:32:17.940 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:17.948 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:32:17.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.134+0000 7f5e42200640 1 Processor -- start 2024-04-17T10:32:18.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.134+0000 7f5e42200640 1 -- start start 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e42200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e3c1466e0 0x7f5e3c06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e42200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c146da0 0x7f5e3c071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e42200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c161740 con 0x7f5e3c071850 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e42200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c161a20 con 0x7f5e3c1466e0 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e42200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c161d00 con 0x7f5e3c146da0 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e3b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e3c1466e0 0x7f5e3c06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e3b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5e3c1466e0 0x7f5e3c06eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35890/0 (socket says 172.21.15.67:35890) 2024-04-17T10:32:18.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e3b400640 1 -- 172.21.15.67:0/1521190668 learned_addr learned my addr 172.21.15.67:0/1521190668 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e3b400640 1 -- 172.21.15.67:0/1521190668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c146da0 msgr2=0x7f5e3c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e3b400640 1 --2- 172.21.15.67:0/1521190668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c146da0 0x7f5e3c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e3b400640 1 -- 172.21.15.67:0/1521190668 >> v1:172.21.15.86:6789/0 conn(0x7f5e3c071850 legacy=0x7f5e3c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:18.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.138+0000 7f5e3b400640 1 -- 172.21.15.67:0/1521190668 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c161fe0 con 0x7f5e3c1466e0 2024-04-17T10:32:18.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e3b400640 1 --2- 172.21.15.67:0/1521190668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c06eb70 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009870 tx=0x7f5e2c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b6e9033e1c8084 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e3aa00640 1 -- 172.21.15.67:0/1521190668 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0b0070 con 0x7f5e3c1466e0 2024-04-17T10:32:18.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e3aa00640 1 -- 172.21.15.67:0/1521190668 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c0ab600 con 0x7f5e3c1466e0 2024-04-17T10:32:18.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e3aa00640 1 -- 172.21.15.67:0/1521190668 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0ab9d0 con 0x7f5e3c1466e0 2024-04-17T10:32:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- 172.21.15.67:0/1521190668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 msgr2=0x7f5e3c06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 --2- 172.21.15.67:0/1521190668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c06eb70 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c009870 tx=0x7f5e2c01c520 comp rx=0 tx=0).stop 2024-04-17T10:32:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- 172.21.15.67:0/1521190668 shutdown_connections 2024-04-17T10:32:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 --2- 172.21.15.67:0/1521190668 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c146da0 0x7f5e3c071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 --2- 172.21.15.67:0/1521190668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c06eb70 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- 172.21.15.67:0/1521190668 >> 172.21.15.67:0/1521190668 conn(0x7f5e3c06ac90 msgr2=0x7f5e3c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- 172.21.15.67:0/1521190668 shutdown_connections 2024-04-17T10:32:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- 172.21.15.67:0/1521190668 wait complete. 2024-04-17T10:32:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 Processor -- start 2024-04-17T10:32:18.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- start start 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c071850 0x7f5e3c0cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c0af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e3c146da0 0x7f5e3c0af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c16aa40 con 0x7f5e3c1466e0 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0af200 con 0x7f5e3c146da0 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e42200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c16abc0 con 0x7f5e3c071850 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.142+0000 7f5e33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c0af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c0af3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35904/0 (socket says 172.21.15.67:35904) 2024-04-17T10:32:18.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 -- 172.21.15.67:0/3740307143 learned_addr learned my addr 172.21.15.67:0/3740307143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e3be00640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e3c146da0 0x7f5e3c0af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e3b400640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c071850 0x7f5e3c0cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 -- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c071850 msgr2=0x7f5e3c0cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c071850 0x7f5e3c0cb7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 -- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e3c146da0 msgr2=0x7f5e3c0af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e3c146da0 0x7f5e3c0af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 -- 172.21.15.67:0/3740307143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c165050 con 0x7f5e3c1466e0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33e00640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c0af3b0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400b3f0 tx=0x7f5e2400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33400640 1 -- 172.21.15.67:0/3740307143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24015020 con 0x7f5e3c1466e0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33400640 1 -- 172.21.15.67:0/3740307143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e24011830 con 0x7f5e3c1466e0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33400640 1 -- 172.21.15.67:0/3740307143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e24011c20 con 0x7f5e3c1466e0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c165330 con 0x7f5e3c1466e0 2024-04-17T10:32:18.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c1658f0 con 0x7f5e3c1466e0 2024-04-17T10:32:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c072d50 con 0x7f5e3c1466e0 2024-04-17T10:32:18.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33400640 1 -- 172.21.15.67:0/3740307143 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5e24018020 con 0x7f5e3c1466e0 2024-04-17T10:32:18.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33400640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e18066f60 0x7f5e18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.146+0000 7f5e33400640 1 -- 172.21.15.67:0/3740307143 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5e240851c0 con 0x7f5e3c1466e0 2024-04-17T10:32:18.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.150+0000 7f5e3b400640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e18066f60 0x7f5e18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.150+0000 7f5e3b400640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e18066f60 0x7f5e18069420 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0ad000 tx=0x7f5e2c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.150+0000 7f5e33400640 1 -- 172.21.15.67:0/3740307143 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5e2401f020 con 0x7f5e3c1466e0 2024-04-17T10:32:18.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.346+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e3c074e30 con 0x7f5e3c1466e0 2024-04-17T10:32:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.346+0000 7f5e33400640 1 -- 172.21.15.67:0/3740307143 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f5e240172c0 con 0x7f5e3c1466e0 2024-04-17T10:32:18.339 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:18.339 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:31:37.749417+0000","modified":"2024-04-17T10:31:40.768850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7698":{"gid":7698,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3523329517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3523329517},{"type":"v1","addr":"172.21.15.67:6835","nonce":3523329517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-17T10:32:18.339 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 64 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e18066f60 msgr2=0x7f5e18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e18066f60 0x7f5e18069420 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c0ad000 tx=0x7f5e2c0a6000 comp rx=0 tx=0).stop 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 msgr2=0x7f5e3c0af3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c0af3b0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f5e2400b3f0 tx=0x7f5e2400b8b0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 shutdown_connections 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5e18066f60 0x7f5e18069420 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5e3c146da0 0x7f5e3c0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e3c1466e0 0x7f5e3c0af3b0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 --2- 172.21.15.67:0/3740307143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5e3c071850 0x7f5e3c0cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 >> 172.21.15.67:0/3740307143 conn(0x7f5e3c06ac90 msgr2=0x7f5e3c0b1350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 shutdown_connections 2024-04-17T10:32:18.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.350+0000 7f5e42200640 1 -- 172.21.15.67:0/3740307143 wait complete. 2024-04-17T10:32:18.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:18.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85ab200640 1 Processor -- start 2024-04-17T10:32:18.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85ab200640 1 -- start start 2024-04-17T10:32:18.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85ab200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85ac15dc60 0x7f85ac099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85ab200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f85ac09a830 0x7f85ac06cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85ab200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac09a370 con 0x7f85ac099a50 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85ab200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac0d0bd0 con 0x7f85ac15dc60 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85ab200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac06bda0 con 0x7f85ac09a830 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f85ac099a50 0x7f85ac0d03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46920/0 (socket says 172.21.15.67:46920) 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a9e00640 1 -- 172.21.15.67:0/849598500 learned_addr learned my addr 172.21.15.67:0/849598500 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a1400640 1 --2- 172.21.15.67:0/849598500 >> v2:172.21.15.67:3300/0 conn(0x7f85ac15dc60 0x7f85ac099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85aa800640 1 --2- 172.21.15.67:0/849598500 >> v2:172.21.15.86:3300/0 conn(0x7f85ac09a830 0x7f85ac06cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a1400640 1 -- 172.21.15.67:0/849598500 >> v2:172.21.15.86:3300/0 conn(0x7f85ac09a830 msgr2=0x7f85ac06cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:18.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a1400640 1 --2- 172.21.15.67:0/849598500 >> v2:172.21.15.86:3300/0 conn(0x7f85ac09a830 0x7f85ac06cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a1400640 1 -- 172.21.15.67:0/849598500 >> v1:172.21.15.67:6789/0 conn(0x7f85ac099a50 legacy=0x7f85ac0d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a1400640 1 -- 172.21.15.67:0/849598500 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac06c080 con 0x7f85ac15dc60 2024-04-17T10:32:18.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85a9400640 1 -- 172.21.15.67:0/849598500 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945494875 0 0) 0x7f85ac09a370 con 0x7f85ac099a50 2024-04-17T10:32:18.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.514+0000 7f85aa800640 1 --2- 172.21.15.67:0/849598500 >> v2:172.21.15.86:3300/0 conn(0x7f85ac09a830 0x7f85ac06cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:18.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85a1400640 1 --2- 172.21.15.67:0/849598500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac15dc60 0x7f85ac099e30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0d2470 tx=0x7f859401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a84b84214c2116 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85a9400640 1 -- 172.21.15.67:0/849598500 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940ab650 con 0x7f85ac15dc60 2024-04-17T10:32:18.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85a9400640 1 -- 172.21.15.67:0/849598500 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85940b0070 con 0x7f85ac15dc60 2024-04-17T10:32:18.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85a9400640 1 -- 172.21.15.67:0/849598500 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940aba10 con 0x7f85ac15dc60 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 -- 172.21.15.67:0/849598500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac15dc60 msgr2=0x7f85ac099e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 --2- 172.21.15.67:0/849598500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac15dc60 0x7f85ac099e30 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0d2470 tx=0x7f859401c450 comp rx=0 tx=0).stop 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 -- 172.21.15.67:0/849598500 shutdown_connections 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 --2- 172.21.15.67:0/849598500 >> v2:172.21.15.86:3300/0 conn(0x7f85ac09a830 0x7f85ac06cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 --2- 172.21.15.67:0/849598500 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac15dc60 0x7f85ac099e30 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 -- 172.21.15.67:0/849598500 >> 172.21.15.67:0/849598500 conn(0x7f85ac0d1610 msgr2=0x7f85ac0d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 -- 172.21.15.67:0/849598500 shutdown_connections 2024-04-17T10:32:18.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 -- 172.21.15.67:0/849598500 wait complete. 2024-04-17T10:32:18.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 Processor -- start 2024-04-17T10:32:18.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.518+0000 7f85ab200640 1 -- start start 2024-04-17T10:32:18.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85ac099a50 0x7f85ac154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac09a830 0x7f85ac0caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85ac15dc60 0x7f85ac157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac16acb0 con 0x7f85ac09a830 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac16ab30 con 0x7f85ac15dc60 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac16ae30 con 0x7f85ac099a50 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac09a830 0x7f85ac0caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac09a830 0x7f85ac0caa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35924/0 (socket says 172.21.15.67:35924) 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 -- 172.21.15.67:0/3855345195 learned_addr learned my addr 172.21.15.67:0/3855345195 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a9e00640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85ac099a50 0x7f85ac154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85aa800640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85ac15dc60 0x7f85ac157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 -- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85ac099a50 msgr2=0x7f85ac154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85ac099a50 0x7f85ac154940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 -- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85ac15dc60 msgr2=0x7f85ac157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85ac15dc60 0x7f85ac157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 -- 172.21.15.67:0/3855345195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0cb4f0 con 0x7f85ac09a830 2024-04-17T10:32:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a1400640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac09a830 0x7f85ac0caa00 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f85940ad040 tx=0x7f859401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a2a00640 1 -- 172.21.15.67:0/3855345195 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594006e20 con 0x7f85ac09a830 2024-04-17T10:32:18.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a2a00640 1 -- 172.21.15.67:0/3855345195 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8594004230 con 0x7f85ac09a830 2024-04-17T10:32:18.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac0c9190 con 0x7f85ac09a830 2024-04-17T10:32:18.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85a2a00640 1 -- 172.21.15.67:0/3855345195 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8594003620 con 0x7f85ac09a830 2024-04-17T10:32:18.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac0c9750 con 0x7f85ac09a830 2024-04-17T10:32:18.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.522+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac074dc0 con 0x7f85ac09a830 2024-04-17T10:32:18.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.526+0000 7f85a2a00640 1 -- 172.21.15.67:0/3855345195 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f85940037c0 con 0x7f85ac09a830 2024-04-17T10:32:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.526+0000 7f85a2a00640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8588066f60 0x7f8588069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.526+0000 7f85a2a00640 1 -- 172.21.15.67:0/3855345195 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f859411feb0 con 0x7f85ac09a830 2024-04-17T10:32:18.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.526+0000 7f85a9e00640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8588066f60 0x7f8588069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.526+0000 7f85a9e00640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8588066f60 0x7f8588069420 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f859c000f70 tx=0x7f859c007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.530+0000 7f85a2a00640 1 -- 172.21.15.67:0/3855345195 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8594016070 con 0x7f85ac09a830 2024-04-17T10:32:18.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.694+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85ac0d0590 con 0x7f85ac09a830 2024-04-17T10:32:18.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.694+0000 7f85a2a00640 1 -- 172.21.15.67:0/3855345195 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f85940f06d0 con 0x7f85ac09a830 2024-04-17T10:32:18.686 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:18.686 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:39.352312+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:31:36.162943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-17T10:31:36.740175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:18.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8588066f60 msgr2=0x7f8588069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8588066f60 0x7f8588069420 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f859c000f70 tx=0x7f859c007df0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac09a830 msgr2=0x7f85ac0caa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac09a830 0x7f85ac0caa00 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f85940ad040 tx=0x7f859401ac90 comp rx=0 tx=0).stop 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 shutdown_connections 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8588066f60 0x7f8588069420 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f85ac15dc60 0x7f85ac157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85ac09a830 0x7f85ac0caa00 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 --2- 172.21.15.67:0/3855345195 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f85ac099a50 0x7f85ac154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 >> 172.21.15.67:0/3855345195 conn(0x7f85ac0d1610 msgr2=0x7f85ac15a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 shutdown_connections 2024-04-17T10:32:18.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.698+0000 7f85ab200640 1 -- 172.21.15.67:0/3855345195 wait complete. 2024-04-17T10:32:18.707 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:31:36.162943+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:31:36.740175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:32:18.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:32:18.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.858+0000 7f974cc00640 1 Processor -- start 2024-04-17T10:32:18.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.858+0000 7f974cc00640 1 -- start start 2024-04-17T10:32:18.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f974cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f97480cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f974cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97480cce60 0x7f974806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f974cc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480b4990 con 0x7f974806e7b0 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f974cc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97480cd2c0 con 0x7f97480cce60 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f974cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97480af340 con 0x7f97480aef60 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97480cce60 0x7f974806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f97480cce60 0x7f974806e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35936/0 (socket says 172.21.15.67:35936) 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745400640 1 -- 172.21.15.67:0/2219209707 learned_addr learned my addr 172.21.15.67:0/2219209707 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745e00640 1 --2- 172.21.15.67:0/2219209707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f97480cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745400640 1 -- 172.21.15.67:0/2219209707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 msgr2=0x7f97480cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745400640 1 --2- 172.21.15.67:0/2219209707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f97480cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745400640 1 -- 172.21.15.67:0/2219209707 >> v1:172.21.15.86:6789/0 conn(0x7f974806e7b0 legacy=0x7f97480713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.862+0000 7f9745400640 1 -- 172.21.15.67:0/2219209707 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974815ae90 con 0x7f97480cce60 2024-04-17T10:32:18.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f9745400640 1 --2- 172.21.15.67:0/2219209707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f974806e270 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f9734009870 tx=0x7f973401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecad3be08fbaf54d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f9744a00640 1 -- 172.21.15.67:0/2219209707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340b0070 con 0x7f97480cce60 2024-04-17T10:32:18.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f9744a00640 1 -- 172.21.15.67:0/2219209707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97340ab600 con 0x7f97480cce60 2024-04-17T10:32:18.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f9744a00640 1 -- 172.21.15.67:0/2219209707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97340ab9d0 con 0x7f97480cce60 2024-04-17T10:32:18.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 -- 172.21.15.67:0/2219209707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 msgr2=0x7f974806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 --2- 172.21.15.67:0/2219209707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f974806e270 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f9734009870 tx=0x7f973401c4d0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 -- 172.21.15.67:0/2219209707 shutdown_connections 2024-04-17T10:32:18.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 --2- 172.21.15.67:0/2219209707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f974806e270 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 --2- 172.21.15.67:0/2219209707 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f97480cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 -- 172.21.15.67:0/2219209707 >> 172.21.15.67:0/2219209707 conn(0x7f97480b1740 msgr2=0x7f97480b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:18.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 -- 172.21.15.67:0/2219209707 shutdown_connections 2024-04-17T10:32:18.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 -- 172.21.15.67:0/2219209707 wait complete. 2024-04-17T10:32:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 Processor -- start 2024-04-17T10:32:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.866+0000 7f974cc00640 1 -- start start 2024-04-17T10:32:18.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974806e7b0 0x7f9748156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f974814cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f9748150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974815bc10 con 0x7f97480cce60 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974815b410 con 0x7f974806e7b0 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97481508d0 con 0x7f97480aef60 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f9748150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f9748150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35952/0 (socket says 172.21.15.67:35952) 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 -- 172.21.15.67:0/2453389168 learned_addr learned my addr 172.21.15.67:0/2453389168 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9745e00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974806e7b0 0x7f9748156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9745400640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f974814cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 -- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 msgr2=0x7f974814cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f974814cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 -- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974806e7b0 msgr2=0x7f9748156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:18.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974806e7b0 0x7f9748156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:18.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 -- 172.21.15.67:0/2453389168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9748150bb0 con 0x7f97480cce60 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f9746800640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f9748150080 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f974000c730 tx=0x7f974000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f973e000640 1 -- 172.21.15.67:0/2453389168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740003b80 con 0x7f97480cce60 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f973e000640 1 -- 172.21.15.67:0/2453389168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9740003d20 con 0x7f97480cce60 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f973e000640 1 -- 172.21.15.67:0/2453389168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9740005310 con 0x7f97480cce60 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9748150e90 con 0x7f97480cce60 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9748151450 con 0x7f97480cce60 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f974804ef30 con 0x7f97480cce60 2024-04-17T10:32:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.870+0000 7f973e000640 1 -- 172.21.15.67:0/2453389168 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f97400056c0 con 0x7f97480cce60 2024-04-17T10:32:18.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.874+0000 7f973e000640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9728066f60 0x7f9728069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:18.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.874+0000 7f973e000640 1 -- 172.21.15.67:0/2453389168 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9740085090 con 0x7f97480cce60 2024-04-17T10:32:18.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.874+0000 7f9745e00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9728066f60 0x7f9728069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.874+0000 7f9745e00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9728066f60 0x7f9728069420 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f974815e2b0 tx=0x7f9738008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:18.878+0000 7f973e000640 1 -- 172.21.15.67:0/2453389168 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f97480b4990 con 0x7f97480cce60 2024-04-17T10:32:19.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.038+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f974806d370 con 0x7f97480cce60 2024-04-17T10:32:19.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.038+0000 7f973e000640 1 -- 172.21.15.67:0/2453389168 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v96) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9740055930 con 0x7f97480cce60 2024-04-17T10:32:19.032 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:32:19.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9728066f60 msgr2=0x7f9728069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9728066f60 0x7f9728069420 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f974815e2b0 tx=0x7f9738008040 comp rx=0 tx=0).stop 2024-04-17T10:32:19.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 msgr2=0x7f9748150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f9748150080 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f974000c730 tx=0x7f974000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 shutdown_connections 2024-04-17T10:32:19.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9728066f60 0x7f9728069420 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97480cce60 0x7f9748150080 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97480aef60 0x7f974814cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 --2- 172.21.15.67:0/2453389168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f974806e7b0 0x7f9748156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 >> 172.21.15.67:0/2453389168 conn(0x7f97480b1740 msgr2=0x7f97480af970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:19.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 shutdown_connections 2024-04-17T10:32:19.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.042+0000 7f974cc00640 1 -- 172.21.15.67:0/2453389168 wait complete. 2024-04-17T10:32:19.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:32:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd48400640 1 Processor -- start 2024-04-17T10:32:19.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd48400640 1 -- start start 2024-04-17T10:32:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd48400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd4006e980 0x7fbd401653b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd48400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd400b4990 con 0x7fbd400aef60 2024-04-17T10:32:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd48400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd4015ae90 con 0x7fbd400cd030 2024-04-17T10:32:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd48400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd400af340 con 0x7fbd4006e980 2024-04-17T10:32:19.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd46400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbd4006e980 0x7fbd401653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd45a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbd400aef60 0x7fbd400cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46924/0 (socket says 172.21.15.67:46924) 2024-04-17T10:32:19.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd45a00640 1 -- 172.21.15.67:0/846437439 learned_addr learned my addr 172.21.15.67:0/846437439 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd46400640 1 -- 172.21.15.67:0/846437439 >> v1:172.21.15.86:6789/0 conn(0x7fbd400cd030 legacy=0x7fbd4006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd46400640 1 -- 172.21.15.67:0/846437439 >> v1:172.21.15.67:6789/0 conn(0x7fbd400aef60 legacy=0x7fbd400cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.206+0000 7fbd46400640 1 -- 172.21.15.67:0/846437439 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4006ee80 con 0x7fbd4006e980 2024-04-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd46400640 1 --2- 172.21.15.67:0/846437439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4006e980 0x7fbd401653b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009d90 tx=0x7fbd3801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afb8196a0042864 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd3fe00640 1 -- 172.21.15.67:0/846437439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380b2070 con 0x7fbd4006e980 2024-04-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd3fe00640 1 -- 172.21.15.67:0/846437439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd380a57a0 con 0x7fbd4006e980 2024-04-17T10:32:19.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd3fe00640 1 -- 172.21.15.67:0/846437439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380a5b70 con 0x7fbd4006e980 2024-04-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 -- 172.21.15.67:0/846437439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4006e980 msgr2=0x7fbd401653b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 --2- 172.21.15.67:0/846437439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4006e980 0x7fbd401653b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009d90 tx=0x7fbd3801c8e0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 -- 172.21.15.67:0/846437439 shutdown_connections 2024-04-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 --2- 172.21.15.67:0/846437439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd4006e980 0x7fbd401653b0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 -- 172.21.15.67:0/846437439 >> 172.21.15.67:0/846437439 conn(0x7fbd400b1740 msgr2=0x7fbd400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 -- 172.21.15.67:0/846437439 shutdown_connections 2024-04-17T10:32:19.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 -- 172.21.15.67:0/846437439 wait complete. 2024-04-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 Processor -- start 2024-04-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.210+0000 7fbd48400640 1 -- start start 2024-04-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd4006e980 0x7fbd40152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd400aef60 0x7fbd401471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd400cd030 0x7fbd400adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd401686a0 con 0x7fbd400cd030 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40168520 con 0x7fbd400aef60 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd40168820 con 0x7fbd4006e980 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd400cd030 0x7fbd400adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd400cd030 0x7fbd400adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35980/0 (socket says 172.21.15.67:35980) 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 -- 172.21.15.67:0/2652269594 learned_addr learned my addr 172.21.15.67:0/2652269594 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd45000640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd400aef60 0x7fbd401471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd45a00640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd4006e980 0x7fbd40152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 -- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd4006e980 msgr2=0x7fbd40152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd4006e980 0x7fbd40152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 -- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd400aef60 msgr2=0x7fbd401471a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd400aef60 0x7fbd401471a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 -- 172.21.15.67:0/2652269594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd40152d90 con 0x7fbd400cd030 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd46400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd400cd030 0x7fbd400adfb0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbd3801a510 tx=0x7fbd38019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd3d600640 1 -- 172.21.15.67:0/2652269594 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380b2050 con 0x7fbd400cd030 2024-04-17T10:32:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd3d600640 1 -- 172.21.15.67:0/2652269594 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd380be070 con 0x7fbd400cd030 2024-04-17T10:32:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd3d600640 1 -- 172.21.15.67:0/2652269594 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd380ada50 con 0x7fbd400cd030 2024-04-17T10:32:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd40153070 con 0x7fbd400cd030 2024-04-17T10:32:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd40153630 con 0x7fbd400cd030 2024-04-17T10:32:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.214+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4004ef30 con 0x7fbd400cd030 2024-04-17T10:32:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.218+0000 7fbd3d600640 1 -- 172.21.15.67:0/2652269594 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbd380a5620 con 0x7fbd400cd030 2024-04-17T10:32:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.218+0000 7fbd3d600640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd20066f60 0x7fbd20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.218+0000 7fbd3d600640 1 -- 172.21.15.67:0/2652269594 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbd38120320 con 0x7fbd400cd030 2024-04-17T10:32:19.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.218+0000 7fbd45a00640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd20066f60 0x7fbd20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.218+0000 7fbd45a00640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd20066f60 0x7fbd20069420 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbd34005890 tx=0x7fbd34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.222+0000 7fbd3d600640 1 -- 172.21.15.67:0/2652269594 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbd38016560 con 0x7fbd400cd030 2024-04-17T10:32:19.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.386+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbd400721d0 con 0x7fbd400cd030 2024-04-17T10:32:19.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.386+0000 7fbd3d600640 1 -- 172.21.15.67:0/2652269594 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v96) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbd380f0bd0 con 0x7fbd400cd030 2024-04-17T10:32:19.379 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:32:19.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd20066f60 msgr2=0x7fbd20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd20066f60 0x7fbd20069420 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fbd34005890 tx=0x7fbd34009040 comp rx=0 tx=0).stop 2024-04-17T10:32:19.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd400cd030 msgr2=0x7fbd400adfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd400cd030 0x7fbd400adfb0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fbd3801a510 tx=0x7fbd38019680 comp rx=0 tx=0).stop 2024-04-17T10:32:19.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 shutdown_connections 2024-04-17T10:32:19.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd20066f60 0x7fbd20069420 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd400cd030 0x7fbd400adfb0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd400aef60 0x7fbd401471a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 --2- 172.21.15.67:0/2652269594 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd4006e980 0x7fbd40152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 >> 172.21.15.67:0/2652269594 conn(0x7fbd400b1740 msgr2=0x7fbd4014c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 shutdown_connections 2024-04-17T10:32:19.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.390+0000 7fbd48400640 1 -- 172.21.15.67:0/2652269594 wait complete. 2024-04-17T10:32:19.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:32:19.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.550+0000 7f1447400640 1 Processor -- start 2024-04-17T10:32:19.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.550+0000 7f1447400640 1 -- start start 2024-04-17T10:32:19.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1447400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1440071af0 0x7f14400ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1447400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f144015c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1447400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f144015cb30 con 0x7f1440159440 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1447400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14400a8650 con 0x7f1440071af0 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1447400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144015ce10 con 0x7f144015be70 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1440071af0 0x7f14400ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1440071af0 0x7f14400ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35986/0 (socket says 172.21.15.67:35986) 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 -- 172.21.15.67:0/1945667537 learned_addr learned my addr 172.21.15.67:0/1945667537 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1445400640 1 --2- 172.21.15.67:0/1945667537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f144015c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 -- 172.21.15.67:0/1945667537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 msgr2=0x7f144015c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 --2- 172.21.15.67:0/1945667537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f144015c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 -- 172.21.15.67:0/1945667537 >> v1:172.21.15.86:6789/0 conn(0x7f1440159440 legacy=0x7f14400713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 -- 172.21.15.67:0/1945667537 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14400cd610 con 0x7f1440071af0 2024-04-17T10:32:19.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f1437e00640 1 --2- 172.21.15.67:0/1945667537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f14400ce0f0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1428009870 tx=0x7f142801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6f0bbcdf1b36bfa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f143fe00640 1 -- 172.21.15.67:0/1945667537 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14280a9620 con 0x7f1440071af0 2024-04-17T10:32:19.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f143fe00640 1 -- 172.21.15.67:0/1945667537 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14280a97c0 con 0x7f1440071af0 2024-04-17T10:32:19.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.554+0000 7f143fe00640 1 -- 172.21.15.67:0/1945667537 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14280a9b90 con 0x7f1440071af0 2024-04-17T10:32:19.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- 172.21.15.67:0/1945667537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 msgr2=0x7f14400ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 --2- 172.21.15.67:0/1945667537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f14400ce0f0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1428009870 tx=0x7f142801c560 comp rx=0 tx=0).stop 2024-04-17T10:32:19.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- 172.21.15.67:0/1945667537 shutdown_connections 2024-04-17T10:32:19.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 --2- 172.21.15.67:0/1945667537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f144015c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 --2- 172.21.15.67:0/1945667537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f14400ce0f0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- 172.21.15.67:0/1945667537 >> 172.21.15.67:0/1945667537 conn(0x7f14400b1740 msgr2=0x7f14400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:19.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- 172.21.15.67:0/1945667537 shutdown_connections 2024-04-17T10:32:19.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- 172.21.15.67:0/1945667537 wait complete. 2024-04-17T10:32:19.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 Processor -- start 2024-04-17T10:32:19.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- start start 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f144015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1440159440 0x7f144014a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f1440159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144015d840 con 0x7f1440071af0 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144015c800 con 0x7f1440159440 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1447400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144015d9c0 con 0x7f144015be70 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f144015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f144015b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35988/0 (socket says 172.21.15.67:35988) 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 -- 172.21.15.67:0/3534048994 learned_addr learned my addr 172.21.15.67:0/3534048994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1445400640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f1440159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1437e00640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1440159440 0x7f144014a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 -- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 msgr2=0x7f1440159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f1440159ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 -- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1440159440 msgr2=0x7f144014a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1440159440 0x7f144014a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.558+0000 7f1444a00640 1 -- 172.21.15.67:0/3534048994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144015a430 con 0x7f1440071af0 2024-04-17T10:32:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f1444a00640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f144015b890 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f143000c420 tx=0x7f143000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f143d600640 1 -- 172.21.15.67:0/3534048994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14300052d0 con 0x7f1440071af0 2024-04-17T10:32:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f143d600640 1 -- 172.21.15.67:0/3534048994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1430005470 con 0x7f1440071af0 2024-04-17T10:32:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144015aad0 con 0x7f1440071af0 2024-04-17T10:32:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f143d600640 1 -- 172.21.15.67:0/3534048994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430007260 con 0x7f1440071af0 2024-04-17T10:32:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144015b090 con 0x7f1440071af0 2024-04-17T10:32:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144004ef30 con 0x7f1440071af0 2024-04-17T10:32:19.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f143d600640 1 -- 172.21.15.67:0/3534048994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1430007400 con 0x7f1440071af0 2024-04-17T10:32:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f143d600640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f141c066f60 0x7f141c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f143d600640 1 -- 172.21.15.67:0/3534048994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1430084650 con 0x7f1440071af0 2024-04-17T10:32:19.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.562+0000 7f1437e00640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f141c066f60 0x7f141c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.566+0000 7f1437e00640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f141c066f60 0x7f141c069420 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f14280ae000 tx=0x7f14280b3040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.566+0000 7f143d600640 1 -- 172.21.15.67:0/3534048994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f144015a430 con 0x7f1440071af0 2024-04-17T10:32:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f144015dda0 con 0x7f1440071af0 2024-04-17T10:32:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f143d600640 1 -- 172.21.15.67:0/3534048994 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v96) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1430054e70 con 0x7f1440071af0 2024-04-17T10:32:19.721 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:32:19.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f141c066f60 msgr2=0x7f141c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f141c066f60 0x7f141c069420 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f14280ae000 tx=0x7f14280b3040 comp rx=0 tx=0).stop 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 msgr2=0x7f144015b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f144015b890 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f143000c420 tx=0x7f143000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 shutdown_connections 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f141c066f60 0x7f141c069420 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f144015be70 0x7f1440159ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1440159440 0x7f144014a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 --2- 172.21.15.67:0/3534048994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1440071af0 0x7f144015b890 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 >> 172.21.15.67:0/3534048994 conn(0x7f14400b1740 msgr2=0x7f14400cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 shutdown_connections 2024-04-17T10:32:19.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.730+0000 7f1447400640 1 -- 172.21.15.67:0/3534048994 wait complete. 2024-04-17T10:32:19.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0093800640 1 Processor -- start 2024-04-17T10:32:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0093800640 1 -- start start 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0093800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0093800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f008c0cf0c0 0x7f008c144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0093800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008c144560 con 0x7f008c06e6e0 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0093800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c15ae40 con 0x7f008c0cf0c0 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0093800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c071440 con 0x7f008c06eac0 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0091800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f008c0cf0c0 0x7f008c144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f008be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0090e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f008c06e6e0 0x7f008c0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:60222/0 (socket says 172.21.15.67:60222) 2024-04-17T10:32:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0091800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f008c0cf0c0 0x7f008c144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:36530/0 (socket says 172.21.15.67:36530) 2024-04-17T10:32:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0090e00640 1 -- 172.21.15.67:0/3803196087 learned_addr learned my addr 172.21.15.67:0/3803196087 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0091800640 1 -- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 msgr2=0x7f008c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0091800640 1 --2- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0091800640 1 -- 172.21.15.67:0/3803196087 >> v1:172.21.15.86:6789/0 conn(0x7f008c06e6e0 legacy=0x7f008c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f0091800640 1 -- 172.21.15.67:0/3803196087 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c071720 con 0x7f008c0cf0c0 2024-04-17T10:32:19.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.894+0000 7f008be00640 1 --2- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:19.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0091800640 1 --2- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c0cf0c0 0x7f008c144020 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0084009d90 tx=0x7f008401c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1759d6f7b935a8b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f008b400640 1 -- 172.21.15.67:0/3803196087 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00840b2070 con 0x7f008c0cf0c0 2024-04-17T10:32:19.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f008b400640 1 -- 172.21.15.67:0/3803196087 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00840a5970 con 0x7f008c0cf0c0 2024-04-17T10:32:19.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f008b400640 1 -- 172.21.15.67:0/3803196087 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00840a5d40 con 0x7f008c0cf0c0 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 -- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c0cf0c0 msgr2=0x7f008c144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 --2- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c0cf0c0 0x7f008c144020 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0084009d90 tx=0x7f008401c8b0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 -- 172.21.15.67:0/3803196087 shutdown_connections 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 --2- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c0cf0c0 0x7f008c144020 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 --2- 172.21.15.67:0/3803196087 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 -- 172.21.15.67:0/3803196087 >> 172.21.15.67:0/3803196087 conn(0x7f008c06ac90 msgr2=0x7f008c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 -- 172.21.15.67:0/3803196087 shutdown_connections 2024-04-17T10:32:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.898+0000 7f0093800640 1 -- 172.21.15.67:0/3803196087 wait complete. 2024-04-17T10:32:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 Processor -- start 2024-04-17T10:32:19.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 -- start start 2024-04-17T10:32:19.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c06e6e0 0x7f008c13a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c0c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f008c0cf0c0 0x7f008c0c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0b3a40 con 0x7f008c0cf0c0 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0b38c0 con 0x7f008c06e6e0 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0b3bc0 con 0x7f008c06eac0 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f008c0cf0c0 0x7f008c0c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f008c0cf0c0 0x7f008c0c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36002/0 (socket says 172.21.15.67:36002) 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 -- 172.21.15.67:0/20315797 learned_addr learned my addr 172.21.15.67:0/20315797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f008be00640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c0c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0090e00640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c06e6e0 0x7f008c13a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 -- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 msgr2=0x7f008c0c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c0c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 -- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c06e6e0 msgr2=0x7f008c13a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c06e6e0 0x7f008c13a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 -- 172.21.15.67:0/20315797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c13f4a0 con 0x7f008c0cf0c0 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0091800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f008c0cf0c0 0x7f008c0c7430 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f008401af60 tx=0x7f0084019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0088c00640 1 -- 172.21.15.67:0/20315797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00840b2050 con 0x7f008c0cf0c0 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0088c00640 1 -- 172.21.15.67:0/20315797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00840be070 con 0x7f008c0cf0c0 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c0c4970 con 0x7f008c0cf0c0 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0088c00640 1 -- 172.21.15.67:0/20315797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00840ada50 con 0x7f008c0cf0c0 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c0c4f30 con 0x7f008c0cf0c0 2024-04-17T10:32:19.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.902+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c072d50 con 0x7f008c0cf0c0 2024-04-17T10:32:19.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.906+0000 7f0088c00640 1 -- 172.21.15.67:0/20315797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f00840a57f0 con 0x7f008c0cf0c0 2024-04-17T10:32:19.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.906+0000 7f0088c00640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f006c066f60 0x7f006c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:19.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.906+0000 7f0088c00640 1 -- 172.21.15.67:0/20315797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0084120350 con 0x7f008c0cf0c0 2024-04-17T10:32:19.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.906+0000 7f0090e00640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f006c066f60 0x7f006c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:19.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.906+0000 7f0090e00640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f006c066f60 0x7f006c069420 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0080000f70 tx=0x7f0080007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:19.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:19.910+0000 7f0088c00640 1 -- 172.21.15.67:0/20315797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0084016560 con 0x7f008c0cf0c0 2024-04-17T10:32:20.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f008c074e30 con 0x7f008c0cf0c0 2024-04-17T10:32:20.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0088c00640 1 -- 172.21.15.67:0/20315797 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f00840f0ae0 con 0x7f008c0cf0c0 2024-04-17T10:32:20.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:20.098 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:31:37.749417+0000","modified":"2024-04-17T10:31:40.768850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7698":{"gid":7698,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3523329517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3523329517},{"type":"v1","addr":"172.21.15.67:6835","nonce":3523329517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-17T10:32:20.098 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 64 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f006c066f60 msgr2=0x7f006c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f006c066f60 0x7f006c069420 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0080000f70 tx=0x7f0080007e90 comp rx=0 tx=0).stop 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f008c0cf0c0 msgr2=0x7f008c0c7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f008c0cf0c0 0x7f008c0c7430 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f008401af60 tx=0x7f0084019650 comp rx=0 tx=0).stop 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 shutdown_connections 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f006c066f60 0x7f006c069420 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f008c0cf0c0 0x7f008c0c7430 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f008c06eac0 0x7f008c0c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 --2- 172.21.15.67:0/20315797 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f008c06e6e0 0x7f008c13a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.106+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 >> 172.21.15.67:0/20315797 conn(0x7f008c06ac90 msgr2=0x7f008c13fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.110+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 shutdown_connections 2024-04-17T10:32:20.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.110+0000 7f0093800640 1 -- 172.21.15.67:0/20315797 wait complete. 2024-04-17T10:32:20.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.266+0000 7fcbf2200640 1 Processor -- start 2024-04-17T10:32:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.266+0000 7fcbf2200640 1 -- start start 2024-04-17T10:32:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbf2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbec071840 0x7fcbec166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbf2200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec0b6640 con 0x7fcbec1446c0 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbf2200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbec1456d0 con 0x7fcbec144aa0 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbf2200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbec0b51a0 con 0x7fcbec071840 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbebe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcbec071840 0x7fcbec166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbeb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcbec1446c0 0x7fcbec144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46926/0 (socket says 172.21.15.67:46926) 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbeb400640 1 -- 172.21.15.67:0/876705730 learned_addr learned my addr 172.21.15.67:0/876705730 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbebe00640 1 -- 172.21.15.67:0/876705730 >> v1:172.21.15.86:6789/0 conn(0x7fcbec144aa0 legacy=0x7fcbec0d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbebe00640 1 -- 172.21.15.67:0/876705730 >> v1:172.21.15.67:6789/0 conn(0x7fcbec1446c0 legacy=0x7fcbec144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.270+0000 7fcbebe00640 1 -- 172.21.15.67:0/876705730 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec0b4870 con 0x7fcbec071840 2024-04-17T10:32:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbebe00640 1 --2- 172.21.15.67:0/876705730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec166a30 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009d90 tx=0x7fcbe401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5db5f6a73770ee server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbeaa00640 1 -- 172.21.15.67:0/876705730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40b2070 con 0x7fcbec071840 2024-04-17T10:32:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbeaa00640 1 -- 172.21.15.67:0/876705730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe40a5910 con 0x7fcbec071840 2024-04-17T10:32:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbeaa00640 1 -- 172.21.15.67:0/876705730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40a5ce0 con 0x7fcbec071840 2024-04-17T10:32:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 -- 172.21.15.67:0/876705730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 msgr2=0x7fcbec166a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 --2- 172.21.15.67:0/876705730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec166a30 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009d90 tx=0x7fcbe401c940 comp rx=0 tx=0).stop 2024-04-17T10:32:20.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 -- 172.21.15.67:0/876705730 shutdown_connections 2024-04-17T10:32:20.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 --2- 172.21.15.67:0/876705730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec166a30 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 -- 172.21.15.67:0/876705730 >> 172.21.15.67:0/876705730 conn(0x7fcbec0b34d0 msgr2=0x7fcbec0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:20.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 -- 172.21.15.67:0/876705730 shutdown_connections 2024-04-17T10:32:20.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 -- 172.21.15.67:0/876705730 wait complete. 2024-04-17T10:32:20.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 Processor -- start 2024-04-17T10:32:20.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 -- start start 2024-04-17T10:32:20.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec13f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.274+0000 7fcbf2200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec1446c0 0x7fcbec14fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbf2200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbec144aa0 0x7fcbec155580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbf2200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec169910 con 0x7fcbec071840 2024-04-17T10:32:20.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbf2200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec169790 con 0x7fcbec1446c0 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbf2200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec169a90 con 0x7fcbec144aa0 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec13f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec13f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36014/0 (socket says 172.21.15.67:36014) 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 -- 172.21.15.67:0/1737003315 learned_addr learned my addr 172.21.15.67:0/1737003315 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbebe00640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbec144aa0 0x7fcbec155580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec1446c0 0x7fcbec14fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 -- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbec144aa0 msgr2=0x7fcbec155580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbec144aa0 0x7fcbec155580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 -- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec1446c0 msgr2=0x7fcbec14fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec1446c0 0x7fcbec14fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 -- 172.21.15.67:0/1737003315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec140fd0 con 0x7fcbec071840 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbeb400640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec13f450 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00c4d0 tx=0x7fcbdc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbe3400640 1 -- 172.21.15.67:0/1737003315 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc017070 con 0x7fcbec071840 2024-04-17T10:32:20.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbe3400640 1 -- 172.21.15.67:0/1737003315 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc005cf0 con 0x7fcbec071840 2024-04-17T10:32:20.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbe3400640 1 -- 172.21.15.67:0/1737003315 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc0072b0 con 0x7fcbec071840 2024-04-17T10:32:20.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec1514d0 con 0x7fcbec071840 2024-04-17T10:32:20.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec14f630 con 0x7fcbec071840 2024-04-17T10:32:20.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec1519d0 con 0x7fcbec071840 2024-04-17T10:32:20.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.278+0000 7fcbe3400640 1 -- 172.21.15.67:0/1737003315 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcbdc01a070 con 0x7fcbec071840 2024-04-17T10:32:20.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.282+0000 7fcbe3400640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbc8066f60 0x7fcbc8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.282+0000 7fcbe3400640 1 -- 172.21.15.67:0/1737003315 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcbdc0844b0 con 0x7fcbec071840 2024-04-17T10:32:20.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.282+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbc8066f60 0x7fcbc8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.282+0000 7fcbe3e00640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbc8066f60 0x7fcbc8069420 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcbec158ab0 tx=0x7fcbd4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.286+0000 7fcbe3400640 1 -- 172.21.15.67:0/1737003315 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcbdc055320 con 0x7fcbec071840 2024-04-17T10:32:20.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.446+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbec070bb0 con 0x7fcbec071840 2024-04-17T10:32:20.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.446+0000 7fcbe3400640 1 -- 172.21.15.67:0/1737003315 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fcbdc054c20 con 0x7fcbec071840 2024-04-17T10:32:20.440 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:20.440 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:39.352312+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:31:36.162943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-17T10:31:36.740175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbc8066f60 msgr2=0x7fcbc8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbc8066f60 0x7fcbc8069420 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcbec158ab0 tx=0x7fcbd4009040 comp rx=0 tx=0).stop 2024-04-17T10:32:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 msgr2=0x7fcbec13f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec13f450 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00c4d0 tx=0x7fcbdc00c990 comp rx=0 tx=0).stop 2024-04-17T10:32:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 shutdown_connections 2024-04-17T10:32:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcbc8066f60 0x7fcbc8069420 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcbec144aa0 0x7fcbec155580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcbec1446c0 0x7fcbec14fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 --2- 172.21.15.67:0/1737003315 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcbec071840 0x7fcbec13f450 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 >> 172.21.15.67:0/1737003315 conn(0x7fcbec0b34d0 msgr2=0x7fcbec154450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 shutdown_connections 2024-04-17T10:32:20.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.450+0000 7fcbf2200640 1 -- 172.21.15.67:0/1737003315 wait complete. 2024-04-17T10:32:20.461 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:32:20.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.610+0000 7f97caa00640 1 Processor -- start 2024-04-17T10:32:20.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.610+0000 7f97caa00640 1 -- start start 2024-04-17T10:32:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97caa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c41667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97caa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c415be60 con 0x7f97c415cc00 2024-04-17T10:32:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97caa00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c4160230 con 0x7f97c40ce1b0 2024-04-17T10:32:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97caa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c415c120 con 0x7f97c415fbd0 2024-04-17T10:32:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97bbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f97c415cc00 0x7f97c415f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41196/0 (socket says 172.21.15.67:41196) 2024-04-17T10:32:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97bbe00640 1 -- 172.21.15.67:0/1946642038 learned_addr learned my addr 172.21.15.67:0/1946642038 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:20.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97c8a00640 1 --2- 172.21.15.67:0/1946642038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c41667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97c3400640 1 -- 172.21.15.67:0/1946642038 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382957296 0 0) 0x7f97c415be60 con 0x7f97c415cc00 2024-04-17T10:32:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97c3400640 1 -- 172.21.15.67:0/1946642038 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a8003530 con 0x7f97c415cc00 2024-04-17T10:32:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97c8a00640 1 -- 172.21.15.67:0/1946642038 >> v1:172.21.15.86:6789/0 conn(0x7f97c40ce1b0 legacy=0x7f97c40ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97c8a00640 1 -- 172.21.15.67:0/1946642038 >> v1:172.21.15.67:6789/0 conn(0x7f97c415cc00 legacy=0x7f97c415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97c3400640 1 -- 172.21.15.67:0/1946642038 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579490182 0 0) 0x7f97c4160230 con 0x7f97c40ce1b0 2024-04-17T10:32:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.614+0000 7f97c8a00640 1 -- 172.21.15.67:0/1946642038 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c415c400 con 0x7f97c415fbd0 2024-04-17T10:32:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97c8a00640 1 --2- 172.21.15.67:0/1946642038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c41667e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009dc0 tx=0x7f97bc01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59a2550e7a74643b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97c3400640 1 -- 172.21.15.67:0/1946642038 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0b4070 con 0x7f97c415fbd0 2024-04-17T10:32:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97c3400640 1 -- 172.21.15.67:0/1946642038 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97bc0af600 con 0x7f97c415fbd0 2024-04-17T10:32:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97c3400640 1 -- 172.21.15.67:0/1946642038 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97bc0af9b0 con 0x7f97c415fbd0 2024-04-17T10:32:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 -- 172.21.15.67:0/1946642038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 msgr2=0x7f97c41667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 --2- 172.21.15.67:0/1946642038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c41667e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f97bc009dc0 tx=0x7f97bc01f170 comp rx=0 tx=0).stop 2024-04-17T10:32:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 -- 172.21.15.67:0/1946642038 shutdown_connections 2024-04-17T10:32:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 --2- 172.21.15.67:0/1946642038 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c41667e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 -- 172.21.15.67:0/1946642038 >> 172.21.15.67:0/1946642038 conn(0x7f97c40b33f0 msgr2=0x7f97c40b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:20.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 -- 172.21.15.67:0/1946642038 shutdown_connections 2024-04-17T10:32:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 -- 172.21.15.67:0/1946642038 wait complete. 2024-04-17T10:32:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 Processor -- start 2024-04-17T10:32:20.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.618+0000 7f97caa00640 1 -- start start 2024-04-17T10:32:20.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40ce1b0 0x7f97c4159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c415cc00 0x7f97c406d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c41598b0 con 0x7f97c415cc00 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4159730 con 0x7f97c40ce1b0 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97c4166d20 con 0x7f97c415fbd0 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c415cc00 0x7f97c406d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c415cc00 0x7f97c406d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33090/0 (socket says 172.21.15.67:33090) 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 -- 172.21.15.67:0/4198792946 learned_addr learned my addr 172.21.15.67:0/4198792946 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97c8a00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97c3e00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40ce1b0 0x7f97c4159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 -- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 msgr2=0x7f97c40cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c40cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 -- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40ce1b0 msgr2=0x7f97c4159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40ce1b0 0x7f97c4159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 -- 172.21.15.67:0/4198792946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c415a160 con 0x7f97c415cc00 2024-04-17T10:32:20.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97bbe00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c415cc00 0x7f97c406d920 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00c480 tx=0x7f97ac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97c0c00640 1 -- 172.21.15.67:0/4198792946 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac017070 con 0x7f97c415cc00 2024-04-17T10:32:20.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97c0c00640 1 -- 172.21.15.67:0/4198792946 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac005cf0 con 0x7f97c415cc00 2024-04-17T10:32:20.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97c0c00640 1 -- 172.21.15.67:0/4198792946 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac007290 con 0x7f97c415cc00 2024-04-17T10:32:20.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c41653a0 con 0x7f97c415cc00 2024-04-17T10:32:20.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c4165960 con 0x7f97c415cc00 2024-04-17T10:32:20.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.622+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c40c6ab0 con 0x7f97c415cc00 2024-04-17T10:32:20.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.626+0000 7f97c0c00640 1 -- 172.21.15.67:0/4198792946 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f97ac01a070 con 0x7f97c415cc00 2024-04-17T10:32:20.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.626+0000 7f97c0c00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97a0066f60 0x7f97a0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.626+0000 7f97c0c00640 1 -- 172.21.15.67:0/4198792946 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f97ac084580 con 0x7f97c415cc00 2024-04-17T10:32:20.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.626+0000 7f97c3e00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97a0066f60 0x7f97a0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.626+0000 7f97c3e00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97a0066f60 0x7f97a0069420 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f97b4000f70 tx=0x7f97b4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.630+0000 7f97c0c00640 1 -- 172.21.15.67:0/4198792946 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f97ac054da0 con 0x7f97c415cc00 2024-04-17T10:32:20.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97c4070ab0 con 0x7f97c415cc00 2024-04-17T10:32:20.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97c0c00640 1 -- 172.21.15.67:0/4198792946 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f97ac058e40 con 0x7f97c415cc00 2024-04-17T10:32:20.818 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:20.818 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:31:37.749417+0000","modified":"2024-04-17T10:31:40.768850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7698":{"gid":7698,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3523329517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3523329517},{"type":"v1","addr":"172.21.15.67:6835","nonce":3523329517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-17T10:32:20.818 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 64 2024-04-17T10:32:20.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97a0066f60 msgr2=0x7f97a0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97a0066f60 0x7f97a0069420 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f97b4000f70 tx=0x7f97b4009040 comp rx=0 tx=0).stop 2024-04-17T10:32:20.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c415cc00 msgr2=0x7f97c406d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c415cc00 0x7f97c406d920 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00c480 tx=0x7f97ac00c940 comp rx=0 tx=0).stop 2024-04-17T10:32:20.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 shutdown_connections 2024-04-17T10:32:20.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f97a0066f60 0x7f97a0069420 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f97c415fbd0 0x7f97c40cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97c415cc00 0x7f97c406d920 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 --2- 172.21.15.67:0/4198792946 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97c40ce1b0 0x7f97c4159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 >> 172.21.15.67:0/4198792946 conn(0x7f97c40b33f0 msgr2=0x7f97c415e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:20.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 shutdown_connections 2024-04-17T10:32:20.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.826+0000 7f97caa00640 1 -- 172.21.15.67:0/4198792946 wait complete. 2024-04-17T10:32:20.839 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cb0200640 1 Processor -- start 2024-04-17T10:32:20.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cb0200640 1 -- start start 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cb0200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ca80714d0 0x7f5ca8168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cb0200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca80b5980 con 0x7f5ca815dd10 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cb0200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca80b0cc0 con 0x7f5ca80b07b0 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cb0200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca80b1280 con 0x7f5ca80714d0 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cae200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5ca80714d0 0x7f5ca8168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cad800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5ca815dd10 0x7f5ca80b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41210/0 (socket says 172.21.15.67:41210) 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cad800640 1 -- 172.21.15.67:0/3821139724 learned_addr learned my addr 172.21.15.67:0/3821139724 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cae200640 1 -- 172.21.15.67:0/3821139724 >> v1:172.21.15.86:6789/0 conn(0x7f5ca80b07b0 legacy=0x7f5ca8070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cae200640 1 -- 172.21.15.67:0/3821139724 >> v1:172.21.15.67:6789/0 conn(0x7f5ca815dd10 legacy=0x7f5ca80b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.990+0000 7f5cae200640 1 -- 172.21.15.67:0/3821139724 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80b0fa0 con 0x7f5ca80714d0 2024-04-17T10:32:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cae200640 1 --2- 172.21.15.67:0/3821139724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca8168c50 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0009d90 tx=0x7f5ca001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ebf94e66c935f52 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5ca7e00640 1 -- 172.21.15.67:0/3821139724 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca00b2070 con 0x7f5ca80714d0 2024-04-17T10:32:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5ca7e00640 1 -- 172.21.15.67:0/3821139724 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca00a5970 con 0x7f5ca80714d0 2024-04-17T10:32:20.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5ca7e00640 1 -- 172.21.15.67:0/3821139724 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca00a5d40 con 0x7f5ca80714d0 2024-04-17T10:32:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cb0200640 1 -- 172.21.15.67:0/3821139724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 msgr2=0x7f5ca8168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cb0200640 1 --2- 172.21.15.67:0/3821139724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca8168c50 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f5ca0009d90 tx=0x7f5ca001c8b0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cb0200640 1 -- 172.21.15.67:0/3821139724 shutdown_connections 2024-04-17T10:32:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cb0200640 1 --2- 172.21.15.67:0/3821139724 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca8168c50 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cb0200640 1 -- 172.21.15.67:0/3821139724 >> 172.21.15.67:0/3821139724 conn(0x7f5ca80b34d0 msgr2=0x7f5ca80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cb0200640 1 -- 172.21.15.67:0/3821139724 shutdown_connections 2024-04-17T10:32:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.994+0000 7f5cb0200640 1 -- 172.21.15.67:0/3821139724 wait complete. 2024-04-17T10:32:20.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 Processor -- start 2024-04-17T10:32:20.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 -- start start 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca80cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca80b07b0 0x7f5ca80d0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca815dd10 0x7f5ca81543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca816bb30 con 0x7f5ca80714d0 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca816b9b0 con 0x7f5ca80b07b0 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca816bcb0 con 0x7f5ca815dd10 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca80cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca80cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33106/0 (socket says 172.21.15.67:33106) 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 -- 172.21.15.67:0/1419356044 learned_addr learned my addr 172.21.15.67:0/1419356044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cae200640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca815dd10 0x7f5ca81543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cace00640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca80b07b0 0x7f5ca80d0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 -- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca815dd10 msgr2=0x7f5ca81543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca815dd10 0x7f5ca81543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 -- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca80b07b0 msgr2=0x7f5ca80d0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca80b07b0 0x7f5ca80d0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 -- 172.21.15.67:0/1419356044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80d05c0 con 0x7f5ca80714d0 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cace00640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca80b07b0 0x7f5ca80d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cad800640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca80cab50 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00c400 tx=0x7f5c9c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5ca5600640 1 -- 172.21.15.67:0/1419356044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c017070 con 0x7f5ca80714d0 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5ca5600640 1 -- 172.21.15.67:0/1419356044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c005cf0 con 0x7f5ca80714d0 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca80cb130 con 0x7f5ca80714d0 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5ca5600640 1 -- 172.21.15.67:0/1419356044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c9c0072d0 con 0x7f5ca80714d0 2024-04-17T10:32:20.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca80cb6f0 con 0x7f5ca80714d0 2024-04-17T10:32:20.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:20.998+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca8070bb0 con 0x7f5ca80714d0 2024-04-17T10:32:20.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.002+0000 7f5ca5600640 1 -- 172.21.15.67:0/1419356044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5c9c01a070 con 0x7f5ca80714d0 2024-04-17T10:32:20.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.002+0000 7f5ca5600640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c88066f60 0x7f5c88069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:20.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.002+0000 7f5ca5600640 1 -- 172.21.15.67:0/1419356044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5c9c0843e0 con 0x7f5ca80714d0 2024-04-17T10:32:20.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.002+0000 7f5cace00640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c88066f60 0x7f5c88069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:20.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.002+0000 7f5cace00640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c88066f60 0x7f5c88069420 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80d0030 tx=0x7f5c94009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:20.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.006+0000 7f5ca5600640 1 -- 172.21.15.67:0/1419356044 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5c9c054ca0 con 0x7f5ca80714d0 2024-04-17T10:32:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.202+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ca8074e30 con 0x7f5ca80714d0 2024-04-17T10:32:21.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.202+0000 7f5ca5600640 1 -- 172.21.15.67:0/1419356044 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f5c9c058d40 con 0x7f5ca80714d0 2024-04-17T10:32:21.195 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:21.195 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":64,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:31:37.749417+0000","modified":"2024-04-17T10:31:40.768850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7698},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7698":{"gid":7698,"name":"a","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6835/3523329517","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3523329517},{"type":"v1","addr":"172.21.15.67:6835","nonce":3523329517}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-04-17T10:32:21.196 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 64 2024-04-17T10:32:21.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c88066f60 msgr2=0x7f5c88069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c88066f60 0x7f5c88069420 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f5ca80d0030 tx=0x7f5c94009040 comp rx=0 tx=0).stop 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 msgr2=0x7f5ca80cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca80cab50 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f5c9c00c400 tx=0x7f5c9c00c8c0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 shutdown_connections 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5c88066f60 0x7f5c88069420 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5ca815dd10 0x7f5ca81543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5ca80b07b0 0x7f5ca80d0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 --2- 172.21.15.67:0/1419356044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5ca80714d0 0x7f5ca80cab50 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 >> 172.21.15.67:0/1419356044 conn(0x7f5ca80b34d0 msgr2=0x7f5ca815cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:21.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 shutdown_connections 2024-04-17T10:32:21.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.206+0000 7f5cb0200640 1 -- 172.21.15.67:0/1419356044 wait complete. 2024-04-17T10:32:21.217 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:21.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc8a5a00640 1 Processor -- start 2024-04-17T10:32:21.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc8a5a00640 1 -- start start 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc8a5a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a0144aa0 0x7fc8a00d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc8a5a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc8a0071840 0x7fc8a00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc8a5a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a01456d0 con 0x7fc8a01446c0 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc8a5a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a00b51a0 con 0x7fc8a0071840 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc8a5a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00d1390 con 0x7fc8a0144aa0 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc8a01446c0 0x7fc8a0144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41224/0 (socket says 172.21.15.67:41224) 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89ea00640 1 -- 172.21.15.67:0/2340496931 learned_addr learned my addr 172.21.15.67:0/2340496931 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:21.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89f400640 1 --2- 172.21.15.67:0/2340496931 >> v2:172.21.15.67:3300/0 conn(0x7fc8a0071840 0x7fc8a00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc896000640 1 --2- 172.21.15.67:0/2340496931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a0144aa0 0x7fc8a00d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89e000640 1 -- 172.21.15.67:0/2340496931 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788068844 0 0) 0x7fc8a01456d0 con 0x7fc8a01446c0 2024-04-17T10:32:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89e000640 1 -- 172.21.15.67:0/2340496931 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc884003530 con 0x7fc8a01446c0 2024-04-17T10:32:21.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89f400640 1 -- 172.21.15.67:0/2340496931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a0144aa0 msgr2=0x7fc8a00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89f400640 1 --2- 172.21.15.67:0/2340496931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a0144aa0 0x7fc8a00d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89f400640 1 -- 172.21.15.67:0/2340496931 >> v1:172.21.15.67:6789/0 conn(0x7fc8a01446c0 legacy=0x7fc8a0144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.366+0000 7fc89f400640 1 -- 172.21.15.67:0/2340496931 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a00b6640 con 0x7fc8a0071840 2024-04-17T10:32:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc89f400640 1 --2- 172.21.15.67:0/2340496931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0071840 0x7fc8a00d0e50 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00b5320 tx=0x7fc89801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b2c3c773de6760 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:21.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc89e000640 1 -- 172.21.15.67:0/2340496931 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8980b1070 con 0x7fc8a0071840 2024-04-17T10:32:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc89e000640 1 -- 172.21.15.67:0/2340496931 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8980ac620 con 0x7fc8a0071840 2024-04-17T10:32:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc89e000640 1 -- 172.21.15.67:0/2340496931 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8980ac9f0 con 0x7fc8a0071840 2024-04-17T10:32:21.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 -- 172.21.15.67:0/2340496931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0071840 msgr2=0x7fc8a00d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/2340496931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0071840 0x7fc8a00d0e50 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00b5320 tx=0x7fc89801c3c0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 -- 172.21.15.67:0/2340496931 shutdown_connections 2024-04-17T10:32:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/2340496931 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0071840 0x7fc8a00d0e50 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/2340496931 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a0144aa0 0x7fc8a00d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 -- 172.21.15.67:0/2340496931 >> 172.21.15.67:0/2340496931 conn(0x7fc8a00b34d0 msgr2=0x7fc8a00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 -- 172.21.15.67:0/2340496931 shutdown_connections 2024-04-17T10:32:21.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.370+0000 7fc8a5a00640 1 -- 172.21.15.67:0/2340496931 wait complete. 2024-04-17T10:32:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 Processor -- start 2024-04-17T10:32:21.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 -- start start 2024-04-17T10:32:21.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a0071840 0x7fc8a0142e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a01446c0 0x7fc8a01651d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0144aa0 0x7fc8a0165710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00b5e20 con 0x7fc8a0144aa0 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a015c0d0 con 0x7fc8a0071840 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a00b5fa0 con 0x7fc8a01446c0 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0144aa0 0x7fc8a0165710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0144aa0 0x7fc8a0165710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33138/0 (socket says 172.21.15.67:33138) 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 -- 172.21.15.67:0/190456537 learned_addr learned my addr 172.21.15.67:0/190456537 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89ea00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a0071840 0x7fc8a0142e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc896000640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a01446c0 0x7fc8a01651d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 -- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a01446c0 msgr2=0x7fc8a01651d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a01446c0 0x7fc8a01651d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 -- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a0071840 msgr2=0x7fc8a0142e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a0071840 0x7fc8a0142e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 -- 172.21.15.67:0/190456537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a015d220 con 0x7fc8a0144aa0 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89ea00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a0071840 0x7fc8a0142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc896000640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a01446c0 0x7fc8a01651d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc89f400640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0144aa0 0x7fc8a0165710 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc8980af040 tx=0x7fc898019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc897400640 1 -- 172.21.15.67:0/190456537 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8980b9070 con 0x7fc8a0144aa0 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc897400640 1 -- 172.21.15.67:0/190456537 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8980b5410 con 0x7fc8a0144aa0 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc897400640 1 -- 172.21.15.67:0/190456537 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8980b1040 con 0x7fc8a0144aa0 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a015d500 con 0x7fc8a0144aa0 2024-04-17T10:32:21.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.374+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a015dac0 con 0x7fc8a0144aa0 2024-04-17T10:32:21.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.378+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a0070bb0 con 0x7fc8a0144aa0 2024-04-17T10:32:21.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.378+0000 7fc897400640 1 -- 172.21.15.67:0/190456537 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc8980bc070 con 0x7fc8a0144aa0 2024-04-17T10:32:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.378+0000 7fc897400640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc87c066f60 0x7fc87c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.378+0000 7fc897400640 1 -- 172.21.15.67:0/190456537 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc89811efe0 con 0x7fc8a0144aa0 2024-04-17T10:32:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.378+0000 7fc89ea00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc87c066f60 0x7fc87c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.378+0000 7fc89ea00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc87c066f60 0x7fc87c069420 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc890005c10 tx=0x7fc890009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:21.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.382+0000 7fc897400640 1 -- 172.21.15.67:0/190456537 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc898016070 con 0x7fc8a0144aa0 2024-04-17T10:32:21.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.554+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8a0166770 con 0x7fc8a0144aa0 2024-04-17T10:32:21.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.554+0000 7fc897400640 1 -- 172.21.15.67:0/190456537 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fc8980efe50 con 0x7fc8a0144aa0 2024-04-17T10:32:21.547 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:21.548 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:31:39.352312+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:31:36.162943+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-04-17T10:31:36.740175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:21.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc87c066f60 msgr2=0x7fc87c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc87c066f60 0x7fc87c069420 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc890005c10 tx=0x7fc890009040 comp rx=0 tx=0).stop 2024-04-17T10:32:21.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0144aa0 msgr2=0x7fc8a0165710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0144aa0 0x7fc8a0165710 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc8980af040 tx=0x7fc898019160 comp rx=0 tx=0).stop 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 shutdown_connections 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc87c066f60 0x7fc87c069420 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc8a0144aa0 0x7fc8a0165710 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc8a01446c0 0x7fc8a01651d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 --2- 172.21.15.67:0/190456537 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc8a0071840 0x7fc8a0142e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 >> 172.21.15.67:0/190456537 conn(0x7fc8a00b34d0 msgr2=0x7fc8a0163a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 shutdown_connections 2024-04-17T10:32:21.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.558+0000 7fc8a5a00640 1 -- 172.21.15.67:0/190456537 wait complete. 2024-04-17T10:32:21.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:32:21.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.718+0000 7f2c89a00640 1 Processor -- start 2024-04-17T10:32:21.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c89a00640 1 -- start start 2024-04-17T10:32:21.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c89a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c840cc250 0x7f2c840cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c89a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c840cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c89a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2c840ce2a0 0x7f2c840cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c89a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c840cd150 con 0x7f2c840cc250 2024-04-17T10:32:21.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c89a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c8406b100 con 0x7f2c840ce2a0 2024-04-17T10:32:21.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c89a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8406b3e0 con 0x7f2c840cb260 2024-04-17T10:32:21.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c840cc250 0x7f2c840cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c840cc250 0x7f2c840cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33148/0 (socket says 172.21.15.67:33148) 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 -- 172.21.15.67:0/1921347160 learned_addr learned my addr 172.21.15.67:0/1921347160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c83400640 1 --2- 172.21.15.67:0/1921347160 >> v2:172.21.15.86:3300/0 conn(0x7f2c840ce2a0 0x7f2c840cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c7be00640 1 --2- 172.21.15.67:0/1921347160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c840cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 -- 172.21.15.67:0/1921347160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 msgr2=0x7f2c840cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 --2- 172.21.15.67:0/1921347160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c840cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 -- 172.21.15.67:0/1921347160 >> v2:172.21.15.86:3300/0 conn(0x7f2c840ce2a0 msgr2=0x7f2c840cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 --2- 172.21.15.67:0/1921347160 >> v2:172.21.15.86:3300/0 conn(0x7f2c840ce2a0 0x7f2c840cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c82a00640 1 -- 172.21.15.67:0/1921347160 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8406b6c0 con 0x7f2c840cc250 2024-04-17T10:32:21.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.722+0000 7f2c7be00640 1 --2- 172.21.15.67:0/1921347160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c840cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:21.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c82a00640 1 --2- 172.21.15.67:0/1921347160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840cc250 0x7f2c840cad20 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f2c841617e0 tx=0x7f2c7401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44463db1ae1baff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:21.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c82000640 1 -- 172.21.15.67:0/1921347160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c740b0070 con 0x7f2c840cc250 2024-04-17T10:32:21.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c82000640 1 -- 172.21.15.67:0/1921347160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c740ab600 con 0x7f2c840cc250 2024-04-17T10:32:21.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c82000640 1 -- 172.21.15.67:0/1921347160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c740ab9d0 con 0x7f2c840cc250 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 -- 172.21.15.67:0/1921347160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840cc250 msgr2=0x7f2c840cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 --2- 172.21.15.67:0/1921347160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840cc250 0x7f2c840cad20 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f2c841617e0 tx=0x7f2c7401c520 comp rx=0 tx=0).stop 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 -- 172.21.15.67:0/1921347160 shutdown_connections 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 --2- 172.21.15.67:0/1921347160 >> v2:172.21.15.86:3300/0 conn(0x7f2c840ce2a0 0x7f2c840cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 --2- 172.21.15.67:0/1921347160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c840cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 --2- 172.21.15.67:0/1921347160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840cc250 0x7f2c840cad20 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 -- 172.21.15.67:0/1921347160 >> 172.21.15.67:0/1921347160 conn(0x7f2c840b22a0 msgr2=0x7f2c840b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:21.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 -- 172.21.15.67:0/1921347160 shutdown_connections 2024-04-17T10:32:21.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 -- 172.21.15.67:0/1921347160 wait complete. 2024-04-17T10:32:21.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 Processor -- start 2024-04-17T10:32:21.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.726+0000 7f2c89a00640 1 -- start start 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c84159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c840cc250 0x7f2c840ad640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840ce2a0 0x7f2c840adb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8406e850 con 0x7f2c840ce2a0 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8406e6d0 con 0x7f2c840cc250 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8406e9d0 con 0x7f2c840cb260 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840ce2a0 0x7f2c840adb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840ce2a0 0x7f2c840adb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33152/0 (socket says 172.21.15.67:33152) 2024-04-17T10:32:21.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 -- 172.21.15.67:0/648191205 learned_addr learned my addr 172.21.15.67:0/648191205 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:21.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c82a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c84159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c7be00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c840cc250 0x7f2c840ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 -- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 msgr2=0x7f2c84159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c84159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 -- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c840cc250 msgr2=0x7f2c840ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c840cc250 0x7f2c840ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 -- 172.21.15.67:0/648191205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840b0df0 con 0x7f2c840ce2a0 2024-04-17T10:32:21.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c83400640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840ce2a0 0x7f2c840adb80 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00ef90 tx=0x7f2c7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:21.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c7aa00640 1 -- 172.21.15.67:0/648191205 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c018020 con 0x7f2c840ce2a0 2024-04-17T10:32:21.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c7aa00640 1 -- 172.21.15.67:0/648191205 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7c014480 con 0x7f2c840ce2a0 2024-04-17T10:32:21.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c8415ee70 con 0x7f2c840ce2a0 2024-04-17T10:32:21.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c7aa00640 1 -- 172.21.15.67:0/648191205 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c009e60 con 0x7f2c840ce2a0 2024-04-17T10:32:21.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c840b05f0 con 0x7f2c840ce2a0 2024-04-17T10:32:21.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c8409e900 con 0x7f2c840ce2a0 2024-04-17T10:32:21.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.730+0000 7f2c7aa00640 1 -- 172.21.15.67:0/648191205 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2c7c01b020 con 0x7f2c840ce2a0 2024-04-17T10:32:21.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.734+0000 7f2c7aa00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c60066f60 0x7f2c60069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.734+0000 7f2c7aa00640 1 -- 172.21.15.67:0/648191205 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2c7c085ce0 con 0x7f2c840ce2a0 2024-04-17T10:32:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.734+0000 7f2c82a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c60066f60 0x7f2c60069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:21.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.734+0000 7f2c82a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c60066f60 0x7f2c60069420 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2c740ad000 tx=0x7f2c74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:21.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.738+0000 7f2c7aa00640 1 -- 172.21.15.67:0/648191205 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2c7c022030 con 0x7f2c840ce2a0 2024-04-17T10:32:21.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.930+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2c84074e30 con 0x7f2c840ce2a0 2024-04-17T10:32:21.923 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:32:21.930+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:32:21.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c7aa00640 1 -- 172.21.15.67:0/648191205 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v65) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2c840b0df0 con 0x7f2c840ce2a0 2024-04-17T10:32:21.925 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:32:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c60066f60 msgr2=0x7f2c60069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c60066f60 0x7f2c60069420 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2c740ad000 tx=0x7f2c74008040 comp rx=0 tx=0).stop 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840ce2a0 msgr2=0x7f2c840adb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840ce2a0 0x7f2c840adb80 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00ef90 tx=0x7f2c7c00c490 comp rx=0 tx=0).stop 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 shutdown_connections 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c60066f60 0x7f2c60069420 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c840ce2a0 0x7f2c840adb80 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c840cc250 0x7f2c840ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 --2- 172.21.15.67:0/648191205 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c840cb260 0x7f2c84159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:21.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.934+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 >> 172.21.15.67:0/648191205 conn(0x7f2c840b22a0 msgr2=0x7f2c8415b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.938+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 shutdown_connections 2024-04-17T10:32:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:21.938+0000 7f2c89a00640 1 -- 172.21.15.67:0/648191205 wait complete. 2024-04-17T10:32:21.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:32:22.000 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:32:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa544a00640 1 Processor -- start 2024-04-17T10:32:22.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa544a00640 1 -- start start 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa544a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa5400aef60 0x7fa5400cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa544a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5400cce60 0x7fa54006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa544a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5400b4990 con 0x7fa54006e7b0 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa544a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400af340 con 0x7fa5400aef60 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa544a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5400b4130 con 0x7fa5400cce60 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa53de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa5400aef60 0x7fa5400cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa53de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa5400aef60 0x7fa5400cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33166/0 (socket says 172.21.15.67:33166) 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa53de00640 1 -- 172.21.15.67:0/3264301331 learned_addr learned my addr 172.21.15.67:0/3264301331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:22.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa537e00640 1 --2- 172.21.15.67:0/3264301331 >> v2:172.21.15.86:3300/0 conn(0x7fa5400cce60 0x7fa54006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa53de00640 1 -- 172.21.15.67:0/3264301331 >> v2:172.21.15.86:3300/0 conn(0x7fa5400cce60 msgr2=0x7fa54006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa53de00640 1 --2- 172.21.15.67:0/3264301331 >> v2:172.21.15.86:3300/0 conn(0x7fa5400cce60 0x7fa54006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa53de00640 1 -- 172.21.15.67:0/3264301331 >> v1:172.21.15.86:6789/0 conn(0x7fa54006e7b0 legacy=0x7fa5400713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:22.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.106+0000 7fa53de00640 1 -- 172.21.15.67:0/3264301331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54015ae90 con 0x7fa5400aef60 2024-04-17T10:32:22.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa53de00640 1 --2- 172.21.15.67:0/3264301331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa5400cc920 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fa5400b4400 tx=0x7fa53001c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3de4ba9fefd46f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:22.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa53d400640 1 -- 172.21.15.67:0/3264301331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300b0070 con 0x7fa5400aef60 2024-04-17T10:32:22.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa53d400640 1 -- 172.21.15.67:0/3264301331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5300ab600 con 0x7fa5400aef60 2024-04-17T10:32:22.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa53d400640 1 -- 172.21.15.67:0/3264301331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5300ab9d0 con 0x7fa5400aef60 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 -- 172.21.15.67:0/3264301331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 msgr2=0x7fa5400cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 --2- 172.21.15.67:0/3264301331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa5400cc920 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fa5400b4400 tx=0x7fa53001c270 comp rx=0 tx=0).stop 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 -- 172.21.15.67:0/3264301331 shutdown_connections 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 --2- 172.21.15.67:0/3264301331 >> v2:172.21.15.86:3300/0 conn(0x7fa5400cce60 0x7fa54006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 --2- 172.21.15.67:0/3264301331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa5400cc920 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 -- 172.21.15.67:0/3264301331 >> 172.21.15.67:0/3264301331 conn(0x7fa5400b1740 msgr2=0x7fa5400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 -- 172.21.15.67:0/3264301331 shutdown_connections 2024-04-17T10:32:22.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.110+0000 7fa544a00640 1 -- 172.21.15.67:0/3264301331 wait complete. 2024-04-17T10:32:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 Processor -- start 2024-04-17T10:32:22.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 -- start start 2024-04-17T10:32:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa54006e7b0 0x7fa5401473e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa54014b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5400cce60 0x7fa540151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54015bc10 con 0x7fa5400aef60 2024-04-17T10:32:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54006eb90 con 0x7fa5400cce60 2024-04-17T10:32:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa540147920 con 0x7fa54006e7b0 2024-04-17T10:32:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa54014b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa54014b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33176/0 (socket says 172.21.15.67:33176) 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 -- 172.21.15.67:0/1851658442 learned_addr learned my addr 172.21.15.67:0/1851658442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa53e800640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5400cce60 0x7fa540151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa53de00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa54006e7b0 0x7fa5401473e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 -- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa54006e7b0 msgr2=0x7fa5401473e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa54006e7b0 0x7fa5401473e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 -- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5400cce60 msgr2=0x7fa540151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5400cce60 0x7fa540151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 -- 172.21.15.67:0/1851658442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa540152750 con 0x7fa5400aef60 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa537e00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa54014b480 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fa52800d5f0 tx=0x7fa52800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:22.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa536000640 1 -- 172.21.15.67:0/1851658442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5280125f0 con 0x7fa5400aef60 2024-04-17T10:32:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa536000640 1 -- 172.21.15.67:0/1851658442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa528019070 con 0x7fa5400aef60 2024-04-17T10:32:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54014a3e0 con 0x7fa5400aef60 2024-04-17T10:32:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa536000640 1 -- 172.21.15.67:0/1851658442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5280129b0 con 0x7fa5400aef60 2024-04-17T10:32:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.114+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54014a9a0 con 0x7fa5400aef60 2024-04-17T10:32:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.118+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54004ef30 con 0x7fa5400aef60 2024-04-17T10:32:22.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.118+0000 7fa536000640 1 -- 172.21.15.67:0/1851658442 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa528012b50 con 0x7fa5400aef60 2024-04-17T10:32:22.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.118+0000 7fa536000640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa51c066f60 0x7fa51c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:22.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.118+0000 7fa536000640 1 -- 172.21.15.67:0/1851658442 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa5280854e0 con 0x7fa5400aef60 2024-04-17T10:32:22.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.118+0000 7fa53de00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa51c066f60 0x7fa51c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:22.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.122+0000 7fa53de00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa51c066f60 0x7fa51c069420 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa5300ad000 tx=0x7fa5300a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:22.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.122+0000 7fa536000640 1 -- 172.21.15.67:0/1851658442 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa528055cb0 con 0x7fa5400aef60 2024-04-17T10:32:22.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.322+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa54015d9a0 con 0x7fa5400aef60 2024-04-17T10:32:22.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa536000640 1 -- 172.21.15.67:0/1851658442 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v66) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa528059d50 con 0x7fa5400aef60 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa51c066f60 msgr2=0x7fa51c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa51c066f60 0x7fa51c069420 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fa5300ad000 tx=0x7fa5300a6000 comp rx=0 tx=0).stop 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 msgr2=0x7fa54014b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa54014b480 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fa52800d5f0 tx=0x7fa52800dab0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 shutdown_connections 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa51c066f60 0x7fa51c069420 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5400cce60 0x7fa540151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5400aef60 0x7fa54014b480 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 --2- 172.21.15.67:0/1851658442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa54006e7b0 0x7fa5401473e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 >> 172.21.15.67:0/1851658442 conn(0x7fa5400b1740 msgr2=0x7fa5400713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 shutdown_connections 2024-04-17T10:32:22.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:22.962+0000 7fa544a00640 1 -- 172.21.15.67:0/1851658442 wait complete. 2024-04-17T10:32:22.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:32:23.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba41800640 1 Processor -- start 2024-04-17T10:32:23.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba41800640 1 -- start start 2024-04-17T10:32:23.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba41800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba3c0cd0a0 0x7fba3c06e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:23.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba41800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c06ea10 0x7fba3c0b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:23.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba41800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c071bb0 con 0x7fba3c144100 2024-04-17T10:32:23.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba41800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3c0b3140 con 0x7fba3c0cd0a0 2024-04-17T10:32:23.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba41800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0b3420 con 0x7fba3c06ea10 2024-04-17T10:32:23.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba3b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c06ea10 0x7fba3c0b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:23.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba33e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba3c0cd0a0 0x7fba3c06e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:23.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba3b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c06ea10 0x7fba3c0b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:58280/0 (socket says 172.21.15.67:58280) 2024-04-17T10:32:23.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.134+0000 7fba3b400640 1 -- 172.21.15.67:0/2092573324 learned_addr learned my addr 172.21.15.67:0/2092573324 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:23.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba33e00640 1 -- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c06ea10 msgr2=0x7fba3c0b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:23.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba33e00640 1 --2- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c06ea10 0x7fba3c0b38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba33e00640 1 -- 172.21.15.67:0/2092573324 >> v1:172.21.15.86:6789/0 conn(0x7fba3c144100 legacy=0x7fba3c071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:23.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba33e00640 1 -- 172.21.15.67:0/2092573324 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c071e90 con 0x7fba3c0cd0a0 2024-04-17T10:32:23.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba3b400640 1 --2- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c06ea10 0x7fba3c0b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:23.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba33e00640 1 --2- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c0cd0a0 0x7fba3c06e4d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fba240097c0 tx=0x7fba2401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dedc008efffd69b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:23.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba3a000640 1 -- 172.21.15.67:0/2092573324 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240b0070 con 0x7fba3c0cd0a0 2024-04-17T10:32:23.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba3a000640 1 -- 172.21.15.67:0/2092573324 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba240ab600 con 0x7fba3c0cd0a0 2024-04-17T10:32:23.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba3a000640 1 -- 172.21.15.67:0/2092573324 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240ab9d0 con 0x7fba3c0cd0a0 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba41800640 1 -- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c0cd0a0 msgr2=0x7fba3c06e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba41800640 1 --2- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c0cd0a0 0x7fba3c06e4d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fba240097c0 tx=0x7fba2401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba41800640 1 -- 172.21.15.67:0/2092573324 shutdown_connections 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba41800640 1 --2- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c06ea10 0x7fba3c0b38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.138+0000 7fba41800640 1 --2- 172.21.15.67:0/2092573324 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c0cd0a0 0x7fba3c06e4d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 -- 172.21.15.67:0/2092573324 >> 172.21.15.67:0/2092573324 conn(0x7fba3c06ac90 msgr2=0x7fba3c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 -- 172.21.15.67:0/2092573324 shutdown_connections 2024-04-17T10:32:23.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 -- 172.21.15.67:0/2092573324 wait complete. 2024-04-17T10:32:23.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 Processor -- start 2024-04-17T10:32:23.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 -- start start 2024-04-17T10:32:23.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ea10 0x7fba3c15b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:23.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c0cd0a0 0x7fba3c159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:23.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c144100 0x7fba3c142530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0b4a10 con 0x7fba3c06ea10 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0b4890 con 0x7fba3c144100 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba41800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0b4b90 con 0x7fba3c0cd0a0 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ea10 0x7fba3c15b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ea10 0x7fba3c15b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33182/0 (socket says 172.21.15.67:33182) 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 -- 172.21.15.67:0/400593779 learned_addr learned my addr 172.21.15.67:0/400593779 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3b400640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c144100 0x7fba3c142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba33e00640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c0cd0a0 0x7fba3c159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 -- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c0cd0a0 msgr2=0x7fba3c159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c0cd0a0 0x7fba3c159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 -- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c144100 msgr2=0x7fba3c142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c144100 0x7fba3c142530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 -- 172.21.15.67:0/400593779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c15bec0 con 0x7fba3c06ea10 2024-04-17T10:32:23.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.142+0000 7fba3aa00640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ea10 0x7fba3c15b980 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c420 tx=0x7fba2c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:23.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba32a00640 1 -- 172.21.15.67:0/400593779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0052d0 con 0x7fba3c06ea10 2024-04-17T10:32:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba32a00640 1 -- 172.21.15.67:0/400593779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c005470 con 0x7fba3c06ea10 2024-04-17T10:32:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba32a00640 1 -- 172.21.15.67:0/400593779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c007260 con 0x7fba3c06ea10 2024-04-17T10:32:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c15c570 con 0x7fba3c06ea10 2024-04-17T10:32:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c15cb30 con 0x7fba3c06ea10 2024-04-17T10:32:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c072da0 con 0x7fba3c06ea10 2024-04-17T10:32:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba32a00640 1 -- 172.21.15.67:0/400593779 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fba2c007400 con 0x7fba3c06ea10 2024-04-17T10:32:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba32a00640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f90 0x7fba18069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba32a00640 1 -- 172.21.15.67:0/400593779 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fba2c084620 con 0x7fba3c06ea10 2024-04-17T10:32:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.146+0000 7fba33e00640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f90 0x7fba18069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.150+0000 7fba33e00640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f90 0x7fba18069450 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fba240ad000 tx=0x7fba24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.150+0000 7fba32a00640 1 -- 172.21.15.67:0/400593779 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fba2c054df0 con 0x7fba3c06ea10 2024-04-17T10:32:23.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.318+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fba3c074e30 con 0x7fba3c06ea10 2024-04-17T10:32:23.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.962+0000 7fba32a00640 1 -- 172.21.15.67:0/400593779 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v98) v1 ==== 181+0+0 (secure 0 0 0) 0x7fba2c058e90 con 0x7fba3c06ea10 2024-04-17T10:32:23.955 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f90 msgr2=0x7fba18069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f90 0x7fba18069450 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fba240ad000 tx=0x7fba24008040 comp rx=0 tx=0).stop 2024-04-17T10:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ea10 msgr2=0x7fba3c15b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:23.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ea10 0x7fba3c15b980 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c420 tx=0x7fba2c00c8e0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 shutdown_connections 2024-04-17T10:32:23.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba18066f90 0x7fba18069450 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba3c144100 0x7fba3c142530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba3c0cd0a0 0x7fba3c159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 --2- 172.21.15.67:0/400593779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba3c06ea10 0x7fba3c15b980 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:23.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.966+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 >> 172.21.15.67:0/400593779 conn(0x7fba3c06ac90 msgr2=0x7fba3c0adb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.970+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 shutdown_connections 2024-04-17T10:32:23.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:23.970+0000 7fba41800640 1 -- 172.21.15.67:0/400593779 wait complete. 2024-04-17T10:32:23.985 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:32:24.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.138+0000 7f63fe600640 1 Processor -- start 2024-04-17T10:32:24.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.138+0000 7f63fe600640 1 -- start start 2024-04-17T10:32:24.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63fe600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63f80cd0a0 0x7f63f806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63fe600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f8071bb0 con 0x7f63f8144100 2024-04-17T10:32:24.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63fe600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f8071e90 con 0x7f63f806ea10 2024-04-17T10:32:24.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63fe600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f80b3140 con 0x7f63f80cd0a0 2024-04-17T10:32:24.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f63f8144100 0x7f63f8071360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41240/0 (socket says 172.21.15.67:41240) 2024-04-17T10:32:24.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7400640 1 -- 172.21.15.67:0/1166596876 learned_addr learned my addr 172.21.15.67:0/1166596876 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:24.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63eea00640 1 --2- 172.21.15.67:0/1166596876 >> v2:172.21.15.86:3300/0 conn(0x7f63f80cd0a0 0x7f63f806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2715909215 0 0) 0x7f63f8071bb0 con 0x7f63f8144100 2024-04-17T10:32:24.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63dc003530 con 0x7f63f8144100 2024-04-17T10:32:24.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236852019 0 0) 0x7f63f8071e90 con 0x7f63f806ea10 2024-04-17T10:32:24.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63f8071bb0 con 0x7f63f806ea10 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 276395982 0 0) 0x7f63dc003530 con 0x7f63f8144100 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 >> v2:172.21.15.86:3300/0 conn(0x7f63f80cd0a0 msgr2=0x7f63f806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 --2- 172.21.15.67:0/1166596876 >> v2:172.21.15.86:3300/0 conn(0x7f63f80cd0a0 0x7f63f806e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 >> v1:172.21.15.86:6789/0 conn(0x7f63f806ea10 legacy=0x7f63f80b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f8072170 con 0x7f63f8144100 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f63e8001680 con 0x7f63f8144100 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 >> v1:172.21.15.67:6789/0 conn(0x7f63f8144100 legacy=0x7f63f8071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc003ba0 0x7f63dc003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc004810 0x7f63dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0051d0 0x7f63dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80b3140 con 0x7f63dc0051d0 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc00c0d0 con 0x7f63dc003ba0 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc00c2d0 con 0x7f63dc004810 2024-04-17T10:32:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7e00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0051d0 0x7f63dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63eea00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc004810 0x7f63dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7400640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc003ba0 0x7f63dc003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7e00640 1 -- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc004810 msgr2=0x7f63dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7e00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc004810 0x7f63dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7e00640 1 -- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc003ba0 msgr2=0x7f63dc003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7e00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc003ba0 0x7f63dc003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.142+0000 7f63f7e00640 1 -- 172.21.15.67:0/1166596876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc003530 con 0x7f63dc0051d0 2024-04-17T10:32:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63f7e00640 1 -- 172.21.15.67:0/1166596876 reap_dead start 2024-04-17T10:32:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63f7e00640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0051d0 0x7f63dc00b9d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f63f0001fd0 tx=0x7f63f001c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd1ec2eea8ba902 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0001c20 con 0x7f63dc0051d0 2024-04-17T10:32:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0003980 con 0x7f63dc0051d0 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63f6a00640 1 -- 172.21.15.67:0/1166596876 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0003d70 con 0x7f63dc0051d0 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63fe600640 1 -- 172.21.15.67:0/1166596876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0051d0 msgr2=0x7f63dc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63fe600640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0051d0 0x7f63dc00b9d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f63f0001fd0 tx=0x7f63f001c900 comp rx=0 tx=0).stop 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63fe600640 1 -- 172.21.15.67:0/1166596876 shutdown_connections 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63fe600640 1 --2- 172.21.15.67:0/1166596876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0051d0 0x7f63dc00b9d0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63fe600640 1 -- 172.21.15.67:0/1166596876 >> 172.21.15.67:0/1166596876 conn(0x7f63f806ac90 msgr2=0x7f63f806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63fe600640 1 -- 172.21.15.67:0/1166596876 shutdown_connections 2024-04-17T10:32:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.146+0000 7f63fe600640 1 -- 172.21.15.67:0/1166596876 wait complete. 2024-04-17T10:32:24.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 Processor -- start 2024-04-17T10:32:24.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 -- start start 2024-04-17T10:32:24.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0051d0 0x7f63f815c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f8140520 0x7f63f8150eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f813e4f0 0x7f63f813e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f806e4d0 con 0x7f63f8140520 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f80b3420 con 0x7f63dc0051d0 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f806e650 con 0x7f63f813e4f0 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f8140520 0x7f63f8150eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f8140520 0x7f63f8150eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33208/0 (socket says 172.21.15.67:33208) 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 -- 172.21.15.67:0/3829445917 learned_addr learned my addr 172.21.15.67:0/3829445917 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:24.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63f7400640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0051d0 0x7f63f815c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63f7e00640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f813e4f0 0x7f63f813e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 -- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f813e4f0 msgr2=0x7f63f813e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f813e4f0 0x7f63f813e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 -- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0051d0 msgr2=0x7f63f815c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0051d0 0x7f63f815c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 -- 172.21.15.67:0/3829445917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f815ca20 con 0x7f63f8140520 2024-04-17T10:32:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63eea00640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f8140520 0x7f63f8150eb0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f63e0011960 tx=0x7f63e0011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63efe00640 1 -- 172.21.15.67:0/3829445917 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e000faf0 con 0x7f63f8140520 2024-04-17T10:32:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63efe00640 1 -- 172.21.15.67:0/3829445917 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e000fc90 con 0x7f63f8140520 2024-04-17T10:32:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63efe00640 1 -- 172.21.15.67:0/3829445917 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e0014440 con 0x7f63f8140520 2024-04-17T10:32:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f81513f0 con 0x7f63f8140520 2024-04-17T10:32:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f8140ce0 con 0x7f63f8140520 2024-04-17T10:32:24.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.150+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f8072da0 con 0x7f63f8140520 2024-04-17T10:32:24.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.154+0000 7f63efe00640 1 -- 172.21.15.67:0/3829445917 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f63e0012040 con 0x7f63f8140520 2024-04-17T10:32:24.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.154+0000 7f63efe00640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d4066f60 0x7f63d4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:24.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.154+0000 7f63efe00640 1 -- 172.21.15.67:0/3829445917 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f63e0085aa0 con 0x7f63f8140520 2024-04-17T10:32:24.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.154+0000 7f63f7400640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d4066f60 0x7f63d4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:24.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.154+0000 7f63f7400640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d4066f60 0x7f63d4069420 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f63e8001060 tx=0x7f63e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:24.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.158+0000 7f63efe00640 1 -- 172.21.15.67:0/3829445917 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f63f815ca20 con 0x7f63f8140520 2024-04-17T10:32:24.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.322+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f63f8074e30 con 0x7f63f8140520 2024-04-17T10:32:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.978+0000 7f63efe00640 1 -- 172.21.15.67:0/3829445917 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v99) v1 ==== 169+0+0 (secure 0 0 0) 0x7f63e0019030 con 0x7f63f8140520 2024-04-17T10:32:24.970 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:32:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d4066f60 msgr2=0x7f63d4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d4066f60 0x7f63d4069420 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f63e8001060 tx=0x7f63e8009040 comp rx=0 tx=0).stop 2024-04-17T10:32:24.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f8140520 msgr2=0x7f63f8150eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f8140520 0x7f63f8150eb0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f63e0011960 tx=0x7f63e0011e20 comp rx=0 tx=0).stop 2024-04-17T10:32:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 shutdown_connections 2024-04-17T10:32:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d4066f60 0x7f63d4069420 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f813e4f0 0x7f63f813e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f8140520 0x7f63f8150eb0 secure :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f63e0011960 tx=0x7f63e0011e20 comp rx=0 tx=0).stop 2024-04-17T10:32:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 --2- 172.21.15.67:0/3829445917 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc0051d0 0x7f63f815c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:24.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 >> 172.21.15.67:0/3829445917 conn(0x7f63f806ac90 msgr2=0x7f63f8141a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 shutdown_connections 2024-04-17T10:32:24.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:24.982+0000 7f63fe600640 1 -- 172.21.15.67:0/3829445917 wait complete. 2024-04-17T10:32:25.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:32:25.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf49e00640 1 Processor -- start 2024-04-17T10:32:25.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf49e00640 1 -- start start 2024-04-17T10:32:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf49e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf4406b3a0 0x7fdf440b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf49e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf441616a0 con 0x7fdf441474c0 2024-04-17T10:32:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf49e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf44161940 con 0x7fdf440b3920 2024-04-17T10:32:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf49e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44161c20 con 0x7fdf4406b3a0 2024-04-17T10:32:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf42a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdf441474c0 0x7fdf4406ac90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41246/0 (socket says 172.21.15.67:41246) 2024-04-17T10:32:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf42a00640 1 -- 172.21.15.67:0/3078679190 learned_addr learned my addr 172.21.15.67:0/3078679190 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:25.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf3be00640 1 --2- 172.21.15.67:0/3078679190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf4406b3a0 0x7fdf440b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf42000640 1 -- 172.21.15.67:0/3078679190 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 542063641 0 0) 0x7fdf441616a0 con 0x7fdf441474c0 2024-04-17T10:32:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf42000640 1 -- 172.21.15.67:0/3078679190 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf28003530 con 0x7fdf441474c0 2024-04-17T10:32:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf42000640 1 -- 172.21.15.67:0/3078679190 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843120844 0 0) 0x7fdf44161940 con 0x7fdf440b3920 2024-04-17T10:32:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf42000640 1 -- 172.21.15.67:0/3078679190 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf441616a0 con 0x7fdf440b3920 2024-04-17T10:32:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf3be00640 1 -- 172.21.15.67:0/3078679190 >> v1:172.21.15.86:6789/0 conn(0x7fdf440b3920 legacy=0x7fdf440d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf3be00640 1 -- 172.21.15.67:0/3078679190 >> v1:172.21.15.67:6789/0 conn(0x7fdf441474c0 legacy=0x7fdf4406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.162+0000 7fdf3be00640 1 -- 172.21.15.67:0/3078679190 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44161f00 con 0x7fdf4406b3a0 2024-04-17T10:32:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf3be00640 1 --2- 172.21.15.67:0/3078679190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf4406b3a0 0x7fdf440b1b90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c0097f0 tx=0x7fdf2c01ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=815dd744ea4a978a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf42000640 1 -- 172.21.15.67:0/3078679190 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c0b5070 con 0x7fdf4406b3a0 2024-04-17T10:32:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf42000640 1 -- 172.21.15.67:0/3078679190 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c0a89d0 con 0x7fdf4406b3a0 2024-04-17T10:32:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf42000640 1 -- 172.21.15.67:0/3078679190 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c0a8d80 con 0x7fdf4406b3a0 2024-04-17T10:32:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf49e00640 1 -- 172.21.15.67:0/3078679190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf4406b3a0 msgr2=0x7fdf440b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3078679190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf4406b3a0 0x7fdf440b1b90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c0097f0 tx=0x7fdf2c01ecd0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf49e00640 1 -- 172.21.15.67:0/3078679190 shutdown_connections 2024-04-17T10:32:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3078679190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf4406b3a0 0x7fdf440b1b90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf49e00640 1 -- 172.21.15.67:0/3078679190 >> 172.21.15.67:0/3078679190 conn(0x7fdf440b5ef0 msgr2=0x7fdf44099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf49e00640 1 -- 172.21.15.67:0/3078679190 shutdown_connections 2024-04-17T10:32:25.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.166+0000 7fdf49e00640 1 -- 172.21.15.67:0/3078679190 wait complete. 2024-04-17T10:32:25.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 Processor -- start 2024-04-17T10:32:25.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 -- start start 2024-04-17T10:32:25.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf4406b3a0 0x7fdf441f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf440b3920 0x7fdf4415ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf441474c0 0x7fdf4413ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf44162420 con 0x7fdf441474c0 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf440b2250 con 0x7fdf4406b3a0 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf440d0b90 con 0x7fdf440b3920 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf441474c0 0x7fdf4413ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf42a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf4406b3a0 0x7fdf441f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf3be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf440b3920 0x7fdf4415ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf441474c0 0x7fdf4413ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33220/0 (socket says 172.21.15.67:33220) 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 -- 172.21.15.67:0/3782710811 learned_addr learned my addr 172.21.15.67:0/3782710811 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 -- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf440b3920 msgr2=0x7fdf4415ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf440b3920 0x7fdf4415ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 -- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf4406b3a0 msgr2=0x7fdf441f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf4406b3a0 0x7fdf441f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 -- 172.21.15.67:0/3782710811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf440af3a0 con 0x7fdf441474c0 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf43400640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf441474c0 0x7fdf4413ebd0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c790 tx=0x7fdf3c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf3b400640 1 -- 172.21.15.67:0/3782710811 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c003b80 con 0x7fdf441474c0 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf3b400640 1 -- 172.21.15.67:0/3782710811 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c003d20 con 0x7fdf441474c0 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf440af610 con 0x7fdf441474c0 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf3b400640 1 -- 172.21.15.67:0/3782710811 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c00aa50 con 0x7fdf441474c0 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf440afbd0 con 0x7fdf441474c0 2024-04-17T10:32:25.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.170+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4404ef30 con 0x7fdf441474c0 2024-04-17T10:32:25.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.174+0000 7fdf3b400640 1 -- 172.21.15.67:0/3782710811 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdf3c00abf0 con 0x7fdf441474c0 2024-04-17T10:32:25.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.174+0000 7fdf3b400640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf20066f60 0x7fdf20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.174+0000 7fdf3b400640 1 -- 172.21.15.67:0/3782710811 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fdf3c084d70 con 0x7fdf441474c0 2024-04-17T10:32:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.174+0000 7fdf42a00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf20066f60 0x7fdf20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.174+0000 7fdf42a00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf20066f60 0x7fdf20069420 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdf34007fd0 tx=0x7fdf34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:25.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.178+0000 7fdf3b400640 1 -- 172.21.15.67:0/3782710811 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdf3c055890 con 0x7fdf441474c0 2024-04-17T10:32:25.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.402+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdf440b13a0 con 0x7fdf441474c0 2024-04-17T10:32:25.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf3b400640 1 -- 172.21.15.67:0/3782710811 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdf3c059930 con 0x7fdf441474c0 2024-04-17T10:32:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf20066f60 msgr2=0x7fdf20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf20066f60 0x7fdf20069420 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fdf34007fd0 tx=0x7fdf34009040 comp rx=0 tx=0).stop 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf441474c0 msgr2=0x7fdf4413ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf441474c0 0x7fdf4413ebd0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c00c790 tx=0x7fdf3c00cc50 comp rx=0 tx=0).stop 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 shutdown_connections 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf20066f60 0x7fdf20069420 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf441474c0 0x7fdf4413ebd0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf440b3920 0x7fdf4415ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 --2- 172.21.15.67:0/3782710811 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf4406b3a0 0x7fdf441f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 >> 172.21.15.67:0/3782710811 conn(0x7fdf440b5ef0 msgr2=0x7fdf4406ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 shutdown_connections 2024-04-17T10:32:25.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.406+0000 7fdf49e00640 1 -- 172.21.15.67:0/3782710811 wait complete. 2024-04-17T10:32:25.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort' 2024-04-17T10:32:25.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe683e00640 1 Processor -- start 2024-04-17T10:32:25.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe683e00640 1 -- start start 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe683e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15cc00 0x7fe67c15f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe683e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe67c15fa00 0x7fe67c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe683e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c15be60 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe683e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe67c160230 con 0x7fe67c15fa00 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe683e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c15c120 con 0x7fe67c15cc00 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe681e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe67c15fa00 0x7fe67c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe681400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe67c0ce1b0 0x7fe67c0ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41254/0 (socket says 172.21.15.67:41254) 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.570+0000 7fe681400640 1 -- 172.21.15.67:0/4075152966 learned_addr learned my addr 172.21.15.67:0/4075152966 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:25.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe680a00640 1 --2- 172.21.15.67:0/4075152966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15cc00 0x7fe67c15f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe681e00640 1 -- 172.21.15.67:0/4075152966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15cc00 msgr2=0x7fe67c15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe681e00640 1 --2- 172.21.15.67:0/4075152966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15cc00 0x7fe67c15f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe681e00640 1 -- 172.21.15.67:0/4075152966 >> v1:172.21.15.67:6789/0 conn(0x7fe67c0ce1b0 legacy=0x7fe67c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe681e00640 1 -- 172.21.15.67:0/4075152966 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c15c400 con 0x7fe67c15fa00 2024-04-17T10:32:25.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe67be00640 1 -- 172.21.15.67:0/4075152966 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65772645 0 0) 0x7fe67c15be60 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe681e00640 1 --2- 172.21.15.67:0/4075152966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c15fa00 0x7fe67c0c6b30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe674009d90 tx=0x7fe67401f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b781ffb1116e4596 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:25.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe67be00640 1 -- 172.21.15.67:0/4075152966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6740b4070 con 0x7fe67c15fa00 2024-04-17T10:32:25.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe67be00640 1 -- 172.21.15.67:0/4075152966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6740af600 con 0x7fe67c15fa00 2024-04-17T10:32:25.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe67be00640 1 -- 172.21.15.67:0/4075152966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6740af9d0 con 0x7fe67c15fa00 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe683e00640 1 -- 172.21.15.67:0/4075152966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c15fa00 msgr2=0x7fe67c0c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.574+0000 7fe683e00640 1 --2- 172.21.15.67:0/4075152966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c15fa00 0x7fe67c0c6b30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fe674009d90 tx=0x7fe67401f150 comp rx=0 tx=0).stop 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- 172.21.15.67:0/4075152966 shutdown_connections 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 --2- 172.21.15.67:0/4075152966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c15fa00 0x7fe67c0c6b30 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 --2- 172.21.15.67:0/4075152966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15cc00 0x7fe67c15f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- 172.21.15.67:0/4075152966 >> 172.21.15.67:0/4075152966 conn(0x7fe67c0b33f0 msgr2=0x7fe67c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- 172.21.15.67:0/4075152966 shutdown_connections 2024-04-17T10:32:25.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- 172.21.15.67:0/4075152966 wait complete. 2024-04-17T10:32:25.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 Processor -- start 2024-04-17T10:32:25.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- start start 2024-04-17T10:32:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c0ce1b0 0x7fe67c1f6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c15cc00 0x7fe67c0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15fa00 0x7fe67c1419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c160010 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c15fe90 con 0x7fe67c15cc00 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe683e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67c159730 con 0x7fe67c15fa00 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c0ce1b0 0x7fe67c1f6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c0ce1b0 0x7fe67c1f6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33224/0 (socket says 172.21.15.67:33224) 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 -- 172.21.15.67:0/1147605499 learned_addr learned my addr 172.21.15.67:0/1147605499 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681e00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15fa00 0x7fe67c1419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe680a00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c15cc00 0x7fe67c0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 -- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15fa00 msgr2=0x7fe67c1419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15fa00 0x7fe67c1419e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 -- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c15cc00 msgr2=0x7fe67c0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c15cc00 0x7fe67c0cccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 -- 172.21.15.67:0/1147605499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0cbc50 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe681400640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c0ce1b0 0x7fe67c1f6bf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe6700011f0 tx=0x7fe67000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.578+0000 7fe679600640 1 -- 172.21.15.67:0/1147605499 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670018070 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe679600640 1 -- 172.21.15.67:0/1147605499 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6700074e0 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe679600640 1 -- 172.21.15.67:0/1147605499 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670004a10 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c0cbf30 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c0cc4f0 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c070ab0 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe679600640 1 -- 172.21.15.67:0/1147605499 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe670004bb0 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe679600640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe65c066f90 0x7fe65c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:25.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe679600640 1 -- 172.21.15.67:0/1147605499 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fe670084180 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe680a00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe65c066f90 0x7fe65c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:25.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.582+0000 7fe680a00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe65c066f90 0x7fe65c069450 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe668009af0 tx=0x7fe668008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:25.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.586+0000 7fe679600640 1 -- 172.21.15.67:0/1147605499 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe6700552f0 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.806+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]} v 0) v1 -- 0x7fe67c074e30 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.978+0000 7fe679600640 1 -- 172.21.15.67:0/1147605499 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]}]=0 v363) v1 ==== 144+0+0 (secure 0 0 0) 0x7fe670054bf0 con 0x7fe67c0ce1b0 2024-04-17T10:32:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe65c066f90 msgr2=0x7fe65c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe65c066f90 0x7fe65c069450 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe668009af0 tx=0x7fe668008040 comp rx=0 tx=0).stop 2024-04-17T10:32:25.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c0ce1b0 msgr2=0x7fe67c1f6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c0ce1b0 0x7fe67c1f6bf0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fe6700011f0 tx=0x7fe67000c9b0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 shutdown_connections 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe65c066f90 0x7fe65c069450 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe67c15fa00 0x7fe67c1419e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe67c15cc00 0x7fe67c0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 --2- 172.21.15.67:0/1147605499 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe67c0ce1b0 0x7fe67c1f6bf0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 >> 172.21.15.67:0/1147605499 conn(0x7fe67c0b33f0 msgr2=0x7fe67c0c74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 shutdown_connections 2024-04-17T10:32:25.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:25.982+0000 7fe683e00640 1 -- 172.21.15.67:0/1147605499 wait complete. 2024-04-17T10:32:25.995 INFO:tasks.cephfs_test_runner:test_scrub_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-04-17T10:32:25.995 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_pause_and_resume (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-04-17T10:32:25.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:26.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6893000640 1 Processor -- start 2024-04-17T10:32:26.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6893000640 1 -- start start 2024-04-17T10:32:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6893000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f688c1620a0 0x7f688c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6893000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c15eb50 con 0x7f688c15e550 2024-04-17T10:32:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6893000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f688c15ee30 con 0x7f688c0ce020 2024-04-17T10:32:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6893000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f688c161650 con 0x7f688c1620a0 2024-04-17T10:32:26.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6891000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f688c15e550 0x7f688c0cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41258/0 (socket says 172.21.15.67:41258) 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6891000640 1 -- 172.21.15.67:0/177232868 learned_addr learned my addr 172.21.15.67:0/177232868 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6883e00640 1 --2- 172.21.15.67:0/177232868 >> v2:172.21.15.86:3300/0 conn(0x7f688c1620a0 0x7f688c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f688b400640 1 -- 172.21.15.67:0/177232868 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168475944 0 0) 0x7f688c15eb50 con 0x7f688c15e550 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f688b400640 1 -- 172.21.15.67:0/177232868 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6870003530 con 0x7f688c15e550 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f688b400640 1 -- 172.21.15.67:0/177232868 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415630757 0 0) 0x7f688c15ee30 con 0x7f688c0ce020 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f688b400640 1 -- 172.21.15.67:0/177232868 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c15eb50 con 0x7f688c0ce020 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6883e00640 1 -- 172.21.15.67:0/177232868 >> v1:172.21.15.86:6789/0 conn(0x7f688c0ce020 legacy=0x7f688c0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6883e00640 1 -- 172.21.15.67:0/177232868 >> v1:172.21.15.67:6789/0 conn(0x7f688c15e550 legacy=0x7f688c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.146+0000 7f6883e00640 1 -- 172.21.15.67:0/177232868 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c1618a0 con 0x7f688c1620a0 2024-04-17T10:32:26.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6883e00640 1 --2- 172.21.15.67:0/177232868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c1620a0 0x7f688c15e010 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f68740097f0 tx=0x7f687401c430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a263d49006c6b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f688b400640 1 -- 172.21.15.67:0/177232868 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68740b0070 con 0x7f688c1620a0 2024-04-17T10:32:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f688b400640 1 -- 172.21.15.67:0/177232868 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68740ab600 con 0x7f688c1620a0 2024-04-17T10:32:26.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f688b400640 1 -- 172.21.15.67:0/177232868 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68740ab9d0 con 0x7f688c1620a0 2024-04-17T10:32:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6893000640 1 -- 172.21.15.67:0/177232868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c1620a0 msgr2=0x7f688c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6893000640 1 --2- 172.21.15.67:0/177232868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c1620a0 0x7f688c15e010 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f68740097f0 tx=0x7f687401c430 comp rx=0 tx=0).stop 2024-04-17T10:32:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6893000640 1 -- 172.21.15.67:0/177232868 shutdown_connections 2024-04-17T10:32:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6893000640 1 --2- 172.21.15.67:0/177232868 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c1620a0 0x7f688c15e010 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6893000640 1 -- 172.21.15.67:0/177232868 >> 172.21.15.67:0/177232868 conn(0x7f688c0b33f0 msgr2=0x7f688c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:26.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6893000640 1 -- 172.21.15.67:0/177232868 shutdown_connections 2024-04-17T10:32:26.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.150+0000 7f6893000640 1 -- 172.21.15.67:0/177232868 wait complete. 2024-04-17T10:32:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 Processor -- start 2024-04-17T10:32:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 -- start start 2024-04-17T10:32:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f688c0ce020 0x7f688c0b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c15e550 0x7f688c152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f688c1620a0 0x7f688c0b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c0c8630 con 0x7f688c0ce020 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688c0c84b0 con 0x7f688c15e550 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f688c16ba00 con 0x7f688c1620a0 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f688c0ce020 0x7f688c0b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f688c0ce020 0x7f688c0b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33240/0 (socket says 172.21.15.67:33240) 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 -- 172.21.15.67:0/3281874487 learned_addr learned my addr 172.21.15.67:0/3281874487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6883e00640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c15e550 0x7f688c152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6891000640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f688c1620a0 0x7f688c0b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 -- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f688c1620a0 msgr2=0x7f688c0b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f688c1620a0 0x7f688c0b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 -- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c15e550 msgr2=0x7f688c152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c15e550 0x7f688c152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 -- 172.21.15.67:0/3281874487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688c0b53d0 con 0x7f688c0ce020 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f688be00640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f688c0ce020 0x7f688c0b46f0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c530 tx=0x7f687c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6888c00640 1 -- 172.21.15.67:0/3281874487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c0052d0 con 0x7f688c0ce020 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6888c00640 1 -- 172.21.15.67:0/3281874487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687c005490 con 0x7f688c0ce020 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6888c00640 1 -- 172.21.15.67:0/3281874487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687c012850 con 0x7f688c0ce020 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688c154d50 con 0x7f688c0ce020 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.154+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688c155310 con 0x7f688c0ce020 2024-04-17T10:32:26.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.158+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f688c070ab0 con 0x7f688c0ce020 2024-04-17T10:32:26.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.158+0000 7f6888c00640 1 -- 172.21.15.67:0/3281874487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f687c0129f0 con 0x7f688c0ce020 2024-04-17T10:32:26.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.158+0000 7f6888c00640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6868066f60 0x7f6868069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.158+0000 7f6888c00640 1 -- 172.21.15.67:0/3281874487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f687c084b80 con 0x7f688c0ce020 2024-04-17T10:32:26.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.158+0000 7f6883e00640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6868066f60 0x7f6868069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.158+0000 7f6883e00640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6868066f60 0x7f6868069420 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f688c0b3380 tx=0x7f68740a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.162+0000 7f6888c00640 1 -- 172.21.15.67:0/3281874487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f687c055c70 con 0x7f688c0ce020 2024-04-17T10:32:26.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.322+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f688c0c75f0 con 0x7f688c0ce020 2024-04-17T10:32:26.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6888c00640 1 -- 172.21.15.67:0/3281874487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+13246 (secure 0 0 0) 0x7f687c055570 con 0x7f688c0ce020 2024-04-17T10:32:26.317 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:26.317 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":99,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:24.975596+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/3523329517":"2024-04-18T10:32:21.934999+0000","172.21.15.67:6833/3523329517":"2024-04-18T10:32:21.934999+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:26.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6868066f60 msgr2=0x7f6868069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6868066f60 0x7f6868069420 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f688c0b3380 tx=0x7f68740a6000 comp rx=0 tx=0).stop 2024-04-17T10:32:26.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f688c0ce020 msgr2=0x7f688c0b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f688c0ce020 0x7f688c0b46f0 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f687c00c530 tx=0x7f687c00c9f0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 shutdown_connections 2024-04-17T10:32:26.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6868066f60 0x7f6868069420 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f688c1620a0 0x7f688c0b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f688c15e550 0x7f688c152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 --2- 172.21.15.67:0/3281874487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f688c0ce020 0x7f688c0b46f0 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 >> 172.21.15.67:0/3281874487 conn(0x7f688c0b33f0 msgr2=0x7f688c0cd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:26.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 shutdown_connections 2024-04-17T10:32:26.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.326+0000 7f6893000640 1 -- 172.21.15.67:0/3281874487 wait complete. 2024-04-17T10:32:26.337 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:32:26.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:32:26.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.490+0000 7f69dd600640 1 Processor -- start 2024-04-17T10:32:26.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.490+0000 7f69dd600640 1 -- start start 2024-04-17T10:32:26.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.490+0000 7f69dd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d80cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.490+0000 7f69dd600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f69d80cce60 0x7f69d806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.490+0000 7f69dd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69d80b4990 con 0x7f69d806e7b0 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.490+0000 7f69dd600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d80cd2c0 con 0x7f69d80cce60 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.490+0000 7f69dd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d80af340 con 0x7f69d80aef60 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f69d806e7b0 0x7f69d80713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41262/0 (socket says 172.21.15.67:41262) 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d7400640 1 -- 172.21.15.67:0/299886799 learned_addr learned my addr 172.21.15.67:0/299886799 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69cfe00640 1 --2- 172.21.15.67:0/299886799 >> v2:172.21.15.86:3300/0 conn(0x7f69d80cce60 0x7f69d806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d6a00640 1 --2- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d80cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d6000640 1 -- 172.21.15.67:0/299886799 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017871978 0 0) 0x7f69d00014a0 con 0x7f69d806e7b0 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d6000640 1 -- 172.21.15.67:0/299886799 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69bc005740 con 0x7f69d806e7b0 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69cfe00640 1 -- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 msgr2=0x7f69d80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69cfe00640 1 --2- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d80cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69cfe00640 1 -- 172.21.15.67:0/299886799 >> v1:172.21.15.67:6789/0 conn(0x7f69d806e7b0 legacy=0x7f69d80713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69cfe00640 1 -- 172.21.15.67:0/299886799 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d815ae90 con 0x7f69d80cce60 2024-04-17T10:32:26.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d6a00640 1 --2- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d80cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:26.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69cfe00640 1 --2- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d806e270 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f69c00097c0 tx=0x7f69c001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a97f3084149a4d35 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d6000640 1 -- 172.21.15.67:0/299886799 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00b0070 con 0x7f69d80cce60 2024-04-17T10:32:26.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d6000640 1 -- 172.21.15.67:0/299886799 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c00ab600 con 0x7f69d80cce60 2024-04-17T10:32:26.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.494+0000 7f69d6000640 1 -- 172.21.15.67:0/299886799 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c00ab9d0 con 0x7f69d80cce60 2024-04-17T10:32:26.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 msgr2=0x7f69d806e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 --2- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d806e270 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f69c00097c0 tx=0x7f69c001c410 comp rx=0 tx=0).stop 2024-04-17T10:32:26.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- 172.21.15.67:0/299886799 shutdown_connections 2024-04-17T10:32:26.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 --2- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d806e270 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 --2- 172.21.15.67:0/299886799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d80cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- 172.21.15.67:0/299886799 >> 172.21.15.67:0/299886799 conn(0x7f69d80b1740 msgr2=0x7f69d80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:26.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- 172.21.15.67:0/299886799 shutdown_connections 2024-04-17T10:32:26.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- 172.21.15.67:0/299886799 wait complete. 2024-04-17T10:32:26.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 Processor -- start 2024-04-17T10:32:26.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- start start 2024-04-17T10:32:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d806e7b0 0x7f69d8151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d8145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d8159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d815bc10 con 0x7f69d806e7b0 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d815b410 con 0x7f69d80cce60 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69dd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d814b270 con 0x7f69d80aef60 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69d6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d806e7b0 0x7f69d8151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69d6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d806e7b0 0x7f69d8151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33250/0 (socket says 172.21.15.67:33250) 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69d6a00640 1 -- 172.21.15.67:0/1866639022 learned_addr learned my addr 172.21.15.67:0/1866639022 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69cfe00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d8145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69d6a00640 1 -- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 msgr2=0x7f69d8145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69d6a00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d8145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69d6a00640 1 -- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 msgr2=0x7f69d8159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.498+0000 7f69d7400640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d8159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69d6a00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d8159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69d6a00640 1 -- 172.21.15.67:0/1866639022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d806f120 con 0x7f69d806e7b0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69cfe00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d8145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69d7400640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d8159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69d6a00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d806e7b0 0x7f69d8151e80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f69c800e9b0 tx=0x7f69c800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69cea00640 1 -- 172.21.15.67:0/1866639022 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c8017070 con 0x7f69d806e7b0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69cea00640 1 -- 172.21.15.67:0/1866639022 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c800ca00 con 0x7f69d806e7b0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69cea00640 1 -- 172.21.15.67:0/1866639022 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c800cdd0 con 0x7f69d806e7b0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d81510b0 con 0x7f69d806e7b0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d814af00 con 0x7f69d806e7b0 2024-04-17T10:32:26.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d804ef30 con 0x7f69d806e7b0 2024-04-17T10:32:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69cea00640 1 -- 172.21.15.67:0/1866639022 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f69c801a070 con 0x7f69d806e7b0 2024-04-17T10:32:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.502+0000 7f69cea00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69b4066f60 0x7f69b4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.506+0000 7f69cea00640 1 -- 172.21.15.67:0/1866639022 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f69c8084d30 con 0x7f69d806e7b0 2024-04-17T10:32:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.506+0000 7f69cfe00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69b4066f60 0x7f69b4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.506+0000 7f69cfe00640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69b4066f60 0x7f69b4069420 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f69c00ad000 tx=0x7f69c0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.510+0000 7f69cea00640 1 -- 172.21.15.67:0/1866639022 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f69c8055f40 con 0x7f69d806e7b0 2024-04-17T10:32:26.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.670+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f69d815ac70 con 0x7f69d806e7b0 2024-04-17T10:32:26.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.670+0000 7f69cea00640 1 -- 172.21.15.67:0/1866639022 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f69c8055840 con 0x7f69d806e7b0 2024-04-17T10:32:26.664 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:32:26.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69b4066f60 msgr2=0x7f69b4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69b4066f60 0x7f69b4069420 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f69c00ad000 tx=0x7f69c0008040 comp rx=0 tx=0).stop 2024-04-17T10:32:26.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d806e7b0 msgr2=0x7f69d8151e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d806e7b0 0x7f69d8151e80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f69c800e9b0 tx=0x7f69c800ee70 comp rx=0 tx=0).stop 2024-04-17T10:32:26.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 shutdown_connections 2024-04-17T10:32:26.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f69b4066f60 0x7f69b4069420 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f69d80cce60 0x7f69d8159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f69d80aef60 0x7f69d8145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 --2- 172.21.15.67:0/1866639022 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69d806e7b0 0x7f69d8151e80 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 >> 172.21.15.67:0/1866639022 conn(0x7f69d80b1740 msgr2=0x7f69d80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 shutdown_connections 2024-04-17T10:32:26.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.674+0000 7f69dd600640 1 -- 172.21.15.67:0/1866639022 wait complete. 2024-04-17T10:32:26.685 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume' 2024-04-17T10:32:26.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaf4e00640 1 Processor -- start 2024-04-17T10:32:26.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaf4e00640 1 -- start start 2024-04-17T10:32:26.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaf4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feaf0071310 0x7feaf0071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaf4e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf006e2f0 con 0x7feaf0145040 2024-04-17T10:32:26.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaf4e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf006d6e0 con 0x7feaf00cc300 2024-04-17T10:32:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaf4e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feaf00a86c0 con 0x7feaf0071310 2024-04-17T10:32:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feae5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feaf0071310 0x7feaf0071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaede00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feaf0145040 0x7feaf006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41264/0 (socket says 172.21.15.67:41264) 2024-04-17T10:32:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feaede00640 1 -- 172.21.15.67:0/2152518668 learned_addr learned my addr 172.21.15.67:0/2152518668 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feae5400640 1 -- 172.21.15.67:0/2152518668 >> v1:172.21.15.86:6789/0 conn(0x7feaf00cc300 legacy=0x7feaf00cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feae5400640 1 -- 172.21.15.67:0/2152518668 >> v1:172.21.15.67:6789/0 conn(0x7feaf0145040 legacy=0x7feaf006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.834+0000 7feae5400640 1 -- 172.21.15.67:0/2152518668 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf006d9c0 con 0x7feaf0071310 2024-04-17T10:32:26.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feae5400640 1 --2- 172.21.15.67:0/2152518668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf0071310 0x7feaf0071770 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fead80097c0 tx=0x7fead801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467c81d0966c86ab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaed400640 1 -- 172.21.15.67:0/2152518668 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80b5070 con 0x7feaf0071310 2024-04-17T10:32:26.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaed400640 1 -- 172.21.15.67:0/2152518668 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead80a8950 con 0x7feaf0071310 2024-04-17T10:32:26.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaed400640 1 -- 172.21.15.67:0/2152518668 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80a8d20 con 0x7feaf0071310 2024-04-17T10:32:26.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaf4e00640 1 -- 172.21.15.67:0/2152518668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf0071310 msgr2=0x7feaf0071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaf4e00640 1 --2- 172.21.15.67:0/2152518668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf0071310 0x7feaf0071770 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fead80097c0 tx=0x7fead801ec50 comp rx=0 tx=0).stop 2024-04-17T10:32:26.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaf4e00640 1 -- 172.21.15.67:0/2152518668 shutdown_connections 2024-04-17T10:32:26.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaf4e00640 1 --2- 172.21.15.67:0/2152518668 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf0071310 0x7feaf0071770 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaf4e00640 1 -- 172.21.15.67:0/2152518668 >> 172.21.15.67:0/2152518668 conn(0x7feaf006ac90 msgr2=0x7feaf006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:26.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaf4e00640 1 -- 172.21.15.67:0/2152518668 shutdown_connections 2024-04-17T10:32:26.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.838+0000 7feaf4e00640 1 -- 172.21.15.67:0/2152518668 wait complete. 2024-04-17T10:32:26.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 Processor -- start 2024-04-17T10:32:26.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 -- start start 2024-04-17T10:32:26.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feaf0071310 0x7feaf013c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf00cc300 0x7feaf0139030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feaf0145040 0x7feaf014eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf006df80 con 0x7feaf00cc300 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf015bf10 con 0x7feaf0071310 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf006d370 con 0x7feaf0145040 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf00cc300 0x7feaf0139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf00cc300 0x7feaf0139030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33262/0 (socket says 172.21.15.67:33262) 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 -- 172.21.15.67:0/855015886 learned_addr learned my addr 172.21.15.67:0/855015886 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaede00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feaf0071310 0x7feaf013c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaee800640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feaf0145040 0x7feaf014eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 -- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feaf0145040 msgr2=0x7feaf014eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feaf0145040 0x7feaf014eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 -- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feaf0071310 msgr2=0x7feaf013c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:26.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feaf0071310 0x7feaf013c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:26.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 -- 172.21.15.67:0/855015886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf0151910 con 0x7feaf00cc300 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae5400640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf00cc300 0x7feaf0139030 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fead801b860 tx=0x7fead801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae6a00640 1 -- 172.21.15.67:0/855015886 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80b5040 con 0x7feaf00cc300 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae6a00640 1 -- 172.21.15.67:0/855015886 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead80bf070 con 0x7feaf00cc300 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf0141340 con 0x7feaf00cc300 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feae6a00640 1 -- 172.21.15.67:0/855015886 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80b0b40 con 0x7feaf00cc300 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf0141900 con 0x7feaf00cc300 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.842+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf0072d50 con 0x7feaf00cc300 2024-04-17T10:32:26.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.846+0000 7feae6a00640 1 -- 172.21.15.67:0/855015886 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fead80b0ce0 con 0x7feaf00cc300 2024-04-17T10:32:26.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.846+0000 7feae6a00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feacc066fb0 0x7feacc069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:26.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.846+0000 7feae6a00640 1 -- 172.21.15.67:0/855015886 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fead8122460 con 0x7feaf00cc300 2024-04-17T10:32:26.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.846+0000 7feaede00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feacc066fb0 0x7feacc069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:26.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.846+0000 7feaede00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feacc066fb0 0x7feacc069470 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7feae00012f0 tx=0x7feae0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:26.850+0000 7feae6a00640 1 -- 172.21.15.67:0/855015886 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fead80187b0 con 0x7feaf00cc300 2024-04-17T10:32:27.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.070+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]} v 0) v1 -- 0x7feaf0074e30 con 0x7feaf00cc300 2024-04-17T10:32:27.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:32:27.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:32:27.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.986+0000 7feae6a00640 1 -- 172.21.15.67:0/855015886 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]}]=0 v365) v1 ==== 158+0+0 (secure 0 0 0) 0x7fead80af3a0 con 0x7feaf00cc300 2024-04-17T10:32:27.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feacc066fb0 msgr2=0x7feacc069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feacc066fb0 0x7feacc069470 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7feae00012f0 tx=0x7feae0009040 comp rx=0 tx=0).stop 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf00cc300 msgr2=0x7feaf0139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf00cc300 0x7feaf0139030 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fead801b860 tx=0x7fead801b9f0 comp rx=0 tx=0).stop 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 shutdown_connections 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feacc066fb0 0x7feacc069470 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feaf0145040 0x7feaf014eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feaf00cc300 0x7feaf0139030 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 --2- 172.21.15.67:0/855015886 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feaf0071310 0x7feaf013c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 >> 172.21.15.67:0/855015886 conn(0x7feaf006ac90 msgr2=0x7feaf015fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 shutdown_connections 2024-04-17T10:32:27.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:27.990+0000 7feaf4e00640 1 -- 172.21.15.67:0/855015886 wait complete. 2024-04-17T10:32:28.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:32:28.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.162+0000 7f5f66e00640 1 Processor -- start 2024-04-17T10:32:28.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.162+0000 7f5f66e00640 1 -- start start 2024-04-17T10:32:28.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.162+0000 7f5f66e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f600ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.162+0000 7f5f66e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f600713e0 0x7f5f60071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.162+0000 7f5f66e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6015cb30 con 0x7f5f6015be70 2024-04-17T10:32:28.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.162+0000 7f5f66e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f600a8650 con 0x7f5f600713e0 2024-04-17T10:32:28.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.162+0000 7f5f66e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60071d80 con 0x7f5f60159440 2024-04-17T10:32:28.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f600ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f57e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f600713e0 0x7f5f60071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f600ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:58406/0 (socket says 172.21.15.67:58406) 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 -- 172.21.15.67:0/1719907645 learned_addr learned my addr 172.21.15.67:0/1719907645 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 -- 172.21.15.67:0/1719907645 >> v2:172.21.15.86:3300/0 conn(0x7f5f600713e0 msgr2=0x7f5f60071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 --2- 172.21.15.67:0/1719907645 >> v2:172.21.15.86:3300/0 conn(0x7f5f600713e0 0x7f5f60071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 -- 172.21.15.67:0/1719907645 >> v1:172.21.15.86:6789/0 conn(0x7f5f6015be70 legacy=0x7f5f6015c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 -- 172.21.15.67:0/1719907645 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600cd610 con 0x7f5f60159440 2024-04-17T10:32:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f57e00640 1 --2- 172.21.15.67:0/1719907645 >> v2:172.21.15.86:3300/0 conn(0x7f5f600713e0 0x7f5f60071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5fe00640 1 --2- 172.21.15.67:0/1719907645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f600ce0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009870 tx=0x7f5f5001c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4d81a9cfc1962c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5f400640 1 -- 172.21.15.67:0/1719907645 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f500a7070 con 0x7f5f60159440 2024-04-17T10:32:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5f400640 1 -- 172.21.15.67:0/1719907645 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f500b0070 con 0x7f5f60159440 2024-04-17T10:32:28.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.166+0000 7f5f5f400640 1 -- 172.21.15.67:0/1719907645 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f500ab890 con 0x7f5f60159440 2024-04-17T10:32:28.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- 172.21.15.67:0/1719907645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 msgr2=0x7f5f600ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 --2- 172.21.15.67:0/1719907645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f600ce0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009870 tx=0x7f5f5001c560 comp rx=0 tx=0).stop 2024-04-17T10:32:28.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- 172.21.15.67:0/1719907645 shutdown_connections 2024-04-17T10:32:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 --2- 172.21.15.67:0/1719907645 >> v2:172.21.15.86:3300/0 conn(0x7f5f600713e0 0x7f5f60071840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 --2- 172.21.15.67:0/1719907645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f600ce0f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- 172.21.15.67:0/1719907645 >> 172.21.15.67:0/1719907645 conn(0x7f5f600b1740 msgr2=0x7f5f600b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- 172.21.15.67:0/1719907645 shutdown_connections 2024-04-17T10:32:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- 172.21.15.67:0/1719907645 wait complete. 2024-04-17T10:32:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 Processor -- start 2024-04-17T10:32:28.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- start start 2024-04-17T10:32:28.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f600713e0 0x7f5f60158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f601499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f6015be70 0x7f5f6014c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6015d9c0 con 0x7f5f6015be70 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6015d840 con 0x7f5f600713e0 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f66e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6015db40 con 0x7f5f60159440 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f6015be70 0x7f5f6014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f6015be70 0x7f5f6014c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33272/0 (socket says 172.21.15.67:33272) 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 -- 172.21.15.67:0/3104140225 learned_addr learned my addr 172.21.15.67:0/3104140225 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f5fe00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f600713e0 0x7f5f60158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f57e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f601499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 -- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 msgr2=0x7f5f601499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f601499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 -- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f600713e0 msgr2=0x7f5f60158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f600713e0 0x7f5f60158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.170+0000 7f5f64e00640 1 -- 172.21.15.67:0/3104140225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600b4250 con 0x7f5f6015be70 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f64e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f6015be70 0x7f5f6014c850 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f5f5800c760 tx=0x7f5f5800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f5cc00640 1 -- 172.21.15.67:0/3104140225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f58003b80 con 0x7f5f6015be70 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f5cc00640 1 -- 172.21.15.67:0/3104140225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f58003d20 con 0x7f5f6015be70 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f600b4530 con 0x7f5f6015be70 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f5cc00640 1 -- 172.21.15.67:0/3104140225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5800aa30 con 0x7f5f6015be70 2024-04-17T10:32:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60158100 con 0x7f5f6015be70 2024-04-17T10:32:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6004ef30 con 0x7f5f6015be70 2024-04-17T10:32:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f5cc00640 1 -- 172.21.15.67:0/3104140225 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5f5800abd0 con 0x7f5f6015be70 2024-04-17T10:32:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f5cc00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f3c066f60 0x7f5f3c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f5cc00640 1 -- 172.21.15.67:0/3104140225 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5f58084da0 con 0x7f5f6015be70 2024-04-17T10:32:28.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.174+0000 7f5f5fe00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f3c066f60 0x7f5f3c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.178+0000 7f5f5fe00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f3c066f60 0x7f5f3c069420 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5f500af000 tx=0x7f5f500a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:28.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.178+0000 7f5f5cc00640 1 -- 172.21.15.67:0/3104140225 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5f58055e80 con 0x7f5f6015be70 2024-04-17T10:32:28.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.342+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5f600b1d00 con 0x7f5f6015be70 2024-04-17T10:32:28.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.346+0000 7f5f5cc00640 1 -- 172.21.15.67:0/3104140225 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5f58055780 con 0x7f5f6015be70 2024-04-17T10:32:28.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f3c066f60 msgr2=0x7f5f3c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f3c066f60 0x7f5f3c069420 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5f500af000 tx=0x7f5f500a7000 comp rx=0 tx=0).stop 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f6015be70 msgr2=0x7f5f6014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f6015be70 0x7f5f6014c850 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f5f5800c760 tx=0x7f5f5800cc20 comp rx=0 tx=0).stop 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 shutdown_connections 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f3c066f60 0x7f5f3c069420 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f6015be70 0x7f5f6014c850 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f60159440 0x7f5f601499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 --2- 172.21.15.67:0/3104140225 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f600713e0 0x7f5f60158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 >> 172.21.15.67:0/3104140225 conn(0x7f5f600b1740 msgr2=0x7f5f6014eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 shutdown_connections 2024-04-17T10:32:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.350+0000 7f5f66e00640 1 -- 172.21.15.67:0/3104140225 wait complete. 2024-04-17T10:32:28.361 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:32:28.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:32:28.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:32:28.515 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:32:28.515 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:32:28.515 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:28.525 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:32:28.526 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:32:28.526 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:28.535 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:32:28.536 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:32:28.536 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:28.576 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:32:28.576 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:32:28.576 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:28.585 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:32:28.585 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:32:28.585 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:28.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.766+0000 7fabd8800640 1 Processor -- start 2024-04-17T10:32:28.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.766+0000 7fabd8800640 1 -- start start 2024-04-17T10:32:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd8800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006e6e0 0x7fabd00ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd8800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabd006eac0 0x7fabd00ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd8800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fabd0143ae0 0x7fabd0143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd8800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd00cf430 con 0x7fabd006eac0 2024-04-17T10:32:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd8800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd015ae40 con 0x7fabd0143ae0 2024-04-17T10:32:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd8800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0071440 con 0x7fabd006e6e0 2024-04-17T10:32:28.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabd006eac0 0x7fabd00ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fabd006eac0 0x7fabd00ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33278/0 (socket says 172.21.15.67:33278) 2024-04-17T10:32:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 -- 172.21.15.67:0/4257358612 learned_addr learned my addr 172.21.15.67:0/4257358612 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd6800640 1 --2- 172.21.15.67:0/4257358612 >> v2:172.21.15.86:3300/0 conn(0x7fabd0143ae0 0x7fabd0143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5e00640 1 --2- 172.21.15.67:0/4257358612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006e6e0 0x7fabd00ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 -- 172.21.15.67:0/4257358612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006e6e0 msgr2=0x7fabd00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 --2- 172.21.15.67:0/4257358612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006e6e0 0x7fabd00ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 -- 172.21.15.67:0/4257358612 >> v2:172.21.15.86:3300/0 conn(0x7fabd0143ae0 msgr2=0x7fabd0143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:28.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 --2- 172.21.15.67:0/4257358612 >> v2:172.21.15.86:3300/0 conn(0x7fabd0143ae0 0x7fabd0143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5400640 1 -- 172.21.15.67:0/4257358612 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd0071720 con 0x7fabd006eac0 2024-04-17T10:32:28.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd6800640 1 --2- 172.21.15.67:0/4257358612 >> v2:172.21.15.86:3300/0 conn(0x7fabd0143ae0 0x7fabd0143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:28.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.770+0000 7fabd5e00640 1 --2- 172.21.15.67:0/4257358612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006e6e0 0x7fabd00ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:28.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd5400640 1 --2- 172.21.15.67:0/4257358612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd006eac0 0x7fabd00ceef0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fabd006ef20 tx=0x7fabc001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccbfc6ad521d992a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:28.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd4a00640 1 -- 172.21.15.67:0/4257358612 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc00ab620 con 0x7fabd006eac0 2024-04-17T10:32:28.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd4a00640 1 -- 172.21.15.67:0/4257358612 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc00b0070 con 0x7fabd006eac0 2024-04-17T10:32:28.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd4a00640 1 -- 172.21.15.67:0/4257358612 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc00ab9e0 con 0x7fabd006eac0 2024-04-17T10:32:28.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 -- 172.21.15.67:0/4257358612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd006eac0 msgr2=0x7fabd00ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 --2- 172.21.15.67:0/4257358612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd006eac0 0x7fabd00ceef0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fabd006ef20 tx=0x7fabc001c460 comp rx=0 tx=0).stop 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 -- 172.21.15.67:0/4257358612 shutdown_connections 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 --2- 172.21.15.67:0/4257358612 >> v2:172.21.15.86:3300/0 conn(0x7fabd0143ae0 0x7fabd0143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 --2- 172.21.15.67:0/4257358612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd006eac0 0x7fabd00ceef0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 --2- 172.21.15.67:0/4257358612 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006e6e0 0x7fabd00ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 -- 172.21.15.67:0/4257358612 >> 172.21.15.67:0/4257358612 conn(0x7fabd006ac90 msgr2=0x7fabd006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 -- 172.21.15.67:0/4257358612 shutdown_connections 2024-04-17T10:32:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 -- 172.21.15.67:0/4257358612 wait complete. 2024-04-17T10:32:28.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 Processor -- start 2024-04-17T10:32:28.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.774+0000 7fabd8800640 1 -- start start 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd006e6e0 0x7fabd015fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006eac0 0x7fabd015a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd0143ae0 0x7fabd00ad9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd015ebf0 con 0x7fabd0143ae0 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd01444b0 con 0x7fabd006e6e0 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd015ed70 con 0x7fabd006eac0 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd0143ae0 0x7fabd00ad9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd0143ae0 0x7fabd00ad9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33286/0 (socket says 172.21.15.67:33286) 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 -- 172.21.15.67:0/3024517928 learned_addr learned my addr 172.21.15.67:0/3024517928 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:28.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd5e00640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd006e6e0 0x7fabd015fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd5400640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006eac0 0x7fabd015a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 -- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006eac0 msgr2=0x7fabd015a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006eac0 0x7fabd015a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 -- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd006e6e0 msgr2=0x7fabd015fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd006e6e0 0x7fabd015fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 -- 172.21.15.67:0/3024517928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd01603e0 con 0x7fabd0143ae0 2024-04-17T10:32:28.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd6800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd0143ae0 0x7fabd00ad9b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00cd50 tx=0x7fabcc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabca000640 1 -- 172.21.15.67:0/3024517928 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00f680 con 0x7fabd0143ae0 2024-04-17T10:32:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabca000640 1 -- 172.21.15.67:0/3024517928 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabcc00f840 con 0x7fabd0143ae0 2024-04-17T10:32:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd01606c0 con 0x7fabd0143ae0 2024-04-17T10:32:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabca000640 1 -- 172.21.15.67:0/3024517928 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabcc00fc30 con 0x7fabd0143ae0 2024-04-17T10:32:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd0160c80 con 0x7fabd0143ae0 2024-04-17T10:32:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0072d50 con 0x7fabd0143ae0 2024-04-17T10:32:28.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.778+0000 7fabca000640 1 -- 172.21.15.67:0/3024517928 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fabcc0213f0 con 0x7fabd0143ae0 2024-04-17T10:32:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.782+0000 7fabca000640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb4066f90 0x7fabb4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:28.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.782+0000 7fabca000640 1 -- 172.21.15.67:0/3024517928 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fabcc085b50 con 0x7fabd0143ae0 2024-04-17T10:32:28.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.782+0000 7fabd5e00640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb4066f90 0x7fabb4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:28.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.782+0000 7fabd5e00640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb4066f90 0x7fabb4069450 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fabd015d680 tx=0x7fabc4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:28.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.786+0000 7fabca000640 1 -- 172.21.15.67:0/3024517928 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fabcc014030 con 0x7fabd0143ae0 2024-04-17T10:32:28.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.986+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabd0143550 con 0x7fabd0143ae0 2024-04-17T10:32:28.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.986+0000 7fabca000640 1 -- 172.21.15.67:0/3024517928 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+3693 (secure 0 0 0) 0x7fabcc0565e0 con 0x7fabd0143ae0 2024-04-17T10:32:28.979 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:28.979 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":66,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1957726114","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1957726114},{"type":"v1","addr":"172.21.15.86:6836","nonce":1957726114}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7204,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/976808521","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":976808521},{"type":"v1","addr":"172.21.15.86:6837","nonce":976808521}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7695,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/737293610","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":737293610},{"type":"v1","addr":"172.21.15.67:6834","nonce":737293610}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":8064,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3540739013","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3540739013},{"type":"v1","addr":"172.21.15.67:6835","nonce":3540739013}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":66}],"filesystems":[]} 2024-04-17T10:32:28.979 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 66 2024-04-17T10:32:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb4066f90 msgr2=0x7fabb4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb4066f90 0x7fabb4069450 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fabd015d680 tx=0x7fabc4008040 comp rx=0 tx=0).stop 2024-04-17T10:32:28.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd0143ae0 msgr2=0x7fabd00ad9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd0143ae0 0x7fabd00ad9b0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fabcc00cd50 tx=0x7fabcc00a650 comp rx=0 tx=0).stop 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 shutdown_connections 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabb4066f90 0x7fabb4069450 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fabd0143ae0 0x7fabd00ad9b0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fabd006eac0 0x7fabd015a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 --2- 172.21.15.67:0/3024517928 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fabd006e6e0 0x7fabd015fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 >> 172.21.15.67:0/3024517928 conn(0x7fabd006ac90 msgr2=0x7fabd006d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 shutdown_connections 2024-04-17T10:32:28.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:28.990+0000 7fabd8800640 1 -- 172.21.15.67:0/3024517928 wait complete. 2024-04-17T10:32:29.002 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:32:29.003 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:32:29.003 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:32:29.003 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:32:29.004 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:32:29.004 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:32:29.004 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:32:29.004 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:32:29.004 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:32:29.004 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:32:29.004 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:32:29.005 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:32:29.005 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:32:29.005 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:32:29.005 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:32:29.005 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:32:29.015 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:32:29.015 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:32:29.016 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:32:29.016 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:32:29.020 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:32:29.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:32:29.021 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:32:29.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:32:29.063 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:32:29.063 DEBUG:teuthology.parallel:result is None 2024-04-17T10:32:29.064 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:32:29.064 DEBUG:teuthology.parallel:result is None 2024-04-17T10:32:29.071 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:32:29.072 DEBUG:teuthology.parallel:result is None 2024-04-17T10:32:29.073 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:32:29.073 DEBUG:teuthology.parallel:result is None 2024-04-17T10:32:29.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:32:29.150 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:32:29.151 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:32:29.163 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:32:29.163 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:32:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efdae800640 1 Processor -- start 2024-04-17T10:32:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efdae800640 1 -- start start 2024-04-17T10:32:29.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efdae800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efdae800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efda81441f0 0x7efda80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efdae800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda815ae40 con 0x7efda806eac0 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efdae800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda80cf430 con 0x7efda81441f0 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efdae800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda8071440 con 0x7efda806e6e0 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efd9fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efda806eac0 0x7efda8143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41294/0 (socket says 172.21.15.67:41294) 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efd9fe00640 1 -- 172.21.15.67:0/1813960318 learned_addr learned my addr 172.21.15.67:0/1813960318 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda7400640 1 --2- 172.21.15.67:0/1813960318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda7e00640 1 --2- 172.21.15.67:0/1813960318 >> v2:172.21.15.86:3300/0 conn(0x7efda81441f0 0x7efda80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda6a00640 1 -- 172.21.15.67:0/1813960318 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199963330 0 0) 0x7efda815ae40 con 0x7efda806eac0 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda6a00640 1 -- 172.21.15.67:0/1813960318 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c003530 con 0x7efda806eac0 2024-04-17T10:32:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda6a00640 1 -- 172.21.15.67:0/1813960318 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1628160968 0 0) 0x7efd8c003530 con 0x7efda806eac0 2024-04-17T10:32:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda7400640 1 -- 172.21.15.67:0/1813960318 >> v2:172.21.15.86:3300/0 conn(0x7efda81441f0 msgr2=0x7efda80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda7400640 1 --2- 172.21.15.67:0/1813960318 >> v2:172.21.15.86:3300/0 conn(0x7efda81441f0 0x7efda80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda7400640 1 -- 172.21.15.67:0/1813960318 >> v1:172.21.15.67:6789/0 conn(0x7efda806eac0 legacy=0x7efda8143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda7400640 1 -- 172.21.15.67:0/1813960318 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda8071720 con 0x7efda806e6e0 2024-04-17T10:32:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.230+0000 7efda7e00640 1 --2- 172.21.15.67:0/1813960318 >> v2:172.21.15.86:3300/0 conn(0x7efda81441f0 0x7efda80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:29.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efda7400640 1 --2- 172.21.15.67:0/1813960318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda80ce9b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd980097c0 tx=0x7efd9801c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dccabffc12dd2775 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:29.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efda6a00640 1 -- 172.21.15.67:0/1813960318 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd980b0070 con 0x7efda806e6e0 2024-04-17T10:32:29.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efda6a00640 1 -- 172.21.15.67:0/1813960318 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd980ab400 con 0x7efda806e6e0 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efda6a00640 1 -- 172.21.15.67:0/1813960318 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90001680 con 0x7efda806e6e0 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 -- 172.21.15.67:0/1813960318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 msgr2=0x7efda80ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 --2- 172.21.15.67:0/1813960318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda80ce9b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efd980097c0 tx=0x7efd9801c300 comp rx=0 tx=0).stop 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 -- 172.21.15.67:0/1813960318 shutdown_connections 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 --2- 172.21.15.67:0/1813960318 >> v2:172.21.15.86:3300/0 conn(0x7efda81441f0 0x7efda80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 --2- 172.21.15.67:0/1813960318 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda80ce9b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 -- 172.21.15.67:0/1813960318 >> 172.21.15.67:0/1813960318 conn(0x7efda806ac90 msgr2=0x7efda806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 -- 172.21.15.67:0/1813960318 shutdown_connections 2024-04-17T10:32:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.234+0000 7efdae800640 1 -- 172.21.15.67:0/1813960318 wait complete. 2024-04-17T10:32:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 Processor -- start 2024-04-17T10:32:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 -- start start 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda813cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efda806eac0 0x7efda80abb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda81441f0 0x7efda80aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda80b3a40 con 0x7efda806eac0 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda80b38c0 con 0x7efda81441f0 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda80b3bc0 con 0x7efda806e6e0 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efda806eac0 0x7efda80abb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efda806eac0 0x7efda80abb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33322/0 (socket says 172.21.15.67:33322) 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 -- 172.21.15.67:0/43345435 learned_addr learned my addr 172.21.15.67:0/43345435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efda7400640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda813cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efda7e00640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda81441f0 0x7efda80aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 -- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 msgr2=0x7efda813cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda813cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 -- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda81441f0 msgr2=0x7efda80aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda81441f0 0x7efda80aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 -- 172.21.15.67:0/43345435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda80c9ea0 con 0x7efda806eac0 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9fe00640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efda806eac0 0x7efda80abb00 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c750 tx=0x7efd9000cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9f400640 1 -- 172.21.15.67:0/43345435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90005ae0 con 0x7efda806eac0 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9f400640 1 -- 172.21.15.67:0/43345435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd90005ca0 con 0x7efda806eac0 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efd9f400640 1 -- 172.21.15.67:0/43345435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd900072b0 con 0x7efda806eac0 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda80ca140 con 0x7efda806eac0 2024-04-17T10:32:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.238+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda80ca700 con 0x7efda806eac0 2024-04-17T10:32:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.242+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda8072d50 con 0x7efda806eac0 2024-04-17T10:32:29.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.242+0000 7efd9f400640 1 -- 172.21.15.67:0/43345435 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efd90007450 con 0x7efda806eac0 2024-04-17T10:32:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.242+0000 7efd9f400640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efd84066f60 0x7efd84069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.242+0000 7efd9f400640 1 -- 172.21.15.67:0/43345435 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5335+0+0 (secure 0 0 0) 0x7efd90084d40 con 0x7efda806eac0 2024-04-17T10:32:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.242+0000 7efda7400640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efd84066f60 0x7efd84069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:29.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.242+0000 7efda7400640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efd84066f60 0x7efd84069420 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efd980ad000 tx=0x7efd980a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:29.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.246+0000 7efd9f400640 1 -- 172.21.15.67:0/43345435 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efd90055e20 con 0x7efda806eac0 2024-04-17T10:32:29.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:29.422+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7efda8074e30 con 0x7efda806eac0 2024-04-17T10:32:29.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.002+0000 7efd9f400640 1 -- 172.21.15.67:0/43345435 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v101) v1 ==== 97+0+0 (secure 0 0 0) 0x7efd90055720 con 0x7efda806eac0 2024-04-17T10:32:29.996 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:32:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efd84066f60 msgr2=0x7efd84069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efd84066f60 0x7efd84069420 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7efd980ad000 tx=0x7efd980a6000 comp rx=0 tx=0).stop 2024-04-17T10:32:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efda806eac0 msgr2=0x7efda80abb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efda806eac0 0x7efda80abb00 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7efd9000c750 tx=0x7efd9000cb50 comp rx=0 tx=0).stop 2024-04-17T10:32:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 shutdown_connections 2024-04-17T10:32:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efd84066f60 0x7efd84069420 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:29.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efda81441f0 0x7efda80aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efda806eac0 0x7efda80abb00 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 --2- 172.21.15.67:0/43345435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efda806e6e0 0x7efda813cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 >> 172.21.15.67:0/43345435 conn(0x7efda806ac90 msgr2=0x7efda80a9410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 shutdown_connections 2024-04-17T10:32:30.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.006+0000 7efdae800640 1 -- 172.21.15.67:0/43345435 wait complete. 2024-04-17T10:32:30.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:32:30.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.170+0000 7f0a66600640 1 Processor -- start 2024-04-17T10:32:30.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.170+0000 7f0a66600640 1 -- start start 2024-04-17T10:32:30.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.170+0000 7f0a66600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0a600cf0c0 0x7f0a601441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.170+0000 7f0a66600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6015ae40 con 0x7f0a6006eac0 2024-04-17T10:32:30.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.170+0000 7f0a66600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a60071440 con 0x7f0a6006e6e0 2024-04-17T10:32:30.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.170+0000 7f0a66600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a60071720 con 0x7f0a600cf0c0 2024-04-17T10:32:30.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a56a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0a6006eac0 0x7f0a60143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41304/0 (socket says 172.21.15.67:41304) 2024-04-17T10:32:30.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a56a00640 1 -- 172.21.15.67:0/834745704 learned_addr learned my addr 172.21.15.67:0/834745704 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:30.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/834745704 >> v2:172.21.15.86:3300/0 conn(0x7f0a600cf0c0 0x7f0a601441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5ea00640 1 -- 172.21.15.67:0/834745704 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931807275 0 0) 0x7f0a6015ae40 con 0x7f0a6006eac0 2024-04-17T10:32:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5ea00640 1 -- 172.21.15.67:0/834745704 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a44005740 con 0x7f0a6006eac0 2024-04-17T10:32:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5fe00640 1 -- 172.21.15.67:0/834745704 >> v1:172.21.15.86:6789/0 conn(0x7f0a6006e6e0 legacy=0x7f0a600ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5fe00640 1 -- 172.21.15.67:0/834745704 >> v1:172.21.15.67:6789/0 conn(0x7f0a6006eac0 legacy=0x7f0a60143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5fe00640 1 -- 172.21.15.67:0/834745704 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60071a00 con 0x7f0a600cf0c0 2024-04-17T10:32:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/834745704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a600cf0c0 0x7f0a601441f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009d90 tx=0x7f0a5801c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f8d3d9b255b25f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5ea00640 1 -- 172.21.15.67:0/834745704 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a580b2070 con 0x7f0a600cf0c0 2024-04-17T10:32:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5ea00640 1 -- 172.21.15.67:0/834745704 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a580a5870 con 0x7f0a600cf0c0 2024-04-17T10:32:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.174+0000 7f0a5ea00640 1 -- 172.21.15.67:0/834745704 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a580a5c40 con 0x7f0a600cf0c0 2024-04-17T10:32:30.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- 172.21.15.67:0/834745704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a600cf0c0 msgr2=0x7f0a601441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 --2- 172.21.15.67:0/834745704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a600cf0c0 0x7f0a601441f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0a58009d90 tx=0x7f0a5801c8b0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- 172.21.15.67:0/834745704 shutdown_connections 2024-04-17T10:32:30.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 --2- 172.21.15.67:0/834745704 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a600cf0c0 0x7f0a601441f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- 172.21.15.67:0/834745704 >> 172.21.15.67:0/834745704 conn(0x7f0a6006ac90 msgr2=0x7f0a6006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:30.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- 172.21.15.67:0/834745704 shutdown_connections 2024-04-17T10:32:30.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- 172.21.15.67:0/834745704 wait complete. 2024-04-17T10:32:30.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 Processor -- start 2024-04-17T10:32:30.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- start start 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6006e6e0 0x7f0a60140820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006eac0 0x7f0a60137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a600cf0c0 0x7f0a600ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6006dd00 con 0x7f0a6006eac0 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6006db80 con 0x7f0a6006e6e0 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a66600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6006de80 con 0x7f0a600cf0c0 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a56a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006eac0 0x7f0a60137680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a56a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006eac0 0x7f0a60137680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33334/0 (socket says 172.21.15.67:33334) 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a56a00640 1 -- 172.21.15.67:0/2813592910 learned_addr learned my addr 172.21.15.67:0/2813592910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a5fe00640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a600cf0c0 0x7f0a600ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.178+0000 7f0a5f400640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6006e6e0 0x7f0a60140820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a56a00640 1 -- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a600cf0c0 msgr2=0x7f0a600ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a56a00640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a600cf0c0 0x7f0a600ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a56a00640 1 -- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6006e6e0 msgr2=0x7f0a60140820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a56a00640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6006e6e0 0x7f0a60140820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a56a00640 1 -- 172.21.15.67:0/2813592910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60139d40 con 0x7f0a6006eac0 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a56a00640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006eac0 0x7f0a60137680 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c4d0 tx=0x7f0a4800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a57e00640 1 -- 172.21.15.67:0/2813592910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48017070 con 0x7f0a6006eac0 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a57e00640 1 -- 172.21.15.67:0/2813592910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a48005cf0 con 0x7f0a6006eac0 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a57e00640 1 -- 172.21.15.67:0/2813592910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48007270 con 0x7f0a6006eac0 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6013fa60 con 0x7f0a6006eac0 2024-04-17T10:32:30.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a60140020 con 0x7f0a6006eac0 2024-04-17T10:32:30.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a60072d50 con 0x7f0a6006eac0 2024-04-17T10:32:30.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a57e00640 1 -- 172.21.15.67:0/2813592910 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0a4801a070 con 0x7f0a6006eac0 2024-04-17T10:32:30.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a57e00640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a3c066f60 0x7f0a3c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.182+0000 7f0a57e00640 1 -- 172.21.15.67:0/2813592910 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f0a48084700 con 0x7f0a6006eac0 2024-04-17T10:32:30.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.186+0000 7f0a5f400640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a3c066f60 0x7f0a3c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.186+0000 7f0a5f400640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a3c066f60 0x7f0a3c069420 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0a50001810 tx=0x7f0a50009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.186+0000 7f0a57e00640 1 -- 172.21.15.67:0/2813592910 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0a480551d0 con 0x7f0a6006eac0 2024-04-17T10:32:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.406+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0a60074e30 con 0x7f0a6006eac0 2024-04-17T10:32:30.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.406+0000 7f0a57e00640 1 -- 172.21.15.67:0/2813592910 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0a48059270 con 0x7f0a6006eac0 2024-04-17T10:32:30.398 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.399 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.400 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.401 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.402 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.403 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.404 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.405 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:32:30.406 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.407 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.408 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:32:30.409 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:32:30.410 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.411 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:30.412 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:32:30.412 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a3c066f60 msgr2=0x7f0a3c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a3c066f60 0x7f0a3c069420 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0a50001810 tx=0x7f0a50009040 comp rx=0 tx=0).stop 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006eac0 msgr2=0x7f0a60137680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006eac0 0x7f0a60137680 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c4d0 tx=0x7f0a4800c990 comp rx=0 tx=0).stop 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 shutdown_connections 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a3c066f60 0x7f0a3c069420 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a600cf0c0 0x7f0a600ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a6006eac0 0x7f0a60137680 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 --2- 172.21.15.67:0/2813592910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a6006e6e0 0x7f0a60140820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 >> 172.21.15.67:0/2813592910 conn(0x7f0a6006ac90 msgr2=0x7f0a600b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 shutdown_connections 2024-04-17T10:32:30.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.410+0000 7f0a66600640 1 -- 172.21.15.67:0/2813592910 wait complete. 2024-04-17T10:32:30.420 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:30.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6623c00640 1 Processor -- start 2024-04-17T10:32:30.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6623c00640 1 -- start start 2024-04-17T10:32:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6623c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f661c06eac0 0x7f661c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6623c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c15ae40 con 0x7f661c06e6e0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6623c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f661c071440 con 0x7f661c0cf0c0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6623c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f661c071720 con 0x7f661c06eac0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6621200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f661c06e6e0 0x7f661c0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57548/0 (socket says 172.21.15.67:57548) 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6621200640 1 -- 172.21.15.67:0/2685808467 learned_addr learned my addr 172.21.15.67:0/2685808467 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f6613e00640 1 --2- 172.21.15.67:0/2685808467 >> v2:172.21.15.86:3300/0 conn(0x7f661c06eac0 0x7f661c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761975380 0 0) 0x7f661c15ae40 con 0x7f661c06e6e0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6600003530 con 0x7f661c06e6e0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3336132701 0 0) 0x7f6600003530 con 0x7f661c06e6e0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 >> v2:172.21.15.86:3300/0 conn(0x7f661c06eac0 msgr2=0x7f661c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 --2- 172.21.15.67:0/2685808467 >> v2:172.21.15.86:3300/0 conn(0x7f661c06eac0 0x7f661c143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 >> v1:172.21.15.86:6789/0 conn(0x7f661c0cf0c0 legacy=0x7f661c144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c071a00 con 0x7f661c06e6e0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f660c001680 con 0x7f661c06e6e0 2024-04-17T10:32:30.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 >> v1:172.21.15.67:6789/0 conn(0x7f661c06e6e0 legacy=0x7f661c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66000039c0 0x7f6600003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6600004810 0x7f6600004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66000051d0 0x7f660000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c071720 con 0x7f6600004810 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f660000c0d0 con 0x7f66000051d0 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.570+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f660000c2d0 con 0x7f66000039c0 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6613e00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6600004810 0x7f6600004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6621200640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66000039c0 0x7f6600003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6621c00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66000051d0 0x7f660000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6613e00640 1 -- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66000039c0 msgr2=0x7f6600003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6613e00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f66000039c0 0x7f6600003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6613e00640 1 -- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66000051d0 msgr2=0x7f660000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6613e00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66000051d0 0x7f660000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6613e00640 1 -- 172.21.15.67:0/2685808467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6600003530 con 0x7f6600004810 2024-04-17T10:32:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6621c00640 1 -- 172.21.15.67:0/2685808467 reap_dead start 2024-04-17T10:32:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f6613e00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6600004810 0x7f6600004c90 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6604009ae0 tx=0x7f660401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f8b6f7c1882dfd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66040b2070 con 0x7f6600004810 2024-04-17T10:32:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66040a55e0 con 0x7f6600004810 2024-04-17T10:32:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.574+0000 7f661be00640 1 -- 172.21.15.67:0/2685808467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66040096b0 con 0x7f6600004810 2024-04-17T10:32:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- 172.21.15.67:0/2685808467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6600004810 msgr2=0x7f6600004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6600004810 0x7f6600004c90 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f6604009ae0 tx=0x7f660401ca70 comp rx=0 tx=0).stop 2024-04-17T10:32:30.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- 172.21.15.67:0/2685808467 shutdown_connections 2024-04-17T10:32:30.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 --2- 172.21.15.67:0/2685808467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6600004810 0x7f6600004c90 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- 172.21.15.67:0/2685808467 >> 172.21.15.67:0/2685808467 conn(0x7f661c06ac90 msgr2=0x7f661c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:30.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- 172.21.15.67:0/2685808467 shutdown_connections 2024-04-17T10:32:30.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- 172.21.15.67:0/2685808467 wait complete. 2024-04-17T10:32:30.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 Processor -- start 2024-04-17T10:32:30.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- start start 2024-04-17T10:32:30.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6600004810 0x7f661c138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f661c0c6830 0x7f661c0c6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f661c0abb40 0x7f661c0abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0ce9b0 con 0x7f661c0abb40 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c071fc0 con 0x7f661c0c6830 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6623c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661c0ceb30 con 0x7f6600004810 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f661c0abb40 0x7f661c0abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6600004810 0x7f661c138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f661c0abb40 0x7f661c0abfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45838/0 (socket says 172.21.15.67:45838) 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 -- 172.21.15.67:0/1610665840 learned_addr learned my addr 172.21.15.67:0/1610665840 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6613e00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f661c0c6830 0x7f661c0c6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 -- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6600004810 msgr2=0x7f661c138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6600004810 0x7f661c138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 -- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f661c0c6830 msgr2=0x7f661c0c6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f661c0c6830 0x7f661c0c6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 -- 172.21.15.67:0/1610665840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0ac870 con 0x7f661c0abb40 2024-04-17T10:32:30.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.578+0000 7f6621c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f661c0abb40 0x7f661c0abfc0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f661c06bb60 tx=0x7f6614012dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6619600640 1 -- 172.21.15.67:0/1610665840 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614010b20 con 0x7f661c0abb40 2024-04-17T10:32:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6619600640 1 -- 172.21.15.67:0/1610665840 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661401b070 con 0x7f661c0abb40 2024-04-17T10:32:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6619600640 1 -- 172.21.15.67:0/1610665840 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6614016480 con 0x7f661c0abb40 2024-04-17T10:32:30.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c0acb50 con 0x7f661c0abb40 2024-04-17T10:32:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c0c49c0 con 0x7f661c0abb40 2024-04-17T10:32:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c072d50 con 0x7f661c0abb40 2024-04-17T10:32:30.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6619600640 1 -- 172.21.15.67:0/1610665840 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6614016620 con 0x7f661c0abb40 2024-04-17T10:32:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6619600640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65f8066f90 0x7f65f8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6619600640 1 -- 172.21.15.67:0/1610665840 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6614088430 con 0x7f661c0abb40 2024-04-17T10:32:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6621200640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65f8066f90 0x7f65f8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.582+0000 7f6621200640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65f8066f90 0x7f65f8069450 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f661c0adf50 tx=0x7f660c000ff0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.586+0000 7f6619600640 1 -- 172.21.15.67:0/1610665840 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6614058f10 con 0x7f661c0abb40 2024-04-17T10:32:30.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.750+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f661c074e30 con 0x7f661c0abb40 2024-04-17T10:32:30.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6619600640 1 -- 172.21.15.67:0/1610665840 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+13118 (secure 0 0 0) 0x7f661405cfb0 con 0x7f661c0abb40 2024-04-17T10:32:30.744 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:30.745 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":103,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:30.017676+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65f8066f90 msgr2=0x7f65f8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65f8066f90 0x7f65f8069450 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f661c0adf50 tx=0x7f660c000ff0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f661c0abb40 msgr2=0x7f661c0abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f661c0abb40 0x7f661c0abfc0 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f661c06bb60 tx=0x7f6614012dd0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 shutdown_connections 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65f8066f90 0x7f65f8069450 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f661c0abb40 0x7f661c0abfc0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f661c0c6830 0x7f661c0c6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 --2- 172.21.15.67:0/1610665840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6600004810 0x7f661c138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 >> 172.21.15.67:0/1610665840 conn(0x7f661c06ac90 msgr2=0x7f661c0a85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 shutdown_connections 2024-04-17T10:32:30.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.754+0000 7f6623c00640 1 -- 172.21.15.67:0/1610665840 wait complete. 2024-04-17T10:32:30.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:32:30.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:32:30.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.918+0000 7f10d6e00640 1 Processor -- start 2024-04-17T10:32:30.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.918+0000 7f10d6e00640 1 -- start start 2024-04-17T10:32:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.918+0000 7f10d6e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f10d0071af0 0x7f10d00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.918+0000 7f10d6e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d015cb30 con 0x7f10d015be70 2024-04-17T10:32:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.918+0000 7f10d6e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d00cd610 con 0x7f10d0159440 2024-04-17T10:32:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.918+0000 7f10d6e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d00a8650 con 0x7f10d0071af0 2024-04-17T10:32:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f10d015be70 0x7f10d015c2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57556/0 (socket says 172.21.15.67:57556) 2024-04-17T10:32:30.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 -- 172.21.15.67:0/4238786715 learned_addr learned my addr 172.21.15.67:0/4238786715 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10c7400640 1 --2- 172.21.15.67:0/4238786715 >> v2:172.21.15.86:3300/0 conn(0x7f10d0071af0 0x7f10d00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617872970 0 0) 0x7f10d015cb30 con 0x7f10d015be70 2024-04-17T10:32:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b4003530 con 0x7f10d015be70 2024-04-17T10:32:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2161693403 0 0) 0x7f10b4003530 con 0x7f10d015be70 2024-04-17T10:32:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 >> v2:172.21.15.86:3300/0 conn(0x7f10d0071af0 msgr2=0x7f10d00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:30.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 --2- 172.21.15.67:0/4238786715 >> v2:172.21.15.86:3300/0 conn(0x7f10d0071af0 0x7f10d00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 >> v1:172.21.15.86:6789/0 conn(0x7f10d0159440 legacy=0x7f10d00713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d00cd8f0 con 0x7f10d015be70 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f10c8001a00 con 0x7f10d015be70 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 >> v1:172.21.15.67:6789/0 conn(0x7f10d015be70 legacy=0x7f10d015c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b40039c0 0x7f10b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b4004810 0x7f10b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b400c040 con 0x7f10b40051d0 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b400c1f0 con 0x7f10b40039c0 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b400c3f0 con 0x7f10b4004810 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cfe00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b40039c0 0x7f10b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10c7400640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b4004810 0x7f10b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 -- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b4004810 msgr2=0x7f10b4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b4004810 0x7f10b4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 -- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b40039c0 msgr2=0x7f10b4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b40039c0 0x7f10b4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10d4e00640 1 -- 172.21.15.67:0/4238786715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4003530 con 0x7f10b40051d0 2024-04-17T10:32:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10cfe00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10b40039c0 0x7f10b4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:30.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.922+0000 7f10c7400640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10b4004810 0x7f10b4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d4e00640 1 -- 172.21.15.67:0/4238786715 reap_dead start 2024-04-17T10:32:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d4e00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10b400b9d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f10c800a840 tx=0x7f10c801d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f6d24da6f9dedd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00013a0 con 0x7f10b40051d0 2024-04-17T10:32:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c80ac640 con 0x7f10b40051d0 2024-04-17T10:32:30.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10cf400640 1 -- 172.21.15.67:0/4238786715 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b80096b0 con 0x7f10b40051d0 2024-04-17T10:32:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 -- 172.21.15.67:0/4238786715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 msgr2=0x7f10b400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10b400b9d0 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f10c800a840 tx=0x7f10c801d360 comp rx=0 tx=0).stop 2024-04-17T10:32:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 -- 172.21.15.67:0/4238786715 shutdown_connections 2024-04-17T10:32:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 --2- 172.21.15.67:0/4238786715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10b400b9d0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 -- 172.21.15.67:0/4238786715 >> 172.21.15.67:0/4238786715 conn(0x7f10d00b1740 msgr2=0x7f10d00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 -- 172.21.15.67:0/4238786715 shutdown_connections 2024-04-17T10:32:30.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 -- 172.21.15.67:0/4238786715 wait complete. 2024-04-17T10:32:30.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 Processor -- start 2024-04-17T10:32:30.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.926+0000 7f10d6e00640 1 -- start start 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10d00b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d00b2970 0x7f10d014f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10d00c47e0 0x7f10d00adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d0159820 con 0x7f10b40051d0 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d015ce10 con 0x7f10d00b2970 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d00ce0f0 con 0x7f10d00c47e0 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10d00b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10d00b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45856/0 (socket says 172.21.15.67:45856) 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 -- 172.21.15.67:0/888878550 learned_addr learned my addr 172.21.15.67:0/888878550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d4e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10d00c47e0 0x7f10d00adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10c7400640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d00b2970 0x7f10d014f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 -- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10d00c47e0 msgr2=0x7f10d00adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10d00c47e0 0x7f10d00adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 -- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d00b2970 msgr2=0x7f10d014f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d00b2970 0x7f10d014f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 -- 172.21.15.67:0/888878550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d014f840 con 0x7f10b40051d0 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10c7400640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d00b2970 0x7f10d014f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d4e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10d00c47e0 0x7f10d00adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:30.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10cfe00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10d00b3f10 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f10c000f5f0 tx=0x7f10c000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10ccc00640 1 -- 172.21.15.67:0/888878550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c00052e0 con 0x7f10b40051d0 2024-04-17T10:32:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10ccc00640 1 -- 172.21.15.67:0/888878550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c0018070 con 0x7f10b40051d0 2024-04-17T10:32:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10ccc00640 1 -- 172.21.15.67:0/888878550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c0003820 con 0x7f10b40051d0 2024-04-17T10:32:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d00b2e70 con 0x7f10b40051d0 2024-04-17T10:32:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d00b3430 con 0x7f10b40051d0 2024-04-17T10:32:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d004ef30 con 0x7f10b40051d0 2024-04-17T10:32:30.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.930+0000 7f10ccc00640 1 -- 172.21.15.67:0/888878550 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f10c00039c0 con 0x7f10b40051d0 2024-04-17T10:32:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.934+0000 7f10ccc00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f10ac066f60 0x7f10ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.934+0000 7f10ccc00640 1 -- 172.21.15.67:0/888878550 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f10c0084c80 con 0x7f10b40051d0 2024-04-17T10:32:30.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.934+0000 7f10c7400640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f10ac066f60 0x7f10ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:30.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.934+0000 7f10c7400640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f10ac066f60 0x7f10ac069420 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f10d00cab40 tx=0x7f10b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:30.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:30.938+0000 7f10ccc00640 1 -- 172.21.15.67:0/888878550 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f10c0055e40 con 0x7f10b40051d0 2024-04-17T10:32:31.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.110+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10d006ddd0 con 0x7f10b40051d0 2024-04-17T10:32:31.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.110+0000 7f10ccc00640 1 -- 172.21.15.67:0/888878550 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10c0055740 con 0x7f10b40051d0 2024-04-17T10:32:31.103 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f10ac066f60 msgr2=0x7f10ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f10ac066f60 0x7f10ac069420 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f10d00cab40 tx=0x7f10b8008040 comp rx=0 tx=0).stop 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 msgr2=0x7f10d00b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10d00b3f10 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f10c000f5f0 tx=0x7f10c000fab0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 shutdown_connections 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f10ac066f60 0x7f10ac069420 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f10d00c47e0 0x7f10d00adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f10d00b2970 0x7f10d014f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 --2- 172.21.15.67:0/888878550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f10b40051d0 0x7f10d00b3f10 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 >> 172.21.15.67:0/888878550 conn(0x7f10d00b1740 msgr2=0x7f10d00cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 shutdown_connections 2024-04-17T10:32:31.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.114+0000 7f10d6e00640 1 -- 172.21.15.67:0/888878550 wait complete. 2024-04-17T10:32:31.124 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:32:31.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f644b400640 1 Processor -- start 2024-04-17T10:32:31.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f644b400640 1 -- start start 2024-04-17T10:32:31.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f644b400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f64441446c0 0x7f6444144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f644b400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64441456d0 con 0x7f6444144aa0 2024-04-17T10:32:31.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f644b400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64440b4870 con 0x7f6444071840 2024-04-17T10:32:31.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f644b400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64440b51a0 con 0x7f64441446c0 2024-04-17T10:32:31.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f643be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6444144aa0 0x7f64440d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57560/0 (socket says 172.21.15.67:57560) 2024-04-17T10:32:31.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f643be00640 1 -- 172.21.15.67:0/718279003 learned_addr learned my addr 172.21.15.67:0/718279003 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:31.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6448a00640 1 --2- 172.21.15.67:0/718279003 >> v2:172.21.15.86:3300/0 conn(0x7f64441446c0 0x7f6444144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733109150 0 0) 0x7f64441456d0 con 0x7f6444144aa0 2024-04-17T10:32:31.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6428003530 con 0x7f6444144aa0 2024-04-17T10:32:31.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031777937 0 0) 0x7f64440b4870 con 0x7f6444071840 2024-04-17T10:32:31.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64441456d0 con 0x7f6444071840 2024-04-17T10:32:31.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257142674 0 0) 0x7f6428003530 con 0x7f6444144aa0 2024-04-17T10:32:31.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6448a00640 1 -- 172.21.15.67:0/718279003 >> v1:172.21.15.86:6789/0 conn(0x7f6444071840 legacy=0x7f64440b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6448a00640 1 -- 172.21.15.67:0/718279003 >> v1:172.21.15.67:6789/0 conn(0x7f6444144aa0 legacy=0x7f64440d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.274+0000 7f6448a00640 1 -- 172.21.15.67:0/718279003 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440b4b50 con 0x7f64441446c0 2024-04-17T10:32:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f6448a00640 1 --2- 172.21.15.67:0/718279003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64441446c0 0x7f6444144ea0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f64340097c0 tx=0x7f643401c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e69585392a799aa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64340b0070 con 0x7f64441446c0 2024-04-17T10:32:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64340ab400 con 0x7f64441446c0 2024-04-17T10:32:31.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f6443e00640 1 -- 172.21.15.67:0/718279003 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c001680 con 0x7f64441446c0 2024-04-17T10:32:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f644b400640 1 -- 172.21.15.67:0/718279003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64441446c0 msgr2=0x7f6444144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f644b400640 1 --2- 172.21.15.67:0/718279003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64441446c0 0x7f6444144ea0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f64340097c0 tx=0x7f643401c390 comp rx=0 tx=0).stop 2024-04-17T10:32:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f644b400640 1 -- 172.21.15.67:0/718279003 shutdown_connections 2024-04-17T10:32:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f644b400640 1 --2- 172.21.15.67:0/718279003 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64441446c0 0x7f6444144ea0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f644b400640 1 -- 172.21.15.67:0/718279003 >> 172.21.15.67:0/718279003 conn(0x7f64440b34d0 msgr2=0x7f64440b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f644b400640 1 -- 172.21.15.67:0/718279003 shutdown_connections 2024-04-17T10:32:31.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.278+0000 7f644b400640 1 -- 172.21.15.67:0/718279003 wait complete. 2024-04-17T10:32:31.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 Processor -- start 2024-04-17T10:32:31.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 -- start start 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6444071840 0x7f64440abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64441446c0 0x7f644413ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6444144aa0 0x7f644413d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6444169bd0 con 0x7f64441446c0 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6444169a50 con 0x7f6444071840 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6444169d50 con 0x7f6444144aa0 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64441446c0 0x7f644413ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64441446c0 0x7f644413ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45864/0 (socket says 172.21.15.67:45864) 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 -- 172.21.15.67:0/115841146 learned_addr learned my addr 172.21.15.67:0/115841146 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:31.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f6448a00640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6444071840 0x7f64440abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f6449400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6444144aa0 0x7f644413d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 -- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6444144aa0 msgr2=0x7f644413d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6444144aa0 0x7f644413d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 -- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6444071840 msgr2=0x7f64440abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6444071840 0x7f64440abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 -- 172.21.15.67:0/115841146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64440aa0d0 con 0x7f64441446c0 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f6448a00640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6444071840 0x7f64440abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f6449400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6444144aa0 0x7f644413d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f643be00640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64441446c0 0x7f644413ce30 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f642c00c730 tx=0x7f642c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f6441600640 1 -- 172.21.15.67:0/115841146 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c017070 con 0x7f64441446c0 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f6441600640 1 -- 172.21.15.67:0/115841146 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642c005b00 con 0x7f64441446c0 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64440aa3b0 con 0x7f64441446c0 2024-04-17T10:32:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f6441600640 1 -- 172.21.15.67:0/115841146 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642c007070 con 0x7f64441446c0 2024-04-17T10:32:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.282+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64440aa970 con 0x7f64441446c0 2024-04-17T10:32:31.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.286+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6444070bb0 con 0x7f64441446c0 2024-04-17T10:32:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.286+0000 7f6441600640 1 -- 172.21.15.67:0/115841146 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f642c01a070 con 0x7f64441446c0 2024-04-17T10:32:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.286+0000 7f6441600640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6420066f60 0x7f6420069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.286+0000 7f6441600640 1 -- 172.21.15.67:0/115841146 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f642c0843b0 con 0x7f64441446c0 2024-04-17T10:32:31.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.286+0000 7f6448a00640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6420066f60 0x7f6420069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.286+0000 7f6448a00640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6420066f60 0x7f6420069420 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f64340ad000 tx=0x7f6434008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:31.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.290+0000 7f6441600640 1 -- 172.21.15.67:0/115841146 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f642c0554e0 con 0x7f64441446c0 2024-04-17T10:32:31.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f6444074e30 con 0x7f64441446c0 2024-04-17T10:32:31.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f6441600640 1 -- 172.21.15.67:0/115841146 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v103) v1 ==== 84+0+65 (secure 0 0 0) 0x7f642c082e20 con 0x7f64441446c0 2024-04-17T10:32:31.445 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:31.445 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:32:31.445 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:32:31.445 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:32:31.445 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:32:31.446 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:32:31.446 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:32:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6420066f60 msgr2=0x7f6420069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6420066f60 0x7f6420069420 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f64340ad000 tx=0x7f6434008040 comp rx=0 tx=0).stop 2024-04-17T10:32:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64441446c0 msgr2=0x7f644413ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64441446c0 0x7f644413ce30 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f642c00c730 tx=0x7f642c00cbf0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 shutdown_connections 2024-04-17T10:32:31.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6420066f60 0x7f6420069420 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6444144aa0 0x7f644413d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64441446c0 0x7f644413ce30 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 --2- 172.21.15.67:0/115841146 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6444071840 0x7f64440abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 >> 172.21.15.67:0/115841146 conn(0x7f64440b34d0 msgr2=0x7f644415d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 shutdown_connections 2024-04-17T10:32:31.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.454+0000 7f644b400640 1 -- 172.21.15.67:0/115841146 wait complete. 2024-04-17T10:32:31.467 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:32:31.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:32:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.618+0000 7f2c60400640 1 Processor -- start 2024-04-17T10:32:31.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.618+0000 7f2c60400640 1 -- start start 2024-04-17T10:32:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c60400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c581447e0 0x7f2c5806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c60400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c58071a70 0x7f2c580cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c60400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5806d9a0 con 0x7f2c58144f00 2024-04-17T10:32:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c60400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c580cc310 con 0x7f2c58071a70 2024-04-17T10:32:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c60400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5806dc80 con 0x7f2c581447e0 2024-04-17T10:32:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c58071a70 0x7f2c580cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2c58071a70 0x7f2c580cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45880/0 (socket says 172.21.15.67:45880) 2024-04-17T10:32:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5e400640 1 -- 172.21.15.67:0/590620789 learned_addr learned my addr 172.21.15.67:0/590620789 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5da00640 1 --2- 172.21.15.67:0/590620789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c581447e0 0x7f2c5806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5e400640 1 -- 172.21.15.67:0/590620789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c581447e0 msgr2=0x7f2c5806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5e400640 1 --2- 172.21.15.67:0/590620789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c581447e0 0x7f2c5806d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5e400640 1 -- 172.21.15.67:0/590620789 >> v1:172.21.15.86:6789/0 conn(0x7f2c58144f00 legacy=0x7f2c58071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:31.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.622+0000 7f2c5e400640 1 -- 172.21.15.67:0/590620789 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5806df60 con 0x7f2c58071a70 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c5e400640 1 --2- 172.21.15.67:0/590620789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580cc7a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009e50 tx=0x7f2c5001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728f06f5cde9a2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c57e00640 1 -- 172.21.15.67:0/590620789 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c500a57d0 con 0x7f2c58071a70 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c57e00640 1 -- 172.21.15.67:0/590620789 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c500a5970 con 0x7f2c58071a70 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c57e00640 1 -- 172.21.15.67:0/590620789 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c500a5d40 con 0x7f2c58071a70 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- 172.21.15.67:0/590620789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 msgr2=0x7f2c580cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 --2- 172.21.15.67:0/590620789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580cc7a0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f2c50009e50 tx=0x7f2c5001c930 comp rx=0 tx=0).stop 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- 172.21.15.67:0/590620789 shutdown_connections 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 --2- 172.21.15.67:0/590620789 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580cc7a0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 --2- 172.21.15.67:0/590620789 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c581447e0 0x7f2c5806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- 172.21.15.67:0/590620789 >> 172.21.15.67:0/590620789 conn(0x7f2c5806ac90 msgr2=0x7f2c5806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:31.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- 172.21.15.67:0/590620789 shutdown_connections 2024-04-17T10:32:31.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- 172.21.15.67:0/590620789 wait complete. 2024-04-17T10:32:31.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 Processor -- start 2024-04-17T10:32:31.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- start start 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c581447e0 0x7f2c58132b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c58144f00 0x7f2c5815aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580cd880 con 0x7f2c58071a70 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.626+0000 7f2c60400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580cd700 con 0x7f2c581447e0 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c60400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580cda00 con 0x7f2c58144f00 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580acb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45890/0 (socket says 172.21.15.67:45890) 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 -- 172.21.15.67:0/1743379656 learned_addr learned my addr 172.21.15.67:0/1743379656 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5e400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c58144f00 0x7f2c5815aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5d000640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c581447e0 0x7f2c58132b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 -- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c58144f00 msgr2=0x7f2c5815aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c58144f00 0x7f2c5815aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 -- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c581447e0 msgr2=0x7f2c58132b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c581447e0 0x7f2c58132b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 -- 172.21.15.67:0/1743379656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c5815afe0 con 0x7f2c58071a70 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5e400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c58144f00 0x7f2c5815aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c5da00640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580acb60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d830 tx=0x7f2c4c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c55600640 1 -- 172.21.15.67:0/1743379656 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00b6c0 con 0x7f2c58071a70 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c55600640 1 -- 172.21.15.67:0/1743379656 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c019070 con 0x7f2c58071a70 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c580ad190 con 0x7f2c58071a70 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c55600640 1 -- 172.21.15.67:0/1743379656 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4c00ba60 con 0x7f2c58071a70 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c580ad750 con 0x7f2c58071a70 2024-04-17T10:32:31.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c58072da0 con 0x7f2c58071a70 2024-04-17T10:32:31.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.630+0000 7f2c55600640 1 -- 172.21.15.67:0/1743379656 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2c4c01e460 con 0x7f2c58071a70 2024-04-17T10:32:31.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.634+0000 7f2c55600640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c38066f60 0x7f2c38069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:31.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.634+0000 7f2c55600640 1 -- 172.21.15.67:0/1743379656 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f2c4c0855f0 con 0x7f2c58071a70 2024-04-17T10:32:31.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.634+0000 7f2c5d000640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c38066f60 0x7f2c38069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:31.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.634+0000 7f2c5d000640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c38066f60 0x7f2c38069420 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2c58136070 tx=0x7f2c44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:31.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.634+0000 7f2c55600640 1 -- 172.21.15.67:0/1743379656 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2c5815afe0 con 0x7f2c58071a70 2024-04-17T10:32:31.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:31.798+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2c5806b1a0 con 0x7f2c58071a70 2024-04-17T10:32:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c55600640 1 -- 172.21.15.67:0/1743379656 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v104) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2c4c056720 con 0x7f2c58071a70 2024-04-17T10:32:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:32:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c38066f60 msgr2=0x7f2c38069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c38066f60 0x7f2c38069420 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2c58136070 tx=0x7f2c44009040 comp rx=0 tx=0).stop 2024-04-17T10:32:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 msgr2=0x7f2c580acb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580acb60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2c4c00d830 tx=0x7f2c4c00dcf0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 shutdown_connections 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2c38066f60 0x7f2c38069420 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2c58144f00 0x7f2c5815aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2c581447e0 0x7f2c58132b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 --2- 172.21.15.67:0/1743379656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2c58071a70 0x7f2c580acb60 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.034+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 >> 172.21.15.67:0/1743379656 conn(0x7f2c5806ac90 msgr2=0x7f2c5813c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.038+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 shutdown_connections 2024-04-17T10:32:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.038+0000 7f2c60400640 1 -- 172.21.15.67:0/1743379656 wait complete. 2024-04-17T10:32:32.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:32:32.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd03e00640 1 Processor -- start 2024-04-17T10:32:32.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd03e00640 1 -- start start 2024-04-17T10:32:32.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd03e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 0x7fccfc161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd03e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccfc15e140 con 0x7fccfc09adf0 2024-04-17T10:32:32.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd03e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccfc099d50 con 0x7fccfc0d06a0 2024-04-17T10:32:32.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd03e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc1626d0 con 0x7fccfc15dd60 2024-04-17T10:32:32.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd00a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fccfc09adf0 0x7fccfc0cff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57562/0 (socket says 172.21.15.67:57562) 2024-04-17T10:32:32.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd00a00640 1 -- 172.21.15.67:0/3577177333 learned_addr learned my addr 172.21.15.67:0/3577177333 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd01400640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 0x7fccfc161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214742478 0 0) 0x7fccfc15e140 con 0x7fccfc09adf0 2024-04-17T10:32:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce4003530 con 0x7fccfc09adf0 2024-04-17T10:32:32.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3817502412 0 0) 0x7fcce4003530 con 0x7fccfc09adf0 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 msgr2=0x7fccfc161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 0x7fccfc161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 >> v1:172.21.15.86:6789/0 conn(0x7fccfc0d06a0 legacy=0x7fccfc06bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc09a030 con 0x7fccfc09adf0 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fcce8001680 con 0x7fccfc09adf0 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 >> v1:172.21.15.67:6789/0 conn(0x7fccfc09adf0 legacy=0x7fccfc0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 0x7fcce40039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcce4003f30 0x7fcce4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fcce40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc1626d0 con 0x7fcce4004d50 2024-04-17T10:32:32.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400ba30 con 0x7fcce4003f30 2024-04-17T10:32:32.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce400bc30 con 0x7fccfc15dd60 2024-04-17T10:32:32.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd01e00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fcce40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd00a00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcce4003f30 0x7fcce4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.210+0000 7fcd01400640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 0x7fcce40039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd01e00640 1 -- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 msgr2=0x7fcce40039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:32.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd01e00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc15dd60 0x7fcce40039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd01e00640 1 -- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcce4003f30 msgr2=0x7fcce4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:32.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd01e00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcce4003f30 0x7fcce4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd01e00640 1 -- 172.21.15.67:0/3577177333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce4003530 con 0x7fcce4004d50 2024-04-17T10:32:32.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd01e00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fcce40051d0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fccf4001f60 tx=0x7fccf401c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9361ff59c029839 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:32.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf4001b60 con 0x7fcce4004d50 2024-04-17T10:32:32.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf40a66f0 con 0x7fcce4004d50 2024-04-17T10:32:32.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fccfbe00640 1 -- 172.21.15.67:0/3577177333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf40a6b20 con 0x7fcce4004d50 2024-04-17T10:32:32.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd03e00640 1 -- 172.21.15.67:0/3577177333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 msgr2=0x7fcce40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:32.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd03e00640 1 --2- 172.21.15.67:0/3577177333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fcce40051d0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fccf4001f60 tx=0x7fccf401c880 comp rx=0 tx=0).stop 2024-04-17T10:32:32.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd01e00640 1 -- 172.21.15.67:0/3577177333 reap_dead start 2024-04-17T10:32:32.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd03e00640 1 -- 172.21.15.67:0/3577177333 shutdown_connections 2024-04-17T10:32:32.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd03e00640 1 -- 172.21.15.67:0/3577177333 >> 172.21.15.67:0/3577177333 conn(0x7fccfc0b34a0 msgr2=0x7fccfc0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:32.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd03e00640 1 -- 172.21.15.67:0/3577177333 shutdown_connections 2024-04-17T10:32:32.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.214+0000 7fcd03e00640 1 -- 172.21.15.67:0/3577177333 wait complete. 2024-04-17T10:32:32.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 Processor -- start 2024-04-17T10:32:32.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 -- start start 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fccfc160620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc0cb760 0x7fccfc157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccfc0c54c0 0x7fccfc0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc09a8d0 con 0x7fcce4004d50 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc09a5f0 con 0x7fccfc0c54c0 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccfc06c4b0 con 0x7fccfc0cb760 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fccfc160620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fccfc160620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45916/0 (socket says 172.21.15.67:45916) 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 -- 172.21.15.67:0/2063726237 learned_addr learned my addr 172.21.15.67:0/2063726237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd00a00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc0cb760 0x7fccfc157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccfc0c54c0 0x7fccfc0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 -- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc0cb760 msgr2=0x7fccfc157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc0cb760 0x7fccfc157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 -- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccfc0c54c0 msgr2=0x7fccfc0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccfc0c54c0 0x7fccfc0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 -- 172.21.15.67:0/2063726237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc0cf990 con 0x7fcce4004d50 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccfc0c54c0 0x7fccfc0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd01400640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fccfc160620 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fccf00119a0 tx=0x7fccf0011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fccf9600640 1 -- 172.21.15.67:0/2063726237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000f9d0 con 0x7fcce4004d50 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fccf9600640 1 -- 172.21.15.67:0/2063726237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf000fb70 con 0x7fcce4004d50 2024-04-17T10:32:32.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fccf9600640 1 -- 172.21.15.67:0/2063726237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0014440 con 0x7fcce4004d50 2024-04-17T10:32:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccfc161150 con 0x7fcce4004d50 2024-04-17T10:32:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.218+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccfc0c9910 con 0x7fcce4004d50 2024-04-17T10:32:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.222+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccfc04ef30 con 0x7fcce4004d50 2024-04-17T10:32:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.222+0000 7fccf9600640 1 -- 172.21.15.67:0/2063726237 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fccf0012040 con 0x7fcce4004d50 2024-04-17T10:32:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.222+0000 7fccf9600640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccdc066f90 0x7fccdc069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:32.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.222+0000 7fccf9600640 1 -- 172.21.15.67:0/2063726237 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5619+0+0 (secure 0 0 0) 0x7fccf0085cb0 con 0x7fcce4004d50 2024-04-17T10:32:32.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.222+0000 7fcd00a00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccdc066f90 0x7fccdc069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:32.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.226+0000 7fcd00a00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccdc066f90 0x7fccdc069450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fccfc15d800 tx=0x7fcce8001c70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:32.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.226+0000 7fccf9600640 1 -- 172.21.15.67:0/2063726237 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fccf002b020 con 0x7fcce4004d50 2024-04-17T10:32:32.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:32.390+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fccfc0b2be0 con 0x7fcce4004d50 2024-04-17T10:32:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fccf9600640 1 -- 172.21.15.67:0/2063726237 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v105) v1 ==== 170+0+0 (secure 0 0 0) 0x7fccf0019030 con 0x7fcce4004d50 2024-04-17T10:32:33.033 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:32:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccdc066f90 msgr2=0x7fccdc069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccdc066f90 0x7fccdc069450 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fccfc15d800 tx=0x7fcce8001c70 comp rx=0 tx=0).stop 2024-04-17T10:32:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 msgr2=0x7fccfc160620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fccfc160620 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fccf00119a0 tx=0x7fccf0011da0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 shutdown_connections 2024-04-17T10:32:33.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccdc066f90 0x7fccdc069450 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fccfc0c54c0 0x7fccfc0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fccfc0cb760 0x7fccfc157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 --2- 172.21.15.67:0/2063726237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcce4004d50 0x7fccfc160620 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.042+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 >> 172.21.15.67:0/2063726237 conn(0x7fccfc0b34a0 msgr2=0x7fccfc06bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:33.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.046+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 shutdown_connections 2024-04-17T10:32:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.046+0000 7fcd03e00640 1 -- 172.21.15.67:0/2063726237 wait complete. 2024-04-17T10:32:33.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:32:33.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.230+0000 7fc646e00640 1 Processor -- start 2024-04-17T10:32:33.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.230+0000 7fc646e00640 1 -- start start 2024-04-17T10:32:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.230+0000 7fc646e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6400722b0 0x7fc64015a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.230+0000 7fc646e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc64015e210 0x7fc64015e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.230+0000 7fc646e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc64015ac40 con 0x7fc640071ed0 2024-04-17T10:32:33.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.230+0000 7fc646e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6400ccbf0 con 0x7fc6400722b0 2024-04-17T10:32:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.230+0000 7fc646e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc64015af20 con 0x7fc64015e210 2024-04-17T10:32:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc644e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc64015e210 0x7fc64015e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc637e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6400722b0 0x7fc64015a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc63fe00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc640071ed0 0x7fc6400aeda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:41672/0 (socket says 172.21.15.67:41672) 2024-04-17T10:32:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc63fe00640 1 -- 172.21.15.67:0/1603865391 learned_addr learned my addr 172.21.15.67:0/1603865391 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc63f400640 1 -- 172.21.15.67:0/1603865391 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147091337 0 0) 0x7fc64015ac40 con 0x7fc640071ed0 2024-04-17T10:32:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc63f400640 1 -- 172.21.15.67:0/1603865391 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc624003530 con 0x7fc640071ed0 2024-04-17T10:32:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc637e00640 1 -- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc64015e210 msgr2=0x7fc64015e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc637e00640 1 --2- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc64015e210 0x7fc64015e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc637e00640 1 -- 172.21.15.67:0/1603865391 >> v1:172.21.15.86:6789/0 conn(0x7fc640071ed0 legacy=0x7fc6400aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc637e00640 1 -- 172.21.15.67:0/1603865391 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64015b200 con 0x7fc6400722b0 2024-04-17T10:32:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc644e00640 1 --2- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc64015e210 0x7fc64015e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc637e00640 1 --2- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6400722b0 0x7fc64015a560 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc6280097c0 tx=0x7fc62801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e76e79ca6e5b1e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc63f400640 1 -- 172.21.15.67:0/1603865391 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280b0070 con 0x7fc6400722b0 2024-04-17T10:32:33.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc63f400640 1 -- 172.21.15.67:0/1603865391 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6280ab600 con 0x7fc6400722b0 2024-04-17T10:32:33.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.234+0000 7fc63f400640 1 -- 172.21.15.67:0/1603865391 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6280ab9d0 con 0x7fc6400722b0 2024-04-17T10:32:33.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6400722b0 msgr2=0x7fc64015a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:33.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 --2- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6400722b0 0x7fc64015a560 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fc6280097c0 tx=0x7fc62801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- 172.21.15.67:0/1603865391 shutdown_connections 2024-04-17T10:32:33.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 --2- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc64015e210 0x7fc64015e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 --2- 172.21.15.67:0/1603865391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6400722b0 0x7fc64015a560 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- 172.21.15.67:0/1603865391 >> 172.21.15.67:0/1603865391 conn(0x7fc6400b1740 msgr2=0x7fc6400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- 172.21.15.67:0/1603865391 shutdown_connections 2024-04-17T10:32:33.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- 172.21.15.67:0/1603865391 wait complete. 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 Processor -- start 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- start start 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc640071ed0 0x7fc640145820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6400722b0 0x7fc6400c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc64015e210 0x7fc6400ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400ce010 con 0x7fc640071ed0 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc640151330 con 0x7fc64015e210 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc646e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400ce190 con 0x7fc6400722b0 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc640071ed0 0x7fc640145820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc640071ed0 0x7fc640145820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45922/0 (socket says 172.21.15.67:45922) 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 -- 172.21.15.67:0/2174987131 learned_addr learned my addr 172.21.15.67:0/2174987131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:33.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc637e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6400722b0 0x7fc6400c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc644e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc64015e210 0x7fc6400ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 -- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6400722b0 msgr2=0x7fc6400c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6400722b0 0x7fc6400c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 -- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc64015e210 msgr2=0x7fc6400ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc64015e210 0x7fc6400ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc63fe00640 1 -- 172.21.15.67:0/2174987131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6400c5b60 con 0x7fc640071ed0 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc637e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6400722b0 0x7fc6400c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.238+0000 7fc644e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc64015e210 0x7fc6400ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:33.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc63fe00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc640071ed0 0x7fc640145820 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c450 tx=0x7fc63000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:33.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc63cc00640 1 -- 172.21.15.67:0/2174987131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630017070 con 0x7fc640071ed0 2024-04-17T10:32:33.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc63cc00640 1 -- 172.21.15.67:0/2174987131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630005cf0 con 0x7fc640071ed0 2024-04-17T10:32:33.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc63cc00640 1 -- 172.21.15.67:0/2174987131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630007290 con 0x7fc640071ed0 2024-04-17T10:32:33.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6400c5e40 con 0x7fc640071ed0 2024-04-17T10:32:33.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6400c6400 con 0x7fc640071ed0 2024-04-17T10:32:33.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64004ef30 con 0x7fc640071ed0 2024-04-17T10:32:33.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc63cc00640 1 -- 172.21.15.67:0/2174987131 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc63001a070 con 0x7fc640071ed0 2024-04-17T10:32:33.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc63cc00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc61c066f60 0x7fc61c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:33.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc63cc00640 1 -- 172.21.15.67:0/2174987131 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fc630084500 con 0x7fc640071ed0 2024-04-17T10:32:33.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.242+0000 7fc637e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc61c066f60 0x7fc61c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:33.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.246+0000 7fc637e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc61c066f60 0x7fc61c069420 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc6280ad000 tx=0x7fc628008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:33.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.250+0000 7fc63cc00640 1 -- 172.21.15.67:0/2174987131 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc630054cf0 con 0x7fc640071ed0 2024-04-17T10:32:33.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:33.446+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc64006ef80 con 0x7fc640071ed0 2024-04-17T10:32:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.042+0000 7fc63cc00640 1 -- 172.21.15.67:0/2174987131 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v69) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc630058d90 con 0x7fc640071ed0 2024-04-17T10:32:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 15 and data pool 16 2024-04-17T10:32:34.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc61c066f60 msgr2=0x7fc61c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc61c066f60 0x7fc61c069420 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc6280ad000 tx=0x7fc628008040 comp rx=0 tx=0).stop 2024-04-17T10:32:34.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc640071ed0 msgr2=0x7fc640145820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc640071ed0 0x7fc640145820 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fc63000c450 tx=0x7fc63000c910 comp rx=0 tx=0).stop 2024-04-17T10:32:34.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 shutdown_connections 2024-04-17T10:32:34.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc61c066f60 0x7fc61c069420 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc64015e210 0x7fc6400ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6400722b0 0x7fc6400c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 --2- 172.21.15.67:0/2174987131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc640071ed0 0x7fc640145820 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 >> 172.21.15.67:0/2174987131 conn(0x7fc6400b1740 msgr2=0x7fc6400b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:34.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 shutdown_connections 2024-04-17T10:32:34.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.046+0000 7fc646e00640 1 -- 172.21.15.67:0/2174987131 wait complete. 2024-04-17T10:32:34.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:34.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec35400640 1 Processor -- start 2024-04-17T10:32:34.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec35400640 1 -- start start 2024-04-17T10:32:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec35400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec30071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec35400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec300cc300 0x7fec300cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec35400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3006e2f0 con 0x7fec30145040 2024-04-17T10:32:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec35400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec300a86c0 con 0x7fec300cc300 2024-04-17T10:32:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec35400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec30071cb0 con 0x7fec30071310 2024-04-17T10:32:34.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec2e800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fec30145040 0x7fec3006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57578/0 (socket says 172.21.15.67:57578) 2024-04-17T10:32:34.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec2e800640 1 -- 172.21.15.67:0/553462604 learned_addr learned my addr 172.21.15.67:0/553462604 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:34.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec27e00640 1 --2- 172.21.15.67:0/553462604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec30071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.218+0000 7fec2f200640 1 --2- 172.21.15.67:0/553462604 >> v2:172.21.15.86:3300/0 conn(0x7fec300cc300 0x7fec300cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec2de00640 1 -- 172.21.15.67:0/553462604 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677442118 0 0) 0x7fec3006e2f0 con 0x7fec30145040 2024-04-17T10:32:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec2de00640 1 -- 172.21.15.67:0/553462604 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec14003530 con 0x7fec30145040 2024-04-17T10:32:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec27e00640 1 -- 172.21.15.67:0/553462604 >> v2:172.21.15.86:3300/0 conn(0x7fec300cc300 msgr2=0x7fec300cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec27e00640 1 --2- 172.21.15.67:0/553462604 >> v2:172.21.15.86:3300/0 conn(0x7fec300cc300 0x7fec300cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec27e00640 1 -- 172.21.15.67:0/553462604 >> v1:172.21.15.67:6789/0 conn(0x7fec30145040 legacy=0x7fec3006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec27e00640 1 -- 172.21.15.67:0/553462604 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3006d6e0 con 0x7fec30071310 2024-04-17T10:32:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec2f200640 1 --2- 172.21.15.67:0/553462604 >> v2:172.21.15.86:3300/0 conn(0x7fec300cc300 0x7fec300cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:34.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec27e00640 1 --2- 172.21.15.67:0/553462604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec30071770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fec180097c0 tx=0x7fec1801ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aae694396483019d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec2de00640 1 -- 172.21.15.67:0/553462604 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec180b5070 con 0x7fec30071310 2024-04-17T10:32:34.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec2de00640 1 -- 172.21.15.67:0/553462604 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec180a8990 con 0x7fec30071310 2024-04-17T10:32:34.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.222+0000 7fec2de00640 1 -- 172.21.15.67:0/553462604 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec180a8d60 con 0x7fec30071310 2024-04-17T10:32:34.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- 172.21.15.67:0/553462604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 msgr2=0x7fec30071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 --2- 172.21.15.67:0/553462604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec30071770 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fec180097c0 tx=0x7fec1801ec90 comp rx=0 tx=0).stop 2024-04-17T10:32:34.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- 172.21.15.67:0/553462604 shutdown_connections 2024-04-17T10:32:34.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 --2- 172.21.15.67:0/553462604 >> v2:172.21.15.86:3300/0 conn(0x7fec300cc300 0x7fec300cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 --2- 172.21.15.67:0/553462604 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec30071770 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- 172.21.15.67:0/553462604 >> 172.21.15.67:0/553462604 conn(0x7fec3006ac90 msgr2=0x7fec3006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:34.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- 172.21.15.67:0/553462604 shutdown_connections 2024-04-17T10:32:34.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- 172.21.15.67:0/553462604 wait complete. 2024-04-17T10:32:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 Processor -- start 2024-04-17T10:32:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- start start 2024-04-17T10:32:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec3015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec300cc300 0x7fec3013b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30145040 0x7fec30139380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3006dca0 con 0x7fec30145040 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3015bf10 con 0x7fec300cc300 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec35400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3006df80 con 0x7fec30071310 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30145040 0x7fec30139380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30145040 0x7fec30139380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45938/0 (socket says 172.21.15.67:45938) 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 -- 172.21.15.67:0/3665993918 learned_addr learned my addr 172.21.15.67:0/3665993918 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2e800640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec3015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec27e00640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec300cc300 0x7fec3013b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 -- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 msgr2=0x7fec3015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec3015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 -- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec300cc300 msgr2=0x7fec3013b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec300cc300 0x7fec3013b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.226+0000 7fec2f200640 1 -- 172.21.15.67:0/3665993918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec30160410 con 0x7fec30145040 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec2f200640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30145040 0x7fec30139380 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fec28009f00 tx=0x7fec2800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec26a00640 1 -- 172.21.15.67:0/3665993918 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28018030 con 0x7fec30145040 2024-04-17T10:32:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec26a00640 1 -- 172.21.15.67:0/3665993918 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2800ce50 con 0x7fec30145040 2024-04-17T10:32:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec26a00640 1 -- 172.21.15.67:0/3665993918 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec28003bc0 con 0x7fec30145040 2024-04-17T10:32:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3013a0a0 con 0x7fec30145040 2024-04-17T10:32:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3013a660 con 0x7fec30145040 2024-04-17T10:32:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec30072d50 con 0x7fec30145040 2024-04-17T10:32:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec26a00640 1 -- 172.21.15.67:0/3665993918 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fec2801b070 con 0x7fec30145040 2024-04-17T10:32:34.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec26a00640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec0c066f90 0x7fec0c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.230+0000 7fec26a00640 1 -- 172.21.15.67:0/3665993918 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec28085c40 con 0x7fec30145040 2024-04-17T10:32:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.234+0000 7fec2e800640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec0c066f90 0x7fec0c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.234+0000 7fec2e800640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec0c066f90 0x7fec0c069450 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fec20005890 tx=0x7fec20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.238+0000 7fec26a00640 1 -- 172.21.15.67:0/3665993918 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fec28022070 con 0x7fec30145040 2024-04-17T10:32:34.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.398+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec30074e30 con 0x7fec30145040 2024-04-17T10:32:34.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec26a00640 1 -- 172.21.15.67:0/3665993918 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16849 (secure 0 0 0) 0x7fec28010040 con 0x7fec30145040 2024-04-17T10:32:34.393 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:34.393 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:33.448480+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:32:31.849728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-17T10:32:32.441190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec0c066f90 msgr2=0x7fec0c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec0c066f90 0x7fec0c069450 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fec20005890 tx=0x7fec20009040 comp rx=0 tx=0).stop 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30145040 msgr2=0x7fec30139380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30145040 0x7fec30139380 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fec28009f00 tx=0x7fec2800c490 comp rx=0 tx=0).stop 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 shutdown_connections 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec0c066f90 0x7fec0c069450 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec30145040 0x7fec30139380 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec300cc300 0x7fec3013b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 --2- 172.21.15.67:0/3665993918 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec30071310 0x7fec3015aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 >> 172.21.15.67:0/3665993918 conn(0x7fec3006ac90 msgr2=0x7fec300ad220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 shutdown_connections 2024-04-17T10:32:34.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.402+0000 7fec35400640 1 -- 172.21.15.67:0/3665993918 wait complete. 2024-04-17T10:32:34.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:34.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.562+0000 7fb69a800640 1 Processor -- start 2024-04-17T10:32:34.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.562+0000 7fb69a800640 1 -- start start 2024-04-17T10:32:34.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb69a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb69406eac0 0x7fb694143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb69a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb69a800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb694144560 con 0x7fb69406e6e0 2024-04-17T10:32:34.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb69a800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69415ae40 con 0x7fb69406eac0 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb69a800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb694071440 con 0x7fb6940cf0c0 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb693400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb69406e6e0 0x7fb6940ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57582/0 (socket says 172.21.15.67:57582) 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb693400640 1 -- 172.21.15.67:0/3292950378 learned_addr learned my addr 172.21.15.67:0/3292950378 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb68be00640 1 --2- 172.21.15.67:0/3292950378 >> v2:172.21.15.67:3300/0 conn(0x7fb69406eac0 0x7fb694143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb693e00640 1 --2- 172.21.15.67:0/3292950378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb692a00640 1 -- 172.21.15.67:0/3292950378 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286010828 0 0) 0x7fb694144560 con 0x7fb69406e6e0 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb692a00640 1 -- 172.21.15.67:0/3292950378 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb678003530 con 0x7fb69406e6e0 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb68be00640 1 -- 172.21.15.67:0/3292950378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 msgr2=0x7fb694144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb68be00640 1 --2- 172.21.15.67:0/3292950378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb68be00640 1 -- 172.21.15.67:0/3292950378 >> v1:172.21.15.67:6789/0 conn(0x7fb69406e6e0 legacy=0x7fb6940ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb68be00640 1 -- 172.21.15.67:0/3292950378 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb694071720 con 0x7fb69406eac0 2024-04-17T10:32:34.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.566+0000 7fb693e00640 1 --2- 172.21.15.67:0/3292950378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb68be00640 1 --2- 172.21.15.67:0/3292950378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb694143ae0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0097c0 tx=0x7fb67c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e214be6bf788ced server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb692a00640 1 -- 172.21.15.67:0/3292950378 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0ab600 con 0x7fb69406eac0 2024-04-17T10:32:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb692a00640 1 -- 172.21.15.67:0/3292950378 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c0b0070 con 0x7fb69406eac0 2024-04-17T10:32:34.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb692a00640 1 -- 172.21.15.67:0/3292950378 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0ab9a0 con 0x7fb69406eac0 2024-04-17T10:32:34.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 -- 172.21.15.67:0/3292950378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 msgr2=0x7fb694143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 --2- 172.21.15.67:0/3292950378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb694143ae0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0097c0 tx=0x7fb67c01c3e0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 -- 172.21.15.67:0/3292950378 shutdown_connections 2024-04-17T10:32:34.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 --2- 172.21.15.67:0/3292950378 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 --2- 172.21.15.67:0/3292950378 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb694143ae0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 -- 172.21.15.67:0/3292950378 >> 172.21.15.67:0/3292950378 conn(0x7fb69406ac90 msgr2=0x7fb69406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:34.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 -- 172.21.15.67:0/3292950378 shutdown_connections 2024-04-17T10:32:34.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 -- 172.21.15.67:0/3292950378 wait complete. 2024-04-17T10:32:34.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 Processor -- start 2024-04-17T10:32:34.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.570+0000 7fb69a800640 1 -- start start 2024-04-17T10:32:34.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb69406e6e0 0x7fb69415a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb69413afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940b3a40 con 0x7fb69406eac0 2024-04-17T10:32:34.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940b38c0 con 0x7fb69406e6e0 2024-04-17T10:32:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940b3bc0 con 0x7fb6940cf0c0 2024-04-17T10:32:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb69413afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb69413afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45946/0 (socket says 172.21.15.67:45946) 2024-04-17T10:32:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 -- 172.21.15.67:0/1637396039 learned_addr learned my addr 172.21.15.67:0/1637396039 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb693400640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb69406e6e0 0x7fb69415a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb693e00640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 -- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 msgr2=0x7fb694157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 -- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb69406e6e0 msgr2=0x7fb69415a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb69406e6e0 0x7fb69415a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 -- 172.21.15.67:0/1637396039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6941425d0 con 0x7fb69406eac0 2024-04-17T10:32:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68be00640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb69413afc0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0097c0 tx=0x7fb67c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68b400640 1 -- 172.21.15.67:0/1637396039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0bf610 con 0x7fb69406eac0 2024-04-17T10:32:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68b400640 1 -- 172.21.15.67:0/1637396039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67c0bf7b0 con 0x7fb69406eac0 2024-04-17T10:32:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68b400640 1 -- 172.21.15.67:0/1637396039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67c0bfb60 con 0x7fb69406eac0 2024-04-17T10:32:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6941428b0 con 0x7fb69406eac0 2024-04-17T10:32:34.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb694142e70 con 0x7fb69406eac0 2024-04-17T10:32:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb694072d50 con 0x7fb69406eac0 2024-04-17T10:32:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.574+0000 7fb68b400640 1 -- 172.21.15.67:0/1637396039 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb67c0bd020 con 0x7fb69406eac0 2024-04-17T10:32:34.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.578+0000 7fb68b400640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb670066f60 0x7fb670069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.578+0000 7fb68b400640 1 -- 172.21.15.67:0/1637396039 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb67c11f3c0 con 0x7fb69406eac0 2024-04-17T10:32:34.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.578+0000 7fb693400640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb670066f60 0x7fb670069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.578+0000 7fb693400640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb670066f60 0x7fb670069420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fb684005890 tx=0x7fb684009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.582+0000 7fb68b400640 1 -- 172.21.15.67:0/1637396039 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb67c0c6030 con 0x7fb69406eac0 2024-04-17T10:32:34.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.742+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb694074e30 con 0x7fb69406eac0 2024-04-17T10:32:34.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.742+0000 7fb68b400640 1 -- 172.21.15.67:0/1637396039 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+16849 (secure 0 0 0) 0x7fb67c016090 con 0x7fb69406eac0 2024-04-17T10:32:34.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:34.736 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:33.448480+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:32:31.849728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-17T10:32:32.441190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb670066f60 msgr2=0x7fb670069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb670066f60 0x7fb670069420 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fb684005890 tx=0x7fb684009040 comp rx=0 tx=0).stop 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 msgr2=0x7fb69413afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb69413afc0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7fb67c0097c0 tx=0x7fb67c01ac20 comp rx=0 tx=0).stop 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 shutdown_connections 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb670066f60 0x7fb670069420 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6940cf0c0 0x7fb694157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb69406eac0 0x7fb69413afc0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 --2- 172.21.15.67:0/1637396039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb69406e6e0 0x7fb69415a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 >> 172.21.15.67:0/1637396039 conn(0x7fb69406ac90 msgr2=0x7fb6940b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 shutdown_connections 2024-04-17T10:32:34.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.746+0000 7fb69a800640 1 -- 172.21.15.67:0/1637396039 wait complete. 2024-04-17T10:32:34.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:32:34.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd979a00640 1 Processor -- start 2024-04-17T10:32:34.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd979a00640 1 -- start start 2024-04-17T10:32:34.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd979a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd974159440 0x7fd9740ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd979a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd97415c580 0x7fd974071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd979a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97415cb30 con 0x7fd9740713e0 2024-04-17T10:32:34.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd979a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9740a8650 con 0x7fd974159440 2024-04-17T10:32:34.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd979a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97415ce10 con 0x7fd97415c580 2024-04-17T10:32:34.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd972a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd974159440 0x7fd9740ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd972a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd974159440 0x7fd9740ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45960/0 (socket says 172.21.15.67:45960) 2024-04-17T10:32:34.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.910+0000 7fd972a00640 1 -- 172.21.15.67:0/515309646 learned_addr learned my addr 172.21.15.67:0/515309646 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:34.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd973400640 1 --2- 172.21.15.67:0/515309646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd97415c580 0x7fd974071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972a00640 1 -- 172.21.15.67:0/515309646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd97415c580 msgr2=0x7fd974071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:34.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972a00640 1 --2- 172.21.15.67:0/515309646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd97415c580 0x7fd974071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972a00640 1 -- 172.21.15.67:0/515309646 >> v1:172.21.15.86:6789/0 conn(0x7fd9740713e0 legacy=0x7fd97415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972a00640 1 -- 172.21.15.67:0/515309646 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9740cd610 con 0x7fd974159440 2024-04-17T10:32:34.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972a00640 1 --2- 172.21.15.67:0/515309646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd974159440 0x7fd9740ce0f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fd964009870 tx=0x7fd96401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb9ba6e8e16cd88 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972000640 1 -- 172.21.15.67:0/515309646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640a7070 con 0x7fd974159440 2024-04-17T10:32:34.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972000640 1 -- 172.21.15.67:0/515309646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9640b0070 con 0x7fd974159440 2024-04-17T10:32:34.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd972000640 1 -- 172.21.15.67:0/515309646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640ab890 con 0x7fd974159440 2024-04-17T10:32:34.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd979a00640 1 -- 172.21.15.67:0/515309646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd974159440 msgr2=0x7fd9740ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd979a00640 1 --2- 172.21.15.67:0/515309646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd974159440 0x7fd9740ce0f0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7fd964009870 tx=0x7fd96401c560 comp rx=0 tx=0).stop 2024-04-17T10:32:34.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd979a00640 1 -- 172.21.15.67:0/515309646 shutdown_connections 2024-04-17T10:32:34.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd979a00640 1 --2- 172.21.15.67:0/515309646 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd97415c580 0x7fd974071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.914+0000 7fd979a00640 1 --2- 172.21.15.67:0/515309646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd974159440 0x7fd9740ce0f0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 -- 172.21.15.67:0/515309646 >> 172.21.15.67:0/515309646 conn(0x7fd9740b1740 msgr2=0x7fd9740b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:34.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 -- 172.21.15.67:0/515309646 shutdown_connections 2024-04-17T10:32:34.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 -- 172.21.15.67:0/515309646 wait complete. 2024-04-17T10:32:34.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 Processor -- start 2024-04-17T10:32:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 -- start start 2024-04-17T10:32:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9740713e0 0x7fd974150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd974159440 0x7fd97415a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd97415c580 0x7fd97413ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97415d9c0 con 0x7fd9740713e0 2024-04-17T10:32:34.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97415d840 con 0x7fd97415c580 2024-04-17T10:32:34.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd979a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97415db40 con 0x7fd974159440 2024-04-17T10:32:34.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9740713e0 0x7fd974150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9740713e0 0x7fd974150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45968/0 (socket says 172.21.15.67:45968) 2024-04-17T10:32:34.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 -- 172.21.15.67:0/2101303396 learned_addr learned my addr 172.21.15.67:0/2101303396 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:34.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd973400640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd97415c580 0x7fd97413ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd96a000640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd974159440 0x7fd97415a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 -- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd974159440 msgr2=0x7fd97415a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd974159440 0x7fd97415a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 -- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd97415c580 msgr2=0x7fd97413ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd97415c580 0x7fd97413ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 -- 172.21.15.67:0/2101303396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd974150b50 con 0x7fd9740713e0 2024-04-17T10:32:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.918+0000 7fd972a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9740713e0 0x7fd974150570 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd9640ad040 tx=0x7fd96401ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd96b400640 1 -- 172.21.15.67:0/2101303396 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640be5f0 con 0x7fd9740713e0 2024-04-17T10:32:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd96b400640 1 -- 172.21.15.67:0/2101303396 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9640be7b0 con 0x7fd9740713e0 2024-04-17T10:32:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd96b400640 1 -- 172.21.15.67:0/2101303396 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9640beba0 con 0x7fd9740713e0 2024-04-17T10:32:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd974150e30 con 0x7fd9740713e0 2024-04-17T10:32:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9741513f0 con 0x7fd9740713e0 2024-04-17T10:32:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97404ef30 con 0x7fd9740713e0 2024-04-17T10:32:34.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd96b400640 1 -- 172.21.15.67:0/2101303396 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd9640b43f0 con 0x7fd9740713e0 2024-04-17T10:32:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd96b400640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd950066f60 0x7fd950069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.922+0000 7fd96b400640 1 -- 172.21.15.67:0/2101303396 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd96411fe80 con 0x7fd9740713e0 2024-04-17T10:32:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.926+0000 7fd96a000640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd950066f60 0x7fd950069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.926+0000 7fd96a000640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd950066f60 0x7fd950069420 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd9740ae0e0 tx=0x7fd95c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:34.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:34.926+0000 7fd96b400640 1 -- 172.21.15.67:0/2101303396 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd9640c4060 con 0x7fd9740713e0 2024-04-17T10:32:35.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:35.130+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd9740b00c0 con 0x7fd9740713e0 2024-04-17T10:32:36.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.062+0000 7fd96b400640 1 -- 172.21.15.67:0/2101303396 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd964016150 con 0x7fd9740713e0 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd950066f60 msgr2=0x7fd950069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd950066f60 0x7fd950069420 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd9740ae0e0 tx=0x7fd95c009040 comp rx=0 tx=0).stop 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9740713e0 msgr2=0x7fd974150570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9740713e0 0x7fd974150570 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd9640ad040 tx=0x7fd96401ada0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 shutdown_connections 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd950066f60 0x7fd950069420 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd97415c580 0x7fd97413ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd974159440 0x7fd97415a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 --2- 172.21.15.67:0/2101303396 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9740713e0 0x7fd974150570 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 >> 172.21.15.67:0/2101303396 conn(0x7fd9740b1740 msgr2=0x7fd9740b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:36.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 shutdown_connections 2024-04-17T10:32:36.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.066+0000 7fd979a00640 1 -- 172.21.15.67:0/2101303396 wait complete. 2024-04-17T10:32:36.082 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:32:36.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:32:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.238+0000 7f9d38200640 1 Processor -- start 2024-04-17T10:32:36.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d38200640 1 -- start start 2024-04-17T10:32:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d38200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d300aef60 0x7f9d300cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d38200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300cce60 0x7f9d3006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d38200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d300b4990 con 0x7f9d3006e7b0 2024-04-17T10:32:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d38200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d300cd2c0 con 0x7f9d300aef60 2024-04-17T10:32:36.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d38200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d300af340 con 0x7f9d300cce60 2024-04-17T10:32:36.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d36200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9d3006e7b0 0x7f9d300713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57584/0 (socket says 172.21.15.67:57584) 2024-04-17T10:32:36.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d35800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9d300aef60 0x7f9d300cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:36.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d36200640 1 -- 172.21.15.67:0/2839310970 learned_addr learned my addr 172.21.15.67:0/2839310970 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d34e00640 1 --2- 172.21.15.67:0/2839310970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300cce60 0x7f9d3006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d2fe00640 1 -- 172.21.15.67:0/2839310970 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352516864 0 0) 0x7f9d280014a0 con 0x7f9d3006e7b0 2024-04-17T10:32:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d2fe00640 1 -- 172.21.15.67:0/2839310970 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d18003530 con 0x7f9d3006e7b0 2024-04-17T10:32:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d35800640 1 -- 172.21.15.67:0/2839310970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300cce60 msgr2=0x7f9d3006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d35800640 1 --2- 172.21.15.67:0/2839310970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300cce60 0x7f9d3006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d35800640 1 -- 172.21.15.67:0/2839310970 >> v1:172.21.15.67:6789/0 conn(0x7f9d3006e7b0 legacy=0x7f9d300713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:36.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.242+0000 7f9d35800640 1 -- 172.21.15.67:0/2839310970 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3015ae90 con 0x7f9d300aef60 2024-04-17T10:32:36.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d35800640 1 --2- 172.21.15.67:0/2839310970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300aef60 0x7f9d300cc920 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9d240097c0 tx=0x7f9d2401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07ef24b9ab20244 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:36.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d2fe00640 1 -- 172.21.15.67:0/2839310970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d240b0070 con 0x7f9d300aef60 2024-04-17T10:32:36.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d2fe00640 1 -- 172.21.15.67:0/2839310970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d240ab600 con 0x7f9d300aef60 2024-04-17T10:32:36.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d2fe00640 1 -- 172.21.15.67:0/2839310970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d240ab9d0 con 0x7f9d300aef60 2024-04-17T10:32:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 -- 172.21.15.67:0/2839310970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300aef60 msgr2=0x7f9d300cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 --2- 172.21.15.67:0/2839310970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300aef60 0x7f9d300cc920 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f9d240097c0 tx=0x7f9d2401c410 comp rx=0 tx=0).stop 2024-04-17T10:32:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 -- 172.21.15.67:0/2839310970 shutdown_connections 2024-04-17T10:32:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 --2- 172.21.15.67:0/2839310970 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300cce60 0x7f9d3006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 --2- 172.21.15.67:0/2839310970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300aef60 0x7f9d300cc920 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 -- 172.21.15.67:0/2839310970 >> 172.21.15.67:0/2839310970 conn(0x7f9d300b1740 msgr2=0x7f9d300b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:36.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 -- 172.21.15.67:0/2839310970 shutdown_connections 2024-04-17T10:32:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 -- 172.21.15.67:0/2839310970 wait complete. 2024-04-17T10:32:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 Processor -- start 2024-04-17T10:32:36.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.246+0000 7f9d38200640 1 -- start start 2024-04-17T10:32:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3006e7b0 0x7f9d30153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300aef60 0x7f9d3013b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300cce60 0x7f9d3014c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3015bc10 con 0x7f9d300cce60 2024-04-17T10:32:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3015b410 con 0x7f9d3006e7b0 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d30147510 con 0x7f9d300aef60 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300cce60 0x7f9d3014c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300cce60 0x7f9d3014c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45994/0 (socket says 172.21.15.67:45994) 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 -- 172.21.15.67:0/1676168687 learned_addr learned my addr 172.21.15.67:0/1676168687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d34e00640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300aef60 0x7f9d3013b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d35800640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3006e7b0 0x7f9d30153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 -- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300aef60 msgr2=0x7f9d3013b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300aef60 0x7f9d3013b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 -- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3006e7b0 msgr2=0x7f9d30153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3006e7b0 0x7f9d30153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 -- 172.21.15.67:0/1676168687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d30151cf0 con 0x7f9d300cce60 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d36200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300cce60 0x7f9d3014c670 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800cb40 tx=0x7f9d2800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d2d600640 1 -- 172.21.15.67:0/1676168687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28017070 con 0x7f9d300cce60 2024-04-17T10:32:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d2d600640 1 -- 172.21.15.67:0/1676168687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2800ac40 con 0x7f9d300cce60 2024-04-17T10:32:36.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d2d600640 1 -- 172.21.15.67:0/1676168687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d28012400 con 0x7f9d300cce60 2024-04-17T10:32:36.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d30151f90 con 0x7f9d300cce60 2024-04-17T10:32:36.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d30152550 con 0x7f9d300cce60 2024-04-17T10:32:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.250+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d3004ef30 con 0x7f9d300cce60 2024-04-17T10:32:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.254+0000 7f9d2d600640 1 -- 172.21.15.67:0/1676168687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9d2801a070 con 0x7f9d300cce60 2024-04-17T10:32:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.254+0000 7f9d2d600640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d10066f60 0x7f9d10069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.254+0000 7f9d2d600640 1 -- 172.21.15.67:0/1676168687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9d28084fd0 con 0x7f9d300cce60 2024-04-17T10:32:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.254+0000 7f9d35800640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d10066f60 0x7f9d10069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.254+0000 7f9d35800640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d10066f60 0x7f9d10069420 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9d240ad000 tx=0x7f9d24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.258+0000 7f9d2d600640 1 -- 172.21.15.67:0/1676168687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9d300b4990 con 0x7f9d300cce60 2024-04-17T10:32:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:36.454+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9d300b3ac0 con 0x7f9d300cce60 2024-04-17T10:32:37.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d2d600640 1 -- 172.21.15.67:0/1676168687 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v73) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d280556d0 con 0x7f9d300cce60 2024-04-17T10:32:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d10066f60 msgr2=0x7f9d10069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d10066f60 0x7f9d10069420 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f9d240ad000 tx=0x7f9d24008040 comp rx=0 tx=0).stop 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300cce60 msgr2=0x7f9d3014c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300cce60 0x7f9d3014c670 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f9d2800cb40 tx=0x7f9d2800a060 comp rx=0 tx=0).stop 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 shutdown_connections 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d10066f60 0x7f9d10069420 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d300cce60 0x7f9d3014c670 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9d300aef60 0x7f9d3013b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 --2- 172.21.15.67:0/1676168687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9d3006e7b0 0x7f9d30153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 >> 172.21.15.67:0/1676168687 conn(0x7f9d300b1740 msgr2=0x7f9d300b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:37.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 shutdown_connections 2024-04-17T10:32:37.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.074+0000 7f9d38200640 1 -- 172.21.15.67:0/1676168687 wait complete. 2024-04-17T10:32:37.089 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:32:37.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e8de00640 1 Processor -- start 2024-04-17T10:32:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e8de00640 1 -- start start 2024-04-17T10:32:37.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e8de00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1e880b07b0 0x7f1e88070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e8de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e88071300 0x7f1e8806c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e8de00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e880b5980 con 0x7f1e8815dd10 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e8de00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e880b1280 con 0x7f1e880b07b0 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e8de00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e880b0cc0 con 0x7f1e88071300 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e87400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e88071300 0x7f1e8806c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e7e000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1e880b07b0 0x7f1e88070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e87400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e88071300 0x7f1e8806c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:50098/0 (socket says 172.21.15.67:50098) 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e86a00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1e8815dd10 0x7f1e880b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:41694/0 (socket says 172.21.15.67:41694) 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e87400640 1 -- 172.21.15.67:0/1385802043 learned_addr learned my addr 172.21.15.67:0/1385802043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:37.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e87400640 1 -- 172.21.15.67:0/1385802043 >> v2:172.21.15.86:3300/0 conn(0x7f1e880b07b0 msgr2=0x7f1e88070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e87400640 1 --2- 172.21.15.67:0/1385802043 >> v2:172.21.15.86:3300/0 conn(0x7f1e880b07b0 0x7f1e88070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e87400640 1 -- 172.21.15.67:0/1385802043 >> v1:172.21.15.86:6789/0 conn(0x7f1e8815dd10 legacy=0x7f1e880b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e87400640 1 -- 172.21.15.67:0/1385802043 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880b0fa0 con 0x7f1e88071300 2024-04-17T10:32:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e86000640 1 -- 172.21.15.67:0/1385802043 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237615508 0 0) 0x7f1e880b5980 con 0x7f1e8815dd10 2024-04-17T10:32:37.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.242+0000 7f1e7e000640 1 --2- 172.21.15.67:0/1385802043 >> v2:172.21.15.86:3300/0 conn(0x7f1e880b07b0 0x7f1e88070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e87400640 1 --2- 172.21.15.67:0/1385802043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e88071300 0x7f1e8806c6d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009e50 tx=0x7f1e8001c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3145df507f79749c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e86000640 1 -- 172.21.15.67:0/1385802043 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800a5750 con 0x7f1e88071300 2024-04-17T10:32:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e86000640 1 -- 172.21.15.67:0/1385802043 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e800a58f0 con 0x7f1e88071300 2024-04-17T10:32:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e86000640 1 -- 172.21.15.67:0/1385802043 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800a5cc0 con 0x7f1e88071300 2024-04-17T10:32:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 -- 172.21.15.67:0/1385802043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e88071300 msgr2=0x7f1e8806c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1385802043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e88071300 0x7f1e8806c6d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009e50 tx=0x7f1e8001c9a0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 -- 172.21.15.67:0/1385802043 shutdown_connections 2024-04-17T10:32:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1385802043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e88071300 0x7f1e8806c6d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1385802043 >> v2:172.21.15.86:3300/0 conn(0x7f1e880b07b0 0x7f1e88070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 -- 172.21.15.67:0/1385802043 >> 172.21.15.67:0/1385802043 conn(0x7f1e880b34d0 msgr2=0x7f1e880b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 -- 172.21.15.67:0/1385802043 shutdown_connections 2024-04-17T10:32:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 -- 172.21.15.67:0/1385802043 wait complete. 2024-04-17T10:32:37.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.246+0000 7f1e8de00640 1 Processor -- start 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 -- start start 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e88071300 0x7f1e880b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e880b07b0 0x7f1e881597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8815dd10 0x7f1e88159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8806d5a0 con 0x7f1e8815dd10 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8806d420 con 0x7f1e88071300 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8806d720 con 0x7f1e880b07b0 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8815dd10 0x7f1e88159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8815dd10 0x7f1e88159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46010/0 (socket says 172.21.15.67:46010) 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 -- 172.21.15.67:0/1478198136 learned_addr learned my addr 172.21.15.67:0/1478198136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e7e000640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e880b07b0 0x7f1e881597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e86a00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e88071300 0x7f1e880b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 -- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e880b07b0 msgr2=0x7f1e881597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e880b07b0 0x7f1e881597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 -- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e88071300 msgr2=0x7f1e880b5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e88071300 0x7f1e880b5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 -- 172.21.15.67:0/1478198136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e880cabf0 con 0x7f1e8815dd10 2024-04-17T10:32:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e86a00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e88071300 0x7f1e880b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:37.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e87400640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8815dd10 0x7f1e88159d30 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f1e800af040 tx=0x7f1e80019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e7fe00640 1 -- 172.21.15.67:0/1478198136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800bc020 con 0x7f1e8815dd10 2024-04-17T10:32:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e7fe00640 1 -- 172.21.15.67:0/1478198136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e800b8670 con 0x7f1e8815dd10 2024-04-17T10:32:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e880cae60 con 0x7f1e8815dd10 2024-04-17T10:32:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e7fe00640 1 -- 172.21.15.67:0/1478198136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e800b8a60 con 0x7f1e8815dd10 2024-04-17T10:32:37.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e880cb420 con 0x7f1e8815dd10 2024-04-17T10:32:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.250+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e880b3240 con 0x7f1e8815dd10 2024-04-17T10:32:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.254+0000 7f1e7fe00640 1 -- 172.21.15.67:0/1478198136 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1e800bf020 con 0x7f1e8815dd10 2024-04-17T10:32:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.254+0000 7f1e7fe00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1e64066f60 0x7f1e64069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.254+0000 7f1e7fe00640 1 -- 172.21.15.67:0/1478198136 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1e801205c0 con 0x7f1e8815dd10 2024-04-17T10:32:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.254+0000 7f1e86a00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1e64066f60 0x7f1e64069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.254+0000 7f1e86a00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1e64066f60 0x7f1e64069420 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000f70 tx=0x7f1e78007dd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.258+0000 7f1e7fe00640 1 -- 172.21.15.67:0/1478198136 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1e800c6030 con 0x7f1e8815dd10 2024-04-17T10:32:37.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.450+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e88074e30 con 0x7f1e8815dd10 2024-04-17T10:32:37.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e7fe00640 1 -- 172.21.15.67:0/1478198136 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f1e80016620 con 0x7f1e8815dd10 2024-04-17T10:32:37.444 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:37.444 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:32:37.445 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:32:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1e64066f60 msgr2=0x7f1e64069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1e64066f60 0x7f1e64069420 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1e78000f70 tx=0x7f1e78007dd0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8815dd10 msgr2=0x7f1e88159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8815dd10 0x7f1e88159d30 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f1e800af040 tx=0x7f1e80019740 comp rx=0 tx=0).stop 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 shutdown_connections 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1e64066f60 0x7f1e64069420 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e8815dd10 0x7f1e88159d30 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1e880b07b0 0x7f1e881597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 --2- 172.21.15.67:0/1478198136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1e88071300 0x7f1e880b5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 >> 172.21.15.67:0/1478198136 conn(0x7f1e880b34d0 msgr2=0x7f1e8815b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 shutdown_connections 2024-04-17T10:32:37.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.454+0000 7f1e8de00640 1 -- 172.21.15.67:0/1478198136 wait complete. 2024-04-17T10:32:37.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:32:37.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.614+0000 7f7e76000640 1 Processor -- start 2024-04-17T10:32:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.614+0000 7f7e76000640 1 -- start start 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e76000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7e7015dc20 0x7f7e700ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e76000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700cf070 0x7f7e700d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e76000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e700d0930 0x7f7e7006c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e76000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700d3580 con 0x7f7e700d0930 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e76000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e700d2970 con 0x7f7e7015dc20 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e76000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700d2c50 con 0x7f7e700cf070 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e700d0930 0x7f7e7006c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7e700d0930 0x7f7e7006c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46026/0 (socket says 172.21.15.67:46026) 2024-04-17T10:32:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 -- 172.21.15.67:0/4205787587 learned_addr learned my addr 172.21.15.67:0/4205787587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6f400640 1 --2- 172.21.15.67:0/4205787587 >> v2:172.21.15.86:3300/0 conn(0x7f7e7015dc20 0x7f7e700ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6ea00640 1 --2- 172.21.15.67:0/4205787587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700cf070 0x7f7e700d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 -- 172.21.15.67:0/4205787587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700cf070 msgr2=0x7f7e700d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 --2- 172.21.15.67:0/4205787587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700cf070 0x7f7e700d03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 -- 172.21.15.67:0/4205787587 >> v2:172.21.15.86:3300/0 conn(0x7f7e7015dc20 msgr2=0x7f7e700ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 --2- 172.21.15.67:0/4205787587 >> v2:172.21.15.86:3300/0 conn(0x7f7e7015dc20 0x7f7e700ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6fe00640 1 -- 172.21.15.67:0/4205787587 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e700d2f30 con 0x7f7e700d0930 2024-04-17T10:32:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6ea00640 1 --2- 172.21.15.67:0/4205787587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700cf070 0x7f7e700d03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:37.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.618+0000 7f7e6f400640 1 --2- 172.21.15.67:0/4205787587 >> v2:172.21.15.86:3300/0 conn(0x7f7e7015dc20 0x7f7e700ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e6fe00640 1 --2- 172.21.15.67:0/4205787587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700d0930 0x7f7e7006c3a0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f7e7006d3d0 tx=0x7f7e6801c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b98adbe6b0ad9149 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e6e000640 1 -- 172.21.15.67:0/4205787587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680a9430 con 0x7f7e700d0930 2024-04-17T10:32:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e6e000640 1 -- 172.21.15.67:0/4205787587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e680ae070 con 0x7f7e700d0930 2024-04-17T10:32:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e6e000640 1 -- 172.21.15.67:0/4205787587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680a97f0 con 0x7f7e700d0930 2024-04-17T10:32:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 -- 172.21.15.67:0/4205787587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700d0930 msgr2=0x7f7e7006c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 --2- 172.21.15.67:0/4205787587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700d0930 0x7f7e7006c3a0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f7e7006d3d0 tx=0x7f7e6801c6a0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 -- 172.21.15.67:0/4205787587 shutdown_connections 2024-04-17T10:32:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 --2- 172.21.15.67:0/4205787587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700d0930 0x7f7e7006c3a0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 --2- 172.21.15.67:0/4205787587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700cf070 0x7f7e700d03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 --2- 172.21.15.67:0/4205787587 >> v2:172.21.15.86:3300/0 conn(0x7f7e7015dc20 0x7f7e700ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 -- 172.21.15.67:0/4205787587 >> 172.21.15.67:0/4205787587 conn(0x7f7e70072a40 msgr2=0x7f7e700b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 -- 172.21.15.67:0/4205787587 shutdown_connections 2024-04-17T10:32:37.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 -- 172.21.15.67:0/4205787587 wait complete. 2024-04-17T10:32:37.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 Processor -- start 2024-04-17T10:32:37.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.622+0000 7f7e76000640 1 -- start start 2024-04-17T10:32:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700cf070 0x7f7e701f9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700d0930 0x7f7e7014c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7015dc20 0x7f7e7014d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700b6e80 con 0x7f7e700cf070 2024-04-17T10:32:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700b6d00 con 0x7f7e7015dc20 2024-04-17T10:32:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e700b7000 con 0x7f7e700d0930 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700cf070 0x7f7e701f9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700cf070 0x7f7e701f9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46028/0 (socket says 172.21.15.67:46028) 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 -- 172.21.15.67:0/3729288015 learned_addr learned my addr 172.21.15.67:0/3729288015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6fe00640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7015dc20 0x7f7e7014d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6ea00640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700d0930 0x7f7e7014c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 -- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700d0930 msgr2=0x7f7e7014c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700d0930 0x7f7e7014c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 -- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7015dc20 msgr2=0x7f7e7014d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7015dc20 0x7f7e7014d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 -- 172.21.15.67:0/3729288015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7014a7b0 con 0x7f7e700cf070 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6fe00640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7015dc20 0x7f7e7014d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:37.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e6f400640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700cf070 0x7f7e701f9dc0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000c7f0 tx=0x7f7e6000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e67400640 1 -- 172.21.15.67:0/3729288015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60015070 con 0x7f7e700cf070 2024-04-17T10:32:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e67400640 1 -- 172.21.15.67:0/3729288015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e60007820 con 0x7f7e700cf070 2024-04-17T10:32:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e67400640 1 -- 172.21.15.67:0/3729288015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60007c10 con 0x7f7e700cf070 2024-04-17T10:32:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e7014aa90 con 0x7f7e700cf070 2024-04-17T10:32:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e7014b050 con 0x7f7e700cf070 2024-04-17T10:32:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e70168630 con 0x7f7e700cf070 2024-04-17T10:32:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.626+0000 7f7e67400640 1 -- 172.21.15.67:0/3729288015 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7e60018070 con 0x7f7e700cf070 2024-04-17T10:32:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.630+0000 7f7e67400640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e50066f60 0x7f7e50069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.630+0000 7f7e67400640 1 -- 172.21.15.67:0/3729288015 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7e60085290 con 0x7f7e700cf070 2024-04-17T10:32:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.630+0000 7f7e6ea00640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e50066f60 0x7f7e50069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.630+0000 7f7e6ea00640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e50066f60 0x7f7e50069420 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7e700c9fa0 tx=0x7f7e5c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.634+0000 7f7e67400640 1 -- 172.21.15.67:0/3729288015 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7e60056100 con 0x7f7e700cf070 2024-04-17T10:32:37.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.798+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e70074e30 con 0x7f7e700cf070 2024-04-17T10:32:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e67400640 1 -- 172.21.15.67:0/3729288015 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16836 (secure 0 0 0) 0x7f7e60055a00 con 0x7f7e700cf070 2024-04-17T10:32:37.792 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:37.793 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":107,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:34.925635+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:32:31.849728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-17T10:32:32.441190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:32:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e50066f60 msgr2=0x7f7e50069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e50066f60 0x7f7e50069420 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7e700c9fa0 tx=0x7f7e5c008040 comp rx=0 tx=0).stop 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700cf070 msgr2=0x7f7e701f9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700cf070 0x7f7e701f9dc0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000c7f0 tx=0x7f7e6000ccb0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 shutdown_connections 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7e50066f60 0x7f7e50069420 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7e7015dc20 0x7f7e7014d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7e700d0930 0x7f7e7014c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 --2- 172.21.15.67:0/3729288015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7e700cf070 0x7f7e701f9dc0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 >> 172.21.15.67:0/3729288015 conn(0x7f7e70072a40 msgr2=0x7f7e700b8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 shutdown_connections 2024-04-17T10:32:37.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.802+0000 7f7e76000640 1 -- 172.21.15.67:0/3729288015 wait complete. 2024-04-17T10:32:37.815 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:32:37.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:32:37.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe55ec00640 1 Processor -- start 2024-04-17T10:32:37.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe55ec00640 1 -- start start 2024-04-17T10:32:37.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe55ec00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5581446c0 0x7fe558144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe55ec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe5580d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe55ec00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5580b6640 con 0x7fe558071840 2024-04-17T10:32:37.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe55ec00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5580b51a0 con 0x7fe5581446c0 2024-04-17T10:32:37.957 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe55ec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5581456d0 con 0x7fe558144aa0 2024-04-17T10:32:37.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe5580d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe5581446c0 0x7fe558144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe5580d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:50138/0 (socket says 172.21.15.67:50138) 2024-04-17T10:32:37.958 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557400640 1 -- 172.21.15.67:0/3972111338 learned_addr learned my addr 172.21.15.67:0/3972111338 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:37.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557400640 1 -- 172.21.15.67:0/3972111338 >> v2:172.21.15.86:3300/0 conn(0x7fe5581446c0 msgr2=0x7fe558144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:37.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557400640 1 --2- 172.21.15.67:0/3972111338 >> v2:172.21.15.86:3300/0 conn(0x7fe5581446c0 0x7fe558144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557400640 1 -- 172.21.15.67:0/3972111338 >> v1:172.21.15.86:6789/0 conn(0x7fe558071840 legacy=0x7fe55815f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.966+0000 7fe557400640 1 -- 172.21.15.67:0/3972111338 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5580b4870 con 0x7fe558144aa0 2024-04-17T10:32:37.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe557400640 1 --2- 172.21.15.67:0/3972111338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe5580d0910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe544009870 tx=0x7fe54401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7de5a58a681982eb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe556a00640 1 -- 172.21.15.67:0/3972111338 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440aa600 con 0x7fe558144aa0 2024-04-17T10:32:37.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe556a00640 1 -- 172.21.15.67:0/3972111338 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5440af070 con 0x7fe558144aa0 2024-04-17T10:32:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe556a00640 1 -- 172.21.15.67:0/3972111338 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440aa9c0 con 0x7fe558144aa0 2024-04-17T10:32:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 -- 172.21.15.67:0/3972111338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 msgr2=0x7fe5580d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 --2- 172.21.15.67:0/3972111338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe5580d0910 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe544009870 tx=0x7fe54401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 -- 172.21.15.67:0/3972111338 shutdown_connections 2024-04-17T10:32:37.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 --2- 172.21.15.67:0/3972111338 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe5580d0910 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 --2- 172.21.15.67:0/3972111338 >> v2:172.21.15.86:3300/0 conn(0x7fe5581446c0 0x7fe558144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 -- 172.21.15.67:0/3972111338 >> 172.21.15.67:0/3972111338 conn(0x7fe5580b34d0 msgr2=0x7fe5580b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:37.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 -- 172.21.15.67:0/3972111338 shutdown_connections 2024-04-17T10:32:37.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.970+0000 7fe55ec00640 1 -- 172.21.15.67:0/3972111338 wait complete. 2024-04-17T10:32:37.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 Processor -- start 2024-04-17T10:32:37.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 -- start start 2024-04-17T10:32:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe558071840 0x7fe558157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5581446c0 0x7fe5581610e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe558143120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5580d1260 con 0x7fe558071840 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55815c0d0 con 0x7fe5581446c0 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe558168a40 con 0x7fe558144aa0 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe558071840 0x7fe558157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe558071840 0x7fe558157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46044/0 (socket says 172.21.15.67:46044) 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 -- 172.21.15.67:0/546211131 learned_addr learned my addr 172.21.15.67:0/546211131 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557400640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5581446c0 0x7fe5581610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55cc00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe558143120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 -- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 msgr2=0x7fe558143120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe558143120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 -- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5581446c0 msgr2=0x7fe5581610e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5581446c0 0x7fe5581610e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 -- 172.21.15.67:0/546211131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe558161620 con 0x7fe558071840 2024-04-17T10:32:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe557e00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe558071840 0x7fe558157e30 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fe54800e930 tx=0x7fe54800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe54fe00640 1 -- 172.21.15.67:0/546211131 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548017070 con 0x7fe558071840 2024-04-17T10:32:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe54fe00640 1 -- 172.21.15.67:0/546211131 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54800ca00 con 0x7fe558071840 2024-04-17T10:32:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe54fe00640 1 -- 172.21.15.67:0/546211131 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54800cdb0 con 0x7fe558071840 2024-04-17T10:32:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5581589b0 con 0x7fe558071840 2024-04-17T10:32:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe558158f70 con 0x7fe558071840 2024-04-17T10:32:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.974+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe558070bb0 con 0x7fe558071840 2024-04-17T10:32:37.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.978+0000 7fe54fe00640 1 -- 172.21.15.67:0/546211131 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe54801a070 con 0x7fe558071840 2024-04-17T10:32:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.978+0000 7fe54fe00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe538066f60 0x7fe538069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.978+0000 7fe54fe00640 1 -- 172.21.15.67:0/546211131 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe548085150 con 0x7fe558071840 2024-04-17T10:32:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.978+0000 7fe557400640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe538066f60 0x7fe538069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.978+0000 7fe557400640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe538066f60 0x7fe538069420 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe5440ae000 tx=0x7fe5440a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:37.982+0000 7fe54fe00640 1 -- 172.21.15.67:0/546211131 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe558161620 con 0x7fe558071840 2024-04-17T10:32:38.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.142+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe558074e30 con 0x7fe538066f60 2024-04-17T10:32:38.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.150+0000 7fe54fe00640 1 -- 172.21.15.67:0/546211131 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+240257 (secure 0 0 0) 0x7fe558074e30 con 0x7fe538066f60 2024-04-17T10:32:38.144 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:38.146 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:32:38.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.150+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe538066f60 msgr2=0x7fe538069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.150+0000 7fe55ec00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe538066f60 0x7fe538069420 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fe5440ae000 tx=0x7fe5440a7000 comp rx=0 tx=0).stop 2024-04-17T10:32:38.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.150+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe558071840 msgr2=0x7fe558157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.150+0000 7fe55ec00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe558071840 0x7fe558157e30 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7fe54800e930 tx=0x7fe54800edf0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 shutdown_connections 2024-04-17T10:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe538066f60 0x7fe538069420 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe558144aa0 0x7fe558143120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe5581446c0 0x7fe5581610e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 --2- 172.21.15.67:0/546211131 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe558071840 0x7fe558157e30 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 >> 172.21.15.67:0/546211131 conn(0x7fe5580b34d0 msgr2=0x7fe55815ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 shutdown_connections 2024-04-17T10:32:38.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.154+0000 7fe55ec00640 1 -- 172.21.15.67:0/546211131 wait complete. 2024-04-17T10:32:38.163 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-04-17T10:32:36.246090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":409172,"kb_used_data":11156,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754565548,"statfs":{"total":773094113280,"available":772675121152,"internally_reserved":0,"allocated":11423744,"data_stored":5929865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":407424561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.209593"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039759+0000","last_change":"2024-04-17T10:32:32.039759+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039759+0000","last_undegraded":"2024-04-17T10:32:32.039759+0000","last_fullsized":"2024-04-17T10:32:32.039759+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.046258+0000","last_change":"2024-04-17T10:32:32.046258+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.046258+0000","last_undegraded":"2024-04-17T10:32:32.046258+0000","last_fullsized":"2024-04-17T10:32:32.046258+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.036871+0000","last_change":"2024-04-17T10:32:32.036871+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.036871+0000","last_undegraded":"2024-04-17T10:32:32.036871+0000","last_fullsized":"2024-04-17T10:32:32.036871+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.043731+0000","last_change":"2024-04-17T10:32:32.043731+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.043731+0000","last_undegraded":"2024-04-17T10:32:32.043731+0000","last_fullsized":"2024-04-17T10:32:32.043731+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039322+0000","last_change":"2024-04-17T10:32:32.039322+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039322+0000","last_undegraded":"2024-04-17T10:32:32.039322+0000","last_fullsized":"2024-04-17T10:32:32.039322+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039999+0000","last_change":"2024-04-17T10:32:32.039999+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039999+0000","last_undegraded":"2024-04-17T10:32:32.039999+0000","last_fullsized":"2024-04-17T10:32:32.039999+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.036884+0000","last_change":"2024-04-17T10:32:32.036884+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.036884+0000","last_undegraded":"2024-04-17T10:32:32.036884+0000","last_fullsized":"2024-04-17T10:32:32.036884+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039783+0000","last_change":"2024-04-17T10:32:32.039783+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039783+0000","last_undegraded":"2024-04-17T10:32:32.039783+0000","last_fullsized":"2024-04-17T10:32:32.039783+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.036669+0000","last_change":"2024-04-17T10:32:32.036669+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.036669+0000","last_undegraded":"2024-04-17T10:32:32.036669+0000","last_fullsized":"2024-04-17T10:32:32.036669+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.048060+0000","last_change":"2024-04-17T10:32:32.048060+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.048060+0000","last_undegraded":"2024-04-17T10:32:32.048060+0000","last_fullsized":"2024-04-17T10:32:32.048060+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.041730+0000","last_change":"2024-04-17T10:32:32.041730+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.041730+0000","last_undegraded":"2024-04-17T10:32:32.041730+0000","last_fullsized":"2024-04-17T10:32:32.041730+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039809+0000","last_change":"2024-04-17T10:32:32.039809+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039809+0000","last_undegraded":"2024-04-17T10:32:32.039809+0000","last_fullsized":"2024-04-17T10:32:32.039809+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.043148+0000","last_change":"2024-04-17T10:32:32.043148+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.043148+0000","last_undegraded":"2024-04-17T10:32:32.043148+0000","last_fullsized":"2024-04-17T10:32:32.043148+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039771+0000","last_change":"2024-04-17T10:32:32.039771+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039771+0000","last_undegraded":"2024-04-17T10:32:32.039771+0000","last_fullsized":"2024-04-17T10:32:32.039771+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.045189+0000","last_change":"2024-04-17T10:32:32.045189+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.045189+0000","last_undegraded":"2024-04-17T10:32:32.045189+0000","last_fullsized":"2024-04-17T10:32:32.045189+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.036662+0000","last_change":"2024-04-17T10:32:32.036662+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.036662+0000","last_undegraded":"2024-04-17T10:32:32.036662+0000","last_fullsized":"2024-04-17T10:32:32.036662+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039731+0000","last_change":"2024-04-17T10:32:32.039731+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039731+0000","last_undegraded":"2024-04-17T10:32:32.039731+0000","last_fullsized":"2024-04-17T10:32:32.039731+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039349+0000","last_change":"2024-04-17T10:32:32.039349+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039349+0000","last_undegraded":"2024-04-17T10:32:32.039349+0000","last_fullsized":"2024-04-17T10:32:32.039349+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039717+0000","last_change":"2024-04-17T10:32:32.039717+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039717+0000","last_undegraded":"2024-04-17T10:32:32.039717+0000","last_fullsized":"2024-04-17T10:32:32.039717+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.044073+0000","last_change":"2024-04-17T10:32:32.044073+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.044073+0000","last_undegraded":"2024-04-17T10:32:32.044073+0000","last_fullsized":"2024-04-17T10:32:32.044073+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.044809+0000","last_change":"2024-04-17T10:32:32.044809+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.044809+0000","last_undegraded":"2024-04-17T10:32:32.044809+0000","last_fullsized":"2024-04-17T10:32:32.044809+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.041990+0000","last_change":"2024-04-17T10:32:32.041990+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.041990+0000","last_undegraded":"2024-04-17T10:32:32.041990+0000","last_fullsized":"2024-04-17T10:32:32.041990+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.044406+0000","last_change":"2024-04-17T10:32:32.044406+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.044406+0000","last_undegraded":"2024-04-17T10:32:32.044406+0000","last_fullsized":"2024-04-17T10:32:32.044406+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.043330+0000","last_change":"2024-04-17T10:32:32.043330+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.043330+0000","last_undegraded":"2024-04-17T10:32:32.043330+0000","last_fullsized":"2024-04-17T10:32:32.043330+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.047617+0000","last_change":"2024-04-17T10:32:32.047617+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.047617+0000","last_undegraded":"2024-04-17T10:32:32.047617+0000","last_fullsized":"2024-04-17T10:32:32.047617+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.039680+0000","last_change":"2024-04-17T10:32:32.039680+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.039680+0000","last_undegraded":"2024-04-17T10:32:32.039680+0000","last_fullsized":"2024-04-17T10:32:32.039680+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.043312+0000","last_change":"2024-04-17T10:32:32.043312+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.043312+0000","last_undegraded":"2024-04-17T10:32:32.043312+0000","last_fullsized":"2024-04-17T10:32:32.043312+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.036683+0000","last_change":"2024-04-17T10:32:32.036683+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.036683+0000","last_undegraded":"2024-04-17T10:32:32.036683+0000","last_fullsized":"2024-04-17T10:32:32.036683+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.044391+0000","last_change":"2024-04-17T10:32:32.044391+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.044391+0000","last_undegraded":"2024-04-17T10:32:32.044391+0000","last_fullsized":"2024-04-17T10:32:32.044391+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.040201+0000","last_change":"2024-04-17T10:32:32.040201+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.040201+0000","last_undegraded":"2024-04-17T10:32:32.040201+0000","last_fullsized":"2024-04-17T10:32:32.040201+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.043078+0000","last_change":"2024-04-17T10:32:32.043078+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.043078+0000","last_undegraded":"2024-04-17T10:32:32.043078+0000","last_fullsized":"2024-04-17T10:32:32.043078+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-17T10:32:33.037359+0000","last_change":"2024-04-17T10:32:33.037359+0000","last_active":"2024-04-17T10:32:33.037359+0000","last_peered":"2024-04-17T10:32:33.037359+0000","last_clean":"2024-04-17T10:32:33.037359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:33.037359+0000","last_undegraded":"2024-04-17T10:32:33.037359+0000","last_fullsized":"2024-04-17T10:32:33.037359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":1,"reported_epoch":104,"state":"creating+peering","last_fresh":"2024-04-17T10:32:32.043668+0000","last_change":"2024-04-17T10:32:32.043668+0000","last_active":"2024-04-17T10:32:32.033005+0000","last_peered":"2024-04-17T10:32:32.033005+0000","last_clean":"2024-04-17T10:32:32.033005+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-17T10:32:32.043668+0000","last_undegraded":"2024-04-17T10:32:32.043668+0000","last_fullsized":"2024-04-17T10:32:32.043668+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":323,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-04-17T10:32:32.042294+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:32:32.042294+0000","last_peered":"2024-04-17T10:32:32.042294+0000","last_clean":"2024-04-17T10:32:32.042294+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:32:32.042294+0000","last_undegraded":"2024-04-17T10:32:32.042294+0000","last_fullsized":"2024-04-17T10:32:32.042294+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574946,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51068,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320772,"statfs":{"total":96636764160,"available":96584470528,"internally_reserved":0,"allocated":1884160,"data_stored":1189442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50476,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321364,"statfs":{"total":96636764160,"available":96585076736,"internally_reserved":0,"allocated":1277952,"data_stored":592865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46320,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325520,"statfs":{"total":96636764160,"available":96589332480,"internally_reserved":0,"allocated":1277952,"data_stored":592865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54644,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317196,"statfs":{"total":96636764160,"available":96580808704,"internally_reserved":0,"allocated":1277952,"data_stored":592865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574945,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54648,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317192,"statfs":{"total":96636764160,"available":96580804608,"internally_reserved":0,"allocated":1277952,"data_stored":592865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574946,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50476,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321364,"statfs":{"total":96636764160,"available":96585076736,"internally_reserved":0,"allocated":1277952,"data_stored":592865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46320,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325520,"statfs":{"total":96636764160,"available":96589332480,"internally_reserved":0,"allocated":1277952,"data_stored":592865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574945,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55220,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316620,"statfs":{"total":96636764160,"available":96580218880,"internally_reserved":0,"allocated":1871872,"data_stored":1183233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:32:38.168 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:32:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2324400640 1 Processor -- start 2024-04-17T10:32:38.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2324400640 1 -- start start 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2324400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c0b07b0 0x7f231c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2324400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f231c071300 0x7f231c06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2324400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f231c0b5980 con 0x7f231c15dd10 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2324400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f231c0b1280 con 0x7f231c071300 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2324400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231c0b0cc0 con 0x7f231c0b07b0 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2321a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f231c15dd10 0x7f231c0b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57586/0 (socket says 172.21.15.67:57586) 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2321a00640 1 -- 172.21.15.67:0/627578573 learned_addr learned my addr 172.21.15.67:0/627578573 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2322400640 1 --2- 172.21.15.67:0/627578573 >> v2:172.21.15.67:3300/0 conn(0x7f231c071300 0x7f231c06c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2321000640 1 --2- 172.21.15.67:0/627578573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c0b07b0 0x7f231c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2322400640 1 -- 172.21.15.67:0/627578573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c0b07b0 msgr2=0x7f231c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2322400640 1 --2- 172.21.15.67:0/627578573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c0b07b0 0x7f231c070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f231be00640 1 -- 172.21.15.67:0/627578573 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836904203 0 0) 0x7f231c0b5980 con 0x7f231c15dd10 2024-04-17T10:32:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2322400640 1 -- 172.21.15.67:0/627578573 >> v1:172.21.15.67:6789/0 conn(0x7f231c15dd10 legacy=0x7f231c0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.358+0000 7f2322400640 1 -- 172.21.15.67:0/627578573 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0b0fa0 con 0x7f231c071300 2024-04-17T10:32:38.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2322400640 1 --2- 172.21.15.67:0/627578573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c06c6d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f2314009d90 tx=0x7f231401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a46adc6c35c73b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:38.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f231be00640 1 -- 172.21.15.67:0/627578573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23140b2070 con 0x7f231c071300 2024-04-17T10:32:38.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f231be00640 1 -- 172.21.15.67:0/627578573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23140a5890 con 0x7f231c071300 2024-04-17T10:32:38.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f231be00640 1 -- 172.21.15.67:0/627578573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23140a5c60 con 0x7f231c071300 2024-04-17T10:32:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 -- 172.21.15.67:0/627578573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 msgr2=0x7f231c06c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 --2- 172.21.15.67:0/627578573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c06c6d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f2314009d90 tx=0x7f231401c8e0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 -- 172.21.15.67:0/627578573 shutdown_connections 2024-04-17T10:32:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 --2- 172.21.15.67:0/627578573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c06c6d0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 --2- 172.21.15.67:0/627578573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c0b07b0 0x7f231c070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 -- 172.21.15.67:0/627578573 >> 172.21.15.67:0/627578573 conn(0x7f231c0b34d0 msgr2=0x7f231c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:38.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 -- 172.21.15.67:0/627578573 shutdown_connections 2024-04-17T10:32:38.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.362+0000 7f2324400640 1 -- 172.21.15.67:0/627578573 wait complete. 2024-04-17T10:32:38.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 Processor -- start 2024-04-17T10:32:38.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 -- start start 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f231c0b07b0 0x7f231c151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c15dd10 0x7f231c148950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c06d5a0 con 0x7f231c071300 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231c06d420 con 0x7f231c0b07b0 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f231c06d720 con 0x7f231c15dd10 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46052/0 (socket says 172.21.15.67:46052) 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 -- 172.21.15.67:0/2836250909 learned_addr learned my addr 172.21.15.67:0/2836250909 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321000640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f231c0b07b0 0x7f231c151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2322400640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c15dd10 0x7f231c148950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 -- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c15dd10 msgr2=0x7f231c148950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c15dd10 0x7f231c148950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 -- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f231c0b07b0 msgr2=0x7f231c151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f231c0b07b0 0x7f231c151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 -- 172.21.15.67:0/2836250909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0ce910 con 0x7f231c071300 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2321a00640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c158a00 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f231000c480 tx=0x7f231000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2319600640 1 -- 172.21.15.67:0/2836250909 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310018070 con 0x7f231c071300 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2319600640 1 -- 172.21.15.67:0/2836250909 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2310007500 con 0x7f231c071300 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2319600640 1 -- 172.21.15.67:0/2836250909 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2310004a30 con 0x7f231c071300 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231c0cebf0 con 0x7f231c071300 2024-04-17T10:32:38.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.366+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231c0cf1b0 con 0x7f231c071300 2024-04-17T10:32:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.370+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f231c070bb0 con 0x7f231c071300 2024-04-17T10:32:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.370+0000 7f2319600640 1 -- 172.21.15.67:0/2836250909 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f231001b070 con 0x7f231c071300 2024-04-17T10:32:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.370+0000 7f2319600640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22fc066f60 0x7f22fc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.370+0000 7f2319600640 1 -- 172.21.15.67:0/2836250909 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f23100845a0 con 0x7f231c071300 2024-04-17T10:32:38.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.370+0000 7f2321000640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22fc066f60 0x7f22fc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.370+0000 7f2321000640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22fc066f60 0x7f22fc069420 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f231c0ce0f0 tx=0x7f2308008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:38.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.374+0000 7f2319600640 1 -- 172.21.15.67:0/2836250909 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2310055410 con 0x7f231c071300 2024-04-17T10:32:38.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.534+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f231c0b4950 con 0x7f22fc066f60 2024-04-17T10:32:38.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.538+0000 7f2319600640 1 -- 172.21.15.67:0/2836250909 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252485 (secure 0 0 0) 0x7f231c0b4950 con 0x7f22fc066f60 2024-04-17T10:32:38.535 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:32:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22fc066f60 msgr2=0x7f22fc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22fc066f60 0x7f22fc069420 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f231c0ce0f0 tx=0x7f2308008040 comp rx=0 tx=0).stop 2024-04-17T10:32:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 msgr2=0x7f231c158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c158a00 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f231000c480 tx=0x7f231000c940 comp rx=0 tx=0).stop 2024-04-17T10:32:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 shutdown_connections 2024-04-17T10:32:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f22fc066f60 0x7f22fc069420 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f231c15dd10 0x7f231c148950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f231c0b07b0 0x7f231c151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 --2- 172.21.15.67:0/2836250909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f231c071300 0x7f231c158a00 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 >> 172.21.15.67:0/2836250909 conn(0x7f231c0b34d0 msgr2=0x7f231c0b5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 shutdown_connections 2024-04-17T10:32:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.542+0000 7f2324400640 1 -- 172.21.15.67:0/2836250909 wait complete. 2024-04-17T10:32:38.555 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-04-17T10:32:38.246524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":409608,"kb_used_data":11592,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754565112,"statfs":{"total":773094113280,"available":772674674688,"internally_reserved":0,"allocated":11870208,"data_stored":6142356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":407424561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001175"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602605+0000","last_change":"2024-04-17T10:32:33.951718+0000","last_active":"2024-04-17T10:32:35.602605+0000","last_peered":"2024-04-17T10:32:35.602605+0000","last_clean":"2024-04-17T10:32:35.602605+0000","last_became_active":"2024-04-17T10:32:33.951303+0000","last_became_peered":"2024-04-17T10:32:33.951303+0000","last_unstale":"2024-04-17T10:32:35.602605+0000","last_undegraded":"2024-04-17T10:32:35.602605+0000","last_fullsized":"2024-04-17T10:32:35.602605+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:25.167725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598106+0000","last_change":"2024-04-17T10:32:33.941848+0000","last_active":"2024-04-17T10:32:35.598106+0000","last_peered":"2024-04-17T10:32:35.598106+0000","last_clean":"2024-04-17T10:32:35.598106+0000","last_became_active":"2024-04-17T10:32:33.940759+0000","last_became_peered":"2024-04-17T10:32:33.940759+0000","last_unstale":"2024-04-17T10:32:35.598106+0000","last_undegraded":"2024-04-17T10:32:35.598106+0000","last_fullsized":"2024-04-17T10:32:35.598106+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:31.042993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602053+0000","last_change":"2024-04-17T10:32:33.951893+0000","last_active":"2024-04-17T10:32:35.602053+0000","last_peered":"2024-04-17T10:32:35.602053+0000","last_clean":"2024-04-17T10:32:35.602053+0000","last_became_active":"2024-04-17T10:32:33.951342+0000","last_became_peered":"2024-04-17T10:32:33.951342+0000","last_unstale":"2024-04-17T10:32:35.602053+0000","last_undegraded":"2024-04-17T10:32:35.602053+0000","last_fullsized":"2024-04-17T10:32:35.602053+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:43:36.272954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601336+0000","last_change":"2024-04-17T10:32:33.934755+0000","last_active":"2024-04-17T10:32:35.601336+0000","last_peered":"2024-04-17T10:32:35.601336+0000","last_clean":"2024-04-17T10:32:35.601336+0000","last_became_active":"2024-04-17T10:32:33.933830+0000","last_became_peered":"2024-04-17T10:32:33.933830+0000","last_unstale":"2024-04-17T10:32:35.601336+0000","last_undegraded":"2024-04-17T10:32:35.601336+0000","last_fullsized":"2024-04-17T10:32:35.601336+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:22.189254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935507+0000","last_change":"2024-04-17T10:32:33.933412+0000","last_active":"2024-04-17T10:32:34.935507+0000","last_peered":"2024-04-17T10:32:34.935507+0000","last_clean":"2024-04-17T10:32:34.935507+0000","last_became_active":"2024-04-17T10:32:33.932791+0000","last_became_peered":"2024-04-17T10:32:33.932791+0000","last_unstale":"2024-04-17T10:32:34.935507+0000","last_undegraded":"2024-04-17T10:32:34.935507+0000","last_fullsized":"2024-04-17T10:32:34.935507+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:35:52.003511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597565+0000","last_change":"2024-04-17T10:32:33.938728+0000","last_active":"2024-04-17T10:32:35.597565+0000","last_peered":"2024-04-17T10:32:35.597565+0000","last_clean":"2024-04-17T10:32:35.597565+0000","last_became_active":"2024-04-17T10:32:33.938455+0000","last_became_peered":"2024-04-17T10:32:33.938455+0000","last_unstale":"2024-04-17T10:32:35.597565+0000","last_undegraded":"2024-04-17T10:32:35.597565+0000","last_fullsized":"2024-04-17T10:32:35.597565+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:35:52.003511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598052+0000","last_change":"2024-04-17T10:32:33.943180+0000","last_active":"2024-04-17T10:32:35.598052+0000","last_peered":"2024-04-17T10:32:35.598052+0000","last_clean":"2024-04-17T10:32:35.598052+0000","last_became_active":"2024-04-17T10:32:33.940659+0000","last_became_peered":"2024-04-17T10:32:33.940659+0000","last_unstale":"2024-04-17T10:32:35.598052+0000","last_undegraded":"2024-04-17T10:32:35.598052+0000","last_fullsized":"2024-04-17T10:32:35.598052+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:39.303969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.600883+0000","last_change":"2024-04-17T10:32:33.944760+0000","last_active":"2024-04-17T10:32:35.600883+0000","last_peered":"2024-04-17T10:32:35.600883+0000","last_clean":"2024-04-17T10:32:35.600883+0000","last_became_active":"2024-04-17T10:32:33.943470+0000","last_became_peered":"2024-04-17T10:32:33.943470+0000","last_unstale":"2024-04-17T10:32:35.600883+0000","last_undegraded":"2024-04-17T10:32:35.600883+0000","last_fullsized":"2024-04-17T10:32:35.600883+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:13.816941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598067+0000","last_change":"2024-04-17T10:32:33.936467+0000","last_active":"2024-04-17T10:32:35.598067+0000","last_peered":"2024-04-17T10:32:35.598067+0000","last_clean":"2024-04-17T10:32:35.598067+0000","last_became_active":"2024-04-17T10:32:33.936121+0000","last_became_peered":"2024-04-17T10:32:33.936121+0000","last_unstale":"2024-04-17T10:32:35.598067+0000","last_undegraded":"2024-04-17T10:32:35.598067+0000","last_fullsized":"2024-04-17T10:32:35.598067+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:39:16.578853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935911+0000","last_change":"2024-04-17T10:32:33.940619+0000","last_active":"2024-04-17T10:32:34.935911+0000","last_peered":"2024-04-17T10:32:34.935911+0000","last_clean":"2024-04-17T10:32:34.935911+0000","last_became_active":"2024-04-17T10:32:33.940315+0000","last_became_peered":"2024-04-17T10:32:33.940315+0000","last_unstale":"2024-04-17T10:32:34.935911+0000","last_undegraded":"2024-04-17T10:32:34.935911+0000","last_fullsized":"2024-04-17T10:32:34.935911+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:54:13.697953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935522+0000","last_change":"2024-04-17T10:32:33.945074+0000","last_active":"2024-04-17T10:32:34.935522+0000","last_peered":"2024-04-17T10:32:34.935522+0000","last_clean":"2024-04-17T10:32:34.935522+0000","last_became_active":"2024-04-17T10:32:33.944784+0000","last_became_peered":"2024-04-17T10:32:33.944784+0000","last_unstale":"2024-04-17T10:32:34.935522+0000","last_undegraded":"2024-04-17T10:32:34.935522+0000","last_fullsized":"2024-04-17T10:32:34.935522+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:04.200987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936744+0000","last_change":"2024-04-17T10:32:33.932942+0000","last_active":"2024-04-17T10:32:34.936744+0000","last_peered":"2024-04-17T10:32:34.936744+0000","last_clean":"2024-04-17T10:32:34.936744+0000","last_became_active":"2024-04-17T10:32:33.932099+0000","last_became_peered":"2024-04-17T10:32:33.932099+0000","last_unstale":"2024-04-17T10:32:34.936744+0000","last_undegraded":"2024-04-17T10:32:34.936744+0000","last_fullsized":"2024-04-17T10:32:34.936744+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:24:07.856261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601099+0000","last_change":"2024-04-17T10:32:33.952243+0000","last_active":"2024-04-17T10:32:35.601099+0000","last_peered":"2024-04-17T10:32:35.601099+0000","last_clean":"2024-04-17T10:32:35.601099+0000","last_became_active":"2024-04-17T10:32:33.951976+0000","last_became_peered":"2024-04-17T10:32:33.951976+0000","last_unstale":"2024-04-17T10:32:35.601099+0000","last_undegraded":"2024-04-17T10:32:35.601099+0000","last_fullsized":"2024-04-17T10:32:35.601099+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:04.200987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597812+0000","last_change":"2024-04-17T10:32:33.947491+0000","last_active":"2024-04-17T10:32:35.597812+0000","last_peered":"2024-04-17T10:32:35.597812+0000","last_clean":"2024-04-17T10:32:35.597812+0000","last_became_active":"2024-04-17T10:32:33.945475+0000","last_became_peered":"2024-04-17T10:32:33.945475+0000","last_unstale":"2024-04-17T10:32:35.597812+0000","last_undegraded":"2024-04-17T10:32:35.597812+0000","last_fullsized":"2024-04-17T10:32:35.597812+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:23:48.123522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597580+0000","last_change":"2024-04-17T10:32:33.943405+0000","last_active":"2024-04-17T10:32:35.597580+0000","last_peered":"2024-04-17T10:32:35.597580+0000","last_clean":"2024-04-17T10:32:35.597580+0000","last_became_active":"2024-04-17T10:32:33.943135+0000","last_became_peered":"2024-04-17T10:32:33.943135+0000","last_unstale":"2024-04-17T10:32:35.597580+0000","last_undegraded":"2024-04-17T10:32:35.597580+0000","last_fullsized":"2024-04-17T10:32:35.597580+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:04:00.118573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597552+0000","last_change":"2024-04-17T10:32:33.947700+0000","last_active":"2024-04-17T10:32:35.597552+0000","last_peered":"2024-04-17T10:32:35.597552+0000","last_clean":"2024-04-17T10:32:35.597552+0000","last_became_active":"2024-04-17T10:32:33.947427+0000","last_became_peered":"2024-04-17T10:32:33.947427+0000","last_unstale":"2024-04-17T10:32:35.597552+0000","last_undegraded":"2024-04-17T10:32:35.597552+0000","last_fullsized":"2024-04-17T10:32:35.597552+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:12:02.375197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602754+0000","last_change":"2024-04-17T10:32:33.956281+0000","last_active":"2024-04-17T10:32:35.602754+0000","last_peered":"2024-04-17T10:32:35.602754+0000","last_clean":"2024-04-17T10:32:35.602754+0000","last_became_active":"2024-04-17T10:32:33.955930+0000","last_became_peered":"2024-04-17T10:32:33.955930+0000","last_unstale":"2024-04-17T10:32:35.602754+0000","last_undegraded":"2024-04-17T10:32:35.602754+0000","last_fullsized":"2024-04-17T10:32:35.602754+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:32:58.907652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598148+0000","last_change":"2024-04-17T10:32:33.942383+0000","last_active":"2024-04-17T10:32:35.598148+0000","last_peered":"2024-04-17T10:32:35.598148+0000","last_clean":"2024-04-17T10:32:35.598148+0000","last_became_active":"2024-04-17T10:32:33.941324+0000","last_became_peered":"2024-04-17T10:32:33.941324+0000","last_unstale":"2024-04-17T10:32:35.598148+0000","last_undegraded":"2024-04-17T10:32:35.598148+0000","last_fullsized":"2024-04-17T10:32:35.598148+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:11:24.040397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602236+0000","last_change":"2024-04-17T10:32:33.955402+0000","last_active":"2024-04-17T10:32:35.602236+0000","last_peered":"2024-04-17T10:32:35.602236+0000","last_clean":"2024-04-17T10:32:35.602236+0000","last_became_active":"2024-04-17T10:32:33.953894+0000","last_became_peered":"2024-04-17T10:32:33.953894+0000","last_unstale":"2024-04-17T10:32:35.602236+0000","last_undegraded":"2024-04-17T10:32:35.602236+0000","last_fullsized":"2024-04-17T10:32:35.602236+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:59:05.587770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936775+0000","last_change":"2024-04-17T10:32:33.935151+0000","last_active":"2024-04-17T10:32:34.936775+0000","last_peered":"2024-04-17T10:32:34.936775+0000","last_clean":"2024-04-17T10:32:34.936775+0000","last_became_active":"2024-04-17T10:32:33.934889+0000","last_became_peered":"2024-04-17T10:32:33.934889+0000","last_unstale":"2024-04-17T10:32:34.936775+0000","last_undegraded":"2024-04-17T10:32:34.936775+0000","last_fullsized":"2024-04-17T10:32:34.936775+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:42.484831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601888+0000","last_change":"2024-04-17T10:32:33.947994+0000","last_active":"2024-04-17T10:32:35.601888+0000","last_peered":"2024-04-17T10:32:35.601888+0000","last_clean":"2024-04-17T10:32:35.601888+0000","last_became_active":"2024-04-17T10:32:33.947614+0000","last_became_peered":"2024-04-17T10:32:33.947614+0000","last_unstale":"2024-04-17T10:32:35.601888+0000","last_undegraded":"2024-04-17T10:32:35.601888+0000","last_fullsized":"2024-04-17T10:32:35.601888+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:24:24.776383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935496+0000","last_change":"2024-04-17T10:32:33.944680+0000","last_active":"2024-04-17T10:32:34.935496+0000","last_peered":"2024-04-17T10:32:34.935496+0000","last_clean":"2024-04-17T10:32:34.935496+0000","last_became_active":"2024-04-17T10:32:33.944409+0000","last_became_peered":"2024-04-17T10:32:33.944409+0000","last_unstale":"2024-04-17T10:32:34.935496+0000","last_undegraded":"2024-04-17T10:32:34.935496+0000","last_fullsized":"2024-04-17T10:32:34.935496+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:55.420302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601325+0000","last_change":"2024-04-17T10:32:33.952759+0000","last_active":"2024-04-17T10:32:35.601325+0000","last_peered":"2024-04-17T10:32:35.601325+0000","last_clean":"2024-04-17T10:32:35.601325+0000","last_became_active":"2024-04-17T10:32:33.952439+0000","last_became_peered":"2024-04-17T10:32:33.952439+0000","last_unstale":"2024-04-17T10:32:35.601325+0000","last_undegraded":"2024-04-17T10:32:35.601325+0000","last_fullsized":"2024-04-17T10:32:35.601325+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:03:34.862329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935681+0000","last_change":"2024-04-17T10:32:33.954228+0000","last_active":"2024-04-17T10:32:34.935681+0000","last_peered":"2024-04-17T10:32:34.935681+0000","last_clean":"2024-04-17T10:32:34.935681+0000","last_became_active":"2024-04-17T10:32:33.953906+0000","last_became_peered":"2024-04-17T10:32:33.953906+0000","last_unstale":"2024-04-17T10:32:34.935681+0000","last_undegraded":"2024-04-17T10:32:34.935681+0000","last_fullsized":"2024-04-17T10:32:34.935681+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:16.704505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936964+0000","last_change":"2024-04-17T10:32:33.947161+0000","last_active":"2024-04-17T10:32:34.936964+0000","last_peered":"2024-04-17T10:32:34.936964+0000","last_clean":"2024-04-17T10:32:34.936964+0000","last_became_active":"2024-04-17T10:32:33.946773+0000","last_became_peered":"2024-04-17T10:32:33.946773+0000","last_unstale":"2024-04-17T10:32:34.936964+0000","last_undegraded":"2024-04-17T10:32:34.936964+0000","last_fullsized":"2024-04-17T10:32:34.936964+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:11:55.555867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936887+0000","last_change":"2024-04-17T10:32:33.939675+0000","last_active":"2024-04-17T10:32:34.936887+0000","last_peered":"2024-04-17T10:32:34.936887+0000","last_clean":"2024-04-17T10:32:34.936887+0000","last_became_active":"2024-04-17T10:32:33.939403+0000","last_became_peered":"2024-04-17T10:32:33.939403+0000","last_unstale":"2024-04-17T10:32:34.936887+0000","last_undegraded":"2024-04-17T10:32:34.936887+0000","last_fullsized":"2024-04-17T10:32:34.936887+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:32.780847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935538+0000","last_change":"2024-04-17T10:32:33.945634+0000","last_active":"2024-04-17T10:32:34.935538+0000","last_peered":"2024-04-17T10:32:34.935538+0000","last_clean":"2024-04-17T10:32:34.935538+0000","last_became_active":"2024-04-17T10:32:33.945385+0000","last_became_peered":"2024-04-17T10:32:33.945385+0000","last_unstale":"2024-04-17T10:32:34.935538+0000","last_undegraded":"2024-04-17T10:32:34.935538+0000","last_fullsized":"2024-04-17T10:32:34.935538+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:22.189254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598076+0000","last_change":"2024-04-17T10:32:33.944168+0000","last_active":"2024-04-17T10:32:35.598076+0000","last_peered":"2024-04-17T10:32:35.598076+0000","last_clean":"2024-04-17T10:32:35.598076+0000","last_became_active":"2024-04-17T10:32:33.943896+0000","last_became_peered":"2024-04-17T10:32:33.943896+0000","last_unstale":"2024-04-17T10:32:35.598076+0000","last_undegraded":"2024-04-17T10:32:35.598076+0000","last_fullsized":"2024-04-17T10:32:35.598076+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:54:13.697953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602052+0000","last_change":"2024-04-17T10:32:33.943097+0000","last_active":"2024-04-17T10:32:35.602052+0000","last_peered":"2024-04-17T10:32:35.602052+0000","last_clean":"2024-04-17T10:32:35.602052+0000","last_became_active":"2024-04-17T10:32:33.941214+0000","last_became_peered":"2024-04-17T10:32:33.941214+0000","last_unstale":"2024-04-17T10:32:35.602052+0000","last_undegraded":"2024-04-17T10:32:35.602052+0000","last_fullsized":"2024-04-17T10:32:35.602052+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:04:40.498902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936401+0000","last_change":"2024-04-17T10:32:33.944517+0000","last_active":"2024-04-17T10:32:34.936401+0000","last_peered":"2024-04-17T10:32:34.936401+0000","last_clean":"2024-04-17T10:32:34.936401+0000","last_became_active":"2024-04-17T10:32:33.944160+0000","last_became_peered":"2024-04-17T10:32:33.944160+0000","last_unstale":"2024-04-17T10:32:34.936401+0000","last_undegraded":"2024-04-17T10:32:34.936401+0000","last_fullsized":"2024-04-17T10:32:34.936401+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:57.540087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597449+0000","last_change":"2024-04-17T10:32:33.952091+0000","last_active":"2024-04-17T10:32:35.597449+0000","last_peered":"2024-04-17T10:32:35.597449+0000","last_clean":"2024-04-17T10:32:35.597449+0000","last_became_active":"2024-04-17T10:32:33.951740+0000","last_became_peered":"2024-04-17T10:32:33.951740+0000","last_unstale":"2024-04-17T10:32:35.597449+0000","last_undegraded":"2024-04-17T10:32:35.597449+0000","last_fullsized":"2024-04-17T10:32:35.597449+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:55.420302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.600961+0000","last_change":"2024-04-17T10:32:33.949692+0000","last_active":"2024-04-17T10:32:35.600961+0000","last_peered":"2024-04-17T10:32:35.600961+0000","last_clean":"2024-04-17T10:32:35.600961+0000","last_became_active":"2024-04-17T10:32:33.949395+0000","last_became_peered":"2024-04-17T10:32:33.949395+0000","last_unstale":"2024-04-17T10:32:35.600961+0000","last_undegraded":"2024-04-17T10:32:35.600961+0000","last_fullsized":"2024-04-17T10:32:35.600961+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:01:55.635710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601645+0000","last_change":"2024-04-17T10:32:33.952837+0000","last_active":"2024-04-17T10:32:35.601645+0000","last_peered":"2024-04-17T10:32:35.601645+0000","last_clean":"2024-04-17T10:32:35.601645+0000","last_became_active":"2024-04-17T10:32:33.945692+0000","last_became_peered":"2024-04-17T10:32:33.945692+0000","last_unstale":"2024-04-17T10:32:35.601645+0000","last_undegraded":"2024-04-17T10:32:35.601645+0000","last_fullsized":"2024-04-17T10:32:35.601645+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:34.424142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"106'4","reported_seq":18,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601890+0000","last_change":"2024-04-17T10:32:33.067529+0000","last_active":"2024-04-17T10:32:35.601890+0000","last_peered":"2024-04-17T10:32:35.601890+0000","last_clean":"2024-04-17T10:32:35.601890+0000","last_became_active":"2024-04-17T10:32:33.067258+0000","last_became_peered":"2024-04-17T10:32:33.067258+0000","last_unstale":"2024-04-17T10:32:35.601890+0000","last_undegraded":"2024-04-17T10:32:35.601890+0000","last_fullsized":"2024-04-17T10:32:35.601890+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:23:31.776493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601114+0000","last_change":"2024-04-17T10:32:33.943367+0000","last_active":"2024-04-17T10:32:35.601114+0000","last_peered":"2024-04-17T10:32:35.601114+0000","last_clean":"2024-04-17T10:32:35.601114+0000","last_became_active":"2024-04-17T10:32:33.943074+0000","last_became_peered":"2024-04-17T10:32:33.943074+0000","last_unstale":"2024-04-17T10:32:35.601114+0000","last_undegraded":"2024-04-17T10:32:35.601114+0000","last_fullsized":"2024-04-17T10:32:35.601114+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:32:16.921026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602192+0000","last_change":"2024-04-17T10:32:33.068213+0000","last_active":"2024-04-17T10:32:35.602192+0000","last_peered":"2024-04-17T10:32:35.602192+0000","last_clean":"2024-04-17T10:32:35.602192+0000","last_became_active":"2024-04-17T10:32:33.067897+0000","last_became_peered":"2024-04-17T10:32:33.067897+0000","last_unstale":"2024-04-17T10:32:35.602192+0000","last_undegraded":"2024-04-17T10:32:35.602192+0000","last_fullsized":"2024-04-17T10:32:35.602192+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:12:30.749292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936416+0000","last_change":"2024-04-17T10:32:33.932477+0000","last_active":"2024-04-17T10:32:34.936416+0000","last_peered":"2024-04-17T10:32:34.936416+0000","last_clean":"2024-04-17T10:32:34.936416+0000","last_became_active":"2024-04-17T10:32:33.932141+0000","last_became_peered":"2024-04-17T10:32:33.932141+0000","last_unstale":"2024-04-17T10:32:34.936416+0000","last_undegraded":"2024-04-17T10:32:34.936416+0000","last_fullsized":"2024-04-17T10:32:34.936416+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:48.846127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936632+0000","last_change":"2024-04-17T10:32:33.063299+0000","last_active":"2024-04-17T10:32:34.936632+0000","last_peered":"2024-04-17T10:32:34.936632+0000","last_clean":"2024-04-17T10:32:34.936632+0000","last_became_active":"2024-04-17T10:32:33.062218+0000","last_became_peered":"2024-04-17T10:32:33.062218+0000","last_unstale":"2024-04-17T10:32:34.936632+0000","last_undegraded":"2024-04-17T10:32:34.936632+0000","last_fullsized":"2024-04-17T10:32:34.936632+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:42:13.538907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935698+0000","last_change":"2024-04-17T10:32:33.941292+0000","last_active":"2024-04-17T10:32:34.935698+0000","last_peered":"2024-04-17T10:32:34.935698+0000","last_clean":"2024-04-17T10:32:34.935698+0000","last_became_active":"2024-04-17T10:32:33.937719+0000","last_became_peered":"2024-04-17T10:32:33.937719+0000","last_unstale":"2024-04-17T10:32:34.935698+0000","last_undegraded":"2024-04-17T10:32:34.935698+0000","last_fullsized":"2024-04-17T10:32:34.935698+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:54:18.612691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935699+0000","last_change":"2024-04-17T10:32:33.065002+0000","last_active":"2024-04-17T10:32:34.935699+0000","last_peered":"2024-04-17T10:32:34.935699+0000","last_clean":"2024-04-17T10:32:34.935699+0000","last_became_active":"2024-04-17T10:32:33.064698+0000","last_became_peered":"2024-04-17T10:32:33.064698+0000","last_unstale":"2024-04-17T10:32:34.935699+0000","last_undegraded":"2024-04-17T10:32:34.935699+0000","last_fullsized":"2024-04-17T10:32:34.935699+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:24.135829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601595+0000","last_change":"2024-04-17T10:32:33.939173+0000","last_active":"2024-04-17T10:32:35.601595+0000","last_peered":"2024-04-17T10:32:35.601595+0000","last_clean":"2024-04-17T10:32:35.601595+0000","last_became_active":"2024-04-17T10:32:33.938870+0000","last_became_peered":"2024-04-17T10:32:33.938870+0000","last_unstale":"2024-04-17T10:32:35.601595+0000","last_undegraded":"2024-04-17T10:32:35.601595+0000","last_fullsized":"2024-04-17T10:32:35.601595+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:54:18.612691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598292+0000","last_change":"2024-04-17T10:32:33.062283+0000","last_active":"2024-04-17T10:32:35.598292+0000","last_peered":"2024-04-17T10:32:35.598292+0000","last_clean":"2024-04-17T10:32:35.598292+0000","last_became_active":"2024-04-17T10:32:33.061964+0000","last_became_peered":"2024-04-17T10:32:33.061964+0000","last_unstale":"2024-04-17T10:32:35.598292+0000","last_undegraded":"2024-04-17T10:32:35.598292+0000","last_fullsized":"2024-04-17T10:32:35.598292+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:50:31.594837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602610+0000","last_change":"2024-04-17T10:32:33.951631+0000","last_active":"2024-04-17T10:32:35.602610+0000","last_peered":"2024-04-17T10:32:35.602610+0000","last_clean":"2024-04-17T10:32:35.602610+0000","last_became_active":"2024-04-17T10:32:33.950967+0000","last_became_peered":"2024-04-17T10:32:33.950967+0000","last_unstale":"2024-04-17T10:32:35.602610+0000","last_undegraded":"2024-04-17T10:32:35.602610+0000","last_fullsized":"2024-04-17T10:32:35.602610+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:51.163103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935193+0000","last_change":"2024-04-17T10:32:33.059122+0000","last_active":"2024-04-17T10:32:34.935193+0000","last_peered":"2024-04-17T10:32:34.935193+0000","last_clean":"2024-04-17T10:32:34.935193+0000","last_became_active":"2024-04-17T10:32:33.058856+0000","last_became_peered":"2024-04-17T10:32:33.058856+0000","last_unstale":"2024-04-17T10:32:34.935193+0000","last_undegraded":"2024-04-17T10:32:34.935193+0000","last_fullsized":"2024-04-17T10:32:34.935193+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:38:07.522644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598270+0000","last_change":"2024-04-17T10:32:33.945193+0000","last_active":"2024-04-17T10:32:35.598270+0000","last_peered":"2024-04-17T10:32:35.598270+0000","last_clean":"2024-04-17T10:32:35.598270+0000","last_became_active":"2024-04-17T10:32:33.944899+0000","last_became_peered":"2024-04-17T10:32:33.944899+0000","last_unstale":"2024-04-17T10:32:35.598270+0000","last_undegraded":"2024-04-17T10:32:35.598270+0000","last_fullsized":"2024-04-17T10:32:35.598270+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:06:57.540087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936467+0000","last_change":"2024-04-17T10:32:33.059086+0000","last_active":"2024-04-17T10:32:34.936467+0000","last_peered":"2024-04-17T10:32:34.936467+0000","last_clean":"2024-04-17T10:32:34.936467+0000","last_became_active":"2024-04-17T10:32:33.058822+0000","last_became_peered":"2024-04-17T10:32:33.058822+0000","last_unstale":"2024-04-17T10:32:34.936467+0000","last_undegraded":"2024-04-17T10:32:34.936467+0000","last_fullsized":"2024-04-17T10:32:34.936467+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:56.631321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601948+0000","last_change":"2024-04-17T10:32:33.953021+0000","last_active":"2024-04-17T10:32:35.601948+0000","last_peered":"2024-04-17T10:32:35.601948+0000","last_clean":"2024-04-17T10:32:35.601948+0000","last_became_active":"2024-04-17T10:32:33.952746+0000","last_became_peered":"2024-04-17T10:32:33.952746+0000","last_unstale":"2024-04-17T10:32:35.601948+0000","last_undegraded":"2024-04-17T10:32:35.601948+0000","last_fullsized":"2024-04-17T10:32:35.601948+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:12:02.375197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.600605+0000","last_change":"2024-04-17T10:32:33.067488+0000","last_active":"2024-04-17T10:32:35.600605+0000","last_peered":"2024-04-17T10:32:35.600605+0000","last_clean":"2024-04-17T10:32:35.600605+0000","last_became_active":"2024-04-17T10:32:33.067186+0000","last_became_peered":"2024-04-17T10:32:33.067186+0000","last_unstale":"2024-04-17T10:32:35.600605+0000","last_undegraded":"2024-04-17T10:32:35.600605+0000","last_fullsized":"2024-04-17T10:32:35.600605+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:24:06.851907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601606+0000","last_change":"2024-04-17T10:32:33.953069+0000","last_active":"2024-04-17T10:32:35.601606+0000","last_peered":"2024-04-17T10:32:35.601606+0000","last_clean":"2024-04-17T10:32:35.601606+0000","last_became_active":"2024-04-17T10:32:33.946984+0000","last_became_peered":"2024-04-17T10:32:33.946984+0000","last_unstale":"2024-04-17T10:32:35.601606+0000","last_undegraded":"2024-04-17T10:32:35.601606+0000","last_fullsized":"2024-04-17T10:32:35.601606+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:16.185473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936856+0000","last_change":"2024-04-17T10:32:33.063258+0000","last_active":"2024-04-17T10:32:34.936856+0000","last_peered":"2024-04-17T10:32:34.936856+0000","last_clean":"2024-04-17T10:32:34.936856+0000","last_became_active":"2024-04-17T10:32:33.062576+0000","last_became_peered":"2024-04-17T10:32:33.062576+0000","last_unstale":"2024-04-17T10:32:34.936856+0000","last_undegraded":"2024-04-17T10:32:34.936856+0000","last_fullsized":"2024-04-17T10:32:34.936856+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:31:13.636365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935405+0000","last_change":"2024-04-17T10:32:33.936608+0000","last_active":"2024-04-17T10:32:34.935405+0000","last_peered":"2024-04-17T10:32:34.935405+0000","last_clean":"2024-04-17T10:32:34.935405+0000","last_became_active":"2024-04-17T10:32:33.936313+0000","last_became_peered":"2024-04-17T10:32:33.936313+0000","last_unstale":"2024-04-17T10:32:34.935405+0000","last_undegraded":"2024-04-17T10:32:34.935405+0000","last_fullsized":"2024-04-17T10:32:34.935405+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:57.635675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602494+0000","last_change":"2024-04-17T10:32:33.055784+0000","last_active":"2024-04-17T10:32:35.602494+0000","last_peered":"2024-04-17T10:32:35.602494+0000","last_clean":"2024-04-17T10:32:35.602494+0000","last_became_active":"2024-04-17T10:32:33.055501+0000","last_became_peered":"2024-04-17T10:32:33.055501+0000","last_unstale":"2024-04-17T10:32:35.602494+0000","last_undegraded":"2024-04-17T10:32:35.602494+0000","last_fullsized":"2024-04-17T10:32:35.602494+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:49:16.704777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601394+0000","last_change":"2024-04-17T10:32:33.942725+0000","last_active":"2024-04-17T10:32:35.601394+0000","last_peered":"2024-04-17T10:32:35.601394+0000","last_clean":"2024-04-17T10:32:35.601394+0000","last_became_active":"2024-04-17T10:32:33.942424+0000","last_became_peered":"2024-04-17T10:32:33.942424+0000","last_unstale":"2024-04-17T10:32:35.601394+0000","last_undegraded":"2024-04-17T10:32:35.601394+0000","last_fullsized":"2024-04-17T10:32:35.601394+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:35:57.612701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"106'3","reported_seq":17,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598374+0000","last_change":"2024-04-17T10:32:33.060308+0000","last_active":"2024-04-17T10:32:35.598374+0000","last_peered":"2024-04-17T10:32:35.598374+0000","last_clean":"2024-04-17T10:32:35.598374+0000","last_became_active":"2024-04-17T10:32:33.060035+0000","last_became_peered":"2024-04-17T10:32:33.060035+0000","last_unstale":"2024-04-17T10:32:35.598374+0000","last_undegraded":"2024-04-17T10:32:35.598374+0000","last_fullsized":"2024-04-17T10:32:35.598374+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:16:42.317953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935687+0000","last_change":"2024-04-17T10:32:33.941088+0000","last_active":"2024-04-17T10:32:34.935687+0000","last_peered":"2024-04-17T10:32:34.935687+0000","last_clean":"2024-04-17T10:32:34.935687+0000","last_became_active":"2024-04-17T10:32:33.939622+0000","last_became_peered":"2024-04-17T10:32:33.939622+0000","last_unstale":"2024-04-17T10:32:34.935687+0000","last_undegraded":"2024-04-17T10:32:34.935687+0000","last_fullsized":"2024-04-17T10:32:34.935687+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:36:39.303969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"106'6","reported_seq":20,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597689+0000","last_change":"2024-04-17T10:32:33.064793+0000","last_active":"2024-04-17T10:32:35.597689+0000","last_peered":"2024-04-17T10:32:35.597689+0000","last_clean":"2024-04-17T10:32:35.597689+0000","last_became_active":"2024-04-17T10:32:33.064519+0000","last_became_peered":"2024-04-17T10:32:33.064519+0000","last_unstale":"2024-04-17T10:32:35.597689+0000","last_undegraded":"2024-04-17T10:32:35.597689+0000","last_fullsized":"2024-04-17T10:32:35.597689+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:37:24.135829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935459+0000","last_change":"2024-04-17T10:32:33.951159+0000","last_active":"2024-04-17T10:32:34.935459+0000","last_peered":"2024-04-17T10:32:34.935459+0000","last_clean":"2024-04-17T10:32:34.935459+0000","last_became_active":"2024-04-17T10:32:33.950841+0000","last_became_peered":"2024-04-17T10:32:33.950841+0000","last_unstale":"2024-04-17T10:32:34.935459+0000","last_undegraded":"2024-04-17T10:32:34.935459+0000","last_fullsized":"2024-04-17T10:32:34.935459+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:37.652561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935757+0000","last_change":"2024-04-17T10:32:33.071215+0000","last_active":"2024-04-17T10:32:34.935757+0000","last_peered":"2024-04-17T10:32:34.935757+0000","last_clean":"2024-04-17T10:32:34.935757+0000","last_became_active":"2024-04-17T10:32:33.070938+0000","last_became_peered":"2024-04-17T10:32:33.070938+0000","last_unstale":"2024-04-17T10:32:34.935757+0000","last_undegraded":"2024-04-17T10:32:34.935757+0000","last_fullsized":"2024-04-17T10:32:34.935757+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:45:10.899971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598297+0000","last_change":"2024-04-17T10:32:33.945863+0000","last_active":"2024-04-17T10:32:35.598297+0000","last_peered":"2024-04-17T10:32:35.598297+0000","last_clean":"2024-04-17T10:32:35.598297+0000","last_became_active":"2024-04-17T10:32:33.942294+0000","last_became_peered":"2024-04-17T10:32:33.942294+0000","last_unstale":"2024-04-17T10:32:35.598297+0000","last_undegraded":"2024-04-17T10:32:35.598297+0000","last_fullsized":"2024-04-17T10:32:35.598297+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:38:08.526998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601100+0000","last_change":"2024-04-17T10:32:33.057182+0000","last_active":"2024-04-17T10:32:35.601100+0000","last_peered":"2024-04-17T10:32:35.601100+0000","last_clean":"2024-04-17T10:32:35.601100+0000","last_became_active":"2024-04-17T10:32:33.056978+0000","last_became_peered":"2024-04-17T10:32:33.056978+0000","last_unstale":"2024-04-17T10:32:35.601100+0000","last_undegraded":"2024-04-17T10:32:35.601100+0000","last_fullsized":"2024-04-17T10:32:35.601100+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:19:41.480477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602166+0000","last_change":"2024-04-17T10:32:33.946165+0000","last_active":"2024-04-17T10:32:35.602166+0000","last_peered":"2024-04-17T10:32:35.602166+0000","last_clean":"2024-04-17T10:32:35.602166+0000","last_became_active":"2024-04-17T10:32:33.945824+0000","last_became_peered":"2024-04-17T10:32:33.945824+0000","last_unstale":"2024-04-17T10:32:35.602166+0000","last_undegraded":"2024-04-17T10:32:35.602166+0000","last_fullsized":"2024-04-17T10:32:35.602166+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:42:10.521384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601219+0000","last_change":"2024-04-17T10:32:33.064385+0000","last_active":"2024-04-17T10:32:35.601219+0000","last_peered":"2024-04-17T10:32:35.601219+0000","last_clean":"2024-04-17T10:32:35.601219+0000","last_became_active":"2024-04-17T10:32:33.064021+0000","last_became_peered":"2024-04-17T10:32:33.064021+0000","last_unstale":"2024-04-17T10:32:35.601219+0000","last_undegraded":"2024-04-17T10:32:35.601219+0000","last_fullsized":"2024-04-17T10:32:35.601219+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:28.506812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602753+0000","last_change":"2024-04-17T10:32:33.953811+0000","last_active":"2024-04-17T10:32:35.602753+0000","last_peered":"2024-04-17T10:32:35.602753+0000","last_clean":"2024-04-17T10:32:35.602753+0000","last_became_active":"2024-04-17T10:32:33.953487+0000","last_became_peered":"2024-04-17T10:32:33.953487+0000","last_unstale":"2024-04-17T10:32:35.602753+0000","last_undegraded":"2024-04-17T10:32:35.602753+0000","last_fullsized":"2024-04-17T10:32:35.602753+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:42:54.505543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"106'8","reported_seq":22,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936183+0000","last_change":"2024-04-17T10:32:33.063023+0000","last_active":"2024-04-17T10:32:34.936183+0000","last_peered":"2024-04-17T10:32:34.936183+0000","last_clean":"2024-04-17T10:32:34.936183+0000","last_became_active":"2024-04-17T10:32:33.062649+0000","last_became_peered":"2024-04-17T10:32:33.062649+0000","last_unstale":"2024-04-17T10:32:34.936183+0000","last_undegraded":"2024-04-17T10:32:34.936183+0000","last_fullsized":"2024-04-17T10:32:34.936183+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:06:28.400806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601041+0000","last_change":"2024-04-17T10:32:33.944985+0000","last_active":"2024-04-17T10:32:35.601041+0000","last_peered":"2024-04-17T10:32:35.601041+0000","last_clean":"2024-04-17T10:32:35.601041+0000","last_became_active":"2024-04-17T10:32:33.944422+0000","last_became_peered":"2024-04-17T10:32:33.944422+0000","last_unstale":"2024-04-17T10:32:35.601041+0000","last_undegraded":"2024-04-17T10:32:35.601041+0000","last_fullsized":"2024-04-17T10:32:35.601041+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:34:52.106188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597977+0000","last_change":"2024-04-17T10:32:33.060855+0000","last_active":"2024-04-17T10:32:35.597977+0000","last_peered":"2024-04-17T10:32:35.597977+0000","last_clean":"2024-04-17T10:32:35.597977+0000","last_became_active":"2024-04-17T10:32:33.060594+0000","last_became_peered":"2024-04-17T10:32:33.060594+0000","last_unstale":"2024-04-17T10:32:35.597977+0000","last_undegraded":"2024-04-17T10:32:35.597977+0000","last_fullsized":"2024-04-17T10:32:35.597977+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:52:03.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597780+0000","last_change":"2024-04-17T10:32:33.943281+0000","last_active":"2024-04-17T10:32:35.597780+0000","last_peered":"2024-04-17T10:32:35.597780+0000","last_clean":"2024-04-17T10:32:35.597780+0000","last_became_active":"2024-04-17T10:32:33.943016+0000","last_became_peered":"2024-04-17T10:32:33.943016+0000","last_unstale":"2024-04-17T10:32:35.597780+0000","last_undegraded":"2024-04-17T10:32:35.597780+0000","last_fullsized":"2024-04-17T10:32:35.597780+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:40:16.704505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"106'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598384+0000","last_change":"2024-04-17T10:32:33.061722+0000","last_active":"2024-04-17T10:32:35.598384+0000","last_peered":"2024-04-17T10:32:35.598384+0000","last_clean":"2024-04-17T10:32:35.598384+0000","last_became_active":"2024-04-17T10:32:33.061456+0000","last_became_peered":"2024-04-17T10:32:33.061456+0000","last_unstale":"2024-04-17T10:32:35.598384+0000","last_undegraded":"2024-04-17T10:32:35.598384+0000","last_fullsized":"2024-04-17T10:32:35.598384+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:47:36.648207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935694+0000","last_change":"2024-04-17T10:32:33.936999+0000","last_active":"2024-04-17T10:32:34.935694+0000","last_peered":"2024-04-17T10:32:34.935694+0000","last_clean":"2024-04-17T10:32:34.935694+0000","last_became_active":"2024-04-17T10:32:33.936673+0000","last_became_peered":"2024-04-17T10:32:33.936673+0000","last_unstale":"2024-04-17T10:32:34.935694+0000","last_undegraded":"2024-04-17T10:32:34.935694+0000","last_fullsized":"2024-04-17T10:32:34.935694+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:39:16.578853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935325+0000","last_change":"2024-04-17T10:32:33.064989+0000","last_active":"2024-04-17T10:32:34.935325+0000","last_peered":"2024-04-17T10:32:34.935325+0000","last_clean":"2024-04-17T10:32:34.935325+0000","last_became_active":"2024-04-17T10:32:33.064715+0000","last_became_peered":"2024-04-17T10:32:33.064715+0000","last_unstale":"2024-04-17T10:32:34.935325+0000","last_undegraded":"2024-04-17T10:32:34.935325+0000","last_fullsized":"2024-04-17T10:32:34.935325+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:26:47.841773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602174+0000","last_change":"2024-04-17T10:32:33.951835+0000","last_active":"2024-04-17T10:32:35.602174+0000","last_peered":"2024-04-17T10:32:35.602174+0000","last_clean":"2024-04-17T10:32:35.602174+0000","last_became_active":"2024-04-17T10:32:33.951185+0000","last_became_peered":"2024-04-17T10:32:33.951185+0000","last_unstale":"2024-04-17T10:32:35.602174+0000","last_undegraded":"2024-04-17T10:32:35.602174+0000","last_fullsized":"2024-04-17T10:32:35.602174+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:20.800654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602103+0000","last_change":"2024-04-17T10:32:33.066217+0000","last_active":"2024-04-17T10:32:35.602103+0000","last_peered":"2024-04-17T10:32:35.602103+0000","last_clean":"2024-04-17T10:32:35.602103+0000","last_became_active":"2024-04-17T10:32:33.065802+0000","last_became_peered":"2024-04-17T10:32:33.065802+0000","last_unstale":"2024-04-17T10:32:35.602103+0000","last_undegraded":"2024-04-17T10:32:35.602103+0000","last_fullsized":"2024-04-17T10:32:35.602103+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:24:27.115573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602490+0000","last_change":"2024-04-17T10:32:33.949029+0000","last_active":"2024-04-17T10:32:35.602490+0000","last_peered":"2024-04-17T10:32:35.602490+0000","last_clean":"2024-04-17T10:32:35.602490+0000","last_became_active":"2024-04-17T10:32:33.948738+0000","last_became_peered":"2024-04-17T10:32:33.948738+0000","last_unstale":"2024-04-17T10:32:35.602490+0000","last_undegraded":"2024-04-17T10:32:35.602490+0000","last_fullsized":"2024-04-17T10:32:35.602490+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:40:25.424849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601253+0000","last_change":"2024-04-17T10:32:33.069484+0000","last_active":"2024-04-17T10:32:35.601253+0000","last_peered":"2024-04-17T10:32:35.601253+0000","last_clean":"2024-04-17T10:32:35.601253+0000","last_became_active":"2024-04-17T10:32:33.069219+0000","last_became_peered":"2024-04-17T10:32:33.069219+0000","last_unstale":"2024-04-17T10:32:35.601253+0000","last_undegraded":"2024-04-17T10:32:35.601253+0000","last_fullsized":"2024-04-17T10:32:35.601253+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:46:23.774209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601841+0000","last_change":"2024-04-17T10:32:33.942121+0000","last_active":"2024-04-17T10:32:35.601841+0000","last_peered":"2024-04-17T10:32:35.601841+0000","last_clean":"2024-04-17T10:32:35.601841+0000","last_became_active":"2024-04-17T10:32:33.941862+0000","last_became_peered":"2024-04-17T10:32:33.941862+0000","last_unstale":"2024-04-17T10:32:35.601841+0000","last_undegraded":"2024-04-17T10:32:35.601841+0000","last_fullsized":"2024-04-17T10:32:35.601841+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:06.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597517+0000","last_change":"2024-04-17T10:32:33.069280+0000","last_active":"2024-04-17T10:32:35.597517+0000","last_peered":"2024-04-17T10:32:35.597517+0000","last_clean":"2024-04-17T10:32:35.597517+0000","last_became_active":"2024-04-17T10:32:33.069023+0000","last_became_peered":"2024-04-17T10:32:33.069023+0000","last_unstale":"2024-04-17T10:32:35.597517+0000","last_undegraded":"2024-04-17T10:32:35.597517+0000","last_fullsized":"2024-04-17T10:32:35.597517+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:50.471640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602305+0000","last_change":"2024-04-17T10:32:33.942756+0000","last_active":"2024-04-17T10:32:35.602305+0000","last_peered":"2024-04-17T10:32:35.602305+0000","last_clean":"2024-04-17T10:32:35.602305+0000","last_became_active":"2024-04-17T10:32:33.942485+0000","last_became_peered":"2024-04-17T10:32:33.942485+0000","last_unstale":"2024-04-17T10:32:35.602305+0000","last_undegraded":"2024-04-17T10:32:35.602305+0000","last_fullsized":"2024-04-17T10:32:35.602305+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:47:57.843089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601013+0000","last_change":"2024-04-17T10:32:33.070408+0000","last_active":"2024-04-17T10:32:35.601013+0000","last_peered":"2024-04-17T10:32:35.601013+0000","last_clean":"2024-04-17T10:32:35.601013+0000","last_became_active":"2024-04-17T10:32:33.070117+0000","last_became_peered":"2024-04-17T10:32:33.070117+0000","last_unstale":"2024-04-17T10:32:35.601013+0000","last_undegraded":"2024-04-17T10:32:35.601013+0000","last_fullsized":"2024-04-17T10:32:35.601013+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:13:51.162015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601376+0000","last_change":"2024-04-17T10:32:33.953532+0000","last_active":"2024-04-17T10:32:35.601376+0000","last_peered":"2024-04-17T10:32:35.601376+0000","last_clean":"2024-04-17T10:32:35.601376+0000","last_became_active":"2024-04-17T10:32:33.953248+0000","last_became_peered":"2024-04-17T10:32:33.953248+0000","last_unstale":"2024-04-17T10:32:35.601376+0000","last_undegraded":"2024-04-17T10:32:35.601376+0000","last_fullsized":"2024-04-17T10:32:35.601376+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:25.988025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935945+0000","last_change":"2024-04-17T10:32:33.071794+0000","last_active":"2024-04-17T10:32:34.935945+0000","last_peered":"2024-04-17T10:32:34.935945+0000","last_clean":"2024-04-17T10:32:34.935945+0000","last_became_active":"2024-04-17T10:32:33.071554+0000","last_became_peered":"2024-04-17T10:32:33.071554+0000","last_unstale":"2024-04-17T10:32:34.935945+0000","last_undegraded":"2024-04-17T10:32:34.935945+0000","last_fullsized":"2024-04-17T10:32:34.935945+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:31.347503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936400+0000","last_change":"2024-04-17T10:32:33.933511+0000","last_active":"2024-04-17T10:32:34.936400+0000","last_peered":"2024-04-17T10:32:34.936400+0000","last_clean":"2024-04-17T10:32:34.936400+0000","last_became_active":"2024-04-17T10:32:33.933160+0000","last_became_peered":"2024-04-17T10:32:33.933160+0000","last_unstale":"2024-04-17T10:32:34.936400+0000","last_undegraded":"2024-04-17T10:32:34.936400+0000","last_fullsized":"2024-04-17T10:32:34.936400+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:38:08.526998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"107'1","reported_seq":15,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602552+0000","last_change":"2024-04-17T10:32:33.065239+0000","last_active":"2024-04-17T10:32:35.602552+0000","last_peered":"2024-04-17T10:32:35.602552+0000","last_clean":"2024-04-17T10:32:35.602552+0000","last_became_active":"2024-04-17T10:32:33.064963+0000","last_became_peered":"2024-04-17T10:32:33.064963+0000","last_unstale":"2024-04-17T10:32:35.602552+0000","last_undegraded":"2024-04-17T10:32:35.602552+0000","last_fullsized":"2024-04-17T10:32:35.602552+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:17:07.379681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601406+0000","last_change":"2024-04-17T10:32:33.939579+0000","last_active":"2024-04-17T10:32:35.601406+0000","last_peered":"2024-04-17T10:32:35.601406+0000","last_clean":"2024-04-17T10:32:35.601406+0000","last_became_active":"2024-04-17T10:32:33.939312+0000","last_became_peered":"2024-04-17T10:32:33.939312+0000","last_unstale":"2024-04-17T10:32:35.601406+0000","last_undegraded":"2024-04-17T10:32:35.601406+0000","last_fullsized":"2024-04-17T10:32:35.601406+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:17:37.404449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598238+0000","last_change":"2024-04-17T10:32:33.056490+0000","last_active":"2024-04-17T10:32:35.598238+0000","last_peered":"2024-04-17T10:32:35.598238+0000","last_clean":"2024-04-17T10:32:35.598238+0000","last_became_active":"2024-04-17T10:32:33.056180+0000","last_became_peered":"2024-04-17T10:32:33.056180+0000","last_unstale":"2024-04-17T10:32:35.598238+0000","last_undegraded":"2024-04-17T10:32:35.598238+0000","last_fullsized":"2024-04-17T10:32:35.598238+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:31.347503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935544+0000","last_change":"2024-04-17T10:32:33.941978+0000","last_active":"2024-04-17T10:32:34.935544+0000","last_peered":"2024-04-17T10:32:34.935544+0000","last_clean":"2024-04-17T10:32:34.935544+0000","last_became_active":"2024-04-17T10:32:33.941594+0000","last_became_peered":"2024-04-17T10:32:33.941594+0000","last_unstale":"2024-04-17T10:32:34.935544+0000","last_undegraded":"2024-04-17T10:32:34.935544+0000","last_fullsized":"2024-04-17T10:32:34.935544+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:17:37.404449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935739+0000","last_change":"2024-04-17T10:32:33.073386+0000","last_active":"2024-04-17T10:32:34.935739+0000","last_peered":"2024-04-17T10:32:34.935739+0000","last_clean":"2024-04-17T10:32:34.935739+0000","last_became_active":"2024-04-17T10:32:33.072505+0000","last_became_peered":"2024-04-17T10:32:33.072505+0000","last_unstale":"2024-04-17T10:32:34.935739+0000","last_undegraded":"2024-04-17T10:32:34.935739+0000","last_fullsized":"2024-04-17T10:32:34.935739+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:52:03.634821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601665+0000","last_change":"2024-04-17T10:32:33.942119+0000","last_active":"2024-04-17T10:32:35.601665+0000","last_peered":"2024-04-17T10:32:35.601665+0000","last_clean":"2024-04-17T10:32:35.601665+0000","last_became_active":"2024-04-17T10:32:33.938318+0000","last_became_peered":"2024-04-17T10:32:33.938318+0000","last_unstale":"2024-04-17T10:32:35.601665+0000","last_undegraded":"2024-04-17T10:32:35.601665+0000","last_fullsized":"2024-04-17T10:32:35.601665+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:46:30.030807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.936424+0000","last_change":"2024-04-17T10:32:33.052821+0000","last_active":"2024-04-17T10:32:34.936424+0000","last_peered":"2024-04-17T10:32:34.936424+0000","last_clean":"2024-04-17T10:32:34.936424+0000","last_became_active":"2024-04-17T10:32:33.052489+0000","last_became_peered":"2024-04-17T10:32:33.052489+0000","last_unstale":"2024-04-17T10:32:34.936424+0000","last_undegraded":"2024-04-17T10:32:34.936424+0000","last_fullsized":"2024-04-17T10:32:34.936424+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:36:54.415948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602117+0000","last_change":"2024-04-17T10:32:33.943559+0000","last_active":"2024-04-17T10:32:35.602117+0000","last_peered":"2024-04-17T10:32:35.602117+0000","last_clean":"2024-04-17T10:32:35.602117+0000","last_became_active":"2024-04-17T10:32:33.943241+0000","last_became_peered":"2024-04-17T10:32:33.943241+0000","last_unstale":"2024-04-17T10:32:35.602117+0000","last_undegraded":"2024-04-17T10:32:35.602117+0000","last_fullsized":"2024-04-17T10:32:35.602117+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:45:16.879493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601489+0000","last_change":"2024-04-17T10:32:33.063439+0000","last_active":"2024-04-17T10:32:35.601489+0000","last_peered":"2024-04-17T10:32:35.601489+0000","last_clean":"2024-04-17T10:32:35.601489+0000","last_became_active":"2024-04-17T10:32:33.063155+0000","last_became_peered":"2024-04-17T10:32:33.063155+0000","last_unstale":"2024-04-17T10:32:35.601489+0000","last_undegraded":"2024-04-17T10:32:35.601489+0000","last_fullsized":"2024-04-17T10:32:35.601489+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:06.536920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.598587+0000","last_change":"2024-04-17T10:32:33.948689+0000","last_active":"2024-04-17T10:32:35.598587+0000","last_peered":"2024-04-17T10:32:35.598587+0000","last_clean":"2024-04-17T10:32:35.598587+0000","last_became_active":"2024-04-17T10:32:33.948198+0000","last_became_peered":"2024-04-17T10:32:33.948198+0000","last_unstale":"2024-04-17T10:32:35.598587+0000","last_undegraded":"2024-04-17T10:32:35.598587+0000","last_fullsized":"2024-04-17T10:32:35.598587+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:45:58.742999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597638+0000","last_change":"2024-04-17T10:32:33.054423+0000","last_active":"2024-04-17T10:32:35.597638+0000","last_peered":"2024-04-17T10:32:35.597638+0000","last_clean":"2024-04-17T10:32:35.597638+0000","last_became_active":"2024-04-17T10:32:33.054135+0000","last_became_peered":"2024-04-17T10:32:33.054135+0000","last_unstale":"2024-04-17T10:32:35.597638+0000","last_undegraded":"2024-04-17T10:32:35.597638+0000","last_fullsized":"2024-04-17T10:32:35.597638+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:12:14.815180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935865+0000","last_change":"2024-04-17T10:32:33.948553+0000","last_active":"2024-04-17T10:32:34.935865+0000","last_peered":"2024-04-17T10:32:34.935865+0000","last_clean":"2024-04-17T10:32:34.935865+0000","last_became_active":"2024-04-17T10:32:33.947864+0000","last_became_peered":"2024-04-17T10:32:33.947864+0000","last_unstale":"2024-04-17T10:32:34.935865+0000","last_undegraded":"2024-04-17T10:32:34.935865+0000","last_fullsized":"2024-04-17T10:32:34.935865+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:25:06.520182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":14,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.601105+0000","last_change":"2024-04-17T10:32:33.063681+0000","last_active":"2024-04-17T10:32:35.601105+0000","last_peered":"2024-04-17T10:32:35.601105+0000","last_clean":"2024-04-17T10:32:35.601105+0000","last_became_active":"2024-04-17T10:32:33.063230+0000","last_became_peered":"2024-04-17T10:32:33.063230+0000","last_unstale":"2024-04-17T10:32:35.601105+0000","last_undegraded":"2024-04-17T10:32:35.601105+0000","last_fullsized":"2024-04-17T10:32:35.601105+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:03:09.416438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:34.935253+0000","last_change":"2024-04-17T10:32:33.950515+0000","last_active":"2024-04-17T10:32:34.935253+0000","last_peered":"2024-04-17T10:32:34.935253+0000","last_clean":"2024-04-17T10:32:34.935253+0000","last_became_active":"2024-04-17T10:32:33.950225+0000","last_became_peered":"2024-04-17T10:32:33.950225+0000","last_unstale":"2024-04-17T10:32:34.935253+0000","last_undegraded":"2024-04-17T10:32:34.935253+0000","last_fullsized":"2024-04-17T10:32:34.935253+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":105,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:33.037359+0000","last_clean_scrub_stamp":"2024-04-17T10:32:33.037359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:16:43.322307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"106'2","reported_seq":16,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.602228+0000","last_change":"2024-04-17T10:32:33.069848+0000","last_active":"2024-04-17T10:32:35.602228+0000","last_peered":"2024-04-17T10:32:35.602228+0000","last_clean":"2024-04-17T10:32:35.602228+0000","last_became_active":"2024-04-17T10:32:33.068353+0000","last_became_peered":"2024-04-17T10:32:33.068353+0000","last_unstale":"2024-04-17T10:32:35.602228+0000","last_undegraded":"2024-04-17T10:32:35.602228+0000","last_fullsized":"2024-04-17T10:32:35.602228+0000","mapping_epoch":104,"log_start":"0'0","ondisk_log_start":"0'0","created":104,"last_epoch_clean":105,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:32:32.033005+0000","last_clean_scrub_stamp":"2024-04-17T10:32:32.033005+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:49:05.697194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":329,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-04-17T10:32:35.597612+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:32:35.597612+0000","last_peered":"2024-04-17T10:32:35.597612+0000","last_clean":"2024-04-17T10:32:35.597612+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:32:35.597612+0000","last_undegraded":"2024-04-17T10:32:35.597612+0000","last_fullsized":"2024-04-17T10:32:35.597612+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574947,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51112,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320728,"statfs":{"total":96636764160,"available":96584425472,"internally_reserved":0,"allocated":1929216,"data_stored":1210128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574945,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50544,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321296,"statfs":{"total":96636764160,"available":96585007104,"internally_reserved":0,"allocated":1347584,"data_stored":621816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574946,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46372,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325468,"statfs":{"total":96636764160,"available":96589279232,"internally_reserved":0,"allocated":1327104,"data_stored":619600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574946,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54696,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317144,"statfs":{"total":96636764160,"available":96580755456,"internally_reserved":0,"allocated":1335296,"data_stored":619692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574946,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54696,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317144,"statfs":{"total":96636764160,"available":96580755456,"internally_reserved":0,"allocated":1331200,"data_stored":620182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574947,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50532,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321308,"statfs":{"total":96636764160,"available":96585019392,"internally_reserved":0,"allocated":1331200,"data_stored":619634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46376,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325464,"statfs":{"total":96636764160,"available":96589275136,"internally_reserved":0,"allocated":1331200,"data_stored":619634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574946,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55280,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316560,"statfs":{"total":96636764160,"available":96580157440,"internally_reserved":0,"allocated":1937408,"data_stored":1211670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:32:38.560 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:32:38.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:32:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.750+0000 7fe33c400640 1 Processor -- start 2024-04-17T10:32:38.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.750+0000 7fe33c400640 1 -- start start 2024-04-17T10:32:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe33c400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe334071ed0 0x7fe33415a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe33c400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe33415e210 0x7fe3341508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe33c400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe33415ac40 con 0x7fe3340722b0 2024-04-17T10:32:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe33c400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3340ccbf0 con 0x7fe334071ed0 2024-04-17T10:32:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe33c400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe33415af20 con 0x7fe33415e210 2024-04-17T10:32:38.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe339a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe334071ed0 0x7fe33415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe339a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe334071ed0 0x7fe33415a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46068/0 (socket says 172.21.15.67:46068) 2024-04-17T10:32:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe339a00640 1 -- 172.21.15.67:0/3701327507 learned_addr learned my addr 172.21.15.67:0/3701327507 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe33a400640 1 --2- 172.21.15.67:0/3701327507 >> v2:172.21.15.86:3300/0 conn(0x7fe33415e210 0x7fe3341508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe339a00640 1 -- 172.21.15.67:0/3701327507 >> v2:172.21.15.86:3300/0 conn(0x7fe33415e210 msgr2=0x7fe3341508d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:38.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe339a00640 1 --2- 172.21.15.67:0/3701327507 >> v2:172.21.15.86:3300/0 conn(0x7fe33415e210 0x7fe3341508d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe339a00640 1 -- 172.21.15.67:0/3701327507 >> v1:172.21.15.86:6789/0 conn(0x7fe3340722b0 legacy=0x7fe3340aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.754+0000 7fe339a00640 1 -- 172.21.15.67:0/3701327507 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33415b200 con 0x7fe334071ed0 2024-04-17T10:32:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe339a00640 1 --2- 172.21.15.67:0/3701327507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33415a560 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe33406ef50 tx=0x7fe32801e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b3a2d6f244e550 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe333e00640 1 -- 172.21.15.67:0/3701327507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280b2070 con 0x7fe334071ed0 2024-04-17T10:32:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe333e00640 1 -- 172.21.15.67:0/3701327507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe328004230 con 0x7fe334071ed0 2024-04-17T10:32:38.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe333e00640 1 -- 172.21.15.67:0/3701327507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328003620 con 0x7fe334071ed0 2024-04-17T10:32:38.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- 172.21.15.67:0/3701327507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 msgr2=0x7fe33415a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 --2- 172.21.15.67:0/3701327507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33415a560 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fe33406ef50 tx=0x7fe32801e9d0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- 172.21.15.67:0/3701327507 shutdown_connections 2024-04-17T10:32:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 --2- 172.21.15.67:0/3701327507 >> v2:172.21.15.86:3300/0 conn(0x7fe33415e210 0x7fe3341508d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 --2- 172.21.15.67:0/3701327507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33415a560 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- 172.21.15.67:0/3701327507 >> 172.21.15.67:0/3701327507 conn(0x7fe3340b1740 msgr2=0x7fe3340b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- 172.21.15.67:0/3701327507 shutdown_connections 2024-04-17T10:32:38.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- 172.21.15.67:0/3701327507 wait complete. 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 Processor -- start 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- start start 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33414b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3340722b0 0x7fe334149a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe33415e210 0x7fe3340b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340ce420 con 0x7fe334071ed0 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340ce2a0 con 0x7fe33415e210 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe33c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3340ce5a0 con 0x7fe3340722b0 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe339a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33414b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe339a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33414b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46082/0 (socket says 172.21.15.67:46082) 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.758+0000 7fe339a00640 1 -- 172.21.15.67:0/2548712736 learned_addr learned my addr 172.21.15.67:0/2548712736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:38.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe33a400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe33415e210 0x7fe3340b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339000640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3340722b0 0x7fe334149a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339a00640 1 -- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3340722b0 msgr2=0x7fe334149a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339a00640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3340722b0 0x7fe334149a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339a00640 1 -- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe33415e210 msgr2=0x7fe3340b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339a00640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe33415e210 0x7fe3340b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339a00640 1 -- 172.21.15.67:0/2548712736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33414a460 con 0x7fe334071ed0 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe33a400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe33415e210 0x7fe3340b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339000640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3340722b0 0x7fe334149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe339a00640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33414b500 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe3280b0040 tx=0x7fe32801b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe331600640 1 -- 172.21.15.67:0/2548712736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3280b2040 con 0x7fe334071ed0 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe331600640 1 -- 172.21.15.67:0/2548712736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe328004030 con 0x7fe334071ed0 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe331600640 1 -- 172.21.15.67:0/2548712736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe328006de0 con 0x7fe334071ed0 2024-04-17T10:32:38.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33414a740 con 0x7fe334071ed0 2024-04-17T10:32:38.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33414ad00 con 0x7fe334071ed0 2024-04-17T10:32:38.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33404ef30 con 0x7fe334071ed0 2024-04-17T10:32:38.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe331600640 1 -- 172.21.15.67:0/2548712736 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe3280c2020 con 0x7fe334071ed0 2024-04-17T10:32:38.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe331600640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe314066f60 0x7fe314069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:38.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.762+0000 7fe331600640 1 -- 172.21.15.67:0/2548712736 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe328121480 con 0x7fe334071ed0 2024-04-17T10:32:38.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.766+0000 7fe339000640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe314066f60 0x7fe314069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:38.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.766+0000 7fe339000640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe314066f60 0x7fe314069420 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe3340ae960 tx=0x7fe320009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:38.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.766+0000 7fe331600640 1 -- 172.21.15.67:0/2548712736 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe3280c9020 con 0x7fe334071ed0 2024-04-17T10:32:38.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.986+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe334165cd0 con 0x7fe334071ed0 2024-04-17T10:32:38.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe331600640 1 -- 172.21.15.67:0/2548712736 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fe3280a8040 con 0x7fe334071ed0 2024-04-17T10:32:38.982 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:32:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:32:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:32:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:32:38.982 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:32:38.982 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:32:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe314066f60 msgr2=0x7fe314069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe314066f60 0x7fe314069420 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fe3340ae960 tx=0x7fe320009040 comp rx=0 tx=0).stop 2024-04-17T10:32:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 msgr2=0x7fe33414b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33414b500 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fe3280b0040 tx=0x7fe32801b770 comp rx=0 tx=0).stop 2024-04-17T10:32:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 shutdown_connections 2024-04-17T10:32:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe314066f60 0x7fe314069420 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe33415e210 0x7fe3340b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3340722b0 0x7fe334149a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 --2- 172.21.15.67:0/2548712736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe334071ed0 0x7fe33414b500 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:38.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.990+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 >> 172.21.15.67:0/2548712736 conn(0x7fe3340b1740 msgr2=0x7fe3340ce960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:38.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.994+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 shutdown_connections 2024-04-17T10:32:38.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:38.994+0000 7fe33c400640 1 -- 172.21.15.67:0/2548712736 wait complete. 2024-04-17T10:32:39.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:32:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71f200640 1 Processor -- start 2024-04-17T10:32:39.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71f200640 1 -- start start 2024-04-17T10:32:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71f200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa71815dd10 0x7fa7180b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa71806c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71f200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7180b0cc0 con 0x7fa7180b57b0 2024-04-17T10:32:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71f200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7180b1280 con 0x7fa71815dd10 2024-04-17T10:32:39.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7180b0fa0 con 0x7fa7180714d0 2024-04-17T10:32:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa71806c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa71806c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:50198/0 (socket says 172.21.15.67:50198) 2024-04-17T10:32:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71d200640 1 -- 172.21.15.67:0/1543604235 learned_addr learned my addr 172.21.15.67:0/1543604235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa717e00640 1 --2- 172.21.15.67:0/1543604235 >> v2:172.21.15.86:3300/0 conn(0x7fa71815dd10 0x7fa7180b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71d200640 1 -- 172.21.15.67:0/1543604235 >> v2:172.21.15.86:3300/0 conn(0x7fa71815dd10 msgr2=0x7fa7180b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71d200640 1 --2- 172.21.15.67:0/1543604235 >> v2:172.21.15.86:3300/0 conn(0x7fa71815dd10 0x7fa7180b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71d200640 1 -- 172.21.15.67:0/1543604235 >> v1:172.21.15.86:6789/0 conn(0x7fa7180b57b0 legacy=0x7fa718070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.154+0000 7fa71d200640 1 -- 172.21.15.67:0/1543604235 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71806bed0 con 0x7fa7180714d0 2024-04-17T10:32:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71d200640 1 --2- 172.21.15.67:0/1543604235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa71806c6d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa710009e50 tx=0x7fa71001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fe422f2d5d6d262 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa717400640 1 -- 172.21.15.67:0/1543604235 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7100a5810 con 0x7fa7180714d0 2024-04-17T10:32:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa717400640 1 -- 172.21.15.67:0/1543604235 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7100a59b0 con 0x7fa7180714d0 2024-04-17T10:32:39.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa717400640 1 -- 172.21.15.67:0/1543604235 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7100a5d80 con 0x7fa7180714d0 2024-04-17T10:32:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 -- 172.21.15.67:0/1543604235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 msgr2=0x7fa71806c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 --2- 172.21.15.67:0/1543604235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa71806c6d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa710009e50 tx=0x7fa71001c970 comp rx=0 tx=0).stop 2024-04-17T10:32:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 -- 172.21.15.67:0/1543604235 shutdown_connections 2024-04-17T10:32:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 --2- 172.21.15.67:0/1543604235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa71806c6d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 --2- 172.21.15.67:0/1543604235 >> v2:172.21.15.86:3300/0 conn(0x7fa71815dd10 0x7fa7180b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 -- 172.21.15.67:0/1543604235 >> 172.21.15.67:0/1543604235 conn(0x7fa7180b34d0 msgr2=0x7fa7180b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 -- 172.21.15.67:0/1543604235 shutdown_connections 2024-04-17T10:32:39.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.158+0000 7fa71f200640 1 -- 172.21.15.67:0/1543604235 wait complete. 2024-04-17T10:32:39.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 Processor -- start 2024-04-17T10:32:39.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 -- start start 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa718151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71815dd10 0x7fa718153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718153e50 0x7fa71815a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71806d7b0 con 0x7fa71815dd10 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa71806d630 con 0x7fa718153e50 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa71806d930 con 0x7fa7180714d0 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71815dd10 0x7fa718153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71815dd10 0x7fa718153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46096/0 (socket says 172.21.15.67:46096) 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 -- 172.21.15.67:0/2136129809 learned_addr learned my addr 172.21.15.67:0/2136129809 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa717e00640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa718151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71d200640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718153e50 0x7fa71815a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 -- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 msgr2=0x7fa718151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa718151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 -- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718153e50 msgr2=0x7fa71815a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718153e50 0x7fa71815a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 -- 172.21.15.67:0/2136129809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa718157fe0 con 0x7fa71815dd10 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa70fe00640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71815dd10 0x7fa718153910 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c420 tx=0x7fa70000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa714c00640 1 -- 172.21.15.67:0/2136129809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7000052d0 con 0x7fa71815dd10 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa714c00640 1 -- 172.21.15.67:0/2136129809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa700005470 con 0x7fa71815dd10 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa714c00640 1 -- 172.21.15.67:0/2136129809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa700007260 con 0x7fa71815dd10 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7181582c0 con 0x7fa71815dd10 2024-04-17T10:32:39.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.162+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7180cafc0 con 0x7fa71815dd10 2024-04-17T10:32:39.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.166+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa718070bb0 con 0x7fa71815dd10 2024-04-17T10:32:39.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.166+0000 7fa714c00640 1 -- 172.21.15.67:0/2136129809 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa700007400 con 0x7fa71815dd10 2024-04-17T10:32:39.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.166+0000 7fa714c00640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6f4066f60 0x7fa6f4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.166+0000 7fa714c00640 1 -- 172.21.15.67:0/2136129809 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa7000846b0 con 0x7fa71815dd10 2024-04-17T10:32:39.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.166+0000 7fa717e00640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6f4066f60 0x7fa6f4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.166+0000 7fa717e00640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6f4066f60 0x7fa6f4069420 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa718155420 tx=0x7fa708009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.170+0000 7fa714c00640 1 -- 172.21.15.67:0/2136129809 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa700054e40 con 0x7fa71815dd10 2024-04-17T10:32:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa718074e30 con 0x7fa71815dd10 2024-04-17T10:32:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa714c00640 1 -- 172.21.15.67:0/2136129809 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7fa700058ee0 con 0x7fa71815dd10 2024-04-17T10:32:39.358 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:32:39.358 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:32:39.358 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:32:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6f4066f60 msgr2=0x7fa6f4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6f4066f60 0x7fa6f4069420 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa718155420 tx=0x7fa708009190 comp rx=0 tx=0).stop 2024-04-17T10:32:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71815dd10 msgr2=0x7fa718153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71815dd10 0x7fa718153910 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fa70000c420 tx=0x7fa70000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 shutdown_connections 2024-04-17T10:32:39.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa6f4066f60 0x7fa6f4069420 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa718153e50 0x7fa71815a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa71815dd10 0x7fa718153910 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 --2- 172.21.15.67:0/2136129809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7180714d0 0x7fa718151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 >> 172.21.15.67:0/2136129809 conn(0x7fa7180b34d0 msgr2=0x7fa718073700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 shutdown_connections 2024-04-17T10:32:39.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.366+0000 7fa71f200640 1 -- 172.21.15.67:0/2136129809 wait complete. 2024-04-17T10:32:39.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 73, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:32:33.448829+0000', 'modified': '2024-04-17T10:32:36.459180+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8139}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8139': {'gid': 8139, 'name': 'a', 'rank': 0, 'incarnation': 69, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6834/2012054720', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 2012054720}, {'type': 'v1', 'addr': '172.21.15.67:6834', 'nonce': 2012054720}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:32:39.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:32:39.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-17T10:32:39.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f2476200640 1 Processor -- start 2024-04-17T10:32:39.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f2476200640 1 -- start start 2024-04-17T10:32:39.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f2476200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2470145040 0x7f247006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f2476200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247006e2f0 con 0x7f2470071a20 2024-04-17T10:32:39.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f2476200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247006d6e0 con 0x7f247006ef10 2024-04-17T10:32:39.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f2476200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24700a86c0 con 0x7f2470145040 2024-04-17T10:32:39.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f246fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2470071a20 0x7f24700cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57602/0 (socket says 172.21.15.67:57602) 2024-04-17T10:32:39.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f246fe00640 1 -- 172.21.15.67:0/114027585 learned_addr learned my addr 172.21.15.67:0/114027585 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:39.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.934+0000 7f246f400640 1 --2- 172.21.15.67:0/114027585 >> v2:172.21.15.86:3300/0 conn(0x7f2470145040 0x7f247006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246ea00640 1 -- 172.21.15.67:0/114027585 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034777121 0 0) 0x7f247006e2f0 con 0x7f2470071a20 2024-04-17T10:32:39.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246ea00640 1 -- 172.21.15.67:0/114027585 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454003530 con 0x7f2470071a20 2024-04-17T10:32:39.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246f400640 1 -- 172.21.15.67:0/114027585 >> v1:172.21.15.86:6789/0 conn(0x7f247006ef10 legacy=0x7f2470071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246ea00640 1 -- 172.21.15.67:0/114027585 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745370606 0 0) 0x7f247006d6e0 con 0x7f247006ef10 2024-04-17T10:32:39.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246f400640 1 -- 172.21.15.67:0/114027585 >> v1:172.21.15.67:6789/0 conn(0x7f2470071a20 legacy=0x7f24700cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246f400640 1 -- 172.21.15.67:0/114027585 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247006d9c0 con 0x7f2470145040 2024-04-17T10:32:39.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246f400640 1 --2- 172.21.15.67:0/114027585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 0x7f247006e9d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f24600097c0 tx=0x7f246001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a2d2b946e5d5bfe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246ea00640 1 -- 172.21.15.67:0/114027585 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24600b5070 con 0x7f2470145040 2024-04-17T10:32:39.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246ea00640 1 -- 172.21.15.67:0/114027585 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24600a8990 con 0x7f2470145040 2024-04-17T10:32:39.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.938+0000 7f246ea00640 1 -- 172.21.15.67:0/114027585 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24600a8d60 con 0x7f2470145040 2024-04-17T10:32:39.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- 172.21.15.67:0/114027585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 msgr2=0x7f247006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 --2- 172.21.15.67:0/114027585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 0x7f247006e9d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f24600097c0 tx=0x7f246001ec90 comp rx=0 tx=0).stop 2024-04-17T10:32:39.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- 172.21.15.67:0/114027585 shutdown_connections 2024-04-17T10:32:39.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 --2- 172.21.15.67:0/114027585 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 0x7f247006e9d0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- 172.21.15.67:0/114027585 >> 172.21.15.67:0/114027585 conn(0x7f247006ac90 msgr2=0x7f247006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- 172.21.15.67:0/114027585 shutdown_connections 2024-04-17T10:32:39.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- 172.21.15.67:0/114027585 wait complete. 2024-04-17T10:32:39.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 Processor -- start 2024-04-17T10:32:39.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- start start 2024-04-17T10:32:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247006ef10 0x7f24700b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2470071a20 0x7f24701f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 0x7f247015f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247006df80 con 0x7f2470071a20 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247015bf10 con 0x7f2470145040 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2476200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247006d370 con 0x7f247006ef10 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2470071a20 0x7f24701f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2470071a20 0x7f24701f2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46100/0 (socket says 172.21.15.67:46100) 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 -- 172.21.15.67:0/2797309284 learned_addr learned my addr 172.21.15.67:0/2797309284 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f246f400640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247006ef10 0x7f24700b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f246fe00640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 0x7f247015f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 -- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247006ef10 msgr2=0x7f24700b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247006ef10 0x7f24700b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 -- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 msgr2=0x7f247015f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 0x7f247015f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.942+0000 7f2466a00640 1 -- 172.21.15.67:0/2797309284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24701f6870 con 0x7f2470071a20 2024-04-17T10:32:39.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2466a00640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2470071a20 0x7f24701f2bd0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f245800c4d0 tx=0x7f245800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2467e00640 1 -- 172.21.15.67:0/2797309284 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458017070 con 0x7f2470071a20 2024-04-17T10:32:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2467e00640 1 -- 172.21.15.67:0/2797309284 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2458005480 con 0x7f2470071a20 2024-04-17T10:32:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2467e00640 1 -- 172.21.15.67:0/2797309284 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458003840 con 0x7f2470071a20 2024-04-17T10:32:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24701f3110 con 0x7f2470071a20 2024-04-17T10:32:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24701f6df0 con 0x7f2470071a20 2024-04-17T10:32:39.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2467e00640 1 -- 172.21.15.67:0/2797309284 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f245801a070 con 0x7f2470071a20 2024-04-17T10:32:39.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2467e00640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f244c066f60 0x7f244c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f2467e00640 1 -- 172.21.15.67:0/2797309284 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24580845a0 con 0x7f2470071a20 2024-04-17T10:32:39.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f246f400640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f244c066f60 0x7f244c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f247962a000 1 Processor -- start 2024-04-17T10:32:39.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.946+0000 7f247962a000 1 -- start start 2024-04-17T10:32:39.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246f400640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f244c066f60 0x7f244c069420 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f24600b2000 tx=0x7f2460008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b37d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51aed9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51aee310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c519d55f0 con 0x555c51aedf30 2024-04-17T10:32:39.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c5193cda0 con 0x555c51b38240 2024-04-17T10:32:39.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c518c5150 con 0x555c51b37920 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51aee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51aee310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46110/0 (socket says 172.21.15.67:46110) 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 -- 172.21.15.67:0/8813169 learned_addr learned my addr 172.21.15.67:0/8813169 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f2466a00640 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51aed9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246f400640 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b37d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 -- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 msgr2=0x555c51b37d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b37d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 -- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 msgr2=0x555c51aed9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51aed9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 -- 172.21.15.67:0/8813169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c518deca0 con 0x555c51aedf30 2024-04-17T10:32:39.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f246fe00640 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51aee310 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2468002050 tx=0x7f246800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdbe0d4b99e6c17 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f245fe00640 1 -- 172.21.15.67:0/8813169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468019070 con 0x555c51aedf30 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f245fe00640 1 -- 172.21.15.67:0/8813169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246800ee50 con 0x555c51aedf30 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f245fe00640 1 -- 172.21.15.67:0/8813169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468014680 con 0x555c51aedf30 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- 172.21.15.67:0/8813169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 msgr2=0x555c51aee310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51aee310 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f2468002050 tx=0x7f246800e530 comp rx=0 tx=0).stop 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- 172.21.15.67:0/8813169 shutdown_connections 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51aee310 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51aed9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 --2- 172.21.15.67:0/8813169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b37d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- 172.21.15.67:0/8813169 >> 172.21.15.67:0/8813169 conn(0x555c51af42a0 msgr2=0x555c51aec2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- 172.21.15.67:0/8813169 shutdown_connections 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- 172.21.15.67:0/8813169 wait complete. 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 -- start start 2024-04-17T10:32:39.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.950+0000 7f247962a000 1 Processor -- start 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51b3ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b3f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51af1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c518f7990 con 0x555c51aedf30 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c51938e40 con 0x555c51b38240 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c518c8900 con 0x555c51b37920 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 10 client.0 _ll_register_callbacks cb 0x555c519f01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51b3ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51b3ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46122/0 (socket says 172.21.15.67:46122) 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 -- 172.21.15.67:0/1664305167 learned_addr learned my addr 172.21.15.67:0/1664305167 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f2466a00640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b3f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246fe00640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51af1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 -- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 msgr2=0x555c51b3f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b3f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 -- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 msgr2=0x555c51af1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51af1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 -- 172.21.15.67:0/1664305167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c518f5630 con 0x555c51aedf30 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246fe00640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51af1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f246f400640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51b3ecf0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f246001c8c0 tx=0x7f246001d4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d0bceeb708729e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 1 -- 172.21.15.67:0/1664305167 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24600b5040 con 0x555c51aedf30 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 1 -- 172.21.15.67:0/1664305167 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24600bbcc0 con 0x555c51aedf30 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f246800ee50 con 0x555c51aedf30 2024-04-17T10:32:39.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 1 -- 172.21.15.67:0/1664305167 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24600d5050 con 0x555c51aedf30 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 10 client.8271 fetch_fsmap learned FSMap version 73 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x555c518c53a0 con 0x555c51aedf30 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 1 -- 172.21.15.67:0/1664305167 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246800ee50 con 0x555c51aedf30 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 1 -- 172.21.15.67:0/1664305167 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f24600d8020 con 0x555c51aedf30 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 10 client.8271 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 10 client.8271 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 20 client.8271 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3160 0x555c51b48d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 4 client.8271 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c518d8a50 con 0x555c51af3160 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f2466a00640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3160 0x555c51b48d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f2466a00640 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3160 0x555c51b48d50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3696da94f976295d server_cookie=d1cadcc8f3a214c9 in_seq=0 out_seq=0 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 10 client.8271 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:32:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 1 -- 172.21.15.67:0/1664305167 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555c518d8a50 con 0x555c51af3160 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f245cc00640 10 client.8271 handle_command_reply: tid=0 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 client.8271 shutdown 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 msgr2=0x555c51b3ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.954+0000 7f247962a000 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51b3ecf0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f246001c8c0 tx=0x7f246001d4d0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 shutdown_connections 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3160 0x555c51b48d50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51af1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51b37920 0x555c51b3f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 --2- 172.21.15.67:0/1664305167 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51aedf30 0x555c51b3ecf0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f245cc00640 0 client.8271 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 >> 172.21.15.67:0/1664305167 conn(0x555c51af42a0 msgr2=0x555c51aec9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 shutdown_connections 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 1 -- 172.21.15.67:0/1664305167 wait complete. 2024-04-17T10:32:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.958+0000 7f247962a000 20 client.8271 trim_cache size 0 max 16384 2024-04-17T10:32:39.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 Processor -- start 2024-04-17T10:32:39.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- start start 2024-04-17T10:32:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51af1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b37920 0x555c51af1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51b418a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24701ff250 con 0x555c51b37920 2024-04-17T10:32:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247013df70 con 0x555c51b38240 2024-04-17T10:32:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24701fec50 con 0x555c51aedf30 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b37920 0x555c51af1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b37920 0x555c51af1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46130/0 (socket says 172.21.15.67:46130) 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 -- 172.21.15.67:0/1401568791 learned_addr learned my addr 172.21.15.67:0/1401568791 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f246f400640 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51af1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f246fe00640 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51b418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 -- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 msgr2=0x555c51af1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51af1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 -- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 msgr2=0x555c51b418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51b418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 -- 172.21.15.67:0/1401568791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c51b38070 con 0x555c51b37920 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f2466a00640 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b37920 0x555c51af1870 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f245800ce90 tx=0x7f245801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af766d5bd23471c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f245f400640 1 -- 172.21.15.67:0/1401568791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245805b170 con 0x555c51b37920 2024-04-17T10:32:39.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f245f400640 1 -- 172.21.15.67:0/1401568791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2458051460 con 0x555c51b37920 2024-04-17T10:32:39.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f245f400640 1 -- 172.21.15.67:0/1401568791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458055250 con 0x555c51b37920 2024-04-17T10:32:39.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- 172.21.15.67:0/1401568791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b37920 msgr2=0x555c51af1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b37920 0x555c51af1870 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f245800ce90 tx=0x7f245801e040 comp rx=0 tx=0).stop 2024-04-17T10:32:39.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- 172.21.15.67:0/1401568791 shutdown_connections 2024-04-17T10:32:39.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b38240 0x555c51b418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b37920 0x555c51af1870 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 --2- 172.21.15.67:0/1401568791 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51af1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- 172.21.15.67:0/1401568791 >> 172.21.15.67:0/1401568791 conn(0x555c51af42a0 msgr2=0x555c51b24010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- 172.21.15.67:0/1401568791 shutdown_connections 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- 172.21.15.67:0/1401568791 wait complete. 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 -- start start 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.974+0000 7f247962a000 1 Processor -- start 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51b448a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b37920 0x555c51b98460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b38240 0x555c51b989a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c519413d0 con 0x555c51b38240 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c519f8d50 con 0x555c51b37920 2024-04-17T10:32:39.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x555c518d7980 con 0x555c51aedf30 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 10 client.0 _ll_register_callbacks cb 0x7f2468001ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b38240 0x555c51b989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b38240 0x555c51b989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46142/0 (socket says 172.21.15.67:46142) 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 -- 172.21.15.67:0/3511655930 learned_addr learned my addr 172.21.15.67:0/3511655930 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246f400640 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51b448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f2466a00640 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b37920 0x555c51b98460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 -- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 msgr2=0x555c51b448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51b448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 -- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b37920 msgr2=0x555c51b98460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b37920 0x555c51b98460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 -- 172.21.15.67:0/3511655930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c519be600 con 0x555c51b38240 2024-04-17T10:32:39.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246fe00640 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b38240 0x555c51b989a0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f246800ea30 tx=0x7f2468001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6040fa3912d52a74 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 1 -- 172.21.15.67:0/3511655930 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468024020 con 0x555c51b38240 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 1 -- 172.21.15.67:0/3511655930 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24680149f0 con 0x555c51b38240 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2458051460 con 0x555c51b38240 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 1 -- 172.21.15.67:0/3511655930 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2468019040 con 0x555c51b38240 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 10 client.8277 fetch_fsmap learned FSMap version 73 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x555c518d19b0 con 0x555c51b38240 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 1 -- 172.21.15.67:0/3511655930 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2458051460 con 0x555c51b38240 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 1 -- 172.21.15.67:0/3511655930 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f246801f040 con 0x555c51b38240 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 10 client.8277 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:32:39.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 10 client.8277 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 20 client.8277 populate_metadata read hostname 'smithi067' 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3e20 0x555c51b98ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 4 client.8277 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555c5194a2e0 con 0x555c51af3e20 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246f400640 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3e20 0x555c51b98ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 10 client.8277 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f246f400640 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3e20 0x555c51b98ee0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51d5a95afd6ddfd8 server_cookie=1716c38cd2a60718 in_seq=0 out_seq=0 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 1 -- 172.21.15.67:0/3511655930 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x555c5194a2e0 con 0x555c51af3e20 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 10 client.8277 handle_command_reply: tid=0 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 client.8277 shutdown 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b38240 msgr2=0x555c51b989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b38240 0x555c51b989a0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f246800ea30 tx=0x7f2468001260 comp rx=0 tx=0).stop 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 shutdown_connections 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x555c51af3e20 0x555c51b98ee0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x555c51b38240 0x555c51b989a0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x555c51b37920 0x555c51b98460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 --2- 172.21.15.67:0/3511655930 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x555c51aedf30 0x555c51b448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f245cc00640 0 client.8277 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.978+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 >> 172.21.15.67:0/3511655930 conn(0x555c51af42a0 msgr2=0x555c51af0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 shutdown_connections 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f247962a000 1 -- 172.21.15.67:0/3511655930 wait complete. 2024-04-17T10:32:39.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f247962a000 20 client.8277 trim_cache size 0 max 16384 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 8139, 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 5.9357017130000003, 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 73, 2024-04-17T10:32:39.978 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 107, 2024-04-17T10:32:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 107, 2024-04-17T10:32:39.979 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 10.818194126 2024-04-17T10:32:39.979 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f244c066f60 msgr2=0x7f244c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f244c066f60 0x7f244c069420 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f24600b2000 tx=0x7f2460008040 comp rx=0 tx=0).stop 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2470071a20 msgr2=0x7f24701f2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2470071a20 0x7f24701f2bd0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f245800c4d0 tx=0x7f245800c990 comp rx=0 tx=0).stop 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 shutdown_connections 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f244c066f60 0x7f244c069420 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2470145040 0x7f247015f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2470071a20 0x7f24701f2bd0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 --2- 172.21.15.67:0/2797309284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f247006ef10 0x7f24700b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 >> 172.21.15.67:0/2797309284 conn(0x7f247006ac90 msgr2=0x7f247006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 shutdown_connections 2024-04-17T10:32:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:39.982+0000 7f2476200640 1 -- 172.21.15.67:0/2797309284 wait complete. 2024-04-17T10:32:39.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:32:40.000 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T10:32:40.010+0000 7ff672600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:32:40.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac13200640 1 Processor -- start 2024-04-17T10:32:40.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac13200640 1 -- start start 2024-04-17T10:32:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac13200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c15dd10 0x7fac0c0b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:40.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac13200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0b0cc0 con 0x7fac0c0714d0 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac13200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c0b0fa0 con 0x7fac0c0b57b0 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac13200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0b1280 con 0x7fac0c15dd10 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac11200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac0c0714d0 0x7fac0c06c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57604/0 (socket says 172.21.15.67:57604) 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac11200640 1 -- 172.21.15.67:0/1119046744 learned_addr learned my addr 172.21.15.67:0/1119046744 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0be00640 1 --2- 172.21.15.67:0/1119046744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c15dd10 0x7fac0c0b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178749673 0 0) 0x7fac0c0b0cc0 con 0x7fac0c0714d0 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf0003530 con 0x7fac0c0714d0 2024-04-17T10:32:40.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317773733 0 0) 0x7fac0c0b0fa0 con 0x7fac0c0b57b0 2024-04-17T10:32:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac0c0b0cc0 con 0x7fac0c0b57b0 2024-04-17T10:32:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2454665628 0 0) 0x7fabf0003530 con 0x7fac0c0714d0 2024-04-17T10:32:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0be00640 1 -- 172.21.15.67:0/1119046744 >> v1:172.21.15.86:6789/0 conn(0x7fac0c0b57b0 legacy=0x7fac0c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0be00640 1 -- 172.21.15.67:0/1119046744 >> v1:172.21.15.67:6789/0 conn(0x7fac0c0714d0 legacy=0x7fac0c06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.142+0000 7fac0be00640 1 -- 172.21.15.67:0/1119046744 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c06bed0 con 0x7fac0c15dd10 2024-04-17T10:32:40.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac0be00640 1 --2- 172.21.15.67:0/1119046744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c15dd10 0x7fac0c0b5270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0097f0 tx=0x7fabfc01c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ff600331f37d26b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:40.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0b0070 con 0x7fac0c15dd10 2024-04-17T10:32:40.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabfc0ab400 con 0x7fac0c15dd10 2024-04-17T10:32:40.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac0b400640 1 -- 172.21.15.67:0/1119046744 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac04001a00 con 0x7fac0c15dd10 2024-04-17T10:32:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac13200640 1 -- 172.21.15.67:0/1119046744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c15dd10 msgr2=0x7fac0c0b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac13200640 1 --2- 172.21.15.67:0/1119046744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c15dd10 0x7fac0c0b5270 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0097f0 tx=0x7fabfc01c340 comp rx=0 tx=0).stop 2024-04-17T10:32:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac13200640 1 -- 172.21.15.67:0/1119046744 shutdown_connections 2024-04-17T10:32:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac13200640 1 --2- 172.21.15.67:0/1119046744 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c15dd10 0x7fac0c0b5270 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac13200640 1 -- 172.21.15.67:0/1119046744 >> 172.21.15.67:0/1119046744 conn(0x7fac0c0b34d0 msgr2=0x7fac0c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac13200640 1 -- 172.21.15.67:0/1119046744 shutdown_connections 2024-04-17T10:32:40.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.146+0000 7fac13200640 1 -- 172.21.15.67:0/1119046744 wait complete. 2024-04-17T10:32:40.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 Processor -- start 2024-04-17T10:32:40.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 -- start start 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0c0714d0 0x7fac0c14ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c15dd10 0x7fac0c0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c1fa920 0x7fac0c1fadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c06da00 con 0x7fac0c15dd10 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c06d880 con 0x7fac0c0714d0 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c16bc70 con 0x7fac0c1fa920 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c15dd10 0x7fac0c0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c15dd10 0x7fac0c0cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:46144/0 (socket says 172.21.15.67:46144) 2024-04-17T10:32:40.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 -- 172.21.15.67:0/1564557119 learned_addr learned my addr 172.21.15.67:0/1564557119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac0be00640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0c0714d0 0x7fac0c14ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac11200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c1fa920 0x7fac0c1fadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 -- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c1fa920 msgr2=0x7fac0c1fadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c1fa920 0x7fac0c1fadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 -- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0c0714d0 msgr2=0x7fac0c14ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0c0714d0 0x7fac0c14ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 -- 172.21.15.67:0/1564557119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c14f0b0 con 0x7fac0c15dd10 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac11200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c1fa920 0x7fac0c1fadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac03400640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c15dd10 0x7fac0c0cd100 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c490 tx=0x7fabf400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac08c00640 1 -- 172.21.15.67:0/1564557119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf40052d0 con 0x7fac0c15dd10 2024-04-17T10:32:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac08c00640 1 -- 172.21.15.67:0/1564557119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf4005470 con 0x7fac0c15dd10 2024-04-17T10:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac08c00640 1 -- 172.21.15.67:0/1564557119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf4012870 con 0x7fac0c15dd10 2024-04-17T10:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.150+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0c14f390 con 0x7fac0c15dd10 2024-04-17T10:32:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.154+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c1532e0 con 0x7fac0c15dd10 2024-04-17T10:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.154+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c070bb0 con 0x7fac0c15dd10 2024-04-17T10:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.154+0000 7fac08c00640 1 -- 172.21.15.67:0/1564557119 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fabf4012a10 con 0x7fac0c15dd10 2024-04-17T10:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.154+0000 7fac08c00640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabe8066f60 0x7fabe8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.154+0000 7fac08c00640 1 -- 172.21.15.67:0/1564557119 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fabf4084e60 con 0x7fac0c15dd10 2024-04-17T10:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.154+0000 7fac0be00640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabe8066f60 0x7fabe8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:40.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.154+0000 7fac0be00640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabe8066f60 0x7fabe8069420 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0ad000 tx=0x7fabfc0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:40.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.158+0000 7fac08c00640 1 -- 172.21.15.67:0/1564557119 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fabf4055c40 con 0x7fac0c15dd10 2024-04-17T10:32:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.358+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fac0c073de0 con 0x7fac0c15dd10 2024-04-17T10:32:40.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.358+0000 7fac08c00640 1 -- 172.21.15.67:0/1564557119 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v73) v1 ==== 53+0+73 (secure 0 0 0) 0x7fabf4055540 con 0x7fac0c15dd10 2024-04-17T10:32:40.351 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:32:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabe8066f60 msgr2=0x7fabe8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabe8066f60 0x7fabe8069420 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fabfc0ad000 tx=0x7fabfc0a6000 comp rx=0 tx=0).stop 2024-04-17T10:32:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c15dd10 msgr2=0x7fac0c0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c15dd10 0x7fac0c0cd100 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fabf400c490 tx=0x7fabf400c950 comp rx=0 tx=0).stop 2024-04-17T10:32:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 shutdown_connections 2024-04-17T10:32:40.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fabe8066f60 0x7fabe8069420 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:40.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac0c1fa920 0x7fac0c1fadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:40.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac0c15dd10 0x7fac0c0cd100 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:40.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 --2- 172.21.15.67:0/1564557119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac0c0714d0 0x7fac0c14ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:40.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 >> 172.21.15.67:0/1564557119 conn(0x7fac0c0b34d0 msgr2=0x7fac0c0cf090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:40.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 shutdown_connections 2024-04-17T10:32:40.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:40.362+0000 7fac13200640 1 -- 172.21.15.67:0/1564557119 wait complete. 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:32:40.372 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:32:40.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.373 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:32:40.429 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:32:40.430 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:32:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:32:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:32:40.431 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:32:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:32:40.431 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:32:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:32:40.431 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41986sec preferred_lft 41986sec 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:32:40.432 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:32:40.433 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:32:40.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.434 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:32:40.489 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:32:40.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.524 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:32:40.579 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:32:40.580 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:32:40.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.581 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:32:40.581 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:40.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:32:40.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:32:40.582 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:32:40.582 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:40.677 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.677 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:32:40.690 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:32:40.690 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:32:40.691 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:32:40.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:32:40.755 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:32:40.755 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:32:40.811 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:32:40.812 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:32:40.812 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:32:40.863 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:32:40.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:32:40.953 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:32:40.953 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:32:40.968 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:32:40.968 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:32:40.968 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:32:40.968 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:32:40.968 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:32:40.972 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:32:40.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:32:40.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:40.993 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:32:41.057 INFO:teuthology.orchestra.run.smithi067.stdout:client6983 2024-04-17T10:32:41.057 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:32:41.057 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:32:41.110 INFO:tasks.cephfs.test_scrub_checks:mountpoint: /home/ubuntu/cephtest/mnt.0 2024-04-17T10:32:41.110 INFO:tasks.cephfs.test_scrub_checks:client_path: /home/ubuntu/cephtest/mnt.0/scrub_control_test_path 2024-04-17T10:32:41.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:41.111 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:41.111 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:41.111 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.0 2024-04-17T10:32:41.111 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:41.111 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:41.111 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:41.111 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197102 s, 53.2 MB/s 2024-04-17T10:32:41.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.216 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172359 s, 60.8 MB/s 2024-04-17T10:32:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176987 s, 59.2 MB/s 2024-04-17T10:32:41.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158338 s, 66.2 MB/s 2024-04-17T10:32:41.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.272 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169485 s, 61.9 MB/s 2024-04-17T10:32:41.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.289 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155969 s, 67.2 MB/s 2024-04-17T10:32:41.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147787 s, 71.0 MB/s 2024-04-17T10:32:41.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.321 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139635 s, 75.1 MB/s 2024-04-17T10:32:41.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149561 s, 70.1 MB/s 2024-04-17T10:32:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162922 s, 64.4 MB/s 2024-04-17T10:32:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.372 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140926 s, 74.4 MB/s 2024-04-17T10:32:41.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.390 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.390 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161053 s, 65.1 MB/s 2024-04-17T10:32:41.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.407 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155239 s, 67.5 MB/s 2024-04-17T10:32:41.424 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.424 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.424 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150309 s, 69.8 MB/s 2024-04-17T10:32:41.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146978 s, 71.3 MB/s 2024-04-17T10:32:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159705 s, 65.7 MB/s 2024-04-17T10:32:41.475 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.475 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.475 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149035 s, 70.4 MB/s 2024-04-17T10:32:41.491 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.492 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147279 s, 71.2 MB/s 2024-04-17T10:32:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153262 s, 68.4 MB/s 2024-04-17T10:32:41.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.527 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016806 s, 62.4 MB/s 2024-04-17T10:32:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134131 s, 78.2 MB/s 2024-04-17T10:32:41.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.558 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136906 s, 76.6 MB/s 2024-04-17T10:32:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.576 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167379 s, 62.6 MB/s 2024-04-17T10:32:41.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.594 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157641 s, 66.5 MB/s 2024-04-17T10:32:41.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137077 s, 76.5 MB/s 2024-04-17T10:32:41.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.624 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132011 s, 79.4 MB/s 2024-04-17T10:32:41.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.641 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150767 s, 69.5 MB/s 2024-04-17T10:32:41.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.660 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165261 s, 63.4 MB/s 2024-04-17T10:32:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.677 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152003 s, 69.0 MB/s 2024-04-17T10:32:41.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.702 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019879 s, 52.7 MB/s 2024-04-17T10:32:41.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.715 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145549 s, 72.0 MB/s 2024-04-17T10:32:41.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.733 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170088 s, 61.6 MB/s 2024-04-17T10:32:41.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:41.735 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:41.735 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:41.735 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.1 2024-04-17T10:32:41.735 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:41.735 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:41.735 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:41.735 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.805 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161895 s, 64.8 MB/s 2024-04-17T10:32:41.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138446 s, 75.7 MB/s 2024-04-17T10:32:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.836 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138001 s, 76.0 MB/s 2024-04-17T10:32:41.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.853 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142369 s, 73.7 MB/s 2024-04-17T10:32:41.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.869 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013918 s, 75.3 MB/s 2024-04-17T10:32:41.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.884 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134409 s, 78.0 MB/s 2024-04-17T10:32:41.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.900 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149845 s, 70.0 MB/s 2024-04-17T10:32:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.915 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133001 s, 78.8 MB/s 2024-04-17T10:32:41.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.932 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142987 s, 73.3 MB/s 2024-04-17T10:32:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015413 s, 68.0 MB/s 2024-04-17T10:32:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018988 s, 55.2 MB/s 2024-04-17T10:32:41.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:41.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:41.986 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138342 s, 75.8 MB/s 2024-04-17T10:32:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.001 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142696 s, 73.5 MB/s 2024-04-17T10:32:42.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.018 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015018 s, 69.8 MB/s 2024-04-17T10:32:42.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.034 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136926 s, 76.6 MB/s 2024-04-17T10:32:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.050 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142834 s, 73.4 MB/s 2024-04-17T10:32:42.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.067 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153681 s, 68.2 MB/s 2024-04-17T10:32:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138939 s, 75.5 MB/s 2024-04-17T10:32:42.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.099 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147089 s, 71.3 MB/s 2024-04-17T10:32:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.114 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132091 s, 79.4 MB/s 2024-04-17T10:32:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.129 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134278 s, 78.1 MB/s 2024-04-17T10:32:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013345 s, 78.6 MB/s 2024-04-17T10:32:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.162 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157791 s, 66.5 MB/s 2024-04-17T10:32:42.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.178 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144655 s, 72.5 MB/s 2024-04-17T10:32:42.194 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.194 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.194 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141412 s, 74.2 MB/s 2024-04-17T10:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.209 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137205 s, 76.4 MB/s 2024-04-17T10:32:42.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.227 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157413 s, 66.6 MB/s 2024-04-17T10:32:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.247 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179349 s, 58.5 MB/s 2024-04-17T10:32:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.266 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178061 s, 58.9 MB/s 2024-04-17T10:32:42.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.282 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143464 s, 73.1 MB/s 2024-04-17T10:32:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.299 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151223 s, 69.3 MB/s 2024-04-17T10:32:42.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140041 s, 74.9 MB/s 2024-04-17T10:32:42.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:42.317 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:42.317 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:42.317 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.2 2024-04-17T10:32:42.317 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:42.317 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:42.317 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:42.317 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:42.390 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131364 s, 79.8 MB/s 2024-04-17T10:32:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.406 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136342 s, 76.9 MB/s 2024-04-17T10:32:42.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180375 s, 58.1 MB/s 2024-04-17T10:32:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138441 s, 75.7 MB/s 2024-04-17T10:32:42.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.457 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133381 s, 78.6 MB/s 2024-04-17T10:32:42.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139591 s, 75.1 MB/s 2024-04-17T10:32:42.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144673 s, 72.5 MB/s 2024-04-17T10:32:42.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.504 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137484 s, 76.3 MB/s 2024-04-17T10:32:42.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197385 s, 53.1 MB/s 2024-04-17T10:32:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.540 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132082 s, 79.4 MB/s 2024-04-17T10:32:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.560 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179607 s, 58.4 MB/s 2024-04-17T10:32:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.580 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181032 s, 57.9 MB/s 2024-04-17T10:32:42.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.597 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156883 s, 66.8 MB/s 2024-04-17T10:32:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013841 s, 75.8 MB/s 2024-04-17T10:32:42.628 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.628 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.628 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136951 s, 76.6 MB/s 2024-04-17T10:32:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147203 s, 71.2 MB/s 2024-04-17T10:32:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150168 s, 69.8 MB/s 2024-04-17T10:32:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148851 s, 70.4 MB/s 2024-04-17T10:32:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.694 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139942 s, 74.9 MB/s 2024-04-17T10:32:42.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132598 s, 79.1 MB/s 2024-04-17T10:32:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.725 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135572 s, 77.3 MB/s 2024-04-17T10:32:42.745 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.746 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187119 s, 56.0 MB/s 2024-04-17T10:32:42.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133364 s, 78.6 MB/s 2024-04-17T10:32:42.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150152 s, 69.8 MB/s 2024-04-17T10:32:42.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.793 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135262 s, 77.5 MB/s 2024-04-17T10:32:42.810 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.810 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.810 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154044 s, 68.1 MB/s 2024-04-17T10:32:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.825 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131759 s, 79.6 MB/s 2024-04-17T10:32:42.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.842 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147128 s, 71.3 MB/s 2024-04-17T10:32:42.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.859 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.859 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151161 s, 69.4 MB/s 2024-04-17T10:32:42.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.874 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138396 s, 75.8 MB/s 2024-04-17T10:32:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.890 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.891 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014885 s, 70.4 MB/s 2024-04-17T10:32:42.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.907 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014686 s, 71.4 MB/s 2024-04-17T10:32:42.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:42.908 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:42.909 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:42.909 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.3 2024-04-17T10:32:42.909 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:42.909 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:42.909 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:42.909 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:42.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148335 s, 70.7 MB/s 2024-04-17T10:32:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145618 s, 72.0 MB/s 2024-04-17T10:32:43.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.012 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139674 s, 75.1 MB/s 2024-04-17T10:32:43.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.029 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151291 s, 69.3 MB/s 2024-04-17T10:32:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.045 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142792 s, 73.4 MB/s 2024-04-17T10:32:43.060 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142056 s, 73.8 MB/s 2024-04-17T10:32:43.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.081 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178155 s, 58.9 MB/s 2024-04-17T10:32:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131373 s, 79.8 MB/s 2024-04-17T10:32:43.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.113 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153967 s, 68.1 MB/s 2024-04-17T10:32:43.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.128 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131587 s, 79.7 MB/s 2024-04-17T10:32:43.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.144 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150318 s, 69.8 MB/s 2024-04-17T10:32:43.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.161 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148592 s, 70.6 MB/s 2024-04-17T10:32:43.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.176 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131726 s, 79.6 MB/s 2024-04-17T10:32:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135716 s, 77.3 MB/s 2024-04-17T10:32:43.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151898 s, 69.0 MB/s 2024-04-17T10:32:43.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.225 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148802 s, 70.5 MB/s 2024-04-17T10:32:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.241 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149206 s, 70.3 MB/s 2024-04-17T10:32:43.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.257 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135029 s, 77.7 MB/s 2024-04-17T10:32:43.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.272 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129865 s, 80.7 MB/s 2024-04-17T10:32:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120432 s, 87.1 MB/s 2024-04-17T10:32:43.301 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.301 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.301 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139001 s, 75.4 MB/s 2024-04-17T10:32:43.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121751 s, 86.1 MB/s 2024-04-17T10:32:43.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136676 s, 76.7 MB/s 2024-04-17T10:32:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147265 s, 71.2 MB/s 2024-04-17T10:32:43.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.362 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131238 s, 79.9 MB/s 2024-04-17T10:32:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.377 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125779 s, 83.4 MB/s 2024-04-17T10:32:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.397 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186263 s, 56.3 MB/s 2024-04-17T10:32:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.414 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155652 s, 67.4 MB/s 2024-04-17T10:32:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.430 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134112 s, 78.2 MB/s 2024-04-17T10:32:43.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.444 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128711 s, 81.5 MB/s 2024-04-17T10:32:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124196 s, 84.4 MB/s 2024-04-17T10:32:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.474 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135582 s, 77.3 MB/s 2024-04-17T10:32:43.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:43.475 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:43.475 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:43.475 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.4 2024-04-17T10:32:43.475 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:43.475 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:43.475 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:43.475 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141117 s, 74.3 MB/s 2024-04-17T10:32:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153741 s, 68.2 MB/s 2024-04-17T10:32:43.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.580 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184916 s, 56.7 MB/s 2024-04-17T10:32:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.596 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141908 s, 73.9 MB/s 2024-04-17T10:32:43.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152344 s, 68.8 MB/s 2024-04-17T10:32:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183649 s, 57.1 MB/s 2024-04-17T10:32:43.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.649 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139083 s, 75.4 MB/s 2024-04-17T10:32:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142761 s, 73.4 MB/s 2024-04-17T10:32:43.682 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.683 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.683 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157107 s, 66.7 MB/s 2024-04-17T10:32:43.702 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.703 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180819 s, 58.0 MB/s 2024-04-17T10:32:43.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.718 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132686 s, 79.0 MB/s 2024-04-17T10:32:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159271 s, 65.8 MB/s 2024-04-17T10:32:43.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142492 s, 73.6 MB/s 2024-04-17T10:32:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132248 s, 79.3 MB/s 2024-04-17T10:32:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.782 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135177 s, 77.6 MB/s 2024-04-17T10:32:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.798 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142875 s, 73.4 MB/s 2024-04-17T10:32:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.814 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144825 s, 72.4 MB/s 2024-04-17T10:32:43.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.830 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144108 s, 72.8 MB/s 2024-04-17T10:32:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.846 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013813 s, 75.9 MB/s 2024-04-17T10:32:43.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013195 s, 79.5 MB/s 2024-04-17T10:32:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150404 s, 69.7 MB/s 2024-04-17T10:32:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184201 s, 56.9 MB/s 2024-04-17T10:32:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.914 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135243 s, 77.5 MB/s 2024-04-17T10:32:43.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.931 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152083 s, 68.9 MB/s 2024-04-17T10:32:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.946 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136583 s, 76.8 MB/s 2024-04-17T10:32:43.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.962 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131472 s, 79.8 MB/s 2024-04-17T10:32:43.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185856 s, 56.4 MB/s 2024-04-17T10:32:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:43.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01514 s, 69.3 MB/s 2024-04-17T10:32:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167455 s, 62.6 MB/s 2024-04-17T10:32:44.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.034 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154782 s, 67.7 MB/s 2024-04-17T10:32:44.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.050 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134712 s, 77.8 MB/s 2024-04-17T10:32:44.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.065 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135298 s, 77.5 MB/s 2024-04-17T10:32:44.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:44.066 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:44.066 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:44.067 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.5 2024-04-17T10:32:44.067 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:44.067 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:44.067 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:44.067 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.136 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144058 s, 72.8 MB/s 2024-04-17T10:32:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.152 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140262 s, 74.8 MB/s 2024-04-17T10:32:44.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.168 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142682 s, 73.5 MB/s 2024-04-17T10:32:44.183 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.184 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.184 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143514 s, 73.1 MB/s 2024-04-17T10:32:44.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.200 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139221 s, 75.3 MB/s 2024-04-17T10:32:44.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.216 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143179 s, 73.2 MB/s 2024-04-17T10:32:44.231 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.231 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.231 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132033 s, 79.4 MB/s 2024-04-17T10:32:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151079 s, 69.4 MB/s 2024-04-17T10:32:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.263 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132427 s, 79.2 MB/s 2024-04-17T10:32:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.278 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132203 s, 79.3 MB/s 2024-04-17T10:32:44.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.294 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140214 s, 74.8 MB/s 2024-04-17T10:32:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.310 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137901 s, 76.0 MB/s 2024-04-17T10:32:44.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.327 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151381 s, 69.3 MB/s 2024-04-17T10:32:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179766 s, 58.3 MB/s 2024-04-17T10:32:44.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.362 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132528 s, 79.1 MB/s 2024-04-17T10:32:44.382 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.382 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.382 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184812 s, 56.7 MB/s 2024-04-17T10:32:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.398 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142251 s, 73.7 MB/s 2024-04-17T10:32:44.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.414 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135027 s, 77.7 MB/s 2024-04-17T10:32:44.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.429 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013459 s, 77.9 MB/s 2024-04-17T10:32:44.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152487 s, 68.8 MB/s 2024-04-17T10:32:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.463 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157166 s, 66.7 MB/s 2024-04-17T10:32:44.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.479 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140649 s, 74.6 MB/s 2024-04-17T10:32:44.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.494 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129461 s, 81.0 MB/s 2024-04-17T10:32:44.509 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.509 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.509 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129273 s, 81.1 MB/s 2024-04-17T10:32:44.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.524 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138138 s, 75.9 MB/s 2024-04-17T10:32:44.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.542 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157739 s, 66.5 MB/s 2024-04-17T10:32:44.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.558 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138115 s, 75.9 MB/s 2024-04-17T10:32:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017864 s, 58.7 MB/s 2024-04-17T10:32:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139692 s, 75.1 MB/s 2024-04-17T10:32:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140986 s, 74.4 MB/s 2024-04-17T10:32:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.624 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135437 s, 77.4 MB/s 2024-04-17T10:32:44.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132511 s, 79.1 MB/s 2024-04-17T10:32:44.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:44.641 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:44.641 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:44.641 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.6 2024-04-17T10:32:44.641 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:44.641 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:44.641 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:44.641 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:44.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136864 s, 76.6 MB/s 2024-04-17T10:32:44.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.729 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124967 s, 83.9 MB/s 2024-04-17T10:32:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.744 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136548 s, 76.8 MB/s 2024-04-17T10:32:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136912 s, 76.6 MB/s 2024-04-17T10:32:44.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.774 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121448 s, 86.3 MB/s 2024-04-17T10:32:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129442 s, 81.0 MB/s 2024-04-17T10:32:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.804 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138503 s, 75.7 MB/s 2024-04-17T10:32:44.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148428 s, 70.6 MB/s 2024-04-17T10:32:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131887 s, 79.5 MB/s 2024-04-17T10:32:44.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.852 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014409 s, 72.8 MB/s 2024-04-17T10:32:44.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.866 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129679 s, 80.9 MB/s 2024-04-17T10:32:44.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.882 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134338 s, 78.1 MB/s 2024-04-17T10:32:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179979 s, 58.3 MB/s 2024-04-17T10:32:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186712 s, 56.2 MB/s 2024-04-17T10:32:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136942 s, 76.6 MB/s 2024-04-17T10:32:44.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.954 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149205 s, 70.3 MB/s 2024-04-17T10:32:44.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144753 s, 72.4 MB/s 2024-04-17T10:32:44.988 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:44.988 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:44.988 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135658 s, 77.3 MB/s 2024-04-17T10:32:45.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.002 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146269 s, 71.7 MB/s 2024-04-17T10:32:45.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.018 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141899 s, 73.9 MB/s 2024-04-17T10:32:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154874 s, 67.7 MB/s 2024-04-17T10:32:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.051 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142415 s, 73.6 MB/s 2024-04-17T10:32:45.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.066 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131226 s, 79.9 MB/s 2024-04-17T10:32:45.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.081 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131147 s, 80.0 MB/s 2024-04-17T10:32:45.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134528 s, 77.9 MB/s 2024-04-17T10:32:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.111 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133493 s, 78.5 MB/s 2024-04-17T10:32:45.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.127 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134601 s, 77.9 MB/s 2024-04-17T10:32:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.142 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013301 s, 78.8 MB/s 2024-04-17T10:32:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124022 s, 84.5 MB/s 2024-04-17T10:32:45.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137369 s, 76.3 MB/s 2024-04-17T10:32:45.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147788 s, 71.0 MB/s 2024-04-17T10:32:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132414 s, 79.2 MB/s 2024-04-17T10:32:45.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:45.205 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:45.205 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:45.205 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.7 2024-04-17T10:32:45.205 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:45.205 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:45.205 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:45.205 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:45.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165989 s, 63.2 MB/s 2024-04-17T10:32:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136107 s, 77.0 MB/s 2024-04-17T10:32:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.310 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152955 s, 68.6 MB/s 2024-04-17T10:32:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.326 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135837 s, 77.2 MB/s 2024-04-17T10:32:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130602 s, 80.3 MB/s 2024-04-17T10:32:45.357 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.357 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.357 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146848 s, 71.4 MB/s 2024-04-17T10:32:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.372 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013375 s, 78.4 MB/s 2024-04-17T10:32:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.389 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155186 s, 67.6 MB/s 2024-04-17T10:32:45.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.407 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146437 s, 71.6 MB/s 2024-04-17T10:32:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150674 s, 69.6 MB/s 2024-04-17T10:32:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.438 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132185 s, 79.3 MB/s 2024-04-17T10:32:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149341 s, 70.2 MB/s 2024-04-17T10:32:45.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.470 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137725 s, 76.1 MB/s 2024-04-17T10:32:45.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.486 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145006 s, 72.3 MB/s 2024-04-17T10:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.501 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012961 s, 80.9 MB/s 2024-04-17T10:32:45.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136956 s, 76.6 MB/s 2024-04-17T10:32:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.535 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140125 s, 74.8 MB/s 2024-04-17T10:32:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128175 s, 81.8 MB/s 2024-04-17T10:32:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012721 s, 82.4 MB/s 2024-04-17T10:32:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156922 s, 66.8 MB/s 2024-04-17T10:32:45.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.595 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013701 s, 76.5 MB/s 2024-04-17T10:32:45.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.615 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185266 s, 56.6 MB/s 2024-04-17T10:32:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129637 s, 80.9 MB/s 2024-04-17T10:32:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.646 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140467 s, 74.6 MB/s 2024-04-17T10:32:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148739 s, 70.5 MB/s 2024-04-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137728 s, 76.1 MB/s 2024-04-17T10:32:45.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.693 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132112 s, 79.4 MB/s 2024-04-17T10:32:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.709 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144631 s, 72.5 MB/s 2024-04-17T10:32:45.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149135 s, 70.3 MB/s 2024-04-17T10:32:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.742 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139753 s, 75.0 MB/s 2024-04-17T10:32:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141874 s, 73.9 MB/s 2024-04-17T10:32:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.778 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188734 s, 55.6 MB/s 2024-04-17T10:32:45.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:45.780 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:45.780 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:45.780 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.8 2024-04-17T10:32:45.780 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:45.780 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:45.780 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:45.780 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.851 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142226 s, 73.7 MB/s 2024-04-17T10:32:45.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.867 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132823 s, 78.9 MB/s 2024-04-17T10:32:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.886 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178098 s, 58.9 MB/s 2024-04-17T10:32:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.901 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133632 s, 78.5 MB/s 2024-04-17T10:32:45.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.918 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.918 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144774 s, 72.4 MB/s 2024-04-17T10:32:45.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.933 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.933 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135548 s, 77.4 MB/s 2024-04-17T10:32:45.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.948 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139207 s, 75.3 MB/s 2024-04-17T10:32:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144996 s, 72.3 MB/s 2024-04-17T10:32:45.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133679 s, 78.4 MB/s 2024-04-17T10:32:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143225 s, 73.2 MB/s 2024-04-17T10:32:46.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.012 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01413 s, 74.2 MB/s 2024-04-17T10:32:46.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133895 s, 78.3 MB/s 2024-04-17T10:32:46.042 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.042 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.042 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131629 s, 79.7 MB/s 2024-04-17T10:32:46.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.058 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133445 s, 78.6 MB/s 2024-04-17T10:32:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184894 s, 56.7 MB/s 2024-04-17T10:32:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149934 s, 69.9 MB/s 2024-04-17T10:32:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.109 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012773 s, 82.1 MB/s 2024-04-17T10:32:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013586 s, 77.2 MB/s 2024-04-17T10:32:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.145 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183497 s, 57.1 MB/s 2024-04-17T10:32:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129827 s, 80.8 MB/s 2024-04-17T10:32:46.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.175 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139508 s, 75.2 MB/s 2024-04-17T10:32:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.193 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147612 s, 71.0 MB/s 2024-04-17T10:32:46.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.209 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144268 s, 72.7 MB/s 2024-04-17T10:32:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.225 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144357 s, 72.6 MB/s 2024-04-17T10:32:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136004 s, 77.1 MB/s 2024-04-17T10:32:46.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.257 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147259 s, 71.2 MB/s 2024-04-17T10:32:46.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.273 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147994 s, 70.9 MB/s 2024-04-17T10:32:46.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.289 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138037 s, 76.0 MB/s 2024-04-17T10:32:46.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.304 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135932 s, 77.1 MB/s 2024-04-17T10:32:46.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.320 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137367 s, 76.3 MB/s 2024-04-17T10:32:46.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.340 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188096 s, 55.7 MB/s 2024-04-17T10:32:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139713 s, 75.1 MB/s 2024-04-17T10:32:46.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:46.357 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:46.358 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:46.358 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.9 2024-04-17T10:32:46.358 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:46.358 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:46.358 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:46.358 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.427 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145723 s, 72.0 MB/s 2024-04-17T10:32:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.443 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141824 s, 73.9 MB/s 2024-04-17T10:32:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135146 s, 77.6 MB/s 2024-04-17T10:32:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.474 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135115 s, 77.6 MB/s 2024-04-17T10:32:46.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.495 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187883 s, 55.8 MB/s 2024-04-17T10:32:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136829 s, 76.6 MB/s 2024-04-17T10:32:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132768 s, 79.0 MB/s 2024-04-17T10:32:46.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167277 s, 62.7 MB/s 2024-04-17T10:32:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.559 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135997 s, 77.1 MB/s 2024-04-17T10:32:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.576 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.576 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146382 s, 71.6 MB/s 2024-04-17T10:32:46.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.591 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135735 s, 77.3 MB/s 2024-04-17T10:32:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.607 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142157 s, 73.8 MB/s 2024-04-17T10:32:46.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.622 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133513 s, 78.5 MB/s 2024-04-17T10:32:46.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151514 s, 69.2 MB/s 2024-04-17T10:32:46.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.655 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140144 s, 74.8 MB/s 2024-04-17T10:32:46.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.672 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156724 s, 66.9 MB/s 2024-04-17T10:32:46.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157958 s, 66.4 MB/s 2024-04-17T10:32:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.704 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125364 s, 83.6 MB/s 2024-04-17T10:32:46.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.720 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137745 s, 76.1 MB/s 2024-04-17T10:32:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.735 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013456 s, 77.9 MB/s 2024-04-17T10:32:46.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.753 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156465 s, 67.0 MB/s 2024-04-17T10:32:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.769 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146597 s, 71.5 MB/s 2024-04-17T10:32:46.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.785 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014054 s, 74.6 MB/s 2024-04-17T10:32:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.801 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132706 s, 79.0 MB/s 2024-04-17T10:32:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.814 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125692 s, 83.4 MB/s 2024-04-17T10:32:46.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.831 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.831 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132666 s, 79.0 MB/s 2024-04-17T10:32:46.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.845 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139776 s, 75.0 MB/s 2024-04-17T10:32:46.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.862 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015099 s, 69.4 MB/s 2024-04-17T10:32:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.877 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013217 s, 79.3 MB/s 2024-04-17T10:32:46.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.893 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136844 s, 76.6 MB/s 2024-04-17T10:32:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.907 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126376 s, 83.0 MB/s 2024-04-17T10:32:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144584 s, 72.5 MB/s 2024-04-17T10:32:46.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:46.925 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:46.925 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:46.925 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.10 2024-04-17T10:32:46.925 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:46.925 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:46.925 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:46.925 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169668 s, 61.8 MB/s 2024-04-17T10:32:47.013 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.014 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137058 s, 76.5 MB/s 2024-04-17T10:32:47.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.029 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135403 s, 77.4 MB/s 2024-04-17T10:32:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.044 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132173 s, 79.3 MB/s 2024-04-17T10:32:47.059 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.060 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.060 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136098 s, 77.0 MB/s 2024-04-17T10:32:47.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.075 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137647 s, 76.2 MB/s 2024-04-17T10:32:47.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.091 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142649 s, 73.5 MB/s 2024-04-17T10:32:47.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.107 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140134 s, 74.8 MB/s 2024-04-17T10:32:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142647 s, 73.5 MB/s 2024-04-17T10:32:47.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.140 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.140 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148459 s, 70.6 MB/s 2024-04-17T10:32:47.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146255 s, 71.7 MB/s 2024-04-17T10:32:47.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150474 s, 69.7 MB/s 2024-04-17T10:32:47.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.190 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148952 s, 70.4 MB/s 2024-04-17T10:32:47.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138329 s, 75.8 MB/s 2024-04-17T10:32:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173631 s, 60.4 MB/s 2024-04-17T10:32:47.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129731 s, 80.8 MB/s 2024-04-17T10:32:47.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.254 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128051 s, 81.9 MB/s 2024-04-17T10:32:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152328 s, 68.8 MB/s 2024-04-17T10:32:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129753 s, 80.8 MB/s 2024-04-17T10:32:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.304 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166903 s, 62.8 MB/s 2024-04-17T10:32:47.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.320 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137473 s, 76.3 MB/s 2024-04-17T10:32:47.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.340 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186497 s, 56.2 MB/s 2024-04-17T10:32:47.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128768 s, 81.4 MB/s 2024-04-17T10:32:47.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.372 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150382 s, 69.7 MB/s 2024-04-17T10:32:47.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.388 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014602 s, 71.8 MB/s 2024-04-17T10:32:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145104 s, 72.3 MB/s 2024-04-17T10:32:47.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137413 s, 76.3 MB/s 2024-04-17T10:32:47.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.435 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132435 s, 79.2 MB/s 2024-04-17T10:32:47.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.451 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137928 s, 76.0 MB/s 2024-04-17T10:32:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.467 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145759 s, 71.9 MB/s 2024-04-17T10:32:47.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.482 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135693 s, 77.3 MB/s 2024-04-17T10:32:47.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01386 s, 75.7 MB/s 2024-04-17T10:32:47.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:47.499 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:47.499 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:47.499 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.11 2024-04-17T10:32:47.500 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:47.500 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:47.500 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:47.500 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.575 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179284 s, 58.5 MB/s 2024-04-17T10:32:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.593 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159149 s, 65.9 MB/s 2024-04-17T10:32:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179227 s, 58.5 MB/s 2024-04-17T10:32:47.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.627 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133277 s, 78.7 MB/s 2024-04-17T10:32:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.644 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142694 s, 73.5 MB/s 2024-04-17T10:32:47.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.659 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139602 s, 75.1 MB/s 2024-04-17T10:32:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138395 s, 75.8 MB/s 2024-04-17T10:32:47.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.691 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.691 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138684 s, 75.6 MB/s 2024-04-17T10:32:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.706 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140714 s, 74.5 MB/s 2024-04-17T10:32:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01371 s, 76.5 MB/s 2024-04-17T10:32:47.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.738 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142638 s, 73.5 MB/s 2024-04-17T10:32:47.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.753 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133442 s, 78.6 MB/s 2024-04-17T10:32:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138818 s, 75.5 MB/s 2024-04-17T10:32:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.786 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150723 s, 69.6 MB/s 2024-04-17T10:32:47.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.803 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148596 s, 70.6 MB/s 2024-04-17T10:32:47.817 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134578 s, 77.9 MB/s 2024-04-17T10:32:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.833 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133321 s, 78.7 MB/s 2024-04-17T10:32:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013619 s, 77.0 MB/s 2024-04-17T10:32:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.864 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139149 s, 75.4 MB/s 2024-04-17T10:32:47.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.884 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.884 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01762 s, 59.5 MB/s 2024-04-17T10:32:47.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125807 s, 83.3 MB/s 2024-04-17T10:32:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.913 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136645 s, 76.7 MB/s 2024-04-17T10:32:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153967 s, 68.1 MB/s 2024-04-17T10:32:47.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.946 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139741 s, 75.0 MB/s 2024-04-17T10:32:47.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133115 s, 78.8 MB/s 2024-04-17T10:32:47.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.982 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185717 s, 56.5 MB/s 2024-04-17T10:32:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:47.999 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152069 s, 69.0 MB/s 2024-04-17T10:32:48.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147594 s, 71.0 MB/s 2024-04-17T10:32:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.031 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138389 s, 75.8 MB/s 2024-04-17T10:32:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130548 s, 80.3 MB/s 2024-04-17T10:32:48.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.062 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148579 s, 70.6 MB/s 2024-04-17T10:32:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.083 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.083 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183595 s, 57.1 MB/s 2024-04-17T10:32:48.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:48.084 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:48.084 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:48.084 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.12 2024-04-17T10:32:48.084 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:48.084 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:48.084 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:48.084 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014951 s, 70.1 MB/s 2024-04-17T10:32:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149489 s, 70.1 MB/s 2024-04-17T10:32:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151693 s, 69.1 MB/s 2024-04-17T10:32:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144216 s, 72.7 MB/s 2024-04-17T10:32:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126391 s, 83.0 MB/s 2024-04-17T10:32:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133835 s, 78.3 MB/s 2024-04-17T10:32:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.252 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150326 s, 69.8 MB/s 2024-04-17T10:32:48.267 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.267 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.267 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012961 s, 80.9 MB/s 2024-04-17T10:32:48.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.282 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129864 s, 80.7 MB/s 2024-04-17T10:32:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134831 s, 77.8 MB/s 2024-04-17T10:32:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019327 s, 54.3 MB/s 2024-04-17T10:32:48.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.335 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146639 s, 71.5 MB/s 2024-04-17T10:32:48.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.351 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141928 s, 73.9 MB/s 2024-04-17T10:32:48.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.367 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147545 s, 71.1 MB/s 2024-04-17T10:32:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157982 s, 66.4 MB/s 2024-04-17T10:32:48.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.401 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141939 s, 73.9 MB/s 2024-04-17T10:32:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150136 s, 69.8 MB/s 2024-04-17T10:32:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144652 s, 72.5 MB/s 2024-04-17T10:32:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143801 s, 72.9 MB/s 2024-04-17T10:32:48.465 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014123 s, 74.2 MB/s 2024-04-17T10:32:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.482 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147521 s, 71.1 MB/s 2024-04-17T10:32:48.497 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.497 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135634 s, 77.3 MB/s 2024-04-17T10:32:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.513 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132971 s, 78.9 MB/s 2024-04-17T10:32:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.529 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144583 s, 72.5 MB/s 2024-04-17T10:32:48.545 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.545 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.545 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143031 s, 73.3 MB/s 2024-04-17T10:32:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161669 s, 64.9 MB/s 2024-04-17T10:32:48.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.578 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133005 s, 78.8 MB/s 2024-04-17T10:32:48.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.596 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159174 s, 65.9 MB/s 2024-04-17T10:32:48.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.611 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.611 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131843 s, 79.5 MB/s 2024-04-17T10:32:48.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.625 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127611 s, 82.2 MB/s 2024-04-17T10:32:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.641 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014183 s, 73.9 MB/s 2024-04-17T10:32:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.658 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014563 s, 72.0 MB/s 2024-04-17T10:32:48.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:48.659 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:48.659 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:48.659 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.13 2024-04-17T10:32:48.659 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:48.659 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:48.659 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:48.659 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:48.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.730 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131583 s, 79.7 MB/s 2024-04-17T10:32:48.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.746 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143496 s, 73.1 MB/s 2024-04-17T10:32:48.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019032 s, 55.1 MB/s 2024-04-17T10:32:48.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.784 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152101 s, 68.9 MB/s 2024-04-17T10:32:48.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145717 s, 72.0 MB/s 2024-04-17T10:32:48.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134732 s, 77.8 MB/s 2024-04-17T10:32:48.834 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171182 s, 61.3 MB/s 2024-04-17T10:32:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.850 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132972 s, 78.9 MB/s 2024-04-17T10:32:48.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.865 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127954 s, 81.9 MB/s 2024-04-17T10:32:48.884 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.884 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.884 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01347 s, 77.8 MB/s 2024-04-17T10:32:48.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.895 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134986 s, 77.7 MB/s 2024-04-17T10:32:48.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139086 s, 75.4 MB/s 2024-04-17T10:32:48.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.926 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132214 s, 79.3 MB/s 2024-04-17T10:32:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.946 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185712 s, 56.5 MB/s 2024-04-17T10:32:48.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.963 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150837 s, 69.5 MB/s 2024-04-17T10:32:48.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154209 s, 68.0 MB/s 2024-04-17T10:32:48.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:48.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:48.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131898 s, 79.5 MB/s 2024-04-17T10:32:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.016 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185026 s, 56.7 MB/s 2024-04-17T10:32:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131749 s, 79.6 MB/s 2024-04-17T10:32:49.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.047 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149943 s, 69.9 MB/s 2024-04-17T10:32:49.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.064 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153659 s, 68.2 MB/s 2024-04-17T10:32:49.084 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.084 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.084 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177672 s, 59.0 MB/s 2024-04-17T10:32:49.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.100 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147212 s, 71.2 MB/s 2024-04-17T10:32:49.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.116 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133806 s, 78.4 MB/s 2024-04-17T10:32:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.135 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174287 s, 60.2 MB/s 2024-04-17T10:32:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.150 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.151 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138491 s, 75.7 MB/s 2024-04-17T10:32:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167042 s, 62.8 MB/s 2024-04-17T10:32:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.193 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213495 s, 49.1 MB/s 2024-04-17T10:32:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132579 s, 79.1 MB/s 2024-04-17T10:32:49.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161088 s, 65.1 MB/s 2024-04-17T10:32:49.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138361 s, 75.8 MB/s 2024-04-17T10:32:49.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.258 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135488 s, 77.4 MB/s 2024-04-17T10:32:49.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:49.259 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:49.259 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:49.259 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.14 2024-04-17T10:32:49.259 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:49.259 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:49.259 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:49.260 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:49.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141225 s, 74.2 MB/s 2024-04-17T10:32:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128392 s, 81.7 MB/s 2024-04-17T10:32:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.360 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127648 s, 82.1 MB/s 2024-04-17T10:32:49.377 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.377 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.377 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015142 s, 69.2 MB/s 2024-04-17T10:32:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.394 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149468 s, 70.2 MB/s 2024-04-17T10:32:49.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139625 s, 75.1 MB/s 2024-04-17T10:32:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.425 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138419 s, 75.8 MB/s 2024-04-17T10:32:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.440 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134331 s, 78.1 MB/s 2024-04-17T10:32:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185663 s, 56.5 MB/s 2024-04-17T10:32:49.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.476 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013784 s, 76.1 MB/s 2024-04-17T10:32:49.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.492 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137393 s, 76.3 MB/s 2024-04-17T10:32:49.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.512 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187478 s, 55.9 MB/s 2024-04-17T10:32:49.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.529 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149508 s, 70.1 MB/s 2024-04-17T10:32:49.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012835 s, 81.7 MB/s 2024-04-17T10:32:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.558 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122295 s, 85.7 MB/s 2024-04-17T10:32:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.571 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113856 s, 92.1 MB/s 2024-04-17T10:32:49.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.588 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150591 s, 69.6 MB/s 2024-04-17T10:32:49.604 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.604 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.604 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144465 s, 72.6 MB/s 2024-04-17T10:32:49.620 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.620 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.620 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140906 s, 74.4 MB/s 2024-04-17T10:32:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136748 s, 76.7 MB/s 2024-04-17T10:32:49.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.650 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127804 s, 82.0 MB/s 2024-04-17T10:32:49.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.666 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134717 s, 77.8 MB/s 2024-04-17T10:32:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017481 s, 60.0 MB/s 2024-04-17T10:32:49.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.703 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01686 s, 62.2 MB/s 2024-04-17T10:32:49.719 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.719 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.719 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142175 s, 73.8 MB/s 2024-04-17T10:32:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.736 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149074 s, 70.3 MB/s 2024-04-17T10:32:49.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.752 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145316 s, 72.2 MB/s 2024-04-17T10:32:49.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013313 s, 78.8 MB/s 2024-04-17T10:32:49.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137621 s, 76.2 MB/s 2024-04-17T10:32:49.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.797 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.798 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128443 s, 81.6 MB/s 2024-04-17T10:32:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.813 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134397 s, 78.0 MB/s 2024-04-17T10:32:49.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.829 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144423 s, 72.6 MB/s 2024-04-17T10:32:49.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:49.830 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:49.831 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:49.831 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.15 2024-04-17T10:32:49.831 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:49.831 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:49.831 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:49.831 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154013 s, 68.1 MB/s 2024-04-17T10:32:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191477 s, 54.8 MB/s 2024-04-17T10:32:49.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.941 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166625 s, 62.9 MB/s 2024-04-17T10:32:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013588 s, 77.2 MB/s 2024-04-17T10:32:49.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.972 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134457 s, 78.0 MB/s 2024-04-17T10:32:49.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:49.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:49.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132662 s, 79.0 MB/s 2024-04-17T10:32:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162607 s, 64.5 MB/s 2024-04-17T10:32:50.022 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.022 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.022 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154449 s, 67.9 MB/s 2024-04-17T10:32:50.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.038 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140883 s, 74.4 MB/s 2024-04-17T10:32:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.054 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143141 s, 73.3 MB/s 2024-04-17T10:32:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.070 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136551 s, 76.8 MB/s 2024-04-17T10:32:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.087 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150595 s, 69.6 MB/s 2024-04-17T10:32:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.107 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189045 s, 55.5 MB/s 2024-04-17T10:32:50.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.124 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015318 s, 68.5 MB/s 2024-04-17T10:32:50.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.141 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150451 s, 69.7 MB/s 2024-04-17T10:32:50.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139493 s, 75.2 MB/s 2024-04-17T10:32:50.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.174 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154448 s, 67.9 MB/s 2024-04-17T10:32:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137248 s, 76.4 MB/s 2024-04-17T10:32:50.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143284 s, 73.2 MB/s 2024-04-17T10:32:50.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.221 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139287 s, 75.3 MB/s 2024-04-17T10:32:50.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.238 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146295 s, 71.7 MB/s 2024-04-17T10:32:50.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.252 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126444 s, 82.9 MB/s 2024-04-17T10:32:50.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.267 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123539 s, 84.9 MB/s 2024-04-17T10:32:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130552 s, 80.3 MB/s 2024-04-17T10:32:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137164 s, 76.4 MB/s 2024-04-17T10:32:50.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133486 s, 78.6 MB/s 2024-04-17T10:32:50.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.327 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131838 s, 79.5 MB/s 2024-04-17T10:32:50.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184191 s, 56.9 MB/s 2024-04-17T10:32:50.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.362 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131714 s, 79.6 MB/s 2024-04-17T10:32:50.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138375 s, 75.8 MB/s 2024-04-17T10:32:50.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.393 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136915 s, 76.6 MB/s 2024-04-17T10:32:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130205 s, 80.5 MB/s 2024-04-17T10:32:50.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:50.410 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:50.410 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:50.410 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.16 2024-04-17T10:32:50.410 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:50.410 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:50.410 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:50.410 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:50.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.480 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153515 s, 68.3 MB/s 2024-04-17T10:32:50.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135638 s, 77.3 MB/s 2024-04-17T10:32:50.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.511 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133738 s, 78.4 MB/s 2024-04-17T10:32:50.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.528 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152259 s, 68.9 MB/s 2024-04-17T10:32:50.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141745 s, 74.0 MB/s 2024-04-17T10:32:50.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178826 s, 58.6 MB/s 2024-04-17T10:32:50.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.577 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012111 s, 86.6 MB/s 2024-04-17T10:32:50.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.597 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180395 s, 58.1 MB/s 2024-04-17T10:32:50.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138095 s, 75.9 MB/s 2024-04-17T10:32:50.628 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.628 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.628 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137892 s, 76.0 MB/s 2024-04-17T10:32:50.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.644 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141302 s, 74.2 MB/s 2024-04-17T10:32:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.659 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126112 s, 83.1 MB/s 2024-04-17T10:32:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138425 s, 75.8 MB/s 2024-04-17T10:32:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145005 s, 72.3 MB/s 2024-04-17T10:32:50.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179538 s, 58.4 MB/s 2024-04-17T10:32:50.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.727 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152478 s, 68.8 MB/s 2024-04-17T10:32:50.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.743 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134262 s, 78.1 MB/s 2024-04-17T10:32:50.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.758 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135798 s, 77.2 MB/s 2024-04-17T10:32:50.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.772 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116589 s, 89.9 MB/s 2024-04-17T10:32:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125618 s, 83.5 MB/s 2024-04-17T10:32:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163505 s, 64.1 MB/s 2024-04-17T10:32:50.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.819 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.819 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132401 s, 79.2 MB/s 2024-04-17T10:32:50.834 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.834 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.834 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138046 s, 76.0 MB/s 2024-04-17T10:32:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.850 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135236 s, 77.5 MB/s 2024-04-17T10:32:50.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.865 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140555 s, 74.6 MB/s 2024-04-17T10:32:50.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.881 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139007 s, 75.4 MB/s 2024-04-17T10:32:50.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.897 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150182 s, 69.8 MB/s 2024-04-17T10:32:50.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.914 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146406 s, 71.6 MB/s 2024-04-17T10:32:50.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.931 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144241 s, 72.7 MB/s 2024-04-17T10:32:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.950 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176391 s, 59.4 MB/s 2024-04-17T10:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.964 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01299 s, 80.7 MB/s 2024-04-17T10:32:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:50.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138662 s, 75.6 MB/s 2024-04-17T10:32:50.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:50.982 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:50.982 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:50.982 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.17 2024-04-17T10:32:50.983 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:50.983 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:50.983 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:50.983 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:51.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.054 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162324 s, 64.6 MB/s 2024-04-17T10:32:51.074 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.075 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181735 s, 57.7 MB/s 2024-04-17T10:32:51.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.091 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147446 s, 71.1 MB/s 2024-04-17T10:32:51.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.106 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132241 s, 79.3 MB/s 2024-04-17T10:32:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.121 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133604 s, 78.5 MB/s 2024-04-17T10:32:51.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141472 s, 74.1 MB/s 2024-04-17T10:32:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153753 s, 68.2 MB/s 2024-04-17T10:32:51.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.174 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181186 s, 57.9 MB/s 2024-04-17T10:32:51.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147727 s, 71.0 MB/s 2024-04-17T10:32:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.209 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016689 s, 62.8 MB/s 2024-04-17T10:32:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179322 s, 58.5 MB/s 2024-04-17T10:32:51.244 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.244 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.244 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140186 s, 74.8 MB/s 2024-04-17T10:32:51.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.260 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137303 s, 76.4 MB/s 2024-04-17T10:32:51.275 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.275 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.275 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136344 s, 76.9 MB/s 2024-04-17T10:32:51.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.292 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.292 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141893 s, 73.9 MB/s 2024-04-17T10:32:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.307 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.308 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013688 s, 76.6 MB/s 2024-04-17T10:32:51.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.323 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140715 s, 74.5 MB/s 2024-04-17T10:32:51.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.339 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138031 s, 76.0 MB/s 2024-04-17T10:32:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.354 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135225 s, 77.5 MB/s 2024-04-17T10:32:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147537 s, 71.1 MB/s 2024-04-17T10:32:51.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144866 s, 72.4 MB/s 2024-04-17T10:32:51.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.402 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.402 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012981 s, 80.8 MB/s 2024-04-17T10:32:51.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154482 s, 67.9 MB/s 2024-04-17T10:32:51.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.435 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141317 s, 74.2 MB/s 2024-04-17T10:32:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.450 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139553 s, 75.1 MB/s 2024-04-17T10:32:51.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136673 s, 76.7 MB/s 2024-04-17T10:32:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.481 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129861 s, 80.7 MB/s 2024-04-17T10:32:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136702 s, 76.7 MB/s 2024-04-17T10:32:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.516 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182684 s, 57.4 MB/s 2024-04-17T10:32:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.531 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131191 s, 79.9 MB/s 2024-04-17T10:32:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.551 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017878 s, 58.7 MB/s 2024-04-17T10:32:51.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.567 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141799 s, 73.9 MB/s 2024-04-17T10:32:51.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:51.568 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:51.568 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:51.568 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.18 2024-04-17T10:32:51.568 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:51.568 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:51.569 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:51.569 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:51.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.643 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145707 s, 72.0 MB/s 2024-04-17T10:32:51.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.659 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140638 s, 74.6 MB/s 2024-04-17T10:32:51.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.676 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147643 s, 71.0 MB/s 2024-04-17T10:32:51.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.692 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144011 s, 72.8 MB/s 2024-04-17T10:32:51.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.709 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155611 s, 67.4 MB/s 2024-04-17T10:32:51.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.724 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136666 s, 76.7 MB/s 2024-04-17T10:32:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013096 s, 80.1 MB/s 2024-04-17T10:32:51.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.758 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170383 s, 61.5 MB/s 2024-04-17T10:32:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.774 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139664 s, 75.1 MB/s 2024-04-17T10:32:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.790 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143051 s, 73.3 MB/s 2024-04-17T10:32:51.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.806 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138625 s, 75.6 MB/s 2024-04-17T10:32:51.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.823 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142805 s, 73.4 MB/s 2024-04-17T10:32:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.838 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145239 s, 72.2 MB/s 2024-04-17T10:32:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134642 s, 77.9 MB/s 2024-04-17T10:32:51.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.870 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143856 s, 72.9 MB/s 2024-04-17T10:32:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147265 s, 71.2 MB/s 2024-04-17T10:32:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.900 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011778 s, 89.0 MB/s 2024-04-17T10:32:51.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.916 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139144 s, 75.4 MB/s 2024-04-17T10:32:51.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.933 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154028 s, 68.1 MB/s 2024-04-17T10:32:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.948 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132435 s, 79.2 MB/s 2024-04-17T10:32:51.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151491 s, 69.2 MB/s 2024-04-17T10:32:51.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140202 s, 74.8 MB/s 2024-04-17T10:32:51.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:51.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:51.996 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135976 s, 77.1 MB/s 2024-04-17T10:32:52.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130473 s, 80.4 MB/s 2024-04-17T10:32:52.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.024 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113874 s, 92.1 MB/s 2024-04-17T10:32:52.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.039 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132585 s, 79.1 MB/s 2024-04-17T10:32:52.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.055 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142434 s, 73.6 MB/s 2024-04-17T10:32:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.070 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132849 s, 78.9 MB/s 2024-04-17T10:32:52.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186539 s, 56.2 MB/s 2024-04-17T10:32:52.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.106 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135427 s, 77.4 MB/s 2024-04-17T10:32:52.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.122 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144547 s, 72.5 MB/s 2024-04-17T10:32:52.138 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.139 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152204 s, 68.9 MB/s 2024-04-17T10:32:52.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:52.140 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:52.140 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:52.140 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.19 2024-04-17T10:32:52.140 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:52.141 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:52.141 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:52.141 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:52.215 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.216 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187222 s, 56.0 MB/s 2024-04-17T10:32:52.234 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.234 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.234 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169125 s, 62.0 MB/s 2024-04-17T10:32:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189258 s, 55.4 MB/s 2024-04-17T10:32:52.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139205 s, 75.3 MB/s 2024-04-17T10:32:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184588 s, 56.8 MB/s 2024-04-17T10:32:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140344 s, 74.7 MB/s 2024-04-17T10:32:52.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.324 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147016 s, 71.3 MB/s 2024-04-17T10:32:52.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.339 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139385 s, 75.2 MB/s 2024-04-17T10:32:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.361 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185398 s, 56.6 MB/s 2024-04-17T10:32:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.375 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137593 s, 76.2 MB/s 2024-04-17T10:32:52.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150203 s, 69.8 MB/s 2024-04-17T10:32:52.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.409 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146403 s, 71.6 MB/s 2024-04-17T10:32:52.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.425 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015264 s, 68.7 MB/s 2024-04-17T10:32:52.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136126 s, 77.0 MB/s 2024-04-17T10:32:52.455 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129805 s, 80.8 MB/s 2024-04-17T10:32:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.477 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189282 s, 55.4 MB/s 2024-04-17T10:32:52.491 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.492 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134584 s, 77.9 MB/s 2024-04-17T10:32:52.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.507 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132292 s, 79.3 MB/s 2024-04-17T10:32:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122824 s, 85.4 MB/s 2024-04-17T10:32:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143389 s, 73.1 MB/s 2024-04-17T10:32:52.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.552 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129681 s, 80.9 MB/s 2024-04-17T10:32:52.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140727 s, 74.5 MB/s 2024-04-17T10:32:52.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139767 s, 75.0 MB/s 2024-04-17T10:32:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152218 s, 68.9 MB/s 2024-04-17T10:32:52.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.617 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148494 s, 70.6 MB/s 2024-04-17T10:32:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148624 s, 70.6 MB/s 2024-04-17T10:32:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135965 s, 77.1 MB/s 2024-04-17T10:32:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140078 s, 74.9 MB/s 2024-04-17T10:32:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.680 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137027 s, 76.5 MB/s 2024-04-17T10:32:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.696 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136985 s, 76.5 MB/s 2024-04-17T10:32:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173155 s, 60.6 MB/s 2024-04-17T10:32:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.732 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141175 s, 74.3 MB/s 2024-04-17T10:32:52.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:52.733 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:52.733 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:52.733 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.20 2024-04-17T10:32:52.733 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:52.733 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:52.733 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:52.733 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:52.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.806 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147531 s, 71.1 MB/s 2024-04-17T10:32:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.824 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167486 s, 62.6 MB/s 2024-04-17T10:32:52.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.841 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152184 s, 68.9 MB/s 2024-04-17T10:32:52.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.858 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153139 s, 68.5 MB/s 2024-04-17T10:32:52.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.877 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168002 s, 62.4 MB/s 2024-04-17T10:32:52.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131397 s, 79.8 MB/s 2024-04-17T10:32:52.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.907 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130545 s, 80.3 MB/s 2024-04-17T10:32:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135503 s, 77.4 MB/s 2024-04-17T10:32:52.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138748 s, 75.6 MB/s 2024-04-17T10:32:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.954 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141247 s, 74.2 MB/s 2024-04-17T10:32:52.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150559 s, 69.6 MB/s 2024-04-17T10:32:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:52.990 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178752 s, 58.7 MB/s 2024-04-17T10:32:53.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190625 s, 55.0 MB/s 2024-04-17T10:32:53.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.028 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149355 s, 70.2 MB/s 2024-04-17T10:32:53.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.049 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197321 s, 53.1 MB/s 2024-04-17T10:32:53.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.065 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140685 s, 74.5 MB/s 2024-04-17T10:32:53.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146266 s, 71.7 MB/s 2024-04-17T10:32:53.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128786 s, 81.4 MB/s 2024-04-17T10:32:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.112 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142829 s, 73.4 MB/s 2024-04-17T10:32:53.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015467 s, 67.8 MB/s 2024-04-17T10:32:53.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.146 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149998 s, 69.9 MB/s 2024-04-17T10:32:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.162 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142934 s, 73.4 MB/s 2024-04-17T10:32:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.178 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013571 s, 77.3 MB/s 2024-04-17T10:32:53.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.195 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151998 s, 69.0 MB/s 2024-04-17T10:32:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.212 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152986 s, 68.5 MB/s 2024-04-17T10:32:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.228 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139456 s, 75.2 MB/s 2024-04-17T10:32:53.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.243 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139564 s, 75.1 MB/s 2024-04-17T10:32:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.260 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145694 s, 72.0 MB/s 2024-04-17T10:32:53.275 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.275 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.275 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132557 s, 79.1 MB/s 2024-04-17T10:32:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.290 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131337 s, 79.8 MB/s 2024-04-17T10:32:53.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139809 s, 75.0 MB/s 2024-04-17T10:32:53.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.322 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142101 s, 73.8 MB/s 2024-04-17T10:32:53.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:53.324 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:53.324 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:53.324 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.21 2024-04-17T10:32:53.324 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:53.324 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:53.324 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:53.324 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:53.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.396 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139891 s, 75.0 MB/s 2024-04-17T10:32:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.411 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129113 s, 81.2 MB/s 2024-04-17T10:32:53.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.427 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149819 s, 70.0 MB/s 2024-04-17T10:32:53.442 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139536 s, 75.1 MB/s 2024-04-17T10:32:53.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013699 s, 76.5 MB/s 2024-04-17T10:32:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.473 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128432 s, 81.6 MB/s 2024-04-17T10:32:53.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.490 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015546 s, 67.4 MB/s 2024-04-17T10:32:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.506 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139411 s, 75.2 MB/s 2024-04-17T10:32:53.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125827 s, 83.3 MB/s 2024-04-17T10:32:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013708 s, 76.5 MB/s 2024-04-17T10:32:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.552 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141892 s, 73.9 MB/s 2024-04-17T10:32:53.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151541 s, 69.2 MB/s 2024-04-17T10:32:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.586 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146131 s, 71.8 MB/s 2024-04-17T10:32:53.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.602 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146033 s, 71.8 MB/s 2024-04-17T10:32:53.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.622 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186016 s, 56.4 MB/s 2024-04-17T10:32:53.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149096 s, 70.3 MB/s 2024-04-17T10:32:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.654 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134531 s, 77.9 MB/s 2024-04-17T10:32:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168169 s, 62.4 MB/s 2024-04-17T10:32:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152332 s, 68.8 MB/s 2024-04-17T10:32:53.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147531 s, 71.1 MB/s 2024-04-17T10:32:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185457 s, 56.5 MB/s 2024-04-17T10:32:53.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.744 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150943 s, 69.5 MB/s 2024-04-17T10:32:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133934 s, 78.3 MB/s 2024-04-17T10:32:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131812 s, 79.6 MB/s 2024-04-17T10:32:53.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.790 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138607 s, 75.7 MB/s 2024-04-17T10:32:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.808 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163743 s, 64.0 MB/s 2024-04-17T10:32:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.830 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170932 s, 61.3 MB/s 2024-04-17T10:32:53.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.844 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.844 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148897 s, 70.4 MB/s 2024-04-17T10:32:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148717 s, 70.5 MB/s 2024-04-17T10:32:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.878 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156255 s, 67.1 MB/s 2024-04-17T10:32:53.893 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.893 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.894 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137237 s, 76.4 MB/s 2024-04-17T10:32:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015728 s, 66.7 MB/s 2024-04-17T10:32:53.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:53.912 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:53.912 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:53.913 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.22 2024-04-17T10:32:53.913 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:53.913 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:53.913 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:53.913 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:53.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.983 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.983 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131436 s, 79.8 MB/s 2024-04-17T10:32:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135954 s, 77.1 MB/s 2024-04-17T10:32:54.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153377 s, 68.4 MB/s 2024-04-17T10:32:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154841 s, 67.7 MB/s 2024-04-17T10:32:54.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.048 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135507 s, 77.4 MB/s 2024-04-17T10:32:54.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.065 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149798 s, 70.0 MB/s 2024-04-17T10:32:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.081 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148288 s, 70.7 MB/s 2024-04-17T10:32:54.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.098 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149873 s, 70.0 MB/s 2024-04-17T10:32:54.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.115 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152787 s, 68.6 MB/s 2024-04-17T10:32:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.134 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175871 s, 59.6 MB/s 2024-04-17T10:32:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152359 s, 68.8 MB/s 2024-04-17T10:32:54.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179156 s, 58.5 MB/s 2024-04-17T10:32:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.189 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153006 s, 68.5 MB/s 2024-04-17T10:32:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.204 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139604 s, 75.1 MB/s 2024-04-17T10:32:54.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153792 s, 68.2 MB/s 2024-04-17T10:32:54.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.238 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140167 s, 74.8 MB/s 2024-04-17T10:32:54.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131055 s, 80.0 MB/s 2024-04-17T10:32:54.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.273 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178164 s, 58.9 MB/s 2024-04-17T10:32:54.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.290 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157002 s, 66.8 MB/s 2024-04-17T10:32:54.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137948 s, 76.0 MB/s 2024-04-17T10:32:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140014 s, 74.9 MB/s 2024-04-17T10:32:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01336 s, 78.5 MB/s 2024-04-17T10:32:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.354 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153254 s, 68.4 MB/s 2024-04-17T10:32:54.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146291 s, 71.7 MB/s 2024-04-17T10:32:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158021 s, 66.4 MB/s 2024-04-17T10:32:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144958 s, 72.3 MB/s 2024-04-17T10:32:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.421 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148053 s, 70.8 MB/s 2024-04-17T10:32:54.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.439 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155551 s, 67.4 MB/s 2024-04-17T10:32:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154639 s, 67.8 MB/s 2024-04-17T10:32:54.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.471 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136471 s, 76.8 MB/s 2024-04-17T10:32:54.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.487 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142006 s, 73.8 MB/s 2024-04-17T10:32:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.504 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152135 s, 68.9 MB/s 2024-04-17T10:32:54.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:54.506 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:54.506 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:54.506 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.23 2024-04-17T10:32:54.506 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:54.506 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:54.506 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:54.506 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:54.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.579 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137542 s, 76.2 MB/s 2024-04-17T10:32:54.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.595 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140117 s, 74.8 MB/s 2024-04-17T10:32:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134248 s, 78.1 MB/s 2024-04-17T10:32:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.626 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143341 s, 73.2 MB/s 2024-04-17T10:32:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184056 s, 57.0 MB/s 2024-04-17T10:32:54.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134608 s, 77.9 MB/s 2024-04-17T10:32:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144456 s, 72.6 MB/s 2024-04-17T10:32:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.697 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018032 s, 58.2 MB/s 2024-04-17T10:32:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156633 s, 66.9 MB/s 2024-04-17T10:32:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.730 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128672 s, 81.5 MB/s 2024-04-17T10:32:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.745 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139062 s, 75.4 MB/s 2024-04-17T10:32:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.762 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149089 s, 70.3 MB/s 2024-04-17T10:32:54.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153448 s, 68.3 MB/s 2024-04-17T10:32:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139569 s, 75.1 MB/s 2024-04-17T10:32:54.810 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141825 s, 73.9 MB/s 2024-04-17T10:32:54.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130658 s, 80.3 MB/s 2024-04-17T10:32:54.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.847 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191079 s, 54.9 MB/s 2024-04-17T10:32:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132342 s, 79.2 MB/s 2024-04-17T10:32:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136629 s, 76.7 MB/s 2024-04-17T10:32:54.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.893 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142388 s, 73.6 MB/s 2024-04-17T10:32:54.912 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.912 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.912 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180187 s, 58.2 MB/s 2024-04-17T10:32:54.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.926 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117611 s, 89.2 MB/s 2024-04-17T10:32:54.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.943 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151682 s, 69.1 MB/s 2024-04-17T10:32:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.960 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153257 s, 68.4 MB/s 2024-04-17T10:32:54.975 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.975 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.975 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013383 s, 78.4 MB/s 2024-04-17T10:32:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:54.996 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189084 s, 55.5 MB/s 2024-04-17T10:32:55.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132058 s, 79.4 MB/s 2024-04-17T10:32:55.026 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.026 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.026 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134993 s, 77.7 MB/s 2024-04-17T10:32:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177798 s, 59.0 MB/s 2024-04-17T10:32:55.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.063 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147891 s, 70.9 MB/s 2024-04-17T10:32:55.077 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132662 s, 79.0 MB/s 2024-04-17T10:32:55.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.093 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013121 s, 79.9 MB/s 2024-04-17T10:32:55.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:55.094 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:55.094 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:55.094 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.24 2024-04-17T10:32:55.094 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:55.094 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:55.094 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:55.094 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177828 s, 59.0 MB/s 2024-04-17T10:32:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.183 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135117 s, 77.6 MB/s 2024-04-17T10:32:55.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.200 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.200 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014891 s, 70.4 MB/s 2024-04-17T10:32:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.215 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135576 s, 77.3 MB/s 2024-04-17T10:32:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154083 s, 68.1 MB/s 2024-04-17T10:32:55.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.248 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01437 s, 73.0 MB/s 2024-04-17T10:32:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.264 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141293 s, 74.2 MB/s 2024-04-17T10:32:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.287 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206536 s, 50.8 MB/s 2024-04-17T10:32:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153949 s, 68.1 MB/s 2024-04-17T10:32:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133055 s, 78.8 MB/s 2024-04-17T10:32:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144069 s, 72.8 MB/s 2024-04-17T10:32:55.351 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.351 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.351 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142559 s, 73.6 MB/s 2024-04-17T10:32:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.366 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.367 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136658 s, 76.7 MB/s 2024-04-17T10:32:55.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.384 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157678 s, 66.5 MB/s 2024-04-17T10:32:55.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.398 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122687 s, 85.5 MB/s 2024-04-17T10:32:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.418 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181776 s, 57.7 MB/s 2024-04-17T10:32:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.434 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138129 s, 75.9 MB/s 2024-04-17T10:32:55.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.449 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.449 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132256 s, 79.3 MB/s 2024-04-17T10:32:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.467 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015535 s, 67.5 MB/s 2024-04-17T10:32:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.483 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149226 s, 70.3 MB/s 2024-04-17T10:32:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131739 s, 79.6 MB/s 2024-04-17T10:32:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155395 s, 67.5 MB/s 2024-04-17T10:32:55.530 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.531 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013452 s, 77.9 MB/s 2024-04-17T10:32:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.546 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137453 s, 76.3 MB/s 2024-04-17T10:32:55.561 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.562 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135929 s, 77.1 MB/s 2024-04-17T10:32:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197198 s, 53.2 MB/s 2024-04-17T10:32:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133922 s, 78.3 MB/s 2024-04-17T10:32:55.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.618 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017739 s, 59.1 MB/s 2024-04-17T10:32:55.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.634 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140744 s, 74.5 MB/s 2024-04-17T10:32:55.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.650 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150824 s, 69.5 MB/s 2024-04-17T10:32:55.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180157 s, 58.2 MB/s 2024-04-17T10:32:55.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.688 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159324 s, 65.8 MB/s 2024-04-17T10:32:55.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:55.689 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:55.690 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:55.690 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.25 2024-04-17T10:32:55.690 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:55.690 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:55.690 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:55.690 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:55.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.764 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152253 s, 68.9 MB/s 2024-04-17T10:32:55.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133572 s, 78.5 MB/s 2024-04-17T10:32:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.795 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141233 s, 74.2 MB/s 2024-04-17T10:32:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.811 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136875 s, 76.6 MB/s 2024-04-17T10:32:55.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133758 s, 78.4 MB/s 2024-04-17T10:32:55.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.841 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137173 s, 76.4 MB/s 2024-04-17T10:32:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150548 s, 69.7 MB/s 2024-04-17T10:32:55.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.875 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156003 s, 67.2 MB/s 2024-04-17T10:32:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.896 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185043 s, 56.7 MB/s 2024-04-17T10:32:55.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.915 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170416 s, 61.5 MB/s 2024-04-17T10:32:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014391 s, 72.9 MB/s 2024-04-17T10:32:55.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139149 s, 75.4 MB/s 2024-04-17T10:32:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.962 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134046 s, 78.2 MB/s 2024-04-17T10:32:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145851 s, 71.9 MB/s 2024-04-17T10:32:55.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:55.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147831 s, 70.9 MB/s 2024-04-17T10:32:56.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146436 s, 71.6 MB/s 2024-04-17T10:32:56.026 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.026 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.026 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013319 s, 78.7 MB/s 2024-04-17T10:32:56.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.043 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148564 s, 70.6 MB/s 2024-04-17T10:32:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.059 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148772 s, 70.5 MB/s 2024-04-17T10:32:56.074 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.075 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134259 s, 78.1 MB/s 2024-04-17T10:32:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.091 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145402 s, 72.1 MB/s 2024-04-17T10:32:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.106 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135572 s, 77.3 MB/s 2024-04-17T10:32:56.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143175 s, 73.2 MB/s 2024-04-17T10:32:56.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131212 s, 79.9 MB/s 2024-04-17T10:32:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.153 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139537 s, 75.1 MB/s 2024-04-17T10:32:56.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.170 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148514 s, 70.6 MB/s 2024-04-17T10:32:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142877 s, 73.4 MB/s 2024-04-17T10:32:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152451 s, 68.8 MB/s 2024-04-17T10:32:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133669 s, 78.4 MB/s 2024-04-17T10:32:56.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132706 s, 79.0 MB/s 2024-04-17T10:32:56.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.249 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139749 s, 75.0 MB/s 2024-04-17T10:32:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.266 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153728 s, 68.2 MB/s 2024-04-17T10:32:56.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:56.267 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:56.268 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:56.268 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.26 2024-04-17T10:32:56.268 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:56.268 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:56.268 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:56.268 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.344 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143259 s, 73.2 MB/s 2024-04-17T10:32:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.360 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144797 s, 72.4 MB/s 2024-04-17T10:32:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.375 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138645 s, 75.6 MB/s 2024-04-17T10:32:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170157 s, 61.6 MB/s 2024-04-17T10:32:56.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144909 s, 72.4 MB/s 2024-04-17T10:32:56.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.430 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186105 s, 56.3 MB/s 2024-04-17T10:32:56.447 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.447 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.447 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147228 s, 71.2 MB/s 2024-04-17T10:32:56.464 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.465 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.465 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156142 s, 67.2 MB/s 2024-04-17T10:32:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.480 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135927 s, 77.1 MB/s 2024-04-17T10:32:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149006 s, 70.4 MB/s 2024-04-17T10:32:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123313 s, 85.0 MB/s 2024-04-17T10:32:56.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134097 s, 78.2 MB/s 2024-04-17T10:32:56.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147863 s, 70.9 MB/s 2024-04-17T10:32:56.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.559 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013867 s, 75.6 MB/s 2024-04-17T10:32:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132064 s, 79.4 MB/s 2024-04-17T10:32:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.589 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137594 s, 76.2 MB/s 2024-04-17T10:32:56.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.607 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159422 s, 65.8 MB/s 2024-04-17T10:32:56.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.624 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151865 s, 69.0 MB/s 2024-04-17T10:32:56.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.641 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150594 s, 69.6 MB/s 2024-04-17T10:32:56.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.656 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133047 s, 78.8 MB/s 2024-04-17T10:32:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.672 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145518 s, 72.1 MB/s 2024-04-17T10:32:56.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.689 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150193 s, 69.8 MB/s 2024-04-17T10:32:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017423 s, 60.2 MB/s 2024-04-17T10:32:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155499 s, 67.4 MB/s 2024-04-17T10:32:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132765 s, 79.0 MB/s 2024-04-17T10:32:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.756 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143285 s, 73.2 MB/s 2024-04-17T10:32:56.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.773 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142063 s, 73.8 MB/s 2024-04-17T10:32:56.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.789 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146487 s, 71.6 MB/s 2024-04-17T10:32:56.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.806 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152587 s, 68.7 MB/s 2024-04-17T10:32:56.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.823 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157084 s, 66.8 MB/s 2024-04-17T10:32:56.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.840 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149248 s, 70.3 MB/s 2024-04-17T10:32:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.857 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150091 s, 69.9 MB/s 2024-04-17T10:32:56.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:56.858 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:56.858 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:56.858 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.27 2024-04-17T10:32:56.859 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:56.859 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:56.859 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:56.859 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148598 s, 70.6 MB/s 2024-04-17T10:32:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133424 s, 78.6 MB/s 2024-04-17T10:32:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.958 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141237 s, 74.2 MB/s 2024-04-17T10:32:56.975 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.976 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152081 s, 68.9 MB/s 2024-04-17T10:32:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154861 s, 67.7 MB/s 2024-04-17T10:32:57.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.010 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015515 s, 67.6 MB/s 2024-04-17T10:32:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.025 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131954 s, 79.5 MB/s 2024-04-17T10:32:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.041 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139805 s, 75.0 MB/s 2024-04-17T10:32:57.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.057 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143678 s, 73.0 MB/s 2024-04-17T10:32:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.073 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140132 s, 74.8 MB/s 2024-04-17T10:32:57.089 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145834 s, 71.9 MB/s 2024-04-17T10:32:57.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.105 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141809 s, 73.9 MB/s 2024-04-17T10:32:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140635 s, 74.6 MB/s 2024-04-17T10:32:57.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135204 s, 77.6 MB/s 2024-04-17T10:32:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.152 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138714 s, 75.6 MB/s 2024-04-17T10:32:57.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.168 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136589 s, 76.8 MB/s 2024-04-17T10:32:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143728 s, 73.0 MB/s 2024-04-17T10:32:57.200 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.201 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150038 s, 69.9 MB/s 2024-04-17T10:32:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148768 s, 70.5 MB/s 2024-04-17T10:32:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.234 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014933 s, 70.2 MB/s 2024-04-17T10:32:57.250 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.250 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.250 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138641 s, 75.6 MB/s 2024-04-17T10:32:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.266 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013993 s, 74.9 MB/s 2024-04-17T10:32:57.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.283 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154516 s, 67.9 MB/s 2024-04-17T10:32:57.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.300 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151491 s, 69.2 MB/s 2024-04-17T10:32:57.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.317 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147785 s, 71.0 MB/s 2024-04-17T10:32:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.334 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152909 s, 68.6 MB/s 2024-04-17T10:32:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.350 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141535 s, 74.1 MB/s 2024-04-17T10:32:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.365 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132177 s, 79.3 MB/s 2024-04-17T10:32:57.382 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.382 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.382 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154426 s, 67.9 MB/s 2024-04-17T10:32:57.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.401 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176216 s, 59.5 MB/s 2024-04-17T10:32:57.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156324 s, 67.1 MB/s 2024-04-17T10:32:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.436 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014916 s, 70.3 MB/s 2024-04-17T10:32:57.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:57.437 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:57.438 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:57.438 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.28 2024-04-17T10:32:57.438 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:57.438 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:57.438 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:57.438 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.506 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135786 s, 77.2 MB/s 2024-04-17T10:32:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.524 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157567 s, 66.5 MB/s 2024-04-17T10:32:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.540 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142696 s, 73.5 MB/s 2024-04-17T10:32:57.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.555 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133117 s, 78.8 MB/s 2024-04-17T10:32:57.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.570 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128709 s, 81.5 MB/s 2024-04-17T10:32:57.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.587 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150845 s, 69.5 MB/s 2024-04-17T10:32:57.604 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.604 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.604 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154788 s, 67.7 MB/s 2024-04-17T10:32:57.620 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.621 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147526 s, 71.1 MB/s 2024-04-17T10:32:57.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.636 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.636 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132799 s, 79.0 MB/s 2024-04-17T10:32:57.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.651 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138375 s, 75.8 MB/s 2024-04-17T10:32:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154044 s, 68.1 MB/s 2024-04-17T10:32:57.683 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.683 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.683 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130537 s, 80.3 MB/s 2024-04-17T10:32:57.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.704 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188543 s, 55.6 MB/s 2024-04-17T10:32:57.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.720 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145165 s, 72.2 MB/s 2024-04-17T10:32:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.736 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134424 s, 78.0 MB/s 2024-04-17T10:32:57.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.751 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139284 s, 75.3 MB/s 2024-04-17T10:32:57.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:32:57.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:32:57.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138532 s, 75.7 MB/s 2024-04-17T10:32:57.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144729 s, 72.5 MB/s 2024-04-17T10:32:57.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146546 s, 71.6 MB/s 2024-04-17T10:32:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198269 s, 52.9 MB/s 2024-04-17T10:32:57.839 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.840 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166135 s, 63.1 MB/s 2024-04-17T10:32:57.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.854 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120816 s, 86.8 MB/s 2024-04-17T10:32:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.869 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138766 s, 75.6 MB/s 2024-04-17T10:32:57.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.889 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177484 s, 59.1 MB/s 2024-04-17T10:32:57.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.903 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130451 s, 80.4 MB/s 2024-04-17T10:32:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.925 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195589 s, 53.6 MB/s 2024-04-17T10:32:57.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.944 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169565 s, 61.8 MB/s 2024-04-17T10:32:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175584 s, 59.7 MB/s 2024-04-17T10:32:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.978 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133139 s, 78.8 MB/s 2024-04-17T10:32:57.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:57.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:57.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151538 s, 69.2 MB/s 2024-04-17T10:32:58.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138111 s, 75.9 MB/s 2024-04-17T10:32:58.026 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139549 s, 75.1 MB/s 2024-04-17T10:32:58.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:58.028 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:58.029 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:58.029 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.29 2024-04-17T10:32:58.029 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:58.029 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:58.029 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:58.029 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137128 s, 76.5 MB/s 2024-04-17T10:32:58.119 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.119 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.119 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182238 s, 57.5 MB/s 2024-04-17T10:32:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136565 s, 76.8 MB/s 2024-04-17T10:32:58.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181394 s, 57.8 MB/s 2024-04-17T10:32:58.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.170 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142034 s, 73.8 MB/s 2024-04-17T10:32:58.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143972 s, 72.8 MB/s 2024-04-17T10:32:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134407 s, 78.0 MB/s 2024-04-17T10:32:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.217 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137178 s, 76.4 MB/s 2024-04-17T10:32:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135114 s, 77.6 MB/s 2024-04-17T10:32:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.248 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140125 s, 74.8 MB/s 2024-04-17T10:32:58.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.264 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143621 s, 73.0 MB/s 2024-04-17T10:32:58.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.282 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154142 s, 68.0 MB/s 2024-04-17T10:32:58.298 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.298 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.298 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014315 s, 73.3 MB/s 2024-04-17T10:32:58.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150572 s, 69.6 MB/s 2024-04-17T10:32:58.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.332 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014634 s, 71.7 MB/s 2024-04-17T10:32:58.346 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133747 s, 78.4 MB/s 2024-04-17T10:32:58.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.363 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.363 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140233 s, 74.8 MB/s 2024-04-17T10:32:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.376 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115842 s, 90.5 MB/s 2024-04-17T10:32:58.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150057 s, 69.9 MB/s 2024-04-17T10:32:58.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141058 s, 74.3 MB/s 2024-04-17T10:32:58.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.425 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147901 s, 70.9 MB/s 2024-04-17T10:32:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144815 s, 72.4 MB/s 2024-04-17T10:32:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151701 s, 69.1 MB/s 2024-04-17T10:32:58.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139278 s, 75.3 MB/s 2024-04-17T10:32:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137693 s, 76.2 MB/s 2024-04-17T10:32:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154519 s, 67.9 MB/s 2024-04-17T10:32:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.522 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139241 s, 75.3 MB/s 2024-04-17T10:32:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133649 s, 78.5 MB/s 2024-04-17T10:32:58.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131601 s, 79.7 MB/s 2024-04-17T10:32:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146762 s, 71.4 MB/s 2024-04-17T10:32:58.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.584 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135119 s, 77.6 MB/s 2024-04-17T10:32:58.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.601 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147019 s, 71.3 MB/s 2024-04-17T10:32:58.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:58.602 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:58.602 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:58.602 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.30 2024-04-17T10:32:58.602 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:58.602 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:58.603 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:58.603 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014923 s, 70.3 MB/s 2024-04-17T10:32:58.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.688 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013159 s, 79.7 MB/s 2024-04-17T10:32:58.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.705 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147919 s, 70.9 MB/s 2024-04-17T10:32:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.725 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190685 s, 55.0 MB/s 2024-04-17T10:32:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.743 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156859 s, 66.8 MB/s 2024-04-17T10:32:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141242 s, 74.2 MB/s 2024-04-17T10:32:58.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.774 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129749 s, 80.8 MB/s 2024-04-17T10:32:58.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.790 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148272 s, 70.7 MB/s 2024-04-17T10:32:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.806 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141926 s, 73.9 MB/s 2024-04-17T10:32:58.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.822 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145032 s, 72.3 MB/s 2024-04-17T10:32:58.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.843 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188719 s, 55.6 MB/s 2024-04-17T10:32:58.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.860 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150479 s, 69.7 MB/s 2024-04-17T10:32:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.876 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146065 s, 71.8 MB/s 2024-04-17T10:32:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149448 s, 70.2 MB/s 2024-04-17T10:32:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143226 s, 73.2 MB/s 2024-04-17T10:32:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136104 s, 77.0 MB/s 2024-04-17T10:32:58.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.941 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154175 s, 68.0 MB/s 2024-04-17T10:32:58.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.962 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189787 s, 55.3 MB/s 2024-04-17T10:32:58.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133553 s, 78.5 MB/s 2024-04-17T10:32:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148398 s, 70.7 MB/s 2024-04-17T10:32:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.009 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139509 s, 75.2 MB/s 2024-04-17T10:32:59.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183686 s, 57.1 MB/s 2024-04-17T10:32:59.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.045 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133662 s, 78.4 MB/s 2024-04-17T10:32:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144837 s, 72.4 MB/s 2024-04-17T10:32:59.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.077 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134518 s, 78.0 MB/s 2024-04-17T10:32:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138398 s, 75.8 MB/s 2024-04-17T10:32:59.108 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.108 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.108 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143026 s, 73.3 MB/s 2024-04-17T10:32:59.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013628 s, 76.9 MB/s 2024-04-17T10:32:59.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.143 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176329 s, 59.5 MB/s 2024-04-17T10:32:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.159 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142716 s, 73.5 MB/s 2024-04-17T10:32:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.179 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182018 s, 57.6 MB/s 2024-04-17T10:32:59.197 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163748 s, 64.0 MB/s 2024-04-17T10:32:59.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:59.199 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:32:59.199 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:32:59.199 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.31 2024-04-17T10:32:59.199 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:32:59.199 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:32:59.199 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:32:59.199 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:32:59.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.267 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134921 s, 77.7 MB/s 2024-04-17T10:32:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149714 s, 70.0 MB/s 2024-04-17T10:32:59.298 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.298 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.299 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134671 s, 77.9 MB/s 2024-04-17T10:32:59.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149751 s, 70.0 MB/s 2024-04-17T10:32:59.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140214 s, 74.8 MB/s 2024-04-17T10:32:59.346 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134734 s, 77.8 MB/s 2024-04-17T10:32:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.362 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138493 s, 75.7 MB/s 2024-04-17T10:32:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141158 s, 74.3 MB/s 2024-04-17T10:32:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.395 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151463 s, 69.2 MB/s 2024-04-17T10:32:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013351 s, 78.5 MB/s 2024-04-17T10:32:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017741 s, 59.1 MB/s 2024-04-17T10:32:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146585 s, 71.5 MB/s 2024-04-17T10:32:59.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181516 s, 57.8 MB/s 2024-04-17T10:32:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.486 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165438 s, 63.4 MB/s 2024-04-17T10:32:59.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.502 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154956 s, 67.7 MB/s 2024-04-17T10:32:59.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.518 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139453 s, 75.2 MB/s 2024-04-17T10:32:59.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175492 s, 59.8 MB/s 2024-04-17T10:32:59.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.553 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134951 s, 77.7 MB/s 2024-04-17T10:32:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139647 s, 75.1 MB/s 2024-04-17T10:32:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130547 s, 80.3 MB/s 2024-04-17T10:32:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.598 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136471 s, 76.8 MB/s 2024-04-17T10:32:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155959 s, 67.2 MB/s 2024-04-17T10:32:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.637 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191251 s, 54.8 MB/s 2024-04-17T10:32:59.652 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.652 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.652 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134686 s, 77.9 MB/s 2024-04-17T10:32:59.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.668 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140588 s, 74.6 MB/s 2024-04-17T10:32:59.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.685 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147483 s, 71.1 MB/s 2024-04-17T10:32:59.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.701 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148922 s, 70.4 MB/s 2024-04-17T10:32:59.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.717 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138387 s, 75.8 MB/s 2024-04-17T10:32:59.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.733 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139004 s, 75.4 MB/s 2024-04-17T10:32:59.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.753 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185762 s, 56.4 MB/s 2024-04-17T10:32:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126506 s, 82.9 MB/s 2024-04-17T10:32:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:32:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:32:59.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140207 s, 74.8 MB/s 2024-04-17T10:32:59.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:32:59.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub start /scrub_control_test_path recursive 2024-04-17T10:32:59.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dc6600640 1 Processor -- start 2024-04-17T10:32:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dc6600640 1 -- start start 2024-04-17T10:32:59.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dc6600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0144100 0x7f9dc00cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dc6600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9dc006ebe0 0x7f9dc00b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dc6600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dc0071bb0 con 0x7f9dc0071360 2024-04-17T10:32:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dc6600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dc00b3140 con 0x7f9dc006ebe0 2024-04-17T10:32:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dc6600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc00b3420 con 0x7f9dc0144100 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbfe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9dc006ebe0 0x7f9dc00b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbfe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9dc006ebe0 0x7f9dc00b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:57904/0 (socket says 172.21.15.67:57904) 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbfe00640 1 -- 172.21.15.67:0/1115226577 learned_addr learned my addr 172.21.15.67:0/1115226577 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbf400640 1 --2- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0144100 0x7f9dc00cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9db6a00640 1 --1- 172.21.15.67:0/1115226577 >> v1:172.21.15.86:6789/0 conn(0x7f9dc0071360 0x7f9dc006e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:60654/0 (socket says 172.21.15.67:60654) 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbfe00640 1 -- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0144100 msgr2=0x7f9dc00cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbfe00640 1 --2- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0144100 0x7f9dc00cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbfe00640 1 -- 172.21.15.67:0/1115226577 >> v1:172.21.15.86:6789/0 conn(0x7f9dc0071360 legacy=0x7f9dc006e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:59.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbfe00640 1 -- 172.21.15.67:0/1115226577 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0071e90 con 0x7f9dc006ebe0 2024-04-17T10:32:59.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.982+0000 7f9dbf400640 1 --2- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0144100 0x7f9dc00cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:59.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dbfe00640 1 --2- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 0x7f9dc00b38b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009e50 tx=0x7f9db801ca00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52188b63ca6dd1b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:59.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dbea00640 1 -- 172.21.15.67:0/1115226577 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80a58f0 con 0x7f9dc006ebe0 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dbea00640 1 -- 172.21.15.67:0/1115226577 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db80a5a90 con 0x7f9dc006ebe0 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dbea00640 1 -- 172.21.15.67:0/1115226577 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db80a5e60 con 0x7f9dc006ebe0 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 -- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 msgr2=0x7f9dc00b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 --2- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 0x7f9dc00b38b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f9db8009e50 tx=0x7f9db801ca00 comp rx=0 tx=0).stop 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 -- 172.21.15.67:0/1115226577 shutdown_connections 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 --2- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 0x7f9dc00b38b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 --2- 172.21.15.67:0/1115226577 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0144100 0x7f9dc00cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 -- 172.21.15.67:0/1115226577 >> 172.21.15.67:0/1115226577 conn(0x7f9dc006ac90 msgr2=0x7f9dc006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 -- 172.21.15.67:0/1115226577 shutdown_connections 2024-04-17T10:32:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.986+0000 7f9dc6600640 1 -- 172.21.15.67:0/1115226577 wait complete. 2024-04-17T10:32:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 Processor -- start 2024-04-17T10:32:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 -- start start 2024-04-17T10:32:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 0x7f9dc01373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0071360 0x7f9dc0134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc0144100 0x7f9dc01350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc00b4600 con 0x7f9dc0144100 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0071950 con 0x7f9dc006ebe0 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc00b4780 con 0x7f9dc0071360 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc0144100 0x7f9dc01350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0071360 0x7f9dc0134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0071360 0x7f9dc0134ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:41132/0 (socket says 172.21.15.67:41132) 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 -- 172.21.15.67:0/3651729808 learned_addr learned my addr 172.21.15.67:0/3651729808 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 0x7f9dc01373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 -- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 msgr2=0x7f9dc01373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 0x7f9dc01373d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 -- 172.21.15.67:0/3651729808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc0144100 msgr2=0x7f9dc01350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc0144100 0x7f9dc01350e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 -- 172.21.15.67:0/3651729808 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc0136330 con 0x7f9dc0071360 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db6a00640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0071360 0x7f9dc0134ba0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9da800c450 tx=0x7f9da800c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db7e00640 1 -- 172.21.15.67:0/3651729808 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da8018070 con 0x7f9dc0071360 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db7e00640 1 -- 172.21.15.67:0/3651729808 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da8007480 con 0x7f9dc0071360 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dc0136610 con 0x7f9dc0071360 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9db7e00640 1 -- 172.21.15.67:0/3651729808 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da80047f0 con 0x7f9dc0071360 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.990+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dc0136bd0 con 0x7f9dc0071360 2024-04-17T10:32:59.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9db7e00640 1 -- 172.21.15.67:0/3651729808 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9da801b070 con 0x7f9dc0071360 2024-04-17T10:32:59.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9db7e00640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d9c066f60 0x7f9d9c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9db7e00640 1 -- 172.21.15.67:0/3651729808 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9da8084330 con 0x7f9dc0071360 2024-04-17T10:32:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dbf400640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d9c066f60 0x7f9d9c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 Processor -- start 2024-04-17T10:32:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 -- start start 2024-04-17T10:32:59.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dbf400640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d9c066f60 0x7f9d9c069420 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009580 tx=0x7f9db0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:59.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b33f0 0x56219c9b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c969fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219c96a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219c7b7d50 con 0x56219c9b33f0 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219c73fe40 con 0x56219c9b3d10 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56219c7b3df0 con 0x56219c96a520 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b33f0 0x56219c9b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b33f0 0x56219c9b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57980/0 (socket says 172.21.15.67:57980) 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dbf400640 1 -- 172.21.15.67:0/1451068835 learned_addr learned my addr 172.21.15.67:0/1451068835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9db6a00640 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c969fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.994+0000 7f9dbfe00640 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219c96a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 -- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 msgr2=0x56219c96a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219c96a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 -- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 msgr2=0x56219c969fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c969fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 -- 172.21.15.67:0/1451068835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56219c754d90 con 0x56219c9b33f0 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9db6a00640 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c969fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b33f0 0x56219c9b37d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f9db0011df0 tx=0x7f9db0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193bfd85fe133645 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dafe00640 1 -- 172.21.15.67:0/1451068835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0025070 con 0x56219c9b33f0 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dafe00640 1 -- 172.21.15.67:0/1451068835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db001b070 con 0x56219c9b33f0 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dafe00640 1 -- 172.21.15.67:0/1451068835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db00213f0 con 0x56219c9b33f0 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- 172.21.15.67:0/1451068835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b33f0 msgr2=0x56219c9b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b33f0 0x56219c9b37d0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f9db0011df0 tx=0x7f9db0014000 comp rx=0 tx=0).stop 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- 172.21.15.67:0/1451068835 shutdown_connections 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219c96a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c969fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/1451068835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b33f0 0x56219c9b37d0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:32:59.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- 172.21.15.67:0/1451068835 >> 172.21.15.67:0/1451068835 conn(0x56219c9664d0 msgr2=0x56219c9688c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:32:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- 172.21.15.67:0/1451068835 shutdown_connections 2024-04-17T10:32:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- 172.21.15.67:0/1451068835 wait complete. 2024-04-17T10:32:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- start start 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 Processor -- start 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c970190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b33f0 0x56219c9a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c9a1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219c75d610 con 0x56219c96a520 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219c7729a0 con 0x56219c9b3d10 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56219c7bc380 con 0x56219c9b33f0 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dc9b09000 10 client.0 _ll_register_callbacks cb 0x56219c86c0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c970190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c970190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57984/0 (socket says 172.21.15.67:57984) 2024-04-17T10:33:00.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbf400640 1 -- 172.21.15.67:0/40527314 learned_addr learned my addr 172.21.15.67:0/40527314 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9db6a00640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b33f0 0x56219c9a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:32:59.998+0000 7f9dbfe00640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c9a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dbf400640 1 -- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b33f0 msgr2=0x56219c9a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dbf400640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b33f0 0x56219c9a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dbf400640 1 -- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 msgr2=0x56219c9a1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dbf400640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c9a1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dbf400640 1 -- 172.21.15.67:0/40527314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56219c7b2d40 con 0x56219c96a520 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9db6a00640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b33f0 0x56219c9a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dbf400640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c970190 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x56219c75a640 tx=0x7f9db0004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39701379745d8bd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 1 -- 172.21.15.67:0/40527314 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0025070 con 0x56219c96a520 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 1 -- 172.21.15.67:0/40527314 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9db0003820 con 0x56219c96a520 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9db001b070 con 0x56219c96a520 2024-04-17T10:33:00.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 1 -- 172.21.15.67:0/40527314 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9db003b050 con 0x56219c96a520 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 10 client.8292 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56219c83af00 con 0x56219c96a520 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 1 -- 172.21.15.67:0/40527314 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db001b070 con 0x56219c96a520 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 1 -- 172.21.15.67:0/40527314 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9db003e020 con 0x56219c96a520 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 10 client.8292 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 10 client.8292 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 20 client.8292 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9ba4c0 0x56219c9ba8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 4 client.8292 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56219c753a20 con 0x56219c9ba4c0 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9db6a00640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9ba4c0 0x56219c9ba8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9db6a00640 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9ba4c0 0x56219c9ba8d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cf35fe1a86b1f26 server_cookie=54aea6ded9e1b6ec in_seq=0 out_seq=0 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 10 client.8292 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 1 -- 172.21.15.67:0/40527314 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56219c753a20 con 0x56219c9ba4c0 2024-04-17T10:33:00.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 10 client.8292 handle_command_reply: tid=0 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 client.8292 shutdown 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 msgr2=0x56219c970190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c970190 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x56219c75a640 tx=0x7f9db0004740 comp rx=0 tx=0).stop 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 shutdown_connections 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9ba4c0 0x56219c9ba8d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b3d10 0x56219c9a1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b33f0 0x56219c9a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dacc00640 0 client.8292 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/40527314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c970190 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 >> 172.21.15.67:0/40527314 conn(0x56219c9664d0 msgr2=0x56219c969150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 shutdown_connections 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 1 -- 172.21.15.67:0/40527314 wait complete. 2024-04-17T10:33:00.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.002+0000 7f9dc9b09000 20 client.8292 trim_cache size 0 max 16384 2024-04-17T10:33:00.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.018+0000 7f9dc9b09000 1 Processor -- start 2024-04-17T10:33:00.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.018+0000 7f9dc9b09000 1 -- start start 2024-04-17T10:33:00.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c9688c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219c968e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b3d10 0x56219c9b6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0142ec0 con 0x56219c96a520 2024-04-17T10:33:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0142080 con 0x56219c9b33f0 2024-04-17T10:33:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc0140c70 con 0x56219c9b3d10 2024-04-17T10:33:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c9688c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c9688c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58000/0 (socket says 172.21.15.67:58000) 2024-04-17T10:33:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 -- 172.21.15.67:0/2903873313 learned_addr learned my addr 172.21.15.67:0/2903873313 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbfe00640 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b3d10 0x56219c9b6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9db6a00640 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219c968e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 -- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b3d10 msgr2=0x56219c9b6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b3d10 0x56219c9b6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 -- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 msgr2=0x56219c968e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219c968e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 -- 172.21.15.67:0/2903873313 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56219c9a1970 con 0x56219c96a520 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dbf400640 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c9688c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f9db00046a0 tx=0x7f9db002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f798205cea8cc1fd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9daf400640 1 -- 172.21.15.67:0/2903873313 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0003620 con 0x56219c96a520 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9daf400640 1 -- 172.21.15.67:0/2903873313 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db00037c0 con 0x56219c96a520 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9daf400640 1 -- 172.21.15.67:0/2903873313 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db0003d10 con 0x56219c96a520 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- 172.21.15.67:0/2903873313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 msgr2=0x56219c9688c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c9688c0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f9db00046a0 tx=0x7f9db002c040 comp rx=0 tx=0).stop 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- 172.21.15.67:0/2903873313 shutdown_connections 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c9b3d10 0x56219c9b6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219c968e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/2903873313 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c96a520 0x56219c9688c0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- 172.21.15.67:0/2903873313 >> 172.21.15.67:0/2903873313 conn(0x56219c9664d0 msgr2=0x56219c96f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- 172.21.15.67:0/2903873313 shutdown_connections 2024-04-17T10:33:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- 172.21.15.67:0/2903873313 wait complete. 2024-04-17T10:33:00.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 -- start start 2024-04-17T10:33:00.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.022+0000 7f9dc9b09000 1 Processor -- start 2024-04-17T10:33:00.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219ca18180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219ca186c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b3d10 0x56219c9b8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219c76edc0 con 0x56219c9b3d10 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dc014a230 con 0x56219c9b33f0 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56219c7b7060 con 0x56219c96a520 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 10 client.0 _ll_register_callbacks cb 0x7f9db0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219ca18180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b3d10 0x56219c9b8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9db6a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219ca186c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219ca18180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:41152/0 (socket says 172.21.15.67:41152) 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 -- 172.21.15.67:0/3080052074 learned_addr learned my addr 172.21.15.67:0/3080052074 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 -- 172.21.15.67:0/3080052074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 msgr2=0x56219ca186c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219ca186c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 -- 172.21.15.67:0/3080052074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b3d10 msgr2=0x56219c9b8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b3d10 0x56219c9b8420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 -- 172.21.15.67:0/3080052074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56219c811330 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dbf400640 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219ca18180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9db00047c0 tx=0x7f9db0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8f2b757a9fdd257 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dacc00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3301/0 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dacc00640 1 -- 172.21.15.67:0/3080052074 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db004d060 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9db00037c0 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dacc00640 1 -- 172.21.15.67:0/3080052074 <== mon.2 v2:172.21.15.86:3301/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9db0003b80 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dacc00640 1 -- 172.21.15.67:0/3080052074 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9db0053050 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 10 client.7001 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56219c7845f0 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dacc00640 1 -- 172.21.15.67:0/3080052074 <== mon.2 v2:172.21.15.86:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db00037c0 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dacc00640 1 -- 172.21.15.67:0/3080052074 <== mon.2 v2:172.21.15.86:3301/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9db0056020 con 0x56219c96a520 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 10 client.7001 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 10 client.7001 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 20 client.7001 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9b8d80 0x56219c9b9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 4 client.7001 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-04-17T10:33:00.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x56219c7c5290 con 0x56219c9b8d80 2024-04-17T10:33:00.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9db6a00640 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9b8d80 0x56219c9b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9dacc00640 10 client.7001 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.026+0000 7f9db6a00640 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9b8d80 0x56219c9b9190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7951a3f75882d0bb server_cookie=8afe21eab8d32766 in_seq=0 out_seq=0 2024-04-17T10:33:00.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dacc00640 1 -- 172.21.15.67:0/3080052074 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56219c7c5290 con 0x56219c9b8d80 2024-04-17T10:33:00.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dacc00640 10 client.7001 handle_command_reply: tid=0 2024-04-17T10:33:00.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 client.7001 shutdown 2024-04-17T10:33:00.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 msgr2=0x56219ca18180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219ca18180 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9db00047c0 tx=0x7f9db0018000 comp rx=0 tx=0).stop 2024-04-17T10:33:00.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 shutdown_connections 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x56219c9b8d80 0x56219c9b9190 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56219c9b3d10 0x56219c9b8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56219c9b33f0 0x56219ca186c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 --2- 172.21.15.67:0/3080052074 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56219c96a520 0x56219ca18180 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dacc00640 0 client.7001 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 >> 172.21.15.67:0/3080052074 conn(0x56219c9664d0 msgr2=0x56219c9b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 shutdown_connections 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 1 -- 172.21.15.67:0/3080052074 wait complete. 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc9b09000 20 client.7001 trim_cache size 0 max 16384 2024-04-17T10:33:00.023 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:00.024 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:33:00.024 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "85eff2e3-62e3-496b-a911-123cb3de8df8", 2024-04-17T10:33:00.024 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:33:00.024 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:00.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d9c066f60 msgr2=0x7f9d9c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d9c066f60 0x7f9d9c069420 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9db0009580 tx=0x7f9db0009190 comp rx=0 tx=0).stop 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0071360 msgr2=0x7f9dc0134ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0071360 0x7f9dc0134ba0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9da800c450 tx=0x7f9da800c910 comp rx=0 tx=0).stop 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 shutdown_connections 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9d9c066f60 0x7f9d9c069420 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9dc0144100 0x7f9dc01350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9dc0071360 0x7f9dc0134ba0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 --2- 172.21.15.67:0/3651729808 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9dc006ebe0 0x7f9dc01373d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 >> 172.21.15.67:0/3651729808 conn(0x7f9dc006ac90 msgr2=0x7f9dc013b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 shutdown_connections 2024-04-17T10:33:00.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.030+0000 7f9dc6600640 1 -- 172.21.15.67:0/3651729808 wait complete. 2024-04-17T10:33:00.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:00.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub pause 2024-04-17T10:33:00.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce98e00640 1 Processor -- start 2024-04-17T10:33:00.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce98e00640 1 -- start start 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce98e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9415dcf0 0x7fce940cf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce98e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fce940ce7a0 0x7fce940b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce98e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce940d0040 con 0x7fce9406cac0 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce98e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce940cfde0 con 0x7fce940ce7a0 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce98e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce940d0320 con 0x7fce9415dcf0 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce89400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fce9406cac0 0x7fce940ce090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:33624/0 (socket says 172.21.15.67:33624) 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce89400640 1 -- 172.21.15.67:0/2166326913 learned_addr learned my addr 172.21.15.67:0/2166326913 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce92800640 1 --2- 172.21.15.67:0/2166326913 >> v2:172.21.15.67:3300/0 conn(0x7fce940ce7a0 0x7fce940b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce91e00640 1 --2- 172.21.15.67:0/2166326913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9415dcf0 0x7fce940cf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce92800640 1 -- 172.21.15.67:0/2166326913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9415dcf0 msgr2=0x7fce940cf8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce92800640 1 --2- 172.21.15.67:0/2166326913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9415dcf0 0x7fce940cf8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce92800640 1 -- 172.21.15.67:0/2166326913 >> v1:172.21.15.67:6789/0 conn(0x7fce9406cac0 legacy=0x7fce940ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce91400640 1 -- 172.21.15.67:0/2166326913 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014425543 0 0) 0x7fce940d0040 con 0x7fce9406cac0 2024-04-17T10:33:00.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.242+0000 7fce92800640 1 -- 172.21.15.67:0/2166326913 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce940d0600 con 0x7fce940ce7a0 2024-04-17T10:33:00.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce92800640 1 --2- 172.21.15.67:0/2166326913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce940ce7a0 0x7fce940b1480 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fce8c009dc0 tx=0x7fce8c01c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6896bd7d5e8bb24e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce91400640 1 -- 172.21.15.67:0/2166326913 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c0b2070 con 0x7fce940ce7a0 2024-04-17T10:33:00.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce91400640 1 -- 172.21.15.67:0/2166326913 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c0a59e0 con 0x7fce940ce7a0 2024-04-17T10:33:00.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce91400640 1 -- 172.21.15.67:0/2166326913 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c0a5db0 con 0x7fce940ce7a0 2024-04-17T10:33:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 -- 172.21.15.67:0/2166326913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce940ce7a0 msgr2=0x7fce940b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 --2- 172.21.15.67:0/2166326913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce940ce7a0 0x7fce940b1480 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fce8c009dc0 tx=0x7fce8c01c920 comp rx=0 tx=0).stop 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 -- 172.21.15.67:0/2166326913 shutdown_connections 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 --2- 172.21.15.67:0/2166326913 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce940ce7a0 0x7fce940b1480 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 --2- 172.21.15.67:0/2166326913 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce9415dcf0 0x7fce940cf8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 -- 172.21.15.67:0/2166326913 >> 172.21.15.67:0/2166326913 conn(0x7fce940b33a0 msgr2=0x7fce940b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 -- 172.21.15.67:0/2166326913 shutdown_connections 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.246+0000 7fce98e00640 1 -- 172.21.15.67:0/2166326913 wait complete. 2024-04-17T10:33:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 Processor -- start 2024-04-17T10:33:00.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 -- start start 2024-04-17T10:33:00.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9406cac0 0x7fce941558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce940ce7a0 0x7fce941502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9415dcf0 0x7fce94148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce940b2350 con 0x7fce9415dcf0 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce940b21d0 con 0x7fce9406cac0 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce940b24d0 con 0x7fce940ce7a0 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9415dcf0 0x7fce94148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9415dcf0 0x7fce94148b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58016/0 (socket says 172.21.15.67:58016) 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 -- 172.21.15.67:0/3314651289 learned_addr learned my addr 172.21.15.67:0/3314651289 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce91e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9406cac0 0x7fce941558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce89400640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce940ce7a0 0x7fce941502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 -- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce940ce7a0 msgr2=0x7fce941502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce940ce7a0 0x7fce941502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 -- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9406cac0 msgr2=0x7fce941558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9406cac0 0x7fce941558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 -- 172.21.15.67:0/3314651289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce94153620 con 0x7fce9415dcf0 2024-04-17T10:33:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce92800640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9415dcf0 0x7fce94148b00 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0b0040 tx=0x7fce8c0196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce8aa00640 1 -- 172.21.15.67:0/3314651289 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c0b2050 con 0x7fce9415dcf0 2024-04-17T10:33:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce8aa00640 1 -- 172.21.15.67:0/3314651289 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c0b3040 con 0x7fce9415dcf0 2024-04-17T10:33:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce8aa00640 1 -- 172.21.15.67:0/3314651289 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c0ad8b0 con 0x7fce9415dcf0 2024-04-17T10:33:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce941538c0 con 0x7fce9415dcf0 2024-04-17T10:33:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.250+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce94153e80 con 0x7fce9415dcf0 2024-04-17T10:33:00.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce8aa00640 1 -- 172.21.15.67:0/3314651289 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fce8c0c0030 con 0x7fce9415dcf0 2024-04-17T10:33:00.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce8aa00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce70066f90 0x7fce70069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce8aa00640 1 -- 172.21.15.67:0/3314651289 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fce8c11f7b0 con 0x7fce9415dcf0 2024-04-17T10:33:00.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce91e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce70066f90 0x7fce70069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce9c389000 1 Processor -- start 2024-04-17T10:33:00.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce9c389000 1 -- start start 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce91e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce70066f90 0x7fce70069450 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fce84009ac0 tx=0x7fce84009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.254+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x560765313e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765314360 0x5607652ce850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x5607652cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607651b11f0 con 0x560765314360 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560765117e90 con 0x5607652ced90 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56076509ff50 con 0x560765313a40 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765314360 0x5607652ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x560765313e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x560765313e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:41170/0 (socket says 172.21.15.67:41170) 2024-04-17T10:33:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 -- 172.21.15.67:0/4074814284 learned_addr learned my addr 172.21.15.67:0/4074814284 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765314360 0x5607652ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58018/0 (socket says 172.21.15.67:58018) 2024-04-17T10:33:00.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce92800640 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x5607652cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 -- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 msgr2=0x5607652cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x5607652cf170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 -- 172.21.15.67:0/4074814284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765314360 msgr2=0x5607652ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765314360 0x5607652ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 -- 172.21.15.67:0/4074814284 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607650ef8b0 con 0x560765313a40 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce91e00640 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x560765313e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fce84006aa0 tx=0x7fce84014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a031534f8f152b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce82a00640 1 -- 172.21.15.67:0/4074814284 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8401b070 con 0x560765313a40 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce82a00640 1 -- 172.21.15.67:0/4074814284 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce84027070 con 0x560765313a40 2024-04-17T10:33:00.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce82a00640 1 -- 172.21.15.67:0/4074814284 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84003a30 con 0x560765313a40 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 msgr2=0x560765313e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x560765313e20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fce84006aa0 tx=0x7fce84014000 comp rx=0 tx=0).stop 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- 172.21.15.67:0/4074814284 shutdown_connections 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x5607652cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765314360 0x5607652ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- 172.21.15.67:0/4074814284 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x560765313e20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- 172.21.15.67:0/4074814284 >> 172.21.15.67:0/4074814284 conn(0x5607652cad40 msgr2=0x5607652cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- 172.21.15.67:0/4074814284 shutdown_connections 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- 172.21.15.67:0/4074814284 wait complete. 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- start start 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 Processor -- start 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5607652ced90 0x56076531b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x56076531b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x5607652d4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607650d2540 con 0x560765313a40 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560765113f30 con 0x560765314360 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607650e2420 con 0x5607652ced90 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.258+0000 7fce9c389000 10 client.0 _ll_register_callbacks cb 0x5607651cc180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x56076531b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x56076531b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58030/0 (socket says 172.21.15.67:58030) 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 -- 172.21.15.67:0/3559467845 learned_addr learned my addr 172.21.15.67:0/3559467845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce92800640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x5607652d4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce91e00640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5607652ced90 0x56076531b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 -- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5607652ced90 msgr2=0x56076531b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5607652ced90 0x56076531b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 -- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 msgr2=0x5607652d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x5607652d4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 -- 172.21.15.67:0/3559467845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607650a01a0 con 0x560765313a40 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce92800640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x5607652d4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce89400640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x56076531b700 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0011f0 tx=0x7fce7c0016d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1e5202d0bbfcc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 1 -- 172.21.15.67:0/3559467845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c0145b0 con 0x560765313a40 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 1 -- 172.21.15.67:0/3559467845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fce7c00c910 con 0x560765313a40 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce84027070 con 0x560765313a40 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 1 -- 172.21.15.67:0/3559467845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce7c00cb10 con 0x560765313a40 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 10 client.8313 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x56076519a840 con 0x560765313a40 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 1 -- 172.21.15.67:0/3559467845 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce84027070 con 0x560765313a40 2024-04-17T10:33:00.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 1 -- 172.21.15.67:0/3559467845 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fce7c00ccb0 con 0x560765313a40 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 10 client.8313 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 10 client.8313 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 20 client.8313 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5607652d5ba0 0x5607652d5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 4 client.8313 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607650b3be0 con 0x5607652d5ba0 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce91e00640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5607652d5ba0 0x5607652d5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce91e00640 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5607652d5ba0 0x5607652d5fb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71e084664ea8762f server_cookie=bfe95b6ae33efff9 in_seq=0 out_seq=0 2024-04-17T10:33:00.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 10 client.8313 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 1 -- 172.21.15.67:0/3559467845 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5607650b3be0 con 0x5607652d5ba0 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 10 client.8313 handle_command_reply: tid=0 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 client.8313 shutdown 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 msgr2=0x56076531b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x56076531b700 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0011f0 tx=0x7fce7c0016d0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 shutdown_connections 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5607652d5ba0 0x5607652d5fb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x5607652d4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x56076531b700 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 --2- 172.21.15.67:0/3559467845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5607652ced90 0x56076531b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce77400640 0 client.8313 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 >> 172.21.15.67:0/3559467845 conn(0x5607652cad40 msgr2=0x5607652cd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 shutdown_connections 2024-04-17T10:33:00.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 1 -- 172.21.15.67:0/3559467845 wait complete. 2024-04-17T10:33:00.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.262+0000 7fce9c389000 20 client.8313 trim_cache size 0 max 16384 2024-04-17T10:33:00.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 Processor -- start 2024-04-17T10:33:00.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 -- start start 2024-04-17T10:33:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x560765370290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x5607653707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765314360 0x560765370d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce94202fc0 con 0x560765313a40 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce942021a0 con 0x5607652ced90 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9414f550 con 0x560765314360 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x5607653707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce89400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x5607653707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58034/0 (socket says 172.21.15.67:58034) 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.278+0000 7fce89400640 1 -- 172.21.15.67:0/1138178705 learned_addr learned my addr 172.21.15.67:0/1138178705 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce92800640 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765314360 0x560765370d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce91e00640 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x560765370290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce89400640 1 -- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765314360 msgr2=0x560765370d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce89400640 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765314360 0x560765370d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce89400640 1 -- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 msgr2=0x560765370290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce89400640 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x560765370290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce89400640 1 -- 172.21.15.67:0/1138178705 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607652d4d40 con 0x560765313a40 2024-04-17T10:33:00.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce89400640 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x5607653707d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fce7c001ec0 tx=0x7fce7c00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625939d01661d340 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce82000640 1 -- 172.21.15.67:0/1138178705 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c02e070 con 0x560765313a40 2024-04-17T10:33:00.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce82000640 1 -- 172.21.15.67:0/1138178705 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00b040 con 0x560765313a40 2024-04-17T10:33:00.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce82000640 1 -- 172.21.15.67:0/1138178705 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00c560 con 0x560765313a40 2024-04-17T10:33:00.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- 172.21.15.67:0/1138178705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 msgr2=0x5607653707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x5607653707d0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fce7c001ec0 tx=0x7fce7c00f000 comp rx=0 tx=0).stop 2024-04-17T10:33:00.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- 172.21.15.67:0/1138178705 shutdown_connections 2024-04-17T10:33:00.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765314360 0x560765370d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x560765313a40 0x5607653707d0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 --2- 172.21.15.67:0/1138178705 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5607652ced90 0x560765370290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- 172.21.15.67:0/1138178705 >> 172.21.15.67:0/1138178705 conn(0x5607652cad40 msgr2=0x560765312160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- 172.21.15.67:0/1138178705 shutdown_connections 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- 172.21.15.67:0/1138178705 wait complete. 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- start start 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 Processor -- start 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607652ced90 0x5607653744b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x5607653749f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x560765318ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607652cabc0 con 0x5607652ced90 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce94201c00 con 0x560765314360 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607650d9e50 con 0x560765313a40 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce9c389000 10 client.0 _ll_register_callbacks cb 0x5607650daa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce91e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607652ced90 0x5607653744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce91e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607652ced90 0x5607653744b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58048/0 (socket says 172.21.15.67:58048) 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce91e00640 1 -- 172.21.15.67:0/1756240949 learned_addr learned my addr 172.21.15.67:0/1756240949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce89400640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x5607653749f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.282+0000 7fce92800640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x560765318ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce91e00640 1 -- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 msgr2=0x5607653749f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce91e00640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x5607653749f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce91e00640 1 -- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 msgr2=0x560765318ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce91e00640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x560765318ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce91e00640 1 -- 172.21.15.67:0/1756240949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560765171a10 con 0x5607652ced90 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce92800640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x560765318ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce91e00640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607652ced90 0x5607653744b0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fce84024040 tx=0x7fce84004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178e1229025c94aa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 1 -- 172.21.15.67:0/1756240949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8401b040 con 0x5607652ced90 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce7c00b040 con 0x5607652ced90 2024-04-17T10:33:00.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 1 -- 172.21.15.67:0/1756240949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fce84041c70 con 0x5607652ced90 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 1 -- 172.21.15.67:0/1756240949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce84048050 con 0x5607652ced90 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 10 client.8319 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x5607650acb00 con 0x5607652ced90 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 1 -- 172.21.15.67:0/1756240949 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00b040 con 0x5607652ced90 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 1 -- 172.21.15.67:0/1756240949 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fce8404b020 con 0x5607652ced90 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 10 client.8319 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 10 client.8319 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 20 client.8319 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x560765319bc0 0x560765319fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 4 client.8319 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "scrub pause"}) v1 -- 0x5607651253d0 con 0x560765319bc0 2024-04-17T10:33:00.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce89400640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x560765319bc0 0x560765319fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 10 client.8319 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce89400640 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x560765319bc0 0x560765319fa0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c76a365bfe8827a2 server_cookie=3987c7291c2d5f0 in_seq=0 out_seq=0 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 1 -- 172.21.15.67:0/1756240949 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x5607651253d0 con 0x560765319bc0 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 10 client.8319 handle_command_reply: tid=0 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 client.8319 shutdown 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607652ced90 msgr2=0x5607653744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607652ced90 0x5607653744b0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7fce84024040 tx=0x7fce84004740 comp rx=0 tx=0).stop 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 shutdown_connections 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x560765319bc0 0x560765319fa0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x560765314360 0x560765318ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x560765313a40 0x5607653749f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce77400640 0 client.8319 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 --2- 172.21.15.67:0/1756240949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5607652ced90 0x5607653744b0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 >> 172.21.15.67:0/1756240949 conn(0x5607652cad40 msgr2=0x5607653129f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 shutdown_connections 2024-04-17T10:33:00.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 1 -- 172.21.15.67:0/1756240949 wait complete. 2024-04-17T10:33:00.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce9c389000 20 client.8319 trim_cache size 0 max 16384 2024-04-17T10:33:00.283 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:00.283 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-04-17T10:33:00.283 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce70066f90 msgr2=0x7fce70069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce70066f90 0x7fce70069450 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fce84009ac0 tx=0x7fce84009190 comp rx=0 tx=0).stop 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9415dcf0 msgr2=0x7fce94148b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9415dcf0 0x7fce94148b00 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fce8c0b0040 tx=0x7fce8c0196c0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 shutdown_connections 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fce70066f90 0x7fce70069450 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fce9415dcf0 0x7fce94148b00 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fce940ce7a0 0x7fce941502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 --2- 172.21.15.67:0/3314651289 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fce9406cac0 0x7fce941558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 >> 172.21.15.67:0/3314651289 conn(0x7fce940b33a0 msgr2=0x7fce9415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 shutdown_connections 2024-04-17T10:33:00.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.286+0000 7fce98e00640 1 -- 172.21.15.67:0/3314651289 wait complete. 2024-04-17T10:33:00.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:00.297 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-17T10:33:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50dba00640 1 Processor -- start 2024-04-17T10:33:00.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50dba00640 1 -- start start 2024-04-17T10:33:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50dba00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50d41583f0 0x7f50d4072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50dba00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d415f360 con 0x7f50d40728c0 2024-04-17T10:33:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50dba00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d40b3160 con 0x7f50d40b5300 2024-04-17T10:33:00.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50dba00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d40b3440 con 0x7f50d41583f0 2024-04-17T10:33:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50d9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50d41583f0 0x7f50d4072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50d3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f50d40728c0 0x7f50d40b4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44704/0 (socket says 172.21.15.67:44704) 2024-04-17T10:33:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50d3e00640 1 -- 172.21.15.67:0/3361650800 learned_addr learned my addr 172.21.15.67:0/3361650800 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50d9000640 1 -- 172.21.15.67:0/3361650800 >> v1:172.21.15.86:6789/0 conn(0x7f50d40b5300 legacy=0x7f50d406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50d9000640 1 -- 172.21.15.67:0/3361650800 >> v1:172.21.15.67:6789/0 conn(0x7f50d40728c0 legacy=0x7f50d40b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.450+0000 7f50d9000640 1 -- 172.21.15.67:0/3361650800 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d40b3720 con 0x7f50d41583f0 2024-04-17T10:33:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50d9000640 1 --2- 172.21.15.67:0/3361650800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d41583f0 0x7f50d4072380 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f50c80097c0 tx=0x7f50c801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b54bf80c47ebbfa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50d3400640 1 -- 172.21.15.67:0/3361650800 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80b1070 con 0x7f50d41583f0 2024-04-17T10:33:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50d3400640 1 -- 172.21.15.67:0/3361650800 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c80ac5f0 con 0x7f50d41583f0 2024-04-17T10:33:00.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50d3400640 1 -- 172.21.15.67:0/3361650800 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80ac9c0 con 0x7f50d41583f0 2024-04-17T10:33:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50dba00640 1 -- 172.21.15.67:0/3361650800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d41583f0 msgr2=0x7f50d4072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50dba00640 1 --2- 172.21.15.67:0/3361650800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d41583f0 0x7f50d4072380 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f50c80097c0 tx=0x7f50c801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50dba00640 1 -- 172.21.15.67:0/3361650800 shutdown_connections 2024-04-17T10:33:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50dba00640 1 --2- 172.21.15.67:0/3361650800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d41583f0 0x7f50d4072380 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50dba00640 1 -- 172.21.15.67:0/3361650800 >> 172.21.15.67:0/3361650800 conn(0x7f50d40b2570 msgr2=0x7f50d4071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50dba00640 1 -- 172.21.15.67:0/3361650800 shutdown_connections 2024-04-17T10:33:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.454+0000 7f50dba00640 1 -- 172.21.15.67:0/3361650800 wait complete. 2024-04-17T10:33:00.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 Processor -- start 2024-04-17T10:33:00.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 -- start start 2024-04-17T10:33:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d40728c0 0x7f50d415a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d40b5300 0x7f50d41681f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d41583f0 0x7f50d415d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d416bb00 con 0x7f50d40b5300 2024-04-17T10:33:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40b4200 con 0x7f50d40728c0 2024-04-17T10:33:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d416bc80 con 0x7f50d41583f0 2024-04-17T10:33:00.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d40b5300 0x7f50d41681f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d40728c0 0x7f50d415a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d40b5300 0x7f50d41681f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41702/0 (socket says 172.21.15.67:41702) 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 -- 172.21.15.67:0/3332795737 learned_addr learned my addr 172.21.15.67:0/3332795737 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d9a00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d41583f0 0x7f50d415d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 -- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d41583f0 msgr2=0x7f50d415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d41583f0 0x7f50d415d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 -- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d40728c0 msgr2=0x7f50d415a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d40728c0 0x7f50d415a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 -- 172.21.15.67:0/3332795737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d415af30 con 0x7f50d40b5300 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d9000640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d40728c0 0x7f50d415a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d3e00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d40b5300 0x7f50d41681f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f50c000c500 tx=0x7f50c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d0c00640 1 -- 172.21.15.67:0/3332795737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c0017070 con 0x7f50d40b5300 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d0c00640 1 -- 172.21.15.67:0/3332795737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c0005cf0 con 0x7f50d40b5300 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d415b210 con 0x7f50d40b5300 2024-04-17T10:33:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50d0c00640 1 -- 172.21.15.67:0/3332795737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c00072f0 con 0x7f50d40b5300 2024-04-17T10:33:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.458+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d415cec0 con 0x7f50d40b5300 2024-04-17T10:33:00.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d0c00640 1 -- 172.21.15.67:0/3332795737 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f50c001a070 con 0x7f50d40b5300 2024-04-17T10:33:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d0c00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f50b4066f60 0x7f50b4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d0c00640 1 -- 172.21.15.67:0/3332795737 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50c00844e0 con 0x7f50d40b5300 2024-04-17T10:33:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 Processor -- start 2024-04-17T10:33:00.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9000640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f50b4066f60 0x7f50b4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 -- start start 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9000640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f50b4066f60 0x7f50b4069420 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f50c80ae000 tx=0x7f50c8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f2491bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f249171b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491720a0 0x55f249172480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f248fbeff0 con 0x55f2491720a0 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f248f47150 con 0x55f2491bb9d0 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f248fbb090 con 0x55f2491bc2f0 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491720a0 0x55f249172480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491720a0 0x55f249172480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41704/0 (socket says 172.21.15.67:41704) 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 -- 172.21.15.67:0/163189560 learned_addr learned my addr 172.21.15.67:0/163189560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9000640 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f2491bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d3e00640 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f249171b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 -- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 msgr2=0x55f249171b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f249171b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 -- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 msgr2=0x55f2491bbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f2491bbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.462+0000 7f50d9a00640 1 -- 172.21.15.67:0/163189560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2491550d0 con 0x55f2491720a0 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491720a0 0x55f249172480 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f50cc001f50 tx=0x7f50cc0016b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d06e6d82c8408 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50c5600640 1 -- 172.21.15.67:0/163189560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc00fe50 con 0x55f2491720a0 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50c5600640 1 -- 172.21.15.67:0/163189560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc018070 con 0x55f2491720a0 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50c5600640 1 -- 172.21.15.67:0/163189560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc003b80 con 0x55f2491720a0 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- 172.21.15.67:0/163189560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491720a0 msgr2=0x55f249172480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491720a0 0x55f249172480 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f50cc001f50 tx=0x7f50cc0016b0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- 172.21.15.67:0/163189560 shutdown_connections 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491720a0 0x55f249172480 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f249171b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 --2- 172.21.15.67:0/163189560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f2491bbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- 172.21.15.67:0/163189560 >> 172.21.15.67:0/163189560 conn(0x55f24916e020 msgr2=0x55f249170440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- 172.21.15.67:0/163189560 shutdown_connections 2024-04-17T10:33:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- 172.21.15.67:0/163189560 wait complete. 2024-04-17T10:33:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- start start 2024-04-17T10:33:00.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 Processor -- start 2024-04-17T10:33:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bb9d0 0x55f2491a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f2491a1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f248f83250 con 0x55f2491bc2f0 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f248f79690 con 0x55f2491720a0 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f248f47a40 con 0x55f2491bb9d0 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50def50000 10 client.0 _ll_register_callbacks cb 0x55f24915ce90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f2491a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bb9d0 0x55f2491a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f2491a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41706/0 (socket says 172.21.15.67:41706) 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491a07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39616/0 (socket says 172.21.15.67:39616) 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 -- 172.21.15.67:0/1612775607 learned_addr learned my addr 172.21.15.67:0/1612775607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 -- 172.21.15.67:0/1612775607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bb9d0 msgr2=0x55f2491a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bb9d0 0x55f2491a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 -- 172.21.15.67:0/1612775607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 msgr2=0x55f2491a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 -- 172.21.15.67:0/1612775607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f24906c1c0 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.466+0000 7f50d9a00640 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f2491a1270 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f50cc000f90 tx=0x7f50cc001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18af8190f23d93e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 1 -- 172.21.15.67:0/1612775607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0054e0 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 1 -- 172.21.15.67:0/1612775607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50cc005680 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50cc018070 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 1 -- 172.21.15.67:0/1612775607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50cc02e050 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 10 client.8334 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55f249068170 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 1 -- 172.21.15.67:0/1612775607 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc018070 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 1 -- 172.21.15.67:0/1612775607 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f50cc031020 con 0x55f2491bc2f0 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 10 client.8334 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 10 client.8334 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 20 client.8334 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cbc80 0x55f2491ce070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 4 client.8334 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f248f5ac60 con 0x55f2491cbc80 2024-04-17T10:33:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50d9000640 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cbc80 0x55f2491ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50d9000640 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cbc80 0x55f2491ce070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f238bd3c8d407395 server_cookie=d8cb47a73070dbc9 in_seq=0 out_seq=0 2024-04-17T10:33:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 10 client.8334 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 1 -- 172.21.15.67:0/1612775607 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f248f5ac60 con 0x55f2491cbc80 2024-04-17T10:33:00.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 10 client.8334 handle_command_reply: tid=0 2024-04-17T10:33:00.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 client.8334 shutdown 2024-04-17T10:33:00.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 msgr2=0x55f2491a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f2491a1270 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f50cc000f90 tx=0x7f50cc001260 comp rx=0 tx=0).stop 2024-04-17T10:33:00.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 shutdown_connections 2024-04-17T10:33:00.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cbc80 0x55f2491ce070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f2491a1270 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bb9d0 0x55f2491a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 --2- 172.21.15.67:0/1612775607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50ba000640 0 client.8334 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 >> 172.21.15.67:0/1612775607 conn(0x55f24916e020 msgr2=0x55f249170ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 shutdown_connections 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 1 -- 172.21.15.67:0/1612775607 wait complete. 2024-04-17T10:33:00.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.470+0000 7f50def50000 20 client.8334 trim_cache size 0 max 16384 2024-04-17T10:33:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 Processor -- start 2024-04-17T10:33:00.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 -- start start 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491720a0 0x55f2492180f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f249218630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f249218b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d4203530 con 0x55f2491bc2f0 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d4202730 con 0x55f2491bb9d0 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d41fae50 con 0x55f2491720a0 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f249218b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f249218b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41712/0 (socket says 172.21.15.67:41712) 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 -- 172.21.15.67:0/4007823041 learned_addr learned my addr 172.21.15.67:0/4007823041 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9000640 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491720a0 0x55f2492180f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d3e00640 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f249218630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 -- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491720a0 msgr2=0x55f2492180f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491720a0 0x55f2492180f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 -- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 msgr2=0x55f249218630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f249218630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 -- 172.21.15.67:0/4007823041 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f249005de0 con 0x55f2491bc2f0 2024-04-17T10:33:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.486+0000 7f50d9a00640 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f249218b70 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f50cc001f50 tx=0x7f50cc01f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ccc4c401e13cdcf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50c4c00640 1 -- 172.21.15.67:0/4007823041 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc037070 con 0x55f2491bc2f0 2024-04-17T10:33:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50c4c00640 1 -- 172.21.15.67:0/4007823041 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc018040 con 0x55f2491bc2f0 2024-04-17T10:33:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50c4c00640 1 -- 172.21.15.67:0/4007823041 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc0052e0 con 0x55f2491bc2f0 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- 172.21.15.67:0/4007823041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 msgr2=0x55f249218b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f249218b70 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f50cc001f50 tx=0x7f50cc01f070 comp rx=0 tx=0).stop 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- 172.21.15.67:0/4007823041 shutdown_connections 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bc2f0 0x55f249218b70 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491bb9d0 0x55f249218630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 --2- 172.21.15.67:0/4007823041 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491720a0 0x55f2492180f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- 172.21.15.67:0/4007823041 >> 172.21.15.67:0/4007823041 conn(0x55f24916e020 msgr2=0x55f2491c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- 172.21.15.67:0/4007823041 shutdown_connections 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- 172.21.15.67:0/4007823041 wait complete. 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- start start 2024-04-17T10:33:00.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 Processor -- start 2024-04-17T10:33:00.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bb9d0 0x55f2491ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f2491cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f248f59e00 con 0x55f2491bb9d0 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d4156420 con 0x55f2491720a0 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f248fbe300 con 0x55f2491bc2f0 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 10 client.0 _ll_register_callbacks cb 0x55f24913d040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bb9d0 0x55f2491ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bb9d0 0x55f2491ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41718/0 (socket says 172.21.15.67:41718) 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 -- 172.21.15.67:0/2290733350 learned_addr learned my addr 172.21.15.67:0/2290733350 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d9000640 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d9a00640 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f2491cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 -- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 msgr2=0x55f2491cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f2491cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 -- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 msgr2=0x55f2491ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 -- 172.21.15.67:0/2290733350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f248f608e0 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50d3e00640 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bb9d0 0x55f2491ce8c0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f50c0054a00 tx=0x7f50c0051020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855201f580b785c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50ba000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50ba000640 1 -- 172.21.15.67:0/2290733350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c00530c0 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50ba000640 1 -- 172.21.15.67:0/2290733350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f50c0055170 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.490+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50cc018040 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50ba000640 1 -- 172.21.15.67:0/2290733350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50c005ced0 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 10 client.8340 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x55f2490188f0 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50ba000640 1 -- 172.21.15.67:0/2290733350 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc018040 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50ba000640 1 -- 172.21.15.67:0/2290733350 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f50c0007490 con 0x55f2491bb9d0 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 10 client.8340 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:00.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 10 client.8340 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 20 client.8340 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cfa40 0x55f249223f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 4 client.8340 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f248fcc530 con 0x55f2491cfa40 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50d9000640 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cfa40 0x55f249223f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50d9000640 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cfa40 0x55f249223f60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5c8f690024650fe server_cookie=9e5e63bf9456837e in_seq=0 out_seq=0 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50ba000640 10 client.8340 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50ba000640 1 -- 172.21.15.67:0/2290733350 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x55f248fcc530 con 0x55f2491cfa40 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50ba000640 10 client.8340 handle_command_reply: tid=0 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 client.8340 shutdown 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bb9d0 msgr2=0x55f2491ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bb9d0 0x55f2491ce8c0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f50c0054a00 tx=0x7f50c0051020 comp rx=0 tx=0).stop 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 shutdown_connections 2024-04-17T10:33:00.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x55f2491cfa40 0x55f249223f60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55f2491bc2f0 0x55f2491cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55f2491bb9d0 0x55f2491ce8c0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 --2- 172.21.15.67:0/2290733350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55f2491720a0 0x55f2491ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50ba000640 0 client.8340 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 >> 172.21.15.67:0/2290733350 conn(0x55f24916e020 msgr2=0x55f2491ba0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 shutdown_connections 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 1 -- 172.21.15.67:0/2290733350 wait complete. 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50def50000 20 client.8340 trim_cache size 0 max 16384 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:00.489 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout: "85eff2e3-62e3-496b-a911-123cb3de8df8": { 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_control_test_path", 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "85eff2e3-62e3-496b-a911-123cb3de8df8", 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:00.490 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f50b4066f60 msgr2=0x7f50b4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f50b4066f60 0x7f50b4069420 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f50c80ae000 tx=0x7f50c8008040 comp rx=0 tx=0).stop 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d40b5300 msgr2=0x7f50d41681f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d40b5300 0x7f50d41681f0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f50c000c500 tx=0x7f50c000c9c0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 shutdown_connections 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f50b4066f60 0x7f50b4069420 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f50d41583f0 0x7f50d415d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50d40b5300 0x7f50d41681f0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 --2- 172.21.15.67:0/3332795737 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f50d40728c0 0x7f50d415a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 >> 172.21.15.67:0/3332795737 conn(0x7f50d40b2570 msgr2=0x7f50d406ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 shutdown_connections 2024-04-17T10:33:00.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.494+0000 7f50dba00640 1 -- 172.21.15.67:0/3332795737 wait complete. 2024-04-17T10:33:00.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:00.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78fe800640 1 Processor -- start 2024-04-17T10:33:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78fe800640 1 -- start start 2024-04-17T10:33:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78fe800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f8145040 0x7f78f806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78fe800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f78f806ef10 0x7f78f8071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78fe800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f806e2f0 con 0x7f78f8071850 2024-04-17T10:33:00.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78fe800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78f80a86c0 con 0x7f78f806ef10 2024-04-17T10:33:00.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78fe800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f815bf10 con 0x7f78f8145040 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78f7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f8145040 0x7f78f806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78eea00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f78f806ef10 0x7f78f8071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78f7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f8145040 0x7f78f806e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:46502/0 (socket says 172.21.15.67:46502) 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78f7400640 1 -- 172.21.15.67:0/1056690589 learned_addr learned my addr 172.21.15.67:0/1056690589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78f7e00640 1 --1- 172.21.15.67:0/1056690589 >> v1:172.21.15.86:6789/0 conn(0x7f78f8071850 0x7f78f815f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:51054/0 (socket says 172.21.15.67:51054) 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78f7400640 1 -- 172.21.15.67:0/1056690589 >> v2:172.21.15.86:3300/0 conn(0x7f78f806ef10 msgr2=0x7f78f8071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.654+0000 7f78f7400640 1 --2- 172.21.15.67:0/1056690589 >> v2:172.21.15.86:3300/0 conn(0x7f78f806ef10 0x7f78f8071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78f7400640 1 -- 172.21.15.67:0/1056690589 >> v1:172.21.15.86:6789/0 conn(0x7f78f8071850 legacy=0x7f78f815f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78f7400640 1 -- 172.21.15.67:0/1056690589 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f806d6e0 con 0x7f78f8145040 2024-04-17T10:33:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78f6a00640 1 -- 172.21.15.67:0/1056690589 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555117883 0 0) 0x7f78f806e2f0 con 0x7f78f8071850 2024-04-17T10:33:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78eea00640 1 --2- 172.21.15.67:0/1056690589 >> v2:172.21.15.86:3300/0 conn(0x7f78f806ef10 0x7f78f8071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78f7400640 1 --2- 172.21.15.67:0/1056690589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f8145040 0x7f78f806e9d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009870 tx=0x7f78e801ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a364d4ecd41d46a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78f6a00640 1 -- 172.21.15.67:0/1056690589 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e80b5070 con 0x7f78f8145040 2024-04-17T10:33:00.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78f6a00640 1 -- 172.21.15.67:0/1056690589 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e80a8a50 con 0x7f78f8145040 2024-04-17T10:33:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78f6a00640 1 -- 172.21.15.67:0/1056690589 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e80a8e20 con 0x7f78f8145040 2024-04-17T10:33:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 -- 172.21.15.67:0/1056690589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f8145040 msgr2=0x7f78f806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 --2- 172.21.15.67:0/1056690589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f8145040 0x7f78f806e9d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f78e8009870 tx=0x7f78e801ed50 comp rx=0 tx=0).stop 2024-04-17T10:33:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 -- 172.21.15.67:0/1056690589 shutdown_connections 2024-04-17T10:33:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 --2- 172.21.15.67:0/1056690589 >> v2:172.21.15.86:3300/0 conn(0x7f78f806ef10 0x7f78f8071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 --2- 172.21.15.67:0/1056690589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f8145040 0x7f78f806e9d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 -- 172.21.15.67:0/1056690589 >> 172.21.15.67:0/1056690589 conn(0x7f78f806ac90 msgr2=0x7f78f806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 -- 172.21.15.67:0/1056690589 shutdown_connections 2024-04-17T10:33:00.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.658+0000 7f78fe800640 1 -- 172.21.15.67:0/1056690589 wait complete. 2024-04-17T10:33:00.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 Processor -- start 2024-04-17T10:33:00.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 -- start start 2024-04-17T10:33:00.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f806ef10 0x7f78f80ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f8071850 0x7f78f815adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78f8145040 0x7f78f8142d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f806dca0 con 0x7f78f8145040 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f8071ca0 con 0x7f78f8071850 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f806df80 con 0x7f78f806ef10 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f806ef10 0x7f78f80ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78f8145040 0x7f78f8142d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f806ef10 0x7f78f80ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:46510/0 (socket says 172.21.15.67:46510) 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 -- 172.21.15.67:0/1231142843 learned_addr learned my addr 172.21.15.67:0/1231142843 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78eea00640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f8071850 0x7f78f815adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 -- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f8071850 msgr2=0x7f78f815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f8071850 0x7f78f815adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 -- 172.21.15.67:0/1231142843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78f8145040 msgr2=0x7f78f8142d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78f8145040 0x7f78f8142d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 -- 172.21.15.67:0/1231142843 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f80b2ac0 con 0x7f78f806ef10 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78f7400640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f806ef10 0x7f78f80ae2d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f78e80b3040 tx=0x7f78e801baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78efe00640 1 -- 172.21.15.67:0/1231142843 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e80b5070 con 0x7f78f806ef10 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78efe00640 1 -- 172.21.15.67:0/1231142843 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78e80b0de0 con 0x7f78f806ef10 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78efe00640 1 -- 172.21.15.67:0/1231142843 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78e80bb670 con 0x7f78f806ef10 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.662+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f80b2da0 con 0x7f78f806ef10 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.666+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f80b3360 con 0x7f78f806ef10 2024-04-17T10:33:00.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.666+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f8072d50 con 0x7f78f806ef10 2024-04-17T10:33:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.666+0000 7f78efe00640 1 -- 172.21.15.67:0/1231142843 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f78e80c2060 con 0x7f78f806ef10 2024-04-17T10:33:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.666+0000 7f78efe00640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78d4066f60 0x7f78d4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.666+0000 7f78efe00640 1 -- 172.21.15.67:0/1231142843 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f78e8122510 con 0x7f78f806ef10 2024-04-17T10:33:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.666+0000 7f78eea00640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78d4066f60 0x7f78d4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:00.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.666+0000 7f78eea00640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78d4066f60 0x7f78d4069420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009580 tx=0x7f78e0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:00.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.670+0000 7f78efe00640 1 -- 172.21.15.67:0/1231142843 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f78e80188b0 con 0x7f78f806ef10 2024-04-17T10:33:00.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78f8074e30 con 0x7f78f806ef10 2024-04-17T10:33:00.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78efe00640 1 -- 172.21.15.67:0/1231142843 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f78e80f2ca0 con 0x7f78f806ef10 2024-04-17T10:33:00.857 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:00.857 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:00.858 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78d4066f60 msgr2=0x7f78d4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78d4066f60 0x7f78d4069420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f78e0009580 tx=0x7f78e0009190 comp rx=0 tx=0).stop 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f806ef10 msgr2=0x7f78f80ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f806ef10 0x7f78f80ae2d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f78e80b3040 tx=0x7f78e801baf0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 shutdown_connections 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f78d4066f60 0x7f78d4069420 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f78f8145040 0x7f78f8142d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f78f8071850 0x7f78f815adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 --2- 172.21.15.67:0/1231142843 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f78f806ef10 0x7f78f80ae2d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:00.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 >> 172.21.15.67:0/1231142843 conn(0x7f78f806ac90 msgr2=0x7f78f80b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:00.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 shutdown_connections 2024-04-17T10:33:00.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:00.866+0000 7f78fe800640 1 -- 172.21.15.67:0/1231142843 wait complete. 2024-04-17T10:33:00.878 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:33:00.878 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:01.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.026+0000 7f29c9000640 1 Processor -- start 2024-04-17T10:33:01.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.026+0000 7f29c9000640 1 -- start start 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.026+0000 7f29c9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29c40af670 0x7f29c4162010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.026+0000 7f29c9000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40ce130 0x7f29c41412a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c9000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c4162550 con 0x7f29c406bfd0 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c9000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c406ac90 con 0x7f29c40af670 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c9000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c406af30 con 0x7f29c40ce130 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29c40af670 0x7f29c4162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f29c40af670 0x7f29c4162010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41748/0 (socket says 172.21.15.67:41748) 2024-04-17T10:33:01.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 -- 172.21.15.67:0/2932474824 learned_addr learned my addr 172.21.15.67:0/2932474824 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:01.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c2800640 1 --2- 172.21.15.67:0/2932474824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40ce130 0x7f29c41412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:01.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 -- 172.21.15.67:0/2932474824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40ce130 msgr2=0x7f29c41412a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:01.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 --2- 172.21.15.67:0/2932474824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40ce130 0x7f29c41412a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 -- 172.21.15.67:0/2932474824 >> v1:172.21.15.86:6789/0 conn(0x7f29c406bfd0 legacy=0x7f29c40cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:01.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 -- 172.21.15.67:0/2932474824 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c406b210 con 0x7f29c40af670 2024-04-17T10:33:01.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1e00640 1 --2- 172.21.15.67:0/2932474824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c40af670 0x7f29c4162010 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f29b40098a0 tx=0x7f29b401c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4797f8abf62f35c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:01.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1400640 1 -- 172.21.15.67:0/2932474824 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b40b0070 con 0x7f29c40af670 2024-04-17T10:33:01.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1400640 1 -- 172.21.15.67:0/2932474824 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b40ab600 con 0x7f29c40af670 2024-04-17T10:33:01.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.030+0000 7f29c1400640 1 -- 172.21.15.67:0/2932474824 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b40ab9d0 con 0x7f29c40af670 2024-04-17T10:33:01.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- 172.21.15.67:0/2932474824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c40af670 msgr2=0x7f29c4162010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:01.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 --2- 172.21.15.67:0/2932474824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c40af670 0x7f29c4162010 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f29b40098a0 tx=0x7f29b401c2f0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- 172.21.15.67:0/2932474824 shutdown_connections 2024-04-17T10:33:01.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 --2- 172.21.15.67:0/2932474824 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40ce130 0x7f29c41412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 --2- 172.21.15.67:0/2932474824 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c40af670 0x7f29c4162010 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- 172.21.15.67:0/2932474824 >> 172.21.15.67:0/2932474824 conn(0x7f29c40cc8d0 msgr2=0x7f29c40ccce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:01.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- 172.21.15.67:0/2932474824 shutdown_connections 2024-04-17T10:33:01.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- 172.21.15.67:0/2932474824 wait complete. 2024-04-17T10:33:01.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 Processor -- start 2024-04-17T10:33:01.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- start start 2024-04-17T10:33:01.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c406bfd0 0x7f29c413a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:01.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40af670 0x7f29c40cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:01.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c40ce130 0x7f29c413cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:01.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4141d70 con 0x7f29c406bfd0 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4141bf0 con 0x7f29c40ce130 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c9000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c41473f0 con 0x7f29c40af670 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c406bfd0 0x7f29c413a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c406bfd0 0x7f29c413a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41752/0 (socket says 172.21.15.67:41752) 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c1e00640 1 -- 172.21.15.67:0/3567593411 learned_addr learned my addr 172.21.15.67:0/3567593411 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29bbe00640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40af670 0x7f29c40cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c2800640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c40ce130 0x7f29c413cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c1e00640 1 -- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40af670 msgr2=0x7f29c40cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c1e00640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40af670 0x7f29c40cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c1e00640 1 -- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c40ce130 msgr2=0x7f29c413cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.034+0000 7f29c1e00640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c40ce130 0x7f29c413cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29c1e00640 1 -- 172.21.15.67:0/3567593411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40b14f0 con 0x7f29c406bfd0 2024-04-17T10:33:01.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29c1e00640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c406bfd0 0x7f29c413a6e0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f29b401cb90 tx=0x7f29b4019090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:01.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29ba000640 1 -- 172.21.15.67:0/3567593411 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b40ba070 con 0x7f29c406bfd0 2024-04-17T10:33:01.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29ba000640 1 -- 172.21.15.67:0/3567593411 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b40b4640 con 0x7f29c406bfd0 2024-04-17T10:33:01.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c413ade0 con 0x7f29c406bfd0 2024-04-17T10:33:01.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29ba000640 1 -- 172.21.15.67:0/3567593411 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b40b0040 con 0x7f29c406bfd0 2024-04-17T10:33:01.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c413b3a0 con 0x7f29c406bfd0 2024-04-17T10:33:01.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c409e180 con 0x7f29c406bfd0 2024-04-17T10:33:01.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29ba000640 1 -- 172.21.15.67:0/3567593411 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f29b40bd070 con 0x7f29c406bfd0 2024-04-17T10:33:01.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29ba000640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a0066f90 0x7f29a0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:01.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29ba000640 1 -- 172.21.15.67:0/3567593411 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f29b411e1b0 con 0x7f29c406bfd0 2024-04-17T10:33:01.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.038+0000 7f29bbe00640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a0066f90 0x7f29a0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:01.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.042+0000 7f29bbe00640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a0066f90 0x7f29a0069450 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f29c406f550 tx=0x7f29ac009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:01.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.042+0000 7f29ba000640 1 -- 172.21.15.67:0/3567593411 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f29c4162550 con 0x7f29c406bfd0 2024-04-17T10:33:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f29c4074e30 con 0x7f29c406bfd0 2024-04-17T10:33:01.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29ba000640 1 -- 172.21.15.67:0/3567593411 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1444 (secure 0 0 0) 0x7f29b40ee940 con 0x7f29c406bfd0 2024-04-17T10:33:01.262 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:01.262 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":512,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":907,"data_bytes":928066852,"bytes_used":2222022656,"bytes_avail":770872090624,"bytes_total":773094113280,"write_bytes_sec":47982327,"read_op_per_sec":0,"write_op_per_sec":47},"fsmap":{"epoch":73,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8139}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-17T10:32:22.243227+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:01.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a0066f90 msgr2=0x7f29a0069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:01.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a0066f90 0x7f29a0069450 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f29c406f550 tx=0x7f29ac009060 comp rx=0 tx=0).stop 2024-04-17T10:33:01.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c406bfd0 msgr2=0x7f29c413a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:01.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c406bfd0 0x7f29c413a6e0 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f29b401cb90 tx=0x7f29b4019090 comp rx=0 tx=0).stop 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 shutdown_connections 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f29a0066f90 0x7f29a0069450 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29c40ce130 0x7f29c413cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f29c40af670 0x7f29c40cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.270+0000 7f29c9000640 1 --2- 172.21.15.67:0/3567593411 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29c406bfd0 0x7f29c413a6e0 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.274+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 >> 172.21.15.67:0/3567593411 conn(0x7f29c40cc8d0 msgr2=0x7f29c406dc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.274+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 shutdown_connections 2024-04-17T10:33:01.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:01.274+0000 7f29c9000640 1 -- 172.21.15.67:0/3567593411 wait complete. 2024-04-17T10:33:01.283 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:33:02.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:02.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e3400640 1 Processor -- start 2024-04-17T10:33:02.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e3400640 1 -- start start 2024-04-17T10:33:02.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75dc06e6e0 0x7f75dc0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e3400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc15ae40 con 0x7f75dc1441f0 2024-04-17T10:33:02.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e3400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc071440 con 0x7f75dc06eac0 2024-04-17T10:33:02.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e3400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc071720 con 0x7f75dc06e6e0 2024-04-17T10:33:02.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f75dc06e6e0 0x7f75dc0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e1400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75dc1441f0 0x7f75dc0b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44718/0 (socket says 172.21.15.67:44718) 2024-04-17T10:33:02.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.446+0000 7f75e1400640 1 -- 172.21.15.67:0/668919964 learned_addr learned my addr 172.21.15.67:0/668919964 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:02.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75db400640 1 -- 172.21.15.67:0/668919964 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109141700 0 0) 0x7f75dc15ae40 con 0x7f75dc1441f0 2024-04-17T10:33:02.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75db400640 1 -- 172.21.15.67:0/668919964 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c4003530 con 0x7f75dc1441f0 2024-04-17T10:33:02.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75e0a00640 1 -- 172.21.15.67:0/668919964 >> v1:172.21.15.86:6789/0 conn(0x7f75dc06eac0 legacy=0x7f75dc143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75e0a00640 1 -- 172.21.15.67:0/668919964 >> v1:172.21.15.67:6789/0 conn(0x7f75dc1441f0 legacy=0x7f75dc0b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75e0a00640 1 -- 172.21.15.67:0/668919964 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc071a00 con 0x7f75dc06e6e0 2024-04-17T10:33:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75e0a00640 1 --2- 172.21.15.67:0/668919964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc06e6e0 0x7f75dc0ce9b0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f75d00097c0 tx=0x7f75d001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f305902018a1bd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75db400640 1 -- 172.21.15.67:0/668919964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00b0070 con 0x7f75dc06e6e0 2024-04-17T10:33:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75db400640 1 -- 172.21.15.67:0/668919964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d00ab600 con 0x7f75dc06e6e0 2024-04-17T10:33:02.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.450+0000 7f75db400640 1 -- 172.21.15.67:0/668919964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d00ab9b0 con 0x7f75dc06e6e0 2024-04-17T10:33:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- 172.21.15.67:0/668919964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc06e6e0 msgr2=0x7f75dc0ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 --2- 172.21.15.67:0/668919964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc06e6e0 0x7f75dc0ce9b0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f75d00097c0 tx=0x7f75d001c3e0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- 172.21.15.67:0/668919964 shutdown_connections 2024-04-17T10:33:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 --2- 172.21.15.67:0/668919964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc06e6e0 0x7f75dc0ce9b0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- 172.21.15.67:0/668919964 >> 172.21.15.67:0/668919964 conn(0x7f75dc06ac90 msgr2=0x7f75dc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- 172.21.15.67:0/668919964 shutdown_connections 2024-04-17T10:33:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- 172.21.15.67:0/668919964 wait complete. 2024-04-17T10:33:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 Processor -- start 2024-04-17T10:33:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- start start 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc06e6e0 0x7f75dc142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc06eac0 0x7f75dc1f8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc1441f0 0x7f75dc1f8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc06d4f0 con 0x7f75dc1441f0 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc06d370 con 0x7f75dc06e6e0 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e3400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc06d670 con 0x7f75dc06eac0 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc1441f0 0x7f75dc1f8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc1441f0 0x7f75dc1f8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41756/0 (socket says 172.21.15.67:41756) 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 -- 172.21.15.67:0/1356506963 learned_addr learned my addr 172.21.15.67:0/1356506963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e0a00640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc06e6e0 0x7f75dc142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75dbe00640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc06eac0 0x7f75dc1f8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 -- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc06eac0 msgr2=0x7f75dc1f8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc06eac0 0x7f75dc1f8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 -- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc06e6e0 msgr2=0x7f75dc142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc06e6e0 0x7f75dc142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 -- 172.21.15.67:0/1356506963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc13c980 con 0x7f75dc1441f0 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75dbe00640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc06eac0 0x7f75dc1f8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.454+0000 7f75e1400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc1441f0 0x7f75dc1f8b70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f75d400caa0 tx=0x7f75d400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75d8c00640 1 -- 172.21.15.67:0/1356506963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4017070 con 0x7f75dc1441f0 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75d8c00640 1 -- 172.21.15.67:0/1356506963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d400ac00 con 0x7f75dc1441f0 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75d8c00640 1 -- 172.21.15.67:0/1356506963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d4012400 con 0x7f75dc1441f0 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc142bf0 con 0x7f75dc1441f0 2024-04-17T10:33:02.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc1431b0 con 0x7f75dc1441f0 2024-04-17T10:33:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc072d50 con 0x7f75dc1441f0 2024-04-17T10:33:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75d8c00640 1 -- 172.21.15.67:0/1356506963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f75d401a070 con 0x7f75dc1441f0 2024-04-17T10:33:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75d8c00640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75bc066f60 0x7f75bc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75d8c00640 1 -- 172.21.15.67:0/1356506963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f75d4084f60 con 0x7f75dc1441f0 2024-04-17T10:33:02.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.458+0000 7f75e0a00640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75bc066f60 0x7f75bc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.462+0000 7f75e0a00640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75bc066f60 0x7f75bc069420 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f75d00ad000 tx=0x7f75d00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:02.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.462+0000 7f75d8c00640 1 -- 172.21.15.67:0/1356506963 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f75d40556f0 con 0x7f75dc1441f0 2024-04-17T10:33:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.658+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75dc074e30 con 0x7f75dc1441f0 2024-04-17T10:33:02.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.658+0000 7f75d8c00640 1 -- 172.21.15.67:0/1356506963 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f75d4059790 con 0x7f75dc1441f0 2024-04-17T10:33:02.650 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:02.650 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:02.651 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75bc066f60 msgr2=0x7f75bc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75bc066f60 0x7f75bc069420 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f75d00ad000 tx=0x7f75d00a6000 comp rx=0 tx=0).stop 2024-04-17T10:33:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc1441f0 msgr2=0x7f75dc1f8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc1441f0 0x7f75dc1f8b70 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f75d400caa0 tx=0x7f75d400cf60 comp rx=0 tx=0).stop 2024-04-17T10:33:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 shutdown_connections 2024-04-17T10:33:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f75bc066f60 0x7f75bc069420 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f75dc1441f0 0x7f75dc1f8b70 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75dc06eac0 0x7f75dc1f8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 --2- 172.21.15.67:0/1356506963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f75dc06e6e0 0x7f75dc142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 >> 172.21.15.67:0/1356506963 conn(0x7f75dc06ac90 msgr2=0x7f75dc0b1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 shutdown_connections 2024-04-17T10:33:02.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.662+0000 7f75e3400640 1 -- 172.21.15.67:0/1356506963 wait complete. 2024-04-17T10:33:02.671 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:33:02.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:02.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02d6600640 1 Processor -- start 2024-04-17T10:33:02.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02d6600640 1 -- start start 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02d6600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02d00b07b0 0x7f02d0070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02d6600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d0071300 0x7f02d006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02d6600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d00b5980 con 0x7f02d015dd10 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02d6600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d00b1280 con 0x7f02d00b07b0 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02d6600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d00b0cc0 con 0x7f02d0071300 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02c6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02d00b07b0 0x7f02d0070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02c6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f02d00b07b0 0x7f02d0070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41766/0 (socket says 172.21.15.67:41766) 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02c6a00640 1 -- 172.21.15.67:0/124724977 learned_addr learned my addr 172.21.15.67:0/124724977 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02cfe00640 1 --2- 172.21.15.67:0/124724977 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d0071300 0x7f02d006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02c6a00640 1 -- 172.21.15.67:0/124724977 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d0071300 msgr2=0x7f02d006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:02.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02c6a00640 1 --2- 172.21.15.67:0/124724977 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d0071300 0x7f02d006c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02c6a00640 1 -- 172.21.15.67:0/124724977 >> v1:172.21.15.86:6789/0 conn(0x7f02d015dd10 legacy=0x7f02d00b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.822+0000 7f02c6a00640 1 -- 172.21.15.67:0/124724977 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00b0fa0 con 0x7f02d00b07b0 2024-04-17T10:33:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02c6a00640 1 --2- 172.21.15.67:0/124724977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d00b07b0 0x7f02d0070dc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009870 tx=0x7f02b801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9240028f098f38 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02cea00640 1 -- 172.21.15.67:0/124724977 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80b1070 con 0x7f02d00b07b0 2024-04-17T10:33:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02cea00640 1 -- 172.21.15.67:0/124724977 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b80ac620 con 0x7f02d00b07b0 2024-04-17T10:33:02.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02cea00640 1 -- 172.21.15.67:0/124724977 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b80ac9f0 con 0x7f02d00b07b0 2024-04-17T10:33:02.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 -- 172.21.15.67:0/124724977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d00b07b0 msgr2=0x7f02d0070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 --2- 172.21.15.67:0/124724977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d00b07b0 0x7f02d0070dc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009870 tx=0x7f02b801c4d0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 -- 172.21.15.67:0/124724977 shutdown_connections 2024-04-17T10:33:02.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 --2- 172.21.15.67:0/124724977 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d0071300 0x7f02d006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 --2- 172.21.15.67:0/124724977 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d00b07b0 0x7f02d0070dc0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 -- 172.21.15.67:0/124724977 >> 172.21.15.67:0/124724977 conn(0x7f02d00b34d0 msgr2=0x7f02d00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 -- 172.21.15.67:0/124724977 shutdown_connections 2024-04-17T10:33:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.826+0000 7f02d6600640 1 -- 172.21.15.67:0/124724977 wait complete. 2024-04-17T10:33:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 Processor -- start 2024-04-17T10:33:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 -- start start 2024-04-17T10:33:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d0071300 0x7f02d0155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d00b07b0 0x7f02d00cfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02d015dd10 0x7f02d0141270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d006d5a0 con 0x7f02d0071300 2024-04-17T10:33:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d006d420 con 0x7f02d015dd10 2024-04-17T10:33:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d006d720 con 0x7f02d00b07b0 2024-04-17T10:33:02.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d0071300 0x7f02d0155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d0071300 0x7f02d0155ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41782/0 (socket says 172.21.15.67:41782) 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 -- 172.21.15.67:0/1928915288 learned_addr learned my addr 172.21.15.67:0/1928915288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cfe00640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02d015dd10 0x7f02d0141270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02c6a00640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d00b07b0 0x7f02d00cfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 -- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d00b07b0 msgr2=0x7f02d00cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d00b07b0 0x7f02d00cfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 -- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02d015dd10 msgr2=0x7f02d0141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02d015dd10 0x7f02d0141270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 -- 172.21.15.67:0/1928915288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d00d0210 con 0x7f02d0071300 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02c6a00640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d00b07b0 0x7f02d00cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02cf400640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d0071300 0x7f02d0155ff0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c730 tx=0x7f02c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02c7e00640 1 -- 172.21.15.67:0/1928915288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0016020 con 0x7f02d0071300 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02c7e00640 1 -- 172.21.15.67:0/1928915288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c0006d60 con 0x7f02d0071300 2024-04-17T10:33:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02c7e00640 1 -- 172.21.15.67:0/1928915288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c0003bc0 con 0x7f02d0071300 2024-04-17T10:33:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d01417b0 con 0x7f02d0071300 2024-04-17T10:33:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d0156990 con 0x7f02d0071300 2024-04-17T10:33:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.830+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d00b6600 con 0x7f02d0071300 2024-04-17T10:33:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.834+0000 7f02c7e00640 1 -- 172.21.15.67:0/1928915288 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f02c000a850 con 0x7f02d0071300 2024-04-17T10:33:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.834+0000 7f02c7e00640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02ac066f60 0x7f02ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.834+0000 7f02c7e00640 1 -- 172.21.15.67:0/1928915288 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f02c0084dd0 con 0x7f02d0071300 2024-04-17T10:33:02.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.834+0000 7f02c6a00640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02ac066f60 0x7f02ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:02.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.834+0000 7f02c6a00640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02ac066f60 0x7f02ac069420 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f02b80ae000 tx=0x7f02b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:02.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:02.838+0000 7f02c7e00640 1 -- 172.21.15.67:0/1928915288 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f02c0020020 con 0x7f02d0071300 2024-04-17T10:33:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f02d0074e30 con 0x7f02d0071300 2024-04-17T10:33:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02c7e00640 1 -- 172.21.15.67:0/1928915288 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1446 (secure 0 0 0) 0x7f02d00d0210 con 0x7f02d0071300 2024-04-17T10:33:03.050 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:03.050 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":513,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076514771,"bytes_used":2583994368,"bytes_avail":770510118912,"bytes_total":773094113280,"write_bytes_sec":60353013,"read_op_per_sec":0,"write_op_per_sec":60},"fsmap":{"epoch":73,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8139}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-17T10:32:22.243227+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02ac066f60 msgr2=0x7f02ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02ac066f60 0x7f02ac069420 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f02b80ae000 tx=0x7f02b8008040 comp rx=0 tx=0).stop 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d0071300 msgr2=0x7f02d0155ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d0071300 0x7f02d0155ff0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f02c000c730 tx=0x7f02c000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 shutdown_connections 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f02ac066f60 0x7f02ac069420 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f02d015dd10 0x7f02d0141270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f02d00b07b0 0x7f02d00cfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 --2- 172.21.15.67:0/1928915288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f02d0071300 0x7f02d0155ff0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 >> 172.21.15.67:0/1928915288 conn(0x7f02d00b34d0 msgr2=0x7f02d0151660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 shutdown_connections 2024-04-17T10:33:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:03.058+0000 7f02d6600640 1 -- 172.21.15.67:0/1928915288 wait complete. 2024-04-17T10:33:03.070 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:33:04.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dd6400640 1 Processor -- start 2024-04-17T10:33:04.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dd6400640 1 -- start start 2024-04-17T10:33:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dd6400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd015f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dd6400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1dd015fa00 0x7f1dd00c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dd6400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd015be60 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dd6400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd0160230 con 0x7f1dd015fa00 2024-04-17T10:33:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dd6400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd015c120 con 0x7f1dd015cc00 2024-04-17T10:33:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dc6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd015f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dcfe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1dd015fa00 0x7f1dd00c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dc6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd015f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:46558/0 (socket says 172.21.15.67:46558) 2024-04-17T10:33:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dc6a00640 1 -- 172.21.15.67:0/3727563605 learned_addr learned my addr 172.21.15.67:0/3727563605 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.234+0000 7f1dcf400640 1 --1- 172.21.15.67:0/3727563605 >> v1:172.21.15.86:6789/0 conn(0x7f1dd00ce1b0 0x7f1dd00ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:51088/0 (socket says 172.21.15.67:51088) 2024-04-17T10:33:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dc6a00640 1 -- 172.21.15.67:0/3727563605 >> v2:172.21.15.86:3300/0 conn(0x7f1dd015fa00 msgr2=0x7f1dd00c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dc6a00640 1 --2- 172.21.15.67:0/3727563605 >> v2:172.21.15.86:3300/0 conn(0x7f1dd015fa00 0x7f1dd00c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dc6a00640 1 -- 172.21.15.67:0/3727563605 >> v1:172.21.15.86:6789/0 conn(0x7f1dd00ce1b0 legacy=0x7f1dd00ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dc6a00640 1 -- 172.21.15.67:0/3727563605 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd015c400 con 0x7f1dd015cc00 2024-04-17T10:33:04.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dcfe00640 1 --2- 172.21.15.67:0/3727563605 >> v2:172.21.15.86:3300/0 conn(0x7f1dd015fa00 0x7f1dd00c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:04.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dc6a00640 1 --2- 172.21.15.67:0/3727563605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd015f4c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009870 tx=0x7f1db801ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae95afc01036cdc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:04.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dcea00640 1 -- 172.21.15.67:0/3727563605 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80b5070 con 0x7f1dd015cc00 2024-04-17T10:33:04.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dcea00640 1 -- 172.21.15.67:0/3727563605 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80a8a50 con 0x7f1dd015cc00 2024-04-17T10:33:04.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dcea00640 1 -- 172.21.15.67:0/3727563605 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db80a8e20 con 0x7f1dd015cc00 2024-04-17T10:33:04.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 -- 172.21.15.67:0/3727563605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 msgr2=0x7f1dd015f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 --2- 172.21.15.67:0/3727563605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd015f4c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1db8009870 tx=0x7f1db801ed50 comp rx=0 tx=0).stop 2024-04-17T10:33:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 -- 172.21.15.67:0/3727563605 shutdown_connections 2024-04-17T10:33:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 --2- 172.21.15.67:0/3727563605 >> v2:172.21.15.86:3300/0 conn(0x7f1dd015fa00 0x7f1dd00c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 --2- 172.21.15.67:0/3727563605 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd015f4c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 -- 172.21.15.67:0/3727563605 >> 172.21.15.67:0/3727563605 conn(0x7f1dd00b33f0 msgr2=0x7f1dd00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 -- 172.21.15.67:0/3727563605 shutdown_connections 2024-04-17T10:33:04.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.238+0000 7f1dd6400640 1 -- 172.21.15.67:0/3727563605 wait complete. 2024-04-17T10:33:04.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 Processor -- start 2024-04-17T10:33:04.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 -- start start 2024-04-17T10:33:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dd00ce1b0 0x7f1dd01f7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dd015fa00 0x7f1dd006c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0160010 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd015fe90 con 0x7f1dd015fa00 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0159730 con 0x7f1dd015cc00 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dd00ce1b0 0x7f1dd01f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dd00ce1b0 0x7f1dd01f7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41784/0 (socket says 172.21.15.67:41784) 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 -- 172.21.15.67:0/263040574 learned_addr learned my addr 172.21.15.67:0/263040574 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dc6a00640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcfe00640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dd015fa00 0x7f1dd006c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 -- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 msgr2=0x7f1dd014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 -- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dd015fa00 msgr2=0x7f1dd006c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dd015fa00 0x7f1dd006c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 -- 172.21.15.67:0/263040574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd01f7860 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dcf400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dd00ce1b0 0x7f1dd01f7280 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000c370 tx=0x7f1dc000c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dc7e00640 1 -- 172.21.15.67:0/263040574 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc00074e0 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dc7e00640 1 -- 172.21.15.67:0/263040574 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc0007680 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd01f7b40 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dc7e00640 1 -- 172.21.15.67:0/263040574 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc0004a30 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.242+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd01f8100 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.246+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd00b6020 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.246+0000 7f1dc7e00640 1 -- 172.21.15.67:0/263040574 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1dc001a020 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.246+0000 7f1dc7e00640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dac066f60 0x7f1dac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.246+0000 7f1dc7e00640 1 -- 172.21.15.67:0/263040574 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1dc00848d0 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.246+0000 7f1dc6a00640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dac066f60 0x7f1dac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.246+0000 7f1dc6a00640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dac066f60 0x7f1dac069420 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1db80b2000 tx=0x7f1db8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:04.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.250+0000 7f1dc7e00640 1 -- 172.21.15.67:0/263040574 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1dd00b6020 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.450+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dd0070ab0 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.450+0000 7f1dc7e00640 1 -- 172.21.15.67:0/263040574 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f1dc00556b0 con 0x7f1dd00ce1b0 2024-04-17T10:33:04.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:04.443 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dac066f60 msgr2=0x7f1dac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dac066f60 0x7f1dac069420 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1db80b2000 tx=0x7f1db8009040 comp rx=0 tx=0).stop 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dd00ce1b0 msgr2=0x7f1dd01f7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dd00ce1b0 0x7f1dd01f7280 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f1dc000c370 tx=0x7f1dc000c770 comp rx=0 tx=0).stop 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 shutdown_connections 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dac066f60 0x7f1dac069420 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dd015fa00 0x7f1dd006c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dd015cc00 0x7f1dd014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 --2- 172.21.15.67:0/263040574 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dd00ce1b0 0x7f1dd01f7280 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 >> 172.21.15.67:0/263040574 conn(0x7f1dd00b33f0 msgr2=0x7f1dd006d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 shutdown_connections 2024-04-17T10:33:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.454+0000 7f1dd6400640 1 -- 172.21.15.67:0/263040574 wait complete. 2024-04-17T10:33:04.463 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:33:04.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:04.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.610+0000 7f3896e00640 1 Processor -- start 2024-04-17T10:33:04.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3896e00640 1 -- start start 2024-04-17T10:33:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3896e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f389015dd10 0x7f38900b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3896e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38900b57b0 0x7f3890070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3896e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900b0cc0 con 0x7f38900b07b0 2024-04-17T10:33:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3896e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38900b1280 con 0x7f389015dd10 2024-04-17T10:33:04.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3896e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900b0fa0 con 0x7f38900b57b0 2024-04-17T10:33:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3887400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38900b57b0 0x7f3890070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3894e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f38900b07b0 0x7f38901618c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:51094/0 (socket says 172.21.15.67:51094) 2024-04-17T10:33:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3894e00640 1 -- 172.21.15.67:0/911460387 learned_addr learned my addr 172.21.15.67:0/911460387 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f388fe00640 1 --2- 172.21.15.67:0/911460387 >> v2:172.21.15.86:3300/0 conn(0x7f389015dd10 0x7f38900b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f388f400640 1 -- 172.21.15.67:0/911460387 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089230838 0 0) 0x7f38900b0cc0 con 0x7f38900b07b0 2024-04-17T10:33:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f388f400640 1 -- 172.21.15.67:0/911460387 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3874003530 con 0x7f38900b07b0 2024-04-17T10:33:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3887400640 1 -- 172.21.15.67:0/911460387 >> v2:172.21.15.86:3300/0 conn(0x7f389015dd10 msgr2=0x7f38900b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3887400640 1 --2- 172.21.15.67:0/911460387 >> v2:172.21.15.86:3300/0 conn(0x7f389015dd10 0x7f38900b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3887400640 1 -- 172.21.15.67:0/911460387 >> v1:172.21.15.86:6789/0 conn(0x7f38900b07b0 legacy=0x7f38901618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.614+0000 7f3887400640 1 -- 172.21.15.67:0/911460387 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389006bed0 con 0x7f38900b57b0 2024-04-17T10:33:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3887400640 1 --2- 172.21.15.67:0/911460387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38900b57b0 0x7f3890070dc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f38780097c0 tx=0x7f387801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8743b4913efabb34 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f388f400640 1 -- 172.21.15.67:0/911460387 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38780b1070 con 0x7f38900b57b0 2024-04-17T10:33:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f388f400640 1 -- 172.21.15.67:0/911460387 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38780ac620 con 0x7f38900b57b0 2024-04-17T10:33:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f388f400640 1 -- 172.21.15.67:0/911460387 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38780ac9f0 con 0x7f38900b57b0 2024-04-17T10:33:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 -- 172.21.15.67:0/911460387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38900b57b0 msgr2=0x7f3890070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 --2- 172.21.15.67:0/911460387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38900b57b0 0x7f3890070dc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f38780097c0 tx=0x7f387801c3e0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 -- 172.21.15.67:0/911460387 shutdown_connections 2024-04-17T10:33:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 --2- 172.21.15.67:0/911460387 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f38900b57b0 0x7f3890070dc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 --2- 172.21.15.67:0/911460387 >> v2:172.21.15.86:3300/0 conn(0x7f389015dd10 0x7f38900b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 -- 172.21.15.67:0/911460387 >> 172.21.15.67:0/911460387 conn(0x7f38900b34d0 msgr2=0x7f38900b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 -- 172.21.15.67:0/911460387 shutdown_connections 2024-04-17T10:33:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.618+0000 7f3896e00640 1 -- 172.21.15.67:0/911460387 wait complete. 2024-04-17T10:33:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 Processor -- start 2024-04-17T10:33:04.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 -- start start 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38900b07b0 0x7f389014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f389015dd10 0x7f38900cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3890141140 0x7f3890151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389016ae70 con 0x7f3890141140 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3890071710 con 0x7f38900b07b0 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389016aff0 con 0x7f389015dd10 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3890141140 0x7f3890151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3890141140 0x7f3890151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41788/0 (socket says 172.21.15.67:41788) 2024-04-17T10:33:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 -- 172.21.15.67:0/4028537664 learned_addr learned my addr 172.21.15.67:0/4028537664 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3887400640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f389015dd10 0x7f38900cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f388fe00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38900b07b0 0x7f389014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 -- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f389015dd10 msgr2=0x7f38900cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f389015dd10 0x7f38900cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 -- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38900b07b0 msgr2=0x7f389014f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38900b07b0 0x7f389014f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 -- 172.21.15.67:0/4028537664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389014f820 con 0x7f3890141140 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f388fe00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38900b07b0 0x7f389014f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3894e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3890141140 0x7f3890151360 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f388800cbe0 tx=0x7f388800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f388cc00640 1 -- 172.21.15.67:0/4028537664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888017070 con 0x7f3890141140 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f388cc00640 1 -- 172.21.15.67:0/4028537664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388800ad40 con 0x7f3890141140 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389014fb00 con 0x7f3890141140 2024-04-17T10:33:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f388cc00640 1 -- 172.21.15.67:0/4028537664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3888012600 con 0x7f3890141140 2024-04-17T10:33:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38901500c0 con 0x7f3890141140 2024-04-17T10:33:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.622+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3890070bb0 con 0x7f3890141140 2024-04-17T10:33:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.626+0000 7f388cc00640 1 -- 172.21.15.67:0/4028537664 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f388801a070 con 0x7f3890141140 2024-04-17T10:33:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.626+0000 7f388cc00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f386c066f90 0x7f386c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.626+0000 7f388cc00640 1 -- 172.21.15.67:0/4028537664 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3888084fe0 con 0x7f3890141140 2024-04-17T10:33:04.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.626+0000 7f388fe00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f386c066f90 0x7f386c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:04.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.626+0000 7f388fe00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f386c066f90 0x7f386c069450 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3880009a40 tx=0x7f3880008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.630+0000 7f388cc00640 1 -- 172.21.15.67:0/4028537664 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f38880556c0 con 0x7f3890141140 2024-04-17T10:33:04.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3890074e30 con 0x7f3890141140 2024-04-17T10:33:04.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f388cc00640 1 -- 172.21.15.67:0/4028537664 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1446 (secure 0 0 0) 0x7f3888059760 con 0x7f3890141140 2024-04-17T10:33:04.841 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:04.841 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":515,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076514771,"bytes_used":2583994368,"bytes_avail":770510118912,"bytes_total":773094113280,"write_bytes_sec":36358863,"read_op_per_sec":0,"write_op_per_sec":36},"fsmap":{"epoch":73,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8139}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":18,"modified":"2024-04-17T10:32:22.243227+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f386c066f90 msgr2=0x7f386c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f386c066f90 0x7f386c069450 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3880009a40 tx=0x7f3880008040 comp rx=0 tx=0).stop 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3890141140 msgr2=0x7f3890151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3890141140 0x7f3890151360 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f388800cbe0 tx=0x7f388800a430 comp rx=0 tx=0).stop 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 shutdown_connections 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f386c066f90 0x7f386c069450 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3890141140 0x7f3890151360 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f389015dd10 0x7f38900cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 --2- 172.21.15.67:0/4028537664 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38900b07b0 0x7f389014f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 >> 172.21.15.67:0/4028537664 conn(0x7f38900b34d0 msgr2=0x7f389015b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 shutdown_connections 2024-04-17T10:33:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:04.850+0000 7f3896e00640 1 -- 172.21.15.67:0/4028537664 wait complete. 2024-04-17T10:33:04.861 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:33:05.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.018+0000 7faf15e00640 1 Processor -- start 2024-04-17T10:33:06.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.018+0000 7faf15e00640 1 -- start start 2024-04-17T10:33:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.018+0000 7faf15e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faf100ce2a0 0x7faf100cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.018+0000 7faf15e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1006b100 con 0x7faf100cc840 2024-04-17T10:33:06.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.018+0000 7faf15e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1006b3e0 con 0x7faf100cc250 2024-04-17T10:33:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.018+0000 7faf15e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf100cd150 con 0x7faf100ce2a0 2024-04-17T10:33:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.018+0000 7faf0f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7faf100cc840 0x7faf100cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44726/0 (socket says 172.21.15.67:44726) 2024-04-17T10:33:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf0f400640 1 -- 172.21.15.67:0/3294484514 learned_addr learned my addr 172.21.15.67:0/3294484514 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf07e00640 1 --2- 172.21.15.67:0/3294484514 >> v2:172.21.15.86:3300/0 conn(0x7faf100ce2a0 0x7faf100cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf0e000640 1 -- 172.21.15.67:0/3294484514 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 195759023 0 0) 0x7faf1006b100 con 0x7faf100cc840 2024-04-17T10:33:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf0e000640 1 -- 172.21.15.67:0/3294484514 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faef4003530 con 0x7faf100cc840 2024-04-17T10:33:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf07e00640 1 -- 172.21.15.67:0/3294484514 >> v1:172.21.15.86:6789/0 conn(0x7faf100cc250 legacy=0x7faf100cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf07e00640 1 -- 172.21.15.67:0/3294484514 >> v1:172.21.15.67:6789/0 conn(0x7faf100cc840 legacy=0x7faf100cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf0e000640 1 -- 172.21.15.67:0/3294484514 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299482784 0 0) 0x7faef4003530 con 0x7faf100cc840 2024-04-17T10:33:06.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf07e00640 1 -- 172.21.15.67:0/3294484514 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1006b6c0 con 0x7faf100ce2a0 2024-04-17T10:33:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf07e00640 1 --2- 172.21.15.67:0/3294484514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 0x7faf100cd5e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faef80097c0 tx=0x7faef801c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34141fa9da05dddc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.022+0000 7faf0e000640 1 -- 172.21.15.67:0/3294484514 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef80b0070 con 0x7faf100ce2a0 2024-04-17T10:33:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf0e000640 1 -- 172.21.15.67:0/3294484514 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef80ab400 con 0x7faf100ce2a0 2024-04-17T10:33:06.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf0e000640 1 -- 172.21.15.67:0/3294484514 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf08001a00 con 0x7faf100ce2a0 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- 172.21.15.67:0/3294484514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 msgr2=0x7faf100cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 --2- 172.21.15.67:0/3294484514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 0x7faf100cd5e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faef80097c0 tx=0x7faef801c3c0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- 172.21.15.67:0/3294484514 shutdown_connections 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 --2- 172.21.15.67:0/3294484514 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 0x7faf100cd5e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- 172.21.15.67:0/3294484514 >> 172.21.15.67:0/3294484514 conn(0x7faf100b22a0 msgr2=0x7faf100b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- 172.21.15.67:0/3294484514 shutdown_connections 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- 172.21.15.67:0/3294484514 wait complete. 2024-04-17T10:33:06.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 Processor -- start 2024-04-17T10:33:06.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- start start 2024-04-17T10:33:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf100cc250 0x7faf10159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf100cc840 0x7faf100c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 0x7faf100c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1016c320 con 0x7faf100cc840 2024-04-17T10:33:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf100cdf30 con 0x7faf100ce2a0 2024-04-17T10:33:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf15e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1016c4a0 con 0x7faf100cc250 2024-04-17T10:33:06.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf100cc840 0x7faf100c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf07e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf100cc840 0x7faf100c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41796/0 (socket says 172.21.15.67:41796) 2024-04-17T10:33:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf07e00640 1 -- 172.21.15.67:0/2340508037 learned_addr learned my addr 172.21.15.67:0/2340508037 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf0f400640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 0x7faf100c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.026+0000 7faf0ea00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf100cc250 0x7faf10159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07e00640 1 -- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf100cc250 msgr2=0x7faf10159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf100cc250 0x7faf10159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07e00640 1 -- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 msgr2=0x7faf100c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 0x7faf100c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07e00640 1 -- 172.21.15.67:0/2340508037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf101603e0 con 0x7faf100cc840 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf100cc840 0x7faf100c77a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7faef8018fd0 tx=0x7faef8019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07400640 1 -- 172.21.15.67:0/2340508037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef80b9020 con 0x7faf100cc840 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07400640 1 -- 172.21.15.67:0/2340508037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef80038a0 con 0x7faf100cc840 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf100c7ce0 con 0x7faf100cc840 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07400640 1 -- 172.21.15.67:0/2340508037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef80b0040 con 0x7faf100cc840 2024-04-17T10:33:06.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf10160680 con 0x7faf100cc840 2024-04-17T10:33:06.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1006cd30 con 0x7faf100cc840 2024-04-17T10:33:06.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07400640 1 -- 172.21.15.67:0/2340508037 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7faef80bc020 con 0x7faf100cc840 2024-04-17T10:33:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07400640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faeec066f60 0x7faeec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.030+0000 7faf07400640 1 -- 172.21.15.67:0/2340508037 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7faef811ee70 con 0x7faf100cc840 2024-04-17T10:33:06.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.034+0000 7faf0ea00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faeec066f60 0x7faeec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.034+0000 7faf0ea00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faeec066f60 0x7faeec069420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7faf10160d40 tx=0x7faf00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.034+0000 7faf07400640 1 -- 172.21.15.67:0/2340508037 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7faef80c3030 con 0x7faf100cc840 2024-04-17T10:33:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf1009e900 con 0x7faf100cc840 2024-04-17T10:33:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf07400640 1 -- 172.21.15.67:0/2340508037 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7faef8015fb0 con 0x7faf100cc840 2024-04-17T10:33:06.224 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:06.225 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faeec066f60 msgr2=0x7faeec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faeec066f60 0x7faeec069420 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7faf10160d40 tx=0x7faf00009040 comp rx=0 tx=0).stop 2024-04-17T10:33:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf100cc840 msgr2=0x7faf100c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf100cc840 0x7faf100c77a0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7faef8018fd0 tx=0x7faef8019160 comp rx=0 tx=0).stop 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 shutdown_connections 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faeec066f60 0x7faeec069420 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faf100ce2a0 0x7faf100c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faf100cc840 0x7faf100c77a0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 --2- 172.21.15.67:0/2340508037 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faf100cc250 0x7faf10159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 >> 172.21.15.67:0/2340508037 conn(0x7faf100b22a0 msgr2=0x7faf1015c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 shutdown_connections 2024-04-17T10:33:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.234+0000 7faf15e00640 1 -- 172.21.15.67:0/2340508037 wait complete. 2024-04-17T10:33:06.245 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:33:06.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2e4c00640 1 Processor -- start 2024-04-17T10:33:06.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2e4c00640 1 -- start start 2024-04-17T10:33:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2e4c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd2e0071310 0x7fd2e0071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2e4c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e00cc300 0x7fd2e00cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2e4c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e006e2f0 con 0x7fd2e0145040 2024-04-17T10:33:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2e4c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e00a86c0 con 0x7fd2e0071310 2024-04-17T10:33:06.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2e4c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e0071cb0 con 0x7fd2e00cc300 2024-04-17T10:33:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2d5400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd2e0071310 0x7fd2e0071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2de800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e00cc300 0x7fd2e00cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2dde00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fd2e0145040 0x7fd2e006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:51110/0 (socket says 172.21.15.67:51110) 2024-04-17T10:33:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2dde00640 1 -- 172.21.15.67:0/4202837153 learned_addr learned my addr 172.21.15.67:0/4202837153 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2de800640 1 -- 172.21.15.67:0/4202837153 >> v2:172.21.15.86:3300/0 conn(0x7fd2e0071310 msgr2=0x7fd2e0071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2de800640 1 --2- 172.21.15.67:0/4202837153 >> v2:172.21.15.86:3300/0 conn(0x7fd2e0071310 0x7fd2e0071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2de800640 1 -- 172.21.15.67:0/4202837153 >> v1:172.21.15.86:6789/0 conn(0x7fd2e0145040 legacy=0x7fd2e006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2de800640 1 -- 172.21.15.67:0/4202837153 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e006d6e0 con 0x7fd2e00cc300 2024-04-17T10:33:06.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.394+0000 7fd2d5400640 1 --2- 172.21.15.67:0/4202837153 >> v2:172.21.15.86:3300/0 conn(0x7fd2e0071310 0x7fd2e0071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:06.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2de800640 1 --2- 172.21.15.67:0/4202837153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e00cc300 0x7fd2e00cc730 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009d90 tx=0x7fd2d801f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e23f2b4339698493 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2dd400640 1 -- 172.21.15.67:0/4202837153 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80b4070 con 0x7fd2e00cc300 2024-04-17T10:33:06.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2dd400640 1 -- 172.21.15.67:0/4202837153 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d80af600 con 0x7fd2e00cc300 2024-04-17T10:33:06.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2dd400640 1 -- 172.21.15.67:0/4202837153 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d80af9d0 con 0x7fd2e00cc300 2024-04-17T10:33:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 -- 172.21.15.67:0/4202837153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e00cc300 msgr2=0x7fd2e00cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/4202837153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e00cc300 0x7fd2e00cc730 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8009d90 tx=0x7fd2d801f150 comp rx=0 tx=0).stop 2024-04-17T10:33:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 -- 172.21.15.67:0/4202837153 shutdown_connections 2024-04-17T10:33:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/4202837153 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e00cc300 0x7fd2e00cc730 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/4202837153 >> v2:172.21.15.86:3300/0 conn(0x7fd2e0071310 0x7fd2e0071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 -- 172.21.15.67:0/4202837153 >> 172.21.15.67:0/4202837153 conn(0x7fd2e006ac90 msgr2=0x7fd2e006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 -- 172.21.15.67:0/4202837153 shutdown_connections 2024-04-17T10:33:06.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.398+0000 7fd2e4c00640 1 -- 172.21.15.67:0/4202837153 wait complete. 2024-04-17T10:33:06.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 Processor -- start 2024-04-17T10:33:06.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 -- start start 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e0071310 0x7fd2e013c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2e00cc300 0x7fd2e0136190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e0145040 0x7fd2e013eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e006df80 con 0x7fd2e0071310 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e006dca0 con 0x7fd2e00cc300 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e006d370 con 0x7fd2e0145040 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e0071310 0x7fd2e013c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e0071310 0x7fd2e013c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41804/0 (socket says 172.21.15.67:41804) 2024-04-17T10:33:06.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 -- 172.21.15.67:0/363352658 learned_addr learned my addr 172.21.15.67:0/363352658 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2d5400640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2e00cc300 0x7fd2e0136190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2de800640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e0145040 0x7fd2e013eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 -- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e0145040 msgr2=0x7fd2e013eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e0145040 0x7fd2e013eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 -- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2e00cc300 msgr2=0x7fd2e0136190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2e00cc300 0x7fd2e0136190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 -- 172.21.15.67:0/363352658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e013c900 con 0x7fd2e0071310 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2dde00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e0071310 0x7fd2e013c000 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c3a0 tx=0x7fd2d000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2d6a00640 1 -- 172.21.15.67:0/363352658 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0017070 con 0x7fd2e0071310 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2d6a00640 1 -- 172.21.15.67:0/363352658 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0005cf0 con 0x7fd2e0071310 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2e013cbe0 con 0x7fd2e0071310 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2d6a00640 1 -- 172.21.15.67:0/363352658 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00072f0 con 0x7fd2e0071310 2024-04-17T10:33:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.402+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e013d1a0 con 0x7fd2e0071310 2024-04-17T10:33:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.406+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e0072d50 con 0x7fd2e0071310 2024-04-17T10:33:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.406+0000 7fd2d6a00640 1 -- 172.21.15.67:0/363352658 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd2d001a070 con 0x7fd2e0071310 2024-04-17T10:33:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.406+0000 7fd2d6a00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd2bc066f60 0x7fd2bc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.406+0000 7fd2d6a00640 1 -- 172.21.15.67:0/363352658 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd2d00845e0 con 0x7fd2e0071310 2024-04-17T10:33:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.406+0000 7fd2d5400640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd2bc066f60 0x7fd2bc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.406+0000 7fd2d5400640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd2bc066f60 0x7fd2bc069420 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009580 tx=0x7fd2c8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.410+0000 7fd2d6a00640 1 -- 172.21.15.67:0/363352658 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd2d0054ce0 con 0x7fd2e0071310 2024-04-17T10:33:06.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.626+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd2e00b0170 con 0x7fd2e0071310 2024-04-17T10:33:06.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2d6a00640 1 -- 172.21.15.67:0/363352658 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7fd2d0058d80 con 0x7fd2e0071310 2024-04-17T10:33:06.620 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:06.620 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":517,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076514771,"bytes_used":2583994368,"bytes_avail":770510118912,"bytes_total":773094113280,"write_bytes_sec":36358298,"read_op_per_sec":0,"write_op_per_sec":36},"fsmap":{"epoch":73,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8139}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-17T10:33:06.252954+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd2bc066f60 msgr2=0x7fd2bc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd2bc066f60 0x7fd2bc069420 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fd2c8009580 tx=0x7fd2c8009190 comp rx=0 tx=0).stop 2024-04-17T10:33:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e0071310 msgr2=0x7fd2e013c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e0071310 0x7fd2e013c000 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c3a0 tx=0x7fd2d000c860 comp rx=0 tx=0).stop 2024-04-17T10:33:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 shutdown_connections 2024-04-17T10:33:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd2bc066f60 0x7fd2bc069420 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd2e0145040 0x7fd2e013eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd2e00cc300 0x7fd2e0136190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 --2- 172.21.15.67:0/363352658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2e0071310 0x7fd2e013c000 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 >> 172.21.15.67:0/363352658 conn(0x7fd2e006ac90 msgr2=0x7fd2e00b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 shutdown_connections 2024-04-17T10:33:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.630+0000 7fd2e4c00640 1 -- 172.21.15.67:0/363352658 wait complete. 2024-04-17T10:33:06.641 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:33:06.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:06.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub resume 2024-04-17T10:33:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 Processor -- start 2024-04-17T10:33:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 -- start start 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c300cd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0c300cf0c0 0x7f0c300cd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c3006e390 0x7f0c3006e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c300cc8b0 con 0x7f0c3006e390 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c300ce020 con 0x7f0c300cf0c0 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c35200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300ce300 con 0x7f0c3015a590 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c3006e390 0x7f0c3006e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c3006e390 0x7f0c3006e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41818/0 (socket says 172.21.15.67:41818) 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 -- 172.21.15.67:0/3794746415 learned_addr learned my addr 172.21.15.67:0/3794746415 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2f400640 1 --2- 172.21.15.67:0/3794746415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c300cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c27e00640 1 --2- 172.21.15.67:0/3794746415 >> v2:172.21.15.86:3300/0 conn(0x7f0c300cf0c0 0x7f0c300cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 -- 172.21.15.67:0/3794746415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 msgr2=0x7f0c300cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/3794746415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c300cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 -- 172.21.15.67:0/3794746415 >> v2:172.21.15.86:3300/0 conn(0x7f0c300cf0c0 msgr2=0x7f0c300cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/3794746415 >> v2:172.21.15.86:3300/0 conn(0x7f0c300cf0c0 0x7f0c300cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.794+0000 7f0c2fe00640 1 -- 172.21.15.67:0/3794746415 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c300ce5e0 con 0x7f0c3006e390 2024-04-17T10:33:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/3794746415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c3006e820 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f0c300ce230 tx=0x7f0c2801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f08c3b8f4ba5ea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c2ea00640 1 -- 172.21.15.67:0/3794746415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280a5760 con 0x7f0c3006e390 2024-04-17T10:33:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c2ea00640 1 -- 172.21.15.67:0/3794746415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c280a5900 con 0x7f0c3006e390 2024-04-17T10:33:06.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c2ea00640 1 -- 172.21.15.67:0/3794746415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280a5cd0 con 0x7f0c3006e390 2024-04-17T10:33:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 -- 172.21.15.67:0/3794746415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 msgr2=0x7f0c3006e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 --2- 172.21.15.67:0/3794746415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c3006e820 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f0c300ce230 tx=0x7f0c2801c9f0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 -- 172.21.15.67:0/3794746415 shutdown_connections 2024-04-17T10:33:06.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 --2- 172.21.15.67:0/3794746415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c3006e820 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 --2- 172.21.15.67:0/3794746415 >> v2:172.21.15.86:3300/0 conn(0x7f0c300cf0c0 0x7f0c300cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 --2- 172.21.15.67:0/3794746415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c300cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 -- 172.21.15.67:0/3794746415 >> 172.21.15.67:0/3794746415 conn(0x7f0c3006ac90 msgr2=0x7f0c3006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 -- 172.21.15.67:0/3794746415 shutdown_connections 2024-04-17T10:33:06.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.798+0000 7f0c35200640 1 -- 172.21.15.67:0/3794746415 wait complete. 2024-04-17T10:33:06.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 Processor -- start 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 -- start start 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c30155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c300cf0c0 0x7f0c301fa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c3006d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30071e20 con 0x7f0c3006e390 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30071ca0 con 0x7f0c300cf0c0 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30071fa0 con 0x7f0c3015a590 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c30155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c30155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41822/0 (socket says 172.21.15.67:41822) 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 -- 172.21.15.67:0/2717178891 learned_addr learned my addr 172.21.15.67:0/2717178891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c27e00640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c300cf0c0 0x7f0c301fa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c3006d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 -- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 msgr2=0x7f0c3006d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c3006d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 -- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c300cf0c0 msgr2=0x7f0c301fa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c300cf0c0 0x7f0c301fa3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 -- 172.21.15.67:0/2717178891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c30155c00 con 0x7f0c3006e390 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c2f400640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c30155620 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000d800 tx=0x7f0c2000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c27400640 1 -- 172.21.15.67:0/2717178891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2000b6c0 con 0x7f0c3006e390 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c27400640 1 -- 172.21.15.67:0/2717178891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c20019070 con 0x7f0c3006e390 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c27400640 1 -- 172.21.15.67:0/2717178891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2000ba60 con 0x7f0c3006e390 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c30147d80 con 0x7f0c3006e390 2024-04-17T10:33:06.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.802+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c301fa900 con 0x7f0c3006e390 2024-04-17T10:33:06.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.806+0000 7f0c27400640 1 -- 172.21.15.67:0/2717178891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0c2000bc00 con 0x7f0c3006e390 2024-04-17T10:33:06.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.806+0000 7f0c27400640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c0c066f60 0x7f0c0c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.806+0000 7f0c27400640 1 -- 172.21.15.67:0/2717178891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0c20085490 con 0x7f0c3006e390 2024-04-17T10:33:06.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.806+0000 7f0c27e00640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c0c066f60 0x7f0c0c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.806+0000 7f0c39c6a000 1 Processor -- start 2024-04-17T10:33:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.806+0000 7f0c39c6a000 1 -- start start 2024-04-17T10:33:06.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.806+0000 7f0c27e00640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c0c066f60 0x7f0c0c069420 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0c300cbfe0 tx=0x7f0c18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db59d0 0x556f15db5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15d6bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15d6c0a0 0x556f15d6c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f15bb8ff0 con 0x556f15d6c0a0 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f15b41150 con 0x556f15db59d0 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f15bb5090 con 0x556f15db62f0 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15d6c0a0 0x556f15d6c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15d6c0a0 0x556f15d6c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41836/0 (socket says 172.21.15.67:41836) 2024-04-17T10:33:06.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 -- 172.21.15.67:0/2653596464 learned_addr learned my addr 172.21.15.67:0/2653596464 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2f400640 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db59d0 0x556f15db5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c27e00640 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15d6bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 -- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 msgr2=0x556f15d6bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15d6bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 -- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db59d0 msgr2=0x556f15db5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db59d0 0x556f15db5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 -- 172.21.15.67:0/2653596464 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f15d4f0d0 con 0x556f15d6c0a0 2024-04-17T10:33:06.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15d6c0a0 0x556f15d6c480 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0c280af040 tx=0x7f0c28004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a56f82d7e2351e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c1fe00640 1 -- 172.21.15.67:0/2653596464 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280a5760 con 0x556f15d6c0a0 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c1fe00640 1 -- 172.21.15.67:0/2653596464 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c280a5900 con 0x556f15d6c0a0 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c1fe00640 1 -- 172.21.15.67:0/2653596464 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280a5cf0 con 0x556f15d6c0a0 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2653596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15d6c0a0 msgr2=0x556f15d6c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15d6c0a0 0x556f15d6c480 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f0c280af040 tx=0x7f0c28004ac0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2653596464 shutdown_connections 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15d6c0a0 0x556f15d6c480 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15d6bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2653596464 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db59d0 0x556f15db5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2653596464 >> 172.21.15.67:0/2653596464 conn(0x556f15d68020 msgr2=0x556f15d6a440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2653596464 shutdown_connections 2024-04-17T10:33:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2653596464 wait complete. 2024-04-17T10:33:06.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 -- start start 2024-04-17T10:33:06.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.810+0000 7f0c39c6a000 1 Processor -- start 2024-04-17T10:33:06.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15d9a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db59d0 0x556f15d9ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db62f0 0x556f15d9b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f15b7d250 con 0x556f15db62f0 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f15b73690 con 0x556f15d6c0a0 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f15b41a40 con 0x556f15db59d0 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 10 client.0 _ll_register_callbacks cb 0x556f15d56e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db62f0 0x556f15d9b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c27e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db59d0 0x556f15d9ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db62f0 0x556f15d9b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41842/0 (socket says 172.21.15.67:41842) 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 -- 172.21.15.67:0/1862047936 learned_addr learned my addr 172.21.15.67:0/1862047936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2f400640 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15d9a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 -- 172.21.15.67:0/1862047936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db59d0 msgr2=0x556f15d9ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db59d0 0x556f15d9ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 -- 172.21.15.67:0/1862047936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 msgr2=0x556f15d9a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15d9a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 -- 172.21.15.67:0/1862047936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f15c661c0 con 0x556f15db62f0 2024-04-17T10:33:06.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db62f0 0x556f15d9b270 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0c2801ced0 tx=0x7f0c2801b230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea0a3f24e5852eb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:06.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 1 -- 172.21.15.67:0/1862047936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c280a5b70 con 0x556f15db62f0 2024-04-17T10:33:06.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c280a5900 con 0x556f15db62f0 2024-04-17T10:33:06.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 1 -- 172.21.15.67:0/1862047936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0c280a5d10 con 0x556f15db62f0 2024-04-17T10:33:06.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 1 -- 172.21.15.67:0/1862047936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c280cf050 con 0x556f15db62f0 2024-04-17T10:33:06.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 10 client.8391 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:06.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x556f15c62170 con 0x556f15db62f0 2024-04-17T10:33:06.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 1 -- 172.21.15.67:0/1862047936 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c280a5900 con 0x556f15db62f0 2024-04-17T10:33:06.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 1 -- 172.21.15.67:0/1862047936 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0c280d2020 con 0x556f15db62f0 2024-04-17T10:33:06.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 10 client.8391 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:06.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 10 client.8391 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:06.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 20 client.8391 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:06.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc5c80 0x556f15dc8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 4 client.8391 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:06.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f15b54c60 con 0x556f15dc5c80 2024-04-17T10:33:06.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2f400640 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc5c80 0x556f15dc8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c2f400640 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc5c80 0x556f15dc8070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af0ebf51a62cc2d0 server_cookie=c17476528439c246 in_seq=0 out_seq=0 2024-04-17T10:33:06.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 10 client.8391 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:06.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 1 -- 172.21.15.67:0/1862047936 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f0c20082a90 con 0x556f15dc5c80 2024-04-17T10:33:06.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 10 client.8391 handle_command_reply: tid=0 2024-04-17T10:33:06.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 client.8391 shutdown 2024-04-17T10:33:06.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db62f0 msgr2=0x556f15d9b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db62f0 0x556f15d9b270 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f0c2801ced0 tx=0x7f0c2801b230 comp rx=0 tx=0).stop 2024-04-17T10:33:06.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 shutdown_connections 2024-04-17T10:33:06.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc5c80 0x556f15dc8070 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db62f0 0x556f15d9b270 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db59d0 0x556f15d9ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c1cc00640 0 client.8391 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:06.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/1862047936 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15d9a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.814+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 >> 172.21.15.67:0/1862047936 conn(0x556f15d68020 msgr2=0x556f15d6aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.818+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 shutdown_connections 2024-04-17T10:33:06.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.818+0000 7f0c39c6a000 1 -- 172.21.15.67:0/1862047936 wait complete. 2024-04-17T10:33:06.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.818+0000 7f0c39c6a000 20 client.8391 trim_cache size 0 max 16384 2024-04-17T10:33:06.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 Processor -- start 2024-04-17T10:33:06.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 -- start start 2024-04-17T10:33:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15d6c0a0 0x556f15d6a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15d6a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db62f0 0x556f15d6ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30203000 con 0x556f15db59d0 2024-04-17T10:33:06.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30202200 con 0x556f15db62f0 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301453e0 con 0x556f15d6c0a0 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.830+0000 7f0c27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15d6a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15d6a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41858/0 (socket says 172.21.15.67:41858) 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 -- 172.21.15.67:0/3144086371 learned_addr learned my addr 172.21.15.67:0/3144086371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db62f0 0x556f15d6ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c2f400640 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15d6c0a0 0x556f15d6a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 -- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15d6c0a0 msgr2=0x556f15d6a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15d6c0a0 0x556f15d6a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 -- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db62f0 msgr2=0x556f15d6ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db62f0 0x556f15d6ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 -- 172.21.15.67:0/3144086371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f15bffde0 con 0x556f15db59d0 2024-04-17T10:33:06.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db62f0 0x556f15d6ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c2f400640 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15d6c0a0 0x556f15d6a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15d6a950 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0c18010ae0 tx=0x7f0c18016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4afddc099525143 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c1f400640 1 -- 172.21.15.67:0/3144086371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1801c070 con 0x556f15db59d0 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c1f400640 1 -- 172.21.15.67:0/3144086371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1801d040 con 0x556f15db59d0 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c1f400640 1 -- 172.21.15.67:0/3144086371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18024470 con 0x556f15db59d0 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- 172.21.15.67:0/3144086371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 msgr2=0x556f15d6a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15d6a950 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0c18010ae0 tx=0x7f0c18016000 comp rx=0 tx=0).stop 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- 172.21.15.67:0/3144086371 shutdown_connections 2024-04-17T10:33:06.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15db62f0 0x556f15d6ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15d6a950 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/3144086371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15d6c0a0 0x556f15d6a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- 172.21.15.67:0/3144086371 >> 172.21.15.67:0/3144086371 conn(0x556f15d68020 msgr2=0x556f15dc2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- 172.21.15.67:0/3144086371 shutdown_connections 2024-04-17T10:33:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- 172.21.15.67:0/3144086371 wait complete. 2024-04-17T10:33:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- start start 2024-04-17T10:33:06.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 Processor -- start 2024-04-17T10:33:06.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15e15720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15e15c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15dc7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f15b53e00 con 0x556f15db59d0 2024-04-17T10:33:06.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c301459b0 con 0x556f15d6c0a0 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f15bb8300 con 0x556f15db62f0 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c39c6a000 10 client.0 _ll_register_callbacks cb 0x556f15d37040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15e15c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15e15c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41870/0 (socket says 172.21.15.67:41870) 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 -- 172.21.15.67:0/2317629891 learned_addr learned my addr 172.21.15.67:0/2317629891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c2f400640 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15e15720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c2fe00640 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15dc7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 -- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 msgr2=0x556f15dc7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.834+0000 7f0c27e00640 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15dc7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c27e00640 1 -- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 msgr2=0x556f15e15720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c27e00640 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15e15720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c27e00640 1 -- 172.21.15.67:0/2317629891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f15b5a8e0 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c27e00640 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15e15c60 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0c180077a0 tx=0x7f0c18004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40e8e0c06e5480ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 1 -- 172.21.15.67:0/2317629891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c18039060 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 1 -- 172.21.15.67:0/2317629891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0c18024900 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c1801d040 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 1 -- 172.21.15.67:0/2317629891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c1803f050 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 10 client.8397 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x556f15c128f0 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 1 -- 172.21.15.67:0/2317629891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1801d040 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 1 -- 172.21.15.67:0/2317629891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0c18042020 con 0x556f15db59d0 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 10 client.8397 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 10 client.8397 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:06.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 20 client.8397 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc8950 0x556f15dc8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 4 client.8397 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "scrub resume"}) v1 -- 0x556f15bc6530 con 0x556f15dc8950 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c2f400640 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc8950 0x556f15dc8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 10 client.8397 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c2f400640 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc8950 0x556f15dc8d30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ec9e38a15cdc1e6 server_cookie=35f48caa3caca36e in_seq=0 out_seq=0 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 1 -- 172.21.15.67:0/2317629891 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x556f15bc6530 con 0x556f15dc8950 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 10 client.8397 handle_command_reply: tid=0 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 client.8397 shutdown 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 msgr2=0x556f15e15c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15e15c60 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0c180077a0 tx=0x7f0c18004740 comp rx=0 tx=0).stop 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 shutdown_connections 2024-04-17T10:33:06.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x556f15dc8950 0x556f15dc8d30 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x556f15db62f0 0x556f15dc7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x556f15db59d0 0x556f15e15c60 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c1cc00640 0 client.8397 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 --2- 172.21.15.67:0/2317629891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x556f15d6c0a0 0x556f15e15720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 >> 172.21.15.67:0/2317629891 conn(0x556f15d68020 msgr2=0x556f15db40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 shutdown_connections 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 1 -- 172.21.15.67:0/2317629891 wait complete. 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c39c6a000 20 client.8397 trim_cache size 0 max 16384 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-04-17T10:33:06.837 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:06.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c0c066f60 msgr2=0x7f0c0c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c0c066f60 0x7f0c0c069420 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f0c300cbfe0 tx=0x7f0c18008040 comp rx=0 tx=0).stop 2024-04-17T10:33:06.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 msgr2=0x7f0c30155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c30155620 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f0c2000d800 tx=0x7f0c2000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 shutdown_connections 2024-04-17T10:33:06.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c0c066f60 0x7f0c0c069420 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c3015a590 0x7f0c3006d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c300cf0c0 0x7f0c301fa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 --2- 172.21.15.67:0/2717178891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c3006e390 0x7f0c30155620 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 >> 172.21.15.67:0/2717178891 conn(0x7f0c3006ac90 msgr2=0x7f0c300b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:06.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 shutdown_connections 2024-04-17T10:33:06.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:06.838+0000 7f0c35200640 1 -- 172.21.15.67:0/2717178891 wait complete. 2024-04-17T10:33:06.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:06.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.7:0 scrub status 2024-04-17T10:33:06.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa564000640 1 Processor -- start 2024-04-17T10:33:06.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa564000640 1 -- start start 2024-04-17T10:33:06.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa564000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa55c0cd030 0x7fa55c06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa564000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa55c06e7b0 0x7fa55c0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa564000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa55c0b4990 con 0x7fa55c0aef60 2024-04-17T10:33:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa564000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa55c0af340 con 0x7fa55c06e7b0 2024-04-17T10:33:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa564000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa55c06ec00 con 0x7fa55c0cd030 2024-04-17T10:33:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa562000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa55c06e7b0 0x7fa55c0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa561600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa55c0aef60 0x7fa55c0cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44728/0 (socket says 172.21.15.67:44728) 2024-04-17T10:33:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa561600640 1 -- 172.21.15.67:0/162095611 learned_addr learned my addr 172.21.15.67:0/162095611 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa560c00640 1 --2- 172.21.15.67:0/162095611 >> v2:172.21.15.86:3300/0 conn(0x7fa55c0cd030 0x7fa55c06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa562000640 1 -- 172.21.15.67:0/162095611 >> v2:172.21.15.86:3300/0 conn(0x7fa55c0cd030 msgr2=0x7fa55c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa562000640 1 --2- 172.21.15.67:0/162095611 >> v2:172.21.15.86:3300/0 conn(0x7fa55c0cd030 0x7fa55c06e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa562000640 1 -- 172.21.15.67:0/162095611 >> v1:172.21.15.67:6789/0 conn(0x7fa55c0aef60 legacy=0x7fa55c0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa55be00640 1 -- 172.21.15.67:0/162095611 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907283929 0 0) 0x7fa550001120 con 0x7fa55c0aef60 2024-04-17T10:33:06.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.002+0000 7fa562000640 1 -- 172.21.15.67:0/162095611 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c15ae90 con 0x7fa55c06e7b0 2024-04-17T10:33:06.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa562000640 1 --2- 172.21.15.67:0/162095611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c0b3120 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa55c15e2b0 tx=0x7fa55401eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54106f94b5dadce6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa55be00640 1 -- 172.21.15.67:0/162095611 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5540b4070 con 0x7fa55c06e7b0 2024-04-17T10:33:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa55be00640 1 -- 172.21.15.67:0/162095611 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5540a7a70 con 0x7fa55c06e7b0 2024-04-17T10:33:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa55be00640 1 -- 172.21.15.67:0/162095611 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5540a7e40 con 0x7fa55c06e7b0 2024-04-17T10:33:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 -- 172.21.15.67:0/162095611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 msgr2=0x7fa55c0b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 --2- 172.21.15.67:0/162095611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c0b3120 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7fa55c15e2b0 tx=0x7fa55401eab0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 -- 172.21.15.67:0/162095611 shutdown_connections 2024-04-17T10:33:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 --2- 172.21.15.67:0/162095611 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c0b3120 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 --2- 172.21.15.67:0/162095611 >> v2:172.21.15.86:3300/0 conn(0x7fa55c0cd030 0x7fa55c06e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 -- 172.21.15.67:0/162095611 >> 172.21.15.67:0/162095611 conn(0x7fa55c0b1740 msgr2=0x7fa55c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 -- 172.21.15.67:0/162095611 shutdown_connections 2024-04-17T10:33:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 -- 172.21.15.67:0/162095611 wait complete. 2024-04-17T10:33:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 Processor -- start 2024-04-17T10:33:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.006+0000 7fa564000640 1 -- start start 2024-04-17T10:33:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55c0aef60 0x7fa55c14ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa55c0cd030 0x7fa55c159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c15bc10 con 0x7fa55c06e7b0 2024-04-17T10:33:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c0b3e70 con 0x7fa55c0cd030 2024-04-17T10:33:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c14f4e0 con 0x7fa55c0aef60 2024-04-17T10:33:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41898/0 (socket says 172.21.15.67:41898) 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 -- 172.21.15.67:0/1236108494 learned_addr learned my addr 172.21.15.67:0/1236108494 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa562000640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa55c0cd030 0x7fa55c159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa560c00640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55c0aef60 0x7fa55c14ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 -- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55c0aef60 msgr2=0x7fa55c14ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55c0aef60 0x7fa55c14ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 -- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa55c0cd030 msgr2=0x7fa55c159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa55c0cd030 0x7fa55c159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 -- 172.21.15.67:0/1236108494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa55c14ac30 con 0x7fa55c06e7b0 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa561600640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c151610 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c7b0 tx=0x7fa55000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa559600640 1 -- 172.21.15.67:0/1236108494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550018070 con 0x7fa55c06e7b0 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa559600640 1 -- 172.21.15.67:0/1236108494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5500072f0 con 0x7fa55c06e7b0 2024-04-17T10:33:07.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa559600640 1 -- 172.21.15.67:0/1236108494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5500047f0 con 0x7fa55c06e7b0 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa55c14f240 con 0x7fa55c06e7b0 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.010+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55c153410 con 0x7fa55c06e7b0 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa559600640 1 -- 172.21.15.67:0/1236108494 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa550004990 con 0x7fa55c06e7b0 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa559600640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa53c066f60 0x7fa53c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa559600640 1 -- 172.21.15.67:0/1236108494 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa5500844a0 con 0x7fa55c06e7b0 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa560c00640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa53c066f60 0x7fa53c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 Processor -- start 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 -- start start 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa560c00640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa53c066f60 0x7fa53c069420 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0ae980 tx=0x7fa548008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x5640674c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x56406747ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640672c7ff0 con 0x5640674c49d0 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564067250150 con 0x56406747b0a0 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5640672c4090 con 0x5640674c52f0 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x5640674c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x5640674c4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41906/0 (socket says 172.21.15.67:41906) 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 -- 172.21.15.67:0/2560546321 learned_addr learned my addr 172.21.15.67:0/2560546321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa560c00640 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x56406747ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa562000640 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 -- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 msgr2=0x56406747ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x56406747ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 -- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 msgr2=0x56406747b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 -- 172.21.15.67:0/2560546321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56406745e0d0 con 0x5640674c49d0 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa561600640 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x5640674c4db0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa550054bc0 tx=0x7fa550054610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd966d766782ab20 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa54e000640 1 -- 172.21.15.67:0/2560546321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55005d290 con 0x5640674c49d0 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa54e000640 1 -- 172.21.15.67:0/2560546321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa550052f30 con 0x5640674c49d0 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.014+0000 7fa54e000640 1 -- 172.21.15.67:0/2560546321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa550004c80 con 0x5640674c49d0 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- 172.21.15.67:0/2560546321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 msgr2=0x5640674c4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x5640674c4db0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fa550054bc0 tx=0x7fa550054610 comp rx=0 tx=0).stop 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- 172.21.15.67:0/2560546321 shutdown_connections 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x56406747ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 --2- 172.21.15.67:0/2560546321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x5640674c4db0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- 172.21.15.67:0/2560546321 >> 172.21.15.67:0/2560546321 conn(0x564067477020 msgr2=0x564067479440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- 172.21.15.67:0/2560546321 shutdown_connections 2024-04-17T10:33:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- 172.21.15.67:0/2560546321 wait complete. 2024-04-17T10:33:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- start start 2024-04-17T10:33:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 Processor -- start 2024-04-17T10:33:07.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56406747b0a0 0x5640674a97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674a9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5640674c52f0 0x5640674aa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56406728c250 con 0x56406747b0a0 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x564067282690 con 0x5640674c52f0 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x564067250a40 con 0x5640674c49d0 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 10 client.0 _ll_register_callbacks cb 0x564067465e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56406747b0a0 0x5640674a97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56406747b0a0 0x5640674a97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41910/0 (socket says 172.21.15.67:41910) 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 -- 172.21.15.67:0/2879517963 learned_addr learned my addr 172.21.15.67:0/2879517963 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa562000640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5640674c52f0 0x5640674aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa560c00640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674a9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 -- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 msgr2=0x5640674a9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674a9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 -- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5640674c52f0 msgr2=0x5640674aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5640674c52f0 0x5640674aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 -- 172.21.15.67:0/2879517963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640673751c0 con 0x56406747b0a0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa560c00640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa561600640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56406747b0a0 0x5640674a97f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa550054700 tx=0x7fa550000ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbc258f0b9dd618 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa542a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa542a00640 1 -- 172.21.15.67:0/2879517963 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55005ce00 con 0x56406747b0a0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa542a00640 1 -- 172.21.15.67:0/2879517963 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa55005cfa0 con 0x56406747b0a0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa550052f30 con 0x56406747b0a0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa542a00640 1 -- 172.21.15.67:0/2879517963 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa550068b40 con 0x56406747b0a0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 10 client.8412 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x564067371170 con 0x56406747b0a0 2024-04-17T10:33:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.018+0000 7fa542a00640 1 -- 172.21.15.67:0/2879517963 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa550052f30 con 0x56406747b0a0 2024-04-17T10:33:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa542a00640 1 -- 172.21.15.67:0/2879517963 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa550068ce0 con 0x56406747b0a0 2024-04-17T10:33:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 10 client.8412 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 10 client.8412 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 20 client.8412 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674aaf50 0x5640674d6c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 4 client.8412 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564067263c60 con 0x5640674aaf50 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa560c00640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674aaf50 0x5640674d6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa542a00640 10 client.8412 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa560c00640 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674aaf50 0x5640674d6c90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6fcc2007c59c2a8 server_cookie=122bcd5d2ac0482c in_seq=0 out_seq=0 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa542a00640 1 -- 172.21.15.67:0/2879517963 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564067263c60 con 0x5640674aaf50 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa542a00640 10 client.8412 handle_command_reply: tid=0 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 client.8412 shutdown 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56406747b0a0 msgr2=0x5640674a97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56406747b0a0 0x5640674a97f0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fa550054700 tx=0x7fa550000ee0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 shutdown_connections 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674aaf50 0x5640674d6c90 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5640674c52f0 0x5640674aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674a9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 --2- 172.21.15.67:0/2879517963 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56406747b0a0 0x5640674a97f0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa542a00640 0 client.8412 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 >> 172.21.15.67:0/2879517963 conn(0x564067477020 msgr2=0x564067479ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 shutdown_connections 2024-04-17T10:33:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 1 -- 172.21.15.67:0/2879517963 wait complete. 2024-04-17T10:33:07.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.022+0000 7fa56743e000 20 client.8412 trim_cache size 0 max 16384 2024-04-17T10:33:07.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 Processor -- start 2024-04-17T10:33:07.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- start start 2024-04-17T10:33:07.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c52f0 0x5640674c4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c203310 con 0x5640674c52f0 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c2024d0 con 0x56406747b0a0 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c155040 con 0x5640674c49d0 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c52f0 0x5640674c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c52f0 0x5640674c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41926/0 (socket says 172.21.15.67:41926) 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 -- 172.21.15.67:0/1477018043 learned_addr learned my addr 172.21.15.67:0/1477018043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa561600640 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa560c00640 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 -- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 msgr2=0x5640674c3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674c3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 -- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 msgr2=0x56406747a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 -- 172.21.15.67:0/1477018043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56406730ede0 con 0x5640674c52f0 2024-04-17T10:33:07.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa561600640 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:07.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa562000640 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c52f0 0x5640674c4040 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fa55400d010 tx=0x7fa554004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b29612eeeb148a76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa54d600640 1 -- 172.21.15.67:0/1477018043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5540b4040 con 0x5640674c52f0 2024-04-17T10:33:07.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa54d600640 1 -- 172.21.15.67:0/1477018043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5540b5040 con 0x5640674c52f0 2024-04-17T10:33:07.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa54d600640 1 -- 172.21.15.67:0/1477018043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5540af880 con 0x5640674c52f0 2024-04-17T10:33:07.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- 172.21.15.67:0/1477018043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c52f0 msgr2=0x5640674c4040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c52f0 0x5640674c4040 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7fa55400d010 tx=0x7fa554004ac0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- 172.21.15.67:0/1477018043 shutdown_connections 2024-04-17T10:33:07.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c52f0 0x5640674c4040 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c49d0 0x5640674c3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 --2- 172.21.15.67:0/1477018043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x56406747a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- 172.21.15.67:0/1477018043 >> 172.21.15.67:0/1477018043 conn(0x564067477020 msgr2=0x5640674c33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.038+0000 7fa56743e000 1 -- 172.21.15.67:0/1477018043 shutdown_connections 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- 172.21.15.67:0/1477018043 wait complete. 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- start start 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 Processor -- start 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x564067524f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x564067525470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x5640674d74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x564067262e00 con 0x5640674c49d0 2024-04-17T10:33:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa55c201ed0 con 0x56406747b0a0 2024-04-17T10:33:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5640672c7300 con 0x5640674c52f0 2024-04-17T10:33:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 10 client.0 _ll_register_callbacks cb 0x564067446040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x564067525470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x564067525470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41928/0 (socket says 172.21.15.67:41928) 2024-04-17T10:33:07.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 -- 172.21.15.67:0/3774592926 learned_addr learned my addr 172.21.15.67:0/3774592926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa562000640 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x5640674d74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa561600640 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x564067524f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 -- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 msgr2=0x5640674d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x5640674d74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 -- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 msgr2=0x564067524f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x564067524f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 -- 172.21.15.67:0/3774592926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640672698e0 con 0x5640674c49d0 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa560c00640 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x564067525470 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa55c15a580 tx=0x7fa548004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e411067ca69ce8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa542a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa542a00640 1 -- 172.21.15.67:0/3774592926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54801c040 con 0x5640674c49d0 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5540b5040 con 0x5640674c49d0 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa542a00640 1 -- 172.21.15.67:0/3774592926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa54802ad10 con 0x5640674c49d0 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa542a00640 1 -- 172.21.15.67:0/3774592926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=73) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa548039050 con 0x5640674c49d0 2024-04-17T10:33:07.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 10 client.8418 fetch_fsmap learned FSMap version 73 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=73,monmap=2+}) v3 -- 0x5640673218f0 con 0x5640674c49d0 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa542a00640 1 -- 172.21.15.67:0/3774592926 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5540b5040 con 0x5640674c49d0 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa542a00640 1 -- 172.21.15.67:0/3774592926 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 73) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa54803c020 con 0x5640674c49d0 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 10 client.8418 fetch_fsmap finished waiting for FSMap version 73 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 10 client.8418 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.a 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 20 client.8418 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674d8060 0x5640674d8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 4 client.8418 mds_command: new command op to 8139 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 --> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5640672d5530 con 0x5640674d8060 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.042+0000 7fa561600640 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674d8060 0x5640674d8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa542a00640 10 client.8418 ms_handle_connect on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa561600640 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674d8060 0x5640674d8470 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55aa1e4850ad38ed server_cookie=40ad536ee3ccf261 in_seq=0 out_seq=0 2024-04-17T10:33:07.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa542a00640 1 -- 172.21.15.67:0/3774592926 <== mds.0 v2:172.21.15.67:6832/2012054720 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5640672d5530 con 0x5640674d8060 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa542a00640 10 client.8418 handle_command_reply: tid=0 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 client.8418 shutdown 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 msgr2=0x564067525470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x564067525470 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa55c15a580 tx=0x7fa548004740 comp rx=0 tx=0).stop 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 shutdown_connections 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:6832/2012054720,v1:172.21.15.67:6834/2012054720] conn(0x5640674d8060 0x5640674d8470 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5640674c52f0 0x5640674d74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5640674c49d0 0x564067525470 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 --2- 172.21.15.67:0/3774592926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56406747b0a0 0x564067524f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa542a00640 0 client.8418 ms_handle_reset on v2:172.21.15.67:6832/2012054720 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 >> 172.21.15.67:0/3774592926 conn(0x564067477020 msgr2=0x5640674c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 shutdown_connections 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 1 -- 172.21.15.67:0/3774592926 wait complete. 2024-04-17T10:33:07.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa56743e000 20 client.8418 trim_cache size 0 max 16384 2024-04-17T10:33:07.046 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:07.046 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-04-17T10:33:07.046 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:33:07.046 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa53c066f60 msgr2=0x7fa53c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa53c066f60 0x7fa53c069420 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa55c0ae980 tx=0x7fa548008040 comp rx=0 tx=0).stop 2024-04-17T10:33:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 msgr2=0x7fa55c151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c151610 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7fa55000c7b0 tx=0x7fa55000cc70 comp rx=0 tx=0).stop 2024-04-17T10:33:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 shutdown_connections 2024-04-17T10:33:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa53c066f60 0x7fa53c069420 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa55c0cd030 0x7fa55c159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa55c0aef60 0x7fa55c14ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 --2- 172.21.15.67:0/1236108494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa55c06e7b0 0x7fa55c151610 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 >> 172.21.15.67:0/1236108494 conn(0x7fa55c0b1740 msgr2=0x7fa55c0b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 shutdown_connections 2024-04-17T10:33:07.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.046+0000 7fa564000640 1 -- 172.21.15.67:0/1236108494 wait complete. 2024-04-17T10:33:07.056 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55cd400640 1 Processor -- start 2024-04-17T10:33:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55cd400640 1 -- start start 2024-04-17T10:33:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55cd400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f55c815fbd0 0x7f55c81667e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55cd400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c815be60 con 0x7f55c815cc00 2024-04-17T10:33:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55cd400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c8160230 con 0x7f55c80ce1b0 2024-04-17T10:33:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55cd400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c8160000 con 0x7f55c815fbd0 2024-04-17T10:33:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55bfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f55c815cc00 0x7f55c815f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44730/0 (socket says 172.21.15.67:44730) 2024-04-17T10:33:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55bfe00640 1 -- 172.21.15.67:0/1328313823 learned_addr learned my addr 172.21.15.67:0/1328313823 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55c7200640 1 --2- 172.21.15.67:0/1328313823 >> v2:172.21.15.86:3300/0 conn(0x7f55c815fbd0 0x7f55c81667e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55c5e00640 1 -- 172.21.15.67:0/1328313823 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667369489 0 0) 0x7f55c815be60 con 0x7f55c815cc00 2024-04-17T10:33:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55c5e00640 1 -- 172.21.15.67:0/1328313823 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55ac003530 con 0x7f55c815cc00 2024-04-17T10:33:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55c7200640 1 -- 172.21.15.67:0/1328313823 >> v1:172.21.15.86:6789/0 conn(0x7f55c80ce1b0 legacy=0x7f55c80ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55c7200640 1 -- 172.21.15.67:0/1328313823 >> v1:172.21.15.67:6789/0 conn(0x7f55c815cc00 legacy=0x7f55c815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.206+0000 7f55c7200640 1 -- 172.21.15.67:0/1328313823 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c815c120 con 0x7f55c815fbd0 2024-04-17T10:33:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55c7200640 1 --2- 172.21.15.67:0/1328313823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815fbd0 0x7f55c81667e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009d90 tx=0x7f55c001f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89b331097fe66209 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55c5e00640 1 -- 172.21.15.67:0/1328313823 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c00b4070 con 0x7f55c815fbd0 2024-04-17T10:33:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55c5e00640 1 -- 172.21.15.67:0/1328313823 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c00af600 con 0x7f55c815fbd0 2024-04-17T10:33:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55c5e00640 1 -- 172.21.15.67:0/1328313823 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c00af9d0 con 0x7f55c815fbd0 2024-04-17T10:33:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55cd400640 1 -- 172.21.15.67:0/1328313823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815fbd0 msgr2=0x7f55c81667e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55cd400640 1 --2- 172.21.15.67:0/1328313823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815fbd0 0x7f55c81667e0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009d90 tx=0x7f55c001f150 comp rx=0 tx=0).stop 2024-04-17T10:33:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55cd400640 1 -- 172.21.15.67:0/1328313823 shutdown_connections 2024-04-17T10:33:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55cd400640 1 --2- 172.21.15.67:0/1328313823 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815fbd0 0x7f55c81667e0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55cd400640 1 -- 172.21.15.67:0/1328313823 >> 172.21.15.67:0/1328313823 conn(0x7f55c80b33f0 msgr2=0x7f55c80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55cd400640 1 -- 172.21.15.67:0/1328313823 shutdown_connections 2024-04-17T10:33:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.210+0000 7f55cd400640 1 -- 172.21.15.67:0/1328313823 wait complete. 2024-04-17T10:33:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 Processor -- start 2024-04-17T10:33:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 -- start start 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55c80ce1b0 0x7f55c814c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815cc00 0x7f55c81f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c815fbd0 0x7f55c81f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c8159730 con 0x7f55c815fbd0 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c815c6e0 con 0x7f55c815cc00 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c81598b0 con 0x7f55c80ce1b0 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c815fbd0 0x7f55c81f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c815fbd0 0x7f55c81f3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41942/0 (socket says 172.21.15.67:41942) 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 -- 172.21.15.67:0/1220704336 learned_addr learned my addr 172.21.15.67:0/1220704336 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55bfe00640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815cc00 0x7f55c81f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c6800640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55c80ce1b0 0x7f55c814c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 -- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55c80ce1b0 msgr2=0x7f55c814c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55c80ce1b0 0x7f55c814c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 -- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815cc00 msgr2=0x7f55c81f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815cc00 0x7f55c81f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 -- 172.21.15.67:0/1220704336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c81f8330 con 0x7f55c815fbd0 2024-04-17T10:33:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55c7200640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c815fbd0 0x7f55c81f3e80 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f55c00b2040 tx=0x7f55c001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55bea00640 1 -- 172.21.15.67:0/1220704336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c00b4040 con 0x7f55c815fbd0 2024-04-17T10:33:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55bea00640 1 -- 172.21.15.67:0/1220704336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c00c1070 con 0x7f55c815fbd0 2024-04-17T10:33:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55bea00640 1 -- 172.21.15.67:0/1220704336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0009450 con 0x7f55c815fbd0 2024-04-17T10:33:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c81f4860 con 0x7f55c815fbd0 2024-04-17T10:33:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.214+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c81f75f0 con 0x7f55c815fbd0 2024-04-17T10:33:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.218+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c8070ab0 con 0x7f55c815fbd0 2024-04-17T10:33:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.218+0000 7f55bea00640 1 -- 172.21.15.67:0/1220704336 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f55c00ba6b0 con 0x7f55c815fbd0 2024-04-17T10:33:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.218+0000 7f55bea00640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f55a4066f60 0x7f55a4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.218+0000 7f55bea00640 1 -- 172.21.15.67:0/1220704336 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f55c0123950 con 0x7f55c815fbd0 2024-04-17T10:33:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.218+0000 7f55c6800640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f55a4066f60 0x7f55a4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.218+0000 7f55c6800640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f55a4066f60 0x7f55a4069420 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f55b8007fd0 tx=0x7f55b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.222+0000 7f55bea00640 1 -- 172.21.15.67:0/1220704336 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f55c81f8330 con 0x7f55c815fbd0 2024-04-17T10:33:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55c8074e30 con 0x7f55c815fbd0 2024-04-17T10:33:07.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55bea00640 1 -- 172.21.15.67:0/1220704336 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f55c0018cb0 con 0x7f55c815fbd0 2024-04-17T10:33:07.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:07.413 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:07.414 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f55a4066f60 msgr2=0x7f55a4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f55a4066f60 0x7f55a4069420 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f55b8007fd0 tx=0x7f55b8009040 comp rx=0 tx=0).stop 2024-04-17T10:33:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c815fbd0 msgr2=0x7f55c81f3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c815fbd0 0x7f55c81f3e80 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f55c00b2040 tx=0x7f55c001bef0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 shutdown_connections 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f55a4066f60 0x7f55a4069420 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f55c815fbd0 0x7f55c81f3e80 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f55c815cc00 0x7f55c81f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 --2- 172.21.15.67:0/1220704336 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f55c80ce1b0 0x7f55c814c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 >> 172.21.15.67:0/1220704336 conn(0x7f55c80b33f0 msgr2=0x7f55c80b2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 shutdown_connections 2024-04-17T10:33:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.422+0000 7f55cd400640 1 -- 172.21.15.67:0/1220704336 wait complete. 2024-04-17T10:33:07.435 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:33:07.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:07.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.582+0000 7f408f800640 1 Processor -- start 2024-04-17T10:33:07.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.582+0000 7f408f800640 1 -- start start 2024-04-17T10:33:07.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f408f800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4088144aa0 0x7f40880d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f408f800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4088071840 0x7f40880d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f408f800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40881456d0 con 0x7f40881446c0 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f408f800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40880b51a0 con 0x7f4088144aa0 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f408f800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40880d1390 con 0x7f4088071840 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f407fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4088144aa0 0x7f40880d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f407fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4088144aa0 0x7f40880d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41946/0 (socket says 172.21.15.67:41946) 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f407fe00640 1 -- 172.21.15.67:0/973087387 learned_addr learned my addr 172.21.15.67:0/973087387 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f408d800640 1 --2- 172.21.15.67:0/973087387 >> v2:172.21.15.86:3300/0 conn(0x7f4088071840 0x7f40880d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f407fe00640 1 -- 172.21.15.67:0/973087387 >> v2:172.21.15.86:3300/0 conn(0x7f4088071840 msgr2=0x7f40880d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f407fe00640 1 --2- 172.21.15.67:0/973087387 >> v2:172.21.15.86:3300/0 conn(0x7f4088071840 0x7f40880d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f407fe00640 1 -- 172.21.15.67:0/973087387 >> v1:172.21.15.86:6789/0 conn(0x7f40881446c0 legacy=0x7f4088144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.586+0000 7f407fe00640 1 -- 172.21.15.67:0/973087387 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40880b6640 con 0x7f4088144aa0 2024-04-17T10:33:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f407fe00640 1 --2- 172.21.15.67:0/973087387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880d0910 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f408806f0e0 tx=0x7f407001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f7f6cc64fb6caf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f4087e00640 1 -- 172.21.15.67:0/973087387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40700aa430 con 0x7f4088144aa0 2024-04-17T10:33:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f4087e00640 1 -- 172.21.15.67:0/973087387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40700af070 con 0x7f4088144aa0 2024-04-17T10:33:07.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f4087e00640 1 -- 172.21.15.67:0/973087387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40700aa7f0 con 0x7f4088144aa0 2024-04-17T10:33:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 -- 172.21.15.67:0/973087387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 msgr2=0x7f40880d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 --2- 172.21.15.67:0/973087387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880d0910 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f408806f0e0 tx=0x7f407001c4d0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 -- 172.21.15.67:0/973087387 shutdown_connections 2024-04-17T10:33:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 --2- 172.21.15.67:0/973087387 >> v2:172.21.15.86:3300/0 conn(0x7f4088071840 0x7f40880d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 --2- 172.21.15.67:0/973087387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880d0910 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 -- 172.21.15.67:0/973087387 >> 172.21.15.67:0/973087387 conn(0x7f40880b34d0 msgr2=0x7f40880b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 -- 172.21.15.67:0/973087387 shutdown_connections 2024-04-17T10:33:07.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 -- 172.21.15.67:0/973087387 wait complete. 2024-04-17T10:33:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 Processor -- start 2024-04-17T10:33:07.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.590+0000 7f408f800640 1 -- start start 2024-04-17T10:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4088071840 0x7f408813fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40881446c0 0x7f40880af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40880b5e20 con 0x7f4088144aa0 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408815c0d0 con 0x7f4088071840 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40880b5fa0 con 0x7f40881446c0 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4088071840 0x7f408813fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41958/0 (socket says 172.21.15.67:41958) 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 -- 172.21.15.67:0/620237802 learned_addr learned my addr 172.21.15.67:0/620237802 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f407fe00640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40881446c0 0x7f40880af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 -- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40881446c0 msgr2=0x7f40880af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40881446c0 0x7f40880af600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 -- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4088071840 msgr2=0x7f408813fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4088071840 0x7f408813fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 -- 172.21.15.67:0/620237802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408815a610 con 0x7f4088144aa0 2024-04-17T10:33:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f407fe00640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40881446c0 0x7f40880af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408d800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880afb40 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f408000efc0 tx=0x7f408000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f4085600640 1 -- 172.21.15.67:0/620237802 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080018020 con 0x7f4088144aa0 2024-04-17T10:33:07.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f4085600640 1 -- 172.21.15.67:0/620237802 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4080014480 con 0x7f4088144aa0 2024-04-17T10:33:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f4085600640 1 -- 172.21.15.67:0/620237802 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4080014870 con 0x7f4088144aa0 2024-04-17T10:33:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408815a8f0 con 0x7f4088144aa0 2024-04-17T10:33:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408815aeb0 con 0x7f4088144aa0 2024-04-17T10:33:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4088070bb0 con 0x7f4088144aa0 2024-04-17T10:33:07.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.594+0000 7f4085600640 1 -- 172.21.15.67:0/620237802 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f408001b020 con 0x7f4088144aa0 2024-04-17T10:33:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.598+0000 7f4085600640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4064066f60 0x7f4064069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.598+0000 7f4085600640 1 -- 172.21.15.67:0/620237802 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4080085fe0 con 0x7f4088144aa0 2024-04-17T10:33:07.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.598+0000 7f408ce00640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4064066f60 0x7f4064069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:07.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.598+0000 7f408ce00640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4064066f60 0x7f4064069420 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f40881f7f10 tx=0x7f4078009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:07.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.602+0000 7f4085600640 1 -- 172.21.15.67:0/620237802 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4080022030 con 0x7f4088144aa0 2024-04-17T10:33:07.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.818+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f408815beb0 con 0x7f4088144aa0 2024-04-17T10:33:07.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.818+0000 7f4085600640 1 -- 172.21.15.67:0/620237802 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7f4080010040 con 0x7f4088144aa0 2024-04-17T10:33:07.810 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:07.810 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":518,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076514771,"bytes_used":2583994368,"bytes_avail":770510118912,"bytes_total":773094113280,"write_bytes_sec":36358298,"read_op_per_sec":0,"write_op_per_sec":36},"fsmap":{"epoch":73,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8139}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-17T10:33:06.252954+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.818+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4064066f60 msgr2=0x7f4064069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.818+0000 7f408f800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4064066f60 0x7f4064069420 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f40881f7f10 tx=0x7f4078009040 comp rx=0 tx=0).stop 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.818+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 msgr2=0x7f40880afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.818+0000 7f408f800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880afb40 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f408000efc0 tx=0x7f408000c490 comp rx=0 tx=0).stop 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.818+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 shutdown_connections 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.822+0000 7f408f800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4064066f60 0x7f4064069420 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.822+0000 7f408f800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4088144aa0 0x7f40880afb40 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.822+0000 7f408f800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f40881446c0 0x7f40880af600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.822+0000 7f408f800640 1 --2- 172.21.15.67:0/620237802 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4088071840 0x7f408813fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:07.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.822+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 >> 172.21.15.67:0/620237802 conn(0x7f40880b34d0 msgr2=0x7f4088140a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.822+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 shutdown_connections 2024-04-17T10:33:07.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:07.822+0000 7f408f800640 1 -- 172.21.15.67:0/620237802 wait complete. 2024-04-17T10:33:07.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:33:08.832 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:08.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f829ce00640 1 Processor -- start 2024-04-17T10:33:08.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f829ce00640 1 -- start start 2024-04-17T10:33:08.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f829ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82980b07b0 0x7f8298070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:08.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f829ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82980b5980 con 0x7f8298071300 2024-04-17T10:33:08.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f829ce00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82980b0cc0 con 0x7f829815dd10 2024-04-17T10:33:08.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f829ce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8298071770 con 0x7f82980b07b0 2024-04-17T10:33:08.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f8295400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f82980b07b0 0x7f8298070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:08.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f8296800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8298071300 0x7f829806c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44736/0 (socket says 172.21.15.67:44736) 2024-04-17T10:33:08.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f8296800640 1 -- 172.21.15.67:0/1520919052 learned_addr learned my addr 172.21.15.67:0/1520919052 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:08.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f8295400640 1 -- 172.21.15.67:0/1520919052 >> v1:172.21.15.86:6789/0 conn(0x7f829815dd10 legacy=0x7f82980b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:08.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f8295400640 1 -- 172.21.15.67:0/1520919052 >> v1:172.21.15.67:6789/0 conn(0x7f8298071300 legacy=0x7f829806c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:08.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f8295400640 1 -- 172.21.15.67:0/1520919052 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82980b0fa0 con 0x7f82980b07b0 2024-04-17T10:33:08.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.994+0000 7f8294a00640 1 -- 172.21.15.67:0/1520919052 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084835424 0 0) 0x7f82980b5980 con 0x7f8298071300 2024-04-17T10:33:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f8295400640 1 --2- 172.21.15.67:0/1520919052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82980b07b0 0x7f8298070dc0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f82840097c0 tx=0x7f828401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a1b9da80dc3c3b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f8294a00640 1 -- 172.21.15.67:0/1520919052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840b0070 con 0x7f82980b07b0 2024-04-17T10:33:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f8294a00640 1 -- 172.21.15.67:0/1520919052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82840ab600 con 0x7f82980b07b0 2024-04-17T10:33:08.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f8294a00640 1 -- 172.21.15.67:0/1520919052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82840ab9d0 con 0x7f82980b07b0 2024-04-17T10:33:08.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f829ce00640 1 -- 172.21.15.67:0/1520919052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82980b07b0 msgr2=0x7f8298070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:08.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f829ce00640 1 --2- 172.21.15.67:0/1520919052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82980b07b0 0x7f8298070dc0 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f82840097c0 tx=0x7f828401c3d0 comp rx=0 tx=0).stop 2024-04-17T10:33:08.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f829ce00640 1 -- 172.21.15.67:0/1520919052 shutdown_connections 2024-04-17T10:33:08.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f829ce00640 1 --2- 172.21.15.67:0/1520919052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82980b07b0 0x7f8298070dc0 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:08.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f829ce00640 1 -- 172.21.15.67:0/1520919052 >> 172.21.15.67:0/1520919052 conn(0x7f82980b34d0 msgr2=0x7f82980b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:08.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f829ce00640 1 -- 172.21.15.67:0/1520919052 shutdown_connections 2024-04-17T10:33:08.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:08.998+0000 7f829ce00640 1 -- 172.21.15.67:0/1520919052 wait complete. 2024-04-17T10:33:08.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 Processor -- start 2024-04-17T10:33:08.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 -- start start 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8298071300 0x7f829813f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82980b07b0 0x7f8298150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f829815dd10 0x7f8298151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829806d770 con 0x7f8298071300 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829806d5f0 con 0x7f82980b07b0 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829806d8f0 con 0x7f829815dd10 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8298071300 0x7f829813f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8298071300 0x7f829813f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41986/0 (socket says 172.21.15.67:41986) 2024-04-17T10:33:08.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 -- 172.21.15.67:0/1734767040 learned_addr learned my addr 172.21.15.67:0/1734767040 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295400640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82980b07b0 0x7f8298150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8296800640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f829815dd10 0x7f8298151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 -- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f829815dd10 msgr2=0x7f8298151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f829815dd10 0x7f8298151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 -- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82980b07b0 msgr2=0x7f8298150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82980b07b0 0x7f8298150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 -- 172.21.15.67:0/1734767040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82980cbde0 con 0x7f8298071300 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f8295e00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8298071300 0x7f829813f020 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f828800b410 tx=0x7f828800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f828e000640 1 -- 172.21.15.67:0/1734767040 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8288015030 con 0x7f8298071300 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f828e000640 1 -- 172.21.15.67:0/1734767040 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8288004020 con 0x7f8298071300 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f828e000640 1 -- 172.21.15.67:0/1734767040 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8288004450 con 0x7f8298071300 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82980cc0c0 con 0x7f8298071300 2024-04-17T10:33:08.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82980cc680 con 0x7f8298071300 2024-04-17T10:33:08.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.002+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8298070bb0 con 0x7f8298071300 2024-04-17T10:33:08.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.006+0000 7f828e000640 1 -- 172.21.15.67:0/1734767040 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8288018070 con 0x7f8298071300 2024-04-17T10:33:08.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.006+0000 7f828e000640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8278066f90 0x7f8278069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:08.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.006+0000 7f828e000640 1 -- 172.21.15.67:0/1734767040 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8288084e00 con 0x7f8298071300 2024-04-17T10:33:08.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.006+0000 7f8295400640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8278066f90 0x7f8278069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:08.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.006+0000 7f8295400640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8278066f90 0x7f8278069450 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f82840ad000 tx=0x7f82840a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:09.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.010+0000 7f828e000640 1 -- 172.21.15.67:0/1734767040 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8288055be0 con 0x7f8298071300 2024-04-17T10:33:09.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.206+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8298074e30 con 0x7f8298071300 2024-04-17T10:33:09.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.206+0000 7f828e000640 1 -- 172.21.15.67:0/1734767040 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f82880554e0 con 0x7f8298071300 2024-04-17T10:33:09.199 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:09.199 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:09.199 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8278066f90 msgr2=0x7f8278069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8278066f90 0x7f8278069450 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f82840ad000 tx=0x7f82840a6000 comp rx=0 tx=0).stop 2024-04-17T10:33:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8298071300 msgr2=0x7f829813f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8298071300 0x7f829813f020 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f828800b410 tx=0x7f828800b8d0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 shutdown_connections 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8278066f90 0x7f8278069450 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f829815dd10 0x7f8298151490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82980b07b0 0x7f8298150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 --2- 172.21.15.67:0/1734767040 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8298071300 0x7f829813f020 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 >> 172.21.15.67:0/1734767040 conn(0x7f82980b34d0 msgr2=0x7f829815b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 shutdown_connections 2024-04-17T10:33:09.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.210+0000 7f829ce00640 1 -- 172.21.15.67:0/1734767040 wait complete. 2024-04-17T10:33:09.220 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:33:09.220 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:09.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df8200640 1 Processor -- start 2024-04-17T10:33:09.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df8200640 1 -- start start 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df8200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9df01446c0 0x7f9df0144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df8200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9df0071840 0x7f9df00b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df8200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df01456d0 con 0x7f9df0144aa0 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df8200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df00b51a0 con 0x7f9df0071840 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df8200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df00b4870 con 0x7f9df01446c0 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df6200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9df0071840 0x7f9df00b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df4e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9df0144aa0 0x7f9df00d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44750/0 (socket says 172.21.15.67:44750) 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df4e00640 1 -- 172.21.15.67:0/1407408389 learned_addr learned my addr 172.21.15.67:0/1407408389 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:09.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df5800640 1 --2- 172.21.15.67:0/1407408389 >> v2:172.21.15.86:3300/0 conn(0x7f9df01446c0 0x7f9df0144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:09.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df6200640 1 -- 172.21.15.67:0/1407408389 >> v2:172.21.15.86:3300/0 conn(0x7f9df01446c0 msgr2=0x7f9df0144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:09.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df6200640 1 --2- 172.21.15.67:0/1407408389 >> v2:172.21.15.86:3300/0 conn(0x7f9df01446c0 0x7f9df0144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df6200640 1 -- 172.21.15.67:0/1407408389 >> v1:172.21.15.67:6789/0 conn(0x7f9df0144aa0 legacy=0x7f9df00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.370+0000 7f9df6200640 1 -- 172.21.15.67:0/1407408389 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00b4b50 con 0x7f9df0071840 2024-04-17T10:33:09.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df6200640 1 --2- 172.21.15.67:0/1407408389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df0071840 0x7f9df00b5630 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f9df00b5320 tx=0x7f9de801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f49a1e2d12018d0b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9defe00640 1 -- 172.21.15.67:0/1407408389 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80b2070 con 0x7f9df0071840 2024-04-17T10:33:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9defe00640 1 -- 172.21.15.67:0/1407408389 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de80a58d0 con 0x7f9df0071840 2024-04-17T10:33:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9defe00640 1 -- 172.21.15.67:0/1407408389 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de80a5ca0 con 0x7f9df0071840 2024-04-17T10:33:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 -- 172.21.15.67:0/1407408389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df0071840 msgr2=0x7f9df00b5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 --2- 172.21.15.67:0/1407408389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df0071840 0x7f9df00b5630 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f9df00b5320 tx=0x7f9de801c900 comp rx=0 tx=0).stop 2024-04-17T10:33:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 -- 172.21.15.67:0/1407408389 shutdown_connections 2024-04-17T10:33:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 --2- 172.21.15.67:0/1407408389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df0071840 0x7f9df00b5630 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 --2- 172.21.15.67:0/1407408389 >> v2:172.21.15.86:3300/0 conn(0x7f9df01446c0 0x7f9df0144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 -- 172.21.15.67:0/1407408389 >> 172.21.15.67:0/1407408389 conn(0x7f9df00b34d0 msgr2=0x7f9df00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 -- 172.21.15.67:0/1407408389 shutdown_connections 2024-04-17T10:33:09.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.374+0000 7f9df8200640 1 -- 172.21.15.67:0/1407408389 wait complete. 2024-04-17T10:33:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 Processor -- start 2024-04-17T10:33:09.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 -- start start 2024-04-17T10:33:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df0071840 0x7f9df006d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df01446c0 0x7f9df01652c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df0144aa0 0x7f9df006bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00b6500 con 0x7f9df01446c0 2024-04-17T10:33:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00b6380 con 0x7f9df0071840 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00b6680 con 0x7f9df0144aa0 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df01446c0 0x7f9df01652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df01446c0 0x7f9df01652c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42002/0 (socket says 172.21.15.67:42002) 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 -- 172.21.15.67:0/3412074397 learned_addr learned my addr 172.21.15.67:0/3412074397 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df5800640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df0071840 0x7f9df006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df6200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df0144aa0 0x7f9df006bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 -- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df0144aa0 msgr2=0x7f9df006bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df0144aa0 0x7f9df006bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 -- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df0071840 msgr2=0x7f9df006d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df0071840 0x7f9df006d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 -- 172.21.15.67:0/3412074397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df006c860 con 0x7f9df01446c0 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df6200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df0144aa0 0x7f9df006bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df4e00640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df01446c0 0x7f9df01652c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00c3d0 tx=0x7f9ddc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:09.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9ded600640 1 -- 172.21.15.67:0/3412074397 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc017070 con 0x7f9df01446c0 2024-04-17T10:33:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9ded600640 1 -- 172.21.15.67:0/3412074397 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ddc005cf0 con 0x7f9df01446c0 2024-04-17T10:33:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9ded600640 1 -- 172.21.15.67:0/3412074397 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc0072f0 con 0x7f9df01446c0 2024-04-17T10:33:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df006cb40 con 0x7f9df01446c0 2024-04-17T10:33:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.378+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df006d100 con 0x7f9df01446c0 2024-04-17T10:33:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.382+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df0070bb0 con 0x7f9df01446c0 2024-04-17T10:33:09.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.382+0000 7f9ded600640 1 -- 172.21.15.67:0/3412074397 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9ddc01a070 con 0x7f9df01446c0 2024-04-17T10:33:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.382+0000 7f9ded600640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9dd0066f60 0x7f9dd0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.382+0000 7f9ded600640 1 -- 172.21.15.67:0/3412074397 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9ddc084480 con 0x7f9df01446c0 2024-04-17T10:33:09.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.382+0000 7f9df5800640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9dd0066f60 0x7f9dd0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.382+0000 7f9df5800640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9dd0066f60 0x7f9dd0069420 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009af0 tx=0x7f9de4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:09.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.386+0000 7f9ded600640 1 -- 172.21.15.67:0/3412074397 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9ddc0552f0 con 0x7f9df01446c0 2024-04-17T10:33:09.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.602+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f9df0073920 con 0x7f9df01446c0 2024-04-17T10:33:09.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9ded600640 1 -- 172.21.15.67:0/3412074397 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7f9ddc054bf0 con 0x7f9df01446c0 2024-04-17T10:33:09.596 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:09.596 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":520,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076583876,"bytes_used":2584133632,"bytes_avail":770509979648,"bytes_total":773094113280,"write_bytes_sec":36364007,"read_op_per_sec":0,"write_op_per_sec":36},"fsmap":{"epoch":73,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8139}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-04-17T10:33:06.252954+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9dd0066f60 msgr2=0x7f9dd0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9dd0066f60 0x7f9dd0069420 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9de4009af0 tx=0x7f9de4008040 comp rx=0 tx=0).stop 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df01446c0 msgr2=0x7f9df01652c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df01446c0 0x7f9df01652c0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f9ddc00c3d0 tx=0x7f9ddc00c890 comp rx=0 tx=0).stop 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 shutdown_connections 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9dd0066f60 0x7f9dd0069420 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9df0144aa0 0x7f9df006bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9df01446c0 0x7f9df01652c0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 --2- 172.21.15.67:0/3412074397 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9df0071840 0x7f9df006d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 >> 172.21.15.67:0/3412074397 conn(0x7f9df00b34d0 msgr2=0x7f9df015d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 shutdown_connections 2024-04-17T10:33:09.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:09.606+0000 7f9df8200640 1 -- 172.21.15.67:0/3412074397 wait complete. 2024-04-17T10:33:09.618 INFO:tasks.cephfs.filesystem.ceph_manager:{'a': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:33:10.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:10.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3942000640 1 Processor -- start 2024-04-17T10:33:10.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3942000640 1 -- start start 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3942000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f393c06eac0 0x7f393c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3942000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3942000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393c144560 con 0x7f393c06e6e0 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3942000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393c15ae40 con 0x7f393c06eac0 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3942000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c071440 con 0x7f393c0cf0c0 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3932a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f393c06eac0 0x7f393c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f393b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f393c06e6e0 0x7f393c0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49666/0 (socket says 172.21.15.67:49666) 2024-04-17T10:33:10.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f393b400640 1 -- 172.21.15.67:0/2961619807 learned_addr learned my addr 172.21.15.67:0/2961619807 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3932a00640 1 -- 172.21.15.67:0/2961619807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 msgr2=0x7f393c144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f393be00640 1 --2- 172.21.15.67:0/2961619807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f393aa00640 1 -- 172.21.15.67:0/2961619807 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998153154 0 0) 0x7f393c144560 con 0x7f393c06e6e0 2024-04-17T10:33:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3932a00640 1 --2- 172.21.15.67:0/2961619807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3932a00640 1 -- 172.21.15.67:0/2961619807 >> v1:172.21.15.67:6789/0 conn(0x7f393c06e6e0 legacy=0x7f393c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:10.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f3932a00640 1 -- 172.21.15.67:0/2961619807 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c071720 con 0x7f393c06eac0 2024-04-17T10:33:10.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.782+0000 7f393be00640 1 --2- 172.21.15.67:0/2961619807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3932a00640 1 --2- 172.21.15.67:0/2961619807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c143ae0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f39240097c0 tx=0x7f392401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a62166f11e150d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f393aa00640 1 -- 172.21.15.67:0/2961619807 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240ab600 con 0x7f393c06eac0 2024-04-17T10:33:10.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f393aa00640 1 -- 172.21.15.67:0/2961619807 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39240b0070 con 0x7f393c06eac0 2024-04-17T10:33:10.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f393aa00640 1 -- 172.21.15.67:0/2961619807 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240ab9c0 con 0x7f393c06eac0 2024-04-17T10:33:10.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 -- 172.21.15.67:0/2961619807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 msgr2=0x7f393c143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 --2- 172.21.15.67:0/2961619807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c143ae0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f39240097c0 tx=0x7f392401c3e0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 -- 172.21.15.67:0/2961619807 shutdown_connections 2024-04-17T10:33:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 --2- 172.21.15.67:0/2961619807 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 --2- 172.21.15.67:0/2961619807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c143ae0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 -- 172.21.15.67:0/2961619807 >> 172.21.15.67:0/2961619807 conn(0x7f393c06ac90 msgr2=0x7f393c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 -- 172.21.15.67:0/2961619807 shutdown_connections 2024-04-17T10:33:10.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.786+0000 7f3942000640 1 -- 172.21.15.67:0/2961619807 wait complete. 2024-04-17T10:33:10.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 Processor -- start 2024-04-17T10:33:10.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 -- start start 2024-04-17T10:33:10.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393c06e6e0 0x7f393c0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:10.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c0aab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:10.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c0ad2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0b3a40 con 0x7f393c06eac0 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0b38c0 con 0x7f393c06e6e0 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393c0b3bc0 con 0x7f393c0cf0c0 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c0aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c0aab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57202/0 (socket says 172.21.15.67:57202) 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 -- 172.21.15.67:0/2524943801 learned_addr learned my addr 172.21.15.67:0/2524943801 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f393be00640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c0ad2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f393b400640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393c06e6e0 0x7f393c0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 -- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 msgr2=0x7f393c0ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c0ad2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 -- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393c06e6e0 msgr2=0x7f393c0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393c06e6e0 0x7f393c0c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 -- 172.21.15.67:0/2524943801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c0adc90 con 0x7f393c06eac0 2024-04-17T10:33:10.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3932a00640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c0aab50 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f39240097c0 tx=0x7f392401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:10.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3933e00640 1 -- 172.21.15.67:0/2524943801 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924004220 con 0x7f393c06eac0 2024-04-17T10:33:10.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3933e00640 1 -- 172.21.15.67:0/2524943801 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39240043c0 con 0x7f393c06eac0 2024-04-17T10:33:10.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3933e00640 1 -- 172.21.15.67:0/2524943801 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924003820 con 0x7f393c06eac0 2024-04-17T10:33:10.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c0b0000 con 0x7f393c06eac0 2024-04-17T10:33:10.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c0b05c0 con 0x7f393c06eac0 2024-04-17T10:33:10.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.790+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c072d50 con 0x7f393c06eac0 2024-04-17T10:33:10.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.794+0000 7f3933e00640 1 -- 172.21.15.67:0/2524943801 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f39240aa070 con 0x7f393c06eac0 2024-04-17T10:33:10.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.794+0000 7f3933e00640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:10.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.794+0000 7f3933e00640 1 -- 172.21.15.67:0/2524943801 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f392411f5a0 con 0x7f393c06eac0 2024-04-17T10:33:10.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.794+0000 7f393b400640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:10.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.794+0000 7f393b400640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f392c000f70 tx=0x7f392c007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:10.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.798+0000 7f3933e00640 1 -- 172.21.15.67:0/2524943801 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3924016090 con 0x7f393c06eac0 2024-04-17T10:33:10.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.994+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f393c074e30 con 0x7f393c06eac0 2024-04-17T10:33:10.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.994+0000 7f3933e00640 1 -- 172.21.15.67:0/2524943801 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f39240f0530 con 0x7f393c06eac0 2024-04-17T10:33:10.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:10.987 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:10.987 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 msgr2=0x7f3918069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f392c000f70 tx=0x7f392c007b60 comp rx=0 tx=0).stop 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 msgr2=0x7f393c0aab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c0aab50 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f39240097c0 tx=0x7f392401ac20 comp rx=0 tx=0).stop 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 shutdown_connections 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393c0cf0c0 0x7f393c0ad2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393c06eac0 0x7f393c0aab50 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 --2- 172.21.15.67:0/2524943801 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393c06e6e0 0x7f393c0c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 >> 172.21.15.67:0/2524943801 conn(0x7f393c06ac90 msgr2=0x7f393c1438b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 shutdown_connections 2024-04-17T10:33:10.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:10.998+0000 7f3942000640 1 -- 172.21.15.67:0/2524943801 wait complete. 2024-04-17T10:33:11.009 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:33:11.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.158+0000 7f393ea00640 1 Processor -- start 2024-04-17T10:33:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.158+0000 7f393ea00640 1 -- start start 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f393ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f393806e6e0 0x7f39380ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f393ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806eac0 0x7f39380ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f393ea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393815ae40 con 0x7f3938143ae0 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f393ea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39380cf430 con 0x7f393806e6e0 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f393ea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3938071440 con 0x7f393806eac0 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f393ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3938143ae0 0x7f3938143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49670/0 (socket says 172.21.15.67:49670) 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f393ca00640 1 -- 172.21.15.67:0/3697677812 learned_addr learned my addr 172.21.15.67:0/3697677812 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937e00640 1 --2- 172.21.15.67:0/3697677812 >> v2:172.21.15.67:3300/0 conn(0x7f393806e6e0 0x7f39380ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937400640 1 --2- 172.21.15.67:0/3697677812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806eac0 0x7f39380ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3936a00640 1 -- 172.21.15.67:0/3697677812 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659676048 0 0) 0x7f393815ae40 con 0x7f3938143ae0 2024-04-17T10:33:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937e00640 1 -- 172.21.15.67:0/3697677812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806eac0 msgr2=0x7f39380ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937e00640 1 --2- 172.21.15.67:0/3697677812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806eac0 0x7f39380ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937e00640 1 -- 172.21.15.67:0/3697677812 >> v1:172.21.15.67:6789/0 conn(0x7f3938143ae0 legacy=0x7f3938143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937e00640 1 -- 172.21.15.67:0/3697677812 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3938071720 con 0x7f393806e6e0 2024-04-17T10:33:11.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937400640 1 --2- 172.21.15.67:0/3697677812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806eac0 0x7f39380ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:11.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3937e00640 1 --2- 172.21.15.67:0/3697677812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393806e6e0 0x7f39380ce9b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f39280097c0 tx=0x7f392801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fd385163054c80 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:11.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3936a00640 1 -- 172.21.15.67:0/3697677812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280b1070 con 0x7f393806e6e0 2024-04-17T10:33:11.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3936a00640 1 -- 172.21.15.67:0/3697677812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39280a8050 con 0x7f393806e6e0 2024-04-17T10:33:11.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.162+0000 7f3936a00640 1 -- 172.21.15.67:0/3697677812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39280ac860 con 0x7f393806e6e0 2024-04-17T10:33:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- 172.21.15.67:0/3697677812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393806e6e0 msgr2=0x7f39380ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 --2- 172.21.15.67:0/3697677812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393806e6e0 0x7f39380ce9b0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f39280097c0 tx=0x7f392801c3e0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- 172.21.15.67:0/3697677812 shutdown_connections 2024-04-17T10:33:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 --2- 172.21.15.67:0/3697677812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806eac0 0x7f39380ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 --2- 172.21.15.67:0/3697677812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f393806e6e0 0x7f39380ce9b0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- 172.21.15.67:0/3697677812 >> 172.21.15.67:0/3697677812 conn(0x7f393806ac90 msgr2=0x7f393806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:11.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- 172.21.15.67:0/3697677812 shutdown_connections 2024-04-17T10:33:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- 172.21.15.67:0/3697677812 wait complete. 2024-04-17T10:33:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 Processor -- start 2024-04-17T10:33:11.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- start start 2024-04-17T10:33:11.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806e6e0 0x7f3938159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393806eac0 0x7f3938135910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3938143ae0 0x7f39380ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39380b3a40 con 0x7f3938143ae0 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39380b38c0 con 0x7f393806eac0 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39380b3bc0 con 0x7f393806e6e0 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3938143ae0 0x7f39380ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3938143ae0 0x7f39380ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57210/0 (socket says 172.21.15.67:57210) 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ca00640 1 -- 172.21.15.67:0/2047909731 learned_addr learned my addr 172.21.15.67:0/2047909731 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f3937400640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393806eac0 0x7f3938135910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f3937e00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806e6e0 0x7f3938159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ca00640 1 -- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806e6e0 msgr2=0x7f3938159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.166+0000 7f393ca00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806e6e0 0x7f3938159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f393ca00640 1 -- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393806eac0 msgr2=0x7f3938135910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f393ca00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393806eac0 0x7f3938135910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f393ca00640 1 -- 172.21.15.67:0/2047909731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393813a8d0 con 0x7f3938143ae0 2024-04-17T10:33:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f3937400640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393806eac0 0x7f3938135910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f393ca00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3938143ae0 0x7f39380ace00 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f393000c700 tx=0x7f393000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:11.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f392fe00640 1 -- 172.21.15.67:0/2047909731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930018070 con 0x7f3938143ae0 2024-04-17T10:33:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f392fe00640 1 -- 172.21.15.67:0/2047909731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393000ce10 con 0x7f3938143ae0 2024-04-17T10:33:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f392fe00640 1 -- 172.21.15.67:0/2047909731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393000a860 con 0x7f3938143ae0 2024-04-17T10:33:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393813e320 con 0x7f3938143ae0 2024-04-17T10:33:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393815a620 con 0x7f3938143ae0 2024-04-17T10:33:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3938072d50 con 0x7f3938143ae0 2024-04-17T10:33:11.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f392fe00640 1 -- 172.21.15.67:0/2047909731 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f393000aa00 con 0x7f3938143ae0 2024-04-17T10:33:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f392fe00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f392fe00640 1 -- 172.21.15.67:0/2047909731 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3930084830 con 0x7f3938143ae0 2024-04-17T10:33:11.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.170+0000 7f3937e00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.174+0000 7f3937e00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f39280ae000 tx=0x7f39280a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:11.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.174+0000 7f392fe00640 1 -- 172.21.15.67:0/2047909731 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3930055610 con 0x7f3938143ae0 2024-04-17T10:33:11.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.398+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3938074e30 con 0x7f3938143ae0 2024-04-17T10:33:11.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.398+0000 7f392fe00640 1 -- 172.21.15.67:0/2047909731 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1575 (secure 0 0 0) 0x7f3930054f10 con 0x7f3938143ae0 2024-04-17T10:33:11.391 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:11.391 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":522,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":107,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076583876,"bytes_used":2584133632,"bytes_avail":770509979648,"bytes_total":773094113280,"write_bytes_sec":12375757,"read_op_per_sec":0,"write_op_per_sec":12},"fsmap":{"epoch":73,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":8139}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":20,"modified":"2024-04-17T10:33:10.254211+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 msgr2=0x7f3918069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f39280ae000 tx=0x7f39280a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3938143ae0 msgr2=0x7f39380ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3938143ae0 0x7f39380ace00 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f393000c700 tx=0x7f393000cbc0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 shutdown_connections 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3918066f60 0x7f3918069420 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3938143ae0 0x7f39380ace00 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f393806eac0 0x7f3938135910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 --2- 172.21.15.67:0/2047909731 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f393806e6e0 0x7f3938159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 >> 172.21.15.67:0/2047909731 conn(0x7f393806ac90 msgr2=0x7f393806d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 shutdown_connections 2024-04-17T10:33:11.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.402+0000 7f393ea00640 1 -- 172.21.15.67:0/2047909731 wait complete. 2024-04-17T10:33:11.413 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:33:11.413 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:33:11.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:33:11.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:33:11.575 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:33:11.575 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:33:11.575 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:11.584 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:33:11.584 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:11.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:11.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0998c00640 1 Processor -- start 2024-04-17T10:33:11.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0998c00640 1 -- start start 2024-04-17T10:33:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0998c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0994099a50 0x7f099415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0998c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f099409a740 0x7f09940cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0998c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09940cde20 con 0x7f09940d03a0 2024-04-17T10:33:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0998c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09940cf300 con 0x7f099409a740 2024-04-17T10:33:11.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0998c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09940cf5e0 con 0x7f0994099a50 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f099409a740 0x7f09940cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f099409a740 0x7f09940cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57222/0 (socket says 172.21.15.67:57222) 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993e00640 1 -- 172.21.15.67:0/1621748607 learned_addr learned my addr 172.21.15.67:0/1621748607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993400640 1 --2- 172.21.15.67:0/1621748607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0994099a50 0x7f099415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993e00640 1 -- 172.21.15.67:0/1621748607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0994099a50 msgr2=0x7f099415c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993e00640 1 --2- 172.21.15.67:0/1621748607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0994099a50 0x7f099415c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993e00640 1 -- 172.21.15.67:0/1621748607 >> v1:172.21.15.86:6789/0 conn(0x7f09940d03a0 legacy=0x7f0994099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.830+0000 7f0993e00640 1 -- 172.21.15.67:0/1621748607 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940cf8c0 con 0x7f099409a740 2024-04-17T10:33:11.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0993e00640 1 --2- 172.21.15.67:0/1621748607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f09940cc1a0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f098c009e50 tx=0x7f098c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94b7b560ff163b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:11.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0992000640 1 -- 172.21.15.67:0/1621748607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c0a5700 con 0x7f099409a740 2024-04-17T10:33:11.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0992000640 1 -- 172.21.15.67:0/1621748607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c0a58a0 con 0x7f099409a740 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0992000640 1 -- 172.21.15.67:0/1621748607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c0a5c70 con 0x7f099409a740 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 -- 172.21.15.67:0/1621748607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 msgr2=0x7f09940cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 --2- 172.21.15.67:0/1621748607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f09940cc1a0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f098c009e50 tx=0x7f098c01ca30 comp rx=0 tx=0).stop 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 -- 172.21.15.67:0/1621748607 shutdown_connections 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 --2- 172.21.15.67:0/1621748607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f09940cc1a0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 --2- 172.21.15.67:0/1621748607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0994099a50 0x7f099415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 -- 172.21.15.67:0/1621748607 >> 172.21.15.67:0/1621748607 conn(0x7f09940d1690 msgr2=0x7f09940d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 -- 172.21.15.67:0/1621748607 shutdown_connections 2024-04-17T10:33:11.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 -- 172.21.15.67:0/1621748607 wait complete. 2024-04-17T10:33:11.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 Processor -- start 2024-04-17T10:33:11.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.834+0000 7f0998c00640 1 -- start start 2024-04-17T10:33:11.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0994099a50 0x7f0994152550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f0994164ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09940d03a0 0x7f0994165530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09941686b0 con 0x7f099409a740 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09940cd100 con 0x7f0994099a50 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0994168830 con 0x7f09940d03a0 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f0994164ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f0994164ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57232/0 (socket says 172.21.15.67:57232) 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 -- 172.21.15.67:0/3054460631 learned_addr learned my addr 172.21.15.67:0/3054460631 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0993400640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0994099a50 0x7f0994152550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0993e00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09940d03a0 0x7f0994165530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 -- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09940d03a0 msgr2=0x7f0994165530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09940d03a0 0x7f0994165530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 -- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0994099a50 msgr2=0x7f0994152550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0994099a50 0x7f0994152550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 -- 172.21.15.67:0/3054460631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09940b02c0 con 0x7f099409a740 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0992a00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f0994164ff0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f098000c420 tx=0x7f098000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f098b400640 1 -- 172.21.15.67:0/3054460631 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09800052d0 con 0x7f099409a740 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f098b400640 1 -- 172.21.15.67:0/3054460631 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0980005470 con 0x7f099409a740 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f098b400640 1 -- 172.21.15.67:0/3054460631 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0980007260 con 0x7f099409a740 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09940b05a0 con 0x7f099409a740 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0994153a20 con 0x7f099409a740 2024-04-17T10:33:11.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.838+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0994074dc0 con 0x7f099409a740 2024-04-17T10:33:11.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.842+0000 7f098b400640 1 -- 172.21.15.67:0/3054460631 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0980007400 con 0x7f099409a740 2024-04-17T10:33:11.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.842+0000 7f098b400640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0974066f90 0x7f0974069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:11.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.842+0000 7f098b400640 1 -- 172.21.15.67:0/3054460631 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f09800845d0 con 0x7f099409a740 2024-04-17T10:33:11.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.842+0000 7f0993400640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0974066f90 0x7f0974069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:11.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.842+0000 7f0993400640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0974066f90 0x7f0974069450 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f099409ad70 tx=0x7f0984008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:11.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:11.846+0000 7f098b400640 1 -- 172.21.15.67:0/3054460631 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0980054df0 con 0x7f099409a740 2024-04-17T10:33:11.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.006+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0994153bf0 con 0x7f099409a740 2024-04-17T10:33:11.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.006+0000 7f098b400640 1 -- 172.21.15.67:0/3054460631 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16836 (secure 0 0 0) 0x7f0980058e90 con 0x7f099409a740 2024-04-17T10:33:12.000 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:12.000 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":107,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:34.925635+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:32:31.849728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-17T10:32:32.441190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0974066f90 msgr2=0x7f0974069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0974066f90 0x7f0974069450 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f099409ad70 tx=0x7f0984008040 comp rx=0 tx=0).stop 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 msgr2=0x7f0994164ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f0994164ff0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f098000c420 tx=0x7f098000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 shutdown_connections 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0974066f90 0x7f0974069450 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09940d03a0 0x7f0994165530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f099409a740 0x7f0994164ff0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 --2- 172.21.15.67:0/3054460631 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0994099a50 0x7f0994152550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 >> 172.21.15.67:0/3054460631 conn(0x7f09940d1690 msgr2=0x7f0994070c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:12.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 shutdown_connections 2024-04-17T10:33:12.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.010+0000 7f0998c00640 1 -- 172.21.15.67:0/3054460631 wait complete. 2024-04-17T10:33:12.022 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:32:31.849728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:32:32.441190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:33:12.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:33:12.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.174+0000 7f4c92200640 1 Processor -- start 2024-04-17T10:33:12.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c92200640 1 -- start start 2024-04-17T10:33:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c92200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c8c0aef60 0x7f4c8c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c92200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c92200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c0b4990 con 0x7f4c8c0cce60 2024-04-17T10:33:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c92200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8c0af340 con 0x7f4c8c0aef60 2024-04-17T10:33:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c92200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c15ae90 con 0x7f4c8c06e980 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c8c0aef60 0x7f4c8c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4c8c0aef60 0x7f4c8c0cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57242/0 (socket says 172.21.15.67:57242) 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8b400640 1 -- 172.21.15.67:0/2118967143 learned_addr learned my addr 172.21.15.67:0/2118967143 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8be00640 1 --2- 172.21.15.67:0/2118967143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8b400640 1 -- 172.21.15.67:0/2118967143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 msgr2=0x7f4c8c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8b400640 1 --2- 172.21.15.67:0/2118967143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c0b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8b400640 1 -- 172.21.15.67:0/2118967143 >> v1:172.21.15.86:6789/0 conn(0x7f4c8c0cce60 legacy=0x7f4c8c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:12.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.178+0000 7f4c8b400640 1 -- 172.21.15.67:0/2118967143 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c06ee80 con 0x7f4c8c0aef60 2024-04-17T10:33:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c8b400640 1 --2- 172.21.15.67:0/2118967143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c0cc920 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009870 tx=0x7f4c7c01c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3764f5b2ad1eab3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c8aa00640 1 -- 172.21.15.67:0/2118967143 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0b0070 con 0x7f4c8c0aef60 2024-04-17T10:33:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c8aa00640 1 -- 172.21.15.67:0/2118967143 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c0ab600 con 0x7f4c8c0aef60 2024-04-17T10:33:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c8aa00640 1 -- 172.21.15.67:0/2118967143 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c0ab9d0 con 0x7f4c8c0aef60 2024-04-17T10:33:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 -- 172.21.15.67:0/2118967143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 msgr2=0x7f4c8c0cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 --2- 172.21.15.67:0/2118967143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c0cc920 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009870 tx=0x7f4c7c01c2b0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 -- 172.21.15.67:0/2118967143 shutdown_connections 2024-04-17T10:33:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 --2- 172.21.15.67:0/2118967143 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c0b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 --2- 172.21.15.67:0/2118967143 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c0cc920 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 -- 172.21.15.67:0/2118967143 >> 172.21.15.67:0/2118967143 conn(0x7f4c8c0b1740 msgr2=0x7f4c8c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 -- 172.21.15.67:0/2118967143 shutdown_connections 2024-04-17T10:33:12.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 -- 172.21.15.67:0/2118967143 wait complete. 2024-04-17T10:33:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 Processor -- start 2024-04-17T10:33:12.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.182+0000 7f4c92200640 1 -- start start 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c14d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c8c0cce60 0x7f4c8c142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0b3ff0 con 0x7f4c8c0aef60 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0b3e70 con 0x7f4c8c0cce60 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c1673c0 con 0x7f4c8c06e980 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c14d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c14d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57248/0 (socket says 172.21.15.67:57248) 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 -- 172.21.15.67:0/3211659525 learned_addr learned my addr 172.21.15.67:0/3211659525 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c8b400640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c8be00640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c8c0cce60 0x7f4c8c142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 -- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 msgr2=0x7f4c8c14f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c14f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 -- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c8c0cce60 msgr2=0x7f4c8c142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c8c0cce60 0x7f4c8c142050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 -- 172.21.15.67:0/3211659525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c14d590 con 0x7f4c8c0aef60 2024-04-17T10:33:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83e00640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c14d050 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c450 tx=0x7f4c7400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83400640 1 -- 172.21.15.67:0/3211659525 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74012830 con 0x7f4c8c0aef60 2024-04-17T10:33:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83400640 1 -- 172.21.15.67:0/3211659525 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74017070 con 0x7f4c8c0aef60 2024-04-17T10:33:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83400640 1 -- 172.21.15.67:0/3211659525 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74012c30 con 0x7f4c8c0aef60 2024-04-17T10:33:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c14d860 con 0x7f4c8c0aef60 2024-04-17T10:33:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c14de20 con 0x7f4c8c0aef60 2024-04-17T10:33:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c14e160 con 0x7f4c8c0aef60 2024-04-17T10:33:12.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.186+0000 7f4c83400640 1 -- 172.21.15.67:0/3211659525 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4c7401c460 con 0x7f4c8c0aef60 2024-04-17T10:33:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.190+0000 7f4c83400640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c68066f90 0x7f4c68069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.190+0000 7f4c83400640 1 -- 172.21.15.67:0/3211659525 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4c74084e10 con 0x7f4c8c0aef60 2024-04-17T10:33:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.190+0000 7f4c8b400640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c68066f90 0x7f4c68069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.190+0000 7f4c8b400640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c68066f90 0x7f4c68069450 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0ad000 tx=0x7f4c7c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.194+0000 7f4c83400640 1 -- 172.21.15.67:0/3211659525 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4c8c14e160 con 0x7f4c8c0aef60 2024-04-17T10:33:12.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.358+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4c8c14c510 con 0x7f4c8c0aef60 2024-04-17T10:33:12.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c83400640 1 -- 172.21.15.67:0/3211659525 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4c74088030 con 0x7f4c8c0aef60 2024-04-17T10:33:12.352 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c68066f90 msgr2=0x7f4c68069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c68066f90 0x7f4c68069450 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c0ad000 tx=0x7f4c7c008040 comp rx=0 tx=0).stop 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 msgr2=0x7f4c8c14d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c14d050 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c450 tx=0x7f4c7400c910 comp rx=0 tx=0).stop 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 shutdown_connections 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c68066f90 0x7f4c68069450 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4c8c0cce60 0x7f4c8c142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4c8c0aef60 0x7f4c8c14d050 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 --2- 172.21.15.67:0/3211659525 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4c8c06e980 0x7f4c8c14f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 >> 172.21.15.67:0/3211659525 conn(0x7f4c8c0b1740 msgr2=0x7f4c8c06d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:12.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 shutdown_connections 2024-04-17T10:33:12.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.362+0000 7f4c92200640 1 -- 172.21.15.67:0/3211659525 wait complete. 2024-04-17T10:33:12.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:33:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2966000640 1 Processor -- start 2024-04-17T10:33:12.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2966000640 1 -- start start 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2966000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f296015ddc0 0x7f2960162320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2966000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296015e1a0 con 0x7f29600ae0a0 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2966000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29600b4a30 con 0x7f29600ae500 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2966000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29600b35b0 con 0x7f296015ddc0 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f295f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f296015ddc0 0x7f2960162320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2957e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f29600ae0a0 0x7f2960072380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49684/0 (socket says 172.21.15.67:49684) 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f2957e00640 1 -- 172.21.15.67:0/3622087581 learned_addr learned my addr 172.21.15.67:0/3622087581 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:12.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f295f400640 1 -- 172.21.15.67:0/3622087581 >> v1:172.21.15.86:6789/0 conn(0x7f29600ae500 legacy=0x7f29600b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:12.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f295f400640 1 -- 172.21.15.67:0/3622087581 >> v1:172.21.15.67:6789/0 conn(0x7f29600ae0a0 legacy=0x7f2960072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f295f400640 1 -- 172.21.15.67:0/3622087581 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960162860 con 0x7f296015ddc0 2024-04-17T10:33:12.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.526+0000 7f295ea00640 1 -- 172.21.15.67:0/3622087581 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257785791 0 0) 0x7f296015e1a0 con 0x7f29600ae0a0 2024-04-17T10:33:12.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f295f400640 1 --2- 172.21.15.67:0/3622087581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f296015ddc0 0x7f2960162320 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f29500097c0 tx=0x7f295001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3154bf82207f977a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f295ea00640 1 -- 172.21.15.67:0/3622087581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29500b0070 con 0x7f296015ddc0 2024-04-17T10:33:12.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f295ea00640 1 -- 172.21.15.67:0/3622087581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29500ab600 con 0x7f296015ddc0 2024-04-17T10:33:12.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f295ea00640 1 -- 172.21.15.67:0/3622087581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29500ab9d0 con 0x7f296015ddc0 2024-04-17T10:33:12.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f2966000640 1 -- 172.21.15.67:0/3622087581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f296015ddc0 msgr2=0x7f2960162320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f2966000640 1 --2- 172.21.15.67:0/3622087581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f296015ddc0 0x7f2960162320 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f29500097c0 tx=0x7f295001c3d0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f2966000640 1 -- 172.21.15.67:0/3622087581 shutdown_connections 2024-04-17T10:33:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f2966000640 1 --2- 172.21.15.67:0/3622087581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f296015ddc0 0x7f2960162320 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f2966000640 1 -- 172.21.15.67:0/3622087581 >> 172.21.15.67:0/3622087581 conn(0x7f29600b2570 msgr2=0x7f2960071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f2966000640 1 -- 172.21.15.67:0/3622087581 shutdown_connections 2024-04-17T10:33:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.530+0000 7f2966000640 1 -- 172.21.15.67:0/3622087581 wait complete. 2024-04-17T10:33:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 Processor -- start 2024-04-17T10:33:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 -- start start 2024-04-17T10:33:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29600ae0a0 0x7f29601fa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29600ae500 0x7f2960153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f296015ddc0 0x7f2960154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296006c180 con 0x7f29600ae0a0 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296006c000 con 0x7f29600ae500 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2960072a90 con 0x7f296015ddc0 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29600ae0a0 0x7f29601fa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29600ae0a0 0x7f29601fa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57274/0 (socket says 172.21.15.67:57274) 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 -- 172.21.15.67:0/1321327221 learned_addr learned my addr 172.21.15.67:0/1321327221 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2957e00640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29600ae500 0x7f2960153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295fe00640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f296015ddc0 0x7f2960154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 -- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f296015ddc0 msgr2=0x7f2960154340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f296015ddc0 0x7f2960154340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 -- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29600ae500 msgr2=0x7f2960153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29600ae500 0x7f2960153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 -- 172.21.15.67:0/1321327221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29600ceef0 con 0x7f29600ae0a0 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2957e00640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29600ae500 0x7f2960153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f295f400640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29600ae0a0 0x7f29601fa870 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f29500ae040 tx=0x7f295001ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2957400640 1 -- 172.21.15.67:0/1321327221 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29500b0050 con 0x7f29600ae0a0 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2957400640 1 -- 172.21.15.67:0/1321327221 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29500bc070 con 0x7f29600ae0a0 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2957400640 1 -- 172.21.15.67:0/1321327221 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29500b5830 con 0x7f29600ae0a0 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29600cf1d0 con 0x7f29600ae0a0 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.534+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296015e2d0 con 0x7f29600ae0a0 2024-04-17T10:33:12.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.538+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296004ef30 con 0x7f29600ae0a0 2024-04-17T10:33:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.538+0000 7f2957400640 1 -- 172.21.15.67:0/1321327221 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f29500b59d0 con 0x7f29600ae0a0 2024-04-17T10:33:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.538+0000 7f2957400640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f293c066f60 0x7f293c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.538+0000 7f2957400640 1 -- 172.21.15.67:0/1321327221 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f295011fb10 con 0x7f29600ae0a0 2024-04-17T10:33:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.538+0000 7f2957e00640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f293c066f60 0x7f293c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.538+0000 7f2957e00640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f293c066f60 0x7f293c069420 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f296015d7a0 tx=0x7f2948007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.542+0000 7f2957400640 1 -- 172.21.15.67:0/1321327221 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2950016050 con 0x7f29600ae0a0 2024-04-17T10:33:12.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.706+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2960066ea0 con 0x7f29600ae0a0 2024-04-17T10:33:12.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.706+0000 7f2957400640 1 -- 172.21.15.67:0/1321327221 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29500f08f0 con 0x7f29600ae0a0 2024-04-17T10:33:12.699 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:33:12.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f293c066f60 msgr2=0x7f293c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f293c066f60 0x7f293c069420 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f296015d7a0 tx=0x7f2948007c20 comp rx=0 tx=0).stop 2024-04-17T10:33:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29600ae0a0 msgr2=0x7f29601fa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29600ae0a0 0x7f29601fa870 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f29500ae040 tx=0x7f295001ac10 comp rx=0 tx=0).stop 2024-04-17T10:33:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 shutdown_connections 2024-04-17T10:33:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f293c066f60 0x7f293c069420 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f296015ddc0 0x7f2960154340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f29600ae500 0x7f2960153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 --2- 172.21.15.67:0/1321327221 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f29600ae0a0 0x7f29601fa870 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 >> 172.21.15.67:0/1321327221 conn(0x7f29600b2570 msgr2=0x7f29600b0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:12.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 shutdown_connections 2024-04-17T10:33:12.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.710+0000 7f2966000640 1 -- 172.21.15.67:0/1321327221 wait complete. 2024-04-17T10:33:12.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:33:12.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fe5e00640 1 Processor -- start 2024-04-17T10:33:12.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fe5e00640 1 -- start start 2024-04-17T10:33:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fe5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fe015c580 0x7f7fe00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fe5e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe015cb30 con 0x7f7fe0159440 2024-04-17T10:33:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fe5e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00cd610 con 0x7f7fe0071af0 2024-04-17T10:33:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fe5e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe00a8650 con 0x7f7fe015c580 2024-04-17T10:33:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fdea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7fe0159440 0x7f7fe00713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49688/0 (socket says 172.21.15.67:49688) 2024-04-17T10:33:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fdea00640 1 -- 172.21.15.67:0/2981871394 learned_addr learned my addr 172.21.15.67:0/2981871394 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:12.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fdf400640 1 --2- 172.21.15.67:0/2981871394 >> v2:172.21.15.67:3300/0 conn(0x7f7fe015c580 0x7f7fe00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fde000640 1 -- 172.21.15.67:0/2981871394 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204280672 0 0) 0x7f7fe015cb30 con 0x7f7fe0159440 2024-04-17T10:33:12.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fde000640 1 -- 172.21.15.67:0/2981871394 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc4003530 con 0x7f7fe0159440 2024-04-17T10:33:12.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fdf400640 1 -- 172.21.15.67:0/2981871394 >> v1:172.21.15.86:6789/0 conn(0x7f7fe0071af0 legacy=0x7f7fe015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:12.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fdf400640 1 -- 172.21.15.67:0/2981871394 >> v1:172.21.15.67:6789/0 conn(0x7f7fe0159440 legacy=0x7f7fe00713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.874+0000 7f7fdf400640 1 -- 172.21.15.67:0/2981871394 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00cd8f0 con 0x7f7fe015c580 2024-04-17T10:33:12.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fdf400640 1 --2- 172.21.15.67:0/2981871394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe015c580 0x7f7fe00ce0f0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009d90 tx=0x7f7fd801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd9db3c922d6c22 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fde000640 1 -- 172.21.15.67:0/2981871394 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80b2070 con 0x7f7fe015c580 2024-04-17T10:33:12.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fde000640 1 -- 172.21.15.67:0/2981871394 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd80a5940 con 0x7f7fe015c580 2024-04-17T10:33:12.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fde000640 1 -- 172.21.15.67:0/2981871394 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd80a5d10 con 0x7f7fe015c580 2024-04-17T10:33:12.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2981871394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe015c580 msgr2=0x7f7fe00ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2981871394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe015c580 0x7f7fe00ce0f0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f7fd8009d90 tx=0x7f7fd801c970 comp rx=0 tx=0).stop 2024-04-17T10:33:12.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2981871394 shutdown_connections 2024-04-17T10:33:12.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2981871394 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe015c580 0x7f7fe00ce0f0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2981871394 >> 172.21.15.67:0/2981871394 conn(0x7f7fe00b1740 msgr2=0x7f7fe00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:12.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2981871394 shutdown_connections 2024-04-17T10:33:12.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.878+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2981871394 wait complete. 2024-04-17T10:33:12.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 Processor -- start 2024-04-17T10:33:12.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 -- start start 2024-04-17T10:33:12.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe0071af0 0x7f7fe00ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe0159440 0x7f7fe006aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe015c580 0x7f7fe014bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe015dc10 con 0x7f7fe0071af0 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe015da90 con 0x7f7fe015c580 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe015dd90 con 0x7f7fe0159440 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe0071af0 0x7f7fe00ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe0071af0 0x7f7fe00ae0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57290/0 (socket says 172.21.15.67:57290) 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 -- 172.21.15.67:0/2449649434 learned_addr learned my addr 172.21.15.67:0/2449649434 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdf400640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe015c580 0x7f7fe014bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fd6000640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe0159440 0x7f7fe006aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 -- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe0159440 msgr2=0x7f7fe006aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe0159440 0x7f7fe006aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 -- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe015c580 msgr2=0x7f7fe014bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe015c580 0x7f7fe014bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 -- 172.21.15.67:0/2449649434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00b2fc0 con 0x7f7fe0071af0 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fdea00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe0071af0 0x7f7fe00ae0b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c420 tx=0x7f7fd000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fd7e00640 1 -- 172.21.15.67:0/2449649434 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0017070 con 0x7f7fe0071af0 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fd7e00640 1 -- 172.21.15.67:0/2449649434 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0005cf0 con 0x7f7fe0071af0 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe00b32a0 con 0x7f7fe0071af0 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fd7e00640 1 -- 172.21.15.67:0/2449649434 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd00072d0 con 0x7f7fe0071af0 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe00b3860 con 0x7f7fe0071af0 2024-04-17T10:33:12.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.882+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe004ef30 con 0x7f7fe0071af0 2024-04-17T10:33:12.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.886+0000 7f7fd7e00640 1 -- 172.21.15.67:0/2449649434 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7fd001a070 con 0x7f7fe0071af0 2024-04-17T10:33:12.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.886+0000 7f7fd7e00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f60 0x7f7fbc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:12.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.886+0000 7f7fd7e00640 1 -- 172.21.15.67:0/2449649434 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7fd00845b0 con 0x7f7fe0071af0 2024-04-17T10:33:12.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.886+0000 7f7fd6000640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f60 0x7f7fbc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:12.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.886+0000 7f7fd6000640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f60 0x7f7fbc069420 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7fe01f9b50 tx=0x7f7fc8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:12.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:12.890+0000 7f7fd7e00640 1 -- 172.21.15.67:0/2449649434 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7fd0054d40 con 0x7f7fe0071af0 2024-04-17T10:33:13.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.050+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7fe006c3e0 con 0x7f7fe0071af0 2024-04-17T10:33:13.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.050+0000 7f7fd7e00640 1 -- 172.21.15.67:0/2449649434 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7fd0058de0 con 0x7f7fe0071af0 2024-04-17T10:33:13.043 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:33:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f60 msgr2=0x7f7fbc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f60 0x7f7fbc069420 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7fe01f9b50 tx=0x7f7fc8009040 comp rx=0 tx=0).stop 2024-04-17T10:33:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe0071af0 msgr2=0x7f7fe00ae0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe0071af0 0x7f7fe00ae0b0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c420 tx=0x7f7fd000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 shutdown_connections 2024-04-17T10:33:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f60 0x7f7fbc069420 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fe015c580 0x7f7fe014bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fe0159440 0x7f7fe006aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 --2- 172.21.15.67:0/2449649434 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fe0071af0 0x7f7fe00ae0b0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 >> 172.21.15.67:0/2449649434 conn(0x7f7fe00b1740 msgr2=0x7f7fe00afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:13.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 shutdown_connections 2024-04-17T10:33:13.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.054+0000 7f7fe5e00640 1 -- 172.21.15.67:0/2449649434 wait complete. 2024-04-17T10:33:13.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:33:13.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.214+0000 7f767d800640 1 Processor -- start 2024-04-17T10:33:13.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.214+0000 7f767d800640 1 -- start start 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f767d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7678161160 0x7f7678099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f767d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7678099f90 0x7f7678161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f767d800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76780d0760 con 0x7f76780b2570 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f767d800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f767809a400 con 0x7f7678161160 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f767d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76780cfc70 con 0x7f7678099f90 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f7677e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7678161160 0x7f7678099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f7677e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7678161160 0x7f7678099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57304/0 (socket says 172.21.15.67:57304) 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f7677e00640 1 -- 172.21.15.67:0/1356103121 learned_addr learned my addr 172.21.15.67:0/1356103121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f767ce00640 1 --2- 172.21.15.67:0/1356103121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7678099f90 0x7f7678161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:13.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f7677e00640 1 -- 172.21.15.67:0/1356103121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7678099f90 msgr2=0x7f7678161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:13.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f7677e00640 1 --2- 172.21.15.67:0/1356103121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7678099f90 0x7f7678161dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f7677e00640 1 -- 172.21.15.67:0/1356103121 >> v1:172.21.15.86:6789/0 conn(0x7f76780b2570 legacy=0x7f76781616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.218+0000 7f7677e00640 1 -- 172.21.15.67:0/1356103121 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76780cc2a0 con 0x7f7678161160 2024-04-17T10:33:13.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f7677e00640 1 --2- 172.21.15.67:0/1356103121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f7678099a50 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f7668009870 tx=0x7f766801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484c02dfcb7283dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:13.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f7677400640 1 -- 172.21.15.67:0/1356103121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76680a7070 con 0x7f7678161160 2024-04-17T10:33:13.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f7677400640 1 -- 172.21.15.67:0/1356103121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76680b0070 con 0x7f7678161160 2024-04-17T10:33:13.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f7677400640 1 -- 172.21.15.67:0/1356103121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76680ab890 con 0x7f7678161160 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 -- 172.21.15.67:0/1356103121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 msgr2=0x7f7678099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 --2- 172.21.15.67:0/1356103121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f7678099a50 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f7668009870 tx=0x7f766801c560 comp rx=0 tx=0).stop 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 -- 172.21.15.67:0/1356103121 shutdown_connections 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 --2- 172.21.15.67:0/1356103121 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7678099f90 0x7f7678161dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 --2- 172.21.15.67:0/1356103121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f7678099a50 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 -- 172.21.15.67:0/1356103121 >> 172.21.15.67:0/1356103121 conn(0x7f76780d1690 msgr2=0x7f76780d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 -- 172.21.15.67:0/1356103121 shutdown_connections 2024-04-17T10:33:13.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 -- 172.21.15.67:0/1356103121 wait complete. 2024-04-17T10:33:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 Processor -- start 2024-04-17T10:33:13.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 -- start start 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7678099f90 0x7f76780c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76780b2570 0x7f76780b6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.222+0000 7f767d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f76780c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76780b2bc0 con 0x7f7678161160 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767d800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76780cc5d0 con 0x7f7678099f90 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f767806bda0 con 0x7f76780b2570 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f76780c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f76780c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57306/0 (socket says 172.21.15.67:57306) 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 -- 172.21.15.67:0/347214795 learned_addr learned my addr 172.21.15.67:0/347214795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f766f400640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76780b2570 0x7f76780b6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:13.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f7677e00640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7678099f90 0x7f76780c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 -- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76780b2570 msgr2=0x7f76780b6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76780b2570 0x7f76780b6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 -- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7678099f90 msgr2=0x7f76780c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7678099f90 0x7f76780c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 -- 172.21.15.67:0/347214795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76780c56c0 con 0x7f7678161160 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767ce00640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f76780c8760 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f7670007e40 tx=0x7f767000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f7674c00640 1 -- 172.21.15.67:0/347214795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670014620 con 0x7f7678161160 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f7674c00640 1 -- 172.21.15.67:0/347214795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7670019070 con 0x7f7678161160 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f7674c00640 1 -- 172.21.15.67:0/347214795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7670014a20 con 0x7f7678161160 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76780c59a0 con 0x7f7678161160 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76780c5f60 con 0x7f7678161160 2024-04-17T10:33:13.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7678074dc0 con 0x7f7678161160 2024-04-17T10:33:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.226+0000 7f7674c00640 1 -- 172.21.15.67:0/347214795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7670014bc0 con 0x7f7678161160 2024-04-17T10:33:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.230+0000 7f7674c00640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7654066f90 0x7f7654069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:13.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.230+0000 7f7674c00640 1 -- 172.21.15.67:0/347214795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f76700859a0 con 0x7f7678161160 2024-04-17T10:33:13.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.230+0000 7f7677e00640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7654066f90 0x7f7654069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:13.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.230+0000 7f7677e00640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7654066f90 0x7f7654069450 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f76680af000 tx=0x7f76680a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:13.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.234+0000 7f7674c00640 1 -- 172.21.15.67:0/347214795 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7670056130 con 0x7f7678161160 2024-04-17T10:33:13.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f767809ac60 con 0x7f7678161160 2024-04-17T10:33:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f7674c00640 1 -- 172.21.15.67:0/347214795 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) v1 ==== 80+0+0 (secure 0 0 0) 0x7f767005a1d0 con 0x7f7678161160 2024-04-17T10:33:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7654066f90 msgr2=0x7f7654069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7654066f90 0x7f7654069450 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f76680af000 tx=0x7f76680a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 msgr2=0x7f76780c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f76780c8760 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f7670007e40 tx=0x7f767000e4b0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 shutdown_connections 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7654066f90 0x7f7654069450 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7678161160 0x7f76780c8760 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76780b2570 0x7f76780b6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 --2- 172.21.15.67:0/347214795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7678099f90 0x7f76780c6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 >> 172.21.15.67:0/347214795 conn(0x7f76780d1690 msgr2=0x7f7678155f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 shutdown_connections 2024-04-17T10:33:13.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:13.406+0000 7f767d800640 1 -- 172.21.15.67:0/347214795 wait complete. 2024-04-17T10:33:13.418 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:33:13.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:13.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:33:14.164 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:33:14.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:14.165 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:33:14.174 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:33:14.174 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:14.184 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:33:14.184 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:33:14.184 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:14.227 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:33:14.227 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:33:14.227 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:14.236 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:33:14.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:14.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde0be00640 1 Processor -- start 2024-04-17T10:33:14.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde0be00640 1 -- start start 2024-04-17T10:33:14.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde0be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde041620a0 0x7fde0415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde0be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde0415e550 0x7fde040cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde0be00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0415eb50 con 0x7fde040ce020 2024-04-17T10:33:14.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde0be00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde04161650 con 0x7fde041620a0 2024-04-17T10:33:14.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde0be00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0415ee30 con 0x7fde0415e550 2024-04-17T10:33:14.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde0415e550 0x7fde040cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde08a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fde041620a0 0x7fde0415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde0415e550 0x7fde040cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:36880/0 (socket says 172.21.15.67:36880) 2024-04-17T10:33:14.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09e00640 1 -- 172.21.15.67:0/2907008077 learned_addr learned my addr 172.21.15.67:0/2907008077 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:14.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09400640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fde040ce020 0x7fde040ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:50562/0 (socket says 172.21.15.67:50562) 2024-04-17T10:33:14.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09e00640 1 -- 172.21.15.67:0/2907008077 >> v2:172.21.15.86:3300/0 conn(0x7fde041620a0 msgr2=0x7fde0415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:14.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09e00640 1 --2- 172.21.15.67:0/2907008077 >> v2:172.21.15.86:3300/0 conn(0x7fde041620a0 0x7fde0415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09e00640 1 -- 172.21.15.67:0/2907008077 >> v1:172.21.15.86:6789/0 conn(0x7fde040ce020 legacy=0x7fde040ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde09e00640 1 -- 172.21.15.67:0/2907008077 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde041618a0 con 0x7fde0415e550 2024-04-17T10:33:14.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.422+0000 7fde08a00640 1 --2- 172.21.15.67:0/2907008077 >> v2:172.21.15.86:3300/0 conn(0x7fde041620a0 0x7fde0415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:14.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde09e00640 1 --2- 172.21.15.67:0/2907008077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde0415e550 0x7fde040cf510 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e80 tx=0x7fddfc01c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95304a0655d91df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:14.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde03e00640 1 -- 172.21.15.67:0/2907008077 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0a58d0 con 0x7fde0415e550 2024-04-17T10:33:14.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde03e00640 1 -- 172.21.15.67:0/2907008077 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc0a5a70 con 0x7fde0415e550 2024-04-17T10:33:14.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde03e00640 1 -- 172.21.15.67:0/2907008077 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc0a5e40 con 0x7fde0415e550 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 -- 172.21.15.67:0/2907008077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde0415e550 msgr2=0x7fde040cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 --2- 172.21.15.67:0/2907008077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde0415e550 0x7fde040cf510 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009e80 tx=0x7fddfc01c9e0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 -- 172.21.15.67:0/2907008077 shutdown_connections 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 --2- 172.21.15.67:0/2907008077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde0415e550 0x7fde040cf510 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 --2- 172.21.15.67:0/2907008077 >> v2:172.21.15.86:3300/0 conn(0x7fde041620a0 0x7fde0415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 -- 172.21.15.67:0/2907008077 >> 172.21.15.67:0/2907008077 conn(0x7fde040b33f0 msgr2=0x7fde040b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 -- 172.21.15.67:0/2907008077 shutdown_connections 2024-04-17T10:33:14.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.426+0000 7fde0be00640 1 -- 172.21.15.67:0/2907008077 wait complete. 2024-04-17T10:33:14.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 Processor -- start 2024-04-17T10:33:14.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 -- start start 2024-04-17T10:33:14.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde040ce020 0x7fde040b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0415e550 0x7fde041f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde041620a0 0x7fde041f7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde040c8630 con 0x7fde0415e550 2024-04-17T10:33:14.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde040c84b0 con 0x7fde041620a0 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0416a9f0 con 0x7fde040ce020 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0415e550 0x7fde041f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0415e550 0x7fde041f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57308/0 (socket says 172.21.15.67:57308) 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 -- 172.21.15.67:0/812463920 learned_addr learned my addr 172.21.15.67:0/812463920 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde09400640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde040ce020 0x7fde040b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde09e00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde041620a0 0x7fde041f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 -- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde040ce020 msgr2=0x7fde040b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde040ce020 0x7fde040b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 -- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde041620a0 msgr2=0x7fde041f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde041620a0 0x7fde041f7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 -- 172.21.15.67:0/812463920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde040b1c90 con 0x7fde0415e550 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde09e00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde041620a0 0x7fde041f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:14.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde08a00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0415e550 0x7fde041f70f0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fddf000e980 tx=0x7fddf000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde01600640 1 -- 172.21.15.67:0/812463920 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0017070 con 0x7fde0415e550 2024-04-17T10:33:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde01600640 1 -- 172.21.15.67:0/812463920 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf000ca00 con 0x7fde0415e550 2024-04-17T10:33:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde01600640 1 -- 172.21.15.67:0/812463920 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf000cdd0 con 0x7fde0415e550 2024-04-17T10:33:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde040b1f70 con 0x7fde0415e550 2024-04-17T10:33:14.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.430+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde040b2530 con 0x7fde0415e550 2024-04-17T10:33:14.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.434+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde04070ab0 con 0x7fde0415e550 2024-04-17T10:33:14.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.434+0000 7fde01600640 1 -- 172.21.15.67:0/812463920 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fddf001a070 con 0x7fde0415e550 2024-04-17T10:33:14.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.434+0000 7fde01600640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdde4066f60 0x7fdde4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.434+0000 7fde01600640 1 -- 172.21.15.67:0/812463920 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fddf00853d0 con 0x7fde0415e550 2024-04-17T10:33:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.434+0000 7fde09400640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdde4066f60 0x7fdde4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.434+0000 7fde09400640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdde4066f60 0x7fdde4069420 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fde04159440 tx=0x7fddf8007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:14.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.438+0000 7fde01600640 1 -- 172.21.15.67:0/812463920 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fddf0055b60 con 0x7fde0415e550 2024-04-17T10:33:14.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde040c7a20 con 0x7fde0415e550 2024-04-17T10:33:14.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde01600640 1 -- 172.21.15.67:0/812463920 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7fddf0059c00 con 0x7fde0415e550 2024-04-17T10:33:14.625 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:14.625 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:14.626 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:14.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdde4066f60 msgr2=0x7fdde4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdde4066f60 0x7fdde4069420 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fde04159440 tx=0x7fddf8007e90 comp rx=0 tx=0).stop 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0415e550 msgr2=0x7fde041f70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0415e550 0x7fde041f70f0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7fddf000e980 tx=0x7fddf000ee40 comp rx=0 tx=0).stop 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 shutdown_connections 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdde4066f60 0x7fdde4069420 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fde041620a0 0x7fde041f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fde0415e550 0x7fde041f70f0 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 --2- 172.21.15.67:0/812463920 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fde040ce020 0x7fde040b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.634+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 >> 172.21.15.67:0/812463920 conn(0x7fde040b33f0 msgr2=0x7fde0415f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.638+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 shutdown_connections 2024-04-17T10:33:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.638+0000 7fde0be00640 1 -- 172.21.15.67:0/812463920 wait complete. 2024-04-17T10:33:14.647 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:14.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.794+0000 7f210f600640 1 Processor -- start 2024-04-17T10:33:14.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.794+0000 7f210f600640 1 -- start start 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210f600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f210806e6e0 0x7f21080ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210f600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f210806eac0 0x7f21080ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210f600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210815ae40 con 0x7f2108143ae0 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210f600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21080cf430 con 0x7f210806e6e0 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210f600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2108071440 con 0x7f210806eac0 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210cc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f210806e6e0 0x7f21080ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210cc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f210806e6e0 0x7f21080ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:57862/0 (socket says 172.21.15.67:57862) 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210cc00640 1 -- 172.21.15.67:0/3237816671 learned_addr learned my addr 172.21.15.67:0/3237816671 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:14.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f2107e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f210806eac0 0x7f21080ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210d600640 1 --1- 172.21.15.67:0/3237816671 >> v1:172.21.15.86:6789/0 conn(0x7f2108143ae0 0x7f2108143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:50578/0 (socket says 172.21.15.67:50578) 2024-04-17T10:33:14.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210cc00640 1 -- 172.21.15.67:0/3237816671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f210806eac0 msgr2=0x7f21080ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:14.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210cc00640 1 --2- 172.21.15.67:0/3237816671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f210806eac0 0x7f21080ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210cc00640 1 -- 172.21.15.67:0/3237816671 >> v1:172.21.15.86:6789/0 conn(0x7f2108143ae0 legacy=0x7f2108143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.798+0000 7f210cc00640 1 -- 172.21.15.67:0/3237816671 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2108071720 con 0x7f210806e6e0 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210cc00640 1 --2- 172.21.15.67:0/3237816671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806e6e0 0x7f21080ce9b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00bb10 tx=0x7f20fc01eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac338e9da8e48fda server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f2107400640 1 -- 172.21.15.67:0/3237816671 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0b4070 con 0x7f210806e6e0 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f2107400640 1 -- 172.21.15.67:0/3237816671 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc0a7b40 con 0x7f210806e6e0 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f2107400640 1 -- 172.21.15.67:0/3237816671 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0af440 con 0x7f210806e6e0 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 -- 172.21.15.67:0/3237816671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806e6e0 msgr2=0x7f21080ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 --2- 172.21.15.67:0/3237816671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806e6e0 0x7f21080ce9b0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00bb10 tx=0x7f20fc01eb70 comp rx=0 tx=0).stop 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 -- 172.21.15.67:0/3237816671 shutdown_connections 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 --2- 172.21.15.67:0/3237816671 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f210806eac0 0x7f21080ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 --2- 172.21.15.67:0/3237816671 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806e6e0 0x7f21080ce9b0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 -- 172.21.15.67:0/3237816671 >> 172.21.15.67:0/3237816671 conn(0x7f210806ac90 msgr2=0x7f210806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 -- 172.21.15.67:0/3237816671 shutdown_connections 2024-04-17T10:33:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 -- 172.21.15.67:0/3237816671 wait complete. 2024-04-17T10:33:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 Processor -- start 2024-04-17T10:33:14.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.802+0000 7f210f600640 1 -- start start 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e6e0 0x7f21080ab430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806eac0 0x7f2108154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2108143ae0 0x7f21080acd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210806dd00 con 0x7f210806e6e0 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f210806db80 con 0x7f210806eac0 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210806de80 con 0x7f2108143ae0 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e6e0 0x7f21080ab430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e6e0 0x7f21080ab430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57316/0 (socket says 172.21.15.67:57316) 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 -- 172.21.15.67:0/1310325036 learned_addr learned my addr 172.21.15.67:0/1310325036 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210d600640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2108143ae0 0x7f21080acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f2107e00640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806eac0 0x7f2108154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 -- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2108143ae0 msgr2=0x7f21080acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2108143ae0 0x7f21080acd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 -- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806eac0 msgr2=0x7f2108154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806eac0 0x7f2108154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 -- 172.21.15.67:0/1310325036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21080ade90 con 0x7f210806e6e0 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210cc00640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e6e0 0x7f21080ab430 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00b2c0 tx=0x7f20fc01b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f2104c00640 1 -- 172.21.15.67:0/1310325036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0be070 con 0x7f210806e6e0 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f2104c00640 1 -- 172.21.15.67:0/1310325036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc0afc10 con 0x7f210806e6e0 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2108154bd0 con 0x7f210806e6e0 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f2104c00640 1 -- 172.21.15.67:0/1310325036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc0b4040 con 0x7f210806e6e0 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21080ad8d0 con 0x7f210806e6e0 2024-04-17T10:33:14.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2108072d50 con 0x7f210806e6e0 2024-04-17T10:33:14.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.806+0000 7f2104c00640 1 -- 172.21.15.67:0/1310325036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f20fc0c1070 con 0x7f210806e6e0 2024-04-17T10:33:14.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.810+0000 7f2104c00640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20e8066f60 0x7f20e8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:14.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.810+0000 7f2104c00640 1 -- 172.21.15.67:0/1310325036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f20fc1224e0 con 0x7f210806e6e0 2024-04-17T10:33:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.810+0000 7f2107e00640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20e8066f60 0x7f20e8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:14.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.810+0000 7f2107e00640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20e8066f60 0x7f20e8069420 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f210815a8b0 tx=0x7f20f4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:14.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.810+0000 7f2104c00640 1 -- 172.21.15.67:0/1310325036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f20fc018820 con 0x7f210806e6e0 2024-04-17T10:33:14.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.974+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21080a9170 con 0x7f210806e6e0 2024-04-17T10:33:14.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.974+0000 7f2104c00640 1 -- 172.21.15.67:0/1310325036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16836 (secure 0 0 0) 0x7f20fc0f2e20 con 0x7f210806e6e0 2024-04-17T10:33:14.967 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:14.967 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":107,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:34.925635+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:32:31.849728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-17T10:32:32.441190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20e8066f60 msgr2=0x7f20e8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20e8066f60 0x7f20e8069420 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f210815a8b0 tx=0x7f20f4009190 comp rx=0 tx=0).stop 2024-04-17T10:33:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e6e0 msgr2=0x7f21080ab430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e6e0 0x7f21080ab430 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f20fc00b2c0 tx=0x7f20fc01b910 comp rx=0 tx=0).stop 2024-04-17T10:33:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 shutdown_connections 2024-04-17T10:33:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f20e8066f60 0x7f20e8069420 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2108143ae0 0x7f21080acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f210806eac0 0x7f2108154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 --2- 172.21.15.67:0/1310325036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f210806e6e0 0x7f21080ab430 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 >> 172.21.15.67:0/1310325036 conn(0x7f210806ac90 msgr2=0x7f210813aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 shutdown_connections 2024-04-17T10:33:14.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:14.978+0000 7f210f600640 1 -- 172.21.15.67:0/1310325036 wait complete. 2024-04-17T10:33:14.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:32:31.849728+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '106', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:32:32.441190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '107', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:33:14.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:33:15.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c57e00640 1 Processor -- start 2024-04-17T10:33:15.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c57e00640 1 -- start start 2024-04-17T10:33:15.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c57e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0c5015dd60 0x7f0c50161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c57e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c5009adf0 0x7f0c50161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c57e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5015e140 con 0x7f0c500cff90 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c57e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c501626d0 con 0x7f0c5009adf0 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c57e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c50099d50 con 0x7f0c5015dd60 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c54a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0c5009adf0 0x7f0c50161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c55e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0c500cff90 0x7f0c500d0420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49704/0 (socket says 172.21.15.67:49704) 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c55e00640 1 -- 172.21.15.67:0/2208056850 learned_addr learned my addr 172.21.15.67:0/2208056850 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.142+0000 7f0c55400640 1 --2- 172.21.15.67:0/2208056850 >> v2:172.21.15.86:3300/0 conn(0x7f0c5015dd60 0x7f0c50161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c54a00640 1 -- 172.21.15.67:0/2208056850 >> v2:172.21.15.86:3300/0 conn(0x7f0c5015dd60 msgr2=0x7f0c50161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c54a00640 1 --2- 172.21.15.67:0/2208056850 >> v2:172.21.15.86:3300/0 conn(0x7f0c5015dd60 0x7f0c50161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c54a00640 1 -- 172.21.15.67:0/2208056850 >> v1:172.21.15.67:6789/0 conn(0x7f0c500cff90 legacy=0x7f0c500d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c54a00640 1 -- 172.21.15.67:0/2208056850 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5009a030 con 0x7f0c5009adf0 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c55400640 1 --2- 172.21.15.67:0/2208056850 >> v2:172.21.15.86:3300/0 conn(0x7f0c5015dd60 0x7f0c50161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:15.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c4fe00640 1 -- 172.21.15.67:0/2208056850 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762448382 0 0) 0x7f0c5015e140 con 0x7f0c500cff90 2024-04-17T10:33:15.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c54a00640 1 --2- 172.21.15.67:0/2208056850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50161dc0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0c5006f620 tx=0x7f0c3c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75944c0cdfca639c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c4fe00640 1 -- 172.21.15.67:0/2208056850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c0ab650 con 0x7f0c5009adf0 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c4fe00640 1 -- 172.21.15.67:0/2208056850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3c0b0070 con 0x7f0c5009adf0 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c4fe00640 1 -- 172.21.15.67:0/2208056850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c0aba10 con 0x7f0c5009adf0 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c57e00640 1 -- 172.21.15.67:0/2208056850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 msgr2=0x7f0c50161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c57e00640 1 --2- 172.21.15.67:0/2208056850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50161dc0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0c5006f620 tx=0x7f0c3c01c450 comp rx=0 tx=0).stop 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c57e00640 1 -- 172.21.15.67:0/2208056850 shutdown_connections 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c57e00640 1 --2- 172.21.15.67:0/2208056850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50161dc0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c57e00640 1 --2- 172.21.15.67:0/2208056850 >> v2:172.21.15.86:3300/0 conn(0x7f0c5015dd60 0x7f0c50161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c57e00640 1 -- 172.21.15.67:0/2208056850 >> 172.21.15.67:0/2208056850 conn(0x7f0c500b34a0 msgr2=0x7f0c500b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.146+0000 7f0c57e00640 1 -- 172.21.15.67:0/2208056850 shutdown_connections 2024-04-17T10:33:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 -- 172.21.15.67:0/2208056850 wait complete. 2024-04-17T10:33:15.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 Processor -- start 2024-04-17T10:33:15.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 -- start start 2024-04-17T10:33:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50159ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c500cff90 0x7f0c50157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c5015dd60 0x7f0c501602c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5006c730 con 0x7f0c5009adf0 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5006c5b0 con 0x7f0c500cff90 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5006c8b0 con 0x7f0c5015dd60 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50159ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50159ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57320/0 (socket says 172.21.15.67:57320) 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 -- 172.21.15.67:0/4275050774 learned_addr learned my addr 172.21.15.67:0/4275050774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c54a00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c500cff90 0x7f0c50157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c5015dd60 0x7f0c501602c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 -- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c5015dd60 msgr2=0x7f0c501602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c5015dd60 0x7f0c501602c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 -- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c500cff90 msgr2=0x7f0c50157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c500cff90 0x7f0c50157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 -- 172.21.15.67:0/4275050774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c501594b0 con 0x7f0c5009adf0 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c54a00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c500cff90 0x7f0c50157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c5015dd60 0x7f0c501602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c55400640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50159ea0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400d830 tx=0x7f0c4400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c4d600640 1 -- 172.21.15.67:0/4275050774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44018020 con 0x7f0c5009adf0 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c4d600640 1 -- 172.21.15.67:0/4275050774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c440096c0 con 0x7f0c5009adf0 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c50158410 con 0x7f0c5009adf0 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c4d600640 1 -- 172.21.15.67:0/4275050774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44009ab0 con 0x7f0c5009adf0 2024-04-17T10:33:15.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.150+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c501589d0 con 0x7f0c5009adf0 2024-04-17T10:33:15.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.154+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c5004ef30 con 0x7f0c5009adf0 2024-04-17T10:33:15.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.154+0000 7f0c4d600640 1 -- 172.21.15.67:0/4275050774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0c4401b020 con 0x7f0c5009adf0 2024-04-17T10:33:15.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.154+0000 7f0c4d600640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c30066f60 0x7f0c30069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.154+0000 7f0c4d600640 1 -- 172.21.15.67:0/4275050774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0c44085960 con 0x7f0c5009adf0 2024-04-17T10:33:15.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.154+0000 7f0c54a00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c30066f60 0x7f0c30069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.154+0000 7f0c54a00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c30066f60 0x7f0c30069420 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0af000 tx=0x7f0c3c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.158+0000 7f0c4d600640 1 -- 172.21.15.67:0/4275050774 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0c44022020 con 0x7f0c5009adf0 2024-04-17T10:33:15.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.318+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c500b42d0 con 0x7f0c5009adf0 2024-04-17T10:33:15.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.318+0000 7f0c4d600640 1 -- 172.21.15.67:0/4275050774 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c440567d0 con 0x7f0c5009adf0 2024-04-17T10:33:15.311 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c30066f60 msgr2=0x7f0c30069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c30066f60 0x7f0c30069420 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0af000 tx=0x7f0c3c008040 comp rx=0 tx=0).stop 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 msgr2=0x7f0c50159ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50159ea0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400d830 tx=0x7f0c4400dcf0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 shutdown_connections 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0c30066f60 0x7f0c30069420 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0c5015dd60 0x7f0c501602c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c500cff90 0x7f0c50157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 --2- 172.21.15.67:0/4275050774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0c5009adf0 0x7f0c50159ea0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 >> 172.21.15.67:0/4275050774 conn(0x7f0c500b34a0 msgr2=0x7f0c500b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 shutdown_connections 2024-04-17T10:33:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.322+0000 7f0c57e00640 1 -- 172.21.15.67:0/4275050774 wait complete. 2024-04-17T10:33:15.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:33:15.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e87600640 1 Processor -- start 2024-04-17T10:33:15.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e87600640 1 -- start start 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e87600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4e800713e0 0x7f4e80162160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e87600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4e8015f200 0x7f4e8006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e87600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e8015f680 con 0x7f4e800717c0 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e87600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8006d6a0 con 0x7f4e8015f200 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e87600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e8006d8d0 con 0x7f4e800713e0 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e85600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4e800717c0 0x7f4e800ae4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49710/0 (socket says 172.21.15.67:49710) 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e85600640 1 -- 172.21.15.67:0/2394400045 learned_addr learned my addr 172.21.15.67:0/2394400045 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e77e00640 1 --2- 172.21.15.67:0/2394400045 >> v2:172.21.15.67:3300/0 conn(0x7f4e8015f200 0x7f4e8006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e84c00640 1 --2- 172.21.15.67:0/2394400045 >> v2:172.21.15.86:3300/0 conn(0x7f4e800713e0 0x7f4e80162160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e77e00640 1 -- 172.21.15.67:0/2394400045 >> v2:172.21.15.86:3300/0 conn(0x7f4e800713e0 msgr2=0x7f4e80162160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:15.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e77e00640 1 --2- 172.21.15.67:0/2394400045 >> v2:172.21.15.86:3300/0 conn(0x7f4e800713e0 0x7f4e80162160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e77e00640 1 -- 172.21.15.67:0/2394400045 >> v1:172.21.15.67:6789/0 conn(0x7f4e800717c0 legacy=0x7f4e800ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e77e00640 1 -- 172.21.15.67:0/2394400045 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8006dbb0 con 0x7f4e8015f200 2024-04-17T10:33:15.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.486+0000 7f4e84c00640 1 --2- 172.21.15.67:0/2394400045 >> v2:172.21.15.86:3300/0 conn(0x7f4e800713e0 0x7f4e80162160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e77e00640 1 --2- 172.21.15.67:0/2394400045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8015f200 0x7f4e8006d160 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4e80072240 tx=0x7f4e6801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e05cb54a4d0fd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e7fe00640 1 -- 172.21.15.67:0/2394400045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680ab650 con 0x7f4e8015f200 2024-04-17T10:33:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e7fe00640 1 -- 172.21.15.67:0/2394400045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e680b0070 con 0x7f4e8015f200 2024-04-17T10:33:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e7fe00640 1 -- 172.21.15.67:0/2394400045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680aba10 con 0x7f4e8015f200 2024-04-17T10:33:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 -- 172.21.15.67:0/2394400045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8015f200 msgr2=0x7f4e8006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 --2- 172.21.15.67:0/2394400045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8015f200 0x7f4e8006d160 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4e80072240 tx=0x7f4e6801c450 comp rx=0 tx=0).stop 2024-04-17T10:33:15.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 -- 172.21.15.67:0/2394400045 shutdown_connections 2024-04-17T10:33:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 --2- 172.21.15.67:0/2394400045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e8015f200 0x7f4e8006d160 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 --2- 172.21.15.67:0/2394400045 >> v2:172.21.15.86:3300/0 conn(0x7f4e800713e0 0x7f4e80162160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 -- 172.21.15.67:0/2394400045 >> 172.21.15.67:0/2394400045 conn(0x7f4e800b1740 msgr2=0x7f4e800b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 -- 172.21.15.67:0/2394400045 shutdown_connections 2024-04-17T10:33:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.490+0000 7f4e87600640 1 -- 172.21.15.67:0/2394400045 wait complete. 2024-04-17T10:33:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 Processor -- start 2024-04-17T10:33:15.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 -- start start 2024-04-17T10:33:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e800713e0 0x7f4e8015d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e800717c0 0x7f4e800cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e8015f200 0x7f4e800ce510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80071d50 con 0x7f4e800717c0 2024-04-17T10:33:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80071bd0 con 0x7f4e800713e0 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e80147640 con 0x7f4e8015f200 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e800717c0 0x7f4e800cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e800717c0 0x7f4e800cdfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57340/0 (socket says 172.21.15.67:57340) 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 -- 172.21.15.67:0/4053488235 learned_addr learned my addr 172.21.15.67:0/4053488235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e84c00640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e800713e0 0x7f4e8015d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e85600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e8015f200 0x7f4e800ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 -- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e8015f200 msgr2=0x7f4e800ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e8015f200 0x7f4e800ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 -- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e800713e0 msgr2=0x7f4e8015d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e800713e0 0x7f4e8015d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 -- 172.21.15.67:0/4053488235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e80161d20 con 0x7f4e800717c0 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e77e00640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e800717c0 0x7f4e800cdfd0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f4e68019060 tx=0x7f4e680191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e7d600640 1 -- 172.21.15.67:0/4053488235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680abda0 con 0x7f4e800717c0 2024-04-17T10:33:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e7d600640 1 -- 172.21.15.67:0/4053488235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e680bf450 con 0x7f4e800717c0 2024-04-17T10:33:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e7d600640 1 -- 172.21.15.67:0/4053488235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680bf800 con 0x7f4e800717c0 2024-04-17T10:33:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e800ccf30 con 0x7f4e800717c0 2024-04-17T10:33:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e800cd4f0 con 0x7f4e800717c0 2024-04-17T10:33:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.494+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8004ef30 con 0x7f4e800717c0 2024-04-17T10:33:15.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.498+0000 7f4e7d600640 1 -- 172.21.15.67:0/4053488235 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4e680c83f0 con 0x7f4e800717c0 2024-04-17T10:33:15.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.498+0000 7f4e7d600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e5c066f90 0x7f4e5c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.498+0000 7f4e7d600640 1 -- 172.21.15.67:0/4053488235 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4e6811fbb0 con 0x7f4e800717c0 2024-04-17T10:33:15.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.498+0000 7f4e84c00640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e5c066f90 0x7f4e5c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.498+0000 7f4e84c00640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e5c066f90 0x7f4e5c069450 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4e700045f0 tx=0x7f4e70008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.502+0000 7f4e7d600640 1 -- 172.21.15.67:0/4053488235 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4e68016070 con 0x7f4e800717c0 2024-04-17T10:33:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4e8013d460 con 0x7f4e800717c0 2024-04-17T10:33:15.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e7d600640 1 -- 172.21.15.67:0/4053488235 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4e680f0b40 con 0x7f4e800717c0 2024-04-17T10:33:15.656 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:33:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e5c066f90 msgr2=0x7f4e5c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e5c066f90 0x7f4e5c069450 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4e700045f0 tx=0x7f4e70008040 comp rx=0 tx=0).stop 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e800717c0 msgr2=0x7f4e800cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e800717c0 0x7f4e800cdfd0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f4e68019060 tx=0x7f4e680191f0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 shutdown_connections 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e5c066f90 0x7f4e5c069450 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4e8015f200 0x7f4e800ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e800717c0 0x7f4e800cdfd0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 --2- 172.21.15.67:0/4053488235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4e800713e0 0x7f4e8015d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 >> 172.21.15.67:0/4053488235 conn(0x7f4e800b1740 msgr2=0x7f4e8006ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 shutdown_connections 2024-04-17T10:33:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.666+0000 7f4e87600640 1 -- 172.21.15.67:0/4053488235 wait complete. 2024-04-17T10:33:15.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:33:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.830+0000 7fe7ee800640 1 Processor -- start 2024-04-17T10:33:15.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.830+0000 7fe7ee800640 1 -- start start 2024-04-17T10:33:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.830+0000 7fe7ee800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 0x7fe7e80cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.830+0000 7fe7ee800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80b4990 con 0x7fe7e80cce60 2024-04-17T10:33:15.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.830+0000 7fe7ee800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e815ae90 con 0x7fe7e806e980 2024-04-17T10:33:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.830+0000 7fe7ee800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e80cd2c0 con 0x7fe7e80aef60 2024-04-17T10:33:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7dea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7e80cce60 0x7fe7e806e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49726/0 (socket says 172.21.15.67:49726) 2024-04-17T10:33:15.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7dea00640 1 -- 172.21.15.67:0/948765883 learned_addr learned my addr 172.21.15.67:0/948765883 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7400640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 0x7fe7e80cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396964608 0 0) 0x7fe7d0001120 con 0x7fe7e80cce60 2024-04-17T10:33:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7cc005740 con 0x7fe7e80cce60 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2558523665 0 0) 0x7fe7cc005740 con 0x7fe7e80cce60 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 msgr2=0x7fe7e80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 0x7fe7e80cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 >> v1:172.21.15.86:6789/0 conn(0x7fe7e806e980 legacy=0x7fe7e80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e806ee80 con 0x7fe7e80cce60 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fe7d0001850 con 0x7fe7e80cce60 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 >> v1:172.21.15.67:6789/0 conn(0x7fe7e80cce60 legacy=0x7fe7e806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7cc005c00 0x7fe7cc006000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc0065d0 0x7fe7cc006aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 0x7fe7cc0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc00dc90 con 0x7fe7cc005c00 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc00de10 con 0x7fe7cc0065d0 2024-04-17T10:33:15.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7cc00e010 con 0x7fe7e80aef60 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7400640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7cc005c00 0x7fe7cc006000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7dea00640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc0065d0 0x7fe7cc006aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7e00640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 0x7fe7cc0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7400640 1 -- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 msgr2=0x7fe7cc0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7400640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e80aef60 0x7fe7cc0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7400640 1 -- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc0065d0 msgr2=0x7fe7cc006aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7400640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc0065d0 0x7fe7cc006aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.834+0000 7fe7e7400640 1 -- 172.21.15.67:0/948765883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc005740 con 0x7fe7cc005c00 2024-04-17T10:33:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7e7400640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7cc005c00 0x7fe7cc006000 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009ae0 tx=0x7fe7d801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bac2422364087ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80096b0 con 0x7fe7cc005c00 2024-04-17T10:33:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d80a6450 con 0x7fe7cc005c00 2024-04-17T10:33:15.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7e6a00640 1 -- 172.21.15.67:0/948765883 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80a6840 con 0x7fe7cc005c00 2024-04-17T10:33:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 -- 172.21.15.67:0/948765883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7cc005c00 msgr2=0x7fe7cc006000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 --2- 172.21.15.67:0/948765883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7cc005c00 0x7fe7cc006000 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009ae0 tx=0x7fe7d801ca70 comp rx=0 tx=0).stop 2024-04-17T10:33:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7e7e00640 1 -- 172.21.15.67:0/948765883 reap_dead start 2024-04-17T10:33:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 -- 172.21.15.67:0/948765883 shutdown_connections 2024-04-17T10:33:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 -- 172.21.15.67:0/948765883 >> 172.21.15.67:0/948765883 conn(0x7fe7e80b1740 msgr2=0x7fe7e80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 -- 172.21.15.67:0/948765883 shutdown_connections 2024-04-17T10:33:15.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 -- 172.21.15.67:0/948765883 wait complete. 2024-04-17T10:33:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 Processor -- start 2024-04-17T10:33:15.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 -- start start 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc005c00 0x7fe7e8151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e813fef0 0x7fe7e813c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.838+0000 7fe7ee800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e815cfa0 0x7fe7e815d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7ee800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e806d160 con 0x7fe7e815cfa0 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7ee800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e80af340 con 0x7fe7cc005c00 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7ee800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e806d2e0 con 0x7fe7e813fef0 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e815cfa0 0x7fe7e815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e815cfa0 0x7fe7e815d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57362/0 (socket says 172.21.15.67:57362) 2024-04-17T10:33:15.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 -- 172.21.15.67:0/430913119 learned_addr learned my addr 172.21.15.67:0/430913119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7400640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc005c00 0x7fe7e8151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7dea00640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e813fef0 0x7fe7e813c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 -- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e813fef0 msgr2=0x7fe7e813c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e813fef0 0x7fe7e813c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 -- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc005c00 msgr2=0x7fe7e8151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc005c00 0x7fe7e8151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 -- 172.21.15.67:0/430913119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e8152fa0 con 0x7fe7e815cfa0 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7400640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc005c00 0x7fe7e8151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7dea00640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e813fef0 0x7fe7e813c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7e7e00640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e815cfa0 0x7fe7e815d450 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80b2e30 tx=0x7fe7e000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7dfe00640 1 -- 172.21.15.67:0/430913119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0003980 con 0x7fe7e815cfa0 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7dfe00640 1 -- 172.21.15.67:0/430913119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e0018070 con 0x7fe7e815cfa0 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7dfe00640 1 -- 172.21.15.67:0/430913119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0003d60 con 0x7fe7e815cfa0 2024-04-17T10:33:15.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e8140350 con 0x7fe7e815cfa0 2024-04-17T10:33:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e8152700 con 0x7fe7e815cfa0 2024-04-17T10:33:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e804ef30 con 0x7fe7e815cfa0 2024-04-17T10:33:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.842+0000 7fe7dfe00640 1 -- 172.21.15.67:0/430913119 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe7e00050d0 con 0x7fe7e815cfa0 2024-04-17T10:33:15.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.846+0000 7fe7dfe00640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7c4066f60 0x7fe7c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.846+0000 7fe7dfe00640 1 -- 172.21.15.67:0/430913119 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe7e00850d0 con 0x7fe7e815cfa0 2024-04-17T10:33:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.846+0000 7fe7e7400640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7c4066f60 0x7fe7c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:15.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.846+0000 7fe7e7400640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7c4066f60 0x7fe7c4069420 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80ad000 tx=0x7fe7d80046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:15.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:15.850+0000 7fe7dfe00640 1 -- 172.21.15.67:0/430913119 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe7e0055f30 con 0x7fe7e815cfa0 2024-04-17T10:33:16.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.010+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe7e80b0f90 con 0x7fe7e815cfa0 2024-04-17T10:33:16.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.010+0000 7fe7dfe00640 1 -- 172.21.15.67:0/430913119 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe7e0055830 con 0x7fe7e815cfa0 2024-04-17T10:33:16.002 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7c4066f60 msgr2=0x7fe7c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7c4066f60 0x7fe7c4069420 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80ad000 tx=0x7fe7d80046d0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e815cfa0 msgr2=0x7fe7e815d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e815cfa0 0x7fe7e815d450 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80b2e30 tx=0x7fe7e000eb70 comp rx=0 tx=0).stop 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 shutdown_connections 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7c4066f60 0x7fe7c4069420 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e815cfa0 0x7fe7e815d450 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7e813fef0 0x7fe7e813c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 --2- 172.21.15.67:0/430913119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7cc005c00 0x7fe7e8151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 >> 172.21.15.67:0/430913119 conn(0x7fe7e80b1740 msgr2=0x7fe7e80afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:16.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 shutdown_connections 2024-04-17T10:33:16.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.014+0000 7fe7ee800640 1 -- 172.21.15.67:0/430913119 wait complete. 2024-04-17T10:33:16.023 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:16.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2fc00640 1 Processor -- start 2024-04-17T10:33:16.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2fc00640 1 -- start start 2024-04-17T10:33:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2fc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe280cff90 0x7efe280ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2fc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2815cf10 con 0x7efe2806b6a0 2024-04-17T10:33:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2fc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2815c300 con 0x7efe2815b970 2024-04-17T10:33:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2fc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2815c5e0 con 0x7efe280cff90 2024-04-17T10:33:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe280cff90 0x7efe280ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe1fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efe2806b6a0 0x7efe280cc680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49736/0 (socket says 172.21.15.67:49736) 2024-04-17T10:33:16.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe1fe00640 1 -- 172.21.15.67:0/1533379443 learned_addr learned my addr 172.21.15.67:0/1533379443 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2dc00640 1 -- 172.21.15.67:0/1533379443 >> v1:172.21.15.86:6789/0 conn(0x7efe2815b970 legacy=0x7efe280cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2dc00640 1 -- 172.21.15.67:0/1533379443 >> v1:172.21.15.67:6789/0 conn(0x7efe2806b6a0 legacy=0x7efe280cc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.174+0000 7efe2dc00640 1 -- 172.21.15.67:0/1533379443 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2815c8c0 con 0x7efe280cff90 2024-04-17T10:33:16.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2dc00640 1 --2- 172.21.15.67:0/1533379443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280ccd90 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7efe20009d90 tx=0x7efe280cdae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93161d7e557ea47c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe27e00640 1 -- 172.21.15.67:0/1533379443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe200b2070 con 0x7efe280cff90 2024-04-17T10:33:16.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe27e00640 1 -- 172.21.15.67:0/1533379443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe200a58b0 con 0x7efe280cff90 2024-04-17T10:33:16.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe27e00640 1 -- 172.21.15.67:0/1533379443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe200a5c80 con 0x7efe280cff90 2024-04-17T10:33:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2fc00640 1 -- 172.21.15.67:0/1533379443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 msgr2=0x7efe280ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1533379443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280ccd90 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7efe20009d90 tx=0x7efe280cdae0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2fc00640 1 -- 172.21.15.67:0/1533379443 shutdown_connections 2024-04-17T10:33:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1533379443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280ccd90 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2fc00640 1 -- 172.21.15.67:0/1533379443 >> 172.21.15.67:0/1533379443 conn(0x7efe280b22a0 msgr2=0x7efe280b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2fc00640 1 -- 172.21.15.67:0/1533379443 shutdown_connections 2024-04-17T10:33:16.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.178+0000 7efe2fc00640 1 -- 172.21.15.67:0/1533379443 wait complete. 2024-04-17T10:33:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 Processor -- start 2024-04-17T10:33:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 -- start start 2024-04-17T10:33:16.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe2806b6a0 0x7efe28156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2815b970 0x7efe2814baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28169920 con 0x7efe280cff90 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe281697a0 con 0x7efe2806b6a0 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe28169aa0 con 0x7efe2815b970 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57378/0 (socket says 172.21.15.67:57378) 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 -- 172.21.15.67:0/1639812177 learned_addr learned my addr 172.21.15.67:0/1639812177 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2d200640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe2806b6a0 0x7efe28156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2dc00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2815b970 0x7efe2814baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 -- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2815b970 msgr2=0x7efe2814baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2815b970 0x7efe2814baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 -- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe2806b6a0 msgr2=0x7efe28156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe2806b6a0 0x7efe28156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 -- 172.21.15.67:0/1639812177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe280ae700 con 0x7efe280cff90 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe1fe00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280adf90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7efe1000c500 tx=0x7efe1000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe25600640 1 -- 172.21.15.67:0/1639812177 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10017070 con 0x7efe280cff90 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe25600640 1 -- 172.21.15.67:0/1639812177 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe10005cf0 con 0x7efe280cff90 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe280ae9e0 con 0x7efe280cff90 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe25600640 1 -- 172.21.15.67:0/1639812177 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe100072f0 con 0x7efe280cff90 2024-04-17T10:33:16.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.182+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe280aefa0 con 0x7efe280cff90 2024-04-17T10:33:16.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.186+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2809e900 con 0x7efe280cff90 2024-04-17T10:33:16.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.186+0000 7efe25600640 1 -- 172.21.15.67:0/1639812177 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efe1001a070 con 0x7efe280cff90 2024-04-17T10:33:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.186+0000 7efe25600640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe04066f90 0x7efe04069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.186+0000 7efe25600640 1 -- 172.21.15.67:0/1639812177 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7efe100844a0 con 0x7efe280cff90 2024-04-17T10:33:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.186+0000 7efe2d200640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe04066f90 0x7efe04069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.186+0000 7efe2d200640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe04066f90 0x7efe04069450 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7efe18007fd0 tx=0x7efe18009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.190+0000 7efe25600640 1 -- 172.21.15.67:0/1639812177 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efe10054cd0 con 0x7efe280cff90 2024-04-17T10:33:16.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.386+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe28074e30 con 0x7efe280cff90 2024-04-17T10:33:16.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.386+0000 7efe25600640 1 -- 172.21.15.67:0/1639812177 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7efe10058d70 con 0x7efe280cff90 2024-04-17T10:33:16.379 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:16.379 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:16.379 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:16.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe04066f90 msgr2=0x7efe04069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe04066f90 0x7efe04069450 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7efe18007fd0 tx=0x7efe18009040 comp rx=0 tx=0).stop 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 msgr2=0x7efe280adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280adf90 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7efe1000c500 tx=0x7efe1000c9c0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 shutdown_connections 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efe04066f90 0x7efe04069450 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efe2815b970 0x7efe2814baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe280cff90 0x7efe280adf90 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 --2- 172.21.15.67:0/1639812177 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efe2806b6a0 0x7efe28156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 >> 172.21.15.67:0/1639812177 conn(0x7efe280b22a0 msgr2=0x7efe280cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 shutdown_connections 2024-04-17T10:33:16.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.390+0000 7efe2fc00640 1 -- 172.21.15.67:0/1639812177 wait complete. 2024-04-17T10:33:16.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9322a00640 1 Processor -- start 2024-04-17T10:33:16.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9322a00640 1 -- start start 2024-04-17T10:33:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9322a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06e6e0 0x7f931c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9322a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f931c1441f0 0x7f931c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9322a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931c15ae40 con 0x7f931c06eac0 2024-04-17T10:33:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9322a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c0cf430 con 0x7f931c1441f0 2024-04-17T10:33:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9322a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c071440 con 0x7f931c06e6e0 2024-04-17T10:33:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9320a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f931c1441f0 0x7f931c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9313e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f931c06eac0 0x7f931c143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49744/0 (socket says 172.21.15.67:49744) 2024-04-17T10:33:16.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9313e00640 1 -- 172.21.15.67:0/2028735395 learned_addr learned my addr 172.21.15.67:0/2028735395 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9320a00640 1 -- 172.21.15.67:0/2028735395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06e6e0 msgr2=0x7f931c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f931be00640 1 --2- 172.21.15.67:0/2028735395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06e6e0 0x7f931c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f931b400640 1 -- 172.21.15.67:0/2028735395 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462163481 0 0) 0x7f931c15ae40 con 0x7f931c06eac0 2024-04-17T10:33:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9320a00640 1 --2- 172.21.15.67:0/2028735395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06e6e0 0x7f931c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9320a00640 1 -- 172.21.15.67:0/2028735395 >> v1:172.21.15.67:6789/0 conn(0x7f931c06eac0 legacy=0x7f931c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f9320a00640 1 -- 172.21.15.67:0/2028735395 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c071720 con 0x7f931c1441f0 2024-04-17T10:33:16.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.550+0000 7f931be00640 1 --2- 172.21.15.67:0/2028735395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06e6e0 0x7f931c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9320a00640 1 --2- 172.21.15.67:0/2028735395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0ceef0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9314009d90 tx=0x7f931401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95adb21ba65bf386 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f931b400640 1 -- 172.21.15.67:0/2028735395 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93140b2070 con 0x7f931c1441f0 2024-04-17T10:33:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f931b400640 1 -- 172.21.15.67:0/2028735395 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93140a5970 con 0x7f931c1441f0 2024-04-17T10:33:16.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f931b400640 1 -- 172.21.15.67:0/2028735395 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93140a5d40 con 0x7f931c1441f0 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 -- 172.21.15.67:0/2028735395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 msgr2=0x7f931c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 --2- 172.21.15.67:0/2028735395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0ceef0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f9314009d90 tx=0x7f931401c8b0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 -- 172.21.15.67:0/2028735395 shutdown_connections 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 --2- 172.21.15.67:0/2028735395 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0ceef0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 --2- 172.21.15.67:0/2028735395 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06e6e0 0x7f931c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 -- 172.21.15.67:0/2028735395 >> 172.21.15.67:0/2028735395 conn(0x7f931c06ac90 msgr2=0x7f931c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 -- 172.21.15.67:0/2028735395 shutdown_connections 2024-04-17T10:33:16.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.554+0000 7f9322a00640 1 -- 172.21.15.67:0/2028735395 wait complete. 2024-04-17T10:33:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 Processor -- start 2024-04-17T10:33:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 -- start start 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f931c06e6e0 0x7f931c0abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06eac0 0x7f931c13d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0aa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0b3a40 con 0x7f931c1441f0 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0b38c0 con 0x7f931c06e6e0 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0b3bc0 con 0x7f931c06eac0 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0aa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0aa8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57398/0 (socket says 172.21.15.67:57398) 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 -- 172.21.15.67:0/1021739355 learned_addr learned my addr 172.21.15.67:0/1021739355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9313e00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06eac0 0x7f931c13d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f931be00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f931c06e6e0 0x7f931c0abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 -- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06eac0 msgr2=0x7f931c13d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06eac0 0x7f931c13d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 -- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f931c06e6e0 msgr2=0x7f931c0abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f931c06e6e0 0x7f931c0abef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 -- 172.21.15.67:0/1021739355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0ac5f0 con 0x7f931c1441f0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9320a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0aa8f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f93140b0040 tx=0x7f9314019650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9318c00640 1 -- 172.21.15.67:0/1021739355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93140b2050 con 0x7f931c1441f0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9318c00640 1 -- 172.21.15.67:0/1021739355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93140b3040 con 0x7f931c1441f0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c0ac8d0 con 0x7f931c1441f0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9318c00640 1 -- 172.21.15.67:0/1021739355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93140ad930 con 0x7f931c1441f0 2024-04-17T10:33:16.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c0ace90 con 0x7f931c1441f0 2024-04-17T10:33:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.558+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c072d50 con 0x7f931c1441f0 2024-04-17T10:33:16.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.562+0000 7f9318c00640 1 -- 172.21.15.67:0/1021739355 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f93140c0020 con 0x7f931c1441f0 2024-04-17T10:33:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.562+0000 7f9318c00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92f8066f90 0x7f92f8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.562+0000 7f9318c00640 1 -- 172.21.15.67:0/1021739355 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f931411f970 con 0x7f931c1441f0 2024-04-17T10:33:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.562+0000 7f931be00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92f8066f90 0x7f92f8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.562+0000 7f931be00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92f8066f90 0x7f92f8069450 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f930c00a9a0 tx=0x7f930c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.566+0000 7f9318c00640 1 -- 172.21.15.67:0/1021739355 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f93140c7020 con 0x7f931c1441f0 2024-04-17T10:33:16.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.726+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f931c074e30 con 0x7f931c1441f0 2024-04-17T10:33:16.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9318c00640 1 -- 172.21.15.67:0/1021739355 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16836 (secure 0 0 0) 0x7f9314016560 con 0x7f931c1441f0 2024-04-17T10:33:16.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:16.720 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":107,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:34.925635+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:32:31.849728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-17T10:32:32.441190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:16.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92f8066f90 msgr2=0x7f92f8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92f8066f90 0x7f92f8069450 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f930c00a9a0 tx=0x7f930c008040 comp rx=0 tx=0).stop 2024-04-17T10:33:16.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 msgr2=0x7f931c0aa8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0aa8f0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f93140b0040 tx=0x7f9314019650 comp rx=0 tx=0).stop 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 shutdown_connections 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92f8066f90 0x7f92f8069450 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f931c1441f0 0x7f931c0aa8f0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f931c06eac0 0x7f931c13d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 --2- 172.21.15.67:0/1021739355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f931c06e6e0 0x7f931c0abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 >> 172.21.15.67:0/1021739355 conn(0x7f931c06ac90 msgr2=0x7f931c140870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 shutdown_connections 2024-04-17T10:33:16.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.730+0000 7f9322a00640 1 -- 172.21.15.67:0/1021739355 wait complete. 2024-04-17T10:33:16.743 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:33:16.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:16.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967c800640 1 Processor -- start 2024-04-17T10:33:16.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967c800640 1 -- start start 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967c800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f967406e6e0 0x7f96740ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96740ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967c800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f967415ae40 con 0x7f967406eac0 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967c800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96740cf430 con 0x7f967406e6e0 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9674071440 con 0x7f96741441f0 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f9679e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f967406e6e0 0x7f96740ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f9679e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f967406e6e0 0x7f96740ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57412/0 (socket says 172.21.15.67:57412) 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f9679e00640 1 -- 172.21.15.67:0/75568672 learned_addr learned my addr 172.21.15.67:0/75568672 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f967a800640 1 --2- 172.21.15.67:0/75568672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96740ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f9679e00640 1 -- 172.21.15.67:0/75568672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 msgr2=0x7f96740ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f9679e00640 1 --2- 172.21.15.67:0/75568672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96740ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f9679e00640 1 -- 172.21.15.67:0/75568672 >> v1:172.21.15.86:6789/0 conn(0x7f967406eac0 legacy=0x7f9674143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:16.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.894+0000 7f9679e00640 1 -- 172.21.15.67:0/75568672 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674071720 con 0x7f967406e6e0 2024-04-17T10:33:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f9679e00640 1 --2- 172.21.15.67:0/75568672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406e6e0 0x7f96740ce9b0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f966800bb10 tx=0x7f966801eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8f05a97f3d8c2d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f9678a00640 1 -- 172.21.15.67:0/75568672 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96680b4070 con 0x7f967406e6e0 2024-04-17T10:33:16.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f9678a00640 1 -- 172.21.15.67:0/75568672 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96680a7a50 con 0x7f967406e6e0 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f9678a00640 1 -- 172.21.15.67:0/75568672 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96680a7e20 con 0x7f967406e6e0 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 -- 172.21.15.67:0/75568672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406e6e0 msgr2=0x7f96740ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 --2- 172.21.15.67:0/75568672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406e6e0 0x7f96740ce9b0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f966800bb10 tx=0x7f966801eb70 comp rx=0 tx=0).stop 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 -- 172.21.15.67:0/75568672 shutdown_connections 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 --2- 172.21.15.67:0/75568672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96740ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 --2- 172.21.15.67:0/75568672 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406e6e0 0x7f96740ce9b0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 -- 172.21.15.67:0/75568672 >> 172.21.15.67:0/75568672 conn(0x7f967406ac90 msgr2=0x7f967406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 -- 172.21.15.67:0/75568672 shutdown_connections 2024-04-17T10:33:16.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 -- 172.21.15.67:0/75568672 wait complete. 2024-04-17T10:33:16.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 Processor -- start 2024-04-17T10:33:16.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.898+0000 7f967c800640 1 -- start start 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f967406e6e0 0x7f967415a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406eac0 0x7f96741393c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96741f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967406dd00 con 0x7f967406eac0 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967406db80 con 0x7f967406e6e0 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967406de80 con 0x7f96741441f0 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406eac0 0x7f96741393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f967406e6e0 0x7f967415a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406eac0 0x7f96741393c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57416/0 (socket says 172.21.15.67:57416) 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 -- 172.21.15.67:0/4147953997 learned_addr learned my addr 172.21.15.67:0/4147953997 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967a800640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96741f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 -- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 msgr2=0x7f96741f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96741f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 -- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f967406e6e0 msgr2=0x7f967415a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f967406e6e0 0x7f967415a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 -- 172.21.15.67:0/4147953997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9674138320 con 0x7f967406eac0 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f9679400640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406eac0 0x7f96741393c0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f966400c420 tx=0x7f966400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f966e000640 1 -- 172.21.15.67:0/4147953997 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96640052d0 con 0x7f967406eac0 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f966e000640 1 -- 172.21.15.67:0/4147953997 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9664005470 con 0x7f967406eac0 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f966e000640 1 -- 172.21.15.67:0/4147953997 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9664007260 con 0x7f967406eac0 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9674138600 con 0x7f967406eac0 2024-04-17T10:33:16.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9674138bc0 con 0x7f967406eac0 2024-04-17T10:33:16.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.902+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9674072d50 con 0x7f967406eac0 2024-04-17T10:33:16.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.906+0000 7f966e000640 1 -- 172.21.15.67:0/4147953997 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9664007400 con 0x7f967406eac0 2024-04-17T10:33:16.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.906+0000 7f966e000640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9658066f60 0x7f9658069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:16.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.906+0000 7f966e000640 1 -- 172.21.15.67:0/4147953997 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9664084610 con 0x7f967406eac0 2024-04-17T10:33:16.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.906+0000 7f9679e00640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9658066f60 0x7f9658069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:16.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.906+0000 7f9679e00640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9658066f60 0x7f9658069420 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f96680b1000 tx=0x7f96680a9070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:16.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:16.910+0000 7f966e000640 1 -- 172.21.15.67:0/4147953997 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9664054e30 con 0x7f967406eac0 2024-04-17T10:33:17.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9674074e30 con 0x7f967406eac0 2024-04-17T10:33:17.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f966e000640 1 -- 172.21.15.67:0/4147953997 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7f9664058ed0 con 0x7f967406eac0 2024-04-17T10:33:17.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:17.098 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9658066f60 msgr2=0x7f9658069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9658066f60 0x7f9658069420 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f96680b1000 tx=0x7f96680a9070 comp rx=0 tx=0).stop 2024-04-17T10:33:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406eac0 msgr2=0x7f96741393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406eac0 0x7f96741393c0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f966400c420 tx=0x7f966400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 shutdown_connections 2024-04-17T10:33:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9658066f60 0x7f9658069420 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f96741441f0 0x7f96741f6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f967406eac0 0x7f96741393c0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 --2- 172.21.15.67:0/4147953997 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f967406e6e0 0x7f967415a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 >> 172.21.15.67:0/4147953997 conn(0x7f967406ac90 msgr2=0x7f96740af550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 shutdown_connections 2024-04-17T10:33:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.106+0000 7f967c800640 1 -- 172.21.15.67:0/4147953997 wait complete. 2024-04-17T10:33:17.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:17.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdee5400640 1 Processor -- start 2024-04-17T10:33:17.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdee5400640 1 -- start start 2024-04-17T10:33:17.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdee5400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdee015dd10 0x7fdee00b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdee5400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdee00714d0 0x7fdee006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdee5400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee00b0cc0 con 0x7fdee00b57b0 2024-04-17T10:33:17.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdee5400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee00b1280 con 0x7fdee00714d0 2024-04-17T10:33:17.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdee5400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee00b0fa0 con 0x7fdee015dd10 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fded5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdee00b57b0 0x7fdee0070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49756/0 (socket says 172.21.15.67:49756) 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdedf200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdee00714d0 0x7fdee006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fded5e00640 1 -- 172.21.15.67:0/4040554336 learned_addr learned my addr 172.21.15.67:0/4040554336 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdede800640 1 --2- 172.21.15.67:0/4040554336 >> v2:172.21.15.86:3300/0 conn(0x7fdee015dd10 0x7fdee00b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdedf200640 1 -- 172.21.15.67:0/4040554336 >> v2:172.21.15.86:3300/0 conn(0x7fdee015dd10 msgr2=0x7fdee00b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdedf200640 1 --2- 172.21.15.67:0/4040554336 >> v2:172.21.15.86:3300/0 conn(0x7fdee015dd10 0x7fdee00b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdedf200640 1 -- 172.21.15.67:0/4040554336 >> v1:172.21.15.67:6789/0 conn(0x7fdee00b57b0 legacy=0x7fdee0070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.270+0000 7fdedf200640 1 -- 172.21.15.67:0/4040554336 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee006bed0 con 0x7fdee00714d0 2024-04-17T10:33:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdedf200640 1 --2- 172.21.15.67:0/4040554336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00714d0 0x7fdee006c6d0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fdee00d2900 tx=0x7fded801c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64872cc3bafcaa5f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdedde00640 1 -- 172.21.15.67:0/4040554336 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded80b2070 con 0x7fdee00714d0 2024-04-17T10:33:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdedde00640 1 -- 172.21.15.67:0/4040554336 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded80a5910 con 0x7fdee00714d0 2024-04-17T10:33:17.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdedde00640 1 -- 172.21.15.67:0/4040554336 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded80a5ce0 con 0x7fdee00714d0 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 -- 172.21.15.67:0/4040554336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00714d0 msgr2=0x7fdee006c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 --2- 172.21.15.67:0/4040554336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00714d0 0x7fdee006c6d0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fdee00d2900 tx=0x7fded801c870 comp rx=0 tx=0).stop 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 -- 172.21.15.67:0/4040554336 shutdown_connections 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 --2- 172.21.15.67:0/4040554336 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00714d0 0x7fdee006c6d0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 --2- 172.21.15.67:0/4040554336 >> v2:172.21.15.86:3300/0 conn(0x7fdee015dd10 0x7fdee00b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 -- 172.21.15.67:0/4040554336 >> 172.21.15.67:0/4040554336 conn(0x7fdee00b34d0 msgr2=0x7fdee00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 -- 172.21.15.67:0/4040554336 shutdown_connections 2024-04-17T10:33:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.274+0000 7fdee5400640 1 -- 172.21.15.67:0/4040554336 wait complete. 2024-04-17T10:33:17.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 Processor -- start 2024-04-17T10:33:17.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 -- start start 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdee00714d0 0x7fdee014e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee015dd10 0x7fdee013d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00d03a0 0x7fdee0157480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee006d7b0 con 0x7fdee00d03a0 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee006d630 con 0x7fdee015dd10 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee006d930 con 0x7fdee00714d0 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00d03a0 0x7fdee0157480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00d03a0 0x7fdee0157480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57436/0 (socket says 172.21.15.67:57436) 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 -- 172.21.15.67:0/3887231866 learned_addr learned my addr 172.21.15.67:0/3887231866 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:17.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdede800640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdee00714d0 0x7fdee014e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fded5e00640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee015dd10 0x7fdee013d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 -- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdee00714d0 msgr2=0x7fdee014e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdee00714d0 0x7fdee014e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 -- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee015dd10 msgr2=0x7fdee013d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee015dd10 0x7fdee013d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 -- 172.21.15.67:0/3887231866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee014d0d0 con 0x7fdee00d03a0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdedf200640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00d03a0 0x7fdee0157480 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fded8019480 tx=0x7fded8019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fded7400640 1 -- 172.21.15.67:0/3887231866 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded80b2050 con 0x7fdee00d03a0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fded7400640 1 -- 172.21.15.67:0/3887231866 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded80be070 con 0x7fdee00d03a0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee014d3b0 con 0x7fdee00d03a0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fded7400640 1 -- 172.21.15.67:0/3887231866 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded80ada70 con 0x7fdee00d03a0 2024-04-17T10:33:17.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.278+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee014d970 con 0x7fdee00d03a0 2024-04-17T10:33:17.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.282+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee014ce20 con 0x7fdee00d03a0 2024-04-17T10:33:17.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.282+0000 7fded7400640 1 -- 172.21.15.67:0/3887231866 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fded80adc10 con 0x7fdee00d03a0 2024-04-17T10:33:17.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.282+0000 7fded7400640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdebc066f60 0x7fdebc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.282+0000 7fded7400640 1 -- 172.21.15.67:0/3887231866 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fded8120430 con 0x7fdee00d03a0 2024-04-17T10:33:17.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.282+0000 7fdede800640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdebc066f60 0x7fdebc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.282+0000 7fdede800640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdebc066f60 0x7fdebc069420 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fded00079c0 tx=0x7fded0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.286+0000 7fded7400640 1 -- 172.21.15.67:0/3887231866 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fded8016520 con 0x7fdee00d03a0 2024-04-17T10:33:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.482+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdee0070bb0 con 0x7fdee00d03a0 2024-04-17T10:33:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.482+0000 7fded7400640 1 -- 172.21.15.67:0/3887231866 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+4937 (secure 0 0 0) 0x7fded80f0bc0 con 0x7fdee00d03a0 2024-04-17T10:33:17.474 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:17.474 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":73,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:32:33.448829+0000","modified":"2024-04-17T10:32:36.459180+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8139},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8139":{"gid":8139,"name":"a","rank":0,"incarnation":69,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/2012054720","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2012054720},{"type":"v1","addr":"172.21.15.67:6834","nonce":2012054720}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-04-17T10:33:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 73 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.482+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdebc066f60 msgr2=0x7fdebc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.482+0000 7fdee5400640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdebc066f60 0x7fdebc069420 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fded00079c0 tx=0x7fded0008040 comp rx=0 tx=0).stop 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.482+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00d03a0 msgr2=0x7fdee0157480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.482+0000 7fdee5400640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00d03a0 0x7fdee0157480 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fded8019480 tx=0x7fded8019610 comp rx=0 tx=0).stop 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 shutdown_connections 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdebc066f60 0x7fdebc069420 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdee00d03a0 0x7fdee0157480 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdee015dd10 0x7fdee013d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 --2- 172.21.15.67:0/3887231866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdee00714d0 0x7fdee014e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 >> 172.21.15.67:0/3887231866 conn(0x7fdee00b34d0 msgr2=0x7fdee0150060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 shutdown_connections 2024-04-17T10:33:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.486+0000 7fdee5400640 1 -- 172.21.15.67:0/3887231866 wait complete. 2024-04-17T10:33:17.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505c00640 1 Processor -- start 2024-04-17T10:33:17.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505c00640 1 -- start start 2024-04-17T10:33:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe500161160 0x7fe500099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5000d0760 con 0x7fe500099f90 2024-04-17T10:33:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5001624e0 con 0x7fe5000b2570 2024-04-17T10:33:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5000cfc70 con 0x7fe500161160 2024-04-17T10:33:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe4ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe500161160 0x7fe500099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe500099f90 0x7fe500161dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49772/0 (socket says 172.21.15.67:49772) 2024-04-17T10:33:17.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe505200640 1 -- 172.21.15.67:0/3453693966 learned_addr learned my addr 172.21.15.67:0/3453693966 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe4ffe00640 1 -- 172.21.15.67:0/3453693966 >> v1:172.21.15.86:6789/0 conn(0x7fe5000b2570 legacy=0x7fe5001616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe4ffe00640 1 -- 172.21.15.67:0/3453693966 >> v1:172.21.15.67:6789/0 conn(0x7fe500099f90 legacy=0x7fe500161dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe4ffe00640 1 -- 172.21.15.67:0/3453693966 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5000cc2a0 con 0x7fe500161160 2024-04-17T10:33:17.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.646+0000 7fe4fea00640 1 -- 172.21.15.67:0/3453693966 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036755814 0 0) 0x7fe5000d0760 con 0x7fe500099f90 2024-04-17T10:33:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe4ffe00640 1 --2- 172.21.15.67:0/3453693966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500161160 0x7fe500099a50 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00097c0 tx=0x7fe4f001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1273bba38e793c57 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe4fea00640 1 -- 172.21.15.67:0/3453693966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00b1070 con 0x7fe500161160 2024-04-17T10:33:17.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe4fea00640 1 -- 172.21.15.67:0/3453693966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f00ac410 con 0x7fe500161160 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe4fea00640 1 -- 172.21.15.67:0/3453693966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00ac7e0 con 0x7fe500161160 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 -- 172.21.15.67:0/3453693966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500161160 msgr2=0x7fe500099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 --2- 172.21.15.67:0/3453693966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500161160 0x7fe500099a50 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00097c0 tx=0x7fe4f001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 -- 172.21.15.67:0/3453693966 shutdown_connections 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 --2- 172.21.15.67:0/3453693966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500161160 0x7fe500099a50 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 -- 172.21.15.67:0/3453693966 >> 172.21.15.67:0/3453693966 conn(0x7fe5000d1690 msgr2=0x7fe5000d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 -- 172.21.15.67:0/3453693966 shutdown_connections 2024-04-17T10:33:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 -- 172.21.15.67:0/3453693966 wait complete. 2024-04-17T10:33:17.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 Processor -- start 2024-04-17T10:33:17.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.650+0000 7fe505c00640 1 -- start start 2024-04-17T10:33:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500099f90 0x7fe500147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5000b2570 0x7fe50014ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe500161160 0x7fe50014cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe50009a3a0 con 0x7fe500099f90 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5000cc5d0 con 0x7fe500161160 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5000b2bc0 con 0x7fe5000b2570 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500099f90 0x7fe500147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500099f90 0x7fe500147950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57454/0 (socket says 172.21.15.67:57454) 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 -- 172.21.15.67:0/1354661878 learned_addr learned my addr 172.21.15.67:0/1354661878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ff400640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5000b2570 0x7fe50014ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505200640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe500161160 0x7fe50014cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 -- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5000b2570 msgr2=0x7fe50014ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5000b2570 0x7fe50014ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 -- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe500161160 msgr2=0x7fe50014cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe500161160 0x7fe50014cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 -- 172.21.15.67:0/1354661878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5001533c0 con 0x7fe500099f90 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4ffe00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500099f90 0x7fe500147950 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00190b0 tx=0x7fe4f0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4f7e00640 1 -- 172.21.15.67:0/1354661878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00b9070 con 0x7fe500099f90 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4f7e00640 1 -- 172.21.15.67:0/1354661878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f00ace40 con 0x7fe500099f90 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe500153bd0 con 0x7fe500099f90 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe4f7e00640 1 -- 172.21.15.67:0/1354661878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00b1040 con 0x7fe500099f90 2024-04-17T10:33:17.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe500154170 con 0x7fe500099f90 2024-04-17T10:33:17.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.654+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe500074dc0 con 0x7fe500099f90 2024-04-17T10:33:17.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.658+0000 7fe4f7e00640 1 -- 172.21.15.67:0/1354661878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe4f00bc070 con 0x7fe500099f90 2024-04-17T10:33:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.658+0000 7fe4f7e00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe4e0066f60 0x7fe4e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.658+0000 7fe4f7e00640 1 -- 172.21.15.67:0/1354661878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe4f011efe0 con 0x7fe500099f90 2024-04-17T10:33:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.658+0000 7fe4ff400640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe4e0066f60 0x7fe4e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.658+0000 7fe4ff400640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe4e0066f60 0x7fe4e0069420 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe500070700 tx=0x7fe4ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.658+0000 7fe4f7e00640 1 -- 172.21.15.67:0/1354661878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe4f0016090 con 0x7fe500099f90 2024-04-17T10:33:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.826+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe50009ac60 con 0x7fe500099f90 2024-04-17T10:33:17.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.826+0000 7fe4f7e00640 1 -- 172.21.15.67:0/1354661878 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16836 (secure 0 0 0) 0x7fe4f00ef880 con 0x7fe500099f90 2024-04-17T10:33:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:17.819 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":107,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:32:34.925635+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:32:31.849728+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"106","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-04-17T10:32:32.441190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"107","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe4e0066f60 msgr2=0x7fe4e0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe4e0066f60 0x7fe4e0069420 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe500070700 tx=0x7fe4ec009040 comp rx=0 tx=0).stop 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500099f90 msgr2=0x7fe500147950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500099f90 0x7fe500147950 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fe4f00190b0 tx=0x7fe4f0019240 comp rx=0 tx=0).stop 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 shutdown_connections 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe4e0066f60 0x7fe4e0069420 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe500161160 0x7fe50014cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe5000b2570 0x7fe50014ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 --2- 172.21.15.67:0/1354661878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe500099f90 0x7fe500147950 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 >> 172.21.15.67:0/1354661878 conn(0x7fe5000d1690 msgr2=0x7fe50006c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 shutdown_connections 2024-04-17T10:33:17.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.830+0000 7fe505c00640 1 -- 172.21.15.67:0/1354661878 wait complete. 2024-04-17T10:33:17.841 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:33:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.994+0000 7f955fe00640 1 Processor -- start 2024-04-17T10:33:17.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.994+0000 7f955fe00640 1 -- start start 2024-04-17T10:33:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.994+0000 7f955fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f95600ceb30 0x7f956015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.994+0000 7f955fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f956015f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.994+0000 7f955fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560160150 con 0x7f95601450b0 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.994+0000 7f955fe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956006d6e0 con 0x7f95600ceb30 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.994+0000 7f955fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956006d930 con 0x7f956015f3e0 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f95601450b0 0x7f95600ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49774/0 (socket says 172.21.15.67:49774) 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955ea00640 1 -- 172.21.15.67:0/303254719 learned_addr learned my addr 172.21.15.67:0/303254719 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955f400640 1 --2- 172.21.15.67:0/303254719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f956015f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f9557e00640 1 --2- 172.21.15.67:0/303254719 >> v2:172.21.15.86:3300/0 conn(0x7f95600ceb30 0x7f956015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955e000640 1 -- 172.21.15.67:0/303254719 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119162568 0 0) 0x7f9560160150 con 0x7f95601450b0 2024-04-17T10:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955e000640 1 -- 172.21.15.67:0/303254719 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9544003530 con 0x7f95601450b0 2024-04-17T10:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955f400640 1 -- 172.21.15.67:0/303254719 >> v2:172.21.15.86:3300/0 conn(0x7f95600ceb30 msgr2=0x7f956015bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955f400640 1 --2- 172.21.15.67:0/303254719 >> v2:172.21.15.86:3300/0 conn(0x7f95600ceb30 0x7f956015bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955f400640 1 -- 172.21.15.67:0/303254719 >> v1:172.21.15.67:6789/0 conn(0x7f95601450b0 legacy=0x7f95600ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955f400640 1 -- 172.21.15.67:0/303254719 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956006dc10 con 0x7f956015f3e0 2024-04-17T10:33:17.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f9557e00640 1 --2- 172.21.15.67:0/303254719 >> v2:172.21.15.86:3300/0 conn(0x7f95600ceb30 0x7f956015bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:17.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:17.998+0000 7f955f400640 1 --2- 172.21.15.67:0/303254719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f956015f870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9558009d90 tx=0x7f955801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85fe8c487d5ffb72 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955e000640 1 -- 172.21.15.67:0/303254719 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95580b2070 con 0x7f956015f3e0 2024-04-17T10:33:17.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955e000640 1 -- 172.21.15.67:0/303254719 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95580a58b0 con 0x7f956015f3e0 2024-04-17T10:33:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955e000640 1 -- 172.21.15.67:0/303254719 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95580a5c80 con 0x7f956015f3e0 2024-04-17T10:33:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- 172.21.15.67:0/303254719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 msgr2=0x7f956015f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 --2- 172.21.15.67:0/303254719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f956015f870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f9558009d90 tx=0x7f955801c8e0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- 172.21.15.67:0/303254719 shutdown_connections 2024-04-17T10:33:17.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 --2- 172.21.15.67:0/303254719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f956015f870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 --2- 172.21.15.67:0/303254719 >> v2:172.21.15.86:3300/0 conn(0x7f95600ceb30 0x7f956015bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- 172.21.15.67:0/303254719 >> 172.21.15.67:0/303254719 conn(0x7f9560067070 msgr2=0x7f956006ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:17.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- 172.21.15.67:0/303254719 shutdown_connections 2024-04-17T10:33:17.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- 172.21.15.67:0/303254719 wait complete. 2024-04-17T10:33:17.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 Processor -- start 2024-04-17T10:33:17.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- start start 2024-04-17T10:33:17.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95600ceb30 0x7f956014e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95601450b0 0x7f95600caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f95600cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956006eda0 con 0x7f95601450b0 2024-04-17T10:33:17.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956006ec20 con 0x7f95600ceb30 2024-04-17T10:33:17.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956006ef20 con 0x7f956015f3e0 2024-04-17T10:33:17.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f9557e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95601450b0 0x7f95600caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f9557e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95601450b0 0x7f95600caf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57466/0 (socket says 172.21.15.67:57466) 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f9557e00640 1 -- 172.21.15.67:0/2412756581 learned_addr learned my addr 172.21.15.67:0/2412756581 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955ea00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95600ceb30 0x7f956014e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.002+0000 7f955f400640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f95600cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9557e00640 1 -- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 msgr2=0x7f95600cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9557e00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f95600cd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9557e00640 1 -- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95600ceb30 msgr2=0x7f956014e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9557e00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95600ceb30 0x7f956014e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9557e00640 1 -- 172.21.15.67:0/2412756581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95600cbd00 con 0x7f95601450b0 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9557e00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95601450b0 0x7f95600caf10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f954800e9d0 tx=0x7f954800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9556a00640 1 -- 172.21.15.67:0/2412756581 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9548017070 con 0x7f95601450b0 2024-04-17T10:33:17.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9556a00640 1 -- 172.21.15.67:0/2412756581 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954800ca00 con 0x7f95601450b0 2024-04-17T10:33:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9556a00640 1 -- 172.21.15.67:0/2412756581 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954800cdf0 con 0x7f95601450b0 2024-04-17T10:33:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956013f890 con 0x7f95601450b0 2024-04-17T10:33:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956013fe50 con 0x7f95601450b0 2024-04-17T10:33:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9560074dc0 con 0x7f95601450b0 2024-04-17T10:33:17.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9556a00640 1 -- 172.21.15.67:0/2412756581 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f954801a070 con 0x7f95601450b0 2024-04-17T10:33:17.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9556a00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f953c066f90 0x7f953c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:17.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.006+0000 7f9556a00640 1 -- 172.21.15.67:0/2412756581 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9548085140 con 0x7f95601450b0 2024-04-17T10:33:17.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.010+0000 7f955ea00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f953c066f90 0x7f953c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:18.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.010+0000 7f955ea00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f953c066f90 0x7f953c069450 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9550007fd0 tx=0x7f9550009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:18.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.010+0000 7f9556a00640 1 -- 172.21.15.67:0/2412756581 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9548055840 con 0x7f95601450b0 2024-04-17T10:33:18.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.206+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9560166400 con 0x7f95601450b0 2024-04-17T10:33:18.199 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:33:18.206+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:33:18.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.210+0000 7f9556a00640 1 -- 172.21.15.67:0/2412756581 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v74) v1 ==== 161+0+0 (secure 0 0 0) 0x7f95480598e0 con 0x7f95601450b0 2024-04-17T10:33:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:33:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f953c066f90 msgr2=0x7f953c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f953c066f90 0x7f953c069450 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9550007fd0 tx=0x7f9550009040 comp rx=0 tx=0).stop 2024-04-17T10:33:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95601450b0 msgr2=0x7f95600caf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95601450b0 0x7f95600caf10 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f954800e9d0 tx=0x7f954800ee90 comp rx=0 tx=0).stop 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 shutdown_connections 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f953c066f90 0x7f953c069450 secure :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f9550007fd0 tx=0x7f9550009040 comp rx=0 tx=0).stop 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f956015f3e0 0x7f95600cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f95601450b0 0x7f95600caf10 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 --2- 172.21.15.67:0/2412756581 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95600ceb30 0x7f956014e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 >> 172.21.15.67:0/2412756581 conn(0x7f9560067070 msgr2=0x7f956015c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 shutdown_connections 2024-04-17T10:33:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.214+0000 7f955fe00640 1 -- 172.21.15.67:0/2412756581 wait complete. 2024-04-17T10:33:18.226 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:33:18.277 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:33:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.378+0000 7fed1c000640 1 Processor -- start 2024-04-17T10:33:18.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.378+0000 7fed1c000640 1 -- start start 2024-04-17T10:33:18.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed1c000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fed14159440 0x7fed140ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed1c000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed1415cb30 con 0x7fed1415c580 2024-04-17T10:33:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed1c000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed140cd610 con 0x7fed140713e0 2024-04-17T10:33:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed1c000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed140a8650 con 0x7fed14159440 2024-04-17T10:33:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed1a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fed1415c580 0x7fed1415d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49790/0 (socket says 172.21.15.67:49790) 2024-04-17T10:33:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed1a000640 1 -- 172.21.15.67:0/363377979 learned_addr learned my addr 172.21.15.67:0/363377979 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:18.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed19600640 1 --2- 172.21.15.67:0/363377979 >> v2:172.21.15.86:3300/0 conn(0x7fed14159440 0x7fed140ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:18.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed13e00640 1 -- 172.21.15.67:0/363377979 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174824646 0 0) 0x7fed1415cb30 con 0x7fed1415c580 2024-04-17T10:33:18.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed13e00640 1 -- 172.21.15.67:0/363377979 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecfc003530 con 0x7fed1415c580 2024-04-17T10:33:18.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed19600640 1 -- 172.21.15.67:0/363377979 >> v1:172.21.15.86:6789/0 conn(0x7fed140713e0 legacy=0x7fed1415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:18.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed19600640 1 -- 172.21.15.67:0/363377979 >> v1:172.21.15.67:6789/0 conn(0x7fed1415c580 legacy=0x7fed1415d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:18.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.382+0000 7fed19600640 1 -- 172.21.15.67:0/363377979 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed140cd8f0 con 0x7fed14159440 2024-04-17T10:33:18.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed19600640 1 --2- 172.21.15.67:0/363377979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 0x7fed140ce0f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fed080097c0 tx=0x7fed1415df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=634a41ddb32d109e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed13e00640 1 -- 172.21.15.67:0/363377979 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080b1070 con 0x7fed14159440 2024-04-17T10:33:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed13e00640 1 -- 172.21.15.67:0/363377979 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed080ac410 con 0x7fed14159440 2024-04-17T10:33:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed13e00640 1 -- 172.21.15.67:0/363377979 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080ac7e0 con 0x7fed14159440 2024-04-17T10:33:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 -- 172.21.15.67:0/363377979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 msgr2=0x7fed140ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 --2- 172.21.15.67:0/363377979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 0x7fed140ce0f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fed080097c0 tx=0x7fed1415df50 comp rx=0 tx=0).stop 2024-04-17T10:33:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 -- 172.21.15.67:0/363377979 shutdown_connections 2024-04-17T10:33:18.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 --2- 172.21.15.67:0/363377979 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 0x7fed140ce0f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 -- 172.21.15.67:0/363377979 >> 172.21.15.67:0/363377979 conn(0x7fed140b1740 msgr2=0x7fed140b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 -- 172.21.15.67:0/363377979 shutdown_connections 2024-04-17T10:33:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 -- 172.21.15.67:0/363377979 wait complete. 2024-04-17T10:33:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 Processor -- start 2024-04-17T10:33:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.386+0000 7fed1c000640 1 -- start start 2024-04-17T10:33:18.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed140713e0 0x7fed1414eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 0x7fed140b2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1415c580 0x7fed140b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed14169640 con 0x7fed1415c580 2024-04-17T10:33:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed141694c0 con 0x7fed14159440 2024-04-17T10:33:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed141697c0 con 0x7fed140713e0 2024-04-17T10:33:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1415c580 0x7fed140b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:18.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1415c580 0x7fed140b2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57492/0 (socket says 172.21.15.67:57492) 2024-04-17T10:33:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 -- 172.21.15.67:0/2363149255 learned_addr learned my addr 172.21.15.67:0/2363149255 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed19600640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed140713e0 0x7fed1414eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed18c00640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 0x7fed140b2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 -- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed140713e0 msgr2=0x7fed1414eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed140713e0 0x7fed1414eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:18.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 -- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 msgr2=0x7fed140b2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:18.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 0x7fed140b2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:18.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 -- 172.21.15.67:0/2363149255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1414f990 con 0x7fed1415c580 2024-04-17T10:33:18.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1a000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1415c580 0x7fed140b2ed0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00cb70 tx=0x7fed0c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:18.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed11600640 1 -- 172.21.15.67:0/2363149255 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c017070 con 0x7fed1415c580 2024-04-17T10:33:18.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed11600640 1 -- 172.21.15.67:0/2363149255 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0c00ad40 con 0x7fed1415c580 2024-04-17T10:33:18.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed141512f0 con 0x7fed1415c580 2024-04-17T10:33:18.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed11600640 1 -- 172.21.15.67:0/2363149255 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0c012620 con 0x7fed1415c580 2024-04-17T10:33:18.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1415a8b0 con 0x7fed1415c580 2024-04-17T10:33:18.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.390+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1404ef30 con 0x7fed1415c580 2024-04-17T10:33:18.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.394+0000 7fed11600640 1 -- 172.21.15.67:0/2363149255 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fed0c01a070 con 0x7fed1415c580 2024-04-17T10:33:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.394+0000 7fed11600640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066fb0 0x7fecf4069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.394+0000 7fed11600640 1 -- 172.21.15.67:0/2363149255 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fed0c084ff0 con 0x7fed1415c580 2024-04-17T10:33:18.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.394+0000 7fed19600640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066fb0 0x7fecf4069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.394+0000 7fed19600640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066fb0 0x7fecf4069470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fed080ae000 tx=0x7fed08004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:18.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.398+0000 7fed11600640 1 -- 172.21.15.67:0/2363149255 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fed1414f990 con 0x7fed1415c580 2024-04-17T10:33:18.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:18.594+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fed140b1260 con 0x7fed1415c580 2024-04-17T10:33:19.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed11600640 1 -- 172.21.15.67:0/2363149255 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7fed0c0557b0 con 0x7fed1415c580 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066fb0 msgr2=0x7fecf4069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066fb0 0x7fecf4069470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fed080ae000 tx=0x7fed08004740 comp rx=0 tx=0).stop 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1415c580 msgr2=0x7fed140b2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1415c580 0x7fed140b2ed0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00cb70 tx=0x7fed0c00a430 comp rx=0 tx=0).stop 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 shutdown_connections 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecf4066fb0 0x7fecf4069470 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed1415c580 0x7fed140b2ed0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fed14159440 0x7fed140b2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 --2- 172.21.15.67:0/2363149255 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fed140713e0 0x7fed1414eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 >> 172.21.15.67:0/2363149255 conn(0x7fed140b1740 msgr2=0x7fed140ae1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 shutdown_connections 2024-04-17T10:33:19.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.218+0000 7fed1c000640 1 -- 172.21.15.67:0/2363149255 wait complete. 2024-04-17T10:33:19.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:33:19.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f3232200640 1 Processor -- start 2024-04-17T10:33:19.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f3232200640 1 -- start start 2024-04-17T10:33:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f3232200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f322c0cd2f0 0x7f322c1600b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f3232200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f322c09a7b0 0x7f322c0cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f3232200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0cc290 con 0x7f322c099b30 2024-04-17T10:33:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f3232200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0cce60 con 0x7f322c09a7b0 2024-04-17T10:33:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f3232200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0cc530 con 0x7f322c0cd2f0 2024-04-17T10:33:19.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f322c09a7b0 0x7f322c0cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f322c099b30 0x7f322c099f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49792/0 (socket says 172.21.15.67:49792) 2024-04-17T10:33:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322b400640 1 -- 172.21.15.67:0/174531440 learned_addr learned my addr 172.21.15.67:0/174531440 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:19.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322aa00640 1 --2- 172.21.15.67:0/174531440 >> v2:172.21.15.86:3300/0 conn(0x7f322c0cd2f0 0x7f322c1600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322be00640 1 -- 172.21.15.67:0/174531440 >> v2:172.21.15.86:3300/0 conn(0x7f322c0cd2f0 msgr2=0x7f322c1600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322be00640 1 --2- 172.21.15.67:0/174531440 >> v2:172.21.15.86:3300/0 conn(0x7f322c0cd2f0 0x7f322c1600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322be00640 1 -- 172.21.15.67:0/174531440 >> v1:172.21.15.67:6789/0 conn(0x7f322c099b30 legacy=0x7f322c099f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:19.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.394+0000 7f322be00640 1 -- 172.21.15.67:0/174531440 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0cc810 con 0x7f322c09a7b0 2024-04-17T10:33:19.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.398+0000 7f322be00640 1 --2- 172.21.15.67:0/174531440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c09a7b0 0x7f322c0cf3c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f322c073190 tx=0x7f322401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63111862626ce030 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:19.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.398+0000 7f322a000640 1 -- 172.21.15.67:0/174531440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32240b2070 con 0x7f322c09a7b0 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.398+0000 7f322a000640 1 -- 172.21.15.67:0/174531440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32240a5760 con 0x7f322c09a7b0 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.398+0000 7f322a000640 1 -- 172.21.15.67:0/174531440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32240a5b30 con 0x7f322c09a7b0 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- 172.21.15.67:0/174531440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c09a7b0 msgr2=0x7f322c0cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 --2- 172.21.15.67:0/174531440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c09a7b0 0x7f322c0cf3c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7f322c073190 tx=0x7f322401c8a0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- 172.21.15.67:0/174531440 shutdown_connections 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 --2- 172.21.15.67:0/174531440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c09a7b0 0x7f322c0cf3c0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 --2- 172.21.15.67:0/174531440 >> v2:172.21.15.86:3300/0 conn(0x7f322c0cd2f0 0x7f322c1600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- 172.21.15.67:0/174531440 >> 172.21.15.67:0/174531440 conn(0x7f322c0b32e0 msgr2=0x7f322c0b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- 172.21.15.67:0/174531440 shutdown_connections 2024-04-17T10:33:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- 172.21.15.67:0/174531440 wait complete. 2024-04-17T10:33:19.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 Processor -- start 2024-04-17T10:33:19.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- start start 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c099b30 0x7f322c0c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f322c09a7b0 0x7f322c1f8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f322c0cd2f0 0x7f322c15de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c168aa0 con 0x7f322c099b30 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c168920 con 0x7f322c0cd2f0 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f3232200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c168c20 con 0x7f322c09a7b0 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c099b30 0x7f322c0c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:19.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c099b30 0x7f322c0c5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57504/0 (socket says 172.21.15.67:57504) 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 -- 172.21.15.67:0/438932101 learned_addr learned my addr 172.21.15.67:0/438932101 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322be00640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f322c0cd2f0 0x7f322c15de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322aa00640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f322c09a7b0 0x7f322c1f8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 -- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f322c09a7b0 msgr2=0x7f322c1f8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f322c09a7b0 0x7f322c1f8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 -- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f322c0cd2f0 msgr2=0x7f322c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f322c0cd2f0 0x7f322c15de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.402+0000 7f322b400640 1 -- 172.21.15.67:0/438932101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c1f52d0 con 0x7f322c099b30 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f322b400640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c099b30 0x7f322c0c5de0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f321c00c370 tx=0x7f321c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3223400640 1 -- 172.21.15.67:0/438932101 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c017070 con 0x7f322c099b30 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3223400640 1 -- 172.21.15.67:0/438932101 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321c005cf0 con 0x7f322c099b30 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3223400640 1 -- 172.21.15.67:0/438932101 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321c0072f0 con 0x7f322c099b30 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c0c6860 con 0x7f322c099b30 2024-04-17T10:33:19.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c1f5850 con 0x7f322c099b30 2024-04-17T10:33:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c09f900 con 0x7f322c099b30 2024-04-17T10:33:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3223400640 1 -- 172.21.15.67:0/438932101 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f321c01a070 con 0x7f322c099b30 2024-04-17T10:33:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3223400640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f320c066f60 0x7f320c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f3223400640 1 -- 172.21.15.67:0/438932101 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f321c084520 con 0x7f322c099b30 2024-04-17T10:33:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.406+0000 7f322aa00640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f320c066f60 0x7f320c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:19.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.410+0000 7f322aa00640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f320c066f60 0x7f320c069420 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f322c0cb8c0 tx=0x7f3218008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:19.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.414+0000 7f3223400640 1 -- 172.21.15.67:0/438932101 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f321c054ce0 con 0x7f322c099b30 2024-04-17T10:33:19.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:19.578+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f322c074e30 con 0x7f322c099b30 2024-04-17T10:33:20.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.406+0000 7f3223400640 1 -- 172.21.15.67:0/438932101 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) v1 ==== 181+0+0 (secure 0 0 0) 0x7f321c058d80 con 0x7f322c099b30 2024-04-17T10:33:20.396 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:33:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.406+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f320c066f60 msgr2=0x7f320c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.406+0000 7f3232200640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f320c066f60 0x7f320c069420 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f322c0cb8c0 tx=0x7f3218008040 comp rx=0 tx=0).stop 2024-04-17T10:33:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.406+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c099b30 msgr2=0x7f322c0c5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.406+0000 7f3232200640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c099b30 0x7f322c0c5de0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f321c00c370 tx=0x7f321c00c830 comp rx=0 tx=0).stop 2024-04-17T10:33:20.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 shutdown_connections 2024-04-17T10:33:20.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f320c066f60 0x7f320c069420 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f322c0cd2f0 0x7f322c15de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f322c09a7b0 0x7f322c1f8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 --2- 172.21.15.67:0/438932101 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f322c099b30 0x7f322c0c5de0 secure :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f321c00c370 tx=0x7f321c00c830 comp rx=0 tx=0).stop 2024-04-17T10:33:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 >> 172.21.15.67:0/438932101 conn(0x7f322c0b32e0 msgr2=0x7f322c0b25e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 shutdown_connections 2024-04-17T10:33:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.410+0000 7f3232200640 1 -- 172.21.15.67:0/438932101 wait complete. 2024-04-17T10:33:20.430 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:33:20.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 Processor -- start 2024-04-17T10:33:20.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 -- start start 2024-04-17T10:33:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f88d41447e0 0x7f88d406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88d4144f00 0x7f88d4071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d40718a0 0x7f88d415f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d4071cf0 con 0x7f88d4144f00 2024-04-17T10:33:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88d40cc310 con 0x7f88d41447e0 2024-04-17T10:33:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88da000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d406d9a0 con 0x7f88d40718a0 2024-04-17T10:33:20.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88d4144f00 0x7f88d4071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88d4144f00 0x7f88d4071360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56046/0 (socket says 172.21.15.67:56046) 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 -- 172.21.15.67:0/2941765948 learned_addr learned my addr 172.21.15.67:0/2941765948 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88d3400640 1 --2- 172.21.15.67:0/2941765948 >> v2:172.21.15.86:3300/0 conn(0x7f88d41447e0 0x7f88d406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 -- 172.21.15.67:0/2941765948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d40718a0 msgr2=0x7f88d415f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88d3e00640 1 --2- 172.21.15.67:0/2941765948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d40718a0 0x7f88d415f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 --2- 172.21.15.67:0/2941765948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d40718a0 0x7f88d415f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 -- 172.21.15.67:0/2941765948 >> v2:172.21.15.86:3300/0 conn(0x7f88d41447e0 msgr2=0x7f88d406d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 --2- 172.21.15.67:0/2941765948 >> v2:172.21.15.86:3300/0 conn(0x7f88d41447e0 0x7f88d406d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88caa00640 1 -- 172.21.15.67:0/2941765948 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d406dc80 con 0x7f88d4144f00 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88d3400640 1 --2- 172.21.15.67:0/2941765948 >> v2:172.21.15.86:3300/0 conn(0x7f88d41447e0 0x7f88d406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:20.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.590+0000 7f88d3e00640 1 --2- 172.21.15.67:0/2941765948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d40718a0 0x7f88d415f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88caa00640 1 --2- 172.21.15.67:0/2941765948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4144f00 0x7f88d4071360 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f88d4072410 tx=0x7f88bc01c1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76188c2cb2d54d64 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88d2a00640 1 -- 172.21.15.67:0/2941765948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0b0070 con 0x7f88d4144f00 2024-04-17T10:33:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88d2a00640 1 -- 172.21.15.67:0/2941765948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc0ab600 con 0x7f88d4144f00 2024-04-17T10:33:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88d2a00640 1 -- 172.21.15.67:0/2941765948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0ab9d0 con 0x7f88d4144f00 2024-04-17T10:33:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 -- 172.21.15.67:0/2941765948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4144f00 msgr2=0x7f88d4071360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 --2- 172.21.15.67:0/2941765948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4144f00 0x7f88d4071360 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f88d4072410 tx=0x7f88bc01c1a0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 -- 172.21.15.67:0/2941765948 shutdown_connections 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 --2- 172.21.15.67:0/2941765948 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d40718a0 0x7f88d415f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 --2- 172.21.15.67:0/2941765948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d4144f00 0x7f88d4071360 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 --2- 172.21.15.67:0/2941765948 >> v2:172.21.15.86:3300/0 conn(0x7f88d41447e0 0x7f88d406d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 -- 172.21.15.67:0/2941765948 >> 172.21.15.67:0/2941765948 conn(0x7f88d406ac90 msgr2=0x7f88d406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 -- 172.21.15.67:0/2941765948 shutdown_connections 2024-04-17T10:33:20.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.594+0000 7f88da000640 1 -- 172.21.15.67:0/2941765948 wait complete. 2024-04-17T10:33:20.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 Processor -- start 2024-04-17T10:33:20.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 -- start start 2024-04-17T10:33:20.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88d40718a0 0x7f88d415aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d41447e0 0x7f88d413aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d4144f00 0x7f88d415bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d40cc900 con 0x7f88d41447e0 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88d40cc780 con 0x7f88d40718a0 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d40cca80 con 0x7f88d4144f00 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d41447e0 0x7f88d413aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d41447e0 0x7f88d413aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56058/0 (socket says 172.21.15.67:56058) 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 -- 172.21.15.67:0/1346123355 learned_addr learned my addr 172.21.15.67:0/1346123355 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88d3e00640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d4144f00 0x7f88d415bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88d3400640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88d40718a0 0x7f88d415aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:20.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 -- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d4144f00 msgr2=0x7f88d415bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:20.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d4144f00 0x7f88d415bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 -- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88d40718a0 msgr2=0x7f88d415aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:20.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88d40718a0 0x7f88d415aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:20.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 -- 172.21.15.67:0/1346123355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d4139f50 con 0x7f88d41447e0 2024-04-17T10:33:20.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88caa00640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d41447e0 0x7f88d413aff0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0ae040 tx=0x7f88bc018f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:20.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88cbe00640 1 -- 172.21.15.67:0/1346123355 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0b0040 con 0x7f88d41447e0 2024-04-17T10:33:20.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88cbe00640 1 -- 172.21.15.67:0/1346123355 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc0bb070 con 0x7f88d41447e0 2024-04-17T10:33:20.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d413a230 con 0x7f88d41447e0 2024-04-17T10:33:20.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88cbe00640 1 -- 172.21.15.67:0/1346123355 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0b68a0 con 0x7f88d41447e0 2024-04-17T10:33:20.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.598+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d413a7f0 con 0x7f88d41447e0 2024-04-17T10:33:20.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.602+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d4072da0 con 0x7f88d41447e0 2024-04-17T10:33:20.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.602+0000 7f88cbe00640 1 -- 172.21.15.67:0/1346123355 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f88bc0b6a40 con 0x7f88d41447e0 2024-04-17T10:33:20.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.602+0000 7f88cbe00640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88b0066f60 0x7f88b0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:20.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.602+0000 7f88cbe00640 1 -- 172.21.15.67:0/1346123355 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f88bc11f440 con 0x7f88d41447e0 2024-04-17T10:33:20.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.602+0000 7f88d3400640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88b0066f60 0x7f88b0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:20.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.602+0000 7f88d3400640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88b0066f60 0x7f88b0069420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f88c400aae0 tx=0x7f88c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.606+0000 7f88cbe00640 1 -- 172.21.15.67:0/1346123355 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f88bc016110 con 0x7f88d41447e0 2024-04-17T10:33:20.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:20.766+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f88d40b2d10 con 0x7f88d41447e0 2024-04-17T10:33:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.406+0000 7f88cbe00640 1 -- 172.21.15.67:0/1346123355 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) v1 ==== 169+0+0 (secure 0 0 0) 0x7f88bc0efd30 con 0x7f88d41447e0 2024-04-17T10:33:21.398 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:33:21.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88b0066f60 msgr2=0x7f88b0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88b0066f60 0x7f88b0069420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f88c400aae0 tx=0x7f88c4008040 comp rx=0 tx=0).stop 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d41447e0 msgr2=0x7f88d413aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d41447e0 0x7f88d413aff0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0ae040 tx=0x7f88bc018f40 comp rx=0 tx=0).stop 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 shutdown_connections 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f88b0066f60 0x7f88b0069420 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88d4144f00 0x7f88d415bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88d41447e0 0x7f88d413aff0 secure :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f88bc0ae040 tx=0x7f88bc018f40 comp rx=0 tx=0).stop 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 --2- 172.21.15.67:0/1346123355 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f88d40718a0 0x7f88d415aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.414+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 >> 172.21.15.67:0/1346123355 conn(0x7f88d406ac90 msgr2=0x7f88d41400f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:21.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.418+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 shutdown_connections 2024-04-17T10:33:21.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.422+0000 7f88da000640 1 -- 172.21.15.67:0/1346123355 wait complete. 2024-04-17T10:33:21.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:33:21.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.662+0000 7fad2c600640 1 Processor -- start 2024-04-17T10:33:21.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.662+0000 7fad2c600640 1 -- start start 2024-04-17T10:33:21.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2c600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad24072a90 0x7fad24166a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:21.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2c600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2406b170 con 0x7fad2415bcc0 2024-04-17T10:33:21.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2c600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad2406b450 con 0x7fad2406ac90 2024-04-17T10:33:21.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2c600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad240afb90 con 0x7fad24072a90 2024-04-17T10:33:21.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2a600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad24072a90 0x7fad24166a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:21.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad29c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fad2415bcc0 0x7fad2415cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40720/0 (socket says 172.21.15.67:40720) 2024-04-17T10:33:21.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad29c00640 1 -- 172.21.15.67:0/690310809 learned_addr learned my addr 172.21.15.67:0/690310809 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:21.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2a600640 1 -- 172.21.15.67:0/690310809 >> v1:172.21.15.86:6789/0 conn(0x7fad2406ac90 legacy=0x7fad24072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:21.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2a600640 1 -- 172.21.15.67:0/690310809 >> v1:172.21.15.67:6789/0 conn(0x7fad2415bcc0 legacy=0x7fad2415cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad2a600640 1 -- 172.21.15.67:0/690310809 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad2406b730 con 0x7fad24072a90 2024-04-17T10:33:21.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.666+0000 7fad23e00640 1 -- 172.21.15.67:0/690310809 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723960533 0 0) 0x7fad2406b170 con 0x7fad2415bcc0 2024-04-17T10:33:21.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2a600640 1 --2- 172.21.15.67:0/690310809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad24166a90 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009d90 tx=0x7fad1c01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7f9614b2b1f428 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:21.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad23e00640 1 -- 172.21.15.67:0/690310809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0b2070 con 0x7fad24072a90 2024-04-17T10:33:21.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad23e00640 1 -- 172.21.15.67:0/690310809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad1c0a59e0 con 0x7fad24072a90 2024-04-17T10:33:21.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad23e00640 1 -- 172.21.15.67:0/690310809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad1c0a5db0 con 0x7fad24072a90 2024-04-17T10:33:21.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- 172.21.15.67:0/690310809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 msgr2=0x7fad24166a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 --2- 172.21.15.67:0/690310809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad24166a90 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fad1c009d90 tx=0x7fad1c01c900 comp rx=0 tx=0).stop 2024-04-17T10:33:21.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- 172.21.15.67:0/690310809 shutdown_connections 2024-04-17T10:33:21.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 --2- 172.21.15.67:0/690310809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad24166a90 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- 172.21.15.67:0/690310809 >> 172.21.15.67:0/690310809 conn(0x7fad240b2570 msgr2=0x7fad24071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:21.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- 172.21.15.67:0/690310809 shutdown_connections 2024-04-17T10:33:21.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- 172.21.15.67:0/690310809 wait complete. 2024-04-17T10:33:21.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 Processor -- start 2024-04-17T10:33:21.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- start start 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2406ac90 0x7fad240adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad2415a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad2415bcc0 0x7fad24147460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24169b80 con 0x7fad24072a90 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24169a00 con 0x7fad2415bcc0 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad2c600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad24169d00 con 0x7fad2406ac90 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad29200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad2415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad29200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad2415a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56078/0 (socket says 172.21.15.67:56078) 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.670+0000 7fad29200640 1 -- 172.21.15.67:0/3207382198 learned_addr learned my addr 172.21.15.67:0/3207382198 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad2a600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad2415bcc0 0x7fad24147460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad29c00640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2406ac90 0x7fad240adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad29200640 1 -- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2406ac90 msgr2=0x7fad240adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad29200640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2406ac90 0x7fad240adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad29200640 1 -- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad2415bcc0 msgr2=0x7fad24147460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad29200640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad2415bcc0 0x7fad24147460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad29200640 1 -- 172.21.15.67:0/3207382198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad240ace60 con 0x7fad24072a90 2024-04-17T10:33:21.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad29200640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad2415a830 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fad1000c480 tx=0x7fad1000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:21.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad21600640 1 -- 172.21.15.67:0/3207382198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10012830 con 0x7fad24072a90 2024-04-17T10:33:21.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad21600640 1 -- 172.21.15.67:0/3207382198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad10017070 con 0x7fad24072a90 2024-04-17T10:33:21.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad240ad140 con 0x7fad24072a90 2024-04-17T10:33:21.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad21600640 1 -- 172.21.15.67:0/3207382198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad10012c30 con 0x7fad24072a90 2024-04-17T10:33:21.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad240ad700 con 0x7fad24072a90 2024-04-17T10:33:21.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad2404ef30 con 0x7fad24072a90 2024-04-17T10:33:21.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad21600640 1 -- 172.21.15.67:0/3207382198 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fad1001c460 con 0x7fad24072a90 2024-04-17T10:33:21.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad21600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad04066f60 0x7fad04069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:21.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.674+0000 7fad21600640 1 -- 172.21.15.67:0/3207382198 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fad10084bf0 con 0x7fad24072a90 2024-04-17T10:33:21.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.678+0000 7fad29c00640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad04066f60 0x7fad04069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:21.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.678+0000 7fad29c00640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad04066f60 0x7fad04069420 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fad18000f70 tx=0x7fad18007df0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:21.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.678+0000 7fad21600640 1 -- 172.21.15.67:0/3207382198 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fad10055710 con 0x7fad24072a90 2024-04-17T10:33:21.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.842+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fad24066ea0 con 0x7fad24072a90 2024-04-17T10:33:21.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.846+0000 7fad21600640 1 -- 172.21.15.67:0/3207382198 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fad100597b0 con 0x7fad24072a90 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad04066f60 msgr2=0x7fad04069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad04066f60 0x7fad04069420 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fad18000f70 tx=0x7fad18007df0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 msgr2=0x7fad2415a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad2415a830 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fad1000c480 tx=0x7fad1000c940 comp rx=0 tx=0).stop 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 shutdown_connections 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad04066f60 0x7fad04069420 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad2415bcc0 0x7fad24147460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad24072a90 0x7fad2415a830 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 --2- 172.21.15.67:0/3207382198 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad2406ac90 0x7fad240adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 >> 172.21.15.67:0/3207382198 conn(0x7fad240b2570 msgr2=0x7fad2415c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 shutdown_connections 2024-04-17T10:33:21.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:21.850+0000 7fad2c600640 1 -- 172.21.15.67:0/3207382198 wait complete. 2024-04-17T10:33:21.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume' 2024-04-17T10:33:22.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e3a00640 1 Processor -- start 2024-04-17T10:33:22.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e3a00640 1 -- start start 2024-04-17T10:33:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e3a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76dc0ce1b0 0x7f76dc0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e3a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc0cead0 con 0x7f76dc15fbd0 2024-04-17T10:33:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e3a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76dc15be60 con 0x7f76dc15cc00 2024-04-17T10:33:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e3a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76dc15ffe0 con 0x7f76dc0ce1b0 2024-04-17T10:33:22.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e1a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f76dc15fbd0 0x7f76dc0c6b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40732/0 (socket says 172.21.15.67:40732) 2024-04-17T10:33:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e1a00640 1 -- 172.21.15.67:0/1067935856 learned_addr learned my addr 172.21.15.67:0/1067935856 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e1000640 1 --2- 172.21.15.67:0/1067935856 >> v2:172.21.15.67:3300/0 conn(0x7f76dc0ce1b0 0x7f76dc0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e1000640 1 -- 172.21.15.67:0/1067935856 >> v1:172.21.15.86:6789/0 conn(0x7f76dc15cc00 legacy=0x7f76dc15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e1000640 1 -- 172.21.15.67:0/1067935856 >> v1:172.21.15.67:6789/0 conn(0x7f76dc15fbd0 legacy=0x7f76dc0c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76e1000640 1 -- 172.21.15.67:0/1067935856 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc160230 con 0x7f76dc0ce1b0 2024-04-17T10:33:22.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.010+0000 7f76dbe00640 1 -- 172.21.15.67:0/1067935856 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1568636517 0 0) 0x7f76dc0cead0 con 0x7f76dc15fbd0 2024-04-17T10:33:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e1000640 1 --2- 172.21.15.67:0/1067935856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0ce1b0 0x7f76dc0ce590 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0097c0 tx=0x7f76cc01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eaaf8513a7f984c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76dbe00640 1 -- 172.21.15.67:0/1067935856 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0b5070 con 0x7f76dc0ce1b0 2024-04-17T10:33:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76dbe00640 1 -- 172.21.15.67:0/1067935856 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc0a8990 con 0x7f76dc0ce1b0 2024-04-17T10:33:22.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76dbe00640 1 -- 172.21.15.67:0/1067935856 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0a8d60 con 0x7f76dc0ce1b0 2024-04-17T10:33:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e3a00640 1 -- 172.21.15.67:0/1067935856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0ce1b0 msgr2=0x7f76dc0ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e3a00640 1 --2- 172.21.15.67:0/1067935856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0ce1b0 0x7f76dc0ce590 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0097c0 tx=0x7f76cc01ec90 comp rx=0 tx=0).stop 2024-04-17T10:33:22.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e3a00640 1 -- 172.21.15.67:0/1067935856 shutdown_connections 2024-04-17T10:33:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e3a00640 1 --2- 172.21.15.67:0/1067935856 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc0ce1b0 0x7f76dc0ce590 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e3a00640 1 -- 172.21.15.67:0/1067935856 >> 172.21.15.67:0/1067935856 conn(0x7f76dc0b33f0 msgr2=0x7f76dc0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e3a00640 1 -- 172.21.15.67:0/1067935856 shutdown_connections 2024-04-17T10:33:22.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.014+0000 7f76e3a00640 1 -- 172.21.15.67:0/1067935856 wait complete. 2024-04-17T10:33:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 Processor -- start 2024-04-17T10:33:22.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 -- start start 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76dc0ce1b0 0x7f76dc14eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc15cc00 0x7f76dc0ca710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76dc15fbd0 0x7f76dc0b1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc15c6e0 con 0x7f76dc15cc00 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc15c400 con 0x7f76dc0ce1b0 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc159730 con 0x7f76dc15fbd0 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc15cc00 0x7f76dc0ca710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc15cc00 0x7f76dc0ca710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56096/0 (socket says 172.21.15.67:56096) 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 -- 172.21.15.67:0/2148461596 learned_addr learned my addr 172.21.15.67:0/2148461596 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e1000640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76dc0ce1b0 0x7f76dc14eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e1a00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76dc15fbd0 0x7f76dc0b1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 -- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76dc15fbd0 msgr2=0x7f76dc0b1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76dc15fbd0 0x7f76dc0b1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 -- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76dc0ce1b0 msgr2=0x7f76dc14eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76dc0ce1b0 0x7f76dc14eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 -- 172.21.15.67:0/2148461596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76dc14f410 con 0x7f76dc15cc00 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d3e00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc15cc00 0x7f76dc0ca710 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f76c400c500 tx=0x7f76c400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d9600640 1 -- 172.21.15.67:0/2148461596 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4016020 con 0x7f76dc15cc00 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d9600640 1 -- 172.21.15.67:0/2148461596 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c4005220 con 0x7f76dc15cc00 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76d9600640 1 -- 172.21.15.67:0/2148461596 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c4007030 con 0x7f76dc15cc00 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc0b6260 con 0x7f76dc15cc00 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc0c9f10 con 0x7f76dc15cc00 2024-04-17T10:33:22.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.018+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc070ab0 con 0x7f76dc15cc00 2024-04-17T10:33:22.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.022+0000 7f76d9600640 1 -- 172.21.15.67:0/2148461596 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f76c40071f0 con 0x7f76dc15cc00 2024-04-17T10:33:22.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.022+0000 7f76d9600640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f76b8066f60 0x7f76b8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.022+0000 7f76d9600640 1 -- 172.21.15.67:0/2148461596 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f76c4084860 con 0x7f76dc15cc00 2024-04-17T10:33:22.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.022+0000 7f76e1000640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f76b8066f60 0x7f76b8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.022+0000 7f76e1000640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f76b8066f60 0x7f76b8069420 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0b2000 tx=0x7f76cc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.026+0000 7f76d9600640 1 -- 172.21.15.67:0/2148461596 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f76c4020020 con 0x7f76dc15cc00 2024-04-17T10:33:22.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.241+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]} v 0) v1 -- 0x7f76dc074e30 con 0x7f76dc15cc00 2024-04-17T10:33:22.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.421+0000 7f76d9600640 1 -- 172.21.15.67:0/2148461596 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]}]=0 v409) v1 ==== 155+0+0 (secure 0 0 0) 0x7f76c4055950 con 0x7f76dc15cc00 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f76b8066f60 msgr2=0x7f76b8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f76b8066f60 0x7f76b8069420 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f76cc0b2000 tx=0x7f76cc008040 comp rx=0 tx=0).stop 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc15cc00 msgr2=0x7f76dc0ca710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc15cc00 0x7f76dc0ca710 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f76c400c500 tx=0x7f76c400c9c0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 shutdown_connections 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f76b8066f60 0x7f76b8069420 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f76dc15fbd0 0x7f76dc0b1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f76dc15cc00 0x7f76dc0ca710 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 --2- 172.21.15.67:0/2148461596 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f76dc0ce1b0 0x7f76dc14eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 >> 172.21.15.67:0/2148461596 conn(0x7f76dc0b33f0 msgr2=0x7f76dc0738a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 shutdown_connections 2024-04-17T10:33:22.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.425+0000 7f76e3a00640 1 -- 172.21.15.67:0/2148461596 wait complete. 2024-04-17T10:33:22.436 INFO:tasks.cephfs_test_runner:test_scrub_pause_and_resume (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-04-17T10:33:22.437 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-04-17T10:33:22.437 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:22.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a9c00640 1 Processor -- start 2024-04-17T10:33:22.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a9c00640 1 -- start start 2024-04-17T10:33:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a9c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a4145040 0x7f72a406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a9c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f72a4071a20 0x7f72a40cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a9c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a406e2f0 con 0x7f72a406ef10 2024-04-17T10:33:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a9c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a40a86c0 con 0x7f72a4071a20 2024-04-17T10:33:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a9c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a415bf10 con 0x7f72a4145040 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a4145040 0x7f72a406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a4145040 0x7f72a406e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:37094/0 (socket says 172.21.15.67:37094) 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a2a00640 1 -- 172.21.15.67:0/940736410 learned_addr learned my addr 172.21.15.67:0/940736410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.585+0000 7f72a3400640 1 --2- 172.21.15.67:0/940736410 >> v2:172.21.15.86:3300/0 conn(0x7f72a4071a20 0x7f72a40cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2a00640 1 -- 172.21.15.67:0/940736410 >> v2:172.21.15.86:3300/0 conn(0x7f72a4071a20 msgr2=0x7f72a40cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2a00640 1 --2- 172.21.15.67:0/940736410 >> v2:172.21.15.86:3300/0 conn(0x7f72a4071a20 0x7f72a40cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2a00640 1 -- 172.21.15.67:0/940736410 >> v1:172.21.15.86:6789/0 conn(0x7f72a406ef10 legacy=0x7f72a4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2a00640 1 -- 172.21.15.67:0/940736410 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a406d6e0 con 0x7f72a4145040 2024-04-17T10:33:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2a00640 1 --2- 172.21.15.67:0/940736410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a4145040 0x7f72a406e9d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7294009870 tx=0x7f729401ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47ecde818055e64f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2000640 1 -- 172.21.15.67:0/940736410 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940b5070 con 0x7f72a4145040 2024-04-17T10:33:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2000640 1 -- 172.21.15.67:0/940736410 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72940a8a50 con 0x7f72a4145040 2024-04-17T10:33:22.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.589+0000 7f72a2000640 1 -- 172.21.15.67:0/940736410 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940a8e20 con 0x7f72a4145040 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- 172.21.15.67:0/940736410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a4145040 msgr2=0x7f72a406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 --2- 172.21.15.67:0/940736410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a4145040 0x7f72a406e9d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7294009870 tx=0x7f729401ed50 comp rx=0 tx=0).stop 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- 172.21.15.67:0/940736410 shutdown_connections 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 --2- 172.21.15.67:0/940736410 >> v2:172.21.15.86:3300/0 conn(0x7f72a4071a20 0x7f72a40cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 --2- 172.21.15.67:0/940736410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a4145040 0x7f72a406e9d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- 172.21.15.67:0/940736410 >> 172.21.15.67:0/940736410 conn(0x7f72a406ac90 msgr2=0x7f72a406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- 172.21.15.67:0/940736410 shutdown_connections 2024-04-17T10:33:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- 172.21.15.67:0/940736410 wait complete. 2024-04-17T10:33:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 Processor -- start 2024-04-17T10:33:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- start start 2024-04-17T10:33:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a406ef10 0x7f72a415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72a4071a20 0x7f72a413e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72a4145040 0x7f72a415d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a406df80 con 0x7f72a4145040 2024-04-17T10:33:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a406dca0 con 0x7f72a4071a20 2024-04-17T10:33:22.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a9c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a406d370 con 0x7f72a406ef10 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72a4145040 0x7f72a415d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a406ef10 0x7f72a415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72a4145040 0x7f72a415d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56112/0 (socket says 172.21.15.67:56112) 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 -- 172.21.15.67:0/1337014000 learned_addr learned my addr 172.21.15.67:0/1337014000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f729be00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72a4071a20 0x7f72a413e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 -- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a406ef10 msgr2=0x7f72a415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a406ef10 0x7f72a415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 -- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72a4071a20 msgr2=0x7f72a413e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72a4071a20 0x7f72a413e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a3400640 1 -- 172.21.15.67:0/1337014000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a413ecf0 con 0x7f72a4145040 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.593+0000 7f72a2a00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a406ef10 0x7f72a415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f72a3400640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72a4145040 0x7f72a415d8b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f729c00efc0 tx=0x7f729c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f729aa00640 1 -- 172.21.15.67:0/1337014000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c018020 con 0x7f72a4145040 2024-04-17T10:33:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f729aa00640 1 -- 172.21.15.67:0/1337014000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f729c014480 con 0x7f72a4145040 2024-04-17T10:33:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a4142a80 con 0x7f72a4145040 2024-04-17T10:33:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f729aa00640 1 -- 172.21.15.67:0/1337014000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f729c014870 con 0x7f72a4145040 2024-04-17T10:33:22.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a4143040 con 0x7f72a4145040 2024-04-17T10:33:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a4143240 con 0x7f72a4145040 2024-04-17T10:33:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f729aa00640 1 -- 172.21.15.67:0/1337014000 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f729c01b020 con 0x7f72a4145040 2024-04-17T10:33:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f729aa00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7280066f60 0x7f7280069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f729aa00640 1 -- 172.21.15.67:0/1337014000 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f729c085d00 con 0x7f72a4145040 2024-04-17T10:33:22.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.597+0000 7f72a2a00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7280066f60 0x7f7280069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.601+0000 7f72a2a00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7280066f60 0x7f7280069420 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f72940b2000 tx=0x7f7294009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.601+0000 7f729aa00640 1 -- 172.21.15.67:0/1337014000 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f72a4143240 con 0x7f72a4145040 2024-04-17T10:33:22.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72a4074e30 con 0x7f72a4145040 2024-04-17T10:33:22.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f729aa00640 1 -- 172.21.15.67:0/1337014000 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+13255 (secure 0 0 0) 0x7f729c022020 con 0x7f72a4145040 2024-04-17T10:33:22.761 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:22.761 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":110,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:21.408149+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6834/2012054720":"2024-04-18T10:33:18.210754+0000","172.21.15.67:6832/2012054720":"2024-04-18T10:33:18.210754+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7280066f60 msgr2=0x7f7280069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7280066f60 0x7f7280069420 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f72940b2000 tx=0x7f7294009040 comp rx=0 tx=0).stop 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72a4145040 msgr2=0x7f72a415d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72a4145040 0x7f72a415d8b0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f729c00efc0 tx=0x7f729c00c490 comp rx=0 tx=0).stop 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 shutdown_connections 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7280066f60 0x7f7280069420 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72a4145040 0x7f72a415d8b0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72a4071a20 0x7f72a413e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 --2- 172.21.15.67:0/1337014000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72a406ef10 0x7f72a415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 >> 172.21.15.67:0/1337014000 conn(0x7f72a406ac90 msgr2=0x7f72a40af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:22.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 shutdown_connections 2024-04-17T10:33:22.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.769+0000 7f72a9c00640 1 -- 172.21.15.67:0/1337014000 wait complete. 2024-04-17T10:33:22.783 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:33:22.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:33:22.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.933+0000 7f422d200640 1 Processor -- start 2024-04-17T10:33:22.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f422d200640 1 -- start start 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f422d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280aef60 0x7f42280cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f422d200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f422806e980 0x7f42280b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f422d200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42280b4990 con 0x7f42280cce60 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f422d200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42280cd2c0 con 0x7f422806e980 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f422d200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42280af340 con 0x7f42280aef60 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f421d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f42280cce60 0x7f422806e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40744/0 (socket says 172.21.15.67:40744) 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f421d400640 1 -- 172.21.15.67:0/18591828 learned_addr learned my addr 172.21.15.67:0/18591828 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4225e00640 1 --2- 172.21.15.67:0/18591828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280aef60 0x7f42280cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4226800640 1 --2- 172.21.15.67:0/18591828 >> v2:172.21.15.86:3300/0 conn(0x7f422806e980 0x7f42280b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4225400640 1 -- 172.21.15.67:0/18591828 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120579375 0 0) 0x7f4210001120 con 0x7f42280cce60 2024-04-17T10:33:22.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4225400640 1 -- 172.21.15.67:0/18591828 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f420c005740 con 0x7f42280cce60 2024-04-17T10:33:22.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4225e00640 1 -- 172.21.15.67:0/18591828 >> v2:172.21.15.86:3300/0 conn(0x7f422806e980 msgr2=0x7f42280b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:22.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4225e00640 1 --2- 172.21.15.67:0/18591828 >> v2:172.21.15.86:3300/0 conn(0x7f422806e980 0x7f42280b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4225e00640 1 -- 172.21.15.67:0/18591828 >> v1:172.21.15.67:6789/0 conn(0x7f42280cce60 legacy=0x7f422806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.937+0000 7f4225e00640 1 -- 172.21.15.67:0/18591828 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422815ae90 con 0x7f42280aef60 2024-04-17T10:33:22.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f4225e00640 1 --2- 172.21.15.67:0/18591828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280aef60 0x7f42280cc920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4218009d90 tx=0x7f421801c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=806a397cfa21a2c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f4225400640 1 -- 172.21.15.67:0/18591828 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42180b2070 con 0x7f42280aef60 2024-04-17T10:33:22.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f4225400640 1 -- 172.21.15.67:0/18591828 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42180a5890 con 0x7f42280aef60 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f4225400640 1 -- 172.21.15.67:0/18591828 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42180a5c60 con 0x7f42280aef60 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 -- 172.21.15.67:0/18591828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280aef60 msgr2=0x7f42280cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 --2- 172.21.15.67:0/18591828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280aef60 0x7f42280cc920 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4218009d90 tx=0x7f421801c8e0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 -- 172.21.15.67:0/18591828 shutdown_connections 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 --2- 172.21.15.67:0/18591828 >> v2:172.21.15.86:3300/0 conn(0x7f422806e980 0x7f42280b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 --2- 172.21.15.67:0/18591828 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280aef60 0x7f42280cc920 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 -- 172.21.15.67:0/18591828 >> 172.21.15.67:0/18591828 conn(0x7f42280b1740 msgr2=0x7f42280b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 -- 172.21.15.67:0/18591828 shutdown_connections 2024-04-17T10:33:22.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 -- 172.21.15.67:0/18591828 wait complete. 2024-04-17T10:33:22.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 Processor -- start 2024-04-17T10:33:22.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.941+0000 7f422d200640 1 -- start start 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422806e980 0x7f4228140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280aef60 0x7f4228148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280cce60 0x7f4228159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422815bc10 con 0x7f422806e980 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42280b3e70 con 0x7f42280aef60 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4228142f80 con 0x7f42280cce60 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422806e980 0x7f4228140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422806e980 0x7f4228140680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56118/0 (socket says 172.21.15.67:56118) 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 -- 172.21.15.67:0/1104514926 learned_addr learned my addr 172.21.15.67:0/1104514926 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f421d400640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280aef60 0x7f4228148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4226800640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280cce60 0x7f4228159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 -- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280cce60 msgr2=0x7f4228159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280cce60 0x7f4228159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 -- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280aef60 msgr2=0x7f4228148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280aef60 0x7f4228148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 -- 172.21.15.67:0/1104514926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228143410 con 0x7f422806e980 2024-04-17T10:33:22.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f4225e00640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422806e980 0x7f4228140680 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f421801a510 tx=0x7f4218019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f421ea00640 1 -- 172.21.15.67:0/1104514926 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42180b2050 con 0x7f422806e980 2024-04-17T10:33:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f421ea00640 1 -- 172.21.15.67:0/1104514926 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42180be070 con 0x7f422806e980 2024-04-17T10:33:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f421ea00640 1 -- 172.21.15.67:0/1104514926 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218009450 con 0x7f422806e980 2024-04-17T10:33:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4228142370 con 0x7f422806e980 2024-04-17T10:33:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4228142930 con 0x7f422806e980 2024-04-17T10:33:22.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.945+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422804ef30 con 0x7f422806e980 2024-04-17T10:33:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.949+0000 7f421ea00640 1 -- 172.21.15.67:0/1104514926 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f42180adcc0 con 0x7f422806e980 2024-04-17T10:33:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.949+0000 7f421ea00640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4204066f90 0x7f4204069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:22.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.949+0000 7f421ea00640 1 -- 172.21.15.67:0/1104514926 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f421811fb20 con 0x7f422806e980 2024-04-17T10:33:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.949+0000 7f421d400640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4204066f90 0x7f4204069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:22.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.949+0000 7f421d400640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4204066f90 0x7f4204069450 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4228151830 tx=0x7f4210009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:22.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:22.953+0000 7f421ea00640 1 -- 172.21.15.67:0/1104514926 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f42180c7020 con 0x7f422806e980 2024-04-17T10:33:23.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42280cc8f0 con 0x7f422806e980 2024-04-17T10:33:23.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f421ea00640 1 -- 172.21.15.67:0/1104514926 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4218016560 con 0x7f422806e980 2024-04-17T10:33:23.109 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4204066f90 msgr2=0x7f4204069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4204066f90 0x7f4204069450 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f4228151830 tx=0x7f4210009040 comp rx=0 tx=0).stop 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422806e980 msgr2=0x7f4228140680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422806e980 0x7f4228140680 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f421801a510 tx=0x7f4218019680 comp rx=0 tx=0).stop 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 shutdown_connections 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4204066f90 0x7f4204069450 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42280cce60 0x7f4228159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:23.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280aef60 0x7f4228148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 --2- 172.21.15.67:0/1104514926 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422806e980 0x7f4228140680 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 >> 172.21.15.67:0/1104514926 conn(0x7f42280b1740 msgr2=0x7f422815c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 shutdown_connections 2024-04-17T10:33:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.117+0000 7f422d200640 1 -- 172.21.15.67:0/1104514926 wait complete. 2024-04-17T10:33:23.131 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort' 2024-04-17T10:33:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a46000640 1 Processor -- start 2024-04-17T10:33:23.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a46000640 1 -- start start 2024-04-17T10:33:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a46000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2a40163060 0x7f2a40071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a46000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a400719a0 con 0x7f2a400cef80 2024-04-17T10:33:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a46000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a40071c80 con 0x7f2a400b4970 2024-04-17T10:33:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a46000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a40162580 con 0x7f2a40163060 2024-04-17T10:33:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2a40163060 0x7f2a40071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2a400cef80 0x7f2a400b4260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40746/0 (socket says 172.21.15.67:40746) 2024-04-17T10:33:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3f400640 1 -- 172.21.15.67:0/4043672678 learned_addr learned my addr 172.21.15.67:0/4043672678 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3ea00640 1 -- 172.21.15.67:0/4043672678 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358743638 0 0) 0x7f2a400719a0 con 0x7f2a400cef80 2024-04-17T10:33:23.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3ea00640 1 -- 172.21.15.67:0/4043672678 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a24003530 con 0x7f2a400cef80 2024-04-17T10:33:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3fe00640 1 -- 172.21.15.67:0/4043672678 >> v1:172.21.15.86:6789/0 conn(0x7f2a400b4970 legacy=0x7f2a400b2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3fe00640 1 -- 172.21.15.67:0/4043672678 >> v1:172.21.15.67:6789/0 conn(0x7f2a400cef80 legacy=0x7f2a400b4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:23.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.281+0000 7f2a3fe00640 1 -- 172.21.15.67:0/4043672678 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a40162860 con 0x7f2a40163060 2024-04-17T10:33:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a3fe00640 1 --2- 172.21.15.67:0/4043672678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a40163060 0x7f2a40071310 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009d90 tx=0x7f2a3801c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68293ee9371f1d70 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a3ea00640 1 -- 172.21.15.67:0/4043672678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a380b2070 con 0x7f2a40163060 2024-04-17T10:33:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a3ea00640 1 -- 172.21.15.67:0/4043672678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a380a59a0 con 0x7f2a40163060 2024-04-17T10:33:23.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a3ea00640 1 -- 172.21.15.67:0/4043672678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a380a5d70 con 0x7f2a40163060 2024-04-17T10:33:23.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a46000640 1 -- 172.21.15.67:0/4043672678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a40163060 msgr2=0x7f2a40071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:23.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a46000640 1 --2- 172.21.15.67:0/4043672678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a40163060 0x7f2a40071310 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009d90 tx=0x7f2a3801c940 comp rx=0 tx=0).stop 2024-04-17T10:33:23.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a46000640 1 -- 172.21.15.67:0/4043672678 shutdown_connections 2024-04-17T10:33:23.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a46000640 1 --2- 172.21.15.67:0/4043672678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a40163060 0x7f2a40071310 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:23.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a46000640 1 -- 172.21.15.67:0/4043672678 >> 172.21.15.67:0/4043672678 conn(0x7f2a4006ac90 msgr2=0x7f2a4006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:23.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a46000640 1 -- 172.21.15.67:0/4043672678 shutdown_connections 2024-04-17T10:33:23.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.285+0000 7f2a46000640 1 -- 172.21.15.67:0/4043672678 wait complete. 2024-04-17T10:33:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 Processor -- start 2024-04-17T10:33:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 -- start start 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a400b4970 0x7f2a40161340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a400cef80 0x7f2a401452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a40163060 0x7f2a401533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4015f1c0 con 0x7f2a400b4970 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4015f040 con 0x7f2a400cef80 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4015f340 con 0x7f2a40163060 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a400b4970 0x7f2a40161340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a400b4970 0x7f2a40161340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56130/0 (socket says 172.21.15.67:56130) 2024-04-17T10:33:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 -- 172.21.15.67:0/3530847487 learned_addr learned my addr 172.21.15.67:0/3530847487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a37e00640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a400cef80 0x7f2a401452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3fe00640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a40163060 0x7f2a401533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 -- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a40163060 msgr2=0x7f2a401533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a40163060 0x7f2a401533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 -- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a400cef80 msgr2=0x7f2a401452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a400cef80 0x7f2a401452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 -- 172.21.15.67:0/3530847487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a40161880 con 0x7f2a400b4970 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a3f400640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a400b4970 0x7f2a40161340 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000c4d0 tx=0x7f2a3000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a37400640 1 -- 172.21.15.67:0/3530847487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30017070 con 0x7f2a400b4970 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a37400640 1 -- 172.21.15.67:0/3530847487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a30005cf0 con 0x7f2a400b4970 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a401602a0 con 0x7f2a400b4970 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a37400640 1 -- 172.21.15.67:0/3530847487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a30007270 con 0x7f2a400b4970 2024-04-17T10:33:23.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.289+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a40160860 con 0x7f2a400b4970 2024-04-17T10:33:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.293+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a40072d50 con 0x7f2a400b4970 2024-04-17T10:33:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.293+0000 7f2a37400640 1 -- 172.21.15.67:0/3530847487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2a3001a070 con 0x7f2a400b4970 2024-04-17T10:33:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.293+0000 7f2a37400640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2a1c066f60 0x7f2a1c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.293+0000 7f2a37400640 1 -- 172.21.15.67:0/3530847487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f2a300841d0 con 0x7f2a400b4970 2024-04-17T10:33:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.293+0000 7f2a37e00640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2a1c066f60 0x7f2a1c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.293+0000 7f2a37e00640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2a1c066f60 0x7f2a1c069420 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2a4006bc20 tx=0x7f2a28009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:23.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.297+0000 7f2a37400640 1 -- 172.21.15.67:0/3530847487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2a30054cf0 con 0x7f2a400b4970 2024-04-17T10:33:23.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:23.517+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]} v 0) v1 -- 0x7f2a40074e30 con 0x7f2a400b4970 2024-04-17T10:33:24.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a37400640 1 -- 172.21.15.67:0/3530847487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]}]=0 v411) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2a30058d90 con 0x7f2a400b4970 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2a1c066f60 msgr2=0x7f2a1c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2a1c066f60 0x7f2a1c069420 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f2a4006bc20 tx=0x7f2a28009040 comp rx=0 tx=0).stop 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a400b4970 msgr2=0x7f2a40161340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a400b4970 0x7f2a40161340 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f2a3000c4d0 tx=0x7f2a3000c990 comp rx=0 tx=0).stop 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 shutdown_connections 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2a1c066f60 0x7f2a1c069420 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2a40163060 0x7f2a401533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2a400cef80 0x7f2a401452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 --2- 172.21.15.67:0/3530847487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a400b4970 0x7f2a40161340 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 >> 172.21.15.67:0/3530847487 conn(0x7f2a4006ac90 msgr2=0x7f2a400cd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:24.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 shutdown_connections 2024-04-17T10:33:24.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.433+0000 7f2a46000640 1 -- 172.21.15.67:0/3530847487 wait complete. 2024-04-17T10:33:24.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:33:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f952a200640 1 Processor -- start 2024-04-17T10:33:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f952a200640 1 -- start start 2024-04-17T10:33:24.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f952a200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f95240cd030 0x7f952406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f952a200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95240b4990 con 0x7f95240aef60 2024-04-17T10:33:24.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f952a200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f952406ec00 con 0x7f952406e7b0 2024-04-17T10:33:24.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f952a200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95240af340 con 0x7f95240cd030 2024-04-17T10:33:24.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9523400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f95240aef60 0x7f95240cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40754/0 (socket says 172.21.15.67:40754) 2024-04-17T10:33:24.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9523400640 1 -- 172.21.15.67:0/3213232056 learned_addr learned my addr 172.21.15.67:0/3213232056 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523291870 0 0) 0x7f95140014a0 con 0x7f95240aef60 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f951aa00640 1 --2- 172.21.15.67:0/3213232056 >> v2:172.21.15.86:3300/0 conn(0x7f95240cd030 0x7f952406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9508003530 con 0x7f95240aef60 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2132930891 0 0) 0x7f9508003530 con 0x7f95240aef60 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 >> v2:172.21.15.86:3300/0 conn(0x7f95240cd030 msgr2=0x7f952406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 --2- 172.21.15.67:0/3213232056 >> v2:172.21.15.86:3300/0 conn(0x7f95240cd030 0x7f952406e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 >> v1:172.21.15.86:6789/0 conn(0x7f952406e7b0 legacy=0x7f95240b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952415ae90 con 0x7f95240aef60 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f9514001bd0 con 0x7f95240aef60 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 >> v1:172.21.15.67:6789/0 conn(0x7f95240aef60 legacy=0x7f95240cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508003a10 0x7f9508003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f9508004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508005220 0x7f950800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95240af340 con 0x7f9508004810 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950800c170 con 0x7f9508005220 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950800c370 con 0x7f9508003a10 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f951aa00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f9508004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9523e00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508005220 0x7f950800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9523400640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508003a10 0x7f9508003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f951aa00640 1 -- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508003a10 msgr2=0x7f9508003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f951aa00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9508003a10 0x7f9508003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f951aa00640 1 -- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508005220 msgr2=0x7f950800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f951aa00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9508005220 0x7f950800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f9523e00640 1 -- 172.21.15.67:0/3213232056 reap_dead start 2024-04-17T10:33:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.609+0000 7f951aa00640 1 -- 172.21.15.67:0/3213232056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9508003530 con 0x7f9508004810 2024-04-17T10:33:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f951aa00640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f9508004c90 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f95240b2a20 tx=0x7f950c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14822e928bb1bfb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c0096b0 con 0x7f9508004810 2024-04-17T10:33:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950c0a8450 con 0x7f9508004810 2024-04-17T10:33:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f9522a00640 1 -- 172.21.15.67:0/3213232056 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c0a8840 con 0x7f9508004810 2024-04-17T10:33:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f952a200640 1 -- 172.21.15.67:0/3213232056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 msgr2=0x7f9508004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f952a200640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f9508004c90 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f95240b2a20 tx=0x7f950c01ca10 comp rx=0 tx=0).stop 2024-04-17T10:33:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f952a200640 1 -- 172.21.15.67:0/3213232056 shutdown_connections 2024-04-17T10:33:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f952a200640 1 --2- 172.21.15.67:0/3213232056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f9508004c90 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f952a200640 1 -- 172.21.15.67:0/3213232056 >> 172.21.15.67:0/3213232056 conn(0x7f95240b1740 msgr2=0x7f95240b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f952a200640 1 -- 172.21.15.67:0/3213232056 shutdown_connections 2024-04-17T10:33:24.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.613+0000 7f952a200640 1 -- 172.21.15.67:0/3213232056 wait complete. 2024-04-17T10:33:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 Processor -- start 2024-04-17T10:33:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 -- start start 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f95241f7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f952414de10 0x7f9524147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95240b07a0 0x7f95240b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95240cceb0 con 0x7f9508004810 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95240ccd30 con 0x7f95240b07a0 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952415bc10 con 0x7f952414de10 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f95241f7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:24.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f95241f7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56144/0 (socket says 172.21.15.67:56144) 2024-04-17T10:33:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 -- 172.21.15.67:0/1443220241 learned_addr learned my addr 172.21.15.67:0/1443220241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 -- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f952414de10 msgr2=0x7f9524147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f952414de10 0x7f9524147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 -- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95240b07a0 msgr2=0x7f95240b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95240b07a0 0x7f95240b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 -- 172.21.15.67:0/1443220241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f952415ef70 con 0x7f9508004810 2024-04-17T10:33:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f9523400640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f95241f7940 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f9514007a10 tx=0x7f951400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f951be00640 1 -- 172.21.15.67:0/1443220241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f951400ee50 con 0x7f9508004810 2024-04-17T10:33:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f951be00640 1 -- 172.21.15.67:0/1443220241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9514018070 con 0x7f9508004810 2024-04-17T10:33:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f951be00640 1 -- 172.21.15.67:0/1443220241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9514013660 con 0x7f9508004810 2024-04-17T10:33:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95241488c0 con 0x7f9508004810 2024-04-17T10:33:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.617+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952414e2f0 con 0x7f9508004810 2024-04-17T10:33:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.621+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f952404ef30 con 0x7f9508004810 2024-04-17T10:33:24.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.621+0000 7f951be00640 1 -- 172.21.15.67:0/1443220241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9514013800 con 0x7f9508004810 2024-04-17T10:33:24.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.621+0000 7f951be00640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9500066f90 0x7f9500069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:24.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.621+0000 7f951be00640 1 -- 172.21.15.67:0/1443220241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f95140855b0 con 0x7f9508004810 2024-04-17T10:33:24.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.621+0000 7f951aa00640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9500066f90 0x7f9500069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:24.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.621+0000 7f951aa00640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9500066f90 0x7f9500069450 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f950c0af000 tx=0x7f950c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.625+0000 7f951be00640 1 -- 172.21.15.67:0/1443220241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f95241488c0 con 0x7f9508004810 2024-04-17T10:33:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.785+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f95240b0050 con 0x7f9508004810 2024-04-17T10:33:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.789+0000 7f951be00640 1 -- 172.21.15.67:0/1443220241 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f95240b4990 con 0x7f9508004810 2024-04-17T10:33:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9500066f90 msgr2=0x7f9500069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9500066f90 0x7f9500069450 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f950c0af000 tx=0x7f950c008040 comp rx=0 tx=0).stop 2024-04-17T10:33:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 msgr2=0x7f95241f7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f95241f7940 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f9514007a10 tx=0x7f951400e530 comp rx=0 tx=0).stop 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 shutdown_connections 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9500066f90 0x7f9500069450 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f95240b07a0 0x7f95240b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f952414de10 0x7f9524147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 --2- 172.21.15.67:0/1443220241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9508004810 0x7f95241f7940 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 >> 172.21.15.67:0/1443220241 conn(0x7f95240b1740 msgr2=0x7f95240b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 shutdown_connections 2024-04-17T10:33:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:24.793+0000 7f952a200640 1 -- 172.21.15.67:0/1443220241 wait complete. 2024-04-17T10:33:24.805 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:33:24.805 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:33:24.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:33:24.963 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:33:24.964 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:33:24.964 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:24.973 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:33:24.973 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:33:24.973 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:24.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:33:24.983 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:33:24.983 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:25.027 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:33:25.027 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:33:25.027 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:25.036 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:33:25.036 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:33:25.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:25.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.221+0000 7f06ffa00640 1 Processor -- start 2024-04-17T10:33:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06ffa00640 1 -- start start 2024-04-17T10:33:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06ffa00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f06f80714d0 0x7f06f8168c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06ffa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b5980 con 0x7f06f815dd10 2024-04-17T10:33:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06ffa00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f80b0cc0 con 0x7f06f80b07b0 2024-04-17T10:33:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06ffa00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06f80b1280 con 0x7f06f80714d0 2024-04-17T10:33:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06fd000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f06f815dd10 0x7f06f80b5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40764/0 (socket says 172.21.15.67:40764) 2024-04-17T10:33:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06fd000640 1 -- 172.21.15.67:0/823149526 learned_addr learned my addr 172.21.15.67:0/823149526 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06fda00640 1 --2- 172.21.15.67:0/823149526 >> v2:172.21.15.86:3300/0 conn(0x7f06f80714d0 0x7f06f8168c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06f7400640 1 -- 172.21.15.67:0/823149526 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 682638001 0 0) 0x7f06f80b5980 con 0x7f06f815dd10 2024-04-17T10:33:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06f7400640 1 -- 172.21.15.67:0/823149526 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06e0003530 con 0x7f06f815dd10 2024-04-17T10:33:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06fda00640 1 -- 172.21.15.67:0/823149526 >> v1:172.21.15.86:6789/0 conn(0x7f06f80b07b0 legacy=0x7f06f8070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06fda00640 1 -- 172.21.15.67:0/823149526 >> v1:172.21.15.67:6789/0 conn(0x7f06f815dd10 legacy=0x7f06f80b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06f7400640 1 -- 172.21.15.67:0/823149526 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1802919402 0 0) 0x7f06e0003530 con 0x7f06f815dd10 2024-04-17T10:33:25.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.225+0000 7f06fda00640 1 -- 172.21.15.67:0/823149526 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f80b0fa0 con 0x7f06f80714d0 2024-04-17T10:33:25.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06fda00640 1 --2- 172.21.15.67:0/823149526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f80714d0 0x7f06f8168c50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009d90 tx=0x7f06f001c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d0c1729555cc44 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:25.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06f7400640 1 -- 172.21.15.67:0/823149526 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06f00b2070 con 0x7f06f80714d0 2024-04-17T10:33:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06f7400640 1 -- 172.21.15.67:0/823149526 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06f00a5610 con 0x7f06f80714d0 2024-04-17T10:33:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06f7400640 1 -- 172.21.15.67:0/823149526 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec001680 con 0x7f06f80714d0 2024-04-17T10:33:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 -- 172.21.15.67:0/823149526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f80714d0 msgr2=0x7f06f8168c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 --2- 172.21.15.67:0/823149526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f80714d0 0x7f06f8168c50 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f06f0009d90 tx=0x7f06f001c7d0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 -- 172.21.15.67:0/823149526 shutdown_connections 2024-04-17T10:33:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 --2- 172.21.15.67:0/823149526 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f80714d0 0x7f06f8168c50 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 -- 172.21.15.67:0/823149526 >> 172.21.15.67:0/823149526 conn(0x7f06f80b34d0 msgr2=0x7f06f80b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 -- 172.21.15.67:0/823149526 shutdown_connections 2024-04-17T10:33:25.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 -- 172.21.15.67:0/823149526 wait complete. 2024-04-17T10:33:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 Processor -- start 2024-04-17T10:33:25.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.229+0000 7f06ffa00640 1 -- start start 2024-04-17T10:33:25.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80714d0 0x7f06f814c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f80b07b0 0x7f06f8152fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815dd10 0x7f06f8153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f816bb30 con 0x7f06f80714d0 2024-04-17T10:33:25.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f816b9b0 con 0x7f06f815dd10 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f816bcb0 con 0x7f06f80b07b0 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80714d0 0x7f06f814c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80714d0 0x7f06f814c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56146/0 (socket says 172.21.15.67:56146) 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 -- 172.21.15.67:0/637176095 learned_addr learned my addr 172.21.15.67:0/637176095 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fda00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815dd10 0x7f06f8153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06f7e00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f80b07b0 0x7f06f8152fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 -- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f80b07b0 msgr2=0x7f06f8152fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f80b07b0 0x7f06f8152fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 -- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815dd10 msgr2=0x7f06f8153500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815dd10 0x7f06f8153500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 -- 172.21.15.67:0/637176095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f8152370 con 0x7f06f80714d0 2024-04-17T10:33:25.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06fd000640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80714d0 0x7f06f814c870 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00c520 tx=0x7f06ec00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06f4c00640 1 -- 172.21.15.67:0/637176095 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec017070 con 0x7f06f80714d0 2024-04-17T10:33:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06f4c00640 1 -- 172.21.15.67:0/637176095 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec005d10 con 0x7f06f80714d0 2024-04-17T10:33:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06f4c00640 1 -- 172.21.15.67:0/637176095 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec0072b0 con 0x7f06f80714d0 2024-04-17T10:33:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f81589f0 con 0x7f06f80714d0 2024-04-17T10:33:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f8158fb0 con 0x7f06f80714d0 2024-04-17T10:33:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f8159670 con 0x7f06f80714d0 2024-04-17T10:33:25.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.233+0000 7f06f4c00640 1 -- 172.21.15.67:0/637176095 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f06ec01a070 con 0x7f06f80714d0 2024-04-17T10:33:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.237+0000 7f06f4c00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.237+0000 7f06f4c00640 1 -- 172.21.15.67:0/637176095 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f06ec0843e0 con 0x7f06f80714d0 2024-04-17T10:33:25.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.237+0000 7f06f7e00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.237+0000 7f06f7e00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f06f8070e70 tx=0x7f06e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:25.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.241+0000 7f06f4c00640 1 -- 172.21.15.67:0/637176095 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f06f8159670 con 0x7f06f80714d0 2024-04-17T10:33:25.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.433+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06f8070bb0 con 0x7f06f80714d0 2024-04-17T10:33:25.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06f4c00640 1 -- 172.21.15.67:0/637176095 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+3693 (secure 0 0 0) 0x7f06ec054df0 con 0x7f06f80714d0 2024-04-17T10:33:25.428 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:25.428 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":75,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7591,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/977762372","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":977762372},{"type":"v1","addr":"172.21.15.67:6835","nonce":977762372}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7600,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/1954911271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":1954911271},{"type":"v1","addr":"172.21.15.86:6836","nonce":1954911271}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7603,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/2103586930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":2103586930},{"type":"v1","addr":"172.21.15.86:6837","nonce":2103586930}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":8589,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/238390079","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":238390079},{"type":"v1","addr":"172.21.15.67:6834","nonce":238390079}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":75}],"filesystems":[]} 2024-04-17T10:33:25.428 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 75 2024-04-17T10:33:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 msgr2=0x7f06d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f06f8070e70 tx=0x7f06e4009040 comp rx=0 tx=0).stop 2024-04-17T10:33:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80714d0 msgr2=0x7f06f814c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80714d0 0x7f06f814c870 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f06ec00c520 tx=0x7f06ec00c9e0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 shutdown_connections 2024-04-17T10:33:25.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f06d8066f60 0x7f06d8069420 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06f815dd10 0x7f06f8153500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06f80b07b0 0x7f06f8152fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 --2- 172.21.15.67:0/637176095 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06f80714d0 0x7f06f814c870 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 >> 172.21.15.67:0/637176095 conn(0x7f06f80b34d0 msgr2=0x7f06f80d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 shutdown_connections 2024-04-17T10:33:25.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.437+0000 7f06ffa00640 1 -- 172.21.15.67:0/637176095 wait complete. 2024-04-17T10:33:25.449 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:33:25.449 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:33:25.449 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:33:25.450 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:33:25.450 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:33:25.450 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:33:25.450 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:33:25.450 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:33:25.450 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:33:25.450 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:33:25.451 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:33:25.451 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:33:25.451 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:33:25.451 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:33:25.451 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:33:25.451 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:33:25.462 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:33:25.462 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:33:25.463 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:33:25.463 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:33:25.464 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:33:25.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:33:25.465 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:33:25.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:33:25.510 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:33:25.510 DEBUG:teuthology.parallel:result is None 2024-04-17T10:33:25.511 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:33:25.511 DEBUG:teuthology.parallel:result is None 2024-04-17T10:33:25.511 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:33:25.511 DEBUG:teuthology.parallel:result is None 2024-04-17T10:33:25.512 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:33:25.512 DEBUG:teuthology.parallel:result is None 2024-04-17T10:33:25.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:33:25.597 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:33:25.598 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:33:25.600 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:33:25.600 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:33:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.665+0000 7fc0ce000640 1 Processor -- start 2024-04-17T10:33:25.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.665+0000 7fc0ce000640 1 -- start start 2024-04-17T10:33:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0ce000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80cc250 0x7fc0c80cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0ce000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc0c80cb260 0x7fc0c80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0ce000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c80ce2a0 0x7fc0c80cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0ce000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c80cd150 con 0x7fc0c80ce2a0 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0ce000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c806b100 con 0x7fc0c80cb260 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0ce000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c806b3e0 con 0x7fc0c80cc250 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c80ce2a0 0x7fc0c80cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc0c80ce2a0 0x7fc0c80cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56174/0 (socket says 172.21.15.67:56174) 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 -- 172.21.15.67:0/3108090207 learned_addr learned my addr 172.21.15.67:0/3108090207 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0bea00640 1 --2- 172.21.15.67:0/3108090207 >> v2:172.21.15.86:3300/0 conn(0x7fc0c80cb260 0x7fc0c80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7400640 1 --2- 172.21.15.67:0/3108090207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80cc250 0x7fc0c80cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 -- 172.21.15.67:0/3108090207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80cc250 msgr2=0x7fc0c80cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 --2- 172.21.15.67:0/3108090207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80cc250 0x7fc0c80cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 -- 172.21.15.67:0/3108090207 >> v2:172.21.15.86:3300/0 conn(0x7fc0c80cb260 msgr2=0x7fc0c80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 --2- 172.21.15.67:0/3108090207 >> v2:172.21.15.86:3300/0 conn(0x7fc0c80cb260 0x7fc0c80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.669+0000 7fc0c7e00640 1 -- 172.21.15.67:0/3108090207 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c806b6c0 con 0x7fc0c80ce2a0 2024-04-17T10:33:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0c7e00640 1 --2- 172.21.15.67:0/3108090207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80ce2a0 0x7fc0c80cdb20 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80d1730 tx=0x7fc0c001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a6d961ed7a6267 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:25.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0c6a00640 1 -- 172.21.15.67:0/3108090207 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c00a5670 con 0x7fc0c80ce2a0 2024-04-17T10:33:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0c6a00640 1 -- 172.21.15.67:0/3108090207 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c00a5810 con 0x7fc0c80ce2a0 2024-04-17T10:33:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0c6a00640 1 -- 172.21.15.67:0/3108090207 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c00a5be0 con 0x7fc0c80ce2a0 2024-04-17T10:33:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- 172.21.15.67:0/3108090207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80ce2a0 msgr2=0x7fc0c80cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 --2- 172.21.15.67:0/3108090207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80ce2a0 0x7fc0c80cdb20 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80d1730 tx=0x7fc0c001c9f0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- 172.21.15.67:0/3108090207 shutdown_connections 2024-04-17T10:33:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 --2- 172.21.15.67:0/3108090207 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80ce2a0 0x7fc0c80cdb20 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 --2- 172.21.15.67:0/3108090207 >> v2:172.21.15.86:3300/0 conn(0x7fc0c80cb260 0x7fc0c80cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 --2- 172.21.15.67:0/3108090207 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80cc250 0x7fc0c80cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- 172.21.15.67:0/3108090207 >> 172.21.15.67:0/3108090207 conn(0x7fc0c80b22a0 msgr2=0x7fc0c80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- 172.21.15.67:0/3108090207 shutdown_connections 2024-04-17T10:33:25.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- 172.21.15.67:0/3108090207 wait complete. 2024-04-17T10:33:25.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 Processor -- start 2024-04-17T10:33:25.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- start start 2024-04-17T10:33:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80cb260 0x7fc0c8158970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c80cc250 0x7fc0c814f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80ce2a0 0x7fc0c80b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c806e850 con 0x7fc0c80cb260 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c806e6d0 con 0x7fc0c80cc250 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0ce000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c806e9d0 con 0x7fc0c80ce2a0 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0c7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80cb260 0x7fc0c8158970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0c7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80cb260 0x7fc0c8158970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56190/0 (socket says 172.21.15.67:56190) 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.673+0000 7fc0c7400640 1 -- 172.21.15.67:0/1632628467 learned_addr learned my addr 172.21.15.67:0/1632628467 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0c7e00640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80ce2a0 0x7fc0c80b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0bea00640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c80cc250 0x7fc0c814f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0c7400640 1 -- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80ce2a0 msgr2=0x7fc0c80b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0c7400640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80ce2a0 0x7fc0c80b00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0c7400640 1 -- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c80cc250 msgr2=0x7fc0c814f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0c7400640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c80cc250 0x7fc0c814f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:25.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0c7400640 1 -- 172.21.15.67:0/1632628467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80c55a0 con 0x7fc0c80cb260 2024-04-17T10:33:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0c7400640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80cb260 0x7fc0c8158970 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800e930 tx=0x7fc0b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0bfe00640 1 -- 172.21.15.67:0/1632628467 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b8017070 con 0x7fc0c80cb260 2024-04-17T10:33:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0bfe00640 1 -- 172.21.15.67:0/1632628467 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b800ca00 con 0x7fc0c80cb260 2024-04-17T10:33:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0bfe00640 1 -- 172.21.15.67:0/1632628467 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b800cdb0 con 0x7fc0c80cb260 2024-04-17T10:33:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c81591c0 con 0x7fc0c80cb260 2024-04-17T10:33:25.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c8159780 con 0x7fc0c80cb260 2024-04-17T10:33:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c809e900 con 0x7fc0c80cb260 2024-04-17T10:33:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0bfe00640 1 -- 172.21.15.67:0/1632628467 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc0b801a070 con 0x7fc0c80cb260 2024-04-17T10:33:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0bfe00640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc0a4066f60 0x7fc0a4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.677+0000 7fc0bfe00640 1 -- 172.21.15.67:0/1632628467 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5335+0+0 (secure 0 0 0) 0x7fc0b8084e40 con 0x7fc0c80cb260 2024-04-17T10:33:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.681+0000 7fc0bea00640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc0a4066f60 0x7fc0a4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:25.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.681+0000 7fc0bea00640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc0a4066f60 0x7fc0a4069420 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b6840 tx=0x7fc0b0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:25.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.681+0000 7fc0bfe00640 1 -- 172.21.15.67:0/1632628467 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc0b8055a20 con 0x7fc0c80cb260 2024-04-17T10:33:25.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:25.857+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc0c8074e30 con 0x7fc0c80cb260 2024-04-17T10:33:26.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.445+0000 7fc0bfe00640 1 -- 172.21.15.67:0/1632628467 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v112) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc0b8059ac0 con 0x7fc0c80cb260 2024-04-17T10:33:26.439 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:33:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc0a4066f60 msgr2=0x7fc0a4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc0a4066f60 0x7fc0a4069420 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80b6840 tx=0x7fc0b0009190 comp rx=0 tx=0).stop 2024-04-17T10:33:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80cb260 msgr2=0x7fc0c8158970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:26.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80cb260 0x7fc0c8158970 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fc0b800e930 tx=0x7fc0b800edf0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 shutdown_connections 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc0a4066f60 0x7fc0a4069420 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc0c80ce2a0 0x7fc0c80b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc0c80cc250 0x7fc0c814f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 --2- 172.21.15.67:0/1632628467 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc0c80cb260 0x7fc0c8158970 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 >> 172.21.15.67:0/1632628467 conn(0x7fc0c80b22a0 msgr2=0x7fc0c80caa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 shutdown_connections 2024-04-17T10:33:26.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.449+0000 7fc0ce000640 1 -- 172.21.15.67:0/1632628467 wait complete. 2024-04-17T10:33:26.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:33:26.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 Processor -- start 2024-04-17T10:33:26.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 -- start start 2024-04-17T10:33:26.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd64c0aefa0 0x7fd64c06df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd64c15c9d0 0x7fd64c0cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c071650 0x7fd64c071ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c06dac0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64c0ce1b0 con 0x7fd64c15c9d0 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0ce490 con 0x7fd64c071650 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd651600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd64c0aefa0 0x7fd64c06df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd651600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd64c0aefa0 0x7fd64c06df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56204/0 (socket says 172.21.15.67:56204) 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd651600640 1 -- 172.21.15.67:0/4001619526 learned_addr learned my addr 172.21.15.67:0/4001619526 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd652000640 1 --2- 172.21.15.67:0/4001619526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c071650 0x7fd64c071ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.613+0000 7fd650c00640 1 --2- 172.21.15.67:0/4001619526 >> v2:172.21.15.86:3300/0 conn(0x7fd64c15c9d0 0x7fd64c0cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd651600640 1 -- 172.21.15.67:0/4001619526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c071650 msgr2=0x7fd64c071ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd651600640 1 --2- 172.21.15.67:0/4001619526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c071650 0x7fd64c071ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd651600640 1 -- 172.21.15.67:0/4001619526 >> v2:172.21.15.86:3300/0 conn(0x7fd64c15c9d0 msgr2=0x7fd64c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:26.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd651600640 1 --2- 172.21.15.67:0/4001619526 >> v2:172.21.15.86:3300/0 conn(0x7fd64c15c9d0 0x7fd64c0cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd651600640 1 -- 172.21.15.67:0/4001619526 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c06d470 con 0x7fd64c0aefa0 2024-04-17T10:33:26.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd650c00640 1 --2- 172.21.15.67:0/4001619526 >> v2:172.21.15.86:3300/0 conn(0x7fd64c15c9d0 0x7fd64c0cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:26.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd651600640 1 --2- 172.21.15.67:0/4001619526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c06df50 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0723c0 tx=0x7fd64001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952be787962da839 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:26.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd64be00640 1 -- 172.21.15.67:0/4001619526 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400b5070 con 0x7fd64c0aefa0 2024-04-17T10:33:26.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd64be00640 1 -- 172.21.15.67:0/4001619526 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6400a89d0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.617+0000 7fd64be00640 1 -- 172.21.15.67:0/4001619526 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6400a8da0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- 172.21.15.67:0/4001619526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 msgr2=0x7fd64c06df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 --2- 172.21.15.67:0/4001619526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c06df50 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0723c0 tx=0x7fd64001ed10 comp rx=0 tx=0).stop 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- 172.21.15.67:0/4001619526 shutdown_connections 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 --2- 172.21.15.67:0/4001619526 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c071650 0x7fd64c071ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 --2- 172.21.15.67:0/4001619526 >> v2:172.21.15.86:3300/0 conn(0x7fd64c15c9d0 0x7fd64c0cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 --2- 172.21.15.67:0/4001619526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c06df50 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- 172.21.15.67:0/4001619526 >> 172.21.15.67:0/4001619526 conn(0x7fd64c06ac90 msgr2=0x7fd64c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- 172.21.15.67:0/4001619526 shutdown_connections 2024-04-17T10:33:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- 172.21.15.67:0/4001619526 wait complete. 2024-04-17T10:33:26.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 Processor -- start 2024-04-17T10:33:26.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- start start 2024-04-17T10:33:26.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd64c071650 0x7fd64c0ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c156480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c15c9d0 0x7fd64c144380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c15cea0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c0cde30 con 0x7fd64c071650 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64c15d020 con 0x7fd64c15c9d0 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c156480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c156480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56214/0 (socket says 172.21.15.67:56214) 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 -- 172.21.15.67:0/1393470835 learned_addr learned my addr 172.21.15.67:0/1393470835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd652000640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c15c9d0 0x7fd64c144380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd651600640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd64c071650 0x7fd64c0ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 -- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c15c9d0 msgr2=0x7fd64c144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c15c9d0 0x7fd64c144380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 -- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd64c071650 msgr2=0x7fd64c0ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd64c071650 0x7fd64c0ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd650c00640 1 -- 172.21.15.67:0/1393470835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c0ad9b0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.621+0000 7fd651600640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd64c071650 0x7fd64c0ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd650c00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c156480 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fd63800d800 tx=0x7fd63800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd649600640 1 -- 172.21.15.67:0/1393470835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638018020 con 0x7fd64c0aefa0 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd649600640 1 -- 172.21.15.67:0/1393470835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6380096c0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64c0adc90 con 0x7fd64c0aefa0 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd649600640 1 -- 172.21.15.67:0/1393470835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd638009ab0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64c0b31a0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd64c074dc0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd649600640 1 -- 172.21.15.67:0/1393470835 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd63801b020 con 0x7fd64c0aefa0 2024-04-17T10:33:26.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd649600640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd62c066f60 0x7fd62c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:26.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd649600640 1 -- 172.21.15.67:0/1393470835 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fd6380853d0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.625+0000 7fd651600640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd62c066f60 0x7fd62c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:26.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.629+0000 7fd651600640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd62c066f60 0x7fd62c069420 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd6400b2000 tx=0x7fd6400aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:26.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.629+0000 7fd649600640 1 -- 172.21.15.67:0/1393470835 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd638022020 con 0x7fd64c0aefa0 2024-04-17T10:33:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.853+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd64c15b3a0 con 0x7fd64c0aefa0 2024-04-17T10:33:26.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.853+0000 7fd649600640 1 -- 172.21.15.67:0/1393470835 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd638056590 con 0x7fd64c0aefa0 2024-04-17T10:33:26.846 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:26.846 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:26.846 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:33:26.847 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:33:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:33:26.849 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.850 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.851 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.852 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.853 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.854 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.855 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.856 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.857 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:33:26.858 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.859 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.860 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:26.861 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:33:26.861 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd62c066f60 msgr2=0x7fd62c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd62c066f60 0x7fd62c069420 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd6400b2000 tx=0x7fd6400aa070 comp rx=0 tx=0).stop 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 msgr2=0x7fd64c156480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c156480 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7fd63800d800 tx=0x7fd63800dcc0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 shutdown_connections 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd62c066f60 0x7fd62c069420 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd64c15c9d0 0x7fd64c144380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd64c0aefa0 0x7fd64c156480 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 --2- 172.21.15.67:0/1393470835 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd64c071650 0x7fd64c0ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 >> 172.21.15.67:0/1393470835 conn(0x7fd64c06ac90 msgr2=0x7fd64c15ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 shutdown_connections 2024-04-17T10:33:26.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:26.857+0000 7fd652a00640 1 -- 172.21.15.67:0/1393470835 wait complete. 2024-04-17T10:33:26.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b7400640 1 Processor -- start 2024-04-17T10:33:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b7400640 1 -- start start 2024-04-17T10:33:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b7400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3b01620a0 0x7fc3b015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b00cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b7400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3b015eb50 con 0x7fc3b00ce020 2024-04-17T10:33:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b7400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3b0161650 con 0x7fc3b01620a0 2024-04-17T10:33:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b7400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b015ee30 con 0x7fc3b015e550 2024-04-17T10:33:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3b5400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b00cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.017+0000 7fc3a7e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3b01620a0 0x7fc3b015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3a7e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3b01620a0 0x7fc3b015e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:41664/0 (socket says 172.21.15.67:41664) 2024-04-17T10:33:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3a7e00640 1 -- 172.21.15.67:0/1939059562 learned_addr learned my addr 172.21.15.67:0/1939059562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3b5400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b00cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:37186/0 (socket says 172.21.15.67:37186) 2024-04-17T10:33:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3b4a00640 1 --1- 172.21.15.67:0/1939059562 >> v1:172.21.15.86:6789/0 conn(0x7fc3b00ce020 0x7fc3b00ce400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:41072/0 (socket says 172.21.15.67:41072) 2024-04-17T10:33:27.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3a7e00640 1 -- 172.21.15.67:0/1939059562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 msgr2=0x7fc3b00cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3a7e00640 1 --2- 172.21.15.67:0/1939059562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b00cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3a7e00640 1 -- 172.21.15.67:0/1939059562 >> v1:172.21.15.86:6789/0 conn(0x7fc3b00ce020 legacy=0x7fc3b00ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3a7e00640 1 -- 172.21.15.67:0/1939059562 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b01618a0 con 0x7fc3b01620a0 2024-04-17T10:33:27.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3afe00640 1 -- 172.21.15.67:0/1939059562 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318023054 0 0) 0x7fc3b015eb50 con 0x7fc3b00ce020 2024-04-17T10:33:27.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3a7e00640 1 --2- 172.21.15.67:0/1939059562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015e010 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc3980098a0 tx=0x7fc39801c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6797a6b68c63488 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3afe00640 1 -- 172.21.15.67:0/1939059562 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3980ab620 con 0x7fc3b01620a0 2024-04-17T10:33:27.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3afe00640 1 -- 172.21.15.67:0/1939059562 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3980b0070 con 0x7fc3b01620a0 2024-04-17T10:33:27.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.021+0000 7fc3afe00640 1 -- 172.21.15.67:0/1939059562 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3980ab9e0 con 0x7fc3b01620a0 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- 172.21.15.67:0/1939059562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 msgr2=0x7fc3b015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1939059562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015e010 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fc3980098a0 tx=0x7fc39801c510 comp rx=0 tx=0).stop 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- 172.21.15.67:0/1939059562 shutdown_connections 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1939059562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b00cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1939059562 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015e010 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- 172.21.15.67:0/1939059562 >> 172.21.15.67:0/1939059562 conn(0x7fc3b00b33f0 msgr2=0x7fc3b00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- 172.21.15.67:0/1939059562 shutdown_connections 2024-04-17T10:33:27.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- 172.21.15.67:0/1939059562 wait complete. 2024-04-17T10:33:27.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 Processor -- start 2024-04-17T10:33:27.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- start start 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b00ce020 0x7fc3b015a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b015b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00c8630 con 0x7fc3b00ce020 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b00c84b0 con 0x7fc3b01620a0 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b7400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b016a9f0 con 0x7fc3b015e550 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b00ce020 0x7fc3b015a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b00ce020 0x7fc3b015a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56222/0 (socket says 172.21.15.67:56222) 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 -- 172.21.15.67:0/1693744978 learned_addr learned my addr 172.21.15.67:0/1693744978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:27.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3a7e00640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b015b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b5400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 -- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 msgr2=0x7fc3b015b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b015b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 -- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 msgr2=0x7fc3b015cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 -- 172.21.15.67:0/1693744978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b01fa360 con 0x7fc3b00ce020 2024-04-17T10:33:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b5400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:27.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.025+0000 7fc3b4a00640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b00ce020 0x7fc3b015a630 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00011f0 tx=0x7fc3a000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3ad600640 1 -- 172.21.15.67:0/1693744978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0018070 con 0x7fc3b00ce020 2024-04-17T10:33:27.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3ad600640 1 -- 172.21.15.67:0/1693744978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a00074e0 con 0x7fc3b00ce020 2024-04-17T10:33:27.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3ad600640 1 -- 172.21.15.67:0/1693744978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0004a10 con 0x7fc3b00ce020 2024-04-17T10:33:27.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b01fa640 con 0x7fc3b00ce020 2024-04-17T10:33:27.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b01fac00 con 0x7fc3b00ce020 2024-04-17T10:33:27.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3b0070ab0 con 0x7fc3b00ce020 2024-04-17T10:33:27.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3ad600640 1 -- 172.21.15.67:0/1693744978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc3a0004bb0 con 0x7fc3b00ce020 2024-04-17T10:33:27.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3ad600640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc38c066f60 0x7fc38c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3ad600640 1 -- 172.21.15.67:0/1693744978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc3a0084150 con 0x7fc3b00ce020 2024-04-17T10:33:27.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.029+0000 7fc3a7e00640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc38c066f60 0x7fc38c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.033+0000 7fc3a7e00640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc38c066f60 0x7fc38c069420 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc3980af000 tx=0x7fc3980a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.033+0000 7fc3ad600640 1 -- 172.21.15.67:0/1693744978 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc3a0055310 con 0x7fc3b00ce020 2024-04-17T10:33:27.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3b00cf040 con 0x7fc3b00ce020 2024-04-17T10:33:27.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3ad600640 1 -- 172.21.15.67:0/1693744978 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+13126 (secure 0 0 0) 0x7fc3a0054c10 con 0x7fc3b00ce020 2024-04-17T10:33:27.189 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:27.189 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":114,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:26.464361+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":105,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc38c066f60 msgr2=0x7fc38c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc38c066f60 0x7fc38c069420 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fc3980af000 tx=0x7fc3980a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b00ce020 msgr2=0x7fc3b015a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b00ce020 0x7fc3b015a630 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00011f0 tx=0x7fc3a000c9b0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 shutdown_connections 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc38c066f60 0x7fc38c069420 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3b01620a0 0x7fc3b015cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3b015e550 0x7fc3b015b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 --2- 172.21.15.67:0/1693744978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3b00ce020 0x7fc3b015a630 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 >> 172.21.15.67:0/1693744978 conn(0x7fc3b00b33f0 msgr2=0x7fc3b00ce400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 shutdown_connections 2024-04-17T10:33:27.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.197+0000 7fc3b7400640 1 -- 172.21.15.67:0/1693744978 wait complete. 2024-04-17T10:33:27.209 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:33:27.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:33:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f7740600640 1 Processor -- start 2024-04-17T10:33:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f7740600640 1 -- start start 2024-04-17T10:33:27.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f7740600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f773815dd60 0x7f7738161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f7740600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77380d06a0 0x7f7738161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f7740600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773815e140 con 0x7f773809adf0 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f7740600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77381626d0 con 0x7f77380d06a0 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f7740600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7738099d50 con 0x7f773815dd60 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.361+0000 7f773e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77380d06a0 0x7f7738161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77380d06a0 0x7f7738161dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56236/0 (socket says 172.21.15.67:56236) 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773e600640 1 -- 172.21.15.67:0/820947420 learned_addr learned my addr 172.21.15.67:0/820947420 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773dc00640 1 --2- 172.21.15.67:0/820947420 >> v2:172.21.15.86:3300/0 conn(0x7f773815dd60 0x7f7738161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773e600640 1 -- 172.21.15.67:0/820947420 >> v2:172.21.15.86:3300/0 conn(0x7f773815dd60 msgr2=0x7f7738161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773e600640 1 --2- 172.21.15.67:0/820947420 >> v2:172.21.15.86:3300/0 conn(0x7f773815dd60 0x7f7738161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773e600640 1 -- 172.21.15.67:0/820947420 >> v1:172.21.15.86:6789/0 conn(0x7f773809adf0 legacy=0x7f77380cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:27.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773e600640 1 -- 172.21.15.67:0/820947420 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773809a030 con 0x7f77380d06a0 2024-04-17T10:33:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f773e600640 1 --2- 172.21.15.67:0/820947420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f7738161dc0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f773806f620 tx=0x7f773001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d781f0d3c9eab59 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f7737e00640 1 -- 172.21.15.67:0/820947420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77300a5840 con 0x7f77380d06a0 2024-04-17T10:33:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f7737e00640 1 -- 172.21.15.67:0/820947420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77300a59e0 con 0x7f77380d06a0 2024-04-17T10:33:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f7737e00640 1 -- 172.21.15.67:0/820947420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77300a5db0 con 0x7f77380d06a0 2024-04-17T10:33:27.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.365+0000 7f7740600640 1 -- 172.21.15.67:0/820947420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 msgr2=0x7f7738161dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 --2- 172.21.15.67:0/820947420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f7738161dc0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f773806f620 tx=0x7f773001c980 comp rx=0 tx=0).stop 2024-04-17T10:33:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- 172.21.15.67:0/820947420 shutdown_connections 2024-04-17T10:33:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 --2- 172.21.15.67:0/820947420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f7738161dc0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 --2- 172.21.15.67:0/820947420 >> v2:172.21.15.86:3300/0 conn(0x7f773815dd60 0x7f7738161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- 172.21.15.67:0/820947420 >> 172.21.15.67:0/820947420 conn(0x7f77380b34a0 msgr2=0x7f77380b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- 172.21.15.67:0/820947420 shutdown_connections 2024-04-17T10:33:27.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- 172.21.15.67:0/820947420 wait complete. 2024-04-17T10:33:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 Processor -- start 2024-04-17T10:33:27.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- start start 2024-04-17T10:33:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f773809adf0 0x7f7738153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f77381529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f773815dd60 0x7f77380c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773806c730 con 0x7f77380d06a0 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773806c5b0 con 0x7f773815dd60 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7740600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773806c8b0 con 0x7f773809adf0 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f77381529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f77381529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56248/0 (socket says 172.21.15.67:56248) 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 -- 172.21.15.67:0/607889538 learned_addr learned my addr 172.21.15.67:0/607889538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773e600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f773815dd60 0x7f77380c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773dc00640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f773809adf0 0x7f7738153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 -- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f773809adf0 msgr2=0x7f7738153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f773809adf0 0x7f7738153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 -- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f773815dd60 msgr2=0x7f77380c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f773815dd60 0x7f77380c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 -- 172.21.15.67:0/607889538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77380c8340 con 0x7f77380d06a0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f773d200640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f77381529d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f772400c450 tx=0x7f772400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.369+0000 7f7735600640 1 -- 172.21.15.67:0/607889538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724012830 con 0x7f77380d06a0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7735600640 1 -- 172.21.15.67:0/607889538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7724017070 con 0x7f77380d06a0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7735600640 1 -- 172.21.15.67:0/607889538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724012c30 con 0x7f77380d06a0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7738151c10 con 0x7f77380d06a0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77381521d0 con 0x7f77380d06a0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773804ef30 con 0x7f77380d06a0 2024-04-17T10:33:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7735600640 1 -- 172.21.15.67:0/607889538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f772401c460 con 0x7f77380d06a0 2024-04-17T10:33:27.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7735600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7718066f60 0x7f7718069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f7735600640 1 -- 172.21.15.67:0/607889538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f7724084bb0 con 0x7f77380d06a0 2024-04-17T10:33:27.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.373+0000 7f773dc00640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7718066f60 0x7f7718069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.377+0000 7f773dc00640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7718066f60 0x7f7718069420 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f77380b3100 tx=0x7f772c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.377+0000 7f7735600640 1 -- 172.21.15.67:0/607889538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7724055690 con 0x7f77380d06a0 2024-04-17T10:33:27.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.545+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f77380cf050 con 0x7f77380d06a0 2024-04-17T10:33:27.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.545+0000 7f7735600640 1 -- 172.21.15.67:0/607889538 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v114) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7724059730 con 0x7f77380d06a0 2024-04-17T10:33:27.538 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7718066f60 msgr2=0x7f7718069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7718066f60 0x7f7718069420 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f77380b3100 tx=0x7f772c008040 comp rx=0 tx=0).stop 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 msgr2=0x7f77381529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f77381529d0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f772400c450 tx=0x7f772400c910 comp rx=0 tx=0).stop 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 shutdown_connections 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7718066f60 0x7f7718069420 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f773815dd60 0x7f77380c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77380d06a0 0x7f77381529d0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 --2- 172.21.15.67:0/607889538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f773809adf0 0x7f7738153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 >> 172.21.15.67:0/607889538 conn(0x7f77380b34a0 msgr2=0x7f77380b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 shutdown_connections 2024-04-17T10:33:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.549+0000 7f7740600640 1 -- 172.21.15.67:0/607889538 wait complete. 2024-04-17T10:33:27.559 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:33:27.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf2d600640 1 Processor -- start 2024-04-17T10:33:27.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf2d600640 1 -- start start 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf2d600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf281446c0 0x7fdf28144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf2d600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdf28071840 0x7fdf280b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf2d600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf281456d0 con 0x7fdf28144aa0 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf2d600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf280b51a0 con 0x7fdf281446c0 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf2d600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf280b4870 con 0x7fdf28071840 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf26a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf281446c0 0x7fdf28144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf26a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdf281446c0 0x7fdf28144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56254/0 (socket says 172.21.15.67:56254) 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf26a00640 1 -- 172.21.15.67:0/3995789595 learned_addr learned my addr 172.21.15.67:0/3995789595 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:27.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf27400640 1 --2- 172.21.15.67:0/3995789595 >> v2:172.21.15.86:3300/0 conn(0x7fdf28071840 0x7fdf280b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf26a00640 1 -- 172.21.15.67:0/3995789595 >> v2:172.21.15.86:3300/0 conn(0x7fdf28071840 msgr2=0x7fdf280b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:27.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf26a00640 1 --2- 172.21.15.67:0/3995789595 >> v2:172.21.15.86:3300/0 conn(0x7fdf28071840 0x7fdf280b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf26a00640 1 -- 172.21.15.67:0/3995789595 >> v1:172.21.15.86:6789/0 conn(0x7fdf28144aa0 legacy=0x7fdf280d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.709+0000 7fdf26a00640 1 -- 172.21.15.67:0/3995789595 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf280b4b50 con 0x7fdf281446c0 2024-04-17T10:33:27.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf26a00640 1 --2- 172.21.15.67:0/3995789595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf28144ea0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fdf280b5320 tx=0x7fdf1801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee7583b31060717a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf26000640 1 -- 172.21.15.67:0/3995789595 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf180ab600 con 0x7fdf281446c0 2024-04-17T10:33:27.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf26000640 1 -- 172.21.15.67:0/3995789595 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf180b0070 con 0x7fdf281446c0 2024-04-17T10:33:27.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf26000640 1 -- 172.21.15.67:0/3995789595 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf180ab9c0 con 0x7fdf281446c0 2024-04-17T10:33:27.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 -- 172.21.15.67:0/3995789595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 msgr2=0x7fdf28144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3995789595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf28144ea0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fdf280b5320 tx=0x7fdf1801c4f0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 -- 172.21.15.67:0/3995789595 shutdown_connections 2024-04-17T10:33:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3995789595 >> v2:172.21.15.86:3300/0 conn(0x7fdf28071840 0x7fdf280b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3995789595 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf28144ea0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 -- 172.21.15.67:0/3995789595 >> 172.21.15.67:0/3995789595 conn(0x7fdf280b34d0 msgr2=0x7fdf280b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 -- 172.21.15.67:0/3995789595 shutdown_connections 2024-04-17T10:33:27.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.713+0000 7fdf2d600640 1 -- 172.21.15.67:0/3995789595 wait complete. 2024-04-17T10:33:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 Processor -- start 2024-04-17T10:33:27.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 -- start start 2024-04-17T10:33:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf28071840 0x7fdf28159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf280c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf28144aa0 0x7fdf281fa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280b6500 con 0x7fdf281446c0 2024-04-17T10:33:27.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280b6380 con 0x7fdf28144aa0 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf280b6680 con 0x7fdf28071840 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf280c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf280c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56270/0 (socket says 172.21.15.67:56270) 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 -- 172.21.15.67:0/3860065587 learned_addr learned my addr 172.21.15.67:0/3860065587 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf26a00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf28071840 0x7fdf28159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf27400640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf28144aa0 0x7fdf281fa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 -- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf28071840 msgr2=0x7fdf28159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf28071840 0x7fdf28159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 -- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf28144aa0 msgr2=0x7fdf281fa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf28144aa0 0x7fdf281fa3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 -- 172.21.15.67:0/3860065587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf281598d0 con 0x7fdf281446c0 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf26a00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf28071840 0x7fdf28159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf27400640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf28144aa0 0x7fdf281fa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:27.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1fe00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf280c6ec0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fdf1000c3a0 tx=0x7fdf1000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1ea00640 1 -- 172.21.15.67:0/3860065587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100052d0 con 0x7fdf281446c0 2024-04-17T10:33:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1ea00640 1 -- 172.21.15.67:0/3860065587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf10005470 con 0x7fdf281446c0 2024-04-17T10:33:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf1ea00640 1 -- 172.21.15.67:0/3860065587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10007260 con 0x7fdf281446c0 2024-04-17T10:33:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2806c7b0 con 0x7fdf281446c0 2024-04-17T10:33:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.717+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2806cd70 con 0x7fdf281446c0 2024-04-17T10:33:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.721+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf28070bb0 con 0x7fdf281446c0 2024-04-17T10:33:27.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.721+0000 7fdf1ea00640 1 -- 172.21.15.67:0/3860065587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdf10007400 con 0x7fdf281446c0 2024-04-17T10:33:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.721+0000 7fdf1ea00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf04066f60 0x7fdf04069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.721+0000 7fdf1ea00640 1 -- 172.21.15.67:0/3860065587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fdf10084380 con 0x7fdf281446c0 2024-04-17T10:33:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.721+0000 7fdf26a00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf04066f60 0x7fdf04069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:27.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.721+0000 7fdf26a00640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf04066f60 0x7fdf04069420 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdf180af000 tx=0x7fdf180a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.725+0000 7fdf1ea00640 1 -- 172.21.15.67:0/3860065587 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdf100554b0 con 0x7fdf281446c0 2024-04-17T10:33:27.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:33:27.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:33:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdf280b41a0 con 0x7fdf281446c0 2024-04-17T10:33:27.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf1ea00640 1 -- 172.21.15.67:0/3860065587 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v114) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdf10059e20 con 0x7fdf281446c0 2024-04-17T10:33:27.880 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:27.881 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:33:27.881 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:33:27.881 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:33:27.881 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:33:27.881 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:27.881 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf04066f60 msgr2=0x7fdf04069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf04066f60 0x7fdf04069420 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdf180af000 tx=0x7fdf180a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 msgr2=0x7fdf280c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf280c6ec0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fdf1000c3a0 tx=0x7fdf1000c860 comp rx=0 tx=0).stop 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 shutdown_connections 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdf04066f60 0x7fdf04069420 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf28144aa0 0x7fdf281fa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf281446c0 0x7fdf280c6ec0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 --2- 172.21.15.67:0/3860065587 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf28071840 0x7fdf28159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 >> 172.21.15.67:0/3860065587 conn(0x7fdf280b34d0 msgr2=0x7fdf2815de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 shutdown_connections 2024-04-17T10:33:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:27.889+0000 7fdf2d600640 1 -- 172.21.15.67:0/3860065587 wait complete. 2024-04-17T10:33:27.902 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:33:27.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:33:28.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.097+0000 7fc3efc00640 1 Processor -- start 2024-04-17T10:33:28.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.097+0000 7fc3efc00640 1 -- start start 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3efc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc3e8144100 0x7fc3e80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3efc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3e8071360 0x7fc3e806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3efc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e8071bb0 con 0x7fc3e806ea10 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3efc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80b3140 con 0x7fc3e8071360 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3efc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80b3420 con 0x7fc3e8144100 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3e8071360 0x7fc3e806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3e8071360 0x7fc3e806e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56278/0 (socket says 172.21.15.67:56278) 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3e7e00640 1 -- 172.21.15.67:0/254127692 learned_addr learned my addr 172.21.15.67:0/254127692 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3ed200640 1 --2- 172.21.15.67:0/254127692 >> v2:172.21.15.86:3300/0 conn(0x7fc3e8144100 0x7fc3e80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3e7e00640 1 -- 172.21.15.67:0/254127692 >> v2:172.21.15.86:3300/0 conn(0x7fc3e8144100 msgr2=0x7fc3e80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:28.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3e7e00640 1 --2- 172.21.15.67:0/254127692 >> v2:172.21.15.86:3300/0 conn(0x7fc3e8144100 0x7fc3e80cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3e7e00640 1 -- 172.21.15.67:0/254127692 >> v1:172.21.15.86:6789/0 conn(0x7fc3e806ea10 legacy=0x7fc3e815e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.101+0000 7fc3e7e00640 1 -- 172.21.15.67:0/254127692 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8071e90 con 0x7fc3e8071360 2024-04-17T10:33:28.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3e7e00640 1 --2- 172.21.15.67:0/254127692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e806e4d0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fc3e806f870 tx=0x7fc3d401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac4b139e38e5310 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:28.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3e7400640 1 -- 172.21.15.67:0/254127692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40ab430 con 0x7fc3e8071360 2024-04-17T10:33:28.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3e7400640 1 -- 172.21.15.67:0/254127692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d40b0070 con 0x7fc3e8071360 2024-04-17T10:33:28.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3e7400640 1 -- 172.21.15.67:0/254127692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40ab7f0 con 0x7fc3e8071360 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- 172.21.15.67:0/254127692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 msgr2=0x7fc3e806e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 --2- 172.21.15.67:0/254127692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e806e4d0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fc3e806f870 tx=0x7fc3d401c520 comp rx=0 tx=0).stop 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- 172.21.15.67:0/254127692 shutdown_connections 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 --2- 172.21.15.67:0/254127692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e806e4d0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 --2- 172.21.15.67:0/254127692 >> v2:172.21.15.86:3300/0 conn(0x7fc3e8144100 0x7fc3e80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- 172.21.15.67:0/254127692 >> 172.21.15.67:0/254127692 conn(0x7fc3e806ac90 msgr2=0x7fc3e806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- 172.21.15.67:0/254127692 shutdown_connections 2024-04-17T10:33:28.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- 172.21.15.67:0/254127692 wait complete. 2024-04-17T10:33:28.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 Processor -- start 2024-04-17T10:33:28.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- start start 2024-04-17T10:33:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3e806ea10 0x7fc3e815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e8142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3e8144100 0x7fc3e81431e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8165920 con 0x7fc3e8071360 2024-04-17T10:33:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e81657a0 con 0x7fc3e8144100 2024-04-17T10:33:28.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.105+0000 7fc3efc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8165aa0 con 0x7fc3e806ea10 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e8142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e8142ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56294/0 (socket says 172.21.15.67:56294) 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 -- 172.21.15.67:0/3848059347 learned_addr learned my addr 172.21.15.67:0/3848059347 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3ed200640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3e806ea10 0x7fc3e815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3edc00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3e8144100 0x7fc3e81431e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 -- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3e806ea10 msgr2=0x7fc3e815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3e806ea10 0x7fc3e815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 -- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3e8144100 msgr2=0x7fc3e81431e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3e8144100 0x7fc3e81431e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 -- 172.21.15.67:0/3848059347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e81f58a0 con 0x7fc3e8071360 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3ed200640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3e806ea10 0x7fc3e815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:28.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e7e00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e8142ca0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4019130 tx=0x7fc3d401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e4c00640 1 -- 172.21.15.67:0/3848059347 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40be5f0 con 0x7fc3e8071360 2024-04-17T10:33:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e4c00640 1 -- 172.21.15.67:0/3848059347 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d40be790 con 0x7fc3e8071360 2024-04-17T10:33:28.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e4c00640 1 -- 172.21.15.67:0/3848059347 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d40beb40 con 0x7fc3e8071360 2024-04-17T10:33:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e81f5b80 con 0x7fc3e8071360 2024-04-17T10:33:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e81f6140 con 0x7fc3e8071360 2024-04-17T10:33:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e8072da0 con 0x7fc3e8071360 2024-04-17T10:33:28.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.109+0000 7fc3e4c00640 1 -- 172.21.15.67:0/3848059347 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc3d40b43f0 con 0x7fc3e8071360 2024-04-17T10:33:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.113+0000 7fc3e4c00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc3c8066f60 0x7fc3c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.113+0000 7fc3e4c00640 1 -- 172.21.15.67:0/3848059347 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc3d40a8030 con 0x7fc3e8071360 2024-04-17T10:33:28.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.113+0000 7fc3ed200640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc3c8066f60 0x7fc3c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.113+0000 7fc3ed200640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc3c8066f60 0x7fc3c8069420 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00b9f0 tx=0x7fc3dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:28.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.117+0000 7fc3e4c00640 1 -- 172.21.15.67:0/3848059347 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc3d4016110 con 0x7fc3e8071360 2024-04-17T10:33:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.281+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc3e80b2cf0 con 0x7fc3e8071360 2024-04-17T10:33:28.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.461+0000 7fc3e4c00640 1 -- 172.21.15.67:0/3848059347 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v115) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc3d40f0360 con 0x7fc3e8071360 2024-04-17T10:33:28.454 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:33:28.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc3c8066f60 msgr2=0x7fc3c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc3c8066f60 0x7fc3c8069420 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00b9f0 tx=0x7fc3dc008040 comp rx=0 tx=0).stop 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 msgr2=0x7fc3e8142ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e8142ca0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4019130 tx=0x7fc3d401ad60 comp rx=0 tx=0).stop 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 shutdown_connections 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc3c8066f60 0x7fc3c8069420 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc3e8144100 0x7fc3e81431e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e8071360 0x7fc3e8142ca0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 --2- 172.21.15.67:0/3848059347 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3e806ea10 0x7fc3e815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 >> 172.21.15.67:0/3848059347 conn(0x7fc3e806ac90 msgr2=0x7fc3e80b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 shutdown_connections 2024-04-17T10:33:28.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.465+0000 7fc3efc00640 1 -- 172.21.15.67:0/3848059347 wait complete. 2024-04-17T10:33:28.481 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:33:28.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.637+0000 7f7d43200640 1 Processor -- start 2024-04-17T10:33:28.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.637+0000 7f7d43200640 1 -- start start 2024-04-17T10:33:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d43200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d4409a160 0x7f7d440b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d43200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d440cf9d0 con 0x7f7d44160f40 2024-04-17T10:33:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d43200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4406cce0 con 0x7f7d4406c7b0 2024-04-17T10:33:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d43200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d440cfc40 con 0x7f7d4409a160 2024-04-17T10:33:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d3be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d4409a160 0x7f7d440b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d41e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7d44160f40 0x7f7d44099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40778/0 (socket says 172.21.15.67:40778) 2024-04-17T10:33:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d41e00640 1 -- 172.21.15.67:0/2538029012 learned_addr learned my addr 172.21.15.67:0/2538029012 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d3be00640 1 -- 172.21.15.67:0/2538029012 >> v1:172.21.15.86:6789/0 conn(0x7f7d4406c7b0 legacy=0x7f7d440d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d3be00640 1 -- 172.21.15.67:0/2538029012 >> v1:172.21.15.67:6789/0 conn(0x7f7d44160f40 legacy=0x7f7d44099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d41400640 1 -- 172.21.15.67:0/2538029012 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53163068 0 0) 0x7f7d440cf9d0 con 0x7f7d44160f40 2024-04-17T10:33:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.641+0000 7f7d3be00640 1 -- 172.21.15.67:0/2538029012 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440cff20 con 0x7f7d4409a160 2024-04-17T10:33:28.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d3be00640 1 --2- 172.21.15.67:0/2538029012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d440b2640 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097c0 tx=0x7f7d2c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd37d176426e624 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d41400640 1 -- 172.21.15.67:0/2538029012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0b0070 con 0x7f7d4409a160 2024-04-17T10:33:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d41400640 1 -- 172.21.15.67:0/2538029012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c0ab600 con 0x7f7d4409a160 2024-04-17T10:33:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d41400640 1 -- 172.21.15.67:0/2538029012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0ab9d0 con 0x7f7d4409a160 2024-04-17T10:33:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- 172.21.15.67:0/2538029012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 msgr2=0x7f7d440b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 --2- 172.21.15.67:0/2538029012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d440b2640 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097c0 tx=0x7f7d2c01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- 172.21.15.67:0/2538029012 shutdown_connections 2024-04-17T10:33:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 --2- 172.21.15.67:0/2538029012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d440b2640 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- 172.21.15.67:0/2538029012 >> 172.21.15.67:0/2538029012 conn(0x7f7d440b3190 msgr2=0x7f7d440b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- 172.21.15.67:0/2538029012 shutdown_connections 2024-04-17T10:33:28.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- 172.21.15.67:0/2538029012 wait complete. 2024-04-17T10:33:28.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 Processor -- start 2024-04-17T10:33:28.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- start start 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d4406c7b0 0x7f7d440c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d441520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d44160f40 0x7f7d441574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4416bc10 con 0x7f7d4409a160 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4416ba90 con 0x7f7d4406c7b0 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.645+0000 7f7d43200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4416bd90 con 0x7f7d44160f40 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d441520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d441520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56306/0 (socket says 172.21.15.67:56306) 2024-04-17T10:33:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 -- 172.21.15.67:0/2760242321 learned_addr learned my addr 172.21.15.67:0/2760242321 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d41e00640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d4406c7b0 0x7f7d440c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d42800640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d44160f40 0x7f7d441574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 -- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d44160f40 msgr2=0x7f7d441574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d44160f40 0x7f7d441574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 -- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d4406c7b0 msgr2=0x7f7d440c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d4406c7b0 0x7f7d440c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 -- 172.21.15.67:0/2760242321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440cb3b0 con 0x7f7d4409a160 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3be00640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d441520f0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097c0 tx=0x7f7d2c01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3a000640 1 -- 172.21.15.67:0/2760242321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0b0040 con 0x7f7d4409a160 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3a000640 1 -- 172.21.15.67:0/2760242321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c004020 con 0x7f7d4409a160 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d440cb690 con 0x7f7d4409a160 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3a000640 1 -- 172.21.15.67:0/2760242321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c004430 con 0x7f7d4409a160 2024-04-17T10:33:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d440ca060 con 0x7f7d4409a160 2024-04-17T10:33:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d44074dc0 con 0x7f7d4409a160 2024-04-17T10:33:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.649+0000 7f7d3a000640 1 -- 172.21.15.67:0/2760242321 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7d2c0bf060 con 0x7f7d4409a160 2024-04-17T10:33:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.653+0000 7f7d3a000640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d20066f90 0x7f7d20069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:28.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.653+0000 7f7d3a000640 1 -- 172.21.15.67:0/2760242321 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f7d2c11ef00 con 0x7f7d4409a160 2024-04-17T10:33:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.653+0000 7f7d41e00640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d20066f90 0x7f7d20069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.653+0000 7f7d41e00640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d20066f90 0x7f7d20069450 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7d34000f70 tx=0x7f7d34007240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:28.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.653+0000 7f7d3a000640 1 -- 172.21.15.67:0/2760242321 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7d2c016090 con 0x7f7d4409a160 2024-04-17T10:33:28.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:28.821+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7d4409ad70 con 0x7f7d4409a160 2024-04-17T10:33:29.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.465+0000 7f7d3a000640 1 -- 172.21.15.67:0/2760242321 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v116) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7d2c0efa10 con 0x7f7d4409a160 2024-04-17T10:33:29.458 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:33:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d20066f90 msgr2=0x7f7d20069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d20066f90 0x7f7d20069450 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f7d34000f70 tx=0x7f7d34007240 comp rx=0 tx=0).stop 2024-04-17T10:33:29.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 msgr2=0x7f7d441520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d441520f0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0097c0 tx=0x7f7d2c01ace0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 shutdown_connections 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d20066f90 0x7f7d20069450 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d44160f40 0x7f7d441574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d4409a160 0x7f7d441520f0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 --2- 172.21.15.67:0/2760242321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d4406c7b0 0x7f7d440c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 >> 172.21.15.67:0/2760242321 conn(0x7f7d440b3190 msgr2=0x7f7d44070280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 shutdown_connections 2024-04-17T10:33:29.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.469+0000 7f7d43200640 1 -- 172.21.15.67:0/2760242321 wait complete. 2024-04-17T10:33:29.495 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:33:29.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b90000640 1 Processor -- start 2024-04-17T10:33:29.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b90000640 1 -- start start 2024-04-17T10:33:29.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b90000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3b880ce8a0 0x7f3b8806bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:29.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b90000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8806c430 con 0x7f3b8815db80 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b90000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8806c710 con 0x7f3b8809a660 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b90000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8806cfb0 con 0x7f3b880ce8a0 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b8d600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3b8815db80 0x7f3b880ce190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40788/0 (socket says 172.21.15.67:40788) 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b8d600640 1 -- 172.21.15.67:0/3261590215 learned_addr learned my addr 172.21.15.67:0/3261590215 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b8cc00640 1 --2- 172.21.15.67:0/3261590215 >> v2:172.21.15.86:3300/0 conn(0x7f3b880ce8a0 0x7f3b8806bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b87e00640 1 -- 172.21.15.67:0/3261590215 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484075366 0 0) 0x7f3b8806c430 con 0x7f3b8815db80 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b87e00640 1 -- 172.21.15.67:0/3261590215 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b70003530 con 0x7f3b8815db80 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b8cc00640 1 -- 172.21.15.67:0/3261590215 >> v1:172.21.15.86:6789/0 conn(0x7f3b8809a660 legacy=0x7f3b880b1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b8cc00640 1 -- 172.21.15.67:0/3261590215 >> v1:172.21.15.67:6789/0 conn(0x7f3b8815db80 legacy=0x7f3b880ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:29.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.649+0000 7f3b8cc00640 1 -- 172.21.15.67:0/3261590215 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8806c9f0 con 0x7f3b880ce8a0 2024-04-17T10:33:29.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b8cc00640 1 --2- 172.21.15.67:0/3261590215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b880ce8a0 0x7f3b8806bda0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3b740097c0 tx=0x7f3b7401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bd016eb45e1e33a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:29.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b87e00640 1 -- 172.21.15.67:0/3261590215 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b740b1070 con 0x7f3b880ce8a0 2024-04-17T10:33:29.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b87e00640 1 -- 172.21.15.67:0/3261590215 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b740ac5f0 con 0x7f3b880ce8a0 2024-04-17T10:33:29.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b87e00640 1 -- 172.21.15.67:0/3261590215 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b740ac9c0 con 0x7f3b880ce8a0 2024-04-17T10:33:29.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b90000640 1 -- 172.21.15.67:0/3261590215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b880ce8a0 msgr2=0x7f3b8806bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:29.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b90000640 1 --2- 172.21.15.67:0/3261590215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b880ce8a0 0x7f3b8806bda0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3b740097c0 tx=0x7f3b7401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b90000640 1 -- 172.21.15.67:0/3261590215 shutdown_connections 2024-04-17T10:33:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b90000640 1 --2- 172.21.15.67:0/3261590215 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b880ce8a0 0x7f3b8806bda0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b90000640 1 -- 172.21.15.67:0/3261590215 >> 172.21.15.67:0/3261590215 conn(0x7f3b880b3410 msgr2=0x7f3b880b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b90000640 1 -- 172.21.15.67:0/3261590215 shutdown_connections 2024-04-17T10:33:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.653+0000 7f3b90000640 1 -- 172.21.15.67:0/3261590215 wait complete. 2024-04-17T10:33:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 Processor -- start 2024-04-17T10:33:29.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 -- start start 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8809a660 0x7f3b880b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b880ce8a0 0x7f3b88154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8815db80 0x7f3b8815b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8816bac0 con 0x7f3b8815db80 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8809aa70 con 0x7f3b8809a660 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8816bc40 con 0x7f3b880ce8a0 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8815db80 0x7f3b8815b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:29.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8815db80 0x7f3b8815b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56318/0 (socket says 172.21.15.67:56318) 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 -- 172.21.15.67:0/2637969176 learned_addr learned my addr 172.21.15.67:0/2637969176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8d600640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8809a660 0x7f3b880b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8cc00640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b880ce8a0 0x7f3b88154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 -- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b880ce8a0 msgr2=0x7f3b88154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b880ce8a0 0x7f3b88154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 -- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8809a660 msgr2=0x7f3b880b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8809a660 0x7f3b880b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 -- 172.21.15.67:0/2637969176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b880b51c0 con 0x7f3b8815db80 2024-04-17T10:33:29.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b8e000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8815db80 0x7f3b8815b6d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c7a0 tx=0x7f3b8000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:29.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b85600640 1 -- 172.21.15.67:0/2637969176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b80003b80 con 0x7f3b8815db80 2024-04-17T10:33:29.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b85600640 1 -- 172.21.15.67:0/2637969176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80003d20 con 0x7f3b8815db80 2024-04-17T10:33:29.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b85600640 1 -- 172.21.15.67:0/2637969176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b8000aa90 con 0x7f3b8815db80 2024-04-17T10:33:29.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b881553f0 con 0x7f3b8815db80 2024-04-17T10:33:29.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.657+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b880b5740 con 0x7f3b8815db80 2024-04-17T10:33:29.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.661+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8804ef30 con 0x7f3b8815db80 2024-04-17T10:33:29.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.661+0000 7f3b85600640 1 -- 172.21.15.67:0/2637969176 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3b8000ac30 con 0x7f3b8815db80 2024-04-17T10:33:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.661+0000 7f3b85600640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b68066f90 0x7f3b68069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.661+0000 7f3b85600640 1 -- 172.21.15.67:0/2637969176 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f3b80085a50 con 0x7f3b8815db80 2024-04-17T10:33:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.661+0000 7f3b8d600640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b68066f90 0x7f3b68069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:29.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.661+0000 7f3b8d600640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b68066f90 0x7f3b68069450 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0011f0 tx=0x7f3b7c007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:29.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.665+0000 7f3b85600640 1 -- 172.21.15.67:0/2637969176 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3b800562c0 con 0x7f3b8815db80 2024-04-17T10:33:29.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:29.861+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3b8809b270 con 0x7f3b8815db80 2024-04-17T10:33:30.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.509+0000 7f3b85600640 1 -- 172.21.15.67:0/2637969176 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v78) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3b8005a360 con 0x7f3b8815db80 2024-04-17T10:33:30.511 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 17 and data pool 18 2024-04-17T10:33:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b68066f90 msgr2=0x7f3b68069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b68066f90 0x7f3b68069450 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3b7c0011f0 tx=0x7f3b7c007d40 comp rx=0 tx=0).stop 2024-04-17T10:33:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8815db80 msgr2=0x7f3b8815b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8815db80 0x7f3b8815b6d0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c7a0 tx=0x7f3b8000cc60 comp rx=0 tx=0).stop 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 shutdown_connections 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b68066f90 0x7f3b68069450 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b8815db80 0x7f3b8815b6d0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b880ce8a0 0x7f3b88154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 --2- 172.21.15.67:0/2637969176 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b8809a660 0x7f3b880b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 >> 172.21.15.67:0/2637969176 conn(0x7f3b880b3410 msgr2=0x7f3b881590d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 shutdown_connections 2024-04-17T10:33:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.525+0000 7f3b90000640 1 -- 172.21.15.67:0/2637969176 wait complete. 2024-04-17T10:33:30.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:30.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1069600640 1 Processor -- start 2024-04-17T10:33:30.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1069600640 1 -- start start 2024-04-17T10:33:30.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1069600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1064145040 0x7f106406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:30.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1069600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1064071a20 0x7f10640cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:30.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1069600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106406e2f0 con 0x7f106406ef10 2024-04-17T10:33:30.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1069600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10640a86c0 con 0x7f1064145040 2024-04-17T10:33:30.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1069600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106415bf10 con 0x7f1064071a20 2024-04-17T10:33:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1062a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1064145040 0x7f106406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1062a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1064145040 0x7f106406e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42360/0 (socket says 172.21.15.67:42360) 2024-04-17T10:33:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1062a00640 1 -- 172.21.15.67:0/3845850956 learned_addr learned my addr 172.21.15.67:0/3845850956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1063400640 1 --2- 172.21.15.67:0/3845850956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1064071a20 0x7f10640cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1062a00640 1 -- 172.21.15.67:0/3845850956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1064071a20 msgr2=0x7f10640cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1062a00640 1 --2- 172.21.15.67:0/3845850956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1064071a20 0x7f10640cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1062a00640 1 -- 172.21.15.67:0/3845850956 >> v1:172.21.15.86:6789/0 conn(0x7f106406ef10 legacy=0x7f1064071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.697+0000 7f1062a00640 1 -- 172.21.15.67:0/3845850956 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106406d6e0 con 0x7f1064145040 2024-04-17T10:33:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1062a00640 1 --2- 172.21.15.67:0/3845850956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064145040 0x7f106406e9d0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f1054009870 tx=0x7f105401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90bb02f38893d4e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1062000640 1 -- 172.21.15.67:0/3845850956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10540b5070 con 0x7f1064145040 2024-04-17T10:33:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1062000640 1 -- 172.21.15.67:0/3845850956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10540a8a50 con 0x7f1064145040 2024-04-17T10:33:30.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1062000640 1 -- 172.21.15.67:0/3845850956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10540a8e20 con 0x7f1064145040 2024-04-17T10:33:30.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 -- 172.21.15.67:0/3845850956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064145040 msgr2=0x7f106406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 --2- 172.21.15.67:0/3845850956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064145040 0x7f106406e9d0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f1054009870 tx=0x7f105401ed50 comp rx=0 tx=0).stop 2024-04-17T10:33:30.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 -- 172.21.15.67:0/3845850956 shutdown_connections 2024-04-17T10:33:30.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 --2- 172.21.15.67:0/3845850956 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1064071a20 0x7f10640cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 --2- 172.21.15.67:0/3845850956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064145040 0x7f106406e9d0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 -- 172.21.15.67:0/3845850956 >> 172.21.15.67:0/3845850956 conn(0x7f106406ac90 msgr2=0x7f106406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:30.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 -- 172.21.15.67:0/3845850956 shutdown_connections 2024-04-17T10:33:30.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.701+0000 7f1069600640 1 -- 172.21.15.67:0/3845850956 wait complete. 2024-04-17T10:33:30.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 Processor -- start 2024-04-17T10:33:30.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 -- start start 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f106406ef10 0x7f106415aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064071a20 0x7f1064140220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1064145040 0x7f1064140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106406df80 con 0x7f1064071a20 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106406dca0 con 0x7f1064145040 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106406d370 con 0x7f106406ef10 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064071a20 0x7f1064140220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1062a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f106406ef10 0x7f106415aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064071a20 0x7f1064140220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42364/0 (socket says 172.21.15.67:42364) 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 -- 172.21.15.67:0/1503314290 learned_addr learned my addr 172.21.15.67:0/1503314290 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1063400640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1064145040 0x7f1064140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 -- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f106406ef10 msgr2=0x7f106415aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f106406ef10 0x7f106415aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 -- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1064145040 msgr2=0x7f1064140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1064145040 0x7f1064140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 -- 172.21.15.67:0/1503314290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10641f7300 con 0x7f1064071a20 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1062a00640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f106406ef10 0x7f106415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105a000640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064071a20 0x7f1064140220 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c420 tx=0x7f104c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105b400640 1 -- 172.21.15.67:0/1503314290 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c0052d0 con 0x7f1064071a20 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105b400640 1 -- 172.21.15.67:0/1503314290 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c005470 con 0x7f1064071a20 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f105b400640 1 -- 172.21.15.67:0/1503314290 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c007260 con 0x7f1064071a20 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10641f75e0 con 0x7f1064071a20 2024-04-17T10:33:30.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10641f7ba0 con 0x7f1064071a20 2024-04-17T10:33:30.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.705+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1064072d50 con 0x7f1064071a20 2024-04-17T10:33:30.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.709+0000 7f105b400640 1 -- 172.21.15.67:0/1503314290 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f104c007400 con 0x7f1064071a20 2024-04-17T10:33:30.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.709+0000 7f105b400640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1040066f90 0x7f1040069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:30.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.709+0000 7f105b400640 1 -- 172.21.15.67:0/1503314290 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f104c084660 con 0x7f1064071a20 2024-04-17T10:33:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.709+0000 7f1062a00640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1040066f90 0x7f1040069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:30.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.709+0000 7f1062a00640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1040066f90 0x7f1040069450 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f10540b2000 tx=0x7f1054009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:30.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.713+0000 7f105b400640 1 -- 172.21.15.67:0/1503314290 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f104c054df0 con 0x7f1064071a20 2024-04-17T10:33:30.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.877+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1064145500 con 0x7f1064071a20 2024-04-17T10:33:30.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.877+0000 7f105b400640 1 -- 172.21.15.67:0/1503314290 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16917 (secure 0 0 0) 0x7f104c058e90 con 0x7f1064071a20 2024-04-17T10:33:30.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:30.875 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":117,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:29.865231+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:33:28.333663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-17T10:33:28.871385+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1040066f90 msgr2=0x7f1040069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1040066f90 0x7f1040069450 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f10540b2000 tx=0x7f1054009040 comp rx=0 tx=0).stop 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064071a20 msgr2=0x7f1064140220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064071a20 0x7f1064140220 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f104c00c420 tx=0x7f104c00c8e0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 shutdown_connections 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1040066f90 0x7f1040069450 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1064145040 0x7f1064140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1064071a20 0x7f1064140220 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 --2- 172.21.15.67:0/1503314290 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f106406ef10 0x7f106415aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 >> 172.21.15.67:0/1503314290 conn(0x7f106406ac90 msgr2=0x7f10640b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 shutdown_connections 2024-04-17T10:33:30.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:30.881+0000 7f1069600640 1 -- 172.21.15.67:0/1503314290 wait complete. 2024-04-17T10:33:30.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:31.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.041+0000 7f0e20e00640 1 Processor -- start 2024-04-17T10:33:31.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.041+0000 7f0e20e00640 1 -- start start 2024-04-17T10:33:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e20e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e20e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0e1c071c60 0x7f0e1c0720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e20e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e1c0ad8f0 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e20e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1c15dfa0 con 0x7f0e1c071c60 2024-04-17T10:33:31.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e20e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c06faa0 con 0x7f0e1c06c410 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e11400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e1a800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0e1c071c60 0x7f0e1c0720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e11400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c06f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:34528/0 (socket says 172.21.15.67:34528) 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e11400640 1 -- 172.21.15.67:0/1525630322 learned_addr learned my addr 172.21.15.67:0/1525630322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e1a800640 1 -- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 msgr2=0x7f0e1c06f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e1a800640 1 --2- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c06f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e1a800640 1 -- 172.21.15.67:0/1525630322 >> v1:172.21.15.86:6789/0 conn(0x7f0e1c15dbc0 legacy=0x7f0e1c06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e1a800640 1 -- 172.21.15.67:0/1525630322 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c0adbd0 con 0x7f0e1c071c60 2024-04-17T10:33:31.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.045+0000 7f0e11400640 1 --2- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c06f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:31.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e1a800640 1 --2- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 0x7f0e1c0720c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0e140097c0 tx=0x7f0e1401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=504c76e8d6f4f351 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:31.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e19400640 1 -- 172.21.15.67:0/1525630322 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e140b0070 con 0x7f0e1c071c60 2024-04-17T10:33:31.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e19400640 1 -- 172.21.15.67:0/1525630322 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e140ab600 con 0x7f0e1c071c60 2024-04-17T10:33:31.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e19400640 1 -- 172.21.15.67:0/1525630322 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e140ab9d0 con 0x7f0e1c071c60 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 msgr2=0x7f0e1c0720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 --2- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 0x7f0e1c0720c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0e140097c0 tx=0x7f0e1401c410 comp rx=0 tx=0).stop 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- 172.21.15.67:0/1525630322 shutdown_connections 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 --2- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 0x7f0e1c0720c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 --2- 172.21.15.67:0/1525630322 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c06f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- 172.21.15.67:0/1525630322 >> 172.21.15.67:0/1525630322 conn(0x7f0e1c0af670 msgr2=0x7f0e1c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- 172.21.15.67:0/1525630322 shutdown_connections 2024-04-17T10:33:31.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- 172.21.15.67:0/1525630322 wait complete. 2024-04-17T10:33:31.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 Processor -- start 2024-04-17T10:33:31.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- start start 2024-04-17T10:33:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c157430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 0x7f0e1c14ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e1c15dbc0 0x7f0e1c0cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c16a100 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c06af70 con 0x7f0e1c071c60 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.049+0000 7f0e20e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1c16a280 con 0x7f0e1c06c410 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e1c15dbc0 0x7f0e1c0cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e1c15dbc0 0x7f0e1c0cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42376/0 (socket says 172.21.15.67:42376) 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 -- 172.21.15.67:0/2510994743 learned_addr learned my addr 172.21.15.67:0/2510994743 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e19e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c157430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e11400640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 0x7f0e1c14ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 -- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 msgr2=0x7f0e1c157430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c157430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 -- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 msgr2=0x7f0e1c14ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 0x7f0e1c14ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 -- 172.21.15.67:0/2510994743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c14cee0 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e1a800640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e1c15dbc0 0x7f0e1c0cdd60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0e1401a040 tx=0x7f0e140191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e12a00640 1 -- 172.21.15.67:0/2510994743 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e140b8020 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e12a00640 1 -- 172.21.15.67:0/2510994743 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e140b4610 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1c14d1c0 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e12a00640 1 -- 172.21.15.67:0/2510994743 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e140b0040 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1c14d780 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1c09e260 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e12a00640 1 -- 172.21.15.67:0/2510994743 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0e140bb020 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.053+0000 7f0e12a00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0df8066f60 0x7f0df8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.057+0000 7f0e12a00640 1 -- 172.21.15.67:0/2510994743 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0e1411eed0 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.057+0000 7f0e19e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0df8066f60 0x7f0df8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.057+0000 7f0e19e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0df8066f60 0x7f0df8069420 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c001770 tx=0x7f0e0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:31.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.061+0000 7f0e12a00640 1 -- 172.21.15.67:0/2510994743 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0e140c2030 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.221+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e1c0d08e0 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e12a00640 1 -- 172.21.15.67:0/2510994743 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+16917 (secure 0 0 0) 0x7f0e14016090 con 0x7f0e1c15dbc0 2024-04-17T10:33:31.216 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:31.216 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":117,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:29.865231+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:33:28.333663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-17T10:33:28.871385+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0df8066f60 msgr2=0x7f0df8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0df8066f60 0x7f0df8069420 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c001770 tx=0x7f0e0c009040 comp rx=0 tx=0).stop 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e1c15dbc0 msgr2=0x7f0e1c0cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e1c15dbc0 0x7f0e1c0cdd60 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f0e1401a040 tx=0x7f0e140191b0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 shutdown_connections 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0df8066f60 0x7f0df8069420 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e1c15dbc0 0x7f0e1c0cdd60 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0e1c071c60 0x7f0e1c14ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 --2- 172.21.15.67:0/2510994743 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0e1c06c410 0x7f0e1c157430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 >> 172.21.15.67:0/2510994743 conn(0x7f0e1c0af670 msgr2=0x7f0e1c0b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:31.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 shutdown_connections 2024-04-17T10:33:31.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.225+0000 7f0e20e00640 1 -- 172.21.15.67:0/2510994743 wait complete. 2024-04-17T10:33:31.237 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:33:31.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ef4a00640 1 Processor -- start 2024-04-17T10:33:31.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ef4a00640 1 -- start start 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ef4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef00722b0 0x7f3ef015a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ef4a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ef015e210 0x7f3ef015e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ef4a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ef015ac40 con 0x7f3ef0071ed0 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ef4a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ef00ccbf0 con 0x7f3ef015e210 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ef4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef015af20 con 0x7f3ef00722b0 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3eee800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ef015e210 0x7f3ef015e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3eee800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ef015e210 0x7f3ef015e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42378/0 (socket says 172.21.15.67:42378) 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3eee800640 1 -- 172.21.15.67:0/2694608878 learned_addr learned my addr 172.21.15.67:0/2694608878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ee5400640 1 --2- 172.21.15.67:0/2694608878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef00722b0 0x7f3ef015a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3eee800640 1 -- 172.21.15.67:0/2694608878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef00722b0 msgr2=0x7f3ef015a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3eee800640 1 --2- 172.21.15.67:0/2694608878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef00722b0 0x7f3ef015a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3eee800640 1 -- 172.21.15.67:0/2694608878 >> v1:172.21.15.86:6789/0 conn(0x7f3ef0071ed0 legacy=0x7f3ef00aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3eee800640 1 -- 172.21.15.67:0/2694608878 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef015b200 con 0x7f3ef015e210 2024-04-17T10:33:31.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.389+0000 7f3ee5400640 1 --2- 172.21.15.67:0/2694608878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef00722b0 0x7f3ef015a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3eee800640 1 --2- 172.21.15.67:0/2694608878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef015e210 0x7f3ef015e6a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009870 tx=0x7f3ee801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8905c0f98da4bab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3eed400640 1 -- 172.21.15.67:0/2694608878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80b0070 con 0x7f3ef015e210 2024-04-17T10:33:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3eed400640 1 -- 172.21.15.67:0/2694608878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee80ab600 con 0x7f3ef015e210 2024-04-17T10:33:31.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3eed400640 1 -- 172.21.15.67:0/2694608878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee80ab9d0 con 0x7f3ef015e210 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 -- 172.21.15.67:0/2694608878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef015e210 msgr2=0x7f3ef015e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/2694608878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef015e210 0x7f3ef015e6a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009870 tx=0x7f3ee801c560 comp rx=0 tx=0).stop 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 -- 172.21.15.67:0/2694608878 shutdown_connections 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/2694608878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef015e210 0x7f3ef015e6a0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/2694608878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef00722b0 0x7f3ef015a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 -- 172.21.15.67:0/2694608878 >> 172.21.15.67:0/2694608878 conn(0x7f3ef00b1740 msgr2=0x7f3ef00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 -- 172.21.15.67:0/2694608878 shutdown_connections 2024-04-17T10:33:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.393+0000 7f3ef4a00640 1 -- 172.21.15.67:0/2694608878 wait complete. 2024-04-17T10:33:31.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 Processor -- start 2024-04-17T10:33:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 -- start start 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef0071ed0 0x7f3ef014a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef00722b0 0x7f3ef015c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef015e210 0x7f3ef014eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef0151470 con 0x7f3ef00722b0 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef01512f0 con 0x7f3ef0071ed0 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef0165ef0 con 0x7f3ef015e210 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef00722b0 0x7f3ef015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef00722b0 0x7f3ef015c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42384/0 (socket says 172.21.15.67:42384) 2024-04-17T10:33:31.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 -- 172.21.15.67:0/4165208482 learned_addr learned my addr 172.21.15.67:0/4165208482 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3eee800640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef015e210 0x7f3ef014eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3eede00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef0071ed0 0x7f3ef014a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 -- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef015e210 msgr2=0x7f3ef014eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef015e210 0x7f3ef014eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 -- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef0071ed0 msgr2=0x7f3ef014a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef0071ed0 0x7f3ef014a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 -- 172.21.15.67:0/4165208482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef0148590 con 0x7f3ef00722b0 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3eee800640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef015e210 0x7f3ef014eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3eede00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef0071ed0 0x7f3ef014a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee5400640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef00722b0 0x7f3ef015c530 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800e810 tx=0x7f3ed800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee6a00640 1 -- 172.21.15.67:0/4165208482 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8017070 con 0x7f3ef00722b0 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee6a00640 1 -- 172.21.15.67:0/4165208482 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed800c8d0 con 0x7f3ef00722b0 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ee6a00640 1 -- 172.21.15.67:0/4165208482 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed800f040 con 0x7f3ef00722b0 2024-04-17T10:33:31.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ef015ca70 con 0x7f3ef00722b0 2024-04-17T10:33:31.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.397+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef00b07f0 con 0x7f3ef00722b0 2024-04-17T10:33:31.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.401+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef004ef30 con 0x7f3ef00722b0 2024-04-17T10:33:31.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.401+0000 7f3ee6a00640 1 -- 172.21.15.67:0/4165208482 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3ed801a070 con 0x7f3ef00722b0 2024-04-17T10:33:31.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.401+0000 7f3ee6a00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ecc066f60 0x7f3ecc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:31.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.401+0000 7f3ee6a00640 1 -- 172.21.15.67:0/4165208482 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ed8084d40 con 0x7f3ef00722b0 2024-04-17T10:33:31.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.401+0000 7f3eede00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ecc066f60 0x7f3ecc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:31.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.401+0000 7f3eede00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ecc066f60 0x7f3ecc069420 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0007fd0 tx=0x7f3ee0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:31.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.405+0000 7f3ee6a00640 1 -- 172.21.15.67:0/4165208482 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3ed8055c50 con 0x7f3ef00722b0 2024-04-17T10:33:31.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:31.601+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3ef00ce1b0 con 0x7f3ef00722b0 2024-04-17T10:33:32.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.521+0000 7f3ee6a00640 1 -- 172.21.15.67:0/4165208482 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v81) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3ed8055550 con 0x7f3ef00722b0 2024-04-17T10:33:32.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ecc066f60 msgr2=0x7f3ecc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:32.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ecc066f60 0x7f3ecc069420 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3ee0007fd0 tx=0x7f3ee0009040 comp rx=0 tx=0).stop 2024-04-17T10:33:32.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef00722b0 msgr2=0x7f3ef015c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:32.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef00722b0 0x7f3ef015c530 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800e810 tx=0x7f3ed800ecd0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 shutdown_connections 2024-04-17T10:33:32.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ecc066f60 0x7f3ecc069420 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ef015e210 0x7f3ef014eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ef00722b0 0x7f3ef015c530 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 --2- 172.21.15.67:0/4165208482 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ef0071ed0 0x7f3ef014a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 >> 172.21.15.67:0/4165208482 conn(0x7f3ef00b1740 msgr2=0x7f3ef00b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 shutdown_connections 2024-04-17T10:33:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.525+0000 7f3ef4a00640 1 -- 172.21.15.67:0/4165208482 wait complete. 2024-04-17T10:33:32.548 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:33:32.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:33:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f81e00640 1 Processor -- start 2024-04-17T10:33:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f81e00640 1 -- start start 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f81e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c071ed0 0x7f5f7c15a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f81e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f7c15e210 0x7f5f7c1508d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f81e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f7c15ac40 con 0x7f5f7c0722b0 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f81e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7c0ccbf0 con 0x7f5f7c15e210 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f81e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c15af20 con 0x7f5f7c071ed0 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f72000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5f7c0722b0 0x7f5f7c0aeda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59352/0 (socket says 172.21.15.67:59352) 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f72000640 1 -- 172.21.15.67:0/2806968478 learned_addr learned my addr 172.21.15.67:0/2806968478 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:32.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7aa00640 1 --2- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c071ed0 0x7f5f7c15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:32.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7b400640 1 --2- 172.21.15.67:0/2806968478 >> v2:172.21.15.86:3300/0 conn(0x7f5f7c15e210 0x7f5f7c1508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:32.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7a000640 1 -- 172.21.15.67:0/2806968478 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130914088 0 0) 0x7f5f7c15ac40 con 0x7f5f7c0722b0 2024-04-17T10:33:32.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7a000640 1 -- 172.21.15.67:0/2806968478 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f60003530 con 0x7f5f7c0722b0 2024-04-17T10:33:32.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7b400640 1 -- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c071ed0 msgr2=0x7f5f7c15a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:32.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7b400640 1 --2- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c071ed0 0x7f5f7c15a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7b400640 1 -- 172.21.15.67:0/2806968478 >> v1:172.21.15.67:6789/0 conn(0x7f5f7c0722b0 legacy=0x7f5f7c0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:32.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7b400640 1 -- 172.21.15.67:0/2806968478 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c15b200 con 0x7f5f7c15e210 2024-04-17T10:33:32.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.709+0000 7f5f7aa00640 1 --2- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c071ed0 0x7f5f7c15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:32.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f7b400640 1 --2- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c15e210 0x7f5f7c1508d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009d90 tx=0x7f5f7401c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73bf0bb13836f4c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:32.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f7a000640 1 -- 172.21.15.67:0/2806968478 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f740b1070 con 0x7f5f7c15e210 2024-04-17T10:33:32.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f7a000640 1 -- 172.21.15.67:0/2806968478 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f740ac600 con 0x7f5f7c15e210 2024-04-17T10:33:32.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f7a000640 1 -- 172.21.15.67:0/2806968478 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f740ac9d0 con 0x7f5f7c15e210 2024-04-17T10:33:32.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 -- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c15e210 msgr2=0x7f5f7c1508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:32.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 --2- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c15e210 0x7f5f7c1508d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f5f74009d90 tx=0x7f5f7401c970 comp rx=0 tx=0).stop 2024-04-17T10:33:32.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 -- 172.21.15.67:0/2806968478 shutdown_connections 2024-04-17T10:33:32.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 --2- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c15e210 0x7f5f7c1508d0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 --2- 172.21.15.67:0/2806968478 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c071ed0 0x7f5f7c15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 -- 172.21.15.67:0/2806968478 >> 172.21.15.67:0/2806968478 conn(0x7f5f7c0b1740 msgr2=0x7f5f7c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:32.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 -- 172.21.15.67:0/2806968478 shutdown_connections 2024-04-17T10:33:32.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.713+0000 7f5f81e00640 1 -- 172.21.15.67:0/2806968478 wait complete. 2024-04-17T10:33:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 Processor -- start 2024-04-17T10:33:32.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 -- start start 2024-04-17T10:33:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c071ed0 0x7f5f7c15d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:32.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c0722b0 0x7f5f7c13d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c15e210 0x7f5f7c163b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0ce3a0 con 0x7f5f7c071ed0 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0ce220 con 0x7f5f7c0722b0 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7c0ce520 con 0x7f5f7c15e210 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c071ed0 0x7f5f7c15d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c071ed0 0x7f5f7c15d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42394/0 (socket says 172.21.15.67:42394) 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 -- 172.21.15.67:0/1882080878 learned_addr learned my addr 172.21.15.67:0/1882080878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7b400640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c15e210 0x7f5f7c163b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f72000640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c0722b0 0x7f5f7c13d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:32.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 -- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c15e210 msgr2=0x7f5f7c163b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c15e210 0x7f5f7c163b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 -- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c0722b0 msgr2=0x7f5f7c13d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c0722b0 0x7f5f7c13d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 -- 172.21.15.67:0/1882080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c15c140 con 0x7f5f7c071ed0 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f72000640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c0722b0 0x7f5f7c13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f7aa00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c071ed0 0x7f5f7c15d1e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00e9b0 tx=0x7f5f6c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f73e00640 1 -- 172.21.15.67:0/1882080878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c016070 con 0x7f5f7c071ed0 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f73e00640 1 -- 172.21.15.67:0/1882080878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6c00cca0 con 0x7f5f7c071ed0 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7c15c420 con 0x7f5f7c071ed0 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f73e00640 1 -- 172.21.15.67:0/1882080878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6c00b040 con 0x7f5f7c071ed0 2024-04-17T10:33:32.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.717+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7c15c9e0 con 0x7f5f7c071ed0 2024-04-17T10:33:32.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.721+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f7c04ef30 con 0x7f5f7c071ed0 2024-04-17T10:33:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.721+0000 7f5f73e00640 1 -- 172.21.15.67:0/1882080878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5f6c019070 con 0x7f5f7c071ed0 2024-04-17T10:33:32.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.721+0000 7f5f73e00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f58066f90 0x7f5f58069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:32.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.721+0000 7f5f73e00640 1 -- 172.21.15.67:0/1882080878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5f6c084bd0 con 0x7f5f7c071ed0 2024-04-17T10:33:32.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.721+0000 7f5f72000640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f58066f90 0x7f5f58069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:32.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.721+0000 7f5f72000640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f58066f90 0x7f5f58069450 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c163690 tx=0x7f5f64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:32.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.725+0000 7f5f73e00640 1 -- 172.21.15.67:0/1882080878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5f6c055400 con 0x7f5f7c071ed0 2024-04-17T10:33:32.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:32.925+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5f7c0af8f0 con 0x7f5f7c071ed0 2024-04-17T10:33:33.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f73e00640 1 -- 172.21.15.67:0/1882080878 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5f6c0594a0 con 0x7f5f7c071ed0 2024-04-17T10:33:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f58066f90 msgr2=0x7f5f58069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f58066f90 0x7f5f58069450 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c163690 tx=0x7f5f64009040 comp rx=0 tx=0).stop 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c071ed0 msgr2=0x7f5f7c15d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c071ed0 0x7f5f7c15d1e0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f5f6c00e9b0 tx=0x7f5f6c00ee70 comp rx=0 tx=0).stop 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 shutdown_connections 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f58066f90 0x7f5f58069450 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f7c15e210 0x7f5f7c163b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f7c0722b0 0x7f5f7c13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 --2- 172.21.15.67:0/1882080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f7c071ed0 0x7f5f7c15d1e0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 >> 172.21.15.67:0/1882080878 conn(0x7f5f7c0b1740 msgr2=0x7f5f7c06d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 shutdown_connections 2024-04-17T10:33:33.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.525+0000 7f5f81e00640 1 -- 172.21.15.67:0/1882080878 wait complete. 2024-04-17T10:33:33.539 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:33:33.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:33.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.689+0000 7f870d200640 1 Processor -- start 2024-04-17T10:33:33.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.689+0000 7f870d200640 1 -- start start 2024-04-17T10:33:33.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f870d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8708071310 0x7f8708071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:33.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f870d200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f87080cc300 0x7f87080cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f870d200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f870806e2f0 con 0x7f8708145040 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f870d200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87080a86c0 con 0x7f87080cc300 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f870d200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8708071cb0 con 0x7f8708071310 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8705e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8708145040 0x7f870806e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59364/0 (socket says 172.21.15.67:59364) 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8705e00640 1 -- 172.21.15.67:0/1944086415 learned_addr learned my addr 172.21.15.67:0/1944086415 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8706800640 1 --2- 172.21.15.67:0/1944086415 >> v2:172.21.15.67:3300/0 conn(0x7f87080cc300 0x7f87080cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f86fd400640 1 --2- 172.21.15.67:0/1944086415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8708071310 0x7f8708071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8705400640 1 -- 172.21.15.67:0/1944086415 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062882004 0 0) 0x7f870806e2f0 con 0x7f8708145040 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8705400640 1 -- 172.21.15.67:0/1944086415 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ec003530 con 0x7f8708145040 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8706800640 1 -- 172.21.15.67:0/1944086415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8708071310 msgr2=0x7f8708071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8706800640 1 --2- 172.21.15.67:0/1944086415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8708071310 0x7f8708071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8706800640 1 -- 172.21.15.67:0/1944086415 >> v1:172.21.15.67:6789/0 conn(0x7f8708145040 legacy=0x7f870806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.693+0000 7f8706800640 1 -- 172.21.15.67:0/1944086415 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f870806d6e0 con 0x7f87080cc300 2024-04-17T10:33:33.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f8706800640 1 --2- 172.21.15.67:0/1944086415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87080cc300 0x7f87080cc750 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8700009d90 tx=0x7f870001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef324b38e10bb0f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:33.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f8705400640 1 -- 172.21.15.67:0/1944086415 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000b4070 con 0x7f87080cc300 2024-04-17T10:33:33.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f8705400640 1 -- 172.21.15.67:0/1944086415 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87000af600 con 0x7f87080cc300 2024-04-17T10:33:33.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f8705400640 1 -- 172.21.15.67:0/1944086415 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000af9b0 con 0x7f87080cc300 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- 172.21.15.67:0/1944086415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87080cc300 msgr2=0x7f87080cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 --2- 172.21.15.67:0/1944086415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87080cc300 0x7f87080cc750 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f8700009d90 tx=0x7f870001f150 comp rx=0 tx=0).stop 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- 172.21.15.67:0/1944086415 shutdown_connections 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 --2- 172.21.15.67:0/1944086415 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87080cc300 0x7f87080cc750 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 --2- 172.21.15.67:0/1944086415 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8708071310 0x7f8708071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- 172.21.15.67:0/1944086415 >> 172.21.15.67:0/1944086415 conn(0x7f870806ac90 msgr2=0x7f870806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- 172.21.15.67:0/1944086415 shutdown_connections 2024-04-17T10:33:33.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- 172.21.15.67:0/1944086415 wait complete. 2024-04-17T10:33:33.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 Processor -- start 2024-04-17T10:33:33.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- start start 2024-04-17T10:33:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8708071310 0x7f87080c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87080cc300 0x7f87080ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8708145040 0x7f870815ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870806dca0 con 0x7f8708145040 2024-04-17T10:33:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870815bf10 con 0x7f8708071310 2024-04-17T10:33:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f870d200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f870806df80 con 0x7f87080cc300 2024-04-17T10:33:33.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.697+0000 7f8706800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8708145040 0x7f870815ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:33.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8708145040 0x7f870815ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42412/0 (socket says 172.21.15.67:42412) 2024-04-17T10:33:33.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 -- 172.21.15.67:0/1008998272 learned_addr learned my addr 172.21.15.67:0/1008998272 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f86fd400640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87080cc300 0x7f87080ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8705e00640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8708071310 0x7f87080c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 -- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87080cc300 msgr2=0x7f87080ad670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87080cc300 0x7f87080ad670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 -- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8708071310 msgr2=0x7f87080c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8708071310 0x7f87080c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 -- 172.21.15.67:0/1008998272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87080ae0c0 con 0x7f8708145040 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8705e00640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8708071310 0x7f87080c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f8706800640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8708145040 0x7f870815ab80 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f87000b2040 tx=0x7f870001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f86fea00640 1 -- 172.21.15.67:0/1008998272 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000b4040 con 0x7f8708145040 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f86fea00640 1 -- 172.21.15.67:0/1008998272 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87000c1070 con 0x7f8708145040 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87080ae3a0 con 0x7f8708145040 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f86fea00640 1 -- 172.21.15.67:0/1008998272 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700009450 con 0x7f8708145040 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87081402a0 con 0x7f8708145040 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8708072d50 con 0x7f8708145040 2024-04-17T10:33:33.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f86fea00640 1 -- 172.21.15.67:0/1008998272 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f87000ba690 con 0x7f8708145040 2024-04-17T10:33:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f86fea00640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86e4066f60 0x7f86e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.701+0000 7f86fea00640 1 -- 172.21.15.67:0/1008998272 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8700123b60 con 0x7f8708145040 2024-04-17T10:33:33.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.705+0000 7f8705e00640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86e4066f60 0x7f86e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:33.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.705+0000 7f8705e00640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86e4066f60 0x7f86e4069420 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f86f8005890 tx=0x7f86f8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:33.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.705+0000 7f86fea00640 1 -- 172.21.15.67:0/1008998272 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8700018cb0 con 0x7f8708145040 2024-04-17T10:33:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8708074e30 con 0x7f8708145040 2024-04-17T10:33:33.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f86fea00640 1 -- 172.21.15.67:0/1008998272 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f87000f4a00 con 0x7f8708145040 2024-04-17T10:33:33.897 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:33.897 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:33:33.897 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86e4066f60 msgr2=0x7f86e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86e4066f60 0x7f86e4069420 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f86f8005890 tx=0x7f86f8009040 comp rx=0 tx=0).stop 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8708145040 msgr2=0x7f870815ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8708145040 0x7f870815ab80 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f87000b2040 tx=0x7f870001bef0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 shutdown_connections 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86e4066f60 0x7f86e4069420 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8708145040 0x7f870815ab80 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87080cc300 0x7f87080ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 --2- 172.21.15.67:0/1008998272 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8708071310 0x7f87080c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 >> 172.21.15.67:0/1008998272 conn(0x7f870806ac90 msgr2=0x7f870815c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 shutdown_connections 2024-04-17T10:33:33.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:33.905+0000 7f870d200640 1 -- 172.21.15.67:0/1008998272 wait complete. 2024-04-17T10:33:33.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:33:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.065+0000 7f63e2a00640 1 Processor -- start 2024-04-17T10:33:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.065+0000 7f63e2a00640 1 -- start start 2024-04-17T10:33:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63e2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63e2a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63dc06b6a0 0x7f63dc0cd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63e2a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63dc0cc8b0 con 0x7f63dc0cf880 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63e2a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63dc0cd4f0 con 0x7f63dc06b6a0 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63e2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0ccb90 con 0x7f63dc1597a0 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63e0a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f63dc0cf880 0x7f63dc15bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59376/0 (socket says 172.21.15.67:59376) 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63e0a00640 1 -- 172.21.15.67:0/3202957883 learned_addr learned my addr 172.21.15.67:0/3202957883 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63dbe00640 1 --2- 172.21.15.67:0/3202957883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63d3e00640 1 --2- 172.21.15.67:0/3202957883 >> v2:172.21.15.86:3300/0 conn(0x7f63dc06b6a0 0x7f63dc0cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63db400640 1 -- 172.21.15.67:0/3202957883 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181896931 0 0) 0x7f63dc0cc8b0 con 0x7f63dc0cf880 2024-04-17T10:33:34.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63db400640 1 -- 172.21.15.67:0/3202957883 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c0003530 con 0x7f63dc0cf880 2024-04-17T10:33:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63db400640 1 -- 172.21.15.67:0/3202957883 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2654143987 0 0) 0x7f63c0003530 con 0x7f63dc0cf880 2024-04-17T10:33:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63dbe00640 1 -- 172.21.15.67:0/3202957883 >> v2:172.21.15.86:3300/0 conn(0x7f63dc06b6a0 msgr2=0x7f63dc0cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63dbe00640 1 --2- 172.21.15.67:0/3202957883 >> v2:172.21.15.86:3300/0 conn(0x7f63dc06b6a0 0x7f63dc0cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63dbe00640 1 -- 172.21.15.67:0/3202957883 >> v1:172.21.15.67:6789/0 conn(0x7f63dc0cf880 legacy=0x7f63dc15bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.069+0000 7f63dbe00640 1 -- 172.21.15.67:0/3202957883 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc0cd180 con 0x7f63dc1597a0 2024-04-17T10:33:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63dbe00640 1 --2- 172.21.15.67:0/3202957883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0cc260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0097c0 tx=0x7f63cc01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23a794d00f981bff server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63db400640 1 -- 172.21.15.67:0/3202957883 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63cc0b0070 con 0x7f63dc1597a0 2024-04-17T10:33:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63db400640 1 -- 172.21.15.67:0/3202957883 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc0ab400 con 0x7f63dc1597a0 2024-04-17T10:33:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63db400640 1 -- 172.21.15.67:0/3202957883 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d4001a00 con 0x7f63dc1597a0 2024-04-17T10:33:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 -- 172.21.15.67:0/3202957883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 msgr2=0x7f63dc0cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3202957883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0cc260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0097c0 tx=0x7f63cc01c360 comp rx=0 tx=0).stop 2024-04-17T10:33:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 -- 172.21.15.67:0/3202957883 shutdown_connections 2024-04-17T10:33:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3202957883 >> v2:172.21.15.86:3300/0 conn(0x7f63dc06b6a0 0x7f63dc0cd980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3202957883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0cc260 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 -- 172.21.15.67:0/3202957883 >> 172.21.15.67:0/3202957883 conn(0x7f63dc0b22a0 msgr2=0x7f63dc0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 -- 172.21.15.67:0/3202957883 shutdown_connections 2024-04-17T10:33:34.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 -- 172.21.15.67:0/3202957883 wait complete. 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 Processor -- start 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.073+0000 7f63e2a00640 1 -- start start 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc06b6a0 0x7f63dc155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0cf880 0x7f63dc0c2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0c3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0b3020 con 0x7f63dc0cf880 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc0b2ea0 con 0x7f63dc06b6a0 2024-04-17T10:33:34.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63dc168dd0 con 0x7f63dc1597a0 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0cf880 0x7f63dc0c2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0cf880 0x7f63dc0c2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42428/0 (socket says 172.21.15.67:42428) 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 -- 172.21.15.67:0/3399170327 learned_addr learned my addr 172.21.15.67:0/3399170327 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e0a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0c3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63dbe00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc06b6a0 0x7f63dc155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 -- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 msgr2=0x7f63dc0c3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0c3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 -- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc06b6a0 msgr2=0x7f63dc155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc06b6a0 0x7f63dc155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 -- 172.21.15.67:0/3399170327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63dc1546a0 con 0x7f63dc0cf880 2024-04-17T10:33:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e0a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d3e00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0cf880 0x7f63dc0c2b00 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f63c400e8a0 tx=0x7f63c400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d8c00640 1 -- 172.21.15.67:0/3399170327 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4017070 con 0x7f63dc0cf880 2024-04-17T10:33:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d8c00640 1 -- 172.21.15.67:0/3399170327 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c400c940 con 0x7f63dc0cf880 2024-04-17T10:33:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d8c00640 1 -- 172.21.15.67:0/3399170327 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c400ccf0 con 0x7f63dc0cf880 2024-04-17T10:33:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63dc154980 con 0x7f63dc0cf880 2024-04-17T10:33:34.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63dc154f40 con 0x7f63dc0cf880 2024-04-17T10:33:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63dc0c4170 con 0x7f63dc0cf880 2024-04-17T10:33:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.077+0000 7f63d8c00640 1 -- 172.21.15.67:0/3399170327 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f63c401a070 con 0x7f63dc0cf880 2024-04-17T10:33:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.081+0000 7f63d8c00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63b8066f60 0x7f63b8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.081+0000 7f63d8c00640 1 -- 172.21.15.67:0/3399170327 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f63c40852e0 con 0x7f63dc0cf880 2024-04-17T10:33:34.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.081+0000 7f63dbe00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63b8066f60 0x7f63b8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.081+0000 7f63dbe00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63b8066f60 0x7f63b8069420 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ad000 tx=0x7f63cc0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.085+0000 7f63d8c00640 1 -- 172.21.15.67:0/3399170327 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f63c4055ba0 con 0x7f63dc0cf880 2024-04-17T10:33:34.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.245+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63dc09e900 con 0x7f63dc0cf880 2024-04-17T10:33:34.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63d8c00640 1 -- 172.21.15.67:0/3399170327 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16904 (secure 0 0 0) 0x7f63c4059c40 con 0x7f63dc0cf880 2024-04-17T10:33:34.239 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:34.240 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":118,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:31.509307+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:33:28.333663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-17T10:33:28.871385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63b8066f60 msgr2=0x7f63b8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63b8066f60 0x7f63b8069420 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f63cc0ad000 tx=0x7f63cc0a6000 comp rx=0 tx=0).stop 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0cf880 msgr2=0x7f63dc0c2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0cf880 0x7f63dc0c2b00 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f63c400e8a0 tx=0x7f63c400eca0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 shutdown_connections 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63b8066f60 0x7f63b8069420 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63dc1597a0 0x7f63dc0c3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63dc0cf880 0x7f63dc0c2b00 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 --2- 172.21.15.67:0/3399170327 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63dc06b6a0 0x7f63dc155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 >> 172.21.15.67:0/3399170327 conn(0x7f63dc0b22a0 msgr2=0x7f63dc0ae150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:34.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 shutdown_connections 2024-04-17T10:33:34.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.249+0000 7f63e2a00640 1 -- 172.21.15.67:0/3399170327 wait complete. 2024-04-17T10:33:34.261 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:33:34.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:33:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.409+0000 7ff717e00640 1 Processor -- start 2024-04-17T10:33:34.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.409+0000 7ff717e00640 1 -- start start 2024-04-17T10:33:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff717e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7181450b0 0x7ff71815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff717e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff717e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff71815f3e0 0x7ff71815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff717e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7180cedc0 con 0x7ff7181450b0 2024-04-17T10:33:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff717e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71806d6e0 con 0x7ff71815f3e0 2024-04-17T10:33:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff717e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff718160150 con 0x7ff7180ce420 2024-04-17T10:33:34.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7181450b0 0x7ff71815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff7181450b0 0x7ff71815bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42438/0 (socket says 172.21.15.67:42438) 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 -- 172.21.15.67:0/3507875788 learned_addr learned my addr 172.21.15.67:0/3507875788 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff717400640 1 --2- 172.21.15.67:0/3507875788 >> v2:172.21.15.86:3300/0 conn(0x7ff71815f3e0 0x7ff71815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff70e000640 1 --2- 172.21.15.67:0/3507875788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 -- 172.21.15.67:0/3507875788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 msgr2=0x7ff7180ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 --2- 172.21.15.67:0/3507875788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 -- 172.21.15.67:0/3507875788 >> v2:172.21.15.86:3300/0 conn(0x7ff71815f3e0 msgr2=0x7ff71815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 --2- 172.21.15.67:0/3507875788 >> v2:172.21.15.86:3300/0 conn(0x7ff71815f3e0 0x7ff71815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 -- 172.21.15.67:0/3507875788 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71806d930 con 0x7ff7181450b0 2024-04-17T10:33:34.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.413+0000 7ff716a00640 1 --2- 172.21.15.67:0/3507875788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7181450b0 0x7ff71815bd80 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff718160360 tx=0x7ff70801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b5c0fbe64e8e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff716000640 1 -- 172.21.15.67:0/3507875788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7080a56d0 con 0x7ff7181450b0 2024-04-17T10:33:34.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff716000640 1 -- 172.21.15.67:0/3507875788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7080a5870 con 0x7ff7181450b0 2024-04-17T10:33:34.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff716000640 1 -- 172.21.15.67:0/3507875788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7080a5c40 con 0x7ff7181450b0 2024-04-17T10:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- 172.21.15.67:0/3507875788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7181450b0 msgr2=0x7ff71815bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 --2- 172.21.15.67:0/3507875788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7181450b0 0x7ff71815bd80 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff718160360 tx=0x7ff70801c960 comp rx=0 tx=0).stop 2024-04-17T10:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- 172.21.15.67:0/3507875788 shutdown_connections 2024-04-17T10:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 --2- 172.21.15.67:0/3507875788 >> v2:172.21.15.86:3300/0 conn(0x7ff71815f3e0 0x7ff71815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 --2- 172.21.15.67:0/3507875788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 --2- 172.21.15.67:0/3507875788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7181450b0 0x7ff71815bd80 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- 172.21.15.67:0/3507875788 >> 172.21.15.67:0/3507875788 conn(0x7ff718067070 msgr2=0x7ff71806ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- 172.21.15.67:0/3507875788 shutdown_connections 2024-04-17T10:33:34.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- 172.21.15.67:0/3507875788 wait complete. 2024-04-17T10:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 Processor -- start 2024-04-17T10:33:34.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- start start 2024-04-17T10:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7181450b0 0x7ff7181594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71815f3e0 0x7ff71815a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71815fed0 con 0x7ff71815f3e0 2024-04-17T10:33:34.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71815fd50 con 0x7ff7181450b0 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71806ec20 con 0x7ff7180ce420 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71815f3e0 0x7ff71815a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71815f3e0 0x7ff71815a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42444/0 (socket says 172.21.15.67:42444) 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff717400640 1 -- 172.21.15.67:0/2208526996 learned_addr learned my addr 172.21.15.67:0/2208526996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff716a00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.417+0000 7ff70e000640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7181450b0 0x7ff7181594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717400640 1 -- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 msgr2=0x7ff7180cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717400640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180cb640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717400640 1 -- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7181450b0 msgr2=0x7ff7181594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717400640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7181450b0 0x7ff7181594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717400640 1 -- 172.21.15.67:0/2208526996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7180cb360 con 0x7ff71815f3e0 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717400640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71815f3e0 0x7ff71815a910 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff71000d590 tx=0x7ff71000da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff70f400640 1 -- 172.21.15.67:0/2208526996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7100125f0 con 0x7ff71815f3e0 2024-04-17T10:33:34.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff70f400640 1 -- 172.21.15.67:0/2208526996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff710019070 con 0x7ff71815f3e0 2024-04-17T10:33:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff70f400640 1 -- 172.21.15.67:0/2208526996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7100129f0 con 0x7ff71815f3e0 2024-04-17T10:33:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71813f720 con 0x7ff71815f3e0 2024-04-17T10:33:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71813fce0 con 0x7ff71815f3e0 2024-04-17T10:33:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71813fee0 con 0x7ff71815f3e0 2024-04-17T10:33:34.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff70f400640 1 -- 172.21.15.67:0/2208526996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff710012b90 con 0x7ff71815f3e0 2024-04-17T10:33:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff70f400640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6f4066f60 0x7ff6f4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.421+0000 7ff70f400640 1 -- 172.21.15.67:0/2208526996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff7100853e0 con 0x7ff71815f3e0 2024-04-17T10:33:34.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.425+0000 7ff716a00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6f4066f60 0x7ff6f4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.425+0000 7ff716a00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6f4066f60 0x7ff6f4069420 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff7080ae000 tx=0x7ff7080a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.425+0000 7ff70f400640 1 -- 172.21.15.67:0/2208526996 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff71813fee0 con 0x7ff71815f3e0 2024-04-17T10:33:34.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.589+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff7180cbc40 con 0x7ff6f4066f60 2024-04-17T10:33:34.586 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.593+0000 7ff70f400640 1 -- 172.21.15.67:0/2208526996 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252486 (secure 0 0 0) 0x7ff7180cbc40 con 0x7ff6f4066f60 2024-04-17T10:33:34.590 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:34.591 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:33:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6f4066f60 msgr2=0x7ff6f4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6f4066f60 0x7ff6f4069420 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7ff7080ae000 tx=0x7ff7080a7070 comp rx=0 tx=0).stop 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71815f3e0 msgr2=0x7ff71815a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71815f3e0 0x7ff71815a910 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7ff71000d590 tx=0x7ff71000da50 comp rx=0 tx=0).stop 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 shutdown_connections 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6f4066f60 0x7ff6f4069420 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff71815f3e0 0x7ff71815a910 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7181450b0 0x7ff7181594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 --2- 172.21.15.67:0/2208526996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7180ce420 0x7ff7180cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 >> 172.21.15.67:0/2208526996 conn(0x7ff718067070 msgr2=0x7ff71815c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 shutdown_connections 2024-04-17T10:33:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.597+0000 7ff717e00640 1 -- 172.21.15.67:0/2208526996 wait complete. 2024-04-17T10:33:34.610 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-04-17T10:33:34.260365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423156,"kb_used_data":12676,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754551564,"statfs":{"total":773094113280,"available":772660801536,"internally_reserved":0,"allocated":12980224,"data_stored":6662376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":420204081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.794602"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514773+0000","last_change":"2024-04-17T10:33:30.517880+0000","last_active":"2024-04-17T10:33:31.514773+0000","last_peered":"2024-04-17T10:33:31.514773+0000","last_clean":"2024-04-17T10:33:31.514773+0000","last_became_active":"2024-04-17T10:33:30.517617+0000","last_became_peered":"2024-04-17T10:33:30.517617+0000","last_unstale":"2024-04-17T10:33:31.514773+0000","last_undegraded":"2024-04-17T10:33:31.514773+0000","last_fullsized":"2024-04-17T10:33:31.514773+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:55.336606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613281+0000","last_change":"2024-04-17T10:33:30.514070+0000","last_active":"2024-04-17T10:33:31.613281+0000","last_peered":"2024-04-17T10:33:31.613281+0000","last_clean":"2024-04-17T10:33:31.613281+0000","last_became_active":"2024-04-17T10:33:30.513292+0000","last_became_peered":"2024-04-17T10:33:30.513292+0000","last_unstale":"2024-04-17T10:33:31.613281+0000","last_undegraded":"2024-04-17T10:33:31.613281+0000","last_fullsized":"2024-04-17T10:33:31.613281+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:51.485725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613129+0000","last_change":"2024-04-17T10:33:30.519522+0000","last_active":"2024-04-17T10:33:31.613129+0000","last_peered":"2024-04-17T10:33:31.613129+0000","last_clean":"2024-04-17T10:33:31.613129+0000","last_became_active":"2024-04-17T10:33:30.519240+0000","last_became_peered":"2024-04-17T10:33:30.519240+0000","last_unstale":"2024-04-17T10:33:31.613129+0000","last_undegraded":"2024-04-17T10:33:31.613129+0000","last_fullsized":"2024-04-17T10:33:31.613129+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:58.535058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609739+0000","last_change":"2024-04-17T10:33:30.524405+0000","last_active":"2024-04-17T10:33:31.609739+0000","last_peered":"2024-04-17T10:33:31.609739+0000","last_clean":"2024-04-17T10:33:31.609739+0000","last_became_active":"2024-04-17T10:33:30.524051+0000","last_became_peered":"2024-04-17T10:33:30.524051+0000","last_unstale":"2024-04-17T10:33:31.609739+0000","last_undegraded":"2024-04-17T10:33:31.609739+0000","last_fullsized":"2024-04-17T10:33:31.609739+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:48.741482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514609+0000","last_change":"2024-04-17T10:33:30.515073+0000","last_active":"2024-04-17T10:33:31.514609+0000","last_peered":"2024-04-17T10:33:31.514609+0000","last_clean":"2024-04-17T10:33:31.514609+0000","last_became_active":"2024-04-17T10:33:30.514021+0000","last_became_peered":"2024-04-17T10:33:30.514021+0000","last_unstale":"2024-04-17T10:33:31.514609+0000","last_undegraded":"2024-04-17T10:33:31.514609+0000","last_fullsized":"2024-04-17T10:33:31.514609+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:47.592056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529806+0000","last_change":"2024-04-17T10:33:30.525132+0000","last_active":"2024-04-17T10:33:31.529806+0000","last_peered":"2024-04-17T10:33:31.529806+0000","last_clean":"2024-04-17T10:33:31.529806+0000","last_became_active":"2024-04-17T10:33:30.524815+0000","last_became_peered":"2024-04-17T10:33:30.524815+0000","last_unstale":"2024-04-17T10:33:31.529806+0000","last_undegraded":"2024-04-17T10:33:31.529806+0000","last_fullsized":"2024-04-17T10:33:31.529806+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:57:54.256415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529723+0000","last_change":"2024-04-17T10:33:30.528241+0000","last_active":"2024-04-17T10:33:31.529723+0000","last_peered":"2024-04-17T10:33:31.529723+0000","last_clean":"2024-04-17T10:33:31.529723+0000","last_became_active":"2024-04-17T10:33:30.527987+0000","last_became_peered":"2024-04-17T10:33:30.527987+0000","last_unstale":"2024-04-17T10:33:31.529723+0000","last_undegraded":"2024-04-17T10:33:31.529723+0000","last_fullsized":"2024-04-17T10:33:31.529723+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:53.733211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612560+0000","last_change":"2024-04-17T10:33:30.520211+0000","last_active":"2024-04-17T10:33:31.612560+0000","last_peered":"2024-04-17T10:33:31.612560+0000","last_clean":"2024-04-17T10:33:31.612560+0000","last_became_active":"2024-04-17T10:33:30.519711+0000","last_became_peered":"2024-04-17T10:33:30.519711+0000","last_unstale":"2024-04-17T10:33:31.612560+0000","last_undegraded":"2024-04-17T10:33:31.612560+0000","last_fullsized":"2024-04-17T10:33:31.612560+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:38.216709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.610095+0000","last_change":"2024-04-17T10:33:30.524736+0000","last_active":"2024-04-17T10:33:31.610095+0000","last_peered":"2024-04-17T10:33:31.610095+0000","last_clean":"2024-04-17T10:33:31.610095+0000","last_became_active":"2024-04-17T10:33:30.523635+0000","last_became_peered":"2024-04-17T10:33:30.523635+0000","last_unstale":"2024-04-17T10:33:31.610095+0000","last_undegraded":"2024-04-17T10:33:31.610095+0000","last_fullsized":"2024-04-17T10:33:31.610095+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:53.545591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613011+0000","last_change":"2024-04-17T10:33:30.519167+0000","last_active":"2024-04-17T10:33:31.613011+0000","last_peered":"2024-04-17T10:33:31.613011+0000","last_clean":"2024-04-17T10:33:31.613011+0000","last_became_active":"2024-04-17T10:33:30.518906+0000","last_became_peered":"2024-04-17T10:33:30.518906+0000","last_unstale":"2024-04-17T10:33:31.613011+0000","last_undegraded":"2024-04-17T10:33:31.613011+0000","last_fullsized":"2024-04-17T10:33:31.613011+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:12.614426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514322+0000","last_change":"2024-04-17T10:33:30.513613+0000","last_active":"2024-04-17T10:33:31.514322+0000","last_peered":"2024-04-17T10:33:31.514322+0000","last_clean":"2024-04-17T10:33:31.514322+0000","last_became_active":"2024-04-17T10:33:30.513312+0000","last_became_peered":"2024-04-17T10:33:30.513312+0000","last_unstale":"2024-04-17T10:33:31.514322+0000","last_undegraded":"2024-04-17T10:33:31.514322+0000","last_fullsized":"2024-04-17T10:33:31.514322+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:21.853803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530247+0000","last_change":"2024-04-17T10:33:30.506874+0000","last_active":"2024-04-17T10:33:31.530247+0000","last_peered":"2024-04-17T10:33:31.530247+0000","last_clean":"2024-04-17T10:33:31.530247+0000","last_became_active":"2024-04-17T10:33:30.506086+0000","last_became_peered":"2024-04-17T10:33:30.506086+0000","last_unstale":"2024-04-17T10:33:31.530247+0000","last_undegraded":"2024-04-17T10:33:31.530247+0000","last_fullsized":"2024-04-17T10:33:31.530247+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.562886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612836+0000","last_change":"2024-04-17T10:33:30.520353+0000","last_active":"2024-04-17T10:33:31.612836+0000","last_peered":"2024-04-17T10:33:31.612836+0000","last_clean":"2024-04-17T10:33:31.612836+0000","last_became_active":"2024-04-17T10:33:30.519575+0000","last_became_peered":"2024-04-17T10:33:30.519575+0000","last_unstale":"2024-04-17T10:33:31.612836+0000","last_undegraded":"2024-04-17T10:33:31.612836+0000","last_fullsized":"2024-04-17T10:33:31.612836+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:00:22.416979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612825+0000","last_change":"2024-04-17T10:33:30.520547+0000","last_active":"2024-04-17T10:33:31.612825+0000","last_peered":"2024-04-17T10:33:31.612825+0000","last_clean":"2024-04-17T10:33:31.612825+0000","last_became_active":"2024-04-17T10:33:30.520259+0000","last_became_peered":"2024-04-17T10:33:30.520259+0000","last_unstale":"2024-04-17T10:33:31.612825+0000","last_undegraded":"2024-04-17T10:33:31.612825+0000","last_fullsized":"2024-04-17T10:33:31.612825+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.599913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513878+0000","last_change":"2024-04-17T10:33:30.514355+0000","last_active":"2024-04-17T10:33:31.513878+0000","last_peered":"2024-04-17T10:33:31.513878+0000","last_clean":"2024-04-17T10:33:31.513878+0000","last_became_active":"2024-04-17T10:33:30.514033+0000","last_became_peered":"2024-04-17T10:33:30.514033+0000","last_unstale":"2024-04-17T10:33:31.513878+0000","last_undegraded":"2024-04-17T10:33:31.513878+0000","last_fullsized":"2024-04-17T10:33:31.513878+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:32.701908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529651+0000","last_change":"2024-04-17T10:33:30.511617+0000","last_active":"2024-04-17T10:33:31.529651+0000","last_peered":"2024-04-17T10:33:31.529651+0000","last_clean":"2024-04-17T10:33:31.529651+0000","last_became_active":"2024-04-17T10:33:30.511282+0000","last_became_peered":"2024-04-17T10:33:30.511282+0000","last_unstale":"2024-04-17T10:33:31.529651+0000","last_undegraded":"2024-04-17T10:33:31.529651+0000","last_fullsized":"2024-04-17T10:33:31.529651+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:51:34.981246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613126+0000","last_change":"2024-04-17T10:33:30.517765+0000","last_active":"2024-04-17T10:33:31.613126+0000","last_peered":"2024-04-17T10:33:31.613126+0000","last_clean":"2024-04-17T10:33:31.613126+0000","last_became_active":"2024-04-17T10:33:30.517451+0000","last_became_peered":"2024-04-17T10:33:30.517451+0000","last_unstale":"2024-04-17T10:33:31.613126+0000","last_undegraded":"2024-04-17T10:33:31.613126+0000","last_fullsized":"2024-04-17T10:33:31.613126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:30.853095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613388+0000","last_change":"2024-04-17T10:33:30.516067+0000","last_active":"2024-04-17T10:33:31.613388+0000","last_peered":"2024-04-17T10:33:31.613388+0000","last_clean":"2024-04-17T10:33:31.613388+0000","last_became_active":"2024-04-17T10:33:30.515706+0000","last_became_peered":"2024-04-17T10:33:30.515706+0000","last_unstale":"2024-04-17T10:33:31.613388+0000","last_undegraded":"2024-04-17T10:33:31.613388+0000","last_fullsized":"2024-04-17T10:33:31.613388+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:14.541115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524624+0000","last_change":"2024-04-17T10:33:30.524529+0000","last_active":"2024-04-17T10:33:31.524624+0000","last_peered":"2024-04-17T10:33:31.524624+0000","last_clean":"2024-04-17T10:33:31.524624+0000","last_became_active":"2024-04-17T10:33:30.524272+0000","last_became_peered":"2024-04-17T10:33:30.524272+0000","last_unstale":"2024-04-17T10:33:31.524624+0000","last_undegraded":"2024-04-17T10:33:31.524624+0000","last_fullsized":"2024-04-17T10:33:31.524624+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:44.795215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518614+0000","last_change":"2024-04-17T10:33:30.512314+0000","last_active":"2024-04-17T10:33:31.518614+0000","last_peered":"2024-04-17T10:33:31.518614+0000","last_clean":"2024-04-17T10:33:31.518614+0000","last_became_active":"2024-04-17T10:33:30.512010+0000","last_became_peered":"2024-04-17T10:33:30.512010+0000","last_unstale":"2024-04-17T10:33:31.518614+0000","last_undegraded":"2024-04-17T10:33:31.518614+0000","last_fullsized":"2024-04-17T10:33:31.518614+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:27.471947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609841+0000","last_change":"2024-04-17T10:33:30.527262+0000","last_active":"2024-04-17T10:33:31.609841+0000","last_peered":"2024-04-17T10:33:31.609841+0000","last_clean":"2024-04-17T10:33:31.609841+0000","last_became_active":"2024-04-17T10:33:30.526790+0000","last_became_peered":"2024-04-17T10:33:30.526790+0000","last_unstale":"2024-04-17T10:33:31.609841+0000","last_undegraded":"2024-04-17T10:33:31.609841+0000","last_fullsized":"2024-04-17T10:33:31.609841+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:38.216709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514275+0000","last_change":"2024-04-17T10:33:30.521026+0000","last_active":"2024-04-17T10:33:31.514275+0000","last_peered":"2024-04-17T10:33:31.514275+0000","last_clean":"2024-04-17T10:33:31.514275+0000","last_became_active":"2024-04-17T10:33:30.520723+0000","last_became_peered":"2024-04-17T10:33:30.520723+0000","last_unstale":"2024-04-17T10:33:31.514275+0000","last_undegraded":"2024-04-17T10:33:31.514275+0000","last_fullsized":"2024-04-17T10:33:31.514275+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:02.016724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609493+0000","last_change":"2024-04-17T10:33:30.530596+0000","last_active":"2024-04-17T10:33:31.609493+0000","last_peered":"2024-04-17T10:33:31.609493+0000","last_clean":"2024-04-17T10:33:31.609493+0000","last_became_active":"2024-04-17T10:33:30.530312+0000","last_became_peered":"2024-04-17T10:33:30.530312+0000","last_unstale":"2024-04-17T10:33:31.609493+0000","last_undegraded":"2024-04-17T10:33:31.609493+0000","last_fullsized":"2024-04-17T10:33:31.609493+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:57.706709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513491+0000","last_change":"2024-04-17T10:33:30.523226+0000","last_active":"2024-04-17T10:33:31.513491+0000","last_peered":"2024-04-17T10:33:31.513491+0000","last_clean":"2024-04-17T10:33:31.513491+0000","last_became_active":"2024-04-17T10:33:30.522944+0000","last_became_peered":"2024-04-17T10:33:30.522944+0000","last_unstale":"2024-04-17T10:33:31.513491+0000","last_undegraded":"2024-04-17T10:33:31.513491+0000","last_fullsized":"2024-04-17T10:33:31.513491+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:30:06.352801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514584+0000","last_change":"2024-04-17T10:33:30.519734+0000","last_active":"2024-04-17T10:33:31.514584+0000","last_peered":"2024-04-17T10:33:31.514584+0000","last_clean":"2024-04-17T10:33:31.514584+0000","last_became_active":"2024-04-17T10:33:30.519440+0000","last_became_peered":"2024-04-17T10:33:30.519440+0000","last_unstale":"2024-04-17T10:33:31.514584+0000","last_undegraded":"2024-04-17T10:33:31.514584+0000","last_fullsized":"2024-04-17T10:33:31.514584+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:13.368923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518720+0000","last_change":"2024-04-17T10:33:30.518449+0000","last_active":"2024-04-17T10:33:31.518720+0000","last_peered":"2024-04-17T10:33:31.518720+0000","last_clean":"2024-04-17T10:33:31.518720+0000","last_became_active":"2024-04-17T10:33:30.518180+0000","last_became_peered":"2024-04-17T10:33:30.518180+0000","last_unstale":"2024-04-17T10:33:31.518720+0000","last_undegraded":"2024-04-17T10:33:31.518720+0000","last_fullsized":"2024-04-17T10:33:31.518720+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:20.469351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524604+0000","last_change":"2024-04-17T10:33:30.529753+0000","last_active":"2024-04-17T10:33:31.524604+0000","last_peered":"2024-04-17T10:33:31.524604+0000","last_clean":"2024-04-17T10:33:31.524604+0000","last_became_active":"2024-04-17T10:33:30.529285+0000","last_became_peered":"2024-04-17T10:33:30.529285+0000","last_unstale":"2024-04-17T10:33:31.524604+0000","last_undegraded":"2024-04-17T10:33:31.524604+0000","last_fullsized":"2024-04-17T10:33:31.524604+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:04.491386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530141+0000","last_change":"2024-04-17T10:33:30.516445+0000","last_active":"2024-04-17T10:33:31.530141+0000","last_peered":"2024-04-17T10:33:31.530141+0000","last_clean":"2024-04-17T10:33:31.530141+0000","last_became_active":"2024-04-17T10:33:30.516056+0000","last_became_peered":"2024-04-17T10:33:30.516056+0000","last_unstale":"2024-04-17T10:33:31.530141+0000","last_undegraded":"2024-04-17T10:33:31.530141+0000","last_fullsized":"2024-04-17T10:33:31.530141+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:48.556618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518575+0000","last_change":"2024-04-17T10:33:30.526282+0000","last_active":"2024-04-17T10:33:31.518575+0000","last_peered":"2024-04-17T10:33:31.518575+0000","last_clean":"2024-04-17T10:33:31.518575+0000","last_became_active":"2024-04-17T10:33:30.525850+0000","last_became_peered":"2024-04-17T10:33:30.525850+0000","last_unstale":"2024-04-17T10:33:31.518575+0000","last_undegraded":"2024-04-17T10:33:31.518575+0000","last_fullsized":"2024-04-17T10:33:31.518575+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:10.126907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612903+0000","last_change":"2024-04-17T10:33:30.513650+0000","last_active":"2024-04-17T10:33:31.612903+0000","last_peered":"2024-04-17T10:33:31.612903+0000","last_clean":"2024-04-17T10:33:31.612903+0000","last_became_active":"2024-04-17T10:33:30.513394+0000","last_became_peered":"2024-04-17T10:33:30.513394+0000","last_unstale":"2024-04-17T10:33:31.612903+0000","last_undegraded":"2024-04-17T10:33:31.612903+0000","last_fullsized":"2024-04-17T10:33:31.612903+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:23.766536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513633+0000","last_change":"2024-04-17T10:33:30.512740+0000","last_active":"2024-04-17T10:33:31.513633+0000","last_peered":"2024-04-17T10:33:31.513633+0000","last_clean":"2024-04-17T10:33:31.513633+0000","last_became_active":"2024-04-17T10:33:30.512393+0000","last_became_peered":"2024-04-17T10:33:30.512393+0000","last_unstale":"2024-04-17T10:33:31.513633+0000","last_undegraded":"2024-04-17T10:33:31.513633+0000","last_fullsized":"2024-04-17T10:33:31.513633+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:14:01.311909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513544+0000","last_change":"2024-04-17T10:33:30.522878+0000","last_active":"2024-04-17T10:33:31.513544+0000","last_peered":"2024-04-17T10:33:31.513544+0000","last_clean":"2024-04-17T10:33:31.513544+0000","last_became_active":"2024-04-17T10:33:30.522568+0000","last_became_peered":"2024-04-17T10:33:30.522568+0000","last_unstale":"2024-04-17T10:33:31.513544+0000","last_undegraded":"2024-04-17T10:33:31.513544+0000","last_fullsized":"2024-04-17T10:33:31.513544+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:47.891876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612640+0000","last_change":"2024-04-17T10:33:30.517703+0000","last_active":"2024-04-17T10:33:31.612640+0000","last_peered":"2024-04-17T10:33:31.612640+0000","last_clean":"2024-04-17T10:33:31.612640+0000","last_became_active":"2024-04-17T10:33:30.517367+0000","last_became_peered":"2024-04-17T10:33:30.517367+0000","last_unstale":"2024-04-17T10:33:31.612640+0000","last_undegraded":"2024-04-17T10:33:31.612640+0000","last_fullsized":"2024-04-17T10:33:31.612640+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:31.291283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"117'4","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524670+0000","last_change":"2024-04-17T10:33:29.488987+0000","last_active":"2024-04-17T10:33:31.524670+0000","last_peered":"2024-04-17T10:33:31.524670+0000","last_clean":"2024-04-17T10:33:31.524670+0000","last_became_active":"2024-04-17T10:33:29.488649+0000","last_became_peered":"2024-04-17T10:33:29.488649+0000","last_unstale":"2024-04-17T10:33:31.524670+0000","last_undegraded":"2024-04-17T10:33:31.524670+0000","last_fullsized":"2024-04-17T10:33:31.524670+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:03.802935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530094+0000","last_change":"2024-04-17T10:33:30.525973+0000","last_active":"2024-04-17T10:33:31.530094+0000","last_peered":"2024-04-17T10:33:31.530094+0000","last_clean":"2024-04-17T10:33:31.530094+0000","last_became_active":"2024-04-17T10:33:30.525740+0000","last_became_peered":"2024-04-17T10:33:30.525740+0000","last_unstale":"2024-04-17T10:33:31.530094+0000","last_undegraded":"2024-04-17T10:33:31.530094+0000","last_fullsized":"2024-04-17T10:33:31.530094+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:59.558573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612997+0000","last_change":"2024-04-17T10:33:29.490892+0000","last_active":"2024-04-17T10:33:31.612997+0000","last_peered":"2024-04-17T10:33:31.612997+0000","last_clean":"2024-04-17T10:33:31.612997+0000","last_became_active":"2024-04-17T10:33:29.490624+0000","last_became_peered":"2024-04-17T10:33:29.490624+0000","last_unstale":"2024-04-17T10:33:31.612997+0000","last_undegraded":"2024-04-17T10:33:31.612997+0000","last_fullsized":"2024-04-17T10:33:31.612997+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:43.542422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514383+0000","last_change":"2024-04-17T10:33:30.521938+0000","last_active":"2024-04-17T10:33:31.514383+0000","last_peered":"2024-04-17T10:33:31.514383+0000","last_clean":"2024-04-17T10:33:31.514383+0000","last_became_active":"2024-04-17T10:33:30.521512+0000","last_became_peered":"2024-04-17T10:33:30.521512+0000","last_unstale":"2024-04-17T10:33:31.514383+0000","last_undegraded":"2024-04-17T10:33:31.514383+0000","last_fullsized":"2024-04-17T10:33:31.514383+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:57.706709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609914+0000","last_change":"2024-04-17T10:33:29.494875+0000","last_active":"2024-04-17T10:33:31.609914+0000","last_peered":"2024-04-17T10:33:31.609914+0000","last_clean":"2024-04-17T10:33:31.609914+0000","last_became_active":"2024-04-17T10:33:29.494626+0000","last_became_peered":"2024-04-17T10:33:29.494626+0000","last_unstale":"2024-04-17T10:33:31.609914+0000","last_undegraded":"2024-04-17T10:33:31.609914+0000","last_fullsized":"2024-04-17T10:33:31.609914+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:49.924443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.610136+0000","last_change":"2024-04-17T10:33:30.524503+0000","last_active":"2024-04-17T10:33:31.610136+0000","last_peered":"2024-04-17T10:33:31.610136+0000","last_clean":"2024-04-17T10:33:31.610136+0000","last_became_active":"2024-04-17T10:33:30.523960+0000","last_became_peered":"2024-04-17T10:33:30.523960+0000","last_unstale":"2024-04-17T10:33:31.610136+0000","last_undegraded":"2024-04-17T10:33:31.610136+0000","last_fullsized":"2024-04-17T10:33:31.610136+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:57.278375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612747+0000","last_change":"2024-04-17T10:33:29.495956+0000","last_active":"2024-04-17T10:33:31.612747+0000","last_peered":"2024-04-17T10:33:31.612747+0000","last_clean":"2024-04-17T10:33:31.612747+0000","last_became_active":"2024-04-17T10:33:29.495664+0000","last_became_peered":"2024-04-17T10:33:29.495664+0000","last_unstale":"2024-04-17T10:33:31.612747+0000","last_undegraded":"2024-04-17T10:33:31.612747+0000","last_fullsized":"2024-04-17T10:33:31.612747+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:20.135789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612353+0000","last_change":"2024-04-17T10:33:30.519779+0000","last_active":"2024-04-17T10:33:31.612353+0000","last_peered":"2024-04-17T10:33:31.612353+0000","last_clean":"2024-04-17T10:33:31.612353+0000","last_became_active":"2024-04-17T10:33:30.519499+0000","last_became_peered":"2024-04-17T10:33:30.519499+0000","last_unstale":"2024-04-17T10:33:31.612353+0000","last_undegraded":"2024-04-17T10:33:31.612353+0000","last_fullsized":"2024-04-17T10:33:31.612353+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:04.491386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514630+0000","last_change":"2024-04-17T10:33:29.484585+0000","last_active":"2024-04-17T10:33:31.514630+0000","last_peered":"2024-04-17T10:33:31.514630+0000","last_clean":"2024-04-17T10:33:31.514630+0000","last_became_active":"2024-04-17T10:33:29.483420+0000","last_became_peered":"2024-04-17T10:33:29.483420+0000","last_unstale":"2024-04-17T10:33:31.514630+0000","last_undegraded":"2024-04-17T10:33:31.514630+0000","last_fullsized":"2024-04-17T10:33:31.514630+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:45.687248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514052+0000","last_change":"2024-04-17T10:33:30.521856+0000","last_active":"2024-04-17T10:33:31.514052+0000","last_peered":"2024-04-17T10:33:31.514052+0000","last_clean":"2024-04-17T10:33:31.514052+0000","last_became_active":"2024-04-17T10:33:30.521588+0000","last_became_peered":"2024-04-17T10:33:30.521588+0000","last_unstale":"2024-04-17T10:33:31.514052+0000","last_undegraded":"2024-04-17T10:33:31.514052+0000","last_fullsized":"2024-04-17T10:33:31.514052+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:17.229608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529919+0000","last_change":"2024-04-17T10:33:29.499168+0000","last_active":"2024-04-17T10:33:31.529919+0000","last_peered":"2024-04-17T10:33:31.529919+0000","last_clean":"2024-04-17T10:33:31.529919+0000","last_became_active":"2024-04-17T10:33:29.498878+0000","last_became_peered":"2024-04-17T10:33:29.498878+0000","last_unstale":"2024-04-17T10:33:31.529919+0000","last_undegraded":"2024-04-17T10:33:31.529919+0000","last_fullsized":"2024-04-17T10:33:31.529919+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:01.294357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612977+0000","last_change":"2024-04-17T10:33:30.518406+0000","last_active":"2024-04-17T10:33:31.612977+0000","last_peered":"2024-04-17T10:33:31.612977+0000","last_clean":"2024-04-17T10:33:31.612977+0000","last_became_active":"2024-04-17T10:33:30.518092+0000","last_became_peered":"2024-04-17T10:33:30.518092+0000","last_unstale":"2024-04-17T10:33:31.612977+0000","last_undegraded":"2024-04-17T10:33:31.612977+0000","last_fullsized":"2024-04-17T10:33:31.612977+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:53.545591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513624+0000","last_change":"2024-04-17T10:33:29.482197+0000","last_active":"2024-04-17T10:33:31.513624+0000","last_peered":"2024-04-17T10:33:31.513624+0000","last_clean":"2024-04-17T10:33:31.513624+0000","last_became_active":"2024-04-17T10:33:29.481902+0000","last_became_peered":"2024-04-17T10:33:29.481902+0000","last_unstale":"2024-04-17T10:33:31.513624+0000","last_undegraded":"2024-04-17T10:33:31.513624+0000","last_fullsized":"2024-04-17T10:33:31.513624+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:11.604372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612485+0000","last_change":"2024-04-17T10:33:30.530577+0000","last_active":"2024-04-17T10:33:31.612485+0000","last_peered":"2024-04-17T10:33:31.612485+0000","last_clean":"2024-04-17T10:33:31.612485+0000","last_became_active":"2024-04-17T10:33:30.530268+0000","last_became_peered":"2024-04-17T10:33:30.530268+0000","last_unstale":"2024-04-17T10:33:31.612485+0000","last_undegraded":"2024-04-17T10:33:31.612485+0000","last_fullsized":"2024-04-17T10:33:31.612485+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:56.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530039+0000","last_change":"2024-04-17T10:33:29.498168+0000","last_active":"2024-04-17T10:33:31.530039+0000","last_peered":"2024-04-17T10:33:31.530039+0000","last_clean":"2024-04-17T10:33:31.530039+0000","last_became_active":"2024-04-17T10:33:29.497688+0000","last_became_peered":"2024-04-17T10:33:29.497688+0000","last_unstale":"2024-04-17T10:33:31.530039+0000","last_undegraded":"2024-04-17T10:33:31.530039+0000","last_fullsized":"2024-04-17T10:33:31.530039+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:52:32.030486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524373+0000","last_change":"2024-04-17T10:33:30.511638+0000","last_active":"2024-04-17T10:33:31.524373+0000","last_peered":"2024-04-17T10:33:31.524373+0000","last_clean":"2024-04-17T10:33:31.524373+0000","last_became_active":"2024-04-17T10:33:30.511369+0000","last_became_peered":"2024-04-17T10:33:30.511369+0000","last_unstale":"2024-04-17T10:33:31.524373+0000","last_undegraded":"2024-04-17T10:33:31.524373+0000","last_fullsized":"2024-04-17T10:33:31.524373+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:14:09.976313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524642+0000","last_change":"2024-04-17T10:33:29.485622+0000","last_active":"2024-04-17T10:33:31.524642+0000","last_peered":"2024-04-17T10:33:31.524642+0000","last_clean":"2024-04-17T10:33:31.524642+0000","last_became_active":"2024-04-17T10:33:29.485350+0000","last_became_peered":"2024-04-17T10:33:29.485350+0000","last_unstale":"2024-04-17T10:33:31.524642+0000","last_undegraded":"2024-04-17T10:33:31.524642+0000","last_fullsized":"2024-04-17T10:33:31.524642+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:12.191975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514265+0000","last_change":"2024-04-17T10:33:30.520954+0000","last_active":"2024-04-17T10:33:31.514265+0000","last_peered":"2024-04-17T10:33:31.514265+0000","last_clean":"2024-04-17T10:33:31.514265+0000","last_became_active":"2024-04-17T10:33:30.520672+0000","last_became_peered":"2024-04-17T10:33:30.520672+0000","last_unstale":"2024-04-17T10:33:31.514265+0000","last_undegraded":"2024-04-17T10:33:31.514265+0000","last_fullsized":"2024-04-17T10:33:31.514265+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:29.525176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514474+0000","last_change":"2024-04-17T10:33:29.486831+0000","last_active":"2024-04-17T10:33:31.514474+0000","last_peered":"2024-04-17T10:33:31.514474+0000","last_clean":"2024-04-17T10:33:31.514474+0000","last_became_active":"2024-04-17T10:33:29.486204+0000","last_became_peered":"2024-04-17T10:33:29.486204+0000","last_unstale":"2024-04-17T10:33:31.514474+0000","last_undegraded":"2024-04-17T10:33:31.514474+0000","last_fullsized":"2024-04-17T10:33:31.514474+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:31.691854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613416+0000","last_change":"2024-04-17T10:33:30.519388+0000","last_active":"2024-04-17T10:33:31.613416+0000","last_peered":"2024-04-17T10:33:31.613416+0000","last_clean":"2024-04-17T10:33:31.613416+0000","last_became_active":"2024-04-17T10:33:30.519104+0000","last_became_peered":"2024-04-17T10:33:30.519104+0000","last_unstale":"2024-04-17T10:33:31.613416+0000","last_undegraded":"2024-04-17T10:33:31.613416+0000","last_fullsized":"2024-04-17T10:33:31.613416+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:00.490108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"117'3","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514374+0000","last_change":"2024-04-17T10:33:29.485972+0000","last_active":"2024-04-17T10:33:31.514374+0000","last_peered":"2024-04-17T10:33:31.514374+0000","last_clean":"2024-04-17T10:33:31.514374+0000","last_became_active":"2024-04-17T10:33:29.485542+0000","last_became_peered":"2024-04-17T10:33:29.485542+0000","last_unstale":"2024-04-17T10:33:31.514374+0000","last_undegraded":"2024-04-17T10:33:31.514374+0000","last_fullsized":"2024-04-17T10:33:31.514374+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:29.843042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514655+0000","last_change":"2024-04-17T10:33:30.513626+0000","last_active":"2024-04-17T10:33:31.514655+0000","last_peered":"2024-04-17T10:33:31.514655+0000","last_clean":"2024-04-17T10:33:31.514655+0000","last_became_active":"2024-04-17T10:33:30.513332+0000","last_became_peered":"2024-04-17T10:33:30.513332+0000","last_unstale":"2024-04-17T10:33:31.514655+0000","last_undegraded":"2024-04-17T10:33:31.514655+0000","last_fullsized":"2024-04-17T10:33:31.514655+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:48.741482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"117'6","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612840+0000","last_change":"2024-04-17T10:33:29.490370+0000","last_active":"2024-04-17T10:33:31.612840+0000","last_peered":"2024-04-17T10:33:31.612840+0000","last_clean":"2024-04-17T10:33:31.612840+0000","last_became_active":"2024-04-17T10:33:29.490109+0000","last_became_peered":"2024-04-17T10:33:29.490109+0000","last_unstale":"2024-04-17T10:33:31.612840+0000","last_undegraded":"2024-04-17T10:33:31.612840+0000","last_fullsized":"2024-04-17T10:33:31.612840+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:20.586624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524468+0000","last_change":"2024-04-17T10:33:30.523893+0000","last_active":"2024-04-17T10:33:31.524468+0000","last_peered":"2024-04-17T10:33:31.524468+0000","last_clean":"2024-04-17T10:33:31.524468+0000","last_became_active":"2024-04-17T10:33:30.523553+0000","last_became_peered":"2024-04-17T10:33:30.523553+0000","last_unstale":"2024-04-17T10:33:31.524468+0000","last_undegraded":"2024-04-17T10:33:31.524468+0000","last_fullsized":"2024-04-17T10:33:31.524468+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:46.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612819+0000","last_change":"2024-04-17T10:33:29.494255+0000","last_active":"2024-04-17T10:33:31.612819+0000","last_peered":"2024-04-17T10:33:31.612819+0000","last_clean":"2024-04-17T10:33:31.612819+0000","last_became_active":"2024-04-17T10:33:29.493978+0000","last_became_peered":"2024-04-17T10:33:29.493978+0000","last_unstale":"2024-04-17T10:33:31.612819+0000","last_undegraded":"2024-04-17T10:33:31.612819+0000","last_fullsized":"2024-04-17T10:33:31.612819+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:20.843749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612365+0000","last_change":"2024-04-17T10:33:30.530914+0000","last_active":"2024-04-17T10:33:31.612365+0000","last_peered":"2024-04-17T10:33:31.612365+0000","last_clean":"2024-04-17T10:33:31.612365+0000","last_became_active":"2024-04-17T10:33:30.530657+0000","last_became_peered":"2024-04-17T10:33:30.530657+0000","last_unstale":"2024-04-17T10:33:31.612365+0000","last_undegraded":"2024-04-17T10:33:31.612365+0000","last_fullsized":"2024-04-17T10:33:31.612365+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:44.795215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613137+0000","last_change":"2024-04-17T10:33:29.492252+0000","last_active":"2024-04-17T10:33:31.613137+0000","last_peered":"2024-04-17T10:33:31.613137+0000","last_clean":"2024-04-17T10:33:31.613137+0000","last_became_active":"2024-04-17T10:33:29.490092+0000","last_became_peered":"2024-04-17T10:33:29.490092+0000","last_unstale":"2024-04-17T10:33:31.613137+0000","last_undegraded":"2024-04-17T10:33:31.613137+0000","last_fullsized":"2024-04-17T10:33:31.613137+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:46.582002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518342+0000","last_change":"2024-04-17T10:33:30.513713+0000","last_active":"2024-04-17T10:33:31.518342+0000","last_peered":"2024-04-17T10:33:31.518342+0000","last_clean":"2024-04-17T10:33:31.518342+0000","last_became_active":"2024-04-17T10:33:30.513439+0000","last_became_peered":"2024-04-17T10:33:30.513439+0000","last_unstale":"2024-04-17T10:33:31.518342+0000","last_undegraded":"2024-04-17T10:33:31.518342+0000","last_fullsized":"2024-04-17T10:33:31.518342+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:40:13.007807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613130+0000","last_change":"2024-04-17T10:33:29.483114+0000","last_active":"2024-04-17T10:33:31.613130+0000","last_peered":"2024-04-17T10:33:31.613130+0000","last_clean":"2024-04-17T10:33:31.613130+0000","last_became_active":"2024-04-17T10:33:29.482291+0000","last_became_peered":"2024-04-17T10:33:29.482291+0000","last_unstale":"2024-04-17T10:33:31.613130+0000","last_undegraded":"2024-04-17T10:33:31.613130+0000","last_fullsized":"2024-04-17T10:33:31.613130+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:49.924443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514076+0000","last_change":"2024-04-17T10:33:30.521974+0000","last_active":"2024-04-17T10:33:31.514076+0000","last_peered":"2024-04-17T10:33:31.514076+0000","last_clean":"2024-04-17T10:33:31.514076+0000","last_became_active":"2024-04-17T10:33:30.521654+0000","last_became_peered":"2024-04-17T10:33:30.521654+0000","last_unstale":"2024-04-17T10:33:31.514076+0000","last_undegraded":"2024-04-17T10:33:31.514076+0000","last_fullsized":"2024-04-17T10:33:31.514076+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:31.627592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613188+0000","last_change":"2024-04-17T10:33:29.498174+0000","last_active":"2024-04-17T10:33:31.613188+0000","last_peered":"2024-04-17T10:33:31.613188+0000","last_clean":"2024-04-17T10:33:31.613188+0000","last_became_active":"2024-04-17T10:33:29.497727+0000","last_became_peered":"2024-04-17T10:33:29.497727+0000","last_unstale":"2024-04-17T10:33:31.613188+0000","last_undegraded":"2024-04-17T10:33:31.613188+0000","last_fullsized":"2024-04-17T10:33:31.613188+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:46:54.161898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514477+0000","last_change":"2024-04-17T10:33:30.521113+0000","last_active":"2024-04-17T10:33:31.514477+0000","last_peered":"2024-04-17T10:33:31.514477+0000","last_clean":"2024-04-17T10:33:31.514477+0000","last_became_active":"2024-04-17T10:33:30.520808+0000","last_became_peered":"2024-04-17T10:33:30.520808+0000","last_unstale":"2024-04-17T10:33:31.514477+0000","last_undegraded":"2024-04-17T10:33:31.514477+0000","last_fullsized":"2024-04-17T10:33:31.514477+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:21.596678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518723+0000","last_change":"2024-04-17T10:33:29.488532+0000","last_active":"2024-04-17T10:33:31.518723+0000","last_peered":"2024-04-17T10:33:31.518723+0000","last_clean":"2024-04-17T10:33:31.518723+0000","last_became_active":"2024-04-17T10:33:29.488234+0000","last_became_peered":"2024-04-17T10:33:29.488234+0000","last_unstale":"2024-04-17T10:33:31.518723+0000","last_undegraded":"2024-04-17T10:33:31.518723+0000","last_fullsized":"2024-04-17T10:33:31.518723+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:32:10.059619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530257+0000","last_change":"2024-04-17T10:33:30.515255+0000","last_active":"2024-04-17T10:33:31.530257+0000","last_peered":"2024-04-17T10:33:31.530257+0000","last_clean":"2024-04-17T10:33:31.530257+0000","last_became_active":"2024-04-17T10:33:30.512245+0000","last_became_peered":"2024-04-17T10:33:30.512245+0000","last_unstale":"2024-04-17T10:33:31.530257+0000","last_undegraded":"2024-04-17T10:33:31.530257+0000","last_fullsized":"2024-04-17T10:33:31.530257+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:27.370635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609941+0000","last_change":"2024-04-17T10:33:29.488911+0000","last_active":"2024-04-17T10:33:31.609941+0000","last_peered":"2024-04-17T10:33:31.609941+0000","last_clean":"2024-04-17T10:33:31.609941+0000","last_became_active":"2024-04-17T10:33:29.488581+0000","last_became_peered":"2024-04-17T10:33:29.488581+0000","last_unstale":"2024-04-17T10:33:31.609941+0000","last_undegraded":"2024-04-17T10:33:31.609941+0000","last_fullsized":"2024-04-17T10:33:31.609941+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:53:13.663196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514723+0000","last_change":"2024-04-17T10:33:30.529062+0000","last_active":"2024-04-17T10:33:31.514723+0000","last_peered":"2024-04-17T10:33:31.514723+0000","last_clean":"2024-04-17T10:33:31.514723+0000","last_became_active":"2024-04-17T10:33:30.528793+0000","last_became_peered":"2024-04-17T10:33:30.528793+0000","last_unstale":"2024-04-17T10:33:31.514723+0000","last_undegraded":"2024-04-17T10:33:31.514723+0000","last_fullsized":"2024-04-17T10:33:31.514723+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:38.216709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609838+0000","last_change":"2024-04-17T10:33:29.495528+0000","last_active":"2024-04-17T10:33:31.609838+0000","last_peered":"2024-04-17T10:33:31.609838+0000","last_clean":"2024-04-17T10:33:31.609838+0000","last_became_active":"2024-04-17T10:33:29.495252+0000","last_became_peered":"2024-04-17T10:33:29.495252+0000","last_unstale":"2024-04-17T10:33:31.609838+0000","last_undegraded":"2024-04-17T10:33:31.609838+0000","last_fullsized":"2024-04-17T10:33:31.609838+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:46.582002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529784+0000","last_change":"2024-04-17T10:33:30.513432+0000","last_active":"2024-04-17T10:33:31.529784+0000","last_peered":"2024-04-17T10:33:31.529784+0000","last_clean":"2024-04-17T10:33:31.529784+0000","last_became_active":"2024-04-17T10:33:30.513088+0000","last_became_peered":"2024-04-17T10:33:30.513088+0000","last_unstale":"2024-04-17T10:33:31.529784+0000","last_undegraded":"2024-04-17T10:33:31.529784+0000","last_fullsized":"2024-04-17T10:33:31.529784+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:18:58.424993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529786+0000","last_change":"2024-04-17T10:33:29.495116+0000","last_active":"2024-04-17T10:33:31.529786+0000","last_peered":"2024-04-17T10:33:31.529786+0000","last_clean":"2024-04-17T10:33:31.529786+0000","last_became_active":"2024-04-17T10:33:29.494706+0000","last_became_peered":"2024-04-17T10:33:29.494706+0000","last_unstale":"2024-04-17T10:33:31.529786+0000","last_undegraded":"2024-04-17T10:33:31.529786+0000","last_fullsized":"2024-04-17T10:33:31.529786+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:01:53.711266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612988+0000","last_change":"2024-04-17T10:33:30.515873+0000","last_active":"2024-04-17T10:33:31.612988+0000","last_peered":"2024-04-17T10:33:31.612988+0000","last_clean":"2024-04-17T10:33:31.612988+0000","last_became_active":"2024-04-17T10:33:30.515500+0000","last_became_peered":"2024-04-17T10:33:30.515500+0000","last_unstale":"2024-04-17T10:33:31.612988+0000","last_undegraded":"2024-04-17T10:33:31.612988+0000","last_fullsized":"2024-04-17T10:33:31.612988+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:57.278375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609422+0000","last_change":"2024-04-17T10:33:29.488855+0000","last_active":"2024-04-17T10:33:31.609422+0000","last_peered":"2024-04-17T10:33:31.609422+0000","last_clean":"2024-04-17T10:33:31.609422+0000","last_became_active":"2024-04-17T10:33:29.488538+0000","last_became_peered":"2024-04-17T10:33:29.488538+0000","last_unstale":"2024-04-17T10:33:31.609422+0000","last_undegraded":"2024-04-17T10:33:31.609422+0000","last_fullsized":"2024-04-17T10:33:31.609422+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:54.326552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613168+0000","last_change":"2024-04-17T10:33:30.513047+0000","last_active":"2024-04-17T10:33:31.613168+0000","last_peered":"2024-04-17T10:33:31.613168+0000","last_clean":"2024-04-17T10:33:31.613168+0000","last_became_active":"2024-04-17T10:33:30.512503+0000","last_became_peered":"2024-04-17T10:33:30.512503+0000","last_unstale":"2024-04-17T10:33:31.613168+0000","last_undegraded":"2024-04-17T10:33:31.613168+0000","last_fullsized":"2024-04-17T10:33:31.613168+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:36:59.103325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613242+0000","last_change":"2024-04-17T10:33:29.493323+0000","last_active":"2024-04-17T10:33:31.613242+0000","last_peered":"2024-04-17T10:33:31.613242+0000","last_clean":"2024-04-17T10:33:31.613242+0000","last_became_active":"2024-04-17T10:33:29.491574+0000","last_became_peered":"2024-04-17T10:33:29.491574+0000","last_unstale":"2024-04-17T10:33:31.613242+0000","last_undegraded":"2024-04-17T10:33:31.613242+0000","last_fullsized":"2024-04-17T10:33:31.613242+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:54.326552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513932+0000","last_change":"2024-04-17T10:33:30.516151+0000","last_active":"2024-04-17T10:33:31.513932+0000","last_peered":"2024-04-17T10:33:31.513932+0000","last_clean":"2024-04-17T10:33:31.513932+0000","last_became_active":"2024-04-17T10:33:30.515877+0000","last_became_peered":"2024-04-17T10:33:30.515877+0000","last_unstale":"2024-04-17T10:33:31.513932+0000","last_undegraded":"2024-04-17T10:33:31.513932+0000","last_fullsized":"2024-04-17T10:33:31.513932+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:14:41.699457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609262+0000","last_change":"2024-04-17T10:33:29.491280+0000","last_active":"2024-04-17T10:33:31.609262+0000","last_peered":"2024-04-17T10:33:31.609262+0000","last_clean":"2024-04-17T10:33:31.609262+0000","last_became_active":"2024-04-17T10:33:29.490947+0000","last_became_peered":"2024-04-17T10:33:29.490947+0000","last_unstale":"2024-04-17T10:33:31.609262+0000","last_undegraded":"2024-04-17T10:33:31.609262+0000","last_fullsized":"2024-04-17T10:33:31.609262+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:20.843749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530304+0000","last_change":"2024-04-17T10:33:30.524459+0000","last_active":"2024-04-17T10:33:31.530304+0000","last_peered":"2024-04-17T10:33:31.530304+0000","last_clean":"2024-04-17T10:33:31.530304+0000","last_became_active":"2024-04-17T10:33:30.524159+0000","last_became_peered":"2024-04-17T10:33:30.524159+0000","last_unstale":"2024-04-17T10:33:31.530304+0000","last_undegraded":"2024-04-17T10:33:31.530304+0000","last_fullsized":"2024-04-17T10:33:31.530304+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:17.625384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530226+0000","last_change":"2024-04-17T10:33:29.492646+0000","last_active":"2024-04-17T10:33:31.530226+0000","last_peered":"2024-04-17T10:33:31.530226+0000","last_clean":"2024-04-17T10:33:31.530226+0000","last_became_active":"2024-04-17T10:33:29.492187+0000","last_became_peered":"2024-04-17T10:33:29.492187+0000","last_unstale":"2024-04-17T10:33:31.530226+0000","last_undegraded":"2024-04-17T10:33:31.530226+0000","last_fullsized":"2024-04-17T10:33:31.530226+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:46.098736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612568+0000","last_change":"2024-04-17T10:33:30.519616+0000","last_active":"2024-04-17T10:33:31.612568+0000","last_peered":"2024-04-17T10:33:31.612568+0000","last_clean":"2024-04-17T10:33:31.612568+0000","last_became_active":"2024-04-17T10:33:30.519334+0000","last_became_peered":"2024-04-17T10:33:30.519334+0000","last_unstale":"2024-04-17T10:33:31.612568+0000","last_undegraded":"2024-04-17T10:33:31.612568+0000","last_fullsized":"2024-04-17T10:33:31.612568+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:21:04.234855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"118'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.526385+0000","last_change":"2024-04-17T10:33:29.487245+0000","last_active":"2024-04-17T10:33:31.526385+0000","last_peered":"2024-04-17T10:33:31.526385+0000","last_clean":"2024-04-17T10:33:31.526385+0000","last_became_active":"2024-04-17T10:33:29.486874+0000","last_became_peered":"2024-04-17T10:33:29.486874+0000","last_unstale":"2024-04-17T10:33:31.526385+0000","last_undegraded":"2024-04-17T10:33:31.526385+0000","last_fullsized":"2024-04-17T10:33:31.526385+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:13.128031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609940+0000","last_change":"2024-04-17T10:33:30.522497+0000","last_active":"2024-04-17T10:33:31.609940+0000","last_peered":"2024-04-17T10:33:31.609940+0000","last_clean":"2024-04-17T10:33:31.609940+0000","last_became_active":"2024-04-17T10:33:30.522138+0000","last_became_peered":"2024-04-17T10:33:30.522138+0000","last_unstale":"2024-04-17T10:33:31.609940+0000","last_undegraded":"2024-04-17T10:33:31.609940+0000","last_fullsized":"2024-04-17T10:33:31.609940+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:23.766536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524704+0000","last_change":"2024-04-17T10:33:29.485210+0000","last_active":"2024-04-17T10:33:31.524704+0000","last_peered":"2024-04-17T10:33:31.524704+0000","last_clean":"2024-04-17T10:33:31.524704+0000","last_became_active":"2024-04-17T10:33:29.484745+0000","last_became_peered":"2024-04-17T10:33:29.484745+0000","last_unstale":"2024-04-17T10:33:31.524704+0000","last_undegraded":"2024-04-17T10:33:31.524704+0000","last_fullsized":"2024-04-17T10:33:31.524704+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:56:56.669526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518835+0000","last_change":"2024-04-17T10:33:30.516350+0000","last_active":"2024-04-17T10:33:31.518835+0000","last_peered":"2024-04-17T10:33:31.518835+0000","last_clean":"2024-04-17T10:33:31.518835+0000","last_became_active":"2024-04-17T10:33:30.516089+0000","last_became_peered":"2024-04-17T10:33:30.516089+0000","last_unstale":"2024-04-17T10:33:31.518835+0000","last_undegraded":"2024-04-17T10:33:31.518835+0000","last_fullsized":"2024-04-17T10:33:31.518835+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:07:53.969041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530310+0000","last_change":"2024-04-17T10:33:29.497772+0000","last_active":"2024-04-17T10:33:31.530310+0000","last_peered":"2024-04-17T10:33:31.530310+0000","last_clean":"2024-04-17T10:33:31.530310+0000","last_became_active":"2024-04-17T10:33:29.496779+0000","last_became_peered":"2024-04-17T10:33:29.496779+0000","last_unstale":"2024-04-17T10:33:31.530310+0000","last_undegraded":"2024-04-17T10:33:31.530310+0000","last_fullsized":"2024-04-17T10:33:31.530310+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:50.156210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529878+0000","last_change":"2024-04-17T10:33:30.506486+0000","last_active":"2024-04-17T10:33:31.529878+0000","last_peered":"2024-04-17T10:33:31.529878+0000","last_clean":"2024-04-17T10:33:31.529878+0000","last_became_active":"2024-04-17T10:33:30.506241+0000","last_became_peered":"2024-04-17T10:33:30.506241+0000","last_unstale":"2024-04-17T10:33:31.529878+0000","last_undegraded":"2024-04-17T10:33:31.529878+0000","last_fullsized":"2024-04-17T10:33:31.529878+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:02.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514764+0000","last_change":"2024-04-17T10:33:29.481339+0000","last_active":"2024-04-17T10:33:31.514764+0000","last_peered":"2024-04-17T10:33:31.514764+0000","last_clean":"2024-04-17T10:33:31.514764+0000","last_became_active":"2024-04-17T10:33:29.480897+0000","last_became_peered":"2024-04-17T10:33:29.480897+0000","last_unstale":"2024-04-17T10:33:31.514764+0000","last_undegraded":"2024-04-17T10:33:31.514764+0000","last_fullsized":"2024-04-17T10:33:31.514764+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:03.481333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613147+0000","last_change":"2024-04-17T10:33:30.512815+0000","last_active":"2024-04-17T10:33:31.613147+0000","last_peered":"2024-04-17T10:33:31.613147+0000","last_clean":"2024-04-17T10:33:31.613147+0000","last_became_active":"2024-04-17T10:33:30.512189+0000","last_became_peered":"2024-04-17T10:33:30.512189+0000","last_unstale":"2024-04-17T10:33:31.613147+0000","last_undegraded":"2024-04-17T10:33:31.613147+0000","last_fullsized":"2024-04-17T10:33:31.613147+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:31.393021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514174+0000","last_change":"2024-04-17T10:33:29.476128+0000","last_active":"2024-04-17T10:33:31.514174+0000","last_peered":"2024-04-17T10:33:31.514174+0000","last_clean":"2024-04-17T10:33:31.514174+0000","last_became_active":"2024-04-17T10:33:29.475862+0000","last_became_peered":"2024-04-17T10:33:29.475862+0000","last_unstale":"2024-04-17T10:33:31.514174+0000","last_undegraded":"2024-04-17T10:33:31.514174+0000","last_fullsized":"2024-04-17T10:33:31.514174+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:14:08.966259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609536+0000","last_change":"2024-04-17T10:33:30.526594+0000","last_active":"2024-04-17T10:33:31.609536+0000","last_peered":"2024-04-17T10:33:31.609536+0000","last_clean":"2024-04-17T10:33:31.609536+0000","last_became_active":"2024-04-17T10:33:30.526312+0000","last_became_peered":"2024-04-17T10:33:30.526312+0000","last_unstale":"2024-04-17T10:33:31.609536+0000","last_undegraded":"2024-04-17T10:33:31.609536+0000","last_fullsized":"2024-04-17T10:33:31.609536+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:13.368923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529695+0000","last_change":"2024-04-17T10:33:29.488144+0000","last_active":"2024-04-17T10:33:31.529695+0000","last_peered":"2024-04-17T10:33:31.529695+0000","last_clean":"2024-04-17T10:33:31.529695+0000","last_became_active":"2024-04-17T10:33:29.487882+0000","last_became_peered":"2024-04-17T10:33:29.487882+0000","last_unstale":"2024-04-17T10:33:31.529695+0000","last_undegraded":"2024-04-17T10:33:31.529695+0000","last_fullsized":"2024-04-17T10:33:31.529695+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:32:53.424723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612624+0000","last_change":"2024-04-17T10:33:30.517444+0000","last_active":"2024-04-17T10:33:31.612624+0000","last_peered":"2024-04-17T10:33:31.612624+0000","last_clean":"2024-04-17T10:33:31.612624+0000","last_became_active":"2024-04-17T10:33:30.515775+0000","last_became_peered":"2024-04-17T10:33:30.515775+0000","last_unstale":"2024-04-17T10:33:31.612624+0000","last_undegraded":"2024-04-17T10:33:31.612624+0000","last_fullsized":"2024-04-17T10:33:31.612624+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:07.313743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609214+0000","last_change":"2024-04-17T10:33:29.491041+0000","last_active":"2024-04-17T10:33:31.609214+0000","last_peered":"2024-04-17T10:33:31.609214+0000","last_clean":"2024-04-17T10:33:31.609214+0000","last_became_active":"2024-04-17T10:33:29.490574+0000","last_became_peered":"2024-04-17T10:33:29.490574+0000","last_unstale":"2024-04-17T10:33:31.609214+0000","last_undegraded":"2024-04-17T10:33:31.609214+0000","last_fullsized":"2024-04-17T10:33:31.609214+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:20.586624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609837+0000","last_change":"2024-04-17T10:33:30.527050+0000","last_active":"2024-04-17T10:33:31.609837+0000","last_peered":"2024-04-17T10:33:31.609837+0000","last_clean":"2024-04-17T10:33:31.609837+0000","last_became_active":"2024-04-17T10:33:30.526471+0000","last_became_peered":"2024-04-17T10:33:30.526471+0000","last_unstale":"2024-04-17T10:33:31.609837+0000","last_undegraded":"2024-04-17T10:33:31.609837+0000","last_fullsized":"2024-04-17T10:33:31.609837+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.599913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"117'8","reported_seq":22,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612682+0000","last_change":"2024-04-17T10:33:29.493929+0000","last_active":"2024-04-17T10:33:31.612682+0000","last_peered":"2024-04-17T10:33:31.612682+0000","last_clean":"2024-04-17T10:33:31.612682+0000","last_became_active":"2024-04-17T10:33:29.492762+0000","last_became_peered":"2024-04-17T10:33:29.492762+0000","last_unstale":"2024-04-17T10:33:31.612682+0000","last_undegraded":"2024-04-17T10:33:31.612682+0000","last_fullsized":"2024-04-17T10:33:31.612682+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:01.006670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":350,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609339+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:33:31.609339+0000","last_peered":"2024-04-17T10:33:31.609339+0000","last_clean":"2024-04-17T10:33:31.609339+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:33:31.609339+0000","last_undegraded":"2024-04-17T10:33:31.609339+0000","last_fullsized":"2024-04-17T10:33:31.609339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574958,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51260,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320580,"statfs":{"total":96636764160,"available":96584273920,"internally_reserved":0,"allocated":2080768,"data_stored":1278256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50676,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321164,"statfs":{"total":96636764160,"available":96584871936,"internally_reserved":0,"allocated":1474560,"data_stored":687694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46516,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325324,"statfs":{"total":96636764160,"available":96589131776,"internally_reserved":0,"allocated":1478656,"data_stored":687272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574957,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58984,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312856,"statfs":{"total":96636764160,"available":96576364544,"internally_reserved":0,"allocated":1470464,"data_stored":682032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54828,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317012,"statfs":{"total":96636764160,"available":96580620288,"internally_reserved":0,"allocated":1474560,"data_stored":682066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54824,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317016,"statfs":{"total":96636764160,"available":96580624384,"internally_reserved":0,"allocated":1470464,"data_stored":687112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50656,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1462272,"data_stored":680442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574957,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55412,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316428,"statfs":{"total":96636764160,"available":96580022272,"internally_reserved":0,"allocated":2068480,"data_stored":1277502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:33:34.615 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:33:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.801+0000 7f9720c00640 1 Processor -- start 2024-04-17T10:33:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.801+0000 7f9720c00640 1 -- start start 2024-04-17T10:33:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9720c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f971c0cc250 0x7f971c0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9720c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f971c0cb260 0x7f971c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9720c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0ce2a0 0x7f971c0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9720c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971c0cd150 con 0x7f971c0cb260 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9720c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971c06b100 con 0x7f971c0cc250 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9720c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971c06b3e0 con 0x7f971c0ce2a0 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f971c0cb260 0x7f971c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f971c0cb260 0x7f971c0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42452/0 (socket says 172.21.15.67:42452) 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 -- 172.21.15.67:0/3885559562 learned_addr learned my addr 172.21.15.67:0/3885559562 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f971a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0ce2a0 0x7f971c0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9719e00640 1 --2- 172.21.15.67:0/3885559562 >> v2:172.21.15.86:3300/0 conn(0x7f971c0cc250 0x7f971c0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 -- 172.21.15.67:0/3885559562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0ce2a0 msgr2=0x7f971c0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 --2- 172.21.15.67:0/3885559562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0ce2a0 0x7f971c0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 -- 172.21.15.67:0/3885559562 >> v2:172.21.15.86:3300/0 conn(0x7f971c0cc250 msgr2=0x7f971c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 --2- 172.21.15.67:0/3885559562 >> v2:172.21.15.86:3300/0 conn(0x7f971c0cc250 0x7f971c0cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.805+0000 7f9711400640 1 -- 172.21.15.67:0/3885559562 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c06b6c0 con 0x7f971c0cb260 2024-04-17T10:33:34.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9711400640 1 --2- 172.21.15.67:0/3885559562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0cb260 0x7f971c0cd5e0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f971c068850 tx=0x7f970401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=627b507311d1c781 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9719400640 1 -- 172.21.15.67:0/3885559562 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97040b0070 con 0x7f971c0cb260 2024-04-17T10:33:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9719400640 1 -- 172.21.15.67:0/3885559562 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97040ab600 con 0x7f971c0cb260 2024-04-17T10:33:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9719400640 1 -- 172.21.15.67:0/3885559562 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97040ab9d0 con 0x7f971c0cb260 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 -- 172.21.15.67:0/3885559562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0cb260 msgr2=0x7f971c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 --2- 172.21.15.67:0/3885559562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0cb260 0x7f971c0cd5e0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f971c068850 tx=0x7f970401c520 comp rx=0 tx=0).stop 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 -- 172.21.15.67:0/3885559562 shutdown_connections 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 --2- 172.21.15.67:0/3885559562 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0ce2a0 0x7f971c0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 --2- 172.21.15.67:0/3885559562 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0cb260 0x7f971c0cd5e0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 --2- 172.21.15.67:0/3885559562 >> v2:172.21.15.86:3300/0 conn(0x7f971c0cc250 0x7f971c0cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 -- 172.21.15.67:0/3885559562 >> 172.21.15.67:0/3885559562 conn(0x7f971c0b22a0 msgr2=0x7f971c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 -- 172.21.15.67:0/3885559562 shutdown_connections 2024-04-17T10:33:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 -- 172.21.15.67:0/3885559562 wait complete. 2024-04-17T10:33:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 Processor -- start 2024-04-17T10:33:34.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.809+0000 7f9720c00640 1 -- start start 2024-04-17T10:33:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971c0cb260 0x7f971c15dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0cc250 0x7f971c14ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0ce2a0 0x7f971c15a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c06e850 con 0x7f971c0ce2a0 2024-04-17T10:33:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c06e6d0 con 0x7f971c0cb260 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971c06e9d0 con 0x7f971c0cc250 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9719e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971c0cb260 0x7f971c15dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f971a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0ce2a0 0x7f971c15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9711400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0cc250 0x7f971c14ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9719e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971c0cb260 0x7f971c15dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:47458/0 (socket says 172.21.15.67:47458) 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9719e00640 1 -- 172.21.15.67:0/1689508375 learned_addr learned my addr 172.21.15.67:0/1689508375 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f971a800640 1 -- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0cc250 msgr2=0x7f971c14ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f971a800640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0cc250 0x7f971c14ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f971a800640 1 -- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971c0cb260 msgr2=0x7f971c15dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f971a800640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971c0cb260 0x7f971c15dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f971a800640 1 -- 172.21.15.67:0/1689508375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c14f3a0 con 0x7f971c0ce2a0 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9711400640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0cc250 0x7f971c14ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f971a800640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0ce2a0 0x7f971c15a210 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f971c0ce6b0 tx=0x7f971400c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9712a00640 1 -- 172.21.15.67:0/1689508375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714018020 con 0x7f971c0ce2a0 2024-04-17T10:33:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9712a00640 1 -- 172.21.15.67:0/1689508375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971400cc20 con 0x7f971c0ce2a0 2024-04-17T10:33:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9712a00640 1 -- 172.21.15.67:0/1689508375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714014480 con 0x7f971c0ce2a0 2024-04-17T10:33:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c0c98a0 con 0x7f971c0ce2a0 2024-04-17T10:33:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c0c9e60 con 0x7f971c0ce2a0 2024-04-17T10:33:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971c06cd30 con 0x7f971c0ce2a0 2024-04-17T10:33:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.813+0000 7f9712a00640 1 -- 172.21.15.67:0/1689508375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f971401b020 con 0x7f971c0ce2a0 2024-04-17T10:33:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.817+0000 7f9712a00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96f8066f60 0x7f96f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.817+0000 7f9712a00640 1 -- 172.21.15.67:0/1689508375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9714085c80 con 0x7f971c0ce2a0 2024-04-17T10:33:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.817+0000 7f9719e00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96f8066f60 0x7f96f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.817+0000 7f9719e00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96f8066f60 0x7f96f8069420 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f970c009ba0 tx=0x7f970c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:34.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.821+0000 7f9712a00640 1 -- 172.21.15.67:0/1689508375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9714022030 con 0x7f971c0ce2a0 2024-04-17T10:33:34.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.977+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f971c0cafe0 con 0x7f96f8066f60 2024-04-17T10:33:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.985+0000 7f9712a00640 1 -- 172.21.15.67:0/1689508375 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252486 (secure 0 0 0) 0x7f971c0cafe0 con 0x7f96f8066f60 2024-04-17T10:33:34.979 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:34.980 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96f8066f60 msgr2=0x7f96f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96f8066f60 0x7f96f8069420 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f970c009ba0 tx=0x7f970c009190 comp rx=0 tx=0).stop 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0ce2a0 msgr2=0x7f971c15a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0ce2a0 0x7f971c15a210 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f971c0ce6b0 tx=0x7f971400c0c0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 shutdown_connections 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f96f8066f60 0x7f96f8069420 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f971c0ce2a0 0x7f971c15a210 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f971c0cc250 0x7f971c14ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 --2- 172.21.15.67:0/1689508375 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f971c0cb260 0x7f971c15dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 >> 172.21.15.67:0/1689508375 conn(0x7f971c0b22a0 msgr2=0x7f971c1549a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 shutdown_connections 2024-04-17T10:33:34.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:34.989+0000 7f9720c00640 1 -- 172.21.15.67:0/1689508375 wait complete. 2024-04-17T10:33:35.000 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-04-17T10:33:34.260365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423156,"kb_used_data":12676,"kb_used_omap":12,"kb_used_meta":410355,"kb_avail":754551564,"statfs":{"total":773094113280,"available":772660801536,"internally_reserved":0,"allocated":12980224,"data_stored":6662376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":420204081},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.794602"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514773+0000","last_change":"2024-04-17T10:33:30.517880+0000","last_active":"2024-04-17T10:33:31.514773+0000","last_peered":"2024-04-17T10:33:31.514773+0000","last_clean":"2024-04-17T10:33:31.514773+0000","last_became_active":"2024-04-17T10:33:30.517617+0000","last_became_peered":"2024-04-17T10:33:30.517617+0000","last_unstale":"2024-04-17T10:33:31.514773+0000","last_undegraded":"2024-04-17T10:33:31.514773+0000","last_fullsized":"2024-04-17T10:33:31.514773+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:55.336606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613281+0000","last_change":"2024-04-17T10:33:30.514070+0000","last_active":"2024-04-17T10:33:31.613281+0000","last_peered":"2024-04-17T10:33:31.613281+0000","last_clean":"2024-04-17T10:33:31.613281+0000","last_became_active":"2024-04-17T10:33:30.513292+0000","last_became_peered":"2024-04-17T10:33:30.513292+0000","last_unstale":"2024-04-17T10:33:31.613281+0000","last_undegraded":"2024-04-17T10:33:31.613281+0000","last_fullsized":"2024-04-17T10:33:31.613281+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:51.485725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613129+0000","last_change":"2024-04-17T10:33:30.519522+0000","last_active":"2024-04-17T10:33:31.613129+0000","last_peered":"2024-04-17T10:33:31.613129+0000","last_clean":"2024-04-17T10:33:31.613129+0000","last_became_active":"2024-04-17T10:33:30.519240+0000","last_became_peered":"2024-04-17T10:33:30.519240+0000","last_unstale":"2024-04-17T10:33:31.613129+0000","last_undegraded":"2024-04-17T10:33:31.613129+0000","last_fullsized":"2024-04-17T10:33:31.613129+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:58.535058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609739+0000","last_change":"2024-04-17T10:33:30.524405+0000","last_active":"2024-04-17T10:33:31.609739+0000","last_peered":"2024-04-17T10:33:31.609739+0000","last_clean":"2024-04-17T10:33:31.609739+0000","last_became_active":"2024-04-17T10:33:30.524051+0000","last_became_peered":"2024-04-17T10:33:30.524051+0000","last_unstale":"2024-04-17T10:33:31.609739+0000","last_undegraded":"2024-04-17T10:33:31.609739+0000","last_fullsized":"2024-04-17T10:33:31.609739+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:48.741482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514609+0000","last_change":"2024-04-17T10:33:30.515073+0000","last_active":"2024-04-17T10:33:31.514609+0000","last_peered":"2024-04-17T10:33:31.514609+0000","last_clean":"2024-04-17T10:33:31.514609+0000","last_became_active":"2024-04-17T10:33:30.514021+0000","last_became_peered":"2024-04-17T10:33:30.514021+0000","last_unstale":"2024-04-17T10:33:31.514609+0000","last_undegraded":"2024-04-17T10:33:31.514609+0000","last_fullsized":"2024-04-17T10:33:31.514609+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:47.592056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529806+0000","last_change":"2024-04-17T10:33:30.525132+0000","last_active":"2024-04-17T10:33:31.529806+0000","last_peered":"2024-04-17T10:33:31.529806+0000","last_clean":"2024-04-17T10:33:31.529806+0000","last_became_active":"2024-04-17T10:33:30.524815+0000","last_became_peered":"2024-04-17T10:33:30.524815+0000","last_unstale":"2024-04-17T10:33:31.529806+0000","last_undegraded":"2024-04-17T10:33:31.529806+0000","last_fullsized":"2024-04-17T10:33:31.529806+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:57:54.256415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529723+0000","last_change":"2024-04-17T10:33:30.528241+0000","last_active":"2024-04-17T10:33:31.529723+0000","last_peered":"2024-04-17T10:33:31.529723+0000","last_clean":"2024-04-17T10:33:31.529723+0000","last_became_active":"2024-04-17T10:33:30.527987+0000","last_became_peered":"2024-04-17T10:33:30.527987+0000","last_unstale":"2024-04-17T10:33:31.529723+0000","last_undegraded":"2024-04-17T10:33:31.529723+0000","last_fullsized":"2024-04-17T10:33:31.529723+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:53.733211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612560+0000","last_change":"2024-04-17T10:33:30.520211+0000","last_active":"2024-04-17T10:33:31.612560+0000","last_peered":"2024-04-17T10:33:31.612560+0000","last_clean":"2024-04-17T10:33:31.612560+0000","last_became_active":"2024-04-17T10:33:30.519711+0000","last_became_peered":"2024-04-17T10:33:30.519711+0000","last_unstale":"2024-04-17T10:33:31.612560+0000","last_undegraded":"2024-04-17T10:33:31.612560+0000","last_fullsized":"2024-04-17T10:33:31.612560+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:38.216709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.610095+0000","last_change":"2024-04-17T10:33:30.524736+0000","last_active":"2024-04-17T10:33:31.610095+0000","last_peered":"2024-04-17T10:33:31.610095+0000","last_clean":"2024-04-17T10:33:31.610095+0000","last_became_active":"2024-04-17T10:33:30.523635+0000","last_became_peered":"2024-04-17T10:33:30.523635+0000","last_unstale":"2024-04-17T10:33:31.610095+0000","last_undegraded":"2024-04-17T10:33:31.610095+0000","last_fullsized":"2024-04-17T10:33:31.610095+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:53.545591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613011+0000","last_change":"2024-04-17T10:33:30.519167+0000","last_active":"2024-04-17T10:33:31.613011+0000","last_peered":"2024-04-17T10:33:31.613011+0000","last_clean":"2024-04-17T10:33:31.613011+0000","last_became_active":"2024-04-17T10:33:30.518906+0000","last_became_peered":"2024-04-17T10:33:30.518906+0000","last_unstale":"2024-04-17T10:33:31.613011+0000","last_undegraded":"2024-04-17T10:33:31.613011+0000","last_fullsized":"2024-04-17T10:33:31.613011+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:12.614426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514322+0000","last_change":"2024-04-17T10:33:30.513613+0000","last_active":"2024-04-17T10:33:31.514322+0000","last_peered":"2024-04-17T10:33:31.514322+0000","last_clean":"2024-04-17T10:33:31.514322+0000","last_became_active":"2024-04-17T10:33:30.513312+0000","last_became_peered":"2024-04-17T10:33:30.513312+0000","last_unstale":"2024-04-17T10:33:31.514322+0000","last_undegraded":"2024-04-17T10:33:31.514322+0000","last_fullsized":"2024-04-17T10:33:31.514322+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:21.853803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530247+0000","last_change":"2024-04-17T10:33:30.506874+0000","last_active":"2024-04-17T10:33:31.530247+0000","last_peered":"2024-04-17T10:33:31.530247+0000","last_clean":"2024-04-17T10:33:31.530247+0000","last_became_active":"2024-04-17T10:33:30.506086+0000","last_became_peered":"2024-04-17T10:33:30.506086+0000","last_unstale":"2024-04-17T10:33:31.530247+0000","last_undegraded":"2024-04-17T10:33:31.530247+0000","last_fullsized":"2024-04-17T10:33:31.530247+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:54:26.562886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612836+0000","last_change":"2024-04-17T10:33:30.520353+0000","last_active":"2024-04-17T10:33:31.612836+0000","last_peered":"2024-04-17T10:33:31.612836+0000","last_clean":"2024-04-17T10:33:31.612836+0000","last_became_active":"2024-04-17T10:33:30.519575+0000","last_became_peered":"2024-04-17T10:33:30.519575+0000","last_unstale":"2024-04-17T10:33:31.612836+0000","last_undegraded":"2024-04-17T10:33:31.612836+0000","last_fullsized":"2024-04-17T10:33:31.612836+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:00:22.416979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612825+0000","last_change":"2024-04-17T10:33:30.520547+0000","last_active":"2024-04-17T10:33:31.612825+0000","last_peered":"2024-04-17T10:33:31.612825+0000","last_clean":"2024-04-17T10:33:31.612825+0000","last_became_active":"2024-04-17T10:33:30.520259+0000","last_became_peered":"2024-04-17T10:33:30.520259+0000","last_unstale":"2024-04-17T10:33:31.612825+0000","last_undegraded":"2024-04-17T10:33:31.612825+0000","last_fullsized":"2024-04-17T10:33:31.612825+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.599913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513878+0000","last_change":"2024-04-17T10:33:30.514355+0000","last_active":"2024-04-17T10:33:31.513878+0000","last_peered":"2024-04-17T10:33:31.513878+0000","last_clean":"2024-04-17T10:33:31.513878+0000","last_became_active":"2024-04-17T10:33:30.514033+0000","last_became_peered":"2024-04-17T10:33:30.514033+0000","last_unstale":"2024-04-17T10:33:31.513878+0000","last_undegraded":"2024-04-17T10:33:31.513878+0000","last_fullsized":"2024-04-17T10:33:31.513878+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:32.701908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529651+0000","last_change":"2024-04-17T10:33:30.511617+0000","last_active":"2024-04-17T10:33:31.529651+0000","last_peered":"2024-04-17T10:33:31.529651+0000","last_clean":"2024-04-17T10:33:31.529651+0000","last_became_active":"2024-04-17T10:33:30.511282+0000","last_became_peered":"2024-04-17T10:33:30.511282+0000","last_unstale":"2024-04-17T10:33:31.529651+0000","last_undegraded":"2024-04-17T10:33:31.529651+0000","last_fullsized":"2024-04-17T10:33:31.529651+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:51:34.981246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613126+0000","last_change":"2024-04-17T10:33:30.517765+0000","last_active":"2024-04-17T10:33:31.613126+0000","last_peered":"2024-04-17T10:33:31.613126+0000","last_clean":"2024-04-17T10:33:31.613126+0000","last_became_active":"2024-04-17T10:33:30.517451+0000","last_became_peered":"2024-04-17T10:33:30.517451+0000","last_unstale":"2024-04-17T10:33:31.613126+0000","last_undegraded":"2024-04-17T10:33:31.613126+0000","last_fullsized":"2024-04-17T10:33:31.613126+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:30.853095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613388+0000","last_change":"2024-04-17T10:33:30.516067+0000","last_active":"2024-04-17T10:33:31.613388+0000","last_peered":"2024-04-17T10:33:31.613388+0000","last_clean":"2024-04-17T10:33:31.613388+0000","last_became_active":"2024-04-17T10:33:30.515706+0000","last_became_peered":"2024-04-17T10:33:30.515706+0000","last_unstale":"2024-04-17T10:33:31.613388+0000","last_undegraded":"2024-04-17T10:33:31.613388+0000","last_fullsized":"2024-04-17T10:33:31.613388+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:14.541115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524624+0000","last_change":"2024-04-17T10:33:30.524529+0000","last_active":"2024-04-17T10:33:31.524624+0000","last_peered":"2024-04-17T10:33:31.524624+0000","last_clean":"2024-04-17T10:33:31.524624+0000","last_became_active":"2024-04-17T10:33:30.524272+0000","last_became_peered":"2024-04-17T10:33:30.524272+0000","last_unstale":"2024-04-17T10:33:31.524624+0000","last_undegraded":"2024-04-17T10:33:31.524624+0000","last_fullsized":"2024-04-17T10:33:31.524624+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:44.795215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518614+0000","last_change":"2024-04-17T10:33:30.512314+0000","last_active":"2024-04-17T10:33:31.518614+0000","last_peered":"2024-04-17T10:33:31.518614+0000","last_clean":"2024-04-17T10:33:31.518614+0000","last_became_active":"2024-04-17T10:33:30.512010+0000","last_became_peered":"2024-04-17T10:33:30.512010+0000","last_unstale":"2024-04-17T10:33:31.518614+0000","last_undegraded":"2024-04-17T10:33:31.518614+0000","last_fullsized":"2024-04-17T10:33:31.518614+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:29:27.471947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609841+0000","last_change":"2024-04-17T10:33:30.527262+0000","last_active":"2024-04-17T10:33:31.609841+0000","last_peered":"2024-04-17T10:33:31.609841+0000","last_clean":"2024-04-17T10:33:31.609841+0000","last_became_active":"2024-04-17T10:33:30.526790+0000","last_became_peered":"2024-04-17T10:33:30.526790+0000","last_unstale":"2024-04-17T10:33:31.609841+0000","last_undegraded":"2024-04-17T10:33:31.609841+0000","last_fullsized":"2024-04-17T10:33:31.609841+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:38.216709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514275+0000","last_change":"2024-04-17T10:33:30.521026+0000","last_active":"2024-04-17T10:33:31.514275+0000","last_peered":"2024-04-17T10:33:31.514275+0000","last_clean":"2024-04-17T10:33:31.514275+0000","last_became_active":"2024-04-17T10:33:30.520723+0000","last_became_peered":"2024-04-17T10:33:30.520723+0000","last_unstale":"2024-04-17T10:33:31.514275+0000","last_undegraded":"2024-04-17T10:33:31.514275+0000","last_fullsized":"2024-04-17T10:33:31.514275+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:02.016724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609493+0000","last_change":"2024-04-17T10:33:30.530596+0000","last_active":"2024-04-17T10:33:31.609493+0000","last_peered":"2024-04-17T10:33:31.609493+0000","last_clean":"2024-04-17T10:33:31.609493+0000","last_became_active":"2024-04-17T10:33:30.530312+0000","last_became_peered":"2024-04-17T10:33:30.530312+0000","last_unstale":"2024-04-17T10:33:31.609493+0000","last_undegraded":"2024-04-17T10:33:31.609493+0000","last_fullsized":"2024-04-17T10:33:31.609493+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:57.706709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513491+0000","last_change":"2024-04-17T10:33:30.523226+0000","last_active":"2024-04-17T10:33:31.513491+0000","last_peered":"2024-04-17T10:33:31.513491+0000","last_clean":"2024-04-17T10:33:31.513491+0000","last_became_active":"2024-04-17T10:33:30.522944+0000","last_became_peered":"2024-04-17T10:33:30.522944+0000","last_unstale":"2024-04-17T10:33:31.513491+0000","last_undegraded":"2024-04-17T10:33:31.513491+0000","last_fullsized":"2024-04-17T10:33:31.513491+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:30:06.352801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514584+0000","last_change":"2024-04-17T10:33:30.519734+0000","last_active":"2024-04-17T10:33:31.514584+0000","last_peered":"2024-04-17T10:33:31.514584+0000","last_clean":"2024-04-17T10:33:31.514584+0000","last_became_active":"2024-04-17T10:33:30.519440+0000","last_became_peered":"2024-04-17T10:33:30.519440+0000","last_unstale":"2024-04-17T10:33:31.514584+0000","last_undegraded":"2024-04-17T10:33:31.514584+0000","last_fullsized":"2024-04-17T10:33:31.514584+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:13.368923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518720+0000","last_change":"2024-04-17T10:33:30.518449+0000","last_active":"2024-04-17T10:33:31.518720+0000","last_peered":"2024-04-17T10:33:31.518720+0000","last_clean":"2024-04-17T10:33:31.518720+0000","last_became_active":"2024-04-17T10:33:30.518180+0000","last_became_peered":"2024-04-17T10:33:30.518180+0000","last_unstale":"2024-04-17T10:33:31.518720+0000","last_undegraded":"2024-04-17T10:33:31.518720+0000","last_fullsized":"2024-04-17T10:33:31.518720+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:12:20.469351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524604+0000","last_change":"2024-04-17T10:33:30.529753+0000","last_active":"2024-04-17T10:33:31.524604+0000","last_peered":"2024-04-17T10:33:31.524604+0000","last_clean":"2024-04-17T10:33:31.524604+0000","last_became_active":"2024-04-17T10:33:30.529285+0000","last_became_peered":"2024-04-17T10:33:30.529285+0000","last_unstale":"2024-04-17T10:33:31.524604+0000","last_undegraded":"2024-04-17T10:33:31.524604+0000","last_fullsized":"2024-04-17T10:33:31.524604+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:04.491386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530141+0000","last_change":"2024-04-17T10:33:30.516445+0000","last_active":"2024-04-17T10:33:31.530141+0000","last_peered":"2024-04-17T10:33:31.530141+0000","last_clean":"2024-04-17T10:33:31.530141+0000","last_became_active":"2024-04-17T10:33:30.516056+0000","last_became_peered":"2024-04-17T10:33:30.516056+0000","last_unstale":"2024-04-17T10:33:31.530141+0000","last_undegraded":"2024-04-17T10:33:31.530141+0000","last_fullsized":"2024-04-17T10:33:31.530141+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:00:48.556618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518575+0000","last_change":"2024-04-17T10:33:30.526282+0000","last_active":"2024-04-17T10:33:31.518575+0000","last_peered":"2024-04-17T10:33:31.518575+0000","last_clean":"2024-04-17T10:33:31.518575+0000","last_became_active":"2024-04-17T10:33:30.525850+0000","last_became_peered":"2024-04-17T10:33:30.525850+0000","last_unstale":"2024-04-17T10:33:31.518575+0000","last_undegraded":"2024-04-17T10:33:31.518575+0000","last_fullsized":"2024-04-17T10:33:31.518575+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:55:10.126907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612903+0000","last_change":"2024-04-17T10:33:30.513650+0000","last_active":"2024-04-17T10:33:31.612903+0000","last_peered":"2024-04-17T10:33:31.612903+0000","last_clean":"2024-04-17T10:33:31.612903+0000","last_became_active":"2024-04-17T10:33:30.513394+0000","last_became_peered":"2024-04-17T10:33:30.513394+0000","last_unstale":"2024-04-17T10:33:31.612903+0000","last_undegraded":"2024-04-17T10:33:31.612903+0000","last_fullsized":"2024-04-17T10:33:31.612903+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:23.766536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513633+0000","last_change":"2024-04-17T10:33:30.512740+0000","last_active":"2024-04-17T10:33:31.513633+0000","last_peered":"2024-04-17T10:33:31.513633+0000","last_clean":"2024-04-17T10:33:31.513633+0000","last_became_active":"2024-04-17T10:33:30.512393+0000","last_became_peered":"2024-04-17T10:33:30.512393+0000","last_unstale":"2024-04-17T10:33:31.513633+0000","last_undegraded":"2024-04-17T10:33:31.513633+0000","last_fullsized":"2024-04-17T10:33:31.513633+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:14:01.311909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513544+0000","last_change":"2024-04-17T10:33:30.522878+0000","last_active":"2024-04-17T10:33:31.513544+0000","last_peered":"2024-04-17T10:33:31.513544+0000","last_clean":"2024-04-17T10:33:31.513544+0000","last_became_active":"2024-04-17T10:33:30.522568+0000","last_became_peered":"2024-04-17T10:33:30.522568+0000","last_unstale":"2024-04-17T10:33:31.513544+0000","last_undegraded":"2024-04-17T10:33:31.513544+0000","last_fullsized":"2024-04-17T10:33:31.513544+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:47.891876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612640+0000","last_change":"2024-04-17T10:33:30.517703+0000","last_active":"2024-04-17T10:33:31.612640+0000","last_peered":"2024-04-17T10:33:31.612640+0000","last_clean":"2024-04-17T10:33:31.612640+0000","last_became_active":"2024-04-17T10:33:30.517367+0000","last_became_peered":"2024-04-17T10:33:30.517367+0000","last_unstale":"2024-04-17T10:33:31.612640+0000","last_undegraded":"2024-04-17T10:33:31.612640+0000","last_fullsized":"2024-04-17T10:33:31.612640+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:04:31.291283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"117'4","reported_seq":18,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524670+0000","last_change":"2024-04-17T10:33:29.488987+0000","last_active":"2024-04-17T10:33:31.524670+0000","last_peered":"2024-04-17T10:33:31.524670+0000","last_clean":"2024-04-17T10:33:31.524670+0000","last_became_active":"2024-04-17T10:33:29.488649+0000","last_became_peered":"2024-04-17T10:33:29.488649+0000","last_unstale":"2024-04-17T10:33:31.524670+0000","last_undegraded":"2024-04-17T10:33:31.524670+0000","last_fullsized":"2024-04-17T10:33:31.524670+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:18:03.802935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530094+0000","last_change":"2024-04-17T10:33:30.525973+0000","last_active":"2024-04-17T10:33:31.530094+0000","last_peered":"2024-04-17T10:33:31.530094+0000","last_clean":"2024-04-17T10:33:31.530094+0000","last_became_active":"2024-04-17T10:33:30.525740+0000","last_became_peered":"2024-04-17T10:33:30.525740+0000","last_unstale":"2024-04-17T10:33:31.530094+0000","last_undegraded":"2024-04-17T10:33:31.530094+0000","last_fullsized":"2024-04-17T10:33:31.530094+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:59.558573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612997+0000","last_change":"2024-04-17T10:33:29.490892+0000","last_active":"2024-04-17T10:33:31.612997+0000","last_peered":"2024-04-17T10:33:31.612997+0000","last_clean":"2024-04-17T10:33:31.612997+0000","last_became_active":"2024-04-17T10:33:29.490624+0000","last_became_peered":"2024-04-17T10:33:29.490624+0000","last_unstale":"2024-04-17T10:33:31.612997+0000","last_undegraded":"2024-04-17T10:33:31.612997+0000","last_fullsized":"2024-04-17T10:33:31.612997+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:24:43.542422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514383+0000","last_change":"2024-04-17T10:33:30.521938+0000","last_active":"2024-04-17T10:33:31.514383+0000","last_peered":"2024-04-17T10:33:31.514383+0000","last_clean":"2024-04-17T10:33:31.514383+0000","last_became_active":"2024-04-17T10:33:30.521512+0000","last_became_peered":"2024-04-17T10:33:30.521512+0000","last_unstale":"2024-04-17T10:33:31.514383+0000","last_undegraded":"2024-04-17T10:33:31.514383+0000","last_fullsized":"2024-04-17T10:33:31.514383+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:20:57.706709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609914+0000","last_change":"2024-04-17T10:33:29.494875+0000","last_active":"2024-04-17T10:33:31.609914+0000","last_peered":"2024-04-17T10:33:31.609914+0000","last_clean":"2024-04-17T10:33:31.609914+0000","last_became_active":"2024-04-17T10:33:29.494626+0000","last_became_peered":"2024-04-17T10:33:29.494626+0000","last_unstale":"2024-04-17T10:33:31.609914+0000","last_undegraded":"2024-04-17T10:33:31.609914+0000","last_fullsized":"2024-04-17T10:33:31.609914+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:49.924443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.610136+0000","last_change":"2024-04-17T10:33:30.524503+0000","last_active":"2024-04-17T10:33:31.610136+0000","last_peered":"2024-04-17T10:33:31.610136+0000","last_clean":"2024-04-17T10:33:31.610136+0000","last_became_active":"2024-04-17T10:33:30.523960+0000","last_became_peered":"2024-04-17T10:33:30.523960+0000","last_unstale":"2024-04-17T10:33:31.610136+0000","last_undegraded":"2024-04-17T10:33:31.610136+0000","last_fullsized":"2024-04-17T10:33:31.610136+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:57.278375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612747+0000","last_change":"2024-04-17T10:33:29.495956+0000","last_active":"2024-04-17T10:33:31.612747+0000","last_peered":"2024-04-17T10:33:31.612747+0000","last_clean":"2024-04-17T10:33:31.612747+0000","last_became_active":"2024-04-17T10:33:29.495664+0000","last_became_peered":"2024-04-17T10:33:29.495664+0000","last_unstale":"2024-04-17T10:33:31.612747+0000","last_undegraded":"2024-04-17T10:33:31.612747+0000","last_fullsized":"2024-04-17T10:33:31.612747+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:10:20.135789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612353+0000","last_change":"2024-04-17T10:33:30.519779+0000","last_active":"2024-04-17T10:33:31.612353+0000","last_peered":"2024-04-17T10:33:31.612353+0000","last_clean":"2024-04-17T10:33:31.612353+0000","last_became_active":"2024-04-17T10:33:30.519499+0000","last_became_peered":"2024-04-17T10:33:30.519499+0000","last_unstale":"2024-04-17T10:33:31.612353+0000","last_undegraded":"2024-04-17T10:33:31.612353+0000","last_fullsized":"2024-04-17T10:33:31.612353+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:04.491386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514630+0000","last_change":"2024-04-17T10:33:29.484585+0000","last_active":"2024-04-17T10:33:31.514630+0000","last_peered":"2024-04-17T10:33:31.514630+0000","last_clean":"2024-04-17T10:33:31.514630+0000","last_became_active":"2024-04-17T10:33:29.483420+0000","last_became_peered":"2024-04-17T10:33:29.483420+0000","last_unstale":"2024-04-17T10:33:31.514630+0000","last_undegraded":"2024-04-17T10:33:31.514630+0000","last_fullsized":"2024-04-17T10:33:31.514630+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:45.687248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514052+0000","last_change":"2024-04-17T10:33:30.521856+0000","last_active":"2024-04-17T10:33:31.514052+0000","last_peered":"2024-04-17T10:33:31.514052+0000","last_clean":"2024-04-17T10:33:31.514052+0000","last_became_active":"2024-04-17T10:33:30.521588+0000","last_became_peered":"2024-04-17T10:33:30.521588+0000","last_unstale":"2024-04-17T10:33:31.514052+0000","last_undegraded":"2024-04-17T10:33:31.514052+0000","last_fullsized":"2024-04-17T10:33:31.514052+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:17.229608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529919+0000","last_change":"2024-04-17T10:33:29.499168+0000","last_active":"2024-04-17T10:33:31.529919+0000","last_peered":"2024-04-17T10:33:31.529919+0000","last_clean":"2024-04-17T10:33:31.529919+0000","last_became_active":"2024-04-17T10:33:29.498878+0000","last_became_peered":"2024-04-17T10:33:29.498878+0000","last_unstale":"2024-04-17T10:33:31.529919+0000","last_undegraded":"2024-04-17T10:33:31.529919+0000","last_fullsized":"2024-04-17T10:33:31.529919+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:01.294357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612977+0000","last_change":"2024-04-17T10:33:30.518406+0000","last_active":"2024-04-17T10:33:31.612977+0000","last_peered":"2024-04-17T10:33:31.612977+0000","last_clean":"2024-04-17T10:33:31.612977+0000","last_became_active":"2024-04-17T10:33:30.518092+0000","last_became_peered":"2024-04-17T10:33:30.518092+0000","last_unstale":"2024-04-17T10:33:31.612977+0000","last_undegraded":"2024-04-17T10:33:31.612977+0000","last_fullsized":"2024-04-17T10:33:31.612977+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:33:53.545591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513624+0000","last_change":"2024-04-17T10:33:29.482197+0000","last_active":"2024-04-17T10:33:31.513624+0000","last_peered":"2024-04-17T10:33:31.513624+0000","last_clean":"2024-04-17T10:33:31.513624+0000","last_became_active":"2024-04-17T10:33:29.481902+0000","last_became_peered":"2024-04-17T10:33:29.481902+0000","last_unstale":"2024-04-17T10:33:31.513624+0000","last_undegraded":"2024-04-17T10:33:31.513624+0000","last_fullsized":"2024-04-17T10:33:31.513624+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:11.604372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612485+0000","last_change":"2024-04-17T10:33:30.530577+0000","last_active":"2024-04-17T10:33:31.612485+0000","last_peered":"2024-04-17T10:33:31.612485+0000","last_clean":"2024-04-17T10:33:31.612485+0000","last_became_active":"2024-04-17T10:33:30.530268+0000","last_became_peered":"2024-04-17T10:33:30.530268+0000","last_unstale":"2024-04-17T10:33:31.612485+0000","last_undegraded":"2024-04-17T10:33:31.612485+0000","last_fullsized":"2024-04-17T10:33:31.612485+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:21:56.144774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530039+0000","last_change":"2024-04-17T10:33:29.498168+0000","last_active":"2024-04-17T10:33:31.530039+0000","last_peered":"2024-04-17T10:33:31.530039+0000","last_clean":"2024-04-17T10:33:31.530039+0000","last_became_active":"2024-04-17T10:33:29.497688+0000","last_became_peered":"2024-04-17T10:33:29.497688+0000","last_unstale":"2024-04-17T10:33:31.530039+0000","last_undegraded":"2024-04-17T10:33:31.530039+0000","last_fullsized":"2024-04-17T10:33:31.530039+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:52:32.030486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524373+0000","last_change":"2024-04-17T10:33:30.511638+0000","last_active":"2024-04-17T10:33:31.524373+0000","last_peered":"2024-04-17T10:33:31.524373+0000","last_clean":"2024-04-17T10:33:31.524373+0000","last_became_active":"2024-04-17T10:33:30.511369+0000","last_became_peered":"2024-04-17T10:33:30.511369+0000","last_unstale":"2024-04-17T10:33:31.524373+0000","last_undegraded":"2024-04-17T10:33:31.524373+0000","last_fullsized":"2024-04-17T10:33:31.524373+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:14:09.976313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524642+0000","last_change":"2024-04-17T10:33:29.485622+0000","last_active":"2024-04-17T10:33:31.524642+0000","last_peered":"2024-04-17T10:33:31.524642+0000","last_clean":"2024-04-17T10:33:31.524642+0000","last_became_active":"2024-04-17T10:33:29.485350+0000","last_became_peered":"2024-04-17T10:33:29.485350+0000","last_unstale":"2024-04-17T10:33:31.524642+0000","last_undegraded":"2024-04-17T10:33:31.524642+0000","last_fullsized":"2024-04-17T10:33:31.524642+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:33:12.191975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514265+0000","last_change":"2024-04-17T10:33:30.520954+0000","last_active":"2024-04-17T10:33:31.514265+0000","last_peered":"2024-04-17T10:33:31.514265+0000","last_clean":"2024-04-17T10:33:31.514265+0000","last_became_active":"2024-04-17T10:33:30.520672+0000","last_became_peered":"2024-04-17T10:33:30.520672+0000","last_unstale":"2024-04-17T10:33:31.514265+0000","last_undegraded":"2024-04-17T10:33:31.514265+0000","last_fullsized":"2024-04-17T10:33:31.514265+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:37:29.525176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514474+0000","last_change":"2024-04-17T10:33:29.486831+0000","last_active":"2024-04-17T10:33:31.514474+0000","last_peered":"2024-04-17T10:33:31.514474+0000","last_clean":"2024-04-17T10:33:31.514474+0000","last_became_active":"2024-04-17T10:33:29.486204+0000","last_became_peered":"2024-04-17T10:33:29.486204+0000","last_unstale":"2024-04-17T10:33:31.514474+0000","last_undegraded":"2024-04-17T10:33:31.514474+0000","last_fullsized":"2024-04-17T10:33:31.514474+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:44:31.691854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613416+0000","last_change":"2024-04-17T10:33:30.519388+0000","last_active":"2024-04-17T10:33:31.613416+0000","last_peered":"2024-04-17T10:33:31.613416+0000","last_clean":"2024-04-17T10:33:31.613416+0000","last_became_active":"2024-04-17T10:33:30.519104+0000","last_became_peered":"2024-04-17T10:33:30.519104+0000","last_unstale":"2024-04-17T10:33:31.613416+0000","last_undegraded":"2024-04-17T10:33:31.613416+0000","last_fullsized":"2024-04-17T10:33:31.613416+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:00.490108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"117'3","reported_seq":17,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514374+0000","last_change":"2024-04-17T10:33:29.485972+0000","last_active":"2024-04-17T10:33:31.514374+0000","last_peered":"2024-04-17T10:33:31.514374+0000","last_clean":"2024-04-17T10:33:31.514374+0000","last_became_active":"2024-04-17T10:33:29.485542+0000","last_became_peered":"2024-04-17T10:33:29.485542+0000","last_unstale":"2024-04-17T10:33:31.514374+0000","last_undegraded":"2024-04-17T10:33:31.514374+0000","last_fullsized":"2024-04-17T10:33:31.514374+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:22:29.843042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514655+0000","last_change":"2024-04-17T10:33:30.513626+0000","last_active":"2024-04-17T10:33:31.514655+0000","last_peered":"2024-04-17T10:33:31.514655+0000","last_clean":"2024-04-17T10:33:31.514655+0000","last_became_active":"2024-04-17T10:33:30.513332+0000","last_became_peered":"2024-04-17T10:33:30.513332+0000","last_unstale":"2024-04-17T10:33:31.514655+0000","last_undegraded":"2024-04-17T10:33:31.514655+0000","last_fullsized":"2024-04-17T10:33:31.514655+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:39:48.741482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"117'6","reported_seq":20,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612840+0000","last_change":"2024-04-17T10:33:29.490370+0000","last_active":"2024-04-17T10:33:31.612840+0000","last_peered":"2024-04-17T10:33:31.612840+0000","last_clean":"2024-04-17T10:33:31.612840+0000","last_became_active":"2024-04-17T10:33:29.490109+0000","last_became_peered":"2024-04-17T10:33:29.490109+0000","last_unstale":"2024-04-17T10:33:31.612840+0000","last_undegraded":"2024-04-17T10:33:31.612840+0000","last_fullsized":"2024-04-17T10:33:31.612840+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:20.586624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524468+0000","last_change":"2024-04-17T10:33:30.523893+0000","last_active":"2024-04-17T10:33:31.524468+0000","last_peered":"2024-04-17T10:33:31.524468+0000","last_clean":"2024-04-17T10:33:31.524468+0000","last_became_active":"2024-04-17T10:33:30.523553+0000","last_became_peered":"2024-04-17T10:33:30.523553+0000","last_unstale":"2024-04-17T10:33:31.524468+0000","last_undegraded":"2024-04-17T10:33:31.524468+0000","last_fullsized":"2024-04-17T10:33:31.524468+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:46.697302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612819+0000","last_change":"2024-04-17T10:33:29.494255+0000","last_active":"2024-04-17T10:33:31.612819+0000","last_peered":"2024-04-17T10:33:31.612819+0000","last_clean":"2024-04-17T10:33:31.612819+0000","last_became_active":"2024-04-17T10:33:29.493978+0000","last_became_peered":"2024-04-17T10:33:29.493978+0000","last_unstale":"2024-04-17T10:33:31.612819+0000","last_undegraded":"2024-04-17T10:33:31.612819+0000","last_fullsized":"2024-04-17T10:33:31.612819+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:20.843749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612365+0000","last_change":"2024-04-17T10:33:30.530914+0000","last_active":"2024-04-17T10:33:31.612365+0000","last_peered":"2024-04-17T10:33:31.612365+0000","last_clean":"2024-04-17T10:33:31.612365+0000","last_became_active":"2024-04-17T10:33:30.530657+0000","last_became_peered":"2024-04-17T10:33:30.530657+0000","last_unstale":"2024-04-17T10:33:31.612365+0000","last_undegraded":"2024-04-17T10:33:31.612365+0000","last_fullsized":"2024-04-17T10:33:31.612365+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:05:44.795215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613137+0000","last_change":"2024-04-17T10:33:29.492252+0000","last_active":"2024-04-17T10:33:31.613137+0000","last_peered":"2024-04-17T10:33:31.613137+0000","last_clean":"2024-04-17T10:33:31.613137+0000","last_became_active":"2024-04-17T10:33:29.490092+0000","last_became_peered":"2024-04-17T10:33:29.490092+0000","last_unstale":"2024-04-17T10:33:31.613137+0000","last_undegraded":"2024-04-17T10:33:31.613137+0000","last_fullsized":"2024-04-17T10:33:31.613137+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:46.582002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518342+0000","last_change":"2024-04-17T10:33:30.513713+0000","last_active":"2024-04-17T10:33:31.518342+0000","last_peered":"2024-04-17T10:33:31.518342+0000","last_clean":"2024-04-17T10:33:31.518342+0000","last_became_active":"2024-04-17T10:33:30.513439+0000","last_became_peered":"2024-04-17T10:33:30.513439+0000","last_unstale":"2024-04-17T10:33:31.518342+0000","last_undegraded":"2024-04-17T10:33:31.518342+0000","last_fullsized":"2024-04-17T10:33:31.518342+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:40:13.007807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613130+0000","last_change":"2024-04-17T10:33:29.483114+0000","last_active":"2024-04-17T10:33:31.613130+0000","last_peered":"2024-04-17T10:33:31.613130+0000","last_clean":"2024-04-17T10:33:31.613130+0000","last_became_active":"2024-04-17T10:33:29.482291+0000","last_became_peered":"2024-04-17T10:33:29.482291+0000","last_unstale":"2024-04-17T10:33:31.613130+0000","last_undegraded":"2024-04-17T10:33:31.613130+0000","last_fullsized":"2024-04-17T10:33:31.613130+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:43:49.924443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514076+0000","last_change":"2024-04-17T10:33:30.521974+0000","last_active":"2024-04-17T10:33:31.514076+0000","last_peered":"2024-04-17T10:33:31.514076+0000","last_clean":"2024-04-17T10:33:31.514076+0000","last_became_active":"2024-04-17T10:33:30.521654+0000","last_became_peered":"2024-04-17T10:33:30.521654+0000","last_unstale":"2024-04-17T10:33:31.514076+0000","last_undegraded":"2024-04-17T10:33:31.514076+0000","last_fullsized":"2024-04-17T10:33:31.514076+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:20:31.627592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613188+0000","last_change":"2024-04-17T10:33:29.498174+0000","last_active":"2024-04-17T10:33:31.613188+0000","last_peered":"2024-04-17T10:33:31.613188+0000","last_clean":"2024-04-17T10:33:31.613188+0000","last_became_active":"2024-04-17T10:33:29.497727+0000","last_became_peered":"2024-04-17T10:33:29.497727+0000","last_unstale":"2024-04-17T10:33:31.613188+0000","last_undegraded":"2024-04-17T10:33:31.613188+0000","last_fullsized":"2024-04-17T10:33:31.613188+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:46:54.161898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514477+0000","last_change":"2024-04-17T10:33:30.521113+0000","last_active":"2024-04-17T10:33:31.514477+0000","last_peered":"2024-04-17T10:33:31.514477+0000","last_clean":"2024-04-17T10:33:31.514477+0000","last_became_active":"2024-04-17T10:33:30.520808+0000","last_became_peered":"2024-04-17T10:33:30.520808+0000","last_unstale":"2024-04-17T10:33:31.514477+0000","last_undegraded":"2024-04-17T10:33:31.514477+0000","last_fullsized":"2024-04-17T10:33:31.514477+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:21.596678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518723+0000","last_change":"2024-04-17T10:33:29.488532+0000","last_active":"2024-04-17T10:33:31.518723+0000","last_peered":"2024-04-17T10:33:31.518723+0000","last_clean":"2024-04-17T10:33:31.518723+0000","last_became_active":"2024-04-17T10:33:29.488234+0000","last_became_peered":"2024-04-17T10:33:29.488234+0000","last_unstale":"2024-04-17T10:33:31.518723+0000","last_undegraded":"2024-04-17T10:33:31.518723+0000","last_fullsized":"2024-04-17T10:33:31.518723+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:32:10.059619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530257+0000","last_change":"2024-04-17T10:33:30.515255+0000","last_active":"2024-04-17T10:33:31.530257+0000","last_peered":"2024-04-17T10:33:31.530257+0000","last_clean":"2024-04-17T10:33:31.530257+0000","last_became_active":"2024-04-17T10:33:30.512245+0000","last_became_peered":"2024-04-17T10:33:30.512245+0000","last_unstale":"2024-04-17T10:33:31.530257+0000","last_undegraded":"2024-04-17T10:33:31.530257+0000","last_fullsized":"2024-04-17T10:33:31.530257+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:35:27.370635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"117'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609941+0000","last_change":"2024-04-17T10:33:29.488911+0000","last_active":"2024-04-17T10:33:31.609941+0000","last_peered":"2024-04-17T10:33:31.609941+0000","last_clean":"2024-04-17T10:33:31.609941+0000","last_became_active":"2024-04-17T10:33:29.488581+0000","last_became_peered":"2024-04-17T10:33:29.488581+0000","last_unstale":"2024-04-17T10:33:31.609941+0000","last_undegraded":"2024-04-17T10:33:31.609941+0000","last_fullsized":"2024-04-17T10:33:31.609941+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:53:13.663196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514723+0000","last_change":"2024-04-17T10:33:30.529062+0000","last_active":"2024-04-17T10:33:31.514723+0000","last_peered":"2024-04-17T10:33:31.514723+0000","last_clean":"2024-04-17T10:33:31.514723+0000","last_became_active":"2024-04-17T10:33:30.528793+0000","last_became_peered":"2024-04-17T10:33:30.528793+0000","last_unstale":"2024-04-17T10:33:31.514723+0000","last_undegraded":"2024-04-17T10:33:31.514723+0000","last_fullsized":"2024-04-17T10:33:31.514723+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:38.216709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609838+0000","last_change":"2024-04-17T10:33:29.495528+0000","last_active":"2024-04-17T10:33:31.609838+0000","last_peered":"2024-04-17T10:33:31.609838+0000","last_clean":"2024-04-17T10:33:31.609838+0000","last_became_active":"2024-04-17T10:33:29.495252+0000","last_became_peered":"2024-04-17T10:33:29.495252+0000","last_unstale":"2024-04-17T10:33:31.609838+0000","last_undegraded":"2024-04-17T10:33:31.609838+0000","last_fullsized":"2024-04-17T10:33:31.609838+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:21:46.582002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529784+0000","last_change":"2024-04-17T10:33:30.513432+0000","last_active":"2024-04-17T10:33:31.529784+0000","last_peered":"2024-04-17T10:33:31.529784+0000","last_clean":"2024-04-17T10:33:31.529784+0000","last_became_active":"2024-04-17T10:33:30.513088+0000","last_became_peered":"2024-04-17T10:33:30.513088+0000","last_unstale":"2024-04-17T10:33:31.529784+0000","last_undegraded":"2024-04-17T10:33:31.529784+0000","last_fullsized":"2024-04-17T10:33:31.529784+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:18:58.424993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529786+0000","last_change":"2024-04-17T10:33:29.495116+0000","last_active":"2024-04-17T10:33:31.529786+0000","last_peered":"2024-04-17T10:33:31.529786+0000","last_clean":"2024-04-17T10:33:31.529786+0000","last_became_active":"2024-04-17T10:33:29.494706+0000","last_became_peered":"2024-04-17T10:33:29.494706+0000","last_unstale":"2024-04-17T10:33:31.529786+0000","last_undegraded":"2024-04-17T10:33:31.529786+0000","last_fullsized":"2024-04-17T10:33:31.529786+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:01:53.711266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612988+0000","last_change":"2024-04-17T10:33:30.515873+0000","last_active":"2024-04-17T10:33:31.612988+0000","last_peered":"2024-04-17T10:33:31.612988+0000","last_clean":"2024-04-17T10:33:31.612988+0000","last_became_active":"2024-04-17T10:33:30.515500+0000","last_became_peered":"2024-04-17T10:33:30.515500+0000","last_unstale":"2024-04-17T10:33:31.612988+0000","last_undegraded":"2024-04-17T10:33:31.612988+0000","last_fullsized":"2024-04-17T10:33:31.612988+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:05:57.278375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609422+0000","last_change":"2024-04-17T10:33:29.488855+0000","last_active":"2024-04-17T10:33:31.609422+0000","last_peered":"2024-04-17T10:33:31.609422+0000","last_clean":"2024-04-17T10:33:31.609422+0000","last_became_active":"2024-04-17T10:33:29.488538+0000","last_became_peered":"2024-04-17T10:33:29.488538+0000","last_unstale":"2024-04-17T10:33:31.609422+0000","last_undegraded":"2024-04-17T10:33:31.609422+0000","last_fullsized":"2024-04-17T10:33:31.609422+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:54.326552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613168+0000","last_change":"2024-04-17T10:33:30.513047+0000","last_active":"2024-04-17T10:33:31.613168+0000","last_peered":"2024-04-17T10:33:31.613168+0000","last_clean":"2024-04-17T10:33:31.613168+0000","last_became_active":"2024-04-17T10:33:30.512503+0000","last_became_peered":"2024-04-17T10:33:30.512503+0000","last_unstale":"2024-04-17T10:33:31.613168+0000","last_undegraded":"2024-04-17T10:33:31.613168+0000","last_fullsized":"2024-04-17T10:33:31.613168+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:36:59.103325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613242+0000","last_change":"2024-04-17T10:33:29.493323+0000","last_active":"2024-04-17T10:33:31.613242+0000","last_peered":"2024-04-17T10:33:31.613242+0000","last_clean":"2024-04-17T10:33:31.613242+0000","last_became_active":"2024-04-17T10:33:29.491574+0000","last_became_peered":"2024-04-17T10:33:29.491574+0000","last_unstale":"2024-04-17T10:33:31.613242+0000","last_undegraded":"2024-04-17T10:33:31.613242+0000","last_fullsized":"2024-04-17T10:33:31.613242+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:33:54.326552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.513932+0000","last_change":"2024-04-17T10:33:30.516151+0000","last_active":"2024-04-17T10:33:31.513932+0000","last_peered":"2024-04-17T10:33:31.513932+0000","last_clean":"2024-04-17T10:33:31.513932+0000","last_became_active":"2024-04-17T10:33:30.515877+0000","last_became_peered":"2024-04-17T10:33:30.515877+0000","last_unstale":"2024-04-17T10:33:31.513932+0000","last_undegraded":"2024-04-17T10:33:31.513932+0000","last_fullsized":"2024-04-17T10:33:31.513932+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:14:41.699457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609262+0000","last_change":"2024-04-17T10:33:29.491280+0000","last_active":"2024-04-17T10:33:31.609262+0000","last_peered":"2024-04-17T10:33:31.609262+0000","last_clean":"2024-04-17T10:33:31.609262+0000","last_became_active":"2024-04-17T10:33:29.490947+0000","last_became_peered":"2024-04-17T10:33:29.490947+0000","last_unstale":"2024-04-17T10:33:31.609262+0000","last_undegraded":"2024-04-17T10:33:31.609262+0000","last_fullsized":"2024-04-17T10:33:31.609262+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:41:20.843749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530304+0000","last_change":"2024-04-17T10:33:30.524459+0000","last_active":"2024-04-17T10:33:31.530304+0000","last_peered":"2024-04-17T10:33:31.530304+0000","last_clean":"2024-04-17T10:33:31.530304+0000","last_became_active":"2024-04-17T10:33:30.524159+0000","last_became_peered":"2024-04-17T10:33:30.524159+0000","last_unstale":"2024-04-17T10:33:31.530304+0000","last_undegraded":"2024-04-17T10:33:31.530304+0000","last_fullsized":"2024-04-17T10:33:31.530304+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:51:17.625384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"117'2","reported_seq":16,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530226+0000","last_change":"2024-04-17T10:33:29.492646+0000","last_active":"2024-04-17T10:33:31.530226+0000","last_peered":"2024-04-17T10:33:31.530226+0000","last_clean":"2024-04-17T10:33:31.530226+0000","last_became_active":"2024-04-17T10:33:29.492187+0000","last_became_peered":"2024-04-17T10:33:29.492187+0000","last_unstale":"2024-04-17T10:33:31.530226+0000","last_undegraded":"2024-04-17T10:33:31.530226+0000","last_fullsized":"2024-04-17T10:33:31.530226+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:09:46.098736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612568+0000","last_change":"2024-04-17T10:33:30.519616+0000","last_active":"2024-04-17T10:33:31.612568+0000","last_peered":"2024-04-17T10:33:31.612568+0000","last_clean":"2024-04-17T10:33:31.612568+0000","last_became_active":"2024-04-17T10:33:30.519334+0000","last_became_peered":"2024-04-17T10:33:30.519334+0000","last_unstale":"2024-04-17T10:33:31.612568+0000","last_undegraded":"2024-04-17T10:33:31.612568+0000","last_fullsized":"2024-04-17T10:33:31.612568+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:21:04.234855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"118'1","reported_seq":15,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.526385+0000","last_change":"2024-04-17T10:33:29.487245+0000","last_active":"2024-04-17T10:33:31.526385+0000","last_peered":"2024-04-17T10:33:31.526385+0000","last_clean":"2024-04-17T10:33:31.526385+0000","last_became_active":"2024-04-17T10:33:29.486874+0000","last_became_peered":"2024-04-17T10:33:29.486874+0000","last_unstale":"2024-04-17T10:33:31.526385+0000","last_undegraded":"2024-04-17T10:33:31.526385+0000","last_fullsized":"2024-04-17T10:33:31.526385+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:50:13.128031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609940+0000","last_change":"2024-04-17T10:33:30.522497+0000","last_active":"2024-04-17T10:33:31.609940+0000","last_peered":"2024-04-17T10:33:31.609940+0000","last_clean":"2024-04-17T10:33:31.609940+0000","last_became_active":"2024-04-17T10:33:30.522138+0000","last_became_peered":"2024-04-17T10:33:30.522138+0000","last_unstale":"2024-04-17T10:33:31.609940+0000","last_undegraded":"2024-04-17T10:33:31.609940+0000","last_fullsized":"2024-04-17T10:33:31.609940+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:23.766536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.524704+0000","last_change":"2024-04-17T10:33:29.485210+0000","last_active":"2024-04-17T10:33:31.524704+0000","last_peered":"2024-04-17T10:33:31.524704+0000","last_clean":"2024-04-17T10:33:31.524704+0000","last_became_active":"2024-04-17T10:33:29.484745+0000","last_became_peered":"2024-04-17T10:33:29.484745+0000","last_unstale":"2024-04-17T10:33:31.524704+0000","last_undegraded":"2024-04-17T10:33:31.524704+0000","last_fullsized":"2024-04-17T10:33:31.524704+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:56:56.669526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.518835+0000","last_change":"2024-04-17T10:33:30.516350+0000","last_active":"2024-04-17T10:33:31.518835+0000","last_peered":"2024-04-17T10:33:31.518835+0000","last_clean":"2024-04-17T10:33:31.518835+0000","last_became_active":"2024-04-17T10:33:30.516089+0000","last_became_peered":"2024-04-17T10:33:30.516089+0000","last_unstale":"2024-04-17T10:33:31.518835+0000","last_undegraded":"2024-04-17T10:33:31.518835+0000","last_fullsized":"2024-04-17T10:33:31.518835+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:07:53.969041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.530310+0000","last_change":"2024-04-17T10:33:29.497772+0000","last_active":"2024-04-17T10:33:31.530310+0000","last_peered":"2024-04-17T10:33:31.530310+0000","last_clean":"2024-04-17T10:33:31.530310+0000","last_became_active":"2024-04-17T10:33:29.496779+0000","last_became_peered":"2024-04-17T10:33:29.496779+0000","last_unstale":"2024-04-17T10:33:31.530310+0000","last_undegraded":"2024-04-17T10:33:31.530310+0000","last_fullsized":"2024-04-17T10:33:31.530310+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:40:50.156210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529878+0000","last_change":"2024-04-17T10:33:30.506486+0000","last_active":"2024-04-17T10:33:31.529878+0000","last_peered":"2024-04-17T10:33:31.529878+0000","last_clean":"2024-04-17T10:33:31.529878+0000","last_became_active":"2024-04-17T10:33:30.506241+0000","last_became_peered":"2024-04-17T10:33:30.506241+0000","last_unstale":"2024-04-17T10:33:31.529878+0000","last_undegraded":"2024-04-17T10:33:31.529878+0000","last_fullsized":"2024-04-17T10:33:31.529878+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:02.483608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514764+0000","last_change":"2024-04-17T10:33:29.481339+0000","last_active":"2024-04-17T10:33:31.514764+0000","last_peered":"2024-04-17T10:33:31.514764+0000","last_clean":"2024-04-17T10:33:31.514764+0000","last_became_active":"2024-04-17T10:33:29.480897+0000","last_became_peered":"2024-04-17T10:33:29.480897+0000","last_unstale":"2024-04-17T10:33:31.514764+0000","last_undegraded":"2024-04-17T10:33:31.514764+0000","last_fullsized":"2024-04-17T10:33:31.514764+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:30:03.481333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.613147+0000","last_change":"2024-04-17T10:33:30.512815+0000","last_active":"2024-04-17T10:33:31.613147+0000","last_peered":"2024-04-17T10:33:31.613147+0000","last_clean":"2024-04-17T10:33:31.613147+0000","last_became_active":"2024-04-17T10:33:30.512189+0000","last_became_peered":"2024-04-17T10:33:30.512189+0000","last_unstale":"2024-04-17T10:33:31.613147+0000","last_undegraded":"2024-04-17T10:33:31.613147+0000","last_fullsized":"2024-04-17T10:33:31.613147+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:45:31.393021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.514174+0000","last_change":"2024-04-17T10:33:29.476128+0000","last_active":"2024-04-17T10:33:31.514174+0000","last_peered":"2024-04-17T10:33:31.514174+0000","last_clean":"2024-04-17T10:33:31.514174+0000","last_became_active":"2024-04-17T10:33:29.475862+0000","last_became_peered":"2024-04-17T10:33:29.475862+0000","last_unstale":"2024-04-17T10:33:31.514174+0000","last_undegraded":"2024-04-17T10:33:31.514174+0000","last_fullsized":"2024-04-17T10:33:31.514174+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:14:08.966259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609536+0000","last_change":"2024-04-17T10:33:30.526594+0000","last_active":"2024-04-17T10:33:31.609536+0000","last_peered":"2024-04-17T10:33:31.609536+0000","last_clean":"2024-04-17T10:33:31.609536+0000","last_became_active":"2024-04-17T10:33:30.526312+0000","last_became_peered":"2024-04-17T10:33:30.526312+0000","last_unstale":"2024-04-17T10:33:31.609536+0000","last_undegraded":"2024-04-17T10:33:31.609536+0000","last_fullsized":"2024-04-17T10:33:31.609536+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:16:13.368923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.529695+0000","last_change":"2024-04-17T10:33:29.488144+0000","last_active":"2024-04-17T10:33:31.529695+0000","last_peered":"2024-04-17T10:33:31.529695+0000","last_clean":"2024-04-17T10:33:31.529695+0000","last_became_active":"2024-04-17T10:33:29.487882+0000","last_became_peered":"2024-04-17T10:33:29.487882+0000","last_unstale":"2024-04-17T10:33:31.529695+0000","last_undegraded":"2024-04-17T10:33:31.529695+0000","last_fullsized":"2024-04-17T10:33:31.529695+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:32:53.424723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612624+0000","last_change":"2024-04-17T10:33:30.517444+0000","last_active":"2024-04-17T10:33:31.612624+0000","last_peered":"2024-04-17T10:33:31.612624+0000","last_clean":"2024-04-17T10:33:31.612624+0000","last_became_active":"2024-04-17T10:33:30.515775+0000","last_became_peered":"2024-04-17T10:33:30.515775+0000","last_unstale":"2024-04-17T10:33:31.612624+0000","last_undegraded":"2024-04-17T10:33:31.612624+0000","last_fullsized":"2024-04-17T10:33:31.612624+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:07.313743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":14,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609214+0000","last_change":"2024-04-17T10:33:29.491041+0000","last_active":"2024-04-17T10:33:31.609214+0000","last_peered":"2024-04-17T10:33:31.609214+0000","last_clean":"2024-04-17T10:33:31.609214+0000","last_became_active":"2024-04-17T10:33:29.490574+0000","last_became_peered":"2024-04-17T10:33:29.490574+0000","last_unstale":"2024-04-17T10:33:31.609214+0000","last_undegraded":"2024-04-17T10:33:31.609214+0000","last_fullsized":"2024-04-17T10:33:31.609214+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:04:20.586624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":12,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609837+0000","last_change":"2024-04-17T10:33:30.527050+0000","last_active":"2024-04-17T10:33:31.609837+0000","last_peered":"2024-04-17T10:33:31.609837+0000","last_clean":"2024-04-17T10:33:31.609837+0000","last_became_active":"2024-04-17T10:33:30.526471+0000","last_became_peered":"2024-04-17T10:33:30.526471+0000","last_unstale":"2024-04-17T10:33:31.609837+0000","last_undegraded":"2024-04-17T10:33:31.609837+0000","last_fullsized":"2024-04-17T10:33:31.609837+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":116,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:29.466312+0000","last_clean_scrub_stamp":"2024-04-17T10:33:29.466312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:17:30.599913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"117'8","reported_seq":22,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.612682+0000","last_change":"2024-04-17T10:33:29.493929+0000","last_active":"2024-04-17T10:33:31.612682+0000","last_peered":"2024-04-17T10:33:31.612682+0000","last_clean":"2024-04-17T10:33:31.612682+0000","last_became_active":"2024-04-17T10:33:29.492762+0000","last_became_peered":"2024-04-17T10:33:29.492762+0000","last_unstale":"2024-04-17T10:33:31.612682+0000","last_undegraded":"2024-04-17T10:33:31.612682+0000","last_fullsized":"2024-04-17T10:33:31.612682+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":115,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:33:28.456259+0000","last_clean_scrub_stamp":"2024-04-17T10:33:28.456259+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:00:01.006670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":350,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-04-17T10:33:31.609339+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:33:31.609339+0000","last_peered":"2024-04-17T10:33:31.609339+0000","last_clean":"2024-04-17T10:33:31.609339+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:33:31.609339+0000","last_undegraded":"2024-04-17T10:33:31.609339+0000","last_fullsized":"2024-04-17T10:33:31.609339+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574958,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51260,"kb_used_data":2032,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320580,"statfs":{"total":96636764160,"available":96584273920,"internally_reserved":0,"allocated":2080768,"data_stored":1278256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574956,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50676,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321164,"statfs":{"total":96636764160,"available":96584871936,"internally_reserved":0,"allocated":1474560,"data_stored":687694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574957,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46516,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325324,"statfs":{"total":96636764160,"available":96589131776,"internally_reserved":0,"allocated":1478656,"data_stored":687272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574957,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58984,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312856,"statfs":{"total":96636764160,"available":96576364544,"internally_reserved":0,"allocated":1470464,"data_stored":682032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574957,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54828,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317012,"statfs":{"total":96636764160,"available":96580620288,"internally_reserved":0,"allocated":1474560,"data_stored":682066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574958,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54824,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317016,"statfs":{"total":96636764160,"available":96580624384,"internally_reserved":0,"allocated":1470464,"data_stored":687112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50656,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321184,"statfs":{"total":96636764160,"available":96584892416,"internally_reserved":0,"allocated":1462272,"data_stored":680442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574957,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55412,"kb_used_data":2020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316428,"statfs":{"total":96636764160,"available":96580022272,"internally_reserved":0,"allocated":2068480,"data_stored":1277502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:33:35.004 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:33:35.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:33:35.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff643600640 1 Processor -- start 2024-04-17T10:33:35.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff643600640 1 -- start start 2024-04-17T10:33:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff643600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 0x7ff63c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff643600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c15cb30 con 0x7ff63c159440 2024-04-17T10:33:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff643600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63c0cd610 con 0x7ff63c071af0 2024-04-17T10:33:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff643600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0a8650 con 0x7ff63c15c580 2024-04-17T10:33:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff640c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff63c159440 0x7ff63c0713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59384/0 (socket says 172.21.15.67:59384) 2024-04-17T10:33:35.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.201+0000 7ff640c00640 1 -- 172.21.15.67:0/1189680781 learned_addr learned my addr 172.21.15.67:0/1189680781 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff641600640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 0x7ff63c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771953366 0 0) 0x7ff63c15cb30 con 0x7ff63c159440 2024-04-17T10:33:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff620003530 con 0x7ff63c159440 2024-04-17T10:33:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097172371 0 0) 0x7ff620003530 con 0x7ff63c159440 2024-04-17T10:33:35.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 msgr2=0x7ff63c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 0x7ff63c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 >> v1:172.21.15.86:6789/0 conn(0x7ff63c071af0 legacy=0x7ff63c15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0cd8f0 con 0x7ff63c159440 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7ff62c001680 con 0x7ff63c159440 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 >> v1:172.21.15.67:6789/0 conn(0x7ff63c159440 legacy=0x7ff63c0713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6200039f0 0x7ff620003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620004810 0x7ff620004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 0x7ff6200051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62000ba80 con 0x7ff620004810 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62000bc00 con 0x7ff6200039f0 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62000be00 con 0x7ff63c15c580 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff633e00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620004810 0x7ff620004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff640c00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6200039f0 0x7ff620003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff641600640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 0x7ff6200051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff633e00640 1 -- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 msgr2=0x7ff6200051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff633e00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff63c15c580 0x7ff6200051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff633e00640 1 -- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6200039f0 msgr2=0x7ff620003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff633e00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6200039f0 0x7ff620003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff633e00640 1 -- 172.21.15.67:0/1189680781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff620003530 con 0x7ff620004810 2024-04-17T10:33:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.205+0000 7ff640c00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6200039f0 0x7ff620003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:35.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff633e00640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620004810 0x7ff620004c90 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0a8510 tx=0x7ff62401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b975d4b6eb2c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:35.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6240013a0 con 0x7ff620004810 2024-04-17T10:33:35.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6240052f0 con 0x7ff620004810 2024-04-17T10:33:35.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff63be00640 1 -- 172.21.15.67:0/1189680781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6240a4e60 con 0x7ff620004810 2024-04-17T10:33:35.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- 172.21.15.67:0/1189680781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620004810 msgr2=0x7ff620004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 --2- 172.21.15.67:0/1189680781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff620004810 0x7ff620004c90 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0a8510 tx=0x7ff62401c3a0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff641600640 1 -- 172.21.15.67:0/1189680781 reap_dead start 2024-04-17T10:33:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- 172.21.15.67:0/1189680781 shutdown_connections 2024-04-17T10:33:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- 172.21.15.67:0/1189680781 >> 172.21.15.67:0/1189680781 conn(0x7ff63c0b1740 msgr2=0x7ff63c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- 172.21.15.67:0/1189680781 shutdown_connections 2024-04-17T10:33:35.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- 172.21.15.67:0/1189680781 wait complete. 2024-04-17T10:33:35.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 Processor -- start 2024-04-17T10:33:35.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- start start 2024-04-17T10:33:35.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff620004810 0x7ff63c1542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff63c154af0 0x7ff63c154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63c0cc250 0x7ff63c0cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c159820 con 0x7ff63c0cc250 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c15ce10 con 0x7ff63c154af0 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.209+0000 7ff643600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c0ce0f0 con 0x7ff620004810 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63c0cc250 0x7ff63c0cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63c0cc250 0x7ff63c0cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42468/0 (socket says 172.21.15.67:42468) 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 -- 172.21.15.67:0/554629008 learned_addr learned my addr 172.21.15.67:0/554629008 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff633e00640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff63c154af0 0x7ff63c154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff640c00640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff620004810 0x7ff63c1542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 -- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff620004810 msgr2=0x7ff63c1542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff620004810 0x7ff63c1542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 -- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff63c154af0 msgr2=0x7ff63c154f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff63c154af0 0x7ff63c154f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 -- 172.21.15.67:0/554629008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c0a9070 con 0x7ff63c0cc250 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff640c00640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff620004810 0x7ff63c1542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff641600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63c0cc250 0x7ff63c0cc6e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0b4d50 tx=0x7ff6340115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff639600640 1 -- 172.21.15.67:0/554629008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6340039a0 con 0x7ff63c0cc250 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff639600640 1 -- 172.21.15.67:0/554629008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63401b070 con 0x7ff63c0cc250 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c0a9350 con 0x7ff63c0cc250 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff639600640 1 -- 172.21.15.67:0/554629008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff634003b80 con 0x7ff63c0cc250 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c1f2fe0 con 0x7ff63c0cc250 2024-04-17T10:33:35.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c1f31e0 con 0x7ff63c0cc250 2024-04-17T10:33:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.213+0000 7ff639600640 1 -- 172.21.15.67:0/554629008 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff634004030 con 0x7ff63c0cc250 2024-04-17T10:33:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.217+0000 7ff639600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff618066f90 0x7ff618069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.217+0000 7ff639600640 1 -- 172.21.15.67:0/554629008 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff634085a30 con 0x7ff63c0cc250 2024-04-17T10:33:35.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.217+0000 7ff640c00640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff618066f90 0x7ff618069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.217+0000 7ff640c00640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff618066f90 0x7ff618069450 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff62c001020 tx=0x7ff62c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:35.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.217+0000 7ff639600640 1 -- 172.21.15.67:0/554629008 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff63c1f31e0 con 0x7ff63c0cc250 2024-04-17T10:33:35.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.441+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff63c06c3e0 con 0x7ff63c0cc250 2024-04-17T10:33:35.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff639600640 1 -- 172.21.15.67:0/554629008 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff634056810 con 0x7ff63c0cc250 2024-04-17T10:33:35.436 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:33:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:33:35.436 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:33:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:33:35.437 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:33:35.437 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff618066f90 msgr2=0x7ff618069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff618066f90 0x7ff618069450 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7ff62c001020 tx=0x7ff62c009040 comp rx=0 tx=0).stop 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63c0cc250 msgr2=0x7ff63c0cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63c0cc250 0x7ff63c0cc6e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0b4d50 tx=0x7ff6340115a0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 shutdown_connections 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff618066f90 0x7ff618069450 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff63c0cc250 0x7ff63c0cc6e0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff63c154af0 0x7ff63c154f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 --2- 172.21.15.67:0/554629008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff620004810 0x7ff63c1542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 >> 172.21.15.67:0/554629008 conn(0x7ff63c0b1740 msgr2=0x7ff63c15cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:35.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 shutdown_connections 2024-04-17T10:33:35.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.445+0000 7ff643600640 1 -- 172.21.15.67:0/554629008 wait complete. 2024-04-17T10:33:35.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbb05600640 1 Processor -- start 2024-04-17T10:33:35.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbb05600640 1 -- start start 2024-04-17T10:33:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbb05600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb000cf0c0 0x7fbb001441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbb05600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0015ae40 con 0x7fbb0006eac0 2024-04-17T10:33:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbb05600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb00071440 con 0x7fbb0006e6e0 2024-04-17T10:33:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbb05600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb00071720 con 0x7fbb000cf0c0 2024-04-17T10:33:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbaff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbb000cf0c0 0x7fbb001441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbaf7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb0006eac0 0x7fbb00143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59396/0 (socket says 172.21.15.67:59396) 2024-04-17T10:33:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbaf7e00640 1 -- 172.21.15.67:0/3332701405 learned_addr learned my addr 172.21.15.67:0/3332701405 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbaff400640 1 -- 172.21.15.67:0/3332701405 >> v1:172.21.15.86:6789/0 conn(0x7fbb0006e6e0 legacy=0x7fbb000ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbaff400640 1 -- 172.21.15.67:0/3332701405 >> v1:172.21.15.67:6789/0 conn(0x7fbb0006eac0 legacy=0x7fbb00143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbafe000640 1 -- 172.21.15.67:0/3332701405 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951980011 0 0) 0x7fbb0015ae40 con 0x7fbb0006eac0 2024-04-17T10:33:35.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.609+0000 7fbaff400640 1 -- 172.21.15.67:0/3332701405 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb00071a00 con 0x7fbb000cf0c0 2024-04-17T10:33:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbaff400640 1 --2- 172.21.15.67:0/3332701405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb000cf0c0 0x7fbb001441f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009d90 tx=0x7fbaf801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d9428a15da0d05d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbafe000640 1 -- 172.21.15.67:0/3332701405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80b2070 con 0x7fbb000cf0c0 2024-04-17T10:33:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbafe000640 1 -- 172.21.15.67:0/3332701405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf80a5970 con 0x7fbb000cf0c0 2024-04-17T10:33:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbafe000640 1 -- 172.21.15.67:0/3332701405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80a5d40 con 0x7fbb000cf0c0 2024-04-17T10:33:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbb05600640 1 -- 172.21.15.67:0/3332701405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb000cf0c0 msgr2=0x7fbb001441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbb05600640 1 --2- 172.21.15.67:0/3332701405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb000cf0c0 0x7fbb001441f0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009d90 tx=0x7fbaf801c8b0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbb05600640 1 -- 172.21.15.67:0/3332701405 shutdown_connections 2024-04-17T10:33:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbb05600640 1 --2- 172.21.15.67:0/3332701405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb000cf0c0 0x7fbb001441f0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbb05600640 1 -- 172.21.15.67:0/3332701405 >> 172.21.15.67:0/3332701405 conn(0x7fbb0006ac90 msgr2=0x7fbb0006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbb05600640 1 -- 172.21.15.67:0/3332701405 shutdown_connections 2024-04-17T10:33:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.613+0000 7fbb05600640 1 -- 172.21.15.67:0/3332701405 wait complete. 2024-04-17T10:33:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 Processor -- start 2024-04-17T10:33:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 -- start start 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0006e6e0 0x7fbb0014cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0006eac0 0x7fbb000b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb000cf0c0 0x7fbb0013d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb001680b0 con 0x7fbb0006eac0 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb00167f30 con 0x7fbb0006e6e0 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb00168230 con 0x7fbb000cf0c0 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaf7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0006eac0 0x7fbb000b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbafea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0006e6e0 0x7fbb0014cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaf7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0006eac0 0x7fbb000b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42486/0 (socket says 172.21.15.67:42486) 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaf7e00640 1 -- 172.21.15.67:0/1895033271 learned_addr learned my addr 172.21.15.67:0/1895033271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaff400640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb000cf0c0 0x7fbb0013d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbafea00640 1 -- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb000cf0c0 msgr2=0x7fbb0013d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbafea00640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb000cf0c0 0x7fbb0013d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbafea00640 1 -- 172.21.15.67:0/1895033271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0006eac0 msgr2=0x7fbb000b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbafea00640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0006eac0 0x7fbb000b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbafea00640 1 -- 172.21.15.67:0/1895033271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb000b1e20 con 0x7fbb0006e6e0 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaf7e00640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0006eac0 0x7fbb000b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaff400640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb000cf0c0 0x7fbb0013d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbafea00640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0006e6e0 0x7fbb0014cf40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0000f70 tx=0x7fbaf000c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaf6a00640 1 -- 172.21.15.67:0/1895033271 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0017070 con 0x7fbb0006e6e0 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaf6a00640 1 -- 172.21.15.67:0/1895033271 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf00052d0 con 0x7fbb0006e6e0 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbaf6a00640 1 -- 172.21.15.67:0/1895033271 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf00054c0 con 0x7fbb0006e6e0 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.617+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0014d480 con 0x7fbb0006e6e0 2024-04-17T10:33:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.621+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb000b1b40 con 0x7fbb0006e6e0 2024-04-17T10:33:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.621+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb00072d50 con 0x7fbb0006e6e0 2024-04-17T10:33:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.621+0000 7fbaf6a00640 1 -- 172.21.15.67:0/1895033271 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbaf001a070 con 0x7fbb0006e6e0 2024-04-17T10:33:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.621+0000 7fbaf6a00640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbadc066f60 0x7fbadc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.621+0000 7fbaf6a00640 1 -- 172.21.15.67:0/1895033271 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbaf0084540 con 0x7fbb0006e6e0 2024-04-17T10:33:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.621+0000 7fbaf7e00640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbadc066f60 0x7fbadc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.625+0000 7fbaf7e00640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbadc066f60 0x7fbadc069420 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbae8007b20 tx=0x7fbae8007a70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.625+0000 7fbaf6a00640 1 -- 172.21.15.67:0/1895033271 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbaf0054cd0 con 0x7fbb0006e6e0 2024-04-17T10:33:35.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb000b2c60 con 0x7fbb0006e6e0 2024-04-17T10:33:35.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbaf6a00640 1 -- 172.21.15.67:0/1895033271 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fbaf0058d70 con 0x7fbb0006e6e0 2024-04-17T10:33:35.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:35.817 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:33:35.817 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:33:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbadc066f60 msgr2=0x7fbadc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbadc066f60 0x7fbadc069420 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbae8007b20 tx=0x7fbae8007a70 comp rx=0 tx=0).stop 2024-04-17T10:33:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0006e6e0 msgr2=0x7fbb0014cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0006e6e0 0x7fbb0014cf40 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0000f70 tx=0x7fbaf000c890 comp rx=0 tx=0).stop 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 shutdown_connections 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbadc066f60 0x7fbadc069420 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb000cf0c0 0x7fbb0013d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb0006eac0 0x7fbb000b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 --2- 172.21.15.67:0/1895033271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0006e6e0 0x7fbb0014cf40 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 >> 172.21.15.67:0/1895033271 conn(0x7fbb0006ac90 msgr2=0x7fbb000ad9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 shutdown_connections 2024-04-17T10:33:35.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:35.825+0000 7fbb05600640 1 -- 172.21.15.67:0/1895033271 wait complete. 2024-04-17T10:33:36.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:33:29.865690+0000', 'modified': '2024-04-17T10:33:32.930770+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8670}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8670': {'gid': 8670, 'name': 'c', 'rank': 0, 'incarnation': 78, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6834/397092831', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6833', 'nonce': 397092831}, {'type': 'v1', 'addr': '172.21.15.67:6834', 'nonce': 397092831}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:33:36.241 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:33:36.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T10:33:36.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa752800640 1 Processor -- start 2024-04-17T10:33:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa752800640 1 -- start start 2024-04-17T10:33:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa752800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa74c144aa0 0x7fa74c0d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa752800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa74c071840 0x7fa74c0d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa752800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74c1456d0 con 0x7fa74c1446c0 2024-04-17T10:33:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa752800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c0b51a0 con 0x7fa74c144aa0 2024-04-17T10:33:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa752800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa74c0d1390 con 0x7fa74c071840 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa742a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa74c144aa0 0x7fa74c0d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa742a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa74c144aa0 0x7fa74c0d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42498/0 (socket says 172.21.15.67:42498) 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa742a00640 1 -- 172.21.15.67:0/1390056120 learned_addr learned my addr 172.21.15.67:0/1390056120 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa74be00640 1 --2- 172.21.15.67:0/1390056120 >> v2:172.21.15.86:3300/0 conn(0x7fa74c071840 0x7fa74c0d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa742a00640 1 -- 172.21.15.67:0/1390056120 >> v2:172.21.15.86:3300/0 conn(0x7fa74c071840 msgr2=0x7fa74c0d0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa742a00640 1 --2- 172.21.15.67:0/1390056120 >> v2:172.21.15.86:3300/0 conn(0x7fa74c071840 0x7fa74c0d0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa742a00640 1 -- 172.21.15.67:0/1390056120 >> v1:172.21.15.86:6789/0 conn(0x7fa74c1446c0 legacy=0x7fa74c144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa742a00640 1 -- 172.21.15.67:0/1390056120 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c0b6640 con 0x7fa74c144aa0 2024-04-17T10:33:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.393+0000 7fa74be00640 1 --2- 172.21.15.67:0/1390056120 >> v2:172.21.15.86:3300/0 conn(0x7fa74c071840 0x7fa74c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:36.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa742a00640 1 --2- 172.21.15.67:0/1390056120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c144aa0 0x7fa74c0d0910 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fa74c06f0e0 tx=0x7fa73401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bb8320ca513123 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa74aa00640 1 -- 172.21.15.67:0/1390056120 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340aa430 con 0x7fa74c144aa0 2024-04-17T10:33:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa74aa00640 1 -- 172.21.15.67:0/1390056120 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7340af070 con 0x7fa74c144aa0 2024-04-17T10:33:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa74aa00640 1 -- 172.21.15.67:0/1390056120 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340aa7f0 con 0x7fa74c144aa0 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 -- 172.21.15.67:0/1390056120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c144aa0 msgr2=0x7fa74c0d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 --2- 172.21.15.67:0/1390056120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c144aa0 0x7fa74c0d0910 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fa74c06f0e0 tx=0x7fa73401c4d0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 -- 172.21.15.67:0/1390056120 shutdown_connections 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 --2- 172.21.15.67:0/1390056120 >> v2:172.21.15.86:3300/0 conn(0x7fa74c071840 0x7fa74c0d0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 --2- 172.21.15.67:0/1390056120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c144aa0 0x7fa74c0d0910 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 -- 172.21.15.67:0/1390056120 >> 172.21.15.67:0/1390056120 conn(0x7fa74c0b34d0 msgr2=0x7fa74c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 -- 172.21.15.67:0/1390056120 shutdown_connections 2024-04-17T10:33:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.397+0000 7fa752800640 1 -- 172.21.15.67:0/1390056120 wait complete. 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 Processor -- start 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 -- start start 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c071840 0x7fa74c1fb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa74c1446c0 0x7fa74c143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa74c144aa0 0x7fa74c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c0b5e20 con 0x7fa74c071840 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c15c0d0 con 0x7fa74c1446c0 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c0b5fa0 con 0x7fa74c144aa0 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c071840 0x7fa74c1fb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c071840 0x7fa74c1fb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42506/0 (socket says 172.21.15.67:42506) 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 -- 172.21.15.67:0/2432112243 learned_addr learned my addr 172.21.15.67:0/2432112243 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa742a00640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa74c1446c0 0x7fa74c143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74be00640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa74c144aa0 0x7fa74c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 -- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa74c144aa0 msgr2=0x7fa74c1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa74c144aa0 0x7fa74c1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 -- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa74c1446c0 msgr2=0x7fa74c143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa74c1446c0 0x7fa74c143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 -- 172.21.15.67:0/2432112243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c130be0 con 0x7fa74c071840 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74be00640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa74c144aa0 0x7fa74c1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa74b400640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c071840 0x7fa74c1fb070 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c450 tx=0x7fa73c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa743e00640 1 -- 172.21.15.67:0/2432112243 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c016020 con 0x7fa74c071840 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa743e00640 1 -- 172.21.15.67:0/2432112243 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c0052d0 con 0x7fa74c071840 2024-04-17T10:33:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa743e00640 1 -- 172.21.15.67:0/2432112243 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c007260 con 0x7fa74c071840 2024-04-17T10:33:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa74c130ec0 con 0x7fa74c071840 2024-04-17T10:33:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.401+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa74c1568e0 con 0x7fa74c071840 2024-04-17T10:33:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.405+0000 7fa743e00640 1 -- 172.21.15.67:0/2432112243 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa73c007400 con 0x7fa74c071840 2024-04-17T10:33:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.405+0000 7fa743e00640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f90 0x7fa728069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.405+0000 7fa743e00640 1 -- 172.21.15.67:0/2432112243 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa73c084b60 con 0x7fa74c071840 2024-04-17T10:33:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.405+0000 7fa755c9e000 1 Processor -- start 2024-04-17T10:33:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.405+0000 7fa742a00640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f90 0x7fa728069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.405+0000 7fa755c9e000 1 -- start start 2024-04-17T10:33:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f90 0x7fa728069450 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa7340ae000 tx=0x7fa7340a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f6240 0x55fb633ab9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb633ac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb632935f0 con 0x55fb633f6240 2024-04-17T10:33:36.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb631fada0 con 0x55fb633abf30 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb63183150 con 0x55fb633f5920 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f6240 0x55fb633ab9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f6240 0x55fb633ab9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42510/0 (socket says 172.21.15.67:42510) 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 -- 172.21.15.67:0/1003701445 learned_addr learned my addr 172.21.15.67:0/1003701445 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa74be00640 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb633ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa74b400640 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 -- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 msgr2=0x55fb633f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 -- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 msgr2=0x55fb633ac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb633ac310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 -- 172.21.15.67:0/1003701445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb6319cca0 con 0x55fb633f6240 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa742a00640 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f6240 0x55fb633ab9f0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fa7340ac040 tx=0x7fa734004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d03da1adbba42d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa73be00640 1 -- 172.21.15.67:0/1003701445 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340c8630 con 0x55fb633f6240 2024-04-17T10:33:36.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa73be00640 1 -- 172.21.15.67:0/1003701445 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7340af040 con 0x55fb633f6240 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa73be00640 1 -- 172.21.15.67:0/1003701445 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340c89d0 con 0x55fb633f6240 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- 172.21.15.67:0/1003701445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f6240 msgr2=0x55fb633ab9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f6240 0x55fb633ab9f0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fa7340ac040 tx=0x7fa734004740 comp rx=0 tx=0).stop 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- 172.21.15.67:0/1003701445 shutdown_connections 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb633ac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f6240 0x55fb633ab9f0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1003701445 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- 172.21.15.67:0/1003701445 >> 172.21.15.67:0/1003701445 conn(0x55fb633b22a0 msgr2=0x55fb633aa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- 172.21.15.67:0/1003701445 shutdown_connections 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- 172.21.15.67:0/1003701445 wait complete. 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 -- start start 2024-04-17T10:33:36.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.409+0000 7fa755c9e000 1 Processor -- start 2024-04-17T10:33:36.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb633fcc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb633afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb631b5990 con 0x55fb633abf30 2024-04-17T10:33:36.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb631f6e40 con 0x55fb633f6240 2024-04-17T10:33:36.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb63186900 con 0x55fb633f5920 2024-04-17T10:33:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 10 client.0 _ll_register_callbacks cb 0x55fb632ae1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb633fcc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb633fcc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42518/0 (socket says 172.21.15.67:42518) 2024-04-17T10:33:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 -- 172.21.15.67:0/2672215202 learned_addr learned my addr 172.21.15.67:0/2672215202 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa742a00640 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74be00640 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb633afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 -- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 msgr2=0x55fb633fd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633fd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 -- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 msgr2=0x55fb633afaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb633afaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 -- 172.21.15.67:0/2672215202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb631b3630 con 0x55fb633abf30 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa74b400640 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb633fcc80 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00ce10 tx=0x7fa73c0559d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9717c661d765e4db server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 1 -- 172.21.15.67:0/2672215202 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c05d9e0 con 0x55fb633abf30 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 1 -- 172.21.15.67:0/2672215202 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa73c051940 con 0x55fb633abf30 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7340af040 con 0x55fb633abf30 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 1 -- 172.21.15.67:0/2672215202 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa73c073170 con 0x55fb633abf30 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 10 client.8802 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55fb631833a0 con 0x55fb633abf30 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 1 -- 172.21.15.67:0/2672215202 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7340af040 con 0x55fb633abf30 2024-04-17T10:33:36.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 1 -- 172.21.15.67:0/2672215202 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa73c054720 con 0x55fb633abf30 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 10 client.8802 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 10 client.8802 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 20 client.8802 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb633b1160 0x55fb63406ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 4 client.8802 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb63196a50 con 0x55fb633b1160 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa742a00640 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb633b1160 0x55fb63406ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa742a00640 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb633b1160 0x55fb63406ce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfd2bb4248bffaad server_cookie=a3c8165dbb4b04b7 in_seq=0 out_seq=0 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 10 client.8802 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 1 -- 172.21.15.67:0/2672215202 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fb63196a50 con 0x55fb633b1160 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.413+0000 7fa738c00640 10 client.8802 handle_command_reply: tid=0 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 client.8802 shutdown 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 msgr2=0x55fb633fcc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb633fcc80 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00ce10 tx=0x7fa73c0559d0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 shutdown_connections 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb633b1160 0x55fb63406ce0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb633afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb633fd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa738c00640 0 client.8802 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2672215202 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb633fcc80 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 >> 172.21.15.67:0/2672215202 conn(0x55fb633b22a0 msgr2=0x55fb633aa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 shutdown_connections 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 1 -- 172.21.15.67:0/2672215202 wait complete. 2024-04-17T10:33:36.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.417+0000 7fa755c9e000 20 client.8802 trim_cache size 0 max 16384 2024-04-17T10:33:36.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 Processor -- start 2024-04-17T10:33:36.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- start start 2024-04-17T10:33:36.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb634513a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb634518e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb63451e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c202310 con 0x55fb633abf30 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c2017c0 con 0x55fb633f6240 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74c201d40 con 0x55fb633f5920 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb634513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb634513a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42526/0 (socket says 172.21.15.67:42526) 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 -- 172.21.15.67:0/2513554785 learned_addr learned my addr 172.21.15.67:0/2513554785 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa742a00640 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb634518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74be00640 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb63451e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 -- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 msgr2=0x55fb634518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb634518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 -- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 msgr2=0x55fb63451e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb63451e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 -- 172.21.15.67:0/2513554785 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb633f6070 con 0x55fb633abf30 2024-04-17T10:33:36.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74be00640 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb63451e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa74b400640 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb634513a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fa73c005630 tx=0x7fa73c06c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1009923f889b16f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa73b400640 1 -- 172.21.15.67:0/2513554785 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c05d9e0 con 0x55fb633abf30 2024-04-17T10:33:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa73b400640 1 -- 172.21.15.67:0/2513554785 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c081070 con 0x55fb633abf30 2024-04-17T10:33:36.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa73b400640 1 -- 172.21.15.67:0/2513554785 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73c054720 con 0x55fb633abf30 2024-04-17T10:33:36.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- 172.21.15.67:0/2513554785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 msgr2=0x55fb634513a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb634513a0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fa73c005630 tx=0x7fa73c06c000 comp rx=0 tx=0).stop 2024-04-17T10:33:36.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- 172.21.15.67:0/2513554785 shutdown_connections 2024-04-17T10:33:36.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633f6240 0x55fb63451e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f5920 0x55fb634518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 --2- 172.21.15.67:0/2513554785 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633abf30 0x55fb634513a0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- 172.21.15.67:0/2513554785 >> 172.21.15.67:0/2513554785 conn(0x55fb633b22a0 msgr2=0x55fb633af220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- 172.21.15.67:0/2513554785 shutdown_connections 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- 172.21.15.67:0/2513554785 wait complete. 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 -- start start 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.433+0000 7fa755c9e000 1 Processor -- start 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb63439ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f5920 0x55fb6343a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f6240 0x55fb6343ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb631ff3d0 con 0x55fb633f5920 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb632b6d50 con 0x55fb633abf30 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb63195980 con 0x55fb633f6240 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 10 client.0 _ll_register_callbacks cb 0x7fa734006ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f5920 0x55fb6343a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f5920 0x55fb6343a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42530/0 (socket says 172.21.15.67:42530) 2024-04-17T10:33:36.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 -- 172.21.15.67:0/1901715351 learned_addr learned my addr 172.21.15.67:0/1901715351 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa74b400640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb63439ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa74be00640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f6240 0x55fb6343ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 -- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f6240 msgr2=0x55fb6343ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f6240 0x55fb6343ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 -- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 msgr2=0x55fb63439ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb63439ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 -- 172.21.15.67:0/1901715351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb6327c600 con 0x55fb633f5920 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa74be00640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f6240 0x55fb6343ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa742a00640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f5920 0x55fb6343a940 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fa73401a100 tx=0x7fa7340046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c27ba477805ae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 1 -- 172.21.15.67:0/1901715351 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340d61a0 con 0x55fb633f5920 2024-04-17T10:33:36.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 1 -- 172.21.15.67:0/1901715351 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa7340cd040 con 0x55fb633f5920 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa73c081070 con 0x55fb633f5920 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 1 -- 172.21.15.67:0/1901715351 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7340e3050 con 0x55fb633f5920 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 10 client.8808 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55fb6318f9b0 con 0x55fb633f5920 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 1 -- 172.21.15.67:0/1901715351 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73c081070 con 0x55fb633f5920 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 1 -- 172.21.15.67:0/1901715351 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa7340e6020 con 0x55fb633f5920 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 10 client.8808 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 10 client.8808 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 20 client.8808 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb6343b3c0 0x55fb6343b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 4 client.8808 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fb632082e0 con 0x55fb6343b3c0 2024-04-17T10:33:36.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa74b400640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb6343b3c0 0x55fb6343b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa74b400640 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb6343b3c0 0x55fb6343b7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f07b5d96a493eda server_cookie=cf1810a3ad7bdead in_seq=0 out_seq=0 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 10 client.8808 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 1 -- 172.21.15.67:0/1901715351 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55fb632082e0 con 0x55fb6343b3c0 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 10 client.8808 handle_command_reply: tid=0 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 client.8808 shutdown 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f5920 msgr2=0x55fb6343a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f5920 0x55fb6343a940 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fa73401a100 tx=0x7fa7340046d0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 shutdown_connections 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55fb6343b3c0 0x55fb6343b7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55fb633f6240 0x55fb6343ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fb633f5920 0x55fb6343a940 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa738c00640 0 client.8808 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 --2- 172.21.15.67:0/1901715351 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55fb633abf30 0x55fb63439ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.437+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 >> 172.21.15.67:0/1901715351 conn(0x55fb633b22a0 msgr2=0x55fb633fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 shutdown_connections 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa755c9e000 1 -- 172.21.15.67:0/1901715351 wait complete. 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa755c9e000 20 client.8808 trim_cache size 0 max 16384 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 8670, 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:33:36.436 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:33:36.437 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:33:36.437 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 5.9271857450000001, 2024-04-17T10:33:36.437 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 82, 2024-04-17T10:33:36.437 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 118, 2024-04-17T10:33:36.437 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 118, 2024-04-17T10:33:36.437 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 10.830308307999999 2024-04-17T10:33:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f90 msgr2=0x7fa728069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f90 0x7fa728069450 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa7340ae000 tx=0x7fa7340a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c071840 msgr2=0x7fa74c1fb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c071840 0x7fa74c1fb070 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fa73c00c450 tx=0x7fa73c00c910 comp rx=0 tx=0).stop 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 shutdown_connections 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f90 0x7fa728069450 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa74c144aa0 0x7fa74c1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa74c1446c0 0x7fa74c143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 --2- 172.21.15.67:0/2432112243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa74c071840 0x7fa74c1fb070 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 >> 172.21.15.67:0/2432112243 conn(0x7fa74c0b34d0 msgr2=0x7fa74c06d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 shutdown_connections 2024-04-17T10:33:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.441+0000 7fa752800640 1 -- 172.21.15.67:0/2432112243 wait complete. 2024-04-17T10:33:36.440 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T10:33:36.449+0000 7f7825c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:33:36.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:33:36.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.601+0000 7fcd81600640 1 Processor -- start 2024-04-17T10:33:36.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.601+0000 7fcd81600640 1 -- start start 2024-04-17T10:33:36.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.601+0000 7fcd81600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd7c099a50 0x7fcd7c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd81600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c0cde20 con 0x7fcd7c09a740 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd81600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7c0cf300 con 0x7fcd7c0d03a0 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd81600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7c0cf5e0 con 0x7fcd7c099a50 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd7c099a50 0x7fcd7c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd80c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcd7c09a740 0x7fcd7c0cc1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59402/0 (socket says 172.21.15.67:59402) 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd7c099a50 0x7fcd7c15c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42542/0 (socket says 172.21.15.67:42542) 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7be00640 1 -- 172.21.15.67:0/2414107466 learned_addr learned my addr 172.21.15.67:0/2414107466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7be00640 1 -- 172.21.15.67:0/2414107466 >> v1:172.21.15.86:6789/0 conn(0x7fcd7c0d03a0 legacy=0x7fcd7c099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7be00640 1 -- 172.21.15.67:0/2414107466 >> v1:172.21.15.67:6789/0 conn(0x7fcd7c09a740 legacy=0x7fcd7c0cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7be00640 1 -- 172.21.15.67:0/2414107466 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c0cf8c0 con 0x7fcd7c099a50 2024-04-17T10:33:36.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7b400640 1 -- 172.21.15.67:0/2414107466 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575538514 0 0) 0x7fcd7c0cde20 con 0x7fcd7c09a740 2024-04-17T10:33:36.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7be00640 1 --2- 172.21.15.67:0/2414107466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c15c7a0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009870 tx=0x7fcd6c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a21acb8e91eb7629 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7b400640 1 -- 172.21.15.67:0/2414107466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0b2070 con 0x7fcd7c099a50 2024-04-17T10:33:36.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7b400640 1 -- 172.21.15.67:0/2414107466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c0a9070 con 0x7fcd7c099a50 2024-04-17T10:33:36.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.605+0000 7fcd7b400640 1 -- 172.21.15.67:0/2414107466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0ad880 con 0x7fcd7c099a50 2024-04-17T10:33:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- 172.21.15.67:0/2414107466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 msgr2=0x7fcd7c15c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 --2- 172.21.15.67:0/2414107466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c15c7a0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009870 tx=0x7fcd6c01c560 comp rx=0 tx=0).stop 2024-04-17T10:33:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- 172.21.15.67:0/2414107466 shutdown_connections 2024-04-17T10:33:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 --2- 172.21.15.67:0/2414107466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c15c7a0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- 172.21.15.67:0/2414107466 >> 172.21.15.67:0/2414107466 conn(0x7fcd7c0d1690 msgr2=0x7fcd7c0d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- 172.21.15.67:0/2414107466 shutdown_connections 2024-04-17T10:33:36.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- 172.21.15.67:0/2414107466 wait complete. 2024-04-17T10:33:36.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 Processor -- start 2024-04-17T10:33:36.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- start start 2024-04-17T10:33:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd7c09a740 0x7fcd7c151af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd7c0d03a0 0x7fcd7c0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c169a50 con 0x7fcd7c099a50 2024-04-17T10:33:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c1698d0 con 0x7fcd7c09a740 2024-04-17T10:33:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd81600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c169bd0 con 0x7fcd7c0d03a0 2024-04-17T10:33:36.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c0b7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42556/0 (socket says 172.21.15.67:42556) 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 -- 172.21.15.67:0/2371491968 learned_addr learned my addr 172.21.15.67:0/2371491968 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd73400640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd7c09a740 0x7fcd7c151af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd80c00640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd7c0d03a0 0x7fcd7c0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 -- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd7c0d03a0 msgr2=0x7fcd7c0c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd7c0d03a0 0x7fcd7c0c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 -- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd7c09a740 msgr2=0x7fcd7c151af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd7c09a740 0x7fcd7c151af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.609+0000 7fcd7be00640 1 -- 172.21.15.67:0/2371491968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c152030 con 0x7fcd7c099a50 2024-04-17T10:33:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd7be00640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c0b7000 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0b0040 tx=0x7fcd6c01ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd78c00640 1 -- 172.21.15.67:0/2371491968 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0b2040 con 0x7fcd7c099a50 2024-04-17T10:33:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd78c00640 1 -- 172.21.15.67:0/2371491968 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c0adcd0 con 0x7fcd7c099a50 2024-04-17T10:33:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd78c00640 1 -- 172.21.15.67:0/2371491968 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c0b8680 con 0x7fcd7c099a50 2024-04-17T10:33:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c157930 con 0x7fcd7c099a50 2024-04-17T10:33:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c157ef0 con 0x7fcd7c099a50 2024-04-17T10:33:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c074dc0 con 0x7fcd7c099a50 2024-04-17T10:33:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd78c00640 1 -- 172.21.15.67:0/2371491968 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcd6c0c0030 con 0x7fcd7c099a50 2024-04-17T10:33:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd78c00640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd58066f60 0x7fcd58069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd78c00640 1 -- 172.21.15.67:0/2371491968 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcd6c11ea20 con 0x7fcd7c099a50 2024-04-17T10:33:36.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.613+0000 7fcd73400640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd58066f60 0x7fcd58069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:36.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.617+0000 7fcd73400640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd58066f60 0x7fcd58069420 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c151790 tx=0x7fcd64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:36.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.617+0000 7fcd78c00640 1 -- 172.21.15.67:0/2371491968 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcd6c0c7060 con 0x7fcd7c099a50 2024-04-17T10:33:36.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.813+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcd7c0d24a0 con 0x7fcd7c099a50 2024-04-17T10:33:36.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.813+0000 7fcd78c00640 1 -- 172.21.15.67:0/2371491968 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v82) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcd7c152030 con 0x7fcd7c099a50 2024-04-17T10:33:36.809 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd58066f60 msgr2=0x7fcd58069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd58066f60 0x7fcd58069420 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcd7c151790 tx=0x7fcd64009040 comp rx=0 tx=0).stop 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 msgr2=0x7fcd7c0b7000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c0b7000 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c0b0040 tx=0x7fcd6c01ada0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 shutdown_connections 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcd58066f60 0x7fcd58069420 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd7c0d03a0 0x7fcd7c0c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd7c09a740 0x7fcd7c151af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 --2- 172.21.15.67:0/2371491968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd7c099a50 0x7fcd7c0b7000 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 >> 172.21.15.67:0/2371491968 conn(0x7fcd7c0d1690 msgr2=0x7fcd7c0af780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:36.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 shutdown_connections 2024-04-17T10:33:36.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:36.817+0000 7fcd81600640 1 -- 172.21.15.67:0/2371491968 wait complete. 2024-04-17T10:33:36.827 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:33:36.828 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:33:36.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:36.828 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:33:36.837 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:33:36.838 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41930sec preferred_lft 41930sec 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:33:36.839 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:33:36.840 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:33:36.841 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:33:36.841 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:33:36.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:36.842 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:33:36.901 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:33:36.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:36.903 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:33:36.959 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:33:36.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:36.960 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:33:36.960 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:36.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:33:36.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:33:36.960 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:33:36.960 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:37.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:37.053 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:33:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:33:37.061 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:33:37.062 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:33:37.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:37.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:33:37.127 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:33:37.127 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:33:37.179 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:33:37.180 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:33:37.180 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:33:37.230 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:33:37.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:37.231 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:33:37.320 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:33:37.320 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:33:37.332 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:33:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:33:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:33:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:33:37.333 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:33:37.336 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:33:37.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:37.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:33:37.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:37.358 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:33:37.419 INFO:teuthology.orchestra.run.smithi067.stdout:client7352 2024-04-17T10:33:37.420 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:33:37.420 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:33:37.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:37.472 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:37.472 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:37.472 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.0 2024-04-17T10:33:37.472 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:37.472 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:37.472 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:37.472 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:37.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184699 s, 56.8 MB/s 2024-04-17T10:33:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.581 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02096 s, 50.0 MB/s 2024-04-17T10:33:37.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.599 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155634 s, 67.4 MB/s 2024-04-17T10:33:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172581 s, 60.8 MB/s 2024-04-17T10:33:37.637 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.637 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.637 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170375 s, 61.5 MB/s 2024-04-17T10:33:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.654 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153626 s, 68.3 MB/s 2024-04-17T10:33:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.670 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146567 s, 71.5 MB/s 2024-04-17T10:33:37.686 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.686 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.687 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139503 s, 75.2 MB/s 2024-04-17T10:33:37.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.702 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134311 s, 78.1 MB/s 2024-04-17T10:33:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.719 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157487 s, 66.6 MB/s 2024-04-17T10:33:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.737 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160962 s, 65.1 MB/s 2024-04-17T10:33:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169239 s, 62.0 MB/s 2024-04-17T10:33:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.772 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150657 s, 69.6 MB/s 2024-04-17T10:33:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179847 s, 58.3 MB/s 2024-04-17T10:33:37.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146695 s, 71.5 MB/s 2024-04-17T10:33:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156138 s, 67.2 MB/s 2024-04-17T10:33:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.844 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155639 s, 67.4 MB/s 2024-04-17T10:33:37.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.863 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176213 s, 59.5 MB/s 2024-04-17T10:33:37.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.881 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015502 s, 67.6 MB/s 2024-04-17T10:33:37.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.905 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218438 s, 48.0 MB/s 2024-04-17T10:33:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172577 s, 60.8 MB/s 2024-04-17T10:33:37.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.942 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164043 s, 63.9 MB/s 2024-04-17T10:33:37.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175212 s, 59.8 MB/s 2024-04-17T10:33:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159655 s, 65.7 MB/s 2024-04-17T10:33:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:37.996 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149274 s, 70.2 MB/s 2024-04-17T10:33:38.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.012 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148169 s, 70.8 MB/s 2024-04-17T10:33:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138247 s, 75.8 MB/s 2024-04-17T10:33:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.051 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209811 s, 50.0 MB/s 2024-04-17T10:33:38.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.067 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154161 s, 68.0 MB/s 2024-04-17T10:33:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160948 s, 65.1 MB/s 2024-04-17T10:33:38.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.101 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136334 s, 76.9 MB/s 2024-04-17T10:33:38.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.118 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150081 s, 69.9 MB/s 2024-04-17T10:33:38.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:38.119 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:38.119 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:38.119 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.1 2024-04-17T10:33:38.120 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:38.120 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:38.120 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:38.120 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147022 s, 71.3 MB/s 2024-04-17T10:33:38.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152355 s, 68.8 MB/s 2024-04-17T10:33:38.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.225 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015881 s, 66.0 MB/s 2024-04-17T10:33:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.244 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165 s, 63.6 MB/s 2024-04-17T10:33:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134112 s, 78.2 MB/s 2024-04-17T10:33:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178679 s, 58.7 MB/s 2024-04-17T10:33:38.295 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.296 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152112 s, 68.9 MB/s 2024-04-17T10:33:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.313 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016001 s, 65.5 MB/s 2024-04-17T10:33:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.330 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139956 s, 74.9 MB/s 2024-04-17T10:33:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.346 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152062 s, 69.0 MB/s 2024-04-17T10:33:38.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.364 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162585 s, 64.5 MB/s 2024-04-17T10:33:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144359 s, 72.6 MB/s 2024-04-17T10:33:38.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.401 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192652 s, 54.4 MB/s 2024-04-17T10:33:38.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134382 s, 78.0 MB/s 2024-04-17T10:33:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.432 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.433 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139366 s, 75.2 MB/s 2024-04-17T10:33:38.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.448 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140093 s, 74.8 MB/s 2024-04-17T10:33:38.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.467 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170755 s, 61.4 MB/s 2024-04-17T10:33:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.484 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148914 s, 70.4 MB/s 2024-04-17T10:33:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146773 s, 71.4 MB/s 2024-04-17T10:33:38.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.518 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157849 s, 66.4 MB/s 2024-04-17T10:33:38.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.534 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136619 s, 76.8 MB/s 2024-04-17T10:33:38.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.552 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158137 s, 66.3 MB/s 2024-04-17T10:33:38.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144135 s, 72.7 MB/s 2024-04-17T10:33:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174641 s, 60.0 MB/s 2024-04-17T10:33:38.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.603 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139045 s, 75.4 MB/s 2024-04-17T10:33:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148249 s, 70.7 MB/s 2024-04-17T10:33:38.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.636 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147826 s, 70.9 MB/s 2024-04-17T10:33:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153059 s, 68.5 MB/s 2024-04-17T10:33:38.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183482 s, 57.1 MB/s 2024-04-17T10:33:38.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.688 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135387 s, 77.5 MB/s 2024-04-17T10:33:38.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.704 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139102 s, 75.4 MB/s 2024-04-17T10:33:38.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.720 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.720 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014155 s, 74.1 MB/s 2024-04-17T10:33:38.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:38.721 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:38.721 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:38.722 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.2 2024-04-17T10:33:38.722 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:38.722 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:38.722 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:38.722 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:38.791 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.791 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.791 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149405 s, 70.2 MB/s 2024-04-17T10:33:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163955 s, 64.0 MB/s 2024-04-17T10:33:38.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.825 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144223 s, 72.7 MB/s 2024-04-17T10:33:38.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.841 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135012 s, 77.7 MB/s 2024-04-17T10:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.860 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179179 s, 58.5 MB/s 2024-04-17T10:33:38.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.876 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133613 s, 78.5 MB/s 2024-04-17T10:33:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.891 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136183 s, 77.0 MB/s 2024-04-17T10:33:38.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.906 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135046 s, 77.6 MB/s 2024-04-17T10:33:38.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138561 s, 75.7 MB/s 2024-04-17T10:33:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.941 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174091 s, 60.2 MB/s 2024-04-17T10:33:38.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.957 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137087 s, 76.5 MB/s 2024-04-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.973 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142975 s, 73.3 MB/s 2024-04-17T10:33:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159125 s, 65.9 MB/s 2024-04-17T10:33:39.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.006 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134349 s, 78.0 MB/s 2024-04-17T10:33:39.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.024 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161075 s, 65.1 MB/s 2024-04-17T10:33:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.041 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151607 s, 69.2 MB/s 2024-04-17T10:33:39.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.057 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014276 s, 73.5 MB/s 2024-04-17T10:33:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.073 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145856 s, 71.9 MB/s 2024-04-17T10:33:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151878 s, 69.0 MB/s 2024-04-17T10:33:39.109 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177172 s, 59.2 MB/s 2024-04-17T10:33:39.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.125 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136344 s, 76.9 MB/s 2024-04-17T10:33:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.144 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165159 s, 63.5 MB/s 2024-04-17T10:33:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146943 s, 71.4 MB/s 2024-04-17T10:33:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.176 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141354 s, 74.2 MB/s 2024-04-17T10:33:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012506 s, 83.8 MB/s 2024-04-17T10:33:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142126 s, 73.8 MB/s 2024-04-17T10:33:39.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141583 s, 74.1 MB/s 2024-04-17T10:33:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188027 s, 55.8 MB/s 2024-04-17T10:33:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.258 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137895 s, 76.0 MB/s 2024-04-17T10:33:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142411 s, 73.6 MB/s 2024-04-17T10:33:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.291 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149016 s, 70.4 MB/s 2024-04-17T10:33:39.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.308 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153848 s, 68.2 MB/s 2024-04-17T10:33:39.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:39.309 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:39.309 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:39.309 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.3 2024-04-17T10:33:39.309 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:39.310 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:39.310 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:39.310 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.380 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153795 s, 68.2 MB/s 2024-04-17T10:33:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.396 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013704 s, 76.5 MB/s 2024-04-17T10:33:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.415 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018106 s, 57.9 MB/s 2024-04-17T10:33:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.431 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138748 s, 75.6 MB/s 2024-04-17T10:33:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131741 s, 79.6 MB/s 2024-04-17T10:33:39.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.463 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146566 s, 71.5 MB/s 2024-04-17T10:33:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.479 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149834 s, 70.0 MB/s 2024-04-17T10:33:39.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.495 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137112 s, 76.5 MB/s 2024-04-17T10:33:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.512 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154486 s, 67.9 MB/s 2024-04-17T10:33:39.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.527 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131744 s, 79.6 MB/s 2024-04-17T10:33:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.543 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144247 s, 72.7 MB/s 2024-04-17T10:33:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147595 s, 71.0 MB/s 2024-04-17T10:33:39.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.580 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180995 s, 57.9 MB/s 2024-04-17T10:33:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.598 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158417 s, 66.2 MB/s 2024-04-17T10:33:39.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148769 s, 70.5 MB/s 2024-04-17T10:33:39.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.630 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013543 s, 77.4 MB/s 2024-04-17T10:33:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.647 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150758 s, 69.6 MB/s 2024-04-17T10:33:39.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131009 s, 80.0 MB/s 2024-04-17T10:33:39.679 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.679 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.679 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012838 s, 81.7 MB/s 2024-04-17T10:33:39.691 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.691 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.691 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132913 s, 78.9 MB/s 2024-04-17T10:33:39.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013487 s, 77.7 MB/s 2024-04-17T10:33:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139116 s, 75.4 MB/s 2024-04-17T10:33:39.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.737 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129677 s, 80.9 MB/s 2024-04-17T10:33:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.755 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015736 s, 66.6 MB/s 2024-04-17T10:33:39.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.770 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131501 s, 79.7 MB/s 2024-04-17T10:33:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.785 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134165 s, 78.2 MB/s 2024-04-17T10:33:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.799 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122136 s, 85.9 MB/s 2024-04-17T10:33:39.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148052 s, 70.8 MB/s 2024-04-17T10:33:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.831 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134509 s, 78.0 MB/s 2024-04-17T10:33:39.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.846 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132766 s, 79.0 MB/s 2024-04-17T10:33:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.863 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157231 s, 66.7 MB/s 2024-04-17T10:33:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.879 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139286 s, 75.3 MB/s 2024-04-17T10:33:39.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:39.881 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:39.881 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:39.881 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.4 2024-04-17T10:33:39.881 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:39.881 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:39.881 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:39.881 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.951 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143217 s, 73.2 MB/s 2024-04-17T10:33:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.967 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150803 s, 69.5 MB/s 2024-04-17T10:33:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132441 s, 79.2 MB/s 2024-04-17T10:33:39.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:39.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:39.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136394 s, 76.9 MB/s 2024-04-17T10:33:40.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.014 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139653 s, 75.1 MB/s 2024-04-17T10:33:40.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143481 s, 73.1 MB/s 2024-04-17T10:33:40.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.045 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013158 s, 79.7 MB/s 2024-04-17T10:33:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.062 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151908 s, 69.0 MB/s 2024-04-17T10:33:40.077 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.077 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136658 s, 76.7 MB/s 2024-04-17T10:33:40.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.093 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135097 s, 77.6 MB/s 2024-04-17T10:33:40.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156035 s, 67.2 MB/s 2024-04-17T10:33:40.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.126 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138378 s, 75.8 MB/s 2024-04-17T10:33:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180654 s, 58.0 MB/s 2024-04-17T10:33:40.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.161 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132336 s, 79.2 MB/s 2024-04-17T10:33:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142864 s, 73.4 MB/s 2024-04-17T10:33:40.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.196 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175528 s, 59.7 MB/s 2024-04-17T10:33:40.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.211 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133204 s, 78.7 MB/s 2024-04-17T10:33:40.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.227 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143318 s, 73.2 MB/s 2024-04-17T10:33:40.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135542 s, 77.4 MB/s 2024-04-17T10:33:40.259 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.260 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149557 s, 70.1 MB/s 2024-04-17T10:33:40.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.276 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147544 s, 71.1 MB/s 2024-04-17T10:33:40.292 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.292 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.292 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147831 s, 70.9 MB/s 2024-04-17T10:33:40.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.310 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155803 s, 67.3 MB/s 2024-04-17T10:33:40.326 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.326 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.326 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146643 s, 71.5 MB/s 2024-04-17T10:33:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.342 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141285 s, 74.2 MB/s 2024-04-17T10:33:40.358 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.359 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146133 s, 71.8 MB/s 2024-04-17T10:33:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.374 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013591 s, 77.2 MB/s 2024-04-17T10:33:40.390 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.390 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.390 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140002 s, 74.9 MB/s 2024-04-17T10:33:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.409 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171564 s, 61.1 MB/s 2024-04-17T10:33:40.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156643 s, 66.9 MB/s 2024-04-17T10:33:40.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130803 s, 80.2 MB/s 2024-04-17T10:33:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128604 s, 81.5 MB/s 2024-04-17T10:33:40.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:40.457 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:40.457 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:40.457 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.5 2024-04-17T10:33:40.458 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:40.458 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:40.458 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:40.458 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:40.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.528 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015305 s, 68.5 MB/s 2024-04-17T10:33:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138586 s, 75.7 MB/s 2024-04-17T10:33:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180346 s, 58.1 MB/s 2024-04-17T10:33:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.580 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150685 s, 69.6 MB/s 2024-04-17T10:33:40.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.595 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130954 s, 80.1 MB/s 2024-04-17T10:33:40.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155675 s, 67.4 MB/s 2024-04-17T10:33:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143637 s, 73.0 MB/s 2024-04-17T10:33:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.647 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155147 s, 67.6 MB/s 2024-04-17T10:33:40.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.661 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129921 s, 80.7 MB/s 2024-04-17T10:33:40.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.682 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.682 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183989 s, 57.0 MB/s 2024-04-17T10:33:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.697 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139324 s, 75.3 MB/s 2024-04-17T10:33:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147859 s, 70.9 MB/s 2024-04-17T10:33:40.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.729 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139398 s, 75.2 MB/s 2024-04-17T10:33:40.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.746 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145836 s, 71.9 MB/s 2024-04-17T10:33:40.761 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.762 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138335 s, 75.8 MB/s 2024-04-17T10:33:40.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.777 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133895 s, 78.3 MB/s 2024-04-17T10:33:40.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.793 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143816 s, 72.9 MB/s 2024-04-17T10:33:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.810 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152621 s, 68.7 MB/s 2024-04-17T10:33:40.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.825 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013438 s, 78.0 MB/s 2024-04-17T10:33:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.841 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143477 s, 73.1 MB/s 2024-04-17T10:33:40.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.857 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144277 s, 72.7 MB/s 2024-04-17T10:33:40.873 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.873 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.873 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143238 s, 73.2 MB/s 2024-04-17T10:33:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.889 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141568 s, 74.1 MB/s 2024-04-17T10:33:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.906 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151876 s, 69.0 MB/s 2024-04-17T10:33:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150507 s, 69.7 MB/s 2024-04-17T10:33:40.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.941 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154794 s, 67.7 MB/s 2024-04-17T10:33:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151906 s, 69.0 MB/s 2024-04-17T10:33:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.973 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137951 s, 76.0 MB/s 2024-04-17T10:33:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015243 s, 68.8 MB/s 2024-04-17T10:33:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151716 s, 69.1 MB/s 2024-04-17T10:33:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013424 s, 78.1 MB/s 2024-04-17T10:33:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.037 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013313 s, 78.8 MB/s 2024-04-17T10:33:41.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:41.039 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:41.039 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:41.039 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.6 2024-04-17T10:33:41.039 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:41.039 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:41.039 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:41.039 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139677 s, 75.1 MB/s 2024-04-17T10:33:41.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.124 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151202 s, 69.3 MB/s 2024-04-17T10:33:41.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.143 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169105 s, 62.0 MB/s 2024-04-17T10:33:41.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.159 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148367 s, 70.7 MB/s 2024-04-17T10:33:41.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.176 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149694 s, 70.0 MB/s 2024-04-17T10:33:41.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137031 s, 76.5 MB/s 2024-04-17T10:33:41.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129518 s, 81.0 MB/s 2024-04-17T10:33:41.223 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156455 s, 67.0 MB/s 2024-04-17T10:33:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.239 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133921 s, 78.3 MB/s 2024-04-17T10:33:41.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.258 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175236 s, 59.8 MB/s 2024-04-17T10:33:41.275 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.276 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154975 s, 67.7 MB/s 2024-04-17T10:33:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.291 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136555 s, 76.8 MB/s 2024-04-17T10:33:41.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.310 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151044 s, 69.4 MB/s 2024-04-17T10:33:41.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.324 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138342 s, 75.8 MB/s 2024-04-17T10:33:41.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.341 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151308 s, 69.3 MB/s 2024-04-17T10:33:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136317 s, 76.9 MB/s 2024-04-17T10:33:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013721 s, 76.4 MB/s 2024-04-17T10:33:41.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140257 s, 74.8 MB/s 2024-04-17T10:33:41.402 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.402 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.402 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133656 s, 78.5 MB/s 2024-04-17T10:33:41.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149845 s, 70.0 MB/s 2024-04-17T10:33:41.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.434 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132895 s, 78.9 MB/s 2024-04-17T10:33:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.450 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139434 s, 75.2 MB/s 2024-04-17T10:33:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.465 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135471 s, 77.4 MB/s 2024-04-17T10:33:41.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.483 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151645 s, 69.1 MB/s 2024-04-17T10:33:41.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.500 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.500 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153401 s, 68.4 MB/s 2024-04-17T10:33:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154484 s, 67.9 MB/s 2024-04-17T10:33:41.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.533 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.533 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138446 s, 75.7 MB/s 2024-04-17T10:33:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140837 s, 74.5 MB/s 2024-04-17T10:33:41.565 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.565 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.566 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015281 s, 68.6 MB/s 2024-04-17T10:33:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.580 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128195 s, 81.8 MB/s 2024-04-17T10:33:41.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.597 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152848 s, 68.6 MB/s 2024-04-17T10:33:41.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.615 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152917 s, 68.6 MB/s 2024-04-17T10:33:41.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:41.616 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:41.616 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:41.616 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.7 2024-04-17T10:33:41.616 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:41.616 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:41.616 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:41.616 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.686 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136136 s, 77.0 MB/s 2024-04-17T10:33:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.703 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153719 s, 68.2 MB/s 2024-04-17T10:33:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.719 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137089 s, 76.5 MB/s 2024-04-17T10:33:41.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.735 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143644 s, 73.0 MB/s 2024-04-17T10:33:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143962 s, 72.8 MB/s 2024-04-17T10:33:41.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138162 s, 75.9 MB/s 2024-04-17T10:33:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.784 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152271 s, 68.9 MB/s 2024-04-17T10:33:41.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143885 s, 72.9 MB/s 2024-04-17T10:33:41.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160971 s, 65.1 MB/s 2024-04-17T10:33:41.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.834 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138137 s, 75.9 MB/s 2024-04-17T10:33:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.849 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134687 s, 77.9 MB/s 2024-04-17T10:33:41.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.864 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137632 s, 76.2 MB/s 2024-04-17T10:33:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013189 s, 79.5 MB/s 2024-04-17T10:33:41.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.896 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145333 s, 72.2 MB/s 2024-04-17T10:33:41.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131214 s, 79.9 MB/s 2024-04-17T10:33:41.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143136 s, 73.3 MB/s 2024-04-17T10:33:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.942 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136841 s, 76.6 MB/s 2024-04-17T10:33:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016805 s, 62.4 MB/s 2024-04-17T10:33:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137283 s, 76.4 MB/s 2024-04-17T10:33:41.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:41.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:41.992 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137365 s, 76.3 MB/s 2024-04-17T10:33:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.009 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157055 s, 66.8 MB/s 2024-04-17T10:33:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155627 s, 67.4 MB/s 2024-04-17T10:33:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.043 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147628 s, 71.0 MB/s 2024-04-17T10:33:42.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157851 s, 66.4 MB/s 2024-04-17T10:33:42.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151328 s, 69.3 MB/s 2024-04-17T10:33:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01537 s, 68.2 MB/s 2024-04-17T10:33:42.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.112 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150631 s, 69.6 MB/s 2024-04-17T10:33:42.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152294 s, 68.9 MB/s 2024-04-17T10:33:42.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.147 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159311 s, 65.8 MB/s 2024-04-17T10:33:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.164 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154311 s, 68.0 MB/s 2024-04-17T10:33:42.180 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.180 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.180 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143285 s, 73.2 MB/s 2024-04-17T10:33:42.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165787 s, 63.2 MB/s 2024-04-17T10:33:42.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:42.200 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:42.200 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:42.200 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.8 2024-04-17T10:33:42.200 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:42.200 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:42.200 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:42.200 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144152 s, 72.7 MB/s 2024-04-17T10:33:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137232 s, 76.4 MB/s 2024-04-17T10:33:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.305 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168873 s, 62.1 MB/s 2024-04-17T10:33:42.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.320 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133933 s, 78.3 MB/s 2024-04-17T10:33:42.335 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.335 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.335 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133993 s, 78.3 MB/s 2024-04-17T10:33:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.352 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150095 s, 69.9 MB/s 2024-04-17T10:33:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.368 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134739 s, 77.8 MB/s 2024-04-17T10:33:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135243 s, 77.5 MB/s 2024-04-17T10:33:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.399 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.400 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150198 s, 69.8 MB/s 2024-04-17T10:33:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015273 s, 68.7 MB/s 2024-04-17T10:33:42.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.432 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132372 s, 79.2 MB/s 2024-04-17T10:33:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159111 s, 65.9 MB/s 2024-04-17T10:33:42.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.463 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123326 s, 85.0 MB/s 2024-04-17T10:33:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.480 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148835 s, 70.5 MB/s 2024-04-17T10:33:42.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138904 s, 75.5 MB/s 2024-04-17T10:33:42.513 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.513 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.513 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158774 s, 66.0 MB/s 2024-04-17T10:33:42.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.530 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.530 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014323 s, 73.2 MB/s 2024-04-17T10:33:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185224 s, 56.6 MB/s 2024-04-17T10:33:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.567 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156561 s, 67.0 MB/s 2024-04-17T10:33:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.584 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139866 s, 75.0 MB/s 2024-04-17T10:33:42.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.601 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153534 s, 68.3 MB/s 2024-04-17T10:33:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.616 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137599 s, 76.2 MB/s 2024-04-17T10:33:42.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129671 s, 80.9 MB/s 2024-04-17T10:33:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139537 s, 75.1 MB/s 2024-04-17T10:33:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013653 s, 76.8 MB/s 2024-04-17T10:33:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158752 s, 66.1 MB/s 2024-04-17T10:33:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.695 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013222 s, 79.3 MB/s 2024-04-17T10:33:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.712 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154668 s, 67.8 MB/s 2024-04-17T10:33:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.728 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134943 s, 77.7 MB/s 2024-04-17T10:33:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180033 s, 58.2 MB/s 2024-04-17T10:33:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.761 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118247 s, 88.7 MB/s 2024-04-17T10:33:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166493 s, 63.0 MB/s 2024-04-17T10:33:42.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:42.781 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:42.781 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:42.781 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.9 2024-04-17T10:33:42.781 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:42.781 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:42.781 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:42.781 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:42.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.854 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136051 s, 77.1 MB/s 2024-04-17T10:33:42.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.873 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154358 s, 67.9 MB/s 2024-04-17T10:33:42.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132369 s, 79.2 MB/s 2024-04-17T10:33:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.904 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149047 s, 70.4 MB/s 2024-04-17T10:33:42.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.920 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014168 s, 74.0 MB/s 2024-04-17T10:33:42.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151939 s, 69.0 MB/s 2024-04-17T10:33:42.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.954 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015518 s, 67.6 MB/s 2024-04-17T10:33:42.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.972 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160786 s, 65.2 MB/s 2024-04-17T10:33:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:42.988 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144168 s, 72.7 MB/s 2024-04-17T10:33:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.005 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153552 s, 68.3 MB/s 2024-04-17T10:33:43.020 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.020 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.020 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133589 s, 78.5 MB/s 2024-04-17T10:33:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.038 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148776 s, 70.5 MB/s 2024-04-17T10:33:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.053 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144081 s, 72.8 MB/s 2024-04-17T10:33:43.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.069 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135852 s, 77.2 MB/s 2024-04-17T10:33:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.084 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135677 s, 77.3 MB/s 2024-04-17T10:33:43.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.101 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152092 s, 68.9 MB/s 2024-04-17T10:33:43.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.117 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137142 s, 76.5 MB/s 2024-04-17T10:33:43.132 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.132 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.132 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132835 s, 78.9 MB/s 2024-04-17T10:33:43.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.151 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145302 s, 72.2 MB/s 2024-04-17T10:33:43.164 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.164 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.164 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142919 s, 73.4 MB/s 2024-04-17T10:33:43.180 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.181 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.181 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152579 s, 68.7 MB/s 2024-04-17T10:33:43.197 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150409 s, 69.7 MB/s 2024-04-17T10:33:43.212 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131345 s, 79.8 MB/s 2024-04-17T10:33:43.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.230 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154496 s, 67.9 MB/s 2024-04-17T10:33:43.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.247 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152021 s, 69.0 MB/s 2024-04-17T10:33:43.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.262 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139502 s, 75.2 MB/s 2024-04-17T10:33:43.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.277 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132913 s, 78.9 MB/s 2024-04-17T10:33:43.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154041 s, 68.1 MB/s 2024-04-17T10:33:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.312 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.312 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152457 s, 68.8 MB/s 2024-04-17T10:33:43.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.327 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141633 s, 74.0 MB/s 2024-04-17T10:33:43.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.345 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.345 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152488 s, 68.8 MB/s 2024-04-17T10:33:43.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.361 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142826 s, 73.4 MB/s 2024-04-17T10:33:43.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:43.362 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:43.362 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:43.362 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.10 2024-04-17T10:33:43.362 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:43.362 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:43.362 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:43.362 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150907 s, 69.5 MB/s 2024-04-17T10:33:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137551 s, 76.2 MB/s 2024-04-17T10:33:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130146 s, 80.6 MB/s 2024-04-17T10:33:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.478 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131663 s, 79.6 MB/s 2024-04-17T10:33:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.493 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129472 s, 81.0 MB/s 2024-04-17T10:33:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.509 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141813 s, 73.9 MB/s 2024-04-17T10:33:43.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151154 s, 69.4 MB/s 2024-04-17T10:33:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159755 s, 65.6 MB/s 2024-04-17T10:33:43.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141496 s, 74.1 MB/s 2024-04-17T10:33:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.574 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125674 s, 83.4 MB/s 2024-04-17T10:33:43.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.589 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135143 s, 77.6 MB/s 2024-04-17T10:33:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165427 s, 63.4 MB/s 2024-04-17T10:33:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.624 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140734 s, 74.5 MB/s 2024-04-17T10:33:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135549 s, 77.4 MB/s 2024-04-17T10:33:43.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.655 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146361 s, 71.6 MB/s 2024-04-17T10:33:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132623 s, 79.1 MB/s 2024-04-17T10:33:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.686 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133285 s, 78.7 MB/s 2024-04-17T10:33:43.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.701 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135473 s, 77.4 MB/s 2024-04-17T10:33:43.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.718 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156266 s, 67.1 MB/s 2024-04-17T10:33:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.735 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147296 s, 71.2 MB/s 2024-04-17T10:33:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.752 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154119 s, 68.0 MB/s 2024-04-17T10:33:43.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.770 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158902 s, 66.0 MB/s 2024-04-17T10:33:43.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.790 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186053 s, 56.4 MB/s 2024-04-17T10:33:43.806 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.807 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.807 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145907 s, 71.9 MB/s 2024-04-17T10:33:43.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176026 s, 59.6 MB/s 2024-04-17T10:33:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177668 s, 59.0 MB/s 2024-04-17T10:33:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.863 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155884 s, 67.3 MB/s 2024-04-17T10:33:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136154 s, 77.0 MB/s 2024-04-17T10:33:43.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.894 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142472 s, 73.6 MB/s 2024-04-17T10:33:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146416 s, 71.6 MB/s 2024-04-17T10:33:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.928 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153567 s, 68.3 MB/s 2024-04-17T10:33:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:43.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:43.945 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148244 s, 70.7 MB/s 2024-04-17T10:33:43.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:43.946 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:43.946 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:43.946 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.11 2024-04-17T10:33:43.946 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:43.946 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:43.946 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:43.946 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:44.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.014 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136658 s, 76.7 MB/s 2024-04-17T10:33:44.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136234 s, 77.0 MB/s 2024-04-17T10:33:44.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.047 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015222 s, 68.9 MB/s 2024-04-17T10:33:44.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130639 s, 80.3 MB/s 2024-04-17T10:33:44.077 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.077 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144229 s, 72.7 MB/s 2024-04-17T10:33:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146502 s, 71.6 MB/s 2024-04-17T10:33:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.111 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150397 s, 69.7 MB/s 2024-04-17T10:33:44.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140957 s, 74.4 MB/s 2024-04-17T10:33:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.143 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143495 s, 73.1 MB/s 2024-04-17T10:33:44.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.159 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135489 s, 77.4 MB/s 2024-04-17T10:33:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.174 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133349 s, 78.6 MB/s 2024-04-17T10:33:44.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.189 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134975 s, 77.7 MB/s 2024-04-17T10:33:44.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.205 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138439 s, 75.7 MB/s 2024-04-17T10:33:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.220 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130254 s, 80.5 MB/s 2024-04-17T10:33:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140917 s, 74.4 MB/s 2024-04-17T10:33:44.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176077 s, 59.6 MB/s 2024-04-17T10:33:44.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143852 s, 72.9 MB/s 2024-04-17T10:33:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144236 s, 72.7 MB/s 2024-04-17T10:33:44.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.302 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013415 s, 78.2 MB/s 2024-04-17T10:33:44.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.319 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148846 s, 70.4 MB/s 2024-04-17T10:33:44.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.334 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138353 s, 75.8 MB/s 2024-04-17T10:33:44.351 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.352 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153821 s, 68.2 MB/s 2024-04-17T10:33:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.367 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135731 s, 77.3 MB/s 2024-04-17T10:33:44.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.383 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141802 s, 73.9 MB/s 2024-04-17T10:33:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.400 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150608 s, 69.6 MB/s 2024-04-17T10:33:44.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151893 s, 69.0 MB/s 2024-04-17T10:33:44.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.435 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164972 s, 63.6 MB/s 2024-04-17T10:33:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.450 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135611 s, 77.3 MB/s 2024-04-17T10:33:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139491 s, 75.2 MB/s 2024-04-17T10:33:44.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.483 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150535 s, 69.7 MB/s 2024-04-17T10:33:44.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01307 s, 80.2 MB/s 2024-04-17T10:33:44.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.515 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149448 s, 70.2 MB/s 2024-04-17T10:33:44.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:44.516 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:44.516 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:44.516 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.12 2024-04-17T10:33:44.516 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:44.516 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:44.516 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:44.517 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.588 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151829 s, 69.1 MB/s 2024-04-17T10:33:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.605 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155014 s, 67.6 MB/s 2024-04-17T10:33:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133104 s, 78.8 MB/s 2024-04-17T10:33:44.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.640 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171874 s, 61.0 MB/s 2024-04-17T10:33:44.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.655 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013233 s, 79.2 MB/s 2024-04-17T10:33:44.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.672 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015856 s, 66.1 MB/s 2024-04-17T10:33:44.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.688 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141824 s, 73.9 MB/s 2024-04-17T10:33:44.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137715 s, 76.1 MB/s 2024-04-17T10:33:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.721 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154315 s, 68.0 MB/s 2024-04-17T10:33:44.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.736 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136998 s, 76.5 MB/s 2024-04-17T10:33:44.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149659 s, 70.1 MB/s 2024-04-17T10:33:44.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116587 s, 89.9 MB/s 2024-04-17T10:33:44.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.782 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141368 s, 74.2 MB/s 2024-04-17T10:33:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.796 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120546 s, 87.0 MB/s 2024-04-17T10:33:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.813 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153717 s, 68.2 MB/s 2024-04-17T10:33:44.831 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.831 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.831 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159364 s, 65.8 MB/s 2024-04-17T10:33:44.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.846 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013585 s, 77.2 MB/s 2024-04-17T10:33:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130563 s, 80.3 MB/s 2024-04-17T10:33:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.876 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133452 s, 78.6 MB/s 2024-04-17T10:33:44.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135137 s, 77.6 MB/s 2024-04-17T10:33:44.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.906 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118811 s, 88.3 MB/s 2024-04-17T10:33:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140728 s, 74.5 MB/s 2024-04-17T10:33:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.937 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144505 s, 72.6 MB/s 2024-04-17T10:33:44.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141365 s, 74.2 MB/s 2024-04-17T10:33:44.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153988 s, 68.1 MB/s 2024-04-17T10:33:44.990 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:44.991 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188341 s, 55.7 MB/s 2024-04-17T10:33:45.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.007 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143008 s, 73.3 MB/s 2024-04-17T10:33:45.023 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.023 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.023 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014375 s, 72.9 MB/s 2024-04-17T10:33:45.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.039 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141124 s, 74.3 MB/s 2024-04-17T10:33:45.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.055 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142545 s, 73.6 MB/s 2024-04-17T10:33:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.070 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133347 s, 78.6 MB/s 2024-04-17T10:33:45.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.085 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132859 s, 78.9 MB/s 2024-04-17T10:33:45.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:45.086 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:45.087 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:45.087 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.13 2024-04-17T10:33:45.087 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:45.087 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:45.087 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:45.087 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:45.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153096 s, 68.5 MB/s 2024-04-17T10:33:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.177 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152458 s, 68.8 MB/s 2024-04-17T10:33:45.192 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.193 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138277 s, 75.8 MB/s 2024-04-17T10:33:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133323 s, 78.6 MB/s 2024-04-17T10:33:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.225 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151583 s, 69.2 MB/s 2024-04-17T10:33:45.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.241 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143831 s, 72.9 MB/s 2024-04-17T10:33:45.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.258 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154804 s, 67.7 MB/s 2024-04-17T10:33:45.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.274 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141385 s, 74.2 MB/s 2024-04-17T10:33:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.290 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013717 s, 76.4 MB/s 2024-04-17T10:33:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014525 s, 72.2 MB/s 2024-04-17T10:33:45.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.322 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142476 s, 73.6 MB/s 2024-04-17T10:33:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.337 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133176 s, 78.7 MB/s 2024-04-17T10:33:45.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.352 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137358 s, 76.3 MB/s 2024-04-17T10:33:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.369 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150939 s, 69.5 MB/s 2024-04-17T10:33:45.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.386 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151743 s, 69.1 MB/s 2024-04-17T10:33:45.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.404 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153749 s, 68.2 MB/s 2024-04-17T10:33:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152422 s, 68.8 MB/s 2024-04-17T10:33:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138515 s, 75.7 MB/s 2024-04-17T10:33:45.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152429 s, 68.8 MB/s 2024-04-17T10:33:45.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128427 s, 81.6 MB/s 2024-04-17T10:33:45.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.485 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149177 s, 70.3 MB/s 2024-04-17T10:33:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187476 s, 55.9 MB/s 2024-04-17T10:33:45.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136234 s, 77.0 MB/s 2024-04-17T10:33:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.540 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172782 s, 60.7 MB/s 2024-04-17T10:33:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207199 s, 50.6 MB/s 2024-04-17T10:33:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.579 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148521 s, 70.6 MB/s 2024-04-17T10:33:45.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.596 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151723 s, 69.1 MB/s 2024-04-17T10:33:45.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.612 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014764 s, 71.0 MB/s 2024-04-17T10:33:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172153 s, 60.9 MB/s 2024-04-17T10:33:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.647 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136213 s, 77.0 MB/s 2024-04-17T10:33:45.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.665 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156484 s, 67.0 MB/s 2024-04-17T10:33:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142662 s, 73.5 MB/s 2024-04-17T10:33:45.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:45.682 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:45.682 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:45.682 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.14 2024-04-17T10:33:45.682 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:45.682 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:45.683 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:45.683 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:45.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.754 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133762 s, 78.4 MB/s 2024-04-17T10:33:45.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.770 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137278 s, 76.4 MB/s 2024-04-17T10:33:45.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.785 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135561 s, 77.4 MB/s 2024-04-17T10:33:45.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.801 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146033 s, 71.8 MB/s 2024-04-17T10:33:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154479 s, 67.9 MB/s 2024-04-17T10:33:45.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.832 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120126 s, 87.3 MB/s 2024-04-17T10:33:45.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139063 s, 75.4 MB/s 2024-04-17T10:33:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.863 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127699 s, 82.1 MB/s 2024-04-17T10:33:45.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.879 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146002 s, 71.8 MB/s 2024-04-17T10:33:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.897 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164582 s, 63.7 MB/s 2024-04-17T10:33:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.913 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141954 s, 73.9 MB/s 2024-04-17T10:33:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151271 s, 69.3 MB/s 2024-04-17T10:33:45.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.946 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143735 s, 73.0 MB/s 2024-04-17T10:33:45.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.962 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140283 s, 74.7 MB/s 2024-04-17T10:33:45.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134836 s, 77.8 MB/s 2024-04-17T10:33:45.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:45.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01585 s, 66.2 MB/s 2024-04-17T10:33:46.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.010 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137518 s, 76.3 MB/s 2024-04-17T10:33:46.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180842 s, 58.0 MB/s 2024-04-17T10:33:46.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.047 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151305 s, 69.3 MB/s 2024-04-17T10:33:46.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.064 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148882 s, 70.4 MB/s 2024-04-17T10:33:46.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132768 s, 79.0 MB/s 2024-04-17T10:33:46.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135752 s, 77.2 MB/s 2024-04-17T10:33:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137813 s, 76.1 MB/s 2024-04-17T10:33:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.125 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132427 s, 79.2 MB/s 2024-04-17T10:33:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.140 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133597 s, 78.5 MB/s 2024-04-17T10:33:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.158 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155313 s, 67.5 MB/s 2024-04-17T10:33:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.174 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146991 s, 71.3 MB/s 2024-04-17T10:33:46.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.190 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136702 s, 76.7 MB/s 2024-04-17T10:33:46.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.205 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132177 s, 79.3 MB/s 2024-04-17T10:33:46.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.221 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151507 s, 69.2 MB/s 2024-04-17T10:33:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.237 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135638 s, 77.3 MB/s 2024-04-17T10:33:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134771 s, 77.8 MB/s 2024-04-17T10:33:46.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:46.253 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:46.253 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:46.254 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.15 2024-04-17T10:33:46.254 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:46.254 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:46.254 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:46.254 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.323 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151393 s, 69.3 MB/s 2024-04-17T10:33:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.339 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135713 s, 77.3 MB/s 2024-04-17T10:33:46.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142497 s, 73.6 MB/s 2024-04-17T10:33:46.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138604 s, 75.7 MB/s 2024-04-17T10:33:46.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.386 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136562 s, 76.8 MB/s 2024-04-17T10:33:46.405 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.406 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176942 s, 59.3 MB/s 2024-04-17T10:33:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.423 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146849 s, 71.4 MB/s 2024-04-17T10:33:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.440 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156787 s, 66.9 MB/s 2024-04-17T10:33:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01444 s, 72.6 MB/s 2024-04-17T10:33:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.471 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136175 s, 77.0 MB/s 2024-04-17T10:33:46.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.487 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141036 s, 74.3 MB/s 2024-04-17T10:33:46.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.502 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135633 s, 77.3 MB/s 2024-04-17T10:33:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.519 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150394 s, 69.7 MB/s 2024-04-17T10:33:46.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.536 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146153 s, 71.7 MB/s 2024-04-17T10:33:46.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153775 s, 68.2 MB/s 2024-04-17T10:33:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.570 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143137 s, 73.3 MB/s 2024-04-17T10:33:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135326 s, 77.5 MB/s 2024-04-17T10:33:46.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.602 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149463 s, 70.2 MB/s 2024-04-17T10:33:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.616 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127751 s, 82.1 MB/s 2024-04-17T10:33:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128815 s, 81.4 MB/s 2024-04-17T10:33:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139271 s, 75.3 MB/s 2024-04-17T10:33:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133977 s, 78.3 MB/s 2024-04-17T10:33:46.677 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139429 s, 75.2 MB/s 2024-04-17T10:33:46.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.693 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134585 s, 77.9 MB/s 2024-04-17T10:33:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115075 s, 91.1 MB/s 2024-04-17T10:33:46.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133088 s, 78.8 MB/s 2024-04-17T10:33:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.742 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182499 s, 57.5 MB/s 2024-04-17T10:33:46.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.757 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134629 s, 77.9 MB/s 2024-04-17T10:33:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.772 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134088 s, 78.2 MB/s 2024-04-17T10:33:46.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.789 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154753 s, 67.8 MB/s 2024-04-17T10:33:46.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.805 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139016 s, 75.4 MB/s 2024-04-17T10:33:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.822 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152059 s, 69.0 MB/s 2024-04-17T10:33:46.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:46.824 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:46.824 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:46.824 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.16 2024-04-17T10:33:46.824 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:46.824 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:46.824 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:46.824 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:46.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.894 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137535 s, 76.2 MB/s 2024-04-17T10:33:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149132 s, 70.3 MB/s 2024-04-17T10:33:46.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.927 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143945 s, 72.8 MB/s 2024-04-17T10:33:46.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.944 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157569 s, 66.5 MB/s 2024-04-17T10:33:46.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151884 s, 69.0 MB/s 2024-04-17T10:33:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.978 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152698 s, 68.7 MB/s 2024-04-17T10:33:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:46.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180098 s, 58.2 MB/s 2024-04-17T10:33:47.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.013 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.013 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131131 s, 80.0 MB/s 2024-04-17T10:33:47.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152997 s, 68.5 MB/s 2024-04-17T10:33:47.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.093 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152064 s, 69.0 MB/s 2024-04-17T10:33:47.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151157 s, 69.4 MB/s 2024-04-17T10:33:47.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154311 s, 68.0 MB/s 2024-04-17T10:33:47.097 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.098 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150973 s, 69.5 MB/s 2024-04-17T10:33:47.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.114 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150044 s, 69.9 MB/s 2024-04-17T10:33:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.131 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148904 s, 70.4 MB/s 2024-04-17T10:33:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.147 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143152 s, 73.2 MB/s 2024-04-17T10:33:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.164 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150777 s, 69.5 MB/s 2024-04-17T10:33:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.181 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155606 s, 67.4 MB/s 2024-04-17T10:33:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150944 s, 69.5 MB/s 2024-04-17T10:33:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186565 s, 56.2 MB/s 2024-04-17T10:33:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.236 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152443 s, 68.8 MB/s 2024-04-17T10:33:47.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153586 s, 68.3 MB/s 2024-04-17T10:33:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.269 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015121 s, 69.3 MB/s 2024-04-17T10:33:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.285 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137384 s, 76.3 MB/s 2024-04-17T10:33:47.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195811 s, 53.6 MB/s 2024-04-17T10:33:47.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.323 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145492 s, 72.1 MB/s 2024-04-17T10:33:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139933 s, 74.9 MB/s 2024-04-17T10:33:47.358 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.358 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.358 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178441 s, 58.8 MB/s 2024-04-17T10:33:47.373 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.373 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.373 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133363 s, 78.6 MB/s 2024-04-17T10:33:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.389 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139558 s, 75.1 MB/s 2024-04-17T10:33:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.409 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181999 s, 57.6 MB/s 2024-04-17T10:33:47.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.425 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143217 s, 73.2 MB/s 2024-04-17T10:33:47.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:47.426 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:47.426 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:47.426 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.17 2024-04-17T10:33:47.426 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:47.426 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:47.426 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:47.426 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:47.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.495 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139874 s, 75.0 MB/s 2024-04-17T10:33:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.511 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142398 s, 73.6 MB/s 2024-04-17T10:33:47.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134962 s, 77.7 MB/s 2024-04-17T10:33:47.541 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.541 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.541 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134942 s, 77.7 MB/s 2024-04-17T10:33:47.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.559 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156529 s, 67.0 MB/s 2024-04-17T10:33:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.574 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135007 s, 77.7 MB/s 2024-04-17T10:33:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.590 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141492 s, 74.1 MB/s 2024-04-17T10:33:47.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.607 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147463 s, 71.1 MB/s 2024-04-17T10:33:47.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.622 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135935 s, 77.1 MB/s 2024-04-17T10:33:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155848 s, 67.3 MB/s 2024-04-17T10:33:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.657 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154273 s, 68.0 MB/s 2024-04-17T10:33:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129301 s, 81.1 MB/s 2024-04-17T10:33:47.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.692 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191375 s, 54.8 MB/s 2024-04-17T10:33:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.709 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149641 s, 70.1 MB/s 2024-04-17T10:33:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.725 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137719 s, 76.1 MB/s 2024-04-17T10:33:47.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140311 s, 74.7 MB/s 2024-04-17T10:33:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.761 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190922 s, 54.9 MB/s 2024-04-17T10:33:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.776 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126332 s, 83.0 MB/s 2024-04-17T10:33:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142215 s, 73.7 MB/s 2024-04-17T10:33:47.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.808 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141963 s, 73.9 MB/s 2024-04-17T10:33:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.824 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139907 s, 74.9 MB/s 2024-04-17T10:33:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148282 s, 70.7 MB/s 2024-04-17T10:33:47.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.857 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014705 s, 71.3 MB/s 2024-04-17T10:33:47.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.875 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160352 s, 65.4 MB/s 2024-04-17T10:33:47.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015187 s, 69.0 MB/s 2024-04-17T10:33:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.908 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.908 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138996 s, 75.4 MB/s 2024-04-17T10:33:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130682 s, 80.2 MB/s 2024-04-17T10:33:47.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.943 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189187 s, 55.4 MB/s 2024-04-17T10:33:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.959 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.959 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138012 s, 76.0 MB/s 2024-04-17T10:33:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159338 s, 65.8 MB/s 2024-04-17T10:33:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133451 s, 78.6 MB/s 2024-04-17T10:33:48.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.007 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129754 s, 80.8 MB/s 2024-04-17T10:33:48.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:48.008 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:48.008 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:48.008 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.18 2024-04-17T10:33:48.008 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:48.008 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:48.008 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:48.008 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133285 s, 78.7 MB/s 2024-04-17T10:33:48.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.099 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157091 s, 66.7 MB/s 2024-04-17T10:33:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170723 s, 61.4 MB/s 2024-04-17T10:33:48.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.135 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150027 s, 69.9 MB/s 2024-04-17T10:33:48.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179821 s, 58.3 MB/s 2024-04-17T10:33:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.172 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155233 s, 67.5 MB/s 2024-04-17T10:33:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136055 s, 77.1 MB/s 2024-04-17T10:33:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.205 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153458 s, 68.3 MB/s 2024-04-17T10:33:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152384 s, 68.8 MB/s 2024-04-17T10:33:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149029 s, 70.4 MB/s 2024-04-17T10:33:48.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.256 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154413 s, 67.9 MB/s 2024-04-17T10:33:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.272 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144999 s, 72.3 MB/s 2024-04-17T10:33:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146125 s, 71.8 MB/s 2024-04-17T10:33:48.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.305 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146349 s, 71.6 MB/s 2024-04-17T10:33:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120314 s, 87.2 MB/s 2024-04-17T10:33:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.337 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163639 s, 64.1 MB/s 2024-04-17T10:33:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.352 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135182 s, 77.6 MB/s 2024-04-17T10:33:48.369 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.369 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.369 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153916 s, 68.1 MB/s 2024-04-17T10:33:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.384 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132269 s, 79.3 MB/s 2024-04-17T10:33:48.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.401 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153548 s, 68.3 MB/s 2024-04-17T10:33:48.416 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.416 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.416 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013566 s, 77.3 MB/s 2024-04-17T10:33:48.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.431 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.432 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132444 s, 79.2 MB/s 2024-04-17T10:33:48.449 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.449 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.449 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157074 s, 66.8 MB/s 2024-04-17T10:33:48.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149505 s, 70.1 MB/s 2024-04-17T10:33:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.481 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132324 s, 79.2 MB/s 2024-04-17T10:33:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153189 s, 68.4 MB/s 2024-04-17T10:33:48.514 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.515 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148921 s, 70.4 MB/s 2024-04-17T10:33:48.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.532 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015615 s, 67.2 MB/s 2024-04-17T10:33:48.548 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.548 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.548 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143878 s, 72.9 MB/s 2024-04-17T10:33:48.565 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.566 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148816 s, 70.5 MB/s 2024-04-17T10:33:48.581 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.581 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.581 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014219 s, 73.7 MB/s 2024-04-17T10:33:48.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.598 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153049 s, 68.5 MB/s 2024-04-17T10:33:48.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:48.599 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:48.600 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:48.600 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.19 2024-04-17T10:33:48.600 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:48.600 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:48.600 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:48.600 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:48.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141085 s, 74.3 MB/s 2024-04-17T10:33:48.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017379 s, 60.3 MB/s 2024-04-17T10:33:48.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145754 s, 71.9 MB/s 2024-04-17T10:33:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136071 s, 77.1 MB/s 2024-04-17T10:33:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.739 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151429 s, 69.2 MB/s 2024-04-17T10:33:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.754 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136692 s, 76.7 MB/s 2024-04-17T10:33:48.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.769 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013332 s, 78.7 MB/s 2024-04-17T10:33:48.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.784 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133448 s, 78.6 MB/s 2024-04-17T10:33:48.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144428 s, 72.6 MB/s 2024-04-17T10:33:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185692 s, 56.5 MB/s 2024-04-17T10:33:48.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.836 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134389 s, 78.0 MB/s 2024-04-17T10:33:48.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.852 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134488 s, 78.0 MB/s 2024-04-17T10:33:48.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.867 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140261 s, 74.8 MB/s 2024-04-17T10:33:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176609 s, 59.4 MB/s 2024-04-17T10:33:48.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.905 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167081 s, 62.8 MB/s 2024-04-17T10:33:48.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133755 s, 78.4 MB/s 2024-04-17T10:33:48.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158069 s, 66.3 MB/s 2024-04-17T10:33:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.954 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132767 s, 79.0 MB/s 2024-04-17T10:33:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.969 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136587 s, 76.8 MB/s 2024-04-17T10:33:48.984 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.985 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138603 s, 75.7 MB/s 2024-04-17T10:33:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:48.999 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132175 s, 79.3 MB/s 2024-04-17T10:33:49.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.017 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153983 s, 68.1 MB/s 2024-04-17T10:33:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133549 s, 78.5 MB/s 2024-04-17T10:33:49.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.048 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143074 s, 73.3 MB/s 2024-04-17T10:33:49.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.063 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135801 s, 77.2 MB/s 2024-04-17T10:33:49.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168129 s, 62.4 MB/s 2024-04-17T10:33:49.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130414 s, 80.4 MB/s 2024-04-17T10:33:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.112 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139438 s, 75.2 MB/s 2024-04-17T10:33:49.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.126 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116929 s, 89.7 MB/s 2024-04-17T10:33:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132303 s, 79.3 MB/s 2024-04-17T10:33:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119496 s, 87.7 MB/s 2024-04-17T10:33:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142519 s, 73.6 MB/s 2024-04-17T10:33:49.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:49.172 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:49.172 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:49.172 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.20 2024-04-17T10:33:49.172 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:49.172 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:49.172 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:49.172 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.243 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151315 s, 69.3 MB/s 2024-04-17T10:33:49.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.260 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147702 s, 71.0 MB/s 2024-04-17T10:33:49.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152359 s, 68.8 MB/s 2024-04-17T10:33:49.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151482 s, 69.2 MB/s 2024-04-17T10:33:49.309 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.310 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143232 s, 73.2 MB/s 2024-04-17T10:33:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.323 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116915 s, 89.7 MB/s 2024-04-17T10:33:49.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.339 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138207 s, 75.9 MB/s 2024-04-17T10:33:49.358 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.358 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.358 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174182 s, 60.2 MB/s 2024-04-17T10:33:49.373 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.373 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.373 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132999 s, 78.8 MB/s 2024-04-17T10:33:49.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.390 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149452 s, 70.2 MB/s 2024-04-17T10:33:49.405 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.405 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.405 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139023 s, 75.4 MB/s 2024-04-17T10:33:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140882 s, 74.4 MB/s 2024-04-17T10:33:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134988 s, 77.7 MB/s 2024-04-17T10:33:49.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176399 s, 59.4 MB/s 2024-04-17T10:33:49.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142967 s, 73.3 MB/s 2024-04-17T10:33:49.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154624 s, 67.8 MB/s 2024-04-17T10:33:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139744 s, 75.0 MB/s 2024-04-17T10:33:49.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.520 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013368 s, 78.4 MB/s 2024-04-17T10:33:49.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014669 s, 71.5 MB/s 2024-04-17T10:33:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153726 s, 68.2 MB/s 2024-04-17T10:33:49.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013623 s, 77.0 MB/s 2024-04-17T10:33:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.586 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144641 s, 72.5 MB/s 2024-04-17T10:33:49.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.601 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013394 s, 78.3 MB/s 2024-04-17T10:33:49.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.618 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136797 s, 76.7 MB/s 2024-04-17T10:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154293 s, 68.0 MB/s 2024-04-17T10:33:49.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.654 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018783 s, 55.8 MB/s 2024-04-17T10:33:49.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149994 s, 69.9 MB/s 2024-04-17T10:33:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.687 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148381 s, 70.7 MB/s 2024-04-17T10:33:49.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.703 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.703 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144419 s, 72.6 MB/s 2024-04-17T10:33:49.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.718 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133245 s, 78.7 MB/s 2024-04-17T10:33:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.735 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142905 s, 73.4 MB/s 2024-04-17T10:33:49.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.752 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158507 s, 66.2 MB/s 2024-04-17T10:33:49.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:49.754 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:49.754 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:49.754 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.21 2024-04-17T10:33:49.754 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:49.754 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:49.754 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:49.754 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:49.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.822 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142443 s, 73.6 MB/s 2024-04-17T10:33:49.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.840 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160199 s, 65.5 MB/s 2024-04-17T10:33:49.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.857 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154803 s, 67.7 MB/s 2024-04-17T10:33:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151084 s, 69.4 MB/s 2024-04-17T10:33:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.889 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134765 s, 77.8 MB/s 2024-04-17T10:33:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012926 s, 81.1 MB/s 2024-04-17T10:33:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141024 s, 74.4 MB/s 2024-04-17T10:33:49.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.941 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.941 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018745 s, 55.9 MB/s 2024-04-17T10:33:49.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.958 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.958 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152858 s, 68.6 MB/s 2024-04-17T10:33:49.973 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.973 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.973 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136362 s, 76.9 MB/s 2024-04-17T10:33:49.989 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:49.989 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:49.989 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142534 s, 73.6 MB/s 2024-04-17T10:33:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.005 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013818 s, 75.9 MB/s 2024-04-17T10:33:50.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.021 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147817 s, 70.9 MB/s 2024-04-17T10:33:50.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.037 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138848 s, 75.5 MB/s 2024-04-17T10:33:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.053 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136745 s, 76.7 MB/s 2024-04-17T10:33:50.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142562 s, 73.6 MB/s 2024-04-17T10:33:50.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.086 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152123 s, 68.9 MB/s 2024-04-17T10:33:50.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.101 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132983 s, 78.9 MB/s 2024-04-17T10:33:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.116 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136436 s, 76.9 MB/s 2024-04-17T10:33:50.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.135 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173283 s, 60.5 MB/s 2024-04-17T10:33:50.152 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.153 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153399 s, 68.4 MB/s 2024-04-17T10:33:50.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.168 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138404 s, 75.8 MB/s 2024-04-17T10:33:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.185 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153009 s, 68.5 MB/s 2024-04-17T10:33:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152361 s, 68.8 MB/s 2024-04-17T10:33:50.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143149 s, 73.3 MB/s 2024-04-17T10:33:50.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155328 s, 67.5 MB/s 2024-04-17T10:33:50.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181846 s, 57.7 MB/s 2024-04-17T10:33:50.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133343 s, 78.6 MB/s 2024-04-17T10:33:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139853 s, 75.0 MB/s 2024-04-17T10:33:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146533 s, 71.6 MB/s 2024-04-17T10:33:50.317 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.318 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133443 s, 78.6 MB/s 2024-04-17T10:33:50.335 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.335 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.335 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153062 s, 68.5 MB/s 2024-04-17T10:33:50.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:50.336 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:50.336 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:50.336 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.22 2024-04-17T10:33:50.337 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:50.337 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:50.337 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:50.337 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01733 s, 60.5 MB/s 2024-04-17T10:33:50.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.430 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018023 s, 58.2 MB/s 2024-04-17T10:33:50.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.445 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138687 s, 75.6 MB/s 2024-04-17T10:33:50.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140313 s, 74.7 MB/s 2024-04-17T10:33:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.481 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180234 s, 58.2 MB/s 2024-04-17T10:33:50.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153043 s, 68.5 MB/s 2024-04-17T10:33:50.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017512 s, 59.9 MB/s 2024-04-17T10:33:50.533 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.533 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.533 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135768 s, 77.2 MB/s 2024-04-17T10:33:50.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.551 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015787 s, 66.4 MB/s 2024-04-17T10:33:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161649 s, 64.9 MB/s 2024-04-17T10:33:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154355 s, 67.9 MB/s 2024-04-17T10:33:50.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.606 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177838 s, 59.0 MB/s 2024-04-17T10:33:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.624 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161372 s, 65.0 MB/s 2024-04-17T10:33:50.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.641 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150942 s, 69.5 MB/s 2024-04-17T10:33:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.657 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141841 s, 73.9 MB/s 2024-04-17T10:33:50.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192876 s, 54.4 MB/s 2024-04-17T10:33:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.698 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183109 s, 57.3 MB/s 2024-04-17T10:33:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136375 s, 76.9 MB/s 2024-04-17T10:33:50.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.731 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149395 s, 70.2 MB/s 2024-04-17T10:33:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136563 s, 76.8 MB/s 2024-04-17T10:33:50.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.762 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141424 s, 74.1 MB/s 2024-04-17T10:33:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015263 s, 68.7 MB/s 2024-04-17T10:33:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.796 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015161 s, 69.2 MB/s 2024-04-17T10:33:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.814 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163273 s, 64.2 MB/s 2024-04-17T10:33:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.830 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138458 s, 75.7 MB/s 2024-04-17T10:33:50.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.843 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115951 s, 90.4 MB/s 2024-04-17T10:33:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.858 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132646 s, 79.1 MB/s 2024-04-17T10:33:50.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.874 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138432 s, 75.7 MB/s 2024-04-17T10:33:50.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.890 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139195 s, 75.3 MB/s 2024-04-17T10:33:50.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.907 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158585 s, 66.1 MB/s 2024-04-17T10:33:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137263 s, 76.4 MB/s 2024-04-17T10:33:50.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:50.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:50.940 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149858 s, 70.0 MB/s 2024-04-17T10:33:50.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:50.941 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:50.941 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:50.941 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.23 2024-04-17T10:33:50.941 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:50.941 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:50.941 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:50.941 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:51.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.014 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177684 s, 59.0 MB/s 2024-04-17T10:33:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146906 s, 71.4 MB/s 2024-04-17T10:33:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144292 s, 72.7 MB/s 2024-04-17T10:33:51.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.064 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153403 s, 68.4 MB/s 2024-04-17T10:33:51.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.080 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150938 s, 69.5 MB/s 2024-04-17T10:33:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137785 s, 76.1 MB/s 2024-04-17T10:33:51.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.111 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.111 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135849 s, 77.2 MB/s 2024-04-17T10:33:51.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.129 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157191 s, 66.7 MB/s 2024-04-17T10:33:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.144 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135399 s, 77.4 MB/s 2024-04-17T10:33:51.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138282 s, 75.8 MB/s 2024-04-17T10:33:51.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.175 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131436 s, 79.8 MB/s 2024-04-17T10:33:51.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145255 s, 72.2 MB/s 2024-04-17T10:33:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173053 s, 60.6 MB/s 2024-04-17T10:33:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142008 s, 73.8 MB/s 2024-04-17T10:33:51.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.245 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174702 s, 60.0 MB/s 2024-04-17T10:33:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155786 s, 67.3 MB/s 2024-04-17T10:33:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147344 s, 71.2 MB/s 2024-04-17T10:33:51.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.300 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197548 s, 53.1 MB/s 2024-04-17T10:33:51.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013096 s, 80.1 MB/s 2024-04-17T10:33:51.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135876 s, 77.2 MB/s 2024-04-17T10:33:51.346 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.347 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141865 s, 73.9 MB/s 2024-04-17T10:33:51.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.362 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136067 s, 77.1 MB/s 2024-04-17T10:33:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136833 s, 76.6 MB/s 2024-04-17T10:33:51.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.395 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155781 s, 67.3 MB/s 2024-04-17T10:33:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.409 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120808 s, 86.8 MB/s 2024-04-17T10:33:51.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153247 s, 68.4 MB/s 2024-04-17T10:33:51.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.443 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147422 s, 71.1 MB/s 2024-04-17T10:33:51.460 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.460 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.460 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152934 s, 68.6 MB/s 2024-04-17T10:33:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.476 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143428 s, 73.1 MB/s 2024-04-17T10:33:51.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.492 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149117 s, 70.3 MB/s 2024-04-17T10:33:51.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.508 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137915 s, 76.0 MB/s 2024-04-17T10:33:51.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.525 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152383 s, 68.8 MB/s 2024-04-17T10:33:51.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:51.527 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:51.527 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:51.527 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.24 2024-04-17T10:33:51.527 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:51.527 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:51.527 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:51.527 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:51.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.598 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139993 s, 74.9 MB/s 2024-04-17T10:33:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013164 s, 79.7 MB/s 2024-04-17T10:33:51.628 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.628 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.628 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132621 s, 79.1 MB/s 2024-04-17T10:33:51.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.645 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143562 s, 73.0 MB/s 2024-04-17T10:33:51.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.665 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190593 s, 55.0 MB/s 2024-04-17T10:33:51.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.681 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140744 s, 74.5 MB/s 2024-04-17T10:33:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.698 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152326 s, 68.8 MB/s 2024-04-17T10:33:51.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141829 s, 73.9 MB/s 2024-04-17T10:33:51.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.733 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173876 s, 60.3 MB/s 2024-04-17T10:33:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014433 s, 72.7 MB/s 2024-04-17T10:33:51.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.766 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146488 s, 71.6 MB/s 2024-04-17T10:33:51.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014705 s, 71.3 MB/s 2024-04-17T10:33:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.802 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180375 s, 58.1 MB/s 2024-04-17T10:33:51.819 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.819 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.819 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148625 s, 70.6 MB/s 2024-04-17T10:33:51.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.837 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159249 s, 65.8 MB/s 2024-04-17T10:33:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.853 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145291 s, 72.2 MB/s 2024-04-17T10:33:51.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.868 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135926 s, 77.1 MB/s 2024-04-17T10:33:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.889 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184306 s, 56.9 MB/s 2024-04-17T10:33:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.904 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141228 s, 74.2 MB/s 2024-04-17T10:33:51.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154578 s, 67.8 MB/s 2024-04-17T10:33:51.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151922 s, 69.0 MB/s 2024-04-17T10:33:51.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.956 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153663 s, 68.2 MB/s 2024-04-17T10:33:51.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.971 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013781 s, 76.1 MB/s 2024-04-17T10:33:51.991 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:51.992 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018519 s, 56.6 MB/s 2024-04-17T10:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.007 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137836 s, 76.1 MB/s 2024-04-17T10:33:52.023 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.023 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.023 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013826 s, 75.8 MB/s 2024-04-17T10:33:52.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.041 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016382 s, 64.0 MB/s 2024-04-17T10:33:52.056 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.056 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.056 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132811 s, 79.0 MB/s 2024-04-17T10:33:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.073 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156387 s, 67.1 MB/s 2024-04-17T10:33:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.089 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139484 s, 75.2 MB/s 2024-04-17T10:33:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.103 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011978 s, 87.5 MB/s 2024-04-17T10:33:52.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.121 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157474 s, 66.6 MB/s 2024-04-17T10:33:52.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:52.122 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:52.122 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:52.122 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.25 2024-04-17T10:33:52.122 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:52.122 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:52.122 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:52.122 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.199 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187275 s, 56.0 MB/s 2024-04-17T10:33:52.215 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.215 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.215 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014044 s, 74.7 MB/s 2024-04-17T10:33:52.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.232 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158314 s, 66.2 MB/s 2024-04-17T10:33:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186448 s, 56.2 MB/s 2024-04-17T10:33:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133745 s, 78.4 MB/s 2024-04-17T10:33:52.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.285 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154341 s, 67.9 MB/s 2024-04-17T10:33:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149965 s, 69.9 MB/s 2024-04-17T10:33:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.320 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158233 s, 66.3 MB/s 2024-04-17T10:33:52.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.337 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154892 s, 67.7 MB/s 2024-04-17T10:33:52.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015448 s, 67.9 MB/s 2024-04-17T10:33:52.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.370 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136035 s, 77.1 MB/s 2024-04-17T10:33:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.386 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142019 s, 73.8 MB/s 2024-04-17T10:33:52.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.403 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152677 s, 68.7 MB/s 2024-04-17T10:33:52.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146691 s, 71.5 MB/s 2024-04-17T10:33:52.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.436 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142665 s, 73.5 MB/s 2024-04-17T10:33:52.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.453 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148616 s, 70.6 MB/s 2024-04-17T10:33:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139409 s, 75.2 MB/s 2024-04-17T10:33:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.485 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147911 s, 70.9 MB/s 2024-04-17T10:33:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.500 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134765 s, 77.8 MB/s 2024-04-17T10:33:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.515 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136377 s, 76.9 MB/s 2024-04-17T10:33:52.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.532 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144673 s, 72.5 MB/s 2024-04-17T10:33:52.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.547 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139204 s, 75.3 MB/s 2024-04-17T10:33:52.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182415 s, 57.5 MB/s 2024-04-17T10:33:52.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.589 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019068 s, 55.0 MB/s 2024-04-17T10:33:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013564 s, 77.3 MB/s 2024-04-17T10:33:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.619 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137266 s, 76.4 MB/s 2024-04-17T10:33:52.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.635 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139659 s, 75.1 MB/s 2024-04-17T10:33:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.651 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141132 s, 74.3 MB/s 2024-04-17T10:33:52.666 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.666 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.666 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013285 s, 78.9 MB/s 2024-04-17T10:33:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.684 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157413 s, 66.6 MB/s 2024-04-17T10:33:52.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.699 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136417 s, 76.9 MB/s 2024-04-17T10:33:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139431 s, 75.2 MB/s 2024-04-17T10:33:52.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:52.716 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:52.716 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:52.717 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.26 2024-04-17T10:33:52.717 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:52.717 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:52.717 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:52.717 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:52.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.789 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118858 s, 88.2 MB/s 2024-04-17T10:33:52.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.805 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014973 s, 70.0 MB/s 2024-04-17T10:33:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128975 s, 81.3 MB/s 2024-04-17T10:33:52.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.836 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139193 s, 75.3 MB/s 2024-04-17T10:33:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.851 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013878 s, 75.6 MB/s 2024-04-17T10:33:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159173 s, 65.9 MB/s 2024-04-17T10:33:52.885 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.885 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.885 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145495 s, 72.1 MB/s 2024-04-17T10:33:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137948 s, 76.0 MB/s 2024-04-17T10:33:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.916 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131521 s, 79.7 MB/s 2024-04-17T10:33:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141966 s, 73.9 MB/s 2024-04-17T10:33:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151605 s, 69.2 MB/s 2024-04-17T10:33:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142055 s, 73.8 MB/s 2024-04-17T10:33:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139116 s, 75.4 MB/s 2024-04-17T10:33:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:52.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156408 s, 67.0 MB/s 2024-04-17T10:33:53.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152948 s, 68.6 MB/s 2024-04-17T10:33:53.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017576 s, 59.7 MB/s 2024-04-17T10:33:53.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.052 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159827 s, 65.6 MB/s 2024-04-17T10:33:53.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.068 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138328 s, 75.8 MB/s 2024-04-17T10:33:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015055 s, 69.6 MB/s 2024-04-17T10:33:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.101 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140428 s, 74.7 MB/s 2024-04-17T10:33:53.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.116 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140894 s, 74.4 MB/s 2024-04-17T10:33:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188427 s, 55.6 MB/s 2024-04-17T10:33:53.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01482 s, 70.8 MB/s 2024-04-17T10:33:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152831 s, 68.6 MB/s 2024-04-17T10:33:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137147 s, 76.5 MB/s 2024-04-17T10:33:53.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.204 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154891 s, 67.7 MB/s 2024-04-17T10:33:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.219 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136751 s, 76.7 MB/s 2024-04-17T10:33:53.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118167 s, 88.7 MB/s 2024-04-17T10:33:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208268 s, 50.3 MB/s 2024-04-17T10:33:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138156 s, 75.9 MB/s 2024-04-17T10:33:53.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.288 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014686 s, 71.4 MB/s 2024-04-17T10:33:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.304 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146907 s, 71.4 MB/s 2024-04-17T10:33:53.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:53.305 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:53.306 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:53.306 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.27 2024-04-17T10:33:53.306 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:53.306 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:53.306 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:53.306 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:53.380 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.381 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160531 s, 65.3 MB/s 2024-04-17T10:33:53.397 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.397 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.397 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144783 s, 72.4 MB/s 2024-04-17T10:33:53.416 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.416 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179835 s, 58.3 MB/s 2024-04-17T10:33:53.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.433 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144761 s, 72.4 MB/s 2024-04-17T10:33:53.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.453 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182349 s, 57.5 MB/s 2024-04-17T10:33:53.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.470 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152341 s, 68.8 MB/s 2024-04-17T10:33:53.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.487 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151079 s, 69.4 MB/s 2024-04-17T10:33:53.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.503 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140072 s, 74.9 MB/s 2024-04-17T10:33:53.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.518 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137942 s, 76.0 MB/s 2024-04-17T10:33:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.538 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173857 s, 60.3 MB/s 2024-04-17T10:33:53.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.556 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157672 s, 66.5 MB/s 2024-04-17T10:33:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156483 s, 67.0 MB/s 2024-04-17T10:33:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.588 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139567 s, 75.1 MB/s 2024-04-17T10:33:53.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.606 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154133 s, 68.0 MB/s 2024-04-17T10:33:53.621 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.621 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.621 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013883 s, 75.5 MB/s 2024-04-17T10:33:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178041 s, 58.9 MB/s 2024-04-17T10:33:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.658 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146838 s, 71.4 MB/s 2024-04-17T10:33:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141314 s, 74.2 MB/s 2024-04-17T10:33:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143317 s, 73.2 MB/s 2024-04-17T10:33:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181481 s, 57.8 MB/s 2024-04-17T10:33:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.725 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141562 s, 74.1 MB/s 2024-04-17T10:33:53.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.743 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152093 s, 68.9 MB/s 2024-04-17T10:33:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.760 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153469 s, 68.3 MB/s 2024-04-17T10:33:53.775 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.775 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.775 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135284 s, 77.5 MB/s 2024-04-17T10:33:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.796 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191141 s, 54.9 MB/s 2024-04-17T10:33:53.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.813 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.813 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154508 s, 67.9 MB/s 2024-04-17T10:33:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.829 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138965 s, 75.5 MB/s 2024-04-17T10:33:53.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.845 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142877 s, 73.4 MB/s 2024-04-17T10:33:53.860 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138256 s, 75.8 MB/s 2024-04-17T10:33:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.877 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149679 s, 70.1 MB/s 2024-04-17T10:33:53.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.896 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167135 s, 62.7 MB/s 2024-04-17T10:33:53.912 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.913 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147185 s, 71.2 MB/s 2024-04-17T10:33:53.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:53.914 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:53.914 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:53.914 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.28 2024-04-17T10:33:53.914 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:53.914 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:53.914 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:53.914 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:53.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:53.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:53.986 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169071 s, 62.0 MB/s 2024-04-17T10:33:54.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.001 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136521 s, 76.8 MB/s 2024-04-17T10:33:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.018 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152094 s, 68.9 MB/s 2024-04-17T10:33:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.033 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133462 s, 78.6 MB/s 2024-04-17T10:33:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.049 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138525 s, 75.7 MB/s 2024-04-17T10:33:54.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.065 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141574 s, 74.1 MB/s 2024-04-17T10:33:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.086 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019021 s, 55.1 MB/s 2024-04-17T10:33:54.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.102 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148288 s, 70.7 MB/s 2024-04-17T10:33:54.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.120 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154606 s, 67.8 MB/s 2024-04-17T10:33:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.139 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178611 s, 58.7 MB/s 2024-04-17T10:33:54.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136698 s, 76.7 MB/s 2024-04-17T10:33:54.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146359 s, 71.6 MB/s 2024-04-17T10:33:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133587 s, 78.5 MB/s 2024-04-17T10:33:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146677 s, 71.5 MB/s 2024-04-17T10:33:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138184 s, 75.9 MB/s 2024-04-17T10:33:54.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.238 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180382 s, 58.1 MB/s 2024-04-17T10:33:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.254 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140188 s, 74.8 MB/s 2024-04-17T10:33:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.270 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138291 s, 75.8 MB/s 2024-04-17T10:33:54.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014758 s, 71.1 MB/s 2024-04-17T10:33:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.302 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138087 s, 75.9 MB/s 2024-04-17T10:33:54.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.319 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148023 s, 70.8 MB/s 2024-04-17T10:33:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.334 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137024 s, 76.5 MB/s 2024-04-17T10:33:54.349 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.350 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137885 s, 76.0 MB/s 2024-04-17T10:33:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.366 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.367 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015053 s, 69.7 MB/s 2024-04-17T10:33:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135432 s, 77.4 MB/s 2024-04-17T10:33:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142016 s, 73.8 MB/s 2024-04-17T10:33:54.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.413 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134907 s, 77.7 MB/s 2024-04-17T10:33:54.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.429 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141473 s, 74.1 MB/s 2024-04-17T10:33:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.444 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132405 s, 79.2 MB/s 2024-04-17T10:33:54.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.462 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152001 s, 69.0 MB/s 2024-04-17T10:33:54.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.477 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138023 s, 76.0 MB/s 2024-04-17T10:33:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150578 s, 69.6 MB/s 2024-04-17T10:33:54.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:54.495 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:54.495 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:54.495 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.29 2024-04-17T10:33:54.495 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:54.495 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:54.495 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:54.495 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:54.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150717 s, 69.6 MB/s 2024-04-17T10:33:54.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.585 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149215 s, 70.3 MB/s 2024-04-17T10:33:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.602 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156677 s, 66.9 MB/s 2024-04-17T10:33:54.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.618 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139815 s, 75.0 MB/s 2024-04-17T10:33:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118276 s, 88.7 MB/s 2024-04-17T10:33:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.648 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146652 s, 71.5 MB/s 2024-04-17T10:33:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.663 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133274 s, 78.7 MB/s 2024-04-17T10:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.679 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144321 s, 72.7 MB/s 2024-04-17T10:33:54.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.699 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178302 s, 58.8 MB/s 2024-04-17T10:33:54.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.716 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146921 s, 71.4 MB/s 2024-04-17T10:33:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.732 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155021 s, 67.6 MB/s 2024-04-17T10:33:54.748 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.748 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.748 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140578 s, 74.6 MB/s 2024-04-17T10:33:54.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.764 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135956 s, 77.1 MB/s 2024-04-17T10:33:54.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.780 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139592 s, 75.1 MB/s 2024-04-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139792 s, 75.0 MB/s 2024-04-17T10:33:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.811 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141011 s, 74.4 MB/s 2024-04-17T10:33:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.827 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147164 s, 71.3 MB/s 2024-04-17T10:33:54.842 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.843 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132688 s, 79.0 MB/s 2024-04-17T10:33:54.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.859 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.859 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140813 s, 74.5 MB/s 2024-04-17T10:33:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.877 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164577 s, 63.7 MB/s 2024-04-17T10:33:54.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136289 s, 76.9 MB/s 2024-04-17T10:33:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.913 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188528 s, 55.6 MB/s 2024-04-17T10:33:54.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.928 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136691 s, 76.7 MB/s 2024-04-17T10:33:54.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.944 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139071 s, 75.4 MB/s 2024-04-17T10:33:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146494 s, 71.6 MB/s 2024-04-17T10:33:54.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.978 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152299 s, 68.8 MB/s 2024-04-17T10:33:54.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:54.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:54.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154594 s, 67.8 MB/s 2024-04-17T10:33:55.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.012 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.012 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151827 s, 69.1 MB/s 2024-04-17T10:33:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150414 s, 69.7 MB/s 2024-04-17T10:33:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153423 s, 68.3 MB/s 2024-04-17T10:33:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134329 s, 78.1 MB/s 2024-04-17T10:33:55.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157332 s, 66.6 MB/s 2024-04-17T10:33:55.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:55.080 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:55.080 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:55.080 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.30 2024-04-17T10:33:55.080 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:55.080 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:55.080 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:55.081 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:55.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152489 s, 68.8 MB/s 2024-04-17T10:33:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139046 s, 75.4 MB/s 2024-04-17T10:33:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.189 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156132 s, 67.2 MB/s 2024-04-17T10:33:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.204 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138584 s, 75.7 MB/s 2024-04-17T10:33:55.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157965 s, 66.4 MB/s 2024-04-17T10:33:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.239 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147454 s, 71.1 MB/s 2024-04-17T10:33:55.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.254 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013659 s, 76.8 MB/s 2024-04-17T10:33:55.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.270 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145085 s, 72.3 MB/s 2024-04-17T10:33:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167965 s, 62.4 MB/s 2024-04-17T10:33:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155253 s, 67.5 MB/s 2024-04-17T10:33:55.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.321 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124318 s, 84.3 MB/s 2024-04-17T10:33:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152984 s, 68.5 MB/s 2024-04-17T10:33:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154956 s, 67.7 MB/s 2024-04-17T10:33:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145792 s, 71.9 MB/s 2024-04-17T10:33:55.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.389 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155566 s, 67.4 MB/s 2024-04-17T10:33:55.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.404 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137844 s, 76.1 MB/s 2024-04-17T10:33:55.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142732 s, 73.5 MB/s 2024-04-17T10:33:55.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.436 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134205 s, 78.1 MB/s 2024-04-17T10:33:55.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.451 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137667 s, 76.2 MB/s 2024-04-17T10:33:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149388 s, 70.2 MB/s 2024-04-17T10:33:55.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.485 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152065 s, 69.0 MB/s 2024-04-17T10:33:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.501 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141937 s, 73.9 MB/s 2024-04-17T10:33:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.518 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153373 s, 68.4 MB/s 2024-04-17T10:33:55.535 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.535 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.535 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151859 s, 69.0 MB/s 2024-04-17T10:33:55.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.550 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133162 s, 78.7 MB/s 2024-04-17T10:33:55.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.566 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.566 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141971 s, 73.9 MB/s 2024-04-17T10:33:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014626 s, 71.7 MB/s 2024-04-17T10:33:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.597 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122765 s, 85.4 MB/s 2024-04-17T10:33:55.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149834 s, 70.0 MB/s 2024-04-17T10:33:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152034 s, 69.0 MB/s 2024-04-17T10:33:55.652 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.652 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.652 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185129 s, 56.6 MB/s 2024-04-17T10:33:55.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.668 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144103 s, 72.8 MB/s 2024-04-17T10:33:55.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.31 2024-04-17T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:33:55.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.738 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139893 s, 75.0 MB/s 2024-04-17T10:33:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.758 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180584 s, 58.1 MB/s 2024-04-17T10:33:55.778 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.778 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.778 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180082 s, 58.2 MB/s 2024-04-17T10:33:55.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.793 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135655 s, 77.3 MB/s 2024-04-17T10:33:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143895 s, 72.9 MB/s 2024-04-17T10:33:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152834 s, 68.6 MB/s 2024-04-17T10:33:55.842 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.842 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.842 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013993 s, 74.9 MB/s 2024-04-17T10:33:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.859 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144678 s, 72.5 MB/s 2024-04-17T10:33:55.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.875 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.875 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014739 s, 71.1 MB/s 2024-04-17T10:33:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.891 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146525 s, 71.6 MB/s 2024-04-17T10:33:55.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.908 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148319 s, 70.7 MB/s 2024-04-17T10:33:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.924 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014383 s, 72.9 MB/s 2024-04-17T10:33:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151761 s, 69.1 MB/s 2024-04-17T10:33:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.957 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135658 s, 77.3 MB/s 2024-04-17T10:33:55.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.972 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013961 s, 75.1 MB/s 2024-04-17T10:33:55.988 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:55.988 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:55.988 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147247 s, 71.2 MB/s 2024-04-17T10:33:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.003 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131333 s, 79.8 MB/s 2024-04-17T10:33:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.022 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168643 s, 62.2 MB/s 2024-04-17T10:33:56.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.039 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152042 s, 69.0 MB/s 2024-04-17T10:33:56.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.054 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134377 s, 78.0 MB/s 2024-04-17T10:33:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.071 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015359 s, 68.3 MB/s 2024-04-17T10:33:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.088 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153321 s, 68.4 MB/s 2024-04-17T10:33:56.104 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.105 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144849 s, 72.4 MB/s 2024-04-17T10:33:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.121 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140418 s, 74.7 MB/s 2024-04-17T10:33:56.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.136 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140942 s, 74.4 MB/s 2024-04-17T10:33:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135057 s, 77.6 MB/s 2024-04-17T10:33:56.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.168 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141757 s, 74.0 MB/s 2024-04-17T10:33:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152681 s, 68.7 MB/s 2024-04-17T10:33:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015355 s, 68.3 MB/s 2024-04-17T10:33:56.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.219 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149636 s, 70.1 MB/s 2024-04-17T10:33:56.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146404 s, 71.6 MB/s 2024-04-17T10:33:56.250 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:33:56.250 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:33:56.250 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138397 s, 75.8 MB/s 2024-04-17T10:33:56.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:56.252 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.8:0 scrub start /scrub_control_test_path recursive 2024-04-17T10:33:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce9200640 1 Processor -- start 2024-04-17T10:33:56.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce9200640 1 -- start start 2024-04-17T10:33:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce9200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ce415da40 0x7f3ce40d2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce9200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ce406cc00 0x7f3ce40d0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce9200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce415de20 con 0x7f3ce4099a50 2024-04-17T10:33:56.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce9200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce40d0ac0 con 0x7f3ce415da40 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce9200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce406be40 con 0x7f3ce406cc00 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ce415da40 0x7f3ce40d2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3ce4099a50 0x7f3ce4099e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35630/0 (socket says 172.21.15.67:35630) 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce3e00640 1 -- 172.21.15.67:0/2012992767 learned_addr learned my addr 172.21.15.67:0/2012992767 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/2012992767 >> v2:172.21.15.86:3300/0 conn(0x7f3ce406cc00 0x7f3ce40d0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce3400640 1 -- 172.21.15.67:0/2012992767 >> v2:172.21.15.86:3300/0 conn(0x7f3ce406cc00 msgr2=0x7f3ce40d0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce3400640 1 --2- 172.21.15.67:0/2012992767 >> v2:172.21.15.86:3300/0 conn(0x7f3ce406cc00 0x7f3ce40d0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce3400640 1 -- 172.21.15.67:0/2012992767 >> v1:172.21.15.67:6789/0 conn(0x7f3ce4099a50 legacy=0x7f3ce4099e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce2a00640 1 -- 172.21.15.67:0/2012992767 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872447657 0 0) 0x7f3ce415de20 con 0x7f3ce4099a50 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3ce3400640 1 -- 172.21.15.67:0/2012992767 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce406c120 con 0x7f3ce415da40 2024-04-17T10:33:56.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.453+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/2012992767 >> v2:172.21.15.86:3300/0 conn(0x7f3ce406cc00 0x7f3ce40d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce3400640 1 --2- 172.21.15.67:0/2012992767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce40d2280 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40684a0 tx=0x7f3cd401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992cbf0e2cb60e5b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce2a00640 1 -- 172.21.15.67:0/2012992767 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40ab650 con 0x7f3ce415da40 2024-04-17T10:33:56.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce2a00640 1 -- 172.21.15.67:0/2012992767 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd40b0070 con 0x7f3ce415da40 2024-04-17T10:33:56.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce2a00640 1 -- 172.21.15.67:0/2012992767 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40aba10 con 0x7f3ce415da40 2024-04-17T10:33:56.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 -- 172.21.15.67:0/2012992767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 msgr2=0x7f3ce40d2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 --2- 172.21.15.67:0/2012992767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce40d2280 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f3ce40684a0 tx=0x7f3cd401c450 comp rx=0 tx=0).stop 2024-04-17T10:33:56.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 -- 172.21.15.67:0/2012992767 shutdown_connections 2024-04-17T10:33:56.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 --2- 172.21.15.67:0/2012992767 >> v2:172.21.15.86:3300/0 conn(0x7f3ce406cc00 0x7f3ce40d0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 --2- 172.21.15.67:0/2012992767 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce40d2280 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 -- 172.21.15.67:0/2012992767 >> 172.21.15.67:0/2012992767 conn(0x7f3ce40d16b0 msgr2=0x7f3ce40d1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 -- 172.21.15.67:0/2012992767 shutdown_connections 2024-04-17T10:33:56.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 -- 172.21.15.67:0/2012992767 wait complete. 2024-04-17T10:33:56.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 Processor -- start 2024-04-17T10:33:56.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.457+0000 7f3ce9200640 1 -- start start 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce406cc00 0x7f3ce41554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce4099a50 0x7f3ce415b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce415be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce406d190 con 0x7f3ce415da40 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce406d010 con 0x7f3ce406cc00 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40cc510 con 0x7f3ce4099a50 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce415be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce415be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42176/0 (socket says 172.21.15.67:42176) 2024-04-17T10:33:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 -- 172.21.15.67:0/4238376609 learned_addr learned my addr 172.21.15.67:0/4238376609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce4099a50 0x7f3ce415b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3400640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce406cc00 0x7f3ce41554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 -- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce4099a50 msgr2=0x7f3ce415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce4099a50 0x7f3ce415b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 -- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce406cc00 msgr2=0x7f3ce41554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce406cc00 0x7f3ce41554a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 -- 172.21.15.67:0/4238376609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce40c8900 con 0x7f3ce415da40 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce3e00640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce415be60 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c7a0 tx=0x7f3cdc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3cdb400640 1 -- 172.21.15.67:0/4238376609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc018070 con 0x7f3ce415da40 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3cdb400640 1 -- 172.21.15.67:0/4238376609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc00a640 con 0x7f3ce415da40 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce4155da0 con 0x7f3ce415da40 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3cdb400640 1 -- 172.21.15.67:0/4238376609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00aa70 con 0x7f3ce415da40 2024-04-17T10:33:56.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.461+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce4156360 con 0x7f3ce415da40 2024-04-17T10:33:56.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdb400640 1 -- 172.21.15.67:0/4238376609 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3cdc01b070 con 0x7f3ce415da40 2024-04-17T10:33:56.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdb400640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc0066f60 0x7f3cc0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdb400640 1 -- 172.21.15.67:0/4238376609 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3cdc0846f0 con 0x7f3ce415da40 2024-04-17T10:33:56.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3ce3400640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc0066f60 0x7f3cc0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 Processor -- start 2024-04-17T10:33:56.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 -- start start 2024-04-17T10:33:56.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3ce3400640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc0066f60 0x7f3cc0069420 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40af000 tx=0x7f3cd40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e487d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544e48d10 0x55a544dfefe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544dff520 0x55a544dff900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a544c4cd50 con 0x55a544e48d10 2024-04-17T10:33:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a544bd4e40 con 0x55a544e483f0 2024-04-17T10:33:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a544c48df0 con 0x55a544dff520 2024-04-17T10:33:56.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544e48d10 0x55a544dfefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3ce3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544e48d10 0x55a544dfefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42178/0 (socket says 172.21.15.67:42178) 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 -- 172.21.15.67:0/2147449966 learned_addr learned my addr 172.21.15.67:0/2147449966 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3ce3e00640 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544dff520 0x55a544dff900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 -- 172.21.15.67:0/2147449966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544dff520 msgr2=0x55a544dff900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544dff520 0x55a544dff900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 -- 172.21.15.67:0/2147449966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 msgr2=0x55a544e487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e487d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 -- 172.21.15.67:0/2147449966 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a544be9d90 con 0x55a544e48d10 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.465+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544e48d10 0x55a544dfefe0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009900 tx=0x7f3ccc009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f3ff511e2ff243 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cd3e00640 1 -- 172.21.15.67:0/2147449966 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc019070 con 0x55a544e48d10 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cd3e00640 1 -- 172.21.15.67:0/2147449966 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc011aa0 con 0x55a544e48d10 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cd3e00640 1 -- 172.21.15.67:0/2147449966 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc00f040 con 0x55a544e48d10 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- 172.21.15.67:0/2147449966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544e48d10 msgr2=0x55a544dfefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544e48d10 0x55a544dfefe0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009900 tx=0x7f3ccc009dc0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- 172.21.15.67:0/2147449966 shutdown_connections 2024-04-17T10:33:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544dff520 0x55a544dff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544e48d10 0x55a544dfefe0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/2147449966 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- 172.21.15.67:0/2147449966 >> 172.21.15.67:0/2147449966 conn(0x55a544dfb4d0 msgr2=0x55a544dfd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- 172.21.15.67:0/2147449966 shutdown_connections 2024-04-17T10:33:56.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- 172.21.15.67:0/2147449966 wait complete. 2024-04-17T10:33:56.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- start start 2024-04-17T10:33:56.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 Processor -- start 2024-04-17T10:33:56.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544e05040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e36600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a544bf2610 con 0x55a544dff520 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a544c079a0 con 0x55a544e483f0 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a544c51380 con 0x55a544e48d10 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cedbc7000 10 client.0 _ll_register_callbacks cb 0x55a544d010d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544e05040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544e05040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42192/0 (socket says 172.21.15.67:42192) 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 -- 172.21.15.67:0/4034410183 learned_addr learned my addr 172.21.15.67:0/4034410183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3e00640 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e36600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 -- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 msgr2=0x55a544e36600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e36600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 -- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 msgr2=0x55a544e360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 -- 172.21.15.67:0/4034410183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a544c47d40 con 0x55a544dff520 2024-04-17T10:33:56.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3ce3400640 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544e05040 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3cd401a080 tx=0x7f3cd401ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98eb6706a9c6c56d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.469+0000 7f3cd0c00640 1 -- 172.21.15.67:0/4034410183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40cf060 con 0x55a544dff520 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ccc011aa0 con 0x55a544dff520 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 1 -- 172.21.15.67:0/4034410183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3cd40c14d0 con 0x55a544dff520 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 1 -- 172.21.15.67:0/4034410183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cd40d5050 con 0x55a544dff520 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 10 client.8832 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55a544ccff00 con 0x55a544dff520 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 1 -- 172.21.15.67:0/4034410183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc011aa0 con 0x55a544dff520 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 1 -- 172.21.15.67:0/4034410183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3cd40d8020 con 0x55a544dff520 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 10 client.8832 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 10 client.8832 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 20 client.8832 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4f570 0x55a544e4f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 4 client.8832 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a544be8a20 con 0x55a544e4f570 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4f570 0x55a544e4f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 10 client.8832 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4f570 0x55a544e4f980 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2a6ec6e8f40ae7a server_cookie=ad283f0f25e9a624 in_seq=0 out_seq=0 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 1 -- 172.21.15.67:0/4034410183 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a544be8a20 con 0x55a544e4f570 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 10 client.8832 handle_command_reply: tid=0 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 client.8832 shutdown 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 msgr2=0x55a544e05040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544e05040 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f3cd401a080 tx=0x7f3cd401ac90 comp rx=0 tx=0).stop 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 shutdown_connections 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4f570 0x55a544e4f980 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e36600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544e360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/4034410183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544e05040 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cd0c00640 0 client.8832 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 >> 172.21.15.67:0/4034410183 conn(0x55a544dfb4d0 msgr2=0x55a544dfe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 shutdown_connections 2024-04-17T10:33:56.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 1 -- 172.21.15.67:0/4034410183 wait complete. 2024-04-17T10:33:56.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.473+0000 7f3cedbc7000 20 client.8832 trim_cache size 0 max 16384 2024-04-17T10:33:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 Processor -- start 2024-04-17T10:33:56.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 -- start start 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544dfec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544dfd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544dfde00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce4202f30 con 0x55a544dff520 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce4202130 con 0x55a544e483f0 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40b0370 con 0x55a544e48d10 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544dfec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544dfec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42204/0 (socket says 172.21.15.67:42204) 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 -- 172.21.15.67:0/1403090929 learned_addr learned my addr 172.21.15.67:0/1403090929 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3e00640 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544dfde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544dfd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 -- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 msgr2=0x55a544dfde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544dfde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 -- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 msgr2=0x55a544dfd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544dfd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 -- 172.21.15.67:0/1403090929 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a544ca6330 con 0x55a544dff520 2024-04-17T10:33:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.489+0000 7f3ce3400640 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544dfec30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40c1b70 tx=0x7f3cd40c6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f62202648d16079 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cd3400640 1 -- 172.21.15.67:0/1403090929 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40de070 con 0x55a544dff520 2024-04-17T10:33:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cd3400640 1 -- 172.21.15.67:0/1403090929 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd40b0040 con 0x55a544dff520 2024-04-17T10:33:56.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cd3400640 1 -- 172.21.15.67:0/1403090929 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40bc070 con 0x55a544dff520 2024-04-17T10:33:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1403090929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 msgr2=0x55a544dfec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544dfec30 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40c1b70 tx=0x7f3cd40c6000 comp rx=0 tx=0).stop 2024-04-17T10:33:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1403090929 shutdown_connections 2024-04-17T10:33:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544dfde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544dfd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1403090929 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544dfec30 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1403090929 >> 172.21.15.67:0/1403090929 conn(0x55a544dfb4d0 msgr2=0x55a544e04490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1403090929 shutdown_connections 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1403090929 wait complete. 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- start start 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 Processor -- start 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544ea96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544ea9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e4d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a544c03dc0 con 0x55a544dff520 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce40b0970 con 0x55a544e483f0 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a544c4c060 con 0x55a544e48d10 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cedbc7000 10 client.0 _ll_register_callbacks cb 0x55a544c52270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544ea9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544ea9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:40024/0 (socket says 172.21.15.67:40024) 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 -- 172.21.15.67:0/1337158070 learned_addr learned my addr 172.21.15.67:0/1337158070 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3ce3e00640 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e4d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3ce3400640 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544ea96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 -- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 msgr2=0x55a544e4d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e4d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 -- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 msgr2=0x55a544ea96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544ea96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 -- 172.21.15.67:0/1337158070 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a544c195f0 con 0x55a544e483f0 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cdbe00640 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544ea9be0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc011980 tx=0x7f3ccc004740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=360a57b6e9e0baa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.493+0000 7f3cd0c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3300/0 2024-04-17T10:33:56.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 1 -- 172.21.15.67:0/1337158070 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc019040 con 0x55a544e483f0 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 1 -- 172.21.15.67:0/1337158070 <== mon.1 v2:172.21.15.86:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3ccc011cd0 con 0x55a544e483f0 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cd40b0040 con 0x55a544e483f0 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 1 -- 172.21.15.67:0/1337158070 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ccc03a050 con 0x55a544e483f0 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 10 client.8245 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55a544bf3fe0 con 0x55a544e483f0 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 1 -- 172.21.15.67:0/1337158070 <== mon.1 v2:172.21.15.86:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd40b0040 con 0x55a544e483f0 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 1 -- 172.21.15.67:0/1337158070 <== mon.1 v2:172.21.15.86:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3ccc03d020 con 0x55a544e483f0 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 10 client.8245 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 10 client.8245 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 20 client.8245 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4e180 0x55a544e4e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 4 client.8245 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x55a544c5a290 con 0x55a544e4e180 2024-04-17T10:33:56.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3ce3400640 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4e180 0x55a544e4e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 10 client.8245 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3ce3400640 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4e180 0x55a544e4e590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfd379b76a6d8b9f server_cookie=8ae9e3c3572b3b0c in_seq=0 out_seq=0 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 1 -- 172.21.15.67:0/1337158070 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a544c5a290 con 0x55a544e4e180 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cd0c00640 10 client.8245 handle_command_reply: tid=0 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.497+0000 7f3cedbc7000 1 client.8245 shutdown 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 msgr2=0x55a544ea9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544ea9be0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc011980 tx=0x7f3ccc004740 comp rx=0 tx=0).stop 2024-04-17T10:33:56.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 shutdown_connections 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a544e4e180 0x55a544e4e590 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a544e48d10 0x55a544e4d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a544e483f0 0x55a544ea9be0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 --2- 172.21.15.67:0/1337158070 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a544dff520 0x55a544ea96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cd0c00640 0 client.8245 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 >> 172.21.15.67:0/1337158070 conn(0x55a544dfb4d0 msgr2=0x55a544e47150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 shutdown_connections 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 1 -- 172.21.15.67:0/1337158070 wait complete. 2024-04-17T10:33:56.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3cedbc7000 20 client.8245 trim_cache size 0 max 16384 2024-04-17T10:33:56.495 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:56.495 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:33:56.495 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "3662aba3-3c3a-4153-aeeb-8bd2c974cc63", 2024-04-17T10:33:56.495 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:33:56.495 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc0066f60 msgr2=0x7f3cc0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc0066f60 0x7f3cc0069420 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40af000 tx=0x7f3cd40a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 msgr2=0x7f3ce415be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce415be60 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c7a0 tx=0x7f3cdc00cc60 comp rx=0 tx=0).stop 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 shutdown_connections 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc0066f60 0x7f3cc0069420 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce415da40 0x7f3ce415be60 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce4099a50 0x7f3ce415b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 --2- 172.21.15.67:0/4238376609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce406cc00 0x7f3ce41554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 >> 172.21.15.67:0/4238376609 conn(0x7f3ce40d16b0 msgr2=0x7f3ce40b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 shutdown_connections 2024-04-17T10:33:56.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.501+0000 7f3ce9200640 1 -- 172.21.15.67:0/4238376609 wait complete. 2024-04-17T10:33:56.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:56.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.8:0 scrub pause 2024-04-17T10:33:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 Processor -- start 2024-04-17T10:33:56.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 -- start start 2024-04-17T10:33:56.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f909c1547f0 0x7f909c161420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c070400 0x7f909c0ce4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f909c0cea30 0x7f909c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c15df60 con 0x7f909c0cea30 2024-04-17T10:33:56.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f909c161a00 con 0x7f909c1547f0 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a3200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c161ce0 con 0x7f909c070400 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f909c0cea30 0x7f909c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f909c0cea30 0x7f909c06ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42236/0 (socket says 172.21.15.67:42236) 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a2800640 1 -- 172.21.15.67:0/3529421542 learned_addr learned my addr 172.21.15.67:0/3529421542 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.705+0000 7f90a1e00640 1 --2- 172.21.15.67:0/3529421542 >> v2:172.21.15.86:3300/0 conn(0x7f909c1547f0 0x7f909c161420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a1400640 1 --2- 172.21.15.67:0/3529421542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c070400 0x7f909c0ce4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a2800640 1 -- 172.21.15.67:0/3529421542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c070400 msgr2=0x7f909c0ce4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a2800640 1 --2- 172.21.15.67:0/3529421542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c070400 0x7f909c0ce4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a2800640 1 -- 172.21.15.67:0/3529421542 >> v2:172.21.15.86:3300/0 conn(0x7f909c1547f0 msgr2=0x7f909c161420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a2800640 1 --2- 172.21.15.67:0/3529421542 >> v2:172.21.15.86:3300/0 conn(0x7f909c1547f0 0x7f909c161420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a2800640 1 -- 172.21.15.67:0/3529421542 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c161fc0 con 0x7f909c0cea30 2024-04-17T10:33:56.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a2800640 1 --2- 172.21.15.67:0/3529421542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c0cea30 0x7f909c06ac90 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f909c06ba70 tx=0x7f909801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecfb9f2e43345284 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a0a00640 1 -- 172.21.15.67:0/3529421542 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980a5890 con 0x7f909c0cea30 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a0a00640 1 -- 172.21.15.67:0/3529421542 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90980a5a30 con 0x7f909c0cea30 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a0a00640 1 -- 172.21.15.67:0/3529421542 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980a5e00 con 0x7f909c0cea30 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 -- 172.21.15.67:0/3529421542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c0cea30 msgr2=0x7f909c06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 --2- 172.21.15.67:0/3529421542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c0cea30 0x7f909c06ac90 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f909c06ba70 tx=0x7f909801c9f0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 -- 172.21.15.67:0/3529421542 shutdown_connections 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 --2- 172.21.15.67:0/3529421542 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c0cea30 0x7f909c06ac90 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 --2- 172.21.15.67:0/3529421542 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c070400 0x7f909c0ce4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 --2- 172.21.15.67:0/3529421542 >> v2:172.21.15.86:3300/0 conn(0x7f909c1547f0 0x7f909c161420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 -- 172.21.15.67:0/3529421542 >> 172.21.15.67:0/3529421542 conn(0x7f909c06e420 msgr2=0x7f909c06e830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.709+0000 7f90a3200640 1 -- 172.21.15.67:0/3529421542 shutdown_connections 2024-04-17T10:33:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 -- 172.21.15.67:0/3529421542 wait complete. 2024-04-17T10:33:56.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 Processor -- start 2024-04-17T10:33:56.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 -- start start 2024-04-17T10:33:56.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c070400 0x7f909c14c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c0cea30 0x7f909c14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f909c1547f0 0x7f909c0b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0af520 con 0x7f909c070400 2024-04-17T10:33:56.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c06b5e0 con 0x7f909c1547f0 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c0af6a0 con 0x7f909c0cea30 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c070400 0x7f909c14c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c070400 0x7f909c14c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42252/0 (socket says 172.21.15.67:42252) 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 -- 172.21.15.67:0/422754891 learned_addr learned my addr 172.21.15.67:0/422754891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1400640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c0cea30 0x7f909c14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a2800640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f909c1547f0 0x7f909c0b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 -- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c0cea30 msgr2=0x7f909c14dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c0cea30 0x7f909c14dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 -- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f909c1547f0 msgr2=0x7f909c0b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f909c1547f0 0x7f909c0b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 -- 172.21.15.67:0/422754891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c14ce00 con 0x7f909c070400 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a2800640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f909c1547f0 0x7f909c0b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a1e00640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c070400 0x7f909c14c5d0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f909000e960 tx=0x7f909000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f9096000640 1 -- 172.21.15.67:0/422754891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090017070 con 0x7f909c070400 2024-04-17T10:33:56.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f9096000640 1 -- 172.21.15.67:0/422754891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909000ca00 con 0x7f909c070400 2024-04-17T10:33:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909c14d0e0 con 0x7f909c070400 2024-04-17T10:33:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f9096000640 1 -- 172.21.15.67:0/422754891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000cdb0 con 0x7f909c070400 2024-04-17T10:33:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.713+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909c14d6a0 con 0x7f909c070400 2024-04-17T10:33:56.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.717+0000 7f9096000640 1 -- 172.21.15.67:0/422754891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f909001a070 con 0x7f909c070400 2024-04-17T10:33:56.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.717+0000 7f9096000640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9080066f60 0x7f9080069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.717+0000 7f9096000640 1 -- 172.21.15.67:0/422754891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90900855b0 con 0x7f909c070400 2024-04-17T10:33:56.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.717+0000 7f90a7c83000 1 Processor -- start 2024-04-17T10:33:56.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.717+0000 7f90a1400640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9080066f60 0x7f9080069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.717+0000 7f90a7c83000 1 -- start start 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.717+0000 7f90a1400640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9080066f60 0x7f9080069420 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f909c1457b0 tx=0x7f908c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f3a40 0x5580506f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506ae850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506aed90 0x5580506af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580505911f0 con 0x5580506f3a40 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580504f7e90 con 0x5580506aed90 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55805047ff50 con 0x5580506f4360 2024-04-17T10:33:56.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f3a40 0x5580506f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f3a40 0x5580506f3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42266/0 (socket says 172.21.15.67:42266) 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 -- 172.21.15.67:0/3617116813 learned_addr learned my addr 172.21.15.67:0/3617116813 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 -- 172.21.15.67:0/3617116813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 msgr2=0x5580506ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 --2- 172.21.15.67:0/3617116813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 -- 172.21.15.67:0/3617116813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506aed90 msgr2=0x5580506af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 --2- 172.21.15.67:0/3617116813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506aed90 0x5580506af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 -- 172.21.15.67:0/3617116813 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580504cf8b0 con 0x5580506f3a40 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a1e00640 1 --2- 172.21.15.67:0/3617116813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f3a40 0x5580506f3e20 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f909000d010 tx=0x7f9090055d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489c3673ed066a9c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f908aa00640 1 -- 172.21.15.67:0/3617116813 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090021040 con 0x5580506f3a40 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f908aa00640 1 -- 172.21.15.67:0/3617116813 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090053fb0 con 0x5580506f3a40 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f908aa00640 1 -- 172.21.15.67:0/3617116813 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90900850c0 con 0x5580506f3a40 2024-04-17T10:33:56.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- 172.21.15.67:0/3617116813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f3a40 msgr2=0x5580506f3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3617116813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f3a40 0x5580506f3e20 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f909000d010 tx=0x7f9090055d70 comp rx=0 tx=0).stop 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- 172.21.15.67:0/3617116813 shutdown_connections 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3617116813 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506aed90 0x5580506af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3617116813 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3617116813 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f3a40 0x5580506f3e20 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- 172.21.15.67:0/3617116813 >> 172.21.15.67:0/3617116813 conn(0x5580506aad40 msgr2=0x5580506ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- 172.21.15.67:0/3617116813 shutdown_connections 2024-04-17T10:33:56.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- 172.21.15.67:0/3617116813 wait complete. 2024-04-17T10:33:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- start start 2024-04-17T10:33:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 Processor -- start 2024-04-17T10:33:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506aed90 0x5580506fb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506fb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f4360 0x5580506b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580504b2540 con 0x5580506f4360 2024-04-17T10:33:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580504f3f30 con 0x5580506f3a40 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580504b9e50 con 0x5580506aed90 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.721+0000 7f90a7c83000 10 client.0 _ll_register_callbacks cb 0x5580505ac180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f4360 0x5580506b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f4360 0x5580506b4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42268/0 (socket says 172.21.15.67:42268) 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a1400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506fb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 -- 172.21.15.67:0/1796741353 learned_addr learned my addr 172.21.15.67:0/1796741353 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a1e00640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506aed90 0x5580506fb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 -- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506aed90 msgr2=0x5580506fb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506aed90 0x5580506fb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 -- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 msgr2=0x5580506fb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506fb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 -- 172.21.15.67:0/1796741353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580504801a0 con 0x5580506f4360 2024-04-17T10:33:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a1400640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506fb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a1e00640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506aed90 0x5580506fb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a2800640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f4360 0x5580506b4ee0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f909801d290 tx=0x7f9098019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3ef6e4588d18cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 1 -- 172.21.15.67:0/1796741353 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90980a5a90 con 0x5580506f4360 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 1 -- 172.21.15.67:0/1796741353 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f90980a5c30 con 0x5580506f4360 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9090053fb0 con 0x5580506f4360 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 1 -- 172.21.15.67:0/1796741353 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90980c7050 con 0x5580506f4360 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 10 client.8850 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55805057a840 con 0x5580506f4360 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 1 -- 172.21.15.67:0/1796741353 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9090053fb0 con 0x5580506f4360 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 1 -- 172.21.15.67:0/1796741353 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f90980ca020 con 0x5580506f4360 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 10 client.8850 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 10 client.8850 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:33:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 20 client.8850 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506b5c50 0x5580506b6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 4 client.8850 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558050493be0 con 0x5580506b5c50 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a1e00640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506b5c50 0x5580506b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a1e00640 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506b5c50 0x5580506b6030 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15c110cf95696272 server_cookie=c18388783ede31ac in_seq=0 out_seq=0 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 10 client.8850 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 1 -- 172.21.15.67:0/1796741353 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558050493be0 con 0x5580506b5c50 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 10 client.8850 handle_command_reply: tid=0 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 client.8850 shutdown 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f4360 msgr2=0x5580506b4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f4360 0x5580506b4ee0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f909801d290 tx=0x7f9098019790 comp rx=0 tx=0).stop 2024-04-17T10:33:56.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 shutdown_connections 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506b5c50 0x5580506b6030 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506f4360 0x5580506b4ee0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506fb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f907f400640 0 client.8850 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 --2- 172.21.15.67:0/1796741353 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506aed90 0x5580506fb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 >> 172.21.15.67:0/1796741353 conn(0x5580506aad40 msgr2=0x5580506ad9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 shutdown_connections 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 1 -- 172.21.15.67:0/1796741353 wait complete. 2024-04-17T10:33:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.725+0000 7f90a7c83000 20 client.8850 trim_cache size 0 max 16384 2024-04-17T10:33:56.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 Processor -- start 2024-04-17T10:33:56.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 -- start start 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580506ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506ad670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506f6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c203930 con 0x5580506aed90 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c202b30 con 0x5580506f3a40 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f909c153af0 con 0x5580506f4360 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580506ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580506ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42270/0 (socket says 172.21.15.67:42270) 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 -- 172.21.15.67:0/4274576056 learned_addr learned my addr 172.21.15.67:0/4274576056 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1400640 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a2800640 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 -- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 msgr2=0x5580506f6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506f6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 -- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 msgr2=0x5580506ad670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506ad670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1e00640 1 -- 172.21.15.67:0/4274576056 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558050551a10 con 0x5580506aed90 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a2800640 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.741+0000 7f90a1400640 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580506ad130 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f909000c520 tx=0x7f9090085050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c9a8668b787a98 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f908a000640 1 -- 172.21.15.67:0/4274576056 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090071020 con 0x5580506aed90 2024-04-17T10:33:56.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f908a000640 1 -- 172.21.15.67:0/4274576056 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90900850c0 con 0x5580506aed90 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f908a000640 1 -- 172.21.15.67:0/4274576056 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090021040 con 0x5580506aed90 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- 172.21.15.67:0/4274576056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 msgr2=0x5580506ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580506ad130 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f909000c520 tx=0x7f9090085050 comp rx=0 tx=0).stop 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- 172.21.15.67:0/4274576056 shutdown_connections 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f4360 0x5580506f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f3a40 0x5580506ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 --2- 172.21.15.67:0/4274576056 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580506ad130 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- 172.21.15.67:0/4274576056 >> 172.21.15.67:0/4274576056 conn(0x5580506aad40 msgr2=0x5580506f2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- 172.21.15.67:0/4274576056 shutdown_connections 2024-04-17T10:33:56.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- 172.21.15.67:0/4274576056 wait complete. 2024-04-17T10:33:56.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- start start 2024-04-17T10:33:56.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 Processor -- start 2024-04-17T10:33:56.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580507544b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f3a40 0x5580507549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f4360 0x5580506f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580504f3f30 con 0x5580506aed90 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909c202530 con 0x5580506f4360 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5580504bb720 con 0x5580506f3a40 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 10 client.0 _ll_register_callbacks cb 0x5580504baa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580507544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580507544b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42284/0 (socket says 172.21.15.67:42284) 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 -- 172.21.15.67:0/3665699962 learned_addr learned my addr 172.21.15.67:0/3665699962 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1400640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f3a40 0x5580507549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a2800640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f4360 0x5580506f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 -- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f3a40 msgr2=0x5580507549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f3a40 0x5580507549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 -- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f4360 msgr2=0x5580506f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f4360 0x5580506f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 -- 172.21.15.67:0/3665699962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55805048cb00 con 0x5580506aed90 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1400640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f3a40 0x5580507549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a2800640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f4360 0x5580506f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a1e00640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580507544b0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f9090055c40 tx=0x7f9090000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4917ff01e86f0e27 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f907f400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f907f400640 1 -- 172.21.15.67:0/3665699962 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090076070 con 0x5580506aed90 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f907f400640 1 -- 172.21.15.67:0/3665699962 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9090055100 con 0x5580506aed90 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.745+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90900850c0 con 0x5580506aed90 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f907f400640 1 -- 172.21.15.67:0/3665699962 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9090071030 con 0x5580506aed90 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 10 client.8856 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55805049ef80 con 0x5580506aed90 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f907f400640 1 -- 172.21.15.67:0/3665699962 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90900850c0 con 0x5580506aed90 2024-04-17T10:33:56.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f907f400640 1 -- 172.21.15.67:0/3665699962 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f9090051a00 con 0x5580506aed90 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 10 client.8856 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 10 client.8856 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 20 client.8856 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506f9890 0x5580506f9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 4 client.8856 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "scrub pause"}) v1 -- 0x5580505053d0 con 0x5580506f9890 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a1400640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506f9890 0x5580506f9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a1400640 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506f9890 0x5580506f9ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99750462bd145692 server_cookie=4662b4c3db0999ce in_seq=0 out_seq=0 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f907f400640 10 client.8856 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f907f400640 1 -- 172.21.15.67:0/3665699962 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x5580505053d0 con 0x5580506f9890 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f907f400640 10 client.8856 handle_command_reply: tid=0 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 client.8856 shutdown 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 msgr2=0x5580507544b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580507544b0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f9090055c40 tx=0x7f9090000bd0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 shutdown_connections 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5580506f9890 0x5580506f9ca0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5580506f4360 0x5580506f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5580506f3a40 0x5580507549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f907f400640 0 client.8856 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 --2- 172.21.15.67:0/3665699962 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5580506aed90 0x5580507544b0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 >> 172.21.15.67:0/3665699962 conn(0x5580506aad40 msgr2=0x5580506f2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 shutdown_connections 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 1 -- 172.21.15.67:0/3665699962 wait complete. 2024-04-17T10:33:56.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a7c83000 20 client.8856 trim_cache size 0 max 16384 2024-04-17T10:33:56.747 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:56.747 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-04-17T10:33:56.747 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9080066f60 msgr2=0x7f9080069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9080066f60 0x7f9080069420 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f909c1457b0 tx=0x7f908c008040 comp rx=0 tx=0).stop 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c070400 msgr2=0x7f909c14c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c070400 0x7f909c14c5d0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f909000e960 tx=0x7f909000ee20 comp rx=0 tx=0).stop 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 shutdown_connections 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9080066f60 0x7f9080069420 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f909c1547f0 0x7f909c0b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f909c0cea30 0x7f909c14dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 --2- 172.21.15.67:0/422754891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f909c070400 0x7f909c14c5d0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 >> 172.21.15.67:0/422754891 conn(0x7f909c06e420 msgr2=0x7f909c0afad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 shutdown_connections 2024-04-17T10:33:56.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.749+0000 7f90a3200640 1 -- 172.21.15.67:0/422754891 wait complete. 2024-04-17T10:33:56.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:33:56.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.8:0 scrub status 2024-04-17T10:33:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.909+0000 7fec85400640 1 Processor -- start 2024-04-17T10:33:56.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.909+0000 7fec85400640 1 -- start start 2024-04-17T10:33:56.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec85400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fec800aef60 0x7fec800cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec85400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec800cce60 0x7fec8006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec85400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800b4990 con 0x7fec8006e7b0 2024-04-17T10:33:56.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec85400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec800af340 con 0x7fec800cce60 2024-04-17T10:33:56.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec85400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec800b4130 con 0x7fec800aef60 2024-04-17T10:33:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec77e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec800cce60 0x7fec8006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec77e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fec800cce60 0x7fec8006e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42296/0 (socket says 172.21.15.67:42296) 2024-04-17T10:33:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec77e00640 1 -- 172.21.15.67:0/1921153076 learned_addr learned my addr 172.21.15.67:0/1921153076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec7e800640 1 --2- 172.21.15.67:0/1921153076 >> v2:172.21.15.86:3300/0 conn(0x7fec800aef60 0x7fec800cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec77e00640 1 -- 172.21.15.67:0/1921153076 >> v2:172.21.15.86:3300/0 conn(0x7fec800aef60 msgr2=0x7fec800cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec77e00640 1 --2- 172.21.15.67:0/1921153076 >> v2:172.21.15.86:3300/0 conn(0x7fec800aef60 0x7fec800cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec77e00640 1 -- 172.21.15.67:0/1921153076 >> v1:172.21.15.86:6789/0 conn(0x7fec8006e7b0 legacy=0x7fec800713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.913+0000 7fec77e00640 1 -- 172.21.15.67:0/1921153076 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8015ae90 con 0x7fec800cce60 2024-04-17T10:33:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec77e00640 1 --2- 172.21.15.67:0/1921153076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800cce60 0x7fec8006e270 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fec8006bdc0 tx=0x7fec6801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c06944b97ca30c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec7de00640 1 -- 172.21.15.67:0/1921153076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680b0070 con 0x7fec800cce60 2024-04-17T10:33:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec7de00640 1 -- 172.21.15.67:0/1921153076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec680ab600 con 0x7fec800cce60 2024-04-17T10:33:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec7de00640 1 -- 172.21.15.67:0/1921153076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680ab9d0 con 0x7fec800cce60 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 -- 172.21.15.67:0/1921153076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800cce60 msgr2=0x7fec8006e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 --2- 172.21.15.67:0/1921153076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800cce60 0x7fec8006e270 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fec8006bdc0 tx=0x7fec6801c490 comp rx=0 tx=0).stop 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 -- 172.21.15.67:0/1921153076 shutdown_connections 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 --2- 172.21.15.67:0/1921153076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800cce60 0x7fec8006e270 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 --2- 172.21.15.67:0/1921153076 >> v2:172.21.15.86:3300/0 conn(0x7fec800aef60 0x7fec800cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 -- 172.21.15.67:0/1921153076 >> 172.21.15.67:0/1921153076 conn(0x7fec800b1740 msgr2=0x7fec800b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 -- 172.21.15.67:0/1921153076 shutdown_connections 2024-04-17T10:33:56.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 -- 172.21.15.67:0/1921153076 wait complete. 2024-04-17T10:33:56.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 Processor -- start 2024-04-17T10:33:56.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.917+0000 7fec85400640 1 -- start start 2024-04-17T10:33:56.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec8006e7b0 0x7fec801f6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800aef60 0x7fec801427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec800cce60 0x7fec80159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015bc10 con 0x7fec800aef60 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8006eb90 con 0x7fec8006e7b0 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015dd40 con 0x7fec800cce60 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800aef60 0x7fec801427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800aef60 0x7fec801427f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42310/0 (socket says 172.21.15.67:42310) 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 -- 172.21.15.67:0/3051372265 learned_addr learned my addr 172.21.15.67:0/3051372265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec7e800640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec8006e7b0 0x7fec801f6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec7f200640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec800cce60 0x7fec80159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 -- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec800cce60 msgr2=0x7fec80159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec800cce60 0x7fec80159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 -- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec8006e7b0 msgr2=0x7fec801f6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec8006e7b0 0x7fec801f6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 -- 172.21.15.67:0/3051372265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80145e40 con 0x7fec800aef60 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec7f200640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec800cce60 0x7fec80159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec77e00640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800aef60 0x7fec801427f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fec680190a0 tx=0x7fec68019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec76a00640 1 -- 172.21.15.67:0/3051372265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680b0040 con 0x7fec800aef60 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec76a00640 1 -- 172.21.15.67:0/3051372265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec680b9070 con 0x7fec800aef60 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec76a00640 1 -- 172.21.15.67:0/3051372265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680b4880 con 0x7fec800aef60 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec80146120 con 0x7fec800aef60 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec80149d20 con 0x7fec800aef60 2024-04-17T10:33:56.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.921+0000 7fec76a00640 1 -- 172.21.15.67:0/3051372265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fec680b4a20 con 0x7fec800aef60 2024-04-17T10:33:56.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec76a00640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec5c066f60 0x7fec5c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec76a00640 1 -- 172.21.15.67:0/3051372265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec6811eff0 con 0x7fec800aef60 2024-04-17T10:33:56.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec7e800640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec5c066f60 0x7fec5c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 Processor -- start 2024-04-17T10:33:56.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- start start 2024-04-17T10:33:56.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec7e800640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec5c066f60 0x7fec5c069420 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fec70009b60 tx=0x7fec70009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485dba4db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba52f0 0x56485db5ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485db5b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56485d9a7ff0 con 0x56485dba52f0 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56485d930150 con 0x56485dba49d0 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56485d9a4090 con 0x56485db5b0a0 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba52f0 0x56485db5ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba52f0 0x56485db5ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42324/0 (socket says 172.21.15.67:42324) 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 -- 172.21.15.67:0/929579933 learned_addr learned my addr 172.21.15.67:0/929579933 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec7e800640 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485dba4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec7f200640 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485db5b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 -- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 msgr2=0x56485db5b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485db5b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 -- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 msgr2=0x56485dba4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485dba4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 -- 172.21.15.67:0/929579933 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56485db3e0d0 con 0x56485dba52f0 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec7e800640 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485dba4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec77e00640 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba52f0 0x56485db5ab60 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fec680f7900 tx=0x7fec680efe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b6670ca97564281 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec6f400640 1 -- 172.21.15.67:0/929579933 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680f3870 con 0x56485dba52f0 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec6f400640 1 -- 172.21.15.67:0/929579933 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec680f7e30 con 0x56485dba52f0 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec6f400640 1 -- 172.21.15.67:0/929579933 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec680eda20 con 0x56485dba52f0 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- 172.21.15.67:0/929579933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba52f0 msgr2=0x56485db5ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba52f0 0x56485db5ab60 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fec680f7900 tx=0x7fec680efe20 comp rx=0 tx=0).stop 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- 172.21.15.67:0/929579933 shutdown_connections 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485db5b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba52f0 0x56485db5ab60 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 --2- 172.21.15.67:0/929579933 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485dba4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- 172.21.15.67:0/929579933 >> 172.21.15.67:0/929579933 conn(0x56485db57020 msgr2=0x56485db59440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- 172.21.15.67:0/929579933 shutdown_connections 2024-04-17T10:33:56.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.925+0000 7fec887e5000 1 -- 172.21.15.67:0/929579933 wait complete. 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 -- start start 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 Processor -- start 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485db5b0a0 0x56485db897f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485db89d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485db8a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56485d96c250 con 0x56485db5b0a0 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x56485d962690 con 0x56485dba49d0 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56485d930a40 con 0x56485dba52f0 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 10 client.0 _ll_register_callbacks cb 0x56485db45e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:56.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485db5b0a0 0x56485db897f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485db5b0a0 0x56485db897f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42336/0 (socket says 172.21.15.67:42336) 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 -- 172.21.15.67:0/2056501677 learned_addr learned my addr 172.21.15.67:0/2056501677 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec77e00640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485db89d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7f200640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485db8a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 -- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 msgr2=0x56485db8a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485db8a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 -- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 msgr2=0x56485db89d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485db89d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 -- 172.21.15.67:0/2056501677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56485da551c0 con 0x56485db5b0a0 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec77e00640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485db89d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec7e800640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485db5b0a0 0x56485db897f0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fec70011fd0 tx=0x7fec700046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab9eae634f1dced8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec63e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:56.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec63e00640 1 -- 172.21.15.67:0/2056501677 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70027020 con 0x56485db5b0a0 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec63e00640 1 -- 172.21.15.67:0/2056501677 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec70023450 con 0x56485db5b0a0 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec680f7e30 con 0x56485db5b0a0 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec63e00640 1 -- 172.21.15.67:0/2056501677 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec70023650 con 0x56485db5b0a0 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 10 client.8868 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x56485da51170 con 0x56485db5b0a0 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec63e00640 1 -- 172.21.15.67:0/2056501677 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec680f7e30 con 0x56485db5b0a0 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec63e00640 1 -- 172.21.15.67:0/2056501677 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fec700237f0 con 0x56485db5b0a0 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 10 client.8868 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 10 client.8868 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 20 client.8868 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dbb4c80 0x56485dbb7070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 4 client.8868 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56485d943c60 con 0x56485dbb4c80 2024-04-17T10:33:56.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.929+0000 7fec77e00640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dbb4c80 0x56485dbb7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec77e00640 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dbb4c80 0x56485dbb7070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ded23eefff11707 server_cookie=b97d7522d1379093 in_seq=0 out_seq=0 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec63e00640 10 client.8868 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec63e00640 1 -- 172.21.15.67:0/2056501677 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56485d943c60 con 0x56485dbb4c80 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec63e00640 10 client.8868 handle_command_reply: tid=0 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 client.8868 shutdown 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485db5b0a0 msgr2=0x56485db897f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485db5b0a0 0x56485db897f0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fec70011fd0 tx=0x7fec700046d0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 shutdown_connections 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dbb4c80 0x56485dbb7070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485db8a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba49d0 0x56485db89d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 --2- 172.21.15.67:0/2056501677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485db5b0a0 0x56485db897f0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec63e00640 0 client.8868 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 >> 172.21.15.67:0/2056501677 conn(0x56485db57020 msgr2=0x56485db59ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 shutdown_connections 2024-04-17T10:33:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 1 -- 172.21.15.67:0/2056501677 wait complete. 2024-04-17T10:33:56.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.933+0000 7fec887e5000 20 client.8868 trim_cache size 0 max 16384 2024-04-17T10:33:56.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 Processor -- start 2024-04-17T10:33:56.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- start start 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485db5b0a0 0x56485dc00ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba49d0 0x56485dc01420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485dc01960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec802034a0 con 0x56485dba49d0 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec80202680 con 0x56485db5b0a0 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015c720 con 0x56485dba52f0 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba49d0 0x56485dc01420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba49d0 0x56485dc01420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42338/0 (socket says 172.21.15.67:42338) 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 -- 172.21.15.67:0/4101975781 learned_addr learned my addr 172.21.15.67:0/4101975781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec7f200640 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485dc01960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec7e800640 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485db5b0a0 0x56485dc00ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 -- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 msgr2=0x56485dc01960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485dc01960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 -- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485db5b0a0 msgr2=0x56485dc00ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485db5b0a0 0x56485dc00ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 -- 172.21.15.67:0/4101975781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56485d9eede0 con 0x56485dba49d0 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec77e00640 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba49d0 0x56485dc01420 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fec680f3a30 tx=0x7fec680f2000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a1635ca67f38f0a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec6ea00640 1 -- 172.21.15.67:0/4101975781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6811d0b0 con 0x56485dba49d0 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec6ea00640 1 -- 172.21.15.67:0/4101975781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec680ab540 con 0x56485dba49d0 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec6ea00640 1 -- 172.21.15.67:0/4101975781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec681070a0 con 0x56485dba49d0 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- 172.21.15.67:0/4101975781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba49d0 msgr2=0x56485dc01420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba49d0 0x56485dc01420 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fec680f3a30 tx=0x7fec680f2000 comp rx=0 tx=0).stop 2024-04-17T10:33:56.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- 172.21.15.67:0/4101975781 shutdown_connections 2024-04-17T10:33:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485dba52f0 0x56485dc01960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dba49d0 0x56485dc01420 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 --2- 172.21.15.67:0/4101975781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485db5b0a0 0x56485dc00ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- 172.21.15.67:0/4101975781 >> 172.21.15.67:0/4101975781 conn(0x56485db57020 msgr2=0x56485dba30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- 172.21.15.67:0/4101975781 shutdown_connections 2024-04-17T10:33:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- 172.21.15.67:0/4101975781 wait complete. 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 -- start start 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.949+0000 7fec887e5000 1 Processor -- start 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485dbb7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba52f0 0x56485dbb78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dbb7e00 0x56485dbb81e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56485d942e00 con 0x56485dbb7e00 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015cd20 con 0x56485dba52f0 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56485d9a7300 con 0x56485db5b0a0 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 10 client.0 _ll_register_callbacks cb 0x56485db26040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dbb7e00 0x56485dbb81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dbb7e00 0x56485dbb81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42340/0 (socket says 172.21.15.67:42340) 2024-04-17T10:33:56.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 -- 172.21.15.67:0/918605339 learned_addr learned my addr 172.21.15.67:0/918605339 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec77e00640 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba52f0 0x56485dbb78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7e800640 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485dbb7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 -- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 msgr2=0x56485dbb7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485dbb7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 -- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba52f0 msgr2=0x56485dbb78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba52f0 0x56485dbb78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 -- 172.21.15.67:0/918605339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56485d9498e0 con 0x56485dbb7e00 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7f200640 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dbb7e00 0x56485dbb81e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fec78005990 tx=0x7fec780157e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899a6eaacd479579 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 1 -- 172.21.15.67:0/918605339 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78017640 con 0x56485dbb7e00 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 1 -- 172.21.15.67:0/918605339 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fec78020c40 con 0x56485dbb7e00 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec680ab540 con 0x56485dbb7e00 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 1 -- 172.21.15.67:0/918605339 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec78020e40 con 0x56485dbb7e00 2024-04-17T10:33:56.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 10 client.8874 fetch_fsmap learned FSMap version 82 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x56485da018f0 con 0x56485dbb7e00 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 1 -- 172.21.15.67:0/918605339 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec680ab540 con 0x56485dbb7e00 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 1 -- 172.21.15.67:0/918605339 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fec780177e0 con 0x56485dbb7e00 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 10 client.8874 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 10 client.8874 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 20 client.8874 populate_metadata read hostname 'smithi067' 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dc0af50 0x56485dc0d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 4 client.8874 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56485d9b5530 con 0x56485dc0af50 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7e800640 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dc0af50 0x56485dc0d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec7e800640 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dc0af50 0x56485dc0d340 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f479dd5e0784e414 server_cookie=3117c5c45b136df5 in_seq=0 out_seq=0 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 10 client.8874 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 1 -- 172.21.15.67:0/918605339 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x56485d9b5530 con 0x56485dc0af50 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 10 client.8874 handle_command_reply: tid=0 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 client.8874 shutdown 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dbb7e00 msgr2=0x56485dbb81e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dbb7e00 0x56485dbb81e0 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fec78005990 tx=0x7fec780157e0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 shutdown_connections 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x56485dc0af50 0x56485dc0d340 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56485dbb7e00 0x56485dbb81e0 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x56485dba52f0 0x56485dbb78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 --2- 172.21.15.67:0/918605339 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56485db5b0a0 0x56485dbb7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec63e00640 0 client.8874 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 >> 172.21.15.67:0/918605339 conn(0x56485db57020 msgr2=0x56485dba3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 shutdown_connections 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 1 -- 172.21.15.67:0/918605339 wait complete. 2024-04-17T10:33:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.953+0000 7fec887e5000 20 client.8874 trim_cache size 0 max 16384 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: "3662aba3-3c3a-4153-aeeb-8bd2c974cc63": { 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_control_test_path", 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "3662aba3-3c3a-4153-aeeb-8bd2c974cc63", 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:33:56.952 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:33:56.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec5c066f60 msgr2=0x7fec5c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec5c066f60 0x7fec5c069420 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fec70009b60 tx=0x7fec70009190 comp rx=0 tx=0).stop 2024-04-17T10:33:56.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800aef60 msgr2=0x7fec801427f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:56.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800aef60 0x7fec801427f0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fec680190a0 tx=0x7fec68019230 comp rx=0 tx=0).stop 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 shutdown_connections 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fec5c066f60 0x7fec5c069420 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fec800cce60 0x7fec80159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec800aef60 0x7fec801427f0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 --2- 172.21.15.67:0/3051372265 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fec8006e7b0 0x7fec801f6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 >> 172.21.15.67:0/3051372265 conn(0x7fec800b1740 msgr2=0x7fec800b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 shutdown_connections 2024-04-17T10:33:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:56.957+0000 7fec85400640 1 -- 172.21.15.67:0/3051372265 wait complete. 2024-04-17T10:33:56.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:57.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f081b600640 1 Processor -- start 2024-04-17T10:33:57.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f081b600640 1 -- start start 2024-04-17T10:33:57.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f081b600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0814144aa0 0x7f08140d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f081b600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08141456d0 con 0x7f08141446c0 2024-04-17T10:33:57.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f081b600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140b4870 con 0x7f0814071840 2024-04-17T10:33:57.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f081b600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08140b51a0 con 0x7f0814144aa0 2024-04-17T10:33:57.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0818c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f08141446c0 0x7f0814144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35634/0 (socket says 172.21.15.67:35634) 2024-04-17T10:33:57.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0818c00640 1 -- 172.21.15.67:0/1105028568 learned_addr learned my addr 172.21.15.67:0/1105028568 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:57.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813e00640 1 --2- 172.21.15.67:0/1105028568 >> v2:172.21.15.86:3300/0 conn(0x7f0814144aa0 0x7f08140d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813400640 1 -- 172.21.15.67:0/1105028568 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315517011 0 0) 0x7f08141456d0 con 0x7f08141446c0 2024-04-17T10:33:57.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813400640 1 -- 172.21.15.67:0/1105028568 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc003530 con 0x7f08141446c0 2024-04-17T10:33:57.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813400640 1 -- 172.21.15.67:0/1105028568 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221326240 0 0) 0x7f08140b4870 con 0x7f0814071840 2024-04-17T10:33:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813400640 1 -- 172.21.15.67:0/1105028568 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08141456d0 con 0x7f0814071840 2024-04-17T10:33:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813e00640 1 -- 172.21.15.67:0/1105028568 >> v1:172.21.15.86:6789/0 conn(0x7f0814071840 legacy=0x7f08140b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813e00640 1 -- 172.21.15.67:0/1105028568 >> v1:172.21.15.67:6789/0 conn(0x7f08141446c0 legacy=0x7f0814144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.117+0000 7f0813e00640 1 -- 172.21.15.67:0/1105028568 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08140b4b50 con 0x7f0814144aa0 2024-04-17T10:33:57.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f0813e00640 1 --2- 172.21.15.67:0/1105028568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0814144aa0 0x7f08140d0910 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f08000097c0 tx=0x7f080001c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db1286144dfab0a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:57.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f0813400640 1 -- 172.21.15.67:0/1105028568 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000ab610 con 0x7f0814144aa0 2024-04-17T10:33:57.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f0813400640 1 -- 172.21.15.67:0/1105028568 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08000b0070 con 0x7f0814144aa0 2024-04-17T10:33:57.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f0813400640 1 -- 172.21.15.67:0/1105028568 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08000ab9d0 con 0x7f0814144aa0 2024-04-17T10:33:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f081b600640 1 -- 172.21.15.67:0/1105028568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0814144aa0 msgr2=0x7f08140d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f081b600640 1 --2- 172.21.15.67:0/1105028568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0814144aa0 0x7f08140d0910 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f08000097c0 tx=0x7f080001c470 comp rx=0 tx=0).stop 2024-04-17T10:33:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f081b600640 1 -- 172.21.15.67:0/1105028568 shutdown_connections 2024-04-17T10:33:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f081b600640 1 --2- 172.21.15.67:0/1105028568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0814144aa0 0x7f08140d0910 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f081b600640 1 -- 172.21.15.67:0/1105028568 >> 172.21.15.67:0/1105028568 conn(0x7f08140b34d0 msgr2=0x7f08140b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f081b600640 1 -- 172.21.15.67:0/1105028568 shutdown_connections 2024-04-17T10:33:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.121+0000 7f081b600640 1 -- 172.21.15.67:0/1105028568 wait complete. 2024-04-17T10:33:57.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 Processor -- start 2024-04-17T10:33:57.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 -- start start 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814071840 0x7f0814158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08141446c0 0x7f08140aede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0814144aa0 0x7f08140c67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814169bd0 con 0x7f0814071840 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814169a50 con 0x7f08141446c0 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0814169d50 con 0x7f0814144aa0 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814071840 0x7f0814158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814071840 0x7f0814158710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42342/0 (socket says 172.21.15.67:42342) 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 -- 172.21.15.67:0/3099628117 learned_addr learned my addr 172.21.15.67:0/3099628117 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0813e00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08141446c0 0x7f08140aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0819600640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0814144aa0 0x7f08140c67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 -- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0814144aa0 msgr2=0x7f08140c67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0814144aa0 0x7f08140c67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 -- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08141446c0 msgr2=0x7f08140aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08141446c0 0x7f08140aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 -- 172.21.15.67:0/3099628117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081406c850 con 0x7f0814071840 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0813e00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08141446c0 0x7f08140aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0818c00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814071840 0x7f0814158710 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f080800c4d0 tx=0x7f080800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0810c00640 1 -- 172.21.15.67:0/3099628117 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808017070 con 0x7f0814071840 2024-04-17T10:33:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0810c00640 1 -- 172.21.15.67:0/3099628117 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808005cf0 con 0x7f0814071840 2024-04-17T10:33:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f0810c00640 1 -- 172.21.15.67:0/3099628117 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080072b0 con 0x7f0814071840 2024-04-17T10:33:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f081406cb30 con 0x7f0814071840 2024-04-17T10:33:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f081406d0f0 con 0x7f0814071840 2024-04-17T10:33:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.125+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814070bb0 con 0x7f0814071840 2024-04-17T10:33:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.129+0000 7f0810c00640 1 -- 172.21.15.67:0/3099628117 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f080801a070 con 0x7f0814071840 2024-04-17T10:33:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.129+0000 7f0810c00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07f4066f60 0x7f07f4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.129+0000 7f0810c00640 1 -- 172.21.15.67:0/3099628117 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0808084530 con 0x7f0814071840 2024-04-17T10:33:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.129+0000 7f0813e00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07f4066f60 0x7f07f4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.129+0000 7f0813e00640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07f4066f60 0x7f07f4069420 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f08000af000 tx=0x7f08000a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.133+0000 7f0810c00640 1 -- 172.21.15.67:0/3099628117 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0808054d60 con 0x7f0814071840 2024-04-17T10:33:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.333+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08140b41a0 con 0x7f0814071840 2024-04-17T10:33:57.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.333+0000 7f0810c00640 1 -- 172.21.15.67:0/3099628117 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f0808058e00 con 0x7f0814071840 2024-04-17T10:33:57.326 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:57.326 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:33:57.326 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07f4066f60 msgr2=0x7f07f4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07f4066f60 0x7f07f4069420 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f08000af000 tx=0x7f08000a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814071840 msgr2=0x7f0814158710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814071840 0x7f0814158710 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f080800c4d0 tx=0x7f080800c990 comp rx=0 tx=0).stop 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 shutdown_connections 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f07f4066f60 0x7f07f4069420 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0814144aa0 0x7f08140c67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08141446c0 0x7f08140aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 --2- 172.21.15.67:0/3099628117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0814071840 0x7f0814158710 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 >> 172.21.15.67:0/3099628117 conn(0x7f08140b34d0 msgr2=0x7f0814143470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 shutdown_connections 2024-04-17T10:33:57.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.337+0000 7f081b600640 1 -- 172.21.15.67:0/3099628117 wait complete. 2024-04-17T10:33:57.346 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:33:57.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.493+0000 7f2f9f600640 1 Processor -- start 2024-04-17T10:33:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.493+0000 7f2f9f600640 1 -- start start 2024-04-17T10:33:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9f600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f981446c0 0x7f2f98144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9f600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f98144aa0 0x7f2f980d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9f600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f980b6640 con 0x7f2f98071840 2024-04-17T10:33:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9f600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f980b51a0 con 0x7f2f981446c0 2024-04-17T10:33:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9f600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f981456d0 con 0x7f2f98144aa0 2024-04-17T10:33:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f981446c0 0x7f2f98144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9d600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2f98071840 0x7f2f9815f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35646/0 (socket says 172.21.15.67:35646) 2024-04-17T10:33:57.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9d600640 1 -- 172.21.15.67:0/3371106387 learned_addr learned my addr 172.21.15.67:0/3371106387 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:57.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f8fe00640 1 --2- 172.21.15.67:0/3371106387 >> v2:172.21.15.86:3300/0 conn(0x7f2f98144aa0 0x7f2f980d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9cc00640 1 -- 172.21.15.67:0/3371106387 >> v2:172.21.15.86:3300/0 conn(0x7f2f98144aa0 msgr2=0x7f2f980d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:57.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9cc00640 1 --2- 172.21.15.67:0/3371106387 >> v2:172.21.15.86:3300/0 conn(0x7f2f98144aa0 0x7f2f980d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9cc00640 1 -- 172.21.15.67:0/3371106387 >> v1:172.21.15.67:6789/0 conn(0x7f2f98071840 legacy=0x7f2f9815f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f9cc00640 1 -- 172.21.15.67:0/3371106387 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f980b4870 con 0x7f2f981446c0 2024-04-17T10:33:57.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.497+0000 7f2f97e00640 1 -- 172.21.15.67:0/3371106387 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476267648 0 0) 0x7f2f980b6640 con 0x7f2f98071840 2024-04-17T10:33:57.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9cc00640 1 --2- 172.21.15.67:0/3371106387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f981446c0 0x7f2f98144ea0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f2f9815f620 tx=0x7f2f8801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e43ccf351ca6e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:57.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f97e00640 1 -- 172.21.15.67:0/3371106387 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f880ab650 con 0x7f2f981446c0 2024-04-17T10:33:57.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f97e00640 1 -- 172.21.15.67:0/3371106387 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f880b0070 con 0x7f2f981446c0 2024-04-17T10:33:57.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f97e00640 1 -- 172.21.15.67:0/3371106387 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f880aba10 con 0x7f2f981446c0 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 -- 172.21.15.67:0/3371106387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f981446c0 msgr2=0x7f2f98144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3371106387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f981446c0 0x7f2f98144ea0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f2f9815f620 tx=0x7f2f8801c450 comp rx=0 tx=0).stop 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 -- 172.21.15.67:0/3371106387 shutdown_connections 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3371106387 >> v2:172.21.15.86:3300/0 conn(0x7f2f98144aa0 0x7f2f980d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3371106387 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f981446c0 0x7f2f98144ea0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 -- 172.21.15.67:0/3371106387 >> 172.21.15.67:0/3371106387 conn(0x7f2f980b34d0 msgr2=0x7f2f980b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 -- 172.21.15.67:0/3371106387 shutdown_connections 2024-04-17T10:33:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 -- 172.21.15.67:0/3371106387 wait complete. 2024-04-17T10:33:57.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 Processor -- start 2024-04-17T10:33:57.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.501+0000 7f2f9f600640 1 -- start start 2024-04-17T10:33:57.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98071840 0x7f2f981404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981446c0 0x7f2f98153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f98144aa0 0x7f2f9806c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f980d1260 con 0x7f2f98144aa0 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9815c0d0 con 0x7f2f981446c0 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f98168a40 con 0x7f2f98071840 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f98144aa0 0x7f2f9806c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f98144aa0 0x7f2f9806c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42348/0 (socket says 172.21.15.67:42348) 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 -- 172.21.15.67:0/3560585442 learned_addr learned my addr 172.21.15.67:0/3560585442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f8fe00640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981446c0 0x7f2f98153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9cc00640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98071840 0x7f2f981404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 -- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98071840 msgr2=0x7f2f981404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98071840 0x7f2f981404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 -- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981446c0 msgr2=0x7f2f98153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981446c0 0x7f2f98153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 -- 172.21.15.67:0/3560585442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9806d070 con 0x7f2f98144aa0 2024-04-17T10:33:57.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9cc00640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98071840 0x7f2f981404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9d600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f98144aa0 0x7f2f9806c8d0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000cb20 tx=0x7f2f9000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f95600640 1 -- 172.21.15.67:0/3560585442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9000aca0 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f95600640 1 -- 172.21.15.67:0/3560585442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f90012400 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9806d350 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f95600640 1 -- 172.21.15.67:0/3560585442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9000ae60 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9806d910 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f98070bb0 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.505+0000 7f2f95600640 1 -- 172.21.15.67:0/3560585442 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2f900127f0 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.509+0000 7f2f95600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f74066f60 0x7f2f74069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.509+0000 7f2f95600640 1 -- 172.21.15.67:0/3560585442 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2f90085850 con 0x7f2f98144aa0 2024-04-17T10:33:57.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.509+0000 7f2f9cc00640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f74066f60 0x7f2f74069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:57.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.509+0000 7f2f9cc00640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f74066f60 0x7f2f74069420 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2f880af000 tx=0x7f2f880a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:57.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.513+0000 7f2f95600640 1 -- 172.21.15.67:0/3560585442 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2f90056630 con 0x7f2f98144aa0 2024-04-17T10:33:57.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.729+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2f98074e30 con 0x7f2f98144aa0 2024-04-17T10:33:57.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.729+0000 7f2f95600640 1 -- 172.21.15.67:0/3560585442 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1444 (secure 0 0 0) 0x7f2f90055f30 con 0x7f2f98144aa0 2024-04-17T10:33:57.723 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:57.723 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":568,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":827,"data_bytes":844168908,"bytes_used":2070249472,"bytes_avail":771023863808,"bytes_total":773094113280,"write_bytes_sec":47894754,"read_op_per_sec":0,"write_op_per_sec":47},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-17T10:33:18.256212+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f74066f60 msgr2=0x7f2f74069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f74066f60 0x7f2f74069420 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f2f880af000 tx=0x7f2f880a7000 comp rx=0 tx=0).stop 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f98144aa0 msgr2=0x7f2f9806c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f98144aa0 0x7f2f9806c8d0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f2f9000cb20 tx=0x7f2f9000a060 comp rx=0 tx=0).stop 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 shutdown_connections 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f74066f60 0x7f2f74069420 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f98144aa0 0x7f2f9806c8d0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f981446c0 0x7f2f98153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 --2- 172.21.15.67:0/3560585442 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f98071840 0x7f2f981404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:57.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 >> 172.21.15.67:0/3560585442 conn(0x7f2f980b34d0 msgr2=0x7f2f98155850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:57.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 shutdown_connections 2024-04-17T10:33:57.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:57.733+0000 7f2f9f600640 1 -- 172.21.15.67:0/3560585442 wait complete. 2024-04-17T10:33:57.743 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:33:57.880 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:33:57.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:33:58.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:33:58.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a93e00640 1 Processor -- start 2024-04-17T10:33:58.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a93e00640 1 -- start start 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a93e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0a8c0ce1b0 0x7f0a8c0ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a93e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a8c15fbd0 0x7f0a8c0c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a93e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a8c0cead0 con 0x7f0a8c15cc00 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a93e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8c15ffe0 con 0x7f0a8c15fbd0 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a93e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8c15be60 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a90a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0a8c15cc00 0x7f0a8c15f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35660/0 (socket says 172.21.15.67:35660) 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a90a00640 1 -- 172.21.15.67:0/2228145750 learned_addr learned my addr 172.21.15.67:0/2228145750 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:58.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a91e00640 1 --2- 172.21.15.67:0/2228145750 >> v2:172.21.15.67:3300/0 conn(0x7f0a8c15fbd0 0x7f0a8c0c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a91400640 1 --2- 172.21.15.67:0/2228145750 >> v2:172.21.15.86:3300/0 conn(0x7f0a8c0ce1b0 0x7f0a8c0ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a8be00640 1 -- 172.21.15.67:0/2228145750 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526584151 0 0) 0x7f0a8c0cead0 con 0x7f0a8c15cc00 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a8be00640 1 -- 172.21.15.67:0/2228145750 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a74003530 con 0x7f0a8c15cc00 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a91e00640 1 -- 172.21.15.67:0/2228145750 >> v2:172.21.15.86:3300/0 conn(0x7f0a8c0ce1b0 msgr2=0x7f0a8c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a91e00640 1 --2- 172.21.15.67:0/2228145750 >> v2:172.21.15.86:3300/0 conn(0x7f0a8c0ce1b0 0x7f0a8c0ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a91e00640 1 -- 172.21.15.67:0/2228145750 >> v1:172.21.15.67:6789/0 conn(0x7f0a8c15cc00 legacy=0x7f0a8c15f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a91e00640 1 -- 172.21.15.67:0/2228145750 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c160230 con 0x7f0a8c15fbd0 2024-04-17T10:33:58.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.905+0000 7f0a91400640 1 --2- 172.21.15.67:0/2228145750 >> v2:172.21.15.86:3300/0 conn(0x7f0a8c0ce1b0 0x7f0a8c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:33:58.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a91e00640 1 --2- 172.21.15.67:0/2228145750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c15fbd0 0x7f0a8c0c6b30 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0d1770 tx=0x7f0a8401f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9a79cea4fdecf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:58.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a8be00640 1 -- 172.21.15.67:0/2228145750 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840b4070 con 0x7f0a8c15fbd0 2024-04-17T10:33:58.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a8be00640 1 -- 172.21.15.67:0/2228145750 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a840af600 con 0x7f0a8c15fbd0 2024-04-17T10:33:58.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a8be00640 1 -- 172.21.15.67:0/2228145750 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840af9d0 con 0x7f0a8c15fbd0 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 -- 172.21.15.67:0/2228145750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c15fbd0 msgr2=0x7f0a8c0c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 --2- 172.21.15.67:0/2228145750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c15fbd0 0x7f0a8c0c6b30 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0d1770 tx=0x7f0a8401f110 comp rx=0 tx=0).stop 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 -- 172.21.15.67:0/2228145750 shutdown_connections 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 --2- 172.21.15.67:0/2228145750 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c15fbd0 0x7f0a8c0c6b30 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 --2- 172.21.15.67:0/2228145750 >> v2:172.21.15.86:3300/0 conn(0x7f0a8c0ce1b0 0x7f0a8c0ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 -- 172.21.15.67:0/2228145750 >> 172.21.15.67:0/2228145750 conn(0x7f0a8c0b33f0 msgr2=0x7f0a8c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 -- 172.21.15.67:0/2228145750 shutdown_connections 2024-04-17T10:33:58.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.909+0000 7f0a93e00640 1 -- 172.21.15.67:0/2228145750 wait complete. 2024-04-17T10:33:58.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 Processor -- start 2024-04-17T10:33:58.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 -- start start 2024-04-17T10:33:58.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c0ce1b0 0x7f0a8c14f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:58.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a8c15cc00 0x7f0a8c150990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:58.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a8c15fbd0 0x7f0a8c1525f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:58.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c15c6e0 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c15c400 con 0x7f0a8c15fbd0 2024-04-17T10:33:58.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c159730 con 0x7f0a8c15cc00 2024-04-17T10:33:58.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c0ce1b0 0x7f0a8c14f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:58.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c0ce1b0 0x7f0a8c14f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42362/0 (socket says 172.21.15.67:42362) 2024-04-17T10:33:58.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 -- 172.21.15.67:0/31321935 learned_addr learned my addr 172.21.15.67:0/31321935 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 -- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a8c15cc00 msgr2=0x7f0a8c150990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a8c15cc00 0x7f0a8c150990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 -- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a8c15fbd0 msgr2=0x7f0a8c1525f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a8c15fbd0 0x7f0a8c1525f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 -- 172.21.15.67:0/31321935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c1f7230 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a91400640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c0ce1b0 0x7f0a8c14f380 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000e810 tx=0x7f0a8000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a89600640 1 -- 172.21.15.67:0/31321935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80017070 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a89600640 1 -- 172.21.15.67:0/31321935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8000c8d0 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a89600640 1 -- 172.21.15.67:0/31321935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8000f040 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c1f7510 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.913+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c1f7ad0 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.917+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c070ab0 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.917+0000 7f0a89600640 1 -- 172.21.15.67:0/31321935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0a8001a070 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.917+0000 7f0a89600640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a6c066f60 0x7f0a6c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:58.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.917+0000 7f0a89600640 1 -- 172.21.15.67:0/31321935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f0a80084e40 con 0x7f0a8c0ce1b0 2024-04-17T10:33:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.917+0000 7f0a90a00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a6c066f60 0x7f0a6c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:58.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.917+0000 7f0a90a00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a6c066f60 0x7f0a6c069420 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0a78007fd0 tx=0x7f0a78007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:58.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:58.921+0000 7f0a89600640 1 -- 172.21.15.67:0/31321935 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0a80055cb0 con 0x7f0a8c0ce1b0 2024-04-17T10:33:59.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a8c0cf040 con 0x7f0a8c0ce1b0 2024-04-17T10:33:59.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a89600640 1 -- 172.21.15.67:0/31321935 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f0a800555b0 con 0x7f0a8c0ce1b0 2024-04-17T10:33:59.109 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:59.109 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:33:59.109 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a6c066f60 msgr2=0x7f0a6c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a6c066f60 0x7f0a6c069420 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0a78007fd0 tx=0x7f0a78007900 comp rx=0 tx=0).stop 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c0ce1b0 msgr2=0x7f0a8c14f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c0ce1b0 0x7f0a8c14f380 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000e810 tx=0x7f0a8000ecd0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 shutdown_connections 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0a6c066f60 0x7f0a6c069420 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0a8c15fbd0 0x7f0a8c1525f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0a8c15cc00 0x7f0a8c150990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 --2- 172.21.15.67:0/31321935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a8c0ce1b0 0x7f0a8c14f380 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 >> 172.21.15.67:0/31321935 conn(0x7f0a8c0b33f0 msgr2=0x7f0a8c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:59.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 shutdown_connections 2024-04-17T10:33:59.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.117+0000 7f0a93e00640 1 -- 172.21.15.67:0/31321935 wait complete. 2024-04-17T10:33:59.129 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:33:59.130 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:33:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.277+0000 7f7fe4200640 1 Processor -- start 2024-04-17T10:33:59.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.277+0000 7f7fe4200640 1 -- start start 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.277+0000 7f7fe4200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fdc06eac0 0x7f7fdc143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe4200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7fdc0cf0c0 0x7f7fdc144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe4200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc144560 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe4200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc15ae40 con 0x7f7fdc06eac0 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe4200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc071440 con 0x7f7fdc0cf0c0 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe0e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7fdc06eac0 0x7f7fdc143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe1800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7fdc06e6e0 0x7f7fdc0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:35668/0 (socket says 172.21.15.67:35668) 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe1800640 1 -- 172.21.15.67:0/2714858756 learned_addr learned my addr 172.21.15.67:0/2714858756 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe2200640 1 --2- 172.21.15.67:0/2714858756 >> v2:172.21.15.86:3300/0 conn(0x7f7fdc0cf0c0 0x7f7fdc144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:59.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe0e00640 1 -- 172.21.15.67:0/2714858756 >> v2:172.21.15.86:3300/0 conn(0x7f7fdc0cf0c0 msgr2=0x7f7fdc144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:33:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe0e00640 1 --2- 172.21.15.67:0/2714858756 >> v2:172.21.15.86:3300/0 conn(0x7f7fdc0cf0c0 0x7f7fdc144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe0e00640 1 -- 172.21.15.67:0/2714858756 >> v1:172.21.15.67:6789/0 conn(0x7f7fdc06e6e0 legacy=0x7f7fdc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe0e00640 1 -- 172.21.15.67:0/2714858756 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc071720 con 0x7f7fdc06eac0 2024-04-17T10:33:59.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe2200640 1 --2- 172.21.15.67:0/2714858756 >> v2:172.21.15.86:3300/0 conn(0x7f7fdc0cf0c0 0x7f7fdc144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:33:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fe0e00640 1 --2- 172.21.15.67:0/2714858756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06eac0 0x7f7fdc143ae0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0ce3b0 tx=0x7f7fc801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ddf617491717b87 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fdbe00640 1 -- 172.21.15.67:0/2714858756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc80b1070 con 0x7f7fdc06eac0 2024-04-17T10:33:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fdbe00640 1 -- 172.21.15.67:0/2714858756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc80a8050 con 0x7f7fdc06eac0 2024-04-17T10:33:59.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.281+0000 7f7fdbe00640 1 -- 172.21.15.67:0/2714858756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc80ac860 con 0x7f7fdc06eac0 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- 172.21.15.67:0/2714858756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06eac0 msgr2=0x7f7fdc143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 --2- 172.21.15.67:0/2714858756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06eac0 0x7f7fdc143ae0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0ce3b0 tx=0x7f7fc801c3c0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- 172.21.15.67:0/2714858756 shutdown_connections 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 --2- 172.21.15.67:0/2714858756 >> v2:172.21.15.86:3300/0 conn(0x7f7fdc0cf0c0 0x7f7fdc144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 --2- 172.21.15.67:0/2714858756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06eac0 0x7f7fdc143ae0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- 172.21.15.67:0/2714858756 >> 172.21.15.67:0/2714858756 conn(0x7f7fdc06ac90 msgr2=0x7f7fdc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- 172.21.15.67:0/2714858756 shutdown_connections 2024-04-17T10:33:59.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- 172.21.15.67:0/2714858756 wait complete. 2024-04-17T10:33:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 Processor -- start 2024-04-17T10:33:59.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- start start 2024-04-17T10:33:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06e6e0 0x7f7fdc14d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fdc06eac0 0x7f7fdc0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fdc0cf0c0 0x7f7fdc0afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc0b3a40 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc0b38c0 con 0x7f7fdc0cf0c0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe4200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc0b3bc0 con 0x7f7fdc06eac0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06e6e0 0x7f7fdc14d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06e6e0 0x7f7fdc14d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42380/0 (socket says 172.21.15.67:42380) 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 -- 172.21.15.67:0/1877078080 learned_addr learned my addr 172.21.15.67:0/1877078080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe0e00640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fdc06eac0 0x7f7fdc0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe2200640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fdc0cf0c0 0x7f7fdc0afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 -- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fdc06eac0 msgr2=0x7f7fdc0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fdc06eac0 0x7f7fdc0c46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 -- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fdc0cf0c0 msgr2=0x7f7fdc0afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fdc0cf0c0 0x7f7fdc0afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.285+0000 7f7fe1800640 1 -- 172.21.15.67:0/1877078080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0c5050 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fe1800640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06e6e0 0x7f7fdc14d350 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c450 tx=0x7f7fd000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fd9600640 1 -- 172.21.15.67:0/1877078080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0017070 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fd9600640 1 -- 172.21.15.67:0/1877078080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0005cf0 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc0c5330 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fd9600640 1 -- 172.21.15.67:0/1877078080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0007290 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc0c58f0 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc072d50 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fd9600640 1 -- 172.21.15.67:0/1877078080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7fd001a070 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fd9600640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f90 0x7f7fbc069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:33:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fd9600640 1 -- 172.21.15.67:0/1877078080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7fd00844f0 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.289+0000 7f7fe0e00640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f90 0x7f7fbc069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:33:59.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.293+0000 7f7fe0e00640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f90 0x7f7fbc069450 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80ae000 tx=0x7f7fc8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:33:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.293+0000 7f7fd9600640 1 -- 172.21.15.67:0/1877078080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7fd0055360 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.513+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7fdc0b2c60 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.513+0000 7f7fd9600640 1 -- 172.21.15.67:0/1877078080 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1446 (secure 0 0 0) 0x7f7fd0054c60 con 0x7f7fdc06e6e0 2024-04-17T10:33:59.506 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:33:59.506 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":570,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076506864,"bytes_used":2597904384,"bytes_avail":770496208896,"bytes_total":773094113280,"write_bytes_sec":67254935,"read_op_per_sec":0,"write_op_per_sec":67},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-17T10:33:18.256212+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:33:59.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f90 msgr2=0x7f7fbc069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f90 0x7f7fbc069450 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80ae000 tx=0x7f7fc8008040 comp rx=0 tx=0).stop 2024-04-17T10:33:59.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06e6e0 msgr2=0x7f7fdc14d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:33:59.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06e6e0 0x7f7fdc14d350 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c450 tx=0x7f7fd000c910 comp rx=0 tx=0).stop 2024-04-17T10:33:59.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 shutdown_connections 2024-04-17T10:33:59.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7fbc066f90 0x7f7fbc069450 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7fdc0cf0c0 0x7f7fdc0afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7fdc06eac0 0x7f7fdc0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 --2- 172.21.15.67:0/1877078080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fdc06e6e0 0x7f7fdc14d350 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:33:59.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 >> 172.21.15.67:0/1877078080 conn(0x7f7fdc06ac90 msgr2=0x7f7fdc159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:33:59.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 shutdown_connections 2024-04-17T10:33:59.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:33:59.517+0000 7f7fe4200640 1 -- 172.21.15.67:0/1877078080 wait complete. 2024-04-17T10:33:59.527 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:34:00.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:00.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.681+0000 7f5426000640 1 Processor -- start 2024-04-17T10:34:00.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.681+0000 7f5426000640 1 -- start start 2024-04-17T10:34:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5426000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f54200cf0c0 0x7f54201441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5426000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f542015ae40 con 0x7f542006eac0 2024-04-17T10:34:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5426000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5420071440 con 0x7f542006e6e0 2024-04-17T10:34:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5426000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5420071720 con 0x7f54200cf0c0 2024-04-17T10:34:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f542006eac0 0x7f5420143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34338/0 (socket says 172.21.15.67:34338) 2024-04-17T10:34:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 -- 172.21.15.67:0/2892502383 learned_addr learned my addr 172.21.15.67:0/2892502383 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:00.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541fe00640 1 --2- 172.21.15.67:0/2892502383 >> v2:172.21.15.86:3300/0 conn(0x7f54200cf0c0 0x7f54201441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030792650 0 0) 0x7f542015ae40 con 0x7f542006eac0 2024-04-17T10:34:00.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5404003530 con 0x7f542006eac0 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 753856710 0 0) 0x7f5404003530 con 0x7f542006eac0 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 >> v2:172.21.15.86:3300/0 conn(0x7f54200cf0c0 msgr2=0x7f54201441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 --2- 172.21.15.67:0/2892502383 >> v2:172.21.15.86:3300/0 conn(0x7f54200cf0c0 0x7f54201441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 >> v1:172.21.15.86:6789/0 conn(0x7f542006e6e0 legacy=0x7f54200ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5420071a00 con 0x7f542006eac0 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f5408001680 con 0x7f542006eac0 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 >> v1:172.21.15.67:6789/0 conn(0x7f542006eac0 legacy=0x7f5420143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54040039c0 0x7f5404003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5404004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54040051d0 0x7f540400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5420071720 con 0x7f5404004810 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540400c170 con 0x7f54040051d0 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f540400c370 con 0x7f54040039c0 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5404004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541f400640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54040039c0 0x7f5404003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541fe00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54040051d0 0x7f540400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 -- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54040039c0 msgr2=0x7f5404003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54040039c0 0x7f5404003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 -- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54040051d0 msgr2=0x7f540400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54040051d0 0x7f540400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f5417e00640 1 -- 172.21.15.67:0/2892502383 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5404003530 con 0x7f5404004810 2024-04-17T10:34:00.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.685+0000 7f541fe00640 1 -- 172.21.15.67:0/2892502383 reap_dead start 2024-04-17T10:34:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5417e00640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5404004c90 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f540800c240 tx=0x7f540801cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2862aa10a98c46fe server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408007d20 con 0x7f5404004810 2024-04-17T10:34:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54080a9070 con 0x7f5404004810 2024-04-17T10:34:00.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f541ea00640 1 -- 172.21.15.67:0/2892502383 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5408005d80 con 0x7f5404004810 2024-04-17T10:34:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 -- 172.21.15.67:0/2892502383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 msgr2=0x7f5404004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5404004c90 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f540800c240 tx=0x7f540801cbd0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 -- 172.21.15.67:0/2892502383 shutdown_connections 2024-04-17T10:34:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 --2- 172.21.15.67:0/2892502383 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5404004c90 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 -- 172.21.15.67:0/2892502383 >> 172.21.15.67:0/2892502383 conn(0x7f542006ac90 msgr2=0x7f542006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 -- 172.21.15.67:0/2892502383 shutdown_connections 2024-04-17T10:34:00.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 -- 172.21.15.67:0/2892502383 wait complete. 2024-04-17T10:34:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.689+0000 7f5426000640 1 Processor -- start 2024-04-17T10:34:00.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 -- start start 2024-04-17T10:34:00.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5420159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54201ea810 0x7f54201eac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54201401a0 0x7f5420140650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5420144470 con 0x7f5404004810 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54201442f0 con 0x7f54201401a0 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f542015e3e0 con 0x7f54201ea810 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5420159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5420159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40542/0 (socket says 172.21.15.67:40542) 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 -- 172.21.15.67:0/3795417816 learned_addr learned my addr 172.21.15.67:0/3795417816 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5417e00640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54201ea810 0x7f54201eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541fe00640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54201401a0 0x7f5420140650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 -- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54201ea810 msgr2=0x7f54201eac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54201ea810 0x7f54201eac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 -- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54201401a0 msgr2=0x7f5420140650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:00.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54201401a0 0x7f5420140650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 -- 172.21.15.67:0/3795417816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f542013e400 con 0x7f5404004810 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541fe00640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54201401a0 0x7f5420140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f541f400640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5420159af0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f541000f770 tx=0x7f541000fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5417400640 1 -- 172.21.15.67:0/3795417816 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54100050d0 con 0x7f5404004810 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5417400640 1 -- 172.21.15.67:0/3795417816 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5410018070 con 0x7f5404004810 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5417400640 1 -- 172.21.15.67:0/3795417816 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410005490 con 0x7f5404004810 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f542013e6e0 con 0x7f5404004810 2024-04-17T10:34:00.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f542013eca0 con 0x7f5404004810 2024-04-17T10:34:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.693+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5420072d50 con 0x7f5404004810 2024-04-17T10:34:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.697+0000 7f5417400640 1 -- 172.21.15.67:0/3795417816 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f541000d650 con 0x7f5404004810 2024-04-17T10:34:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.697+0000 7f5417400640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f53fc067130 0x7f53fc0695f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:00.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.697+0000 7f5417400640 1 -- 172.21.15.67:0/3795417816 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5410084d80 con 0x7f5404004810 2024-04-17T10:34:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.697+0000 7f5417e00640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f53fc067130 0x7f53fc0695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:00.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.697+0000 7f5417e00640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f53fc067130 0x7f53fc0695f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f54080ab000 tx=0x7f54080b1040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:00.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.701+0000 7f5417400640 1 -- 172.21.15.67:0/3795417816 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5410055bf0 con 0x7f5404004810 2024-04-17T10:34:00.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5420074e30 con 0x7f5404004810 2024-04-17T10:34:00.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5417400640 1 -- 172.21.15.67:0/3795417816 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f54100554f0 con 0x7f5404004810 2024-04-17T10:34:00.888 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:00.888 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:00.889 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:00.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f53fc067130 msgr2=0x7f53fc0695f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f53fc067130 0x7f53fc0695f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f54080ab000 tx=0x7f54080b1040 comp rx=0 tx=0).stop 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 msgr2=0x7f5420159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5420159af0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f541000f770 tx=0x7f541000fc30 comp rx=0 tx=0).stop 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 shutdown_connections 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f53fc067130 0x7f53fc0695f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f54201401a0 0x7f5420140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f54201ea810 0x7f54201eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 --2- 172.21.15.67:0/3795417816 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5404004810 0x7f5420159af0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 >> 172.21.15.67:0/3795417816 conn(0x7f542006ac90 msgr2=0x7f54200b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 shutdown_connections 2024-04-17T10:34:00.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:00.897+0000 7f5426000640 1 -- 172.21.15.67:0/3795417816 wait complete. 2024-04-17T10:34:00.909 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:34:00.910 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb955800640 1 Processor -- start 2024-04-17T10:34:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb955800640 1 -- start start 2024-04-17T10:34:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb955800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb955800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9500714d0 0x7fb95006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb955800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9500b0cc0 con 0x7fb9500b57b0 2024-04-17T10:34:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb955800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9500b1280 con 0x7fb9500714d0 2024-04-17T10:34:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb955800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9500b0fa0 con 0x7fb95015dd10 2024-04-17T10:34:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb947e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb9500b57b0 0x7fb950070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34354/0 (socket says 172.21.15.67:34354) 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb947e00640 1 -- 172.21.15.67:0/3418636614 learned_addr learned my addr 172.21.15.67:0/3418636614 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb94f400640 1 --2- 172.21.15.67:0/3418636614 >> v2:172.21.15.67:3300/0 conn(0x7fb9500714d0 0x7fb95006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb94ea00640 1 --2- 172.21.15.67:0/3418636614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb94f400640 1 -- 172.21.15.67:0/3418636614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 msgr2=0x7fb9500b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb94f400640 1 --2- 172.21.15.67:0/3418636614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb94f400640 1 -- 172.21.15.67:0/3418636614 >> v1:172.21.15.67:6789/0 conn(0x7fb9500b57b0 legacy=0x7fb950070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb94f400640 1 -- 172.21.15.67:0/3418636614 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95006bed0 con 0x7fb9500714d0 2024-04-17T10:34:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.057+0000 7fb94e000640 1 -- 172.21.15.67:0/3418636614 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951850851 0 0) 0x7fb9500b0cc0 con 0x7fb9500b57b0 2024-04-17T10:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb94f400640 1 --2- 172.21.15.67:0/3418636614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95006c6d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb948009d90 tx=0x7fb94801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceebc9b9f865e281 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb94e000640 1 -- 172.21.15.67:0/3418636614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9480b2070 con 0x7fb9500714d0 2024-04-17T10:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb94e000640 1 -- 172.21.15.67:0/3418636614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9480a5950 con 0x7fb9500714d0 2024-04-17T10:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb94e000640 1 -- 172.21.15.67:0/3418636614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9480a5d20 con 0x7fb9500714d0 2024-04-17T10:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 -- 172.21.15.67:0/3418636614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 msgr2=0x7fb95006c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 --2- 172.21.15.67:0/3418636614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95006c6d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fb948009d90 tx=0x7fb94801c8b0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 -- 172.21.15.67:0/3418636614 shutdown_connections 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 --2- 172.21.15.67:0/3418636614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95006c6d0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 --2- 172.21.15.67:0/3418636614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 -- 172.21.15.67:0/3418636614 >> 172.21.15.67:0/3418636614 conn(0x7fb9500b34d0 msgr2=0x7fb9500b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 -- 172.21.15.67:0/3418636614 shutdown_connections 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.061+0000 7fb955800640 1 -- 172.21.15.67:0/3418636614 wait complete. 2024-04-17T10:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 Processor -- start 2024-04-17T10:34:01.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 -- start start 2024-04-17T10:34:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95014c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9500cab50 0x7fb9501528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95006d7b0 con 0x7fb9500714d0 2024-04-17T10:34:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb95006d630 con 0x7fb9500cab50 2024-04-17T10:34:01.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95006d930 con 0x7fb95015dd10 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95014c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95014c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40560/0 (socket says 172.21.15.67:40560) 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 -- 172.21.15.67:0/2409355490 learned_addr learned my addr 172.21.15.67:0/2409355490 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 -- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 msgr2=0x7fb9500cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb947e00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94f400640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9500cab50 0x7fb9501528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 -- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9500cab50 msgr2=0x7fb9501528d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9500cab50 0x7fb9501528d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 -- 172.21.15.67:0/2409355490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9500ce860 con 0x7fb9500714d0 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94f400640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9500cab50 0x7fb9501528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb947e00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb94ea00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95014c4c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fb94000e8a0 tx=0x7fb94000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:01.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb946a00640 1 -- 172.21.15.67:0/2409355490 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb940017070 con 0x7fb9500714d0 2024-04-17T10:34:01.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb946a00640 1 -- 172.21.15.67:0/2409355490 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94000c940 con 0x7fb9500714d0 2024-04-17T10:34:01.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9500cead0 con 0x7fb9500714d0 2024-04-17T10:34:01.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb946a00640 1 -- 172.21.15.67:0/2409355490 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb94000ccf0 con 0x7fb9500714d0 2024-04-17T10:34:01.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9500cf090 con 0x7fb9500714d0 2024-04-17T10:34:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.065+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb950070bb0 con 0x7fb9500714d0 2024-04-17T10:34:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.069+0000 7fb946a00640 1 -- 172.21.15.67:0/2409355490 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb94001a070 con 0x7fb9500714d0 2024-04-17T10:34:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.069+0000 7fb946a00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb92c066f90 0x7fb92c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.069+0000 7fb946a00640 1 -- 172.21.15.67:0/2409355490 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb940085140 con 0x7fb9500714d0 2024-04-17T10:34:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.069+0000 7fb947e00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb92c066f90 0x7fb92c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:01.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.069+0000 7fb947e00640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb92c066f90 0x7fb92c069450 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb9501fa480 tx=0x7fb938009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:01.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.073+0000 7fb946a00640 1 -- 172.21.15.67:0/2409355490 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb940055f20 con 0x7fb9500714d0 2024-04-17T10:34:01.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.289+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb950074e30 con 0x7fb9500714d0 2024-04-17T10:34:01.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb946a00640 1 -- 172.21.15.67:0/2409355490 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1446 (secure 0 0 0) 0x7fb940055820 con 0x7fb9500714d0 2024-04-17T10:34:01.284 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:01.284 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":572,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076506864,"bytes_used":2597904384,"bytes_avail":770496208896,"bytes_total":773094113280,"write_bytes_sec":42998565,"read_op_per_sec":0,"write_op_per_sec":43},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":21,"modified":"2024-04-17T10:33:18.256212+0000","services":{"mds":{"daemons":{"summary":"","b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:01.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb92c066f90 msgr2=0x7fb92c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb92c066f90 0x7fb92c069450 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fb9501fa480 tx=0x7fb938009040 comp rx=0 tx=0).stop 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 msgr2=0x7fb95014c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95014c4c0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fb94000e8a0 tx=0x7fb94000eca0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 shutdown_connections 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb92c066f90 0x7fb92c069450 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9500cab50 0x7fb9501528d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb95015dd10 0x7fb9500cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 --2- 172.21.15.67:0/2409355490 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9500714d0 0x7fb95014c4c0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 >> 172.21.15.67:0/2409355490 conn(0x7fb9500b34d0 msgr2=0x7fb95015c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 shutdown_connections 2024-04-17T10:34:01.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:01.293+0000 7fb955800640 1 -- 172.21.15.67:0/2409355490 wait complete. 2024-04-17T10:34:01.305 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:34:02.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:02.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.465+0000 7fdf18e00640 1 Processor -- start 2024-04-17T10:34:02.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.465+0000 7fdf18e00640 1 -- start start 2024-04-17T10:34:02.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf18e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1406eac0 0x7fdf14143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf18e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdf140cf0c0 0x7fdf14144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf18e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf14144560 con 0x7fdf1406e6e0 2024-04-17T10:34:02.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf18e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf1415ae40 con 0x7fdf140cf0c0 2024-04-17T10:34:02.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf18e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf14071440 con 0x7fdf1406eac0 2024-04-17T10:34:02.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf11e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdf1406e6e0 0x7fdf140ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34360/0 (socket says 172.21.15.67:34360) 2024-04-17T10:34:02.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf11e00640 1 -- 172.21.15.67:0/2268737717 learned_addr learned my addr 172.21.15.67:0/2268737717 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:02.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf0be00640 1 --2- 172.21.15.67:0/2268737717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1406eac0 0x7fdf14143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf12800640 1 --2- 172.21.15.67:0/2268737717 >> v2:172.21.15.86:3300/0 conn(0x7fdf140cf0c0 0x7fdf14144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf11400640 1 -- 172.21.15.67:0/2268737717 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019410647 0 0) 0x7fdf14144560 con 0x7fdf1406e6e0 2024-04-17T10:34:02.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf11400640 1 -- 172.21.15.67:0/2268737717 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdef8003530 con 0x7fdf1406e6e0 2024-04-17T10:34:02.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf0be00640 1 -- 172.21.15.67:0/2268737717 >> v2:172.21.15.86:3300/0 conn(0x7fdf140cf0c0 msgr2=0x7fdf14144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:02.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf0be00640 1 --2- 172.21.15.67:0/2268737717 >> v2:172.21.15.86:3300/0 conn(0x7fdf140cf0c0 0x7fdf14144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf0be00640 1 -- 172.21.15.67:0/2268737717 >> v1:172.21.15.67:6789/0 conn(0x7fdf1406e6e0 legacy=0x7fdf140ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.469+0000 7fdf0be00640 1 -- 172.21.15.67:0/2268737717 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf14071720 con 0x7fdf1406eac0 2024-04-17T10:34:02.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf0be00640 1 --2- 172.21.15.67:0/2268737717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1406eac0 0x7fdf14143ae0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0097c0 tx=0x7fdefc01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa2171a26ad4d32e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:02.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf11400640 1 -- 172.21.15.67:0/2268737717 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0ab600 con 0x7fdf1406eac0 2024-04-17T10:34:02.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf11400640 1 -- 172.21.15.67:0/2268737717 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc0b0070 con 0x7fdf1406eac0 2024-04-17T10:34:02.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf11400640 1 -- 172.21.15.67:0/2268737717 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0ab9c0 con 0x7fdf1406eac0 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 -- 172.21.15.67:0/2268737717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1406eac0 msgr2=0x7fdf14143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2268737717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1406eac0 0x7fdf14143ae0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdefc0097c0 tx=0x7fdefc01c3e0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 -- 172.21.15.67:0/2268737717 shutdown_connections 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2268737717 >> v2:172.21.15.86:3300/0 conn(0x7fdf140cf0c0 0x7fdf14144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2268737717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf1406eac0 0x7fdf14143ae0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 -- 172.21.15.67:0/2268737717 >> 172.21.15.67:0/2268737717 conn(0x7fdf1406ac90 msgr2=0x7fdf1406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 -- 172.21.15.67:0/2268737717 shutdown_connections 2024-04-17T10:34:02.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 -- 172.21.15.67:0/2268737717 wait complete. 2024-04-17T10:34:02.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 Processor -- start 2024-04-17T10:34:02.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.473+0000 7fdf18e00640 1 -- start start 2024-04-17T10:34:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf1406e6e0 0x7fdf1413bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1406eac0 0x7fdf14140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf140cf0c0 0x7fdf141416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140b3a40 con 0x7fdf1406eac0 2024-04-17T10:34:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140b38c0 con 0x7fdf1406e6e0 2024-04-17T10:34:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140b3bc0 con 0x7fdf140cf0c0 2024-04-17T10:34:02.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1406eac0 0x7fdf14140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1406eac0 0x7fdf14140ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40572/0 (socket says 172.21.15.67:40572) 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 -- 172.21.15.67:0/2257723876 learned_addr learned my addr 172.21.15.67:0/2257723876 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf12800640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf140cf0c0 0x7fdf141416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf11e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf1406e6e0 0x7fdf1413bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 -- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf140cf0c0 msgr2=0x7fdf141416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf140cf0c0 0x7fdf141416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 -- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf1406e6e0 msgr2=0x7fdf1413bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf1406e6e0 0x7fdf1413bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 -- 172.21.15.67:0/2257723876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1413fa30 con 0x7fdf1406eac0 2024-04-17T10:34:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0be00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1406eac0 0x7fdf14140ad0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdefc01a010 tx=0x7fdefc01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0a000640 1 -- 172.21.15.67:0/2257723876 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0bf410 con 0x7fdf1406eac0 2024-04-17T10:34:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0a000640 1 -- 172.21.15.67:0/2257723876 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdefc0bf5b0 con 0x7fdf1406eac0 2024-04-17T10:34:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1413fd10 con 0x7fdf1406eac0 2024-04-17T10:34:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0a000640 1 -- 172.21.15.67:0/2257723876 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdefc0bf9a0 con 0x7fdf1406eac0 2024-04-17T10:34:02.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf141402d0 con 0x7fdf1406eac0 2024-04-17T10:34:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf14072d50 con 0x7fdf1406eac0 2024-04-17T10:34:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.477+0000 7fdf0a000640 1 -- 172.21.15.67:0/2257723876 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdefc0bd020 con 0x7fdf1406eac0 2024-04-17T10:34:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.481+0000 7fdf0a000640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdef0066f60 0x7fdef0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.481+0000 7fdf0a000640 1 -- 172.21.15.67:0/2257723876 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdefc11f620 con 0x7fdf1406eac0 2024-04-17T10:34:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.481+0000 7fdf11e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdef0066f60 0x7fdef0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.481+0000 7fdf11e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdef0066f60 0x7fdef0069420 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdf04005890 tx=0x7fdf04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.485+0000 7fdf0a000640 1 -- 172.21.15.67:0/2257723876 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdefc016090 con 0x7fdf1406eac0 2024-04-17T10:34:02.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.681+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf140b2890 con 0x7fdf1406eac0 2024-04-17T10:34:02.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.681+0000 7fdf0a000640 1 -- 172.21.15.67:0/2257723876 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fdefc0efdb0 con 0x7fdf1406eac0 2024-04-17T10:34:02.675 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:02.675 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:02.675 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdef0066f60 msgr2=0x7fdef0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdef0066f60 0x7fdef0069420 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fdf04005890 tx=0x7fdf04009040 comp rx=0 tx=0).stop 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1406eac0 msgr2=0x7fdf14140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1406eac0 0x7fdf14140ad0 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdefc01a010 tx=0x7fdefc01ac20 comp rx=0 tx=0).stop 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 shutdown_connections 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdef0066f60 0x7fdef0069420 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdf140cf0c0 0x7fdf141416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf1406eac0 0x7fdf14140ad0 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 --2- 172.21.15.67:0/2257723876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdf1406e6e0 0x7fdf1413bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 >> 172.21.15.67:0/2257723876 conn(0x7fdf1406ac90 msgr2=0x7fdf14144000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 shutdown_connections 2024-04-17T10:34:02.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.685+0000 7fdf18e00640 1 -- 172.21.15.67:0/2257723876 wait complete. 2024-04-17T10:34:02.696 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:34:02.696 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:02.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e72e00640 1 Processor -- start 2024-04-17T10:34:02.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e72e00640 1 -- start start 2024-04-17T10:34:02.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e72e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0cc250 0x7f3e6c0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e72e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e6c0cb260 0x7f3e6c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e72e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6c06b100 con 0x7f3e6c0ce2a0 2024-04-17T10:34:02.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e72e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6c0cd150 con 0x7f3e6c0cb260 2024-04-17T10:34:02.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e72e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c06b3e0 con 0x7f3e6c0cc250 2024-04-17T10:34:02.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e70e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3e6c0ce2a0 0x7f3e6c0cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34370/0 (socket says 172.21.15.67:34370) 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e70e00640 1 -- 172.21.15.67:0/1094501607 learned_addr learned my addr 172.21.15.67:0/1094501607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e63400640 1 --2- 172.21.15.67:0/1094501607 >> v2:172.21.15.67:3300/0 conn(0x7f3e6c0cb260 0x7f3e6c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e6be00640 1 --2- 172.21.15.67:0/1094501607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0cc250 0x7f3e6c0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e63400640 1 -- 172.21.15.67:0/1094501607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0cc250 msgr2=0x7f3e6c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e63400640 1 --2- 172.21.15.67:0/1094501607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0cc250 0x7f3e6c0cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e63400640 1 -- 172.21.15.67:0/1094501607 >> v1:172.21.15.67:6789/0 conn(0x7f3e6c0ce2a0 legacy=0x7f3e6c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e63400640 1 -- 172.21.15.67:0/1094501607 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c06b6c0 con 0x7f3e6c0cb260 2024-04-17T10:34:02.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.845+0000 7f3e6b400640 1 -- 172.21.15.67:0/1094501607 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949962971 0 0) 0x7f3e6c06b100 con 0x7f3e6c0ce2a0 2024-04-17T10:34:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e63400640 1 --2- 172.21.15.67:0/1094501607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0cd5e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009d90 tx=0x7f3e5401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dce9bc87774eae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e6b400640 1 -- 172.21.15.67:0/1094501607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e540b2070 con 0x7f3e6c0cb260 2024-04-17T10:34:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e6b400640 1 -- 172.21.15.67:0/1094501607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e540a5830 con 0x7f3e6c0cb260 2024-04-17T10:34:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e6b400640 1 -- 172.21.15.67:0/1094501607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e540a5c00 con 0x7f3e6c0cb260 2024-04-17T10:34:02.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 -- 172.21.15.67:0/1094501607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 msgr2=0x7f3e6c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 --2- 172.21.15.67:0/1094501607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0cd5e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009d90 tx=0x7f3e5401c970 comp rx=0 tx=0).stop 2024-04-17T10:34:02.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 -- 172.21.15.67:0/1094501607 shutdown_connections 2024-04-17T10:34:02.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 --2- 172.21.15.67:0/1094501607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0cd5e0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 --2- 172.21.15.67:0/1094501607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0cc250 0x7f3e6c0cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 -- 172.21.15.67:0/1094501607 >> 172.21.15.67:0/1094501607 conn(0x7f3e6c0b22a0 msgr2=0x7f3e6c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:02.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 -- 172.21.15.67:0/1094501607 shutdown_connections 2024-04-17T10:34:02.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.849+0000 7f3e72e00640 1 -- 172.21.15.67:0/1094501607 wait complete. 2024-04-17T10:34:02.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 Processor -- start 2024-04-17T10:34:02.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 -- start start 2024-04-17T10:34:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0c4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e6c0cc250 0x7f3e6c15f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0ce2a0 0x7f3e6c15d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c0cdef0 con 0x7f3e6c0cb260 2024-04-17T10:34:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c0cdd70 con 0x7f3e6c0cc250 2024-04-17T10:34:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6c16b100 con 0x7f3e6c0ce2a0 2024-04-17T10:34:02.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0c4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0c4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40598/0 (socket says 172.21.15.67:40598) 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 -- 172.21.15.67:0/3530742553 learned_addr learned my addr 172.21.15.67:0/3530742553 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e70e00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0ce2a0 0x7f3e6c15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e63400640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e6c0cc250 0x7f3e6c15f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 -- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0ce2a0 msgr2=0x7f3e6c15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0ce2a0 0x7f3e6c15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 -- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e6c0cc250 msgr2=0x7f3e6c15f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e6c0cc250 0x7f3e6c15f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 -- 172.21.15.67:0/3530742553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c0afce0 con 0x7f3e6c0cb260 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e6be00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0c4140 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c00e930 tx=0x7f3e5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e68c00640 1 -- 172.21.15.67:0/3530742553 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c017070 con 0x7f3e6c0cb260 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e68c00640 1 -- 172.21.15.67:0/3530742553 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c00ca00 con 0x7f3e6c0cb260 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e68c00640 1 -- 172.21.15.67:0/3530742553 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c00cdb0 con 0x7f3e6c0cb260 2024-04-17T10:34:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6c0c47d0 con 0x7f3e6c0cb260 2024-04-17T10:34:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.853+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6c14f8e0 con 0x7f3e6c0cb260 2024-04-17T10:34:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.857+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e6c06cd30 con 0x7f3e6c0cb260 2024-04-17T10:34:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.857+0000 7f3e68c00640 1 -- 172.21.15.67:0/3530742553 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3e5c01a070 con 0x7f3e6c0cb260 2024-04-17T10:34:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.857+0000 7f3e68c00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e48066f60 0x7f3e48069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.857+0000 7f3e68c00640 1 -- 172.21.15.67:0/3530742553 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3e5c085290 con 0x7f3e6c0cb260 2024-04-17T10:34:02.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.857+0000 7f3e63400640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e48066f60 0x7f3e48069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:02.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.857+0000 7f3e63400640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e48066f60 0x7f3e48069420 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3e540af000 tx=0x7f3e540a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:02.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:02.861+0000 7f3e68c00640 1 -- 172.21.15.67:0/3530742553 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3e5c056130 con 0x7f3e6c0cb260 2024-04-17T10:34:03.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3e6c074e30 con 0x7f3e6c0cb260 2024-04-17T10:34:03.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e68c00640 1 -- 172.21.15.67:0/3530742553 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7f3e5c055a30 con 0x7f3e6c0cb260 2024-04-17T10:34:03.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:03.072 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":573,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076583881,"bytes_used":2598060032,"bytes_avail":770496053248,"bytes_total":773094113280,"write_bytes_sec":43005623,"read_op_per_sec":0,"write_op_per_sec":43},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-17T10:34:02.266878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e48066f60 msgr2=0x7f3e48069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e48066f60 0x7f3e48069420 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f3e540af000 tx=0x7f3e540a7070 comp rx=0 tx=0).stop 2024-04-17T10:34:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 msgr2=0x7f3e6c0c4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0c4140 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c00e930 tx=0x7f3e5c00edf0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 shutdown_connections 2024-04-17T10:34:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e48066f60 0x7f3e48069420 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e6c0ce2a0 0x7f3e6c15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e6c0cc250 0x7f3e6c15f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 --2- 172.21.15.67:0/3530742553 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e6c0cb260 0x7f3e6c0c4140 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 >> 172.21.15.67:0/3530742553 conn(0x7f3e6c0b22a0 msgr2=0x7f3e6c0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 shutdown_connections 2024-04-17T10:34:03.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.081+0000 7f3e72e00640 1 -- 172.21.15.67:0/3530742553 wait complete. 2024-04-17T10:34:03.093 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:34:03.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:03.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.8:0 scrub abort 2024-04-17T10:34:03.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe496000640 1 Processor -- start 2024-04-17T10:34:03.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe496000640 1 -- start start 2024-04-17T10:34:03.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe496000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4901583f0 0x7fe490072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe496000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4900728c0 0x7fe4900b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe496000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49015f360 con 0x7fe4900b5130 2024-04-17T10:34:03.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe496000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe49006bca0 con 0x7fe4900728c0 2024-04-17T10:34:03.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe496000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900b3160 con 0x7fe4901583f0 2024-04-17T10:34:03.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe487e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe4900728c0 0x7fe4900b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe48fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe4900b5130 0x7fe490161980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34372/0 (socket says 172.21.15.67:34372) 2024-04-17T10:34:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe48fe00640 1 -- 172.21.15.67:0/2215378306 learned_addr learned my addr 172.21.15.67:0/2215378306 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe48f400640 1 --2- 172.21.15.67:0/2215378306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4901583f0 0x7fe490072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe487e00640 1 -- 172.21.15.67:0/2215378306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4901583f0 msgr2=0x7fe490072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe487e00640 1 --2- 172.21.15.67:0/2215378306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4901583f0 0x7fe490072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe487e00640 1 -- 172.21.15.67:0/2215378306 >> v1:172.21.15.67:6789/0 conn(0x7fe4900b5130 legacy=0x7fe490161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe487e00640 1 -- 172.21.15.67:0/2215378306 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4900b3440 con 0x7fe4900728c0 2024-04-17T10:34:03.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.245+0000 7fe48ea00640 1 -- 172.21.15.67:0/2215378306 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975868906 0 0) 0x7fe49015f360 con 0x7fe4900b5130 2024-04-17T10:34:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe487e00640 1 --2- 172.21.15.67:0/2215378306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4900728c0 0x7fe4900b4bf0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe4780097c0 tx=0x7fe47801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b5d4b56a6c2709 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe48ea00640 1 -- 172.21.15.67:0/2215378306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780b0070 con 0x7fe4900728c0 2024-04-17T10:34:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe48ea00640 1 -- 172.21.15.67:0/2215378306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4780ab600 con 0x7fe4900728c0 2024-04-17T10:34:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe48ea00640 1 -- 172.21.15.67:0/2215378306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780ab9d0 con 0x7fe4900728c0 2024-04-17T10:34:03.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 -- 172.21.15.67:0/2215378306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4900728c0 msgr2=0x7fe4900b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 --2- 172.21.15.67:0/2215378306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4900728c0 0x7fe4900b4bf0 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7fe4780097c0 tx=0x7fe47801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 -- 172.21.15.67:0/2215378306 shutdown_connections 2024-04-17T10:34:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 --2- 172.21.15.67:0/2215378306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4900728c0 0x7fe4900b4bf0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 --2- 172.21.15.67:0/2215378306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4901583f0 0x7fe490072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 -- 172.21.15.67:0/2215378306 >> 172.21.15.67:0/2215378306 conn(0x7fe4900b2570 msgr2=0x7fe490071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 -- 172.21.15.67:0/2215378306 shutdown_connections 2024-04-17T10:34:03.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.249+0000 7fe496000640 1 -- 172.21.15.67:0/2215378306 wait complete. 2024-04-17T10:34:03.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 Processor -- start 2024-04-17T10:34:03.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 -- start start 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4900728c0 0x7fe49014ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4900b5130 0x7fe490153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4901583f0 0x7fe4901535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900b4200 con 0x7fe4901583f0 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4901587d0 con 0x7fe4900b5130 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4900b5540 con 0x7fe4900728c0 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4901583f0 0x7fe4901535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4901583f0 0x7fe4901535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40620/0 (socket says 172.21.15.67:40620) 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 -- 172.21.15.67:0/381836062 learned_addr learned my addr 172.21.15.67:0/381836062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48f400640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4900728c0 0x7fe49014ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe487e00640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4900b5130 0x7fe490153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 -- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4900728c0 msgr2=0x7fe49014ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4900728c0 0x7fe49014ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 -- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4900b5130 msgr2=0x7fe490153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4900b5130 0x7fe490153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 -- 172.21.15.67:0/381836062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49014efd0 con 0x7fe4901583f0 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48f400640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4900728c0 0x7fe49014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe487e00640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4900b5130 0x7fe490153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe48fe00640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4901583f0 0x7fe4901535a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c7a0 tx=0x7fe48800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe487400640 1 -- 172.21.15.67:0/381836062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488018070 con 0x7fe4901583f0 2024-04-17T10:34:03.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe487400640 1 -- 172.21.15.67:0/381836062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48800a640 con 0x7fe4901583f0 2024-04-17T10:34:03.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe487400640 1 -- 172.21.15.67:0/381836062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48800aa70 con 0x7fe4901583f0 2024-04-17T10:34:03.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49014f270 con 0x7fe4901583f0 2024-04-17T10:34:03.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.253+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49014f830 con 0x7fe4901583f0 2024-04-17T10:34:03.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487400640 1 -- 172.21.15.67:0/381836062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe48801b070 con 0x7fe4901583f0 2024-04-17T10:34:03.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487400640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe46c066f60 0x7fe46c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487400640 1 -- 172.21.15.67:0/381836062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe488084830 con 0x7fe4901583f0 2024-04-17T10:34:03.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe48f400640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe46c066f60 0x7fe46c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 Processor -- start 2024-04-17T10:34:03.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 -- start start 2024-04-17T10:34:03.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe48f400640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe46c066f60 0x7fe46c069420 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe4901576d0 tx=0x7fe480008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c35237d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c35237e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3522601f0 con 0x55c3523c3360 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3521c6e90 con 0x55c35237dd90 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c35214ef50 con 0x55c3523c2a40 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c35237d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c35237d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40632/0 (socket says 172.21.15.67:40632) 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 -- 172.21.15.67:0/4111036880 learned_addr learned my addr 172.21.15.67:0/4111036880 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe48f400640 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe48fe00640 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c35237e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 -- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 msgr2=0x55c3523c2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523c2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 -- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 msgr2=0x55c35237e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c35237e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.257+0000 7fe487e00640 1 -- 172.21.15.67:0/4111036880 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c35219e8b0 con 0x55c3523c3360 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe487e00640 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c35237d850 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe47801ab50 tx=0x7fe478004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92842845fad1d811 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe47fe00640 1 -- 172.21.15.67:0/4111036880 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4780b0070 con 0x55c3523c3360 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe47fe00640 1 -- 172.21.15.67:0/4111036880 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4780b1040 con 0x55c3523c3360 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe47fe00640 1 -- 172.21.15.67:0/4111036880 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478003620 con 0x55c3523c3360 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- 172.21.15.67:0/4111036880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 msgr2=0x55c35237d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c35237d850 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7fe47801ab50 tx=0x7fe478004740 comp rx=0 tx=0).stop 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- 172.21.15.67:0/4111036880 shutdown_connections 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c35237e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c35237d850 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 --2- 172.21.15.67:0/4111036880 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523c2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- 172.21.15.67:0/4111036880 >> 172.21.15.67:0/4111036880 conn(0x55c352379d40 msgr2=0x55c35237c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- 172.21.15.67:0/4111036880 shutdown_connections 2024-04-17T10:34:03.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- 172.21.15.67:0/4111036880 wait complete. 2024-04-17T10:34:03.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- start start 2024-04-17T10:34:03.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 Processor -- start 2024-04-17T10:34:03.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c3523ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523ca700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c352383d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c352181540 con 0x55c3523c3360 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3521c2f30 con 0x55c35237dd90 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c352172260 con 0x55c3523c2a40 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe49940e000 10 client.0 _ll_register_callbacks cb 0x55c35227b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c352383d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c352383d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40646/0 (socket says 172.21.15.67:40646) 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 -- 172.21.15.67:0/2626848330 learned_addr learned my addr 172.21.15.67:0/2626848330 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe487e00640 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48f400640 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c3523ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 -- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 msgr2=0x55c3523ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 -- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 msgr2=0x55c3523ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c3523ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.261+0000 7fe48fe00640 1 -- 172.21.15.67:0/2626848330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c35214f1a0 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe48fe00640 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c352383d80 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c770 tx=0x7fe488055610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1c289346e88ce2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 1 -- 172.21.15.67:0/2626848330 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48805d390 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 1 -- 172.21.15.67:0/2626848330 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe48805b070 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe4780b1040 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 1 -- 172.21.15.67:0/2626848330 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe488059060 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 10 client.8952 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55c352249840 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 1 -- 172.21.15.67:0/2626848330 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4780b1040 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 1 -- 172.21.15.67:0/2626848330 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe488054360 con 0x55c3523c3360 2024-04-17T10:34:03.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 10 client.8952 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 10 client.8952 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 20 client.8952 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352384b20 0x55c352384f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 4 client.8952 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c352162be0 con 0x55c352384b20 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe48f400640 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352384b20 0x55c352384f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 10 client.8952 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe48f400640 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352384b20 0x55c352384f00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fffc862937a9dd36 server_cookie=2b4e1d047d6e6085 in_seq=0 out_seq=0 2024-04-17T10:34:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 1 -- 172.21.15.67:0/2626848330 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c352162be0 con 0x55c352384b20 2024-04-17T10:34:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 10 client.8952 handle_command_reply: tid=0 2024-04-17T10:34:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 client.8952 shutdown 2024-04-17T10:34:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 msgr2=0x55c352383d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c352383d80 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c770 tx=0x7fe488055610 comp rx=0 tx=0).stop 2024-04-17T10:34:03.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 shutdown_connections 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352384b20 0x55c352384f00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c352383d80 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c3523ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 --2- 172.21.15.67:0/2626848330 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c35237dd90 0x55c3523ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe47cc00640 0 client.8952 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 >> 172.21.15.67:0/2626848330 conn(0x55c352379d40 msgr2=0x55c35237c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 shutdown_connections 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 1 -- 172.21.15.67:0/2626848330 wait complete. 2024-04-17T10:34:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.265+0000 7fe49940e000 20 client.8952 trim_cache size 0 max 16384 2024-04-17T10:34:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 Processor -- start 2024-04-17T10:34:03.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 -- start start 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35237dd90 0x55c35241f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c35241f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c3360 0x55c35241fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe490203120 con 0x55c35237dd90 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4902023b0 con 0x55c3523c3360 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe49014c770 con 0x55c3523c2a40 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35237dd90 0x55c35241f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35237dd90 0x55c35241f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40650/0 (socket says 172.21.15.67:40650) 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 -- 172.21.15.67:0/3130811502 learned_addr learned my addr 172.21.15.67:0/3130811502 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 -- 172.21.15.67:0/3130811502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 msgr2=0x55c35241f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 --2- 172.21.15.67:0/3130811502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c35241f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 -- 172.21.15.67:0/3130811502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c3360 msgr2=0x55c35241fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 --2- 172.21.15.67:0/3130811502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c3360 0x55c35241fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 -- 172.21.15.67:0/3130811502 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c352220a10 con 0x55c35237dd90 2024-04-17T10:34:03.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe48f400640 1 --2- 172.21.15.67:0/3130811502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35237dd90 0x55c35241f080 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fe48000fd50 tx=0x7fe480014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d17818bf0177c8e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe47f400640 1 -- 172.21.15.67:0/3130811502 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48001b040 con 0x55c35237dd90 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe47f400640 1 -- 172.21.15.67:0/3130811502 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48001c040 con 0x55c35237dd90 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe47f400640 1 -- 172.21.15.67:0/3130811502 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480029d00 con 0x55c35237dd90 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 -- 172.21.15.67:0/3130811502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35237dd90 msgr2=0x55c35241f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 --2- 172.21.15.67:0/3130811502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35237dd90 0x55c35241f080 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fe48000fd50 tx=0x7fe480014000 comp rx=0 tx=0).stop 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 -- 172.21.15.67:0/3130811502 shutdown_connections 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 --2- 172.21.15.67:0/3130811502 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c3360 0x55c35241fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 --2- 172.21.15.67:0/3130811502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c3523c2a40 0x55c35241f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 --2- 172.21.15.67:0/3130811502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c35237dd90 0x55c35241f080 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.281+0000 7fe49940e000 1 -- 172.21.15.67:0/3130811502 >> 172.21.15.67:0/3130811502 conn(0x55c352379d40 msgr2=0x55c3523d08a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- 172.21.15.67:0/3130811502 shutdown_connections 2024-04-17T10:34:03.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- 172.21.15.67:0/3130811502 wait complete. 2024-04-17T10:34:03.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- start start 2024-04-17T10:34:03.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 Processor -- start 2024-04-17T10:34:03.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c35237dd90 0x55c3523c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c2a40 0x55c3523c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c3523c8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3521937c0 con 0x55c3523c3360 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49014cd40 con 0x55c3523c2a40 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c3521c61a0 con 0x55c35237dd90 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 10 client.0 _ll_register_callbacks cb 0x55c352189a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c3523c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c3523c8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40652/0 (socket says 172.21.15.67:40652) 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 -- 172.21.15.67:0/477120515 learned_addr learned my addr 172.21.15.67:0/477120515 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe487e00640 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c2a40 0x55c3523c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48f400640 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c35237dd90 0x55c3523c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 -- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c35237dd90 msgr2=0x55c3523c7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c35237dd90 0x55c3523c7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 -- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c2a40 msgr2=0x55c3523c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c2a40 0x55c3523c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 -- 172.21.15.67:0/477120515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c35215bb00 con 0x55c3523c3360 2024-04-17T10:34:03.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48fe00640 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c3523c8520 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe48805d0f0 tx=0x7fe488082a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e762692f3e493410 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe47cc00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe47cc00640 1 -- 172.21.15.67:0/477120515 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe488050c60 con 0x55c3523c3360 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe47cc00640 1 -- 172.21.15.67:0/477120515 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fe48807dcd0 con 0x55c3523c3360 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe48001c040 con 0x55c3523c3360 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe47cc00640 1 -- 172.21.15.67:0/477120515 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe488050e00 con 0x55c3523c3360 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 10 client.8958 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55c35216df80 con 0x55c3523c3360 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe47cc00640 1 -- 172.21.15.67:0/477120515 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48001c040 con 0x55c3523c3360 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe47cc00640 1 -- 172.21.15.67:0/477120515 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe488053dd0 con 0x55c3523c3360 2024-04-17T10:34:03.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 10 client.8958 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 10 client.8958 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 20 client.8958 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352428cd0 0x55c35242b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 4 client.8958 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "scrub abort"}) v1 -- 0x55c3521d43d0 con 0x55c352428cd0 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.285+0000 7fe48f400640 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352428cd0 0x55c35242b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe48f400640 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352428cd0 0x55c35242b0c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2466437dda196369 server_cookie=605d8592b6643ef0 in_seq=0 out_seq=0 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe47cc00640 10 client.8958 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe47cc00640 1 -- 172.21.15.67:0/477120515 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x55c3521d43d0 con 0x55c352428cd0 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe47cc00640 10 client.8958 handle_command_reply: tid=0 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 client.8958 shutdown 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 msgr2=0x55c3523c8520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c3523c8520 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fe48805d0f0 tx=0x7fe488082a00 comp rx=0 tx=0).stop 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 shutdown_connections 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55c352428cd0 0x55c35242b0c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55c3523c3360 0x55c3523c8520 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55c3523c2a40 0x55c3523c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 --2- 172.21.15.67:0/477120515 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55c35237dd90 0x55c3523c7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe47cc00640 0 client.8958 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 >> 172.21.15.67:0/477120515 conn(0x55c352379d40 msgr2=0x55c3523c10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 shutdown_connections 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 1 -- 172.21.15.67:0/477120515 wait complete. 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe49940e000 20 client.8958 trim_cache size 0 max 16384 2024-04-17T10:34:03.284 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-04-17T10:34:03.285 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe46c066f60 msgr2=0x7fe46c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe46c066f60 0x7fe46c069420 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fe4901576d0 tx=0x7fe480008040 comp rx=0 tx=0).stop 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4901583f0 msgr2=0x7fe4901535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4901583f0 0x7fe4901535a0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7fe48800c7a0 tx=0x7fe48800cc60 comp rx=0 tx=0).stop 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 shutdown_connections 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe46c066f60 0x7fe46c069420 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe4901583f0 0x7fe4901535a0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe4900b5130 0x7fe490153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 --2- 172.21.15.67:0/381836062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe4900728c0 0x7fe49014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 >> 172.21.15.67:0/381836062 conn(0x7fe4900b2570 msgr2=0x7fe490151880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 shutdown_connections 2024-04-17T10:34:03.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.289+0000 7fe496000640 1 -- 172.21.15.67:0/381836062 wait complete. 2024-04-17T10:34:03.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:03.300 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.8:0 scrub status 2024-04-17T10:34:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.449+0000 7f3c80400640 1 Processor -- start 2024-04-17T10:34:03.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.449+0000 7f3c80400640 1 -- start start 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c80400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c780aef60 0x7f3c780cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c80400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c80400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c7806e7b0 0x7f3c780713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c80400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780cd2c0 con 0x7f3c780aef60 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c80400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780af340 con 0x7f3c7806e7b0 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c80400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780b4990 con 0x7f3c780cce60 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c780aef60 0x7f3c780cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c780aef60 0x7f3c780cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40658/0 (socket says 172.21.15.67:40658) 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 -- 172.21.15.67:0/3426914732 learned_addr learned my addr 172.21.15.67:0/3426914732 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7d000640 1 --2- 172.21.15.67:0/3426914732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7e400640 1 --2- 172.21.15.67:0/3426914732 >> v2:172.21.15.86:3300/0 conn(0x7f3c7806e7b0 0x7f3c780713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 -- 172.21.15.67:0/3426914732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 msgr2=0x7f3c7806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 --2- 172.21.15.67:0/3426914732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 -- 172.21.15.67:0/3426914732 >> v2:172.21.15.86:3300/0 conn(0x7f3c7806e7b0 msgr2=0x7f3c780713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 --2- 172.21.15.67:0/3426914732 >> v2:172.21.15.86:3300/0 conn(0x7f3c7806e7b0 0x7f3c780713e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 -- 172.21.15.67:0/3426914732 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7815ae90 con 0x7f3c780aef60 2024-04-17T10:34:03.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.453+0000 7f3c7da00640 1 --2- 172.21.15.67:0/3426914732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780cc920 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3c780b2900 tx=0x7f3c6c01c1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24c79750c42097cb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c77e00640 1 -- 172.21.15.67:0/3426914732 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c0b0070 con 0x7f3c780aef60 2024-04-17T10:34:03.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c77e00640 1 -- 172.21.15.67:0/3426914732 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c0ab600 con 0x7f3c780aef60 2024-04-17T10:34:03.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c77e00640 1 -- 172.21.15.67:0/3426914732 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6c0ab9d0 con 0x7f3c780aef60 2024-04-17T10:34:03.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- 172.21.15.67:0/3426914732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 msgr2=0x7f3c780cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 --2- 172.21.15.67:0/3426914732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780cc920 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f3c780b2900 tx=0x7f3c6c01c1d0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- 172.21.15.67:0/3426914732 shutdown_connections 2024-04-17T10:34:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 --2- 172.21.15.67:0/3426914732 >> v2:172.21.15.86:3300/0 conn(0x7f3c7806e7b0 0x7f3c780713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 --2- 172.21.15.67:0/3426914732 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 --2- 172.21.15.67:0/3426914732 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780cc920 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- 172.21.15.67:0/3426914732 >> 172.21.15.67:0/3426914732 conn(0x7f3c780b1740 msgr2=0x7f3c780b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- 172.21.15.67:0/3426914732 shutdown_connections 2024-04-17T10:34:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- 172.21.15.67:0/3426914732 wait complete. 2024-04-17T10:34:03.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 Processor -- start 2024-04-17T10:34:03.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- start start 2024-04-17T10:34:03.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c7806e7b0 0x7f3c781f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7813bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7815bc10 con 0x7f3c780aef60 2024-04-17T10:34:03.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7815b410 con 0x7f3c7806e7b0 2024-04-17T10:34:03.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c80400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7814ea30 con 0x7f3c780cce60 2024-04-17T10:34:03.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40660/0 (socket says 172.21.15.67:40660) 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c7d000640 1 -- 172.21.15.67:0/1399542427 learned_addr learned my addr 172.21.15.67:0/1399542427 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.457+0000 7f3c7da00640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c7806e7b0 0x7f3c781f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c7e400640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7813bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c7d000640 1 -- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 msgr2=0x7f3c7813bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7813bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c7d000640 1 -- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c7806e7b0 msgr2=0x7f3c781f9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c7806e7b0 0x7f3c781f9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c7d000640 1 -- 172.21.15.67:0/1399542427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7814ec80 con 0x7f3c780aef60 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780a84b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400d800 tx=0x7f3c6400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c75600640 1 -- 172.21.15.67:0/1399542427 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6400b6c0 con 0x7f3c780aef60 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c75600640 1 -- 172.21.15.67:0/1399542427 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c64019070 con 0x7f3c780aef60 2024-04-17T10:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7814ef60 con 0x7f3c780aef60 2024-04-17T10:34:03.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c75600640 1 -- 172.21.15.67:0/1399542427 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6400ba60 con 0x7f3c780aef60 2024-04-17T10:34:03.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7814f520 con 0x7f3c780aef60 2024-04-17T10:34:03.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.461+0000 7f3c75600640 1 -- 172.21.15.67:0/1399542427 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3c6400bc00 con 0x7f3c780aef60 2024-04-17T10:34:03.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c75600640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c58066f60 0x7f3c58069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c75600640 1 -- 172.21.15.67:0/1399542427 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3c640855f0 con 0x7f3c780aef60 2024-04-17T10:34:03.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 Processor -- start 2024-04-17T10:34:03.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- start start 2024-04-17T10:34:03.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7da00640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c58066f60 0x7f3c58069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d9a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d95a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06d7a6ff0 con 0x55a06d9a42f0 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06d72f150 con 0x55a06d9a39d0 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a06d7a3090 con 0x55a06d95a0a0 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7da00640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c58066f60 0x7f3c58069420 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ad000 tx=0x7f3c6c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40664/0 (socket says 172.21.15.67:40664) 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 -- 172.21.15.67:0/1169702443 learned_addr learned my addr 172.21.15.67:0/1169702443 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7da00640 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d9a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7e400640 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d95a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 -- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 msgr2=0x55a06d95a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d95a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 -- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 msgr2=0x55a06d9a3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d9a3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 -- 172.21.15.67:0/1169702443 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a06d93d0d0 con 0x55a06d9a42f0 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959b60 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3c6405deb0 tx=0x7f3c640559f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5544bf52d3faeb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c6a000640 1 -- 172.21.15.67:0/1169702443 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6405c0c0 con 0x55a06d9a42f0 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c6a000640 1 -- 172.21.15.67:0/1169702443 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c64054080 con 0x55a06d9a42f0 2024-04-17T10:34:03.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c6a000640 1 -- 172.21.15.67:0/1169702443 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c640551d0 con 0x55a06d9a42f0 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- 172.21.15.67:0/1169702443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 msgr2=0x55a06d959b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959b60 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f3c6405deb0 tx=0x7f3c640559f0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- 172.21.15.67:0/1169702443 shutdown_connections 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d95a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959b60 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1169702443 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d9a3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- 172.21.15.67:0/1169702443 >> 172.21.15.67:0/1169702443 conn(0x55a06d956020 msgr2=0x55a06d958440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- 172.21.15.67:0/1169702443 shutdown_connections 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- 172.21.15.67:0/1169702443 wait complete. 2024-04-17T10:34:03.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.465+0000 7f3c839a2000 1 -- start start 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 Processor -- start 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9887f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a39d0 0x55a06d988d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a42f0 0x55a06d989270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06d76b250 con 0x55a06d9a39d0 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06d761690 con 0x55a06d9a42f0 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a06d72fa40 con 0x55a06d95a0a0 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 10 client.0 _ll_register_callbacks cb 0x55a06d944e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a39d0 0x55a06d988d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7da00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9887f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a39d0 0x55a06d988d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40670/0 (socket says 172.21.15.67:40670) 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 -- 172.21.15.67:0/1726838521 learned_addr learned my addr 172.21.15.67:0/1726838521 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7e400640 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a42f0 0x55a06d989270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 -- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 msgr2=0x55a06d9887f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9887f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 -- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a42f0 msgr2=0x55a06d989270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a42f0 0x55a06d989270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 -- 172.21.15.67:0/1726838521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a06d8541c0 con 0x55a06d9a39d0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7e400640 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a42f0 0x55a06d989270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7d000640 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a39d0 0x55a06d988d30 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3c64055a80 tx=0x7f3c64085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d9f4126ce619360 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c5ea00640 1 -- 172.21.15.67:0/1726838521 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64071020 con 0x55a06d9a39d0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c5ea00640 1 -- 172.21.15.67:0/1726838521 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3c640553d0 con 0x55a06d9a39d0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c64054080 con 0x55a06d9a39d0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c5ea00640 1 -- 172.21.15.67:0/1726838521 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c6407c050 con 0x55a06d9a39d0 2024-04-17T10:34:03.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 10 client.8970 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55a06d850170 con 0x55a06d9a39d0 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c5ea00640 1 -- 172.21.15.67:0/1726838521 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c64054080 con 0x55a06d9a39d0 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c5ea00640 1 -- 172.21.15.67:0/1726838521 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3c6407f040 con 0x55a06d9a39d0 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 10 client.8970 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 10 client.8970 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 20 client.8970 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b3c80 0x55a06d9b6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 4 client.8970 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a06d742c60 con 0x55a06d9b3c80 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7da00640 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b3c80 0x55a06d9b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c7da00640 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b3c80 0x55a06d9b6070 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f50f4c57aa8335d server_cookie=eb59c34bf6e61b84 in_seq=0 out_seq=0 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.469+0000 7f3c5ea00640 10 client.8970 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c5ea00640 1 -- 172.21.15.67:0/1726838521 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a06d742c60 con 0x55a06d9b3c80 2024-04-17T10:34:03.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c5ea00640 10 client.8970 handle_command_reply: tid=0 2024-04-17T10:34:03.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 client.8970 shutdown 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a39d0 msgr2=0x55a06d988d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a39d0 0x55a06d988d30 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f3c64055a80 tx=0x7f3c64085040 comp rx=0 tx=0).stop 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 shutdown_connections 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b3c80 0x55a06d9b6070 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a42f0 0x55a06d989270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a39d0 0x55a06d988d30 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 --2- 172.21.15.67:0/1726838521 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9887f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c5ea00640 0 client.8970 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 >> 172.21.15.67:0/1726838521 conn(0x55a06d956020 msgr2=0x55a06d958ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 shutdown_connections 2024-04-17T10:34:03.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 1 -- 172.21.15.67:0/1726838521 wait complete. 2024-04-17T10:34:03.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.473+0000 7f3c839a2000 20 client.8970 trim_cache size 0 max 16384 2024-04-17T10:34:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.485+0000 7f3c839a2000 1 Processor -- start 2024-04-17T10:34:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.485+0000 7f3c839a2000 1 -- start start 2024-04-17T10:34:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9a2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d959420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c782035d0 con 0x55a06d9a42f0 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c782027e0 con 0x55a06d9a39d0 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c78150d40 con 0x55a06d95a0a0 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40684/0 (socket says 172.21.15.67:40684) 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 -- 172.21.15.67:0/922658961 learned_addr learned my addr 172.21.15.67:0/922658961 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7da00640 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9a2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7d000640 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d959420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 -- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 msgr2=0x55a06d9a2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9a2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 -- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 msgr2=0x55a06d959420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d959420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 -- 172.21.15.67:0/922658961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a06d7edde0 con 0x55a06d9a42f0 2024-04-17T10:34:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7da00640 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7d000640 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d959420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c7e400640 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959960 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f3c700039d0 tx=0x7f3c70003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7431fe22ab80dce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c69600640 1 -- 172.21.15.67:0/922658961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7001b070 con 0x55a06d9a42f0 2024-04-17T10:34:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c69600640 1 -- 172.21.15.67:0/922658961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70017440 con 0x55a06d9a42f0 2024-04-17T10:34:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c69600640 1 -- 172.21.15.67:0/922658961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70017810 con 0x55a06d9a42f0 2024-04-17T10:34:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- 172.21.15.67:0/922658961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 msgr2=0x55a06d959960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959960 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f3c700039d0 tx=0x7f3c70003e90 comp rx=0 tx=0).stop 2024-04-17T10:34:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- 172.21.15.67:0/922658961 shutdown_connections 2024-04-17T10:34:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d959960 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d9a39d0 0x55a06d959420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 --2- 172.21.15.67:0/922658961 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d95a0a0 0x55a06d9a2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- 172.21.15.67:0/922658961 >> 172.21.15.67:0/922658961 conn(0x55a06d956020 msgr2=0x55a06d9a20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- 172.21.15.67:0/922658961 shutdown_connections 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- 172.21.15.67:0/922658961 wait complete. 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 -- start start 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.489+0000 7f3c839a2000 1 Processor -- start 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d95a0a0 0x55a06da04310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d9a39d0 0x55a06da04850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d9b6380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06d741e00 con 0x55a06d9a42f0 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c782021e0 con 0x55a06d95a0a0 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a06d7a6300 con 0x55a06d9a39d0 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 10 client.0 _ll_register_callbacks cb 0x55a06d925040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d9b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d9a39d0 0x55a06da04850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7da00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d95a0a0 0x55a06da04310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7d000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d9a39d0 0x55a06da04850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:32822/0 (socket says 172.21.15.67:32822) 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7d000640 1 -- 172.21.15.67:0/2509248246 learned_addr learned my addr 172.21.15.67:0/2509248246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d9b6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40700/0 (socket says 172.21.15.67:40700) 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 -- 172.21.15.67:0/2509248246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d9a39d0 msgr2=0x55a06da04850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d9a39d0 0x55a06da04850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 -- 172.21.15.67:0/2509248246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d95a0a0 msgr2=0x55a06da04310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d95a0a0 0x55a06da04310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 -- 172.21.15.67:0/2509248246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a06d7488e0 con 0x55a06d9a42f0 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7e400640 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d9b6380 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f3c7001d000 tx=0x7f3c70004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f83f6f5238772a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 1 -- 172.21.15.67:0/2509248246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7001b050 con 0x55a06d9a42f0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 1 -- 172.21.15.67:0/2509248246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3c70020450 con 0x55a06d9a42f0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c70017440 con 0x55a06d9a42f0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 1 -- 172.21.15.67:0/2509248246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c70030050 con 0x55a06d9a42f0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 10 client.8976 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55a06d8008f0 con 0x55a06d9a42f0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 1 -- 172.21.15.67:0/2509248246 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70017440 con 0x55a06d9a42f0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 1 -- 172.21.15.67:0/2509248246 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f3c70033020 con 0x55a06d9a42f0 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 10 client.8976 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 10 client.8976 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 20 client.8976 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b6ea0 0x55a06d9b72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 4 client.8976 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a06d7b4530 con 0x55a06d9b6ea0 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7da00640 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b6ea0 0x55a06d9b72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c7da00640 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b6ea0 0x55a06d9b72b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=132f398cb61a1f17 server_cookie=af68dcb35d426781 in_seq=0 out_seq=0 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 10 client.8976 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 1 -- 172.21.15.67:0/2509248246 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+69 (crc 0 0 0) 0x55a06d7b4530 con 0x55a06d9b6ea0 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 10 client.8976 handle_command_reply: tid=0 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 client.8976 shutdown 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 msgr2=0x55a06d9b6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d9b6380 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f3c7001d000 tx=0x7f3c70004ac0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 shutdown_connections 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55a06d9b6ea0 0x55a06d9b72b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55a06d9a42f0 0x55a06d9b6380 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55a06d9a39d0 0x55a06da04850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 --2- 172.21.15.67:0/2509248246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55a06d95a0a0 0x55a06da04310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c5ea00640 0 client.8976 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 >> 172.21.15.67:0/2509248246 conn(0x55a06d956020 msgr2=0x55a06d9a2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 shutdown_connections 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.493+0000 7f3c839a2000 1 -- 172.21.15.67:0/2509248246 wait complete. 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c839a2000 20 client.8976 trim_cache size 0 max 16384 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:03.491 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-17T10:34:03.492 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:34:03.492 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c58066f60 msgr2=0x7f3c58069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c58066f60 0x7f3c58069420 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ad000 tx=0x7f3c6c008040 comp rx=0 tx=0).stop 2024-04-17T10:34:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 msgr2=0x7f3c780a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780a84b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f3c6400d800 tx=0x7f3c6400dcc0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 shutdown_connections 2024-04-17T10:34:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c58066f60 0x7f3c58069420 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c780cce60 0x7f3c7813bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c780aef60 0x7f3c780a84b0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 --2- 172.21.15.67:0/1399542427 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c7806e7b0 0x7f3c781f9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 >> 172.21.15.67:0/1399542427 conn(0x7f3c780b1740 msgr2=0x7f3c780af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 shutdown_connections 2024-04-17T10:34:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.497+0000 7f3c80400640 1 -- 172.21.15.67:0/1399542427 wait complete. 2024-04-17T10:34:03.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:03.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a31000640 1 Processor -- start 2024-04-17T10:34:03.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a31000640 1 -- start start 2024-04-17T10:34:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a31000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c145040 0x7f5a2c06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a31000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a2c06ef10 0x7f5a2c071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a31000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a2c06e2f0 con 0x7f5a2c071850 2024-04-17T10:34:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a31000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a2c0a86c0 con 0x7f5a2c06ef10 2024-04-17T10:34:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a31000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c15bf10 con 0x7f5a2c145040 2024-04-17T10:34:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a2c06ef10 0x7f5a2c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a23e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a2c06ef10 0x7f5a2c071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40712/0 (socket says 172.21.15.67:40712) 2024-04-17T10:34:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a23e00640 1 -- 172.21.15.67:0/1560337059 learned_addr learned my addr 172.21.15.67:0/1560337059 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a29e00640 1 --2- 172.21.15.67:0/1560337059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c145040 0x7f5a2c06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a23e00640 1 -- 172.21.15.67:0/1560337059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c145040 msgr2=0x7f5a2c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a23e00640 1 --2- 172.21.15.67:0/1560337059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c145040 0x7f5a2c06e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a23e00640 1 -- 172.21.15.67:0/1560337059 >> v1:172.21.15.86:6789/0 conn(0x7f5a2c071850 legacy=0x7f5a2c15f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.657+0000 7f5a23e00640 1 -- 172.21.15.67:0/1560337059 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c06d6e0 con 0x7f5a2c06ef10 2024-04-17T10:34:03.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a23e00640 1 --2- 172.21.15.67:0/1560337059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c06ef10 0x7f5a2c071310 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009870 tx=0x7f5a1401ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ecfb31870d5416 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a29400640 1 -- 172.21.15.67:0/1560337059 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a140b5070 con 0x7f5a2c06ef10 2024-04-17T10:34:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a29400640 1 -- 172.21.15.67:0/1560337059 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a140a8a50 con 0x7f5a2c06ef10 2024-04-17T10:34:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a29400640 1 -- 172.21.15.67:0/1560337059 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a140a8e20 con 0x7f5a2c06ef10 2024-04-17T10:34:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 -- 172.21.15.67:0/1560337059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c06ef10 msgr2=0x7f5a2c071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 --2- 172.21.15.67:0/1560337059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c06ef10 0x7f5a2c071310 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f5a14009870 tx=0x7f5a1401ed50 comp rx=0 tx=0).stop 2024-04-17T10:34:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 -- 172.21.15.67:0/1560337059 shutdown_connections 2024-04-17T10:34:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 --2- 172.21.15.67:0/1560337059 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c06ef10 0x7f5a2c071310 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 --2- 172.21.15.67:0/1560337059 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c145040 0x7f5a2c06e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 -- 172.21.15.67:0/1560337059 >> 172.21.15.67:0/1560337059 conn(0x7f5a2c06ac90 msgr2=0x7f5a2c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 -- 172.21.15.67:0/1560337059 shutdown_connections 2024-04-17T10:34:03.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.661+0000 7f5a31000640 1 -- 172.21.15.67:0/1560337059 wait complete. 2024-04-17T10:34:03.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 Processor -- start 2024-04-17T10:34:03.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 -- start start 2024-04-17T10:34:03.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c06ef10 0x7f5a2c135880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c071850 0x7f5a2c1f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a2c145040 0x7f5a2c13e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c06dca0 con 0x7f5a2c071850 2024-04-17T10:34:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c071ca0 con 0x7f5a2c145040 2024-04-17T10:34:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2c06df80 con 0x7f5a2c06ef10 2024-04-17T10:34:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c071850 0x7f5a2c1f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c071850 0x7f5a2c1f1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40722/0 (socket says 172.21.15.67:40722) 2024-04-17T10:34:03.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 -- 172.21.15.67:0/854908246 learned_addr learned my addr 172.21.15.67:0/854908246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a29e00640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c06ef10 0x7f5a2c135880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a2a800640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a2c145040 0x7f5a2c13e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 -- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c06ef10 msgr2=0x7f5a2c135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c06ef10 0x7f5a2c135880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 -- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a2c145040 msgr2=0x7f5a2c13e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a2c145040 0x7f5a2c13e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 -- 172.21.15.67:0/854908246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a2c137720 con 0x7f5a2c071850 2024-04-17T10:34:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a29e00640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c06ef10 0x7f5a2c135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a23e00640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c071850 0x7f5a2c1f1c10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f5a140b3040 tx=0x7f5a1401baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a22000640 1 -- 172.21.15.67:0/854908246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a140c0070 con 0x7f5a2c071850 2024-04-17T10:34:03.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a22000640 1 -- 172.21.15.67:0/854908246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a140ae550 con 0x7f5a2c071850 2024-04-17T10:34:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2c1385e0 con 0x7f5a2c071850 2024-04-17T10:34:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a22000640 1 -- 172.21.15.67:0/854908246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a140b5070 con 0x7f5a2c071850 2024-04-17T10:34:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2c138ba0 con 0x7f5a2c071850 2024-04-17T10:34:03.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.665+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2c072d50 con 0x7f5a2c071850 2024-04-17T10:34:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.669+0000 7f5a22000640 1 -- 172.21.15.67:0/854908246 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5a140c3070 con 0x7f5a2c071850 2024-04-17T10:34:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.669+0000 7f5a22000640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5a08066f60 0x7f5a08069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.669+0000 7f5a22000640 1 -- 172.21.15.67:0/854908246 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5a14122b30 con 0x7f5a2c071850 2024-04-17T10:34:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.669+0000 7f5a29e00640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5a08066f60 0x7f5a08069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.669+0000 7f5a29e00640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5a08066f60 0x7f5a08069420 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00b9f0 tx=0x7f5a1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:03.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.673+0000 7f5a22000640 1 -- 172.21.15.67:0/854908246 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5a140188b0 con 0x7f5a2c071850 2024-04-17T10:34:03.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a2c074e30 con 0x7f5a2c071850 2024-04-17T10:34:03.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a22000640 1 -- 172.21.15.67:0/854908246 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f5a140f3910 con 0x7f5a2c071850 2024-04-17T10:34:03.861 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:03.861 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:03.861 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:03.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5a08066f60 msgr2=0x7f5a08069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5a08066f60 0x7f5a08069420 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c00b9f0 tx=0x7f5a1c008040 comp rx=0 tx=0).stop 2024-04-17T10:34:03.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c071850 msgr2=0x7f5a2c1f1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:03.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c071850 0x7f5a2c1f1c10 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f5a140b3040 tx=0x7f5a1401baf0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 shutdown_connections 2024-04-17T10:34:03.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5a08066f60 0x7f5a08069420 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5a2c145040 0x7f5a2c13e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a2c071850 0x7f5a2c1f1c10 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 --2- 172.21.15.67:0/854908246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5a2c06ef10 0x7f5a2c135880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:03.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 >> 172.21.15.67:0/854908246 conn(0x7f5a2c06ac90 msgr2=0x7f5a2c0af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:03.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 shutdown_connections 2024-04-17T10:34:03.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:03.869+0000 7f5a31000640 1 -- 172.21.15.67:0/854908246 wait complete. 2024-04-17T10:34:03.882 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:34:03.882 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff440e00640 1 Processor -- start 2024-04-17T10:34:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff440e00640 1 -- start start 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff440e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff43c0ce020 0x7ff43c1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff440e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0cf510 0x7ff43c0cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff440e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43c15eb50 con 0x7ff43c0ce400 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff440e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff43c161650 con 0x7ff43c0ce020 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff440e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c1625e0 con 0x7ff43c0cf510 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.029+0000 7ff439e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff43c0ce020 0x7ff43c1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff439e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff43c0ce020 0x7ff43c1620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40726/0 (socket says 172.21.15.67:40726) 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff439e00640 1 -- 172.21.15.67:0/482236052 learned_addr learned my addr 172.21.15.67:0/482236052 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff43a800640 1 --2- 172.21.15.67:0/482236052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0cf510 0x7ff43c0cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff439e00640 1 -- 172.21.15.67:0/482236052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0cf510 msgr2=0x7ff43c0cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff439e00640 1 --2- 172.21.15.67:0/482236052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0cf510 0x7ff43c0cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff439e00640 1 -- 172.21.15.67:0/482236052 >> v1:172.21.15.86:6789/0 conn(0x7ff43c0ce400 legacy=0x7ff43c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff439e00640 1 -- 172.21.15.67:0/482236052 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c15ee30 con 0x7ff43c0ce020 2024-04-17T10:34:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff439e00640 1 --2- 172.21.15.67:0/482236052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce020 0x7ff43c1620a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0098a0 tx=0x7ff42c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc05b6e14667529 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff438a00640 1 -- 172.21.15.67:0/482236052 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c0ab620 con 0x7ff43c0ce020 2024-04-17T10:34:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff438a00640 1 -- 172.21.15.67:0/482236052 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c0b0070 con 0x7ff43c0ce020 2024-04-17T10:34:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.033+0000 7ff438a00640 1 -- 172.21.15.67:0/482236052 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c0ab9e0 con 0x7ff43c0ce020 2024-04-17T10:34:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- 172.21.15.67:0/482236052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce020 msgr2=0x7ff43c1620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 --2- 172.21.15.67:0/482236052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce020 0x7ff43c1620a0 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0098a0 tx=0x7ff42c01c510 comp rx=0 tx=0).stop 2024-04-17T10:34:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- 172.21.15.67:0/482236052 shutdown_connections 2024-04-17T10:34:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 --2- 172.21.15.67:0/482236052 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0cf510 0x7ff43c0cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 --2- 172.21.15.67:0/482236052 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce020 0x7ff43c1620a0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- 172.21.15.67:0/482236052 >> 172.21.15.67:0/482236052 conn(0x7ff43c0b33f0 msgr2=0x7ff43c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- 172.21.15.67:0/482236052 shutdown_connections 2024-04-17T10:34:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- 172.21.15.67:0/482236052 wait complete. 2024-04-17T10:34:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 Processor -- start 2024-04-17T10:34:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- start start 2024-04-17T10:34:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0ce020 0x7ff43c0b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce400 0x7ff43c156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cf510 0x7ff43c071260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0ce990 con 0x7ff43c0ce400 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c0ce810 con 0x7ff43c0cf510 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff440e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c15e720 con 0x7ff43c0ce020 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce400 0x7ff43c156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce400 0x7ff43c156650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40732/0 (socket says 172.21.15.67:40732) 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 -- 172.21.15.67:0/3580916512 learned_addr learned my addr 172.21.15.67:0/3580916512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0ce020 0x7ff43c0b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff43a800640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cf510 0x7ff43c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 -- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0ce020 msgr2=0x7ff43c0b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0ce020 0x7ff43c0b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 -- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cf510 msgr2=0x7ff43c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cf510 0x7ff43c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 -- 172.21.15.67:0/3580916512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c156ea0 con 0x7ff43c0ce400 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.037+0000 7ff439400640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce400 0x7ff43c156650 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff42800c420 tx=0x7ff42800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff432000640 1 -- 172.21.15.67:0/3580916512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4280052d0 con 0x7ff43c0ce400 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff432000640 1 -- 172.21.15.67:0/3580916512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff428005470 con 0x7ff43c0ce400 2024-04-17T10:34:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff432000640 1 -- 172.21.15.67:0/3580916512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff428007260 con 0x7ff43c0ce400 2024-04-17T10:34:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff43c157180 con 0x7ff43c0ce400 2024-04-17T10:34:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43c157740 con 0x7ff43c0ce400 2024-04-17T10:34:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43c070ab0 con 0x7ff43c0ce400 2024-04-17T10:34:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff432000640 1 -- 172.21.15.67:0/3580916512 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff428007400 con 0x7ff43c0ce400 2024-04-17T10:34:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff432000640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff41c066f60 0x7ff41c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff432000640 1 -- 172.21.15.67:0/3580916512 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff4280846e0 con 0x7ff43c0ce400 2024-04-17T10:34:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.041+0000 7ff439e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff41c066f60 0x7ff41c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.045+0000 7ff439e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff41c066f60 0x7ff41c069420 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0af000 tx=0x7ff42c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.045+0000 7ff432000640 1 -- 172.21.15.67:0/3580916512 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff428054e70 con 0x7ff43c0ce400 2024-04-17T10:34:04.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.265+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff43c074e30 con 0x7ff43c0ce400 2024-04-17T10:34:04.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff432000640 1 -- 172.21.15.67:0/3580916512 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7ff428058f10 con 0x7ff43c0ce400 2024-04-17T10:34:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:04.260 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":575,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076583881,"bytes_used":2598060032,"bytes_avail":770496053248,"bytes_total":773094113280,"write_bytes_sec":43005623,"read_op_per_sec":0,"write_op_per_sec":43},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-17T10:34:02.266878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff41c066f60 msgr2=0x7ff41c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff41c066f60 0x7ff41c069420 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7ff42c0af000 tx=0x7ff42c0a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:04.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce400 msgr2=0x7ff43c156650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce400 0x7ff43c156650 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7ff42800c420 tx=0x7ff42800c8e0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 shutdown_connections 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff41c066f60 0x7ff41c069420 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff43c0cf510 0x7ff43c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff43c0ce400 0x7ff43c156650 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 --2- 172.21.15.67:0/3580916512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff43c0ce020 0x7ff43c0b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 >> 172.21.15.67:0/3580916512 conn(0x7ff43c0b33f0 msgr2=0x7ff43c0cc140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 shutdown_connections 2024-04-17T10:34:04.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.269+0000 7ff440e00640 1 -- 172.21.15.67:0/3580916512 wait complete. 2024-04-17T10:34:04.280 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:34:04.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:04.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.8:0 scrub resume 2024-04-17T10:34:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2f000640 1 Processor -- start 2024-04-17T10:34:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2f000640 1 -- start start 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2f000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a280aef60 0x7f1a280cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2f000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a2806e980 0x7f1a280b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2f000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a280b4990 con 0x7f1a280cce60 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2f000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a280af340 con 0x7f1a280aef60 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2f000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2815ae90 con 0x7f1a2806e980 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a280aef60 0x7f1a280cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1a280aef60 0x7f1a280cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40748/0 (socket says 172.21.15.67:40748) 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a27e00640 1 -- 172.21.15.67:0/388274496 learned_addr learned my addr 172.21.15.67:0/388274496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a2d000640 1 --2- 172.21.15.67:0/388274496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a2806e980 0x7f1a280b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a27e00640 1 -- 172.21.15.67:0/388274496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a2806e980 msgr2=0x7f1a280b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a27e00640 1 --2- 172.21.15.67:0/388274496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a2806e980 0x7f1a280b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a27e00640 1 -- 172.21.15.67:0/388274496 >> v1:172.21.15.86:6789/0 conn(0x7f1a280cce60 legacy=0x7f1a2806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.433+0000 7f1a27e00640 1 -- 172.21.15.67:0/388274496 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2806ee80 con 0x7f1a280aef60 2024-04-17T10:34:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a27e00640 1 --2- 172.21.15.67:0/388274496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280aef60 0x7f1a280cc920 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009870 tx=0x7f1a1801c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78bc41a3bc03a97f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a26a00640 1 -- 172.21.15.67:0/388274496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180b0070 con 0x7f1a280aef60 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a26a00640 1 -- 172.21.15.67:0/388274496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180ab600 con 0x7f1a280aef60 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a26a00640 1 -- 172.21.15.67:0/388274496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180ab9d0 con 0x7f1a280aef60 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 -- 172.21.15.67:0/388274496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280aef60 msgr2=0x7f1a280cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 --2- 172.21.15.67:0/388274496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280aef60 0x7f1a280cc920 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009870 tx=0x7f1a1801c2b0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 -- 172.21.15.67:0/388274496 shutdown_connections 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 --2- 172.21.15.67:0/388274496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a2806e980 0x7f1a280b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 --2- 172.21.15.67:0/388274496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280aef60 0x7f1a280cc920 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 -- 172.21.15.67:0/388274496 >> 172.21.15.67:0/388274496 conn(0x7f1a280b1740 msgr2=0x7f1a280b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 -- 172.21.15.67:0/388274496 shutdown_connections 2024-04-17T10:34:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 -- 172.21.15.67:0/388274496 wait complete. 2024-04-17T10:34:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 Processor -- start 2024-04-17T10:34:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.437+0000 7f1a2f000640 1 -- start start 2024-04-17T10:34:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a2806e980 0x7f1a281fa710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a280aef60 0x7f1a281492f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280cce60 0x7f1a28142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280b3ff0 con 0x7f1a280cce60 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a280b3e70 con 0x7f1a2806e980 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a281673c0 con 0x7f1a280aef60 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280cce60 0x7f1a28142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280cce60 0x7f1a28142c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40756/0 (socket says 172.21.15.67:40756) 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a27e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a2806e980 0x7f1a281fa710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 -- 172.21.15.67:0/2397799086 learned_addr learned my addr 172.21.15.67:0/2397799086 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a27400640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a280aef60 0x7f1a281492f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 -- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a280aef60 msgr2=0x7f1a281492f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a280aef60 0x7f1a281492f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 -- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a2806e980 msgr2=0x7f1a281fa710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a2806e980 0x7f1a281fa710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 -- 172.21.15.67:0/2397799086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a281431c0 con 0x7f1a280cce60 2024-04-17T10:34:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2d000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280cce60 0x7f1a28142c80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f1a20007e90 tx=0x7f1a2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a1fe00640 1 -- 172.21.15.67:0/2397799086 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20014420 con 0x7f1a280cce60 2024-04-17T10:34:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a1fe00640 1 -- 172.21.15.67:0/2397799086 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a20019070 con 0x7f1a280cce60 2024-04-17T10:34:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a28145c40 con 0x7f1a280cce60 2024-04-17T10:34:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a1fe00640 1 -- 172.21.15.67:0/2397799086 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20014800 con 0x7f1a280cce60 2024-04-17T10:34:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.441+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a28153fc0 con 0x7f1a280cce60 2024-04-17T10:34:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a1fe00640 1 -- 172.21.15.67:0/2397799086 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1a200149a0 con 0x7f1a280cce60 2024-04-17T10:34:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a1fe00640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a08066f60 0x7f1a08069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a1fe00640 1 -- 172.21.15.67:0/2397799086 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a20085840 con 0x7f1a280cce60 2024-04-17T10:34:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a08066f60 0x7f1a08069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 Processor -- start 2024-04-17T10:34:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 -- start start 2024-04-17T10:34:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a08066f60 0x7f1a08069420 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1a180ad000 tx=0x7f1a18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007579d0 0x560900757db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090070db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090070e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56090055aff0 con 0x5609007579d0 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609004e3150 con 0x5609007582f0 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x560900557090 con 0x56090070e0a0 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007579d0 0x560900757db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007579d0 0x560900757db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40772/0 (socket says 172.21.15.67:40772) 2024-04-17T10:34:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 -- 172.21.15.67:0/563041045 learned_addr learned my addr 172.21.15.67:0/563041045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a2d000640 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090070e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27400640 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090070db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 -- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 msgr2=0x56090070e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090070e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 -- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 msgr2=0x56090070db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090070db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 -- 172.21.15.67:0/563041045 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609006f10d0 con 0x5609007579d0 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27400640 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090070db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a27e00640 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007579d0 0x560900757db0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f1a1801a960 tx=0x7f1a180a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb621e235fcbae29 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a1cc00640 1 -- 172.21.15.67:0/563041045 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180c7070 con 0x5609007579d0 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a1cc00640 1 -- 172.21.15.67:0/563041045 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180b1040 con 0x5609007579d0 2024-04-17T10:34:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.445+0000 7f1a1cc00640 1 -- 172.21.15.67:0/563041045 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180b7830 con 0x5609007579d0 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- 172.21.15.67:0/563041045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007579d0 msgr2=0x560900757db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007579d0 0x560900757db0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f1a1801a960 tx=0x7f1a180a6000 comp rx=0 tx=0).stop 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- 172.21.15.67:0/563041045 shutdown_connections 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090070e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090070db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 --2- 172.21.15.67:0/563041045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007579d0 0x560900757db0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- 172.21.15.67:0/563041045 >> 172.21.15.67:0/563041045 conn(0x56090070a020 msgr2=0x56090070c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- 172.21.15.67:0/563041045 shutdown_connections 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- 172.21.15.67:0/563041045 wait complete. 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- start start 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 Processor -- start 2024-04-17T10:34:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090073c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x56090073cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007582f0 0x56090073d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x56090051f250 con 0x5609007582f0 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x560900515690 con 0x5609007579d0 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5609004e3a40 con 0x56090070e0a0 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 10 client.0 _ll_register_callbacks cb 0x5609006f8e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007582f0 0x56090073d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007582f0 0x56090073d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40780/0 (socket says 172.21.15.67:40780) 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 -- 172.21.15.67:0/1942602250 learned_addr learned my addr 172.21.15.67:0/1942602250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a27e00640 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090073c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a27400640 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x56090073cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 -- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 msgr2=0x56090073c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090073c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 -- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 msgr2=0x56090073cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x56090073cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 -- 172.21.15.67:0/1942602250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609006081c0 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a2d000640 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007582f0 0x56090073d270 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f1a20055d60 tx=0x7f1a2005a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1fcd219b2a22cb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a11600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a11600640 1 -- 172.21.15.67:0/1942602250 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2005e430 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a11600640 1 -- 172.21.15.67:0/1942602250 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a2007ed70 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a180b1040 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a11600640 1 -- 172.21.15.67:0/1942602250 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a20054420 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 10 client.9000 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x560900604170 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.449+0000 7f1a11600640 1 -- 172.21.15.67:0/1942602250 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180b1040 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a11600640 1 -- 172.21.15.67:0/1942602250 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1a200554b0 con 0x5609007582f0 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 10 client.9000 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 10 client.9000 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 20 client.9000 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x560900767c80 0x56090076a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 4 client.9000 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5609004f6c60 con 0x560900767c80 2024-04-17T10:34:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a27e00640 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x560900767c80 0x56090076a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a27e00640 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x560900767c80 0x56090076a070 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e249b14f1f0c6816 server_cookie=a39a988ae6c1ac8a in_seq=0 out_seq=0 2024-04-17T10:34:04.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a11600640 10 client.9000 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a11600640 1 -- 172.21.15.67:0/1942602250 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5609004f6c60 con 0x560900767c80 2024-04-17T10:34:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a11600640 10 client.9000 handle_command_reply: tid=0 2024-04-17T10:34:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 client.9000 shutdown 2024-04-17T10:34:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007582f0 msgr2=0x56090073d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007582f0 0x56090073d270 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f1a20055d60 tx=0x7f1a2005a1c0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 shutdown_connections 2024-04-17T10:34:04.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x560900767c80 0x56090076a070 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5609007582f0 0x56090073d270 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x56090073cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a11600640 0 client.9000 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 --2- 172.21.15.67:0/1942602250 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x56090070e0a0 0x56090073c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 >> 172.21.15.67:0/1942602250 conn(0x56090070a020 msgr2=0x56090070cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 shutdown_connections 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 1 -- 172.21.15.67:0/1942602250 wait complete. 2024-04-17T10:34:04.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.453+0000 7f1a3250d000 20 client.9000 trim_cache size 0 max 16384 2024-04-17T10:34:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 Processor -- start 2024-04-17T10:34:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- start start 2024-04-17T10:34:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x5609007b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x5609007b4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007582f0 0x5609007b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a282033f0 con 0x56090070e0a0 2024-04-17T10:34:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a282025d0 con 0x5609007579d0 2024-04-17T10:34:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2814a5c0 con 0x5609007582f0 2024-04-17T10:34:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x5609007b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x5609007b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40786/0 (socket says 172.21.15.67:40786) 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 -- 172.21.15.67:0/3594816028 learned_addr learned my addr 172.21.15.67:0/3594816028 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a2d000640 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007582f0 0x5609007b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27400640 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x5609007b4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 -- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007582f0 msgr2=0x5609007b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007582f0 0x5609007b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 -- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 msgr2=0x5609007b4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x5609007b4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 -- 172.21.15.67:0/3594816028 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609005a1de0 con 0x56090070e0a0 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a2d000640 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007582f0 0x5609007b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27400640 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x5609007b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x5609007b4300 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f1a18008010 tx=0x7f1a180046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21bbc030d3d9bef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a13e00640 1 -- 172.21.15.67:0/3594816028 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180c7040 con 0x56090070e0a0 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a13e00640 1 -- 172.21.15.67:0/3594816028 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180d9070 con 0x56090070e0a0 2024-04-17T10:34:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a13e00640 1 -- 172.21.15.67:0/3594816028 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180b7ac0 con 0x56090070e0a0 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- 172.21.15.67:0/3594816028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 msgr2=0x5609007b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x5609007b4300 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f1a18008010 tx=0x7f1a180046d0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- 172.21.15.67:0/3594816028 shutdown_connections 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007582f0 0x5609007b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007579d0 0x5609007b4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3594816028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x5609007b4300 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- 172.21.15.67:0/3594816028 >> 172.21.15.67:0/3594816028 conn(0x56090070a020 msgr2=0x5609007560d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.469+0000 7f1a3250d000 1 -- 172.21.15.67:0/3594816028 shutdown_connections 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- 172.21.15.67:0/3594816028 wait complete. 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- start start 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 Processor -- start 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x56090076a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007579d0 0x56090076acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090076b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609004f5e00 con 0x56090070e0a0 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a28202070 con 0x5609007582f0 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x56090055a300 con 0x5609007579d0 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 10 client.0 _ll_register_callbacks cb 0x5609006d9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x56090076a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x56090076a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40790/0 (socket says 172.21.15.67:40790) 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 -- 172.21.15.67:0/3725303787 learned_addr learned my addr 172.21.15.67:0/3725303787 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27400640 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007579d0 0x56090076acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a2d000640 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090076b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 -- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007579d0 msgr2=0x56090076acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007579d0 0x56090076acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 -- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 msgr2=0x56090076b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090076b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 -- 172.21.15.67:0/3725303787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609004fc8e0 con 0x56090070e0a0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27e00640 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x56090076a770 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1a180ae040 tx=0x7f1a180b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=793a968785f89e43 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a11600640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a11600640 1 -- 172.21.15.67:0/3725303787 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a180e3070 con 0x56090070e0a0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a11600640 1 -- 172.21.15.67:0/3725303787 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1a180b78c0 con 0x56090070e0a0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1a180d9070 con 0x56090070e0a0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a11600640 1 -- 172.21.15.67:0/3725303787 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a180c7040 con 0x56090070e0a0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 10 client.9006 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x5609005b48f0 con 0x56090070e0a0 2024-04-17T10:34:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a11600640 1 -- 172.21.15.67:0/3725303787 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a180d9070 con 0x56090070e0a0 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a11600640 1 -- 172.21.15.67:0/3725303787 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f1a180e6020 con 0x56090070e0a0 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 10 client.9006 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 10 client.9006 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 20 client.9006 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5609007bdf50 0x5609007c0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 4 client.9006 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "scrub resume"}) v1 -- 0x560900568530 con 0x5609007bdf50 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.473+0000 7f1a27400640 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5609007bdf50 0x5609007c0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a11600640 10 client.9006 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a27400640 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5609007bdf50 0x5609007c0340 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=323383995f3c4daa server_cookie=46074526f67e40b4 in_seq=0 out_seq=0 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a11600640 1 -- 172.21.15.67:0/3725303787 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x560900568530 con 0x5609007bdf50 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a11600640 10 client.9006 handle_command_reply: tid=0 2024-04-17T10:34:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 client.9006 shutdown 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 msgr2=0x56090076a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x56090076a770 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f1a180ae040 tx=0x7f1a180b9040 comp rx=0 tx=0).stop 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 shutdown_connections 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x5609007bdf50 0x5609007c0340 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5609007582f0 0x56090076b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5609007579d0 0x56090076acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 --2- 172.21.15.67:0/3725303787 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x56090070e0a0 0x56090076a770 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a11600640 0 client.9006 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 >> 172.21.15.67:0/3725303787 conn(0x56090070a020 msgr2=0x560900756960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 shutdown_connections 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 1 -- 172.21.15.67:0/3725303787 wait complete. 2024-04-17T10:34:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a3250d000 20 client.9006 trim_cache size 0 max 16384 2024-04-17T10:34:04.473 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:04.473 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-04-17T10:34:04.473 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a08066f60 msgr2=0x7f1a08069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a08066f60 0x7f1a08069420 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f1a180ad000 tx=0x7f1a18008040 comp rx=0 tx=0).stop 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280cce60 msgr2=0x7f1a28142c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280cce60 0x7f1a28142c80 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f1a20007e90 tx=0x7f1a2000e4b0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 shutdown_connections 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1a08066f60 0x7f1a08069420 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1a280cce60 0x7f1a28142c80 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1a280aef60 0x7f1a281492f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 --2- 172.21.15.67:0/2397799086 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1a2806e980 0x7f1a281fa710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 >> 172.21.15.67:0/2397799086 conn(0x7f1a280b1740 msgr2=0x7f1a280afcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 shutdown_connections 2024-04-17T10:34:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.477+0000 7f1a2f000640 1 -- 172.21.15.67:0/2397799086 wait complete. 2024-04-17T10:34:04.487 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-04-17T10:34:04.487 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph tell mds.8:0 scrub status 2024-04-17T10:34:04.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.641+0000 7f8a23a00640 1 Processor -- start 2024-04-17T10:34:04.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.641+0000 7f8a23a00640 1 -- start start 2024-04-17T10:34:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a23a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8a1c06b3a0 0x7f8a1c0b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a23a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c1616a0 con 0x7f8a1c0b3920 2024-04-17T10:34:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a23a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1c161940 con 0x7f8a1c1474c0 2024-04-17T10:34:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a23a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a1c161c20 con 0x7f8a1c06b3a0 2024-04-17T10:34:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a21a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8a1c0b3920 0x7f8a1c0d0480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34380/0 (socket says 172.21.15.67:34380) 2024-04-17T10:34:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a21a00640 1 -- 172.21.15.67:0/861447253 learned_addr learned my addr 172.21.15.67:0/861447253 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a13e00640 1 --2- 172.21.15.67:0/861447253 >> v2:172.21.15.86:3300/0 conn(0x7f8a1c06b3a0 0x7f8a1c0b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a1be00640 1 -- 172.21.15.67:0/861447253 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029532174 0 0) 0x7f8a1c1616a0 con 0x7f8a1c0b3920 2024-04-17T10:34:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a1be00640 1 -- 172.21.15.67:0/861447253 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a00003530 con 0x7f8a1c0b3920 2024-04-17T10:34:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a1be00640 1 -- 172.21.15.67:0/861447253 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148868329 0 0) 0x7f8a1c161940 con 0x7f8a1c1474c0 2024-04-17T10:34:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a1be00640 1 -- 172.21.15.67:0/861447253 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a1c1616a0 con 0x7f8a1c1474c0 2024-04-17T10:34:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a13e00640 1 -- 172.21.15.67:0/861447253 >> v1:172.21.15.86:6789/0 conn(0x7f8a1c1474c0 legacy=0x7f8a1c06ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a13e00640 1 -- 172.21.15.67:0/861447253 >> v1:172.21.15.67:6789/0 conn(0x7f8a1c0b3920 legacy=0x7f8a1c0d0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.645+0000 7f8a13e00640 1 -- 172.21.15.67:0/861447253 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c161f00 con 0x7f8a1c06b3a0 2024-04-17T10:34:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a13e00640 1 --2- 172.21.15.67:0/861447253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c0b1b90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8a040097c0 tx=0x7f8a0401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=796a0989db8a24ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a1be00640 1 -- 172.21.15.67:0/861447253 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a040b5070 con 0x7f8a1c06b3a0 2024-04-17T10:34:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a1be00640 1 -- 172.21.15.67:0/861447253 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a040a8990 con 0x7f8a1c06b3a0 2024-04-17T10:34:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a1be00640 1 -- 172.21.15.67:0/861447253 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a040a8d60 con 0x7f8a1c06b3a0 2024-04-17T10:34:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 -- 172.21.15.67:0/861447253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 msgr2=0x7f8a1c0b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 --2- 172.21.15.67:0/861447253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c0b1b90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f8a040097c0 tx=0x7f8a0401ec90 comp rx=0 tx=0).stop 2024-04-17T10:34:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 -- 172.21.15.67:0/861447253 shutdown_connections 2024-04-17T10:34:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 --2- 172.21.15.67:0/861447253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c0b1b90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 -- 172.21.15.67:0/861447253 >> 172.21.15.67:0/861447253 conn(0x7f8a1c0b5ef0 msgr2=0x7f8a1c099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 -- 172.21.15.67:0/861447253 shutdown_connections 2024-04-17T10:34:04.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 -- 172.21.15.67:0/861447253 wait complete. 2024-04-17T10:34:04.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 Processor -- start 2024-04-17T10:34:04.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 -- start start 2024-04-17T10:34:04.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.649+0000 7f8a23a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c1350f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a23a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8a1c0b3920 0x7f8a1c0aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a23a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1c1474c0 0x7f8a1c0cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a23a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c162420 con 0x7f8a1c1474c0 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a23a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0b2250 con 0x7f8a1c06b3a0 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a23a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0d0b90 con 0x7f8a1c0b3920 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1c1474c0 0x7f8a1c0cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1c1474c0 0x7f8a1c0cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40804/0 (socket says 172.21.15.67:40804) 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 -- 172.21.15.67:0/3486352461 learned_addr learned my addr 172.21.15.67:0/3486352461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a13e00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8a1c0b3920 0x7f8a1c0aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21000640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c1350f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 -- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8a1c0b3920 msgr2=0x7f8a1c0aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8a1c0b3920 0x7f8a1c0aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 -- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 msgr2=0x7f8a1c1350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c1350f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 -- 172.21.15.67:0/3486352461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0ad970 con 0x7f8a1c1474c0 2024-04-17T10:34:04.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21000640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c1350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:04.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a21a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1c1474c0 0x7f8a1c0cd100 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400cbc0 tx=0x7f8a1400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a19600640 1 -- 172.21.15.67:0/3486352461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14017070 con 0x7f8a1c1474c0 2024-04-17T10:34:04.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a19600640 1 -- 172.21.15.67:0/3486352461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1400ace0 con 0x7f8a1c1474c0 2024-04-17T10:34:04.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a19600640 1 -- 172.21.15.67:0/3486352461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a14012600 con 0x7f8a1c1474c0 2024-04-17T10:34:04.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1c0adc50 con 0x7f8a1c1474c0 2024-04-17T10:34:04.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1c0ae210 con 0x7f8a1c1474c0 2024-04-17T10:34:04.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.653+0000 7f8a19600640 1 -- 172.21.15.67:0/3486352461 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8a1401a070 con 0x7f8a1c1474c0 2024-04-17T10:34:04.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a19600640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89f8066f60 0x7f89f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a19600640 1 -- 172.21.15.67:0/3486352461 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a14084f70 con 0x7f8a1c1474c0 2024-04-17T10:34:04.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a21000640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89f8066f60 0x7f89f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 Processor -- start 2024-04-17T10:34:04.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 -- start start 2024-04-17T10:34:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a21000640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89f8066f60 0x7f89f8069420 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c001640 tx=0x7f8a0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85ca8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85ca92f0 0x55ba85c5eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85c5f0a0 0x55ba85c5f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba85aabff0 con 0x55ba85ca92f0 2024-04-17T10:34:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba85a34150 con 0x55ba85c5f0a0 2024-04-17T10:34:04.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba85aa8090 con 0x55ba85ca89d0 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85ca92f0 0x55ba85c5eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85ca92f0 0x55ba85c5eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40812/0 (socket says 172.21.15.67:40812) 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 -- 172.21.15.67:0/3318179020 learned_addr learned my addr 172.21.15.67:0/3318179020 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a21000640 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85ca8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a21a00640 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85c5f0a0 0x55ba85c5f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 -- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 msgr2=0x55ba85ca8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85ca8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 -- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85c5f0a0 msgr2=0x55ba85c5f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85c5f0a0 0x55ba85c5f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 -- 172.21.15.67:0/3318179020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba85c420d0 con 0x55ba85ca92f0 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a13e00640 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85ca92f0 0x55ba85c5eb60 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8a0401b8a0 tx=0x7f8a040aa070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d9cd54a30a41e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.657+0000 7f8a11600640 1 -- 172.21.15.67:0/3318179020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a040c1070 con 0x55ba85ca92f0 2024-04-17T10:34:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a11600640 1 -- 172.21.15.67:0/3318179020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a040b6040 con 0x55ba85ca92f0 2024-04-17T10:34:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a11600640 1 -- 172.21.15.67:0/3318179020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a040b5050 con 0x55ba85ca92f0 2024-04-17T10:34:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- 172.21.15.67:0/3318179020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85ca92f0 msgr2=0x55ba85c5eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85ca92f0 0x55ba85c5eb60 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f8a0401b8a0 tx=0x7f8a040aa070 comp rx=0 tx=0).stop 2024-04-17T10:34:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- 172.21.15.67:0/3318179020 shutdown_connections 2024-04-17T10:34:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85c5f0a0 0x55ba85c5f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85ca92f0 0x55ba85c5eb60 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/3318179020 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85ca8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- 172.21.15.67:0/3318179020 >> 172.21.15.67:0/3318179020 conn(0x55ba85c5b020 msgr2=0x55ba85c5d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- 172.21.15.67:0/3318179020 shutdown_connections 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- 172.21.15.67:0/3318179020 wait complete. 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- start start 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 Processor -- start 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85c8d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85c8dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85c8e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba85a70250 con 0x55ba85c5f0a0 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba85a66690 con 0x55ba85ca92f0 2024-04-17T10:34:04.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba85a34a40 con 0x55ba85ca89d0 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85c8d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 10 client.0 _ll_register_callbacks cb 0x55ba85c49e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85c8d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40820/0 (socket says 172.21.15.67:40820) 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 -- 172.21.15.67:0/421424466 learned_addr learned my addr 172.21.15.67:0/421424466 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a13e00640 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85c8dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21a00640 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85c8e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 -- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 msgr2=0x55ba85c8dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85c8dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 -- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 msgr2=0x55ba85c8e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85c8e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 -- 172.21.15.67:0/421424466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba85b591c0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a21000640 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85c8d7f0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c00dde0 tx=0x7f8a0c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c25442fd4e1d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a0a000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a0a000640 1 -- 172.21.15.67:0/421424466 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c027060 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a0a000640 1 -- 172.21.15.67:0/421424466 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a0c022c90 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.661+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a040b6040 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a0a000640 1 -- 172.21.15.67:0/421424466 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a0c005110 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 10 client.9018 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55ba85b55170 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a0a000640 1 -- 172.21.15.67:0/421424466 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a040b6040 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a0a000640 1 -- 172.21.15.67:0/421424466 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8a0c0052b0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 10 client.9018 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 10 client.9018 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 20 client.9018 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cb8c80 0x55ba85cbb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 4 client.9018 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba85a47c60 con 0x55ba85cb8c80 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a13e00640 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cb8c80 0x55ba85cbb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a13e00640 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cb8c80 0x55ba85cbb070 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2909e80beeceab5 server_cookie=88497db980969b76 in_seq=0 out_seq=0 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a0a000640 10 client.9018 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a0a000640 1 -- 172.21.15.67:0/421424466 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ba85a47c60 con 0x55ba85cb8c80 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a0a000640 10 client.9018 handle_command_reply: tid=0 2024-04-17T10:34:04.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 client.9018 shutdown 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 msgr2=0x55ba85c8d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85c8d7f0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c00dde0 tx=0x7f8a0c000bd0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 shutdown_connections 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cb8c80 0x55ba85cbb070 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85c8e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85c8dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/421424466 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85c8d7f0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a0a000640 0 client.9018 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 >> 172.21.15.67:0/421424466 conn(0x55ba85c5b020 msgr2=0x55ba85c5dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 shutdown_connections 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 1 -- 172.21.15.67:0/421424466 wait complete. 2024-04-17T10:34:04.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.665+0000 7f8a26dd0000 20 client.9018 trim_cache size 0 max 16384 2024-04-17T10:34:04.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 Processor -- start 2024-04-17T10:34:04.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 -- start start 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d05300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca89d0 0x55ba85d05840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca92f0 0x55ba85d05d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c200320 con 0x55ba85c5f0a0 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c1ff550 con 0x55ba85ca89d0 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cb430 con 0x55ba85ca92f0 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d05300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d05300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40824/0 (socket says 172.21.15.67:40824) 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 -- 172.21.15.67:0/4083631685 learned_addr learned my addr 172.21.15.67:0/4083631685 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 -- 172.21.15.67:0/4083631685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca92f0 msgr2=0x55ba85d05d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 --2- 172.21.15.67:0/4083631685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca92f0 0x55ba85d05d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 -- 172.21.15.67:0/4083631685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca89d0 msgr2=0x55ba85d05840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 --2- 172.21.15.67:0/4083631685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca89d0 0x55ba85d05840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 -- 172.21.15.67:0/4083631685 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba85af2de0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a21000640 1 --2- 172.21.15.67:0/4083631685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d05300 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c007d50 tx=0x7f8a0c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aeedcf1641dcc5d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a10c00640 1 -- 172.21.15.67:0/4083631685 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c0052b0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a10c00640 1 -- 172.21.15.67:0/4083631685 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c0037c0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a10c00640 1 -- 172.21.15.67:0/4083631685 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c003b70 con 0x55ba85c5f0a0 2024-04-17T10:34:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 -- 172.21.15.67:0/4083631685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 msgr2=0x55ba85d05300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.681+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/4083631685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d05300 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c007d50 tx=0x7f8a0c01f000 comp rx=0 tx=0).stop 2024-04-17T10:34:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- 172.21.15.67:0/4083631685 shutdown_connections 2024-04-17T10:34:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/4083631685 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca92f0 0x55ba85d05d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/4083631685 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca89d0 0x55ba85d05840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/4083631685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d05300 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- 172.21.15.67:0/4083631685 >> 172.21.15.67:0/4083631685 conn(0x55ba85c5b020 msgr2=0x55ba85cb5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- 172.21.15.67:0/4083631685 shutdown_connections 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- 172.21.15.67:0/4083631685 wait complete. 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- start start 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 Processor -- start 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d09310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85d09850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85cbb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba85a46e00 con 0x55ba85c5f0a0 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1c0cba30 con 0x55ba85ca92f0 2024-04-17T10:34:04.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba85aab300 con 0x55ba85ca89d0 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 10 client.0 _ll_register_callbacks cb 0x55ba85c2a040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d09310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d09310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40828/0 (socket says 172.21.15.67:40828) 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 -- 172.21.15.67:0/933603645 learned_addr learned my addr 172.21.15.67:0/933603645 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a13e00640 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85d09850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21a00640 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85cbb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 -- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 msgr2=0x55ba85d09850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85d09850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 -- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 msgr2=0x55ba85cbb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85cbb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 -- 172.21.15.67:0/933603645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba85a4d8e0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a21000640 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d09310 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c009010 tx=0x7f8a0c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523deeb87710c5ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a0a000640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a0a000640 1 -- 172.21.15.67:0/933603645 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0c003620 con 0x55ba85c5f0a0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a0a000640 1 -- 172.21.15.67:0/933603645 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8a0c021450 con 0x55ba85c5f0a0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a0c0037c0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a0a000640 1 -- 172.21.15.67:0/933603645 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a0c014c80 con 0x55ba85c5f0a0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 10 client.9024 fetch_fsmap learned FSMap version 82 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x55ba85b058f0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a0a000640 1 -- 172.21.15.67:0/933603645 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a0c0037c0 con 0x55ba85c5f0a0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a0a000640 1 -- 172.21.15.67:0/933603645 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 82) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8a0c047020 con 0x55ba85c5f0a0 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 10 client.9024 fetch_fsmap finished waiting for FSMap version 82 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 10 client.9024 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 20 client.9024 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:04.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cbbd40 0x55ba85cbc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 4 client.9024 mds_command: new command op to 8670 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.685+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 --> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ba85ab9530 con 0x55ba85cbbd40 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a13e00640 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cbbd40 0x55ba85cbc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a0a000640 10 client.9024 ms_handle_connect on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a13e00640 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cbbd40 0x55ba85cbc150 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3b4954a3a9aba80 server_cookie=cfdc40984abc1b49 in_seq=0 out_seq=0 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a0a000640 1 -- 172.21.15.67:0/933603645 <== mds.0 v2:172.21.15.67:6833/397092831 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ba85ab9530 con 0x55ba85cbbd40 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a0a000640 10 client.9024 handle_command_reply: tid=0 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 client.9024 shutdown 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 msgr2=0x55ba85d09310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d09310 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c009010 tx=0x7f8a0c012000 comp rx=0 tx=0).stop 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 shutdown_connections 2024-04-17T10:34:04.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:6833/397092831,v1:172.21.15.67:6834/397092831] conn(0x55ba85cbbd40 0x55ba85cbc150 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ba85ca92f0 0x55ba85cbb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ba85ca89d0 0x55ba85d09850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 --2- 172.21.15.67:0/933603645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ba85c5f0a0 0x55ba85d09310 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a0a000640 0 client.9024 ms_handle_reset on v2:172.21.15.67:6833/397092831 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 >> 172.21.15.67:0/933603645 conn(0x55ba85c5b020 msgr2=0x55ba85ca70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 shutdown_connections 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 1 -- 172.21.15.67:0/933603645 wait complete. 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a26dd0000 20 client.9024 trim_cache size 0 max 16384 2024-04-17T10:34:04.686 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:04.687 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "no active scrubs running", 2024-04-17T10:34:04.687 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": {} 2024-04-17T10:34:04.687 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:04.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89f8066f60 msgr2=0x7f89f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89f8066f60 0x7f89f8069420 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f8a0c001640 tx=0x7f8a0c009040 comp rx=0 tx=0).stop 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1c1474c0 msgr2=0x7f8a1c0cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1c1474c0 0x7f8a1c0cd100 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400cbc0 tx=0x7f8a1400a430 comp rx=0 tx=0).stop 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 shutdown_connections 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89f8066f60 0x7f89f8069420 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a1c1474c0 0x7f8a1c0cd100 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8a1c0b3920 0x7f8a1c0aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 --2- 172.21.15.67:0/3486352461 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8a1c06b3a0 0x7f8a1c1350f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 >> 172.21.15.67:0/3486352461 conn(0x7f8a1c0b5ef0 msgr2=0x7f8a1c073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 shutdown_connections 2024-04-17T10:34:04.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.689+0000 7f8a23a00640 1 -- 172.21.15.67:0/3486352461 wait complete. 2024-04-17T10:34:04.699 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-17T10:34:04.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:04.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 Processor -- start 2024-04-17T10:34:04.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 -- start start 2024-04-17T10:34:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e015b970 0x7f65e00cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f65e006b6a0 0x7f65e00cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65e00b2dc0 0x7f65e00ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e015cf10 con 0x7f65e00b2dc0 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e015c300 con 0x7f65e006b6a0 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e8000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e015c5e0 con 0x7f65e015b970 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65e00b2dc0 0x7f65e00ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f65e00b2dc0 0x7f65e00ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40832/0 (socket says 172.21.15.67:40832) 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 -- 172.21.15.67:0/3730715992 learned_addr learned my addr 172.21.15.67:0/3730715992 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e4c00640 1 --2- 172.21.15.67:0/3730715992 >> v2:172.21.15.86:3300/0 conn(0x7f65e006b6a0 0x7f65e00cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e5600640 1 --2- 172.21.15.67:0/3730715992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e015b970 0x7f65e00cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 -- 172.21.15.67:0/3730715992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e015b970 msgr2=0x7f65e00cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 --2- 172.21.15.67:0/3730715992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e015b970 0x7f65e00cf880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 -- 172.21.15.67:0/3730715992 >> v2:172.21.15.86:3300/0 conn(0x7f65e006b6a0 msgr2=0x7f65e00cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:04.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 --2- 172.21.15.67:0/3730715992 >> v2:172.21.15.86:3300/0 conn(0x7f65e006b6a0 0x7f65e00cfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.849+0000 7f65e6000640 1 -- 172.21.15.67:0/3730715992 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e015c8c0 con 0x7f65e00b2dc0 2024-04-17T10:34:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e6000640 1 --2- 172.21.15.67:0/3730715992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e00ce2a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f65e00d17d0 tx=0x7f65d801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96105441bc00c96 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65dfe00640 1 -- 172.21.15.67:0/3730715992 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d80a56f0 con 0x7f65e00b2dc0 2024-04-17T10:34:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65dfe00640 1 -- 172.21.15.67:0/3730715992 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d80a5890 con 0x7f65e00b2dc0 2024-04-17T10:34:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65dfe00640 1 -- 172.21.15.67:0/3730715992 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65d80a5c60 con 0x7f65e00b2dc0 2024-04-17T10:34:04.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 -- 172.21.15.67:0/3730715992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 msgr2=0x7f65e00ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 --2- 172.21.15.67:0/3730715992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e00ce2a0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f65e00d17d0 tx=0x7f65d801c960 comp rx=0 tx=0).stop 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 -- 172.21.15.67:0/3730715992 shutdown_connections 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 --2- 172.21.15.67:0/3730715992 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e00ce2a0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 --2- 172.21.15.67:0/3730715992 >> v2:172.21.15.86:3300/0 conn(0x7f65e006b6a0 0x7f65e00cfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 --2- 172.21.15.67:0/3730715992 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e015b970 0x7f65e00cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 -- 172.21.15.67:0/3730715992 >> 172.21.15.67:0/3730715992 conn(0x7f65e00b22a0 msgr2=0x7f65e00b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 -- 172.21.15.67:0/3730715992 shutdown_connections 2024-04-17T10:34:04.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 -- 172.21.15.67:0/3730715992 wait complete. 2024-04-17T10:34:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.853+0000 7f65e8000640 1 Processor -- start 2024-04-17T10:34:04.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 -- start start 2024-04-17T10:34:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e006b6a0 0x7f65e0156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e0159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65e015b970 0x7f65e01596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00cc9f0 con 0x7f65e00b2dc0 2024-04-17T10:34:04.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00cc870 con 0x7f65e015b970 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e00ccb70 con 0x7f65e006b6a0 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e0159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e0159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40836/0 (socket says 172.21.15.67:40836) 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 -- 172.21.15.67:0/62145379 learned_addr learned my addr 172.21.15.67:0/62145379 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e5600640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e006b6a0 0x7f65e0156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e6000640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65e015b970 0x7f65e01596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 -- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e006b6a0 msgr2=0x7f65e0156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e006b6a0 0x7f65e0156d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 -- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65e015b970 msgr2=0x7f65e01596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65e015b970 0x7f65e01596a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:04.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 -- 172.21.15.67:0/62145379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0155c70 con 0x7f65e00b2dc0 2024-04-17T10:34:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e4c00640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e0159160 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e8e0 tx=0x7f65cc00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65dd600640 1 -- 172.21.15.67:0/62145379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc017070 con 0x7f65e00b2dc0 2024-04-17T10:34:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65dd600640 1 -- 172.21.15.67:0/62145379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc00ca00 con 0x7f65e00b2dc0 2024-04-17T10:34:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65dd600640 1 -- 172.21.15.67:0/62145379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00cdd0 con 0x7f65e00b2dc0 2024-04-17T10:34:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e0155f50 con 0x7f65e00b2dc0 2024-04-17T10:34:04.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e0156510 con 0x7f65e00b2dc0 2024-04-17T10:34:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.857+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e009e900 con 0x7f65e00b2dc0 2024-04-17T10:34:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.861+0000 7f65dd600640 1 -- 172.21.15.67:0/62145379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f65cc01a070 con 0x7f65e00b2dc0 2024-04-17T10:34:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.861+0000 7f65dd600640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65c0066f60 0x7f65c0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.861+0000 7f65dd600640 1 -- 172.21.15.67:0/62145379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f65cc0855f0 con 0x7f65e00b2dc0 2024-04-17T10:34:04.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.861+0000 7f65e5600640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65c0066f60 0x7f65c0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:04.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.861+0000 7f65e5600640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65c0066f60 0x7f65c0069420 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f65e00b6480 tx=0x7f65d4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:04.865+0000 7f65dd600640 1 -- 172.21.15.67:0/62145379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f65cc055d80 con 0x7f65e00b2dc0 2024-04-17T10:34:05.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.061+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65e0074e30 con 0x7f65e00b2dc0 2024-04-17T10:34:05.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.061+0000 7f65dd600640 1 -- 172.21.15.67:0/62145379 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f65cc059e20 con 0x7f65e00b2dc0 2024-04-17T10:34:05.053 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:05.053 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:05.054 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65c0066f60 msgr2=0x7f65c0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65c0066f60 0x7f65c0069420 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f65e00b6480 tx=0x7f65d4009190 comp rx=0 tx=0).stop 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 msgr2=0x7f65e0159160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e0159160 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00e8e0 tx=0x7f65cc00eda0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 shutdown_connections 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f65c0066f60 0x7f65c0069420 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f65e015b970 0x7f65e01596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65e00b2dc0 0x7f65e0159160 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 --2- 172.21.15.67:0/62145379 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f65e006b6a0 0x7f65e0156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 >> 172.21.15.67:0/62145379 conn(0x7f65e00b22a0 msgr2=0x7f65e0157cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 shutdown_connections 2024-04-17T10:34:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.065+0000 7f65e8000640 1 -- 172.21.15.67:0/62145379 wait complete. 2024-04-17T10:34:05.075 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:34:05.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:05.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.221+0000 7f241b400640 1 Processor -- start 2024-04-17T10:34:05.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.221+0000 7f241b400640 1 -- start start 2024-04-17T10:34:05.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f241b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24141466e0 0x7f241406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f241b400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2414146da0 0x7f2414071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f241b400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2414161740 con 0x7f2414071850 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f241b400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2414161a20 con 0x7f2414146da0 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f241b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2414161d00 con 0x7f24141466e0 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2414071850 0x7f241406d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34388/0 (socket says 172.21.15.67:34388) 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 -- 172.21.15.67:0/347040029 learned_addr learned my addr 172.21.15.67:0/347040029 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413e00640 1 --2- 172.21.15.67:0/347040029 >> v2:172.21.15.86:3300/0 conn(0x7f2414146da0 0x7f2414071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2418a00640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24141466e0 0x7f241406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861446702 0 0) 0x7f2414161740 con 0x7f2414071850 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23fc003530 con 0x7f2414071850 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 720251746 0 0) 0x7f23fc003530 con 0x7f2414071850 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24141466e0 msgr2=0x7f241406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:05.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24141466e0 0x7f241406eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 >> v2:172.21.15.86:3300/0 conn(0x7f2414146da0 msgr2=0x7f2414071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 --2- 172.21.15.67:0/347040029 >> v2:172.21.15.86:3300/0 conn(0x7f2414146da0 0x7f2414071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2414161fe0 con 0x7f2414071850 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f240c001a00 con 0x7f2414071850 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413e00640 1 --2- 172.21.15.67:0/347040029 >> v2:172.21.15.86:3300/0 conn(0x7f2414146da0 0x7f2414071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 >> v1:172.21.15.67:6789/0 conn(0x7f2414071850 legacy=0x7f241406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23fc0039c0 0x7f23fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23fc0043b0 0x7f23fc004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f23fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414161a20 con 0x7f23fc004f00 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414161d00 con 0x7f23fc0043b0 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc00bf10 con 0x7f23fc0039c0 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2418a00640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f24141466e0 0x7f241406eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f23fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2413e00640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23fc0043b0 0x7f23fc004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2418a00640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23fc0039c0 0x7f23fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 -- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23fc0039c0 msgr2=0x7f23fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23fc0039c0 0x7f23fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 -- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23fc0043b0 msgr2=0x7f23fc004890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23fc0043b0 0x7f23fc004890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2419400640 1 -- 172.21.15.67:0/347040029 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc003530 con 0x7f23fc004f00 2024-04-17T10:34:05.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.225+0000 7f2418a00640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23fc0039c0 0x7f23fc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f2419400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f23fc00b9d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f240c00a880 tx=0x7f240c01d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3c417a38ecae978 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c00a4a0 con 0x7f23fc004f00 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c0ad630 con 0x7f23fc004f00 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f2413400640 1 -- 172.21.15.67:0/347040029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c0adaa0 con 0x7f23fc004f00 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f241b400640 1 -- 172.21.15.67:0/347040029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 msgr2=0x7f23fc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f241b400640 1 --2- 172.21.15.67:0/347040029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f23fc00b9d0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f240c00a880 tx=0x7f240c01d3a0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f2419400640 1 -- 172.21.15.67:0/347040029 reap_dead start 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f241b400640 1 -- 172.21.15.67:0/347040029 shutdown_connections 2024-04-17T10:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f241b400640 1 -- 172.21.15.67:0/347040029 >> 172.21.15.67:0/347040029 conn(0x7f241406ac90 msgr2=0x7f241406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:05.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f241b400640 1 -- 172.21.15.67:0/347040029 shutdown_connections 2024-04-17T10:34:05.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.229+0000 7f241b400640 1 -- 172.21.15.67:0/347040029 wait complete. 2024-04-17T10:34:05.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 Processor -- start 2024-04-17T10:34:05.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 -- start start 2024-04-17T10:34:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f241415d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2414133c90 0x7f2414134110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2414160280 0x7f2414160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24140aef70 con 0x7f23fc004f00 2024-04-17T10:34:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24140aedf0 con 0x7f2414160280 2024-04-17T10:34:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24140af0f0 con 0x7f2414133c90 2024-04-17T10:34:05.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f241415d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f241415d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40860/0 (socket says 172.21.15.67:40860) 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 -- 172.21.15.67:0/2896191302 learned_addr learned my addr 172.21.15.67:0/2896191302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2413e00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2414133c90 0x7f2414134110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 -- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2414133c90 msgr2=0x7f2414134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2419400640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2414160280 0x7f2414160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2414133c90 0x7f2414134110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 -- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2414160280 msgr2=0x7f2414160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2414160280 0x7f2414160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 -- 172.21.15.67:0/2896191302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2414142030 con 0x7f23fc004f00 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2413e00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2414133c90 0x7f2414134110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2418a00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f241415d0b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f2408011610 tx=0x7f2408011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2410c00640 1 -- 172.21.15.67:0/2896191302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240800f8f0 con 0x7f23fc004f00 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2410c00640 1 -- 172.21.15.67:0/2896191302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240800fa90 con 0x7f23fc004f00 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f2410c00640 1 -- 172.21.15.67:0/2896191302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240800fe40 con 0x7f23fc004f00 2024-04-17T10:34:05.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2414134850 con 0x7f23fc004f00 2024-04-17T10:34:05.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.233+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2414131c50 con 0x7f23fc004f00 2024-04-17T10:34:05.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.237+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2414072d50 con 0x7f23fc004f00 2024-04-17T10:34:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.237+0000 7f2410c00640 1 -- 172.21.15.67:0/2896191302 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2408015460 con 0x7f23fc004f00 2024-04-17T10:34:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.237+0000 7f2410c00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23f4066f90 0x7f23f4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.237+0000 7f2410c00640 1 -- 172.21.15.67:0/2896191302 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f24080854e0 con 0x7f23fc004f00 2024-04-17T10:34:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.237+0000 7f2413e00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23f4066f90 0x7f23f4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:05.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.237+0000 7f2413e00640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23f4066f90 0x7f23f4069450 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f24140b3240 tx=0x7f2400008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:05.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.241+0000 7f2410c00640 1 -- 172.21.15.67:0/2896191302 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2408019030 con 0x7f23fc004f00 2024-04-17T10:34:05.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.465+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2414074e30 con 0x7f23fc004f00 2024-04-17T10:34:05.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.465+0000 7f2410c00640 1 -- 172.21.15.67:0/2896191302 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7f24080562c0 con 0x7f23fc004f00 2024-04-17T10:34:05.458 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:05.458 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":576,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076583881,"bytes_used":2598060032,"bytes_avail":770496053248,"bytes_total":773094113280,"write_bytes_sec":19366372,"read_op_per_sec":0,"write_op_per_sec":19},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-17T10:34:02.266878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23f4066f90 msgr2=0x7f23f4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23f4066f90 0x7f23f4069450 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f24140b3240 tx=0x7f2400008040 comp rx=0 tx=0).stop 2024-04-17T10:34:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 msgr2=0x7f241415d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f241415d0b0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f2408011610 tx=0x7f2408011ad0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 shutdown_connections 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23f4066f90 0x7f23f4069450 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2414160280 0x7f2414160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2414133c90 0x7f2414134110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 --2- 172.21.15.67:0/2896191302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23fc004f00 0x7f241415d0b0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 >> 172.21.15.67:0/2896191302 conn(0x7f241406ac90 msgr2=0x7f24140cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 shutdown_connections 2024-04-17T10:34:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:05.469+0000 7f241b400640 1 -- 172.21.15.67:0/2896191302 wait complete. 2024-04-17T10:34:05.479 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:34:06.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:06.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.641+0000 7f2f08400640 1 Processor -- start 2024-04-17T10:34:06.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.641+0000 7f2f08400640 1 -- start start 2024-04-17T10:34:06.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.641+0000 7f2f08400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f0015dbc0 0x7f2f0006f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:06.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.641+0000 7f2f08400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f0006db90 0x7f2f00071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:06.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f08400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f000ad8f0 con 0x7f2f000721a0 2024-04-17T10:34:06.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f08400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0015dfa0 con 0x7f2f0006db90 2024-04-17T10:34:06.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f08400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f0006faa0 con 0x7f2f0015dbc0 2024-04-17T10:34:06.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f0006db90 0x7f2f00071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f0006db90 0x7f2f00071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40872/0 (socket says 172.21.15.67:40872) 2024-04-17T10:34:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 -- 172.21.15.67:0/1388690471 learned_addr learned my addr 172.21.15.67:0/1388690471 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05a00640 1 --2- 172.21.15.67:0/1388690471 >> v2:172.21.15.86:3300/0 conn(0x7f2f0015dbc0 0x7f2f0006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 -- 172.21.15.67:0/1388690471 >> v2:172.21.15.86:3300/0 conn(0x7f2f0015dbc0 msgr2=0x7f2f0006f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:06.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 --2- 172.21.15.67:0/1388690471 >> v2:172.21.15.86:3300/0 conn(0x7f2f0015dbc0 0x7f2f0006f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 -- 172.21.15.67:0/1388690471 >> v1:172.21.15.86:6789/0 conn(0x7f2f000721a0 legacy=0x7f2f0006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 -- 172.21.15.67:0/1388690471 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000adbd0 con 0x7f2f0006db90 2024-04-17T10:34:06.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05a00640 1 --2- 172.21.15.67:0/1388690471 >> v2:172.21.15.86:3300/0 conn(0x7f2f0015dbc0 0x7f2f0006f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:06.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2f05000640 1 --2- 172.21.15.67:0/1388690471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00071c60 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2f000d0f20 tx=0x7f2eec01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc04eee8dfc9336 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:06.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2effe00640 1 -- 172.21.15.67:0/1388690471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0b0070 con 0x7f2f0006db90 2024-04-17T10:34:06.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2effe00640 1 -- 172.21.15.67:0/1388690471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec0ab600 con 0x7f2f0006db90 2024-04-17T10:34:06.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.645+0000 7f2effe00640 1 -- 172.21.15.67:0/1388690471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0ab9d0 con 0x7f2f0006db90 2024-04-17T10:34:06.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- 172.21.15.67:0/1388690471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 msgr2=0x7f2f00071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:06.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 --2- 172.21.15.67:0/1388690471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00071c60 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f2f000d0f20 tx=0x7f2eec01c450 comp rx=0 tx=0).stop 2024-04-17T10:34:06.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- 172.21.15.67:0/1388690471 shutdown_connections 2024-04-17T10:34:06.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 --2- 172.21.15.67:0/1388690471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00071c60 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 --2- 172.21.15.67:0/1388690471 >> v2:172.21.15.86:3300/0 conn(0x7f2f0015dbc0 0x7f2f0006f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- 172.21.15.67:0/1388690471 >> 172.21.15.67:0/1388690471 conn(0x7f2f000af670 msgr2=0x7f2f000b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:06.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- 172.21.15.67:0/1388690471 shutdown_connections 2024-04-17T10:34:06.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- 172.21.15.67:0/1388690471 wait complete. 2024-04-17T10:34:06.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 Processor -- start 2024-04-17T10:34:06.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- start start 2024-04-17T10:34:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:06.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f000721a0 0x7f2f0014fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f0015dbc0 0x7f2f00150430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0006c6b0 con 0x7f2f0006db90 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0006c530 con 0x7f2f0015dbc0 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f08400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0006af70 con 0x7f2f000721a0 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00155190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40884/0 (socket says 172.21.15.67:40884) 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 -- 172.21.15.67:0/270375568 learned_addr learned my addr 172.21.15.67:0/270375568 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05000640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f000721a0 0x7f2f0014fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f06400640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f0015dbc0 0x7f2f00150430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 -- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f000721a0 msgr2=0x7f2f0014fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f000721a0 0x7f2f0014fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 -- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f0015dbc0 msgr2=0x7f2f00150430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f0015dbc0 0x7f2f00150430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05a00640 1 -- 172.21.15.67:0/270375568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f001536a0 con 0x7f2f0006db90 2024-04-17T10:34:06.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.649+0000 7f2f05000640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f000721a0 0x7f2f0014fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2f05a00640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00155190 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c7f0 tx=0x7f2ef400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2efd600640 1 -- 172.21.15.67:0/270375568 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4007a30 con 0x7f2f0006db90 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2efd600640 1 -- 172.21.15.67:0/270375568 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4007bd0 con 0x7f2f0006db90 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2efd600640 1 -- 172.21.15.67:0/270375568 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40113f0 con 0x7f2f0006db90 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f00153980 con 0x7f2f0006db90 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f00154460 con 0x7f2f0006db90 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0009e260 con 0x7f2f0006db90 2024-04-17T10:34:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2efd600640 1 -- 172.21.15.67:0/270375568 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2ef4011590 con 0x7f2f0006db90 2024-04-17T10:34:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.653+0000 7f2efd600640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ee0066f60 0x7f2ee0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.657+0000 7f2efd600640 1 -- 172.21.15.67:0/270375568 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f2ef4085370 con 0x7f2f0006db90 2024-04-17T10:34:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.657+0000 7f2f05000640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ee0066f60 0x7f2ee0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:06.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.657+0000 7f2f05000640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ee0066f60 0x7f2ee0069420 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0ad000 tx=0x7f2eec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.661+0000 7f2efd600640 1 -- 172.21.15.67:0/270375568 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2ef4015020 con 0x7f2f0006db90 2024-04-17T10:34:06.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.853+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f00074e30 con 0x7f2f0006db90 2024-04-17T10:34:06.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.853+0000 7f2efd600640 1 -- 172.21.15.67:0/270375568 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f2ef4056280 con 0x7f2f0006db90 2024-04-17T10:34:06.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:06.847 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:06.848 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:06.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ee0066f60 msgr2=0x7f2ee0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:06.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ee0066f60 0x7f2ee0069420 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0ad000 tx=0x7f2eec008040 comp rx=0 tx=0).stop 2024-04-17T10:34:06.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 msgr2=0x7f2f00155190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:06.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00155190 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c7f0 tx=0x7f2ef400ccb0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 shutdown_connections 2024-04-17T10:34:06.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ee0066f60 0x7f2ee0069420 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f0015dbc0 0x7f2f00150430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f000721a0 0x7f2f0014fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 --2- 172.21.15.67:0/270375568 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f0006db90 0x7f2f00155190 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:06.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 >> 172.21.15.67:0/270375568 conn(0x7f2f000af670 msgr2=0x7f2f000b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:06.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 shutdown_connections 2024-04-17T10:34:06.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:06.857+0000 7f2f08400640 1 -- 172.21.15.67:0/270375568 wait complete. 2024-04-17T10:34:06.869 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:34:06.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff028600640 1 Processor -- start 2024-04-17T10:34:07.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff028600640 1 -- start start 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff028600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff020145040 0x7ff02006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff028600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff020071a20 0x7ff0200cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff028600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02006e2f0 con 0x7ff02006ef10 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff028600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0200a86c0 con 0x7ff020071a20 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff028600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff02015bf10 con 0x7ff020145040 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff026600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff020071a20 0x7ff0200cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff025200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff02006ef10 0x7ff020071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34396/0 (socket says 172.21.15.67:34396) 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff025200640 1 -- 172.21.15.67:0/3715099471 learned_addr learned my addr 172.21.15.67:0/3715099471 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:07.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff025c00640 1 --2- 172.21.15.67:0/3715099471 >> v2:172.21.15.86:3300/0 conn(0x7ff020145040 0x7ff02006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff026600640 1 -- 172.21.15.67:0/3715099471 >> v2:172.21.15.86:3300/0 conn(0x7ff020145040 msgr2=0x7ff02006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff026600640 1 --2- 172.21.15.67:0/3715099471 >> v2:172.21.15.86:3300/0 conn(0x7ff020145040 0x7ff02006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff026600640 1 -- 172.21.15.67:0/3715099471 >> v1:172.21.15.67:6789/0 conn(0x7ff02006ef10 legacy=0x7ff020071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff026600640 1 -- 172.21.15.67:0/3715099471 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02006d6e0 con 0x7ff020071a20 2024-04-17T10:34:07.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.017+0000 7ff025c00640 1 --2- 172.21.15.67:0/3715099471 >> v2:172.21.15.86:3300/0 conn(0x7ff020145040 0x7ff02006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff026600640 1 --2- 172.21.15.67:0/3715099471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff0200cc300 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7ff020072410 tx=0x7ff01801f110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=333a3135960151cf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff01fe00640 1 -- 172.21.15.67:0/3715099471 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0180b4070 con 0x7ff020071a20 2024-04-17T10:34:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff01fe00640 1 -- 172.21.15.67:0/3715099471 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0180af600 con 0x7ff020071a20 2024-04-17T10:34:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff01fe00640 1 -- 172.21.15.67:0/3715099471 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0180af9d0 con 0x7ff020071a20 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 -- 172.21.15.67:0/3715099471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 msgr2=0x7ff0200cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 --2- 172.21.15.67:0/3715099471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff0200cc300 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7ff020072410 tx=0x7ff01801f110 comp rx=0 tx=0).stop 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 -- 172.21.15.67:0/3715099471 shutdown_connections 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 --2- 172.21.15.67:0/3715099471 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff0200cc300 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 --2- 172.21.15.67:0/3715099471 >> v2:172.21.15.86:3300/0 conn(0x7ff020145040 0x7ff02006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 -- 172.21.15.67:0/3715099471 >> 172.21.15.67:0/3715099471 conn(0x7ff02006ac90 msgr2=0x7ff02006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 -- 172.21.15.67:0/3715099471 shutdown_connections 2024-04-17T10:34:07.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.021+0000 7ff028600640 1 -- 172.21.15.67:0/3715099471 wait complete. 2024-04-17T10:34:07.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 Processor -- start 2024-04-17T10:34:07.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 -- start start 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff02006ef10 0x7ff02015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff02015e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff020145040 0x7ff02013d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02006df80 con 0x7ff020071a20 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff02006dce0 con 0x7ff02006ef10 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02006d370 con 0x7ff020145040 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff02015e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff02015e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40906/0 (socket says 172.21.15.67:40906) 2024-04-17T10:34:07.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 -- 172.21.15.67:0/1035764112 learned_addr learned my addr 172.21.15.67:0/1035764112 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025c00640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff02006ef10 0x7ff02015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 -- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff020145040 msgr2=0x7ff02013d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff026600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff020145040 0x7ff02013d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff020145040 0x7ff02013d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 -- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff02006ef10 msgr2=0x7ff02015dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff02006ef10 0x7ff02015dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 -- 172.21.15.67:0/1035764112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02015ec00 con 0x7ff020071a20 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025c00640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff02006ef10 0x7ff02015dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff026600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff020145040 0x7ff02013d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff025200640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff02015e0e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c450 tx=0x7ff00c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff01d600640 1 -- 172.21.15.67:0/1035764112 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c017070 con 0x7ff020071a20 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff01d600640 1 -- 172.21.15.67:0/1035764112 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c005cf0 con 0x7ff020071a20 2024-04-17T10:34:07.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff01d600640 1 -- 172.21.15.67:0/1035764112 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c007290 con 0x7ff020071a20 2024-04-17T10:34:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff02015e920 con 0x7ff020071a20 2024-04-17T10:34:07.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.025+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff020138d90 con 0x7ff020071a20 2024-04-17T10:34:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.029+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff020072d50 con 0x7ff020071a20 2024-04-17T10:34:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.029+0000 7ff01d600640 1 -- 172.21.15.67:0/1035764112 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff00c01a070 con 0x7ff020071a20 2024-04-17T10:34:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.029+0000 7ff01d600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff000066f90 0x7ff000069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.029+0000 7ff01d600640 1 -- 172.21.15.67:0/1035764112 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff00c0844c0 con 0x7ff020071a20 2024-04-17T10:34:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.029+0000 7ff025c00640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff000066f90 0x7ff000069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:07.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.029+0000 7ff025c00640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff000066f90 0x7ff000069450 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff014008890 tx=0x7ff014008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:07.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.033+0000 7ff01d600640 1 -- 172.21.15.67:0/1035764112 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff00c055340 con 0x7ff020071a20 2024-04-17T10:34:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.249+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7ff0200b40c0 con 0x7ff020071a20 2024-04-17T10:34:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.249+0000 7ff01d600640 1 -- 172.21.15.67:0/1035764112 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7ff00c054c40 con 0x7ff020071a20 2024-04-17T10:34:07.243 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:07.243 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":578,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076583881,"bytes_used":2598060032,"bytes_avail":770496053248,"bytes_total":773094113280,"write_bytes_sec":19366343,"read_op_per_sec":0,"write_op_per_sec":19},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":22,"modified":"2024-04-17T10:34:02.266878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff000066f90 msgr2=0x7ff000069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff000066f90 0x7ff000069450 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7ff014008890 tx=0x7ff014008040 comp rx=0 tx=0).stop 2024-04-17T10:34:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 msgr2=0x7ff02015e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff02015e0e0 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00c450 tx=0x7ff00c00c910 comp rx=0 tx=0).stop 2024-04-17T10:34:07.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 shutdown_connections 2024-04-17T10:34:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff000066f90 0x7ff000069450 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff020145040 0x7ff02013d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff020071a20 0x7ff02015e0e0 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 --2- 172.21.15.67:0/1035764112 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff02006ef10 0x7ff02015dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 >> 172.21.15.67:0/1035764112 conn(0x7ff02006ac90 msgr2=0x7ff02013f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 shutdown_connections 2024-04-17T10:34:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:07.253+0000 7ff028600640 1 -- 172.21.15.67:0/1035764112 wait complete. 2024-04-17T10:34:07.266 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:34:08.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda3e200640 1 Processor -- start 2024-04-17T10:34:08.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda3e200640 1 -- start start 2024-04-17T10:34:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda3e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda3815dd10 0x7fda380b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda3e200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fda380b57b0 0x7fda38070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda3e200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda380b0cc0 con 0x7fda380b07b0 2024-04-17T10:34:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda3e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda380b1280 con 0x7fda3815dd10 2024-04-17T10:34:08.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda3e200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda380b0fa0 con 0x7fda380b57b0 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda37400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda3815dd10 0x7fda380b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda37400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda3815dd10 0x7fda380b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40908/0 (socket says 172.21.15.67:40908) 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda37400640 1 -- 172.21.15.67:0/1429310404 learned_addr learned my addr 172.21.15.67:0/1429310404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda36a00640 1 --2- 172.21.15.67:0/1429310404 >> v2:172.21.15.86:3300/0 conn(0x7fda380b57b0 0x7fda38070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda37400640 1 -- 172.21.15.67:0/1429310404 >> v2:172.21.15.86:3300/0 conn(0x7fda380b57b0 msgr2=0x7fda38070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda37400640 1 --2- 172.21.15.67:0/1429310404 >> v2:172.21.15.86:3300/0 conn(0x7fda380b57b0 0x7fda38070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda37400640 1 -- 172.21.15.67:0/1429310404 >> v1:172.21.15.86:6789/0 conn(0x7fda380b07b0 legacy=0x7fda381618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:08.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.429+0000 7fda37400640 1 -- 172.21.15.67:0/1429310404 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3806bed0 con 0x7fda3815dd10 2024-04-17T10:34:08.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda37400640 1 --2- 172.21.15.67:0/1429310404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380b5270 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fda38068850 tx=0x7fda2801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d310c2f6169fa619 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:08.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda36000640 1 -- 172.21.15.67:0/1429310404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280ab600 con 0x7fda3815dd10 2024-04-17T10:34:08.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda36000640 1 -- 172.21.15.67:0/1429310404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda280b0070 con 0x7fda3815dd10 2024-04-17T10:34:08.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda36000640 1 -- 172.21.15.67:0/1429310404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280ab9c0 con 0x7fda3815dd10 2024-04-17T10:34:08.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 -- 172.21.15.67:0/1429310404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 msgr2=0x7fda380b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 --2- 172.21.15.67:0/1429310404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380b5270 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7fda38068850 tx=0x7fda2801c420 comp rx=0 tx=0).stop 2024-04-17T10:34:08.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 -- 172.21.15.67:0/1429310404 shutdown_connections 2024-04-17T10:34:08.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 --2- 172.21.15.67:0/1429310404 >> v2:172.21.15.86:3300/0 conn(0x7fda380b57b0 0x7fda38070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 --2- 172.21.15.67:0/1429310404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380b5270 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 -- 172.21.15.67:0/1429310404 >> 172.21.15.67:0/1429310404 conn(0x7fda380b34d0 msgr2=0x7fda380b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:08.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 -- 172.21.15.67:0/1429310404 shutdown_connections 2024-04-17T10:34:08.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 -- 172.21.15.67:0/1429310404 wait complete. 2024-04-17T10:34:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.433+0000 7fda3e200640 1 Processor -- start 2024-04-17T10:34:08.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 -- start start 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda380b07b0 0x7fda380cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda380ccfc0 0x7fda381489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3816ae70 con 0x7fda3815dd10 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda38071710 con 0x7fda380b07b0 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3816aff0 con 0x7fda380ccfc0 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380cca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40918/0 (socket says 172.21.15.67:40918) 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 -- 172.21.15.67:0/89970661 learned_addr learned my addr 172.21.15.67:0/89970661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda37400640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda380b07b0 0x7fda380cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda37e00640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda380ccfc0 0x7fda381489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 -- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda380ccfc0 msgr2=0x7fda381489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda380ccfc0 0x7fda381489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 -- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda380b07b0 msgr2=0x7fda380cab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda380b07b0 0x7fda380cab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 -- 172.21.15.67:0/89970661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda380cb130 con 0x7fda3815dd10 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda37e00640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda380ccfc0 0x7fda381489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda36a00640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380cca80 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fda2400d830 tx=0x7fda2400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:08.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda2f400640 1 -- 172.21.15.67:0/89970661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2400b6c0 con 0x7fda3815dd10 2024-04-17T10:34:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda2f400640 1 -- 172.21.15.67:0/89970661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda24019070 con 0x7fda3815dd10 2024-04-17T10:34:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda2f400640 1 -- 172.21.15.67:0/89970661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2400ba60 con 0x7fda3815dd10 2024-04-17T10:34:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda380cd520 con 0x7fda3815dd10 2024-04-17T10:34:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda38148f20 con 0x7fda3815dd10 2024-04-17T10:34:08.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.437+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda38070bb0 con 0x7fda3815dd10 2024-04-17T10:34:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.441+0000 7fda2f400640 1 -- 172.21.15.67:0/89970661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fda2401e460 con 0x7fda3815dd10 2024-04-17T10:34:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.441+0000 7fda2f400640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda18066f90 0x7fda18069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.441+0000 7fda2f400640 1 -- 172.21.15.67:0/89970661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fda240857f0 con 0x7fda3815dd10 2024-04-17T10:34:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.441+0000 7fda37400640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda18066f90 0x7fda18069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.441+0000 7fda37400640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda18066f90 0x7fda18069450 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fda280af000 tx=0x7fda280a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:08.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.445+0000 7fda2f400640 1 -- 172.21.15.67:0/89970661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fda240565d0 con 0x7fda3815dd10 2024-04-17T10:34:08.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.637+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda38074e30 con 0x7fda3815dd10 2024-04-17T10:34:08.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda2f400640 1 -- 172.21.15.67:0/89970661 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fda24055ed0 con 0x7fda3815dd10 2024-04-17T10:34:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:08.632 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:08.632 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda18066f90 msgr2=0x7fda18069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda18066f90 0x7fda18069450 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fda280af000 tx=0x7fda280a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 msgr2=0x7fda380cca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380cca80 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7fda2400d830 tx=0x7fda2400dcf0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 shutdown_connections 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda18066f90 0x7fda18069450 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fda380ccfc0 0x7fda381489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda3815dd10 0x7fda380cca80 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 --2- 172.21.15.67:0/89970661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fda380b07b0 0x7fda380cab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 >> 172.21.15.67:0/89970661 conn(0x7fda380b34d0 msgr2=0x7fda380d16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 shutdown_connections 2024-04-17T10:34:08.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.641+0000 7fda3e200640 1 -- 172.21.15.67:0/89970661 wait complete. 2024-04-17T10:34:08.653 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-04-17T10:34:08.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:08.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd80800640 1 Processor -- start 2024-04-17T10:34:08.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd80800640 1 -- start start 2024-04-17T10:34:08.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd80800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd7806b6a0 0x7fbd780cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd80800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd780cf880 0x7fbd780cfd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd80800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd780cc8b0 con 0x7fbd781597a0 2024-04-17T10:34:08.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd80800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd780cd4f0 con 0x7fbd7806b6a0 2024-04-17T10:34:08.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd80800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd780ccb90 con 0x7fbd780cf880 2024-04-17T10:34:08.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7d400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd7806b6a0 0x7fbd780cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd780cf880 0x7fbd780cfd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7de00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbd781597a0 0x7fbd780cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:40188/0 (socket says 172.21.15.67:40188) 2024-04-17T10:34:08.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7de00640 1 -- 172.21.15.67:0/666127287 learned_addr learned my addr 172.21.15.67:0/666127287 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7d400640 1 -- 172.21.15.67:0/666127287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd780cf880 msgr2=0x7fbd780cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7d400640 1 --2- 172.21.15.67:0/666127287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd780cf880 0x7fbd780cfd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7d400640 1 -- 172.21.15.67:0/666127287 >> v1:172.21.15.86:6789/0 conn(0x7fbd781597a0 legacy=0x7fbd780cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.801+0000 7fbd7d400640 1 -- 172.21.15.67:0/666127287 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd780cd180 con 0x7fbd7806b6a0 2024-04-17T10:34:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd7d400640 1 --2- 172.21.15.67:0/666127287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 0x7fbd780cc260 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbd680097c0 tx=0x7fbd6801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c573b759ca492765 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd7ca00640 1 -- 172.21.15.67:0/666127287 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd680b1070 con 0x7fbd7806b6a0 2024-04-17T10:34:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd7ca00640 1 -- 172.21.15.67:0/666127287 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd680ac410 con 0x7fbd7806b6a0 2024-04-17T10:34:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd7ca00640 1 -- 172.21.15.67:0/666127287 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd680ac7e0 con 0x7fbd7806b6a0 2024-04-17T10:34:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 -- 172.21.15.67:0/666127287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 msgr2=0x7fbd780cc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 --2- 172.21.15.67:0/666127287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 0x7fbd780cc260 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fbd680097c0 tx=0x7fbd6801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 -- 172.21.15.67:0/666127287 shutdown_connections 2024-04-17T10:34:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 --2- 172.21.15.67:0/666127287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd780cf880 0x7fbd780cfd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 --2- 172.21.15.67:0/666127287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 0x7fbd780cc260 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 -- 172.21.15.67:0/666127287 >> 172.21.15.67:0/666127287 conn(0x7fbd780b22a0 msgr2=0x7fbd780b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 -- 172.21.15.67:0/666127287 shutdown_connections 2024-04-17T10:34:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.805+0000 7fbd80800640 1 -- 172.21.15.67:0/666127287 wait complete. 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 Processor -- start 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 -- start start 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 0x7fbd78157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd780cf880 0x7fbd780c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd781597a0 0x7fbd78158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd781685c0 con 0x7fbd780cf880 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7815ca50 con 0x7fbd7806b6a0 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd78168740 con 0x7fbd781597a0 2024-04-17T10:34:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd780cf880 0x7fbd780c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd780cf880 0x7fbd780c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40930/0 (socket says 172.21.15.67:40930) 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 -- 172.21.15.67:0/3732463715 learned_addr learned my addr 172.21.15.67:0/3732463715 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7e800640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd781597a0 0x7fbd78158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7de00640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 0x7fbd78157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 -- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd781597a0 msgr2=0x7fbd78158910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd781597a0 0x7fbd78158910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 -- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 msgr2=0x7fbd78157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 0x7fbd78157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 -- 172.21.15.67:0/3732463715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7814eee0 con 0x7fbd780cf880 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd7d400640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd780cf880 0x7fbd780c8a10 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fbd680af040 tx=0x7fbd68019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd72000640 1 -- 172.21.15.67:0/3732463715 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd680b1040 con 0x7fbd780cf880 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd72000640 1 -- 172.21.15.67:0/3732463715 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd680bb070 con 0x7fbd780cf880 2024-04-17T10:34:08.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd78157870 con 0x7fbd780cf880 2024-04-17T10:34:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd72000640 1 -- 172.21.15.67:0/3732463715 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd68006e20 con 0x7fbd780cf880 2024-04-17T10:34:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.809+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd78157e30 con 0x7fbd780cf880 2024-04-17T10:34:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.813+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd7809e900 con 0x7fbd780cf880 2024-04-17T10:34:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.813+0000 7fbd72000640 1 -- 172.21.15.67:0/3732463715 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbd680c0460 con 0x7fbd780cf880 2024-04-17T10:34:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.813+0000 7fbd72000640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd5c066f60 0x7fbd5c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.813+0000 7fbd72000640 1 -- 172.21.15.67:0/3732463715 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbd6811ed40 con 0x7fbd780cf880 2024-04-17T10:34:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.813+0000 7fbd7de00640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd5c066f60 0x7fbd5c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.813+0000 7fbd7de00640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd5c066f60 0x7fbd5c069420 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbd780c3bf0 tx=0x7fbd6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:08.817+0000 7fbd72000640 1 -- 172.21.15.67:0/3732463715 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbd7814eee0 con 0x7fbd780cf880 2024-04-17T10:34:09.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.033+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbd780c7170 con 0x7fbd780cf880 2024-04-17T10:34:09.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.033+0000 7fbd72000640 1 -- 172.21.15.67:0/3732463715 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1575 (secure 0 0 0) 0x7fbd68016090 con 0x7fbd780cf880 2024-04-17T10:34:09.027 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:09.027 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":579,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":118,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076588106,"bytes_used":2598068224,"bytes_avail":770496045056,"bytes_total":773094113280,"write_bytes_sec":19367034,"read_op_per_sec":0,"write_op_per_sec":20},"fsmap":{"epoch":82,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":8670}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-04-17T10:34:08.268434+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd5c066f60 msgr2=0x7fbd5c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd5c066f60 0x7fbd5c069420 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fbd780c3bf0 tx=0x7fbd6c009040 comp rx=0 tx=0).stop 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd780cf880 msgr2=0x7fbd780c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd780cf880 0x7fbd780c8a10 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fbd680af040 tx=0x7fbd68019240 comp rx=0 tx=0).stop 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 shutdown_connections 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd5c066f60 0x7fbd5c069420 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd781597a0 0x7fbd78158910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd780cf880 0x7fbd780c8a10 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 --2- 172.21.15.67:0/3732463715 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd7806b6a0 0x7fbd78157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 >> 172.21.15.67:0/3732463715 conn(0x7fbd780b22a0 msgr2=0x7fbd7815b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:09.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 shutdown_connections 2024-04-17T10:34:09.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.037+0000 7fbd80800640 1 -- 172.21.15.67:0/3732463715 wait complete. 2024-04-17T10:34:09.048 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:34:09.049 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:34:09.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:34:09.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:34:09.250 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:34:09.251 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:34:09.251 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:09.260 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:34:09.260 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:09.313 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:09.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8817200640 1 Processor -- start 2024-04-17T10:34:09.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8817200640 1 -- start start 2024-04-17T10:34:09.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8817200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f881006b6a0 0x7f88100cc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8817200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100cc8b0 con 0x7f88101597a0 2024-04-17T10:34:09.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8817200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100ccb90 con 0x7f88100cf880 2024-04-17T10:34:09.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8817200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88100cd4f0 con 0x7f881006b6a0 2024-04-17T10:34:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f88101597a0 0x7f88100cd980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34402/0 (socket says 172.21.15.67:34402) 2024-04-17T10:34:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880fe00640 1 -- 172.21.15.67:0/623833741 learned_addr learned my addr 172.21.15.67:0/623833741 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328496045 0 0) 0x7f88100cc8b0 con 0x7f88101597a0 2024-04-17T10:34:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8807e00640 1 --2- 172.21.15.67:0/623833741 >> v2:172.21.15.86:3300/0 conn(0x7f881006b6a0 0x7f88100cc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f4003530 con 0x7f88101597a0 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4064429891 0 0) 0x7f87f4003530 con 0x7f88101597a0 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 >> v2:172.21.15.86:3300/0 conn(0x7f881006b6a0 msgr2=0x7f88100cc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 --2- 172.21.15.67:0/623833741 >> v2:172.21.15.86:3300/0 conn(0x7f881006b6a0 0x7f88100cc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 >> v1:172.21.15.86:6789/0 conn(0x7f88100cf880 legacy=0x7f881015bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100cd180 con 0x7f88101597a0 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f8800001680 con 0x7f88101597a0 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 >> v1:172.21.15.67:6789/0 conn(0x7f88101597a0 legacy=0x7f88100cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87f40039c0 0x7f87f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f4004810 0x7f87f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87f40051d0 0x7f87f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100cd4f0 con 0x7f87f40051d0 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f400c070 con 0x7f87f4004810 2024-04-17T10:34:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f400c270 con 0x7f87f40039c0 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8815200640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87f40051d0 0x7f87f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f880fe00640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87f40039c0 0x7f87f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.505+0000 7f8807e00640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f4004810 0x7f87f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8815200640 1 -- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87f40039c0 msgr2=0x7f87f4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8815200640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f87f40039c0 0x7f87f4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8815200640 1 -- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f4004810 msgr2=0x7f87f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8815200640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f4004810 0x7f87f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8815200640 1 -- 172.21.15.67:0/623833741 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4003530 con 0x7f87f40051d0 2024-04-17T10:34:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8815200640 1 -- 172.21.15.67:0/623833741 reap_dead start 2024-04-17T10:34:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8815200640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87f40051d0 0x7f87f400b9d0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f8808001e70 tx=0x7f880801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f56651c24b8ecaa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808001a90 con 0x7f87f40051d0 2024-04-17T10:34:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8808003bc0 con 0x7f87f40051d0 2024-04-17T10:34:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f880f400640 1 -- 172.21.15.67:0/623833741 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8808005020 con 0x7f87f40051d0 2024-04-17T10:34:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8817200640 1 -- 172.21.15.67:0/623833741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87f40051d0 msgr2=0x7f87f400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8817200640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87f40051d0 0x7f87f400b9d0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f8808001e70 tx=0x7f880801c780 comp rx=0 tx=0).stop 2024-04-17T10:34:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8817200640 1 -- 172.21.15.67:0/623833741 shutdown_connections 2024-04-17T10:34:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8817200640 1 --2- 172.21.15.67:0/623833741 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87f40051d0 0x7f87f400b9d0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8817200640 1 -- 172.21.15.67:0/623833741 >> 172.21.15.67:0/623833741 conn(0x7f88100b22a0 msgr2=0x7f88100b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8817200640 1 -- 172.21.15.67:0/623833741 shutdown_connections 2024-04-17T10:34:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.509+0000 7f8817200640 1 -- 172.21.15.67:0/623833741 wait complete. 2024-04-17T10:34:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 Processor -- start 2024-04-17T10:34:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 -- start start 2024-04-17T10:34:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f40051d0 0x7f8810157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881015bda0 0x7f8810159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88100aeec0 0x7f88100af350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100b3020 con 0x7f881015bda0 2024-04-17T10:34:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88100b2ea0 con 0x7f87f40051d0 2024-04-17T10:34:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88100cc260 con 0x7f88100aeec0 2024-04-17T10:34:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881015bda0 0x7f8810159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881015bda0 0x7f8810159b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40954/0 (socket says 172.21.15.67:40954) 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 -- 172.21.15.67:0/2906033430 learned_addr learned my addr 172.21.15.67:0/2906033430 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f880fe00640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f40051d0 0x7f8810157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8815200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88100aeec0 0x7f88100af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 -- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88100aeec0 msgr2=0x7f88100af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88100aeec0 0x7f88100af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 -- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f40051d0 msgr2=0x7f8810157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f40051d0 0x7f8810157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 -- 172.21.15.67:0/2906033430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881015a480 con 0x7f881015bda0 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8815200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88100aeec0 0x7f88100af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8807e00640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881015bda0 0x7f8810159b80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f87f8011960 tx=0x7f87f8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f880cc00640 1 -- 172.21.15.67:0/2906033430 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f800faf0 con 0x7f881015bda0 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f880cc00640 1 -- 172.21.15.67:0/2906033430 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f800fc90 con 0x7f881015bda0 2024-04-17T10:34:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881015a760 con 0x7f881015bda0 2024-04-17T10:34:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f880cc00640 1 -- 172.21.15.67:0/2906033430 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f8014680 con 0x7f881015bda0 2024-04-17T10:34:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.513+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881015ad20 con 0x7f881015bda0 2024-04-17T10:34:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.517+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881009e900 con 0x7f881015bda0 2024-04-17T10:34:09.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.517+0000 7f880cc00640 1 -- 172.21.15.67:0/2906033430 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f87f8012040 con 0x7f881015bda0 2024-04-17T10:34:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.517+0000 7f880cc00640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f87ec066f90 0x7f87ec069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.517+0000 7f880cc00640 1 -- 172.21.15.67:0/2906033430 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f87f80856f0 con 0x7f881015bda0 2024-04-17T10:34:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.517+0000 7f880fe00640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f87ec066f90 0x7f87ec069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.517+0000 7f880fe00640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f87ec066f90 0x7f87ec069450 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f88100afa80 tx=0x7f8800001c70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.521+0000 7f880cc00640 1 -- 172.21.15.67:0/2906033430 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f87f8019030 con 0x7f881015bda0 2024-04-17T10:34:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.689+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88100ced50 con 0x7f881015bda0 2024-04-17T10:34:09.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f880cc00640 1 -- 172.21.15.67:0/2906033430 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16904 (secure 0 0 0) 0x7f87f8056560 con 0x7f881015bda0 2024-04-17T10:34:09.684 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:09.684 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":118,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:31.509307+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:33:28.333663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-17T10:33:28.871385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f87ec066f90 msgr2=0x7f87ec069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f87ec066f90 0x7f87ec069450 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f88100afa80 tx=0x7f8800001c70 comp rx=0 tx=0).stop 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881015bda0 msgr2=0x7f8810159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881015bda0 0x7f8810159b80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f87f8011960 tx=0x7f87f8011e20 comp rx=0 tx=0).stop 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 shutdown_connections 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f87ec066f90 0x7f87ec069450 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f88100aeec0 0x7f88100af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f881015bda0 0x7f8810159b80 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 --2- 172.21.15.67:0/2906033430 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f87f40051d0 0x7f8810157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 >> 172.21.15.67:0/2906033430 conn(0x7f88100b22a0 msgr2=0x7f88100b1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 shutdown_connections 2024-04-17T10:34:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.693+0000 7f8817200640 1 -- 172.21.15.67:0/2906033430 wait complete. 2024-04-17T10:34:09.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:33:28.333663+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:33:28.871385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:34:09.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:34:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.857+0000 7f7d4d400640 1 Processor -- start 2024-04-17T10:34:09.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.857+0000 7f7d4d400640 1 -- start start 2024-04-17T10:34:09.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d4d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d481583f0 0x7f7d48072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d4d400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d480b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d4d400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4815f360 con 0x7f7d480b5130 2024-04-17T10:34:09.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d4d400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4806bca0 con 0x7f7d481583f0 2024-04-17T10:34:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d4d400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480b3160 con 0x7f7d480728c0 2024-04-17T10:34:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d46800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7d481583f0 0x7f7d48072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d47200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7d480b5130 0x7f7d48161980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34410/0 (socket says 172.21.15.67:34410) 2024-04-17T10:34:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d47200640 1 -- 172.21.15.67:0/1908531796 learned_addr learned my addr 172.21.15.67:0/1908531796 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d3fe00640 1 --2- 172.21.15.67:0/1908531796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d480b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d46800640 1 -- 172.21.15.67:0/1908531796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 msgr2=0x7f7d480b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:09.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d46800640 1 --2- 172.21.15.67:0/1908531796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d480b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d46800640 1 -- 172.21.15.67:0/1908531796 >> v1:172.21.15.67:6789/0 conn(0x7f7d480b5130 legacy=0x7f7d48161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d46800640 1 -- 172.21.15.67:0/1908531796 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d480b3440 con 0x7f7d481583f0 2024-04-17T10:34:09.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.861+0000 7f7d3fe00640 1 --2- 172.21.15.67:0/1908531796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d480b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:09.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d46800640 1 --2- 172.21.15.67:0/1908531796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d481583f0 0x7f7d48072380 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f7d380097c0 tx=0x7f7d3801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe7ecd1073b2272 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:09.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d45e00640 1 -- 172.21.15.67:0/1908531796 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d380b0070 con 0x7f7d481583f0 2024-04-17T10:34:09.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d45e00640 1 -- 172.21.15.67:0/1908531796 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d380ab600 con 0x7f7d481583f0 2024-04-17T10:34:09.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d45e00640 1 -- 172.21.15.67:0/1908531796 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d380ab9d0 con 0x7f7d481583f0 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 -- 172.21.15.67:0/1908531796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d481583f0 msgr2=0x7f7d48072380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 --2- 172.21.15.67:0/1908531796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d481583f0 0x7f7d48072380 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f7d380097c0 tx=0x7f7d3801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 -- 172.21.15.67:0/1908531796 shutdown_connections 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 --2- 172.21.15.67:0/1908531796 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d480b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 --2- 172.21.15.67:0/1908531796 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d481583f0 0x7f7d48072380 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 -- 172.21.15.67:0/1908531796 >> 172.21.15.67:0/1908531796 conn(0x7f7d480b2570 msgr2=0x7f7d48071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 -- 172.21.15.67:0/1908531796 shutdown_connections 2024-04-17T10:34:09.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 -- 172.21.15.67:0/1908531796 wait complete. 2024-04-17T10:34:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 Processor -- start 2024-04-17T10:34:09.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.865+0000 7f7d4d400640 1 -- start start 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d481512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d480b5130 0x7f7d48151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d481583f0 0x7f7d481f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480b4200 con 0x7f7d480b5130 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d481587d0 con 0x7f7d481583f0 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480b5540 con 0x7f7d480728c0 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d480b5130 0x7f7d48151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d480b5130 0x7f7d48151900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40970/0 (socket says 172.21.15.67:40970) 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 -- 172.21.15.67:0/123632616 learned_addr learned my addr 172.21.15.67:0/123632616 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d46800640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d481512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d47200640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d481583f0 0x7f7d481f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 -- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 msgr2=0x7f7d481512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d481512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 -- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d481583f0 msgr2=0x7f7d481f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d481583f0 0x7f7d481f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 -- 172.21.15.67:0/123632616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d481557d0 con 0x7f7d480b5130 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d47200640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d481583f0 0x7f7d481f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3fe00640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d480b5130 0x7f7d48151900 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000c7c0 tx=0x7f7d3000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3ea00640 1 -- 172.21.15.67:0/123632616 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30007a30 con 0x7f7d480b5130 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3ea00640 1 -- 172.21.15.67:0/123632616 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30007bd0 con 0x7f7d480b5130 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3ea00640 1 -- 172.21.15.67:0/123632616 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d300113f0 con 0x7f7d480b5130 2024-04-17T10:34:09.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d48155ab0 con 0x7f7d480b5130 2024-04-17T10:34:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d48156070 con 0x7f7d480b5130 2024-04-17T10:34:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4804ef30 con 0x7f7d480b5130 2024-04-17T10:34:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.869+0000 7f7d3ea00640 1 -- 172.21.15.67:0/123632616 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7d30011590 con 0x7f7d480b5130 2024-04-17T10:34:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.873+0000 7f7d3ea00640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d24066f60 0x7f7d24069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.873+0000 7f7d3ea00640 1 -- 172.21.15.67:0/123632616 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7d300853f0 con 0x7f7d480b5130 2024-04-17T10:34:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.873+0000 7f7d46800640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d24066f60 0x7f7d24069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.873+0000 7f7d46800640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d24066f60 0x7f7d24069420 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7d380ad000 tx=0x7f7d380a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:09.877+0000 7f7d3ea00640 1 -- 172.21.15.67:0/123632616 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7d30015020 con 0x7f7d480b5130 2024-04-17T10:34:10.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d480ae1d0 con 0x7f7d480b5130 2024-04-17T10:34:10.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d3ea00640 1 -- 172.21.15.67:0/123632616 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d30055b80 con 0x7f7d480b5130 2024-04-17T10:34:10.028 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:34:10.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d24066f60 msgr2=0x7f7d24069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d24066f60 0x7f7d24069420 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7d380ad000 tx=0x7f7d380a6000 comp rx=0 tx=0).stop 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d480b5130 msgr2=0x7f7d48151900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d480b5130 0x7f7d48151900 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000c7c0 tx=0x7f7d3000cc80 comp rx=0 tx=0).stop 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 shutdown_connections 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7d24066f60 0x7f7d24069420 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7d481583f0 0x7f7d481f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7d480b5130 0x7f7d48151900 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 --2- 172.21.15.67:0/123632616 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7d480728c0 0x7f7d481512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 >> 172.21.15.67:0/123632616 conn(0x7f7d480b2570 msgr2=0x7f7d480b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 shutdown_connections 2024-04-17T10:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.037+0000 7f7d4d400640 1 -- 172.21.15.67:0/123632616 wait complete. 2024-04-17T10:34:10.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47cae00640 1 Processor -- start 2024-04-17T10:34:10.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47cae00640 1 -- start start 2024-04-17T10:34:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47cae00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f47c4071af0 0x7f47c40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47cae00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f47c415be70 0x7f47c415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47cae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c415cb30 con 0x7f47c4159440 2024-04-17T10:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47cae00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c40a8650 con 0x7f47c4071af0 2024-04-17T10:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47cae00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c415ce10 con 0x7f47c415be70 2024-04-17T10:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47c3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f47c4159440 0x7f47c40713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:34418/0 (socket says 172.21.15.67:34418) 2024-04-17T10:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47c3e00640 1 -- 172.21.15.67:0/915052409 learned_addr learned my addr 172.21.15.67:0/915052409 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:10.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.201+0000 7f47bbe00640 1 --2- 172.21.15.67:0/915052409 >> v2:172.21.15.67:3300/0 conn(0x7f47c4071af0 0x7f47c40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47c8e00640 1 --2- 172.21.15.67:0/915052409 >> v2:172.21.15.86:3300/0 conn(0x7f47c415be70 0x7f47c415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47bbe00640 1 -- 172.21.15.67:0/915052409 >> v2:172.21.15.86:3300/0 conn(0x7f47c415be70 msgr2=0x7f47c415c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47bbe00640 1 --2- 172.21.15.67:0/915052409 >> v2:172.21.15.86:3300/0 conn(0x7f47c415be70 0x7f47c415c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47bbe00640 1 -- 172.21.15.67:0/915052409 >> v1:172.21.15.67:6789/0 conn(0x7f47c4159440 legacy=0x7f47c40713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47bbe00640 1 -- 172.21.15.67:0/915052409 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c40cd610 con 0x7f47c4071af0 2024-04-17T10:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47c3400640 1 -- 172.21.15.67:0/915052409 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215609152 0 0) 0x7f47c415cb30 con 0x7f47c4159440 2024-04-17T10:34:10.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47c8e00640 1 --2- 172.21.15.67:0/915052409 >> v2:172.21.15.86:3300/0 conn(0x7f47c415be70 0x7f47c415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47bbe00640 1 --2- 172.21.15.67:0/915052409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c4071af0 0x7f47c40ce0f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f47c40ce690 tx=0x7f47ac01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef23eabc0183d5c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47c3400640 1 -- 172.21.15.67:0/915052409 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0ab680 con 0x7f47c4071af0 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47c3400640 1 -- 172.21.15.67:0/915052409 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac0b0070 con 0x7f47c4071af0 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47c3400640 1 -- 172.21.15.67:0/915052409 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0aba40 con 0x7f47c4071af0 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 -- 172.21.15.67:0/915052409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c4071af0 msgr2=0x7f47c40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 --2- 172.21.15.67:0/915052409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c4071af0 0x7f47c40ce0f0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f47c40ce690 tx=0x7f47ac01c480 comp rx=0 tx=0).stop 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 -- 172.21.15.67:0/915052409 shutdown_connections 2024-04-17T10:34:10.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 --2- 172.21.15.67:0/915052409 >> v2:172.21.15.86:3300/0 conn(0x7f47c415be70 0x7f47c415c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 --2- 172.21.15.67:0/915052409 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c4071af0 0x7f47c40ce0f0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 -- 172.21.15.67:0/915052409 >> 172.21.15.67:0/915052409 conn(0x7f47c40b1740 msgr2=0x7f47c40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 -- 172.21.15.67:0/915052409 shutdown_connections 2024-04-17T10:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.205+0000 7f47cae00640 1 -- 172.21.15.67:0/915052409 wait complete. 2024-04-17T10:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 Processor -- start 2024-04-17T10:34:10.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 -- start start 2024-04-17T10:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4071af0 0x7f47c40b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4159440 0x7f47c4152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c415be70 0x7f47c414dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c415d840 con 0x7f47c415be70 2024-04-17T10:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c41598e0 con 0x7f47c4159440 2024-04-17T10:34:10.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47cae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c415d9c0 con 0x7f47c4071af0 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c415be70 0x7f47c414dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c415be70 0x7f47c414dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40980/0 (socket says 172.21.15.67:40980) 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 -- 172.21.15.67:0/1227370764 learned_addr learned my addr 172.21.15.67:0/1227370764 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c3e00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4071af0 0x7f47c40b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47bbe00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4159440 0x7f47c4152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 -- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4071af0 msgr2=0x7f47c40b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4071af0 0x7f47c40b3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 -- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4159440 msgr2=0x7f47c4152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4159440 0x7f47c4152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 -- 172.21.15.67:0/1227370764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c4152dd0 con 0x7f47c415be70 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c8e00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c415be70 0x7f47c414dfc0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00cd50 tx=0x7f47bc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c0c00640 1 -- 172.21.15.67:0/1227370764 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc00f680 con 0x7f47c415be70 2024-04-17T10:34:10.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c0c00640 1 -- 172.21.15.67:0/1227370764 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc00f840 con 0x7f47c415be70 2024-04-17T10:34:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.209+0000 7f47c0c00640 1 -- 172.21.15.67:0/1227370764 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc00fc30 con 0x7f47c415be70 2024-04-17T10:34:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c4152110 con 0x7f47c415be70 2024-04-17T10:34:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c40b3660 con 0x7f47c415be70 2024-04-17T10:34:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c404ef30 con 0x7f47c415be70 2024-04-17T10:34:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47c0c00640 1 -- 172.21.15.67:0/1227370764 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f47bc00e3e0 con 0x7f47c415be70 2024-04-17T10:34:10.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47c0c00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47c0c00640 1 -- 172.21.15.67:0/1227370764 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f47bc085f20 con 0x7f47c415be70 2024-04-17T10:34:10.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47c3e00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.213+0000 7f47c3e00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f47b4007fd0 tx=0x7f47b4007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.217+0000 7f47c0c00640 1 -- 172.21.15.67:0/1227370764 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f47bc014030 con 0x7f47c415be70 2024-04-17T10:34:10.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.377+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f47c4158560 con 0x7f47c415be70 2024-04-17T10:34:10.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.377+0000 7f47c0c00640 1 -- 172.21.15.67:0/1227370764 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7f47bc056620 con 0x7f47c415be70 2024-04-17T10:34:10.371 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:34:10.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 msgr2=0x7f47a0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f47b4007fd0 tx=0x7f47b4007d60 comp rx=0 tx=0).stop 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c415be70 msgr2=0x7f47c414dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c415be70 0x7f47c414dfc0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f47bc00cd50 tx=0x7f47bc00a650 comp rx=0 tx=0).stop 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 shutdown_connections 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f47a0066f60 0x7f47a0069420 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47c415be70 0x7f47c414dfc0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f47c4159440 0x7f47c4152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 --2- 172.21.15.67:0/1227370764 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f47c4071af0 0x7f47c40b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 >> 172.21.15.67:0/1227370764 conn(0x7f47c40b1740 msgr2=0x7f47c40713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 shutdown_connections 2024-04-17T10:34:10.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.381+0000 7f47cae00640 1 -- 172.21.15.67:0/1227370764 wait complete. 2024-04-17T10:34:10.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:34:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b6c00640 1 Processor -- start 2024-04-17T10:34:10.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b6c00640 1 -- start start 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b6c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f70b0147360 0x7f70b006e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b6c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70b0071f00 0x7f70b00ade00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b6c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b0161730 con 0x7f70b0147740 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b6c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b01619d0 con 0x7f70b0071f00 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b6c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b0161cb0 con 0x7f70b0147360 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70b0071f00 0x7f70b00ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b4c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70b0071f00 0x7f70b00ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39938/0 (socket says 172.21.15.67:39938) 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70b4c00640 1 -- 172.21.15.67:0/3785398446 learned_addr learned my addr 172.21.15.67:0/3785398446 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:10.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.545+0000 7f70afe00640 1 --2- 172.21.15.67:0/3785398446 >> v2:172.21.15.86:3300/0 conn(0x7f70b0147360 0x7f70b006e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b4c00640 1 -- 172.21.15.67:0/3785398446 >> v2:172.21.15.86:3300/0 conn(0x7f70b0147360 msgr2=0x7f70b006e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:10.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b4c00640 1 --2- 172.21.15.67:0/3785398446 >> v2:172.21.15.86:3300/0 conn(0x7f70b0147360 0x7f70b006e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b4c00640 1 -- 172.21.15.67:0/3785398446 >> v1:172.21.15.86:6789/0 conn(0x7f70b0147740 legacy=0x7f70b00717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b4c00640 1 -- 172.21.15.67:0/3785398446 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b0161f90 con 0x7f70b0071f00 2024-04-17T10:34:10.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b4c00640 1 --2- 172.21.15.67:0/3785398446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0071f00 0x7f70b00ade00 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f70b006ba90 tx=0x7f70a801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8f394421b5adc2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70aea00640 1 -- 172.21.15.67:0/3785398446 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a80a58a0 con 0x7f70b0071f00 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70aea00640 1 -- 172.21.15.67:0/3785398446 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a80a5a40 con 0x7f70b0071f00 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70aea00640 1 -- 172.21.15.67:0/3785398446 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a80a5e10 con 0x7f70b0071f00 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b6c00640 1 -- 172.21.15.67:0/3785398446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0071f00 msgr2=0x7f70b00ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b6c00640 1 --2- 172.21.15.67:0/3785398446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0071f00 0x7f70b00ade00 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f70b006ba90 tx=0x7f70a801c9c0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b6c00640 1 -- 172.21.15.67:0/3785398446 shutdown_connections 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b6c00640 1 --2- 172.21.15.67:0/3785398446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0071f00 0x7f70b00ade00 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b6c00640 1 --2- 172.21.15.67:0/3785398446 >> v2:172.21.15.86:3300/0 conn(0x7f70b0147360 0x7f70b006e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.549+0000 7f70b6c00640 1 -- 172.21.15.67:0/3785398446 >> 172.21.15.67:0/3785398446 conn(0x7f70b006ac90 msgr2=0x7f70b006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:10.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 -- 172.21.15.67:0/3785398446 shutdown_connections 2024-04-17T10:34:10.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 -- 172.21.15.67:0/3785398446 wait complete. 2024-04-17T10:34:10.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 Processor -- start 2024-04-17T10:34:10.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 -- start start 2024-04-17T10:34:10.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b0071f00 0x7f70b013a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0147360 0x7f70b015bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b0147740 0x7f70b013af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b00aeee0 con 0x7f70b0147360 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b00aed60 con 0x7f70b0147740 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b00af060 con 0x7f70b0071f00 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0147360 0x7f70b015bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0147360 0x7f70b015bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39944/0 (socket says 172.21.15.67:39944) 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 -- 172.21.15.67:0/246422769 learned_addr learned my addr 172.21.15.67:0/246422769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70afe00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b0071f00 0x7f70b013a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b4c00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b0147740 0x7f70b013af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 -- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b0071f00 msgr2=0x7f70b013a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b0071f00 0x7f70b013a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 -- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b0147740 msgr2=0x7f70b013af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b0147740 0x7f70b013af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 -- 172.21.15.67:0/246422769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b01384b0 con 0x7f70b0147360 2024-04-17T10:34:10.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70af400640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0147360 0x7f70b015bc60 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c390 tx=0x7f709c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70a7e00640 1 -- 172.21.15.67:0/246422769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c017070 con 0x7f70b0147360 2024-04-17T10:34:10.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70a7e00640 1 -- 172.21.15.67:0/246422769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709c0052d0 con 0x7f70b0147360 2024-04-17T10:34:10.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70a7e00640 1 -- 172.21.15.67:0/246422769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709c003840 con 0x7f70b0147360 2024-04-17T10:34:10.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.553+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b01398c0 con 0x7f70b0147360 2024-04-17T10:34:10.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.557+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b0139e80 con 0x7f70b0147360 2024-04-17T10:34:10.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.557+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b004ef30 con 0x7f70b0147360 2024-04-17T10:34:10.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.557+0000 7f70a7e00640 1 -- 172.21.15.67:0/246422769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f709c01a070 con 0x7f70b0147360 2024-04-17T10:34:10.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.557+0000 7f70a7e00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7090066f90 0x7f7090069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.557+0000 7f70a7e00640 1 -- 172.21.15.67:0/246422769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f709c084350 con 0x7f70b0147360 2024-04-17T10:34:10.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.557+0000 7f70afe00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7090066f90 0x7f7090069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.557+0000 7f70afe00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7090066f90 0x7f7090069450 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009b60 tx=0x7f70a0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.561+0000 7f70a7e00640 1 -- 172.21.15.67:0/246422769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f709c005480 con 0x7f70b0147360 2024-04-17T10:34:10.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.725+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f70b00cc3d0 con 0x7f70b0147360 2024-04-17T10:34:10.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.725+0000 7f70a7e00640 1 -- 172.21.15.67:0/246422769 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f709c054b70 con 0x7f70b0147360 2024-04-17T10:34:10.718 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:34:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7090066f90 msgr2=0x7f7090069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7090066f90 0x7f7090069450 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f70a0009b60 tx=0x7f70a0009190 comp rx=0 tx=0).stop 2024-04-17T10:34:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0147360 msgr2=0x7f70b015bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0147360 0x7f70b015bc60 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c390 tx=0x7f709c00c850 comp rx=0 tx=0).stop 2024-04-17T10:34:10.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 shutdown_connections 2024-04-17T10:34:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7090066f90 0x7f7090069450 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70b0147740 0x7f70b013af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70b0147360 0x7f70b015bc60 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 --2- 172.21.15.67:0/246422769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f70b0071f00 0x7f70b013a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 >> 172.21.15.67:0/246422769 conn(0x7f70b006ac90 msgr2=0x7f70b015f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 shutdown_connections 2024-04-17T10:34:10.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.729+0000 7f70b6c00640 1 -- 172.21.15.67:0/246422769 wait complete. 2024-04-17T10:34:10.739 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:34:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a8600640 1 Processor -- start 2024-04-17T10:34:10.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a8600640 1 -- start start 2024-04-17T10:34:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a8600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006eac0 0x7f71a0143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a8600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71a00cf0c0 0x7f71a0144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a8600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a0144560 con 0x7f71a006e6e0 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a8600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a015ae40 con 0x7f71a00cf0c0 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a8600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a0071440 con 0x7f71a006eac0 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a6600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71a00cf0c0 0x7f71a0144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a6600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71a00cf0c0 0x7f71a0144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39948/0 (socket says 172.21.15.67:39948) 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a6600640 1 -- 172.21.15.67:0/781793982 learned_addr learned my addr 172.21.15.67:0/781793982 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a5200640 1 --2- 172.21.15.67:0/781793982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006eac0 0x7f71a0143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a6600640 1 -- 172.21.15.67:0/781793982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006eac0 msgr2=0x7f71a0143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a6600640 1 --2- 172.21.15.67:0/781793982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006eac0 0x7f71a0143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a6600640 1 -- 172.21.15.67:0/781793982 >> v1:172.21.15.86:6789/0 conn(0x7f71a006e6e0 legacy=0x7f71a00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.889+0000 7f71a6600640 1 -- 172.21.15.67:0/781793982 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a0071720 con 0x7f71a00cf0c0 2024-04-17T10:34:10.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a6600640 1 --2- 172.21.15.67:0/781793982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a00cf0c0 0x7f71a0144020 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f7198009e50 tx=0x7f719801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6f4d4a02bcf906 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f719fe00640 1 -- 172.21.15.67:0/781793982 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71980a5830 con 0x7f71a00cf0c0 2024-04-17T10:34:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f719fe00640 1 -- 172.21.15.67:0/781793982 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71980a59d0 con 0x7f71a00cf0c0 2024-04-17T10:34:10.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f719fe00640 1 -- 172.21.15.67:0/781793982 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71980a5da0 con 0x7f71a00cf0c0 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 -- 172.21.15.67:0/781793982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a00cf0c0 msgr2=0x7f71a0144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 --2- 172.21.15.67:0/781793982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a00cf0c0 0x7f71a0144020 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f7198009e50 tx=0x7f719801c970 comp rx=0 tx=0).stop 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 -- 172.21.15.67:0/781793982 shutdown_connections 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 --2- 172.21.15.67:0/781793982 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a00cf0c0 0x7f71a0144020 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 --2- 172.21.15.67:0/781793982 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006eac0 0x7f71a0143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 -- 172.21.15.67:0/781793982 >> 172.21.15.67:0/781793982 conn(0x7f71a006ac90 msgr2=0x7f71a006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 -- 172.21.15.67:0/781793982 shutdown_connections 2024-04-17T10:34:10.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.893+0000 7f71a8600640 1 -- 172.21.15.67:0/781793982 wait complete. 2024-04-17T10:34:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 Processor -- start 2024-04-17T10:34:10.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 -- start start 2024-04-17T10:34:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006e6e0 0x7f71a01f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a006eac0 0x7f71a00cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00cf0c0 0x7f71a00a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a00b3a40 con 0x7f71a006eac0 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a00b38c0 con 0x7f71a00cf0c0 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a00b3bc0 con 0x7f71a006e6e0 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a006eac0 0x7f71a00cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a006eac0 0x7f71a00cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39954/0 (socket says 172.21.15.67:39954) 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 -- 172.21.15.67:0/1796109851 learned_addr learned my addr 172.21.15.67:0/1796109851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5c00640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006e6e0 0x7f71a01f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a6600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00cf0c0 0x7f71a00a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 -- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006e6e0 msgr2=0x7f71a01f7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006e6e0 0x7f71a01f7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 -- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00cf0c0 msgr2=0x7f71a00a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:10.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00cf0c0 0x7f71a00a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 -- 172.21.15.67:0/1796109851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a014b6a0 con 0x7f71a006eac0 2024-04-17T10:34:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a5200640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a006eac0 0x7f71a00cba80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f718c00d800 tx=0x7f718c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f719d600640 1 -- 172.21.15.67:0/1796109851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c00b6c0 con 0x7f71a006eac0 2024-04-17T10:34:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f719d600640 1 -- 172.21.15.67:0/1796109851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718c019070 con 0x7f71a006eac0 2024-04-17T10:34:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f719d600640 1 -- 172.21.15.67:0/1796109851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718c00bae0 con 0x7f71a006eac0 2024-04-17T10:34:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a01f7330 con 0x7f71a006eac0 2024-04-17T10:34:10.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.897+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a01f78f0 con 0x7f71a006eac0 2024-04-17T10:34:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.901+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a0072d50 con 0x7f71a006eac0 2024-04-17T10:34:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.901+0000 7f719d600640 1 -- 172.21.15.67:0/1796109851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f718c00bc80 con 0x7f71a006eac0 2024-04-17T10:34:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.901+0000 7f719d600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7180066f60 0x7f7180069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.901+0000 7f719d600640 1 -- 172.21.15.67:0/1796109851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f718c085670 con 0x7f71a006eac0 2024-04-17T10:34:10.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.901+0000 7f71a5c00640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7180066f60 0x7f7180069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:10.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.901+0000 7f71a5c00640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7180066f60 0x7f7180069420 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7194001810 tx=0x7f7194009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:10.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:10.905+0000 7f719d600640 1 -- 172.21.15.67:0/1796109851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f718c055e00 con 0x7f71a006eac0 2024-04-17T10:34:11.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f71a0074e30 con 0x7f71a006eac0 2024-04-17T10:34:11.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f719d600640 1 -- 172.21.15.67:0/1796109851 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v118) v1 ==== 80+0+0 (secure 0 0 0) 0x7f718c059ea0 con 0x7f71a006eac0 2024-04-17T10:34:11.072 INFO:teuthology.orchestra.run.smithi067.stderr:listed 0 entries 2024-04-17T10:34:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7180066f60 msgr2=0x7f7180069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7180066f60 0x7f7180069420 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f7194001810 tx=0x7f7194009040 comp rx=0 tx=0).stop 2024-04-17T10:34:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a006eac0 msgr2=0x7f71a00cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a006eac0 0x7f71a00cba80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f718c00d800 tx=0x7f718c00dcc0 comp rx=0 tx=0).stop 2024-04-17T10:34:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 shutdown_connections 2024-04-17T10:34:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7180066f60 0x7f7180069420 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:11.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00cf0c0 0x7f71a00a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a006eac0 0x7f71a00cba80 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 --2- 172.21.15.67:0/1796109851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a006e6e0 0x7f71a01f7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 >> 172.21.15.67:0/1796109851 conn(0x7f71a006ac90 msgr2=0x7f71a0140de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 shutdown_connections 2024-04-17T10:34:11.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:11.081+0000 7f71a8600640 1 -- 172.21.15.67:0/1796109851 wait complete. 2024-04-17T10:34:11.092 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:34:11.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:11.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:34:11.821 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:34:11.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:11.821 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:34:11.830 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:34:11.830 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:11.839 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:34:11.840 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:34:11.840 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:11.882 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:34:11.883 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:34:11.883 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:11.891 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:34:11.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:12.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1dade00640 1 Processor -- start 2024-04-17T10:34:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1dade00640 1 -- start start 2024-04-17T10:34:12.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1dade00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 0x7f1da80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1dade00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1da80cc840 0x7f1da80cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1dade00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1da806b100 con 0x7f1da80cc250 2024-04-17T10:34:12.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1dade00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da80cd150 con 0x7f1da80cc840 2024-04-17T10:34:12.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1dade00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da806b3e0 con 0x7f1da80ce2a0 2024-04-17T10:34:12.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1da80cc250 0x7f1da80cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58626/0 (socket says 172.21.15.67:58626) 2024-04-17T10:34:12.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6a00640 1 -- 172.21.15.67:0/3356326622 learned_addr learned my addr 172.21.15.67:0/3356326622 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6000640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 0x7f1da80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da7400640 1 --2- 172.21.15.67:0/3356326622 >> v2:172.21.15.86:3300/0 conn(0x7f1da80cc840 0x7f1da80cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719502535 0 0) 0x7f1da806b100 con 0x7f1da80cc250 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d8c003530 con 0x7f1da80cc250 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358763179 0 0) 0x7f1d8c003530 con 0x7f1da80cc250 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 msgr2=0x7f1da80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 0x7f1da80cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 >> v2:172.21.15.86:3300/0 conn(0x7f1da80cc840 msgr2=0x7f1da80cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 --2- 172.21.15.67:0/3356326622 >> v2:172.21.15.86:3300/0 conn(0x7f1da80cc840 0x7f1da80cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da806b6c0 con 0x7f1da80cc250 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f1d98001680 con 0x7f1da80cc250 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 >> v1:172.21.15.67:6789/0 conn(0x7f1da80cc250 legacy=0x7f1da80cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8c0039c0 0x7f1d8c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1d8c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 0x7f1d8c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80cd150 con 0x7f1d8c004810 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da806b3e0 con 0x7f1d8c0039c0 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c00b9d0 con 0x7f1da80ce2a0 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6000640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1d8c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6a00640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8c0039c0 0x7f1d8c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da7400640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 0x7f1d8c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6000640 1 -- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 msgr2=0x7f1d8c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6000640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80ce2a0 0x7f1d8c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6000640 1 -- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8c0039c0 msgr2=0x7f1d8c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6000640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d8c0039c0 0x7f1d8c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.077+0000 7f1da6000640 1 -- 172.21.15.67:0/3356326622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c003530 con 0x7f1d8c004810 2024-04-17T10:34:12.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1da6000640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1d8c004c90 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009b10 tx=0x7f1d9401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123e98004de08603 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d940b2070 con 0x7f1d8c004810 2024-04-17T10:34:12.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d940a5580 con 0x7f1d8c004810 2024-04-17T10:34:12.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1da5600640 1 -- 172.21.15.67:0/3356326622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d940096b0 con 0x7f1d8c004810 2024-04-17T10:34:12.074 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1dade00640 1 -- 172.21.15.67:0/3356326622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 msgr2=0x7f1d8c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1dade00640 1 --2- 172.21.15.67:0/3356326622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1d8c004c90 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009b10 tx=0x7f1d9401ca70 comp rx=0 tx=0).stop 2024-04-17T10:34:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1da7400640 1 -- 172.21.15.67:0/3356326622 reap_dead start 2024-04-17T10:34:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1dade00640 1 -- 172.21.15.67:0/3356326622 shutdown_connections 2024-04-17T10:34:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1dade00640 1 -- 172.21.15.67:0/3356326622 >> 172.21.15.67:0/3356326622 conn(0x7f1da80b22a0 msgr2=0x7f1da80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1dade00640 1 -- 172.21.15.67:0/3356326622 shutdown_connections 2024-04-17T10:34:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.081+0000 7f1dade00640 1 -- 172.21.15.67:0/3356326622 wait complete. 2024-04-17T10:34:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 Processor -- start 2024-04-17T10:34:12.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 -- start start 2024-04-17T10:34:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1da815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da80c69b0 0x7f1da80c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80b44a0 0x7f1da80b4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da806ae10 con 0x7f1d8c004810 2024-04-17T10:34:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da806ac90 con 0x7f1da80c69b0 2024-04-17T10:34:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da80cd5e0 con 0x7f1da80b44a0 2024-04-17T10:34:12.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1da815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1da815dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39966/0 (socket says 172.21.15.67:39966) 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 -- 172.21.15.67:0/1257687142 learned_addr learned my addr 172.21.15.67:0/1257687142 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da7400640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80b44a0 0x7f1da80b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6000640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da80c69b0 0x7f1da80c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 -- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80b44a0 msgr2=0x7f1da80b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80b44a0 0x7f1da80b4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 -- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da80c69b0 msgr2=0x7f1da80c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da80c69b0 0x7f1da80c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 -- 172.21.15.67:0/1257687142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da80b51d0 con 0x7f1d8c004810 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da7400640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80b44a0 0x7f1da80b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6000640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da80c69b0 0x7f1da80c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1da6a00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1da815dba0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f1d98001880 tx=0x7f1d980126f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1d9f400640 1 -- 172.21.15.67:0/1257687142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9801c070 con 0x7f1d8c004810 2024-04-17T10:34:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1d9f400640 1 -- 172.21.15.67:0/1257687142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d98012e90 con 0x7f1d8c004810 2024-04-17T10:34:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da80b54b0 con 0x7f1d8c004810 2024-04-17T10:34:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1d9f400640 1 -- 172.21.15.67:0/1257687142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d98014990 con 0x7f1d8c004810 2024-04-17T10:34:12.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.085+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da80b5a70 con 0x7f1d8c004810 2024-04-17T10:34:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.089+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da806cd30 con 0x7f1d8c004810 2024-04-17T10:34:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.089+0000 7f1d9f400640 1 -- 172.21.15.67:0/1257687142 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1d9801f070 con 0x7f1d8c004810 2024-04-17T10:34:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.089+0000 7f1d9f400640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d88066f90 0x7f1d88069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.089+0000 7f1d9f400640 1 -- 172.21.15.67:0/1257687142 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1d98087ac0 con 0x7f1d8c004810 2024-04-17T10:34:12.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.089+0000 7f1da6000640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d88066f90 0x7f1d88069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.089+0000 7f1da6000640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d88066f90 0x7f1d88069450 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1d940af000 tx=0x7f1d940046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.093+0000 7f1d9f400640 1 -- 172.21.15.67:0/1257687142 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1d980588a0 con 0x7f1d8c004810 2024-04-17T10:34:12.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.289+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1da809e900 con 0x7f1d8c004810 2024-04-17T10:34:12.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.289+0000 7f1d9f400640 1 -- 172.21.15.67:0/1257687142 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f1d98087e20 con 0x7f1d8c004810 2024-04-17T10:34:12.283 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:12.283 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:12.283 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d88066f90 msgr2=0x7f1d88069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d88066f90 0x7f1d88069450 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f1d940af000 tx=0x7f1d940046d0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 msgr2=0x7f1da815dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1da815dba0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f1d98001880 tx=0x7f1d980126f0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 shutdown_connections 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1d88066f90 0x7f1d88069450 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1da80b44a0 0x7f1da80b4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1da80c69b0 0x7f1da80c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 --2- 172.21.15.67:0/1257687142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d8c004810 0x7f1da815dba0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 >> 172.21.15.67:0/1257687142 conn(0x7f1da80b22a0 msgr2=0x7f1da806e4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 shutdown_connections 2024-04-17T10:34:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.293+0000 7f1dade00640 1 -- 172.21.15.67:0/1257687142 wait complete. 2024-04-17T10:34:12.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:12.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35cc600640 1 Processor -- start 2024-04-17T10:34:12.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35cc600640 1 -- start start 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35cc600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35c415dc20 0x7f35c40ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35cc600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c40b6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35cc600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c40d2970 con 0x7f35c40cf070 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35cc600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c40d3580 con 0x7f35c415dc20 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35cc600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40d2c50 con 0x7f35c40d0b00 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c9c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f35c415dc20 0x7f35c40ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c9200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f35c40cf070 0x7f35c40d03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58638/0 (socket says 172.21.15.67:58638) 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c9200640 1 -- 172.21.15.67:0/851190154 learned_addr learned my addr 172.21.15.67:0/851190154 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35ca600640 1 --2- 172.21.15.67:0/851190154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c40b6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c9c00640 1 -- 172.21.15.67:0/851190154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 msgr2=0x7f35c40b6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c9c00640 1 --2- 172.21.15.67:0/851190154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c40b6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c9c00640 1 -- 172.21.15.67:0/851190154 >> v1:172.21.15.67:6789/0 conn(0x7f35c40cf070 legacy=0x7f35c40d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c9c00640 1 -- 172.21.15.67:0/851190154 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c40d2f30 con 0x7f35c415dc20 2024-04-17T10:34:12.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.453+0000 7f35c3e00640 1 -- 172.21.15.67:0/851190154 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093716587 0 0) 0x7f35c40d2970 con 0x7f35c40cf070 2024-04-17T10:34:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35c9c00640 1 --2- 172.21.15.67:0/851190154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c40ceb30 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f35b80097c0 tx=0x7f35b801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155363c7d62e602 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35c3e00640 1 -- 172.21.15.67:0/851190154 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b80b2070 con 0x7f35c415dc20 2024-04-17T10:34:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35c3e00640 1 -- 172.21.15.67:0/851190154 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b80ad680 con 0x7f35c415dc20 2024-04-17T10:34:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35c3e00640 1 -- 172.21.15.67:0/851190154 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b80ada50 con 0x7f35c415dc20 2024-04-17T10:34:12.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 -- 172.21.15.67:0/851190154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 msgr2=0x7f35c40ceb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 --2- 172.21.15.67:0/851190154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c40ceb30 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7f35b80097c0 tx=0x7f35b801c3e0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 -- 172.21.15.67:0/851190154 shutdown_connections 2024-04-17T10:34:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 --2- 172.21.15.67:0/851190154 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c40b6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 --2- 172.21.15.67:0/851190154 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c40ceb30 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 -- 172.21.15.67:0/851190154 >> 172.21.15.67:0/851190154 conn(0x7f35c4072a40 msgr2=0x7f35c40b55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 -- 172.21.15.67:0/851190154 shutdown_connections 2024-04-17T10:34:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.457+0000 7f35cc600640 1 -- 172.21.15.67:0/851190154 wait complete. 2024-04-17T10:34:12.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 Processor -- start 2024-04-17T10:34:12.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 -- start start 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c40cf070 0x7f35c40c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c415ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c41542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40b7bd0 con 0x7f35c415dc20 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40b7a50 con 0x7f35c40cf070 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c40b7d50 con 0x7f35c40d0b00 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c41542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c41542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39976/0 (socket says 172.21.15.67:39976) 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 -- 172.21.15.67:0/1588151703 learned_addr learned my addr 172.21.15.67:0/1588151703 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35c9c00640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c40cf070 0x7f35c40c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35c9200640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c415ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 -- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 msgr2=0x7f35c415ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c415ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 -- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c40cf070 msgr2=0x7f35c40c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c40cf070 0x7f35c40c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 -- 172.21.15.67:0/1588151703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c415f650 con 0x7f35c415dc20 2024-04-17T10:34:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35ca600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c41542c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00ef90 tx=0x7f35bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35c1600640 1 -- 172.21.15.67:0/1588151703 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc018020 con 0x7f35c415dc20 2024-04-17T10:34:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35c1600640 1 -- 172.21.15.67:0/1588151703 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35bc014480 con 0x7f35c415dc20 2024-04-17T10:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c415f930 con 0x7f35c415dc20 2024-04-17T10:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35c1600640 1 -- 172.21.15.67:0/1588151703 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc014870 con 0x7f35c415dc20 2024-04-17T10:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.461+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c415fef0 con 0x7f35c415dc20 2024-04-17T10:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.465+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c4074e30 con 0x7f35c415dc20 2024-04-17T10:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.465+0000 7f35c1600640 1 -- 172.21.15.67:0/1588151703 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f35bc01b020 con 0x7f35c415dc20 2024-04-17T10:34:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.465+0000 7f35c1600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35a4066f90 0x7f35a4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.465+0000 7f35c1600640 1 -- 172.21.15.67:0/1588151703 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f35bc085ee0 con 0x7f35c415dc20 2024-04-17T10:34:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.465+0000 7f35c9c00640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35a4066f90 0x7f35a4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.469+0000 7f35c9c00640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35a4066f90 0x7f35a4069450 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f35b80af000 tx=0x7f35b80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.469+0000 7f35c1600640 1 -- 172.21.15.67:0/1588151703 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f35bc022030 con 0x7f35c415dc20 2024-04-17T10:34:12.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.633+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35c415e410 con 0x7f35c415dc20 2024-04-17T10:34:12.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35c1600640 1 -- 172.21.15.67:0/1588151703 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16904 (secure 0 0 0) 0x7f35bc010040 con 0x7f35c415dc20 2024-04-17T10:34:12.628 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:12.628 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":118,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:31.509307+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:33:28.333663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-17T10:33:28.871385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35a4066f90 msgr2=0x7f35a4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35a4066f90 0x7f35a4069450 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f35b80af000 tx=0x7f35b80a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 msgr2=0x7f35c41542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c41542c0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f35bc00ef90 tx=0x7f35bc00c490 comp rx=0 tx=0).stop 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 shutdown_connections 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f35a4066f90 0x7f35a4069450 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35c415dc20 0x7f35c41542c0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f35c40d0b00 0x7f35c415ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 --2- 172.21.15.67:0/1588151703 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f35c40cf070 0x7f35c40c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 >> 172.21.15.67:0/1588151703 conn(0x7f35c4072a40 msgr2=0x7f35c40d00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 shutdown_connections 2024-04-17T10:34:12.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.637+0000 7f35cc600640 1 -- 172.21.15.67:0/1588151703 wait complete. 2024-04-17T10:34:12.649 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:33:28.333663+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '117', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:33:28.871385+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '118', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:34:12.649 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:34:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b94a00640 1 Processor -- start 2024-04-17T10:34:12.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b94a00640 1 -- start start 2024-04-17T10:34:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b94a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6b90099a50 0x7f6b900cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b94a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 0x7f6b900b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b94a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b900cc3e0 con 0x7f6b900b2a30 2024-04-17T10:34:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b94a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b900cd9c0 con 0x7f6b90099a50 2024-04-17T10:34:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b94a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cc6c0 con 0x7f6b900cf200 2024-04-17T10:34:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b85e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6b900b2a30 0x7f6b90099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58648/0 (socket says 172.21.15.67:58648) 2024-04-17T10:34:12.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b85e00640 1 -- 172.21.15.67:0/3839374002 learned_addr learned my addr 172.21.15.67:0/3839374002 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8f200640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 0x7f6b900b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8e800640 1 --2- 172.21.15.67:0/3839374002 >> v2:172.21.15.86:3300/0 conn(0x7f6b90099a50 0x7f6b900cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816058375 0 0) 0x7f6b900cc3e0 con 0x7f6b900b2a30 2024-04-17T10:34:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b74003530 con 0x7f6b900b2a30 2024-04-17T10:34:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454245341 0 0) 0x7f6b74003530 con 0x7f6b900b2a30 2024-04-17T10:34:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 msgr2=0x7f6b900b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 0x7f6b900b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 >> v2:172.21.15.86:3300/0 conn(0x7f6b90099a50 msgr2=0x7f6b900cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 --2- 172.21.15.67:0/3839374002 >> v2:172.21.15.86:3300/0 conn(0x7f6b90099a50 0x7f6b900cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b900cc9a0 con 0x7f6b900b2a30 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.801+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f6b78001680 con 0x7f6b900b2a30 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 >> v1:172.21.15.67:6789/0 conn(0x7f6b900b2a30 legacy=0x7f6b90099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 0x7f6b740039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74003f90 0x7f6b74004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cd9c0 con 0x7f6b74004d50 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900cc6c0 con 0x7f6b74003f90 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7400b9d0 con 0x7f6b900cf200 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8f200640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8e800640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 0x7f6b740039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b85e00640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74003f90 0x7f6b74004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8f200640 1 -- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 msgr2=0x7f6b740039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8f200640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b900cf200 0x7f6b740039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8f200640 1 -- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74003f90 msgr2=0x7f6b74004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8f200640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b74003f90 0x7f6b74004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8f200640 1 -- 172.21.15.67:0/3839374002 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b74003530 con 0x7f6b74004d50 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8f200640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b740051d0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800e170 tx=0x7f6b8801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f01393676a7a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b880b2070 con 0x7f6b74004d50 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b880a5c70 con 0x7f6b74004d50 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.805+0000 7f6b8de00640 1 -- 172.21.15.67:0/3839374002 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88009c10 con 0x7f6b74004d50 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- 172.21.15.67:0/3839374002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 msgr2=0x7f6b740051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 --2- 172.21.15.67:0/3839374002 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b740051d0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800e170 tx=0x7f6b8801cc60 comp rx=0 tx=0).stop 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8f200640 1 -- 172.21.15.67:0/3839374002 reap_dead start 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- 172.21.15.67:0/3839374002 shutdown_connections 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- 172.21.15.67:0/3839374002 >> 172.21.15.67:0/3839374002 conn(0x7f6b900d1630 msgr2=0x7f6b900d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- 172.21.15.67:0/3839374002 shutdown_connections 2024-04-17T10:34:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- 172.21.15.67:0/3839374002 wait complete. 2024-04-17T10:34:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 Processor -- start 2024-04-17T10:34:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- start start 2024-04-17T10:34:12.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b901f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b901f6870 0x7f6b900afb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b9015d560 0x7f6b9015d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900d2370 con 0x7f6b74004d50 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900ccf60 con 0x7f6b9015d560 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b94a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b900d24f0 con 0x7f6b901f6870 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b901f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b901f3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39988/0 (socket says 172.21.15.67:39988) 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b85e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b901f6870 0x7f6b900afb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 -- 172.21.15.67:0/1687971238 learned_addr learned my addr 172.21.15.67:0/1687971238 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8f200640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b9015d560 0x7f6b9015d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 -- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b901f6870 msgr2=0x7f6b900afb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b901f6870 0x7f6b900afb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 -- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b9015d560 msgr2=0x7f6b9015d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b9015d560 0x7f6b9015d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.809+0000 7f6b8e800640 1 -- 172.21.15.67:0/1687971238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b900c87a0 con 0x7f6b74004d50 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b8e800640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b901f3cb0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f6b80011aa0 tx=0x7f6b80011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b87400640 1 -- 172.21.15.67:0/1687971238 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000fb80 con 0x7f6b74004d50 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b87400640 1 -- 172.21.15.67:0/1687971238 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8000fd20 con 0x7f6b74004d50 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b87400640 1 -- 172.21.15.67:0/1687971238 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80014680 con 0x7f6b74004d50 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b901f6e30 con 0x7f6b74004d50 2024-04-17T10:34:12.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b901f73f0 con 0x7f6b74004d50 2024-04-17T10:34:12.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9004f780 con 0x7f6b74004d50 2024-04-17T10:34:12.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b87400640 1 -- 172.21.15.67:0/1687971238 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6b80012040 con 0x7f6b74004d50 2024-04-17T10:34:12.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b87400640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b6c066f60 0x7f6b6c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:12.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b87400640 1 -- 172.21.15.67:0/1687971238 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6b800857b0 con 0x7f6b74004d50 2024-04-17T10:34:12.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.813+0000 7f6b85e00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b6c066f60 0x7f6b6c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:12.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.817+0000 7f6b85e00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b6c066f60 0x7f6b6c069420 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6b90070540 tx=0x7f6b78001e20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.817+0000 7f6b87400640 1 -- 172.21.15.67:0/1687971238 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6b80019030 con 0x7f6b74004d50 2024-04-17T10:34:12.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.981+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6b90070290 con 0x7f6b74004d50 2024-04-17T10:34:12.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.981+0000 7f6b87400640 1 -- 172.21.15.67:0/1687971238 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v118) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6b80055f40 con 0x7f6b74004d50 2024-04-17T10:34:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:34:12.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b6c066f60 msgr2=0x7f6b6c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b6c066f60 0x7f6b6c069420 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f6b90070540 tx=0x7f6b78001e20 comp rx=0 tx=0).stop 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 msgr2=0x7f6b901f3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b901f3cb0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f6b80011aa0 tx=0x7f6b80011f60 comp rx=0 tx=0).stop 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 shutdown_connections 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b6c066f60 0x7f6b6c069420 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b9015d560 0x7f6b9015d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b901f6870 0x7f6b900afb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 --2- 172.21.15.67:0/1687971238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b74004d50 0x7f6b901f3cb0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 >> 172.21.15.67:0/1687971238 conn(0x7f6b900d1630 msgr2=0x7f6b90070a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 shutdown_connections 2024-04-17T10:34:12.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:12.985+0000 7f6b94a00640 1 -- 172.21.15.67:0/1687971238 wait complete. 2024-04-17T10:34:12.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:34:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.145+0000 7f7a3ec00640 1 Processor -- start 2024-04-17T10:34:13.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.145+0000 7f7a3ec00640 1 -- start start 2024-04-17T10:34:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3ec00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a3806d150 0x7f7a3815f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3ec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a3815dfd0 0x7f7a3815e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3ec00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7a38099a50 0x7f7a38099ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3ec00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3809a400 con 0x7f7a3806d150 2024-04-17T10:34:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3ec00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a3815eb20 con 0x7f7a38099a50 2024-04-17T10:34:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3ec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3815ee00 con 0x7f7a3815dfd0 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a3806d150 0x7f7a3815f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7a3806d150 0x7f7a3815f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40000/0 (socket says 172.21.15.67:40000) 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 -- 172.21.15.67:0/3111981031 learned_addr learned my addr 172.21.15.67:0/3111981031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3ce00640 1 --2- 172.21.15.67:0/3111981031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a3815dfd0 0x7f7a3815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3e200640 1 --2- 172.21.15.67:0/3111981031 >> v2:172.21.15.86:3300/0 conn(0x7f7a38099a50 0x7f7a38099ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 -- 172.21.15.67:0/3111981031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a3815dfd0 msgr2=0x7f7a3815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 --2- 172.21.15.67:0/3111981031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a3815dfd0 0x7f7a3815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 -- 172.21.15.67:0/3111981031 >> v2:172.21.15.86:3300/0 conn(0x7f7a38099a50 msgr2=0x7f7a38099ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 --2- 172.21.15.67:0/3111981031 >> v2:172.21.15.86:3300/0 conn(0x7f7a38099a50 0x7f7a38099ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.149+0000 7f7a3d800640 1 -- 172.21.15.67:0/3111981031 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380b5230 con 0x7f7a3806d150 2024-04-17T10:34:13.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3d800640 1 --2- 172.21.15.67:0/3111981031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a3815f400 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f7a380b4010 tx=0x7f7a2c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1196835a9ab99b5a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a37e00640 1 -- 172.21.15.67:0/3111981031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0b2070 con 0x7f7a3806d150 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a37e00640 1 -- 172.21.15.67:0/3111981031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c0a9070 con 0x7f7a3806d150 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a37e00640 1 -- 172.21.15.67:0/3111981031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0ad880 con 0x7f7a3806d150 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 -- 172.21.15.67:0/3111981031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 msgr2=0x7f7a3815f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/3111981031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a3815f400 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f7a380b4010 tx=0x7f7a2c01c490 comp rx=0 tx=0).stop 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 -- 172.21.15.67:0/3111981031 shutdown_connections 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/3111981031 >> v2:172.21.15.86:3300/0 conn(0x7f7a38099a50 0x7f7a38099ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/3111981031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a3815dfd0 0x7f7a3815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/3111981031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a3815f400 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 -- 172.21.15.67:0/3111981031 >> 172.21.15.67:0/3111981031 conn(0x7f7a380b31b0 msgr2=0x7f7a380b35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 -- 172.21.15.67:0/3111981031 shutdown_connections 2024-04-17T10:34:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 -- 172.21.15.67:0/3111981031 wait complete. 2024-04-17T10:34:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 Processor -- start 2024-04-17T10:34:13.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.153+0000 7f7a3ec00640 1 -- start start 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a38143c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a38099a50 0x7f7a38145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a3815dfd0 0x7f7a3814fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38158260 con 0x7f7a3806d150 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a381580e0 con 0x7f7a3815dfd0 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a381583e0 con 0x7f7a38099a50 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a38143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a38143c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40006/0 (socket says 172.21.15.67:40006) 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 -- 172.21.15.67:0/128751860 learned_addr learned my addr 172.21.15.67:0/128751860 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ce00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a38099a50 0x7f7a38145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3e200640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a3815dfd0 0x7f7a3814fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 -- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a38099a50 msgr2=0x7f7a38145a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a38099a50 0x7f7a38145a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 -- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a3815dfd0 msgr2=0x7f7a3814fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a3815dfd0 0x7f7a3814fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 -- 172.21.15.67:0/128751860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3814f470 con 0x7f7a3806d150 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3e200640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a3815dfd0 0x7f7a3814fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3d800640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a38143c50 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c01ab40 tx=0x7f7a2c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a35600640 1 -- 172.21.15.67:0/128751860 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0b2040 con 0x7f7a3806d150 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a35600640 1 -- 172.21.15.67:0/128751860 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2c0be070 con 0x7f7a3806d150 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a38145f70 con 0x7f7a3806d150 2024-04-17T10:34:13.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a35600640 1 -- 172.21.15.67:0/128751860 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0b8660 con 0x7f7a3806d150 2024-04-17T10:34:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a38158a70 con 0x7f7a3806d150 2024-04-17T10:34:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3804f780 con 0x7f7a3806d150 2024-04-17T10:34:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.157+0000 7f7a35600640 1 -- 172.21.15.67:0/128751860 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7a2c0ab070 con 0x7f7a3806d150 2024-04-17T10:34:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.161+0000 7f7a35600640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a18066f60 0x7f7a18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.161+0000 7f7a35600640 1 -- 172.21.15.67:0/128751860 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7a2c11f300 con 0x7f7a3806d150 2024-04-17T10:34:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.161+0000 7f7a3ce00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a18066f60 0x7f7a18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.161+0000 7f7a3ce00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a18066f60 0x7f7a18069420 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7a38070be0 tx=0x7f7a24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.165+0000 7f7a35600640 1 -- 172.21.15.67:0/128751860 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7a2c016110 con 0x7f7a3806d150 2024-04-17T10:34:13.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.329+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a3809a610 con 0x7f7a3806d150 2024-04-17T10:34:13.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.329+0000 7f7a35600640 1 -- 172.21.15.67:0/128751860 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v118) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7a2c0f00e0 con 0x7f7a3806d150 2024-04-17T10:34:13.322 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a18066f60 msgr2=0x7f7a18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a18066f60 0x7f7a18069420 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f7a38070be0 tx=0x7f7a24008040 comp rx=0 tx=0).stop 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 msgr2=0x7f7a38143c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a38143c50 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c01ab40 tx=0x7f7a2c01acd0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 shutdown_connections 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7a18066f60 0x7f7a18069420 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7a3815dfd0 0x7f7a3814fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7a38099a50 0x7f7a38145a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 --2- 172.21.15.67:0/128751860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a3806d150 0x7f7a38143c50 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 >> 172.21.15.67:0/128751860 conn(0x7f7a380b31b0 msgr2=0x7f7a380b25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:13.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 shutdown_connections 2024-04-17T10:34:13.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.333+0000 7f7a3ec00640 1 -- 172.21.15.67:0/128751860 wait complete. 2024-04-17T10:34:13.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:34:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.493+0000 7f1957200640 1 Processor -- start 2024-04-17T10:34:13.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.493+0000 7f1957200640 1 -- start start 2024-04-17T10:34:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1957200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f19580cdd70 0x7f19580d0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1957200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f195809ab50 con 0x7f195809a160 2024-04-17T10:34:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1957200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19580b0f90 con 0x7f195815eec0 2024-04-17T10:34:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1957200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19580b18c0 con 0x7f19580cdd70 2024-04-17T10:34:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f195809a160 0x7f19580b1d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58664/0 (socket says 172.21.15.67:58664) 2024-04-17T10:34:13.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 -- 172.21.15.67:0/4108763350 learned_addr learned my addr 172.21.15.67:0/4108763350 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f194d400640 1 --2- 172.21.15.67:0/4108763350 >> v2:172.21.15.86:3300/0 conn(0x7f19580cdd70 0x7f19580d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268788827 0 0) 0x7f195809ab50 con 0x7f195809a160 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f193c003530 con 0x7f195809a160 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 437310412 0 0) 0x7f193c003530 con 0x7f195809a160 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 >> v2:172.21.15.86:3300/0 conn(0x7f19580cdd70 msgr2=0x7f19580d0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 --2- 172.21.15.67:0/4108763350 >> v2:172.21.15.86:3300/0 conn(0x7f19580cdd70 0x7f19580d0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 >> v1:172.21.15.86:6789/0 conn(0x7f195815eec0 legacy=0x7f1958099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19580b1270 con 0x7f195809a160 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f1950001a00 con 0x7f195809a160 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 >> v1:172.21.15.67:6789/0 conn(0x7f195809a160 legacy=0x7f19580b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193c0039c0 0x7f193c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193c004810 0x7f193c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f193c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19580b18c0 con 0x7f193c0052f0 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c00c1c0 con 0x7f193c0039c0 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193c00c3c0 con 0x7f193c004810 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f193c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1955e00640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193c0039c0 0x7f193c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f194d400640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193c004810 0x7f193c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 -- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193c004810 msgr2=0x7f193c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:13.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f193c004810 0x7f193c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 -- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193c0039c0 msgr2=0x7f193c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f193c0039c0 0x7f193c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.497+0000 7f1956800640 1 -- 172.21.15.67:0/4108763350 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c003530 con 0x7f193c0052f0 2024-04-17T10:34:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1956800640 1 -- 172.21.15.67:0/4108763350 reap_dead start 2024-04-17T10:34:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1956800640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f193c00b9d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f1950002240 tx=0x7f195001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792b95b2543561a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195000a4a0 con 0x7f193c0052f0 2024-04-17T10:34:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19500ad650 con 0x7f193c0052f0 2024-04-17T10:34:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1955400640 1 -- 172.21.15.67:0/4108763350 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19500ada40 con 0x7f193c0052f0 2024-04-17T10:34:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 -- 172.21.15.67:0/4108763350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 msgr2=0x7f193c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f193c00b9d0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f1950002240 tx=0x7f195001d360 comp rx=0 tx=0).stop 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 -- 172.21.15.67:0/4108763350 shutdown_connections 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 --2- 172.21.15.67:0/4108763350 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f193c00b9d0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 -- 172.21.15.67:0/4108763350 >> 172.21.15.67:0/4108763350 conn(0x7f19580d16d0 msgr2=0x7f19580d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 -- 172.21.15.67:0/4108763350 shutdown_connections 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 -- 172.21.15.67:0/4108763350 wait complete. 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 Processor -- start 2024-04-17T10:34:13.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.501+0000 7f1957200640 1 -- start start 2024-04-17T10:34:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f195814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19581f59c0 0x7f19580cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1958154fe0 0x7f19581f6f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19580d0050 con 0x7f193c0052f0 2024-04-17T10:34:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f195809a9a0 con 0x7f19581f59c0 2024-04-17T10:34:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19580d01d0 con 0x7f1958154fe0 2024-04-17T10:34:13.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f195814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f195814d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40032/0 (socket says 172.21.15.67:40032) 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 -- 172.21.15.67:0/4031842119 learned_addr learned my addr 172.21.15.67:0/4031842119 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1956800640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1958154fe0 0x7f19581f6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f194d400640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19581f59c0 0x7f19580cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 -- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1958154fe0 msgr2=0x7f19581f6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1958154fe0 0x7f19581f6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 -- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19581f59c0 msgr2=0x7f19580cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19581f59c0 0x7f19580cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 -- 172.21.15.67:0/4031842119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19581f5e80 con 0x7f193c0052f0 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1955e00640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f195814d6d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f194800f620 tx=0x7f194800fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f194ea00640 1 -- 172.21.15.67:0/4031842119 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19480052e0 con 0x7f193c0052f0 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f194ea00640 1 -- 172.21.15.67:0/4031842119 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1948018070 con 0x7f193c0052f0 2024-04-17T10:34:13.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19581f6160 con 0x7f193c0052f0 2024-04-17T10:34:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f194ea00640 1 -- 172.21.15.67:0/4031842119 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948003820 con 0x7f193c0052f0 2024-04-17T10:34:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19581f6720 con 0x7f193c0052f0 2024-04-17T10:34:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f195804f780 con 0x7f193c0052f0 2024-04-17T10:34:13.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.505+0000 7f194ea00640 1 -- 172.21.15.67:0/4031842119 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f19480039c0 con 0x7f193c0052f0 2024-04-17T10:34:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.509+0000 7f194ea00640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.509+0000 7f194ea00640 1 -- 172.21.15.67:0/4031842119 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f1948084ff0 con 0x7f193c0052f0 2024-04-17T10:34:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.509+0000 7f194d400640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.509+0000 7f194d400640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1958154690 tx=0x7f1940008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.513+0000 7f194ea00640 1 -- 172.21.15.67:0/4031842119 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1948055e10 con 0x7f193c0052f0 2024-04-17T10:34:13.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.673+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19581543e0 con 0x7f193c0052f0 2024-04-17T10:34:13.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.673+0000 7f194ea00640 1 -- 172.21.15.67:0/4031842119 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v118) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1948005480 con 0x7f193c0052f0 2024-04-17T10:34:13.666 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 msgr2=0x7f1934069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f1958154690 tx=0x7f1940008040 comp rx=0 tx=0).stop 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 msgr2=0x7f195814d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f195814d6d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f194800f620 tx=0x7f194800fae0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 shutdown_connections 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1934066f60 0x7f1934069420 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1958154fe0 0x7f19581f6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f19581f59c0 0x7f19580cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 --2- 172.21.15.67:0/4031842119 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f193c0052f0 0x7f195814d6d0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 >> 172.21.15.67:0/4031842119 conn(0x7f19580d16d0 msgr2=0x7f1958157d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 shutdown_connections 2024-04-17T10:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.677+0000 7f1957200640 1 -- 172.21.15.67:0/4031842119 wait complete. 2024-04-17T10:34:13.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 Processor -- start 2024-04-17T10:34:13.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 -- start start 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422815bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42280ce420 0x7f42280ce880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f422815f3e0 0x7f422815f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42280cedc0 con 0x7f42280ce420 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422806d6e0 con 0x7f422815f3e0 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422de00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4228160150 con 0x7f42281450b0 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42280ce420 0x7f42280ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f42280ce420 0x7f42280ce880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40036/0 (socket says 172.21.15.67:40036) 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 -- 172.21.15.67:0/2259204003 learned_addr learned my addr 172.21.15.67:0/2259204003 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422ca00640 1 --2- 172.21.15.67:0/2259204003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422815bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f422d400640 1 --2- 172.21.15.67:0/2259204003 >> v2:172.21.15.86:3300/0 conn(0x7f422815f3e0 0x7f422815f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 -- 172.21.15.67:0/2259204003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 msgr2=0x7f422815bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 --2- 172.21.15.67:0/2259204003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422815bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 -- 172.21.15.67:0/2259204003 >> v2:172.21.15.86:3300/0 conn(0x7f422815f3e0 msgr2=0x7f422815f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 --2- 172.21.15.67:0/2259204003 >> v2:172.21.15.86:3300/0 conn(0x7f422815f3e0 0x7f422815f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.837+0000 7f4227e00640 1 -- 172.21.15.67:0/2259204003 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422806d930 con 0x7f42280ce420 2024-04-17T10:34:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f4227e00640 1 --2- 172.21.15.67:0/2259204003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280ce420 0x7f42280ce880 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f4228160360 tx=0x7f421401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844bbcb37f76b9a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f4227400640 1 -- 172.21.15.67:0/2259204003 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42140b0070 con 0x7f42280ce420 2024-04-17T10:34:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f4227400640 1 -- 172.21.15.67:0/2259204003 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42140ab600 con 0x7f42280ce420 2024-04-17T10:34:13.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f4227400640 1 -- 172.21.15.67:0/2259204003 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42140ab9d0 con 0x7f42280ce420 2024-04-17T10:34:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 -- 172.21.15.67:0/2259204003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280ce420 msgr2=0x7f42280ce880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 --2- 172.21.15.67:0/2259204003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280ce420 0x7f42280ce880 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f4228160360 tx=0x7f421401c490 comp rx=0 tx=0).stop 2024-04-17T10:34:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 -- 172.21.15.67:0/2259204003 shutdown_connections 2024-04-17T10:34:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 --2- 172.21.15.67:0/2259204003 >> v2:172.21.15.86:3300/0 conn(0x7f422815f3e0 0x7f422815f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 --2- 172.21.15.67:0/2259204003 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42280ce420 0x7f42280ce880 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 --2- 172.21.15.67:0/2259204003 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422815bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 -- 172.21.15.67:0/2259204003 >> 172.21.15.67:0/2259204003 conn(0x7f4228067070 msgr2=0x7f422806ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 -- 172.21.15.67:0/2259204003 shutdown_connections 2024-04-17T10:34:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 -- 172.21.15.67:0/2259204003 wait complete. 2024-04-17T10:34:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 Processor -- start 2024-04-17T10:34:13.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.841+0000 7f422de00640 1 -- start start 2024-04-17T10:34:13.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280ce420 0x7f4228135940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422813eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422815f3e0 0x7f4228137130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422815fed0 con 0x7f422815f3e0 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422815fd50 con 0x7f42280ce420 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f422806ec20 con 0x7f42281450b0 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422815f3e0 0x7f4228137130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280ce420 0x7f4228135940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f4227e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422813eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422815f3e0 0x7f4228137130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40046/0 (socket says 172.21.15.67:40046) 2024-04-17T10:34:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 -- 172.21.15.67:0/648296452 learned_addr learned my addr 172.21.15.67:0/648296452 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 -- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 msgr2=0x7f422813eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422813eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 -- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280ce420 msgr2=0x7f4228135940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280ce420 0x7f4228135940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:13.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 -- 172.21.15.67:0/648296452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422813f8c0 con 0x7f422815f3e0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422d400640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422815f3e0 0x7f4228137130 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f4220007e60 tx=0x7f422000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f4224c00640 1 -- 172.21.15.67:0/648296452 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220014420 con 0x7f422815f3e0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f4224c00640 1 -- 172.21.15.67:0/648296452 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4220019070 con 0x7f422815f3e0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f4224c00640 1 -- 172.21.15.67:0/648296452 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4220014820 con 0x7f422815f3e0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422813fba0 con 0x7f422815f3e0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4228140160 con 0x7f422815f3e0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4228074dc0 con 0x7f422815f3e0 2024-04-17T10:34:13.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.845+0000 7f4224c00640 1 -- 172.21.15.67:0/648296452 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f42200149c0 con 0x7f422815f3e0 2024-04-17T10:34:13.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.849+0000 7f4224c00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4208066f60 0x7f4208069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:13.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.849+0000 7f4224c00640 1 -- 172.21.15.67:0/648296452 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4220085880 con 0x7f422815f3e0 2024-04-17T10:34:13.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.849+0000 7f422ca00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4208066f60 0x7f4208069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:13.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.849+0000 7f422ca00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4208066f60 0x7f4208069420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f421c007a60 tx=0x7f421c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:13.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:13.853+0000 7f4224c00640 1 -- 172.21.15.67:0/648296452 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f42200560b0 con 0x7f422815f3e0 2024-04-17T10:34:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42280b0920 con 0x7f422815f3e0 2024-04-17T10:34:14.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f4224c00640 1 -- 172.21.15.67:0/648296452 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7f422005a150 con 0x7f422815f3e0 2024-04-17T10:34:14.040 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:14.040 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:14.040 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4208066f60 msgr2=0x7f4208069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4208066f60 0x7f4208069420 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f421c007a60 tx=0x7f421c008040 comp rx=0 tx=0).stop 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422815f3e0 msgr2=0x7f4228137130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422815f3e0 0x7f4228137130 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f4220007e60 tx=0x7f422000e4b0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 shutdown_connections 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4208066f60 0x7f4208069420 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f422815f3e0 0x7f4228137130 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42281450b0 0x7f422813eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 --2- 172.21.15.67:0/648296452 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42280ce420 0x7f4228135940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 >> 172.21.15.67:0/648296452 conn(0x7f4228067070 msgr2=0x7f42280c4730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 shutdown_connections 2024-04-17T10:34:14.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.049+0000 7f422de00640 1 -- 172.21.15.67:0/648296452 wait complete. 2024-04-17T10:34:14.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.209+0000 7ff455400640 1 Processor -- start 2024-04-17T10:34:14.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.209+0000 7ff455400640 1 -- start start 2024-04-17T10:34:14.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff455400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff45006e6e0 0x7ff4500ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff455400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff45006eac0 0x7ff4500ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff455400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45015ae40 con 0x7ff450143ae0 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff455400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4500cf430 con 0x7ff45006eac0 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff455400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff450071440 con 0x7ff45006e6e0 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff447e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff45006eac0 0x7ff4500ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff44f200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff450143ae0 0x7ff450143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58674/0 (socket says 172.21.15.67:58674) 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff44f200640 1 -- 172.21.15.67:0/3611228664 learned_addr learned my addr 172.21.15.67:0/3611228664 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff44e800640 1 --2- 172.21.15.67:0/3611228664 >> v2:172.21.15.86:3300/0 conn(0x7ff45006e6e0 0x7ff4500ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff447e00640 1 -- 172.21.15.67:0/3611228664 >> v2:172.21.15.86:3300/0 conn(0x7ff45006e6e0 msgr2=0x7ff4500ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff447e00640 1 --2- 172.21.15.67:0/3611228664 >> v2:172.21.15.86:3300/0 conn(0x7ff45006e6e0 0x7ff4500ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff447e00640 1 -- 172.21.15.67:0/3611228664 >> v1:172.21.15.67:6789/0 conn(0x7ff450143ae0 legacy=0x7ff450143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff447e00640 1 -- 172.21.15.67:0/3611228664 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450071720 con 0x7ff45006eac0 2024-04-17T10:34:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff44e800640 1 --2- 172.21.15.67:0/3611228664 >> v2:172.21.15.86:3300/0 conn(0x7ff45006e6e0 0x7ff4500ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:14.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff44de00640 1 -- 172.21.15.67:0/3611228664 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668488475 0 0) 0x7ff45015ae40 con 0x7ff450143ae0 2024-04-17T10:34:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.213+0000 7ff447e00640 1 --2- 172.21.15.67:0/3611228664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006eac0 0x7ff4500ceef0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7ff4500cf620 tx=0x7ff43801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8cf5be408069f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff44de00640 1 -- 172.21.15.67:0/3611228664 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4380b1070 con 0x7ff45006eac0 2024-04-17T10:34:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff44de00640 1 -- 172.21.15.67:0/3611228664 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4380ac620 con 0x7ff45006eac0 2024-04-17T10:34:14.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff44de00640 1 -- 172.21.15.67:0/3611228664 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4380ac9f0 con 0x7ff45006eac0 2024-04-17T10:34:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- 172.21.15.67:0/3611228664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006eac0 msgr2=0x7ff4500ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 --2- 172.21.15.67:0/3611228664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006eac0 0x7ff4500ceef0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7ff4500cf620 tx=0x7ff43801c3c0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- 172.21.15.67:0/3611228664 shutdown_connections 2024-04-17T10:34:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 --2- 172.21.15.67:0/3611228664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006eac0 0x7ff4500ceef0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 --2- 172.21.15.67:0/3611228664 >> v2:172.21.15.86:3300/0 conn(0x7ff45006e6e0 0x7ff4500ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- 172.21.15.67:0/3611228664 >> 172.21.15.67:0/3611228664 conn(0x7ff45006ac90 msgr2=0x7ff45006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:14.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- 172.21.15.67:0/3611228664 shutdown_connections 2024-04-17T10:34:14.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- 172.21.15.67:0/3611228664 wait complete. 2024-04-17T10:34:14.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 Processor -- start 2024-04-17T10:34:14.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- start start 2024-04-17T10:34:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006e6e0 0x7ff45013f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45006eac0 0x7ff4500cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff450143ae0 0x7ff4500c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4500b3a40 con 0x7ff45006e6e0 2024-04-17T10:34:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4500b38c0 con 0x7ff450143ae0 2024-04-17T10:34:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff455400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4500b3bc0 con 0x7ff45006eac0 2024-04-17T10:34:14.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff44e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006e6e0 0x7ff45013f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff44e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006e6e0 0x7ff45013f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40062/0 (socket says 172.21.15.67:40062) 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff44e800640 1 -- 172.21.15.67:0/197513163 learned_addr learned my addr 172.21.15.67:0/197513163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff447e00640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45006eac0 0x7ff4500cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.217+0000 7ff44f200640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff450143ae0 0x7ff4500c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff44e800640 1 -- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45006eac0 msgr2=0x7ff4500cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff44e800640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45006eac0 0x7ff4500cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff44e800640 1 -- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff450143ae0 msgr2=0x7ff4500c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff44e800640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff450143ae0 0x7ff4500c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff44e800640 1 -- 172.21.15.67:0/197513163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4500c50c0 con 0x7ff45006e6e0 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff44f200640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff450143ae0 0x7ff4500c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:14.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff44e800640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006e6e0 0x7ff45013f1b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ff44000e810 tx=0x7ff44000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff446a00640 1 -- 172.21.15.67:0/197513163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440017070 con 0x7ff45006e6e0 2024-04-17T10:34:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff446a00640 1 -- 172.21.15.67:0/197513163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44000c8d0 con 0x7ff45006e6e0 2024-04-17T10:34:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45013f6f0 con 0x7ff45006e6e0 2024-04-17T10:34:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff446a00640 1 -- 172.21.15.67:0/197513163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44000f040 con 0x7ff45006e6e0 2024-04-17T10:34:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4500c5340 con 0x7ff45006e6e0 2024-04-17T10:34:14.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff450072d50 con 0x7ff45006e6e0 2024-04-17T10:34:14.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.221+0000 7ff446a00640 1 -- 172.21.15.67:0/197513163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff44001a070 con 0x7ff45006e6e0 2024-04-17T10:34:14.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.225+0000 7ff446a00640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff42c066f90 0x7ff42c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.225+0000 7ff446a00640 1 -- 172.21.15.67:0/197513163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ff440084d40 con 0x7ff45006e6e0 2024-04-17T10:34:14.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.225+0000 7ff447e00640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff42c066f90 0x7ff42c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.225+0000 7ff447e00640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff42c066f90 0x7ff42c069450 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff4380ae000 tx=0x7ff4380a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.229+0000 7ff446a00640 1 -- 172.21.15.67:0/197513163 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff4500c50c0 con 0x7ff45006e6e0 2024-04-17T10:34:14.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.393+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff450074e30 con 0x7ff45006e6e0 2024-04-17T10:34:14.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.393+0000 7ff446a00640 1 -- 172.21.15.67:0/197513163 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16904 (secure 0 0 0) 0x7ff4400554d0 con 0x7ff45006e6e0 2024-04-17T10:34:14.387 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:14.387 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":118,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:31.509307+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:33:28.333663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-17T10:33:28.871385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:14.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff42c066f90 msgr2=0x7ff42c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff42c066f90 0x7ff42c069450 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff4380ae000 tx=0x7ff4380a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:14.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006e6e0 msgr2=0x7ff45013f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006e6e0 0x7ff45013f1b0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ff44000e810 tx=0x7ff44000ecd0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 shutdown_connections 2024-04-17T10:34:14.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff42c066f90 0x7ff42c069450 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff450143ae0 0x7ff4500c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff45006eac0 0x7ff4500cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 --2- 172.21.15.67:0/197513163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff45006e6e0 0x7ff45013f1b0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 >> 172.21.15.67:0/197513163 conn(0x7ff45006ac90 msgr2=0x7ff45013f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:14.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 shutdown_connections 2024-04-17T10:34:14.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.397+0000 7ff455400640 1 -- 172.21.15.67:0/197513163 wait complete. 2024-04-17T10:34:14.409 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:34:14.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:14.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.557+0000 7fbcf6800640 1 Processor -- start 2024-04-17T10:34:14.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.557+0000 7fbcf6800640 1 -- start start 2024-04-17T10:34:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cfc00 0x7fbcf009a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf6800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf015de60 con 0x7fbcf00cc3c0 2024-04-17T10:34:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf6800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf015e0d0 con 0x7fbcf00cffe0 2024-04-17T10:34:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf6800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf006bda0 con 0x7fbcf00cfc00 2024-04-17T10:34:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf5e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbcf00cc3c0 0x7fbcf00cc830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58688/0 (socket says 172.21.15.67:58688) 2024-04-17T10:34:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf5e00640 1 -- 172.21.15.67:0/3926074141 learned_addr learned my addr 172.21.15.67:0/3926074141 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:14.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf5400640 1 --2- 172.21.15.67:0/3926074141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cfc00 0x7fbcf009a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcefe00640 1 -- 172.21.15.67:0/3926074141 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125608622 0 0) 0x7fbcf015de60 con 0x7fbcf00cc3c0 2024-04-17T10:34:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcefe00640 1 -- 172.21.15.67:0/3926074141 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd8003530 con 0x7fbcf00cc3c0 2024-04-17T10:34:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf5400640 1 -- 172.21.15.67:0/3926074141 >> v1:172.21.15.86:6789/0 conn(0x7fbcf00cffe0 legacy=0x7fbcf00cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf5400640 1 -- 172.21.15.67:0/3926074141 >> v1:172.21.15.67:6789/0 conn(0x7fbcf00cc3c0 legacy=0x7fbcf00cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.561+0000 7fbcf5400640 1 -- 172.21.15.67:0/3926074141 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf015e3b0 con 0x7fbcf00cfc00 2024-04-17T10:34:14.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf5400640 1 --2- 172.21.15.67:0/3926074141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cfc00 0x7fbcf009a450 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097f0 tx=0x7fbce401c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9671b8ddaeb7302 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcefe00640 1 -- 172.21.15.67:0/3926074141 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40b1070 con 0x7fbcf00cfc00 2024-04-17T10:34:14.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcefe00640 1 -- 172.21.15.67:0/3926074141 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40ac630 con 0x7fbcf00cfc00 2024-04-17T10:34:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcefe00640 1 -- 172.21.15.67:0/3926074141 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce40a8050 con 0x7fbcf00cfc00 2024-04-17T10:34:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 -- 172.21.15.67:0/3926074141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cfc00 msgr2=0x7fbcf009a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3926074141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cfc00 0x7fbcf009a450 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbce40097f0 tx=0x7fbce401c4e0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 -- 172.21.15.67:0/3926074141 shutdown_connections 2024-04-17T10:34:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3926074141 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cfc00 0x7fbcf009a450 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 -- 172.21.15.67:0/3926074141 >> 172.21.15.67:0/3926074141 conn(0x7fbcf00d1620 msgr2=0x7fbcf00d1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 -- 172.21.15.67:0/3926074141 shutdown_connections 2024-04-17T10:34:14.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 -- 172.21.15.67:0/3926074141 wait complete. 2024-04-17T10:34:14.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 Processor -- start 2024-04-17T10:34:14.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 -- start start 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.565+0000 7fbcf6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00cc3c0 0x7fbcf015ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf00cfc00 0x7fbcf01575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cffe0 0x7fbcf00704d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00d0540 con 0x7fbcf00cfc00 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00d03c0 con 0x7fbcf00cc3c0 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00cd350 con 0x7fbcf00cffe0 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf00cfc00 0x7fbcf01575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf00cfc00 0x7fbcf01575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40068/0 (socket says 172.21.15.67:40068) 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 -- 172.21.15.67:0/3193939653 learned_addr learned my addr 172.21.15.67:0/3193939653 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf5e00640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cffe0 0x7fbcf00704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf5400640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00cc3c0 0x7fbcf015ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 -- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cffe0 msgr2=0x7fbcf00704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cffe0 0x7fbcf00704d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 -- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00cc3c0 msgr2=0x7fbcf015ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00cc3c0 0x7fbcf015ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 -- 172.21.15.67:0/3193939653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf00b4250 con 0x7fbcf00cfc00 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf5400640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00cc3c0 0x7fbcf015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf4a00640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf00cfc00 0x7fbcf01575a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00c370 tx=0x7fbcdc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbced600640 1 -- 172.21.15.67:0/3193939653 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0052d0 con 0x7fbcf00cfc00 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbced600640 1 -- 172.21.15.67:0/3193939653 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcdc005470 con 0x7fbcf00cfc00 2024-04-17T10:34:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbced600640 1 -- 172.21.15.67:0/3193939653 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc007260 con 0x7fbcf00cfc00 2024-04-17T10:34:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf00b4530 con 0x7fbcf00cfc00 2024-04-17T10:34:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf00b4af0 con 0x7fbcf00cfc00 2024-04-17T10:34:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf0074dc0 con 0x7fbcf00cfc00 2024-04-17T10:34:14.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.569+0000 7fbced600640 1 -- 172.21.15.67:0/3193939653 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbcdc007400 con 0x7fbcf00cfc00 2024-04-17T10:34:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.573+0000 7fbced600640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd0066f90 0x7fbcd0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.573+0000 7fbced600640 1 -- 172.21.15.67:0/3193939653 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fbcdc0846a0 con 0x7fbcf00cfc00 2024-04-17T10:34:14.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.573+0000 7fbcf5400640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd0066f90 0x7fbcd0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.573+0000 7fbcf5400640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd0066f90 0x7fbcd0069450 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbce40ae000 tx=0x7fbce40a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.577+0000 7fbced600640 1 -- 172.21.15.67:0/3193939653 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbcdc055480 con 0x7fbcf00cfc00 2024-04-17T10:34:14.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcf015b7c0 con 0x7fbcf00cfc00 2024-04-17T10:34:14.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbced600640 1 -- 172.21.15.67:0/3193939653 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fbcdc054d80 con 0x7fbcf00cfc00 2024-04-17T10:34:14.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:14.764 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:14.764 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd0066f90 msgr2=0x7fbcd0069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd0066f90 0x7fbcd0069450 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fbce40ae000 tx=0x7fbce40a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf00cfc00 msgr2=0x7fbcf01575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf00cfc00 0x7fbcf01575a0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00c370 tx=0x7fbcdc00c830 comp rx=0 tx=0).stop 2024-04-17T10:34:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 shutdown_connections 2024-04-17T10:34:14.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbcd0066f90 0x7fbcd0069450 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbcf00cffe0 0x7fbcf00704d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf00cfc00 0x7fbcf01575a0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 --2- 172.21.15.67:0/3193939653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbcf00cc3c0 0x7fbcf015ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 >> 172.21.15.67:0/3193939653 conn(0x7fbcf00d1620 msgr2=0x7fbcf00d1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 shutdown_connections 2024-04-17T10:34:14.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.773+0000 7fbcf6800640 1 -- 172.21.15.67:0/3193939653 wait complete. 2024-04-17T10:34:14.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:14.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfc800640 1 Processor -- start 2024-04-17T10:34:14.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfc800640 1 -- start start 2024-04-17T10:34:14.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfc800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fadf406e6e0 0x7fadf40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfc800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fadf41441f0 0x7fadf40ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfc800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf415ae40 con 0x7fadf406eac0 2024-04-17T10:34:14.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfc800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf40cf430 con 0x7fadf41441f0 2024-04-17T10:34:14.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfc800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf4071440 con 0x7fadf406e6e0 2024-04-17T10:34:14.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadf9400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fadf406eac0 0x7fadf4143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58696/0 (socket says 172.21.15.67:58696) 2024-04-17T10:34:14.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadf9400640 1 -- 172.21.15.67:0/2248888176 learned_addr learned my addr 172.21.15.67:0/2248888176 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:14.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.933+0000 7fadfa800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fadf41441f0 0x7fadf40ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadf9e00640 1 --2- 172.21.15.67:0/2248888176 >> v2:172.21.15.86:3300/0 conn(0x7fadf406e6e0 0x7fadf40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadfa800640 1 -- 172.21.15.67:0/2248888176 >> v2:172.21.15.86:3300/0 conn(0x7fadf406e6e0 msgr2=0x7fadf40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadfa800640 1 --2- 172.21.15.67:0/2248888176 >> v2:172.21.15.86:3300/0 conn(0x7fadf406e6e0 0x7fadf40ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadfa800640 1 -- 172.21.15.67:0/2248888176 >> v1:172.21.15.67:6789/0 conn(0x7fadf406eac0 legacy=0x7fadf4143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadf8a00640 1 -- 172.21.15.67:0/2248888176 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032852267 0 0) 0x7fadf415ae40 con 0x7fadf406eac0 2024-04-17T10:34:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadfa800640 1 -- 172.21.15.67:0/2248888176 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf4071720 con 0x7fadf41441f0 2024-04-17T10:34:14.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadf9e00640 1 --2- 172.21.15.67:0/2248888176 >> v2:172.21.15.86:3300/0 conn(0x7fadf406e6e0 0x7fadf40ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadfa800640 1 --2- 172.21.15.67:0/2248888176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf40ceef0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fadf40cf620 tx=0x7fadf001c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8863781c53396389 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadf8a00640 1 -- 172.21.15.67:0/2248888176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00b2070 con 0x7fadf41441f0 2024-04-17T10:34:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadf8a00640 1 -- 172.21.15.67:0/2248888176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf00a5930 con 0x7fadf41441f0 2024-04-17T10:34:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.937+0000 7fadf8a00640 1 -- 172.21.15.67:0/2248888176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00a5d00 con 0x7fadf41441f0 2024-04-17T10:34:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- 172.21.15.67:0/2248888176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 msgr2=0x7fadf40ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 --2- 172.21.15.67:0/2248888176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf40ceef0 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fadf40cf620 tx=0x7fadf001c870 comp rx=0 tx=0).stop 2024-04-17T10:34:14.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- 172.21.15.67:0/2248888176 shutdown_connections 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 --2- 172.21.15.67:0/2248888176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf40ceef0 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 --2- 172.21.15.67:0/2248888176 >> v2:172.21.15.86:3300/0 conn(0x7fadf406e6e0 0x7fadf40ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- 172.21.15.67:0/2248888176 >> 172.21.15.67:0/2248888176 conn(0x7fadf406ac90 msgr2=0x7fadf406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- 172.21.15.67:0/2248888176 shutdown_connections 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- 172.21.15.67:0/2248888176 wait complete. 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 Processor -- start 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- start start 2024-04-17T10:34:14.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf406e6e0 0x7fadf415a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf406eac0 0x7fadf40acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf41f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40b3a40 con 0x7fadf41441f0 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40b38c0 con 0x7fadf406eac0 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfc800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40b3bc0 con 0x7fadf406e6e0 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf41f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfa800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf41f45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40082/0 (socket says 172.21.15.67:40082) 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadfa800640 1 -- 172.21.15.67:0/539077183 learned_addr learned my addr 172.21.15.67:0/539077183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:14.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadf9400640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf406eac0 0x7fadf40acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.941+0000 7fadf9e00640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf406e6e0 0x7fadf415a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfa800640 1 -- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf406e6e0 msgr2=0x7fadf415a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfa800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf406e6e0 0x7fadf415a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfa800640 1 -- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf406eac0 msgr2=0x7fadf40acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfa800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf406eac0 0x7fadf40acec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfa800640 1 -- 172.21.15.67:0/539077183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf40c4ca0 con 0x7fadf41441f0 2024-04-17T10:34:14.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfa800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf41f45d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fadf0019480 tx=0x7fadf0019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadee000640 1 -- 172.21.15.67:0/539077183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00b2050 con 0x7fadf41441f0 2024-04-17T10:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadee000640 1 -- 172.21.15.67:0/539077183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf00b3040 con 0x7fadf41441f0 2024-04-17T10:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadee000640 1 -- 172.21.15.67:0/539077183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf00ad850 con 0x7fadf41441f0 2024-04-17T10:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf40c4f80 con 0x7fadf41441f0 2024-04-17T10:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf40c5540 con 0x7fadf41441f0 2024-04-17T10:34:14.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf4072d50 con 0x7fadf41441f0 2024-04-17T10:34:14.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadee000640 1 -- 172.21.15.67:0/539077183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fadf00c0020 con 0x7fadf41441f0 2024-04-17T10:34:14.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadee000640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadd8066f60 0x7fadd8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:14.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadee000640 1 -- 172.21.15.67:0/539077183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fadf011fd30 con 0x7fadf41441f0 2024-04-17T10:34:14.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.945+0000 7fadf9e00640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadd8066f60 0x7fadd8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:14.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.949+0000 7fadf9e00640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadd8066f60 0x7fadd8069420 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fade80098b0 tx=0x7fade8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:14.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:14.953+0000 7fadee000640 1 -- 172.21.15.67:0/539077183 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fadf0016520 con 0x7fadf41441f0 2024-04-17T10:34:15.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.145+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadf4074e30 con 0x7fadf41441f0 2024-04-17T10:34:15.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.145+0000 7fadee000640 1 -- 172.21.15.67:0/539077183 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4934 (secure 0 0 0) 0x7fadf00f0c30 con 0x7fadf41441f0 2024-04-17T10:34:15.139 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:15.140 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":82,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:33:29.865690+0000","modified":"2024-04-17T10:33:32.930770+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8670},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8670":{"gid":8670,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/397092831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":397092831},{"type":"v1","addr":"172.21.15.67:6834","nonce":397092831}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-04-17T10:34:15.140 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 82 2024-04-17T10:34:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadd8066f60 msgr2=0x7fadd8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadd8066f60 0x7fadd8069420 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fade80098b0 tx=0x7fade8008040 comp rx=0 tx=0).stop 2024-04-17T10:34:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 msgr2=0x7fadf41f45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf41f45d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fadf0019480 tx=0x7fadf0019610 comp rx=0 tx=0).stop 2024-04-17T10:34:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 shutdown_connections 2024-04-17T10:34:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fadd8066f60 0x7fadd8069420 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fadf41441f0 0x7fadf41f45d0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fadf406eac0 0x7fadf40acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 --2- 172.21.15.67:0/539077183 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fadf406e6e0 0x7fadf415a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 >> 172.21.15.67:0/539077183 conn(0x7fadf406ac90 msgr2=0x7fadf4142d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:15.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 shutdown_connections 2024-04-17T10:34:15.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.149+0000 7fadfc800640 1 -- 172.21.15.67:0/539077183 wait complete. 2024-04-17T10:34:15.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.309+0000 7f4ebae00640 1 Processor -- start 2024-04-17T10:34:15.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.309+0000 7f4ebae00640 1 -- start start 2024-04-17T10:34:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4ebae00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4eb406b6a0 0x7f4eb40cc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4ebae00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb415cf10 con 0x7f4eb40cff90 2024-04-17T10:34:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4ebae00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb415c300 con 0x7f4eb415b970 2024-04-17T10:34:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4ebae00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb415c5e0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eb8e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4eb40cff90 0x7f4eb40ccbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58700/0 (socket says 172.21.15.67:58700) 2024-04-17T10:34:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eb8e00640 1 -- 172.21.15.67:0/644508148 learned_addr learned my addr 172.21.15.67:0/644508148 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eabe00640 1 --2- 172.21.15.67:0/644508148 >> v2:172.21.15.86:3300/0 conn(0x7f4eb406b6a0 0x7f4eb40cc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eb3400640 1 -- 172.21.15.67:0/644508148 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880221371 0 0) 0x7f4eb415cf10 con 0x7f4eb40cff90 2024-04-17T10:34:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eb3400640 1 -- 172.21.15.67:0/644508148 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e98003530 con 0x7f4eb40cff90 2024-04-17T10:34:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eabe00640 1 -- 172.21.15.67:0/644508148 >> v1:172.21.15.86:6789/0 conn(0x7f4eb415b970 legacy=0x7f4eb40cf880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eabe00640 1 -- 172.21.15.67:0/644508148 >> v1:172.21.15.67:6789/0 conn(0x7f4eb40cff90 legacy=0x7f4eb40ccbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eb3400640 1 -- 172.21.15.67:0/644508148 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081767156 0 0) 0x7f4eb415c300 con 0x7f4eb415b970 2024-04-17T10:34:15.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.313+0000 7f4eabe00640 1 -- 172.21.15.67:0/644508148 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb415c8c0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4eabe00640 1 --2- 172.21.15.67:0/644508148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb406b6a0 0x7f4eb40cc680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0097c0 tx=0x7f4eb40cdae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d8f953224da687c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:15.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4eb3400640 1 -- 172.21.15.67:0/644508148 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c0b0070 con 0x7f4eb406b6a0 2024-04-17T10:34:15.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4eb3400640 1 -- 172.21.15.67:0/644508148 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c0ab600 con 0x7f4eb406b6a0 2024-04-17T10:34:15.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4eb3400640 1 -- 172.21.15.67:0/644508148 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c0ab9d0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- 172.21.15.67:0/644508148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb406b6a0 msgr2=0x7f4eb40cc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 --2- 172.21.15.67:0/644508148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb406b6a0 0x7f4eb40cc680 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0097c0 tx=0x7f4eb40cdae0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- 172.21.15.67:0/644508148 shutdown_connections 2024-04-17T10:34:15.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 --2- 172.21.15.67:0/644508148 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb406b6a0 0x7f4eb40cc680 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- 172.21.15.67:0/644508148 >> 172.21.15.67:0/644508148 conn(0x7f4eb40b22a0 msgr2=0x7f4eb40b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:15.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- 172.21.15.67:0/644508148 shutdown_connections 2024-04-17T10:34:15.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- 172.21.15.67:0/644508148 wait complete. 2024-04-17T10:34:15.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 Processor -- start 2024-04-17T10:34:15.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- start start 2024-04-17T10:34:15.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4eb406b6a0 0x7f4eb41537c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40cff90 0x7f4eb4152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eb415b970 0x7f4eb406f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb4169920 con 0x7f4eb406b6a0 2024-04-17T10:34:15.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41697a0 con 0x7f4eb40cff90 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.317+0000 7f4ebae00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb4169aa0 con 0x7f4eb415b970 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4eb406b6a0 0x7f4eb41537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4eb406b6a0 0x7f4eb41537c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40086/0 (socket says 172.21.15.67:40086) 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 -- 172.21.15.67:0/104877847 learned_addr learned my addr 172.21.15.67:0/104877847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eabe00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40cff90 0x7f4eb4152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb8e00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eb415b970 0x7f4eb406f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 -- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eb415b970 msgr2=0x7f4eb406f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eb415b970 0x7f4eb406f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 -- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40cff90 msgr2=0x7f4eb4152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40cff90 0x7f4eb4152f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 -- 172.21.15.67:0/104877847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb40700d0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb3e00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4eb406b6a0 0x7f4eb41537c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4001ec0 tx=0x7f4ea400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb0c00640 1 -- 172.21.15.67:0/104877847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4017070 con 0x7f4eb406b6a0 2024-04-17T10:34:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb0c00640 1 -- 172.21.15.67:0/104877847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea4003600 con 0x7f4eb406b6a0 2024-04-17T10:34:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb40703b0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb0c00640 1 -- 172.21.15.67:0/104877847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea40039f0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb40ceeb0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb409e900 con 0x7f4eb406b6a0 2024-04-17T10:34:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.321+0000 7f4eb0c00640 1 -- 172.21.15.67:0/104877847 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4ea401a070 con 0x7f4eb406b6a0 2024-04-17T10:34:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.325+0000 7f4eb0c00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e90066f60 0x7f4e90069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.325+0000 7f4eb0c00640 1 -- 172.21.15.67:0/104877847 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4ea40845e0 con 0x7f4eb406b6a0 2024-04-17T10:34:15.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.325+0000 7f4eabe00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e90066f60 0x7f4e90069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.325+0000 7f4eabe00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e90066f60 0x7f4e90069420 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0ad000 tx=0x7f4e9c004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:15.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.329+0000 7f4eb0c00640 1 -- 172.21.15.67:0/104877847 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4ea4055450 con 0x7f4eb406b6a0 2024-04-17T10:34:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.489+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4eb4074e30 con 0x7f4eb406b6a0 2024-04-17T10:34:15.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.489+0000 7f4eb0c00640 1 -- 172.21.15.67:0/104877847 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v118) v1 ==== 74+0+16904 (secure 0 0 0) 0x7f4ea4054d50 con 0x7f4eb406b6a0 2024-04-17T10:34:15.483 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:15.483 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":118,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:33:31.509307+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:33:28.333663+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"117","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-04-17T10:33:28.871385+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"118","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e90066f60 msgr2=0x7f4e90069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e90066f60 0x7f4e90069420 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c0ad000 tx=0x7f4e9c004740 comp rx=0 tx=0).stop 2024-04-17T10:34:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4eb406b6a0 msgr2=0x7f4eb41537c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4eb406b6a0 0x7f4eb41537c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4001ec0 tx=0x7f4ea400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 shutdown_connections 2024-04-17T10:34:15.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4e90066f60 0x7f4e90069420 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4eb415b970 0x7f4eb406f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4eb40cff90 0x7f4eb4152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 --2- 172.21.15.67:0/104877847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4eb406b6a0 0x7f4eb41537c0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 >> 172.21.15.67:0/104877847 conn(0x7f4eb40b22a0 msgr2=0x7f4eb40b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 shutdown_connections 2024-04-17T10:34:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.493+0000 7f4ebae00640 1 -- 172.21.15.67:0/104877847 wait complete. 2024-04-17T10:34:15.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:34:15.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.653+0000 7fb80cc00640 1 Processor -- start 2024-04-17T10:34:15.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.653+0000 7fb80cc00640 1 -- start start 2024-04-17T10:34:15.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb80cc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb8080ce020 0x7fb8081620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb80cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8080ce400 0x7fb80815e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb80cc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80815eb50 con 0x7fb80815e550 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb80cc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb808161650 con 0x7fb8080ce400 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb80cc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8081625e0 con 0x7fb8080ce020 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb7fd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb8080ce400 0x7fb80815e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb806800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb80815e550 0x7fb8080cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58706/0 (socket says 172.21.15.67:58706) 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb806800640 1 -- 172.21.15.67:0/3813379287 learned_addr learned my addr 172.21.15.67:0/3813379287 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb805e00640 1 --2- 172.21.15.67:0/3813379287 >> v2:172.21.15.86:3300/0 conn(0x7fb8080ce020 0x7fb8081620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb7fd400640 1 -- 172.21.15.67:0/3813379287 >> v2:172.21.15.86:3300/0 conn(0x7fb8080ce020 msgr2=0x7fb8081620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb7fd400640 1 --2- 172.21.15.67:0/3813379287 >> v2:172.21.15.86:3300/0 conn(0x7fb8080ce020 0x7fb8081620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb7fd400640 1 -- 172.21.15.67:0/3813379287 >> v1:172.21.15.67:6789/0 conn(0x7fb80815e550 legacy=0x7fb8080cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.657+0000 7fb7fd400640 1 -- 172.21.15.67:0/3813379287 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80815ee30 con 0x7fb8080ce400 2024-04-17T10:34:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb7fd400640 1 --2- 172.21.15.67:0/3813379287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8080ce400 0x7fb80815e010 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00097c0 tx=0x7fb7f001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3b24ba58e67728 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb805400640 1 -- 172.21.15.67:0/3813379287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00b1070 con 0x7fb8080ce400 2024-04-17T10:34:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb805400640 1 -- 172.21.15.67:0/3813379287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f00ac640 con 0x7fb8080ce400 2024-04-17T10:34:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb805400640 1 -- 172.21.15.67:0/3813379287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00aca10 con 0x7fb8080ce400 2024-04-17T10:34:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 -- 172.21.15.67:0/3813379287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8080ce400 msgr2=0x7fb80815e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3813379287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8080ce400 0x7fb80815e010 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7fb7f00097c0 tx=0x7fb7f001c420 comp rx=0 tx=0).stop 2024-04-17T10:34:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 -- 172.21.15.67:0/3813379287 shutdown_connections 2024-04-17T10:34:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3813379287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8080ce400 0x7fb80815e010 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3813379287 >> v2:172.21.15.86:3300/0 conn(0x7fb8080ce020 0x7fb8081620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 -- 172.21.15.67:0/3813379287 >> 172.21.15.67:0/3813379287 conn(0x7fb8080b33f0 msgr2=0x7fb8080b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:15.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 -- 172.21.15.67:0/3813379287 shutdown_connections 2024-04-17T10:34:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 -- 172.21.15.67:0/3813379287 wait complete. 2024-04-17T10:34:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 Processor -- start 2024-04-17T10:34:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.661+0000 7fb80cc00640 1 -- start start 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8080ce020 0x7fb808144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8080ce400 0x7fb808152fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80815e550 0x7fb808154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80806c0f0 con 0x7fb80815e550 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80806bf70 con 0x7fb8080ce400 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080c84b0 con 0x7fb8080ce020 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80815e550 0x7fb808154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80815e550 0x7fb808154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40106/0 (socket says 172.21.15.67:40106) 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 -- 172.21.15.67:0/3159853076 learned_addr learned my addr 172.21.15.67:0/3159853076 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb7fd400640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8080ce400 0x7fb808152fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb805e00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8080ce020 0x7fb808144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 -- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8080ce020 msgr2=0x7fb808144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8080ce020 0x7fb808144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 -- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8080ce400 msgr2=0x7fb808152fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8080ce400 0x7fb808152fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 -- 172.21.15.67:0/3159853076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808156e60 con 0x7fb80815e550 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb806800640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80815e550 0x7fb808154e60 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c3c0 tx=0x7fb80000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb7fea00640 1 -- 172.21.15.67:0/3159853076 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb800017070 con 0x7fb80815e550 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb7fea00640 1 -- 172.21.15.67:0/3159853076 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb800005cf0 con 0x7fb80815e550 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb808157140 con 0x7fb80815e550 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb7fea00640 1 -- 172.21.15.67:0/3159853076 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8000072f0 con 0x7fb80815e550 2024-04-17T10:34:15.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80815a150 con 0x7fb80815e550 2024-04-17T10:34:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb808070ab0 con 0x7fb80815e550 2024-04-17T10:34:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.665+0000 7fb7fea00640 1 -- 172.21.15.67:0/3159853076 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb80001a070 con 0x7fb80815e550 2024-04-17T10:34:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.669+0000 7fb7fea00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7e4066f60 0x7fb7e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:15.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.669+0000 7fb7fea00640 1 -- 172.21.15.67:0/3159853076 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb800084470 con 0x7fb80815e550 2024-04-17T10:34:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.669+0000 7fb805e00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7e4066f60 0x7fb7e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:15.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.669+0000 7fb805e00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7e4066f60 0x7fb7e4069420 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003b90 tx=0x7fb7f80098d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:15.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.673+0000 7fb7fea00640 1 -- 172.21.15.67:0/3159853076 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb800054c90 con 0x7fb80815e550 2024-04-17T10:34:15.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.865+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb8081687c0 con 0x7fb80815e550 2024-04-17T10:34:15.859 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:34:15.869+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:34:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb7fea00640 1 -- 172.21.15.67:0/3159853076 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb800058d30 con 0x7fb80815e550 2024-04-17T10:34:15.863 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:34:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7e4066f60 msgr2=0x7fb7e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7e4066f60 0x7fb7e4069420 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003b90 tx=0x7fb7f80098d0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80815e550 msgr2=0x7fb808154e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:15.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80815e550 0x7fb808154e60 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7fb80000c3c0 tx=0x7fb80000c880 comp rx=0 tx=0).stop 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 shutdown_connections 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb7e4066f60 0x7fb7e4069420 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb80815e550 0x7fb808154e60 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8080ce400 0x7fb808152fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 --2- 172.21.15.67:0/3159853076 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8080ce020 0x7fb808144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 >> 172.21.15.67:0/3159853076 conn(0x7fb8080b33f0 msgr2=0x7fb8080d0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 shutdown_connections 2024-04-17T10:34:15.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:15.873+0000 7fb80cc00640 1 -- 172.21.15.67:0/3159853076 wait complete. 2024-04-17T10:34:15.889 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:34:15.933 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:34:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.041+0000 7fd243e00640 1 Processor -- start 2024-04-17T10:34:16.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.041+0000 7fd243e00640 1 -- start start 2024-04-17T10:34:16.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd243e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd23c072a90 0x7fd23c0b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:16.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd243e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c0b5130 0x7fd23c06ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:16.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd243e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23c15f360 con 0x7fd23c1583f0 2024-04-17T10:34:16.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd243e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd23c0b3160 con 0x7fd23c072a90 2024-04-17T10:34:16.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd243e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c0b3440 con 0x7fd23c0b5130 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd241e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c0b5130 0x7fd23c06ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd240a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd23c072a90 0x7fd23c0b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd241e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c0b5130 0x7fd23c06ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:37878/0 (socket says 172.21.15.67:37878) 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd241e00640 1 -- 172.21.15.67:0/1299880162 learned_addr learned my addr 172.21.15.67:0/1299880162 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd241e00640 1 -- 172.21.15.67:0/1299880162 >> v2:172.21.15.86:3300/0 conn(0x7fd23c072a90 msgr2=0x7fd23c0b4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd241e00640 1 --2- 172.21.15.67:0/1299880162 >> v2:172.21.15.86:3300/0 conn(0x7fd23c072a90 0x7fd23c0b4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd241e00640 1 -- 172.21.15.67:0/1299880162 >> v1:172.21.15.86:6789/0 conn(0x7fd23c1583f0 legacy=0x7fd23c072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:16.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.045+0000 7fd241e00640 1 -- 172.21.15.67:0/1299880162 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c0b3720 con 0x7fd23c0b5130 2024-04-17T10:34:16.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd241e00640 1 --2- 172.21.15.67:0/1299880162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c0b5130 0x7fd23c06ac90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd234009e50 tx=0x7fd23401ca00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ec6a9f8ae67fe9c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:16.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd23be00640 1 -- 172.21.15.67:0/1299880162 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340a58d0 con 0x7fd23c0b5130 2024-04-17T10:34:16.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd23be00640 1 -- 172.21.15.67:0/1299880162 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2340a5a70 con 0x7fd23c0b5130 2024-04-17T10:34:16.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd23be00640 1 -- 172.21.15.67:0/1299880162 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340a5e40 con 0x7fd23c0b5130 2024-04-17T10:34:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- 172.21.15.67:0/1299880162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c0b5130 msgr2=0x7fd23c06ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 --2- 172.21.15.67:0/1299880162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c0b5130 0x7fd23c06ac90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd234009e50 tx=0x7fd23401ca00 comp rx=0 tx=0).stop 2024-04-17T10:34:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- 172.21.15.67:0/1299880162 shutdown_connections 2024-04-17T10:34:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 --2- 172.21.15.67:0/1299880162 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c0b5130 0x7fd23c06ac90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 --2- 172.21.15.67:0/1299880162 >> v2:172.21.15.86:3300/0 conn(0x7fd23c072a90 0x7fd23c0b4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- 172.21.15.67:0/1299880162 >> 172.21.15.67:0/1299880162 conn(0x7fd23c0b2570 msgr2=0x7fd23c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:16.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- 172.21.15.67:0/1299880162 shutdown_connections 2024-04-17T10:34:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- 172.21.15.67:0/1299880162 wait complete. 2024-04-17T10:34:16.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 Processor -- start 2024-04-17T10:34:16.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- start start 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c072a90 0x7fd23c146b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd23c0b5130 0x7fd23c14d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c1583f0 0x7fd23c154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c0b5550 con 0x7fd23c1583f0 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c0b4200 con 0x7fd23c0b5130 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd243e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23c06b9e0 con 0x7fd23c072a90 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd241e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c1583f0 0x7fd23c154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:16.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.049+0000 7fd240a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd23c0b5130 0x7fd23c14d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd241400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c072a90 0x7fd23c146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd240a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd23c0b5130 0x7fd23c14d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:52682/0 (socket says 172.21.15.67:52682) 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd240a00640 1 -- 172.21.15.67:0/2864109111 learned_addr learned my addr 172.21.15.67:0/2864109111 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd241e00640 1 -- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c072a90 msgr2=0x7fd23c146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd241e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c072a90 0x7fd23c146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd241e00640 1 -- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd23c0b5130 msgr2=0x7fd23c14d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd241e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd23c0b5130 0x7fd23c14d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd241e00640 1 -- 172.21.15.67:0/2864109111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c1567f0 con 0x7fd23c1583f0 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd240a00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd23c0b5130 0x7fd23c14d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd241e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c1583f0 0x7fd23c154330 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fd2340ab000 tx=0x7fd2340197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd239600640 1 -- 172.21.15.67:0/2864109111 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340bc030 con 0x7fd23c1583f0 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23c155750 con 0x7fd23c1583f0 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd239600640 1 -- 172.21.15.67:0/2864109111 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd234007c30 con 0x7fd23c1583f0 2024-04-17T10:34:16.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd239600640 1 -- 172.21.15.67:0/2864109111 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340a58d0 con 0x7fd23c1583f0 2024-04-17T10:34:16.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23c155d10 con 0x7fd23c1583f0 2024-04-17T10:34:16.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23c04ef30 con 0x7fd23c1583f0 2024-04-17T10:34:16.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.053+0000 7fd239600640 1 -- 172.21.15.67:0/2864109111 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd2340bf070 con 0x7fd23c1583f0 2024-04-17T10:34:16.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.057+0000 7fd239600640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd21c066f60 0x7fd21c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:16.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.057+0000 7fd239600640 1 -- 172.21.15.67:0/2864109111 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd23411fe50 con 0x7fd23c1583f0 2024-04-17T10:34:16.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.057+0000 7fd241400640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd21c066f60 0x7fd21c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.057+0000 7fd241400640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd21c066f60 0x7fd21c069420 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd23c1490e0 tx=0x7fd230009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:16.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.061+0000 7fd239600640 1 -- 172.21.15.67:0/2864109111 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd2340c7020 con 0x7fd23c1583f0 2024-04-17T10:34:16.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.257+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd23c15de90 con 0x7fd23c1583f0 2024-04-17T10:34:16.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.877+0000 7fd239600640 1 -- 172.21.15.67:0/2864109111 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v84) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd234016620 con 0x7fd23c1583f0 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd21c066f60 msgr2=0x7fd21c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd21c066f60 0x7fd21c069420 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd23c1490e0 tx=0x7fd230009040 comp rx=0 tx=0).stop 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c1583f0 msgr2=0x7fd23c154330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c1583f0 0x7fd23c154330 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fd2340ab000 tx=0x7fd2340197a0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 shutdown_connections 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd21c066f60 0x7fd21c069420 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd23c1583f0 0x7fd23c154330 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd23c0b5130 0x7fd23c14d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 --2- 172.21.15.67:0/2864109111 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd23c072a90 0x7fd23c146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:16.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 >> 172.21.15.67:0/2864109111 conn(0x7fd23c0b2570 msgr2=0x7fd23c0ae060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:16.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 shutdown_connections 2024-04-17T10:34:16.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:16.881+0000 7fd243e00640 1 -- 172.21.15.67:0/2864109111 wait complete. 2024-04-17T10:34:16.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:34:17.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.093+0000 7f306d000640 1 Processor -- start 2024-04-17T10:34:17.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.093+0000 7f306d000640 1 -- start start 2024-04-17T10:34:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f306d000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f306806d3c0 0x7f3068071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f306d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3068071cb0 con 0x7f30681447e0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f306d000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306806d9a0 con 0x7f30680718a0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f306d000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30680cc310 con 0x7f306806d3c0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f30681447e0 0x7f3068144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58736/0 (socket says 172.21.15.67:58736) 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 -- 172.21.15.67:0/1453869168 learned_addr learned my addr 172.21.15.67:0/1453869168 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f305d400640 1 --2- 172.21.15.67:0/1453869168 >> v2:172.21.15.86:3300/0 conn(0x7f306806d3c0 0x7f3068071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152708920 0 0) 0x7f3058001120 con 0x7f30681447e0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f304c003530 con 0x7f30681447e0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1215500195 0 0) 0x7f304c003530 con 0x7f30681447e0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 >> v2:172.21.15.86:3300/0 conn(0x7f306806d3c0 msgr2=0x7f3068071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 --2- 172.21.15.67:0/1453869168 >> v2:172.21.15.86:3300/0 conn(0x7f306806d3c0 0x7f3068071360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 >> v1:172.21.15.86:6789/0 conn(0x7f30680718a0 legacy=0x7f30680cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306806dc80 con 0x7f30681447e0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f3058001850 con 0x7f30681447e0 2024-04-17T10:34:17.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f305d400640 1 --2- 172.21.15.67:0/1453869168 >> v2:172.21.15.86:3300/0 conn(0x7f306806d3c0 0x7f3068071360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 >> v1:172.21.15.67:6789/0 conn(0x7f30681447e0 legacy=0x7f3068144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c0039c0 0x7f304c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c004810 0x7f304c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304c0051d0 0x7f304c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680cc310 con 0x7f304c0039c0 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c00c170 con 0x7f304c004810 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c00c370 con 0x7f304c0051d0 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c0039c0 0x7f304c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f305d400640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c004810 0x7f304c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3066800640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304c0051d0 0x7f304c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 -- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304c0051d0 msgr2=0x7f304c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:17.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304c0051d0 0x7f304c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 -- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c004810 msgr2=0x7f304c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:17.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c004810 0x7f304c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3065e00640 1 -- 172.21.15.67:0/1453869168 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c003530 con 0x7f304c0039c0 2024-04-17T10:34:17.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f305d400640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c004810 0x7f304c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:17.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3066800640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f304c0051d0 0x7f304c00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:17.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.097+0000 7f3066800640 1 -- 172.21.15.67:0/1453869168 reap_dead start 2024-04-17T10:34:17.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f3065e00640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c0039c0 0x7f304c003de0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f305800c4c0 tx=0x7f305801cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ed15b25ea2a226e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:17.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305800c0e0 con 0x7f304c0039c0 2024-04-17T10:34:17.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30580a9070 con 0x7f304c0039c0 2024-04-17T10:34:17.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f3065400640 1 -- 172.21.15.67:0/1453869168 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3058005d80 con 0x7f304c0039c0 2024-04-17T10:34:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f306d000640 1 -- 172.21.15.67:0/1453869168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c0039c0 msgr2=0x7f304c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f306d000640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c0039c0 0x7f304c003de0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f305800c4c0 tx=0x7f305801cc70 comp rx=0 tx=0).stop 2024-04-17T10:34:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f306d000640 1 -- 172.21.15.67:0/1453869168 shutdown_connections 2024-04-17T10:34:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f306d000640 1 --2- 172.21.15.67:0/1453869168 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c0039c0 0x7f304c003de0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f306d000640 1 -- 172.21.15.67:0/1453869168 >> 172.21.15.67:0/1453869168 conn(0x7f306806ac90 msgr2=0x7f306806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f306d000640 1 -- 172.21.15.67:0/1453869168 shutdown_connections 2024-04-17T10:34:17.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.101+0000 7f306d000640 1 -- 172.21.15.67:0/1453869168 wait complete. 2024-04-17T10:34:17.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 Processor -- start 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 -- start start 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c0039c0 0x7f306813cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3068154540 0x7f306813ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f306813e540 0x7f306813b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306815fca0 con 0x7f306813e540 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3068145520 con 0x7f304c0039c0 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306815fe20 con 0x7f3068154540 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f306813e540 0x7f306813b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f306813e540 0x7f306813b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40132/0 (socket says 172.21.15.67:40132) 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 -- 172.21.15.67:0/1679300129 learned_addr learned my addr 172.21.15.67:0/1679300129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f305d400640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3068154540 0x7f306813ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3065e00640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c0039c0 0x7f306813cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 -- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3068154540 msgr2=0x7f306813ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3068154540 0x7f306813ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 -- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c0039c0 msgr2=0x7f306813cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c0039c0 0x7f306813cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 -- 172.21.15.67:0/1679300129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306813d220 con 0x7f306813e540 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3065e00640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c0039c0 0x7f306813cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f3066800640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f306813e540 0x7f306813b630 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f306806f870 tx=0x7f306000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f305ea00640 1 -- 172.21.15.67:0/1679300129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060003980 con 0x7f306813e540 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f305ea00640 1 -- 172.21.15.67:0/1679300129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3060018070 con 0x7f306813e540 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f305ea00640 1 -- 172.21.15.67:0/1679300129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060003d60 con 0x7f306813e540 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306813b3d0 con 0x7f306813e540 2024-04-17T10:34:17.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306813d780 con 0x7f306813e540 2024-04-17T10:34:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.105+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3068072da0 con 0x7f306813e540 2024-04-17T10:34:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.109+0000 7f305ea00640 1 -- 172.21.15.67:0/1679300129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f30600050d0 con 0x7f306813e540 2024-04-17T10:34:17.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.109+0000 7f305ea00640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3044066f90 0x7f3044069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:17.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.109+0000 7f305ea00640 1 -- 172.21.15.67:0/1679300129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3060084cc0 con 0x7f306813e540 2024-04-17T10:34:17.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.109+0000 7f3065e00640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3044066f90 0x7f3044069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:17.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.109+0000 7f3065e00640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3044066f90 0x7f3044069450 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f305800c490 tx=0x7f30580a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:17.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.113+0000 7f305ea00640 1 -- 172.21.15.67:0/1679300129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f306008b080 con 0x7f306813e540 2024-04-17T10:34:17.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.277+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f30680cd460 con 0x7f306813e540 2024-04-17T10:34:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f305ea00640 1 -- 172.21.15.67:0/1679300129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v120) v1 ==== 181+0+0 (secure 0 0 0) 0x7f3060022020 con 0x7f306813e540 2024-04-17T10:34:17.872 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3044066f90 msgr2=0x7f3044069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3044066f90 0x7f3044069450 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f305800c490 tx=0x7f30580a7070 comp rx=0 tx=0).stop 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f306813e540 msgr2=0x7f306813b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f306813e540 0x7f306813b630 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f306806f870 tx=0x7f306000eb90 comp rx=0 tx=0).stop 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 shutdown_connections 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3044066f90 0x7f3044069450 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f306813e540 0x7f306813b630 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3068154540 0x7f306813ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 --2- 172.21.15.67:0/1679300129 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f304c0039c0 0x7f306813cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.881+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 >> 172.21.15.67:0/1679300129 conn(0x7f306806ac90 msgr2=0x7f30680cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:17.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.885+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 shutdown_connections 2024-04-17T10:34:17.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:17.885+0000 7f306d000640 1 -- 172.21.15.67:0/1679300129 wait complete. 2024-04-17T10:34:17.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:34:18.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc600e00640 1 Processor -- start 2024-04-17T10:34:18.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc600e00640 1 -- start start 2024-04-17T10:34:18.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc600e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc0b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:18.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc600e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc0adde0 con 0x7fc5fc0ae880 2024-04-17T10:34:18.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc600e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc161d90 con 0x7fc5fc1440e0 2024-04-17T10:34:18.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc600e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc162060 con 0x7fc5fc0b2800 2024-04-17T10:34:18.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5fa800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5fc0ae880 0x7fc5fc0b42f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:58742/0 (socket says 172.21.15.67:58742) 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5fa800640 1 -- 172.21.15.67:0/1934458706 learned_addr learned my addr 172.21.15.67:0/1934458706 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5f1400640 1 --2- 172.21.15.67:0/1934458706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc0b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5f9400640 1 -- 172.21.15.67:0/1934458706 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739369484 0 0) 0x7fc5fc0adde0 con 0x7fc5fc0ae880 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5f9400640 1 -- 172.21.15.67:0/1934458706 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5e0003530 con 0x7fc5fc0ae880 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5f1400640 1 -- 172.21.15.67:0/1934458706 >> v1:172.21.15.86:6789/0 conn(0x7fc5fc1440e0 legacy=0x7fc5fc161680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5f9400640 1 -- 172.21.15.67:0/1934458706 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131074575 0 0) 0x7fc5e0003530 con 0x7fc5fc0ae880 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5f1400640 1 -- 172.21.15.67:0/1934458706 >> v1:172.21.15.67:6789/0 conn(0x7fc5fc0ae880 legacy=0x7fc5fc0b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.057+0000 7fc5f1400640 1 -- 172.21.15.67:0/1934458706 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc162340 con 0x7fc5fc0b2800 2024-04-17T10:34:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc5f1400640 1 --2- 172.21.15.67:0/1934458706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc0b2c60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40097f0 tx=0x7fc5e401c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acda1aae92b5c06f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc5f9400640 1 -- 172.21.15.67:0/1934458706 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e40b0070 con 0x7fc5fc0b2800 2024-04-17T10:34:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc5f9400640 1 -- 172.21.15.67:0/1934458706 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e40ab400 con 0x7fc5fc0b2800 2024-04-17T10:34:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc5f9400640 1 -- 172.21.15.67:0/1934458706 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4001a00 con 0x7fc5fc0b2800 2024-04-17T10:34:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc600e00640 1 -- 172.21.15.67:0/1934458706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 msgr2=0x7fc5fc0b2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc600e00640 1 --2- 172.21.15.67:0/1934458706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc0b2c60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40097f0 tx=0x7fc5e401c400 comp rx=0 tx=0).stop 2024-04-17T10:34:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc600e00640 1 -- 172.21.15.67:0/1934458706 shutdown_connections 2024-04-17T10:34:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc600e00640 1 --2- 172.21.15.67:0/1934458706 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc0b2c60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc600e00640 1 -- 172.21.15.67:0/1934458706 >> 172.21.15.67:0/1934458706 conn(0x7fc5fc06ac90 msgr2=0x7fc5fc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc600e00640 1 -- 172.21.15.67:0/1934458706 shutdown_connections 2024-04-17T10:34:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.061+0000 7fc600e00640 1 -- 172.21.15.67:0/1934458706 wait complete. 2024-04-17T10:34:18.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 Processor -- start 2024-04-17T10:34:18.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 -- start start 2024-04-17T10:34:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5fc0ae880 0x7fc5fc15ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc136340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5fc1440e0 0x7fc5fc0afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc06e750 con 0x7fc5fc0ae880 2024-04-17T10:34:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc06e5d0 con 0x7fc5fc1440e0 2024-04-17T10:34:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc0b4a00 con 0x7fc5fc0b2800 2024-04-17T10:34:18.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5fc0ae880 0x7fc5fc15ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5fc0ae880 0x7fc5fc15ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40148/0 (socket says 172.21.15.67:40148) 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 -- 172.21.15.67:0/2810855118 learned_addr learned my addr 172.21.15.67:0/2810855118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5fa800640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5fc1440e0 0x7fc5fc0afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f1400640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc136340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 -- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 msgr2=0x7fc5fc136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc136340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 -- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5fc1440e0 msgr2=0x7fc5fc0afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5fc1440e0 0x7fc5fc0afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 -- 172.21.15.67:0/2810855118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc06df20 con 0x7fc5fc0ae880 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f9e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5fc0ae880 0x7fc5fc15ebb0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00c3c0 tx=0x7fc5ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f2a00640 1 -- 172.21.15.67:0/2810855118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec017070 con 0x7fc5fc0ae880 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f2a00640 1 -- 172.21.15.67:0/2810855118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec0052d0 con 0x7fc5fc0ae880 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc5f2a00640 1 -- 172.21.15.67:0/2810855118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec003840 con 0x7fc5fc0ae880 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc06d960 con 0x7fc5fc0ae880 2024-04-17T10:34:18.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.065+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc15e0d0 con 0x7fc5fc0ae880 2024-04-17T10:34:18.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.069+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc072da0 con 0x7fc5fc0ae880 2024-04-17T10:34:18.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.069+0000 7fc5f2a00640 1 -- 172.21.15.67:0/2810855118 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc5ec01a070 con 0x7fc5fc0ae880 2024-04-17T10:34:18.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.069+0000 7fc5f2a00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc5d8066f60 0x7fc5d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:18.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.069+0000 7fc5f2a00640 1 -- 172.21.15.67:0/2810855118 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 5749+0+0 (secure 0 0 0) 0x7fc5ec084330 con 0x7fc5fc0ae880 2024-04-17T10:34:18.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.069+0000 7fc5f1400640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc5d8066f60 0x7fc5d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:18.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.069+0000 7fc5f1400640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc5d8066f60 0x7fc5d8069420 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40ad000 tx=0x7fc5e4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:18.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.073+0000 7fc5f2a00640 1 -- 172.21.15.67:0/2810855118 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc5ec005480 con 0x7fc5fc0ae880 2024-04-17T10:34:18.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.237+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc5fc06b480 con 0x7fc5fc0ae880 2024-04-17T10:34:18.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.885+0000 7fc5f2a00640 1 -- 172.21.15.67:0/2810855118 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v121) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc5ec054c20 con 0x7fc5fc0ae880 2024-04-17T10:34:18.879 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:34:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc5d8066f60 msgr2=0x7fc5d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc5d8066f60 0x7fc5d8069420 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc5e40ad000 tx=0x7fc5e4008040 comp rx=0 tx=0).stop 2024-04-17T10:34:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5fc0ae880 msgr2=0x7fc5fc15ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:18.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5fc0ae880 0x7fc5fc15ebb0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00c3c0 tx=0x7fc5ec00c880 comp rx=0 tx=0).stop 2024-04-17T10:34:18.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 shutdown_connections 2024-04-17T10:34:18.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc5d8066f60 0x7fc5d8069420 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:18.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc5fc1440e0 0x7fc5fc0afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:18.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc5fc0b2800 0x7fc5fc136340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:18.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 --2- 172.21.15.67:0/2810855118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5fc0ae880 0x7fc5fc15ebb0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:18.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.893+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 >> 172.21.15.67:0/2810855118 conn(0x7fc5fc06ac90 msgr2=0x7fc5fc13b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.897+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 shutdown_connections 2024-04-17T10:34:18.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:18.897+0000 7fc600e00640 1 -- 172.21.15.67:0/2810855118 wait complete. 2024-04-17T10:34:18.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:34:19.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec2e00640 1 Processor -- start 2024-04-17T10:34:19.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec2e00640 1 -- start start 2024-04-17T10:34:19.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec2e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ebc15dd60 0x7f6ebc161880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec2e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0d06a0 0x7f6ebc161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec2e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ebc15e140 con 0x7f6ebc09adf0 2024-04-17T10:34:19.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec2e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ebc1626d0 con 0x7f6ebc15dd60 2024-04-17T10:34:19.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec2e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc099d50 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec0e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0d06a0 0x7f6ebc161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ebbe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ebc15dd60 0x7f6ebc161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec0e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0d06a0 0x7f6ebc161dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:37924/0 (socket says 172.21.15.67:37924) 2024-04-17T10:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.089+0000 7f6ec0e00640 1 -- 172.21.15.67:0/302372607 learned_addr learned my addr 172.21.15.67:0/302372607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebbe00640 1 -- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0d06a0 msgr2=0x7f6ebc161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0d06a0 0x7f6ebc161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebbe00640 1 -- 172.21.15.67:0/302372607 >> v1:172.21.15.86:6789/0 conn(0x7f6ebc09adf0 legacy=0x7f6ebc0cff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebbe00640 1 -- 172.21.15.67:0/302372607 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc09a030 con 0x7f6ebc15dd60 2024-04-17T10:34:19.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ec0e00640 1 --2- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0d06a0 0x7f6ebc161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:19.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc161880 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0097c0 tx=0x7f6eac01c470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c95d45c0f34574 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:19.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebb400640 1 -- 172.21.15.67:0/302372607 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac0a7070 con 0x7f6ebc15dd60 2024-04-17T10:34:19.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebb400640 1 -- 172.21.15.67:0/302372607 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eac0b0070 con 0x7f6ebc15dd60 2024-04-17T10:34:19.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ebb400640 1 -- 172.21.15.67:0/302372607 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eac0ab890 con 0x7f6ebc15dd60 2024-04-17T10:34:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ec2e00640 1 -- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc15dd60 msgr2=0x7f6ebc161880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.093+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc161880 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0097c0 tx=0x7f6eac01c470 comp rx=0 tx=0).stop 2024-04-17T10:34:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- 172.21.15.67:0/302372607 shutdown_connections 2024-04-17T10:34:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc0d06a0 0x7f6ebc161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/302372607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc161880 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- 172.21.15.67:0/302372607 >> 172.21.15.67:0/302372607 conn(0x7f6ebc0b34a0 msgr2=0x7f6ebc0b38b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- 172.21.15.67:0/302372607 shutdown_connections 2024-04-17T10:34:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- 172.21.15.67:0/302372607 wait complete. 2024-04-17T10:34:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 Processor -- start 2024-04-17T10:34:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- start start 2024-04-17T10:34:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc09adf0 0x7f6ebc202a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0d06a0 0x7f6ebc0c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc0c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc06c730 con 0x7f6ebc15dd60 2024-04-17T10:34:19.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc06c5b0 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec2e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc06c8b0 con 0x7f6ebc09adf0 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc0c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc0c54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40164/0 (socket says 172.21.15.67:40164) 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ec0e00640 1 -- 172.21.15.67:0/1195531316 learned_addr learned my addr 172.21.15.67:0/1195531316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6eb3400640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0d06a0 0x7f6ebc0c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.097+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc09adf0 0x7f6ebc202a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb3400640 1 -- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc09adf0 msgr2=0x7f6ebc202a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb3400640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc09adf0 0x7f6ebc202a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb3400640 1 -- 172.21.15.67:0/1195531316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc15dd60 msgr2=0x7f6ebc0c54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb3400640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc0c54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb3400640 1 -- 172.21.15.67:0/1195531316 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc0c4340 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6ec0e00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc0c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:19.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc09adf0 0x7f6ebc202a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:19.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb3400640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0d06a0 0x7f6ebc0c8a80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400c730 tx=0x7f6ea400cb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:19.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1195531316 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4016020 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1195531316 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ea4006d60 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1195531316 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ea4003bc0 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc0c4620 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc0caba0 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc04ef30 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.101+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1195531316 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6ea40050d0 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.105+0000 7f6eb8c00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f60 0x7f6e98069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.105+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1195531316 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6ea40849e0 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.105+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f60 0x7f6e98069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.105+0000 7f6ebbe00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f60 0x7f6e98069420 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0af000 tx=0x7f6eac0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:19.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.109+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1195531316 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6ea4020020 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.281+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6ebc0b2da0 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.285+0000 7f6eb8c00640 1 -- 172.21.15.67:0/1195531316 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ea4055b50 con 0x7f6ebc0d06a0 2024-04-17T10:34:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f60 msgr2=0x7f6e98069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f60 0x7f6e98069420 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f6eac0af000 tx=0x7f6eac0a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0d06a0 msgr2=0x7f6ebc0c8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0d06a0 0x7f6ebc0c8a80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f6ea400c730 tx=0x7f6ea400cb30 comp rx=0 tx=0).stop 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 shutdown_connections 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e98066f60 0x7f6e98069420 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ebc15dd60 0x7f6ebc0c54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ebc0d06a0 0x7f6ebc0c8a80 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 --2- 172.21.15.67:0/1195531316 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ebc09adf0 0x7f6ebc202a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 >> 172.21.15.67:0/1195531316 conn(0x7f6ebc0b34a0 msgr2=0x7f6ebc161f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 shutdown_connections 2024-04-17T10:34:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.289+0000 7f6ec2e00640 1 -- 172.21.15.67:0/1195531316 wait complete. 2024-04-17T10:34:19.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort' 2024-04-17T10:34:19.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cbd400640 1 Processor -- start 2024-04-17T10:34:19.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cbd400640 1 -- start start 2024-04-17T10:34:19.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cbd400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4cb806e6e0 0x7f4cb80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cbd400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4cb806eac0 0x7f4cb80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cbd400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cb815ae40 con 0x7f4cb8143ae0 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cbd400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb80cf430 con 0x7f4cb806e6e0 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cbd400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cb8071440 con 0x7f4cb806eac0 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4cb806e6e0 0x7f4cb80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4cb806e6e0 0x7f4cb80ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40178/0 (socket says 172.21.15.67:40178) 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb6800640 1 -- 172.21.15.67:0/2150117142 learned_addr learned my addr 172.21.15.67:0/2150117142 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb5e00640 1 --2- 172.21.15.67:0/2150117142 >> v2:172.21.15.86:3300/0 conn(0x7f4cb806eac0 0x7f4cb80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb6800640 1 -- 172.21.15.67:0/2150117142 >> v2:172.21.15.86:3300/0 conn(0x7f4cb806eac0 msgr2=0x7f4cb80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb6800640 1 --2- 172.21.15.67:0/2150117142 >> v2:172.21.15.86:3300/0 conn(0x7f4cb806eac0 0x7f4cb80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb6800640 1 -- 172.21.15.67:0/2150117142 >> v1:172.21.15.86:6789/0 conn(0x7f4cb8143ae0 legacy=0x7f4cb8143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.449+0000 7f4cb6800640 1 -- 172.21.15.67:0/2150117142 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb8071720 con 0x7f4cb806e6e0 2024-04-17T10:34:19.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cb6800640 1 --2- 172.21.15.67:0/2150117142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb806e6e0 0x7f4cb80ce9b0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f4cb806ef80 tx=0x7f4ca801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a8b9437b94d5de server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:19.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cb5400640 1 -- 172.21.15.67:0/2150117142 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca80b2070 con 0x7f4cb806e6e0 2024-04-17T10:34:19.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cb5400640 1 -- 172.21.15.67:0/2150117142 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ca80a9070 con 0x7f4cb806e6e0 2024-04-17T10:34:19.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cb5400640 1 -- 172.21.15.67:0/2150117142 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ca80ad880 con 0x7f4cb806e6e0 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 -- 172.21.15.67:0/2150117142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb806e6e0 msgr2=0x7f4cb80ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 --2- 172.21.15.67:0/2150117142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb806e6e0 0x7f4cb80ce9b0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f4cb806ef80 tx=0x7f4ca801c460 comp rx=0 tx=0).stop 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 -- 172.21.15.67:0/2150117142 shutdown_connections 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 --2- 172.21.15.67:0/2150117142 >> v2:172.21.15.86:3300/0 conn(0x7f4cb806eac0 0x7f4cb80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 --2- 172.21.15.67:0/2150117142 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb806e6e0 0x7f4cb80ce9b0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 -- 172.21.15.67:0/2150117142 >> 172.21.15.67:0/2150117142 conn(0x7f4cb806ac90 msgr2=0x7f4cb806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 -- 172.21.15.67:0/2150117142 shutdown_connections 2024-04-17T10:34:19.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.453+0000 7f4cbd400640 1 -- 172.21.15.67:0/2150117142 wait complete. 2024-04-17T10:34:19.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 Processor -- start 2024-04-17T10:34:19.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 -- start start 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cb806e6e0 0x7f4cb81569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4cb806eac0 0x7f4cb813c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb8143ae0 0x7f4cb8141ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb806dd00 con 0x7f4cb8143ae0 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb806db80 con 0x7f4cb806e6e0 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb806de80 con 0x7f4cb806eac0 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb8143ae0 0x7f4cb8141ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cb806e6e0 0x7f4cb81569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb8143ae0 0x7f4cb8141ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40184/0 (socket says 172.21.15.67:40184) 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 -- 172.21.15.67:0/3277737512 learned_addr learned my addr 172.21.15.67:0/3277737512 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb5e00640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4cb806eac0 0x7f4cb813c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 -- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4cb806eac0 msgr2=0x7f4cb813c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4cb806eac0 0x7f4cb813c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 -- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cb806e6e0 msgr2=0x7f4cb81569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cb806e6e0 0x7f4cb81569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 -- 172.21.15.67:0/3277737512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb80b2610 con 0x7f4cb8143ae0 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cb7200640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb8143ae0 0x7f4cb8141ca0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000c730 tx=0x7f4cb000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4caea00640 1 -- 172.21.15.67:0/3277737512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb0003b80 con 0x7f4cb8143ae0 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4caea00640 1 -- 172.21.15.67:0/3277737512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb0003d20 con 0x7f4cb8143ae0 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cb813b1a0 con 0x7f4cb8143ae0 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4caea00640 1 -- 172.21.15.67:0/3277737512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb000aa30 con 0x7f4cb8143ae0 2024-04-17T10:34:19.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.457+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cb813b760 con 0x7f4cb8143ae0 2024-04-17T10:34:19.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.461+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cb8072d50 con 0x7f4cb8143ae0 2024-04-17T10:34:19.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.461+0000 7f4caea00640 1 -- 172.21.15.67:0/3277737512 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4cb000abd0 con 0x7f4cb8143ae0 2024-04-17T10:34:19.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.461+0000 7f4caea00640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c98066f60 0x7f4c98069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:19.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.461+0000 7f4caea00640 1 -- 172.21.15.67:0/3277737512 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f4cb0084b50 con 0x7f4cb8143ae0 2024-04-17T10:34:19.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.461+0000 7f4cb6800640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c98066f60 0x7f4c98069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:19.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.461+0000 7f4cb6800640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c98066f60 0x7f4c98069420 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80af000 tx=0x7f4ca80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:19.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.465+0000 7f4caea00640 1 -- 172.21.15.67:0/3277737512 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4cb0055710 con 0x7f4cb8143ae0 2024-04-17T10:34:19.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.685+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]} v 0) v1 -- 0x7f4cb8074e30 con 0x7f4cb8143ae0 2024-04-17T10:34:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4caea00640 1 -- 172.21.15.67:0/3277737512 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]}]=0 v452) v1 ==== 166+0+0 (secure 0 0 0) 0x7f4cb00597b0 con 0x7f4cb8143ae0 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c98066f60 msgr2=0x7f4c98069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c98066f60 0x7f4c98069420 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4ca80af000 tx=0x7f4ca80a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb8143ae0 msgr2=0x7f4cb8141ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb8143ae0 0x7f4cb8141ca0 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f4cb000c730 tx=0x7f4cb000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 shutdown_connections 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4c98066f60 0x7f4c98069420 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4cb8143ae0 0x7f4cb8141ca0 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4cb806eac0 0x7f4cb813c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 --2- 172.21.15.67:0/3277737512 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4cb806e6e0 0x7f4cb81569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 >> 172.21.15.67:0/3277737512 conn(0x7f4cb806ac90 msgr2=0x7f4cb80addb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 shutdown_connections 2024-04-17T10:34:19.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:19.889+0000 7f4cbd400640 1 -- 172.21.15.67:0/3277737512 wait complete. 2024-04-17T10:34:19.902 INFO:tasks.cephfs_test_runner:test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-04-17T10:34:19.902 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_task_status_on_mds_failover (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-04-17T10:34:19.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:20.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.053+0000 7f3d61a00640 1 Processor -- start 2024-04-17T10:34:20.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.053+0000 7f3d61a00640 1 -- start start 2024-04-17T10:34:20.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d61a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3d5c06e6e0 0x7f3d5c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d61a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d5c06eac0 0x7f3d5c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d61a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c15ae40 con 0x7f3d5c143ae0 2024-04-17T10:34:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d61a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5c0cf430 con 0x7f3d5c06eac0 2024-04-17T10:34:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d61a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5c071440 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d53e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d5c06eac0 0x7f3d5c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d53e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3d5c06eac0 0x7f3d5c0ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40192/0 (socket says 172.21.15.67:40192) 2024-04-17T10:34:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d53e00640 1 -- 172.21.15.67:0/712523710 learned_addr learned my addr 172.21.15.67:0/712523710 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d5aa00640 1 --2- 172.21.15.67:0/712523710 >> v2:172.21.15.86:3300/0 conn(0x7f3d5c06e6e0 0x7f3d5c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d53e00640 1 -- 172.21.15.67:0/712523710 >> v2:172.21.15.86:3300/0 conn(0x7f3d5c06e6e0 msgr2=0x7f3d5c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d53e00640 1 --2- 172.21.15.67:0/712523710 >> v2:172.21.15.86:3300/0 conn(0x7f3d5c06e6e0 0x7f3d5c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d53e00640 1 -- 172.21.15.67:0/712523710 >> v1:172.21.15.86:6789/0 conn(0x7f3d5c143ae0 legacy=0x7f3d5c143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d53e00640 1 -- 172.21.15.67:0/712523710 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c071720 con 0x7f3d5c06eac0 2024-04-17T10:34:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.057+0000 7f3d5a000640 1 -- 172.21.15.67:0/712523710 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400647382 0 0) 0x7f3d5c15ae40 con 0x7f3d5c143ae0 2024-04-17T10:34:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d53e00640 1 --2- 172.21.15.67:0/712523710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06eac0 0x7f3d5c0ceef0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0cf620 tx=0x7f3d4401eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7002bdedc8348e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d5a000640 1 -- 172.21.15.67:0/712523710 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d440b4070 con 0x7f3d5c06eac0 2024-04-17T10:34:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d5a000640 1 -- 172.21.15.67:0/712523710 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d440a7b00 con 0x7f3d5c06eac0 2024-04-17T10:34:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d5a000640 1 -- 172.21.15.67:0/712523710 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d440af440 con 0x7f3d5c06eac0 2024-04-17T10:34:20.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 -- 172.21.15.67:0/712523710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06eac0 msgr2=0x7f3d5c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 --2- 172.21.15.67:0/712523710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06eac0 0x7f3d5c0ceef0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0cf620 tx=0x7f3d4401eb30 comp rx=0 tx=0).stop 2024-04-17T10:34:20.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 -- 172.21.15.67:0/712523710 shutdown_connections 2024-04-17T10:34:20.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 --2- 172.21.15.67:0/712523710 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06eac0 0x7f3d5c0ceef0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 --2- 172.21.15.67:0/712523710 >> v2:172.21.15.86:3300/0 conn(0x7f3d5c06e6e0 0x7f3d5c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 -- 172.21.15.67:0/712523710 >> 172.21.15.67:0/712523710 conn(0x7f3d5c06ac90 msgr2=0x7f3d5c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:20.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 -- 172.21.15.67:0/712523710 shutdown_connections 2024-04-17T10:34:20.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 -- 172.21.15.67:0/712523710 wait complete. 2024-04-17T10:34:20.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 Processor -- start 2024-04-17T10:34:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.061+0000 7f3d61a00640 1 -- start start 2024-04-17T10:34:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06e6e0 0x7f3d5c142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5c06eac0 0x7f3d5c149dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5c143ae0 0x7f3d5c14a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c06dd00 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c06db80 con 0x7f3d5c143ae0 2024-04-17T10:34:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c06de80 con 0x7f3d5c06eac0 2024-04-17T10:34:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06e6e0 0x7f3d5c142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06e6e0 0x7f3d5c142a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40208/0 (socket says 172.21.15.67:40208) 2024-04-17T10:34:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 -- 172.21.15.67:0/4198201200 learned_addr learned my addr 172.21.15.67:0/4198201200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:20.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5b400640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5c143ae0 0x7f3d5c14a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d53e00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5c06eac0 0x7f3d5c149dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 -- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5c06eac0 msgr2=0x7f3d5c149dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5c06eac0 0x7f3d5c149dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 -- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5c143ae0 msgr2=0x7f3d5c14a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5c143ae0 0x7f3d5c14a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 -- 172.21.15.67:0/4198201200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c0caa20 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d5aa00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06e6e0 0x7f3d5c142a20 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00e930 tx=0x7f3d4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d52a00640 1 -- 172.21.15.67:0/4198201200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c017070 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d52a00640 1 -- 172.21.15.67:0/4198201200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c00ca00 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d52a00640 1 -- 172.21.15.67:0/4198201200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c00cdb0 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c0cad00 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c0cb2c0 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5c072d50 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.065+0000 7f3d52a00640 1 -- 172.21.15.67:0/4198201200 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3d4c01a070 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.069+0000 7f3d52a00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d38066f60 0x7f3d38069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.069+0000 7f3d52a00640 1 -- 172.21.15.67:0/4198201200 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f3d4c085210 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.069+0000 7f3d53e00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d38066f60 0x7f3d38069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.069+0000 7f3d53e00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d38066f60 0x7f3d38069420 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3d440b1000 tx=0x7f3d44008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.073+0000 7f3d52a00640 1 -- 172.21.15.67:0/4198201200 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3d4c055de0 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.233+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d5c1654f0 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.233+0000 7f3d52a00640 1 -- 172.21.15.67:0/4198201200 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+13253 (secure 0 0 0) 0x7f3d4c059e80 con 0x7f3d5c06e6e0 2024-04-17T10:34:20.226 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:20.226 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":121,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:34:18.885437+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6834/397092831":"2024-04-18T10:34:15.872063+0000","172.21.15.67:6833/397092831":"2024-04-18T10:34:15.872063+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d38066f60 msgr2=0x7f3d38069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d38066f60 0x7f3d38069420 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f3d440b1000 tx=0x7f3d44008040 comp rx=0 tx=0).stop 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06e6e0 msgr2=0x7f3d5c142a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06e6e0 0x7f3d5c142a20 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00e930 tx=0x7f3d4c00edf0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 shutdown_connections 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d38066f60 0x7f3d38069420 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3d5c143ae0 0x7f3d5c14a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3d5c06eac0 0x7f3d5c149dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 --2- 172.21.15.67:0/4198201200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3d5c06e6e0 0x7f3d5c142a20 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 >> 172.21.15.67:0/4198201200 conn(0x7f3d5c06ac90 msgr2=0x7f3d5c0af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:20.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 shutdown_connections 2024-04-17T10:34:20.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.237+0000 7f3d61a00640 1 -- 172.21.15.67:0/4198201200 wait complete. 2024-04-17T10:34:20.247 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:34:20.247 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:34:20.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f7e00640 1 Processor -- start 2024-04-17T10:34:20.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f7e00640 1 -- start start 2024-04-17T10:34:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f7e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f63f0071af0 0x7f63f00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63f015be70 0x7f63f015c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f015cb30 con 0x7f63f0159440 2024-04-17T10:34:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f00a8650 con 0x7f63f015be70 2024-04-17T10:34:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f7e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f015ce10 con 0x7f63f0071af0 2024-04-17T10:34:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f63f015be70 0x7f63f015c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f63f0159440 0x7f63f00713e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42660/0 (socket says 172.21.15.67:42660) 2024-04-17T10:34:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f5400640 1 -- 172.21.15.67:0/693469244 learned_addr learned my addr 172.21.15.67:0/693469244 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f4a00640 1 --2- 172.21.15.67:0/693469244 >> v2:172.21.15.86:3300/0 conn(0x7f63f0071af0 0x7f63f00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f5e00640 1 -- 172.21.15.67:0/693469244 >> v2:172.21.15.86:3300/0 conn(0x7f63f0071af0 msgr2=0x7f63f00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f5e00640 1 --2- 172.21.15.67:0/693469244 >> v2:172.21.15.86:3300/0 conn(0x7f63f0071af0 0x7f63f00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f5e00640 1 -- 172.21.15.67:0/693469244 >> v1:172.21.15.67:6789/0 conn(0x7f63f0159440 legacy=0x7f63f00713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f5e00640 1 -- 172.21.15.67:0/693469244 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f00cd610 con 0x7f63f015be70 2024-04-17T10:34:20.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.401+0000 7f63f4a00640 1 --2- 172.21.15.67:0/693469244 >> v2:172.21.15.86:3300/0 conn(0x7f63f0071af0 0x7f63f00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:20.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f5e00640 1 --2- 172.21.15.67:0/693469244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f015be70 0x7f63f015c2c0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f63f00cf770 tx=0x7f63e801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45e0ee864d460f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63efe00640 1 -- 172.21.15.67:0/693469244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e80b2070 con 0x7f63f015be70 2024-04-17T10:34:20.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63efe00640 1 -- 172.21.15.67:0/693469244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e80a59f0 con 0x7f63f015be70 2024-04-17T10:34:20.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63efe00640 1 -- 172.21.15.67:0/693469244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e80a5dc0 con 0x7f63f015be70 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 -- 172.21.15.67:0/693469244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f015be70 msgr2=0x7f63f015c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 --2- 172.21.15.67:0/693469244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f015be70 0x7f63f015c2c0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f63f00cf770 tx=0x7f63e801c930 comp rx=0 tx=0).stop 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 -- 172.21.15.67:0/693469244 shutdown_connections 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 --2- 172.21.15.67:0/693469244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f015be70 0x7f63f015c2c0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 --2- 172.21.15.67:0/693469244 >> v2:172.21.15.86:3300/0 conn(0x7f63f0071af0 0x7f63f00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 -- 172.21.15.67:0/693469244 >> 172.21.15.67:0/693469244 conn(0x7f63f00b1740 msgr2=0x7f63f00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 -- 172.21.15.67:0/693469244 shutdown_connections 2024-04-17T10:34:20.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.405+0000 7f63f7e00640 1 -- 172.21.15.67:0/693469244 wait complete. 2024-04-17T10:34:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 Processor -- start 2024-04-17T10:34:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 -- start start 2024-04-17T10:34:20.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f0071af0 0x7f63f01530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f0159440 0x7f63f014f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f015be70 0x7f63f0165d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f015d840 con 0x7f63f0159440 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f01598e0 con 0x7f63f015be70 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f015d9c0 con 0x7f63f0071af0 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f0159440 0x7f63f014f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f0159440 0x7f63f014f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34974/0 (socket says 172.21.15.67:34974) 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 -- 172.21.15.67:0/2171823614 learned_addr learned my addr 172.21.15.67:0/2171823614 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f5400640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f0071af0 0x7f63f01530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f5e00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f015be70 0x7f63f0165d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 -- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f0071af0 msgr2=0x7f63f01530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f0071af0 0x7f63f01530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 -- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f015be70 msgr2=0x7f63f0165d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f015be70 0x7f63f0165d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 -- 172.21.15.67:0/2171823614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f014f860 con 0x7f63f0159440 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f4a00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f0159440 0x7f63f014f320 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00e810 tx=0x7f63dc00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63ed600640 1 -- 172.21.15.67:0/2171823614 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc017070 con 0x7f63f0159440 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63ed600640 1 -- 172.21.15.67:0/2171823614 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63dc00c8d0 con 0x7f63f0159440 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63ed600640 1 -- 172.21.15.67:0/2171823614 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63dc00f040 con 0x7f63f0159440 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f0166280 con 0x7f63f0159440 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f0153990 con 0x7f63f0159440 2024-04-17T10:34:20.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.409+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f004ef30 con 0x7f63f0159440 2024-04-17T10:34:20.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.413+0000 7f63ed600640 1 -- 172.21.15.67:0/2171823614 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f63dc01a070 con 0x7f63f0159440 2024-04-17T10:34:20.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.413+0000 7f63ed600640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d0066f60 0x7f63d0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.413+0000 7f63ed600640 1 -- 172.21.15.67:0/2171823614 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f63dc0849e0 con 0x7f63f0159440 2024-04-17T10:34:20.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.413+0000 7f63f5400640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d0066f60 0x7f63d0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.413+0000 7f63f5400640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d0066f60 0x7f63d0069420 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f63f00cd140 tx=0x7f63e4007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.417+0000 7f63ed600640 1 -- 172.21.15.67:0/2171823614 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f63dc055500 con 0x7f63f0159440 2024-04-17T10:34:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.577+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63f0152940 con 0x7f63f0159440 2024-04-17T10:34:20.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.577+0000 7f63ed600640 1 -- 172.21.15.67:0/2171823614 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63dc0595a0 con 0x7f63f0159440 2024-04-17T10:34:20.570 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:34:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d0066f60 msgr2=0x7f63d0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d0066f60 0x7f63d0069420 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f63f00cd140 tx=0x7f63e4007e90 comp rx=0 tx=0).stop 2024-04-17T10:34:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f0159440 msgr2=0x7f63f014f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f0159440 0x7f63f014f320 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f63dc00e810 tx=0x7f63dc00ecd0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 shutdown_connections 2024-04-17T10:34:20.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f63d0066f60 0x7f63d0069420 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f63f015be70 0x7f63f0165d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f63f0159440 0x7f63f014f320 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 --2- 172.21.15.67:0/2171823614 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f63f0071af0 0x7f63f01530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 >> 172.21.15.67:0/2171823614 conn(0x7f63f00b1740 msgr2=0x7f63f00af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 shutdown_connections 2024-04-17T10:34:20.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.581+0000 7f63f7e00640 1 -- 172.21.15.67:0/2171823614 wait complete. 2024-04-17T10:34:20.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover' 2024-04-17T10:34:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.741+0000 7f5687600640 1 Processor -- start 2024-04-17T10:34:20.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.741+0000 7f5687600640 1 -- start start 2024-04-17T10:34:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5687600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f5680071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5687600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568006e2f0 con 0x7f5680145040 2024-04-17T10:34:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5687600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f568006d6e0 con 0x7f56800cc300 2024-04-17T10:34:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5687600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56800a86c0 con 0x7f5680071310 2024-04-17T10:34:20.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5684c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f5680145040 0x7f568006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42670/0 (socket says 172.21.15.67:42670) 2024-04-17T10:34:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5684c00640 1 -- 172.21.15.67:0/1359763411 learned_addr learned my addr 172.21.15.67:0/1359763411 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5677e00640 1 --2- 172.21.15.67:0/1359763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f5680071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053555413 0 0) 0x7f568006e2f0 con 0x7f5680145040 2024-04-17T10:34:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5664003530 con 0x7f5680145040 2024-04-17T10:34:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564277331 0 0) 0x7f568006d6e0 con 0x7f56800cc300 2024-04-17T10:34:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f568006e2f0 con 0x7f56800cc300 2024-04-17T10:34:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3141144765 0 0) 0x7f5664003530 con 0x7f5680145040 2024-04-17T10:34:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5677e00640 1 -- 172.21.15.67:0/1359763411 >> v1:172.21.15.86:6789/0 conn(0x7f56800cc300 legacy=0x7f56800cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5677e00640 1 -- 172.21.15.67:0/1359763411 >> v1:172.21.15.67:6789/0 conn(0x7f5680145040 legacy=0x7f568006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.745+0000 7f5677e00640 1 -- 172.21.15.67:0/1359763411 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568006d9c0 con 0x7f5680071310 2024-04-17T10:34:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5677e00640 1 --2- 172.21.15.67:0/1359763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f5680071770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f56680097f0 tx=0x7f566801ebf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31d8b640cde8a838 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56680b5070 con 0x7f5680071310 2024-04-17T10:34:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56680a8670 con 0x7f5680071310 2024-04-17T10:34:20.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f567fe00640 1 -- 172.21.15.67:0/1359763411 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5670001680 con 0x7f5680071310 2024-04-17T10:34:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- 172.21.15.67:0/1359763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 msgr2=0x7f5680071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 --2- 172.21.15.67:0/1359763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f5680071770 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f56680097f0 tx=0x7f566801ebf0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- 172.21.15.67:0/1359763411 shutdown_connections 2024-04-17T10:34:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 --2- 172.21.15.67:0/1359763411 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f5680071770 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- 172.21.15.67:0/1359763411 >> 172.21.15.67:0/1359763411 conn(0x7f568006ac90 msgr2=0x7f568006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- 172.21.15.67:0/1359763411 shutdown_connections 2024-04-17T10:34:20.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- 172.21.15.67:0/1359763411 wait complete. 2024-04-17T10:34:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 Processor -- start 2024-04-17T10:34:20.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- start start 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f568014d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56800cc300 0x7f568014c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680145040 0x7f5680143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568015bf10 con 0x7f56800cc300 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568006df80 con 0x7f5680145040 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5687600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f568006d370 con 0x7f5680071310 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5677e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56800cc300 0x7f568014c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5677e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56800cc300 0x7f568014c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34980/0 (socket says 172.21.15.67:34980) 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5677e00640 1 -- 172.21.15.67:0/432536617 learned_addr learned my addr 172.21.15.67:0/432536617 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5684c00640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f568014d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.749+0000 7f5685600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680145040 0x7f5680143cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5677e00640 1 -- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 msgr2=0x7f568014d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5677e00640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f568014d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5677e00640 1 -- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680145040 msgr2=0x7f5680143cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5677e00640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680145040 0x7f5680143cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5677e00640 1 -- 172.21.15.67:0/432536617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568015d280 con 0x7f56800cc300 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5677e00640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56800cc300 0x7f568014c8e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f56680097f0 tx=0x7f566801b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f567d600640 1 -- 172.21.15.67:0/432536617 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56680b5040 con 0x7f56800cc300 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f567d600640 1 -- 172.21.15.67:0/432536617 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56680b0440 con 0x7f56800cc300 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568015d560 con 0x7f56800cc300 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f567d600640 1 -- 172.21.15.67:0/432536617 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5668006e20 con 0x7f56800cc300 2024-04-17T10:34:20.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568015db20 con 0x7f56800cc300 2024-04-17T10:34:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5680072d50 con 0x7f56800cc300 2024-04-17T10:34:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f567d600640 1 -- 172.21.15.67:0/432536617 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f56680c1020 con 0x7f56800cc300 2024-04-17T10:34:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f567d600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f565c066f60 0x7f565c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:20.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f567d600640 1 -- 172.21.15.67:0/432536617 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f5668121aa0 con 0x7f56800cc300 2024-04-17T10:34:20.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.753+0000 7f5684c00640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f565c066f60 0x7f565c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:20.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.757+0000 7f5684c00640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f565c066f60 0x7f565c069420 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5670000fe0 tx=0x7f5670001720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:20.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.757+0000 7f567d600640 1 -- 172.21.15.67:0/432536617 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5668018750 con 0x7f56800cc300 2024-04-17T10:34:20.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:20.985+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]} v 0) v1 -- 0x7f5680074e30 con 0x7f56800cc300 2024-04-17T10:34:21.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.897+0000 7f567d600640 1 -- 172.21.15.67:0/432536617 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]}]=0 v454) v1 ==== 169+0+0 (secure 0 0 0) 0x7f56680f2530 con 0x7f56800cc300 2024-04-17T10:34:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f565c066f60 msgr2=0x7f565c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f565c066f60 0x7f565c069420 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f5670000fe0 tx=0x7f5670001720 comp rx=0 tx=0).stop 2024-04-17T10:34:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56800cc300 msgr2=0x7f568014c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56800cc300 0x7f568014c8e0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f56680097f0 tx=0x7f566801b990 comp rx=0 tx=0).stop 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 shutdown_connections 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f565c066f60 0x7f565c069420 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5680145040 0x7f5680143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f56800cc300 0x7f568014c8e0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 --2- 172.21.15.67:0/432536617 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5680071310 0x7f568014d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 >> 172.21.15.67:0/432536617 conn(0x7f568006ac90 msgr2=0x7f56800af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 shutdown_connections 2024-04-17T10:34:21.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:21.901+0000 7f5687600640 1 -- 172.21.15.67:0/432536617 wait complete. 2024-04-17T10:34:21.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:34:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.073+0000 7f8f38c00640 1 Processor -- start 2024-04-17T10:34:22.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.073+0000 7f8f38c00640 1 -- start start 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f38c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f340cd030 0x7f8f3406e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f38c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8f3406e7b0 0x7f8f340b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f38c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340b4990 con 0x7f8f340aef60 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f38c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f340af340 con 0x7f8f340cd030 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f38c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3406ec00 con 0x7f8f3406e7b0 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f340cd030 0x7f8f3406e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8f340cd030 0x7f8f3406e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34996/0 (socket says 172.21.15.67:34996) 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 -- 172.21.15.67:0/2974216608 learned_addr learned my addr 172.21.15.67:0/2974216608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f32800640 1 --2- 172.21.15.67:0/2974216608 >> v2:172.21.15.86:3300/0 conn(0x7f8f3406e7b0 0x7f8f340b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 -- 172.21.15.67:0/2974216608 >> v2:172.21.15.86:3300/0 conn(0x7f8f3406e7b0 msgr2=0x7f8f340b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 --2- 172.21.15.67:0/2974216608 >> v2:172.21.15.86:3300/0 conn(0x7f8f3406e7b0 0x7f8f340b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 -- 172.21.15.67:0/2974216608 >> v1:172.21.15.86:6789/0 conn(0x7f8f340aef60 legacy=0x7f8f340cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 -- 172.21.15.67:0/2974216608 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3415ae90 con 0x7f8f340cd030 2024-04-17T10:34:22.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.077+0000 7f8f2be00640 1 --2- 172.21.15.67:0/2974216608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f3406e270 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f8f340b2e60 tx=0x7f8f1c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8deaaad39b41f8c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:22.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f31400640 1 -- 172.21.15.67:0/2974216608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c0b2070 con 0x7f8f340cd030 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f31400640 1 -- 172.21.15.67:0/2974216608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c0ad680 con 0x7f8f340cd030 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f31400640 1 -- 172.21.15.67:0/2974216608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c0ada50 con 0x7f8f340cd030 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- 172.21.15.67:0/2974216608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 msgr2=0x7f8f3406e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2974216608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f3406e270 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f8f340b2e60 tx=0x7f8f1c01c490 comp rx=0 tx=0).stop 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- 172.21.15.67:0/2974216608 shutdown_connections 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2974216608 >> v2:172.21.15.86:3300/0 conn(0x7f8f3406e7b0 0x7f8f340b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2974216608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f3406e270 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- 172.21.15.67:0/2974216608 >> 172.21.15.67:0/2974216608 conn(0x7f8f340b1740 msgr2=0x7f8f340b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- 172.21.15.67:0/2974216608 shutdown_connections 2024-04-17T10:34:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- 172.21.15.67:0/2974216608 wait complete. 2024-04-17T10:34:22.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 Processor -- start 2024-04-17T10:34:22.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- start start 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3406e7b0 0x7f8f3414da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340aef60 0x7f8f34136ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f34159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3415bc10 con 0x7f8f340cd030 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340b3e70 con 0x7f8f3406e7b0 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f38c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3414df80 con 0x7f8f340aef60 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f32800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f34159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f32800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f34159850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35010/0 (socket says 172.21.15.67:35010) 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f32800640 1 -- 172.21.15.67:0/2215299058 learned_addr learned my addr 172.21.15.67:0/2215299058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f31e00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3406e7b0 0x7f8f3414da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.081+0000 7f8f2be00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340aef60 0x7f8f34136ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f32800640 1 -- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340aef60 msgr2=0x7f8f34136ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f32800640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340aef60 0x7f8f34136ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f32800640 1 -- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3406e7b0 msgr2=0x7f8f3414da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f32800640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3406e7b0 0x7f8f3414da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f32800640 1 -- 172.21.15.67:0/2215299058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3413a5a0 con 0x7f8f340cd030 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f32800640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f34159850 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00efc0 tx=0x7f8f2c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f2a000640 1 -- 172.21.15.67:0/2215299058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c018020 con 0x7f8f340cd030 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f2a000640 1 -- 172.21.15.67:0/2215299058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c014480 con 0x7f8f340cd030 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3413a2c0 con 0x7f8f340cd030 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f2a000640 1 -- 172.21.15.67:0/2215299058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c014870 con 0x7f8f340cd030 2024-04-17T10:34:22.077 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3414d240 con 0x7f8f340cd030 2024-04-17T10:34:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3404ef30 con 0x7f8f340cd030 2024-04-17T10:34:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f2a000640 1 -- 172.21.15.67:0/2215299058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8f2c01b020 con 0x7f8f340cd030 2024-04-17T10:34:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f2a000640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f10066f90 0x7f8f10069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f2a000640 1 -- 172.21.15.67:0/2215299058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f8f2c085820 con 0x7f8f340cd030 2024-04-17T10:34:22.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.085+0000 7f8f31e00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f10066f90 0x7f8f10069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.089+0000 7f8f31e00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f10066f90 0x7f8f10069450 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8f24007fd0 tx=0x7f8f24009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:22.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.089+0000 7f8f2a000640 1 -- 172.21.15.67:0/2215299058 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8f2c022070 con 0x7f8f340cd030 2024-04-17T10:34:22.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.253+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8f340cee40 con 0x7f8f340cd030 2024-04-17T10:34:22.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f2a000640 1 -- 172.21.15.67:0/2215299058 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8f2c010040 con 0x7f8f340cd030 2024-04-17T10:34:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f10066f90 msgr2=0x7f8f10069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f10066f90 0x7f8f10069450 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8f24007fd0 tx=0x7f8f24009040 comp rx=0 tx=0).stop 2024-04-17T10:34:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 msgr2=0x7f8f34159850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f34159850 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c00efc0 tx=0x7f8f2c00c490 comp rx=0 tx=0).stop 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 shutdown_connections 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8f10066f90 0x7f8f10069450 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f340cd030 0x7f8f34159850 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f340aef60 0x7f8f34136ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 --2- 172.21.15.67:0/2215299058 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f3406e7b0 0x7f8f3414da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 >> 172.21.15.67:0/2215299058 conn(0x7f8f340b1740 msgr2=0x7f8f340b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 shutdown_connections 2024-04-17T10:34:22.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.261+0000 7f8f38c00640 1 -- 172.21.15.67:0/2215299058 wait complete. 2024-04-17T10:34:22.273 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:34:22.274 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:34:22.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:34:22.439 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:34:22.439 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:34:22.439 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:22.448 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:34:22.448 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:34:22.448 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:22.457 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:34:22.457 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:34:22.457 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:22.499 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:34:22.499 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:34:22.499 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:22.511 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:34:22.511 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:34:22.511 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:22.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 Processor -- start 2024-04-17T10:34:22.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 -- start start 2024-04-17T10:34:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f090c0ce400 0x7f090c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f090c15e550 0x7f090c0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090c161650 con 0x7f090c0ce400 2024-04-17T10:34:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f090c1625e0 con 0x7f090c15e550 2024-04-17T10:34:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0910e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c15eb50 con 0x7f090c0ce020 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f090c0ce400 0x7f090c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f090c0ce400 0x7f090c15e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35014/0 (socket says 172.21.15.67:35014) 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 -- 172.21.15.67:0/3082232391 learned_addr learned my addr 172.21.15.67:0/3082232391 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0909e00640 1 --2- 172.21.15.67:0/3082232391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f090a800640 1 --2- 172.21.15.67:0/3082232391 >> v2:172.21.15.86:3300/0 conn(0x7f090c15e550 0x7f090c0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 -- 172.21.15.67:0/3082232391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 msgr2=0x7f090c1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 --2- 172.21.15.67:0/3082232391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 -- 172.21.15.67:0/3082232391 >> v2:172.21.15.86:3300/0 conn(0x7f090c15e550 msgr2=0x7f090c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 --2- 172.21.15.67:0/3082232391 >> v2:172.21.15.86:3300/0 conn(0x7f090c15e550 0x7f090c0cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0903e00640 1 -- 172.21.15.67:0/3082232391 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c15ee30 con 0x7f090c0ce400 2024-04-17T10:34:22.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.693+0000 7f0909e00640 1 --2- 172.21.15.67:0/3082232391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0903e00640 1 --2- 172.21.15.67:0/3082232391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c0ce400 0x7f090c15e010 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f090c0d2840 tx=0x7f08f401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3dfac20b3f2518 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0909400640 1 -- 172.21.15.67:0/3082232391 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f40b1070 con 0x7f090c0ce400 2024-04-17T10:34:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0909400640 1 -- 172.21.15.67:0/3082232391 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f40ac620 con 0x7f090c0ce400 2024-04-17T10:34:22.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0909400640 1 -- 172.21.15.67:0/3082232391 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f40ac9f0 con 0x7f090c0ce400 2024-04-17T10:34:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 -- 172.21.15.67:0/3082232391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c0ce400 msgr2=0x7f090c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 --2- 172.21.15.67:0/3082232391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c0ce400 0x7f090c15e010 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f090c0d2840 tx=0x7f08f401c4f0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 -- 172.21.15.67:0/3082232391 shutdown_connections 2024-04-17T10:34:22.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 --2- 172.21.15.67:0/3082232391 >> v2:172.21.15.86:3300/0 conn(0x7f090c15e550 0x7f090c0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 --2- 172.21.15.67:0/3082232391 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c0ce400 0x7f090c15e010 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 --2- 172.21.15.67:0/3082232391 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 -- 172.21.15.67:0/3082232391 >> 172.21.15.67:0/3082232391 conn(0x7f090c0b33f0 msgr2=0x7f090c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 -- 172.21.15.67:0/3082232391 shutdown_connections 2024-04-17T10:34:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.697+0000 7f0910e00640 1 -- 172.21.15.67:0/3082232391 wait complete. 2024-04-17T10:34:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 Processor -- start 2024-04-17T10:34:22.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 -- start start 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f090c0ce400 0x7f090c14b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c15e550 0x7f090c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c06c0f0 con 0x7f090c15e550 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090c06bf70 con 0x7f090c0ce400 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090c0c84b0 con 0x7f090c0ce020 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c15e550 0x7f090c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c15e550 0x7f090c15d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35020/0 (socket says 172.21.15.67:35020) 2024-04-17T10:34:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 -- 172.21.15.67:0/1271923163 learned_addr learned my addr 172.21.15.67:0/1271923163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0903e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f090c0ce400 0x7f090c14b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0909e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 -- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 msgr2=0x7f090c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 -- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f090c0ce400 msgr2=0x7f090c14b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f090c0ce400 0x7f090c14b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 -- 172.21.15.67:0/1271923163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f090c159df0 con 0x7f090c15e550 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f090a800640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c15e550 0x7f090c15d460 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f0904007e70 tx=0x7f090400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0902000640 1 -- 172.21.15.67:0/1271923163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904014620 con 0x7f090c15e550 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0902000640 1 -- 172.21.15.67:0/1271923163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0904019070 con 0x7f090c15e550 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0902000640 1 -- 172.21.15.67:0/1271923163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0904014a20 con 0x7f090c15e550 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f090c15a060 con 0x7f090c15e550 2024-04-17T10:34:22.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f090c15a620 con 0x7f090c15e550 2024-04-17T10:34:22.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.701+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f090c070ab0 con 0x7f090c15e550 2024-04-17T10:34:22.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.705+0000 7f0902000640 1 -- 172.21.15.67:0/1271923163 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0904014bc0 con 0x7f090c15e550 2024-04-17T10:34:22.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.705+0000 7f0902000640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08e8066f90 0x7f08e8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:22.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.705+0000 7f0902000640 1 -- 172.21.15.67:0/1271923163 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f09040856c0 con 0x7f090c15e550 2024-04-17T10:34:22.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.705+0000 7f0909e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08e8066f90 0x7f08e8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:22.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.705+0000 7f0909e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08e8066f90 0x7f08e8069450 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0045f0 tx=0x7f08fc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:22.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.709+0000 7f0902000640 1 -- 172.21.15.67:0/1271923163 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0904056150 con 0x7f090c15e550 2024-04-17T10:34:22.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.905+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f090c074e30 con 0x7f090c15e550 2024-04-17T10:34:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.905+0000 7f0902000640 1 -- 172.21.15.67:0/1271923163 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f090405a1f0 con 0x7f090c15e550 2024-04-17T10:34:22.898 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:22.898 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":84,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7244,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6836/377638456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":377638456},{"type":"v1","addr":"172.21.15.86:6836","nonce":377638456}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":7247,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1683565836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":1683565836},{"type":"v1","addr":"172.21.15.86:6837","nonce":1683565836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":8089,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1662984555","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1662984555},{"type":"v1","addr":"172.21.15.67:6835","nonce":1662984555}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":9180,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2972683578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":2972683578},{"type":"v1","addr":"172.21.15.67:6834","nonce":2972683578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[]} 2024-04-17T10:34:22.898 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 84 2024-04-17T10:34:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08e8066f90 msgr2=0x7f08e8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08e8066f90 0x7f08e8069450 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f08fc0045f0 tx=0x7f08fc008040 comp rx=0 tx=0).stop 2024-04-17T10:34:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c15e550 msgr2=0x7f090c15d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c15e550 0x7f090c15d460 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f0904007e70 tx=0x7f090400e4b0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 shutdown_connections 2024-04-17T10:34:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08e8066f90 0x7f08e8069450 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f090c15e550 0x7f090c15d460 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f090c0ce400 0x7f090c14b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 --2- 172.21.15.67:0/1271923163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f090c0ce020 0x7f090c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:22.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 >> 172.21.15.67:0/1271923163 conn(0x7f090c0b33f0 msgr2=0x7f090c0cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:22.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 shutdown_connections 2024-04-17T10:34:22.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:22.909+0000 7f0910e00640 1 -- 172.21.15.67:0/1271923163 wait complete. 2024-04-17T10:34:22.920 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:34:22.920 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:34:22.920 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:34:22.920 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:34:22.920 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:34:22.920 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:34:22.921 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:34:22.921 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:34:22.921 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:34:22.921 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:34:22.921 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:34:22.921 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:34:22.922 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:34:22.922 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-17T10:34:22.922 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:34:22.922 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:34:22.932 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:34:22.932 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:34:22.933 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:34:22.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:34:22.937 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:34:22.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:34:22.938 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:34:22.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:34:22.978 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:34:22.978 DEBUG:teuthology.parallel:result is None 2024-04-17T10:34:22.979 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:34:22.980 DEBUG:teuthology.parallel:result is None 2024-04-17T10:34:22.986 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:34:22.986 DEBUG:teuthology.parallel:result is None 2024-04-17T10:34:22.988 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:34:22.988 DEBUG:teuthology.parallel:result is None 2024-04-17T10:34:22.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:34:23.065 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:34:23.065 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:34:23.081 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:34:23.081 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:34:23.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a4600640 1 Processor -- start 2024-04-17T10:34:23.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a4600640 1 -- start start 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a4600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f619c071310 0x7f619c071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a4600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f619c0cc300 0x7f619c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a4600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c06e2f0 con 0x7f619c145040 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a4600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c0a86c0 con 0x7f619c071310 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a4600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c071cb0 con 0x7f619c0cc300 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f619c071310 0x7f619c071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a1200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f619c071310 0x7f619c071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35074/0 (socket says 172.21.15.67:35074) 2024-04-17T10:34:23.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a1200640 1 -- 172.21.15.67:0/3674111615 learned_addr learned my addr 172.21.15.67:0/3674111615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a1c00640 1 --1- 172.21.15.67:0/3674111615 >> v1:172.21.15.67:6789/0 conn(0x7f619c145040 0x7f619c06e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42698/0 (socket says 172.21.15.67:42698) 2024-04-17T10:34:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.145+0000 7f61a2600640 1 --2- 172.21.15.67:0/3674111615 >> v2:172.21.15.86:3300/0 conn(0x7f619c0cc300 0x7f619c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a1200640 1 -- 172.21.15.67:0/3674111615 >> v2:172.21.15.86:3300/0 conn(0x7f619c0cc300 msgr2=0x7f619c0cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a1200640 1 --2- 172.21.15.67:0/3674111615 >> v2:172.21.15.86:3300/0 conn(0x7f619c0cc300 0x7f619c0cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a1200640 1 -- 172.21.15.67:0/3674111615 >> v1:172.21.15.67:6789/0 conn(0x7f619c145040 legacy=0x7f619c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a1200640 1 -- 172.21.15.67:0/3674111615 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c06d6e0 con 0x7f619c071310 2024-04-17T10:34:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f619be00640 1 -- 172.21.15.67:0/3674111615 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941033640 0 0) 0x7f619c06e2f0 con 0x7f619c145040 2024-04-17T10:34:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a1200640 1 --2- 172.21.15.67:0/3674111615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c071770 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f619c0af390 tx=0x7f618801ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ef90f73c725752 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f619be00640 1 -- 172.21.15.67:0/3674111615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61880b5070 con 0x7f619c071310 2024-04-17T10:34:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f619be00640 1 -- 172.21.15.67:0/3674111615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61880a8a10 con 0x7f619c071310 2024-04-17T10:34:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f619be00640 1 -- 172.21.15.67:0/3674111615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61880a8de0 con 0x7f619c071310 2024-04-17T10:34:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 -- 172.21.15.67:0/3674111615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 msgr2=0x7f619c071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 --2- 172.21.15.67:0/3674111615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c071770 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f619c0af390 tx=0x7f618801ed10 comp rx=0 tx=0).stop 2024-04-17T10:34:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 -- 172.21.15.67:0/3674111615 shutdown_connections 2024-04-17T10:34:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 --2- 172.21.15.67:0/3674111615 >> v2:172.21.15.86:3300/0 conn(0x7f619c0cc300 0x7f619c0cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 --2- 172.21.15.67:0/3674111615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c071770 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 -- 172.21.15.67:0/3674111615 >> 172.21.15.67:0/3674111615 conn(0x7f619c06ac90 msgr2=0x7f619c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:23.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 -- 172.21.15.67:0/3674111615 shutdown_connections 2024-04-17T10:34:23.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.149+0000 7f61a4600640 1 -- 172.21.15.67:0/3674111615 wait complete. 2024-04-17T10:34:23.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 Processor -- start 2024-04-17T10:34:23.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 -- start start 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c0c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f619c0cc300 0x7f619c15d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f619c145040 0x7f619c15dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c06df80 con 0x7f619c071310 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c06dca0 con 0x7f619c0cc300 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c06d370 con 0x7f619c145040 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c0c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c0c4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35080/0 (socket says 172.21.15.67:35080) 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 -- 172.21.15.67:0/3314077691 learned_addr learned my addr 172.21.15.67:0/3314077691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1200640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f619c0cc300 0x7f619c15d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:23.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a2600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f619c145040 0x7f619c15dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 -- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f619c145040 msgr2=0x7f619c15dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f619c145040 0x7f619c15dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 -- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f619c0cc300 msgr2=0x7f619c15d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f619c0cc300 0x7f619c15d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 -- 172.21.15.67:0/3314077691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c131f60 con 0x7f619c071310 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1200640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f619c0cc300 0x7f619c15d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a2600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f619c145040 0x7f619c15dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a1c00640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c0c4e10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f61900011f0 tx=0x7f619000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f6199600640 1 -- 172.21.15.67:0/3314077691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190018070 con 0x7f619c071310 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f6199600640 1 -- 172.21.15.67:0/3314077691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61900074e0 con 0x7f619c071310 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f6199600640 1 -- 172.21.15.67:0/3314077691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6190004a10 con 0x7f619c071310 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c132240 con 0x7f619c071310 2024-04-17T10:34:23.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.153+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c0c5350 con 0x7f619c071310 2024-04-17T10:34:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.157+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c072d50 con 0x7f619c071310 2024-04-17T10:34:23.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.157+0000 7f6199600640 1 -- 172.21.15.67:0/3314077691 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6190004bb0 con 0x7f619c071310 2024-04-17T10:34:23.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.157+0000 7f6199600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f617c066f60 0x7f617c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:23.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.157+0000 7f6199600640 1 -- 172.21.15.67:0/3314077691 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5335+0+0 (secure 0 0 0) 0x7f6190084350 con 0x7f619c071310 2024-04-17T10:34:23.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.157+0000 7f61a1200640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f617c066f60 0x7f617c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:23.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.161+0000 7f61a1200640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f617c066f60 0x7f617c069420 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f61880b2000 tx=0x7f61880aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:23.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.161+0000 7f6199600640 1 -- 172.21.15.67:0/3314077691 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6190055430 con 0x7f619c071310 2024-04-17T10:34:23.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:23.345+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f619c074e30 con 0x7f619c071310 2024-04-17T10:34:24.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.277+0000 7f6199600640 1 -- 172.21.15.67:0/3314077691 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v126) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6190054d30 con 0x7f619c071310 2024-04-17T10:34:24.271 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:34:24.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f617c066f60 msgr2=0x7f617c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f617c066f60 0x7f617c069420 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f61880b2000 tx=0x7f61880aa070 comp rx=0 tx=0).stop 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 msgr2=0x7f619c0c4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c0c4e10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f61900011f0 tx=0x7f619000c9b0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 shutdown_connections 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f617c066f60 0x7f617c069420 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f619c145040 0x7f619c15dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f619c0cc300 0x7f619c15d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 --2- 172.21.15.67:0/3314077691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f619c071310 0x7f619c0c4e10 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 >> 172.21.15.67:0/3314077691 conn(0x7f619c06ac90 msgr2=0x7f619c0b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:24.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 shutdown_connections 2024-04-17T10:34:24.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.281+0000 7f61a4600640 1 -- 172.21.15.67:0/3314077691 wait complete. 2024-04-17T10:34:24.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:34:24.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c49000640 1 Processor -- start 2024-04-17T10:34:24.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c49000640 1 -- start start 2024-04-17T10:34:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c49000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c440d10c0 0x7f8c440d1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c49000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c440ceb60 con 0x7f8c440ce3c0 2024-04-17T10:34:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c49000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c440cee40 con 0x7f8c4415dbd0 2024-04-17T10:34:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c49000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c440d3ee0 con 0x7f8c440d10c0 2024-04-17T10:34:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c3be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c440d10c0 0x7f8c440d1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c42800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8c440ce3c0 0x7f8c440b6af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42700/0 (socket says 172.21.15.67:42700) 2024-04-17T10:34:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c42800640 1 -- 172.21.15.67:0/778877698 learned_addr learned my addr 172.21.15.67:0/778877698 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c3be00640 1 -- 172.21.15.67:0/778877698 >> v1:172.21.15.86:6789/0 conn(0x7f8c4415dbd0 legacy=0x7f8c440d09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c3be00640 1 -- 172.21.15.67:0/778877698 >> v1:172.21.15.67:6789/0 conn(0x7f8c440ce3c0 legacy=0x7f8c440b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c3be00640 1 -- 172.21.15.67:0/778877698 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c440d3b70 con 0x7f8c440d10c0 2024-04-17T10:34:24.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.453+0000 7f8c41400640 1 -- 172.21.15.67:0/778877698 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456245431 0 0) 0x7f8c440ceb60 con 0x7f8c440ce3c0 2024-04-17T10:34:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c3be00640 1 --2- 172.21.15.67:0/778877698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440d10c0 0x7f8c440d1710 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0097c0 tx=0x7f8c2c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7731794cc6e11e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:24.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c41400640 1 -- 172.21.15.67:0/778877698 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c0b1070 con 0x7f8c440d10c0 2024-04-17T10:34:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c41400640 1 -- 172.21.15.67:0/778877698 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c2c0ac410 con 0x7f8c440d10c0 2024-04-17T10:34:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c41400640 1 -- 172.21.15.67:0/778877698 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c0ac7e0 con 0x7f8c440d10c0 2024-04-17T10:34:24.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c49000640 1 -- 172.21.15.67:0/778877698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440d10c0 msgr2=0x7f8c440d1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c49000640 1 --2- 172.21.15.67:0/778877698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440d10c0 0x7f8c440d1710 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0097c0 tx=0x7f8c2c01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c49000640 1 -- 172.21.15.67:0/778877698 shutdown_connections 2024-04-17T10:34:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c49000640 1 --2- 172.21.15.67:0/778877698 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440d10c0 0x7f8c440d1710 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c49000640 1 -- 172.21.15.67:0/778877698 >> 172.21.15.67:0/778877698 conn(0x7f8c440727c0 msgr2=0x7f8c44072bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c49000640 1 -- 172.21.15.67:0/778877698 shutdown_connections 2024-04-17T10:34:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.457+0000 7f8c49000640 1 -- 172.21.15.67:0/778877698 wait complete. 2024-04-17T10:34:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 Processor -- start 2024-04-17T10:34:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 -- start start 2024-04-17T10:34:24.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ce3c0 0x7f8c441596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c440d10c0 0x7f8c44159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c4415dbd0 0x7f8c4414bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c440b7790 con 0x7f8c440ce3c0 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c440b7610 con 0x7f8c440d10c0 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c4416bfa0 con 0x7f8c4415dbd0 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ce3c0 0x7f8c441596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ce3c0 0x7f8c441596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35096/0 (socket says 172.21.15.67:35096) 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 -- 172.21.15.67:0/2988787357 learned_addr learned my addr 172.21.15.67:0/2988787357 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c3be00640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c440d10c0 0x7f8c44159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c42800640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c4415dbd0 0x7f8c4414bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 -- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c4415dbd0 msgr2=0x7f8c4414bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c4415dbd0 0x7f8c4414bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 -- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c440d10c0 msgr2=0x7f8c44159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c440d10c0 0x7f8c44159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 -- 172.21.15.67:0/2988787357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c4415a300 con 0x7f8c440ce3c0 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c42800640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c4415dbd0 0x7f8c4414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c41e00640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ce3c0 0x7f8c441596b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400c3c0 tx=0x7f8c3400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c3a000640 1 -- 172.21.15.67:0/2988787357 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34017070 con 0x7f8c440ce3c0 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c3a000640 1 -- 172.21.15.67:0/2988787357 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c340052d0 con 0x7f8c440ce3c0 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c3a000640 1 -- 172.21.15.67:0/2988787357 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34003840 con 0x7f8c440ce3c0 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c4415a5a0 con 0x7f8c440ce3c0 2024-04-17T10:34:24.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.461+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c4415ab60 con 0x7f8c440ce3c0 2024-04-17T10:34:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.465+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c4408afe0 con 0x7f8c440ce3c0 2024-04-17T10:34:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.465+0000 7f8c3a000640 1 -- 172.21.15.67:0/2988787357 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8c3401a070 con 0x7f8c440ce3c0 2024-04-17T10:34:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.465+0000 7f8c3a000640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c20066f60 0x7f8c20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.465+0000 7f8c3a000640 1 -- 172.21.15.67:0/2988787357 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8c34084180 con 0x7f8c440ce3c0 2024-04-17T10:34:24.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.465+0000 7f8c3be00640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c20066f60 0x7f8c20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.465+0000 7f8c3be00640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c20066f60 0x7f8c20069420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0ae000 tx=0x7f8c2c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:24.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.469+0000 7f8c3a000640 1 -- 172.21.15.67:0/2988787357 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8c34005480 con 0x7f8c440ce3c0 2024-04-17T10:34:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.689+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8c441552d0 con 0x7f8c440ce3c0 2024-04-17T10:34:24.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.689+0000 7f8c3a000640 1 -- 172.21.15.67:0/2988787357 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8c34054c60 con 0x7f8c440ce3c0 2024-04-17T10:34:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:24.681 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:34:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:34:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:34:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.682 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.683 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.684 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.685 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.686 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.687 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:34:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:34:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.688 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:34:24.689 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:34:24.690 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:34:24.691 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:34:24.692 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:34:24.693 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.694 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:34:24.695 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.689+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c20066f60 msgr2=0x7f8c20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.689+0000 7f8c49000640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c20066f60 0x7f8c20069420 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c0ae000 tx=0x7f8c2c008040 comp rx=0 tx=0).stop 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.689+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ce3c0 msgr2=0x7f8c441596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.689+0000 7f8c49000640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ce3c0 0x7f8c441596b0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400c3c0 tx=0x7f8c3400c880 comp rx=0 tx=0).stop 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 shutdown_connections 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c20066f60 0x7f8c20069420 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c4415dbd0 0x7f8c4414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c440d10c0 0x7f8c44159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 --2- 172.21.15.67:0/2988787357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c440ce3c0 0x7f8c441596b0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 >> 172.21.15.67:0/2988787357 conn(0x7f8c440727c0 msgr2=0x7f8c440b43d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:24.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 shutdown_connections 2024-04-17T10:34:24.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.693+0000 7f8c49000640 1 -- 172.21.15.67:0/2988787357 wait complete. 2024-04-17T10:34:24.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:24.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d6800640 1 Processor -- start 2024-04-17T10:34:24.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d6800640 1 -- start start 2024-04-17T10:34:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d6800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d00cde20 con 0x7ff6d0099a50 2024-04-17T10:34:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d6800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d00cf300 con 0x7ff6d009a740 2024-04-17T10:34:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d6800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d00cf5e0 con 0x7ff6d00d03a0 2024-04-17T10:34:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff6d0099a50 0x7ff6d0099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42716/0 (socket says 172.21.15.67:42716) 2024-04-17T10:34:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d5400640 1 -- 172.21.15.67:0/3486483142 learned_addr learned my addr 172.21.15.67:0/3486483142 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:24.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d4a00640 1 --2- 172.21.15.67:0/3486483142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6cfe00640 1 -- 172.21.15.67:0/3486483142 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811768628 0 0) 0x7ff6d00cde20 con 0x7ff6d0099a50 2024-04-17T10:34:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6cfe00640 1 -- 172.21.15.67:0/3486483142 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b8003530 con 0x7ff6d0099a50 2024-04-17T10:34:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6cfe00640 1 -- 172.21.15.67:0/3486483142 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106715447 0 0) 0x7ff6d00cf300 con 0x7ff6d009a740 2024-04-17T10:34:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6cfe00640 1 -- 172.21.15.67:0/3486483142 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d00cde20 con 0x7ff6d009a740 2024-04-17T10:34:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d4a00640 1 -- 172.21.15.67:0/3486483142 >> v1:172.21.15.86:6789/0 conn(0x7ff6d009a740 legacy=0x7ff6d00cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d4a00640 1 -- 172.21.15.67:0/3486483142 >> v1:172.21.15.67:6789/0 conn(0x7ff6d0099a50 legacy=0x7ff6d0099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.853+0000 7ff6d4a00640 1 -- 172.21.15.67:0/3486483142 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d00cf8c0 con 0x7ff6d00d03a0 2024-04-17T10:34:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d4a00640 1 --2- 172.21.15.67:0/3486483142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d015c7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0097f0 tx=0x7ff6bc01c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b99636ec8389b16b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6cfe00640 1 -- 172.21.15.67:0/3486483142 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0b0070 con 0x7ff6d00d03a0 2024-04-17T10:34:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6cfe00640 1 -- 172.21.15.67:0/3486483142 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6bc0ab600 con 0x7ff6d00d03a0 2024-04-17T10:34:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6cfe00640 1 -- 172.21.15.67:0/3486483142 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc0ab9d0 con 0x7ff6d00d03a0 2024-04-17T10:34:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 -- 172.21.15.67:0/3486483142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 msgr2=0x7ff6d015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3486483142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d015c7a0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0097f0 tx=0x7ff6bc01c4e0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 -- 172.21.15.67:0/3486483142 shutdown_connections 2024-04-17T10:34:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3486483142 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d015c7a0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 -- 172.21.15.67:0/3486483142 >> 172.21.15.67:0/3486483142 conn(0x7ff6d00d1690 msgr2=0x7ff6d00d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 -- 172.21.15.67:0/3486483142 shutdown_connections 2024-04-17T10:34:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 -- 172.21.15.67:0/3486483142 wait complete. 2024-04-17T10:34:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 Processor -- start 2024-04-17T10:34:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.857+0000 7ff6d6800640 1 -- start start 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d0099a50 0x7ff6d0152150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6d009a740 0x7ff6d01579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d0155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0169a50 con 0x7ff6d0099a50 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d01698d0 con 0x7ff6d009a740 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0169bd0 con 0x7ff6d00d03a0 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d0099a50 0x7ff6d0152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d0099a50 0x7ff6d0152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35098/0 (socket says 172.21.15.67:35098) 2024-04-17T10:34:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 -- 172.21.15.67:0/3899917271 learned_addr learned my addr 172.21.15.67:0/3899917271 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d4a00640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6d009a740 0x7ff6d01579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5e00640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d0155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 -- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 msgr2=0x7ff6d0155360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d0155360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 -- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6d009a740 msgr2=0x7ff6d01579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6d009a740 0x7ff6d01579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 -- 172.21.15.67:0/3899917271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d00b1b00 con 0x7ff6d0099a50 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5e00640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d0155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d5400640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d0099a50 0x7ff6d0152150 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400c420 tx=0x7ff6c400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6cd600640 1 -- 172.21.15.67:0/3899917271 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4017070 con 0x7ff6d0099a50 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6cd600640 1 -- 172.21.15.67:0/3899917271 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6c4005cf0 con 0x7ff6d0099a50 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d00b1de0 con 0x7ff6d0099a50 2024-04-17T10:34:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6cd600640 1 -- 172.21.15.67:0/3899917271 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4007310 con 0x7ff6d0099a50 2024-04-17T10:34:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d00b23a0 con 0x7ff6d0099a50 2024-04-17T10:34:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.861+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d0074dc0 con 0x7ff6d0099a50 2024-04-17T10:34:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.865+0000 7ff6cd600640 1 -- 172.21.15.67:0/3899917271 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff6c401a070 con 0x7ff6d0099a50 2024-04-17T10:34:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.865+0000 7ff6cd600640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6b0066f60 0x7ff6b0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.865+0000 7ff6cd600640 1 -- 172.21.15.67:0/3899917271 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5249+0+0 (secure 0 0 0) 0x7ff6c4084260 con 0x7ff6d0099a50 2024-04-17T10:34:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.865+0000 7ff6d4a00640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6b0066f60 0x7ff6b0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.865+0000 7ff6d4a00640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6b0066f60 0x7ff6b0069420 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ad000 tx=0x7ff6bc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:24.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:24.869+0000 7ff6cd600640 1 -- 172.21.15.67:0/3899917271 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff6c4054d30 con 0x7ff6d0099a50 2024-04-17T10:34:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.033+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d009af30 con 0x7ff6d0099a50 2024-04-17T10:34:25.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.033+0000 7ff6cd600640 1 -- 172.21.15.67:0/3899917271 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+13126 (secure 0 0 0) 0x7ff6c4058dd0 con 0x7ff6d0099a50 2024-04-17T10:34:25.027 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:25.027 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":126,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:34:24.280964+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6b0066f60 msgr2=0x7ff6b0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6b0066f60 0x7ff6b0069420 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc0ad000 tx=0x7ff6bc008040 comp rx=0 tx=0).stop 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d0099a50 msgr2=0x7ff6d0152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d0099a50 0x7ff6d0152150 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff6c400c420 tx=0x7ff6c400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 shutdown_connections 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff6b0066f60 0x7ff6b0069420 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff6d00d03a0 0x7ff6d0155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff6d009a740 0x7ff6d01579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 --2- 172.21.15.67:0/3899917271 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6d0099a50 0x7ff6d0152150 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 >> 172.21.15.67:0/3899917271 conn(0x7ff6d00d1690 msgr2=0x7ff6d00af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 shutdown_connections 2024-04-17T10:34:25.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.037+0000 7ff6d6800640 1 -- 172.21.15.67:0/3899917271 wait complete. 2024-04-17T10:34:25.048 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:34:25.048 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:34:25.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9c2000640 1 Processor -- start 2024-04-17T10:34:25.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9c2000640 1 -- start start 2024-04-17T10:34:25.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9c2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9bc15bd60 0x7fc9bc099b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9c2000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc9bc0cfdf0 0x7fc9bc09a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9c2000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9bc0cca60 con 0x7fc9bc15b980 2024-04-17T10:34:25.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9c2000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9bc0ccd40 con 0x7fc9bc15bd60 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9c2000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9bc0cd020 con 0x7fc9bc0cfdf0 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9b2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc9bc15bd60 0x7fc9bc099b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9bb400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc9bc15b980 0x7fc9bc0cc1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42728/0 (socket says 172.21.15.67:42728) 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9bb400640 1 -- 172.21.15.67:0/2630403899 learned_addr learned my addr 172.21.15.67:0/2630403899 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9bbe00640 1 --2- 172.21.15.67:0/2630403899 >> v2:172.21.15.86:3300/0 conn(0x7fc9bc0cfdf0 0x7fc9bc09a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9b2a00640 1 -- 172.21.15.67:0/2630403899 >> v2:172.21.15.86:3300/0 conn(0x7fc9bc0cfdf0 msgr2=0x7fc9bc09a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9b2a00640 1 --2- 172.21.15.67:0/2630403899 >> v2:172.21.15.86:3300/0 conn(0x7fc9bc0cfdf0 0x7fc9bc09a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9b2a00640 1 -- 172.21.15.67:0/2630403899 >> v1:172.21.15.67:6789/0 conn(0x7fc9bc15b980 legacy=0x7fc9bc0cc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.201+0000 7fc9b2a00640 1 -- 172.21.15.67:0/2630403899 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc0cd300 con 0x7fc9bc15bd60 2024-04-17T10:34:25.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9b2a00640 1 --2- 172.21.15.67:0/2630403899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc099b30 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0b42a0 tx=0x7fc9a401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db629d62e10e9924 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9baa00640 1 -- 172.21.15.67:0/2630403899 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40b5070 con 0x7fc9bc15bd60 2024-04-17T10:34:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9baa00640 1 -- 172.21.15.67:0/2630403899 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a40a8970 con 0x7fc9bc15bd60 2024-04-17T10:34:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9baa00640 1 -- 172.21.15.67:0/2630403899 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a40a8d40 con 0x7fc9bc15bd60 2024-04-17T10:34:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 -- 172.21.15.67:0/2630403899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 msgr2=0x7fc9bc099b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 --2- 172.21.15.67:0/2630403899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc099b30 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0b42a0 tx=0x7fc9a401ec70 comp rx=0 tx=0).stop 2024-04-17T10:34:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 -- 172.21.15.67:0/2630403899 shutdown_connections 2024-04-17T10:34:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 --2- 172.21.15.67:0/2630403899 >> v2:172.21.15.86:3300/0 conn(0x7fc9bc0cfdf0 0x7fc9bc09a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 --2- 172.21.15.67:0/2630403899 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc099b30 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 -- 172.21.15.67:0/2630403899 >> 172.21.15.67:0/2630403899 conn(0x7fc9bc0b33a0 msgr2=0x7fc9bc0b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 -- 172.21.15.67:0/2630403899 shutdown_connections 2024-04-17T10:34:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.205+0000 7fc9c2000640 1 -- 172.21.15.67:0/2630403899 wait complete. 2024-04-17T10:34:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 Processor -- start 2024-04-17T10:34:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 -- start start 2024-04-17T10:34:25.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9bc0cfdf0 0x7fc9bc0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9bc15b980 0x7fc9bc155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc0cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0cdc80 con 0x7fc9bc15bd60 2024-04-17T10:34:25.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0cdb00 con 0x7fc9bc0cfdf0 2024-04-17T10:34:25.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc0cde00 con 0x7fc9bc15b980 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc0cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc0cafd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35126/0 (socket says 172.21.15.67:35126) 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 -- 172.21.15.67:0/109335733 learned_addr learned my addr 172.21.15.67:0/109335733 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bb400640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9bc0cfdf0 0x7fc9bc0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9b2a00640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9bc15b980 0x7fc9bc155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 -- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9bc15b980 msgr2=0x7fc9bc155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9bc15b980 0x7fc9bc155e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 -- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9bc0cfdf0 msgr2=0x7fc9bc0c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9bc0cfdf0 0x7fc9bc0c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 -- 172.21.15.67:0/109335733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9bc1563b0 con 0x7fc9bc15bd60 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9bbe00640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc0cafd0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400efc0 tx=0x7fc9b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9b3e00640 1 -- 172.21.15.67:0/109335733 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4018020 con 0x7fc9bc15bd60 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9b3e00640 1 -- 172.21.15.67:0/109335733 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b4014480 con 0x7fc9bc15bd60 2024-04-17T10:34:25.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9b3e00640 1 -- 172.21.15.67:0/109335733 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4014870 con 0x7fc9bc15bd60 2024-04-17T10:34:25.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc0b63a0 con 0x7fc9bc15bd60 2024-04-17T10:34:25.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc0b6c00 con 0x7fc9bc15bd60 2024-04-17T10:34:25.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.209+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc04ef30 con 0x7fc9bc15bd60 2024-04-17T10:34:25.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.213+0000 7fc9b3e00640 1 -- 172.21.15.67:0/109335733 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc9b401b020 con 0x7fc9bc15bd60 2024-04-17T10:34:25.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.213+0000 7fc9b3e00640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998066f60 0x7fc998069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.213+0000 7fc9b3e00640 1 -- 172.21.15.67:0/109335733 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fc9b4085b70 con 0x7fc9bc15bd60 2024-04-17T10:34:25.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.213+0000 7fc9bb400640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998066f60 0x7fc998069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.213+0000 7fc9bb400640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998066f60 0x7fc998069420 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac000f70 tx=0x7fc9ac007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.217+0000 7fc9b3e00640 1 -- 172.21.15.67:0/109335733 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc9b4022030 con 0x7fc9bc15bd60 2024-04-17T10:34:25.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc9bc0afb60 con 0x7fc9bc15bd60 2024-04-17T10:34:25.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9b3e00640 1 -- 172.21.15.67:0/109335733 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9b4010040 con 0x7fc9bc15bd60 2024-04-17T10:34:25.372 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:34:25.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998066f60 msgr2=0x7fc998069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998066f60 0x7fc998069420 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac000f70 tx=0x7fc9ac007e90 comp rx=0 tx=0).stop 2024-04-17T10:34:25.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 msgr2=0x7fc9bc0cafd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc0cafd0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400efc0 tx=0x7fc9b400c490 comp rx=0 tx=0).stop 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 shutdown_connections 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc998066f60 0x7fc998069420 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc9bc15bd60 0x7fc9bc0cafd0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc9bc15b980 0x7fc9bc155e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 --2- 172.21.15.67:0/109335733 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc9bc0cfdf0 0x7fc9bc0c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 >> 172.21.15.67:0/109335733 conn(0x7fc9bc0b33a0 msgr2=0x7fc9bc06f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 shutdown_connections 2024-04-17T10:34:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.381+0000 7fc9c2000640 1 -- 172.21.15.67:0/109335733 wait complete. 2024-04-17T10:34:25.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:34:25.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.541+0000 7fb9d2200640 1 Processor -- start 2024-04-17T10:34:25.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.541+0000 7fb9d2200640 1 -- start start 2024-04-17T10:34:25.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9d2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9cc0ce020 0x7fb9cc1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9d2200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb9cc0ce400 0x7fb9cc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9d2200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9cc15eb50 con 0x7fb9cc15e550 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9d2200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9cc161650 con 0x7fb9cc0ce020 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9d2200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9cc1625e0 con 0x7fb9cc0ce400 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9cb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9cc0ce020 0x7fb9cc1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9cb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb9cc0ce020 0x7fb9cc1620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35136/0 (socket says 172.21.15.67:35136) 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9cb400640 1 -- 172.21.15.67:0/2259417483 learned_addr learned my addr 172.21.15.67:0/2259417483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9c2a00640 1 --2- 172.21.15.67:0/2259417483 >> v2:172.21.15.86:3300/0 conn(0x7fb9cc0ce400 0x7fb9cc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9cb400640 1 -- 172.21.15.67:0/2259417483 >> v2:172.21.15.86:3300/0 conn(0x7fb9cc0ce400 msgr2=0x7fb9cc15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9cb400640 1 --2- 172.21.15.67:0/2259417483 >> v2:172.21.15.86:3300/0 conn(0x7fb9cc0ce400 0x7fb9cc15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9cb400640 1 -- 172.21.15.67:0/2259417483 >> v1:172.21.15.86:6789/0 conn(0x7fb9cc15e550 legacy=0x7fb9cc0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9cb400640 1 -- 172.21.15.67:0/2259417483 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc15ee30 con 0x7fb9cc0ce020 2024-04-17T10:34:25.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.545+0000 7fb9c2a00640 1 --2- 172.21.15.67:0/2259417483 >> v2:172.21.15.86:3300/0 conn(0x7fb9cc0ce400 0x7fb9cc15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:25.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9cb400640 1 --2- 172.21.15.67:0/2259417483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce020 0x7fb9cc1620a0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009870 tx=0x7fb9bc01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67075440083ccc6d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9caa00640 1 -- 172.21.15.67:0/2259417483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc0b1070 con 0x7fb9cc0ce020 2024-04-17T10:34:25.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9caa00640 1 -- 172.21.15.67:0/2259417483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc0ac630 con 0x7fb9cc0ce020 2024-04-17T10:34:25.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9caa00640 1 -- 172.21.15.67:0/2259417483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc0a8050 con 0x7fb9cc0ce020 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 -- 172.21.15.67:0/2259417483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce020 msgr2=0x7fb9cc1620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 --2- 172.21.15.67:0/2259417483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce020 0x7fb9cc1620a0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc009870 tx=0x7fb9bc01c4e0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 -- 172.21.15.67:0/2259417483 shutdown_connections 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 --2- 172.21.15.67:0/2259417483 >> v2:172.21.15.86:3300/0 conn(0x7fb9cc0ce400 0x7fb9cc15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 --2- 172.21.15.67:0/2259417483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce020 0x7fb9cc1620a0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 -- 172.21.15.67:0/2259417483 >> 172.21.15.67:0/2259417483 conn(0x7fb9cc0b33f0 msgr2=0x7fb9cc0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 -- 172.21.15.67:0/2259417483 shutdown_connections 2024-04-17T10:34:25.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 -- 172.21.15.67:0/2259417483 wait complete. 2024-04-17T10:34:25.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 Processor -- start 2024-04-17T10:34:25.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.549+0000 7fb9d2200640 1 -- start start 2024-04-17T10:34:25.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9cc0ce020 0x7fb9cc0c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc15e550 0x7fb9cc159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc0ce990 con 0x7fb9cc0ce400 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc0ce810 con 0x7fb9cc15e550 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9cc0c84b0 con 0x7fb9cc0ce020 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc15b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35138/0 (socket says 172.21.15.67:35138) 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 -- 172.21.15.67:0/3515068892 learned_addr learned my addr 172.21.15.67:0/3515068892 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9cb400640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9cc0ce020 0x7fb9cc0c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9cbe00640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc15e550 0x7fb9cc159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 -- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9cc0ce020 msgr2=0x7fb9cc0c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9cc0ce020 0x7fb9cc0c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 -- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc15e550 msgr2=0x7fb9cc159c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc15e550 0x7fb9cc159c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 -- 172.21.15.67:0/3515068892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9cc15a4a0 con 0x7fb9cc0ce400 2024-04-17T10:34:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c2a00640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc15b540 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009fc0 tx=0x7fb9b400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c3e00640 1 -- 172.21.15.67:0/3515068892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b401a070 con 0x7fb9cc0ce400 2024-04-17T10:34:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c3e00640 1 -- 172.21.15.67:0/3515068892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b4003bc0 con 0x7fb9cc0ce400 2024-04-17T10:34:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9cc15a780 con 0x7fb9cc0ce400 2024-04-17T10:34:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c3e00640 1 -- 172.21.15.67:0/3515068892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4004030 con 0x7fb9cc0ce400 2024-04-17T10:34:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9cc15ad40 con 0x7fb9cc0ce400 2024-04-17T10:34:25.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9cc070ab0 con 0x7fb9cc0ce400 2024-04-17T10:34:25.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.553+0000 7fb9c3e00640 1 -- 172.21.15.67:0/3515068892 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb9b401d070 con 0x7fb9cc0ce400 2024-04-17T10:34:25.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.557+0000 7fb9c3e00640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb9a8066f60 0x7fb9a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.557+0000 7fb9c3e00640 1 -- 172.21.15.67:0/3515068892 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fb9b40851f0 con 0x7fb9cc0ce400 2024-04-17T10:34:25.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.557+0000 7fb9cb400640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb9a8066f60 0x7fb9a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.557+0000 7fb9cb400640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb9a8066f60 0x7fb9a8069420 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0ae000 tx=0x7fb9bc0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.561+0000 7fb9c3e00640 1 -- 172.21.15.67:0/3515068892 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb9b4010070 con 0x7fb9cc0ce400 2024-04-17T10:34:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb9cc0cf040 con 0x7fb9cc0ce400 2024-04-17T10:34:25.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9c3e00640 1 -- 172.21.15.67:0/3515068892 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v126) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb9b4055c40 con 0x7fb9cc0ce400 2024-04-17T10:34:25.715 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:25.716 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:34:25.716 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:34:25.716 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:34:25.716 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:34:25.716 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:25.716 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:34:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb9a8066f60 msgr2=0x7fb9a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb9a8066f60 0x7fb9a8069420 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc0ae000 tx=0x7fb9bc0a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce400 msgr2=0x7fb9cc15b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc15b540 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4009fc0 tx=0x7fb9b400d560 comp rx=0 tx=0).stop 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 shutdown_connections 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb9a8066f60 0x7fb9a8069420 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb9cc15e550 0x7fb9cc159c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb9cc0ce400 0x7fb9cc15b540 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 --2- 172.21.15.67:0/3515068892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9cc0ce020 0x7fb9cc0c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 >> 172.21.15.67:0/3515068892 conn(0x7fb9cc0b33f0 msgr2=0x7fb9cc073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 shutdown_connections 2024-04-17T10:34:25.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.725+0000 7fb9d2200640 1 -- 172.21.15.67:0/3515068892 wait complete. 2024-04-17T10:34:25.736 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:34:25.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:34:25.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.889+0000 7f6173e00640 1 Processor -- start 2024-04-17T10:34:25.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.889+0000 7f6173e00640 1 -- start start 2024-04-17T10:34:25.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.889+0000 7f6173e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f616c1447e0 0x7f616c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.889+0000 7f6173e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f616c071a70 0x7f616c0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.889+0000 7f6173e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c06d9a0 con 0x7f616c144f00 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6173e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0cc310 con 0x7f616c071a70 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6173e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c06dc80 con 0x7f616c1447e0 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f616c071a70 0x7f616c0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6170a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f616c144f00 0x7f616c071360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42744/0 (socket says 172.21.15.67:42744) 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f616c071a70 0x7f616c0cc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35146/0 (socket says 172.21.15.67:35146) 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6170a00640 1 -- 172.21.15.67:0/1432588854 learned_addr learned my addr 172.21.15.67:0/1432588854 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171400640 1 --2- 172.21.15.67:0/1432588854 >> v2:172.21.15.86:3300/0 conn(0x7f616c1447e0 0x7f616c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171e00640 1 -- 172.21.15.67:0/1432588854 >> v2:172.21.15.86:3300/0 conn(0x7f616c1447e0 msgr2=0x7f616c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171e00640 1 --2- 172.21.15.67:0/1432588854 >> v2:172.21.15.86:3300/0 conn(0x7f616c1447e0 0x7f616c06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171e00640 1 -- 172.21.15.67:0/1432588854 >> v1:172.21.15.67:6789/0 conn(0x7f616c144f00 legacy=0x7f616c071360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171e00640 1 -- 172.21.15.67:0/1432588854 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c06df60 con 0x7f616c071a70 2024-04-17T10:34:25.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f6171e00640 1 --2- 172.21.15.67:0/1432588854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c071a70 0x7f616c0cc7a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f616c072410 tx=0x7f616401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891a9f6d414934af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f616be00640 1 -- 172.21.15.67:0/1432588854 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640b2070 con 0x7f616c071a70 2024-04-17T10:34:25.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f616be00640 1 -- 172.21.15.67:0/1432588854 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61640a5950 con 0x7f616c071a70 2024-04-17T10:34:25.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.893+0000 7f616be00640 1 -- 172.21.15.67:0/1432588854 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640a5d20 con 0x7f616c071a70 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- 172.21.15.67:0/1432588854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c071a70 msgr2=0x7f616c0cc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 --2- 172.21.15.67:0/1432588854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c071a70 0x7f616c0cc7a0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f616c072410 tx=0x7f616401c870 comp rx=0 tx=0).stop 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- 172.21.15.67:0/1432588854 shutdown_connections 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 --2- 172.21.15.67:0/1432588854 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c071a70 0x7f616c0cc7a0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 --2- 172.21.15.67:0/1432588854 >> v2:172.21.15.86:3300/0 conn(0x7f616c1447e0 0x7f616c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- 172.21.15.67:0/1432588854 >> 172.21.15.67:0/1432588854 conn(0x7f616c06ac90 msgr2=0x7f616c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- 172.21.15.67:0/1432588854 shutdown_connections 2024-04-17T10:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- 172.21.15.67:0/1432588854 wait complete. 2024-04-17T10:34:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 Processor -- start 2024-04-17T10:34:25.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- start start 2024-04-17T10:34:25.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f616c071a70 0x7f616c141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c1447e0 0x7f616c13b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f616c144f00 0x7f616c15aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0cd880 con 0x7f616c1447e0 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0cd700 con 0x7f616c071a70 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6173e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0cda00 con 0x7f616c144f00 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c1447e0 0x7f616c13b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c1447e0 0x7f616c13b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35162/0 (socket says 172.21.15.67:35162) 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 -- 172.21.15.67:0/93695140 learned_addr learned my addr 172.21.15.67:0/93695140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6171400640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f616c071a70 0x7f616c141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6171e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f616c144f00 0x7f616c15aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 -- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f616c144f00 msgr2=0x7f616c15aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f616c144f00 0x7f616c15aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 -- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f616c071a70 msgr2=0x7f616c141590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f616c071a70 0x7f616c141590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6170a00640 1 -- 172.21.15.67:0/93695140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c141e50 con 0x7f616c1447e0 2024-04-17T10:34:25.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.897+0000 7f6171e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f616c144f00 0x7f616c15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6170a00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c1447e0 0x7f616c13b950 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f615800c370 tx=0x7f615800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6169600640 1 -- 172.21.15.67:0/93695140 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6158017070 con 0x7f616c1447e0 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6169600640 1 -- 172.21.15.67:0/93695140 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6158005cf0 con 0x7f616c1447e0 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c142110 con 0x7f616c1447e0 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6169600640 1 -- 172.21.15.67:0/93695140 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61580072f0 con 0x7f616c1447e0 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c1426d0 con 0x7f616c1447e0 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c072da0 con 0x7f616c1447e0 2024-04-17T10:34:25.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6169600640 1 -- 172.21.15.67:0/93695140 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f615801a070 con 0x7f616c1447e0 2024-04-17T10:34:25.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6169600640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f614c066f60 0x7f614c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:25.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6169600640 1 -- 172.21.15.67:0/93695140 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f6158084200 con 0x7f616c1447e0 2024-04-17T10:34:25.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.901+0000 7f6171400640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f614c066f60 0x7f614c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:25.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.905+0000 7f6171400640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f614c066f60 0x7f614c069420 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6160007a60 tx=0x7f6160008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:25.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:25.905+0000 7f6169600640 1 -- 172.21.15.67:0/93695140 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6158054ce0 con 0x7f616c1447e0 2024-04-17T10:34:26.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.069+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f616c074e30 con 0x7f616c1447e0 2024-04-17T10:34:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.925+0000 7f6169600640 1 -- 172.21.15.67:0/93695140 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v127) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6158058d80 con 0x7f616c1447e0 2024-04-17T10:34:26.916 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:34:26.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f614c066f60 msgr2=0x7f614c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f614c066f60 0x7f614c069420 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f6160007a60 tx=0x7f6160008040 comp rx=0 tx=0).stop 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c1447e0 msgr2=0x7f616c13b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c1447e0 0x7f616c13b950 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f615800c370 tx=0x7f615800c830 comp rx=0 tx=0).stop 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 shutdown_connections 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f614c066f60 0x7f614c069420 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f616c144f00 0x7f616c15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f616c1447e0 0x7f616c13b950 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 --2- 172.21.15.67:0/93695140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f616c071a70 0x7f616c141590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 >> 172.21.15.67:0/93695140 conn(0x7f616c06ac90 msgr2=0x7f616c0b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 shutdown_connections 2024-04-17T10:34:26.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:26.929+0000 7f6173e00640 1 -- 172.21.15.67:0/93695140 wait complete. 2024-04-17T10:34:26.948 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:34:27.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aef200640 1 Processor -- start 2024-04-17T10:34:27.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aef200640 1 -- start start 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aef200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af00b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aef200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3af006c7b0 0x7f3af00cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aef200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af00cf9d0 con 0x7f3af0099a50 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aef200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af006cce0 con 0x7f3af006c7b0 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aef200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00cfc40 con 0x7f3af0160f40 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3ae7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3af006c7b0 0x7f3af00cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3ae7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3af006c7b0 0x7f3af00cbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35174/0 (socket says 172.21.15.67:35174) 2024-04-17T10:34:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3ae7e00640 1 -- 172.21.15.67:0/2759325638 learned_addr learned my addr 172.21.15.67:0/2759325638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:27.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aee800640 1 --1- 172.21.15.67:0/2759325638 >> v1:172.21.15.67:6789/0 conn(0x7f3af0099a50 0x7f3af00d0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42756/0 (socket says 172.21.15.67:42756) 2024-04-17T10:34:27.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aede00640 1 --2- 172.21.15.67:0/2759325638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af00b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:27.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3ae7e00640 1 -- 172.21.15.67:0/2759325638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 msgr2=0x7f3af00b2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:27.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3ae7e00640 1 --2- 172.21.15.67:0/2759325638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af00b2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3ae7e00640 1 -- 172.21.15.67:0/2759325638 >> v1:172.21.15.67:6789/0 conn(0x7f3af0099a50 legacy=0x7f3af00d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:27.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3ae7e00640 1 -- 172.21.15.67:0/2759325638 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af00cff20 con 0x7f3af006c7b0 2024-04-17T10:34:27.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.109+0000 7f3aed400640 1 -- 172.21.15.67:0/2759325638 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352140338 0 0) 0x7f3af00cf9d0 con 0x7f3af0099a50 2024-04-17T10:34:27.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3ae7e00640 1 --2- 172.21.15.67:0/2759325638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af006c7b0 0x7f3af00cbcf0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009870 tx=0x7f3ad801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de9874cadbd9187f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aed400640 1 -- 172.21.15.67:0/2759325638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad80b1070 con 0x7f3af006c7b0 2024-04-17T10:34:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aed400640 1 -- 172.21.15.67:0/2759325638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad80ac410 con 0x7f3af006c7b0 2024-04-17T10:34:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aed400640 1 -- 172.21.15.67:0/2759325638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad80ac7e0 con 0x7f3af006c7b0 2024-04-17T10:34:27.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 -- 172.21.15.67:0/2759325638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af006c7b0 msgr2=0x7f3af00cbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 --2- 172.21.15.67:0/2759325638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af006c7b0 0x7f3af00cbcf0 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009870 tx=0x7f3ad801c560 comp rx=0 tx=0).stop 2024-04-17T10:34:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 -- 172.21.15.67:0/2759325638 shutdown_connections 2024-04-17T10:34:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 --2- 172.21.15.67:0/2759325638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af006c7b0 0x7f3af00cbcf0 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 --2- 172.21.15.67:0/2759325638 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af00b2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 -- 172.21.15.67:0/2759325638 >> 172.21.15.67:0/2759325638 conn(0x7f3af00b3190 msgr2=0x7f3af00b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 -- 172.21.15.67:0/2759325638 shutdown_connections 2024-04-17T10:34:27.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.113+0000 7f3aef200640 1 -- 172.21.15.67:0/2759325638 wait complete. 2024-04-17T10:34:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 Processor -- start 2024-04-17T10:34:27.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 -- start start 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af006c7b0 0x7f3af01506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af0099a50 0x7f3af0156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af015ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af01614a0 con 0x7f3af0099a50 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0161320 con 0x7f3af006c7b0 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af016aa80 con 0x7f3af0160f40 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af0099a50 0x7f3af0156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af0099a50 0x7f3af0156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35176/0 (socket says 172.21.15.67:35176) 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 -- 172.21.15.67:0/876013959 learned_addr learned my addr 172.21.15.67:0/876013959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aede00640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af006c7b0 0x7f3af01506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aee800640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af015ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 -- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 msgr2=0x7f3af015ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:27.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af015ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 -- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af006c7b0 msgr2=0x7f3af01506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af006c7b0 0x7f3af01506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 -- 172.21.15.67:0/876013959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af0157580 con 0x7f3af0099a50 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aede00640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af006c7b0 0x7f3af01506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae7e00640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af0099a50 0x7f3af0156380 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009870 tx=0x7f3ad8019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae6000640 1 -- 172.21.15.67:0/876013959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad80b1040 con 0x7f3af0099a50 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae6000640 1 -- 172.21.15.67:0/876013959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad80acd80 con 0x7f3af0099a50 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3ae6000640 1 -- 172.21.15.67:0/876013959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad8004240 con 0x7f3af0099a50 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af01511c0 con 0x7f3af0099a50 2024-04-17T10:34:27.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.117+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af0151780 con 0x7f3af0099a50 2024-04-17T10:34:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.121+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af0074dc0 con 0x7f3af0099a50 2024-04-17T10:34:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.121+0000 7f3ae6000640 1 -- 172.21.15.67:0/876013959 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3ad80be060 con 0x7f3af0099a50 2024-04-17T10:34:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.121+0000 7f3ae6000640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3acc066f60 0x7f3acc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.121+0000 7f3ae6000640 1 -- 172.21.15.67:0/876013959 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f3ad811df90 con 0x7f3af0099a50 2024-04-17T10:34:27.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.121+0000 7f3aede00640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3acc066f60 0x7f3acc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:27.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.121+0000 7f3aede00640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3acc066f60 0x7f3acc069420 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f3af00c8120 tx=0x7f3ae0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:27.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.125+0000 7f3ae6000640 1 -- 172.21.15.67:0/876013959 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3ad8016150 con 0x7f3af0099a50 2024-04-17T10:34:27.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.293+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3af00cdad0 con 0x7f3af0099a50 2024-04-17T10:34:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.925+0000 7f3ae6000640 1 -- 172.21.15.67:0/876013959 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v128) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3ad80be350 con 0x7f3af0099a50 2024-04-17T10:34:27.919 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:34:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3acc066f60 msgr2=0x7f3acc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3acc066f60 0x7f3acc069420 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f3af00c8120 tx=0x7f3ae0008040 comp rx=0 tx=0).stop 2024-04-17T10:34:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af0099a50 msgr2=0x7f3af0156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af0099a50 0x7f3af0156380 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009870 tx=0x7f3ad8019300 comp rx=0 tx=0).stop 2024-04-17T10:34:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 shutdown_connections 2024-04-17T10:34:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3acc066f60 0x7f3acc069420 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3af0160f40 0x7f3af015ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3af0099a50 0x7f3af0156380 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 --2- 172.21.15.67:0/876013959 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3af006c7b0 0x7f3af01506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:27.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.929+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 >> 172.21.15.67:0/876013959 conn(0x7f3af00b3190 msgr2=0x7f3af00d0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.933+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 shutdown_connections 2024-04-17T10:34:27.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:27.933+0000 7f3aef200640 1 -- 172.21.15.67:0/876013959 wait complete. 2024-04-17T10:34:27.936 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:34:27.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:34:27.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:34:28.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cce400640 1 Processor -- start 2024-04-17T10:34:28.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cce400640 1 -- start start 2024-04-17T10:34:28.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cce400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1cc80cf8a0 0x7f1cc80ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:28.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cce400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc80d0040 con 0x7f1cc815dcf0 2024-04-17T10:34:28.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cce400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc80d0320 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cce400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc80d0600 con 0x7f1cc80cf8a0 2024-04-17T10:34:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cbfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1cc80cf8a0 0x7f1cc80ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cbfe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1cc80cf8a0 0x7f1cc80ce090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35190/0 (socket says 172.21.15.67:35190) 2024-04-17T10:34:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cbfe00640 1 -- 172.21.15.67:0/1070846366 learned_addr learned my addr 172.21.15.67:0/1070846366 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cbfe00640 1 -- 172.21.15.67:0/1070846366 >> v1:172.21.15.86:6789/0 conn(0x7f1cc80ce5d0 legacy=0x7f1cc80b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cbfe00640 1 -- 172.21.15.67:0/1070846366 >> v1:172.21.15.67:6789/0 conn(0x7f1cc815dcf0 legacy=0x7f1cc806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:28.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.165+0000 7f1cbfe00640 1 -- 172.21.15.67:0/1070846366 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80d08e0 con 0x7f1cc80cf8a0 2024-04-17T10:34:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cbfe00640 1 --2- 172.21.15.67:0/1070846366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80cf8a0 0x7f1cc80ce090 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009870 tx=0x7f1cb001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14bdb28f11352143 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cc6a00640 1 -- 172.21.15.67:0/1070846366 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00b1070 con 0x7f1cc80cf8a0 2024-04-17T10:34:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cc6a00640 1 -- 172.21.15.67:0/1070846366 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb00ac630 con 0x7f1cc80cf8a0 2024-04-17T10:34:28.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cc6a00640 1 -- 172.21.15.67:0/1070846366 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb00a8050 con 0x7f1cc80cf8a0 2024-04-17T10:34:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cce400640 1 -- 172.21.15.67:0/1070846366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80cf8a0 msgr2=0x7f1cc80ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cce400640 1 --2- 172.21.15.67:0/1070846366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80cf8a0 0x7f1cc80ce090 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009870 tx=0x7f1cb001c520 comp rx=0 tx=0).stop 2024-04-17T10:34:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cce400640 1 -- 172.21.15.67:0/1070846366 shutdown_connections 2024-04-17T10:34:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cce400640 1 --2- 172.21.15.67:0/1070846366 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80cf8a0 0x7f1cc80ce090 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cce400640 1 -- 172.21.15.67:0/1070846366 >> 172.21.15.67:0/1070846366 conn(0x7f1cc80b33a0 msgr2=0x7f1cc80b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cce400640 1 -- 172.21.15.67:0/1070846366 shutdown_connections 2024-04-17T10:34:28.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.169+0000 7f1cce400640 1 -- 172.21.15.67:0/1070846366 wait complete. 2024-04-17T10:34:28.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 Processor -- start 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 -- start start 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80ce5d0 0x7f1cc813bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cc80cf8a0 0x7f1cc8154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cc815dcf0 0x7f1cc81550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80b2770 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80b25f0 con 0x7f1cc80cf8a0 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc816b740 con 0x7f1cc815dcf0 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80ce5d0 0x7f1cc813bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80ce5d0 0x7f1cc813bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35202/0 (socket says 172.21.15.67:35202) 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 -- 172.21.15.67:0/4062179717 learned_addr learned my addr 172.21.15.67:0/4062179717 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cbfe00640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cc80cf8a0 0x7f1cc8154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7e00640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cc815dcf0 0x7f1cc81550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:28.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 -- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cc815dcf0 msgr2=0x7f1cc81550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cc815dcf0 0x7f1cc81550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 -- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cc80cf8a0 msgr2=0x7f1cc8154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cc80cf8a0 0x7f1cc8154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 -- 172.21.15.67:0/4062179717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc813c450 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cc7400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80ce5d0 0x7f1cc813bf10 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800c3a0 tx=0x7f1cb800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cbf400640 1 -- 172.21.15.67:0/4062179717 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb80052d0 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cbf400640 1 -- 172.21.15.67:0/4062179717 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb8005470 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cbf400640 1 -- 172.21.15.67:0/4062179717 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8007260 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc8139450 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc814a4a0 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.173+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc804ef30 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.177+0000 7f1cbf400640 1 -- 172.21.15.67:0/4062179717 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1cb8007400 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.177+0000 7f1cbf400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ca4066f60 0x7f1ca4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:28.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.177+0000 7f1cbf400640 1 -- 172.21.15.67:0/4062179717 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f1cb8084690 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.177+0000 7f1cbfe00640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ca4066f60 0x7f1ca4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:28.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.177+0000 7f1cbfe00640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ca4066f60 0x7f1ca4069420 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00ae000 tx=0x7f1cb0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:28.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.181+0000 7f1cbf400640 1 -- 172.21.15.67:0/4062179717 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1cc813c450 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.377+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1cc8159b00 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.948 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 19 and data pool 20 2024-04-17T10:34:28.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.957+0000 7f1cbf400640 1 -- 172.21.15.67:0/4062179717 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 19 and data pool 20 v86) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1cb80554d0 con 0x7f1cc80ce5d0 2024-04-17T10:34:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ca4066f60 msgr2=0x7f1ca4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ca4066f60 0x7f1ca4069420 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00ae000 tx=0x7f1cb0008040 comp rx=0 tx=0).stop 2024-04-17T10:34:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80ce5d0 msgr2=0x7f1cc813bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:28.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80ce5d0 0x7f1cc813bf10 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800c3a0 tx=0x7f1cb800c860 comp rx=0 tx=0).stop 2024-04-17T10:34:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 shutdown_connections 2024-04-17T10:34:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ca4066f60 0x7f1ca4069420 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1cc815dcf0 0x7f1cc81550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1cc80cf8a0 0x7f1cc8154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 --2- 172.21.15.67:0/4062179717 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1cc80ce5d0 0x7f1cc813bf10 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 >> 172.21.15.67:0/4062179717 conn(0x7f1cc80b33a0 msgr2=0x7f1cc80cdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:28.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 shutdown_connections 2024-04-17T10:34:28.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:28.961+0000 7f1cce400640 1 -- 172.21.15.67:0/4062179717 wait complete. 2024-04-17T10:34:28.995 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:29.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5c000640 1 Processor -- start 2024-04-17T10:34:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5c000640 1 -- start start 2024-04-17T10:34:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc5406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5c000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcc54071a20 0x7fcc540cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5c000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc5406e2f0 con 0x7fcc5406ef10 2024-04-17T10:34:29.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5c000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc540a86c0 con 0x7fcc54071a20 2024-04-17T10:34:29.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5415bf10 con 0x7fcc54145040 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5a000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcc54071a20 0x7fcc540cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc58c00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcc5406ef10 0x7fcc54071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:44010/0 (socket says 172.21.15.67:44010) 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc58c00640 1 -- 172.21.15.67:0/188327840 learned_addr learned my addr 172.21.15.67:0/188327840 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc59600640 1 --2- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc5406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5a000640 1 -- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 msgr2=0x7fcc5406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5a000640 1 --2- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc5406e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5a000640 1 -- 172.21.15.67:0/188327840 >> v1:172.21.15.86:6789/0 conn(0x7fcc5406ef10 legacy=0x7fcc54071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc5a000640 1 -- 172.21.15.67:0/188327840 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5406d6e0 con 0x7fcc54071a20 2024-04-17T10:34:29.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.149+0000 7fcc59600640 1 --2- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc5406e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:29.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5a000640 1 --2- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540cc300 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009d90 tx=0x7fcc4c01f150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15556f61e2b1afdd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc53e00640 1 -- 172.21.15.67:0/188327840 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0b4070 con 0x7fcc54071a20 2024-04-17T10:34:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc53e00640 1 -- 172.21.15.67:0/188327840 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c0af600 con 0x7fcc54071a20 2024-04-17T10:34:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc53e00640 1 -- 172.21.15.67:0/188327840 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0af9d0 con 0x7fcc54071a20 2024-04-17T10:34:29.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 -- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 msgr2=0x7fcc540cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 --2- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540cc300 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c009d90 tx=0x7fcc4c01f150 comp rx=0 tx=0).stop 2024-04-17T10:34:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 -- 172.21.15.67:0/188327840 shutdown_connections 2024-04-17T10:34:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 --2- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540cc300 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 --2- 172.21.15.67:0/188327840 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc5406e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 -- 172.21.15.67:0/188327840 >> 172.21.15.67:0/188327840 conn(0x7fcc5406ac90 msgr2=0x7fcc5406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 -- 172.21.15.67:0/188327840 shutdown_connections 2024-04-17T10:34:29.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.153+0000 7fcc5c000640 1 -- 172.21.15.67:0/188327840 wait complete. 2024-04-17T10:34:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 Processor -- start 2024-04-17T10:34:29.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 -- start start 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc5406ef10 0x7fcc5415b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540a8f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc540a9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5406df80 con 0x7fcc5406ef10 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5406dca0 con 0x7fcc54071a20 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5406d370 con 0x7fcc54145040 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc5406ef10 0x7fcc5415b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc5406ef10 0x7fcc5415b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35216/0 (socket says 172.21.15.67:35216) 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 -- 172.21.15.67:0/3051674709 learned_addr learned my addr 172.21.15.67:0/3051674709 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc58c00640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540a8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5a000640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc540a9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 -- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 msgr2=0x7fcc540a9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc540a9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 -- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 msgr2=0x7fcc540a8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540a8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 -- 172.21.15.67:0/3051674709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc541e80e0 con 0x7fcc5406ef10 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc58c00640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc59600640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc5406ef10 0x7fcc5415b0e0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800e980 tx=0x7fcc4800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc51600640 1 -- 172.21.15.67:0/3051674709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48017070 con 0x7fcc5406ef10 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc51600640 1 -- 172.21.15.67:0/3051674709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4800ca00 con 0x7fcc5406ef10 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc51600640 1 -- 172.21.15.67:0/3051674709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4800cdd0 con 0x7fcc5406ef10 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc54142540 con 0x7fcc5406ef10 2024-04-17T10:34:29.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5415b620 con 0x7fcc5406ef10 2024-04-17T10:34:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.157+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc54072d50 con 0x7fcc5406ef10 2024-04-17T10:34:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.161+0000 7fcc51600640 1 -- 172.21.15.67:0/3051674709 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcc4801a070 con 0x7fcc5406ef10 2024-04-17T10:34:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.161+0000 7fcc51600640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc3406b370 0x7fcc3406d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.161+0000 7fcc51600640 1 -- 172.21.15.67:0/3051674709 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcc48085380 con 0x7fcc5406ef10 2024-04-17T10:34:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.161+0000 7fcc58c00640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc3406b370 0x7fcc3406d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.161+0000 7fcc58c00640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc3406b370 0x7fcc3406d830 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fcc541f4f70 tx=0x7fcc40007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.165+0000 7fcc51600640 1 -- 172.21.15.67:0/3051674709 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcc48055ba0 con 0x7fcc5406ef10 2024-04-17T10:34:29.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.329+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc54145500 con 0x7fcc5406ef10 2024-04-17T10:34:29.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.329+0000 7fcc51600640 1 -- 172.21.15.67:0/3051674709 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16865 (secure 0 0 0) 0x7fcc48059c40 con 0x7fcc5406ef10 2024-04-17T10:34:29.321 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:29.321 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":129,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:34:28.382508+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:34:26.120176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-17T10:34:27.343196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.329+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc3406b370 msgr2=0x7fcc3406d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.329+0000 7fcc5c000640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc3406b370 0x7fcc3406d830 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fcc541f4f70 tx=0x7fcc40007e90 comp rx=0 tx=0).stop 2024-04-17T10:34:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.329+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc5406ef10 msgr2=0x7fcc5415b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.329+0000 7fcc5c000640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc5406ef10 0x7fcc5415b0e0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800e980 tx=0x7fcc4800ee40 comp rx=0 tx=0).stop 2024-04-17T10:34:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 shutdown_connections 2024-04-17T10:34:29.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcc3406b370 0x7fcc3406d830 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcc54145040 0x7fcc540a9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc54071a20 0x7fcc540a8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 --2- 172.21.15.67:0/3051674709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc5406ef10 0x7fcc5415b0e0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 >> 172.21.15.67:0/3051674709 conn(0x7fcc5406ac90 msgr2=0x7fcc540b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:29.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 shutdown_connections 2024-04-17T10:34:29.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.333+0000 7fcc5c000640 1 -- 172.21.15.67:0/3051674709 wait complete. 2024-04-17T10:34:29.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:29.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42b5000640 1 Processor -- start 2024-04-17T10:34:29.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42b5000640 1 -- start start 2024-04-17T10:34:29.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42b5000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b0145040 0x7f42b006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42b5000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b006e2f0 con 0x7f42b0071a20 2024-04-17T10:34:29.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42b5000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b006d6e0 con 0x7f42b006ef10 2024-04-17T10:34:29.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42b5000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b00a86c0 con 0x7f42b0145040 2024-04-17T10:34:29.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ae800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f42b0071a20 0x7f42b00cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:42776/0 (socket says 172.21.15.67:42776) 2024-04-17T10:34:29.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ae800640 1 -- 172.21.15.67:0/3161933873 learned_addr learned my addr 172.21.15.67:0/3161933873 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ade00640 1 --2- 172.21.15.67:0/3161933873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b0145040 0x7f42b006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ad400640 1 -- 172.21.15.67:0/3161933873 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743659195 0 0) 0x7f42b006e2f0 con 0x7f42b0071a20 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ad400640 1 -- 172.21.15.67:0/3161933873 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4294003530 con 0x7f42b0071a20 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ad400640 1 -- 172.21.15.67:0/3161933873 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995186778 0 0) 0x7f42b006d6e0 con 0x7f42b006ef10 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ad400640 1 -- 172.21.15.67:0/3161933873 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b006e2f0 con 0x7f42b006ef10 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ade00640 1 -- 172.21.15.67:0/3161933873 >> v1:172.21.15.86:6789/0 conn(0x7f42b006ef10 legacy=0x7f42b0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ade00640 1 -- 172.21.15.67:0/3161933873 >> v1:172.21.15.67:6789/0 conn(0x7f42b0071a20 legacy=0x7f42b00cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.493+0000 7f42ade00640 1 -- 172.21.15.67:0/3161933873 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b006d9c0 con 0x7f42b0145040 2024-04-17T10:34:29.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.497+0000 7f42ade00640 1 --2- 172.21.15.67:0/3161933873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b0145040 0x7f42b006e9d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f42a00097f0 tx=0x7f42a001ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f71633f7c9186f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.497+0000 7f42ad400640 1 -- 172.21.15.67:0/3161933873 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00b5070 con 0x7f42b0145040 2024-04-17T10:34:29.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.497+0000 7f42ad400640 1 -- 172.21.15.67:0/3161933873 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a00a89d0 con 0x7f42b0145040 2024-04-17T10:34:29.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.497+0000 7f42ad400640 1 -- 172.21.15.67:0/3161933873 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a00a8da0 con 0x7f42b0145040 2024-04-17T10:34:29.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- 172.21.15.67:0/3161933873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b0145040 msgr2=0x7f42b006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 --2- 172.21.15.67:0/3161933873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b0145040 0x7f42b006e9d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f42a00097f0 tx=0x7f42a001ecd0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- 172.21.15.67:0/3161933873 shutdown_connections 2024-04-17T10:34:29.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 --2- 172.21.15.67:0/3161933873 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b0145040 0x7f42b006e9d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- 172.21.15.67:0/3161933873 >> 172.21.15.67:0/3161933873 conn(0x7f42b006ac90 msgr2=0x7f42b006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:29.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- 172.21.15.67:0/3161933873 shutdown_connections 2024-04-17T10:34:29.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- 172.21.15.67:0/3161933873 wait complete. 2024-04-17T10:34:29.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 Processor -- start 2024-04-17T10:34:29.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- start start 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b006ef10 0x7f42b013ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b0071a20 0x7f42b015e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42b0145040 0x7f42b013fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b006df80 con 0x7f42b0071a20 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b015bf10 con 0x7f42b0145040 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42b5000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b006d370 con 0x7f42b006ef10 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b0071a20 0x7f42b015e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b0071a20 0x7f42b015e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35228/0 (socket says 172.21.15.67:35228) 2024-04-17T10:34:29.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 -- 172.21.15.67:0/2864576424 learned_addr learned my addr 172.21.15.67:0/2864576424 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 -- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b006ef10 msgr2=0x7f42b013ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42ade00640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b006ef10 0x7f42b013ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42ae800640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42b0145040 0x7f42b013fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b006ef10 0x7f42b013ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 -- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42b0145040 msgr2=0x7f42b013fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42b0145040 0x7f42b013fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 -- 172.21.15.67:0/2864576424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b0143b70 con 0x7f42b0071a20 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42ade00640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b006ef10 0x7f42b013ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42ae800640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42b0145040 0x7f42b013fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a5400640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b0071a20 0x7f42b015e670 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f429800c500 tx=0x7f429800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.501+0000 7f42a6a00640 1 -- 172.21.15.67:0/2864576424 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42980052d0 con 0x7f42b0071a20 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42a6a00640 1 -- 172.21.15.67:0/2864576424 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4298005490 con 0x7f42b0071a20 2024-04-17T10:34:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42a6a00640 1 -- 172.21.15.67:0/2864576424 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4298012850 con 0x7f42b0071a20 2024-04-17T10:34:29.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b0143de0 con 0x7f42b0071a20 2024-04-17T10:34:29.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b015db90 con 0x7f42b0071a20 2024-04-17T10:34:29.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b0072d50 con 0x7f42b0071a20 2024-04-17T10:34:29.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42a6a00640 1 -- 172.21.15.67:0/2864576424 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f42980129f0 con 0x7f42b0071a20 2024-04-17T10:34:29.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42a6a00640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f428c066f60 0x7f428c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42a6a00640 1 -- 172.21.15.67:0/2864576424 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f4298084ee0 con 0x7f42b0071a20 2024-04-17T10:34:29.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42ade00640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f428c066f60 0x7f428c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.505+0000 7f42ade00640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f428c066f60 0x7f428c069420 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f42a00b2000 tx=0x7f42a00aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.509+0000 7f42a6a00640 1 -- 172.21.15.67:0/2864576424 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4298055cd0 con 0x7f42b0071a20 2024-04-17T10:34:29.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42b00ce140 con 0x7f42b0071a20 2024-04-17T10:34:29.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42a6a00640 1 -- 172.21.15.67:0/2864576424 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16865 (secure 0 0 0) 0x7f42980555d0 con 0x7f42b0071a20 2024-04-17T10:34:29.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":129,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:34:28.382508+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:34:26.120176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-17T10:34:27.343196+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:29.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f428c066f60 msgr2=0x7f428c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f428c066f60 0x7f428c069420 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f42a00b2000 tx=0x7f42a00aa070 comp rx=0 tx=0).stop 2024-04-17T10:34:29.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b0071a20 msgr2=0x7f42b015e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b0071a20 0x7f42b015e670 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f429800c500 tx=0x7f429800c9c0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 shutdown_connections 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f428c066f60 0x7f428c069420 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f42b0145040 0x7f42b013fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b0071a20 0x7f42b015e670 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 --2- 172.21.15.67:0/2864576424 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f42b006ef10 0x7f42b013ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 >> 172.21.15.67:0/2864576424 conn(0x7f42b006ac90 msgr2=0x7f42b0071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 shutdown_connections 2024-04-17T10:34:29.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.677+0000 7f42b5000640 1 -- 172.21.15.67:0/2864576424 wait complete. 2024-04-17T10:34:29.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:34:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.841+0000 7f71a6400640 1 Processor -- start 2024-04-17T10:34:29.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.841+0000 7f71a6400640 1 -- start start 2024-04-17T10:34:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f71a6400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71a0159440 0x7f71a00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f71a6400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f71a015c580 0x7f71a0071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f71a6400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71a015cb30 con 0x7f71a00713e0 2024-04-17T10:34:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f71a6400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a00a8650 con 0x7f71a0159440 2024-04-17T10:34:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f71a6400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71a015ce10 con 0x7f71a015c580 2024-04-17T10:34:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71a0159440 0x7f71a00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f71a0159440 0x7f71a00ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35232/0 (socket says 172.21.15.67:35232) 2024-04-17T10:34:29.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719f400640 1 -- 172.21.15.67:0/195302277 learned_addr learned my addr 172.21.15.67:0/195302277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719fe00640 1 --2- 172.21.15.67:0/195302277 >> v2:172.21.15.86:3300/0 conn(0x7f71a015c580 0x7f71a0071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719f400640 1 -- 172.21.15.67:0/195302277 >> v2:172.21.15.86:3300/0 conn(0x7f71a015c580 msgr2=0x7f71a0071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719f400640 1 --2- 172.21.15.67:0/195302277 >> v2:172.21.15.86:3300/0 conn(0x7f71a015c580 0x7f71a0071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719f400640 1 -- 172.21.15.67:0/195302277 >> v1:172.21.15.86:6789/0 conn(0x7f71a00713e0 legacy=0x7f71a015be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719f400640 1 -- 172.21.15.67:0/195302277 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a00cd610 con 0x7f71a0159440 2024-04-17T10:34:29.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.845+0000 7f719fe00640 1 --2- 172.21.15.67:0/195302277 >> v2:172.21.15.86:3300/0 conn(0x7f71a015c580 0x7f71a0071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:29.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f719f400640 1 --2- 172.21.15.67:0/195302277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a0159440 0x7f71a00ce0f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f71a00ce660 tx=0x7f719001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6555c90f30b07b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f719ea00640 1 -- 172.21.15.67:0/195302277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71900ab430 con 0x7f71a0159440 2024-04-17T10:34:29.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f719ea00640 1 -- 172.21.15.67:0/195302277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71900b0070 con 0x7f71a0159440 2024-04-17T10:34:29.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f719ea00640 1 -- 172.21.15.67:0/195302277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71900ab7f0 con 0x7f71a0159440 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- 172.21.15.67:0/195302277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a0159440 msgr2=0x7f71a00ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 --2- 172.21.15.67:0/195302277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a0159440 0x7f71a00ce0f0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f71a00ce660 tx=0x7f719001c520 comp rx=0 tx=0).stop 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- 172.21.15.67:0/195302277 shutdown_connections 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 --2- 172.21.15.67:0/195302277 >> v2:172.21.15.86:3300/0 conn(0x7f71a015c580 0x7f71a0071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 --2- 172.21.15.67:0/195302277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a0159440 0x7f71a00ce0f0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- 172.21.15.67:0/195302277 >> 172.21.15.67:0/195302277 conn(0x7f71a00b1740 msgr2=0x7f71a00b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- 172.21.15.67:0/195302277 shutdown_connections 2024-04-17T10:34:29.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- 172.21.15.67:0/195302277 wait complete. 2024-04-17T10:34:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 Processor -- start 2024-04-17T10:34:29.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- start start 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00713e0 0x7f71a014dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a0159440 0x7f71a014d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a015c580 0x7f71a01513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a015d9c0 con 0x7f71a015c580 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71a015d840 con 0x7f71a00713e0 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f71a6400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71a015db40 con 0x7f71a0159440 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.849+0000 7f719fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a015c580 0x7f71a01513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a015c580 0x7f71a01513a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35248/0 (socket says 172.21.15.67:35248) 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 -- 172.21.15.67:0/868392344 learned_addr learned my addr 172.21.15.67:0/868392344 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:29.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00713e0 0x7f71a014dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f7197e00640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a0159440 0x7f71a014d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 -- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a0159440 msgr2=0x7f71a014d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a0159440 0x7f71a014d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 -- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00713e0 msgr2=0x7f71a014dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00713e0 0x7f71a014dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 -- 172.21.15.67:0/868392344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a014e4a0 con 0x7f71a015c580 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f719fe00640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a015c580 0x7f71a01513a0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f719800cd80 tx=0x7f719800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f7197400640 1 -- 172.21.15.67:0/868392344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719800f680 con 0x7f71a015c580 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f7197400640 1 -- 172.21.15.67:0/868392344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f719800f840 con 0x7f71a015c580 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f7197400640 1 -- 172.21.15.67:0/868392344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f719800fc30 con 0x7f71a015c580 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71a014e780 con 0x7f71a015c580 2024-04-17T10:34:29.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71a014ed40 con 0x7f71a015c580 2024-04-17T10:34:29.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71a004ef30 con 0x7f71a015c580 2024-04-17T10:34:29.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.853+0000 7f7197400640 1 -- 172.21.15.67:0/868392344 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f719800e3e0 con 0x7f71a015c580 2024-04-17T10:34:29.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.857+0000 7f7197400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f717c066f60 0x7f717c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:29.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.857+0000 7f7197400640 1 -- 172.21.15.67:0/868392344 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7198085df0 con 0x7f71a015c580 2024-04-17T10:34:29.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.857+0000 7f719f400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f717c066f60 0x7f717c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:29.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.857+0000 7f719f400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f717c066f60 0x7f717c069420 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f71900af000 tx=0x7f71900a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:29.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:29.861+0000 7f7197400640 1 -- 172.21.15.67:0/868392344 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7198014030 con 0x7f71a015c580 2024-04-17T10:34:30.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.057+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f71a00afe70 con 0x7f71a015c580 2024-04-17T10:34:30.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f7197400640 1 -- 172.21.15.67:0/868392344 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v89) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7198056610 con 0x7f71a015c580 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f717c066f60 msgr2=0x7f717c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f717c066f60 0x7f717c069420 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f71900af000 tx=0x7f71900a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a015c580 msgr2=0x7f71a01513a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a015c580 0x7f71a01513a0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f719800cd80 tx=0x7f719800a650 comp rx=0 tx=0).stop 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 shutdown_connections 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f717c066f60 0x7f717c069420 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f71a015c580 0x7f71a01513a0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f71a0159440 0x7f71a014d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:30.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 --2- 172.21.15.67:0/868392344 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f71a00713e0 0x7f71a014dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:30.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 >> 172.21.15.67:0/868392344 conn(0x7f71a00b1740 msgr2=0x7f71a01586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:30.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 shutdown_connections 2024-04-17T10:34:30.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:30.969+0000 7f71a6400640 1 -- 172.21.15.67:0/868392344 wait complete. 2024-04-17T10:34:30.986 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:34:30.986 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:34:31.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.141+0000 7fd6ee600640 1 Processor -- start 2024-04-17T10:34:31.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6ee600640 1 -- start start 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6ee600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e80cd030 0x7fd6e806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6ee600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6e806e7b0 0x7fd6e80b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6ee600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e80b4990 con 0x7fd6e80aef60 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6ee600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e80af340 con 0x7fd6e806e7b0 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6ee600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e815ae90 con 0x7fd6e80cd030 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6e806e7b0 0x7fd6e80b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6e806e7b0 0x7fd6e80b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48002/0 (socket says 172.21.15.67:48002) 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e7e00640 1 -- 172.21.15.67:0/38641803 learned_addr learned my addr 172.21.15.67:0/38641803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e6a00640 1 --2- 172.21.15.67:0/38641803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e80cd030 0x7fd6e806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e7e00640 1 -- 172.21.15.67:0/38641803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e80cd030 msgr2=0x7fd6e806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e7e00640 1 --2- 172.21.15.67:0/38641803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e80cd030 0x7fd6e806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e7e00640 1 -- 172.21.15.67:0/38641803 >> v1:172.21.15.86:6789/0 conn(0x7fd6e80aef60 legacy=0x7fd6e80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:31.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.145+0000 7fd6e7e00640 1 -- 172.21.15.67:0/38641803 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e806ee80 con 0x7fd6e806e7b0 2024-04-17T10:34:31.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6e7e00640 1 --2- 172.21.15.67:0/38641803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e806e7b0 0x7fd6e80b3120 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009e50 tx=0x7fd6e001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed0cd6781781a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:31.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6e6000640 1 -- 172.21.15.67:0/38641803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00a5750 con 0x7fd6e806e7b0 2024-04-17T10:34:31.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6e6000640 1 -- 172.21.15.67:0/38641803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e00a58f0 con 0x7fd6e806e7b0 2024-04-17T10:34:31.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6e6000640 1 -- 172.21.15.67:0/38641803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e00a5cc0 con 0x7fd6e806e7b0 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 -- 172.21.15.67:0/38641803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e806e7b0 msgr2=0x7fd6e80b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 --2- 172.21.15.67:0/38641803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e806e7b0 0x7fd6e80b3120 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009e50 tx=0x7fd6e001c9a0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 -- 172.21.15.67:0/38641803 shutdown_connections 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 --2- 172.21.15.67:0/38641803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e806e7b0 0x7fd6e80b3120 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 --2- 172.21.15.67:0/38641803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e80cd030 0x7fd6e806e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 -- 172.21.15.67:0/38641803 >> 172.21.15.67:0/38641803 conn(0x7fd6e80b1740 msgr2=0x7fd6e80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 -- 172.21.15.67:0/38641803 shutdown_connections 2024-04-17T10:34:31.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 -- 172.21.15.67:0/38641803 wait complete. 2024-04-17T10:34:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 Processor -- start 2024-04-17T10:34:31.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.149+0000 7fd6ee600640 1 -- start start 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e806e7b0 0x7fd6e8154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e80aef60 0x7fd6e8138e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e80cd030 0x7fd6e80b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e815bc10 con 0x7fd6e80aef60 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80b3e70 con 0x7fd6e80cd030 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e814e8c0 con 0x7fd6e806e7b0 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e80aef60 0x7fd6e8138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e806e7b0 0x7fd6e8154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e80aef60 0x7fd6e8138e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48012/0 (socket says 172.21.15.67:48012) 2024-04-17T10:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 -- 172.21.15.67:0/393296210 learned_addr learned my addr 172.21.15.67:0/393296210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e7e00640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e80cd030 0x7fd6e80b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 -- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e806e7b0 msgr2=0x7fd6e8154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e806e7b0 0x7fd6e8154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 -- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e80cd030 msgr2=0x7fd6e80b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e80cd030 0x7fd6e80b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 -- 172.21.15.67:0/393296210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e814cfb0 con 0x7fd6e80aef60 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6e6a00640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e80aef60 0x7fd6e8138e70 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400d620 tx=0x7fd6d400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6df400640 1 -- 172.21.15.67:0/393296210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400b620 con 0x7fd6e80aef60 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6df400640 1 -- 172.21.15.67:0/393296210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4019070 con 0x7fd6e80aef60 2024-04-17T10:34:31.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6df400640 1 -- 172.21.15.67:0/393296210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d400b9e0 con 0x7fd6e80aef60 2024-04-17T10:34:31.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e814d290 con 0x7fd6e80aef60 2024-04-17T10:34:31.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e814d850 con 0x7fd6e80aef60 2024-04-17T10:34:31.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e804ef30 con 0x7fd6e80aef60 2024-04-17T10:34:31.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.153+0000 7fd6df400640 1 -- 172.21.15.67:0/393296210 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd6d400bb80 con 0x7fd6e80aef60 2024-04-17T10:34:31.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.157+0000 7fd6df400640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6c8066f60 0x7fd6c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:31.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.157+0000 7fd6df400640 1 -- 172.21.15.67:0/393296210 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd6d4085460 con 0x7fd6e80aef60 2024-04-17T10:34:31.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.157+0000 7fd6e7400640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6c8066f60 0x7fd6c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:31.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.157+0000 7fd6e7400640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6c8066f60 0x7fd6c8069420 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8001810 tx=0x7fd6d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:31.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.161+0000 7fd6df400640 1 -- 172.21.15.67:0/393296210 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd6d4055c90 con 0x7fd6e80aef60 2024-04-17T10:34:31.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.357+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd6e814eac0 con 0x7fd6e80aef60 2024-04-17T10:34:31.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.973+0000 7fd6df400640 1 -- 172.21.15.67:0/393296210 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v90) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd6d4059d30 con 0x7fd6e80aef60 2024-04-17T10:34:31.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6c8066f60 msgr2=0x7fd6c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:31.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6c8066f60 0x7fd6c8069420 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8001810 tx=0x7fd6d8009040 comp rx=0 tx=0).stop 2024-04-17T10:34:31.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e80aef60 msgr2=0x7fd6e8138e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:31.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e80aef60 0x7fd6e8138e70 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400d620 tx=0x7fd6d400dae0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 shutdown_connections 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd6c8066f60 0x7fd6c8069420 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd6e80cd030 0x7fd6e80b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6e80aef60 0x7fd6e8138e70 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 --2- 172.21.15.67:0/393296210 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd6e806e7b0 0x7fd6e8154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 >> 172.21.15.67:0/393296210 conn(0x7fd6e80b1740 msgr2=0x7fd6e80cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 shutdown_connections 2024-04-17T10:34:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:31.977+0000 7fd6ee600640 1 -- 172.21.15.67:0/393296210 wait complete. 2024-04-17T10:34:31.993 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:34:31.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:32.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.145+0000 7f8bad200640 1 Processor -- start 2024-04-17T10:34:32.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.145+0000 7f8bad200640 1 -- start start 2024-04-17T10:34:32.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8bad200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8ba815b870 0x7f8ba815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8bad200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba80720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8bad200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba806aea0 con 0x7f8ba806f560 2024-04-17T10:34:32.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8bad200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba806b7d0 con 0x7f8ba815b870 2024-04-17T10:34:32.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8bad200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba806b180 con 0x7f8ba8071c60 2024-04-17T10:34:32.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba80720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba5e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8ba815b870 0x7f8ba815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba80720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38398/0 (socket says 172.21.15.67:38398) 2024-04-17T10:34:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba6800640 1 -- 172.21.15.67:0/2068288971 learned_addr learned my addr 172.21.15.67:0/2068288971 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8b9fe00640 1 --1- 172.21.15.67:0/2068288971 >> v1:172.21.15.86:6789/0 conn(0x7f8ba806f560 0x7f8ba815bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:44096/0 (socket says 172.21.15.67:44096) 2024-04-17T10:34:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba5e00640 1 -- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 msgr2=0x7f8ba80720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:32.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba5e00640 1 --2- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba80720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba5e00640 1 -- 172.21.15.67:0/2068288971 >> v1:172.21.15.86:6789/0 conn(0x7f8ba806f560 legacy=0x7f8ba815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba5e00640 1 -- 172.21.15.67:0/2068288971 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba806b460 con 0x7f8ba815b870 2024-04-17T10:34:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba6800640 1 --2- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba80720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:32.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.149+0000 7f8ba5400640 1 -- 172.21.15.67:0/2068288971 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944705480 0 0) 0x7f8ba806aea0 con 0x7f8ba806f560 2024-04-17T10:34:32.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8ba5e00640 1 --2- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 0x7f8ba815fe10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8b980097c0 tx=0x7f8b9801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8da8e8f3e20a8f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8ba5400640 1 -- 172.21.15.67:0/2068288971 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980b0070 con 0x7f8ba815b870 2024-04-17T10:34:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8ba5400640 1 -- 172.21.15.67:0/2068288971 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b980ab600 con 0x7f8ba815b870 2024-04-17T10:34:32.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8ba5400640 1 -- 172.21.15.67:0/2068288971 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980ab9d0 con 0x7f8ba815b870 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 -- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 msgr2=0x7f8ba815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 --2- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 0x7f8ba815fe10 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f8b980097c0 tx=0x7f8b9801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 -- 172.21.15.67:0/2068288971 shutdown_connections 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 --2- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba80720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 --2- 172.21.15.67:0/2068288971 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 0x7f8ba815fe10 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 -- 172.21.15.67:0/2068288971 >> 172.21.15.67:0/2068288971 conn(0x7f8ba80af670 msgr2=0x7f8ba80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 -- 172.21.15.67:0/2068288971 shutdown_connections 2024-04-17T10:34:32.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 -- 172.21.15.67:0/2068288971 wait complete. 2024-04-17T10:34:32.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 Processor -- start 2024-04-17T10:34:32.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.153+0000 7f8bad200640 1 -- start start 2024-04-17T10:34:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba806f560 0x7f8ba8139190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba814d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 0x7f8ba81517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba806c6c0 con 0x7f8ba806f560 2024-04-17T10:34:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba81687a0 con 0x7f8ba815b870 2024-04-17T10:34:32.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba813bcb0 con 0x7f8ba8071c60 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba806f560 0x7f8ba8139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba806f560 0x7f8ba8139190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48018/0 (socket says 172.21.15.67:48018) 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 -- 172.21.15.67:0/1651061693 learned_addr learned my addr 172.21.15.67:0/1651061693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba6800640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 0x7f8ba81517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba814d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 -- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 msgr2=0x7f8ba814d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba814d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 -- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 msgr2=0x7f8ba81517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 0x7f8ba81517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 -- 172.21.15.67:0/1651061693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba813bf90 con 0x7f8ba806f560 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba814d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8ba5e00640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba806f560 0x7f8ba8139190 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f8b980190b0 tx=0x7f8b98019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8b9e000640 1 -- 172.21.15.67:0/1651061693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980b9070 con 0x7f8ba806f560 2024-04-17T10:34:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8b9e000640 1 -- 172.21.15.67:0/1651061693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b980b5480 con 0x7f8ba806f560 2024-04-17T10:34:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba813c270 con 0x7f8ba806f560 2024-04-17T10:34:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8b9e000640 1 -- 172.21.15.67:0/1651061693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980b0040 con 0x7f8ba806f560 2024-04-17T10:34:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba813c830 con 0x7f8ba806f560 2024-04-17T10:34:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.157+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba809e260 con 0x7f8ba806f560 2024-04-17T10:34:32.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.161+0000 7f8b9e000640 1 -- 172.21.15.67:0/1651061693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8b980bc070 con 0x7f8ba806f560 2024-04-17T10:34:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.161+0000 7f8b9e000640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b84066f60 0x7f8b84069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.161+0000 7f8b9e000640 1 -- 172.21.15.67:0/1651061693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8b9811f310 con 0x7f8ba806f560 2024-04-17T10:34:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.161+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b84066f60 0x7f8b84069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.161+0000 7f8b9fe00640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b84066f60 0x7f8b84069420 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8ba81fa160 tx=0x7f8b90007c60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.165+0000 7f8b9e000640 1 -- 172.21.15.67:0/1651061693 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8b980c3030 con 0x7f8ba806f560 2024-04-17T10:34:32.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.361+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ba81398d0 con 0x7f8ba806f560 2024-04-17T10:34:32.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.361+0000 7f8b9e000640 1 -- 172.21.15.67:0/1651061693 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f8b98016090 con 0x7f8ba806f560 2024-04-17T10:34:32.355 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:32.355 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:34:32.355 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b84066f60 msgr2=0x7f8b84069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b84066f60 0x7f8b84069420 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f8ba81fa160 tx=0x7f8b90007c60 comp rx=0 tx=0).stop 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba806f560 msgr2=0x7f8ba8139190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba806f560 0x7f8ba8139190 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f8b980190b0 tx=0x7f8b98019240 comp rx=0 tx=0).stop 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 shutdown_connections 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b84066f60 0x7f8b84069420 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ba815b870 0x7f8ba81517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ba8071c60 0x7f8ba814d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 --2- 172.21.15.67:0/1651061693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba806f560 0x7f8ba8139190 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 >> 172.21.15.67:0/1651061693 conn(0x7f8ba80af670 msgr2=0x7f8ba80ad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 shutdown_connections 2024-04-17T10:34:32.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.365+0000 7f8bad200640 1 -- 172.21.15.67:0/1651061693 wait complete. 2024-04-17T10:34:32.376 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:34:32.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.525+0000 7f08d0a00640 1 Processor -- start 2024-04-17T10:34:32.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.525+0000 7f08d0a00640 1 -- start start 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.525+0000 7f08d0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08cc06eac0 0x7f08cc143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.525+0000 7f08d0a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f08cc0cf0c0 0x7f08cc144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.525+0000 7f08d0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc144560 con 0x7f08cc06e6e0 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.525+0000 7f08d0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08cc15ae40 con 0x7f08cc06eac0 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.525+0000 7f08d0a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08cc071440 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08cc06eac0 0x7f08cc143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f08cc06eac0 0x7f08cc143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48026/0 (socket says 172.21.15.67:48026) 2024-04-17T10:34:32.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 -- 172.21.15.67:0/236867176 learned_addr learned my addr 172.21.15.67:0/236867176 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c9e00640 1 --1- 172.21.15.67:0/236867176 >> v1:172.21.15.67:6789/0 conn(0x7f08cc06e6e0 0x7f08cc0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37760/0 (socket says 172.21.15.67:37760) 2024-04-17T10:34:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08ca800640 1 --2- 172.21.15.67:0/236867176 >> v2:172.21.15.86:3300/0 conn(0x7f08cc0cf0c0 0x7f08cc144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 -- 172.21.15.67:0/236867176 >> v2:172.21.15.86:3300/0 conn(0x7f08cc0cf0c0 msgr2=0x7f08cc144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 --2- 172.21.15.67:0/236867176 >> v2:172.21.15.86:3300/0 conn(0x7f08cc0cf0c0 0x7f08cc144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 -- 172.21.15.67:0/236867176 >> v1:172.21.15.67:6789/0 conn(0x7f08cc06e6e0 legacy=0x7f08cc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 -- 172.21.15.67:0/236867176 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc071720 con 0x7f08cc06eac0 2024-04-17T10:34:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c9400640 1 -- 172.21.15.67:0/236867176 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873968159 0 0) 0x7f08cc144560 con 0x7f08cc06e6e0 2024-04-17T10:34:32.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c1400640 1 --2- 172.21.15.67:0/236867176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc06eac0 0x7f08cc143ae0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0ce3b0 tx=0x7f08b401eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce918718ae35faf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c9400640 1 -- 172.21.15.67:0/236867176 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b40b4070 con 0x7f08cc06eac0 2024-04-17T10:34:32.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c9400640 1 -- 172.21.15.67:0/236867176 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08b40a7b20 con 0x7f08cc06eac0 2024-04-17T10:34:32.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.529+0000 7f08c9400640 1 -- 172.21.15.67:0/236867176 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b40af440 con 0x7f08cc06eac0 2024-04-17T10:34:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- 172.21.15.67:0/236867176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc06eac0 msgr2=0x7f08cc143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 --2- 172.21.15.67:0/236867176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc06eac0 0x7f08cc143ae0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0ce3b0 tx=0x7f08b401eb30 comp rx=0 tx=0).stop 2024-04-17T10:34:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- 172.21.15.67:0/236867176 shutdown_connections 2024-04-17T10:34:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 --2- 172.21.15.67:0/236867176 >> v2:172.21.15.86:3300/0 conn(0x7f08cc0cf0c0 0x7f08cc144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 --2- 172.21.15.67:0/236867176 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc06eac0 0x7f08cc143ae0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- 172.21.15.67:0/236867176 >> 172.21.15.67:0/236867176 conn(0x7f08cc06ac90 msgr2=0x7f08cc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:32.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- 172.21.15.67:0/236867176 shutdown_connections 2024-04-17T10:34:32.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- 172.21.15.67:0/236867176 wait complete. 2024-04-17T10:34:32.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 Processor -- start 2024-04-17T10:34:32.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- start start 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08cc06e6e0 0x7f08cc13dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08cc06eac0 0x7f08cc0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc0cf0c0 0x7f08cc0cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc06dd00 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc06db80 con 0x7f08cc06e6e0 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08d0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc06de80 con 0x7f08cc06eac0 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc0cf0c0 0x7f08cc0cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc0cf0c0 0x7f08cc0cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48032/0 (socket says 172.21.15.67:48032) 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 -- 172.21.15.67:0/2230075329 learned_addr learned my addr 172.21.15.67:0/2230075329 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08c9e00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08cc06e6e0 0x7f08cc13dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08c1400640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08cc06eac0 0x7f08cc0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 -- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08cc06eac0 msgr2=0x7f08cc0cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08cc06eac0 0x7f08cc0cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 -- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08cc06e6e0 msgr2=0x7f08cc13dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08cc06e6e0 0x7f08cc13dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 -- 172.21.15.67:0/2230075329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc0c5410 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08c1400640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08cc06eac0 0x7f08cc0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.533+0000 7f08ca800640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc0cf0c0 0x7f08cc0cb7b0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f08c4007e90 tx=0x7f08c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08c2a00640 1 -- 172.21.15.67:0/2230075329 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4014420 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08c2a00640 1 -- 172.21.15.67:0/2230075329 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c4019070 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08c2a00640 1 -- 172.21.15.67:0/2230075329 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4014800 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc0c56f0 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc0c5cb0 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc072d50 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08c2a00640 1 -- 172.21.15.67:0/2230075329 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f08c40149a0 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08c2a00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08a8066f60 0x7f08a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08c2a00640 1 -- 172.21.15.67:0/2230075329 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f08c40859e0 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.537+0000 7f08c9e00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08a8066f60 0x7f08a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.541+0000 7f08c9e00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08a8066f60 0x7f08a8069420 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f08bc007fd0 tx=0x7f08bc007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.541+0000 7f08c2a00640 1 -- 172.21.15.67:0/2230075329 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f08c40567c0 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.705+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08cc13e7a0 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.705+0000 7f08c2a00640 1 -- 172.21.15.67:0/2230075329 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16852 (secure 0 0 0) 0x7f08c40560c0 con 0x7f08cc0cf0c0 2024-04-17T10:34:32.698 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:32.698 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":130,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:34:29.959972+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:34:26.120176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-17T10:34:27.343196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:34:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08a8066f60 msgr2=0x7f08a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08a8066f60 0x7f08a8069420 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f08bc007fd0 tx=0x7f08bc007d60 comp rx=0 tx=0).stop 2024-04-17T10:34:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc0cf0c0 msgr2=0x7f08cc0cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc0cf0c0 0x7f08cc0cb7b0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f08c4007e90 tx=0x7f08c400e4b0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 shutdown_connections 2024-04-17T10:34:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f08a8066f60 0x7f08a8069420 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08cc0cf0c0 0x7f08cc0cb7b0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f08cc06eac0 0x7f08cc0cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 --2- 172.21.15.67:0/2230075329 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f08cc06e6e0 0x7f08cc13dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 >> 172.21.15.67:0/2230075329 conn(0x7f08cc06ac90 msgr2=0x7f08cc13f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 shutdown_connections 2024-04-17T10:34:32.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.709+0000 7f08d0a00640 1 -- 172.21.15.67:0/2230075329 wait complete. 2024-04-17T10:34:32.719 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:34:32.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:34:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 Processor -- start 2024-04-17T10:34:32.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 -- start start 2024-04-17T10:34:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62780ce1b0 0x7f62780ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f627815cc00 0x7f627815f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f627815fa00 0x7f6278070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62780cead0 con 0x7f627815fa00 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f627815be60 con 0x7f627815cc00 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f627e600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6278160230 con 0x7f62780ce1b0 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f627815fa00 0x7f6278070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f627815fa00 0x7f6278070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48036/0 (socket says 172.21.15.67:48036) 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 -- 172.21.15.67:0/890876229 learned_addr learned my addr 172.21.15.67:0/890876229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277400640 1 --2- 172.21.15.67:0/890876229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62780ce1b0 0x7f62780ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f626ea00640 1 --2- 172.21.15.67:0/890876229 >> v2:172.21.15.86:3300/0 conn(0x7f627815cc00 0x7f627815f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 -- 172.21.15.67:0/890876229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62780ce1b0 msgr2=0x7f62780ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 --2- 172.21.15.67:0/890876229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62780ce1b0 0x7f62780ce590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 -- 172.21.15.67:0/890876229 >> v2:172.21.15.86:3300/0 conn(0x7f627815cc00 msgr2=0x7f627815f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 --2- 172.21.15.67:0/890876229 >> v2:172.21.15.86:3300/0 conn(0x7f627815cc00 0x7f627815f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.869+0000 7f6277e00640 1 -- 172.21.15.67:0/890876229 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f627815c120 con 0x7f627815fa00 2024-04-17T10:34:32.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f6277e00640 1 --2- 172.21.15.67:0/890876229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815fa00 0x7f6278070cc0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f627806e1b0 tx=0x7f627001f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c844cf7b1a1e910 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f6276a00640 1 -- 172.21.15.67:0/890876229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62700b4070 con 0x7f627815fa00 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f6276a00640 1 -- 172.21.15.67:0/890876229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62700af400 con 0x7f627815fa00 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f6276a00640 1 -- 172.21.15.67:0/890876229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62700af7d0 con 0x7f627815fa00 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 -- 172.21.15.67:0/890876229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815fa00 msgr2=0x7f6278070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 --2- 172.21.15.67:0/890876229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815fa00 0x7f6278070cc0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f627806e1b0 tx=0x7f627001f1e0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 -- 172.21.15.67:0/890876229 shutdown_connections 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 --2- 172.21.15.67:0/890876229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815fa00 0x7f6278070cc0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 --2- 172.21.15.67:0/890876229 >> v2:172.21.15.86:3300/0 conn(0x7f627815cc00 0x7f627815f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 --2- 172.21.15.67:0/890876229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f62780ce1b0 0x7f62780ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 -- 172.21.15.67:0/890876229 >> 172.21.15.67:0/890876229 conn(0x7f62780b33f0 msgr2=0x7f62780b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 -- 172.21.15.67:0/890876229 shutdown_connections 2024-04-17T10:34:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.873+0000 7f627e600640 1 -- 172.21.15.67:0/890876229 wait complete. 2024-04-17T10:34:32.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 Processor -- start 2024-04-17T10:34:32.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 -- start start 2024-04-17T10:34:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62780ce1b0 0x7f6278158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815cc00 0x7f62780b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f627815fa00 0x7f6278157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627815fe70 con 0x7f627815cc00 2024-04-17T10:34:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627815c6e0 con 0x7f62780ce1b0 2024-04-17T10:34:32.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627815fff0 con 0x7f627815fa00 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815cc00 0x7f62780b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815cc00 0x7f62780b7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48038/0 (socket says 172.21.15.67:48038) 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 -- 172.21.15.67:0/3303324975 learned_addr learned my addr 172.21.15.67:0/3303324975 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f6277e00640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f627815fa00 0x7f6278157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f6277400640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62780ce1b0 0x7f6278158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 -- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f627815fa00 msgr2=0x7f6278157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f627815fa00 0x7f6278157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 -- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62780ce1b0 msgr2=0x7f6278158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62780ce1b0 0x7f6278158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 -- 172.21.15.67:0/3303324975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278158e50 con 0x7f627815cc00 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626ea00640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815cc00 0x7f62780b7220 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f626000d800 tx=0x7f626000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626fe00640 1 -- 172.21.15.67:0/3303324975 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626000b6c0 con 0x7f627815cc00 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626fe00640 1 -- 172.21.15.67:0/3303324975 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6260019070 con 0x7f627815cc00 2024-04-17T10:34:32.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f626fe00640 1 -- 172.21.15.67:0/3303324975 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f626000ba60 con 0x7f627815cc00 2024-04-17T10:34:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6278156dd0 con 0x7f627815cc00 2024-04-17T10:34:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.877+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6278157390 con 0x7f627815cc00 2024-04-17T10:34:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.881+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6278070ab0 con 0x7f627815cc00 2024-04-17T10:34:32.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.881+0000 7f626fe00640 1 -- 172.21.15.67:0/3303324975 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f626000bc00 con 0x7f627815cc00 2024-04-17T10:34:32.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.881+0000 7f626fe00640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6254066f60 0x7f6254069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:32.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.881+0000 7f626fe00640 1 -- 172.21.15.67:0/3303324975 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f62600854c0 con 0x7f627815cc00 2024-04-17T10:34:32.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.881+0000 7f6277400640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6254066f60 0x7f6254069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:32.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.881+0000 7f6277400640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6254066f60 0x7f6254069420 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f62680045f0 tx=0x7f6268008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:32.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:32.885+0000 7f626fe00640 1 -- 172.21.15.67:0/3303324975 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6260055d80 con 0x7f627815cc00 2024-04-17T10:34:33.038 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.045+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62780cf040 con 0x7f6254066f60 2024-04-17T10:34:33.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.053+0000 7f626fe00640 1 -- 172.21.15.67:0/3303324975 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252486 (secure 0 0 0) 0x7f62780cf040 con 0x7f6254066f60 2024-04-17T10:34:33.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:34:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6254066f60 msgr2=0x7f6254069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6254066f60 0x7f6254069420 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f62680045f0 tx=0x7f6268008040 comp rx=0 tx=0).stop 2024-04-17T10:34:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815cc00 msgr2=0x7f62780b7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815cc00 0x7f62780b7220 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f626000d800 tx=0x7f626000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 shutdown_connections 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6254066f60 0x7f6254069420 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f627815fa00 0x7f6278157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f627815cc00 0x7f62780b7220 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 --2- 172.21.15.67:0/3303324975 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f62780ce1b0 0x7f6278158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 >> 172.21.15.67:0/3303324975 conn(0x7f62780b33f0 msgr2=0x7f62780c6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 shutdown_connections 2024-04-17T10:34:33.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.057+0000 7f627e600640 1 -- 172.21.15.67:0/3303324975 wait complete. 2024-04-17T10:34:33.067 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-04-17T10:34:32.273749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":441056,"kb_used_data":13928,"kb_used_omap":12,"kb_used_meta":426995,"kb_avail":754533664,"statfs":{"total":773094113280,"available":772642471936,"internally_reserved":0,"allocated":14262272,"data_stored":7268140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":437243441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.347648"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110835+0000","last_change":"2024-04-17T10:34:28.974594+0000","last_active":"2024-04-17T10:34:30.110835+0000","last_peered":"2024-04-17T10:34:30.110835+0000","last_clean":"2024-04-17T10:34:30.110835+0000","last_became_active":"2024-04-17T10:34:28.974307+0000","last_became_peered":"2024-04-17T10:34:28.974307+0000","last_unstale":"2024-04-17T10:34:30.110835+0000","last_undegraded":"2024-04-17T10:34:30.110835+0000","last_fullsized":"2024-04-17T10:34:30.110835+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:00.478597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966754+0000","last_change":"2024-04-17T10:34:28.958804+0000","last_active":"2024-04-17T10:34:29.966754+0000","last_peered":"2024-04-17T10:34:29.966754+0000","last_clean":"2024-04-17T10:34:29.966754+0000","last_became_active":"2024-04-17T10:34:28.958536+0000","last_became_peered":"2024-04-17T10:34:28.958536+0000","last_unstale":"2024-04-17T10:34:29.966754+0000","last_undegraded":"2024-04-17T10:34:29.966754+0000","last_fullsized":"2024-04-17T10:34:29.966754+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:47.018491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108277+0000","last_change":"2024-04-17T10:34:28.975736+0000","last_active":"2024-04-17T10:34:30.108277+0000","last_peered":"2024-04-17T10:34:30.108277+0000","last_clean":"2024-04-17T10:34:30.108277+0000","last_became_active":"2024-04-17T10:34:28.975409+0000","last_became_peered":"2024-04-17T10:34:28.975409+0000","last_unstale":"2024-04-17T10:34:30.108277+0000","last_undegraded":"2024-04-17T10:34:30.108277+0000","last_fullsized":"2024-04-17T10:34:30.108277+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:44:14.364854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966778+0000","last_change":"2024-04-17T10:34:28.970318+0000","last_active":"2024-04-17T10:34:29.966778+0000","last_peered":"2024-04-17T10:34:29.966778+0000","last_clean":"2024-04-17T10:34:29.966778+0000","last_became_active":"2024-04-17T10:34:28.970019+0000","last_became_peered":"2024-04-17T10:34:28.970019+0000","last_unstale":"2024-04-17T10:34:29.966778+0000","last_undegraded":"2024-04-17T10:34:29.966778+0000","last_fullsized":"2024-04-17T10:34:29.966778+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:25.024758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111024+0000","last_change":"2024-04-17T10:34:28.980623+0000","last_active":"2024-04-17T10:34:30.111024+0000","last_peered":"2024-04-17T10:34:30.111024+0000","last_clean":"2024-04-17T10:34:30.111024+0000","last_became_active":"2024-04-17T10:34:28.980140+0000","last_became_peered":"2024-04-17T10:34:28.980140+0000","last_unstale":"2024-04-17T10:34:30.111024+0000","last_undegraded":"2024-04-17T10:34:30.111024+0000","last_fullsized":"2024-04-17T10:34:30.111024+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:03.361676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967753+0000","last_change":"2024-04-17T10:34:28.982156+0000","last_active":"2024-04-17T10:34:29.967753+0000","last_peered":"2024-04-17T10:34:29.967753+0000","last_clean":"2024-04-17T10:34:29.967753+0000","last_became_active":"2024-04-17T10:34:28.981840+0000","last_became_peered":"2024-04-17T10:34:28.981840+0000","last_unstale":"2024-04-17T10:34:29.967753+0000","last_undegraded":"2024-04-17T10:34:29.967753+0000","last_fullsized":"2024-04-17T10:34:29.967753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:52:33.443118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108458+0000","last_change":"2024-04-17T10:34:28.987629+0000","last_active":"2024-04-17T10:34:30.108458+0000","last_peered":"2024-04-17T10:34:30.108458+0000","last_clean":"2024-04-17T10:34:30.108458+0000","last_became_active":"2024-04-17T10:34:28.986810+0000","last_became_peered":"2024-04-17T10:34:28.986810+0000","last_unstale":"2024-04-17T10:34:30.108458+0000","last_undegraded":"2024-04-17T10:34:30.108458+0000","last_fullsized":"2024-04-17T10:34:30.108458+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:04.516924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.965885+0000","last_change":"2024-04-17T10:34:28.964128+0000","last_active":"2024-04-17T10:34:29.965885+0000","last_peered":"2024-04-17T10:34:29.965885+0000","last_clean":"2024-04-17T10:34:29.965885+0000","last_became_active":"2024-04-17T10:34:28.963815+0000","last_became_peered":"2024-04-17T10:34:28.963815+0000","last_unstale":"2024-04-17T10:34:29.965885+0000","last_undegraded":"2024-04-17T10:34:29.965885+0000","last_fullsized":"2024-04-17T10:34:29.965885+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:52.397412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968269+0000","last_change":"2024-04-17T10:34:28.966082+0000","last_active":"2024-04-17T10:34:29.968269+0000","last_peered":"2024-04-17T10:34:29.968269+0000","last_clean":"2024-04-17T10:34:29.968269+0000","last_became_active":"2024-04-17T10:34:28.965745+0000","last_became_peered":"2024-04-17T10:34:28.965745+0000","last_unstale":"2024-04-17T10:34:29.968269+0000","last_undegraded":"2024-04-17T10:34:29.968269+0000","last_fullsized":"2024-04-17T10:34:29.968269+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:58.020446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110170+0000","last_change":"2024-04-17T10:34:28.979428+0000","last_active":"2024-04-17T10:34:30.110170+0000","last_peered":"2024-04-17T10:34:30.110170+0000","last_clean":"2024-04-17T10:34:30.110170+0000","last_became_active":"2024-04-17T10:34:28.979012+0000","last_became_peered":"2024-04-17T10:34:28.979012+0000","last_unstale":"2024-04-17T10:34:30.110170+0000","last_undegraded":"2024-04-17T10:34:30.110170+0000","last_fullsized":"2024-04-17T10:34:30.110170+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:24.731406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966635+0000","last_change":"2024-04-17T10:34:28.978199+0000","last_active":"2024-04-17T10:34:29.966635+0000","last_peered":"2024-04-17T10:34:29.966635+0000","last_clean":"2024-04-17T10:34:29.966635+0000","last_became_active":"2024-04-17T10:34:28.977904+0000","last_became_peered":"2024-04-17T10:34:28.977904+0000","last_unstale":"2024-04-17T10:34:29.966635+0000","last_undegraded":"2024-04-17T10:34:29.966635+0000","last_fullsized":"2024-04-17T10:34:29.966635+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:58.020446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110724+0000","last_change":"2024-04-17T10:34:28.975443+0000","last_active":"2024-04-17T10:34:30.110724+0000","last_peered":"2024-04-17T10:34:30.110724+0000","last_clean":"2024-04-17T10:34:30.110724+0000","last_became_active":"2024-04-17T10:34:28.973158+0000","last_became_peered":"2024-04-17T10:34:28.973158+0000","last_unstale":"2024-04-17T10:34:30.110724+0000","last_undegraded":"2024-04-17T10:34:30.110724+0000","last_fullsized":"2024-04-17T10:34:30.110724+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:15.691481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967899+0000","last_change":"2024-04-17T10:34:28.985762+0000","last_active":"2024-04-17T10:34:29.967899+0000","last_peered":"2024-04-17T10:34:29.967899+0000","last_clean":"2024-04-17T10:34:29.967899+0000","last_became_active":"2024-04-17T10:34:28.985474+0000","last_became_peered":"2024-04-17T10:34:28.985474+0000","last_unstale":"2024-04-17T10:34:29.967899+0000","last_undegraded":"2024-04-17T10:34:29.967899+0000","last_fullsized":"2024-04-17T10:34:29.967899+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:52.195084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110165+0000","last_change":"2024-04-17T10:34:28.987434+0000","last_active":"2024-04-17T10:34:30.110165+0000","last_peered":"2024-04-17T10:34:30.110165+0000","last_clean":"2024-04-17T10:34:30.110165+0000","last_became_active":"2024-04-17T10:34:28.987125+0000","last_became_peered":"2024-04-17T10:34:28.987125+0000","last_unstale":"2024-04-17T10:34:30.110165+0000","last_undegraded":"2024-04-17T10:34:30.110165+0000","last_fullsized":"2024-04-17T10:34:30.110165+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:11.076299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109797+0000","last_change":"2024-04-17T10:34:28.986015+0000","last_active":"2024-04-17T10:34:30.109797+0000","last_peered":"2024-04-17T10:34:30.109797+0000","last_clean":"2024-04-17T10:34:30.109797+0000","last_became_active":"2024-04-17T10:34:28.985713+0000","last_became_peered":"2024-04-17T10:34:28.985713+0000","last_unstale":"2024-04-17T10:34:30.109797+0000","last_undegraded":"2024-04-17T10:34:30.109797+0000","last_fullsized":"2024-04-17T10:34:30.109797+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:05.775616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967522+0000","last_change":"2024-04-17T10:34:28.973205+0000","last_active":"2024-04-17T10:34:29.967522+0000","last_peered":"2024-04-17T10:34:29.967522+0000","last_clean":"2024-04-17T10:34:29.967522+0000","last_became_active":"2024-04-17T10:34:28.972667+0000","last_became_peered":"2024-04-17T10:34:28.972667+0000","last_unstale":"2024-04-17T10:34:29.967522+0000","last_undegraded":"2024-04-17T10:34:29.967522+0000","last_fullsized":"2024-04-17T10:34:29.967522+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:33.384275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968330+0000","last_change":"2024-04-17T10:34:28.970943+0000","last_active":"2024-04-17T10:34:29.968330+0000","last_peered":"2024-04-17T10:34:29.968330+0000","last_clean":"2024-04-17T10:34:29.968330+0000","last_became_active":"2024-04-17T10:34:28.970652+0000","last_became_peered":"2024-04-17T10:34:28.970652+0000","last_unstale":"2024-04-17T10:34:29.968330+0000","last_undegraded":"2024-04-17T10:34:29.968330+0000","last_fullsized":"2024-04-17T10:34:29.968330+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:47.018491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111129+0000","last_change":"2024-04-17T10:34:28.977580+0000","last_active":"2024-04-17T10:34:30.111129+0000","last_peered":"2024-04-17T10:34:30.111129+0000","last_clean":"2024-04-17T10:34:30.111129+0000","last_became_active":"2024-04-17T10:34:28.977270+0000","last_became_peered":"2024-04-17T10:34:28.977270+0000","last_unstale":"2024-04-17T10:34:30.111129+0000","last_undegraded":"2024-04-17T10:34:30.111129+0000","last_fullsized":"2024-04-17T10:34:30.111129+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:18:44.864368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108020+0000","last_change":"2024-04-17T10:34:28.986292+0000","last_active":"2024-04-17T10:34:30.108020+0000","last_peered":"2024-04-17T10:34:30.108020+0000","last_clean":"2024-04-17T10:34:30.108020+0000","last_became_active":"2024-04-17T10:34:28.985852+0000","last_became_peered":"2024-04-17T10:34:28.985852+0000","last_unstale":"2024-04-17T10:34:30.108020+0000","last_undegraded":"2024-04-17T10:34:30.108020+0000","last_fullsized":"2024-04-17T10:34:30.108020+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:21.004154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111268+0000","last_change":"2024-04-17T10:34:28.991008+0000","last_active":"2024-04-17T10:34:30.111268+0000","last_peered":"2024-04-17T10:34:30.111268+0000","last_clean":"2024-04-17T10:34:30.111268+0000","last_became_active":"2024-04-17T10:34:28.990696+0000","last_became_peered":"2024-04-17T10:34:28.990696+0000","last_unstale":"2024-04-17T10:34:30.111268+0000","last_undegraded":"2024-04-17T10:34:30.111268+0000","last_fullsized":"2024-04-17T10:34:30.111268+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:30.067543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966870+0000","last_change":"2024-04-17T10:34:28.961096+0000","last_active":"2024-04-17T10:34:29.966870+0000","last_peered":"2024-04-17T10:34:29.966870+0000","last_clean":"2024-04-17T10:34:29.966870+0000","last_became_active":"2024-04-17T10:34:28.960767+0000","last_became_peered":"2024-04-17T10:34:28.960767+0000","last_unstale":"2024-04-17T10:34:29.966870+0000","last_undegraded":"2024-04-17T10:34:29.966870+0000","last_fullsized":"2024-04-17T10:34:29.966870+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:52:16.087256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110026+0000","last_change":"2024-04-17T10:34:28.962758+0000","last_active":"2024-04-17T10:34:30.110026+0000","last_peered":"2024-04-17T10:34:30.110026+0000","last_clean":"2024-04-17T10:34:30.110026+0000","last_became_active":"2024-04-17T10:34:28.962088+0000","last_became_peered":"2024-04-17T10:34:28.962088+0000","last_unstale":"2024-04-17T10:34:30.110026+0000","last_undegraded":"2024-04-17T10:34:30.110026+0000","last_fullsized":"2024-04-17T10:34:30.110026+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:29.753156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966161+0000","last_change":"2024-04-17T10:34:28.982852+0000","last_active":"2024-04-17T10:34:29.966161+0000","last_peered":"2024-04-17T10:34:29.966161+0000","last_clean":"2024-04-17T10:34:29.966161+0000","last_became_active":"2024-04-17T10:34:28.982223+0000","last_became_peered":"2024-04-17T10:34:28.982223+0000","last_unstale":"2024-04-17T10:34:29.966161+0000","last_undegraded":"2024-04-17T10:34:29.966161+0000","last_fullsized":"2024-04-17T10:34:29.966161+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:52.718288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967565+0000","last_change":"2024-04-17T10:34:28.966086+0000","last_active":"2024-04-17T10:34:29.967565+0000","last_peered":"2024-04-17T10:34:29.967565+0000","last_clean":"2024-04-17T10:34:29.967565+0000","last_became_active":"2024-04-17T10:34:28.965757+0000","last_became_peered":"2024-04-17T10:34:28.965757+0000","last_unstale":"2024-04-17T10:34:29.967565+0000","last_undegraded":"2024-04-17T10:34:29.967565+0000","last_fullsized":"2024-04-17T10:34:29.967565+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:23.954687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108664+0000","last_change":"2024-04-17T10:34:28.975208+0000","last_active":"2024-04-17T10:34:30.108664+0000","last_peered":"2024-04-17T10:34:30.108664+0000","last_clean":"2024-04-17T10:34:30.108664+0000","last_became_active":"2024-04-17T10:34:28.974913+0000","last_became_peered":"2024-04-17T10:34:28.974913+0000","last_unstale":"2024-04-17T10:34:30.108664+0000","last_undegraded":"2024-04-17T10:34:30.108664+0000","last_fullsized":"2024-04-17T10:34:30.108664+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:01.538678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108466+0000","last_change":"2024-04-17T10:34:28.984601+0000","last_active":"2024-04-17T10:34:30.108466+0000","last_peered":"2024-04-17T10:34:30.108466+0000","last_clean":"2024-04-17T10:34:30.108466+0000","last_became_active":"2024-04-17T10:34:28.984350+0000","last_became_peered":"2024-04-17T10:34:28.984350+0000","last_unstale":"2024-04-17T10:34:30.108466+0000","last_undegraded":"2024-04-17T10:34:30.108466+0000","last_fullsized":"2024-04-17T10:34:30.108466+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:52.397412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108176+0000","last_change":"2024-04-17T10:34:28.991515+0000","last_active":"2024-04-17T10:34:30.108176+0000","last_peered":"2024-04-17T10:34:30.108176+0000","last_clean":"2024-04-17T10:34:30.108176+0000","last_became_active":"2024-04-17T10:34:28.991177+0000","last_became_peered":"2024-04-17T10:34:28.991177+0000","last_unstale":"2024-04-17T10:34:30.108176+0000","last_undegraded":"2024-04-17T10:34:30.108176+0000","last_fullsized":"2024-04-17T10:34:30.108176+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:58:51.767495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110114+0000","last_change":"2024-04-17T10:34:28.979151+0000","last_active":"2024-04-17T10:34:30.110114+0000","last_peered":"2024-04-17T10:34:30.110114+0000","last_clean":"2024-04-17T10:34:30.110114+0000","last_became_active":"2024-04-17T10:34:28.978652+0000","last_became_peered":"2024-04-17T10:34:28.978652+0000","last_unstale":"2024-04-17T10:34:30.110114+0000","last_undegraded":"2024-04-17T10:34:30.110114+0000","last_fullsized":"2024-04-17T10:34:30.110114+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:52:55.987893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107845+0000","last_change":"2024-04-17T10:34:28.976197+0000","last_active":"2024-04-17T10:34:30.107845+0000","last_peered":"2024-04-17T10:34:30.107845+0000","last_clean":"2024-04-17T10:34:30.107845+0000","last_became_active":"2024-04-17T10:34:28.975904+0000","last_became_peered":"2024-04-17T10:34:28.975904+0000","last_unstale":"2024-04-17T10:34:30.107845+0000","last_undegraded":"2024-04-17T10:34:30.107845+0000","last_fullsized":"2024-04-17T10:34:30.107845+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:49.628136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107628+0000","last_change":"2024-04-17T10:34:28.974772+0000","last_active":"2024-04-17T10:34:30.107628+0000","last_peered":"2024-04-17T10:34:30.107628+0000","last_clean":"2024-04-17T10:34:30.107628+0000","last_became_active":"2024-04-17T10:34:28.974450+0000","last_became_peered":"2024-04-17T10:34:28.974450+0000","last_unstale":"2024-04-17T10:34:30.107628+0000","last_undegraded":"2024-04-17T10:34:30.107628+0000","last_fullsized":"2024-04-17T10:34:30.107628+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:00.766283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.965895+0000","last_change":"2024-04-17T10:34:28.968028+0000","last_active":"2024-04-17T10:34:29.965895+0000","last_peered":"2024-04-17T10:34:29.965895+0000","last_clean":"2024-04-17T10:34:29.965895+0000","last_became_active":"2024-04-17T10:34:28.967736+0000","last_became_peered":"2024-04-17T10:34:28.967736+0000","last_unstale":"2024-04-17T10:34:29.965895+0000","last_undegraded":"2024-04-17T10:34:29.965895+0000","last_fullsized":"2024-04-17T10:34:29.965895+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:01.538678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109893+0000","last_change":"2024-04-17T10:34:28.977742+0000","last_active":"2024-04-17T10:34:30.109893+0000","last_peered":"2024-04-17T10:34:30.109893+0000","last_clean":"2024-04-17T10:34:30.109893+0000","last_became_active":"2024-04-17T10:34:28.977285+0000","last_became_peered":"2024-04-17T10:34:28.977285+0000","last_unstale":"2024-04-17T10:34:30.109893+0000","last_undegraded":"2024-04-17T10:34:30.109893+0000","last_fullsized":"2024-04-17T10:34:30.109893+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:35.778397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966058+0000","last_change":"2024-04-17T10:34:28.983315+0000","last_active":"2024-04-17T10:34:29.966058+0000","last_peered":"2024-04-17T10:34:29.966058+0000","last_clean":"2024-04-17T10:34:29.966058+0000","last_became_active":"2024-04-17T10:34:28.982983+0000","last_became_peered":"2024-04-17T10:34:28.982983+0000","last_unstale":"2024-04-17T10:34:29.966058+0000","last_undegraded":"2024-04-17T10:34:29.966058+0000","last_fullsized":"2024-04-17T10:34:29.966058+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:11.721543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108582+0000","last_change":"2024-04-17T10:34:27.949473+0000","last_active":"2024-04-17T10:34:30.108582+0000","last_peered":"2024-04-17T10:34:30.108582+0000","last_clean":"2024-04-17T10:34:30.108582+0000","last_became_active":"2024-04-17T10:34:27.949190+0000","last_became_peered":"2024-04-17T10:34:27.949190+0000","last_unstale":"2024-04-17T10:34:30.108582+0000","last_undegraded":"2024-04-17T10:34:30.108582+0000","last_fullsized":"2024-04-17T10:34:30.108582+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:28.054550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110083+0000","last_change":"2024-04-17T10:34:28.962284+0000","last_active":"2024-04-17T10:34:30.110083+0000","last_peered":"2024-04-17T10:34:30.110083+0000","last_clean":"2024-04-17T10:34:30.110083+0000","last_became_active":"2024-04-17T10:34:28.961711+0000","last_became_peered":"2024-04-17T10:34:28.961711+0000","last_unstale":"2024-04-17T10:34:30.110083+0000","last_undegraded":"2024-04-17T10:34:30.110083+0000","last_fullsized":"2024-04-17T10:34:30.110083+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:20.878852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111386+0000","last_change":"2024-04-17T10:34:27.953510+0000","last_active":"2024-04-17T10:34:30.111386+0000","last_peered":"2024-04-17T10:34:30.111386+0000","last_clean":"2024-04-17T10:34:30.111386+0000","last_became_active":"2024-04-17T10:34:27.952580+0000","last_became_peered":"2024-04-17T10:34:27.952580+0000","last_unstale":"2024-04-17T10:34:30.111386+0000","last_undegraded":"2024-04-17T10:34:30.111386+0000","last_fullsized":"2024-04-17T10:34:30.111386+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:59.938245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966557+0000","last_change":"2024-04-17T10:34:28.983422+0000","last_active":"2024-04-17T10:34:29.966557+0000","last_peered":"2024-04-17T10:34:29.966557+0000","last_clean":"2024-04-17T10:34:29.966557+0000","last_became_active":"2024-04-17T10:34:28.982854+0000","last_became_peered":"2024-04-17T10:34:28.982854+0000","last_unstale":"2024-04-17T10:34:29.966557+0000","last_undegraded":"2024-04-17T10:34:29.966557+0000","last_fullsized":"2024-04-17T10:34:29.966557+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:55:00.945480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108272+0000","last_change":"2024-04-17T10:34:27.953236+0000","last_active":"2024-04-17T10:34:30.108272+0000","last_peered":"2024-04-17T10:34:30.108272+0000","last_clean":"2024-04-17T10:34:30.108272+0000","last_became_active":"2024-04-17T10:34:27.952976+0000","last_became_peered":"2024-04-17T10:34:27.952976+0000","last_unstale":"2024-04-17T10:34:30.108272+0000","last_undegraded":"2024-04-17T10:34:30.108272+0000","last_fullsized":"2024-04-17T10:34:30.108272+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:51.711052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110279+0000","last_change":"2024-04-17T10:34:28.987978+0000","last_active":"2024-04-17T10:34:30.110279+0000","last_peered":"2024-04-17T10:34:30.110279+0000","last_clean":"2024-04-17T10:34:30.110279+0000","last_became_active":"2024-04-17T10:34:28.987675+0000","last_became_peered":"2024-04-17T10:34:28.987675+0000","last_unstale":"2024-04-17T10:34:30.110279+0000","last_undegraded":"2024-04-17T10:34:30.110279+0000","last_fullsized":"2024-04-17T10:34:30.110279+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:29.314968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108157+0000","last_change":"2024-04-17T10:34:27.948168+0000","last_active":"2024-04-17T10:34:30.108157+0000","last_peered":"2024-04-17T10:34:30.108157+0000","last_clean":"2024-04-17T10:34:30.108157+0000","last_became_active":"2024-04-17T10:34:27.947102+0000","last_became_peered":"2024-04-17T10:34:27.947102+0000","last_unstale":"2024-04-17T10:34:30.108157+0000","last_undegraded":"2024-04-17T10:34:30.108157+0000","last_fullsized":"2024-04-17T10:34:30.108157+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:55.879630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967234+0000","last_change":"2024-04-17T10:34:28.968675+0000","last_active":"2024-04-17T10:34:29.967234+0000","last_peered":"2024-04-17T10:34:29.967234+0000","last_clean":"2024-04-17T10:34:29.967234+0000","last_became_active":"2024-04-17T10:34:28.968367+0000","last_became_peered":"2024-04-17T10:34:28.968367+0000","last_unstale":"2024-04-17T10:34:29.967234+0000","last_undegraded":"2024-04-17T10:34:29.967234+0000","last_fullsized":"2024-04-17T10:34:29.967234+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:50:10.822976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966468+0000","last_change":"2024-04-17T10:34:27.935854+0000","last_active":"2024-04-17T10:34:29.966468+0000","last_peered":"2024-04-17T10:34:29.966468+0000","last_clean":"2024-04-17T10:34:29.966468+0000","last_became_active":"2024-04-17T10:34:27.935517+0000","last_became_peered":"2024-04-17T10:34:27.935517+0000","last_unstale":"2024-04-17T10:34:29.966468+0000","last_undegraded":"2024-04-17T10:34:29.966468+0000","last_fullsized":"2024-04-17T10:34:29.966468+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:59.759048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967329+0000","last_change":"2024-04-17T10:34:28.970218+0000","last_active":"2024-04-17T10:34:29.967329+0000","last_peered":"2024-04-17T10:34:29.967329+0000","last_clean":"2024-04-17T10:34:29.967329+0000","last_became_active":"2024-04-17T10:34:28.969894+0000","last_became_peered":"2024-04-17T10:34:28.969894+0000","last_unstale":"2024-04-17T10:34:29.967329+0000","last_undegraded":"2024-04-17T10:34:29.967329+0000","last_fullsized":"2024-04-17T10:34:29.967329+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:33.384275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110775+0000","last_change":"2024-04-17T10:34:27.938224+0000","last_active":"2024-04-17T10:34:30.110775+0000","last_peered":"2024-04-17T10:34:30.110775+0000","last_clean":"2024-04-17T10:34:30.110775+0000","last_became_active":"2024-04-17T10:34:27.937929+0000","last_became_peered":"2024-04-17T10:34:27.937929+0000","last_unstale":"2024-04-17T10:34:30.110775+0000","last_undegraded":"2024-04-17T10:34:30.110775+0000","last_fullsized":"2024-04-17T10:34:30.110775+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:55.879630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111344+0000","last_change":"2024-04-17T10:34:28.986619+0000","last_active":"2024-04-17T10:34:30.111344+0000","last_peered":"2024-04-17T10:34:30.111344+0000","last_clean":"2024-04-17T10:34:30.111344+0000","last_became_active":"2024-04-17T10:34:28.986332+0000","last_became_peered":"2024-04-17T10:34:28.986332+0000","last_unstale":"2024-04-17T10:34:30.111344+0000","last_undegraded":"2024-04-17T10:34:30.111344+0000","last_fullsized":"2024-04-17T10:34:30.111344+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:18:09.832833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967445+0000","last_change":"2024-04-17T10:34:27.949955+0000","last_active":"2024-04-17T10:34:29.967445+0000","last_peered":"2024-04-17T10:34:29.967445+0000","last_clean":"2024-04-17T10:34:29.967445+0000","last_became_active":"2024-04-17T10:34:27.949460+0000","last_became_peered":"2024-04-17T10:34:27.949460+0000","last_unstale":"2024-04-17T10:34:29.967445+0000","last_undegraded":"2024-04-17T10:34:29.967445+0000","last_fullsized":"2024-04-17T10:34:29.967445+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:48.940362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108886+0000","last_change":"2024-04-17T10:34:28.983387+0000","last_active":"2024-04-17T10:34:30.108886+0000","last_peered":"2024-04-17T10:34:30.108886+0000","last_clean":"2024-04-17T10:34:30.108886+0000","last_became_active":"2024-04-17T10:34:28.982989+0000","last_became_peered":"2024-04-17T10:34:28.982989+0000","last_unstale":"2024-04-17T10:34:30.108886+0000","last_undegraded":"2024-04-17T10:34:30.108886+0000","last_fullsized":"2024-04-17T10:34:30.108886+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:29.879975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107421+0000","last_change":"2024-04-17T10:34:27.947754+0000","last_active":"2024-04-17T10:34:30.107421+0000","last_peered":"2024-04-17T10:34:30.107421+0000","last_clean":"2024-04-17T10:34:30.107421+0000","last_became_active":"2024-04-17T10:34:27.947434+0000","last_became_peered":"2024-04-17T10:34:27.947434+0000","last_unstale":"2024-04-17T10:34:30.107421+0000","last_undegraded":"2024-04-17T10:34:30.107421+0000","last_fullsized":"2024-04-17T10:34:30.107421+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:35.671346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109784+0000","last_change":"2024-04-17T10:34:28.981842+0000","last_active":"2024-04-17T10:34:30.109784+0000","last_peered":"2024-04-17T10:34:30.109784+0000","last_clean":"2024-04-17T10:34:30.109784+0000","last_became_active":"2024-04-17T10:34:28.981557+0000","last_became_peered":"2024-04-17T10:34:28.981557+0000","last_unstale":"2024-04-17T10:34:30.109784+0000","last_undegraded":"2024-04-17T10:34:30.109784+0000","last_fullsized":"2024-04-17T10:34:30.109784+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:02.696728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966724+0000","last_change":"2024-04-17T10:34:27.953065+0000","last_active":"2024-04-17T10:34:29.966724+0000","last_peered":"2024-04-17T10:34:29.966724+0000","last_clean":"2024-04-17T10:34:29.966724+0000","last_became_active":"2024-04-17T10:34:27.952542+0000","last_became_peered":"2024-04-17T10:34:27.952542+0000","last_unstale":"2024-04-17T10:34:29.966724+0000","last_undegraded":"2024-04-17T10:34:29.966724+0000","last_fullsized":"2024-04-17T10:34:29.966724+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:51.390177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967636+0000","last_change":"2024-04-17T10:34:28.972014+0000","last_active":"2024-04-17T10:34:29.967636+0000","last_peered":"2024-04-17T10:34:29.967636+0000","last_clean":"2024-04-17T10:34:29.967636+0000","last_became_active":"2024-04-17T10:34:28.971704+0000","last_became_peered":"2024-04-17T10:34:28.971704+0000","last_unstale":"2024-04-17T10:34:29.967636+0000","last_undegraded":"2024-04-17T10:34:29.967636+0000","last_fullsized":"2024-04-17T10:34:29.967636+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:37:12.984255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110677+0000","last_change":"2024-04-17T10:34:27.957469+0000","last_active":"2024-04-17T10:34:30.110677+0000","last_peered":"2024-04-17T10:34:30.110677+0000","last_clean":"2024-04-17T10:34:30.110677+0000","last_became_active":"2024-04-17T10:34:27.957144+0000","last_became_peered":"2024-04-17T10:34:27.957144+0000","last_unstale":"2024-04-17T10:34:30.110677+0000","last_undegraded":"2024-04-17T10:34:30.110677+0000","last_fullsized":"2024-04-17T10:34:30.110677+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:02.267626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107720+0000","last_change":"2024-04-17T10:34:28.975429+0000","last_active":"2024-04-17T10:34:30.107720+0000","last_peered":"2024-04-17T10:34:30.107720+0000","last_clean":"2024-04-17T10:34:30.107720+0000","last_became_active":"2024-04-17T10:34:28.975066+0000","last_became_peered":"2024-04-17T10:34:28.975066+0000","last_unstale":"2024-04-17T10:34:30.107720+0000","last_undegraded":"2024-04-17T10:34:30.107720+0000","last_fullsized":"2024-04-17T10:34:30.107720+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:52.896649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"129'3","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108272+0000","last_change":"2024-04-17T10:34:27.948950+0000","last_active":"2024-04-17T10:34:30.108272+0000","last_peered":"2024-04-17T10:34:30.108272+0000","last_clean":"2024-04-17T10:34:30.108272+0000","last_became_active":"2024-04-17T10:34:27.945716+0000","last_became_peered":"2024-04-17T10:34:27.945716+0000","last_unstale":"2024-04-17T10:34:30.108272+0000","last_undegraded":"2024-04-17T10:34:30.108272+0000","last_fullsized":"2024-04-17T10:34:30.108272+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:59.938245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108063+0000","last_change":"2024-04-17T10:34:28.975574+0000","last_active":"2024-04-17T10:34:30.108063+0000","last_peered":"2024-04-17T10:34:30.108063+0000","last_clean":"2024-04-17T10:34:30.108063+0000","last_became_active":"2024-04-17T10:34:28.975192+0000","last_became_peered":"2024-04-17T10:34:28.975192+0000","last_unstale":"2024-04-17T10:34:30.108063+0000","last_undegraded":"2024-04-17T10:34:30.108063+0000","last_fullsized":"2024-04-17T10:34:30.108063+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:11.440956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"129'6","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108284+0000","last_change":"2024-04-17T10:34:27.947263+0000","last_active":"2024-04-17T10:34:30.108284+0000","last_peered":"2024-04-17T10:34:30.108284+0000","last_clean":"2024-04-17T10:34:30.108284+0000","last_became_active":"2024-04-17T10:34:27.946998+0000","last_became_peered":"2024-04-17T10:34:27.946998+0000","last_unstale":"2024-04-17T10:34:30.108284+0000","last_undegraded":"2024-04-17T10:34:30.108284+0000","last_fullsized":"2024-04-17T10:34:30.108284+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:24.017523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110252+0000","last_change":"2024-04-17T10:34:28.971565+0000","last_active":"2024-04-17T10:34:30.110252+0000","last_peered":"2024-04-17T10:34:30.110252+0000","last_clean":"2024-04-17T10:34:30.110252+0000","last_became_active":"2024-04-17T10:34:28.971280+0000","last_became_peered":"2024-04-17T10:34:28.971280+0000","last_unstale":"2024-04-17T10:34:30.110252+0000","last_undegraded":"2024-04-17T10:34:30.110252+0000","last_fullsized":"2024-04-17T10:34:30.110252+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:28:32.874971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110335+0000","last_change":"2024-04-17T10:34:27.960133+0000","last_active":"2024-04-17T10:34:30.110335+0000","last_peered":"2024-04-17T10:34:30.110335+0000","last_clean":"2024-04-17T10:34:30.110335+0000","last_became_active":"2024-04-17T10:34:27.959866+0000","last_became_peered":"2024-04-17T10:34:27.959866+0000","last_unstale":"2024-04-17T10:34:30.110335+0000","last_undegraded":"2024-04-17T10:34:30.110335+0000","last_fullsized":"2024-04-17T10:34:30.110335+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:11.592722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108837+0000","last_change":"2024-04-17T10:34:28.983729+0000","last_active":"2024-04-17T10:34:30.108837+0000","last_peered":"2024-04-17T10:34:30.108837+0000","last_clean":"2024-04-17T10:34:30.108837+0000","last_became_active":"2024-04-17T10:34:28.983332+0000","last_became_peered":"2024-04-17T10:34:28.983332+0000","last_unstale":"2024-04-17T10:34:30.108837+0000","last_undegraded":"2024-04-17T10:34:30.108837+0000","last_fullsized":"2024-04-17T10:34:30.108837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:28.986152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107426+0000","last_change":"2024-04-17T10:34:27.952616+0000","last_active":"2024-04-17T10:34:30.107426+0000","last_peered":"2024-04-17T10:34:30.107426+0000","last_clean":"2024-04-17T10:34:30.107426+0000","last_became_active":"2024-04-17T10:34:27.952361+0000","last_became_peered":"2024-04-17T10:34:27.952361+0000","last_unstale":"2024-04-17T10:34:30.107426+0000","last_undegraded":"2024-04-17T10:34:30.107426+0000","last_fullsized":"2024-04-17T10:34:30.107426+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:46.196119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111366+0000","last_change":"2024-04-17T10:34:28.975608+0000","last_active":"2024-04-17T10:34:30.111366+0000","last_peered":"2024-04-17T10:34:30.111366+0000","last_clean":"2024-04-17T10:34:30.111366+0000","last_became_active":"2024-04-17T10:34:28.974801+0000","last_became_peered":"2024-04-17T10:34:28.974801+0000","last_unstale":"2024-04-17T10:34:30.111366+0000","last_undegraded":"2024-04-17T10:34:30.111366+0000","last_fullsized":"2024-04-17T10:34:30.111366+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:58:51.767495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"129'8","reported_seq":22,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967582+0000","last_change":"2024-04-17T10:34:27.947171+0000","last_active":"2024-04-17T10:34:29.967582+0000","last_peered":"2024-04-17T10:34:29.967582+0000","last_clean":"2024-04-17T10:34:29.967582+0000","last_became_active":"2024-04-17T10:34:27.946849+0000","last_became_peered":"2024-04-17T10:34:27.946849+0000","last_unstale":"2024-04-17T10:34:29.967582+0000","last_undegraded":"2024-04-17T10:34:29.967582+0000","last_fullsized":"2024-04-17T10:34:29.967582+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:57.944745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107631+0000","last_change":"2024-04-17T10:34:28.980083+0000","last_active":"2024-04-17T10:34:30.107631+0000","last_peered":"2024-04-17T10:34:30.107631+0000","last_clean":"2024-04-17T10:34:30.107631+0000","last_became_active":"2024-04-17T10:34:28.979637+0000","last_became_peered":"2024-04-17T10:34:28.979637+0000","last_unstale":"2024-04-17T10:34:30.107631+0000","last_undegraded":"2024-04-17T10:34:30.107631+0000","last_fullsized":"2024-04-17T10:34:30.107631+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:31.502413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966842+0000","last_change":"2024-04-17T10:34:27.953712+0000","last_active":"2024-04-17T10:34:29.966842+0000","last_peered":"2024-04-17T10:34:29.966842+0000","last_clean":"2024-04-17T10:34:29.966842+0000","last_became_active":"2024-04-17T10:34:27.953463+0000","last_became_peered":"2024-04-17T10:34:27.953463+0000","last_unstale":"2024-04-17T10:34:29.966842+0000","last_undegraded":"2024-04-17T10:34:29.966842+0000","last_fullsized":"2024-04-17T10:34:29.966842+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:10.714307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108033+0000","last_change":"2024-04-17T10:34:28.987847+0000","last_active":"2024-04-17T10:34:30.108033+0000","last_peered":"2024-04-17T10:34:30.108033+0000","last_clean":"2024-04-17T10:34:30.108033+0000","last_became_active":"2024-04-17T10:34:28.987362+0000","last_became_peered":"2024-04-17T10:34:28.987362+0000","last_unstale":"2024-04-17T10:34:30.108033+0000","last_undegraded":"2024-04-17T10:34:30.108033+0000","last_fullsized":"2024-04-17T10:34:30.108033+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:18:09.832833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108528+0000","last_change":"2024-04-17T10:34:27.949802+0000","last_active":"2024-04-17T10:34:30.108528+0000","last_peered":"2024-04-17T10:34:30.108528+0000","last_clean":"2024-04-17T10:34:30.108528+0000","last_became_active":"2024-04-17T10:34:27.949528+0000","last_became_peered":"2024-04-17T10:34:27.949528+0000","last_unstale":"2024-04-17T10:34:30.108528+0000","last_undegraded":"2024-04-17T10:34:30.108528+0000","last_fullsized":"2024-04-17T10:34:30.108528+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:10.823560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108609+0000","last_change":"2024-04-17T10:34:28.989985+0000","last_active":"2024-04-17T10:34:30.108609+0000","last_peered":"2024-04-17T10:34:30.108609+0000","last_clean":"2024-04-17T10:34:30.108609+0000","last_became_active":"2024-04-17T10:34:28.987071+0000","last_became_peered":"2024-04-17T10:34:28.987071+0000","last_unstale":"2024-04-17T10:34:30.108609+0000","last_undegraded":"2024-04-17T10:34:30.108609+0000","last_fullsized":"2024-04-17T10:34:30.108609+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:56.269481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968040+0000","last_change":"2024-04-17T10:34:27.949219+0000","last_active":"2024-04-17T10:34:29.968040+0000","last_peered":"2024-04-17T10:34:29.968040+0000","last_clean":"2024-04-17T10:34:29.968040+0000","last_became_active":"2024-04-17T10:34:27.948908+0000","last_became_peered":"2024-04-17T10:34:27.948908+0000","last_unstale":"2024-04-17T10:34:29.968040+0000","last_undegraded":"2024-04-17T10:34:29.968040+0000","last_fullsized":"2024-04-17T10:34:29.968040+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:59.759048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110524+0000","last_change":"2024-04-17T10:34:28.980366+0000","last_active":"2024-04-17T10:34:30.110524+0000","last_peered":"2024-04-17T10:34:30.110524+0000","last_clean":"2024-04-17T10:34:30.110524+0000","last_became_active":"2024-04-17T10:34:28.980114+0000","last_became_peered":"2024-04-17T10:34:28.980114+0000","last_unstale":"2024-04-17T10:34:30.110524+0000","last_undegraded":"2024-04-17T10:34:30.110524+0000","last_fullsized":"2024-04-17T10:34:30.110524+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:43:12.890477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111327+0000","last_change":"2024-04-17T10:34:27.952979+0000","last_active":"2024-04-17T10:34:30.111327+0000","last_peered":"2024-04-17T10:34:30.111327+0000","last_clean":"2024-04-17T10:34:30.111327+0000","last_became_active":"2024-04-17T10:34:27.952545+0000","last_became_peered":"2024-04-17T10:34:27.952545+0000","last_unstale":"2024-04-17T10:34:30.111327+0000","last_undegraded":"2024-04-17T10:34:30.111327+0000","last_fullsized":"2024-04-17T10:34:30.111327+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:51.711052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111393+0000","last_change":"2024-04-17T10:34:28.972194+0000","last_active":"2024-04-17T10:34:30.111393+0000","last_peered":"2024-04-17T10:34:30.111393+0000","last_clean":"2024-04-17T10:34:30.111393+0000","last_became_active":"2024-04-17T10:34:28.970371+0000","last_became_peered":"2024-04-17T10:34:28.970371+0000","last_unstale":"2024-04-17T10:34:30.111393+0000","last_undegraded":"2024-04-17T10:34:30.111393+0000","last_fullsized":"2024-04-17T10:34:30.111393+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:04.516924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966589+0000","last_change":"2024-04-17T10:34:27.945066+0000","last_active":"2024-04-17T10:34:29.966589+0000","last_peered":"2024-04-17T10:34:29.966589+0000","last_clean":"2024-04-17T10:34:29.966589+0000","last_became_active":"2024-04-17T10:34:27.944583+0000","last_became_peered":"2024-04-17T10:34:27.944583+0000","last_unstale":"2024-04-17T10:34:29.966589+0000","last_undegraded":"2024-04-17T10:34:29.966589+0000","last_fullsized":"2024-04-17T10:34:29.966589+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:52.175957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110402+0000","last_change":"2024-04-17T10:34:28.987393+0000","last_active":"2024-04-17T10:34:30.110402+0000","last_peered":"2024-04-17T10:34:30.110402+0000","last_clean":"2024-04-17T10:34:30.110402+0000","last_became_active":"2024-04-17T10:34:28.987111+0000","last_became_peered":"2024-04-17T10:34:28.987111+0000","last_unstale":"2024-04-17T10:34:30.110402+0000","last_undegraded":"2024-04-17T10:34:30.110402+0000","last_fullsized":"2024-04-17T10:34:30.110402+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:31.163780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110182+0000","last_change":"2024-04-17T10:34:27.950378+0000","last_active":"2024-04-17T10:34:30.110182+0000","last_peered":"2024-04-17T10:34:30.110182+0000","last_clean":"2024-04-17T10:34:30.110182+0000","last_became_active":"2024-04-17T10:34:27.950033+0000","last_became_peered":"2024-04-17T10:34:27.950033+0000","last_unstale":"2024-04-17T10:34:30.110182+0000","last_undegraded":"2024-04-17T10:34:30.110182+0000","last_fullsized":"2024-04-17T10:34:30.110182+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:04.390056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111321+0000","last_change":"2024-04-17T10:34:28.984017+0000","last_active":"2024-04-17T10:34:30.111321+0000","last_peered":"2024-04-17T10:34:30.111321+0000","last_clean":"2024-04-17T10:34:30.111321+0000","last_became_active":"2024-04-17T10:34:28.983702+0000","last_became_peered":"2024-04-17T10:34:28.983702+0000","last_unstale":"2024-04-17T10:34:30.111321+0000","last_undegraded":"2024-04-17T10:34:30.111321+0000","last_fullsized":"2024-04-17T10:34:30.111321+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:21.004154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108095+0000","last_change":"2024-04-17T10:34:27.955926+0000","last_active":"2024-04-17T10:34:30.108095+0000","last_peered":"2024-04-17T10:34:30.108095+0000","last_clean":"2024-04-17T10:34:30.108095+0000","last_became_active":"2024-04-17T10:34:27.955613+0000","last_became_peered":"2024-04-17T10:34:27.955613+0000","last_unstale":"2024-04-17T10:34:30.108095+0000","last_undegraded":"2024-04-17T10:34:30.108095+0000","last_fullsized":"2024-04-17T10:34:30.108095+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:24.825272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966864+0000","last_change":"2024-04-17T10:34:28.979339+0000","last_active":"2024-04-17T10:34:29.966864+0000","last_peered":"2024-04-17T10:34:29.966864+0000","last_clean":"2024-04-17T10:34:29.966864+0000","last_became_active":"2024-04-17T10:34:28.979069+0000","last_became_peered":"2024-04-17T10:34:28.979069+0000","last_unstale":"2024-04-17T10:34:29.966864+0000","last_undegraded":"2024-04-17T10:34:29.966864+0000","last_fullsized":"2024-04-17T10:34:29.966864+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:25.832508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967620+0000","last_change":"2024-04-17T10:34:27.956132+0000","last_active":"2024-04-17T10:34:29.967620+0000","last_peered":"2024-04-17T10:34:29.967620+0000","last_clean":"2024-04-17T10:34:29.967620+0000","last_became_active":"2024-04-17T10:34:27.955624+0000","last_became_peered":"2024-04-17T10:34:27.955624+0000","last_unstale":"2024-04-17T10:34:29.967620+0000","last_undegraded":"2024-04-17T10:34:29.967620+0000","last_fullsized":"2024-04-17T10:34:29.967620+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:52.175957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110992+0000","last_change":"2024-04-17T10:34:28.984158+0000","last_active":"2024-04-17T10:34:30.110992+0000","last_peered":"2024-04-17T10:34:30.110992+0000","last_clean":"2024-04-17T10:34:30.110992+0000","last_became_active":"2024-04-17T10:34:28.983850+0000","last_became_peered":"2024-04-17T10:34:28.983850+0000","last_unstale":"2024-04-17T10:34:30.110992+0000","last_undegraded":"2024-04-17T10:34:30.110992+0000","last_fullsized":"2024-04-17T10:34:30.110992+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:32:26.493336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111001+0000","last_change":"2024-04-17T10:34:27.958486+0000","last_active":"2024-04-17T10:34:30.111001+0000","last_peered":"2024-04-17T10:34:30.111001+0000","last_clean":"2024-04-17T10:34:30.111001+0000","last_became_active":"2024-04-17T10:34:27.954907+0000","last_became_peered":"2024-04-17T10:34:27.954907+0000","last_unstale":"2024-04-17T10:34:30.111001+0000","last_undegraded":"2024-04-17T10:34:30.111001+0000","last_fullsized":"2024-04-17T10:34:30.111001+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:10.656666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107652+0000","last_change":"2024-04-17T10:34:28.969270+0000","last_active":"2024-04-17T10:34:30.107652+0000","last_peered":"2024-04-17T10:34:30.107652+0000","last_clean":"2024-04-17T10:34:30.107652+0000","last_became_active":"2024-04-17T10:34:28.968928+0000","last_became_peered":"2024-04-17T10:34:28.968928+0000","last_unstale":"2024-04-17T10:34:30.107652+0000","last_undegraded":"2024-04-17T10:34:30.107652+0000","last_fullsized":"2024-04-17T10:34:30.107652+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:45.570663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967544+0000","last_change":"2024-04-17T10:34:27.949187+0000","last_active":"2024-04-17T10:34:29.967544+0000","last_peered":"2024-04-17T10:34:29.967544+0000","last_clean":"2024-04-17T10:34:29.967544+0000","last_became_active":"2024-04-17T10:34:27.948931+0000","last_became_peered":"2024-04-17T10:34:27.948931+0000","last_unstale":"2024-04-17T10:34:29.967544+0000","last_undegraded":"2024-04-17T10:34:29.967544+0000","last_fullsized":"2024-04-17T10:34:29.967544+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:44.563427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108622+0000","last_change":"2024-04-17T10:34:28.985318+0000","last_active":"2024-04-17T10:34:30.108622+0000","last_peered":"2024-04-17T10:34:30.108622+0000","last_clean":"2024-04-17T10:34:30.108622+0000","last_became_active":"2024-04-17T10:34:28.983874+0000","last_became_peered":"2024-04-17T10:34:28.983874+0000","last_unstale":"2024-04-17T10:34:30.108622+0000","last_undegraded":"2024-04-17T10:34:30.108622+0000","last_fullsized":"2024-04-17T10:34:30.108622+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:58.674364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967239+0000","last_change":"2024-04-17T10:34:27.949100+0000","last_active":"2024-04-17T10:34:29.967239+0000","last_peered":"2024-04-17T10:34:29.967239+0000","last_clean":"2024-04-17T10:34:29.967239+0000","last_became_active":"2024-04-17T10:34:27.948265+0000","last_became_peered":"2024-04-17T10:34:27.948265+0000","last_unstale":"2024-04-17T10:34:29.967239+0000","last_undegraded":"2024-04-17T10:34:29.967239+0000","last_fullsized":"2024-04-17T10:34:29.967239+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:30.495177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110479+0000","last_change":"2024-04-17T10:34:28.987298+0000","last_active":"2024-04-17T10:34:30.110479+0000","last_peered":"2024-04-17T10:34:30.110479+0000","last_clean":"2024-04-17T10:34:30.110479+0000","last_became_active":"2024-04-17T10:34:28.986389+0000","last_became_peered":"2024-04-17T10:34:28.986389+0000","last_unstale":"2024-04-17T10:34:30.110479+0000","last_undegraded":"2024-04-17T10:34:30.110479+0000","last_fullsized":"2024-04-17T10:34:30.110479+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:53.892216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967946+0000","last_change":"2024-04-17T10:34:27.950978+0000","last_active":"2024-04-17T10:34:29.967946+0000","last_peered":"2024-04-17T10:34:29.967946+0000","last_clean":"2024-04-17T10:34:29.967946+0000","last_became_active":"2024-04-17T10:34:27.950693+0000","last_became_peered":"2024-04-17T10:34:27.950693+0000","last_unstale":"2024-04-17T10:34:29.967946+0000","last_undegraded":"2024-04-17T10:34:29.967946+0000","last_fullsized":"2024-04-17T10:34:29.967946+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:11.995752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968156+0000","last_change":"2024-04-17T10:34:28.965007+0000","last_active":"2024-04-17T10:34:29.968156+0000","last_peered":"2024-04-17T10:34:29.968156+0000","last_clean":"2024-04-17T10:34:29.968156+0000","last_became_active":"2024-04-17T10:34:28.964702+0000","last_became_peered":"2024-04-17T10:34:28.964702+0000","last_unstale":"2024-04-17T10:34:29.968156+0000","last_undegraded":"2024-04-17T10:34:29.968156+0000","last_fullsized":"2024-04-17T10:34:29.968156+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:55:00.945480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966912+0000","last_change":"2024-04-17T10:34:27.959806+0000","last_active":"2024-04-17T10:34:29.966912+0000","last_peered":"2024-04-17T10:34:29.966912+0000","last_clean":"2024-04-17T10:34:29.966912+0000","last_became_active":"2024-04-17T10:34:27.959486+0000","last_became_peered":"2024-04-17T10:34:27.959486+0000","last_unstale":"2024-04-17T10:34:29.966912+0000","last_undegraded":"2024-04-17T10:34:29.966912+0000","last_fullsized":"2024-04-17T10:34:29.966912+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:00.531443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968111+0000","last_change":"2024-04-17T10:34:28.969624+0000","last_active":"2024-04-17T10:34:29.968111+0000","last_peered":"2024-04-17T10:34:29.968111+0000","last_clean":"2024-04-17T10:34:29.968111+0000","last_became_active":"2024-04-17T10:34:28.969314+0000","last_became_peered":"2024-04-17T10:34:28.969314+0000","last_unstale":"2024-04-17T10:34:29.968111+0000","last_undegraded":"2024-04-17T10:34:29.968111+0000","last_fullsized":"2024-04-17T10:34:29.968111+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:56.886865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108694+0000","last_change":"2024-04-17T10:34:27.946338+0000","last_active":"2024-04-17T10:34:30.108694+0000","last_peered":"2024-04-17T10:34:30.108694+0000","last_clean":"2024-04-17T10:34:30.108694+0000","last_became_active":"2024-04-17T10:34:27.945265+0000","last_became_peered":"2024-04-17T10:34:27.945265+0000","last_unstale":"2024-04-17T10:34:30.108694+0000","last_undegraded":"2024-04-17T10:34:30.108694+0000","last_fullsized":"2024-04-17T10:34:30.108694+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.013210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968125+0000","last_change":"2024-04-17T10:34:28.989082+0000","last_active":"2024-04-17T10:34:29.968125+0000","last_peered":"2024-04-17T10:34:29.968125+0000","last_clean":"2024-04-17T10:34:29.968125+0000","last_became_active":"2024-04-17T10:34:28.988817+0000","last_became_peered":"2024-04-17T10:34:28.988817+0000","last_unstale":"2024-04-17T10:34:29.968125+0000","last_undegraded":"2024-04-17T10:34:29.968125+0000","last_fullsized":"2024-04-17T10:34:29.968125+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:52.718288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967552+0000","last_change":"2024-04-17T10:34:27.951045+0000","last_active":"2024-04-17T10:34:29.967552+0000","last_peered":"2024-04-17T10:34:29.967552+0000","last_clean":"2024-04-17T10:34:29.967552+0000","last_became_active":"2024-04-17T10:34:27.950758+0000","last_became_peered":"2024-04-17T10:34:27.950758+0000","last_unstale":"2024-04-17T10:34:29.967552+0000","last_undegraded":"2024-04-17T10:34:29.967552+0000","last_fullsized":"2024-04-17T10:34:29.967552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:30.495177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109018+0000","last_change":"2024-04-17T10:34:28.984105+0000","last_active":"2024-04-17T10:34:30.109018+0000","last_peered":"2024-04-17T10:34:30.109018+0000","last_clean":"2024-04-17T10:34:30.109018+0000","last_became_active":"2024-04-17T10:34:28.983649+0000","last_became_peered":"2024-04-17T10:34:28.983649+0000","last_unstale":"2024-04-17T10:34:30.109018+0000","last_undegraded":"2024-04-17T10:34:30.109018+0000","last_fullsized":"2024-04-17T10:34:30.109018+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:42.467741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110922+0000","last_change":"2024-04-17T10:34:27.949819+0000","last_active":"2024-04-17T10:34:30.110922+0000","last_peered":"2024-04-17T10:34:30.110922+0000","last_clean":"2024-04-17T10:34:30.110922+0000","last_became_active":"2024-04-17T10:34:27.949452+0000","last_became_peered":"2024-04-17T10:34:27.949452+0000","last_unstale":"2024-04-17T10:34:30.110922+0000","last_undegraded":"2024-04-17T10:34:30.110922+0000","last_fullsized":"2024-04-17T10:34:30.110922+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.013210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108445+0000","last_change":"2024-04-17T10:34:28.975707+0000","last_active":"2024-04-17T10:34:30.108445+0000","last_peered":"2024-04-17T10:34:30.108445+0000","last_clean":"2024-04-17T10:34:30.108445+0000","last_became_active":"2024-04-17T10:34:28.975096+0000","last_became_peered":"2024-04-17T10:34:28.975096+0000","last_unstale":"2024-04-17T10:34:30.108445+0000","last_undegraded":"2024-04-17T10:34:30.108445+0000","last_fullsized":"2024-04-17T10:34:30.108445+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:01:34.835074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108783+0000","last_change":"2024-04-17T10:34:27.955786+0000","last_active":"2024-04-17T10:34:30.108783+0000","last_peered":"2024-04-17T10:34:30.108783+0000","last_clean":"2024-04-17T10:34:30.108783+0000","last_became_active":"2024-04-17T10:34:27.955527+0000","last_became_peered":"2024-04-17T10:34:27.955527+0000","last_unstale":"2024-04-17T10:34:30.108783+0000","last_undegraded":"2024-04-17T10:34:30.108783+0000","last_fullsized":"2024-04-17T10:34:30.108783+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:46.011255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":371,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107342+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:34:30.107342+0000","last_peered":"2024-04-17T10:34:30.107342+0000","last_clean":"2024-04-17T10:34:30.107342+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:34:30.107342+0000","last_undegraded":"2024-04-17T10:34:30.107342+0000","last_fullsized":"2024-04-17T10:34:30.107342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574970,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51416,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320424,"statfs":{"total":96636764160,"available":96584114176,"internally_reserved":0,"allocated":2236416,"data_stored":1352393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574968,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54984,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316856,"statfs":{"total":96636764160,"available":96580460544,"internally_reserved":0,"allocated":1634304,"data_stored":760995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574969,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50828,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321012,"statfs":{"total":96636764160,"available":96584716288,"internally_reserved":0,"allocated":1630208,"data_stored":760345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574969,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59144,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312696,"statfs":{"total":96636764160,"available":96576200704,"internally_reserved":0,"allocated":1626112,"data_stored":760323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574969,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59152,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312688,"statfs":{"total":96636764160,"available":96576192512,"internally_reserved":0,"allocated":1642496,"data_stored":762005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574970,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54988,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316852,"statfs":{"total":96636764160,"available":96580456448,"internally_reserved":0,"allocated":1634304,"data_stored":760939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50820,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321020,"statfs":{"total":96636764160,"available":96584724480,"internally_reserved":0,"allocated":1630208,"data_stored":760393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574969,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59724,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312116,"statfs":{"total":96636764160,"available":96575606784,"internally_reserved":0,"allocated":2228224,"data_stored":1350747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:34:33.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:34:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c78e00640 1 Processor -- start 2024-04-17T10:34:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c78e00640 1 -- start start 2024-04-17T10:34:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c78e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c74099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c78e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6c74099f90 0x7f6c74161dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c78e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740d0760 con 0x7f6c740b2570 2024-04-17T10:34:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c78e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7409a400 con 0x7f6c74099f90 2024-04-17T10:34:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c78e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740cfc70 con 0x7f6c74161160 2024-04-17T10:34:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c72a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6c740b2570 0x7f6c741616c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37772/0 (socket says 172.21.15.67:37772) 2024-04-17T10:34:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c72a00640 1 -- 172.21.15.67:0/3811426352 learned_addr learned my addr 172.21.15.67:0/3811426352 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c73400640 1 --2- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c74099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c73e00640 1 --2- 172.21.15.67:0/3811426352 >> v2:172.21.15.86:3300/0 conn(0x7f6c74099f90 0x7f6c74161dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c72000640 1 -- 172.21.15.67:0/3811426352 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586006505 0 0) 0x7f6c740d0760 con 0x7f6c740b2570 2024-04-17T10:34:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c72000640 1 -- 172.21.15.67:0/3811426352 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c58003530 con 0x7f6c740b2570 2024-04-17T10:34:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c73e00640 1 -- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 msgr2=0x7f6c74099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c73e00640 1 --2- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c74099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c73e00640 1 -- 172.21.15.67:0/3811426352 >> v1:172.21.15.67:6789/0 conn(0x7f6c740b2570 legacy=0x7f6c741616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.261+0000 7f6c73e00640 1 -- 172.21.15.67:0/3811426352 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740cc2a0 con 0x7f6c74099f90 2024-04-17T10:34:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c73e00640 1 --2- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c74161dd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009d90 tx=0x7f6c6c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c62cefcc47b610bc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c72000640 1 -- 172.21.15.67:0/3811426352 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0b2070 con 0x7f6c74099f90 2024-04-17T10:34:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c72000640 1 -- 172.21.15.67:0/3811426352 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c0a5940 con 0x7f6c74099f90 2024-04-17T10:34:33.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c72000640 1 -- 172.21.15.67:0/3811426352 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c0a5d10 con 0x7f6c74099f90 2024-04-17T10:34:33.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 -- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 msgr2=0x7f6c74161dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c74161dd0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009d90 tx=0x7f6c6c01c970 comp rx=0 tx=0).stop 2024-04-17T10:34:33.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 -- 172.21.15.67:0/3811426352 shutdown_connections 2024-04-17T10:34:33.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c74161dd0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3811426352 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c74099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 -- 172.21.15.67:0/3811426352 >> 172.21.15.67:0/3811426352 conn(0x7f6c740d1690 msgr2=0x7f6c740d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:33.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 -- 172.21.15.67:0/3811426352 shutdown_connections 2024-04-17T10:34:33.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.265+0000 7f6c78e00640 1 -- 172.21.15.67:0/3811426352 wait complete. 2024-04-17T10:34:33.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 Processor -- start 2024-04-17T10:34:33.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 -- start start 2024-04-17T10:34:33.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c7414e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c740b2570 0x7f6c740b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c7414db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740b2bc0 con 0x7f6c740b2570 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740cc5d0 con 0x7f6c74099f90 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7406bda0 con 0x7f6c74161160 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c740b2570 0x7f6c740b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c740b2570 0x7f6c740b6530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48054/0 (socket says 172.21.15.67:48054) 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 -- 172.21.15.67:0/3096946576 learned_addr learned my addr 172.21.15.67:0/3096946576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c73400640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c7414e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c73e00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c7414db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 -- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 msgr2=0x7f6c7414db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c7414db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 -- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 msgr2=0x7f6c7414e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c7414e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 -- 172.21.15.67:0/3096946576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7414efa0 con 0x7f6c740b2570 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c73400640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c7414e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c72a00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c740b2570 0x7f6c740b6530 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000c3c0 tx=0x7f6c6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c6b400640 1 -- 172.21.15.67:0/3096946576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c60017070 con 0x7f6c740b2570 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c6b400640 1 -- 172.21.15.67:0/3096946576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c60005cf0 con 0x7f6c740b2570 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c6b400640 1 -- 172.21.15.67:0/3096946576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c600072f0 con 0x7f6c740b2570 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7414f280 con 0x7f6c740b2570 2024-04-17T10:34:33.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7414f840 con 0x7f6c740b2570 2024-04-17T10:34:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.269+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c74074dc0 con 0x7f6c740b2570 2024-04-17T10:34:33.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.273+0000 7f6c6b400640 1 -- 172.21.15.67:0/3096946576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6c6001a070 con 0x7f6c740b2570 2024-04-17T10:34:33.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.273+0000 7f6c6b400640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c54066f60 0x7f6c54069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.273+0000 7f6c6b400640 1 -- 172.21.15.67:0/3096946576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6c60084600 con 0x7f6c740b2570 2024-04-17T10:34:33.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.273+0000 7f6c73400640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c54066f60 0x7f6c54069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.273+0000 7f6c73400640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c54066f60 0x7f6c54069420 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009f50 tx=0x7f6c64009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:33.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.277+0000 7f6c6b400640 1 -- 172.21.15.67:0/3096946576 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6c60054d00 con 0x7f6c740b2570 2024-04-17T10:34:33.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.437+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6c740cfa50 con 0x7f6c54066f60 2024-04-17T10:34:33.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.445+0000 7f6c6b400640 1 -- 172.21.15.67:0/3096946576 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252486 (secure 0 0 0) 0x7f6c740cfa50 con 0x7f6c54066f60 2024-04-17T10:34:33.438 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:33.440 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c54066f60 msgr2=0x7f6c54069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c54066f60 0x7f6c54069420 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009f50 tx=0x7f6c64009190 comp rx=0 tx=0).stop 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c740b2570 msgr2=0x7f6c740b6530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c740b2570 0x7f6c740b6530 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000c3c0 tx=0x7f6c6000c880 comp rx=0 tx=0).stop 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 shutdown_connections 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6c54066f60 0x7f6c54069420 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6c74161160 0x7f6c7414db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6c740b2570 0x7f6c740b6530 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 --2- 172.21.15.67:0/3096946576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6c74099f90 0x7f6c7414e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 >> 172.21.15.67:0/3096946576 conn(0x7f6c740d1690 msgr2=0x7f6c7406c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:33.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 shutdown_connections 2024-04-17T10:34:33.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.449+0000 7f6c78e00640 1 -- 172.21.15.67:0/3096946576 wait complete. 2024-04-17T10:34:33.460 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-04-17T10:34:32.273749+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":441056,"kb_used_data":13928,"kb_used_omap":12,"kb_used_meta":426995,"kb_avail":754533664,"statfs":{"total":773094113280,"available":772642471936,"internally_reserved":0,"allocated":14262272,"data_stored":7268140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":437243441},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.347648"},"pg_stats":[{"pgid":"20.3f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110835+0000","last_change":"2024-04-17T10:34:28.974594+0000","last_active":"2024-04-17T10:34:30.110835+0000","last_peered":"2024-04-17T10:34:30.110835+0000","last_clean":"2024-04-17T10:34:30.110835+0000","last_became_active":"2024-04-17T10:34:28.974307+0000","last_became_peered":"2024-04-17T10:34:28.974307+0000","last_unstale":"2024-04-17T10:34:30.110835+0000","last_undegraded":"2024-04-17T10:34:30.110835+0000","last_fullsized":"2024-04-17T10:34:30.110835+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:01:00.478597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966754+0000","last_change":"2024-04-17T10:34:28.958804+0000","last_active":"2024-04-17T10:34:29.966754+0000","last_peered":"2024-04-17T10:34:29.966754+0000","last_clean":"2024-04-17T10:34:29.966754+0000","last_became_active":"2024-04-17T10:34:28.958536+0000","last_became_peered":"2024-04-17T10:34:28.958536+0000","last_unstale":"2024-04-17T10:34:29.966754+0000","last_undegraded":"2024-04-17T10:34:29.966754+0000","last_fullsized":"2024-04-17T10:34:29.966754+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:47.018491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108277+0000","last_change":"2024-04-17T10:34:28.975736+0000","last_active":"2024-04-17T10:34:30.108277+0000","last_peered":"2024-04-17T10:34:30.108277+0000","last_clean":"2024-04-17T10:34:30.108277+0000","last_became_active":"2024-04-17T10:34:28.975409+0000","last_became_peered":"2024-04-17T10:34:28.975409+0000","last_unstale":"2024-04-17T10:34:30.108277+0000","last_undegraded":"2024-04-17T10:34:30.108277+0000","last_fullsized":"2024-04-17T10:34:30.108277+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:44:14.364854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.3c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966778+0000","last_change":"2024-04-17T10:34:28.970318+0000","last_active":"2024-04-17T10:34:29.966778+0000","last_peered":"2024-04-17T10:34:29.966778+0000","last_clean":"2024-04-17T10:34:29.966778+0000","last_became_active":"2024-04-17T10:34:28.970019+0000","last_became_peered":"2024-04-17T10:34:28.970019+0000","last_unstale":"2024-04-17T10:34:29.966778+0000","last_undegraded":"2024-04-17T10:34:29.966778+0000","last_fullsized":"2024-04-17T10:34:29.966778+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:25.024758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.3b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111024+0000","last_change":"2024-04-17T10:34:28.980623+0000","last_active":"2024-04-17T10:34:30.111024+0000","last_peered":"2024-04-17T10:34:30.111024+0000","last_clean":"2024-04-17T10:34:30.111024+0000","last_became_active":"2024-04-17T10:34:28.980140+0000","last_became_peered":"2024-04-17T10:34:28.980140+0000","last_unstale":"2024-04-17T10:34:30.111024+0000","last_undegraded":"2024-04-17T10:34:30.111024+0000","last_fullsized":"2024-04-17T10:34:30.111024+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:38:03.361676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.3a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967753+0000","last_change":"2024-04-17T10:34:28.982156+0000","last_active":"2024-04-17T10:34:29.967753+0000","last_peered":"2024-04-17T10:34:29.967753+0000","last_clean":"2024-04-17T10:34:29.967753+0000","last_became_active":"2024-04-17T10:34:28.981840+0000","last_became_peered":"2024-04-17T10:34:28.981840+0000","last_unstale":"2024-04-17T10:34:29.967753+0000","last_undegraded":"2024-04-17T10:34:29.967753+0000","last_fullsized":"2024-04-17T10:34:29.967753+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:52:33.443118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.39","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108458+0000","last_change":"2024-04-17T10:34:28.987629+0000","last_active":"2024-04-17T10:34:30.108458+0000","last_peered":"2024-04-17T10:34:30.108458+0000","last_clean":"2024-04-17T10:34:30.108458+0000","last_became_active":"2024-04-17T10:34:28.986810+0000","last_became_peered":"2024-04-17T10:34:28.986810+0000","last_unstale":"2024-04-17T10:34:30.108458+0000","last_undegraded":"2024-04-17T10:34:30.108458+0000","last_fullsized":"2024-04-17T10:34:30.108458+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:04.516924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.38","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.965885+0000","last_change":"2024-04-17T10:34:28.964128+0000","last_active":"2024-04-17T10:34:29.965885+0000","last_peered":"2024-04-17T10:34:29.965885+0000","last_clean":"2024-04-17T10:34:29.965885+0000","last_became_active":"2024-04-17T10:34:28.963815+0000","last_became_peered":"2024-04-17T10:34:28.963815+0000","last_unstale":"2024-04-17T10:34:29.965885+0000","last_undegraded":"2024-04-17T10:34:29.965885+0000","last_fullsized":"2024-04-17T10:34:29.965885+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:52.397412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.37","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968269+0000","last_change":"2024-04-17T10:34:28.966082+0000","last_active":"2024-04-17T10:34:29.968269+0000","last_peered":"2024-04-17T10:34:29.968269+0000","last_clean":"2024-04-17T10:34:29.968269+0000","last_became_active":"2024-04-17T10:34:28.965745+0000","last_became_peered":"2024-04-17T10:34:28.965745+0000","last_unstale":"2024-04-17T10:34:29.968269+0000","last_undegraded":"2024-04-17T10:34:29.968269+0000","last_fullsized":"2024-04-17T10:34:29.968269+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:58.020446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.36","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110170+0000","last_change":"2024-04-17T10:34:28.979428+0000","last_active":"2024-04-17T10:34:30.110170+0000","last_peered":"2024-04-17T10:34:30.110170+0000","last_clean":"2024-04-17T10:34:30.110170+0000","last_became_active":"2024-04-17T10:34:28.979012+0000","last_became_peered":"2024-04-17T10:34:28.979012+0000","last_unstale":"2024-04-17T10:34:30.110170+0000","last_undegraded":"2024-04-17T10:34:30.110170+0000","last_fullsized":"2024-04-17T10:34:30.110170+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:25:24.731406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.35","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966635+0000","last_change":"2024-04-17T10:34:28.978199+0000","last_active":"2024-04-17T10:34:29.966635+0000","last_peered":"2024-04-17T10:34:29.966635+0000","last_clean":"2024-04-17T10:34:29.966635+0000","last_became_active":"2024-04-17T10:34:28.977904+0000","last_became_peered":"2024-04-17T10:34:28.977904+0000","last_unstale":"2024-04-17T10:34:29.966635+0000","last_undegraded":"2024-04-17T10:34:29.966635+0000","last_fullsized":"2024-04-17T10:34:29.966635+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:58.020446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.34","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110724+0000","last_change":"2024-04-17T10:34:28.975443+0000","last_active":"2024-04-17T10:34:30.110724+0000","last_peered":"2024-04-17T10:34:30.110724+0000","last_clean":"2024-04-17T10:34:30.110724+0000","last_became_active":"2024-04-17T10:34:28.973158+0000","last_became_peered":"2024-04-17T10:34:28.973158+0000","last_unstale":"2024-04-17T10:34:30.110724+0000","last_undegraded":"2024-04-17T10:34:30.110724+0000","last_fullsized":"2024-04-17T10:34:30.110724+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:20:15.691481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.33","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967899+0000","last_change":"2024-04-17T10:34:28.985762+0000","last_active":"2024-04-17T10:34:29.967899+0000","last_peered":"2024-04-17T10:34:29.967899+0000","last_clean":"2024-04-17T10:34:29.967899+0000","last_became_active":"2024-04-17T10:34:28.985474+0000","last_became_peered":"2024-04-17T10:34:28.985474+0000","last_unstale":"2024-04-17T10:34:29.967899+0000","last_undegraded":"2024-04-17T10:34:29.967899+0000","last_fullsized":"2024-04-17T10:34:29.967899+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:52.195084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.32","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110165+0000","last_change":"2024-04-17T10:34:28.987434+0000","last_active":"2024-04-17T10:34:30.110165+0000","last_peered":"2024-04-17T10:34:30.110165+0000","last_clean":"2024-04-17T10:34:30.110165+0000","last_became_active":"2024-04-17T10:34:28.987125+0000","last_became_peered":"2024-04-17T10:34:28.987125+0000","last_unstale":"2024-04-17T10:34:30.110165+0000","last_undegraded":"2024-04-17T10:34:30.110165+0000","last_fullsized":"2024-04-17T10:34:30.110165+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:05:11.076299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.31","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109797+0000","last_change":"2024-04-17T10:34:28.986015+0000","last_active":"2024-04-17T10:34:30.109797+0000","last_peered":"2024-04-17T10:34:30.109797+0000","last_clean":"2024-04-17T10:34:30.109797+0000","last_became_active":"2024-04-17T10:34:28.985713+0000","last_became_peered":"2024-04-17T10:34:28.985713+0000","last_unstale":"2024-04-17T10:34:30.109797+0000","last_undegraded":"2024-04-17T10:34:30.109797+0000","last_fullsized":"2024-04-17T10:34:30.109797+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:46:05.775616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.30","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967522+0000","last_change":"2024-04-17T10:34:28.973205+0000","last_active":"2024-04-17T10:34:29.967522+0000","last_peered":"2024-04-17T10:34:29.967522+0000","last_clean":"2024-04-17T10:34:29.967522+0000","last_became_active":"2024-04-17T10:34:28.972667+0000","last_became_peered":"2024-04-17T10:34:28.972667+0000","last_unstale":"2024-04-17T10:34:29.967522+0000","last_undegraded":"2024-04-17T10:34:29.967522+0000","last_fullsized":"2024-04-17T10:34:29.967522+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:33.384275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968330+0000","last_change":"2024-04-17T10:34:28.970943+0000","last_active":"2024-04-17T10:34:29.968330+0000","last_peered":"2024-04-17T10:34:29.968330+0000","last_clean":"2024-04-17T10:34:29.968330+0000","last_became_active":"2024-04-17T10:34:28.970652+0000","last_became_peered":"2024-04-17T10:34:28.970652+0000","last_unstale":"2024-04-17T10:34:29.968330+0000","last_undegraded":"2024-04-17T10:34:29.968330+0000","last_fullsized":"2024-04-17T10:34:29.968330+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:47.018491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.2e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111129+0000","last_change":"2024-04-17T10:34:28.977580+0000","last_active":"2024-04-17T10:34:30.111129+0000","last_peered":"2024-04-17T10:34:30.111129+0000","last_clean":"2024-04-17T10:34:30.111129+0000","last_became_active":"2024-04-17T10:34:28.977270+0000","last_became_peered":"2024-04-17T10:34:28.977270+0000","last_unstale":"2024-04-17T10:34:30.111129+0000","last_undegraded":"2024-04-17T10:34:30.111129+0000","last_fullsized":"2024-04-17T10:34:30.111129+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:18:44.864368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108020+0000","last_change":"2024-04-17T10:34:28.986292+0000","last_active":"2024-04-17T10:34:30.108020+0000","last_peered":"2024-04-17T10:34:30.108020+0000","last_clean":"2024-04-17T10:34:30.108020+0000","last_became_active":"2024-04-17T10:34:28.985852+0000","last_became_peered":"2024-04-17T10:34:28.985852+0000","last_unstale":"2024-04-17T10:34:30.108020+0000","last_undegraded":"2024-04-17T10:34:30.108020+0000","last_fullsized":"2024-04-17T10:34:30.108020+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:21.004154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.2c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111268+0000","last_change":"2024-04-17T10:34:28.991008+0000","last_active":"2024-04-17T10:34:30.111268+0000","last_peered":"2024-04-17T10:34:30.111268+0000","last_clean":"2024-04-17T10:34:30.111268+0000","last_became_active":"2024-04-17T10:34:28.990696+0000","last_became_peered":"2024-04-17T10:34:28.990696+0000","last_unstale":"2024-04-17T10:34:30.111268+0000","last_undegraded":"2024-04-17T10:34:30.111268+0000","last_fullsized":"2024-04-17T10:34:30.111268+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:51:30.067543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.2b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966870+0000","last_change":"2024-04-17T10:34:28.961096+0000","last_active":"2024-04-17T10:34:29.966870+0000","last_peered":"2024-04-17T10:34:29.966870+0000","last_clean":"2024-04-17T10:34:29.966870+0000","last_became_active":"2024-04-17T10:34:28.960767+0000","last_became_peered":"2024-04-17T10:34:28.960767+0000","last_unstale":"2024-04-17T10:34:29.966870+0000","last_undegraded":"2024-04-17T10:34:29.966870+0000","last_fullsized":"2024-04-17T10:34:29.966870+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:52:16.087256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110026+0000","last_change":"2024-04-17T10:34:28.962758+0000","last_active":"2024-04-17T10:34:30.110026+0000","last_peered":"2024-04-17T10:34:30.110026+0000","last_clean":"2024-04-17T10:34:30.110026+0000","last_became_active":"2024-04-17T10:34:28.962088+0000","last_became_peered":"2024-04-17T10:34:28.962088+0000","last_unstale":"2024-04-17T10:34:30.110026+0000","last_undegraded":"2024-04-17T10:34:30.110026+0000","last_fullsized":"2024-04-17T10:34:30.110026+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:05:29.753156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.29","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966161+0000","last_change":"2024-04-17T10:34:28.982852+0000","last_active":"2024-04-17T10:34:29.966161+0000","last_peered":"2024-04-17T10:34:29.966161+0000","last_clean":"2024-04-17T10:34:29.966161+0000","last_became_active":"2024-04-17T10:34:28.982223+0000","last_became_peered":"2024-04-17T10:34:28.982223+0000","last_unstale":"2024-04-17T10:34:29.966161+0000","last_undegraded":"2024-04-17T10:34:29.966161+0000","last_fullsized":"2024-04-17T10:34:29.966161+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:52.718288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.28","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967565+0000","last_change":"2024-04-17T10:34:28.966086+0000","last_active":"2024-04-17T10:34:29.967565+0000","last_peered":"2024-04-17T10:34:29.967565+0000","last_clean":"2024-04-17T10:34:29.967565+0000","last_became_active":"2024-04-17T10:34:28.965757+0000","last_became_peered":"2024-04-17T10:34:28.965757+0000","last_unstale":"2024-04-17T10:34:29.967565+0000","last_undegraded":"2024-04-17T10:34:29.967565+0000","last_fullsized":"2024-04-17T10:34:29.967565+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:23.954687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.27","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108664+0000","last_change":"2024-04-17T10:34:28.975208+0000","last_active":"2024-04-17T10:34:30.108664+0000","last_peered":"2024-04-17T10:34:30.108664+0000","last_clean":"2024-04-17T10:34:30.108664+0000","last_became_active":"2024-04-17T10:34:28.974913+0000","last_became_peered":"2024-04-17T10:34:28.974913+0000","last_unstale":"2024-04-17T10:34:30.108664+0000","last_undegraded":"2024-04-17T10:34:30.108664+0000","last_fullsized":"2024-04-17T10:34:30.108664+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:01.538678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.26","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108466+0000","last_change":"2024-04-17T10:34:28.984601+0000","last_active":"2024-04-17T10:34:30.108466+0000","last_peered":"2024-04-17T10:34:30.108466+0000","last_clean":"2024-04-17T10:34:30.108466+0000","last_became_active":"2024-04-17T10:34:28.984350+0000","last_became_peered":"2024-04-17T10:34:28.984350+0000","last_unstale":"2024-04-17T10:34:30.108466+0000","last_undegraded":"2024-04-17T10:34:30.108466+0000","last_fullsized":"2024-04-17T10:34:30.108466+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:52.397412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.25","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108176+0000","last_change":"2024-04-17T10:34:28.991515+0000","last_active":"2024-04-17T10:34:30.108176+0000","last_peered":"2024-04-17T10:34:30.108176+0000","last_clean":"2024-04-17T10:34:30.108176+0000","last_became_active":"2024-04-17T10:34:28.991177+0000","last_became_peered":"2024-04-17T10:34:28.991177+0000","last_unstale":"2024-04-17T10:34:30.108176+0000","last_undegraded":"2024-04-17T10:34:30.108176+0000","last_fullsized":"2024-04-17T10:34:30.108176+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:58:51.767495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.24","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110114+0000","last_change":"2024-04-17T10:34:28.979151+0000","last_active":"2024-04-17T10:34:30.110114+0000","last_peered":"2024-04-17T10:34:30.110114+0000","last_clean":"2024-04-17T10:34:30.110114+0000","last_became_active":"2024-04-17T10:34:28.978652+0000","last_became_peered":"2024-04-17T10:34:28.978652+0000","last_unstale":"2024-04-17T10:34:30.110114+0000","last_undegraded":"2024-04-17T10:34:30.110114+0000","last_fullsized":"2024-04-17T10:34:30.110114+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:52:55.987893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.23","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107845+0000","last_change":"2024-04-17T10:34:28.976197+0000","last_active":"2024-04-17T10:34:30.107845+0000","last_peered":"2024-04-17T10:34:30.107845+0000","last_clean":"2024-04-17T10:34:30.107845+0000","last_became_active":"2024-04-17T10:34:28.975904+0000","last_became_peered":"2024-04-17T10:34:28.975904+0000","last_unstale":"2024-04-17T10:34:30.107845+0000","last_undegraded":"2024-04-17T10:34:30.107845+0000","last_fullsized":"2024-04-17T10:34:30.107845+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:41:49.628136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.22","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107628+0000","last_change":"2024-04-17T10:34:28.974772+0000","last_active":"2024-04-17T10:34:30.107628+0000","last_peered":"2024-04-17T10:34:30.107628+0000","last_clean":"2024-04-17T10:34:30.107628+0000","last_became_active":"2024-04-17T10:34:28.974450+0000","last_became_peered":"2024-04-17T10:34:28.974450+0000","last_unstale":"2024-04-17T10:34:30.107628+0000","last_undegraded":"2024-04-17T10:34:30.107628+0000","last_fullsized":"2024-04-17T10:34:30.107628+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:03:00.766283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.21","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.965895+0000","last_change":"2024-04-17T10:34:28.968028+0000","last_active":"2024-04-17T10:34:29.965895+0000","last_peered":"2024-04-17T10:34:29.965895+0000","last_clean":"2024-04-17T10:34:29.965895+0000","last_became_active":"2024-04-17T10:34:28.967736+0000","last_became_peered":"2024-04-17T10:34:28.967736+0000","last_unstale":"2024-04-17T10:34:29.965895+0000","last_undegraded":"2024-04-17T10:34:29.965895+0000","last_fullsized":"2024-04-17T10:34:29.965895+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:01.538678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.20","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109893+0000","last_change":"2024-04-17T10:34:28.977742+0000","last_active":"2024-04-17T10:34:30.109893+0000","last_peered":"2024-04-17T10:34:30.109893+0000","last_clean":"2024-04-17T10:34:30.109893+0000","last_became_active":"2024-04-17T10:34:28.977285+0000","last_became_peered":"2024-04-17T10:34:28.977285+0000","last_unstale":"2024-04-17T10:34:30.109893+0000","last_undegraded":"2024-04-17T10:34:30.109893+0000","last_fullsized":"2024-04-17T10:34:30.109893+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:07:35.778397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966058+0000","last_change":"2024-04-17T10:34:28.983315+0000","last_active":"2024-04-17T10:34:29.966058+0000","last_peered":"2024-04-17T10:34:29.966058+0000","last_clean":"2024-04-17T10:34:29.966058+0000","last_became_active":"2024-04-17T10:34:28.982983+0000","last_became_peered":"2024-04-17T10:34:28.982983+0000","last_unstale":"2024-04-17T10:34:29.966058+0000","last_undegraded":"2024-04-17T10:34:29.966058+0000","last_fullsized":"2024-04-17T10:34:29.966058+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:11.721543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1f","version":"129'4","reported_seq":18,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108582+0000","last_change":"2024-04-17T10:34:27.949473+0000","last_active":"2024-04-17T10:34:30.108582+0000","last_peered":"2024-04-17T10:34:30.108582+0000","last_clean":"2024-04-17T10:34:30.108582+0000","last_became_active":"2024-04-17T10:34:27.949190+0000","last_became_peered":"2024-04-17T10:34:27.949190+0000","last_unstale":"2024-04-17T10:34:30.108582+0000","last_undegraded":"2024-04-17T10:34:30.108582+0000","last_fullsized":"2024-04-17T10:34:30.108582+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:18:28.054550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110083+0000","last_change":"2024-04-17T10:34:28.962284+0000","last_active":"2024-04-17T10:34:30.110083+0000","last_peered":"2024-04-17T10:34:30.110083+0000","last_clean":"2024-04-17T10:34:30.110083+0000","last_became_active":"2024-04-17T10:34:28.961711+0000","last_became_peered":"2024-04-17T10:34:28.961711+0000","last_unstale":"2024-04-17T10:34:30.110083+0000","last_undegraded":"2024-04-17T10:34:30.110083+0000","last_fullsized":"2024-04-17T10:34:30.110083+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:01:20.878852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1e","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111386+0000","last_change":"2024-04-17T10:34:27.953510+0000","last_active":"2024-04-17T10:34:30.111386+0000","last_peered":"2024-04-17T10:34:30.111386+0000","last_clean":"2024-04-17T10:34:30.111386+0000","last_became_active":"2024-04-17T10:34:27.952580+0000","last_became_peered":"2024-04-17T10:34:27.952580+0000","last_unstale":"2024-04-17T10:34:30.111386+0000","last_undegraded":"2024-04-17T10:34:30.111386+0000","last_fullsized":"2024-04-17T10:34:30.111386+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:59.938245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966557+0000","last_change":"2024-04-17T10:34:28.983422+0000","last_active":"2024-04-17T10:34:29.966557+0000","last_peered":"2024-04-17T10:34:29.966557+0000","last_clean":"2024-04-17T10:34:29.966557+0000","last_became_active":"2024-04-17T10:34:28.982854+0000","last_became_peered":"2024-04-17T10:34:28.982854+0000","last_unstale":"2024-04-17T10:34:29.966557+0000","last_undegraded":"2024-04-17T10:34:29.966557+0000","last_fullsized":"2024-04-17T10:34:29.966557+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:55:00.945480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1d","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108272+0000","last_change":"2024-04-17T10:34:27.953236+0000","last_active":"2024-04-17T10:34:30.108272+0000","last_peered":"2024-04-17T10:34:30.108272+0000","last_clean":"2024-04-17T10:34:30.108272+0000","last_became_active":"2024-04-17T10:34:27.952976+0000","last_became_peered":"2024-04-17T10:34:27.952976+0000","last_unstale":"2024-04-17T10:34:30.108272+0000","last_undegraded":"2024-04-17T10:34:30.108272+0000","last_fullsized":"2024-04-17T10:34:30.108272+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:51.711052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110279+0000","last_change":"2024-04-17T10:34:28.987978+0000","last_active":"2024-04-17T10:34:30.110279+0000","last_peered":"2024-04-17T10:34:30.110279+0000","last_clean":"2024-04-17T10:34:30.110279+0000","last_became_active":"2024-04-17T10:34:28.987675+0000","last_became_peered":"2024-04-17T10:34:28.987675+0000","last_unstale":"2024-04-17T10:34:30.110279+0000","last_undegraded":"2024-04-17T10:34:30.110279+0000","last_fullsized":"2024-04-17T10:34:30.110279+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:23:29.314968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.1c","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108157+0000","last_change":"2024-04-17T10:34:27.948168+0000","last_active":"2024-04-17T10:34:30.108157+0000","last_peered":"2024-04-17T10:34:30.108157+0000","last_clean":"2024-04-17T10:34:30.108157+0000","last_became_active":"2024-04-17T10:34:27.947102+0000","last_became_peered":"2024-04-17T10:34:27.947102+0000","last_unstale":"2024-04-17T10:34:30.108157+0000","last_undegraded":"2024-04-17T10:34:30.108157+0000","last_fullsized":"2024-04-17T10:34:30.108157+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:55.879630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967234+0000","last_change":"2024-04-17T10:34:28.968675+0000","last_active":"2024-04-17T10:34:29.967234+0000","last_peered":"2024-04-17T10:34:29.967234+0000","last_clean":"2024-04-17T10:34:29.967234+0000","last_became_active":"2024-04-17T10:34:28.968367+0000","last_became_peered":"2024-04-17T10:34:28.968367+0000","last_unstale":"2024-04-17T10:34:29.967234+0000","last_undegraded":"2024-04-17T10:34:29.967234+0000","last_fullsized":"2024-04-17T10:34:29.967234+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:50:10.822976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1b","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966468+0000","last_change":"2024-04-17T10:34:27.935854+0000","last_active":"2024-04-17T10:34:29.966468+0000","last_peered":"2024-04-17T10:34:29.966468+0000","last_clean":"2024-04-17T10:34:29.966468+0000","last_became_active":"2024-04-17T10:34:27.935517+0000","last_became_peered":"2024-04-17T10:34:27.935517+0000","last_unstale":"2024-04-17T10:34:29.966468+0000","last_undegraded":"2024-04-17T10:34:29.966468+0000","last_fullsized":"2024-04-17T10:34:29.966468+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:59.759048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967329+0000","last_change":"2024-04-17T10:34:28.970218+0000","last_active":"2024-04-17T10:34:29.967329+0000","last_peered":"2024-04-17T10:34:29.967329+0000","last_clean":"2024-04-17T10:34:29.967329+0000","last_became_active":"2024-04-17T10:34:28.969894+0000","last_became_peered":"2024-04-17T10:34:28.969894+0000","last_unstale":"2024-04-17T10:34:29.967329+0000","last_undegraded":"2024-04-17T10:34:29.967329+0000","last_fullsized":"2024-04-17T10:34:29.967329+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:48:33.384275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1a","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110775+0000","last_change":"2024-04-17T10:34:27.938224+0000","last_active":"2024-04-17T10:34:30.110775+0000","last_peered":"2024-04-17T10:34:30.110775+0000","last_clean":"2024-04-17T10:34:30.110775+0000","last_became_active":"2024-04-17T10:34:27.937929+0000","last_became_peered":"2024-04-17T10:34:27.937929+0000","last_unstale":"2024-04-17T10:34:30.110775+0000","last_undegraded":"2024-04-17T10:34:30.110775+0000","last_fullsized":"2024-04-17T10:34:30.110775+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:55.879630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111344+0000","last_change":"2024-04-17T10:34:28.986619+0000","last_active":"2024-04-17T10:34:30.111344+0000","last_peered":"2024-04-17T10:34:30.111344+0000","last_clean":"2024-04-17T10:34:30.111344+0000","last_became_active":"2024-04-17T10:34:28.986332+0000","last_became_peered":"2024-04-17T10:34:28.986332+0000","last_unstale":"2024-04-17T10:34:30.111344+0000","last_undegraded":"2024-04-17T10:34:30.111344+0000","last_fullsized":"2024-04-17T10:34:30.111344+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:18:09.832833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967445+0000","last_change":"2024-04-17T10:34:27.949955+0000","last_active":"2024-04-17T10:34:29.967445+0000","last_peered":"2024-04-17T10:34:29.967445+0000","last_clean":"2024-04-17T10:34:29.967445+0000","last_became_active":"2024-04-17T10:34:27.949460+0000","last_became_peered":"2024-04-17T10:34:27.949460+0000","last_unstale":"2024-04-17T10:34:29.967445+0000","last_undegraded":"2024-04-17T10:34:29.967445+0000","last_fullsized":"2024-04-17T10:34:29.967445+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:48.940362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108886+0000","last_change":"2024-04-17T10:34:28.983387+0000","last_active":"2024-04-17T10:34:30.108886+0000","last_peered":"2024-04-17T10:34:30.108886+0000","last_clean":"2024-04-17T10:34:30.108886+0000","last_became_active":"2024-04-17T10:34:28.982989+0000","last_became_peered":"2024-04-17T10:34:28.982989+0000","last_unstale":"2024-04-17T10:34:30.108886+0000","last_undegraded":"2024-04-17T10:34:30.108886+0000","last_fullsized":"2024-04-17T10:34:30.108886+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:01:29.879975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107421+0000","last_change":"2024-04-17T10:34:27.947754+0000","last_active":"2024-04-17T10:34:30.107421+0000","last_peered":"2024-04-17T10:34:30.107421+0000","last_clean":"2024-04-17T10:34:30.107421+0000","last_became_active":"2024-04-17T10:34:27.947434+0000","last_became_peered":"2024-04-17T10:34:27.947434+0000","last_unstale":"2024-04-17T10:34:30.107421+0000","last_undegraded":"2024-04-17T10:34:30.107421+0000","last_fullsized":"2024-04-17T10:34:30.107421+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:02:35.671346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109784+0000","last_change":"2024-04-17T10:34:28.981842+0000","last_active":"2024-04-17T10:34:30.109784+0000","last_peered":"2024-04-17T10:34:30.109784+0000","last_clean":"2024-04-17T10:34:30.109784+0000","last_became_active":"2024-04-17T10:34:28.981557+0000","last_became_peered":"2024-04-17T10:34:28.981557+0000","last_unstale":"2024-04-17T10:34:30.109784+0000","last_undegraded":"2024-04-17T10:34:30.109784+0000","last_fullsized":"2024-04-17T10:34:30.109784+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:22:02.696728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966724+0000","last_change":"2024-04-17T10:34:27.953065+0000","last_active":"2024-04-17T10:34:29.966724+0000","last_peered":"2024-04-17T10:34:29.966724+0000","last_clean":"2024-04-17T10:34:29.966724+0000","last_became_active":"2024-04-17T10:34:27.952542+0000","last_became_peered":"2024-04-17T10:34:27.952542+0000","last_unstale":"2024-04-17T10:34:29.966724+0000","last_undegraded":"2024-04-17T10:34:29.966724+0000","last_fullsized":"2024-04-17T10:34:29.966724+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:02:51.390177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967636+0000","last_change":"2024-04-17T10:34:28.972014+0000","last_active":"2024-04-17T10:34:29.967636+0000","last_peered":"2024-04-17T10:34:29.967636+0000","last_clean":"2024-04-17T10:34:29.967636+0000","last_became_active":"2024-04-17T10:34:28.971704+0000","last_became_peered":"2024-04-17T10:34:28.971704+0000","last_unstale":"2024-04-17T10:34:29.967636+0000","last_undegraded":"2024-04-17T10:34:29.967636+0000","last_fullsized":"2024-04-17T10:34:29.967636+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:37:12.984255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110677+0000","last_change":"2024-04-17T10:34:27.957469+0000","last_active":"2024-04-17T10:34:30.110677+0000","last_peered":"2024-04-17T10:34:30.110677+0000","last_clean":"2024-04-17T10:34:30.110677+0000","last_became_active":"2024-04-17T10:34:27.957144+0000","last_became_peered":"2024-04-17T10:34:27.957144+0000","last_unstale":"2024-04-17T10:34:30.110677+0000","last_undegraded":"2024-04-17T10:34:30.110677+0000","last_fullsized":"2024-04-17T10:34:30.110677+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:19:02.267626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107720+0000","last_change":"2024-04-17T10:34:28.975429+0000","last_active":"2024-04-17T10:34:30.107720+0000","last_peered":"2024-04-17T10:34:30.107720+0000","last_clean":"2024-04-17T10:34:30.107720+0000","last_became_active":"2024-04-17T10:34:28.975066+0000","last_became_peered":"2024-04-17T10:34:28.975066+0000","last_unstale":"2024-04-17T10:34:30.107720+0000","last_undegraded":"2024-04-17T10:34:30.107720+0000","last_fullsized":"2024-04-17T10:34:30.107720+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:33:52.896649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.15","version":"129'3","reported_seq":17,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108272+0000","last_change":"2024-04-17T10:34:27.948950+0000","last_active":"2024-04-17T10:34:30.108272+0000","last_peered":"2024-04-17T10:34:30.108272+0000","last_clean":"2024-04-17T10:34:30.108272+0000","last_became_active":"2024-04-17T10:34:27.945716+0000","last_became_peered":"2024-04-17T10:34:27.945716+0000","last_unstale":"2024-04-17T10:34:30.108272+0000","last_undegraded":"2024-04-17T10:34:30.108272+0000","last_fullsized":"2024-04-17T10:34:30.108272+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:59.938245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.13","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108063+0000","last_change":"2024-04-17T10:34:28.975574+0000","last_active":"2024-04-17T10:34:30.108063+0000","last_peered":"2024-04-17T10:34:30.108063+0000","last_clean":"2024-04-17T10:34:30.108063+0000","last_became_active":"2024-04-17T10:34:28.975192+0000","last_became_peered":"2024-04-17T10:34:28.975192+0000","last_unstale":"2024-04-17T10:34:30.108063+0000","last_undegraded":"2024-04-17T10:34:30.108063+0000","last_fullsized":"2024-04-17T10:34:30.108063+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:11:11.440956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.14","version":"129'6","reported_seq":20,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108284+0000","last_change":"2024-04-17T10:34:27.947263+0000","last_active":"2024-04-17T10:34:30.108284+0000","last_peered":"2024-04-17T10:34:30.108284+0000","last_clean":"2024-04-17T10:34:30.108284+0000","last_became_active":"2024-04-17T10:34:27.946998+0000","last_became_peered":"2024-04-17T10:34:27.946998+0000","last_unstale":"2024-04-17T10:34:30.108284+0000","last_undegraded":"2024-04-17T10:34:30.108284+0000","last_fullsized":"2024-04-17T10:34:30.108284+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:55:24.017523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.14","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110252+0000","last_change":"2024-04-17T10:34:28.971565+0000","last_active":"2024-04-17T10:34:30.110252+0000","last_peered":"2024-04-17T10:34:30.110252+0000","last_clean":"2024-04-17T10:34:30.110252+0000","last_became_active":"2024-04-17T10:34:28.971280+0000","last_became_peered":"2024-04-17T10:34:28.971280+0000","last_unstale":"2024-04-17T10:34:30.110252+0000","last_undegraded":"2024-04-17T10:34:30.110252+0000","last_fullsized":"2024-04-17T10:34:30.110252+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:28:32.874971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.13","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110335+0000","last_change":"2024-04-17T10:34:27.960133+0000","last_active":"2024-04-17T10:34:30.110335+0000","last_peered":"2024-04-17T10:34:30.110335+0000","last_clean":"2024-04-17T10:34:30.110335+0000","last_became_active":"2024-04-17T10:34:27.959866+0000","last_became_peered":"2024-04-17T10:34:27.959866+0000","last_unstale":"2024-04-17T10:34:30.110335+0000","last_undegraded":"2024-04-17T10:34:30.110335+0000","last_fullsized":"2024-04-17T10:34:30.110335+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:51:11.592722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108837+0000","last_change":"2024-04-17T10:34:28.983729+0000","last_active":"2024-04-17T10:34:30.108837+0000","last_peered":"2024-04-17T10:34:30.108837+0000","last_clean":"2024-04-17T10:34:30.108837+0000","last_became_active":"2024-04-17T10:34:28.983332+0000","last_became_peered":"2024-04-17T10:34:28.983332+0000","last_unstale":"2024-04-17T10:34:30.108837+0000","last_undegraded":"2024-04-17T10:34:30.108837+0000","last_fullsized":"2024-04-17T10:34:30.108837+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:28.986152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107426+0000","last_change":"2024-04-17T10:34:27.952616+0000","last_active":"2024-04-17T10:34:30.107426+0000","last_peered":"2024-04-17T10:34:30.107426+0000","last_clean":"2024-04-17T10:34:30.107426+0000","last_became_active":"2024-04-17T10:34:27.952361+0000","last_became_peered":"2024-04-17T10:34:27.952361+0000","last_unstale":"2024-04-17T10:34:30.107426+0000","last_undegraded":"2024-04-17T10:34:30.107426+0000","last_fullsized":"2024-04-17T10:34:30.107426+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:40:46.196119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111366+0000","last_change":"2024-04-17T10:34:28.975608+0000","last_active":"2024-04-17T10:34:30.111366+0000","last_peered":"2024-04-17T10:34:30.111366+0000","last_clean":"2024-04-17T10:34:30.111366+0000","last_became_active":"2024-04-17T10:34:28.974801+0000","last_became_peered":"2024-04-17T10:34:28.974801+0000","last_unstale":"2024-04-17T10:34:30.111366+0000","last_undegraded":"2024-04-17T10:34:30.111366+0000","last_fullsized":"2024-04-17T10:34:30.111366+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:58:51.767495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.10","version":"129'8","reported_seq":22,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967582+0000","last_change":"2024-04-17T10:34:27.947171+0000","last_active":"2024-04-17T10:34:29.967582+0000","last_peered":"2024-04-17T10:34:29.967582+0000","last_clean":"2024-04-17T10:34:29.967582+0000","last_became_active":"2024-04-17T10:34:27.946849+0000","last_became_peered":"2024-04-17T10:34:27.946849+0000","last_unstale":"2024-04-17T10:34:29.967582+0000","last_undegraded":"2024-04-17T10:34:29.967582+0000","last_fullsized":"2024-04-17T10:34:29.967582+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:26:57.944745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107631+0000","last_change":"2024-04-17T10:34:28.980083+0000","last_active":"2024-04-17T10:34:30.107631+0000","last_peered":"2024-04-17T10:34:30.107631+0000","last_clean":"2024-04-17T10:34:30.107631+0000","last_became_active":"2024-04-17T10:34:28.979637+0000","last_became_peered":"2024-04-17T10:34:28.979637+0000","last_unstale":"2024-04-17T10:34:30.107631+0000","last_undegraded":"2024-04-17T10:34:30.107631+0000","last_fullsized":"2024-04-17T10:34:30.107631+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:31.502413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.f","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966842+0000","last_change":"2024-04-17T10:34:27.953712+0000","last_active":"2024-04-17T10:34:29.966842+0000","last_peered":"2024-04-17T10:34:29.966842+0000","last_clean":"2024-04-17T10:34:29.966842+0000","last_became_active":"2024-04-17T10:34:27.953463+0000","last_became_peered":"2024-04-17T10:34:27.953463+0000","last_unstale":"2024-04-17T10:34:29.966842+0000","last_undegraded":"2024-04-17T10:34:29.966842+0000","last_fullsized":"2024-04-17T10:34:29.966842+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:58:10.714307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108033+0000","last_change":"2024-04-17T10:34:28.987847+0000","last_active":"2024-04-17T10:34:30.108033+0000","last_peered":"2024-04-17T10:34:30.108033+0000","last_clean":"2024-04-17T10:34:30.108033+0000","last_became_active":"2024-04-17T10:34:28.987362+0000","last_became_peered":"2024-04-17T10:34:28.987362+0000","last_unstale":"2024-04-17T10:34:30.108033+0000","last_undegraded":"2024-04-17T10:34:30.108033+0000","last_fullsized":"2024-04-17T10:34:30.108033+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:18:09.832833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.e","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108528+0000","last_change":"2024-04-17T10:34:27.949802+0000","last_active":"2024-04-17T10:34:30.108528+0000","last_peered":"2024-04-17T10:34:30.108528+0000","last_clean":"2024-04-17T10:34:30.108528+0000","last_became_active":"2024-04-17T10:34:27.949528+0000","last_became_peered":"2024-04-17T10:34:27.949528+0000","last_unstale":"2024-04-17T10:34:30.108528+0000","last_undegraded":"2024-04-17T10:34:30.108528+0000","last_fullsized":"2024-04-17T10:34:30.108528+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:10.823560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108609+0000","last_change":"2024-04-17T10:34:28.989985+0000","last_active":"2024-04-17T10:34:30.108609+0000","last_peered":"2024-04-17T10:34:30.108609+0000","last_clean":"2024-04-17T10:34:30.108609+0000","last_became_active":"2024-04-17T10:34:28.987071+0000","last_became_peered":"2024-04-17T10:34:28.987071+0000","last_unstale":"2024-04-17T10:34:30.108609+0000","last_undegraded":"2024-04-17T10:34:30.108609+0000","last_fullsized":"2024-04-17T10:34:30.108609+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:56.269481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.d","version":"129'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968040+0000","last_change":"2024-04-17T10:34:27.949219+0000","last_active":"2024-04-17T10:34:29.968040+0000","last_peered":"2024-04-17T10:34:29.968040+0000","last_clean":"2024-04-17T10:34:29.968040+0000","last_became_active":"2024-04-17T10:34:27.948908+0000","last_became_peered":"2024-04-17T10:34:27.948908+0000","last_unstale":"2024-04-17T10:34:29.968040+0000","last_undegraded":"2024-04-17T10:34:29.968040+0000","last_fullsized":"2024-04-17T10:34:29.968040+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:59.759048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.b","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110524+0000","last_change":"2024-04-17T10:34:28.980366+0000","last_active":"2024-04-17T10:34:30.110524+0000","last_peered":"2024-04-17T10:34:30.110524+0000","last_clean":"2024-04-17T10:34:30.110524+0000","last_became_active":"2024-04-17T10:34:28.980114+0000","last_became_peered":"2024-04-17T10:34:28.980114+0000","last_unstale":"2024-04-17T10:34:30.110524+0000","last_undegraded":"2024-04-17T10:34:30.110524+0000","last_fullsized":"2024-04-17T10:34:30.110524+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:43:12.890477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111327+0000","last_change":"2024-04-17T10:34:27.952979+0000","last_active":"2024-04-17T10:34:30.111327+0000","last_peered":"2024-04-17T10:34:30.111327+0000","last_clean":"2024-04-17T10:34:30.111327+0000","last_became_active":"2024-04-17T10:34:27.952545+0000","last_became_peered":"2024-04-17T10:34:27.952545+0000","last_unstale":"2024-04-17T10:34:30.111327+0000","last_undegraded":"2024-04-17T10:34:30.111327+0000","last_fullsized":"2024-04-17T10:34:30.111327+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:51.711052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111393+0000","last_change":"2024-04-17T10:34:28.972194+0000","last_active":"2024-04-17T10:34:30.111393+0000","last_peered":"2024-04-17T10:34:30.111393+0000","last_clean":"2024-04-17T10:34:30.111393+0000","last_became_active":"2024-04-17T10:34:28.970371+0000","last_became_peered":"2024-04-17T10:34:28.970371+0000","last_unstale":"2024-04-17T10:34:30.111393+0000","last_undegraded":"2024-04-17T10:34:30.111393+0000","last_fullsized":"2024-04-17T10:34:30.111393+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:51:04.516924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.b","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966589+0000","last_change":"2024-04-17T10:34:27.945066+0000","last_active":"2024-04-17T10:34:29.966589+0000","last_peered":"2024-04-17T10:34:29.966589+0000","last_clean":"2024-04-17T10:34:29.966589+0000","last_became_active":"2024-04-17T10:34:27.944583+0000","last_became_peered":"2024-04-17T10:34:27.944583+0000","last_unstale":"2024-04-17T10:34:29.966589+0000","last_undegraded":"2024-04-17T10:34:29.966589+0000","last_fullsized":"2024-04-17T10:34:29.966589+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:52.175957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.d","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110402+0000","last_change":"2024-04-17T10:34:28.987393+0000","last_active":"2024-04-17T10:34:30.110402+0000","last_peered":"2024-04-17T10:34:30.110402+0000","last_clean":"2024-04-17T10:34:30.110402+0000","last_became_active":"2024-04-17T10:34:28.987111+0000","last_became_peered":"2024-04-17T10:34:28.987111+0000","last_unstale":"2024-04-17T10:34:30.110402+0000","last_undegraded":"2024-04-17T10:34:30.110402+0000","last_fullsized":"2024-04-17T10:34:30.110402+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:45:31.163780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110182+0000","last_change":"2024-04-17T10:34:27.950378+0000","last_active":"2024-04-17T10:34:30.110182+0000","last_peered":"2024-04-17T10:34:30.110182+0000","last_clean":"2024-04-17T10:34:30.110182+0000","last_became_active":"2024-04-17T10:34:27.950033+0000","last_became_peered":"2024-04-17T10:34:27.950033+0000","last_unstale":"2024-04-17T10:34:30.110182+0000","last_undegraded":"2024-04-17T10:34:30.110182+0000","last_fullsized":"2024-04-17T10:34:30.110182+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:30:04.390056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111321+0000","last_change":"2024-04-17T10:34:28.984017+0000","last_active":"2024-04-17T10:34:30.111321+0000","last_peered":"2024-04-17T10:34:30.111321+0000","last_clean":"2024-04-17T10:34:30.111321+0000","last_became_active":"2024-04-17T10:34:28.983702+0000","last_became_peered":"2024-04-17T10:34:28.983702+0000","last_unstale":"2024-04-17T10:34:30.111321+0000","last_undegraded":"2024-04-17T10:34:30.111321+0000","last_fullsized":"2024-04-17T10:34:30.111321+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:14:21.004154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108095+0000","last_change":"2024-04-17T10:34:27.955926+0000","last_active":"2024-04-17T10:34:30.108095+0000","last_peered":"2024-04-17T10:34:30.108095+0000","last_clean":"2024-04-17T10:34:30.108095+0000","last_became_active":"2024-04-17T10:34:27.955613+0000","last_became_peered":"2024-04-17T10:34:27.955613+0000","last_unstale":"2024-04-17T10:34:30.108095+0000","last_undegraded":"2024-04-17T10:34:30.108095+0000","last_fullsized":"2024-04-17T10:34:30.108095+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:24.825272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966864+0000","last_change":"2024-04-17T10:34:28.979339+0000","last_active":"2024-04-17T10:34:29.966864+0000","last_peered":"2024-04-17T10:34:29.966864+0000","last_clean":"2024-04-17T10:34:29.966864+0000","last_became_active":"2024-04-17T10:34:28.979069+0000","last_became_peered":"2024-04-17T10:34:28.979069+0000","last_unstale":"2024-04-17T10:34:29.966864+0000","last_undegraded":"2024-04-17T10:34:29.966864+0000","last_fullsized":"2024-04-17T10:34:29.966864+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:36:25.832508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967620+0000","last_change":"2024-04-17T10:34:27.956132+0000","last_active":"2024-04-17T10:34:29.967620+0000","last_peered":"2024-04-17T10:34:29.967620+0000","last_clean":"2024-04-17T10:34:29.967620+0000","last_became_active":"2024-04-17T10:34:27.955624+0000","last_became_peered":"2024-04-17T10:34:27.955624+0000","last_unstale":"2024-04-17T10:34:29.967620+0000","last_undegraded":"2024-04-17T10:34:29.967620+0000","last_fullsized":"2024-04-17T10:34:29.967620+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:02:52.175957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110992+0000","last_change":"2024-04-17T10:34:28.984158+0000","last_active":"2024-04-17T10:34:30.110992+0000","last_peered":"2024-04-17T10:34:30.110992+0000","last_clean":"2024-04-17T10:34:30.110992+0000","last_became_active":"2024-04-17T10:34:28.983850+0000","last_became_peered":"2024-04-17T10:34:28.983850+0000","last_unstale":"2024-04-17T10:34:30.110992+0000","last_undegraded":"2024-04-17T10:34:30.110992+0000","last_fullsized":"2024-04-17T10:34:30.110992+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:32:26.493336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.7","version":"129'2","reported_seq":16,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.111001+0000","last_change":"2024-04-17T10:34:27.958486+0000","last_active":"2024-04-17T10:34:30.111001+0000","last_peered":"2024-04-17T10:34:30.111001+0000","last_clean":"2024-04-17T10:34:30.111001+0000","last_became_active":"2024-04-17T10:34:27.954907+0000","last_became_peered":"2024-04-17T10:34:27.954907+0000","last_unstale":"2024-04-17T10:34:30.111001+0000","last_undegraded":"2024-04-17T10:34:30.111001+0000","last_fullsized":"2024-04-17T10:34:30.111001+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:34:10.656666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107652+0000","last_change":"2024-04-17T10:34:28.969270+0000","last_active":"2024-04-17T10:34:30.107652+0000","last_peered":"2024-04-17T10:34:30.107652+0000","last_clean":"2024-04-17T10:34:30.107652+0000","last_became_active":"2024-04-17T10:34:28.968928+0000","last_became_peered":"2024-04-17T10:34:28.968928+0000","last_unstale":"2024-04-17T10:34:30.107652+0000","last_undegraded":"2024-04-17T10:34:30.107652+0000","last_fullsized":"2024-04-17T10:34:30.107652+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:45.570663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.6","version":"130'1","reported_seq":15,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967544+0000","last_change":"2024-04-17T10:34:27.949187+0000","last_active":"2024-04-17T10:34:29.967544+0000","last_peered":"2024-04-17T10:34:29.967544+0000","last_clean":"2024-04-17T10:34:29.967544+0000","last_became_active":"2024-04-17T10:34:27.948931+0000","last_became_peered":"2024-04-17T10:34:27.948931+0000","last_unstale":"2024-04-17T10:34:29.967544+0000","last_undegraded":"2024-04-17T10:34:29.967544+0000","last_fullsized":"2024-04-17T10:34:29.967544+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:44.563427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108622+0000","last_change":"2024-04-17T10:34:28.985318+0000","last_active":"2024-04-17T10:34:30.108622+0000","last_peered":"2024-04-17T10:34:30.108622+0000","last_clean":"2024-04-17T10:34:30.108622+0000","last_became_active":"2024-04-17T10:34:28.983874+0000","last_became_peered":"2024-04-17T10:34:28.983874+0000","last_unstale":"2024-04-17T10:34:30.108622+0000","last_undegraded":"2024-04-17T10:34:30.108622+0000","last_fullsized":"2024-04-17T10:34:30.108622+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:25:58.674364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967239+0000","last_change":"2024-04-17T10:34:27.949100+0000","last_active":"2024-04-17T10:34:29.967239+0000","last_peered":"2024-04-17T10:34:29.967239+0000","last_clean":"2024-04-17T10:34:29.967239+0000","last_became_active":"2024-04-17T10:34:27.948265+0000","last_became_peered":"2024-04-17T10:34:27.948265+0000","last_unstale":"2024-04-17T10:34:29.967239+0000","last_undegraded":"2024-04-17T10:34:29.967239+0000","last_fullsized":"2024-04-17T10:34:29.967239+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:30.495177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110479+0000","last_change":"2024-04-17T10:34:28.987298+0000","last_active":"2024-04-17T10:34:30.110479+0000","last_peered":"2024-04-17T10:34:30.110479+0000","last_clean":"2024-04-17T10:34:30.110479+0000","last_became_active":"2024-04-17T10:34:28.986389+0000","last_became_peered":"2024-04-17T10:34:28.986389+0000","last_unstale":"2024-04-17T10:34:30.110479+0000","last_undegraded":"2024-04-17T10:34:30.110479+0000","last_fullsized":"2024-04-17T10:34:30.110479+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:54:53.892216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967946+0000","last_change":"2024-04-17T10:34:27.950978+0000","last_active":"2024-04-17T10:34:29.967946+0000","last_peered":"2024-04-17T10:34:29.967946+0000","last_clean":"2024-04-17T10:34:29.967946+0000","last_became_active":"2024-04-17T10:34:27.950693+0000","last_became_peered":"2024-04-17T10:34:27.950693+0000","last_unstale":"2024-04-17T10:34:29.967946+0000","last_undegraded":"2024-04-17T10:34:29.967946+0000","last_fullsized":"2024-04-17T10:34:29.967946+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:17:11.995752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968156+0000","last_change":"2024-04-17T10:34:28.965007+0000","last_active":"2024-04-17T10:34:29.968156+0000","last_peered":"2024-04-17T10:34:29.968156+0000","last_clean":"2024-04-17T10:34:29.968156+0000","last_became_active":"2024-04-17T10:34:28.964702+0000","last_became_peered":"2024-04-17T10:34:28.964702+0000","last_unstale":"2024-04-17T10:34:29.968156+0000","last_undegraded":"2024-04-17T10:34:29.968156+0000","last_fullsized":"2024-04-17T10:34:29.968156+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:55:00.945480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.966912+0000","last_change":"2024-04-17T10:34:27.959806+0000","last_active":"2024-04-17T10:34:29.966912+0000","last_peered":"2024-04-17T10:34:29.966912+0000","last_clean":"2024-04-17T10:34:29.966912+0000","last_became_active":"2024-04-17T10:34:27.959486+0000","last_became_peered":"2024-04-17T10:34:27.959486+0000","last_unstale":"2024-04-17T10:34:29.966912+0000","last_undegraded":"2024-04-17T10:34:29.966912+0000","last_fullsized":"2024-04-17T10:34:29.966912+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:08:00.531443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968111+0000","last_change":"2024-04-17T10:34:28.969624+0000","last_active":"2024-04-17T10:34:29.968111+0000","last_peered":"2024-04-17T10:34:29.968111+0000","last_clean":"2024-04-17T10:34:29.968111+0000","last_became_active":"2024-04-17T10:34:28.969314+0000","last_became_peered":"2024-04-17T10:34:28.969314+0000","last_unstale":"2024-04-17T10:34:29.968111+0000","last_undegraded":"2024-04-17T10:34:29.968111+0000","last_fullsized":"2024-04-17T10:34:29.968111+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:19:56.886865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108694+0000","last_change":"2024-04-17T10:34:27.946338+0000","last_active":"2024-04-17T10:34:30.108694+0000","last_peered":"2024-04-17T10:34:30.108694+0000","last_clean":"2024-04-17T10:34:30.108694+0000","last_became_active":"2024-04-17T10:34:27.945265+0000","last_became_peered":"2024-04-17T10:34:27.945265+0000","last_unstale":"2024-04-17T10:34:30.108694+0000","last_undegraded":"2024-04-17T10:34:30.108694+0000","last_fullsized":"2024-04-17T10:34:30.108694+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.013210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.968125+0000","last_change":"2024-04-17T10:34:28.989082+0000","last_active":"2024-04-17T10:34:29.968125+0000","last_peered":"2024-04-17T10:34:29.968125+0000","last_clean":"2024-04-17T10:34:29.968125+0000","last_became_active":"2024-04-17T10:34:28.988817+0000","last_became_peered":"2024-04-17T10:34:28.988817+0000","last_unstale":"2024-04-17T10:34:29.968125+0000","last_undegraded":"2024-04-17T10:34:29.968125+0000","last_fullsized":"2024-04-17T10:34:29.968125+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:58:52.718288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:29.967552+0000","last_change":"2024-04-17T10:34:27.951045+0000","last_active":"2024-04-17T10:34:29.967552+0000","last_peered":"2024-04-17T10:34:29.967552+0000","last_clean":"2024-04-17T10:34:29.967552+0000","last_became_active":"2024-04-17T10:34:27.950758+0000","last_became_peered":"2024-04-17T10:34:27.950758+0000","last_unstale":"2024-04-17T10:34:29.967552+0000","last_undegraded":"2024-04-17T10:34:29.967552+0000","last_fullsized":"2024-04-17T10:34:29.967552+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:53:30.495177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.7","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.109018+0000","last_change":"2024-04-17T10:34:28.984105+0000","last_active":"2024-04-17T10:34:30.109018+0000","last_peered":"2024-04-17T10:34:30.109018+0000","last_clean":"2024-04-17T10:34:30.109018+0000","last_became_active":"2024-04-17T10:34:28.983649+0000","last_became_peered":"2024-04-17T10:34:28.983649+0000","last_unstale":"2024-04-17T10:34:30.109018+0000","last_undegraded":"2024-04-17T10:34:30.109018+0000","last_fullsized":"2024-04-17T10:34:30.109018+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:49:42.467741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.110922+0000","last_change":"2024-04-17T10:34:27.949819+0000","last_active":"2024-04-17T10:34:30.110922+0000","last_peered":"2024-04-17T10:34:30.110922+0000","last_clean":"2024-04-17T10:34:30.110922+0000","last_became_active":"2024-04-17T10:34:27.949452+0000","last_became_peered":"2024-04-17T10:34:27.949452+0000","last_unstale":"2024-04-17T10:34:30.110922+0000","last_undegraded":"2024-04-17T10:34:30.110922+0000","last_fullsized":"2024-04-17T10:34:30.110922+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:03:57.013210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.15","version":"0'0","reported_seq":12,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108445+0000","last_change":"2024-04-17T10:34:28.975707+0000","last_active":"2024-04-17T10:34:30.108445+0000","last_peered":"2024-04-17T10:34:30.108445+0000","last_clean":"2024-04-17T10:34:30.108445+0000","last_became_active":"2024-04-17T10:34:28.975096+0000","last_became_peered":"2024-04-17T10:34:28.975096+0000","last_unstale":"2024-04-17T10:34:30.108445+0000","last_undegraded":"2024-04-17T10:34:30.108445+0000","last_fullsized":"2024-04-17T10:34:30.108445+0000","mapping_epoch":128,"log_start":"0'0","ondisk_log_start":"0'0","created":128,"last_epoch_clean":129,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:27.928185+0000","last_clean_scrub_stamp":"2024-04-17T10:34:27.928185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:01:34.835074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":14,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.108783+0000","last_change":"2024-04-17T10:34:27.955786+0000","last_active":"2024-04-17T10:34:30.108783+0000","last_peered":"2024-04-17T10:34:30.108783+0000","last_clean":"2024-04-17T10:34:30.108783+0000","last_became_active":"2024-04-17T10:34:27.955527+0000","last_became_peered":"2024-04-17T10:34:27.955527+0000","last_unstale":"2024-04-17T10:34:30.108783+0000","last_undegraded":"2024-04-17T10:34:30.108783+0000","last_fullsized":"2024-04-17T10:34:30.108783+0000","mapping_epoch":127,"log_start":"0'0","ondisk_log_start":"0'0","created":127,"last_epoch_clean":128,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:34:26.920950+0000","last_clean_scrub_stamp":"2024-04-17T10:34:26.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:01:46.011255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":371,"reported_epoch":130,"state":"active+clean","last_fresh":"2024-04-17T10:34:30.107342+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:34:30.107342+0000","last_peered":"2024-04-17T10:34:30.107342+0000","last_clean":"2024-04-17T10:34:30.107342+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:34:30.107342+0000","last_undegraded":"2024-04-17T10:34:30.107342+0000","last_fullsized":"2024-04-17T10:34:30.107342+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":20,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":19,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574970,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51416,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320424,"statfs":{"total":96636764160,"available":96584114176,"internally_reserved":0,"allocated":2236416,"data_stored":1352393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574968,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54984,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316856,"statfs":{"total":96636764160,"available":96580460544,"internally_reserved":0,"allocated":1634304,"data_stored":760995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574969,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50828,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321012,"statfs":{"total":96636764160,"available":96584716288,"internally_reserved":0,"allocated":1630208,"data_stored":760345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574969,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59144,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312696,"statfs":{"total":96636764160,"available":96576200704,"internally_reserved":0,"allocated":1626112,"data_stored":760323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574969,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59152,"kb_used_data":1604,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312688,"statfs":{"total":96636764160,"available":96576192512,"internally_reserved":0,"allocated":1642496,"data_stored":762005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574970,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54988,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316852,"statfs":{"total":96636764160,"available":96580456448,"internally_reserved":0,"allocated":1634304,"data_stored":760939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50820,"kb_used_data":1592,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321020,"statfs":{"total":96636764160,"available":96584724480,"internally_reserved":0,"allocated":1630208,"data_stored":760393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574969,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59724,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312116,"statfs":{"total":96636764160,"available":96575606784,"internally_reserved":0,"allocated":2228224,"data_stored":1350747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:34:33.463 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:34:33.464 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:34:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb042c00640 1 Processor -- start 2024-04-17T10:34:33.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb042c00640 1 -- start start 2024-04-17T10:34:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb042c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb03c09a160 0x7fb03c0b2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb042c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0cf9d0 con 0x7fb03c160f40 2024-04-17T10:34:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb042c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c06cce0 con 0x7fb03c06c7b0 2024-04-17T10:34:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb042c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c0cfc40 con 0x7fb03c09a160 2024-04-17T10:34:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb041800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb03c160f40 0x7fb03c099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37778/0 (socket says 172.21.15.67:37778) 2024-04-17T10:34:33.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb041800640 1 -- 172.21.15.67:0/3000094586 learned_addr learned my addr 172.21.15.67:0/3000094586 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:33.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb040e00640 1 --2- 172.21.15.67:0/3000094586 >> v2:172.21.15.86:3300/0 conn(0x7fb03c09a160 0x7fb03c0b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb03be00640 1 -- 172.21.15.67:0/3000094586 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303210385 0 0) 0x7fb03c0cf9d0 con 0x7fb03c160f40 2024-04-17T10:34:33.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb03be00640 1 -- 172.21.15.67:0/3000094586 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb024003530 con 0x7fb03c160f40 2024-04-17T10:34:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb040e00640 1 -- 172.21.15.67:0/3000094586 >> v1:172.21.15.86:6789/0 conn(0x7fb03c06c7b0 legacy=0x7fb03c0d0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb040e00640 1 -- 172.21.15.67:0/3000094586 >> v1:172.21.15.67:6789/0 conn(0x7fb03c160f40 legacy=0x7fb03c099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.661+0000 7fb040e00640 1 -- 172.21.15.67:0/3000094586 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0cff20 con 0x7fb03c09a160 2024-04-17T10:34:33.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb040e00640 1 --2- 172.21.15.67:0/3000094586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c09a160 0x7fb03c0b2640 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb0280097c0 tx=0x7fb02801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c46647361c04918e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:33.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb03be00640 1 -- 172.21.15.67:0/3000094586 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0280b0070 con 0x7fb03c09a160 2024-04-17T10:34:33.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb03be00640 1 -- 172.21.15.67:0/3000094586 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0280ab600 con 0x7fb03c09a160 2024-04-17T10:34:33.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb03be00640 1 -- 172.21.15.67:0/3000094586 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0280ab9d0 con 0x7fb03c09a160 2024-04-17T10:34:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 -- 172.21.15.67:0/3000094586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c09a160 msgr2=0x7fb03c0b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 --2- 172.21.15.67:0/3000094586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c09a160 0x7fb03c0b2640 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb0280097c0 tx=0x7fb02801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 -- 172.21.15.67:0/3000094586 shutdown_connections 2024-04-17T10:34:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 --2- 172.21.15.67:0/3000094586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c09a160 0x7fb03c0b2640 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 -- 172.21.15.67:0/3000094586 >> 172.21.15.67:0/3000094586 conn(0x7fb03c0b3190 msgr2=0x7fb03c0b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 -- 172.21.15.67:0/3000094586 shutdown_connections 2024-04-17T10:34:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 -- 172.21.15.67:0/3000094586 wait complete. 2024-04-17T10:34:33.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 Processor -- start 2024-04-17T10:34:33.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.665+0000 7fb042c00640 1 -- start start 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb03c06c7b0 0x7fb03c14ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb03c09a160 0x7fb03c0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c160f40 0x7fb03c154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c16bc10 con 0x7fb03c06c7b0 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c16ba90 con 0x7fb03c160f40 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c16bd90 con 0x7fb03c09a160 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb03c06c7b0 0x7fb03c14ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb03c06c7b0 0x7fb03c14ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48064/0 (socket says 172.21.15.67:48064) 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 -- 172.21.15.67:0/2058247709 learned_addr learned my addr 172.21.15.67:0/2058247709 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042200640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c160f40 0x7fb03c154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb040e00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb03c09a160 0x7fb03c0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 -- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb03c09a160 msgr2=0x7fb03c0ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb03c09a160 0x7fb03c0ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 -- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c160f40 msgr2=0x7fb03c154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c160f40 0x7fb03c154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 -- 172.21.15.67:0/2058247709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c0cafa0 con 0x7fb03c06c7b0 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb041800640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb03c06c7b0 0x7fb03c14ffd0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb03000c450 tx=0x7fb03000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb039600640 1 -- 172.21.15.67:0/2058247709 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb030017070 con 0x7fb03c06c7b0 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb039600640 1 -- 172.21.15.67:0/2058247709 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb030005cf0 con 0x7fb03c06c7b0 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb039600640 1 -- 172.21.15.67:0/2058247709 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb030007270 con 0x7fb03c06c7b0 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c0cb280 con 0x7fb03c06c7b0 2024-04-17T10:34:33.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c150730 con 0x7fb03c06c7b0 2024-04-17T10:34:33.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.669+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03c074dc0 con 0x7fb03c06c7b0 2024-04-17T10:34:33.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.673+0000 7fb039600640 1 -- 172.21.15.67:0/2058247709 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb03001a070 con 0x7fb03c06c7b0 2024-04-17T10:34:33.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.673+0000 7fb039600640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb01c066f60 0x7fb01c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:33.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.673+0000 7fb039600640 1 -- 172.21.15.67:0/2058247709 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb030084470 con 0x7fb03c06c7b0 2024-04-17T10:34:33.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.673+0000 7fb040e00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb01c066f60 0x7fb01c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:33.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.673+0000 7fb040e00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb01c066f60 0x7fb01c069420 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb0280ad000 tx=0x7fb028008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:33.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.677+0000 7fb039600640 1 -- 172.21.15.67:0/2058247709 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb030054c00 con 0x7fb03c06c7b0 2024-04-17T10:34:33.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.897+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb03c153150 con 0x7fb03c06c7b0 2024-04-17T10:34:33.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.897+0000 7fb039600640 1 -- 172.21.15.67:0/2058247709 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb030058ca0 con 0x7fb03c06c7b0 2024-04-17T10:34:33.889 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:34:33.890 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:34:33.890 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:34:33.890 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:34:33.890 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:34:33.890 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb01c066f60 msgr2=0x7fb01c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb01c066f60 0x7fb01c069420 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fb0280ad000 tx=0x7fb028008040 comp rx=0 tx=0).stop 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb03c06c7b0 msgr2=0x7fb03c14ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb03c06c7b0 0x7fb03c14ffd0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fb03000c450 tx=0x7fb03000c910 comp rx=0 tx=0).stop 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 shutdown_connections 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb01c066f60 0x7fb01c069420 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb03c160f40 0x7fb03c154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb03c09a160 0x7fb03c0ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 --2- 172.21.15.67:0/2058247709 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb03c06c7b0 0x7fb03c14ffd0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 >> 172.21.15.67:0/2058247709 conn(0x7fb03c0b3190 msgr2=0x7fb03c073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 shutdown_connections 2024-04-17T10:34:33.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:33.901+0000 7fb042c00640 1 -- 172.21.15.67:0/2058247709 wait complete. 2024-04-17T10:34:33.908 INFO:tasks.ceph.mds.d.smithi086.stderr:2024-04-17T10:34:33.914+0000 7f7ada600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:34:33.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:34.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb847c00640 1 Processor -- start 2024-04-17T10:34:34.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb847c00640 1 -- start start 2024-04-17T10:34:34.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb847c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb840071770 0x7fb8400cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb847c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84006e2f0 con 0x7fb840145040 2024-04-17T10:34:34.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb847c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84006d6e0 con 0x7fb840071310 2024-04-17T10:34:34.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb847c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8400a86c0 con 0x7fb840071770 2024-04-17T10:34:34.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb845200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb840145040 0x7fb84006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37788/0 (socket says 172.21.15.67:37788) 2024-04-17T10:34:34.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.061+0000 7fb845200640 1 -- 172.21.15.67:0/3053178377 learned_addr learned my addr 172.21.15.67:0/3053178377 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:34.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb845c00640 1 --2- 172.21.15.67:0/3053178377 >> v2:172.21.15.67:3300/0 conn(0x7fb840071770 0x7fb8400cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb845c00640 1 -- 172.21.15.67:0/3053178377 >> v1:172.21.15.86:6789/0 conn(0x7fb840071310 legacy=0x7fb8400cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb845c00640 1 -- 172.21.15.67:0/3053178377 >> v1:172.21.15.67:6789/0 conn(0x7fb840145040 legacy=0x7fb84006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb845c00640 1 -- 172.21.15.67:0/3053178377 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84006d9c0 con 0x7fb840071770 2024-04-17T10:34:34.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb83f400640 1 -- 172.21.15.67:0/3053178377 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579949484 0 0) 0x7fb84006d6e0 con 0x7fb840071310 2024-04-17T10:34:34.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb845c00640 1 --2- 172.21.15.67:0/3053178377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb8400cca10 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fb838009d90 tx=0x7fb83801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e83ae83545e6f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb83f400640 1 -- 172.21.15.67:0/3053178377 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380b2070 con 0x7fb840071770 2024-04-17T10:34:34.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb83f400640 1 -- 172.21.15.67:0/3053178377 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8380a58f0 con 0x7fb840071770 2024-04-17T10:34:34.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.065+0000 7fb83f400640 1 -- 172.21.15.67:0/3053178377 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8380a5cc0 con 0x7fb840071770 2024-04-17T10:34:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- 172.21.15.67:0/3053178377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 msgr2=0x7fb8400cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 --2- 172.21.15.67:0/3053178377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb8400cca10 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fb838009d90 tx=0x7fb83801c890 comp rx=0 tx=0).stop 2024-04-17T10:34:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- 172.21.15.67:0/3053178377 shutdown_connections 2024-04-17T10:34:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 --2- 172.21.15.67:0/3053178377 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb8400cca10 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- 172.21.15.67:0/3053178377 >> 172.21.15.67:0/3053178377 conn(0x7fb84006ac90 msgr2=0x7fb84006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- 172.21.15.67:0/3053178377 shutdown_connections 2024-04-17T10:34:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- 172.21.15.67:0/3053178377 wait complete. 2024-04-17T10:34:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 Processor -- start 2024-04-17T10:34:34.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- start start 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb840071310 0x7fb840141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb84015e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb840145040 0x7fb84015ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb840071b80 con 0x7fb840071770 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84006df80 con 0x7fb840145040 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb847c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84006d370 con 0x7fb840071310 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb84015e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb84015e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48084/0 (socket says 172.21.15.67:48084) 2024-04-17T10:34:34.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 -- 172.21.15.67:0/3733103319 learned_addr learned my addr 172.21.15.67:0/3733103319 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb845200640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb840071310 0x7fb840141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb845c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb840145040 0x7fb84015ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 -- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb840071310 msgr2=0x7fb840141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb840071310 0x7fb840141730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 -- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb840145040 msgr2=0x7fb84015ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb840145040 0x7fb84015ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 -- 172.21.15.67:0/3733103319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8400c46e0 con 0x7fb840071770 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb845c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb840145040 0x7fb84015ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.069+0000 7fb83fe00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb84015e150 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c480 tx=0x7fb82c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb83cc00640 1 -- 172.21.15.67:0/3733103319 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c005480 con 0x7fb840071770 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb83cc00640 1 -- 172.21.15.67:0/3733103319 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb82c003600 con 0x7fb840071770 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8400adb70 con 0x7fb840071770 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb83cc00640 1 -- 172.21.15.67:0/3733103319 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb82c0039f0 con 0x7fb840071770 2024-04-17T10:34:34.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8400b2f70 con 0x7fb840071770 2024-04-17T10:34:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb840072d50 con 0x7fb840071770 2024-04-17T10:34:34.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb83cc00640 1 -- 172.21.15.67:0/3733103319 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb82c003b90 con 0x7fb840071770 2024-04-17T10:34:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb83cc00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb820066f90 0x7fb820069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb83cc00640 1 -- 172.21.15.67:0/3733103319 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb82c084dc0 con 0x7fb840071770 2024-04-17T10:34:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb845200640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb820066f90 0x7fb820069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.073+0000 7fb845200640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb820066f90 0x7fb820069450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb8400721d0 tx=0x7fb83400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.077+0000 7fb83cc00640 1 -- 172.21.15.67:0/3733103319 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb82c055cd0 con 0x7fb840071770 2024-04-17T10:34:34.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.273+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8400cc4a0 con 0x7fb840071770 2024-04-17T10:34:34.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.273+0000 7fb83cc00640 1 -- 172.21.15.67:0/3733103319 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7fb82c0555d0 con 0x7fb840071770 2024-04-17T10:34:34.266 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:34.266 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:34:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:34:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb820066f90 msgr2=0x7fb820069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb820066f90 0x7fb820069450 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fb8400721d0 tx=0x7fb83400b040 comp rx=0 tx=0).stop 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 msgr2=0x7fb84015e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb84015e150 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fb82c00c480 tx=0x7fb82c00c940 comp rx=0 tx=0).stop 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 shutdown_connections 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb820066f90 0x7fb820069450 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb840145040 0x7fb84015ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb840071770 0x7fb84015e150 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 --2- 172.21.15.67:0/3733103319 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb840071310 0x7fb840141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 >> 172.21.15.67:0/3733103319 conn(0x7fb84006ac90 msgr2=0x7fb84006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 shutdown_connections 2024-04-17T10:34:34.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.277+0000 7fb847c00640 1 -- 172.21.15.67:0/3733103319 wait complete. 2024-04-17T10:34:34.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 90, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:34:28.382849+0000', 'modified': '2024-04-17T10:34:31.362617+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9270': {'gid': 9270, 'name': 'd', 'rank': 0, 'incarnation': 86, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.86:6836/538421223', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6834', 'nonce': 538421223}, {'type': 'v1', 'addr': '172.21.15.86:6836', 'nonce': 538421223}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:34:34.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:34:34.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-17T10:34:34.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.841+0000 7fcf97e00640 1 Processor -- start 2024-04-17T10:34:34.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf97e00640 1 -- start start 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf97e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf9006e6e0 0x7fcf900ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf97e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9006eac0 0x7fcf900ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf97e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf9015ae40 con 0x7fcf90143ae0 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf97e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf900cf430 con 0x7fcf9006e6e0 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf97e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90071440 con 0x7fcf9006eac0 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf95e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcf90143ae0 0x7fcf90143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37792/0 (socket says 172.21.15.67:37792) 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf95e00640 1 -- 172.21.15.67:0/842670730 learned_addr learned my addr 172.21.15.67:0/842670730 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf95400640 1 --2- 172.21.15.67:0/842670730 >> v2:172.21.15.86:3300/0 conn(0x7fcf9006e6e0 0x7fcf900ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf94a00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9006eac0 0x7fcf900ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033562685 0 0) 0x7fcf9015ae40 con 0x7fcf90143ae0 2024-04-17T10:34:34.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf78003530 con 0x7fcf90143ae0 2024-04-17T10:34:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381663389 0 0) 0x7fcf78003530 con 0x7fcf90143ae0 2024-04-17T10:34:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9006eac0 msgr2=0x7fcf900ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf9006eac0 0x7fcf900ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 >> v2:172.21.15.86:3300/0 conn(0x7fcf9006e6e0 msgr2=0x7fcf900ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 --2- 172.21.15.67:0/842670730 >> v2:172.21.15.86:3300/0 conn(0x7fcf9006e6e0 0x7fcf900ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90071720 con 0x7fcf90143ae0 2024-04-17T10:34:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fcf88001a00 con 0x7fcf90143ae0 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 >> v1:172.21.15.67:6789/0 conn(0x7fcf90143ae0 legacy=0x7fcf90143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf780039c0 0x7fcf78003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf78004810 0x7fcf78004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf780051d0 0x7fcf7800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf900cf430 con 0x7fcf78004810 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90071440 con 0x7fcf780051d0 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7800bf90 con 0x7fcf780039c0 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf94a00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf78004810 0x7fcf78004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf95400640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf780039c0 0x7fcf78003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf95e00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf780051d0 0x7fcf7800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf94a00640 1 -- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf780039c0 msgr2=0x7fcf78003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf94a00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf780039c0 0x7fcf78003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf94a00640 1 -- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf780051d0 msgr2=0x7fcf7800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf94a00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf780051d0 0x7fcf7800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.845+0000 7fcf94a00640 1 -- 172.21.15.67:0/842670730 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78003530 con 0x7fcf78004810 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf94a00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf78004810 0x7fcf78004c90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c0099a0 tx=0x7fcf7c01c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ad62c5a7aaecaf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c009670 con 0x7fcf78004810 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c0a93f0 con 0x7fcf78004810 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf8fe00640 1 -- 172.21.15.67:0/842670730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0a97a0 con 0x7fcf78004810 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf97e00640 1 -- 172.21.15.67:0/842670730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf78004810 msgr2=0x7fcf78004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf97e00640 1 --2- 172.21.15.67:0/842670730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf78004810 0x7fcf78004c90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c0099a0 tx=0x7fcf7c01c900 comp rx=0 tx=0).stop 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf95e00640 1 -- 172.21.15.67:0/842670730 reap_dead start 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf97e00640 1 -- 172.21.15.67:0/842670730 shutdown_connections 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf97e00640 1 -- 172.21.15.67:0/842670730 >> 172.21.15.67:0/842670730 conn(0x7fcf9006ac90 msgr2=0x7fcf9006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf97e00640 1 -- 172.21.15.67:0/842670730 shutdown_connections 2024-04-17T10:34:34.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.849+0000 7fcf97e00640 1 -- 172.21.15.67:0/842670730 wait complete. 2024-04-17T10:34:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 Processor -- start 2024-04-17T10:34:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 -- start start 2024-04-17T10:34:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf78004810 0x7fcf9015a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf901605f0 0x7fcf90160a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf90165710 0x7fcf90165b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9006dd00 con 0x7fcf90165710 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9006db80 con 0x7fcf78004810 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9006de80 con 0x7fcf901605f0 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf90165710 0x7fcf90165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf90165710 0x7fcf90165b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48104/0 (socket says 172.21.15.67:48104) 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 -- 172.21.15.67:0/3485590725 learned_addr learned my addr 172.21.15.67:0/3485590725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf94a00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf901605f0 0x7fcf90160a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95400640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf78004810 0x7fcf9015a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 -- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf901605f0 msgr2=0x7fcf90160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf901605f0 0x7fcf90160a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 -- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf78004810 msgr2=0x7fcf9015a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf78004810 0x7fcf9015a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 -- 172.21.15.67:0/3485590725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9012f490 con 0x7fcf90165710 2024-04-17T10:34:34.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf95e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf90165710 0x7fcf90165b90 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fcf88002240 tx=0x7fcf88007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf8d600640 1 -- 172.21.15.67:0/3485590725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8800aa70 con 0x7fcf90165710 2024-04-17T10:34:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf8d600640 1 -- 172.21.15.67:0/3485590725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88018070 con 0x7fcf90165710 2024-04-17T10:34:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf8d600640 1 -- 172.21.15.67:0/3485590725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880133f0 con 0x7fcf90165710 2024-04-17T10:34:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9012f770 con 0x7fcf90165710 2024-04-17T10:34:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.853+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9012fd30 con 0x7fcf90165710 2024-04-17T10:34:34.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf8d600640 1 -- 172.21.15.67:0/3485590725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcf88013590 con 0x7fcf90165710 2024-04-17T10:34:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf8d600640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf70066f60 0x7fcf70069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf8d600640 1 -- 172.21.15.67:0/3485590725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf88085720 con 0x7fcf90165710 2024-04-17T10:34:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf95400640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf70066f60 0x7fcf70069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 Processor -- start 2024-04-17T10:34:34.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 -- start start 2024-04-17T10:34:34.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf95400640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf70066f60 0x7fcf70069420 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcf84009be0 tx=0x7fcf84009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de28920 0x55ad7de28d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7ddde9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7dddf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dcc65f0 con 0x55ad7dddef30 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dc2dda0 con 0x55ad7de28920 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dbb6150 con 0x55ad7de29240 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.857+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7dddf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7dddf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48108/0 (socket says 172.21.15.67:48108) 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 -- 172.21.15.67:0/647236124 learned_addr learned my addr 172.21.15.67:0/647236124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf94a00640 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7ddde9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95400640 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de28920 0x55ad7de28d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 -- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 msgr2=0x55ad7ddde9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7ddde9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 -- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de28920 msgr2=0x55ad7de28d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de28920 0x55ad7de28d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 -- 172.21.15.67:0/647236124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad7dbcfca0 con 0x55ad7dddef30 2024-04-17T10:34:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf94a00640 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7ddde9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95400640 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de28920 0x55ad7de28d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7dddf310 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fcf88055a70 tx=0x7fcf880557d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7f8e3680a53c48e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf82000640 1 -- 172.21.15.67:0/647236124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8805e510 con 0x55ad7dddef30 2024-04-17T10:34:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf82000640 1 -- 172.21.15.67:0/647236124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88054070 con 0x55ad7dddef30 2024-04-17T10:34:34.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf82000640 1 -- 172.21.15.67:0/647236124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880551c0 con 0x55ad7dddef30 2024-04-17T10:34:34.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/647236124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 msgr2=0x55ad7dddf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7dddf310 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fcf88055a70 tx=0x7fcf880557d0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/647236124 shutdown_connections 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7dddf310 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7ddde9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/647236124 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de28920 0x55ad7de28d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/647236124 >> 172.21.15.67:0/647236124 conn(0x55ad7dde52a0 msgr2=0x55ad7dddd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/647236124 shutdown_connections 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/647236124 wait complete. 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- start start 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 Processor -- start 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7de2fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de301c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de29240 0x55ad7dde2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dbe8990 con 0x55ad7dddef30 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dc29e40 con 0x55ad7de29240 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dbb9900 con 0x55ad7de28920 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf9b2eb000 10 client.0 _ll_register_callbacks cb 0x55ad7dce11c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7de2fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7de2fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48122/0 (socket says 172.21.15.67:48122) 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95400640 1 -- 172.21.15.67:0/1454285915 learned_addr learned my addr 172.21.15.67:0/1454285915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf95e00640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de29240 0x55ad7dde2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.861+0000 7fcf94a00640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf95400640 1 -- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 msgr2=0x55ad7de301c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf95400640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de301c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf95400640 1 -- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de29240 msgr2=0x55ad7dde2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf95400640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de29240 0x55ad7dde2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf95400640 1 -- 172.21.15.67:0/1454285915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad7dbe6630 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf94a00640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf95400640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7de2fc80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fcf84022eb0 tx=0x7fcf840046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa09b2c21d87395 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 1 -- 172.21.15.67:0/1454285915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8401b070 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 1 -- 172.21.15.67:0/1454285915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf84014e20 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf88054070 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 1 -- 172.21.15.67:0/1454285915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf84035050 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 10 client.9411 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55ad7dbb63a0 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 1 -- 172.21.15.67:0/1454285915 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf88054070 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 1 -- 172.21.15.67:0/1454285915 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcf84038020 con 0x55ad7dddef30 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 10 client.9411 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 10 client.9411 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:34:34.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 20 client.9411 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:34.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7dde34b0 0x55ad7dde3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 4 client.9411 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:34.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad7dbc9a50 con 0x55ad7dde34b0 2024-04-17T10:34:34.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf94a00640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7dde34b0 0x55ad7dde3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 10 client.9411 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf94a00640 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7dde34b0 0x55ad7dde3890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8fb9c5619bb651a7 server_cookie=5dd99206769ef03e in_seq=0 out_seq=0 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 1 -- 172.21.15.67:0/1454285915 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ad7dbc9a50 con 0x55ad7dde34b0 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 10 client.9411 handle_command_reply: tid=0 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 client.9411 shutdown 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 msgr2=0x55ad7de2fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7de2fc80 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fcf84022eb0 tx=0x7fcf840046d0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 shutdown_connections 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7dde34b0 0x55ad7dde3890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7de29240 0x55ad7dde2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf76a00640 0 client.9411 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1454285915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7dddef30 0x55ad7de2fc80 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.865+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 >> 172.21.15.67:0/1454285915 conn(0x55ad7dde52a0 msgr2=0x55ad7dddd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.869+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 shutdown_connections 2024-04-17T10:34:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.869+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1454285915 wait complete. 2024-04-17T10:34:34.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.869+0000 7fcf9b2eb000 20 client.9411 trim_cache size 0 max 16384 2024-04-17T10:34:34.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.881+0000 7fcf9b2eb000 1 Processor -- start 2024-04-17T10:34:34.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.881+0000 7fcf9b2eb000 1 -- start start 2024-04-17T10:34:34.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7dde2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de28920 0x55ad7dde2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7de32860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90202d10 con 0x55ad7de28920 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90132cc0 con 0x55ad7dddef30 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf90202710 con 0x55ad7de29240 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de28920 0x55ad7dde2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de28920 0x55ad7dde2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48124/0 (socket says 172.21.15.67:48124) 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 -- 172.21.15.67:0/3453495586 learned_addr learned my addr 172.21.15.67:0/3453495586 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf95400640 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7dde2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf95e00640 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7de32860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 -- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 msgr2=0x55ad7de32860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7de32860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 -- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 msgr2=0x55ad7dde2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7dde2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 -- 172.21.15.67:0/3453495586 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad7dcaf600 con 0x55ad7de28920 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf94a00640 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de28920 0x55ad7dde2870 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c000c10 tx=0x7fcf7c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7108e6e03aed27a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf81600640 1 -- 172.21.15.67:0/3453495586 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0044e0 con 0x55ad7de28920 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf81600640 1 -- 172.21.15.67:0/3453495586 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c003620 con 0x55ad7de28920 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf81600640 1 -- 172.21.15.67:0/3453495586 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0039d0 con 0x55ad7de28920 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/3453495586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de28920 msgr2=0x55ad7dde2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de28920 0x55ad7dde2870 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c000c10 tx=0x7fcf7c0046d0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/3453495586 shutdown_connections 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de29240 0x55ad7de32860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de28920 0x55ad7dde2870 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/3453495586 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7dde2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/3453495586 >> 172.21.15.67:0/3453495586 conn(0x55ad7dde52a0 msgr2=0x55ad7de14b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/3453495586 shutdown_connections 2024-04-17T10:34:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/3453495586 wait complete. 2024-04-17T10:34:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 -- start start 2024-04-17T10:34:34.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 Processor -- start 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.885+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7de357e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de44330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de29240 0x55ad7de44870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dc323d0 con 0x55ad7de29240 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dce9d50 con 0x55ad7dddef30 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad7dbc8980 con 0x55ad7de28920 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 10 client.0 _ll_register_callbacks cb 0x7fcf88052010 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:34.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de29240 0x55ad7de44870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de29240 0x55ad7de44870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48126/0 (socket says 172.21.15.67:48126) 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 -- 172.21.15.67:0/1389414485 learned_addr learned my addr 172.21.15.67:0/1389414485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95400640 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7de357e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf94a00640 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de44330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 -- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 msgr2=0x55ad7de44330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de44330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 -- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 msgr2=0x55ad7de357e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7de357e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 -- 172.21.15.67:0/1389414485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad7dbc29b0 con 0x55ad7de29240 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95e00640 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de29240 0x55ad7de44870 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fcf88055a70 tx=0x7fcf88001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=896cf098f89e2fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:34.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 1 -- 172.21.15.67:0/1389414485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf880556d0 con 0x55ad7de29240 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 1 -- 172.21.15.67:0/1389414485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fcf88079c70 con 0x55ad7de29240 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf7c003620 con 0x55ad7de29240 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 1 -- 172.21.15.67:0/1389414485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf8805e510 con 0x55ad7de29240 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 10 client.9417 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55ad7dbd50b0 con 0x55ad7de29240 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 1 -- 172.21.15.67:0/1389414485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c003620 con 0x55ad7de29240 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 1 -- 172.21.15.67:0/1389414485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fcf8806db90 con 0x55ad7de29240 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 10 client.9417 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 10 client.9417 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 20 client.9417 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7de44db0 0x55ad7de894c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 4 client.9417 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ad7dc3b2e0 con 0x55ad7de44db0 2024-04-17T10:34:34.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95400640 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7de44db0 0x55ad7de894c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf95400640 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7de44db0 0x55ad7de894c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b3cddc9051861e1 server_cookie=1687d8850710fe35 in_seq=0 out_seq=0 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 10 client.9417 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 1 -- 172.21.15.67:0/1389414485 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55ad7dc3b2e0 con 0x55ad7de44db0 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 10 client.9417 handle_command_reply: tid=0 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 client.9417 shutdown 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de29240 msgr2=0x55ad7de44870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de29240 0x55ad7de44870 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fcf88055a70 tx=0x7fcf88001d00 comp rx=0 tx=0).stop 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 shutdown_connections 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ad7de44db0 0x55ad7de894c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ad7de29240 0x55ad7de44870 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ad7de28920 0x55ad7de44330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 --2- 172.21.15.67:0/1389414485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ad7dddef30 0x55ad7de357e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf76a00640 0 client.9417 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:34.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 >> 172.21.15.67:0/1389414485 conn(0x55ad7dde52a0 msgr2=0x55ad7dde1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 shutdown_connections 2024-04-17T10:34:34.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 1 -- 172.21.15.67:0/1389414485 wait complete. 2024-04-17T10:34:34.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.889+0000 7fcf9b2eb000 20 client.9417 trim_cache size 0 max 16384 2024-04-17T10:34:34.886 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:34.886 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 9270, 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 5.9325329760000001, 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 90, 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 130, 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 130, 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 11.797581418 2024-04-17T10:34:34.887 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf70066f60 msgr2=0x7fcf70069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf70066f60 0x7fcf70069420 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fcf84009be0 tx=0x7fcf84009190 comp rx=0 tx=0).stop 2024-04-17T10:34:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf90165710 msgr2=0x7fcf90165b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:34.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf90165710 0x7fcf90165b90 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fcf88002240 tx=0x7fcf88007b10 comp rx=0 tx=0).stop 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 shutdown_connections 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf70066f60 0x7fcf70069420 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf90165710 0x7fcf90165b90 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf901605f0 0x7fcf90160a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 --2- 172.21.15.67:0/3485590725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf78004810 0x7fcf9015a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 >> 172.21.15.67:0/3485590725 conn(0x7fcf9006ac90 msgr2=0x7fcf900b3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 shutdown_connections 2024-04-17T10:34:34.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:34.893+0000 7fcf97e00640 1 -- 172.21.15.67:0/3485590725 wait complete. 2024-04-17T10:34:34.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:34:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc915400640 1 Processor -- start 2024-04-17T10:34:35.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc915400640 1 -- start start 2024-04-17T10:34:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc915400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc910145040 0x7fc91006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc915400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc91006ef10 0x7fc910071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:35.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc915400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91006e2f0 con 0x7fc910071850 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc915400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9100a86c0 con 0x7fc91006ef10 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc915400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91015bf10 con 0x7fc910145040 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc905e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc91006ef10 0x7fc910071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc905e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc91006ef10 0x7fc910071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48134/0 (socket says 172.21.15.67:48134) 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc905e00640 1 -- 172.21.15.67:0/1865932609 learned_addr learned my addr 172.21.15.67:0/1865932609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc90e800640 1 --2- 172.21.15.67:0/1865932609 >> v2:172.21.15.86:3300/0 conn(0x7fc910145040 0x7fc91006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc905e00640 1 -- 172.21.15.67:0/1865932609 >> v2:172.21.15.86:3300/0 conn(0x7fc910145040 msgr2=0x7fc91006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc905e00640 1 --2- 172.21.15.67:0/1865932609 >> v2:172.21.15.86:3300/0 conn(0x7fc910145040 0x7fc91006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc905e00640 1 -- 172.21.15.67:0/1865932609 >> v1:172.21.15.86:6789/0 conn(0x7fc910071850 legacy=0x7fc91015f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:35.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.053+0000 7fc905e00640 1 -- 172.21.15.67:0/1865932609 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc910071c60 con 0x7fc91006ef10 2024-04-17T10:34:35.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc905e00640 1 --2- 172.21.15.67:0/1865932609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc91006ef10 0x7fc910071310 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc910072410 tx=0x7fc8f801f1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff45a95eefd26a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc90de00640 1 -- 172.21.15.67:0/1865932609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80b4070 con 0x7fc91006ef10 2024-04-17T10:34:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc90de00640 1 -- 172.21.15.67:0/1865932609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f80af400 con 0x7fc91006ef10 2024-04-17T10:34:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc90de00640 1 -- 172.21.15.67:0/1865932609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80af7d0 con 0x7fc91006ef10 2024-04-17T10:34:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 -- 172.21.15.67:0/1865932609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc91006ef10 msgr2=0x7fc910071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:35.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 --2- 172.21.15.67:0/1865932609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc91006ef10 0x7fc910071310 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc910072410 tx=0x7fc8f801f1a0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 -- 172.21.15.67:0/1865932609 shutdown_connections 2024-04-17T10:34:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 --2- 172.21.15.67:0/1865932609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc91006ef10 0x7fc910071310 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 --2- 172.21.15.67:0/1865932609 >> v2:172.21.15.86:3300/0 conn(0x7fc910145040 0x7fc91006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 -- 172.21.15.67:0/1865932609 >> 172.21.15.67:0/1865932609 conn(0x7fc91006ac90 msgr2=0x7fc91006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 -- 172.21.15.67:0/1865932609 shutdown_connections 2024-04-17T10:34:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.057+0000 7fc915400640 1 -- 172.21.15.67:0/1865932609 wait complete. 2024-04-17T10:34:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 Processor -- start 2024-04-17T10:34:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 -- start start 2024-04-17T10:34:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc91006ef10 0x7fc910141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc910071850 0x7fc91015e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:35.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc910145040 0x7fc910137c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91006df80 con 0x7fc910145040 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91006dca0 con 0x7fc91006ef10 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91006d370 con 0x7fc910071850 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc910145040 0x7fc910137c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc910145040 0x7fc910137c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48140/0 (socket says 172.21.15.67:48140) 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 -- 172.21.15.67:0/1420798235 learned_addr learned my addr 172.21.15.67:0/1420798235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90e800640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc91006ef10 0x7fc910141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc905e00640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc910071850 0x7fc91015e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 -- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc910071850 msgr2=0x7fc91015e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc910071850 0x7fc91015e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 -- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc91006ef10 msgr2=0x7fc910141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc91006ef10 0x7fc910141c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 -- 172.21.15.67:0/1420798235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9101429a0 con 0x7fc910145040 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc90f200640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc910145040 0x7fc910137c90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c3f0 tx=0x7fc90800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc907400640 1 -- 172.21.15.67:0/1420798235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908011790 con 0x7fc910145040 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc907400640 1 -- 172.21.15.67:0/1420798235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc908017070 con 0x7fc910145040 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc907400640 1 -- 172.21.15.67:0/1420798235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc908011b70 con 0x7fc910145040 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc910142c80 con 0x7fc910145040 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc910143240 con 0x7fc910145040 2024-04-17T10:34:35.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.061+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc910072d50 con 0x7fc910145040 2024-04-17T10:34:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.065+0000 7fc907400640 1 -- 172.21.15.67:0/1420798235 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc908011d10 con 0x7fc910145040 2024-04-17T10:34:35.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.065+0000 7fc907400640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc8ec066f60 0x7fc8ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.065+0000 7fc907400640 1 -- 172.21.15.67:0/1420798235 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc908084ef0 con 0x7fc910145040 2024-04-17T10:34:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.065+0000 7fc90e800640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc8ec066f60 0x7fc8ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:35.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.065+0000 7fc90e800640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc8ec066f60 0x7fc8ec069420 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc9000079c0 tx=0x7fc900008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:35.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.069+0000 7fc907400640 1 -- 172.21.15.67:0/1420798235 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc908055680 con 0x7fc910145040 2024-04-17T10:34:35.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc9100cba10 con 0x7fc910145040 2024-04-17T10:34:35.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc907400640 1 -- 172.21.15.67:0/1420798235 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v90) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc908059720 con 0x7fc910145040 2024-04-17T10:34:35.261 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc8ec066f60 msgr2=0x7fc8ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc8ec066f60 0x7fc8ec069420 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fc9000079c0 tx=0x7fc900008040 comp rx=0 tx=0).stop 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc910145040 msgr2=0x7fc910137c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc910145040 0x7fc910137c90 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c3f0 tx=0x7fc90800c8b0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 shutdown_connections 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc8ec066f60 0x7fc8ec069420 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc910145040 0x7fc910137c90 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc910071850 0x7fc91015e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 --2- 172.21.15.67:0/1420798235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc91006ef10 0x7fc910141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.269+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 >> 172.21.15.67:0/1420798235 conn(0x7fc91006ac90 msgr2=0x7fc9100b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.273+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 shutdown_connections 2024-04-17T10:34:35.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:35.273+0000 7fc915400640 1 -- 172.21.15.67:0/1420798235 wait complete. 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:34:35.282 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:34:35.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.283 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:34:35.291 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:34:35.292 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41871sec preferred_lft 41871sec 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.293 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:34:35.294 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:34:35.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.295 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:34:35.356 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:34:35.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.358 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:34:35.418 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:34:35.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.419 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:34:35.419 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:35.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:34:35.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:34:35.419 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:34:35.419 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:35.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.508 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:34:35.516 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:34:35.516 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:34:35.517 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:34:35.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.517 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:34:35.578 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:34:35.578 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:34:35.630 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:34:35.630 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:34:35.631 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:34:35.681 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:34:35.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:34:35.775 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:34:35.775 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:34:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:34:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:34:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:34:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:34:35.788 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:34:35.791 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:34:35.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:34:35.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:35.812 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:34:35.873 INFO:teuthology.orchestra.run.smithi067.stdout:client7739 2024-04-17T10:34:35.873 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:34:35.873 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:34:35.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:36.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.112+0000 7f378de00640 1 Processor -- start 2024-04-17T10:34:36.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.112+0000 7f378de00640 1 -- start start 2024-04-17T10:34:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f378de00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f37880cf0c0 0x7f37881441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f378de00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f378815ae40 con 0x7f378806e6e0 2024-04-17T10:34:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f378de00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3788071440 con 0x7f378806eac0 2024-04-17T10:34:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f378de00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3788071720 con 0x7f37880cf0c0 2024-04-17T10:34:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f378806e6e0 0x7f37880ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37800/0 (socket says 172.21.15.67:37800) 2024-04-17T10:34:36.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786a00640 1 -- 172.21.15.67:0/3029243440 learned_addr learned my addr 172.21.15.67:0/3029243440 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3787400640 1 --2- 172.21.15.67:0/3029243440 >> v2:172.21.15.86:3300/0 conn(0x7f37880cf0c0 0x7f37881441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263674665 0 0) 0x7f378815ae40 con 0x7f378806e6e0 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f376c003530 con 0x7f378806e6e0 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4188895938 0 0) 0x7f376c003530 con 0x7f378806e6e0 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 >> v2:172.21.15.86:3300/0 conn(0x7f37880cf0c0 msgr2=0x7f37881441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 --2- 172.21.15.67:0/3029243440 >> v2:172.21.15.86:3300/0 conn(0x7f37880cf0c0 0x7f37881441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 >> v1:172.21.15.86:6789/0 conn(0x7f378806eac0 legacy=0x7f3788143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3788071a00 con 0x7f378806e6e0 2024-04-17T10:34:36.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f3778001680 con 0x7f378806e6e0 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 >> v1:172.21.15.67:6789/0 conn(0x7f378806e6e0 legacy=0x7f37880ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f376c003a10 0x7f376c003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f376c004810 0x7f376c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c005220 0x7f376c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3788071720 con 0x7f376c004810 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c00c170 con 0x7f376c005220 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c00c370 con 0x7f376c003a10 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f377e000640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f376c004810 0x7f376c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3786a00640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f376c003a10 0x7f376c003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3787400640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c005220 0x7f376c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f377e000640 1 -- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f376c003a10 msgr2=0x7f376c003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f377e000640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f376c003a10 0x7f376c003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f377e000640 1 -- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c005220 msgr2=0x7f376c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f377e000640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c005220 0x7f376c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f377e000640 1 -- 172.21.15.67:0/3029243440 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c003530 con 0x7f376c004810 2024-04-17T10:34:36.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.116+0000 7f3787400640 1 -- 172.21.15.67:0/3029243440 reap_dead start 2024-04-17T10:34:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f377e000640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f376c004810 0x7f376c004c90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f3770001750 tx=0x7f377001c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b2f2fc0e128e52 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700013a0 con 0x7f376c004810 2024-04-17T10:34:36.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37700052f0 con 0x7f376c004810 2024-04-17T10:34:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f3786000640 1 -- 172.21.15.67:0/3029243440 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3770003840 con 0x7f376c004810 2024-04-17T10:34:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f378de00640 1 -- 172.21.15.67:0/3029243440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f376c004810 msgr2=0x7f376c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f378de00640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f376c004810 0x7f376c004c90 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f3770001750 tx=0x7f377001c390 comp rx=0 tx=0).stop 2024-04-17T10:34:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f378de00640 1 -- 172.21.15.67:0/3029243440 shutdown_connections 2024-04-17T10:34:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f378de00640 1 --2- 172.21.15.67:0/3029243440 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f376c004810 0x7f376c004c90 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f378de00640 1 -- 172.21.15.67:0/3029243440 >> 172.21.15.67:0/3029243440 conn(0x7f378806ac90 msgr2=0x7f378806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f378de00640 1 -- 172.21.15.67:0/3029243440 shutdown_connections 2024-04-17T10:34:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.120+0000 7f378de00640 1 -- 172.21.15.67:0/3029243440 wait complete. 2024-04-17T10:34:36.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 Processor -- start 2024-04-17T10:34:36.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 -- start start 2024-04-17T10:34:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c004810 0x7f3788159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37880b0820 0x7f37880b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37881f8300 0x7f37881f8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3788144470 con 0x7f37881f8300 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37881442f0 con 0x7f376c004810 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378815e3e0 con 0x7f37880b0820 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37881f8300 0x7f37881f8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37881f8300 0x7f37881f8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48152/0 (socket says 172.21.15.67:48152) 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 -- 172.21.15.67:0/3705992891 learned_addr learned my addr 172.21.15.67:0/3705992891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f377e000640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37880b0820 0x7f37880b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3786a00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c004810 0x7f3788159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 -- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37880b0820 msgr2=0x7f37880b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37880b0820 0x7f37880b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 -- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c004810 msgr2=0x7f3788159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c004810 0x7f3788159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 -- 172.21.15.67:0/3705992891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37881f9010 con 0x7f37881f8300 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f3787400640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37881f8300 0x7f37881f8780 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f378806ef50 tx=0x7f378000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f377fe00640 1 -- 172.21.15.67:0/3705992891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378000fd80 con 0x7f37881f8300 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f377fe00640 1 -- 172.21.15.67:0/3705992891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3780015430 con 0x7f37881f8300 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f377fe00640 1 -- 172.21.15.67:0/3705992891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780015820 con 0x7f37881f8300 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37881f6050 con 0x7f37881f8300 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378815a3a0 con 0x7f37881f8300 2024-04-17T10:34:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.124+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3788072d50 con 0x7f37881f8300 2024-04-17T10:34:36.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.128+0000 7f377fe00640 1 -- 172.21.15.67:0/3705992891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f37800159c0 con 0x7f37881f8300 2024-04-17T10:34:36.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.128+0000 7f377fe00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3764066f90 0x7f3764069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.128+0000 7f377fe00640 1 -- 172.21.15.67:0/3705992891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3780085ce0 con 0x7f37881f8300 2024-04-17T10:34:36.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.128+0000 7f3786a00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3764066f90 0x7f3764069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.128+0000 7f3786a00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3764066f90 0x7f3764069450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3778001060 tx=0x7f3778000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:36.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.132+0000 7f377fe00640 1 -- 172.21.15.67:0/3705992891 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3780024070 con 0x7f37881f8300 2024-04-17T10:34:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.328+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3788141630 con 0x7f37881f8300 2024-04-17T10:34:36.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.328+0000 7f377fe00640 1 -- 172.21.15.67:0/3705992891 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f3780019030 con 0x7f37881f8300 2024-04-17T10:34:36.322 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:36.322 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:34:36.323 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:34:36.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3764066f90 msgr2=0x7f3764069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3764066f90 0x7f3764069450 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f3778001060 tx=0x7f3778000f90 comp rx=0 tx=0).stop 2024-04-17T10:34:36.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37881f8300 msgr2=0x7f37881f8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37881f8300 0x7f37881f8780 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f378806ef50 tx=0x7f378000f580 comp rx=0 tx=0).stop 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 shutdown_connections 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3764066f90 0x7f3764069450 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37881f8300 0x7f37881f8780 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f37880b0820 0x7f37880b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 --2- 172.21.15.67:0/3705992891 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f376c004810 0x7f3788159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 >> 172.21.15.67:0/3705992891 conn(0x7f378806ac90 msgr2=0x7f378815d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 shutdown_connections 2024-04-17T10:34:36.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.332+0000 7f378de00640 1 -- 172.21.15.67:0/3705992891 wait complete. 2024-04-17T10:34:36.344 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:36.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91ac00640 1 Processor -- start 2024-04-17T10:34:36.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91ac00640 1 -- start start 2024-04-17T10:34:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91ac00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb91415f500 0x7fb91415f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91ac00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91415fe20 con 0x7fb91415cbd0 2024-04-17T10:34:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91ac00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb914099c60 con 0x7fb9140ccc40 2024-04-17T10:34:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91ac00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb914099f40 con 0x7fb91415f500 2024-04-17T10:34:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91a200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb91415cbd0 0x7fb9140cf900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37804/0 (socket says 172.21.15.67:37804) 2024-04-17T10:34:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb91a200640 1 -- 172.21.15.67:0/1660098672 learned_addr learned my addr 172.21.15.67:0/1660098672 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:36.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb919800640 1 --2- 172.21.15.67:0/1660098672 >> v2:172.21.15.86:3300/0 conn(0x7fb91415f500 0x7fb91415f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb913e00640 1 -- 172.21.15.67:0/1660098672 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560601826 0 0) 0x7fb91415fe20 con 0x7fb91415cbd0 2024-04-17T10:34:36.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.492+0000 7fb913e00640 1 -- 172.21.15.67:0/1660098672 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8fc003530 con 0x7fb91415cbd0 2024-04-17T10:34:36.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb919800640 1 -- 172.21.15.67:0/1660098672 >> v1:172.21.15.86:6789/0 conn(0x7fb9140ccc40 legacy=0x7fb9140cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb919800640 1 -- 172.21.15.67:0/1660098672 >> v1:172.21.15.67:6789/0 conn(0x7fb91415cbd0 legacy=0x7fb9140cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb919800640 1 -- 172.21.15.67:0/1660098672 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91409a220 con 0x7fb91415f500 2024-04-17T10:34:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb919800640 1 --2- 172.21.15.67:0/1660098672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415f500 0x7fb91415f8e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb9080097c0 tx=0x7fb90801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ec62b714d850b1f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb913e00640 1 -- 172.21.15.67:0/1660098672 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9080b1070 con 0x7fb91415f500 2024-04-17T10:34:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb913e00640 1 -- 172.21.15.67:0/1660098672 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9080ac5f0 con 0x7fb91415f500 2024-04-17T10:34:36.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb913e00640 1 -- 172.21.15.67:0/1660098672 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9080ac9c0 con 0x7fb91415f500 2024-04-17T10:34:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb91ac00640 1 -- 172.21.15.67:0/1660098672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415f500 msgr2=0x7fb91415f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb91ac00640 1 --2- 172.21.15.67:0/1660098672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415f500 0x7fb91415f8e0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fb9080097c0 tx=0x7fb90801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb91ac00640 1 -- 172.21.15.67:0/1660098672 shutdown_connections 2024-04-17T10:34:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb91ac00640 1 --2- 172.21.15.67:0/1660098672 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415f500 0x7fb91415f8e0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.496+0000 7fb91ac00640 1 -- 172.21.15.67:0/1660098672 >> 172.21.15.67:0/1660098672 conn(0x7fb9140d1620 msgr2=0x7fb9140d1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 -- 172.21.15.67:0/1660098672 shutdown_connections 2024-04-17T10:34:36.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 -- 172.21.15.67:0/1660098672 wait complete. 2024-04-17T10:34:36.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 Processor -- start 2024-04-17T10:34:36.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 -- start start 2024-04-17T10:34:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9140ccc40 0x7fb9140b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415cbd0 0x7fb914139050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415f500 0x7fb914139590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91416a060 con 0x7fb91415f500 2024-04-17T10:34:36.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9140cd430 con 0x7fb91415cbd0 2024-04-17T10:34:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91ac00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91416a1e0 con 0x7fb9140ccc40 2024-04-17T10:34:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415f500 0x7fb914139590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415f500 0x7fb914139590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48154/0 (socket says 172.21.15.67:48154) 2024-04-17T10:34:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 -- 172.21.15.67:0/3051351237 learned_addr learned my addr 172.21.15.67:0/3051351237 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb918e00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415cbd0 0x7fb914139050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb919800640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9140ccc40 0x7fb9140b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 -- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9140ccc40 msgr2=0x7fb9140b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9140ccc40 0x7fb9140b6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 -- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415cbd0 msgr2=0x7fb914139050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415cbd0 0x7fb914139050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 -- 172.21.15.67:0/3051351237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140b68c0 con 0x7fb91415f500 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb919800640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9140ccc40 0x7fb9140b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb918e00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415cbd0 0x7fb914139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb91a200640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415f500 0x7fb914139590 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00ca70 tx=0x7fb90c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.500+0000 7fb911600640 1 -- 172.21.15.67:0/3051351237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c017070 con 0x7fb91415f500 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb911600640 1 -- 172.21.15.67:0/3051351237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c00ab70 con 0x7fb91415f500 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb911600640 1 -- 172.21.15.67:0/3051351237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c012400 con 0x7fb91415f500 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9140b7120 con 0x7fb91415f500 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9140b6e40 con 0x7fb91415f500 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb914074dc0 con 0x7fb91415f500 2024-04-17T10:34:36.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb911600640 1 -- 172.21.15.67:0/3051351237 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb90c01a070 con 0x7fb91415f500 2024-04-17T10:34:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb911600640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8f4066f60 0x7fb8f4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb911600640 1 -- 172.21.15.67:0/3051351237 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fb90c084d50 con 0x7fb91415f500 2024-04-17T10:34:36.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.504+0000 7fb919800640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8f4066f60 0x7fb8f4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:36.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.508+0000 7fb919800640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8f4066f60 0x7fb8f4069420 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb9080ae000 tx=0x7fb9080a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.508+0000 7fb911600640 1 -- 172.21.15.67:0/3051351237 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb90c055570 con 0x7fb91415f500 2024-04-17T10:34:36.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.704+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9140ce990 con 0x7fb91415f500 2024-04-17T10:34:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.704+0000 7fb911600640 1 -- 172.21.15.67:0/3051351237 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7fb90c059610 con 0x7fb91415f500 2024-04-17T10:34:36.698 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:36.698 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:34:36.698 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8f4066f60 msgr2=0x7fb8f4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8f4066f60 0x7fb8f4069420 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fb9080ae000 tx=0x7fb9080a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415f500 msgr2=0x7fb914139590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415f500 0x7fb914139590 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fb90c00ca70 tx=0x7fb90c00cf30 comp rx=0 tx=0).stop 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 shutdown_connections 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8f4066f60 0x7fb8f4069420 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb91415f500 0x7fb914139590 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb91415cbd0 0x7fb914139050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 --2- 172.21.15.67:0/3051351237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb9140ccc40 0x7fb9140b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 >> 172.21.15.67:0/3051351237 conn(0x7fb9140d1620 msgr2=0x7fb9140cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 shutdown_connections 2024-04-17T10:34:36.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:36.708+0000 7fb91ac00640 1 -- 172.21.15.67:0/3051351237 wait complete. 2024-04-17T10:34:36.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:36.718 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:36.719 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:36.719 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.0 2024-04-17T10:34:36.719 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:36.719 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:36.719 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:36.719 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.763 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189169 s, 55.4 MB/s 2024-04-17T10:34:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181802 s, 57.7 MB/s 2024-04-17T10:34:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149271 s, 70.2 MB/s 2024-04-17T10:34:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146263 s, 71.7 MB/s 2024-04-17T10:34:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.830 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012454 s, 84.2 MB/s 2024-04-17T10:34:36.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.850 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179047 s, 58.6 MB/s 2024-04-17T10:34:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.868 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158214 s, 66.3 MB/s 2024-04-17T10:34:36.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.888 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184554 s, 56.8 MB/s 2024-04-17T10:34:36.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.905 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144716 s, 72.5 MB/s 2024-04-17T10:34:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144733 s, 72.4 MB/s 2024-04-17T10:34:36.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.937 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149628 s, 70.1 MB/s 2024-04-17T10:34:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161362 s, 65.0 MB/s 2024-04-17T10:34:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.973 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154461 s, 67.9 MB/s 2024-04-17T10:34:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:36.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:36.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197734 s, 53.0 MB/s 2024-04-17T10:34:37.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.010 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138913 s, 75.5 MB/s 2024-04-17T10:34:37.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154592 s, 67.8 MB/s 2024-04-17T10:34:37.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164827 s, 63.6 MB/s 2024-04-17T10:34:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.062 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143151 s, 73.2 MB/s 2024-04-17T10:34:37.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015123 s, 69.3 MB/s 2024-04-17T10:34:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153365 s, 68.4 MB/s 2024-04-17T10:34:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.113 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152723 s, 68.7 MB/s 2024-04-17T10:34:37.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.129 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141255 s, 74.2 MB/s 2024-04-17T10:34:37.145 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.146 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014856 s, 70.6 MB/s 2024-04-17T10:34:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149476 s, 70.1 MB/s 2024-04-17T10:34:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.178 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136905 s, 76.6 MB/s 2024-04-17T10:34:37.194 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.194 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.194 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149848 s, 70.0 MB/s 2024-04-17T10:34:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.211 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143674 s, 73.0 MB/s 2024-04-17T10:34:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203947 s, 51.4 MB/s 2024-04-17T10:34:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143086 s, 73.3 MB/s 2024-04-17T10:34:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.268 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166343 s, 63.0 MB/s 2024-04-17T10:34:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143629 s, 73.0 MB/s 2024-04-17T10:34:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.303 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017357 s, 60.4 MB/s 2024-04-17T10:34:37.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:37.305 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:37.305 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:37.305 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.1 2024-04-17T10:34:37.305 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:37.305 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:37.305 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:37.305 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.375 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141049 s, 74.3 MB/s 2024-04-17T10:34:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.393 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164304 s, 63.8 MB/s 2024-04-17T10:34:37.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.410 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015766 s, 66.5 MB/s 2024-04-17T10:34:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136869 s, 76.6 MB/s 2024-04-17T10:34:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136496 s, 76.8 MB/s 2024-04-17T10:34:37.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.459 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151363 s, 69.3 MB/s 2024-04-17T10:34:37.475 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.475 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.475 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149914 s, 69.9 MB/s 2024-04-17T10:34:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154635 s, 67.8 MB/s 2024-04-17T10:34:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014704 s, 71.3 MB/s 2024-04-17T10:34:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.527 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015557 s, 67.4 MB/s 2024-04-17T10:34:37.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.543 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143255 s, 73.2 MB/s 2024-04-17T10:34:37.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014554 s, 72.0 MB/s 2024-04-17T10:34:37.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.575 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143266 s, 73.2 MB/s 2024-04-17T10:34:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.591 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144936 s, 72.3 MB/s 2024-04-17T10:34:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151941 s, 69.0 MB/s 2024-04-17T10:34:37.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.623 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136578 s, 76.8 MB/s 2024-04-17T10:34:37.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141736 s, 74.0 MB/s 2024-04-17T10:34:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.655 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135785 s, 77.2 MB/s 2024-04-17T10:34:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147086 s, 71.3 MB/s 2024-04-17T10:34:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.688 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151312 s, 69.3 MB/s 2024-04-17T10:34:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.705 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149392 s, 70.2 MB/s 2024-04-17T10:34:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151548 s, 69.2 MB/s 2024-04-17T10:34:37.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.738 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142952 s, 73.4 MB/s 2024-04-17T10:34:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.754 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141257 s, 74.2 MB/s 2024-04-17T10:34:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.770 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.770 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142741 s, 73.5 MB/s 2024-04-17T10:34:37.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.786 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144959 s, 72.3 MB/s 2024-04-17T10:34:37.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.803 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157993 s, 66.4 MB/s 2024-04-17T10:34:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152198 s, 68.9 MB/s 2024-04-17T10:34:37.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.836 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139484 s, 75.2 MB/s 2024-04-17T10:34:37.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.854 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162092 s, 64.7 MB/s 2024-04-17T10:34:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139623 s, 75.1 MB/s 2024-04-17T10:34:37.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149011 s, 70.4 MB/s 2024-04-17T10:34:37.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:37.888 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:37.888 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:37.888 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.2 2024-04-17T10:34:37.888 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:37.888 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:37.888 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:37.888 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.963 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140215 s, 74.8 MB/s 2024-04-17T10:34:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153373 s, 68.4 MB/s 2024-04-17T10:34:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:37.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137606 s, 76.2 MB/s 2024-04-17T10:34:38.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01473 s, 71.2 MB/s 2024-04-17T10:34:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.028 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147773 s, 71.0 MB/s 2024-04-17T10:34:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.043 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135683 s, 77.3 MB/s 2024-04-17T10:34:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.060 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149566 s, 70.1 MB/s 2024-04-17T10:34:38.077 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158443 s, 66.2 MB/s 2024-04-17T10:34:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148298 s, 70.7 MB/s 2024-04-17T10:34:38.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.113 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173917 s, 60.3 MB/s 2024-04-17T10:34:38.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.128 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133041 s, 78.8 MB/s 2024-04-17T10:34:38.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.144 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138794 s, 75.5 MB/s 2024-04-17T10:34:38.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.161 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149306 s, 70.2 MB/s 2024-04-17T10:34:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136417 s, 76.9 MB/s 2024-04-17T10:34:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150894 s, 69.5 MB/s 2024-04-17T10:34:38.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.209 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142254 s, 73.7 MB/s 2024-04-17T10:34:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015337 s, 68.4 MB/s 2024-04-17T10:34:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142034 s, 73.8 MB/s 2024-04-17T10:34:38.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.259 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144969 s, 72.3 MB/s 2024-04-17T10:34:38.275 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.275 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.275 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149036 s, 70.4 MB/s 2024-04-17T10:34:38.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.291 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138337 s, 75.8 MB/s 2024-04-17T10:34:38.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136355 s, 76.9 MB/s 2024-04-17T10:34:38.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.325 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165492 s, 63.4 MB/s 2024-04-17T10:34:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.344 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179059 s, 58.6 MB/s 2024-04-17T10:34:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135053 s, 77.6 MB/s 2024-04-17T10:34:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140062 s, 74.9 MB/s 2024-04-17T10:34:38.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148287 s, 70.7 MB/s 2024-04-17T10:34:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145622 s, 72.0 MB/s 2024-04-17T10:34:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.423 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134156 s, 78.2 MB/s 2024-04-17T10:34:38.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.440 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147028 s, 71.3 MB/s 2024-04-17T10:34:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148034 s, 70.8 MB/s 2024-04-17T10:34:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138791 s, 75.6 MB/s 2024-04-17T10:34:38.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:38.473 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:38.473 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:38.473 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.3 2024-04-17T10:34:38.473 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:38.474 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:38.474 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:38.474 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.546 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130819 s, 80.2 MB/s 2024-04-17T10:34:38.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.562 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144031 s, 72.8 MB/s 2024-04-17T10:34:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.578 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138997 s, 75.4 MB/s 2024-04-17T10:34:38.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.593 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132123 s, 79.4 MB/s 2024-04-17T10:34:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018793 s, 55.8 MB/s 2024-04-17T10:34:38.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.629 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137526 s, 76.2 MB/s 2024-04-17T10:34:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.646 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148125 s, 70.8 MB/s 2024-04-17T10:34:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.667 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189686 s, 55.3 MB/s 2024-04-17T10:34:38.682 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.682 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.683 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140466 s, 74.6 MB/s 2024-04-17T10:34:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.699 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146135 s, 71.8 MB/s 2024-04-17T10:34:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135921 s, 77.1 MB/s 2024-04-17T10:34:38.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.730 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.730 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139877 s, 75.0 MB/s 2024-04-17T10:34:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.751 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018863 s, 55.6 MB/s 2024-04-17T10:34:38.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.768 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014913 s, 70.3 MB/s 2024-04-17T10:34:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139162 s, 75.3 MB/s 2024-04-17T10:34:38.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.799 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135164 s, 77.6 MB/s 2024-04-17T10:34:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.815 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145425 s, 72.1 MB/s 2024-04-17T10:34:38.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.833 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164495 s, 63.7 MB/s 2024-04-17T10:34:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.849 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143023 s, 73.3 MB/s 2024-04-17T10:34:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.865 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136239 s, 77.0 MB/s 2024-04-17T10:34:38.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.881 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142149 s, 73.8 MB/s 2024-04-17T10:34:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.895 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126704 s, 82.8 MB/s 2024-04-17T10:34:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013952 s, 75.2 MB/s 2024-04-17T10:34:38.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.928 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149697 s, 70.0 MB/s 2024-04-17T10:34:38.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192843 s, 54.4 MB/s 2024-04-17T10:34:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.966 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150673 s, 69.6 MB/s 2024-04-17T10:34:38.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.982 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140275 s, 74.8 MB/s 2024-04-17T10:34:38.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:38.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:38.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147056 s, 71.3 MB/s 2024-04-17T10:34:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150406 s, 69.7 MB/s 2024-04-17T10:34:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154526 s, 67.9 MB/s 2024-04-17T10:34:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.048 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137996 s, 76.0 MB/s 2024-04-17T10:34:39.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.065 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153436 s, 68.3 MB/s 2024-04-17T10:34:39.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:39.066 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:39.066 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:39.066 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.4 2024-04-17T10:34:39.067 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:39.067 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:39.067 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:39.067 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:39.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.139 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138875 s, 75.5 MB/s 2024-04-17T10:34:39.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131623 s, 79.7 MB/s 2024-04-17T10:34:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172028 s, 61.0 MB/s 2024-04-17T10:34:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135564 s, 77.3 MB/s 2024-04-17T10:34:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157086 s, 66.8 MB/s 2024-04-17T10:34:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151424 s, 69.2 MB/s 2024-04-17T10:34:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.237 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.238 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133807 s, 78.4 MB/s 2024-04-17T10:34:39.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.254 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138928 s, 75.5 MB/s 2024-04-17T10:34:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134173 s, 78.2 MB/s 2024-04-17T10:34:39.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150954 s, 69.5 MB/s 2024-04-17T10:34:39.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.300 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131167 s, 79.9 MB/s 2024-04-17T10:34:39.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131408 s, 79.8 MB/s 2024-04-17T10:34:39.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.333 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154429 s, 67.9 MB/s 2024-04-17T10:34:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.353 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184223 s, 56.9 MB/s 2024-04-17T10:34:39.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.370 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015201 s, 69.0 MB/s 2024-04-17T10:34:39.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.386 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143031 s, 73.3 MB/s 2024-04-17T10:34:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013801 s, 76.0 MB/s 2024-04-17T10:34:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.418 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146704 s, 71.5 MB/s 2024-04-17T10:34:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.438 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185377 s, 56.6 MB/s 2024-04-17T10:34:39.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.454 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135805 s, 77.2 MB/s 2024-04-17T10:34:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.473 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180291 s, 58.2 MB/s 2024-04-17T10:34:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134474 s, 78.0 MB/s 2024-04-17T10:34:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.506 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150454 s, 69.7 MB/s 2024-04-17T10:34:39.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.523 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149268 s, 70.2 MB/s 2024-04-17T10:34:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.538 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.539 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014412 s, 72.8 MB/s 2024-04-17T10:34:39.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132925 s, 78.9 MB/s 2024-04-17T10:34:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.570 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143167 s, 73.2 MB/s 2024-04-17T10:34:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.586 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145032 s, 72.3 MB/s 2024-04-17T10:34:39.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.602 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135416 s, 77.4 MB/s 2024-04-17T10:34:39.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.617 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137708 s, 76.1 MB/s 2024-04-17T10:34:39.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.633 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140991 s, 74.4 MB/s 2024-04-17T10:34:39.648 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.648 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.648 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130331 s, 80.5 MB/s 2024-04-17T10:34:39.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:39.649 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:39.649 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:39.650 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.5 2024-04-17T10:34:39.650 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:39.650 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:39.650 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:39.650 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:39.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179972 s, 58.3 MB/s 2024-04-17T10:34:39.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.742 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139421 s, 75.2 MB/s 2024-04-17T10:34:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.758 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139437 s, 75.2 MB/s 2024-04-17T10:34:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.775 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151389 s, 69.3 MB/s 2024-04-17T10:34:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.790 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131953 s, 79.5 MB/s 2024-04-17T10:34:39.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.805 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.805 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137459 s, 76.3 MB/s 2024-04-17T10:34:39.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136073 s, 77.1 MB/s 2024-04-17T10:34:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.836 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134107 s, 78.2 MB/s 2024-04-17T10:34:39.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.855 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171888 s, 61.0 MB/s 2024-04-17T10:34:39.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.872 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149101 s, 70.3 MB/s 2024-04-17T10:34:39.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.888 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014264 s, 73.5 MB/s 2024-04-17T10:34:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137696 s, 76.2 MB/s 2024-04-17T10:34:39.919 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.919 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.919 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144298 s, 72.7 MB/s 2024-04-17T10:34:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.935 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138323 s, 75.8 MB/s 2024-04-17T10:34:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.950 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135287 s, 77.5 MB/s 2024-04-17T10:34:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.966 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014682 s, 71.4 MB/s 2024-04-17T10:34:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013192 s, 79.5 MB/s 2024-04-17T10:34:39.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:39.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.000 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160972 s, 65.1 MB/s 2024-04-17T10:34:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.016 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150606 s, 69.6 MB/s 2024-04-17T10:34:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139491 s, 75.2 MB/s 2024-04-17T10:34:40.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.049 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147513 s, 71.1 MB/s 2024-04-17T10:34:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.063 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.064 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132328 s, 79.2 MB/s 2024-04-17T10:34:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150453 s, 69.7 MB/s 2024-04-17T10:34:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.098 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134059 s, 78.2 MB/s 2024-04-17T10:34:40.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.112 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146725 s, 71.5 MB/s 2024-04-17T10:34:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.128 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148667 s, 70.5 MB/s 2024-04-17T10:34:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.146 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153164 s, 68.5 MB/s 2024-04-17T10:34:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.162 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148562 s, 70.6 MB/s 2024-04-17T10:34:40.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.179 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150268 s, 69.8 MB/s 2024-04-17T10:34:40.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.195 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139557 s, 75.1 MB/s 2024-04-17T10:34:40.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.210 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134166 s, 78.2 MB/s 2024-04-17T10:34:40.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.228 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015537 s, 67.5 MB/s 2024-04-17T10:34:40.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:40.229 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:40.229 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:40.229 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.6 2024-04-17T10:34:40.229 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:40.229 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:40.229 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:40.229 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.299 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139917 s, 74.9 MB/s 2024-04-17T10:34:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.317 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.317 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156603 s, 67.0 MB/s 2024-04-17T10:34:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.333 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146654 s, 71.5 MB/s 2024-04-17T10:34:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.348 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129654 s, 80.9 MB/s 2024-04-17T10:34:40.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.365 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153364 s, 68.4 MB/s 2024-04-17T10:34:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.380 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141713 s, 74.0 MB/s 2024-04-17T10:34:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.396 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013973 s, 75.0 MB/s 2024-04-17T10:34:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.412 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144415 s, 72.6 MB/s 2024-04-17T10:34:40.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.429 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146157 s, 71.7 MB/s 2024-04-17T10:34:40.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151034 s, 69.4 MB/s 2024-04-17T10:34:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.463 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160677 s, 65.3 MB/s 2024-04-17T10:34:40.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.479 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014054 s, 74.6 MB/s 2024-04-17T10:34:40.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150491 s, 69.7 MB/s 2024-04-17T10:34:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133604 s, 78.5 MB/s 2024-04-17T10:34:40.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.528 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014685 s, 71.4 MB/s 2024-04-17T10:34:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.543 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139322 s, 75.3 MB/s 2024-04-17T10:34:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183388 s, 57.2 MB/s 2024-04-17T10:34:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.579 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137769 s, 76.1 MB/s 2024-04-17T10:34:40.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.594 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133367 s, 78.6 MB/s 2024-04-17T10:34:40.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147297 s, 71.2 MB/s 2024-04-17T10:34:40.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.626 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139327 s, 75.3 MB/s 2024-04-17T10:34:40.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135359 s, 77.5 MB/s 2024-04-17T10:34:40.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.659 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155362 s, 67.5 MB/s 2024-04-17T10:34:40.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.674 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129611 s, 80.9 MB/s 2024-04-17T10:34:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144372 s, 72.6 MB/s 2024-04-17T10:34:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148411 s, 70.7 MB/s 2024-04-17T10:34:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148251 s, 70.7 MB/s 2024-04-17T10:34:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148566 s, 70.6 MB/s 2024-04-17T10:34:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.755 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137787 s, 76.1 MB/s 2024-04-17T10:34:40.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.771 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142148 s, 73.8 MB/s 2024-04-17T10:34:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.788 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148164 s, 70.8 MB/s 2024-04-17T10:34:40.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.802 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121414 s, 86.4 MB/s 2024-04-17T10:34:40.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:40.803 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:40.803 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:40.803 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.7 2024-04-17T10:34:40.803 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:40.803 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:40.803 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:40.804 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:40.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.877 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163461 s, 64.1 MB/s 2024-04-17T10:34:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.895 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157163 s, 66.7 MB/s 2024-04-17T10:34:40.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143389 s, 73.1 MB/s 2024-04-17T10:34:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.927 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147097 s, 71.3 MB/s 2024-04-17T10:34:40.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.942 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.942 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132659 s, 79.0 MB/s 2024-04-17T10:34:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.958 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134596 s, 77.9 MB/s 2024-04-17T10:34:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174613 s, 60.1 MB/s 2024-04-17T10:34:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:40.994 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148714 s, 70.5 MB/s 2024-04-17T10:34:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148496 s, 70.6 MB/s 2024-04-17T10:34:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146742 s, 71.5 MB/s 2024-04-17T10:34:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.044 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155535 s, 67.4 MB/s 2024-04-17T10:34:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.064 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181215 s, 57.9 MB/s 2024-04-17T10:34:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135053 s, 77.6 MB/s 2024-04-17T10:34:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013902 s, 75.4 MB/s 2024-04-17T10:34:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134544 s, 77.9 MB/s 2024-04-17T10:34:41.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.127 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144317 s, 72.7 MB/s 2024-04-17T10:34:41.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.142 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131398 s, 79.8 MB/s 2024-04-17T10:34:41.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.157 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133715 s, 78.4 MB/s 2024-04-17T10:34:41.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123713 s, 84.8 MB/s 2024-04-17T10:34:41.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136204 s, 77.0 MB/s 2024-04-17T10:34:41.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153416 s, 68.3 MB/s 2024-04-17T10:34:41.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.219 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142401 s, 73.6 MB/s 2024-04-17T10:34:41.234 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134691 s, 77.9 MB/s 2024-04-17T10:34:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.251 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146314 s, 71.7 MB/s 2024-04-17T10:34:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.267 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139525 s, 75.2 MB/s 2024-04-17T10:34:41.284 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.284 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.284 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150007 s, 69.9 MB/s 2024-04-17T10:34:41.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.300 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147154 s, 71.3 MB/s 2024-04-17T10:34:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.320 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181663 s, 57.7 MB/s 2024-04-17T10:34:41.336 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.337 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014565 s, 72.0 MB/s 2024-04-17T10:34:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184043 s, 57.0 MB/s 2024-04-17T10:34:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.373 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147837 s, 70.9 MB/s 2024-04-17T10:34:41.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.389 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137389 s, 76.3 MB/s 2024-04-17T10:34:41.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:41.391 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:41.391 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:41.391 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.8 2024-04-17T10:34:41.391 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:41.391 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:41.391 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:41.391 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187867 s, 55.8 MB/s 2024-04-17T10:34:41.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.483 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137847 s, 76.1 MB/s 2024-04-17T10:34:41.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.499 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144727 s, 72.5 MB/s 2024-04-17T10:34:41.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.519 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.519 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184677 s, 56.8 MB/s 2024-04-17T10:34:41.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153058 s, 68.5 MB/s 2024-04-17T10:34:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152608 s, 68.7 MB/s 2024-04-17T10:34:41.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.571 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015023 s, 69.8 MB/s 2024-04-17T10:34:41.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.586 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142298 s, 73.7 MB/s 2024-04-17T10:34:41.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.603 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153074 s, 68.5 MB/s 2024-04-17T10:34:41.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.620 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142468 s, 73.6 MB/s 2024-04-17T10:34:41.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.635 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134231 s, 78.1 MB/s 2024-04-17T10:34:41.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.650 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139888 s, 75.0 MB/s 2024-04-17T10:34:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.666 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.666 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136253 s, 77.0 MB/s 2024-04-17T10:34:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.682 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136691 s, 76.7 MB/s 2024-04-17T10:34:41.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.700 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017338 s, 60.5 MB/s 2024-04-17T10:34:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.716 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138877 s, 75.5 MB/s 2024-04-17T10:34:41.731 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.732 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134862 s, 77.8 MB/s 2024-04-17T10:34:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135332 s, 77.5 MB/s 2024-04-17T10:34:41.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.763 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142157 s, 73.8 MB/s 2024-04-17T10:34:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146083 s, 71.8 MB/s 2024-04-17T10:34:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.795 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136653 s, 76.7 MB/s 2024-04-17T10:34:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.810 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131335 s, 79.8 MB/s 2024-04-17T10:34:41.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.827 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150736 s, 69.6 MB/s 2024-04-17T10:34:41.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.843 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145894 s, 71.9 MB/s 2024-04-17T10:34:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.858 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137807 s, 76.1 MB/s 2024-04-17T10:34:41.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.879 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183757 s, 57.1 MB/s 2024-04-17T10:34:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135624 s, 77.3 MB/s 2024-04-17T10:34:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.914 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180249 s, 58.2 MB/s 2024-04-17T10:34:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130077 s, 80.6 MB/s 2024-04-17T10:34:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183481 s, 57.1 MB/s 2024-04-17T10:34:41.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.962 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.962 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114863 s, 91.3 MB/s 2024-04-17T10:34:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:41.978 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139787 s, 75.0 MB/s 2024-04-17T10:34:41.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:41.979 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:41.979 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:41.979 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.9 2024-04-17T10:34:41.980 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:41.980 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:41.980 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:41.980 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:42.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.056 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.056 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144268 s, 72.7 MB/s 2024-04-17T10:34:42.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.071 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013754 s, 76.2 MB/s 2024-04-17T10:34:42.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177699 s, 59.0 MB/s 2024-04-17T10:34:42.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.106 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014201 s, 73.8 MB/s 2024-04-17T10:34:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.125 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017273 s, 60.7 MB/s 2024-04-17T10:34:42.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.141 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138203 s, 75.9 MB/s 2024-04-17T10:34:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.161 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183812 s, 57.0 MB/s 2024-04-17T10:34:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158524 s, 66.1 MB/s 2024-04-17T10:34:42.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.199 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182249 s, 57.5 MB/s 2024-04-17T10:34:42.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.217 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159025 s, 65.9 MB/s 2024-04-17T10:34:42.231 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.231 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.232 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134163 s, 78.2 MB/s 2024-04-17T10:34:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.248 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143746 s, 72.9 MB/s 2024-04-17T10:34:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.263 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01349 s, 77.7 MB/s 2024-04-17T10:34:42.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.278 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013343 s, 78.6 MB/s 2024-04-17T10:34:42.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.294 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136729 s, 76.7 MB/s 2024-04-17T10:34:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.310 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147596 s, 71.0 MB/s 2024-04-17T10:34:42.328 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.328 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.328 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157145 s, 66.7 MB/s 2024-04-17T10:34:42.342 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.343 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013062 s, 80.3 MB/s 2024-04-17T10:34:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.359 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150349 s, 69.7 MB/s 2024-04-17T10:34:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.376 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145403 s, 72.1 MB/s 2024-04-17T10:34:42.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141956 s, 73.9 MB/s 2024-04-17T10:34:42.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146802 s, 71.4 MB/s 2024-04-17T10:34:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.423 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136757 s, 76.7 MB/s 2024-04-17T10:34:42.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.440 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145698 s, 72.0 MB/s 2024-04-17T10:34:42.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146794 s, 71.4 MB/s 2024-04-17T10:34:42.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136925 s, 76.6 MB/s 2024-04-17T10:34:42.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.488 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146705 s, 71.5 MB/s 2024-04-17T10:34:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148844 s, 70.4 MB/s 2024-04-17T10:34:42.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013886 s, 75.5 MB/s 2024-04-17T10:34:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152226 s, 68.9 MB/s 2024-04-17T10:34:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.553 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134917 s, 77.7 MB/s 2024-04-17T10:34:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143592 s, 73.0 MB/s 2024-04-17T10:34:42.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:42.570 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:42.571 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:42.571 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.10 2024-04-17T10:34:42.571 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:42.571 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:42.571 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:42.571 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138885 s, 75.5 MB/s 2024-04-17T10:34:42.659 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150912 s, 69.5 MB/s 2024-04-17T10:34:42.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.676 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146618 s, 71.5 MB/s 2024-04-17T10:34:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.692 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014976 s, 70.0 MB/s 2024-04-17T10:34:42.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.708 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134879 s, 77.7 MB/s 2024-04-17T10:34:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.724 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140241 s, 74.8 MB/s 2024-04-17T10:34:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.741 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151818 s, 69.1 MB/s 2024-04-17T10:34:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149686 s, 70.1 MB/s 2024-04-17T10:34:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138985 s, 75.4 MB/s 2024-04-17T10:34:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.789 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141182 s, 74.3 MB/s 2024-04-17T10:34:42.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.804 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137313 s, 76.4 MB/s 2024-04-17T10:34:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.820 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135713 s, 77.3 MB/s 2024-04-17T10:34:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139248 s, 75.3 MB/s 2024-04-17T10:34:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.852 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146787 s, 71.4 MB/s 2024-04-17T10:34:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.868 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141097 s, 74.3 MB/s 2024-04-17T10:34:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.883 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135398 s, 77.4 MB/s 2024-04-17T10:34:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133043 s, 78.8 MB/s 2024-04-17T10:34:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138323 s, 75.8 MB/s 2024-04-17T10:34:42.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.929 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128242 s, 81.8 MB/s 2024-04-17T10:34:42.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.944 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138334 s, 75.8 MB/s 2024-04-17T10:34:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143916 s, 72.9 MB/s 2024-04-17T10:34:42.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147929 s, 70.9 MB/s 2024-04-17T10:34:42.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:42.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:42.994 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148505 s, 70.6 MB/s 2024-04-17T10:34:43.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148951 s, 70.4 MB/s 2024-04-17T10:34:43.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.025 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127537 s, 82.2 MB/s 2024-04-17T10:34:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.041 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136139 s, 77.0 MB/s 2024-04-17T10:34:43.056 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.057 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143425 s, 73.1 MB/s 2024-04-17T10:34:43.074 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.074 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.074 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154103 s, 68.0 MB/s 2024-04-17T10:34:43.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149463 s, 70.2 MB/s 2024-04-17T10:34:43.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.107 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145039 s, 72.3 MB/s 2024-04-17T10:34:43.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.122 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129545 s, 80.9 MB/s 2024-04-17T10:34:43.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.140 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157961 s, 66.4 MB/s 2024-04-17T10:34:43.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:43.142 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:43.142 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:43.142 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.11 2024-04-17T10:34:43.142 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:43.142 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:43.143 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:43.143 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:43.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.210 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138594 s, 75.7 MB/s 2024-04-17T10:34:43.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.225 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132658 s, 79.0 MB/s 2024-04-17T10:34:43.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.247 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197726 s, 53.0 MB/s 2024-04-17T10:34:43.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.262 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138334 s, 75.8 MB/s 2024-04-17T10:34:43.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143932 s, 72.9 MB/s 2024-04-17T10:34:43.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144127 s, 72.8 MB/s 2024-04-17T10:34:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.311 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146294 s, 71.7 MB/s 2024-04-17T10:34:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190415 s, 55.1 MB/s 2024-04-17T10:34:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148976 s, 70.4 MB/s 2024-04-17T10:34:43.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174482 s, 60.1 MB/s 2024-04-17T10:34:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137407 s, 76.3 MB/s 2024-04-17T10:34:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149102 s, 70.3 MB/s 2024-04-17T10:34:43.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.421 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151672 s, 69.1 MB/s 2024-04-17T10:34:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.433 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154308 s, 68.0 MB/s 2024-04-17T10:34:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.448 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132852 s, 78.9 MB/s 2024-04-17T10:34:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134098 s, 78.2 MB/s 2024-04-17T10:34:43.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.480 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150799 s, 69.5 MB/s 2024-04-17T10:34:43.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140818 s, 74.5 MB/s 2024-04-17T10:34:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.512 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137218 s, 76.4 MB/s 2024-04-17T10:34:43.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.527 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.527 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138931 s, 75.5 MB/s 2024-04-17T10:34:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145137 s, 72.2 MB/s 2024-04-17T10:34:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.560 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146627 s, 71.5 MB/s 2024-04-17T10:34:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.576 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141018 s, 74.4 MB/s 2024-04-17T10:34:43.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.591 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138344 s, 75.8 MB/s 2024-04-17T10:34:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151375 s, 69.3 MB/s 2024-04-17T10:34:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.623 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132688 s, 79.0 MB/s 2024-04-17T10:34:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143637 s, 73.0 MB/s 2024-04-17T10:34:43.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.655 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.655 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141822 s, 73.9 MB/s 2024-04-17T10:34:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.670 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127163 s, 82.5 MB/s 2024-04-17T10:34:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.684 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128229 s, 81.8 MB/s 2024-04-17T10:34:43.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.700 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139632 s, 75.1 MB/s 2024-04-17T10:34:43.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.717 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150408 s, 69.7 MB/s 2024-04-17T10:34:43.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:43.719 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:43.719 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:43.719 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.12 2024-04-17T10:34:43.719 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:43.719 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:43.719 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:43.719 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.795 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013397 s, 78.3 MB/s 2024-04-17T10:34:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.812 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149596 s, 70.1 MB/s 2024-04-17T10:34:43.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.827 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140291 s, 74.7 MB/s 2024-04-17T10:34:43.844 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.845 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153946 s, 68.1 MB/s 2024-04-17T10:34:43.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149527 s, 70.1 MB/s 2024-04-17T10:34:43.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.881 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.882 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181256 s, 57.9 MB/s 2024-04-17T10:34:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144177 s, 72.7 MB/s 2024-04-17T10:34:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.916 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161604 s, 64.9 MB/s 2024-04-17T10:34:43.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.932 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146988 s, 71.3 MB/s 2024-04-17T10:34:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156426 s, 67.0 MB/s 2024-04-17T10:34:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141864 s, 73.9 MB/s 2024-04-17T10:34:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.982 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151689 s, 69.1 MB/s 2024-04-17T10:34:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:43.999 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150275 s, 69.8 MB/s 2024-04-17T10:34:44.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.014 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136655 s, 76.7 MB/s 2024-04-17T10:34:44.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138187 s, 75.9 MB/s 2024-04-17T10:34:44.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.047 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.047 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014968 s, 70.1 MB/s 2024-04-17T10:34:44.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.062 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138411 s, 75.8 MB/s 2024-04-17T10:34:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.078 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137329 s, 76.4 MB/s 2024-04-17T10:34:44.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.094 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137774 s, 76.1 MB/s 2024-04-17T10:34:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.114 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182933 s, 57.3 MB/s 2024-04-17T10:34:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.127 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115345 s, 90.9 MB/s 2024-04-17T10:34:44.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.142 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127839 s, 82.0 MB/s 2024-04-17T10:34:44.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130256 s, 80.5 MB/s 2024-04-17T10:34:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147864 s, 70.9 MB/s 2024-04-17T10:34:44.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.190 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154332 s, 67.9 MB/s 2024-04-17T10:34:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.207 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015003 s, 69.9 MB/s 2024-04-17T10:34:44.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133839 s, 78.3 MB/s 2024-04-17T10:34:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.239 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152256 s, 68.9 MB/s 2024-04-17T10:34:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139772 s, 75.0 MB/s 2024-04-17T10:34:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.276 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188584 s, 55.6 MB/s 2024-04-17T10:34:44.292 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151571 s, 69.2 MB/s 2024-04-17T10:34:44.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.308 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139473 s, 75.2 MB/s 2024-04-17T10:34:44.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:44.310 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:44.310 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:44.310 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.13 2024-04-17T10:34:44.310 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:44.310 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:44.311 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:44.311 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.380 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153104 s, 68.5 MB/s 2024-04-17T10:34:44.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.396 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140127 s, 74.8 MB/s 2024-04-17T10:34:44.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.412 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148906 s, 70.4 MB/s 2024-04-17T10:34:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.427 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013307 s, 78.8 MB/s 2024-04-17T10:34:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.442 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136841 s, 76.6 MB/s 2024-04-17T10:34:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137029 s, 76.5 MB/s 2024-04-17T10:34:44.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.477 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168152 s, 62.4 MB/s 2024-04-17T10:34:44.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.494 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150134 s, 69.8 MB/s 2024-04-17T10:34:44.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.512 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.512 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168513 s, 62.2 MB/s 2024-04-17T10:34:44.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.528 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137747 s, 76.1 MB/s 2024-04-17T10:34:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142458 s, 73.6 MB/s 2024-04-17T10:34:44.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.561 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.561 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149862 s, 70.0 MB/s 2024-04-17T10:34:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127408 s, 82.3 MB/s 2024-04-17T10:34:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.593 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163837 s, 64.0 MB/s 2024-04-17T10:34:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137593 s, 76.2 MB/s 2024-04-17T10:34:44.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.625 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141412 s, 74.2 MB/s 2024-04-17T10:34:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149589 s, 70.1 MB/s 2024-04-17T10:34:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139603 s, 75.1 MB/s 2024-04-17T10:34:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132738 s, 79.0 MB/s 2024-04-17T10:34:44.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.689 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149325 s, 70.2 MB/s 2024-04-17T10:34:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.704 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136001 s, 77.1 MB/s 2024-04-17T10:34:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015462 s, 67.8 MB/s 2024-04-17T10:34:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.737 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135552 s, 77.4 MB/s 2024-04-17T10:34:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.754 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151536 s, 69.2 MB/s 2024-04-17T10:34:44.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.771 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151838 s, 69.1 MB/s 2024-04-17T10:34:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.787 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.788 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149168 s, 70.3 MB/s 2024-04-17T10:34:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.803 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141334 s, 74.2 MB/s 2024-04-17T10:34:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.819 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.820 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01444 s, 72.6 MB/s 2024-04-17T10:34:44.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.837 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015725 s, 66.7 MB/s 2024-04-17T10:34:44.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.853 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134477 s, 78.0 MB/s 2024-04-17T10:34:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.868 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143052 s, 73.3 MB/s 2024-04-17T10:34:44.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165231 s, 63.5 MB/s 2024-04-17T10:34:44.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:44.888 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:44.888 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:44.889 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.14 2024-04-17T10:34:44.889 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:44.889 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:44.889 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:44.889 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:44.959 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.959 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.959 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152383 s, 68.8 MB/s 2024-04-17T10:34:44.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.978 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.978 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167703 s, 62.5 MB/s 2024-04-17T10:34:44.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:44.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:44.993 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131782 s, 79.6 MB/s 2024-04-17T10:34:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.009 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143394 s, 73.1 MB/s 2024-04-17T10:34:45.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.025 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139694 s, 75.1 MB/s 2024-04-17T10:34:45.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.041 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143873 s, 72.9 MB/s 2024-04-17T10:34:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.057 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138903 s, 75.5 MB/s 2024-04-17T10:34:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.076 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170942 s, 61.3 MB/s 2024-04-17T10:34:45.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.092 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142097 s, 73.8 MB/s 2024-04-17T10:34:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.108 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014524 s, 72.2 MB/s 2024-04-17T10:34:45.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133298 s, 78.7 MB/s 2024-04-17T10:34:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.139 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144183 s, 72.7 MB/s 2024-04-17T10:34:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139545 s, 75.1 MB/s 2024-04-17T10:34:45.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.172 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153546 s, 68.3 MB/s 2024-04-17T10:34:45.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.188 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138387 s, 75.8 MB/s 2024-04-17T10:34:45.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.205 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151269 s, 69.3 MB/s 2024-04-17T10:34:45.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151577 s, 69.2 MB/s 2024-04-17T10:34:45.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162046 s, 64.7 MB/s 2024-04-17T10:34:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.256 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149975 s, 69.9 MB/s 2024-04-17T10:34:45.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133042 s, 78.8 MB/s 2024-04-17T10:34:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.285 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011995 s, 87.4 MB/s 2024-04-17T10:34:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.303 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161711 s, 64.8 MB/s 2024-04-17T10:34:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.323 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181439 s, 57.8 MB/s 2024-04-17T10:34:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.340 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158279 s, 66.2 MB/s 2024-04-17T10:34:45.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.360 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178233 s, 58.8 MB/s 2024-04-17T10:34:45.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.376 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147069 s, 71.3 MB/s 2024-04-17T10:34:45.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134062 s, 78.2 MB/s 2024-04-17T10:34:45.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143655 s, 73.0 MB/s 2024-04-17T10:34:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01528 s, 68.6 MB/s 2024-04-17T10:34:45.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.445 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183636 s, 57.1 MB/s 2024-04-17T10:34:45.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147294 s, 71.2 MB/s 2024-04-17T10:34:45.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.479 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152914 s, 68.6 MB/s 2024-04-17T10:34:45.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:45.480 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:45.480 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:45.480 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.15 2024-04-17T10:34:45.480 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:45.480 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:45.480 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:45.480 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142065 s, 73.8 MB/s 2024-04-17T10:34:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.570 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134142 s, 78.2 MB/s 2024-04-17T10:34:45.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.588 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168481 s, 62.2 MB/s 2024-04-17T10:34:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183616 s, 57.1 MB/s 2024-04-17T10:34:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.625 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.626 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153972 s, 68.1 MB/s 2024-04-17T10:34:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.643 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155876 s, 67.3 MB/s 2024-04-17T10:34:45.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170432 s, 61.5 MB/s 2024-04-17T10:34:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142048 s, 73.8 MB/s 2024-04-17T10:34:45.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.693 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140571 s, 74.6 MB/s 2024-04-17T10:34:45.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.711 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153277 s, 68.4 MB/s 2024-04-17T10:34:45.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139185 s, 75.3 MB/s 2024-04-17T10:34:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.741 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133407 s, 78.6 MB/s 2024-04-17T10:34:45.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.757 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013399 s, 78.3 MB/s 2024-04-17T10:34:45.775 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.775 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.775 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170559 s, 61.5 MB/s 2024-04-17T10:34:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.792 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144664 s, 72.5 MB/s 2024-04-17T10:34:45.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015773 s, 66.5 MB/s 2024-04-17T10:34:45.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.827 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161687 s, 64.9 MB/s 2024-04-17T10:34:45.845 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.846 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164884 s, 63.6 MB/s 2024-04-17T10:34:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.862 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.863 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150276 s, 69.8 MB/s 2024-04-17T10:34:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.877 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.878 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132738 s, 79.0 MB/s 2024-04-17T10:34:45.893 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.893 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.893 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139403 s, 75.2 MB/s 2024-04-17T10:34:45.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.910 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151475 s, 69.2 MB/s 2024-04-17T10:34:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.925 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128159 s, 81.8 MB/s 2024-04-17T10:34:45.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.945 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179974 s, 58.3 MB/s 2024-04-17T10:34:45.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.963 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166308 s, 63.1 MB/s 2024-04-17T10:34:45.984 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:45.984 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:45.984 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192348 s, 54.5 MB/s 2024-04-17T10:34:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.000 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014043 s, 74.7 MB/s 2024-04-17T10:34:46.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.019 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168228 s, 62.3 MB/s 2024-04-17T10:34:46.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.036 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149879 s, 70.0 MB/s 2024-04-17T10:34:46.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.053 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154507 s, 67.9 MB/s 2024-04-17T10:34:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138323 s, 75.8 MB/s 2024-04-17T10:34:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014913 s, 70.3 MB/s 2024-04-17T10:34:46.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:46.087 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:46.087 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:46.087 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.16 2024-04-17T10:34:46.087 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:46.087 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:46.087 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:46.087 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157426 s, 66.6 MB/s 2024-04-17T10:34:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.177 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153092 s, 68.5 MB/s 2024-04-17T10:34:46.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.196 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168084 s, 62.4 MB/s 2024-04-17T10:34:46.212 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.212 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.212 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138882 s, 75.5 MB/s 2024-04-17T10:34:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.228 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142716 s, 73.5 MB/s 2024-04-17T10:34:46.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.246 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160975 s, 65.1 MB/s 2024-04-17T10:34:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140866 s, 74.4 MB/s 2024-04-17T10:34:46.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.277 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137982 s, 76.0 MB/s 2024-04-17T10:34:46.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143589 s, 73.0 MB/s 2024-04-17T10:34:46.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.308 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.308 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131718 s, 79.6 MB/s 2024-04-17T10:34:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.324 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136837 s, 76.6 MB/s 2024-04-17T10:34:46.341 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.341 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.341 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154239 s, 68.0 MB/s 2024-04-17T10:34:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.357 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135808 s, 77.2 MB/s 2024-04-17T10:34:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.373 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014309 s, 73.3 MB/s 2024-04-17T10:34:46.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.393 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181727 s, 57.7 MB/s 2024-04-17T10:34:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139937 s, 74.9 MB/s 2024-04-17T10:34:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156453 s, 67.0 MB/s 2024-04-17T10:34:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.441 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139359 s, 75.2 MB/s 2024-04-17T10:34:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147637 s, 71.0 MB/s 2024-04-17T10:34:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.473 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134233 s, 78.1 MB/s 2024-04-17T10:34:46.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.488 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134202 s, 78.1 MB/s 2024-04-17T10:34:46.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146359 s, 71.6 MB/s 2024-04-17T10:34:46.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188251 s, 55.7 MB/s 2024-04-17T10:34:46.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.542 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147351 s, 71.2 MB/s 2024-04-17T10:34:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.558 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140662 s, 74.5 MB/s 2024-04-17T10:34:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180169 s, 58.2 MB/s 2024-04-17T10:34:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134845 s, 77.8 MB/s 2024-04-17T10:34:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134952 s, 77.7 MB/s 2024-04-17T10:34:46.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.623 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131685 s, 79.6 MB/s 2024-04-17T10:34:46.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165089 s, 63.5 MB/s 2024-04-17T10:34:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.657 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133725 s, 78.4 MB/s 2024-04-17T10:34:46.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.672 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132947 s, 78.9 MB/s 2024-04-17T10:34:46.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:46.673 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:46.673 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:46.674 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.17 2024-04-17T10:34:46.674 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:46.674 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:46.674 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:46.674 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:46.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143591 s, 73.0 MB/s 2024-04-17T10:34:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.764 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149661 s, 70.1 MB/s 2024-04-17T10:34:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140711 s, 74.5 MB/s 2024-04-17T10:34:46.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.796 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.796 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148902 s, 70.4 MB/s 2024-04-17T10:34:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.812 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138032 s, 76.0 MB/s 2024-04-17T10:34:46.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119644 s, 87.6 MB/s 2024-04-17T10:34:46.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.841 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132184 s, 79.3 MB/s 2024-04-17T10:34:46.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.856 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138024 s, 76.0 MB/s 2024-04-17T10:34:46.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.872 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143259 s, 73.2 MB/s 2024-04-17T10:34:46.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148138 s, 70.8 MB/s 2024-04-17T10:34:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.904 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138587 s, 75.7 MB/s 2024-04-17T10:34:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163886 s, 64.0 MB/s 2024-04-17T10:34:46.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.939 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150775 s, 69.5 MB/s 2024-04-17T10:34:46.959 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.959 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.959 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178831 s, 58.6 MB/s 2024-04-17T10:34:46.974 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.975 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140698 s, 74.5 MB/s 2024-04-17T10:34:46.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:46.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:46.992 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153121 s, 68.5 MB/s 2024-04-17T10:34:47.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.007 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.007 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135552 s, 77.4 MB/s 2024-04-17T10:34:47.022 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.023 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.023 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135102 s, 77.6 MB/s 2024-04-17T10:34:47.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.039 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143508 s, 73.1 MB/s 2024-04-17T10:34:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.054 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135374 s, 77.5 MB/s 2024-04-17T10:34:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.070 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146741 s, 71.5 MB/s 2024-04-17T10:34:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.086 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140635 s, 74.6 MB/s 2024-04-17T10:34:47.103 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.103 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.103 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150426 s, 69.7 MB/s 2024-04-17T10:34:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.120 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150745 s, 69.6 MB/s 2024-04-17T10:34:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155865 s, 67.3 MB/s 2024-04-17T10:34:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148418 s, 70.7 MB/s 2024-04-17T10:34:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151344 s, 69.3 MB/s 2024-04-17T10:34:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.188 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154821 s, 67.7 MB/s 2024-04-17T10:34:47.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179543 s, 58.4 MB/s 2024-04-17T10:34:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013911 s, 75.4 MB/s 2024-04-17T10:34:47.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.241 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153239 s, 68.4 MB/s 2024-04-17T10:34:47.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.257 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014403 s, 72.8 MB/s 2024-04-17T10:34:47.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:47.258 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:47.258 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:47.259 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.18 2024-04-17T10:34:47.259 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:47.259 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:47.259 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:47.259 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:47.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.333 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169442 s, 61.9 MB/s 2024-04-17T10:34:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.351 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152702 s, 68.7 MB/s 2024-04-17T10:34:47.366 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.366 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.366 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137092 s, 76.5 MB/s 2024-04-17T10:34:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189181 s, 55.4 MB/s 2024-04-17T10:34:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.403 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140016 s, 74.9 MB/s 2024-04-17T10:34:47.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.422 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178366 s, 58.8 MB/s 2024-04-17T10:34:47.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131678 s, 79.6 MB/s 2024-04-17T10:34:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.458 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187156 s, 56.0 MB/s 2024-04-17T10:34:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164534 s, 63.7 MB/s 2024-04-17T10:34:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.492 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146051 s, 71.8 MB/s 2024-04-17T10:34:47.513 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.513 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.513 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183964 s, 57.0 MB/s 2024-04-17T10:34:47.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.528 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013599 s, 77.1 MB/s 2024-04-17T10:34:47.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143418 s, 73.1 MB/s 2024-04-17T10:34:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.561 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015134 s, 69.3 MB/s 2024-04-17T10:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.577 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141701 s, 74.0 MB/s 2024-04-17T10:34:47.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.595 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161256 s, 65.0 MB/s 2024-04-17T10:34:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170931 s, 61.3 MB/s 2024-04-17T10:34:47.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.630 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140872 s, 74.4 MB/s 2024-04-17T10:34:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.645 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135156 s, 77.6 MB/s 2024-04-17T10:34:47.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.660 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135908 s, 77.2 MB/s 2024-04-17T10:34:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.676 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013174 s, 79.6 MB/s 2024-04-17T10:34:47.691 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.691 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.691 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013949 s, 75.2 MB/s 2024-04-17T10:34:47.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144117 s, 72.8 MB/s 2024-04-17T10:34:47.723 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.724 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143911 s, 72.9 MB/s 2024-04-17T10:34:47.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148349 s, 70.7 MB/s 2024-04-17T10:34:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.756 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140733 s, 74.5 MB/s 2024-04-17T10:34:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.773 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158176 s, 66.3 MB/s 2024-04-17T10:34:47.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.789 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137133 s, 76.5 MB/s 2024-04-17T10:34:47.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.804 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136116 s, 77.0 MB/s 2024-04-17T10:34:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.822 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152217 s, 68.9 MB/s 2024-04-17T10:34:47.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.838 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147189 s, 71.2 MB/s 2024-04-17T10:34:47.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.857 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.857 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172741 s, 60.7 MB/s 2024-04-17T10:34:47.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:47.859 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:47.859 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:47.859 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.19 2024-04-17T10:34:47.859 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:47.859 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:47.859 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:47.859 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.929 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.930 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130578 s, 80.3 MB/s 2024-04-17T10:34:47.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.945 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135009 s, 77.7 MB/s 2024-04-17T10:34:47.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139383 s, 75.2 MB/s 2024-04-17T10:34:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147641 s, 71.0 MB/s 2024-04-17T10:34:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:47.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:47.994 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142527 s, 73.6 MB/s 2024-04-17T10:34:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158433 s, 66.2 MB/s 2024-04-17T10:34:48.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135671 s, 77.3 MB/s 2024-04-17T10:34:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.043 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138593 s, 75.7 MB/s 2024-04-17T10:34:48.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.059 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.059 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139177 s, 75.3 MB/s 2024-04-17T10:34:48.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.075 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147594 s, 71.0 MB/s 2024-04-17T10:34:48.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.091 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137102 s, 76.5 MB/s 2024-04-17T10:34:48.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.108 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152047 s, 69.0 MB/s 2024-04-17T10:34:48.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.124 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140718 s, 74.5 MB/s 2024-04-17T10:34:48.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.141 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158252 s, 66.3 MB/s 2024-04-17T10:34:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220736 s, 47.5 MB/s 2024-04-17T10:34:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158182 s, 66.3 MB/s 2024-04-17T10:34:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.204 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194581 s, 53.9 MB/s 2024-04-17T10:34:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150535 s, 69.7 MB/s 2024-04-17T10:34:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.238 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149512 s, 70.1 MB/s 2024-04-17T10:34:48.254 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149703 s, 70.0 MB/s 2024-04-17T10:34:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.274 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150779 s, 69.5 MB/s 2024-04-17T10:34:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.288 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147345 s, 71.2 MB/s 2024-04-17T10:34:48.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.305 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149315 s, 70.2 MB/s 2024-04-17T10:34:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.321 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139834 s, 75.0 MB/s 2024-04-17T10:34:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.336 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137131 s, 76.5 MB/s 2024-04-17T10:34:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172251 s, 60.9 MB/s 2024-04-17T10:34:48.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139048 s, 75.4 MB/s 2024-04-17T10:34:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.389 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161331 s, 65.0 MB/s 2024-04-17T10:34:48.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.404 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133753 s, 78.4 MB/s 2024-04-17T10:34:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116828 s, 89.8 MB/s 2024-04-17T10:34:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.434 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152365 s, 68.8 MB/s 2024-04-17T10:34:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.450 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138958 s, 75.5 MB/s 2024-04-17T10:34:48.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:48.452 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:48.452 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:48.452 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.20 2024-04-17T10:34:48.452 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:48.452 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:48.452 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:48.452 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:48.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169385 s, 61.9 MB/s 2024-04-17T10:34:48.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.542 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147008 s, 71.3 MB/s 2024-04-17T10:34:48.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.557 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137275 s, 76.4 MB/s 2024-04-17T10:34:48.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.573 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013915 s, 75.4 MB/s 2024-04-17T10:34:48.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.588 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132928 s, 78.9 MB/s 2024-04-17T10:34:48.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.603 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131563 s, 79.7 MB/s 2024-04-17T10:34:48.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.618 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.618 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131793 s, 79.6 MB/s 2024-04-17T10:34:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.635 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148327 s, 70.7 MB/s 2024-04-17T10:34:48.651 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.652 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148363 s, 70.7 MB/s 2024-04-17T10:34:48.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.667 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138851 s, 75.5 MB/s 2024-04-17T10:34:48.683 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.683 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.683 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140236 s, 74.8 MB/s 2024-04-17T10:34:48.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.698 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013068 s, 80.2 MB/s 2024-04-17T10:34:48.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145563 s, 72.0 MB/s 2024-04-17T10:34:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.729 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135866 s, 77.2 MB/s 2024-04-17T10:34:48.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.746 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150409 s, 69.7 MB/s 2024-04-17T10:34:48.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.762 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.762 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137589 s, 76.2 MB/s 2024-04-17T10:34:48.778 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.778 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.778 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144762 s, 72.4 MB/s 2024-04-17T10:34:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.795 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149691 s, 70.0 MB/s 2024-04-17T10:34:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.812 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151564 s, 69.2 MB/s 2024-04-17T10:34:48.828 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.828 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.828 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147595 s, 71.0 MB/s 2024-04-17T10:34:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180315 s, 58.2 MB/s 2024-04-17T10:34:48.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.863 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136296 s, 76.9 MB/s 2024-04-17T10:34:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.881 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152292 s, 68.9 MB/s 2024-04-17T10:34:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.897 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143355 s, 73.1 MB/s 2024-04-17T10:34:48.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.914 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150693 s, 69.6 MB/s 2024-04-17T10:34:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.930 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148318 s, 70.7 MB/s 2024-04-17T10:34:48.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.947 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150493 s, 69.7 MB/s 2024-04-17T10:34:48.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.963 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.963 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145421 s, 72.1 MB/s 2024-04-17T10:34:48.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.980 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147972 s, 70.9 MB/s 2024-04-17T10:34:48.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:48.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:48.997 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151985 s, 69.0 MB/s 2024-04-17T10:34:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016384 s, 64.0 MB/s 2024-04-17T10:34:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.031 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147049 s, 71.3 MB/s 2024-04-17T10:34:49.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:49.033 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:49.033 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:49.033 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.21 2024-04-17T10:34:49.033 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:49.033 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:49.033 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:49.033 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.102 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136172 s, 77.0 MB/s 2024-04-17T10:34:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.119 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014824 s, 70.7 MB/s 2024-04-17T10:34:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153896 s, 68.1 MB/s 2024-04-17T10:34:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.152 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141946 s, 73.9 MB/s 2024-04-17T10:34:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.167 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135514 s, 77.4 MB/s 2024-04-17T10:34:49.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.183 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136131 s, 77.0 MB/s 2024-04-17T10:34:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.200 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.201 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157306 s, 66.7 MB/s 2024-04-17T10:34:49.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154631 s, 67.8 MB/s 2024-04-17T10:34:49.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013515 s, 77.6 MB/s 2024-04-17T10:34:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145215 s, 72.2 MB/s 2024-04-17T10:34:49.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.268 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173849 s, 60.3 MB/s 2024-04-17T10:34:49.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.283 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133941 s, 78.3 MB/s 2024-04-17T10:34:49.303 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.304 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180611 s, 58.1 MB/s 2024-04-17T10:34:49.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.319 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.319 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139457 s, 75.2 MB/s 2024-04-17T10:34:49.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170266 s, 61.6 MB/s 2024-04-17T10:34:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.353 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.354 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136559 s, 76.8 MB/s 2024-04-17T10:34:49.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159903 s, 65.6 MB/s 2024-04-17T10:34:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.388 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014752 s, 71.1 MB/s 2024-04-17T10:34:49.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.403 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137845 s, 76.1 MB/s 2024-04-17T10:34:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146486 s, 71.6 MB/s 2024-04-17T10:34:49.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152003 s, 69.0 MB/s 2024-04-17T10:34:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.453 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144864 s, 72.4 MB/s 2024-04-17T10:34:49.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.474 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189066 s, 55.5 MB/s 2024-04-17T10:34:49.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131885 s, 79.5 MB/s 2024-04-17T10:34:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148109 s, 70.8 MB/s 2024-04-17T10:34:49.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.525 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018174 s, 57.7 MB/s 2024-04-17T10:34:49.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.540 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130919 s, 80.1 MB/s 2024-04-17T10:34:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.556 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137598 s, 76.2 MB/s 2024-04-17T10:34:49.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.574 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163685 s, 64.1 MB/s 2024-04-17T10:34:49.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.589 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133501 s, 78.5 MB/s 2024-04-17T10:34:49.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.606 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148117 s, 70.8 MB/s 2024-04-17T10:34:49.621 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.621 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.621 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137945 s, 76.0 MB/s 2024-04-17T10:34:49.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:49.623 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:49.623 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:49.623 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.22 2024-04-17T10:34:49.623 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:49.623 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:49.623 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:49.624 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.699 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192396 s, 54.5 MB/s 2024-04-17T10:34:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.715 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138742 s, 75.6 MB/s 2024-04-17T10:34:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149605 s, 70.1 MB/s 2024-04-17T10:34:49.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.748 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144993 s, 72.3 MB/s 2024-04-17T10:34:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.763 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136837 s, 76.6 MB/s 2024-04-17T10:34:49.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139398 s, 75.2 MB/s 2024-04-17T10:34:49.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.795 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.795 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143916 s, 72.9 MB/s 2024-04-17T10:34:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.812 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153853 s, 68.2 MB/s 2024-04-17T10:34:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.827 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132506 s, 79.1 MB/s 2024-04-17T10:34:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.843 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139686 s, 75.1 MB/s 2024-04-17T10:34:49.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.859 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.859 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141217 s, 74.3 MB/s 2024-04-17T10:34:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.874 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136205 s, 77.0 MB/s 2024-04-17T10:34:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.891 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149607 s, 70.1 MB/s 2024-04-17T10:34:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.908 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149273 s, 70.2 MB/s 2024-04-17T10:34:49.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133253 s, 78.7 MB/s 2024-04-17T10:34:49.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.939 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141529 s, 74.1 MB/s 2024-04-17T10:34:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142541 s, 73.6 MB/s 2024-04-17T10:34:49.969 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130055 s, 80.6 MB/s 2024-04-17T10:34:49.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:49.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:49.985 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136646 s, 76.7 MB/s 2024-04-17T10:34:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.001 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144842 s, 72.4 MB/s 2024-04-17T10:34:50.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137394 s, 76.3 MB/s 2024-04-17T10:34:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145199 s, 72.2 MB/s 2024-04-17T10:34:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.049 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140252 s, 74.8 MB/s 2024-04-17T10:34:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.067 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160857 s, 65.2 MB/s 2024-04-17T10:34:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130511 s, 80.3 MB/s 2024-04-17T10:34:50.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.099 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150948 s, 69.5 MB/s 2024-04-17T10:34:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.115 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014582 s, 71.9 MB/s 2024-04-17T10:34:50.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136212 s, 77.0 MB/s 2024-04-17T10:34:50.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.146 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137996 s, 76.0 MB/s 2024-04-17T10:34:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.163 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014956 s, 70.1 MB/s 2024-04-17T10:34:50.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.180 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.180 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152049 s, 69.0 MB/s 2024-04-17T10:34:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01689 s, 62.1 MB/s 2024-04-17T10:34:50.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:50.199 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:50.200 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:50.200 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.23 2024-04-17T10:34:50.200 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:50.200 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:50.200 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:50.200 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185256 s, 56.6 MB/s 2024-04-17T10:34:50.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.294 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136461 s, 76.8 MB/s 2024-04-17T10:34:50.309 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.309 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.309 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134782 s, 77.8 MB/s 2024-04-17T10:34:50.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.327 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155467 s, 67.4 MB/s 2024-04-17T10:34:50.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.343 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142999 s, 73.3 MB/s 2024-04-17T10:34:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.360 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151709 s, 69.1 MB/s 2024-04-17T10:34:50.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.377 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145916 s, 71.9 MB/s 2024-04-17T10:34:50.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.391 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131401 s, 79.8 MB/s 2024-04-17T10:34:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.407 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013993 s, 74.9 MB/s 2024-04-17T10:34:50.424 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.424 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.424 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155678 s, 67.4 MB/s 2024-04-17T10:34:50.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.439 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125937 s, 83.3 MB/s 2024-04-17T10:34:50.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142256 s, 73.7 MB/s 2024-04-17T10:34:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.470 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136031 s, 77.1 MB/s 2024-04-17T10:34:50.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.485 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130354 s, 80.4 MB/s 2024-04-17T10:34:50.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.501 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145486 s, 72.1 MB/s 2024-04-17T10:34:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.516 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127933 s, 82.0 MB/s 2024-04-17T10:34:50.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.533 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.533 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149964 s, 69.9 MB/s 2024-04-17T10:34:50.548 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.548 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.548 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139727 s, 75.0 MB/s 2024-04-17T10:34:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184327 s, 56.9 MB/s 2024-04-17T10:34:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.586 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156725 s, 66.9 MB/s 2024-04-17T10:34:50.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.603 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.603 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151033 s, 69.4 MB/s 2024-04-17T10:34:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.619 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147309 s, 71.2 MB/s 2024-04-17T10:34:50.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178164 s, 58.9 MB/s 2024-04-17T10:34:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175014 s, 59.9 MB/s 2024-04-17T10:34:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144453 s, 72.6 MB/s 2024-04-17T10:34:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.688 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123456 s, 84.9 MB/s 2024-04-17T10:34:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.709 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184348 s, 56.9 MB/s 2024-04-17T10:34:50.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141675 s, 74.0 MB/s 2024-04-17T10:34:50.741 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.741 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.741 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144153 s, 72.7 MB/s 2024-04-17T10:34:50.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.757 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.757 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145901 s, 71.9 MB/s 2024-04-17T10:34:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.774 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150738 s, 69.6 MB/s 2024-04-17T10:34:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135057 s, 77.6 MB/s 2024-04-17T10:34:50.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:50.791 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:50.791 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:50.791 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.24 2024-04-17T10:34:50.791 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:50.791 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:50.791 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:50.791 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:50.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.866 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134857 s, 77.8 MB/s 2024-04-17T10:34:50.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.883 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151729 s, 69.1 MB/s 2024-04-17T10:34:50.899 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.899 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.899 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147437 s, 71.1 MB/s 2024-04-17T10:34:50.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.916 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143517 s, 73.1 MB/s 2024-04-17T10:34:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153867 s, 68.1 MB/s 2024-04-17T10:34:50.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145764 s, 71.9 MB/s 2024-04-17T10:34:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146982 s, 71.3 MB/s 2024-04-17T10:34:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.982 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014923 s, 70.3 MB/s 2024-04-17T10:34:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138585 s, 75.7 MB/s 2024-04-17T10:34:51.013 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.014 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.014 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143931 s, 72.9 MB/s 2024-04-17T10:34:51.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.030 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143712 s, 73.0 MB/s 2024-04-17T10:34:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146646 s, 71.5 MB/s 2024-04-17T10:34:51.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133958 s, 78.3 MB/s 2024-04-17T10:34:51.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155476 s, 67.4 MB/s 2024-04-17T10:34:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145447 s, 72.1 MB/s 2024-04-17T10:34:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015071 s, 69.6 MB/s 2024-04-17T10:34:51.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161255 s, 65.0 MB/s 2024-04-17T10:34:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.146 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147613 s, 71.0 MB/s 2024-04-17T10:34:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.167 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188208 s, 55.7 MB/s 2024-04-17T10:34:51.183 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.183 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.183 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143841 s, 72.9 MB/s 2024-04-17T10:34:51.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134495 s, 78.0 MB/s 2024-04-17T10:34:51.213 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.213 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.213 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131231 s, 79.9 MB/s 2024-04-17T10:34:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.229 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138708 s, 75.6 MB/s 2024-04-17T10:34:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.246 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150563 s, 69.6 MB/s 2024-04-17T10:34:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.262 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145644 s, 72.0 MB/s 2024-04-17T10:34:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151657 s, 69.1 MB/s 2024-04-17T10:34:51.295 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.296 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148981 s, 70.4 MB/s 2024-04-17T10:34:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.312 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149481 s, 70.1 MB/s 2024-04-17T10:34:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.329 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149382 s, 70.2 MB/s 2024-04-17T10:34:51.345 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.345 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.345 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139243 s, 75.3 MB/s 2024-04-17T10:34:51.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.361 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146677 s, 71.5 MB/s 2024-04-17T10:34:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150982 s, 69.5 MB/s 2024-04-17T10:34:51.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:51.379 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:51.380 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:51.380 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.25 2024-04-17T10:34:51.380 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:51.380 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:51.380 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:51.380 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149552 s, 70.1 MB/s 2024-04-17T10:34:51.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152311 s, 68.8 MB/s 2024-04-17T10:34:51.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.486 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155596 s, 67.4 MB/s 2024-04-17T10:34:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.501 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136772 s, 76.7 MB/s 2024-04-17T10:34:51.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137408 s, 76.3 MB/s 2024-04-17T10:34:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01866 s, 56.2 MB/s 2024-04-17T10:34:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145024 s, 72.3 MB/s 2024-04-17T10:34:51.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138915 s, 75.5 MB/s 2024-04-17T10:34:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.585 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014297 s, 73.3 MB/s 2024-04-17T10:34:51.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.602 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.602 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151487 s, 69.2 MB/s 2024-04-17T10:34:51.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.619 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.619 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014963 s, 70.1 MB/s 2024-04-17T10:34:51.636 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.636 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.637 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160155 s, 65.5 MB/s 2024-04-17T10:34:51.652 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.653 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.653 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144872 s, 72.4 MB/s 2024-04-17T10:34:51.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.668 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.668 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136152 s, 77.0 MB/s 2024-04-17T10:34:51.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.684 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141541 s, 74.1 MB/s 2024-04-17T10:34:51.699 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137982 s, 76.0 MB/s 2024-04-17T10:34:51.715 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.716 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.716 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139638 s, 75.1 MB/s 2024-04-17T10:34:51.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.732 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148848 s, 70.4 MB/s 2024-04-17T10:34:51.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.755 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183703 s, 57.1 MB/s 2024-04-17T10:34:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134066 s, 78.2 MB/s 2024-04-17T10:34:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014057 s, 74.6 MB/s 2024-04-17T10:34:51.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.798 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137865 s, 76.1 MB/s 2024-04-17T10:34:51.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.815 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144804 s, 72.4 MB/s 2024-04-17T10:34:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187237 s, 56.0 MB/s 2024-04-17T10:34:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.851 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014252 s, 73.6 MB/s 2024-04-17T10:34:51.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.867 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.867 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143117 s, 73.3 MB/s 2024-04-17T10:34:51.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.883 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145475 s, 72.1 MB/s 2024-04-17T10:34:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.902 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168737 s, 62.1 MB/s 2024-04-17T10:34:51.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.917 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132048 s, 79.4 MB/s 2024-04-17T10:34:51.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.932 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134343 s, 78.1 MB/s 2024-04-17T10:34:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148887 s, 70.4 MB/s 2024-04-17T10:34:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:51.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143416 s, 73.1 MB/s 2024-04-17T10:34:51.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:51.967 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:51.967 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:51.967 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.26 2024-04-17T10:34:51.967 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:51.967 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:51.967 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:51.967 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:52.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.039 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132613 s, 79.1 MB/s 2024-04-17T10:34:52.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.054 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134061 s, 78.2 MB/s 2024-04-17T10:34:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.074 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018583 s, 56.4 MB/s 2024-04-17T10:34:52.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.091 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152058 s, 69.0 MB/s 2024-04-17T10:34:52.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.107 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139044 s, 75.4 MB/s 2024-04-17T10:34:52.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.124 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155887 s, 67.3 MB/s 2024-04-17T10:34:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.141 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151591 s, 69.2 MB/s 2024-04-17T10:34:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131652 s, 79.6 MB/s 2024-04-17T10:34:52.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147076 s, 71.3 MB/s 2024-04-17T10:34:52.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.188 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013581 s, 77.2 MB/s 2024-04-17T10:34:52.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.204 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141791 s, 74.0 MB/s 2024-04-17T10:34:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.217 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114706 s, 91.4 MB/s 2024-04-17T10:34:52.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013757 s, 76.2 MB/s 2024-04-17T10:34:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184914 s, 56.7 MB/s 2024-04-17T10:34:52.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.270 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151627 s, 69.2 MB/s 2024-04-17T10:34:52.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.286 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.286 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014792 s, 70.9 MB/s 2024-04-17T10:34:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.302 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137617 s, 76.2 MB/s 2024-04-17T10:34:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.318 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147236 s, 71.2 MB/s 2024-04-17T10:34:52.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.334 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.334 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142671 s, 73.5 MB/s 2024-04-17T10:34:52.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.352 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153688 s, 68.2 MB/s 2024-04-17T10:34:52.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.372 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.372 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182605 s, 57.4 MB/s 2024-04-17T10:34:52.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139414 s, 75.2 MB/s 2024-04-17T10:34:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150704 s, 69.6 MB/s 2024-04-17T10:34:52.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136113 s, 77.0 MB/s 2024-04-17T10:34:52.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.440 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.440 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184252 s, 56.9 MB/s 2024-04-17T10:34:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.457 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014993 s, 69.9 MB/s 2024-04-17T10:34:52.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.474 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154269 s, 68.0 MB/s 2024-04-17T10:34:52.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.490 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.490 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143516 s, 73.1 MB/s 2024-04-17T10:34:52.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.505 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.505 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013421 s, 78.1 MB/s 2024-04-17T10:34:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.522 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146279 s, 71.7 MB/s 2024-04-17T10:34:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.538 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146921 s, 71.4 MB/s 2024-04-17T10:34:52.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142865 s, 73.4 MB/s 2024-04-17T10:34:52.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:52.556 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:52.556 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:52.556 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.27 2024-04-17T10:34:52.556 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:52.556 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:52.556 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:52.556 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014112 s, 74.3 MB/s 2024-04-17T10:34:52.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.646 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138718 s, 75.6 MB/s 2024-04-17T10:34:52.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.661 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.661 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133673 s, 78.4 MB/s 2024-04-17T10:34:52.677 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142701 s, 73.5 MB/s 2024-04-17T10:34:52.692 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.693 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132262 s, 79.3 MB/s 2024-04-17T10:34:52.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.708 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137964 s, 76.0 MB/s 2024-04-17T10:34:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155488 s, 67.4 MB/s 2024-04-17T10:34:52.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.742 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014681 s, 71.4 MB/s 2024-04-17T10:34:52.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.758 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.758 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139493 s, 75.2 MB/s 2024-04-17T10:34:52.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.774 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.774 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130141 s, 80.6 MB/s 2024-04-17T10:34:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.787 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124478 s, 84.2 MB/s 2024-04-17T10:34:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.803 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.804 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148375 s, 70.7 MB/s 2024-04-17T10:34:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146259 s, 71.7 MB/s 2024-04-17T10:34:52.836 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.837 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147344 s, 71.2 MB/s 2024-04-17T10:34:52.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.854 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155162 s, 67.6 MB/s 2024-04-17T10:34:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.870 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138943 s, 75.5 MB/s 2024-04-17T10:34:52.885 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.885 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.885 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013755 s, 76.2 MB/s 2024-04-17T10:34:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.901 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139233 s, 75.3 MB/s 2024-04-17T10:34:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185848 s, 56.4 MB/s 2024-04-17T10:34:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147556 s, 71.1 MB/s 2024-04-17T10:34:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137579 s, 76.2 MB/s 2024-04-17T10:34:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.971 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158579 s, 66.1 MB/s 2024-04-17T10:34:52.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:52.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:52.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140559 s, 74.6 MB/s 2024-04-17T10:34:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.003 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014375 s, 72.9 MB/s 2024-04-17T10:34:53.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.019 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141127 s, 74.3 MB/s 2024-04-17T10:34:53.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.034 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.034 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133179 s, 78.7 MB/s 2024-04-17T10:34:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.051 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151189 s, 69.4 MB/s 2024-04-17T10:34:53.072 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.073 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196482 s, 53.4 MB/s 2024-04-17T10:34:53.089 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.089 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.089 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144027 s, 72.8 MB/s 2024-04-17T10:34:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.106 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153448 s, 68.3 MB/s 2024-04-17T10:34:53.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145538 s, 72.0 MB/s 2024-04-17T10:34:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144585 s, 72.5 MB/s 2024-04-17T10:34:53.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:53.141 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:53.141 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:53.141 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.28 2024-04-17T10:34:53.141 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:53.141 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:53.141 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:53.141 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:53.214 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.215 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185451 s, 56.5 MB/s 2024-04-17T10:34:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.230 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014181 s, 73.9 MB/s 2024-04-17T10:34:53.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.247 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01474 s, 71.1 MB/s 2024-04-17T10:34:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.265 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158698 s, 66.1 MB/s 2024-04-17T10:34:53.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.281 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147778 s, 71.0 MB/s 2024-04-17T10:34:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132729 s, 79.0 MB/s 2024-04-17T10:34:53.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.313 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149267 s, 70.2 MB/s 2024-04-17T10:34:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161848 s, 64.8 MB/s 2024-04-17T10:34:53.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180345 s, 58.1 MB/s 2024-04-17T10:34:53.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.367 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.368 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150542 s, 69.7 MB/s 2024-04-17T10:34:53.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.383 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137238 s, 76.4 MB/s 2024-04-17T10:34:53.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.400 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154951 s, 67.7 MB/s 2024-04-17T10:34:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148982 s, 70.4 MB/s 2024-04-17T10:34:53.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.433 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143977 s, 72.8 MB/s 2024-04-17T10:34:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.450 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151134 s, 69.4 MB/s 2024-04-17T10:34:53.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139557 s, 75.1 MB/s 2024-04-17T10:34:53.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.483 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015657 s, 67.0 MB/s 2024-04-17T10:34:53.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.499 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141556 s, 74.1 MB/s 2024-04-17T10:34:53.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204875 s, 51.2 MB/s 2024-04-17T10:34:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136716 s, 76.7 MB/s 2024-04-17T10:34:53.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.557 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183513 s, 57.1 MB/s 2024-04-17T10:34:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.573 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140953 s, 74.4 MB/s 2024-04-17T10:34:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.590 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153612 s, 68.3 MB/s 2024-04-17T10:34:53.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157748 s, 66.5 MB/s 2024-04-17T10:34:53.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.627 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170879 s, 61.4 MB/s 2024-04-17T10:34:53.641 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013307 s, 78.8 MB/s 2024-04-17T10:34:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138777 s, 75.6 MB/s 2024-04-17T10:34:53.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.674 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145734 s, 72.0 MB/s 2024-04-17T10:34:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.689 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141415 s, 74.1 MB/s 2024-04-17T10:34:53.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183975 s, 57.0 MB/s 2024-04-17T10:34:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157019 s, 66.8 MB/s 2024-04-17T10:34:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185133 s, 56.6 MB/s 2024-04-17T10:34:53.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:53.749 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:53.749 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:53.749 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.29 2024-04-17T10:34:53.749 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:53.749 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:53.749 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:53.749 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134941 s, 77.7 MB/s 2024-04-17T10:34:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.838 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183915 s, 57.0 MB/s 2024-04-17T10:34:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.855 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014872 s, 70.5 MB/s 2024-04-17T10:34:53.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.870 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132709 s, 79.0 MB/s 2024-04-17T10:34:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155008 s, 67.6 MB/s 2024-04-17T10:34:53.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.903 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143029 s, 73.3 MB/s 2024-04-17T10:34:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.920 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153063 s, 68.5 MB/s 2024-04-17T10:34:53.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162819 s, 64.4 MB/s 2024-04-17T10:34:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132167 s, 79.3 MB/s 2024-04-17T10:34:53.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147691 s, 71.0 MB/s 2024-04-17T10:34:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156002 s, 67.2 MB/s 2024-04-17T10:34:54.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.003 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013905 s, 75.4 MB/s 2024-04-17T10:34:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138749 s, 75.6 MB/s 2024-04-17T10:34:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143444 s, 73.1 MB/s 2024-04-17T10:34:54.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.054 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177777 s, 59.0 MB/s 2024-04-17T10:34:54.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.071 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152758 s, 68.6 MB/s 2024-04-17T10:34:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156106 s, 67.2 MB/s 2024-04-17T10:34:54.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.105 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.105 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141473 s, 74.1 MB/s 2024-04-17T10:34:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.121 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140512 s, 74.6 MB/s 2024-04-17T10:34:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01512 s, 69.4 MB/s 2024-04-17T10:34:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.156 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169774 s, 61.8 MB/s 2024-04-17T10:34:54.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.175 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172683 s, 60.7 MB/s 2024-04-17T10:34:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138751 s, 75.6 MB/s 2024-04-17T10:34:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.210 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174247 s, 60.2 MB/s 2024-04-17T10:34:54.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145165 s, 72.2 MB/s 2024-04-17T10:34:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145764 s, 71.9 MB/s 2024-04-17T10:34:54.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.259 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.259 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143266 s, 73.2 MB/s 2024-04-17T10:34:54.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.274 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139458 s, 75.2 MB/s 2024-04-17T10:34:54.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.288 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115134 s, 91.1 MB/s 2024-04-17T10:34:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.305 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156247 s, 67.1 MB/s 2024-04-17T10:34:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.322 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153003 s, 68.5 MB/s 2024-04-17T10:34:54.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138085 s, 75.9 MB/s 2024-04-17T10:34:54.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:54.339 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:54.339 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:54.339 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.30 2024-04-17T10:34:54.340 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:54.340 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:54.340 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:54.340 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.410 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.411 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140398 s, 74.7 MB/s 2024-04-17T10:34:54.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.427 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152105 s, 68.9 MB/s 2024-04-17T10:34:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.443 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01371 s, 76.5 MB/s 2024-04-17T10:34:54.459 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.459 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.459 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141049 s, 74.3 MB/s 2024-04-17T10:34:54.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.476 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.476 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015557 s, 67.4 MB/s 2024-04-17T10:34:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015351 s, 68.3 MB/s 2024-04-17T10:34:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151309 s, 69.3 MB/s 2024-04-17T10:34:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139333 s, 75.3 MB/s 2024-04-17T10:34:54.542 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.543 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147578 s, 71.1 MB/s 2024-04-17T10:34:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187788 s, 55.8 MB/s 2024-04-17T10:34:54.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.580 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.580 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151352 s, 69.3 MB/s 2024-04-17T10:34:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.596 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139079 s, 75.4 MB/s 2024-04-17T10:34:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153562 s, 68.3 MB/s 2024-04-17T10:34:54.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.629 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.629 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140967 s, 74.4 MB/s 2024-04-17T10:34:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.646 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153759 s, 68.2 MB/s 2024-04-17T10:34:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144078 s, 72.8 MB/s 2024-04-17T10:34:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.678 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140646 s, 74.6 MB/s 2024-04-17T10:34:54.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.694 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134723 s, 77.8 MB/s 2024-04-17T10:34:54.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.711 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.711 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152068 s, 69.0 MB/s 2024-04-17T10:34:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139417 s, 75.2 MB/s 2024-04-17T10:34:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.744 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015614 s, 67.2 MB/s 2024-04-17T10:34:54.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.764 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.764 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185192 s, 56.6 MB/s 2024-04-17T10:34:54.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170179 s, 61.6 MB/s 2024-04-17T10:34:54.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148953 s, 70.4 MB/s 2024-04-17T10:34:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.815 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136454 s, 76.8 MB/s 2024-04-17T10:34:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.832 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153208 s, 68.4 MB/s 2024-04-17T10:34:54.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143105 s, 73.3 MB/s 2024-04-17T10:34:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.864 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145988 s, 71.8 MB/s 2024-04-17T10:34:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139628 s, 75.1 MB/s 2024-04-17T10:34:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146922 s, 71.4 MB/s 2024-04-17T10:34:54.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.915 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162314 s, 64.6 MB/s 2024-04-17T10:34:54.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:54.931 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:54.931 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143957 s, 72.8 MB/s 2024-04-17T10:34:54.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:54.932 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:34:54.932 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:34:54.933 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.31 2024-04-17T10:34:54.933 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:34:54.933 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:34:54.933 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:34:54.933 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:34:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.003 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149989 s, 69.9 MB/s 2024-04-17T10:34:55.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.019 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014303 s, 73.3 MB/s 2024-04-17T10:34:55.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.039 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.039 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183112 s, 57.3 MB/s 2024-04-17T10:34:55.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.055 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142131 s, 73.8 MB/s 2024-04-17T10:34:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.070 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139041 s, 75.4 MB/s 2024-04-17T10:34:55.087 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.088 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.088 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153139 s, 68.5 MB/s 2024-04-17T10:34:55.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.101 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.101 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116691 s, 89.9 MB/s 2024-04-17T10:34:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.117 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147927 s, 70.9 MB/s 2024-04-17T10:34:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140342 s, 74.7 MB/s 2024-04-17T10:34:55.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.150 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139951 s, 74.9 MB/s 2024-04-17T10:34:55.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.165 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.165 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136881 s, 76.6 MB/s 2024-04-17T10:34:55.180 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.180 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.180 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135861 s, 77.2 MB/s 2024-04-17T10:34:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138546 s, 75.7 MB/s 2024-04-17T10:34:55.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.211 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134984 s, 77.7 MB/s 2024-04-17T10:34:55.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.226 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.226 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129061 s, 81.2 MB/s 2024-04-17T10:34:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153347 s, 68.4 MB/s 2024-04-17T10:34:55.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.259 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138163 s, 75.9 MB/s 2024-04-17T10:34:55.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.276 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.276 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152469 s, 68.8 MB/s 2024-04-17T10:34:55.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.291 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.291 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133351 s, 78.6 MB/s 2024-04-17T10:34:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130843 s, 80.1 MB/s 2024-04-17T10:34:55.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.324 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.325 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167533 s, 62.6 MB/s 2024-04-17T10:34:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145508 s, 72.1 MB/s 2024-04-17T10:34:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132461 s, 79.2 MB/s 2024-04-17T10:34:55.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.377 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.377 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188289 s, 55.7 MB/s 2024-04-17T10:34:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014453 s, 72.6 MB/s 2024-04-17T10:34:55.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.414 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189651 s, 55.3 MB/s 2024-04-17T10:34:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.429 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139966 s, 74.9 MB/s 2024-04-17T10:34:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.445 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141789 s, 74.0 MB/s 2024-04-17T10:34:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.462 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149817 s, 70.0 MB/s 2024-04-17T10:34:55.478 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.479 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148075 s, 70.8 MB/s 2024-04-17T10:34:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181515 s, 57.8 MB/s 2024-04-17T10:34:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:34:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:34:55.515 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147091 s, 71.3 MB/s 2024-04-17T10:34:55.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:55.516 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.9:0 scrub start /scrub_control_test_path recursive 2024-04-17T10:34:55.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffba2800640 1 Processor -- start 2024-04-17T10:34:55.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffba2800640 1 -- start start 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffba2800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb9c06d3c0 0x7ffb9c071360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffba2800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c0cc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffba2800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c06d9a0 con 0x7ffb9c1447e0 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffba2800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c071cf0 con 0x7ffb9c06d3c0 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffba2800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0cc310 con 0x7ffb9c0718a0 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb9c06d3c0 0x7ffb9c071360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffb9c06d3c0 0x7ffb9c071360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35948/0 (socket says 172.21.15.67:35948) 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9aa00640 1 -- 172.21.15.67:0/2004157236 learned_addr learned my addr 172.21.15.67:0/2004157236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2004157236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c0cc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9aa00640 1 -- 172.21.15.67:0/2004157236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 msgr2=0x7ffb9c0cc7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/2004157236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c0cc7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9aa00640 1 -- 172.21.15.67:0/2004157236 >> v1:172.21.15.86:6789/0 conn(0x7ffb9c1447e0 legacy=0x7ffb9c144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.712+0000 7ffb9aa00640 1 -- 172.21.15.67:0/2004157236 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c06dc80 con 0x7ffb9c06d3c0 2024-04-17T10:34:55.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/2004157236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c06d3c0 0x7ffb9c071360 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7ffb88009870 tx=0x7ffb8801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4118148a237be48 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffb9a000640 1 -- 172.21.15.67:0/2004157236 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb880aa600 con 0x7ffb9c06d3c0 2024-04-17T10:34:55.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffb9a000640 1 -- 172.21.15.67:0/2004157236 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb880af070 con 0x7ffb9c06d3c0 2024-04-17T10:34:55.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffb9a000640 1 -- 172.21.15.67:0/2004157236 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb880aa9c0 con 0x7ffb9c06d3c0 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 -- 172.21.15.67:0/2004157236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c06d3c0 msgr2=0x7ffb9c071360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 --2- 172.21.15.67:0/2004157236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c06d3c0 0x7ffb9c071360 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7ffb88009870 tx=0x7ffb8801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 -- 172.21.15.67:0/2004157236 shutdown_connections 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 --2- 172.21.15.67:0/2004157236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c0cc7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 --2- 172.21.15.67:0/2004157236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c06d3c0 0x7ffb9c071360 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 -- 172.21.15.67:0/2004157236 >> 172.21.15.67:0/2004157236 conn(0x7ffb9c06ac90 msgr2=0x7ffb9c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 -- 172.21.15.67:0/2004157236 shutdown_connections 2024-04-17T10:34:55.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 -- 172.21.15.67:0/2004157236 wait complete. 2024-04-17T10:34:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 Processor -- start 2024-04-17T10:34:55.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.716+0000 7ffba2800640 1 -- start start 2024-04-17T10:34:55.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb9c06d3c0 0x7ffb9c0ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c14e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c1447e0 0x7ffb9c13bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0cd880 con 0x7ffb9c1447e0 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0cd700 con 0x7ffb9c06d3c0 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0cda00 con 0x7ffb9c0718a0 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c1447e0 0x7ffb9c13bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb9c06d3c0 0x7ffb9c0ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9aa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c14e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c1447e0 0x7ffb9c13bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35954/0 (socket says 172.21.15.67:35954) 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 -- 172.21.15.67:0/2931373476 learned_addr learned my addr 172.21.15.67:0/2931373476 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 -- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 msgr2=0x7ffb9c14e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c14e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 -- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb9c06d3c0 msgr2=0x7ffb9c0ad570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb9c06d3c0 0x7ffb9c0ad570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 -- 172.21.15.67:0/2931373476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c0adab0 con 0x7ffb9c1447e0 2024-04-17T10:34:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c14e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c1447e0 0x7ffb9c13bea0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009dd0 tx=0x7ffb9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb93400640 1 -- 172.21.15.67:0/2931373476 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94018020 con 0x7ffb9c1447e0 2024-04-17T10:34:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb93400640 1 -- 172.21.15.67:0/2931373476 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94014480 con 0x7ffb9c1447e0 2024-04-17T10:34:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffb93400640 1 -- 172.21.15.67:0/2931373476 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94014870 con 0x7ffb9c1447e0 2024-04-17T10:34:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c0add50 con 0x7ffb9c1447e0 2024-04-17T10:34:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.720+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c0b2d60 con 0x7ffb9c1447e0 2024-04-17T10:34:55.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb93400640 1 -- 172.21.15.67:0/2931373476 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ffb9401b020 con 0x7ffb9c1447e0 2024-04-17T10:34:55.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb93400640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffb7c066f60 0x7ffb7c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb93400640 1 -- 172.21.15.67:0/2931373476 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffb94085c50 con 0x7ffb9c1447e0 2024-04-17T10:34:55.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 Processor -- start 2024-04-17T10:34:55.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 -- start start 2024-04-17T10:34:55.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffb7c066f60 0x7ffb7c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffb7c066f60 0x7ffb7c069420 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c001810 tx=0x7ffb8c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd0257d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfdbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bce29d50 con 0x5600bd025d10 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bcdb1e40 con 0x5600bd0253f0 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600bce25df0 con 0x5600bcfdc520 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfdbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfdbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35970/0 (socket says 172.21.15.67:35970) 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 -- 172.21.15.67:0/3688407822 learned_addr learned my addr 172.21.15.67:0/3688407822 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9b400640 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd0257d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9be00640 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 -- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 msgr2=0x5600bcfdc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 -- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 msgr2=0x5600bd0257d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd0257d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.724+0000 7ffb9aa00640 1 -- 172.21.15.67:0/3688407822 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600bcdc6d90 con 0x5600bd025d10 2024-04-17T10:34:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfdbfe0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7ffb8801a0d0 tx=0x7ffb88004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fd86b0b735eac9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb87e00640 1 -- 172.21.15.67:0/3688407822 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb880aadb0 con 0x5600bd025d10 2024-04-17T10:34:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb87e00640 1 -- 172.21.15.67:0/3688407822 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb880af040 con 0x5600bd025d10 2024-04-17T10:34:55.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb87e00640 1 -- 172.21.15.67:0/3688407822 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb880bd5f0 con 0x5600bd025d10 2024-04-17T10:34:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- 172.21.15.67:0/3688407822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 msgr2=0x5600bcfdbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfdbfe0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7ffb8801a0d0 tx=0x7ffb88004740 comp rx=0 tx=0).stop 2024-04-17T10:34:55.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- 172.21.15.67:0/3688407822 shutdown_connections 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfdbfe0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/3688407822 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd0257d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- 172.21.15.67:0/3688407822 >> 172.21.15.67:0/3688407822 conn(0x5600bcfd84d0 msgr2=0x5600bcfda8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- 172.21.15.67:0/3688407822 shutdown_connections 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- 172.21.15.67:0/3688407822 wait complete. 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- start start 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 Processor -- start 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bcfdc520 0x5600bd02c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd0253f0 0x5600bd02ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bcdcf610 con 0x5600bd025d10 2024-04-17T10:34:55.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bcde49a0 con 0x5600bcfdc520 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600bce2e380 con 0x5600bd0253f0 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffba5ce9000 10 client.0 _ll_register_callbacks cb 0x5600bcede0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35972/0 (socket says 172.21.15.67:35972) 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 -- 172.21.15.67:0/2529895256 learned_addr learned my addr 172.21.15.67:0/2529895256 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bcfdc520 0x5600bd02c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd0253f0 0x5600bd02ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 -- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd0253f0 msgr2=0x5600bd02ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd0253f0 0x5600bd02ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 -- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bcfdc520 msgr2=0x5600bd02c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bcfdc520 0x5600bd02c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 -- 172.21.15.67:0/2529895256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600bce24d40 con 0x5600bd025d10 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe6bf0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009a40 tx=0x7ffb9405ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71f1f6cbc16537a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.728+0000 7ffb84c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:55.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 1 -- 172.21.15.67:0/2529895256 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9405e7b0 con 0x5600bd025d10 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb880af040 con 0x5600bd025d10 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 1 -- 172.21.15.67:0/2529895256 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffb940557a0 con 0x5600bd025d10 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 1 -- 172.21.15.67:0/2529895256 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb940559a0 con 0x5600bd025d10 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 10 client.9447 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x5600bceacf00 con 0x5600bd025d10 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 1 -- 172.21.15.67:0/2529895256 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb880af040 con 0x5600bd025d10 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 1 -- 172.21.15.67:0/2529895256 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ffb9406b070 con 0x5600bd025d10 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 10 client.9447 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 10 client.9447 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.d 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 20 client.9447 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bcfe78d0 0x5600bcfe7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 4 client.9447 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600bcdc5a20 con 0x5600bcfe78d0 2024-04-17T10:34:55.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bcfe78d0 0x5600bcfe7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bcfe78d0 0x5600bcfe7cb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c13b5ead3c9f3bb4 server_cookie=6e50cb813b7f8fb0 in_seq=0 out_seq=0 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 10 client.9447 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 1 -- 172.21.15.67:0/2529895256 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5600bcdc5a20 con 0x5600bcfe78d0 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 10 client.9447 handle_command_reply: tid=0 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 client.9447 shutdown 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 msgr2=0x5600bcfe6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe6bf0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009a40 tx=0x7ffb9405ea40 comp rx=0 tx=0).stop 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 shutdown_connections 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bcfe78d0 0x5600bcfe7cb0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe6bf0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd0253f0 0x5600bd02ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffb84c00640 0 client.9447 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2529895256 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bcfdc520 0x5600bd02c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 >> 172.21.15.67:0/2529895256 conn(0x5600bcfd84d0 msgr2=0x5600bcfdb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 shutdown_connections 2024-04-17T10:34:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2529895256 wait complete. 2024-04-17T10:34:55.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.732+0000 7ffba5ce9000 20 client.9447 trim_cache size 0 max 16384 2024-04-17T10:34:55.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 Processor -- start 2024-04-17T10:34:55.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 -- start start 2024-04-17T10:34:55.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bcfe1da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c203070 con 0x5600bd025d10 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0b43a0 con 0x5600bd0253f0 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0b37a0 con 0x5600bcfdc520 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35974/0 (socket says 172.21.15.67:35974) 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 -- 172.21.15.67:0/1589844021 learned_addr learned my addr 172.21.15.67:0/1589844021 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9b400640 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bcfe1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 -- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 msgr2=0x5600bcfdbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 -- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 msgr2=0x5600bcfe1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bcfe1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 -- 172.21.15.67:0/1589844021 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600bce83330 con 0x5600bd025d10 2024-04-17T10:34:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.748+0000 7ffb9be00640 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe22e0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7ffb9405a360 tx=0x7ffb9405a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80511edc1a3fcec2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb87400640 1 -- 172.21.15.67:0/1589844021 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9405e7b0 con 0x5600bd025d10 2024-04-17T10:34:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb87400640 1 -- 172.21.15.67:0/1589844021 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94082070 con 0x5600bd025d10 2024-04-17T10:34:55.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb87400640 1 -- 172.21.15.67:0/1589844021 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb94083e90 con 0x5600bd025d10 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- 172.21.15.67:0/1589844021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 msgr2=0x5600bcfe22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe22e0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7ffb9405a360 tx=0x7ffb9405a000 comp rx=0 tx=0).stop 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- 172.21.15.67:0/1589844021 shutdown_connections 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bd025d10 0x5600bcfe22e0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bcfe1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/1589844021 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bcfdc520 0x5600bcfdbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- 172.21.15.67:0/1589844021 >> 172.21.15.67:0/1589844021 conn(0x5600bcfd84d0 msgr2=0x5600bd023e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- 172.21.15.67:0/1589844021 shutdown_connections 2024-04-17T10:34:55.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- 172.21.15.67:0/1589844021 wait complete. 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- start start 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 Processor -- start 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bcfdc520 0x5600bd029540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd042c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd025d10 0x5600bd043140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bcde0dc0 con 0x5600bcfdc520 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0b3da0 con 0x5600bd0253f0 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5600bce29060 con 0x5600bd025d10 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 10 client.0 _ll_register_callbacks cb 0x7ffb880096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bcfdc520 0x5600bd029540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bcfdc520 0x5600bd029540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35988/0 (socket says 172.21.15.67:35988) 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 -- 172.21.15.67:0/2612073554 learned_addr learned my addr 172.21.15.67:0/2612073554 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd042c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9be00640 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd025d10 0x5600bd043140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 -- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd025d10 msgr2=0x5600bd043140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd025d10 0x5600bd043140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 -- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 msgr2=0x5600bd042c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd042c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 -- 172.21.15.67:0/2612073554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600bcdf65f0 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb9b400640 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bcfdc520 0x5600bd029540 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x5600bcdd0940 tx=0x7ffb8c000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a3dd8907bb7a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb84c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb84c00640 1 -- 172.21.15.67:0/2612073554 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c0069f0 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffb84c00640 1 -- 172.21.15.67:0/2612073554 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffb8c010040 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.752+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb94082070 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffb84c00640 1 -- 172.21.15.67:0/2612073554 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb8c037050 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 10 client.9453 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x5600bcdd0fe0 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffb84c00640 1 -- 172.21.15.67:0/2612073554 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb94082070 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffb84c00640 1 -- 172.21.15.67:0/2612073554 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ffb8c03a020 con 0x5600bcfdc520 2024-04-17T10:34:55.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 10 client.9453 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 10 client.9453 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.d 2024-04-17T10:34:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 20 client.9453 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bd043680 0x5600bd08e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 4 client.9453 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-04-17T10:34:55.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x5600bce37290 con 0x5600bd043680 2024-04-17T10:34:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bd043680 0x5600bd08e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffb9aa00640 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bd043680 0x5600bd08e070 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=525fdac5dca9ea54 server_cookie=69944191e26a5ebf in_seq=0 out_seq=0 2024-04-17T10:34:55.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.756+0000 7ffb84c00640 10 client.9453 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffb84c00640 1 -- 172.21.15.67:0/2612073554 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5600bce37290 con 0x5600bd043680 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffb84c00640 10 client.9453 handle_command_reply: tid=0 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 client.9453 shutdown 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bcfdc520 msgr2=0x5600bd029540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bcfdc520 0x5600bd029540 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x5600bcdd0940 tx=0x7ffb8c000bd0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 shutdown_connections 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5600bd043680 0x5600bd08e070 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5600bd025d10 0x5600bd043140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5600bd0253f0 0x5600bd042c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 --2- 172.21.15.67:0/2612073554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5600bcfdc520 0x5600bd029540 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffb84c00640 0 client.9453 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 >> 172.21.15.67:0/2612073554 conn(0x5600bcfd84d0 msgr2=0x5600bd023e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 shutdown_connections 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 1 -- 172.21.15.67:0/2612073554 wait complete. 2024-04-17T10:34:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba5ce9000 20 client.9453 trim_cache size 0 max 16384 2024-04-17T10:34:55.753 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0, 2024-04-17T10:34:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "scrub_tag": "3598e48a-86ce-44d6-8826-5b2339eddced", 2024-04-17T10:34:55.753 INFO:teuthology.orchestra.run.smithi067.stdout: "mode": "asynchronous" 2024-04-17T10:34:55.753 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffb7c066f60 msgr2=0x7ffb7c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffb7c066f60 0x7ffb7c069420 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c001810 tx=0x7ffb8c009040 comp rx=0 tx=0).stop 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c1447e0 msgr2=0x7ffb9c13bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c1447e0 0x7ffb9c13bea0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7ffb94009dd0 tx=0x7ffb9400c490 comp rx=0 tx=0).stop 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 shutdown_connections 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffb7c066f60 0x7ffb7c069420 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffb9c1447e0 0x7ffb9c13bea0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffb9c0718a0 0x7ffb9c14e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 --2- 172.21.15.67:0/2931373476 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffb9c06d3c0 0x7ffb9c0ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 >> 172.21.15.67:0/2931373476 conn(0x7ffb9c06ac90 msgr2=0x7ffb9c143940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 shutdown_connections 2024-04-17T10:34:55.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.760+0000 7ffba2800640 1 -- 172.21.15.67:0/2931373476 wait complete. 2024-04-17T10:34:55.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:55.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.9:0 scrub pause 2024-04-17T10:34:55.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.936+0000 7f72c9000640 1 Processor -- start 2024-04-17T10:34:55.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.936+0000 7f72c9000640 1 -- start start 2024-04-17T10:34:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c9000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f72c4159440 0x7f72c40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c9000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c4071870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c9000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72c415cb30 con 0x7f72c40713e0 2024-04-17T10:34:55.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c9000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72c40a8650 con 0x7f72c4159440 2024-04-17T10:34:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c9000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c415ce10 con 0x7f72c415c580 2024-04-17T10:34:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f72c4159440 0x7f72c40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72b9400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f72c40713e0 0x7f72c415be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47078/0 (socket says 172.21.15.67:47078) 2024-04-17T10:34:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72b9400640 1 -- 172.21.15.67:0/2933134842 learned_addr learned my addr 172.21.15.67:0/2933134842 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c2800640 1 --2- 172.21.15.67:0/2933134842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c4071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c1e00640 1 -- 172.21.15.67:0/2933134842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 msgr2=0x7f72c4071870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c1e00640 1 --2- 172.21.15.67:0/2933134842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c4071870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c1e00640 1 -- 172.21.15.67:0/2933134842 >> v1:172.21.15.67:6789/0 conn(0x7f72c40713e0 legacy=0x7f72c415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c1e00640 1 -- 172.21.15.67:0/2933134842 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c40cd610 con 0x7f72c4159440 2024-04-17T10:34:55.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.940+0000 7f72c2800640 1 --2- 172.21.15.67:0/2933134842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c4071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:34:55.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c1e00640 1 --2- 172.21.15.67:0/2933134842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c4159440 0x7f72c40ce0f0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f72b40097c0 tx=0x7f72b401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41d5f198827a29b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c1400640 1 -- 172.21.15.67:0/2933134842 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40b1070 con 0x7f72c4159440 2024-04-17T10:34:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c1400640 1 -- 172.21.15.67:0/2933134842 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b40ac410 con 0x7f72c4159440 2024-04-17T10:34:55.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c1400640 1 -- 172.21.15.67:0/2933134842 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40ac7e0 con 0x7f72c4159440 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 -- 172.21.15.67:0/2933134842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c4159440 msgr2=0x7f72c40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 --2- 172.21.15.67:0/2933134842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c4159440 0x7f72c40ce0f0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f72b40097c0 tx=0x7f72b401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 -- 172.21.15.67:0/2933134842 shutdown_connections 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 --2- 172.21.15.67:0/2933134842 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c4071870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 --2- 172.21.15.67:0/2933134842 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c4159440 0x7f72c40ce0f0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 -- 172.21.15.67:0/2933134842 >> 172.21.15.67:0/2933134842 conn(0x7f72c40b1740 msgr2=0x7f72c40b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 -- 172.21.15.67:0/2933134842 shutdown_connections 2024-04-17T10:34:55.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 -- 172.21.15.67:0/2933134842 wait complete. 2024-04-17T10:34:55.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 Processor -- start 2024-04-17T10:34:55.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 -- start start 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c40713e0 0x7f72c413c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72c4159440 0x7f72c4157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.944+0000 7f72c9000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c41580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c9000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c415d9c0 con 0x7f72c40713e0 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c9000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c415d840 con 0x7f72c4159440 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c9000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c415db40 con 0x7f72c415c580 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c40713e0 0x7f72c413c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c40713e0 0x7f72c413c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36004/0 (socket says 172.21.15.67:36004) 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 -- 172.21.15.67:0/1615795405 learned_addr learned my addr 172.21.15.67:0/1615795405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c2800640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c41580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72b9400640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72c4159440 0x7f72c4157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 -- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 msgr2=0x7f72c41580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c41580d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 -- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72c4159440 msgr2=0x7f72c4157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72c4159440 0x7f72c4157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 -- 172.21.15.67:0/1615795405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c413cc90 con 0x7f72c40713e0 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c40713e0 0x7f72c413c260 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f72b40af040 tx=0x7f72b4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72baa00640 1 -- 172.21.15.67:0/1615795405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40b1040 con 0x7f72c40713e0 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72baa00640 1 -- 172.21.15.67:0/1615795405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b40bb070 con 0x7f72c40713e0 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72baa00640 1 -- 172.21.15.67:0/1615795405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40b5670 con 0x7f72c40713e0 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c413cf70 con 0x7f72c40713e0 2024-04-17T10:34:55.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c413a6a0 con 0x7f72c40713e0 2024-04-17T10:34:55.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.948+0000 7f72baa00640 1 -- 172.21.15.67:0/1615795405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f72b40b5810 con 0x7f72c40713e0 2024-04-17T10:34:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72baa00640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f72a0066f60 0x7f72a0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72baa00640 1 -- 172.21.15.67:0/1615795405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f72b411eca0 con 0x7f72c40713e0 2024-04-17T10:34:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b9400640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f72a0066f60 0x7f72a0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 Processor -- start 2024-04-17T10:34:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- start start 2024-04-17T10:34:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b9400640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f72a0066f60 0x7f72a0069420 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f72c41587d0 tx=0x7f72ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc0ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecbc9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecbca170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565ecaac1f0 con 0x5565ecc0f360 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eca12e90 con 0x5565ecc0ea40 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565ec99af50 con 0x5565ecbc9d90 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecbc9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecbc9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36010/0 (socket says 172.21.15.67:36010) 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc0ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b9400640 1 -- 172.21.15.67:0/1973540908 learned_addr learned my addr 172.21.15.67:0/1973540908 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c2800640 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecbca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c1e00640 1 -- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 msgr2=0x5565ecbca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecbca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c1e00640 1 -- 172.21.15.67:0/1973540908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 msgr2=0x5565ecbc9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecbc9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c1e00640 1 -- 172.21.15.67:0/1973540908 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565ec9ea8b0 con 0x5565ecc0ea40 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b9400640 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecbc9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc0ee20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f72b411cfa0 tx=0x7f72b40eece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb9ebba475f01ea server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b2a00640 1 -- 172.21.15.67:0/1973540908 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40ef380 con 0x5565ecc0ea40 2024-04-17T10:34:55.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b2a00640 1 -- 172.21.15.67:0/1973540908 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b40f34d0 con 0x5565ecc0ea40 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72b2a00640 1 -- 172.21.15.67:0/1973540908 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b40f7a90 con 0x5565ecc0ea40 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 msgr2=0x5565ecc0ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc0ee20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f72b411cfa0 tx=0x7f72b40eece0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- 172.21.15.67:0/1973540908 shutdown_connections 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecbca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecbc9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1973540908 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc0ee20 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- 172.21.15.67:0/1973540908 >> 172.21.15.67:0/1973540908 conn(0x5565ecbc5d40 msgr2=0x5565ecbc8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- 172.21.15.67:0/1973540908 shutdown_connections 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.952+0000 7f72cc59e000 1 -- 172.21.15.67:0/1973540908 wait complete. 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 -- start start 2024-04-17T10:34:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 Processor -- start 2024-04-17T10:34:55.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc161c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc16700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecbcfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565ec9cd540 con 0x5565ecc0ea40 2024-04-17T10:34:55.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565eca0ef30 con 0x5565ecbc9d90 2024-04-17T10:34:55.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565eca12e90 con 0x5565ecc0f360 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 10 client.0 _ll_register_callbacks cb 0x5565ecac7180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc16700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc16700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36012/0 (socket says 172.21.15.67:36012) 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 -- 172.21.15.67:0/2931704246 learned_addr learned my addr 172.21.15.67:0/2931704246 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72c2800640 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecbcfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72c1e00640 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc161c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 -- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 msgr2=0x5565ecbcfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecbcfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 -- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 msgr2=0x5565ecc161c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc161c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 -- 172.21.15.67:0/2931704246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565ec99b1a0 con 0x5565ecc0ea40 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72b9400640 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc16700 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0076b0 tx=0x7f72ac001260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9d203922ea842a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72a7400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72a7400640 1 -- 172.21.15.67:0/2931704246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac027060 con 0x5565ecc0ea40 2024-04-17T10:34:55.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72b40f34d0 con 0x5565ecc0ea40 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72a7400640 1 -- 172.21.15.67:0/2931704246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f72ac010040 con 0x5565ecc0ea40 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72a7400640 1 -- 172.21.15.67:0/2931704246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72ac02d050 con 0x5565ecc0ea40 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 10 client.9468 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x5565eca95840 con 0x5565ecc0ea40 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72a7400640 1 -- 172.21.15.67:0/2931704246 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b40f34d0 con 0x5565ecc0ea40 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72a7400640 1 -- 172.21.15.67:0/2931704246 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f72ac030020 con 0x5565ecc0ea40 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 10 client.9468 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 10 client.9468 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.d 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 20 client.9468 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecbd0c30 0x5565ecbd1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 4 client.9468 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565ec9aebe0 con 0x5565ecbd0c30 2024-04-17T10:34:55.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.956+0000 7f72c1e00640 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecbd0c30 0x5565ecbd1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72c1e00640 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecbd0c30 0x5565ecbd1040 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b79ef3a56b8d09 server_cookie=ca5d5bbec4f260a1 in_seq=0 out_seq=0 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72a7400640 10 client.9468 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72a7400640 1 -- 172.21.15.67:0/2931704246 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5565ec9aebe0 con 0x5565ecbd0c30 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72a7400640 10 client.9468 handle_command_reply: tid=0 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 client.9468 shutdown 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 msgr2=0x5565ecc16700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc16700 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f72ac0076b0 tx=0x7f72ac001260 comp rx=0 tx=0).stop 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 shutdown_connections 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecbd0c30 0x5565ecbd1040 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecbcfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc16700 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 --2- 172.21.15.67:0/2931704246 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc161c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72a7400640 0 client.9468 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 >> 172.21.15.67:0/2931704246 conn(0x5565ecbc5d40 msgr2=0x5565ecbc89c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 shutdown_connections 2024-04-17T10:34:55.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 1 -- 172.21.15.67:0/2931704246 wait complete. 2024-04-17T10:34:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.960+0000 7f72cc59e000 20 client.9468 trim_cache size 0 max 16384 2024-04-17T10:34:55.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 Processor -- start 2024-04-17T10:34:55.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- start start 2024-04-17T10:34:55.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc6b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc6b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecc6bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4203240 con 0x5565ecc0ea40 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c42023e0 con 0x5565ecbc9d90 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72c413b980 con 0x5565ecc0f360 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc6b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72c1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc6b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc6b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36020/0 (socket says 172.21.15.67:36020) 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 -- 172.21.15.67:0/4011856608 learned_addr learned my addr 172.21.15.67:0/4011856608 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72c2800640 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecc6bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 -- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 msgr2=0x5565ecc6bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecc6bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 -- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 msgr2=0x5565ecc6b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc6b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 -- 172.21.15.67:0/4011856608 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565eca6ca10 con 0x5565ecc0ea40 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72c1e00640 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc6b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b9400640 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc6b9e0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f72ac000f90 tx=0x7f72ac01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9ed9f1197256d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b2000640 1 -- 172.21.15.67:0/4011856608 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac014800 con 0x5565ecc0ea40 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b2000640 1 -- 172.21.15.67:0/4011856608 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac019070 con 0x5565ecc0ea40 2024-04-17T10:34:55.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72b2000640 1 -- 172.21.15.67:0/4011856608 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ac022650 con 0x5565ecc0ea40 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- 172.21.15.67:0/4011856608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 msgr2=0x5565ecc6b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc6b9e0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f72ac000f90 tx=0x7f72ac01d000 comp rx=0 tx=0).stop 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- 172.21.15.67:0/4011856608 shutdown_connections 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecc0f360 0x5565ecc6bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0ea40 0x5565ecc6b9e0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 --2- 172.21.15.67:0/4011856608 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecbc9d90 0x5565ecc6b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- 172.21.15.67:0/4011856608 >> 172.21.15.67:0/4011856608 conn(0x5565ecbc5d40 msgr2=0x5565ecc0d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- 172.21.15.67:0/4011856608 shutdown_connections 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- 172.21.15.67:0/4011856608 wait complete. 2024-04-17T10:34:55.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 -- start start 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.976+0000 7f72cc59e000 1 Processor -- start 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecc6f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc6f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecc13ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565ecacfd10 con 0x5565ecc0f360 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4201ee0 con 0x5565ecc0ea40 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565eca0ef30 con 0x5565ecbc9d90 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 10 client.0 _ll_register_callbacks cb 0x5565ec9d5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecc13ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecc13ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36026/0 (socket says 172.21.15.67:36026) 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 -- 172.21.15.67:0/1169753809 learned_addr learned my addr 172.21.15.67:0/1169753809 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72b9400640 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc6f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecc6f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 -- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 msgr2=0x5565ecc6f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecc6f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 -- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 msgr2=0x5565ecc6f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc6f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 -- 172.21.15.67:0/1169753809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565ec9a7b00 con 0x5565ecc0f360 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c2800640 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecc13ed0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f72bc016a50 tx=0x7f72bc016b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287f4af2d8266e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 1 -- 172.21.15.67:0/1169753809 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc004030 con 0x5565ecc0f360 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 1 -- 172.21.15.67:0/1169753809 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f72bc0041d0 con 0x5565ecc0f360 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72ac019070 con 0x5565ecc0f360 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 1 -- 172.21.15.67:0/1169753809 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72bc028050 con 0x5565ecc0f360 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 10 client.9474 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x5565ec9b9f80 con 0x5565ecc0f360 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 1 -- 172.21.15.67:0/1169753809 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac019070 con 0x5565ecc0f360 2024-04-17T10:34:55.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 1 -- 172.21.15.67:0/1169753809 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f72bc013880 con 0x5565ecc0f360 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 10 client.9474 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 10 client.9474 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.d 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 20 client.9474 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecc14d20 0x5565ecc15130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 4 client.9474 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "scrub pause"}) v1 -- 0x5565eca203d0 con 0x5565ecc14d20 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecc14d20 0x5565ecc15130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 10 client.9474 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72c1e00640 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecc14d20 0x5565ecc15130 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9019bce2d342e18 server_cookie=c4730a42a78458b8 in_seq=0 out_seq=0 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 1 -- 172.21.15.67:0/1169753809 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x5565eca203d0 con 0x5565ecc14d20 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 10 client.9474 handle_command_reply: tid=0 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 client.9474 shutdown 2024-04-17T10:34:55.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 msgr2=0x5565ecc13ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecc13ed0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f72bc016a50 tx=0x7f72bc016b40 comp rx=0 tx=0).stop 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 shutdown_connections 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x5565ecc14d20 0x5565ecc15130 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5565ecc0f360 0x5565ecc13ed0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x5565ecc0ea40 0x5565ecc6f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 --2- 172.21.15.67:0/1169753809 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x5565ecbc9d90 0x5565ecc6f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72a7400640 0 client.9474 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 >> 172.21.15.67:0/1169753809 conn(0x5565ecbc5d40 msgr2=0x5565ecc0da80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.980+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 shutdown_connections 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72cc59e000 1 -- 172.21.15.67:0/1169753809 wait complete. 2024-04-17T10:34:55.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72cc59e000 20 client.9474 trim_cache size 0 max 16384 2024-04-17T10:34:55.980 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:55.980 INFO:teuthology.orchestra.run.smithi067.stdout: "return_code": 0 2024-04-17T10:34:55.980 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f72a0066f60 msgr2=0x7f72a0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f72a0066f60 0x7f72a0069420 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f72c41587d0 tx=0x7f72ac009040 comp rx=0 tx=0).stop 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c40713e0 msgr2=0x7f72c413c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c40713e0 0x7f72c413c260 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f72b40af040 tx=0x7f72b4019240 comp rx=0 tx=0).stop 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 shutdown_connections 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f72a0066f60 0x7f72a0069420 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f72c415c580 0x7f72c41580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f72c4159440 0x7f72c4157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 --2- 172.21.15.67:0/1615795405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f72c40713e0 0x7f72c413c260 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 >> 172.21.15.67:0/1615795405 conn(0x7f72c40b1740 msgr2=0x7f72c4154d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 shutdown_connections 2024-04-17T10:34:55.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:55.984+0000 7f72c9000640 1 -- 172.21.15.67:0/1615795405 wait complete. 2024-04-17T10:34:55.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:34:55.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.9:0 scrub status 2024-04-17T10:34:56.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.144+0000 7fd55e200640 1 Processor -- start 2024-04-17T10:34:56.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.144+0000 7fd55e200640 1 -- start start 2024-04-17T10:34:56.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd55e200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd558147360 0x7fd55806e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd55e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd558147740 0x7fd5580717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd55e200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd558161730 con 0x7fd55815ede0 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd55e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5581619d0 con 0x7fd558147740 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd55e200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd558161cb0 con 0x7fd558147360 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd54ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd558147740 0x7fd5580717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd54ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd558147740 0x7fd5580717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36036/0 (socket says 172.21.15.67:36036) 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd54ea00640 1 -- 172.21.15.67:0/2982002730 learned_addr learned my addr 172.21.15.67:0/2982002730 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd557400640 1 --2- 172.21.15.67:0/2982002730 >> v2:172.21.15.86:3300/0 conn(0x7fd558147360 0x7fd55806e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd54ea00640 1 -- 172.21.15.67:0/2982002730 >> v2:172.21.15.86:3300/0 conn(0x7fd558147360 msgr2=0x7fd55806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2982002730 >> v2:172.21.15.86:3300/0 conn(0x7fd558147360 0x7fd55806e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd54ea00640 1 -- 172.21.15.67:0/2982002730 >> v1:172.21.15.86:6789/0 conn(0x7fd55815ede0 legacy=0x7fd55806d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.148+0000 7fd54ea00640 1 -- 172.21.15.67:0/2982002730 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558161f90 con 0x7fd558147740 2024-04-17T10:34:56.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2982002730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147740 0x7fd5580717f0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fd55806ba90 tx=0x7fd54001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f634ddac38be3be server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd556a00640 1 -- 172.21.15.67:0/2982002730 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400ab600 con 0x7fd558147740 2024-04-17T10:34:56.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd556a00640 1 -- 172.21.15.67:0/2982002730 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5400b0070 con 0x7fd558147740 2024-04-17T10:34:56.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd556a00640 1 -- 172.21.15.67:0/2982002730 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400ab9c0 con 0x7fd558147740 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- 172.21.15.67:0/2982002730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147740 msgr2=0x7fd5580717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 --2- 172.21.15.67:0/2982002730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147740 0x7fd5580717f0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fd55806ba90 tx=0x7fd54001c4f0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- 172.21.15.67:0/2982002730 shutdown_connections 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 --2- 172.21.15.67:0/2982002730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147740 0x7fd5580717f0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 --2- 172.21.15.67:0/2982002730 >> v2:172.21.15.86:3300/0 conn(0x7fd558147360 0x7fd55806e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- 172.21.15.67:0/2982002730 >> 172.21.15.67:0/2982002730 conn(0x7fd55806ac90 msgr2=0x7fd55806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- 172.21.15.67:0/2982002730 shutdown_connections 2024-04-17T10:34:56.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- 172.21.15.67:0/2982002730 wait complete. 2024-04-17T10:34:56.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 Processor -- start 2024-04-17T10:34:56.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- start start 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147360 0x7fd55813c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd558147740 0x7fd55813c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd55815ede0 0x7fd5580cefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558072140 con 0x7fd558147360 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55806eff0 con 0x7fd558147740 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.152+0000 7fd55e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd55816ac40 con 0x7fd55815ede0 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147360 0x7fd55813c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147360 0x7fd55813c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36042/0 (socket says 172.21.15.67:36042) 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 -- 172.21.15.67:0/2216343687 learned_addr learned my addr 172.21.15.67:0/2216343687 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557e00640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd55815ede0 0x7fd5580cefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd558147740 0x7fd55813c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 -- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd55815ede0 msgr2=0x7fd5580cefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd55815ede0 0x7fd5580cefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 -- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd558147740 msgr2=0x7fd55813c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd558147740 0x7fd55813c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 -- 172.21.15.67:0/2216343687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55813cd20 con 0x7fd558147360 2024-04-17T10:34:56.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd557400640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147360 0x7fd55813c140 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fd54800e930 tx=0x7fd54800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd54fe00640 1 -- 172.21.15.67:0/2216343687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548017070 con 0x7fd558147360 2024-04-17T10:34:56.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd54fe00640 1 -- 172.21.15.67:0/2216343687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd54800ca00 con 0x7fd558147360 2024-04-17T10:34:56.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd54fe00640 1 -- 172.21.15.67:0/2216343687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54800cdb0 con 0x7fd558147360 2024-04-17T10:34:56.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5581f76a0 con 0x7fd558147360 2024-04-17T10:34:56.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5581f7c60 con 0x7fd558147360 2024-04-17T10:34:56.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.156+0000 7fd54fe00640 1 -- 172.21.15.67:0/2216343687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd54801a070 con 0x7fd558147360 2024-04-17T10:34:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd54fe00640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd534066f90 0x7fd534069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd54fe00640 1 -- 172.21.15.67:0/2216343687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd548085140 con 0x7fd558147360 2024-04-17T10:34:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd534066f90 0x7fd534069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 Processor -- start 2024-04-17T10:34:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- start start 2024-04-17T10:34:56.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd534066f90 0x7fd534069450 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd5400af000 tx=0x7fd5400a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f7eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f7a4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7a5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f5f1ff0 con 0x55ef2f7a50a0 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f57a150 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f5ee090 con 0x55ef2f7ef2f0 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f7eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f7eedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39542/0 (socket says 172.21.15.67:39542) 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 -- 172.21.15.67:0/2304933911 learned_addr learned my addr 172.21.15.67:0/2304933911 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f7a4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557e00640 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 -- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 msgr2=0x55ef2f7a4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f7a4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 -- 172.21.15.67:0/2304933911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 msgr2=0x55ef2f7a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 -- 172.21.15.67:0/2304933911 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef2f7880d0 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f7a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd557400640 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f7eedb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd548051e80 tx=0x7fd548055a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb9f4a911f3784bd server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd547e00640 1 -- 172.21.15.67:0/2304933911 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548021040 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd547e00640 1 -- 172.21.15.67:0/2304933911 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548081070 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd547e00640 1 -- 172.21.15.67:0/2304933911 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd54805e080 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 msgr2=0x55ef2f7eedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f7eedb0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd548051e80 tx=0x7fd548055a20 comp rx=0 tx=0).stop 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- 172.21.15.67:0/2304933911 shutdown_connections 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f7a4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2304933911 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f7eedb0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- 172.21.15.67:0/2304933911 >> 172.21.15.67:0/2304933911 conn(0x55ef2f7a1020 msgr2=0x55ef2f7a3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- 172.21.15.67:0/2304933911 shutdown_connections 2024-04-17T10:34:56.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.160+0000 7fd5617a1000 1 -- 172.21.15.67:0/2304933911 wait complete. 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 -- start start 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 Processor -- start 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7d37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ee9d0 0x55ef2f7d3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f7d4270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f5b6250 con 0x55ef2f7a50a0 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f5ac690 con 0x55ef2f7ef2f0 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f57aa40 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 10 client.0 _ll_register_callbacks cb 0x55ef2f78fe90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd557400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7d37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd557400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7d37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36058/0 (socket says 172.21.15.67:36058) 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd557400640 1 -- 172.21.15.67:0/2464784023 learned_addr learned my addr 172.21.15.67:0/2464784023 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ee9d0 0x55ef2f7d3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd557e00640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f7d4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd54ea00640 1 -- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 msgr2=0x55ef2f7d4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f7d4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd54ea00640 1 -- 172.21.15.67:0/2464784023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 msgr2=0x55ef2f7d37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7d37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd54ea00640 1 -- 172.21.15.67:0/2464784023 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef2f69f1c0 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd557400640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7d37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd54ea00640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ee9d0 0x55ef2f7d3d30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5400ac000 tx=0x7fd54001ad30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f117216f06cf4085 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd544c00640 10 client.0 ms_handle_connect on v2:172.21.15.86:3301/0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd544c00640 1 -- 172.21.15.67:0/2464784023 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400ce060 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd544c00640 1 -- 172.21.15.67:0/2464784023 <== mon.2 v2:172.21.15.86:3301/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5400c1970 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd548081070 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd544c00640 1 -- 172.21.15.67:0/2464784023 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5400d4050 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 10 client.7787 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55ef2f69b170 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd544c00640 1 -- 172.21.15.67:0/2464784023 <== mon.2 v2:172.21.15.86:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548081070 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd544c00640 1 -- 172.21.15.67:0/2464784023 <== mon.2 v2:172.21.15.86:3301/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd5400d7020 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 10 client.7787 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 10 client.7787 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.d 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 20 client.7787 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f7fec80 0x55ef2f801070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 4 client.7787 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef2f58dc60 con 0x55ef2f7fec80 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.164+0000 7fd557400640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f7fec80 0x55ef2f801070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd557400640 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f7fec80 0x55ef2f801070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2314e5175545a4ee server_cookie=cdf86e0c2a23573b in_seq=0 out_seq=0 2024-04-17T10:34:56.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd544c00640 10 client.7787 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:56.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd544c00640 1 -- 172.21.15.67:0/2464784023 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ef2f58dc60 con 0x55ef2f7fec80 2024-04-17T10:34:56.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd544c00640 10 client.7787 handle_command_reply: tid=0 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 client.7787 shutdown 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ee9d0 msgr2=0x55ef2f7d3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ee9d0 0x55ef2f7d3d30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd5400ac000 tx=0x7fd54001ad30 comp rx=0 tx=0).stop 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 shutdown_connections 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f7fec80 0x55ef2f801070 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f7d4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ee9d0 0x55ef2f7d3d30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 --2- 172.21.15.67:0/2464784023 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f7d37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd544c00640 0 client.7787 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 >> 172.21.15.67:0/2464784023 conn(0x55ef2f7a1020 msgr2=0x55ef2f7a3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 shutdown_connections 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 1 -- 172.21.15.67:0/2464784023 wait complete. 2024-04-17T10:34:56.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.168+0000 7fd5617a1000 20 client.7787 trim_cache size 0 max 16384 2024-04-17T10:34:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 Processor -- start 2024-04-17T10:34:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- start start 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f84b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f84b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f84bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558142270 con 0x55ef2f7a50a0 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581414b0 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581401a0 con 0x55ef2f7ef2f0 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f84b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f84b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36062/0 (socket says 172.21.15.67:36062) 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 -- 172.21.15.67:0/609954005 learned_addr learned my addr 172.21.15.67:0/609954005 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557e00640 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f84bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd54ea00640 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f84b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 -- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 msgr2=0x55ef2f84bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f84bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 -- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 msgr2=0x55ef2f84b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f84b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 -- 172.21.15.67:0/609954005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef2f638de0 con 0x55ef2f7a50a0 2024-04-17T10:34:56.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd557400640 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f84b300 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fd548059c80 tx=0x7fd54801e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a7ddcd1106c2b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd547400640 1 -- 172.21.15.67:0/609954005 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548021040 con 0x55ef2f7a50a0 2024-04-17T10:34:56.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd547400640 1 -- 172.21.15.67:0/609954005 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548078070 con 0x55ef2f7a50a0 2024-04-17T10:34:56.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd547400640 1 -- 172.21.15.67:0/609954005 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5480550b0 con 0x55ef2f7a50a0 2024-04-17T10:34:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- 172.21.15.67:0/609954005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 msgr2=0x55ef2f84b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f84b300 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fd548059c80 tx=0x7fd54801e000 comp rx=0 tx=0).stop 2024-04-17T10:34:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- 172.21.15.67:0/609954005 shutdown_connections 2024-04-17T10:34:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7ef2f0 0x55ef2f84bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f84b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 --2- 172.21.15.67:0/609954005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7a50a0 0x55ef2f84b300 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- 172.21.15.67:0/609954005 >> 172.21.15.67:0/609954005 conn(0x55ef2f7a1020 msgr2=0x55ef2f7fbfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- 172.21.15.67:0/609954005 shutdown_connections 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- 172.21.15.67:0/609954005 wait complete. 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 -- start start 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.184+0000 7fd5617a1000 1 Processor -- start 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7a50a0 0x55ef2f801770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f801cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f8021f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f58ce00 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5581407a0 con 0x55ef2f7ef2f0 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef2f5f1300 con 0x55ef2f7a50a0 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 10 client.0 _ll_register_callbacks cb 0x55ef2f770040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f801cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f801cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36066/0 (socket says 172.21.15.67:36066) 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 -- 172.21.15.67:0/4211140852 learned_addr learned my addr 172.21.15.67:0/4211140852 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd557400640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7a50a0 0x55ef2f801770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd557e00640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f8021f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 -- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7a50a0 msgr2=0x55ef2f801770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7a50a0 0x55ef2f801770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 -- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 msgr2=0x55ef2f8021f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f8021f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 -- 172.21.15.67:0/4211140852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef2f5938e0 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd557e00640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f8021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd54ea00640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f801cb0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fd5400ad040 tx=0x7fd540008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4814507d3b108baa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 1 -- 172.21.15.67:0/4211140852 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5400bf890 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd548078070 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 1 -- 172.21.15.67:0/4211140852 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fd5400bfa30 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 1 -- 172.21.15.67:0/4211140852 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5400e9050 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 10 client.9492 fetch_fsmap learned FSMap version 90 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=90,monmap=2+}) v3 -- 0x55ef2f64b8f0 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 1 -- 172.21.15.67:0/4211140852 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548078070 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 1 -- 172.21.15.67:0/4211140852 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 90) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd5400ec020 con 0x55ef2f7ee9d0 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 10 client.9492 fetch_fsmap finished waiting for FSMap version 90 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 10 client.9492 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.d 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 20 client.9492 populate_metadata read hostname 'smithi067' 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f854f50 0x55ef2f857340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 4 client.9492 mds_command: new command op to 9270 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 --> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ef2f5ff530 con 0x55ef2f854f50 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd557400640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f854f50 0x55ef2f857340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd557400640 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f854f50 0x55ef2f857340 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457a83dd46e582bf server_cookie=ab519a42478919ff in_seq=0 out_seq=0 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 10 client.9492 ms_handle_connect on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 1 -- 172.21.15.67:0/4211140852 <== mds.0 v2:172.21.15.86:6834/538421223 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x55ef2f5ff530 con 0x55ef2f854f50 2024-04-17T10:34:56.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 10 client.9492 handle_command_reply: tid=0 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 client.9492 shutdown 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7ee9d0 msgr2=0x55ef2f801cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f801cb0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fd5400ad040 tx=0x7fd540008040 comp rx=0 tx=0).stop 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 shutdown_connections 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:6834/538421223,v1:172.21.15.86:6836/538421223] conn(0x55ef2f854f50 0x55ef2f857340 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55ef2f7ef2f0 0x55ef2f8021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ef2f7ee9d0 0x55ef2f801cb0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 --2- 172.21.15.67:0/4211140852 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55ef2f7a50a0 0x55ef2f801770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd544c00640 0 client.9492 ms_handle_reset on v2:172.21.15.86:6834/538421223 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.188+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 >> 172.21.15.67:0/4211140852 conn(0x55ef2f7a1020 msgr2=0x55ef2f7ed070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 shutdown_connections 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd5617a1000 1 -- 172.21.15.67:0/4211140852 wait complete. 2024-04-17T10:34:56.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd5617a1000 20 client.9492 trim_cache size 0 max 16384 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: "scrubs": { 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: "3598e48a-86ce-44d6-8826-5b2339eddced": { 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: "path": "/scrub_control_test_path", 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: "tag": "3598e48a-86ce-44d6-8826-5b2339eddced", 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: "options": "recursive" 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:34:56.187 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd534066f90 msgr2=0x7fd534069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd534066f90 0x7fd534069450 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd5400af000 tx=0x7fd5400a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147360 msgr2=0x7fd55813c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147360 0x7fd55813c140 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fd54800e930 tx=0x7fd54800edf0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 shutdown_connections 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd534066f90 0x7fd534069450 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd55815ede0 0x7fd5580cefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd558147740 0x7fd55813c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 --2- 172.21.15.67:0/2216343687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd558147360 0x7fd55813c140 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 >> 172.21.15.67:0/2216343687 conn(0x7fd55806ac90 msgr2=0x7fd558144960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 shutdown_connections 2024-04-17T10:34:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.192+0000 7fd55e200640 1 -- 172.21.15.67:0/2216343687 wait complete. 2024-04-17T10:34:56.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac3a400640 1 Processor -- start 2024-04-17T10:34:56.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac3a400640 1 -- start start 2024-04-17T10:34:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac3a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac341446c0 0x7fac34144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac3a400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fac34071840 0x7fac340b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac3a400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac341456d0 con 0x7fac34144aa0 2024-04-17T10:34:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac3a400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340b51a0 con 0x7fac341446c0 2024-04-17T10:34:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac3a400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac340b4870 con 0x7fac34071840 2024-04-17T10:34:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac33400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fac341446c0 0x7fac34144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac2be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fac34144aa0 0x7fac340d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47092/0 (socket says 172.21.15.67:47092) 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac2be00640 1 -- 172.21.15.67:0/2682508288 learned_addr learned my addr 172.21.15.67:0/2682508288 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac33e00640 1 --2- 172.21.15.67:0/2682508288 >> v2:172.21.15.86:3300/0 conn(0x7fac34071840 0x7fac340b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac33400640 1 -- 172.21.15.67:0/2682508288 >> v2:172.21.15.86:3300/0 conn(0x7fac34071840 msgr2=0x7fac340b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac33400640 1 --2- 172.21.15.67:0/2682508288 >> v2:172.21.15.86:3300/0 conn(0x7fac34071840 0x7fac340b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac33400640 1 -- 172.21.15.67:0/2682508288 >> v1:172.21.15.67:6789/0 conn(0x7fac34144aa0 legacy=0x7fac340d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac33400640 1 -- 172.21.15.67:0/2682508288 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac340b4b50 con 0x7fac341446c0 2024-04-17T10:34:56.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.352+0000 7fac32a00640 1 -- 172.21.15.67:0/2682508288 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335956779 0 0) 0x7fac341456d0 con 0x7fac34144aa0 2024-04-17T10:34:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac33400640 1 --2- 172.21.15.67:0/2682508288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac341446c0 0x7fac34144ea0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fac340b5320 tx=0x7fac2401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac7455ad61a9b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac32a00640 1 -- 172.21.15.67:0/2682508288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac240b0070 con 0x7fac341446c0 2024-04-17T10:34:56.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac32a00640 1 -- 172.21.15.67:0/2682508288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac240ab600 con 0x7fac341446c0 2024-04-17T10:34:56.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac32a00640 1 -- 172.21.15.67:0/2682508288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac240ab9d0 con 0x7fac341446c0 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 -- 172.21.15.67:0/2682508288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac341446c0 msgr2=0x7fac34144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 --2- 172.21.15.67:0/2682508288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac341446c0 0x7fac34144ea0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fac340b5320 tx=0x7fac2401c430 comp rx=0 tx=0).stop 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 -- 172.21.15.67:0/2682508288 shutdown_connections 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 --2- 172.21.15.67:0/2682508288 >> v2:172.21.15.86:3300/0 conn(0x7fac34071840 0x7fac340b5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 --2- 172.21.15.67:0/2682508288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac341446c0 0x7fac34144ea0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 -- 172.21.15.67:0/2682508288 >> 172.21.15.67:0/2682508288 conn(0x7fac340b34d0 msgr2=0x7fac340b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 -- 172.21.15.67:0/2682508288 shutdown_connections 2024-04-17T10:34:56.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.356+0000 7fac3a400640 1 -- 172.21.15.67:0/2682508288 wait complete. 2024-04-17T10:34:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 Processor -- start 2024-04-17T10:34:56.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 -- start start 2024-04-17T10:34:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac34071840 0x7fac341f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac341446c0 0x7fac34159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac34144aa0 0x7fac3415da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340b6500 con 0x7fac34144aa0 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac340b6380 con 0x7fac341446c0 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac340b6680 con 0x7fac34071840 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac34144aa0 0x7fac3415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac34144aa0 0x7fac3415da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36080/0 (socket says 172.21.15.67:36080) 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 -- 172.21.15.67:0/4103856619 learned_addr learned my addr 172.21.15.67:0/4103856619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac2be00640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac341446c0 0x7fac34159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac34071840 0x7fac341f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 -- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac34071840 msgr2=0x7fac341f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac34071840 0x7fac341f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 -- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac341446c0 msgr2=0x7fac34159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac341446c0 0x7fac34159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 -- 172.21.15.67:0/4103856619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac341664d0 con 0x7fac34144aa0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac33e00640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac34144aa0 0x7fac3415da60 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00efc0 tx=0x7fac2c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac2b400640 1 -- 172.21.15.67:0/4103856619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c018020 con 0x7fac34144aa0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac2b400640 1 -- 172.21.15.67:0/4103856619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac2c014480 con 0x7fac34144aa0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac2b400640 1 -- 172.21.15.67:0/4103856619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2c014870 con 0x7fac34144aa0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3413fd90 con 0x7fac34144aa0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.360+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac34140030 con 0x7fac34144aa0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.364+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac34070bb0 con 0x7fac34144aa0 2024-04-17T10:34:56.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.364+0000 7fac2b400640 1 -- 172.21.15.67:0/4103856619 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fac2c01b020 con 0x7fac34144aa0 2024-04-17T10:34:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.364+0000 7fac2b400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.364+0000 7fac2b400640 1 -- 172.21.15.67:0/4103856619 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fac2c085b30 con 0x7fac34144aa0 2024-04-17T10:34:56.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.364+0000 7fac33400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.368+0000 7fac33400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fac3415f620 tx=0x7fac24008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.368+0000 7fac2b400640 1 -- 172.21.15.67:0/4103856619 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fac2c022070 con 0x7fac34144aa0 2024-04-17T10:34:56.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.564+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac340b41a0 con 0x7fac34144aa0 2024-04-17T10:34:56.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.564+0000 7fac2b400640 1 -- 172.21.15.67:0/4103856619 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7fac2c010040 con 0x7fac34144aa0 2024-04-17T10:34:56.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:56.558 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:34:56.558 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 msgr2=0x7fac10069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fac3415f620 tx=0x7fac24008040 comp rx=0 tx=0).stop 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac34144aa0 msgr2=0x7fac3415da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac34144aa0 0x7fac3415da60 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fac2c00efc0 tx=0x7fac2c00c490 comp rx=0 tx=0).stop 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 shutdown_connections 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fac10066f60 0x7fac10069420 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fac34144aa0 0x7fac3415da60 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fac341446c0 0x7fac34159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 --2- 172.21.15.67:0/4103856619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fac34071840 0x7fac341f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 >> 172.21.15.67:0/4103856619 conn(0x7fac340b34d0 msgr2=0x7fac3415cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 shutdown_connections 2024-04-17T10:34:56.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.568+0000 7fac3a400640 1 -- 172.21.15.67:0/4103856619 wait complete. 2024-04-17T10:34:56.579 DEBUG:tasks.cephfs.test_scrub_checks:current active=['d'] 2024-04-17T10:34:56.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:56.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.724+0000 7f336ea00640 1 Processor -- start 2024-04-17T10:34:56.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.724+0000 7f336ea00640 1 -- start start 2024-04-17T10:34:56.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f336ea00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33680cf3c0 0x7f33680b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f336ea00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33680ce090 con 0x7f33680cdaf0 2024-04-17T10:34:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f336ea00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33680ce3d0 con 0x7f336815dfe0 2024-04-17T10:34:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f336ea00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33680ce950 con 0x7f33680cf3c0 2024-04-17T10:34:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f336ca00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f33680cdaf0 0x7f33680cb8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47108/0 (socket says 172.21.15.67:47108) 2024-04-17T10:34:56.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f336ca00640 1 -- 172.21.15.67:0/722198706 learned_addr learned my addr 172.21.15.67:0/722198706 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f3367400640 1 --2- 172.21.15.67:0/722198706 >> v2:172.21.15.86:3300/0 conn(0x7f33680cf3c0 0x7f33680b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f3366a00640 1 -- 172.21.15.67:0/722198706 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923095323 0 0) 0x7f33680ce090 con 0x7f33680cdaf0 2024-04-17T10:34:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f3366a00640 1 -- 172.21.15.67:0/722198706 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f334c003530 con 0x7f33680cdaf0 2024-04-17T10:34:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f3367400640 1 -- 172.21.15.67:0/722198706 >> v1:172.21.15.86:6789/0 conn(0x7f336815dfe0 legacy=0x7f33680cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f3367400640 1 -- 172.21.15.67:0/722198706 >> v1:172.21.15.67:6789/0 conn(0x7f33680cdaf0 legacy=0x7f33680cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.728+0000 7f3367400640 1 -- 172.21.15.67:0/722198706 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680afc90 con 0x7f33680cf3c0 2024-04-17T10:34:56.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f3367400640 1 --2- 172.21.15.67:0/722198706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 0x7f33680b2dc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f33540097c0 tx=0x7f335401c350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88988fff8dcbb603 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f3366a00640 1 -- 172.21.15.67:0/722198706 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540b0070 con 0x7f33680cf3c0 2024-04-17T10:34:56.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f3366a00640 1 -- 172.21.15.67:0/722198706 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33540ab600 con 0x7f33680cf3c0 2024-04-17T10:34:56.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f3366a00640 1 -- 172.21.15.67:0/722198706 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33540ab9b0 con 0x7f33680cf3c0 2024-04-17T10:34:56.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 -- 172.21.15.67:0/722198706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 msgr2=0x7f33680b2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 --2- 172.21.15.67:0/722198706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 0x7f33680b2dc0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f33540097c0 tx=0x7f335401c350 comp rx=0 tx=0).stop 2024-04-17T10:34:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 -- 172.21.15.67:0/722198706 shutdown_connections 2024-04-17T10:34:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 --2- 172.21.15.67:0/722198706 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 0x7f33680b2dc0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 -- 172.21.15.67:0/722198706 >> 172.21.15.67:0/722198706 conn(0x7f33680b22a0 msgr2=0x7f33680b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 -- 172.21.15.67:0/722198706 shutdown_connections 2024-04-17T10:34:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 -- 172.21.15.67:0/722198706 wait complete. 2024-04-17T10:34:56.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 Processor -- start 2024-04-17T10:34:56.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.732+0000 7f336ea00640 1 -- start start 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33680cdaf0 0x7f3368152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 0x7f3368157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33680c6a80 0x7f33680c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3368169120 con 0x7f33680cdaf0 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3368168fa0 con 0x7f33680cf3c0 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33681692a0 con 0x7f33680c6a80 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33680cdaf0 0x7f3368152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33680cdaf0 0x7f3368152d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36096/0 (socket says 172.21.15.67:36096) 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33680c6a80 0x7f33680c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 -- 172.21.15.67:0/3853503098 learned_addr learned my addr 172.21.15.67:0/3853503098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367400640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 0x7f3368157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 -- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33680c6a80 msgr2=0x7f33680c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33680c6a80 0x7f33680c3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 -- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 msgr2=0x7f3368157b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 0x7f3368157b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 -- 172.21.15.67:0/3853503098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680aff70 con 0x7f33680cdaf0 2024-04-17T10:34:56.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f3367e00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33680cdaf0 0x7f3368152d80 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f335800c3c0 tx=0x7f335800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f335fe00640 1 -- 172.21.15.67:0/3853503098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358017070 con 0x7f33680cdaf0 2024-04-17T10:34:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f335fe00640 1 -- 172.21.15.67:0/3853503098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33580052d0 con 0x7f33680cdaf0 2024-04-17T10:34:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f335fe00640 1 -- 172.21.15.67:0/3853503098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358003840 con 0x7f33680cdaf0 2024-04-17T10:34:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33680b0210 con 0x7f33680cdaf0 2024-04-17T10:34:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33680c7500 con 0x7f33680cdaf0 2024-04-17T10:34:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.736+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336809e900 con 0x7f33680cdaf0 2024-04-17T10:34:56.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.740+0000 7f335fe00640 1 -- 172.21.15.67:0/3853503098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f335801a070 con 0x7f33680cdaf0 2024-04-17T10:34:56.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.740+0000 7f335fe00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3348066f60 0x7f3348069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:56.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.740+0000 7f335fe00640 1 -- 172.21.15.67:0/3853503098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3358084450 con 0x7f33680cdaf0 2024-04-17T10:34:56.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.740+0000 7f3367400640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3348066f60 0x7f3348069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:56.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.740+0000 7f3367400640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3348066f60 0x7f3348069420 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f33540ad000 tx=0x7f33540a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:56.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.744+0000 7f335fe00640 1 -- 172.21.15.67:0/3853503098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3358005480 con 0x7f33680cdaf0 2024-04-17T10:34:56.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f33680ca8e0 con 0x7f33680cdaf0 2024-04-17T10:34:56.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f335fe00640 1 -- 172.21.15.67:0/3853503098 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1444 (secure 0 0 0) 0x7f3358054c70 con 0x7f33680cdaf0 2024-04-17T10:34:56.952 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:56.952 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":627,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":130,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":869,"data_bytes":888087621,"bytes_used":2193080320,"bytes_avail":770901032960,"bytes_total":773094113280,"write_bytes_sec":47811983,"read_op_per_sec":0,"write_op_per_sec":47},"fsmap":{"epoch":90,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"d","status":"up:active","gid":9270}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":24,"modified":"2024-04-17T10:34:16.270424+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3348066f60 msgr2=0x7f3348069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3348066f60 0x7f3348069420 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f33540ad000 tx=0x7f33540a6000 comp rx=0 tx=0).stop 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33680cdaf0 msgr2=0x7f3368152d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33680cdaf0 0x7f3368152d80 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f335800c3c0 tx=0x7f335800c880 comp rx=0 tx=0).stop 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 shutdown_connections 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3348066f60 0x7f3348069420 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33680c6a80 0x7f33680c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33680cf3c0 0x7f3368157b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 --2- 172.21.15.67:0/3853503098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33680cdaf0 0x7f3368152d80 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 >> 172.21.15.67:0/3853503098 conn(0x7f33680b22a0 msgr2=0x7f33680b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 shutdown_connections 2024-04-17T10:34:56.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:56.960+0000 7f336ea00640 1 -- 172.21.15.67:0/3853503098 wait complete. 2024-04-17T10:34:56.972 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:34:57.973 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:58.095 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:34:58.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:34:58.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8398c00640 1 Processor -- start 2024-04-17T10:34:58.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8398c00640 1 -- start start 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8398c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f839415d950 0x7f839406ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8398c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839415dd30 con 0x7f8394070990 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8398c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8394161640 con 0x7f83940ce520 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8398c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8394161920 con 0x7f839415d950 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8393400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f839415d950 0x7f839406ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8393e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8394070990 0x7f83940b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47124/0 (socket says 172.21.15.67:47124) 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8393e00640 1 -- 172.21.15.67:0/846300371 learned_addr learned my addr 172.21.15.67:0/846300371 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:34:58.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8393400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f839415d950 0x7f839406ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36108/0 (socket says 172.21.15.67:36108) 2024-04-17T10:34:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8393400640 1 -- 172.21.15.67:0/846300371 >> v1:172.21.15.86:6789/0 conn(0x7f83940ce520 legacy=0x7f8394070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8393400640 1 -- 172.21.15.67:0/846300371 >> v1:172.21.15.67:6789/0 conn(0x7f8394070990 legacy=0x7f83940b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.136+0000 7f8393400640 1 -- 172.21.15.67:0/846300371 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8394161c00 con 0x7f839415d950 2024-04-17T10:34:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8393400640 1 --2- 172.21.15.67:0/846300371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f839406ac90 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f83840097c0 tx=0x7f838401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcb29c4b31d665a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8392000640 1 -- 172.21.15.67:0/846300371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840b2070 con 0x7f839415d950 2024-04-17T10:34:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8392000640 1 -- 172.21.15.67:0/846300371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83840ad680 con 0x7f839415d950 2024-04-17T10:34:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8392000640 1 -- 172.21.15.67:0/846300371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840ada50 con 0x7f839415d950 2024-04-17T10:34:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8398c00640 1 -- 172.21.15.67:0/846300371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 msgr2=0x7f839406ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8398c00640 1 --2- 172.21.15.67:0/846300371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f839406ac90 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f83840097c0 tx=0x7f838401c470 comp rx=0 tx=0).stop 2024-04-17T10:34:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8398c00640 1 -- 172.21.15.67:0/846300371 shutdown_connections 2024-04-17T10:34:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8398c00640 1 --2- 172.21.15.67:0/846300371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f839406ac90 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8398c00640 1 -- 172.21.15.67:0/846300371 >> 172.21.15.67:0/846300371 conn(0x7f83940b23f0 msgr2=0x7f83940b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:58.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8398c00640 1 -- 172.21.15.67:0/846300371 shutdown_connections 2024-04-17T10:34:58.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.140+0000 7f8398c00640 1 -- 172.21.15.67:0/846300371 wait complete. 2024-04-17T10:34:58.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 Processor -- start 2024-04-17T10:34:58.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 -- start start 2024-04-17T10:34:58.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8394070990 0x7f839415ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83940ce520 0x7f8394152f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f8394153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839416ba60 con 0x7f839415d950 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839406b5e0 con 0x7f8394070990 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8398c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839416bbe0 con 0x7f83940ce520 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f8394153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8394070990 0x7f839415ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f8394153460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36116/0 (socket says 172.21.15.67:36116) 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 -- 172.21.15.67:0/1461066203 learned_addr learned my addr 172.21.15.67:0/1461066203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:58.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8392a00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83940ce520 0x7f8394152f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 -- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83940ce520 msgr2=0x7f8394152f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83940ce520 0x7f8394152f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 -- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8394070990 msgr2=0x7f839415ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8394070990 0x7f839415ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 -- 172.21.15.67:0/1461066203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839415f950 con 0x7f839415d950 2024-04-17T10:34:58.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.144+0000 7f8393e00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f8394153460 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f838c00cbf0 tx=0x7f838c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f838b400640 1 -- 172.21.15.67:0/1461066203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c017070 con 0x7f839415d950 2024-04-17T10:34:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f838b400640 1 -- 172.21.15.67:0/1461066203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838c00ad40 con 0x7f839415d950 2024-04-17T10:34:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f838b400640 1 -- 172.21.15.67:0/1461066203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838c012600 con 0x7f839415d950 2024-04-17T10:34:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839415fc30 con 0x7f839415d950 2024-04-17T10:34:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83941601f0 con 0x7f839415d950 2024-04-17T10:34:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8394074dc0 con 0x7f839415d950 2024-04-17T10:34:58.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f838b400640 1 -- 172.21.15.67:0/1461066203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f838c01a070 con 0x7f839415d950 2024-04-17T10:34:58.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f838b400640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8374066f60 0x7f8374069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f838b400640 1 -- 172.21.15.67:0/1461066203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f838c084ea0 con 0x7f839415d950 2024-04-17T10:34:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.148+0000 7f8393400640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8374066f60 0x7f8374069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.152+0000 7f8393400640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8374066f60 0x7f8374069420 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f83840af000 tx=0x7f83840a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:58.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.152+0000 7f838b400640 1 -- 172.21.15.67:0/1461066203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f838c055630 con 0x7f839415d950 2024-04-17T10:34:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f839406fdc0 con 0x7f839415d950 2024-04-17T10:34:58.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f838b400640 1 -- 172.21.15.67:0/1461066203 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f838c0596d0 con 0x7f839415d950 2024-04-17T10:34:58.341 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:58.341 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:34:58.341 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8374066f60 msgr2=0x7f8374069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8374066f60 0x7f8374069420 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f83840af000 tx=0x7f83840a7000 comp rx=0 tx=0).stop 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 msgr2=0x7f8394153460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f8394153460 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f838c00cbf0 tx=0x7f838c00a430 comp rx=0 tx=0).stop 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 shutdown_connections 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8374066f60 0x7f8374069420 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f839415d950 0x7f8394153460 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f83940ce520 0x7f8394152f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 --2- 172.21.15.67:0/1461066203 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8394070990 0x7f839415ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 >> 172.21.15.67:0/1461066203 conn(0x7f83940b23f0 msgr2=0x7f83940b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.348+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 shutdown_connections 2024-04-17T10:34:58.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.352+0000 7f8398c00640 1 -- 172.21.15.67:0/1461066203 wait complete. 2024-04-17T10:34:58.363 DEBUG:tasks.cephfs.test_scrub_checks:current active=['d'] 2024-04-17T10:34:58.363 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:34:58.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62c400640 1 Processor -- start 2024-04-17T10:34:58.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62c400640 1 -- start start 2024-04-17T10:34:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa62415dd10 0x7fa6240b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62c400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6240714d0 0x7fa62406c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62c400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6240b0cc0 con 0x7fa6240b57b0 2024-04-17T10:34:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62c400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6240b1280 con 0x7fa6240714d0 2024-04-17T10:34:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6240b0fa0 con 0x7fa62415dd10 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa629a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa62415dd10 0x7fa6240b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62a400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6240714d0 0x7fa62406c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62a400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa6240714d0 0x7fa62406c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:39634/0 (socket says 172.21.15.67:39634) 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa629a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa62415dd10 0x7fa6240b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:37614/0 (socket says 172.21.15.67:37614) 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62a400640 1 -- 172.21.15.67:0/2375663002 learned_addr learned my addr 172.21.15.67:0/2375663002 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa629a00640 1 -- 172.21.15.67:0/2375663002 >> v2:172.21.15.86:3300/0 conn(0x7fa6240714d0 msgr2=0x7fa62406c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa629a00640 1 --2- 172.21.15.67:0/2375663002 >> v2:172.21.15.86:3300/0 conn(0x7fa6240714d0 0x7fa62406c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa629a00640 1 -- 172.21.15.67:0/2375663002 >> v1:172.21.15.86:6789/0 conn(0x7fa6240b57b0 legacy=0x7fa624070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa629a00640 1 -- 172.21.15.67:0/2375663002 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62406bed0 con 0x7fa62415dd10 2024-04-17T10:34:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa623e00640 1 -- 172.21.15.67:0/2375663002 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681355847 0 0) 0x7fa6240b0cc0 con 0x7fa6240b57b0 2024-04-17T10:34:58.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.512+0000 7fa62a400640 1 --2- 172.21.15.67:0/2375663002 >> v2:172.21.15.86:3300/0 conn(0x7fa6240714d0 0x7fa62406c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:34:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa629a00640 1 --2- 172.21.15.67:0/2375663002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa62415dd10 0x7fa6240b5270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6180097c0 tx=0x7fa61801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26a7e94618e8eb2f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa623e00640 1 -- 172.21.15.67:0/2375663002 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180b1070 con 0x7fa62415dd10 2024-04-17T10:34:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa623e00640 1 -- 172.21.15.67:0/2375663002 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6180ac620 con 0x7fa62415dd10 2024-04-17T10:34:58.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa623e00640 1 -- 172.21.15.67:0/2375663002 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180ac9f0 con 0x7fa62415dd10 2024-04-17T10:34:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 -- 172.21.15.67:0/2375663002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa62415dd10 msgr2=0x7fa6240b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 --2- 172.21.15.67:0/2375663002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa62415dd10 0x7fa6240b5270 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa6180097c0 tx=0x7fa61801c3e0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 -- 172.21.15.67:0/2375663002 shutdown_connections 2024-04-17T10:34:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 --2- 172.21.15.67:0/2375663002 >> v2:172.21.15.86:3300/0 conn(0x7fa6240714d0 0x7fa62406c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 --2- 172.21.15.67:0/2375663002 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa62415dd10 0x7fa6240b5270 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 -- 172.21.15.67:0/2375663002 >> 172.21.15.67:0/2375663002 conn(0x7fa6240b34d0 msgr2=0x7fa6240b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 -- 172.21.15.67:0/2375663002 shutdown_connections 2024-04-17T10:34:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.516+0000 7fa62c400640 1 -- 172.21.15.67:0/2375663002 wait complete. 2024-04-17T10:34:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 Processor -- start 2024-04-17T10:34:58.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 -- start start 2024-04-17T10:34:58.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6240714d0 0x7fa624149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa62415dd10 0x7fa624160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6241608b0 0x7fa6240b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62406d7b0 con 0x7fa6240714d0 2024-04-17T10:34:58.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62406d630 con 0x7fa62415dd10 2024-04-17T10:34:58.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62406d930 con 0x7fa6241608b0 2024-04-17T10:34:58.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6240714d0 0x7fa624149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6240714d0 0x7fa624149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36120/0 (socket says 172.21.15.67:36120) 2024-04-17T10:34:58.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 -- 172.21.15.67:0/3311031573 learned_addr learned my addr 172.21.15.67:0/3311031573 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:58.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62a400640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6241608b0 0x7fa6240b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629000640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa62415dd10 0x7fa624160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 -- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6241608b0 msgr2=0x7fa6240b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6241608b0 0x7fa6240b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 -- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa62415dd10 msgr2=0x7fa624160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa62415dd10 0x7fa624160370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 -- 172.21.15.67:0/3311031573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6240cada0 con 0x7fa6240714d0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629000640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa62415dd10 0x7fa624160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa629a00640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6240714d0 0x7fa624149280 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa6180097c0 tx=0x7fa618019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa621600640 1 -- 172.21.15.67:0/3311031573 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180b9020 con 0x7fa6240714d0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa621600640 1 -- 172.21.15.67:0/3311031573 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6180b5410 con 0x7fa6240714d0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6240cb080 con 0x7fa6240714d0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa621600640 1 -- 172.21.15.67:0/3311031573 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180b1040 con 0x7fa6240714d0 2024-04-17T10:34:58.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.520+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6240cb640 con 0x7fa6240714d0 2024-04-17T10:34:58.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.524+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa624070bb0 con 0x7fa6240714d0 2024-04-17T10:34:58.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.524+0000 7fa621600640 1 -- 172.21.15.67:0/3311031573 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa6180bc020 con 0x7fa6240714d0 2024-04-17T10:34:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.524+0000 7fa621600640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa604066f60 0x7fa604069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.524+0000 7fa621600640 1 -- 172.21.15.67:0/3311031573 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fa61811ed90 con 0x7fa6240714d0 2024-04-17T10:34:58.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.524+0000 7fa629000640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa604066f60 0x7fa604069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:58.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.524+0000 7fa629000640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa604066f60 0x7fa604069420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa610000f70 tx=0x7fa610009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.528+0000 7fa621600640 1 -- 172.21.15.67:0/3311031573 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa6180c3030 con 0x7fa6240714d0 2024-04-17T10:34:58.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa624074e30 con 0x7fa6240714d0 2024-04-17T10:34:58.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa621600640 1 -- 172.21.15.67:0/3311031573 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1446 (secure 0 0 0) 0x7fa618016090 con 0x7fa6240714d0 2024-04-17T10:34:58.754 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:34:58.754 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":629,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":130,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076430860,"bytes_used":2607521792,"bytes_avail":770486591488,"bytes_total":773094113280,"write_bytes_sec":63505234,"read_op_per_sec":0,"write_op_per_sec":63},"fsmap":{"epoch":90,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"d","status":"up:active","gid":9270}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":24,"modified":"2024-04-17T10:34:16.270424+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa604066f60 msgr2=0x7fa604069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa604066f60 0x7fa604069420 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fa610000f70 tx=0x7fa610009040 comp rx=0 tx=0).stop 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6240714d0 msgr2=0x7fa624149280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6240714d0 0x7fa624149280 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7fa6180097c0 tx=0x7fa618019180 comp rx=0 tx=0).stop 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 shutdown_connections 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa604066f60 0x7fa604069420 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6241608b0 0x7fa6240b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa62415dd10 0x7fa624160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 --2- 172.21.15.67:0/3311031573 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6240714d0 0x7fa624149280 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:58.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 >> 172.21.15.67:0/3311031573 conn(0x7fa6240b34d0 msgr2=0x7fa624149ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:58.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 shutdown_connections 2024-04-17T10:34:58.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:58.752+0000 7fa62c400640 1 -- 172.21.15.67:0/3311031573 wait complete. 2024-04-17T10:34:58.764 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:34:59.764 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:34:59.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.924+0000 7f9a5e400640 1 Processor -- start 2024-04-17T10:34:59.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.924+0000 7f9a5e400640 1 -- start start 2024-04-17T10:34:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a58099a50 0x7f9a5815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5e400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9a580d03a0 0x7f9a58099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5e400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a580cde20 con 0x7f9a5809a740 2024-04-17T10:34:59.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5e400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a580cd1b0 con 0x7f9a58099a50 2024-04-17T10:34:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5e400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a5809a370 con 0x7f9a580d03a0 2024-04-17T10:34:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a58099a50 0x7f9a5815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9a58099a50 0x7f9a5815c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36128/0 (socket says 172.21.15.67:36128) 2024-04-17T10:34:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5d000640 1 -- 172.21.15.67:0/274693468 learned_addr learned my addr 172.21.15.67:0/274693468 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a4fe00640 1 --2- 172.21.15.67:0/274693468 >> v2:172.21.15.86:3300/0 conn(0x7f9a580d03a0 0x7f9a58099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5d000640 1 -- 172.21.15.67:0/274693468 >> v2:172.21.15.86:3300/0 conn(0x7f9a580d03a0 msgr2=0x7f9a58099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:34:59.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5d000640 1 --2- 172.21.15.67:0/274693468 >> v2:172.21.15.86:3300/0 conn(0x7f9a580d03a0 0x7f9a58099e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5d000640 1 -- 172.21.15.67:0/274693468 >> v1:172.21.15.86:6789/0 conn(0x7f9a5809a740 legacy=0x7f9a5815f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:59.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.928+0000 7f9a5d000640 1 -- 172.21.15.67:0/274693468 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a580cf300 con 0x7f9a58099a50 2024-04-17T10:34:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5d000640 1 --2- 172.21.15.67:0/274693468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58099a50 0x7f9a5815c7a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9a5809acd0 tx=0x7f9a4801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81e6904688b0690 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a57e00640 1 -- 172.21.15.67:0/274693468 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a480b0070 con 0x7f9a58099a50 2024-04-17T10:34:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a57e00640 1 -- 172.21.15.67:0/274693468 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a480ab600 con 0x7f9a58099a50 2024-04-17T10:34:59.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a57e00640 1 -- 172.21.15.67:0/274693468 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a480ab9d0 con 0x7f9a58099a50 2024-04-17T10:34:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 -- 172.21.15.67:0/274693468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58099a50 msgr2=0x7f9a5815c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 --2- 172.21.15.67:0/274693468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58099a50 0x7f9a5815c7a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f9a5809acd0 tx=0x7f9a4801c490 comp rx=0 tx=0).stop 2024-04-17T10:34:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 -- 172.21.15.67:0/274693468 shutdown_connections 2024-04-17T10:34:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 --2- 172.21.15.67:0/274693468 >> v2:172.21.15.86:3300/0 conn(0x7f9a580d03a0 0x7f9a58099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 --2- 172.21.15.67:0/274693468 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a58099a50 0x7f9a5815c7a0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:59.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 -- 172.21.15.67:0/274693468 >> 172.21.15.67:0/274693468 conn(0x7f9a580d1690 msgr2=0x7f9a580d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:34:59.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 -- 172.21.15.67:0/274693468 shutdown_connections 2024-04-17T10:34:59.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 -- 172.21.15.67:0/274693468 wait complete. 2024-04-17T10:34:59.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 Processor -- start 2024-04-17T10:34:59.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.932+0000 7f9a5e400640 1 -- start start 2024-04-17T10:34:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a58099a50 0x7f9a580b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a5809a740 0x7f9a580b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:59.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a580d03a0 0x7f9a580b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a581686b0 con 0x7f9a580d03a0 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a580cfe80 con 0x7f9a58099a50 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a58168830 con 0x7f9a5809a740 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a580d03a0 0x7f9a580b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a580d03a0 0x7f9a580b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36130/0 (socket says 172.21.15.67:36130) 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 -- 172.21.15.67:0/3944836910 learned_addr learned my addr 172.21.15.67:0/3944836910 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5d000640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a58099a50 0x7f9a580b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a4fe00640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a5809a740 0x7f9a580b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 -- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a5809a740 msgr2=0x7f9a580b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a5809a740 0x7f9a580b4690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 -- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a58099a50 msgr2=0x7f9a580b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a58099a50 0x7f9a580b6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:34:59.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 -- 172.21.15.67:0/3944836910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5815e200 con 0x7f9a580d03a0 2024-04-17T10:34:59.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5da00640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a580d03a0 0x7f9a580b4bd0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000c730 tx=0x7f9a5000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a55600640 1 -- 172.21.15.67:0/3944836910 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50003b80 con 0x7f9a580d03a0 2024-04-17T10:34:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a55600640 1 -- 172.21.15.67:0/3944836910 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a50003d20 con 0x7f9a580d03a0 2024-04-17T10:34:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5815e4e0 con 0x7f9a580d03a0 2024-04-17T10:34:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a55600640 1 -- 172.21.15.67:0/3944836910 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000aa30 con 0x7f9a580d03a0 2024-04-17T10:34:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a580c9e20 con 0x7f9a580d03a0 2024-04-17T10:34:59.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a5815ea00 con 0x7f9a580d03a0 2024-04-17T10:34:59.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.936+0000 7f9a55600640 1 -- 172.21.15.67:0/3944836910 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9a5000abd0 con 0x7f9a580d03a0 2024-04-17T10:34:59.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.940+0000 7f9a55600640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a34066f60 0x7f9a34069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:34:59.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.940+0000 7f9a55600640 1 -- 172.21.15.67:0/3944836910 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f9a50084fb0 con 0x7f9a580d03a0 2024-04-17T10:34:59.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.940+0000 7f9a5d000640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a34066f60 0x7f9a34069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:34:59.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.940+0000 7f9a5d000640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a34066f60 0x7f9a34069420 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9a480ad000 tx=0x7f9a480a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:34:59.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:34:59.944+0000 7f9a55600640 1 -- 172.21.15.67:0/3944836910 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9a50055740 con 0x7f9a580d03a0 2024-04-17T10:35:00.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.176+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a58074dc0 con 0x7f9a580d03a0 2024-04-17T10:35:00.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.176+0000 7f9a55600640 1 -- 172.21.15.67:0/3944836910 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f9a500597e0 con 0x7f9a580d03a0 2024-04-17T10:35:00.171 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:00.171 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:00.171 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:35:00.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a34066f60 msgr2=0x7f9a34069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a34066f60 0x7f9a34069420 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f9a480ad000 tx=0x7f9a480a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a580d03a0 msgr2=0x7f9a580b4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a580d03a0 0x7f9a580b4bd0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000c730 tx=0x7f9a5000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 shutdown_connections 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a34066f60 0x7f9a34069420 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a580d03a0 0x7f9a580b4bd0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a5809a740 0x7f9a580b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 --2- 172.21.15.67:0/3944836910 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a58099a50 0x7f9a580b6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 >> 172.21.15.67:0/3944836910 conn(0x7f9a580d1690 msgr2=0x7f9a580ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 shutdown_connections 2024-04-17T10:35:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.180+0000 7f9a5e400640 1 -- 172.21.15.67:0/3944836910 wait complete. 2024-04-17T10:35:00.192 DEBUG:tasks.cephfs.test_scrub_checks:current active=['d'] 2024-04-17T10:35:00.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:35:00.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f264d000640 1 Processor -- start 2024-04-17T10:35:00.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f264d000640 1 -- start start 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f264d000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2648099a50 0x7f264815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f264d000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f264809a740 0x7f26480cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f264d000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26480cde20 con 0x7f26480d03a0 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f264d000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26480cf300 con 0x7f2648099a50 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f264d000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26480cf5e0 con 0x7f264809a740 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f2647400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2648099a50 0x7f264815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f263fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f26480d03a0 0x7f2648099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38482/0 (socket says 172.21.15.67:38482) 2024-04-17T10:35:00.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.340+0000 7f263fe00640 1 -- 172.21.15.67:0/3049366971 learned_addr learned my addr 172.21.15.67:0/3049366971 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:00.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2647e00640 1 --2- 172.21.15.67:0/3049366971 >> v2:172.21.15.86:3300/0 conn(0x7f264809a740 0x7f26480cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2647400640 1 -- 172.21.15.67:0/3049366971 >> v2:172.21.15.86:3300/0 conn(0x7f264809a740 msgr2=0x7f26480cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:00.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2647400640 1 --2- 172.21.15.67:0/3049366971 >> v2:172.21.15.86:3300/0 conn(0x7f264809a740 0x7f26480cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2647400640 1 -- 172.21.15.67:0/3049366971 >> v1:172.21.15.67:6789/0 conn(0x7f26480d03a0 legacy=0x7f2648099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2647400640 1 -- 172.21.15.67:0/3049366971 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26480cf8c0 con 0x7f2648099a50 2024-04-17T10:35:00.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2647e00640 1 --2- 172.21.15.67:0/3049366971 >> v2:172.21.15.86:3300/0 conn(0x7f264809a740 0x7f26480cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:00.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2647400640 1 --2- 172.21.15.67:0/3049366971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f264815c7a0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2648072860 tx=0x7f263801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9d0545b3165aeae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:00.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2646a00640 1 -- 172.21.15.67:0/3049366971 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380b0070 con 0x7f2648099a50 2024-04-17T10:35:00.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2646a00640 1 -- 172.21.15.67:0/3049366971 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26380ab600 con 0x7f2648099a50 2024-04-17T10:35:00.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f2646a00640 1 -- 172.21.15.67:0/3049366971 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380ab9d0 con 0x7f2648099a50 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 -- 172.21.15.67:0/3049366971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 msgr2=0x7f264815c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 --2- 172.21.15.67:0/3049366971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f264815c7a0 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f2648072860 tx=0x7f263801c480 comp rx=0 tx=0).stop 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 -- 172.21.15.67:0/3049366971 shutdown_connections 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 --2- 172.21.15.67:0/3049366971 >> v2:172.21.15.86:3300/0 conn(0x7f264809a740 0x7f26480cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 --2- 172.21.15.67:0/3049366971 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f264815c7a0 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 -- 172.21.15.67:0/3049366971 >> 172.21.15.67:0/3049366971 conn(0x7f26480d1690 msgr2=0x7f26480d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 -- 172.21.15.67:0/3049366971 shutdown_connections 2024-04-17T10:35:00.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.344+0000 7f264d000640 1 -- 172.21.15.67:0/3049366971 wait complete. 2024-04-17T10:35:00.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 Processor -- start 2024-04-17T10:35:00.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 -- start start 2024-04-17T10:35:00.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f2648154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f264809a740 0x7f26481547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26480d03a0 0x7f264815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26481686b0 con 0x7f2648099a50 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26480cd140 con 0x7f264809a740 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2648168830 con 0x7f26480d03a0 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f2648154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f2648154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58706/0 (socket says 172.21.15.67:58706) 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 -- 172.21.15.67:0/74889277 learned_addr learned my addr 172.21.15.67:0/74889277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f263fe00640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f264809a740 0x7f26481547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647e00640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26480d03a0 0x7f264815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 -- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26480d03a0 msgr2=0x7f264815a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26480d03a0 0x7f264815a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 -- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f264809a740 msgr2=0x7f26481547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f264809a740 0x7f26481547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 -- 172.21.15.67:0/74889277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26481faa60 con 0x7f2648099a50 2024-04-17T10:35:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f2647400640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f2648154270 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f26380ae040 tx=0x7f263801acc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:00.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f263f400640 1 -- 172.21.15.67:0/74889277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380b0050 con 0x7f2648099a50 2024-04-17T10:35:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f263f400640 1 -- 172.21.15.67:0/74889277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26380bc070 con 0x7f2648099a50 2024-04-17T10:35:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f263f400640 1 -- 172.21.15.67:0/74889277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26380b5a50 con 0x7f2648099a50 2024-04-17T10:35:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.348+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264806bfb0 con 0x7f2648099a50 2024-04-17T10:35:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.352+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264806c570 con 0x7f2648099a50 2024-04-17T10:35:00.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.352+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2648074dc0 con 0x7f2648099a50 2024-04-17T10:35:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.352+0000 7f263f400640 1 -- 172.21.15.67:0/74889277 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f26380ab4c0 con 0x7f2648099a50 2024-04-17T10:35:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.352+0000 7f263f400640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2624066f60 0x7f2624069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.352+0000 7f263f400640 1 -- 172.21.15.67:0/74889277 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f263811fb60 con 0x7f2648099a50 2024-04-17T10:35:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.352+0000 7f263fe00640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2624066f60 0x7f2624069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.352+0000 7f263fe00640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2624066f60 0x7f2624069420 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f264815b9e0 tx=0x7f2630009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:00.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.356+0000 7f263f400640 1 -- 172.21.15.67:0/74889277 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2638016070 con 0x7f2648099a50 2024-04-17T10:35:00.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.580+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2648071300 con 0x7f2648099a50 2024-04-17T10:35:00.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.580+0000 7f263f400640 1 -- 172.21.15.67:0/74889277 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1631 (secure 0 0 0) 0x7f26380f0940 con 0x7f2648099a50 2024-04-17T10:35:00.572 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:00.573 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":631,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":130,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076430860,"bytes_used":2607521792,"bytes_avail":770486591488,"bytes_total":773094113280,"write_bytes_sec":39334577,"read_op_per_sec":0,"write_op_per_sec":39},"fsmap":{"epoch":90,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"d","status":"up:active","gid":9270}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-04-17T10:35:00.280094+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}}}}}},"progress_events":{}} 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.580+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2624066f60 msgr2=0x7f2624069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.580+0000 7f264d000640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2624066f60 0x7f2624069420 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f264815b9e0 tx=0x7f2630009040 comp rx=0 tx=0).stop 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.580+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 msgr2=0x7f2648154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.580+0000 7f264d000640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f2648154270 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f26380ae040 tx=0x7f263801acc0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 shutdown_connections 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2624066f60 0x7f2624069420 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26480d03a0 0x7f264815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f264809a740 0x7f26481547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 --2- 172.21.15.67:0/74889277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2648099a50 0x7f2648154270 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 >> 172.21.15.67:0/74889277 conn(0x7f26480d1690 msgr2=0x7f2648099e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 shutdown_connections 2024-04-17T10:35:00.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.584+0000 7f264d000640 1 -- 172.21.15.67:0/74889277 wait complete. 2024-04-17T10:35:00.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'d': 'paused paths [/scrub_control_test_path]'} 2024-04-17T10:35:00.594 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-17T10:35:00.594 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:35:00.609 INFO:tasks.ceph.mds.d:Stopped 2024-04-17T10:35:00.609 INFO:tasks.cephfs.test_scrub_checks:Waiting for promotion of one of the original standbys {'a', 'b', 'c'} 2024-04-17T10:35:00.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-04-17T10:35:00.762 INFO:teuthology.orchestra.run.smithi067.stdout:{"mds_beacon_grace":"15.000000"} 2024-04-17T10:35:00.762 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-04-17T10:35:00.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:00.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d03e00640 1 Processor -- start 2024-04-17T10:35:00.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d03e00640 1 -- start start 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d03e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cfc071790 0x7f6cfc15ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d03e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc071b70 0x7f6cfc06e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d03e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cfc0b3ab0 con 0x7f6cfc06ebb0 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d03e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cfc161700 con 0x7f6cfc071790 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d03e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc1619e0 con 0x7f6cfc071b70 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d01400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cfc071790 0x7f6cfc15ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d01400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6cfc071790 0x7f6cfc15ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58708/0 (socket says 172.21.15.67:58708) 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d01400640 1 -- 172.21.15.67:0/3318143331 learned_addr learned my addr 172.21.15.67:0/3318143331 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:00.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d00a00640 1 --2- 172.21.15.67:0/3318143331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc071b70 0x7f6cfc06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d01400640 1 -- 172.21.15.67:0/3318143331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc071b70 msgr2=0x7f6cfc06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:00.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.912+0000 7f6d01400640 1 --2- 172.21.15.67:0/3318143331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc071b70 0x7f6cfc06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d01400640 1 -- 172.21.15.67:0/3318143331 >> v1:172.21.15.86:6789/0 conn(0x7f6cfc06ebb0 legacy=0x7f6cfc0b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d01400640 1 -- 172.21.15.67:0/3318143331 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc161cc0 con 0x7f6cfc071790 2024-04-17T10:35:00.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d00a00640 1 --2- 172.21.15.67:0/3318143331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc071b70 0x7f6cfc06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:00.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d01400640 1 --2- 172.21.15.67:0/3318143331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071790 0x7f6cfc15ecf0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009870 tx=0x7f6cf001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385fb45bc4f5f3b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:00.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6cfbe00640 1 -- 172.21.15.67:0/3318143331 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf00aa600 con 0x7f6cfc071790 2024-04-17T10:35:00.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6cfbe00640 1 -- 172.21.15.67:0/3318143331 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf00af070 con 0x7f6cfc071790 2024-04-17T10:35:00.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6cfbe00640 1 -- 172.21.15.67:0/3318143331 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf00aa9c0 con 0x7f6cfc071790 2024-04-17T10:35:00.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 -- 172.21.15.67:0/3318143331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071790 msgr2=0x7f6cfc15ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3318143331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071790 0x7f6cfc15ecf0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009870 tx=0x7f6cf001c560 comp rx=0 tx=0).stop 2024-04-17T10:35:00.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 -- 172.21.15.67:0/3318143331 shutdown_connections 2024-04-17T10:35:00.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3318143331 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc071b70 0x7f6cfc06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3318143331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071790 0x7f6cfc15ecf0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 -- 172.21.15.67:0/3318143331 >> 172.21.15.67:0/3318143331 conn(0x7f6cfc0cfa30 msgr2=0x7f6cfc0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:00.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 -- 172.21.15.67:0/3318143331 shutdown_connections 2024-04-17T10:35:00.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.916+0000 7f6d03e00640 1 -- 172.21.15.67:0/3318143331 wait complete. 2024-04-17T10:35:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 Processor -- start 2024-04-17T10:35:00.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 -- start start 2024-04-17T10:35:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc06ebb0 0x7f6cfc1399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cfc071790 0x7f6cfc167fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071b70 0x7f6cfc151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc0720d0 con 0x7f6cfc071b70 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc071f50 con 0x7f6cfc071790 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cfc16aa00 con 0x7f6cfc06ebb0 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071b70 0x7f6cfc151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071b70 0x7f6cfc151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58720/0 (socket says 172.21.15.67:58720) 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 -- 172.21.15.67:0/3526795049 learned_addr learned my addr 172.21.15.67:0/3526795049 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d00a00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cfc071790 0x7f6cfc167fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01400640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc06ebb0 0x7f6cfc1399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 -- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc06ebb0 msgr2=0x7f6cfc1399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc06ebb0 0x7f6cfc1399a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 -- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cfc071790 msgr2=0x7f6cfc167fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cfc071790 0x7f6cfc167fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:00.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 -- 172.21.15.67:0/3526795049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc13d1c0 con 0x7f6cfc071b70 2024-04-17T10:35:00.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01400640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc06ebb0 0x7f6cfc1399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:00.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d00a00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cfc071790 0x7f6cfc167fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d01e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071b70 0x7f6cfc151e60 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c7a0 tx=0x7f6cf400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6cf9600640 1 -- 172.21.15.67:0/3526795049 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4003b80 con 0x7f6cfc071b70 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6cf9600640 1 -- 172.21.15.67:0/3526795049 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf4003d20 con 0x7f6cfc071b70 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6cf9600640 1 -- 172.21.15.67:0/3526795049 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf400aa90 con 0x7f6cfc071b70 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cfc167200 con 0x7f6cfc071b70 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.920+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cfc1677c0 con 0x7f6cfc071b70 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.924+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cfc074e30 con 0x7f6cfc071b70 2024-04-17T10:35:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.924+0000 7f6cf9600640 1 -- 172.21.15.67:0/3526795049 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6cf400ac30 con 0x7f6cfc071b70 2024-04-17T10:35:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.924+0000 7f6cf9600640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cdc066f60 0x7f6cdc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.924+0000 7f6cf9600640 1 -- 172.21.15.67:0/3526795049 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f6cf4085100 con 0x7f6cfc071b70 2024-04-17T10:35:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.924+0000 7f6d01400640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cdc066f60 0x7f6cdc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:00.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.924+0000 7f6d01400640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cdc066f60 0x7f6cdc069420 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00ae000 tx=0x7f6cf00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:00.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:00.928+0000 7f6cf9600640 1 -- 172.21.15.67:0/3526795049 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6cf4055f70 con 0x7f6cfc071b70 2024-04-17T10:35:01.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cfc0729a0 con 0x7f6cfc071b70 2024-04-17T10:35:01.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6cf9600640 1 -- 172.21.15.67:0/3526795049 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f6cf4055870 con 0x7f6cfc071b70 2024-04-17T10:35:01.124 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:01.125 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:01.125 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:35:01.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cdc066f60 msgr2=0x7f6cdc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:01.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cdc066f60 0x7f6cdc069420 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f6cf00ae000 tx=0x7f6cf00a7000 comp rx=0 tx=0).stop 2024-04-17T10:35:01.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071b70 msgr2=0x7f6cfc151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071b70 0x7f6cfc151e60 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6cf400c7a0 tx=0x7f6cf400cc60 comp rx=0 tx=0).stop 2024-04-17T10:35:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 shutdown_connections 2024-04-17T10:35:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cdc066f60 0x7f6cdc069420 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6cfc071b70 0x7f6cfc151e60 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6cfc071790 0x7f6cfc167fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 --2- 172.21.15.67:0/3526795049 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6cfc06ebb0 0x7f6cfc1399a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:01.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 >> 172.21.15.67:0/3526795049 conn(0x7f6cfc0cfa30 msgr2=0x7f6cfc0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:01.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.132+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 shutdown_connections 2024-04-17T10:35:01.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:01.136+0000 7f6d03e00640 1 -- 172.21.15.67:0/3526795049 wait complete. 2024-04-17T10:35:01.146 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-04-17T10:35:06.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:06.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a8f200640 1 Processor -- start 2024-04-17T10:35:06.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a8f200640 1 -- start start 2024-04-17T10:35:06.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:06.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a8f200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8806b100 con 0x7f3a880cc250 2024-04-17T10:35:06.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a8f200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8806b3e0 con 0x7f3a880cc840 2024-04-17T10:35:06.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a8f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a880cd150 con 0x7f3a880ce2a0 2024-04-17T10:35:06.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a87e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3a880cc250 0x7f3a880cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:38490/0 (socket says 172.21.15.67:38490) 2024-04-17T10:35:06.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a87e00640 1 -- 172.21.15.67:0/1355329407 learned_addr learned my addr 172.21.15.67:0/1355329407 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:06.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/1355329407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a87400640 1 -- 172.21.15.67:0/1355329407 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273375272 0 0) 0x7f3a8806b100 con 0x7f3a880cc250 2024-04-17T10:35:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a87400640 1 -- 172.21.15.67:0/1355329407 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6c003530 con 0x7f3a880cc250 2024-04-17T10:35:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a87400640 1 -- 172.21.15.67:0/1355329407 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562453703 0 0) 0x7f3a8806b3e0 con 0x7f3a880cc840 2024-04-17T10:35:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a87400640 1 -- 172.21.15.67:0/1355329407 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a8806b100 con 0x7f3a880cc840 2024-04-17T10:35:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a7fe00640 1 -- 172.21.15.67:0/1355329407 >> v1:172.21.15.86:6789/0 conn(0x7f3a880cc840 legacy=0x7f3a880cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a7fe00640 1 -- 172.21.15.67:0/1355329407 >> v1:172.21.15.67:6789/0 conn(0x7f3a880cc250 legacy=0x7f3a880cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:06.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.308+0000 7f3a7fe00640 1 -- 172.21.15.67:0/1355329407 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8806b6c0 con 0x7f3a880ce2a0 2024-04-17T10:35:06.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/1355329407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880cd5e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a700097f0 tx=0x7f3a7001c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e34cf39214f684b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:06.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a87400640 1 -- 172.21.15.67:0/1355329407 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a700ab5f0 con 0x7f3a880ce2a0 2024-04-17T10:35:06.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a87400640 1 -- 172.21.15.67:0/1355329407 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a700b0070 con 0x7f3a880ce2a0 2024-04-17T10:35:06.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a87400640 1 -- 172.21.15.67:0/1355329407 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a700ab9b0 con 0x7f3a880ce2a0 2024-04-17T10:35:06.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a8f200640 1 -- 172.21.15.67:0/1355329407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 msgr2=0x7f3a880cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:06.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a8f200640 1 --2- 172.21.15.67:0/1355329407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880cd5e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3a700097f0 tx=0x7f3a7001c4e0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a8f200640 1 -- 172.21.15.67:0/1355329407 shutdown_connections 2024-04-17T10:35:06.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a8f200640 1 --2- 172.21.15.67:0/1355329407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880cd5e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a8f200640 1 -- 172.21.15.67:0/1355329407 >> 172.21.15.67:0/1355329407 conn(0x7f3a880b22a0 msgr2=0x7f3a880b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:06.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a8f200640 1 -- 172.21.15.67:0/1355329407 shutdown_connections 2024-04-17T10:35:06.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.312+0000 7f3a8f200640 1 -- 172.21.15.67:0/1355329407 wait complete. 2024-04-17T10:35:06.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 Processor -- start 2024-04-17T10:35:06.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 -- start start 2024-04-17T10:35:06.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a880cc250 0x7f3a88160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:06.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a880cc840 0x7f3a8814f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:06.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:06.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8816c320 con 0x7f3a880cc250 2024-04-17T10:35:06.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a880cdf30 con 0x7f3a880cc840 2024-04-17T10:35:06.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8816c4a0 con 0x7f3a880ce2a0 2024-04-17T10:35:06.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a880cc250 0x7f3a88160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a880cc250 0x7f3a88160600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58724/0 (socket says 172.21.15.67:58724) 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 -- 172.21.15.67:0/3175294485 learned_addr learned my addr 172.21.15.67:0/3175294485 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a880cc840 0x7f3a8814f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8d200640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 -- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 msgr2=0x7f3a880b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 -- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a880cc840 msgr2=0x7f3a8814f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a880cc840 0x7f3a8814f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 -- 172.21.15.67:0/3175294485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a880b3750 con 0x7f3a880cc250 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a87e00640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a880cc250 0x7f3a88160600 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800c4d0 tx=0x7f3a7800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a84c00640 1 -- 172.21.15.67:0/3175294485 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a78017070 con 0x7f3a880cc250 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a84c00640 1 -- 172.21.15.67:0/3175294485 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a78005cf0 con 0x7f3a880cc250 2024-04-17T10:35:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a84c00640 1 -- 172.21.15.67:0/3175294485 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a780072b0 con 0x7f3a880cc250 2024-04-17T10:35:06.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a880b59e0 con 0x7f3a880cc250 2024-04-17T10:35:06.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a880b5fa0 con 0x7f3a880cc250 2024-04-17T10:35:06.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.316+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8806cd30 con 0x7f3a880cc250 2024-04-17T10:35:06.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.320+0000 7f3a84c00640 1 -- 172.21.15.67:0/3175294485 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3a7801a070 con 0x7f3a880cc250 2024-04-17T10:35:06.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.320+0000 7f3a84c00640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a64066f60 0x7f3a64069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:06.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.320+0000 7f3a84c00640 1 -- 172.21.15.67:0/3175294485 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3a78084670 con 0x7f3a880cc250 2024-04-17T10:35:06.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.320+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a64066f60 0x7f3a64069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:06.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.320+0000 7f3a7fe00640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a64066f60 0x7f3a64069420 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3a700af000 tx=0x7f3a700a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:06.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.324+0000 7f3a84c00640 1 -- 172.21.15.67:0/3175294485 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3a78055450 con 0x7f3a880cc250 2024-04-17T10:35:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a8809e900 con 0x7f3a880cc250 2024-04-17T10:35:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a84c00640 1 -- 172.21.15.67:0/3175294485 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f3a78054d50 con 0x7f3a880cc250 2024-04-17T10:35:06.516 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:06.516 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a64066f60 msgr2=0x7f3a64069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a64066f60 0x7f3a64069420 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f3a700af000 tx=0x7f3a700a7000 comp rx=0 tx=0).stop 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a880cc250 msgr2=0x7f3a88160600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a880cc250 0x7f3a88160600 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f3a7800c4d0 tx=0x7f3a7800c990 comp rx=0 tx=0).stop 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 shutdown_connections 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3a64066f60 0x7f3a64069420 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3a880ce2a0 0x7f3a880b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3a880cc840 0x7f3a8814f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 --2- 172.21.15.67:0/3175294485 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3a880cc250 0x7f3a88160600 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 >> 172.21.15.67:0/3175294485 conn(0x7f3a880b22a0 msgr2=0x7f3a88159b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 shutdown_connections 2024-04-17T10:35:06.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:06.524+0000 7f3a8f200640 1 -- 172.21.15.67:0/3175294485 wait complete. 2024-04-17T10:35:06.537 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-04-17T10:35:11.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b36000640 1 Processor -- start 2024-04-17T10:35:11.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b36000640 1 -- start start 2024-04-17T10:35:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b36000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b30144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b36000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b30144aa0 0x7f3b300d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b36000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b300b6640 con 0x7f3b30071840 2024-04-17T10:35:11.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b36000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b300b51a0 con 0x7f3b30144aa0 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b36000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b301456d0 con 0x7f3b301446c0 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b30144aa0 0x7f3b300d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b27e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b30144aa0 0x7f3b300d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51622/0 (socket says 172.21.15.67:51622) 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b27e00640 1 -- 172.21.15.67:0/551223006 learned_addr learned my addr 172.21.15.67:0/551223006 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b2f400640 1 --2- 172.21.15.67:0/551223006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b30144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b27e00640 1 -- 172.21.15.67:0/551223006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 msgr2=0x7f3b30144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b27e00640 1 --2- 172.21.15.67:0/551223006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b30144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b27e00640 1 -- 172.21.15.67:0/551223006 >> v1:172.21.15.86:6789/0 conn(0x7f3b30071840 legacy=0x7f3b3015f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:11.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.700+0000 7f3b27e00640 1 -- 172.21.15.67:0/551223006 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b300b4870 con 0x7f3b30144aa0 2024-04-17T10:35:11.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b27e00640 1 --2- 172.21.15.67:0/551223006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30144aa0 0x7f3b300d0910 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e4dc26f8d0eaab server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:11.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b2ea00640 1 -- 172.21.15.67:0/551223006 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b180aa600 con 0x7f3b30144aa0 2024-04-17T10:35:11.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b2ea00640 1 -- 172.21.15.67:0/551223006 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b180af070 con 0x7f3b30144aa0 2024-04-17T10:35:11.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b2ea00640 1 -- 172.21.15.67:0/551223006 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b180aa9c0 con 0x7f3b30144aa0 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 -- 172.21.15.67:0/551223006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30144aa0 msgr2=0x7f3b300d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 --2- 172.21.15.67:0/551223006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30144aa0 0x7f3b300d0910 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009870 tx=0x7f3b1801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 -- 172.21.15.67:0/551223006 shutdown_connections 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 --2- 172.21.15.67:0/551223006 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30144aa0 0x7f3b300d0910 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 --2- 172.21.15.67:0/551223006 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b30144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 -- 172.21.15.67:0/551223006 >> 172.21.15.67:0/551223006 conn(0x7f3b300b34d0 msgr2=0x7f3b300b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 -- 172.21.15.67:0/551223006 shutdown_connections 2024-04-17T10:35:11.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.704+0000 7f3b36000640 1 -- 172.21.15.67:0/551223006 wait complete. 2024-04-17T10:35:11.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 Processor -- start 2024-04-17T10:35:11.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 -- start start 2024-04-17T10:35:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30071840 0x7f3b30133d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b301606d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:11.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b30144aa0 0x7f3b3015c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b300d1260 con 0x7f3b30071840 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3015c0d0 con 0x7f3b30144aa0 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b30168a40 con 0x7f3b301446c0 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30071840 0x7f3b30133d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30071840 0x7f3b30133d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51636/0 (socket says 172.21.15.67:51636) 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 -- 172.21.15.67:0/2231486994 learned_addr learned my addr 172.21.15.67:0/2231486994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b27e00640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b301606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2fe00640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b30144aa0 0x7f3b3015c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 -- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 msgr2=0x7f3b301606d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b301606d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 -- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b30144aa0 msgr2=0x7f3b3015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:11.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b30144aa0 0x7f3b3015c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 -- 172.21.15.67:0/2231486994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b30134760 con 0x7f3b30071840 2024-04-17T10:35:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2fe00640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b30144aa0 0x7f3b3015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b2f400640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30071840 0x7f3b30133d50 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e930 tx=0x7f3b2000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b27400640 1 -- 172.21.15.67:0/2231486994 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20017070 con 0x7f3b30071840 2024-04-17T10:35:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b27400640 1 -- 172.21.15.67:0/2231486994 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2000ca00 con 0x7f3b30071840 2024-04-17T10:35:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b27400640 1 -- 172.21.15.67:0/2231486994 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2000cdb0 con 0x7f3b30071840 2024-04-17T10:35:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b30143120 con 0x7f3b30071840 2024-04-17T10:35:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b301436e0 con 0x7f3b30071840 2024-04-17T10:35:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.708+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b30070bb0 con 0x7f3b30071840 2024-04-17T10:35:11.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.712+0000 7f3b27400640 1 -- 172.21.15.67:0/2231486994 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3b2001a070 con 0x7f3b30071840 2024-04-17T10:35:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.712+0000 7f3b27400640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f90 0x7f3b0c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.712+0000 7f3b27400640 1 -- 172.21.15.67:0/2231486994 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3b20085260 con 0x7f3b30071840 2024-04-17T10:35:11.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.712+0000 7f3b27e00640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f90 0x7f3b0c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:11.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.712+0000 7f3b27e00640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f90 0x7f3b0c069450 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3b180ae000 tx=0x7f3b180a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:11.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.716+0000 7f3b27400640 1 -- 172.21.15.67:0/2231486994 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3b20055a20 con 0x7f3b30071840 2024-04-17T10:35:11.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.912+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b30074e30 con 0x7f3b30071840 2024-04-17T10:35:11.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.912+0000 7f3b27400640 1 -- 172.21.15.67:0/2231486994 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+4931 (secure 0 0 0) 0x7f3b20059ac0 con 0x7f3b30071840 2024-04-17T10:35:11.906 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:11.906 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":90,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9255,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:34:31.362617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9270":{"gid":9270,"name":"d","rank":0,"incarnation":86,"state":"up:active","state_seq":3,"addr":"172.21.15.86:6836/538421223","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":538421223},{"type":"v1","addr":"172.21.15.86:6836","nonce":538421223}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:11.906 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 90 2024-04-17T10:35:11.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f90 msgr2=0x7f3b0c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:11.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f90 0x7f3b0c069450 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f3b180ae000 tx=0x7f3b180a7000 comp rx=0 tx=0).stop 2024-04-17T10:35:11.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30071840 msgr2=0x7f3b30133d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30071840 0x7f3b30133d50 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f3b2000e930 tx=0x7f3b2000edf0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 shutdown_connections 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f90 0x7f3b0c069450 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b30144aa0 0x7f3b3015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b301446c0 0x7f3b301606d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 --2- 172.21.15.67:0/2231486994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b30071840 0x7f3b30133d50 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 >> 172.21.15.67:0/2231486994 conn(0x7f3b300b34d0 msgr2=0x7f3b300b2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 shutdown_connections 2024-04-17T10:35:11.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:11.916+0000 7f3b36000640 1 -- 172.21.15.67:0/2231486994 wait complete. 2024-04-17T10:35:11.927 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-04-17T10:35:16.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f5202200640 1 Processor -- start 2024-04-17T10:35:17.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f5202200640 1 -- start start 2024-04-17T10:35:17.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f5202200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f51fc15a060 0x7f51fc15e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f5202200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc06e740 con 0x7f51fc0ad530 2024-04-17T10:35:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f5202200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc06ea20 con 0x7f51fc06c3c0 2024-04-17T10:35:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f5202200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51fc0ad0a0 con 0x7f51fc15a060 2024-04-17T10:35:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f51fc0ad530 0x7f51fc071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41642/0 (socket says 172.21.15.67:41642) 2024-04-17T10:35:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fbe00640 1 -- 172.21.15.67:0/3552095071 learned_addr learned my addr 172.21.15.67:0/3552095071 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fb400640 1 --2- 172.21.15.67:0/3552095071 >> v2:172.21.15.86:3300/0 conn(0x7f51fc15a060 0x7f51fc15e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fa000640 1 -- 172.21.15.67:0/3552095071 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584450627 0 0) 0x7f51fc06ea20 con 0x7f51fc06c3c0 2024-04-17T10:35:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fa000640 1 -- 172.21.15.67:0/3552095071 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e0003530 con 0x7f51fc06c3c0 2024-04-17T10:35:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fa000640 1 -- 172.21.15.67:0/3552095071 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501034016 0 0) 0x7f51fc06e740 con 0x7f51fc0ad530 2024-04-17T10:35:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fa000640 1 -- 172.21.15.67:0/3552095071 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51fc06ea20 con 0x7f51fc0ad530 2024-04-17T10:35:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fb400640 1 -- 172.21.15.67:0/3552095071 >> v1:172.21.15.86:6789/0 conn(0x7f51fc06c3c0 legacy=0x7f51fc0b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fb400640 1 -- 172.21.15.67:0/3552095071 >> v1:172.21.15.67:6789/0 conn(0x7f51fc0ad530 legacy=0x7f51fc071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.160+0000 7f51fb400640 1 -- 172.21.15.67:0/3552095071 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc06ed00 con 0x7f51fc15a060 2024-04-17T10:35:17.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.164+0000 7f51fb400640 1 --2- 172.21.15.67:0/3552095071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc15a060 0x7f51fc15e640 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0097c0 tx=0x7f51ec01c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9c4ea9e96fe4cc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:17.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.164+0000 7f51fa000640 1 -- 172.21.15.67:0/3552095071 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0b0070 con 0x7f51fc15a060 2024-04-17T10:35:17.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.164+0000 7f51fa000640 1 -- 172.21.15.67:0/3552095071 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec0ab600 con 0x7f51fc15a060 2024-04-17T10:35:17.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.164+0000 7f51fa000640 1 -- 172.21.15.67:0/3552095071 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0ab9d0 con 0x7f51fc15a060 2024-04-17T10:35:17.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 -- 172.21.15.67:0/3552095071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc15a060 msgr2=0x7f51fc15e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 --2- 172.21.15.67:0/3552095071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc15a060 0x7f51fc15e640 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0097c0 tx=0x7f51ec01c3d0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 -- 172.21.15.67:0/3552095071 shutdown_connections 2024-04-17T10:35:17.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 --2- 172.21.15.67:0/3552095071 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc15a060 0x7f51fc15e640 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 -- 172.21.15.67:0/3552095071 >> 172.21.15.67:0/3552095071 conn(0x7f51fc0af670 msgr2=0x7f51fc0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:17.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 -- 172.21.15.67:0/3552095071 shutdown_connections 2024-04-17T10:35:17.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 -- 172.21.15.67:0/3552095071 wait complete. 2024-04-17T10:35:17.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 Processor -- start 2024-04-17T10:35:17.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 -- start start 2024-04-17T10:35:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.168+0000 7f5202200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc06c3c0 0x7f51fc13e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0ad530 0x7f51fc145f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc06c3c0 0x7f51fc13e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51fc15a060 0x7f51fc1464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0724f0 con 0x7f51fc15a060 2024-04-17T10:35:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc072370 con 0x7f51fc06c3c0 2024-04-17T10:35:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc0ad940 con 0x7f51fc0ad530 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc06c3c0 0x7f51fc13e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:51586/0 (socket says 172.21.15.67:51586) 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 -- 172.21.15.67:0/1171456302 learned_addr learned my addr 172.21.15.67:0/1171456302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51faa00640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0ad530 0x7f51fc145f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fbe00640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51fc15a060 0x7f51fc1464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 -- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0ad530 msgr2=0x7f51fc145f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0ad530 0x7f51fc145f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 -- 172.21.15.67:0/1171456302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51fc15a060 msgr2=0x7f51fc1464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51fc15a060 0x7f51fc1464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 -- 172.21.15.67:0/1171456302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc15bd80 con 0x7f51fc06c3c0 2024-04-17T10:35:17.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51fb400640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc06c3c0 0x7f51fc13e1f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f51ec01a000 tx=0x7f51ec019170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:17.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51f3400640 1 -- 172.21.15.67:0/1171456302 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0b9020 con 0x7f51fc06c3c0 2024-04-17T10:35:17.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51f3400640 1 -- 172.21.15.67:0/1171456302 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec0b5480 con 0x7f51fc06c3c0 2024-04-17T10:35:17.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc15b2b0 con 0x7f51fc06c3c0 2024-04-17T10:35:17.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f51f3400640 1 -- 172.21.15.67:0/1171456302 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0b0040 con 0x7f51fc06c3c0 2024-04-17T10:35:17.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc15b870 con 0x7f51fc06c3c0 2024-04-17T10:35:17.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.172+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51fc0b5400 con 0x7f51fc06c3c0 2024-04-17T10:35:17.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.176+0000 7f51f3400640 1 -- 172.21.15.67:0/1171456302 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f51ec0bc020 con 0x7f51fc06c3c0 2024-04-17T10:35:17.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.176+0000 7f51f3400640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f51dc066f60 0x7f51dc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.176+0000 7f51f3400640 1 -- 172.21.15.67:0/1171456302 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f51ec11f530 con 0x7f51fc06c3c0 2024-04-17T10:35:17.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.176+0000 7f51faa00640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f51dc066f60 0x7f51dc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.176+0000 7f51faa00640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f51dc066f60 0x7f51dc069420 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f51e8000c10 tx=0x7f51e8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:17.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.180+0000 7f51f3400640 1 -- 172.21.15.67:0/1171456302 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f51ec016050 con 0x7f51fc06c3c0 2024-04-17T10:35:17.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.552+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51fc074e30 con 0x7f51fc06c3c0 2024-04-17T10:35:17.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f51f3400640 1 -- 172.21.15.67:0/1171456302 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4146 (secure 0 0 0) 0x7f51ec0f03d0 con 0x7f51fc06c3c0 2024-04-17T10:35:17.547 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:17.547 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:35:15.983015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9255":{"gid":9255,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:17.547 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T10:35:17.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f51dc066f60 msgr2=0x7f51dc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f51dc066f60 0x7f51dc069420 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f51e8000c10 tx=0x7f51e8009040 comp rx=0 tx=0).stop 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc06c3c0 msgr2=0x7f51fc13e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc06c3c0 0x7f51fc13e1f0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f51ec01a000 tx=0x7f51ec019170 comp rx=0 tx=0).stop 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 shutdown_connections 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f51dc066f60 0x7f51dc069420 secure :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f51e8000c10 tx=0x7f51e8009040 comp rx=0 tx=0).stop 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f51fc15a060 0x7f51fc1464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f51fc0ad530 0x7f51fc145f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 --2- 172.21.15.67:0/1171456302 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f51fc06c3c0 0x7f51fc13e1f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 >> 172.21.15.67:0/1171456302 conn(0x7f51fc0af670 msgr2=0x7f51fc073670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:17.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 shutdown_connections 2024-04-17T10:35:17.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.556+0000 7f5202200640 1 -- 172.21.15.67:0/1171456302 wait complete. 2024-04-17T10:35:17.581 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s and 0 retries 2024-04-17T10:35:17.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefc6200640 1 Processor -- start 2024-04-17T10:35:17.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefc6200640 1 -- start start 2024-04-17T10:35:17.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefc6200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fefc015bb50 0x7fefc00ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefc6200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc00b0ef0 0x7fefc00ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefc6200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefc015fea0 con 0x7fefc015b770 2024-04-17T10:35:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefc6200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefc0160180 con 0x7fefc015bb50 2024-04-17T10:35:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefc6200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00b1a60 con 0x7fefc00b0ef0 2024-04-17T10:35:17.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefbea00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fefc015bb50 0x7fefc00ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefbf400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fefc015b770 0x7fefc00cc1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41656/0 (socket says 172.21.15.67:41656) 2024-04-17T10:35:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefbf400640 1 -- 172.21.15.67:0/4057587386 learned_addr learned my addr 172.21.15.67:0/4057587386 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.812+0000 7fefbfe00640 1 --2- 172.21.15.67:0/4057587386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc00b0ef0 0x7fefc00ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.816+0000 7fefbfe00640 1 -- 172.21.15.67:0/4057587386 >> v2:172.21.15.86:3300/0 conn(0x7fefc015bb50 msgr2=0x7fefc00ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.816+0000 7fefbfe00640 1 --2- 172.21.15.67:0/4057587386 >> v2:172.21.15.86:3300/0 conn(0x7fefc015bb50 0x7fefc00ce0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.816+0000 7fefbfe00640 1 -- 172.21.15.67:0/4057587386 >> v1:172.21.15.67:6789/0 conn(0x7fefc015b770 legacy=0x7fefc00cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.816+0000 7fefbfe00640 1 -- 172.21.15.67:0/4057587386 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc00b1d40 con 0x7fefc00b0ef0 2024-04-17T10:35:17.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefbfe00640 1 --2- 172.21.15.67:0/4057587386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc00b0ef0 0x7fefc00ce5e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009d90 tx=0x7fefb801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31b39717c43bab38 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:17.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefbe000640 1 -- 172.21.15.67:0/4057587386 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80b2070 con 0x7fefc00b0ef0 2024-04-17T10:35:17.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefbe000640 1 -- 172.21.15.67:0/4057587386 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb80a5940 con 0x7fefc00b0ef0 2024-04-17T10:35:17.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefbe000640 1 -- 172.21.15.67:0/4057587386 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80a5d10 con 0x7fefc00b0ef0 2024-04-17T10:35:17.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 -- 172.21.15.67:0/4057587386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc00b0ef0 msgr2=0x7fefc00ce5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 --2- 172.21.15.67:0/4057587386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc00b0ef0 0x7fefc00ce5e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009d90 tx=0x7fefb801c970 comp rx=0 tx=0).stop 2024-04-17T10:35:17.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 -- 172.21.15.67:0/4057587386 shutdown_connections 2024-04-17T10:35:17.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 --2- 172.21.15.67:0/4057587386 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc00b0ef0 0x7fefc00ce5e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 --2- 172.21.15.67:0/4057587386 >> v2:172.21.15.86:3300/0 conn(0x7fefc015bb50 0x7fefc00ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 -- 172.21.15.67:0/4057587386 >> 172.21.15.67:0/4057587386 conn(0x7fefc00b3410 msgr2=0x7fefc00b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:17.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 -- 172.21.15.67:0/4057587386 shutdown_connections 2024-04-17T10:35:17.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.820+0000 7fefc6200640 1 -- 172.21.15.67:0/4057587386 wait complete. 2024-04-17T10:35:17.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 Processor -- start 2024-04-17T10:35:17.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 -- start start 2024-04-17T10:35:17.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefc00b0ef0 0x7fefc01fac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc015b770 0x7fefc0158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefc015bb50 0x7fefc00c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00b3120 con 0x7fefc00b0ef0 2024-04-17T10:35:17.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00b2fa0 con 0x7fefc015bb50 2024-04-17T10:35:17.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefc6200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefc0168830 con 0x7fefc015b770 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc015b770 0x7fefc0158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefc015bb50 0x7fefc00c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefc015bb50 0x7fefc00c8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:51614/0 (socket says 172.21.15.67:51614) 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 -- 172.21.15.67:0/141237402 learned_addr learned my addr 172.21.15.67:0/141237402 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc015b770 0x7fefc0158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:42088/0 (socket says 172.21.15.67:42088) 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbf400640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefc00b0ef0 0x7fefc01fac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 -- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc015b770 msgr2=0x7fefc0158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc015b770 0x7fefc0158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 -- 172.21.15.67:0/141237402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefc00b0ef0 msgr2=0x7fefc01fac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:17.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefc00b0ef0 0x7fefc01fac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:17.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 -- 172.21.15.67:0/141237402 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefc01599b0 con 0x7fefc015bb50 2024-04-17T10:35:17.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.824+0000 7fefbfe00640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefc015bb50 0x7fefc00c8990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fefb80b0040 tx=0x7fefb8019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:17.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefb7400640 1 -- 172.21.15.67:0/141237402 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80b2050 con 0x7fefc015bb50 2024-04-17T10:35:17.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefb7400640 1 -- 172.21.15.67:0/141237402 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb80be070 con 0x7fefc015bb50 2024-04-17T10:35:17.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefb7400640 1 -- 172.21.15.67:0/141237402 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80adc40 con 0x7fefc015bb50 2024-04-17T10:35:17.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc0159c90 con 0x7fefc015bb50 2024-04-17T10:35:17.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc015a250 con 0x7fefc015bb50 2024-04-17T10:35:17.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc009fa00 con 0x7fefc015bb50 2024-04-17T10:35:17.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefb7400640 1 -- 172.21.15.67:0/141237402 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fefb80c3460 con 0x7fefc015bb50 2024-04-17T10:35:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefb7400640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef9c066fb0 0x7fef9c069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.828+0000 7fefb7400640 1 -- 172.21.15.67:0/141237402 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fefb8120560 con 0x7fefc015bb50 2024-04-17T10:35:17.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.832+0000 7fefbf400640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef9c066fb0 0x7fef9c069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:17.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.832+0000 7fefbf400640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef9c066fb0 0x7fef9c069470 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fefb0005890 tx=0x7fefb0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:17.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:17.832+0000 7fefb7400640 1 -- 172.21.15.67:0/141237402 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fefb8016560 con 0x7fefc015bb50 2024-04-17T10:35:18.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.036+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefc0074e30 con 0x7fefc015bb50 2024-04-17T10:35:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefb7400640 1 -- 172.21.15.67:0/141237402 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4146 (secure 0 0 0) 0x7fefb80f12e0 con 0x7fefc015bb50 2024-04-17T10:35:18.031 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:18.031 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:35:15.983015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9255":{"gid":9255,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:18.031 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T10:35:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef9c066fb0 msgr2=0x7fef9c069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef9c066fb0 0x7fef9c069470 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fefb0005890 tx=0x7fefb0009040 comp rx=0 tx=0).stop 2024-04-17T10:35:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefc015bb50 msgr2=0x7fefc00c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefc015bb50 0x7fefc00c8990 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fefb80b0040 tx=0x7fefb8019710 comp rx=0 tx=0).stop 2024-04-17T10:35:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 shutdown_connections 2024-04-17T10:35:18.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fef9c066fb0 0x7fef9c069470 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fefc015bb50 0x7fefc00c8990 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fefc015b770 0x7fefc0158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 --2- 172.21.15.67:0/141237402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefc00b0ef0 0x7fefc01fac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 >> 172.21.15.67:0/141237402 conn(0x7fefc00b3410 msgr2=0x7fefc01582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 shutdown_connections 2024-04-17T10:35:18.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.040+0000 7fefc6200640 1 -- 172.21.15.67:0/141237402 wait complete. 2024-04-17T10:35:18.052 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-04-17T10:35:18.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-17T10:35:18.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.200+0000 7f6b42c00640 1 Processor -- start 2024-04-17T10:35:18.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.200+0000 7f6b42c00640 1 -- start start 2024-04-17T10:35:18.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b42c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b42c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c0cc530 con 0x7f6b3c15e440 2024-04-17T10:35:18.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b42c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c0ce930 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b42c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c0cbc40 con 0x7f6b3c0b2dc0 2024-04-17T10:35:18.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b3be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6b3c15e440 0x7f6b3c15e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41664/0 (socket says 172.21.15.67:41664) 2024-04-17T10:35:18.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b3be00640 1 -- 172.21.15.67:0/1468476915 learned_addr learned my addr 172.21.15.67:0/1468476915 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:18.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b33400640 1 --2- 172.21.15.67:0/1468476915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b3b400640 1 -- 172.21.15.67:0/1468476915 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998615448 0 0) 0x7f6b3c0cc530 con 0x7f6b3c15e440 2024-04-17T10:35:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b3b400640 1 -- 172.21.15.67:0/1468476915 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b20003530 con 0x7f6b3c15e440 2024-04-17T10:35:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b3b400640 1 -- 172.21.15.67:0/1468476915 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977888612 0 0) 0x7f6b3c0ce930 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b33400640 1 -- 172.21.15.67:0/1468476915 >> v1:172.21.15.86:6789/0 conn(0x7f6b3c0cc0d0 legacy=0x7f6b3c06e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b33400640 1 -- 172.21.15.67:0/1468476915 >> v1:172.21.15.67:6789/0 conn(0x7f6b3c15e440 legacy=0x7f6b3c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.204+0000 7f6b33400640 1 -- 172.21.15.67:0/1468476915 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0cb310 con 0x7f6b3c0b2dc0 2024-04-17T10:35:18.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b33400640 1 --2- 172.21.15.67:0/1468476915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0ce2a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6b240097f0 tx=0x7f6b2401c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcc4bd00bd98f41c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:18.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b3b400640 1 -- 172.21.15.67:0/1468476915 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b240aa610 con 0x7f6b3c0b2dc0 2024-04-17T10:35:18.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b3b400640 1 -- 172.21.15.67:0/1468476915 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b240af070 con 0x7f6b3c0b2dc0 2024-04-17T10:35:18.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b3b400640 1 -- 172.21.15.67:0/1468476915 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b240aa9d0 con 0x7f6b3c0b2dc0 2024-04-17T10:35:18.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 -- 172.21.15.67:0/1468476915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0b2dc0 msgr2=0x7f6b3c0ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 --2- 172.21.15.67:0/1468476915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0ce2a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6b240097f0 tx=0x7f6b2401c4b0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 -- 172.21.15.67:0/1468476915 shutdown_connections 2024-04-17T10:35:18.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 --2- 172.21.15.67:0/1468476915 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0ce2a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 -- 172.21.15.67:0/1468476915 >> 172.21.15.67:0/1468476915 conn(0x7f6b3c0b22a0 msgr2=0x7f6b3c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 -- 172.21.15.67:0/1468476915 shutdown_connections 2024-04-17T10:35:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 -- 172.21.15.67:0/1468476915 wait complete. 2024-04-17T10:35:18.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 Processor -- start 2024-04-17T10:35:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.208+0000 7f6b42c00640 1 -- start start 2024-04-17T10:35:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0cc0d0 0x7f6b3c0c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c15e440 0x7f6b3c0c2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c15ab70 con 0x7f6b3c15e440 2024-04-17T10:35:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c06f1f0 con 0x7f6b3c0b2dc0 2024-04-17T10:35:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b3c15acf0 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b40c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c15e440 0x7f6b3c0c2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b3be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0cc0d0 0x7f6b3c0c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0cc0d0 0x7f6b3c0c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:42114/0 (socket says 172.21.15.67:42114) 2024-04-17T10:35:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b40c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c15e440 0x7f6b3c0c2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51662/0 (socket says 172.21.15.67:51662) 2024-04-17T10:35:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 -- 172.21.15.67:0/2492076583 learned_addr learned my addr 172.21.15.67:0/2492076583 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:18.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b3be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0c97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:51628/0 (socket says 172.21.15.67:51628) 2024-04-17T10:35:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 -- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c0b2dc0 msgr2=0x7f6b3c0c97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0c97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 -- 172.21.15.67:0/2492076583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c15e440 msgr2=0x7f6b3c0c2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c15e440 0x7f6b3c0c2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 -- 172.21.15.67:0/2492076583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c15d2b0 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b3be00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b40c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c15e440 0x7f6b3c0c2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b33400640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0cc0d0 0x7f6b3c0c77a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b240ac040 tx=0x7f6b2401acf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b38c00640 1 -- 172.21.15.67:0/2492076583 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b240be410 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b38c00640 1 -- 172.21.15.67:0/2492076583 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b240be5d0 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b38c00640 1 -- 172.21.15.67:0/2492076583 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b240be9c0 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c15d590 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c0c9d20 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.212+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b3c09e900 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.216+0000 7f6b38c00640 1 -- 172.21.15.67:0/2492076583 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6b240bc020 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.216+0000 7f6b38c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b18066f60 0x7f6b18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.216+0000 7f6b38c00640 1 -- 172.21.15.67:0/2492076583 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6b2411f750 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.216+0000 7f6b3be00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b18066f60 0x7f6b18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.216+0000 7f6b3be00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b18066f60 0x7f6b18069420 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c002240 tx=0x7f6b2c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:18.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.220+0000 7f6b38c00640 1 -- 172.21.15.67:0/2492076583 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6b240a7070 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.440+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f6b3c074e30 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.440+0000 7f6b38c00640 1 -- 172.21.15.67:0/2492076583 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1377 (secure 0 0 0) 0x7f6b2411de20 con 0x7f6b3c0cc0d0 2024-04-17T10:35:18.433 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:18.433 INFO:teuthology.orchestra.run.smithi067.stdout:{"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":649,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":131,"num_osds":8,"num_up_osds":8,"osd_up_since":1713349481,"num_in_osds":8,"osd_in_since":1713349471,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076722292,"bytes_used":2608103424,"bytes_avail":770486009856,"bytes_total":773094113280},"fsmap":{"epoch":94,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"a","status":"up:active","gid":9255}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":26,"modified":"2024-04-17T10:35:14.283397+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b18066f60 msgr2=0x7f6b18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b18066f60 0x7f6b18069420 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c002240 tx=0x7f6b2c009040 comp rx=0 tx=0).stop 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0cc0d0 msgr2=0x7f6b3c0c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0cc0d0 0x7f6b3c0c77a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6b240ac040 tx=0x7f6b2401acf0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 shutdown_connections 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6b18066f60 0x7f6b18069420 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b3c15e440 0x7f6b3c0c2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6b3c0cc0d0 0x7f6b3c0c77a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 --2- 172.21.15.67:0/2492076583 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6b3c0b2dc0 0x7f6b3c0c97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 >> 172.21.15.67:0/2492076583 conn(0x7f6b3c0b22a0 msgr2=0x7f6b3c0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 shutdown_connections 2024-04-17T10:35:18.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.444+0000 7f6b42c00640 1 -- 172.21.15.67:0/2492076583 wait complete. 2024-04-17T10:35:18.454 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-04-17T10:35:18.455 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:35:18.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:35:18.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:35:18.617 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:35:18.618 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:35:18.618 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:18.627 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:35:18.627 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:18.679 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3375c00640 1 Processor -- start 2024-04-17T10:35:18.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3375c00640 1 -- start start 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3375c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33700d0030 0x7f33700cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3375c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337006b6a0 0x7f337006f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3375c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33700700a0 con 0x7f337015de50 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3375c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33700cad20 con 0x7f33700d0030 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3375c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3370070380 con 0x7f337006b6a0 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3366000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33700d0030 0x7f33700cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3366000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33700d0030 0x7f33700cc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51668/0 (socket says 172.21.15.67:51668) 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3366000640 1 -- 172.21.15.67:0/3413591670 learned_addr learned my addr 172.21.15.67:0/3413591670 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f336ea00640 1 --1- 172.21.15.67:0/3413591670 >> v1:172.21.15.67:6789/0 conn(0x7f337015de50 0x7f33700cf920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41680/0 (socket says 172.21.15.67:41680) 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f336f400640 1 --2- 172.21.15.67:0/3413591670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337006b6a0 0x7f337006f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3366000640 1 -- 172.21.15.67:0/3413591670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337006b6a0 msgr2=0x7f337006f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:18.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3366000640 1 --2- 172.21.15.67:0/3413591670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337006b6a0 0x7f337006f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3366000640 1 -- 172.21.15.67:0/3413591670 >> v1:172.21.15.67:6789/0 conn(0x7f337015de50 legacy=0x7f33700cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f3366000640 1 -- 172.21.15.67:0/3413591670 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33700caf70 con 0x7f33700d0030 2024-04-17T10:35:18.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f336e000640 1 -- 172.21.15.67:0/3413591670 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33182918 0 0) 0x7f33700700a0 con 0x7f337015de50 2024-04-17T10:35:18.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.872+0000 7f336f400640 1 --2- 172.21.15.67:0/3413591670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337006b6a0 0x7f337006f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3366000640 1 --2- 172.21.15.67:0/3413591670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33700cc7d0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f3358009870 tx=0x7f335801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e012c9326266b82 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f336e000640 1 -- 172.21.15.67:0/3413591670 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33580b0070 con 0x7f33700d0030 2024-04-17T10:35:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f336e000640 1 -- 172.21.15.67:0/3413591670 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33580ab600 con 0x7f33700d0030 2024-04-17T10:35:18.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f336e000640 1 -- 172.21.15.67:0/3413591670 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33580ab9d0 con 0x7f33700d0030 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 -- 172.21.15.67:0/3413591670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 msgr2=0x7f33700cc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 --2- 172.21.15.67:0/3413591670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33700cc7d0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f3358009870 tx=0x7f335801c440 comp rx=0 tx=0).stop 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 -- 172.21.15.67:0/3413591670 shutdown_connections 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 --2- 172.21.15.67:0/3413591670 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337006b6a0 0x7f337006f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 --2- 172.21.15.67:0/3413591670 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33700cc7d0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 -- 172.21.15.67:0/3413591670 >> 172.21.15.67:0/3413591670 conn(0x7f33700b22a0 msgr2=0x7f33700b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 -- 172.21.15.67:0/3413591670 shutdown_connections 2024-04-17T10:35:18.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.876+0000 7f3375c00640 1 -- 172.21.15.67:0/3413591670 wait complete. 2024-04-17T10:35:18.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 Processor -- start 2024-04-17T10:35:18.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 -- start start 2024-04-17T10:35:18.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f337006b6a0 0x7f337015bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33701559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337015de50 0x7f3370155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700cc350 con 0x7f33700d0030 2024-04-17T10:35:18.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700cc1d0 con 0x7f337006b6a0 2024-04-17T10:35:18.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337016a7f0 con 0x7f337015de50 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33701559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33701559e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51682/0 (socket says 172.21.15.67:51682) 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 -- 172.21.15.67:0/780383292 learned_addr learned my addr 172.21.15.67:0/780383292 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f336ea00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f337006b6a0 0x7f337015bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f336f400640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337015de50 0x7f3370155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 -- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337015de50 msgr2=0x7f3370155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337015de50 0x7f3370155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 -- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f337006b6a0 msgr2=0x7f337015bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f337006b6a0 0x7f337015bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 -- 172.21.15.67:0/780383292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33700aef70 con 0x7f33700d0030 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f336ea00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f337006b6a0 0x7f337015bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:18.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f336f400640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337015de50 0x7f3370155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:18.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3366000640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33701559e0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f33580ae040 tx=0x7f335801ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:18.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3367400640 1 -- 172.21.15.67:0/780383292 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33580b0040 con 0x7f33700d0030 2024-04-17T10:35:18.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3367400640 1 -- 172.21.15.67:0/780383292 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3358004030 con 0x7f33700d0030 2024-04-17T10:35:18.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33700af250 con 0x7f33700d0030 2024-04-17T10:35:18.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3367400640 1 -- 172.21.15.67:0/780383292 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358004400 con 0x7f33700d0030 2024-04-17T10:35:18.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33700af810 con 0x7f33700d0030 2024-04-17T10:35:18.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.880+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337009e900 con 0x7f33700d0030 2024-04-17T10:35:18.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.884+0000 7f3367400640 1 -- 172.21.15.67:0/780383292 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f33580aa7d0 con 0x7f33700d0030 2024-04-17T10:35:18.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.884+0000 7f3367400640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f334c066f60 0x7f334c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:18.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.884+0000 7f3367400640 1 -- 172.21.15.67:0/780383292 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f335811fea0 con 0x7f33700d0030 2024-04-17T10:35:18.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.884+0000 7f336ea00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f334c066f60 0x7f334c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:18.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.884+0000 7f336ea00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f334c066f60 0x7f334c069420 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3360000f70 tx=0x7f3360009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:18.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:18.888+0000 7f3367400640 1 -- 172.21.15.67:0/780383292 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3358016150 con 0x7f33700d0030 2024-04-17T10:35:18.908 INFO:tasks.ceph.mds.a.smithi067.stderr:2024-04-17T10:35:18.916+0000 7ff054800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:35:19.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.052+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3370074e30 con 0x7f33700d0030 2024-04-17T10:35:19.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.052+0000 7f3367400640 1 -- 172.21.15.67:0/780383292 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16979 (secure 0 0 0) 0x7f33580f0c30 con 0x7f33700d0030 2024-04-17T10:35:19.046 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:19.046 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":131,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:13.952681+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:34:26.120176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-17T10:34:27.343196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6836/538421223":"2024-04-18T10:35:13.952580+0000","172.21.15.86:6834/538421223":"2024-04-18T10:35:13.952580+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:19.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f334c066f60 msgr2=0x7f334c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f334c066f60 0x7f334c069420 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3360000f70 tx=0x7f3360009040 comp rx=0 tx=0).stop 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 msgr2=0x7f33701559e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33701559e0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f33580ae040 tx=0x7f335801ac80 comp rx=0 tx=0).stop 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 shutdown_connections 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f334c066f60 0x7f334c069420 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f337015de50 0x7f3370155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33700d0030 0x7f33701559e0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 --2- 172.21.15.67:0/780383292 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f337006b6a0 0x7f337015bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 >> 172.21.15.67:0/780383292 conn(0x7f33700b22a0 msgr2=0x7f33700b0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 shutdown_connections 2024-04-17T10:35:19.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.056+0000 7f3375c00640 1 -- 172.21.15.67:0/780383292 wait complete. 2024-04-17T10:35:19.068 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:34:26.120176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:34:27.343196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:35:19.068 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:35:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.220+0000 7f2f36600640 1 Processor -- start 2024-04-17T10:35:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.220+0000 7f2f36600640 1 -- start start 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.220+0000 7f2f36600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300cd030 0x7f2f3006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.220+0000 7f2f36600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f3006e7b0 0x7f2f300b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.220+0000 7f2f36600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300b4990 con 0x7f2f300aef60 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.220+0000 7f2f36600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f300af340 con 0x7f2f3006e7b0 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.220+0000 7f2f36600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3015ae90 con 0x7f2f300cd030 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f3006e7b0 0x7f2f300b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f3006e7b0 0x7f2f300b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51686/0 (socket says 172.21.15.67:51686) 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 -- 172.21.15.67:0/278005937 learned_addr learned my addr 172.21.15.67:0/278005937 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f27e00640 1 --2- 172.21.15.67:0/278005937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300cd030 0x7f2f3006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 -- 172.21.15.67:0/278005937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300cd030 msgr2=0x7f2f3006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 --2- 172.21.15.67:0/278005937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300cd030 0x7f2f3006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 -- 172.21.15.67:0/278005937 >> v1:172.21.15.86:6789/0 conn(0x7f2f300aef60 legacy=0x7f2f300cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 -- 172.21.15.67:0/278005937 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3006ee80 con 0x7f2f3006e7b0 2024-04-17T10:35:19.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2fe00640 1 --2- 172.21.15.67:0/278005937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f300b3120 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009e50 tx=0x7f2f2801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2533e4e24f8cb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2ea00640 1 -- 172.21.15.67:0/278005937 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f280a5750 con 0x7f2f3006e7b0 2024-04-17T10:35:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2ea00640 1 -- 172.21.15.67:0/278005937 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f280a58f0 con 0x7f2f3006e7b0 2024-04-17T10:35:19.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.224+0000 7f2f2ea00640 1 -- 172.21.15.67:0/278005937 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f280a5cc0 con 0x7f2f3006e7b0 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- 172.21.15.67:0/278005937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 msgr2=0x7f2f300b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 --2- 172.21.15.67:0/278005937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f300b3120 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009e50 tx=0x7f2f2801c9a0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- 172.21.15.67:0/278005937 shutdown_connections 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 --2- 172.21.15.67:0/278005937 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f300b3120 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 --2- 172.21.15.67:0/278005937 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300cd030 0x7f2f3006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- 172.21.15.67:0/278005937 >> 172.21.15.67:0/278005937 conn(0x7f2f300b1740 msgr2=0x7f2f300b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- 172.21.15.67:0/278005937 shutdown_connections 2024-04-17T10:35:19.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- 172.21.15.67:0/278005937 wait complete. 2024-04-17T10:35:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 Processor -- start 2024-04-17T10:35:19.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- start start 2024-04-17T10:35:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f3014aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300aef60 0x7f2f30148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f300cd030 0x7f2f30148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3015bc10 con 0x7f2f3006e7b0 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300b3e70 con 0x7f2f300cd030 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f36600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3014ca00 con 0x7f2f300aef60 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f3014aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f3014aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51692/0 (socket says 172.21.15.67:51692) 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 -- 172.21.15.67:0/1763063676 learned_addr learned my addr 172.21.15.67:0/1763063676 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2fe00640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f300cd030 0x7f2f30148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f27e00640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300aef60 0x7f2f30148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 -- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300aef60 msgr2=0x7f2f30148520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300aef60 0x7f2f30148520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 -- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f300cd030 msgr2=0x7f2f30148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f300cd030 0x7f2f30148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.228+0000 7f2f2f400640 1 -- 172.21.15.67:0/1763063676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300ae7e0 con 0x7f2f3006e7b0 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f2f400640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f3014aba0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000c420 tx=0x7f2f2000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f27400640 1 -- 172.21.15.67:0/1763063676 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f200052d0 con 0x7f2f3006e7b0 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f27400640 1 -- 172.21.15.67:0/1763063676 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f20005470 con 0x7f2f3006e7b0 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f27400640 1 -- 172.21.15.67:0/1763063676 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f20007260 con 0x7f2f3006e7b0 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f30152ea0 con 0x7f2f3006e7b0 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f30153460 con 0x7f2f3006e7b0 2024-04-17T10:35:19.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f3004ef30 con 0x7f2f3006e7b0 2024-04-17T10:35:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f27400640 1 -- 172.21.15.67:0/1763063676 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2f20007400 con 0x7f2f3006e7b0 2024-04-17T10:35:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f27400640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f0c066f60 0x7f2f0c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f27400640 1 -- 172.21.15.67:0/1763063676 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2f20084710 con 0x7f2f3006e7b0 2024-04-17T10:35:19.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.232+0000 7f2f27e00640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f0c066f60 0x7f2f0c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.236+0000 7f2f27e00640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f0c066f60 0x7f2f0c069420 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2f300cf630 tx=0x7f2f18009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.236+0000 7f2f27400640 1 -- 172.21.15.67:0/1763063676 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2f300b4990 con 0x7f2f3006e7b0 2024-04-17T10:35:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2f3014fee0 con 0x7f2f3006e7b0 2024-04-17T10:35:19.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f27400640 1 -- 172.21.15.67:0/1763063676 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2f20054e50 con 0x7f2f3006e7b0 2024-04-17T10:35:19.393 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:35:19.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f0c066f60 msgr2=0x7f2f0c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f0c066f60 0x7f2f0c069420 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f2f300cf630 tx=0x7f2f18009190 comp rx=0 tx=0).stop 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 msgr2=0x7f2f3014aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f3014aba0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f2f2000c420 tx=0x7f2f2000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 shutdown_connections 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f0c066f60 0x7f2f0c069420 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f300cd030 0x7f2f30148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f300aef60 0x7f2f30148520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 --2- 172.21.15.67:0/1763063676 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f3006e7b0 0x7f2f3014aba0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.400+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 >> 172.21.15.67:0/1763063676 conn(0x7f2f300b1740 msgr2=0x7f2f300cbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.404+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 shutdown_connections 2024-04-17T10:35:19.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.404+0000 7f2f36600640 1 -- 172.21.15.67:0/1763063676 wait complete. 2024-04-17T10:35:19.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:35:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 Processor -- start 2024-04-17T10:35:19.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 -- start start 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc071ed0 0x7fdffc15a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdffc0722b0 0x7fdffc0aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdffc15e210 0x7fdffc15e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdffc0ccbf0 con 0x7fdffc15e210 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdffc0af2e0 con 0x7fdffc0722b0 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fe002000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc15ac40 con 0x7fdffc071ed0 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdffc15e210 0x7fdffc15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdffc15e210 0x7fdffc15e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51702/0 (socket says 172.21.15.67:51702) 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 -- 172.21.15.67:0/1512550320 learned_addr learned my addr 172.21.15.67:0/1512550320 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffb400640 1 --2- 172.21.15.67:0/1512550320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc071ed0 0x7fdffc15a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdff3e00640 1 --2- 172.21.15.67:0/1512550320 >> v2:172.21.15.86:3300/0 conn(0x7fdffc0722b0 0x7fdffc0aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 -- 172.21.15.67:0/1512550320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc071ed0 msgr2=0x7fdffc15a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 --2- 172.21.15.67:0/1512550320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc071ed0 0x7fdffc15a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 -- 172.21.15.67:0/1512550320 >> v2:172.21.15.86:3300/0 conn(0x7fdffc0722b0 msgr2=0x7fdffc0aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 --2- 172.21.15.67:0/1512550320 >> v2:172.21.15.86:3300/0 conn(0x7fdffc0722b0 0x7fdffc0aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.568+0000 7fdffbe00640 1 -- 172.21.15.67:0/1512550320 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc15af20 con 0x7fdffc15e210 2024-04-17T10:35:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fdffbe00640 1 --2- 172.21.15.67:0/1512550320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc15e210 0x7fdffc15e670 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fdffc072880 tx=0x7fdff401c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de19992f4cefa92 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fdffaa00640 1 -- 172.21.15.67:0/1512550320 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff40a55f0 con 0x7fdffc15e210 2024-04-17T10:35:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fdffaa00640 1 -- 172.21.15.67:0/1512550320 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff40a5790 con 0x7fdffc15e210 2024-04-17T10:35:19.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fdffaa00640 1 -- 172.21.15.67:0/1512550320 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff40a5b60 con 0x7fdffc15e210 2024-04-17T10:35:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 -- 172.21.15.67:0/1512550320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc15e210 msgr2=0x7fdffc15e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 --2- 172.21.15.67:0/1512550320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc15e210 0x7fdffc15e670 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fdffc072880 tx=0x7fdff401c9f0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 -- 172.21.15.67:0/1512550320 shutdown_connections 2024-04-17T10:35:19.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 --2- 172.21.15.67:0/1512550320 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc15e210 0x7fdffc15e670 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 --2- 172.21.15.67:0/1512550320 >> v2:172.21.15.86:3300/0 conn(0x7fdffc0722b0 0x7fdffc0aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 --2- 172.21.15.67:0/1512550320 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc071ed0 0x7fdffc15a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 -- 172.21.15.67:0/1512550320 >> 172.21.15.67:0/1512550320 conn(0x7fdffc0b1740 msgr2=0x7fdffc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 -- 172.21.15.67:0/1512550320 shutdown_connections 2024-04-17T10:35:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.572+0000 7fe002000640 1 -- 172.21.15.67:0/1512550320 wait complete. 2024-04-17T10:35:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 Processor -- start 2024-04-17T10:35:19.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 -- start start 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc071ed0 0x7fdffc14fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0722b0 0x7fdffc13b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc15e210 0x7fdffc1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc150a50 con 0x7fdffc071ed0 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc1508d0 con 0x7fdffc15e210 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdffc150bd0 con 0x7fdffc0722b0 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc071ed0 0x7fdffc14fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc071ed0 0x7fdffc14fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51716/0 (socket says 172.21.15.67:51716) 2024-04-17T10:35:19.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 -- 172.21.15.67:0/218558012 learned_addr learned my addr 172.21.15.67:0/218558012 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdff3e00640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0722b0 0x7fdffc13b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffbe00640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc15e210 0x7fdffc1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 -- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0722b0 msgr2=0x7fdffc13b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0722b0 0x7fdffc13b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 -- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc15e210 msgr2=0x7fdffc1594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc15e210 0x7fdffc1594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 -- 172.21.15.67:0/218558012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdffc14cf50 con 0x7fdffc071ed0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdffb400640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc071ed0 0x7fdffc14fee0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e8e0 tx=0x7fdfec00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdff3400640 1 -- 172.21.15.67:0/218558012 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec017070 con 0x7fdffc071ed0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdff3400640 1 -- 172.21.15.67:0/218558012 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfec00ca00 con 0x7fdffc071ed0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fdff3400640 1 -- 172.21.15.67:0/218558012 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00cdd0 con 0x7fdffc071ed0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdffc14d230 con 0x7fdffc071ed0 2024-04-17T10:35:19.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdffc159a30 con 0x7fdffc071ed0 2024-04-17T10:35:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.576+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdffc04ef30 con 0x7fdffc071ed0 2024-04-17T10:35:19.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.580+0000 7fdff3400640 1 -- 172.21.15.67:0/218558012 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdfec01a070 con 0x7fdffc071ed0 2024-04-17T10:35:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.580+0000 7fdff3400640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdfd8066f60 0x7fdfd8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.580+0000 7fdff3400640 1 -- 172.21.15.67:0/218558012 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdfec0856f0 con 0x7fdffc071ed0 2024-04-17T10:35:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.580+0000 7fdff3e00640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdfd8066f60 0x7fdfd8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.580+0000 7fdff3e00640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdfd8066f60 0x7fdfd8069420 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdffc1f8ca0 tx=0x7fdfe4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.584+0000 7fdff3400640 1 -- 172.21.15.67:0/218558012 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdfec056480 con 0x7fdffc071ed0 2024-04-17T10:35:19.736 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.744+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdffc13b880 con 0x7fdffc071ed0 2024-04-17T10:35:19.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.744+0000 7fdff3400640 1 -- 172.21.15.67:0/218558012 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdfec055d80 con 0x7fdffc071ed0 2024-04-17T10:35:19.737 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:35:19.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdfd8066f60 msgr2=0x7fdfd8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdfd8066f60 0x7fdfd8069420 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdffc1f8ca0 tx=0x7fdfe4008040 comp rx=0 tx=0).stop 2024-04-17T10:35:19.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc071ed0 msgr2=0x7fdffc14fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc071ed0 0x7fdffc14fee0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fdfec00e8e0 tx=0x7fdfec00eda0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 shutdown_connections 2024-04-17T10:35:19.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdfd8066f60 0x7fdfd8069420 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdffc15e210 0x7fdffc1594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdffc0722b0 0x7fdffc13b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 --2- 172.21.15.67:0/218558012 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdffc071ed0 0x7fdffc14fee0 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 >> 172.21.15.67:0/218558012 conn(0x7fdffc0b1740 msgr2=0x7fdffc06ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:19.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 shutdown_connections 2024-04-17T10:35:19.740 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.748+0000 7fe002000640 1 -- 172.21.15.67:0/218558012 wait complete. 2024-04-17T10:35:19.758 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:35:19.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.908+0000 7ff9fe200640 1 Processor -- start 2024-04-17T10:35:19.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.908+0000 7ff9fe200640 1 -- start start 2024-04-17T10:35:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9fe200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff9f80cd030 0x7ff9f806e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9fe200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f80b4990 con 0x7ff9f80aef60 2024-04-17T10:35:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9fe200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9f806ec00 con 0x7ff9f806e7b0 2024-04-17T10:35:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9fe200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9f80af340 con 0x7ff9f80cd030 2024-04-17T10:35:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff9f80aef60 0x7ff9f80cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41684/0 (socket says 172.21.15.67:41684) 2024-04-17T10:35:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f7400640 1 -- 172.21.15.67:0/2568961550 learned_addr learned my addr 172.21.15.67:0/2568961550 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:19.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9efe00640 1 --2- 172.21.15.67:0/2568961550 >> v2:172.21.15.86:3300/0 conn(0x7ff9f80cd030 0x7ff9f806e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747088146 0 0) 0x7ff9e8001120 con 0x7ff9f80aef60 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc003530 con 0x7ff9f80aef60 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4213364531 0 0) 0x7ff9dc003530 con 0x7ff9f80aef60 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 >> v2:172.21.15.86:3300/0 conn(0x7ff9f80cd030 msgr2=0x7ff9f806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 --2- 172.21.15.67:0/2568961550 >> v2:172.21.15.86:3300/0 conn(0x7ff9f80cd030 0x7ff9f806e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 >> v1:172.21.15.86:6789/0 conn(0x7ff9f806e7b0 legacy=0x7ff9f80b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f815ae90 con 0x7ff9f80aef60 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7ff9e8001850 con 0x7ff9f80aef60 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 >> v1:172.21.15.67:6789/0 conn(0x7ff9f80aef60 legacy=0x7ff9f80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9dc0039c0 0x7ff9dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9dc0051d0 0x7ff9dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80af340 con 0x7ff9dc004810 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc00c170 con 0x7ff9dc0051d0 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9dc00e580 con 0x7ff9dc0039c0 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9efe00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f7400640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9dc0039c0 0x7ff9dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f7e00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9dc0051d0 0x7ff9dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9efe00640 1 -- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9dc0039c0 msgr2=0x7ff9dc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9efe00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9dc0039c0 0x7ff9dc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9efe00640 1 -- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9dc0051d0 msgr2=0x7ff9dc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9efe00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9dc0051d0 0x7ff9dc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9f7e00640 1 -- 172.21.15.67:0/2568961550 reap_dead start 2024-04-17T10:35:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.912+0000 7ff9efe00640 1 -- 172.21.15.67:0/2568961550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc003530 con 0x7ff9dc004810 2024-04-17T10:35:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9efe00640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9dc004c90 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009ab0 tx=0x7ff9e001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1b3f1607293e49 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00b2070 con 0x7ff9dc004810 2024-04-17T10:35:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e00a5510 con 0x7ff9dc004810 2024-04-17T10:35:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9f6a00640 1 -- 172.21.15.67:0/2568961550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e00096b0 con 0x7ff9dc004810 2024-04-17T10:35:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 -- 172.21.15.67:0/2568961550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 msgr2=0x7ff9dc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9dc004c90 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7ff9e0009ab0 tx=0x7ff9e001ca50 comp rx=0 tx=0).stop 2024-04-17T10:35:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 -- 172.21.15.67:0/2568961550 shutdown_connections 2024-04-17T10:35:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 --2- 172.21.15.67:0/2568961550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9dc004c90 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 -- 172.21.15.67:0/2568961550 >> 172.21.15.67:0/2568961550 conn(0x7ff9f80b1740 msgr2=0x7ff9f80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 -- 172.21.15.67:0/2568961550 shutdown_connections 2024-04-17T10:35:19.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 -- 172.21.15.67:0/2568961550 wait complete. 2024-04-17T10:35:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 Processor -- start 2024-04-17T10:35:19.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.916+0000 7ff9fe200640 1 -- start start 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9f813d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9f80a9e20 0x7ff9f80aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f81e6fe0 0x7ff9f81e7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80cceb0 con 0x7ff9dc004810 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f80ccd30 con 0x7ff9f81e6fe0 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f815bc10 con 0x7ff9f80a9e20 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9f813d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9f813d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51736/0 (socket says 172.21.15.67:51736) 2024-04-17T10:35:19.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 -- 172.21.15.67:0/43112171 learned_addr learned my addr 172.21.15.67:0/43112171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9efe00640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9f80a9e20 0x7ff9f80aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7e00640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f81e6fe0 0x7ff9f81e7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 -- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9f80a9e20 msgr2=0x7ff9f80aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9f80a9e20 0x7ff9f80aa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 -- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f81e6fe0 msgr2=0x7ff9f81e7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f81e6fe0 0x7ff9f81e7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 -- 172.21.15.67:0/43112171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9f80ac040 con 0x7ff9dc004810 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9efe00640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9f80a9e20 0x7ff9f80aa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9f7400640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9f813d170 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8007330 tx=0x7ff9e80077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9ef400640 1 -- 172.21.15.67:0/43112171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e800e6e0 con 0x7ff9dc004810 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9ef400640 1 -- 172.21.15.67:0/43112171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9e8018070 con 0x7ff9dc004810 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f80ac320 con 0x7ff9dc004810 2024-04-17T10:35:19.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9ef400640 1 -- 172.21.15.67:0/43112171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e800eae0 con 0x7ff9dc004810 2024-04-17T10:35:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f80ac8e0 con 0x7ff9dc004810 2024-04-17T10:35:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9f804ef30 con 0x7ff9dc004810 2024-04-17T10:35:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.920+0000 7ff9ef400640 1 -- 172.21.15.67:0/43112171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff9e800ec80 con 0x7ff9dc004810 2024-04-17T10:35:19.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.924+0000 7ff9ef400640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9d4066f60 0x7ff9d4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.924+0000 7ff9ef400640 1 -- 172.21.15.67:0/43112171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7ff9e80852b0 con 0x7ff9dc004810 2024-04-17T10:35:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.924+0000 7ff9efe00640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9d4066f60 0x7ff9d4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:19.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.924+0000 7ff9efe00640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9d4066f60 0x7ff9d4069420 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00af000 tx=0x7ff9e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:19.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:19.928+0000 7ff9ef400640 1 -- 172.21.15.67:0/43112171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff9f80b4990 con 0x7ff9dc004810 2024-04-17T10:35:20.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff9f8150270 con 0x7ff9dc004810 2024-04-17T10:35:20.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9ef400640 1 -- 172.21.15.67:0/43112171 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff9e8056040 con 0x7ff9dc004810 2024-04-17T10:35:20.080 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:35:20.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9d4066f60 msgr2=0x7ff9d4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9d4066f60 0x7ff9d4069420 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7ff9e00af000 tx=0x7ff9e0008040 comp rx=0 tx=0).stop 2024-04-17T10:35:20.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 msgr2=0x7ff9f813d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9f813d170 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7ff9e8007330 tx=0x7ff9e80077f0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 shutdown_connections 2024-04-17T10:35:20.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff9d4066f60 0x7ff9d4069420 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9f81e6fe0 0x7ff9f81e7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9f80a9e20 0x7ff9f80aa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 --2- 172.21.15.67:0/43112171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9dc004810 0x7ff9f813d170 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 >> 172.21.15.67:0/43112171 conn(0x7ff9f80b1740 msgr2=0x7ff9f8150a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 shutdown_connections 2024-04-17T10:35:20.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.088+0000 7ff9fe200640 1 -- 172.21.15.67:0/43112171 wait complete. 2024-04-17T10:35:20.102 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:35:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.252+0000 7f9167200640 1 Processor -- start 2024-04-17T10:35:20.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.252+0000 7f9167200640 1 -- start start 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.252+0000 7f9167200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9160071310 0x7f9160071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.252+0000 7f9167200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916006e2f0 con 0x7f91600cc300 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.252+0000 7f9167200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f916006d6e0 con 0x7f9160145040 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.252+0000 7f9167200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91600a86c0 con 0x7f9160071310 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.252+0000 7f9165200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f91600cc300 0x7f91600cc750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41692/0 (socket says 172.21.15.67:41692) 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f9165200640 1 -- 172.21.15.67:0/3524944191 learned_addr learned my addr 172.21.15.67:0/3524944191 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f9157400640 1 --2- 172.21.15.67:0/3524944191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9160071310 0x7f9160071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f915f400640 1 -- 172.21.15.67:0/3524944191 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810453298 0 0) 0x7f916006e2f0 con 0x7f91600cc300 2024-04-17T10:35:20.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f915f400640 1 -- 172.21.15.67:0/3524944191 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9144003530 con 0x7f91600cc300 2024-04-17T10:35:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f915f400640 1 -- 172.21.15.67:0/3524944191 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109380209 0 0) 0x7f916006d6e0 con 0x7f9160145040 2024-04-17T10:35:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f915f400640 1 -- 172.21.15.67:0/3524944191 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916006e2f0 con 0x7f9160145040 2024-04-17T10:35:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f9157400640 1 -- 172.21.15.67:0/3524944191 >> v1:172.21.15.86:6789/0 conn(0x7f9160145040 legacy=0x7f916006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f9157400640 1 -- 172.21.15.67:0/3524944191 >> v1:172.21.15.67:6789/0 conn(0x7f91600cc300 legacy=0x7f91600cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f9157400640 1 -- 172.21.15.67:0/3524944191 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916006d9c0 con 0x7f9160071310 2024-04-17T10:35:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f9157400640 1 --2- 172.21.15.67:0/3524944191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9160071310 0x7f9160071770 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91480097f0 tx=0x7f914801ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a4218be09c61629 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.256+0000 7f915f400640 1 -- 172.21.15.67:0/3524944191 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91480b5070 con 0x7f9160071310 2024-04-17T10:35:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f915f400640 1 -- 172.21.15.67:0/3524944191 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91480a89d0 con 0x7f9160071310 2024-04-17T10:35:20.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f915f400640 1 -- 172.21.15.67:0/3524944191 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91480a8da0 con 0x7f9160071310 2024-04-17T10:35:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- 172.21.15.67:0/3524944191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9160071310 msgr2=0x7f9160071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 --2- 172.21.15.67:0/3524944191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9160071310 0x7f9160071770 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91480097f0 tx=0x7f914801ecd0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- 172.21.15.67:0/3524944191 shutdown_connections 2024-04-17T10:35:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 --2- 172.21.15.67:0/3524944191 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9160071310 0x7f9160071770 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- 172.21.15.67:0/3524944191 >> 172.21.15.67:0/3524944191 conn(0x7f916006ac90 msgr2=0x7f916006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- 172.21.15.67:0/3524944191 shutdown_connections 2024-04-17T10:35:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- 172.21.15.67:0/3524944191 wait complete. 2024-04-17T10:35:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 Processor -- start 2024-04-17T10:35:20.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- start start 2024-04-17T10:35:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9160071310 0x7f916015b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91600cc300 0x7f91601e5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9160145040 0x7f91600b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916015bf10 con 0x7f9160145040 2024-04-17T10:35:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916006df80 con 0x7f9160071310 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9167200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916006d370 con 0x7f91600cc300 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9165200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9160145040 0x7f91600b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9165200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9160145040 0x7f91600b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51746/0 (socket says 172.21.15.67:51746) 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.260+0000 7f9165200640 1 -- 172.21.15.67:0/156427126 learned_addr learned my addr 172.21.15.67:0/156427126 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915fe00640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9160071310 0x7f916015b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9157400640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91600cc300 0x7f91601e5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9165200640 1 -- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91600cc300 msgr2=0x7f91601e5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9165200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91600cc300 0x7f91601e5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9165200640 1 -- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9160071310 msgr2=0x7f916015b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9165200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9160071310 0x7f916015b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9165200640 1 -- 172.21.15.67:0/156427126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91600b4830 con 0x7f9160145040 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915fe00640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9160071310 0x7f916015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9165200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9160145040 0x7f91600b0140 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f915800cba0 tx=0x7f915800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915cc00640 1 -- 172.21.15.67:0/156427126 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158017070 con 0x7f9160145040 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915cc00640 1 -- 172.21.15.67:0/156427126 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915800ace0 con 0x7f9160145040 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915cc00640 1 -- 172.21.15.67:0/156427126 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158012620 con 0x7f9160145040 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91600b3f90 con 0x7f9160145040 2024-04-17T10:35:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91600b4550 con 0x7f9160145040 2024-04-17T10:35:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9160072d50 con 0x7f9160145040 2024-04-17T10:35:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915cc00640 1 -- 172.21.15.67:0/156427126 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f915801a070 con 0x7f9160145040 2024-04-17T10:35:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915cc00640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f913c066f90 0x7f913c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.264+0000 7f915cc00640 1 -- 172.21.15.67:0/156427126 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f9158084fc0 con 0x7f9160145040 2024-04-17T10:35:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.268+0000 7f915fe00640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f913c066f90 0x7f913c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.268+0000 7f915fe00640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f913c066f90 0x7f913c069450 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f91500011f0 tx=0x7f9150009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.268+0000 7f915cc00640 1 -- 172.21.15.67:0/156427126 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9158055700 con 0x7f9160145040 2024-04-17T10:35:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.444+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f91600b2da0 con 0x7f9160145040 2024-04-17T10:35:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.444+0000 7f915cc00640 1 -- 172.21.15.67:0/156427126 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v131) v1 ==== 80+0+120 (secure 0 0 0) 0x7f91580597a0 con 0x7f9160145040 2024-04-17T10:35:20.437 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.86:6836/538421223 2024-04-18T10:35:13.952580+0000 2024-04-17T10:35:20.437 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.86:6834/538421223 2024-04-18T10:35:13.952580+0000 2024-04-17T10:35:20.437 INFO:teuthology.orchestra.run.smithi067.stderr:listed 2 entries 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f913c066f90 msgr2=0x7f913c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f913c066f90 0x7f913c069450 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f91500011f0 tx=0x7f9150009040 comp rx=0 tx=0).stop 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9160145040 msgr2=0x7f91600b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9160145040 0x7f91600b0140 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f915800cba0 tx=0x7f915800a430 comp rx=0 tx=0).stop 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 shutdown_connections 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f913c066f90 0x7f913c069450 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9160145040 0x7f91600b0140 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f91600cc300 0x7f91601e5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 --2- 172.21.15.67:0/156427126 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9160071310 0x7f916015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 >> 172.21.15.67:0/156427126 conn(0x7f916006ac90 msgr2=0x7f916015f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:20.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 shutdown_connections 2024-04-17T10:35:20.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.448+0000 7f9167200640 1 -- 172.21.15.67:0/156427126 wait complete. 2024-04-17T10:35:20.458 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:35:20.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:20.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:35:20.486 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-04-17T10:35:20.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:20.486 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-17T10:35:20.539 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:35:20.539 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:20.549 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:35:20.549 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:35:20.549 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:20.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:35:20.591 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:35:20.591 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:20.600 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:35:20.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.780+0000 7fa464c00640 1 Processor -- start 2024-04-17T10:35:20.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.780+0000 7fa464c00640 1 -- start start 2024-04-17T10:35:20.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa464c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa46015dd10 0x7fa4600b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa464c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4600b57b0 0x7fa460070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa464c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4600b07b0 0x7fa460071300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa464c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4600b1280 con 0x7fa4600b07b0 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa464c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4600b0cc0 con 0x7fa46015dd10 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa464c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4600b0fa0 con 0x7fa4600b57b0 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4600b07b0 0x7fa460071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa4600b07b0 0x7fa460071300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56166/0 (socket says 172.21.15.67:56166) 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 -- 172.21.15.67:0/936325381 learned_addr learned my addr 172.21.15.67:0/936325381 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45de00640 1 --2- 172.21.15.67:0/936325381 >> v2:172.21.15.86:3300/0 conn(0x7fa46015dd10 0x7fa4600b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 -- 172.21.15.67:0/936325381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4600b57b0 msgr2=0x7fa460070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa457e00640 1 --2- 172.21.15.67:0/936325381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4600b57b0 0x7fa460070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 --2- 172.21.15.67:0/936325381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4600b57b0 0x7fa460070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 -- 172.21.15.67:0/936325381 >> v2:172.21.15.86:3300/0 conn(0x7fa46015dd10 msgr2=0x7fa4600b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 --2- 172.21.15.67:0/936325381 >> v2:172.21.15.86:3300/0 conn(0x7fa46015dd10 0x7fa4600b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45e800640 1 -- 172.21.15.67:0/936325381 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46006bed0 con 0x7fa4600b07b0 2024-04-17T10:35:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa45de00640 1 --2- 172.21.15.67:0/936325381 >> v2:172.21.15.86:3300/0 conn(0x7fa46015dd10 0x7fa4600b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:20.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.784+0000 7fa457e00640 1 --2- 172.21.15.67:0/936325381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4600b57b0 0x7fa460070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa45e800640 1 --2- 172.21.15.67:0/936325381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4600b07b0 0x7fa460071300 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa4600d2900 tx=0x7fa45801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754b56d1f5364f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa45d400640 1 -- 172.21.15.67:0/936325381 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4580a5790 con 0x7fa4600b07b0 2024-04-17T10:35:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa45d400640 1 -- 172.21.15.67:0/936325381 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4580a5930 con 0x7fa4600b07b0 2024-04-17T10:35:20.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa45d400640 1 -- 172.21.15.67:0/936325381 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4580a5d00 con 0x7fa4600b07b0 2024-04-17T10:35:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 -- 172.21.15.67:0/936325381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4600b07b0 msgr2=0x7fa460071300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 --2- 172.21.15.67:0/936325381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4600b07b0 0x7fa460071300 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa4600d2900 tx=0x7fa45801c930 comp rx=0 tx=0).stop 2024-04-17T10:35:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 -- 172.21.15.67:0/936325381 shutdown_connections 2024-04-17T10:35:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 --2- 172.21.15.67:0/936325381 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4600b07b0 0x7fa460071300 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 --2- 172.21.15.67:0/936325381 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa4600b57b0 0x7fa460070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 --2- 172.21.15.67:0/936325381 >> v2:172.21.15.86:3300/0 conn(0x7fa46015dd10 0x7fa4600b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 -- 172.21.15.67:0/936325381 >> 172.21.15.67:0/936325381 conn(0x7fa4600b34d0 msgr2=0x7fa4600b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 -- 172.21.15.67:0/936325381 shutdown_connections 2024-04-17T10:35:20.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 -- 172.21.15.67:0/936325381 wait complete. 2024-04-17T10:35:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 Processor -- start 2024-04-17T10:35:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.788+0000 7fa464c00640 1 -- start start 2024-04-17T10:35:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4600b07b0 0x7fa46015e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46015dd10 0x7fa46015e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa460148040 0x7fa460136180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa460162460 con 0x7fa46015dd10 2024-04-17T10:35:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4601622e0 con 0x7fa4600b07b0 2024-04-17T10:35:20.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4601625e0 con 0x7fa460148040 2024-04-17T10:35:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46015dd10 0x7fa46015e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa45de00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4600b07b0 0x7fa46015e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46015dd10 0x7fa46015e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56172/0 (socket says 172.21.15.67:56172) 2024-04-17T10:35:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 -- 172.21.15.67:0/1409308286 learned_addr learned my addr 172.21.15.67:0/1409308286 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:20.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa45e800640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa460148040 0x7fa460136180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 -- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa460148040 msgr2=0x7fa460136180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa460148040 0x7fa460136180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 -- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4600b07b0 msgr2=0x7fa46015e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4600b07b0 0x7fa46015e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 -- 172.21.15.67:0/1409308286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4601485b0 con 0x7fa46015dd10 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa457e00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46015dd10 0x7fa46015e880 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fa44800c7f0 tx=0x7fa44800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa456000640 1 -- 172.21.15.67:0/1409308286 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448015070 con 0x7fa46015dd10 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa456000640 1 -- 172.21.15.67:0/1409308286 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa448007a30 con 0x7fa46015dd10 2024-04-17T10:35:20.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa456000640 1 -- 172.21.15.67:0/1409308286 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448007e00 con 0x7fa46015dd10 2024-04-17T10:35:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa460148890 con 0x7fa46015dd10 2024-04-17T10:35:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa460148e50 con 0x7fa46015dd10 2024-04-17T10:35:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.792+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa460070bb0 con 0x7fa46015dd10 2024-04-17T10:35:20.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.796+0000 7fa456000640 1 -- 172.21.15.67:0/1409308286 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa448018070 con 0x7fa46015dd10 2024-04-17T10:35:20.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.796+0000 7fa456000640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa43c066f90 0x7fa43c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:20.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.796+0000 7fa456000640 1 -- 172.21.15.67:0/1409308286 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa4480850f0 con 0x7fa46015dd10 2024-04-17T10:35:20.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.796+0000 7fa45de00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa43c066f90 0x7fa43c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:20.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.796+0000 7fa45de00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa43c066f90 0x7fa43c069450 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:20.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.800+0000 7fa456000640 1 -- 172.21.15.67:0/1409308286 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa448055830 con 0x7fa46015dd10 2024-04-17T10:35:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa460074e30 con 0x7fa46015dd10 2024-04-17T10:35:20.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa456000640 1 -- 172.21.15.67:0/1409308286 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4146 (secure 0 0 0) 0x7fa4480598d0 con 0x7fa46015dd10 2024-04-17T10:35:20.987 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:20.987 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:35:15.983015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9255":{"gid":9255,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:20.988 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T10:35:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa43c066f90 msgr2=0x7fa43c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa43c066f90 0x7fa43c069450 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa450006fd0 tx=0x7fa450008040 comp rx=0 tx=0).stop 2024-04-17T10:35:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46015dd10 msgr2=0x7fa46015e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:20.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46015dd10 0x7fa46015e880 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fa44800c7f0 tx=0x7fa44800ccb0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 shutdown_connections 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa43c066f90 0x7fa43c069450 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa460148040 0x7fa460136180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa46015dd10 0x7fa46015e880 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 --2- 172.21.15.67:0/1409308286 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa4600b07b0 0x7fa46015e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 >> 172.21.15.67:0/1409308286 conn(0x7fa4600b34d0 msgr2=0x7fa460156cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 shutdown_connections 2024-04-17T10:35:20.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:20.996+0000 7fa464c00640 1 -- 172.21.15.67:0/1409308286 wait complete. 2024-04-17T10:35:21.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:21.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ae1000640 1 Processor -- start 2024-04-17T10:35:21.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ae1000640 1 -- start start 2024-04-17T10:35:21.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ae1000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc1620a0 0x7f3adc15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ae1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3adc15e550 0x7f3adc0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ae1000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3adc15eb50 con 0x7f3adc0ce020 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ae1000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3adc161650 con 0x7f3adc15e550 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ae1000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc15ee30 con 0x7f3adc1620a0 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ada800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3adc15e550 0x7f3adc0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ada800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3adc15e550 0x7f3adc0cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56178/0 (socket says 172.21.15.67:56178) 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.156+0000 7f3ada800640 1 -- 172.21.15.67:0/3034538803 learned_addr learned my addr 172.21.15.67:0/3034538803 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ad3e00640 1 --2- 172.21.15.67:0/3034538803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc1620a0 0x7f3adc15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ada800640 1 -- 172.21.15.67:0/3034538803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc1620a0 msgr2=0x7f3adc15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ada800640 1 --2- 172.21.15.67:0/3034538803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc1620a0 0x7f3adc15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ada800640 1 -- 172.21.15.67:0/3034538803 >> v1:172.21.15.86:6789/0 conn(0x7f3adc0ce020 legacy=0x7f3adc0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ada800640 1 -- 172.21.15.67:0/3034538803 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc1618a0 con 0x7f3adc15e550 2024-04-17T10:35:21.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ada800640 1 --2- 172.21.15.67:0/3034538803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc15e550 0x7f3adc0cf510 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009e50 tx=0x7f3ad401ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d00b8c640c56978 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ad9400640 1 -- 172.21.15.67:0/3034538803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40a57e0 con 0x7f3adc15e550 2024-04-17T10:35:21.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ad9400640 1 -- 172.21.15.67:0/3034538803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad40a5980 con 0x7f3adc15e550 2024-04-17T10:35:21.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ad9400640 1 -- 172.21.15.67:0/3034538803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40a5d50 con 0x7f3adc15e550 2024-04-17T10:35:21.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ae1000640 1 -- 172.21.15.67:0/3034538803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc15e550 msgr2=0x7f3adc0cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.160+0000 7f3ae1000640 1 --2- 172.21.15.67:0/3034538803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc15e550 0x7f3adc0cf510 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009e50 tx=0x7f3ad401ca00 comp rx=0 tx=0).stop 2024-04-17T10:35:21.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- 172.21.15.67:0/3034538803 shutdown_connections 2024-04-17T10:35:21.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 --2- 172.21.15.67:0/3034538803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc15e550 0x7f3adc0cf510 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 --2- 172.21.15.67:0/3034538803 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc1620a0 0x7f3adc15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- 172.21.15.67:0/3034538803 >> 172.21.15.67:0/3034538803 conn(0x7f3adc0b33f0 msgr2=0x7f3adc0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:21.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- 172.21.15.67:0/3034538803 shutdown_connections 2024-04-17T10:35:21.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- 172.21.15.67:0/3034538803 wait complete. 2024-04-17T10:35:21.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 Processor -- start 2024-04-17T10:35:21.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- start start 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc0ce020 0x7f3adc1574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3adc15e550 0x7f3adc150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1620a0 0x7f3adc151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc0c8630 con 0x7f3adc1620a0 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc0c84b0 con 0x7f3adc15e550 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ae1000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3adc16a9f0 con 0x7f3adc0ce020 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1620a0 0x7f3adc151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1620a0 0x7f3adc151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56182/0 (socket says 172.21.15.67:56182) 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 -- 172.21.15.67:0/1476028861 learned_addr learned my addr 172.21.15.67:0/1476028861 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ad9e00640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc0ce020 0x7f3adc1574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ad3e00640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3adc15e550 0x7f3adc150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 -- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc0ce020 msgr2=0x7f3adc1574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc0ce020 0x7f3adc1574f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 -- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3adc15e550 msgr2=0x7f3adc150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3adc15e550 0x7f3adc150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 -- 172.21.15.67:0/1476028861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3adc157040 con 0x7f3adc1620a0 2024-04-17T10:35:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.164+0000 7f3ada800640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1620a0 0x7f3adc151810 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40af040 tx=0x7f3ad40197a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ad2000640 1 -- 172.21.15.67:0/1476028861 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4009150 con 0x7f3adc1620a0 2024-04-17T10:35:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ad2000640 1 -- 172.21.15.67:0/1476028861 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad40b8470 con 0x7f3adc1620a0 2024-04-17T10:35:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ad2000640 1 -- 172.21.15.67:0/1476028861 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad40b8860 con 0x7f3adc1620a0 2024-04-17T10:35:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3adc150a50 con 0x7f3adc1620a0 2024-04-17T10:35:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3adc151010 con 0x7f3adc1620a0 2024-04-17T10:35:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc070ab0 con 0x7f3adc1620a0 2024-04-17T10:35:21.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ad2000640 1 -- 172.21.15.67:0/1476028861 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3ad40c0030 con 0x7f3adc1620a0 2024-04-17T10:35:21.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ad2000640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ab8066f60 0x7f3ab8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ad2000640 1 -- 172.21.15.67:0/1476028861 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3ad411fc50 con 0x7f3adc1620a0 2024-04-17T10:35:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.168+0000 7f3ad9e00640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ab8066f60 0x7f3ab8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.172+0000 7f3ad9e00640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ab8066f60 0x7f3ab8069420 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000f70 tx=0x7f3acc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.172+0000 7f3ad2000640 1 -- 172.21.15.67:0/1476028861 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3ad40c7020 con 0x7f3adc1620a0 2024-04-17T10:35:21.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.340+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3adc074e30 con 0x7f3adc1620a0 2024-04-17T10:35:21.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.340+0000 7f3ad2000640 1 -- 172.21.15.67:0/1476028861 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16979 (secure 0 0 0) 0x7f3ad4016620 con 0x7f3adc1620a0 2024-04-17T10:35:21.333 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:21.333 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":131,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:13.952681+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:34:26.120176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-17T10:34:27.343196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6836/538421223":"2024-04-18T10:35:13.952580+0000","172.21.15.86:6834/538421223":"2024-04-18T10:35:13.952580+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:21.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ab8066f60 msgr2=0x7f3ab8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ab8066f60 0x7f3ab8069420 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000f70 tx=0x7f3acc009040 comp rx=0 tx=0).stop 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1620a0 msgr2=0x7f3adc151810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1620a0 0x7f3adc151810 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f3ad40af040 tx=0x7f3ad40197a0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 shutdown_connections 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ab8066f60 0x7f3ab8069420 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3adc1620a0 0x7f3adc151810 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3adc15e550 0x7f3adc150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 --2- 172.21.15.67:0/1476028861 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3adc0ce020 0x7f3adc1574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 >> 172.21.15.67:0/1476028861 conn(0x7f3adc0b33f0 msgr2=0x7f3adc073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 shutdown_connections 2024-04-17T10:35:21.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.344+0000 7f3ae1000640 1 -- 172.21.15.67:0/1476028861 wait complete. 2024-04-17T10:35:21.355 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:34:26.120176+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:34:27.343196+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:35:21.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:35:21.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f45600640 1 Processor -- start 2024-04-17T10:35:21.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f45600640 1 -- start start 2024-04-17T10:35:21.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f45600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f40071af0 0x7f6f400ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f45600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f4015be70 0x7f6f4015c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f45600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4015cb30 con 0x7f6f40159440 2024-04-17T10:35:21.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f45600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f400a8650 con 0x7f6f40071af0 2024-04-17T10:35:21.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f45600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4015ce10 con 0x7f6f4015be70 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f40071af0 0x7f6f400ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f40071af0 0x7f6f400ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56186/0 (socket says 172.21.15.67:56186) 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3e000640 1 -- 172.21.15.67:0/136887178 learned_addr learned my addr 172.21.15.67:0/136887178 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3f400640 1 --2- 172.21.15.67:0/136887178 >> v2:172.21.15.86:3300/0 conn(0x7f6f4015be70 0x7f6f4015c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3e000640 1 -- 172.21.15.67:0/136887178 >> v2:172.21.15.86:3300/0 conn(0x7f6f4015be70 msgr2=0x7f6f4015c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3e000640 1 --2- 172.21.15.67:0/136887178 >> v2:172.21.15.86:3300/0 conn(0x7f6f4015be70 0x7f6f4015c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3e000640 1 -- 172.21.15.67:0/136887178 >> v1:172.21.15.86:6789/0 conn(0x7f6f40159440 legacy=0x7f6f400713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.508+0000 7f6f3e000640 1 -- 172.21.15.67:0/136887178 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f400cd610 con 0x7f6f40071af0 2024-04-17T10:35:21.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f3e000640 1 --2- 172.21.15.67:0/136887178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40071af0 0x7f6f400ce0f0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f6f400ce690 tx=0x7f6f2c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8aadc49102076d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f3d600640 1 -- 172.21.15.67:0/136887178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0aa680 con 0x7f6f40071af0 2024-04-17T10:35:21.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f3d600640 1 -- 172.21.15.67:0/136887178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c0af070 con 0x7f6f40071af0 2024-04-17T10:35:21.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f3d600640 1 -- 172.21.15.67:0/136887178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0aaa40 con 0x7f6f40071af0 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 -- 172.21.15.67:0/136887178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40071af0 msgr2=0x7f6f400ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 --2- 172.21.15.67:0/136887178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40071af0 0x7f6f400ce0f0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f6f400ce690 tx=0x7f6f2c01c520 comp rx=0 tx=0).stop 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 -- 172.21.15.67:0/136887178 shutdown_connections 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 --2- 172.21.15.67:0/136887178 >> v2:172.21.15.86:3300/0 conn(0x7f6f4015be70 0x7f6f4015c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 --2- 172.21.15.67:0/136887178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40071af0 0x7f6f400ce0f0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 -- 172.21.15.67:0/136887178 >> 172.21.15.67:0/136887178 conn(0x7f6f400b1740 msgr2=0x7f6f400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 -- 172.21.15.67:0/136887178 shutdown_connections 2024-04-17T10:35:21.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 -- 172.21.15.67:0/136887178 wait complete. 2024-04-17T10:35:21.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.512+0000 7f6f45600640 1 Processor -- start 2024-04-17T10:35:21.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 -- start start 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40071af0 0x7f6f4013e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40159440 0x7f6f4014ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f4015be70 0x7f6f40159e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4015d840 con 0x7f6f40159440 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f401598e0 con 0x7f6f4015be70 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4015d9c0 con 0x7f6f40071af0 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40159440 0x7f6f4014ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40159440 0x7f6f4014ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56202/0 (socket says 172.21.15.67:56202) 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 -- 172.21.15.67:0/3561782780 learned_addr learned my addr 172.21.15.67:0/3561782780 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3ea00640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40071af0 0x7f6f4013e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3f400640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f4015be70 0x7f6f40159e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 -- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40071af0 msgr2=0x7f6f4013e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40071af0 0x7f6f4013e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 -- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f4015be70 msgr2=0x7f6f40159e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f4015be70 0x7f6f40159e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 -- 172.21.15.67:0/3561782780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f401534e0 con 0x7f6f40159440 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3f400640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f4015be70 0x7f6f40159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f3e000640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40159440 0x7f6f4014ccb0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c01a150 tx=0x7f6f2c01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f36a00640 1 -- 172.21.15.67:0/3561782780 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0bd5f0 con 0x7f6f40159440 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f36a00640 1 -- 172.21.15.67:0/3561782780 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c0bd790 con 0x7f6f40159440 2024-04-17T10:35:21.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f36a00640 1 -- 172.21.15.67:0/3561782780 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c0bdb60 con 0x7f6f40159440 2024-04-17T10:35:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f401537c0 con 0x7f6f40159440 2024-04-17T10:35:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f40157d90 con 0x7f6f40159440 2024-04-17T10:35:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.516+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4004ef30 con 0x7f6f40159440 2024-04-17T10:35:21.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.520+0000 7f6f36a00640 1 -- 172.21.15.67:0/3561782780 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6f2c0a8050 con 0x7f6f40159440 2024-04-17T10:35:21.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.520+0000 7f6f36a00640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f20066f60 0x7f6f20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.520+0000 7f6f36a00640 1 -- 172.21.15.67:0/3561782780 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f6f2c11f970 con 0x7f6f40159440 2024-04-17T10:35:21.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.520+0000 7f6f3ea00640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f20066f60 0x7f6f20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.520+0000 7f6f3ea00640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f20066f60 0x7f6f20069420 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6f30001650 tx=0x7f6f30009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.524+0000 7f6f36a00640 1 -- 172.21.15.67:0/3561782780 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6f2c0c3060 con 0x7f6f40159440 2024-04-17T10:35:21.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.684+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f401525e0 con 0x7f6f40159440 2024-04-17T10:35:21.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f36a00640 1 -- 172.21.15.67:0/3561782780 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f2c0a6040 con 0x7f6f40159440 2024-04-17T10:35:21.679 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f20066f60 msgr2=0x7f6f20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f20066f60 0x7f6f20069420 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6f30001650 tx=0x7f6f30009040 comp rx=0 tx=0).stop 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40159440 msgr2=0x7f6f4014ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40159440 0x7f6f4014ccb0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c01a150 tx=0x7f6f2c01ad60 comp rx=0 tx=0).stop 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 shutdown_connections 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6f20066f60 0x7f6f20069420 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f4015be70 0x7f6f40159e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f40159440 0x7f6f4014ccb0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 --2- 172.21.15.67:0/3561782780 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f40071af0 0x7f6f4013e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 >> 172.21.15.67:0/3561782780 conn(0x7f6f400b1740 msgr2=0x7f6f40152b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:21.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 shutdown_connections 2024-04-17T10:35:21.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.688+0000 7f6f45600640 1 -- 172.21.15.67:0/3561782780 wait complete. 2024-04-17T10:35:21.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:35:21.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f45e00640 1 Processor -- start 2024-04-17T10:35:21.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f45e00640 1 -- start start 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f45e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400cd030 0x7f2f4006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f45e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f4006e7b0 0x7f2f400b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f45e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400b4990 con 0x7f2f400aef60 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f45e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f400af340 con 0x7f2f4006e7b0 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f45e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4015ae90 con 0x7f2f400cd030 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f3f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f4006e7b0 0x7f2f400b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f3f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2f4006e7b0 0x7f2f400b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56218/0 (socket says 172.21.15.67:56218) 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f3f400640 1 -- 172.21.15.67:0/2040390850 learned_addr learned my addr 172.21.15.67:0/2040390850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:21.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f36000640 1 --2- 172.21.15.67:0/2040390850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400cd030 0x7f2f4006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f3f400640 1 -- 172.21.15.67:0/2040390850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400cd030 msgr2=0x7f2f4006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:21.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f3f400640 1 --2- 172.21.15.67:0/2040390850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400cd030 0x7f2f4006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f3f400640 1 -- 172.21.15.67:0/2040390850 >> v1:172.21.15.86:6789/0 conn(0x7f2f400aef60 legacy=0x7f2f400cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.852+0000 7f2f3f400640 1 -- 172.21.15.67:0/2040390850 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4006ee80 con 0x7f2f4006e7b0 2024-04-17T10:35:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f3f400640 1 --2- 172.21.15.67:0/2040390850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006e7b0 0x7f2f400b3120 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800d150 tx=0x7f2f3801eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2e2b24436ee6ef server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f3e000640 1 -- 172.21.15.67:0/2040390850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800b2d0 con 0x7f2f4006e7b0 2024-04-17T10:35:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f3e000640 1 -- 172.21.15.67:0/2040390850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3800b470 con 0x7f2f4006e7b0 2024-04-17T10:35:21.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f3e000640 1 -- 172.21.15.67:0/2040390850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f380a7b80 con 0x7f2f4006e7b0 2024-04-17T10:35:21.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 -- 172.21.15.67:0/2040390850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006e7b0 msgr2=0x7f2f400b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2040390850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006e7b0 0x7f2f400b3120 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800d150 tx=0x7f2f3801eba0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 -- 172.21.15.67:0/2040390850 shutdown_connections 2024-04-17T10:35:21.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2040390850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f4006e7b0 0x7f2f400b3120 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2040390850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400cd030 0x7f2f4006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 -- 172.21.15.67:0/2040390850 >> 172.21.15.67:0/2040390850 conn(0x7f2f400b1740 msgr2=0x7f2f400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:21.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 -- 172.21.15.67:0/2040390850 shutdown_connections 2024-04-17T10:35:21.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.856+0000 7f2f45e00640 1 -- 172.21.15.67:0/2040390850 wait complete. 2024-04-17T10:35:21.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 Processor -- start 2024-04-17T10:35:21.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 -- start start 2024-04-17T10:35:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f4006e7b0 0x7f2f4013d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400aef60 0x7f2f40139d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f400cd030 0x7f2f4015bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4015bc10 con 0x7f2f400cd030 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400b3e70 con 0x7f2f4006e7b0 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4015d040 con 0x7f2f400aef60 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f400cd030 0x7f2f4015bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f400cd030 0x7f2f4015bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56234/0 (socket says 172.21.15.67:56234) 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 -- 172.21.15.67:0/2757300035 learned_addr learned my addr 172.21.15.67:0/2757300035 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f36000640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400aef60 0x7f2f40139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3ea00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f4006e7b0 0x7f2f4013d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 -- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400aef60 msgr2=0x7f2f40139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400aef60 0x7f2f40139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 -- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f4006e7b0 msgr2=0x7f2f4013d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f4006e7b0 0x7f2f4013d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:21.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 -- 172.21.15.67:0/2757300035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4015c4f0 con 0x7f2f400cd030 2024-04-17T10:35:21.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f3f400640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f400cd030 0x7f2f4015bfb0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f2f380b1040 tx=0x7f2f3801b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f37e00640 1 -- 172.21.15.67:0/2757300035 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800b720 con 0x7f2f400cd030 2024-04-17T10:35:21.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f37e00640 1 -- 172.21.15.67:0/2757300035 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f380be070 con 0x7f2f400cd030 2024-04-17T10:35:21.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f4015c7a0 con 0x7f2f400cd030 2024-04-17T10:35:21.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f37e00640 1 -- 172.21.15.67:0/2757300035 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f38009c90 con 0x7f2f400cd030 2024-04-17T10:35:21.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4015cd60 con 0x7f2f400cd030 2024-04-17T10:35:21.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.860+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f4004ef30 con 0x7f2f400cd030 2024-04-17T10:35:21.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.864+0000 7f2f37e00640 1 -- 172.21.15.67:0/2757300035 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2f380b6070 con 0x7f2f400cd030 2024-04-17T10:35:21.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.864+0000 7f2f37e00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f1c066f60 0x7f2f1c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:21.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.864+0000 7f2f37e00640 1 -- 172.21.15.67:0/2757300035 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f2f38120250 con 0x7f2f400cd030 2024-04-17T10:35:21.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.864+0000 7f2f3ea00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f1c066f60 0x7f2f1c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:21.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.864+0000 7f2f3ea00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f1c066f60 0x7f2f1c069420 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2f30001810 tx=0x7f2f30009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:21.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:21.868+0000 7f2f37e00640 1 -- 172.21.15.67:0/2757300035 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2f38018820 con 0x7f2f400cd030 2024-04-17T10:35:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.028+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f400cedc0 con 0x7f2f400cd030 2024-04-17T10:35:22.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.028+0000 7f2f37e00640 1 -- 172.21.15.67:0/2757300035 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v131) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f380f0ae0 con 0x7f2f400cd030 2024-04-17T10:35:22.022 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:35:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f1c066f60 msgr2=0x7f2f1c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f1c066f60 0x7f2f1c069420 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2f30001810 tx=0x7f2f30009040 comp rx=0 tx=0).stop 2024-04-17T10:35:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f400cd030 msgr2=0x7f2f4015bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f400cd030 0x7f2f4015bfb0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f2f380b1040 tx=0x7f2f3801b940 comp rx=0 tx=0).stop 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 shutdown_connections 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2f1c066f60 0x7f2f1c069420 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f400cd030 0x7f2f4015bfb0 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f400aef60 0x7f2f40139d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 --2- 172.21.15.67:0/2757300035 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2f4006e7b0 0x7f2f4013d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 >> 172.21.15.67:0/2757300035 conn(0x7f2f400b1740 msgr2=0x7f2f400b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 shutdown_connections 2024-04-17T10:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.032+0000 7f2f45e00640 1 -- 172.21.15.67:0/2757300035 wait complete. 2024-04-17T10:35:22.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:35:22.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.192+0000 7fb6ad000640 1 Processor -- start 2024-04-17T10:35:22.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6ad000640 1 -- start start 2024-04-17T10:35:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6ad000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6ad000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6a80b2a30 0x7fb6a8099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6ad000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a809a370 con 0x7fb6a80cf200 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6ad000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a80cd9c0 con 0x7fb6a80b2a30 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6ad000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a80cc3e0 con 0x7fb6a8099a50 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb69fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6a80b2a30 0x7fb6a8099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6a7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb6a80cf200 0x7fb6a80b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48782/0 (socket says 172.21.15.67:48782) 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6a7e00640 1 -- 172.21.15.67:0/699699060 learned_addr learned my addr 172.21.15.67:0/699699060 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6a7400640 1 --2- 172.21.15.67:0/699699060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6a6a00640 1 -- 172.21.15.67:0/699699060 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720249658 0 0) 0x7fb6a809a370 con 0x7fb6a80cf200 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb6a6a00640 1 -- 172.21.15.67:0/699699060 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68c003530 con 0x7fb6a80cf200 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb69fe00640 1 -- 172.21.15.67:0/699699060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 msgr2=0x7fb6a80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:22.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb69fe00640 1 --2- 172.21.15.67:0/699699060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb69fe00640 1 -- 172.21.15.67:0/699699060 >> v1:172.21.15.67:6789/0 conn(0x7fb6a80cf200 legacy=0x7fb6a80b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.196+0000 7fb69fe00640 1 -- 172.21.15.67:0/699699060 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a80cc6c0 con 0x7fb6a80b2a30 2024-04-17T10:35:22.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb69fe00640 1 --2- 172.21.15.67:0/699699060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a80b2a30 0x7fb6a8099e30 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fb6900097c0 tx=0x7fb69001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5feb18930b33cb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6a6a00640 1 -- 172.21.15.67:0/699699060 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6900a7070 con 0x7fb6a80b2a30 2024-04-17T10:35:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6a6a00640 1 -- 172.21.15.67:0/699699060 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6900b0070 con 0x7fb6a80b2a30 2024-04-17T10:35:22.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6a6a00640 1 -- 172.21.15.67:0/699699060 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6900ab890 con 0x7fb6a80b2a30 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 -- 172.21.15.67:0/699699060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a80b2a30 msgr2=0x7fb6a8099e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 --2- 172.21.15.67:0/699699060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a80b2a30 0x7fb6a8099e30 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fb6900097c0 tx=0x7fb69001c470 comp rx=0 tx=0).stop 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 -- 172.21.15.67:0/699699060 shutdown_connections 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 --2- 172.21.15.67:0/699699060 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a80b2a30 0x7fb6a8099e30 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 --2- 172.21.15.67:0/699699060 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a80cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 -- 172.21.15.67:0/699699060 >> 172.21.15.67:0/699699060 conn(0x7fb6a80d1630 msgr2=0x7fb6a80d1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 -- 172.21.15.67:0/699699060 shutdown_connections 2024-04-17T10:35:22.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 -- 172.21.15.67:0/699699060 wait complete. 2024-04-17T10:35:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 Processor -- start 2024-04-17T10:35:22.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.200+0000 7fb6ad000640 1 -- start start 2024-04-17T10:35:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a81513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80cf200 0x7fb6a8159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a815a440 0x7fb6a8156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a80d2370 con 0x7fb6a815a440 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a80b1b30 con 0x7fb6a80cf200 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a80d24f0 con 0x7fb6a8099a50 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a815a440 0x7fb6a8156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a815a440 0x7fb6a8156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56254/0 (socket says 172.21.15.67:56254) 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 -- 172.21.15.67:0/3773881073 learned_addr learned my addr 172.21.15.67:0/3773881073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb69fe00640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80cf200 0x7fb6a8159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7400640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a81513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 -- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 msgr2=0x7fb6a81513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a81513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 -- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80cf200 msgr2=0x7fb6a8159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80cf200 0x7fb6a8159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 -- 172.21.15.67:0/3773881073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a81558a0 con 0x7fb6a815a440 2024-04-17T10:35:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6a7e00640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a815a440 0x7fb6a8156ce0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000caa0 tx=0x7fb6a000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb69f400640 1 -- 172.21.15.67:0/3773881073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0017070 con 0x7fb6a815a440 2024-04-17T10:35:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a8155b80 con 0x7fb6a815a440 2024-04-17T10:35:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb69f400640 1 -- 172.21.15.67:0/3773881073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a000ac00 con 0x7fb6a815a440 2024-04-17T10:35:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb69f400640 1 -- 172.21.15.67:0/3773881073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0012400 con 0x7fb6a815a440 2024-04-17T10:35:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a8156140 con 0x7fb6a815a440 2024-04-17T10:35:22.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.204+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a804f780 con 0x7fb6a815a440 2024-04-17T10:35:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.208+0000 7fb69f400640 1 -- 172.21.15.67:0/3773881073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb6a001a070 con 0x7fb6a815a440 2024-04-17T10:35:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.208+0000 7fb69f400640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb684066f90 0x7fb684069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.208+0000 7fb69f400640 1 -- 172.21.15.67:0/3773881073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb6a0084f70 con 0x7fb6a815a440 2024-04-17T10:35:22.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.208+0000 7fb6a7400640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb684066f90 0x7fb684069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.208+0000 7fb6a7400640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb684066f90 0x7fb684069450 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8158d30 tx=0x7fb698009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.212+0000 7fb69f400640 1 -- 172.21.15.67:0/3773881073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb6a00556d0 con 0x7fb6a815a440 2024-04-17T10:35:22.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.372+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb6a80712f0 con 0x7fb6a815a440 2024-04-17T10:35:22.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.372+0000 7fb69f400640 1 -- 172.21.15.67:0/3773881073 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v131) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb6a0059770 con 0x7fb6a815a440 2024-04-17T10:35:22.366 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:35:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb684066f90 msgr2=0x7fb684069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb684066f90 0x7fb684069450 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8158d30 tx=0x7fb698009190 comp rx=0 tx=0).stop 2024-04-17T10:35:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a815a440 msgr2=0x7fb6a8156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a815a440 0x7fb6a8156ce0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fb6a000caa0 tx=0x7fb6a000cf60 comp rx=0 tx=0).stop 2024-04-17T10:35:22.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 shutdown_connections 2024-04-17T10:35:22.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb684066f90 0x7fb684069450 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6a815a440 0x7fb6a8156ce0 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb6a80cf200 0x7fb6a8159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 --2- 172.21.15.67:0/3773881073 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb6a8099a50 0x7fb6a81513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 >> 172.21.15.67:0/3773881073 conn(0x7fb6a80d1630 msgr2=0x7fb6a80cfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:22.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 shutdown_connections 2024-04-17T10:35:22.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.376+0000 7fb6ad000640 1 -- 172.21.15.67:0/3773881073 wait complete. 2024-04-17T10:35:22.387 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:22.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 Processor -- start 2024-04-17T10:35:22.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 -- start start 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa98c0ce400 0x7fa98c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa98c15e550 0x7fa98c0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c161650 con 0x7fa98c0ce400 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98c1625e0 con 0x7fa98c15e550 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa993000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c15eb50 con 0x7fa98c0ce020 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa983400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa98c0ce400 0x7fa98c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa983400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa98c0ce400 0x7fa98c15e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56262/0 (socket says 172.21.15.67:56262) 2024-04-17T10:35:22.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa983400640 1 -- 172.21.15.67:0/3091600311 learned_addr learned my addr 172.21.15.67:0/3091600311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa991000640 1 --2- 172.21.15.67:0/3091600311 >> v2:172.21.15.86:3300/0 conn(0x7fa98c15e550 0x7fa98c0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.536+0000 7fa98be00640 1 --2- 172.21.15.67:0/3091600311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa983400640 1 -- 172.21.15.67:0/3091600311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 msgr2=0x7fa98c1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:22.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa983400640 1 --2- 172.21.15.67:0/3091600311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa983400640 1 -- 172.21.15.67:0/3091600311 >> v2:172.21.15.86:3300/0 conn(0x7fa98c15e550 msgr2=0x7fa98c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa983400640 1 --2- 172.21.15.67:0/3091600311 >> v2:172.21.15.86:3300/0 conn(0x7fa98c15e550 0x7fa98c0cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa983400640 1 -- 172.21.15.67:0/3091600311 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c15ee30 con 0x7fa98c0ce400 2024-04-17T10:35:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa983400640 1 --2- 172.21.15.67:0/3091600311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c15e010 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0d2840 tx=0x7fa97401c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462929d3584fe88 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa98b400640 1 -- 172.21.15.67:0/3091600311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9740b1070 con 0x7fa98c0ce400 2024-04-17T10:35:22.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa98b400640 1 -- 172.21.15.67:0/3091600311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9740ac620 con 0x7fa98c0ce400 2024-04-17T10:35:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa98b400640 1 -- 172.21.15.67:0/3091600311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9740ac9f0 con 0x7fa98c0ce400 2024-04-17T10:35:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 -- 172.21.15.67:0/3091600311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 msgr2=0x7fa98c15e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 --2- 172.21.15.67:0/3091600311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c15e010 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fa98c0d2840 tx=0x7fa97401c4f0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 -- 172.21.15.67:0/3091600311 shutdown_connections 2024-04-17T10:35:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 --2- 172.21.15.67:0/3091600311 >> v2:172.21.15.86:3300/0 conn(0x7fa98c15e550 0x7fa98c0cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 --2- 172.21.15.67:0/3091600311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c15e010 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 --2- 172.21.15.67:0/3091600311 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 -- 172.21.15.67:0/3091600311 >> 172.21.15.67:0/3091600311 conn(0x7fa98c0b33f0 msgr2=0x7fa98c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.540+0000 7fa993000640 1 -- 172.21.15.67:0/3091600311 shutdown_connections 2024-04-17T10:35:22.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 -- 172.21.15.67:0/3091600311 wait complete. 2024-04-17T10:35:22.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 Processor -- start 2024-04-17T10:35:22.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 -- start start 2024-04-17T10:35:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c14b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa98c15e550 0x7fa98c15d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c06c0f0 con 0x7fa98c0ce400 2024-04-17T10:35:22.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c06bf70 con 0x7fa98c15e550 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa993000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa98c0c84b0 con 0x7fa98c0ce020 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c14b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c14b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56276/0 (socket says 172.21.15.67:56276) 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 -- 172.21.15.67:0/592214756 learned_addr learned my addr 172.21.15.67:0/592214756 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa98be00640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa991000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa98c15e550 0x7fa98c15d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 -- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 msgr2=0x7fa98c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 -- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa98c15e550 msgr2=0x7fa98c15d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa98c15e550 0x7fa98c15d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 -- 172.21.15.67:0/592214756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98c159df0 con 0x7fa98c0ce400 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa991000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa98c15e550 0x7fa98c15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa983400640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c14b670 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fa97401cd90 tx=0x7fa974019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa988c00640 1 -- 172.21.15.67:0/592214756 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9740b1040 con 0x7fa98c0ce400 2024-04-17T10:35:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa988c00640 1 -- 172.21.15.67:0/592214756 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9740ba070 con 0x7fa98c0ce400 2024-04-17T10:35:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.544+0000 7fa988c00640 1 -- 172.21.15.67:0/592214756 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9740b5680 con 0x7fa98c0ce400 2024-04-17T10:35:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa98c15a060 con 0x7fa98c0ce400 2024-04-17T10:35:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa98c15a620 con 0x7fa98c0ce400 2024-04-17T10:35:22.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa98c070ab0 con 0x7fa98c0ce400 2024-04-17T10:35:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa988c00640 1 -- 172.21.15.67:0/592214756 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa9740b5820 con 0x7fa98c0ce400 2024-04-17T10:35:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa988c00640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa968066f90 0x7fa968069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa988c00640 1 -- 172.21.15.67:0/592214756 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa97411f060 con 0x7fa98c0ce400 2024-04-17T10:35:22.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa98be00640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa968066f90 0x7fa968069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.548+0000 7fa98be00640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa968066f90 0x7fa968069450 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0045f0 tx=0x7fa97c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.552+0000 7fa988c00640 1 -- 172.21.15.67:0/592214756 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa974016110 con 0x7fa98c0ce400 2024-04-17T10:35:22.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.748+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa98c074e30 con 0x7fa98c0ce400 2024-04-17T10:35:22.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa988c00640 1 -- 172.21.15.67:0/592214756 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4146 (secure 0 0 0) 0x7fa9740efdf0 con 0x7fa98c0ce400 2024-04-17T10:35:22.743 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:22.744 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:35:15.983015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9255":{"gid":9255,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:22.744 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T10:35:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa968066f90 msgr2=0x7fa968069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa968066f90 0x7fa968069450 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa97c0045f0 tx=0x7fa97c008040 comp rx=0 tx=0).stop 2024-04-17T10:35:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 msgr2=0x7fa98c14b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c14b670 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fa97401cd90 tx=0x7fa974019290 comp rx=0 tx=0).stop 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 shutdown_connections 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa968066f90 0x7fa968069450 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa98c15e550 0x7fa98c15d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa98c0ce400 0x7fa98c14b670 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 --2- 172.21.15.67:0/592214756 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa98c0ce020 0x7fa98c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 >> 172.21.15.67:0/592214756 conn(0x7fa98c0b33f0 msgr2=0x7fa98c0cd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 shutdown_connections 2024-04-17T10:35:22.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.752+0000 7fa993000640 1 -- 172.21.15.67:0/592214756 wait complete. 2024-04-17T10:35:22.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:22.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.912+0000 7f14cd600640 1 Processor -- start 2024-04-17T10:35:22.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14cd600640 1 -- start start 2024-04-17T10:35:22.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14cd600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f14c8071310 0x7f14c8071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14cd600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c806e2f0 con 0x7f14c8145040 2024-04-17T10:35:22.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14cd600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c806d6e0 con 0x7f14c80cc300 2024-04-17T10:35:22.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14cd600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c80a86c0 con 0x7f14c8071310 2024-04-17T10:35:22.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f14c8145040 0x7f14c806e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48796/0 (socket says 172.21.15.67:48796) 2024-04-17T10:35:22.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6a00640 1 -- 172.21.15.67:0/2213967321 learned_addr learned my addr 172.21.15.67:0/2213967321 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:22.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14be000640 1 --2- 172.21.15.67:0/2213967321 >> v2:172.21.15.86:3300/0 conn(0x7f14c8071310 0x7f14c8071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021478672 0 0) 0x7f14c806e2f0 con 0x7f14c8145040 2024-04-17T10:35:22.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ac003530 con 0x7f14c8145040 2024-04-17T10:35:22.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903671066 0 0) 0x7f14ac003530 con 0x7f14c8145040 2024-04-17T10:35:22.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 >> v2:172.21.15.86:3300/0 conn(0x7f14c8071310 msgr2=0x7f14c8071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 --2- 172.21.15.67:0/2213967321 >> v2:172.21.15.86:3300/0 conn(0x7f14c8071310 0x7f14c8071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 >> v1:172.21.15.86:6789/0 conn(0x7f14c80cc300 legacy=0x7f14c80cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c806d9c0 con 0x7f14c8145040 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f14b8001680 con 0x7f14c8145040 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 >> v1:172.21.15.67:6789/0 conn(0x7f14c8145040 legacy=0x7f14c806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ac0039c0 0x7f14ac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac004810 0x7f14ac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ac0051d0 0x7f14ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac00c050 con 0x7f14ac0051d0 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac00c200 con 0x7f14ac0039c0 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac00c400 con 0x7f14ac004810 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14be000640 1 --2- 172.21.15.67:0/2213967321 >> v2:172.21.15.86:3300/0 conn(0x7f14c8071310 0x7f14c8071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c6a00640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ac0039c0 0x7f14ac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c7400640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ac0051d0 0x7f14ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14be000640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac004810 0x7f14ac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c7400640 1 -- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac004810 msgr2=0x7f14ac004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:22.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c7400640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac004810 0x7f14ac004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c7400640 1 -- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ac0039c0 msgr2=0x7f14ac003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:22.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c7400640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14ac0039c0 0x7f14ac003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14c7400640 1 -- 172.21.15.67:0/2213967321 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac003530 con 0x7f14ac0051d0 2024-04-17T10:35:22.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.916+0000 7f14be000640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac004810 0x7f14ac004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:22.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14c7400640 1 -- 172.21.15.67:0/2213967321 reap_dead start 2024-04-17T10:35:22.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14c7400640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ac0051d0 0x7f14ac00b9d0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f14c0001e70 tx=0x7f14c001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c96d232716370f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c0001a90 con 0x7f14ac0051d0 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c0003bc0 con 0x7f14ac0051d0 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14c6000640 1 -- 172.21.15.67:0/2213967321 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c0005020 con 0x7f14ac0051d0 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14cd600640 1 -- 172.21.15.67:0/2213967321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ac0051d0 msgr2=0x7f14ac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14cd600640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ac0051d0 0x7f14ac00b9d0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f14c0001e70 tx=0x7f14c001c780 comp rx=0 tx=0).stop 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14cd600640 1 -- 172.21.15.67:0/2213967321 shutdown_connections 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14cd600640 1 --2- 172.21.15.67:0/2213967321 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14ac0051d0 0x7f14ac00b9d0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14cd600640 1 -- 172.21.15.67:0/2213967321 >> 172.21.15.67:0/2213967321 conn(0x7f14c806ac90 msgr2=0x7f14c806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14cd600640 1 -- 172.21.15.67:0/2213967321 shutdown_connections 2024-04-17T10:35:22.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.920+0000 7f14cd600640 1 -- 172.21.15.67:0/2213967321 wait complete. 2024-04-17T10:35:22.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 Processor -- start 2024-04-17T10:35:22.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 -- start start 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac0051d0 0x7f14c81f1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c813cf20 0x7f14c8138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c813d3a0 0x7f14c81f3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c806df80 con 0x7f14c813cf20 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c815bf10 con 0x7f14c813d3a0 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c8145420 con 0x7f14ac0051d0 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c813cf20 0x7f14c8138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c813cf20 0x7f14c8138960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56288/0 (socket says 172.21.15.67:56288) 2024-04-17T10:35:22.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 -- 172.21.15.67:0/1388699480 learned_addr learned my addr 172.21.15.67:0/1388699480 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 -- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac0051d0 msgr2=0x7f14c81f1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac0051d0 0x7f14c81f1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac0051d0 0x7f14c81f1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14c7400640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c813d3a0 0x7f14c81f3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 -- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c813d3a0 msgr2=0x7f14c81f3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c813d3a0 0x7f14c81f3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 -- 172.21.15.67:0/1388699480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c813a4c0 con 0x7f14c813cf20 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac0051d0 0x7f14c81f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14c7400640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c813d3a0 0x7f14c81f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14be000640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c813cf20 0x7f14c8138960 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f14b0004660 tx=0x7f14b0011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14bf400640 1 -- 172.21.15.67:0/1388699480 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b000f960 con 0x7f14c813cf20 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14bf400640 1 -- 172.21.15.67:0/1388699480 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b000fb00 con 0x7f14c813cf20 2024-04-17T10:35:22.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14bf400640 1 -- 172.21.15.67:0/1388699480 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b0015460 con 0x7f14c813cf20 2024-04-17T10:35:22.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c81396e0 con 0x7f14c813cf20 2024-04-17T10:35:22.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.924+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c813a760 con 0x7f14c813cf20 2024-04-17T10:35:22.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.928+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c8072d50 con 0x7f14c813cf20 2024-04-17T10:35:22.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.928+0000 7f14bf400640 1 -- 172.21.15.67:0/1388699480 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f14b0015600 con 0x7f14c813cf20 2024-04-17T10:35:22.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.928+0000 7f14bf400640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14a4066f60 0x7f14a4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:22.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.928+0000 7f14bf400640 1 -- 172.21.15.67:0/1388699480 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f14b0085c20 con 0x7f14c813cf20 2024-04-17T10:35:22.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.928+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14a4066f60 0x7f14a4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:22.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.928+0000 7f14c6a00640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14a4066f60 0x7f14a4069420 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000c10 tx=0x7f14b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:22.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:22.932+0000 7f14bf400640 1 -- 172.21.15.67:0/1388699480 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f14b0019030 con 0x7f14c813cf20 2024-04-17T10:35:23.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.116+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14c8074e30 con 0x7f14c813cf20 2024-04-17T10:35:23.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.116+0000 7f14bf400640 1 -- 172.21.15.67:0/1388699480 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16979 (secure 0 0 0) 0x7f14b0056a50 con 0x7f14c813cf20 2024-04-17T10:35:23.109 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:23.109 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":131,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:13.952681+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:34:26.120176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-17T10:34:27.343196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6836/538421223":"2024-04-18T10:35:13.952580+0000","172.21.15.86:6834/538421223":"2024-04-18T10:35:13.952580+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14a4066f60 msgr2=0x7f14a4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14a4066f60 0x7f14a4069420 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f14b8000c10 tx=0x7f14b8009040 comp rx=0 tx=0).stop 2024-04-17T10:35:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c813cf20 msgr2=0x7f14c8138960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c813cf20 0x7f14c8138960 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f14b0004660 tx=0x7f14b0011b30 comp rx=0 tx=0).stop 2024-04-17T10:35:23.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 shutdown_connections 2024-04-17T10:35:23.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14a4066f60 0x7f14a4069420 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f14c813d3a0 0x7f14c81f3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f14c813cf20 0x7f14c8138960 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 --2- 172.21.15.67:0/1388699480 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f14ac0051d0 0x7f14c81f1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 >> 172.21.15.67:0/1388699480 conn(0x7f14c806ac90 msgr2=0x7f14c80cc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:23.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 shutdown_connections 2024-04-17T10:35:23.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.120+0000 7f14cd600640 1 -- 172.21.15.67:0/1388699480 wait complete. 2024-04-17T10:35:23.138 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-17T10:35:23.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 Processor -- start 2024-04-17T10:35:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 -- start start 2024-04-17T10:35:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d006eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52d0146da0 0x7f52d0071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f52d0071850 0x7f52d006d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d0161740 con 0x7f52d0146da0 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52d0161a20 con 0x7f52d0071850 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d7800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0161d00 con 0x7f52d01466e0 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52c7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52d0146da0 0x7f52d0071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52c7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f52d0146da0 0x7f52d0071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56290/0 (socket says 172.21.15.67:56290) 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52c7e00640 1 -- 172.21.15.67:0/2084990000 learned_addr learned my addr 172.21.15.67:0/2084990000 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d4e00640 1 --2- 172.21.15.67:0/2084990000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d006eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.288+0000 7f52d5800640 1 --2- 172.21.15.67:0/2084990000 >> v2:172.21.15.86:3300/0 conn(0x7f52d0071850 0x7f52d006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52c7e00640 1 -- 172.21.15.67:0/2084990000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 msgr2=0x7f52d006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52c7e00640 1 --2- 172.21.15.67:0/2084990000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d006eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52c7e00640 1 -- 172.21.15.67:0/2084990000 >> v2:172.21.15.86:3300/0 conn(0x7f52d0071850 msgr2=0x7f52d006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52c7e00640 1 --2- 172.21.15.67:0/2084990000 >> v2:172.21.15.86:3300/0 conn(0x7f52d0071850 0x7f52d006d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52c7e00640 1 -- 172.21.15.67:0/2084990000 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d0161fe0 con 0x7f52d0146da0 2024-04-17T10:35:23.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52c7e00640 1 --2- 172.21.15.67:0/2084990000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0146da0 0x7f52d0071310 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f52d015e0b0 tx=0x7f52b801c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e1674dad07dd4a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52cfe00640 1 -- 172.21.15.67:0/2084990000 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80b0070 con 0x7f52d0146da0 2024-04-17T10:35:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52cfe00640 1 -- 172.21.15.67:0/2084990000 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b80ab600 con 0x7f52d0146da0 2024-04-17T10:35:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.292+0000 7f52cfe00640 1 -- 172.21.15.67:0/2084990000 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b80ab9d0 con 0x7f52d0146da0 2024-04-17T10:35:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- 172.21.15.67:0/2084990000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0146da0 msgr2=0x7f52d0071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 --2- 172.21.15.67:0/2084990000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0146da0 0x7f52d0071310 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f52d015e0b0 tx=0x7f52b801c260 comp rx=0 tx=0).stop 2024-04-17T10:35:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- 172.21.15.67:0/2084990000 shutdown_connections 2024-04-17T10:35:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 --2- 172.21.15.67:0/2084990000 >> v2:172.21.15.86:3300/0 conn(0x7f52d0071850 0x7f52d006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 --2- 172.21.15.67:0/2084990000 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0146da0 0x7f52d0071310 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 --2- 172.21.15.67:0/2084990000 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d006eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- 172.21.15.67:0/2084990000 >> 172.21.15.67:0/2084990000 conn(0x7f52d006ac90 msgr2=0x7f52d006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- 172.21.15.67:0/2084990000 shutdown_connections 2024-04-17T10:35:23.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- 172.21.15.67:0/2084990000 wait complete. 2024-04-17T10:35:23.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 Processor -- start 2024-04-17T10:35:23.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- start start 2024-04-17T10:35:23.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0071850 0x7f52d00ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d012dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0146da0 0x7f52d015e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ae170 con 0x7f52d0071850 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00adff0 con 0x7f52d0146da0 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d7800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d00ae2f0 con 0x7f52d01466e0 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0071850 0x7f52d00ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0071850 0x7f52d00ae7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56304/0 (socket says 172.21.15.67:56304) 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 -- 172.21.15.67:0/1875305944 learned_addr learned my addr 172.21.15.67:0/1875305944 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52c7e00640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d012dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d5800640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0146da0 0x7f52d015e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 -- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 msgr2=0x7f52d012dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d012dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 -- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0146da0 msgr2=0x7f52d015e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0146da0 0x7f52d015e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 -- 172.21.15.67:0/1875305944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00cd240 con 0x7f52d0071850 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52d4e00640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0071850 0x7f52d00ae7c0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f52c000d800 tx=0x7f52c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.296+0000 7f52cd600640 1 -- 172.21.15.67:0/1875305944 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c000b6c0 con 0x7f52d0071850 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52cd600640 1 -- 172.21.15.67:0/1875305944 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c0019070 con 0x7f52d0071850 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d013cea0 con 0x7f52d0071850 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52cd600640 1 -- 172.21.15.67:0/1875305944 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c000ba60 con 0x7f52d0071850 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d00b01d0 con 0x7f52d0071850 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d00aed60 con 0x7f52d0071850 2024-04-17T10:35:23.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52cd600640 1 -- 172.21.15.67:0/1875305944 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f52c000bc00 con 0x7f52d0071850 2024-04-17T10:35:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52cd600640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f52ac066f60 0x7f52ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52cd600640 1 -- 172.21.15.67:0/1875305944 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f52c0085700 con 0x7f52d0071850 2024-04-17T10:35:23.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52c7e00640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f52ac066f60 0x7f52ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.300+0000 7f52c7e00640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f52ac066f60 0x7f52ac069420 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f52b80ad000 tx=0x7f52b80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:23.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.304+0000 7f52cd600640 1 -- 172.21.15.67:0/1875305944 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f52c0056400 con 0x7f52d0071850 2024-04-17T10:35:23.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52d0072d50 con 0x7f52d0071850 2024-04-17T10:35:23.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52cd600640 1 -- 172.21.15.67:0/1875305944 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4146 (secure 0 0 0) 0x7f52c0055d00 con 0x7f52d0071850 2024-04-17T10:35:23.504 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:23.504 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:35:15.983015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9255":{"gid":9255,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:23.504 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T10:35:23.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f52ac066f60 msgr2=0x7f52ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f52ac066f60 0x7f52ac069420 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f52b80ad000 tx=0x7f52b80a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:23.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0071850 msgr2=0x7f52d00ae7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0071850 0x7f52d00ae7c0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f52c000d800 tx=0x7f52c000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 shutdown_connections 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f52ac066f60 0x7f52ac069420 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f52d0146da0 0x7f52d015e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f52d01466e0 0x7f52d012dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 --2- 172.21.15.67:0/1875305944 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f52d0071850 0x7f52d00ae7c0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 >> 172.21.15.67:0/1875305944 conn(0x7f52d006ac90 msgr2=0x7f52d0143c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 shutdown_connections 2024-04-17T10:35:23.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.512+0000 7f52d7800640 1 -- 172.21.15.67:0/1875305944 wait complete. 2024-04-17T10:35:23.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.672+0000 7f4dd9400640 1 Processor -- start 2024-04-17T10:35:23.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.672+0000 7f4dd9400640 1 -- start start 2024-04-17T10:35:23.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd406e6e0 0x7f4dd40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd9400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4dd406eac0 0x7f4dd40ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd9400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd415ae40 con 0x7f4dd4143ae0 2024-04-17T10:35:23.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd9400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd40cf430 con 0x7f4dd406eac0 2024-04-17T10:35:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd9400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd4071440 con 0x7f4dd406e6e0 2024-04-17T10:35:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd3200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4dd4143ae0 0x7f4dd4143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48812/0 (socket says 172.21.15.67:48812) 2024-04-17T10:35:23.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd3200640 1 -- 172.21.15.67:0/2859532942 learned_addr learned my addr 172.21.15.67:0/2859532942 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dc9e00640 1 --2- 172.21.15.67:0/2859532942 >> v2:172.21.15.86:3300/0 conn(0x7f4dd406eac0 0x7f4dd40ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd2800640 1 --2- 172.21.15.67:0/2859532942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd406e6e0 0x7f4dd40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd1e00640 1 -- 172.21.15.67:0/2859532942 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450478014 0 0) 0x7f4dd415ae40 con 0x7f4dd4143ae0 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd1e00640 1 -- 172.21.15.67:0/2859532942 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db8003530 con 0x7f4dd4143ae0 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd1e00640 1 -- 172.21.15.67:0/2859532942 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065694271 0 0) 0x7f4db8003530 con 0x7f4dd4143ae0 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd2800640 1 -- 172.21.15.67:0/2859532942 >> v2:172.21.15.86:3300/0 conn(0x7f4dd406eac0 msgr2=0x7f4dd40ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd2800640 1 --2- 172.21.15.67:0/2859532942 >> v2:172.21.15.86:3300/0 conn(0x7f4dd406eac0 0x7f4dd40ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd2800640 1 -- 172.21.15.67:0/2859532942 >> v1:172.21.15.67:6789/0 conn(0x7f4dd4143ae0 legacy=0x7f4dd4143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dd2800640 1 -- 172.21.15.67:0/2859532942 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd4071720 con 0x7f4dd406e6e0 2024-04-17T10:35:23.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.676+0000 7f4dc9e00640 1 --2- 172.21.15.67:0/2859532942 >> v2:172.21.15.86:3300/0 conn(0x7f4dd406eac0 0x7f4dd40ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:23.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd2800640 1 --2- 172.21.15.67:0/2859532942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd406e6e0 0x7f4dd40ce9b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40097c0 tx=0x7f4dc401c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22cdd4926346cf33 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd1e00640 1 -- 172.21.15.67:0/2859532942 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc40b0070 con 0x7f4dd406e6e0 2024-04-17T10:35:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd1e00640 1 -- 172.21.15.67:0/2859532942 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc40ab400 con 0x7f4dd406e6e0 2024-04-17T10:35:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd1e00640 1 -- 172.21.15.67:0/2859532942 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc001a00 con 0x7f4dd406e6e0 2024-04-17T10:35:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 -- 172.21.15.67:0/2859532942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd406e6e0 msgr2=0x7f4dd40ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 --2- 172.21.15.67:0/2859532942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd406e6e0 0x7f4dd40ce9b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40097c0 tx=0x7f4dc401c300 comp rx=0 tx=0).stop 2024-04-17T10:35:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 -- 172.21.15.67:0/2859532942 shutdown_connections 2024-04-17T10:35:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 --2- 172.21.15.67:0/2859532942 >> v2:172.21.15.86:3300/0 conn(0x7f4dd406eac0 0x7f4dd40ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 --2- 172.21.15.67:0/2859532942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd406e6e0 0x7f4dd40ce9b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 -- 172.21.15.67:0/2859532942 >> 172.21.15.67:0/2859532942 conn(0x7f4dd406ac90 msgr2=0x7f4dd406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 -- 172.21.15.67:0/2859532942 shutdown_connections 2024-04-17T10:35:23.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 -- 172.21.15.67:0/2859532942 wait complete. 2024-04-17T10:35:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 Processor -- start 2024-04-17T10:35:23.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.680+0000 7f4dd9400640 1 -- start start 2024-04-17T10:35:23.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dd406e6e0 0x7f4dd41409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dd406eac0 0x7f4dd4138740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd4143ae0 0x7f4dd415a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b3a40 con 0x7f4dd406eac0 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b38c0 con 0x7f4dd406e6e0 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd40b3bc0 con 0x7f4dd4143ae0 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dd406eac0 0x7f4dd4138740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dd406eac0 0x7f4dd4138740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56318/0 (socket says 172.21.15.67:56318) 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 -- 172.21.15.67:0/1884494441 learned_addr learned my addr 172.21.15.67:0/1884494441 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd2800640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dd406e6e0 0x7f4dd41409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd3200640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd4143ae0 0x7f4dd415a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 -- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd4143ae0 msgr2=0x7f4dd415a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd4143ae0 0x7f4dd415a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 -- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dd406e6e0 msgr2=0x7f4dd41409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dd406e6e0 0x7f4dd41409d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 -- 172.21.15.67:0/1884494441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd41410d0 con 0x7f4dd406eac0 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dc9e00640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dd406eac0 0x7f4dd4138740 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00ebe0 tx=0x7f4dbc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dcb400640 1 -- 172.21.15.67:0/1884494441 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc017070 con 0x7f4dd406eac0 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dcb400640 1 -- 172.21.15.67:0/1884494441 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc00cc30 con 0x7f4dd406eac0 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd41413b0 con 0x7f4dd406eac0 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dcb400640 1 -- 172.21.15.67:0/1884494441 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc013470 con 0x7f4dd406eac0 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd4141970 con 0x7f4dd406eac0 2024-04-17T10:35:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.684+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd4072d50 con 0x7f4dd406eac0 2024-04-17T10:35:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.688+0000 7f4dcb400640 1 -- 172.21.15.67:0/1884494441 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4dbc01a070 con 0x7f4dd406eac0 2024-04-17T10:35:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.688+0000 7f4dcb400640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4db0066f60 0x7f4db0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.688+0000 7f4dcb400640 1 -- 172.21.15.67:0/1884494441 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4dbc085450 con 0x7f4dd406eac0 2024-04-17T10:35:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.688+0000 7f4dd2800640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4db0066f60 0x7f4db0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:23.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.688+0000 7f4dd2800640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4db0066f60 0x7f4db0069420 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40ad000 tx=0x7f4dc40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:23.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.692+0000 7f4dcb400640 1 -- 172.21.15.67:0/1884494441 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4dbc055c60 con 0x7f4dd406eac0 2024-04-17T10:35:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.884+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dd4074e30 con 0x7f4dd406eac0 2024-04-17T10:35:23.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.884+0000 7f4dcb400640 1 -- 172.21.15.67:0/1884494441 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4146 (secure 0 0 0) 0x7f4dbc059d00 con 0x7f4dd406eac0 2024-04-17T10:35:23.877 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:23.878 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":94,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:34:28.382849+0000","modified":"2024-04-17T10:35:15.983015+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9255},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9255":{"gid":9255,"name":"a","rank":0,"incarnation":91,"state":"up:active","state_seq":16,"addr":"172.21.15.67:6835/1493271033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1493271033},{"type":"v1","addr":"172.21.15.67:6835","nonce":1493271033}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-04-17T10:35:23.878 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 94 2024-04-17T10:35:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4db0066f60 msgr2=0x7f4db0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4db0066f60 0x7f4db0069420 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40ad000 tx=0x7f4dc40a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:23.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dd406eac0 msgr2=0x7f4dd4138740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dd406eac0 0x7f4dd4138740 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc00ebe0 tx=0x7f4dbc00c520 comp rx=0 tx=0).stop 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 shutdown_connections 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4db0066f60 0x7f4db0069420 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4dd4143ae0 0x7f4dd415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dd406eac0 0x7f4dd4138740 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 --2- 172.21.15.67:0/1884494441 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dd406e6e0 0x7f4dd41409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 >> 172.21.15.67:0/1884494441 conn(0x7f4dd406ac90 msgr2=0x7f4dd4142e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 shutdown_connections 2024-04-17T10:35:23.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:23.888+0000 7f4dd9400640 1 -- 172.21.15.67:0/1884494441 wait complete. 2024-04-17T10:35:23.898 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:24.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.048+0000 7f3dd1000640 1 Processor -- start 2024-04-17T10:35:24.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.048+0000 7f3dd1000640 1 -- start start 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dd1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dcc06e6e0 0x7f3dcc0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dd1000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3dcc06eac0 0x7f3dcc0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dd1000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc15ae40 con 0x7f3dcc143ae0 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dd1000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dcc0cf430 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dd1000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dcc071440 con 0x7f3dcc06eac0 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dcc06e6e0 0x7f3dcc0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc9e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dcc06e6e0 0x7f3dcc0ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56330/0 (socket says 172.21.15.67:56330) 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc9e00640 1 -- 172.21.15.67:0/3369455835 learned_addr learned my addr 172.21.15.67:0/3369455835 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:24.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc3e00640 1 --2- 172.21.15.67:0/3369455835 >> v2:172.21.15.86:3300/0 conn(0x7f3dcc06eac0 0x7f3dcc0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc9e00640 1 -- 172.21.15.67:0/3369455835 >> v2:172.21.15.86:3300/0 conn(0x7f3dcc06eac0 msgr2=0x7f3dcc0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc9e00640 1 --2- 172.21.15.67:0/3369455835 >> v2:172.21.15.86:3300/0 conn(0x7f3dcc06eac0 0x7f3dcc0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc9e00640 1 -- 172.21.15.67:0/3369455835 >> v1:172.21.15.86:6789/0 conn(0x7f3dcc143ae0 legacy=0x7f3dcc143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.052+0000 7f3dc9e00640 1 -- 172.21.15.67:0/3369455835 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc071720 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dc9e00640 1 --2- 172.21.15.67:0/3369455835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0ce9b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc06ef80 tx=0x7f3dbc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f8d1e4d5584b45 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dc9400640 1 -- 172.21.15.67:0/3369455835 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0ab660 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dc9400640 1 -- 172.21.15.67:0/3369455835 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0b0070 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dc9400640 1 -- 172.21.15.67:0/3369455835 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0aba20 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 -- 172.21.15.67:0/3369455835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 msgr2=0x7f3dcc0ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 --2- 172.21.15.67:0/3369455835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0ce9b0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc06ef80 tx=0x7f3dbc01c460 comp rx=0 tx=0).stop 2024-04-17T10:35:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 -- 172.21.15.67:0/3369455835 shutdown_connections 2024-04-17T10:35:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 --2- 172.21.15.67:0/3369455835 >> v2:172.21.15.86:3300/0 conn(0x7f3dcc06eac0 0x7f3dcc0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 --2- 172.21.15.67:0/3369455835 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0ce9b0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 -- 172.21.15.67:0/3369455835 >> 172.21.15.67:0/3369455835 conn(0x7f3dcc06ac90 msgr2=0x7f3dcc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:24.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 -- 172.21.15.67:0/3369455835 shutdown_connections 2024-04-17T10:35:24.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 -- 172.21.15.67:0/3369455835 wait complete. 2024-04-17T10:35:24.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 Processor -- start 2024-04-17T10:35:24.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.056+0000 7f3dd1000640 1 -- start start 2024-04-17T10:35:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dcc06eac0 0x7f3dcc159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dcc143ae0 0x7f3dcc13a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0b3a40 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0b38c0 con 0x7f3dcc143ae0 2024-04-17T10:35:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0b3bc0 con 0x7f3dcc06eac0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0adb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56336/0 (socket says 172.21.15.67:56336) 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 -- 172.21.15.67:0/1426347766 learned_addr learned my addr 172.21.15.67:0/1426347766 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc3e00640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dcc06eac0 0x7f3dcc159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dca800640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dcc143ae0 0x7f3dcc13a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 -- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dcc06eac0 msgr2=0x7f3dcc159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dcc06eac0 0x7f3dcc159d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 -- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dcc143ae0 msgr2=0x7f3dcc13a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dcc143ae0 0x7f3dcc13a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 -- 172.21.15.67:0/1426347766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc0ae140 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc9e00640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0adb60 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0ad040 tx=0x7f3dbc01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc2000640 1 -- 172.21.15.67:0/1426347766 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0be5f0 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc2000640 1 -- 172.21.15.67:0/1426347766 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc0be7b0 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dcc14b1f0 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc2000640 1 -- 172.21.15.67:0/1426347766 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0beba0 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dcc14ba90 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dcc14b5e0 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.060+0000 7f3dc2000640 1 -- 172.21.15.67:0/1426347766 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3dbc0b43f0 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.064+0000 7f3dc2000640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3da8066f60 0x7f3da8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.064+0000 7f3dc2000640 1 -- 172.21.15.67:0/1426347766 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3dbc120010 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.064+0000 7f3dc3e00640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3da8066f60 0x7f3da8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.064+0000 7f3dc3e00640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3da8066f60 0x7f3da8069420 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc13f890 tx=0x7f3db4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.068+0000 7f3dc2000640 1 -- 172.21.15.67:0/1426347766 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3dbc0c4060 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3dcc074e30 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dc2000640 1 -- 172.21.15.67:0/1426347766 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+16979 (secure 0 0 0) 0x7f3dbc016110 con 0x7f3dcc06e6e0 2024-04-17T10:35:24.224 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:24.224 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":131,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:13.952681+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:34:26.120176+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_data","create_time":"2024-04-17T10:34:27.343196+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6836/538421223":"2024-04-18T10:35:13.952580+0000","172.21.15.86:6834/538421223":"2024-04-18T10:35:13.952580+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3da8066f60 msgr2=0x7f3da8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3da8066f60 0x7f3da8069420 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc13f890 tx=0x7f3db4009190 comp rx=0 tx=0).stop 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 msgr2=0x7f3dcc0adb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0adb60 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc0ad040 tx=0x7f3dbc01aca0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 shutdown_connections 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3da8066f60 0x7f3da8069420 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dcc143ae0 0x7f3dcc13a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dcc06eac0 0x7f3dcc159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 --2- 172.21.15.67:0/1426347766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dcc06e6e0 0x7f3dcc0adb60 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 >> 172.21.15.67:0/1426347766 conn(0x7f3dcc06ac90 msgr2=0x7f3dcc0aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 shutdown_connections 2024-04-17T10:35:24.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.232+0000 7f3dd1000640 1 -- 172.21.15.67:0/1426347766 wait complete. 2024-04-17T10:35:24.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:35:24.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb63e00640 1 Processor -- start 2024-04-17T10:35:24.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb63e00640 1 -- start start 2024-04-17T10:35:24.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb63e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb5c0ce020 0x7feb5c1620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb63e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0cf510 0x7feb5c0cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb63e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c15eb50 con 0x7feb5c0ce400 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb63e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5c161650 con 0x7feb5c0ce020 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb63e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c1625e0 con 0x7feb5c0cf510 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb5c0ce020 0x7feb5c1620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feb5c0ce020 0x7feb5c1620a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56344/0 (socket says 172.21.15.67:56344) 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61400640 1 -- 172.21.15.67:0/2135711965 learned_addr learned my addr 172.21.15.67:0/2135711965 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:24.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb60a00640 1 --1- 172.21.15.67:0/2135711965 >> v1:172.21.15.67:6789/0 conn(0x7feb5c0ce400 0x7feb5c15e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48814/0 (socket says 172.21.15.67:48814) 2024-04-17T10:35:24.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61e00640 1 --2- 172.21.15.67:0/2135711965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0cf510 0x7feb5c0cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61400640 1 -- 172.21.15.67:0/2135711965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0cf510 msgr2=0x7feb5c0cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:24.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61400640 1 --2- 172.21.15.67:0/2135711965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0cf510 0x7feb5c0cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61400640 1 -- 172.21.15.67:0/2135711965 >> v1:172.21.15.67:6789/0 conn(0x7feb5c0ce400 legacy=0x7feb5c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb61400640 1 -- 172.21.15.67:0/2135711965 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c15ee30 con 0x7feb5c0ce020 2024-04-17T10:35:24.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.396+0000 7feb5be00640 1 -- 172.21.15.67:0/2135711965 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290411923 0 0) 0x7feb5c15eb50 con 0x7feb5c0ce400 2024-04-17T10:35:24.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb61400640 1 --2- 172.21.15.67:0/2135711965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce020 0x7feb5c1620a0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7feb500098a0 tx=0x7feb5001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637965ede041ad6e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb5be00640 1 -- 172.21.15.67:0/2135711965 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb500ab620 con 0x7feb5c0ce020 2024-04-17T10:35:24.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb5be00640 1 -- 172.21.15.67:0/2135711965 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb500b0070 con 0x7feb5c0ce020 2024-04-17T10:35:24.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb5be00640 1 -- 172.21.15.67:0/2135711965 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb500ab9e0 con 0x7feb5c0ce020 2024-04-17T10:35:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 -- 172.21.15.67:0/2135711965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce020 msgr2=0x7feb5c1620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 --2- 172.21.15.67:0/2135711965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce020 0x7feb5c1620a0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7feb500098a0 tx=0x7feb5001c510 comp rx=0 tx=0).stop 2024-04-17T10:35:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 -- 172.21.15.67:0/2135711965 shutdown_connections 2024-04-17T10:35:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 --2- 172.21.15.67:0/2135711965 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0cf510 0x7feb5c0cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 --2- 172.21.15.67:0/2135711965 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce020 0x7feb5c1620a0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 -- 172.21.15.67:0/2135711965 >> 172.21.15.67:0/2135711965 conn(0x7feb5c0b33f0 msgr2=0x7feb5c0b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:24.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 -- 172.21.15.67:0/2135711965 shutdown_connections 2024-04-17T10:35:24.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.400+0000 7feb63e00640 1 -- 172.21.15.67:0/2135711965 wait complete. 2024-04-17T10:35:24.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 Processor -- start 2024-04-17T10:35:24.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 -- start start 2024-04-17T10:35:24.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0ce020 0x7feb5c15ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce400 0x7feb5c157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb5c0cf510 0x7feb5c154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c06c0f0 con 0x7feb5c0ce400 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c06bf70 con 0x7feb5c0cf510 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c15e720 con 0x7feb5c0ce020 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce400 0x7feb5c157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce400 0x7feb5c157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56346/0 (socket says 172.21.15.67:56346) 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 -- 172.21.15.67:0/3241963465 learned_addr learned my addr 172.21.15.67:0/3241963465 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb61400640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0ce020 0x7feb5c15ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb61e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb5c0cf510 0x7feb5c154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 -- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0ce020 msgr2=0x7feb5c15ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0ce020 0x7feb5c15ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 -- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb5c0cf510 msgr2=0x7feb5c154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb5c0cf510 0x7feb5c154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 -- 172.21.15.67:0/3241963465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c15b260 con 0x7feb5c0ce400 2024-04-17T10:35:24.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb61e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb5c0cf510 0x7feb5c154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:24.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb60a00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce400 0x7feb5c157790 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7feb4800c520 tx=0x7feb4800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb59600640 1 -- 172.21.15.67:0/3241963465 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb48018070 con 0x7feb5c0ce400 2024-04-17T10:35:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb59600640 1 -- 172.21.15.67:0/3241963465 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb480074e0 con 0x7feb5c0ce400 2024-04-17T10:35:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c15b540 con 0x7feb5c0ce400 2024-04-17T10:35:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb59600640 1 -- 172.21.15.67:0/3241963465 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb480049f0 con 0x7feb5c0ce400 2024-04-17T10:35:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c15bf90 con 0x7feb5c0ce400 2024-04-17T10:35:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.404+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c15c190 con 0x7feb5c0ce400 2024-04-17T10:35:24.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.408+0000 7feb59600640 1 -- 172.21.15.67:0/3241963465 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7feb4801b070 con 0x7feb5c0ce400 2024-04-17T10:35:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.408+0000 7feb59600640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb3c066f60 0x7feb3c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.408+0000 7feb59600640 1 -- 172.21.15.67:0/3241963465 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6187+0+0 (secure 0 0 0) 0x7feb480846a0 con 0x7feb5c0ce400 2024-04-17T10:35:24.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.408+0000 7feb61400640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb3c066f60 0x7feb3c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.408+0000 7feb61400640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb3c066f60 0x7feb3c069420 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feb500af000 tx=0x7feb500a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.412+0000 7feb59600640 1 -- 172.21.15.67:0/3241963465 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7feb5c15c190 con 0x7feb5c0ce400 2024-04-17T10:35:24.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.608+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7feb5c070ab0 con 0x7feb5c0ce400 2024-04-17T10:35:24.600 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:35:24.608+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:35:24.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.608+0000 7feb59600640 1 -- 172.21.15.67:0/3241963465 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v95) v1 ==== 161+0+0 (secure 0 0 0) 0x7feb48054de0 con 0x7feb5c0ce400 2024-04-17T10:35:24.602 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:35:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb3c066f60 msgr2=0x7feb3c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb3c066f60 0x7feb3c069420 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feb500af000 tx=0x7feb500a7000 comp rx=0 tx=0).stop 2024-04-17T10:35:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce400 msgr2=0x7feb5c157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce400 0x7feb5c157790 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7feb4800c520 tx=0x7feb4800c9e0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 shutdown_connections 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feb3c066f60 0x7feb3c069420 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feb5c0cf510 0x7feb5c154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb5c0ce400 0x7feb5c157790 secure :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7feb4800c520 tx=0x7feb4800c9e0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 --2- 172.21.15.67:0/3241963465 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feb5c0ce020 0x7feb5c15ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 >> 172.21.15.67:0/3241963465 conn(0x7feb5c0b33f0 msgr2=0x7feb5c157f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 shutdown_connections 2024-04-17T10:35:24.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.612+0000 7feb63e00640 1 -- 172.21.15.67:0/3241963465 wait complete. 2024-04-17T10:35:24.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-17T10:35:24.673 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:35:24.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.780+0000 7fd4d7200640 1 Processor -- start 2024-04-17T10:35:24.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.780+0000 7fd4d7200640 1 -- start start 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4d7200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d01583f0 0x7fd4d0072380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4d7200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4d00728c0 0x7fd4d00b4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4d7200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d015f360 con 0x7fd4d00b5130 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4d7200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d006bca0 con 0x7fd4d00728c0 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4d7200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b3160 con 0x7fd4d01583f0 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4c7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4d00728c0 0x7fd4d00b4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4c7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd4d00728c0 0x7fd4d00b4bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56370/0 (socket says 172.21.15.67:56370) 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4c7400640 1 -- 172.21.15.67:0/1072847704 learned_addr learned my addr 172.21.15.67:0/1072847704 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:24.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4cfe00640 1 --2- 172.21.15.67:0/1072847704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d01583f0 0x7fd4d0072380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4c7400640 1 -- 172.21.15.67:0/1072847704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d01583f0 msgr2=0x7fd4d0072380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:24.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4c7400640 1 --2- 172.21.15.67:0/1072847704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d01583f0 0x7fd4d0072380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4c7400640 1 -- 172.21.15.67:0/1072847704 >> v1:172.21.15.86:6789/0 conn(0x7fd4d00b5130 legacy=0x7fd4d0161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.784+0000 7fd4c7400640 1 -- 172.21.15.67:0/1072847704 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00b3440 con 0x7fd4d00728c0 2024-04-17T10:35:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4c7400640 1 --2- 172.21.15.67:0/1072847704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d00b4bf0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009870 tx=0x7fd4b801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d397d66bde1586 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4cf400640 1 -- 172.21.15.67:0/1072847704 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80b0070 con 0x7fd4d00728c0 2024-04-17T10:35:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4cf400640 1 -- 172.21.15.67:0/1072847704 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b80ab600 con 0x7fd4d00728c0 2024-04-17T10:35:24.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4cf400640 1 -- 172.21.15.67:0/1072847704 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80ab9d0 con 0x7fd4d00728c0 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 -- 172.21.15.67:0/1072847704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 msgr2=0x7fd4d00b4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 --2- 172.21.15.67:0/1072847704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d00b4bf0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fd4b8009870 tx=0x7fd4b801c560 comp rx=0 tx=0).stop 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 -- 172.21.15.67:0/1072847704 shutdown_connections 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 --2- 172.21.15.67:0/1072847704 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d00b4bf0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 --2- 172.21.15.67:0/1072847704 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d01583f0 0x7fd4d0072380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 -- 172.21.15.67:0/1072847704 >> 172.21.15.67:0/1072847704 conn(0x7fd4d00b2570 msgr2=0x7fd4d0071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 -- 172.21.15.67:0/1072847704 shutdown_connections 2024-04-17T10:35:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 -- 172.21.15.67:0/1072847704 wait complete. 2024-04-17T10:35:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 Processor -- start 2024-04-17T10:35:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.788+0000 7fd4d7200640 1 -- start start 2024-04-17T10:35:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d0071f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d00b5130 0x7fd4d0152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d01583f0 0x7fd4d014fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b4200 con 0x7fd4d00728c0 2024-04-17T10:35:24.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d01587d0 con 0x7fd4d01583f0 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b5540 con 0x7fd4d00b5130 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d0071f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d0071f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56376/0 (socket says 172.21.15.67:56376) 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 -- 172.21.15.67:0/3202045054 learned_addr learned my addr 172.21.15.67:0/3202045054 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4c7400640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d00b5130 0x7fd4d0152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d5200640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d01583f0 0x7fd4d014fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 -- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d00b5130 msgr2=0x7fd4d0152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d00b5130 0x7fd4d0152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 -- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d01583f0 msgr2=0x7fd4d014fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d01583f0 0x7fd4d014fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 -- 172.21.15.67:0/3202045054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0153030 con 0x7fd4d00728c0 2024-04-17T10:35:24.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4cfe00640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d0071f30 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000e870 tx=0x7fd4c000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4ccc00640 1 -- 172.21.15.67:0/3202045054 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c0017070 con 0x7fd4d00728c0 2024-04-17T10:35:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4ccc00640 1 -- 172.21.15.67:0/3202045054 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c000c8e0 con 0x7fd4d00728c0 2024-04-17T10:35:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4ccc00640 1 -- 172.21.15.67:0/3202045054 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c000cc90 con 0x7fd4d00728c0 2024-04-17T10:35:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d0153310 con 0x7fd4d00728c0 2024-04-17T10:35:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d01538d0 con 0x7fd4d00728c0 2024-04-17T10:35:24.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d004ef30 con 0x7fd4d00728c0 2024-04-17T10:35:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.792+0000 7fd4ccc00640 1 -- 172.21.15.67:0/3202045054 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd4c001a070 con 0x7fd4d00728c0 2024-04-17T10:35:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.796+0000 7fd4ccc00640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4ac066f60 0x7fd4ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:24.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.796+0000 7fd4ccc00640 1 -- 172.21.15.67:0/3202045054 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6273+0+0 (secure 0 0 0) 0x7fd4c00853c0 con 0x7fd4d00728c0 2024-04-17T10:35:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.796+0000 7fd4c7400640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4ac066f60 0x7fd4ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:24.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.796+0000 7fd4c7400640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4ac066f60 0x7fd4ac069420 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80ad000 tx=0x7fd4b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:24.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.800+0000 7fd4ccc00640 1 -- 172.21.15.67:0/3202045054 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd4c0055b30 con 0x7fd4d00728c0 2024-04-17T10:35:24.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:24.996+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd4d0154b90 con 0x7fd4d00728c0 2024-04-17T10:35:25.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.612+0000 7fd4ccc00640 1 -- 172.21.15.67:0/3202045054 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v96) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd4c0059bd0 con 0x7fd4d00728c0 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4ac066f60 msgr2=0x7fd4ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4ac066f60 0x7fd4ac069420 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80ad000 tx=0x7fd4b8008040 comp rx=0 tx=0).stop 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 msgr2=0x7fd4d0071f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d0071f30 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fd4c000e870 tx=0x7fd4c000ec70 comp rx=0 tx=0).stop 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 shutdown_connections 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd4ac066f60 0x7fd4ac069420 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd4d01583f0 0x7fd4d014fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd4d00b5130 0x7fd4d0152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 --2- 172.21.15.67:0/3202045054 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd4d00728c0 0x7fd4d0071f30 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 >> 172.21.15.67:0/3202045054 conn(0x7fd4d00b2570 msgr2=0x7fd4d01550b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:25.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 shutdown_connections 2024-04-17T10:35:25.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.616+0000 7fd4d7200640 1 -- 172.21.15.67:0/3202045054 wait complete. 2024-04-17T10:35:25.629 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-17T10:35:25.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.824+0000 7f4ded600640 1 Processor -- start 2024-04-17T10:35:25.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.824+0000 7f4ded600640 1 -- start start 2024-04-17T10:35:25.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4ded600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de8144100 0x7f4de80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:25.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4ded600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4de8071360 0x7f4de806e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:25.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4ded600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4de806ea10 0x7f4de815e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:25.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4ded600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de80b3140 con 0x7f4de8071360 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4ded600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de80b3420 con 0x7f4de806ea10 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4ded600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de8071bb0 con 0x7f4de8144100 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4de8071360 0x7f4de806e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4de8071360 0x7f4de806e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56386/0 (socket says 172.21.15.67:56386) 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 -- 172.21.15.67:0/3292345936 learned_addr learned my addr 172.21.15.67:0/3292345936 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4de6a00640 1 --2- 172.21.15.67:0/3292345936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de8144100 0x7f4de80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4de7400640 1 --2- 172.21.15.67:0/3292345936 >> v2:172.21.15.86:3300/0 conn(0x7f4de806ea10 0x7f4de815e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 -- 172.21.15.67:0/3292345936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de8144100 msgr2=0x7f4de80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 --2- 172.21.15.67:0/3292345936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de8144100 0x7f4de80cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 -- 172.21.15.67:0/3292345936 >> v2:172.21.15.86:3300/0 conn(0x7f4de806ea10 msgr2=0x7f4de815e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 --2- 172.21.15.67:0/3292345936 >> v2:172.21.15.86:3300/0 conn(0x7f4de806ea10 0x7f4de815e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 -- 172.21.15.67:0/3292345936 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de8071e90 con 0x7f4de8071360 2024-04-17T10:35:25.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.828+0000 7f4dde000640 1 --2- 172.21.15.67:0/3292345936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de806e4d0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f4de8072690 tx=0x7f4dd001e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48805854a27af76 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:25.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4de6000640 1 -- 172.21.15.67:0/3292345936 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd00b4070 con 0x7f4de8071360 2024-04-17T10:35:25.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4de6000640 1 -- 172.21.15.67:0/3292345936 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd00a78d0 con 0x7f4de8071360 2024-04-17T10:35:25.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4de6000640 1 -- 172.21.15.67:0/3292345936 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd00a7ca0 con 0x7f4de8071360 2024-04-17T10:35:25.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- 172.21.15.67:0/3292345936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 msgr2=0x7f4de806e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:25.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 --2- 172.21.15.67:0/3292345936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de806e4d0 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f4de8072690 tx=0x7f4dd001e930 comp rx=0 tx=0).stop 2024-04-17T10:35:25.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- 172.21.15.67:0/3292345936 shutdown_connections 2024-04-17T10:35:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 --2- 172.21.15.67:0/3292345936 >> v2:172.21.15.86:3300/0 conn(0x7f4de806ea10 0x7f4de815e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 --2- 172.21.15.67:0/3292345936 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de806e4d0 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 --2- 172.21.15.67:0/3292345936 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de8144100 0x7f4de80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- 172.21.15.67:0/3292345936 >> 172.21.15.67:0/3292345936 conn(0x7f4de806ac90 msgr2=0x7f4de806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- 172.21.15.67:0/3292345936 shutdown_connections 2024-04-17T10:35:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- 172.21.15.67:0/3292345936 wait complete. 2024-04-17T10:35:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 Processor -- start 2024-04-17T10:35:25.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- start start 2024-04-17T10:35:25.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de806ea10 0x7f4de81f9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:25.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de8140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:25.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4de8144100 0x7f4de80b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de806d540 con 0x7f4de8071360 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de806d3c0 con 0x7f4de8144100 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4ded600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de806d6c0 con 0x7f4de806ea10 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4dde000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de8140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4dde000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de8140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56400/0 (socket says 172.21.15.67:56400) 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4dde000640 1 -- 172.21.15.67:0/938728890 learned_addr learned my addr 172.21.15.67:0/938728890 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4de6a00640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de806ea10 0x7f4de81f9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.832+0000 7f4de7400640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4de8144100 0x7f4de80b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4dde000640 1 -- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de806ea10 msgr2=0x7f4de81f9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4dde000640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de806ea10 0x7f4de81f9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4dde000640 1 -- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4de8144100 msgr2=0x7f4de80b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4dde000640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4de8144100 0x7f4de80b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:25.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4dde000640 1 -- 172.21.15.67:0/938728890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de81fa200 con 0x7f4de8071360 2024-04-17T10:35:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4dde000640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de8140f90 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f4dd001cfe0 tx=0x7f4dd001b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ddf400640 1 -- 172.21.15.67:0/938728890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd00b4070 con 0x7f4de8071360 2024-04-17T10:35:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ddf400640 1 -- 172.21.15.67:0/938728890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd00b6070 con 0x7f4de8071360 2024-04-17T10:35:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ddf400640 1 -- 172.21.15.67:0/938728890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd00afaf0 con 0x7f4de8071360 2024-04-17T10:35:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de8141a10 con 0x7f4de8071360 2024-04-17T10:35:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de8141fd0 con 0x7f4de8071360 2024-04-17T10:35:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de8072da0 con 0x7f4de8071360 2024-04-17T10:35:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ddf400640 1 -- 172.21.15.67:0/938728890 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4dd00afc90 con 0x7f4de8071360 2024-04-17T10:35:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ddf400640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc4066f90 0x7f4dc4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4ddf400640 1 -- 172.21.15.67:0/938728890 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 6273+0+0 (secure 0 0 0) 0x7f4dd011f230 con 0x7f4de8071360 2024-04-17T10:35:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.836+0000 7f4de6a00640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc4066f90 0x7f4dc4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.840+0000 7f4de6a00640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc4066f90 0x7f4dc4069450 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009a40 tx=0x7f4dd8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:25.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:25.840+0000 7f4ddf400640 1 -- 172.21.15.67:0/938728890 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4dd00187e0 con 0x7f4de8071360 2024-04-17T10:35:25.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:26.004+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4de813bac0 con 0x7f4de8071360 2024-04-17T10:35:27.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.008+0000 7f4ddf400640 1 -- 172.21.15.67:0/938728890 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v133) v1 ==== 181+0+0 (secure 0 0 0) 0x7f4dd00efac0 con 0x7f4de8071360 2024-04-17T10:35:27.002 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-04-17T10:35:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc4066f90 msgr2=0x7f4dc4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc4066f90 0x7f4dc4069450 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8009a40 tx=0x7f4dd8009190 comp rx=0 tx=0).stop 2024-04-17T10:35:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 msgr2=0x7f4de8140f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de8140f90 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f4dd001cfe0 tx=0x7f4dd001b6d0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 shutdown_connections 2024-04-17T10:35:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc4066f90 0x7f4dc4069450 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4de8144100 0x7f4de80b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de8071360 0x7f4de8140f90 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 --2- 172.21.15.67:0/938728890 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de806ea10 0x7f4de81f9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 >> 172.21.15.67:0/938728890 conn(0x7f4de806ac90 msgr2=0x7f4de80afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:27.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 shutdown_connections 2024-04-17T10:35:27.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.016+0000 7f4ded600640 1 -- 172.21.15.67:0/938728890 wait complete. 2024-04-17T10:35:27.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-17T10:35:27.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbff0c00640 1 Processor -- start 2024-04-17T10:35:27.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbff0c00640 1 -- start start 2024-04-17T10:35:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbff0c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbfec0cfc20 0x7fbfec070da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbff0c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec071430 con 0x7fbfec15dfa0 2024-04-17T10:35:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbff0c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfec071710 con 0x7fbfec15b670 2024-04-17T10:35:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbff0c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfec06c4a0 con 0x7fbfec0cfc20 2024-04-17T10:35:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfe2a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbfec15dfa0 0x7fbfec0cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48842/0 (socket says 172.21.15.67:48842) 2024-04-17T10:35:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfe2a00640 1 -- 172.21.15.67:0/954360096 learned_addr learned my addr 172.21.15.67:0/954360096 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:27.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfebe00640 1 --2- 172.21.15.67:0/954360096 >> v2:172.21.15.86:3300/0 conn(0x7fbfec0cfc20 0x7fbfec070da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfeaa00640 1 -- 172.21.15.67:0/954360096 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558901204 0 0) 0x7fbfec071430 con 0x7fbfec15dfa0 2024-04-17T10:35:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfeaa00640 1 -- 172.21.15.67:0/954360096 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfd0003530 con 0x7fbfec15dfa0 2024-04-17T10:35:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfebe00640 1 -- 172.21.15.67:0/954360096 >> v1:172.21.15.86:6789/0 conn(0x7fbfec15b670 legacy=0x7fbfec15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfeaa00640 1 -- 172.21.15.67:0/954360096 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399742417 0 0) 0x7fbfec071710 con 0x7fbfec15b670 2024-04-17T10:35:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfebe00640 1 -- 172.21.15.67:0/954360096 >> v1:172.21.15.67:6789/0 conn(0x7fbfec15dfa0 legacy=0x7fbfec0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:27.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.196+0000 7fbfebe00640 1 -- 172.21.15.67:0/954360096 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0719f0 con 0x7fbfec0cfc20 2024-04-17T10:35:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbfebe00640 1 --2- 172.21.15.67:0/954360096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec0cfc20 0x7fbfec070da0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbfec06c620 tx=0x7fbfe401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=786dfa0e96c17777 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbfeaa00640 1 -- 172.21.15.67:0/954360096 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40b1070 con 0x7fbfec0cfc20 2024-04-17T10:35:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbfeaa00640 1 -- 172.21.15.67:0/954360096 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe40ac410 con 0x7fbfec0cfc20 2024-04-17T10:35:27.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbfeaa00640 1 -- 172.21.15.67:0/954360096 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40ac7c0 con 0x7fbfec0cfc20 2024-04-17T10:35:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbff0c00640 1 -- 172.21.15.67:0/954360096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec0cfc20 msgr2=0x7fbfec070da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbff0c00640 1 --2- 172.21.15.67:0/954360096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec0cfc20 0x7fbfec070da0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fbfec06c620 tx=0x7fbfe401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbff0c00640 1 -- 172.21.15.67:0/954360096 shutdown_connections 2024-04-17T10:35:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbff0c00640 1 --2- 172.21.15.67:0/954360096 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec0cfc20 0x7fbfec070da0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbff0c00640 1 -- 172.21.15.67:0/954360096 >> 172.21.15.67:0/954360096 conn(0x7fbfec06f5b0 msgr2=0x7fbfec06f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbff0c00640 1 -- 172.21.15.67:0/954360096 shutdown_connections 2024-04-17T10:35:27.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.200+0000 7fbff0c00640 1 -- 172.21.15.67:0/954360096 wait complete. 2024-04-17T10:35:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 Processor -- start 2024-04-17T10:35:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 -- start start 2024-04-17T10:35:27.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfec0cfc20 0x7fbfec0b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec15b670 0x7fbfec154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfec15dfa0 0x7fbfec0b6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec16bc60 con 0x7fbfec15dfa0 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec16bae0 con 0x7fbfec15b670 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec16bde0 con 0x7fbfec0cfc20 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfec15dfa0 0x7fbfec0b6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfec15dfa0 0x7fbfec0b6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56416/0 (socket says 172.21.15.67:56416) 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 -- 172.21.15.67:0/680649673 learned_addr learned my addr 172.21.15.67:0/680649673 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfe2a00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec15b670 0x7fbfec154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfeb400640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfec0cfc20 0x7fbfec0b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 -- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfec0cfc20 msgr2=0x7fbfec0b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:27.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfec0cfc20 0x7fbfec0b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 -- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec15b670 msgr2=0x7fbfec154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec15b670 0x7fbfec154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 -- 172.21.15.67:0/680649673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec0b77c0 con 0x7fbfec15dfa0 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfebe00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfec15dfa0 0x7fbfec0b6120 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40190b0 tx=0x7fbfe4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfe3e00640 1 -- 172.21.15.67:0/680649673 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe40b1040 con 0x7fbfec15dfa0 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfe3e00640 1 -- 172.21.15.67:0/680649673 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe40bb070 con 0x7fbfec15dfa0 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfec153300 con 0x7fbfec15dfa0 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbfe3e00640 1 -- 172.21.15.67:0/680649673 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4003620 con 0x7fbfec15dfa0 2024-04-17T10:35:27.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.204+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfec0cba20 con 0x7fbfec15dfa0 2024-04-17T10:35:27.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.208+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfec074dc0 con 0x7fbfec15dfa0 2024-04-17T10:35:27.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.208+0000 7fbfe3e00640 1 -- 172.21.15.67:0/680649673 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbfe40037c0 con 0x7fbfec15dfa0 2024-04-17T10:35:27.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.208+0000 7fbfe3e00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbfc8066f60 0x7fbfc8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:27.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.208+0000 7fbfe3e00640 1 -- 172.21.15.67:0/680649673 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5835+0+0 (secure 0 0 0) 0x7fbfe411e7a0 con 0x7fbfec15dfa0 2024-04-17T10:35:27.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.208+0000 7fbfeb400640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbfc8066f60 0x7fbfc8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:27.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.208+0000 7fbfeb400640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbfc8066f60 0x7fbfc8069420 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc000f70 tx=0x7fbfdc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:27.204 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.212+0000 7fbfe3e00640 1 -- 172.21.15.67:0/680649673 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbfe4016090 con 0x7fbfec15dfa0 2024-04-17T10:35:27.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:27.376+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbfec0cef80 con 0x7fbfec15dfa0 2024-04-17T10:35:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.012+0000 7fbfe3e00640 1 -- 172.21.15.67:0/680649673 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v134) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbfe40ef690 con 0x7fbfec15dfa0 2024-04-17T10:35:28.004 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-04-17T10:35:28.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbfc8066f60 msgr2=0x7fbfc8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbfc8066f60 0x7fbfc8069420 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc000f70 tx=0x7fbfdc009040 comp rx=0 tx=0).stop 2024-04-17T10:35:28.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfec15dfa0 msgr2=0x7fbfec0b6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfec15dfa0 0x7fbfec0b6120 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40190b0 tx=0x7fbfe4019240 comp rx=0 tx=0).stop 2024-04-17T10:35:28.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 shutdown_connections 2024-04-17T10:35:28.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbfc8066f60 0x7fbfc8069420 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbfec15dfa0 0x7fbfec0b6120 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbfec15b670 0x7fbfec154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 --2- 172.21.15.67:0/680649673 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbfec0cfc20 0x7fbfec0b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 >> 172.21.15.67:0/680649673 conn(0x7fbfec06f5b0 msgr2=0x7fbfec1580d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 shutdown_connections 2024-04-17T10:35:28.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.016+0000 7fbff0c00640 1 -- 172.21.15.67:0/680649673 wait complete. 2024-04-17T10:35:28.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-17T10:35:28.169 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:35:28.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:35:28.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab82e00640 1 Processor -- start 2024-04-17T10:35:28.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab82e00640 1 -- start start 2024-04-17T10:35:28.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab82e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab7c0aef60 0x7fab7c0cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab82e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c0b4990 con 0x7fab7c06e980 2024-04-17T10:35:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab82e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c15ae90 con 0x7fab7c0cce60 2024-04-17T10:35:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab82e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7c0af340 con 0x7fab7c0aef60 2024-04-17T10:35:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab7be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab7c0aef60 0x7fab7c0cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab80e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fab7c06e980 0x7fab7c0b3120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48846/0 (socket says 172.21.15.67:48846) 2024-04-17T10:35:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab7be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fab7c0aef60 0x7fab7c0cc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56426/0 (socket says 172.21.15.67:56426) 2024-04-17T10:35:28.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab80e00640 1 -- 172.21.15.67:0/2143392051 learned_addr learned my addr 172.21.15.67:0/2143392051 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab7be00640 1 -- 172.21.15.67:0/2143392051 >> v1:172.21.15.86:6789/0 conn(0x7fab7c0cce60 legacy=0x7fab7c06e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab7be00640 1 -- 172.21.15.67:0/2143392051 >> v1:172.21.15.67:6789/0 conn(0x7fab7c06e980 legacy=0x7fab7c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.228+0000 7fab7be00640 1 -- 172.21.15.67:0/2143392051 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c06ee80 con 0x7fab7c0aef60 2024-04-17T10:35:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab7be00640 1 --2- 172.21.15.67:0/2143392051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0aef60 0x7fab7c0cc920 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0097c0 tx=0x7fab6c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bff65da08f3c673 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab7aa00640 1 -- 172.21.15.67:0/2143392051 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0b0070 con 0x7fab7c0aef60 2024-04-17T10:35:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab7aa00640 1 -- 172.21.15.67:0/2143392051 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c0ab600 con 0x7fab7c0aef60 2024-04-17T10:35:28.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab7aa00640 1 -- 172.21.15.67:0/2143392051 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0ab9d0 con 0x7fab7c0aef60 2024-04-17T10:35:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab82e00640 1 -- 172.21.15.67:0/2143392051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0aef60 msgr2=0x7fab7c0cc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab82e00640 1 --2- 172.21.15.67:0/2143392051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0aef60 0x7fab7c0cc920 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0097c0 tx=0x7fab6c01c410 comp rx=0 tx=0).stop 2024-04-17T10:35:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab82e00640 1 -- 172.21.15.67:0/2143392051 shutdown_connections 2024-04-17T10:35:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab82e00640 1 --2- 172.21.15.67:0/2143392051 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0aef60 0x7fab7c0cc920 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab82e00640 1 -- 172.21.15.67:0/2143392051 >> 172.21.15.67:0/2143392051 conn(0x7fab7c0b1740 msgr2=0x7fab7c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab82e00640 1 -- 172.21.15.67:0/2143392051 shutdown_connections 2024-04-17T10:35:28.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.232+0000 7fab82e00640 1 -- 172.21.15.67:0/2143392051 wait complete. 2024-04-17T10:35:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 Processor -- start 2024-04-17T10:35:28.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 -- start start 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c06e980 0x7fab7c13fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c0aef60 0x7fab7c1f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0cce60 0x7fab7c1402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c06d2e0 con 0x7fab7c0cce60 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c06d160 con 0x7fab7c06e980 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c06d460 con 0x7fab7c0aef60 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0cce60 0x7fab7c1402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0cce60 0x7fab7c1402f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56432/0 (socket says 172.21.15.67:56432) 2024-04-17T10:35:28.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 -- 172.21.15.67:0/2044508068 learned_addr learned my addr 172.21.15.67:0/2044508068 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab7be00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c06e980 0x7fab7c13fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab7b400640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c0aef60 0x7fab7c1f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 -- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c0aef60 msgr2=0x7fab7c1f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c0aef60 0x7fab7c1f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 -- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c06e980 msgr2=0x7fab7c13fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c06e980 0x7fab7c13fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 -- 172.21.15.67:0/2044508068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c1529a0 con 0x7fab7c0cce60 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab80e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0cce60 0x7fab7c1402f0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fab7400cbc0 tx=0x7fab7400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab73e00640 1 -- 172.21.15.67:0/2044508068 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74017070 con 0x7fab7c0cce60 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab73e00640 1 -- 172.21.15.67:0/2044508068 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7400ad40 con 0x7fab7c0cce60 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7c152c80 con 0x7fab7c0cce60 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab73e00640 1 -- 172.21.15.67:0/2044508068 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74012600 con 0x7fab7c0cce60 2024-04-17T10:35:28.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7c153240 con 0x7fab7c0cce60 2024-04-17T10:35:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.236+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7c04ef30 con 0x7fab7c0cce60 2024-04-17T10:35:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.240+0000 7fab73e00640 1 -- 172.21.15.67:0/2044508068 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fab7401a070 con 0x7fab7c0cce60 2024-04-17T10:35:28.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.240+0000 7fab73e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fab5c066f60 0x7fab5c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.240+0000 7fab73e00640 1 -- 172.21.15.67:0/2044508068 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fab74084b30 con 0x7fab7c0cce60 2024-04-17T10:35:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.240+0000 7fab7be00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fab5c066f60 0x7fab5c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.240+0000 7fab7be00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fab5c066f60 0x7fab5c069420 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0ad000 tx=0x7fab6c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:28.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.244+0000 7fab73e00640 1 -- 172.21.15.67:0/2044508068 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fab7c0b4990 con 0x7fab7c0cce60 2024-04-17T10:35:28.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.416+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fab7c14ca10 con 0x7fab7c0cce60 2024-04-17T10:35:28.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.420+0000 7fab73e00640 1 -- 172.21.15.67:0/2044508068 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fab74055680 con 0x7fab7c0cce60 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fab5c066f60 msgr2=0x7fab5c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fab5c066f60 0x7fab5c069420 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0ad000 tx=0x7fab6c008040 comp rx=0 tx=0).stop 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0cce60 msgr2=0x7fab7c1402f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0cce60 0x7fab7c1402f0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fab7400cbc0 tx=0x7fab7400a430 comp rx=0 tx=0).stop 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 shutdown_connections 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fab5c066f60 0x7fab5c069420 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab7c0cce60 0x7fab7c1402f0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fab7c0aef60 0x7fab7c1f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 --2- 172.21.15.67:0/2044508068 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab7c06e980 0x7fab7c13fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 >> 172.21.15.67:0/2044508068 conn(0x7fab7c0b1740 msgr2=0x7fab7c0b0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:28.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 shutdown_connections 2024-04-17T10:35:28.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.424+0000 7fab82e00640 1 -- 172.21.15.67:0/2044508068 wait complete. 2024-04-17T10:35:28.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover' 2024-04-17T10:35:28.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 Processor -- start 2024-04-17T10:35:28.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 -- start start 2024-04-17T10:35:28.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd05415e440 0x7fd0540b2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0540ce2a0 0x7fd0540cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd05415e820 0x7fd05406e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0540cbc40 con 0x7fd0540ce2a0 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0540ce930 con 0x7fd05415e440 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd058c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0540cb310 con 0x7fd05415e820 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0540ce2a0 0x7fd0540cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0540ce2a0 0x7fd0540cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56436/0 (socket says 172.21.15.67:56436) 2024-04-17T10:35:28.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 -- 172.21.15.67:0/1795440102 learned_addr learned my addr 172.21.15.67:0/1795440102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd052800640 1 --2- 172.21.15.67:0/1795440102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd05415e820 0x7fd05406e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 -- 172.21.15.67:0/1795440102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd05415e820 msgr2=0x7fd05406e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd051e00640 1 --2- 172.21.15.67:0/1795440102 >> v2:172.21.15.86:3300/0 conn(0x7fd05415e440 0x7fd0540b2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 --2- 172.21.15.67:0/1795440102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd05415e820 0x7fd05406e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 -- 172.21.15.67:0/1795440102 >> v2:172.21.15.86:3300/0 conn(0x7fd05415e440 msgr2=0x7fd0540b2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 --2- 172.21.15.67:0/1795440102 >> v2:172.21.15.86:3300/0 conn(0x7fd05415e440 0x7fd0540b2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd049400640 1 -- 172.21.15.67:0/1795440102 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0540cb5f0 con 0x7fd0540ce2a0 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd052800640 1 --2- 172.21.15.67:0/1795440102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd05415e820 0x7fd05406e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.632+0000 7fd051e00640 1 --2- 172.21.15.67:0/1795440102 >> v2:172.21.15.86:3300/0 conn(0x7fd05415e440 0x7fd0540b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:28.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd049400640 1 --2- 172.21.15.67:0/1795440102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0540ce2a0 0x7fd0540cc0d0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fd0540730a0 tx=0x7fd04401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210f0cf841fd4ce5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd051400640 1 -- 172.21.15.67:0/1795440102 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0440b0070 con 0x7fd0540ce2a0 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd051400640 1 -- 172.21.15.67:0/1795440102 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0440ab600 con 0x7fd0540ce2a0 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd051400640 1 -- 172.21.15.67:0/1795440102 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0440ab9d0 con 0x7fd0540ce2a0 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 -- 172.21.15.67:0/1795440102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0540ce2a0 msgr2=0x7fd0540cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 --2- 172.21.15.67:0/1795440102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0540ce2a0 0x7fd0540cc0d0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fd0540730a0 tx=0x7fd04401c520 comp rx=0 tx=0).stop 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 -- 172.21.15.67:0/1795440102 shutdown_connections 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 --2- 172.21.15.67:0/1795440102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd05415e820 0x7fd05406e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 --2- 172.21.15.67:0/1795440102 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0540ce2a0 0x7fd0540cc0d0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 --2- 172.21.15.67:0/1795440102 >> v2:172.21.15.86:3300/0 conn(0x7fd05415e440 0x7fd0540b2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 -- 172.21.15.67:0/1795440102 >> 172.21.15.67:0/1795440102 conn(0x7fd0540b22a0 msgr2=0x7fd0540b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 -- 172.21.15.67:0/1795440102 shutdown_connections 2024-04-17T10:35:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.636+0000 7fd058c00640 1 -- 172.21.15.67:0/1795440102 wait complete. 2024-04-17T10:35:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 Processor -- start 2024-04-17T10:35:28.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 -- start start 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0540ce2a0 0x7fd05414fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd05415e440 0x7fd05415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05415e820 0x7fd054156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0540cd0e0 con 0x7fd05415e820 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05415ad80 con 0x7fd05415e440 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0540cd260 con 0x7fd0540ce2a0 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05415e820 0x7fd054156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05415e820 0x7fd054156130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56446/0 (socket says 172.21.15.67:56446) 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 -- 172.21.15.67:0/946178277 learned_addr learned my addr 172.21.15.67:0/946178277 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd051e00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0540ce2a0 0x7fd05414fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd049400640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd05415e440 0x7fd05415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 -- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0540ce2a0 msgr2=0x7fd05414fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0540ce2a0 0x7fd05414fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 -- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd05415e440 msgr2=0x7fd05415ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd05415e440 0x7fd05415ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 -- 172.21.15.67:0/946178277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054150320 con 0x7fd05415e820 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd051e00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0540ce2a0 0x7fd05414fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd049400640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd05415e440 0x7fd05415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd052800640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05415e820 0x7fd054156130 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00cd50 tx=0x7fd04c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:28.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd04aa00640 1 -- 172.21.15.67:0/946178277 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c00f680 con 0x7fd05415e820 2024-04-17T10:35:28.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd04aa00640 1 -- 172.21.15.67:0/946178277 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c00f840 con 0x7fd05415e820 2024-04-17T10:35:28.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05415c2c0 con 0x7fd05415e820 2024-04-17T10:35:28.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd04aa00640 1 -- 172.21.15.67:0/946178277 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c00fc30 con 0x7fd05415e820 2024-04-17T10:35:28.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05415c880 con 0x7fd05415e820 2024-04-17T10:35:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.640+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05409e900 con 0x7fd05415e820 2024-04-17T10:35:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.644+0000 7fd04aa00640 1 -- 172.21.15.67:0/946178277 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd04c0213f0 con 0x7fd05415e820 2024-04-17T10:35:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.644+0000 7fd04aa00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd030066f60 0x7fd030069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.644+0000 7fd04aa00640 1 -- 172.21.15.67:0/946178277 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7fd04c085cf0 con 0x7fd05415e820 2024-04-17T10:35:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.644+0000 7fd051e00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd030066f60 0x7fd030069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:28.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.644+0000 7fd051e00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd030066f60 0x7fd030069420 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd03c008930 tx=0x7fd03c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:28.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.648+0000 7fd04aa00640 1 -- 172.21.15.67:0/946178277 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd04c014030 con 0x7fd05415e820 2024-04-17T10:35:28.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:28.868+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]} v 0) v1 -- 0x7fd054070ca0 con 0x7fd05415e820 2024-04-17T10:35:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd04aa00640 1 -- 172.21.15.67:0/946178277 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]}]=0 v506) v1 ==== 166+0+0 (secure 0 0 0) 0x7fd04c056d80 con 0x7fd05415e820 2024-04-17T10:35:29.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd030066f60 msgr2=0x7fd030069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd030066f60 0x7fd030069420 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd03c008930 tx=0x7fd03c008040 comp rx=0 tx=0).stop 2024-04-17T10:35:29.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05415e820 msgr2=0x7fd054156130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05415e820 0x7fd054156130 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00cd50 tx=0x7fd04c00a650 comp rx=0 tx=0).stop 2024-04-17T10:35:29.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 shutdown_connections 2024-04-17T10:35:29.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd030066f60 0x7fd030069420 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05415e820 0x7fd054156130 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd05415e440 0x7fd05415ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 --2- 172.21.15.67:0/946178277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0540ce2a0 0x7fd05414fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 >> 172.21.15.67:0/946178277 conn(0x7fd0540b22a0 msgr2=0x7fd0540cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 shutdown_connections 2024-04-17T10:35:29.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.024+0000 7fd058c00640 1 -- 172.21.15.67:0/946178277 wait complete. 2024-04-17T10:35:29.038 INFO:tasks.cephfs_test_runner:test_scrub_task_status_on_mds_failover (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-04-17T10:35:29.038 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_when_mds_is_inactive (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-04-17T10:35:29.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:29.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8087200640 1 Processor -- start 2024-04-17T10:35:29.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8087200640 1 -- start start 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8087200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8080071840 0x7f8080166a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8087200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80800b6640 con 0x7f8080144aa0 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8087200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80801456d0 con 0x7f80801446c0 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8087200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80800b51a0 con 0x7f8080071840 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f807f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8080144aa0 0x7f80800d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48852/0 (socket says 172.21.15.67:48852) 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f807f400640 1 -- 172.21.15.67:0/2101180647 learned_addr learned my addr 172.21.15.67:0/2101180647 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8085200640 1 --2- 172.21.15.67:0/2101180647 >> v2:172.21.15.67:3300/0 conn(0x7f8080071840 0x7f8080166a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f807ea00640 1 -- 172.21.15.67:0/2101180647 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660259178 0 0) 0x7f80800b6640 con 0x7f8080144aa0 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f807ea00640 1 -- 172.21.15.67:0/2101180647 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8064003530 con 0x7f8080144aa0 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8085200640 1 -- 172.21.15.67:0/2101180647 >> v1:172.21.15.86:6789/0 conn(0x7f80801446c0 legacy=0x7f8080144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8085200640 1 -- 172.21.15.67:0/2101180647 >> v1:172.21.15.67:6789/0 conn(0x7f8080144aa0 legacy=0x7f80800d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.188+0000 7f8085200640 1 -- 172.21.15.67:0/2101180647 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80800b4870 con 0x7f8080071840 2024-04-17T10:35:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8085200640 1 --2- 172.21.15.67:0/2101180647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8080071840 0x7f8080166a30 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f8078009d90 tx=0x7f807801c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4faba59fdd8a26f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f807ea00640 1 -- 172.21.15.67:0/2101180647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780b2070 con 0x7f8080071840 2024-04-17T10:35:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f807ea00640 1 -- 172.21.15.67:0/2101180647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80780a58d0 con 0x7f8080071840 2024-04-17T10:35:29.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f807ea00640 1 -- 172.21.15.67:0/2101180647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80780a5ca0 con 0x7f8080071840 2024-04-17T10:35:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8087200640 1 -- 172.21.15.67:0/2101180647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8080071840 msgr2=0x7f8080166a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8087200640 1 --2- 172.21.15.67:0/2101180647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8080071840 0x7f8080166a30 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f8078009d90 tx=0x7f807801c900 comp rx=0 tx=0).stop 2024-04-17T10:35:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8087200640 1 -- 172.21.15.67:0/2101180647 shutdown_connections 2024-04-17T10:35:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8087200640 1 --2- 172.21.15.67:0/2101180647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8080071840 0x7f8080166a30 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8087200640 1 -- 172.21.15.67:0/2101180647 >> 172.21.15.67:0/2101180647 conn(0x7f80800b34d0 msgr2=0x7f80800b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8087200640 1 -- 172.21.15.67:0/2101180647 shutdown_connections 2024-04-17T10:35:29.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.192+0000 7f8087200640 1 -- 172.21.15.67:0/2101180647 wait complete. 2024-04-17T10:35:29.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 Processor -- start 2024-04-17T10:35:29.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 -- start start 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8080071840 0x7f8080164ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80801446c0 0x7f808015e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8080144aa0 0x7f808015eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8080169910 con 0x7f80801446c0 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8080169790 con 0x7f8080071840 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8080169a90 con 0x7f8080144aa0 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80801446c0 0x7f808015e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8080071840 0x7f8080164ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80801446c0 0x7f808015e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56470/0 (socket says 172.21.15.67:56470) 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 -- 172.21.15.67:0/403574169 learned_addr learned my addr 172.21.15.67:0/403574169 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8085200640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8080144aa0 0x7f808015eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 -- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8080144aa0 msgr2=0x7f808015eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8080144aa0 0x7f808015eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 -- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8080071840 msgr2=0x7f8080164ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8080071840 0x7f8080164ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 -- 172.21.15.67:0/403574169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8080165010 con 0x7f80801446c0 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f807f400640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80801446c0 0x7f808015e600 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f806c0010f0 tx=0x7f806c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8077e00640 1 -- 172.21.15.67:0/403574169 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c017070 con 0x7f80801446c0 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8077e00640 1 -- 172.21.15.67:0/403574169 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c005d10 con 0x7f80801446c0 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8077e00640 1 -- 172.21.15.67:0/403574169 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806c007310 con 0x7f80801446c0 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8080166a60 con 0x7f80801446c0 2024-04-17T10:35:29.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80801fa8f0 con 0x7f80801446c0 2024-04-17T10:35:29.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.196+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8080070bb0 con 0x7f80801446c0 2024-04-17T10:35:29.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.200+0000 7f8077e00640 1 -- 172.21.15.67:0/403574169 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f806c01a070 con 0x7f80801446c0 2024-04-17T10:35:29.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.200+0000 7f8077e00640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8060066f60 0x7f8060069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.200+0000 7f8077e00640 1 -- 172.21.15.67:0/403574169 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f806c084260 con 0x7f80801446c0 2024-04-17T10:35:29.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.200+0000 7f807fe00640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8060066f60 0x7f8060069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.200+0000 7f807fe00640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8060066f60 0x7f8060069420 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f80800b09e0 tx=0x7f8070007e40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.204+0000 7f8077e00640 1 -- 172.21.15.67:0/403574169 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f806c054d30 con 0x7f80801446c0 2024-04-17T10:35:29.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80800b41a0 con 0x7f80801446c0 2024-04-17T10:35:29.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8077e00640 1 -- 172.21.15.67:0/403574169 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v134) v1 ==== 74+0+13383 (secure 0 0 0) 0x7f806c058dd0 con 0x7f80801446c0 2024-04-17T10:35:29.361 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:29.361 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":134,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:28.014660+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6835/1493271033":"2024-04-18T10:35:24.613806+0000","172.21.15.67:6833/1493271033":"2024-04-18T10:35:24.613806+0000","172.21.15.86:6836/538421223":"2024-04-18T10:35:13.952580+0000","172.21.15.86:6834/538421223":"2024-04-18T10:35:13.952580+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8060066f60 msgr2=0x7f8060069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8060066f60 0x7f8060069420 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f80800b09e0 tx=0x7f8070007e40 comp rx=0 tx=0).stop 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80801446c0 msgr2=0x7f808015e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80801446c0 0x7f808015e600 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f806c0010f0 tx=0x7f806c00c890 comp rx=0 tx=0).stop 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 shutdown_connections 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8060066f60 0x7f8060069420 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8080144aa0 0x7f808015eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f80801446c0 0x7f808015e600 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 --2- 172.21.15.67:0/403574169 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8080071840 0x7f8080164ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.368+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 >> 172.21.15.67:0/403574169 conn(0x7f80800b34d0 msgr2=0x7f808015c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.372+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 shutdown_connections 2024-04-17T10:35:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.372+0000 7f8087200640 1 -- 172.21.15.67:0/403574169 wait complete. 2024-04-17T10:35:29.382 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:35:29.382 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:35:29.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.532+0000 7f97a0e00640 1 Processor -- start 2024-04-17T10:35:29.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.532+0000 7f97a0e00640 1 -- start start 2024-04-17T10:35:29.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.532+0000 7f97a0e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 0x7f979c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.532+0000 7f97a0e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f979c0713e0 0x7f979c071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f97a0e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c15cb30 con 0x7f979c15be70 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f97a0e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c0a8650 con 0x7f979c0713e0 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f97a0e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979c071d80 con 0x7f979c159440 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f979a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f979c15be70 0x7f979c15c2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:48854/0 (socket says 172.21.15.67:48854) 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f979a800640 1 -- 172.21.15.67:0/2350902479 learned_addr learned my addr 172.21.15.67:0/2350902479 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9791400640 1 --2- 172.21.15.67:0/2350902479 >> v2:172.21.15.86:3300/0 conn(0x7f979c0713e0 0x7f979c071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799e00640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 0x7f979c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286502586 0 0) 0x7f979c15cb30 con 0x7f979c15be70 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9780003530 con 0x7f979c15be70 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1878490718 0 0) 0x7f9780003530 con 0x7f979c15be70 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 msgr2=0x7f979c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 0x7f979c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 >> v2:172.21.15.86:3300/0 conn(0x7f979c0713e0 msgr2=0x7f979c071840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 --2- 172.21.15.67:0/2350902479 >> v2:172.21.15.86:3300/0 conn(0x7f979c0713e0 0x7f979c071840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c0cd610 con 0x7f979c15be70 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f9794001a00 con 0x7f979c15be70 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 >> v1:172.21.15.67:6789/0 conn(0x7f979c15be70 legacy=0x7f979c15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800039c0 0x7f9780003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9780004810 0x7f9780004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 0x7f97800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978000b9d0 con 0x7f9780004810 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978000bb50 con 0x7f97800039c0 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f978000bcd0 con 0x7f979c159440 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9791400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9780004810 0x7f9780004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f979a800640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 0x7f97800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9799e00640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800039c0 0x7f9780003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9791400640 1 -- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 msgr2=0x7f97800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9791400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f979c159440 0x7f97800051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9791400640 1 -- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800039c0 msgr2=0x7f9780003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9791400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f97800039c0 0x7f9780003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.536+0000 7f9791400640 1 -- 172.21.15.67:0/2350902479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9780003530 con 0x7f9780004810 2024-04-17T10:35:29.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f9791400640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9780004810 0x7f9780004c90 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f9784009ae0 tx=0x7f978401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8da9b614cce72d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840b2070 con 0x7f9780004810 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97840a55c0 con 0x7f9780004810 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f9799400640 1 -- 172.21.15.67:0/2350902479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840096b0 con 0x7f9780004810 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 -- 172.21.15.67:0/2350902479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9780004810 msgr2=0x7f9780004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 --2- 172.21.15.67:0/2350902479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9780004810 0x7f9780004c90 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f9784009ae0 tx=0x7f978401cab0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f979a800640 1 -- 172.21.15.67:0/2350902479 reap_dead start 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 -- 172.21.15.67:0/2350902479 shutdown_connections 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 -- 172.21.15.67:0/2350902479 >> 172.21.15.67:0/2350902479 conn(0x7f979c0b1740 msgr2=0x7f979c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 -- 172.21.15.67:0/2350902479 shutdown_connections 2024-04-17T10:35:29.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 -- 172.21.15.67:0/2350902479 wait complete. 2024-04-17T10:35:29.533 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 Processor -- start 2024-04-17T10:35:29.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.540+0000 7f97a0e00640 1 -- start start 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9780004810 0x7f979c1f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f979c0c47e0 0x7f979c14a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979c0c6410 0x7f979c159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c0cdbd0 con 0x7f979c0c47e0 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c15ce10 con 0x7f979c0c6410 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979c159820 con 0x7f9780004810 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f979c0c47e0 0x7f979c14a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f979c0c47e0 0x7f979c14a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56492/0 (socket says 172.21.15.67:56492) 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 -- 172.21.15.67:0/1727994370 learned_addr learned my addr 172.21.15.67:0/1727994370 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9799e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9780004810 0x7f979c1f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f979a800640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979c0c6410 0x7f979c159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 -- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9780004810 msgr2=0x7f979c1f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9780004810 0x7f979c1f1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 -- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979c0c6410 msgr2=0x7f979c159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979c0c6410 0x7f979c159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 -- 172.21.15.67:0/1727994370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c0c57d0 con 0x7f979c0c47e0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9791400640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f979c0c47e0 0x7f979c14a1b0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9784009ae0 tx=0x7f9780005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9792a00640 1 -- 172.21.15.67:0/1727994370 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840b2040 con 0x7f979c0c47e0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9792a00640 1 -- 172.21.15.67:0/1727994370 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978400f000 con 0x7f979c0c47e0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9792a00640 1 -- 172.21.15.67:0/1727994370 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97840096b0 con 0x7f979c0c47e0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c0c4c60 con 0x7f979c0c47e0 2024-04-17T10:35:29.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c0c5210 con 0x7f979c0c47e0 2024-04-17T10:35:29.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c04ef30 con 0x7f979c0c47e0 2024-04-17T10:35:29.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.544+0000 7f9792a00640 1 -- 172.21.15.67:0/1727994370 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f97840c0030 con 0x7f979c0c47e0 2024-04-17T10:35:29.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.548+0000 7f9792a00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9778066f60 0x7f9778069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.548+0000 7f9792a00640 1 -- 172.21.15.67:0/1727994370 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f978411ea20 con 0x7f979c0c47e0 2024-04-17T10:35:29.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.548+0000 7f9799e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9778066f60 0x7f9778069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.548+0000 7f9799e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9778066f60 0x7f9778069420 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f978c009a80 tx=0x7f978c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.552+0000 7f9792a00640 1 -- 172.21.15.67:0/1727994370 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f97840c7060 con 0x7f979c0c47e0 2024-04-17T10:35:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.712+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f979c15b9f0 con 0x7f979c0c47e0 2024-04-17T10:35:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.712+0000 7f9792a00640 1 -- 172.21.15.67:0/1727994370 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v134) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97840167c0 con 0x7f979c0c47e0 2024-04-17T10:35:29.706 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:35:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9778066f60 msgr2=0x7f9778069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9778066f60 0x7f9778069420 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f978c009a80 tx=0x7f978c009190 comp rx=0 tx=0).stop 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f979c0c47e0 msgr2=0x7f979c14a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f979c0c47e0 0x7f979c14a1b0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f9784009ae0 tx=0x7f9780005840 comp rx=0 tx=0).stop 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 shutdown_connections 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9778066f60 0x7f9778069420 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f979c0c6410 0x7f979c159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f979c0c47e0 0x7f979c14a1b0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 --2- 172.21.15.67:0/1727994370 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9780004810 0x7f979c1f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 >> 172.21.15.67:0/1727994370 conn(0x7f979c0b1740 msgr2=0x7f979c15ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 shutdown_connections 2024-04-17T10:35:29.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.716+0000 7f97a0e00640 1 -- 172.21.15.67:0/1727994370 wait complete. 2024-04-17T10:35:29.726 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive' 2024-04-17T10:35:29.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 Processor -- start 2024-04-17T10:35:29.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 -- start start 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f726c06eac0 0x7f726c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f726c143ae0 0x7f726c143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f726c0cf430 con 0x7f726c143ae0 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f726c15ae40 con 0x7f726c06eac0 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7273600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c071440 con 0x7f726c06e6e0 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f726c143ae0 0x7f726c143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f726c143ae0 0x7f726c143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56498/0 (socket says 172.21.15.67:56498) 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 -- 172.21.15.67:0/334688084 learned_addr learned my addr 172.21.15.67:0/334688084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7263e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f726c06eac0 0x7f726c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7270c00640 1 --2- 172.21.15.67:0/334688084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 -- 172.21.15.67:0/334688084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 msgr2=0x7f726c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 --2- 172.21.15.67:0/334688084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 -- 172.21.15.67:0/334688084 >> v2:172.21.15.86:3300/0 conn(0x7f726c06eac0 msgr2=0x7f726c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 --2- 172.21.15.67:0/334688084 >> v2:172.21.15.86:3300/0 conn(0x7f726c06eac0 0x7f726c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.876+0000 7f7271600640 1 -- 172.21.15.67:0/334688084 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c071720 con 0x7f726c143ae0 2024-04-17T10:35:29.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7271600640 1 --2- 172.21.15.67:0/334688084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c143f70 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f726c06bac0 tx=0x7f726401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6f3c2697f17446 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f726be00640 1 -- 172.21.15.67:0/334688084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72640a57b0 con 0x7f726c143ae0 2024-04-17T10:35:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f726be00640 1 -- 172.21.15.67:0/334688084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72640a5950 con 0x7f726c143ae0 2024-04-17T10:35:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f726be00640 1 -- 172.21.15.67:0/334688084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72640a5d20 con 0x7f726c143ae0 2024-04-17T10:35:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 -- 172.21.15.67:0/334688084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 msgr2=0x7f726c143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 --2- 172.21.15.67:0/334688084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c143f70 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f726c06bac0 tx=0x7f726401c930 comp rx=0 tx=0).stop 2024-04-17T10:35:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 -- 172.21.15.67:0/334688084 shutdown_connections 2024-04-17T10:35:29.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 --2- 172.21.15.67:0/334688084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c143f70 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 --2- 172.21.15.67:0/334688084 >> v2:172.21.15.86:3300/0 conn(0x7f726c06eac0 0x7f726c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 --2- 172.21.15.67:0/334688084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 -- 172.21.15.67:0/334688084 >> 172.21.15.67:0/334688084 conn(0x7f726c06ac90 msgr2=0x7f726c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 -- 172.21.15.67:0/334688084 shutdown_connections 2024-04-17T10:35:29.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.880+0000 7f7273600640 1 -- 172.21.15.67:0/334688084 wait complete. 2024-04-17T10:35:29.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 Processor -- start 2024-04-17T10:35:29.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 -- start start 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c1589a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726c06eac0 0x7f726c0afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c13cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c15ebf0 con 0x7f726c143ae0 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726c1444b0 con 0x7f726c06eac0 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726c15ed70 con 0x7f726c06e6e0 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c13cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c13cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56504/0 (socket says 172.21.15.67:56504) 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 -- 172.21.15.67:0/2173310173 learned_addr learned my addr 172.21.15.67:0/2173310173 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:29.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7263e00640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726c06eac0 0x7f726c0afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7270c00640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c1589a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 -- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 msgr2=0x7f726c1589a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c1589a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 -- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726c06eac0 msgr2=0x7f726c0afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726c06eac0 0x7f726c0afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 -- 172.21.15.67:0/2173310173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c159080 con 0x7f726c143ae0 2024-04-17T10:35:29.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7271600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c13cf30 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f72640af040 tx=0x7f72640196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7269600640 1 -- 172.21.15.67:0/2173310173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72640bc020 con 0x7f726c143ae0 2024-04-17T10:35:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7269600640 1 -- 172.21.15.67:0/2173310173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72640b8670 con 0x7f726c143ae0 2024-04-17T10:35:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7269600640 1 -- 172.21.15.67:0/2173310173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7264009510 con 0x7f726c143ae0 2024-04-17T10:35:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c159360 con 0x7f726c143ae0 2024-04-17T10:35:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.884+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c159920 con 0x7f726c143ae0 2024-04-17T10:35:29.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.888+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726c072d50 con 0x7f726c143ae0 2024-04-17T10:35:29.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.888+0000 7f7269600640 1 -- 172.21.15.67:0/2173310173 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f72640bf020 con 0x7f726c143ae0 2024-04-17T10:35:29.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.888+0000 7f7269600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7248066f60 0x7f7248069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:29.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.888+0000 7f7269600640 1 -- 172.21.15.67:0/2173310173 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f726411fe70 con 0x7f726c143ae0 2024-04-17T10:35:29.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.888+0000 7f7270c00640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7248066f60 0x7f7248069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:29.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.888+0000 7f7270c00640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7248066f60 0x7f7248069420 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f725c009b60 tx=0x7f725c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:29.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:29.892+0000 7f7269600640 1 -- 172.21.15.67:0/2173310173 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f72640c6030 con 0x7f726c143ae0 2024-04-17T10:35:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:30.112+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive"]} v 0) v1 -- 0x7f726c0af830 con 0x7f726c143ae0 2024-04-17T10:35:31.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7269600640 1 -- 172.21.15.67:0/2173310173 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive"]}]=0 v508) v1 ==== 162+0+0 (secure 0 0 0) 0x7f72640165e0 con 0x7f726c143ae0 2024-04-17T10:35:31.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7248066f60 msgr2=0x7f7248069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7248066f60 0x7f7248069420 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f725c009b60 tx=0x7f725c009190 comp rx=0 tx=0).stop 2024-04-17T10:35:31.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 msgr2=0x7f726c13cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c13cf30 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f72640af040 tx=0x7f72640196d0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 shutdown_connections 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7248066f60 0x7f7248069420 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f726c143ae0 0x7f726c13cf30 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f726c06eac0 0x7f726c0afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 --2- 172.21.15.67:0/2173310173 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f726c06e6e0 0x7f726c1589a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 >> 172.21.15.67:0/2173310173 conn(0x7f726c06ac90 msgr2=0x7f726c06b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 shutdown_connections 2024-04-17T10:35:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.036+0000 7f7273600640 1 -- 172.21.15.67:0/2173310173 wait complete. 2024-04-17T10:35:31.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-17T10:35:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f266ce00640 1 Processor -- start 2024-04-17T10:35:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f266ce00640 1 -- start start 2024-04-17T10:35:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f266ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f26680cf8a0 0x7f26680ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f266ce00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26680d0040 con 0x7f266815dcf0 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f266ce00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26680d0320 con 0x7f26680ce5d0 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f266ce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26680d0600 con 0x7f26680cf8a0 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2665e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f266815dcf0 0x7f266806cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59932/0 (socket says 172.21.15.67:59932) 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2665e00640 1 -- 172.21.15.67:0/2533998657 learned_addr learned my addr 172.21.15.67:0/2533998657 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2665400640 1 --2- 172.21.15.67:0/2533998657 >> v2:172.21.15.67:3300/0 conn(0x7f26680cf8a0 0x7f26680ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2664a00640 1 -- 172.21.15.67:0/2533998657 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660792641 0 0) 0x7f26680d0040 con 0x7f266815dcf0 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2664a00640 1 -- 172.21.15.67:0/2533998657 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f264c003530 con 0x7f266815dcf0 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2665400640 1 -- 172.21.15.67:0/2533998657 >> v1:172.21.15.86:6789/0 conn(0x7f26680ce5d0 legacy=0x7f26680b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2665400640 1 -- 172.21.15.67:0/2533998657 >> v1:172.21.15.67:6789/0 conn(0x7f266815dcf0 legacy=0x7f266806cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.208 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.212+0000 7f2665400640 1 -- 172.21.15.67:0/2533998657 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26680d08e0 con 0x7f26680cf8a0 2024-04-17T10:35:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f2665400640 1 --2- 172.21.15.67:0/2533998657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680cf8a0 0x7f26680ce090 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f26540097c0 tx=0x7f265401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918574295b07314f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f2664a00640 1 -- 172.21.15.67:0/2533998657 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26540b0070 con 0x7f26680cf8a0 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f2664a00640 1 -- 172.21.15.67:0/2533998657 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2654004230 con 0x7f26680cf8a0 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f2664a00640 1 -- 172.21.15.67:0/2533998657 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2654003620 con 0x7f26680cf8a0 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f266ce00640 1 -- 172.21.15.67:0/2533998657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680cf8a0 msgr2=0x7f26680ce090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f266ce00640 1 --2- 172.21.15.67:0/2533998657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680cf8a0 0x7f26680ce090 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f26540097c0 tx=0x7f265401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f266ce00640 1 -- 172.21.15.67:0/2533998657 shutdown_connections 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f266ce00640 1 --2- 172.21.15.67:0/2533998657 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680cf8a0 0x7f26680ce090 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f266ce00640 1 -- 172.21.15.67:0/2533998657 >> 172.21.15.67:0/2533998657 conn(0x7f26680b33a0 msgr2=0x7f26680b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f266ce00640 1 -- 172.21.15.67:0/2533998657 shutdown_connections 2024-04-17T10:35:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.216+0000 7f266ce00640 1 -- 172.21.15.67:0/2533998657 wait complete. 2024-04-17T10:35:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 Processor -- start 2024-04-17T10:35:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 -- start start 2024-04-17T10:35:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680ce5d0 0x7f2668160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26680cf8a0 0x7f26680cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f266815dcf0 0x7f26680c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26680b2730 con 0x7f26680ce5d0 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26680b25b0 con 0x7f26680cf8a0 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f266816b740 con 0x7f266815dcf0 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680ce5d0 0x7f2668160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680ce5d0 0x7f2668160630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51856/0 (socket says 172.21.15.67:51856) 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 -- 172.21.15.67:0/2952852798 learned_addr learned my addr 172.21.15.67:0/2952852798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665400640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26680cf8a0 0x7f26680cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2666800640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f266815dcf0 0x7f26680c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 -- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f266815dcf0 msgr2=0x7f26680c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f266815dcf0 0x7f26680c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 -- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26680cf8a0 msgr2=0x7f26680cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26680cf8a0 0x7f26680cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 -- 172.21.15.67:0/2952852798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f266815f590 con 0x7f26680ce5d0 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f2665e00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680ce5d0 0x7f2668160630 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f265800c420 tx=0x7f265800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f265e000640 1 -- 172.21.15.67:0/2952852798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658017070 con 0x7f26680ce5d0 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f265e000640 1 -- 172.21.15.67:0/2952852798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2658005cf0 con 0x7f26680ce5d0 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f266815f870 con 0x7f26680ce5d0 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f265e000640 1 -- 172.21.15.67:0/2952852798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26580072d0 con 0x7f26680ce5d0 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f266815fe30 con 0x7f26680ce5d0 2024-04-17T10:35:31.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.220+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f266804ef30 con 0x7f26680ce5d0 2024-04-17T10:35:31.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.224+0000 7f265e000640 1 -- 172.21.15.67:0/2952852798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f265801a070 con 0x7f26680ce5d0 2024-04-17T10:35:31.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.224+0000 7f265e000640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2648066f60 0x7f2648069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.224+0000 7f265e000640 1 -- 172.21.15.67:0/2952852798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f2658084300 con 0x7f26680ce5d0 2024-04-17T10:35:31.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.224+0000 7f2665400640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2648066f60 0x7f2648069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.224+0000 7f2665400640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2648066f60 0x7f2648069420 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f26540ad000 tx=0x7f26540a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:31.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.228+0000 7f265e000640 1 -- 172.21.15.67:0/2952852798 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2658054d40 con 0x7f26680ce5d0 2024-04-17T10:35:31.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.392+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f26680712b0 con 0x7f26680ce5d0 2024-04-17T10:35:31.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f265e000640 1 -- 172.21.15.67:0/2952852798 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2658058de0 con 0x7f26680ce5d0 2024-04-17T10:35:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2648066f60 msgr2=0x7f2648069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2648066f60 0x7f2648069420 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f26540ad000 tx=0x7f26540a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680ce5d0 msgr2=0x7f2668160630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680ce5d0 0x7f2668160630 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f265800c420 tx=0x7f265800c8e0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 shutdown_connections 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2648066f60 0x7f2648069420 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f266815dcf0 0x7f26680c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26680cf8a0 0x7f26680cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 --2- 172.21.15.67:0/2952852798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26680ce5d0 0x7f2668160630 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 >> 172.21.15.67:0/2952852798 conn(0x7f26680b33a0 msgr2=0x7f266806cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 shutdown_connections 2024-04-17T10:35:31.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.396+0000 7f266ce00640 1 -- 172.21.15.67:0/2952852798 wait complete. 2024-04-17T10:35:31.411 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:35:31.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:35:31.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:35:31.601 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:35:31.602 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:35:31.602 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:31.610 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:35:31.610 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:35:31.611 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:31.620 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:35:31.620 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:35:31.620 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:31.662 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350324k,nr_inodes=4087581,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3577 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274992k,nr_inodes=818748,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274996k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-17T10:35:31.663 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-04-17T10:35:31.663 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:31.672 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=16350304k,nr_inodes=4087576,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=5614 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3274988k,nr_inodes=818747,mode=700,uid=1000,gid=1271 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3274992k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-17T10:35:31.672 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-17T10:35:31.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:31.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.856+0000 7f410cc00640 1 Processor -- start 2024-04-17T10:35:31.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.856+0000 7f410cc00640 1 -- start start 2024-04-17T10:35:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f410cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4108161c10 0x7f410809a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f410cc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41080cc8a0 con 0x7f410815d920 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f410cc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41080ccb80 con 0x7f41080cc110 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f410cc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410809ab30 con 0x7f4108161c10 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f40ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4108161c10 0x7f410809a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f40ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4108161c10 0x7f410809a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51864/0 (socket says 172.21.15.67:51864) 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f4107400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f410815d920 0x7f4108161500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59948/0 (socket says 172.21.15.67:59948) 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f40ffe00640 1 -- 172.21.15.67:0/3754902155 learned_addr learned my addr 172.21.15.67:0/3754902155 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f40ffe00640 1 -- 172.21.15.67:0/3754902155 >> v1:172.21.15.86:6789/0 conn(0x7f41080cc110 legacy=0x7f4108070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f40ffe00640 1 -- 172.21.15.67:0/3754902155 >> v1:172.21.15.67:6789/0 conn(0x7f410815d920 legacy=0x7f4108161500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.860+0000 7f40ffe00640 1 -- 172.21.15.67:0/3754902155 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41080cce60 con 0x7f4108161c10 2024-04-17T10:35:31.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f40ffe00640 1 --2- 172.21.15.67:0/3754902155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f410809a5f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009870 tx=0x7f40f001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af41454d0f573037 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f4106a00640 1 -- 172.21.15.67:0/3754902155 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00ab5f0 con 0x7f4108161c10 2024-04-17T10:35:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f4106a00640 1 -- 172.21.15.67:0/3754902155 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f00b0070 con 0x7f4108161c10 2024-04-17T10:35:31.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f4106a00640 1 -- 172.21.15.67:0/3754902155 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00ab9b0 con 0x7f4108161c10 2024-04-17T10:35:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- 172.21.15.67:0/3754902155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 msgr2=0x7f410809a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 --2- 172.21.15.67:0/3754902155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f410809a5f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009870 tx=0x7f40f001c520 comp rx=0 tx=0).stop 2024-04-17T10:35:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- 172.21.15.67:0/3754902155 shutdown_connections 2024-04-17T10:35:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 --2- 172.21.15.67:0/3754902155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f410809a5f0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- 172.21.15.67:0/3754902155 >> 172.21.15.67:0/3754902155 conn(0x7f41080d1600 msgr2=0x7f41080d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- 172.21.15.67:0/3754902155 shutdown_connections 2024-04-17T10:35:31.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- 172.21.15.67:0/3754902155 wait complete. 2024-04-17T10:35:31.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 Processor -- start 2024-04-17T10:35:31.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- start start 2024-04-17T10:35:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41080cc110 0x7f41081521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410815d920 0x7f41080c6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.859 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f4108157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41080cc520 con 0x7f4108161c10 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108071260 con 0x7f410815d920 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.864+0000 7f410cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41080d04c0 con 0x7f41080cc110 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f4108157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f4108157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51880/0 (socket says 172.21.15.67:51880) 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 -- 172.21.15.67:0/656098695 learned_addr learned my addr 172.21.15.67:0/656098695 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f40ffe00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410815d920 0x7f41080c6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107400640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41080cc110 0x7f41081521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 -- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41080cc110 msgr2=0x7f41081521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41080cc110 0x7f41081521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 -- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410815d920 msgr2=0x7f41080c6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410815d920 0x7f41080c6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:31.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 -- 172.21.15.67:0/656098695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4108152700 con 0x7f4108161c10 2024-04-17T10:35:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f40ffe00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410815d920 0x7f41080c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f4107e00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f4108157610 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f410000b990 tx=0x7f410000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f40ff400640 1 -- 172.21.15.67:0/656098695 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100019070 con 0x7f4108161c10 2024-04-17T10:35:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f40ff400640 1 -- 172.21.15.67:0/656098695 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410000cba0 con 0x7f4108161c10 2024-04-17T10:35:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f40ff400640 1 -- 172.21.15.67:0/656098695 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100014440 con 0x7f4108161c10 2024-04-17T10:35:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41081580d0 con 0x7f4108161c10 2024-04-17T10:35:31.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41080ae4d0 con 0x7f4108161c10 2024-04-17T10:35:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4108074dc0 con 0x7f4108161c10 2024-04-17T10:35:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f40ff400640 1 -- 172.21.15.67:0/656098695 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f410001c070 con 0x7f4108161c10 2024-04-17T10:35:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.868+0000 7f40ff400640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e4066f90 0x7f40e4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:31.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.872+0000 7f40ff400640 1 -- 172.21.15.67:0/656098695 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f4100084cc0 con 0x7f4108161c10 2024-04-17T10:35:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.872+0000 7f4107400640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e4066f90 0x7f40e4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:31.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.872+0000 7f4107400640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e4066f90 0x7f40e4069450 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f40f8000f70 tx=0x7f40f8007cd0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:31.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:31.872+0000 7f40ff400640 1 -- 172.21.15.67:0/656098695 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f410008b080 con 0x7f4108161c10 2024-04-17T10:35:32.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.072+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41081622a0 con 0x7f4108161c10 2024-04-17T10:35:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f40ff400640 1 -- 172.21.15.67:0/656098695 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+2902 (secure 0 0 0) 0x7f4100055d50 con 0x7f4108161c10 2024-04-17T10:35:32.067 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:32.067 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":96,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8617,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/85119230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6835","nonce":85119230},{"type":"v1","addr":"172.21.15.86:6837","nonce":85119230}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9252,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/2568325228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2568325228},{"type":"v1","addr":"172.21.15.67:6834","nonce":2568325228}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85},{"gid":9681,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1282374450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":1282374450},{"type":"v1","addr":"172.21.15.67:6835","nonce":1282374450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":96}],"filesystems":[]} 2024-04-17T10:35:32.067 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 96 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e4066f90 msgr2=0x7f40e4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e4066f90 0x7f40e4069450 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f40f8000f70 tx=0x7f40f8007cd0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 msgr2=0x7f4108157610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f4108157610 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f410000b990 tx=0x7f410000be50 comp rx=0 tx=0).stop 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 shutdown_connections 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e4066f90 0x7f40e4069450 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4108161c10 0x7f4108157610 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410815d920 0x7f41080c6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 --2- 172.21.15.67:0/656098695 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41080cc110 0x7f41081521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 >> 172.21.15.67:0/656098695 conn(0x7f41080d1600 msgr2=0x7f41080b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 shutdown_connections 2024-04-17T10:35:32.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.076+0000 7f410cc00640 1 -- 172.21.15.67:0/656098695 wait complete. 2024-04-17T10:35:32.089 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-17T10:35:32.089 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-17T10:35:32.089 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-17T10:35:32.090 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:35:32.090 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-17T10:35:32.090 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-17T10:35:32.090 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-17T10:35:32.090 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:35:32.090 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-17T10:35:32.090 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-17T10:35:32.091 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-17T10:35:32.091 INFO:teuthology.orchestra.run:waiting for 300 2024-04-17T10:35:32.091 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-17T10:35:32.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-17T10:35:32.094 INFO:tasks.ceph.mds.d:Started 2024-04-17T10:35:32.094 DEBUG:teuthology.parallel:result is None 2024-04-17T10:35:32.102 INFO:tasks.ceph.mds.c:Stopped 2024-04-17T10:35:32.103 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-17T10:35:32.104 INFO:tasks.ceph.mds.a:Stopped 2024-04-17T10:35:32.104 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-17T10:35:32.105 INFO:tasks.ceph.mds.b:Stopped 2024-04-17T10:35:32.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-17T10:35:32.153 INFO:tasks.ceph.mds.b:Started 2024-04-17T10:35:32.153 DEBUG:teuthology.parallel:result is None 2024-04-17T10:35:32.154 INFO:tasks.ceph.mds.c:Started 2024-04-17T10:35:32.154 DEBUG:teuthology.parallel:result is None 2024-04-17T10:35:32.155 INFO:tasks.ceph.mds.a:Started 2024-04-17T10:35:32.155 DEBUG:teuthology.parallel:result is None 2024-04-17T10:35:32.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-17T10:35:32.186 INFO:tasks.ceph.mds.d.smithi086.stdout:starting mds.d at 2024-04-17T10:35:32.242 INFO:tasks.ceph.mds.b.smithi086.stdout:starting mds.b at 2024-04-17T10:35:32.242 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:35:32.243 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2024-04-17T10:35:32.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f71400640 1 Processor -- start 2024-04-17T10:35:32.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f71400640 1 -- start start 2024-04-17T10:35:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f71400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9f6c1446c0 0x7f9f6c144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f71400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c144aa0 0x7f9f6c0d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f71400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6c0b6640 con 0x7f9f6c071840 2024-04-17T10:35:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f71400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f6c0b51a0 con 0x7f9f6c1446c0 2024-04-17T10:35:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f71400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c1456d0 con 0x7f9f6c144aa0 2024-04-17T10:35:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f61e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c144aa0 0x7f9f6c0d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f6a800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9f6c1446c0 0x7f9f6c144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f61e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c144aa0 0x7f9f6c0d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:39000/0 (socket says 172.21.15.67:39000) 2024-04-17T10:35:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f61e00640 1 -- 172.21.15.67:0/1307939254 learned_addr learned my addr 172.21.15.67:0/1307939254 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f6b200640 1 --1- 172.21.15.67:0/1307939254 >> v1:172.21.15.86:6789/0 conn(0x7f9f6c071840 0x7f9f6c15f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:60398/0 (socket says 172.21.15.67:60398) 2024-04-17T10:35:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f61e00640 1 -- 172.21.15.67:0/1307939254 >> v2:172.21.15.86:3300/0 conn(0x7f9f6c1446c0 msgr2=0x7f9f6c144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f61e00640 1 --2- 172.21.15.67:0/1307939254 >> v2:172.21.15.86:3300/0 conn(0x7f9f6c1446c0 0x7f9f6c144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f61e00640 1 -- 172.21.15.67:0/1307939254 >> v1:172.21.15.86:6789/0 conn(0x7f9f6c071840 legacy=0x7f9f6c15f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:32.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.312+0000 7f9f61e00640 1 -- 172.21.15.67:0/1307939254 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0b4870 con 0x7f9f6c144aa0 2024-04-17T10:35:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f61e00640 1 --2- 172.21.15.67:0/1307939254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c144aa0 0x7f9f6c0d0910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009870 tx=0x7f9f5401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1ca07330618956c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f69e00640 1 -- 172.21.15.67:0/1307939254 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f540aa600 con 0x7f9f6c144aa0 2024-04-17T10:35:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f69e00640 1 -- 172.21.15.67:0/1307939254 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f540af070 con 0x7f9f6c144aa0 2024-04-17T10:35:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f69e00640 1 -- 172.21.15.67:0/1307939254 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f540aa9c0 con 0x7f9f6c144aa0 2024-04-17T10:35:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 -- 172.21.15.67:0/1307939254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c144aa0 msgr2=0x7f9f6c0d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 --2- 172.21.15.67:0/1307939254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c144aa0 0x7f9f6c0d0910 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009870 tx=0x7f9f5401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 -- 172.21.15.67:0/1307939254 shutdown_connections 2024-04-17T10:35:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 --2- 172.21.15.67:0/1307939254 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c144aa0 0x7f9f6c0d0910 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 --2- 172.21.15.67:0/1307939254 >> v2:172.21.15.86:3300/0 conn(0x7f9f6c1446c0 0x7f9f6c144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 -- 172.21.15.67:0/1307939254 >> 172.21.15.67:0/1307939254 conn(0x7f9f6c0b34d0 msgr2=0x7f9f6c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 -- 172.21.15.67:0/1307939254 shutdown_connections 2024-04-17T10:35:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.316+0000 7f9f71400640 1 -- 172.21.15.67:0/1307939254 wait complete. 2024-04-17T10:35:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 Processor -- start 2024-04-17T10:35:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 -- start start 2024-04-17T10:35:32.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c071840 0x7f9f6c0b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:32.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c1446c0 0x7f9f6c0a9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:32.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6c144aa0 0x7f9f6c15b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:32.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c0d1260 con 0x7f9f6c144aa0 2024-04-17T10:35:32.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c15c0d0 con 0x7f9f6c1446c0 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6c168a40 con 0x7f9f6c071840 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f6b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6c144aa0 0x7f9f6c15b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f6b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6c144aa0 0x7f9f6c15b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51914/0 (socket says 172.21.15.67:51914) 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f6a800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c071840 0x7f9f6c0b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f6b200640 1 -- 172.21.15.67:0/4024702584 learned_addr learned my addr 172.21.15.67:0/4024702584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f61e00640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c1446c0 0x7f9f6c0a9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f61e00640 1 -- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c071840 msgr2=0x7f9f6c0b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f61e00640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c071840 0x7f9f6c0b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f61e00640 1 -- 172.21.15.67:0/4024702584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6c144aa0 msgr2=0x7f9f6c15b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f61e00640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6c144aa0 0x7f9f6c15b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f61e00640 1 -- 172.21.15.67:0/4024702584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c0aa6b0 con 0x7f9f6c1446c0 2024-04-17T10:35:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f6a800640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c071840 0x7f9f6c0b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f61e00640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c1446c0 0x7f9f6c0a9e60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c06bcc0 tx=0x7f9f5401ace0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f63400640 1 -- 172.21.15.67:0/4024702584 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f540bd640 con 0x7f9f6c1446c0 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f63400640 1 -- 172.21.15.67:0/4024702584 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f540bd7e0 con 0x7f9f6c1446c0 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f63400640 1 -- 172.21.15.67:0/4024702584 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f540bdbd0 con 0x7f9f6c1446c0 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6c0aa990 con 0x7f9f6c1446c0 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.320+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6c0a8930 con 0x7f9f6c1446c0 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.324+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6c070bb0 con 0x7f9f6c1446c0 2024-04-17T10:35:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.324+0000 7f9f63400640 1 -- 172.21.15.67:0/4024702584 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9f540a9070 con 0x7f9f6c1446c0 2024-04-17T10:35:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.324+0000 7f9f63400640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f48066f90 0x7f9f48069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.324+0000 7f9f63400640 1 -- 172.21.15.67:0/4024702584 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 5421+0+0 (secure 0 0 0) 0x7f9f5411eca0 con 0x7f9f6c1446c0 2024-04-17T10:35:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.324+0000 7f9f6a800640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f48066f90 0x7f9f48069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.324+0000 7f9f6a800640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f48066f90 0x7f9f48069450 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009f50 tx=0x7f9f5c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:32.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.328+0000 7f9f63400640 1 -- 172.21.15.67:0/4024702584 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9f540c4070 con 0x7f9f6c1446c0 2024-04-17T10:35:32.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:32.504+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f9f6c074e30 con 0x7f9f6c1446c0 2024-04-17T10:35:33.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.051+0000 7f9f63400640 1 -- 172.21.15.67:0/4024702584 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9f54016150 con 0x7f9f6c1446c0 2024-04-17T10:35:33.046 INFO:teuthology.orchestra.run.smithi067.stderr: removed all blocklist entries 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f48066f90 msgr2=0x7f9f48069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f48066f90 0x7f9f48069450 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009f50 tx=0x7f9f5c009190 comp rx=0 tx=0).stop 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c1446c0 msgr2=0x7f9f6c0a9e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c1446c0 0x7f9f6c0a9e60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c06bcc0 tx=0x7f9f5401ace0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 shutdown_connections 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9f48066f90 0x7f9f48069450 secure :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c009f50 tx=0x7f9f5c009190 comp rx=0 tx=0).stop 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9f6c144aa0 0x7f9f6c15b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9f6c1446c0 0x7f9f6c0a9e60 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 --2- 172.21.15.67:0/4024702584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9f6c071840 0x7f9f6c0b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 >> 172.21.15.67:0/4024702584 conn(0x7f9f6c0b34d0 msgr2=0x7f9f6c0b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:33.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 shutdown_connections 2024-04-17T10:35:33.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.055+0000 7f9f71400640 1 -- 172.21.15.67:0/4024702584 wait complete. 2024-04-17T10:35:33.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-17T10:35:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9ab6600640 1 Processor -- start 2024-04-17T10:35:33.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9ab6600640 1 -- start start 2024-04-17T10:35:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9ab6600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ab01446c0 0x7f9ab0144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9ab6600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0144aa0 0x7f9ab00d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9ab6600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab00b6640 con 0x7f9ab0071840 2024-04-17T10:35:33.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9ab6600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00b51a0 con 0x7f9ab01446c0 2024-04-17T10:35:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9ab6600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab01456d0 con 0x7f9ab0144aa0 2024-04-17T10:35:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aaf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ab01446c0 0x7f9ab0144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aaf400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9ab01446c0 0x7f9ab0144ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51930/0 (socket says 172.21.15.67:51930) 2024-04-17T10:35:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aaf400640 1 -- 172.21.15.67:0/2423770019 learned_addr learned my addr 172.21.15.67:0/2423770019 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:33.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aa6a00640 1 --2- 172.21.15.67:0/2423770019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0144aa0 0x7f9ab00d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aaf400640 1 -- 172.21.15.67:0/2423770019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0144aa0 msgr2=0x7f9ab00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aaf400640 1 --2- 172.21.15.67:0/2423770019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0144aa0 0x7f9ab00d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aaf400640 1 -- 172.21.15.67:0/2423770019 >> v1:172.21.15.86:6789/0 conn(0x7f9ab0071840 legacy=0x7f9ab015f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aaf400640 1 -- 172.21.15.67:0/2423770019 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab00b4870 con 0x7f9ab01446c0 2024-04-17T10:35:33.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.223+0000 7f9aa6a00640 1 --2- 172.21.15.67:0/2423770019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0144aa0 0x7f9ab00d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9aaf400640 1 --2- 172.21.15.67:0/2423770019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab01446c0 0x7f9ab0144ea0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009e50 tx=0x7f9aa001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f10bdb8ec22ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9aaea00640 1 -- 172.21.15.67:0/2423770019 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00a5830 con 0x7f9ab01446c0 2024-04-17T10:35:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9aaea00640 1 -- 172.21.15.67:0/2423770019 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa00a59d0 con 0x7f9ab01446c0 2024-04-17T10:35:33.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9aaea00640 1 -- 172.21.15.67:0/2423770019 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00a5da0 con 0x7f9ab01446c0 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 -- 172.21.15.67:0/2423770019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab01446c0 msgr2=0x7f9ab0144ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 --2- 172.21.15.67:0/2423770019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab01446c0 0x7f9ab0144ea0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0009e50 tx=0x7f9aa001c970 comp rx=0 tx=0).stop 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 -- 172.21.15.67:0/2423770019 shutdown_connections 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 --2- 172.21.15.67:0/2423770019 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab0144aa0 0x7f9ab00d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 --2- 172.21.15.67:0/2423770019 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab01446c0 0x7f9ab0144ea0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 -- 172.21.15.67:0/2423770019 >> 172.21.15.67:0/2423770019 conn(0x7f9ab00b34d0 msgr2=0x7f9ab00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 -- 172.21.15.67:0/2423770019 shutdown_connections 2024-04-17T10:35:33.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 -- 172.21.15.67:0/2423770019 wait complete. 2024-04-17T10:35:33.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.227+0000 7f9ab6600640 1 Processor -- start 2024-04-17T10:35:33.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 -- start start 2024-04-17T10:35:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab0071840 0x7f9ab0143320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab01446c0 0x7f9ab0165af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab0144aa0 0x7f9ab0166030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00d1260 con 0x7f9ab0071840 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab015c0d0 con 0x7f9ab0144aa0 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0168a40 con 0x7f9ab01446c0 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab0071840 0x7f9ab0143320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab0071840 0x7f9ab0143320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51944/0 (socket says 172.21.15.67:51944) 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 -- 172.21.15.67:0/1191171165 learned_addr learned my addr 172.21.15.67:0/1191171165 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aa6a00640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab01446c0 0x7f9ab0165af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aafe00640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab0144aa0 0x7f9ab0166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 -- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab01446c0 msgr2=0x7f9ab0165af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab01446c0 0x7f9ab0165af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 -- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab0144aa0 msgr2=0x7f9ab0166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab0144aa0 0x7f9ab0166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 -- 172.21.15.67:0/1191171165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab0153de0 con 0x7f9ab0071840 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aafe00640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab0144aa0 0x7f9ab0166030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aa6a00640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab01446c0 0x7f9ab0165af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aaf400640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab0071840 0x7f9ab0143320 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00af040 tx=0x7f9aa0019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aa7e00640 1 -- 172.21.15.67:0/1191171165 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00bc020 con 0x7f9ab0071840 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aa7e00640 1 -- 172.21.15.67:0/1191171165 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa00b8880 con 0x7f9ab0071840 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab01540c0 con 0x7f9ab0071840 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9aa7e00640 1 -- 172.21.15.67:0/1191171165 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00b8c70 con 0x7f9ab0071840 2024-04-17T10:35:33.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab0154680 con 0x7f9ab0071840 2024-04-17T10:35:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.231+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab0070bb0 con 0x7f9ab0071840 2024-04-17T10:35:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.235+0000 7f9aa7e00640 1 -- 172.21.15.67:0/1191171165 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9aa00bf020 con 0x7f9ab0071840 2024-04-17T10:35:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.235+0000 7f9aa7e00640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a8c066f90 0x7f9a8c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.235+0000 7f9aa7e00640 1 -- 172.21.15.67:0/1191171165 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9aa011fec0 con 0x7f9ab0071840 2024-04-17T10:35:33.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.235+0000 7f9aa6a00640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a8c066f90 0x7f9a8c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.235+0000 7f9aa6a00640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a8c066f90 0x7f9a8c069450 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0152ec0 tx=0x7f9a98008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.239+0000 7f9aa7e00640 1 -- 172.21.15.67:0/1191171165 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9aa00c6070 con 0x7f9ab0071840 2024-04-17T10:35:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f9ab0074e30 con 0x7f9ab0071840 2024-04-17T10:35:33.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9aa7e00640 1 -- 172.21.15.67:0/1191171165 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f9aa0016620 con 0x7f9ab0071840 2024-04-17T10:35:33.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:33.452 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:35:33.452 INFO:teuthology.orchestra.run.smithi067.stdout: "auth_dump": [ 2024-04-17T10:35:33.452 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.452 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.a", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mA10xLhAARsxn23HfCWhLjnmVTuYW8g==", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.b", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9m3VmJOhAAApuUrZW1jZjnNHcM+hQxaw==", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.453 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.c", 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mAFOFNRAAd1KhTq/OBqp8xpvU8DrAOQ==", 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.454 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mds.d", 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mHwotBhAAW1yB3LOVv9azY2D+RETdGQ==", 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.0", 2024-04-17T10:35:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA+ox9mxsduDRAAEQsyMUKX65t0ENVWZ+r1vQ==", 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.1", 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBBox9mLBLdFxAAPULzmqsYA+xTpCvap+UCig==", 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.456 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.2", 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBEox9mdoa7IBAA7U0LB95vFe6UVEm20s+ryA==", 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.3", 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBHox9mwB1tKhAAYWYR5bLv6+5IBpA8vBzfkw==", 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.457 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.4", 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBNox9m+tXzIBAAY/Sne7WbZYsQon8kRPtubQ==", 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.458 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.5", 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBQox9mrnhBKxAAjAHi90VVwusMOTjVWKI85g==", 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.6", 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBTox9mWurzNRAAlXqJK9d4qHrCJhZdaqFJCw==", 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.459 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "osd.7", 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQBXox9mznJnBBAAceM1wAIIkCl+WbyiBPXWsQ==", 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow profile osd", 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile osd", 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.460 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.0", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww==", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow r", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.1", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m906XExAAfEq0noB22+WB+NWudXy5wA==", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:35:33.461 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.2", 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9mSogMEBAAqYt6qC6sAeGCSoioCWFiGA==", 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.462 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.3", 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA7ox9m1KeRGRAAO47GGiCMv6+MJX5lOw863g==", 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow", 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow r", 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow rw", 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow rwx" 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "client.admin", 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mpZAdEhAA7trb/2rih1G+vVdAKJtXWA==", 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.463 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "mgr": "allow *", 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow *", 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.x", 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mJUoLKhAALyny4EwA8Hw4Ptx7p+2rSg==", 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2024-04-17T10:35:33.464 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: "entity": "mgr.y", 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: "key": "AQA6ox9mMDQpJhAAeTQtZoX3zdTs6Vw7ay7CFw==", 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": { 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: "mds": "allow *", 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: "mon": "allow profile mgr", 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: "osd": "allow *" 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-04-17T10:35:33.465 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a8c066f90 msgr2=0x7f9a8c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a8c066f90 0x7f9a8c069450 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0152ec0 tx=0x7f9a98008040 comp rx=0 tx=0).stop 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab0071840 msgr2=0x7f9ab0143320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab0071840 0x7f9ab0143320 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00af040 tx=0x7f9aa0019710 comp rx=0 tx=0).stop 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 shutdown_connections 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a8c066f90 0x7f9a8c069450 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9ab0144aa0 0x7f9ab0166030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9ab01446c0 0x7f9ab0165af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 --2- 172.21.15.67:0/1191171165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ab0071840 0x7f9ab0143320 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 >> 172.21.15.67:0/1191171165 conn(0x7f9ab00b34d0 msgr2=0x7f9ab015c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 shutdown_connections 2024-04-17T10:35:33.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.459+0000 7f9ab6600640 1 -- 172.21.15.67:0/1191171165 wait complete. 2024-04-17T10:35:33.474 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.619+0000 7fa995000640 1 Processor -- start 2024-04-17T10:35:33.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.619+0000 7fa995000640 1 -- start start 2024-04-17T10:35:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa995000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9900d0200 0x7fa990070680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa995000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa990071240 0x7fa990099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa995000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99006f610 con 0x7fa990099f90 2024-04-17T10:35:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa995000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99006ff40 con 0x7fa990071240 2024-04-17T10:35:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa995000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa99009ac20 con 0x7fa9900d0200 2024-04-17T10:35:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa990099f90 0x7fa99015f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59970/0 (socket says 172.21.15.67:59970) 2024-04-17T10:35:33.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98fe00640 1 -- 172.21.15.67:0/545200688 learned_addr learned my addr 172.21.15.67:0/545200688 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:33.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98f400640 1 --2- 172.21.15.67:0/545200688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9900d0200 0x7fa990070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa987e00640 1 --2- 172.21.15.67:0/545200688 >> v2:172.21.15.86:3300/0 conn(0x7fa990071240 0x7fa990099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98ea00640 1 -- 172.21.15.67:0/545200688 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094692024 0 0) 0x7fa99006f610 con 0x7fa990099f90 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98ea00640 1 -- 172.21.15.67:0/545200688 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa974003530 con 0x7fa990099f90 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98f400640 1 -- 172.21.15.67:0/545200688 >> v2:172.21.15.86:3300/0 conn(0x7fa990071240 msgr2=0x7fa990099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98f400640 1 --2- 172.21.15.67:0/545200688 >> v2:172.21.15.86:3300/0 conn(0x7fa990071240 0x7fa990099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98f400640 1 -- 172.21.15.67:0/545200688 >> v1:172.21.15.67:6789/0 conn(0x7fa990099f90 legacy=0x7fa99015f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa98f400640 1 -- 172.21.15.67:0/545200688 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99006f8f0 con 0x7fa9900d0200 2024-04-17T10:35:33.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.623+0000 7fa987e00640 1 --2- 172.21.15.67:0/545200688 >> v2:172.21.15.86:3300/0 conn(0x7fa990071240 0x7fa990099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:33.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa98f400640 1 --2- 172.21.15.67:0/545200688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9900d0200 0x7fa990070680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9800097c0 tx=0x7fa98001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5034ed1644d7875 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa98ea00640 1 -- 172.21.15.67:0/545200688 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9800b0070 con 0x7fa9900d0200 2024-04-17T10:35:33.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa98ea00640 1 -- 172.21.15.67:0/545200688 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9800ab600 con 0x7fa9900d0200 2024-04-17T10:35:33.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa98ea00640 1 -- 172.21.15.67:0/545200688 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9800ab9d0 con 0x7fa9900d0200 2024-04-17T10:35:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 -- 172.21.15.67:0/545200688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9900d0200 msgr2=0x7fa990070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 --2- 172.21.15.67:0/545200688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9900d0200 0x7fa990070680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa9800097c0 tx=0x7fa98001c410 comp rx=0 tx=0).stop 2024-04-17T10:35:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 -- 172.21.15.67:0/545200688 shutdown_connections 2024-04-17T10:35:33.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 --2- 172.21.15.67:0/545200688 >> v2:172.21.15.86:3300/0 conn(0x7fa990071240 0x7fa990099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 --2- 172.21.15.67:0/545200688 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa9900d0200 0x7fa990070680 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 -- 172.21.15.67:0/545200688 >> 172.21.15.67:0/545200688 conn(0x7fa9900d1600 msgr2=0x7fa9900d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 -- 172.21.15.67:0/545200688 shutdown_connections 2024-04-17T10:35:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 -- 172.21.15.67:0/545200688 wait complete. 2024-04-17T10:35:33.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 Processor -- start 2024-04-17T10:35:33.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.627+0000 7fa995000640 1 -- start start 2024-04-17T10:35:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa990071240 0x7fa9900c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990099f90 0x7fa990155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9900d0200 0x7fa9900b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900cd170 con 0x7fa990099f90 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9900ccff0 con 0x7fa9900d0200 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa990168850 con 0x7fa990071240 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990099f90 0x7fa990155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990099f90 0x7fa990155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51960/0 (socket says 172.21.15.67:51960) 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 -- 172.21.15.67:0/1539200084 learned_addr learned my addr 172.21.15.67:0/1539200084 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa98f400640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa990071240 0x7fa9900c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa98fe00640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9900d0200 0x7fa9900b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 -- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa990071240 msgr2=0x7fa9900c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa990071240 0x7fa9900c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 -- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9900d0200 msgr2=0x7fa9900b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9900d0200 0x7fa9900b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 -- 172.21.15.67:0/1539200084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9900c9f60 con 0x7fa990099f90 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa98fe00640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9900d0200 0x7fa9900b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:33.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987e00640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990099f90 0x7fa990155810 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fa97800e9b0 tx=0x7fa97800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987400640 1 -- 172.21.15.67:0/1539200084 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978017070 con 0x7fa990099f90 2024-04-17T10:35:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987400640 1 -- 172.21.15.67:0/1539200084 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97800c9a0 con 0x7fa990099f90 2024-04-17T10:35:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987400640 1 -- 172.21.15.67:0/1539200084 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97800cd70 con 0x7fa990099f90 2024-04-17T10:35:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9900ca240 con 0x7fa990099f90 2024-04-17T10:35:33.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9900ca800 con 0x7fa990099f90 2024-04-17T10:35:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa990074dc0 con 0x7fa990099f90 2024-04-17T10:35:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.631+0000 7fa987400640 1 -- 172.21.15.67:0/1539200084 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa97801a070 con 0x7fa990099f90 2024-04-17T10:35:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.635+0000 7fa987400640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa96c066f60 0x7fa96c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.635+0000 7fa987400640 1 -- 172.21.15.67:0/1539200084 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5249+0+0 (secure 0 0 0) 0x7fa978084ff0 con 0x7fa990099f90 2024-04-17T10:35:33.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.635+0000 7fa98f400640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa96c066f60 0x7fa96c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.635+0000 7fa98f400640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa96c066f60 0x7fa96c069420 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa9800ad000 tx=0x7fa9800a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.639+0000 7fa987400640 1 -- 172.21.15.67:0/1539200084 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa978055b30 con 0x7fa990099f90 2024-04-17T10:35:33.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.799+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9900b7580 con 0x7fa990099f90 2024-04-17T10:35:33.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa987400640 1 -- 172.21.15.67:0/1539200084 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v137) v1 ==== 74+0+13126 (secure 0 0 0) 0x7fa978059bd0 con 0x7fa990099f90 2024-04-17T10:35:33.795 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:33.795 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":137,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:33.058527+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa96c066f60 msgr2=0x7fa96c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa96c066f60 0x7fa96c069420 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fa9800ad000 tx=0x7fa9800a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990099f90 msgr2=0x7fa990155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990099f90 0x7fa990155810 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fa97800e9b0 tx=0x7fa97800ee70 comp rx=0 tx=0).stop 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 shutdown_connections 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa96c066f60 0x7fa96c069420 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa9900d0200 0x7fa9900b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa990099f90 0x7fa990155810 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 --2- 172.21.15.67:0/1539200084 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa990071240 0x7fa9900c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 >> 172.21.15.67:0/1539200084 conn(0x7fa9900d1600 msgr2=0x7fa9900aef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:33.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 shutdown_connections 2024-04-17T10:35:33.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.803+0000 7fa995000640 1 -- 172.21.15.67:0/1539200084 wait complete. 2024-04-17T10:35:33.817 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:35:33.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:35:33.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.967+0000 7f9a47a00640 1 Processor -- start 2024-04-17T10:35:33.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a47a00640 1 -- start start 2024-04-17T10:35:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a47a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 0x7f9a400ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a47a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a4015cb30 con 0x7f9a4015be70 2024-04-17T10:35:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a47a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a400cd610 con 0x7f9a40159440 2024-04-17T10:35:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a47a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a400a8650 con 0x7f9a40071af0 2024-04-17T10:35:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9a4015be70 0x7f9a4015c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59984/0 (socket says 172.21.15.67:59984) 2024-04-17T10:35:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45a00640 1 -- 172.21.15.67:0/3132159507 learned_addr learned my addr 172.21.15.67:0/3132159507 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3fe00640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 0x7f9a400ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10816157 0 0) 0x7f9a4015cb30 con 0x7f9a4015be70 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a28003530 con 0x7f9a4015be70 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 463603995 0 0) 0x7f9a28003530 con 0x7f9a4015be70 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 msgr2=0x7f9a400ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 0x7f9a400ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 >> v1:172.21.15.86:6789/0 conn(0x7f9a40159440 legacy=0x7f9a400713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a400cd8f0 con 0x7f9a4015be70 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f9a38001a00 con 0x7f9a4015be70 2024-04-17T10:35:33.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 >> v1:172.21.15.67:6789/0 conn(0x7f9a4015be70 legacy=0x7f9a4015c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a280039f0 0x7f9a28003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 0x7f9a28004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a28004d50 0x7f9a280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2800b9d0 con 0x7f9a280039f0 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2800bb50 con 0x7f9a28004d50 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2800bd50 con 0x7f9a40071af0 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45000640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a280039f0 0x7f9a28003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a3fe00640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 0x7f9a28004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45a00640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a28004d50 0x7f9a280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45000640 1 -- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 msgr2=0x7f9a28004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45000640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a40071af0 0x7f9a28004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45000640 1 -- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a28004d50 msgr2=0x7f9a280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45000640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a28004d50 0x7f9a280051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.971+0000 7f9a45000640 1 -- 172.21.15.67:0/3132159507 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a28003530 con 0x7f9a280039f0 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a45000640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a280039f0 0x7f9a28003dd0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f9a34001910 tx=0x7f9a3401c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab598f9f5c64df8f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a340b0070 con 0x7f9a280039f0 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a340ab620 con 0x7f9a280039f0 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a3f400640 1 -- 172.21.15.67:0/3132159507 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a34001510 con 0x7f9a280039f0 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a47a00640 1 -- 172.21.15.67:0/3132159507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a280039f0 msgr2=0x7f9a28003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a47a00640 1 --2- 172.21.15.67:0/3132159507 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a280039f0 0x7f9a28003dd0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f9a34001910 tx=0x7f9a3401c4c0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a45a00640 1 -- 172.21.15.67:0/3132159507 reap_dead start 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a47a00640 1 -- 172.21.15.67:0/3132159507 shutdown_connections 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a47a00640 1 -- 172.21.15.67:0/3132159507 >> 172.21.15.67:0/3132159507 conn(0x7f9a400b1740 msgr2=0x7f9a400b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a47a00640 1 -- 172.21.15.67:0/3132159507 shutdown_connections 2024-04-17T10:35:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.975+0000 7f9a47a00640 1 -- 172.21.15.67:0/3132159507 wait complete. 2024-04-17T10:35:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 Processor -- start 2024-04-17T10:35:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 -- start start 2024-04-17T10:35:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a280039f0 0x7f9a401616b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a401502c0 0x7f9a40157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40157f60 0x7f9a400afd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4015dbd0 con 0x7f9a40157f60 2024-04-17T10:35:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4015da50 con 0x7f9a280039f0 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4015dd50 con 0x7f9a401502c0 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40157f60 0x7f9a400afd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40157f60 0x7f9a400afd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51978/0 (socket says 172.21.15.67:51978) 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 -- 172.21.15.67:0/255317772 learned_addr learned my addr 172.21.15.67:0/255317772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45000640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a280039f0 0x7f9a401616b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a3fe00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a401502c0 0x7f9a40157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 -- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a401502c0 msgr2=0x7f9a40157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a401502c0 0x7f9a40157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 -- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a280039f0 msgr2=0x7f9a401616b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a280039f0 0x7f9a401616b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 -- 172.21.15.67:0/255317772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a40156df0 con 0x7f9a40157f60 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45000640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a280039f0 0x7f9a401616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a45a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40157f60 0x7f9a400afd60 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f9a400ce690 tx=0x7f9a38007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a3cc00640 1 -- 172.21.15.67:0/255317772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a3800abf0 con 0x7f9a40157f60 2024-04-17T10:35:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a3cc00640 1 -- 172.21.15.67:0/255317772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a38019070 con 0x7f9a40157f60 2024-04-17T10:35:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a3cc00640 1 -- 172.21.15.67:0/255317772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a380133f0 con 0x7f9a40157f60 2024-04-17T10:35:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a40155f90 con 0x7f9a40157f60 2024-04-17T10:35:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a40156550 con 0x7f9a40157f60 2024-04-17T10:35:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.979+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4004ef30 con 0x7f9a40157f60 2024-04-17T10:35:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.983+0000 7f9a3cc00640 1 -- 172.21.15.67:0/255317772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9a38013590 con 0x7f9a40157f60 2024-04-17T10:35:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.983+0000 7f9a3cc00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a20066f60 0x7f9a20069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.983+0000 7f9a3cc00640 1 -- 172.21.15.67:0/255317772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f9a380854d0 con 0x7f9a40157f60 2024-04-17T10:35:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.983+0000 7f9a45000640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a20066f60 0x7f9a20069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:33.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.983+0000 7f9a45000640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a20066f60 0x7f9a20069420 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9a340ad000 tx=0x7f9a34001330 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:33.987+0000 7f9a3cc00640 1 -- 172.21.15.67:0/255317772 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9a38056600 con 0x7f9a40157f60 2024-04-17T10:35:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.151+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9a400ccf20 con 0x7f9a40157f60 2024-04-17T10:35:34.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.151+0000 7f9a3cc00640 1 -- 172.21.15.67:0/255317772 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v137) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9a38055f00 con 0x7f9a40157f60 2024-04-17T10:35:34.145 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:35:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a20066f60 msgr2=0x7f9a20069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a20066f60 0x7f9a20069420 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f9a340ad000 tx=0x7f9a34001330 comp rx=0 tx=0).stop 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40157f60 msgr2=0x7f9a400afd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40157f60 0x7f9a400afd60 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f9a400ce690 tx=0x7f9a38007bd0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 shutdown_connections 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9a20066f60 0x7f9a20069420 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9a40157f60 0x7f9a400afd60 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9a401502c0 0x7f9a40157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 --2- 172.21.15.67:0/255317772 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9a280039f0 0x7f9a401616b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 >> 172.21.15.67:0/255317772 conn(0x7f9a400b1740 msgr2=0x7f9a4015df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 shutdown_connections 2024-04-17T10:35:34.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.155+0000 7f9a47a00640 1 -- 172.21.15.67:0/255317772 wait complete. 2024-04-17T10:35:34.167 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-17T10:35:34.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13e00640 1 Processor -- start 2024-04-17T10:35:34.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13e00640 1 -- start start 2024-04-17T10:35:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f1406c7b0 0x7f8f1406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8f140d0840 0x7f8f140cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1409a7d0 con 0x7f8f140d03e0 2024-04-17T10:35:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f140cf6d0 con 0x7f8f140d0840 2024-04-17T10:35:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1409a4f0 con 0x7f8f1406c7b0 2024-04-17T10:35:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f0a000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8f140d03e0 0x7f8f14099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59988/0 (socket says 172.21.15.67:59988) 2024-04-17T10:35:34.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f0a000640 1 -- 172.21.15.67:0/2664760681 learned_addr learned my addr 172.21.15.67:0/2664760681 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13400640 1 --2- 172.21.15.67:0/2664760681 >> v2:172.21.15.86:3300/0 conn(0x7f8f140d0840 0x7f8f140cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f12a00640 1 --2- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f1406c7b0 0x7f8f1406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f12000640 1 -- 172.21.15.67:0/2664760681 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138523868 0 0) 0x7f8f1409a7d0 con 0x7f8f140d03e0 2024-04-17T10:35:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f12000640 1 -- 172.21.15.67:0/2664760681 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef8003530 con 0x7f8f140d03e0 2024-04-17T10:35:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13400640 1 -- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f1406c7b0 msgr2=0x7f8f1406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13400640 1 --2- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f1406c7b0 0x7f8f1406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13400640 1 -- 172.21.15.67:0/2664760681 >> v1:172.21.15.67:6789/0 conn(0x7f8f140d03e0 legacy=0x7f8f14099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.315+0000 7f8f13400640 1 -- 172.21.15.67:0/2664760681 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f140cf900 con 0x7f8f140d0840 2024-04-17T10:35:34.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13400640 1 --2- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f140d0840 0x7f8f140cbc70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009d90 tx=0x7f8f0c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a6883825dbd43f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:34.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f12000640 1 -- 172.21.15.67:0/2664760681 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0b2070 con 0x7f8f140d0840 2024-04-17T10:35:34.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f12000640 1 -- 172.21.15.67:0/2664760681 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c0a5920 con 0x7f8f140d0840 2024-04-17T10:35:34.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f12000640 1 -- 172.21.15.67:0/2664760681 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0a5cf0 con 0x7f8f140d0840 2024-04-17T10:35:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 -- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f140d0840 msgr2=0x7f8f140cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 --2- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f140d0840 0x7f8f140cbc70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009d90 tx=0x7f8f0c01c970 comp rx=0 tx=0).stop 2024-04-17T10:35:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 -- 172.21.15.67:0/2664760681 shutdown_connections 2024-04-17T10:35:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 --2- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f140d0840 0x7f8f140cbc70 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 --2- 172.21.15.67:0/2664760681 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f1406c7b0 0x7f8f1406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 -- 172.21.15.67:0/2664760681 >> 172.21.15.67:0/2664760681 conn(0x7f8f140d1620 msgr2=0x7f8f140d1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:34.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 -- 172.21.15.67:0/2664760681 shutdown_connections 2024-04-17T10:35:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.319+0000 7f8f13e00640 1 -- 172.21.15.67:0/2664760681 wait complete. 2024-04-17T10:35:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 Processor -- start 2024-04-17T10:35:34.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 -- start start 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1406c7b0 0x7f8f14153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f140d03e0 0x7f8f14161dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f140d0840 0x7f8f14154f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f140cc5c0 con 0x7f8f140d0840 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1406d0d0 con 0x7f8f1406c7b0 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1409a160 con 0x7f8f140d03e0 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f140d0840 0x7f8f14154f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f12a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1406c7b0 0x7f8f14153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f140d0840 0x7f8f14154f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51990/0 (socket says 172.21.15.67:51990) 2024-04-17T10:35:34.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 -- 172.21.15.67:0/3544258343 learned_addr learned my addr 172.21.15.67:0/3544258343 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f0a000640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f140d03e0 0x7f8f14161dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 -- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f140d03e0 msgr2=0x7f8f14161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f140d03e0 0x7f8f14161dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 -- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1406c7b0 msgr2=0x7f8f14153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1406c7b0 0x7f8f14153bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 -- 172.21.15.67:0/3544258343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f141541f0 con 0x7f8f140d0840 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13400640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f140d0840 0x7f8f14154f70 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009d90 tx=0x7f8f0c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f0b400640 1 -- 172.21.15.67:0/3544258343 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0b2050 con 0x7f8f140d0840 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f0b400640 1 -- 172.21.15.67:0/3544258343 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0c0be070 con 0x7f8f140d0840 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f14154490 con 0x7f8f140d0840 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f0b400640 1 -- 172.21.15.67:0/3544258343 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0ad870 con 0x7f8f140d0840 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.323+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f14154a50 con 0x7f8f140d0840 2024-04-17T10:35:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.327+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f14074dc0 con 0x7f8f140d0840 2024-04-17T10:35:34.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.327+0000 7f8f0b400640 1 -- 172.21.15.67:0/3544258343 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8f0c0ada10 con 0x7f8f140d0840 2024-04-17T10:35:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.327+0000 7f8f0b400640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ef0066f60 0x7f8ef0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.327+0000 7f8f0b400640 1 -- 172.21.15.67:0/3544258343 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f8f0c11fe20 con 0x7f8f140d0840 2024-04-17T10:35:34.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.327+0000 7f8f12a00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ef0066f60 0x7f8ef0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.327+0000 7f8f12a00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ef0066f60 0x7f8ef0069420 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009ac0 tx=0x7f8f04009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:34.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.331+0000 7f8f0b400640 1 -- 172.21.15.67:0/3544258343 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8f0c016560 con 0x7f8f140d0840 2024-04-17T10:35:34.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.495+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8f140cb570 con 0x7f8f140d0840 2024-04-17T10:35:34.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.495+0000 7f8f0b400640 1 -- 172.21.15.67:0/3544258343 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v137) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8f0c0add00 con 0x7f8f140d0840 2024-04-17T10:35:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:34.490 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-04-17T10:35:34.490 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-04-17T10:35:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-04-17T10:35:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-04-17T10:35:34.491 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-04-17T10:35:34.491 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-04-17T10:35:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ef0066f60 msgr2=0x7f8ef0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ef0066f60 0x7f8ef0069420 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f8f04009ac0 tx=0x7f8f04009190 comp rx=0 tx=0).stop 2024-04-17T10:35:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f140d0840 msgr2=0x7f8f14154f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f140d0840 0x7f8f14154f70 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009d90 tx=0x7f8f0c019710 comp rx=0 tx=0).stop 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 shutdown_connections 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ef0066f60 0x7f8ef0069420 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f140d0840 0x7f8f14154f70 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8f140d03e0 0x7f8f14161dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 --2- 172.21.15.67:0/3544258343 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8f1406c7b0 0x7f8f14153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 >> 172.21.15.67:0/3544258343 conn(0x7f8f140d1620 msgr2=0x7f8f14070680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 shutdown_connections 2024-04-17T10:35:34.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.499+0000 7f8f13e00640 1 -- 172.21.15.67:0/3544258343 wait complete. 2024-04-17T10:35:34.511 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-17T10:35:34.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-17T10:35:34.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1bf4a00640 1 Processor -- start 2024-04-17T10:35:34.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1bf4a00640 1 -- start start 2024-04-17T10:35:34.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1bf4a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1bf00d10c0 0x7f1bf0168b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1bf4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf015f150 con 0x7f1bf006cd70 2024-04-17T10:35:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1bf4a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf015e8b0 con 0x7f1bf0099b30 2024-04-17T10:35:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1bf4a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf006c0f0 con 0x7f1bf00d10c0 2024-04-17T10:35:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1be5400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1bf006cd70 0x7f1bf015de90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59996/0 (socket says 172.21.15.67:59996) 2024-04-17T10:35:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.663+0000 7f1be5400640 1 -- 172.21.15.67:0/3756206729 learned_addr learned my addr 172.21.15.67:0/3756206729 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 --2- 172.21.15.67:0/3756206729 >> v2:172.21.15.86:3300/0 conn(0x7f1bf00d10c0 0x7f1bf0168b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456361353 0 0) 0x7f1bf015f150 con 0x7f1bf006cd70 2024-04-17T10:35:34.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd4003530 con 0x7f1bf006cd70 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2256058542 0 0) 0x7f1bd4003530 con 0x7f1bf006cd70 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 >> v2:172.21.15.86:3300/0 conn(0x7f1bf00d10c0 msgr2=0x7f1bf0168b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 --2- 172.21.15.67:0/3756206729 >> v2:172.21.15.86:3300/0 conn(0x7f1bf00d10c0 0x7f1bf0168b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 >> v1:172.21.15.86:6789/0 conn(0x7f1bf0099b30 legacy=0x7f1bf0099f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf015eb90 con 0x7f1bf006cd70 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f1bd8001680 con 0x7f1bf006cd70 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 --2- 172.21.15.67:0/3756206729 >> v2:172.21.15.86:3300/0 conn(0x7f1bf00d10c0 0x7f1bf0168b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 >> v1:172.21.15.67:6789/0 conn(0x7f1bf006cd70 legacy=0x7f1bf015de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd4003a10 0x7f1bd4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd4004810 0x7f1bd4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bd400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd400c0a0 con 0x7f1bd40051d0 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd400c250 con 0x7f1bd4004810 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd400c450 con 0x7f1bd4003a10 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bd400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1be5400640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd4004810 0x7f1bd4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bede00640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd4003a10 0x7f1bd4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 -- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd4003a10 msgr2=0x7f1bd4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bd4003a10 0x7f1bd4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 -- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd4004810 msgr2=0x7f1bd4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd4004810 0x7f1bd4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1bee800640 1 -- 172.21.15.67:0/3756206729 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd4003530 con 0x7f1bd40051d0 2024-04-17T10:35:34.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.667+0000 7f1be5400640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bd4004810 0x7f1bd4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:34.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bee800640 1 -- 172.21.15.67:0/3756206729 reap_dead start 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bee800640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bd400b9d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f1be800e350 tx=0x7f1be801ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b37871ca838a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be80b2070 con 0x7f1bd40051d0 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be80ad440 con 0x7f1bd40051d0 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bed400640 1 -- 172.21.15.67:0/3756206729 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8009de0 con 0x7f1bd40051d0 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3756206729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 msgr2=0x7f1bd400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bd400b9d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f1be800e350 tx=0x7f1be801ce80 comp rx=0 tx=0).stop 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3756206729 shutdown_connections 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3756206729 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bd400b9d0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3756206729 >> 172.21.15.67:0/3756206729 conn(0x7f1bf00b32e0 msgr2=0x7f1bf00b36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3756206729 shutdown_connections 2024-04-17T10:35:34.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3756206729 wait complete. 2024-04-17T10:35:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 Processor -- start 2024-04-17T10:35:34.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- start start 2024-04-17T10:35:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bf00b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf0151c10 0x7f1bf01f3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf00b66e0 0x7f1bf00b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf00cfbb0 con 0x7f1bd40051d0 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0099f10 con 0x7f1bf0151c10 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.671+0000 7f1bf4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf009a090 con 0x7f1bf00b66e0 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bf00b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bf00b2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52000/0 (socket says 172.21.15.67:52000) 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 -- 172.21.15.67:0/3023570235 learned_addr learned my addr 172.21.15.67:0/3023570235 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bee800640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf00b66e0 0x7f1bf00b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1be5400640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf0151c10 0x7f1bf01f3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 -- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf00b66e0 msgr2=0x7f1bf00b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf00b66e0 0x7f1bf00b6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 -- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf0151c10 msgr2=0x7f1bf01f3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf0151c10 0x7f1bf01f3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 -- 172.21.15.67:0/3023570235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf0159260 con 0x7f1bd40051d0 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bede00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bf00b2b50 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f1be000e4d0 tx=0x7f1be000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1be6a00640 1 -- 172.21.15.67:0/3023570235 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be00052e0 con 0x7f1bd40051d0 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1be6a00640 1 -- 172.21.15.67:0/3023570235 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be0018070 con 0x7f1bd40051d0 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf00b4410 con 0x7f1bd40051d0 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1be6a00640 1 -- 172.21.15.67:0/3023570235 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be0003820 con 0x7f1bd40051d0 2024-04-17T10:35:34.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf00b49d0 con 0x7f1bd40051d0 2024-04-17T10:35:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf009f900 con 0x7f1bd40051d0 2024-04-17T10:35:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1be6a00640 1 -- 172.21.15.67:0/3023570235 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1be00039c0 con 0x7f1bd40051d0 2024-04-17T10:35:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.675+0000 7f1be6a00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bcc066f60 0x7f1bcc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.679+0000 7f1be6a00640 1 -- 172.21.15.67:0/3023570235 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5249+0+0 (secure 0 0 0) 0x7f1be0084bf0 con 0x7f1bd40051d0 2024-04-17T10:35:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.679+0000 7f1be5400640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bcc066f60 0x7f1bcc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:34.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.679+0000 7f1be5400640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bcc066f60 0x7f1bcc069420 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8001000 tx=0x7f1bd8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.683+0000 7f1be6a00640 1 -- 172.21.15.67:0/3023570235 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1be0055c90 con 0x7f1bd40051d0 2024-04-17T10:35:34.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:34.847+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1bf0074e30 con 0x7f1bd40051d0 2024-04-17T10:35:35.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.055+0000 7f1be6a00640 1 -- 172.21.15.67:0/3023570235 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v138) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1be0005480 con 0x7f1bd40051d0 2024-04-17T10:35:35.051 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bcc066f60 msgr2=0x7f1bcc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bcc066f60 0x7f1bcc069420 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1bd8001000 tx=0x7f1bd8009040 comp rx=0 tx=0).stop 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 msgr2=0x7f1bf00b2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bf00b2b50 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f1be000e4d0 tx=0x7f1be000e990 comp rx=0 tx=0).stop 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 shutdown_connections 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1bcc066f60 0x7f1bcc069420 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1bf00b66e0 0x7f1bf00b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1bf0151c10 0x7f1bf01f3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 --2- 172.21.15.67:0/3023570235 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1bd40051d0 0x7f1bf00b2b50 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 >> 172.21.15.67:0/3023570235 conn(0x7f1bf00b32e0 msgr2=0x7f1bf006fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 shutdown_connections 2024-04-17T10:35:35.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.059+0000 7f1bf4a00640 1 -- 172.21.15.67:0/3023570235 wait complete. 2024-04-17T10:35:35.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-17T10:35:35.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dc2400640 1 Processor -- start 2024-04-17T10:35:35.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dc2400640 1 -- start start 2024-04-17T10:35:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dc2400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3dbc159440 0x7f3dbc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dc2400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc15cb30 con 0x7f3dbc15c580 2024-04-17T10:35:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dc2400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dbc0cd610 con 0x7f3dbc0713e0 2024-04-17T10:35:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dc2400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dbc0a8650 con 0x7f3dbc159440 2024-04-17T10:35:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3dbc15c580 0x7f3dbc15d030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60006/0 (socket says 172.21.15.67:60006) 2024-04-17T10:35:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbbe00640 1 -- 172.21.15.67:0/2511155844 learned_addr learned my addr 172.21.15.67:0/2511155844 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:35.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbb400640 1 --2- 172.21.15.67:0/2511155844 >> v2:172.21.15.67:3300/0 conn(0x7f3dbc159440 0x7f3dbc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbaa00640 1 -- 172.21.15.67:0/2511155844 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954899492 0 0) 0x7f3dbc15cb30 con 0x7f3dbc15c580 2024-04-17T10:35:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbaa00640 1 -- 172.21.15.67:0/2511155844 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da0003530 con 0x7f3dbc15c580 2024-04-17T10:35:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbaa00640 1 -- 172.21.15.67:0/2511155844 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839736477 0 0) 0x7f3dbc0cd610 con 0x7f3dbc0713e0 2024-04-17T10:35:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbb400640 1 -- 172.21.15.67:0/2511155844 >> v1:172.21.15.86:6789/0 conn(0x7f3dbc0713e0 legacy=0x7f3dbc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:35.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbb400640 1 -- 172.21.15.67:0/2511155844 >> v1:172.21.15.67:6789/0 conn(0x7f3dbc15c580 legacy=0x7f3dbc15d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:35.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.235+0000 7f3dbb400640 1 -- 172.21.15.67:0/2511155844 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc0cd8f0 con 0x7f3dbc159440 2024-04-17T10:35:35.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dbb400640 1 --2- 172.21.15.67:0/2511155844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc159440 0x7f3dbc0ce0f0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009d90 tx=0x7f3dbc15df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5de82c54f4432a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:35.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dbaa00640 1 -- 172.21.15.67:0/2511155844 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0b2070 con 0x7f3dbc159440 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dbaa00640 1 -- 172.21.15.67:0/2511155844 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac0a5940 con 0x7f3dbc159440 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dbaa00640 1 -- 172.21.15.67:0/2511155844 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0a5d10 con 0x7f3dbc159440 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dc2400640 1 -- 172.21.15.67:0/2511155844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc159440 msgr2=0x7f3dbc0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dc2400640 1 --2- 172.21.15.67:0/2511155844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc159440 0x7f3dbc0ce0f0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009d90 tx=0x7f3dbc15df50 comp rx=0 tx=0).stop 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dc2400640 1 -- 172.21.15.67:0/2511155844 shutdown_connections 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dc2400640 1 --2- 172.21.15.67:0/2511155844 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc159440 0x7f3dbc0ce0f0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dc2400640 1 -- 172.21.15.67:0/2511155844 >> 172.21.15.67:0/2511155844 conn(0x7f3dbc0b1740 msgr2=0x7f3dbc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dc2400640 1 -- 172.21.15.67:0/2511155844 shutdown_connections 2024-04-17T10:35:35.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.239+0000 7f3dc2400640 1 -- 172.21.15.67:0/2511155844 wait complete. 2024-04-17T10:35:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 Processor -- start 2024-04-17T10:35:35.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 -- start start 2024-04-17T10:35:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc0713e0 0x7f3dbc0acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dbc159440 0x7f3dbc1399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dbc15c580 0x7f3dbc159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc169640 con 0x7f3dbc0713e0 2024-04-17T10:35:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc1694c0 con 0x7f3dbc15c580 2024-04-17T10:35:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dbc1697c0 con 0x7f3dbc159440 2024-04-17T10:35:35.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc0713e0 0x7f3dbc0acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc0713e0 0x7f3dbc0acbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52022/0 (socket says 172.21.15.67:52022) 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 -- 172.21.15.67:0/19512234 learned_addr learned my addr 172.21.15.67:0/19512234 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 -- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dbc159440 msgr2=0x7f3dbc1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dbc159440 0x7f3dbc1399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 -- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dbc15c580 msgr2=0x7f3dbc159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dbc15c580 0x7f3dbc159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 -- 172.21.15.67:0/19512234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dbc0ad130 con 0x7f3dbc0713e0 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dbb400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc0713e0 0x7f3dbc0acbf0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f3dac01a5a0 tx=0x7f3dac019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3db3e00640 1 -- 172.21.15.67:0/19512234 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac0b2040 con 0x7f3dbc0713e0 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3db3e00640 1 -- 172.21.15.67:0/19512234 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dac003bc0 con 0x7f3dbc0713e0 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dbc139f00 con 0x7f3dbc0713e0 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3db3e00640 1 -- 172.21.15.67:0/19512234 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac004030 con 0x7f3dbc0713e0 2024-04-17T10:35:35.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dbc0ad6b0 con 0x7f3dbc0713e0 2024-04-17T10:35:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.243+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dbc04ef30 con 0x7f3dbc0713e0 2024-04-17T10:35:35.239 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.247+0000 7f3db3e00640 1 -- 172.21.15.67:0/19512234 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3dac0b5070 con 0x7f3dbc0713e0 2024-04-17T10:35:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.247+0000 7f3db3e00640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d98066f60 0x7f3d98069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:35.240 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.247+0000 7f3db3e00640 1 -- 172.21.15.67:0/19512234 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5619+0+0 (secure 0 0 0) 0x7f3dac120130 con 0x7f3dbc0713e0 2024-04-17T10:35:35.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.247+0000 7f3db2a00640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d98066f60 0x7f3d98069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:35.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.247+0000 7f3db2a00640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d98066f60 0x7f3d98069420 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3da4007fd0 tx=0x7f3da4007940 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:35.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.251+0000 7f3db3e00640 1 -- 172.21.15.67:0/19512234 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3dac0c5070 con 0x7f3dbc0713e0 2024-04-17T10:35:35.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:35.411+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3dbc0ce440 con 0x7f3dbc0713e0 2024-04-17T10:35:36.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.059+0000 7f3db3e00640 1 -- 172.21.15.67:0/19512234 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v139) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3dac0bc030 con 0x7f3dbc0713e0 2024-04-17T10:35:36.054 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d98066f60 msgr2=0x7f3d98069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d98066f60 0x7f3d98069420 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3da4007fd0 tx=0x7f3da4007940 comp rx=0 tx=0).stop 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc0713e0 msgr2=0x7f3dbc0acbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc0713e0 0x7f3dbc0acbf0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f3dac01a5a0 tx=0x7f3dac019710 comp rx=0 tx=0).stop 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 shutdown_connections 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3d98066f60 0x7f3d98069420 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3dbc15c580 0x7f3dbc159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3dbc159440 0x7f3dbc1399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 --2- 172.21.15.67:0/19512234 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3dbc0713e0 0x7f3dbc0acbf0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 >> 172.21.15.67:0/19512234 conn(0x7f3dbc0b1740 msgr2=0x7f3dbc0af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 shutdown_connections 2024-04-17T10:35:36.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.063+0000 7f3dc2400640 1 -- 172.21.15.67:0/19512234 wait complete. 2024-04-17T10:35:36.096 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-17T10:35:36.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f750c600640 1 Processor -- start 2024-04-17T10:35:36.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f750c600640 1 -- start start 2024-04-17T10:35:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f750c600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 0x7f75041653b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f750c600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75040b4990 con 0x7f75040cd030 2024-04-17T10:35:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f750c600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750415ae90 con 0x7f75040aef60 2024-04-17T10:35:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f750c600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75040af340 con 0x7f750406e980 2024-04-17T10:35:36.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f7509200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f75040cd030 0x7f750406e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60016/0 (socket says 172.21.15.67:60016) 2024-04-17T10:35:36.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.247+0000 7f7509200640 1 -- 172.21.15.67:0/75267287 learned_addr learned my addr 172.21.15.67:0/75267287 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f750a600640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 0x7f75041653b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419931019 0 0) 0x7f74f0001120 con 0x7f75040cd030 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74ec003530 con 0x7f75040cd030 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390099905 0 0) 0x7f750415ae90 con 0x7f75040aef60 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f0001120 con 0x7f75040aef60 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 266422013 0 0) 0x7f74ec003530 con 0x7f75040cd030 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 msgr2=0x7f75041653b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 0x7f75041653b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 >> v1:172.21.15.86:6789/0 conn(0x7f75040aef60 legacy=0x7f75040cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750406ee80 con 0x7f75040cd030 2024-04-17T10:35:36.245 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f74f00017f0 con 0x7f75040cd030 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 >> v1:172.21.15.67:6789/0 conn(0x7f75040cd030 legacy=0x7f750406e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec003b80 0x7f74ec003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 0x7f74ec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec004d50 0x7f74ec0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75040af340 con 0x7f74ec004d50 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec00ffe0 con 0x7f74ec003b80 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec010160 con 0x7f750406e980 2024-04-17T10:35:36.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f750a600640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec004d50 0x7f74ec0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7509c00640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec003b80 0x7f74ec003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f7509200640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 0x7f74ec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f750a600640 1 -- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 msgr2=0x7f74ec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:36.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f750a600640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f750406e980 0x7f74ec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f750a600640 1 -- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec003b80 msgr2=0x7f74ec003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f750a600640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec003b80 0x7f74ec003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.251+0000 7f750a600640 1 -- 172.21.15.67:0/75267287 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec003530 con 0x7f74ec004d50 2024-04-17T10:35:36.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750a600640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec004d50 0x7f74ec0051d0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009f90 tx=0x7f74fc01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9c80e8e7384124 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:36.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0b2070 con 0x7f74ec004d50 2024-04-17T10:35:36.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc0a5c50 con 0x7f74ec004d50 2024-04-17T10:35:36.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f7503e00640 1 -- 172.21.15.67:0/75267287 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc009c10 con 0x7f74ec004d50 2024-04-17T10:35:36.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 -- 172.21.15.67:0/75267287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec004d50 msgr2=0x7f74ec0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 --2- 172.21.15.67:0/75267287 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74ec004d50 0x7f74ec0051d0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f74fc009f90 tx=0x7f74fc01cc40 comp rx=0 tx=0).stop 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750a600640 1 -- 172.21.15.67:0/75267287 reap_dead start 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 -- 172.21.15.67:0/75267287 shutdown_connections 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 -- 172.21.15.67:0/75267287 >> 172.21.15.67:0/75267287 conn(0x7f75040b1740 msgr2=0x7f75040b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 -- 172.21.15.67:0/75267287 shutdown_connections 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 -- 172.21.15.67:0/75267287 wait complete. 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 Processor -- start 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 -- start start 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec004d50 0x7f7504143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.255+0000 7f750c600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7504155440 0x7f7504156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750c600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75041558c0 0x7f75041572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750c600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750415b410 con 0x7f7504155440 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750c600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75040b4130 con 0x7f74ec004d50 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750c600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750415bc10 con 0x7f75041558c0 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7504155440 0x7f7504156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7504155440 0x7f7504156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52028/0 (socket says 172.21.15.67:52028) 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 -- 172.21.15.67:0/1764183259 learned_addr learned my addr 172.21.15.67:0/1764183259 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750a600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75041558c0 0x7f75041572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509c00640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec004d50 0x7f7504143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 -- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75041558c0 msgr2=0x7f75041572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75041558c0 0x7f75041572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 -- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec004d50 msgr2=0x7f7504143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec004d50 0x7f7504143020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 -- 172.21.15.67:0/1764183259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7504155e20 con 0x7f7504155440 2024-04-17T10:35:36.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750a600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75041558c0 0x7f75041572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:36.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7509200640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7504155440 0x7f7504156d70 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f74f00072c0 tx=0x7f74f0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:36.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7501600640 1 -- 172.21.15.67:0/1764183259 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000f960 con 0x7f7504155440 2024-04-17T10:35:36.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7501600640 1 -- 172.21.15.67:0/1764183259 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f0018070 con 0x7f7504155440 2024-04-17T10:35:36.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7501600640 1 -- 172.21.15.67:0/1764183259 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f000fd80 con 0x7f7504155440 2024-04-17T10:35:36.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7504156100 con 0x7f7504155440 2024-04-17T10:35:36.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75041566c0 con 0x7f7504155440 2024-04-17T10:35:36.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750404ef30 con 0x7f7504155440 2024-04-17T10:35:36.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.259+0000 7f7501600640 1 -- 172.21.15.67:0/1764183259 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f74f001d460 con 0x7f7504155440 2024-04-17T10:35:36.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.263+0000 7f7501600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f74e4066f90 0x7f74e4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:36.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.263+0000 7f7501600640 1 -- 172.21.15.67:0/1764183259 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f74f0085b10 con 0x7f7504155440 2024-04-17T10:35:36.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.263+0000 7f7509c00640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f74e4066f90 0x7f74e4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:36.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.263+0000 7f7509c00640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f74e4066f90 0x7f74e4069450 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f750415ad50 tx=0x7f74f8006ff0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:36.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.267+0000 7f7501600640 1 -- 172.21.15.67:0/1764183259 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f75040b4990 con 0x7f7504155440 2024-04-17T10:35:36.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:36.463+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f75040af8f0 con 0x7f7504155440 2024-04-17T10:35:37.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.087+0000 7f7501600640 1 -- 172.21.15.67:0/1764183259 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 21 and data pool 22 v99) v1 ==== 174+0+0 (secure 0 0 0) 0x7f74f0056410 con 0x7f7504155440 2024-04-17T10:35:37.084 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 21 and data pool 22 2024-04-17T10:35:37.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f74e4066f90 msgr2=0x7f74e4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f74e4066f90 0x7f74e4069450 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f750415ad50 tx=0x7f74f8006ff0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7504155440 msgr2=0x7f7504156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7504155440 0x7f7504156d70 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f74f00072c0 tx=0x7f74f0007780 comp rx=0 tx=0).stop 2024-04-17T10:35:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 shutdown_connections 2024-04-17T10:35:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f74e4066f90 0x7f74e4069450 secure :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f750415ad50 tx=0x7f74f8006ff0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f75041558c0 0x7f75041572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7504155440 0x7f7504156d70 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 --2- 172.21.15.67:0/1764183259 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74ec004d50 0x7f7504143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.095+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 >> 172.21.15.67:0/1764183259 conn(0x7f75040b1740 msgr2=0x7f7504153d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:37.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.099+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 shutdown_connections 2024-04-17T10:35:37.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.099+0000 7f750c600640 1 -- 172.21.15.67:0/1764183259 wait complete. 2024-04-17T10:35:37.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:37.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.275+0000 7f33a9600640 1 Processor -- start 2024-04-17T10:35:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a9600640 1 -- start start 2024-04-17T10:35:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a9600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33a406e6e0 0x7f33a40ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a9600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a41441f0 0x7f33a40ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a9600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a415ae40 con 0x7f33a406eac0 2024-04-17T10:35:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a9600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a40cf430 con 0x7f33a406e6e0 2024-04-17T10:35:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a9600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a4071440 con 0x7f33a41441f0 2024-04-17T10:35:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a41441f0 0x7f33a40ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a2a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33a406e6e0 0x7f33a40ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f339be00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f33a406eac0 0x7f33a4143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:60426/0 (socket says 172.21.15.67:60426) 2024-04-17T10:35:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f339be00640 1 -- 172.21.15.67:0/48602906 learned_addr learned my addr 172.21.15.67:0/48602906 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a2a00640 1 -- 172.21.15.67:0/48602906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a41441f0 msgr2=0x7f33a40ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a2a00640 1 --2- 172.21.15.67:0/48602906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a41441f0 0x7f33a40ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a2a00640 1 -- 172.21.15.67:0/48602906 >> v1:172.21.15.86:6789/0 conn(0x7f33a406eac0 legacy=0x7f33a4143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.279+0000 7f33a2a00640 1 -- 172.21.15.67:0/48602906 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a4071720 con 0x7f33a406e6e0 2024-04-17T10:35:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a2a00640 1 --2- 172.21.15.67:0/48602906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a406e6e0 0x7f33a40ce9b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f33940097c0 tx=0x7f339401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e34d0b60a11b66af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a2000640 1 -- 172.21.15.67:0/48602906 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940b0070 con 0x7f33a406e6e0 2024-04-17T10:35:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a2000640 1 -- 172.21.15.67:0/48602906 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33940ab600 con 0x7f33a406e6e0 2024-04-17T10:35:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a2000640 1 -- 172.21.15.67:0/48602906 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33940ab9d0 con 0x7f33a406e6e0 2024-04-17T10:35:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 -- 172.21.15.67:0/48602906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a406e6e0 msgr2=0x7f33a40ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 --2- 172.21.15.67:0/48602906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a406e6e0 0x7f33a40ce9b0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f33940097c0 tx=0x7f339401c3e0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 -- 172.21.15.67:0/48602906 shutdown_connections 2024-04-17T10:35:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 --2- 172.21.15.67:0/48602906 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a41441f0 0x7f33a40ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 --2- 172.21.15.67:0/48602906 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a406e6e0 0x7f33a40ce9b0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 -- 172.21.15.67:0/48602906 >> 172.21.15.67:0/48602906 conn(0x7f33a406ac90 msgr2=0x7f33a406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 -- 172.21.15.67:0/48602906 shutdown_connections 2024-04-17T10:35:37.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 -- 172.21.15.67:0/48602906 wait complete. 2024-04-17T10:35:37.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.283+0000 7f33a9600640 1 Processor -- start 2024-04-17T10:35:37.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 -- start start 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a406e6e0 0x7f33a4138090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a406eac0 0x7f33a4152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a41441f0 0x7f33a4152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40b3a40 con 0x7f33a406eac0 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40b38c0 con 0x7f33a41441f0 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40b3bc0 con 0x7f33a406e6e0 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a406eac0 0x7f33a4152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a406eac0 0x7f33a4152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52036/0 (socket says 172.21.15.67:52036) 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 -- 172.21.15.67:0/1341910407 learned_addr learned my addr 172.21.15.67:0/1341910407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a2a00640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a406e6e0 0x7f33a4138090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a3400640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a41441f0 0x7f33a4152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 -- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a406e6e0 msgr2=0x7f33a4138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a406e6e0 0x7f33a4138090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 -- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a41441f0 msgr2=0x7f33a4152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a41441f0 0x7f33a4152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 -- 172.21.15.67:0/1341910407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a41545a0 con 0x7f33a406eac0 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339be00640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a406eac0 0x7f33a4152320 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f338c00c420 tx=0x7f338c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339aa00640 1 -- 172.21.15.67:0/1341910407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c017070 con 0x7f33a406eac0 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339aa00640 1 -- 172.21.15.67:0/1341910407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c005cf0 con 0x7f33a406eac0 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f339aa00640 1 -- 172.21.15.67:0/1341910407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c0072f0 con 0x7f33a406eac0 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a4154880 con 0x7f33a406eac0 2024-04-17T10:35:37.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a4154e40 con 0x7f33a406eac0 2024-04-17T10:35:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.287+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a4072d50 con 0x7f33a406eac0 2024-04-17T10:35:37.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.291+0000 7f339aa00640 1 -- 172.21.15.67:0/1341910407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f338c01a070 con 0x7f33a406eac0 2024-04-17T10:35:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.291+0000 7f339aa00640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.291+0000 7f339aa00640 1 -- 172.21.15.67:0/1341910407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f338c084540 con 0x7f33a406eac0 2024-04-17T10:35:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.291+0000 7f33a2a00640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.291+0000 7f33a2a00640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f33940ad000 tx=0x7f33940a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.295+0000 7f339aa00640 1 -- 172.21.15.67:0/1341910407 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f338c054cd0 con 0x7f33a406eac0 2024-04-17T10:35:37.450 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.455+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33a40b2c60 con 0x7f33a406eac0 2024-04-17T10:35:37.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f339aa00640 1 -- 172.21.15.67:0/1341910407 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16857 (secure 0 0 0) 0x7f338c058d70 con 0x7f33a406eac0 2024-04-17T10:35:37.451 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:37.452 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":140,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:36.469896+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:35:34.898907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-17T10:35:35.464996+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 msgr2=0x7f3380069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f33940ad000 tx=0x7f33940a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a406eac0 msgr2=0x7f33a4152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a406eac0 0x7f33a4152320 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f338c00c420 tx=0x7f338c00c8e0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 shutdown_connections 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a41441f0 0x7f33a4152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a406eac0 0x7f33a4152320 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 --2- 172.21.15.67:0/1341910407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a406e6e0 0x7f33a4138090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 >> 172.21.15.67:0/1341910407 conn(0x7f33a406ac90 msgr2=0x7f33a40cc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 shutdown_connections 2024-04-17T10:35:37.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.459+0000 7f33a9600640 1 -- 172.21.15.67:0/1341910407 wait complete. 2024-04-17T10:35:37.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:37.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeba200640 1 Processor -- start 2024-04-17T10:35:37.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeba200640 1 -- start start 2024-04-17T10:35:37.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeba200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdeb4099a50 0x7fdeb415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeba200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdeb40d03a0 0x7fdeb4099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeba200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb40cde20 con 0x7fdeb409a740 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeba200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb40cd1b0 con 0x7fdeb40d03a0 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeba200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb409a370 con 0x7fdeb4099a50 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb3e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdeb40d03a0 0x7fdeb4099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb9800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fdeb409a740 0x7fdeb415f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60028/0 (socket says 172.21.15.67:60028) 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb8e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fdeb4099a50 0x7fdeb415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb9800640 1 -- 172.21.15.67:0/643976036 learned_addr learned my addr 172.21.15.67:0/643976036 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb3400640 1 -- 172.21.15.67:0/643976036 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262427443 0 0) 0x7fdeb40cde20 con 0x7fdeb409a740 2024-04-17T10:35:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb3e00640 1 -- 172.21.15.67:0/643976036 >> v2:172.21.15.86:3300/0 conn(0x7fdeb4099a50 msgr2=0x7fdeb415c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb3e00640 1 --2- 172.21.15.67:0/643976036 >> v2:172.21.15.86:3300/0 conn(0x7fdeb4099a50 0x7fdeb415c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb3e00640 1 -- 172.21.15.67:0/643976036 >> v1:172.21.15.67:6789/0 conn(0x7fdeb409a740 legacy=0x7fdeb415f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb3e00640 1 -- 172.21.15.67:0/643976036 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb40cf300 con 0x7fdeb40d03a0 2024-04-17T10:35:37.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.623+0000 7fdeb8e00640 1 --2- 172.21.15.67:0/643976036 >> v2:172.21.15.86:3300/0 conn(0x7fdeb4099a50 0x7fdeb415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeb3e00640 1 --2- 172.21.15.67:0/643976036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb40d03a0 0x7fdeb4099e30 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fdeb409a710 tx=0x7fdea001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a031e29bd563c4f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeb3400640 1 -- 172.21.15.67:0/643976036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea00b0070 con 0x7fdeb40d03a0 2024-04-17T10:35:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeb3400640 1 -- 172.21.15.67:0/643976036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea00ab600 con 0x7fdeb40d03a0 2024-04-17T10:35:37.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeb3400640 1 -- 172.21.15.67:0/643976036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea00ab9d0 con 0x7fdeb40d03a0 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 -- 172.21.15.67:0/643976036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb40d03a0 msgr2=0x7fdeb4099e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 --2- 172.21.15.67:0/643976036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb40d03a0 0x7fdeb4099e30 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fdeb409a710 tx=0x7fdea001c3f0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 -- 172.21.15.67:0/643976036 shutdown_connections 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 --2- 172.21.15.67:0/643976036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb40d03a0 0x7fdeb4099e30 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 --2- 172.21.15.67:0/643976036 >> v2:172.21.15.86:3300/0 conn(0x7fdeb4099a50 0x7fdeb415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 -- 172.21.15.67:0/643976036 >> 172.21.15.67:0/643976036 conn(0x7fdeb40d1690 msgr2=0x7fdeb40d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 -- 172.21.15.67:0/643976036 shutdown_connections 2024-04-17T10:35:37.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.627+0000 7fdeba200640 1 -- 172.21.15.67:0/643976036 wait complete. 2024-04-17T10:35:37.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 Processor -- start 2024-04-17T10:35:37.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 -- start start 2024-04-17T10:35:37.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdeb4099a50 0x7fdeb406cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb409a740 0x7fdeb415b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdeb40d03a0 0x7fdeb414ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb41686b0 con 0x7fdeb409a740 2024-04-17T10:35:37.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb40cfe80 con 0x7fdeb40d03a0 2024-04-17T10:35:37.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb4168830 con 0x7fdeb4099a50 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb409a740 0x7fdeb415b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb409a740 0x7fdeb415b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52056/0 (socket says 172.21.15.67:52056) 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 -- 172.21.15.67:0/3460899889 learned_addr learned my addr 172.21.15.67:0/3460899889 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb8e00640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdeb4099a50 0x7fdeb406cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb9800640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdeb40d03a0 0x7fdeb414ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 -- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdeb4099a50 msgr2=0x7fdeb406cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdeb4099a50 0x7fdeb406cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 -- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdeb40d03a0 msgr2=0x7fdeb414ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdeb40d03a0 0x7fdeb414ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 -- 172.21.15.67:0/3460899889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb406bed0 con 0x7fdeb409a740 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb9800640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdeb40d03a0 0x7fdeb414ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:37.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb3e00640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb409a740 0x7fdeb415b400 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fdea001a020 tx=0x7fdea0019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb0c00640 1 -- 172.21.15.67:0/3460899889 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea00ba070 con 0x7fdeb409a740 2024-04-17T10:35:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb0c00640 1 -- 172.21.15.67:0/3460899889 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea00b5460 con 0x7fdeb409a740 2024-04-17T10:35:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeb0c00640 1 -- 172.21.15.67:0/3460899889 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea00b0040 con 0x7fdeb409a740 2024-04-17T10:35:37.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb406c1b0 con 0x7fdeb409a740 2024-04-17T10:35:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb406c770 con 0x7fdeb409a740 2024-04-17T10:35:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.631+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb4074dc0 con 0x7fdeb409a740 2024-04-17T10:35:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.635+0000 7fdeb0c00640 1 -- 172.21.15.67:0/3460899889 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdea00bd070 con 0x7fdeb409a740 2024-04-17T10:35:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.635+0000 7fdeb0c00640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde94066f60 0x7fde94069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.635+0000 7fdeb0c00640 1 -- 172.21.15.67:0/3460899889 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fdea011f600 con 0x7fdeb409a740 2024-04-17T10:35:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.635+0000 7fdeb8e00640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde94066f60 0x7fde94069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.635+0000 7fdeb8e00640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde94066f60 0x7fde94069420 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009580 tx=0x7fdea8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.639+0000 7fdeb0c00640 1 -- 172.21.15.67:0/3460899889 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdea0016070 con 0x7fdeb409a740 2024-04-17T10:35:37.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.799+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeb40d24a0 con 0x7fdeb409a740 2024-04-17T10:35:37.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.799+0000 7fdeb0c00640 1 -- 172.21.15.67:0/3460899889 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16857 (secure 0 0 0) 0x7fdea00efe20 con 0x7fdeb409a740 2024-04-17T10:35:37.794 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:37.794 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":140,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:36.469896+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:35:34.898907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-17T10:35:35.464996+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde94066f60 msgr2=0x7fde94069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde94066f60 0x7fde94069420 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009580 tx=0x7fdea8009190 comp rx=0 tx=0).stop 2024-04-17T10:35:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb409a740 msgr2=0x7fdeb415b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb409a740 0x7fdeb415b400 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fdea001a020 tx=0x7fdea0019190 comp rx=0 tx=0).stop 2024-04-17T10:35:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 shutdown_connections 2024-04-17T10:35:37.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fde94066f60 0x7fde94069420 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdeb40d03a0 0x7fdeb414ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdeb409a740 0x7fdeb415b400 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 --2- 172.21.15.67:0/3460899889 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdeb4099a50 0x7fdeb406cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 >> 172.21.15.67:0/3460899889 conn(0x7fdeb40d1690 msgr2=0x7fdeb40ce0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 shutdown_connections 2024-04-17T10:35:37.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.803+0000 7fdeba200640 1 -- 172.21.15.67:0/3460899889 wait complete. 2024-04-17T10:35:37.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-17T10:35:37.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822fe00640 1 Processor -- start 2024-04-17T10:35:37.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822fe00640 1 -- start start 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82281474c0 0x7f82280b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82280b3920 0x7f822806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822fe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82281616a0 con 0x7f822806b1d0 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822fe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822806b640 con 0x7f82280b3920 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82280d1490 con 0x7f82281474c0 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822de00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f822806b1d0 0x7f822815dfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60032/0 (socket says 172.21.15.67:60032) 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822de00640 1 -- 172.21.15.67:0/799424493 learned_addr learned my addr 172.21.15.67:0/799424493 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:37.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822d400640 1 --2- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82281474c0 0x7f82280b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822ca00640 1 --2- 172.21.15.67:0/799424493 >> v2:172.21.15.86:3300/0 conn(0x7f82280b3920 0x7f822806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f8227e00640 1 -- 172.21.15.67:0/799424493 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799224247 0 0) 0x7f82281616a0 con 0x7f822806b1d0 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f8227e00640 1 -- 172.21.15.67:0/799424493 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8210003530 con 0x7f822806b1d0 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822ca00640 1 -- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82281474c0 msgr2=0x7f82280b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822ca00640 1 --2- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82281474c0 0x7f82280b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822ca00640 1 -- 172.21.15.67:0/799424493 >> v1:172.21.15.67:6789/0 conn(0x7f822806b1d0 legacy=0x7f822815dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822ca00640 1 -- 172.21.15.67:0/799424493 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228161940 con 0x7f82280b3920 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.967+0000 7f822d400640 1 --2- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82281474c0 0x7f82280b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822ca00640 1 --2- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82280b3920 0x7f822806ac90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f82140097c0 tx=0x7f821401ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=902ca7c406c42a16 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f8227e00640 1 -- 172.21.15.67:0/799424493 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140b5070 con 0x7f82280b3920 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f8227e00640 1 -- 172.21.15.67:0/799424493 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82140a8990 con 0x7f82280b3920 2024-04-17T10:35:37.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f8227e00640 1 -- 172.21.15.67:0/799424493 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140a8d60 con 0x7f82280b3920 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 -- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82280b3920 msgr2=0x7f822806ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 --2- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82280b3920 0x7f822806ac90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f82140097c0 tx=0x7f821401ec90 comp rx=0 tx=0).stop 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 -- 172.21.15.67:0/799424493 shutdown_connections 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 --2- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82280b3920 0x7f822806ac90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 --2- 172.21.15.67:0/799424493 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82281474c0 0x7f82280b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 -- 172.21.15.67:0/799424493 >> 172.21.15.67:0/799424493 conn(0x7f82280b5ef0 msgr2=0x7f8228099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 -- 172.21.15.67:0/799424493 shutdown_connections 2024-04-17T10:35:37.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.971+0000 7f822fe00640 1 -- 172.21.15.67:0/799424493 wait complete. 2024-04-17T10:35:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 Processor -- start 2024-04-17T10:35:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 -- start start 2024-04-17T10:35:37.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822806b1d0 0x7f822815d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82280b3920 0x7f82280ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82281474c0 0x7f8228145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82280b2250 con 0x7f82280b3920 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8228161f00 con 0x7f82281474c0 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8228162420 con 0x7f822806b1d0 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82280b3920 0x7f82280ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82280b3920 0x7f82280ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52060/0 (socket says 172.21.15.67:52060) 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 -- 172.21.15.67:0/2726845214 learned_addr learned my addr 172.21.15.67:0/2726845214 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822de00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82281474c0 0x7f8228145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822d400640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822806b1d0 0x7f822815d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 -- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822806b1d0 msgr2=0x7f822815d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822806b1d0 0x7f822815d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 -- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82281474c0 msgr2=0x7f8228145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82281474c0 0x7f8228145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 -- 172.21.15.67:0/2726845214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82280cf060 con 0x7f82280b3920 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822ca00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82280b3920 0x7f82280ce610 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f82140b3040 tx=0x7f821401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f8225600640 1 -- 172.21.15.67:0/2726845214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140bf070 con 0x7f82280b3920 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f8225600640 1 -- 172.21.15.67:0/2726845214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82140b0700 con 0x7f82280b3920 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f8225600640 1 -- 172.21.15.67:0/2726845214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82140b5040 con 0x7f82280b3920 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82280cf340 con 0x7f82280b3920 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82280cf900 con 0x7f82280b3920 2024-04-17T10:35:37.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.975+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822804ef30 con 0x7f82280b3920 2024-04-17T10:35:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.979+0000 7f8225600640 1 -- 172.21.15.67:0/2726845214 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f82140c2070 con 0x7f82280b3920 2024-04-17T10:35:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.979+0000 7f8225600640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8208066f90 0x7f8208069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:37.972 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.979+0000 7f8225600640 1 -- 172.21.15.67:0/2726845214 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f8214121710 con 0x7f82280b3920 2024-04-17T10:35:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.979+0000 7f822d400640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8208066f90 0x7f8208069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:37.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.979+0000 7f822d400640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8208066f90 0x7f8208069450 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f821c009580 tx=0x7f821c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:37.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:37.983+0000 7f8225600640 1 -- 172.21.15.67:0/2726845214 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f82140187f0 con 0x7f82280b3920 2024-04-17T10:35:38.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:38.183+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f822809a310 con 0x7f82280b3920 2024-04-17T10:35:39.050 INFO:tasks.ceph.mds.c.smithi067.stderr:2024-04-17T10:35:39.059+0000 7f55f7e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-17T10:35:39.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.099+0000 7f8225600640 1 -- 172.21.15.67:0/2726845214 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v102) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82140f1ea0 con 0x7f82280b3920 2024-04-17T10:35:39.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8208066f90 msgr2=0x7f8208069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8208066f90 0x7f8208069450 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f821c009580 tx=0x7f821c009190 comp rx=0 tx=0).stop 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82280b3920 msgr2=0x7f82280ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82280b3920 0x7f82280ce610 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f82140b3040 tx=0x7f821401ba30 comp rx=0 tx=0).stop 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 shutdown_connections 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8208066f90 0x7f8208069450 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82281474c0 0x7f8228145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82280b3920 0x7f82280ce610 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 --2- 172.21.15.67:0/2726845214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f822806b1d0 0x7f822815d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 >> 172.21.15.67:0/2726845214 conn(0x7f82280b5ef0 msgr2=0x7f82280b51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 shutdown_connections 2024-04-17T10:35:39.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.103+0000 7f822fe00640 1 -- 172.21.15.67:0/2726845214 wait complete. 2024-04-17T10:35:39.119 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-17T10:35:39.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-17T10:35:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f4701e00640 1 Processor -- start 2024-04-17T10:35:39.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f4701e00640 1 -- start start 2024-04-17T10:35:39.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f4701e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc159440 0x7f46fc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f4701e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46fc15c580 0x7f46fc071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f4701e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc15cb30 con 0x7f46fc0713e0 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f4701e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc0a8650 con 0x7f46fc15c580 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f4701e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc071d80 con 0x7f46fc159440 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46fb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46fc15c580 0x7f46fc071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46fb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f46fc15c580 0x7f46fc071840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52066/0 (socket says 172.21.15.67:52066) 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46fb400640 1 -- 172.21.15.67:0/721923066 learned_addr learned my addr 172.21.15.67:0/721923066 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:39.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46faa00640 1 --2- 172.21.15.67:0/721923066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc159440 0x7f46fc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46fb400640 1 -- 172.21.15.67:0/721923066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc159440 msgr2=0x7f46fc0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46fb400640 1 --2- 172.21.15.67:0/721923066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc159440 0x7f46fc0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46fb400640 1 -- 172.21.15.67:0/721923066 >> v1:172.21.15.86:6789/0 conn(0x7f46fc0713e0 legacy=0x7f46fc15be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:39.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.275+0000 7f46fb400640 1 -- 172.21.15.67:0/721923066 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0cd610 con 0x7f46fc15c580 2024-04-17T10:35:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f46fb400640 1 --2- 172.21.15.67:0/721923066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc071840 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009e50 tx=0x7f46f401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ab0584d54400ca server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f46fa000640 1 -- 172.21.15.67:0/721923066 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40a5810 con 0x7f46fc15c580 2024-04-17T10:35:39.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f46fa000640 1 -- 172.21.15.67:0/721923066 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f40a59b0 con 0x7f46fc15c580 2024-04-17T10:35:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f46fa000640 1 -- 172.21.15.67:0/721923066 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40a5d80 con 0x7f46fc15c580 2024-04-17T10:35:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 -- 172.21.15.67:0/721923066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 msgr2=0x7f46fc071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:39.273 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 --2- 172.21.15.67:0/721923066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc071840 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009e50 tx=0x7f46f401ca30 comp rx=0 tx=0).stop 2024-04-17T10:35:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 -- 172.21.15.67:0/721923066 shutdown_connections 2024-04-17T10:35:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 --2- 172.21.15.67:0/721923066 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc071840 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 --2- 172.21.15.67:0/721923066 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc159440 0x7f46fc0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 -- 172.21.15.67:0/721923066 >> 172.21.15.67:0/721923066 conn(0x7f46fc0b1740 msgr2=0x7f46fc0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 -- 172.21.15.67:0/721923066 shutdown_connections 2024-04-17T10:35:39.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.279+0000 7f4701e00640 1 -- 172.21.15.67:0/721923066 wait complete. 2024-04-17T10:35:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 Processor -- start 2024-04-17T10:35:39.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 -- start start 2024-04-17T10:35:39.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc0713e0 0x7f46fc0acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46fc159440 0x7f46fc15b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc15b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc15d9c0 con 0x7f46fc15c580 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc15d840 con 0x7f46fc159440 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc15db40 con 0x7f46fc0713e0 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc15b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc15b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52068/0 (socket says 172.21.15.67:52068) 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 -- 172.21.15.67:0/926442993 learned_addr learned my addr 172.21.15.67:0/926442993 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46f2000640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46fc159440 0x7f46fc15b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46faa00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc0713e0 0x7f46fc0acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 -- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc0713e0 msgr2=0x7f46fc0acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc0713e0 0x7f46fc0acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 -- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46fc159440 msgr2=0x7f46fc15b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46fc159440 0x7f46fc15b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:39.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 -- 172.21.15.67:0/926442993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0ad4f0 con 0x7f46fc15c580 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46fb400640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc15b890 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f46f40af040 tx=0x7f46f40197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46f3e00640 1 -- 172.21.15.67:0/926442993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40bc020 con 0x7f46fc15c580 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46f3e00640 1 -- 172.21.15.67:0/926442993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f40b8670 con 0x7f46fc15c580 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc0ad7d0 con 0x7f46fc15c580 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f46f3e00640 1 -- 172.21.15.67:0/926442993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40b8a60 con 0x7f46fc15c580 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc0add90 con 0x7f46fc15c580 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.283+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc0ae590 con 0x7f46fc15c580 2024-04-17T10:35:39.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.287+0000 7f46f3e00640 1 -- 172.21.15.67:0/926442993 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f46f40bf020 con 0x7f46fc15c580 2024-04-17T10:35:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.287+0000 7f46f3e00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f46d8066f60 0x7f46d8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:39.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.287+0000 7f46f3e00640 1 -- 172.21.15.67:0/926442993 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f46f411ffe0 con 0x7f46fc15c580 2024-04-17T10:35:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.287+0000 7f46faa00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f46d8066f60 0x7f46d8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:39.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.287+0000 7f46faa00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f46d8066f60 0x7f46d8069420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f46ec007a60 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:39.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.291+0000 7f46f3e00640 1 -- 172.21.15.67:0/926442993 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f46f40c6030 con 0x7f46fc15c580 2024-04-17T10:35:39.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:39.491+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f46fc04ef30 con 0x7f46fc15c580 2024-04-17T10:35:40.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.103+0000 7f46f3e00640 1 -- 172.21.15.67:0/926442993 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v103) v1 ==== 122+0+0 (secure 0 0 0) 0x7f46f4016620 con 0x7f46fc15c580 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f46d8066f60 msgr2=0x7f46d8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f46d8066f60 0x7f46d8069420 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f46ec007a60 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 msgr2=0x7f46fc15b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc15b890 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f46f40af040 tx=0x7f46f40197d0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 shutdown_connections 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f46d8066f60 0x7f46d8069420 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46fc15c580 0x7f46fc15b890 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f46fc159440 0x7f46fc15b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 --2- 172.21.15.67:0/926442993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f46fc0713e0 0x7f46fc0acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 >> 172.21.15.67:0/926442993 conn(0x7f46fc0b1740 msgr2=0x7f46fc0af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:40.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 shutdown_connections 2024-04-17T10:35:40.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.107+0000 7f4701e00640 1 -- 172.21.15.67:0/926442993 wait complete. 2024-04-17T10:35:40.476 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-17T10:35:40.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.639+0000 7f5f39000640 1 Processor -- start 2024-04-17T10:35:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.639+0000 7f5f39000640 1 -- start start 2024-04-17T10:35:40.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f39000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34071310 0x7f5f34071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:40.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f39000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f340cc300 0x7f5f340cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:40.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f39000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3406e2f0 con 0x7f5f34145040 2024-04-17T10:35:40.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f39000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f340a86c0 con 0x7f5f340cc300 2024-04-17T10:35:40.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f39000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f34071cb0 con 0x7f5f34071310 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34071310 0x7f5f34071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f2be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34071310 0x7f5f34071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:43820/0 (socket says 172.21.15.67:43820) 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f32800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5f340cc300 0x7f5f340cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f2be00640 1 -- 172.21.15.67:0/216933647 learned_addr learned my addr 172.21.15.67:0/216933647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f2be00640 1 -- 172.21.15.67:0/216933647 >> v2:172.21.15.86:3300/0 conn(0x7f5f340cc300 msgr2=0x7f5f340cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f2be00640 1 --2- 172.21.15.67:0/216933647 >> v2:172.21.15.86:3300/0 conn(0x7f5f340cc300 0x7f5f340cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f2be00640 1 -- 172.21.15.67:0/216933647 >> v1:172.21.15.86:6789/0 conn(0x7f5f34145040 legacy=0x7f5f3406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f2be00640 1 -- 172.21.15.67:0/216933647 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3406d6e0 con 0x7f5f34071310 2024-04-17T10:35:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.643+0000 7f5f32800640 1 --2- 172.21.15.67:0/216933647 >> v2:172.21.15.86:3300/0 conn(0x7f5f340cc300 0x7f5f340cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:35:40.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f2be00640 1 --2- 172.21.15.67:0/216933647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34071310 0x7f5f34071770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009870 tx=0x7f5f1c01ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c20b6068e9a2af3c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:40.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f31400640 1 -- 172.21.15.67:0/216933647 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0b5070 con 0x7f5f34071310 2024-04-17T10:35:40.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f31400640 1 -- 172.21.15.67:0/216933647 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c0a8a50 con 0x7f5f34071310 2024-04-17T10:35:40.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f31400640 1 -- 172.21.15.67:0/216933647 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0a8e20 con 0x7f5f34071310 2024-04-17T10:35:40.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 -- 172.21.15.67:0/216933647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34071310 msgr2=0x7f5f34071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 --2- 172.21.15.67:0/216933647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34071310 0x7f5f34071770 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009870 tx=0x7f5f1c01ed50 comp rx=0 tx=0).stop 2024-04-17T10:35:40.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 -- 172.21.15.67:0/216933647 shutdown_connections 2024-04-17T10:35:40.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 --2- 172.21.15.67:0/216933647 >> v2:172.21.15.86:3300/0 conn(0x7f5f340cc300 0x7f5f340cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 --2- 172.21.15.67:0/216933647 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f34071310 0x7f5f34071770 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 -- 172.21.15.67:0/216933647 >> 172.21.15.67:0/216933647 conn(0x7f5f3406ac90 msgr2=0x7f5f3406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:40.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 -- 172.21.15.67:0/216933647 shutdown_connections 2024-04-17T10:35:40.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 -- 172.21.15.67:0/216933647 wait complete. 2024-04-17T10:35:40.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 Processor -- start 2024-04-17T10:35:40.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 -- start start 2024-04-17T10:35:40.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.647+0000 7f5f39000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f34071310 0x7f5f3414d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:40.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f340cc300 0x7f5f3413eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:40.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f34145040 0x7f5f3415e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:40.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3406dca0 con 0x7f5f34071310 2024-04-17T10:35:40.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3415bf10 con 0x7f5f34145040 2024-04-17T10:35:40.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3406df80 con 0x7f5f340cc300 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f34071310 0x7f5f3414d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f34071310 0x7f5f3414d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49228/0 (socket says 172.21.15.67:49228) 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 -- 172.21.15.67:0/3191728362 learned_addr learned my addr 172.21.15.67:0/3191728362 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f2be00640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f340cc300 0x7f5f3413eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f32800640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f34145040 0x7f5f3415e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 -- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f340cc300 msgr2=0x7f5f3413eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f340cc300 0x7f5f3413eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 -- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f34145040 msgr2=0x7f5f3415e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f34145040 0x7f5f3415e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 -- 172.21.15.67:0/3191728362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3415d330 con 0x7f5f34071310 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f31e00640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f34071310 0x7f5f3414d7d0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c3a0 tx=0x7f5f2400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f2a000640 1 -- 172.21.15.67:0/3191728362 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f240052d0 con 0x7f5f34071310 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f2a000640 1 -- 172.21.15.67:0/3191728362 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f24005470 con 0x7f5f34071310 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f2a000640 1 -- 172.21.15.67:0/3191728362 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f24007260 con 0x7f5f34071310 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f3415d610 con 0x7f5f34071310 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f34153b40 con 0x7f5f34071310 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f34072d50 con 0x7f5f34071310 2024-04-17T10:35:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.651+0000 7f5f2a000640 1 -- 172.21.15.67:0/3191728362 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5f24007400 con 0x7f5f34071310 2024-04-17T10:35:40.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.655+0000 7f5f2a000640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f10066f60 0x7f5f10069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:40.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.655+0000 7f5f2a000640 1 -- 172.21.15.67:0/3191728362 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5f240844a0 con 0x7f5f34071310 2024-04-17T10:35:40.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.655+0000 7f5f2be00640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f10066f60 0x7f5f10069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:40.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.655+0000 7f5f2be00640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f10066f60 0x7f5f10069420 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0b2000 tx=0x7f5f1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.659+0000 7f5f2a000640 1 -- 172.21.15.67:0/3191728362 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5f24054cc0 con 0x7f5f34071310 2024-04-17T10:35:40.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.851+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f34074e30 con 0x7f5f34071310 2024-04-17T10:35:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f2a000640 1 -- 172.21.15.67:0/3191728362 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f5f24058d60 con 0x7f5f34071310 2024-04-17T10:35:40.843 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:40.843 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:35:39.497533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9777":{"gid":9777,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/1952434688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1952434688},{"type":"v1","addr":"172.21.15.67:6834","nonce":1952434688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:35:40.843 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 103 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f10066f60 msgr2=0x7f5f10069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f10066f60 0x7f5f10069420 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0b2000 tx=0x7f5f1c009040 comp rx=0 tx=0).stop 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f34071310 msgr2=0x7f5f3414d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f34071310 0x7f5f3414d7d0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f5f2400c3a0 tx=0x7f5f2400c7a0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 shutdown_connections 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5f10066f60 0x7f5f10069420 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5f34145040 0x7f5f3415e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5f340cc300 0x7f5f3413eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 --2- 172.21.15.67:0/3191728362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5f34071310 0x7f5f3414d7d0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:40.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 >> 172.21.15.67:0/3191728362 conn(0x7f5f3406ac90 msgr2=0x7f5f340b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:40.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 shutdown_connections 2024-04-17T10:35:40.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:40.855+0000 7f5f39000640 1 -- 172.21.15.67:0/3191728362 wait complete. 2024-04-17T10:35:40.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:35:41.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7543a00640 1 Processor -- start 2024-04-17T10:35:41.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7543a00640 1 -- start start 2024-04-17T10:35:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7543a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f753c1446c0 0x7f753c144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7543a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f753c144aa0 0x7f753c0d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7543a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f753c0b6640 con 0x7f753c071840 2024-04-17T10:35:41.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7543a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f753c0b51a0 con 0x7f753c144aa0 2024-04-17T10:35:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7543a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f753c1456d0 con 0x7f753c1446c0 2024-04-17T10:35:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f753be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f753c144aa0 0x7f753c0d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f753be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f753c144aa0 0x7f753c0d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49234/0 (socket says 172.21.15.67:49234) 2024-04-17T10:35:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f753be00640 1 -- 172.21.15.67:0/1608012903 learned_addr learned my addr 172.21.15.67:0/1608012903 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f7541000640 1 --2- 172.21.15.67:0/1608012903 >> v2:172.21.15.86:3300/0 conn(0x7f753c1446c0 0x7f753c144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f753be00640 1 -- 172.21.15.67:0/1608012903 >> v2:172.21.15.86:3300/0 conn(0x7f753c1446c0 msgr2=0x7f753c144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:41.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f753be00640 1 --2- 172.21.15.67:0/1608012903 >> v2:172.21.15.86:3300/0 conn(0x7f753c1446c0 0x7f753c144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f753be00640 1 -- 172.21.15.67:0/1608012903 >> v1:172.21.15.86:6789/0 conn(0x7f753c071840 legacy=0x7f753c15f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:41.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.019+0000 7f753be00640 1 -- 172.21.15.67:0/1608012903 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0b4870 con 0x7f753c144aa0 2024-04-17T10:35:41.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f753be00640 1 --2- 172.21.15.67:0/1608012903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c144aa0 0x7f753c0d0910 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f753c15f620 tx=0x7f752801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83025bca93a979f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f753b400640 1 -- 172.21.15.67:0/1608012903 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75280aa5f0 con 0x7f753c144aa0 2024-04-17T10:35:41.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f753b400640 1 -- 172.21.15.67:0/1608012903 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75280af070 con 0x7f753c144aa0 2024-04-17T10:35:41.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f753b400640 1 -- 172.21.15.67:0/1608012903 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75280aa9b0 con 0x7f753c144aa0 2024-04-17T10:35:41.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 -- 172.21.15.67:0/1608012903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c144aa0 msgr2=0x7f753c0d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 --2- 172.21.15.67:0/1608012903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c144aa0 0x7f753c0d0910 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f753c15f620 tx=0x7f752801c490 comp rx=0 tx=0).stop 2024-04-17T10:35:41.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 -- 172.21.15.67:0/1608012903 shutdown_connections 2024-04-17T10:35:41.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 --2- 172.21.15.67:0/1608012903 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c144aa0 0x7f753c0d0910 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 --2- 172.21.15.67:0/1608012903 >> v2:172.21.15.86:3300/0 conn(0x7f753c1446c0 0x7f753c144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 -- 172.21.15.67:0/1608012903 >> 172.21.15.67:0/1608012903 conn(0x7f753c0b34d0 msgr2=0x7f753c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:41.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 -- 172.21.15.67:0/1608012903 shutdown_connections 2024-04-17T10:35:41.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.023+0000 7f7543a00640 1 -- 172.21.15.67:0/1608012903 wait complete. 2024-04-17T10:35:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 Processor -- start 2024-04-17T10:35:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 -- start start 2024-04-17T10:35:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c071840 0x7f753c158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f753c1446c0 0x7f753c15acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f753c144aa0 0x7f753c0aa570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0d1260 con 0x7f753c071840 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c15c0d0 con 0x7f753c144aa0 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753c168a40 con 0x7f753c1446c0 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c071840 0x7f753c158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c071840 0x7f753c158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49250/0 (socket says 172.21.15.67:49250) 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 -- 172.21.15.67:0/2381629151 learned_addr learned my addr 172.21.15.67:0/2381629151 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541a00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f753c144aa0 0x7f753c0aa570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f753be00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f753c1446c0 0x7f753c15acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 -- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f753c1446c0 msgr2=0x7f753c15acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f753c1446c0 0x7f753c15acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 -- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f753c144aa0 msgr2=0x7f753c0aa570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f753c144aa0 0x7f753c0aa570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 -- 172.21.15.67:0/2381629151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c1594d0 con 0x7f753c071840 2024-04-17T10:35:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7541000640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c071840 0x7f753c158f90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f753000d800 tx=0x7f753000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7538c00640 1 -- 172.21.15.67:0/2381629151 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753000b6c0 con 0x7f753c071840 2024-04-17T10:35:41.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7538c00640 1 -- 172.21.15.67:0/2381629151 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7530019070 con 0x7f753c071840 2024-04-17T10:35:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7538c00640 1 -- 172.21.15.67:0/2381629151 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753000ba60 con 0x7f753c071840 2024-04-17T10:35:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753c0aaf50 con 0x7f753c071840 2024-04-17T10:35:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753c0a89a0 con 0x7f753c071840 2024-04-17T10:35:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.027+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753c070bb0 con 0x7f753c071840 2024-04-17T10:35:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.031+0000 7f7538c00640 1 -- 172.21.15.67:0/2381629151 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f753000bc00 con 0x7f753c071840 2024-04-17T10:35:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.031+0000 7f7538c00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f751c066f60 0x7f751c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.031+0000 7f7538c00640 1 -- 172.21.15.67:0/2381629151 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f7530085600 con 0x7f753c071840 2024-04-17T10:35:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.031+0000 7f753be00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f751c066f60 0x7f751c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.031+0000 7f753be00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f751c066f60 0x7f751c069420 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f75280ae000 tx=0x7f75280a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.035+0000 7f7538c00640 1 -- 172.21.15.67:0/2381629151 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7530055d00 con 0x7f753c071840 2024-04-17T10:35:41.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.203+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f753c074e30 con 0x7f753c071840 2024-04-17T10:35:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.203+0000 7f7538c00640 1 -- 172.21.15.67:0/2381629151 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16844 (secure 0 0 0) 0x7f7530059da0 con 0x7f753c071840 2024-04-17T10:35:41.193 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:41.193 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":141,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:35:38.092538+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:35:34.898907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-17T10:35:35.464996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:35:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f751c066f60 msgr2=0x7f751c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f751c066f60 0x7f751c069420 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f75280ae000 tx=0x7f75280a7000 comp rx=0 tx=0).stop 2024-04-17T10:35:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c071840 msgr2=0x7f753c158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c071840 0x7f753c158f90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f753000d800 tx=0x7f753000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 shutdown_connections 2024-04-17T10:35:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f751c066f60 0x7f751c069420 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f753c144aa0 0x7f753c0aa570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f753c1446c0 0x7f753c15acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 --2- 172.21.15.67:0/2381629151 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f753c071840 0x7f753c158f90 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 >> 172.21.15.67:0/2381629151 conn(0x7f753c0b34d0 msgr2=0x7f753c0b1440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 shutdown_connections 2024-04-17T10:35:41.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.207+0000 7f7543a00640 1 -- 172.21.15.67:0/2381629151 wait complete. 2024-04-17T10:35:41.215 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-17T10:35:41.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:35:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b31400640 1 Processor -- start 2024-04-17T10:35:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b31400640 1 -- start start 2024-04-17T10:35:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b31400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b2c15dd10 0x7f3b2c0b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b31400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0b0cc0 con 0x7f3b2c0714d0 2024-04-17T10:35:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b31400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b2c0b0fa0 con 0x7f3b2c0b57b0 2024-04-17T10:35:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b31400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b2c0b1280 con 0x7f3b2c15dd10 2024-04-17T10:35:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b2a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3b2c15dd10 0x7f3b2c0b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b2b200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3b2c0714d0 0x7f3b2c06c6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47136/0 (socket says 172.21.15.67:47136) 2024-04-17T10:35:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b2b200640 1 -- 172.21.15.67:0/4020623998 learned_addr learned my addr 172.21.15.67:0/4020623998 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b2a800640 1 -- 172.21.15.67:0/4020623998 >> v1:172.21.15.86:6789/0 conn(0x7f3b2c0b57b0 legacy=0x7f3b2c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b2a800640 1 -- 172.21.15.67:0/4020623998 >> v1:172.21.15.67:6789/0 conn(0x7f3b2c0714d0 legacy=0x7f3b2c06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.367+0000 7f3b2a800640 1 -- 172.21.15.67:0/4020623998 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c06bed0 con 0x7f3b2c15dd10 2024-04-17T10:35:41.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b2a800640 1 --2- 172.21.15.67:0/4020623998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c0b5270 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0097c0 tx=0x7f3b1c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b188daa259731dc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b29400640 1 -- 172.21.15.67:0/4020623998 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0b2070 con 0x7f3b2c15dd10 2024-04-17T10:35:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b29400640 1 -- 172.21.15.67:0/4020623998 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c0ad680 con 0x7f3b2c15dd10 2024-04-17T10:35:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b29400640 1 -- 172.21.15.67:0/4020623998 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c0ada50 con 0x7f3b2c15dd10 2024-04-17T10:35:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 -- 172.21.15.67:0/4020623998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 msgr2=0x7f3b2c0b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 --2- 172.21.15.67:0/4020623998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c0b5270 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0097c0 tx=0x7f3b1c01c3a0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 -- 172.21.15.67:0/4020623998 shutdown_connections 2024-04-17T10:35:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 --2- 172.21.15.67:0/4020623998 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c0b5270 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 -- 172.21.15.67:0/4020623998 >> 172.21.15.67:0/4020623998 conn(0x7f3b2c0b34d0 msgr2=0x7f3b2c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 -- 172.21.15.67:0/4020623998 shutdown_connections 2024-04-17T10:35:41.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 -- 172.21.15.67:0/4020623998 wait complete. 2024-04-17T10:35:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 Processor -- start 2024-04-17T10:35:41.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.371+0000 7f3b31400640 1 -- start start 2024-04-17T10:35:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b2c0714d0 0x7f3b2c14c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b2c168f20 0x7f3b2c151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c06da00 con 0x7f3b2c15dd10 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c06d880 con 0x7f3b2c0714d0 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c16bc70 con 0x7f3b2c168f20 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c160910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49262/0 (socket says 172.21.15.67:49262) 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 -- 172.21.15.67:0/2409936407 learned_addr learned my addr 172.21.15.67:0/2409936407 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b2a800640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b2c0714d0 0x7f3b2c14c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b2b200640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b2c168f20 0x7f3b2c151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 -- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b2c168f20 msgr2=0x7f3b2c151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b2c168f20 0x7f3b2c151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 -- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b2c0714d0 msgr2=0x7f3b2c14c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b2c0714d0 0x7f3b2c14c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 -- 172.21.15.67:0/2409936407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c161260 con 0x7f3b2c15dd10 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b2a800640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b2c0714d0 0x7f3b2c14c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:41.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b29e00640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c160910 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c480 tx=0x7f3b1800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b22a00640 1 -- 172.21.15.67:0/2409936407 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18012830 con 0x7f3b2c15dd10 2024-04-17T10:35:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b22a00640 1 -- 172.21.15.67:0/2409936407 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b18017070 con 0x7f3b2c15dd10 2024-04-17T10:35:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c0cb600 con 0x7f3b2c15dd10 2024-04-17T10:35:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b22a00640 1 -- 172.21.15.67:0/2409936407 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18012c30 con 0x7f3b2c15dd10 2024-04-17T10:35:41.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c0cbbc0 con 0x7f3b2c15dd10 2024-04-17T10:35:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c070bb0 con 0x7f3b2c15dd10 2024-04-17T10:35:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.375+0000 7f3b22a00640 1 -- 172.21.15.67:0/2409936407 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3b1801c460 con 0x7f3b2c15dd10 2024-04-17T10:35:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.379+0000 7f3b22a00640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f60 0x7f3b0c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.379+0000 7f3b22a00640 1 -- 172.21.15.67:0/2409936407 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f3b18085080 con 0x7f3b2c15dd10 2024-04-17T10:35:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.379+0000 7f3b2a800640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f60 0x7f3b0c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.379+0000 7f3b2a800640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f60 0x7f3b0c069420 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0af000 tx=0x7f3b1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.383+0000 7f3b22a00640 1 -- 172.21.15.67:0/2409936407 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3b18055ef0 con 0x7f3b2c15dd10 2024-04-17T10:35:41.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.539+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b2c074e30 con 0x7f3b0c066f60 2024-04-17T10:35:41.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.547+0000 7f3b22a00640 1 -- 172.21.15.67:0/2409936407 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252414 (secure 0 0 0) 0x7f3b2c074e30 con 0x7f3b0c066f60 2024-04-17T10:35:41.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:41.542 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:35:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f60 msgr2=0x7f3b0c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f60 0x7f3b0c069420 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c0af000 tx=0x7f3b1c008040 comp rx=0 tx=0).stop 2024-04-17T10:35:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 msgr2=0x7f3b2c160910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c160910 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c480 tx=0x7f3b1800c940 comp rx=0 tx=0).stop 2024-04-17T10:35:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 shutdown_connections 2024-04-17T10:35:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3b0c066f60 0x7f3b0c069420 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3b2c168f20 0x7f3b2c151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3b2c15dd10 0x7f3b2c160910 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 --2- 172.21.15.67:0/2409936407 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3b2c0714d0 0x7f3b2c14c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 >> 172.21.15.67:0/2409936407 conn(0x7f3b2c0b34d0 msgr2=0x7f3b2c0cf360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 shutdown_connections 2024-04-17T10:35:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.551+0000 7f3b31400640 1 -- 172.21.15.67:0/2409936407 wait complete. 2024-04-17T10:35:41.559 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-04-17T10:35:40.289677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":468268,"kb_used_data":14828,"kb_used_omap":12,"kb_used_meta":453299,"kb_avail":754506452,"statfs":{"total":773094113280,"available":772614606848,"internally_reserved":0,"allocated":15183872,"data_stored":7713533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":464178737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.227988"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114170+0000","last_change":"2024-04-17T10:35:37.103191+0000","last_active":"2024-04-17T10:35:37.114170+0000","last_peered":"2024-04-17T10:35:37.114170+0000","last_clean":"2024-04-17T10:35:37.114170+0000","last_became_active":"2024-04-17T10:35:37.102855+0000","last_became_peered":"2024-04-17T10:35:37.102855+0000","last_unstale":"2024-04-17T10:35:37.114170+0000","last_undegraded":"2024-04-17T10:35:37.114170+0000","last_fullsized":"2024-04-17T10:35:37.114170+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:59.713807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113399+0000","last_change":"2024-04-17T10:35:37.111687+0000","last_active":"2024-04-17T10:35:37.113399+0000","last_peered":"2024-04-17T10:35:37.113399+0000","last_clean":"2024-04-17T10:35:37.113399+0000","last_became_active":"2024-04-17T10:35:37.111407+0000","last_became_peered":"2024-04-17T10:35:37.111407+0000","last_unstale":"2024-04-17T10:35:37.113399+0000","last_undegraded":"2024-04-17T10:35:37.113399+0000","last_fullsized":"2024-04-17T10:35:37.113399+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:01:29.568978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110152+0000","last_change":"2024-04-17T10:35:37.109484+0000","last_active":"2024-04-17T10:35:37.110152+0000","last_peered":"2024-04-17T10:35:37.110152+0000","last_clean":"2024-04-17T10:35:37.110152+0000","last_became_active":"2024-04-17T10:35:37.108923+0000","last_became_peered":"2024-04-17T10:35:37.108923+0000","last_unstale":"2024-04-17T10:35:37.110152+0000","last_undegraded":"2024-04-17T10:35:37.110152+0000","last_fullsized":"2024-04-17T10:35:37.110152+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:40.033640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102374+0000","last_change":"2024-04-17T10:35:37.097969+0000","last_active":"2024-04-17T10:35:37.102374+0000","last_peered":"2024-04-17T10:35:37.102374+0000","last_clean":"2024-04-17T10:35:37.102374+0000","last_became_active":"2024-04-17T10:35:37.097676+0000","last_became_peered":"2024-04-17T10:35:37.097676+0000","last_unstale":"2024-04-17T10:35:37.102374+0000","last_undegraded":"2024-04-17T10:35:37.102374+0000","last_fullsized":"2024-04-17T10:35:37.102374+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:38.015381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114132+0000","last_change":"2024-04-17T10:35:37.110215+0000","last_active":"2024-04-17T10:35:37.114132+0000","last_peered":"2024-04-17T10:35:37.114132+0000","last_clean":"2024-04-17T10:35:37.114132+0000","last_became_active":"2024-04-17T10:35:37.107323+0000","last_became_peered":"2024-04-17T10:35:37.107323+0000","last_unstale":"2024-04-17T10:35:37.114132+0000","last_undegraded":"2024-04-17T10:35:37.114132+0000","last_fullsized":"2024-04-17T10:35:37.114132+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:48.289465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114010+0000","last_change":"2024-04-17T10:35:37.107605+0000","last_active":"2024-04-17T10:35:37.114010+0000","last_peered":"2024-04-17T10:35:37.114010+0000","last_clean":"2024-04-17T10:35:37.114010+0000","last_became_active":"2024-04-17T10:35:37.107273+0000","last_became_peered":"2024-04-17T10:35:37.107273+0000","last_unstale":"2024-04-17T10:35:37.114010+0000","last_undegraded":"2024-04-17T10:35:37.114010+0000","last_fullsized":"2024-04-17T10:35:37.114010+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:13.596314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110565+0000","last_change":"2024-04-17T10:35:37.101896+0000","last_active":"2024-04-17T10:35:37.110565+0000","last_peered":"2024-04-17T10:35:37.110565+0000","last_clean":"2024-04-17T10:35:37.110565+0000","last_became_active":"2024-04-17T10:35:37.101616+0000","last_became_peered":"2024-04-17T10:35:37.101616+0000","last_unstale":"2024-04-17T10:35:37.110565+0000","last_undegraded":"2024-04-17T10:35:37.110565+0000","last_fullsized":"2024-04-17T10:35:37.110565+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:41.067908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114022+0000","last_change":"2024-04-17T10:35:37.110162+0000","last_active":"2024-04-17T10:35:37.114022+0000","last_peered":"2024-04-17T10:35:37.114022+0000","last_clean":"2024-04-17T10:35:37.114022+0000","last_became_active":"2024-04-17T10:35:37.109915+0000","last_became_peered":"2024-04-17T10:35:37.109915+0000","last_unstale":"2024-04-17T10:35:37.114022+0000","last_undegraded":"2024-04-17T10:35:37.114022+0000","last_fullsized":"2024-04-17T10:35:37.114022+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:39:05.072399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110139+0000","last_change":"2024-04-17T10:35:37.103180+0000","last_active":"2024-04-17T10:35:37.110139+0000","last_peered":"2024-04-17T10:35:37.110139+0000","last_clean":"2024-04-17T10:35:37.110139+0000","last_became_active":"2024-04-17T10:35:37.102911+0000","last_became_peered":"2024-04-17T10:35:37.102911+0000","last_unstale":"2024-04-17T10:35:37.110139+0000","last_undegraded":"2024-04-17T10:35:37.110139+0000","last_fullsized":"2024-04-17T10:35:37.110139+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:05:21.313210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.097296+0000","last_change":"2024-04-17T10:35:37.091924+0000","last_active":"2024-04-17T10:35:37.097296+0000","last_peered":"2024-04-17T10:35:37.097296+0000","last_clean":"2024-04-17T10:35:37.097296+0000","last_became_active":"2024-04-17T10:35:37.091312+0000","last_became_peered":"2024-04-17T10:35:37.091312+0000","last_unstale":"2024-04-17T10:35:37.097296+0000","last_undegraded":"2024-04-17T10:35:37.097296+0000","last_fullsized":"2024-04-17T10:35:37.097296+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:04:37.360473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113752+0000","last_change":"2024-04-17T10:35:37.108750+0000","last_active":"2024-04-17T10:35:37.113752+0000","last_peered":"2024-04-17T10:35:37.113752+0000","last_clean":"2024-04-17T10:35:37.113752+0000","last_became_active":"2024-04-17T10:35:37.108420+0000","last_became_peered":"2024-04-17T10:35:37.108420+0000","last_unstale":"2024-04-17T10:35:37.113752+0000","last_undegraded":"2024-04-17T10:35:37.113752+0000","last_fullsized":"2024-04-17T10:35:37.113752+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:57.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102344+0000","last_change":"2024-04-17T10:35:37.097401+0000","last_active":"2024-04-17T10:35:37.102344+0000","last_peered":"2024-04-17T10:35:37.102344+0000","last_clean":"2024-04-17T10:35:37.102344+0000","last_became_active":"2024-04-17T10:35:37.094229+0000","last_became_peered":"2024-04-17T10:35:37.094229+0000","last_unstale":"2024-04-17T10:35:37.102344+0000","last_undegraded":"2024-04-17T10:35:37.102344+0000","last_fullsized":"2024-04-17T10:35:37.102344+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:20:37.121559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.092188+0000","last_change":"2024-04-17T10:35:37.089668+0000","last_active":"2024-04-17T10:35:37.092188+0000","last_peered":"2024-04-17T10:35:37.092188+0000","last_clean":"2024-04-17T10:35:37.092188+0000","last_became_active":"2024-04-17T10:35:37.089371+0000","last_became_peered":"2024-04-17T10:35:37.089371+0000","last_unstale":"2024-04-17T10:35:37.092188+0000","last_undegraded":"2024-04-17T10:35:37.092188+0000","last_fullsized":"2024-04-17T10:35:37.092188+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:32:24.858936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.098266+0000","last_change":"2024-04-17T10:35:37.093465+0000","last_active":"2024-04-17T10:35:37.098266+0000","last_peered":"2024-04-17T10:35:37.098266+0000","last_clean":"2024-04-17T10:35:37.098266+0000","last_became_active":"2024-04-17T10:35:37.093183+0000","last_became_peered":"2024-04-17T10:35:37.093183+0000","last_unstale":"2024-04-17T10:35:37.098266+0000","last_undegraded":"2024-04-17T10:35:37.098266+0000","last_fullsized":"2024-04-17T10:35:37.098266+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:04.404888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109460+0000","last_change":"2024-04-17T10:35:37.106785+0000","last_active":"2024-04-17T10:35:37.109460+0000","last_peered":"2024-04-17T10:35:37.109460+0000","last_clean":"2024-04-17T10:35:37.109460+0000","last_became_active":"2024-04-17T10:35:37.106338+0000","last_became_peered":"2024-04-17T10:35:37.106338+0000","last_unstale":"2024-04-17T10:35:37.109460+0000","last_undegraded":"2024-04-17T10:35:37.109460+0000","last_fullsized":"2024-04-17T10:35:37.109460+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:32.866813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109351+0000","last_change":"2024-04-17T10:35:37.103250+0000","last_active":"2024-04-17T10:35:37.109351+0000","last_peered":"2024-04-17T10:35:37.109351+0000","last_clean":"2024-04-17T10:35:37.109351+0000","last_became_active":"2024-04-17T10:35:37.102934+0000","last_became_peered":"2024-04-17T10:35:37.102934+0000","last_unstale":"2024-04-17T10:35:37.109351+0000","last_undegraded":"2024-04-17T10:35:37.109351+0000","last_fullsized":"2024-04-17T10:35:37.109351+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:00.673501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.101283+0000","last_change":"2024-04-17T10:35:37.094519+0000","last_active":"2024-04-17T10:35:37.101283+0000","last_peered":"2024-04-17T10:35:37.101283+0000","last_clean":"2024-04-17T10:35:37.101283+0000","last_became_active":"2024-04-17T10:35:37.094229+0000","last_became_peered":"2024-04-17T10:35:37.094229+0000","last_unstale":"2024-04-17T10:35:37.101283+0000","last_undegraded":"2024-04-17T10:35:37.101283+0000","last_fullsized":"2024-04-17T10:35:37.101283+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:06.809771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119711+0000","last_change":"2024-04-17T10:35:37.118400+0000","last_active":"2024-04-17T10:35:37.119711+0000","last_peered":"2024-04-17T10:35:37.119711+0000","last_clean":"2024-04-17T10:35:37.119711+0000","last_became_active":"2024-04-17T10:35:37.118144+0000","last_became_peered":"2024-04-17T10:35:37.118144+0000","last_unstale":"2024-04-17T10:35:37.119711+0000","last_undegraded":"2024-04-17T10:35:37.119711+0000","last_fullsized":"2024-04-17T10:35:37.119711+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:22.500261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102576+0000","last_change":"2024-04-17T10:35:37.098036+0000","last_active":"2024-04-17T10:35:37.102576+0000","last_peered":"2024-04-17T10:35:37.102576+0000","last_clean":"2024-04-17T10:35:37.102576+0000","last_became_active":"2024-04-17T10:35:37.097461+0000","last_became_peered":"2024-04-17T10:35:37.097461+0000","last_unstale":"2024-04-17T10:35:37.102576+0000","last_undegraded":"2024-04-17T10:35:37.102576+0000","last_fullsized":"2024-04-17T10:35:37.102576+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:13.596314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.100187+0000","last_change":"2024-04-17T10:35:37.097351+0000","last_active":"2024-04-17T10:35:37.100187+0000","last_peered":"2024-04-17T10:35:37.100187+0000","last_clean":"2024-04-17T10:35:37.100187+0000","last_became_active":"2024-04-17T10:35:37.096992+0000","last_became_peered":"2024-04-17T10:35:37.096992+0000","last_unstale":"2024-04-17T10:35:37.100187+0000","last_undegraded":"2024-04-17T10:35:37.100187+0000","last_fullsized":"2024-04-17T10:35:37.100187+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:42.313204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110359+0000","last_change":"2024-04-17T10:35:37.103635+0000","last_active":"2024-04-17T10:35:37.110359+0000","last_peered":"2024-04-17T10:35:37.110359+0000","last_clean":"2024-04-17T10:35:37.110359+0000","last_became_active":"2024-04-17T10:35:37.103375+0000","last_became_peered":"2024-04-17T10:35:37.103375+0000","last_unstale":"2024-04-17T10:35:37.110359+0000","last_undegraded":"2024-04-17T10:35:37.110359+0000","last_fullsized":"2024-04-17T10:35:37.110359+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:07:49.964511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114260+0000","last_change":"2024-04-17T10:35:37.105178+0000","last_active":"2024-04-17T10:35:37.114260+0000","last_peered":"2024-04-17T10:35:37.114260+0000","last_clean":"2024-04-17T10:35:37.114260+0000","last_became_active":"2024-04-17T10:35:37.104835+0000","last_became_peered":"2024-04-17T10:35:37.104835+0000","last_unstale":"2024-04-17T10:35:37.114260+0000","last_undegraded":"2024-04-17T10:35:37.114260+0000","last_fullsized":"2024-04-17T10:35:37.114260+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:46:22.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.111858+0000","last_change":"2024-04-17T10:35:37.107537+0000","last_active":"2024-04-17T10:35:37.111858+0000","last_peered":"2024-04-17T10:35:37.111858+0000","last_clean":"2024-04-17T10:35:37.111858+0000","last_became_active":"2024-04-17T10:35:37.107259+0000","last_became_peered":"2024-04-17T10:35:37.107259+0000","last_unstale":"2024-04-17T10:35:37.111858+0000","last_undegraded":"2024-04-17T10:35:37.111858+0000","last_fullsized":"2024-04-17T10:35:37.111858+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:50.603148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113647+0000","last_change":"2024-04-17T10:35:37.108739+0000","last_active":"2024-04-17T10:35:37.113647+0000","last_peered":"2024-04-17T10:35:37.113647+0000","last_clean":"2024-04-17T10:35:37.113647+0000","last_became_active":"2024-04-17T10:35:37.108406+0000","last_became_peered":"2024-04-17T10:35:37.108406+0000","last_unstale":"2024-04-17T10:35:37.113647+0000","last_undegraded":"2024-04-17T10:35:37.113647+0000","last_fullsized":"2024-04-17T10:35:37.113647+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:35:01.032056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116164+0000","last_change":"2024-04-17T10:35:37.110990+0000","last_active":"2024-04-17T10:35:37.116164+0000","last_peered":"2024-04-17T10:35:37.116164+0000","last_clean":"2024-04-17T10:35:37.116164+0000","last_became_active":"2024-04-17T10:35:37.105458+0000","last_became_peered":"2024-04-17T10:35:37.105458+0000","last_unstale":"2024-04-17T10:35:37.116164+0000","last_undegraded":"2024-04-17T10:35:37.116164+0000","last_fullsized":"2024-04-17T10:35:37.116164+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:36.302526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116110+0000","last_change":"2024-04-17T10:35:37.110999+0000","last_active":"2024-04-17T10:35:37.116110+0000","last_peered":"2024-04-17T10:35:37.116110+0000","last_clean":"2024-04-17T10:35:37.116110+0000","last_became_active":"2024-04-17T10:35:37.104269+0000","last_became_peered":"2024-04-17T10:35:37.104269+0000","last_unstale":"2024-04-17T10:35:37.116110+0000","last_undegraded":"2024-04-17T10:35:37.116110+0000","last_fullsized":"2024-04-17T10:35:37.116110+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:51.313852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.111202+0000","last_change":"2024-04-17T10:35:37.104308+0000","last_active":"2024-04-17T10:35:37.111202+0000","last_peered":"2024-04-17T10:35:37.111202+0000","last_clean":"2024-04-17T10:35:37.111202+0000","last_became_active":"2024-04-17T10:35:37.104061+0000","last_became_peered":"2024-04-17T10:35:37.104061+0000","last_unstale":"2024-04-17T10:35:37.111202+0000","last_undegraded":"2024-04-17T10:35:37.111202+0000","last_fullsized":"2024-04-17T10:35:37.111202+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:50.321997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.115082+0000","last_change":"2024-04-17T10:35:37.110709+0000","last_active":"2024-04-17T10:35:37.115082+0000","last_peered":"2024-04-17T10:35:37.115082+0000","last_clean":"2024-04-17T10:35:37.115082+0000","last_became_active":"2024-04-17T10:35:37.110439+0000","last_became_peered":"2024-04-17T10:35:37.110439+0000","last_unstale":"2024-04-17T10:35:37.115082+0000","last_undegraded":"2024-04-17T10:35:37.115082+0000","last_fullsized":"2024-04-17T10:35:37.115082+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:19.576363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.108002+0000","last_change":"2024-04-17T10:35:37.106870+0000","last_active":"2024-04-17T10:35:37.108002+0000","last_peered":"2024-04-17T10:35:37.108002+0000","last_clean":"2024-04-17T10:35:37.108002+0000","last_became_active":"2024-04-17T10:35:37.106421+0000","last_became_peered":"2024-04-17T10:35:37.106421+0000","last_unstale":"2024-04-17T10:35:37.108002+0000","last_undegraded":"2024-04-17T10:35:37.108002+0000","last_fullsized":"2024-04-17T10:35:37.108002+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:38.015381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.112498+0000","last_change":"2024-04-17T10:35:37.103984+0000","last_active":"2024-04-17T10:35:37.112498+0000","last_peered":"2024-04-17T10:35:37.112498+0000","last_clean":"2024-04-17T10:35:37.112498+0000","last_became_active":"2024-04-17T10:35:37.103707+0000","last_became_peered":"2024-04-17T10:35:37.103707+0000","last_unstale":"2024-04-17T10:35:37.112498+0000","last_undegraded":"2024-04-17T10:35:37.112498+0000","last_fullsized":"2024-04-17T10:35:37.112498+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:39.637819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105673+0000","last_change":"2024-04-17T10:35:37.097693+0000","last_active":"2024-04-17T10:35:37.105673+0000","last_peered":"2024-04-17T10:35:37.105673+0000","last_clean":"2024-04-17T10:35:37.105673+0000","last_became_active":"2024-04-17T10:35:37.097411+0000","last_became_peered":"2024-04-17T10:35:37.097411+0000","last_unstale":"2024-04-17T10:35:37.105673+0000","last_undegraded":"2024-04-17T10:35:37.105673+0000","last_fullsized":"2024-04-17T10:35:37.105673+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:39.398472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113997+0000","last_change":"2024-04-17T10:35:37.109167+0000","last_active":"2024-04-17T10:35:37.113997+0000","last_peered":"2024-04-17T10:35:37.113997+0000","last_clean":"2024-04-17T10:35:37.113997+0000","last_became_active":"2024-04-17T10:35:37.108883+0000","last_became_peered":"2024-04-17T10:35:37.108883+0000","last_unstale":"2024-04-17T10:35:37.113997+0000","last_undegraded":"2024-04-17T10:35:37.113997+0000","last_fullsized":"2024-04-17T10:35:37.113997+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:55:46.629651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.108694+0000","last_change":"2024-04-17T10:35:37.104788+0000","last_active":"2024-04-17T10:35:37.108694+0000","last_peered":"2024-04-17T10:35:37.108694+0000","last_clean":"2024-04-17T10:35:37.108694+0000","last_became_active":"2024-04-17T10:35:37.104197+0000","last_became_peered":"2024-04-17T10:35:37.104197+0000","last_unstale":"2024-04-17T10:35:37.108694+0000","last_undegraded":"2024-04-17T10:35:37.108694+0000","last_fullsized":"2024-04-17T10:35:37.108694+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:13.776507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"140'4","reported_seq":16,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.122006+0000","last_change":"2024-04-17T10:35:36.089282+0000","last_active":"2024-04-17T10:35:37.122006+0000","last_peered":"2024-04-17T10:35:37.122006+0000","last_clean":"2024-04-17T10:35:37.122006+0000","last_became_active":"2024-04-17T10:35:36.089023+0000","last_became_peered":"2024-04-17T10:35:36.089023+0000","last_unstale":"2024-04-17T10:35:37.122006+0000","last_undegraded":"2024-04-17T10:35:37.122006+0000","last_fullsized":"2024-04-17T10:35:37.122006+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:50.307672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109650+0000","last_change":"2024-04-17T10:35:37.107293+0000","last_active":"2024-04-17T10:35:37.109650+0000","last_peered":"2024-04-17T10:35:37.109650+0000","last_clean":"2024-04-17T10:35:37.109650+0000","last_became_active":"2024-04-17T10:35:37.106407+0000","last_became_peered":"2024-04-17T10:35:37.106407+0000","last_unstale":"2024-04-17T10:35:37.109650+0000","last_undegraded":"2024-04-17T10:35:37.109650+0000","last_fullsized":"2024-04-17T10:35:37.109650+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:02.027623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119190+0000","last_change":"2024-04-17T10:35:36.091651+0000","last_active":"2024-04-17T10:35:37.119190+0000","last_peered":"2024-04-17T10:35:37.119190+0000","last_clean":"2024-04-17T10:35:37.119190+0000","last_became_active":"2024-04-17T10:35:36.090988+0000","last_became_peered":"2024-04-17T10:35:36.090988+0000","last_unstale":"2024-04-17T10:35:37.119190+0000","last_undegraded":"2024-04-17T10:35:37.119190+0000","last_fullsized":"2024-04-17T10:35:37.119190+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:38.392292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113893+0000","last_change":"2024-04-17T10:35:37.109028+0000","last_active":"2024-04-17T10:35:37.113893+0000","last_peered":"2024-04-17T10:35:37.113893+0000","last_clean":"2024-04-17T10:35:37.113893+0000","last_became_active":"2024-04-17T10:35:37.108783+0000","last_became_peered":"2024-04-17T10:35:37.108783+0000","last_unstale":"2024-04-17T10:35:37.113893+0000","last_undegraded":"2024-04-17T10:35:37.113893+0000","last_fullsized":"2024-04-17T10:35:37.113893+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:43.471471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119735+0000","last_change":"2024-04-17T10:35:36.094939+0000","last_active":"2024-04-17T10:35:37.119735+0000","last_peered":"2024-04-17T10:35:37.119735+0000","last_clean":"2024-04-17T10:35:37.119735+0000","last_became_active":"2024-04-17T10:35:36.094630+0000","last_became_peered":"2024-04-17T10:35:36.094630+0000","last_unstale":"2024-04-17T10:35:37.119735+0000","last_undegraded":"2024-04-17T10:35:37.119735+0000","last_fullsized":"2024-04-17T10:35:37.119735+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:05:20.307030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110093+0000","last_change":"2024-04-17T10:35:37.107091+0000","last_active":"2024-04-17T10:35:37.110093+0000","last_peered":"2024-04-17T10:35:37.110093+0000","last_clean":"2024-04-17T10:35:37.110093+0000","last_became_active":"2024-04-17T10:35:37.106811+0000","last_became_peered":"2024-04-17T10:35:37.106811+0000","last_unstale":"2024-04-17T10:35:37.110093+0000","last_undegraded":"2024-04-17T10:35:37.110093+0000","last_fullsized":"2024-04-17T10:35:37.110093+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:32.647661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.122001+0000","last_change":"2024-04-17T10:35:36.087964+0000","last_active":"2024-04-17T10:35:37.122001+0000","last_peered":"2024-04-17T10:35:37.122001+0000","last_clean":"2024-04-17T10:35:37.122001+0000","last_became_active":"2024-04-17T10:35:36.087225+0000","last_became_peered":"2024-04-17T10:35:36.087225+0000","last_unstale":"2024-04-17T10:35:37.122001+0000","last_undegraded":"2024-04-17T10:35:37.122001+0000","last_fullsized":"2024-04-17T10:35:37.122001+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:39:04.066220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105318+0000","last_change":"2024-04-17T10:35:37.103106+0000","last_active":"2024-04-17T10:35:37.105318+0000","last_peered":"2024-04-17T10:35:37.105318+0000","last_clean":"2024-04-17T10:35:37.105318+0000","last_became_active":"2024-04-17T10:35:37.102826+0000","last_became_peered":"2024-04-17T10:35:37.102826+0000","last_unstale":"2024-04-17T10:35:37.105318+0000","last_undegraded":"2024-04-17T10:35:37.105318+0000","last_fullsized":"2024-04-17T10:35:37.105318+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:33.074742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113898+0000","last_change":"2024-04-17T10:35:36.088024+0000","last_active":"2024-04-17T10:35:37.113898+0000","last_peered":"2024-04-17T10:35:37.113898+0000","last_clean":"2024-04-17T10:35:37.113898+0000","last_became_active":"2024-04-17T10:35:36.087589+0000","last_became_peered":"2024-04-17T10:35:36.087589+0000","last_unstale":"2024-04-17T10:35:37.113898+0000","last_undegraded":"2024-04-17T10:35:37.113898+0000","last_fullsized":"2024-04-17T10:35:37.113898+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:11.646151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102492+0000","last_change":"2024-04-17T10:35:37.099028+0000","last_active":"2024-04-17T10:35:37.102492+0000","last_peered":"2024-04-17T10:35:37.102492+0000","last_clean":"2024-04-17T10:35:37.102492+0000","last_became_active":"2024-04-17T10:35:37.098779+0000","last_became_peered":"2024-04-17T10:35:37.098779+0000","last_unstale":"2024-04-17T10:35:37.102492+0000","last_undegraded":"2024-04-17T10:35:37.102492+0000","last_fullsized":"2024-04-17T10:35:37.102492+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:50.603148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121448+0000","last_change":"2024-04-17T10:35:36.093492+0000","last_active":"2024-04-17T10:35:37.121448+0000","last_peered":"2024-04-17T10:35:37.121448+0000","last_clean":"2024-04-17T10:35:37.121448+0000","last_became_active":"2024-04-17T10:35:36.092707+0000","last_became_peered":"2024-04-17T10:35:36.092707+0000","last_unstale":"2024-04-17T10:35:37.121448+0000","last_undegraded":"2024-04-17T10:35:37.121448+0000","last_fullsized":"2024-04-17T10:35:37.121448+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:40:53.106577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102206+0000","last_change":"2024-04-17T10:35:37.100996+0000","last_active":"2024-04-17T10:35:37.102206+0000","last_peered":"2024-04-17T10:35:37.102206+0000","last_clean":"2024-04-17T10:35:37.102206+0000","last_became_active":"2024-04-17T10:35:37.100706+0000","last_became_peered":"2024-04-17T10:35:37.100706+0000","last_unstale":"2024-04-17T10:35:37.102206+0000","last_undegraded":"2024-04-17T10:35:37.102206+0000","last_fullsized":"2024-04-17T10:35:37.102206+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:34.031060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.098571+0000","last_change":"2024-04-17T10:35:36.079916+0000","last_active":"2024-04-17T10:35:37.098571+0000","last_peered":"2024-04-17T10:35:37.098571+0000","last_clean":"2024-04-17T10:35:37.098571+0000","last_became_active":"2024-04-17T10:35:36.079636+0000","last_became_peered":"2024-04-17T10:35:36.079636+0000","last_unstale":"2024-04-17T10:35:37.098571+0000","last_undegraded":"2024-04-17T10:35:37.098571+0000","last_fullsized":"2024-04-17T10:35:37.098571+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:39.027460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110600+0000","last_change":"2024-04-17T10:35:37.099921+0000","last_active":"2024-04-17T10:35:37.110600+0000","last_peered":"2024-04-17T10:35:37.110600+0000","last_clean":"2024-04-17T10:35:37.110600+0000","last_became_active":"2024-04-17T10:35:37.099602+0000","last_became_peered":"2024-04-17T10:35:37.099602+0000","last_unstale":"2024-04-17T10:35:37.110600+0000","last_undegraded":"2024-04-17T10:35:37.110600+0000","last_fullsized":"2024-04-17T10:35:37.110600+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:06.637859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085546+0000","last_change":"2024-04-17T10:35:36.086559+0000","last_active":"2024-04-17T10:35:37.085546+0000","last_peered":"2024-04-17T10:35:37.085546+0000","last_clean":"2024-04-17T10:35:37.085546+0000","last_became_active":"2024-04-17T10:35:36.086108+0000","last_became_peered":"2024-04-17T10:35:36.086108+0000","last_unstale":"2024-04-17T10:35:37.085546+0000","last_undegraded":"2024-04-17T10:35:37.085546+0000","last_fullsized":"2024-04-17T10:35:37.085546+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:58.707627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.106014+0000","last_change":"2024-04-17T10:35:37.104776+0000","last_active":"2024-04-17T10:35:37.106014+0000","last_peered":"2024-04-17T10:35:37.106014+0000","last_clean":"2024-04-17T10:35:37.106014+0000","last_became_active":"2024-04-17T10:35:37.104514+0000","last_became_peered":"2024-04-17T10:35:37.104514+0000","last_unstale":"2024-04-17T10:35:37.106014+0000","last_undegraded":"2024-04-17T10:35:37.106014+0000","last_fullsized":"2024-04-17T10:35:37.106014+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:20:37.121559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.092498+0000","last_change":"2024-04-17T10:35:36.093199+0000","last_active":"2024-04-17T10:35:37.092498+0000","last_peered":"2024-04-17T10:35:37.092498+0000","last_clean":"2024-04-17T10:35:37.092498+0000","last_became_active":"2024-04-17T10:35:36.092893+0000","last_became_peered":"2024-04-17T10:35:36.092893+0000","last_unstale":"2024-04-17T10:35:37.092498+0000","last_undegraded":"2024-04-17T10:35:37.092498+0000","last_fullsized":"2024-04-17T10:35:37.092498+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:07:48.958331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113729+0000","last_change":"2024-04-17T10:35:37.108279+0000","last_active":"2024-04-17T10:35:37.113729+0000","last_peered":"2024-04-17T10:35:37.113729+0000","last_clean":"2024-04-17T10:35:37.113729+0000","last_became_active":"2024-04-17T10:35:37.107712+0000","last_became_peered":"2024-04-17T10:35:37.107712+0000","last_unstale":"2024-04-17T10:35:37.113729+0000","last_undegraded":"2024-04-17T10:35:37.113729+0000","last_fullsized":"2024-04-17T10:35:37.113729+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:11:53.706069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099563+0000","last_change":"2024-04-17T10:35:36.083117+0000","last_active":"2024-04-17T10:35:37.099563+0000","last_peered":"2024-04-17T10:35:37.099563+0000","last_clean":"2024-04-17T10:35:37.099563+0000","last_became_active":"2024-04-17T10:35:36.082819+0000","last_became_peered":"2024-04-17T10:35:36.082819+0000","last_unstale":"2024-04-17T10:35:37.099563+0000","last_undegraded":"2024-04-17T10:35:37.099563+0000","last_fullsized":"2024-04-17T10:35:37.099563+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:23.125545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119774+0000","last_change":"2024-04-17T10:35:37.118909+0000","last_active":"2024-04-17T10:35:37.119774+0000","last_peered":"2024-04-17T10:35:37.119774+0000","last_clean":"2024-04-17T10:35:37.119774+0000","last_became_active":"2024-04-17T10:35:37.118679+0000","last_became_peered":"2024-04-17T10:35:37.118679+0000","last_unstale":"2024-04-17T10:35:37.119774+0000","last_undegraded":"2024-04-17T10:35:37.119774+0000","last_fullsized":"2024-04-17T10:35:37.119774+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:42.970481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"140'3","reported_seq":15,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121642+0000","last_change":"2024-04-17T10:35:36.094567+0000","last_active":"2024-04-17T10:35:37.121642+0000","last_peered":"2024-04-17T10:35:37.121642+0000","last_clean":"2024-04-17T10:35:37.121642+0000","last_became_active":"2024-04-17T10:35:36.094065+0000","last_became_peered":"2024-04-17T10:35:36.094065+0000","last_unstale":"2024-04-17T10:35:37.121642+0000","last_undegraded":"2024-04-17T10:35:37.121642+0000","last_fullsized":"2024-04-17T10:35:37.121642+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:06.081208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105754+0000","last_change":"2024-04-17T10:35:37.104084+0000","last_active":"2024-04-17T10:35:37.105754+0000","last_peered":"2024-04-17T10:35:37.105754+0000","last_clean":"2024-04-17T10:35:37.105754+0000","last_became_active":"2024-04-17T10:35:37.103774+0000","last_became_peered":"2024-04-17T10:35:37.103774+0000","last_unstale":"2024-04-17T10:35:37.105754+0000","last_undegraded":"2024-04-17T10:35:37.105754+0000","last_fullsized":"2024-04-17T10:35:37.105754+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:06.809771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121906+0000","last_change":"2024-04-17T10:35:36.088996+0000","last_active":"2024-04-17T10:35:37.121906+0000","last_peered":"2024-04-17T10:35:37.121906+0000","last_clean":"2024-04-17T10:35:37.121906+0000","last_became_active":"2024-04-17T10:35:36.088680+0000","last_became_peered":"2024-04-17T10:35:36.088680+0000","last_unstale":"2024-04-17T10:35:37.121906+0000","last_undegraded":"2024-04-17T10:35:37.121906+0000","last_fullsized":"2024-04-17T10:35:37.121906+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:46:21.966356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.108898+0000","last_change":"2024-04-17T10:35:37.106008+0000","last_active":"2024-04-17T10:35:37.108898+0000","last_peered":"2024-04-17T10:35:37.108898+0000","last_clean":"2024-04-17T10:35:37.108898+0000","last_became_active":"2024-04-17T10:35:37.105679+0000","last_became_peered":"2024-04-17T10:35:37.105679+0000","last_unstale":"2024-04-17T10:35:37.108898+0000","last_undegraded":"2024-04-17T10:35:37.108898+0000","last_fullsized":"2024-04-17T10:35:37.108898+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:54:04.123300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085645+0000","last_change":"2024-04-17T10:35:36.084061+0000","last_active":"2024-04-17T10:35:37.085645+0000","last_peered":"2024-04-17T10:35:37.085645+0000","last_clean":"2024-04-17T10:35:37.085645+0000","last_became_active":"2024-04-17T10:35:36.083775+0000","last_became_peered":"2024-04-17T10:35:36.083775+0000","last_unstale":"2024-04-17T10:35:37.085645+0000","last_undegraded":"2024-04-17T10:35:37.085645+0000","last_fullsized":"2024-04-17T10:35:37.085645+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:16.962060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114161+0000","last_change":"2024-04-17T10:35:37.110618+0000","last_active":"2024-04-17T10:35:37.114161+0000","last_peered":"2024-04-17T10:35:37.114161+0000","last_clean":"2024-04-17T10:35:37.114161+0000","last_became_active":"2024-04-17T10:35:37.110358+0000","last_became_peered":"2024-04-17T10:35:37.110358+0000","last_unstale":"2024-04-17T10:35:37.114161+0000","last_undegraded":"2024-04-17T10:35:37.114161+0000","last_fullsized":"2024-04-17T10:35:37.114161+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:51.309229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085205+0000","last_change":"2024-04-17T10:35:36.091614+0000","last_active":"2024-04-17T10:35:37.085205+0000","last_peered":"2024-04-17T10:35:37.085205+0000","last_clean":"2024-04-17T10:35:37.085205+0000","last_became_active":"2024-04-17T10:35:36.088453+0000","last_became_peered":"2024-04-17T10:35:36.088453+0000","last_unstale":"2024-04-17T10:35:37.085205+0000","last_undegraded":"2024-04-17T10:35:37.085205+0000","last_fullsized":"2024-04-17T10:35:37.085205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:51.993595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110117+0000","last_change":"2024-04-17T10:35:37.109456+0000","last_active":"2024-04-17T10:35:37.110117+0000","last_peered":"2024-04-17T10:35:37.110117+0000","last_clean":"2024-04-17T10:35:37.110117+0000","last_became_active":"2024-04-17T10:35:37.109176+0000","last_became_peered":"2024-04-17T10:35:37.109176+0000","last_unstale":"2024-04-17T10:35:37.110117+0000","last_undegraded":"2024-04-17T10:35:37.110117+0000","last_fullsized":"2024-04-17T10:35:37.110117+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:21.025884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"140'8","reported_seq":20,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121894+0000","last_change":"2024-04-17T10:35:36.088905+0000","last_active":"2024-04-17T10:35:37.121894+0000","last_peered":"2024-04-17T10:35:37.121894+0000","last_clean":"2024-04-17T10:35:37.121894+0000","last_became_active":"2024-04-17T10:35:36.088593+0000","last_became_peered":"2024-04-17T10:35:36.088593+0000","last_unstale":"2024-04-17T10:35:37.121894+0000","last_undegraded":"2024-04-17T10:35:37.121894+0000","last_fullsized":"2024-04-17T10:35:37.121894+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:20.019704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099688+0000","last_change":"2024-04-17T10:35:37.093004+0000","last_active":"2024-04-17T10:35:37.099688+0000","last_peered":"2024-04-17T10:35:37.099688+0000","last_clean":"2024-04-17T10:35:37.099688+0000","last_became_active":"2024-04-17T10:35:37.092745+0000","last_became_peered":"2024-04-17T10:35:37.092745+0000","last_unstale":"2024-04-17T10:35:37.099688+0000","last_undegraded":"2024-04-17T10:35:37.099688+0000","last_fullsized":"2024-04-17T10:35:37.099688+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:50.321997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121933+0000","last_change":"2024-04-17T10:35:36.090837+0000","last_active":"2024-04-17T10:35:37.121933+0000","last_peered":"2024-04-17T10:35:37.121933+0000","last_clean":"2024-04-17T10:35:37.121933+0000","last_became_active":"2024-04-17T10:35:36.090538+0000","last_became_peered":"2024-04-17T10:35:36.090538+0000","last_unstale":"2024-04-17T10:35:37.121933+0000","last_undegraded":"2024-04-17T10:35:37.121933+0000","last_fullsized":"2024-04-17T10:35:37.121933+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:04:00.312420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102603+0000","last_change":"2024-04-17T10:35:37.098266+0000","last_active":"2024-04-17T10:35:37.102603+0000","last_peered":"2024-04-17T10:35:37.102603+0000","last_clean":"2024-04-17T10:35:37.102603+0000","last_became_active":"2024-04-17T10:35:37.097738+0000","last_became_peered":"2024-04-17T10:35:37.097738+0000","last_unstale":"2024-04-17T10:35:37.102603+0000","last_undegraded":"2024-04-17T10:35:37.102603+0000","last_fullsized":"2024-04-17T10:35:37.102603+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:20:24.797449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.118580+0000","last_change":"2024-04-17T10:35:36.093074+0000","last_active":"2024-04-17T10:35:37.118580+0000","last_peered":"2024-04-17T10:35:37.118580+0000","last_clean":"2024-04-17T10:35:37.118580+0000","last_became_active":"2024-04-17T10:35:36.092772+0000","last_became_peered":"2024-04-17T10:35:36.092772+0000","last_unstale":"2024-04-17T10:35:37.118580+0000","last_undegraded":"2024-04-17T10:35:37.118580+0000","last_fullsized":"2024-04-17T10:35:37.118580+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:47:59.667321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114419+0000","last_change":"2024-04-17T10:35:37.110167+0000","last_active":"2024-04-17T10:35:37.114419+0000","last_peered":"2024-04-17T10:35:37.114419+0000","last_clean":"2024-04-17T10:35:37.114419+0000","last_became_active":"2024-04-17T10:35:37.104639+0000","last_became_peered":"2024-04-17T10:35:37.104639+0000","last_unstale":"2024-04-17T10:35:37.114419+0000","last_undegraded":"2024-04-17T10:35:37.114419+0000","last_fullsized":"2024-04-17T10:35:37.114419+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:40:54.112757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121077+0000","last_change":"2024-04-17T10:35:36.093466+0000","last_active":"2024-04-17T10:35:37.121077+0000","last_peered":"2024-04-17T10:35:37.121077+0000","last_clean":"2024-04-17T10:35:37.121077+0000","last_became_active":"2024-04-17T10:35:36.093108+0000","last_became_peered":"2024-04-17T10:35:36.093108+0000","last_unstale":"2024-04-17T10:35:37.121077+0000","last_undegraded":"2024-04-17T10:35:37.121077+0000","last_fullsized":"2024-04-17T10:35:37.121077+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:47.980225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.111155+0000","last_change":"2024-04-17T10:35:37.102014+0000","last_active":"2024-04-17T10:35:37.111155+0000","last_peered":"2024-04-17T10:35:37.111155+0000","last_clean":"2024-04-17T10:35:37.111155+0000","last_became_active":"2024-04-17T10:35:37.101738+0000","last_became_peered":"2024-04-17T10:35:37.101738+0000","last_unstale":"2024-04-17T10:35:37.111155+0000","last_undegraded":"2024-04-17T10:35:37.111155+0000","last_fullsized":"2024-04-17T10:35:37.111155+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:16:30.273467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085540+0000","last_change":"2024-04-17T10:35:36.089996+0000","last_active":"2024-04-17T10:35:37.085540+0000","last_peered":"2024-04-17T10:35:37.085540+0000","last_clean":"2024-04-17T10:35:37.085540+0000","last_became_active":"2024-04-17T10:35:36.089182+0000","last_became_peered":"2024-04-17T10:35:36.089182+0000","last_unstale":"2024-04-17T10:35:37.085540+0000","last_undegraded":"2024-04-17T10:35:37.085540+0000","last_fullsized":"2024-04-17T10:35:37.085540+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:37.196771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.104810+0000","last_change":"2024-04-17T10:35:37.102543+0000","last_active":"2024-04-17T10:35:37.104810+0000","last_peered":"2024-04-17T10:35:37.104810+0000","last_clean":"2024-04-17T10:35:37.104810+0000","last_became_active":"2024-04-17T10:35:37.096287+0000","last_became_peered":"2024-04-17T10:35:37.096287+0000","last_unstale":"2024-04-17T10:35:37.104810+0000","last_undegraded":"2024-04-17T10:35:37.104810+0000","last_fullsized":"2024-04-17T10:35:37.104810+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:48.289465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.122366+0000","last_change":"2024-04-17T10:35:36.087132+0000","last_active":"2024-04-17T10:35:37.122366+0000","last_peered":"2024-04-17T10:35:37.122366+0000","last_clean":"2024-04-17T10:35:37.122366+0000","last_became_active":"2024-04-17T10:35:36.086802+0000","last_became_peered":"2024-04-17T10:35:36.086802+0000","last_unstale":"2024-04-17T10:35:37.122366+0000","last_undegraded":"2024-04-17T10:35:37.122366+0000","last_fullsized":"2024-04-17T10:35:37.122366+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:01:47.106270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109479+0000","last_change":"2024-04-17T10:35:37.107132+0000","last_active":"2024-04-17T10:35:37.109479+0000","last_peered":"2024-04-17T10:35:37.109479+0000","last_clean":"2024-04-17T10:35:37.109479+0000","last_became_active":"2024-04-17T10:35:37.106868+0000","last_became_peered":"2024-04-17T10:35:37.106868+0000","last_unstale":"2024-04-17T10:35:37.109479+0000","last_undegraded":"2024-04-17T10:35:37.109479+0000","last_fullsized":"2024-04-17T10:35:37.109479+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:29:41.010378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102189+0000","last_change":"2024-04-17T10:35:36.084917+0000","last_active":"2024-04-17T10:35:37.102189+0000","last_peered":"2024-04-17T10:35:37.102189+0000","last_clean":"2024-04-17T10:35:37.102189+0000","last_became_active":"2024-04-17T10:35:36.083136+0000","last_became_peered":"2024-04-17T10:35:36.083136+0000","last_unstale":"2024-04-17T10:35:37.102189+0000","last_undegraded":"2024-04-17T10:35:37.102189+0000","last_fullsized":"2024-04-17T10:35:37.102189+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:03.398709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109775+0000","last_change":"2024-04-17T10:35:37.105951+0000","last_active":"2024-04-17T10:35:37.109775+0000","last_peered":"2024-04-17T10:35:37.109775+0000","last_clean":"2024-04-17T10:35:37.109775+0000","last_became_active":"2024-04-17T10:35:37.105695+0000","last_became_peered":"2024-04-17T10:35:37.105695+0000","last_unstale":"2024-04-17T10:35:37.109775+0000","last_undegraded":"2024-04-17T10:35:37.109775+0000","last_fullsized":"2024-04-17T10:35:37.109775+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:51.309229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.088002+0000","last_change":"2024-04-17T10:35:36.087908+0000","last_active":"2024-04-17T10:35:37.088002+0000","last_peered":"2024-04-17T10:35:37.088002+0000","last_clean":"2024-04-17T10:35:37.088002+0000","last_became_active":"2024-04-17T10:35:36.087618+0000","last_became_peered":"2024-04-17T10:35:36.087618+0000","last_unstale":"2024-04-17T10:35:37.088002+0000","last_undegraded":"2024-04-17T10:35:37.088002+0000","last_fullsized":"2024-04-17T10:35:37.088002+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:16:15.343014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109346+0000","last_change":"2024-04-17T10:35:37.105926+0000","last_active":"2024-04-17T10:35:37.109346+0000","last_peered":"2024-04-17T10:35:37.109346+0000","last_clean":"2024-04-17T10:35:37.109346+0000","last_became_active":"2024-04-17T10:35:37.105646+0000","last_became_peered":"2024-04-17T10:35:37.105646+0000","last_unstale":"2024-04-17T10:35:37.109346+0000","last_undegraded":"2024-04-17T10:35:37.109346+0000","last_fullsized":"2024-04-17T10:35:37.109346+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:43.913804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.092837+0000","last_change":"2024-04-17T10:35:36.088396+0000","last_active":"2024-04-17T10:35:37.092837+0000","last_peered":"2024-04-17T10:35:37.092837+0000","last_clean":"2024-04-17T10:35:37.092837+0000","last_became_active":"2024-04-17T10:35:36.087752+0000","last_became_peered":"2024-04-17T10:35:36.087752+0000","last_unstale":"2024-04-17T10:35:37.092837+0000","last_undegraded":"2024-04-17T10:35:37.092837+0000","last_fullsized":"2024-04-17T10:35:37.092837+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:16.962060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114153+0000","last_change":"2024-04-17T10:35:37.107607+0000","last_active":"2024-04-17T10:35:37.114153+0000","last_peered":"2024-04-17T10:35:37.114153+0000","last_clean":"2024-04-17T10:35:37.114153+0000","last_became_active":"2024-04-17T10:35:37.107339+0000","last_became_peered":"2024-04-17T10:35:37.107339+0000","last_unstale":"2024-04-17T10:35:37.114153+0000","last_undegraded":"2024-04-17T10:35:37.114153+0000","last_fullsized":"2024-04-17T10:35:37.114153+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:20:24.797449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114389+0000","last_change":"2024-04-17T10:35:36.090786+0000","last_active":"2024-04-17T10:35:37.114389+0000","last_peered":"2024-04-17T10:35:37.114389+0000","last_clean":"2024-04-17T10:35:37.114389+0000","last_became_active":"2024-04-17T10:35:36.089211+0000","last_became_peered":"2024-04-17T10:35:36.089211+0000","last_unstale":"2024-04-17T10:35:37.114389+0000","last_undegraded":"2024-04-17T10:35:37.114389+0000","last_fullsized":"2024-04-17T10:35:37.114389+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:15:28.133381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.096020+0000","last_change":"2024-04-17T10:35:37.090765+0000","last_active":"2024-04-17T10:35:37.096020+0000","last_peered":"2024-04-17T10:35:37.096020+0000","last_clean":"2024-04-17T10:35:37.096020+0000","last_became_active":"2024-04-17T10:35:37.090501+0000","last_became_peered":"2024-04-17T10:35:37.090501+0000","last_unstale":"2024-04-17T10:35:37.096020+0000","last_undegraded":"2024-04-17T10:35:37.096020+0000","last_fullsized":"2024-04-17T10:35:37.096020+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:12.652331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099661+0000","last_change":"2024-04-17T10:35:36.089744+0000","last_active":"2024-04-17T10:35:37.099661+0000","last_peered":"2024-04-17T10:35:37.099661+0000","last_clean":"2024-04-17T10:35:37.099661+0000","last_became_active":"2024-04-17T10:35:36.089476+0000","last_became_peered":"2024-04-17T10:35:36.089476+0000","last_unstale":"2024-04-17T10:35:37.099661+0000","last_undegraded":"2024-04-17T10:35:37.099661+0000","last_fullsized":"2024-04-17T10:35:37.099661+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:27.127929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116133+0000","last_change":"2024-04-17T10:35:37.111848+0000","last_active":"2024-04-17T10:35:37.116133+0000","last_peered":"2024-04-17T10:35:37.116133+0000","last_clean":"2024-04-17T10:35:37.116133+0000","last_became_active":"2024-04-17T10:35:37.111586+0000","last_became_peered":"2024-04-17T10:35:37.111586+0000","last_unstale":"2024-04-17T10:35:37.116133+0000","last_undegraded":"2024-04-17T10:35:37.116133+0000","last_fullsized":"2024-04-17T10:35:37.116133+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:32:24.858936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.095476+0000","last_change":"2024-04-17T10:35:36.083940+0000","last_active":"2024-04-17T10:35:37.095476+0000","last_peered":"2024-04-17T10:35:37.095476+0000","last_clean":"2024-04-17T10:35:37.095476+0000","last_became_active":"2024-04-17T10:35:36.082901+0000","last_became_peered":"2024-04-17T10:35:36.082901+0000","last_unstale":"2024-04-17T10:35:37.095476+0000","last_undegraded":"2024-04-17T10:35:37.095476+0000","last_fullsized":"2024-04-17T10:35:37.095476+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:09.890350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.095127+0000","last_change":"2024-04-17T10:35:37.092021+0000","last_active":"2024-04-17T10:35:37.095127+0000","last_peered":"2024-04-17T10:35:37.095127+0000","last_clean":"2024-04-17T10:35:37.095127+0000","last_became_active":"2024-04-17T10:35:37.091586+0000","last_became_peered":"2024-04-17T10:35:37.091586+0000","last_unstale":"2024-04-17T10:35:37.095127+0000","last_undegraded":"2024-04-17T10:35:37.095127+0000","last_fullsized":"2024-04-17T10:35:37.095127+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:53.265621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.086568+0000","last_change":"2024-04-17T10:35:36.091248+0000","last_active":"2024-04-17T10:35:37.086568+0000","last_peered":"2024-04-17T10:35:37.086568+0000","last_clean":"2024-04-17T10:35:37.086568+0000","last_became_active":"2024-04-17T10:35:36.091008+0000","last_became_peered":"2024-04-17T10:35:36.091008+0000","last_unstale":"2024-04-17T10:35:37.086568+0000","last_undegraded":"2024-04-17T10:35:37.086568+0000","last_fullsized":"2024-04-17T10:35:37.086568+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:35.296346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105334+0000","last_change":"2024-04-17T10:35:37.097301+0000","last_active":"2024-04-17T10:35:37.105334+0000","last_peered":"2024-04-17T10:35:37.105334+0000","last_clean":"2024-04-17T10:35:37.105334+0000","last_became_active":"2024-04-17T10:35:37.096913+0000","last_became_peered":"2024-04-17T10:35:37.096913+0000","last_unstale":"2024-04-17T10:35:37.105334+0000","last_undegraded":"2024-04-17T10:35:37.105334+0000","last_fullsized":"2024-04-17T10:35:37.105334+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:59.902902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099358+0000","last_change":"2024-04-17T10:35:36.090314+0000","last_active":"2024-04-17T10:35:37.099358+0000","last_peered":"2024-04-17T10:35:37.099358+0000","last_clean":"2024-04-17T10:35:37.099358+0000","last_became_active":"2024-04-17T10:35:36.090030+0000","last_became_peered":"2024-04-17T10:35:36.090030+0000","last_unstale":"2024-04-17T10:35:37.099358+0000","last_undegraded":"2024-04-17T10:35:37.099358+0000","last_fullsized":"2024-04-17T10:35:37.099358+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:30.198188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.094585+0000","last_change":"2024-04-17T10:35:37.090132+0000","last_active":"2024-04-17T10:35:37.094585+0000","last_peered":"2024-04-17T10:35:37.094585+0000","last_clean":"2024-04-17T10:35:37.094585+0000","last_became_active":"2024-04-17T10:35:37.089836+0000","last_became_peered":"2024-04-17T10:35:37.089836+0000","last_unstale":"2024-04-17T10:35:37.094585+0000","last_undegraded":"2024-04-17T10:35:37.094585+0000","last_fullsized":"2024-04-17T10:35:37.094585+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:59.902902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.100607+0000","last_change":"2024-04-17T10:35:36.077405+0000","last_active":"2024-04-17T10:35:37.100607+0000","last_peered":"2024-04-17T10:35:37.100607+0000","last_clean":"2024-04-17T10:35:37.100607+0000","last_became_active":"2024-04-17T10:35:36.077141+0000","last_became_peered":"2024-04-17T10:35:36.077141+0000","last_unstale":"2024-04-17T10:35:37.100607+0000","last_undegraded":"2024-04-17T10:35:37.100607+0000","last_fullsized":"2024-04-17T10:35:37.100607+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:07.895511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.115247+0000","last_change":"2024-04-17T10:35:37.112239+0000","last_active":"2024-04-17T10:35:37.115247+0000","last_peered":"2024-04-17T10:35:37.115247+0000","last_clean":"2024-04-17T10:35:37.115247+0000","last_became_active":"2024-04-17T10:35:37.111944+0000","last_became_peered":"2024-04-17T10:35:37.111944+0000","last_unstale":"2024-04-17T10:35:37.115247+0000","last_undegraded":"2024-04-17T10:35:37.115247+0000","last_fullsized":"2024-04-17T10:35:37.115247+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:31.204368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.098721+0000","last_change":"2024-04-17T10:35:36.083966+0000","last_active":"2024-04-17T10:35:37.098721+0000","last_peered":"2024-04-17T10:35:37.098721+0000","last_clean":"2024-04-17T10:35:37.098721+0000","last_became_active":"2024-04-17T10:35:36.083616+0000","last_became_peered":"2024-04-17T10:35:36.083616+0000","last_unstale":"2024-04-17T10:35:37.098721+0000","last_undegraded":"2024-04-17T10:35:37.098721+0000","last_fullsized":"2024-04-17T10:35:37.098721+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:01:28.562798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116041+0000","last_change":"2024-04-17T10:35:37.110975+0000","last_active":"2024-04-17T10:35:37.116041+0000","last_peered":"2024-04-17T10:35:37.116041+0000","last_clean":"2024-04-17T10:35:37.116041+0000","last_became_active":"2024-04-17T10:35:37.110718+0000","last_became_peered":"2024-04-17T10:35:37.110718+0000","last_unstale":"2024-04-17T10:35:37.116041+0000","last_undegraded":"2024-04-17T10:35:37.116041+0000","last_fullsized":"2024-04-17T10:35:37.116041+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:48.986405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.088090+0000","last_change":"2024-04-17T10:35:36.093527+0000","last_active":"2024-04-17T10:35:37.088090+0000","last_peered":"2024-04-17T10:35:37.088090+0000","last_clean":"2024-04-17T10:35:37.088090+0000","last_became_active":"2024-04-17T10:35:36.093234+0000","last_became_peered":"2024-04-17T10:35:36.093234+0000","last_unstale":"2024-04-17T10:35:37.088090+0000","last_undegraded":"2024-04-17T10:35:37.088090+0000","last_fullsized":"2024-04-17T10:35:37.088090+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:52.259442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113503+0000","last_change":"2024-04-17T10:35:37.108330+0000","last_active":"2024-04-17T10:35:37.113503+0000","last_peered":"2024-04-17T10:35:37.113503+0000","last_clean":"2024-04-17T10:35:37.113503+0000","last_became_active":"2024-04-17T10:35:37.108027+0000","last_became_peered":"2024-04-17T10:35:37.108027+0000","last_unstale":"2024-04-17T10:35:37.113503+0000","last_undegraded":"2024-04-17T10:35:37.113503+0000","last_fullsized":"2024-04-17T10:35:37.113503+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:36:20.016329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"140'6","reported_seq":18,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.126711+0000","last_change":"2024-04-17T10:35:36.088856+0000","last_active":"2024-04-17T10:35:37.126711+0000","last_peered":"2024-04-17T10:35:37.126711+0000","last_clean":"2024-04-17T10:35:37.126711+0000","last_became_active":"2024-04-17T10:35:36.088591+0000","last_became_peered":"2024-04-17T10:35:36.088591+0000","last_unstale":"2024-04-17T10:35:37.126711+0000","last_undegraded":"2024-04-17T10:35:37.126711+0000","last_fullsized":"2024-04-17T10:35:37.126711+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:47:36.588267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":391,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.088152+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:35:37.088152+0000","last_peered":"2024-04-17T10:35:37.088152+0000","last_clean":"2024-04-17T10:35:37.088152+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:35:37.088152+0000","last_undegraded":"2024-04-17T10:35:37.088152+0000","last_fullsized":"2024-04-17T10:35:37.088152+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574983,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55676,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316164,"statfs":{"total":96636764160,"available":96579751936,"internally_reserved":0,"allocated":2338816,"data_stored":1405782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574981,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55092,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316748,"statfs":{"total":96636764160,"available":96580349952,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50936,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320904,"statfs":{"total":96636764160,"available":96584605696,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59252,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312588,"statfs":{"total":96636764160,"available":96576090112,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81404,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290436,"statfs":{"total":96636764160,"available":96553406464,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55096,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316744,"statfs":{"total":96636764160,"available":96580345856,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50940,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320900,"statfs":{"total":96636764160,"available":96584601600,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574982,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59872,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311968,"statfs":{"total":96636764160,"available":96575455232,"internally_reserved":0,"allocated":2375680,"data_stored":1415267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:35:41.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-17T10:35:41.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.751+0000 7f0515800640 1 Processor -- start 2024-04-17T10:35:41.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.751+0000 7f0515800640 1 -- start start 2024-04-17T10:35:41.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f0515800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05100cc250 0x7f05100cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f0515800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f05100cdcf0 0x7f05100cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f0515800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051006b100 con 0x7f05100cb260 2024-04-17T10:35:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f0515800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05100cd150 con 0x7f05100cc250 2024-04-17T10:35:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f0515800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051006b3e0 con 0x7f05100cdcf0 2024-04-17T10:35:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050ea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f05100cc250 0x7f05100cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f0507e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f05100cb260 0x7f05100cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47140/0 (socket says 172.21.15.67:47140) 2024-04-17T10:35:41.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f0507e00640 1 -- 172.21.15.67:0/3973053637 learned_addr learned my addr 172.21.15.67:0/3973053637 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050f400640 1 --2- 172.21.15.67:0/3973053637 >> v2:172.21.15.86:3300/0 conn(0x7f05100cdcf0 0x7f05100cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050ea00640 1 -- 172.21.15.67:0/3973053637 >> v2:172.21.15.86:3300/0 conn(0x7f05100cdcf0 msgr2=0x7f05100cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050ea00640 1 --2- 172.21.15.67:0/3973053637 >> v2:172.21.15.86:3300/0 conn(0x7f05100cdcf0 0x7f05100cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050ea00640 1 -- 172.21.15.67:0/3973053637 >> v1:172.21.15.67:6789/0 conn(0x7f05100cb260 legacy=0x7f05100cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050e000640 1 -- 172.21.15.67:0/3973053637 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4914032 0 0) 0x7f051006b100 con 0x7f05100cb260 2024-04-17T10:35:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050ea00640 1 -- 172.21.15.67:0/3973053637 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051006b6c0 con 0x7f05100cc250 2024-04-17T10:35:41.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.755+0000 7f050f400640 1 --2- 172.21.15.67:0/3973053637 >> v2:172.21.15.86:3300/0 conn(0x7f05100cdcf0 0x7f05100cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f050ea00640 1 --2- 172.21.15.67:0/3973053637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f05100cad20 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f0510071b20 tx=0x7f050001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=232f1a81462e53ad server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f050e000640 1 -- 172.21.15.67:0/3973053637 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05000b0070 con 0x7f05100cc250 2024-04-17T10:35:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f050e000640 1 -- 172.21.15.67:0/3973053637 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05000ab600 con 0x7f05100cc250 2024-04-17T10:35:41.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f050e000640 1 -- 172.21.15.67:0/3973053637 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05000ab9d0 con 0x7f05100cc250 2024-04-17T10:35:41.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 -- 172.21.15.67:0/3973053637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 msgr2=0x7f05100cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 --2- 172.21.15.67:0/3973053637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f05100cad20 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f0510071b20 tx=0x7f050001c480 comp rx=0 tx=0).stop 2024-04-17T10:35:41.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 -- 172.21.15.67:0/3973053637 shutdown_connections 2024-04-17T10:35:41.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 --2- 172.21.15.67:0/3973053637 >> v2:172.21.15.86:3300/0 conn(0x7f05100cdcf0 0x7f05100cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 --2- 172.21.15.67:0/3973053637 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f05100cad20 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 -- 172.21.15.67:0/3973053637 >> 172.21.15.67:0/3973053637 conn(0x7f05100b22a0 msgr2=0x7f05100b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 -- 172.21.15.67:0/3973053637 shutdown_connections 2024-04-17T10:35:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 -- 172.21.15.67:0/3973053637 wait complete. 2024-04-17T10:35:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 Processor -- start 2024-04-17T10:35:41.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.759+0000 7f0515800640 1 -- start start 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05100cb260 0x7f051015eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f0510153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05100cdcf0 0x7f0510153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05100d02e0 con 0x7f05100cc250 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05100d0160 con 0x7f05100cb260 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051016b100 con 0x7f05100cdcf0 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f0510153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f0510153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49276/0 (socket says 172.21.15.67:49276) 2024-04-17T10:35:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 -- 172.21.15.67:0/2516582498 learned_addr learned my addr 172.21.15.67:0/2516582498 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:41.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f050ea00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05100cb260 0x7f051015eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f050f400640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05100cdcf0 0x7f0510153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 -- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05100cdcf0 msgr2=0x7f0510153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05100cdcf0 0x7f0510153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 -- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05100cb260 msgr2=0x7f051015eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05100cb260 0x7f051015eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 -- 172.21.15.67:0/2516582498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0510152b90 con 0x7f05100cc250 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f050ea00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05100cb260 0x7f051015eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0507e00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f0510153600 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c520 tx=0x7f04f800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0506a00640 1 -- 172.21.15.67:0/2516582498 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8018070 con 0x7f05100cc250 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0506a00640 1 -- 172.21.15.67:0/2516582498 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f80074e0 con 0x7f05100cc250 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0506a00640 1 -- 172.21.15.67:0/2516582498 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f80049f0 con 0x7f05100cc250 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05101577a0 con 0x7f05100cc250 2024-04-17T10:35:41.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0510157d60 con 0x7f05100cc250 2024-04-17T10:35:41.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0510154840 con 0x7f05100cc250 2024-04-17T10:35:41.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.763+0000 7f0506a00640 1 -- 172.21.15.67:0/2516582498 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f04f801b070 con 0x7f05100cc250 2024-04-17T10:35:41.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.767+0000 7f0506a00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:41.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.767+0000 7f0506a00640 1 -- 172.21.15.67:0/2516582498 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f04f80845f0 con 0x7f05100cc250 2024-04-17T10:35:41.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.767+0000 7f050ea00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:41.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.767+0000 7f050ea00640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f05000ad000 tx=0x7f05000a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:41.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.771+0000 7f0506a00640 1 -- 172.21.15.67:0/2516582498 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f04f8055500 con 0x7f05100cc250 2024-04-17T10:35:41.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.927+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 --> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f051009e900 con 0x7f04ec066f60 2024-04-17T10:35:41.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.935+0000 7f0506a00640 1 -- 172.21.15.67:0/2516582498 <== mgr.4100 v2:172.21.15.86:6832/4206025909 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+252414 (secure 0 0 0) 0x7f051009e900 con 0x7f04ec066f60 2024-04-17T10:35:41.925 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:41.927 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 msgr2=0x7f04ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f05000ad000 tx=0x7f05000a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 msgr2=0x7f0510153600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f0510153600 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f04f800c520 tx=0x7f04f800c9e0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 shutdown_connections 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f04ec066f60 0x7f04ec069420 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f05100cdcf0 0x7f0510153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f05100cc250 0x7f0510153600 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 --2- 172.21.15.67:0/2516582498 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f05100cb260 0x7f051015eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 >> 172.21.15.67:0/2516582498 conn(0x7f05100b22a0 msgr2=0x7f05100cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:41.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 shutdown_connections 2024-04-17T10:35:41.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:41.939+0000 7f0515800640 1 -- 172.21.15.67:0/2516582498 wait complete. 2024-04-17T10:35:41.947 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-04-17T10:35:40.289677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":53,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":468268,"kb_used_data":14828,"kb_used_omap":12,"kb_used_meta":453299,"kb_avail":754506452,"statfs":{"total":773094113280,"available":772614606848,"internally_reserved":0,"allocated":15183872,"data_stored":7713533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":464178737},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.227988"},"pg_stats":[{"pgid":"22.3f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114170+0000","last_change":"2024-04-17T10:35:37.103191+0000","last_active":"2024-04-17T10:35:37.114170+0000","last_peered":"2024-04-17T10:35:37.114170+0000","last_clean":"2024-04-17T10:35:37.114170+0000","last_became_active":"2024-04-17T10:35:37.102855+0000","last_became_peered":"2024-04-17T10:35:37.102855+0000","last_unstale":"2024-04-17T10:35:37.114170+0000","last_undegraded":"2024-04-17T10:35:37.114170+0000","last_fullsized":"2024-04-17T10:35:37.114170+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:59.713807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113399+0000","last_change":"2024-04-17T10:35:37.111687+0000","last_active":"2024-04-17T10:35:37.113399+0000","last_peered":"2024-04-17T10:35:37.113399+0000","last_clean":"2024-04-17T10:35:37.113399+0000","last_became_active":"2024-04-17T10:35:37.111407+0000","last_became_peered":"2024-04-17T10:35:37.111407+0000","last_unstale":"2024-04-17T10:35:37.113399+0000","last_undegraded":"2024-04-17T10:35:37.113399+0000","last_fullsized":"2024-04-17T10:35:37.113399+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:01:29.568978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110152+0000","last_change":"2024-04-17T10:35:37.109484+0000","last_active":"2024-04-17T10:35:37.110152+0000","last_peered":"2024-04-17T10:35:37.110152+0000","last_clean":"2024-04-17T10:35:37.110152+0000","last_became_active":"2024-04-17T10:35:37.108923+0000","last_became_peered":"2024-04-17T10:35:37.108923+0000","last_unstale":"2024-04-17T10:35:37.110152+0000","last_undegraded":"2024-04-17T10:35:37.110152+0000","last_fullsized":"2024-04-17T10:35:37.110152+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:40.033640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.3c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102374+0000","last_change":"2024-04-17T10:35:37.097969+0000","last_active":"2024-04-17T10:35:37.102374+0000","last_peered":"2024-04-17T10:35:37.102374+0000","last_clean":"2024-04-17T10:35:37.102374+0000","last_became_active":"2024-04-17T10:35:37.097676+0000","last_became_peered":"2024-04-17T10:35:37.097676+0000","last_unstale":"2024-04-17T10:35:37.102374+0000","last_undegraded":"2024-04-17T10:35:37.102374+0000","last_fullsized":"2024-04-17T10:35:37.102374+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:38.015381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.3b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114132+0000","last_change":"2024-04-17T10:35:37.110215+0000","last_active":"2024-04-17T10:35:37.114132+0000","last_peered":"2024-04-17T10:35:37.114132+0000","last_clean":"2024-04-17T10:35:37.114132+0000","last_became_active":"2024-04-17T10:35:37.107323+0000","last_became_peered":"2024-04-17T10:35:37.107323+0000","last_unstale":"2024-04-17T10:35:37.114132+0000","last_undegraded":"2024-04-17T10:35:37.114132+0000","last_fullsized":"2024-04-17T10:35:37.114132+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:48.289465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114010+0000","last_change":"2024-04-17T10:35:37.107605+0000","last_active":"2024-04-17T10:35:37.114010+0000","last_peered":"2024-04-17T10:35:37.114010+0000","last_clean":"2024-04-17T10:35:37.114010+0000","last_became_active":"2024-04-17T10:35:37.107273+0000","last_became_peered":"2024-04-17T10:35:37.107273+0000","last_unstale":"2024-04-17T10:35:37.114010+0000","last_undegraded":"2024-04-17T10:35:37.114010+0000","last_fullsized":"2024-04-17T10:35:37.114010+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:13.596314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.39","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110565+0000","last_change":"2024-04-17T10:35:37.101896+0000","last_active":"2024-04-17T10:35:37.110565+0000","last_peered":"2024-04-17T10:35:37.110565+0000","last_clean":"2024-04-17T10:35:37.110565+0000","last_became_active":"2024-04-17T10:35:37.101616+0000","last_became_peered":"2024-04-17T10:35:37.101616+0000","last_unstale":"2024-04-17T10:35:37.110565+0000","last_undegraded":"2024-04-17T10:35:37.110565+0000","last_fullsized":"2024-04-17T10:35:37.110565+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:02:41.067908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.38","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114022+0000","last_change":"2024-04-17T10:35:37.110162+0000","last_active":"2024-04-17T10:35:37.114022+0000","last_peered":"2024-04-17T10:35:37.114022+0000","last_clean":"2024-04-17T10:35:37.114022+0000","last_became_active":"2024-04-17T10:35:37.109915+0000","last_became_peered":"2024-04-17T10:35:37.109915+0000","last_unstale":"2024-04-17T10:35:37.114022+0000","last_undegraded":"2024-04-17T10:35:37.114022+0000","last_fullsized":"2024-04-17T10:35:37.114022+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:39:05.072399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.37","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110139+0000","last_change":"2024-04-17T10:35:37.103180+0000","last_active":"2024-04-17T10:35:37.110139+0000","last_peered":"2024-04-17T10:35:37.110139+0000","last_clean":"2024-04-17T10:35:37.110139+0000","last_became_active":"2024-04-17T10:35:37.102911+0000","last_became_peered":"2024-04-17T10:35:37.102911+0000","last_unstale":"2024-04-17T10:35:37.110139+0000","last_undegraded":"2024-04-17T10:35:37.110139+0000","last_fullsized":"2024-04-17T10:35:37.110139+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:05:21.313210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.36","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.097296+0000","last_change":"2024-04-17T10:35:37.091924+0000","last_active":"2024-04-17T10:35:37.097296+0000","last_peered":"2024-04-17T10:35:37.097296+0000","last_clean":"2024-04-17T10:35:37.097296+0000","last_became_active":"2024-04-17T10:35:37.091312+0000","last_became_peered":"2024-04-17T10:35:37.091312+0000","last_unstale":"2024-04-17T10:35:37.097296+0000","last_undegraded":"2024-04-17T10:35:37.097296+0000","last_fullsized":"2024-04-17T10:35:37.097296+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:04:37.360473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.35","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113752+0000","last_change":"2024-04-17T10:35:37.108750+0000","last_active":"2024-04-17T10:35:37.113752+0000","last_peered":"2024-04-17T10:35:37.113752+0000","last_clean":"2024-04-17T10:35:37.113752+0000","last_became_active":"2024-04-17T10:35:37.108420+0000","last_became_peered":"2024-04-17T10:35:37.108420+0000","last_unstale":"2024-04-17T10:35:37.113752+0000","last_undegraded":"2024-04-17T10:35:37.113752+0000","last_fullsized":"2024-04-17T10:35:37.113752+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:42:57.763543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.34","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102344+0000","last_change":"2024-04-17T10:35:37.097401+0000","last_active":"2024-04-17T10:35:37.102344+0000","last_peered":"2024-04-17T10:35:37.102344+0000","last_clean":"2024-04-17T10:35:37.102344+0000","last_became_active":"2024-04-17T10:35:37.094229+0000","last_became_peered":"2024-04-17T10:35:37.094229+0000","last_unstale":"2024-04-17T10:35:37.102344+0000","last_undegraded":"2024-04-17T10:35:37.102344+0000","last_fullsized":"2024-04-17T10:35:37.102344+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:20:37.121559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.33","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.092188+0000","last_change":"2024-04-17T10:35:37.089668+0000","last_active":"2024-04-17T10:35:37.092188+0000","last_peered":"2024-04-17T10:35:37.092188+0000","last_clean":"2024-04-17T10:35:37.092188+0000","last_became_active":"2024-04-17T10:35:37.089371+0000","last_became_peered":"2024-04-17T10:35:37.089371+0000","last_unstale":"2024-04-17T10:35:37.092188+0000","last_undegraded":"2024-04-17T10:35:37.092188+0000","last_fullsized":"2024-04-17T10:35:37.092188+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:32:24.858936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.32","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.098266+0000","last_change":"2024-04-17T10:35:37.093465+0000","last_active":"2024-04-17T10:35:37.098266+0000","last_peered":"2024-04-17T10:35:37.098266+0000","last_clean":"2024-04-17T10:35:37.098266+0000","last_became_active":"2024-04-17T10:35:37.093183+0000","last_became_peered":"2024-04-17T10:35:37.093183+0000","last_unstale":"2024-04-17T10:35:37.098266+0000","last_undegraded":"2024-04-17T10:35:37.098266+0000","last_fullsized":"2024-04-17T10:35:37.098266+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:04.404888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.31","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109460+0000","last_change":"2024-04-17T10:35:37.106785+0000","last_active":"2024-04-17T10:35:37.109460+0000","last_peered":"2024-04-17T10:35:37.109460+0000","last_clean":"2024-04-17T10:35:37.109460+0000","last_became_active":"2024-04-17T10:35:37.106338+0000","last_became_peered":"2024-04-17T10:35:37.106338+0000","last_unstale":"2024-04-17T10:35:37.109460+0000","last_undegraded":"2024-04-17T10:35:37.109460+0000","last_fullsized":"2024-04-17T10:35:37.109460+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:26:32.866813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.30","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109351+0000","last_change":"2024-04-17T10:35:37.103250+0000","last_active":"2024-04-17T10:35:37.109351+0000","last_peered":"2024-04-17T10:35:37.109351+0000","last_clean":"2024-04-17T10:35:37.109351+0000","last_became_active":"2024-04-17T10:35:37.102934+0000","last_became_peered":"2024-04-17T10:35:37.102934+0000","last_unstale":"2024-04-17T10:35:37.109351+0000","last_undegraded":"2024-04-17T10:35:37.109351+0000","last_fullsized":"2024-04-17T10:35:37.109351+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:48:00.673501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.101283+0000","last_change":"2024-04-17T10:35:37.094519+0000","last_active":"2024-04-17T10:35:37.101283+0000","last_peered":"2024-04-17T10:35:37.101283+0000","last_clean":"2024-04-17T10:35:37.101283+0000","last_became_active":"2024-04-17T10:35:37.094229+0000","last_became_peered":"2024-04-17T10:35:37.094229+0000","last_unstale":"2024-04-17T10:35:37.101283+0000","last_undegraded":"2024-04-17T10:35:37.101283+0000","last_fullsized":"2024-04-17T10:35:37.101283+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:06.809771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.2e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119711+0000","last_change":"2024-04-17T10:35:37.118400+0000","last_active":"2024-04-17T10:35:37.119711+0000","last_peered":"2024-04-17T10:35:37.119711+0000","last_clean":"2024-04-17T10:35:37.119711+0000","last_became_active":"2024-04-17T10:35:37.118144+0000","last_became_peered":"2024-04-17T10:35:37.118144+0000","last_unstale":"2024-04-17T10:35:37.119711+0000","last_undegraded":"2024-04-17T10:35:37.119711+0000","last_fullsized":"2024-04-17T10:35:37.119711+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:45:22.500261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102576+0000","last_change":"2024-04-17T10:35:37.098036+0000","last_active":"2024-04-17T10:35:37.102576+0000","last_peered":"2024-04-17T10:35:37.102576+0000","last_clean":"2024-04-17T10:35:37.102576+0000","last_became_active":"2024-04-17T10:35:37.097461+0000","last_became_peered":"2024-04-17T10:35:37.097461+0000","last_unstale":"2024-04-17T10:35:37.102576+0000","last_undegraded":"2024-04-17T10:35:37.102576+0000","last_fullsized":"2024-04-17T10:35:37.102576+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:47:13.596314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.2c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.100187+0000","last_change":"2024-04-17T10:35:37.097351+0000","last_active":"2024-04-17T10:35:37.100187+0000","last_peered":"2024-04-17T10:35:37.100187+0000","last_clean":"2024-04-17T10:35:37.100187+0000","last_became_active":"2024-04-17T10:35:37.096992+0000","last_became_peered":"2024-04-17T10:35:37.096992+0000","last_unstale":"2024-04-17T10:35:37.100187+0000","last_undegraded":"2024-04-17T10:35:37.100187+0000","last_fullsized":"2024-04-17T10:35:37.100187+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:36:42.313204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.2b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110359+0000","last_change":"2024-04-17T10:35:37.103635+0000","last_active":"2024-04-17T10:35:37.110359+0000","last_peered":"2024-04-17T10:35:37.110359+0000","last_clean":"2024-04-17T10:35:37.110359+0000","last_became_active":"2024-04-17T10:35:37.103375+0000","last_became_peered":"2024-04-17T10:35:37.103375+0000","last_unstale":"2024-04-17T10:35:37.110359+0000","last_undegraded":"2024-04-17T10:35:37.110359+0000","last_fullsized":"2024-04-17T10:35:37.110359+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:07:49.964511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.2a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114260+0000","last_change":"2024-04-17T10:35:37.105178+0000","last_active":"2024-04-17T10:35:37.114260+0000","last_peered":"2024-04-17T10:35:37.114260+0000","last_clean":"2024-04-17T10:35:37.114260+0000","last_became_active":"2024-04-17T10:35:37.104835+0000","last_became_peered":"2024-04-17T10:35:37.104835+0000","last_unstale":"2024-04-17T10:35:37.114260+0000","last_undegraded":"2024-04-17T10:35:37.114260+0000","last_fullsized":"2024-04-17T10:35:37.114260+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:46:22.972535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.29","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.111858+0000","last_change":"2024-04-17T10:35:37.107537+0000","last_active":"2024-04-17T10:35:37.111858+0000","last_peered":"2024-04-17T10:35:37.111858+0000","last_clean":"2024-04-17T10:35:37.111858+0000","last_became_active":"2024-04-17T10:35:37.107259+0000","last_became_peered":"2024-04-17T10:35:37.107259+0000","last_unstale":"2024-04-17T10:35:37.111858+0000","last_undegraded":"2024-04-17T10:35:37.111858+0000","last_fullsized":"2024-04-17T10:35:37.111858+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:50.603148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.28","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113647+0000","last_change":"2024-04-17T10:35:37.108739+0000","last_active":"2024-04-17T10:35:37.113647+0000","last_peered":"2024-04-17T10:35:37.113647+0000","last_clean":"2024-04-17T10:35:37.113647+0000","last_became_active":"2024-04-17T10:35:37.108406+0000","last_became_peered":"2024-04-17T10:35:37.108406+0000","last_unstale":"2024-04-17T10:35:37.113647+0000","last_undegraded":"2024-04-17T10:35:37.113647+0000","last_fullsized":"2024-04-17T10:35:37.113647+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:35:01.032056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.27","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116164+0000","last_change":"2024-04-17T10:35:37.110990+0000","last_active":"2024-04-17T10:35:37.116164+0000","last_peered":"2024-04-17T10:35:37.116164+0000","last_clean":"2024-04-17T10:35:37.116164+0000","last_became_active":"2024-04-17T10:35:37.105458+0000","last_became_peered":"2024-04-17T10:35:37.105458+0000","last_unstale":"2024-04-17T10:35:37.116164+0000","last_undegraded":"2024-04-17T10:35:37.116164+0000","last_fullsized":"2024-04-17T10:35:37.116164+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:36.302526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.26","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116110+0000","last_change":"2024-04-17T10:35:37.110999+0000","last_active":"2024-04-17T10:35:37.116110+0000","last_peered":"2024-04-17T10:35:37.116110+0000","last_clean":"2024-04-17T10:35:37.116110+0000","last_became_active":"2024-04-17T10:35:37.104269+0000","last_became_peered":"2024-04-17T10:35:37.104269+0000","last_unstale":"2024-04-17T10:35:37.116110+0000","last_undegraded":"2024-04-17T10:35:37.116110+0000","last_fullsized":"2024-04-17T10:35:37.116110+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:51.313852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.25","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.111202+0000","last_change":"2024-04-17T10:35:37.104308+0000","last_active":"2024-04-17T10:35:37.111202+0000","last_peered":"2024-04-17T10:35:37.111202+0000","last_clean":"2024-04-17T10:35:37.111202+0000","last_became_active":"2024-04-17T10:35:37.104061+0000","last_became_peered":"2024-04-17T10:35:37.104061+0000","last_unstale":"2024-04-17T10:35:37.111202+0000","last_undegraded":"2024-04-17T10:35:37.111202+0000","last_fullsized":"2024-04-17T10:35:37.111202+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:50.321997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.24","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.115082+0000","last_change":"2024-04-17T10:35:37.110709+0000","last_active":"2024-04-17T10:35:37.115082+0000","last_peered":"2024-04-17T10:35:37.115082+0000","last_clean":"2024-04-17T10:35:37.115082+0000","last_became_active":"2024-04-17T10:35:37.110439+0000","last_became_peered":"2024-04-17T10:35:37.110439+0000","last_unstale":"2024-04-17T10:35:37.115082+0000","last_undegraded":"2024-04-17T10:35:37.115082+0000","last_fullsized":"2024-04-17T10:35:37.115082+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:12:19.576363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.23","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.108002+0000","last_change":"2024-04-17T10:35:37.106870+0000","last_active":"2024-04-17T10:35:37.108002+0000","last_peered":"2024-04-17T10:35:37.108002+0000","last_clean":"2024-04-17T10:35:37.108002+0000","last_became_active":"2024-04-17T10:35:37.106421+0000","last_became_peered":"2024-04-17T10:35:37.106421+0000","last_unstale":"2024-04-17T10:35:37.108002+0000","last_undegraded":"2024-04-17T10:35:37.108002+0000","last_fullsized":"2024-04-17T10:35:37.108002+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:02:38.015381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.22","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.112498+0000","last_change":"2024-04-17T10:35:37.103984+0000","last_active":"2024-04-17T10:35:37.112498+0000","last_peered":"2024-04-17T10:35:37.112498+0000","last_clean":"2024-04-17T10:35:37.112498+0000","last_became_active":"2024-04-17T10:35:37.103707+0000","last_became_peered":"2024-04-17T10:35:37.103707+0000","last_unstale":"2024-04-17T10:35:37.112498+0000","last_undegraded":"2024-04-17T10:35:37.112498+0000","last_fullsized":"2024-04-17T10:35:37.112498+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:54:39.637819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.21","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105673+0000","last_change":"2024-04-17T10:35:37.097693+0000","last_active":"2024-04-17T10:35:37.105673+0000","last_peered":"2024-04-17T10:35:37.105673+0000","last_clean":"2024-04-17T10:35:37.105673+0000","last_became_active":"2024-04-17T10:35:37.097411+0000","last_became_peered":"2024-04-17T10:35:37.097411+0000","last_unstale":"2024-04-17T10:35:37.105673+0000","last_undegraded":"2024-04-17T10:35:37.105673+0000","last_fullsized":"2024-04-17T10:35:37.105673+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:39.398472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.20","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113997+0000","last_change":"2024-04-17T10:35:37.109167+0000","last_active":"2024-04-17T10:35:37.113997+0000","last_peered":"2024-04-17T10:35:37.113997+0000","last_clean":"2024-04-17T10:35:37.113997+0000","last_became_active":"2024-04-17T10:35:37.108883+0000","last_became_peered":"2024-04-17T10:35:37.108883+0000","last_unstale":"2024-04-17T10:35:37.113997+0000","last_undegraded":"2024-04-17T10:35:37.113997+0000","last_fullsized":"2024-04-17T10:35:37.113997+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:55:46.629651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.108694+0000","last_change":"2024-04-17T10:35:37.104788+0000","last_active":"2024-04-17T10:35:37.108694+0000","last_peered":"2024-04-17T10:35:37.108694+0000","last_clean":"2024-04-17T10:35:37.108694+0000","last_became_active":"2024-04-17T10:35:37.104197+0000","last_became_peered":"2024-04-17T10:35:37.104197+0000","last_unstale":"2024-04-17T10:35:37.108694+0000","last_undegraded":"2024-04-17T10:35:37.108694+0000","last_fullsized":"2024-04-17T10:35:37.108694+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:59:13.776507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1f","version":"140'4","reported_seq":16,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.122006+0000","last_change":"2024-04-17T10:35:36.089282+0000","last_active":"2024-04-17T10:35:37.122006+0000","last_peered":"2024-04-17T10:35:37.122006+0000","last_clean":"2024-04-17T10:35:37.122006+0000","last_became_active":"2024-04-17T10:35:36.089023+0000","last_became_peered":"2024-04-17T10:35:36.089023+0000","last_unstale":"2024-04-17T10:35:37.122006+0000","last_undegraded":"2024-04-17T10:35:37.122006+0000","last_fullsized":"2024-04-17T10:35:37.122006+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:39:50.307672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109650+0000","last_change":"2024-04-17T10:35:37.107293+0000","last_active":"2024-04-17T10:35:37.109650+0000","last_peered":"2024-04-17T10:35:37.109650+0000","last_clean":"2024-04-17T10:35:37.109650+0000","last_became_active":"2024-04-17T10:35:37.106407+0000","last_became_peered":"2024-04-17T10:35:37.106407+0000","last_unstale":"2024-04-17T10:35:37.109650+0000","last_undegraded":"2024-04-17T10:35:37.109650+0000","last_fullsized":"2024-04-17T10:35:37.109650+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:56:02.027623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1e","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119190+0000","last_change":"2024-04-17T10:35:36.091651+0000","last_active":"2024-04-17T10:35:37.119190+0000","last_peered":"2024-04-17T10:35:37.119190+0000","last_clean":"2024-04-17T10:35:37.119190+0000","last_became_active":"2024-04-17T10:35:36.090988+0000","last_became_peered":"2024-04-17T10:35:36.090988+0000","last_unstale":"2024-04-17T10:35:37.119190+0000","last_undegraded":"2024-04-17T10:35:37.119190+0000","last_fullsized":"2024-04-17T10:35:37.119190+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:47:38.392292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113893+0000","last_change":"2024-04-17T10:35:37.109028+0000","last_active":"2024-04-17T10:35:37.113893+0000","last_peered":"2024-04-17T10:35:37.113893+0000","last_clean":"2024-04-17T10:35:37.113893+0000","last_became_active":"2024-04-17T10:35:37.108783+0000","last_became_peered":"2024-04-17T10:35:37.108783+0000","last_unstale":"2024-04-17T10:35:37.113893+0000","last_undegraded":"2024-04-17T10:35:37.113893+0000","last_fullsized":"2024-04-17T10:35:37.113893+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:16:43.471471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119735+0000","last_change":"2024-04-17T10:35:36.094939+0000","last_active":"2024-04-17T10:35:37.119735+0000","last_peered":"2024-04-17T10:35:37.119735+0000","last_clean":"2024-04-17T10:35:37.119735+0000","last_became_active":"2024-04-17T10:35:36.094630+0000","last_became_peered":"2024-04-17T10:35:36.094630+0000","last_unstale":"2024-04-17T10:35:37.119735+0000","last_undegraded":"2024-04-17T10:35:37.119735+0000","last_fullsized":"2024-04-17T10:35:37.119735+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:05:20.307030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110093+0000","last_change":"2024-04-17T10:35:37.107091+0000","last_active":"2024-04-17T10:35:37.110093+0000","last_peered":"2024-04-17T10:35:37.110093+0000","last_clean":"2024-04-17T10:35:37.110093+0000","last_became_active":"2024-04-17T10:35:37.106811+0000","last_became_peered":"2024-04-17T10:35:37.106811+0000","last_unstale":"2024-04-17T10:35:37.110093+0000","last_undegraded":"2024-04-17T10:35:37.110093+0000","last_fullsized":"2024-04-17T10:35:37.110093+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:53:32.647661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1c","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.122001+0000","last_change":"2024-04-17T10:35:36.087964+0000","last_active":"2024-04-17T10:35:37.122001+0000","last_peered":"2024-04-17T10:35:37.122001+0000","last_clean":"2024-04-17T10:35:37.122001+0000","last_became_active":"2024-04-17T10:35:36.087225+0000","last_became_peered":"2024-04-17T10:35:36.087225+0000","last_unstale":"2024-04-17T10:35:37.122001+0000","last_undegraded":"2024-04-17T10:35:37.122001+0000","last_fullsized":"2024-04-17T10:35:37.122001+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:39:04.066220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105318+0000","last_change":"2024-04-17T10:35:37.103106+0000","last_active":"2024-04-17T10:35:37.105318+0000","last_peered":"2024-04-17T10:35:37.105318+0000","last_clean":"2024-04-17T10:35:37.105318+0000","last_became_active":"2024-04-17T10:35:37.102826+0000","last_became_peered":"2024-04-17T10:35:37.102826+0000","last_unstale":"2024-04-17T10:35:37.105318+0000","last_undegraded":"2024-04-17T10:35:37.105318+0000","last_fullsized":"2024-04-17T10:35:37.105318+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:10:33.074742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.1b","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113898+0000","last_change":"2024-04-17T10:35:36.088024+0000","last_active":"2024-04-17T10:35:37.113898+0000","last_peered":"2024-04-17T10:35:37.113898+0000","last_clean":"2024-04-17T10:35:37.113898+0000","last_became_active":"2024-04-17T10:35:36.087589+0000","last_became_peered":"2024-04-17T10:35:36.087589+0000","last_unstale":"2024-04-17T10:35:37.113898+0000","last_undegraded":"2024-04-17T10:35:37.113898+0000","last_fullsized":"2024-04-17T10:35:37.113898+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:11.646151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102492+0000","last_change":"2024-04-17T10:35:37.099028+0000","last_active":"2024-04-17T10:35:37.102492+0000","last_peered":"2024-04-17T10:35:37.102492+0000","last_clean":"2024-04-17T10:35:37.102492+0000","last_became_active":"2024-04-17T10:35:37.098779+0000","last_became_peered":"2024-04-17T10:35:37.098779+0000","last_unstale":"2024-04-17T10:35:37.102492+0000","last_undegraded":"2024-04-17T10:35:37.102492+0000","last_fullsized":"2024-04-17T10:35:37.102492+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:50:50.603148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1a","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121448+0000","last_change":"2024-04-17T10:35:36.093492+0000","last_active":"2024-04-17T10:35:37.121448+0000","last_peered":"2024-04-17T10:35:37.121448+0000","last_clean":"2024-04-17T10:35:37.121448+0000","last_became_active":"2024-04-17T10:35:36.092707+0000","last_became_peered":"2024-04-17T10:35:36.092707+0000","last_unstale":"2024-04-17T10:35:37.121448+0000","last_undegraded":"2024-04-17T10:35:37.121448+0000","last_fullsized":"2024-04-17T10:35:37.121448+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:40:53.106577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102206+0000","last_change":"2024-04-17T10:35:37.100996+0000","last_active":"2024-04-17T10:35:37.102206+0000","last_peered":"2024-04-17T10:35:37.102206+0000","last_clean":"2024-04-17T10:35:37.102206+0000","last_became_active":"2024-04-17T10:35:37.100706+0000","last_became_peered":"2024-04-17T10:35:37.100706+0000","last_unstale":"2024-04-17T10:35:37.102206+0000","last_undegraded":"2024-04-17T10:35:37.102206+0000","last_fullsized":"2024-04-17T10:35:37.102206+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:24:34.031060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.098571+0000","last_change":"2024-04-17T10:35:36.079916+0000","last_active":"2024-04-17T10:35:37.098571+0000","last_peered":"2024-04-17T10:35:37.098571+0000","last_clean":"2024-04-17T10:35:37.098571+0000","last_became_active":"2024-04-17T10:35:36.079636+0000","last_became_peered":"2024-04-17T10:35:36.079636+0000","last_unstale":"2024-04-17T10:35:37.098571+0000","last_undegraded":"2024-04-17T10:35:37.098571+0000","last_fullsized":"2024-04-17T10:35:37.098571+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:02:39.027460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110600+0000","last_change":"2024-04-17T10:35:37.099921+0000","last_active":"2024-04-17T10:35:37.110600+0000","last_peered":"2024-04-17T10:35:37.110600+0000","last_clean":"2024-04-17T10:35:37.110600+0000","last_became_active":"2024-04-17T10:35:37.099602+0000","last_became_peered":"2024-04-17T10:35:37.099602+0000","last_unstale":"2024-04-17T10:35:37.110600+0000","last_undegraded":"2024-04-17T10:35:37.110600+0000","last_fullsized":"2024-04-17T10:35:37.110600+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:23:06.637859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085546+0000","last_change":"2024-04-17T10:35:36.086559+0000","last_active":"2024-04-17T10:35:37.085546+0000","last_peered":"2024-04-17T10:35:37.085546+0000","last_clean":"2024-04-17T10:35:37.085546+0000","last_became_active":"2024-04-17T10:35:36.086108+0000","last_became_peered":"2024-04-17T10:35:36.086108+0000","last_unstale":"2024-04-17T10:35:37.085546+0000","last_undegraded":"2024-04-17T10:35:37.085546+0000","last_fullsized":"2024-04-17T10:35:37.085546+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:02:58.707627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.106014+0000","last_change":"2024-04-17T10:35:37.104776+0000","last_active":"2024-04-17T10:35:37.106014+0000","last_peered":"2024-04-17T10:35:37.106014+0000","last_clean":"2024-04-17T10:35:37.106014+0000","last_became_active":"2024-04-17T10:35:37.104514+0000","last_became_peered":"2024-04-17T10:35:37.104514+0000","last_unstale":"2024-04-17T10:35:37.106014+0000","last_undegraded":"2024-04-17T10:35:37.106014+0000","last_fullsized":"2024-04-17T10:35:37.106014+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:20:37.121559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.092498+0000","last_change":"2024-04-17T10:35:36.093199+0000","last_active":"2024-04-17T10:35:37.092498+0000","last_peered":"2024-04-17T10:35:37.092498+0000","last_clean":"2024-04-17T10:35:37.092498+0000","last_became_active":"2024-04-17T10:35:36.092893+0000","last_became_peered":"2024-04-17T10:35:36.092893+0000","last_unstale":"2024-04-17T10:35:37.092498+0000","last_undegraded":"2024-04-17T10:35:37.092498+0000","last_fullsized":"2024-04-17T10:35:37.092498+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:07:48.958331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113729+0000","last_change":"2024-04-17T10:35:37.108279+0000","last_active":"2024-04-17T10:35:37.113729+0000","last_peered":"2024-04-17T10:35:37.113729+0000","last_clean":"2024-04-17T10:35:37.113729+0000","last_became_active":"2024-04-17T10:35:37.107712+0000","last_became_peered":"2024-04-17T10:35:37.107712+0000","last_unstale":"2024-04-17T10:35:37.113729+0000","last_undegraded":"2024-04-17T10:35:37.113729+0000","last_fullsized":"2024-04-17T10:35:37.113729+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:11:53.706069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099563+0000","last_change":"2024-04-17T10:35:36.083117+0000","last_active":"2024-04-17T10:35:37.099563+0000","last_peered":"2024-04-17T10:35:37.099563+0000","last_clean":"2024-04-17T10:35:37.099563+0000","last_became_active":"2024-04-17T10:35:36.082819+0000","last_became_peered":"2024-04-17T10:35:36.082819+0000","last_unstale":"2024-04-17T10:35:37.099563+0000","last_undegraded":"2024-04-17T10:35:37.099563+0000","last_fullsized":"2024-04-17T10:35:37.099563+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:18:23.125545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.119774+0000","last_change":"2024-04-17T10:35:37.118909+0000","last_active":"2024-04-17T10:35:37.119774+0000","last_peered":"2024-04-17T10:35:37.119774+0000","last_clean":"2024-04-17T10:35:37.119774+0000","last_became_active":"2024-04-17T10:35:37.118679+0000","last_became_peered":"2024-04-17T10:35:37.118679+0000","last_unstale":"2024-04-17T10:35:37.119774+0000","last_undegraded":"2024-04-17T10:35:37.119774+0000","last_fullsized":"2024-04-17T10:35:37.119774+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:02:42.970481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.15","version":"140'3","reported_seq":15,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121642+0000","last_change":"2024-04-17T10:35:36.094567+0000","last_active":"2024-04-17T10:35:37.121642+0000","last_peered":"2024-04-17T10:35:37.121642+0000","last_clean":"2024-04-17T10:35:37.121642+0000","last_became_active":"2024-04-17T10:35:36.094065+0000","last_became_peered":"2024-04-17T10:35:36.094065+0000","last_unstale":"2024-04-17T10:35:37.121642+0000","last_undegraded":"2024-04-17T10:35:37.121642+0000","last_fullsized":"2024-04-17T10:35:37.121642+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:28:06.081208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105754+0000","last_change":"2024-04-17T10:35:37.104084+0000","last_active":"2024-04-17T10:35:37.105754+0000","last_peered":"2024-04-17T10:35:37.105754+0000","last_clean":"2024-04-17T10:35:37.105754+0000","last_became_active":"2024-04-17T10:35:37.103774+0000","last_became_peered":"2024-04-17T10:35:37.103774+0000","last_unstale":"2024-04-17T10:35:37.105754+0000","last_undegraded":"2024-04-17T10:35:37.105754+0000","last_fullsized":"2024-04-17T10:35:37.105754+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:27:06.809771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.13","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121906+0000","last_change":"2024-04-17T10:35:36.088996+0000","last_active":"2024-04-17T10:35:37.121906+0000","last_peered":"2024-04-17T10:35:37.121906+0000","last_clean":"2024-04-17T10:35:37.121906+0000","last_became_active":"2024-04-17T10:35:36.088680+0000","last_became_peered":"2024-04-17T10:35:36.088680+0000","last_unstale":"2024-04-17T10:35:37.121906+0000","last_undegraded":"2024-04-17T10:35:37.121906+0000","last_fullsized":"2024-04-17T10:35:37.121906+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T10:46:21.966356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.108898+0000","last_change":"2024-04-17T10:35:37.106008+0000","last_active":"2024-04-17T10:35:37.108898+0000","last_peered":"2024-04-17T10:35:37.108898+0000","last_clean":"2024-04-17T10:35:37.108898+0000","last_became_active":"2024-04-17T10:35:37.105679+0000","last_became_peered":"2024-04-17T10:35:37.105679+0000","last_unstale":"2024-04-17T10:35:37.108898+0000","last_undegraded":"2024-04-17T10:35:37.108898+0000","last_fullsized":"2024-04-17T10:35:37.108898+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:54:04.123300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085645+0000","last_change":"2024-04-17T10:35:36.084061+0000","last_active":"2024-04-17T10:35:37.085645+0000","last_peered":"2024-04-17T10:35:37.085645+0000","last_clean":"2024-04-17T10:35:37.085645+0000","last_became_active":"2024-04-17T10:35:36.083775+0000","last_became_peered":"2024-04-17T10:35:36.083775+0000","last_unstale":"2024-04-17T10:35:37.085645+0000","last_undegraded":"2024-04-17T10:35:37.085645+0000","last_fullsized":"2024-04-17T10:35:37.085645+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:16.962060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114161+0000","last_change":"2024-04-17T10:35:37.110618+0000","last_active":"2024-04-17T10:35:37.114161+0000","last_peered":"2024-04-17T10:35:37.114161+0000","last_clean":"2024-04-17T10:35:37.114161+0000","last_became_active":"2024-04-17T10:35:37.110358+0000","last_became_peered":"2024-04-17T10:35:37.110358+0000","last_unstale":"2024-04-17T10:35:37.114161+0000","last_undegraded":"2024-04-17T10:35:37.114161+0000","last_fullsized":"2024-04-17T10:35:37.114161+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:51.309229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085205+0000","last_change":"2024-04-17T10:35:36.091614+0000","last_active":"2024-04-17T10:35:37.085205+0000","last_peered":"2024-04-17T10:35:37.085205+0000","last_clean":"2024-04-17T10:35:37.085205+0000","last_became_active":"2024-04-17T10:35:36.088453+0000","last_became_peered":"2024-04-17T10:35:36.088453+0000","last_unstale":"2024-04-17T10:35:37.085205+0000","last_undegraded":"2024-04-17T10:35:37.085205+0000","last_fullsized":"2024-04-17T10:35:37.085205+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:19:51.993595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.110117+0000","last_change":"2024-04-17T10:35:37.109456+0000","last_active":"2024-04-17T10:35:37.110117+0000","last_peered":"2024-04-17T10:35:37.110117+0000","last_clean":"2024-04-17T10:35:37.110117+0000","last_became_active":"2024-04-17T10:35:37.109176+0000","last_became_peered":"2024-04-17T10:35:37.109176+0000","last_unstale":"2024-04-17T10:35:37.110117+0000","last_undegraded":"2024-04-17T10:35:37.110117+0000","last_fullsized":"2024-04-17T10:35:37.110117+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:21.025884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.10","version":"140'8","reported_seq":20,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121894+0000","last_change":"2024-04-17T10:35:36.088905+0000","last_active":"2024-04-17T10:35:37.121894+0000","last_peered":"2024-04-17T10:35:37.121894+0000","last_clean":"2024-04-17T10:35:37.121894+0000","last_became_active":"2024-04-17T10:35:36.088593+0000","last_became_peered":"2024-04-17T10:35:36.088593+0000","last_unstale":"2024-04-17T10:35:37.121894+0000","last_undegraded":"2024-04-17T10:35:37.121894+0000","last_fullsized":"2024-04-17T10:35:37.121894+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:44:20.019704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099688+0000","last_change":"2024-04-17T10:35:37.093004+0000","last_active":"2024-04-17T10:35:37.099688+0000","last_peered":"2024-04-17T10:35:37.099688+0000","last_clean":"2024-04-17T10:35:37.099688+0000","last_became_active":"2024-04-17T10:35:37.092745+0000","last_became_peered":"2024-04-17T10:35:37.092745+0000","last_unstale":"2024-04-17T10:35:37.099688+0000","last_undegraded":"2024-04-17T10:35:37.099688+0000","last_fullsized":"2024-04-17T10:35:37.099688+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:50.321997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.f","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121933+0000","last_change":"2024-04-17T10:35:36.090837+0000","last_active":"2024-04-17T10:35:37.121933+0000","last_peered":"2024-04-17T10:35:37.121933+0000","last_clean":"2024-04-17T10:35:37.121933+0000","last_became_active":"2024-04-17T10:35:36.090538+0000","last_became_peered":"2024-04-17T10:35:36.090538+0000","last_unstale":"2024-04-17T10:35:37.121933+0000","last_undegraded":"2024-04-17T10:35:37.121933+0000","last_fullsized":"2024-04-17T10:35:37.121933+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:04:00.312420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102603+0000","last_change":"2024-04-17T10:35:37.098266+0000","last_active":"2024-04-17T10:35:37.102603+0000","last_peered":"2024-04-17T10:35:37.102603+0000","last_clean":"2024-04-17T10:35:37.102603+0000","last_became_active":"2024-04-17T10:35:37.097738+0000","last_became_peered":"2024-04-17T10:35:37.097738+0000","last_unstale":"2024-04-17T10:35:37.102603+0000","last_undegraded":"2024-04-17T10:35:37.102603+0000","last_fullsized":"2024-04-17T10:35:37.102603+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:20:24.797449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.e","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.118580+0000","last_change":"2024-04-17T10:35:36.093074+0000","last_active":"2024-04-17T10:35:37.118580+0000","last_peered":"2024-04-17T10:35:37.118580+0000","last_clean":"2024-04-17T10:35:37.118580+0000","last_became_active":"2024-04-17T10:35:36.092772+0000","last_became_peered":"2024-04-17T10:35:36.092772+0000","last_unstale":"2024-04-17T10:35:37.118580+0000","last_undegraded":"2024-04-17T10:35:37.118580+0000","last_fullsized":"2024-04-17T10:35:37.118580+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:47:59.667321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114419+0000","last_change":"2024-04-17T10:35:37.110167+0000","last_active":"2024-04-17T10:35:37.114419+0000","last_peered":"2024-04-17T10:35:37.114419+0000","last_clean":"2024-04-17T10:35:37.114419+0000","last_became_active":"2024-04-17T10:35:37.104639+0000","last_became_peered":"2024-04-17T10:35:37.104639+0000","last_unstale":"2024-04-17T10:35:37.114419+0000","last_undegraded":"2024-04-17T10:35:37.114419+0000","last_fullsized":"2024-04-17T10:35:37.114419+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:40:54.112757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.d","version":"140'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.121077+0000","last_change":"2024-04-17T10:35:36.093466+0000","last_active":"2024-04-17T10:35:37.121077+0000","last_peered":"2024-04-17T10:35:37.121077+0000","last_clean":"2024-04-17T10:35:37.121077+0000","last_became_active":"2024-04-17T10:35:36.093108+0000","last_became_peered":"2024-04-17T10:35:36.093108+0000","last_unstale":"2024-04-17T10:35:37.121077+0000","last_undegraded":"2024-04-17T10:35:37.121077+0000","last_fullsized":"2024-04-17T10:35:37.121077+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:47.980225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.111155+0000","last_change":"2024-04-17T10:35:37.102014+0000","last_active":"2024-04-17T10:35:37.111155+0000","last_peered":"2024-04-17T10:35:37.111155+0000","last_clean":"2024-04-17T10:35:37.111155+0000","last_became_active":"2024-04-17T10:35:37.101738+0000","last_became_peered":"2024-04-17T10:35:37.101738+0000","last_unstale":"2024-04-17T10:35:37.111155+0000","last_undegraded":"2024-04-17T10:35:37.111155+0000","last_fullsized":"2024-04-17T10:35:37.111155+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:16:30.273467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.085540+0000","last_change":"2024-04-17T10:35:36.089996+0000","last_active":"2024-04-17T10:35:37.085540+0000","last_peered":"2024-04-17T10:35:37.085540+0000","last_clean":"2024-04-17T10:35:37.085540+0000","last_became_active":"2024-04-17T10:35:36.089182+0000","last_became_peered":"2024-04-17T10:35:36.089182+0000","last_unstale":"2024-04-17T10:35:37.085540+0000","last_undegraded":"2024-04-17T10:35:37.085540+0000","last_fullsized":"2024-04-17T10:35:37.085540+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:52:37.196771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.104810+0000","last_change":"2024-04-17T10:35:37.102543+0000","last_active":"2024-04-17T10:35:37.104810+0000","last_peered":"2024-04-17T10:35:37.104810+0000","last_clean":"2024-04-17T10:35:37.104810+0000","last_became_active":"2024-04-17T10:35:37.096287+0000","last_became_peered":"2024-04-17T10:35:37.096287+0000","last_unstale":"2024-04-17T10:35:37.104810+0000","last_undegraded":"2024-04-17T10:35:37.104810+0000","last_fullsized":"2024-04-17T10:35:37.104810+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:57:48.289465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.b","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.122366+0000","last_change":"2024-04-17T10:35:36.087132+0000","last_active":"2024-04-17T10:35:37.122366+0000","last_peered":"2024-04-17T10:35:37.122366+0000","last_clean":"2024-04-17T10:35:37.122366+0000","last_became_active":"2024-04-17T10:35:36.086802+0000","last_became_peered":"2024-04-17T10:35:36.086802+0000","last_unstale":"2024-04-17T10:35:37.122366+0000","last_undegraded":"2024-04-17T10:35:37.122366+0000","last_fullsized":"2024-04-17T10:35:37.122366+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:01:47.106270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109479+0000","last_change":"2024-04-17T10:35:37.107132+0000","last_active":"2024-04-17T10:35:37.109479+0000","last_peered":"2024-04-17T10:35:37.109479+0000","last_clean":"2024-04-17T10:35:37.109479+0000","last_became_active":"2024-04-17T10:35:37.106868+0000","last_became_peered":"2024-04-17T10:35:37.106868+0000","last_unstale":"2024-04-17T10:35:37.109479+0000","last_undegraded":"2024-04-17T10:35:37.109479+0000","last_fullsized":"2024-04-17T10:35:37.109479+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:29:41.010378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.102189+0000","last_change":"2024-04-17T10:35:36.084917+0000","last_active":"2024-04-17T10:35:37.102189+0000","last_peered":"2024-04-17T10:35:37.102189+0000","last_clean":"2024-04-17T10:35:37.102189+0000","last_became_active":"2024-04-17T10:35:36.083136+0000","last_became_peered":"2024-04-17T10:35:36.083136+0000","last_unstale":"2024-04-17T10:35:37.102189+0000","last_undegraded":"2024-04-17T10:35:37.102189+0000","last_fullsized":"2024-04-17T10:35:37.102189+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T13:29:03.398709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109775+0000","last_change":"2024-04-17T10:35:37.105951+0000","last_active":"2024-04-17T10:35:37.109775+0000","last_peered":"2024-04-17T10:35:37.109775+0000","last_clean":"2024-04-17T10:35:37.109775+0000","last_became_active":"2024-04-17T10:35:37.105695+0000","last_became_peered":"2024-04-17T10:35:37.105695+0000","last_unstale":"2024-04-17T10:35:37.109775+0000","last_undegraded":"2024-04-17T10:35:37.109775+0000","last_fullsized":"2024-04-17T10:35:37.109775+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:10:51.309229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.088002+0000","last_change":"2024-04-17T10:35:36.087908+0000","last_active":"2024-04-17T10:35:37.088002+0000","last_peered":"2024-04-17T10:35:37.088002+0000","last_clean":"2024-04-17T10:35:37.088002+0000","last_became_active":"2024-04-17T10:35:36.087618+0000","last_became_peered":"2024-04-17T10:35:36.087618+0000","last_unstale":"2024-04-17T10:35:37.088002+0000","last_undegraded":"2024-04-17T10:35:37.088002+0000","last_fullsized":"2024-04-17T10:35:37.088002+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T21:16:15.343014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.109346+0000","last_change":"2024-04-17T10:35:37.105926+0000","last_active":"2024-04-17T10:35:37.109346+0000","last_peered":"2024-04-17T10:35:37.109346+0000","last_clean":"2024-04-17T10:35:37.109346+0000","last_became_active":"2024-04-17T10:35:37.105646+0000","last_became_peered":"2024-04-17T10:35:37.105646+0000","last_unstale":"2024-04-17T10:35:37.109346+0000","last_undegraded":"2024-04-17T10:35:37.109346+0000","last_fullsized":"2024-04-17T10:35:37.109346+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:08:43.913804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.092837+0000","last_change":"2024-04-17T10:35:36.088396+0000","last_active":"2024-04-17T10:35:37.092837+0000","last_peered":"2024-04-17T10:35:37.092837+0000","last_clean":"2024-04-17T10:35:37.092837+0000","last_became_active":"2024-04-17T10:35:36.087752+0000","last_became_peered":"2024-04-17T10:35:36.087752+0000","last_unstale":"2024-04-17T10:35:37.092837+0000","last_undegraded":"2024-04-17T10:35:37.092837+0000","last_fullsized":"2024-04-17T10:35:37.092837+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:19:16.962060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114153+0000","last_change":"2024-04-17T10:35:37.107607+0000","last_active":"2024-04-17T10:35:37.114153+0000","last_peered":"2024-04-17T10:35:37.114153+0000","last_clean":"2024-04-17T10:35:37.114153+0000","last_became_active":"2024-04-17T10:35:37.107339+0000","last_became_peered":"2024-04-17T10:35:37.107339+0000","last_unstale":"2024-04-17T10:35:37.114153+0000","last_undegraded":"2024-04-17T10:35:37.114153+0000","last_fullsized":"2024-04-17T10:35:37.114153+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:20:24.797449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.7","version":"140'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.114389+0000","last_change":"2024-04-17T10:35:36.090786+0000","last_active":"2024-04-17T10:35:37.114389+0000","last_peered":"2024-04-17T10:35:37.114389+0000","last_clean":"2024-04-17T10:35:37.114389+0000","last_became_active":"2024-04-17T10:35:36.089211+0000","last_became_peered":"2024-04-17T10:35:36.089211+0000","last_unstale":"2024-04-17T10:35:37.114389+0000","last_undegraded":"2024-04-17T10:35:37.114389+0000","last_fullsized":"2024-04-17T10:35:37.114389+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:15:28.133381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.096020+0000","last_change":"2024-04-17T10:35:37.090765+0000","last_active":"2024-04-17T10:35:37.096020+0000","last_peered":"2024-04-17T10:35:37.096020+0000","last_clean":"2024-04-17T10:35:37.096020+0000","last_became_active":"2024-04-17T10:35:37.090501+0000","last_became_peered":"2024-04-17T10:35:37.090501+0000","last_unstale":"2024-04-17T10:35:37.096020+0000","last_undegraded":"2024-04-17T10:35:37.096020+0000","last_fullsized":"2024-04-17T10:35:37.096020+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T17:52:12.652331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099661+0000","last_change":"2024-04-17T10:35:36.089744+0000","last_active":"2024-04-17T10:35:37.099661+0000","last_peered":"2024-04-17T10:35:37.099661+0000","last_clean":"2024-04-17T10:35:37.099661+0000","last_became_active":"2024-04-17T10:35:36.089476+0000","last_became_peered":"2024-04-17T10:35:36.089476+0000","last_unstale":"2024-04-17T10:35:37.099661+0000","last_undegraded":"2024-04-17T10:35:37.099661+0000","last_fullsized":"2024-04-17T10:35:37.099661+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T16:59:27.127929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116133+0000","last_change":"2024-04-17T10:35:37.111848+0000","last_active":"2024-04-17T10:35:37.116133+0000","last_peered":"2024-04-17T10:35:37.116133+0000","last_clean":"2024-04-17T10:35:37.116133+0000","last_became_active":"2024-04-17T10:35:37.111586+0000","last_became_peered":"2024-04-17T10:35:37.111586+0000","last_unstale":"2024-04-17T10:35:37.116133+0000","last_undegraded":"2024-04-17T10:35:37.116133+0000","last_fullsized":"2024-04-17T10:35:37.116133+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:32:24.858936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.095476+0000","last_change":"2024-04-17T10:35:36.083940+0000","last_active":"2024-04-17T10:35:37.095476+0000","last_peered":"2024-04-17T10:35:37.095476+0000","last_clean":"2024-04-17T10:35:37.095476+0000","last_became_active":"2024-04-17T10:35:36.082901+0000","last_became_peered":"2024-04-17T10:35:36.082901+0000","last_unstale":"2024-04-17T10:35:37.095476+0000","last_undegraded":"2024-04-17T10:35:37.095476+0000","last_fullsized":"2024-04-17T10:35:37.095476+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:48:09.890350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.095127+0000","last_change":"2024-04-17T10:35:37.092021+0000","last_active":"2024-04-17T10:35:37.095127+0000","last_peered":"2024-04-17T10:35:37.095127+0000","last_clean":"2024-04-17T10:35:37.095127+0000","last_became_active":"2024-04-17T10:35:37.091586+0000","last_became_peered":"2024-04-17T10:35:37.091586+0000","last_unstale":"2024-04-17T10:35:37.095127+0000","last_undegraded":"2024-04-17T10:35:37.095127+0000","last_fullsized":"2024-04-17T10:35:37.095127+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:53.265621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.086568+0000","last_change":"2024-04-17T10:35:36.091248+0000","last_active":"2024-04-17T10:35:37.086568+0000","last_peered":"2024-04-17T10:35:37.086568+0000","last_clean":"2024-04-17T10:35:37.086568+0000","last_became_active":"2024-04-17T10:35:36.091008+0000","last_became_peered":"2024-04-17T10:35:36.091008+0000","last_unstale":"2024-04-17T10:35:37.086568+0000","last_undegraded":"2024-04-17T10:35:37.086568+0000","last_fullsized":"2024-04-17T10:35:37.086568+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T14:19:35.296346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.105334+0000","last_change":"2024-04-17T10:35:37.097301+0000","last_active":"2024-04-17T10:35:37.105334+0000","last_peered":"2024-04-17T10:35:37.105334+0000","last_clean":"2024-04-17T10:35:37.105334+0000","last_became_active":"2024-04-17T10:35:37.096913+0000","last_became_peered":"2024-04-17T10:35:37.096913+0000","last_unstale":"2024-04-17T10:35:37.105334+0000","last_undegraded":"2024-04-17T10:35:37.105334+0000","last_fullsized":"2024-04-17T10:35:37.105334+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:59.902902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.099358+0000","last_change":"2024-04-17T10:35:36.090314+0000","last_active":"2024-04-17T10:35:37.099358+0000","last_peered":"2024-04-17T10:35:37.099358+0000","last_clean":"2024-04-17T10:35:37.099358+0000","last_became_active":"2024-04-17T10:35:36.090030+0000","last_became_peered":"2024-04-17T10:35:36.090030+0000","last_unstale":"2024-04-17T10:35:37.099358+0000","last_undegraded":"2024-04-17T10:35:37.099358+0000","last_fullsized":"2024-04-17T10:35:37.099358+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:30.198188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.094585+0000","last_change":"2024-04-17T10:35:37.090132+0000","last_active":"2024-04-17T10:35:37.094585+0000","last_peered":"2024-04-17T10:35:37.094585+0000","last_clean":"2024-04-17T10:35:37.094585+0000","last_became_active":"2024-04-17T10:35:37.089836+0000","last_became_peered":"2024-04-17T10:35:37.089836+0000","last_unstale":"2024-04-17T10:35:37.094585+0000","last_undegraded":"2024-04-17T10:35:37.094585+0000","last_fullsized":"2024-04-17T10:35:37.094585+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T19:59:59.902902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.100607+0000","last_change":"2024-04-17T10:35:36.077405+0000","last_active":"2024-04-17T10:35:37.100607+0000","last_peered":"2024-04-17T10:35:37.100607+0000","last_clean":"2024-04-17T10:35:37.100607+0000","last_became_active":"2024-04-17T10:35:36.077141+0000","last_became_peered":"2024-04-17T10:35:36.077141+0000","last_unstale":"2024-04-17T10:35:37.100607+0000","last_undegraded":"2024-04-17T10:35:37.100607+0000","last_fullsized":"2024-04-17T10:35:37.100607+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T15:04:07.895511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.115247+0000","last_change":"2024-04-17T10:35:37.112239+0000","last_active":"2024-04-17T10:35:37.115247+0000","last_peered":"2024-04-17T10:35:37.115247+0000","last_clean":"2024-04-17T10:35:37.115247+0000","last_became_active":"2024-04-17T10:35:37.111944+0000","last_became_peered":"2024-04-17T10:35:37.111944+0000","last_unstale":"2024-04-17T10:35:37.115247+0000","last_undegraded":"2024-04-17T10:35:37.115247+0000","last_fullsized":"2024-04-17T10:35:37.115247+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T12:19:31.204368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.098721+0000","last_change":"2024-04-17T10:35:36.083966+0000","last_active":"2024-04-17T10:35:37.098721+0000","last_peered":"2024-04-17T10:35:37.098721+0000","last_clean":"2024-04-17T10:35:37.098721+0000","last_became_active":"2024-04-17T10:35:36.083616+0000","last_became_peered":"2024-04-17T10:35:36.083616+0000","last_unstale":"2024-04-17T10:35:37.098721+0000","last_undegraded":"2024-04-17T10:35:37.098721+0000","last_fullsized":"2024-04-17T10:35:37.098721+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T22:01:28.562798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.116041+0000","last_change":"2024-04-17T10:35:37.110975+0000","last_active":"2024-04-17T10:35:37.116041+0000","last_peered":"2024-04-17T10:35:37.116041+0000","last_clean":"2024-04-17T10:35:37.116041+0000","last_became_active":"2024-04-17T10:35:37.110718+0000","last_became_peered":"2024-04-17T10:35:37.110718+0000","last_unstale":"2024-04-17T10:35:37.116041+0000","last_undegraded":"2024-04-17T10:35:37.116041+0000","last_fullsized":"2024-04-17T10:35:37.116041+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:34:48.986405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.088090+0000","last_change":"2024-04-17T10:35:36.093527+0000","last_active":"2024-04-17T10:35:37.088090+0000","last_peered":"2024-04-17T10:35:37.088090+0000","last_clean":"2024-04-17T10:35:37.088090+0000","last_became_active":"2024-04-17T10:35:36.093234+0000","last_became_peered":"2024-04-17T10:35:36.093234+0000","last_unstale":"2024-04-17T10:35:37.088090+0000","last_undegraded":"2024-04-17T10:35:37.088090+0000","last_fullsized":"2024-04-17T10:35:37.088090+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:06:52.259442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":9,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.113503+0000","last_change":"2024-04-17T10:35:37.108330+0000","last_active":"2024-04-17T10:35:37.113503+0000","last_peered":"2024-04-17T10:35:37.113503+0000","last_clean":"2024-04-17T10:35:37.113503+0000","last_became_active":"2024-04-17T10:35:37.108027+0000","last_became_peered":"2024-04-17T10:35:37.108027+0000","last_unstale":"2024-04-17T10:35:37.113503+0000","last_undegraded":"2024-04-17T10:35:37.113503+0000","last_fullsized":"2024-04-17T10:35:37.113503+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":139,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:36.063592+0000","last_clean_scrub_stamp":"2024-04-17T10:35:36.063592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T11:36:20.016329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.14","version":"140'6","reported_seq":18,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.126711+0000","last_change":"2024-04-17T10:35:36.088856+0000","last_active":"2024-04-17T10:35:37.126711+0000","last_peered":"2024-04-17T10:35:37.126711+0000","last_clean":"2024-04-17T10:35:37.126711+0000","last_became_active":"2024-04-17T10:35:36.088591+0000","last_became_peered":"2024-04-17T10:35:36.088591+0000","last_unstale":"2024-04-17T10:35:37.126711+0000","last_undegraded":"2024-04-17T10:35:37.126711+0000","last_fullsized":"2024-04-17T10:35:37.126711+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:35:35.057412+0000","last_clean_scrub_stamp":"2024-04-17T10:35:35.057412+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T18:47:36.588267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":391,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-04-17T10:35:37.088152+0000","last_change":"2024-04-17T10:24:43.632176+0000","last_active":"2024-04-17T10:35:37.088152+0000","last_peered":"2024-04-17T10:35:37.088152+0000","last_clean":"2024-04-17T10:35:37.088152+0000","last_became_active":"2024-04-17T10:24:43.631090+0000","last_became_peered":"2024-04-17T10:24:43.631090+0000","last_unstale":"2024-04-17T10:35:37.088152+0000","last_undegraded":"2024-04-17T10:35:37.088152+0000","last_fullsized":"2024-04-17T10:35:37.088152+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-17T10:24:42.514115+0000","last_clean_scrub_stamp":"2024-04-17T10:24:42.514115+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-18T20:34:44.401661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":22,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":21,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574983,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55676,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316164,"statfs":{"total":96636764160,"available":96579751936,"internally_reserved":0,"allocated":2338816,"data_stored":1405782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574981,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55092,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316748,"statfs":{"total":96636764160,"available":96580349952,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50936,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320904,"statfs":{"total":96636764160,"available":96584605696,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574982,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59252,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312588,"statfs":{"total":96636764160,"available":96576090112,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81404,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":94290436,"statfs":{"total":96636764160,"available":96553406464,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81590727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55096,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316744,"statfs":{"total":96636764160,"available":96580345856,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574981,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50940,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320900,"statfs":{"total":96636764160,"available":96584601600,"internally_reserved":0,"allocated":1744896,"data_stored":815414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574982,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59872,"kb_used_data":2320,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311968,"statfs":{"total":96636764160,"available":96575455232,"internally_reserved":0,"allocated":2375680,"data_stored":1415267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-17T10:35:41.951 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-17T10:35:41.951 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-17T10:35:42.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.151+0000 7fc6ae200640 1 Processor -- start 2024-04-17T10:35:42.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.151+0000 7fc6ae200640 1 -- start start 2024-04-17T10:35:42.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6ae200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a80aef60 0x7fc6a80cc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6ae200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fc6a806e980 0x7fc6a80b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6ae200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a80b4990 con 0x7fc6a80cce60 2024-04-17T10:35:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6ae200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a80cd2c0 con 0x7fc6a806e980 2024-04-17T10:35:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6ae200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80af340 con 0x7fc6a80aef60 2024-04-17T10:35:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc69fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc6a80cce60 0x7fc6a806e270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47148/0 (socket says 172.21.15.67:47148) 2024-04-17T10:35:42.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc69fe00640 1 -- 172.21.15.67:0/202419258 learned_addr learned my addr 172.21.15.67:0/202419258 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/202419258 >> v2:172.21.15.86:3300/0 conn(0x7fc6a806e980 0x7fc6a80b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a7400640 1 --2- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a80aef60 0x7fc6a80cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a6a00640 1 -- 172.21.15.67:0/202419258 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793546450 0 0) 0x7fc690001120 con 0x7fc6a80cce60 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a6a00640 1 -- 172.21.15.67:0/202419258 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc68c003530 con 0x7fc6a80cce60 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a7e00640 1 -- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a80aef60 msgr2=0x7fc6a80cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a80aef60 0x7fc6a80cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a7e00640 1 -- 172.21.15.67:0/202419258 >> v1:172.21.15.67:6789/0 conn(0x7fc6a80cce60 legacy=0x7fc6a806e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a6a00640 1 -- 172.21.15.67:0/202419258 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4097510925 0 0) 0x7fc68c003530 con 0x7fc6a80cce60 2024-04-17T10:35:42.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.155+0000 7fc6a7e00640 1 -- 172.21.15.67:0/202419258 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a815ae90 con 0x7fc6a806e980 2024-04-17T10:35:42.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a806e980 0x7fc6a80b3120 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009d90 tx=0x7fc6a001c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d33ac39e212dcf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:42.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6a6a00640 1 -- 172.21.15.67:0/202419258 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc690001850 con 0x7fc6a806e980 2024-04-17T10:35:42.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6a6a00640 1 -- 172.21.15.67:0/202419258 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a00a6500 con 0x7fc6a806e980 2024-04-17T10:35:42.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6a6a00640 1 -- 172.21.15.67:0/202419258 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00a68d0 con 0x7fc6a806e980 2024-04-17T10:35:42.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 -- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a806e980 msgr2=0x7fc6a80b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 --2- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a806e980 0x7fc6a80b3120 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc6a0009d90 tx=0x7fc6a001c800 comp rx=0 tx=0).stop 2024-04-17T10:35:42.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 -- 172.21.15.67:0/202419258 shutdown_connections 2024-04-17T10:35:42.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 --2- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a806e980 0x7fc6a80b3120 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 --2- 172.21.15.67:0/202419258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a80aef60 0x7fc6a80cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 -- 172.21.15.67:0/202419258 >> 172.21.15.67:0/202419258 conn(0x7fc6a80b1740 msgr2=0x7fc6a80b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:42.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 -- 172.21.15.67:0/202419258 shutdown_connections 2024-04-17T10:35:42.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 -- 172.21.15.67:0/202419258 wait complete. 2024-04-17T10:35:42.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 Processor -- start 2024-04-17T10:35:42.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.159+0000 7fc6ae200640 1 -- start start 2024-04-17T10:35:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a806e980 0x7fc6a8150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a80aef60 0x7fc6a8154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a80cce60 0x7fc6a81545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a815bc10 con 0x7fc6a80cce60 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a80b3e70 con 0x7fc6a80aef60 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a813b390 con 0x7fc6a806e980 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a80cce60 0x7fc6a81545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a80cce60 0x7fc6a81545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49286/0 (socket says 172.21.15.67:49286) 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 -- 172.21.15.67:0/190645016 learned_addr learned my addr 172.21.15.67:0/190645016 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc69fe00640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a80aef60 0x7fc6a8154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7400640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a806e980 0x7fc6a8150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 -- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a806e980 msgr2=0x7fc6a8150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a806e980 0x7fc6a8150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 -- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a80aef60 msgr2=0x7fc6a8154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a80aef60 0x7fc6a8154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 -- 172.21.15.67:0/190645016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a813b670 con 0x7fc6a80cce60 2024-04-17T10:35:42.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6a7e00640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a80cce60 0x7fc6a81545b0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fc6a001d0a0 tx=0x7fc6a00195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:42.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc69f400640 1 -- 172.21.15.67:0/190645016 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00be070 con 0x7fc6a80cce60 2024-04-17T10:35:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc69f400640 1 -- 172.21.15.67:0/190645016 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a00b8840 con 0x7fc6a80cce60 2024-04-17T10:35:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a813b950 con 0x7fc6a80cce60 2024-04-17T10:35:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc69f400640 1 -- 172.21.15.67:0/190645016 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a00b8bf0 con 0x7fc6a80cce60 2024-04-17T10:35:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a813b010 con 0x7fc6a80cce60 2024-04-17T10:35:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a804ef30 con 0x7fc6a80cce60 2024-04-17T10:35:42.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.163+0000 7fc69f400640 1 -- 172.21.15.67:0/190645016 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc6a00c1070 con 0x7fc6a80cce60 2024-04-17T10:35:42.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.167+0000 7fc69f400640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc684066f60 0x7fc684069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.167+0000 7fc69f400640 1 -- 172.21.15.67:0/190645016 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fc6a01209c0 con 0x7fc6a80cce60 2024-04-17T10:35:42.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.167+0000 7fc6a7400640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc684066f60 0x7fc684069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.167+0000 7fc6a7400640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc684066f60 0x7fc684069420 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc698009af0 tx=0x7fc698008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:42.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.171+0000 7fc69f400640 1 -- 172.21.15.67:0/190645016 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc6a0016480 con 0x7fc6a80cce60 2024-04-17T10:35:42.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.391+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc6a81516c0 con 0x7fc6a80cce60 2024-04-17T10:35:42.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.391+0000 7fc69f400640 1 -- 172.21.15.67:0/190645016 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc6a00f1830 con 0x7fc6a80cce60 2024-04-17T10:35:42.382 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-04-17T10:35:42.382 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQA7ox9mjzsWChAASGbhMMMi9f+jflcFOrdOww== 2024-04-17T10:35:42.382 INFO:teuthology.orchestra.run.smithi067.stdout: caps mds = "allow" 2024-04-17T10:35:42.382 INFO:teuthology.orchestra.run.smithi067.stdout: caps mon = "allow r" 2024-04-17T10:35:42.382 INFO:teuthology.orchestra.run.smithi067.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-17T10:35:42.383 INFO:teuthology.orchestra.run.smithi067.stderr:updated caps for client.0 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc684066f60 msgr2=0x7fc684069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc684066f60 0x7fc684069420 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fc698009af0 tx=0x7fc698008040 comp rx=0 tx=0).stop 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a80cce60 msgr2=0x7fc6a81545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a80cce60 0x7fc6a81545b0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fc6a001d0a0 tx=0x7fc6a00195a0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 shutdown_connections 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc684066f60 0x7fc684069420 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a80cce60 0x7fc6a81545b0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6a80aef60 0x7fc6a8154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 --2- 172.21.15.67:0/190645016 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6a806e980 0x7fc6a8150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 >> 172.21.15.67:0/190645016 conn(0x7fc6a80b1740 msgr2=0x7fc6a80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 shutdown_connections 2024-04-17T10:35:42.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.395+0000 7fc6ae200640 1 -- 172.21.15.67:0/190645016 wait complete. 2024-04-17T10:35:42.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:35:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4522c00640 1 Processor -- start 2024-04-17T10:35:42.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4522c00640 1 -- start start 2024-04-17T10:35:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4522c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f451c15dd10 0x7f451c0b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4522c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b57b0 0x7f451c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4522c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f451c0b0cc0 con 0x7f451c0b07b0 2024-04-17T10:35:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4522c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f451c0b1280 con 0x7f451c15dd10 2024-04-17T10:35:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4522c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451c0b0fa0 con 0x7f451c0b57b0 2024-04-17T10:35:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4520c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f451c0b07b0 0x7f451c1618c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47164/0 (socket says 172.21.15.67:47164) 2024-04-17T10:35:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4520c00640 1 -- 172.21.15.67:0/1430689277 learned_addr learned my addr 172.21.15.67:0/1430689277 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:42.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451be00640 1 --2- 172.21.15.67:0/1430689277 >> v2:172.21.15.86:3300/0 conn(0x7f451c15dd10 0x7f451c0b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f4513400640 1 --2- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b57b0 0x7f451c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451b400640 1 -- 172.21.15.67:0/1430689277 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910631768 0 0) 0x7f451c0b0cc0 con 0x7f451c0b07b0 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451b400640 1 -- 172.21.15.67:0/1430689277 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4500003530 con 0x7f451c0b07b0 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451b400640 1 -- 172.21.15.67:0/1430689277 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 739689985 0 0) 0x7f4500003530 con 0x7f451c0b07b0 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451be00640 1 -- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b57b0 msgr2=0x7f451c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451be00640 1 --2- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b57b0 0x7f451c070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451be00640 1 -- 172.21.15.67:0/1430689277 >> v1:172.21.15.67:6789/0 conn(0x7f451c0b07b0 legacy=0x7f451c1618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.555+0000 7f451be00640 1 -- 172.21.15.67:0/1430689277 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c06bed0 con 0x7f451c15dd10 2024-04-17T10:35:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f451be00640 1 --2- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c15dd10 0x7f451c0b5270 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f45040097c0 tx=0x7f450401c300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc804168be8ec3f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f451b400640 1 -- 172.21.15.67:0/1430689277 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45040b0070 con 0x7f451c15dd10 2024-04-17T10:35:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f451b400640 1 -- 172.21.15.67:0/1430689277 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45040ab400 con 0x7f451c15dd10 2024-04-17T10:35:42.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f451b400640 1 -- 172.21.15.67:0/1430689277 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514001a00 con 0x7f451c15dd10 2024-04-17T10:35:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 -- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c15dd10 msgr2=0x7f451c0b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 --2- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c15dd10 0x7f451c0b5270 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f45040097c0 tx=0x7f450401c300 comp rx=0 tx=0).stop 2024-04-17T10:35:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 -- 172.21.15.67:0/1430689277 shutdown_connections 2024-04-17T10:35:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 --2- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b57b0 0x7f451c070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 --2- 172.21.15.67:0/1430689277 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c15dd10 0x7f451c0b5270 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 -- 172.21.15.67:0/1430689277 >> 172.21.15.67:0/1430689277 conn(0x7f451c0b34d0 msgr2=0x7f451c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 -- 172.21.15.67:0/1430689277 shutdown_connections 2024-04-17T10:35:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.559+0000 7f4522c00640 1 -- 172.21.15.67:0/1430689277 wait complete. 2024-04-17T10:35:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 Processor -- start 2024-04-17T10:35:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 -- start start 2024-04-17T10:35:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b07b0 0x7f451c14c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451c15dd10 0x7f451c14d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c14cd90 0x7f451c14dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c16ae70 con 0x7f451c15dd10 2024-04-17T10:35:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451c071710 con 0x7f451c14cd90 2024-04-17T10:35:42.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f451c16aff0 con 0x7f451c0b07b0 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451c15dd10 0x7f451c14d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451c15dd10 0x7f451c14d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49298/0 (socket says 172.21.15.67:49298) 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 -- 172.21.15.67:0/1234657524 learned_addr learned my addr 172.21.15.67:0/1234657524 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f451be00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b07b0 0x7f451c14c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4520c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c14cd90 0x7f451c14dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 -- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b07b0 msgr2=0x7f451c14c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b07b0 0x7f451c14c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 -- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c14cd90 msgr2=0x7f451c14dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c14cd90 0x7f451c14dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 -- 172.21.15.67:0/1234657524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c1506a0 con 0x7f451c15dd10 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4513400640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451c15dd10 0x7f451c14d690 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f450c00e930 tx=0x7f450c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4518c00640 1 -- 172.21.15.67:0/1234657524 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c017070 con 0x7f451c15dd10 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4518c00640 1 -- 172.21.15.67:0/1234657524 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c00ca00 con 0x7f451c15dd10 2024-04-17T10:35:42.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4518c00640 1 -- 172.21.15.67:0/1234657524 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c00cdb0 con 0x7f451c15dd10 2024-04-17T10:35:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451c150980 con 0x7f451c15dd10 2024-04-17T10:35:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.563+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451c14c050 con 0x7f451c15dd10 2024-04-17T10:35:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.567+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f451c070bb0 con 0x7f451c15dd10 2024-04-17T10:35:42.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.567+0000 7f4518c00640 1 -- 172.21.15.67:0/1234657524 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f450c01a070 con 0x7f451c15dd10 2024-04-17T10:35:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.567+0000 7f4518c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f44f8066f90 0x7f44f8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.567+0000 7f4518c00640 1 -- 172.21.15.67:0/1234657524 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f450c085520 con 0x7f451c15dd10 2024-04-17T10:35:42.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.567+0000 7f451be00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f44f8066f90 0x7f44f8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:42.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.567+0000 7f451be00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f44f8066f90 0x7f44f8069450 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f45040ad000 tx=0x7f45040a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:42.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.571+0000 7f4518c00640 1 -- 172.21.15.67:0/1234657524 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f450c055d70 con 0x7f451c15dd10 2024-04-17T10:35:42.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f451c074e30 con 0x7f451c15dd10 2024-04-17T10:35:42.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4518c00640 1 -- 172.21.15.67:0/1234657524 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+4941 (secure 0 0 0) 0x7f450c059e10 con 0x7f451c15dd10 2024-04-17T10:35:42.756 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:35:42.756 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":103,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:35:39.497533+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9777},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9777":{"gid":9777,"name":"c","rank":0,"incarnation":99,"state":"up:active","state_seq":3,"addr":"172.21.15.67:6834/1952434688","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":1952434688},{"type":"v1","addr":"172.21.15.67:6834","nonce":1952434688}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:35:42.757 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 103 2024-04-17T10:35:42.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f44f8066f90 msgr2=0x7f44f8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f44f8066f90 0x7f44f8069450 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f45040ad000 tx=0x7f45040a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451c15dd10 msgr2=0x7f451c14d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451c15dd10 0x7f451c14d690 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f450c00e930 tx=0x7f450c00edf0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 shutdown_connections 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f44f8066f90 0x7f44f8069450 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f451c14cd90 0x7f451c14dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f451c15dd10 0x7f451c14d690 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 --2- 172.21.15.67:0/1234657524 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f451c0b07b0 0x7f451c14c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 >> 172.21.15.67:0/1234657524 conn(0x7f451c0b34d0 msgr2=0x7f451c156e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 shutdown_connections 2024-04-17T10:35:42.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:42.767+0000 7f4522c00640 1 -- 172.21.15.67:0/1234657524 wait complete. 2024-04-17T10:35:43.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 103, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:35:39.497533+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9777}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9777': {'gid': 9777, 'name': 'c', 'rank': 0, 'incarnation': 99, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.67:6834/1952434688', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.67:6832', 'nonce': 1952434688}, {'type': 'v1', 'addr': '172.21.15.67:6834', 'nonce': 1952434688}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:35:43.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-17T10:35:43.181 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-17T10:35:43.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f5992400640 1 Processor -- start 2024-04-17T10:35:43.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f5992400640 1 -- start start 2024-04-17T10:35:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f5992400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f598c06eac0 0x7f598c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f5992400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c15ae40 con 0x7f598c0cf0c0 2024-04-17T10:35:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f5992400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c071440 con 0x7f598c06e6e0 2024-04-17T10:35:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f5992400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c071720 con 0x7f598c06eac0 2024-04-17T10:35:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f598be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f598c0cf0c0 0x7f598c144020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47172/0 (socket says 172.21.15.67:47172) 2024-04-17T10:35:43.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f598be00640 1 -- 172.21.15.67:0/1985199749 learned_addr learned my addr 172.21.15.67:0/1985199749 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:35:43.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f5982a00640 1 --2- 172.21.15.67:0/1985199749 >> v2:172.21.15.86:3300/0 conn(0x7f598c06eac0 0x7f598c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.335+0000 7f598aa00640 1 -- 172.21.15.67:0/1985199749 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502501536 0 0) 0x7f598c15ae40 con 0x7f598c0cf0c0 2024-04-17T10:35:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f598aa00640 1 -- 172.21.15.67:0/1985199749 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5970003530 con 0x7f598c0cf0c0 2024-04-17T10:35:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f598aa00640 1 -- 172.21.15.67:0/1985199749 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903528757 0 0) 0x7f5970003530 con 0x7f598c0cf0c0 2024-04-17T10:35:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f5982a00640 1 -- 172.21.15.67:0/1985199749 >> v1:172.21.15.86:6789/0 conn(0x7f598c06e6e0 legacy=0x7f598c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f5982a00640 1 -- 172.21.15.67:0/1985199749 >> v1:172.21.15.67:6789/0 conn(0x7f598c0cf0c0 legacy=0x7f598c144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f5982a00640 1 -- 172.21.15.67:0/1985199749 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c071a00 con 0x7f598c06eac0 2024-04-17T10:35:43.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f5982a00640 1 --2- 172.21.15.67:0/1985199749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c06eac0 0x7f598c143ae0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f59740097c0 tx=0x7f597401c1b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=779358c4f7063ad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f598aa00640 1 -- 172.21.15.67:0/1985199749 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740b0070 con 0x7f598c06eac0 2024-04-17T10:35:43.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f598aa00640 1 -- 172.21.15.67:0/1985199749 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59740ab400 con 0x7f598c06eac0 2024-04-17T10:35:43.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.339+0000 7f598aa00640 1 -- 172.21.15.67:0/1985199749 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5984001a00 con 0x7f598c06eac0 2024-04-17T10:35:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- 172.21.15.67:0/1985199749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c06eac0 msgr2=0x7f598c143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 --2- 172.21.15.67:0/1985199749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c06eac0 0x7f598c143ae0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f59740097c0 tx=0x7f597401c1b0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- 172.21.15.67:0/1985199749 shutdown_connections 2024-04-17T10:35:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 --2- 172.21.15.67:0/1985199749 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c06eac0 0x7f598c143ae0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- 172.21.15.67:0/1985199749 >> 172.21.15.67:0/1985199749 conn(0x7f598c06ac90 msgr2=0x7f598c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- 172.21.15.67:0/1985199749 shutdown_connections 2024-04-17T10:35:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- 172.21.15.67:0/1985199749 wait complete. 2024-04-17T10:35:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 Processor -- start 2024-04-17T10:35:43.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- start start 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c06e6e0 0x7f598c15a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598c06eac0 0x7f598c0aa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c0cf0c0 0x7f598c0aad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c1680b0 con 0x7f598c06e6e0 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c167f30 con 0x7f598c0cf0c0 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5992400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c168230 con 0x7f598c06eac0 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c06e6e0 0x7f598c15a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c06e6e0 0x7f598c15a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49310/0 (socket says 172.21.15.67:49310) 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 -- 172.21.15.67:0/1976817264 learned_addr learned my addr 172.21.15.67:0/1976817264 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f5982a00640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598c06eac0 0x7f598c0aa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598be00640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c0cf0c0 0x7f598c0aad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 -- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598c06eac0 msgr2=0x7f598c0aa7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598c06eac0 0x7f598c0aa7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 -- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c0cf0c0 msgr2=0x7f598c0aad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c0cf0c0 0x7f598c0aad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 -- 172.21.15.67:0/1976817264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c0af040 con 0x7f598c06e6e0 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.343+0000 7f598b400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c06e6e0 0x7f598c15a080 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f597c00c7b0 tx=0x7f597c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.347+0000 7f5983e00640 1 -- 172.21.15.67:0/1976817264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c017070 con 0x7f598c06e6e0 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.347+0000 7f5983e00640 1 -- 172.21.15.67:0/1976817264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c005270 con 0x7f598c06e6e0 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.347+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c0af320 con 0x7f598c06e6e0 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.347+0000 7f5983e00640 1 -- 172.21.15.67:0/1976817264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c00a640 con 0x7f598c06e6e0 2024-04-17T10:35:43.336 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.347+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0af8e0 con 0x7f598c06e6e0 2024-04-17T10:35:43.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.347+0000 7f5983e00640 1 -- 172.21.15.67:0/1976817264 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f597c01a070 con 0x7f598c06e6e0 2024-04-17T10:35:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.347+0000 7f5983e00640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5968066f60 0x7f5968069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f5983e00640 1 -- 172.21.15.67:0/1976817264 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f597c084640 con 0x7f598c06e6e0 2024-04-17T10:35:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 Processor -- start 2024-04-17T10:35:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f5982a00640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5968066f60 0x7f5968069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- start start 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f5982a00640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5968066f60 0x7f5968069420 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f59740ad000 tx=0x7f59740a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e318054d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318055240 0x55e31800a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e31800af30 0x55e31800b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e317ef25f0 con 0x55e318054920 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e317e59da0 con 0x55e31800af30 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e317de2150 con 0x55e318055240 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e318054d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e318054d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49324/0 (socket says 172.21.15.67:49324) 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 -- 172.21.15.67:0/4040719518 learned_addr learned my addr 172.21.15.67:0/4040719518 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f5982a00640 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318055240 0x55e31800a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598be00640 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e31800af30 0x55e31800b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 -- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318055240 msgr2=0x55e31800a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318055240 0x55e31800a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 -- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e31800af30 msgr2=0x55e31800b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e31800af30 0x55e31800b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 -- 172.21.15.67:0/4040719518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e317dfbca0 con 0x55e318054920 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598be00640 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e31800af30 0x55e31800b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f598b400640 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e318054d00 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f597c05d060 tx=0x7f597c001240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7ca11060726238 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f597be00640 1 -- 172.21.15.67:0/4040719518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c053230 con 0x55e318054920 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f597be00640 1 -- 172.21.15.67:0/4040719518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c0515a0 con 0x55e318054920 2024-04-17T10:35:43.342 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f597be00640 1 -- 172.21.15.67:0/4040719518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c05b2b0 con 0x55e318054920 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- 172.21.15.67:0/4040719518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 msgr2=0x55e318054d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e318054d00 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7f597c05d060 tx=0x7f597c001240 comp rx=0 tx=0).stop 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- 172.21.15.67:0/4040719518 shutdown_connections 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e31800af30 0x55e31800b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318055240 0x55e31800a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 --2- 172.21.15.67:0/4040719518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e318054d00 unknown :-1 s=CLOSED pgs=1907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- 172.21.15.67:0/4040719518 >> 172.21.15.67:0/4040719518 conn(0x55e3180112a0 msgr2=0x55e3180092d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- 172.21.15.67:0/4040719518 shutdown_connections 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.351+0000 7f59957da000 1 -- 172.21.15.67:0/4040719518 wait complete. 2024-04-17T10:35:43.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 -- start start 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 Processor -- start 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e31800af30 0x55e31805bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e31805c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e31800eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e317e14990 con 0x55e318054920 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e317e55e40 con 0x55e318055240 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e317de5900 con 0x55e31800af30 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 10 client.0 _ll_register_callbacks cb 0x55e317f0d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e31805c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e31805c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49332/0 (socket says 172.21.15.67:49332) 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 -- 172.21.15.67:0/67287973 learned_addr learned my addr 172.21.15.67:0/67287973 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f598b400640 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e31800af30 0x55e31805bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f598be00640 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e31800eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 -- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e31800af30 msgr2=0x55e31805bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e31800af30 0x55e31805bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 -- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 msgr2=0x55e31800eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e31800eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 -- 172.21.15.67:0/67287973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e317e12630 con 0x55e318054920 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5982a00640 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e31805c2a0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f59740097c0 tx=0x7f597401a9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74758a3acfb7e5c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5978c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5978c00640 1 -- 172.21.15.67:0/67287973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740c8020 con 0x55e318054920 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5978c00640 1 -- 172.21.15.67:0/67287973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f59740c4450 con 0x55e318054920 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f597c0515a0 con 0x55e318054920 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5978c00640 1 -- 172.21.15.67:0/67287973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59740b0040 con 0x55e318054920 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 10 client.9906 fetch_fsmap learned FSMap version 103 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) v3 -- 0x55e317de23a0 con 0x55e318054920 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5978c00640 1 -- 172.21.15.67:0/67287973 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c0515a0 con 0x55e318054920 2024-04-17T10:35:43.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f5978c00640 1 -- 172.21.15.67:0/67287973 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 103) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f59740c4650 con 0x55e318054920 2024-04-17T10:35:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 10 client.9906 fetch_fsmap finished waiting for FSMap version 103 2024-04-17T10:35:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 10 client.9906 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:35:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 20 client.9906 populate_metadata read hostname 'smithi067' 2024-04-17T10:35:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e31800f4b0 0x55e31800f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 4 client.9906 mds_command: new command op to 9777 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-17T10:35:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 --> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e317df5a50 con 0x55e31800f4b0 2024-04-17T10:35:43.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.355+0000 7f598b400640 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e31800f4b0 0x55e31800f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f598b400640 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e31800f4b0 0x55e31800f890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e52ec95330a4b11 server_cookie=fed5c0248b20513b in_seq=0 out_seq=0 2024-04-17T10:35:43.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f5978c00640 10 client.9906 ms_handle_connect on v2:172.21.15.67:6832/1952434688 2024-04-17T10:35:43.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f5978c00640 1 -- 172.21.15.67:0/67287973 <== mds.0 v2:172.21.15.67:6832/1952434688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e317df5a50 con 0x55e31800f4b0 2024-04-17T10:35:43.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f5978c00640 10 client.9906 handle_command_reply: tid=0 2024-04-17T10:35:43.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 client.9906 shutdown 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 msgr2=0x55e31805c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e31805c2a0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f59740097c0 tx=0x7f597401a9f0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 shutdown_connections 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e31800f4b0 0x55e31800f890 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e31800eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e318054920 0x55e31805c2a0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 --2- 172.21.15.67:0/67287973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e31800af30 0x55e31805bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f5978c00640 0 client.9906 ms_handle_reset on v2:172.21.15.67:6832/1952434688 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 >> 172.21.15.67:0/67287973 conn(0x55e3180112a0 msgr2=0x55e3180099e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 shutdown_connections 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 1 -- 172.21.15.67:0/67287973 wait complete. 2024-04-17T10:35:43.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.359+0000 7f59957da000 20 client.9906 trim_cache size 0 max 16384 2024-04-17T10:35:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 Processor -- start 2024-04-17T10:35:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 -- start start 2024-04-17T10:35:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c2020c0 con 0x55e31800af30 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0c9810 con 0x55e318055240 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0cb3a0 con 0x55e318054920 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49344/0 (socket says 172.21.15.67:49344) 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 -- 172.21.15.67:0/396073015 learned_addr learned my addr 172.21.15.67:0/396073015 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598be00640 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f5982a00640 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 -- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 msgr2=0x55e3180b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180b05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 -- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 msgr2=0x55e3180b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 -- 172.21.15.67:0/396073015 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e318055070 con 0x55e31800af30 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f598b400640 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180b00b0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f597c082cd0 tx=0x7f597c084000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0760a32895a2001 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f597b400640 1 -- 172.21.15.67:0/396073015 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c051650 con 0x55e31800af30 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f597b400640 1 -- 172.21.15.67:0/396073015 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c05b0b0 con 0x55e31800af30 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.375+0000 7f597b400640 1 -- 172.21.15.67:0/396073015 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c05d100 con 0x55e31800af30 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/396073015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 msgr2=0x55e3180b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180b00b0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f597c082cd0 tx=0x7f597c084000 comp rx=0 tx=0).stop 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/396073015 shutdown_connections 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180b05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- 172.21.15.67:0/396073015 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180b00b0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/396073015 >> 172.21.15.67:0/396073015 conn(0x55e3180112a0 msgr2=0x55e31800e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/396073015 shutdown_connections 2024-04-17T10:35:43.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/396073015 wait complete. 2024-04-17T10:35:43.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- start start 2024-04-17T10:35:43.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 Processor -- start 2024-04-17T10:35:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180612a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180617e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e317e5e3d0 con 0x55e31800af30 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e317f15d50 con 0x55e318055240 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e317df4980 con 0x55e318054920 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 10 client.0 _ll_register_callbacks cb 0x7f597c054d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180612a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180612a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49360/0 (socket says 172.21.15.67:49360) 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 -- 172.21.15.67:0/3292862774 learned_addr learned my addr 172.21.15.67:0/3292862774 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f5982a00640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180617e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598be00640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 -- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 msgr2=0x55e3180617e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180617e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 -- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 msgr2=0x55e3180b17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 -- 172.21.15.67:0/3292862774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e317edb600 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598be00640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f598b400640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180612a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f597c00a430 tx=0x7f597c01e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c7479caf438798 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f5978c00640 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f5978c00640 1 -- 172.21.15.67:0/3292862774 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f597c054a70 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f5978c00640 1 -- 172.21.15.67:0/3292862774 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f597c082e00 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f597c05b0b0 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f5978c00640 1 -- 172.21.15.67:0/3292862774 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=103) v2 ==== 24+0+0 (secure 0 0 0) 0x7f597c05d100 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 10 client.9912 fetch_fsmap learned FSMap version 103 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=103,monmap=2+}) v3 -- 0x55e317dee9b0 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f5978c00640 1 -- 172.21.15.67:0/3292862774 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597c05b0b0 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f5978c00640 1 -- 172.21.15.67:0/3292862774 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 103) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f597c076070 con 0x55e31800af30 2024-04-17T10:35:43.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 10 client.9912 fetch_fsmap finished waiting for FSMap version 103 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 10 client.9912 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 20 client.9912 populate_metadata read hostname 'smithi067' 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e3180b1d20 0x55e3180b2100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 4 client.9912 mds_command: new command op to 9777 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.379+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 --> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e317e672e0 con 0x55e3180b1d20 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5982a00640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e3180b1d20 0x55e3180b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5982a00640 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e3180b1d20 0x55e3180b2100 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d58af868574cb33d server_cookie=2f3befde5312e591 in_seq=0 out_seq=0 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5978c00640 10 client.9912 ms_handle_connect on v2:172.21.15.67:6832/1952434688 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5978c00640 1 -- 172.21.15.67:0/3292862774 <== mds.0 v2:172.21.15.67:6832/1952434688 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55e317e672e0 con 0x55e3180b1d20 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5978c00640 10 client.9912 handle_command_reply: tid=0 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 client.9912 shutdown 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 msgr2=0x55e3180612a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180612a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f597c00a430 tx=0x7f597c01e040 comp rx=0 tx=0).stop 2024-04-17T10:35:43.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 shutdown_connections 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:6832/1952434688,v1:172.21.15.67:6834/1952434688] conn(0x55e3180b1d20 0x55e3180b2100 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x55e318055240 0x55e3180b17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x55e318054920 0x55e3180617e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5978c00640 0 client.9912 ms_handle_reset on v2:172.21.15.67:6832/1952434688 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 --2- 172.21.15.67:0/3292862774 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e31800af30 0x55e3180612a0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 >> 172.21.15.67:0/3292862774 conn(0x55e3180112a0 msgr2=0x55e31805c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 shutdown_connections 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 1 -- 172.21.15.67:0/3292862774 wait complete. 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f59957da000 20 client.9912 trim_cache size 0 max 16384 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-04-17T10:35:43.376 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "8e8d1faf-cb51-4015-918b-f894d5d28e99", 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 9777, 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 6.2888919369999998, 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 103, 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 141, 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 141, 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 11.126194146 2024-04-17T10:35:43.377 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-04-17T10:35:43.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5968066f60 msgr2=0x7f5968069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5968066f60 0x7f5968069420 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f59740ad000 tx=0x7f59740a6000 comp rx=0 tx=0).stop 2024-04-17T10:35:43.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c06e6e0 msgr2=0x7f598c15a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c06e6e0 0x7f598c15a080 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f597c00c7b0 tx=0x7f597c00cc70 comp rx=0 tx=0).stop 2024-04-17T10:35:43.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 shutdown_connections 2024-04-17T10:35:43.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5968066f60 0x7f5968069420 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f598c0cf0c0 0x7f598c0aad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f598c06eac0 0x7f598c0aa7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 --2- 172.21.15.67:0/1976817264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f598c06e6e0 0x7f598c15a080 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 >> 172.21.15.67:0/1976817264 conn(0x7f598c06ac90 msgr2=0x7f598c0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 shutdown_connections 2024-04-17T10:35:43.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.383+0000 7f5992400640 1 -- 172.21.15.67:0/1976817264 wait complete. 2024-04-17T10:35:43.391 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-17T10:35:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde4000640 1 Processor -- start 2024-04-17T10:35:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde4000640 1 -- start start 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde4000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdddc06eac0 0x7fdddc143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde4000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde4000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc144560 con 0x7fdddc06e6e0 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde4000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc15ae40 con 0x7fdddc06eac0 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde4000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc071440 con 0x7fdddc0cf0c0 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde0c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdddc06eac0 0x7fdddc143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde0c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdddc06eac0 0x7fdddc143ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49376/0 (socket says 172.21.15.67:49376) 2024-04-17T10:35:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde0c00640 1 -- 172.21.15.67:0/125563973 learned_addr learned my addr 172.21.15.67:0/125563973 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde0c00640 1 -- 172.21.15.67:0/125563973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 msgr2=0x7fdddc144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde2000640 1 --2- 172.21.15.67:0/125563973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde0c00640 1 --2- 172.21.15.67:0/125563973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde0c00640 1 -- 172.21.15.67:0/125563973 >> v1:172.21.15.86:6789/0 conn(0x7fdddc06e6e0 legacy=0x7fdddc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:35:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde0c00640 1 -- 172.21.15.67:0/125563973 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc071720 con 0x7fdddc06eac0 2024-04-17T10:35:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.543+0000 7fdde2000640 1 --2- 172.21.15.67:0/125563973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:35:43.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde0c00640 1 --2- 172.21.15.67:0/125563973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06eac0 0x7fdddc143ae0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009870 tx=0x7fddc801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf2080ab817674e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdddbe00640 1 -- 172.21.15.67:0/125563973 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc80ab640 con 0x7fdddc06eac0 2024-04-17T10:35:43.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdddbe00640 1 -- 172.21.15.67:0/125563973 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc80b0070 con 0x7fdddc06eac0 2024-04-17T10:35:43.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdddbe00640 1 -- 172.21.15.67:0/125563973 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc80aba00 con 0x7fdddc06eac0 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 -- 172.21.15.67:0/125563973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06eac0 msgr2=0x7fdddc143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 --2- 172.21.15.67:0/125563973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06eac0 0x7fdddc143ae0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fddc8009870 tx=0x7fddc801c460 comp rx=0 tx=0).stop 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 -- 172.21.15.67:0/125563973 shutdown_connections 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 --2- 172.21.15.67:0/125563973 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 --2- 172.21.15.67:0/125563973 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06eac0 0x7fdddc143ae0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 -- 172.21.15.67:0/125563973 >> 172.21.15.67:0/125563973 conn(0x7fdddc06ac90 msgr2=0x7fdddc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 -- 172.21.15.67:0/125563973 shutdown_connections 2024-04-17T10:35:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.547+0000 7fdde4000640 1 -- 172.21.15.67:0/125563973 wait complete. 2024-04-17T10:35:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 Processor -- start 2024-04-17T10:35:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 -- start start 2024-04-17T10:35:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06e6e0 0x7fdddc0c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdddc06eac0 0x7fdddc13bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0b3a40 con 0x7fdddc06e6e0 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0b38c0 con 0x7fdddc06eac0 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0b3bc0 con 0x7fdddc0cf0c0 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06e6e0 0x7fdddc0c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06e6e0 0x7fdddc0c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49382/0 (socket says 172.21.15.67:49382) 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 -- 172.21.15.67:0/3117897721 learned_addr learned my addr 172.21.15.67:0/3117897721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde2000640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde0c00640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdddc06eac0 0x7fdddc13bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 -- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 msgr2=0x7fdddc140c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc140c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 -- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdddc06eac0 msgr2=0x7fdddc13bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdddc06eac0 0x7fdddc13bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 -- 172.21.15.67:0/3117897721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc13f900 con 0x7fdddc06e6e0 2024-04-17T10:35:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde1600640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06e6e0 0x7fdddc0c8af0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fddd000c480 tx=0x7fddd000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fddd9600640 1 -- 172.21.15.67:0/3117897721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0012830 con 0x7fdddc06e6e0 2024-04-17T10:35:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fddd9600640 1 -- 172.21.15.67:0/3117897721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd0017070 con 0x7fdddc06e6e0 2024-04-17T10:35:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fddd9600640 1 -- 172.21.15.67:0/3117897721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddd0012c30 con 0x7fdddc06e6e0 2024-04-17T10:35:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc13fbc0 con 0x7fdddc06e6e0 2024-04-17T10:35:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.551+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc140180 con 0x7fdddc06e6e0 2024-04-17T10:35:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.555+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc072d50 con 0x7fdddc06e6e0 2024-04-17T10:35:43.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.555+0000 7fddd9600640 1 -- 172.21.15.67:0/3117897721 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fddd001c460 con 0x7fdddc06e6e0 2024-04-17T10:35:43.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.555+0000 7fddd9600640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fddbc066f60 0x7fddbc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:35:43.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.555+0000 7fddd9600640 1 -- 172.21.15.67:0/3117897721 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7fddd0084ef0 con 0x7fdddc06e6e0 2024-04-17T10:35:43.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.555+0000 7fdde0c00640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fddbc066f60 0x7fddbc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:35:43.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.555+0000 7fdde0c00640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fddbc066f60 0x7fddbc069420 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fddc80af000 tx=0x7fddc80a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:35:43.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.559+0000 7fddd9600640 1 -- 172.21.15.67:0/3117897721 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fddd0055710 con 0x7fdddc06e6e0 2024-04-17T10:35:43.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.755+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdddc074e30 con 0x7fdddc06e6e0 2024-04-17T10:35:43.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.755+0000 7fddd9600640 1 -- 172.21.15.67:0/3117897721 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v103) v1 ==== 53+0+73 (secure 0 0 0) 0x7fddd00597b0 con 0x7fdddc06e6e0 2024-04-17T10:35:43.746 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-17T10:35:43.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fddbc066f60 msgr2=0x7fddbc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fddbc066f60 0x7fddbc069420 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fddc80af000 tx=0x7fddc80a7000 comp rx=0 tx=0).stop 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06e6e0 msgr2=0x7fdddc0c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06e6e0 0x7fdddc0c8af0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fddd000c480 tx=0x7fddd000c940 comp rx=0 tx=0).stop 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 shutdown_connections 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fddbc066f60 0x7fddbc069420 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdddc0cf0c0 0x7fdddc140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdddc06eac0 0x7fdddc13bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 --2- 172.21.15.67:0/3117897721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdddc06e6e0 0x7fdddc0c8af0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 >> 172.21.15.67:0/3117897721 conn(0x7fdddc06ac90 msgr2=0x7fdddc0a8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 shutdown_connections 2024-04-17T10:35:43.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:35:43.759+0000 7fdde4000640 1 -- 172.21.15.67:0/3117897721 wait complete. 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-17T10:35:43.768 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:35:43.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:43.769 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-04-17T10:35:43.778 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-17T10:35:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-17T10:35:43.778 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-04-17T10:35:43.779 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41803sec preferred_lft 41803sec 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::ec4:7aff:feb6:fc0b/64 scope link 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 02:42:86:0e:84:6e brd ff:ff:ff:ff:ff:ff 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-17T10:35:43.780 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 5e:1f:84:6d:79:ae brd ff:ff:ff:ff:ff:ff 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::5c1f:84ff:fe6d:79ae/64 scope link 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0e:5c:f6:1f:82:2f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c5c:f6ff:fe1f:822f/64 scope link 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether c6:a6:3d:69:d0:db brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::c4a6:3dff:fe69:d0db/64 scope link 2024-04-17T10:35:43.781 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-04-17T10:35:43.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:43.782 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-17T10:35:43.844 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-04-17T10:35:43.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:43.846 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-04-17T10:35:43.905 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-04-17T10:35:43.905 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-17T10:35:43.905 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-17T10:35:43.905 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-17T10:35:43.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:43.906 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec bash -c ' 2024-04-17T10:35:43.906 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:43.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-17T10:35:43.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-17T10:35:43.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-17T10:35:43.907 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:43.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:43.996 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-04-17T10:35:44.005 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-17T10:35:44.005 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-17T10:35:44.006 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:35:44.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:44.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-04-17T10:35:44.070 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:35:44.070 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-17T10:35:44.122 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-17T10:35:44.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-17T10:35:44.123 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:35:44.178 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-17T10:35:44.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:44.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:35:44.267 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:35:44.267 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:35:44.280 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-17T10:35:44.280 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-17T10:35:44.280 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.86:3300,172.21.15.86:3301" 2024-04-17T10:35:44.280 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with new device syntax: 0@8e8d1faf-cb51-4015-918b-f894d5d28e99.cephfs=/ 2024-04-17T10:35:44.280 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.67:3300/172.21.15.86:3300/172.21.15.86:3301" will pass to kernel 2024-04-17T10:35:44.284 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-17T10:35:44.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:44.284 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-17T10:35:44.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:44.305 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-17T10:35:44.364 INFO:teuthology.orchestra.run.smithi067.stdout:client9205 2024-04-17T10:35:44.364 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:35:44.364 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:35:44.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:44.415 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:44.415 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:44.415 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.0 2024-04-17T10:35:44.415 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:44.415 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:44.415 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:44.415 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:44.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.498 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.498 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176242 s, 59.5 MB/s 2024-04-17T10:35:44.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169455 s, 61.9 MB/s 2024-04-17T10:35:44.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.532 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130835 s, 80.1 MB/s 2024-04-17T10:35:44.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201503 s, 52.0 MB/s 2024-04-17T10:35:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204134 s, 51.4 MB/s 2024-04-17T10:35:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.594 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165815 s, 63.2 MB/s 2024-04-17T10:35:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129925 s, 80.7 MB/s 2024-04-17T10:35:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.627 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155907 s, 67.3 MB/s 2024-04-17T10:35:44.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.644 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158176 s, 66.3 MB/s 2024-04-17T10:35:44.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.660 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141298 s, 74.2 MB/s 2024-04-17T10:35:44.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.676 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.676 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143625 s, 73.0 MB/s 2024-04-17T10:35:44.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.694 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.694 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015766 s, 66.5 MB/s 2024-04-17T10:35:44.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146386 s, 71.6 MB/s 2024-04-17T10:35:44.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.728 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164123 s, 63.9 MB/s 2024-04-17T10:35:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.746 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015733 s, 66.6 MB/s 2024-04-17T10:35:44.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.767 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188919 s, 55.5 MB/s 2024-04-17T10:35:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.785 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167087 s, 62.8 MB/s 2024-04-17T10:35:44.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.802 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015339 s, 68.4 MB/s 2024-04-17T10:35:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139274 s, 75.3 MB/s 2024-04-17T10:35:44.834 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148907 s, 70.4 MB/s 2024-04-17T10:35:44.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.852 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156246 s, 67.1 MB/s 2024-04-17T10:35:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.869 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149306 s, 70.2 MB/s 2024-04-17T10:35:44.884 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.884 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.884 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138161 s, 75.9 MB/s 2024-04-17T10:35:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.901 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144196 s, 72.7 MB/s 2024-04-17T10:35:44.918 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.918 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.918 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156203 s, 67.1 MB/s 2024-04-17T10:35:44.934 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.934 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.934 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143444 s, 73.1 MB/s 2024-04-17T10:35:44.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.950 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139656 s, 75.1 MB/s 2024-04-17T10:35:44.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.966 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.966 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137013 s, 76.5 MB/s 2024-04-17T10:35:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.982 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.983 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149176 s, 70.3 MB/s 2024-04-17T10:35:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:44.998 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138508 s, 75.7 MB/s 2024-04-17T10:35:45.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.016 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.016 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164221 s, 63.9 MB/s 2024-04-17T10:35:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167288 s, 62.7 MB/s 2024-04-17T10:35:45.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:45.036 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:45.036 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:45.037 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.1 2024-04-17T10:35:45.037 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:45.037 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:45.037 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:45.037 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.115 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159241 s, 65.8 MB/s 2024-04-17T10:35:45.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204215 s, 51.3 MB/s 2024-04-17T10:35:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187888 s, 55.8 MB/s 2024-04-17T10:35:45.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.172 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158509 s, 66.2 MB/s 2024-04-17T10:35:45.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.194 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183309 s, 57.2 MB/s 2024-04-17T10:35:45.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.210 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132071 s, 79.4 MB/s 2024-04-17T10:35:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147469 s, 71.1 MB/s 2024-04-17T10:35:45.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138556 s, 75.7 MB/s 2024-04-17T10:35:45.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.258 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.258 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155135 s, 67.6 MB/s 2024-04-17T10:35:45.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.274 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.274 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153781 s, 68.2 MB/s 2024-04-17T10:35:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.296 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205477 s, 51.0 MB/s 2024-04-17T10:35:45.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168701 s, 62.2 MB/s 2024-04-17T10:35:45.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.330 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128414 s, 81.7 MB/s 2024-04-17T10:35:45.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.345 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134867 s, 77.7 MB/s 2024-04-17T10:35:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.361 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147843 s, 70.9 MB/s 2024-04-17T10:35:45.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.383 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181673 s, 57.7 MB/s 2024-04-17T10:35:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.398 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153509 s, 68.3 MB/s 2024-04-17T10:35:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189694 s, 55.3 MB/s 2024-04-17T10:35:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139218 s, 75.3 MB/s 2024-04-17T10:35:45.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.452 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153909 s, 68.1 MB/s 2024-04-17T10:35:45.469 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.469 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.470 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152923 s, 68.6 MB/s 2024-04-17T10:35:45.485 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.486 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137512 s, 76.3 MB/s 2024-04-17T10:35:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.502 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147773 s, 71.0 MB/s 2024-04-17T10:35:45.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135924 s, 77.1 MB/s 2024-04-17T10:35:45.533 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.534 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149297 s, 70.2 MB/s 2024-04-17T10:35:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186118 s, 56.3 MB/s 2024-04-17T10:35:45.571 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.571 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.571 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154985 s, 67.7 MB/s 2024-04-17T10:35:45.587 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.588 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150063 s, 69.9 MB/s 2024-04-17T10:35:45.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191969 s, 54.6 MB/s 2024-04-17T10:35:45.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.624 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.624 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135298 s, 77.5 MB/s 2024-04-17T10:35:45.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.641 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147672 s, 71.0 MB/s 2024-04-17T10:35:45.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.657 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144246 s, 72.7 MB/s 2024-04-17T10:35:45.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:45.658 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:45.658 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:45.658 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.2 2024-04-17T10:35:45.658 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:45.658 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:45.658 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:45.658 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.735 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159857 s, 65.6 MB/s 2024-04-17T10:35:45.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.751 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.752 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147617 s, 71.0 MB/s 2024-04-17T10:35:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.768 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148611 s, 70.6 MB/s 2024-04-17T10:35:45.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.784 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141802 s, 73.9 MB/s 2024-04-17T10:35:45.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138425 s, 75.8 MB/s 2024-04-17T10:35:45.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135649 s, 77.3 MB/s 2024-04-17T10:35:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.833 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157098 s, 66.7 MB/s 2024-04-17T10:35:45.847 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.847 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.847 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131856 s, 79.5 MB/s 2024-04-17T10:35:45.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.863 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.864 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138931 s, 75.5 MB/s 2024-04-17T10:35:45.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145336 s, 72.1 MB/s 2024-04-17T10:35:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151949 s, 69.0 MB/s 2024-04-17T10:35:45.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.916 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.916 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018588 s, 56.4 MB/s 2024-04-17T10:35:45.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.932 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138868 s, 75.5 MB/s 2024-04-17T10:35:45.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.948 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.948 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137827 s, 76.1 MB/s 2024-04-17T10:35:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152555 s, 68.7 MB/s 2024-04-17T10:35:45.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.980 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139724 s, 75.0 MB/s 2024-04-17T10:35:45.996 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:45.997 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141658 s, 74.0 MB/s 2024-04-17T10:35:46.013 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.013 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.013 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149116 s, 70.3 MB/s 2024-04-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.029 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139757 s, 75.0 MB/s 2024-04-17T10:35:46.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.049 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180802 s, 58.0 MB/s 2024-04-17T10:35:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.068 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167285 s, 62.7 MB/s 2024-04-17T10:35:46.084 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.085 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150127 s, 69.8 MB/s 2024-04-17T10:35:46.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.100 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013487 s, 77.7 MB/s 2024-04-17T10:35:46.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.118 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137463 s, 76.3 MB/s 2024-04-17T10:35:46.131 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.131 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.131 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139236 s, 75.3 MB/s 2024-04-17T10:35:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.149 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.150 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165692 s, 63.3 MB/s 2024-04-17T10:35:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.169 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170845 s, 61.4 MB/s 2024-04-17T10:35:46.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160018 s, 65.5 MB/s 2024-04-17T10:35:46.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139026 s, 75.4 MB/s 2024-04-17T10:35:46.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.219 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014712 s, 71.3 MB/s 2024-04-17T10:35:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143823 s, 72.9 MB/s 2024-04-17T10:35:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167909 s, 62.4 MB/s 2024-04-17T10:35:46.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:46.255 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:46.255 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:46.255 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.3 2024-04-17T10:35:46.255 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:46.255 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:46.255 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:46.255 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.328 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150258 s, 69.8 MB/s 2024-04-17T10:35:46.345 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.345 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.346 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157736 s, 66.5 MB/s 2024-04-17T10:35:46.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.362 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.362 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152446 s, 68.8 MB/s 2024-04-17T10:35:46.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.379 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.379 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150179 s, 69.8 MB/s 2024-04-17T10:35:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.400 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186445 s, 56.2 MB/s 2024-04-17T10:35:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.417 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154516 s, 67.9 MB/s 2024-04-17T10:35:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.435 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158152 s, 66.3 MB/s 2024-04-17T10:35:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.452 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153007 s, 68.5 MB/s 2024-04-17T10:35:46.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.467 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136358 s, 76.9 MB/s 2024-04-17T10:35:46.483 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.484 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.484 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013601 s, 77.1 MB/s 2024-04-17T10:35:46.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.499 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.499 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149674 s, 70.1 MB/s 2024-04-17T10:35:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.516 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149336 s, 70.2 MB/s 2024-04-17T10:35:46.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.532 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.532 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142433 s, 73.6 MB/s 2024-04-17T10:35:46.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.547 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013395 s, 78.3 MB/s 2024-04-17T10:35:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137336 s, 76.4 MB/s 2024-04-17T10:35:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.578 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139068 s, 75.4 MB/s 2024-04-17T10:35:46.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.594 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139131 s, 75.4 MB/s 2024-04-17T10:35:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.610 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140782 s, 74.5 MB/s 2024-04-17T10:35:46.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.630 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01848 s, 56.7 MB/s 2024-04-17T10:35:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139227 s, 75.3 MB/s 2024-04-17T10:35:46.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.663 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153398 s, 68.4 MB/s 2024-04-17T10:35:46.679 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.680 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144497 s, 72.6 MB/s 2024-04-17T10:35:46.695 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.695 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.695 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139074 s, 75.4 MB/s 2024-04-17T10:35:46.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.710 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.710 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133995 s, 78.3 MB/s 2024-04-17T10:35:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.726 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139935 s, 74.9 MB/s 2024-04-17T10:35:46.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.747 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.747 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190855 s, 54.9 MB/s 2024-04-17T10:35:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.763 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141055 s, 74.3 MB/s 2024-04-17T10:35:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.779 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141102 s, 74.3 MB/s 2024-04-17T10:35:46.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195592 s, 53.6 MB/s 2024-04-17T10:35:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.816 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138498 s, 75.7 MB/s 2024-04-17T10:35:46.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.833 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147331 s, 71.2 MB/s 2024-04-17T10:35:46.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.848 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013988 s, 75.0 MB/s 2024-04-17T10:35:46.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:46.850 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:46.850 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:46.850 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.4 2024-04-17T10:35:46.850 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:46.850 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:46.850 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:46.850 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:46.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.918 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.918 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143314 s, 73.2 MB/s 2024-04-17T10:35:46.936 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.936 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.936 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162844 s, 64.4 MB/s 2024-04-17T10:35:46.951 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.952 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.952 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138568 s, 75.7 MB/s 2024-04-17T10:35:46.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.968 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148171 s, 70.8 MB/s 2024-04-17T10:35:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:46.985 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150261 s, 69.8 MB/s 2024-04-17T10:35:47.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.002 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152088 s, 68.9 MB/s 2024-04-17T10:35:47.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.017 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136302 s, 76.9 MB/s 2024-04-17T10:35:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142943 s, 73.4 MB/s 2024-04-17T10:35:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.049 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140597 s, 74.6 MB/s 2024-04-17T10:35:47.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.066 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147401 s, 71.1 MB/s 2024-04-17T10:35:47.083 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.083 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.083 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162143 s, 64.7 MB/s 2024-04-17T10:35:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.100 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151396 s, 69.3 MB/s 2024-04-17T10:35:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.116 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137156 s, 76.5 MB/s 2024-04-17T10:35:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.132 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014721 s, 71.2 MB/s 2024-04-17T10:35:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.150 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158251 s, 66.3 MB/s 2024-04-17T10:35:47.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.167 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152201 s, 68.9 MB/s 2024-04-17T10:35:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.182 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135005 s, 77.7 MB/s 2024-04-17T10:35:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.199 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151315 s, 69.3 MB/s 2024-04-17T10:35:47.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.217 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156841 s, 66.9 MB/s 2024-04-17T10:35:47.232 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140517 s, 74.6 MB/s 2024-04-17T10:35:47.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.252 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018099 s, 57.9 MB/s 2024-04-17T10:35:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.270 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159014 s, 65.9 MB/s 2024-04-17T10:35:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.290 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171392 s, 61.2 MB/s 2024-04-17T10:35:47.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137795 s, 76.1 MB/s 2024-04-17T10:35:47.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.321 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136928 s, 76.6 MB/s 2024-04-17T10:35:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.339 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154958 s, 67.7 MB/s 2024-04-17T10:35:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.360 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184635 s, 56.8 MB/s 2024-04-17T10:35:47.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.376 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.376 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155629 s, 67.4 MB/s 2024-04-17T10:35:47.393 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.394 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155333 s, 67.5 MB/s 2024-04-17T10:35:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175541 s, 59.7 MB/s 2024-04-17T10:35:47.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.428 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134412 s, 78.0 MB/s 2024-04-17T10:35:47.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.444 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013595 s, 77.1 MB/s 2024-04-17T10:35:47.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:47.445 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:47.445 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:47.445 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.5 2024-04-17T10:35:47.445 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:47.445 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:47.445 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:47.446 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:47.514 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.515 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.515 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150303 s, 69.8 MB/s 2024-04-17T10:35:47.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.531 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.531 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150807 s, 69.5 MB/s 2024-04-17T10:35:47.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.547 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134491 s, 78.0 MB/s 2024-04-17T10:35:47.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.562 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131023 s, 80.0 MB/s 2024-04-17T10:35:47.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.578 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.578 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144408 s, 72.6 MB/s 2024-04-17T10:35:47.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.594 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.594 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142397 s, 73.6 MB/s 2024-04-17T10:35:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.613 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.614 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180938 s, 58.0 MB/s 2024-04-17T10:35:47.630 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151627 s, 69.2 MB/s 2024-04-17T10:35:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.646 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01392 s, 75.3 MB/s 2024-04-17T10:35:47.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.663 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.663 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148372 s, 70.7 MB/s 2024-04-17T10:35:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.680 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157004 s, 66.8 MB/s 2024-04-17T10:35:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.696 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139253 s, 75.3 MB/s 2024-04-17T10:35:47.711 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.711 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.712 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137211 s, 76.4 MB/s 2024-04-17T10:35:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.728 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141747 s, 74.0 MB/s 2024-04-17T10:35:47.742 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.743 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133337 s, 78.6 MB/s 2024-04-17T10:35:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.760 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015597 s, 67.2 MB/s 2024-04-17T10:35:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.777 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153036 s, 68.5 MB/s 2024-04-17T10:35:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.792 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133922 s, 78.3 MB/s 2024-04-17T10:35:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146433 s, 71.6 MB/s 2024-04-17T10:35:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.825 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146843 s, 71.4 MB/s 2024-04-17T10:35:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.840 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013232 s, 79.2 MB/s 2024-04-17T10:35:47.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.858 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015757 s, 66.5 MB/s 2024-04-17T10:35:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.878 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187445 s, 55.9 MB/s 2024-04-17T10:35:47.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.896 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.896 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154151 s, 68.0 MB/s 2024-04-17T10:35:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.912 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014808 s, 70.8 MB/s 2024-04-17T10:35:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141054 s, 74.3 MB/s 2024-04-17T10:35:47.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.943 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140051 s, 74.9 MB/s 2024-04-17T10:35:47.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.960 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148798 s, 70.5 MB/s 2024-04-17T10:35:47.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147267 s, 71.2 MB/s 2024-04-17T10:35:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:47.992 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134517 s, 78.0 MB/s 2024-04-17T10:35:48.008 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.008 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.008 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146074 s, 71.8 MB/s 2024-04-17T10:35:48.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.024 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142311 s, 73.7 MB/s 2024-04-17T10:35:48.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:48.026 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:48.026 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:48.026 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.6 2024-04-17T10:35:48.026 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:48.026 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:48.026 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:48.026 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:48.097 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.097 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.097 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133352 s, 78.6 MB/s 2024-04-17T10:35:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.113 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014542 s, 72.1 MB/s 2024-04-17T10:35:48.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.130 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.130 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147878 s, 70.9 MB/s 2024-04-17T10:35:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.147 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155362 s, 67.5 MB/s 2024-04-17T10:35:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136982 s, 76.5 MB/s 2024-04-17T10:35:48.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148187 s, 70.8 MB/s 2024-04-17T10:35:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137672 s, 76.2 MB/s 2024-04-17T10:35:48.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.209 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.209 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121464 s, 86.3 MB/s 2024-04-17T10:35:48.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134828 s, 77.8 MB/s 2024-04-17T10:35:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138797 s, 75.5 MB/s 2024-04-17T10:35:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.257 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149345 s, 70.2 MB/s 2024-04-17T10:35:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.273 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141004 s, 74.4 MB/s 2024-04-17T10:35:48.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.290 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.290 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149707 s, 70.0 MB/s 2024-04-17T10:35:48.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.306 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.306 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146914 s, 71.4 MB/s 2024-04-17T10:35:48.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.322 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.322 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140439 s, 74.7 MB/s 2024-04-17T10:35:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.339 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150477 s, 69.7 MB/s 2024-04-17T10:35:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.355 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014318 s, 73.2 MB/s 2024-04-17T10:35:48.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138869 s, 75.5 MB/s 2024-04-17T10:35:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.388 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151762 s, 69.1 MB/s 2024-04-17T10:35:48.405 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.405 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.405 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155025 s, 67.6 MB/s 2024-04-17T10:35:48.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.422 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149017 s, 70.4 MB/s 2024-04-17T10:35:48.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.438 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149625 s, 70.1 MB/s 2024-04-17T10:35:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.455 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150472 s, 69.7 MB/s 2024-04-17T10:35:48.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144053 s, 72.8 MB/s 2024-04-17T10:35:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.488 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145271 s, 72.2 MB/s 2024-04-17T10:35:48.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.504 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.504 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147287 s, 71.2 MB/s 2024-04-17T10:35:48.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.520 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139697 s, 75.1 MB/s 2024-04-17T10:35:48.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.536 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.536 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140088 s, 74.9 MB/s 2024-04-17T10:35:48.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.553 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149458 s, 70.2 MB/s 2024-04-17T10:35:48.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.570 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149658 s, 70.1 MB/s 2024-04-17T10:35:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.585 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136424 s, 76.9 MB/s 2024-04-17T10:35:48.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.601 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013923 s, 75.3 MB/s 2024-04-17T10:35:48.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:48.602 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:48.602 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:48.602 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.7 2024-04-17T10:35:48.602 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:48.602 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:48.603 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:48.603 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:48.674 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.675 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.675 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149398 s, 70.2 MB/s 2024-04-17T10:35:48.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132311 s, 79.3 MB/s 2024-04-17T10:35:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.707 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155214 s, 67.6 MB/s 2024-04-17T10:35:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.722 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132674 s, 79.0 MB/s 2024-04-17T10:35:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.738 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149537 s, 70.1 MB/s 2024-04-17T10:35:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.755 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145577 s, 72.0 MB/s 2024-04-17T10:35:48.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.771 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.772 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148557 s, 70.6 MB/s 2024-04-17T10:35:48.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.790 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167949 s, 62.4 MB/s 2024-04-17T10:35:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.807 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014847 s, 70.6 MB/s 2024-04-17T10:35:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.822 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141489 s, 74.1 MB/s 2024-04-17T10:35:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.839 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142332 s, 73.7 MB/s 2024-04-17T10:35:48.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140663 s, 74.5 MB/s 2024-04-17T10:35:48.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.869 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.870 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134548 s, 77.9 MB/s 2024-04-17T10:35:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.886 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145777 s, 71.9 MB/s 2024-04-17T10:35:48.902 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.902 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.902 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142033 s, 73.8 MB/s 2024-04-17T10:35:48.918 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.918 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.918 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142315 s, 73.7 MB/s 2024-04-17T10:35:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.935 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151044 s, 69.4 MB/s 2024-04-17T10:35:48.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.954 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.955 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180459 s, 58.1 MB/s 2024-04-17T10:35:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014125 s, 74.2 MB/s 2024-04-17T10:35:48.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:48.986 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:48.986 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138608 s, 75.7 MB/s 2024-04-17T10:35:49.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176336 s, 59.5 MB/s 2024-04-17T10:35:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.021 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138469 s, 75.7 MB/s 2024-04-17T10:35:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.038 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145765 s, 71.9 MB/s 2024-04-17T10:35:49.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.054 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.054 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144098 s, 72.8 MB/s 2024-04-17T10:35:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.067 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113665 s, 92.3 MB/s 2024-04-17T10:35:49.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133705 s, 78.4 MB/s 2024-04-17T10:35:49.097 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.098 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135671 s, 77.3 MB/s 2024-04-17T10:35:49.117 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.118 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181668 s, 57.7 MB/s 2024-04-17T10:35:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178159 s, 58.9 MB/s 2024-04-17T10:35:49.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146942 s, 71.4 MB/s 2024-04-17T10:35:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.169 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.170 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144201 s, 72.7 MB/s 2024-04-17T10:35:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.186 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014372 s, 73.0 MB/s 2024-04-17T10:35:49.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:49.188 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:49.188 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:49.188 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.8 2024-04-17T10:35:49.188 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:49.188 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:49.188 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:49.188 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:49.259 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.260 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.260 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154593 s, 67.8 MB/s 2024-04-17T10:35:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.277 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154626 s, 67.8 MB/s 2024-04-17T10:35:49.292 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142035 s, 73.8 MB/s 2024-04-17T10:35:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.313 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183556 s, 57.1 MB/s 2024-04-17T10:35:49.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.328 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.328 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132665 s, 79.0 MB/s 2024-04-17T10:35:49.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.343 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134004 s, 78.2 MB/s 2024-04-17T10:35:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.359 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144571 s, 72.5 MB/s 2024-04-17T10:35:49.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.375 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.376 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144559 s, 72.5 MB/s 2024-04-17T10:35:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.391 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135543 s, 77.4 MB/s 2024-04-17T10:35:49.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.406 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.407 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013805 s, 76.0 MB/s 2024-04-17T10:35:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131046 s, 80.0 MB/s 2024-04-17T10:35:49.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.438 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144602 s, 72.5 MB/s 2024-04-17T10:35:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.454 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142565 s, 73.6 MB/s 2024-04-17T10:35:49.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.471 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150131 s, 69.8 MB/s 2024-04-17T10:35:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.486 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139513 s, 75.2 MB/s 2024-04-17T10:35:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.501 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013585 s, 77.2 MB/s 2024-04-17T10:35:49.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141849 s, 73.9 MB/s 2024-04-17T10:35:49.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.534 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149254 s, 70.3 MB/s 2024-04-17T10:35:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.551 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151314 s, 69.3 MB/s 2024-04-17T10:35:49.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.568 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.568 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156021 s, 67.2 MB/s 2024-04-17T10:35:49.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.584 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.584 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144455 s, 72.6 MB/s 2024-04-17T10:35:49.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.600 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014053 s, 74.6 MB/s 2024-04-17T10:35:49.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.618 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156485 s, 67.0 MB/s 2024-04-17T10:35:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.633 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142935 s, 73.4 MB/s 2024-04-17T10:35:49.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.651 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152149 s, 68.9 MB/s 2024-04-17T10:35:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.667 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150715 s, 69.6 MB/s 2024-04-17T10:35:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.684 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156305 s, 67.1 MB/s 2024-04-17T10:35:49.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.701 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145894 s, 71.9 MB/s 2024-04-17T10:35:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.717 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147615 s, 71.0 MB/s 2024-04-17T10:35:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.734 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150681 s, 69.6 MB/s 2024-04-17T10:35:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.750 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133297 s, 78.7 MB/s 2024-04-17T10:35:49.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.766 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147044 s, 71.3 MB/s 2024-04-17T10:35:49.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:49.767 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:49.767 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:49.767 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.9 2024-04-17T10:35:49.768 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:49.768 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:49.768 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:49.768 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.839 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152501 s, 68.8 MB/s 2024-04-17T10:35:49.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.855 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143678 s, 73.0 MB/s 2024-04-17T10:35:49.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.871 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.871 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140829 s, 74.5 MB/s 2024-04-17T10:35:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143798 s, 72.9 MB/s 2024-04-17T10:35:49.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.904 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147082 s, 71.3 MB/s 2024-04-17T10:35:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.920 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144131 s, 72.8 MB/s 2024-04-17T10:35:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.935 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135363 s, 77.5 MB/s 2024-04-17T10:35:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138086 s, 75.9 MB/s 2024-04-17T10:35:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.968 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150001 s, 69.9 MB/s 2024-04-17T10:35:49.984 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.984 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.985 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149285 s, 70.2 MB/s 2024-04-17T10:35:49.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:49.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:49.999 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133609 s, 78.5 MB/s 2024-04-17T10:35:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.017 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159907 s, 65.6 MB/s 2024-04-17T10:35:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.033 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141848 s, 73.9 MB/s 2024-04-17T10:35:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.048 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130314 s, 80.5 MB/s 2024-04-17T10:35:50.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.065 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.065 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153045 s, 68.5 MB/s 2024-04-17T10:35:50.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014916 s, 70.3 MB/s 2024-04-17T10:35:50.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.099 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153669 s, 68.2 MB/s 2024-04-17T10:35:50.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.121 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198908 s, 52.7 MB/s 2024-04-17T10:35:50.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.136 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142075 s, 73.8 MB/s 2024-04-17T10:35:50.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.153 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.153 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150361 s, 69.7 MB/s 2024-04-17T10:35:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.172 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162999 s, 64.3 MB/s 2024-04-17T10:35:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.187 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137191 s, 76.4 MB/s 2024-04-17T10:35:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143914 s, 72.9 MB/s 2024-04-17T10:35:50.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134654 s, 77.9 MB/s 2024-04-17T10:35:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.234 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136031 s, 77.1 MB/s 2024-04-17T10:35:50.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.250 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.250 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139175 s, 75.3 MB/s 2024-04-17T10:35:50.265 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.265 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.265 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134727 s, 77.8 MB/s 2024-04-17T10:35:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139808 s, 75.0 MB/s 2024-04-17T10:35:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143971 s, 72.8 MB/s 2024-04-17T10:35:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165627 s, 63.3 MB/s 2024-04-17T10:35:50.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.332 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149751 s, 70.0 MB/s 2024-04-17T10:35:50.349 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.349 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.349 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151456 s, 69.2 MB/s 2024-04-17T10:35:50.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:50.351 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:50.351 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:50.351 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.10 2024-04-17T10:35:50.351 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:50.351 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:50.351 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:50.351 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:50.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.421 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.421 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140807 s, 74.5 MB/s 2024-04-17T10:35:50.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142782 s, 73.4 MB/s 2024-04-17T10:35:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.456 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167485 s, 62.6 MB/s 2024-04-17T10:35:50.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140058 s, 74.9 MB/s 2024-04-17T10:35:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.491 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177523 s, 59.1 MB/s 2024-04-17T10:35:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.507 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013384 s, 78.3 MB/s 2024-04-17T10:35:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.523 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.523 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140017 s, 74.9 MB/s 2024-04-17T10:35:50.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.541 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162642 s, 64.5 MB/s 2024-04-17T10:35:50.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.557 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144247 s, 72.7 MB/s 2024-04-17T10:35:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.574 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147374 s, 71.2 MB/s 2024-04-17T10:35:50.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.590 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.590 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144758 s, 72.4 MB/s 2024-04-17T10:35:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.606 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141603 s, 74.1 MB/s 2024-04-17T10:35:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.622 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149165 s, 70.3 MB/s 2024-04-17T10:35:50.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178117 s, 58.9 MB/s 2024-04-17T10:35:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.658 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146734 s, 71.5 MB/s 2024-04-17T10:35:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133971 s, 78.3 MB/s 2024-04-17T10:35:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147902 s, 70.9 MB/s 2024-04-17T10:35:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.706 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140877 s, 74.4 MB/s 2024-04-17T10:35:50.722 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.723 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.723 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149134 s, 70.3 MB/s 2024-04-17T10:35:50.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.740 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.740 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154181 s, 68.0 MB/s 2024-04-17T10:35:50.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.756 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144482 s, 72.6 MB/s 2024-04-17T10:35:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.773 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154207 s, 68.0 MB/s 2024-04-17T10:35:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.788 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131742 s, 79.6 MB/s 2024-04-17T10:35:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142889 s, 73.4 MB/s 2024-04-17T10:35:50.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.821 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153986 s, 68.1 MB/s 2024-04-17T10:35:50.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.838 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143353 s, 73.1 MB/s 2024-04-17T10:35:50.853 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.854 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.854 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014313 s, 73.3 MB/s 2024-04-17T10:35:50.871 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.871 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.871 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154947 s, 67.7 MB/s 2024-04-17T10:35:50.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.888 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152345 s, 68.8 MB/s 2024-04-17T10:35:50.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.904 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141818 s, 73.9 MB/s 2024-04-17T10:35:50.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014855 s, 70.6 MB/s 2024-04-17T10:35:50.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:50.938 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:50.938 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155816 s, 67.3 MB/s 2024-04-17T10:35:50.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:50.939 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:50.939 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:50.940 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.11 2024-04-17T10:35:50.940 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:50.940 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:50.940 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:50.940 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:51.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.010 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138884 s, 75.5 MB/s 2024-04-17T10:35:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.026 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142705 s, 73.5 MB/s 2024-04-17T10:35:51.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.042 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139596 s, 75.1 MB/s 2024-04-17T10:35:51.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.057 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.057 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142205 s, 73.7 MB/s 2024-04-17T10:35:51.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019471 s, 53.9 MB/s 2024-04-17T10:35:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145444 s, 72.1 MB/s 2024-04-17T10:35:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.112 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155833 s, 67.3 MB/s 2024-04-17T10:35:51.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.129 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.129 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148332 s, 70.7 MB/s 2024-04-17T10:35:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.145 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149582 s, 70.1 MB/s 2024-04-17T10:35:51.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.162 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.162 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153333 s, 68.4 MB/s 2024-04-17T10:35:51.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.179 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.179 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151516 s, 69.2 MB/s 2024-04-17T10:35:51.194 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.195 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.195 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135482 s, 77.4 MB/s 2024-04-17T10:35:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.210 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137009 s, 76.5 MB/s 2024-04-17T10:35:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.228 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161943 s, 64.7 MB/s 2024-04-17T10:35:51.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.245 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155608 s, 67.4 MB/s 2024-04-17T10:35:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.263 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158306 s, 66.2 MB/s 2024-04-17T10:35:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141784 s, 74.0 MB/s 2024-04-17T10:35:51.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.294 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.294 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134398 s, 78.0 MB/s 2024-04-17T10:35:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148559 s, 70.6 MB/s 2024-04-17T10:35:51.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.327 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.327 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141327 s, 74.2 MB/s 2024-04-17T10:35:51.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.344 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.344 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149709 s, 70.0 MB/s 2024-04-17T10:35:51.359 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.360 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.360 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136594 s, 76.8 MB/s 2024-04-17T10:35:51.377 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.377 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.377 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153519 s, 68.3 MB/s 2024-04-17T10:35:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134313 s, 78.1 MB/s 2024-04-17T10:35:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.414 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193209 s, 54.3 MB/s 2024-04-17T10:35:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.428 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.429 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138649 s, 75.6 MB/s 2024-04-17T10:35:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156388 s, 67.0 MB/s 2024-04-17T10:35:51.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.462 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.462 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141065 s, 74.3 MB/s 2024-04-17T10:35:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.479 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152834 s, 68.6 MB/s 2024-04-17T10:35:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146675 s, 71.5 MB/s 2024-04-17T10:35:51.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125785 s, 83.4 MB/s 2024-04-17T10:35:51.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.525 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.525 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137675 s, 76.2 MB/s 2024-04-17T10:35:51.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:51.527 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:51.527 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:51.527 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.12 2024-04-17T10:35:51.527 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:51.527 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:51.528 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:51.528 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:51.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.599 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151206 s, 69.3 MB/s 2024-04-17T10:35:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.615 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141784 s, 74.0 MB/s 2024-04-17T10:35:51.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148619 s, 70.6 MB/s 2024-04-17T10:35:51.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.650 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.650 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146748 s, 71.5 MB/s 2024-04-17T10:35:51.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.664 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147017 s, 71.3 MB/s 2024-04-17T10:35:51.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.684 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.684 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182192 s, 57.6 MB/s 2024-04-17T10:35:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.700 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143792 s, 72.9 MB/s 2024-04-17T10:35:51.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.718 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154462 s, 67.9 MB/s 2024-04-17T10:35:51.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.733 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138334 s, 75.8 MB/s 2024-04-17T10:35:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137895 s, 76.0 MB/s 2024-04-17T10:35:51.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.766 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149397 s, 70.2 MB/s 2024-04-17T10:35:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.780 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.781 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134352 s, 78.0 MB/s 2024-04-17T10:35:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190226 s, 55.1 MB/s 2024-04-17T10:35:51.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146069 s, 71.8 MB/s 2024-04-17T10:35:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.833 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137642 s, 76.2 MB/s 2024-04-17T10:35:51.848 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.849 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013619 s, 77.0 MB/s 2024-04-17T10:35:51.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.865 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145635 s, 72.0 MB/s 2024-04-17T10:35:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132376 s, 79.2 MB/s 2024-04-17T10:35:51.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.895 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131426 s, 79.8 MB/s 2024-04-17T10:35:51.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013998 s, 74.9 MB/s 2024-04-17T10:35:51.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.930 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176525 s, 59.4 MB/s 2024-04-17T10:35:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.947 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150457 s, 69.7 MB/s 2024-04-17T10:35:51.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.964 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.964 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154939 s, 67.7 MB/s 2024-04-17T10:35:51.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.979 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.979 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134308 s, 78.1 MB/s 2024-04-17T10:35:51.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:51.994 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:51.995 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132438 s, 79.2 MB/s 2024-04-17T10:35:52.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.011 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014137 s, 74.2 MB/s 2024-04-17T10:35:52.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.028 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.028 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155828 s, 67.3 MB/s 2024-04-17T10:35:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01453 s, 72.2 MB/s 2024-04-17T10:35:52.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.062 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.062 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160322 s, 65.4 MB/s 2024-04-17T10:35:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014868 s, 70.5 MB/s 2024-04-17T10:35:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015562 s, 67.4 MB/s 2024-04-17T10:35:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.113 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150358 s, 69.7 MB/s 2024-04-17T10:35:52.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:52.115 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:52.115 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:52.115 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.13 2024-04-17T10:35:52.115 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:52.115 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:52.115 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:52.115 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:52.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.185 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134743 s, 77.8 MB/s 2024-04-17T10:35:52.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.202 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151313 s, 69.3 MB/s 2024-04-17T10:35:52.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.218 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.218 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142848 s, 73.4 MB/s 2024-04-17T10:35:52.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136828 s, 76.6 MB/s 2024-04-17T10:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140001 s, 74.9 MB/s 2024-04-17T10:35:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169115 s, 62.0 MB/s 2024-04-17T10:35:52.284 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.284 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.284 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139635 s, 75.1 MB/s 2024-04-17T10:35:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.307 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202323 s, 51.8 MB/s 2024-04-17T10:35:52.325 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.325 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.325 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156016 s, 67.2 MB/s 2024-04-17T10:35:52.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.343 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.344 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159448 s, 65.8 MB/s 2024-04-17T10:35:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.358 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128934 s, 81.3 MB/s 2024-04-17T10:35:52.374 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.374 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.374 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137626 s, 76.2 MB/s 2024-04-17T10:35:52.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.394 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.394 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188021 s, 55.8 MB/s 2024-04-17T10:35:52.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.409 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.409 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123974 s, 84.6 MB/s 2024-04-17T10:35:52.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.423 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.423 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131049 s, 80.0 MB/s 2024-04-17T10:35:52.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.439 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.439 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136933 s, 76.6 MB/s 2024-04-17T10:35:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.454 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132849 s, 78.9 MB/s 2024-04-17T10:35:52.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.470 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.470 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143854 s, 72.9 MB/s 2024-04-17T10:35:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.487 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143804 s, 72.9 MB/s 2024-04-17T10:35:52.501 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.502 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134458 s, 78.0 MB/s 2024-04-17T10:35:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.520 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169367 s, 61.9 MB/s 2024-04-17T10:35:52.535 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.535 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.535 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129519 s, 81.0 MB/s 2024-04-17T10:35:52.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.555 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181151 s, 57.9 MB/s 2024-04-17T10:35:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.571 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139347 s, 75.2 MB/s 2024-04-17T10:35:52.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.586 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.586 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133534 s, 78.5 MB/s 2024-04-17T10:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126247 s, 83.1 MB/s 2024-04-17T10:35:52.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.617 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.617 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142241 s, 73.7 MB/s 2024-04-17T10:35:52.633 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.634 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152472 s, 68.8 MB/s 2024-04-17T10:35:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.650 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139666 s, 75.1 MB/s 2024-04-17T10:35:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.665 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141248 s, 74.2 MB/s 2024-04-17T10:35:52.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.686 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.686 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183959 s, 57.0 MB/s 2024-04-17T10:35:52.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.701 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139155 s, 75.4 MB/s 2024-04-17T10:35:52.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:52.703 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:52.703 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:52.703 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.14 2024-04-17T10:35:52.703 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:52.703 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:52.703 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:52.703 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191385 s, 54.8 MB/s 2024-04-17T10:35:52.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157166 s, 66.7 MB/s 2024-04-17T10:35:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.821 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018468 s, 56.8 MB/s 2024-04-17T10:35:52.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.837 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146069 s, 71.8 MB/s 2024-04-17T10:35:52.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.852 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130166 s, 80.6 MB/s 2024-04-17T10:35:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148364 s, 70.7 MB/s 2024-04-17T10:35:52.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.883 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.884 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135289 s, 77.5 MB/s 2024-04-17T10:35:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.900 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014404 s, 72.8 MB/s 2024-04-17T10:35:52.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.920 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185412 s, 56.6 MB/s 2024-04-17T10:35:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.940 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180348 s, 58.1 MB/s 2024-04-17T10:35:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.957 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154302 s, 68.0 MB/s 2024-04-17T10:35:52.973 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.974 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138435 s, 75.7 MB/s 2024-04-17T10:35:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143478 s, 73.1 MB/s 2024-04-17T10:35:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148315 s, 70.7 MB/s 2024-04-17T10:35:53.023 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.023 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.023 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153417 s, 68.3 MB/s 2024-04-17T10:35:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.040 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151407 s, 69.3 MB/s 2024-04-17T10:35:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.055 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135573 s, 77.3 MB/s 2024-04-17T10:35:53.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.071 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.071 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144039 s, 72.8 MB/s 2024-04-17T10:35:53.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156262 s, 67.1 MB/s 2024-04-17T10:35:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.107 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162475 s, 64.5 MB/s 2024-04-17T10:35:53.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137519 s, 76.2 MB/s 2024-04-17T10:35:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.139 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147973 s, 70.9 MB/s 2024-04-17T10:35:53.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142951 s, 73.4 MB/s 2024-04-17T10:35:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.170 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133291 s, 78.7 MB/s 2024-04-17T10:35:53.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.188 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.188 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016297 s, 64.3 MB/s 2024-04-17T10:35:53.204 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.205 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144891 s, 72.4 MB/s 2024-04-17T10:35:53.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.222 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.222 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147639 s, 71.0 MB/s 2024-04-17T10:35:53.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164978 s, 63.6 MB/s 2024-04-17T10:35:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.256 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148787 s, 70.5 MB/s 2024-04-17T10:35:53.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.273 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154594 s, 67.8 MB/s 2024-04-17T10:35:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.289 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139878 s, 75.0 MB/s 2024-04-17T10:35:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.305 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138118 s, 75.9 MB/s 2024-04-17T10:35:53.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:53.306 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:53.307 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:53.307 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.15 2024-04-17T10:35:53.307 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:53.307 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:53.307 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:53.307 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:53.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.378 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.378 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014553 s, 72.1 MB/s 2024-04-17T10:35:53.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.395 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.395 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152555 s, 68.7 MB/s 2024-04-17T10:35:53.411 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.411 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.411 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145202 s, 72.2 MB/s 2024-04-17T10:35:53.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.427 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.427 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142907 s, 73.4 MB/s 2024-04-17T10:35:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.443 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142322 s, 73.7 MB/s 2024-04-17T10:35:53.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.463 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.464 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183036 s, 57.3 MB/s 2024-04-17T10:35:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.480 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.481 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152022 s, 69.0 MB/s 2024-04-17T10:35:53.495 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.496 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.496 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134351 s, 78.0 MB/s 2024-04-17T10:35:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.511 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133608 s, 78.5 MB/s 2024-04-17T10:35:53.528 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.529 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.529 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159332 s, 65.8 MB/s 2024-04-17T10:35:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.544 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138459 s, 75.7 MB/s 2024-04-17T10:35:53.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.560 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.561 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138916 s, 75.5 MB/s 2024-04-17T10:35:53.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.577 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.577 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150492 s, 69.7 MB/s 2024-04-17T10:35:53.592 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.593 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.593 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137272 s, 76.4 MB/s 2024-04-17T10:35:53.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.609 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.609 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141845 s, 73.9 MB/s 2024-04-17T10:35:53.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.626 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.626 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155648 s, 67.4 MB/s 2024-04-17T10:35:53.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.642 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.642 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146341 s, 71.7 MB/s 2024-04-17T10:35:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.658 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135092 s, 77.6 MB/s 2024-04-17T10:35:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.673 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134729 s, 77.8 MB/s 2024-04-17T10:35:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.690 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150605 s, 69.6 MB/s 2024-04-17T10:35:53.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.705 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138116 s, 75.9 MB/s 2024-04-17T10:35:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.721 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136539 s, 76.8 MB/s 2024-04-17T10:35:53.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.737 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.737 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145944 s, 71.8 MB/s 2024-04-17T10:35:53.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.753 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.753 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137018 s, 76.5 MB/s 2024-04-17T10:35:53.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.769 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146914 s, 71.4 MB/s 2024-04-17T10:35:53.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.786 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.786 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151913 s, 69.0 MB/s 2024-04-17T10:35:53.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.802 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140725 s, 74.5 MB/s 2024-04-17T10:35:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014284 s, 73.4 MB/s 2024-04-17T10:35:53.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151763 s, 69.1 MB/s 2024-04-17T10:35:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134415 s, 78.0 MB/s 2024-04-17T10:35:53.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.866 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.866 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139996 s, 74.9 MB/s 2024-04-17T10:35:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144733 s, 72.4 MB/s 2024-04-17T10:35:53.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:53.884 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:53.884 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:53.884 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.16 2024-04-17T10:35:53.884 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:53.884 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:53.884 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:53.885 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.953 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137916 s, 76.0 MB/s 2024-04-17T10:35:53.970 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.971 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155503 s, 67.4 MB/s 2024-04-17T10:35:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:53.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014455 s, 72.5 MB/s 2024-04-17T10:35:54.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.003 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.003 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145867 s, 71.9 MB/s 2024-04-17T10:35:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.019 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143483 s, 73.1 MB/s 2024-04-17T10:35:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140196 s, 74.8 MB/s 2024-04-17T10:35:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.053 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165225 s, 63.5 MB/s 2024-04-17T10:35:54.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.069 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142149 s, 73.8 MB/s 2024-04-17T10:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.085 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.086 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143781 s, 72.9 MB/s 2024-04-17T10:35:54.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.103 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.103 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153289 s, 68.4 MB/s 2024-04-17T10:35:54.119 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.119 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.119 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146254 s, 71.7 MB/s 2024-04-17T10:35:54.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.135 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.135 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145823 s, 71.9 MB/s 2024-04-17T10:35:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140482 s, 74.6 MB/s 2024-04-17T10:35:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.168 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148514 s, 70.6 MB/s 2024-04-17T10:35:54.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.185 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.185 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156599 s, 67.0 MB/s 2024-04-17T10:35:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.202 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146633 s, 71.5 MB/s 2024-04-17T10:35:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135151 s, 77.6 MB/s 2024-04-17T10:35:54.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.233 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.233 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148585 s, 70.6 MB/s 2024-04-17T10:35:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.249 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.250 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145322 s, 72.2 MB/s 2024-04-17T10:35:54.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.266 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.266 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151244 s, 69.3 MB/s 2024-04-17T10:35:54.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.282 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.282 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139898 s, 75.0 MB/s 2024-04-17T10:35:54.298 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.298 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.298 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139648 s, 75.1 MB/s 2024-04-17T10:35:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153501 s, 68.3 MB/s 2024-04-17T10:35:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136072 s, 77.1 MB/s 2024-04-17T10:35:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.347 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.348 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154596 s, 67.8 MB/s 2024-04-17T10:35:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.365 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151707 s, 69.1 MB/s 2024-04-17T10:35:54.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143491 s, 73.1 MB/s 2024-04-17T10:35:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013879 s, 75.6 MB/s 2024-04-17T10:35:54.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.412 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.412 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138399 s, 75.8 MB/s 2024-04-17T10:35:54.432 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.432 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.432 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018708 s, 56.0 MB/s 2024-04-17T10:35:54.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.448 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.448 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139187 s, 75.3 MB/s 2024-04-17T10:35:54.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.468 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.468 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180586 s, 58.1 MB/s 2024-04-17T10:35:54.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:54.469 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:54.469 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:54.469 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.17 2024-04-17T10:35:54.470 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:54.470 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:54.470 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:54.470 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145986 s, 71.8 MB/s 2024-04-17T10:35:54.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.555 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143209 s, 73.2 MB/s 2024-04-17T10:35:54.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.570 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.570 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135898 s, 77.2 MB/s 2024-04-17T10:35:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.590 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.591 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018562 s, 56.5 MB/s 2024-04-17T10:35:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151746 s, 69.1 MB/s 2024-04-17T10:35:54.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.623 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142927 s, 73.4 MB/s 2024-04-17T10:35:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.639 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139554 s, 75.1 MB/s 2024-04-17T10:35:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.656 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148977 s, 70.4 MB/s 2024-04-17T10:35:54.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014109 s, 74.3 MB/s 2024-04-17T10:35:54.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.689 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.689 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151872 s, 69.0 MB/s 2024-04-17T10:35:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.705 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148582 s, 70.6 MB/s 2024-04-17T10:35:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161078 s, 65.1 MB/s 2024-04-17T10:35:54.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.739 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140098 s, 74.8 MB/s 2024-04-17T10:35:54.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.759 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.759 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186245 s, 56.3 MB/s 2024-04-17T10:35:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.777 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152904 s, 68.6 MB/s 2024-04-17T10:35:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150173 s, 69.8 MB/s 2024-04-17T10:35:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135674 s, 77.3 MB/s 2024-04-17T10:35:54.825 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151828 s, 69.1 MB/s 2024-04-17T10:35:54.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.840 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.841 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132131 s, 79.4 MB/s 2024-04-17T10:35:54.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.856 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132392 s, 79.2 MB/s 2024-04-17T10:35:54.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.872 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014887 s, 70.4 MB/s 2024-04-17T10:35:54.887 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.888 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134641 s, 77.9 MB/s 2024-04-17T10:35:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136835 s, 76.6 MB/s 2024-04-17T10:35:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151984 s, 69.0 MB/s 2024-04-17T10:35:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.935 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013053 s, 80.3 MB/s 2024-04-17T10:35:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013854 s, 75.7 MB/s 2024-04-17T10:35:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.968 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155321 s, 67.5 MB/s 2024-04-17T10:35:54.983 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.983 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.983 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128601 s, 81.5 MB/s 2024-04-17T10:35:54.998 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141314 s, 74.2 MB/s 2024-04-17T10:35:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014623 s, 71.7 MB/s 2024-04-17T10:35:55.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.032 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154442 s, 67.9 MB/s 2024-04-17T10:35:55.049 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.050 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015532 s, 67.5 MB/s 2024-04-17T10:35:55.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:55.051 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:55.051 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:55.051 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.18 2024-04-17T10:35:55.051 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:55.051 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:55.051 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:55.051 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.123 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.123 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149761 s, 70.0 MB/s 2024-04-17T10:35:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130062 s, 80.6 MB/s 2024-04-17T10:35:55.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.155 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.155 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161337 s, 65.0 MB/s 2024-04-17T10:35:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138693 s, 75.6 MB/s 2024-04-17T10:35:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.187 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014368 s, 73.0 MB/s 2024-04-17T10:35:55.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141011 s, 74.4 MB/s 2024-04-17T10:35:55.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.219 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.219 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139286 s, 75.3 MB/s 2024-04-17T10:35:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.235 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145664 s, 72.0 MB/s 2024-04-17T10:35:55.251 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.251 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.251 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143152 s, 73.2 MB/s 2024-04-17T10:35:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138926 s, 75.5 MB/s 2024-04-17T10:35:55.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.283 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.283 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147621 s, 71.0 MB/s 2024-04-17T10:35:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138618 s, 75.6 MB/s 2024-04-17T10:35:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136573 s, 76.8 MB/s 2024-04-17T10:35:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141698 s, 74.0 MB/s 2024-04-17T10:35:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175558 s, 59.7 MB/s 2024-04-17T10:35:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191081 s, 54.9 MB/s 2024-04-17T10:35:55.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.387 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142618 s, 73.5 MB/s 2024-04-17T10:35:55.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.408 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.408 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191863 s, 54.7 MB/s 2024-04-17T10:35:55.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.422 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122217 s, 85.8 MB/s 2024-04-17T10:35:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.438 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145653 s, 72.0 MB/s 2024-04-17T10:35:55.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.454 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.455 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147553 s, 71.1 MB/s 2024-04-17T10:35:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.472 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154706 s, 67.8 MB/s 2024-04-17T10:35:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.488 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147039 s, 71.3 MB/s 2024-04-17T10:35:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.508 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177547 s, 59.1 MB/s 2024-04-17T10:35:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.525 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150051 s, 69.9 MB/s 2024-04-17T10:35:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.541 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.541 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014306 s, 73.3 MB/s 2024-04-17T10:35:55.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.556 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130163 s, 80.6 MB/s 2024-04-17T10:35:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.573 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156795 s, 66.9 MB/s 2024-04-17T10:35:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.589 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143217 s, 73.2 MB/s 2024-04-17T10:35:55.605 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.606 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145115 s, 72.3 MB/s 2024-04-17T10:35:55.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.623 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156666 s, 66.9 MB/s 2024-04-17T10:35:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.640 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152025 s, 69.0 MB/s 2024-04-17T10:35:55.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:55.641 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:55.641 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:55.642 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.19 2024-04-17T10:35:55.642 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:55.642 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:55.642 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:55.642 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148165 s, 70.8 MB/s 2024-04-17T10:35:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.732 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160896 s, 65.2 MB/s 2024-04-17T10:35:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.752 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184162 s, 56.9 MB/s 2024-04-17T10:35:55.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.769 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.769 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151332 s, 69.3 MB/s 2024-04-17T10:35:55.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.790 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186996 s, 56.1 MB/s 2024-04-17T10:35:55.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171325 s, 61.2 MB/s 2024-04-17T10:35:55.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.823 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129118 s, 81.2 MB/s 2024-04-17T10:35:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.838 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132076 s, 79.4 MB/s 2024-04-17T10:35:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.855 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148761 s, 70.5 MB/s 2024-04-17T10:35:55.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.872 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.872 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015131 s, 69.3 MB/s 2024-04-17T10:35:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.888 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138995 s, 75.4 MB/s 2024-04-17T10:35:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.905 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015782 s, 66.4 MB/s 2024-04-17T10:35:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154093 s, 68.0 MB/s 2024-04-17T10:35:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.939 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151042 s, 69.4 MB/s 2024-04-17T10:35:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.956 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145464 s, 72.1 MB/s 2024-04-17T10:35:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.976 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018652 s, 56.2 MB/s 2024-04-17T10:35:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:55.993 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148152 s, 70.8 MB/s 2024-04-17T10:35:56.008 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.009 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140807 s, 74.5 MB/s 2024-04-17T10:35:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.024 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.025 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141802 s, 73.9 MB/s 2024-04-17T10:35:56.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.041 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.041 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145616 s, 72.0 MB/s 2024-04-17T10:35:56.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.058 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.058 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014943 s, 70.2 MB/s 2024-04-17T10:35:56.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.073 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.073 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132926 s, 78.9 MB/s 2024-04-17T10:35:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.090 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149968 s, 69.9 MB/s 2024-04-17T10:35:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.111 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187205 s, 56.0 MB/s 2024-04-17T10:35:56.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.127 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147994 s, 70.9 MB/s 2024-04-17T10:35:56.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.143 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138472 s, 75.7 MB/s 2024-04-17T10:35:56.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151985 s, 69.0 MB/s 2024-04-17T10:35:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.175 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139269 s, 75.3 MB/s 2024-04-17T10:35:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139813 s, 75.0 MB/s 2024-04-17T10:35:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.208 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153926 s, 68.1 MB/s 2024-04-17T10:35:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.224 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141505 s, 74.1 MB/s 2024-04-17T10:35:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141258 s, 74.2 MB/s 2024-04-17T10:35:56.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:56.242 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:56.242 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:56.242 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.20 2024-04-17T10:35:56.242 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:56.242 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:56.242 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:56.242 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147221 s, 71.2 MB/s 2024-04-17T10:35:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.331 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142988 s, 73.3 MB/s 2024-04-17T10:35:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.348 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148785 s, 70.5 MB/s 2024-04-17T10:35:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.365 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150708 s, 69.6 MB/s 2024-04-17T10:35:56.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.384 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.384 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169867 s, 61.7 MB/s 2024-04-17T10:35:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.400 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148978 s, 70.4 MB/s 2024-04-17T10:35:56.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01736 s, 60.4 MB/s 2024-04-17T10:35:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.436 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014566 s, 72.0 MB/s 2024-04-17T10:35:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.452 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139663 s, 75.1 MB/s 2024-04-17T10:35:56.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.473 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.473 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193166 s, 54.3 MB/s 2024-04-17T10:35:56.488 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.489 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013743 s, 76.3 MB/s 2024-04-17T10:35:56.507 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.508 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172182 s, 60.9 MB/s 2024-04-17T10:35:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.524 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150348 s, 69.7 MB/s 2024-04-17T10:35:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.539 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125382 s, 83.6 MB/s 2024-04-17T10:35:56.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.555 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.556 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146674 s, 71.5 MB/s 2024-04-17T10:35:56.572 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.572 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.572 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151653 s, 69.1 MB/s 2024-04-17T10:35:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.588 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143262 s, 73.2 MB/s 2024-04-17T10:35:56.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.606 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.606 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155917 s, 67.3 MB/s 2024-04-17T10:35:56.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.622 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.622 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146539 s, 71.6 MB/s 2024-04-17T10:35:56.637 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.638 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136703 s, 76.7 MB/s 2024-04-17T10:35:56.653 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.654 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139508 s, 75.2 MB/s 2024-04-17T10:35:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155468 s, 67.4 MB/s 2024-04-17T10:35:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.686 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013237 s, 79.2 MB/s 2024-04-17T10:35:56.702 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.702 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.702 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014048 s, 74.6 MB/s 2024-04-17T10:35:56.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.718 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.718 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014883 s, 70.5 MB/s 2024-04-17T10:35:56.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.734 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.734 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130887 s, 80.1 MB/s 2024-04-17T10:35:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.749 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137856 s, 76.1 MB/s 2024-04-17T10:35:56.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.765 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138175 s, 75.9 MB/s 2024-04-17T10:35:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.784 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174608 s, 60.1 MB/s 2024-04-17T10:35:56.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.800 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.800 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141818 s, 73.9 MB/s 2024-04-17T10:35:56.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.820 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.820 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182376 s, 57.5 MB/s 2024-04-17T10:35:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134184 s, 78.1 MB/s 2024-04-17T10:35:56.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:56.837 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:56.837 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:56.837 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.21 2024-04-17T10:35:56.837 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:56.837 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:56.837 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:56.837 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:56.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.907 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.908 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156937 s, 66.8 MB/s 2024-04-17T10:35:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.922 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128088 s, 81.9 MB/s 2024-04-17T10:35:56.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.939 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155833 s, 67.3 MB/s 2024-04-17T10:35:56.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.956 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.956 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146748 s, 71.5 MB/s 2024-04-17T10:35:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.971 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131709 s, 79.6 MB/s 2024-04-17T10:35:56.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:56.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:56.987 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143373 s, 73.1 MB/s 2024-04-17T10:35:57.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.002 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133819 s, 78.4 MB/s 2024-04-17T10:35:57.019 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.020 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157146 s, 66.7 MB/s 2024-04-17T10:35:57.036 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.037 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152412 s, 68.8 MB/s 2024-04-17T10:35:57.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.052 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.052 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137042 s, 76.5 MB/s 2024-04-17T10:35:57.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.068 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.068 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140769 s, 74.5 MB/s 2024-04-17T10:35:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.084 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139032 s, 75.4 MB/s 2024-04-17T10:35:57.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.099 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.099 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136927 s, 76.6 MB/s 2024-04-17T10:35:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.114 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130409 s, 80.4 MB/s 2024-04-17T10:35:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.127 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108177 s, 96.9 MB/s 2024-04-17T10:35:57.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.143 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.143 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014667 s, 71.5 MB/s 2024-04-17T10:35:57.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014898 s, 70.4 MB/s 2024-04-17T10:35:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.175 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137457 s, 76.3 MB/s 2024-04-17T10:35:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135963 s, 77.1 MB/s 2024-04-17T10:35:57.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136542 s, 76.8 MB/s 2024-04-17T10:35:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.223 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151982 s, 69.0 MB/s 2024-04-17T10:35:57.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148895 s, 70.4 MB/s 2024-04-17T10:35:57.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.258 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159112 s, 65.9 MB/s 2024-04-17T10:35:57.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.278 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184283 s, 56.9 MB/s 2024-04-17T10:35:57.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.296 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.296 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161968 s, 64.7 MB/s 2024-04-17T10:35:57.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.316 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.316 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178628 s, 58.7 MB/s 2024-04-17T10:35:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149858 s, 70.0 MB/s 2024-04-17T10:35:57.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.352 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.352 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172341 s, 60.8 MB/s 2024-04-17T10:35:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.369 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015004 s, 69.9 MB/s 2024-04-17T10:35:57.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.386 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.387 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152749 s, 68.6 MB/s 2024-04-17T10:35:57.402 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.403 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.403 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144474 s, 72.6 MB/s 2024-04-17T10:35:57.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.419 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.419 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149714 s, 70.0 MB/s 2024-04-17T10:35:57.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:57.421 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:57.421 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:57.421 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.22 2024-04-17T10:35:57.421 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:57.421 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:57.421 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:57.422 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:57.491 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.491 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.491 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147951 s, 70.9 MB/s 2024-04-17T10:35:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.510 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175271 s, 59.8 MB/s 2024-04-17T10:35:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.530 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184303 s, 56.9 MB/s 2024-04-17T10:35:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.546 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137492 s, 76.3 MB/s 2024-04-17T10:35:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.562 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014595 s, 71.8 MB/s 2024-04-17T10:35:57.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179323 s, 58.5 MB/s 2024-04-17T10:35:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.598 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145338 s, 72.1 MB/s 2024-04-17T10:35:57.614 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.615 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014082 s, 74.5 MB/s 2024-04-17T10:35:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.631 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149427 s, 70.2 MB/s 2024-04-17T10:35:57.648 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.648 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.648 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152673 s, 68.7 MB/s 2024-04-17T10:35:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140574 s, 74.6 MB/s 2024-04-17T10:35:57.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.681 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149608 s, 70.1 MB/s 2024-04-17T10:35:57.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.697 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.697 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146208 s, 71.7 MB/s 2024-04-17T10:35:57.712 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.712 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.712 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135849 s, 77.2 MB/s 2024-04-17T10:35:57.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.728 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.728 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014018 s, 74.8 MB/s 2024-04-17T10:35:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.743 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136689 s, 76.7 MB/s 2024-04-17T10:35:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147777 s, 71.0 MB/s 2024-04-17T10:35:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.777 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151525 s, 69.2 MB/s 2024-04-17T10:35:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.794 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155045 s, 67.6 MB/s 2024-04-17T10:35:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.809 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136279 s, 76.9 MB/s 2024-04-17T10:35:57.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.826 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.826 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01418 s, 73.9 MB/s 2024-04-17T10:35:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.842 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145906 s, 71.9 MB/s 2024-04-17T10:35:57.858 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.859 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148631 s, 70.5 MB/s 2024-04-17T10:35:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.874 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138443 s, 75.7 MB/s 2024-04-17T10:35:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.891 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151711 s, 69.1 MB/s 2024-04-17T10:35:57.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.906 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.906 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133858 s, 78.3 MB/s 2024-04-17T10:35:57.922 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.923 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144804 s, 72.4 MB/s 2024-04-17T10:35:57.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.939 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.939 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145138 s, 72.2 MB/s 2024-04-17T10:35:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.955 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141863 s, 73.9 MB/s 2024-04-17T10:35:57.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.971 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.971 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147723 s, 71.0 MB/s 2024-04-17T10:35:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145846 s, 71.9 MB/s 2024-04-17T10:35:58.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.005 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.005 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155306 s, 67.5 MB/s 2024-04-17T10:35:58.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:58.006 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:58.006 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:58.007 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.23 2024-04-17T10:35:58.007 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:58.007 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:58.007 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:58.007 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.079 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145578 s, 72.0 MB/s 2024-04-17T10:35:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149039 s, 70.4 MB/s 2024-04-17T10:35:58.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.112 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.112 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144161 s, 72.7 MB/s 2024-04-17T10:35:58.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.129 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137881 s, 76.0 MB/s 2024-04-17T10:35:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.144 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149831 s, 70.0 MB/s 2024-04-17T10:35:58.159 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.160 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.160 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136568 s, 76.8 MB/s 2024-04-17T10:35:58.175 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.176 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.176 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142018 s, 73.8 MB/s 2024-04-17T10:35:58.197 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.198 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198769 s, 52.8 MB/s 2024-04-17T10:35:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.214 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146794 s, 71.4 MB/s 2024-04-17T10:35:58.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:35:58.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:35:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146578 s, 71.5 MB/s 2024-04-17T10:35:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.247 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146592 s, 71.5 MB/s 2024-04-17T10:35:58.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.264 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.264 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152922 s, 68.6 MB/s 2024-04-17T10:35:58.279 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.280 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141365 s, 74.2 MB/s 2024-04-17T10:35:58.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.300 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.301 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189278 s, 55.4 MB/s 2024-04-17T10:35:58.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.318 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.318 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156558 s, 67.0 MB/s 2024-04-17T10:35:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183947 s, 57.0 MB/s 2024-04-17T10:35:58.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.354 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141873 s, 73.9 MB/s 2024-04-17T10:35:58.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.370 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.370 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141826 s, 73.9 MB/s 2024-04-17T10:35:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135313 s, 77.5 MB/s 2024-04-17T10:35:58.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.401 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.401 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138273 s, 75.8 MB/s 2024-04-17T10:35:58.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.422 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.422 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192786 s, 54.4 MB/s 2024-04-17T10:35:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.437 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129826 s, 80.8 MB/s 2024-04-17T10:35:58.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.453 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.453 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143816 s, 72.9 MB/s 2024-04-17T10:35:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.474 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188045 s, 55.8 MB/s 2024-04-17T10:35:58.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.487 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.487 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114804 s, 91.3 MB/s 2024-04-17T10:35:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.503 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141234 s, 74.2 MB/s 2024-04-17T10:35:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.521 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162502 s, 64.5 MB/s 2024-04-17T10:35:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.538 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148838 s, 70.5 MB/s 2024-04-17T10:35:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.553 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138008 s, 76.0 MB/s 2024-04-17T10:35:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138382 s, 75.8 MB/s 2024-04-17T10:35:58.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.585 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.585 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144232 s, 72.7 MB/s 2024-04-17T10:35:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.599 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118528 s, 88.5 MB/s 2024-04-17T10:35:58.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:58.601 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:58.601 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:58.601 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.24 2024-04-17T10:35:58.601 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:58.601 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:58.601 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:58.601 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:58.669 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.670 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.670 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132724 s, 79.0 MB/s 2024-04-17T10:35:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140958 s, 74.4 MB/s 2024-04-17T10:35:58.702 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.702 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.702 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151007 s, 69.4 MB/s 2024-04-17T10:35:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132093 s, 79.4 MB/s 2024-04-17T10:35:58.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.733 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.733 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142366 s, 73.7 MB/s 2024-04-17T10:35:58.750 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.750 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.750 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147139 s, 71.3 MB/s 2024-04-17T10:35:58.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.766 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.766 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014171 s, 74.0 MB/s 2024-04-17T10:35:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.781 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139979 s, 74.9 MB/s 2024-04-17T10:35:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.798 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149987 s, 69.9 MB/s 2024-04-17T10:35:58.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.814 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141896 s, 73.9 MB/s 2024-04-17T10:35:58.829 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.830 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.830 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137054 s, 76.5 MB/s 2024-04-17T10:35:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.846 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146099 s, 71.8 MB/s 2024-04-17T10:35:58.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.861 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.861 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135904 s, 77.2 MB/s 2024-04-17T10:35:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173563 s, 60.4 MB/s 2024-04-17T10:35:58.897 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153853 s, 68.2 MB/s 2024-04-17T10:35:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015222 s, 68.9 MB/s 2024-04-17T10:35:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130509 s, 80.3 MB/s 2024-04-17T10:35:58.945 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.946 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.946 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145693 s, 72.0 MB/s 2024-04-17T10:35:58.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132827 s, 78.9 MB/s 2024-04-17T10:35:58.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143032 s, 73.3 MB/s 2024-04-17T10:35:58.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:58.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:58.994 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148196 s, 70.8 MB/s 2024-04-17T10:35:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.010 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151157 s, 69.4 MB/s 2024-04-17T10:35:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.029 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161679 s, 64.9 MB/s 2024-04-17T10:35:59.045 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.046 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.046 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154765 s, 67.8 MB/s 2024-04-17T10:35:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.061 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138814 s, 75.5 MB/s 2024-04-17T10:35:59.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.076 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131481 s, 79.8 MB/s 2024-04-17T10:35:59.092 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.093 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.093 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146925 s, 71.4 MB/s 2024-04-17T10:35:59.109 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.110 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151432 s, 69.2 MB/s 2024-04-17T10:35:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143562 s, 73.0 MB/s 2024-04-17T10:35:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144594 s, 72.5 MB/s 2024-04-17T10:35:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.158 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147642 s, 71.0 MB/s 2024-04-17T10:35:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.174 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140628 s, 74.6 MB/s 2024-04-17T10:35:59.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:59.175 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:59.176 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:59.176 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.25 2024-04-17T10:35:59.176 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:59.176 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:59.176 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:59.176 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135793 s, 77.2 MB/s 2024-04-17T10:35:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.262 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152622 s, 68.7 MB/s 2024-04-17T10:35:59.280 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.281 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165167 s, 63.5 MB/s 2024-04-17T10:35:59.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.297 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.297 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150038 s, 69.9 MB/s 2024-04-17T10:35:59.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146673 s, 71.5 MB/s 2024-04-17T10:35:59.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.329 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.330 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139299 s, 75.3 MB/s 2024-04-17T10:35:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.348 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169551 s, 61.8 MB/s 2024-04-17T10:35:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.365 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015263 s, 68.7 MB/s 2024-04-17T10:35:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.381 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135889 s, 77.2 MB/s 2024-04-17T10:35:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.396 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135637 s, 77.3 MB/s 2024-04-17T10:35:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.414 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156907 s, 66.8 MB/s 2024-04-17T10:35:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.430 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146418 s, 71.6 MB/s 2024-04-17T10:35:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.446 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014213 s, 73.8 MB/s 2024-04-17T10:35:59.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.461 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.461 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134558 s, 77.9 MB/s 2024-04-17T10:35:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.477 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140387 s, 74.7 MB/s 2024-04-17T10:35:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.494 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153658 s, 68.2 MB/s 2024-04-17T10:35:59.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.511 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.511 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149594 s, 70.1 MB/s 2024-04-17T10:35:59.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.526 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.526 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131176 s, 79.9 MB/s 2024-04-17T10:35:59.546 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.547 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.547 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190208 s, 55.1 MB/s 2024-04-17T10:35:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.563 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148686 s, 70.5 MB/s 2024-04-17T10:35:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.583 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183665 s, 57.1 MB/s 2024-04-17T10:35:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.600 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151886 s, 69.0 MB/s 2024-04-17T10:35:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.616 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139741 s, 75.0 MB/s 2024-04-17T10:35:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.634 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137867 s, 76.1 MB/s 2024-04-17T10:35:59.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.647 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.647 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013695 s, 76.6 MB/s 2024-04-17T10:35:59.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.664 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.664 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144272 s, 72.7 MB/s 2024-04-17T10:35:59.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.680 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.681 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151819 s, 69.1 MB/s 2024-04-17T10:35:59.696 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.696 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.696 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013642 s, 76.9 MB/s 2024-04-17T10:35:59.711 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.712 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137315 s, 76.4 MB/s 2024-04-17T10:35:59.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.732 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188032 s, 55.8 MB/s 2024-04-17T10:35:59.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.749 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.749 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148752 s, 70.5 MB/s 2024-04-17T10:35:59.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.765 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143848 s, 72.9 MB/s 2024-04-17T10:35:59.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:35:59.767 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:35:59.767 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:35:59.767 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.26 2024-04-17T10:35:59.767 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:35:59.767 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:35:59.767 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:35:59.767 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:35:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139442 s, 75.2 MB/s 2024-04-17T10:35:59.852 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133963 s, 78.3 MB/s 2024-04-17T10:35:59.870 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.871 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160133 s, 65.5 MB/s 2024-04-17T10:35:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.886 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.886 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132533 s, 79.1 MB/s 2024-04-17T10:35:59.902 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.902 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.902 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014924 s, 70.3 MB/s 2024-04-17T10:35:59.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.917 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.918 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133158 s, 78.7 MB/s 2024-04-17T10:35:59.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.932 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.933 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132401 s, 79.2 MB/s 2024-04-17T10:35:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.949 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149541 s, 70.1 MB/s 2024-04-17T10:35:59.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.965 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.965 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139554 s, 75.1 MB/s 2024-04-17T10:35:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.981 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141847 s, 73.9 MB/s 2024-04-17T10:35:59.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:35:59.999 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:35:59.999 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157352 s, 66.6 MB/s 2024-04-17T10:36:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.015 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142029 s, 73.8 MB/s 2024-04-17T10:36:00.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.033 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.033 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154295 s, 68.0 MB/s 2024-04-17T10:36:00.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.050 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.050 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014822 s, 70.7 MB/s 2024-04-17T10:36:00.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.066 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.066 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147885 s, 70.9 MB/s 2024-04-17T10:36:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.086 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179642 s, 58.4 MB/s 2024-04-17T10:36:00.102 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.103 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.103 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149069 s, 70.3 MB/s 2024-04-17T10:36:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.122 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177598 s, 59.0 MB/s 2024-04-17T10:36:00.137 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.138 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.138 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137171 s, 76.4 MB/s 2024-04-17T10:36:00.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.154 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.154 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146874 s, 71.4 MB/s 2024-04-17T10:36:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174226 s, 60.2 MB/s 2024-04-17T10:36:00.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.189 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.189 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134947 s, 77.7 MB/s 2024-04-17T10:36:00.205 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.206 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150117 s, 69.9 MB/s 2024-04-17T10:36:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.223 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155403 s, 67.5 MB/s 2024-04-17T10:36:00.239 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.240 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148141 s, 70.8 MB/s 2024-04-17T10:36:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.255 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138353 s, 75.8 MB/s 2024-04-17T10:36:00.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.271 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.271 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139554 s, 75.1 MB/s 2024-04-17T10:36:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.287 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143444 s, 73.1 MB/s 2024-04-17T10:36:00.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.304 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153772 s, 68.2 MB/s 2024-04-17T10:36:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.321 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150534 s, 69.7 MB/s 2024-04-17T10:36:00.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.337 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.337 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140606 s, 74.6 MB/s 2024-04-17T10:36:00.353 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.353 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.353 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138084 s, 75.9 MB/s 2024-04-17T10:36:00.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:36:00.354 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:36:00.354 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:36:00.355 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.27 2024-04-17T10:36:00.355 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:36:00.355 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:36:00.355 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:36:00.355 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:36:00.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.426 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.426 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136366 s, 76.9 MB/s 2024-04-17T10:36:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155106 s, 67.6 MB/s 2024-04-17T10:36:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.460 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150555 s, 69.6 MB/s 2024-04-17T10:36:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.477 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152356 s, 68.8 MB/s 2024-04-17T10:36:00.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.493 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.493 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139296 s, 75.3 MB/s 2024-04-17T10:36:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.508 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134854 s, 77.8 MB/s 2024-04-17T10:36:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.524 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139641 s, 75.1 MB/s 2024-04-17T10:36:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.539 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135925 s, 77.1 MB/s 2024-04-17T10:36:00.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.557 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.557 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155376 s, 67.5 MB/s 2024-04-17T10:36:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.574 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.575 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154759 s, 67.8 MB/s 2024-04-17T10:36:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191271 s, 54.8 MB/s 2024-04-17T10:36:00.611 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.611 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.611 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147556 s, 71.1 MB/s 2024-04-17T10:36:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133377 s, 78.6 MB/s 2024-04-17T10:36:00.643 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.644 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.644 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149307 s, 70.2 MB/s 2024-04-17T10:36:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.660 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146944 s, 71.4 MB/s 2024-04-17T10:36:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.677 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153725 s, 68.2 MB/s 2024-04-17T10:36:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.693 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137929 s, 76.0 MB/s 2024-04-17T10:36:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141006 s, 74.4 MB/s 2024-04-17T10:36:00.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.724 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.724 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136719 s, 76.7 MB/s 2024-04-17T10:36:00.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.739 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.739 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132882 s, 78.9 MB/s 2024-04-17T10:36:00.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.756 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.756 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152981 s, 68.5 MB/s 2024-04-17T10:36:00.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.772 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.772 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143026 s, 73.3 MB/s 2024-04-17T10:36:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149177 s, 70.3 MB/s 2024-04-17T10:36:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.802 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116651 s, 89.9 MB/s 2024-04-17T10:36:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.818 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146073 s, 71.8 MB/s 2024-04-17T10:36:00.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.835 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.835 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148463 s, 70.6 MB/s 2024-04-17T10:36:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139892 s, 75.0 MB/s 2024-04-17T10:36:00.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.868 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.868 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146156 s, 71.7 MB/s 2024-04-17T10:36:00.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.882 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.882 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133881 s, 78.3 MB/s 2024-04-17T10:36:00.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.898 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.898 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142503 s, 73.6 MB/s 2024-04-17T10:36:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137294 s, 76.4 MB/s 2024-04-17T10:36:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:00.930 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150079 s, 69.9 MB/s 2024-04-17T10:36:00.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:36:00.932 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:36:00.932 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:36:00.932 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.28 2024-04-17T10:36:00.932 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:36:00.932 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:36:00.932 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:36:00.932 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:36:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.002 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.002 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138377 s, 75.8 MB/s 2024-04-17T10:36:01.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.018 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.018 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151039 s, 69.4 MB/s 2024-04-17T10:36:01.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.035 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.035 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146136 s, 71.8 MB/s 2024-04-17T10:36:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143877 s, 72.9 MB/s 2024-04-17T10:36:01.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.069 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.069 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160231 s, 65.4 MB/s 2024-04-17T10:36:01.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.086 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.086 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153779 s, 68.2 MB/s 2024-04-17T10:36:01.104 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.104 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.104 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160226 s, 65.4 MB/s 2024-04-17T10:36:01.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.124 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.124 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017934 s, 58.5 MB/s 2024-04-17T10:36:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.141 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.141 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015086 s, 69.5 MB/s 2024-04-17T10:36:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.157 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147682 s, 71.0 MB/s 2024-04-17T10:36:01.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.172 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.173 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137532 s, 76.2 MB/s 2024-04-17T10:36:01.190 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.191 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.191 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163167 s, 64.3 MB/s 2024-04-17T10:36:01.207 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.207 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.207 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144981 s, 72.3 MB/s 2024-04-17T10:36:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.225 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161155 s, 65.1 MB/s 2024-04-17T10:36:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.242 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150496 s, 69.7 MB/s 2024-04-17T10:36:01.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.257 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138741 s, 75.6 MB/s 2024-04-17T10:36:01.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.273 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.273 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138597 s, 75.7 MB/s 2024-04-17T10:36:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.288 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135829 s, 77.2 MB/s 2024-04-17T10:36:01.304 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.305 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.305 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143756 s, 72.9 MB/s 2024-04-17T10:36:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.321 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152907 s, 68.6 MB/s 2024-04-17T10:36:01.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.338 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.338 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145752 s, 71.9 MB/s 2024-04-17T10:36:01.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.354 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.354 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144838 s, 72.4 MB/s 2024-04-17T10:36:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.371 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153617 s, 68.3 MB/s 2024-04-17T10:36:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.388 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154075 s, 68.1 MB/s 2024-04-17T10:36:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.404 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141948 s, 73.9 MB/s 2024-04-17T10:36:01.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.420 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.420 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139999 s, 74.9 MB/s 2024-04-17T10:36:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.435 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.436 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01388 s, 75.5 MB/s 2024-04-17T10:36:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.451 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.452 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142358 s, 73.7 MB/s 2024-04-17T10:36:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.471 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177757 s, 59.0 MB/s 2024-04-17T10:36:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.486 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134585 s, 77.9 MB/s 2024-04-17T10:36:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.503 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151493 s, 69.2 MB/s 2024-04-17T10:36:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.520 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149907 s, 69.9 MB/s 2024-04-17T10:36:01.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:36:01.522 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:36:01.522 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:36:01.522 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.29 2024-04-17T10:36:01.522 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:36:01.522 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:36:01.522 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:36:01.522 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:36:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.591 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015357 s, 68.3 MB/s 2024-04-17T10:36:01.607 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.608 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.608 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146698 s, 71.5 MB/s 2024-04-17T10:36:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.623 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136095 s, 77.0 MB/s 2024-04-17T10:36:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.638 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137133 s, 76.5 MB/s 2024-04-17T10:36:01.653 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.654 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134205 s, 78.1 MB/s 2024-04-17T10:36:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.671 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015764 s, 66.5 MB/s 2024-04-17T10:36:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.685 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117526 s, 89.2 MB/s 2024-04-17T10:36:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.701 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148027 s, 70.8 MB/s 2024-04-17T10:36:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.717 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.718 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142773 s, 73.4 MB/s 2024-04-17T10:36:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.732 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013241 s, 79.2 MB/s 2024-04-17T10:36:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.748 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141085 s, 74.3 MB/s 2024-04-17T10:36:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.765 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150395 s, 69.7 MB/s 2024-04-17T10:36:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.782 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.783 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155289 s, 67.5 MB/s 2024-04-17T10:36:01.798 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.799 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.799 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014362 s, 73.0 MB/s 2024-04-17T10:36:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.815 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151164 s, 69.4 MB/s 2024-04-17T10:36:01.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.832 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.832 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144669 s, 72.5 MB/s 2024-04-17T10:36:01.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.849 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.849 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154979 s, 67.7 MB/s 2024-04-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.864 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132425 s, 79.2 MB/s 2024-04-17T10:36:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.880 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142744 s, 73.5 MB/s 2024-04-17T10:36:01.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.895 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.895 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134962 s, 77.7 MB/s 2024-04-17T10:36:01.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.911 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.911 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138533 s, 75.7 MB/s 2024-04-17T10:36:01.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.926 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.926 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136685 s, 76.7 MB/s 2024-04-17T10:36:01.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.943 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.943 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148991 s, 70.4 MB/s 2024-04-17T10:36:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.960 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153662 s, 68.2 MB/s 2024-04-17T10:36:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.976 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136892 s, 76.6 MB/s 2024-04-17T10:36:01.991 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:01.992 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138809 s, 75.5 MB/s 2024-04-17T10:36:02.008 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.008 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.008 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146846 s, 71.4 MB/s 2024-04-17T10:36:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.025 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148415 s, 70.7 MB/s 2024-04-17T10:36:02.040 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.040 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.040 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137933 s, 76.0 MB/s 2024-04-17T10:36:02.056 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.056 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.056 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014469 s, 72.5 MB/s 2024-04-17T10:36:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.075 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.076 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176164 s, 59.5 MB/s 2024-04-17T10:36:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.091 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136913 s, 76.6 MB/s 2024-04-17T10:36:02.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:36:02.092 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:36:02.093 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:36:02.093 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.30 2024-04-17T10:36:02.093 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:36:02.093 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:36:02.093 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:36:02.093 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:36:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01738 s, 60.3 MB/s 2024-04-17T10:36:02.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.187 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.187 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162695 s, 64.5 MB/s 2024-04-17T10:36:02.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.203 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.203 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145053 s, 72.3 MB/s 2024-04-17T10:36:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.220 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148429 s, 70.6 MB/s 2024-04-17T10:36:02.236 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.236 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.236 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143089 s, 73.3 MB/s 2024-04-17T10:36:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138067 s, 75.9 MB/s 2024-04-17T10:36:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.273 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186271 s, 56.3 MB/s 2024-04-17T10:36:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019033 s, 55.1 MB/s 2024-04-17T10:36:02.313 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.314 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018235 s, 57.5 MB/s 2024-04-17T10:36:02.331 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.332 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157005 s, 66.8 MB/s 2024-04-17T10:36:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.349 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148195 s, 70.8 MB/s 2024-04-17T10:36:02.365 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.366 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149195 s, 70.3 MB/s 2024-04-17T10:36:02.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.383 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.383 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160667 s, 65.3 MB/s 2024-04-17T10:36:02.399 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.399 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.400 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145983 s, 71.8 MB/s 2024-04-17T10:36:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.415 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136919 s, 76.6 MB/s 2024-04-17T10:36:02.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.433 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.433 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161558 s, 64.9 MB/s 2024-04-17T10:36:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.450 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151267 s, 69.3 MB/s 2024-04-17T10:36:02.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.466 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.466 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140138 s, 74.8 MB/s 2024-04-17T10:36:02.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.482 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.482 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147646 s, 71.0 MB/s 2024-04-17T10:36:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.502 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181142 s, 57.9 MB/s 2024-04-17T10:36:02.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.517 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.517 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133591 s, 78.5 MB/s 2024-04-17T10:36:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.534 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146282 s, 71.7 MB/s 2024-04-17T10:36:02.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.550 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.550 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145669 s, 72.0 MB/s 2024-04-17T10:36:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.567 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151967 s, 69.0 MB/s 2024-04-17T10:36:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.582 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134158 s, 78.2 MB/s 2024-04-17T10:36:02.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.600 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.600 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160669 s, 65.3 MB/s 2024-04-17T10:36:02.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.615 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.615 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134181 s, 78.1 MB/s 2024-04-17T10:36:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153475 s, 68.3 MB/s 2024-04-17T10:36:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.649 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153141 s, 68.5 MB/s 2024-04-17T10:36:02.666 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.666 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.666 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015214 s, 68.9 MB/s 2024-04-17T10:36:02.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.682 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133543 s, 78.5 MB/s 2024-04-17T10:36:02.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.698 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.698 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144667 s, 72.5 MB/s 2024-04-17T10:36:02.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:36:02.699 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-04-17T10:36:02.700 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-04-17T10:36:02.700 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p scrub_control_test_path/dir.31 2024-04-17T10:36:02.700 DEBUG:teuthology.orchestra.run.smithi067:> for ((i = 0; i < 32; i++)); do 2024-04-17T10:36:02.700 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-04-17T10:36:02.700 DEBUG:teuthology.orchestra.run.smithi067:> done 2024-04-17T10:36:02.700 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-04-17T10:36:02.775 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.776 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182718 s, 57.4 MB/s 2024-04-17T10:36:02.791 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.792 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.792 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142537 s, 73.6 MB/s 2024-04-17T10:36:02.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.808 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.808 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144752 s, 72.4 MB/s 2024-04-17T10:36:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.823 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.824 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137618 s, 76.2 MB/s 2024-04-17T10:36:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158535 s, 66.1 MB/s 2024-04-17T10:36:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.856 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131948 s, 79.5 MB/s 2024-04-17T10:36:02.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.876 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.876 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177316 s, 59.1 MB/s 2024-04-17T10:36:02.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.892 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.892 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142549 s, 73.6 MB/s 2024-04-17T10:36:02.909 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.910 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158469 s, 66.2 MB/s 2024-04-17T10:36:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.927 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015333 s, 68.4 MB/s 2024-04-17T10:36:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.944 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154857 s, 67.7 MB/s 2024-04-17T10:36:02.960 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.961 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014749 s, 71.1 MB/s 2024-04-17T10:36:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150022 s, 69.9 MB/s 2024-04-17T10:36:02.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:02.993 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:02.993 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143622 s, 73.0 MB/s 2024-04-17T10:36:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.009 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.010 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014279 s, 73.4 MB/s 2024-04-17T10:36:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155902 s, 67.3 MB/s 2024-04-17T10:36:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150356 s, 69.7 MB/s 2024-04-17T10:36:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136975 s, 76.6 MB/s 2024-04-17T10:36:03.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.076 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.076 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152238 s, 68.9 MB/s 2024-04-17T10:36:03.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.094 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.094 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154866 s, 67.7 MB/s 2024-04-17T10:36:03.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.113 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.113 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173036 s, 60.6 MB/s 2024-04-17T10:36:03.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.128 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.129 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135715 s, 77.3 MB/s 2024-04-17T10:36:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.146 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153669 s, 68.2 MB/s 2024-04-17T10:36:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137035 s, 76.5 MB/s 2024-04-17T10:36:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138808 s, 75.5 MB/s 2024-04-17T10:36:03.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.193 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.193 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141281 s, 74.2 MB/s 2024-04-17T10:36:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.211 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162834 s, 64.4 MB/s 2024-04-17T10:36:03.227 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.228 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145073 s, 72.3 MB/s 2024-04-17T10:36:03.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.243 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.244 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143431 s, 73.1 MB/s 2024-04-17T10:36:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.261 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158849 s, 66.0 MB/s 2024-04-17T10:36:03.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.278 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.278 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149249 s, 70.3 MB/s 2024-04-17T10:36:03.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-04-17T10:36:03.293 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-04-17T10:36:03.293 INFO:teuthology.orchestra.run.smithi067.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013569 s, 77.3 MB/s 2024-04-17T10:36:03.298 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-04-17T10:36:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.491+0000 7f5964000640 1 Processor -- start 2024-04-17T10:36:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.491+0000 7f5964000640 1 -- start start 2024-04-17T10:36:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5964000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f595c0cd030 0x7f595c06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5964000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f595c06e7b0 0x7f595c0b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5964000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595c0b4990 con 0x7f595c0aef60 2024-04-17T10:36:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5964000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595c0af340 con 0x7f595c0cd030 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5964000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595c06ec00 con 0x7f595c06e7b0 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5960c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f595c0cd030 0x7f595c06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5961600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f595c0aef60 0x7f595c0cc920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44690/0 (socket says 172.21.15.67:44690) 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5961600640 1 -- 172.21.15.67:0/9684494 learned_addr learned my addr 172.21.15.67:0/9684494 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5962000640 1 --2- 172.21.15.67:0/9684494 >> v2:172.21.15.86:3300/0 conn(0x7f595c06e7b0 0x7f595c0b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f595be00640 1 -- 172.21.15.67:0/9684494 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357297235 0 0) 0x7f5950001120 con 0x7f595c0aef60 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5960c00640 1 -- 172.21.15.67:0/9684494 >> v2:172.21.15.86:3300/0 conn(0x7f595c06e7b0 msgr2=0x7f595c0b3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5960c00640 1 --2- 172.21.15.67:0/9684494 >> v2:172.21.15.86:3300/0 conn(0x7f595c06e7b0 0x7f595c0b3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5960c00640 1 -- 172.21.15.67:0/9684494 >> v1:172.21.15.67:6789/0 conn(0x7f595c0aef60 legacy=0x7f595c0cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:03.485 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.495+0000 7f5960c00640 1 -- 172.21.15.67:0/9684494 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c15ae90 con 0x7f595c0cd030 2024-04-17T10:36:03.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5960c00640 1 --2- 172.21.15.67:0/9684494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c06e270 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f595c0b2e60 tx=0x7f594801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b425e8717907059 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:03.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f595be00640 1 -- 172.21.15.67:0/9684494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59480b0070 con 0x7f595c0cd030 2024-04-17T10:36:03.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f595be00640 1 -- 172.21.15.67:0/9684494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59480ab600 con 0x7f595c0cd030 2024-04-17T10:36:03.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f595be00640 1 -- 172.21.15.67:0/9684494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59480ab9d0 con 0x7f595c0cd030 2024-04-17T10:36:03.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 -- 172.21.15.67:0/9684494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 msgr2=0x7f595c06e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:03.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 --2- 172.21.15.67:0/9684494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c06e270 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f595c0b2e60 tx=0x7f594801c3f0 comp rx=0 tx=0).stop 2024-04-17T10:36:03.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 -- 172.21.15.67:0/9684494 shutdown_connections 2024-04-17T10:36:03.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 --2- 172.21.15.67:0/9684494 >> v2:172.21.15.86:3300/0 conn(0x7f595c06e7b0 0x7f595c0b3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 --2- 172.21.15.67:0/9684494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c06e270 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 -- 172.21.15.67:0/9684494 >> 172.21.15.67:0/9684494 conn(0x7f595c0b1740 msgr2=0x7f595c0b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 -- 172.21.15.67:0/9684494 shutdown_connections 2024-04-17T10:36:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 -- 172.21.15.67:0/9684494 wait complete. 2024-04-17T10:36:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 Processor -- start 2024-04-17T10:36:03.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.499+0000 7f5964000640 1 -- start start 2024-04-17T10:36:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f595c06e7b0 0x7f595c138fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f595c0aef60 0x7f595c154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:03.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c14e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c15bc10 con 0x7f595c0cd030 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595c0b3e70 con 0x7f595c06e7b0 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595c139bb0 con 0x7f595c0aef60 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c14e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c14e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34482/0 (socket says 172.21.15.67:34482) 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 -- 172.21.15.67:0/877715160 learned_addr learned my addr 172.21.15.67:0/877715160 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5961600640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f595c06e7b0 0x7f595c138fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 -- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f595c0aef60 msgr2=0x7f595c154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f595c0aef60 0x7f595c154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 -- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f595c06e7b0 msgr2=0x7f595c138fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f595c06e7b0 0x7f595c138fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 -- 172.21.15.67:0/877715160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595c139e90 con 0x7f595c0cd030 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5961600640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f595c06e7b0 0x7f595c138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:36:03.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5962000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c14e6d0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f595400efc0 tx=0x7f595400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5959600640 1 -- 172.21.15.67:0/877715160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954018020 con 0x7f595c0cd030 2024-04-17T10:36:03.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5959600640 1 -- 172.21.15.67:0/877715160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5954014480 con 0x7f595c0cd030 2024-04-17T10:36:03.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f595c13a170 con 0x7f595c0cd030 2024-04-17T10:36:03.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5959600640 1 -- 172.21.15.67:0/877715160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5954014870 con 0x7f595c0cd030 2024-04-17T10:36:03.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f595c13a730 con 0x7f595c0cd030 2024-04-17T10:36:03.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595c13a930 con 0x7f595c0cd030 2024-04-17T10:36:03.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.503+0000 7f5959600640 1 -- 172.21.15.67:0/877715160 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f595401b020 con 0x7f595c0cd030 2024-04-17T10:36:03.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.507+0000 7f5959600640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f593c066f90 0x7f593c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:03.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.507+0000 7f5959600640 1 -- 172.21.15.67:0/877715160 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f5954085cc0 con 0x7f595c0cd030 2024-04-17T10:36:03.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.507+0000 7f5961600640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f593c066f90 0x7f593c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:03.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.507+0000 7f5961600640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f593c066f90 0x7f593c069450 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5950007fd0 tx=0x7f5950009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:03.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.511+0000 7f5959600640 1 -- 172.21.15.67:0/877715160 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f595c13a930 con 0x7f595c0cd030 2024-04-17T10:36:03.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:03.707+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f595c04ef30 con 0x7f595c0cd030 2024-04-17T10:36:04.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.379+0000 7f5959600640 1 -- 172.21.15.67:0/877715160 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v104) v1 ==== 105+0+0 (secure 0 0 0) 0x7f595408c080 con 0x7f595c0cd030 2024-04-17T10:36:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f593c066f90 msgr2=0x7f593c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f593c066f90 0x7f593c069450 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f5950007fd0 tx=0x7f5950009040 comp rx=0 tx=0).stop 2024-04-17T10:36:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 msgr2=0x7f595c14e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c14e6d0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f595400efc0 tx=0x7f595400c490 comp rx=0 tx=0).stop 2024-04-17T10:36:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 shutdown_connections 2024-04-17T10:36:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f593c066f90 0x7f593c069450 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f595c0cd030 0x7f595c14e6d0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f595c0aef60 0x7f595c154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 --2- 172.21.15.67:0/877715160 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f595c06e7b0 0x7f595c138fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 >> 172.21.15.67:0/877715160 conn(0x7f595c0b1740 msgr2=0x7f595c0b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:04.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 shutdown_connections 2024-04-17T10:36:04.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.383+0000 7f5964000640 1 -- 172.21.15.67:0/877715160 wait complete. 2024-04-17T10:36:04.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-17T10:36:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.547+0000 7ffa7dc00640 1 Processor -- start 2024-04-17T10:36:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.547+0000 7ffa7dc00640 1 -- start start 2024-04-17T10:36:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa7dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa78071310 0x7ffa78071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa7dc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa780cc300 0x7ffa780cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa7dc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7806e2f0 con 0x7ffa78145040 2024-04-17T10:36:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa7dc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa780a86c0 con 0x7ffa780cc300 2024-04-17T10:36:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa7dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa78071cb0 con 0x7ffa78071310 2024-04-17T10:36:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa780cc300 0x7ffa780cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ffa780cc300 0x7ffa780cc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34488/0 (socket says 172.21.15.67:34488) 2024-04-17T10:36:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 -- 172.21.15.67:0/1812028031 learned_addr learned my addr 172.21.15.67:0/1812028031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa6e000640 1 --2- 172.21.15.67:0/1812028031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa78071310 0x7ffa78071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 -- 172.21.15.67:0/1812028031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa78071310 msgr2=0x7ffa78071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 --2- 172.21.15.67:0/1812028031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa78071310 0x7ffa78071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 -- 172.21.15.67:0/1812028031 >> v1:172.21.15.86:6789/0 conn(0x7ffa78145040 legacy=0x7ffa7806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 -- 172.21.15.67:0/1812028031 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7806d6e0 con 0x7ffa780cc300 2024-04-17T10:36:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.551+0000 7ffa77400640 1 --2- 172.21.15.67:0/1812028031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa780cc300 0x7ffa780cc750 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009870 tx=0x7ffa7001ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38ef49c8e81901c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa76000640 1 -- 172.21.15.67:0/1812028031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa700b5070 con 0x7ffa780cc300 2024-04-17T10:36:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa76000640 1 -- 172.21.15.67:0/1812028031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa700a8a50 con 0x7ffa780cc300 2024-04-17T10:36:04.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa76000640 1 -- 172.21.15.67:0/1812028031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa700a8e20 con 0x7ffa780cc300 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1812028031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa780cc300 msgr2=0x7ffa780cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1812028031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa780cc300 0x7ffa780cc750 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009870 tx=0x7ffa7001ed50 comp rx=0 tx=0).stop 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1812028031 shutdown_connections 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1812028031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa780cc300 0x7ffa780cc750 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1812028031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa78071310 0x7ffa78071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1812028031 >> 172.21.15.67:0/1812028031 conn(0x7ffa7806ac90 msgr2=0x7ffa7806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1812028031 shutdown_connections 2024-04-17T10:36:04.544 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1812028031 wait complete. 2024-04-17T10:36:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 Processor -- start 2024-04-17T10:36:04.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- start start 2024-04-17T10:36:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78071310 0x7ffa7813e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa780cc300 0x7ffa78137eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa78145040 0x7ffa7813b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7806dca0 con 0x7ffa78145040 2024-04-17T10:36:04.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7815bf10 con 0x7ffa78071310 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.555+0000 7ffa7dc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7806df80 con 0x7ffa780cc300 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa78145040 0x7ffa7813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa78145040 0x7ffa7813b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34492/0 (socket says 172.21.15.67:34492) 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 -- 172.21.15.67:0/1313641845 learned_addr learned my addr 172.21.15.67:0/1313641845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa76a00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78071310 0x7ffa7813e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa6e000640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa780cc300 0x7ffa78137eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 -- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa780cc300 msgr2=0x7ffa78137eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa780cc300 0x7ffa78137eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 -- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78071310 msgr2=0x7ffa7813e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78071310 0x7ffa7813e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 -- 172.21.15.67:0/1313641845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7813e960 con 0x7ffa78145040 2024-04-17T10:36:04.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa77400640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa78145040 0x7ffa7813b8d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ffa700b3040 tx=0x7ffa7001baf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa6f400640 1 -- 172.21.15.67:0/1313641845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa700b5070 con 0x7ffa78145040 2024-04-17T10:36:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa6f400640 1 -- 172.21.15.67:0/1313641845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa700b0d80 con 0x7ffa78145040 2024-04-17T10:36:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7813ec40 con 0x7ffa78145040 2024-04-17T10:36:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa6f400640 1 -- 172.21.15.67:0/1313641845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa700bb630 con 0x7ffa78145040 2024-04-17T10:36:04.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7813f200 con 0x7ffa78145040 2024-04-17T10:36:04.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa78072d50 con 0x7ffa78145040 2024-04-17T10:36:04.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.559+0000 7ffa6f400640 1 -- 172.21.15.67:0/1313641845 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ffa700c2060 con 0x7ffa78145040 2024-04-17T10:36:04.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.563+0000 7ffa6f400640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa54066f60 0x7ffa54069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.563+0000 7ffa6f400640 1 -- 172.21.15.67:0/1313641845 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6101+0+0 (secure 0 0 0) 0x7ffa701224f0 con 0x7ffa78145040 2024-04-17T10:36:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.563+0000 7ffa76a00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa54066f60 0x7ffa54069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.563+0000 7ffa76a00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa54066f60 0x7ffa54069420 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ffa781445c0 tx=0x7ffa68009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:04.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.567+0000 7ffa6f400640 1 -- 172.21.15.67:0/1313641845 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ffa700188b0 con 0x7ffa78145040 2024-04-17T10:36:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.767+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ffa78074e30 con 0x7ffa78145040 2024-04-17T10:36:04.756 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:36:04.767+0000 7ff63f200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-17T10:36:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa6f400640 1 -- 172.21.15.67:0/1313641845 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v105) v1 ==== 161+0+0 (secure 0 0 0) 0x7ffa700f2d10 con 0x7ffa78145040 2024-04-17T10:36:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-17T10:36:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa54066f60 msgr2=0x7ffa54069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa54066f60 0x7ffa54069420 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7ffa781445c0 tx=0x7ffa68009040 comp rx=0 tx=0).stop 2024-04-17T10:36:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa78145040 msgr2=0x7ffa7813b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa78145040 0x7ffa7813b8d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7ffa700b3040 tx=0x7ffa7001baf0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 shutdown_connections 2024-04-17T10:36:04.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ffa54066f60 0x7ffa54069420 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa78145040 0x7ffa7813b8d0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ffa780cc300 0x7ffa78137eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 --2- 172.21.15.67:0/1313641845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ffa78071310 0x7ffa7813e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 >> 172.21.15.67:0/1313641845 conn(0x7ffa7806ac90 msgr2=0x7ffa7813ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 shutdown_connections 2024-04-17T10:36:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.771+0000 7ffa7dc00640 1 -- 172.21.15.67:0/1313641845 wait complete. 2024-04-17T10:36:04.784 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-17T10:36:04.831 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2024-04-17T10:36:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebee00640 1 Processor -- start 2024-04-17T10:36:04.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebee00640 1 -- start start 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebee00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb813ec90 0x7f8eb806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebee00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8eb806b1d0 0x7f8eb815f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebee00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb815bf50 con 0x7f8eb813e7a0 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebee00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb806b620 con 0x7f8eb806b1d0 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebee00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb813f0d0 con 0x7f8eb813ec90 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebda00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8eb813e7a0 0x7f8eb8070710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44702/0 (socket says 172.21.15.67:44702) 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebda00640 1 -- 172.21.15.67:0/2648431380 learned_addr learned my addr 172.21.15.67:0/2648431380 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebd000640 1 --2- 172.21.15.67:0/2648431380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb813ec90 0x7f8eb806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebe400640 1 --2- 172.21.15.67:0/2648431380 >> v2:172.21.15.86:3300/0 conn(0x7f8eb806b1d0 0x7f8eb815f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8eb7e00640 1 -- 172.21.15.67:0/2648431380 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 175109811 0 0) 0x7f8eb815bf50 con 0x7f8eb813e7a0 2024-04-17T10:36:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8eb7e00640 1 -- 172.21.15.67:0/2648431380 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea0003530 con 0x7f8eb813e7a0 2024-04-17T10:36:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebd000640 1 -- 172.21.15.67:0/2648431380 >> v2:172.21.15.86:3300/0 conn(0x7f8eb806b1d0 msgr2=0x7f8eb815f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebd000640 1 --2- 172.21.15.67:0/2648431380 >> v2:172.21.15.86:3300/0 conn(0x7f8eb806b1d0 0x7f8eb815f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebd000640 1 -- 172.21.15.67:0/2648431380 >> v1:172.21.15.67:6789/0 conn(0x7f8eb813e7a0 legacy=0x7f8eb8070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.943+0000 7f8ebd000640 1 -- 172.21.15.67:0/2648431380 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb815c230 con 0x7f8eb813ec90 2024-04-17T10:36:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebd000640 1 --2- 172.21.15.67:0/2648431380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb813ec90 0x7f8eb806ac90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097c0 tx=0x7f8ea401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad03edd085d72e6e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8eb7e00640 1 -- 172.21.15.67:0/2648431380 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40b5070 con 0x7f8eb813ec90 2024-04-17T10:36:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8eb7e00640 1 -- 172.21.15.67:0/2648431380 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea40a8990 con 0x7f8eb813ec90 2024-04-17T10:36:04.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8eb7e00640 1 -- 172.21.15.67:0/2648431380 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea40a8d60 con 0x7f8eb813ec90 2024-04-17T10:36:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 -- 172.21.15.67:0/2648431380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb813ec90 msgr2=0x7f8eb806ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 --2- 172.21.15.67:0/2648431380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb813ec90 0x7f8eb806ac90 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40097c0 tx=0x7f8ea401ec90 comp rx=0 tx=0).stop 2024-04-17T10:36:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 -- 172.21.15.67:0/2648431380 shutdown_connections 2024-04-17T10:36:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 --2- 172.21.15.67:0/2648431380 >> v2:172.21.15.86:3300/0 conn(0x7f8eb806b1d0 0x7f8eb815f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 --2- 172.21.15.67:0/2648431380 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb813ec90 0x7f8eb806ac90 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 -- 172.21.15.67:0/2648431380 >> 172.21.15.67:0/2648431380 conn(0x7f8eb80b2310 msgr2=0x7f8eb80b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:04.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 -- 172.21.15.67:0/2648431380 shutdown_connections 2024-04-17T10:36:04.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.947+0000 7f8ebee00640 1 -- 172.21.15.67:0/2648431380 wait complete. 2024-04-17T10:36:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 Processor -- start 2024-04-17T10:36:04.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 -- start start 2024-04-17T10:36:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eb806b1d0 0x7f8eb813cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb813ec90 0x7f8eb815dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb815e2a0 0x7f8eb813f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815cad0 con 0x7f8eb806b1d0 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815c7f0 con 0x7f8eb813ec90 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb815bc40 con 0x7f8eb815e2a0 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eb806b1d0 0x7f8eb813cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eb806b1d0 0x7f8eb813cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34508/0 (socket says 172.21.15.67:34508) 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 -- 172.21.15.67:0/4095464404 learned_addr learned my addr 172.21.15.67:0/4095464404 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebd000640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb813ec90 0x7f8eb815dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 -- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb815e2a0 msgr2=0x7f8eb813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebe400640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb815e2a0 0x7f8eb813f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb815e2a0 0x7f8eb813f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 -- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb813ec90 msgr2=0x7f8eb815dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb813ec90 0x7f8eb815dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 -- 172.21.15.67:0/4095464404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb81f6b30 con 0x7f8eb806b1d0 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebd000640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb813ec90 0x7f8eb815dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebe400640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb815e2a0 0x7f8eb813f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebda00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eb806b1d0 0x7f8eb813cf00 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00c480 tx=0x7f8eac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8eb5600640 1 -- 172.21.15.67:0/4095464404 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac017070 con 0x7f8eb806b1d0 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8eb5600640 1 -- 172.21.15.67:0/4095464404 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eac005cf0 con 0x7f8eb806b1d0 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8eb5600640 1 -- 172.21.15.67:0/4095464404 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac007290 con 0x7f8eb806b1d0 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb81f6dd0 con 0x7f8eb806b1d0 2024-04-17T10:36:04.944 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.951+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb81f7390 con 0x7f8eb806b1d0 2024-04-17T10:36:04.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.955+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb815e6b0 con 0x7f8eb806b1d0 2024-04-17T10:36:04.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.955+0000 7f8eb5600640 1 -- 172.21.15.67:0/4095464404 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8eac01a070 con 0x7f8eb806b1d0 2024-04-17T10:36:04.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.955+0000 7f8eb5600640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e98066f90 0x7f8e98069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:04.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.955+0000 7f8eb5600640 1 -- 172.21.15.67:0/4095464404 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f8eac084680 con 0x7f8eb806b1d0 2024-04-17T10:36:04.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.955+0000 7f8ebd000640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e98066f90 0x7f8e98069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:04.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.955+0000 7f8ebd000640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e98066f90 0x7f8e98069450 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40b2000 tx=0x7f8ea40aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:04.959+0000 7f8eb5600640 1 -- 172.21.15.67:0/4095464404 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8eb815e6b0 con 0x7f8eb806b1d0 2024-04-17T10:36:05.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.155+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f8eb804f780 con 0x7f8eb806b1d0 2024-04-17T10:36:05.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8eb5600640 1 -- 172.21.15.67:0/4095464404 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v106) v1 ==== 121+0+0 (secure 0 0 0) 0x7f8eac0554a0 con 0x7f8eb806b1d0 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e98066f90 msgr2=0x7f8e98069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e98066f90 0x7f8e98069450 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f8ea40b2000 tx=0x7f8ea40aa070 comp rx=0 tx=0).stop 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eb806b1d0 msgr2=0x7f8eb813cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eb806b1d0 0x7f8eb813cf00 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f8eac00c480 tx=0x7f8eac00c940 comp rx=0 tx=0).stop 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 shutdown_connections 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8e98066f90 0x7f8e98069450 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8eb815e2a0 0x7f8eb813f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8eb813ec90 0x7f8eb815dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 --2- 172.21.15.67:0/4095464404 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8eb806b1d0 0x7f8eb813cf00 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 >> 172.21.15.67:0/4095464404 conn(0x7f8eb80b2310 msgr2=0x7f8eb80b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 shutdown_connections 2024-04-17T10:36:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.775+0000 7f8ebee00640 1 -- 172.21.15.67:0/4095464404 wait complete. 2024-04-17T10:36:05.788 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:05.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.939+0000 7f3c08400640 1 Processor -- start 2024-04-17T10:36:05.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.939+0000 7f3c08400640 1 -- start start 2024-04-17T10:36:05.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c08400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c0006e6e0 0x7f3c000ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:05.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c08400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c001441f0 0x7f3c000ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:05.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c08400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0015ae40 con 0x7f3c0006eac0 2024-04-17T10:36:05.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c08400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c000cf430 con 0x7f3c001441f0 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c08400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c00071440 con 0x7f3c0006e6e0 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c001441f0 0x7f3c000ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c001441f0 0x7f3c000ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34520/0 (socket says 172.21.15.67:34520) 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 -- 172.21.15.67:0/2732756442 learned_addr learned my addr 172.21.15.67:0/2732756442 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c05a00640 1 --2- 172.21.15.67:0/2732756442 >> v2:172.21.15.86:3300/0 conn(0x7f3c0006e6e0 0x7f3c000ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 -- 172.21.15.67:0/2732756442 >> v2:172.21.15.86:3300/0 conn(0x7f3c0006e6e0 msgr2=0x7f3c000ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 --2- 172.21.15.67:0/2732756442 >> v2:172.21.15.86:3300/0 conn(0x7f3c0006e6e0 0x7f3c000ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 -- 172.21.15.67:0/2732756442 >> v1:172.21.15.86:6789/0 conn(0x7f3c0006eac0 legacy=0x7f3c00143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:05.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 -- 172.21.15.67:0/2732756442 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c00071720 con 0x7f3c001441f0 2024-04-17T10:36:05.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3c06400640 1 --2- 172.21.15.67:0/2732756442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000ceef0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f3c000cf620 tx=0x7f3bf801eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da122729534cfeac server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:05.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3bffe00640 1 -- 172.21.15.67:0/2732756442 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf800b2d0 con 0x7f3c001441f0 2024-04-17T10:36:05.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3bffe00640 1 -- 172.21.15.67:0/2732756442 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf800b470 con 0x7f3c001441f0 2024-04-17T10:36:05.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.943+0000 7f3bffe00640 1 -- 172.21.15.67:0/2732756442 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf80a7ae0 con 0x7f3c001441f0 2024-04-17T10:36:05.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- 172.21.15.67:0/2732756442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 msgr2=0x7f3c000ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:05.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 --2- 172.21.15.67:0/2732756442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000ceef0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f3c000cf620 tx=0x7f3bf801eaf0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- 172.21.15.67:0/2732756442 shutdown_connections 2024-04-17T10:36:05.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 --2- 172.21.15.67:0/2732756442 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000ceef0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 --2- 172.21.15.67:0/2732756442 >> v2:172.21.15.86:3300/0 conn(0x7f3c0006e6e0 0x7f3c000ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- 172.21.15.67:0/2732756442 >> 172.21.15.67:0/2732756442 conn(0x7f3c0006ac90 msgr2=0x7f3c0006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:05.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- 172.21.15.67:0/2732756442 shutdown_connections 2024-04-17T10:36:05.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- 172.21.15.67:0/2732756442 wait complete. 2024-04-17T10:36:05.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 Processor -- start 2024-04-17T10:36:05.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- start start 2024-04-17T10:36:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0006e6e0 0x7f3c000c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:05.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0006eac0 0x7f3c0013ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0006dd00 con 0x7f3c001441f0 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0006db80 con 0x7f3c0006e6e0 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c08400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0006de80 con 0x7f3c0006eac0 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34528/0 (socket says 172.21.15.67:34528) 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 -- 172.21.15.67:0/1178378779 learned_addr learned my addr 172.21.15.67:0/1178378779 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c05000640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0006eac0 0x7f3c0013ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c05a00640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0006e6e0 0x7f3c000c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 -- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0006eac0 msgr2=0x7f3c0013ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0006eac0 0x7f3c0013ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 -- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0006e6e0 msgr2=0x7f3c000c52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0006e6e0 0x7f3c000c52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:05.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c06400640 1 -- 172.21.15.67:0/1178378779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c001419f0 con 0x7f3c001441f0 2024-04-17T10:36:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.947+0000 7f3c05000640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0006eac0 0x7f3c0013ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:05.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3c06400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000acb00 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b1040 tx=0x7f3bf801b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3bfd600640 1 -- 172.21.15.67:0/1178378779 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8009800 con 0x7f3c001441f0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3bfd600640 1 -- 172.21.15.67:0/1178378779 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf80be070 con 0x7f3c001441f0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3bfd600640 1 -- 172.21.15.67:0/1178378779 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8009c20 con 0x7f3c001441f0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c000c4810 con 0x7f3c001441f0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0013b390 con 0x7f3c001441f0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c00072d50 con 0x7f3c001441f0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3bfd600640 1 -- 172.21.15.67:0/1178378779 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3bf80b6070 con 0x7f3c001441f0 2024-04-17T10:36:05.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3bfd600640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3be0066f60 0x7f3be0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3bfd600640 1 -- 172.21.15.67:0/1178378779 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f3bf8120150 con 0x7f3c001441f0 2024-04-17T10:36:05.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.951+0000 7f3c05a00640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3be0066f60 0x7f3be0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.955+0000 7f3c05a00640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3be0066f60 0x7f3be0069420 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400b9d0 tx=0x7f3bf4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:05.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:05.955+0000 7f3bfd600640 1 -- 172.21.15.67:0/1178378779 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3bf80187a0 con 0x7f3c001441f0 2024-04-17T10:36:06.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c00074e30 con 0x7f3c001441f0 2024-04-17T10:36:06.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3bfd600640 1 -- 172.21.15.67:0/1178378779 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3bf80f1090 con 0x7f3c001441f0 2024-04-17T10:36:06.144 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:06.144 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:06.145 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3be0066f60 msgr2=0x7f3be0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3be0066f60 0x7f3be0069420 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400b9d0 tx=0x7f3bf4008040 comp rx=0 tx=0).stop 2024-04-17T10:36:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 msgr2=0x7f3c000acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000acb00 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80b1040 tx=0x7f3bf801b890 comp rx=0 tx=0).stop 2024-04-17T10:36:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 shutdown_connections 2024-04-17T10:36:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3be0066f60 0x7f3be0069420 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:06.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c001441f0 0x7f3c000acb00 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c0006eac0 0x7f3c0013ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 --2- 172.21.15.67:0/1178378779 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c0006e6e0 0x7f3c000c52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 >> 172.21.15.67:0/1178378779 conn(0x7f3c0006ac90 msgr2=0x7f3c000b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 shutdown_connections 2024-04-17T10:36:06.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:06.155+0000 7f3c08400640 1 -- 172.21.15.67:0/1178378779 wait complete. 2024-04-17T10:36:06.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:06.568 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:07.569 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.723+0000 7f09d6000640 1 Processor -- start 2024-04-17T10:36:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.723+0000 7f09d6000640 1 -- start start 2024-04-17T10:36:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09d6000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cc250 0x7f09d00cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09d6000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f09d00cdcf0 0x7f09d00cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09d6000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d006b100 con 0x7f09d00cb260 2024-04-17T10:36:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09d6000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d00cd150 con 0x7f09d00cdcf0 2024-04-17T10:36:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09d6000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d006b3e0 con 0x7f09d00cc250 2024-04-17T10:36:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09c6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f09d00cb260 0x7f09d00cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44714/0 (socket says 172.21.15.67:44714) 2024-04-17T10:36:07.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09c6a00640 1 -- 172.21.15.67:0/3379701018 learned_addr learned my addr 172.21.15.67:0/3379701018 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:07.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cf400640 1 --2- 172.21.15.67:0/3379701018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cc250 0x7f09d00cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:07.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cfe00640 1 --2- 172.21.15.67:0/3379701018 >> v2:172.21.15.86:3300/0 conn(0x7f09d00cdcf0 0x7f09d00cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:07.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cea00640 1 -- 172.21.15.67:0/3379701018 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229604820 0 0) 0x7f09d006b100 con 0x7f09d00cb260 2024-04-17T10:36:07.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cea00640 1 -- 172.21.15.67:0/3379701018 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b4003530 con 0x7f09d00cb260 2024-04-17T10:36:07.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cf400640 1 -- 172.21.15.67:0/3379701018 >> v2:172.21.15.86:3300/0 conn(0x7f09d00cdcf0 msgr2=0x7f09d00cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:07.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cf400640 1 --2- 172.21.15.67:0/3379701018 >> v2:172.21.15.86:3300/0 conn(0x7f09d00cdcf0 0x7f09d00cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cf400640 1 -- 172.21.15.67:0/3379701018 >> v1:172.21.15.67:6789/0 conn(0x7f09d00cb260 legacy=0x7f09d00cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:07.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cf400640 1 -- 172.21.15.67:0/3379701018 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d006b6c0 con 0x7f09d00cc250 2024-04-17T10:36:07.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.727+0000 7f09cfe00640 1 --2- 172.21.15.67:0/3379701018 >> v2:172.21.15.86:3300/0 conn(0x7f09d00cdcf0 0x7f09d00cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:07.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09cf400640 1 --2- 172.21.15.67:0/3379701018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cc250 0x7f09d00cad20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c00097c0 tx=0x7f09c001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5c7c238a423c512 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:07.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09cea00640 1 -- 172.21.15.67:0/3379701018 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c00b0070 con 0x7f09d00cc250 2024-04-17T10:36:07.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09cea00640 1 -- 172.21.15.67:0/3379701018 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c00ab600 con 0x7f09d00cc250 2024-04-17T10:36:07.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09cea00640 1 -- 172.21.15.67:0/3379701018 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c00ab9b0 con 0x7f09d00cc250 2024-04-17T10:36:07.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 -- 172.21.15.67:0/3379701018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cc250 msgr2=0x7f09d00cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:07.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 --2- 172.21.15.67:0/3379701018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cc250 0x7f09d00cad20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c00097c0 tx=0x7f09c001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 -- 172.21.15.67:0/3379701018 shutdown_connections 2024-04-17T10:36:07.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 --2- 172.21.15.67:0/3379701018 >> v2:172.21.15.86:3300/0 conn(0x7f09d00cdcf0 0x7f09d00cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 --2- 172.21.15.67:0/3379701018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cc250 0x7f09d00cad20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 -- 172.21.15.67:0/3379701018 >> 172.21.15.67:0/3379701018 conn(0x7f09d00b22a0 msgr2=0x7f09d00b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:07.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 -- 172.21.15.67:0/3379701018 shutdown_connections 2024-04-17T10:36:07.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 -- 172.21.15.67:0/3379701018 wait complete. 2024-04-17T10:36:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 Processor -- start 2024-04-17T10:36:07.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 -- start start 2024-04-17T10:36:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.731+0000 7f09d6000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09d00cb260 0x7f09d015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09d00cc250 0x7f09d00ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cdcf0 0x7f09d00b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00d02e0 con 0x7f09d00cc250 2024-04-17T10:36:07.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00d0160 con 0x7f09d00cb260 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d016b100 con 0x7f09d00cdcf0 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09d00cc250 0x7f09d00ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09d00cc250 0x7f09d00ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34530/0 (socket says 172.21.15.67:34530) 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 -- 172.21.15.67:0/632846178 learned_addr learned my addr 172.21.15.67:0/632846178 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09cf400640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09d00cb260 0x7f09d015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09cfe00640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cdcf0 0x7f09d00b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 -- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cdcf0 msgr2=0x7f09d00b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cdcf0 0x7f09d00b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 -- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09d00cb260 msgr2=0x7f09d015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09d00cb260 0x7f09d015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 -- 172.21.15.67:0/632846178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d00c7b70 con 0x7f09d00cc250 2024-04-17T10:36:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c6a00640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09d00cc250 0x7f09d00ca110 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f09b800c450 tx=0x7f09b800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c7e00640 1 -- 172.21.15.67:0/632846178 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8017070 con 0x7f09d00cc250 2024-04-17T10:36:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c7e00640 1 -- 172.21.15.67:0/632846178 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b8005d10 con 0x7f09d00cc250 2024-04-17T10:36:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c7e00640 1 -- 172.21.15.67:0/632846178 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b8007310 con 0x7f09d00cc250 2024-04-17T10:36:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d00c7e50 con 0x7f09d00cc250 2024-04-17T10:36:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d00c8410 con 0x7f09d00cc250 2024-04-17T10:36:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d009e900 con 0x7f09d00cc250 2024-04-17T10:36:07.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.735+0000 7f09c7e00640 1 -- 172.21.15.67:0/632846178 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f09b801a070 con 0x7f09d00cc250 2024-04-17T10:36:07.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.739+0000 7f09c7e00640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f09ac066f60 0x7f09ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:07.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.739+0000 7f09c7e00640 1 -- 172.21.15.67:0/632846178 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f09b8084620 con 0x7f09d00cc250 2024-04-17T10:36:07.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.739+0000 7f09cf400640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f09ac066f60 0x7f09ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:07.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.739+0000 7f09cf400640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f09ac066f60 0x7f09ac069420 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f09c00ad000 tx=0x7f09c00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:07.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.743+0000 7f09c7e00640 1 -- 172.21.15.67:0/632846178 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f09b8054e20 con 0x7f09d00cc250 2024-04-17T10:36:07.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.934+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09d00b28d0 con 0x7f09d00cc250 2024-04-17T10:36:07.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.934+0000 7f09c7e00640 1 -- 172.21.15.67:0/632846178 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f09b8058ec0 con 0x7f09d00cc250 2024-04-17T10:36:07.926 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:07.926 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:07.926 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f09ac066f60 msgr2=0x7f09ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f09ac066f60 0x7f09ac069420 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f09c00ad000 tx=0x7f09c00a6000 comp rx=0 tx=0).stop 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09d00cc250 msgr2=0x7f09d00ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09d00cc250 0x7f09d00ca110 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f09b800c450 tx=0x7f09b800c850 comp rx=0 tx=0).stop 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 shutdown_connections 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f09ac066f60 0x7f09ac069420 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f09d00cdcf0 0x7f09d00b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09d00cc250 0x7f09d00ca110 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 --2- 172.21.15.67:0/632846178 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f09d00cb260 0x7f09d015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 >> 172.21.15.67:0/632846178 conn(0x7f09d00b22a0 msgr2=0x7f09d00b17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:07.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 shutdown_connections 2024-04-17T10:36:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:07.938+0000 7f09d6000640 1 -- 172.21.15.67:0/632846178 wait complete. 2024-04-17T10:36:08.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:08.349 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:09.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f6b400640 1 Processor -- start 2024-04-17T10:36:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f6b400640 1 -- start start 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f6b400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f6415cc00 0x7f7f6415f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f6b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f6415fa00 0x7f7f640c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f6b400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6415be60 con 0x7f7f640ce1b0 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f6b400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f64160230 con 0x7f7f6415cc00 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f6b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6415c120 con 0x7f7f6415fa00 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f5be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f6415cc00 0x7f7f6415f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f5be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f7f6415cc00 0x7f7f6415f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34536/0 (socket says 172.21.15.67:34536) 2024-04-17T10:36:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f5be00640 1 -- 172.21.15.67:0/1731364719 learned_addr learned my addr 172.21.15.67:0/1731364719 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f69400640 1 --2- 172.21.15.67:0/1731364719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f6415fa00 0x7f7f640c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f5be00640 1 -- 172.21.15.67:0/1731364719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f6415fa00 msgr2=0x7f7f640c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f5be00640 1 --2- 172.21.15.67:0/1731364719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f6415fa00 0x7f7f640c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f5be00640 1 -- 172.21.15.67:0/1731364719 >> v1:172.21.15.86:6789/0 conn(0x7f7f640ce1b0 legacy=0x7f7f640ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f5be00640 1 -- 172.21.15.67:0/1731364719 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6415c400 con 0x7f7f6415cc00 2024-04-17T10:36:09.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.510+0000 7f7f69400640 1 --2- 172.21.15.67:0/1731364719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f6415fa00 0x7f7f640c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:36:09.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f5be00640 1 --2- 172.21.15.67:0/1731364719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f6415f4c0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009870 tx=0x7f7f4c01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec3fe17efb19266 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f63e00640 1 -- 172.21.15.67:0/1731364719 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0b5070 con 0x7f7f6415cc00 2024-04-17T10:36:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f63e00640 1 -- 172.21.15.67:0/1731364719 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c0a89d0 con 0x7f7f6415cc00 2024-04-17T10:36:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f63e00640 1 -- 172.21.15.67:0/1731364719 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0a8da0 con 0x7f7f6415cc00 2024-04-17T10:36:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 -- 172.21.15.67:0/1731364719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 msgr2=0x7f7f6415f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1731364719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f6415f4c0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009870 tx=0x7f7f4c01ed10 comp rx=0 tx=0).stop 2024-04-17T10:36:09.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 -- 172.21.15.67:0/1731364719 shutdown_connections 2024-04-17T10:36:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1731364719 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f6415fa00 0x7f7f640c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1731364719 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f6415f4c0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 -- 172.21.15.67:0/1731364719 >> 172.21.15.67:0/1731364719 conn(0x7f7f640b33f0 msgr2=0x7f7f640b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 -- 172.21.15.67:0/1731364719 shutdown_connections 2024-04-17T10:36:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.514+0000 7f7f6b400640 1 -- 172.21.15.67:0/1731364719 wait complete. 2024-04-17T10:36:09.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 Processor -- start 2024-04-17T10:36:09.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 -- start start 2024-04-17T10:36:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f640ce1b0 0x7f7f6415a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f64158520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6415fa00 0x7f7f64149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6415fff0 con 0x7f7f6415cc00 2024-04-17T10:36:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6415fe70 con 0x7f7f6415fa00 2024-04-17T10:36:09.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f64159730 con 0x7f7f640ce1b0 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f64158520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f64158520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34552/0 (socket says 172.21.15.67:34552) 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 -- 172.21.15.67:0/1629153371 learned_addr learned my addr 172.21.15.67:0/1629153371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f68a00640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f640ce1b0 0x7f7f6415a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f69400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6415fa00 0x7f7f64149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 -- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f640ce1b0 msgr2=0x7f7f6415a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f640ce1b0 0x7f7f6415a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 -- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6415fa00 msgr2=0x7f7f64149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6415fa00 0x7f7f64149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 -- 172.21.15.67:0/1629153371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64155810 con 0x7f7f6415cc00 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f69400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6415fa00 0x7f7f64149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f5be00640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f64158520 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0b3040 tx=0x7f7f4c01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f61600640 1 -- 172.21.15.67:0/1629153371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0c0070 con 0x7f7f6415cc00 2024-04-17T10:36:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f61600640 1 -- 172.21.15.67:0/1629153371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c0b0b40 con 0x7f7f6415cc00 2024-04-17T10:36:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f61600640 1 -- 172.21.15.67:0/1629153371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c0b5070 con 0x7f7f6415cc00 2024-04-17T10:36:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f64155af0 con 0x7f7f6415cc00 2024-04-17T10:36:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.518+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6415ad00 con 0x7f7f6415cc00 2024-04-17T10:36:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.522+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f64070ab0 con 0x7f7f6415cc00 2024-04-17T10:36:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.522+0000 7f7f61600640 1 -- 172.21.15.67:0/1629153371 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7f4c0c3070 con 0x7f7f6415cc00 2024-04-17T10:36:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.522+0000 7f7f61600640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f40066f60 0x7f7f40069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.522+0000 7f7f61600640 1 -- 172.21.15.67:0/1629153371 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f7f4c122490 con 0x7f7f6415cc00 2024-04-17T10:36:09.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.522+0000 7f7f68a00640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f40066f60 0x7f7f40069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:09.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.522+0000 7f7f68a00640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f40066f60 0x7f7f40069420 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7f540098e0 tx=0x7f7f54009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:09.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.526+0000 7f7f61600640 1 -- 172.21.15.67:0/1629153371 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7f4c018870 con 0x7f7f6415cc00 2024-04-17T10:36:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.722+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f64155fa0 con 0x7f7f6415cc00 2024-04-17T10:36:09.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.722+0000 7f7f61600640 1 -- 172.21.15.67:0/1629153371 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f7f4c0f2bd0 con 0x7f7f6415cc00 2024-04-17T10:36:09.714 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:09.715 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:09.715 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f40066f60 msgr2=0x7f7f40069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f40066f60 0x7f7f40069420 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f7f540098e0 tx=0x7f7f54009060 comp rx=0 tx=0).stop 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 msgr2=0x7f7f64158520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f64158520 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0b3040 tx=0x7f7f4c01bab0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 shutdown_connections 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f40066f60 0x7f7f40069420 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f6415fa00 0x7f7f64149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f6415cc00 0x7f7f64158520 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 --2- 172.21.15.67:0/1629153371 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f640ce1b0 0x7f7f6415a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 >> 172.21.15.67:0/1629153371 conn(0x7f7f640b33f0 msgr2=0x7f7f6414ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:09.717 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 shutdown_connections 2024-04-17T10:36:09.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:09.726+0000 7f7f6b400640 1 -- 172.21.15.67:0/1629153371 wait complete. 2024-04-17T10:36:10.137 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:10.138 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:11.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:11.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 Processor -- start 2024-04-17T10:36:11.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 -- start start 2024-04-17T10:36:11.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:11.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f861c0cb260 0x7f861c0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f861c0ce2a0 0x7f861c0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861c0cd150 con 0x7f861c0cb260 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f861c06b100 con 0x7f861c0ce2a0 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8621800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c06b3e0 con 0x7f861c0cc250 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f861c0cb260 0x7f861c0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f861c0cb260 0x7f861c0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58486/0 (socket says 172.21.15.67:58486) 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 -- 172.21.15.67:0/3508650536 learned_addr learned my addr 172.21.15.67:0/3508650536 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f861aa00640 1 --2- 172.21.15.67:0/3508650536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f861b400640 1 --2- 172.21.15.67:0/3508650536 >> v2:172.21.15.86:3300/0 conn(0x7f861c0ce2a0 0x7f861c0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 -- 172.21.15.67:0/3508650536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 msgr2=0x7f861c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 --2- 172.21.15.67:0/3508650536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c0cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 -- 172.21.15.67:0/3508650536 >> v2:172.21.15.86:3300/0 conn(0x7f861c0ce2a0 msgr2=0x7f861c0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 --2- 172.21.15.67:0/3508650536 >> v2:172.21.15.86:3300/0 conn(0x7f861c0ce2a0 0x7f861c0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.302+0000 7f8612000640 1 -- 172.21.15.67:0/3508650536 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c06b6c0 con 0x7f861c0cb260 2024-04-17T10:36:11.296 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8612000640 1 --2- 172.21.15.67:0/3508650536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c0cd5e0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f861c1617e0 tx=0x7f860401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7aed36ccf0e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:11.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f861a000640 1 -- 172.21.15.67:0/3508650536 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86040b0070 con 0x7f861c0cb260 2024-04-17T10:36:11.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f861a000640 1 -- 172.21.15.67:0/3508650536 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86040ab600 con 0x7f861c0cb260 2024-04-17T10:36:11.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f861a000640 1 -- 172.21.15.67:0/3508650536 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86040ab9d0 con 0x7f861c0cb260 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 -- 172.21.15.67:0/3508650536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 msgr2=0x7f861c0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 --2- 172.21.15.67:0/3508650536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c0cd5e0 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f861c1617e0 tx=0x7f860401c520 comp rx=0 tx=0).stop 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 -- 172.21.15.67:0/3508650536 shutdown_connections 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 --2- 172.21.15.67:0/3508650536 >> v2:172.21.15.86:3300/0 conn(0x7f861c0ce2a0 0x7f861c0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 --2- 172.21.15.67:0/3508650536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c0cd5e0 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 --2- 172.21.15.67:0/3508650536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c0cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 -- 172.21.15.67:0/3508650536 >> 172.21.15.67:0/3508650536 conn(0x7f861c0b22a0 msgr2=0x7f861c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 -- 172.21.15.67:0/3508650536 shutdown_connections 2024-04-17T10:36:11.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.306+0000 7f8621800640 1 -- 172.21.15.67:0/3508650536 wait complete. 2024-04-17T10:36:11.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 Processor -- start 2024-04-17T10:36:11.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 -- start start 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c15e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f861c0ce2a0 0x7f861c15e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c06e850 con 0x7f861c0cb260 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f861c06e6d0 con 0x7f861c0ce2a0 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f861c06e9d0 con 0x7f861c0cc250 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58494/0 (socket says 172.21.15.67:58494) 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 -- 172.21.15.67:0/3245159342 learned_addr learned my addr 172.21.15.67:0/3245159342 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8612000640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c15e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861b400640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f861c0ce2a0 0x7f861c15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 -- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 msgr2=0x7f861c15e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:11.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c15e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 -- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f861c0ce2a0 msgr2=0x7f861c15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:11.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f861c0ce2a0 0x7f861c15e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 -- 172.21.15.67:0/3245159342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c155230 con 0x7f861c0cb260 2024-04-17T10:36:11.302 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f861aa00640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c151d00 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f860c00d830 tx=0x7f860c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:11.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8613400640 1 -- 172.21.15.67:0/3245159342 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c018020 con 0x7f861c0cb260 2024-04-17T10:36:11.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8613400640 1 -- 172.21.15.67:0/3245159342 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f860c0096a0 con 0x7f861c0cb260 2024-04-17T10:36:11.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f861c155510 con 0x7f861c0cb260 2024-04-17T10:36:11.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8613400640 1 -- 172.21.15.67:0/3245159342 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f860c009a50 con 0x7f861c0cb260 2024-04-17T10:36:11.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f861c155ad0 con 0x7f861c0cb260 2024-04-17T10:36:11.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.310+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f861c06cd30 con 0x7f861c0cb260 2024-04-17T10:36:11.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.314+0000 7f8613400640 1 -- 172.21.15.67:0/3245159342 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f860c01b020 con 0x7f861c0cb260 2024-04-17T10:36:11.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.314+0000 7f8613400640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f85f8066f60 0x7f85f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:11.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.314+0000 7f8613400640 1 -- 172.21.15.67:0/3245159342 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f860c085700 con 0x7f861c0cb260 2024-04-17T10:36:11.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.314+0000 7f8612000640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f85f8066f60 0x7f85f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:11.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.314+0000 7f8612000640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f85f8066f60 0x7f85f8069420 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f86040ad000 tx=0x7f8604008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:11.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.318+0000 7f8613400640 1 -- 172.21.15.67:0/3245159342 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f860c022020 con 0x7f861c0cb260 2024-04-17T10:36:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.514+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f861c074e30 con 0x7f861c0cb260 2024-04-17T10:36:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.514+0000 7f8613400640 1 -- 172.21.15.67:0/3245159342 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f860c055e40 con 0x7f861c0cb260 2024-04-17T10:36:11.506 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:11.506 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:11.506 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f85f8066f60 msgr2=0x7f85f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f85f8066f60 0x7f85f8069420 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f86040ad000 tx=0x7f8604008040 comp rx=0 tx=0).stop 2024-04-17T10:36:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 msgr2=0x7f861c151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c151d00 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f860c00d830 tx=0x7f860c00dcf0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 shutdown_connections 2024-04-17T10:36:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f85f8066f60 0x7f85f8069420 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f861c0ce2a0 0x7f861c15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f861c0cc250 0x7f861c15e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 --2- 172.21.15.67:0/3245159342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f861c0cb260 0x7f861c151d00 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 >> 172.21.15.67:0/3245159342 conn(0x7f861c0b22a0 msgr2=0x7f861c0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 shutdown_connections 2024-04-17T10:36:11.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:11.518+0000 7f8621800640 1 -- 172.21.15.67:0/3245159342 wait complete. 2024-04-17T10:36:11.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:11.929 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:12.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.094+0000 7faa79c00640 1 Processor -- start 2024-04-17T10:36:13.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.094+0000 7faa79c00640 1 -- start start 2024-04-17T10:36:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.094+0000 7faa79c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa74071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.094+0000 7faa79c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faa740cc300 0x7faa740cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.094+0000 7faa79c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7406e2f0 con 0x7faa74145040 2024-04-17T10:36:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.094+0000 7faa79c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa740a86c0 con 0x7faa740cc300 2024-04-17T10:36:13.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.094+0000 7faa79c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa74071cb0 con 0x7faa74071310 2024-04-17T10:36:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa74071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa73400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faa740cc300 0x7faa740cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa74071770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:47430/0 (socket says 172.21.15.67:47430) 2024-04-17T10:36:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 -- 172.21.15.67:0/2484611713 learned_addr learned my addr 172.21.15.67:0/2484611713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:13.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 -- 172.21.15.67:0/2484611713 >> v2:172.21.15.86:3300/0 conn(0x7faa740cc300 msgr2=0x7faa740cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 --2- 172.21.15.67:0/2484611713 >> v2:172.21.15.86:3300/0 conn(0x7faa740cc300 0x7faa740cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 -- 172.21.15.67:0/2484611713 >> v1:172.21.15.86:6789/0 conn(0x7faa74145040 legacy=0x7faa7406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 -- 172.21.15.67:0/2484611713 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7406d6e0 con 0x7faa74071310 2024-04-17T10:36:13.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa73400640 1 --2- 172.21.15.67:0/2484611713 >> v2:172.21.15.86:3300/0 conn(0x7faa740cc300 0x7faa740cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa6be00640 1 --2- 172.21.15.67:0/2484611713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa74071770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009870 tx=0x7faa5c01ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a92e41759305f64 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:13.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.098+0000 7faa72000640 1 -- 172.21.15.67:0/2484611713 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c0b5070 con 0x7faa74071310 2024-04-17T10:36:13.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa72000640 1 -- 172.21.15.67:0/2484611713 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5c0a8a50 con 0x7faa74071310 2024-04-17T10:36:13.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa72000640 1 -- 172.21.15.67:0/2484611713 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c0a8e20 con 0x7faa74071310 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- 172.21.15.67:0/2484611713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 msgr2=0x7faa74071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 --2- 172.21.15.67:0/2484611713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa74071770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faa5c009870 tx=0x7faa5c01ed50 comp rx=0 tx=0).stop 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- 172.21.15.67:0/2484611713 shutdown_connections 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 --2- 172.21.15.67:0/2484611713 >> v2:172.21.15.86:3300/0 conn(0x7faa740cc300 0x7faa740cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 --2- 172.21.15.67:0/2484611713 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa74071770 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- 172.21.15.67:0/2484611713 >> 172.21.15.67:0/2484611713 conn(0x7faa7406ac90 msgr2=0x7faa7406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- 172.21.15.67:0/2484611713 shutdown_connections 2024-04-17T10:36:13.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- 172.21.15.67:0/2484611713 wait complete. 2024-04-17T10:36:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 Processor -- start 2024-04-17T10:36:13.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- start start 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa7415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cc300 0x7faa741f3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa74145040 0x7faa741375c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7406dca0 con 0x7faa74145040 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7415bf10 con 0x7faa740cc300 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa79c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7406df80 con 0x7faa74071310 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa73400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa74145040 0x7faa741375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa73400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa74145040 0x7faa741375c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58500/0 (socket says 172.21.15.67:58500) 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa73400640 1 -- 172.21.15.67:0/2952500555 learned_addr learned my addr 172.21.15.67:0/2952500555 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa72a00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa7415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.102+0000 7faa6be00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cc300 0x7faa741f3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa73400640 1 -- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 msgr2=0x7faa7415a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:13.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa73400640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa7415a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa73400640 1 -- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cc300 msgr2=0x7faa741f3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa73400640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cc300 0x7faa741f3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa73400640 1 -- 172.21.15.67:0/2952500555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7415ad10 con 0x7faa74145040 2024-04-17T10:36:13.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa73400640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa74145040 0x7faa741375c0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00efc0 tx=0x7faa6c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa6aa00640 1 -- 172.21.15.67:0/2952500555 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c018030 con 0x7faa74145040 2024-04-17T10:36:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa6aa00640 1 -- 172.21.15.67:0/2952500555 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c00cde0 con 0x7faa74145040 2024-04-17T10:36:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa6aa00640 1 -- 172.21.15.67:0/2952500555 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c003bc0 con 0x7faa74145040 2024-04-17T10:36:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa740b2c50 con 0x7faa74145040 2024-04-17T10:36:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa740b3210 con 0x7faa74145040 2024-04-17T10:36:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa74072d50 con 0x7faa74145040 2024-04-17T10:36:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa6aa00640 1 -- 172.21.15.67:0/2952500555 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7faa6c01b070 con 0x7faa74145040 2024-04-17T10:36:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.106+0000 7faa6aa00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa50066f60 0x7faa50069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:13.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.110+0000 7faa6aa00640 1 -- 172.21.15.67:0/2952500555 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7faa6c085b00 con 0x7faa74145040 2024-04-17T10:36:13.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.110+0000 7faa72a00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa50066f60 0x7faa50069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:13.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.110+0000 7faa72a00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa50066f60 0x7faa50069420 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7faa64007fd0 tx=0x7faa64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:13.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.114+0000 7faa6aa00640 1 -- 172.21.15.67:0/2952500555 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7faa6c022070 con 0x7faa74145040 2024-04-17T10:36:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.306+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa74160070 con 0x7faa74145040 2024-04-17T10:36:13.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.306+0000 7faa6aa00640 1 -- 172.21.15.67:0/2952500555 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7faa6c085e20 con 0x7faa74145040 2024-04-17T10:36:13.299 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:13.299 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:13.299 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa50066f60 msgr2=0x7faa50069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa50066f60 0x7faa50069420 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7faa64007fd0 tx=0x7faa64009040 comp rx=0 tx=0).stop 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa74145040 msgr2=0x7faa741375c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa74145040 0x7faa741375c0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00efc0 tx=0x7faa6c009f90 comp rx=0 tx=0).stop 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 shutdown_connections 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faa50066f60 0x7faa50069420 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa74145040 0x7faa741375c0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faa740cc300 0x7faa741f3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 --2- 172.21.15.67:0/2952500555 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faa74071310 0x7faa7415a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 >> 172.21.15.67:0/2952500555 conn(0x7faa7406ac90 msgr2=0x7faa7415f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 shutdown_connections 2024-04-17T10:36:13.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:13.310+0000 7faa79c00640 1 -- 172.21.15.67:0/2952500555 wait complete. 2024-04-17T10:36:13.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:13.723 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:14.724 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:14.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.886+0000 7f4d1e000640 1 Processor -- start 2024-04-17T10:36:14.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.886+0000 7f4d1e000640 1 -- start start 2024-04-17T10:36:14.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.886+0000 7f4d1e000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4d1806c410 0x7f4d1806f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:14.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.886+0000 7f4d1e000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d18071c60 0x7f4d180720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:14.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.886+0000 7f4d1e000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d180ad8f0 con 0x7f4d1815dbc0 2024-04-17T10:36:14.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.886+0000 7f4d1e000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d1815dfa0 con 0x7f4d18071c60 2024-04-17T10:36:14.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.886+0000 7f4d1e000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d1806faa0 con 0x7f4d1806c410 2024-04-17T10:36:14.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4d1815dbc0 0x7f4d1806db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57976/0 (socket says 172.21.15.67:57976) 2024-04-17T10:36:14.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17400640 1 -- 172.21.15.67:0/2641460198 learned_addr learned my addr 172.21.15.67:0/2641460198 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:14.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4d18071c60 0x7f4d180720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:14.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d0ea00640 1 --2- 172.21.15.67:0/2641460198 >> v2:172.21.15.86:3300/0 conn(0x7f4d1806c410 0x7f4d1806f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:14.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17e00640 1 -- 172.21.15.67:0/2641460198 >> v2:172.21.15.86:3300/0 conn(0x7f4d1806c410 msgr2=0x7f4d1806f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:14.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17e00640 1 --2- 172.21.15.67:0/2641460198 >> v2:172.21.15.86:3300/0 conn(0x7f4d1806c410 0x7f4d1806f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:14.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17e00640 1 -- 172.21.15.67:0/2641460198 >> v1:172.21.15.67:6789/0 conn(0x7f4d1815dbc0 legacy=0x7f4d1806db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:14.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17e00640 1 -- 172.21.15.67:0/2641460198 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d180adbd0 con 0x7f4d18071c60 2024-04-17T10:36:14.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d0ea00640 1 --2- 172.21.15.67:0/2641460198 >> v2:172.21.15.86:3300/0 conn(0x7f4d1806c410 0x7f4d1806f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:14.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d17e00640 1 --2- 172.21.15.67:0/2641460198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d180720f0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f4d18071c10 tx=0x7f4d1001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6390366393bebc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:14.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d16a00640 1 -- 172.21.15.67:0/2641460198 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d100b2070 con 0x7f4d18071c60 2024-04-17T10:36:14.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d16a00640 1 -- 172.21.15.67:0/2641460198 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d100a5760 con 0x7f4d18071c60 2024-04-17T10:36:14.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.890+0000 7f4d16a00640 1 -- 172.21.15.67:0/2641460198 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d100a5b30 con 0x7f4d18071c60 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- 172.21.15.67:0/2641460198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 msgr2=0x7f4d180720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 --2- 172.21.15.67:0/2641460198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d180720f0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f4d18071c10 tx=0x7f4d1001c8a0 comp rx=0 tx=0).stop 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- 172.21.15.67:0/2641460198 shutdown_connections 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 --2- 172.21.15.67:0/2641460198 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d180720f0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 --2- 172.21.15.67:0/2641460198 >> v2:172.21.15.86:3300/0 conn(0x7f4d1806c410 0x7f4d1806f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- 172.21.15.67:0/2641460198 >> 172.21.15.67:0/2641460198 conn(0x7f4d180af670 msgr2=0x7f4d180b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- 172.21.15.67:0/2641460198 shutdown_connections 2024-04-17T10:36:14.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- 172.21.15.67:0/2641460198 wait complete. 2024-04-17T10:36:14.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 Processor -- start 2024-04-17T10:36:14.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- start start 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d1806c410 0x7f4d18160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d1813a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1815dbc0 0x7f4d180b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1816a100 con 0x7f4d18071c60 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1806af70 con 0x7f4d1806c410 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d1e000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1816a280 con 0x7f4d1815dbc0 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d1813a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:14.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d1813a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58524/0 (socket says 172.21.15.67:58524) 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 -- 172.21.15.67:0/3095628460 learned_addr learned my addr 172.21.15.67:0/3095628460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d17400640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d1806c410 0x7f4d18160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d17e00640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1815dbc0 0x7f4d180b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 -- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1815dbc0 msgr2=0x7f4d180b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1815dbc0 0x7f4d180b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 -- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d1806c410 msgr2=0x7f4d18160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d1806c410 0x7f4d18160960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.894+0000 7f4d0ea00640 1 -- 172.21.15.67:0/3095628460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1815f8c0 con 0x7f4d18071c60 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d0ea00640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d1813a920 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000d800 tx=0x7f4d0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d0fe00640 1 -- 172.21.15.67:0/3095628460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00018020 con 0x7f4d18071c60 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d0fe00640 1 -- 172.21.15.67:0/3095628460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d000096c0 con 0x7f4d18071c60 2024-04-17T10:36:14.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1815fba0 con 0x7f4d18071c60 2024-04-17T10:36:14.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d0fe00640 1 -- 172.21.15.67:0/3095628460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d00009ab0 con 0x7f4d18071c60 2024-04-17T10:36:14.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d18160160 con 0x7f4d18071c60 2024-04-17T10:36:14.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1809e260 con 0x7f4d18071c60 2024-04-17T10:36:14.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d0fe00640 1 -- 172.21.15.67:0/3095628460 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4d0001b020 con 0x7f4d18071c60 2024-04-17T10:36:14.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d0fe00640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:14.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d0fe00640 1 -- 172.21.15.67:0/3095628460 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4d00085640 con 0x7f4d18071c60 2024-04-17T10:36:14.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.898+0000 7f4d17400640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:14.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.902+0000 7f4d17400640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000f70 tx=0x7f4d08007e90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:14.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:14.902+0000 7f4d0fe00640 1 -- 172.21.15.67:0/3095628460 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4d00022020 con 0x7f4d18071c60 2024-04-17T10:36:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d1806df10 con 0x7f4d18071c60 2024-04-17T10:36:15.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d0fe00640 1 -- 172.21.15.67:0/3095628460 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f4d00055e10 con 0x7f4d18071c60 2024-04-17T10:36:15.088 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:15.088 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 msgr2=0x7f4cf4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4d08000f70 tx=0x7f4d08007e90 comp rx=0 tx=0).stop 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 msgr2=0x7f4d1813a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d1813a920 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f4d0000d800 tx=0x7f4d0000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 shutdown_connections 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4cf4066f60 0x7f4cf4069420 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4d1815dbc0 0x7f4d180b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:15.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4d18071c60 0x7f4d1813a920 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:15.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 --2- 172.21.15.67:0/3095628460 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4d1806c410 0x7f4d18160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:15.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 >> 172.21.15.67:0/3095628460 conn(0x7f4d180af670 msgr2=0x7f4d180b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:15.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 shutdown_connections 2024-04-17T10:36:15.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:15.098+0000 7f4d1e000640 1 -- 172.21.15.67:0/3095628460 wait complete. 2024-04-17T10:36:15.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:15.512 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:16.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:16.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.674+0000 7fcf42200640 1 Processor -- start 2024-04-17T10:36:16.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.674+0000 7fcf42200640 1 -- start start 2024-04-17T10:36:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf42200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcf3c1465a0 0x7fcf3c0cd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf42200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf3c147220 0x7fcf3c071210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf42200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c06d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf42200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3c0cdc80 con 0x7fcf3c147220 2024-04-17T10:36:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf42200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3c0cdf60 con 0x7fcf3c1465a0 2024-04-17T10:36:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf42200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c15e1e0 con 0x7fcf3c071750 2024-04-17T10:36:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf3c147220 0x7fcf3c071210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcf3c147220 0x7fcf3c071210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58532/0 (socket says 172.21.15.67:58532) 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 -- 172.21.15.67:0/2100575613 learned_addr learned my addr 172.21.15.67:0/2100575613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf40e00640 1 --2- 172.21.15.67:0/2100575613 >> v2:172.21.15.86:3300/0 conn(0x7fcf3c1465a0 0x7fcf3c0cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf41800640 1 --2- 172.21.15.67:0/2100575613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c06d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 -- 172.21.15.67:0/2100575613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 msgr2=0x7fcf3c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 --2- 172.21.15.67:0/2100575613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c06d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 -- 172.21.15.67:0/2100575613 >> v2:172.21.15.86:3300/0 conn(0x7fcf3c1465a0 msgr2=0x7fcf3c0cd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 --2- 172.21.15.67:0/2100575613 >> v2:172.21.15.86:3300/0 conn(0x7fcf3c1465a0 0x7fcf3c0cd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.678+0000 7fcf33e00640 1 -- 172.21.15.67:0/2100575613 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c0cd970 con 0x7fcf3c147220 2024-04-17T10:36:16.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf33e00640 1 --2- 172.21.15.67:0/2100575613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c147220 0x7fcf3c071210 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c06f320 tx=0x7fcf2401c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fad08c3f3aa1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:16.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf3be00640 1 -- 172.21.15.67:0/2100575613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf240b0070 con 0x7fcf3c147220 2024-04-17T10:36:16.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf3be00640 1 -- 172.21.15.67:0/2100575613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf240ab600 con 0x7fcf3c147220 2024-04-17T10:36:16.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf3be00640 1 -- 172.21.15.67:0/2100575613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf240ab9d0 con 0x7fcf3c147220 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 -- 172.21.15.67:0/2100575613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c147220 msgr2=0x7fcf3c071210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 --2- 172.21.15.67:0/2100575613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c147220 0x7fcf3c071210 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c06f320 tx=0x7fcf2401c260 comp rx=0 tx=0).stop 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 -- 172.21.15.67:0/2100575613 shutdown_connections 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 --2- 172.21.15.67:0/2100575613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c06d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 --2- 172.21.15.67:0/2100575613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c147220 0x7fcf3c071210 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 --2- 172.21.15.67:0/2100575613 >> v2:172.21.15.86:3300/0 conn(0x7fcf3c1465a0 0x7fcf3c0cd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 -- 172.21.15.67:0/2100575613 >> 172.21.15.67:0/2100575613 conn(0x7fcf3c067070 msgr2=0x7fcf3c06ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 -- 172.21.15.67:0/2100575613 shutdown_connections 2024-04-17T10:36:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 -- 172.21.15.67:0/2100575613 wait complete. 2024-04-17T10:36:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 Processor -- start 2024-04-17T10:36:16.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.682+0000 7fcf42200640 1 -- start start 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c1392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c1465a0 0x7fcf3c15c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf3c147220 0x7fcf3c141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c15f3e0 con 0x7fcf3c1465a0 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c146b90 con 0x7fcf3c147220 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3c15f560 con 0x7fcf3c071750 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c1465a0 0x7fcf3c15c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c1465a0 0x7fcf3c15c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58540/0 (socket says 172.21.15.67:58540) 2024-04-17T10:36:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 -- 172.21.15.67:0/1265036373 learned_addr learned my addr 172.21.15.67:0/1265036373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:16.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf40e00640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c1392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:16.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf41800640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf3c147220 0x7fcf3c141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:16.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 -- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 msgr2=0x7fcf3c1392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:16.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c1392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 -- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf3c147220 msgr2=0x7fcf3c141b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:16.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf3c147220 0x7fcf3c141b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 -- 172.21.15.67:0/1265036373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c139890 con 0x7fcf3c1465a0 2024-04-17T10:36:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf33e00640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c1465a0 0x7fcf3c15c7f0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fcf240ae040 tx=0x7fcf24019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf39600640 1 -- 172.21.15.67:0/1265036373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf240ba020 con 0x7fcf3c1465a0 2024-04-17T10:36:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf39600640 1 -- 172.21.15.67:0/1265036373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf240b6640 con 0x7fcf3c1465a0 2024-04-17T10:36:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf39600640 1 -- 172.21.15.67:0/1265036373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf240b0040 con 0x7fcf3c1465a0 2024-04-17T10:36:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3c136920 con 0x7fcf3c1465a0 2024-04-17T10:36:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3c136ee0 con 0x7fcf3c1465a0 2024-04-17T10:36:16.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.686+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf3c1370e0 con 0x7fcf3c1465a0 2024-04-17T10:36:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.690+0000 7fcf39600640 1 -- 172.21.15.67:0/1265036373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcf240bd020 con 0x7fcf3c1465a0 2024-04-17T10:36:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.690+0000 7fcf39600640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf18066f60 0x7fcf18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.690+0000 7fcf39600640 1 -- 172.21.15.67:0/1265036373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcf2411f4b0 con 0x7fcf3c1465a0 2024-04-17T10:36:16.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.690+0000 7fcf40e00640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf18066f60 0x7fcf18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:16.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.690+0000 7fcf40e00640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf18066f60 0x7fcf18069420 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009b60 tx=0x7fcf2c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:16.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.694+0000 7fcf39600640 1 -- 172.21.15.67:0/1265036373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcf3c1370e0 con 0x7fcf3c1465a0 2024-04-17T10:36:16.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.890+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf3c06b710 con 0x7fcf3c1465a0 2024-04-17T10:36:16.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf39600640 1 -- 172.21.15.67:0/1265036373 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fcf240c4030 con 0x7fcf3c1465a0 2024-04-17T10:36:16.883 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:16.883 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:16.883 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf18066f60 msgr2=0x7fcf18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf18066f60 0x7fcf18069420 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009b60 tx=0x7fcf2c009190 comp rx=0 tx=0).stop 2024-04-17T10:36:16.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c1465a0 msgr2=0x7fcf3c15c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c1465a0 0x7fcf3c15c7f0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fcf240ae040 tx=0x7fcf24019000 comp rx=0 tx=0).stop 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 shutdown_connections 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcf18066f60 0x7fcf18069420 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcf3c147220 0x7fcf3c141b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcf3c1465a0 0x7fcf3c15c7f0 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 --2- 172.21.15.67:0/1265036373 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcf3c071750 0x7fcf3c1392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 >> 172.21.15.67:0/1265036373 conn(0x7fcf3c067070 msgr2=0x7fcf3c0b36d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 shutdown_connections 2024-04-17T10:36:16.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:16.894+0000 7fcf42200640 1 -- 172.21.15.67:0/1265036373 wait complete. 2024-04-17T10:36:17.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:17.307 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:18.308 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:18.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcc1200640 1 Processor -- start 2024-04-17T10:36:18.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcc1200640 1 -- start start 2024-04-17T10:36:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcc1200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7efcbc1466e0 0x7efcbc06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcc1200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc071a20 0x7efcbc0adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcc1200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcbc161740 con 0x7efcbc146da0 2024-04-17T10:36:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcc1200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcbc161a20 con 0x7efcbc1466e0 2024-04-17T10:36:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcc1200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc161d00 con 0x7efcbc071a20 2024-04-17T10:36:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcb3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efcbc146da0 0x7efcbc071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57990/0 (socket says 172.21.15.67:57990) 2024-04-17T10:36:18.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcb3e00640 1 -- 172.21.15.67:0/443600258 learned_addr learned my addr 172.21.15.67:0/443600258 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcb9e00640 1 --2- 172.21.15.67:0/443600258 >> v2:172.21.15.86:3300/0 conn(0x7efcbc1466e0 0x7efcbc06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:18.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcba800640 1 --2- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc071a20 0x7efcbc0adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcb9400640 1 -- 172.21.15.67:0/443600258 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418981166 0 0) 0x7efcbc161740 con 0x7efcbc146da0 2024-04-17T10:36:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.470+0000 7efcb9400640 1 -- 172.21.15.67:0/443600258 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efca0003530 con 0x7efcbc146da0 2024-04-17T10:36:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9e00640 1 -- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc071a20 msgr2=0x7efcbc0adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9e00640 1 --2- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc071a20 0x7efcbc0adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9e00640 1 -- 172.21.15.67:0/443600258 >> v1:172.21.15.67:6789/0 conn(0x7efcbc146da0 legacy=0x7efcbc071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:18.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9e00640 1 -- 172.21.15.67:0/443600258 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc161fe0 con 0x7efcbc1466e0 2024-04-17T10:36:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9e00640 1 --2- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc1466e0 0x7efcbc06eb70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efcac0097c0 tx=0x7efcac01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=425161e0d09d7704 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9400640 1 -- 172.21.15.67:0/443600258 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0b0070 con 0x7efcbc1466e0 2024-04-17T10:36:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9400640 1 -- 172.21.15.67:0/443600258 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac0ab600 con 0x7efcbc1466e0 2024-04-17T10:36:18.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcb9400640 1 -- 172.21.15.67:0/443600258 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0ab9b0 con 0x7efcbc1466e0 2024-04-17T10:36:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcc1200640 1 -- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc1466e0 msgr2=0x7efcbc06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcc1200640 1 --2- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc1466e0 0x7efcbc06eb70 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efcac0097c0 tx=0x7efcac01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcc1200640 1 -- 172.21.15.67:0/443600258 shutdown_connections 2024-04-17T10:36:18.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcc1200640 1 --2- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc071a20 0x7efcbc0adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.474+0000 7efcc1200640 1 --2- 172.21.15.67:0/443600258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc1466e0 0x7efcbc06eb70 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 -- 172.21.15.67:0/443600258 >> 172.21.15.67:0/443600258 conn(0x7efcbc06ac90 msgr2=0x7efcbc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 -- 172.21.15.67:0/443600258 shutdown_connections 2024-04-17T10:36:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 -- 172.21.15.67:0/443600258 wait complete. 2024-04-17T10:36:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 Processor -- start 2024-04-17T10:36:18.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 -- start start 2024-04-17T10:36:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc071a20 0x7efcbc15d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1466e0 0x7efcbc1f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc146da0 0x7efcbc0b0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0aeec0 con 0x7efcbc071a20 2024-04-17T10:36:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0aed40 con 0x7efcbc146da0 2024-04-17T10:36:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcc1200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc0af040 con 0x7efcbc1466e0 2024-04-17T10:36:18.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc071a20 0x7efcbc15d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc071a20 0x7efcbc15d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58548/0 (socket says 172.21.15.67:58548) 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 -- 172.21.15.67:0/630104678 learned_addr learned my addr 172.21.15.67:0/630104678 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb3e00640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1466e0 0x7efcbc1f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcba800640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc146da0 0x7efcbc0b0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 -- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1466e0 msgr2=0x7efcbc1f76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1466e0 0x7efcbc1f76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 -- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc146da0 msgr2=0x7efcbc0b0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc146da0 0x7efcbc0b0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 -- 172.21.15.67:0/630104678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc1425e0 con 0x7efcbc071a20 2024-04-17T10:36:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb3e00640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1466e0 0x7efcbc1f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcba800640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc146da0 0x7efcbc0b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb9e00640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc071a20 0x7efcbc15d170 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7efcac000c10 tx=0x7efcac019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.478+0000 7efcb2000640 1 -- 172.21.15.67:0/630104678 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0ba070 con 0x7efcbc071a20 2024-04-17T10:36:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcb2000640 1 -- 172.21.15.67:0/630104678 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcac0ab9b0 con 0x7efcbc071a20 2024-04-17T10:36:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcb2000640 1 -- 172.21.15.67:0/630104678 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac0b0040 con 0x7efcbc071a20 2024-04-17T10:36:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc1428c0 con 0x7efcbc071a20 2024-04-17T10:36:18.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc142e80 con 0x7efcbc071a20 2024-04-17T10:36:18.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc072d50 con 0x7efcbc071a20 2024-04-17T10:36:18.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcb2000640 1 -- 172.21.15.67:0/630104678 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7efcac0bd070 con 0x7efcbc071a20 2024-04-17T10:36:18.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcb2000640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efc98066f60 0x7efc98069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:18.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcb2000640 1 -- 172.21.15.67:0/630104678 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7efcac11e490 con 0x7efcbc071a20 2024-04-17T10:36:18.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.482+0000 7efcb3e00640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efc98066f60 0x7efc98069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:18.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.486+0000 7efcb3e00640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efc98066f60 0x7efc98069420 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7efcbc15f200 tx=0x7efca4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:18.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.486+0000 7efcb2000640 1 -- 172.21.15.67:0/630104678 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7efcac016090 con 0x7efcbc071a20 2024-04-17T10:36:18.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcbc06e6b0 con 0x7efcbc071a20 2024-04-17T10:36:18.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcb2000640 1 -- 172.21.15.67:0/630104678 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7efcac0ef2b0 con 0x7efcbc071a20 2024-04-17T10:36:18.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:18.673 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:18.673 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efc98066f60 msgr2=0x7efc98069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efc98066f60 0x7efc98069420 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7efcbc15f200 tx=0x7efca4009040 comp rx=0 tx=0).stop 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc071a20 msgr2=0x7efcbc15d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc071a20 0x7efcbc15d170 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7efcac000c10 tx=0x7efcac019240 comp rx=0 tx=0).stop 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 shutdown_connections 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7efc98066f60 0x7efc98069420 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7efcbc146da0 0x7efcbc0b0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7efcbc1466e0 0x7efcbc1f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 --2- 172.21.15.67:0/630104678 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efcbc071a20 0x7efcbc15d170 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 >> 172.21.15.67:0/630104678 conn(0x7efcbc06ac90 msgr2=0x7efcbc071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:18.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 shutdown_connections 2024-04-17T10:36:18.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:18.682+0000 7efcc1200640 1 -- 172.21.15.67:0/630104678 wait complete. 2024-04-17T10:36:19.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:19.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:20.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65c800640 1 Processor -- start 2024-04-17T10:36:20.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65c800640 1 -- start start 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65415a060 0x7fb65415e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65c800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6540ad530 0x7fb6540ad940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65c800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6540ade80 con 0x7fb65406c3c0 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65c800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6540ad0a0 con 0x7fb6540ad530 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65406e740 con 0x7fb65415a060 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb65a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb6540ad530 0x7fb6540ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb659400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb65406c3c0 0x7fb6540b34c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57992/0 (socket says 172.21.15.67:57992) 2024-04-17T10:36:20.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb659400640 1 -- 172.21.15.67:0/379029728 learned_addr learned my addr 172.21.15.67:0/379029728 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.258+0000 7fb659e00640 1 --2- 172.21.15.67:0/379029728 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65415a060 0x7fb65415e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb65a800640 1 -- 172.21.15.67:0/379029728 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65415a060 msgr2=0x7fb65415e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb65a800640 1 --2- 172.21.15.67:0/379029728 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65415a060 0x7fb65415e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb65a800640 1 -- 172.21.15.67:0/379029728 >> v1:172.21.15.67:6789/0 conn(0x7fb65406c3c0 legacy=0x7fb6540b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb65a800640 1 -- 172.21.15.67:0/379029728 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65406ea20 con 0x7fb6540ad530 2024-04-17T10:36:20.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb658a00640 1 -- 172.21.15.67:0/379029728 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834280177 0 0) 0x7fb6540ade80 con 0x7fb65406c3c0 2024-04-17T10:36:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb65a800640 1 --2- 172.21.15.67:0/379029728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb6540ad940 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fb650009d90 tx=0x7fb65001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb8e131f2b07e40 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb658a00640 1 -- 172.21.15.67:0/379029728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500b2070 con 0x7fb6540ad530 2024-04-17T10:36:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb658a00640 1 -- 172.21.15.67:0/379029728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6500a5830 con 0x7fb6540ad530 2024-04-17T10:36:20.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb658a00640 1 -- 172.21.15.67:0/379029728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6500a5c00 con 0x7fb6540ad530 2024-04-17T10:36:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb65c800640 1 -- 172.21.15.67:0/379029728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 msgr2=0x7fb6540ad940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.262+0000 7fb65c800640 1 --2- 172.21.15.67:0/379029728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb6540ad940 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fb650009d90 tx=0x7fb65001c8e0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- 172.21.15.67:0/379029728 shutdown_connections 2024-04-17T10:36:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 --2- 172.21.15.67:0/379029728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb6540ad940 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 --2- 172.21.15.67:0/379029728 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65415a060 0x7fb65415e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- 172.21.15.67:0/379029728 >> 172.21.15.67:0/379029728 conn(0x7fb6540af670 msgr2=0x7fb6540b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- 172.21.15.67:0/379029728 shutdown_connections 2024-04-17T10:36:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- 172.21.15.67:0/379029728 wait complete. 2024-04-17T10:36:20.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 Processor -- start 2024-04-17T10:36:20.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- start start 2024-04-17T10:36:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65406c3c0 0x7fb654158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb654152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:20.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb65415a060 0x7fb6541523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6540b3d50 con 0x7fb6540ad530 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6540b3bd0 con 0x7fb65415a060 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb654071c60 con 0x7fb65406c3c0 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb654152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb654152c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58574/0 (socket says 172.21.15.67:58574) 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65406c3c0 0x7fb654158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 -- 172.21.15.67:0/3130383817 learned_addr learned my addr 172.21.15.67:0/3130383817 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb65a800640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb65415a060 0x7fb6541523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 -- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65406c3c0 msgr2=0x7fb654158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65406c3c0 0x7fb654158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 -- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb65415a060 msgr2=0x7fb6541523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb65415a060 0x7fb6541523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 -- 172.21.15.67:0/3130383817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6541575d0 con 0x7fb6540ad530 2024-04-17T10:36:20.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.266+0000 7fb659400640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb654152c70 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fb6440011f0 tx=0x7fb64400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb64e000640 1 -- 172.21.15.67:0/3130383817 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644018070 con 0x7fb6540ad530 2024-04-17T10:36:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb64e000640 1 -- 172.21.15.67:0/3130383817 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6440074e0 con 0x7fb6540ad530 2024-04-17T10:36:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb64e000640 1 -- 172.21.15.67:0/3130383817 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644004a10 con 0x7fb6540ad530 2024-04-17T10:36:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb654156810 con 0x7fb6540ad530 2024-04-17T10:36:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb654156dd0 con 0x7fb6540ad530 2024-04-17T10:36:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb65409e260 con 0x7fb6540ad530 2024-04-17T10:36:20.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb64e000640 1 -- 172.21.15.67:0/3130383817 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb644004bb0 con 0x7fb6540ad530 2024-04-17T10:36:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb64e000640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb638066f60 0x7fb638069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb64e000640 1 -- 172.21.15.67:0/3130383817 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fb644084560 con 0x7fb6540ad530 2024-04-17T10:36:20.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.270+0000 7fb659e00640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb638066f60 0x7fb638069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:20.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.274+0000 7fb659e00640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb638066f60 0x7fb638069420 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb648009ac0 tx=0x7fb648009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:20.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.274+0000 7fb64e000640 1 -- 172.21.15.67:0/3130383817 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb644054d30 con 0x7fb6540ad530 2024-04-17T10:36:20.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.470+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6540b5400 con 0x7fb6540ad530 2024-04-17T10:36:20.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.470+0000 7fb64e000640 1 -- 172.21.15.67:0/3130383817 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fb64405b080 con 0x7fb6540ad530 2024-04-17T10:36:20.461 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:20.461 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:20.462 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb638066f60 msgr2=0x7fb638069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb638066f60 0x7fb638069420 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fb648009ac0 tx=0x7fb648009190 comp rx=0 tx=0).stop 2024-04-17T10:36:20.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 msgr2=0x7fb654152c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb654152c70 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fb6440011f0 tx=0x7fb64400c9b0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 shutdown_connections 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb638066f60 0x7fb638069420 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb65415a060 0x7fb6541523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb6540ad530 0x7fb654152c70 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 --2- 172.21.15.67:0/3130383817 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb65406c3c0 0x7fb654158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 >> 172.21.15.67:0/3130383817 conn(0x7fb6540af670 msgr2=0x7fb6540cfd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 shutdown_connections 2024-04-17T10:36:20.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:20.474+0000 7fb65c800640 1 -- 172.21.15.67:0/3130383817 wait complete. 2024-04-17T10:36:20.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:20.886 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:21.886 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f60c00640 1 Processor -- start 2024-04-17T10:36:22.041 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f60c00640 1 -- start start 2024-04-17T10:36:22.042 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f60c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c071770 0x7f1f5c0cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f60c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5c06e2f0 con 0x7f1f5c071310 2024-04-17T10:36:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f60c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f5c06d6e0 con 0x7f1f5c145040 2024-04-17T10:36:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f60c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c0a86c0 con 0x7f1f5c071770 2024-04-17T10:36:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f51400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1f5c071310 0x7f1f5c0cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59420/0 (socket says 172.21.15.67:59420) 2024-04-17T10:36:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.050+0000 7f1f51400640 1 -- 172.21.15.67:0/2983380248 learned_addr learned my addr 172.21.15.67:0/2983380248 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:22.043 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f5a800640 1 --2- 172.21.15.67:0/2983380248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c071770 0x7f1f5c0cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:22.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f59400640 1 -- 172.21.15.67:0/2983380248 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841837329 0 0) 0x7f1f5c06e2f0 con 0x7f1f5c071310 2024-04-17T10:36:22.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f59400640 1 -- 172.21.15.67:0/2983380248 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f40003530 con 0x7f1f5c071310 2024-04-17T10:36:22.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f59400640 1 -- 172.21.15.67:0/2983380248 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872180039 0 0) 0x7f1f5c06d6e0 con 0x7f1f5c145040 2024-04-17T10:36:22.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f5a800640 1 -- 172.21.15.67:0/2983380248 >> v1:172.21.15.86:6789/0 conn(0x7f1f5c145040 legacy=0x7f1f5c06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:22.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f5a800640 1 -- 172.21.15.67:0/2983380248 >> v1:172.21.15.67:6789/0 conn(0x7f1f5c071310 legacy=0x7f1f5c0cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:22.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f5a800640 1 -- 172.21.15.67:0/2983380248 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c06d9c0 con 0x7f1f5c071770 2024-04-17T10:36:22.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f5a800640 1 --2- 172.21.15.67:0/2983380248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c071770 0x7f1f5c0cca10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009dc0 tx=0x7f1f5401f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48b17237d0c2113 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:22.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f59400640 1 -- 172.21.15.67:0/2983380248 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f540b4070 con 0x7f1f5c071770 2024-04-17T10:36:22.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f59400640 1 -- 172.21.15.67:0/2983380248 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f540af600 con 0x7f1f5c071770 2024-04-17T10:36:22.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.054+0000 7f1f59400640 1 -- 172.21.15.67:0/2983380248 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f540af9d0 con 0x7f1f5c071770 2024-04-17T10:36:22.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- 172.21.15.67:0/2983380248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c071770 msgr2=0x7f1f5c0cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:22.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 --2- 172.21.15.67:0/2983380248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c071770 0x7f1f5c0cca10 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f54009dc0 tx=0x7f1f5401f170 comp rx=0 tx=0).stop 2024-04-17T10:36:22.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- 172.21.15.67:0/2983380248 shutdown_connections 2024-04-17T10:36:22.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 --2- 172.21.15.67:0/2983380248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c071770 0x7f1f5c0cca10 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:22.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- 172.21.15.67:0/2983380248 >> 172.21.15.67:0/2983380248 conn(0x7f1f5c06ac90 msgr2=0x7f1f5c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:22.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- 172.21.15.67:0/2983380248 shutdown_connections 2024-04-17T10:36:22.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- 172.21.15.67:0/2983380248 wait complete. 2024-04-17T10:36:22.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 Processor -- start 2024-04-17T10:36:22.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- start start 2024-04-17T10:36:22.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f5c071310 0x7f1f5c15b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:22.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f5c071770 0x7f1f5c140e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:22.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c145040 0x7f1f5c1f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:22.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c06df80 con 0x7f1f5c071770 2024-04-17T10:36:22.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c15bf10 con 0x7f1f5c071310 2024-04-17T10:36:22.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f60c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c071ba0 con 0x7f1f5c145040 2024-04-17T10:36:22.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f51400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f5c071770 0x7f1f5c140e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:22.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f51400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f5c071770 0x7f1f5c140e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33498/0 (socket says 172.21.15.67:33498) 2024-04-17T10:36:22.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f51400640 1 -- 172.21.15.67:0/3045670754 learned_addr learned my addr 172.21.15.67:0/3045670754 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:22.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f5a800640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c145040 0x7f1f5c1f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:22.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.058+0000 7f1f59e00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f5c071310 0x7f1f5c15b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:22.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f51400640 1 -- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c145040 msgr2=0x7f1f5c1f6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:22.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f51400640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c145040 0x7f1f5c1f6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f51400640 1 -- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f5c071310 msgr2=0x7f1f5c15b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f51400640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f5c071310 0x7f1f5c15b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f51400640 1 -- 172.21.15.67:0/3045670754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c15dc90 con 0x7f1f5c071770 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f51400640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f5c071770 0x7f1f5c140e40 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c480 tx=0x7f1f4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f52a00640 1 -- 172.21.15.67:0/3045670754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44017070 con 0x7f1f5c071770 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f52a00640 1 -- 172.21.15.67:0/3045670754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f44005cf0 con 0x7f1f5c071770 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f5c15df70 con 0x7f1f5c071770 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f52a00640 1 -- 172.21.15.67:0/3045670754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f44007290 con 0x7f1f5c071770 2024-04-17T10:36:22.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f5c15e530 con 0x7f1f5c071770 2024-04-17T10:36:22.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5c072d50 con 0x7f1f5c071770 2024-04-17T10:36:22.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f52a00640 1 -- 172.21.15.67:0/3045670754 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1f4401a070 con 0x7f1f5c071770 2024-04-17T10:36:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f52a00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f38066f60 0x7f1f38069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.062+0000 7f1f52a00640 1 -- 172.21.15.67:0/3045670754 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1f44084510 con 0x7f1f5c071770 2024-04-17T10:36:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.066+0000 7f1f59e00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f38066f60 0x7f1f38069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:22.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.066+0000 7f1f59e00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f38066f60 0x7f1f38069420 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007fd0 tx=0x7f1f4c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:22.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.066+0000 7f1f52a00640 1 -- 172.21.15.67:0/3045670754 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1f44054d80 con 0x7f1f5c071770 2024-04-17T10:36:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.266+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f5c074e30 con 0x7f1f5c071770 2024-04-17T10:36:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.266+0000 7f1f52a00640 1 -- 172.21.15.67:0/3045670754 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1f4405b0d0 con 0x7f1f5c071770 2024-04-17T10:36:22.258 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:22.258 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f38066f60 msgr2=0x7f1f38069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f38066f60 0x7f1f38069420 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f1f4c007fd0 tx=0x7f1f4c009040 comp rx=0 tx=0).stop 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f5c071770 msgr2=0x7f1f5c140e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f5c071770 0x7f1f5c140e40 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f1f4400c480 tx=0x7f1f4400c940 comp rx=0 tx=0).stop 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 shutdown_connections 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f38066f60 0x7f1f38069420 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f5c145040 0x7f1f5c1f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f5c071770 0x7f1f5c140e40 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 --2- 172.21.15.67:0/3045670754 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f5c071310 0x7f1f5c15b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 >> 172.21.15.67:0/3045670754 conn(0x7f1f5c06ac90 msgr2=0x7f1f5c0af3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:22.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 shutdown_connections 2024-04-17T10:36:22.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:22.270+0000 7f1f60c00640 1 -- 172.21.15.67:0/3045670754 wait complete. 2024-04-17T10:36:22.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:22.681 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:23.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:23.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82b5800640 1 Processor -- start 2024-04-17T10:36:23.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82b5800640 1 -- start start 2024-04-17T10:36:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82b5800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f82b00ce1b0 0x7f82b00ce590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82b5800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b00cead0 con 0x7f82b015cc00 2024-04-17T10:36:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82b5800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b015be60 con 0x7f82b015fbd0 2024-04-17T10:36:23.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82b5800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b015ffe0 con 0x7f82b00ce1b0 2024-04-17T10:36:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82a6000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f82b015cc00 0x7f82b015f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59434/0 (socket says 172.21.15.67:59434) 2024-04-17T10:36:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82a6000640 1 -- 172.21.15.67:0/556988535 learned_addr learned my addr 172.21.15.67:0/556988535 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82aea00640 1 --2- 172.21.15.67:0/556988535 >> v2:172.21.15.86:3300/0 conn(0x7f82b00ce1b0 0x7f82b00ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:23.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82aea00640 1 -- 172.21.15.67:0/556988535 >> v1:172.21.15.86:6789/0 conn(0x7f82b015fbd0 legacy=0x7f82b00c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82aea00640 1 -- 172.21.15.67:0/556988535 >> v1:172.21.15.67:6789/0 conn(0x7f82b015cc00 legacy=0x7f82b015f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82ae000640 1 -- 172.21.15.67:0/556988535 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406072351 0 0) 0x7f82b00cead0 con 0x7f82b015cc00 2024-04-17T10:36:23.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.846+0000 7f82aea00640 1 -- 172.21.15.67:0/556988535 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b0160230 con 0x7f82b00ce1b0 2024-04-17T10:36:23.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82aea00640 1 --2- 172.21.15.67:0/556988535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b00ce590 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f82980097c0 tx=0x7f829801ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7095fbd34b774d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:23.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82ae000640 1 -- 172.21.15.67:0/556988535 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980b5070 con 0x7f82b00ce1b0 2024-04-17T10:36:23.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82ae000640 1 -- 172.21.15.67:0/556988535 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82980a8990 con 0x7f82b00ce1b0 2024-04-17T10:36:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82ae000640 1 -- 172.21.15.67:0/556988535 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82980a8d60 con 0x7f82b00ce1b0 2024-04-17T10:36:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82b5800640 1 -- 172.21.15.67:0/556988535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 msgr2=0x7f82b00ce590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:23.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82b5800640 1 --2- 172.21.15.67:0/556988535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b00ce590 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f82980097c0 tx=0x7f829801ec90 comp rx=0 tx=0).stop 2024-04-17T10:36:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82b5800640 1 -- 172.21.15.67:0/556988535 shutdown_connections 2024-04-17T10:36:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82b5800640 1 --2- 172.21.15.67:0/556988535 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b00ce590 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82b5800640 1 -- 172.21.15.67:0/556988535 >> 172.21.15.67:0/556988535 conn(0x7f82b00b33f0 msgr2=0x7f82b00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82b5800640 1 -- 172.21.15.67:0/556988535 shutdown_connections 2024-04-17T10:36:23.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.850+0000 7f82b5800640 1 -- 172.21.15.67:0/556988535 wait complete. 2024-04-17T10:36:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 Processor -- start 2024-04-17T10:36:23.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 -- start start 2024-04-17T10:36:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b0140210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b015cc00 0x7f82b00ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b015fbd0 0x7f82b00cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b015c6e0 con 0x7f82b015cc00 2024-04-17T10:36:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b015c400 con 0x7f82b00ce1b0 2024-04-17T10:36:23.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b0159730 con 0x7f82b015fbd0 2024-04-17T10:36:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b015cc00 0x7f82b00ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b015cc00 0x7f82b00ccd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33502/0 (socket says 172.21.15.67:33502) 2024-04-17T10:36:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 -- 172.21.15.67:0/3585412781 learned_addr learned my addr 172.21.15.67:0/3585412781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82aea00640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b0140210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:23.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82af400640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b015fbd0 0x7f82b00cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 -- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b015fbd0 msgr2=0x7f82b00cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b015fbd0 0x7f82b00cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 -- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 msgr2=0x7f82b0140210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b0140210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 -- 172.21.15.67:0/3585412781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b00b46f0 con 0x7f82b015cc00 2024-04-17T10:36:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82aea00640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b0140210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:23.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a6000640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b015cc00 0x7f82b00ccd20 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f82a000c4a0 tx=0x7f82a000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a7400640 1 -- 172.21.15.67:0/3585412781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a0018070 con 0x7f82b015cc00 2024-04-17T10:36:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a7400640 1 -- 172.21.15.67:0/3585412781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a00074e0 con 0x7f82b015cc00 2024-04-17T10:36:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b00b49d0 con 0x7f82b015cc00 2024-04-17T10:36:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82a7400640 1 -- 172.21.15.67:0/3585412781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a0004a30 con 0x7f82b015cc00 2024-04-17T10:36:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.854+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b00b4f90 con 0x7f82b015cc00 2024-04-17T10:36:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.858+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b0070ab0 con 0x7f82b015cc00 2024-04-17T10:36:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.858+0000 7f82a7400640 1 -- 172.21.15.67:0/3585412781 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f82a001b070 con 0x7f82b015cc00 2024-04-17T10:36:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.858+0000 7f82a7400640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.858+0000 7f82a7400640 1 -- 172.21.15.67:0/3585412781 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f82a0084560 con 0x7f82b015cc00 2024-04-17T10:36:23.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.858+0000 7f82aea00640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:23.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.858+0000 7f82aea00640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f82980b2000 tx=0x7f8298008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:23.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:23.862+0000 7f82a7400640 1 -- 172.21.15.67:0/3585412781 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f82a0054ca0 con 0x7f82b015cc00 2024-04-17T10:36:24.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.058+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82b0074e30 con 0x7f82b015cc00 2024-04-17T10:36:24.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.058+0000 7f82a7400640 1 -- 172.21.15.67:0/3585412781 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f82a0058d40 con 0x7f82b015cc00 2024-04-17T10:36:24.049 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:24.049 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:24.050 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:24.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 msgr2=0x7f828c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:24.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f82980b2000 tx=0x7f8298008040 comp rx=0 tx=0).stop 2024-04-17T10:36:24.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b015cc00 msgr2=0x7f82b00ccd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:24.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b015cc00 0x7f82b00ccd20 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f82a000c4a0 tx=0x7f82a000c960 comp rx=0 tx=0).stop 2024-04-17T10:36:24.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 shutdown_connections 2024-04-17T10:36:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f828c066f60 0x7f828c069420 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f82b015fbd0 0x7f82b00cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b015cc00 0x7f82b00ccd20 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 --2- 172.21.15.67:0/3585412781 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f82b00ce1b0 0x7f82b0140210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 >> 172.21.15.67:0/3585412781 conn(0x7f82b00b33f0 msgr2=0x7f82b00b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 shutdown_connections 2024-04-17T10:36:24.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:24.062+0000 7f82b5800640 1 -- 172.21.15.67:0/3585412781 wait complete. 2024-04-17T10:36:24.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:24.474 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:25.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:25.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.638+0000 7f1b42200640 1 Processor -- start 2024-04-17T10:36:25.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.638+0000 7f1b42200640 1 -- start start 2024-04-17T10:36:25.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b42200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b3c06c410 0x7f1b3c06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:25.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b42200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0ad8f0 con 0x7f1b3c15dbc0 2024-04-17T10:36:25.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b42200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0adbd0 con 0x7f1b3c071c60 2024-04-17T10:36:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b42200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c15dfa0 con 0x7f1b3c06c410 2024-04-17T10:36:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b32a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b3c06c410 0x7f1b3c06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b3b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1b3c15dbc0 0x7f1b3c06db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59436/0 (socket says 172.21.15.67:59436) 2024-04-17T10:36:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b3b400640 1 -- 172.21.15.67:0/1672266244 learned_addr learned my addr 172.21.15.67:0/1672266244 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b32a00640 1 -- 172.21.15.67:0/1672266244 >> v1:172.21.15.86:6789/0 conn(0x7f1b3c071c60 legacy=0x7f1b3c1676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b32a00640 1 -- 172.21.15.67:0/1672266244 >> v1:172.21.15.67:6789/0 conn(0x7f1b3c15dbc0 legacy=0x7f1b3c06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:25.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b32a00640 1 -- 172.21.15.67:0/1672266244 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c0adeb0 con 0x7f1b3c06c410 2024-04-17T10:36:25.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.642+0000 7f1b3aa00640 1 -- 172.21.15.67:0/1672266244 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825089048 0 0) 0x7f1b3c0ad8f0 con 0x7f1b3c15dbc0 2024-04-17T10:36:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b32a00640 1 --2- 172.21.15.67:0/1672266244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c06f560 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f1b240097c0 tx=0x7f1b2401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f70b39a22b5bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b3aa00640 1 -- 172.21.15.67:0/1672266244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240b0070 con 0x7f1b3c06c410 2024-04-17T10:36:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b3aa00640 1 -- 172.21.15.67:0/1672266244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b240ab600 con 0x7f1b3c06c410 2024-04-17T10:36:25.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b3aa00640 1 -- 172.21.15.67:0/1672266244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240ab9d0 con 0x7f1b3c06c410 2024-04-17T10:36:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- 172.21.15.67:0/1672266244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 msgr2=0x7f1b3c06f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 --2- 172.21.15.67:0/1672266244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c06f560 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f1b240097c0 tx=0x7f1b2401c410 comp rx=0 tx=0).stop 2024-04-17T10:36:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- 172.21.15.67:0/1672266244 shutdown_connections 2024-04-17T10:36:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 --2- 172.21.15.67:0/1672266244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c06f560 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- 172.21.15.67:0/1672266244 >> 172.21.15.67:0/1672266244 conn(0x7f1b3c0af670 msgr2=0x7f1b3c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- 172.21.15.67:0/1672266244 shutdown_connections 2024-04-17T10:36:25.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- 172.21.15.67:0/1672266244 wait complete. 2024-04-17T10:36:25.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 Processor -- start 2024-04-17T10:36:25.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- start start 2024-04-17T10:36:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b3c071c60 0x7f1b3c149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b3c15dbc0 0x7f1b3c15f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c16b820 con 0x7f1b3c06c410 2024-04-17T10:36:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c06af70 con 0x7f1b3c071c60 2024-04-17T10:36:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.646+0000 7f1b42200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c16b9a0 con 0x7f1b3c15dbc0 2024-04-17T10:36:25.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:25.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c15cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33506/0 (socket says 172.21.15.67:33506) 2024-04-17T10:36:25.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 -- 172.21.15.67:0/3397936333 learned_addr learned my addr 172.21.15.67:0/3397936333 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:25.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b32a00640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b3c071c60 0x7f1b3c149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:25.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3be00640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b3c15dbc0 0x7f1b3c15f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:25.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 -- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b3c15dbc0 msgr2=0x7f1b3c15f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:25.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b3c15dbc0 0x7f1b3c15f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:25.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 -- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b3c071c60 msgr2=0x7f1b3c149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:25.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b3c071c60 0x7f1b3c149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:25.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 -- 172.21.15.67:0/3397936333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c14a340 con 0x7f1b3c06c410 2024-04-17T10:36:25.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b3b400640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c15cfd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00c4a0 tx=0x7f1b2c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:25.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b33e00640 1 -- 172.21.15.67:0/3397936333 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c018070 con 0x7f1b3c06c410 2024-04-17T10:36:25.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b33e00640 1 -- 172.21.15.67:0/3397936333 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c0074e0 con 0x7f1b3c06c410 2024-04-17T10:36:25.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b33e00640 1 -- 172.21.15.67:0/3397936333 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c004a30 con 0x7f1b3c06c410 2024-04-17T10:36:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c14a620 con 0x7f1b3c06c410 2024-04-17T10:36:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c14abe0 con 0x7f1b3c06c410 2024-04-17T10:36:25.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c09e260 con 0x7f1b3c06c410 2024-04-17T10:36:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.650+0000 7f1b33e00640 1 -- 172.21.15.67:0/3397936333 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1b2c004bd0 con 0x7f1b3c06c410 2024-04-17T10:36:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.654+0000 7f1b33e00640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b18066f60 0x7f1b18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:25.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.654+0000 7f1b33e00640 1 -- 172.21.15.67:0/3397936333 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f1b2c084620 con 0x7f1b3c06c410 2024-04-17T10:36:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.654+0000 7f1b32a00640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b18066f60 0x7f1b18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:25.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.654+0000 7f1b32a00640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b18066f60 0x7f1b18069420 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1b240ad000 tx=0x7f1b240a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:25.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.658+0000 7f1b33e00640 1 -- 172.21.15.67:0/3397936333 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1b2c0553b0 con 0x7f1b3c06c410 2024-04-17T10:36:25.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b3c074e30 con 0x7f1b3c06c410 2024-04-17T10:36:25.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b33e00640 1 -- 172.21.15.67:0/3397936333 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1b2c054cb0 con 0x7f1b3c06c410 2024-04-17T10:36:25.845 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:25.845 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:25.845 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:25.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b18066f60 msgr2=0x7f1b18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b18066f60 0x7f1b18069420 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1b240ad000 tx=0x7f1b240a6000 comp rx=0 tx=0).stop 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 msgr2=0x7f1b3c15cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c15cfd0 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00c4a0 tx=0x7f1b2c00c960 comp rx=0 tx=0).stop 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 shutdown_connections 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1b18066f60 0x7f1b18069420 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1b3c15dbc0 0x7f1b3c15f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1b3c071c60 0x7f1b3c149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 --2- 172.21.15.67:0/3397936333 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c06c410 0x7f1b3c15cfd0 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 >> 172.21.15.67:0/3397936333 conn(0x7f1b3c0af670 msgr2=0x7f1b3c152010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 shutdown_connections 2024-04-17T10:36:25.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:25.854+0000 7f1b42200640 1 -- 172.21.15.67:0/3397936333 wait complete. 2024-04-17T10:36:26.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:26.269 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:27.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.430+0000 7fa27b200640 1 Processor -- start 2024-04-17T10:36:27.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.430+0000 7fa27b200640 1 -- start start 2024-04-17T10:36:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa27b200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa27415ace0 0x7fa2740ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa27b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27406d9f0 con 0x7fa274158170 2024-04-17T10:36:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa27b200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa27406dcd0 con 0x7fa2740ceb70 2024-04-17T10:36:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa27b200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2740cf3c0 con 0x7fa27415ace0 2024-04-17T10:36:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa279e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa274158170 0x7fa27406d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59442/0 (socket says 172.21.15.67:59442) 2024-04-17T10:36:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa279e00640 1 -- 172.21.15.67:0/1795893555 learned_addr learned my addr 172.21.15.67:0/1795893555 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:27.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa279400640 1 --2- 172.21.15.67:0/1795893555 >> v2:172.21.15.86:3300/0 conn(0x7fa27415ace0 0x7fa2740ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:27.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa278a00640 1 -- 172.21.15.67:0/1795893555 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338429164 0 0) 0x7fa27406d9f0 con 0x7fa274158170 2024-04-17T10:36:27.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa278a00640 1 -- 172.21.15.67:0/1795893555 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa25c003530 con 0x7fa274158170 2024-04-17T10:36:27.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa279400640 1 -- 172.21.15.67:0/1795893555 >> v1:172.21.15.86:6789/0 conn(0x7fa2740ceb70 legacy=0x7fa2740cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:27.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa279400640 1 -- 172.21.15.67:0/1795893555 >> v1:172.21.15.67:6789/0 conn(0x7fa274158170 legacy=0x7fa27406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:27.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.434+0000 7fa279400640 1 -- 172.21.15.67:0/1795893555 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27406dfb0 con 0x7fa27415ace0 2024-04-17T10:36:27.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa279400640 1 --2- 172.21.15.67:0/1795893555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa27415ace0 0x7fa2740ce630 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa2640097c0 tx=0x7fa26401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f98f77358d7ca816 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:27.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa278a00640 1 -- 172.21.15.67:0/1795893555 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640aa420 con 0x7fa27415ace0 2024-04-17T10:36:27.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa278a00640 1 -- 172.21.15.67:0/1795893555 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2640af070 con 0x7fa27415ace0 2024-04-17T10:36:27.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa278a00640 1 -- 172.21.15.67:0/1795893555 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640aa7e0 con 0x7fa27415ace0 2024-04-17T10:36:27.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 -- 172.21.15.67:0/1795893555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa27415ace0 msgr2=0x7fa2740ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:27.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 --2- 172.21.15.67:0/1795893555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa27415ace0 0x7fa2740ce630 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa2640097c0 tx=0x7fa26401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 -- 172.21.15.67:0/1795893555 shutdown_connections 2024-04-17T10:36:27.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 --2- 172.21.15.67:0/1795893555 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa27415ace0 0x7fa2740ce630 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 -- 172.21.15.67:0/1795893555 >> 172.21.15.67:0/1795893555 conn(0x7fa274067070 msgr2=0x7fa27406ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:27.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 -- 172.21.15.67:0/1795893555 shutdown_connections 2024-04-17T10:36:27.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 -- 172.21.15.67:0/1795893555 wait complete. 2024-04-17T10:36:27.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 Processor -- start 2024-04-17T10:36:27.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.438+0000 7fa27b200640 1 -- start start 2024-04-17T10:36:27.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2740ceb70 0x7fa2741592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:27.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa274158170 0x7fa2741f9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:27.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa27415ace0 0x7fa2741f9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:27.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2741586d0 con 0x7fa274158170 2024-04-17T10:36:27.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa274158550 con 0x7fa2740ceb70 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa27406ea90 con 0x7fa27415ace0 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa274158170 0x7fa2741f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa274158170 0x7fa2741f9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33520/0 (socket says 172.21.15.67:33520) 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 -- 172.21.15.67:0/3597645257 learned_addr learned my addr 172.21.15.67:0/3597645257 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279e00640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2740ceb70 0x7fa2741592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27a800640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa27415ace0 0x7fa2741f9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 -- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa27415ace0 msgr2=0x7fa2741f9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa27415ace0 0x7fa2741f9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 -- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2740ceb70 msgr2=0x7fa2741592c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2740ceb70 0x7fa2741592c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 -- 172.21.15.67:0/3597645257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa274159980 con 0x7fa274158170 2024-04-17T10:36:27.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa279400640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa274158170 0x7fa2741f9530 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fa2640190b0 tx=0x7fa26401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:27.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa26e000640 1 -- 172.21.15.67:0/3597645257 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640aacd0 con 0x7fa274158170 2024-04-17T10:36:27.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa26e000640 1 -- 172.21.15.67:0/3597645257 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2640be410 con 0x7fa274158170 2024-04-17T10:36:27.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa274159c60 con 0x7fa274158170 2024-04-17T10:36:27.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa26e000640 1 -- 172.21.15.67:0/3597645257 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2640be800 con 0x7fa274158170 2024-04-17T10:36:27.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27415a220 con 0x7fa274158170 2024-04-17T10:36:27.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.442+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa274074dc0 con 0x7fa274158170 2024-04-17T10:36:27.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.446+0000 7fa26e000640 1 -- 172.21.15.67:0/3597645257 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa2640bc020 con 0x7fa274158170 2024-04-17T10:36:27.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.446+0000 7fa26e000640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa258066f60 0x7fa258069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:27.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.446+0000 7fa26e000640 1 -- 172.21.15.67:0/3597645257 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fa26411f6c0 con 0x7fa274158170 2024-04-17T10:36:27.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.446+0000 7fa279e00640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa258066f60 0x7fa258069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:27.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.446+0000 7fa279e00640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa258066f60 0x7fa258069420 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa2680011f0 tx=0x7fa268007d40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:27.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.450+0000 7fa26e000640 1 -- 172.21.15.67:0/3597645257 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa2640c3060 con 0x7fa274158170 2024-04-17T10:36:27.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.646+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa27414b620 con 0x7fa274158170 2024-04-17T10:36:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa26e000640 1 -- 172.21.15.67:0/3597645257 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fa264016090 con 0x7fa274158170 2024-04-17T10:36:27.639 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:27.639 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:27.640 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:27.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa258066f60 msgr2=0x7fa258069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:27.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa258066f60 0x7fa258069420 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa2680011f0 tx=0x7fa268007d40 comp rx=0 tx=0).stop 2024-04-17T10:36:27.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa274158170 msgr2=0x7fa2741f9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa274158170 0x7fa2741f9530 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fa2640190b0 tx=0x7fa26401ace0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 shutdown_connections 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa258066f60 0x7fa258069420 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa27415ace0 0x7fa2741f9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa274158170 0x7fa2741f9530 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 --2- 172.21.15.67:0/3597645257 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa2740ceb70 0x7fa2741592c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 >> 172.21.15.67:0/3597645257 conn(0x7fa274067070 msgr2=0x7fa27415d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 shutdown_connections 2024-04-17T10:36:27.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:27.650+0000 7fa27b200640 1 -- 172.21.15.67:0/3597645257 wait complete. 2024-04-17T10:36:28.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:28.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:28.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:36:28.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:36:29.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:29.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd08a600640 1 Processor -- start 2024-04-17T10:36:29.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd08a600640 1 -- start start 2024-04-17T10:36:29.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd08a600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd084070470 0x7fd084099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:29.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd08a600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840cf330 con 0x7fd084099f90 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd08a600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0840cf610 con 0x7fd08406f770 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd08a600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08409b190 con 0x7fd084070470 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd083e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd084070470 0x7fd084099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd089c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd084099f90 0x7fd08409a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:59454/0 (socket says 172.21.15.67:59454) 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd089c00640 1 -- 172.21.15.67:0/2513139475 learned_addr learned my addr 172.21.15.67:0/2513139475 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd083e00640 1 -- 172.21.15.67:0/2513139475 >> v1:172.21.15.86:6789/0 conn(0x7fd08406f770 legacy=0x7fd08406fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd083e00640 1 -- 172.21.15.67:0/2513139475 >> v1:172.21.15.67:6789/0 conn(0x7fd084099f90 legacy=0x7fd08409a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:29.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.230+0000 7fd083e00640 1 -- 172.21.15.67:0/2513139475 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08409ac70 con 0x7fd084070470 2024-04-17T10:36:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd083e00640 1 --2- 172.21.15.67:0/2513139475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084070470 0x7fd084099a50 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fd0700097c0 tx=0x7fd07001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70a44f7ca45f2b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd083400640 1 -- 172.21.15.67:0/2513139475 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0700b0070 con 0x7fd084070470 2024-04-17T10:36:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd083400640 1 -- 172.21.15.67:0/2513139475 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0700ab600 con 0x7fd084070470 2024-04-17T10:36:29.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd083400640 1 -- 172.21.15.67:0/2513139475 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0700a4e60 con 0x7fd084070470 2024-04-17T10:36:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 -- 172.21.15.67:0/2513139475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084070470 msgr2=0x7fd084099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 --2- 172.21.15.67:0/2513139475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084070470 0x7fd084099a50 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7fd0700097c0 tx=0x7fd07001c3d0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 -- 172.21.15.67:0/2513139475 shutdown_connections 2024-04-17T10:36:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 --2- 172.21.15.67:0/2513139475 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084070470 0x7fd084099a50 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 -- 172.21.15.67:0/2513139475 >> 172.21.15.67:0/2513139475 conn(0x7fd0840d1630 msgr2=0x7fd0840d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 -- 172.21.15.67:0/2513139475 shutdown_connections 2024-04-17T10:36:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 -- 172.21.15.67:0/2513139475 wait complete. 2024-04-17T10:36:29.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 Processor -- start 2024-04-17T10:36:29.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.234+0000 7fd08a600640 1 -- start start 2024-04-17T10:36:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd08406f770 0x7fd084165140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd084070470 0x7fd0841457e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084099f90 0x7fd08415dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd084169760 con 0x7fd084099f90 2024-04-17T10:36:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0840715d0 con 0x7fd08406f770 2024-04-17T10:36:29.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0841698e0 con 0x7fd084070470 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084099f90 0x7fd08415dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084099f90 0x7fd08415dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33536/0 (socket says 172.21.15.67:33536) 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 -- 172.21.15.67:0/139761279 learned_addr learned my addr 172.21.15.67:0/139761279 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089200640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd08406f770 0x7fd084165140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 -- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd084070470 msgr2=0x7fd0841457e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd084070470 0x7fd0841457e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 -- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd08406f770 msgr2=0x7fd084165140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd08406f770 0x7fd084165140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 -- 172.21.15.67:0/139761279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd084145d20 con 0x7fd084099f90 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089200640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd08406f770 0x7fd084165140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:36:29.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd089c00640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084099f90 0x7fd08415dcd0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00cbc0 tx=0x7fd07c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd080c00640 1 -- 172.21.15.67:0/139761279 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c017070 con 0x7fd084099f90 2024-04-17T10:36:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd080c00640 1 -- 172.21.15.67:0/139761279 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c00ad40 con 0x7fd084099f90 2024-04-17T10:36:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd080c00640 1 -- 172.21.15.67:0/139761279 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c012600 con 0x7fd084099f90 2024-04-17T10:36:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0841640a0 con 0x7fd084099f90 2024-04-17T10:36:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd084164660 con 0x7fd084099f90 2024-04-17T10:36:29.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.238+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd084074dc0 con 0x7fd084099f90 2024-04-17T10:36:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.242+0000 7fd080c00640 1 -- 172.21.15.67:0/139761279 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd07c01a070 con 0x7fd084099f90 2024-04-17T10:36:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.242+0000 7fd080c00640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd064066f90 0x7fd064069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.242+0000 7fd080c00640 1 -- 172.21.15.67:0/139761279 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fd07c084e20 con 0x7fd084099f90 2024-04-17T10:36:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.242+0000 7fd089200640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd064066f90 0x7fd064069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.242+0000 7fd089200640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd064066f90 0x7fd064069450 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd07800a850 tx=0x7fd078008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:29.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.246+0000 7fd080c00640 1 -- 172.21.15.67:0/139761279 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd07c0555f0 con 0x7fd084099f90 2024-04-17T10:36:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.442+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd08415e210 con 0x7fd084099f90 2024-04-17T10:36:29.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.442+0000 7fd080c00640 1 -- 172.21.15.67:0/139761279 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fd07c059690 con 0x7fd084099f90 2024-04-17T10:36:29.434 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:29.434 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:29.434 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd064066f90 msgr2=0x7fd064069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd064066f90 0x7fd064069450 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd07800a850 tx=0x7fd078008040 comp rx=0 tx=0).stop 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084099f90 msgr2=0x7fd08415dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084099f90 0x7fd08415dcd0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00cbc0 tx=0x7fd07c00a430 comp rx=0 tx=0).stop 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 shutdown_connections 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd064066f90 0x7fd064069450 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd084099f90 0x7fd08415dcd0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd084070470 0x7fd0841457e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 --2- 172.21.15.67:0/139761279 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd08406f770 0x7fd084165140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 >> 172.21.15.67:0/139761279 conn(0x7fd0840d1630 msgr2=0x7fd08406bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:29.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 shutdown_connections 2024-04-17T10:36:29.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:29.446+0000 7fd08a600640 1 -- 172.21.15.67:0/139761279 wait complete. 2024-04-17T10:36:29.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:29.858 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:30.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.018+0000 7fcb9ce00640 1 Processor -- start 2024-04-17T10:36:31.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.018+0000 7fcb9ce00640 1 -- start start 2024-04-17T10:36:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb9ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb98071790 0x7fcb9815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb9ce00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcb98071b70 0x7fcb9806e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:31.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb9ce00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980b3ab0 con 0x7fcb9806ebb0 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb9ce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9806efc0 con 0x7fcb98071790 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb9ce00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb98161700 con 0x7fcb98071b70 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb95e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb98071790 0x7fcb9815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb95e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcb98071790 0x7fcb9815ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52440/0 (socket says 172.21.15.67:52440) 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb95e00640 1 -- 172.21.15.67:0/1701965954 learned_addr learned my addr 172.21.15.67:0/1701965954 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb8d400640 1 --2- 172.21.15.67:0/1701965954 >> v2:172.21.15.86:3300/0 conn(0x7fcb98071b70 0x7fcb9806e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb95e00640 1 -- 172.21.15.67:0/1701965954 >> v2:172.21.15.86:3300/0 conn(0x7fcb98071b70 msgr2=0x7fcb9806e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb95e00640 1 --2- 172.21.15.67:0/1701965954 >> v2:172.21.15.86:3300/0 conn(0x7fcb98071b70 0x7fcb9806e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb95e00640 1 -- 172.21.15.67:0/1701965954 >> v1:172.21.15.86:6789/0 conn(0x7fcb9806ebb0 legacy=0x7fcb980b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:31.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.022+0000 7fcb95e00640 1 -- 172.21.15.67:0/1701965954 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb981619e0 con 0x7fcb98071790 2024-04-17T10:36:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb95e00640 1 --2- 172.21.15.67:0/1701965954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb9815ecf0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fcb980683a0 tx=0x7fcb8801c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff18001985012d51 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb95400640 1 -- 172.21.15.67:0/1701965954 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb880a57a0 con 0x7fcb98071790 2024-04-17T10:36:31.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb95400640 1 -- 172.21.15.67:0/1701965954 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb880a5940 con 0x7fcb98071790 2024-04-17T10:36:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb95400640 1 -- 172.21.15.67:0/1701965954 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb880a5d10 con 0x7fcb98071790 2024-04-17T10:36:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 -- 172.21.15.67:0/1701965954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 msgr2=0x7fcb9815ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/1701965954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb9815ecf0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7fcb980683a0 tx=0x7fcb8801c9c0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 -- 172.21.15.67:0/1701965954 shutdown_connections 2024-04-17T10:36:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/1701965954 >> v2:172.21.15.86:3300/0 conn(0x7fcb98071b70 0x7fcb9806e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/1701965954 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb9815ecf0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 -- 172.21.15.67:0/1701965954 >> 172.21.15.67:0/1701965954 conn(0x7fcb980cfa30 msgr2=0x7fcb980cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 -- 172.21.15.67:0/1701965954 shutdown_connections 2024-04-17T10:36:31.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 -- 172.21.15.67:0/1701965954 wait complete. 2024-04-17T10:36:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 Processor -- start 2024-04-17T10:36:31.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.026+0000 7fcb9ce00640 1 -- start start 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb9806ebb0 0x7fcb9815de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb98166130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb98071b70 0x7fcb98167c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98072120 con 0x7fcb98071790 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98071fa0 con 0x7fcb9806ebb0 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9806f1e0 con 0x7fcb98071b70 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb98166130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb98166130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52444/0 (socket says 172.21.15.67:52444) 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 -- 172.21.15.67:0/894668769 learned_addr learned my addr 172.21.15.67:0/894668769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb95e00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb9806ebb0 0x7fcb9815de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb96800640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb98071b70 0x7fcb98167c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 -- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb98071b70 msgr2=0x7fcb98167c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:31.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb98071b70 0x7fcb98167c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 -- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb9806ebb0 msgr2=0x7fcb9815de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:31.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb9806ebb0 0x7fcb9815de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 -- 172.21.15.67:0/894668769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980b06c0 con 0x7fcb98071790 2024-04-17T10:36:31.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8d400640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb98166130 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000d830 tx=0x7fcb8000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8ea00640 1 -- 172.21.15.67:0/894668769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8000b6c0 con 0x7fcb98071790 2024-04-17T10:36:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8ea00640 1 -- 172.21.15.67:0/894668769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb80019070 con 0x7fcb98071790 2024-04-17T10:36:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb980b09a0 con 0x7fcb98071790 2024-04-17T10:36:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8ea00640 1 -- 172.21.15.67:0/894668769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8000bae0 con 0x7fcb98071790 2024-04-17T10:36:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb980b0f60 con 0x7fcb98071790 2024-04-17T10:36:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb980729a0 con 0x7fcb98071790 2024-04-17T10:36:31.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.030+0000 7fcb8ea00640 1 -- 172.21.15.67:0/894668769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcb8001e460 con 0x7fcb98071790 2024-04-17T10:36:31.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.034+0000 7fcb8ea00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb74066f60 0x7fcb74069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:31.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.034+0000 7fcb8ea00640 1 -- 172.21.15.67:0/894668769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fcb80085930 con 0x7fcb98071790 2024-04-17T10:36:31.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.034+0000 7fcb95e00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb74066f60 0x7fcb74069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:31.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.034+0000 7fcb95e00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb74066f60 0x7fcb74069420 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fcb880ae000 tx=0x7fcb880a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:31.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.038+0000 7fcb8ea00640 1 -- 172.21.15.67:0/894668769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcb800566c0 con 0x7fcb98071790 2024-04-17T10:36:31.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.230+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb9806b880 con 0x7fcb98071790 2024-04-17T10:36:31.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.230+0000 7fcb8ea00640 1 -- 172.21.15.67:0/894668769 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fcb80055fc0 con 0x7fcb98071790 2024-04-17T10:36:31.223 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:31.223 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:31.223 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:31.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb74066f60 msgr2=0x7fcb74069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:31.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb74066f60 0x7fcb74069420 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fcb880ae000 tx=0x7fcb880a7070 comp rx=0 tx=0).stop 2024-04-17T10:36:31.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 msgr2=0x7fcb98166130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb98166130 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fcb8000d830 tx=0x7fcb8000dcf0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 shutdown_connections 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcb74066f60 0x7fcb74069420 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcb98071b70 0x7fcb98167c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcb98071790 0x7fcb98166130 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 --2- 172.21.15.67:0/894668769 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcb9806ebb0 0x7fcb9815de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 >> 172.21.15.67:0/894668769 conn(0x7fcb980cfa30 msgr2=0x7fcb981601a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 shutdown_connections 2024-04-17T10:36:31.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:31.234+0000 7fcb9ce00640 1 -- 172.21.15.67:0/894668769 wait complete. 2024-04-17T10:36:31.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:31.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:32.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.810+0000 7f432e200640 1 Processor -- start 2024-04-17T10:36:32.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.810+0000 7f432e200640 1 -- start start 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.810+0000 7f432e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4328144980 0x7f432806ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.810+0000 7f432e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f432806b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.810+0000 7f432e200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f432806fbd0 con 0x7f432806eed0 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.810+0000 7f432e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43280707a0 con 0x7f4328144980 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.810+0000 7f432e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43280ce020 con 0x7f4328145080 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4328144980 0x7f432806ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4328144980 0x7f432806ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52454/0 (socket says 172.21.15.67:52454) 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 -- 172.21.15.67:0/2477560654 learned_addr learned my addr 172.21.15.67:0/2477560654 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:32.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432d800640 1 --2- 172.21.15.67:0/2477560654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f432806b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 -- 172.21.15.67:0/2477560654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 msgr2=0x7f432806b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 --2- 172.21.15.67:0/2477560654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f432806b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 -- 172.21.15.67:0/2477560654 >> v1:172.21.15.86:6789/0 conn(0x7f432806eed0 legacy=0x7f432815f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:32.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 -- 172.21.15.67:0/2477560654 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f432806fe70 con 0x7f4328144980 2024-04-17T10:36:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f432ce00640 1 --2- 172.21.15.67:0/2477560654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f432806ac90 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f4318009870 tx=0x7f431801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43fcdc6a0c0c48c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:32.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f4327e00640 1 -- 172.21.15.67:0/2477560654 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43180b5070 con 0x7f4328144980 2024-04-17T10:36:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f4327e00640 1 -- 172.21.15.67:0/2477560654 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43180a8a50 con 0x7f4328144980 2024-04-17T10:36:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.814+0000 7f4327e00640 1 -- 172.21.15.67:0/2477560654 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43180a8e20 con 0x7f4328144980 2024-04-17T10:36:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- 172.21.15.67:0/2477560654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 msgr2=0x7f432806ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 --2- 172.21.15.67:0/2477560654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f432806ac90 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f4318009870 tx=0x7f431801ed50 comp rx=0 tx=0).stop 2024-04-17T10:36:32.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- 172.21.15.67:0/2477560654 shutdown_connections 2024-04-17T10:36:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 --2- 172.21.15.67:0/2477560654 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f432806b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 --2- 172.21.15.67:0/2477560654 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f432806ac90 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- 172.21.15.67:0/2477560654 >> 172.21.15.67:0/2477560654 conn(0x7f432806e3b0 msgr2=0x7f432806e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- 172.21.15.67:0/2477560654 shutdown_connections 2024-04-17T10:36:32.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- 172.21.15.67:0/2477560654 wait complete. 2024-04-17T10:36:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 Processor -- start 2024-04-17T10:36:32.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- start start 2024-04-17T10:36:32.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432806eed0 0x7f432815afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f4328134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f4328143450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4328070c30 con 0x7f4328144980 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4328070430 con 0x7f432806eed0 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432806ba40 con 0x7f4328145080 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f431fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f4328134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432806eed0 0x7f432815afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f431fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f4328134aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52464/0 (socket says 172.21.15.67:52464) 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f431fe00640 1 -- 172.21.15.67:0/909022496 learned_addr learned my addr 172.21.15.67:0/909022496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.818+0000 7f432d800640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f4328143450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f431fe00640 1 -- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 msgr2=0x7f4328143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f431fe00640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f4328143450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f431fe00640 1 -- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432806eed0 msgr2=0x7f432815afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:32.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f431fe00640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432806eed0 0x7f432815afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f431fe00640 1 -- 172.21.15.67:0/909022496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43281e55b0 con 0x7f4328144980 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f431fe00640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f4328134aa0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f431000c420 tx=0x7f431000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f4325600640 1 -- 172.21.15.67:0/909022496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100052d0 con 0x7f4328144980 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f4325600640 1 -- 172.21.15.67:0/909022496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4310005470 con 0x7f4328144980 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f4325600640 1 -- 172.21.15.67:0/909022496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310007260 con 0x7f4328144980 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43281e5890 con 0x7f4328144980 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43281e5e50 con 0x7f4328144980 2024-04-17T10:36:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4328074dc0 con 0x7f4328144980 2024-04-17T10:36:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f4325600640 1 -- 172.21.15.67:0/909022496 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4310007400 con 0x7f4328144980 2024-04-17T10:36:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f4325600640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4304066f60 0x7f4304069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.822+0000 7f4325600640 1 -- 172.21.15.67:0/909022496 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f43100846b0 con 0x7f4328144980 2024-04-17T10:36:32.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.826+0000 7f432ce00640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4304066f60 0x7f4304069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:32.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.826+0000 7f432ce00640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4304066f60 0x7f4304069420 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f43180b2000 tx=0x7f4318009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:32.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:32.826+0000 7f4325600640 1 -- 172.21.15.67:0/909022496 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4310054e70 con 0x7f4328144980 2024-04-17T10:36:33.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.022+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f432806eb10 con 0x7f4328144980 2024-04-17T10:36:33.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.022+0000 7f4325600640 1 -- 172.21.15.67:0/909022496 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f4310058f10 con 0x7f4328144980 2024-04-17T10:36:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:33.014 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:33.015 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4304066f60 msgr2=0x7f4304069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4304066f60 0x7f4304069420 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f43180b2000 tx=0x7f4318009040 comp rx=0 tx=0).stop 2024-04-17T10:36:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 msgr2=0x7f4328134aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f4328134aa0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f431000c420 tx=0x7f431000c8e0 comp rx=0 tx=0).stop 2024-04-17T10:36:33.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 shutdown_connections 2024-04-17T10:36:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4304066f60 0x7f4304069420 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4328145080 0x7f4328143450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4328144980 0x7f4328134aa0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 --2- 172.21.15.67:0/909022496 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f432806eed0 0x7f432815afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 >> 172.21.15.67:0/909022496 conn(0x7f432806e3b0 msgr2=0x7f43281412e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 shutdown_connections 2024-04-17T10:36:33.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:33.026+0000 7f432e200640 1 -- 172.21.15.67:0/909022496 wait complete. 2024-04-17T10:36:33.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:33.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:34.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.602+0000 7fdc02000640 1 Processor -- start 2024-04-17T10:36:34.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.602+0000 7fdc02000640 1 -- start start 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.602+0000 7fdc02000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbfc145040 0x7fdbfc06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.602+0000 7fdc02000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc06ef10 0x7fdbfc071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.602+0000 7fdc02000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbfc06e2f0 con 0x7fdbfc071850 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.602+0000 7fdc02000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbfc0a86c0 con 0x7fdbfc145040 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.602+0000 7fdc02000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc15bf10 con 0x7fdbfc06ef10 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbfc145040 0x7fdbfc06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbfc145040 0x7fdbfc06e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52468/0 (socket says 172.21.15.67:52468) 2024-04-17T10:36:34.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 -- 172.21.15.67:0/1059260981 learned_addr learned my addr 172.21.15.67:0/1059260981 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:34.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbf2a00640 1 --2- 172.21.15.67:0/1059260981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc06ef10 0x7fdbfc071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:34.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 -- 172.21.15.67:0/1059260981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc06ef10 msgr2=0x7fdbfc071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:34.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 --2- 172.21.15.67:0/1059260981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc06ef10 0x7fdbfc071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 -- 172.21.15.67:0/1059260981 >> v1:172.21.15.86:6789/0 conn(0x7fdbfc071850 legacy=0x7fdbfc15f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:34.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 -- 172.21.15.67:0/1059260981 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc06d6e0 con 0x7fdbfc145040 2024-04-17T10:36:34.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfb400640 1 --2- 172.21.15.67:0/1059260981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc145040 0x7fdbfc06e9d0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009870 tx=0x7fdbec01ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cbd945b11b8768e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfaa00640 1 -- 172.21.15.67:0/1059260981 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0b5070 con 0x7fdbfc145040 2024-04-17T10:36:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfaa00640 1 -- 172.21.15.67:0/1059260981 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec0a89d0 con 0x7fdbfc145040 2024-04-17T10:36:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.606+0000 7fdbfaa00640 1 -- 172.21.15.67:0/1059260981 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0a8da0 con 0x7fdbfc145040 2024-04-17T10:36:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- 172.21.15.67:0/1059260981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc145040 msgr2=0x7fdbfc06e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 --2- 172.21.15.67:0/1059260981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc145040 0x7fdbfc06e9d0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009870 tx=0x7fdbec01ed10 comp rx=0 tx=0).stop 2024-04-17T10:36:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- 172.21.15.67:0/1059260981 shutdown_connections 2024-04-17T10:36:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 --2- 172.21.15.67:0/1059260981 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc06ef10 0x7fdbfc071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 --2- 172.21.15.67:0/1059260981 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc145040 0x7fdbfc06e9d0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- 172.21.15.67:0/1059260981 >> 172.21.15.67:0/1059260981 conn(0x7fdbfc06ac90 msgr2=0x7fdbfc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:34.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- 172.21.15.67:0/1059260981 shutdown_connections 2024-04-17T10:36:34.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- 172.21.15.67:0/1059260981 wait complete. 2024-04-17T10:36:34.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 Processor -- start 2024-04-17T10:36:34.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- start start 2024-04-17T10:36:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc06ef10 0x7fdbfc15b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc071850 0x7fdbfc141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbfc145040 0x7fdbfc1419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc06dca0 con 0x7fdbfc06ef10 2024-04-17T10:36:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc071ca0 con 0x7fdbfc145040 2024-04-17T10:36:34.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdc02000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc06df80 con 0x7fdbfc071850 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc06ef10 0x7fdbfc15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc06ef10 0x7fdbfc15b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52470/0 (socket says 172.21.15.67:52470) 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 -- 172.21.15.67:0/1176594942 learned_addr learned my addr 172.21.15.67:0/1176594942 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfbe00640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbfc145040 0x7fdbfc1419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbf2a00640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc071850 0x7fdbfc141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 -- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc071850 msgr2=0x7fdbfc141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc071850 0x7fdbfc141470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 -- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbfc145040 msgr2=0x7fdbfc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbfc145040 0x7fdbfc1419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.610+0000 7fdbfb400640 1 -- 172.21.15.67:0/1176594942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc1405c0 con 0x7fdbfc06ef10 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbfb400640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc06ef10 0x7fdbfc15b070 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b3040 tx=0x7fdbec01bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbf3e00640 1 -- 172.21.15.67:0/1176594942 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0c0070 con 0x7fdbfc06ef10 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbf3e00640 1 -- 172.21.15.67:0/1176594942 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbec0b0a90 con 0x7fdbfc06ef10 2024-04-17T10:36:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbf3e00640 1 -- 172.21.15.67:0/1176594942 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec0b5070 con 0x7fdbfc06ef10 2024-04-17T10:36:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc141fc0 con 0x7fdbfc06ef10 2024-04-17T10:36:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc142a80 con 0x7fdbfc06ef10 2024-04-17T10:36:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc142890 con 0x7fdbfc06ef10 2024-04-17T10:36:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbf3e00640 1 -- 172.21.15.67:0/1176594942 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdbec0ba3e0 con 0x7fdbfc06ef10 2024-04-17T10:36:34.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbf3e00640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd8066f60 0x7fdbd8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:34.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbf3e00640 1 -- 172.21.15.67:0/1176594942 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7fdbec122410 con 0x7fdbfc06ef10 2024-04-17T10:36:34.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.614+0000 7fdbf2a00640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd8066f60 0x7fdbd8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:34.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.618+0000 7fdbf2a00640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd8066f60 0x7fdbd8069420 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc071590 tx=0x7fdbe40098d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:34.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.622+0000 7fdbf3e00640 1 -- 172.21.15.67:0/1176594942 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdbec018870 con 0x7fdbfc06ef10 2024-04-17T10:36:34.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.814+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbfc0b11b0 con 0x7fdbfc06ef10 2024-04-17T10:36:34.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.814+0000 7fdbf3e00640 1 -- 172.21.15.67:0/1176594942 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fdbec0f2b50 con 0x7fdbfc06ef10 2024-04-17T10:36:34.805 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:34.805 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd8066f60 msgr2=0x7fdbd8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd8066f60 0x7fdbd8069420 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc071590 tx=0x7fdbe40098d0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc06ef10 msgr2=0x7fdbfc15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:34.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc06ef10 0x7fdbfc15b070 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0b3040 tx=0x7fdbec01bab0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 shutdown_connections 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd8066f60 0x7fdbd8069420 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbfc145040 0x7fdbfc1419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbfc071850 0x7fdbfc141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 --2- 172.21.15.67:0/1176594942 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbfc06ef10 0x7fdbfc15b070 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 >> 172.21.15.67:0/1176594942 conn(0x7fdbfc06ac90 msgr2=0x7fdbfc0b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 shutdown_connections 2024-04-17T10:36:34.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:34.818+0000 7fdc02000640 1 -- 172.21.15.67:0/1176594942 wait complete. 2024-04-17T10:36:35.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:35.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:36.230 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:36.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bbba00640 1 Processor -- start 2024-04-17T10:36:36.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bbba00640 1 -- start start 2024-04-17T10:36:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bbba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb40cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bbba00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb415cf10 con 0x7f4bb406b6a0 2024-04-17T10:36:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bbba00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb40cf0c0 con 0x7f4bb40cd690 2024-04-17T10:36:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bbba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb415c300 con 0x7f4bb415b970 2024-04-17T10:36:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4babe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4bb406b6a0 0x7f4bb40ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53208/0 (socket says 172.21.15.67:53208) 2024-04-17T10:36:36.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4babe00640 1 -- 172.21.15.67:0/757305566 learned_addr learned my addr 172.21.15.67:0/757305566 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb9000640 1 --2- 172.21.15.67:0/757305566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb40cf880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590446271 0 0) 0x7f4bb415cf10 con 0x7f4bb406b6a0 2024-04-17T10:36:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b98003530 con 0x7f4bb406b6a0 2024-04-17T10:36:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539639790 0 0) 0x7f4bb40cf0c0 con 0x7f4bb40cd690 2024-04-17T10:36:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb415cf10 con 0x7f4bb40cd690 2024-04-17T10:36:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb9000640 1 -- 172.21.15.67:0/757305566 >> v1:172.21.15.86:6789/0 conn(0x7f4bb40cd690 legacy=0x7f4bb40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:36.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb9000640 1 -- 172.21.15.67:0/757305566 >> v1:172.21.15.67:6789/0 conn(0x7f4bb406b6a0 legacy=0x7f4bb40ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846161592 0 0) 0x7f4b98003530 con 0x7f4bb406b6a0 2024-04-17T10:36:36.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.394+0000 7f4bb9000640 1 -- 172.21.15.67:0/757305566 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb415c5e0 con 0x7f4bb415b970 2024-04-17T10:36:36.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bb9000640 1 --2- 172.21.15.67:0/757305566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb40cf880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40097f0 tx=0x7f4ba401c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1193d1a9b94029a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:36.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40b0070 con 0x7f4bb415b970 2024-04-17T10:36:36.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba40ab400 con 0x7f4bb415b970 2024-04-17T10:36:36.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bb3e00640 1 -- 172.21.15.67:0/757305566 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c001680 con 0x7f4bb415b970 2024-04-17T10:36:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bbba00640 1 -- 172.21.15.67:0/757305566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 msgr2=0x7f4bb40cf880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bbba00640 1 --2- 172.21.15.67:0/757305566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb40cf880 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40097f0 tx=0x7f4ba401c370 comp rx=0 tx=0).stop 2024-04-17T10:36:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bbba00640 1 -- 172.21.15.67:0/757305566 shutdown_connections 2024-04-17T10:36:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bbba00640 1 --2- 172.21.15.67:0/757305566 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb40cf880 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bbba00640 1 -- 172.21.15.67:0/757305566 >> 172.21.15.67:0/757305566 conn(0x7f4bb40b22a0 msgr2=0x7f4bb40b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bbba00640 1 -- 172.21.15.67:0/757305566 shutdown_connections 2024-04-17T10:36:36.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.398+0000 7f4bbba00640 1 -- 172.21.15.67:0/757305566 wait complete. 2024-04-17T10:36:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 Processor -- start 2024-04-17T10:36:36.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 -- start start 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb406b6a0 0x7f4bb4149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb40cd690 0x7f4bb41666d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb4151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb40d01d0 con 0x7f4bb40cd690 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb40d0050 con 0x7f4bb406b6a0 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb4169da0 con 0x7f4bb415b970 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb40cd690 0x7f4bb41666d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bb9000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb406b6a0 0x7f4bb4149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:36.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb40cd690 0x7f4bb41666d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52476/0 (socket says 172.21.15.67:52476) 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 -- 172.21.15.67:0/3476878024 learned_addr learned my addr 172.21.15.67:0/3476878024 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bb9a00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb4151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 -- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 msgr2=0x7f4bb4151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb4151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 -- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb406b6a0 msgr2=0x7f4bb4149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb406b6a0 0x7f4bb4149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 -- 172.21.15.67:0/3476878024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb414a0a0 con 0x7f4bb40cd690 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4babe00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb40cd690 0x7f4bb41666d0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c5d0 tx=0x7f4b9c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bb1600640 1 -- 172.21.15.67:0/3476878024 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c017070 con 0x7f4bb40cd690 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bb1600640 1 -- 172.21.15.67:0/3476878024 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c005d10 con 0x7f4bb40cd690 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb406e710 con 0x7f4bb40cd690 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bb1600640 1 -- 172.21.15.67:0/3476878024 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0072b0 con 0x7f4bb40cd690 2024-04-17T10:36:36.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.402+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb406ecd0 con 0x7f4bb40cd690 2024-04-17T10:36:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.406+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb409e900 con 0x7f4bb40cd690 2024-04-17T10:36:36.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.406+0000 7f4bb1600640 1 -- 172.21.15.67:0/3476878024 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4b9c01a070 con 0x7f4bb40cd690 2024-04-17T10:36:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.406+0000 7f4bb1600640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b90066f60 0x7f4b90069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:36.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.406+0000 7f4bb1600640 1 -- 172.21.15.67:0/3476878024 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6187+0+0 (secure 0 0 0) 0x7f4b9c084710 con 0x7f4bb40cd690 2024-04-17T10:36:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.406+0000 7f4bb9000640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b90066f60 0x7f4b90069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:36.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.410+0000 7f4bb9000640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b90066f60 0x7f4b90069420 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40ad000 tx=0x7f4ba40a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:36.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.410+0000 7f4bb1600640 1 -- 172.21.15.67:0/3476878024 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4b9c054ee0 con 0x7f4bb40cd690 2024-04-17T10:36:36.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bb4074e30 con 0x7f4bb40cd690 2024-04-17T10:36:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bb1600640 1 -- 172.21.15.67:0/3476878024 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f4b9c058f80 con 0x7f4bb40cd690 2024-04-17T10:36:36.600 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:36.600 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:36.601 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b90066f60 msgr2=0x7f4b90069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b90066f60 0x7f4b90069420 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f4ba40ad000 tx=0x7f4ba40a6000 comp rx=0 tx=0).stop 2024-04-17T10:36:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb40cd690 msgr2=0x7f4bb41666d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb40cd690 0x7f4bb41666d0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c5d0 tx=0x7f4b9c00ca90 comp rx=0 tx=0).stop 2024-04-17T10:36:36.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 shutdown_connections 2024-04-17T10:36:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4b90066f60 0x7f4b90069420 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4bb415b970 0x7f4bb4151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bb40cd690 0x7f4bb41666d0 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 --2- 172.21.15.67:0/3476878024 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4bb406b6a0 0x7f4bb4149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 >> 172.21.15.67:0/3476878024 conn(0x7f4bb40b22a0 msgr2=0x7f4bb40b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 shutdown_connections 2024-04-17T10:36:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:36.610+0000 7f4bbba00640 1 -- 172.21.15.67:0/3476878024 wait complete. 2024-04-17T10:36:37.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:37.025 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:38.026 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:38.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7fead1c00640 1 Processor -- start 2024-04-17T10:36:38.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7fead1c00640 1 -- start start 2024-04-17T10:36:38.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7fead1c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7feacc06eac0 0x7feacc143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7fead1c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 0x7feacc144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7fead1c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc144560 con 0x7feacc06e6e0 2024-04-17T10:36:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7fead1c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc15ae40 con 0x7feacc06eac0 2024-04-17T10:36:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7fead1c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc071440 con 0x7feacc0cf0c0 2024-04-17T10:36:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7feacaa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feacc06e6e0 0x7feacc0ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53210/0 (socket says 172.21.15.67:53210) 2024-04-17T10:36:38.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.190+0000 7feacaa00640 1 -- 172.21.15.67:0/915884280 learned_addr learned my addr 172.21.15.67:0/915884280 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:38.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacb400640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 0x7feacc144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feac2000640 1 --2- 172.21.15.67:0/915884280 >> v2:172.21.15.86:3300/0 conn(0x7feacc06eac0 0x7feacc143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666191753 0 0) 0x7feacc144560 con 0x7feacc06e6e0 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feab0003530 con 0x7feacc06e6e0 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36603644 0 0) 0x7feab0003530 con 0x7feacc06e6e0 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 msgr2=0x7feacc144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 0x7feacc144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 >> v2:172.21.15.86:3300/0 conn(0x7feacc06eac0 msgr2=0x7feacc143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 --2- 172.21.15.67:0/915884280 >> v2:172.21.15.86:3300/0 conn(0x7feacc06eac0 0x7feacc143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc071720 con 0x7feacc06e6e0 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7feabc001680 con 0x7feacc06e6e0 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 >> v1:172.21.15.67:6789/0 conn(0x7feacc06e6e0 legacy=0x7feacc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab00039c0 0x7feab0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 0x7feab0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab0004d50 0x7feab00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc15ae40 con 0x7feab00039c0 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc071440 con 0x7feab0004d50 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab000b9d0 con 0x7feacc0cf0c0 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacaa00640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab00039c0 0x7feab0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feac2000640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 0x7feab0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacb400640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab0004d50 0x7feab00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacaa00640 1 -- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 msgr2=0x7feab0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacaa00640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feacc0cf0c0 0x7feab0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacaa00640 1 -- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab0004d50 msgr2=0x7feab00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacaa00640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feab0004d50 0x7feab00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.194+0000 7feacaa00640 1 -- 172.21.15.67:0/915884280 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab0003530 con 0x7feab00039c0 2024-04-17T10:36:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7feacaa00640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab00039c0 0x7feab0003de0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c2c0 tx=0x7feabc01cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=debeaeb2f6878db6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc007d60 con 0x7feab00039c0 2024-04-17T10:36:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc0a9020 con 0x7feab00039c0 2024-04-17T10:36:38.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7feaca000640 1 -- 172.21.15.67:0/915884280 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc0a5b80 con 0x7feab00039c0 2024-04-17T10:36:38.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 -- 172.21.15.67:0/915884280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab00039c0 msgr2=0x7feab0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:38.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 --2- 172.21.15.67:0/915884280 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab00039c0 0x7feab0003de0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c2c0 tx=0x7feabc01cd10 comp rx=0 tx=0).stop 2024-04-17T10:36:38.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7feacb400640 1 -- 172.21.15.67:0/915884280 reap_dead start 2024-04-17T10:36:38.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 -- 172.21.15.67:0/915884280 shutdown_connections 2024-04-17T10:36:38.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 -- 172.21.15.67:0/915884280 >> 172.21.15.67:0/915884280 conn(0x7feacc06ac90 msgr2=0x7feacc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 -- 172.21.15.67:0/915884280 shutdown_connections 2024-04-17T10:36:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 -- 172.21.15.67:0/915884280 wait complete. 2024-04-17T10:36:38.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 Processor -- start 2024-04-17T10:36:38.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 -- start start 2024-04-17T10:36:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab00039c0 0x7feacc0ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc0ca6b0 0x7feacc1eb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.198+0000 7fead1c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc1eb9b0 0x7feacc1ebe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7fead1c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0b3a40 con 0x7feacc1eb9b0 2024-04-17T10:36:38.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7fead1c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0b38c0 con 0x7feacc0ca6b0 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7fead1c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc0b3bc0 con 0x7feab00039c0 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacb400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc1eb9b0 0x7feacc1ebe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab00039c0 0x7feacc0ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab00039c0 0x7feacc0ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:35756/0 (socket says 172.21.15.67:35756) 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacaa00640 1 -- 172.21.15.67:0/4137584244 learned_addr learned my addr 172.21.15.67:0/4137584244 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feac2000640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc0ca6b0 0x7feacc1eb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacb400640 1 -- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab00039c0 msgr2=0x7feacc0ca170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacb400640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab00039c0 0x7feacc0ca170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacb400640 1 -- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc0ca6b0 msgr2=0x7feacc1eb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacb400640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc0ca6b0 0x7feacc1eb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacb400640 1 -- 172.21.15.67:0/4137584244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc1f6e90 con 0x7feacc1eb9b0 2024-04-17T10:36:38.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacaa00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab00039c0 0x7feacc0ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feacb400640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc1eb9b0 0x7feacc1ebe30 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7feab000bba0 tx=0x7feac400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feac3400640 1 -- 172.21.15.67:0/4137584244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac400fce0 con 0x7feacc1eb9b0 2024-04-17T10:36:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feac3400640 1 -- 172.21.15.67:0/4137584244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac400fe80 con 0x7feacc1eb9b0 2024-04-17T10:36:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feac3400640 1 -- 172.21.15.67:0/4137584244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4015670 con 0x7feacc1eb9b0 2024-04-17T10:36:38.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc0cab60 con 0x7feacc1eb9b0 2024-04-17T10:36:38.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc0cb120 con 0x7feacc1eb9b0 2024-04-17T10:36:38.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feacc072d50 con 0x7feacc1eb9b0 2024-04-17T10:36:38.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.202+0000 7feac3400640 1 -- 172.21.15.67:0/4137584244 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7feac4015810 con 0x7feacc1eb9b0 2024-04-17T10:36:38.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.206+0000 7feac3400640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feaa8066f90 0x7feaa8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:38.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.206+0000 7feac3400640 1 -- 172.21.15.67:0/4137584244 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7feac4085f60 con 0x7feacc1eb9b0 2024-04-17T10:36:38.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.206+0000 7feacaa00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feaa8066f90 0x7feaa8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:38.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.206+0000 7feacaa00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feaa8066f90 0x7feaa8069450 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c2c0 tx=0x7feabc001a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:38.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.210+0000 7feac3400640 1 -- 172.21.15.67:0/4137584244 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7feac4019030 con 0x7feacc1eb9b0 2024-04-17T10:36:38.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.410+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feacc074e30 con 0x7feacc1eb9b0 2024-04-17T10:36:38.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.410+0000 7feac3400640 1 -- 172.21.15.67:0/4137584244 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7feac4010040 con 0x7feacc1eb9b0 2024-04-17T10:36:38.401 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:38.401 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:38.401 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feaa8066f90 msgr2=0x7feaa8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feaa8066f90 0x7feaa8069450 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7feabc00c2c0 tx=0x7feabc001a50 comp rx=0 tx=0).stop 2024-04-17T10:36:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc1eb9b0 msgr2=0x7feacc1ebe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:38.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc1eb9b0 0x7feacc1ebe30 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7feab000bba0 tx=0x7feac400f1b0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 shutdown_connections 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feaa8066f90 0x7feaa8069450 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feacc1eb9b0 0x7feacc1ebe30 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feacc0ca6b0 0x7feacc1eb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 --2- 172.21.15.67:0/4137584244 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feab00039c0 0x7feacc0ca170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 >> 172.21.15.67:0/4137584244 conn(0x7feacc06ac90 msgr2=0x7feacc0afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 shutdown_connections 2024-04-17T10:36:38.404 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:38.414+0000 7fead1c00640 1 -- 172.21.15.67:0/4137584244 wait complete. 2024-04-17T10:36:38.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:38.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:39.826 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:39.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde7400640 1 Processor -- start 2024-04-17T10:36:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde7400640 1 -- start start 2024-04-17T10:36:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde7400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbde0144aa0 0x7fbde00d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:39.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0071840 0x7fbde00d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde7400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde01456d0 con 0x7fbde01446c0 2024-04-17T10:36:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde7400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde00b51a0 con 0x7fbde0144aa0 2024-04-17T10:36:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde7400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00d1390 con 0x7fbde0071840 2024-04-17T10:36:39.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbddfe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbde0144aa0 0x7fbde00d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde5400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0071840 0x7fbde00d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde5400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0071840 0x7fbde00d0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:35762/0 (socket says 172.21.15.67:35762) 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde4a00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbde01446c0 0x7fbde0144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:35500/0 (socket says 172.21.15.67:35500) 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde5400640 1 -- 172.21.15.67:0/3092608408 learned_addr learned my addr 172.21.15.67:0/3092608408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbddfe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbde0144aa0 0x7fbde00d0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:59492/0 (socket says 172.21.15.67:59492) 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde5400640 1 -- 172.21.15.67:0/3092608408 >> v2:172.21.15.86:3300/0 conn(0x7fbde0144aa0 msgr2=0x7fbde00d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde5400640 1 --2- 172.21.15.67:0/3092608408 >> v2:172.21.15.86:3300/0 conn(0x7fbde0144aa0 0x7fbde00d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde5400640 1 -- 172.21.15.67:0/3092608408 >> v1:172.21.15.86:6789/0 conn(0x7fbde01446c0 legacy=0x7fbde0144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:39.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.990+0000 7fbde5400640 1 -- 172.21.15.67:0/3092608408 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde00b6640 con 0x7fbde0071840 2024-04-17T10:36:39.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.994+0000 7fbde5400640 1 --2- 172.21.15.67:0/3092608408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0071840 0x7fbde00d0e50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009e50 tx=0x7fbdd801c9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2ad365a48e5d0b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:39.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.994+0000 7fbddf400640 1 -- 172.21.15.67:0/3092608408 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80a58a0 con 0x7fbde0071840 2024-04-17T10:36:39.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbddf400640 1 -- 172.21.15.67:0/3092608408 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd80a5a40 con 0x7fbde0071840 2024-04-17T10:36:39.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbddf400640 1 -- 172.21.15.67:0/3092608408 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80a5e10 con 0x7fbde0071840 2024-04-17T10:36:39.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 -- 172.21.15.67:0/3092608408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0071840 msgr2=0x7fbde00d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:39.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 --2- 172.21.15.67:0/3092608408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0071840 0x7fbde00d0e50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009e50 tx=0x7fbdd801c9e0 comp rx=0 tx=0).stop 2024-04-17T10:36:39.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 -- 172.21.15.67:0/3092608408 shutdown_connections 2024-04-17T10:36:39.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 --2- 172.21.15.67:0/3092608408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0071840 0x7fbde00d0e50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:39.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 --2- 172.21.15.67:0/3092608408 >> v2:172.21.15.86:3300/0 conn(0x7fbde0144aa0 0x7fbde00d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 -- 172.21.15.67:0/3092608408 >> 172.21.15.67:0/3092608408 conn(0x7fbde00b34d0 msgr2=0x7fbde00b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 -- 172.21.15.67:0/3092608408 shutdown_connections 2024-04-17T10:36:39.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:39.998+0000 7fbde7400640 1 -- 172.21.15.67:0/3092608408 wait complete. 2024-04-17T10:36:39.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 Processor -- start 2024-04-17T10:36:39.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 -- start start 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde0071840 0x7fbde013dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbde01446c0 0x7fbde0159f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0144aa0 0x7fbde015a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00b5e20 con 0x7fbde0071840 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde015c0d0 con 0x7fbde01446c0 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00b5fa0 con 0x7fbde0144aa0 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde0071840 0x7fbde013dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde0071840 0x7fbde013dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52490/0 (socket says 172.21.15.67:52490) 2024-04-17T10:36:39.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 -- 172.21.15.67:0/2707271477 learned_addr learned my addr 172.21.15.67:0/2707271477 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde5400640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0144aa0 0x7fbde015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbddfe00640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbde01446c0 0x7fbde0159f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 -- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0144aa0 msgr2=0x7fbde015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0144aa0 0x7fbde015a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 -- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbde01446c0 msgr2=0x7fbde0159f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbde01446c0 0x7fbde0159f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 -- 172.21.15.67:0/2707271477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde013e280 con 0x7fbde0071840 2024-04-17T10:36:39.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde4a00640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde0071840 0x7fbde013dd40 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400c420 tx=0x7fbdd400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbddcc00640 1 -- 172.21.15.67:0/2707271477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd40074e0 con 0x7fbde0071840 2024-04-17T10:36:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbddcc00640 1 -- 172.21.15.67:0/2707271477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd40076a0 con 0x7fbde0071840 2024-04-17T10:36:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbddcc00640 1 -- 172.21.15.67:0/2707271477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4004a30 con 0x7fbde0071840 2024-04-17T10:36:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde00a8770 con 0x7fbde0071840 2024-04-17T10:36:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde00a8d30 con 0x7fbde0071840 2024-04-17T10:36:39.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.002+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde0070bb0 con 0x7fbde0071840 2024-04-17T10:36:39.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.006+0000 7fbddcc00640 1 -- 172.21.15.67:0/2707271477 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbdd401a020 con 0x7fbde0071840 2024-04-17T10:36:39.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.006+0000 7fbddcc00640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdc0066f60 0x7fbdc0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:39.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.006+0000 7fbddcc00640 1 -- 172.21.15.67:0/2707271477 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbdd40849c0 con 0x7fbde0071840 2024-04-17T10:36:39.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.006+0000 7fbddfe00640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdc0066f60 0x7fbdc0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:39.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.006+0000 7fbddfe00640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdc0066f60 0x7fbdc0069420 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fbde01429b0 tx=0x7fbdcc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:40.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.010+0000 7fbddcc00640 1 -- 172.21.15.67:0/2707271477 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbdd40550c0 con 0x7fbde0071840 2024-04-17T10:36:40.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.206+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbde0074e30 con 0x7fbde0071840 2024-04-17T10:36:40.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.206+0000 7fbddcc00640 1 -- 172.21.15.67:0/2707271477 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fbdd4059160 con 0x7fbde0071840 2024-04-17T10:36:40.199 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:40.199 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:40.199 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:40.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdc0066f60 msgr2=0x7fbdc0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdc0066f60 0x7fbdc0069420 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fbde01429b0 tx=0x7fbdcc009190 comp rx=0 tx=0).stop 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde0071840 msgr2=0x7fbde013dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde0071840 0x7fbde013dd40 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400c420 tx=0x7fbdd400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 shutdown_connections 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdc0066f60 0x7fbdc0069420 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbde0144aa0 0x7fbde015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbde01446c0 0x7fbde0159f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 --2- 172.21.15.67:0/2707271477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbde0071840 0x7fbde013dd40 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 >> 172.21.15.67:0/2707271477 conn(0x7fbde00b34d0 msgr2=0x7fbde00b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 shutdown_connections 2024-04-17T10:36:40.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:40.210+0000 7fbde7400640 1 -- 172.21.15.67:0/2707271477 wait complete. 2024-04-17T10:36:40.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:40.622 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:41.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 Processor -- start 2024-04-17T10:36:41.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 -- start start 2024-04-17T10:36:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f9e640ce020 0x7f9e641620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e640ce400 0x7f9e6415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e640cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e64161650 con 0x7f9e640ce400 2024-04-17T10:36:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e641625e0 con 0x7f9e640ce020 2024-04-17T10:36:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e6bc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6415eb50 con 0x7f9e6415e550 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e640ce400 0x7f9e6415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e640ce400 0x7f9e6415e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34260/0 (socket says 172.21.15.67:34260) 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 -- 172.21.15.67:0/2536754306 learned_addr learned my addr 172.21.15.67:0/2536754306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e69c00640 1 --2- 172.21.15.67:0/2536754306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e640cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e69200640 1 --2- 172.21.15.67:0/2536754306 >> v2:172.21.15.86:3300/0 conn(0x7f9e640ce020 0x7f9e641620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 -- 172.21.15.67:0/2536754306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 msgr2=0x7f9e640cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 --2- 172.21.15.67:0/2536754306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e640cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 -- 172.21.15.67:0/2536754306 >> v2:172.21.15.86:3300/0 conn(0x7f9e640ce020 msgr2=0x7f9e641620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 --2- 172.21.15.67:0/2536754306 >> v2:172.21.15.86:3300/0 conn(0x7f9e640ce020 0x7f9e641620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.786+0000 7f9e5be00640 1 -- 172.21.15.67:0/2536754306 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6415ee30 con 0x7f9e640ce400 2024-04-17T10:36:41.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e5be00640 1 --2- 172.21.15.67:0/2536754306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e6415e010 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009870 tx=0x7f9e4c01c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5acfaba88891ba server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:41.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e63e00640 1 -- 172.21.15.67:0/2536754306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0b1070 con 0x7f9e640ce400 2024-04-17T10:36:41.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e63e00640 1 -- 172.21.15.67:0/2536754306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c0ac620 con 0x7f9e640ce400 2024-04-17T10:36:41.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e63e00640 1 -- 172.21.15.67:0/2536754306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0ac9f0 con 0x7f9e640ce400 2024-04-17T10:36:41.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 -- 172.21.15.67:0/2536754306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 msgr2=0x7f9e6415e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:41.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/2536754306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e6415e010 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009870 tx=0x7f9e4c01c4f0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 -- 172.21.15.67:0/2536754306 shutdown_connections 2024-04-17T10:36:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/2536754306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e640cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/2536754306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e6415e010 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/2536754306 >> v2:172.21.15.86:3300/0 conn(0x7f9e640ce020 0x7f9e641620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 -- 172.21.15.67:0/2536754306 >> 172.21.15.67:0/2536754306 conn(0x7f9e640b33f0 msgr2=0x7f9e640b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 -- 172.21.15.67:0/2536754306 shutdown_connections 2024-04-17T10:36:41.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.790+0000 7f9e6bc00640 1 -- 172.21.15.67:0/2536754306 wait complete. 2024-04-17T10:36:41.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 Processor -- start 2024-04-17T10:36:41.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 -- start start 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e640ce020 0x7f9e641594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e641509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e64160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6406c0f0 con 0x7f9e640ce400 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6406bf70 con 0x7f9e640ce020 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e640c84b0 con 0x7f9e6415e550 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e5be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e641509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:41.786 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e69200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e640ce020 0x7f9e641594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e69200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e640ce020 0x7f9e641594f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:52510/0 (socket says 172.21.15.67:52510) 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e69200640 1 -- 172.21.15.67:0/388986560 learned_addr learned my addr 172.21.15.67:0/388986560 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e69c00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e64160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e5be00640 1 -- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 msgr2=0x7f9e64160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e5be00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e64160320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e5be00640 1 -- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e640ce020 msgr2=0x7f9e641594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e5be00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e640ce020 0x7f9e641594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e5be00640 1 -- 172.21.15.67:0/388986560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e64159a30 con 0x7f9e640ce400 2024-04-17T10:36:41.787 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e5be00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e641509d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0ac000 tx=0x7f9e4c019290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:41.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e61600640 1 -- 172.21.15.67:0/388986560 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0b1040 con 0x7f9e640ce400 2024-04-17T10:36:41.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e61600640 1 -- 172.21.15.67:0/388986560 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c0ac420 con 0x7f9e640ce400 2024-04-17T10:36:41.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e61600640 1 -- 172.21.15.67:0/388986560 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e4c0b5680 con 0x7f9e640ce400 2024-04-17T10:36:41.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6415d0d0 con 0x7f9e640ce400 2024-04-17T10:36:41.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.794+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e64157170 con 0x7f9e640ce400 2024-04-17T10:36:41.788 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.798+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e64070ab0 con 0x7f9e640ce400 2024-04-17T10:36:41.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.798+0000 7f9e61600640 1 -- 172.21.15.67:0/388986560 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9e4c0bc020 con 0x7f9e640ce400 2024-04-17T10:36:41.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.798+0000 7f9e61600640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e40066f60 0x7f9e40069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:41.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.798+0000 7f9e61600640 1 -- 172.21.15.67:0/388986560 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f9e4c11e650 con 0x7f9e640ce400 2024-04-17T10:36:41.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.798+0000 7f9e69200640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e40066f60 0x7f9e40069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:41.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.798+0000 7f9e69200640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e40066f60 0x7f9e40069420 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9e540045f0 tx=0x7f9e54008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:41.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.802+0000 7f9e61600640 1 -- 172.21.15.67:0/388986560 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9e4c0c3020 con 0x7f9e640ce400 2024-04-17T10:36:41.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e640cf7e0 con 0x7f9e640ce400 2024-04-17T10:36:42.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e61600640 1 -- 172.21.15.67:0/388986560 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f9e4c016110 con 0x7f9e640ce400 2024-04-17T10:36:42.063 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:42.063 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e40066f60 msgr2=0x7f9e40069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e40066f60 0x7f9e40069420 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9e540045f0 tx=0x7f9e54008040 comp rx=0 tx=0).stop 2024-04-17T10:36:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 msgr2=0x7f9e641509d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:42.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e641509d0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0ac000 tx=0x7f9e4c019290 comp rx=0 tx=0).stop 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 shutdown_connections 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e40066f60 0x7f9e40069420 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e6415e550 0x7f9e64160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e640ce400 0x7f9e641509d0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 --2- 172.21.15.67:0/388986560 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e640ce020 0x7f9e641594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 >> 172.21.15.67:0/388986560 conn(0x7f9e640b33f0 msgr2=0x7f9e640c6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 shutdown_connections 2024-04-17T10:36:42.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:41.998+0000 7f9e6bc00640 1 -- 172.21.15.67:0/388986560 wait complete. 2024-04-17T10:36:42.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:42.469 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:43.470 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:43.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067f800640 1 Processor -- start 2024-04-17T10:36:43.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067f800640 1 -- start start 2024-04-17T10:36:43.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067f800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f06781441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067f800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067815ae40 con 0x7f067806eac0 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067f800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0678071440 con 0x7f067806e6e0 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067f800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0678071720 con 0x7f06780cf0c0 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f066fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f067806eac0 0x7f0678143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52776/0 (socket says 172.21.15.67:52776) 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f066fe00640 1 -- 172.21.15.67:0/1725398825 learned_addr learned my addr 172.21.15.67:0/1725398825 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067d800640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f06781441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401043838 0 0) 0x7f067815ae40 con 0x7f067806eac0 2024-04-17T10:36:43.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f065c003530 con 0x7f067806eac0 2024-04-17T10:36:43.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087230059 0 0) 0x7f0678071440 con 0x7f067806e6e0 2024-04-17T10:36:43.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f067815ae40 con 0x7f067806e6e0 2024-04-17T10:36:43.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419896789 0 0) 0x7f065c003530 con 0x7f067806eac0 2024-04-17T10:36:43.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 msgr2=0x7f06781441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f06781441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 >> v1:172.21.15.86:6789/0 conn(0x7f067806e6e0 legacy=0x7f06780ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0678071a00 con 0x7f067806eac0 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f0660001680 con 0x7f067806eac0 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f067d800640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f06781441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 >> v1:172.21.15.67:6789/0 conn(0x7f067806eac0 legacy=0x7f0678143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f065c003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065c004110 0x7f065c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f065c004d50 0x7f065c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678071720 con 0x7f065c004110 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c00dda0 con 0x7f065c004d50 2024-04-17T10:36:43.629 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.634+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c00dfa0 con 0x7f06780cf0c0 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f066fe00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065c004110 0x7f065c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067ce00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f065c003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067d800640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f065c004d50 0x7f065c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067ce00640 1 -- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f065c004d50 msgr2=0x7f065c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067ce00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f065c004d50 0x7f065c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067ce00640 1 -- 172.21.15.67:0/1725398825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065c004110 msgr2=0x7f065c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067ce00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065c004110 0x7f065c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067ce00640 1 -- 172.21.15.67:0/1725398825 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c003530 con 0x7f06780cf0c0 2024-04-17T10:36:43.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f066fe00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f065c004110 0x7f065c004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:43.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.638+0000 7f067ce00640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f065c003b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f06680018d0 tx=0x7f066801c550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd0d93a0187ba39a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668001520 con 0x7f06780cf0c0 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06680052f0 con 0x7f06780cf0c0 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f0677e00640 1 -- 172.21.15.67:0/1725398825 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668003840 con 0x7f06780cf0c0 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 msgr2=0x7f065c003b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 --2- 172.21.15.67:0/1725398825 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f06780cf0c0 0x7f065c003b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f06680018d0 tx=0x7f066801c550 comp rx=0 tx=0).stop 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067d800640 1 -- 172.21.15.67:0/1725398825 reap_dead start 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- 172.21.15.67:0/1725398825 shutdown_connections 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- 172.21.15.67:0/1725398825 >> 172.21.15.67:0/1725398825 conn(0x7f067806ac90 msgr2=0x7f067806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- 172.21.15.67:0/1725398825 shutdown_connections 2024-04-17T10:36:43.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- 172.21.15.67:0/1725398825 wait complete. 2024-04-17T10:36:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 Processor -- start 2024-04-17T10:36:43.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- start start 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780cf0c0 0x7f06781595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06781ecb00 0x7f0678140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0678141330 0x7f06781f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067806db80 con 0x7f06780cf0c0 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0678144600 con 0x7f06781ecb00 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067f800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067806dd00 con 0x7f0678141330 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780cf0c0 0x7f06781595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780cf0c0 0x7f06781595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34292/0 (socket says 172.21.15.67:34292) 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.642+0000 7f067ce00640 1 -- 172.21.15.67:0/1074992184 learned_addr learned my addr 172.21.15.67:0/1074992184 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f066fe00640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06781ecb00 0x7f0678140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067d800640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0678141330 0x7f06781f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067ce00640 1 -- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0678141330 msgr2=0x7f06781f6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067ce00640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0678141330 0x7f06781f6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067ce00640 1 -- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06781ecb00 msgr2=0x7f0678140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067ce00640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06781ecb00 0x7f0678140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067ce00640 1 -- 172.21.15.67:0/1074992184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06780afd00 con 0x7f06780cf0c0 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f066fe00640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06781ecb00 0x7f0678140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067ce00640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780cf0c0 0x7f06781595c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f066801c380 tx=0x7f065c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f0675600640 1 -- 172.21.15.67:0/1074992184 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06680bf070 con 0x7f06780cf0c0 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f0675600640 1 -- 172.21.15.67:0/1074992184 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06680054a0 con 0x7f06780cf0c0 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f0675600640 1 -- 172.21.15.67:0/1074992184 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0668001500 con 0x7f06780cf0c0 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06781ec020 con 0x7f06780cf0c0 2024-04-17T10:36:43.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06781ec5e0 con 0x7f06780cf0c0 2024-04-17T10:36:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0678141740 con 0x7f06780cf0c0 2024-04-17T10:36:43.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.646+0000 7f0675600640 1 -- 172.21.15.67:0/1074992184 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f06680c2070 con 0x7f06780cf0c0 2024-04-17T10:36:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.650+0000 7f0675600640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0654066f90 0x7f0654069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.650+0000 7f0675600640 1 -- 172.21.15.67:0/1074992184 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f066811ead0 con 0x7f06780cf0c0 2024-04-17T10:36:43.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.650+0000 7f066fe00640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0654066f90 0x7f0654069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:43.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.650+0000 7f066fe00640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0654066f90 0x7f0654069450 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0660001210 tx=0x7f0660009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:43.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.654+0000 7f0675600640 1 -- 172.21.15.67:0/1074992184 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0678141740 con 0x7f06780cf0c0 2024-04-17T10:36:43.836 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.846+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0678074e30 con 0x7f06780cf0c0 2024-04-17T10:36:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.846+0000 7f0675600640 1 -- 172.21.15.67:0/1074992184 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f0668016290 con 0x7f06780cf0c0 2024-04-17T10:36:43.837 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:43.837 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:43.837 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.846+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0654066f90 msgr2=0x7f0654069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.846+0000 7f067f800640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0654066f90 0x7f0654069450 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f0660001210 tx=0x7f0660009040 comp rx=0 tx=0).stop 2024-04-17T10:36:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.846+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780cf0c0 msgr2=0x7f06781595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:43.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.846+0000 7f067f800640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780cf0c0 0x7f06781595c0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f066801c380 tx=0x7f065c005840 comp rx=0 tx=0).stop 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 shutdown_connections 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0654066f90 0x7f0654069450 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0678141330 0x7f06781f6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f06781ecb00 0x7f0678140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 --2- 172.21.15.67:0/1074992184 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f06780cf0c0 0x7f06781595c0 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 >> 172.21.15.67:0/1074992184 conn(0x7f067806ac90 msgr2=0x7f06780cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 shutdown_connections 2024-04-17T10:36:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:43.850+0000 7f067f800640 1 -- 172.21.15.67:0/1074992184 wait complete. 2024-04-17T10:36:44.261 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:44.261 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:45.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad47e00640 1 Processor -- start 2024-04-17T10:36:45.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad47e00640 1 -- start start 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad47e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad4015dd10 0x7fad400b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad47e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400714d0 0x7fad4006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad47e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad400b0cc0 con 0x7fad400b57b0 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad47e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad400b1280 con 0x7fad4015dd10 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad47e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad400b0fa0 con 0x7fad400714d0 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad4015dd10 0x7fad400b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fad4015dd10 0x7fad400b5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34294/0 (socket says 172.21.15.67:34294) 2024-04-17T10:36:45.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45400640 1 -- 172.21.15.67:0/336229666 learned_addr learned my addr 172.21.15.67:0/336229666 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45e00640 1 --2- 172.21.15.67:0/336229666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400714d0 0x7fad4006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45400640 1 -- 172.21.15.67:0/336229666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400714d0 msgr2=0x7fad4006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45400640 1 --2- 172.21.15.67:0/336229666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400714d0 0x7fad4006c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45400640 1 -- 172.21.15.67:0/336229666 >> v1:172.21.15.86:6789/0 conn(0x7fad400b57b0 legacy=0x7fad40070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:45.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.426+0000 7fad45400640 1 -- 172.21.15.67:0/336229666 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4006bed0 con 0x7fad4015dd10 2024-04-17T10:36:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad45400640 1 --2- 172.21.15.67:0/336229666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad400b5270 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fad34009870 tx=0x7fad3401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1f249390b8ad48 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad3fe00640 1 -- 172.21.15.67:0/336229666 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad340ab680 con 0x7fad4015dd10 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad3fe00640 1 -- 172.21.15.67:0/336229666 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad340b0070 con 0x7fad4015dd10 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad3fe00640 1 -- 172.21.15.67:0/336229666 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad340aba40 con 0x7fad4015dd10 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 -- 172.21.15.67:0/336229666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 msgr2=0x7fad400b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 --2- 172.21.15.67:0/336229666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad400b5270 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fad34009870 tx=0x7fad3401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 -- 172.21.15.67:0/336229666 shutdown_connections 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 --2- 172.21.15.67:0/336229666 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400714d0 0x7fad4006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 --2- 172.21.15.67:0/336229666 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad400b5270 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 -- 172.21.15.67:0/336229666 >> 172.21.15.67:0/336229666 conn(0x7fad400b34d0 msgr2=0x7fad400b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 -- 172.21.15.67:0/336229666 shutdown_connections 2024-04-17T10:36:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 -- 172.21.15.67:0/336229666 wait complete. 2024-04-17T10:36:45.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.430+0000 7fad47e00640 1 Processor -- start 2024-04-17T10:36:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 -- start start 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad400714d0 0x7fad4014b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad40154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400cd6e0 0x7fad401617c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4006d7b0 con 0x7fad4015dd10 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4006d630 con 0x7fad400714d0 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4006d930 con 0x7fad400cd6e0 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad40154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad40154e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34308/0 (socket says 172.21.15.67:34308) 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 -- 172.21.15.67:0/3384061589 learned_addr learned my addr 172.21.15.67:0/3384061589 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad45400640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad400714d0 0x7fad4014b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad45e00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400cd6e0 0x7fad401617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 -- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400cd6e0 msgr2=0x7fad401617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400cd6e0 0x7fad401617c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 -- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad400714d0 msgr2=0x7fad4014b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad400714d0 0x7fad4014b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 -- 172.21.15.67:0/3384061589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad40067c90 con 0x7fad4015dd10 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad44a00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad40154e20 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00c420 tx=0x7fad2c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad3d600640 1 -- 172.21.15.67:0/3384061589 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c0052d0 con 0x7fad4015dd10 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad3d600640 1 -- 172.21.15.67:0/3384061589 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c005470 con 0x7fad4015dd10 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad3d600640 1 -- 172.21.15.67:0/3384061589 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c007260 con 0x7fad4015dd10 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad40155750 con 0x7fad4015dd10 2024-04-17T10:36:45.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad40155d10 con 0x7fad4015dd10 2024-04-17T10:36:45.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.434+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad40070bb0 con 0x7fad4015dd10 2024-04-17T10:36:45.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.438+0000 7fad3d600640 1 -- 172.21.15.67:0/3384061589 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fad2c007400 con 0x7fad4015dd10 2024-04-17T10:36:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.438+0000 7fad3d600640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad20066f90 0x7fad20069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.438+0000 7fad3d600640 1 -- 172.21.15.67:0/3384061589 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fad2c0845f0 con 0x7fad4015dd10 2024-04-17T10:36:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.438+0000 7fad45400640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad20066f90 0x7fad20069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.438+0000 7fad45400640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad20066f90 0x7fad20069450 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fad340af000 tx=0x7fad340a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:45.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.442+0000 7fad3d600640 1 -- 172.21.15.67:0/3384061589 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fad2c054d80 con 0x7fad4015dd10 2024-04-17T10:36:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad40154c00 con 0x7fad4015dd10 2024-04-17T10:36:45.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad3d600640 1 -- 172.21.15.67:0/3384061589 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fad2c05b0d0 con 0x7fad4015dd10 2024-04-17T10:36:45.632 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:45.632 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad20066f90 msgr2=0x7fad20069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad20066f90 0x7fad20069450 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fad340af000 tx=0x7fad340a7000 comp rx=0 tx=0).stop 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 msgr2=0x7fad40154e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad40154e20 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00c420 tx=0x7fad2c00c8e0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 shutdown_connections 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fad20066f90 0x7fad20069450 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fad400cd6e0 0x7fad401617c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad4015dd10 0x7fad40154e20 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 --2- 172.21.15.67:0/3384061589 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fad400714d0 0x7fad4014b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 >> 172.21.15.67:0/3384061589 conn(0x7fad400b34d0 msgr2=0x7fad4014cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 shutdown_connections 2024-04-17T10:36:45.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:45.642+0000 7fad47e00640 1 -- 172.21.15.67:0/3384061589 wait complete. 2024-04-17T10:36:46.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:46.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:47.057 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:47.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.217+0000 7f3ceaa00640 1 Processor -- start 2024-04-17T10:36:47.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.217+0000 7f3ceaa00640 1 -- start start 2024-04-17T10:36:47.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.217+0000 7f3ceaa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ce4099a50 0x7f3ce415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:47.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.217+0000 7f3ceaa00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce40cde20 con 0x7f3ce409a740 2024-04-17T10:36:47.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.217+0000 7f3ceaa00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ce40cf300 con 0x7f3ce40d03a0 2024-04-17T10:36:47.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.217+0000 7f3ceaa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ce40cf5e0 con 0x7f3ce4099a50 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3cea000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3ce409a740 0x7f3ce40cc1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52788/0 (socket says 172.21.15.67:52788) 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce9600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ce4099a50 0x7f3ce415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3cea000640 1 -- 172.21.15.67:0/284764731 learned_addr learned my addr 172.21.15.67:0/284764731 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce3e00640 1 -- 172.21.15.67:0/284764731 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263215530 0 0) 0x7f3ce40cde20 con 0x7f3ce409a740 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce3e00640 1 -- 172.21.15.67:0/284764731 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc003530 con 0x7f3ce409a740 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce9600640 1 -- 172.21.15.67:0/284764731 >> v1:172.21.15.86:6789/0 conn(0x7f3ce40d03a0 legacy=0x7f3ce4099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce3e00640 1 -- 172.21.15.67:0/284764731 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383701323 0 0) 0x7f3ce40cf300 con 0x7f3ce40d03a0 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce9600640 1 -- 172.21.15.67:0/284764731 >> v1:172.21.15.67:6789/0 conn(0x7f3ce409a740 legacy=0x7f3ce40cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:47.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce9600640 1 -- 172.21.15.67:0/284764731 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce40cf8c0 con 0x7f3ce4099a50 2024-04-17T10:36:47.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce9600640 1 --2- 172.21.15.67:0/284764731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce4099a50 0x7f3ce415c7a0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80097c0 tx=0x7f3cd801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a945dc422dea2be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:47.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce3e00640 1 -- 172.21.15.67:0/284764731 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd80b0070 con 0x7f3ce4099a50 2024-04-17T10:36:47.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce3e00640 1 -- 172.21.15.67:0/284764731 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd80ab600 con 0x7f3ce4099a50 2024-04-17T10:36:47.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.221+0000 7f3ce3e00640 1 -- 172.21.15.67:0/284764731 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd80ab9d0 con 0x7f3ce4099a50 2024-04-17T10:36:47.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- 172.21.15.67:0/284764731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce4099a50 msgr2=0x7f3ce415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:47.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/284764731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce4099a50 0x7f3ce415c7a0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80097c0 tx=0x7f3cd801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- 172.21.15.67:0/284764731 shutdown_connections 2024-04-17T10:36:47.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/284764731 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce4099a50 0x7f3ce415c7a0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- 172.21.15.67:0/284764731 >> 172.21.15.67:0/284764731 conn(0x7f3ce40d1690 msgr2=0x7f3ce40d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:47.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- 172.21.15.67:0/284764731 shutdown_connections 2024-04-17T10:36:47.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- 172.21.15.67:0/284764731 wait complete. 2024-04-17T10:36:47.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 Processor -- start 2024-04-17T10:36:47.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- start start 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce4099a50 0x7f3ce4155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce409a740 0x7f3ce40b48f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce40d03a0 0x7f3ce40b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce4169a50 con 0x7f3ce409a740 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce41698d0 con 0x7f3ce4099a50 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ceaa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce4169bd0 con 0x7f3ce40d03a0 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ce8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce409a740 0x7f3ce40b48f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ce8c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce409a740 0x7f3ce40b48f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34326/0 (socket says 172.21.15.67:34326) 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ce8c00640 1 -- 172.21.15.67:0/3256850236 learned_addr learned my addr 172.21.15.67:0/3256850236 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3ce9600640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce4099a50 0x7f3ce4155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.225+0000 7f3cea000640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce40d03a0 0x7f3ce40b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce8c00640 1 -- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce40d03a0 msgr2=0x7f3ce40b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:47.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce8c00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce40d03a0 0x7f3ce40b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce8c00640 1 -- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce4099a50 msgr2=0x7f3ce4155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:47.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce8c00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce4099a50 0x7f3ce4155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce8c00640 1 -- 172.21.15.67:0/3256850236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce406c850 con 0x7f3ce409a740 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce8c00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce409a740 0x7f3ce40b48f0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000c480 tx=0x7f3cd000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce1600640 1 -- 172.21.15.67:0/3256850236 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd0005480 con 0x7f3ce409a740 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce1600640 1 -- 172.21.15.67:0/3256850236 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd0003600 con 0x7f3ce409a740 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce1600640 1 -- 172.21.15.67:0/3256850236 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd00039f0 con 0x7f3ce409a740 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce406caf0 con 0x7f3ce409a740 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce406d0b0 con 0x7f3ce409a740 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce4074dc0 con 0x7f3ce409a740 2024-04-17T10:36:47.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce1600640 1 -- 172.21.15.67:0/3256850236 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3cd0003b90 con 0x7f3ce409a740 2024-04-17T10:36:47.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce1600640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:47.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.229+0000 7f3ce1600640 1 -- 172.21.15.67:0/3256850236 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f3cd0084da0 con 0x7f3ce409a740 2024-04-17T10:36:47.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.233+0000 7f3ce9600640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:47.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.233+0000 7f3ce9600640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80ad000 tx=0x7f3cd80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:47.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.233+0000 7f3ce1600640 1 -- 172.21.15.67:0/3256850236 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3cd0055530 con 0x7f3ce409a740 2024-04-17T10:36:47.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.429+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ce40d24a0 con 0x7f3ce409a740 2024-04-17T10:36:47.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.429+0000 7f3ce1600640 1 -- 172.21.15.67:0/3256850236 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3cd00595d0 con 0x7f3ce409a740 2024-04-17T10:36:47.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:47.421 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:47.422 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:47.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 msgr2=0x7f3cc4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:47.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80ad000 tx=0x7f3cd80a6000 comp rx=0 tx=0).stop 2024-04-17T10:36:47.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce409a740 msgr2=0x7f3ce40b48f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:47.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce409a740 0x7f3ce40b48f0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f3cd000c480 tx=0x7f3cd000c940 comp rx=0 tx=0).stop 2024-04-17T10:36:47.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 shutdown_connections 2024-04-17T10:36:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3cc4066f60 0x7f3cc4069420 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ce40d03a0 0x7f3ce40b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ce409a740 0x7f3ce40b48f0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 --2- 172.21.15.67:0/3256850236 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ce4099a50 0x7f3ce4155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 >> 172.21.15.67:0/3256850236 conn(0x7f3ce40d1690 msgr2=0x7f3ce40af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 shutdown_connections 2024-04-17T10:36:47.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:47.433+0000 7f3ceaa00640 1 -- 172.21.15.67:0/3256850236 wait complete. 2024-04-17T10:36:47.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:47.845 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:48.846 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.009+0000 7fa7c5200640 1 Processor -- start 2024-04-17T10:36:49.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.009+0000 7fa7c5200640 1 -- start start 2024-04-17T10:36:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.009+0000 7fa7c5200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa7c0145040 0x7fa7c006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.009+0000 7fa7c5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c0071a20 0x7fa7c00cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.009+0000 7fa7c5200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7c006e2f0 con 0x7fa7c006ef10 2024-04-17T10:36:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.009+0000 7fa7c5200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7c00a86c0 con 0x7fa7c0145040 2024-04-17T10:36:49.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.009+0000 7fa7c5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c015bf10 con 0x7fa7c0071a20 2024-04-17T10:36:49.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7be800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c0071a20 0x7fa7c00cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:49.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7bde00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa7c0145040 0x7fa7c006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:49.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7be800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c0071a20 0x7fa7c00cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:39096/0 (socket says 172.21.15.67:39096) 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7be800640 1 -- 172.21.15.67:0/1481835093 learned_addr learned my addr 172.21.15.67:0/1481835093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7b5400640 1 --1- 172.21.15.67:0/1481835093 >> v1:172.21.15.86:6789/0 conn(0x7fa7c006ef10 0x7fa7c0071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:45456/0 (socket says 172.21.15.67:45456) 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7bde00640 1 -- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c0071a20 msgr2=0x7fa7c00cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7bde00640 1 --2- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c0071a20 0x7fa7c00cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7bde00640 1 -- 172.21.15.67:0/1481835093 >> v1:172.21.15.86:6789/0 conn(0x7fa7c006ef10 legacy=0x7fa7c0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7bde00640 1 -- 172.21.15.67:0/1481835093 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c006d6e0 con 0x7fa7c0145040 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7be800640 1 --2- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c0071a20 0x7fa7c00cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:49.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7bd400640 1 -- 172.21.15.67:0/1481835093 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588890031 0 0) 0x7fa7c006e2f0 con 0x7fa7c006ef10 2024-04-17T10:36:49.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.013+0000 7fa7bde00640 1 --2- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 0x7fa7c006e9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00097c0 tx=0x7fa7b001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=249670c40f318d7c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7bd400640 1 -- 172.21.15.67:0/1481835093 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b00b5070 con 0x7fa7c0145040 2024-04-17T10:36:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7bd400640 1 -- 172.21.15.67:0/1481835093 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b00a8990 con 0x7fa7c0145040 2024-04-17T10:36:49.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7bd400640 1 -- 172.21.15.67:0/1481835093 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b00a8d60 con 0x7fa7c0145040 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 msgr2=0x7fa7c006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 --2- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 0x7fa7c006e9d0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00097c0 tx=0x7fa7b001ec90 comp rx=0 tx=0).stop 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- 172.21.15.67:0/1481835093 shutdown_connections 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 --2- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c0071a20 0x7fa7c00cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 --2- 172.21.15.67:0/1481835093 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 0x7fa7c006e9d0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- 172.21.15.67:0/1481835093 >> 172.21.15.67:0/1481835093 conn(0x7fa7c006ac90 msgr2=0x7fa7c006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- 172.21.15.67:0/1481835093 shutdown_connections 2024-04-17T10:36:49.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- 172.21.15.67:0/1481835093 wait complete. 2024-04-17T10:36:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 Processor -- start 2024-04-17T10:36:49.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- start start 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c006ef10 0x7fa7c015aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0071a20 0x7fa7c00b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 0x7fa7c00b4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006df80 con 0x7fa7c0071a20 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006dca0 con 0x7fa7c0145040 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7c5200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c006d370 con 0x7fa7c006ef10 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7b5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0071a20 0x7fa7c00b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7b5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0071a20 0x7fa7c00b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:34334/0 (socket says 172.21.15.67:34334) 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7b5400640 1 -- 172.21.15.67:0/3477540669 learned_addr learned my addr 172.21.15.67:0/3477540669 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:49.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7bde00640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c006ef10 0x7fa7c015aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.017+0000 7fa7be800640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 0x7fa7c00b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b5400640 1 -- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c006ef10 msgr2=0x7fa7c015aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b5400640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c006ef10 0x7fa7c015aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b5400640 1 -- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 msgr2=0x7fa7c00b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b5400640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 0x7fa7c00b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b5400640 1 -- 172.21.15.67:0/3477540669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c01f7860 con 0x7fa7c0071a20 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b5400640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0071a20 0x7fa7c00b3ee0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80011f0 tx=0x7fa7a800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b6a00640 1 -- 172.21.15.67:0/3477540669 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8018070 con 0x7fa7c0071a20 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b6a00640 1 -- 172.21.15.67:0/3477540669 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a80074e0 con 0x7fa7c0071a20 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b6a00640 1 -- 172.21.15.67:0/3477540669 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a8004a10 con 0x7fa7c0071a20 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c00adba0 con 0x7fa7c0071a20 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c00ae160 con 0x7fa7c0071a20 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c0072d50 con 0x7fa7c0071a20 2024-04-17T10:36:49.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b6a00640 1 -- 172.21.15.67:0/3477540669 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa7a8004bb0 con 0x7fa7c0071a20 2024-04-17T10:36:49.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b6a00640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa79c066f90 0x7fa79c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.021+0000 7fa7b6a00640 1 -- 172.21.15.67:0/3477540669 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa7a80844a0 con 0x7fa7c0071a20 2024-04-17T10:36:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.025+0000 7fa7bde00640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa79c066f90 0x7fa79c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:49.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.025+0000 7fa7bde00640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa79c066f90 0x7fa79c069450 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00b2000 tx=0x7fa7b00aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:49.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.029+0000 7fa7b6a00640 1 -- 172.21.15.67:0/3477540669 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa7a8054c40 con 0x7fa7c0071a20 2024-04-17T10:36:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.229+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7c00ae330 con 0x7fa7c0071a20 2024-04-17T10:36:49.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.229+0000 7fa7b6a00640 1 -- 172.21.15.67:0/3477540669 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fa7a8058ce0 con 0x7fa7c0071a20 2024-04-17T10:36:49.222 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:49.222 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:49.222 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:49.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa79c066f90 msgr2=0x7fa79c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:49.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa79c066f90 0x7fa79c069450 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fa7b00b2000 tx=0x7fa7b00aa070 comp rx=0 tx=0).stop 2024-04-17T10:36:49.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0071a20 msgr2=0x7fa7c00b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0071a20 0x7fa7c00b3ee0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fa7a80011f0 tx=0x7fa7a800c9b0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 shutdown_connections 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa79c066f90 0x7fa79c069450 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa7c0145040 0x7fa7c00b4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa7c0071a20 0x7fa7c00b3ee0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 --2- 172.21.15.67:0/3477540669 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7c006ef10 0x7fa7c015aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 >> 172.21.15.67:0/3477540669 conn(0x7fa7c006ac90 msgr2=0x7fa7c00ccd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 shutdown_connections 2024-04-17T10:36:49.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:49.233+0000 7fa7c5200640 1 -- 172.21.15.67:0/3477540669 wait complete. 2024-04-17T10:36:49.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:49.645 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:50.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:50.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.805+0000 7fdbfda00640 1 Processor -- start 2024-04-17T10:36:50.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.805+0000 7fdbfda00640 1 -- start start 2024-04-17T10:36:50.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbfda00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf815dbc0 0x7fdbf806f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:50.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbfda00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbf8072370 0x7fdbf81676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:50.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbfda00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf80ad8f0 con 0x7fdbf806db90 2024-04-17T10:36:50.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbfda00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf815dfa0 con 0x7fdbf8072370 2024-04-17T10:36:50.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbfda00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf806faa0 con 0x7fdbf815dbc0 2024-04-17T10:36:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbf8072370 0x7fdbf81676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf7400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdbf8072370 0x7fdbf81676d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50598/0 (socket says 172.21.15.67:50598) 2024-04-17T10:36:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf7400640 1 -- 172.21.15.67:0/209507812 learned_addr learned my addr 172.21.15.67:0/209507812 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf6a00640 1 --2- 172.21.15.67:0/209507812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf815dbc0 0x7fdbf806f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:50.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf7400640 1 -- 172.21.15.67:0/209507812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf815dbc0 msgr2=0x7fdbf806f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf7400640 1 --2- 172.21.15.67:0/209507812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf815dbc0 0x7fdbf806f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:50.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf7400640 1 -- 172.21.15.67:0/209507812 >> v1:172.21.15.86:6789/0 conn(0x7fdbf806db90 legacy=0x7fdbf8071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:50.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.809+0000 7fdbf7400640 1 -- 172.21.15.67:0/209507812 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80adbd0 con 0x7fdbf8072370 2024-04-17T10:36:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbf7400640 1 --2- 172.21.15.67:0/209507812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf8072370 0x7fdbf81676d0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009e50 tx=0x7fdbf001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de32dcfe41f1b9f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbf6000640 1 -- 172.21.15.67:0/209507812 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00a5660 con 0x7fdbf8072370 2024-04-17T10:36:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbf6000640 1 -- 172.21.15.67:0/209507812 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf00a5800 con 0x7fdbf8072370 2024-04-17T10:36:50.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbf6000640 1 -- 172.21.15.67:0/209507812 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf00a5bd0 con 0x7fdbf8072370 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 -- 172.21.15.67:0/209507812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf8072370 msgr2=0x7fdbf81676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 --2- 172.21.15.67:0/209507812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf8072370 0x7fdbf81676d0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009e50 tx=0x7fdbf001c9a0 comp rx=0 tx=0).stop 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 -- 172.21.15.67:0/209507812 shutdown_connections 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 --2- 172.21.15.67:0/209507812 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf8072370 0x7fdbf81676d0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 --2- 172.21.15.67:0/209507812 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf815dbc0 0x7fdbf806f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 -- 172.21.15.67:0/209507812 >> 172.21.15.67:0/209507812 conn(0x7fdbf80af670 msgr2=0x7fdbf80b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 -- 172.21.15.67:0/209507812 shutdown_connections 2024-04-17T10:36:50.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 -- 172.21.15.67:0/209507812 wait complete. 2024-04-17T10:36:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 Processor -- start 2024-04-17T10:36:50.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.813+0000 7fdbfda00640 1 -- start start 2024-04-17T10:36:50.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf806db90 0x7fdbf8157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf8072370 0x7fdbf8149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbf815dbc0 0x7fdbf80cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf816a7c0 con 0x7fdbf806db90 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf816a640 con 0x7fdbf815dbc0 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf816a940 con 0x7fdbf8072370 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf806db90 0x7fdbf8157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf806db90 0x7fdbf8157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50602/0 (socket says 172.21.15.67:50602) 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 -- 172.21.15.67:0/2695101684 learned_addr learned my addr 172.21.15.67:0/2695101684 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbefe00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf8072370 0x7fdbf8149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf7400640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbf815dbc0 0x7fdbf80cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 -- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf8072370 msgr2=0x7fdbf8149aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf8072370 0x7fdbf8149aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:50.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 -- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbf815dbc0 msgr2=0x7fdbf80cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbf815dbc0 0x7fdbf80cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 -- 172.21.15.67:0/2695101684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf80cca80 con 0x7fdbf806db90 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbefe00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf8072370 0x7fdbf8149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf7400640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbf815dbc0 0x7fdbf80cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbf6a00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf806db90 0x7fdbf8157380 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800e870 tx=0x7fdbe800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbeea00640 1 -- 172.21.15.67:0/2695101684 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe8017070 con 0x7fdbf806db90 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbeea00640 1 -- 172.21.15.67:0/2695101684 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe800c940 con 0x7fdbf806db90 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbeea00640 1 -- 172.21.15.67:0/2695101684 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe800ccf0 con 0x7fdbf806db90 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf80ccd60 con 0x7fdbf806db90 2024-04-17T10:36:50.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf80cd320 con 0x7fdbf806db90 2024-04-17T10:36:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf809e260 con 0x7fdbf806db90 2024-04-17T10:36:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.817+0000 7fdbeea00640 1 -- 172.21.15.67:0/2695101684 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdbe801a070 con 0x7fdbf806db90 2024-04-17T10:36:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.821+0000 7fdbeea00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:50.811 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.821+0000 7fdbeea00640 1 -- 172.21.15.67:0/2695101684 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fdbe80855a0 con 0x7fdbf806db90 2024-04-17T10:36:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.821+0000 7fdbefe00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:50.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.821+0000 7fdbefe00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8156c50 tx=0x7fdbe0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:50.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:50.825+0000 7fdbeea00640 1 -- 172.21.15.67:0/2695101684 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdbe8056260 con 0x7fdbf806db90 2024-04-17T10:36:51.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.021+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbf8074e30 con 0x7fdbf806db90 2024-04-17T10:36:51.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.021+0000 7fdbeea00640 1 -- 172.21.15.67:0/2695101684 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fdbe8055b60 con 0x7fdbf806db90 2024-04-17T10:36:51.015 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:51.015 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:51.015 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:51.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 msgr2=0x7fdbd4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:51.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8156c50 tx=0x7fdbe0009040 comp rx=0 tx=0).stop 2024-04-17T10:36:51.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf806db90 msgr2=0x7fdbf8157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:51.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf806db90 0x7fdbf8157380 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fdbe800e870 tx=0x7fdbe800ec70 comp rx=0 tx=0).stop 2024-04-17T10:36:51.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 shutdown_connections 2024-04-17T10:36:51.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fdbd4066f60 0x7fdbd4069420 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:51.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdbf815dbc0 0x7fdbf80cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:51.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdbf8072370 0x7fdbf8149aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:51.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 --2- 172.21.15.67:0/2695101684 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdbf806db90 0x7fdbf8157380 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:51.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 >> 172.21.15.67:0/2695101684 conn(0x7fdbf80af670 msgr2=0x7fdbf80b4ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:51.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 shutdown_connections 2024-04-17T10:36:51.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:51.025+0000 7fdbfda00640 1 -- 172.21.15.67:0/2695101684 wait complete. 2024-04-17T10:36:51.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:51.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:52.440 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:52.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.601+0000 7ff7d4a00640 1 Processor -- start 2024-04-17T10:36:52.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.601+0000 7ff7d4a00640 1 -- start start 2024-04-17T10:36:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.601+0000 7ff7d4a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff7d0145040 0x7ff7d006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.601+0000 7ff7d4a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d006e2f0 con 0x7ff7d0071a20 2024-04-17T10:36:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.601+0000 7ff7d4a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d006d6e0 con 0x7ff7d006ef10 2024-04-17T10:36:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.601+0000 7ff7d4a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7d00a86c0 con 0x7ff7d0145040 2024-04-17T10:36:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7ce800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff7d0071a20 0x7ff7d00cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44748/0 (socket says 172.21.15.67:44748) 2024-04-17T10:36:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7ce800640 1 -- 172.21.15.67:0/557574573 learned_addr learned my addr 172.21.15.67:0/557574573 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:52.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cde00640 1 --2- 172.21.15.67:0/557574573 >> v2:172.21.15.86:3300/0 conn(0x7ff7d0145040 0x7ff7d006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:52.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cd400640 1 -- 172.21.15.67:0/557574573 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4152826325 0 0) 0x7ff7d006e2f0 con 0x7ff7d0071a20 2024-04-17T10:36:52.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cd400640 1 -- 172.21.15.67:0/557574573 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b4003530 con 0x7ff7d0071a20 2024-04-17T10:36:52.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cde00640 1 -- 172.21.15.67:0/557574573 >> v1:172.21.15.86:6789/0 conn(0x7ff7d006ef10 legacy=0x7ff7d0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:52.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cde00640 1 -- 172.21.15.67:0/557574573 >> v1:172.21.15.67:6789/0 conn(0x7ff7d0071a20 legacy=0x7ff7d00cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:52.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cde00640 1 -- 172.21.15.67:0/557574573 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d006d9c0 con 0x7ff7d0145040 2024-04-17T10:36:52.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cde00640 1 --2- 172.21.15.67:0/557574573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 0x7ff7d006e9d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00097c0 tx=0x7ff7c001ec90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ff43b54b6f81530 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cd400640 1 -- 172.21.15.67:0/557574573 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00b5070 con 0x7ff7d0145040 2024-04-17T10:36:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cd400640 1 -- 172.21.15.67:0/557574573 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00a8990 con 0x7ff7d0145040 2024-04-17T10:36:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.605+0000 7ff7cd400640 1 -- 172.21.15.67:0/557574573 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00a8d40 con 0x7ff7d0145040 2024-04-17T10:36:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- 172.21.15.67:0/557574573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 msgr2=0x7ff7d006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/557574573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 0x7ff7d006e9d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00097c0 tx=0x7ff7c001ec90 comp rx=0 tx=0).stop 2024-04-17T10:36:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- 172.21.15.67:0/557574573 shutdown_connections 2024-04-17T10:36:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/557574573 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 0x7ff7d006e9d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- 172.21.15.67:0/557574573 >> 172.21.15.67:0/557574573 conn(0x7ff7d006ac90 msgr2=0x7ff7d006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- 172.21.15.67:0/557574573 shutdown_connections 2024-04-17T10:36:52.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- 172.21.15.67:0/557574573 wait complete. 2024-04-17T10:36:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 Processor -- start 2024-04-17T10:36:52.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- start start 2024-04-17T10:36:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d006ef10 0x7ff7d00b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d0071a20 0x7ff7d01f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:52.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 0x7ff7d01f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d006df80 con 0x7ff7d0071a20 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d015bf10 con 0x7ff7d0145040 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7d4a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d006d370 con 0x7ff7d006ef10 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7c5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d0071a20 0x7ff7d01f6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7c5400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d0071a20 0x7ff7d01f6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50604/0 (socket says 172.21.15.67:50604) 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7c5400640 1 -- 172.21.15.67:0/2317081554 learned_addr learned my addr 172.21.15.67:0/2317081554 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7cde00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d006ef10 0x7ff7d00b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.609+0000 7ff7ce800640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 0x7ff7d01f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7cde00640 1 -- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 msgr2=0x7ff7d01f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7cde00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 0x7ff7d01f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7cde00640 1 -- 172.21.15.67:0/2317081554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d0071a20 msgr2=0x7ff7d01f6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7cde00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d0071a20 0x7ff7d01f6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:52.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7cde00640 1 -- 172.21.15.67:0/2317081554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d0139990 con 0x7ff7d006ef10 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7c5400640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d0071a20 0x7ff7d01f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7cde00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d006ef10 0x7ff7d00b3280 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00af000 tx=0x7ff7c001ba30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7c6a00640 1 -- 172.21.15.67:0/2317081554 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00bf070 con 0x7ff7d006ef10 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7c6a00640 1 -- 172.21.15.67:0/2317081554 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00b0700 con 0x7ff7d006ef10 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d0139c70 con 0x7ff7d006ef10 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7c6a00640 1 -- 172.21.15.67:0/2317081554 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00b5040 con 0x7ff7d006ef10 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d00ae3d0 con 0x7ff7d006ef10 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d0072d50 con 0x7ff7d006ef10 2024-04-17T10:36:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7c6a00640 1 -- 172.21.15.67:0/2317081554 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff7c00c2070 con 0x7ff7d006ef10 2024-04-17T10:36:52.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7c6a00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7ac066f60 0x7ff7ac069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:52.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.613+0000 7ff7c6a00640 1 -- 172.21.15.67:0/2317081554 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff7c0121760 con 0x7ff7d006ef10 2024-04-17T10:36:52.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.617+0000 7ff7c5400640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7ac066f60 0x7ff7ac069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:52.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.617+0000 7ff7c5400640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7ac066f60 0x7ff7ac069420 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff7d00ace40 tx=0x7ff7b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:52.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.621+0000 7ff7c6a00640 1 -- 172.21.15.67:0/2317081554 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff7c00187f0 con 0x7ff7d006ef10 2024-04-17T10:36:52.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.813+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7d0074e30 con 0x7ff7d006ef10 2024-04-17T10:36:52.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7c6a00640 1 -- 172.21.15.67:0/2317081554 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7ff7c00f1ef0 con 0x7ff7d006ef10 2024-04-17T10:36:52.807 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:52.807 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:52.807 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:52.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7ac066f60 msgr2=0x7ff7ac069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:52.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7ac066f60 0x7ff7ac069420 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7ff7d00ace40 tx=0x7ff7b8009040 comp rx=0 tx=0).stop 2024-04-17T10:36:52.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d006ef10 msgr2=0x7ff7d00b3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:52.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d006ef10 0x7ff7d00b3280 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00af000 tx=0x7ff7c001ba30 comp rx=0 tx=0).stop 2024-04-17T10:36:52.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 shutdown_connections 2024-04-17T10:36:52.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff7ac066f60 0x7ff7ac069420 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff7d0145040 0x7ff7d01f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff7d0071a20 0x7ff7d01f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 --2- 172.21.15.67:0/2317081554 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff7d006ef10 0x7ff7d00b3280 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 >> 172.21.15.67:0/2317081554 conn(0x7ff7d006ac90 msgr2=0x7ff7d00afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 shutdown_connections 2024-04-17T10:36:52.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:52.817+0000 7ff7d4a00640 1 -- 172.21.15.67:0/2317081554 wait complete. 2024-04-17T10:36:53.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:53.232 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:54.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc325000640 1 Processor -- start 2024-04-17T10:36:54.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc325000640 1 -- start start 2024-04-17T10:36:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc325000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc32015dd10 0x7fc3200b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc325000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc32006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:54.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc325000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3200b0cc0 con 0x7fc3200b57b0 2024-04-17T10:36:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc325000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3200b1280 con 0x7fc32015dd10 2024-04-17T10:36:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc325000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3200b0fa0 con 0x7fc3200714d0 2024-04-17T10:36:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc317e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc3200b57b0 0x7fc320070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44756/0 (socket says 172.21.15.67:44756) 2024-04-17T10:36:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc32015dd10 0x7fc3200b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:54.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc317e00640 1 -- 172.21.15.67:0/815304253 learned_addr learned my addr 172.21.15.67:0/815304253 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31e800640 1 --2- 172.21.15.67:0/815304253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc32006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31de00640 1 -- 172.21.15.67:0/815304253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 msgr2=0x7fc32006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31de00640 1 --2- 172.21.15.67:0/815304253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc32006c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31de00640 1 -- 172.21.15.67:0/815304253 >> v1:172.21.15.67:6789/0 conn(0x7fc3200b57b0 legacy=0x7fc320070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31de00640 1 -- 172.21.15.67:0/815304253 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32006bed0 con 0x7fc32015dd10 2024-04-17T10:36:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31e800640 1 --2- 172.21.15.67:0/815304253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc32006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.397+0000 7fc31d400640 1 -- 172.21.15.67:0/815304253 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638924191 0 0) 0x7fc3200b0cc0 con 0x7fc3200b57b0 2024-04-17T10:36:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc31de00640 1 --2- 172.21.15.67:0/815304253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32015dd10 0x7fc3200b5270 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fc3100097c0 tx=0x7fc31001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd64fffe0e4756a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc31d400640 1 -- 172.21.15.67:0/815304253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100b1070 con 0x7fc32015dd10 2024-04-17T10:36:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc31d400640 1 -- 172.21.15.67:0/815304253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3100ac620 con 0x7fc32015dd10 2024-04-17T10:36:54.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc31d400640 1 -- 172.21.15.67:0/815304253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100ac9f0 con 0x7fc32015dd10 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 -- 172.21.15.67:0/815304253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32015dd10 msgr2=0x7fc3200b5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 --2- 172.21.15.67:0/815304253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32015dd10 0x7fc3200b5270 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fc3100097c0 tx=0x7fc31001c3e0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 -- 172.21.15.67:0/815304253 shutdown_connections 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 --2- 172.21.15.67:0/815304253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc32006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 --2- 172.21.15.67:0/815304253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc32015dd10 0x7fc3200b5270 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 -- 172.21.15.67:0/815304253 >> 172.21.15.67:0/815304253 conn(0x7fc3200b34d0 msgr2=0x7fc3200b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 -- 172.21.15.67:0/815304253 shutdown_connections 2024-04-17T10:36:54.394 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.401+0000 7fc325000640 1 -- 172.21.15.67:0/815304253 wait complete. 2024-04-17T10:36:54.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 Processor -- start 2024-04-17T10:36:54.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 -- start start 2024-04-17T10:36:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc320154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32015dd10 0x7fc320160550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320149c90 0x7fc3200ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:54.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32006d7b0 con 0x7fc320149c90 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32006d630 con 0x7fc32015dd10 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32006d930 con 0x7fc3200714d0 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320149c90 0x7fc3200ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320149c90 0x7fc3200ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50614/0 (socket says 172.21.15.67:50614) 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 -- 172.21.15.67:0/2141082598 learned_addr learned my addr 172.21.15.67:0/2141082598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 -- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 msgr2=0x7fc320154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc320154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 -- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32015dd10 msgr2=0x7fc320160550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32015dd10 0x7fc320160550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 -- 172.21.15.67:0/2141082598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc320154980 con 0x7fc320149c90 2024-04-17T10:36:54.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc31e800640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320149c90 0x7fc3200ccfa0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fc318007e70 tx=0x7fc31800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc316000640 1 -- 172.21.15.67:0/2141082598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31801a070 con 0x7fc320149c90 2024-04-17T10:36:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc316000640 1 -- 172.21.15.67:0/2141082598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc318015600 con 0x7fc320149c90 2024-04-17T10:36:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc316000640 1 -- 172.21.15.67:0/2141082598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3180159f0 con 0x7fc320149c90 2024-04-17T10:36:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc320160ea0 con 0x7fc320149c90 2024-04-17T10:36:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.405+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3200cad40 con 0x7fc320149c90 2024-04-17T10:36:54.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.409+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc320070bb0 con 0x7fc320149c90 2024-04-17T10:36:54.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.409+0000 7fc316000640 1 -- 172.21.15.67:0/2141082598 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc31801d070 con 0x7fc320149c90 2024-04-17T10:36:54.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.409+0000 7fc316000640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc2fc066f60 0x7fc2fc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:54.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.409+0000 7fc316000640 1 -- 172.21.15.67:0/2141082598 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fc3180855c0 con 0x7fc320149c90 2024-04-17T10:36:54.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.409+0000 7fc31de00640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc2fc066f60 0x7fc2fc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:54.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.409+0000 7fc31de00640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc2fc066f60 0x7fc2fc069420 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc3100ae000 tx=0x7fc310004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:54.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.413+0000 7fc316000640 1 -- 172.21.15.67:0/2141082598 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc318010070 con 0x7fc320149c90 2024-04-17T10:36:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.613+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3200b41a0 con 0x7fc320149c90 2024-04-17T10:36:54.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.613+0000 7fc316000640 1 -- 172.21.15.67:0/2141082598 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fc318056310 con 0x7fc320149c90 2024-04-17T10:36:54.605 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:54.606 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc2fc066f60 msgr2=0x7fc2fc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc2fc066f60 0x7fc2fc069420 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc3100ae000 tx=0x7fc310004740 comp rx=0 tx=0).stop 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320149c90 msgr2=0x7fc3200ccfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320149c90 0x7fc3200ccfa0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fc318007e70 tx=0x7fc31800d4a0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 shutdown_connections 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc2fc066f60 0x7fc2fc069420 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc320149c90 0x7fc3200ccfa0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc32015dd10 0x7fc320160550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 --2- 172.21.15.67:0/2141082598 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc3200714d0 0x7fc320154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 >> 172.21.15.67:0/2141082598 conn(0x7fc3200b34d0 msgr2=0x7fc3200cf220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:54.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 shutdown_connections 2024-04-17T10:36:54.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:54.617+0000 7fc325000640 1 -- 172.21.15.67:0/2141082598 wait complete. 2024-04-17T10:36:55.029 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:55.030 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:56.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f6804e00640 1 Processor -- start 2024-04-17T10:36:56.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f6804e00640 1 -- start start 2024-04-17T10:36:56.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f6804e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6800145040 0x7f680006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f6804e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f68000cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f6804e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f680006e2f0 con 0x7f680006ef10 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f6804e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68000a86c0 con 0x7f6800145040 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f6804e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680015bf10 con 0x7f6800071a20 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f67fde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6800145040 0x7f680006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f67f7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f680006ef10 0x7f6800071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44768/0 (socket says 172.21.15.67:44768) 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f67f7e00640 1 -- 172.21.15.67:0/156316099 learned_addr learned my addr 172.21.15.67:0/156316099 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.197+0000 7f67fe800640 1 --2- 172.21.15.67:0/156316099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f68000cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fde00640 1 -- 172.21.15.67:0/156316099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 msgr2=0x7f68000cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fde00640 1 --2- 172.21.15.67:0/156316099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f68000cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fde00640 1 -- 172.21.15.67:0/156316099 >> v1:172.21.15.67:6789/0 conn(0x7f680006ef10 legacy=0x7f6800071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fde00640 1 -- 172.21.15.67:0/156316099 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680006d6e0 con 0x7f6800145040 2024-04-17T10:36:56.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fd400640 1 -- 172.21.15.67:0/156316099 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281031965 0 0) 0x7f680006e2f0 con 0x7f680006ef10 2024-04-17T10:36:56.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fde00640 1 --2- 172.21.15.67:0/156316099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6800145040 0x7f680006e9d0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f67f00097c0 tx=0x7f67f001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a491fe0a474cda9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:56.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fd400640 1 -- 172.21.15.67:0/156316099 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00b5070 con 0x7f6800145040 2024-04-17T10:36:56.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fd400640 1 -- 172.21.15.67:0/156316099 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f00a8990 con 0x7f6800145040 2024-04-17T10:36:56.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f67fd400640 1 -- 172.21.15.67:0/156316099 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00a8d60 con 0x7f6800145040 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 -- 172.21.15.67:0/156316099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6800145040 msgr2=0x7f680006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 --2- 172.21.15.67:0/156316099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6800145040 0x7f680006e9d0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f67f00097c0 tx=0x7f67f001ec90 comp rx=0 tx=0).stop 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 -- 172.21.15.67:0/156316099 shutdown_connections 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 --2- 172.21.15.67:0/156316099 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f68000cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 --2- 172.21.15.67:0/156316099 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6800145040 0x7f680006e9d0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 -- 172.21.15.67:0/156316099 >> 172.21.15.67:0/156316099 conn(0x7f680006ac90 msgr2=0x7f680006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 -- 172.21.15.67:0/156316099 shutdown_connections 2024-04-17T10:36:56.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.201+0000 7f6804e00640 1 -- 172.21.15.67:0/156316099 wait complete. 2024-04-17T10:36:56.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 Processor -- start 2024-04-17T10:36:56.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 -- start start 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006ef10 0x7f680015df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f680015f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6800145040 0x7f680015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680006df80 con 0x7f680006ef10 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680006dca0 con 0x7f6800145040 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f680006d370 con 0x7f6800071a20 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006ef10 0x7f680015df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:56.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006ef10 0x7f680015df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50626/0 (socket says 172.21.15.67:50626) 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 -- 172.21.15.67:0/1295647584 learned_addr learned my addr 172.21.15.67:0/1295647584 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fe800640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6800145040 0x7f680015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67f7e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f680015f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 -- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 msgr2=0x7f680015f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f680015f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 -- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6800145040 msgr2=0x7f680015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6800145040 0x7f680015acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 -- 172.21.15.67:0/1295647584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6800134a80 con 0x7f680006ef10 2024-04-17T10:36:56.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67fde00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006ef10 0x7f680015df40 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b3040 tx=0x7f67f001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67f6000640 1 -- 172.21.15.67:0/1295647584 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00bf070 con 0x7f680006ef10 2024-04-17T10:36:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67f6000640 1 -- 172.21.15.67:0/1295647584 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f00b07a0 con 0x7f680006ef10 2024-04-17T10:36:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680015e480 con 0x7f680006ef10 2024-04-17T10:36:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f67f6000640 1 -- 172.21.15.67:0/1295647584 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00b5040 con 0x7f680006ef10 2024-04-17T10:36:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.205+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68000b3c70 con 0x7f680006ef10 2024-04-17T10:36:56.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.209+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6800072d50 con 0x7f680006ef10 2024-04-17T10:36:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.209+0000 7f67f6000640 1 -- 172.21.15.67:0/1295647584 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f67f00c2070 con 0x7f680006ef10 2024-04-17T10:36:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.209+0000 7f67f6000640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f67dc066f60 0x7f67dc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:56.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.209+0000 7f67f6000640 1 -- 172.21.15.67:0/1295647584 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f67f0121a30 con 0x7f680006ef10 2024-04-17T10:36:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.209+0000 7f67f7e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f67dc066f60 0x7f67dc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:56.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.209+0000 7f67f7e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f67dc066f60 0x7f67dc069420 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f680013f700 tx=0x7f67e8007b60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:56.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.213+0000 7f67f6000640 1 -- 172.21.15.67:0/1295647584 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f67f00187f0 con 0x7f680006ef10 2024-04-17T10:36:56.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.409+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6800074e30 con 0x7f680006ef10 2024-04-17T10:36:56.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.409+0000 7f67f6000640 1 -- 172.21.15.67:0/1295647584 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f67f00f2780 con 0x7f680006ef10 2024-04-17T10:36:56.402 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:56.403 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:56.403 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f67dc066f60 msgr2=0x7f67dc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f67dc066f60 0x7f67dc069420 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f680013f700 tx=0x7f67e8007b60 comp rx=0 tx=0).stop 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006ef10 msgr2=0x7f680015df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006ef10 0x7f680015df40 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b3040 tx=0x7f67f001ba30 comp rx=0 tx=0).stop 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 shutdown_connections 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f67dc066f60 0x7f67dc069420 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6800145040 0x7f680015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6800071a20 0x7f680015f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 --2- 172.21.15.67:0/1295647584 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f680006ef10 0x7f680015df40 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 >> 172.21.15.67:0/1295647584 conn(0x7f680006ac90 msgr2=0x7f680015e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 shutdown_connections 2024-04-17T10:36:56.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:56.413+0000 7f6804e00640 1 -- 172.21.15.67:0/1295647584 wait complete. 2024-04-17T10:36:56.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:56.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:57.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e19400640 1 Processor -- start 2024-04-17T10:36:57.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e19400640 1 -- start start 2024-04-17T10:36:57.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e19400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e1406eed0 0x7f2e140cd720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e19400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1406fbd0 con 0x7f2e1406b3a0 2024-04-17T10:36:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e19400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1406fe70 con 0x7f2e14144980 2024-04-17T10:36:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e19400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e140707a0 con 0x7f2e1406eed0 2024-04-17T10:36:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e18a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e1406eed0 0x7f2e140cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e0b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2e1406b3a0 0x7f2e140cd010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44776/0 (socket says 172.21.15.67:44776) 2024-04-17T10:36:57.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.989+0000 7f2e0b400640 1 -- 172.21.15.67:0/1236263420 learned_addr learned my addr 172.21.15.67:0/1236263420 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e18a00640 1 -- 172.21.15.67:0/1236263420 >> v1:172.21.15.86:6789/0 conn(0x7f2e14144980 legacy=0x7f2e1406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e18a00640 1 -- 172.21.15.67:0/1236263420 >> v1:172.21.15.67:6789/0 conn(0x7f2e1406b3a0 legacy=0x7f2e140cd010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e18a00640 1 -- 172.21.15.67:0/1236263420 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14070150 con 0x7f2e1406eed0 2024-04-17T10:36:57.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e13400640 1 -- 172.21.15.67:0/1236263420 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299434159 0 0) 0x7f2e1406fbd0 con 0x7f2e1406b3a0 2024-04-17T10:36:57.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e18a00640 1 --2- 172.21.15.67:0/1236263420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e140cd720 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0097c0 tx=0x7f2e0c01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2762bab6e8164999 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e13400640 1 -- 172.21.15.67:0/1236263420 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c0b5070 con 0x7f2e1406eed0 2024-04-17T10:36:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e13400640 1 -- 172.21.15.67:0/1236263420 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c0a8990 con 0x7f2e1406eed0 2024-04-17T10:36:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.993+0000 7f2e13400640 1 -- 172.21.15.67:0/1236263420 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c0a8d60 con 0x7f2e1406eed0 2024-04-17T10:36:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- 172.21.15.67:0/1236263420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 msgr2=0x7f2e140cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 --2- 172.21.15.67:0/1236263420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e140cd720 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c0097c0 tx=0x7f2e0c01ec90 comp rx=0 tx=0).stop 2024-04-17T10:36:57.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- 172.21.15.67:0/1236263420 shutdown_connections 2024-04-17T10:36:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 --2- 172.21.15.67:0/1236263420 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e140cd720 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- 172.21.15.67:0/1236263420 >> 172.21.15.67:0/1236263420 conn(0x7f2e1406e3b0 msgr2=0x7f2e1406e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- 172.21.15.67:0/1236263420 shutdown_connections 2024-04-17T10:36:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- 172.21.15.67:0/1236263420 wait complete. 2024-04-17T10:36:57.988 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 Processor -- start 2024-04-17T10:36:57.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- start start 2024-04-17T10:36:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1406b3a0 0x7f2e140acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e1415c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e14144980 0x7f2e1415f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1406ba40 con 0x7f2e1406eed0 2024-04-17T10:36:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14070c30 con 0x7f2e1406b3a0 2024-04-17T10:36:57.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e19400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1406bbc0 con 0x7f2e14144980 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e1415c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e1415c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50648/0 (socket says 172.21.15.67:50648) 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 -- 172.21.15.67:0/1697622008 learned_addr learned my addr 172.21.15.67:0/1697622008 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e13e00640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1406b3a0 0x7f2e140acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e18a00640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e14144980 0x7f2e1415f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 -- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e14144980 msgr2=0x7f2e1415f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e14144980 0x7f2e1415f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 -- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1406b3a0 msgr2=0x7f2e140acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1406b3a0 0x7f2e140acce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e0b400640 1 -- 172.21.15.67:0/1697622008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140ad5e0 con 0x7f2e1406eed0 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:57.997+0000 7f2e13e00640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1406b3a0 0x7f2e140acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e0b400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e1415c090 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c4a0 tx=0x7f2dfc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:57.992 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e10c00640 1 -- 172.21.15.67:0/1697622008 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc018070 con 0x7f2e1406eed0 2024-04-17T10:36:57.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e10c00640 1 -- 172.21.15.67:0/1697622008 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dfc0074e0 con 0x7f2e1406eed0 2024-04-17T10:36:57.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e10c00640 1 -- 172.21.15.67:0/1697622008 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc004a30 con 0x7f2e1406eed0 2024-04-17T10:36:57.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e140ad920 con 0x7f2e1406eed0 2024-04-17T10:36:57.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e140ae160 con 0x7f2e1406eed0 2024-04-17T10:36:57.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14074dc0 con 0x7f2e1406eed0 2024-04-17T10:36:57.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e10c00640 1 -- 172.21.15.67:0/1697622008 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2dfc01b070 con 0x7f2e1406eed0 2024-04-17T10:36:57.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e10c00640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2df0066f60 0x7f2df0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:57.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e10c00640 1 -- 172.21.15.67:0/1697622008 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f2dfc0845a0 con 0x7f2e1406eed0 2024-04-17T10:36:57.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.001+0000 7f2e13e00640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2df0066f60 0x7f2df0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:57.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.005+0000 7f2e13e00640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2df0066f60 0x7f2df0069420 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f2e040012f0 tx=0x7f2e04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:57.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.005+0000 7f2e10c00640 1 -- 172.21.15.67:0/1697622008 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2e140ad920 con 0x7f2e1406eed0 2024-04-17T10:36:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e14141a30 con 0x7f2e1406eed0 2024-04-17T10:36:58.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e10c00640 1 -- 172.21.15.67:0/1697622008 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f2dfc054ca0 con 0x7f2e1406eed0 2024-04-17T10:36:58.193 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:58.193 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:58.193 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2df0066f60 msgr2=0x7f2df0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2df0066f60 0x7f2df0069420 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f2e040012f0 tx=0x7f2e04009040 comp rx=0 tx=0).stop 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 msgr2=0x7f2e1415c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e1415c090 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c4a0 tx=0x7f2dfc00c960 comp rx=0 tx=0).stop 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 shutdown_connections 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2df0066f60 0x7f2df0069420 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2e14144980 0x7f2e1415f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e1406eed0 0x7f2e1415c090 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:58.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 --2- 172.21.15.67:0/1697622008 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2e1406b3a0 0x7f2e140acce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 >> 172.21.15.67:0/1697622008 conn(0x7f2e1406e3b0 msgr2=0x7f2e1406e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.201+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 shutdown_connections 2024-04-17T10:36:58.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:58.205+0000 7f2e19400640 1 -- 172.21.15.67:0/1697622008 wait complete. 2024-04-17T10:36:58.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:36:58.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:36:58.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:36:58.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:36:59.617 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:36:59.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.773+0000 7fe7f6800640 1 Processor -- start 2024-04-17T10:36:59.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.773+0000 7fe7f6800640 1 -- start start 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7f6800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7f0145040 0x7fe7f006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7f6800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe7f0071a20 0x7fe7f00cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7f6800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f006e2f0 con 0x7fe7f006ef10 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7f6800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f00a86c0 con 0x7fe7f0145040 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7f6800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f015bf10 con 0x7fe7f0071a20 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7ef400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7f0145040 0x7fe7f006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7e6a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7f006ef10 0x7fe7f0071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44786/0 (socket says 172.21.15.67:44786) 2024-04-17T10:36:59.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7e6a00640 1 -- 172.21.15.67:0/1971815058 learned_addr learned my addr 172.21.15.67:0/1971815058 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:36:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7efe00640 1 --2- 172.21.15.67:0/1971815058 >> v2:172.21.15.86:3300/0 conn(0x7fe7f0071a20 0x7fe7f00cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7ef400640 1 -- 172.21.15.67:0/1971815058 >> v2:172.21.15.86:3300/0 conn(0x7fe7f0071a20 msgr2=0x7fe7f00cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:36:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7ef400640 1 --2- 172.21.15.67:0/1971815058 >> v2:172.21.15.86:3300/0 conn(0x7fe7f0071a20 0x7fe7f00cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7ef400640 1 -- 172.21.15.67:0/1971815058 >> v1:172.21.15.67:6789/0 conn(0x7fe7f006ef10 legacy=0x7fe7f0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:59.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.777+0000 7fe7ef400640 1 -- 172.21.15.67:0/1971815058 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f006d6e0 con 0x7fe7f0145040 2024-04-17T10:36:59.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7ef400640 1 --2- 172.21.15.67:0/1971815058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f006e9d0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00af390 tx=0x7fe7e001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e238a6a661aad82a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:59.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7eea00640 1 -- 172.21.15.67:0/1971815058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00b5070 con 0x7fe7f0145040 2024-04-17T10:36:59.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7eea00640 1 -- 172.21.15.67:0/1971815058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e00a8970 con 0x7fe7f0145040 2024-04-17T10:36:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7eea00640 1 -- 172.21.15.67:0/1971815058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00a8d40 con 0x7fe7f0145040 2024-04-17T10:36:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 -- 172.21.15.67:0/1971815058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 msgr2=0x7fe7f006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 --2- 172.21.15.67:0/1971815058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f006e9d0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fe7f00af390 tx=0x7fe7e001ec70 comp rx=0 tx=0).stop 2024-04-17T10:36:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 -- 172.21.15.67:0/1971815058 shutdown_connections 2024-04-17T10:36:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 --2- 172.21.15.67:0/1971815058 >> v2:172.21.15.86:3300/0 conn(0x7fe7f0071a20 0x7fe7f00cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 --2- 172.21.15.67:0/1971815058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f006e9d0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 -- 172.21.15.67:0/1971815058 >> 172.21.15.67:0/1971815058 conn(0x7fe7f006ac90 msgr2=0x7fe7f006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 -- 172.21.15.67:0/1971815058 shutdown_connections 2024-04-17T10:36:59.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 -- 172.21.15.67:0/1971815058 wait complete. 2024-04-17T10:36:59.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 Processor -- start 2024-04-17T10:36:59.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.781+0000 7fe7f6800640 1 -- start start 2024-04-17T10:36:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7f006ef10 0x7fe7f0139d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7f0071a20 0x7fe7f013cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f0136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f006df80 con 0x7fe7f0145040 2024-04-17T10:36:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f006dce0 con 0x7fe7f006ef10 2024-04-17T10:36:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f006d370 con 0x7fe7f0071a20 2024-04-17T10:36:59.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f0136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f0136000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50664/0 (socket says 172.21.15.67:50664) 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 -- 172.21.15.67:0/4169925340 learned_addr learned my addr 172.21.15.67:0/4169925340 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7e6a00640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7f0071a20 0x7fe7f013cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7ef400640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7f006ef10 0x7fe7f0139d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 -- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7f0071a20 msgr2=0x7fe7f013cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7f0071a20 0x7fe7f013cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 -- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7f006ef10 msgr2=0x7fe7f0139d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7f006ef10 0x7fe7f0139d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 -- 172.21.15.67:0/4169925340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f015b0f0 con 0x7fe7f0145040 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7efe00640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f0136000 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8007e70 tx=0x7fe7e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7e7e00640 1 -- 172.21.15.67:0/4169925340 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8014420 con 0x7fe7f0145040 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7e7e00640 1 -- 172.21.15.67:0/4169925340 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e8019070 con 0x7fe7f0145040 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7e7e00640 1 -- 172.21.15.67:0/4169925340 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8014820 con 0x7fe7f0145040 2024-04-17T10:36:59.777 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f0136540 con 0x7fe7f0145040 2024-04-17T10:36:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f01380f0 con 0x7fe7f0145040 2024-04-17T10:36:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f0072d50 con 0x7fe7f0145040 2024-04-17T10:36:59.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.785+0000 7fe7e7e00640 1 -- 172.21.15.67:0/4169925340 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe7e80149c0 con 0x7fe7f0145040 2024-04-17T10:36:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.789+0000 7fe7e7e00640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7cc066f60 0x7fe7cc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:36:59.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.789+0000 7fe7e7e00640 1 -- 172.21.15.67:0/4169925340 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fe7e8085b00 con 0x7fe7f0145040 2024-04-17T10:36:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.789+0000 7fe7ef400640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7cc066f60 0x7fe7cc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:36:59.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.789+0000 7fe7ef400640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7cc066f60 0x7fe7cc069420 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00b2000 tx=0x7fe7e0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:36:59.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.793+0000 7fe7e7e00640 1 -- 172.21.15.67:0/4169925340 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe7e80568e0 con 0x7fe7f0145040 2024-04-17T10:36:59.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7f00cbdc0 con 0x7fe7f0145040 2024-04-17T10:36:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7e7e00640 1 -- 172.21.15.67:0/4169925340 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fe7e80561e0 con 0x7fe7f0145040 2024-04-17T10:36:59.980 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:36:59.980 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:36:59.980 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7cc066f60 msgr2=0x7fe7cc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7cc066f60 0x7fe7cc069420 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fe7e00b2000 tx=0x7fe7e0008040 comp rx=0 tx=0).stop 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 msgr2=0x7fe7f0136000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f0136000 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8007e70 tx=0x7fe7e800e4b0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 shutdown_connections 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe7cc066f60 0x7fe7cc069420 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f0145040 0x7fe7f0136000 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe7f0071a20 0x7fe7f013cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 --2- 172.21.15.67:0/4169925340 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe7f006ef10 0x7fe7f0139d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:36:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 >> 172.21.15.67:0/4169925340 conn(0x7fe7f006ac90 msgr2=0x7fe7f00cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:36:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 shutdown_connections 2024-04-17T10:36:59.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:36:59.989+0000 7fe7f6800640 1 -- 172.21.15.67:0/4169925340 wait complete. 2024-04-17T10:37:00.404 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:00.404 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:01.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 Processor -- start 2024-04-17T10:37:01.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 -- start start 2024-04-17T10:37:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408158000 0x7f44080adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:01.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f440815e850 0x7f4408071650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4408071b90 0x7f440806e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44080ae1b0 con 0x7f440815e850 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440806d700 con 0x7f4408071b90 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440806d9e0 con 0x7f4408158000 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f440815e850 0x7f4408071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f440815e850 0x7f4408071650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57818/0 (socket says 172.21.15.67:57818) 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 -- 172.21.15.67:0/2851724163 learned_addr learned my addr 172.21.15.67:0/2851724163 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440ce00640 1 --2- 172.21.15.67:0/2851724163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408158000 0x7f44080adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f440d800640 1 --2- 172.21.15.67:0/2851724163 >> v2:172.21.15.86:3300/0 conn(0x7f4408071b90 0x7f440806e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 -- 172.21.15.67:0/2851724163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408158000 msgr2=0x7f44080adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 --2- 172.21.15.67:0/2851724163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408158000 0x7f44080adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 -- 172.21.15.67:0/2851724163 >> v2:172.21.15.86:3300/0 conn(0x7f4408071b90 msgr2=0x7f440806e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 --2- 172.21.15.67:0/2851724163 >> v2:172.21.15.86:3300/0 conn(0x7f4408071b90 0x7f440806e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.565+0000 7f43ffe00640 1 -- 172.21.15.67:0/2851724163 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440806dcc0 con 0x7f440815e850 2024-04-17T10:37:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f43ffe00640 1 --2- 172.21.15.67:0/2851724163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440815e850 0x7f4408071650 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f4408161320 tx=0x7f43f001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11fd246bbd08486 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f4407e00640 1 -- 172.21.15.67:0/2851724163 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00b0070 con 0x7f440815e850 2024-04-17T10:37:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f4407e00640 1 -- 172.21.15.67:0/2851724163 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f00ab600 con 0x7f440815e850 2024-04-17T10:37:01.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f4407e00640 1 -- 172.21.15.67:0/2851724163 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00ab9d0 con 0x7f440815e850 2024-04-17T10:37:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 -- 172.21.15.67:0/2851724163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440815e850 msgr2=0x7f4408071650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 --2- 172.21.15.67:0/2851724163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440815e850 0x7f4408071650 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f4408161320 tx=0x7f43f001c520 comp rx=0 tx=0).stop 2024-04-17T10:37:01.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 -- 172.21.15.67:0/2851724163 shutdown_connections 2024-04-17T10:37:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 --2- 172.21.15.67:0/2851724163 >> v2:172.21.15.86:3300/0 conn(0x7f4408071b90 0x7f440806e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 --2- 172.21.15.67:0/2851724163 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f440815e850 0x7f4408071650 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 --2- 172.21.15.67:0/2851724163 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4408158000 0x7f44080adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 -- 172.21.15.67:0/2851724163 >> 172.21.15.67:0/2851724163 conn(0x7f440806ac90 msgr2=0x7f440806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 -- 172.21.15.67:0/2851724163 shutdown_connections 2024-04-17T10:37:01.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.569+0000 7f440e200640 1 -- 172.21.15.67:0/2851724163 wait complete. 2024-04-17T10:37:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 Processor -- start 2024-04-17T10:37:01.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 -- start start 2024-04-17T10:37:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408071b90 0x7f440815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408158000 0x7f440815fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440815e850 0x7f44081601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44080b46d0 con 0x7f4408158000 2024-04-17T10:37:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44080b4550 con 0x7f4408071b90 2024-04-17T10:37:01.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44081613f0 con 0x7f440815e850 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408158000 0x7f440815fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408158000 0x7f440815fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57830/0 (socket says 172.21.15.67:57830) 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440ce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408071b90 0x7f440815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 -- 172.21.15.67:0/1419264694 learned_addr learned my addr 172.21.15.67:0/1419264694 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440d800640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440815e850 0x7f44081601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 -- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440815e850 msgr2=0x7f44081601a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440815e850 0x7f44081601a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 -- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408071b90 msgr2=0x7f440815c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408071b90 0x7f440815c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 -- 172.21.15.67:0/1419264694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408151750 con 0x7f4408158000 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440ce00640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408071b90 0x7f440815c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f43ffe00640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408158000 0x7f440815fc60 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f43f001abd0 tx=0x7f43f00192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:01.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f4405600640 1 -- 172.21.15.67:0/1419264694 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00b0040 con 0x7f4408158000 2024-04-17T10:37:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f4405600640 1 -- 172.21.15.67:0/1419264694 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f00b4620 con 0x7f4408158000 2024-04-17T10:37:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f4405600640 1 -- 172.21.15.67:0/1419264694 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f00b4a10 con 0x7f4408158000 2024-04-17T10:37:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4408151a30 con 0x7f4408158000 2024-04-17T10:37:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.573+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44080cc480 con 0x7f4408158000 2024-04-17T10:37:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.577+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4408074dc0 con 0x7f4408158000 2024-04-17T10:37:01.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.577+0000 7f4405600640 1 -- 172.21.15.67:0/1419264694 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f43f00ab4c0 con 0x7f4408158000 2024-04-17T10:37:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.577+0000 7f4405600640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f43e4066f90 0x7f43e4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.577+0000 7f4405600640 1 -- 172.21.15.67:0/1419264694 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f43f011ed80 con 0x7f4408158000 2024-04-17T10:37:01.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.577+0000 7f440ce00640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f43e4066f90 0x7f43e4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:01.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.577+0000 7f440ce00640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f43e4066f90 0x7f43e4069450 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009f50 tx=0x7f43f8009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:01.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.581+0000 7f4405600640 1 -- 172.21.15.67:0/1419264694 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f43f0016110 con 0x7f4408158000 2024-04-17T10:37:01.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.773+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44081618d0 con 0x7f4408158000 2024-04-17T10:37:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f4405600640 1 -- 172.21.15.67:0/1419264694 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f43f00ef510 con 0x7f4408158000 2024-04-17T10:37:01.767 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:01.767 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:01.767 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f43e4066f90 msgr2=0x7f43e4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f43e4066f90 0x7f43e4069450 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009f50 tx=0x7f43f8009190 comp rx=0 tx=0).stop 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408158000 msgr2=0x7f440815fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408158000 0x7f440815fc60 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f43f001abd0 tx=0x7f43f00192c0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 shutdown_connections 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f43e4066f90 0x7f43e4069450 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f440815e850 0x7f44081601a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4408158000 0x7f440815fc60 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 --2- 172.21.15.67:0/1419264694 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4408071b90 0x7f440815c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 >> 172.21.15.67:0/1419264694 conn(0x7f440806ac90 msgr2=0x7f44080b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 shutdown_connections 2024-04-17T10:37:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:01.777+0000 7f440e200640 1 -- 172.21.15.67:0/1419264694 wait complete. 2024-04-17T10:37:02.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:02.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:03.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:03.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c78800640 1 Processor -- start 2024-04-17T10:37:03.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c78800640 1 -- start start 2024-04-17T10:37:03.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c78800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 0x7f7c70143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c78800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f7c700cf0c0 0x7f7c70144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c78800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c70144560 con 0x7f7c7006e6e0 2024-04-17T10:37:03.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c78800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7015ae40 con 0x7f7c700cf0c0 2024-04-17T10:37:03.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c78800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70071440 con 0x7f7c7006eac0 2024-04-17T10:37:03.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c75e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7c7006e6e0 0x7f7c700ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40330/0 (socket says 172.21.15.67:40330) 2024-04-17T10:37:03.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c75e00640 1 -- 172.21.15.67:0/1594164862 learned_addr learned my addr 172.21.15.67:0/1594164862 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:03.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c75400640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 0x7f7c70143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c76800640 1 --2- 172.21.15.67:0/1594164862 >> v2:172.21.15.86:3300/0 conn(0x7f7c700cf0c0 0x7f7c70144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756423352 0 0) 0x7f7c70144560 con 0x7f7c7006e6e0 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c58003530 con 0x7f7c7006e6e0 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2712342484 0 0) 0x7f7c58003530 con 0x7f7c7006e6e0 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 msgr2=0x7f7c70143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 0x7f7c70143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 >> v2:172.21.15.86:3300/0 conn(0x7f7c700cf0c0 msgr2=0x7f7c70144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 --2- 172.21.15.67:0/1594164862 >> v2:172.21.15.86:3300/0 conn(0x7f7c700cf0c0 0x7f7c70144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c70071720 con 0x7f7c7006e6e0 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f7c64001680 con 0x7f7c7006e6e0 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c76800640 1 --2- 172.21.15.67:0/1594164862 >> v2:172.21.15.86:3300/0 conn(0x7f7c700cf0c0 0x7f7c70144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:03.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 >> v1:172.21.15.67:6789/0 conn(0x7f7c7006e6e0 legacy=0x7f7c700ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 0x7f7c580039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.353+0000 7f7c74a00640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58003f40 0x7f7c58004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c74a00640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c58004d50 0x7f7c580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7015ae40 con 0x7f7c58004d50 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70071440 con 0x7f7c58003f40 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5800ba20 con 0x7f7c7006eac0 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c75e00640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 0x7f7c580039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c76800640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c58004d50 0x7f7c580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c75400640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58003f40 0x7f7c58004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c76800640 1 -- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 msgr2=0x7f7c580039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c76800640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7006eac0 0x7f7c580039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c76800640 1 -- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58003f40 msgr2=0x7f7c58004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c76800640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58003f40 0x7f7c58004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c76800640 1 -- 172.21.15.67:0/1594164862 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c58003530 con 0x7f7c58004d50 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c76800640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c58004d50 0x7f7c580051d0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00e380 tx=0x7f7c6c01cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a960c3a2fa17b10c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0b0070 con 0x7f7c58004d50 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c0ab600 con 0x7f7c58004d50 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.357+0000 7f7c74a00640 1 -- 172.21.15.67:0/1594164862 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c009de0 con 0x7f7c58004d50 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- 172.21.15.67:0/1594164862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c58004d50 msgr2=0x7f7c580051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 --2- 172.21.15.67:0/1594164862 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c58004d50 0x7f7c580051d0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c00e380 tx=0x7f7c6c01cb10 comp rx=0 tx=0).stop 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c76800640 1 -- 172.21.15.67:0/1594164862 reap_dead start 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- 172.21.15.67:0/1594164862 shutdown_connections 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- 172.21.15.67:0/1594164862 >> 172.21.15.67:0/1594164862 conn(0x7f7c7006ac90 msgr2=0x7f7c7006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- 172.21.15.67:0/1594164862 shutdown_connections 2024-04-17T10:37:03.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- 172.21.15.67:0/1594164862 wait complete. 2024-04-17T10:37:03.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 Processor -- start 2024-04-17T10:37:03.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- start start 2024-04-17T10:37:03.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58004d50 0x7f7c700caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7015b400 0x7f7c700cafd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c700cb510 0x7f7c700cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7006d900 con 0x7f7c700cb510 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7006d780 con 0x7f7c58004d50 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c78800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7006da80 con 0x7f7c7015b400 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c76800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c700cb510 0x7f7c700cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c75e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58004d50 0x7f7c700caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c75400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7015b400 0x7f7c700cafd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c76800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c700cb510 0x7f7c700cb9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57850/0 (socket says 172.21.15.67:57850) 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c75e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58004d50 0x7f7c700caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:38576/0 (socket says 172.21.15.67:38576) 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c76800640 1 -- 172.21.15.67:0/4067921214 learned_addr learned my addr 172.21.15.67:0/4067921214 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:03.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.361+0000 7f7c76800640 1 -- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7015b400 msgr2=0x7f7c700cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c76800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7015b400 0x7f7c700cafd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c76800640 1 -- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58004d50 msgr2=0x7f7c700caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c76800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58004d50 0x7f7c700caa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c76800640 1 -- 172.21.15.67:0/4067921214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700c90c0 con 0x7f7c700cb510 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c76800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c700cb510 0x7f7c700cb9c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0ae040 tx=0x7f7c6c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c6a000640 1 -- 172.21.15.67:0/4067921214 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0b0040 con 0x7f7c700cb510 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c6a000640 1 -- 172.21.15.67:0/4067921214 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c0ab400 con 0x7f7c700cb510 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c6a000640 1 -- 172.21.15.67:0/4067921214 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c009de0 con 0x7f7c700cb510 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c700c9fb0 con 0x7f7c700cb510 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c700ca570 con 0x7f7c700cb510 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70072d50 con 0x7f7c700cb510 2024-04-17T10:37:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c6a000640 1 -- 172.21.15.67:0/4067921214 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7c6c0bf020 con 0x7f7c700cb510 2024-04-17T10:37:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.365+0000 7f7c6a000640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7c54066f90 0x7f7c54069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.369+0000 7f7c6a000640 1 -- 172.21.15.67:0/4067921214 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f7c6c11eca0 con 0x7f7c700cb510 2024-04-17T10:37:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.369+0000 7f7c75e00640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7c54066f90 0x7f7c54069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:03.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.369+0000 7f7c75e00640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7c54066f90 0x7f7c54069450 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7c64001060 tx=0x7f7c64007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:03.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.369+0000 7f7c6a000640 1 -- 172.21.15.67:0/4067921214 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7c6c016bb0 con 0x7f7c700cb510 2024-04-17T10:37:03.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.569+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c70074e30 con 0x7f7c700cb510 2024-04-17T10:37:03.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c6a000640 1 -- 172.21.15.67:0/4067921214 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f7c6c0efab0 con 0x7f7c700cb510 2024-04-17T10:37:03.563 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:03.563 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:03.564 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7c54066f90 msgr2=0x7f7c54069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:03.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7c54066f90 0x7f7c54069450 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f7c64001060 tx=0x7f7c64007900 comp rx=0 tx=0).stop 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c700cb510 msgr2=0x7f7c700cb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c700cb510 0x7f7c700cb9c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0ae040 tx=0x7f7c6c009750 comp rx=0 tx=0).stop 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 shutdown_connections 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7c54066f90 0x7f7c54069450 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c700cb510 0x7f7c700cb9c0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7c7015b400 0x7f7c700cafd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 --2- 172.21.15.67:0/4067921214 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7c58004d50 0x7f7c700caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 >> 172.21.15.67:0/4067921214 conn(0x7f7c7006ac90 msgr2=0x7f7c700af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:03.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 shutdown_connections 2024-04-17T10:37:03.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:03.573+0000 7f7c78800640 1 -- 172.21.15.67:0/4067921214 wait complete. 2024-04-17T10:37:03.987 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:03.988 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:04.989 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:05.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f4105800640 1 Processor -- start 2024-04-17T10:37:05.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f4105800640 1 -- start start 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f4105800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f410006e6e0 0x7f41000ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f4105800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f410006eac0 0x7f41000ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f4105800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f410015ae40 con 0x7f4100143ae0 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f4105800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41000cf430 con 0x7f410006eac0 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f4105800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4100071440 con 0x7f410006e6e0 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f40fea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f410006e6e0 0x7f41000ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f40fea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f410006e6e0 0x7f41000ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:53092/0 (socket says 172.21.15.67:53092) 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f40fea00640 1 -- 172.21.15.67:0/4294434823 learned_addr learned my addr 172.21.15.67:0/4294434823 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.153+0000 7f40fe000640 1 --2- 172.21.15.67:0/4294434823 >> v2:172.21.15.86:3300/0 conn(0x7f410006eac0 0x7f41000ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:05.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fd600640 1 -- 172.21.15.67:0/4294434823 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554541058 0 0) 0x7f410015ae40 con 0x7f4100143ae0 2024-04-17T10:37:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fd600640 1 -- 172.21.15.67:0/4294434823 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e4003530 con 0x7f4100143ae0 2024-04-17T10:37:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fea00640 1 -- 172.21.15.67:0/4294434823 >> v2:172.21.15.86:3300/0 conn(0x7f410006eac0 msgr2=0x7f41000ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fea00640 1 --2- 172.21.15.67:0/4294434823 >> v2:172.21.15.86:3300/0 conn(0x7f410006eac0 0x7f41000ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fea00640 1 -- 172.21.15.67:0/4294434823 >> v1:172.21.15.86:6789/0 conn(0x7f4100143ae0 legacy=0x7f4100143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fea00640 1 -- 172.21.15.67:0/4294434823 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4100071720 con 0x7f410006e6e0 2024-04-17T10:37:05.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fe000640 1 --2- 172.21.15.67:0/4294434823 >> v2:172.21.15.86:3300/0 conn(0x7f410006eac0 0x7f41000ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fea00640 1 --2- 172.21.15.67:0/4294434823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f410006e6e0 0x7f41000ce9b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009870 tx=0x7f40f001c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=351d10a239eab041 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fd600640 1 -- 172.21.15.67:0/4294434823 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00aa600 con 0x7f410006e6e0 2024-04-17T10:37:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fd600640 1 -- 172.21.15.67:0/4294434823 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f00af070 con 0x7f410006e6e0 2024-04-17T10:37:05.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.157+0000 7f40fd600640 1 -- 172.21.15.67:0/4294434823 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00aa9c0 con 0x7f410006e6e0 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- 172.21.15.67:0/4294434823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f410006e6e0 msgr2=0x7f41000ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 --2- 172.21.15.67:0/4294434823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f410006e6e0 0x7f41000ce9b0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009870 tx=0x7f40f001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- 172.21.15.67:0/4294434823 shutdown_connections 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 --2- 172.21.15.67:0/4294434823 >> v2:172.21.15.86:3300/0 conn(0x7f410006eac0 0x7f41000ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 --2- 172.21.15.67:0/4294434823 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f410006e6e0 0x7f41000ce9b0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- 172.21.15.67:0/4294434823 >> 172.21.15.67:0/4294434823 conn(0x7f410006ac90 msgr2=0x7f410006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- 172.21.15.67:0/4294434823 shutdown_connections 2024-04-17T10:37:05.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- 172.21.15.67:0/4294434823 wait complete. 2024-04-17T10:37:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 Processor -- start 2024-04-17T10:37:05.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- start start 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410006e6e0 0x7f4100135b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f410006eac0 0x7f4100142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4100143ae0 0x7f41001562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41000b3a40 con 0x7f410006eac0 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41000b38c0 con 0x7f410006e6e0 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f4105800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41000b3bc0 con 0x7f4100143ae0 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f410006eac0 0x7f4100142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f410006eac0 0x7f4100142390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57858/0 (socket says 172.21.15.67:57858) 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 -- 172.21.15.67:0/2506021603 learned_addr learned my addr 172.21.15.67:0/2506021603 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40ff400640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4100143ae0 0x7f41001562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:05.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fea00640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410006e6e0 0x7f4100135b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 -- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4100143ae0 msgr2=0x7f41001562c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4100143ae0 0x7f41001562c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 -- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410006e6e0 msgr2=0x7f4100135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410006e6e0 0x7f4100135b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 -- 172.21.15.67:0/2506021603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4100167c00 con 0x7f410006eac0 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.161+0000 7f40fe000640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f410006eac0 0x7f4100142390 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00ebe0 tx=0x7f40ec00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f40f6a00640 1 -- 172.21.15.67:0/2506021603 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec017070 con 0x7f410006eac0 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f40f6a00640 1 -- 172.21.15.67:0/2506021603 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ec00cc30 con 0x7f410006eac0 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f40f6a00640 1 -- 172.21.15.67:0/2506021603 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec013470 con 0x7f410006eac0 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4100167ee0 con 0x7f410006eac0 2024-04-17T10:37:05.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4100168780 con 0x7f410006eac0 2024-04-17T10:37:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41001680b0 con 0x7f410006eac0 2024-04-17T10:37:05.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f40f6a00640 1 -- 172.21.15.67:0/2506021603 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f40ec01a070 con 0x7f410006eac0 2024-04-17T10:37:05.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f40f6a00640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e0066f90 0x7f40e0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:05.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f40f6a00640 1 -- 172.21.15.67:0/2506021603 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f40ec0855a0 con 0x7f410006eac0 2024-04-17T10:37:05.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.165+0000 7f40fea00640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e0066f90 0x7f40e0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:05.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.169+0000 7f40fea00640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e0066f90 0x7f40e0069450 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f40f00ae000 tx=0x7f40f00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:05.162 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.169+0000 7f40f6a00640 1 -- 172.21.15.67:0/2506021603 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f40ec056310 con 0x7f410006eac0 2024-04-17T10:37:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4100074e30 con 0x7f410006eac0 2024-04-17T10:37:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f40f6a00640 1 -- 172.21.15.67:0/2506021603 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f40ec055c10 con 0x7f410006eac0 2024-04-17T10:37:05.357 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:05.357 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e0066f90 msgr2=0x7f40e0069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e0066f90 0x7f40e0069450 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f40f00ae000 tx=0x7f40f00a7000 comp rx=0 tx=0).stop 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f410006eac0 msgr2=0x7f4100142390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f410006eac0 0x7f4100142390 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f40ec00ebe0 tx=0x7f40ec00c520 comp rx=0 tx=0).stop 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 shutdown_connections 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f40e0066f90 0x7f40e0069450 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4100143ae0 0x7f41001562c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f410006eac0 0x7f4100142390 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 --2- 172.21.15.67:0/2506021603 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f410006e6e0 0x7f4100135b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.365+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 >> 172.21.15.67:0/2506021603 conn(0x7f410006ac90 msgr2=0x7f4100142d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.369+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 shutdown_connections 2024-04-17T10:37:05.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:05.369+0000 7f4105800640 1 -- 172.21.15.67:0/2506021603 wait complete. 2024-04-17T10:37:05.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:05.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:06.783 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:06.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.941+0000 7f9046e00640 1 Processor -- start 2024-04-17T10:37:06.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046e00640 1 -- start start 2024-04-17T10:37:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90400ce520 0x7f9040168830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904015dd30 con 0x7f904006b3a0 2024-04-17T10:37:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9040161640 con 0x7f904015d950 2024-04-17T10:37:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9040161920 con 0x7f90400ce520 2024-04-17T10:37:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f90400ce520 0x7f9040168830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9045000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f904006b3a0 0x7f9040070280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40338/0 (socket says 172.21.15.67:40338) 2024-04-17T10:37:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9045000640 1 -- 172.21.15.67:0/791582374 learned_addr learned my addr 172.21.15.67:0/791582374 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046400640 1 -- 172.21.15.67:0/791582374 >> v1:172.21.15.86:6789/0 conn(0x7f904015d950 legacy=0x7f904006ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046400640 1 -- 172.21.15.67:0/791582374 >> v1:172.21.15.67:6789/0 conn(0x7f904006b3a0 legacy=0x7f9040070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f903fe00640 1 -- 172.21.15.67:0/791582374 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225622837 0 0) 0x7f904015dd30 con 0x7f904006b3a0 2024-04-17T10:37:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.945+0000 7f9046400640 1 -- 172.21.15.67:0/791582374 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9040161c00 con 0x7f90400ce520 2024-04-17T10:37:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046400640 1 --2- 172.21.15.67:0/791582374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90400ce520 0x7f9040168830 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f9038009d90 tx=0x7f903801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2261e17a06afd26 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f903fe00640 1 -- 172.21.15.67:0/791582374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90380b2070 con 0x7f90400ce520 2024-04-17T10:37:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f903fe00640 1 -- 172.21.15.67:0/791582374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90380a5940 con 0x7f90400ce520 2024-04-17T10:37:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f903fe00640 1 -- 172.21.15.67:0/791582374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90380a5d10 con 0x7f90400ce520 2024-04-17T10:37:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 -- 172.21.15.67:0/791582374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90400ce520 msgr2=0x7f9040168830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 --2- 172.21.15.67:0/791582374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90400ce520 0x7f9040168830 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f9038009d90 tx=0x7f903801c970 comp rx=0 tx=0).stop 2024-04-17T10:37:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 -- 172.21.15.67:0/791582374 shutdown_connections 2024-04-17T10:37:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 --2- 172.21.15.67:0/791582374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f90400ce520 0x7f9040168830 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 -- 172.21.15.67:0/791582374 >> 172.21.15.67:0/791582374 conn(0x7f90400b23f0 msgr2=0x7f90400b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 -- 172.21.15.67:0/791582374 shutdown_connections 2024-04-17T10:37:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 -- 172.21.15.67:0/791582374 wait complete. 2024-04-17T10:37:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 Processor -- start 2024-04-17T10:37:06.942 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.949+0000 7f9046e00640 1 -- start start 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904006b3a0 0x7f904014ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90400ce520 0x7f904014c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f904015d950 0x7f904015c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904016b920 con 0x7f904006b3a0 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904016b7a0 con 0x7f90400ce520 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904016baa0 con 0x7f904015d950 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904006b3a0 0x7f904014ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904006b3a0 0x7f904014ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57872/0 (socket says 172.21.15.67:57872) 2024-04-17T10:37:06.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 -- 172.21.15.67:0/513181267 learned_addr learned my addr 172.21.15.67:0/513181267 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 -- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f904015d950 msgr2=0x7f904015c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f904015d950 0x7f904015c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 -- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90400ce520 msgr2=0x7f904014c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90400ce520 0x7f904014c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 -- 172.21.15.67:0/513181267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90400b02c0 con 0x7f904006b3a0 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9045a00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904006b3a0 0x7f904014ee90 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f903400c3f0 tx=0x7f903400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f903d600640 1 -- 172.21.15.67:0/513181267 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034017070 con 0x7f904006b3a0 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f903d600640 1 -- 172.21.15.67:0/513181267 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90340052d0 con 0x7f904006b3a0 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f903d600640 1 -- 172.21.15.67:0/513181267 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9034003840 con 0x7f904006b3a0 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90400b05a0 con 0x7f904006b3a0 2024-04-17T10:37:06.946 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90400b0b60 con 0x7f904006b3a0 2024-04-17T10:37:06.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9040074dc0 con 0x7f904006b3a0 2024-04-17T10:37:06.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.953+0000 7f903d600640 1 -- 172.21.15.67:0/513181267 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f903401a070 con 0x7f904006b3a0 2024-04-17T10:37:06.947 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.957+0000 7f903d600640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9020066f60 0x7f9020069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.957+0000 7f903d600640 1 -- 172.21.15.67:0/513181267 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f90340845e0 con 0x7f904006b3a0 2024-04-17T10:37:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.957+0000 7f9045000640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9020066f60 0x7f9020069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:06.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.957+0000 7f9045000640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9020066f60 0x7f9020069420 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c10 tx=0x7f902c007910 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:06.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:06.961+0000 7f903d600640 1 -- 172.21.15.67:0/513181267 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9034005480 con 0x7f904006b3a0 2024-04-17T10:37:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.153+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9040073120 con 0x7f904006b3a0 2024-04-17T10:37:07.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.153+0000 7f903d600640 1 -- 172.21.15.67:0/513181267 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f90340553c0 con 0x7f904006b3a0 2024-04-17T10:37:07.147 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:07.147 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:07.147 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9020066f60 msgr2=0x7f9020069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9020066f60 0x7f9020069420 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f902c000c10 tx=0x7f902c007910 comp rx=0 tx=0).stop 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904006b3a0 msgr2=0x7f904014ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904006b3a0 0x7f904014ee90 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f903400c3f0 tx=0x7f903400c8b0 comp rx=0 tx=0).stop 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 shutdown_connections 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9020066f60 0x7f9020069420 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f904015d950 0x7f904015c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:07.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f90400ce520 0x7f904014c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:07.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 --2- 172.21.15.67:0/513181267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f904006b3a0 0x7f904014ee90 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:07.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 >> 172.21.15.67:0/513181267 conn(0x7f90400b23f0 msgr2=0x7f90400b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:07.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 shutdown_connections 2024-04-17T10:37:07.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:07.157+0000 7f9046e00640 1 -- 172.21.15.67:0/513181267 wait complete. 2024-04-17T10:37:07.571 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:07.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:08.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.733+0000 7f9ca0a00640 1 Processor -- start 2024-04-17T10:37:08.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.733+0000 7f9ca0a00640 1 -- start start 2024-04-17T10:37:08.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9ca0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c9c071790 0x7f9c9c15ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9ca0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071b70 0x7f9c9c06e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9ca0a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9c0b3ab0 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9ca0a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9c161700 con 0x7f9c9c071790 2024-04-17T10:37:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9ca0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c1619e0 con 0x7f9c9c071b70 2024-04-17T10:37:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c99e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9c9c071790 0x7f9c9c15ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c9a800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9c9c06ebb0 0x7f9c9c0b4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:40346/0 (socket says 172.21.15.67:40346) 2024-04-17T10:37:08.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c9a800640 1 -- 172.21.15.67:0/1776105892 learned_addr learned my addr 172.21.15.67:0/1776105892 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c99e00640 1 -- 172.21.15.67:0/1776105892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071b70 msgr2=0x7f9c9c06e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c91400640 1 --2- 172.21.15.67:0/1776105892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071b70 0x7f9c9c06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c99e00640 1 --2- 172.21.15.67:0/1776105892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071b70 0x7f9c9c06e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c99e00640 1 -- 172.21.15.67:0/1776105892 >> v1:172.21.15.67:6789/0 conn(0x7f9c9c06ebb0 legacy=0x7f9c9c0b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c99e00640 1 -- 172.21.15.67:0/1776105892 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c161cc0 con 0x7f9c9c071790 2024-04-17T10:37:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c99400640 1 -- 172.21.15.67:0/1776105892 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176288684 0 0) 0x7f9c9c0b3ab0 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.737+0000 7f9c91400640 1 --2- 172.21.15.67:0/1776105892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071b70 0x7f9c9c06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9c99e00640 1 --2- 172.21.15.67:0/1776105892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c071790 0x7f9c9c15ecf0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f9c840097c0 tx=0x7f9c8401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27afe798fc5ce93c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9c99400640 1 -- 172.21.15.67:0/1776105892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c840b1070 con 0x7f9c9c071790 2024-04-17T10:37:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9c99400640 1 -- 172.21.15.67:0/1776105892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c840a8050 con 0x7f9c9c071790 2024-04-17T10:37:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9c99400640 1 -- 172.21.15.67:0/1776105892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c840ac860 con 0x7f9c9c071790 2024-04-17T10:37:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- 172.21.15.67:0/1776105892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c071790 msgr2=0x7f9c9c15ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/1776105892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c071790 0x7f9c9c15ecf0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f9c840097c0 tx=0x7f9c8401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- 172.21.15.67:0/1776105892 shutdown_connections 2024-04-17T10:37:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/1776105892 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071b70 0x7f9c9c06e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/1776105892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c071790 0x7f9c9c15ecf0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- 172.21.15.67:0/1776105892 >> 172.21.15.67:0/1776105892 conn(0x7f9c9c0cfa30 msgr2=0x7f9c9c0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:08.732 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- 172.21.15.67:0/1776105892 shutdown_connections 2024-04-17T10:37:08.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- 172.21.15.67:0/1776105892 wait complete. 2024-04-17T10:37:08.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 Processor -- start 2024-04-17T10:37:08.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- start start 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c06ebb0 0x7f9c9c164210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071790 0x7f9c9c166a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c9c071b70 0x7f9c9c151640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c0720d0 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c071f50 con 0x7f9c9c071b70 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.741+0000 7f9ca0a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c16aa00 con 0x7f9c9c071790 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c06ebb0 0x7f9c9c164210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c06ebb0 0x7f9c9c164210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57880/0 (socket says 172.21.15.67:57880) 2024-04-17T10:37:08.735 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 -- 172.21.15.67:0/2718598867 learned_addr learned my addr 172.21.15.67:0/2718598867 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c91400640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071790 0x7f9c9c166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c9a800640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c9c071b70 0x7f9c9c151640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 -- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071790 msgr2=0x7f9c9c166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071790 0x7f9c9c166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 -- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c9c071b70 msgr2=0x7f9c9c151640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c9c071b70 0x7f9c9c151640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 -- 172.21.15.67:0/2718598867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c155d60 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c91400640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071790 0x7f9c9c166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c9a800640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c9c071b70 0x7f9c9c151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c99e00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c06ebb0 0x7f9c9c164210 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f9c840af040 tx=0x7f9c8401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:08.737 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c92a00640 1 -- 172.21.15.67:0/2718598867 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c840b1050 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c92a00640 1 -- 172.21.15.67:0/2718598867 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c840bd070 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c165c90 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c92a00640 1 -- 172.21.15.67:0/2718598867 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c840b88b0 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c166530 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c074e30 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c92a00640 1 -- 172.21.15.67:0/2718598867 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9c840b8a50 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.745+0000 7f9c92a00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9c78066f60 0x7f9c78069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:08.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.749+0000 7f9c92a00640 1 -- 172.21.15.67:0/2718598867 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f9c8411fb00 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.749+0000 7f9c91400640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9c78066f60 0x7f9c78069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:08.739 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.749+0000 7f9c91400640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9c78066f60 0x7f9c78069420 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0d0360 tx=0x7f9c8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:08.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.749+0000 7f9c92a00640 1 -- 172.21.15.67:0/2718598867 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9c9c155d60 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.945+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c9c06e450 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.945+0000 7f9c92a00640 1 -- 172.21.15.67:0/2718598867 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f9c8411fe20 con 0x7f9c9c06ebb0 2024-04-17T10:37:08.939 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:08.939 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:08.939 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9c78066f60 msgr2=0x7f9c78069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9c78066f60 0x7f9c78069420 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0d0360 tx=0x7f9c8c008040 comp rx=0 tx=0).stop 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c06ebb0 msgr2=0x7f9c9c164210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c06ebb0 0x7f9c9c164210 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f9c840af040 tx=0x7f9c8401ace0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 shutdown_connections 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9c78066f60 0x7f9c78069420 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9c9c071b70 0x7f9c9c151640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9c9c071790 0x7f9c9c166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 --2- 172.21.15.67:0/2718598867 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9c9c06ebb0 0x7f9c9c164210 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 >> 172.21.15.67:0/2718598867 conn(0x7f9c9c0cfa30 msgr2=0x7f9c9c0ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 shutdown_connections 2024-04-17T10:37:08.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:08.949+0000 7f9ca0a00640 1 -- 172.21.15.67:0/2718598867 wait complete. 2024-04-17T10:37:09.363 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:09.364 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:10.364 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.525+0000 7f2ce4e00640 1 Processor -- start 2024-04-17T10:37:10.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.525+0000 7f2ce4e00640 1 -- start start 2024-04-17T10:37:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2ce4e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ce015dbc0 0x7f2ce006f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2ce4e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2ce4e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce00ad8f0 con 0x7f2ce00721a0 2024-04-17T10:37:10.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2ce4e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce015dfa0 con 0x7f2ce015dbc0 2024-04-17T10:37:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2ce4e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce006faa0 con 0x7f2ce006db90 2024-04-17T10:37:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cde800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2ce00721a0 0x7f2ce006b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43608/0 (socket says 172.21.15.67:43608) 2024-04-17T10:37:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cdde00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2ce015dbc0 0x7f2ce006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cde800640 1 -- 172.21.15.67:0/1242462514 learned_addr learned my addr 172.21.15.67:0/1242462514 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:10.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cd7e00640 1 --2- 172.21.15.67:0/1242462514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cdde00640 1 -- 172.21.15.67:0/1242462514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 msgr2=0x7f2ce0071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cdde00640 1 --2- 172.21.15.67:0/1242462514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cdde00640 1 -- 172.21.15.67:0/1242462514 >> v1:172.21.15.67:6789/0 conn(0x7f2ce00721a0 legacy=0x7f2ce006b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cdde00640 1 -- 172.21.15.67:0/1242462514 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce00adbd0 con 0x7f2ce015dbc0 2024-04-17T10:37:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cdd400640 1 -- 172.21.15.67:0/1242462514 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018902276 0 0) 0x7f2ce00ad8f0 con 0x7f2ce00721a0 2024-04-17T10:37:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.529+0000 7f2cd7e00640 1 --2- 172.21.15.67:0/1242462514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:10.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2cdde00640 1 --2- 172.21.15.67:0/1242462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce015dbc0 0x7f2ce006f560 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00097c0 tx=0x7f2cd001c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfa369d4047d762 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:10.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2cdd400640 1 -- 172.21.15.67:0/1242462514 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00b0070 con 0x7f2ce015dbc0 2024-04-17T10:37:10.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2cdd400640 1 -- 172.21.15.67:0/1242462514 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd00ab600 con 0x7f2ce015dbc0 2024-04-17T10:37:10.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2cdd400640 1 -- 172.21.15.67:0/1242462514 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00ab9d0 con 0x7f2ce015dbc0 2024-04-17T10:37:10.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- 172.21.15.67:0/1242462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce015dbc0 msgr2=0x7f2ce006f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:10.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/1242462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce015dbc0 0x7f2ce006f560 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00097c0 tx=0x7f2cd001c410 comp rx=0 tx=0).stop 2024-04-17T10:37:10.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- 172.21.15.67:0/1242462514 shutdown_connections 2024-04-17T10:37:10.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/1242462514 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/1242462514 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce015dbc0 0x7f2ce006f560 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- 172.21.15.67:0/1242462514 >> 172.21.15.67:0/1242462514 conn(0x7f2ce00af670 msgr2=0x7f2ce00b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:10.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- 172.21.15.67:0/1242462514 shutdown_connections 2024-04-17T10:37:10.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- 172.21.15.67:0/1242462514 wait complete. 2024-04-17T10:37:10.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 Processor -- start 2024-04-17T10:37:10.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- start start 2024-04-17T10:37:10.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0160b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce00721a0 0x7f2ce015e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ce015dbc0 0x7f2ce01f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce006c6f0 con 0x7f2ce00721a0 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce006c570 con 0x7f2ce015dbc0 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.533+0000 7f2ce4e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce006af70 con 0x7f2ce006db90 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce00721a0 0x7f2ce015e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce00721a0 0x7f2ce015e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53414/0 (socket says 172.21.15.67:53414) 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 -- 172.21.15.67:0/3543708291 learned_addr learned my addr 172.21.15.67:0/3543708291 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cdde00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0160b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cde800640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ce015dbc0 0x7f2ce01f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 -- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 msgr2=0x7f2ce0160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0160b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 -- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ce015dbc0 msgr2=0x7f2ce01f6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ce015dbc0 0x7f2ce01f6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 -- 172.21.15.67:0/3543708291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce015e660 con 0x7f2ce00721a0 2024-04-17T10:37:10.529 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd7e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce00721a0 0x7f2ce015e120 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c7f0 tx=0x7f2cc800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd6000640 1 -- 172.21.15.67:0/3543708291 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8015070 con 0x7f2ce00721a0 2024-04-17T10:37:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd6000640 1 -- 172.21.15.67:0/3543708291 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc80077c0 con 0x7f2ce00721a0 2024-04-17T10:37:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd6000640 1 -- 172.21.15.67:0/3543708291 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8007bb0 con 0x7f2ce00721a0 2024-04-17T10:37:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce00cef30 con 0x7f2ce00721a0 2024-04-17T10:37:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce015ebc0 con 0x7f2ce00721a0 2024-04-17T10:37:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce009e260 con 0x7f2ce00721a0 2024-04-17T10:37:10.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.537+0000 7f2cd6000640 1 -- 172.21.15.67:0/3543708291 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2cc8018070 con 0x7f2ce00721a0 2024-04-17T10:37:10.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.541+0000 7f2cd6000640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2cbc066f60 0x7f2cbc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:10.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.541+0000 7f2cd6000640 1 -- 172.21.15.67:0/3543708291 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f2cc8085340 con 0x7f2ce00721a0 2024-04-17T10:37:10.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.541+0000 7f2cdde00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2cbc066f60 0x7f2cbc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:10.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.541+0000 7f2cdde00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2cbc066f60 0x7f2cbc069420 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00ad000 tx=0x7f2cd00a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:10.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.545+0000 7f2cd6000640 1 -- 172.21.15.67:0/3543708291 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2ce015e660 con 0x7f2ce00721a0 2024-04-17T10:37:10.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.737+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ce015d3f0 con 0x7f2ce00721a0 2024-04-17T10:37:10.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2cd6000640 1 -- 172.21.15.67:0/3543708291 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f2cc8055a40 con 0x7f2ce00721a0 2024-04-17T10:37:10.732 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:10.732 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:10.732 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2cbc066f60 msgr2=0x7f2cbc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:10.733 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2cbc066f60 0x7f2cbc069420 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00ad000 tx=0x7f2cd00a6000 comp rx=0 tx=0).stop 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce00721a0 msgr2=0x7f2ce015e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce00721a0 0x7f2ce015e120 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c7f0 tx=0x7f2cc800ccb0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 shutdown_connections 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2cbc066f60 0x7f2cbc069420 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ce015dbc0 0x7f2ce01f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ce00721a0 0x7f2ce015e120 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 --2- 172.21.15.67:0/3543708291 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2ce006db90 0x7f2ce0160b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 >> 172.21.15.67:0/3543708291 conn(0x7f2ce00af670 msgr2=0x7f2ce00acee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 shutdown_connections 2024-04-17T10:37:10.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:10.741+0000 7f2ce4e00640 1 -- 172.21.15.67:0/3543708291 wait complete. 2024-04-17T10:37:11.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:11.155 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:12.156 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c44400640 1 Processor -- start 2024-04-17T10:37:12.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c44400640 1 -- start start 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c44400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06e6e0 0x7f8c3c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c44400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c3c06eac0 0x7f8c3c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c44400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c3c15ae40 con 0x7f8c3c143ae0 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c44400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c3c0cf430 con 0x7f8c3c06eac0 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c44400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c071440 con 0x7f8c3c06e6e0 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c41000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c3c06eac0 0x7f8c3c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c41000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8c3c06eac0 0x7f8c3c0ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53424/0 (socket says 172.21.15.67:53424) 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c41000640 1 -- 172.21.15.67:0/2719026619 learned_addr learned my addr 172.21.15.67:0/2719026619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.317+0000 7f8c41a00640 1 --2- 172.21.15.67:0/2719026619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06e6e0 0x7f8c3c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c41000640 1 -- 172.21.15.67:0/2719026619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06e6e0 msgr2=0x7f8c3c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c41000640 1 --2- 172.21.15.67:0/2719026619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06e6e0 0x7f8c3c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c41000640 1 -- 172.21.15.67:0/2719026619 >> v1:172.21.15.86:6789/0 conn(0x7f8c3c143ae0 legacy=0x7f8c3c143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:12.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c41000640 1 -- 172.21.15.67:0/2719026619 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c071720 con 0x7f8c3c06eac0 2024-04-17T10:37:12.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c41000640 1 --2- 172.21.15.67:0/2719026619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c06eac0 0x7f8c3c0ceef0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f8c28009870 tx=0x7f8c2801e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf2c110acfaa1f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:12.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c3be00640 1 -- 172.21.15.67:0/2719026619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c280ab680 con 0x7f8c3c06eac0 2024-04-17T10:37:12.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c3be00640 1 -- 172.21.15.67:0/2719026619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c280ab820 con 0x7f8c3c06eac0 2024-04-17T10:37:12.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c3be00640 1 -- 172.21.15.67:0/2719026619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c280abbf0 con 0x7f8c3c06eac0 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 -- 172.21.15.67:0/2719026619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c06eac0 msgr2=0x7f8c3c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 --2- 172.21.15.67:0/2719026619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c06eac0 0x7f8c3c0ceef0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f8c28009870 tx=0x7f8c2801e6b0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 -- 172.21.15.67:0/2719026619 shutdown_connections 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 --2- 172.21.15.67:0/2719026619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c06eac0 0x7f8c3c0ceef0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 --2- 172.21.15.67:0/2719026619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06e6e0 0x7f8c3c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 -- 172.21.15.67:0/2719026619 >> 172.21.15.67:0/2719026619 conn(0x7f8c3c06ac90 msgr2=0x7f8c3c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 -- 172.21.15.67:0/2719026619 shutdown_connections 2024-04-17T10:37:12.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.321+0000 7f8c44400640 1 -- 172.21.15.67:0/2719026619 wait complete. 2024-04-17T10:37:12.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 Processor -- start 2024-04-17T10:37:12.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 -- start start 2024-04-17T10:37:12.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c3c06e6e0 0x7f8c3c13ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:12.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06eac0 0x7f8c3c0cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:12.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c143ae0 0x7f8c3c0c5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:12.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c06dd00 con 0x7f8c3c143ae0 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c06db80 con 0x7f8c3c06e6e0 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3c06de80 con 0x7f8c3c06eac0 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c143ae0 0x7f8c3c0c5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c143ae0 0x7f8c3c0c5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53434/0 (socket says 172.21.15.67:53434) 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 -- 172.21.15.67:0/514431615 learned_addr learned my addr 172.21.15.67:0/514431615 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c41a00640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c3c06e6e0 0x7f8c3c13ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c41000640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06eac0 0x7f8c3c0cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 -- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06eac0 msgr2=0x7f8c3c0cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06eac0 0x7f8c3c0cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 -- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c3c06e6e0 msgr2=0x7f8c3c13ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c3c06e6e0 0x7f8c3c13ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 -- 172.21.15.67:0/514431615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c3c0cb6a0 con 0x7f8c3c143ae0 2024-04-17T10:37:12.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c41a00640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c3c06e6e0 0x7f8c3c13ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:37:12.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c42400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c143ae0 0x7f8c3c0c5730 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400c760 tx=0x7f8c3400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:12.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c39600640 1 -- 172.21.15.67:0/514431615 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c34003b80 con 0x7f8c3c143ae0 2024-04-17T10:37:12.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c39600640 1 -- 172.21.15.67:0/514431615 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c34003d20 con 0x7f8c3c143ae0 2024-04-17T10:37:12.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c39600640 1 -- 172.21.15.67:0/514431615 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3400aa30 con 0x7f8c3c143ae0 2024-04-17T10:37:12.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.325+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c3c0cb980 con 0x7f8c3c143ae0 2024-04-17T10:37:12.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.329+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3c0c7230 con 0x7f8c3c143ae0 2024-04-17T10:37:12.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.329+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c3c072d50 con 0x7f8c3c143ae0 2024-04-17T10:37:12.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.329+0000 7f8c39600640 1 -- 172.21.15.67:0/514431615 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8c3400abd0 con 0x7f8c3c143ae0 2024-04-17T10:37:12.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.329+0000 7f8c39600640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c1c066f60 0x7f8c1c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:12.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.329+0000 7f8c39600640 1 -- 172.21.15.67:0/514431615 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8c34085070 con 0x7f8c3c143ae0 2024-04-17T10:37:12.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.329+0000 7f8c41a00640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c1c066f60 0x7f8c1c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:12.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.329+0000 7f8c41a00640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c1c066f60 0x7f8c1c069420 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0ae150 tx=0x7f8c30008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:12.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.333+0000 7f8c39600640 1 -- 172.21.15.67:0/514431615 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8c34055e50 con 0x7f8c3c143ae0 2024-04-17T10:37:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c3c0c7400 con 0x7f8c3c143ae0 2024-04-17T10:37:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c39600640 1 -- 172.21.15.67:0/514431615 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f8c34055750 con 0x7f8c3c143ae0 2024-04-17T10:37:12.525 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:12.525 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:12.525 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c1c066f60 msgr2=0x7f8c1c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c1c066f60 0x7f8c1c069420 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c0ae150 tx=0x7f8c30008040 comp rx=0 tx=0).stop 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c143ae0 msgr2=0x7f8c3c0c5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c143ae0 0x7f8c3c0c5730 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f8c3400c760 tx=0x7f8c3400cc20 comp rx=0 tx=0).stop 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 shutdown_connections 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8c1c066f60 0x7f8c1c069420 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8c3c143ae0 0x7f8c3c0c5730 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8c3c06eac0 0x7f8c3c0cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 --2- 172.21.15.67:0/514431615 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8c3c06e6e0 0x7f8c3c13ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 >> 172.21.15.67:0/514431615 conn(0x7f8c3c06ac90 msgr2=0x7f8c3c0ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 shutdown_connections 2024-04-17T10:37:12.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:12.533+0000 7f8c44400640 1 -- 172.21.15.67:0/514431615 wait complete. 2024-04-17T10:37:12.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:12.949 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:13.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:14.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdabce00640 1 Processor -- start 2024-04-17T10:37:14.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdabce00640 1 -- start start 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdabce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdab80cc250 0x7fdab80cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdabce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab80cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdabce00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab806b100 con 0x7fdab80cb260 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdabce00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab80cd150 con 0x7fdab80cc250 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdabce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab806b3e0 con 0x7fdab80cdcf0 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdab80cc250 0x7fdab80cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdab80cc250 0x7fdab80cad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53442/0 (socket says 172.21.15.67:53442) 2024-04-17T10:37:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab5e00640 1 -- 172.21.15.67:0/1032542209 learned_addr learned my addr 172.21.15.67:0/1032542209 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab6800640 1 --2- 172.21.15.67:0/1032542209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab80cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab5e00640 1 -- 172.21.15.67:0/1032542209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 msgr2=0x7fdab80cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab5e00640 1 --2- 172.21.15.67:0/1032542209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab80cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab5e00640 1 -- 172.21.15.67:0/1032542209 >> v1:172.21.15.86:6789/0 conn(0x7fdab80cb260 legacy=0x7fdab80cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab5e00640 1 -- 172.21.15.67:0/1032542209 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab806b6c0 con 0x7fdab80cc250 2024-04-17T10:37:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.113+0000 7fdab6800640 1 --2- 172.21.15.67:0/1032542209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab80cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:14.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdab5e00640 1 --2- 172.21.15.67:0/1032542209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cc250 0x7fdab80cad20 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009870 tx=0x7fdaa801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a2408be7f1c881 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:14.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdab5400640 1 -- 172.21.15.67:0/1032542209 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80b0070 con 0x7fdab80cc250 2024-04-17T10:37:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdab5400640 1 -- 172.21.15.67:0/1032542209 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa80ab600 con 0x7fdab80cc250 2024-04-17T10:37:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdab5400640 1 -- 172.21.15.67:0/1032542209 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80ab9d0 con 0x7fdab80cc250 2024-04-17T10:37:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 -- 172.21.15.67:0/1032542209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cc250 msgr2=0x7fdab80cad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 --2- 172.21.15.67:0/1032542209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cc250 0x7fdab80cad20 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009870 tx=0x7fdaa801c520 comp rx=0 tx=0).stop 2024-04-17T10:37:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 -- 172.21.15.67:0/1032542209 shutdown_connections 2024-04-17T10:37:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 --2- 172.21.15.67:0/1032542209 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab80cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 --2- 172.21.15.67:0/1032542209 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cc250 0x7fdab80cad20 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 -- 172.21.15.67:0/1032542209 >> 172.21.15.67:0/1032542209 conn(0x7fdab80b22a0 msgr2=0x7fdab80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 -- 172.21.15.67:0/1032542209 shutdown_connections 2024-04-17T10:37:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.117+0000 7fdabce00640 1 -- 172.21.15.67:0/1032542209 wait complete. 2024-04-17T10:37:14.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 Processor -- start 2024-04-17T10:37:14.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 -- start start 2024-04-17T10:37:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cb260 0x7fdab8159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab80cc250 0x7fdab80c5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab814a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:14.113 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80d0320 con 0x7fdab80cb260 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab80d01a0 con 0x7fdab80cc250 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab816b100 con 0x7fdab80cdcf0 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cb260 0x7fdab8159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cb260 0x7fdab8159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53456/0 (socket says 172.21.15.67:53456) 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 -- 172.21.15.67:0/589300794 learned_addr learned my addr 172.21.15.67:0/589300794 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab6800640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab814a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdaafe00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab80cc250 0x7fdab80c5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 -- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 msgr2=0x7fdab814a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab814a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 -- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab80cc250 msgr2=0x7fdab80c5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab80cc250 0x7fdab80c5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 -- 172.21.15.67:0/589300794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab80c5f60 con 0x7fdab80cb260 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdab5e00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cb260 0x7fdab8159b20 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80ae040 tx=0x7fdaa80192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:14.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdaae000640 1 -- 172.21.15.67:0/589300794 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80b0040 con 0x7fdab80cb260 2024-04-17T10:37:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdaae000640 1 -- 172.21.15.67:0/589300794 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa80b9070 con 0x7fdab80cb260 2024-04-17T10:37:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab80c6240 con 0x7fdab80cb260 2024-04-17T10:37:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdaae000640 1 -- 172.21.15.67:0/589300794 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa80b4880 con 0x7fdab80cb260 2024-04-17T10:37:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab80c6800 con 0x7fdab80cb260 2024-04-17T10:37:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.121+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab806cd30 con 0x7fdab80cb260 2024-04-17T10:37:14.115 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.125+0000 7fdaae000640 1 -- 172.21.15.67:0/589300794 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fdaa80b4a20 con 0x7fdab80cb260 2024-04-17T10:37:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.125+0000 7fdaae000640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda94066f60 0x7fda94069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.125+0000 7fdaae000640 1 -- 172.21.15.67:0/589300794 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fdaa811f260 con 0x7fdab80cb260 2024-04-17T10:37:14.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.125+0000 7fdaafe00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda94066f60 0x7fda94069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:14.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.125+0000 7fdaafe00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda94066f60 0x7fda94069420 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdab815d8e0 tx=0x7fdaa0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:14.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.129+0000 7fdaae000640 1 -- 172.21.15.67:0/589300794 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fdaa8016110 con 0x7fdab80cb260 2024-04-17T10:37:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdab809e900 con 0x7fdab80cb260 2024-04-17T10:37:14.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdaae000640 1 -- 172.21.15.67:0/589300794 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fdaa80effb0 con 0x7fdab80cb260 2024-04-17T10:37:14.316 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:14.316 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:14.316 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:14.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda94066f60 msgr2=0x7fda94069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda94066f60 0x7fda94069420 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fdab815d8e0 tx=0x7fdaa0009040 comp rx=0 tx=0).stop 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cb260 msgr2=0x7fdab8159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cb260 0x7fdab8159b20 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80ae040 tx=0x7fdaa80192c0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 shutdown_connections 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fda94066f60 0x7fda94069420 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fdab80cdcf0 0x7fdab814a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fdab80cc250 0x7fdab80c5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 --2- 172.21.15.67:0/589300794 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdab80cb260 0x7fdab8159b20 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 >> 172.21.15.67:0/589300794 conn(0x7fdab80b22a0 msgr2=0x7fdab80b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 shutdown_connections 2024-04-17T10:37:14.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:14.325+0000 7fdabce00640 1 -- 172.21.15.67:0/589300794 wait complete. 2024-04-17T10:37:14.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:14.740 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:15.741 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.901+0000 7f605ca00640 1 Processor -- start 2024-04-17T10:37:15.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.901+0000 7f605ca00640 1 -- start start 2024-04-17T10:37:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.901+0000 7f605ca00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f605815d920 0x7f605809a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.901+0000 7f605ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60580cc110 0x7f6058161500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.901+0000 7f605ca00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580cc8a0 con 0x7f6058161a40 2024-04-17T10:37:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.901+0000 7f605ca00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6058161eb0 con 0x7f605815d920 2024-04-17T10:37:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.901+0000 7f605ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605809ab30 con 0x7f60580cc110 2024-04-17T10:37:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f605815d920 0x7f605809a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f604de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60580cc110 0x7f6058161500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f605815d920 0x7f605809a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:57912/0 (socket says 172.21.15.67:57912) 2024-04-17T10:37:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 -- 172.21.15.67:0/3057037919 learned_addr learned my addr 172.21.15.67:0/3057037919 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:15.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 -- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60580cc110 msgr2=0x7f6058161500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 --2- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60580cc110 0x7f6058161500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 -- 172.21.15.67:0/3057037919 >> v1:172.21.15.86:6789/0 conn(0x7f6058161a40 legacy=0x7f6058070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 -- 172.21.15.67:0/3057037919 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580ccb80 con 0x7f605815d920 2024-04-17T10:37:15.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f604de00640 1 --2- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60580cc110 0x7f6058161500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:15.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6056800640 1 --2- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f605815d920 0x7f605809a5f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6048009e50 tx=0x7f604801ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=875444863d6eb717 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6055e00640 1 -- 172.21.15.67:0/3057037919 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480a57f0 con 0x7f605815d920 2024-04-17T10:37:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6055e00640 1 -- 172.21.15.67:0/3057037919 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60480a5990 con 0x7f605815d920 2024-04-17T10:37:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.905+0000 7f6055e00640 1 -- 172.21.15.67:0/3057037919 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480a5d60 con 0x7f605815d920 2024-04-17T10:37:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f605815d920 msgr2=0x7f605809a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 --2- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f605815d920 0x7f605809a5f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6048009e50 tx=0x7f604801ca30 comp rx=0 tx=0).stop 2024-04-17T10:37:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- 172.21.15.67:0/3057037919 shutdown_connections 2024-04-17T10:37:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 --2- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f60580cc110 0x7f6058161500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 --2- 172.21.15.67:0/3057037919 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f605815d920 0x7f605809a5f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- 172.21.15.67:0/3057037919 >> 172.21.15.67:0/3057037919 conn(0x7f60580d1600 msgr2=0x7f60580d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- 172.21.15.67:0/3057037919 shutdown_connections 2024-04-17T10:37:15.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- 172.21.15.67:0/3057037919 wait complete. 2024-04-17T10:37:15.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 Processor -- start 2024-04-17T10:37:15.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- start start 2024-04-17T10:37:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580cc110 0x7f6058153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605815d920 0x7f605806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:15.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6058161a40 0x7f605806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6058071260 con 0x7f60580cc110 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605815ed10 con 0x7f6058161a40 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f605ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60580cc4f0 con 0x7f605815d920 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f6056800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580cc110 0x7f6058153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f6056800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580cc110 0x7f6058153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53472/0 (socket says 172.21.15.67:53472) 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f6056800640 1 -- 172.21.15.67:0/3610274410 learned_addr learned my addr 172.21.15.67:0/3610274410 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f604de00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605815d920 0x7f605806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.909+0000 7f6057200640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6058161a40 0x7f605806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f6056800640 1 -- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605815d920 msgr2=0x7f605806d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f6056800640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605815d920 0x7f605806d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f6056800640 1 -- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6058161a40 msgr2=0x7f605806db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f6056800640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6058161a40 0x7f605806db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f6056800640 1 -- 172.21.15.67:0/3610274410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580af930 con 0x7f60580cc110 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f6057200640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6058161a40 0x7f605806db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f6056800640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580cc110 0x7f6058153f90 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f604801a660 tx=0x7f60480197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:15.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f604f400640 1 -- 172.21.15.67:0/3610274410 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480b8470 con 0x7f60580cc110 2024-04-17T10:37:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f604f400640 1 -- 172.21.15.67:0/3610274410 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60480b8610 con 0x7f60580cc110 2024-04-17T10:37:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f604f400640 1 -- 172.21.15.67:0/3610274410 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60480b8a00 con 0x7f60580cc110 2024-04-17T10:37:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60580afc10 con 0x7f60580cc110 2024-04-17T10:37:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60580b01d0 con 0x7f60580cc110 2024-04-17T10:37:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6058074dc0 con 0x7f60580cc110 2024-04-17T10:37:15.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f604f400640 1 -- 172.21.15.67:0/3610274410 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f60480c0020 con 0x7f60580cc110 2024-04-17T10:37:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f604f400640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6034066f60 0x7f6034069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f604f400640 1 -- 172.21.15.67:0/3610274410 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f604811fa50 con 0x7f60580cc110 2024-04-17T10:37:15.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.913+0000 7f604de00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6034066f60 0x7f6034069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:15.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.917+0000 7f604de00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6034066f60 0x7f6034069420 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6040009580 tx=0x7f6040009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:15.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:15.917+0000 7f604f400640 1 -- 172.21.15.67:0/3610274410 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f60480c7020 con 0x7f60580cc110 2024-04-17T10:37:16.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60580d2510 con 0x7f60580cc110 2024-04-17T10:37:16.104 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f604f400640 1 -- 172.21.15.67:0/3610274410 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6048016620 con 0x7f60580cc110 2024-04-17T10:37:16.104 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:16.104 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:16.105 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:16.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6034066f60 msgr2=0x7f6034069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:16.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6034066f60 0x7f6034069420 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f6040009580 tx=0x7f6040009190 comp rx=0 tx=0).stop 2024-04-17T10:37:16.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580cc110 msgr2=0x7f6058153f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:16.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580cc110 0x7f6058153f90 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f604801a660 tx=0x7f60480197d0 comp rx=0 tx=0).stop 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 shutdown_connections 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6034066f60 0x7f6034069420 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6058161a40 0x7f605806db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f605815d920 0x7f605806d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 --2- 172.21.15.67:0/3610274410 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60580cc110 0x7f6058153f90 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 >> 172.21.15.67:0/3610274410 conn(0x7f60580d1600 msgr2=0x7f60580b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 shutdown_connections 2024-04-17T10:37:16.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:16.113+0000 7f605ca00640 1 -- 172.21.15.67:0/3610274410 wait complete. 2024-04-17T10:37:16.527 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:16.527 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:17.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:17.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.689+0000 7f3aeea00640 1 Processor -- start 2024-04-17T10:37:17.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.689+0000 7f3aeea00640 1 -- start start 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aeea00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ae8099a50 0x7f3ae815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aeea00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3ae80d03a0 0x7f3ae8099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aeea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae809a740 0x7f3ae815f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aeea00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae80cde20 con 0x7f3ae8099a50 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aeea00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae809a370 con 0x7f3ae80d03a0 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aeea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80cf300 con 0x7f3ae809a740 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ae8099a50 0x7f3ae815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ae8099a50 0x7f3ae815c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53488/0 (socket says 172.21.15.67:53488) 2024-04-17T10:37:17.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 -- 172.21.15.67:0/1488893262 learned_addr learned my addr 172.21.15.67:0/1488893262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 -- 172.21.15.67:0/1488893262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae809a740 msgr2=0x7f3ae815f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aecc00640 1 --2- 172.21.15.67:0/1488893262 >> v2:172.21.15.86:3300/0 conn(0x7f3ae80d03a0 0x7f3ae8099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aee000640 1 --2- 172.21.15.67:0/1488893262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae809a740 0x7f3ae815f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 --2- 172.21.15.67:0/1488893262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae809a740 0x7f3ae815f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 -- 172.21.15.67:0/1488893262 >> v2:172.21.15.86:3300/0 conn(0x7f3ae80d03a0 msgr2=0x7f3ae8099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 --2- 172.21.15.67:0/1488893262 >> v2:172.21.15.86:3300/0 conn(0x7f3ae80d03a0 0x7f3ae8099e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aed600640 1 -- 172.21.15.67:0/1488893262 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae80cf5e0 con 0x7f3ae8099a50 2024-04-17T10:37:17.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aecc00640 1 --2- 172.21.15.67:0/1488893262 >> v2:172.21.15.86:3300/0 conn(0x7f3ae80d03a0 0x7f3ae8099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:17.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.693+0000 7f3aee000640 1 --2- 172.21.15.67:0/1488893262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae809a740 0x7f3ae815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:17.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aed600640 1 --2- 172.21.15.67:0/1488893262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae8099a50 0x7f3ae815c7a0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3ae809acd0 tx=0x7f3adc01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def6c449b9ca1096 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:17.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3ae7e00640 1 -- 172.21.15.67:0/1488893262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0b0070 con 0x7f3ae8099a50 2024-04-17T10:37:17.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3ae7e00640 1 -- 172.21.15.67:0/1488893262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc0ab600 con 0x7f3ae8099a50 2024-04-17T10:37:17.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3ae7e00640 1 -- 172.21.15.67:0/1488893262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc0ab9d0 con 0x7f3ae8099a50 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 -- 172.21.15.67:0/1488893262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae8099a50 msgr2=0x7f3ae815c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 --2- 172.21.15.67:0/1488893262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae8099a50 0x7f3ae815c7a0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3ae809acd0 tx=0x7f3adc01c260 comp rx=0 tx=0).stop 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 -- 172.21.15.67:0/1488893262 shutdown_connections 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 --2- 172.21.15.67:0/1488893262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae809a740 0x7f3ae815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 --2- 172.21.15.67:0/1488893262 >> v2:172.21.15.86:3300/0 conn(0x7f3ae80d03a0 0x7f3ae8099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 --2- 172.21.15.67:0/1488893262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae8099a50 0x7f3ae815c7a0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 -- 172.21.15.67:0/1488893262 >> 172.21.15.67:0/1488893262 conn(0x7f3ae80d1690 msgr2=0x7f3ae80d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 -- 172.21.15.67:0/1488893262 shutdown_connections 2024-04-17T10:37:17.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 -- 172.21.15.67:0/1488893262 wait complete. 2024-04-17T10:37:17.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 Processor -- start 2024-04-17T10:37:17.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.697+0000 7f3aeea00640 1 -- start start 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae8099a50 0x7f3ae80b5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae809a740 0x7f3ae8157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae80d03a0 0x7f3ae806bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80cc320 con 0x7f3ae80d03a0 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80cc1a0 con 0x7f3ae809a740 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80cc4a0 con 0x7f3ae8099a50 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae80d03a0 0x7f3ae806bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:17.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae80d03a0 0x7f3ae806bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53496/0 (socket says 172.21.15.67:53496) 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 -- 172.21.15.67:0/2463327216 learned_addr learned my addr 172.21.15.67:0/2463327216 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aed600640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae8099a50 0x7f3ae80b5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aecc00640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae809a740 0x7f3ae8157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 -- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae8099a50 msgr2=0x7f3ae80b5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae8099a50 0x7f3ae80b5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 -- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae809a740 msgr2=0x7f3ae8157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae809a740 0x7f3ae8157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 -- 172.21.15.67:0/2463327216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae806c780 con 0x7f3ae80d03a0 2024-04-17T10:37:17.693 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aee000640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae80d03a0 0x7f3ae806bda0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000cf20 tx=0x7f3ae000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:17.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3ae5600640 1 -- 172.21.15.67:0/2463327216 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae001a070 con 0x7f3ae80d03a0 2024-04-17T10:37:17.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3ae5600640 1 -- 172.21.15.67:0/2463327216 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae000d840 con 0x7f3ae80d03a0 2024-04-17T10:37:17.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3ae5600640 1 -- 172.21.15.67:0/2463327216 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae000dc30 con 0x7f3ae80d03a0 2024-04-17T10:37:17.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae806ca60 con 0x7f3ae80d03a0 2024-04-17T10:37:17.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae815a640 con 0x7f3ae80d03a0 2024-04-17T10:37:17.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae815a840 con 0x7f3ae80d03a0 2024-04-17T10:37:17.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.701+0000 7f3ae5600640 1 -- 172.21.15.67:0/2463327216 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3ae001d070 con 0x7f3ae80d03a0 2024-04-17T10:37:17.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.705+0000 7f3ae5600640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ac8066f90 0x7f3ac8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:17.695 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.705+0000 7f3ae5600640 1 -- 172.21.15.67:0/2463327216 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f3ae0085950 con 0x7f3ae80d03a0 2024-04-17T10:37:17.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.705+0000 7f3aed600640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ac8066f90 0x7f3ac8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:17.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.705+0000 7f3aed600640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ac8066f90 0x7f3ac8069450 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3adc0ad000 tx=0x7f3adc0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:17.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.709+0000 7f3ae5600640 1 -- 172.21.15.67:0/2463327216 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3ae815a840 con 0x7f3ae80d03a0 2024-04-17T10:37:17.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ae8074dc0 con 0x7f3ae80d03a0 2024-04-17T10:37:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3ae5600640 1 -- 172.21.15.67:0/2463327216 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3ae00560e0 con 0x7f3ae80d03a0 2024-04-17T10:37:17.897 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:17.897 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:17.897 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ac8066f90 msgr2=0x7f3ac8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ac8066f90 0x7f3ac8069450 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3adc0ad000 tx=0x7f3adc0a6000 comp rx=0 tx=0).stop 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae80d03a0 msgr2=0x7f3ae806bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae80d03a0 0x7f3ae806bda0 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3ae000cf20 tx=0x7f3ae000a710 comp rx=0 tx=0).stop 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 shutdown_connections 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3ac8066f90 0x7f3ac8069450 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ae80d03a0 0x7f3ae806bda0 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3ae809a740 0x7f3ae8157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 --2- 172.21.15.67:0/2463327216 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3ae8099a50 0x7f3ae80b5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.905+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 >> 172.21.15.67:0/2463327216 conn(0x7f3ae80d1690 msgr2=0x7f3ae80b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.909+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 shutdown_connections 2024-04-17T10:37:17.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:17.909+0000 7f3aeea00640 1 -- 172.21.15.67:0/2463327216 wait complete. 2024-04-17T10:37:18.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:18.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:19.321 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb0ca00640 1 Processor -- start 2024-04-17T10:37:19.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb0ca00640 1 -- start start 2024-04-17T10:37:19.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb0ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806e6e0 0x7fbb080ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:19.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb0ca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb0815ae40 con 0x7fbb0806eac0 2024-04-17T10:37:19.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb0ca00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb08071440 con 0x7fbb081441f0 2024-04-17T10:37:19.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb0ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb08071720 con 0x7fbb0806e6e0 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbafd400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbb0806eac0 0x7fbb08143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:43616/0 (socket says 172.21.15.67:43616) 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbafd400640 1 -- 172.21.15.67:0/1908233690 learned_addr learned my addr 172.21.15.67:0/1908233690 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb05e00640 1 --2- 172.21.15.67:0/1908233690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806e6e0 0x7fbb080ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb05400640 1 -- 172.21.15.67:0/1908233690 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360184628 0 0) 0x7fbb0815ae40 con 0x7fbb0806eac0 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb05400640 1 -- 172.21.15.67:0/1908233690 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaec003530 con 0x7fbb0806eac0 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb05e00640 1 -- 172.21.15.67:0/1908233690 >> v1:172.21.15.86:6789/0 conn(0x7fbb081441f0 legacy=0x7fbb080b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb05e00640 1 -- 172.21.15.67:0/1908233690 >> v1:172.21.15.67:6789/0 conn(0x7fbb0806eac0 legacy=0x7fbb08143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb05e00640 1 -- 172.21.15.67:0/1908233690 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08071a00 con 0x7fbb0806e6e0 2024-04-17T10:37:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.485+0000 7fbb05400640 1 -- 172.21.15.67:0/1908233690 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530200004 0 0) 0x7fbb08071440 con 0x7fbb081441f0 2024-04-17T10:37:19.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb05e00640 1 --2- 172.21.15.67:0/1908233690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806e6e0 0x7fbb080ce9b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80097f0 tx=0x7fbaf801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a082217cfcc31551 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:19.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb05400640 1 -- 172.21.15.67:0/1908233690 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80b0070 con 0x7fbb0806e6e0 2024-04-17T10:37:19.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb05400640 1 -- 172.21.15.67:0/1908233690 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaf80ab600 con 0x7fbb0806e6e0 2024-04-17T10:37:19.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb05400640 1 -- 172.21.15.67:0/1908233690 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80ab9d0 con 0x7fbb0806e6e0 2024-04-17T10:37:19.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1908233690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806e6e0 msgr2=0x7fbb080ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:19.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1908233690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806e6e0 0x7fbb080ce9b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80097f0 tx=0x7fbaf801c420 comp rx=0 tx=0).stop 2024-04-17T10:37:19.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1908233690 shutdown_connections 2024-04-17T10:37:19.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1908233690 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806e6e0 0x7fbb080ce9b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:19.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1908233690 >> 172.21.15.67:0/1908233690 conn(0x7fbb0806ac90 msgr2=0x7fbb0806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:19.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1908233690 shutdown_connections 2024-04-17T10:37:19.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.489+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1908233690 wait complete. 2024-04-17T10:37:19.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 Processor -- start 2024-04-17T10:37:19.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 -- start start 2024-04-17T10:37:19.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0806e6e0 0x7fbb081f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:19.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806eac0 0x7fbb08141db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb081441f0 0x7fbb080c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb081688b0 con 0x7fbb081441f0 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb08168730 con 0x7fbb0806e6e0 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb08168a30 con 0x7fbb0806eac0 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb081441f0 0x7fbb080c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb081441f0 0x7fbb080c5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53504/0 (socket says 172.21.15.67:53504) 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 -- 172.21.15.67:0/1909222895 learned_addr learned my addr 172.21.15.67:0/1909222895 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbafd400640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806eac0 0x7fbb08141db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb05e00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0806e6e0 0x7fbb081f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 -- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806eac0 msgr2=0x7fbb08141db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806eac0 0x7fbb08141db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 -- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0806e6e0 msgr2=0x7fbb081f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0806e6e0 0x7fbb081f6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:19.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 -- 172.21.15.67:0/1909222895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08142520 con 0x7fbb081441f0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb06800640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb081441f0 0x7fbb080c5db0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000c780 tx=0x7fbb0000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbafea00640 1 -- 172.21.15.67:0/1909222895 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00003db0 con 0x7fbb081441f0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbafea00640 1 -- 172.21.15.67:0/1909222895 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0000ce30 con 0x7fbb081441f0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0813eac0 con 0x7fbb081441f0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbafea00640 1 -- 172.21.15.67:0/1909222895 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00005290 con 0x7fbb081441f0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.493+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0813f080 con 0x7fbb081441f0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.497+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb08072d50 con 0x7fbb081441f0 2024-04-17T10:37:19.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.497+0000 7fbafea00640 1 -- 172.21.15.67:0/1909222895 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbb00005430 con 0x7fbb081441f0 2024-04-17T10:37:19.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.497+0000 7fbafea00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae4066f60 0x7fbae4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:19.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.497+0000 7fbafea00640 1 -- 172.21.15.67:0/1909222895 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbb00084fe0 con 0x7fbb081441f0 2024-04-17T10:37:19.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.497+0000 7fbb05e00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae4066f60 0x7fbae4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:19.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.497+0000 7fbb05e00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae4066f60 0x7fbae4069420 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80ad000 tx=0x7fbaf80a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:19.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.501+0000 7fbafea00640 1 -- 172.21.15.67:0/1909222895 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbb00055770 con 0x7fbb081441f0 2024-04-17T10:37:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.697+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb08074e30 con 0x7fbb081441f0 2024-04-17T10:37:19.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.697+0000 7fbafea00640 1 -- 172.21.15.67:0/1909222895 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fbb00059810 con 0x7fbb081441f0 2024-04-17T10:37:19.689 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:19.689 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:19.690 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae4066f60 msgr2=0x7fbae4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae4066f60 0x7fbae4069420 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80ad000 tx=0x7fbaf80a6000 comp rx=0 tx=0).stop 2024-04-17T10:37:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb081441f0 msgr2=0x7fbb080c5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb081441f0 0x7fbb080c5db0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fbb0000c780 tx=0x7fbb0000cc40 comp rx=0 tx=0).stop 2024-04-17T10:37:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 shutdown_connections 2024-04-17T10:37:19.691 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbae4066f60 0x7fbae4069420 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb081441f0 0x7fbb080c5db0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb0806eac0 0x7fbb08141db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 --2- 172.21.15.67:0/1909222895 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb0806e6e0 0x7fbb081f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 >> 172.21.15.67:0/1909222895 conn(0x7fbb0806ac90 msgr2=0x7fbb080cc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 shutdown_connections 2024-04-17T10:37:19.692 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:19.701+0000 7fbb0ca00640 1 -- 172.21.15.67:0/1909222895 wait complete. 2024-04-17T10:37:20.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:20.114 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:21.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8713200640 1 Processor -- start 2024-04-17T10:37:21.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8713200640 1 -- start start 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8713200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8714158000 0x7f87140adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8713200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8714071d60 0x7f871406e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8713200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871406d700 con 0x7f871415e850 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8713200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87140ae1b0 con 0x7f8714158000 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8713200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871406d9e0 con 0x7f8714071d60 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f870be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f871415e850 0x7f8714071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37630/0 (socket says 172.21.15.67:37630) 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8711e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8714158000 0x7f87140adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:21.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f870be00640 1 -- 172.21.15.67:0/3681890545 learned_addr learned my addr 172.21.15.67:0/3681890545 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8711e00640 1 -- 172.21.15.67:0/3681890545 >> v2:172.21.15.86:3300/0 conn(0x7f8714071d60 msgr2=0x7f871406e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8712800640 1 --2- 172.21.15.67:0/3681890545 >> v2:172.21.15.86:3300/0 conn(0x7f8714071d60 0x7f871406e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8711e00640 1 --2- 172.21.15.67:0/3681890545 >> v2:172.21.15.86:3300/0 conn(0x7f8714071d60 0x7f871406e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8711e00640 1 -- 172.21.15.67:0/3681890545 >> v1:172.21.15.67:6789/0 conn(0x7f871415e850 legacy=0x7f8714071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8711e00640 1 -- 172.21.15.67:0/3681890545 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f871406dcc0 con 0x7f8714158000 2024-04-17T10:37:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8711400640 1 -- 172.21.15.67:0/3681890545 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103734091 0 0) 0x7f871406d700 con 0x7f871415e850 2024-04-17T10:37:21.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.273+0000 7f8712800640 1 --2- 172.21.15.67:0/3681890545 >> v2:172.21.15.86:3300/0 conn(0x7f8714071d60 0x7f871406e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8711e00640 1 --2- 172.21.15.67:0/3681890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f87140adc70 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f8714067d50 tx=0x7f870401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4426e20b18ea44af server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8711400640 1 -- 172.21.15.67:0/3681890545 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87040b1070 con 0x7f8714158000 2024-04-17T10:37:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8711400640 1 -- 172.21.15.67:0/3681890545 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87040ac640 con 0x7f8714158000 2024-04-17T10:37:21.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8711400640 1 -- 172.21.15.67:0/3681890545 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87040aca10 con 0x7f8714158000 2024-04-17T10:37:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 -- 172.21.15.67:0/3681890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 msgr2=0x7f87140adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 --2- 172.21.15.67:0/3681890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f87140adc70 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f8714067d50 tx=0x7f870401c480 comp rx=0 tx=0).stop 2024-04-17T10:37:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 -- 172.21.15.67:0/3681890545 shutdown_connections 2024-04-17T10:37:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 --2- 172.21.15.67:0/3681890545 >> v2:172.21.15.86:3300/0 conn(0x7f8714071d60 0x7f871406e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 --2- 172.21.15.67:0/3681890545 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f87140adc70 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 -- 172.21.15.67:0/3681890545 >> 172.21.15.67:0/3681890545 conn(0x7f871406ac90 msgr2=0x7f871406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 -- 172.21.15.67:0/3681890545 shutdown_connections 2024-04-17T10:37:21.271 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.277+0000 7f8713200640 1 -- 172.21.15.67:0/3681890545 wait complete. 2024-04-17T10:37:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 Processor -- start 2024-04-17T10:37:21.272 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 -- start start 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714071d60 0x7f8714154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f871415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871415e850 0x7f8714155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87140b46d0 con 0x7f8714158000 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87140b4550 con 0x7f8714071d60 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f871416a560 con 0x7f871415e850 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f871415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f871415acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53454/0 (socket says 172.21.15.67:53454) 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 -- 172.21.15.67:0/2737516645 learned_addr learned my addr 172.21.15.67:0/2737516645 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:21.274 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 -- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871415e850 msgr2=0x7f8714155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871415e850 0x7f8714155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 -- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714071d60 msgr2=0x7f8714154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714071d60 0x7f8714154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 -- 172.21.15.67:0/2737516645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87141589e0 con 0x7f8714158000 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870be00640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f871415acf0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00c520 tx=0x7f86fc00c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870a000640 1 -- 172.21.15.67:0/2737516645 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc018070 con 0x7f8714158000 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870a000640 1 -- 172.21.15.67:0/2737516645 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc0074e0 con 0x7f8714158000 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f870a000640 1 -- 172.21.15.67:0/2737516645 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc0049f0 con 0x7f8714158000 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87140cca10 con 0x7f8714158000 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.281+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87140ccfd0 con 0x7f8714158000 2024-04-17T10:37:21.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.285+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8714074dc0 con 0x7f8714158000 2024-04-17T10:37:21.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.285+0000 7f870a000640 1 -- 172.21.15.67:0/2737516645 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f86fc004b90 con 0x7f8714158000 2024-04-17T10:37:21.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.285+0000 7f870a000640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86f0066f90 0x7f86f0069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.285+0000 7f870a000640 1 -- 172.21.15.67:0/2737516645 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f86fc0846b0 con 0x7f8714158000 2024-04-17T10:37:21.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.285+0000 7f8711e00640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86f0066f90 0x7f86f0069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.285+0000 7f8711e00640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86f0066f90 0x7f86f0069450 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f87040ae000 tx=0x7f8704008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:21.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.289+0000 7f870a000640 1 -- 172.21.15.67:0/2737516645 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f86fc055400 con 0x7f8714158000 2024-04-17T10:37:21.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.485+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87140cd1a0 con 0x7f8714158000 2024-04-17T10:37:21.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.485+0000 7f870a000640 1 -- 172.21.15.67:0/2737516645 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f86fc054d00 con 0x7f8714158000 2024-04-17T10:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:21.478 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:21.478 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86f0066f90 msgr2=0x7f86f0069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:21.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86f0066f90 0x7f86f0069450 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f87040ae000 tx=0x7f8704008040 comp rx=0 tx=0).stop 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 msgr2=0x7f871415acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f871415acf0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f86fc00c520 tx=0x7f86fc00c9e0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 shutdown_connections 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f86f0066f90 0x7f86f0069450 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f871415e850 0x7f8714155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8714158000 0x7f871415acf0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 --2- 172.21.15.67:0/2737516645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8714071d60 0x7f8714154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 >> 172.21.15.67:0/2737516645 conn(0x7f871406ac90 msgr2=0x7f871415e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 shutdown_connections 2024-04-17T10:37:21.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:21.489+0000 7f8713200640 1 -- 172.21.15.67:0/2737516645 wait complete. 2024-04-17T10:37:21.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:21.902 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:22.903 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:23.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.061+0000 7fe530800640 1 Processor -- start 2024-04-17T10:37:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.061+0000 7fe530800640 1 -- start start 2024-04-17T10:37:23.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe530800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528145040 0x7fe52806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:23.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe530800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe528071a20 0x7fe5280cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:23.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe530800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe52806e2f0 con 0x7fe52806ef10 2024-04-17T10:37:23.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe530800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5280a86c0 con 0x7fe528071a20 2024-04-17T10:37:23.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe530800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52815bf10 con 0x7fe528145040 2024-04-17T10:37:23.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52e800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe528071a20 0x7fe5280cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:23.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528145040 0x7fe52806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:23.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52e800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe528071a20 0x7fe5280cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:40200/0 (socket says 172.21.15.67:40200) 2024-04-17T10:37:23.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52e800640 1 -- 172.21.15.67:0/791561432 learned_addr learned my addr 172.21.15.67:0/791561432 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:23.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52d400640 1 --1- 172.21.15.67:0/791561432 >> v1:172.21.15.86:6789/0 conn(0x7fe52806ef10 0x7fe528071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:52022/0 (socket says 172.21.15.67:52022) 2024-04-17T10:37:23.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52de00640 1 -- 172.21.15.67:0/791561432 >> v2:172.21.15.86:3300/0 conn(0x7fe528071a20 msgr2=0x7fe5280cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:23.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52de00640 1 --2- 172.21.15.67:0/791561432 >> v2:172.21.15.86:3300/0 conn(0x7fe528071a20 0x7fe5280cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52de00640 1 -- 172.21.15.67:0/791561432 >> v1:172.21.15.86:6789/0 conn(0x7fe52806ef10 legacy=0x7fe528071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:23.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52de00640 1 -- 172.21.15.67:0/791561432 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe52806d6e0 con 0x7fe528145040 2024-04-17T10:37:23.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.065+0000 7fe52e800640 1 --2- 172.21.15.67:0/791561432 >> v2:172.21.15.86:3300/0 conn(0x7fe528071a20 0x7fe5280cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:23.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe52de00640 1 --2- 172.21.15.67:0/791561432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528145040 0x7fe52806e9d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0097c0 tx=0x7fe51c01ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deaa26326ebe56fc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:23.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe52ca00640 1 -- 172.21.15.67:0/791561432 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0b5070 con 0x7fe528145040 2024-04-17T10:37:23.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe52ca00640 1 -- 172.21.15.67:0/791561432 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51c0a8990 con 0x7fe528145040 2024-04-17T10:37:23.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe52ca00640 1 -- 172.21.15.67:0/791561432 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51c0a8d60 con 0x7fe528145040 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 -- 172.21.15.67:0/791561432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528145040 msgr2=0x7fe52806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 --2- 172.21.15.67:0/791561432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528145040 0x7fe52806e9d0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0097c0 tx=0x7fe51c01ec90 comp rx=0 tx=0).stop 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 -- 172.21.15.67:0/791561432 shutdown_connections 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 --2- 172.21.15.67:0/791561432 >> v2:172.21.15.86:3300/0 conn(0x7fe528071a20 0x7fe5280cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 --2- 172.21.15.67:0/791561432 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528145040 0x7fe52806e9d0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 -- 172.21.15.67:0/791561432 >> 172.21.15.67:0/791561432 conn(0x7fe52806ac90 msgr2=0x7fe52806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 -- 172.21.15.67:0/791561432 shutdown_connections 2024-04-17T10:37:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 -- 172.21.15.67:0/791561432 wait complete. 2024-04-17T10:37:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 Processor -- start 2024-04-17T10:37:23.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.069+0000 7fe530800640 1 -- start start 2024-04-17T10:37:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52806ef10 0x7fe5281f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528071a20 0x7fe528143f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528145040 0x7fe52815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52806df80 con 0x7fe528145040 2024-04-17T10:37:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe52806dca0 con 0x7fe52806ef10 2024-04-17T10:37:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe52806d370 con 0x7fe528071a20 2024-04-17T10:37:23.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528145040 0x7fe52815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528145040 0x7fe52815b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53470/0 (socket says 172.21.15.67:53470) 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 -- 172.21.15.67:0/612134576 learned_addr learned my addr 172.21.15.67:0/612134576 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52de00640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52806ef10 0x7fe5281f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52d400640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528071a20 0x7fe528143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 -- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528071a20 msgr2=0x7fe528143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528071a20 0x7fe528143f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 -- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52806ef10 msgr2=0x7fe5281f7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52806ef10 0x7fe5281f7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 -- 172.21.15.67:0/612134576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe528144470 con 0x7fe528145040 2024-04-17T10:37:23.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52de00640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52806ef10 0x7fe5281f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe52e800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528145040 0x7fe52815b0c0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fe52400ef90 tx=0x7fe52400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe522000640 1 -- 172.21.15.67:0/612134576 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52400cbe0 con 0x7fe528145040 2024-04-17T10:37:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe522000640 1 -- 172.21.15.67:0/612134576 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52400cd80 con 0x7fe528145040 2024-04-17T10:37:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe522000640 1 -- 172.21.15.67:0/612134576 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524003be0 con 0x7fe528145040 2024-04-17T10:37:23.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5281f7b40 con 0x7fe528145040 2024-04-17T10:37:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5281f8100 con 0x7fe528145040 2024-04-17T10:37:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.073+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe528072d50 con 0x7fe528145040 2024-04-17T10:37:23.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.077+0000 7fe522000640 1 -- 172.21.15.67:0/612134576 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe52401b030 con 0x7fe528145040 2024-04-17T10:37:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.077+0000 7fe522000640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe50c066f60 0x7fe50c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.077+0000 7fe522000640 1 -- 172.21.15.67:0/612134576 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fe524085480 con 0x7fe528145040 2024-04-17T10:37:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.077+0000 7fe52de00640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe50c066f60 0x7fe50c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.077+0000 7fe52de00640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe50c066f60 0x7fe50c069420 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0b2000 tx=0x7fe51c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:23.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.081+0000 7fe522000640 1 -- 172.21.15.67:0/612134576 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe524022020 con 0x7fe528145040 2024-04-17T10:37:23.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.273+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe528145500 con 0x7fe528145040 2024-04-17T10:37:23.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.273+0000 7fe522000640 1 -- 172.21.15.67:0/612134576 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fe524010040 con 0x7fe528145040 2024-04-17T10:37:23.266 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:23.266 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:23.266 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:23.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe50c066f60 msgr2=0x7fe50c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:23.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe50c066f60 0x7fe50c069420 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe51c0b2000 tx=0x7fe51c008040 comp rx=0 tx=0).stop 2024-04-17T10:37:23.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528145040 msgr2=0x7fe52815b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:23.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528145040 0x7fe52815b0c0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fe52400ef90 tx=0x7fe52400c490 comp rx=0 tx=0).stop 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 shutdown_connections 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe50c066f60 0x7fe50c069420 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe528145040 0x7fe52815b0c0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe528071a20 0x7fe528143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 --2- 172.21.15.67:0/612134576 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe52806ef10 0x7fe5281f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 >> 172.21.15.67:0/612134576 conn(0x7fe52806ac90 msgr2=0x7fe5280cc300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 shutdown_connections 2024-04-17T10:37:23.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:23.277+0000 7fe530800640 1 -- 172.21.15.67:0/612134576 wait complete. 2024-04-17T10:37:23.690 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:23.691 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:24.691 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f170ee00640 1 Processor -- start 2024-04-17T10:37:24.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f170ee00640 1 -- start start 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f170ee00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17080b5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f170ee00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17080b57b0 0x7f1708070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f170ee00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17080b0cc0 con 0x7f17080b07b0 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f170ee00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17080b1280 con 0x7f17080b57b0 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f170ee00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17080b0fa0 con 0x7f170815dd10 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f16ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17080b57b0 0x7f1708070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f16ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f17080b57b0 0x7f1708070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53474/0 (socket says 172.21.15.67:53474) 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f16ff400640 1 -- 172.21.15.67:0/3420788096 learned_addr learned my addr 172.21.15.67:0/3420788096 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f1707e00640 1 --2- 172.21.15.67:0/3420788096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17080b5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f16ff400640 1 -- 172.21.15.67:0/3420788096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 msgr2=0x7f17080b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:24.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f16ff400640 1 --2- 172.21.15.67:0/3420788096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17080b5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:24.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f16ff400640 1 -- 172.21.15.67:0/3420788096 >> v1:172.21.15.86:6789/0 conn(0x7f17080b07b0 legacy=0x7f17081618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:24.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.853+0000 7f16ff400640 1 -- 172.21.15.67:0/3420788096 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170806bed0 con 0x7f17080b57b0 2024-04-17T10:37:24.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f16ff400640 1 --2- 172.21.15.67:0/3420788096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17080b57b0 0x7f1708070dc0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009870 tx=0x7f16f801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c4ef8e69837e92 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f1707400640 1 -- 172.21.15.67:0/3420788096 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f80ab600 con 0x7f17080b57b0 2024-04-17T10:37:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f1707400640 1 -- 172.21.15.67:0/3420788096 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f80b0070 con 0x7f17080b57b0 2024-04-17T10:37:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f1707400640 1 -- 172.21.15.67:0/3420788096 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f80ab9c0 con 0x7f17080b57b0 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 -- 172.21.15.67:0/3420788096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17080b57b0 msgr2=0x7f1708070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 --2- 172.21.15.67:0/3420788096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17080b57b0 0x7f1708070dc0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f16f8009870 tx=0x7f16f801c460 comp rx=0 tx=0).stop 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 -- 172.21.15.67:0/3420788096 shutdown_connections 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 --2- 172.21.15.67:0/3420788096 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f17080b57b0 0x7f1708070dc0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 --2- 172.21.15.67:0/3420788096 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17080b5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 -- 172.21.15.67:0/3420788096 >> 172.21.15.67:0/3420788096 conn(0x7f17080b34d0 msgr2=0x7f17080b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 -- 172.21.15.67:0/3420788096 shutdown_connections 2024-04-17T10:37:24.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.857+0000 7f170ee00640 1 -- 172.21.15.67:0/3420788096 wait complete. 2024-04-17T10:37:24.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 Processor -- start 2024-04-17T10:37:24.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 -- start start 2024-04-17T10:37:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17080b07b0 0x7f17081587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17081576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170814eb00 0x7f17080cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f170816ae70 con 0x7f170814eb00 2024-04-17T10:37:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708071710 con 0x7f17080b07b0 2024-04-17T10:37:24.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170816aff0 con 0x7f170815dd10 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170814eb00 0x7f17080cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170814eb00 0x7f17080cf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53486/0 (socket says 172.21.15.67:53486) 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 -- 172.21.15.67:0/321643665 learned_addr learned my addr 172.21.15.67:0/321643665 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f16ff400640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17081576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f1707e00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17080b07b0 0x7f17081587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 -- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 msgr2=0x7f17081576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17081576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 -- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17080b07b0 msgr2=0x7f17081587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17080b07b0 0x7f17081587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 -- 172.21.15.67:0/321643665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1708158e10 con 0x7f170814eb00 2024-04-17T10:37:24.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ce00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170814eb00 0x7f17080cf2c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f170000c780 tx=0x7f170000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f1704c00640 1 -- 172.21.15.67:0/321643665 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700016020 con 0x7f170814eb00 2024-04-17T10:37:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f1704c00640 1 -- 172.21.15.67:0/321643665 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170000a9f0 con 0x7f170814eb00 2024-04-17T10:37:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17081590f0 con 0x7f170814eb00 2024-04-17T10:37:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f1704c00640 1 -- 172.21.15.67:0/321643665 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170000ae20 con 0x7f170814eb00 2024-04-17T10:37:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.861+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17080ceda0 con 0x7f170814eb00 2024-04-17T10:37:24.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.865+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1708070bb0 con 0x7f170814eb00 2024-04-17T10:37:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.865+0000 7f1704c00640 1 -- 172.21.15.67:0/321643665 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1700019020 con 0x7f170814eb00 2024-04-17T10:37:24.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.865+0000 7f1704c00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e4066f60 0x7f16e4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:24.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.865+0000 7f1704c00640 1 -- 172.21.15.67:0/321643665 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1700085160 con 0x7f170814eb00 2024-04-17T10:37:24.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.865+0000 7f1707e00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e4066f60 0x7f16e4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:24.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.865+0000 7f1707e00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e4066f60 0x7f16e4069420 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f16f00079c0 tx=0x7f16f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:24.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:24.869+0000 7f1704c00640 1 -- 172.21.15.67:0/321643665 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1700020030 con 0x7f170814eb00 2024-04-17T10:37:25.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.065+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f170814e8c0 con 0x7f170814eb00 2024-04-17T10:37:25.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.065+0000 7f1704c00640 1 -- 172.21.15.67:0/321643665 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1700055870 con 0x7f170814eb00 2024-04-17T10:37:25.058 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:25.058 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:25.059 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e4066f60 msgr2=0x7f16e4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e4066f60 0x7f16e4069420 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f16f00079c0 tx=0x7f16f0008040 comp rx=0 tx=0).stop 2024-04-17T10:37:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170814eb00 msgr2=0x7f17080cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170814eb00 0x7f17080cf2c0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f170000c780 tx=0x7f170000cc40 comp rx=0 tx=0).stop 2024-04-17T10:37:25.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 shutdown_connections 2024-04-17T10:37:25.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f16e4066f60 0x7f16e4069420 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:25.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f170814eb00 0x7f17080cf2c0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:25.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f170815dd10 0x7f17081576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:25.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 --2- 172.21.15.67:0/321643665 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f17080b07b0 0x7f17081587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:25.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 >> 172.21.15.67:0/321643665 conn(0x7f17080b34d0 msgr2=0x7f170815b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:25.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 shutdown_connections 2024-04-17T10:37:25.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:25.069+0000 7f170ee00640 1 -- 172.21.15.67:0/321643665 wait complete. 2024-04-17T10:37:25.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:25.482 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:26.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:26.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff527200640 1 Processor -- start 2024-04-17T10:37:26.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff527200640 1 -- start start 2024-04-17T10:37:26.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff527200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5200d03a0 0x7ff52015c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:26.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff527200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff52009a740 0x7ff5200cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:26.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff527200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5200cde20 con 0x7ff520099a50 2024-04-17T10:37:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff527200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5200cf300 con 0x7ff5200d03a0 2024-04-17T10:37:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff527200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200cf5e0 con 0x7ff52009a740 2024-04-17T10:37:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff525400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5200d03a0 0x7ff52015c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff525400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5200d03a0 0x7ff52015c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53500/0 (socket says 172.21.15.67:53500) 2024-04-17T10:37:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff525400640 1 -- 172.21.15.67:0/4209975399 learned_addr learned my addr 172.21.15.67:0/4209975399 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff526800640 1 --2- 172.21.15.67:0/4209975399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff52009a740 0x7ff5200cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff525400640 1 -- 172.21.15.67:0/4209975399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff52009a740 msgr2=0x7ff5200cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:26.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff525400640 1 --2- 172.21.15.67:0/4209975399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff52009a740 0x7ff5200cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff525400640 1 -- 172.21.15.67:0/4209975399 >> v1:172.21.15.86:6789/0 conn(0x7ff520099a50 legacy=0x7ff520099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:26.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.645+0000 7ff525400640 1 -- 172.21.15.67:0/4209975399 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5200cf8c0 con 0x7ff5200d03a0 2024-04-17T10:37:26.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff525400640 1 --2- 172.21.15.67:0/4209975399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200d03a0 0x7ff52015c7a0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7ff510009870 tx=0x7ff51001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f2de55d3fbb815 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:26.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff524a00640 1 -- 172.21.15.67:0/4209975399 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5100b2070 con 0x7ff5200d03a0 2024-04-17T10:37:26.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff524a00640 1 -- 172.21.15.67:0/4209975399 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5100a9070 con 0x7ff5200d03a0 2024-04-17T10:37:26.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff524a00640 1 -- 172.21.15.67:0/4209975399 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5100ad880 con 0x7ff5200d03a0 2024-04-17T10:37:26.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 -- 172.21.15.67:0/4209975399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200d03a0 msgr2=0x7ff52015c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:26.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 --2- 172.21.15.67:0/4209975399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200d03a0 0x7ff52015c7a0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7ff510009870 tx=0x7ff51001c560 comp rx=0 tx=0).stop 2024-04-17T10:37:26.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 -- 172.21.15.67:0/4209975399 shutdown_connections 2024-04-17T10:37:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 --2- 172.21.15.67:0/4209975399 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff52009a740 0x7ff5200cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 --2- 172.21.15.67:0/4209975399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5200d03a0 0x7ff52015c7a0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 -- 172.21.15.67:0/4209975399 >> 172.21.15.67:0/4209975399 conn(0x7ff5200d1690 msgr2=0x7ff5200d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 -- 172.21.15.67:0/4209975399 shutdown_connections 2024-04-17T10:37:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 -- 172.21.15.67:0/4209975399 wait complete. 2024-04-17T10:37:26.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 Processor -- start 2024-04-17T10:37:26.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.649+0000 7ff527200640 1 -- start start 2024-04-17T10:37:26.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff520099a50 0x7ff520153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52009a740 0x7ff5200b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5200d03a0 0x7ff5201555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5201686b0 con 0x7ff52009a740 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5200cd180 con 0x7ff520099a50 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff520168830 con 0x7ff5200d03a0 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52009a740 0x7ff5200b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52009a740 0x7ff5200b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53502/0 (socket says 172.21.15.67:53502) 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 -- 172.21.15.67:0/2791993171 learned_addr learned my addr 172.21.15.67:0/2791993171 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525e00640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff520099a50 0x7ff520153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff526800640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5200d03a0 0x7ff5201555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 -- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5200d03a0 msgr2=0x7ff5201555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5200d03a0 0x7ff5201555c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 -- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff520099a50 msgr2=0x7ff520153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff520099a50 0x7ff520153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 -- 172.21.15.67:0/2791993171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff520154520 con 0x7ff52009a740 2024-04-17T10:37:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff525400640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52009a740 0x7ff5200b1f80 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff5100b0040 tx=0x7ff510019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff51a000640 1 -- 172.21.15.67:0/2791993171 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5100b2040 con 0x7ff52009a740 2024-04-17T10:37:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff51a000640 1 -- 172.21.15.67:0/2791993171 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5100be070 con 0x7ff52009a740 2024-04-17T10:37:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff520154800 con 0x7ff52009a740 2024-04-17T10:37:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff51a000640 1 -- 172.21.15.67:0/2791993171 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5100b8420 con 0x7ff52009a740 2024-04-17T10:37:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff520154dc0 con 0x7ff52009a740 2024-04-17T10:37:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.653+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff520074dc0 con 0x7ff52009a740 2024-04-17T10:37:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.657+0000 7ff51a000640 1 -- 172.21.15.67:0/2791993171 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff5100ab070 con 0x7ff52009a740 2024-04-17T10:37:26.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.657+0000 7ff51a000640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff504066f60 0x7ff504069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:26.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.657+0000 7ff51a000640 1 -- 172.21.15.67:0/2791993171 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff51011f100 con 0x7ff52009a740 2024-04-17T10:37:26.648 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.657+0000 7ff525e00640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff504066f60 0x7ff504069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:26.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.657+0000 7ff525e00640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff504066f60 0x7ff504069420 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff514001770 tx=0x7ff514009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.661+0000 7ff51a000640 1 -- 172.21.15.67:0/2791993171 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff510016150 con 0x7ff52009a740 2024-04-17T10:37:26.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5200d24a0 con 0x7ff52009a740 2024-04-17T10:37:26.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff51a000640 1 -- 172.21.15.67:0/2791993171 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7ff5100ef800 con 0x7ff52009a740 2024-04-17T10:37:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:26.848 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:26.849 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:26.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff504066f60 msgr2=0x7ff504069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:26.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff504066f60 0x7ff504069420 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff514001770 tx=0x7ff514009040 comp rx=0 tx=0).stop 2024-04-17T10:37:26.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52009a740 msgr2=0x7ff5200b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:26.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52009a740 0x7ff5200b1f80 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7ff5100b0040 tx=0x7ff510019300 comp rx=0 tx=0).stop 2024-04-17T10:37:26.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 shutdown_connections 2024-04-17T10:37:26.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff504066f60 0x7ff504069420 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff5200d03a0 0x7ff5201555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff52009a740 0x7ff5200b1f80 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 --2- 172.21.15.67:0/2791993171 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff520099a50 0x7ff520153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:26.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 >> 172.21.15.67:0/2791993171 conn(0x7ff5200d1690 msgr2=0x7ff5200af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:26.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 shutdown_connections 2024-04-17T10:37:26.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:26.857+0000 7ff527200640 1 -- 172.21.15.67:0/2791993171 wait complete. 2024-04-17T10:37:27.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:27.272 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:28.273 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:28.399 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:37:28.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:37:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6847200640 1 Processor -- start 2024-04-17T10:37:28.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6847200640 1 -- start start 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6847200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6840145040 0x7f684006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6847200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f684006ef10 0x7f6840071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6847200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684006e2f0 con 0x7f6840071850 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6847200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68400a86c0 con 0x7f6840145040 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6847200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684015bf10 con 0x7f684006ef10 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f683fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6840145040 0x7f684006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6845200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6840071850 0x7f684015f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37646/0 (socket says 172.21.15.67:37646) 2024-04-17T10:37:28.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6845200640 1 -- 172.21.15.67:0/808610836 learned_addr learned my addr 172.21.15.67:0/808610836 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f6837e00640 1 --2- 172.21.15.67:0/808610836 >> v2:172.21.15.86:3300/0 conn(0x7f684006ef10 0x7f6840071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f683fe00640 1 -- 172.21.15.67:0/808610836 >> v2:172.21.15.86:3300/0 conn(0x7f684006ef10 msgr2=0x7f6840071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f683fe00640 1 --2- 172.21.15.67:0/808610836 >> v2:172.21.15.86:3300/0 conn(0x7f684006ef10 0x7f6840071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f683f400640 1 -- 172.21.15.67:0/808610836 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846667175 0 0) 0x7f684006e2f0 con 0x7f6840071850 2024-04-17T10:37:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f683fe00640 1 -- 172.21.15.67:0/808610836 >> v1:172.21.15.67:6789/0 conn(0x7f6840071850 legacy=0x7f684015f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:28.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.436+0000 7f683fe00640 1 -- 172.21.15.67:0/808610836 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6840071c60 con 0x7f6840145040 2024-04-17T10:37:28.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f683fe00640 1 --2- 172.21.15.67:0/808610836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684006e9d0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f68400727b0 tx=0x7f683001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a322cc32f68fd445 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f683f400640 1 -- 172.21.15.67:0/808610836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68300b5070 con 0x7f6840145040 2024-04-17T10:37:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f683f400640 1 -- 172.21.15.67:0/808610836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68300a8970 con 0x7f6840145040 2024-04-17T10:37:28.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f683f400640 1 -- 172.21.15.67:0/808610836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68300a8d40 con 0x7f6840145040 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 -- 172.21.15.67:0/808610836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 msgr2=0x7f684006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 --2- 172.21.15.67:0/808610836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684006e9d0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f68400727b0 tx=0x7f683001ec70 comp rx=0 tx=0).stop 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 -- 172.21.15.67:0/808610836 shutdown_connections 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 --2- 172.21.15.67:0/808610836 >> v2:172.21.15.86:3300/0 conn(0x7f684006ef10 0x7f6840071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 --2- 172.21.15.67:0/808610836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684006e9d0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 -- 172.21.15.67:0/808610836 >> 172.21.15.67:0/808610836 conn(0x7f684006ac90 msgr2=0x7f684006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 -- 172.21.15.67:0/808610836 shutdown_connections 2024-04-17T10:37:28.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.440+0000 7f6847200640 1 -- 172.21.15.67:0/808610836 wait complete. 2024-04-17T10:37:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 Processor -- start 2024-04-17T10:37:28.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 -- start start 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f684006ef10 0x7f684013c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6840071850 0x7f6840142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684006df80 con 0x7f6840145040 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684006dca0 con 0x7f684006ef10 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684006d370 con 0x7f6840071850 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684014f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53514/0 (socket says 172.21.15.67:53514) 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 -- 172.21.15.67:0/2049808866 learned_addr learned my addr 172.21.15.67:0/2049808866 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6837e00640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6840071850 0x7f6840142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f683fe00640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f684006ef10 0x7f684013c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:28.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 -- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6840071850 msgr2=0x7f6840142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6840071850 0x7f6840142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 -- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f684006ef10 msgr2=0x7f684013c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f684006ef10 0x7f684013c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 -- 172.21.15.67:0/2049808866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684014c8f0 con 0x7f6840145040 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6845200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684014f110 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f683800c7a0 tx=0x7f683800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f683cc00640 1 -- 172.21.15.67:0/2049808866 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6838018070 con 0x7f6840145040 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f683cc00640 1 -- 172.21.15.67:0/2049808866 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683800a640 con 0x7f6840145040 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f683cc00640 1 -- 172.21.15.67:0/2049808866 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683800aa70 con 0x7f6840145040 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684014cbd0 con 0x7f6840145040 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.444+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684014d190 con 0x7f6840145040 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.448+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6840072d50 con 0x7f6840145040 2024-04-17T10:37:28.440 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.448+0000 7f683cc00640 1 -- 172.21.15.67:0/2049808866 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f683801b070 con 0x7f6840145040 2024-04-17T10:37:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.448+0000 7f683cc00640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f681c066f60 0x7f681c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.448+0000 7f683cc00640 1 -- 172.21.15.67:0/2049808866 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6838084840 con 0x7f6840145040 2024-04-17T10:37:28.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.448+0000 7f683fe00640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f681c066f60 0x7f681c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:28.443 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.452+0000 7f683fe00640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f681c066f60 0x7f681c069420 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f68300b2000 tx=0x7f68300aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:28.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.452+0000 7f683cc00640 1 -- 172.21.15.67:0/2049808866 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6838054f40 con 0x7f6840145040 2024-04-17T10:37:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.648+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6840074e30 con 0x7f6840145040 2024-04-17T10:37:28.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.648+0000 7f683cc00640 1 -- 172.21.15.67:0/2049808866 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6838058fe0 con 0x7f6840145040 2024-04-17T10:37:28.643 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:28.643 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:28.643 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f681c066f60 msgr2=0x7f681c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f681c066f60 0x7f681c069420 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f68300b2000 tx=0x7f68300aa070 comp rx=0 tx=0).stop 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 msgr2=0x7f684014f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684014f110 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f683800c7a0 tx=0x7f683800cc60 comp rx=0 tx=0).stop 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 shutdown_connections 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f681c066f60 0x7f681c069420 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6840145040 0x7f684014f110 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6840071850 0x7f6840142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 --2- 172.21.15.67:0/2049808866 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f684006ef10 0x7f684013c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 >> 172.21.15.67:0/2049808866 conn(0x7f684006ac90 msgr2=0x7f68400b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 shutdown_connections 2024-04-17T10:37:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:28.652+0000 7f6847200640 1 -- 172.21.15.67:0/2049808866 wait complete. 2024-04-17T10:37:29.066 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:29.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:30.067 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:30.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a6a00640 1 Processor -- start 2024-04-17T10:37:30.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a6a00640 1 -- start start 2024-04-17T10:37:30.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a6a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f33a00ce020 0x7f33a01620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:30.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a6a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33a00ce400 0x7f33a015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a6a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a015eb50 con 0x7f33a015e550 2024-04-17T10:37:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a6a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a0161650 con 0x7f33a00ce400 2024-04-17T10:37:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a6a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a01625e0 con 0x7f33a00ce020 2024-04-17T10:37:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f339f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f33a00ce400 0x7f33a015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a4a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f33a015e550 0x7f33a00cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37654/0 (socket says 172.21.15.67:37654) 2024-04-17T10:37:30.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.228+0000 7f33a4a00640 1 -- 172.21.15.67:0/1834722164 learned_addr learned my addr 172.21.15.67:0/1834722164 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339fe00640 1 --2- 172.21.15.67:0/1834722164 >> v2:172.21.15.86:3300/0 conn(0x7f33a00ce020 0x7f33a01620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339f400640 1 -- 172.21.15.67:0/1834722164 >> v2:172.21.15.86:3300/0 conn(0x7f33a00ce020 msgr2=0x7f33a01620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339f400640 1 --2- 172.21.15.67:0/1834722164 >> v2:172.21.15.86:3300/0 conn(0x7f33a00ce020 0x7f33a01620a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339f400640 1 -- 172.21.15.67:0/1834722164 >> v1:172.21.15.67:6789/0 conn(0x7f33a015e550 legacy=0x7f33a00cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339f400640 1 -- 172.21.15.67:0/1834722164 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a015ee30 con 0x7f33a00ce400 2024-04-17T10:37:30.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339ea00640 1 -- 172.21.15.67:0/1834722164 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622314630 0 0) 0x7f33a015eb50 con 0x7f33a015e550 2024-04-17T10:37:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339f400640 1 --2- 172.21.15.67:0/1834722164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce400 0x7f33a015e010 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097c0 tx=0x7f338c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29efabde443af67c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339ea00640 1 -- 172.21.15.67:0/1834722164 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c0b1070 con 0x7f33a00ce400 2024-04-17T10:37:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339ea00640 1 -- 172.21.15.67:0/1834722164 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c0ac640 con 0x7f33a00ce400 2024-04-17T10:37:30.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.232+0000 7f339ea00640 1 -- 172.21.15.67:0/1834722164 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c0aca10 con 0x7f33a00ce400 2024-04-17T10:37:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- 172.21.15.67:0/1834722164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce400 msgr2=0x7f33a015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 --2- 172.21.15.67:0/1834722164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce400 0x7f33a015e010 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f338c0097c0 tx=0x7f338c01c420 comp rx=0 tx=0).stop 2024-04-17T10:37:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- 172.21.15.67:0/1834722164 shutdown_connections 2024-04-17T10:37:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 --2- 172.21.15.67:0/1834722164 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce400 0x7f33a015e010 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 --2- 172.21.15.67:0/1834722164 >> v2:172.21.15.86:3300/0 conn(0x7f33a00ce020 0x7f33a01620a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- 172.21.15.67:0/1834722164 >> 172.21.15.67:0/1834722164 conn(0x7f33a00b33f0 msgr2=0x7f33a00b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- 172.21.15.67:0/1834722164 shutdown_connections 2024-04-17T10:37:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- 172.21.15.67:0/1834722164 wait complete. 2024-04-17T10:37:30.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 Processor -- start 2024-04-17T10:37:30.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- start start 2024-04-17T10:37:30.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce020 0x7f33a015c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a00ce400 0x7f33a00b71a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a015e550 0x7f33a01539b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a006c0f0 con 0x7f33a00ce020 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a006bf70 con 0x7f33a00ce400 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a6a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a00c84b0 con 0x7f33a015e550 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce020 0x7f33a015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce020 0x7f33a015c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53526/0 (socket says 172.21.15.67:53526) 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 -- 172.21.15.67:0/2816500776 learned_addr learned my addr 172.21.15.67:0/2816500776 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339f400640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a00ce400 0x7f33a00b71a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f33a4a00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a015e550 0x7f33a01539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 -- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a015e550 msgr2=0x7f33a01539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a015e550 0x7f33a01539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 -- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a00ce400 msgr2=0x7f33a00b71a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:30.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a00ce400 0x7f33a00b71a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.236+0000 7f339fe00640 1 -- 172.21.15.67:0/2816500776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a015cf00 con 0x7f33a00ce020 2024-04-17T10:37:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f339fe00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce020 0x7f33a015c920 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f339000d800 tx=0x7f339000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f3397e00640 1 -- 172.21.15.67:0/2816500776 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000b6c0 con 0x7f33a00ce020 2024-04-17T10:37:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f3397e00640 1 -- 172.21.15.67:0/2816500776 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3390019070 con 0x7f33a00ce020 2024-04-17T10:37:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f3397e00640 1 -- 172.21.15.67:0/2816500776 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339000ba60 con 0x7f33a00ce020 2024-04-17T10:37:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a015d1e0 con 0x7f33a00ce020 2024-04-17T10:37:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a0153ef0 con 0x7f33a00ce020 2024-04-17T10:37:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a0070ab0 con 0x7f33a00ce020 2024-04-17T10:37:30.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f3397e00640 1 -- 172.21.15.67:0/2816500776 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f339000bc00 con 0x7f33a00ce020 2024-04-17T10:37:30.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f3397e00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:30.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f3397e00640 1 -- 172.21.15.67:0/2816500776 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f3390085610 con 0x7f33a00ce020 2024-04-17T10:37:30.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.240+0000 7f339f400640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:30.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.244+0000 7f339f400640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f338c0ae000 tx=0x7f338c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:30.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.244+0000 7f3397e00640 1 -- 172.21.15.67:0/2816500776 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3390056400 con 0x7f33a00ce020 2024-04-17T10:37:30.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33a0074e30 con 0x7f33a00ce020 2024-04-17T10:37:30.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f3397e00640 1 -- 172.21.15.67:0/2816500776 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3390055d00 con 0x7f33a00ce020 2024-04-17T10:37:30.437 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:30.437 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:30.437 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:30.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 msgr2=0x7f3380069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f338c0ae000 tx=0x7f338c008040 comp rx=0 tx=0).stop 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce020 msgr2=0x7f33a015c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce020 0x7f33a015c920 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f339000d800 tx=0x7f339000dcc0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 shutdown_connections 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3380066f60 0x7f3380069420 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f33a015e550 0x7f33a01539b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f33a00ce400 0x7f33a00b71a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 --2- 172.21.15.67:0/2816500776 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f33a00ce020 0x7f33a015c920 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 >> 172.21.15.67:0/2816500776 conn(0x7f33a00b33f0 msgr2=0x7f33a00cbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 shutdown_connections 2024-04-17T10:37:30.439 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:30.444+0000 7f33a6a00640 1 -- 172.21.15.67:0/2816500776 wait complete. 2024-04-17T10:37:30.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:30.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:31.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.020+0000 7f9463800640 1 Processor -- start 2024-04-17T10:37:32.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.020+0000 7f9463800640 1 -- start start 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9463800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f945c06e6e0 0x7f945c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9463800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f945c1441f0 0x7f945c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9463800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945c15ae40 con 0x7f945c06eac0 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9463800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c0cf430 con 0x7f945c1441f0 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9463800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f945c071440 con 0x7f945c06e6e0 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9453e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f945c06eac0 0x7f945c143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44418/0 (socket says 172.21.15.67:44418) 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9453e00640 1 -- 172.21.15.67:0/1626049692 learned_addr learned my addr 172.21.15.67:0/1626049692 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9461800640 1 --2- 172.21.15.67:0/1626049692 >> v2:172.21.15.67:3300/0 conn(0x7f945c1441f0 0x7f945c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9460e00640 1 --2- 172.21.15.67:0/1626049692 >> v2:172.21.15.86:3300/0 conn(0x7f945c06e6e0 0x7f945c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f945be00640 1 -- 172.21.15.67:0/1626049692 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965680967 0 0) 0x7f945c15ae40 con 0x7f945c06eac0 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f945be00640 1 -- 172.21.15.67:0/1626049692 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9440005740 con 0x7f945c06eac0 2024-04-17T10:37:32.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9461800640 1 -- 172.21.15.67:0/1626049692 >> v2:172.21.15.86:3300/0 conn(0x7f945c06e6e0 msgr2=0x7f945c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9461800640 1 --2- 172.21.15.67:0/1626049692 >> v2:172.21.15.86:3300/0 conn(0x7f945c06e6e0 0x7f945c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9461800640 1 -- 172.21.15.67:0/1626049692 >> v1:172.21.15.67:6789/0 conn(0x7f945c06eac0 legacy=0x7f945c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.024+0000 7f9461800640 1 -- 172.21.15.67:0/1626049692 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c071720 con 0x7f945c1441f0 2024-04-17T10:37:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9461800640 1 --2- 172.21.15.67:0/1626049692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c0ceef0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f945c0cf620 tx=0x7f945401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d1ed141b031f2f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f945be00640 1 -- 172.21.15.67:0/1626049692 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540b2070 con 0x7f945c1441f0 2024-04-17T10:37:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f945be00640 1 -- 172.21.15.67:0/1626049692 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94540a5830 con 0x7f945c1441f0 2024-04-17T10:37:32.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f945be00640 1 -- 172.21.15.67:0/1626049692 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540a5c00 con 0x7f945c1441f0 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 -- 172.21.15.67:0/1626049692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 msgr2=0x7f945c0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 --2- 172.21.15.67:0/1626049692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c0ceef0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f945c0cf620 tx=0x7f945401c870 comp rx=0 tx=0).stop 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 -- 172.21.15.67:0/1626049692 shutdown_connections 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 --2- 172.21.15.67:0/1626049692 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c0ceef0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 --2- 172.21.15.67:0/1626049692 >> v2:172.21.15.86:3300/0 conn(0x7f945c06e6e0 0x7f945c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 -- 172.21.15.67:0/1626049692 >> 172.21.15.67:0/1626049692 conn(0x7f945c06ac90 msgr2=0x7f945c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 -- 172.21.15.67:0/1626049692 shutdown_connections 2024-04-17T10:37:32.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 -- 172.21.15.67:0/1626049692 wait complete. 2024-04-17T10:37:32.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.028+0000 7f9463800640 1 Processor -- start 2024-04-17T10:37:32.023 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 -- start start 2024-04-17T10:37:32.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f945c06e6e0 0x7f945c13e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:32.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f945c06eac0 0x7f945c13c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:32.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c13cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c06dd00 con 0x7f945c1441f0 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945c06db80 con 0x7f945c06e6e0 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945c06de80 con 0x7f945c06eac0 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c13cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9460e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f945c06e6e0 0x7f945c13e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c13cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59762/0 (socket says 172.21.15.67:59762) 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 -- 172.21.15.67:0/2495330005 learned_addr learned my addr 172.21.15.67:0/2495330005 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9453e00640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f945c06eac0 0x7f945c13c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 -- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f945c06eac0 msgr2=0x7f945c13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f945c06eac0 0x7f945c13c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 -- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f945c06e6e0 msgr2=0x7f945c13e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f945c06e6e0 0x7f945c13e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 -- 172.21.15.67:0/2495330005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c13d2c0 con 0x7f945c1441f0 2024-04-17T10:37:32.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9461800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c13cd80 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f94540b0040 tx=0x7f9454019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:32.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9459600640 1 -- 172.21.15.67:0/2495330005 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540b2050 con 0x7f945c1441f0 2024-04-17T10:37:32.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9459600640 1 -- 172.21.15.67:0/2495330005 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94540be070 con 0x7f945c1441f0 2024-04-17T10:37:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f945c13d540 con 0x7f945c1441f0 2024-04-17T10:37:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9459600640 1 -- 172.21.15.67:0/2495330005 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94540ad870 con 0x7f945c1441f0 2024-04-17T10:37:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f945c13db00 con 0x7f945c1441f0 2024-04-17T10:37:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.032+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f945c072d50 con 0x7f945c1441f0 2024-04-17T10:37:32.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.036+0000 7f9459600640 1 -- 172.21.15.67:0/2495330005 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f94540ada10 con 0x7f945c1441f0 2024-04-17T10:37:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.036+0000 7f9459600640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9438066f60 0x7f9438069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:32.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.036+0000 7f9459600640 1 -- 172.21.15.67:0/2495330005 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f94541203d0 con 0x7f945c1441f0 2024-04-17T10:37:32.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.036+0000 7f9460e00640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9438066f60 0x7f9438069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:32.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.036+0000 7f9460e00640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9438066f60 0x7f9438069420 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f944c009b60 tx=0x7f944c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:32.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.040+0000 7f9459600640 1 -- 172.21.15.67:0/2495330005 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f945c13d2c0 con 0x7f945c1441f0 2024-04-17T10:37:32.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.232+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f945c0b2c60 con 0x7f945c1441f0 2024-04-17T10:37:32.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9459600640 1 -- 172.21.15.67:0/2495330005 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f9454016520 con 0x7f945c1441f0 2024-04-17T10:37:32.227 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:32.227 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:32.228 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:32.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9438066f60 msgr2=0x7f9438069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:32.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9438066f60 0x7f9438069420 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f944c009b60 tx=0x7f944c009190 comp rx=0 tx=0).stop 2024-04-17T10:37:32.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 msgr2=0x7f945c13cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c13cd80 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f94540b0040 tx=0x7f9454019610 comp rx=0 tx=0).stop 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 shutdown_connections 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9438066f60 0x7f9438069420 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f945c1441f0 0x7f945c13cd80 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f945c06eac0 0x7f945c13c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 --2- 172.21.15.67:0/2495330005 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f945c06e6e0 0x7f945c13e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 >> 172.21.15.67:0/2495330005 conn(0x7f945c06ac90 msgr2=0x7f945c142ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 shutdown_connections 2024-04-17T10:37:32.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:32.236+0000 7f9463800640 1 -- 172.21.15.67:0/2495330005 wait complete. 2024-04-17T10:37:32.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:32.651 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:33.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.816+0000 7fa0b1e00640 1 Processor -- start 2024-04-17T10:37:33.810 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.816+0000 7fa0b1e00640 1 -- start start 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0b1e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0ac06db90 0x7fa0ac071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0b1e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ac0ad8f0 con 0x7fa0ac0721a0 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0b1e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0ac15dfa0 con 0x7fa0ac06db90 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0b1e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac06faa0 con 0x7fa0ac15dbc0 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0ac06db90 0x7fa0ac071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa0ac06db90 0x7fa0ac071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59768/0 (socket says 172.21.15.67:59768) 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 -- 172.21.15.67:0/1816388368 learned_addr learned my addr 172.21.15.67:0/1816388368 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0aaa00640 1 --2- 172.21.15.67:0/1816388368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 -- 172.21.15.67:0/1816388368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 msgr2=0x7fa0ac06f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:33.812 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 --2- 172.21.15.67:0/1816388368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac06f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 -- 172.21.15.67:0/1816388368 >> v1:172.21.15.86:6789/0 conn(0x7fa0ac0721a0 legacy=0x7fa0ac06b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:33.813 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 -- 172.21.15.67:0/1816388368 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac0adbd0 con 0x7fa0ac06db90 2024-04-17T10:37:33.814 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.820+0000 7fa0a2000640 1 --2- 172.21.15.67:0/1816388368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac06db90 0x7fa0ac071c60 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fa094009870 tx=0x7fa09401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c02db5caf146eae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:33.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0aa000640 1 -- 172.21.15.67:0/1816388368 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0940b0070 con 0x7fa0ac06db90 2024-04-17T10:37:33.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0aa000640 1 -- 172.21.15.67:0/1816388368 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0940ab600 con 0x7fa0ac06db90 2024-04-17T10:37:33.815 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0aa000640 1 -- 172.21.15.67:0/1816388368 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0940ab9d0 con 0x7fa0ac06db90 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- 172.21.15.67:0/1816388368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac06db90 msgr2=0x7fa0ac071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/1816388368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac06db90 0x7fa0ac071c60 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fa094009870 tx=0x7fa09401c4d0 comp rx=0 tx=0).stop 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- 172.21.15.67:0/1816388368 shutdown_connections 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/1816388368 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac06db90 0x7fa0ac071c60 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/1816388368 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac06f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- 172.21.15.67:0/1816388368 >> 172.21.15.67:0/1816388368 conn(0x7fa0ac0af670 msgr2=0x7fa0ac0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- 172.21.15.67:0/1816388368 shutdown_connections 2024-04-17T10:37:33.816 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- 172.21.15.67:0/1816388368 wait complete. 2024-04-17T10:37:33.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 Processor -- start 2024-04-17T10:37:33.817 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- start start 2024-04-17T10:37:33.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0ac06db90 0x7fa0ac14ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac0721a0 0x7fa0ac15eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac15fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac06c730 con 0x7fa0ac0721a0 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac06c5b0 con 0x7fa0ac06db90 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0b1e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac06af70 con 0x7fa0ac15dbc0 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0a2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac0721a0 0x7fa0ac15eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0a2000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac0721a0 0x7fa0ac15eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59774/0 (socket says 172.21.15.67:59774) 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0a2000640 1 -- 172.21.15.67:0/453293849 learned_addr learned my addr 172.21.15.67:0/453293849 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0aaa00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0ac06db90 0x7fa0ac14ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.824+0000 7fa0ab400640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac15fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a2000640 1 -- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 msgr2=0x7fa0ac15fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a2000640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac15fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a2000640 1 -- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0ac06db90 msgr2=0x7fa0ac14ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:33.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a2000640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0ac06db90 0x7fa0ac14ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a2000640 1 -- 172.21.15.67:0/453293849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac14c500 con 0x7fa0ac0721a0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a2000640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac0721a0 0x7fa0ac15eb30 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fa0940ae040 tx=0x7fa094019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a3e00640 1 -- 172.21.15.67:0/453293849 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0940ba070 con 0x7fa0ac0721a0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a3e00640 1 -- 172.21.15.67:0/453293849 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0940b4420 con 0x7fa0ac0721a0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a3e00640 1 -- 172.21.15.67:0/453293849 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0940b0040 con 0x7fa0ac0721a0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ac14c7e0 con 0x7fa0ac0721a0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ac14cda0 con 0x7fa0ac0721a0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ac09e260 con 0x7fa0ac0721a0 2024-04-17T10:37:33.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a3e00640 1 -- 172.21.15.67:0/453293849 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa0940b4810 con 0x7fa0ac0721a0 2024-04-17T10:37:33.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a3e00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa088066f60 0x7fa088069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:33.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.828+0000 7fa0a3e00640 1 -- 172.21.15.67:0/453293849 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa09411e340 con 0x7fa0ac0721a0 2024-04-17T10:37:33.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.832+0000 7fa0aaa00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa088066f60 0x7fa088069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:33.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.832+0000 7fa0aaa00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa088066f60 0x7fa088069420 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa09c003b90 tx=0x7fa09c0098d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:33.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:33.836+0000 7fa0a3e00640 1 -- 172.21.15.67:0/453293849 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa094016150 con 0x7fa0ac0721a0 2024-04-17T10:37:34.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.032+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0ac074e30 con 0x7fa0ac0721a0 2024-04-17T10:37:34.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.032+0000 7fa0a3e00640 1 -- 172.21.15.67:0/453293849 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fa0940eead0 con 0x7fa0ac0721a0 2024-04-17T10:37:34.027 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:34.027 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:34.027 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:34.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa088066f60 msgr2=0x7fa088069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa088066f60 0x7fa088069420 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fa09c003b90 tx=0x7fa09c0098d0 comp rx=0 tx=0).stop 2024-04-17T10:37:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac0721a0 msgr2=0x7fa0ac15eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac0721a0 0x7fa0ac15eb30 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fa0940ae040 tx=0x7fa094019270 comp rx=0 tx=0).stop 2024-04-17T10:37:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 shutdown_connections 2024-04-17T10:37:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa088066f60 0x7fa088069420 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa0ac15dbc0 0x7fa0ac15fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0ac0721a0 0x7fa0ac15eb30 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 --2- 172.21.15.67:0/453293849 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa0ac06db90 0x7fa0ac14ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 >> 172.21.15.67:0/453293849 conn(0x7fa0ac0af670 msgr2=0x7fa0ac0b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 shutdown_connections 2024-04-17T10:37:34.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:34.036+0000 7fa0b1e00640 1 -- 172.21.15.67:0/453293849 wait complete. 2024-04-17T10:37:34.450 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:34.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:35.451 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:35.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.612+0000 7f23ed800640 1 Processor -- start 2024-04-17T10:37:35.606 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.612+0000 7f23ed800640 1 -- start start 2024-04-17T10:37:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.612+0000 7f23ed800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23e8099a50 0x7f23e80d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.612+0000 7f23ed800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e809a830 0x7f23e806cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23ed800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23e80d0930 con 0x7f23e815dc60 2024-04-17T10:37:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23ed800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80d0bd0 con 0x7f23e8099a50 2024-04-17T10:37:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23ed800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e806bda0 con 0x7f23e809a830 2024-04-17T10:37:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23e8099a50 0x7f23e80d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:35.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f23e8099a50 0x7f23e80d03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59776/0 (socket says 172.21.15.67:59776) 2024-04-17T10:37:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 -- 172.21.15.67:0/430805921 learned_addr learned my addr 172.21.15.67:0/430805921 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23ece00640 1 --2- 172.21.15.67:0/430805921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e809a830 0x7f23e806cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 -- 172.21.15.67:0/430805921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e809a830 msgr2=0x7f23e806cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 --2- 172.21.15.67:0/430805921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e809a830 0x7f23e806cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7400640 1 -- 172.21.15.67:0/430805921 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188933035 0 0) 0x7f23e80d0930 con 0x7f23e815dc60 2024-04-17T10:37:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 -- 172.21.15.67:0/430805921 >> v1:172.21.15.86:6789/0 conn(0x7f23e815dc60 legacy=0x7f23e8099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:35.608 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 -- 172.21.15.67:0/430805921 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e806c080 con 0x7f23e8099a50 2024-04-17T10:37:35.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7e00640 1 --2- 172.21.15.67:0/430805921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e80d03f0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009870 tx=0x7f23d801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83dc2a47c77cef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7400640 1 -- 172.21.15.67:0/430805921 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80aa600 con 0x7f23e8099a50 2024-04-17T10:37:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7400640 1 -- 172.21.15.67:0/430805921 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d80af070 con 0x7f23e8099a50 2024-04-17T10:37:35.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.616+0000 7f23e7400640 1 -- 172.21.15.67:0/430805921 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80aa9c0 con 0x7f23e8099a50 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- 172.21.15.67:0/430805921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 msgr2=0x7f23e80d03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 --2- 172.21.15.67:0/430805921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e80d03f0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009870 tx=0x7f23d801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- 172.21.15.67:0/430805921 shutdown_connections 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 --2- 172.21.15.67:0/430805921 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e809a830 0x7f23e806cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 --2- 172.21.15.67:0/430805921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e80d03f0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- 172.21.15.67:0/430805921 >> 172.21.15.67:0/430805921 conn(0x7f23e80d1610 msgr2=0x7f23e80d1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- 172.21.15.67:0/430805921 shutdown_connections 2024-04-17T10:37:35.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- 172.21.15.67:0/430805921 wait complete. 2024-04-17T10:37:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 Processor -- start 2024-04-17T10:37:35.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- start start 2024-04-17T10:37:35.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e814c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23e809a830 0x7f23e80b1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e815dc60 0x7f23e80b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e816acb0 con 0x7f23e8099a50 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e816ab30 con 0x7f23e809a830 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ed800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e816ae30 con 0x7f23e815dc60 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e814c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e814c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59778/0 (socket says 172.21.15.67:59778) 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 -- 172.21.15.67:0/2544080523 learned_addr learned my addr 172.21.15.67:0/2544080523 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23df400640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23e809a830 0x7f23e80b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23ece00640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e815dc60 0x7f23e80b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 -- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e815dc60 msgr2=0x7f23e80b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e815dc60 0x7f23e80b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 -- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23e809a830 msgr2=0x7f23e80b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23e809a830 0x7f23e80b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.620+0000 7f23e7e00640 1 -- 172.21.15.67:0/2544080523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e814db30 con 0x7f23e8099a50 2024-04-17T10:37:35.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23e7e00640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e814c1b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f23d801a0d0 tx=0x7f23d801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23e4c00640 1 -- 172.21.15.67:0/2544080523 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80b9070 con 0x7f23e8099a50 2024-04-17T10:37:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23e4c00640 1 -- 172.21.15.67:0/2544080523 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d8004220 con 0x7f23e8099a50 2024-04-17T10:37:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e814d570 con 0x7f23e8099a50 2024-04-17T10:37:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23e4c00640 1 -- 172.21.15.67:0/2544080523 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d80aa850 con 0x7f23e8099a50 2024-04-17T10:37:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e81520c0 con 0x7f23e8099a50 2024-04-17T10:37:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e8074dc0 con 0x7f23e8099a50 2024-04-17T10:37:35.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23e4c00640 1 -- 172.21.15.67:0/2544080523 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f23d80a9070 con 0x7f23e8099a50 2024-04-17T10:37:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.624+0000 7f23e4c00640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23c4066f90 0x7f23c4069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.628+0000 7f23e4c00640 1 -- 172.21.15.67:0/2544080523 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f23d811ee90 con 0x7f23e8099a50 2024-04-17T10:37:35.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.628+0000 7f23df400640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23c4066f90 0x7f23c4069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:35.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.628+0000 7f23df400640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23c4066f90 0x7f23c4069450 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f23e80b2650 tx=0x7f23d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:35.623 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.632+0000 7f23e4c00640 1 -- 172.21.15.67:0/2544080523 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f23d8016150 con 0x7f23e8099a50 2024-04-17T10:37:35.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.824+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23e809a610 con 0x7f23e8099a50 2024-04-17T10:37:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23e4c00640 1 -- 172.21.15.67:0/2544080523 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f23d80ef620 con 0x7f23e8099a50 2024-04-17T10:37:35.819 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:35.819 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:35.819 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23c4066f90 msgr2=0x7f23c4069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23c4066f90 0x7f23c4069450 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f23e80b2650 tx=0x7f23d0009040 comp rx=0 tx=0).stop 2024-04-17T10:37:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 msgr2=0x7f23e814c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e814c1b0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f23d801a0d0 tx=0x7f23d801ace0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 shutdown_connections 2024-04-17T10:37:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f23c4066f90 0x7f23c4069450 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f23e815dc60 0x7f23e80b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f23e809a830 0x7f23e80b1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 --2- 172.21.15.67:0/2544080523 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f23e8099a50 0x7f23e814c1b0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 >> 172.21.15.67:0/2544080523 conn(0x7f23e80d1610 msgr2=0x7f23e80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 shutdown_connections 2024-04-17T10:37:35.822 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:35.828+0000 7f23ed800640 1 -- 172.21.15.67:0/2544080523 wait complete. 2024-04-17T10:37:36.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:36.242 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:37.243 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6ebe00640 1 Processor -- start 2024-04-17T10:37:37.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6ebe00640 1 -- start start 2024-04-17T10:37:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6ebe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4145040 0x7fc6e406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6ebe00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e406e2f0 con 0x7fc6e4071a20 2024-04-17T10:37:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6ebe00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6e406d6e0 con 0x7fc6e406ef10 2024-04-17T10:37:37.397 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6ebe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e40a86c0 con 0x7fc6e4145040 2024-04-17T10:37:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e9e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc6e4071a20 0x7fc6e40cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:44430/0 (socket says 172.21.15.67:44430) 2024-04-17T10:37:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e9e00640 1 -- 172.21.15.67:0/2617297179 learned_addr learned my addr 172.21.15.67:0/2617297179 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e9400640 1 --2- 172.21.15.67:0/2617297179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4145040 0x7fc6e406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e3e00640 1 -- 172.21.15.67:0/2617297179 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561826604 0 0) 0x7fc6e406e2f0 con 0x7fc6e4071a20 2024-04-17T10:37:37.398 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e3e00640 1 -- 172.21.15.67:0/2617297179 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6cc003530 con 0x7fc6e4071a20 2024-04-17T10:37:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e3e00640 1 -- 172.21.15.67:0/2617297179 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761811231 0 0) 0x7fc6e406d6e0 con 0x7fc6e406ef10 2024-04-17T10:37:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e9400640 1 -- 172.21.15.67:0/2617297179 >> v1:172.21.15.86:6789/0 conn(0x7fc6e406ef10 legacy=0x7fc6e4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e9400640 1 -- 172.21.15.67:0/2617297179 >> v1:172.21.15.67:6789/0 conn(0x7fc6e4071a20 legacy=0x7fc6e40cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:37.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.404+0000 7fc6e9400640 1 -- 172.21.15.67:0/2617297179 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e406d9c0 con 0x7fc6e4145040 2024-04-17T10:37:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6e9400640 1 --2- 172.21.15.67:0/2617297179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4145040 0x7fc6e406e9d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80097f0 tx=0x7fc6d801eb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc6a3d319409b909 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6e3e00640 1 -- 172.21.15.67:0/2617297179 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d80b5070 con 0x7fc6e4145040 2024-04-17T10:37:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6e3e00640 1 -- 172.21.15.67:0/2617297179 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d80a85d0 con 0x7fc6e4145040 2024-04-17T10:37:37.401 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6e3e00640 1 -- 172.21.15.67:0/2617297179 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc001a00 con 0x7fc6e4145040 2024-04-17T10:37:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6ebe00640 1 -- 172.21.15.67:0/2617297179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4145040 msgr2=0x7fc6e406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/2617297179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4145040 0x7fc6e406e9d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80097f0 tx=0x7fc6d801eb10 comp rx=0 tx=0).stop 2024-04-17T10:37:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6ebe00640 1 -- 172.21.15.67:0/2617297179 shutdown_connections 2024-04-17T10:37:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/2617297179 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4145040 0x7fc6e406e9d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:37.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6ebe00640 1 -- 172.21.15.67:0/2617297179 >> 172.21.15.67:0/2617297179 conn(0x7fc6e406ac90 msgr2=0x7fc6e406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6ebe00640 1 -- 172.21.15.67:0/2617297179 shutdown_connections 2024-04-17T10:37:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.408+0000 7fc6ebe00640 1 -- 172.21.15.67:0/2617297179 wait complete. 2024-04-17T10:37:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 Processor -- start 2024-04-17T10:37:37.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 -- start start 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e406ef10 0x7fc6e41f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4071a20 0x7fc6e41435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e4145040 0x7fc6e415e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e406df80 con 0x7fc6e4145040 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e415bf10 con 0x7fc6e406ef10 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6e406d370 con 0x7fc6e4071a20 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e4145040 0x7fc6e415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e4145040 0x7fc6e415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59794/0 (socket says 172.21.15.67:59794) 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 -- 172.21.15.67:0/4142945422 learned_addr learned my addr 172.21.15.67:0/4142945422 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:37.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e8a00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4071a20 0x7fc6e41435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9400640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e406ef10 0x7fc6e41f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 -- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4071a20 msgr2=0x7fc6e41435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4071a20 0x7fc6e41435d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 -- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e406ef10 msgr2=0x7fc6e41f3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e406ef10 0x7fc6e41f3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 -- 172.21.15.67:0/4142945422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e41f4520 con 0x7fc6e4145040 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e9e00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e4145040 0x7fc6e415e430 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00cb30 tx=0x7fc6dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e1600640 1 -- 172.21.15.67:0/4142945422 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc018070 con 0x7fc6e4145040 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e1600640 1 -- 172.21.15.67:0/4142945422 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc00a830 con 0x7fc6e4145040 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6e41419a0 con 0x7fc6e4145040 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6e1600640 1 -- 172.21.15.67:0/4142945422 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6dc00abe0 con 0x7fc6e4145040 2024-04-17T10:37:37.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.412+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6e41421e0 con 0x7fc6e4145040 2024-04-17T10:37:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.416+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6e4072d50 con 0x7fc6e4145040 2024-04-17T10:37:37.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.416+0000 7fc6e1600640 1 -- 172.21.15.67:0/4142945422 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fc6dc01b070 con 0x7fc6e4145040 2024-04-17T10:37:37.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.416+0000 7fc6e1600640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc6c4066f60 0x7fc6c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:37.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.416+0000 7fc6e1600640 1 -- 172.21.15.67:0/4142945422 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fc6dc084cd0 con 0x7fc6e4145040 2024-04-17T10:37:37.409 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.416+0000 7fc6e9400640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc6c4066f60 0x7fc6c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:37.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.416+0000 7fc6e9400640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc6c4066f60 0x7fc6c4069420 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80b2000 tx=0x7fc6d8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:37.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.420+0000 7fc6e1600640 1 -- 172.21.15.67:0/4142945422 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fc6e41419a0 con 0x7fc6e4145040 2024-04-17T10:37:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.616+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6e4140640 con 0x7fc6e4145040 2024-04-17T10:37:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.616+0000 7fc6e1600640 1 -- 172.21.15.67:0/4142945422 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fc6e41f4520 con 0x7fc6e4145040 2024-04-17T10:37:37.610 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:37.610 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:37.610 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc6c4066f60 msgr2=0x7fc6c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc6c4066f60 0x7fc6c4069420 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fc6d80b2000 tx=0x7fc6d8008040 comp rx=0 tx=0).stop 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e4145040 msgr2=0x7fc6e415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e4145040 0x7fc6e415e430 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7fc6dc00cb30 tx=0x7fc6dc00a060 comp rx=0 tx=0).stop 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 shutdown_connections 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fc6c4066f60 0x7fc6c4069420 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6e4145040 0x7fc6e415e430 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fc6e4071a20 0x7fc6e41435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 --2- 172.21.15.67:0/4142945422 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fc6e406ef10 0x7fc6e41f3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:37.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 >> 172.21.15.67:0/4142945422 conn(0x7fc6e406ac90 msgr2=0x7fc6e4071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 shutdown_connections 2024-04-17T10:37:37.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:37.620+0000 7fc6ebe00640 1 -- 172.21.15.67:0/4142945422 wait complete. 2024-04-17T10:37:38.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:38.034 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:39.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:39.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a8000640 1 Processor -- start 2024-04-17T10:37:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a8000640 1 -- start start 2024-04-17T10:37:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a8000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a006e6e0 0x7f74a00ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:39.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a8000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74a01441f0 0x7f74a00ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a8000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a015ae40 con 0x7f74a006eac0 2024-04-17T10:37:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a8000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a00cf430 con 0x7f74a01441f0 2024-04-17T10:37:39.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a8000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a0071440 con 0x7f74a006e6e0 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74a01441f0 0x7f74a00ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a6000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f74a01441f0 0x7f74a00ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59796/0 (socket says 172.21.15.67:59796) 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a6000640 1 -- 172.21.15.67:0/39244850 learned_addr learned my addr 172.21.15.67:0/39244850 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a5600640 1 --2- 172.21.15.67:0/39244850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a006e6e0 0x7f74a00ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a6000640 1 -- 172.21.15.67:0/39244850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a006e6e0 msgr2=0x7f74a00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a6000640 1 --2- 172.21.15.67:0/39244850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a006e6e0 0x7f74a00ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a6000640 1 -- 172.21.15.67:0/39244850 >> v1:172.21.15.86:6789/0 conn(0x7f74a006eac0 legacy=0x7f74a0143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:39.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.196+0000 7f74a6000640 1 -- 172.21.15.67:0/39244850 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a0071720 con 0x7f74a01441f0 2024-04-17T10:37:39.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a6000640 1 --2- 172.21.15.67:0/39244850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a01441f0 0x7f74a00ceef0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f7498009e50 tx=0x7f749801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c2c9b8bf8866f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f749fe00640 1 -- 172.21.15.67:0/39244850 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74980a5830 con 0x7f74a01441f0 2024-04-17T10:37:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f749fe00640 1 -- 172.21.15.67:0/39244850 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74980a59d0 con 0x7f74a01441f0 2024-04-17T10:37:39.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f749fe00640 1 -- 172.21.15.67:0/39244850 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74980a5da0 con 0x7f74a01441f0 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 -- 172.21.15.67:0/39244850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a01441f0 msgr2=0x7f74a00ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 --2- 172.21.15.67:0/39244850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a01441f0 0x7f74a00ceef0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f7498009e50 tx=0x7f749801c970 comp rx=0 tx=0).stop 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 -- 172.21.15.67:0/39244850 shutdown_connections 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 --2- 172.21.15.67:0/39244850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a01441f0 0x7f74a00ceef0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 --2- 172.21.15.67:0/39244850 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a006e6e0 0x7f74a00ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 -- 172.21.15.67:0/39244850 >> 172.21.15.67:0/39244850 conn(0x7f74a006ac90 msgr2=0x7f74a006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 -- 172.21.15.67:0/39244850 shutdown_connections 2024-04-17T10:37:39.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.200+0000 7f74a8000640 1 -- 172.21.15.67:0/39244850 wait complete. 2024-04-17T10:37:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 Processor -- start 2024-04-17T10:37:39.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 -- start start 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a006e6e0 0x7f74a0159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74a006eac0 0x7f74a00b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a01441f0 0x7f74a013cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00b3a40 con 0x7f74a006e6e0 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00b38c0 con 0x7f74a006eac0 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a00b3bc0 con 0x7f74a01441f0 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a006e6e0 0x7f74a0159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a006e6e0 0x7f74a0159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59800/0 (socket says 172.21.15.67:59800) 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 -- 172.21.15.67:0/1684459841 learned_addr learned my addr 172.21.15.67:0/1684459841 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:39.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a6000640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a01441f0 0x7f74a013cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a4c00640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74a006eac0 0x7f74a00b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 -- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a01441f0 msgr2=0x7f74a013cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a01441f0 0x7f74a013cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 -- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74a006eac0 msgr2=0x7f74a00b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74a006eac0 0x7f74a00b1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 -- 172.21.15.67:0/1684459841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a00b1e60 con 0x7f74a006e6e0 2024-04-17T10:37:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a5600640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a006e6e0 0x7f74a0159f00 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f749400d800 tx=0x7f749400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f749d600640 1 -- 172.21.15.67:0/1684459841 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749400b6c0 con 0x7f74a006e6e0 2024-04-17T10:37:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f749d600640 1 -- 172.21.15.67:0/1684459841 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7494019070 con 0x7f74a006e6e0 2024-04-17T10:37:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f749d600640 1 -- 172.21.15.67:0/1684459841 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749400ba60 con 0x7f74a006e6e0 2024-04-17T10:37:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a00c5350 con 0x7f74a006e6e0 2024-04-17T10:37:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a00c5910 con 0x7f74a006e6e0 2024-04-17T10:37:39.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.204+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74a0072d50 con 0x7f74a006e6e0 2024-04-17T10:37:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.208+0000 7f749d600640 1 -- 172.21.15.67:0/1684459841 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f749400bc00 con 0x7f74a006e6e0 2024-04-17T10:37:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.208+0000 7f749d600640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7480066f60 0x7f7480069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:39.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.208+0000 7f749d600640 1 -- 172.21.15.67:0/1684459841 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f74940855f0 con 0x7f74a006e6e0 2024-04-17T10:37:39.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.208+0000 7f74a4c00640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7480066f60 0x7f7480069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:39.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.208+0000 7f74a4c00640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7480066f60 0x7f7480069420 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f748c001810 tx=0x7f748c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:39.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.212+0000 7f749d600640 1 -- 172.21.15.67:0/1684459841 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7494055d80 con 0x7f74a006e6e0 2024-04-17T10:37:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74a01f4b30 con 0x7f74a006e6e0 2024-04-17T10:37:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f749d600640 1 -- 172.21.15.67:0/1684459841 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f74a01f4b30 con 0x7f74a006e6e0 2024-04-17T10:37:39.400 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:39.401 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:39.401 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7480066f60 msgr2=0x7f7480069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7480066f60 0x7f7480069420 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f748c001810 tx=0x7f748c009040 comp rx=0 tx=0).stop 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a006e6e0 msgr2=0x7f74a0159f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a006e6e0 0x7f74a0159f00 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f749400d800 tx=0x7f749400dcc0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 shutdown_connections 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7480066f60 0x7f7480069420 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f74a01441f0 0x7f74a013cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f74a006eac0 0x7f74a00b1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 --2- 172.21.15.67:0/1684459841 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f74a006e6e0 0x7f74a0159f00 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 >> 172.21.15.67:0/1684459841 conn(0x7f74a006ac90 msgr2=0x7f74a013e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 shutdown_connections 2024-04-17T10:37:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:39.408+0000 7f74a8000640 1 -- 172.21.15.67:0/1684459841 wait complete. 2024-04-17T10:37:39.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:39.825 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:40.827 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 Processor -- start 2024-04-17T10:37:40.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 -- start start 2024-04-17T10:37:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef80d1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7faef80d33d0 0x7faef80b93d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faef80b5370 0x7faef80b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80b9910 con 0x7faef80b5370 2024-04-17T10:37:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80b5d20 con 0x7faef80d33d0 2024-04-17T10:37:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefd800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef80d2330 con 0x7faef8160f30 2024-04-17T10:37:40.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faef80b5370 0x7faef80b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faef80b5370 0x7faef80b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44438/0 (socket says 172.21.15.67:44438) 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 -- 172.21.15.67:0/3105486689 learned_addr learned my addr 172.21.15.67:0/3105486689 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faef7400640 1 --2- 172.21.15.67:0/3105486689 >> v2:172.21.15.86:3300/0 conn(0x7faef80d33d0 0x7faef80b93d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faef7e00640 1 --2- 172.21.15.67:0/3105486689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef80d1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 -- 172.21.15.67:0/3105486689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 msgr2=0x7faef80d1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 --2- 172.21.15.67:0/3105486689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef80d1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 -- 172.21.15.67:0/3105486689 >> v2:172.21.15.86:3300/0 conn(0x7faef80d33d0 msgr2=0x7faef80b93d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 --2- 172.21.15.67:0/3105486689 >> v2:172.21.15.86:3300/0 conn(0x7faef80d33d0 0x7faef80b93d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faefce00640 1 -- 172.21.15.67:0/3105486689 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80d2610 con 0x7faef80b5370 2024-04-17T10:37:40.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.988+0000 7faef7400640 1 --2- 172.21.15.67:0/3105486689 >> v2:172.21.15.86:3300/0 conn(0x7faef80d33d0 0x7faef80b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:37:40.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefce00640 1 --2- 172.21.15.67:0/3105486689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80b5370 0x7faef80b57e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7faef808c5a0 tx=0x7faef001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0f2892dc28bad8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faef6a00640 1 -- 172.21.15.67:0/3105486689 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef00a5780 con 0x7faef80b5370 2024-04-17T10:37:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faef6a00640 1 -- 172.21.15.67:0/3105486689 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef00a5920 con 0x7faef80b5370 2024-04-17T10:37:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faef6a00640 1 -- 172.21.15.67:0/3105486689 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef00a5cf0 con 0x7faef80b5370 2024-04-17T10:37:40.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 -- 172.21.15.67:0/3105486689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80b5370 msgr2=0x7faef80b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 --2- 172.21.15.67:0/3105486689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80b5370 0x7faef80b57e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7faef808c5a0 tx=0x7faef001c9f0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 -- 172.21.15.67:0/3105486689 shutdown_connections 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 --2- 172.21.15.67:0/3105486689 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80b5370 0x7faef80b57e0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 --2- 172.21.15.67:0/3105486689 >> v2:172.21.15.86:3300/0 conn(0x7faef80d33d0 0x7faef80b93d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 --2- 172.21.15.67:0/3105486689 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef80d1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 -- 172.21.15.67:0/3105486689 >> 172.21.15.67:0/3105486689 conn(0x7faef80d0540 msgr2=0x7faef80d0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 -- 172.21.15.67:0/3105486689 shutdown_connections 2024-04-17T10:37:40.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 -- 172.21.15.67:0/3105486689 wait complete. 2024-04-17T10:37:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 Processor -- start 2024-04-17T10:37:40.987 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.992+0000 7faefd800640 1 -- start start 2024-04-17T10:37:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faef80b5370 0x7faef80c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80d33d0 0x7faef80b8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef815fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80b7420 con 0x7faef80d33d0 2024-04-17T10:37:40.989 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef80b72a0 con 0x7faef80b5370 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef80b75a0 con 0x7faef8160f30 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80d33d0 0x7faef80b8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80d33d0 0x7faef80b8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44450/0 (socket says 172.21.15.67:44450) 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 -- 172.21.15.67:0/2291028559 learned_addr learned my addr 172.21.15.67:0/2291028559 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7e00640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faef80b5370 0x7faef80c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefce00640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef815fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 -- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 msgr2=0x7faef815fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef815fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 -- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faef80b5370 msgr2=0x7faef80c9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faef80b5370 0x7faef80c9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 -- 172.21.15.67:0/2291028559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80c99a0 con 0x7faef80d33d0 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7e00640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faef80b5370 0x7faef80c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefce00640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef815fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faef7400640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80d33d0 0x7faef80b8580 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7faee400e810 tx=0x7faee400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faeefe00640 1 -- 172.21.15.67:0/2291028559 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee400f040 con 0x7faef80d33d0 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faeefe00640 1 -- 172.21.15.67:0/2291028559 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee4004030 con 0x7faef80d33d0 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faeefe00640 1 -- 172.21.15.67:0/2291028559 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4004400 con 0x7faef80d33d0 2024-04-17T10:37:40.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef81409a0 con 0x7faef80d33d0 2024-04-17T10:37:40.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef8140f60 con 0x7faef80d33d0 2024-04-17T10:37:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef81412a0 con 0x7faef80d33d0 2024-04-17T10:37:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:40.996+0000 7faeefe00640 1 -- 172.21.15.67:0/2291028559 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7faee400c6d0 con 0x7faef80d33d0 2024-04-17T10:37:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.000+0000 7faeefe00640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faed8066f60 0x7faed8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.000+0000 7faeefe00640 1 -- 172.21.15.67:0/2291028559 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7faee4084f80 con 0x7faef80d33d0 2024-04-17T10:37:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.000+0000 7faef7e00640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faed8066f60 0x7faed8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:40.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.000+0000 7faef7e00640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faed8066f60 0x7faed8069420 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7faef815de90 tx=0x7faee8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:40.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.004+0000 7faeefe00640 1 -- 172.21.15.67:0/2291028559 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7faef81412a0 con 0x7faef80d33d0 2024-04-17T10:37:41.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.196+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faef8074dc0 con 0x7faef80d33d0 2024-04-17T10:37:41.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.196+0000 7faeefe00640 1 -- 172.21.15.67:0/2291028559 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7faee4017020 con 0x7faef80d33d0 2024-04-17T10:37:41.191 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:41.191 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:41.191 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:41.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faed8066f60 msgr2=0x7faed8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:41.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faed8066f60 0x7faed8069420 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7faef815de90 tx=0x7faee8008040 comp rx=0 tx=0).stop 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80d33d0 msgr2=0x7faef80b8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80d33d0 0x7faef80b8580 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7faee400e810 tx=0x7faee400ecd0 comp rx=0 tx=0).stop 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 shutdown_connections 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7faed8066f60 0x7faed8069420 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7faef8160f30 0x7faef815fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faef80d33d0 0x7faef80b8580 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 --2- 172.21.15.67:0/2291028559 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7faef80b5370 0x7faef80c9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 >> 172.21.15.67:0/2291028559 conn(0x7faef80d0540 msgr2=0x7faef80b3c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 shutdown_connections 2024-04-17T10:37:41.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:41.200+0000 7faefd800640 1 -- 172.21.15.67:0/2291028559 wait complete. 2024-04-17T10:37:41.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:41.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:42.616 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7420e00640 1 Processor -- start 2024-04-17T10:37:42.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7420e00640 1 -- start start 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7420e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c15a740 0x7f741c15e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7420e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f741c0aea50 0x7f741c0b4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7420e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741c15ab20 con 0x7f741c0b4bc0 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7420e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f741c0b41f0 con 0x7f741c0aea50 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7420e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741c0ae850 con 0x7f741c15a740 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7413e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f741c0aea50 0x7f741c0b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f741be00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f741c0b4bc0 0x7f741c06ecd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56882/0 (socket says 172.21.15.67:56882) 2024-04-17T10:37:42.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f741be00640 1 -- 172.21.15.67:0/1605185512 learned_addr learned my addr 172.21.15.67:0/1605185512 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f741b400640 1 --2- 172.21.15.67:0/1605185512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c15a740 0x7f741c15e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7413e00640 1 -- 172.21.15.67:0/1605185512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c15a740 msgr2=0x7f741c15e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7413e00640 1 --2- 172.21.15.67:0/1605185512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c15a740 0x7f741c15e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7413e00640 1 -- 172.21.15.67:0/1605185512 >> v1:172.21.15.67:6789/0 conn(0x7f741c0b4bc0 legacy=0x7f741c06ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:42.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.776+0000 7f7413e00640 1 -- 172.21.15.67:0/1605185512 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c15e980 con 0x7f741c0aea50 2024-04-17T10:37:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7413e00640 1 --2- 172.21.15.67:0/1605185512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0aea50 0x7f741c0b4680 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f74040097c0 tx=0x7f740401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecdc35512715398 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f741aa00640 1 -- 172.21.15.67:0/1605185512 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040b1070 con 0x7f741c0aea50 2024-04-17T10:37:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f741aa00640 1 -- 172.21.15.67:0/1605185512 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74040ac410 con 0x7f741c0aea50 2024-04-17T10:37:42.773 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f741aa00640 1 -- 172.21.15.67:0/1605185512 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040ac7e0 con 0x7f741c0aea50 2024-04-17T10:37:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 -- 172.21.15.67:0/1605185512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0aea50 msgr2=0x7f741c0b4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 --2- 172.21.15.67:0/1605185512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0aea50 0x7f741c0b4680 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f74040097c0 tx=0x7f740401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 -- 172.21.15.67:0/1605185512 shutdown_connections 2024-04-17T10:37:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 --2- 172.21.15.67:0/1605185512 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0aea50 0x7f741c0b4680 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 --2- 172.21.15.67:0/1605185512 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c15a740 0x7f741c15e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 -- 172.21.15.67:0/1605185512 >> 172.21.15.67:0/1605185512 conn(0x7f741c071a80 msgr2=0x7f741c071e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 -- 172.21.15.67:0/1605185512 shutdown_connections 2024-04-17T10:37:42.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.780+0000 7f7420e00640 1 -- 172.21.15.67:0/1605185512 wait complete. 2024-04-17T10:37:42.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 Processor -- start 2024-04-17T10:37:42.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 -- start start 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c0aea50 0x7f741c13e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0b4bc0 0x7f741c0ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f741c15a740 0x7f741c0cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c1679b0 con 0x7f741c0b4bc0 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741c167830 con 0x7f741c15a740 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741c167b30 con 0x7f741c0aea50 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0b4bc0 0x7f741c0ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0b4bc0 0x7f741c0ce7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44470/0 (socket says 172.21.15.67:44470) 2024-04-17T10:37:42.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 -- 172.21.15.67:0/3100810575 learned_addr learned my addr 172.21.15.67:0/3100810575 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f741b400640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c0aea50 0x7f741c13e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f741be00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f741c15a740 0x7f741c0cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 -- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c0aea50 msgr2=0x7f741c13e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c0aea50 0x7f741c13e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 -- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f741c15a740 msgr2=0x7f741c0cece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f741c15a740 0x7f741c0cece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 -- 172.21.15.67:0/3100810575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c140190 con 0x7f741c0b4bc0 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f741be00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f741c15a740 0x7f741c0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:42.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0b4bc0 0x7f741c0ce7a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f740401a0d0 tx=0x7f7404019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413400640 1 -- 172.21.15.67:0/3100810575 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040b1040 con 0x7f741c0b4bc0 2024-04-17T10:37:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413400640 1 -- 172.21.15.67:0/3100810575 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74040bb070 con 0x7f741c0b4bc0 2024-04-17T10:37:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7413400640 1 -- 172.21.15.67:0/3100810575 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74040b5630 con 0x7f741c0b4bc0 2024-04-17T10:37:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f741c140470 con 0x7f741c0b4bc0 2024-04-17T10:37:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.784+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f741c09b860 con 0x7f741c0b4bc0 2024-04-17T10:37:42.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.788+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741c074dc0 con 0x7f741c0b4bc0 2024-04-17T10:37:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.788+0000 7f7413400640 1 -- 172.21.15.67:0/3100810575 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f74040b57d0 con 0x7f741c0b4bc0 2024-04-17T10:37:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.788+0000 7f7413400640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f8066f60 0x7f73f8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.788+0000 7f7413400640 1 -- 172.21.15.67:0/3100810575 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f740411ee30 con 0x7f741c0b4bc0 2024-04-17T10:37:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.788+0000 7f741b400640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f8066f60 0x7f73f8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:42.782 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.788+0000 7f741b400640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f8066f60 0x7f73f8069420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f740c00b9d0 tx=0x7f740c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:42.785 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.792+0000 7f7413400640 1 -- 172.21.15.67:0/3100810575 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7404016090 con 0x7f741c0b4bc0 2024-04-17T10:37:42.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.988+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f741c157870 con 0x7f741c0b4bc0 2024-04-17T10:37:42.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.988+0000 7f7413400640 1 -- 172.21.15.67:0/3100810575 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f74040efaf0 con 0x7f741c0b4bc0 2024-04-17T10:37:42.982 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:42.982 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:42.982 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:42.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f8066f60 msgr2=0x7f73f8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:42.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f8066f60 0x7f73f8069420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f740c00b9d0 tx=0x7f740c008040 comp rx=0 tx=0).stop 2024-04-17T10:37:42.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0b4bc0 msgr2=0x7f741c0ce7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:42.984 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0b4bc0 0x7f741c0ce7a0 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f740401a0d0 tx=0x7f7404019240 comp rx=0 tx=0).stop 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 shutdown_connections 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f73f8066f60 0x7f73f8069420 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f741c15a740 0x7f741c0cece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741c0b4bc0 0x7f741c0ce7a0 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 --2- 172.21.15.67:0/3100810575 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f741c0aea50 0x7f741c13e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 >> 172.21.15.67:0/3100810575 conn(0x7f741c071a80 msgr2=0x7f741c158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 shutdown_connections 2024-04-17T10:37:42.985 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:42.992+0000 7f7420e00640 1 -- 172.21.15.67:0/3100810575 wait complete. 2024-04-17T10:37:43.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:43.407 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:44.408 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:44.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.568+0000 7fb8da200640 1 Processor -- start 2024-04-17T10:37:44.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.568+0000 7fb8da200640 1 -- start start 2024-04-17T10:37:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.568+0000 7fb8da200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415f500 0x7fb8d415f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8da200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb8d415cbd0 0x7fb8d40cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8da200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8d415fe20 con 0x7fb8d40ccc40 2024-04-17T10:37:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8da200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8d4099c60 con 0x7fb8d415cbd0 2024-04-17T10:37:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8da200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4099f40 con 0x7fb8d415f500 2024-04-17T10:37:44.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8cbe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb8d40ccc40 0x7fb8d40cf1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56892/0 (socket says 172.21.15.67:56892) 2024-04-17T10:37:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8cbe00640 1 -- 172.21.15.67:0/2250934599 learned_addr learned my addr 172.21.15.67:0/2250934599 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d9800640 1 --2- 172.21.15.67:0/2250934599 >> v2:172.21.15.86:3300/0 conn(0x7fb8d415cbd0 0x7fb8d40cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d8e00640 1 --2- 172.21.15.67:0/2250934599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415f500 0x7fb8d415f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d3e00640 1 -- 172.21.15.67:0/2250934599 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785290857 0 0) 0x7fb8d415fe20 con 0x7fb8d40ccc40 2024-04-17T10:37:44.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d3e00640 1 -- 172.21.15.67:0/2250934599 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8b8003530 con 0x7fb8d40ccc40 2024-04-17T10:37:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d8e00640 1 -- 172.21.15.67:0/2250934599 >> v2:172.21.15.86:3300/0 conn(0x7fb8d415cbd0 msgr2=0x7fb8d40cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d8e00640 1 --2- 172.21.15.67:0/2250934599 >> v2:172.21.15.86:3300/0 conn(0x7fb8d415cbd0 0x7fb8d40cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d8e00640 1 -- 172.21.15.67:0/2250934599 >> v1:172.21.15.67:6789/0 conn(0x7fb8d40ccc40 legacy=0x7fb8d40cf1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d8e00640 1 -- 172.21.15.67:0/2250934599 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d409a220 con 0x7fb8d415f500 2024-04-17T10:37:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d9800640 1 --2- 172.21.15.67:0/2250934599 >> v2:172.21.15.86:3300/0 conn(0x7fb8d415cbd0 0x7fb8d40cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.572+0000 7fb8d8e00640 1 --2- 172.21.15.67:0/2250934599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415f500 0x7fb8d415f8e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40097c0 tx=0x7fb8c401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38f1edfb42b73ae9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8d3e00640 1 -- 172.21.15.67:0/2250934599 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40b1070 con 0x7fb8d415f500 2024-04-17T10:37:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8d3e00640 1 -- 172.21.15.67:0/2250934599 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c40ac410 con 0x7fb8d415f500 2024-04-17T10:37:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8d3e00640 1 -- 172.21.15.67:0/2250934599 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40ac7e0 con 0x7fb8d415f500 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- 172.21.15.67:0/2250934599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415f500 msgr2=0x7fb8d415f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 --2- 172.21.15.67:0/2250934599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415f500 0x7fb8d415f8e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40097c0 tx=0x7fb8c401c4a0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- 172.21.15.67:0/2250934599 shutdown_connections 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 --2- 172.21.15.67:0/2250934599 >> v2:172.21.15.86:3300/0 conn(0x7fb8d415cbd0 0x7fb8d40cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 --2- 172.21.15.67:0/2250934599 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415f500 0x7fb8d415f8e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- 172.21.15.67:0/2250934599 >> 172.21.15.67:0/2250934599 conn(0x7fb8d40d1620 msgr2=0x7fb8d40d1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- 172.21.15.67:0/2250934599 shutdown_connections 2024-04-17T10:37:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- 172.21.15.67:0/2250934599 wait complete. 2024-04-17T10:37:44.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 Processor -- start 2024-04-17T10:37:44.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- start start 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d40ccc40 0x7fb8d41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415cbd0 0x7fb8d40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8d415f500 0x7fb8d415a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d41689c0 con 0x7fb8d415f500 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4168840 con 0x7fb8d40ccc40 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8da200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8d4168b40 con 0x7fb8d415cbd0 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8d9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8d415f500 0x7fb8d415a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8d9800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8d415f500 0x7fb8d415a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44472/0 (socket says 172.21.15.67:44472) 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8d9800640 1 -- 172.21.15.67:0/775863714 learned_addr learned my addr 172.21.15.67:0/775863714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8d8e00640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d40ccc40 0x7fb8d41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.576+0000 7fb8cbe00640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415cbd0 0x7fb8d40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d9800640 1 -- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415cbd0 msgr2=0x7fb8d40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d9800640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415cbd0 0x7fb8d40ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d9800640 1 -- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d40ccc40 msgr2=0x7fb8d41529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d9800640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d40ccc40 0x7fb8d41529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d9800640 1 -- 172.21.15.67:0/775863714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8d415fcd0 con 0x7fb8d415f500 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d8e00640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d40ccc40 0x7fb8d41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d9800640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8d415f500 0x7fb8d415a460 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00efb0 tx=0x7fb8cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d1600640 1 -- 172.21.15.67:0/775863714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc018020 con 0x7fb8d415f500 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d1600640 1 -- 172.21.15.67:0/775863714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8cc014480 con 0x7fb8d415f500 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8d40d0e00 con 0x7fb8d415f500 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d1600640 1 -- 172.21.15.67:0/775863714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8cc014850 con 0x7fb8d415f500 2024-04-17T10:37:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8d40c51f0 con 0x7fb8d415f500 2024-04-17T10:37:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8d4074dc0 con 0x7fb8d415f500 2024-04-17T10:37:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d1600640 1 -- 172.21.15.67:0/775863714 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb8cc01b020 con 0x7fb8d415f500 2024-04-17T10:37:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d1600640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8b0066f60 0x7fb8b0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:44.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.580+0000 7fb8d1600640 1 -- 172.21.15.67:0/775863714 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fb8cc085ec0 con 0x7fb8d415f500 2024-04-17T10:37:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.584+0000 7fb8d8e00640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8b0066f60 0x7fb8b0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:44.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.584+0000 7fb8d8e00640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8b0066f60 0x7fb8b0069420 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40ae000 tx=0x7fb8c4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.588+0000 7fb8d1600640 1 -- 172.21.15.67:0/775863714 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb8cc022030 con 0x7fb8d415f500 2024-04-17T10:37:44.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8d40d2410 con 0x7fb8d415f500 2024-04-17T10:37:44.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8d1600640 1 -- 172.21.15.67:0/775863714 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fb8cc010040 con 0x7fb8d415f500 2024-04-17T10:37:44.776 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:44.776 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:44.776 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8b0066f60 msgr2=0x7fb8b0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8b0066f60 0x7fb8b0069420 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40ae000 tx=0x7fb8c4008040 comp rx=0 tx=0).stop 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8d415f500 msgr2=0x7fb8d415a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8d415f500 0x7fb8d415a460 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc00efb0 tx=0x7fb8cc00c490 comp rx=0 tx=0).stop 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 shutdown_connections 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb8b0066f60 0x7fb8b0069420 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8d415f500 0x7fb8d415a460 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb8d415cbd0 0x7fb8d40ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 --2- 172.21.15.67:0/775863714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb8d40ccc40 0x7fb8d41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 >> 172.21.15.67:0/775863714 conn(0x7fb8d40d1620 msgr2=0x7fb8d415d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 shutdown_connections 2024-04-17T10:37:44.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:44.784+0000 7fb8da200640 1 -- 172.21.15.67:0/775863714 wait complete. 2024-04-17T10:37:45.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:45.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:46.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:46.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.360+0000 7f8b9a000640 1 Processor -- start 2024-04-17T10:37:46.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.360+0000 7f8b9a000640 1 -- start start 2024-04-17T10:37:46.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b9a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 0x7f8b940ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b9a000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b9415ae40 con 0x7f8b9406eac0 2024-04-17T10:37:46.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b9a000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b94071440 con 0x7f8b941441f0 2024-04-17T10:37:46.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b9a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b94071720 con 0x7f8b9406e6e0 2024-04-17T10:37:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b8aa00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8b9406eac0 0x7f8b94143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56902/0 (socket says 172.21.15.67:56902) 2024-04-17T10:37:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b8aa00640 1 -- 172.21.15.67:0/815665788 learned_addr learned my addr 172.21.15.67:0/815665788 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719155776 0 0) 0x7f8b9415ae40 con 0x7f8b9406eac0 2024-04-17T10:37:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93400640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 0x7f8b940ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b78003530 con 0x7f8b9406eac0 2024-04-17T10:37:46.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3941975593 0 0) 0x7f8b78003530 con 0x7f8b9406eac0 2024-04-17T10:37:46.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 msgr2=0x7f8b940ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:46.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 0x7f8b940ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 >> v1:172.21.15.86:6789/0 conn(0x7f8b941441f0 legacy=0x7f8b940b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:46.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b94071a00 con 0x7f8b9406eac0 2024-04-17T10:37:46.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f8b7c001680 con 0x7f8b9406eac0 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 >> v1:172.21.15.67:6789/0 conn(0x7f8b9406eac0 legacy=0x7f8b94143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b780039f0 0x7f8b78003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 0x7f8b78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78004d50 0x7f8b780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b94071720 con 0x7f8b78004d50 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7800bb60 con 0x7f8b780039f0 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7800bd60 con 0x7f8b9406e6e0 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93e00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78004d50 0x7f8b780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93400640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b780039f0 0x7f8b78003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b8aa00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 0x7f8b78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93e00640 1 -- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 msgr2=0x7f8b78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93e00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b9406e6e0 0x7f8b78004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93e00640 1 -- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b780039f0 msgr2=0x7f8b78003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:46.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93e00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b780039f0 0x7f8b78003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.364+0000 7f8b93e00640 1 -- 172.21.15.67:0/815665788 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78003530 con 0x7f8b78004d50 2024-04-17T10:37:46.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b93e00640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78004d50 0x7f8b780051d0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c001780 tx=0x7f8b8c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c562438e5715ad9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:46.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0013a0 con 0x7f8b78004d50 2024-04-17T10:37:46.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c0a9070 con 0x7f8b78004d50 2024-04-17T10:37:46.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b92a00640 1 -- 172.21.15.67:0/815665788 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c003600 con 0x7f8b78004d50 2024-04-17T10:37:46.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 -- 172.21.15.67:0/815665788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78004d50 msgr2=0x7f8b780051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:46.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 --2- 172.21.15.67:0/815665788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b78004d50 0x7f8b780051d0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c001780 tx=0x7f8b8c01c420 comp rx=0 tx=0).stop 2024-04-17T10:37:46.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b93e00640 1 -- 172.21.15.67:0/815665788 reap_dead start 2024-04-17T10:37:46.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 -- 172.21.15.67:0/815665788 shutdown_connections 2024-04-17T10:37:46.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 -- 172.21.15.67:0/815665788 >> 172.21.15.67:0/815665788 conn(0x7f8b9406ac90 msgr2=0x7f8b9406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:46.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 -- 172.21.15.67:0/815665788 shutdown_connections 2024-04-17T10:37:46.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 -- 172.21.15.67:0/815665788 wait complete. 2024-04-17T10:37:46.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 Processor -- start 2024-04-17T10:37:46.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.368+0000 7f8b9a000640 1 -- start start 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b78004d50 0x7f8b940ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b94136580 0x7f8b940a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b940b0450 0x7f8b94169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b940b37c0 con 0x7f8b940b0450 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b94071fc0 con 0x7f8b78004d50 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b940b3940 con 0x7f8b94136580 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b940b0450 0x7f8b94169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b940b0450 0x7f8b94169990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44496/0 (socket says 172.21.15.67:44496) 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 -- 172.21.15.67:0/2172338062 learned_addr learned my addr 172.21.15.67:0/2172338062 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93400640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b78004d50 0x7f8b940ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 -- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b94136580 msgr2=0x7f8b940a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b8aa00640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b94136580 0x7f8b940a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b94136580 0x7f8b940a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 -- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b78004d50 msgr2=0x7f8b940ab2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b78004d50 0x7f8b940ab2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 -- 172.21.15.67:0/2172338062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b940a9b40 con 0x7f8b940b0450 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93400640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b78004d50 0x7f8b940ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b8aa00640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b94136580 0x7f8b940a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:46.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b93e00640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b940b0450 0x7f8b94169990 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c005040 tx=0x7f8b78005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b8be00640 1 -- 172.21.15.67:0/2172338062 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0013a0 con 0x7f8b940b0450 2024-04-17T10:37:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b8be00640 1 -- 172.21.15.67:0/2172338062 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c007040 con 0x7f8b940b0450 2024-04-17T10:37:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b8be00640 1 -- 172.21.15.67:0/2172338062 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c007a40 con 0x7f8b940b0450 2024-04-17T10:37:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b940a9e20 con 0x7f8b940b0450 2024-04-17T10:37:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b94169190 con 0x7f8b940b0450 2024-04-17T10:37:46.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b94072d50 con 0x7f8b940b0450 2024-04-17T10:37:46.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.372+0000 7f8b8be00640 1 -- 172.21.15.67:0/2172338062 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8b8c0be020 con 0x7f8b940b0450 2024-04-17T10:37:46.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.376+0000 7f8b8be00640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b70066f60 0x7f8b70069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:46.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.376+0000 7f8b8be00640 1 -- 172.21.15.67:0/2172338062 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8b8c11e510 con 0x7f8b940b0450 2024-04-17T10:37:46.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.376+0000 7f8b93400640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b70066f60 0x7f8b70069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:46.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.376+0000 7f8b93400640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b70066f60 0x7f8b70069420 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8b940b0e60 tx=0x7f8b8400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:46.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.380+0000 7f8b8be00640 1 -- 172.21.15.67:0/2172338062 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8b8c016130 con 0x7f8b940b0450 2024-04-17T10:37:46.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b94074e30 con 0x7f8b940b0450 2024-04-17T10:37:46.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b8be00640 1 -- 172.21.15.67:0/2172338062 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f8b8c0ef260 con 0x7f8b940b0450 2024-04-17T10:37:46.572 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:46.572 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:46.572 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:46.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b70066f60 msgr2=0x7f8b70069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:46.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b70066f60 0x7f8b70069420 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f8b940b0e60 tx=0x7f8b8400b040 comp rx=0 tx=0).stop 2024-04-17T10:37:46.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b940b0450 msgr2=0x7f8b94169990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:46.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b940b0450 0x7f8b94169990 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c005040 tx=0x7f8b78005840 comp rx=0 tx=0).stop 2024-04-17T10:37:46.574 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 shutdown_connections 2024-04-17T10:37:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8b70066f60 0x7f8b70069420 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b940b0450 0x7f8b94169990 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8b94136580 0x7f8b940a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 --2- 172.21.15.67:0/2172338062 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8b78004d50 0x7f8b940ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 >> 172.21.15.67:0/2172338062 conn(0x7f8b9406ac90 msgr2=0x7f8b94139730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 shutdown_connections 2024-04-17T10:37:46.575 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:46.580+0000 7f8b9a000640 1 -- 172.21.15.67:0/2172338062 wait complete. 2024-04-17T10:37:46.998 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:46.998 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:47.999 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 Processor -- start 2024-04-17T10:37:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 -- start start 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fba2815dbc0 0x7fba2806f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2806db90 0x7fba28071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba280721a0 0x7fba2806b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2815dfa0 con 0x7fba280721a0 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2806faa0 con 0x7fba2815dbc0 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba28072600 con 0x7fba2806db90 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba280721a0 0x7fba2806b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:48.155 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2de00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fba280721a0 0x7fba2806b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44504/0 (socket says 172.21.15.67:44504) 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.160+0000 7fba2de00640 1 -- 172.21.15.67:0/406616985 learned_addr learned my addr 172.21.15.67:0/406616985 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2d400640 1 --2- 172.21.15.67:0/406616985 >> v2:172.21.15.86:3300/0 conn(0x7fba2815dbc0 0x7fba2806f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2ca00640 1 --2- 172.21.15.67:0/406616985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2806db90 0x7fba28071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2de00640 1 -- 172.21.15.67:0/406616985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2806db90 msgr2=0x7fba28071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2de00640 1 --2- 172.21.15.67:0/406616985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2806db90 0x7fba28071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2de00640 1 -- 172.21.15.67:0/406616985 >> v2:172.21.15.86:3300/0 conn(0x7fba2815dbc0 msgr2=0x7fba2806f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2de00640 1 --2- 172.21.15.67:0/406616985 >> v2:172.21.15.86:3300/0 conn(0x7fba2815dbc0 0x7fba2806f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2de00640 1 -- 172.21.15.67:0/406616985 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280ad8f0 con 0x7fba280721a0 2024-04-17T10:37:48.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba2de00640 1 --2- 172.21.15.67:0/406616985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2806b400 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fba280d0f20 tx=0x7fba2001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416727e1313af3b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:48.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba27e00640 1 -- 172.21.15.67:0/406616985 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200a55e0 con 0x7fba280721a0 2024-04-17T10:37:48.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba27e00640 1 -- 172.21.15.67:0/406616985 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba200a5780 con 0x7fba280721a0 2024-04-17T10:37:48.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.164+0000 7fba27e00640 1 -- 172.21.15.67:0/406616985 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200a5b50 con 0x7fba280721a0 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- 172.21.15.67:0/406616985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 msgr2=0x7fba2806b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 --2- 172.21.15.67:0/406616985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2806b400 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fba280d0f20 tx=0x7fba2001c960 comp rx=0 tx=0).stop 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- 172.21.15.67:0/406616985 shutdown_connections 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 --2- 172.21.15.67:0/406616985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2806b400 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 --2- 172.21.15.67:0/406616985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2806db90 0x7fba28071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 --2- 172.21.15.67:0/406616985 >> v2:172.21.15.86:3300/0 conn(0x7fba2815dbc0 0x7fba2806f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- 172.21.15.67:0/406616985 >> 172.21.15.67:0/406616985 conn(0x7fba280af670 msgr2=0x7fba280b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:48.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- 172.21.15.67:0/406616985 shutdown_connections 2024-04-17T10:37:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- 172.21.15.67:0/406616985 wait complete. 2024-04-17T10:37:48.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 Processor -- start 2024-04-17T10:37:48.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- start start 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba2806db90 0x7fba281411e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2815cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2815dbc0 0x7fba2813fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2806c2d0 con 0x7fba280721a0 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2806c150 con 0x7fba2806db90 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2fe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2806c450 con 0x7fba2815dbc0 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2815cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:48.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2815cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44512/0 (socket says 172.21.15.67:44512) 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 -- 172.21.15.67:0/440862655 learned_addr learned my addr 172.21.15.67:0/440862655 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2d400640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba2806db90 0x7fba281411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2de00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2815dbc0 0x7fba2813fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 -- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2815dbc0 msgr2=0x7fba2813fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2815dbc0 0x7fba2813fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 -- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba2806db90 msgr2=0x7fba281411e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba2806db90 0x7fba281411e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 -- 172.21.15.67:0/440862655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba28141720 con 0x7fba280721a0 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2d400640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba2806db90 0x7fba281411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.168+0000 7fba2ca00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2815cd60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fba1400e8b0 tx=0x7fba1400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba25600640 1 -- 172.21.15.67:0/440862655 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14017070 con 0x7fba280721a0 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba25600640 1 -- 172.21.15.67:0/440862655 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1400c940 con 0x7fba280721a0 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba25600640 1 -- 172.21.15.67:0/440862655 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1400cd10 con 0x7fba280721a0 2024-04-17T10:37:48.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2813ecc0 con 0x7fba280721a0 2024-04-17T10:37:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2813f280 con 0x7fba280721a0 2024-04-17T10:37:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2809e260 con 0x7fba280721a0 2024-04-17T10:37:48.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba25600640 1 -- 172.21.15.67:0/440862655 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fba1401a070 con 0x7fba280721a0 2024-04-17T10:37:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba25600640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba08066f90 0x7fba08069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba25600640 1 -- 172.21.15.67:0/440862655 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fba14085270 con 0x7fba280721a0 2024-04-17T10:37:48.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.172+0000 7fba2d400640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba08066f90 0x7fba08069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:48.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.176+0000 7fba2d400640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba08066f90 0x7fba08069450 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fba280b1300 tx=0x7fba1c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:48.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.176+0000 7fba25600640 1 -- 172.21.15.67:0/440862655 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fba14055fc0 con 0x7fba280721a0 2024-04-17T10:37:48.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.376+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba280b0ec0 con 0x7fba280721a0 2024-04-17T10:37:48.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.376+0000 7fba25600640 1 -- 172.21.15.67:0/440862655 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fba140558c0 con 0x7fba280721a0 2024-04-17T10:37:48.369 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:48.370 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:48.370 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:48.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba08066f90 msgr2=0x7fba08069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:48.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba08066f90 0x7fba08069450 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fba280b1300 tx=0x7fba1c008040 comp rx=0 tx=0).stop 2024-04-17T10:37:48.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 msgr2=0x7fba2815cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:48.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2815cd60 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fba1400e8b0 tx=0x7fba1400ed70 comp rx=0 tx=0).stop 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 shutdown_connections 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fba08066f90 0x7fba08069450 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fba2815dbc0 0x7fba2813fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fba280721a0 0x7fba2815cd60 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 --2- 172.21.15.67:0/440862655 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fba2806db90 0x7fba281411e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 >> 172.21.15.67:0/440862655 conn(0x7fba280af670 msgr2=0x7fba280b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 shutdown_connections 2024-04-17T10:37:48.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:48.380+0000 7fba2fe00640 1 -- 172.21.15.67:0/440862655 wait complete. 2024-04-17T10:37:48.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:48.794 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:49.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:49.948 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 Processor -- start 2024-04-17T10:37:49.949 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 -- start start 2024-04-17T10:37:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3e3c06e6e0 0x7f3e3c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:49.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e3c143ae0 0x7f3e3c143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c0cf430 con 0x7f3e3c143ae0 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c15ae40 con 0x7f3e3c06e6e0 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e42800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c071440 con 0x7f3e3c06eac0 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e3c143ae0 0x7f3e3c143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e3c143ae0 0x7f3e3c143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44518/0 (socket says 172.21.15.67:44518) 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 -- 172.21.15.67:0/3206271118 learned_addr learned my addr 172.21.15.67:0/3206271118 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e33e00640 1 --2- 172.21.15.67:0/3206271118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3b400640 1 --2- 172.21.15.67:0/3206271118 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c06e6e0 0x7f3e3c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 -- 172.21.15.67:0/3206271118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 msgr2=0x7f3e3c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 --2- 172.21.15.67:0/3206271118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:49.951 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 -- 172.21.15.67:0/3206271118 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c06e6e0 msgr2=0x7f3e3c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:49.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 --2- 172.21.15.67:0/3206271118 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c06e6e0 0x7f3e3c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:49.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.956+0000 7f3e3be00640 1 -- 172.21.15.67:0/3206271118 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c071720 con 0x7f3e3c143ae0 2024-04-17T10:37:49.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e3be00640 1 --2- 172.21.15.67:0/3206271118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c143f70 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0cf620 tx=0x7f3e3401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418728129a9456a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e3aa00640 1 -- 172.21.15.67:0/3206271118 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e340a57b0 con 0x7f3e3c143ae0 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e3aa00640 1 -- 172.21.15.67:0/3206271118 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e340a5950 con 0x7f3e3c143ae0 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e3aa00640 1 -- 172.21.15.67:0/3206271118 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e340a5d20 con 0x7f3e3c143ae0 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 -- 172.21.15.67:0/3206271118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 msgr2=0x7f3e3c143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 --2- 172.21.15.67:0/3206271118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c143f70 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0cf620 tx=0x7f3e3401c930 comp rx=0 tx=0).stop 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 -- 172.21.15.67:0/3206271118 shutdown_connections 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 --2- 172.21.15.67:0/3206271118 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c143f70 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 --2- 172.21.15.67:0/3206271118 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 --2- 172.21.15.67:0/3206271118 >> v2:172.21.15.86:3300/0 conn(0x7f3e3c06e6e0 0x7f3e3c0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 -- 172.21.15.67:0/3206271118 >> 172.21.15.67:0/3206271118 conn(0x7f3e3c06ac90 msgr2=0x7f3e3c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 -- 172.21.15.67:0/3206271118 shutdown_connections 2024-04-17T10:37:49.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.960+0000 7f3e42800640 1 -- 172.21.15.67:0/3206271118 wait complete. 2024-04-17T10:37:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 Processor -- start 2024-04-17T10:37:49.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 -- start start 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c06e6e0 0x7f3e3c13c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c15ebf0 con 0x7f3e3c143ae0 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1444b0 con 0x7f3e3c06e6e0 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e42800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c15ed70 con 0x7f3e3c06eac0 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c141730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44522/0 (socket says 172.21.15.67:44522) 2024-04-17T10:37:49.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 -- 172.21.15.67:0/1872137905 learned_addr learned my addr 172.21.15.67:0/1872137905 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3b400640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c06e6e0 0x7f3e3c13c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e33e00640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 -- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 msgr2=0x7f3e3c0b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 -- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c06e6e0 msgr2=0x7f3e3c13c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c06e6e0 0x7f3e3c13c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 -- 172.21.15.67:0/1872137905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c0b0ab0 con 0x7f3e3c143ae0 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e3be00640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c141730 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f3e340af040 tx=0x7f3e340196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e33400640 1 -- 172.21.15.67:0/1872137905 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e340b8670 con 0x7f3e3c143ae0 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.964+0000 7f3e33400640 1 -- 172.21.15.67:0/1872137905 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e340be070 con 0x7f3e3c143ae0 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c0af4d0 con 0x7f3e3c143ae0 2024-04-17T10:37:49.960 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e33400640 1 -- 172.21.15.67:0/1872137905 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e340b8a70 con 0x7f3e3c143ae0 2024-04-17T10:37:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c0afa90 con 0x7f3e3c143ae0 2024-04-17T10:37:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c072d50 con 0x7f3e3c143ae0 2024-04-17T10:37:49.961 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e33400640 1 -- 172.21.15.67:0/1872137905 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3e340b8c10 con 0x7f3e3c143ae0 2024-04-17T10:37:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e33400640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e18066f60 0x7f3e18069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e33400640 1 -- 172.21.15.67:0/1872137905 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f3e341203b0 con 0x7f3e3c143ae0 2024-04-17T10:37:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e3b400640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e18066f60 0x7f3e18069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:49.962 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.968+0000 7f3e3b400640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e18066f60 0x7f3e18069420 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c007a60 tx=0x7f3e2c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:49.966 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:49.972+0000 7f3e33400640 1 -- 172.21.15.67:0/1872137905 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3e340165e0 con 0x7f3e3c143ae0 2024-04-17T10:37:50.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.172+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e3c074e30 con 0x7f3e3c143ae0 2024-04-17T10:37:50.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.172+0000 7f3e33400640 1 -- 172.21.15.67:0/1872137905 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3e340f0a20 con 0x7f3e3c143ae0 2024-04-17T10:37:50.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:50.167 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:50.167 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e18066f60 msgr2=0x7f3e18069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e18066f60 0x7f3e18069420 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c007a60 tx=0x7f3e2c008040 comp rx=0 tx=0).stop 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 msgr2=0x7f3e3c141730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c141730 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f3e340af040 tx=0x7f3e340196d0 comp rx=0 tx=0).stop 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 shutdown_connections 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3e18066f60 0x7f3e18069420 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e3c143ae0 0x7f3e3c141730 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3e3c06eac0 0x7f3e3c0b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 --2- 172.21.15.67:0/1872137905 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3e3c06e6e0 0x7f3e3c13c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 >> 172.21.15.67:0/1872137905 conn(0x7f3e3c06ac90 msgr2=0x7f3e3c0ce800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 shutdown_connections 2024-04-17T10:37:50.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:50.176+0000 7f3e42800640 1 -- 172.21.15.67:0/1872137905 wait complete. 2024-04-17T10:37:50.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:50.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:51.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:51.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.748+0000 7f7f2d800640 1 Processor -- start 2024-04-17T10:37:51.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.748+0000 7f7f2d800640 1 -- start start 2024-04-17T10:37:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f2d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 0x7f7f280cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.743 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f2d800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2815eb50 con 0x7f7f281620a0 2024-04-17T10:37:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f2d800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f2815ee30 con 0x7f7f280ce020 2024-04-17T10:37:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f2d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28161650 con 0x7f7f280cf510 2024-04-17T10:37:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f1e000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7f281620a0 0x7f7f2815e010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46360/0 (socket says 172.21.15.67:46360) 2024-04-17T10:37:51.744 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f1e000640 1 -- 172.21.15.67:0/472661039 learned_addr learned my addr 172.21.15.67:0/472661039 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897375159 0 0) 0x7f7f2815eb50 con 0x7f7f281620a0 2024-04-17T10:37:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f27400640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 0x7f7f280cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.745 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f0c003530 con 0x7f7f281620a0 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3544824501 0 0) 0x7f7f0c003530 con 0x7f7f281620a0 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 msgr2=0x7f7f280cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 0x7f7f280cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 >> v1:172.21.15.86:6789/0 conn(0x7f7f280ce020 legacy=0x7f7f280ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f281618a0 con 0x7f7f281620a0 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f7f10001680 con 0x7f7f281620a0 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 >> v1:172.21.15.67:6789/0 conn(0x7f7f281620a0 legacy=0x7f7f2815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0c0039f0 0x7f7f0c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 0x7f7f0c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c004d50 0x7f7f0c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f28161650 con 0x7f7f0c0039f0 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c00bab0 con 0x7f7f0c004d50 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c00bcb0 con 0x7f7f280cf510 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26a00640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0c0039f0 0x7f7f0c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f1e000640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 0x7f7f0c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f27400640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c004d50 0x7f7f0c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26a00640 1 -- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 msgr2=0x7f7f0c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26a00640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f280cf510 0x7f7f0c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26a00640 1 -- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c004d50 msgr2=0x7f7f0c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26a00640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c004d50 0x7f7f0c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.752+0000 7f7f26a00640 1 -- 172.21.15.67:0/472661039 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c003530 con 0x7f7f0c0039f0 2024-04-17T10:37:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f26a00640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0c0039f0 0x7f7f0c003e10 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f7f18001750 tx=0x7f7f1801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac16399f1e1e8a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f180013a0 con 0x7f7f0c0039f0 2024-04-17T10:37:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f180052f0 con 0x7f7f0c0039f0 2024-04-17T10:37:51.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f26000640 1 -- 172.21.15.67:0/472661039 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18003840 con 0x7f7f0c0039f0 2024-04-17T10:37:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f2d800640 1 -- 172.21.15.67:0/472661039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0c0039f0 msgr2=0x7f7f0c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f2d800640 1 --2- 172.21.15.67:0/472661039 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f0c0039f0 0x7f7f0c003e10 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f7f18001750 tx=0x7f7f1801c3b0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f27400640 1 -- 172.21.15.67:0/472661039 reap_dead start 2024-04-17T10:37:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f2d800640 1 -- 172.21.15.67:0/472661039 shutdown_connections 2024-04-17T10:37:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f2d800640 1 -- 172.21.15.67:0/472661039 >> 172.21.15.67:0/472661039 conn(0x7f7f280b33f0 msgr2=0x7f7f280b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:51.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f2d800640 1 -- 172.21.15.67:0/472661039 shutdown_connections 2024-04-17T10:37:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f2d800640 1 -- 172.21.15.67:0/472661039 wait complete. 2024-04-17T10:37:51.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.756+0000 7f7f2d800640 1 Processor -- start 2024-04-17T10:37:51.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 -- start start 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c0039f0 0x7f7f28151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f28155210 0x7f7f28155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f2815bf20 0x7f7f2815c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2815e8a0 con 0x7f7f2815bf20 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2815e720 con 0x7f7f0c0039f0 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280c84b0 con 0x7f7f28155210 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f2815bf20 0x7f7f2815c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f26a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c0039f0 0x7f7f28151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f2815bf20 0x7f7f2815c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50402/0 (socket says 172.21.15.67:50402) 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 -- 172.21.15.67:0/1803381565 learned_addr learned my addr 172.21.15.67:0/1803381565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f1e000640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f28155210 0x7f7f28155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 -- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f28155210 msgr2=0x7f7f28155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f28155210 0x7f7f28155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 -- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c0039f0 msgr2=0x7f7f28151240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c0039f0 0x7f7f28151240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 -- 172.21.15.67:0/1803381565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2815a5c0 con 0x7f7f2815bf20 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f26a00640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c0039f0 0x7f7f28151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f27400640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f2815bf20 0x7f7f2815c3a0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f7f280b79b0 tx=0x7f7f2000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f1f400640 1 -- 172.21.15.67:0/1803381565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2000fe10 con 0x7f7f2815bf20 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f1f400640 1 -- 172.21.15.67:0/1803381565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2001d070 con 0x7f7f2815bf20 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f1f400640 1 -- 172.21.15.67:0/1803381565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f20015670 con 0x7f7f2815bf20 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f2815d4f0 con 0x7f7f2815bf20 2024-04-17T10:37:51.754 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f2815cf30 con 0x7f7f2815bf20 2024-04-17T10:37:51.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.760+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28070ab0 con 0x7f7f2815bf20 2024-04-17T10:37:51.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.764+0000 7f7f1f400640 1 -- 172.21.15.67:0/1803381565 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f7f20015810 con 0x7f7f2815bf20 2024-04-17T10:37:51.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.764+0000 7f7f1f400640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f04066f90 0x7f7f04069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:51.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.764+0000 7f7f1f400640 1 -- 172.21.15.67:0/1803381565 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f7f200859c0 con 0x7f7f2815bf20 2024-04-17T10:37:51.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.764+0000 7f7f26a00640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f04066f90 0x7f7f04069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:51.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.764+0000 7f7f26a00640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f04066f90 0x7f7f04069450 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0057f0 tx=0x7f7f180011a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:51.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.768+0000 7f7f1f400640 1 -- 172.21.15.67:0/1803381565 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7f20010040 con 0x7f7f2815bf20 2024-04-17T10:37:51.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f28074e30 con 0x7f7f2815bf20 2024-04-17T10:37:51.956 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f1f400640 1 -- 172.21.15.67:0/1803381565 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f7f200560c0 con 0x7f7f2815bf20 2024-04-17T10:37:51.956 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:51.956 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:51.956 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f04066f90 msgr2=0x7f7f04069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f04066f90 0x7f7f04069450 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0057f0 tx=0x7f7f180011a0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f2815bf20 msgr2=0x7f7f2815c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f2815bf20 0x7f7f2815c3a0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f7f280b79b0 tx=0x7f7f2000f580 comp rx=0 tx=0).stop 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 shutdown_connections 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f7f04066f90 0x7f7f04069450 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7f2815bf20 0x7f7f2815c3a0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7f28155210 0x7f7f28155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 --2- 172.21.15.67:0/1803381565 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f7f0c0039f0 0x7f7f28151240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 >> 172.21.15.67:0/1803381565 conn(0x7f7f280b33f0 msgr2=0x7f7f28159eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 shutdown_connections 2024-04-17T10:37:51.959 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:51.964+0000 7f7f2d800640 1 -- 172.21.15.67:0/1803381565 wait complete. 2024-04-17T10:37:52.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:52.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:53.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:53.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.540+0000 7effa0400640 1 Processor -- start 2024-04-17T10:37:53.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.540+0000 7effa0400640 1 -- start start 2024-04-17T10:37:53.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7effa0400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff9815fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7effa0400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9806aea0 con 0x7eff9806f560 2024-04-17T10:37:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7effa0400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9806b180 con 0x7eff98071c60 2024-04-17T10:37:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7effa0400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9806b7d0 con 0x7eff9815b870 2024-04-17T10:37:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff9d000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7eff9806f560 0x7eff9815bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46364/0 (socket says 172.21.15.67:46364) 2024-04-17T10:37:53.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff9d000640 1 -- 172.21.15.67:0/1087596774 learned_addr learned my addr 172.21.15.67:0/1087596774 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff9da00640 1 --2- 172.21.15.67:0/1087596774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff9815fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474296655 0 0) 0x7eff9806aea0 con 0x7eff9806f560 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff80003530 con 0x7eff9806f560 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207528547 0 0) 0x7eff9806b180 con 0x7eff98071c60 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff9806aea0 con 0x7eff98071c60 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff9da00640 1 -- 172.21.15.67:0/1087596774 >> v1:172.21.15.86:6789/0 conn(0x7eff98071c60 legacy=0x7eff9806bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff9da00640 1 -- 172.21.15.67:0/1087596774 >> v1:172.21.15.67:6789/0 conn(0x7eff9806f560 legacy=0x7eff9815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:53.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3181959243 0 0) 0x7eff80003530 con 0x7eff9806f560 2024-04-17T10:37:53.538 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.544+0000 7eff9da00640 1 -- 172.21.15.67:0/1087596774 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9806b460 con 0x7eff9815b870 2024-04-17T10:37:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7eff9da00640 1 --2- 172.21.15.67:0/1087596774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff9815fe10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097f0 tx=0x7eff8c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70ba62a3765be9a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0b0070 con 0x7eff9815b870 2024-04-17T10:37:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c0ab400 con 0x7eff9815b870 2024-04-17T10:37:53.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7eff97e00640 1 -- 172.21.15.67:0/1087596774 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84001680 con 0x7eff9815b870 2024-04-17T10:37:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7effa0400640 1 -- 172.21.15.67:0/1087596774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 msgr2=0x7eff9815fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7effa0400640 1 --2- 172.21.15.67:0/1087596774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff9815fe10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097f0 tx=0x7eff8c01c400 comp rx=0 tx=0).stop 2024-04-17T10:37:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7effa0400640 1 -- 172.21.15.67:0/1087596774 shutdown_connections 2024-04-17T10:37:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7effa0400640 1 --2- 172.21.15.67:0/1087596774 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff9815fe10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7effa0400640 1 -- 172.21.15.67:0/1087596774 >> 172.21.15.67:0/1087596774 conn(0x7eff980af670 msgr2=0x7eff980b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7effa0400640 1 -- 172.21.15.67:0/1087596774 shutdown_connections 2024-04-17T10:37:53.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.548+0000 7effa0400640 1 -- 172.21.15.67:0/1087596774 wait complete. 2024-04-17T10:37:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 Processor -- start 2024-04-17T10:37:53.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 -- start start 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9806f560 0x7eff98157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff98071c60 0x7eff98143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff980acee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9816a140 con 0x7eff9806f560 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff98169fc0 con 0x7eff98071c60 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9816a2c0 con 0x7eff9815b870 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9806f560 0x7eff98157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9806f560 0x7eff98157550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50412/0 (socket says 172.21.15.67:50412) 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 -- 172.21.15.67:0/2370962923 learned_addr learned my addr 172.21.15.67:0/2370962923 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9d000640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff98071c60 0x7eff98143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:53.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9e400640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff980acee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 -- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 msgr2=0x7eff980acee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff980acee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 -- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff98071c60 msgr2=0x7eff98143840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff98071c60 0x7eff98143840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 -- 172.21.15.67:0/2370962923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98157cc0 con 0x7eff9806f560 2024-04-17T10:37:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff9da00640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9806f560 0x7eff98157550 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097f0 tx=0x7eff8c01ac40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff95600640 1 -- 172.21.15.67:0/2370962923 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0bb020 con 0x7eff9806f560 2024-04-17T10:37:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff95600640 1 -- 172.21.15.67:0/2370962923 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c0b1040 con 0x7eff9806f560 2024-04-17T10:37:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7eff95600640 1 -- 172.21.15.67:0/2370962923 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0b0050 con 0x7eff9806f560 2024-04-17T10:37:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff98157fa0 con 0x7eff9806f560 2024-04-17T10:37:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff98143d80 con 0x7eff9806f560 2024-04-17T10:37:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.552+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff980adf50 con 0x7eff9806f560 2024-04-17T10:37:53.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.556+0000 7eff95600640 1 -- 172.21.15.67:0/2370962923 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7eff8c0be070 con 0x7eff9806f560 2024-04-17T10:37:53.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.556+0000 7eff95600640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff78066f60 0x7eff78069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:53.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.556+0000 7eff95600640 1 -- 172.21.15.67:0/2370962923 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7eff8c11e800 con 0x7eff9806f560 2024-04-17T10:37:53.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.556+0000 7eff9d000640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff78066f60 0x7eff78069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:53.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.556+0000 7eff9d000640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff78066f60 0x7eff78069420 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7eff9815c5e0 tx=0x7eff84001160 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:53.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.560+0000 7eff95600640 1 -- 172.21.15.67:0/2370962923 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7eff8c015ff0 con 0x7eff9806f560 2024-04-17T10:37:53.747 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff98158170 con 0x7eff9806f560 2024-04-17T10:37:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7eff95600640 1 -- 172.21.15.67:0/2370962923 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7eff8c0eef00 con 0x7eff9806f560 2024-04-17T10:37:53.748 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:53.748 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:53.748 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff78066f60 msgr2=0x7eff78069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff78066f60 0x7eff78069420 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7eff9815c5e0 tx=0x7eff84001160 comp rx=0 tx=0).stop 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9806f560 msgr2=0x7eff98157550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9806f560 0x7eff98157550 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7eff8c0097f0 tx=0x7eff8c01ac40 comp rx=0 tx=0).stop 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 shutdown_connections 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7eff78066f60 0x7eff78069420 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7eff9815b870 0x7eff980acee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7eff98071c60 0x7eff98143840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:53.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 --2- 172.21.15.67:0/2370962923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff9806f560 0x7eff98157550 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 >> 172.21.15.67:0/2370962923 conn(0x7eff980af670 msgr2=0x7eff9815dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 shutdown_connections 2024-04-17T10:37:53.751 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:53.756+0000 7effa0400640 1 -- 172.21.15.67:0/2370962923 wait complete. 2024-04-17T10:37:54.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:54.172 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:55.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:55.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd055800640 1 Processor -- start 2024-04-17T10:37:55.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd055800640 1 -- start start 2024-04-17T10:37:55.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd055800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd050144aa0 0x7fd0500d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:55.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd055800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd050071840 0x7fd0500d0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:55.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd055800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0501456d0 con 0x7fd0501446c0 2024-04-17T10:37:55.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd055800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0500b51a0 con 0x7fd050144aa0 2024-04-17T10:37:55.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd055800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500d1390 con 0x7fd050071840 2024-04-17T10:37:55.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04ea00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd0501446c0 0x7fd050144ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:46372/0 (socket says 172.21.15.67:46372) 2024-04-17T10:37:55.329 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04ea00640 1 -- 172.21.15.67:0/3743215860 learned_addr learned my addr 172.21.15.67:0/3743215860 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:37:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04f400640 1 --2- 172.21.15.67:0/3743215860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd050071840 0x7fd0500d0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd046000640 1 --2- 172.21.15.67:0/3743215860 >> v2:172.21.15.86:3300/0 conn(0x7fd050144aa0 0x7fd0500d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04e000640 1 -- 172.21.15.67:0/3743215860 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430506143 0 0) 0x7fd0501456d0 con 0x7fd0501446c0 2024-04-17T10:37:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04e000640 1 -- 172.21.15.67:0/3743215860 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd034003530 con 0x7fd0501446c0 2024-04-17T10:37:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04e000640 1 -- 172.21.15.67:0/3743215860 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158374184 0 0) 0x7fd034003530 con 0x7fd0501446c0 2024-04-17T10:37:55.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04f400640 1 -- 172.21.15.67:0/3743215860 >> v2:172.21.15.86:3300/0 conn(0x7fd050144aa0 msgr2=0x7fd0500d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:55.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04f400640 1 --2- 172.21.15.67:0/3743215860 >> v2:172.21.15.86:3300/0 conn(0x7fd050144aa0 0x7fd0500d0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04f400640 1 -- 172.21.15.67:0/3743215860 >> v1:172.21.15.67:6789/0 conn(0x7fd0501446c0 legacy=0x7fd050144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:55.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.336+0000 7fd04f400640 1 -- 172.21.15.67:0/3743215860 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500b6640 con 0x7fd050071840 2024-04-17T10:37:55.332 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd04f400640 1 --2- 172.21.15.67:0/3743215860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd050071840 0x7fd0500d0e50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd048009d90 tx=0x7fd04801c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a60426a41d28f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd04e000640 1 -- 172.21.15.67:0/3743215860 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0480b2070 con 0x7fd050071840 2024-04-17T10:37:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd04e000640 1 -- 172.21.15.67:0/3743215860 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0480a5660 con 0x7fd050071840 2024-04-17T10:37:55.333 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd04e000640 1 -- 172.21.15.67:0/3743215860 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0400016a0 con 0x7fd050071840 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 -- 172.21.15.67:0/3743215860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd050071840 msgr2=0x7fd0500d0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 --2- 172.21.15.67:0/3743215860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd050071840 0x7fd0500d0e50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd048009d90 tx=0x7fd04801c860 comp rx=0 tx=0).stop 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 -- 172.21.15.67:0/3743215860 shutdown_connections 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 --2- 172.21.15.67:0/3743215860 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd050071840 0x7fd0500d0e50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 --2- 172.21.15.67:0/3743215860 >> v2:172.21.15.86:3300/0 conn(0x7fd050144aa0 0x7fd0500d0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 -- 172.21.15.67:0/3743215860 >> 172.21.15.67:0/3743215860 conn(0x7fd0500b34d0 msgr2=0x7fd0500b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 -- 172.21.15.67:0/3743215860 shutdown_connections 2024-04-17T10:37:55.334 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.340+0000 7fd055800640 1 -- 172.21.15.67:0/3743215860 wait complete. 2024-04-17T10:37:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 Processor -- start 2024-04-17T10:37:55.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 -- start start 2024-04-17T10:37:55.337 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd050071840 0x7fd0500b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0501446c0 0x7fd0500b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd050144aa0 0x7fd0500b1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500b5e40 con 0x7fd050071840 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05015c0d0 con 0x7fd050144aa0 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0500b5fc0 con 0x7fd0501446c0 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd050071840 0x7fd0500b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd050071840 0x7fd0500b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50416/0 (socket says 172.21.15.67:50416) 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 -- 172.21.15.67:0/1447495090 learned_addr learned my addr 172.21.15.67:0/1447495090 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd046000640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0501446c0 0x7fd0500b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04f400640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd050144aa0 0x7fd0500b1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 -- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0501446c0 msgr2=0x7fd0500b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0501446c0 0x7fd0500b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 -- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd050144aa0 msgr2=0x7fd0500b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd050144aa0 0x7fd0500b1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 -- 172.21.15.67:0/1447495090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0500acca0 con 0x7fd050071840 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04f400640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd050144aa0 0x7fd0500b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd046000640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0501446c0 0x7fd0500b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd04ea00640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd050071840 0x7fd0500b0140 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c730 tx=0x7fd04000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd047400640 1 -- 172.21.15.67:0/1447495090 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd040018070 con 0x7fd050071840 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd047400640 1 -- 172.21.15.67:0/1447495090 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0400072f0 con 0x7fd050071840 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd047400640 1 -- 172.21.15.67:0/1447495090 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0400047f0 con 0x7fd050071840 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0500acf80 con 0x7fd050071840 2024-04-17T10:37:55.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0500ad540 con 0x7fd050071840 2024-04-17T10:37:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.344+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd050070bb0 con 0x7fd050071840 2024-04-17T10:37:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.348+0000 7fd047400640 1 -- 172.21.15.67:0/1447495090 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd04001b070 con 0x7fd050071840 2024-04-17T10:37:55.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.348+0000 7fd047400640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd02c066f60 0x7fd02c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.348+0000 7fd047400640 1 -- 172.21.15.67:0/1447495090 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fd040084770 con 0x7fd050071840 2024-04-17T10:37:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.348+0000 7fd046000640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd02c066f60 0x7fd02c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.348+0000 7fd046000640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd02c066f60 0x7fd02c069420 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd0500b23e0 tx=0x7fd038008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:55.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.352+0000 7fd047400640 1 -- 172.21.15.67:0/1447495090 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd040055550 con 0x7fd050071840 2024-04-17T10:37:55.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd050140b20 con 0x7fd050071840 2024-04-17T10:37:55.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd047400640 1 -- 172.21.15.67:0/1447495090 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fd040054e50 con 0x7fd050071840 2024-04-17T10:37:55.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:55.540 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:55.541 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:55.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd02c066f60 msgr2=0x7fd02c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:55.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd02c066f60 0x7fd02c069420 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fd0500b23e0 tx=0x7fd038008040 comp rx=0 tx=0).stop 2024-04-17T10:37:55.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd050071840 msgr2=0x7fd0500b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:55.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd050071840 0x7fd0500b0140 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fd04000c730 tx=0x7fd04000cbf0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 shutdown_connections 2024-04-17T10:37:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd02c066f60 0x7fd02c069420 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd050144aa0 0x7fd0500b1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0501446c0 0x7fd0500b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 --2- 172.21.15.67:0/1447495090 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd050071840 0x7fd0500b0140 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 >> 172.21.15.67:0/1447495090 conn(0x7fd0500b34d0 msgr2=0x7fd05006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 shutdown_connections 2024-04-17T10:37:55.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:55.548+0000 7fd055800640 1 -- 172.21.15.67:0/1447495090 wait complete. 2024-04-17T10:37:55.964 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:55.965 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:56.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.124+0000 7fecfc400640 1 Processor -- start 2024-04-17T10:37:57.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfc400640 1 -- start start 2024-04-17T10:37:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfc400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40ce2a0 0x7fecf40cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfc400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecf40cc840 0x7fecf40cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfc400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecf406b100 con 0x7fecf40cc250 2024-04-17T10:37:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfc400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecf40cd150 con 0x7fecf40cc840 2024-04-17T10:37:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfc400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf406b3e0 con 0x7fecf40ce2a0 2024-04-17T10:37:57.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfa400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecf40cc840 0x7fecf40cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfa400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fecf40cc840 0x7fecf40cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50430/0 (socket says 172.21.15.67:50430) 2024-04-17T10:37:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfa400640 1 -- 172.21.15.67:0/3400948389 learned_addr learned my addr 172.21.15.67:0/3400948389 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecf9000640 1 --2- 172.21.15.67:0/3400948389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40ce2a0 0x7fecf40cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfa400640 1 -- 172.21.15.67:0/3400948389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40ce2a0 msgr2=0x7fecf40cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:57.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfa400640 1 --2- 172.21.15.67:0/3400948389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40ce2a0 0x7fecf40cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfa400640 1 -- 172.21.15.67:0/3400948389 >> v1:172.21.15.86:6789/0 conn(0x7fecf40cc250 legacy=0x7fecf40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:57.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.128+0000 7fecfa400640 1 -- 172.21.15.67:0/3400948389 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf406b6c0 con 0x7fecf40cc840 2024-04-17T10:37:57.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfa400640 1 --2- 172.21.15.67:0/3400948389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40cc840 0x7fecf40cdb20 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fecf40cd2d0 tx=0x7fecec01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b038079f9ebaaf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecf3e00640 1 -- 172.21.15.67:0/3400948389 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0a57f0 con 0x7fecf40cc840 2024-04-17T10:37:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecf3e00640 1 -- 172.21.15.67:0/3400948389 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec0a5990 con 0x7fecf40cc840 2024-04-17T10:37:57.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecf3e00640 1 -- 172.21.15.67:0/3400948389 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0a5d60 con 0x7fecf40cc840 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 -- 172.21.15.67:0/3400948389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40cc840 msgr2=0x7fecf40cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 --2- 172.21.15.67:0/3400948389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40cc840 0x7fecf40cdb20 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fecf40cd2d0 tx=0x7fecec01ca10 comp rx=0 tx=0).stop 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 -- 172.21.15.67:0/3400948389 shutdown_connections 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 --2- 172.21.15.67:0/3400948389 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40cc840 0x7fecf40cdb20 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 --2- 172.21.15.67:0/3400948389 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40ce2a0 0x7fecf40cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 -- 172.21.15.67:0/3400948389 >> 172.21.15.67:0/3400948389 conn(0x7fecf40b22a0 msgr2=0x7fecf40b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 -- 172.21.15.67:0/3400948389 shutdown_connections 2024-04-17T10:37:57.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 -- 172.21.15.67:0/3400948389 wait complete. 2024-04-17T10:37:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 Processor -- start 2024-04-17T10:37:57.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.132+0000 7fecfc400640 1 -- start start 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf40cc250 0x7fecf4159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40cc840 0x7fecf40c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40ce2a0 0x7fecf415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf416ac80 con 0x7fecf40ce2a0 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf416ab00 con 0x7fecf40cc250 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecf416ae00 con 0x7fecf40cc840 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40ce2a0 0x7fecf415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40ce2a0 0x7fecf415a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50446/0 (socket says 172.21.15.67:50446) 2024-04-17T10:37:57.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 -- 172.21.15.67:0/1943353239 learned_addr learned my addr 172.21.15.67:0/1943353239 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecf9000640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40cc840 0x7fecf40c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecf9a00640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf40cc250 0x7fecf4159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 -- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40cc840 msgr2=0x7fecf40c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40cc840 0x7fecf40c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 -- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf40cc250 msgr2=0x7fecf4159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf40cc250 0x7fecf4159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 -- 172.21.15.67:0/1943353239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecf40c4be0 con 0x7fecf40ce2a0 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecf9000640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40cc840 0x7fecf40c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecf9a00640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf40cc250 0x7fecf4159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfa400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40ce2a0 0x7fecf415a5c0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fecec01a640 tx=0x7fecec0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecf1600640 1 -- 172.21.15.67:0/1943353239 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0a5d40 con 0x7fecf40ce2a0 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecf1600640 1 -- 172.21.15.67:0/1943353239 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecec0be070 con 0x7fecf40ce2a0 2024-04-17T10:37:57.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf40c4ec0 con 0x7fecf40ce2a0 2024-04-17T10:37:57.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecf1600640 1 -- 172.21.15.67:0/1943353239 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecec0b86b0 con 0x7fecf40ce2a0 2024-04-17T10:37:57.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf40c5480 con 0x7fecf40ce2a0 2024-04-17T10:37:57.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.136+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf409e900 con 0x7fecf40ce2a0 2024-04-17T10:37:57.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.140+0000 7fecf1600640 1 -- 172.21.15.67:0/1943353239 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fecec0b8850 con 0x7fecf40ce2a0 2024-04-17T10:37:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.140+0000 7fecf1600640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecd4066f60 0x7fecd4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.140+0000 7fecf1600640 1 -- 172.21.15.67:0/1943353239 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fecec1204b0 con 0x7fecf40ce2a0 2024-04-17T10:37:57.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.140+0000 7fecf9a00640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecd4066f60 0x7fecd4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:57.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.140+0000 7fecf9a00640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecd4066f60 0x7fecd4069420 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fece80016f0 tx=0x7fece8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:57.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.144+0000 7fecf1600640 1 -- 172.21.15.67:0/1943353239 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fecec016600 con 0x7fecf40ce2a0 2024-04-17T10:37:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fecf415e8f0 con 0x7fecf40ce2a0 2024-04-17T10:37:57.335 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecf1600640 1 -- 172.21.15.67:0/1943353239 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fecec0f1320 con 0x7fecf40ce2a0 2024-04-17T10:37:57.336 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:57.336 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:57.336 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecd4066f60 msgr2=0x7fecd4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecd4066f60 0x7fecd4069420 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fece80016f0 tx=0x7fece8009040 comp rx=0 tx=0).stop 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40ce2a0 msgr2=0x7fecf415a5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40ce2a0 0x7fecf415a5c0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fecec01a640 tx=0x7fecec0197b0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 shutdown_connections 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fecd4066f60 0x7fecd4069420 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fecf40ce2a0 0x7fecf415a5c0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fecf40cc840 0x7fecf40c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 --2- 172.21.15.67:0/1943353239 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fecf40cc250 0x7fecf4159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 >> 172.21.15.67:0/1943353239 conn(0x7fecf40b22a0 msgr2=0x7fecf40cacc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:57.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 shutdown_connections 2024-04-17T10:37:57.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:57.344+0000 7fecfc400640 1 -- 172.21.15.67:0/1943353239 wait complete. 2024-04-17T10:37:57.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:57.759 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:37:58.467 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:37:58.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:37:58.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:37:58.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 Processor -- start 2024-04-17T10:37:58.912 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 -- start start 2024-04-17T10:37:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1d08145040 0x7f1d0806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d0806ef10 0x7f1d08071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d08071850 0x7f1d0815f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:58.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d080a86c0 con 0x7f1d08071850 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0815bf10 con 0x7f1d08145040 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d0cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d08071ca0 con 0x7f1d0806ef10 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d08071850 0x7f1d0815f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1d08071850 0x7f1d0815f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50450/0 (socket says 172.21.15.67:50450) 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 -- 172.21.15.67:0/2554316613 learned_addr learned my addr 172.21.15.67:0/2554316613 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1cfd400640 1 --2- 172.21.15.67:0/2554316613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d0806ef10 0x7f1d08071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d05e00640 1 --2- 172.21.15.67:0/2554316613 >> v2:172.21.15.86:3300/0 conn(0x7f1d08145040 0x7f1d0806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 -- 172.21.15.67:0/2554316613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d0806ef10 msgr2=0x7f1d08071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 --2- 172.21.15.67:0/2554316613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d0806ef10 0x7f1d08071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 -- 172.21.15.67:0/2554316613 >> v2:172.21.15.86:3300/0 conn(0x7f1d08145040 msgr2=0x7f1d0806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 --2- 172.21.15.67:0/2554316613 >> v2:172.21.15.86:3300/0 conn(0x7f1d08145040 0x7f1d0806e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d06800640 1 -- 172.21.15.67:0/2554316613 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0806e2f0 con 0x7f1d08071850 2024-04-17T10:37:58.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.920+0000 7f1d05e00640 1 --2- 172.21.15.67:0/2554316613 >> v2:172.21.15.86:3300/0 conn(0x7f1d08145040 0x7f1d0806e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:37:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d06800640 1 --2- 172.21.15.67:0/2554316613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815f510 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f1d08072410 tx=0x7f1d0001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69afa646eede1a2d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d05400640 1 -- 172.21.15.67:0/2554316613 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d000b5070 con 0x7f1d08071850 2024-04-17T10:37:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d05400640 1 -- 172.21.15.67:0/2554316613 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d000a89d0 con 0x7f1d08071850 2024-04-17T10:37:58.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d05400640 1 -- 172.21.15.67:0/2554316613 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d000a8da0 con 0x7f1d08071850 2024-04-17T10:37:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 -- 172.21.15.67:0/2554316613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 msgr2=0x7f1d0815f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/2554316613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815f510 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f1d08072410 tx=0x7f1d0001ed10 comp rx=0 tx=0).stop 2024-04-17T10:37:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 -- 172.21.15.67:0/2554316613 shutdown_connections 2024-04-17T10:37:58.918 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/2554316613 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815f510 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/2554316613 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d0806ef10 0x7f1d08071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/2554316613 >> v2:172.21.15.86:3300/0 conn(0x7f1d08145040 0x7f1d0806e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 -- 172.21.15.67:0/2554316613 >> 172.21.15.67:0/2554316613 conn(0x7f1d0806ac90 msgr2=0x7f1d0806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 -- 172.21.15.67:0/2554316613 shutdown_connections 2024-04-17T10:37:58.919 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.924+0000 7f1d0cc00640 1 -- 172.21.15.67:0/2554316613 wait complete. 2024-04-17T10:37:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 Processor -- start 2024-04-17T10:37:58.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 -- start start 2024-04-17T10:37:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d0806ef10 0x7f1d08138dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d08145040 0x7f1d0813e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0806dca0 con 0x7f1d08071850 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0806d9c0 con 0x7f1d0806ef10 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0806df80 con 0x7f1d08145040 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50460/0 (socket says 172.21.15.67:50460) 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 -- 172.21.15.67:0/1809857043 learned_addr learned my addr 172.21.15.67:0/1809857043 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d05e00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d0806ef10 0x7f1d08138dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d06800640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d08145040 0x7f1d0813e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 -- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d08145040 msgr2=0x7f1d0813e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d08145040 0x7f1d0813e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:58.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 -- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d0806ef10 msgr2=0x7f1d08138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d0806ef10 0x7f1d08138dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 -- 172.21.15.67:0/1809857043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08139450 con 0x7f1d08071850 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfd400640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815aa00 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0007e40 tx=0x7f1cf000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfea00640 1 -- 172.21.15.67:0/1809857043 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0014620 con 0x7f1d08071850 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfea00640 1 -- 172.21.15.67:0/1809857043 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf0019070 con 0x7f1d08071850 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1cfea00640 1 -- 172.21.15.67:0/1809857043 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf0014a20 con 0x7f1d08071850 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d08139730 con 0x7f1d08071850 2024-04-17T10:37:58.924 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.928+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d08139cf0 con 0x7f1d08071850 2024-04-17T10:37:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.932+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d08072d50 con 0x7f1d08071850 2024-04-17T10:37:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.932+0000 7f1cfea00640 1 -- 172.21.15.67:0/1809857043 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1cf0014bc0 con 0x7f1d08071850 2024-04-17T10:37:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.932+0000 7f1cfea00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ce4066f60 0x7f1ce4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:37:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.932+0000 7f1cfea00640 1 -- 172.21.15.67:0/1809857043 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1cf0085a30 con 0x7f1d08071850 2024-04-17T10:37:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.932+0000 7f1d05e00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ce4066f60 0x7f1ce4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:37:58.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.932+0000 7f1d05e00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ce4066f60 0x7f1ce4069420 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1d0815ec10 tx=0x7f1cf8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:37:58.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:58.936+0000 7f1cfea00640 1 -- 172.21.15.67:0/1809857043 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1cf0056130 con 0x7f1d08071850 2024-04-17T10:37:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d08074e30 con 0x7f1d08071850 2024-04-17T10:37:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1cfea00640 1 -- 172.21.15.67:0/1809857043 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1cf005a1d0 con 0x7f1d08071850 2024-04-17T10:37:59.125 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:37:59.125 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:37:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:37:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ce4066f60 msgr2=0x7f1ce4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ce4066f60 0x7f1ce4069420 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1d0815ec10 tx=0x7f1cf8008040 comp rx=0 tx=0).stop 2024-04-17T10:37:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 msgr2=0x7f1d0815aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:37:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815aa00 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0007e40 tx=0x7f1cf000e4b0 comp rx=0 tx=0).stop 2024-04-17T10:37:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 shutdown_connections 2024-04-17T10:37:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1ce4066f60 0x7f1ce4069420 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1d08145040 0x7f1d0813e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1d08071850 0x7f1d0815aa00 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 --2- 172.21.15.67:0/1809857043 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1d0806ef10 0x7f1d08138dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:37:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 >> 172.21.15.67:0/1809857043 conn(0x7f1d0806ac90 msgr2=0x7f1d0806b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:37:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 shutdown_connections 2024-04-17T10:37:59.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:37:59.132+0000 7f1d0cc00640 1 -- 172.21.15.67:0/1809857043 wait complete. 2024-04-17T10:37:59.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:37:59.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:00.549 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:00.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb739400640 1 Processor -- start 2024-04-17T10:38:00.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb739400640 1 -- start start 2024-04-17T10:38:00.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb739400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406e6e0 0x7fb7340ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:00.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb739400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7341441f0 0x7fb7340ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:00.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb739400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73415ae40 con 0x7fb73406eac0 2024-04-17T10:38:00.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb739400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7340cf430 con 0x7fb7341441f0 2024-04-17T10:38:00.701 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb739400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb734071440 con 0x7fb73406e6e0 2024-04-17T10:38:00.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb733200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb7341441f0 0x7fb7340ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:00.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb729e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb73406eac0 0x7fb734143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39800/0 (socket says 172.21.15.67:39800) 2024-04-17T10:38:00.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb729e00640 1 -- 172.21.15.67:0/232289549 learned_addr learned my addr 172.21.15.67:0/232289549 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:00.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb732800640 1 --2- 172.21.15.67:0/232289549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406e6e0 0x7fb7340ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:00.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb733200640 1 -- 172.21.15.67:0/232289549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406e6e0 msgr2=0x7fb7340ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:00.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb733200640 1 --2- 172.21.15.67:0/232289549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406e6e0 0x7fb7340ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.702 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb733200640 1 -- 172.21.15.67:0/232289549 >> v1:172.21.15.67:6789/0 conn(0x7fb73406eac0 legacy=0x7fb734143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:00.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb733200640 1 -- 172.21.15.67:0/232289549 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb734071720 con 0x7fb7341441f0 2024-04-17T10:38:00.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.708+0000 7fb731e00640 1 -- 172.21.15.67:0/232289549 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269973332 0 0) 0x7fb73415ae40 con 0x7fb73406eac0 2024-04-17T10:38:00.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb733200640 1 --2- 172.21.15.67:0/232289549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7341441f0 0x7fb7340ceef0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0097c0 tx=0x7fb72c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90469493d29d6b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:00.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb731e00640 1 -- 172.21.15.67:0/232289549 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0b1070 con 0x7fb7341441f0 2024-04-17T10:38:00.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb731e00640 1 -- 172.21.15.67:0/232289549 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c0a8050 con 0x7fb7341441f0 2024-04-17T10:38:00.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb731e00640 1 -- 172.21.15.67:0/232289549 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c0ac860 con 0x7fb7341441f0 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 -- 172.21.15.67:0/232289549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7341441f0 msgr2=0x7fb7340ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 --2- 172.21.15.67:0/232289549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7341441f0 0x7fb7340ceef0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0097c0 tx=0x7fb72c01c3e0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 -- 172.21.15.67:0/232289549 shutdown_connections 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 --2- 172.21.15.67:0/232289549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb7341441f0 0x7fb7340ceef0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 --2- 172.21.15.67:0/232289549 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406e6e0 0x7fb7340ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 -- 172.21.15.67:0/232289549 >> 172.21.15.67:0/232289549 conn(0x7fb73406ac90 msgr2=0x7fb73406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 -- 172.21.15.67:0/232289549 shutdown_connections 2024-04-17T10:38:00.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.712+0000 7fb739400640 1 -- 172.21.15.67:0/232289549 wait complete. 2024-04-17T10:38:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 Processor -- start 2024-04-17T10:38:00.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 -- start start 2024-04-17T10:38:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb73406e6e0 0x7fb7340aa810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406eac0 0x7fb7340ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:00.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7341441f0 0x7fb7340abf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340b3a40 con 0x7fb73406e6e0 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340b38c0 con 0x7fb7341441f0 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7340b3bc0 con 0x7fb73406eac0 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb73406e6e0 0x7fb7340aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb73406e6e0 0x7fb7340aa810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50514/0 (socket says 172.21.15.67:50514) 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 -- 172.21.15.67:0/1533810258 learned_addr learned my addr 172.21.15.67:0/1533810258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb729e00640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406eac0 0x7fb7340ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb733200640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7341441f0 0x7fb7340abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 -- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406eac0 msgr2=0x7fb7340ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406eac0 0x7fb7340ad6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 -- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7341441f0 msgr2=0x7fb7340abf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7341441f0 0x7fb7340abf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 -- 172.21.15.67:0/1533810258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7340ac4d0 con 0x7fb73406e6e0 2024-04-17T10:38:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb732800640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb73406e6e0 0x7fb7340aa810 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fb72400d800 tx=0x7fb72400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb72b400640 1 -- 172.21.15.67:0/1533810258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72400b6c0 con 0x7fb73406e6e0 2024-04-17T10:38:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb72b400640 1 -- 172.21.15.67:0/1533810258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724019070 con 0x7fb73406e6e0 2024-04-17T10:38:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7340aaef0 con 0x7fb73406e6e0 2024-04-17T10:38:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb72b400640 1 -- 172.21.15.67:0/1533810258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72400ba60 con 0x7fb73406e6e0 2024-04-17T10:38:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7340ab4b0 con 0x7fb73406e6e0 2024-04-17T10:38:00.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.716+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb734072d50 con 0x7fb73406e6e0 2024-04-17T10:38:00.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.720+0000 7fb72b400640 1 -- 172.21.15.67:0/1533810258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb72400bc00 con 0x7fb73406e6e0 2024-04-17T10:38:00.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.720+0000 7fb72b400640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb710066f60 0x7fb710069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:00.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.720+0000 7fb72b400640 1 -- 172.21.15.67:0/1533810258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fb724085790 con 0x7fb73406e6e0 2024-04-17T10:38:00.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.720+0000 7fb729e00640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb710066f60 0x7fb710069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:00.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.720+0000 7fb729e00640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb710066f60 0x7fb710069420 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb7340ae190 tx=0x7fb71c007ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:00.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.724+0000 7fb72b400640 1 -- 172.21.15.67:0/1533810258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb7240564e0 con 0x7fb73406e6e0 2024-04-17T10:38:00.913 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.920+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb734074e30 con 0x7fb73406e6e0 2024-04-17T10:38:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.920+0000 7fb72b400640 1 -- 172.21.15.67:0/1533810258 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fb724083e20 con 0x7fb73406e6e0 2024-04-17T10:38:00.914 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:00.914 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb710066f60 msgr2=0x7fb710069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb710066f60 0x7fb710069420 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb7340ae190 tx=0x7fb71c007ee0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb73406e6e0 msgr2=0x7fb7340aa810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:00.916 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb73406e6e0 0x7fb7340aa810 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fb72400d800 tx=0x7fb72400dcc0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 shutdown_connections 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb710066f60 0x7fb710069420 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb7341441f0 0x7fb7340abf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb73406eac0 0x7fb7340ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 --2- 172.21.15.67:0/1533810258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb73406e6e0 0x7fb7340aa810 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 >> 172.21.15.67:0/1533810258 conn(0x7fb73406ac90 msgr2=0x7fb734143a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 shutdown_connections 2024-04-17T10:38:00.917 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:00.924+0000 7fb739400640 1 -- 172.21.15.67:0/1533810258 wait complete. 2024-04-17T10:38:01.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:01.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:02.339 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:02.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcdef200640 1 Processor -- start 2024-04-17T10:38:02.492 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcdef200640 1 -- start start 2024-04-17T10:38:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcdef200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcde80ce020 0x7fcde81620a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcdef200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde815eb50 con 0x7fcde80cf510 2024-04-17T10:38:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcdef200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde815ee30 con 0x7fcde80ce400 2024-04-17T10:38:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcdef200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde8161650 con 0x7fcde80ce020 2024-04-17T10:38:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcded200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcde80cf510 0x7fcde80cf960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39812/0 (socket says 172.21.15.67:39812) 2024-04-17T10:38:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcded200640 1 -- 172.21.15.67:0/1535298845 learned_addr learned my addr 172.21.15.67:0/1535298845 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:02.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcde7e00640 1 --2- 172.21.15.67:0/1535298845 >> v2:172.21.15.86:3300/0 conn(0x7fcde80ce020 0x7fcde81620a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcde6a00640 1 -- 172.21.15.67:0/1535298845 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116231187 0 0) 0x7fcde815eb50 con 0x7fcde80cf510 2024-04-17T10:38:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcde6a00640 1 -- 172.21.15.67:0/1535298845 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdcc003530 con 0x7fcde80cf510 2024-04-17T10:38:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcde7e00640 1 -- 172.21.15.67:0/1535298845 >> v1:172.21.15.86:6789/0 conn(0x7fcde80ce400 legacy=0x7fcde815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcde7e00640 1 -- 172.21.15.67:0/1535298845 >> v1:172.21.15.67:6789/0 conn(0x7fcde80cf510 legacy=0x7fcde80cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:02.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.500+0000 7fcde7e00640 1 -- 172.21.15.67:0/1535298845 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde81618a0 con 0x7fcde80ce020 2024-04-17T10:38:02.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcde7e00640 1 --2- 172.21.15.67:0/1535298845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80ce020 0x7fcde81620a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80097f0 tx=0x7fcdd801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5cbdf82e0516c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcde6a00640 1 -- 172.21.15.67:0/1535298845 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80b1070 con 0x7fcde80ce020 2024-04-17T10:38:02.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcde6a00640 1 -- 172.21.15.67:0/1535298845 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd80ac410 con 0x7fcde80ce020 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcde6a00640 1 -- 172.21.15.67:0/1535298845 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80ac7e0 con 0x7fcde80ce020 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcdef200640 1 -- 172.21.15.67:0/1535298845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80ce020 msgr2=0x7fcde81620a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcdef200640 1 --2- 172.21.15.67:0/1535298845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80ce020 0x7fcde81620a0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80097f0 tx=0x7fcdd801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcdef200640 1 -- 172.21.15.67:0/1535298845 shutdown_connections 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcdef200640 1 --2- 172.21.15.67:0/1535298845 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80ce020 0x7fcde81620a0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcdef200640 1 -- 172.21.15.67:0/1535298845 >> 172.21.15.67:0/1535298845 conn(0x7fcde80b33f0 msgr2=0x7fcde80b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcdef200640 1 -- 172.21.15.67:0/1535298845 shutdown_connections 2024-04-17T10:38:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.504+0000 7fcdef200640 1 -- 172.21.15.67:0/1535298845 wait complete. 2024-04-17T10:38:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 Processor -- start 2024-04-17T10:38:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 -- start start 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde80ce020 0x7fcde80b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcde80ce400 0x7fcde8147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80cf510 0x7fcde80c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde815e8a0 con 0x7fcde80ce020 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde815e720 con 0x7fcde80cf510 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde80c84b0 con 0x7fcde80ce400 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde80ce020 0x7fcde80b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde80ce020 0x7fcde80b1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50522/0 (socket says 172.21.15.67:50522) 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 -- 172.21.15.67:0/1998613077 learned_addr learned my addr 172.21.15.67:0/1998613077 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcded200640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80cf510 0x7fcde80c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7400640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcde80ce400 0x7fcde8147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:02.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 -- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcde80ce400 msgr2=0x7fcde8147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcde80ce400 0x7fcde8147f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 -- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80cf510 msgr2=0x7fcde80c6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80cf510 0x7fcde80c6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 -- 172.21.15.67:0/1998613077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde80b2800 con 0x7fcde80ce020 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7400640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcde80ce400 0x7fcde8147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcde7e00640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde80ce020 0x7fcde80b1de0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80af040 tx=0x7fcdd8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcddfe00640 1 -- 172.21.15.67:0/1998613077 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80b1040 con 0x7fcde80ce020 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcddfe00640 1 -- 172.21.15.67:0/1998613077 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd80bc070 con 0x7fcde80ce020 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcddfe00640 1 -- 172.21.15.67:0/1998613077 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd80b5410 con 0x7fcde80ce020 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde80b2ae0 con 0x7fcde80ce020 2024-04-17T10:38:02.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.508+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde80b15e0 con 0x7fcde80ce020 2024-04-17T10:38:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.512+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde8070ab0 con 0x7fcde80ce020 2024-04-17T10:38:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.512+0000 7fcddfe00640 1 -- 172.21.15.67:0/1998613077 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcdd80b55b0 con 0x7fcde80ce020 2024-04-17T10:38:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.512+0000 7fcddfe00640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdc8066f60 0x7fcdc8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:02.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.512+0000 7fcddfe00640 1 -- 172.21.15.67:0/1998613077 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fcdd811ed40 con 0x7fcde80ce020 2024-04-17T10:38:02.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.512+0000 7fcde7400640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdc8066f60 0x7fcdc8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:02.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.512+0000 7fcde7400640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdc8066f60 0x7fcdc8069420 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcde814b170 tx=0x7fcdd4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:02.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.516+0000 7fcddfe00640 1 -- 172.21.15.67:0/1998613077 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcdd8016090 con 0x7fcde80ce020 2024-04-17T10:38:02.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.716+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcde8068120 con 0x7fcde80ce020 2024-04-17T10:38:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.716+0000 7fcddfe00640 1 -- 172.21.15.67:0/1998613077 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fcdd80efa90 con 0x7fcde80ce020 2024-04-17T10:38:02.709 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:02.709 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:02.709 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.716+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdc8066f60 msgr2=0x7fcdc8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.716+0000 7fcdef200640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdc8066f60 0x7fcdc8069420 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcde814b170 tx=0x7fcdd4009040 comp rx=0 tx=0).stop 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.716+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde80ce020 msgr2=0x7fcde80b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.716+0000 7fcdef200640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde80ce020 0x7fcde80b1de0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80af040 tx=0x7fcdd8019240 comp rx=0 tx=0).stop 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 shutdown_connections 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcdc8066f60 0x7fcdc8069420 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcde80cf510 0x7fcde80c6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcde80ce400 0x7fcde8147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 --2- 172.21.15.67:0/1998613077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcde80ce020 0x7fcde80b1de0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:02.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 >> 172.21.15.67:0/1998613077 conn(0x7fcde80b33f0 msgr2=0x7fcde80cbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:02.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 shutdown_connections 2024-04-17T10:38:02.712 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:02.720+0000 7fcdef200640 1 -- 172.21.15.67:0/1998613077 wait complete. 2024-04-17T10:38:03.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:03.132 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:04.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.292+0000 7fb2fca00640 1 Processor -- start 2024-04-17T10:38:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.292+0000 7fb2fca00640 1 -- start start 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2fca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2f80ce2a0 0x7fb2f80cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2fca00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb2f80cc840 0x7fb2f80cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2fca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f806b100 con 0x7fb2f80cc250 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2fca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f80cd150 con 0x7fb2f80ce2a0 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2fca00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f806b3e0 con 0x7fb2f80cc840 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2ed400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2f80ce2a0 0x7fb2f80cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2ed400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb2f80ce2a0 0x7fb2f80cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50530/0 (socket says 172.21.15.67:50530) 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2ed400640 1 -- 172.21.15.67:0/3906036227 learned_addr learned my addr 172.21.15.67:0/3906036227 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2f6800640 1 --2- 172.21.15.67:0/3906036227 >> v2:172.21.15.86:3300/0 conn(0x7fb2f80cc840 0x7fb2f80cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2ed400640 1 -- 172.21.15.67:0/3906036227 >> v2:172.21.15.86:3300/0 conn(0x7fb2f80cc840 msgr2=0x7fb2f80cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2ed400640 1 --2- 172.21.15.67:0/3906036227 >> v2:172.21.15.86:3300/0 conn(0x7fb2f80cc840 0x7fb2f80cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2ed400640 1 -- 172.21.15.67:0/3906036227 >> v1:172.21.15.67:6789/0 conn(0x7fb2f80cc250 legacy=0x7fb2f80cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2ed400640 1 -- 172.21.15.67:0/3906036227 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f806b6c0 con 0x7fb2f80ce2a0 2024-04-17T10:38:04.289 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.296+0000 7fb2f5400640 1 -- 172.21.15.67:0/3906036227 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363255966 0 0) 0x7fb2f806b100 con 0x7fb2f80cc250 2024-04-17T10:38:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2ed400640 1 --2- 172.21.15.67:0/3906036227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80ce2a0 0x7fb2f80cd5e0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8071b20 tx=0x7fb2e001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44821bf862185f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2f5400640 1 -- 172.21.15.67:0/3906036227 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00ab430 con 0x7fb2f80ce2a0 2024-04-17T10:38:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2f5400640 1 -- 172.21.15.67:0/3906036227 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e00b0070 con 0x7fb2f80ce2a0 2024-04-17T10:38:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2f5400640 1 -- 172.21.15.67:0/3906036227 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e00ab7f0 con 0x7fb2f80ce2a0 2024-04-17T10:38:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 -- 172.21.15.67:0/3906036227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80ce2a0 msgr2=0x7fb2f80cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 --2- 172.21.15.67:0/3906036227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80ce2a0 0x7fb2f80cd5e0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8071b20 tx=0x7fb2e001c520 comp rx=0 tx=0).stop 2024-04-17T10:38:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 -- 172.21.15.67:0/3906036227 shutdown_connections 2024-04-17T10:38:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 --2- 172.21.15.67:0/3906036227 >> v2:172.21.15.86:3300/0 conn(0x7fb2f80cc840 0x7fb2f80cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 --2- 172.21.15.67:0/3906036227 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80ce2a0 0x7fb2f80cd5e0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 -- 172.21.15.67:0/3906036227 >> 172.21.15.67:0/3906036227 conn(0x7fb2f80b22a0 msgr2=0x7fb2f80b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 -- 172.21.15.67:0/3906036227 shutdown_connections 2024-04-17T10:38:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 -- 172.21.15.67:0/3906036227 wait complete. 2024-04-17T10:38:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 Processor -- start 2024-04-17T10:38:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.300+0000 7fb2fca00640 1 -- start start 2024-04-17T10:38:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80cc250 0x7fb2f80ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80cc840 0x7fb2f815b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f80ce2a0 0x7fb2f815ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f816ac80 con 0x7fb2f80cc250 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f816ab00 con 0x7fb2f80cc840 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f816ae00 con 0x7fb2f80ce2a0 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80cc250 0x7fb2f80ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80cc250 0x7fb2f80ae080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50544/0 (socket says 172.21.15.67:50544) 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 -- 172.21.15.67:0/1113619939 learned_addr learned my addr 172.21.15.67:0/1113619939 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2ed400640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80cc840 0x7fb2f815b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f6800640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f80ce2a0 0x7fb2f815ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 -- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f80ce2a0 msgr2=0x7fb2f815ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f80ce2a0 0x7fb2f815ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 -- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80cc840 msgr2=0x7fb2f815b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80cc840 0x7fb2f815b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 -- 172.21.15.67:0/1113619939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8068060 con 0x7fb2f80cc250 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f6800640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f80ce2a0 0x7fb2f815ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2f5e00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80cc250 0x7fb2f80ae080 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800c4d0 tx=0x7fb2e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2eea00640 1 -- 172.21.15.67:0/1113619939 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8017070 con 0x7fb2f80cc250 2024-04-17T10:38:04.299 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2eea00640 1 -- 172.21.15.67:0/1113619939 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e8005480 con 0x7fb2f80cc250 2024-04-17T10:38:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2eea00640 1 -- 172.21.15.67:0/1113619939 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8003840 con 0x7fb2f80cc250 2024-04-17T10:38:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f8154240 con 0x7fb2f80cc250 2024-04-17T10:38:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f815c610 con 0x7fb2f80cc250 2024-04-17T10:38:04.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f809e900 con 0x7fb2f80cc250 2024-04-17T10:38:04.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.304+0000 7fb2eea00640 1 -- 172.21.15.67:0/1113619939 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb2e801a070 con 0x7fb2f80cc250 2024-04-17T10:38:04.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.308+0000 7fb2eea00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb2d4066f60 0x7fb2d4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:04.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.308+0000 7fb2eea00640 1 -- 172.21.15.67:0/1113619939 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fb2e80846e0 con 0x7fb2f80cc250 2024-04-17T10:38:04.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.308+0000 7fb2ed400640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb2d4066f60 0x7fb2d4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:04.301 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.308+0000 7fb2ed400640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb2d4066f60 0x7fb2d4069420 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00af000 tx=0x7fb2e00a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.312+0000 7fb2eea00640 1 -- 172.21.15.67:0/1113619939 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb2e80554f0 con 0x7fb2f80cc250 2024-04-17T10:38:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.504+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2f80b05a0 con 0x7fb2f80cc250 2024-04-17T10:38:04.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2eea00640 1 -- 172.21.15.67:0/1113619939 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fb2e8054df0 con 0x7fb2f80cc250 2024-04-17T10:38:04.499 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:04.500 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:04.500 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb2d4066f60 msgr2=0x7fb2d4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb2d4066f60 0x7fb2d4069420 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00af000 tx=0x7fb2e00a7000 comp rx=0 tx=0).stop 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80cc250 msgr2=0x7fb2f80ae080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80cc250 0x7fb2f80ae080 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800c4d0 tx=0x7fb2e800c990 comp rx=0 tx=0).stop 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 shutdown_connections 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb2d4066f60 0x7fb2d4069420 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb2f80ce2a0 0x7fb2f815ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb2f80cc840 0x7fb2f815b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 --2- 172.21.15.67:0/1113619939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb2f80cc250 0x7fb2f80ae080 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 >> 172.21.15.67:0/1113619939 conn(0x7fb2f80b22a0 msgr2=0x7fb2f80b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 shutdown_connections 2024-04-17T10:38:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:04.508+0000 7fb2fca00640 1 -- 172.21.15.67:0/1113619939 wait complete. 2024-04-17T10:38:04.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:04.924 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:05.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:06.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.084+0000 7fe617800640 1 Processor -- start 2024-04-17T10:38:06.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.084+0000 7fe617800640 1 -- start start 2024-04-17T10:38:06.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe617800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6100cf0c0 0x7fe6101441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe617800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61015ae40 con 0x7fe61006eac0 2024-04-17T10:38:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe617800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe610071440 con 0x7fe61006e6e0 2024-04-17T10:38:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe617800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe610071720 con 0x7fe6100cf0c0 2024-04-17T10:38:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe615800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe6100cf0c0 0x7fe6101441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe60fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe61006eac0 0x7fe610143ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39836/0 (socket says 172.21.15.67:39836) 2024-04-17T10:38:06.080 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe60fe00640 1 -- 172.21.15.67:0/1788051686 learned_addr learned my addr 172.21.15.67:0/1788051686 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe615800640 1 -- 172.21.15.67:0/1788051686 >> v1:172.21.15.86:6789/0 conn(0x7fe61006e6e0 legacy=0x7fe6100ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe615800640 1 -- 172.21.15.67:0/1788051686 >> v1:172.21.15.67:6789/0 conn(0x7fe61006eac0 legacy=0x7fe610143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe615800640 1 -- 172.21.15.67:0/1788051686 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe610071a00 con 0x7fe6100cf0c0 2024-04-17T10:38:06.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.088+0000 7fe60f400640 1 -- 172.21.15.67:0/1788051686 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013876272 0 0) 0x7fe61015ae40 con 0x7fe61006eac0 2024-04-17T10:38:06.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe615800640 1 --2- 172.21.15.67:0/1788051686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6100cf0c0 0x7fe6101441f0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fe608009d90 tx=0x7fe60801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e74d223754e3fd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:06.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe60f400640 1 -- 172.21.15.67:0/1788051686 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6080b2070 con 0x7fe6100cf0c0 2024-04-17T10:38:06.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe60f400640 1 -- 172.21.15.67:0/1788051686 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6080a5970 con 0x7fe6100cf0c0 2024-04-17T10:38:06.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe60f400640 1 -- 172.21.15.67:0/1788051686 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6080a5d40 con 0x7fe6100cf0c0 2024-04-17T10:38:06.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 -- 172.21.15.67:0/1788051686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6100cf0c0 msgr2=0x7fe6101441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:06.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 --2- 172.21.15.67:0/1788051686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6100cf0c0 0x7fe6101441f0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7fe608009d90 tx=0x7fe60801c8b0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 -- 172.21.15.67:0/1788051686 shutdown_connections 2024-04-17T10:38:06.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 --2- 172.21.15.67:0/1788051686 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6100cf0c0 0x7fe6101441f0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 -- 172.21.15.67:0/1788051686 >> 172.21.15.67:0/1788051686 conn(0x7fe61006ac90 msgr2=0x7fe61006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:06.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 -- 172.21.15.67:0/1788051686 shutdown_connections 2024-04-17T10:38:06.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 -- 172.21.15.67:0/1788051686 wait complete. 2024-04-17T10:38:06.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 Processor -- start 2024-04-17T10:38:06.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 -- start start 2024-04-17T10:38:06.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61006e6e0 0x7fe61014d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:06.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.092+0000 7fe617800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe61006eac0 0x7fe61006d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe617800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6100cf0c0 0x7fe61006d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe617800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6101680b0 con 0x7fe61006e6e0 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe617800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe610167f30 con 0x7fe61006eac0 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe617800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe610168230 con 0x7fe6100cf0c0 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61006e6e0 0x7fe61014d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61006e6e0 0x7fe61014d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50572/0 (socket says 172.21.15.67:50572) 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 -- 172.21.15.67:0/937495299 learned_addr learned my addr 172.21.15.67:0/937495299 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 -- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6100cf0c0 msgr2=0x7fe61006d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6100cf0c0 0x7fe61006d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 -- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe61006eac0 msgr2=0x7fe61006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe61006eac0 0x7fe61006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 -- 172.21.15.67:0/937495299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6100c8c20 con 0x7fe61006e6e0 2024-04-17T10:38:06.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe614e00640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61006e6e0 0x7fe61014d170 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fe60400c530 tx=0x7fe60400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:06.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe60cc00640 1 -- 172.21.15.67:0/937495299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604016020 con 0x7fe61006e6e0 2024-04-17T10:38:06.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe60cc00640 1 -- 172.21.15.67:0/937495299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe604005220 con 0x7fe61006e6e0 2024-04-17T10:38:06.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe60cc00640 1 -- 172.21.15.67:0/937495299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604007030 con 0x7fe61006e6e0 2024-04-17T10:38:06.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6100c8f00 con 0x7fe61006e6e0 2024-04-17T10:38:06.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6100c94c0 con 0x7fe61006e6e0 2024-04-17T10:38:06.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe610072d50 con 0x7fe61006e6e0 2024-04-17T10:38:06.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe60cc00640 1 -- 172.21.15.67:0/937495299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe6040071d0 con 0x7fe61006e6e0 2024-04-17T10:38:06.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.096+0000 7fe60cc00640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe5f0066f60 0x7fe5f0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:06.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.100+0000 7fe60cc00640 1 -- 172.21.15.67:0/937495299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fe604084c70 con 0x7fe61006e6e0 2024-04-17T10:38:06.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.100+0000 7fe60fe00640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe5f0066f60 0x7fe5f0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:06.092 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.100+0000 7fe60fe00640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe5f0066f60 0x7fe5f0069420 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc000c10 tx=0x7fe5fc007910 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:06.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.104+0000 7fe60cc00640 1 -- 172.21.15.67:0/937495299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe604020020 con 0x7fe61006e6e0 2024-04-17T10:38:06.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.296+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6100b2c60 con 0x7fe61006e6e0 2024-04-17T10:38:06.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.296+0000 7fe60cc00640 1 -- 172.21.15.67:0/937495299 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fe604055a50 con 0x7fe61006e6e0 2024-04-17T10:38:06.291 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:06.291 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:06.291 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe5f0066f60 msgr2=0x7fe5f0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe5f0066f60 0x7fe5f0069420 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc000c10 tx=0x7fe5fc007910 comp rx=0 tx=0).stop 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61006e6e0 msgr2=0x7fe61014d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61006e6e0 0x7fe61014d170 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7fe60400c530 tx=0x7fe60400c9f0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 shutdown_connections 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe5f0066f60 0x7fe5f0069420 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe6100cf0c0 0x7fe61006d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe61006eac0 0x7fe61006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 --2- 172.21.15.67:0/937495299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe61006e6e0 0x7fe61014d170 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 >> 172.21.15.67:0/937495299 conn(0x7fe61006ac90 msgr2=0x7fe61006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:06.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 shutdown_connections 2024-04-17T10:38:06.294 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:06.300+0000 7fe617800640 1 -- 172.21.15.67:0/937495299 wait complete. 2024-04-17T10:38:06.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:06.716 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:07.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:07.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.876+0000 7f18d9c00640 1 Processor -- start 2024-04-17T10:38:07.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.876+0000 7f18d9c00640 1 -- start start 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d9c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18d4145040 0x7f18d406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d9c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d406ef10 0x7f18d4071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d9c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d406e2f0 con 0x7f18d4071850 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d9c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d40a86c0 con 0x7f18d4145040 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d9c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d415bf10 con 0x7f18d406ef10 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f18d4145040 0x7f18d406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d3400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f18d4071850 0x7f18d415f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39846/0 (socket says 172.21.15.67:39846) 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d3400640 1 -- 172.21.15.67:0/2280097839 learned_addr learned my addr 172.21.15.67:0/2280097839 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:07.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18cbe00640 1 --2- 172.21.15.67:0/2280097839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d406ef10 0x7f18d4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d2a00640 1 -- 172.21.15.67:0/2280097839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d406ef10 msgr2=0x7f18d4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d2a00640 1 --2- 172.21.15.67:0/2280097839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d406ef10 0x7f18d4071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d2a00640 1 -- 172.21.15.67:0/2280097839 >> v1:172.21.15.67:6789/0 conn(0x7f18d4071850 legacy=0x7f18d415f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d2a00640 1 -- 172.21.15.67:0/2280097839 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d406d6e0 con 0x7f18d4145040 2024-04-17T10:38:07.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.880+0000 7f18d2000640 1 -- 172.21.15.67:0/2280097839 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526075437 0 0) 0x7f18d406e2f0 con 0x7f18d4071850 2024-04-17T10:38:07.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d2a00640 1 --2- 172.21.15.67:0/2280097839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d4145040 0x7f18d406e9d0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f18c40097c0 tx=0x7f18c401ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=354f22fce7c84e1d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:07.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d2000640 1 -- 172.21.15.67:0/2280097839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c40b5070 con 0x7f18d4145040 2024-04-17T10:38:07.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d2000640 1 -- 172.21.15.67:0/2280097839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c40a8990 con 0x7f18d4145040 2024-04-17T10:38:07.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d2000640 1 -- 172.21.15.67:0/2280097839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c40a8d60 con 0x7f18d4145040 2024-04-17T10:38:07.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- 172.21.15.67:0/2280097839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d4145040 msgr2=0x7f18d406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:07.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 --2- 172.21.15.67:0/2280097839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d4145040 0x7f18d406e9d0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f18c40097c0 tx=0x7f18c401ec90 comp rx=0 tx=0).stop 2024-04-17T10:38:07.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- 172.21.15.67:0/2280097839 shutdown_connections 2024-04-17T10:38:07.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 --2- 172.21.15.67:0/2280097839 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d406ef10 0x7f18d4071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:07.876 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 --2- 172.21.15.67:0/2280097839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d4145040 0x7f18d406e9d0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- 172.21.15.67:0/2280097839 >> 172.21.15.67:0/2280097839 conn(0x7f18d406ac90 msgr2=0x7f18d406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- 172.21.15.67:0/2280097839 shutdown_connections 2024-04-17T10:38:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- 172.21.15.67:0/2280097839 wait complete. 2024-04-17T10:38:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 Processor -- start 2024-04-17T10:38:07.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- start start 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d406ef10 0x7f18d40b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d4071850 0x7f18d4141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18d4145040 0x7f18d415afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d406dca0 con 0x7f18d406ef10 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4071ca0 con 0x7f18d4145040 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.884+0000 7f18d9c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d406df80 con 0x7f18d4071850 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d406ef10 0x7f18d40b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d406ef10 0x7f18d40b4010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50588/0 (socket says 172.21.15.67:50588) 2024-04-17T10:38:07.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 -- 172.21.15.67:0/1637635288 learned_addr learned my addr 172.21.15.67:0/1637635288 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 -- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d4071850 msgr2=0x7f18d4141750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d4071850 0x7f18d4141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 -- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18d4145040 msgr2=0x7f18d415afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18d4145040 0x7f18d415afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 -- 172.21.15.67:0/1637635288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d4141c90 con 0x7f18d406ef10 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d2a00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d406ef10 0x7f18d40b4010 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f18c40b3040 tx=0x7f18c401ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18caa00640 1 -- 172.21.15.67:0/1637635288 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c40bf070 con 0x7f18d406ef10 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18caa00640 1 -- 172.21.15.67:0/1637635288 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c40b0720 con 0x7f18d406ef10 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18caa00640 1 -- 172.21.15.67:0/1637635288 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c40b5040 con 0x7f18d406ef10 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d40b4710 con 0x7f18d406ef10 2024-04-17T10:38:07.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d40b2da0 con 0x7f18d406ef10 2024-04-17T10:38:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d4072d50 con 0x7f18d406ef10 2024-04-17T10:38:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18caa00640 1 -- 172.21.15.67:0/1637635288 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f18c40c2070 con 0x7f18d406ef10 2024-04-17T10:38:07.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.888+0000 7f18caa00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18b0066f60 0x7f18b0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.892+0000 7f18caa00640 1 -- 172.21.15.67:0/1637635288 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f18c41218e0 con 0x7f18d406ef10 2024-04-17T10:38:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.892+0000 7f18cbe00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18b0066f60 0x7f18b0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:07.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.892+0000 7f18cbe00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18b0066f60 0x7f18b0069420 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f18d40cd110 tx=0x7f18bc0095c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:07.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:07.892+0000 7f18caa00640 1 -- 172.21.15.67:0/1637635288 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f18c40187f0 con 0x7f18d406ef10 2024-04-17T10:38:08.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18d40aff50 con 0x7f18d406ef10 2024-04-17T10:38:08.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18caa00640 1 -- 172.21.15.67:0/1637635288 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f18c40f26c0 con 0x7f18d406ef10 2024-04-17T10:38:08.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:08.085 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:08.085 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18b0066f60 msgr2=0x7f18b0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18b0066f60 0x7f18b0069420 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f18d40cd110 tx=0x7f18bc0095c0 comp rx=0 tx=0).stop 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d406ef10 msgr2=0x7f18d40b4010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d406ef10 0x7f18d40b4010 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f18c40b3040 tx=0x7f18c401ba30 comp rx=0 tx=0).stop 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 shutdown_connections 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f18b0066f60 0x7f18b0069420 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f18d4145040 0x7f18d415afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f18d4071850 0x7f18d4141750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:08.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 --2- 172.21.15.67:0/1637635288 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18d406ef10 0x7f18d40b4010 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:08.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.092+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 >> 172.21.15.67:0/1637635288 conn(0x7f18d406ac90 msgr2=0x7f18d40cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:08.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.096+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 shutdown_connections 2024-04-17T10:38:08.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:08.096+0000 7f18d9c00640 1 -- 172.21.15.67:0/1637635288 wait complete. 2024-04-17T10:38:08.508 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:08.509 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:09.510 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:09.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f18e00640 1 Processor -- start 2024-04-17T10:38:09.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f18e00640 1 -- start start 2024-04-17T10:38:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f18e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 0x7f2f14144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f18e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2f14071840 0x7f2f140b5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f18e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f141456d0 con 0x7f2f14144aa0 2024-04-17T10:38:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f18e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f140b51a0 con 0x7f2f14071840 2024-04-17T10:38:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f18e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f140b4870 con 0x7f2f141446c0 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f09400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2f14144aa0 0x7f2f140d0910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39852/0 (socket says 172.21.15.67:39852) 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f09400640 1 -- 172.21.15.67:0/842162993 learned_addr learned my addr 172.21.15.67:0/842162993 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f12800640 1 --2- 172.21.15.67:0/842162993 >> v2:172.21.15.86:3300/0 conn(0x7f2f14071840 0x7f2f140b5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11e00640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 0x7f2f14144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3337847765 0 0) 0x7f2f141456d0 con 0x7f2f14144aa0 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ef8003530 con 0x7f2f14144aa0 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417972319 0 0) 0x7f2ef8003530 con 0x7f2f14144aa0 2024-04-17T10:38:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 msgr2=0x7f2f14144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 0x7f2f14144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 >> v2:172.21.15.86:3300/0 conn(0x7f2f14071840 msgr2=0x7f2f140b5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 --2- 172.21.15.67:0/842162993 >> v2:172.21.15.86:3300/0 conn(0x7f2f14071840 0x7f2f140b5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f140b4b50 con 0x7f2f14144aa0 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f2efc001680 con 0x7f2f14144aa0 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 >> v1:172.21.15.67:6789/0 conn(0x7f2f14144aa0 legacy=0x7f2f140d0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef80039c0 0x7f2ef8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 0x7f2ef8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef8004d50 0x7f2ef80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f140b51a0 con 0x7f2ef8004d50 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f140b4870 con 0x7f2ef80039c0 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef800b9d0 con 0x7f2f141446c0 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f12800640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef8004d50 0x7f2ef80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f11e00640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef80039c0 0x7f2ef8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.672+0000 7f2f09400640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 0x7f2ef8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f12800640 1 -- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 msgr2=0x7f2ef8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f12800640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141446c0 0x7f2ef8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f12800640 1 -- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef80039c0 msgr2=0x7f2ef8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f12800640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef80039c0 0x7f2ef8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f12800640 1 -- 172.21.15.67:0/842162993 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8003530 con 0x7f2ef8004d50 2024-04-17T10:38:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f11e00640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef80039c0 0x7f2ef8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f12800640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef8004d50 0x7f2ef80051d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009ae0 tx=0x7f2f0c01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15270b9f18686d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0b2070 con 0x7f2ef8004d50 2024-04-17T10:38:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c0a5590 con 0x7f2ef8004d50 2024-04-17T10:38:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.676+0000 7f2f11400640 1 -- 172.21.15.67:0/842162993 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0096b0 con 0x7f2ef8004d50 2024-04-17T10:38:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- 172.21.15.67:0/842162993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef8004d50 msgr2=0x7f2ef80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 --2- 172.21.15.67:0/842162993 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2ef8004d50 0x7f2ef80051d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c009ae0 tx=0x7f2f0c01cab0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 -- 172.21.15.67:0/842162993 reap_dead start 2024-04-17T10:38:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- 172.21.15.67:0/842162993 shutdown_connections 2024-04-17T10:38:09.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- 172.21.15.67:0/842162993 >> 172.21.15.67:0/842162993 conn(0x7f2f140b34d0 msgr2=0x7f2f140b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:09.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- 172.21.15.67:0/842162993 shutdown_connections 2024-04-17T10:38:09.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- 172.21.15.67:0/842162993 wait complete. 2024-04-17T10:38:09.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 Processor -- start 2024-04-17T10:38:09.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- start start 2024-04-17T10:38:09.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef8004d50 0x7f2f141f3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141589d0 0x7f2f14157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f141f7440 0x7f2f141f78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f140d1020 con 0x7f2f141f7440 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1415c0d0 con 0x7f2ef8004d50 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f18e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f140d11a0 con 0x7f2f141589d0 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f141f7440 0x7f2f141f78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f09400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141589d0 0x7f2f14157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f141f7440 0x7f2f141f78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50618/0 (socket says 172.21.15.67:50618) 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 -- 172.21.15.67:0/189030109 learned_addr learned my addr 172.21.15.67:0/189030109 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f11e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef8004d50 0x7f2f141f3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 -- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141589d0 msgr2=0x7f2f14157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141589d0 0x7f2f14157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 -- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef8004d50 msgr2=0x7f2f141f3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef8004d50 0x7f2f141f3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.680+0000 7f2f12800640 1 -- 172.21.15.67:0/189030109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f141e5370 con 0x7f2f141f7440 2024-04-17T10:38:09.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f12800640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f141f7440 0x7f2f141f78c0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c0b0040 tx=0x7f2efc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:09.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f0aa00640 1 -- 172.21.15.67:0/189030109 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0c0b2040 con 0x7f2f141f7440 2024-04-17T10:38:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f0aa00640 1 -- 172.21.15.67:0/189030109 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c004020 con 0x7f2f141f7440 2024-04-17T10:38:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f0aa00640 1 -- 172.21.15.67:0/189030109 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04009c10 con 0x7f2f141f7440 2024-04-17T10:38:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f141e5650 con 0x7f2f141f7440 2024-04-17T10:38:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f141e5c10 con 0x7f2f141f7440 2024-04-17T10:38:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f14070bb0 con 0x7f2f141f7440 2024-04-17T10:38:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f0aa00640 1 -- 172.21.15.67:0/189030109 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2f0c003620 con 0x7f2f141f7440 2024-04-17T10:38:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f0aa00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ef0066f60 0x7f2ef0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f0aa00640 1 -- 172.21.15.67:0/189030109 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f2f0c11e770 con 0x7f2f141f7440 2024-04-17T10:38:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.684+0000 7f2f11e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ef0066f60 0x7f2ef0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.688+0000 7f2f11e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ef0066f60 0x7f2ef0069420 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009fd0 tx=0x7f2f0400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.688+0000 7f2f0aa00640 1 -- 172.21.15.67:0/189030109 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2f0c0167c0 con 0x7f2f141f7440 2024-04-17T10:38:09.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.884+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f14074e30 con 0x7f2f141f7440 2024-04-17T10:38:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.884+0000 7f2f0aa00640 1 -- 172.21.15.67:0/189030109 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f2f0c0eef40 con 0x7f2f141f7440 2024-04-17T10:38:09.879 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:09.879 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ef0066f60 msgr2=0x7f2ef0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ef0066f60 0x7f2ef0069420 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f2f04009fd0 tx=0x7f2f0400b040 comp rx=0 tx=0).stop 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f141f7440 msgr2=0x7f2f141f78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f141f7440 0x7f2f141f78c0 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c0b0040 tx=0x7f2efc007f90 comp rx=0 tx=0).stop 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 shutdown_connections 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2ef0066f60 0x7f2ef0069420 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2f141f7440 0x7f2f141f78c0 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2f141589d0 0x7f2f14157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 --2- 172.21.15.67:0/189030109 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2ef8004d50 0x7f2f141f3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 >> 172.21.15.67:0/189030109 conn(0x7f2f140b34d0 msgr2=0x7f2f140710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 shutdown_connections 2024-04-17T10:38:09.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:09.888+0000 7f2f18e00640 1 -- 172.21.15.67:0/189030109 wait complete. 2024-04-17T10:38:10.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:10.303 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:11.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:11.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.464+0000 7f8aeca00640 1 Processor -- start 2024-04-17T10:38:11.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.464+0000 7f8aeca00640 1 -- start start 2024-04-17T10:38:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8aeca00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ae8145040 0x7f8ae806e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8aeca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae8071a20 0x7f8ae80cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8aeca00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae806e2f0 con 0x7f8ae806ef10 2024-04-17T10:38:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8aeca00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae80a86c0 con 0x7f8ae8145040 2024-04-17T10:38:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8aeca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae815bf10 con 0x7f8ae8071a20 2024-04-17T10:38:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8adfe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8ae806ef10 0x7f8ae8071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55838/0 (socket says 172.21.15.67:55838) 2024-04-17T10:38:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8adfe00640 1 -- 172.21.15.67:0/2898371766 learned_addr learned my addr 172.21.15.67:0/2898371766 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae5e00640 1 --2- 172.21.15.67:0/2898371766 >> v2:172.21.15.67:3300/0 conn(0x7f8ae8145040 0x7f8ae806e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae6800640 1 --2- 172.21.15.67:0/2898371766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae8071a20 0x7f8ae80cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae5e00640 1 -- 172.21.15.67:0/2898371766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae8071a20 msgr2=0x7f8ae80cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae5e00640 1 --2- 172.21.15.67:0/2898371766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae8071a20 0x7f8ae80cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae5e00640 1 -- 172.21.15.67:0/2898371766 >> v1:172.21.15.67:6789/0 conn(0x7f8ae806ef10 legacy=0x7f8ae8071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae5e00640 1 -- 172.21.15.67:0/2898371766 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae806d6e0 con 0x7f8ae8145040 2024-04-17T10:38:11.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae5400640 1 -- 172.21.15.67:0/2898371766 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589758633 0 0) 0x7f8ae806e2f0 con 0x7f8ae806ef10 2024-04-17T10:38:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.468+0000 7f8ae5e00640 1 --2- 172.21.15.67:0/2898371766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae806e9d0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80097c0 tx=0x7f8ad801ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3eecb1974f82bc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8ae5400640 1 -- 172.21.15.67:0/2898371766 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80b5070 con 0x7f8ae8145040 2024-04-17T10:38:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8ae5400640 1 -- 172.21.15.67:0/2898371766 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad80a8990 con 0x7f8ae8145040 2024-04-17T10:38:11.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8ae5400640 1 -- 172.21.15.67:0/2898371766 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80a8d60 con 0x7f8ae8145040 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- 172.21.15.67:0/2898371766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 msgr2=0x7f8ae806e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 --2- 172.21.15.67:0/2898371766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae806e9d0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80097c0 tx=0x7f8ad801ec90 comp rx=0 tx=0).stop 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- 172.21.15.67:0/2898371766 shutdown_connections 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 --2- 172.21.15.67:0/2898371766 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae8071a20 0x7f8ae80cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 --2- 172.21.15.67:0/2898371766 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae806e9d0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- 172.21.15.67:0/2898371766 >> 172.21.15.67:0/2898371766 conn(0x7f8ae806ac90 msgr2=0x7f8ae806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- 172.21.15.67:0/2898371766 shutdown_connections 2024-04-17T10:38:11.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- 172.21.15.67:0/2898371766 wait complete. 2024-04-17T10:38:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 Processor -- start 2024-04-17T10:38:11.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- start start 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae806ef10 0x7f8ae80ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ae8071a20 0x7f8ae815a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae815f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae806df80 con 0x7f8ae8145040 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae806dca0 con 0x7f8ae8071a20 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.472+0000 7f8aeca00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae806d370 con 0x7f8ae806ef10 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae815f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae5e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae806ef10 0x7f8ae80ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae815f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57494/0 (socket says 172.21.15.67:57494) 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 -- 172.21.15.67:0/1097241619 learned_addr learned my addr 172.21.15.67:0/1097241619 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8adfe00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ae8071a20 0x7f8ae815a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 -- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae806ef10 msgr2=0x7f8ae80ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae806ef10 0x7f8ae80ae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 -- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ae8071a20 msgr2=0x7f8ae815a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ae8071a20 0x7f8ae815a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 -- 172.21.15.67:0/1097241619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae80ad1d0 con 0x7f8ae8145040 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ae6800640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae815f410 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0007e30 tx=0x7f8ae000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ade000640 1 -- 172.21.15.67:0/1097241619 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0014620 con 0x7f8ae8145040 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ade000640 1 -- 172.21.15.67:0/1097241619 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae0019070 con 0x7f8ae8145040 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ade000640 1 -- 172.21.15.67:0/1097241619 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0014a00 con 0x7f8ae8145040 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae80ad4b0 con 0x7f8ae8145040 2024-04-17T10:38:11.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae80ada70 con 0x7f8ae8145040 2024-04-17T10:38:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae80adda0 con 0x7f8ae8145040 2024-04-17T10:38:11.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.476+0000 7f8ade000640 1 -- 172.21.15.67:0/1097241619 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8ae0014ba0 con 0x7f8ae8145040 2024-04-17T10:38:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.480+0000 7f8ade000640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ac4066f60 0x7f8ac4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.480+0000 7f8ade000640 1 -- 172.21.15.67:0/1097241619 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8ae0085990 con 0x7f8ae8145040 2024-04-17T10:38:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.480+0000 7f8ae5e00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ac4066f60 0x7f8ac4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:11.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.480+0000 7f8ae5e00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ac4066f60 0x7f8ac4069420 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80b2000 tx=0x7f8ad8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:11.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.484+0000 7f8ade000640 1 -- 172.21.15.67:0/1097241619 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8ae0056130 con 0x7f8ae8145040 2024-04-17T10:38:11.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.680+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ae8074e30 con 0x7f8ae8145040 2024-04-17T10:38:11.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.680+0000 7f8ade000640 1 -- 172.21.15.67:0/1097241619 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f8ae005a1d0 con 0x7f8ae8145040 2024-04-17T10:38:11.675 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:11.675 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:11.675 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ac4066f60 msgr2=0x7f8ac4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ac4066f60 0x7f8ac4069420 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80b2000 tx=0x7f8ad8008040 comp rx=0 tx=0).stop 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 msgr2=0x7f8ae815f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae815f410 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0007e30 tx=0x7f8ae000e4b0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 shutdown_connections 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ac4066f60 0x7f8ac4069420 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae8145040 0x7f8ae815f410 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8ae8071a20 0x7f8ae815a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 --2- 172.21.15.67:0/1097241619 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8ae806ef10 0x7f8ae80ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 >> 172.21.15.67:0/1097241619 conn(0x7f8ae806ac90 msgr2=0x7f8ae80b04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:11.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 shutdown_connections 2024-04-17T10:38:11.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:11.684+0000 7f8aeca00640 1 -- 172.21.15.67:0/1097241619 wait complete. 2024-04-17T10:38:12.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:12.099 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:13.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bcdc00640 1 Processor -- start 2024-04-17T10:38:13.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bcdc00640 1 -- start start 2024-04-17T10:38:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bcdc00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8bc806e6e0 0x7f8bc80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bcdc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc806eac0 0x7f8bc80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bcdc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc815ae40 con 0x7f8bc8143ae0 2024-04-17T10:38:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bcdc00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80cf430 con 0x7f8bc806e6e0 2024-04-17T10:38:13.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bcdc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8071440 con 0x7f8bc806eac0 2024-04-17T10:38:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bbe000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc806eac0 0x7f8bc80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bc6a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8bc806e6e0 0x7f8bc80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:13.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bc6a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f8bc806e6e0 0x7f8bc80ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:46022/0 (socket says 172.21.15.67:46022) 2024-04-17T10:38:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bc6a00640 1 -- 172.21.15.67:0/2228373854 learned_addr learned my addr 172.21.15.67:0/2228373854 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bbe000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc806eac0 0x7f8bc80ceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38182/0 (socket says 172.21.15.67:38182) 2024-04-17T10:38:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.259+0000 7f8bc7400640 1 --1- 172.21.15.67:0/2228373854 >> v1:172.21.15.86:6789/0 conn(0x7f8bc8143ae0 0x7f8bc8143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:34996/0 (socket says 172.21.15.67:34996) 2024-04-17T10:38:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6a00640 1 -- 172.21.15.67:0/2228373854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc806eac0 msgr2=0x7f8bc80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6a00640 1 --2- 172.21.15.67:0/2228373854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc806eac0 0x7f8bc80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6a00640 1 -- 172.21.15.67:0/2228373854 >> v1:172.21.15.86:6789/0 conn(0x7f8bc8143ae0 legacy=0x7f8bc8143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6a00640 1 -- 172.21.15.67:0/2228373854 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8071720 con 0x7f8bc806e6e0 2024-04-17T10:38:13.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6a00640 1 --2- 172.21.15.67:0/2228373854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806e6e0 0x7f8bc80ce9b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009870 tx=0x7f8bb801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef5e4e4af356ca59 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6000640 1 -- 172.21.15.67:0/2228373854 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80b1070 con 0x7f8bc806e6e0 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6000640 1 -- 172.21.15.67:0/2228373854 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb80ac600 con 0x7f8bc806e6e0 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.263+0000 7f8bc6000640 1 -- 172.21.15.67:0/2228373854 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80ac9d0 con 0x7f8bc806e6e0 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- 172.21.15.67:0/2228373854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806e6e0 msgr2=0x7f8bc80ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/2228373854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806e6e0 0x7f8bc80ce9b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8bb8009870 tx=0x7f8bb801c4a0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- 172.21.15.67:0/2228373854 shutdown_connections 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/2228373854 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc806eac0 0x7f8bc80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/2228373854 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806e6e0 0x7f8bc80ce9b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- 172.21.15.67:0/2228373854 >> 172.21.15.67:0/2228373854 conn(0x7f8bc806ac90 msgr2=0x7f8bc806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- 172.21.15.67:0/2228373854 shutdown_connections 2024-04-17T10:38:13.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- 172.21.15.67:0/2228373854 wait complete. 2024-04-17T10:38:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 Processor -- start 2024-04-17T10:38:13.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- start start 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc806e6e0 0x7f8bc8159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806eac0 0x7f8bc81f69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc8143ae0 0x7f8bc81f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc806dd00 con 0x7f8bc806e6e0 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc806db80 con 0x7f8bc806eac0 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bcdc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc806de80 con 0x7f8bc8143ae0 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc806e6e0 0x7f8bc8159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc806e6e0 0x7f8bc8159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57502/0 (socket says 172.21.15.67:57502) 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 -- 172.21.15.67:0/787277635 learned_addr learned my addr 172.21.15.67:0/787277635 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:13.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc7400640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc8143ae0 0x7f8bc81f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:13.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bbe000640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806eac0 0x7f8bc81f69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 -- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc8143ae0 msgr2=0x7f8bc81f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc8143ae0 0x7f8bc81f6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 -- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806eac0 msgr2=0x7f8bc81f69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806eac0 0x7f8bc81f69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 -- 172.21.15.67:0/787277635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc813ba00 con 0x7f8bc806e6e0 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.267+0000 7f8bc6a00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc806e6e0 0x7f8bc8159550 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f8bb801ab50 tx=0x7f8bb801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bbf400640 1 -- 172.21.15.67:0/787277635 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80bb020 con 0x7f8bc806e6e0 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bbf400640 1 -- 172.21.15.67:0/787277635 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb80b7420 con 0x7f8bc806e6e0 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc813bce0 con 0x7f8bc806e6e0 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bbf400640 1 -- 172.21.15.67:0/787277635 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb80b1040 con 0x7f8bc806e6e0 2024-04-17T10:38:13.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc813c2a0 con 0x7f8bc806e6e0 2024-04-17T10:38:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc8072d50 con 0x7f8bc806e6e0 2024-04-17T10:38:13.265 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bbf400640 1 -- 172.21.15.67:0/787277635 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8bb80be020 con 0x7f8bc806e6e0 2024-04-17T10:38:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bbf400640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ba4066f60 0x7f8ba4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bbf400640 1 -- 172.21.15.67:0/787277635 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8bb811efe0 con 0x7f8bc806e6e0 2024-04-17T10:38:13.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.271+0000 7f8bbe000640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ba4066f60 0x7f8ba4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.275+0000 7f8bbe000640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ba4066f60 0x7f8ba4069420 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8bc813a900 tx=0x7f8bb0009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:13.270 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.275+0000 7f8bbf400640 1 -- 172.21.15.67:0/787277635 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8bb80c5070 con 0x7f8bc806e6e0 2024-04-17T10:38:13.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.471+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bc8074e30 con 0x7f8bc806e6e0 2024-04-17T10:38:13.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.471+0000 7f8bbf400640 1 -- 172.21.15.67:0/787277635 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f8bb8016150 con 0x7f8bc806e6e0 2024-04-17T10:38:13.465 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:13.466 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:13.466 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:13.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ba4066f60 msgr2=0x7f8ba4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:13.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ba4066f60 0x7f8ba4069420 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8bc813a900 tx=0x7f8bb0009190 comp rx=0 tx=0).stop 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc806e6e0 msgr2=0x7f8bc8159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc806e6e0 0x7f8bc8159550 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f8bb801ab50 tx=0x7f8bb801ace0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 shutdown_connections 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8ba4066f60 0x7f8ba4069420 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8bc8143ae0 0x7f8bc81f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8bc806eac0 0x7f8bc81f69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 --2- 172.21.15.67:0/787277635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bc806e6e0 0x7f8bc8159550 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 >> 172.21.15.67:0/787277635 conn(0x7f8bc806ac90 msgr2=0x7f8bc815a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 shutdown_connections 2024-04-17T10:38:13.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:13.475+0000 7f8bcdc00640 1 -- 172.21.15.67:0/787277635 wait complete. 2024-04-17T10:38:13.890 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:13.890 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:14.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:15.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.051+0000 7f262ba00640 1 Processor -- start 2024-04-17T10:38:15.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.051+0000 7f262ba00640 1 -- start start 2024-04-17T10:38:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f262ba00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f26240cef30 0x7f26240aed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f262ba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f262415a490 0x7f262406daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f262ba00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f262415ac30 con 0x7f262406dfe0 2024-04-17T10:38:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f262ba00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f262415af10 con 0x7f26240cef30 2024-04-17T10:38:15.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f262ba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f262415b1f0 con 0x7f262415a490 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f2629000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f26240cef30 0x7f26240aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f261be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f262415a490 0x7f262406daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f2629a00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f262406dfe0 0x7f26240cdf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:35010/0 (socket says 172.21.15.67:35010) 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f261be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f262415a490 0x7f262406daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38192/0 (socket says 172.21.15.67:38192) 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f2629a00640 1 -- 172.21.15.67:0/3261026435 learned_addr learned my addr 172.21.15.67:0/3261026435 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f261be00640 1 -- 172.21.15.67:0/3261026435 >> v2:172.21.15.86:3300/0 conn(0x7f26240cef30 msgr2=0x7f26240aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f261be00640 1 --2- 172.21.15.67:0/3261026435 >> v2:172.21.15.86:3300/0 conn(0x7f26240cef30 0x7f26240aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f261be00640 1 -- 172.21.15.67:0/3261026435 >> v1:172.21.15.86:6789/0 conn(0x7f262406dfe0 legacy=0x7f26240cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f261be00640 1 -- 172.21.15.67:0/3261026435 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262406d580 con 0x7f262415a490 2024-04-17T10:38:15.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.055+0000 7f2629000640 1 --2- 172.21.15.67:0/3261026435 >> v2:172.21.15.86:3300/0 conn(0x7f26240cef30 0x7f26240aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:15.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f261be00640 1 --2- 172.21.15.67:0/3261026435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f262415a490 0x7f262406daa0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f260c0097c0 tx=0x7f260c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=328616313821482 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f2623e00640 1 -- 172.21.15.67:0/3261026435 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260c0b0070 con 0x7f262415a490 2024-04-17T10:38:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f2623e00640 1 -- 172.21.15.67:0/3261026435 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f260c0ab600 con 0x7f262415a490 2024-04-17T10:38:15.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f2623e00640 1 -- 172.21.15.67:0/3261026435 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f260c0ab9d0 con 0x7f262415a490 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- 172.21.15.67:0/3261026435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f262415a490 msgr2=0x7f262406daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 --2- 172.21.15.67:0/3261026435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f262415a490 0x7f262406daa0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f260c0097c0 tx=0x7f260c01c4a0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- 172.21.15.67:0/3261026435 shutdown_connections 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 --2- 172.21.15.67:0/3261026435 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f262415a490 0x7f262406daa0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 --2- 172.21.15.67:0/3261026435 >> v2:172.21.15.86:3300/0 conn(0x7f26240cef30 0x7f26240aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- 172.21.15.67:0/3261026435 >> 172.21.15.67:0/3261026435 conn(0x7f262406ac90 msgr2=0x7f262406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- 172.21.15.67:0/3261026435 shutdown_connections 2024-04-17T10:38:15.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- 172.21.15.67:0/3261026435 wait complete. 2024-04-17T10:38:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 Processor -- start 2024-04-17T10:38:15.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- start start 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f262406dfe0 0x7f2624159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26240cef30 0x7f2624145d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262415a490 0x7f2624148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2624150fe0 con 0x7f262406dfe0 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26240cf520 con 0x7f262415a490 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.059+0000 7f262ba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2624151160 con 0x7f26240cef30 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f262406dfe0 0x7f2624159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:15.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f262406dfe0 0x7f2624159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57510/0 (socket says 172.21.15.67:57510) 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 -- 172.21.15.67:0/3805203306 learned_addr learned my addr 172.21.15.67:0/3805203306 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f261be00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26240cef30 0x7f2624145d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629a00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262415a490 0x7f2624148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 -- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26240cef30 msgr2=0x7f2624145d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26240cef30 0x7f2624145d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 -- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262415a490 msgr2=0x7f2624148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262415a490 0x7f2624148690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 -- 172.21.15.67:0/3805203306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624146a50 con 0x7f262406dfe0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2629000640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f262406dfe0 0x7f2624159b50 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f261400ebe0 tx=0x7f261400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2621600640 1 -- 172.21.15.67:0/3805203306 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614017070 con 0x7f262406dfe0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2621600640 1 -- 172.21.15.67:0/3805203306 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f261400cc30 con 0x7f262406dfe0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2621600640 1 -- 172.21.15.67:0/3805203306 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614013470 con 0x7f262406dfe0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2624146d10 con 0x7f262406dfe0 2024-04-17T10:38:15.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2624158db0 con 0x7f262406dfe0 2024-04-17T10:38:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2624072d50 con 0x7f262406dfe0 2024-04-17T10:38:15.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.063+0000 7f2621600640 1 -- 172.21.15.67:0/3805203306 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f261401a070 con 0x7f262406dfe0 2024-04-17T10:38:15.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.067+0000 7f2621600640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2600066f60 0x7f2600069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:15.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.067+0000 7f2621600640 1 -- 172.21.15.67:0/3805203306 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f26140851a0 con 0x7f262406dfe0 2024-04-17T10:38:15.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.067+0000 7f261be00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2600066f60 0x7f2600069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:15.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.067+0000 7f261be00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2600066f60 0x7f2600069420 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f260c0ad000 tx=0x7f260c0a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:15.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.071+0000 7f2621600640 1 -- 172.21.15.67:0/3805203306 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f26140558a0 con 0x7f262406dfe0 2024-04-17T10:38:15.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.263+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2624074e30 con 0x7f262406dfe0 2024-04-17T10:38:15.259 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f2621600640 1 -- 172.21.15.67:0/3805203306 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f2614059940 con 0x7f262406dfe0 2024-04-17T10:38:15.259 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:15.259 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:15.259 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2600066f60 msgr2=0x7f2600069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2600066f60 0x7f2600069420 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f260c0ad000 tx=0x7f260c0a6000 comp rx=0 tx=0).stop 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f262406dfe0 msgr2=0x7f2624159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f262406dfe0 0x7f2624159b50 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f261400ebe0 tx=0x7f261400c520 comp rx=0 tx=0).stop 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 shutdown_connections 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2600066f60 0x7f2600069420 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f262415a490 0x7f2624148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26240cef30 0x7f2624145d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 --2- 172.21.15.67:0/3805203306 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f262406dfe0 0x7f2624159b50 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:15.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 >> 172.21.15.67:0/3805203306 conn(0x7f262406ac90 msgr2=0x7f2624071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:15.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 shutdown_connections 2024-04-17T10:38:15.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:15.267+0000 7f262ba00640 1 -- 172.21.15.67:0/3805203306 wait complete. 2024-04-17T10:38:15.683 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:15.683 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:16.684 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:16.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3844400640 1 Processor -- start 2024-04-17T10:38:16.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3844400640 1 -- start start 2024-04-17T10:38:16.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3844400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f383c0cc250 0x7f383c0cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3844400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c06b100 con 0x7f383c0cb260 2024-04-17T10:38:16.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3844400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c06b3e0 con 0x7f383c0cdcf0 2024-04-17T10:38:16.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3844400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f383c0cd150 con 0x7f383c0cc250 2024-04-17T10:38:16.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3841000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f383c0cb260 0x7f383c0cd5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55848/0 (socket says 172.21.15.67:55848) 2024-04-17T10:38:16.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3841000640 1 -- 172.21.15.67:0/3734539607 learned_addr learned my addr 172.21.15.67:0/3734539607 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:16.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f3841a00640 1 --2- 172.21.15.67:0/3734539607 >> v2:172.21.15.86:3300/0 conn(0x7f383c0cc250 0x7f383c0cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088271630 0 0) 0x7f383c06b100 con 0x7f383c0cb260 2024-04-17T10:38:16.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.843+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3824003530 con 0x7f383c0cb260 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 640699163 0 0) 0x7f3824003530 con 0x7f383c0cb260 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 >> v2:172.21.15.86:3300/0 conn(0x7f383c0cc250 msgr2=0x7f383c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 --2- 172.21.15.67:0/3734539607 >> v2:172.21.15.86:3300/0 conn(0x7f383c0cc250 0x7f383c0cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 >> v1:172.21.15.86:6789/0 conn(0x7f383c0cdcf0 legacy=0x7f383c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c06b6c0 con 0x7f383c0cb260 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f3828001680 con 0x7f383c0cb260 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 >> v1:172.21.15.67:6789/0 conn(0x7f383c0cb260 legacy=0x7f383c0cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f3824003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3824004810 0x7f3824004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38240051d0 0x7f382400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0cd150 con 0x7f38240051d0 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382400c0a0 con 0x7f38240039c0 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f382400c2a0 con 0x7f3824004810 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3841a00640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f3824003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3842400640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38240051d0 0x7f382400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3841000640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3824004810 0x7f3824004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3841a00640 1 -- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3824004810 msgr2=0x7f3824004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3841a00640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3824004810 0x7f3824004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3841a00640 1 -- 172.21.15.67:0/3734539607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38240051d0 msgr2=0x7f382400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3841a00640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f38240051d0 0x7f382400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3841a00640 1 -- 172.21.15.67:0/3734539607 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3824003530 con 0x7f38240039c0 2024-04-17T10:38:16.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.847+0000 7f3842400640 1 -- 172.21.15.67:0/3734539607 reap_dead start 2024-04-17T10:38:16.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3841a00640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f3824003e00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f383000e170 tx=0x7f383001cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=944f5e6ffc47000c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:16.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38300b2070 con 0x7f38240039c0 2024-04-17T10:38:16.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38300a5c30 con 0x7f38240039c0 2024-04-17T10:38:16.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f383be00640 1 -- 172.21.15.67:0/3734539607 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830009c10 con 0x7f38240039c0 2024-04-17T10:38:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 -- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 msgr2=0x7f3824003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f3824003e00 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f383000e170 tx=0x7f383001cc20 comp rx=0 tx=0).stop 2024-04-17T10:38:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 -- 172.21.15.67:0/3734539607 shutdown_connections 2024-04-17T10:38:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 --2- 172.21.15.67:0/3734539607 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f3824003e00 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 -- 172.21.15.67:0/3734539607 >> 172.21.15.67:0/3734539607 conn(0x7f383c0b22a0 msgr2=0x7f383c0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 -- 172.21.15.67:0/3734539607 shutdown_connections 2024-04-17T10:38:16.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 -- 172.21.15.67:0/3734539607 wait complete. 2024-04-17T10:38:16.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 Processor -- start 2024-04-17T10:38:16.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 -- start start 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f383c156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f383c1f48c0 0x7f383c0b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.851+0000 7f3844400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383c0b08d0 0x7f383c0c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3844400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0ce2a0 con 0x7f383c0b08d0 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3844400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c06aea0 con 0x7f38240039c0 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3844400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c0ce420 con 0x7f383c1f48c0 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f383c1f48c0 0x7f383c0b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f383c156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f383c1f48c0 0x7f383c0b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:38216/0 (socket says 172.21.15.67:38216) 2024-04-17T10:38:16.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 -- 172.21.15.67:0/3619990045 learned_addr learned my addr 172.21.15.67:0/3619990045 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:16.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3842400640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383c0b08d0 0x7f383c0c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 -- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 msgr2=0x7f383c156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f383c156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 -- 172.21.15.67:0/3619990045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383c0b08d0 msgr2=0x7f383c0c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383c0b08d0 0x7f383c0c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 -- 172.21.15.67:0/3619990045 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c157be0 con 0x7f383c1f48c0 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3841000640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f383c1f48c0 0x7f383c0b0390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f38280072c0 tx=0x7f3828007780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3839600640 1 -- 172.21.15.67:0/3619990045 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800f940 con 0x7f383c1f48c0 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3839600640 1 -- 172.21.15.67:0/3619990045 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3828018070 con 0x7f383c1f48c0 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3839600640 1 -- 172.21.15.67:0/3619990045 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382800fd60 con 0x7f383c1f48c0 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c157ec0 con 0x7f383c1f48c0 2024-04-17T10:38:16.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c158480 con 0x7f383c1f48c0 2024-04-17T10:38:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c09e900 con 0x7f383c1f48c0 2024-04-17T10:38:16.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.855+0000 7f3839600640 1 -- 172.21.15.67:0/3619990045 <== mon.2 v2:172.21.15.86:3301/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f382801d460 con 0x7f383c1f48c0 2024-04-17T10:38:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.859+0000 7f3839600640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f381c066f90 0x7f381c069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.859+0000 7f3839600640 1 -- 172.21.15.67:0/3619990045 <== mon.2 v2:172.21.15.86:3301/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f3828085280 con 0x7f383c1f48c0 2024-04-17T10:38:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.859+0000 7f3841a00640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f381c066f90 0x7f381c069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:16.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.859+0000 7f3841a00640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f381c066f90 0x7f381c069450 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f38300af000 tx=0x7f38300a7040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:16.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:16.863+0000 7f3839600640 1 -- 172.21.15.67:0/3619990045 <== mon.2 v2:172.21.15.86:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3828055980 con 0x7f383c1f48c0 2024-04-17T10:38:17.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.055+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f383c074e30 con 0x7f383c1f48c0 2024-04-17T10:38:17.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3839600640 1 -- 172.21.15.67:0/3619990045 <== mon.2 v2:172.21.15.86:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3828059a20 con 0x7f383c1f48c0 2024-04-17T10:38:17.051 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:17.051 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:17.052 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:17.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f381c066f90 msgr2=0x7f381c069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:17.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f381c066f90 0x7f381c069450 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f38300af000 tx=0x7f38300a7040 comp rx=0 tx=0).stop 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f383c1f48c0 msgr2=0x7f383c0b0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f383c1f48c0 0x7f383c0b0390 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f38280072c0 tx=0x7f3828007780 comp rx=0 tx=0).stop 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 shutdown_connections 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f381c066f90 0x7f381c069450 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f383c0b08d0 0x7f383c0c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f383c1f48c0 0x7f383c0b0390 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 --2- 172.21.15.67:0/3619990045 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f38240039c0 0x7f383c156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 >> 172.21.15.67:0/3619990045 conn(0x7f383c0b22a0 msgr2=0x7f383c073880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 shutdown_connections 2024-04-17T10:38:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:17.059+0000 7f3844400640 1 -- 172.21.15.67:0/3619990045 wait complete. 2024-04-17T10:38:17.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:17.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:18.477 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.635+0000 7f3283200640 1 Processor -- start 2024-04-17T10:38:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.635+0000 7f3283200640 1 -- start start 2024-04-17T10:38:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3283200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f32840d03a0 0x7f328415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3283200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3283200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32840cde20 con 0x7f3284099a50 2024-04-17T10:38:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3283200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32840cf300 con 0x7f32840d03a0 2024-04-17T10:38:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3283200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840cf5e0 con 0x7f328409a740 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3281e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f3284099a50 0x7f3284099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55854/0 (socket says 172.21.15.67:55854) 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3281e00640 1 -- 172.21.15.67:0/219692237 learned_addr learned my addr 172.21.15.67:0/219692237 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3279400640 1 --2- 172.21.15.67:0/219692237 >> v2:172.21.15.67:3300/0 conn(0x7f32840d03a0 0x7f328415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3282800640 1 --2- 172.21.15.67:0/219692237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3281400640 1 -- 172.21.15.67:0/219692237 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776528664 0 0) 0x7f32840cde20 con 0x7f3284099a50 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3281400640 1 -- 172.21.15.67:0/219692237 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3268003530 con 0x7f3284099a50 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3279400640 1 -- 172.21.15.67:0/219692237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 msgr2=0x7f32840cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3279400640 1 --2- 172.21.15.67:0/219692237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840cc1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3279400640 1 -- 172.21.15.67:0/219692237 >> v1:172.21.15.67:6789/0 conn(0x7f3284099a50 legacy=0x7f3284099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3279400640 1 -- 172.21.15.67:0/219692237 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840cf8c0 con 0x7f32840d03a0 2024-04-17T10:38:18.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.639+0000 7f3282800640 1 --2- 172.21.15.67:0/219692237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3279400640 1 --2- 172.21.15.67:0/219692237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840d03a0 0x7f328415c7a0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f326c009d90 tx=0x7f326c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bca0566692927d73 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3281400640 1 -- 172.21.15.67:0/219692237 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0b2070 con 0x7f32840d03a0 2024-04-17T10:38:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3281400640 1 -- 172.21.15.67:0/219692237 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c0a5830 con 0x7f32840d03a0 2024-04-17T10:38:18.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3281400640 1 -- 172.21.15.67:0/219692237 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0a5c00 con 0x7f32840d03a0 2024-04-17T10:38:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 -- 172.21.15.67:0/219692237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840d03a0 msgr2=0x7f328415c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:18.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 --2- 172.21.15.67:0/219692237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840d03a0 0x7f328415c7a0 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f326c009d90 tx=0x7f326c01c970 comp rx=0 tx=0).stop 2024-04-17T10:38:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 -- 172.21.15.67:0/219692237 shutdown_connections 2024-04-17T10:38:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 --2- 172.21.15.67:0/219692237 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 --2- 172.21.15.67:0/219692237 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f32840d03a0 0x7f328415c7a0 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 -- 172.21.15.67:0/219692237 >> 172.21.15.67:0/219692237 conn(0x7f32840d1690 msgr2=0x7f32840d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 -- 172.21.15.67:0/219692237 shutdown_connections 2024-04-17T10:38:18.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 -- 172.21.15.67:0/219692237 wait complete. 2024-04-17T10:38:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 Processor -- start 2024-04-17T10:38:18.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.643+0000 7f3283200640 1 -- start start 2024-04-17T10:38:18.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3284099a50 0x7f32841539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32840d03a0 0x7f3284152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32841686b0 con 0x7f3284099a50 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840cd180 con 0x7f32840d03a0 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3284168830 con 0x7f328409a740 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3284099a50 0x7f32841539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3284099a50 0x7f32841539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57556/0 (socket says 172.21.15.67:57556) 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 -- 172.21.15.67:0/2170938374 learned_addr learned my addr 172.21.15.67:0/2170938374 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3279400640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3282800640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32840d03a0 0x7f3284152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 -- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 msgr2=0x7f32840b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:18.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 -- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32840d03a0 msgr2=0x7f3284152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32840d03a0 0x7f3284152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 -- 172.21.15.67:0/2170938374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3284154230 con 0x7f3284099a50 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3282800640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32840d03a0 0x7f3284152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3279400640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3281e00640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3284099a50 0x7f32841539e0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f327400c3c0 tx=0x7f327400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f327aa00640 1 -- 172.21.15.67:0/2170938374 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274017070 con 0x7f3284099a50 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f327aa00640 1 -- 172.21.15.67:0/2170938374 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3274005cf0 con 0x7f3284099a50 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3284154510 con 0x7f3284099a50 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f327aa00640 1 -- 172.21.15.67:0/2170938374 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32740072f0 con 0x7f3284099a50 2024-04-17T10:38:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3284154ad0 con 0x7f3284099a50 2024-04-17T10:38:18.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3284074dc0 con 0x7f3284099a50 2024-04-17T10:38:18.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.647+0000 7f327aa00640 1 -- 172.21.15.67:0/2170938374 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f327401a070 con 0x7f3284099a50 2024-04-17T10:38:18.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.651+0000 7f327aa00640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3260066f60 0x7f3260069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.651+0000 7f327aa00640 1 -- 172.21.15.67:0/2170938374 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f32740846b0 con 0x7f3284099a50 2024-04-17T10:38:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.651+0000 7f3279400640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3260066f60 0x7f3260069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:18.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.651+0000 7f3279400640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3260066f60 0x7f3260069420 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f326c0af000 tx=0x7f326c0a7070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:18.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.655+0000 7f327aa00640 1 -- 172.21.15.67:0/2170938374 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3274055400 con 0x7f3284099a50 2024-04-17T10:38:18.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32840d24a0 con 0x7f3284099a50 2024-04-17T10:38:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f327aa00640 1 -- 172.21.15.67:0/2170938374 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3274054d00 con 0x7f3284099a50 2024-04-17T10:38:18.844 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:18.844 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:18.844 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:18.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3260066f60 msgr2=0x7f3260069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:18.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3260066f60 0x7f3260069420 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f326c0af000 tx=0x7f326c0a7070 comp rx=0 tx=0).stop 2024-04-17T10:38:18.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3284099a50 msgr2=0x7f32841539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:18.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3284099a50 0x7f32841539e0 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f327400c3c0 tx=0x7f327400c880 comp rx=0 tx=0).stop 2024-04-17T10:38:18.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 shutdown_connections 2024-04-17T10:38:18.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3260066f60 0x7f3260069420 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f32840d03a0 0x7f3284152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f328409a740 0x7f32840b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 --2- 172.21.15.67:0/2170938374 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3284099a50 0x7f32841539e0 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:18.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 >> 172.21.15.67:0/2170938374 conn(0x7f32840d1690 msgr2=0x7f3284099e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:18.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 shutdown_connections 2024-04-17T10:38:18.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:18.851+0000 7f3283200640 1 -- 172.21.15.67:0/2170938374 wait complete. 2024-04-17T10:38:19.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:19.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:20.269 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:20.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7016600640 1 Processor -- start 2024-04-17T10:38:20.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7016600640 1 -- start start 2024-04-17T10:38:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7016600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 0x7f7010099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7016600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701009a840 con 0x7f7010099f90 2024-04-17T10:38:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7016600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701009ab20 con 0x7f701009a3b0 2024-04-17T10:38:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7016600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701009b170 con 0x7f7010160f60 2024-04-17T10:38:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f7010099f90 0x7f70100cfb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55286/0 (socket says 172.21.15.67:55286) 2024-04-17T10:38:20.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700fe00640 1 -- 172.21.15.67:0/3508570799 learned_addr learned my addr 172.21.15.67:0/3508570799 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015200640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 0x7f7010099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927715943 0 0) 0x7f701009a840 con 0x7f7010099f90 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff8003530 con 0x7f7010099f90 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948916042 0 0) 0x7f6ff8003530 con 0x7f7010099f90 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 msgr2=0x7f7010099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 0x7f7010099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 >> v1:172.21.15.86:6789/0 conn(0x7f701009a3b0 legacy=0x7f701006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701009ae00 con 0x7f7010099f90 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f6ffc001680 con 0x7f7010099f90 2024-04-17T10:38:20.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 >> v1:172.21.15.67:6789/0 conn(0x7f7010099f90 legacy=0x7f70100cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff80039f0 0x7f6ff8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 0x7f6ff8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8004d50 0x7f6ff80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701009b170 con 0x7f6ff8004d50 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff800b9d0 con 0x7f6ff80039f0 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ff800bbd0 con 0x7f7010160f60 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f700fe00640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 0x7f6ff8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015c00640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8004d50 0x7f6ff80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015200640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff80039f0 0x7f6ff8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015c00640 1 -- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 msgr2=0x7f6ff8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015c00640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f7010160f60 0x7f6ff8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015c00640 1 -- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff80039f0 msgr2=0x7f6ff8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015c00640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff80039f0 0x7f6ff8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015c00640 1 -- 172.21.15.67:0/3508570799 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff8003530 con 0x7f6ff8004d50 2024-04-17T10:38:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.431+0000 7f7015200640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ff80039f0 0x7f6ff8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7015c00640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8004d50 0x7f6ff80051d0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f70100b0f80 tx=0x7f700801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8726be88e4b584c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:20.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70080b1070 con 0x7f6ff8004d50 2024-04-17T10:38:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70080a4a60 con 0x7f6ff8004d50 2024-04-17T10:38:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f700f400640 1 -- 172.21.15.67:0/3508570799 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008001b60 con 0x7f6ff8004d50 2024-04-17T10:38:20.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7016600640 1 -- 172.21.15.67:0/3508570799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8004d50 msgr2=0x7f6ff80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7016600640 1 --2- 172.21.15.67:0/3508570799 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ff8004d50 0x7f6ff80051d0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f70100b0f80 tx=0x7f700801c520 comp rx=0 tx=0).stop 2024-04-17T10:38:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7015c00640 1 -- 172.21.15.67:0/3508570799 reap_dead start 2024-04-17T10:38:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7016600640 1 -- 172.21.15.67:0/3508570799 shutdown_connections 2024-04-17T10:38:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7016600640 1 -- 172.21.15.67:0/3508570799 >> 172.21.15.67:0/3508570799 conn(0x7f70100d1630 msgr2=0x7f70100d1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7016600640 1 -- 172.21.15.67:0/3508570799 shutdown_connections 2024-04-17T10:38:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.435+0000 7f7016600640 1 -- 172.21.15.67:0/3508570799 wait complete. 2024-04-17T10:38:20.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 Processor -- start 2024-04-17T10:38:20.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 -- start start 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff8004d50 0x7f7010147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f701006bd60 0x7f701006c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70101f4150 0x7f70101f45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100d05d0 con 0x7f701006bd60 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100d0450 con 0x7f70101f4150 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100d0750 con 0x7f6ff8004d50 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f701006bd60 0x7f701006c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f701006bd60 0x7f701006c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38186/0 (socket says 172.21.15.67:38186) 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 -- 172.21.15.67:0/2402747725 learned_addr learned my addr 172.21.15.67:0/2402747725 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:20.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7015c00640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70101f4150 0x7f70101f45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7015200640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff8004d50 0x7f7010147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 -- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff8004d50 msgr2=0x7f7010147bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:20.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff8004d50 0x7f7010147bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 -- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70101f4150 msgr2=0x7f70101f45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:20.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70101f4150 0x7f70101f45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 -- 172.21.15.67:0/2402747725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70101f4eb0 con 0x7f701006bd60 2024-04-17T10:38:20.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700fe00640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f701006bd60 0x7f701006c1e0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0017f0 tx=0x7f6ffc012860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700cc00640 1 -- 172.21.15.67:0/2402747725 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc01c070 con 0x7f701006bd60 2024-04-17T10:38:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700cc00640 1 -- 172.21.15.67:0/2402747725 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc014810 con 0x7f701006bd60 2024-04-17T10:38:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f700cc00640 1 -- 172.21.15.67:0/2402747725 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc014c00 con 0x7f701006bd60 2024-04-17T10:38:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70101486b0 con 0x7f701006bd60 2024-04-17T10:38:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.439+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7010148c70 con 0x7f701006bd60 2024-04-17T10:38:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.443+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7010148e70 con 0x7f701006bd60 2024-04-17T10:38:20.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.443+0000 7f700cc00640 1 -- 172.21.15.67:0/2402747725 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6ffc01f070 con 0x7f701006bd60 2024-04-17T10:38:20.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.443+0000 7f700cc00640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ff0066f60 0x7f6ff0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:20.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.443+0000 7f700cc00640 1 -- 172.21.15.67:0/2402747725 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6ffc087b70 con 0x7f701006bd60 2024-04-17T10:38:20.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.443+0000 7f7015200640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ff0066f60 0x7f6ff0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:20.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.447+0000 7f7015200640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ff0066f60 0x7f6ff0069420 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7004009d80 tx=0x7f7004009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:20.442 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.447+0000 7f700cc00640 1 -- 172.21.15.67:0/2402747725 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f7010148e70 con 0x7f701006bd60 2024-04-17T10:38:20.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.643+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7010074dc0 con 0x7f701006bd60 2024-04-17T10:38:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.643+0000 7f700cc00640 1 -- 172.21.15.67:0/2402747725 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6ffc0583a0 con 0x7f701006bd60 2024-04-17T10:38:20.638 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:20.638 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:20.638 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ff0066f60 msgr2=0x7f6ff0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ff0066f60 0x7f6ff0069420 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f7004009d80 tx=0x7f7004009190 comp rx=0 tx=0).stop 2024-04-17T10:38:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f701006bd60 msgr2=0x7f701006c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f701006bd60 0x7f701006c1e0 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0017f0 tx=0x7f6ffc012860 comp rx=0 tx=0).stop 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 shutdown_connections 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ff0066f60 0x7f6ff0069420 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f70101f4150 0x7f70101f45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f701006bd60 0x7f701006c1e0 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 --2- 172.21.15.67:0/2402747725 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ff8004d50 0x7f7010147bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 >> 172.21.15.67:0/2402747725 conn(0x7f70100d1630 msgr2=0x7f7010157ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 shutdown_connections 2024-04-17T10:38:20.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:20.647+0000 7f7016600640 1 -- 172.21.15.67:0/2402747725 wait complete. 2024-04-17T10:38:21.063 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:21.064 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:22.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:22.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.227+0000 7f0071a00640 1 Processor -- start 2024-04-17T10:38:22.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.227+0000 7f0071a00640 1 -- start start 2024-04-17T10:38:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0071a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f006c06c410 0x7f006c06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0071a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c0ad8f0 con 0x7f006c071c60 2024-04-17T10:38:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0071a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c0adbd0 con 0x7f006c15dbc0 2024-04-17T10:38:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0071a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c15dfa0 con 0x7f006c06c410 2024-04-17T10:38:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0062000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f006c06c410 0x7f006c06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f006b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f006c071c60 0x7f006c1676d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55300/0 (socket says 172.21.15.67:55300) 2024-04-17T10:38:22.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f006b400640 1 -- 172.21.15.67:0/2677446632 learned_addr learned my addr 172.21.15.67:0/2677446632 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:22.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0062000640 1 -- 172.21.15.67:0/2677446632 >> v1:172.21.15.86:6789/0 conn(0x7f006c15dbc0 legacy=0x7f006c06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:22.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0062000640 1 -- 172.21.15.67:0/2677446632 >> v1:172.21.15.67:6789/0 conn(0x7f006c071c60 legacy=0x7f006c1676d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:22.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.231+0000 7f0062000640 1 -- 172.21.15.67:0/2677446632 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c0adeb0 con 0x7f006c06c410 2024-04-17T10:38:22.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0062000640 1 --2- 172.21.15.67:0/2677446632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c06c410 0x7f006c06f560 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f00540097c0 tx=0x7f005401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4743fc2563af630a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:22.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f006a000640 1 -- 172.21.15.67:0/2677446632 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00540b0070 con 0x7f006c06c410 2024-04-17T10:38:22.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f006a000640 1 -- 172.21.15.67:0/2677446632 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00540ab600 con 0x7f006c06c410 2024-04-17T10:38:22.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f006a000640 1 -- 172.21.15.67:0/2677446632 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00540ab9d0 con 0x7f006c06c410 2024-04-17T10:38:22.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- 172.21.15.67:0/2677446632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c06c410 msgr2=0x7f006c06f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:22.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 --2- 172.21.15.67:0/2677446632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c06c410 0x7f006c06f560 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f00540097c0 tx=0x7f005401c410 comp rx=0 tx=0).stop 2024-04-17T10:38:22.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- 172.21.15.67:0/2677446632 shutdown_connections 2024-04-17T10:38:22.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 --2- 172.21.15.67:0/2677446632 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c06c410 0x7f006c06f560 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:22.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- 172.21.15.67:0/2677446632 >> 172.21.15.67:0/2677446632 conn(0x7f006c0af670 msgr2=0x7f006c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:22.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- 172.21.15.67:0/2677446632 shutdown_connections 2024-04-17T10:38:22.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- 172.21.15.67:0/2677446632 wait complete. 2024-04-17T10:38:22.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 Processor -- start 2024-04-17T10:38:22.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- start start 2024-04-17T10:38:22.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f006c06c410 0x7f006c1489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:22.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f006c071c60 0x7f006c146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:22.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c15dbc0 0x7f006c154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:22.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c16b820 con 0x7f006c15dbc0 2024-04-17T10:38:22.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c06af70 con 0x7f006c071c60 2024-04-17T10:38:22.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.235+0000 7f0071a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c16b9a0 con 0x7f006c06c410 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c15dbc0 0x7f006c154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c15dbc0 0x7f006c154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38198/0 (socket says 172.21.15.67:38198) 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 -- 172.21.15.67:0/2971306481 learned_addr learned my addr 172.21.15.67:0/2971306481 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0062000640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f006c071c60 0x7f006c146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006aa00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f006c06c410 0x7f006c1489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 -- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f006c06c410 msgr2=0x7f006c1489e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f006c06c410 0x7f006c1489e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 -- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f006c071c60 msgr2=0x7f006c146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f006c071c60 0x7f006c146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 -- 172.21.15.67:0/2971306481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c14a720 con 0x7f006c15dbc0 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006aa00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f006c06c410 0x7f006c1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f006b400640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c15dbc0 0x7f006c154df0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f006400cbc0 tx=0x7f006400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0063400640 1 -- 172.21.15.67:0/2971306481 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064017070 con 0x7f006c15dbc0 2024-04-17T10:38:22.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0063400640 1 -- 172.21.15.67:0/2971306481 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f006400ad40 con 0x7f006c15dbc0 2024-04-17T10:38:22.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0063400640 1 -- 172.21.15.67:0/2971306481 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064012600 con 0x7f006c15dbc0 2024-04-17T10:38:22.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c14aa00 con 0x7f006c15dbc0 2024-04-17T10:38:22.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c14afc0 con 0x7f006c15dbc0 2024-04-17T10:38:22.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c0b32c0 con 0x7f006c15dbc0 2024-04-17T10:38:22.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.239+0000 7f0063400640 1 -- 172.21.15.67:0/2971306481 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f006401a070 con 0x7f006c15dbc0 2024-04-17T10:38:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.243+0000 7f0063400640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0048066f60 0x7f0048069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.243+0000 7f0063400640 1 -- 172.21.15.67:0/2971306481 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f0064084ef0 con 0x7f006c15dbc0 2024-04-17T10:38:22.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.243+0000 7f006aa00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0048066f60 0x7f0048069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:22.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.243+0000 7f006aa00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0048066f60 0x7f0048069420 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f005c000f70 tx=0x7f005c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:22.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.247+0000 7f0063400640 1 -- 172.21.15.67:0/2971306481 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0064055680 con 0x7f006c15dbc0 2024-04-17T10:38:22.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.443+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f006c074e30 con 0x7f006c15dbc0 2024-04-17T10:38:22.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.443+0000 7f0063400640 1 -- 172.21.15.67:0/2971306481 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f0064059720 con 0x7f006c15dbc0 2024-04-17T10:38:22.435 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:22.435 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:22.435 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0048066f60 msgr2=0x7f0048069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0048066f60 0x7f0048069420 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f005c000f70 tx=0x7f005c009040 comp rx=0 tx=0).stop 2024-04-17T10:38:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c15dbc0 msgr2=0x7f006c154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c15dbc0 0x7f006c154df0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f006400cbc0 tx=0x7f006400a430 comp rx=0 tx=0).stop 2024-04-17T10:38:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 shutdown_connections 2024-04-17T10:38:22.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0048066f60 0x7f0048069420 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:22.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f006c15dbc0 0x7f006c154df0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:22.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f006c071c60 0x7f006c146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:22.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 --2- 172.21.15.67:0/2971306481 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f006c06c410 0x7f006c1489e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:22.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 >> 172.21.15.67:0/2971306481 conn(0x7f006c0af670 msgr2=0x7f006c1522d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:22.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 shutdown_connections 2024-04-17T10:38:22.438 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:22.447+0000 7f0071a00640 1 -- 172.21.15.67:0/2971306481 wait complete. 2024-04-17T10:38:22.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:22.859 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:23.861 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.023+0000 7fe388400640 1 Processor -- start 2024-04-17T10:38:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.023+0000 7fe388400640 1 -- start start 2024-04-17T10:38:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.023+0000 7fe388400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fe380071310 0x7fe380071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.023+0000 7fe388400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38006e2f0 con 0x7fe380145040 2024-04-17T10:38:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.023+0000 7fe388400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38006d6e0 con 0x7fe3800cc300 2024-04-17T10:38:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.023+0000 7fe388400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3800a86c0 con 0x7fe380071310 2024-04-17T10:38:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe385a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe380145040 0x7fe38006e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55304/0 (socket says 172.21.15.67:55304) 2024-04-17T10:38:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe385a00640 1 -- 172.21.15.67:0/73136516 learned_addr learned my addr 172.21.15.67:0/73136516 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe385000640 1 --2- 172.21.15.67:0/73136516 >> v2:172.21.15.86:3300/0 conn(0x7fe380071310 0x7fe380071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065221298 0 0) 0x7fe38006e2f0 con 0x7fe380145040 2024-04-17T10:38:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe368003530 con 0x7fe380145040 2024-04-17T10:38:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1150068991 0 0) 0x7fe368003530 con 0x7fe380145040 2024-04-17T10:38:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 >> v2:172.21.15.86:3300/0 conn(0x7fe380071310 msgr2=0x7fe380071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 --2- 172.21.15.67:0/73136516 >> v2:172.21.15.86:3300/0 conn(0x7fe380071310 0x7fe380071770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 >> v1:172.21.15.86:6789/0 conn(0x7fe3800cc300 legacy=0x7fe3800cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38006d9c0 con 0x7fe380145040 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fe374001680 con 0x7fe380145040 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe385000640 1 --2- 172.21.15.67:0/73136516 >> v2:172.21.15.86:3300/0 conn(0x7fe380071310 0x7fe380071770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 >> v1:172.21.15.67:6789/0 conn(0x7fe380145040 legacy=0x7fe38006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680039c0 0x7fe368003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe368004810 0x7fe368004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36800c050 con 0x7fe3680051d0 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36800c200 con 0x7fe368004810 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36800c400 con 0x7fe3680039c0 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe385a00640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680039c0 0x7fe368003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe386400640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.018 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe385000640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe368004810 0x7fe368004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe386400640 1 -- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680039c0 msgr2=0x7fe368003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe386400640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680039c0 0x7fe368003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe386400640 1 -- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe368004810 msgr2=0x7fe368004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe386400640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe368004810 0x7fe368004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe386400640 1 -- 172.21.15.67:0/73136516 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe368003530 con 0x7fe3680051d0 2024-04-17T10:38:24.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.027+0000 7fe385a00640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680039c0 0x7fe368003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe386400640 1 -- 172.21.15.67:0/73136516 reap_dead start 2024-04-17T10:38:24.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe386400640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe378001e70 tx=0x7fe37801c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71037c7373f05bb server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe374007d00 con 0x7fe3680051d0 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe378003bc0 con 0x7fe3680051d0 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe37fe00640 1 -- 172.21.15.67:0/73136516 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378001a90 con 0x7fe3680051d0 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 -- 172.21.15.67:0/73136516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3680051d0 msgr2=0x7fe36800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7fe378001e70 tx=0x7fe37801c780 comp rx=0 tx=0).stop 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 -- 172.21.15.67:0/73136516 shutdown_connections 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 --2- 172.21.15.67:0/73136516 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 -- 172.21.15.67:0/73136516 >> 172.21.15.67:0/73136516 conn(0x7fe38006ac90 msgr2=0x7fe38006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 -- 172.21.15.67:0/73136516 shutdown_connections 2024-04-17T10:38:24.021 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 -- 172.21.15.67:0/73136516 wait complete. 2024-04-17T10:38:24.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 Processor -- start 2024-04-17T10:38:24.022 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.031+0000 7fe388400640 1 -- start start 2024-04-17T10:38:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680051d0 0x7fe380130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3800c46e0 0x7fe3800b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3800b3cf0 0x7fe3801332c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38006df80 con 0x7fe3800b3cf0 2024-04-17T10:38:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38015bf10 con 0x7fe3800c46e0 2024-04-17T10:38:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe380145420 con 0x7fe3680051d0 2024-04-17T10:38:24.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3800b3cf0 0x7fe3801332c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3800b3cf0 0x7fe3801332c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38216/0 (socket says 172.21.15.67:38216) 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 -- 172.21.15.67:0/4109280859 learned_addr learned my addr 172.21.15.67:0/4109280859 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe385a00640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680051d0 0x7fe380130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe385000640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3800c46e0 0x7fe3800b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 -- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680051d0 msgr2=0x7fe380130c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680051d0 0x7fe380130c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 -- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3800c46e0 msgr2=0x7fe3800b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3800c46e0 0x7fe3800b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 -- 172.21.15.67:0/4109280859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380131330 con 0x7fe3800b3cf0 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe385000640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3800c46e0 0x7fe3800b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:24.025 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe386400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3800b3cf0 0x7fe3801332c0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fe37801cc60 tx=0x7fe368005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe37d600640 1 -- 172.21.15.67:0/4109280859 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378001a90 con 0x7fe3800b3cf0 2024-04-17T10:38:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe37d600640 1 -- 172.21.15.67:0/4109280859 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe378005810 con 0x7fe3800b3cf0 2024-04-17T10:38:24.026 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe380144870 con 0x7fe3800b3cf0 2024-04-17T10:38:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe37d600640 1 -- 172.21.15.67:0/4109280859 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378007dc0 con 0x7fe3800b3cf0 2024-04-17T10:38:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe380141aa0 con 0x7fe3800b3cf0 2024-04-17T10:38:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.035+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe380072d50 con 0x7fe3800b3cf0 2024-04-17T10:38:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.039+0000 7fe37d600640 1 -- 172.21.15.67:0/4109280859 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe3780a6480 con 0x7fe3800b3cf0 2024-04-17T10:38:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.039+0000 7fe37d600640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe360066f90 0x7fe360069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:24.027 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.039+0000 7fe37d600640 1 -- 172.21.15.67:0/4109280859 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fe37811e9c0 con 0x7fe3800b3cf0 2024-04-17T10:38:24.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.039+0000 7fe385a00640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe360066f90 0x7fe360069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:24.028 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.039+0000 7fe385a00640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe360066f90 0x7fe360069450 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe374000c10 tx=0x7fe374007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:24.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.043+0000 7fe37d600640 1 -- 172.21.15.67:0/4109280859 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe3780b8060 con 0x7fe3800b3cf0 2024-04-17T10:38:24.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.243+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe380131500 con 0x7fe3800b3cf0 2024-04-17T10:38:24.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.243+0000 7fe37d600640 1 -- 172.21.15.67:0/4109280859 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fe378016490 con 0x7fe3800b3cf0 2024-04-17T10:38:24.235 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:24.235 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:24.235 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe360066f90 msgr2=0x7fe360069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe360066f90 0x7fe360069450 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe374000c10 tx=0x7fe374007900 comp rx=0 tx=0).stop 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3800b3cf0 msgr2=0x7fe3801332c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3800b3cf0 0x7fe3801332c0 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7fe37801cc60 tx=0x7fe368005840 comp rx=0 tx=0).stop 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 shutdown_connections 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe360066f90 0x7fe360069450 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3800b3cf0 0x7fe3801332c0 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe3800c46e0 0x7fe3800b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 --2- 172.21.15.67:0/4109280859 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe3680051d0 0x7fe380130c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:24.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 >> 172.21.15.67:0/4109280859 conn(0x7fe38006ac90 msgr2=0x7fe3800aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:24.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 shutdown_connections 2024-04-17T10:38:24.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:24.247+0000 7fe388400640 1 -- 172.21.15.67:0/4109280859 wait complete. 2024-04-17T10:38:24.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:24.663 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:25.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:25.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.827+0000 7f1c6c000640 1 Processor -- start 2024-04-17T10:38:25.818 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.827+0000 7f1c6c000640 1 -- start start 2024-04-17T10:38:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6406c410 0x7f1c6406f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6c000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1c64071c60 0x7f1c640720c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6c000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640ad8f0 con 0x7f1c6415dbc0 2024-04-17T10:38:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6c000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6415dfa0 con 0x7f1c64071c60 2024-04-17T10:38:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6406faa0 con 0x7f1c6406c410 2024-04-17T10:38:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c69600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1c6415dbc0 0x7f1c6406db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55306/0 (socket says 172.21.15.67:55306) 2024-04-17T10:38:25.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c69600640 1 -- 172.21.15.67:0/1844373190 learned_addr learned my addr 172.21.15.67:0/1844373190 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6a000640 1 --2- 172.21.15.67:0/1844373190 >> v2:172.21.15.67:3300/0 conn(0x7f1c64071c60 0x7f1c640720c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c68c00640 1 --2- 172.21.15.67:0/1844373190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6406c410 0x7f1c6406f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6a000640 1 -- 172.21.15.67:0/1844373190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6406c410 msgr2=0x7f1c6406f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6a000640 1 --2- 172.21.15.67:0/1844373190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6406c410 0x7f1c6406f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6a000640 1 -- 172.21.15.67:0/1844373190 >> v1:172.21.15.67:6789/0 conn(0x7f1c6415dbc0 legacy=0x7f1c6406db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c6a000640 1 -- 172.21.15.67:0/1844373190 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640adbd0 con 0x7f1c64071c60 2024-04-17T10:38:25.821 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.831+0000 7f1c63e00640 1 -- 172.21.15.67:0/1844373190 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710319654 0 0) 0x7f1c640ad8f0 con 0x7f1c6415dbc0 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6a000640 1 --2- 172.21.15.67:0/1844373190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c64071c60 0x7f1c640720c0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009d90 tx=0x7f1c5c01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8f0b0fb4026f828 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c63e00640 1 -- 172.21.15.67:0/1844373190 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0b2070 con 0x7f1c64071c60 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c63e00640 1 -- 172.21.15.67:0/1844373190 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c0a57a0 con 0x7f1c64071c60 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c63e00640 1 -- 172.21.15.67:0/1844373190 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0a5b70 con 0x7f1c64071c60 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- 172.21.15.67:0/1844373190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c64071c60 msgr2=0x7f1c640720c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1844373190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c64071c60 0x7f1c640720c0 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009d90 tx=0x7f1c5c01c8e0 comp rx=0 tx=0).stop 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- 172.21.15.67:0/1844373190 shutdown_connections 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1844373190 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c64071c60 0x7f1c640720c0 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1844373190 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6406c410 0x7f1c6406f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- 172.21.15.67:0/1844373190 >> 172.21.15.67:0/1844373190 conn(0x7f1c640af670 msgr2=0x7f1c640b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- 172.21.15.67:0/1844373190 shutdown_connections 2024-04-17T10:38:25.824 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- 172.21.15.67:0/1844373190 wait complete. 2024-04-17T10:38:25.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 Processor -- start 2024-04-17T10:38:25.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- start start 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6406c410 0x7f1c64149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64071c60 0x7f1c640cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6415dbc0 0x7f1c64151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6416a100 con 0x7f1c6406c410 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6406af70 con 0x7f1c64071c60 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c6c000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6416a280 con 0x7f1c6415dbc0 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.835+0000 7f1c69600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6406c410 0x7f1c64149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6406c410 0x7f1c64149880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38234/0 (socket says 172.21.15.67:38234) 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 -- 172.21.15.67:0/1577602848 learned_addr learned my addr 172.21.15.67:0/1577602848 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c68c00640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64071c60 0x7f1c640cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c6a000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6415dbc0 0x7f1c64151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:25.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 -- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6415dbc0 msgr2=0x7f1c64151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6415dbc0 0x7f1c64151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 -- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64071c60 msgr2=0x7f1c640cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64071c60 0x7f1c640cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 -- 172.21.15.67:0/1577602848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640ccca0 con 0x7f1c6406c410 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c6a000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6415dbc0 0x7f1c64151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c69600640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6406c410 0x7f1c64149880 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800c520 tx=0x7f1c5800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c61600640 1 -- 172.21.15.67:0/1577602848 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c58018070 con 0x7f1c6406c410 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c61600640 1 -- 172.21.15.67:0/1577602848 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c580074e0 con 0x7f1c6406c410 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c61600640 1 -- 172.21.15.67:0/1577602848 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c580049f0 con 0x7f1c6406c410 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c640ccf80 con 0x7f1c6406c410 2024-04-17T10:38:25.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c640cd540 con 0x7f1c6406c410 2024-04-17T10:38:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6409e260 con 0x7f1c6406c410 2024-04-17T10:38:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c61600640 1 -- 172.21.15.67:0/1577602848 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1c58004b90 con 0x7f1c6406c410 2024-04-17T10:38:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.839+0000 7f1c61600640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1c44066f60 0x7f1c44069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:25.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.843+0000 7f1c61600640 1 -- 172.21.15.67:0/1577602848 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1c580845b0 con 0x7f1c6406c410 2024-04-17T10:38:25.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.843+0000 7f1c68c00640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1c44066f60 0x7f1c44069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:25.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.843+0000 7f1c68c00640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1c44066f60 0x7f1c44069420 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1c6415ce30 tx=0x7f1c50008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:25.834 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:25.847+0000 7f1c61600640 1 -- 172.21.15.67:0/1577602848 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1c58054cb0 con 0x7f1c6406c410 2024-04-17T10:38:26.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.039+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c64153780 con 0x7f1c6406c410 2024-04-17T10:38:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c61600640 1 -- 172.21.15.67:0/1577602848 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1c58058d50 con 0x7f1c6406c410 2024-04-17T10:38:26.031 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:26.031 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:26.031 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1c44066f60 msgr2=0x7f1c44069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1c44066f60 0x7f1c44069420 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1c6415ce30 tx=0x7f1c50008040 comp rx=0 tx=0).stop 2024-04-17T10:38:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6406c410 msgr2=0x7f1c64149880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6406c410 0x7f1c64149880 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800c520 tx=0x7f1c5800c9e0 comp rx=0 tx=0).stop 2024-04-17T10:38:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 shutdown_connections 2024-04-17T10:38:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1c44066f60 0x7f1c44069420 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:26.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1c6415dbc0 0x7f1c64151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1c64071c60 0x7f1c640cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 --2- 172.21.15.67:0/1577602848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c6406c410 0x7f1c64149880 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 >> 172.21.15.67:0/1577602848 conn(0x7f1c640af670 msgr2=0x7f1c640b4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 shutdown_connections 2024-04-17T10:38:26.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:26.043+0000 7f1c6c000640 1 -- 172.21.15.67:0/1577602848 wait complete. 2024-04-17T10:38:26.455 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:26.456 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:27.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa580600640 1 Processor -- start 2024-04-17T10:38:27.611 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa580600640 1 -- start start 2024-04-17T10:38:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa580600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 0x7fa5780cad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa580600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa5780cb260 0x7fa5780cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa580600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57806b100 con 0x7fa5780ce2a0 2024-04-17T10:38:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa580600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780cd150 con 0x7fa5780cb260 2024-04-17T10:38:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa580600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa57806b3e0 con 0x7fa5780cc250 2024-04-17T10:38:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57e600640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa5780ce2a0 0x7fa5780cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55316/0 (socket says 172.21.15.67:55316) 2024-04-17T10:38:27.612 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57e600640 1 -- 172.21.15.67:0/3779153909 learned_addr learned my addr 172.21.15.67:0/3779153909 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57d200640 1 --2- 172.21.15.67:0/3779153909 >> v2:172.21.15.86:3300/0 conn(0x7fa5780cb260 0x7fa5780cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 0x7fa5780cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372316433 0 0) 0x7fa57806b100 con 0x7fa5780ce2a0 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa560003530 con 0x7fa5780ce2a0 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4239165587 0 0) 0x7fa560003530 con 0x7fa5780ce2a0 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 msgr2=0x7fa5780cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 0x7fa5780cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 >> v2:172.21.15.86:3300/0 conn(0x7fa5780cb260 msgr2=0x7fa5780cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 --2- 172.21.15.67:0/3779153909 >> v2:172.21.15.86:3300/0 conn(0x7fa5780cb260 0x7fa5780cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57806b6c0 con 0x7fa5780ce2a0 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fa570001a00 con 0x7fa5780ce2a0 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 >> v1:172.21.15.67:6789/0 conn(0x7fa5780ce2a0 legacy=0x7fa5780cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa560003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 0x7fa560004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa560004d50 0x7fa5600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780cd150 con 0x7fa560004d50 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57806b3e0 con 0x7fa5600039c0 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56000b9d0 con 0x7fa5780cc250 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57e600640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa560004d50 0x7fa5600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa560003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57d200640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 0x7fa560004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57dc00640 1 -- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 msgr2=0x7fa560004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa5780cc250 0x7fa560004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57dc00640 1 -- 172.21.15.67:0/3779153909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa560004d50 msgr2=0x7fa5600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa560004d50 0x7fa5600051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57dc00640 1 -- 172.21.15.67:0/3779153909 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa560003530 con 0x7fa5600039c0 2024-04-17T10:38:27.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.623+0000 7fa57e600640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa560004d50 0x7fa5600051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa560003de0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009ae0 tx=0x7fa56c01ca70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef22c78cec2e9d30 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0b2070 con 0x7fa5600039c0 2024-04-17T10:38:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56c0a5660 con 0x7fa5600039c0 2024-04-17T10:38:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa577e00640 1 -- 172.21.15.67:0/3779153909 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c0096b0 con 0x7fa5600039c0 2024-04-17T10:38:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa580600640 1 -- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 msgr2=0x7fa560003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa580600640 1 --2- 172.21.15.67:0/3779153909 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa560003de0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fa56c009ae0 tx=0x7fa56c01ca70 comp rx=0 tx=0).stop 2024-04-17T10:38:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa57e600640 1 -- 172.21.15.67:0/3779153909 reap_dead start 2024-04-17T10:38:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa580600640 1 -- 172.21.15.67:0/3779153909 shutdown_connections 2024-04-17T10:38:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa580600640 1 -- 172.21.15.67:0/3779153909 >> 172.21.15.67:0/3779153909 conn(0x7fa5780b22a0 msgr2=0x7fa5780b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa580600640 1 -- 172.21.15.67:0/3779153909 shutdown_connections 2024-04-17T10:38:27.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.627+0000 7fa580600640 1 -- 172.21.15.67:0/3779153909 wait complete. 2024-04-17T10:38:27.620 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 Processor -- start 2024-04-17T10:38:27.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 -- start start 2024-04-17T10:38:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa57809e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa578150490 0x7fa5780c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5780c8780 0x7fa5780addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780d0260 con 0x7fa5780c8780 2024-04-17T10:38:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780d00e0 con 0x7fa5600039c0 2024-04-17T10:38:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5780d03e0 con 0x7fa578150490 2024-04-17T10:38:27.624 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5780c8780 0x7fa5780addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5780c8780 0x7fa5780addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38256/0 (socket says 172.21.15.67:38256) 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 -- 172.21.15.67:0/3796210609 learned_addr learned my addr 172.21.15.67:0/3796210609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57d200640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa578150490 0x7fa5780c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa57809e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 -- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa578150490 msgr2=0x7fa5780c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa578150490 0x7fa5780c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 -- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 msgr2=0x7fa57809e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa57809e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 -- 172.21.15.67:0/3796210609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5780c8d40 con 0x7fa5780c8780 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa57809e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa57e600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5780c8780 0x7fa5780addf0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fa5780ce220 tx=0x7fa57000af00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa575600640 1 -- 172.21.15.67:0/3796210609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57000a3f0 con 0x7fa5780c8780 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa575600640 1 -- 172.21.15.67:0/3796210609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa570018070 con 0x7fa5780c8780 2024-04-17T10:38:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa575600640 1 -- 172.21.15.67:0/3796210609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5700133f0 con 0x7fa5780c8780 2024-04-17T10:38:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57815dc50 con 0x7fa5780c8780 2024-04-17T10:38:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.631+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57815e210 con 0x7fa5780c8780 2024-04-17T10:38:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.635+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57806cd30 con 0x7fa5780c8780 2024-04-17T10:38:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.635+0000 7fa575600640 1 -- 172.21.15.67:0/3796210609 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa570013590 con 0x7fa5780c8780 2024-04-17T10:38:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.635+0000 7fa575600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f90 0x7fa558069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.635+0000 7fa575600640 1 -- 172.21.15.67:0/3796210609 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa570085810 con 0x7fa5780c8780 2024-04-17T10:38:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.635+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f90 0x7fa558069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.635+0000 7fa57dc00640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f90 0x7fa558069450 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0af000 tx=0x7fa56c0046d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:27.628 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.639+0000 7fa575600640 1 -- 172.21.15.67:0/3796210609 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa5700564d0 con 0x7fa5780c8780 2024-04-17T10:38:27.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.835+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5780b6650 con 0x7fa5780c8780 2024-04-17T10:38:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.835+0000 7fa575600640 1 -- 172.21.15.67:0/3796210609 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fa570055dd0 con 0x7fa5780c8780 2024-04-17T10:38:27.826 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:27.826 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:27.826 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f90 msgr2=0x7fa558069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f90 0x7fa558069450 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0af000 tx=0x7fa56c0046d0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5780c8780 msgr2=0x7fa5780addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5780c8780 0x7fa5780addf0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fa5780ce220 tx=0x7fa57000af00 comp rx=0 tx=0).stop 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 shutdown_connections 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa558066f90 0x7fa558069450 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa5780c8780 0x7fa5780addf0 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa578150490 0x7fa5780c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 --2- 172.21.15.67:0/3796210609 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa5600039c0 0x7fa57809e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 >> 172.21.15.67:0/3796210609 conn(0x7fa5780b22a0 msgr2=0x7fa5780cd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:27.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 shutdown_connections 2024-04-17T10:38:27.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:27.839+0000 7fa580600640 1 -- 172.21.15.67:0/3796210609 wait complete. 2024-04-17T10:38:28.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:28.249 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:28.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:38:28.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:38:29.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:29.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef7c00640 1 Processor -- start 2024-04-17T10:38:29.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef7c00640 1 -- start start 2024-04-17T10:38:29.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef7c00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feef0071310 0x7feef0071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:29.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef7c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef006e2f0 con 0x7feef00cc300 2024-04-17T10:38:29.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef7c00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feef006d6e0 con 0x7feef0145040 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef7c00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feef00a86c0 con 0x7feef0071310 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feeefe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7feef0071310 0x7feef0071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef5c00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7feef00cc300 0x7feef00cc710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55330/0 (socket says 172.21.15.67:55330) 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.415+0000 7feef5c00640 1 -- 172.21.15.67:0/595089658 learned_addr learned my addr 172.21.15.67:0/595089658 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeefe00640 1 -- 172.21.15.67:0/595089658 >> v1:172.21.15.86:6789/0 conn(0x7feef0145040 legacy=0x7feef006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeefe00640 1 -- 172.21.15.67:0/595089658 >> v1:172.21.15.67:6789/0 conn(0x7feef00cc300 legacy=0x7feef00cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeefe00640 1 -- 172.21.15.67:0/595089658 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef006d9c0 con 0x7feef0071310 2024-04-17T10:38:29.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeef400640 1 -- 172.21.15.67:0/595089658 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963592999 0 0) 0x7feef006e2f0 con 0x7feef00cc300 2024-04-17T10:38:29.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeefe00640 1 --2- 172.21.15.67:0/595089658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef0071310 0x7feef0071770 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7feedc0097c0 tx=0x7feedc01ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f313140f06e4a80 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:29.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeef400640 1 -- 172.21.15.67:0/595089658 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc0b5070 con 0x7feef0071310 2024-04-17T10:38:29.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeef400640 1 -- 172.21.15.67:0/595089658 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feedc0a8990 con 0x7feef0071310 2024-04-17T10:38:29.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.419+0000 7feeef400640 1 -- 172.21.15.67:0/595089658 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc0a8d60 con 0x7feef0071310 2024-04-17T10:38:29.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- 172.21.15.67:0/595089658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef0071310 msgr2=0x7feef0071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:29.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 --2- 172.21.15.67:0/595089658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef0071310 0x7feef0071770 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7feedc0097c0 tx=0x7feedc01ec90 comp rx=0 tx=0).stop 2024-04-17T10:38:29.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- 172.21.15.67:0/595089658 shutdown_connections 2024-04-17T10:38:29.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 --2- 172.21.15.67:0/595089658 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef0071310 0x7feef0071770 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:29.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- 172.21.15.67:0/595089658 >> 172.21.15.67:0/595089658 conn(0x7feef006ac90 msgr2=0x7feef006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:29.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- 172.21.15.67:0/595089658 shutdown_connections 2024-04-17T10:38:29.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- 172.21.15.67:0/595089658 wait complete. 2024-04-17T10:38:29.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 Processor -- start 2024-04-17T10:38:29.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- start start 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feef0071310 0x7feef015e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef00cc300 0x7feef013c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feef0145040 0x7feef00b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef006df80 con 0x7feef00cc300 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feef015bf10 con 0x7feef0071310 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef7c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feef006d370 con 0x7feef0145040 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef00cc300 0x7feef013c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:29.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef00cc300 0x7feef013c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38276/0 (socket says 172.21.15.67:38276) 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 -- 172.21.15.67:0/503349125 learned_addr learned my addr 172.21.15.67:0/503349125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef5c00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feef0145040 0x7feef00b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feef5200640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feef0071310 0x7feef015e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 -- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feef0145040 msgr2=0x7feef00b0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feef0145040 0x7feef00b0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 -- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feef0071310 msgr2=0x7feef015e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feef0071310 0x7feef015e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 -- 172.21.15.67:0/503349125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef0160410 con 0x7feef00cc300 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.423+0000 7feeefe00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef00cc300 0x7feef013c090 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7feedc01b8a0 tx=0x7feedc01ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feeecc00640 1 -- 172.21.15.67:0/503349125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc0b5040 con 0x7feef00cc300 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feeecc00640 1 -- 172.21.15.67:0/503349125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feedc0bf070 con 0x7feef00cc300 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feeecc00640 1 -- 172.21.15.67:0/503349125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feedc0b0b60 con 0x7feef00cc300 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feef013c5d0 con 0x7feef00cc300 2024-04-17T10:38:29.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feef013cb30 con 0x7feef00cc300 2024-04-17T10:38:29.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feef0072d50 con 0x7feef00cc300 2024-04-17T10:38:29.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feeecc00640 1 -- 172.21.15.67:0/503349125 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7feedc0b0d00 con 0x7feef00cc300 2024-04-17T10:38:29.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feeecc00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feed0066f60 0x7feed0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:29.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feeecc00640 1 -- 172.21.15.67:0/503349125 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7feedc1229b0 con 0x7feef00cc300 2024-04-17T10:38:29.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.427+0000 7feef5200640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feed0066f60 0x7feed0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:29.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.431+0000 7feef5200640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feed0066f60 0x7feed0069420 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7feef0138970 tx=0x7feee4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:29.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.431+0000 7feeecc00640 1 -- 172.21.15.67:0/503349125 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7feef013c5d0 con 0x7feef00cc300 2024-04-17T10:38:29.618 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.627+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feef0074e30 con 0x7feef00cc300 2024-04-17T10:38:29.619 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feeecc00640 1 -- 172.21.15.67:0/503349125 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7feedc0187f0 con 0x7feef00cc300 2024-04-17T10:38:29.619 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:29.619 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:29.619 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:29.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feed0066f60 msgr2=0x7feed0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:29.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feed0066f60 0x7feed0069420 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7feef0138970 tx=0x7feee4009040 comp rx=0 tx=0).stop 2024-04-17T10:38:29.621 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef00cc300 msgr2=0x7feef013c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef00cc300 0x7feef013c090 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7feedc01b8a0 tx=0x7feedc01ba30 comp rx=0 tx=0).stop 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 shutdown_connections 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7feed0066f60 0x7feed0069420 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7feef0145040 0x7feef00b0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feef00cc300 0x7feef013c090 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 --2- 172.21.15.67:0/503349125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7feef0071310 0x7feef015e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 >> 172.21.15.67:0/503349125 conn(0x7feef006ac90 msgr2=0x7feef006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 shutdown_connections 2024-04-17T10:38:29.622 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:29.631+0000 7feef7c00640 1 -- 172.21.15.67:0/503349125 wait complete. 2024-04-17T10:38:30.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:30.044 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:31.045 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:31.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb73200640 1 Processor -- start 2024-04-17T10:38:31.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb73200640 1 -- start start 2024-04-17T10:38:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb73200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb73200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbb740b2570 0x7fbb740ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb73200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb740d0760 con 0x7fbb74161160 2024-04-17T10:38:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb73200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb740cfc70 con 0x7fbb740b2570 2024-04-17T10:38:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb73200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740cc5d0 con 0x7fbb7409a160 2024-04-17T10:38:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb71e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbb74161160 0x7fbb74099a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:54120/0 (socket says 172.21.15.67:54120) 2024-04-17T10:38:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb71e00640 1 -- 172.21.15.67:0/2632108148 learned_addr learned my addr 172.21.15.67:0/2632108148 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb69400640 1 --2- 172.21.15.67:0/2632108148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:31.201 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb72800640 1 --2- 172.21.15.67:0/2632108148 >> v2:172.21.15.86:3300/0 conn(0x7fbb740b2570 0x7fbb740ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb71400640 1 -- 172.21.15.67:0/2632108148 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231800014 0 0) 0x7fbb740d0760 con 0x7fbb74161160 2024-04-17T10:38:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb71400640 1 -- 172.21.15.67:0/2632108148 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb58003530 con 0x7fbb74161160 2024-04-17T10:38:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb69400640 1 -- 172.21.15.67:0/2632108148 >> v2:172.21.15.86:3300/0 conn(0x7fbb740b2570 msgr2=0x7fbb740ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb69400640 1 --2- 172.21.15.67:0/2632108148 >> v2:172.21.15.86:3300/0 conn(0x7fbb740b2570 0x7fbb740ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb69400640 1 -- 172.21.15.67:0/2632108148 >> v1:172.21.15.86:6789/0 conn(0x7fbb74161160 legacy=0x7fbb74099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb69400640 1 -- 172.21.15.67:0/2632108148 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740cc2a0 con 0x7fbb7409a160 2024-04-17T10:38:31.202 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.211+0000 7fbb72800640 1 --2- 172.21.15.67:0/2632108148 >> v2:172.21.15.86:3300/0 conn(0x7fbb740b2570 0x7fbb740ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb69400640 1 --2- 172.21.15.67:0/2632108148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740cca60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009d90 tx=0x7fbb5c01c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e0fbc8fbe4446 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb71400640 1 -- 172.21.15.67:0/2632108148 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0b2070 con 0x7fbb7409a160 2024-04-17T10:38:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb71400640 1 -- 172.21.15.67:0/2632108148 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c0a5920 con 0x7fbb7409a160 2024-04-17T10:38:31.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb71400640 1 -- 172.21.15.67:0/2632108148 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0a5cf0 con 0x7fbb7409a160 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 -- 172.21.15.67:0/2632108148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 msgr2=0x7fbb740cca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 --2- 172.21.15.67:0/2632108148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740cca60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009d90 tx=0x7fbb5c01c970 comp rx=0 tx=0).stop 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 -- 172.21.15.67:0/2632108148 shutdown_connections 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 --2- 172.21.15.67:0/2632108148 >> v2:172.21.15.86:3300/0 conn(0x7fbb740b2570 0x7fbb740ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 --2- 172.21.15.67:0/2632108148 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740cca60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 -- 172.21.15.67:0/2632108148 >> 172.21.15.67:0/2632108148 conn(0x7fbb740d1690 msgr2=0x7fbb740d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 -- 172.21.15.67:0/2632108148 shutdown_connections 2024-04-17T10:38:31.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.215+0000 7fbb73200640 1 -- 172.21.15.67:0/2632108148 wait complete. 2024-04-17T10:38:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 Processor -- start 2024-04-17T10:38:31.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 -- start start 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb740b2570 0x7fbb74151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb74161160 0x7fbb740c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740b2bc0 con 0x7fbb740b2570 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb740cd8f0 con 0x7fbb74161160 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7406bda0 con 0x7fbb7409a160 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb740b2570 0x7fbb74151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:31.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb740b2570 0x7fbb74151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41368/0 (socket says 172.21.15.67:41368) 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 -- 172.21.15.67:0/2407929865 learned_addr learned my addr 172.21.15.67:0/2407929865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb71e00640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb72800640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb74161160 0x7fbb740c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 -- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 msgr2=0x7fbb740c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 -- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb74161160 msgr2=0x7fbb740c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb74161160 0x7fbb740c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 -- 172.21.15.67:0/2407929865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb741522d0 con 0x7fbb740b2570 2024-04-17T10:38:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb69400640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb740b2570 0x7fbb74151500 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0b0040 tx=0x7fbb5c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb6aa00640 1 -- 172.21.15.67:0/2407929865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0b2050 con 0x7fbb740b2570 2024-04-17T10:38:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb6aa00640 1 -- 172.21.15.67:0/2407929865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c0be070 con 0x7fbb740b2570 2024-04-17T10:38:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb741525b0 con 0x7fbb740b2570 2024-04-17T10:38:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb6aa00640 1 -- 172.21.15.67:0/2407929865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0ad850 con 0x7fbb740b2570 2024-04-17T10:38:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.219+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb74152b70 con 0x7fbb740b2570 2024-04-17T10:38:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.223+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb74074dc0 con 0x7fbb740b2570 2024-04-17T10:38:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.223+0000 7fbb6aa00640 1 -- 172.21.15.67:0/2407929865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbb5c0ad9f0 con 0x7fbb740b2570 2024-04-17T10:38:31.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.223+0000 7fbb6aa00640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb50066f60 0x7fbb50069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.223+0000 7fbb6aa00640 1 -- 172.21.15.67:0/2407929865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbb5c120110 con 0x7fbb740b2570 2024-04-17T10:38:31.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.223+0000 7fbb71e00640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb50066f60 0x7fbb50069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:31.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.223+0000 7fbb71e00640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb50066f60 0x7fbb50069420 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007fd0 tx=0x7fbb64009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:31.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.227+0000 7fbb6aa00640 1 -- 172.21.15.67:0/2407929865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbb5c016560 con 0x7fbb740b2570 2024-04-17T10:38:31.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb7409ac60 con 0x7fbb740b2570 2024-04-17T10:38:31.413 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb6aa00640 1 -- 172.21.15.67:0/2407929865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fbb5c0f0f90 con 0x7fbb740b2570 2024-04-17T10:38:31.413 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:31.413 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:31.414 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb50066f60 msgr2=0x7fbb50069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb50066f60 0x7fbb50069420 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007fd0 tx=0x7fbb64009040 comp rx=0 tx=0).stop 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb740b2570 msgr2=0x7fbb74151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb740b2570 0x7fbb74151500 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0b0040 tx=0x7fbb5c019710 comp rx=0 tx=0).stop 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 shutdown_connections 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbb50066f60 0x7fbb50069420 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbb74161160 0x7fbb740c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbb740b2570 0x7fbb74151500 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 --2- 172.21.15.67:0/2407929865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbb7409a160 0x7fbb740c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:31.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 >> 172.21.15.67:0/2407929865 conn(0x7fbb740d1690 msgr2=0x7fbb7406f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:31.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 shutdown_connections 2024-04-17T10:38:31.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:31.423+0000 7fbb73200640 1 -- 172.21.15.67:0/2407929865 wait complete. 2024-04-17T10:38:31.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:31.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:32.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:32.999+0000 7fedc3400640 1 Processor -- start 2024-04-17T10:38:32.991 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:32.999+0000 7fedc3400640 1 -- start start 2024-04-17T10:38:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc3400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedbc1466e0 0x7fedbc06eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc071a20 0x7fedbc0adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc3400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc161740 con 0x7fedbc146da0 2024-04-17T10:38:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc3400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedbc161a20 con 0x7fedbc1466e0 2024-04-17T10:38:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc3400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc161d00 con 0x7fedbc071a20 2024-04-17T10:38:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedbc1466e0 0x7fedbc06eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:32.993 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc0a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fedbc1466e0 0x7fedbc06eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41382/0 (socket says 172.21.15.67:41382) 2024-04-17T10:38:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc0a00640 1 -- 172.21.15.67:0/1438777460 learned_addr learned my addr 172.21.15.67:0/1438777460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc1400640 1 --2- 172.21.15.67:0/1438777460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc071a20 0x7fedbc0adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc0a00640 1 -- 172.21.15.67:0/1438777460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc071a20 msgr2=0x7fedbc0adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc0a00640 1 --2- 172.21.15.67:0/1438777460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc071a20 0x7fedbc0adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc0a00640 1 -- 172.21.15.67:0/1438777460 >> v1:172.21.15.86:6789/0 conn(0x7fedbc146da0 legacy=0x7fedbc071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc0a00640 1 -- 172.21.15.67:0/1438777460 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc161fe0 con 0x7fedbc1466e0 2024-04-17T10:38:32.994 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.003+0000 7fedc1400640 1 --2- 172.21.15.67:0/1438777460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc071a20 0x7fedbc0adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc0a00640 1 --2- 172.21.15.67:0/1438777460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc1466e0 0x7fedbc06eb70 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fedac009e50 tx=0x7fedac01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5faf7e92f681383d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedbbe00640 1 -- 172.21.15.67:0/1438777460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0a55f0 con 0x7fedbc1466e0 2024-04-17T10:38:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedbbe00640 1 -- 172.21.15.67:0/1438777460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac0a5790 con 0x7fedbc1466e0 2024-04-17T10:38:32.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedbbe00640 1 -- 172.21.15.67:0/1438777460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0a5b60 con 0x7fedbc1466e0 2024-04-17T10:38:32.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 -- 172.21.15.67:0/1438777460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc1466e0 msgr2=0x7fedbc06eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:32.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 --2- 172.21.15.67:0/1438777460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc1466e0 0x7fedbc06eb70 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7fedac009e50 tx=0x7fedac01c9f0 comp rx=0 tx=0).stop 2024-04-17T10:38:32.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 -- 172.21.15.67:0/1438777460 shutdown_connections 2024-04-17T10:38:32.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 --2- 172.21.15.67:0/1438777460 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc071a20 0x7fedbc0adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:32.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 --2- 172.21.15.67:0/1438777460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc1466e0 0x7fedbc06eb70 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:32.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 -- 172.21.15.67:0/1438777460 >> 172.21.15.67:0/1438777460 conn(0x7fedbc06ac90 msgr2=0x7fedbc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 -- 172.21.15.67:0/1438777460 shutdown_connections 2024-04-17T10:38:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 -- 172.21.15.67:0/1438777460 wait complete. 2024-04-17T10:38:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 Processor -- start 2024-04-17T10:38:32.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.007+0000 7fedc3400640 1 -- start start 2024-04-17T10:38:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc071a20 0x7fedbc13e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedbc1466e0 0x7fedbc1454b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc146da0 0x7fedbc1459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0aef00 con 0x7fedbc071a20 2024-04-17T10:38:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0aed80 con 0x7fedbc1466e0 2024-04-17T10:38:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc0af080 con 0x7fedbc146da0 2024-04-17T10:38:33.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc071a20 0x7fedbc13e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc071a20 0x7fedbc13e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41396/0 (socket says 172.21.15.67:41396) 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 -- 172.21.15.67:0/1544128561 learned_addr learned my addr 172.21.15.67:0/1544128561 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc1400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc146da0 0x7fedbc1459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedb3e00640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedbc1466e0 0x7fedbc1454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 -- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc146da0 msgr2=0x7fedbc1459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc146da0 0x7fedbc1459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 -- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedbc1466e0 msgr2=0x7fedbc1454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedbc1466e0 0x7fedbc1454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 -- 172.21.15.67:0/1544128561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc0cc4c0 con 0x7fedbc071a20 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc1400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc146da0 0x7fedbc1459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedb3e00640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedbc1466e0 0x7fedbc1454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc0a00640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc071a20 0x7fedbc13e3a0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fedac01b0a0 tx=0x7fedac019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:33.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedb9600640 1 -- 172.21.15.67:0/1544128561 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0bc030 con 0x7fedbc071a20 2024-04-17T10:38:33.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedb9600640 1 -- 172.21.15.67:0/1544128561 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac0b8470 con 0x7fedbc071a20 2024-04-17T10:38:33.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedb9600640 1 -- 172.21.15.67:0/1544128561 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedac0b88a0 con 0x7fedbc071a20 2024-04-17T10:38:33.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc0cc7a0 con 0x7fedbc071a20 2024-04-17T10:38:33.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc0ccd60 con 0x7fedbc071a20 2024-04-17T10:38:33.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.011+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedbc072d50 con 0x7fedbc071a20 2024-04-17T10:38:33.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.015+0000 7fedb9600640 1 -- 172.21.15.67:0/1544128561 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fedac0bf070 con 0x7fedbc071a20 2024-04-17T10:38:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.015+0000 7fedb9600640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fed98066f60 0x7fed98069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.015+0000 7fedb9600640 1 -- 172.21.15.67:0/1544128561 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fedac11fdf0 con 0x7fedbc071a20 2024-04-17T10:38:33.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.015+0000 7fedb3e00640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fed98066f60 0x7fed98069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:33.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.015+0000 7fedb3e00640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fed98066f60 0x7fed98069420 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fedbc1f9c90 tx=0x7feda4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:33.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.019+0000 7fedb9600640 1 -- 172.21.15.67:0/1544128561 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fedac0165e0 con 0x7fedbc071a20 2024-04-17T10:38:33.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.219+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedbc13e8e0 con 0x7fedbc071a20 2024-04-17T10:38:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.219+0000 7fedb9600640 1 -- 172.21.15.67:0/1544128561 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fedac0f0e50 con 0x7fedbc071a20 2024-04-17T10:38:33.210 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:33.210 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:33.210 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fed98066f60 msgr2=0x7fed98069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fed98066f60 0x7fed98069420 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fedbc1f9c90 tx=0x7feda4009040 comp rx=0 tx=0).stop 2024-04-17T10:38:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc071a20 msgr2=0x7fedbc13e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc071a20 0x7fedbc13e3a0 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7fedac01b0a0 tx=0x7fedac019790 comp rx=0 tx=0).stop 2024-04-17T10:38:33.212 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 shutdown_connections 2024-04-17T10:38:33.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fed98066f60 0x7fed98069420 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:33.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fedbc146da0 0x7fedbc1459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:33.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fedbc1466e0 0x7fedbc1454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:33.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 --2- 172.21.15.67:0/1544128561 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fedbc071a20 0x7fedbc13e3a0 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:33.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 >> 172.21.15.67:0/1544128561 conn(0x7fedbc06ac90 msgr2=0x7fedbc0afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:33.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 shutdown_connections 2024-04-17T10:38:33.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:33.223+0000 7fedc3400640 1 -- 172.21.15.67:0/1544128561 wait complete. 2024-04-17T10:38:33.634 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:33.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:34.636 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.799+0000 7fd93e400640 1 Processor -- start 2024-04-17T10:38:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.799+0000 7fd93e400640 1 -- start start 2024-04-17T10:38:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.799+0000 7fd93e400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd938071770 0x7fd9380cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.799+0000 7fd93e400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93806e2f0 con 0x7fd938071310 2024-04-17T10:38:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.799+0000 7fd93e400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93806d6e0 con 0x7fd938145040 2024-04-17T10:38:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.799+0000 7fd93e400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9380a86c0 con 0x7fd938071770 2024-04-17T10:38:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd937e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd938071770 0x7fd9380cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd92fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd938071310 0x7fd9380cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36504/0 (socket says 172.21.15.67:36504) 2024-04-17T10:38:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd92fe00640 1 -- 172.21.15.67:0/1098850773 learned_addr learned my addr 172.21.15.67:0/1098850773 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:34.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd937e00640 1 -- 172.21.15.67:0/1098850773 >> v1:172.21.15.86:6789/0 conn(0x7fd938145040 legacy=0x7fd93806e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:34.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd937e00640 1 -- 172.21.15.67:0/1098850773 >> v1:172.21.15.67:6789/0 conn(0x7fd938071310 legacy=0x7fd9380cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:34.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd937e00640 1 -- 172.21.15.67:0/1098850773 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93806d9c0 con 0x7fd938071770 2024-04-17T10:38:34.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd937e00640 1 --2- 172.21.15.67:0/1098850773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938071770 0x7fd9380cca10 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fd930009d90 tx=0x7fd93001f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cad629f03ac0a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:34.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd936a00640 1 -- 172.21.15.67:0/1098850773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300b4070 con 0x7fd938071770 2024-04-17T10:38:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd936a00640 1 -- 172.21.15.67:0/1098850773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9300af600 con 0x7fd938071770 2024-04-17T10:38:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.803+0000 7fd936a00640 1 -- 172.21.15.67:0/1098850773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300af9d0 con 0x7fd938071770 2024-04-17T10:38:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- 172.21.15.67:0/1098850773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938071770 msgr2=0x7fd9380cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 --2- 172.21.15.67:0/1098850773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938071770 0x7fd9380cca10 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fd930009d90 tx=0x7fd93001f150 comp rx=0 tx=0).stop 2024-04-17T10:38:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- 172.21.15.67:0/1098850773 shutdown_connections 2024-04-17T10:38:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 --2- 172.21.15.67:0/1098850773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938071770 0x7fd9380cca10 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- 172.21.15.67:0/1098850773 >> 172.21.15.67:0/1098850773 conn(0x7fd93806ac90 msgr2=0x7fd93806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- 172.21.15.67:0/1098850773 shutdown_connections 2024-04-17T10:38:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- 172.21.15.67:0/1098850773 wait complete. 2024-04-17T10:38:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 Processor -- start 2024-04-17T10:38:34.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- start start 2024-04-17T10:38:34.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd938071310 0x7fd9381441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd938071770 0x7fd93813be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938145040 0x7fd93815b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938071b80 con 0x7fd938145040 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93806df80 con 0x7fd938071770 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd93e400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93806d370 con 0x7fd938071310 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938145040 0x7fd93815b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938145040 0x7fd93815b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41428/0 (socket says 172.21.15.67:41428) 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 -- 172.21.15.67:0/1859110264 learned_addr learned my addr 172.21.15.67:0/1859110264 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd92fe00640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd938071770 0x7fd93813be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd938071310 0x7fd9381441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 -- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd938071310 msgr2=0x7fd9381441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd938071310 0x7fd9381441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:34.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 -- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd938071770 msgr2=0x7fd93813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd938071770 0x7fd93813be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937e00640 1 -- 172.21.15.67:0/1859110264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93813c6a0 con 0x7fd938145040 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.807+0000 7fd937400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd938071310 0x7fd9381441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd937e00640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938145040 0x7fd93815b150 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fd9300b2040 tx=0x7fd93001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd92f400640 1 -- 172.21.15.67:0/1859110264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300b4040 con 0x7fd938145040 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd92f400640 1 -- 172.21.15.67:0/1859110264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9300c1070 con 0x7fd938145040 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93813c980 con 0x7fd938145040 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd92f400640 1 -- 172.21.15.67:0/1859110264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9300ba6b0 con 0x7fd938145040 2024-04-17T10:38:34.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93813e5f0 con 0x7fd938145040 2024-04-17T10:38:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938072d50 con 0x7fd938145040 2024-04-17T10:38:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd92f400640 1 -- 172.21.15.67:0/1859110264 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd9300ba850 con 0x7fd938145040 2024-04-17T10:38:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd92f400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd914066f90 0x7fd914069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd92f400640 1 -- 172.21.15.67:0/1859110264 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fd930123e60 con 0x7fd938145040 2024-04-17T10:38:34.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.811+0000 7fd937400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd914066f90 0x7fd914069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:34.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.815+0000 7fd937400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd914066f90 0x7fd914069450 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd928000f70 tx=0x7fd928009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:34.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:34.815+0000 7fd92f400640 1 -- 172.21.15.67:0/1859110264 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd930018cb0 con 0x7fd938145040 2024-04-17T10:38:35.003 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9380ce1c0 con 0x7fd938145040 2024-04-17T10:38:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd92f400640 1 -- 172.21.15.67:0/1859110264 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fd9300f4e20 con 0x7fd938145040 2024-04-17T10:38:35.004 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:35.004 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:35.004 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd914066f90 msgr2=0x7fd914069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd914066f90 0x7fd914069450 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fd928000f70 tx=0x7fd928009040 comp rx=0 tx=0).stop 2024-04-17T10:38:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938145040 msgr2=0x7fd93815b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938145040 0x7fd93815b150 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7fd9300b2040 tx=0x7fd93001bef0 comp rx=0 tx=0).stop 2024-04-17T10:38:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 shutdown_connections 2024-04-17T10:38:35.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd914066f90 0x7fd914069450 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:35.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd938145040 0x7fd93815b150 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:35.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd938071770 0x7fd93813be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:35.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 --2- 172.21.15.67:0/1859110264 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd938071310 0x7fd9381441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:35.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 >> 172.21.15.67:0/1859110264 conn(0x7fd93806ac90 msgr2=0x7fd9380ae370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:35.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 shutdown_connections 2024-04-17T10:38:35.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:35.015+0000 7fd93e400640 1 -- 172.21.15.67:0/1859110264 wait complete. 2024-04-17T10:38:35.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:35.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:36.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:36.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.591+0000 7f9e81000640 1 Processor -- start 2024-04-17T10:38:36.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.591+0000 7f9e81000640 1 -- start start 2024-04-17T10:38:36.583 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e81000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e7c072370 0x7f9e7c06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e81000640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0ad8f0 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e81000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e7c0adbd0 con 0x7f9e7c06c410 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e81000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7c15dfa0 con 0x7f9e7c072370 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e7a800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9e7c072370 0x7f9e7c06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e79e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f9e7c15dbc0 0x7f9e7c06db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36508/0 (socket says 172.21.15.67:36508) 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e79e00640 1 -- 172.21.15.67:0/329924702 learned_addr learned my addr 172.21.15.67:0/329924702 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e7a800640 1 -- 172.21.15.67:0/329924702 >> v1:172.21.15.86:6789/0 conn(0x7f9e7c06c410 legacy=0x7f9e7c071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e7a800640 1 -- 172.21.15.67:0/329924702 >> v1:172.21.15.67:6789/0 conn(0x7f9e7c15dbc0 legacy=0x7f9e7c06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:36.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.595+0000 7f9e7a800640 1 -- 172.21.15.67:0/329924702 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0adeb0 con 0x7f9e7c072370 2024-04-17T10:38:36.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e7a800640 1 --2- 172.21.15.67:0/329924702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c072370 0x7f9e7c06f560 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009d90 tx=0x7f9e7401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edcd8740e9ba77e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:36.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e79400640 1 -- 172.21.15.67:0/329924702 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e740b2070 con 0x7f9e7c072370 2024-04-17T10:38:36.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e79400640 1 -- 172.21.15.67:0/329924702 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e740a57a0 con 0x7f9e7c072370 2024-04-17T10:38:36.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e79400640 1 -- 172.21.15.67:0/329924702 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e740a5b70 con 0x7f9e7c072370 2024-04-17T10:38:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- 172.21.15.67:0/329924702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c072370 msgr2=0x7f9e7c06f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 --2- 172.21.15.67:0/329924702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c072370 0x7f9e7c06f560 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f9e74009d90 tx=0x7f9e7401c8e0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- 172.21.15.67:0/329924702 shutdown_connections 2024-04-17T10:38:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 --2- 172.21.15.67:0/329924702 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c072370 0x7f9e7c06f560 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- 172.21.15.67:0/329924702 >> 172.21.15.67:0/329924702 conn(0x7f9e7c0af670 msgr2=0x7f9e7c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- 172.21.15.67:0/329924702 shutdown_connections 2024-04-17T10:38:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- 172.21.15.67:0/329924702 wait complete. 2024-04-17T10:38:36.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 Processor -- start 2024-04-17T10:38:36.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- start start 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7c06c410 0x7f9e7c146260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7c072370 0x7f9e7c14f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c15dbc0 0x7f9e7c154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c16b110 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c06af70 con 0x7f9e7c072370 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.599+0000 7f9e81000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c16b290 con 0x7f9e7c06c410 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c15dbc0 0x7f9e7c154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c15dbc0 0x7f9e7c154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41450/0 (socket says 172.21.15.67:41450) 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 -- 172.21.15.67:0/4245110711 learned_addr learned my addr 172.21.15.67:0/4245110711 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:36.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e73e00640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7c072370 0x7f9e7c14f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:36.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e79e00640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7c06c410 0x7f9e7c146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 -- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7c06c410 msgr2=0x7f9e7c146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7c06c410 0x7f9e7c146260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 -- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7c072370 msgr2=0x7f9e7c14f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7c072370 0x7f9e7c14f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 -- 172.21.15.67:0/4245110711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c1538c0 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e7a800640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c15dbc0 0x7f9e7c154680 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f9e7401a510 tx=0x7f9e74019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e72000640 1 -- 172.21.15.67:0/4245110711 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e740b2050 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e72000640 1 -- 172.21.15.67:0/4245110711 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e740be070 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e72000640 1 -- 172.21.15.67:0/4245110711 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e740b9450 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c153ba0 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c1467a0 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e7c09e260 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.603+0000 7f9e72000640 1 -- 172.21.15.67:0/4245110711 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f9e740b95f0 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.607+0000 7f9e72000640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e58066fb0 0x7f9e58069470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:36.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.607+0000 7f9e72000640 1 -- 172.21.15.67:0/4245110711 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f9e74120680 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.607+0000 7f9e79e00640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e58066fb0 0x7f9e58069470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:36.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.607+0000 7f9e79e00640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e58066fb0 0x7f9e58069470 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c005890 tx=0x7f9e6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:36.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.611+0000 7f9e72000640 1 -- 172.21.15.67:0/4245110711 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f9e74016560 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.807+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e7c074e30 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.797 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.807+0000 7f9e72000640 1 -- 172.21.15.67:0/4245110711 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f9e740f0e00 con 0x7f9e7c15dbc0 2024-04-17T10:38:36.798 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:36.798 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:36.798 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:36.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e58066fb0 msgr2=0x7f9e58069470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:36.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e58066fb0 0x7f9e58069470 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c005890 tx=0x7f9e6c009040 comp rx=0 tx=0).stop 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c15dbc0 msgr2=0x7f9e7c154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c15dbc0 0x7f9e7c154680 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7f9e7401a510 tx=0x7f9e74019680 comp rx=0 tx=0).stop 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 shutdown_connections 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9e58066fb0 0x7f9e58069470 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e7c15dbc0 0x7f9e7c154680 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f9e7c072370 0x7f9e7c14f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 --2- 172.21.15.67:0/4245110711 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f9e7c06c410 0x7f9e7c146260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 >> 172.21.15.67:0/4245110711 conn(0x7f9e7c0af670 msgr2=0x7f9e7c0b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 shutdown_connections 2024-04-17T10:38:36.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:36.811+0000 7f9e81000640 1 -- 172.21.15.67:0/4245110711 wait complete. 2024-04-17T10:38:37.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:37.221 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:38.222 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f3ba00640 1 Processor -- start 2024-04-17T10:38:38.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f3ba00640 1 -- start start 2024-04-17T10:38:38.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f3ba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34071770 0x7f1f340cca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f3ba00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3406e2f0 con 0x7f1f34145040 2024-04-17T10:38:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f3ba00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3406d6e0 con 0x7f1f34071310 2024-04-17T10:38:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f3ba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340a86c0 con 0x7f1f34071770 2024-04-17T10:38:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f39000640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1f34145040 0x7f1f3406e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36524/0 (socket says 172.21.15.67:36524) 2024-04-17T10:38:38.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f39000640 1 -- 172.21.15.67:0/994214147 learned_addr learned my addr 172.21.15.67:0/994214147 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f39a00640 1 --2- 172.21.15.67:0/994214147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34071770 0x7f1f340cca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f33400640 1 -- 172.21.15.67:0/994214147 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12282874 0 0) 0x7f1f3406e2f0 con 0x7f1f34145040 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f33400640 1 -- 172.21.15.67:0/994214147 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f1c003530 con 0x7f1f34145040 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f33400640 1 -- 172.21.15.67:0/994214147 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824752941 0 0) 0x7f1f3406d6e0 con 0x7f1f34071310 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f33400640 1 -- 172.21.15.67:0/994214147 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f3406e2f0 con 0x7f1f34071310 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f39a00640 1 -- 172.21.15.67:0/994214147 >> v1:172.21.15.86:6789/0 conn(0x7f1f34071310 legacy=0x7f1f340cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f39a00640 1 -- 172.21.15.67:0/994214147 >> v1:172.21.15.67:6789/0 conn(0x7f1f34145040 legacy=0x7f1f3406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:38.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.387+0000 7f1f39a00640 1 -- 172.21.15.67:0/994214147 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3406d9c0 con 0x7f1f34071770 2024-04-17T10:38:38.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f39a00640 1 --2- 172.21.15.67:0/994214147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34071770 0x7f1f340cca10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009dc0 tx=0x7f1f2c01f170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24fc462ac8d4b105 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f33400640 1 -- 172.21.15.67:0/994214147 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c0b4070 con 0x7f1f34071770 2024-04-17T10:38:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f33400640 1 -- 172.21.15.67:0/994214147 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c0af600 con 0x7f1f34071770 2024-04-17T10:38:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f33400640 1 -- 172.21.15.67:0/994214147 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c0af9d0 con 0x7f1f34071770 2024-04-17T10:38:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f3ba00640 1 -- 172.21.15.67:0/994214147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34071770 msgr2=0x7f1f340cca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:38.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/994214147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34071770 0x7f1f340cca10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009dc0 tx=0x7f1f2c01f170 comp rx=0 tx=0).stop 2024-04-17T10:38:38.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f3ba00640 1 -- 172.21.15.67:0/994214147 shutdown_connections 2024-04-17T10:38:38.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/994214147 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34071770 0x7f1f340cca10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:38.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f3ba00640 1 -- 172.21.15.67:0/994214147 >> 172.21.15.67:0/994214147 conn(0x7f1f3406ac90 msgr2=0x7f1f3406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:38.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f3ba00640 1 -- 172.21.15.67:0/994214147 shutdown_connections 2024-04-17T10:38:38.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.391+0000 7f1f3ba00640 1 -- 172.21.15.67:0/994214147 wait complete. 2024-04-17T10:38:38.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 Processor -- start 2024-04-17T10:38:38.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 -- start start 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f34071310 0x7f1f34136070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f34071770 0x7f1f34140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34145040 0x7f1f340adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3406df80 con 0x7f1f34071310 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3415bf10 con 0x7f1f34071770 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34071ba0 con 0x7f1f34145040 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f34071310 0x7f1f34136070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f34071310 0x7f1f34136070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41454/0 (socket says 172.21.15.67:41454) 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 -- 172.21.15.67:0/4051310497 learned_addr learned my addr 172.21.15.67:0/4051310497 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:38.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39a00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34145040 0x7f1f340adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f33e00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f34071770 0x7f1f34140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 -- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34145040 msgr2=0x7f1f340adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34145040 0x7f1f340adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 -- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f34071770 msgr2=0x7f1f34140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f34071770 0x7f1f34140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 -- 172.21.15.67:0/4051310497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f341365b0 con 0x7f1f34071310 2024-04-17T10:38:38.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f39000640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f34071310 0x7f1f34136070 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800c450 tx=0x7f1f2800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:38.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f30c00640 1 -- 172.21.15.67:0/4051310497 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28017070 con 0x7f1f34071310 2024-04-17T10:38:38.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f30c00640 1 -- 172.21.15.67:0/4051310497 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f28005cf0 con 0x7f1f34071310 2024-04-17T10:38:38.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3415ca50 con 0x7f1f34071310 2024-04-17T10:38:38.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f30c00640 1 -- 172.21.15.67:0/4051310497 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f280072b0 con 0x7f1f34071310 2024-04-17T10:38:38.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3415d010 con 0x7f1f34071310 2024-04-17T10:38:38.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.395+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3415d500 con 0x7f1f34071310 2024-04-17T10:38:38.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.399+0000 7f1f30c00640 1 -- 172.21.15.67:0/4051310497 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1f2801a070 con 0x7f1f34071310 2024-04-17T10:38:38.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.399+0000 7f1f30c00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f14066f60 0x7f1f14069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:38.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.399+0000 7f1f30c00640 1 -- 172.21.15.67:0/4051310497 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1f28084500 con 0x7f1f34071310 2024-04-17T10:38:38.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.399+0000 7f1f33e00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f14066f60 0x7f1f14069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:38.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.399+0000 7f1f33e00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f14066f60 0x7f1f14069420 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1f341f6ed0 tx=0x7f1f20009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:38.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.403+0000 7f1f30c00640 1 -- 172.21.15.67:0/4051310497 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1f28054c90 con 0x7f1f34071310 2024-04-17T10:38:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f340cbaa0 con 0x7f1f34071310 2024-04-17T10:38:38.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f30c00640 1 -- 172.21.15.67:0/4051310497 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1f28058d30 con 0x7f1f34071310 2024-04-17T10:38:38.588 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:38.588 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:38.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f14066f60 msgr2=0x7f1f14069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:38.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f14066f60 0x7f1f14069420 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1f341f6ed0 tx=0x7f1f20009040 comp rx=0 tx=0).stop 2024-04-17T10:38:38.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f34071310 msgr2=0x7f1f34136070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f34071310 0x7f1f34136070 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800c450 tx=0x7f1f2800c910 comp rx=0 tx=0).stop 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 shutdown_connections 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1f14066f60 0x7f1f14069420 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1f34145040 0x7f1f340adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1f34071770 0x7f1f34140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 --2- 172.21.15.67:0/4051310497 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1f34071310 0x7f1f34136070 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 >> 172.21.15.67:0/4051310497 conn(0x7f1f3406ac90 msgr2=0x7f1f3413ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 shutdown_connections 2024-04-17T10:38:38.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:38.599+0000 7f1f3ba00640 1 -- 172.21.15.67:0/4051310497 wait complete. 2024-04-17T10:38:39.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:39.011 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:40.013 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5f400640 1 Processor -- start 2024-04-17T10:38:40.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5f400640 1 -- start start 2024-04-17T10:38:40.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5f400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbf581446c0 0x7fbf58144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:40.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5f400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf58144aa0 0x7fbf580d0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:40.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5f400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf580b6640 con 0x7fbf58071840 2024-04-17T10:38:40.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5f400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf580b51a0 con 0x7fbf58144aa0 2024-04-17T10:38:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5f400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf581456d0 con 0x7fbf581446c0 2024-04-17T10:38:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5d400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbf58071840 0x7fbf5815f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36532/0 (socket says 172.21.15.67:36532) 2024-04-17T10:38:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5d400640 1 -- 172.21.15.67:0/2024866783 learned_addr learned my addr 172.21.15.67:0/2024866783 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf4fe00640 1 --2- 172.21.15.67:0/2024866783 >> v2:172.21.15.67:3300/0 conn(0x7fbf58144aa0 0x7fbf580d0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf5ca00640 1 --2- 172.21.15.67:0/2024866783 >> v2:172.21.15.86:3300/0 conn(0x7fbf581446c0 0x7fbf58144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:40.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf4fe00640 1 -- 172.21.15.67:0/2024866783 >> v2:172.21.15.86:3300/0 conn(0x7fbf581446c0 msgr2=0x7fbf58144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf4fe00640 1 --2- 172.21.15.67:0/2024866783 >> v2:172.21.15.86:3300/0 conn(0x7fbf581446c0 0x7fbf58144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf4fe00640 1 -- 172.21.15.67:0/2024866783 >> v1:172.21.15.67:6789/0 conn(0x7fbf58071840 legacy=0x7fbf5815f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf57e00640 1 -- 172.21.15.67:0/2024866783 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938832774 0 0) 0x7fbf580b6640 con 0x7fbf58071840 2024-04-17T10:38:40.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.179+0000 7fbf4fe00640 1 -- 172.21.15.67:0/2024866783 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf580b4870 con 0x7fbf58144aa0 2024-04-17T10:38:40.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf4fe00640 1 --2- 172.21.15.67:0/2024866783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf58144aa0 0x7fbf580d0910 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fbf580d2900 tx=0x7fbf4001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959f986c6b23f9e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:40.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf57e00640 1 -- 172.21.15.67:0/2024866783 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf400ab650 con 0x7fbf58144aa0 2024-04-17T10:38:40.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf57e00640 1 -- 172.21.15.67:0/2024866783 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf400b0070 con 0x7fbf58144aa0 2024-04-17T10:38:40.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf57e00640 1 -- 172.21.15.67:0/2024866783 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf400aba10 con 0x7fbf58144aa0 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 -- 172.21.15.67:0/2024866783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf58144aa0 msgr2=0x7fbf580d0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 --2- 172.21.15.67:0/2024866783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf58144aa0 0x7fbf580d0910 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fbf580d2900 tx=0x7fbf4001c450 comp rx=0 tx=0).stop 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 -- 172.21.15.67:0/2024866783 shutdown_connections 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 --2- 172.21.15.67:0/2024866783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf58144aa0 0x7fbf580d0910 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 --2- 172.21.15.67:0/2024866783 >> v2:172.21.15.86:3300/0 conn(0x7fbf581446c0 0x7fbf58144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 -- 172.21.15.67:0/2024866783 >> 172.21.15.67:0/2024866783 conn(0x7fbf580b34d0 msgr2=0x7fbf580b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 -- 172.21.15.67:0/2024866783 shutdown_connections 2024-04-17T10:38:40.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.183+0000 7fbf5f400640 1 -- 172.21.15.67:0/2024866783 wait complete. 2024-04-17T10:38:40.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 Processor -- start 2024-04-17T10:38:40.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 -- start start 2024-04-17T10:38:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf58071840 0x7fbf580a8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf581446c0 0x7fbf58141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf58144aa0 0x7fbf580aa650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:40.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf580d1260 con 0x7fbf581446c0 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5815c0d0 con 0x7fbf58144aa0 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf58168a40 con 0x7fbf58071840 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf581446c0 0x7fbf58141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf581446c0 0x7fbf58141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41474/0 (socket says 172.21.15.67:41474) 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 -- 172.21.15.67:0/1511655538 learned_addr learned my addr 172.21.15.67:0/1511655538 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5ca00640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf58071840 0x7fbf580a8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5d400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf58144aa0 0x7fbf580aa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 -- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf58071840 msgr2=0x7fbf580a8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf58071840 0x7fbf580a8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 -- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf58144aa0 msgr2=0x7fbf580aa650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf58144aa0 0x7fbf580aa650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 -- 172.21.15.67:0/1511655538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf580a9760 con 0x7fbf581446c0 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5d400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf58144aa0 0x7fbf580aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:40.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf4fe00640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf581446c0 0x7fbf58141e10 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fbf400ad040 tx=0x7fbf4001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:40.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf55600640 1 -- 172.21.15.67:0/1511655538 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf400bf610 con 0x7fbf581446c0 2024-04-17T10:38:40.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf55600640 1 -- 172.21.15.67:0/1511655538 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf400bf7d0 con 0x7fbf581446c0 2024-04-17T10:38:40.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf580a9a40 con 0x7fbf581446c0 2024-04-17T10:38:40.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf55600640 1 -- 172.21.15.67:0/1511655538 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf400bfbc0 con 0x7fbf581446c0 2024-04-17T10:38:40.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.187+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf580aa000 con 0x7fbf581446c0 2024-04-17T10:38:40.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.191+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf58070bb0 con 0x7fbf581446c0 2024-04-17T10:38:40.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.191+0000 7fbf55600640 1 -- 172.21.15.67:0/1511655538 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbf400bd020 con 0x7fbf581446c0 2024-04-17T10:38:40.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.191+0000 7fbf55600640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbf34066f60 0x7fbf34069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:40.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.191+0000 7fbf55600640 1 -- 172.21.15.67:0/1511655538 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbf4011fce0 con 0x7fbf581446c0 2024-04-17T10:38:40.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.191+0000 7fbf5ca00640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbf34066f60 0x7fbf34069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:40.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.191+0000 7fbf5ca00640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbf34066f60 0x7fbf34069420 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbf48007a60 tx=0x7fbf48008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:40.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.195+0000 7fbf55600640 1 -- 172.21.15.67:0/1511655538 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbf40016070 con 0x7fbf581446c0 2024-04-17T10:38:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf580b41a0 con 0x7fbf581446c0 2024-04-17T10:38:40.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf55600640 1 -- 172.21.15.67:0/1511655538 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fbf400f0470 con 0x7fbf581446c0 2024-04-17T10:38:40.383 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:40.384 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:40.384 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:40.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbf34066f60 msgr2=0x7fbf34069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:40.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbf34066f60 0x7fbf34069420 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbf48007a60 tx=0x7fbf48008040 comp rx=0 tx=0).stop 2024-04-17T10:38:40.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf581446c0 msgr2=0x7fbf58141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:40.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf581446c0 0x7fbf58141e10 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fbf400ad040 tx=0x7fbf4001ac90 comp rx=0 tx=0).stop 2024-04-17T10:38:40.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 shutdown_connections 2024-04-17T10:38:40.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbf34066f60 0x7fbf34069420 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbf58144aa0 0x7fbf580aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf581446c0 0x7fbf58141e10 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 --2- 172.21.15.67:0/1511655538 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbf58071840 0x7fbf580a8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 >> 172.21.15.67:0/1511655538 conn(0x7fbf580b34d0 msgr2=0x7fbf580b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 shutdown_connections 2024-04-17T10:38:40.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:40.395+0000 7fbf5f400640 1 -- 172.21.15.67:0/1511655538 wait complete. 2024-04-17T10:38:40.809 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:40.810 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:41.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:41.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.971+0000 7fb3f2200640 1 Processor -- start 2024-04-17T10:38:41.963 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.971+0000 7fb3f2200640 1 -- start start 2024-04-17T10:38:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3f2200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fb3ec06e6e0 0x7fb3ec0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3f2200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3ec06eac0 0x7fb3ec0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3f2200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ec15ae40 con 0x7fb3ec143ae0 2024-04-17T10:38:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3f2200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ec0cf430 con 0x7fb3ec06eac0 2024-04-17T10:38:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3f2200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ec071440 con 0x7fb3ec06e6e0 2024-04-17T10:38:41.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3e2a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb3ec06eac0 0x7fb3ec0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3ebe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb3ec143ae0 0x7fb3ec143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53708/0 (socket says 172.21.15.67:53708) 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3ebe00640 1 -- 172.21.15.67:0/1097393204 learned_addr learned my addr 172.21.15.67:0/1097393204 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3eb400640 1 --2- 172.21.15.67:0/1097393204 >> v2:172.21.15.86:3300/0 conn(0x7fb3ec06e6e0 0x7fb3ec0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3e2a00640 1 -- 172.21.15.67:0/1097393204 >> v2:172.21.15.86:3300/0 conn(0x7fb3ec06e6e0 msgr2=0x7fb3ec0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3e2a00640 1 --2- 172.21.15.67:0/1097393204 >> v2:172.21.15.86:3300/0 conn(0x7fb3ec06e6e0 0x7fb3ec0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3e2a00640 1 -- 172.21.15.67:0/1097393204 >> v1:172.21.15.67:6789/0 conn(0x7fb3ec143ae0 legacy=0x7fb3ec143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3e2a00640 1 -- 172.21.15.67:0/1097393204 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec071720 con 0x7fb3ec06eac0 2024-04-17T10:38:41.965 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.975+0000 7fb3eaa00640 1 -- 172.21.15.67:0/1097393204 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 595838844 0 0) 0x7fb3ec15ae40 con 0x7fb3ec143ae0 2024-04-17T10:38:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3e2a00640 1 --2- 172.21.15.67:0/1097393204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0ceef0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0cf620 tx=0x7fb3d401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf65ee171c4ffce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3eaa00640 1 -- 172.21.15.67:0/1097393204 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d40b0070 con 0x7fb3ec06eac0 2024-04-17T10:38:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3eaa00640 1 -- 172.21.15.67:0/1097393204 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d40ab600 con 0x7fb3ec06eac0 2024-04-17T10:38:41.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3eaa00640 1 -- 172.21.15.67:0/1097393204 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d40ab9d0 con 0x7fb3ec06eac0 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 -- 172.21.15.67:0/1097393204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 msgr2=0x7fb3ec0ceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 --2- 172.21.15.67:0/1097393204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0ceef0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0cf620 tx=0x7fb3d401c3a0 comp rx=0 tx=0).stop 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 -- 172.21.15.67:0/1097393204 shutdown_connections 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 --2- 172.21.15.67:0/1097393204 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0ceef0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 --2- 172.21.15.67:0/1097393204 >> v2:172.21.15.86:3300/0 conn(0x7fb3ec06e6e0 0x7fb3ec0ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 -- 172.21.15.67:0/1097393204 >> 172.21.15.67:0/1097393204 conn(0x7fb3ec06ac90 msgr2=0x7fb3ec06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 -- 172.21.15.67:0/1097393204 shutdown_connections 2024-04-17T10:38:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 -- 172.21.15.67:0/1097393204 wait complete. 2024-04-17T10:38:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 Processor -- start 2024-04-17T10:38:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.979+0000 7fb3f2200640 1 -- start start 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3ec06e6e0 0x7fb3ec13f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3ec143ae0 0x7fb3ec0c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec06dd00 con 0x7fb3ec06eac0 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec06db80 con 0x7fb3ec143ae0 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec06de80 con 0x7fb3ec06e6e0 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0c46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47676/0 (socket says 172.21.15.67:47676) 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 -- 172.21.15.67:0/2038880876 learned_addr learned my addr 172.21.15.67:0/2038880876 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3ebe00640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3ec143ae0 0x7fb3ec0c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3eb400640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3ec06e6e0 0x7fb3ec13f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 -- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3ec06e6e0 msgr2=0x7fb3ec13f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3ec06e6e0 0x7fb3ec13f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:41.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 -- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3ec143ae0 msgr2=0x7fb3ec0c4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3ec143ae0 0x7fb3ec0c4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 -- 172.21.15.67:0/2038880876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec0c6390 con 0x7fb3ec06eac0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e2a00640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0c46e0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4018fb0 tx=0x7fb3d4019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e3e00640 1 -- 172.21.15.67:0/2038880876 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d40b8020 con 0x7fb3ec06eac0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e3e00640 1 -- 172.21.15.67:0/2038880876 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d40b4410 con 0x7fb3ec06eac0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3e3e00640 1 -- 172.21.15.67:0/2038880876 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d40b0040 con 0x7fb3ec06eac0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec0c52f0 con 0x7fb3ec06eac0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec0c58b0 con 0x7fb3ec06eac0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.983+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec072d50 con 0x7fb3ec06eac0 2024-04-17T10:38:41.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.987+0000 7fb3e3e00640 1 -- 172.21.15.67:0/2038880876 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb3d40bb020 con 0x7fb3ec06eac0 2024-04-17T10:38:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.987+0000 7fb3e3e00640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3c8066f60 0x7fb3c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:41.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.987+0000 7fb3e3e00640 1 -- 172.21.15.67:0/2038880876 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fb3d411ee20 con 0x7fb3ec06eac0 2024-04-17T10:38:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.987+0000 7fb3eb400640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3c8066f60 0x7fb3c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:41.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.987+0000 7fb3eb400640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3c8066f60 0x7fb3c8069420 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0b10e0 tx=0x7fb3dc008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:41.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:41.991+0000 7fb3e3e00640 1 -- 172.21.15.67:0/2038880876 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb3d40c2030 con 0x7fb3ec06eac0 2024-04-17T10:38:42.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.187+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3ec074e30 con 0x7fb3ec06eac0 2024-04-17T10:38:42.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.187+0000 7fb3e3e00640 1 -- 172.21.15.67:0/2038880876 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fb3d4016050 con 0x7fb3ec06eac0 2024-04-17T10:38:42.179 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:42.179 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:42.179 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3c8066f60 msgr2=0x7fb3c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3c8066f60 0x7fb3c8069420 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec0b10e0 tx=0x7fb3dc008040 comp rx=0 tx=0).stop 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 msgr2=0x7fb3ec0c46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0c46e0 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4018fb0 tx=0x7fb3d4019140 comp rx=0 tx=0).stop 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 shutdown_connections 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb3c8066f60 0x7fb3c8069420 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb3ec143ae0 0x7fb3ec0c4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3ec06eac0 0x7fb3ec0c46e0 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 --2- 172.21.15.67:0/2038880876 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb3ec06e6e0 0x7fb3ec13f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 >> 172.21.15.67:0/2038880876 conn(0x7fb3ec06ac90 msgr2=0x7fb3ec0b19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 shutdown_connections 2024-04-17T10:38:42.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:42.191+0000 7fb3f2200640 1 -- 172.21.15.67:0/2038880876 wait complete. 2024-04-17T10:38:42.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:42.602 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:43.604 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6689200640 1 Processor -- start 2024-04-17T10:38:43.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6689200640 1 -- start start 2024-04-17T10:38:43.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6689200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f668406eac0 0x7f6684143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:43.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6689200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668415ae40 con 0x7f66840cf0c0 2024-04-17T10:38:43.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6689200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6684071440 con 0x7f668406e6e0 2024-04-17T10:38:43.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6689200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6684071720 con 0x7f668406eac0 2024-04-17T10:38:43.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6682800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f66840cf0c0 0x7f6684144020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53712/0 (socket says 172.21.15.67:53712) 2024-04-17T10:38:43.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6682800640 1 -- 172.21.15.67:0/26056550 learned_addr learned my addr 172.21.15.67:0/26056550 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:43.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6681400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f668406eac0 0x7f6684143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:43.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6680a00640 1 -- 172.21.15.67:0/26056550 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765428273 0 0) 0x7f668415ae40 con 0x7f66840cf0c0 2024-04-17T10:38:43.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6680a00640 1 -- 172.21.15.67:0/26056550 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6668003530 con 0x7f66840cf0c0 2024-04-17T10:38:43.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6681400640 1 -- 172.21.15.67:0/26056550 >> v1:172.21.15.86:6789/0 conn(0x7f668406e6e0 legacy=0x7f66840ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:43.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6681400640 1 -- 172.21.15.67:0/26056550 >> v1:172.21.15.67:6789/0 conn(0x7f66840cf0c0 legacy=0x7f6684144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:43.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.767+0000 7f6681400640 1 -- 172.21.15.67:0/26056550 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6684071a00 con 0x7f668406eac0 2024-04-17T10:38:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6681400640 1 --2- 172.21.15.67:0/26056550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406eac0 0x7f6684143ae0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f66700097c0 tx=0x7f667001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07499ac8c5a6324 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:43.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6680a00640 1 -- 172.21.15.67:0/26056550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700ab600 con 0x7f668406eac0 2024-04-17T10:38:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6680a00640 1 -- 172.21.15.67:0/26056550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66700b0070 con 0x7f668406eac0 2024-04-17T10:38:43.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6680a00640 1 -- 172.21.15.67:0/26056550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66700ab9a0 con 0x7f668406eac0 2024-04-17T10:38:43.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6689200640 1 -- 172.21.15.67:0/26056550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406eac0 msgr2=0x7f6684143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:43.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6689200640 1 --2- 172.21.15.67:0/26056550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406eac0 0x7f6684143ae0 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f66700097c0 tx=0x7f667001c3e0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6689200640 1 -- 172.21.15.67:0/26056550 shutdown_connections 2024-04-17T10:38:43.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6689200640 1 --2- 172.21.15.67:0/26056550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406eac0 0x7f6684143ae0 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6689200640 1 -- 172.21.15.67:0/26056550 >> 172.21.15.67:0/26056550 conn(0x7f668406ac90 msgr2=0x7f668406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:43.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6689200640 1 -- 172.21.15.67:0/26056550 shutdown_connections 2024-04-17T10:38:43.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.771+0000 7f6689200640 1 -- 172.21.15.67:0/26056550 wait complete. 2024-04-17T10:38:43.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 Processor -- start 2024-04-17T10:38:43.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 -- start start 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406e6e0 0x7f66841f7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f668406eac0 0x7f668413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66840cf0c0 0x7f66840c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66841680b0 con 0x7f668406e6e0 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6684167f30 con 0x7f66840cf0c0 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6684168230 con 0x7f668406eac0 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406e6e0 0x7f66841f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406e6e0 0x7f66841f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47692/0 (socket says 172.21.15.67:47692) 2024-04-17T10:38:43.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 -- 172.21.15.67:0/2747848106 learned_addr learned my addr 172.21.15.67:0/2747848106 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6682800640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66840cf0c0 0x7f66840c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681400640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f668406eac0 0x7f668413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 -- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f668406eac0 msgr2=0x7f668413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f668406eac0 0x7f668413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 -- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66840cf0c0 msgr2=0x7f66840c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66840cf0c0 0x7f66840c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 -- 172.21.15.67:0/2747848106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668413f510 con 0x7f668406e6e0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6681e00640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406e6e0 0x7f66841f7500 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f667400c420 tx=0x7f667400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f667a000640 1 -- 172.21.15.67:0/2747848106 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66740052d0 con 0x7f668406e6e0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f667a000640 1 -- 172.21.15.67:0/2747848106 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6674005470 con 0x7f668406e6e0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f667a000640 1 -- 172.21.15.67:0/2747848106 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6674007260 con 0x7f668406e6e0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668413f7f0 con 0x7f668406e6e0 2024-04-17T10:38:43.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668413fdb0 con 0x7f668406e6e0 2024-04-17T10:38:43.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.775+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6684072d50 con 0x7f668406e6e0 2024-04-17T10:38:43.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.779+0000 7f667a000640 1 -- 172.21.15.67:0/2747848106 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6674007400 con 0x7f668406e6e0 2024-04-17T10:38:43.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.779+0000 7f667a000640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6664066f60 0x7f6664069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:43.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.779+0000 7f667a000640 1 -- 172.21.15.67:0/2747848106 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6674084640 con 0x7f668406e6e0 2024-04-17T10:38:43.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.779+0000 7f6681400640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6664066f60 0x7f6664069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:43.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.779+0000 7f6681400640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6664066f60 0x7f6664069420 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f66700af000 tx=0x7f66700a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:43.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.783+0000 7f667a000640 1 -- 172.21.15.67:0/2747848106 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6674054e70 con 0x7f668406e6e0 2024-04-17T10:38:43.967 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6684074e30 con 0x7f668406e6e0 2024-04-17T10:38:43.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f667a000640 1 -- 172.21.15.67:0/2747848106 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6674058f10 con 0x7f668406e6e0 2024-04-17T10:38:43.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:43.968 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:43.969 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:43.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6664066f60 msgr2=0x7f6664069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:43.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6664066f60 0x7f6664069420 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f66700af000 tx=0x7f66700a7000 comp rx=0 tx=0).stop 2024-04-17T10:38:43.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406e6e0 msgr2=0x7f66841f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:43.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406e6e0 0x7f66841f7500 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f667400c420 tx=0x7f667400c8e0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 shutdown_connections 2024-04-17T10:38:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6664066f60 0x7f6664069420 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f66840cf0c0 0x7f66840c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f668406eac0 0x7f668413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 --2- 172.21.15.67:0/2747848106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f668406e6e0 0x7f66841f7500 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 >> 172.21.15.67:0/2747848106 conn(0x7f668406ac90 msgr2=0x7f668406d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 shutdown_connections 2024-04-17T10:38:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:43.979+0000 7f6689200640 1 -- 172.21.15.67:0/2747848106 wait complete. 2024-04-17T10:38:44.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:44.392 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:45.394 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:45.547 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd23000640 1 Processor -- start 2024-04-17T10:38:45.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd23000640 1 -- start start 2024-04-17T10:38:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd23000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fcd1c0b07b0 0x7fcd1c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd23000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd1c071300 0x7fcd1c06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd23000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd1c0b5980 con 0x7fcd1c15dd10 2024-04-17T10:38:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd23000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c071770 con 0x7fcd1c071300 2024-04-17T10:38:45.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd23000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd1c0b1280 con 0x7fcd1c0b07b0 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd21000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd1c071300 0x7fcd1c06c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd21000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcd1c071300 0x7fcd1c06c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47694/0 (socket says 172.21.15.67:47694) 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd21000640 1 -- 172.21.15.67:0/670379647 learned_addr learned my addr 172.21.15.67:0/670379647 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd13e00640 1 --2- 172.21.15.67:0/670379647 >> v2:172.21.15.86:3300/0 conn(0x7fcd1c0b07b0 0x7fcd1c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd21000640 1 -- 172.21.15.67:0/670379647 >> v2:172.21.15.86:3300/0 conn(0x7fcd1c0b07b0 msgr2=0x7fcd1c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd21000640 1 --2- 172.21.15.67:0/670379647 >> v2:172.21.15.86:3300/0 conn(0x7fcd1c0b07b0 0x7fcd1c070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd21000640 1 -- 172.21.15.67:0/670379647 >> v1:172.21.15.86:6789/0 conn(0x7fcd1c15dd10 legacy=0x7fcd1c0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:45.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.559+0000 7fcd21000640 1 -- 172.21.15.67:0/670379647 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0b0cc0 con 0x7fcd1c071300 2024-04-17T10:38:45.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd21000640 1 --2- 172.21.15.67:0/670379647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c071300 0x7fcd1c06c6d0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c068850 tx=0x7fcd1401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beed5e4927dfaec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:45.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd1b400640 1 -- 172.21.15.67:0/670379647 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd140a57f0 con 0x7fcd1c071300 2024-04-17T10:38:45.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd1b400640 1 -- 172.21.15.67:0/670379647 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd140a5990 con 0x7fcd1c071300 2024-04-17T10:38:45.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd1b400640 1 -- 172.21.15.67:0/670379647 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd140a5d60 con 0x7fcd1c071300 2024-04-17T10:38:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 -- 172.21.15.67:0/670379647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c071300 msgr2=0x7fcd1c06c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 --2- 172.21.15.67:0/670379647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c071300 0x7fcd1c06c6d0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c068850 tx=0x7fcd1401c930 comp rx=0 tx=0).stop 2024-04-17T10:38:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 -- 172.21.15.67:0/670379647 shutdown_connections 2024-04-17T10:38:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 --2- 172.21.15.67:0/670379647 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c071300 0x7fcd1c06c6d0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 --2- 172.21.15.67:0/670379647 >> v2:172.21.15.86:3300/0 conn(0x7fcd1c0b07b0 0x7fcd1c070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 -- 172.21.15.67:0/670379647 >> 172.21.15.67:0/670379647 conn(0x7fcd1c0b34d0 msgr2=0x7fcd1c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 -- 172.21.15.67:0/670379647 shutdown_connections 2024-04-17T10:38:45.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.563+0000 7fcd23000640 1 -- 172.21.15.67:0/670379647 wait complete. 2024-04-17T10:38:45.558 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 Processor -- start 2024-04-17T10:38:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 -- start start 2024-04-17T10:38:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd1c071300 0x7fcd1c1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd1c0b07b0 0x7fcd1c0cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c15dd10 0x7fcd1c146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c06d420 con 0x7fcd1c15dd10 2024-04-17T10:38:45.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c15e0f0 con 0x7fcd1c0b07b0 2024-04-17T10:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c06d5a0 con 0x7fcd1c071300 2024-04-17T10:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c15dd10 0x7fcd1c146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c15dd10 0x7fcd1c146eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47698/0 (socket says 172.21.15.67:47698) 2024-04-17T10:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 -- 172.21.15.67:0/1402772830 learned_addr learned my addr 172.21.15.67:0/1402772830 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd1be00640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd1c071300 0x7fcd1c1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd13e00640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd1c0b07b0 0x7fcd1c0cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:45.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 -- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd1c071300 msgr2=0x7fcd1c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd1c071300 0x7fcd1c1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 -- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd1c0b07b0 msgr2=0x7fcd1c0cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd1c0b07b0 0x7fcd1c0cff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 -- 172.21.15.67:0/1402772830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c155910 con 0x7fcd1c15dd10 2024-04-17T10:38:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd21000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c15dd10 0x7fcd1c146eb0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fcd14019540 tx=0x7fcd140196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:45.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd18c00640 1 -- 172.21.15.67:0/1402772830 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd140bc020 con 0x7fcd1c15dd10 2024-04-17T10:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd18c00640 1 -- 172.21.15.67:0/1402772830 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd140b8670 con 0x7fcd1c15dd10 2024-04-17T10:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd18c00640 1 -- 172.21.15.67:0/1402772830 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd140b8a60 con 0x7fcd1c15dd10 2024-04-17T10:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c155bb0 con 0x7fcd1c15dd10 2024-04-17T10:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c156170 con 0x7fcd1c15dd10 2024-04-17T10:38:45.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.567+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c070bb0 con 0x7fcd1c15dd10 2024-04-17T10:38:45.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.571+0000 7fcd18c00640 1 -- 172.21.15.67:0/1402772830 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fcd140bf020 con 0x7fcd1c15dd10 2024-04-17T10:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.571+0000 7fcd18c00640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccf8066f60 0x7fccf8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.571+0000 7fcd18c00640 1 -- 172.21.15.67:0/1402772830 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fcd14120300 con 0x7fcd1c15dd10 2024-04-17T10:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.571+0000 7fcd1be00640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccf8066f60 0x7fccf8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:45.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.571+0000 7fcd1be00640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccf8066f60 0x7fccf8069420 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c001770 tx=0x7fcd0c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:45.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.575+0000 7fcd18c00640 1 -- 172.21.15.67:0/1402772830 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fcd140c6030 con 0x7fcd1c15dd10 2024-04-17T10:38:45.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.771+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd1c074e30 con 0x7fcd1c15dd10 2024-04-17T10:38:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.771+0000 7fcd18c00640 1 -- 172.21.15.67:0/1402772830 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fcd140165e0 con 0x7fcd1c15dd10 2024-04-17T10:38:45.761 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:45.761 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:45.761 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccf8066f60 msgr2=0x7fccf8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccf8066f60 0x7fccf8069420 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c001770 tx=0x7fcd0c009040 comp rx=0 tx=0).stop 2024-04-17T10:38:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c15dd10 msgr2=0x7fcd1c146eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c15dd10 0x7fcd1c146eb0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7fcd14019540 tx=0x7fcd140196d0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.763 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 shutdown_connections 2024-04-17T10:38:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fccf8066f60 0x7fccf8069420 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd1c15dd10 0x7fcd1c146eb0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcd1c0b07b0 0x7fcd1c0cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 --2- 172.21.15.67:0/1402772830 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fcd1c071300 0x7fcd1c1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 >> 172.21.15.67:0/1402772830 conn(0x7fcd1c0b34d0 msgr2=0x7fcd1c15c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 shutdown_connections 2024-04-17T10:38:45.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:45.775+0000 7fcd23000640 1 -- 172.21.15.67:0/1402772830 wait complete. 2024-04-17T10:38:46.185 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:46.185 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:47.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:47.338 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.347+0000 7f010a400640 1 Processor -- start 2024-04-17T10:38:47.339 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.347+0000 7f010a400640 1 -- start start 2024-04-17T10:38:47.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f010a400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f010415ecf0 0x7f010406e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:47.340 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f010a400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f010406ebb0 0x7f01040b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f010a400640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0104161700 con 0x7f0104071790 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f010a400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01041619e0 con 0x7f010415ecf0 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f010a400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0104161cc0 con 0x7f010406ebb0 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f00faa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f010415ecf0 0x7f010406e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f00faa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f010415ecf0 0x7f010406e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47704/0 (socket says 172.21.15.67:47704) 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f00faa00640 1 -- 172.21.15.67:0/4248933248 learned_addr learned my addr 172.21.15.67:0/4248933248 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f0103e00640 1 --2- 172.21.15.67:0/4248933248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f010406ebb0 0x7f01040b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f00faa00640 1 -- 172.21.15.67:0/4248933248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f010406ebb0 msgr2=0x7f01040b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f00faa00640 1 --2- 172.21.15.67:0/4248933248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f010406ebb0 0x7f01040b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f00faa00640 1 -- 172.21.15.67:0/4248933248 >> v1:172.21.15.86:6789/0 conn(0x7f0104071790 legacy=0x7f0104071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f00faa00640 1 -- 172.21.15.67:0/4248933248 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0104161fa0 con 0x7f010415ecf0 2024-04-17T10:38:47.341 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.351+0000 7f0103e00640 1 --2- 172.21.15.67:0/4248933248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f010406ebb0 0x7f01040b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:47.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f00faa00640 1 --2- 172.21.15.67:0/4248933248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010415ecf0 0x7f010406e670 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009870 tx=0x7f00ec01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f52e8c6fb155b602 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:47.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f0102a00640 1 -- 172.21.15.67:0/4248933248 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0a7070 con 0x7f010415ecf0 2024-04-17T10:38:47.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f0102a00640 1 -- 172.21.15.67:0/4248933248 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00ec0b0070 con 0x7f010415ecf0 2024-04-17T10:38:47.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f0102a00640 1 -- 172.21.15.67:0/4248933248 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0ab890 con 0x7f010415ecf0 2024-04-17T10:38:47.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 -- 172.21.15.67:0/4248933248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010415ecf0 msgr2=0x7f010406e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 --2- 172.21.15.67:0/4248933248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010415ecf0 0x7f010406e670 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f00ec009870 tx=0x7f00ec01c490 comp rx=0 tx=0).stop 2024-04-17T10:38:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 -- 172.21.15.67:0/4248933248 shutdown_connections 2024-04-17T10:38:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 --2- 172.21.15.67:0/4248933248 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f010406ebb0 0x7f01040b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 --2- 172.21.15.67:0/4248933248 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010415ecf0 0x7f010406e670 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 -- 172.21.15.67:0/4248933248 >> 172.21.15.67:0/4248933248 conn(0x7f01040cfa30 msgr2=0x7f01040cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 -- 172.21.15.67:0/4248933248 shutdown_connections 2024-04-17T10:38:47.345 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 -- 172.21.15.67:0/4248933248 wait complete. 2024-04-17T10:38:47.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 Processor -- start 2024-04-17T10:38:47.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.355+0000 7f010a400640 1 -- start start 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010406ebb0 0x7f0104156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0104071790 0x7f010415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010415ecf0 0x7f01040ae1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010416aa00 con 0x7f010406ebb0 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01040b3a00 con 0x7f010415ecf0 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010416ab80 con 0x7f0104071790 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010406ebb0 0x7f0104156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010406ebb0 0x7f0104156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47706/0 (socket says 172.21.15.67:47706) 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 -- 172.21.15.67:0/1061214953 learned_addr learned my addr 172.21.15.67:0/1061214953 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 -- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0104071790 msgr2=0x7f010415a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:47.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0104071790 0x7f010415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 -- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010415ecf0 msgr2=0x7f01040ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010415ecf0 0x7f01040ae1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 -- 172.21.15.67:0/1061214953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0104159580 con 0x7f010406ebb0 2024-04-17T10:38:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f0103400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010406ebb0 0x7f0104156ba0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f00f400c480 tx=0x7f00f400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:47.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f00fbe00640 1 -- 172.21.15.67:0/1061214953 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4012830 con 0x7f010406ebb0 2024-04-17T10:38:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f00fbe00640 1 -- 172.21.15.67:0/1061214953 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f4017070 con 0x7f010406ebb0 2024-04-17T10:38:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f00fbe00640 1 -- 172.21.15.67:0/1061214953 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4012c30 con 0x7f010406ebb0 2024-04-17T10:38:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0104159860 con 0x7f010406ebb0 2024-04-17T10:38:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0104159e20 con 0x7f010406ebb0 2024-04-17T10:38:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0104074e30 con 0x7f010406ebb0 2024-04-17T10:38:47.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.359+0000 7f00fbe00640 1 -- 172.21.15.67:0/1061214953 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f00f401c460 con 0x7f010406ebb0 2024-04-17T10:38:47.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.363+0000 7f00fbe00640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00e0066f60 0x7f00e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:47.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.363+0000 7f00fbe00640 1 -- 172.21.15.67:0/1061214953 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f00f40850b0 con 0x7f010406ebb0 2024-04-17T10:38:47.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.363+0000 7f00faa00640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00e0066f60 0x7f00e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:47.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.363+0000 7f00faa00640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00e0066f60 0x7f00e0069420 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0af000 tx=0x7f00ec004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:47.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.367+0000 7f00fbe00640 1 -- 172.21.15.67:0/1061214953 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f00f4055e90 con 0x7f010406ebb0 2024-04-17T10:38:47.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.559+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f010406e450 con 0x7f010406ebb0 2024-04-17T10:38:47.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.559+0000 7f00fbe00640 1 -- 172.21.15.67:0/1061214953 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f00f4055790 con 0x7f010406ebb0 2024-04-17T10:38:47.549 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:47.550 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:47.550 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00e0066f60 msgr2=0x7f00e0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00e0066f60 0x7f00e0069420 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0af000 tx=0x7f00ec004740 comp rx=0 tx=0).stop 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010406ebb0 msgr2=0x7f0104156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010406ebb0 0x7f0104156ba0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f00f400c480 tx=0x7f00f400c940 comp rx=0 tx=0).stop 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 shutdown_connections 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f00e0066f60 0x7f00e0069420 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f010415ecf0 0x7f01040ae1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0104071790 0x7f010415a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 --2- 172.21.15.67:0/1061214953 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f010406ebb0 0x7f0104156ba0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 >> 172.21.15.67:0/1061214953 conn(0x7f01040cfa30 msgr2=0x7f01040acd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 shutdown_connections 2024-04-17T10:38:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:47.563+0000 7f010a400640 1 -- 172.21.15.67:0/1061214953 wait complete. 2024-04-17T10:38:47.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:47.974 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:48.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:49.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.139+0000 7f0b77c00640 1 Processor -- start 2024-04-17T10:38:49.131 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.139+0000 7f0b77c00640 1 -- start start 2024-04-17T10:38:49.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b77c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7015dbc0 0x7f0b7006f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:49.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b77c00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0b70072370 0x7f0b701676d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:49.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b77c00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b700ad8f0 con 0x7f0b7006db90 2024-04-17T10:38:49.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b77c00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7015dfa0 con 0x7f0b70072370 2024-04-17T10:38:49.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b77c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7006faa0 con 0x7f0b7015dbc0 2024-04-17T10:38:49.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b6fe00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0b7006db90 0x7f0b70071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53728/0 (socket says 172.21.15.67:53728) 2024-04-17T10:38:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b6fe00640 1 -- 172.21.15.67:0/2472053456 learned_addr learned my addr 172.21.15.67:0/2472053456 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b75c00640 1 --2- 172.21.15.67:0/2472053456 >> v2:172.21.15.86:3300/0 conn(0x7f0b70072370 0x7f0b701676d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b75200640 1 --2- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7015dbc0 0x7f0b7006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:49.133 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b6f400640 1 -- 172.21.15.67:0/2472053456 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347712141 0 0) 0x7f0b700ad8f0 con 0x7f0b7006db90 2024-04-17T10:38:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b6f400640 1 -- 172.21.15.67:0/2472053456 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b58003530 con 0x7f0b7006db90 2024-04-17T10:38:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b75c00640 1 -- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7015dbc0 msgr2=0x7f0b7006f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b75c00640 1 --2- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7015dbc0 0x7f0b7006f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b75c00640 1 -- 172.21.15.67:0/2472053456 >> v1:172.21.15.67:6789/0 conn(0x7f0b7006db90 legacy=0x7f0b70071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b75c00640 1 -- 172.21.15.67:0/2472053456 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700adbd0 con 0x7f0b70072370 2024-04-17T10:38:49.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.143+0000 7f0b75200640 1 --2- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7015dbc0 0x7f0b7006f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b75c00640 1 --2- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b70072370 0x7f0b701676d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009d90 tx=0x7f0b6801c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=901ba16ccb99cfdf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b6f400640 1 -- 172.21.15.67:0/2472053456 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680b2070 con 0x7f0b70072370 2024-04-17T10:38:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b6f400640 1 -- 172.21.15.67:0/2472053456 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b680a57a0 con 0x7f0b70072370 2024-04-17T10:38:49.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b6f400640 1 -- 172.21.15.67:0/2472053456 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b680a5b50 con 0x7f0b70072370 2024-04-17T10:38:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 -- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b70072370 msgr2=0x7f0b701676d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 --2- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b70072370 0x7f0b701676d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009d90 tx=0x7f0b6801c8e0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 -- 172.21.15.67:0/2472053456 shutdown_connections 2024-04-17T10:38:49.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 --2- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b70072370 0x7f0b701676d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 --2- 172.21.15.67:0/2472053456 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b7015dbc0 0x7f0b7006f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 -- 172.21.15.67:0/2472053456 >> 172.21.15.67:0/2472053456 conn(0x7f0b700af670 msgr2=0x7f0b700b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 -- 172.21.15.67:0/2472053456 shutdown_connections 2024-04-17T10:38:49.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.147+0000 7f0b77c00640 1 -- 172.21.15.67:0/2472053456 wait complete. 2024-04-17T10:38:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 Processor -- start 2024-04-17T10:38:49.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 -- start start 2024-04-17T10:38:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b7006db90 0x7f0b701513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b70072370 0x7f0b70149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7015dbc0 0x7f0b700cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7016a7c0 con 0x7f0b7006db90 2024-04-17T10:38:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7016a640 con 0x7f0b7015dbc0 2024-04-17T10:38:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7016a940 con 0x7f0b70072370 2024-04-17T10:38:49.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b7006db90 0x7f0b701513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b7006db90 0x7f0b701513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47718/0 (socket says 172.21.15.67:47718) 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 -- 172.21.15.67:0/946158622 learned_addr learned my addr 172.21.15.67:0/946158622 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75c00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7015dbc0 0x7f0b700cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b6fe00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b70072370 0x7f0b70149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 -- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b70072370 msgr2=0x7f0b70149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b70072370 0x7f0b70149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 -- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7015dbc0 msgr2=0x7f0b700cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7015dbc0 0x7f0b700cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 -- 172.21.15.67:0/946158622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b70151a80 con 0x7f0b7006db90 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b6fe00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b70072370 0x7f0b70149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b75200640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b7006db90 0x7f0b701513f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e980 tx=0x7f0b6400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b6cc00640 1 -- 172.21.15.67:0/946158622 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64017070 con 0x7f0b7006db90 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b6cc00640 1 -- 172.21.15.67:0/946158622 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6400ca00 con 0x7f0b7006db90 2024-04-17T10:38:49.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b6cc00640 1 -- 172.21.15.67:0/946158622 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400cdd0 con 0x7f0b7006db90 2024-04-17T10:38:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b70151d60 con 0x7f0b7006db90 2024-04-17T10:38:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b70152320 con 0x7f0b7006db90 2024-04-17T10:38:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.151+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7009e260 con 0x7f0b7006db90 2024-04-17T10:38:49.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.155+0000 7f0b6cc00640 1 -- 172.21.15.67:0/946158622 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0b6401a070 con 0x7f0b7006db90 2024-04-17T10:38:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.155+0000 7f0b6cc00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b50066f60 0x7f0b50069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.155+0000 7f0b6cc00640 1 -- 172.21.15.67:0/946158622 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f0b64085220 con 0x7f0b7006db90 2024-04-17T10:38:49.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.155+0000 7f0b6fe00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b50066f60 0x7f0b50069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:49.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.155+0000 7f0b6fe00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b50066f60 0x7f0b50069420 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0b700b5560 tx=0x7f0b5c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:49.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.159+0000 7f0b6cc00640 1 -- 172.21.15.67:0/946158622 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0b64055f70 con 0x7f0b7006db90 2024-04-17T10:38:49.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b70153560 con 0x7f0b7006db90 2024-04-17T10:38:49.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b6cc00640 1 -- 172.21.15.67:0/946158622 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f0b64055870 con 0x7f0b7006db90 2024-04-17T10:38:49.344 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:49.344 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:49.345 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b50066f60 msgr2=0x7f0b50069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b50066f60 0x7f0b50069420 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f0b700b5560 tx=0x7f0b5c009040 comp rx=0 tx=0).stop 2024-04-17T10:38:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b7006db90 msgr2=0x7f0b701513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b7006db90 0x7f0b701513f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f0b6400e980 tx=0x7f0b6400ee40 comp rx=0 tx=0).stop 2024-04-17T10:38:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 shutdown_connections 2024-04-17T10:38:49.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0b50066f60 0x7f0b50069420 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0b7015dbc0 0x7f0b700cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0b70072370 0x7f0b70149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 --2- 172.21.15.67:0/946158622 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0b7006db90 0x7f0b701513f0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 >> 172.21.15.67:0/946158622 conn(0x7f0b700af670 msgr2=0x7f0b701674c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 shutdown_connections 2024-04-17T10:38:49.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:49.355+0000 7f0b77c00640 1 -- 172.21.15.67:0/946158622 wait complete. 2024-04-17T10:38:49.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:49.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:50.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.931+0000 7f4de9200640 1 Processor -- start 2024-04-17T10:38:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.931+0000 7f4de9200640 1 -- start start 2024-04-17T10:38:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de9200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f4de41466e0 0x7f4de406eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de9200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 0x7f4de4071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de9200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de4161740 con 0x7f4de4071850 2024-04-17T10:38:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de9200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de4161a20 con 0x7f4de41466e0 2024-04-17T10:38:50.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de9200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4161d00 con 0x7f4de4146da0 2024-04-17T10:38:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de2800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4de4071850 0x7f4de406d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:55152/0 (socket says 172.21.15.67:55152) 2024-04-17T10:38:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de2800640 1 -- 172.21.15.67:0/1993008125 learned_addr learned my addr 172.21.15.67:0/1993008125 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:38:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4ddbe00640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 0x7f4de4071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1e00640 1 --2- 172.21.15.67:0/1993008125 >> v2:172.21.15.86:3300/0 conn(0x7f4de41466e0 0x7f4de406eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284862572 0 0) 0x7f4de4161740 con 0x7f4de4071850 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc8003530 con 0x7f4de4071850 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 974747847 0 0) 0x7f4dc8003530 con 0x7f4de4071850 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 msgr2=0x7f4de4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 0x7f4de4071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 >> v2:172.21.15.86:3300/0 conn(0x7f4de41466e0 msgr2=0x7f4de406eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 --2- 172.21.15.67:0/1993008125 >> v2:172.21.15.86:3300/0 conn(0x7f4de41466e0 0x7f4de406eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4161fe0 con 0x7f4de4071850 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7f4ddc001a00 con 0x7f4de4071850 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 >> v1:172.21.15.67:6789/0 conn(0x7f4de4071850 legacy=0x7f4de406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dc80039c0 0x7f4dc8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc8004810 0x7f4dc8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.926 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 0x7f4dc80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4161a20 con 0x7f4dc80039c0 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4161d00 con 0x7f4dc8004810 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc800b9d0 con 0x7f4de4146da0 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1e00640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dc80039c0 0x7f4dc8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4ddbe00640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc8004810 0x7f4dc8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de2800640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 0x7f4dc80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1e00640 1 -- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 msgr2=0x7f4dc80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1e00640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de4146da0 0x7f4dc80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1e00640 1 -- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc8004810 msgr2=0x7f4dc8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1e00640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc8004810 0x7f4dc8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4de1e00640 1 -- 172.21.15.67:0/1993008125 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8003530 con 0x7f4dc80039c0 2024-04-17T10:38:50.927 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.935+0000 7f4ddbe00640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc8004810 0x7f4dc8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:50.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de1e00640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dc80039c0 0x7f4dc8003de0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009ae0 tx=0x7f4dd401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23202bf39dfdfe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:50.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd40096b0 con 0x7f4dc80039c0 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd40a8410 con 0x7f4dc80039c0 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de1400640 1 -- 172.21.15.67:0/1993008125 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd40a8800 con 0x7f4dc80039c0 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 -- 172.21.15.67:0/1993008125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dc80039c0 msgr2=0x7f4dc8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 --2- 172.21.15.67:0/1993008125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4dc80039c0 0x7f4dc8003de0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009ae0 tx=0x7f4dd401cab0 comp rx=0 tx=0).stop 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de2800640 1 -- 172.21.15.67:0/1993008125 reap_dead start 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 -- 172.21.15.67:0/1993008125 shutdown_connections 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 -- 172.21.15.67:0/1993008125 >> 172.21.15.67:0/1993008125 conn(0x7f4de406ac90 msgr2=0x7f4de406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 -- 172.21.15.67:0/1993008125 shutdown_connections 2024-04-17T10:38:50.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 -- 172.21.15.67:0/1993008125 wait complete. 2024-04-17T10:38:50.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 Processor -- start 2024-04-17T10:38:50.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.939+0000 7f4de9200640 1 -- start start 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc80039c0 0x7f4de4140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de415ce00 0x7f4de413cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de415d260 0x7f4de413faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4071c60 con 0x7f4de415d260 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40af200 con 0x7f4dc80039c0 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4168a30 con 0x7f4de415ce00 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de415d260 0x7f4de413faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de415d260 0x7f4de413faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49966/0 (socket says 172.21.15.67:49966) 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4ddbe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de415ce00 0x7f4de413cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 -- 172.21.15.67:0/1968670018 learned_addr learned my addr 172.21.15.67:0/1968670018 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de1e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc80039c0 0x7f4de4140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 -- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de415ce00 msgr2=0x7f4de413cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de415ce00 0x7f4de413cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 -- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc80039c0 msgr2=0x7f4de4140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc80039c0 0x7f4de4140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:50.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 -- 172.21.15.67:0/1968670018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40af800 con 0x7f4de415d260 2024-04-17T10:38:50.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de2800640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de415d260 0x7f4de413faf0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f4de406ba90 tx=0x7f4ddc007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:50.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4dda000640 1 -- 172.21.15.67:0/1968670018 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc00ea80 con 0x7f4de415d260 2024-04-17T10:38:50.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4dda000640 1 -- 172.21.15.67:0/1968670018 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ddc018070 con 0x7f4de415d260 2024-04-17T10:38:50.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de40afae0 con 0x7f4de415d260 2024-04-17T10:38:50.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4dda000640 1 -- 172.21.15.67:0/1968670018 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ddc00ee60 con 0x7f4de415d260 2024-04-17T10:38:50.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de40b00a0 con 0x7f4de415d260 2024-04-17T10:38:50.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.943+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de4072d50 con 0x7f4de415d260 2024-04-17T10:38:50.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.947+0000 7f4dda000640 1 -- 172.21.15.67:0/1968670018 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f4ddc01d460 con 0x7f4de415d260 2024-04-17T10:38:50.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.947+0000 7f4dda000640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc0066f60 0x7f4dc0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:50.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.947+0000 7f4dda000640 1 -- 172.21.15.67:0/1968670018 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f4ddc0853d0 con 0x7f4de415d260 2024-04-17T10:38:50.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.947+0000 7f4de1e00640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc0066f60 0x7f4dc0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:50.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.947+0000 7f4de1e00640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc0066f60 0x7f4dc0069420 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40af000 tx=0x7f4dd40b3040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:50.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:50.951+0000 7f4dda000640 1 -- 172.21.15.67:0/1968670018 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f4ddc055ad0 con 0x7f4de415d260 2024-04-17T10:38:51.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4de406b2f0 con 0x7f4de415d260 2024-04-17T10:38:51.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4dda000640 1 -- 172.21.15.67:0/1968670018 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f4ddc059b70 con 0x7f4de415d260 2024-04-17T10:38:51.136 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:51.136 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:51.136 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc0066f60 msgr2=0x7f4dc0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc0066f60 0x7f4dc0069420 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40af000 tx=0x7f4dd40b3040 comp rx=0 tx=0).stop 2024-04-17T10:38:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de415d260 msgr2=0x7f4de413faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de415d260 0x7f4de413faf0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f4de406ba90 tx=0x7f4ddc007bc0 comp rx=0 tx=0).stop 2024-04-17T10:38:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 shutdown_connections 2024-04-17T10:38:51.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f4dc0066f60 0x7f4dc0069420 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:51.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4de415d260 0x7f4de413faf0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:51.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f4de415ce00 0x7f4de413cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:51.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 --2- 172.21.15.67:0/1968670018 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f4dc80039c0 0x7f4de4140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:51.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 >> 172.21.15.67:0/1968670018 conn(0x7f4de406ac90 msgr2=0x7f4de406d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:51.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 shutdown_connections 2024-04-17T10:38:51.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:51.147+0000 7f4de9200640 1 -- 172.21.15.67:0/1968670018 wait complete. 2024-04-17T10:38:51.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:51.559 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:52.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:52.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 Processor -- start 2024-04-17T10:38:52.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 -- start start 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5cf806e6e0 0x7f5cf80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f5cf806eac0 0x7f5cf80ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf8143ae0 0x7f5cf8143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf80cf430 con 0x7f5cf806e6e0 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf815ae40 con 0x7f5cf806eac0 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.723+0000 7f5cfd000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf8071440 con 0x7f5cf8143ae0 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5cf806e6e0 0x7f5cf80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5cf806e6e0 0x7f5cf80ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49974/0 (socket says 172.21.15.67:49974) 2024-04-17T10:38:52.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 -- 172.21.15.67:0/4104238458 learned_addr learned my addr 172.21.15.67:0/4104238458 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf6800640 1 --2- 172.21.15.67:0/4104238458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf8143ae0 0x7f5cf8143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 -- 172.21.15.67:0/4104238458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf8143ae0 msgr2=0x7f5cf8143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cefe00640 1 --2- 172.21.15.67:0/4104238458 >> v2:172.21.15.86:3300/0 conn(0x7f5cf806eac0 0x7f5cf80ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 --2- 172.21.15.67:0/4104238458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf8143ae0 0x7f5cf8143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 -- 172.21.15.67:0/4104238458 >> v2:172.21.15.86:3300/0 conn(0x7f5cf806eac0 msgr2=0x7f5cf80ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 --2- 172.21.15.67:0/4104238458 >> v2:172.21.15.86:3300/0 conn(0x7f5cf806eac0 0x7f5cf80ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 -- 172.21.15.67:0/4104238458 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf8071720 con 0x7f5cf806e6e0 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf6800640 1 --2- 172.21.15.67:0/4104238458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf8143ae0 0x7f5cf8143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:52.716 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cefe00640 1 --2- 172.21.15.67:0/4104238458 >> v2:172.21.15.86:3300/0 conn(0x7f5cf806eac0 0x7f5cf80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:38:52.718 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5e00640 1 --2- 172.21.15.67:0/4104238458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf80ce9b0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f5cf806ef80 tx=0x7f5ce801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19fb5cbc231da04b server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5400640 1 -- 172.21.15.67:0/4104238458 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80ab620 con 0x7f5cf806e6e0 2024-04-17T10:38:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5400640 1 -- 172.21.15.67:0/4104238458 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce80b0070 con 0x7f5cf806e6e0 2024-04-17T10:38:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.727+0000 7f5cf5400640 1 -- 172.21.15.67:0/4104238458 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80ab9e0 con 0x7f5cf806e6e0 2024-04-17T10:38:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- 172.21.15.67:0/4104238458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 msgr2=0x7f5cf80ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:52.719 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 --2- 172.21.15.67:0/4104238458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf80ce9b0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f5cf806ef80 tx=0x7f5ce801c460 comp rx=0 tx=0).stop 2024-04-17T10:38:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- 172.21.15.67:0/4104238458 shutdown_connections 2024-04-17T10:38:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 --2- 172.21.15.67:0/4104238458 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf8143ae0 0x7f5cf8143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 --2- 172.21.15.67:0/4104238458 >> v2:172.21.15.86:3300/0 conn(0x7f5cf806eac0 0x7f5cf80ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 --2- 172.21.15.67:0/4104238458 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf80ce9b0 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- 172.21.15.67:0/4104238458 >> 172.21.15.67:0/4104238458 conn(0x7f5cf806ac90 msgr2=0x7f5cf806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- 172.21.15.67:0/4104238458 shutdown_connections 2024-04-17T10:38:52.720 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- 172.21.15.67:0/4104238458 wait complete. 2024-04-17T10:38:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 Processor -- start 2024-04-17T10:38:52.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- start start 2024-04-17T10:38:52.722 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf814dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf806eac0 0x7f5cf8141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf8143ae0 0x7f5cf8141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf815ebf0 con 0x7f5cf806e6e0 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf81444b0 con 0x7f5cf8143ae0 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cfd000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf815ed70 con 0x7f5cf806eac0 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf814dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf814dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49990/0 (socket says 172.21.15.67:49990) 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 -- 172.21.15.67:0/2958145348 learned_addr learned my addr 172.21.15.67:0/2958145348 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cefe00640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf806eac0 0x7f5cf8141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf6800640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf8143ae0 0x7f5cf8141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 -- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf806eac0 msgr2=0x7f5cf8141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf806eac0 0x7f5cf8141230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 -- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf8143ae0 msgr2=0x7f5cf8141770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf8143ae0 0x7f5cf8141770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.731+0000 7f5cf5e00640 1 -- 172.21.15.67:0/2958145348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf813c670 con 0x7f5cf806e6e0 2024-04-17T10:38:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cf5e00640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf814dd80 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80ad040 tx=0x7f5ce801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:52.724 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cee000640 1 -- 172.21.15.67:0/2958145348 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80abd90 con 0x7f5cf806e6e0 2024-04-17T10:38:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cee000640 1 -- 172.21.15.67:0/2958145348 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce80be3f0 con 0x7f5cf806e6e0 2024-04-17T10:38:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cee000640 1 -- 172.21.15.67:0/2958145348 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce80be7c0 con 0x7f5cf806e6e0 2024-04-17T10:38:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf813c950 con 0x7f5cf806e6e0 2024-04-17T10:38:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf813cf10 con 0x7f5cf806e6e0 2024-04-17T10:38:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf8072d50 con 0x7f5cf806e6e0 2024-04-17T10:38:52.725 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cee000640 1 -- 172.21.15.67:0/2958145348 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f5ce80b43f0 con 0x7f5cf806e6e0 2024-04-17T10:38:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cee000640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5cd4066f60 0x7f5cd4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:52.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.735+0000 7f5cee000640 1 -- 172.21.15.67:0/2958145348 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f5ce811f3a0 con 0x7f5cf806e6e0 2024-04-17T10:38:52.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.739+0000 7f5cefe00640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5cd4066f60 0x7f5cd4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:52.728 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.739+0000 7f5cefe00640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5cd4066f60 0x7f5cd4069420 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5cf815a350 tx=0x7f5ce0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.739+0000 7f5cee000640 1 -- 172.21.15.67:0/2958145348 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f5ce8016110 con 0x7f5cf806e6e0 2024-04-17T10:38:52.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.939+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5cf80b2c60 con 0x7f5cf806e6e0 2024-04-17T10:38:52.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.939+0000 7f5cee000640 1 -- 172.21.15.67:0/2958145348 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f5ce80efaa0 con 0x7f5cf806e6e0 2024-04-17T10:38:52.929 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:52.929 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:52.930 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5cd4066f60 msgr2=0x7f5cd4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:52.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5cd4066f60 0x7f5cd4069420 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5cf815a350 tx=0x7f5ce0008040 comp rx=0 tx=0).stop 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 msgr2=0x7f5cf814dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf814dd80 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80ad040 tx=0x7f5ce801aca0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 shutdown_connections 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f5cd4066f60 0x7f5cd4069420 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f5cf8143ae0 0x7f5cf8141770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f5cf806eac0 0x7f5cf8141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 --2- 172.21.15.67:0/2958145348 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cf806e6e0 0x7f5cf814dd80 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 >> 172.21.15.67:0/2958145348 conn(0x7f5cf806ac90 msgr2=0x7f5cf806b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 shutdown_connections 2024-04-17T10:38:52.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:52.943+0000 7f5cfd000640 1 -- 172.21.15.67:0/2958145348 wait complete. 2024-04-17T10:38:53.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:53.354 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:54.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:54.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3013a00640 1 Processor -- start 2024-04-17T10:38:54.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3013a00640 1 -- start start 2024-04-17T10:38:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3013a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f300c0b07b0 0x7f300c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3013a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c071300 0x7f300c06c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3013a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f300c0b5980 con 0x7f300c15dd10 2024-04-17T10:38:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3013a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300c0b1280 con 0x7f300c0b07b0 2024-04-17T10:38:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3013a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300c0b0cc0 con 0x7f300c071300 2024-04-17T10:38:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3003e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f300c0b07b0 0x7f300c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:54.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3003e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f300c0b07b0 0x7f300c070dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:49992/0 (socket says 172.21.15.67:49992) 2024-04-17T10:38:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3003e00640 1 -- 172.21.15.67:0/2214694972 learned_addr learned my addr 172.21.15.67:0/2214694972 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3011a00640 1 --2- 172.21.15.67:0/2214694972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c071300 0x7f300c06c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3003e00640 1 -- 172.21.15.67:0/2214694972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c071300 msgr2=0x7f300c06c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3003e00640 1 --2- 172.21.15.67:0/2214694972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c071300 0x7f300c06c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3003e00640 1 -- 172.21.15.67:0/2214694972 >> v1:172.21.15.86:6789/0 conn(0x7f300c15dd10 legacy=0x7f300c0b5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:54.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.519+0000 7f3003e00640 1 -- 172.21.15.67:0/2214694972 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c0b0fa0 con 0x7f300c0b07b0 2024-04-17T10:38:54.512 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3003e00640 1 --2- 172.21.15.67:0/2214694972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c0b07b0 0x7f300c070dc0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009870 tx=0x7f2ff401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1bdedf891afb6d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:54.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f300be00640 1 -- 172.21.15.67:0/2214694972 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40b1070 con 0x7f300c0b07b0 2024-04-17T10:38:54.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f300be00640 1 -- 172.21.15.67:0/2214694972 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff40ac620 con 0x7f300c0b07b0 2024-04-17T10:38:54.513 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f300be00640 1 -- 172.21.15.67:0/2214694972 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40ac9f0 con 0x7f300c0b07b0 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 -- 172.21.15.67:0/2214694972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c0b07b0 msgr2=0x7f300c070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 --2- 172.21.15.67:0/2214694972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c0b07b0 0x7f300c070dc0 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f2ff4009870 tx=0x7f2ff401c4d0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 -- 172.21.15.67:0/2214694972 shutdown_connections 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 --2- 172.21.15.67:0/2214694972 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c071300 0x7f300c06c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 --2- 172.21.15.67:0/2214694972 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c0b07b0 0x7f300c070dc0 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 -- 172.21.15.67:0/2214694972 >> 172.21.15.67:0/2214694972 conn(0x7f300c0b34d0 msgr2=0x7f300c0b38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 -- 172.21.15.67:0/2214694972 shutdown_connections 2024-04-17T10:38:54.514 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.523+0000 7f3013a00640 1 -- 172.21.15.67:0/2214694972 wait complete. 2024-04-17T10:38:54.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 Processor -- start 2024-04-17T10:38:54.515 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 -- start start 2024-04-17T10:38:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f300c071300 0x7f300c1574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c0b07b0 0x7f300c150c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c15dd10 0x7f300c141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c06d5a0 con 0x7f300c15dd10 2024-04-17T10:38:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f300c06d420 con 0x7f300c071300 2024-04-17T10:38:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300c06d720 con 0x7f300c0b07b0 2024-04-17T10:38:54.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c15dd10 0x7f300c141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c15dd10 0x7f300c141040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50004/0 (socket says 172.21.15.67:50004) 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 -- 172.21.15.67:0/2895126127 learned_addr learned my addr 172.21.15.67:0/2895126127 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011000640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f300c071300 0x7f300c1574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3003e00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c0b07b0 0x7f300c150c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 -- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c0b07b0 msgr2=0x7f300c150c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c0b07b0 0x7f300c150c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 -- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f300c071300 msgr2=0x7f300c1574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f300c071300 0x7f300c1574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 -- 172.21.15.67:0/2895126127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300c151350 con 0x7f300c15dd10 2024-04-17T10:38:54.518 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3011a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c15dd10 0x7f300c141040 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f300400efc0 tx=0x7f300400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3009600640 1 -- 172.21.15.67:0/2895126127 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3004018020 con 0x7f300c15dd10 2024-04-17T10:38:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3009600640 1 -- 172.21.15.67:0/2895126127 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3004014480 con 0x7f300c15dd10 2024-04-17T10:38:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3009600640 1 -- 172.21.15.67:0/2895126127 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3004014870 con 0x7f300c15dd10 2024-04-17T10:38:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300c151630 con 0x7f300c15dd10 2024-04-17T10:38:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300c151bf0 con 0x7f300c15dd10 2024-04-17T10:38:54.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.527+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300c074e30 con 0x7f300c15dd10 2024-04-17T10:38:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.531+0000 7f3009600640 1 -- 172.21.15.67:0/2895126127 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f300401b020 con 0x7f300c15dd10 2024-04-17T10:38:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.531+0000 7f3009600640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe8066f60 0x7f2fe8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:54.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.531+0000 7f3009600640 1 -- 172.21.15.67:0/2895126127 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f3004085c30 con 0x7f300c15dd10 2024-04-17T10:38:54.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.531+0000 7f3011000640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe8066f60 0x7f2fe8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:54.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.531+0000 7f3011000640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe8066f60 0x7f2fe8069420 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc001770 tx=0x7f2ffc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:54.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.535+0000 7f3009600640 1 -- 172.21.15.67:0/2895126127 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3004022070 con 0x7f300c15dd10 2024-04-17T10:38:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f300c0b41a0 con 0x7f300c15dd10 2024-04-17T10:38:54.723 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3009600640 1 -- 172.21.15.67:0/2895126127 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3004010040 con 0x7f300c15dd10 2024-04-17T10:38:54.723 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:54.724 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:54.724 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe8066f60 msgr2=0x7f2fe8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe8066f60 0x7f2fe8069420 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc001770 tx=0x7f2ffc009040 comp rx=0 tx=0).stop 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c15dd10 msgr2=0x7f300c141040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c15dd10 0x7f300c141040 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f300400efc0 tx=0x7f300400c490 comp rx=0 tx=0).stop 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 shutdown_connections 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2fe8066f60 0x7f2fe8069420 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f300c15dd10 0x7f300c141040 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f300c0b07b0 0x7f300c150c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 --2- 172.21.15.67:0/2895126127 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f300c071300 0x7f300c1574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 >> 172.21.15.67:0/2895126127 conn(0x7f300c0b34d0 msgr2=0x7f300c0d0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:54.726 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 shutdown_connections 2024-04-17T10:38:54.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:54.735+0000 7f3013a00640 1 -- 172.21.15.67:0/2895126127 wait complete. 2024-04-17T10:38:55.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:55.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:56.148 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:56.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.311+0000 7fbd91600640 1 Processor -- start 2024-04-17T10:38:56.303 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd91600640 1 -- start start 2024-04-17T10:38:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd91600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd8c099a50 0x7fbd8c15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd91600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c09a740 0x7fbd8c0cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd91600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd8c0cde20 con 0x7fbd8c0d03a0 2024-04-17T10:38:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd91600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd8c0cf300 con 0x7fbd8c099a50 2024-04-17T10:38:56.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd91600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8c0cf5e0 con 0x7fbd8c09a740 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd8be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd8c099a50 0x7fbd8c15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd8be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbd8c099a50 0x7fbd8c15c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:48414/0 (socket says 172.21.15.67:48414) 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd90c00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c09a740 0x7fbd8c0cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd8be00640 1 -- 172.21.15.67:0/3926583699 learned_addr learned my addr 172.21.15.67:0/3926583699 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd83400640 1 --1- 172.21.15.67:0/3926583699 >> v1:172.21.15.86:6789/0 conn(0x7fbd8c0d03a0 0x7fbd8c099e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:55930/0 (socket says 172.21.15.67:55930) 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd90c00640 1 -- 172.21.15.67:0/3926583699 >> v2:172.21.15.86:3300/0 conn(0x7fbd8c099a50 msgr2=0x7fbd8c15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd90c00640 1 --2- 172.21.15.67:0/3926583699 >> v2:172.21.15.86:3300/0 conn(0x7fbd8c099a50 0x7fbd8c15c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd90c00640 1 -- 172.21.15.67:0/3926583699 >> v1:172.21.15.86:6789/0 conn(0x7fbd8c0d03a0 legacy=0x7fbd8c099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:56.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd90c00640 1 -- 172.21.15.67:0/3926583699 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c0cf8c0 con 0x7fbd8c09a740 2024-04-17T10:38:56.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd8b400640 1 -- 172.21.15.67:0/3926583699 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878055059 0 0) 0x7fbd8c0cde20 con 0x7fbd8c0d03a0 2024-04-17T10:38:56.306 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.315+0000 7fbd8be00640 1 --2- 172.21.15.67:0/3926583699 >> v2:172.21.15.86:3300/0 conn(0x7fbd8c099a50 0x7fbd8c15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd90c00640 1 --2- 172.21.15.67:0/3926583699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c09a740 0x7fbd8c0cc1a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009d90 tx=0x7fbd8401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0e2c8db2e59e987 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd8b400640 1 -- 172.21.15.67:0/3926583699 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840b2070 con 0x7fbd8c09a740 2024-04-17T10:38:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd8b400640 1 -- 172.21.15.67:0/3926583699 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd840a5830 con 0x7fbd8c09a740 2024-04-17T10:38:56.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd8b400640 1 -- 172.21.15.67:0/3926583699 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840a5c00 con 0x7fbd8c09a740 2024-04-17T10:38:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 -- 172.21.15.67:0/3926583699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c09a740 msgr2=0x7fbd8c0cc1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 --2- 172.21.15.67:0/3926583699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c09a740 0x7fbd8c0cc1a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbd84009d90 tx=0x7fbd8401c970 comp rx=0 tx=0).stop 2024-04-17T10:38:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 -- 172.21.15.67:0/3926583699 shutdown_connections 2024-04-17T10:38:56.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 --2- 172.21.15.67:0/3926583699 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c09a740 0x7fbd8c0cc1a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 --2- 172.21.15.67:0/3926583699 >> v2:172.21.15.86:3300/0 conn(0x7fbd8c099a50 0x7fbd8c15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 -- 172.21.15.67:0/3926583699 >> 172.21.15.67:0/3926583699 conn(0x7fbd8c0d1690 msgr2=0x7fbd8c0d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 -- 172.21.15.67:0/3926583699 shutdown_connections 2024-04-17T10:38:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 -- 172.21.15.67:0/3926583699 wait complete. 2024-04-17T10:38:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.319+0000 7fbd91600640 1 Processor -- start 2024-04-17T10:38:56.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 -- start start 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd8c099a50 0x7fbd8c0b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd8c09a740 0x7fbd8c150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c0d03a0 0x7fbd8c152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8c1686b0 con 0x7fbd8c099a50 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8c0cd180 con 0x7fbd8c09a740 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8c168830 con 0x7fbd8c0d03a0 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd8c099a50 0x7fbd8c0b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:56.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd8c099a50 0x7fbd8c0b6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50010/0 (socket says 172.21.15.67:50010) 2024-04-17T10:38:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 -- 172.21.15.67:0/1544949384 learned_addr learned my addr 172.21.15.67:0/1544949384 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd83400640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd8c09a740 0x7fbd8c150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:56.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd90c00640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c0d03a0 0x7fbd8c152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 -- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c0d03a0 msgr2=0x7fbd8c152bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c0d03a0 0x7fbd8c152bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 -- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd8c09a740 msgr2=0x7fbd8c150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd8c09a740 0x7fbd8c150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 -- 172.21.15.67:0/1544949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd8c151370 con 0x7fbd8c099a50 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd83400640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd8c09a740 0x7fbd8c150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd8be00640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd8c099a50 0x7fbd8c0b6e50 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00e960 tx=0x7fbd7c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:56.314 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd88c00640 1 -- 172.21.15.67:0/1544949384 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c017070 con 0x7fbd8c099a50 2024-04-17T10:38:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd88c00640 1 -- 172.21.15.67:0/1544949384 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd7c00ca00 con 0x7fbd8c099a50 2024-04-17T10:38:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd88c00640 1 -- 172.21.15.67:0/1544949384 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c00cdb0 con 0x7fbd8c099a50 2024-04-17T10:38:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8c152530 con 0x7fbd8c099a50 2024-04-17T10:38:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8c06d090 con 0x7fbd8c099a50 2024-04-17T10:38:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.323+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd8c074dc0 con 0x7fbd8c099a50 2024-04-17T10:38:56.315 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.327+0000 7fbd88c00640 1 -- 172.21.15.67:0/1544949384 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbd7c01a070 con 0x7fbd8c099a50 2024-04-17T10:38:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.327+0000 7fbd88c00640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd68066f60 0x7fbd68069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.327+0000 7fbd88c00640 1 -- 172.21.15.67:0/1544949384 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbd7c085420 con 0x7fbd8c099a50 2024-04-17T10:38:56.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.327+0000 7fbd83400640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd68066f60 0x7fbd68069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:56.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.327+0000 7fbd83400640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd68066f60 0x7fbd68069420 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0b7390 tx=0x7fbd74007c20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:56.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.331+0000 7fbd88c00640 1 -- 172.21.15.67:0/1544949384 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbd7c055bb0 con 0x7fbd8c099a50 2024-04-17T10:38:56.516 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.527+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd8c0d24a0 con 0x7fbd8c099a50 2024-04-17T10:38:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.527+0000 7fbd88c00640 1 -- 172.21.15.67:0/1544949384 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fbd7c059c50 con 0x7fbd8c099a50 2024-04-17T10:38:56.517 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:56.517 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:56.517 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.527+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd68066f60 msgr2=0x7fbd68069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.527+0000 7fbd91600640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd68066f60 0x7fbd68069420 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0b7390 tx=0x7fbd74007c20 comp rx=0 tx=0).stop 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.527+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd8c099a50 msgr2=0x7fbd8c0b6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.527+0000 7fbd91600640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd8c099a50 0x7fbd8c0b6e50 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00e960 tx=0x7fbd7c00ee20 comp rx=0 tx=0).stop 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 shutdown_connections 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd68066f60 0x7fbd68069420 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd8c0d03a0 0x7fbd8c152bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.519 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd8c09a740 0x7fbd8c150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 --2- 172.21.15.67:0/1544949384 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd8c099a50 0x7fbd8c0b6e50 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:56.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 >> 172.21.15.67:0/1544949384 conn(0x7fbd8c0d1690 msgr2=0x7fbd8c0af490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:56.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 shutdown_connections 2024-04-17T10:38:56.520 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:56.531+0000 7fbd91600640 1 -- 172.21.15.67:0/1544949384 wait complete. 2024-04-17T10:38:56.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:56.941 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:57.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:58.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e4000640 1 Processor -- start 2024-04-17T10:38:58.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e4000640 1 -- start start 2024-04-17T10:38:58.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e4000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc06eac0 0x7f41dc143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:58.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e4000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41dc0cf0c0 0x7f41dc144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:58.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e4000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc144560 con 0x7f41dc06e6e0 2024-04-17T10:38:58.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e4000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc15ae40 con 0x7f41dc0cf0c0 2024-04-17T10:38:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e4000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc071440 con 0x7f41dc06eac0 2024-04-17T10:38:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41dc0cf0c0 0x7f41dc144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e2000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f41dc0cf0c0 0x7f41dc144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50020/0 (socket says 172.21.15.67:50020) 2024-04-17T10:38:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e2000640 1 -- 172.21.15.67:0/2547061459 learned_addr learned my addr 172.21.15.67:0/2547061459 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e0c00640 1 --2- 172.21.15.67:0/2547061459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc06eac0 0x7f41dc143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e2000640 1 -- 172.21.15.67:0/2547061459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc06eac0 msgr2=0x7f41dc143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e2000640 1 --2- 172.21.15.67:0/2547061459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc06eac0 0x7f41dc143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e2000640 1 -- 172.21.15.67:0/2547061459 >> v1:172.21.15.86:6789/0 conn(0x7f41dc06e6e0 legacy=0x7f41dc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.107+0000 7f41e2000640 1 -- 172.21.15.67:0/2547061459 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc071720 con 0x7f41dc0cf0c0 2024-04-17T10:38:58.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e2000640 1 --2- 172.21.15.67:0/2547061459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc0cf0c0 0x7f41dc144020 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009e50 tx=0x7f41d401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfb9acf36f4015f server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41dbe00640 1 -- 172.21.15.67:0/2547061459 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40a5830 con 0x7f41dc0cf0c0 2024-04-17T10:38:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41dbe00640 1 -- 172.21.15.67:0/2547061459 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d40a59d0 con 0x7f41dc0cf0c0 2024-04-17T10:38:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41dbe00640 1 -- 172.21.15.67:0/2547061459 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d40a5da0 con 0x7f41dc0cf0c0 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 -- 172.21.15.67:0/2547061459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc0cf0c0 msgr2=0x7f41dc144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 --2- 172.21.15.67:0/2547061459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc0cf0c0 0x7f41dc144020 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009e50 tx=0x7f41d401c970 comp rx=0 tx=0).stop 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 -- 172.21.15.67:0/2547061459 shutdown_connections 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 --2- 172.21.15.67:0/2547061459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc0cf0c0 0x7f41dc144020 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 --2- 172.21.15.67:0/2547061459 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc06eac0 0x7f41dc143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 -- 172.21.15.67:0/2547061459 >> 172.21.15.67:0/2547061459 conn(0x7f41dc06ac90 msgr2=0x7f41dc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 -- 172.21.15.67:0/2547061459 shutdown_connections 2024-04-17T10:38:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.111+0000 7f41e4000640 1 -- 172.21.15.67:0/2547061459 wait complete. 2024-04-17T10:38:58.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 Processor -- start 2024-04-17T10:38:58.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 -- start start 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41dc06e6e0 0x7f41dc1586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc06eac0 0x7f41dc0af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc0cf0c0 0x7f41dc0afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0b3a40 con 0x7f41dc06eac0 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0b38c0 con 0x7f41dc06e6e0 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0b3bc0 con 0x7f41dc0cf0c0 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc06eac0 0x7f41dc0af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc06eac0 0x7f41dc0af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50034/0 (socket says 172.21.15.67:50034) 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 -- 172.21.15.67:0/1861945140 learned_addr learned my addr 172.21.15.67:0/1861945140 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:58.105 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e1600640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41dc06e6e0 0x7f41dc1586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e2000640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc0cf0c0 0x7f41dc0afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 -- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc0cf0c0 msgr2=0x7f41dc0afbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc0cf0c0 0x7f41dc0afbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 -- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41dc06e6e0 msgr2=0x7f41dc1586e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41dc06e6e0 0x7f41dc1586e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.106 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 -- 172.21.15.67:0/1861945140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc159280 con 0x7f41dc06eac0 2024-04-17T10:38:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e0c00640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc06eac0 0x7f41dc0af6a0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f41c800e930 tx=0x7f41c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41d9600640 1 -- 172.21.15.67:0/1861945140 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8017070 con 0x7f41dc06eac0 2024-04-17T10:38:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41d9600640 1 -- 172.21.15.67:0/1861945140 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c800ca00 con 0x7f41dc06eac0 2024-04-17T10:38:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41d9600640 1 -- 172.21.15.67:0/1861945140 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c800cdb0 con 0x7f41dc06eac0 2024-04-17T10:38:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc159560 con 0x7f41dc06eac0 2024-04-17T10:38:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.115+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc159b20 con 0x7f41dc06eac0 2024-04-17T10:38:58.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.119+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc072d50 con 0x7f41dc06eac0 2024-04-17T10:38:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.119+0000 7f41d9600640 1 -- 172.21.15.67:0/1861945140 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f41c801a070 con 0x7f41dc06eac0 2024-04-17T10:38:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.119+0000 7f41d9600640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41bc066f60 0x7f41bc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:58.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.119+0000 7f41d9600640 1 -- 172.21.15.67:0/1861945140 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f41c80855b0 con 0x7f41dc06eac0 2024-04-17T10:38:58.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.119+0000 7f41e1600640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41bc066f60 0x7f41bc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:58.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.119+0000 7f41e1600640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41bc066f60 0x7f41bc069420 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f41d0001810 tx=0x7f41d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:58.112 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.123+0000 7f41d9600640 1 -- 172.21.15.67:0/1861945140 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f41c8055e00 con 0x7f41dc06eac0 2024-04-17T10:38:58.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41dc0cee60 con 0x7f41dc06eac0 2024-04-17T10:38:58.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41d9600640 1 -- 172.21.15.67:0/1861945140 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f41c8059ea0 con 0x7f41dc06eac0 2024-04-17T10:38:58.308 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:38:58.308 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:38:58.308 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:38:58.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41bc066f60 msgr2=0x7f41bc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:58.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41bc066f60 0x7f41bc069420 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f41d0001810 tx=0x7f41d0009040 comp rx=0 tx=0).stop 2024-04-17T10:38:58.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc06eac0 msgr2=0x7f41dc0af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc06eac0 0x7f41dc0af6a0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f41c800e930 tx=0x7f41c800edf0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 shutdown_connections 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f41bc066f60 0x7f41bc069420 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f41dc0cf0c0 0x7f41dc0afbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f41dc06eac0 0x7f41dc0af6a0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 --2- 172.21.15.67:0/1861945140 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41dc06e6e0 0x7f41dc1586e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 >> 172.21.15.67:0/1861945140 conn(0x7f41dc06ac90 msgr2=0x7f41dc0cb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 shutdown_connections 2024-04-17T10:38:58.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:58.319+0000 7f41e4000640 1 -- 172.21.15.67:0/1861945140 wait complete. 2024-04-17T10:38:58.709 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:38:58.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:38:58.731 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:38:58.731 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:38:59.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:38:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 Processor -- start 2024-04-17T10:38:59.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 -- start start 2024-04-17T10:38:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f89cc158170 0x7f89cc15ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc06d210 0x7f89cc0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89cc0ceb70 0x7f89cc15e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc0cf3c0 con 0x7f89cc0ceb70 2024-04-17T10:38:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc0cf000 con 0x7f89cc158170 2024-04-17T10:38:59.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.895+0000 7f89d1a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc06d9f0 con 0x7f89cc06d210 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89cc0ceb70 0x7f89cc15e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f89cc0ceb70 0x7f89cc15e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50048/0 (socket says 172.21.15.67:50048) 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 -- 172.21.15.67:0/273189927 learned_addr learned my addr 172.21.15.67:0/273189927 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89cbe00640 1 --2- 172.21.15.67:0/273189927 >> v2:172.21.15.86:3300/0 conn(0x7f89cc158170 0x7f89cc15ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89c3400640 1 --2- 172.21.15.67:0/273189927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc06d210 0x7f89cc0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 -- 172.21.15.67:0/273189927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc06d210 msgr2=0x7f89cc0ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 --2- 172.21.15.67:0/273189927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc06d210 0x7f89cc0ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 -- 172.21.15.67:0/273189927 >> v2:172.21.15.86:3300/0 conn(0x7f89cc158170 msgr2=0x7f89cc15ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 --2- 172.21.15.67:0/273189927 >> v2:172.21.15.86:3300/0 conn(0x7f89cc158170 0x7f89cc15ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:59.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 -- 172.21.15.67:0/273189927 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc06dcd0 con 0x7f89cc0ceb70 2024-04-17T10:38:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89d1000640 1 --2- 172.21.15.67:0/273189927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0ceb70 0x7f89cc15e2d0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0b4e20 tx=0x7f89c401c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0bee3a165ac85bc server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89cb400640 1 -- 172.21.15.67:0/273189927 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c40a5580 con 0x7f89cc0ceb70 2024-04-17T10:38:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89cb400640 1 -- 172.21.15.67:0/273189927 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c40a5720 con 0x7f89cc0ceb70 2024-04-17T10:38:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.899+0000 7f89cb400640 1 -- 172.21.15.67:0/273189927 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c40a5af0 con 0x7f89cc0ceb70 2024-04-17T10:38:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- 172.21.15.67:0/273189927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0ceb70 msgr2=0x7f89cc15e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 --2- 172.21.15.67:0/273189927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0ceb70 0x7f89cc15e2d0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f89cc0b4e20 tx=0x7f89c401c700 comp rx=0 tx=0).stop 2024-04-17T10:38:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- 172.21.15.67:0/273189927 shutdown_connections 2024-04-17T10:38:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 --2- 172.21.15.67:0/273189927 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc0ceb70 0x7f89cc15e2d0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:59.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 --2- 172.21.15.67:0/273189927 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc06d210 0x7f89cc0ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:59.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 --2- 172.21.15.67:0/273189927 >> v2:172.21.15.86:3300/0 conn(0x7f89cc158170 0x7f89cc15ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:59.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- 172.21.15.67:0/273189927 >> 172.21.15.67:0/273189927 conn(0x7f89cc067070 msgr2=0x7f89cc06ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:38:59.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- 172.21.15.67:0/273189927 shutdown_connections 2024-04-17T10:38:59.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- 172.21.15.67:0/273189927 wait complete. 2024-04-17T10:38:59.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 Processor -- start 2024-04-17T10:38:59.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- start start 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89cc06d210 0x7f89cc15ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc0ceb70 0x7f89cc0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc158170 0x7f89cc0c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc06d770 con 0x7f89cc158170 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc06d5f0 con 0x7f89cc06d210 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc158550 con 0x7f89cc0ceb70 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc158170 0x7f89cc0c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89cc06d210 0x7f89cc15ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89cc06d210 0x7f89cc15ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:48454/0 (socket says 172.21.15.67:48454) 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 -- 172.21.15.67:0/3268546030 learned_addr learned my addr 172.21.15.67:0/3268546030 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc158170 0x7f89cc0c6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50064/0 (socket says 172.21.15.67:50064) 2024-04-17T10:38:59.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89c3400640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc0ceb70 0x7f89cc0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 -- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc0ceb70 msgr2=0x7f89cc0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc0ceb70 0x7f89cc0c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 -- 172.21.15.67:0/3268546030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc158170 msgr2=0x7f89cc0c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:38:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc158170 0x7f89cc0c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:38:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89cbe00640 1 -- 172.21.15.67:0/3268546030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc0c5ea0 con 0x7f89cc06d210 2024-04-17T10:38:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.903+0000 7f89d1000640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc158170 0x7f89cc0c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:38:59.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89cbe00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89cc06d210 0x7f89cc15ac30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00d5f0 tx=0x7f89bc00dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89c8c00640 1 -- 172.21.15.67:0/3268546030 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc0125f0 con 0x7f89cc06d210 2024-04-17T10:38:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89c8c00640 1 -- 172.21.15.67:0/3268546030 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89bc019070 con 0x7f89cc06d210 2024-04-17T10:38:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89c8c00640 1 -- 172.21.15.67:0/3268546030 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89bc0129b0 con 0x7f89cc06d210 2024-04-17T10:38:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc0c6180 con 0x7f89cc06d210 2024-04-17T10:38:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc0c6740 con 0x7f89cc06d210 2024-04-17T10:38:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc074dc0 con 0x7f89cc06d210 2024-04-17T10:38:59.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89c8c00640 1 -- 172.21.15.67:0/3268546030 <== mon.1 v2:172.21.15.86:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f89bc012b50 con 0x7f89cc06d210 2024-04-17T10:38:59.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89c8c00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89a8066f60 0x7f89a8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:38:59.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89c8c00640 1 -- 172.21.15.67:0/3268546030 <== mon.1 v2:172.21.15.86:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f89bc0854d0 con 0x7f89cc06d210 2024-04-17T10:38:59.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.907+0000 7f89c3400640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89a8066f60 0x7f89a8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:38:59.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.911+0000 7f89c3400640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89a8066f60 0x7f89a8069420 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f89cc1f9ce0 tx=0x7f89b4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:38:59.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:38:59.911+0000 7f89c8c00640 1 -- 172.21.15.67:0/3268546030 <== mon.1 v2:172.21.15.86:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f89bc055c60 con 0x7f89cc06d210 2024-04-17T10:39:00.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.107+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89cc1652e0 con 0x7f89cc06d210 2024-04-17T10:39:00.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.107+0000 7f89c8c00640 1 -- 172.21.15.67:0/3268546030 <== mon.1 v2:172.21.15.86:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f89bc059d00 con 0x7f89cc06d210 2024-04-17T10:39:00.099 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:00.099 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:00.099 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:00.100 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89a8066f60 msgr2=0x7f89a8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89a8066f60 0x7f89a8069420 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f89cc1f9ce0 tx=0x7f89b4008040 comp rx=0 tx=0).stop 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89cc06d210 msgr2=0x7f89cc15ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89cc06d210 0x7f89cc15ac30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f89bc00d5f0 tx=0x7f89bc00dab0 comp rx=0 tx=0).stop 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 shutdown_connections 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f89a8066f60 0x7f89a8069420 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89cc158170 0x7f89cc0c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f89cc0ceb70 0x7f89cc0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 --2- 172.21.15.67:0/3268546030 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f89cc06d210 0x7f89cc15ac30 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 >> 172.21.15.67:0/3268546030 conn(0x7f89cc067070 msgr2=0x7f89cc163810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 shutdown_connections 2024-04-17T10:39:00.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:00.111+0000 7f89d1a00640 1 -- 172.21.15.67:0/3268546030 wait complete. 2024-04-17T10:39:00.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:00.523 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:01.524 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:01.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.686+0000 7f1e02600640 1 Processor -- start 2024-04-17T10:39:01.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.686+0000 7f1e02600640 1 -- start start 2024-04-17T10:39:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.686+0000 7f1e02600640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1dfc0ce2a0 0x7f1dfc0cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.686+0000 7f1e02600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.686+0000 7f1e02600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dfc06b100 con 0x7f1dfc0cc250 2024-04-17T10:39:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.686+0000 7f1e02600640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dfc0cd150 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.686+0000 7f1e02600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc06b3e0 con 0x7f1dfc0cc840 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1dfc0ce2a0 0x7f1dfc0cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1dfbe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f1dfc0ce2a0 0x7f1dfc0cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37658/0 (socket says 172.21.15.67:37658) 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 -- 172.21.15.67:0/2382506102 learned_addr learned my addr 172.21.15.67:0/2382506102 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1dfb400640 1 --1- 172.21.15.67:0/2382506102 >> v1:172.21.15.86:6789/0 conn(0x7f1dfc0cc250 0x7f1dfc0cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:32868/0 (socket says 172.21.15.67:32868) 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 -- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 msgr2=0x7f1dfc0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 --2- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 -- 172.21.15.67:0/2382506102 >> v1:172.21.15.86:6789/0 conn(0x7f1dfc0cc250 legacy=0x7f1dfc0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1dfaa00640 1 -- 172.21.15.67:0/2382506102 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842294115 0 0) 0x7f1dfc06b100 con 0x7f1dfc0cc250 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 -- 172.21.15.67:0/2382506102 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc06b6c0 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1dfbe00640 1 --2- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.690+0000 7f1df2a00640 1 --2- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc0cd5e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009870 tx=0x7f1de401c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d18e698a732fc8fa server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1dfaa00640 1 -- 172.21.15.67:0/2382506102 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de40b1070 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1dfaa00640 1 -- 172.21.15.67:0/2382506102 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de40a8050 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1dfaa00640 1 -- 172.21.15.67:0/2382506102 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de40ac860 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0ce2a0 msgr2=0x7f1dfc0cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 --2- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc0cd5e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1de4009870 tx=0x7f1de401c4d0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- 172.21.15.67:0/2382506102 shutdown_connections 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 --2- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 --2- 172.21.15.67:0/2382506102 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc0cd5e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- 172.21.15.67:0/2382506102 >> 172.21.15.67:0/2382506102 conn(0x7f1dfc0b22a0 msgr2=0x7f1dfc0b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- 172.21.15.67:0/2382506102 shutdown_connections 2024-04-17T10:39:01.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- 172.21.15.67:0/2382506102 wait complete. 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 Processor -- start 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- start start 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0cc250 0x7f1dfc0c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc16ac80 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc16ab00 con 0x7f1dfc0cc250 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1e02600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dfc16ae00 con 0x7f1dfc0cc840 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.694+0000 7f1dfbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44980/0 (socket says 172.21.15.67:44980) 2024-04-17T10:39:01.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 -- 172.21.15.67:0/1072158851 learned_addr learned my addr 172.21.15.67:0/1072158851 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfb400640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0cc250 0x7f1dfc0c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1df2a00640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 -- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 msgr2=0x7f1dfc0b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 -- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0cc250 msgr2=0x7f1dfc0c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0cc250 0x7f1dfc0c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 -- 172.21.15.67:0/1072158851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dfc1596c0 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1dfbe00640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc159180 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009f00 tx=0x7f1df400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1df3e00640 1 -- 172.21.15.67:0/1072158851 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df4018030 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1df3e00640 1 -- 172.21.15.67:0/1072158851 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df400ce50 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1df3e00640 1 -- 172.21.15.67:0/1072158851 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df4003bc0 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dfc15d780 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dfc15dd40 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dfc06cd30 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.698+0000 7f1df3e00640 1 -- 172.21.15.67:0/1072158851 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f1df401b070 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.702+0000 7f1df3e00640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd8066f90 0x7f1dd8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.702+0000 7f1df3e00640 1 -- 172.21.15.67:0/1072158851 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1df4085bf0 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.702+0000 7f1dfb400640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd8066f90 0x7f1dd8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.702+0000 7f1dfb400640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd8066f90 0x7f1dd8069450 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1dec007fd0 tx=0x7f1dec007d60 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:01.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.706+0000 7f1df3e00640 1 -- 172.21.15.67:0/1072158851 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1df4010040 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.898+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dfc074e30 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.898+0000 7f1df3e00640 1 -- 172.21.15.67:0/1072158851 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1df40562f0 con 0x7f1dfc0ce2a0 2024-04-17T10:39:01.891 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:01.891 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:01.891 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd8066f90 msgr2=0x7f1dd8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd8066f90 0x7f1dd8069450 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1dec007fd0 tx=0x7f1dec007d60 comp rx=0 tx=0).stop 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dfc0ce2a0 msgr2=0x7f1dfc159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc159180 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009f00 tx=0x7f1df400c490 comp rx=0 tx=0).stop 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 shutdown_connections 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1dd8066f90 0x7f1dd8069450 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1dfc0ce2a0 0x7f1dfc159180 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f1dfc0cc840 0x7f1dfc0b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 --2- 172.21.15.67:0/1072158851 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1dfc0cc250 0x7f1dfc0c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:01.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 >> 172.21.15.67:0/1072158851 conn(0x7f1dfc0b22a0 msgr2=0x7f1dfc15e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 shutdown_connections 2024-04-17T10:39:01.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:01.902+0000 7f1e02600640 1 -- 172.21.15.67:0/1072158851 wait complete. 2024-04-17T10:39:02.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:02.314 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:03.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:03.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.478+0000 7f6d24e00640 1 Processor -- start 2024-04-17T10:39:03.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.478+0000 7f6d24e00640 1 -- start start 2024-04-17T10:39:03.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.478+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:03.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.478+0000 7f6d24e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d20071a20 0x7f6d200cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:03.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.478+0000 7f6d24e00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d2006e2f0 con 0x7f6d2006ef10 2024-04-17T10:39:03.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.478+0000 7f6d24e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d200a86c0 con 0x7f6d20071a20 2024-04-17T10:39:03.470 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.478+0000 7f6d24e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2015bf10 con 0x7f6d20145040 2024-04-17T10:39:03.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1e800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6d20071a20 0x7f6d200cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1de00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d17e00640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6d2006ef10 0x7f6d20071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:32876/0 (socket says 172.21.15.67:32876) 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d17e00640 1 -- 172.21.15.67:0/2568453536 learned_addr learned my addr 172.21.15.67:0/2568453536 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1d400640 1 -- 172.21.15.67:0/2568453536 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145757797 0 0) 0x7f6d2006e2f0 con 0x7f6d2006ef10 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1d400640 1 -- 172.21.15.67:0/2568453536 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d04003530 con 0x7f6d2006ef10 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1de00640 1 -- 172.21.15.67:0/2568453536 >> v2:172.21.15.86:3300/0 conn(0x7f6d20071a20 msgr2=0x7f6d200cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1de00640 1 --2- 172.21.15.67:0/2568453536 >> v2:172.21.15.86:3300/0 conn(0x7f6d20071a20 0x7f6d200cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1de00640 1 -- 172.21.15.67:0/2568453536 >> v1:172.21.15.86:6789/0 conn(0x7f6d2006ef10 legacy=0x7f6d20071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1de00640 1 -- 172.21.15.67:0/2568453536 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2006d6e0 con 0x7f6d20145040 2024-04-17T10:39:03.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1e800640 1 --2- 172.21.15.67:0/2568453536 >> v2:172.21.15.86:3300/0 conn(0x7f6d20071a20 0x7f6d200cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:39:03.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.482+0000 7f6d1de00640 1 --2- 172.21.15.67:0/2568453536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2006e9d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6d100097c0 tx=0x7f6d1001ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88694afacfd03360 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:03.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d1d400640 1 -- 172.21.15.67:0/2568453536 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100b5070 con 0x7f6d20145040 2024-04-17T10:39:03.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d1d400640 1 -- 172.21.15.67:0/2568453536 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d100a8990 con 0x7f6d20145040 2024-04-17T10:39:03.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d1d400640 1 -- 172.21.15.67:0/2568453536 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100a8d60 con 0x7f6d20145040 2024-04-17T10:39:03.475 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- 172.21.15.67:0/2568453536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 msgr2=0x7f6d2006e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2568453536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2006e9d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6d100097c0 tx=0x7f6d1001ec90 comp rx=0 tx=0).stop 2024-04-17T10:39:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- 172.21.15.67:0/2568453536 shutdown_connections 2024-04-17T10:39:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2568453536 >> v2:172.21.15.86:3300/0 conn(0x7f6d20071a20 0x7f6d200cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2568453536 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2006e9d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- 172.21.15.67:0/2568453536 >> 172.21.15.67:0/2568453536 conn(0x7f6d2006ac90 msgr2=0x7f6d2006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- 172.21.15.67:0/2568453536 shutdown_connections 2024-04-17T10:39:03.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- 172.21.15.67:0/2568453536 wait complete. 2024-04-17T10:39:03.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 Processor -- start 2024-04-17T10:39:03.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- start start 2024-04-17T10:39:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006ef10 0x7f6d200b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d20071a20 0x7f6d2013a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:03.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2006df80 con 0x7f6d2006ef10 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2006dca0 con 0x7f6d20071a20 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.486+0000 7f6d24e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d2006d370 con 0x7f6d20145040 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006ef10 0x7f6d200b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006ef10 0x7f6d200b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44990/0 (socket says 172.21.15.67:44990) 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d17e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d20071a20 0x7f6d2013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 -- 172.21.15.67:0/2744854645 learned_addr learned my addr 172.21.15.67:0/2744854645 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1e800640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 -- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 msgr2=0x7f6d2015b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2015b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 -- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d20071a20 msgr2=0x7f6d2013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d20071a20 0x7f6d2013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 -- 172.21.15.67:0/2744854645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2013a880 con 0x7f6d2006ef10 2024-04-17T10:39:03.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d1de00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006ef10 0x7f6d200b4370 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f6d100b3040 tx=0x7f6d1001ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d16000640 1 -- 172.21.15.67:0/2744854645 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100bf070 con 0x7f6d2006ef10 2024-04-17T10:39:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d16000640 1 -- 172.21.15.67:0/2744854645 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d100b0660 con 0x7f6d2006ef10 2024-04-17T10:39:03.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d200aceb0 con 0x7f6d2006ef10 2024-04-17T10:39:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d16000640 1 -- 172.21.15.67:0/2744854645 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d100b5040 con 0x7f6d2006ef10 2024-04-17T10:39:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d200ad470 con 0x7f6d2006ef10 2024-04-17T10:39:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d20072d50 con 0x7f6d2006ef10 2024-04-17T10:39:03.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.490+0000 7f6d16000640 1 -- 172.21.15.67:0/2744854645 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6d100b0a50 con 0x7f6d2006ef10 2024-04-17T10:39:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.494+0000 7f6d16000640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cfc066f60 0x7f6cfc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.494+0000 7f6d16000640 1 -- 172.21.15.67:0/2744854645 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6d10121a10 con 0x7f6d2006ef10 2024-04-17T10:39:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.494+0000 7f6d17e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cfc066f60 0x7f6cfc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:03.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.494+0000 7f6d17e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cfc066f60 0x7f6cfc069420 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f6d2015f1d0 tx=0x7f6d08009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:03.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.498+0000 7f6d16000640 1 -- 172.21.15.67:0/2744854645 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6d100187f0 con 0x7f6d2006ef10 2024-04-17T10:39:03.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.694+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d20074e30 con 0x7f6d2006ef10 2024-04-17T10:39:03.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.694+0000 7f6d16000640 1 -- 172.21.15.67:0/2744854645 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6d100f2090 con 0x7f6d2006ef10 2024-04-17T10:39:03.685 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:03.685 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:03.685 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.694+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cfc066f60 msgr2=0x7f6cfc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.694+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cfc066f60 0x7f6cfc069420 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f6d2015f1d0 tx=0x7f6d08009040 comp rx=0 tx=0).stop 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.694+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006ef10 msgr2=0x7f6d200b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.694+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006ef10 0x7f6d200b4370 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f6d100b3040 tx=0x7f6d1001ba30 comp rx=0 tx=0).stop 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 shutdown_connections 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6cfc066f60 0x7f6cfc069420 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6d20145040 0x7f6d2015b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6d20071a20 0x7f6d2013a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 --2- 172.21.15.67:0/2744854645 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6d2006ef10 0x7f6d200b4370 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:03.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 >> 172.21.15.67:0/2744854645 conn(0x7f6d2006ac90 msgr2=0x7f6d2006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:03.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 shutdown_connections 2024-04-17T10:39:03.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:03.698+0000 7f6d24e00640 1 -- 172.21.15.67:0/2744854645 wait complete. 2024-04-17T10:39:04.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:04.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:05.110 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcef3600640 1 Processor -- start 2024-04-17T10:39:05.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcef3600640 1 -- start start 2024-04-17T10:39:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcef3600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec072370 0x7fceec06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcef3600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceec0ad8f0 con 0x7fceec06c410 2024-04-17T10:39:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcef3600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceec0adbd0 con 0x7fceec15dbc0 2024-04-17T10:39:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcef3600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec15dfa0 con 0x7fceec072370 2024-04-17T10:39:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcee3e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fceec06c410 0x7fceec071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36166/0 (socket says 172.21.15.67:36166) 2024-04-17T10:39:05.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcee3e00640 1 -- 172.21.15.67:0/74061490 learned_addr learned my addr 172.21.15.67:0/74061490 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fcef1600640 1 --2- 172.21.15.67:0/74061490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec072370 0x7fceec06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.270+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535190347 0 0) 0x7fceec0ad8f0 con 0x7fceec06c410 2024-04-17T10:39:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced0003530 con 0x7fceec06c410 2024-04-17T10:39:05.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666491248 0 0) 0x7fceec0adbd0 con 0x7fceec15dbc0 2024-04-17T10:39:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec0ad8f0 con 0x7fceec15dbc0 2024-04-17T10:39:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 15771952 0 0) 0x7fced0003530 con 0x7fceec06c410 2024-04-17T10:39:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fcef1600640 1 -- 172.21.15.67:0/74061490 >> v1:172.21.15.86:6789/0 conn(0x7fceec15dbc0 legacy=0x7fceec06db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fcef1600640 1 -- 172.21.15.67:0/74061490 >> v1:172.21.15.67:6789/0 conn(0x7fceec06c410 legacy=0x7fceec071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fcef1600640 1 -- 172.21.15.67:0/74061490 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec0adeb0 con 0x7fceec072370 2024-04-17T10:39:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fcef1600640 1 --2- 172.21.15.67:0/74061490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec072370 0x7fceec06f560 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009dc0 tx=0x7fcee401c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcd6be8f799a5977 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee40ac070 con 0x7fceec072370 2024-04-17T10:39:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee40a6470 con 0x7fceec072370 2024-04-17T10:39:05.266 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.274+0000 7fceebe00640 1 -- 172.21.15.67:0/74061490 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4001680 con 0x7fceec072370 2024-04-17T10:39:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- 172.21.15.67:0/74061490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec072370 msgr2=0x7fceec06f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 --2- 172.21.15.67:0/74061490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec072370 0x7fceec06f560 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcee4009dc0 tx=0x7fcee401c820 comp rx=0 tx=0).stop 2024-04-17T10:39:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- 172.21.15.67:0/74061490 shutdown_connections 2024-04-17T10:39:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 --2- 172.21.15.67:0/74061490 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec072370 0x7fceec06f560 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- 172.21.15.67:0/74061490 >> 172.21.15.67:0/74061490 conn(0x7fceec0af670 msgr2=0x7fceec0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:05.267 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- 172.21.15.67:0/74061490 shutdown_connections 2024-04-17T10:39:05.268 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- 172.21.15.67:0/74061490 wait complete. 2024-04-17T10:39:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 Processor -- start 2024-04-17T10:39:05.269 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- start start 2024-04-17T10:39:05.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceec06c410 0x7fceec141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:05.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fceec072370 0x7fceec0cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:05.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec15dbc0 0x7fceec158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:05.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec16b110 con 0x7fceec072370 2024-04-17T10:39:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceec06af70 con 0x7fceec06c410 2024-04-17T10:39:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef3600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceec16b290 con 0x7fceec15dbc0 2024-04-17T10:39:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fceec072370 0x7fceec0cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:05.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fceec072370 0x7fceec0cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45006/0 (socket says 172.21.15.67:45006) 2024-04-17T10:39:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 -- 172.21.15.67:0/1918050278 learned_addr learned my addr 172.21.15.67:0/1918050278 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef0c00640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceec06c410 0x7fceec141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 -- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec15dbc0 msgr2=0x7fceec158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:05.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef1600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec15dbc0 0x7fceec158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec15dbc0 0x7fceec158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 -- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceec06c410 msgr2=0x7fceec141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceec06c410 0x7fceec141890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 -- 172.21.15.67:0/1918050278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceec141dd0 con 0x7fceec072370 2024-04-17T10:39:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcef0c00640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceec06c410 0x7fceec141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:39:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.278+0000 7fcee3e00640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fceec072370 0x7fceec0cd4b0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fced400c750 tx=0x7fced400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcee9600640 1 -- 172.21.15.67:0/1918050278 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4005ae0 con 0x7fceec072370 2024-04-17T10:39:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceec0cde90 con 0x7fceec072370 2024-04-17T10:39:05.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcee9600640 1 -- 172.21.15.67:0/1918050278 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced4005c80 con 0x7fceec072370 2024-04-17T10:39:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcee9600640 1 -- 172.21.15.67:0/1918050278 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced40072b0 con 0x7fceec072370 2024-04-17T10:39:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceec0ce450 con 0x7fceec072370 2024-04-17T10:39:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceec09e260 con 0x7fceec072370 2024-04-17T10:39:05.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcee9600640 1 -- 172.21.15.67:0/1918050278 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fced4007450 con 0x7fceec072370 2024-04-17T10:39:05.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcee9600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcec8066f90 0x7fcec8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:05.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcee9600640 1 -- 172.21.15.67:0/1918050278 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fced4084ff0 con 0x7fceec072370 2024-04-17T10:39:05.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.282+0000 7fcef0c00640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcec8066f90 0x7fcec8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:05.282 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.286+0000 7fcef0c00640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcec8066f90 0x7fcec8069450 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0016e0 tx=0x7fcedc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:05.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.286+0000 7fcee9600640 1 -- 172.21.15.67:0/1918050278 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fced4055e70 con 0x7fceec072370 2024-04-17T10:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.482+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fceec155300 con 0x7fceec072370 2024-04-17T10:39:05.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.482+0000 7fcee9600640 1 -- 172.21.15.67:0/1918050278 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fced4055770 con 0x7fceec072370 2024-04-17T10:39:05.475 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:05.475 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:05.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcec8066f90 msgr2=0x7fcec8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:05.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcec8066f90 0x7fcec8069450 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fcedc0016e0 tx=0x7fcedc009040 comp rx=0 tx=0).stop 2024-04-17T10:39:05.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fceec072370 msgr2=0x7fceec0cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:05.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fceec072370 0x7fceec0cd4b0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7fced400c750 tx=0x7fced400cc10 comp rx=0 tx=0).stop 2024-04-17T10:39:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 shutdown_connections 2024-04-17T10:39:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fcec8066f90 0x7fcec8069450 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fceec15dbc0 0x7fceec158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fceec072370 0x7fceec0cd4b0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 --2- 172.21.15.67:0/1918050278 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fceec06c410 0x7fceec141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 >> 172.21.15.67:0/1918050278 conn(0x7fceec0af670 msgr2=0x7fceec0b3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 shutdown_connections 2024-04-17T10:39:05.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:05.486+0000 7fcef3600640 1 -- 172.21.15.67:0/1918050278 wait complete. 2024-04-17T10:39:05.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:05.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:06.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:07.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.066+0000 7f26f9a00640 1 Processor -- start 2024-04-17T10:39:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.066+0000 7f26f9a00640 1 -- start start 2024-04-17T10:39:07.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f4099a50 0x7f26f4070390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f26f40d0240 0x7f26f40cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26f406f5d0 con 0x7f26f409a8c0 2024-04-17T10:39:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f406ff00 con 0x7f26f40d0240 2024-04-17T10:39:07.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f406f8b0 con 0x7f26f4099a50 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f26f40d0240 0x7f26f40cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f3e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f4099a50 0x7f26f4070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f26f40d0240 0x7f26f40cbbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:37706/0 (socket says 172.21.15.67:37706) 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9000640 1 -- 172.21.15.67:0/2391333958 learned_addr learned my addr 172.21.15.67:0/2391333958 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9000640 1 -- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f4099a50 msgr2=0x7f26f4070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9000640 1 --2- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f4099a50 0x7f26f4070390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9000640 1 -- 172.21.15.67:0/2391333958 >> v1:172.21.15.86:6789/0 conn(0x7f26f409a8c0 legacy=0x7f26f4099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:07.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f9000640 1 -- 172.21.15.67:0/2391333958 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f406fb90 con 0x7f26f40d0240 2024-04-17T10:39:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f3400640 1 -- 172.21.15.67:0/2391333958 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 405647034 0 0) 0x7f26f406f5d0 con 0x7f26f409a8c0 2024-04-17T10:39:07.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.070+0000 7f26f3e00640 1 --2- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f4099a50 0x7f26f4070390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:39:07.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9000640 1 --2- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f40d0240 0x7f26f40cbbf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009e50 tx=0x7f26ec01ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=580cf37a945771d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f3400640 1 -- 172.21.15.67:0/2391333958 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0a5700 con 0x7f26f40d0240 2024-04-17T10:39:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f3400640 1 -- 172.21.15.67:0/2391333958 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec0a58a0 con 0x7f26f40d0240 2024-04-17T10:39:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f3400640 1 -- 172.21.15.67:0/2391333958 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0a5c70 con 0x7f26f40d0240 2024-04-17T10:39:07.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 -- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f40d0240 msgr2=0x7f26f40cbbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f40d0240 0x7f26f40cbbf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f26ec009e50 tx=0x7f26ec01ca30 comp rx=0 tx=0).stop 2024-04-17T10:39:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 -- 172.21.15.67:0/2391333958 shutdown_connections 2024-04-17T10:39:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f40d0240 0x7f26f40cbbf0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2391333958 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f4099a50 0x7f26f4070390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 -- 172.21.15.67:0/2391333958 >> 172.21.15.67:0/2391333958 conn(0x7f26f40d15a0 msgr2=0x7f26f40d19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 -- 172.21.15.67:0/2391333958 shutdown_connections 2024-04-17T10:39:07.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 -- 172.21.15.67:0/2391333958 wait complete. 2024-04-17T10:39:07.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 Processor -- start 2024-04-17T10:39:07.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.074+0000 7f26f9a00640 1 -- start start 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4099a50 0x7f26f40b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f409a8c0 0x7f26f40b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f40d0240 0x7f26f415cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40cc940 con 0x7f26f40d0240 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f40d0920 con 0x7f26f4099a50 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f416a7f0 con 0x7f26f409a8c0 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f40d0240 0x7f26f415cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f40d0240 0x7f26f415cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45016/0 (socket says 172.21.15.67:45016) 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 -- 172.21.15.67:0/2742083487 learned_addr learned my addr 172.21.15.67:0/2742083487 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f3e00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4099a50 0x7f26f40b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26eb400640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f409a8c0 0x7f26f40b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:07.068 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 -- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f409a8c0 msgr2=0x7f26f40b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:07.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f409a8c0 0x7f26f40b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 -- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4099a50 msgr2=0x7f26f40b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:07.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4099a50 0x7f26f40b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 -- 172.21.15.67:0/2742083487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f415bfa0 con 0x7f26f40d0240 2024-04-17T10:39:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9000640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f40d0240 0x7f26f415cb60 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0af040 tx=0x7f26ec0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f0c00640 1 -- 172.21.15.67:0/2742083487 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec009510 con 0x7f26f40d0240 2024-04-17T10:39:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f0c00640 1 -- 172.21.15.67:0/2742083487 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec0be070 con 0x7f26f40d0240 2024-04-17T10:39:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f40c8810 con 0x7f26f40d0240 2024-04-17T10:39:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f0c00640 1 -- 172.21.15.67:0/2742083487 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0b8ac0 con 0x7f26f40d0240 2024-04-17T10:39:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f40c8d90 con 0x7f26f40d0240 2024-04-17T10:39:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f4074dc0 con 0x7f26f40d0240 2024-04-17T10:39:07.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.078+0000 7f26f0c00640 1 -- 172.21.15.67:0/2742083487 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f26ec0b8c60 con 0x7f26f40d0240 2024-04-17T10:39:07.071 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.082+0000 7f26f0c00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26d0066f60 0x7f26d0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:07.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.082+0000 7f26f0c00640 1 -- 172.21.15.67:0/2742083487 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f26ec120410 con 0x7f26f40d0240 2024-04-17T10:39:07.072 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.082+0000 7f26f3e00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26d0066f60 0x7f26d0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:07.073 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.082+0000 7f26f3e00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26d0066f60 0x7f26d0069420 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009580 tx=0x7f26e4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:07.075 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.086+0000 7f26f0c00640 1 -- 172.21.15.67:0/2742083487 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f26f415bfa0 con 0x7f26f40d0240 2024-04-17T10:39:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.282+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26f40d1310 con 0x7f26f40d0240 2024-04-17T10:39:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.282+0000 7f26f0c00640 1 -- 172.21.15.67:0/2742083487 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f26ec016620 con 0x7f26f40d0240 2024-04-17T10:39:07.276 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:07.276 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26d0066f60 msgr2=0x7f26d0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26d0066f60 0x7f26d0069420 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009580 tx=0x7f26e4009190 comp rx=0 tx=0).stop 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f40d0240 msgr2=0x7f26f415cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f40d0240 0x7f26f415cb60 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0af040 tx=0x7f26ec0197d0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 shutdown_connections 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f26d0066f60 0x7f26d0069420 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f26f40d0240 0x7f26f415cb60 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f26f409a8c0 0x7f26f40b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 --2- 172.21.15.67:0/2742083487 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f26f4099a50 0x7f26f40b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 >> 172.21.15.67:0/2742083487 conn(0x7f26f40d15a0 msgr2=0x7f26f40b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 shutdown_connections 2024-04-17T10:39:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:07.286+0000 7f26f9a00640 1 -- 172.21.15.67:0/2742083487 wait complete. 2024-04-17T10:39:07.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:07.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:08.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:08.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.862+0000 7fbd9b200640 1 Processor -- start 2024-04-17T10:39:08.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd9b200640 1 -- start start 2024-04-17T10:39:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd9b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c15da90 0x7fbd9c15de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd9b200640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c15e750 con 0x7fbd9c066d70 2024-04-17T10:39:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd9b200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd9c15ea30 con 0x7fbd9c06c3b0 2024-04-17T10:39:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd9b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c15ed10 con 0x7fbd9c15da90 2024-04-17T10:39:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd99400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbd9c066d70 0x7fbd9c06bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:36178/0 (socket says 172.21.15.67:36178) 2024-04-17T10:39:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd99400640 1 -- 172.21.15.67:0/2625160560 learned_addr learned my addr 172.21.15.67:0/2625160560 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:08.856 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd99e00640 1 --2- 172.21.15.67:0/2625160560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c15da90 0x7fbd9c15de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:08.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd98a00640 1 -- 172.21.15.67:0/2625160560 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624499610 0 0) 0x7fbd9c15ea30 con 0x7fbd9c06c3b0 2024-04-17T10:39:08.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd98a00640 1 -- 172.21.15.67:0/2625160560 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd80003530 con 0x7fbd9c06c3b0 2024-04-17T10:39:08.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd98a00640 1 -- 172.21.15.67:0/2625160560 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368545783 0 0) 0x7fbd9c15e750 con 0x7fbd9c066d70 2024-04-17T10:39:08.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd98a00640 1 -- 172.21.15.67:0/2625160560 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd9c15ea30 con 0x7fbd9c066d70 2024-04-17T10:39:08.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd99e00640 1 -- 172.21.15.67:0/2625160560 >> v1:172.21.15.86:6789/0 conn(0x7fbd9c06c3b0 legacy=0x7fbd9c15f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:08.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd99e00640 1 -- 172.21.15.67:0/2625160560 >> v1:172.21.15.67:6789/0 conn(0x7fbd9c066d70 legacy=0x7fbd9c06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:08.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.866+0000 7fbd99e00640 1 -- 172.21.15.67:0/2625160560 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c15e3b0 con 0x7fbd9c15da90 2024-04-17T10:39:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd99e00640 1 --2- 172.21.15.67:0/2625160560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c15da90 0x7fbd9c15de70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0097f0 tx=0x7fbd8c01c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f6c25cfec9ecd32 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd98a00640 1 -- 172.21.15.67:0/2625160560 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0b2070 con 0x7fbd9c15da90 2024-04-17T10:39:08.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd98a00640 1 -- 172.21.15.67:0/2625160560 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c0a9070 con 0x7fbd9c15da90 2024-04-17T10:39:08.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd98a00640 1 -- 172.21.15.67:0/2625160560 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0ad880 con 0x7fbd9c15da90 2024-04-17T10:39:08.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd9b200640 1 -- 172.21.15.67:0/2625160560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c15da90 msgr2=0x7fbd9c15de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:08.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2625160560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c15da90 0x7fbd9c15de70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0097f0 tx=0x7fbd8c01c450 comp rx=0 tx=0).stop 2024-04-17T10:39:08.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd9b200640 1 -- 172.21.15.67:0/2625160560 shutdown_connections 2024-04-17T10:39:08.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2625160560 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c15da90 0x7fbd9c15de70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:08.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd9b200640 1 -- 172.21.15.67:0/2625160560 >> 172.21.15.67:0/2625160560 conn(0x7fbd9c071ad0 msgr2=0x7fbd9c071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:08.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd9b200640 1 -- 172.21.15.67:0/2625160560 shutdown_connections 2024-04-17T10:39:08.862 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.870+0000 7fbd9b200640 1 -- 172.21.15.67:0/2625160560 wait complete. 2024-04-17T10:39:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 Processor -- start 2024-04-17T10:39:08.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 -- start start 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9c066d70 0x7fbd9c13dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c06c3b0 0x7fbd9c0cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9c15da90 0x7fbd9c13e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c160aa0 con 0x7fbd9c066d70 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c160920 con 0x7fbd9c15da90 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c16b8a0 con 0x7fbd9c06c3b0 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9c066d70 0x7fbd9c13dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9c066d70 0x7fbd9c13dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45028/0 (socket says 172.21.15.67:45028) 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 -- 172.21.15.67:0/2665635408 learned_addr learned my addr 172.21.15.67:0/2665635408 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:08.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99400640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c06c3b0 0x7fbd9c0cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9a800640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9c15da90 0x7fbd9c13e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 -- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c06c3b0 msgr2=0x7fbd9c0cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c06c3b0 0x7fbd9c0cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 -- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9c15da90 msgr2=0x7fbd9c13e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9c15da90 0x7fbd9c13e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 -- 172.21.15.67:0/2665635408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c13e2e0 con 0x7fbd9c066d70 2024-04-17T10:39:08.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd99e00640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9c066d70 0x7fbd9c13dda0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c019060 tx=0x7fbd8c0191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd92000640 1 -- 172.21.15.67:0/2665635408 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0bd070 con 0x7fbd9c066d70 2024-04-17T10:39:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd92000640 1 -- 172.21.15.67:0/2665635408 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd8c0b9680 con 0x7fbd9c066d70 2024-04-17T10:39:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd92000640 1 -- 172.21.15.67:0/2665635408 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd8c0b2040 con 0x7fbd9c066d70 2024-04-17T10:39:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c06da60 con 0x7fbd9c066d70 2024-04-17T10:39:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c06e020 con 0x7fbd9c066d70 2024-04-17T10:39:08.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.874+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c074dc0 con 0x7fbd9c066d70 2024-04-17T10:39:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.878+0000 7fbd92000640 1 -- 172.21.15.67:0/2665635408 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbd8c0c0070 con 0x7fbd9c066d70 2024-04-17T10:39:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.878+0000 7fbd92000640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd7c066f60 0x7fbd7c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:08.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.878+0000 7fbd92000640 1 -- 172.21.15.67:0/2665635408 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbd8c11fa10 con 0x7fbd9c066d70 2024-04-17T10:39:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.878+0000 7fbd99400640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd7c066f60 0x7fbd7c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:08.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.878+0000 7fbd99400640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd7c066f60 0x7fbd7c069420 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c155130 tx=0x7fbd88009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:08.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:08.882+0000 7fbd92000640 1 -- 172.21.15.67:0/2665635408 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbd8c0160d0 con 0x7fbd9c066d70 2024-04-17T10:39:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd9c066b50 con 0x7fbd9c066d70 2024-04-17T10:39:09.076 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd92000640 1 -- 172.21.15.67:0/2665635408 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fbd8c0f0760 con 0x7fbd9c066d70 2024-04-17T10:39:09.077 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:09.077 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:09.077 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:09.078 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd7c066f60 msgr2=0x7fbd7c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd7c066f60 0x7fbd7c069420 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c155130 tx=0x7fbd88009040 comp rx=0 tx=0).stop 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9c066d70 msgr2=0x7fbd9c13dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9c066d70 0x7fbd9c13dda0 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c019060 tx=0x7fbd8c0191f0 comp rx=0 tx=0).stop 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 shutdown_connections 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbd7c066f60 0x7fbd7c069420 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbd9c15da90 0x7fbd9c13e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbd9c06c3b0 0x7fbd9c0cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 --2- 172.21.15.67:0/2665635408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9c066d70 0x7fbd9c13dda0 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 >> 172.21.15.67:0/2665635408 conn(0x7fbd9c071ad0 msgr2=0x7fbd9c0b05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 shutdown_connections 2024-04-17T10:39:09.079 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:09.086+0000 7fbd9b200640 1 -- 172.21.15.67:0/2665635408 wait complete. 2024-04-17T10:39:09.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:09.500 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:10.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.662+0000 7fb62bc00640 1 Processor -- start 2024-04-17T10:39:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.662+0000 7fb62bc00640 1 -- start start 2024-04-17T10:39:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb62bc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 0x7fb624143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.655 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb62bc00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62415ae40 con 0x7fb62406e6e0 2024-04-17T10:39:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb62bc00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb624071440 con 0x7fb6240cf0c0 2024-04-17T10:39:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb62bc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb624071720 con 0x7fb62406eac0 2024-04-17T10:39:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629200640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fb62406e6e0 0x7fb6240ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52102/0 (socket says 172.21.15.67:52102) 2024-04-17T10:39:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629200640 1 -- 172.21.15.67:0/678242477 learned_addr learned my addr 172.21.15.67:0/678242477 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920113123 0 0) 0x7fb62415ae40 con 0x7fb62406e6e0 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb60c005740 con 0x7fb62406e6e0 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623e00640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 0x7fb624143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587827759 0 0) 0x7fb60c005740 con 0x7fb62406e6e0 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 msgr2=0x7fb624143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 0x7fb624143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 >> v1:172.21.15.86:6789/0 conn(0x7fb6240cf0c0 legacy=0x7fb624144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb624071a00 con 0x7fb62406e6e0 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2133358483 0 0) 0x7fb618001680 con 0x7fb62406e6e0 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 >> v1:172.21.15.67:6789/0 conn(0x7fb62406e6e0 legacy=0x7fb6240ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 0x7fb60c005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006180 0x7fb60c006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.658 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb60c006f60 0x7fb60c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb624071720 con 0x7fb60c006f60 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c00dd70 con 0x7fb60c006180 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb60c00df70 con 0x7fb62406eac0 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629c00640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb60c006f60 0x7fb60c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629200640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 0x7fb60c005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb623e00640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006180 0x7fb60c006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629c00640 1 -- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 msgr2=0x7fb60c005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629c00640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62406eac0 0x7fb60c005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629c00640 1 -- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006180 msgr2=0x7fb60c006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629c00640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006180 0x7fb60c006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.666+0000 7fb629c00640 1 -- 172.21.15.67:0/678242477 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60c005740 con 0x7fb60c006f60 2024-04-17T10:39:10.660 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb629c00640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb60c006f60 0x7fb60c0073e0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fb61c001e70 tx=0x7fb61c01c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b6d89715b33f05 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c001a90 con 0x7fb60c006f60 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c003b80 con 0x7fb60c006f60 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb623400640 1 -- 172.21.15.67:0/678242477 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0050d0 con 0x7fb60c006f60 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb62bc00640 1 -- 172.21.15.67:0/678242477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb60c006f60 msgr2=0x7fb60c0073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb62bc00640 1 --2- 172.21.15.67:0/678242477 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb60c006f60 0x7fb60c0073e0 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fb61c001e70 tx=0x7fb61c01c780 comp rx=0 tx=0).stop 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb629c00640 1 -- 172.21.15.67:0/678242477 reap_dead start 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb62bc00640 1 -- 172.21.15.67:0/678242477 shutdown_connections 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb62bc00640 1 -- 172.21.15.67:0/678242477 >> 172.21.15.67:0/678242477 conn(0x7fb62406ac90 msgr2=0x7fb62406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb62bc00640 1 -- 172.21.15.67:0/678242477 shutdown_connections 2024-04-17T10:39:10.661 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.670+0000 7fb62bc00640 1 -- 172.21.15.67:0/678242477 wait complete. 2024-04-17T10:39:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 Processor -- start 2024-04-17T10:39:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 -- start start 2024-04-17T10:39:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006f60 0x7fb6241dfd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62413e560 0x7fb6240ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62413c280 0x7fb6240ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6240b40c0 con 0x7fb62413c280 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb624071fc0 con 0x7fb60c006f60 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb624143ae0 con 0x7fb62413e560 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62413c280 0x7fb6240ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62413c280 0x7fb6240ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53354/0 (socket says 172.21.15.67:53354) 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 -- 172.21.15.67:0/221277865 learned_addr learned my addr 172.21.15.67:0/221277865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629200640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006f60 0x7fb6241dfd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb623e00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62413e560 0x7fb6240ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 -- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62413e560 msgr2=0x7fb6240ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62413e560 0x7fb6240ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 -- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006f60 msgr2=0x7fb6241dfd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006f60 0x7fb6241dfd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 -- 172.21.15.67:0/221277865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6241e06f0 con 0x7fb62413c280 2024-04-17T10:39:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb623e00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62413e560 0x7fb6240ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb629c00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62413c280 0x7fb6240ace00 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb61c01cc60 tx=0x7fb60c007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:10.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb620c00640 1 -- 172.21.15.67:0/221277865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c003980 con 0x7fb62413c280 2024-04-17T10:39:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb620c00640 1 -- 172.21.15.67:0/221277865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c001210 con 0x7fb62413c280 2024-04-17T10:39:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6241e09d0 con 0x7fb62413c280 2024-04-17T10:39:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb620c00640 1 -- 172.21.15.67:0/221277865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c006d60 con 0x7fb62413c280 2024-04-17T10:39:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6241e0f90 con 0x7fb62413c280 2024-04-17T10:39:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.674+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb624072d50 con 0x7fb62413c280 2024-04-17T10:39:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.678+0000 7fb620c00640 1 -- 172.21.15.67:0/221277865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fb61c0050d0 con 0x7fb62413c280 2024-04-17T10:39:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.678+0000 7fb620c00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb604066f90 0x7fb604069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.678+0000 7fb620c00640 1 -- 172.21.15.67:0/221277865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fb61c11eba0 con 0x7fb62413c280 2024-04-17T10:39:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.678+0000 7fb629200640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb604066f90 0x7fb604069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.678+0000 7fb629200640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb604066f90 0x7fb604069450 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb618000c10 tx=0x7fb618007900 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:10.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.682+0000 7fb620c00640 1 -- 172.21.15.67:0/221277865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fb61c0ac020 con 0x7fb62413c280 2024-04-17T10:39:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb624074e30 con 0x7fb62413c280 2024-04-17T10:39:10.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb620c00640 1 -- 172.21.15.67:0/221277865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fb61c016490 con 0x7fb62413c280 2024-04-17T10:39:10.868 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:10.869 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:10.869 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb604066f90 msgr2=0x7fb604069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb604066f90 0x7fb604069450 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fb618000c10 tx=0x7fb618007900 comp rx=0 tx=0).stop 2024-04-17T10:39:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62413c280 msgr2=0x7fb6240ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62413c280 0x7fb6240ace00 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb61c01cc60 tx=0x7fb60c007a50 comp rx=0 tx=0).stop 2024-04-17T10:39:10.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 shutdown_connections 2024-04-17T10:39:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fb604066f90 0x7fb604069450 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb62413c280 0x7fb6240ace00 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fb62413e560 0x7fb6240ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 --2- 172.21.15.67:0/221277865 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb60c006f60 0x7fb6241dfd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 >> 172.21.15.67:0/221277865 conn(0x7fb62406ac90 msgr2=0x7fb6240b1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 shutdown_connections 2024-04-17T10:39:10.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:10.878+0000 7fb62bc00640 1 -- 172.21.15.67:0/221277865 wait complete. 2024-04-17T10:39:11.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:11.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:12.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:12.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.454+0000 7fd9f2a00640 1 Processor -- start 2024-04-17T10:39:12.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.454+0000 7fd9f2a00640 1 -- start start 2024-04-17T10:39:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f2a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd9ec06eac0 0x7fd9ec143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec0cf0c0 0x7fd9ec144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f2a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec144560 con 0x7fd9ec06e6e0 2024-04-17T10:39:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f2a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ec15ae40 con 0x7fd9ec06eac0 2024-04-17T10:39:12.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec071440 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec0cf0c0 0x7fd9ec144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:12.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9e3400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fd9ec06eac0 0x7fd9ec143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f0a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec0cf0c0 0x7fd9ec144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3301/0 says I am v2:172.21.15.67:41126/0 (socket says 172.21.15.67:41126) 2024-04-17T10:39:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9f0a00640 1 -- 172.21.15.67:0/1848254369 learned_addr learned my addr 172.21.15.67:0/1848254369 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9e3400640 1 -- 172.21.15.67:0/1848254369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec0cf0c0 msgr2=0x7fd9ec144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9e3400640 1 --2- 172.21.15.67:0/1848254369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec0cf0c0 0x7fd9ec144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9e3400640 1 -- 172.21.15.67:0/1848254369 >> v1:172.21.15.86:6789/0 conn(0x7fd9ec06e6e0 legacy=0x7fd9ec0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9e3400640 1 -- 172.21.15.67:0/1848254369 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec071720 con 0x7fd9ec06eac0 2024-04-17T10:39:12.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.458+0000 7fd9eb400640 1 -- 172.21.15.67:0/1848254369 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012276423 0 0) 0x7fd9ec144560 con 0x7fd9ec06e6e0 2024-04-17T10:39:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9e3400640 1 --2- 172.21.15.67:0/1848254369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 0x7fd9ec143ae0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40097c0 tx=0x7fd9d401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a08f7513d27a56c server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9eb400640 1 -- 172.21.15.67:0/1848254369 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40ab600 con 0x7fd9ec06eac0 2024-04-17T10:39:12.451 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9eb400640 1 -- 172.21.15.67:0/1848254369 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d40b0070 con 0x7fd9ec06eac0 2024-04-17T10:39:12.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9eb400640 1 -- 172.21.15.67:0/1848254369 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d40ab9c0 con 0x7fd9ec06eac0 2024-04-17T10:39:12.452 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1848254369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 msgr2=0x7fd9ec143ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1848254369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 0x7fd9ec143ae0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd9d40097c0 tx=0x7fd9d401c3e0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1848254369 shutdown_connections 2024-04-17T10:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1848254369 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec0cf0c0 0x7fd9ec144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1848254369 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 0x7fd9ec143ae0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1848254369 >> 172.21.15.67:0/1848254369 conn(0x7fd9ec06ac90 msgr2=0x7fd9ec06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1848254369 shutdown_connections 2024-04-17T10:39:12.453 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1848254369 wait complete. 2024-04-17T10:39:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 Processor -- start 2024-04-17T10:39:12.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.462+0000 7fd9f2a00640 1 -- start start 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec06e6e0 0x7fd9ec0c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 0x7fd9ec06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ec0cf0c0 0x7fd9ec0c4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0b3a40 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0b38c0 con 0x7fd9ec06eac0 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0b3bc0 con 0x7fd9ec06e6e0 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ec0cf0c0 0x7fd9ec0c4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ec0cf0c0 0x7fd9ec0c4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53368/0 (socket says 172.21.15.67:53368) 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 -- 172.21.15.67:0/1497624136 learned_addr learned my addr 172.21.15.67:0/1497624136 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9ebe00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec06e6e0 0x7fd9ec0c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9e3400640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 0x7fd9ec06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 -- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec06e6e0 msgr2=0x7fd9ec0c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec06e6e0 0x7fd9ec0c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 -- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 msgr2=0x7fd9ec06d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 0x7fd9ec06d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 -- 172.21.15.67:0/1497624136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec0c9ad0 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f0a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ec0cf0c0 0x7fd9ec0c4b80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400efc0 tx=0x7fd9e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9e8c00640 1 -- 172.21.15.67:0/1497624136 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e400cd20 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9e8c00640 1 -- 172.21.15.67:0/1497624136 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e40039a0 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec0c5600 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9e8c00640 1 -- 172.21.15.67:0/1497624136 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4003d90 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec0c5bc0 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.466+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ec072d50 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.470+0000 7fd9e8c00640 1 -- 172.21.15.67:0/1497624136 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd9e4012050 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.470+0000 7fd9e8c00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd9c8066f60 0x7fd9c8069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:12.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.470+0000 7fd9e8c00640 1 -- 172.21.15.67:0/1497624136 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fd9e4085640 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.470+0000 7fd9ebe00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd9c8066f60 0x7fd9c8069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:12.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.470+0000 7fd9ebe00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd9c8066f60 0x7fd9c8069420 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc000f70 tx=0x7fd9dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:12.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.474+0000 7fd9e8c00640 1 -- 172.21.15.67:0/1497624136 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd9e4022020 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.662 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.670+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9ec074e30 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9e8c00640 1 -- 172.21.15.67:0/1497624136 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fd9e4010040 con 0x7fd9ec0cf0c0 2024-04-17T10:39:12.663 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:12.663 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:12.663 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd9c8066f60 msgr2=0x7fd9c8069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd9c8066f60 0x7fd9c8069420 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc000f70 tx=0x7fd9dc009040 comp rx=0 tx=0).stop 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ec0cf0c0 msgr2=0x7fd9ec0c4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ec0cf0c0 0x7fd9ec0c4b80 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400efc0 tx=0x7fd9e400c490 comp rx=0 tx=0).stop 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 shutdown_connections 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd9c8066f60 0x7fd9c8069420 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd9ec0cf0c0 0x7fd9ec0c4b80 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd9ec06eac0 0x7fd9ec06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 --2- 172.21.15.67:0/1497624136 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd9ec06e6e0 0x7fd9ec0c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 >> 172.21.15.67:0/1497624136 conn(0x7fd9ec06ac90 msgr2=0x7fd9ec140440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 shutdown_connections 2024-04-17T10:39:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:12.674+0000 7fd9f2a00640 1 -- 172.21.15.67:0/1497624136 wait complete. 2024-04-17T10:39:13.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:13.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:14.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.250+0000 7f3c32200640 1 Processor -- start 2024-04-17T10:39:14.242 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.250+0000 7f3c32200640 1 -- start start 2024-04-17T10:39:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c32200640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f3c2c06eac0 0x7f3c2c143ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c32200640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c2c0cf0c0 0x7f3c2c144020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c32200640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c2c144560 con 0x7f3c2c06e6e0 2024-04-17T10:39:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c32200640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c15ae40 con 0x7f3c2c0cf0c0 2024-04-17T10:39:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c32200640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2c071440 con 0x7f3c2c06eac0 2024-04-17T10:39:14.243 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c2c0cf0c0 0x7f3c2c144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c2c0cf0c0 0x7f3c2c144020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53384/0 (socket says 172.21.15.67:53384) 2024-04-17T10:39:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 -- 172.21.15.67:0/856010804 learned_addr learned my addr 172.21.15.67:0/856010804 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c22a00640 1 --2- 172.21.15.67:0/856010804 >> v2:172.21.15.86:3300/0 conn(0x7f3c2c06eac0 0x7f3c2c143ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 -- 172.21.15.67:0/856010804 >> v2:172.21.15.86:3300/0 conn(0x7f3c2c06eac0 msgr2=0x7f3c2c143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 --2- 172.21.15.67:0/856010804 >> v2:172.21.15.86:3300/0 conn(0x7f3c2c06eac0 0x7f3c2c143ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 -- 172.21.15.67:0/856010804 >> v1:172.21.15.86:6789/0 conn(0x7f3c2c06e6e0 legacy=0x7f3c2c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 -- 172.21.15.67:0/856010804 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c071720 con 0x7f3c2c0cf0c0 2024-04-17T10:39:14.246 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.254+0000 7f3c2be00640 1 --2- 172.21.15.67:0/856010804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c0cf0c0 0x7f3c2c144020 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c06ef80 tx=0x7f3c2401e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c900699f50c0f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c2aa00640 1 -- 172.21.15.67:0/856010804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240b4070 con 0x7f3c2c0cf0c0 2024-04-17T10:39:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c2aa00640 1 -- 172.21.15.67:0/856010804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c240a7860 con 0x7f3c2c0cf0c0 2024-04-17T10:39:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c2aa00640 1 -- 172.21.15.67:0/856010804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c240a7c30 con 0x7f3c2c0cf0c0 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- 172.21.15.67:0/856010804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c0cf0c0 msgr2=0x7f3c2c144020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 --2- 172.21.15.67:0/856010804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c0cf0c0 0x7f3c2c144020 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f3c2c06ef80 tx=0x7f3c2401e890 comp rx=0 tx=0).stop 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- 172.21.15.67:0/856010804 shutdown_connections 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 --2- 172.21.15.67:0/856010804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c0cf0c0 0x7f3c2c144020 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 --2- 172.21.15.67:0/856010804 >> v2:172.21.15.86:3300/0 conn(0x7f3c2c06eac0 0x7f3c2c143ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- 172.21.15.67:0/856010804 >> 172.21.15.67:0/856010804 conn(0x7f3c2c06ac90 msgr2=0x7f3c2c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- 172.21.15.67:0/856010804 shutdown_connections 2024-04-17T10:39:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- 172.21.15.67:0/856010804 wait complete. 2024-04-17T10:39:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 Processor -- start 2024-04-17T10:39:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- start start 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c2c06e6e0 0x7f3c2c13e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c06eac0 0x7f3c2c0ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c2c0cf0c0 0x7f3c2c13f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c06dd00 con 0x7f3c2c06eac0 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c06db80 con 0x7f3c2c06e6e0 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c32200640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c2c06de80 con 0x7f3c2c0cf0c0 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c22a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c06eac0 0x7f3c2c0ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c22a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c06eac0 0x7f3c2c0ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53394/0 (socket says 172.21.15.67:53394) 2024-04-17T10:39:14.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c22a00640 1 -- 172.21.15.67:0/197640376 learned_addr learned my addr 172.21.15.67:0/197640376 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.258+0000 7f3c2b400640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c2c06e6e0 0x7f3c2c13e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c2be00640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c2c0cf0c0 0x7f3c2c13f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c22a00640 1 -- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c2c0cf0c0 msgr2=0x7f3c2c13f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c22a00640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c2c0cf0c0 0x7f3c2c13f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c22a00640 1 -- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c2c06e6e0 msgr2=0x7f3c2c13e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c22a00640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c2c06e6e0 0x7f3c2c13e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c22a00640 1 -- 172.21.15.67:0/197640376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c0cae90 con 0x7f3c2c06eac0 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c2be00640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c2c0cf0c0 0x7f3c2c13f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c22a00640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c06eac0 0x7f3c2c0ca950 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400e930 tx=0x7f3c1400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c23e00640 1 -- 172.21.15.67:0/197640376 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c14017070 con 0x7f3c2c06eac0 2024-04-17T10:39:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c23e00640 1 -- 172.21.15.67:0/197640376 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1400ca00 con 0x7f3c2c06eac0 2024-04-17T10:39:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c0c82e0 con 0x7f3c2c06eac0 2024-04-17T10:39:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c23e00640 1 -- 172.21.15.67:0/197640376 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1400cdb0 con 0x7f3c2c06eac0 2024-04-17T10:39:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c0c85a0 con 0x7f3c2c06eac0 2024-04-17T10:39:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c072d50 con 0x7f3c2c06eac0 2024-04-17T10:39:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c23e00640 1 -- 172.21.15.67:0/197640376 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f3c1401a070 con 0x7f3c2c06eac0 2024-04-17T10:39:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c23e00640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c08066f60 0x7f3c08069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.262+0000 7f3c23e00640 1 -- 172.21.15.67:0/197640376 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f3c14085270 con 0x7f3c2c06eac0 2024-04-17T10:39:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.266+0000 7f3c2b400640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c08066f60 0x7f3c08069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.266+0000 7f3c2b400640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c08066f60 0x7f3c08069420 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c001ea0 tx=0x7f3c1c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:14.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.266+0000 7f3c23e00640 1 -- 172.21.15.67:0/197640376 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f3c2c0cae90 con 0x7f3c2c06eac0 2024-04-17T10:39:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.478+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c2c074e30 con 0x7f3c2c06eac0 2024-04-17T10:39:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.478+0000 7f3c23e00640 1 -- 172.21.15.67:0/197640376 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f3c14055a30 con 0x7f3c2c06eac0 2024-04-17T10:39:14.469 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:14.469 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:14.470 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c08066f60 msgr2=0x7f3c08069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c08066f60 0x7f3c08069420 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c001ea0 tx=0x7f3c1c009040 comp rx=0 tx=0).stop 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c06eac0 msgr2=0x7f3c2c0ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c06eac0 0x7f3c2c0ca950 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400e930 tx=0x7f3c1400edf0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 shutdown_connections 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f3c08066f60 0x7f3c08069420 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f3c2c0cf0c0 0x7f3c2c13f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c2c06eac0 0x7f3c2c0ca950 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 --2- 172.21.15.67:0/197640376 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f3c2c06e6e0 0x7f3c2c13e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 >> 172.21.15.67:0/197640376 conn(0x7f3c2c06ac90 msgr2=0x7f3c2c0af040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 shutdown_connections 2024-04-17T10:39:14.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:14.482+0000 7f3c32200640 1 -- 172.21.15.67:0/197640376 wait complete. 2024-04-17T10:39:14.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:14.893 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:15.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:16.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.054+0000 7fa60e800640 1 Processor -- start 2024-04-17T10:39:16.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60e800640 1 -- start start 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa608099a50 0x7fa60815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa608099e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60e800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa60809a740 0x7fa60815f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60e800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6080cde20 con 0x7fa608099a50 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60e800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60809a370 con 0x7fa60809a740 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080cf300 con 0x7fa6080d03a0 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa608099a50 0x7fa60815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:16.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa608099a50 0x7fa60815c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53406/0 (socket says 172.21.15.67:53406) 2024-04-17T10:39:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 -- 172.21.15.67:0/1340302439 learned_addr learned my addr 172.21.15.67:0/1340302439 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60de00640 1 --2- 172.21.15.67:0/1340302439 >> v2:172.21.15.86:3300/0 conn(0x7fa60809a740 0x7fa60815f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60ca00640 1 --2- 172.21.15.67:0/1340302439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa608099e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 -- 172.21.15.67:0/1340302439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 msgr2=0x7fa608099e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 --2- 172.21.15.67:0/1340302439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa608099e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 -- 172.21.15.67:0/1340302439 >> v2:172.21.15.86:3300/0 conn(0x7fa60809a740 msgr2=0x7fa60815f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 --2- 172.21.15.67:0/1340302439 >> v2:172.21.15.86:3300/0 conn(0x7fa60809a740 0x7fa60815f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60d400640 1 -- 172.21.15.67:0/1340302439 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6080cf5e0 con 0x7fa608099a50 2024-04-17T10:39:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60de00640 1 --2- 172.21.15.67:0/1340302439 >> v2:172.21.15.86:3300/0 conn(0x7fa60809a740 0x7fa60815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:39:16.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.058+0000 7fa60ca00640 1 --2- 172.21.15.67:0/1340302439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa608099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:39:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60d400640 1 --2- 172.21.15.67:0/1340302439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa60815c7a0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fa6080b3170 tx=0x7fa5fc01c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b9c55483af06e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa607e00640 1 -- 172.21.15.67:0/1340302439 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0b0070 con 0x7fa608099a50 2024-04-17T10:39:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa607e00640 1 -- 172.21.15.67:0/1340302439 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc0ab600 con 0x7fa608099a50 2024-04-17T10:39:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa607e00640 1 -- 172.21.15.67:0/1340302439 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0ab9d0 con 0x7fa608099a50 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 -- 172.21.15.67:0/1340302439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 msgr2=0x7fa60815c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 --2- 172.21.15.67:0/1340302439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa60815c7a0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7fa6080b3170 tx=0x7fa5fc01c260 comp rx=0 tx=0).stop 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 -- 172.21.15.67:0/1340302439 shutdown_connections 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 --2- 172.21.15.67:0/1340302439 >> v2:172.21.15.86:3300/0 conn(0x7fa60809a740 0x7fa60815f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 --2- 172.21.15.67:0/1340302439 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa608099e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 --2- 172.21.15.67:0/1340302439 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa60815c7a0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 -- 172.21.15.67:0/1340302439 >> 172.21.15.67:0/1340302439 conn(0x7fa6080d1690 msgr2=0x7fa6080d1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 -- 172.21.15.67:0/1340302439 shutdown_connections 2024-04-17T10:39:16.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 -- 172.21.15.67:0/1340302439 wait complete. 2024-04-17T10:39:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 Processor -- start 2024-04-17T10:39:16.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.062+0000 7fa60e800640 1 -- start start 2024-04-17T10:39:16.056 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa608151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60809a740 0x7fa6080c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa60815d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080cc320 con 0x7fa608099a50 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080cc1a0 con 0x7fa60809a740 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080cc4a0 con 0x7fa6080d03a0 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa608151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa608151ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53412/0 (socket says 172.21.15.67:53412) 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 -- 172.21.15.67:0/2083373996 learned_addr learned my addr 172.21.15.67:0/2083373996 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60de00640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa60815d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60ca00640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60809a740 0x7fa6080c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 -- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 msgr2=0x7fa60815d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa60815d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 -- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60809a740 msgr2=0x7fa6080c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60809a740 0x7fa6080c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 -- 172.21.15.67:0/2083373996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60815daa0 con 0x7fa608099a50 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60ca00640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60809a740 0x7fa6080c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60d400640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa608151ad0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0ae040 tx=0x7fa5fc019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa605600640 1 -- 172.21.15.67:0/2083373996 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0b0040 con 0x7fa608099a50 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa605600640 1 -- 172.21.15.67:0/2083373996 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5fc0b4640 con 0x7fa608099a50 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60815dd40 con 0x7fa608099a50 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa605600640 1 -- 172.21.15.67:0/2083373996 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0b4a30 con 0x7fa608099a50 2024-04-17T10:39:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60815e300 con 0x7fa608099a50 2024-04-17T10:39:16.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.066+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa608074dc0 con 0x7fa608099a50 2024-04-17T10:39:16.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.070+0000 7fa605600640 1 -- 172.21.15.67:0/2083373996 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa5fc0ab4c0 con 0x7fa608099a50 2024-04-17T10:39:16.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.070+0000 7fa605600640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa5e8066f90 0x7fa5e8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:16.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.070+0000 7fa605600640 1 -- 172.21.15.67:0/2083373996 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa5fc11ef50 con 0x7fa608099a50 2024-04-17T10:39:16.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.070+0000 7fa60ca00640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa5e8066f90 0x7fa5e8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.070+0000 7fa60ca00640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa5e8066f90 0x7fa5e8069450 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa6080af2b0 tx=0x7fa5f4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:16.063 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.074+0000 7fa605600640 1 -- 172.21.15.67:0/2083373996 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa5fc016110 con 0x7fa608099a50 2024-04-17T10:39:16.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.266+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6080d24a0 con 0x7fa608099a50 2024-04-17T10:39:16.258 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.266+0000 7fa605600640 1 -- 172.21.15.67:0/2083373996 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fa5fc0efd30 con 0x7fa608099a50 2024-04-17T10:39:16.258 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:16.258 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:16.259 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa5e8066f90 msgr2=0x7fa5e8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa5e8066f90 0x7fa5e8069450 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fa6080af2b0 tx=0x7fa5f4008040 comp rx=0 tx=0).stop 2024-04-17T10:39:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 msgr2=0x7fa608151ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa608151ad0 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0ae040 tx=0x7fa5fc019000 comp rx=0 tx=0).stop 2024-04-17T10:39:16.260 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 shutdown_connections 2024-04-17T10:39:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa5e8066f90 0x7fa5e8069450 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa6080d03a0 0x7fa60815d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa60809a740 0x7fa6080c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 --2- 172.21.15.67:0/2083373996 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa608099a50 0x7fa608151ad0 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 >> 172.21.15.67:0/2083373996 conn(0x7fa6080d1690 msgr2=0x7fa6081522e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 shutdown_connections 2024-04-17T10:39:16.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:16.270+0000 7fa60e800640 1 -- 172.21.15.67:0/2083373996 wait complete. 2024-04-17T10:39:16.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:16.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:17.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a8a00640 1 Processor -- start 2024-04-17T10:39:17.835 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a8a00640 1 -- start start 2024-04-17T10:39:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a8a00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7ff9a4145040 0x7ff9a406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a8a00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a406e2f0 con 0x7ff9a4071a20 2024-04-17T10:39:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a8a00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a406d6e0 con 0x7ff9a406ef10 2024-04-17T10:39:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a8a00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a40a86c0 con 0x7ff9a4145040 2024-04-17T10:39:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a2800640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff9a4071a20 0x7ff9a40cc300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52112/0 (socket says 172.21.15.67:52112) 2024-04-17T10:39:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a2800640 1 -- 172.21.15.67:0/2405731860 learned_addr learned my addr 172.21.15.67:0/2405731860 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:17.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a1e00640 1 --2- 172.21.15.67:0/2405731860 >> v2:172.21.15.86:3300/0 conn(0x7ff9a4145040 0x7ff9a406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1733467872 0 0) 0x7ff9a406e2f0 con 0x7ff9a4071a20 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff988003530 con 0x7ff9a4071a20 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523323600 0 0) 0x7ff9a406d6e0 con 0x7ff9a406ef10 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a406e2f0 con 0x7ff9a406ef10 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a1e00640 1 -- 172.21.15.67:0/2405731860 >> v1:172.21.15.86:6789/0 conn(0x7ff9a406ef10 legacy=0x7ff9a4071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a1e00640 1 -- 172.21.15.67:0/2405731860 >> v1:172.21.15.67:6789/0 conn(0x7ff9a4071a20 legacy=0x7ff9a40cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a1e00640 1 -- 172.21.15.67:0/2405731860 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a406d9c0 con 0x7ff9a4145040 2024-04-17T10:39:17.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.846+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3902993751 0 0) 0x7ff988003530 con 0x7ff9a4071a20 2024-04-17T10:39:17.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a1e00640 1 --2- 172.21.15.67:0/2405731860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 0x7ff9a406e9d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff9940097c0 tx=0x7ff99401ebb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdf4bde6c5ef3be0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9940b5070 con 0x7ff9a4145040 2024-04-17T10:39:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9940a8630 con 0x7ff9a4145040 2024-04-17T10:39:17.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a0a00640 1 -- 172.21.15.67:0/2405731860 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff99c001a00 con 0x7ff9a4145040 2024-04-17T10:39:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a8a00640 1 -- 172.21.15.67:0/2405731860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 msgr2=0x7ff9a406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/2405731860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 0x7ff9a406e9d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7ff9940097c0 tx=0x7ff99401ebb0 comp rx=0 tx=0).stop 2024-04-17T10:39:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a8a00640 1 -- 172.21.15.67:0/2405731860 shutdown_connections 2024-04-17T10:39:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/2405731860 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 0x7ff9a406e9d0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:17.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a8a00640 1 -- 172.21.15.67:0/2405731860 >> 172.21.15.67:0/2405731860 conn(0x7ff9a406ac90 msgr2=0x7ff9a406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a8a00640 1 -- 172.21.15.67:0/2405731860 shutdown_connections 2024-04-17T10:39:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.850+0000 7ff9a8a00640 1 -- 172.21.15.67:0/2405731860 wait complete. 2024-04-17T10:39:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 Processor -- start 2024-04-17T10:39:17.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 -- start start 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a406ef10 0x7ff9a413e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a4071a20 0x7ff9a4142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 0x7ff9a415f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a406df80 con 0x7ff9a4071a20 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a415bf10 con 0x7ff9a4145040 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a406d370 con 0x7ff9a406ef10 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a4071a20 0x7ff9a4142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a4071a20 0x7ff9a4142300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53414/0 (socket says 172.21.15.67:53414) 2024-04-17T10:39:17.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 -- 172.21.15.67:0/1176602262 learned_addr learned my addr 172.21.15.67:0/1176602262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1e00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a406ef10 0x7ff9a413e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a2800640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 0x7ff9a415f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 -- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a406ef10 msgr2=0x7ff9a413e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a406ef10 0x7ff9a413e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 -- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 msgr2=0x7ff9a415f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 0x7ff9a415f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 -- 172.21.15.67:0/1176602262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a413e4a0 con 0x7ff9a4071a20 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a1400640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a4071a20 0x7ff9a4142300 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7ff99000c500 tx=0x7ff99000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff99a000640 1 -- 172.21.15.67:0/1176602262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9900052d0 con 0x7ff9a4071a20 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff99a000640 1 -- 172.21.15.67:0/1176602262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff990005490 con 0x7ff9a4071a20 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a41412c0 con 0x7ff9a4071a20 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff99a000640 1 -- 172.21.15.67:0/1176602262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff990012850 con 0x7ff9a4071a20 2024-04-17T10:39:17.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.854+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a4141b00 con 0x7ff9a4071a20 2024-04-17T10:39:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.858+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a4072d50 con 0x7ff9a4071a20 2024-04-17T10:39:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.858+0000 7ff99a000640 1 -- 172.21.15.67:0/1176602262 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff9900129f0 con 0x7ff9a4071a20 2024-04-17T10:39:17.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.858+0000 7ff99a000640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff984066f60 0x7ff984069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.858+0000 7ff99a000640 1 -- 172.21.15.67:0/1176602262 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff990084e70 con 0x7ff9a4071a20 2024-04-17T10:39:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.858+0000 7ff9a1e00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff984066f60 0x7ff984069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:17.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.858+0000 7ff9a1e00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff984066f60 0x7ff984069420 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff9940b2000 tx=0x7ff9940aa070 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:17.862+0000 7ff99a000640 1 -- 172.21.15.67:0/1176602262 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff9a41412c0 con 0x7ff9a4071a20 2024-04-17T10:39:18.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9a40cbef0 con 0x7ff9a4071a20 2024-04-17T10:39:18.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff99a000640 1 -- 172.21.15.67:0/1176602262 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7ff990055580 con 0x7ff9a4071a20 2024-04-17T10:39:18.052 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:18.052 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:18.052 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff984066f60 msgr2=0x7ff984069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff984066f60 0x7ff984069420 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff9940b2000 tx=0x7ff9940aa070 comp rx=0 tx=0).stop 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a4071a20 msgr2=0x7ff9a4142300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a4071a20 0x7ff9a4142300 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7ff99000c500 tx=0x7ff99000c9c0 comp rx=0 tx=0).stop 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 shutdown_connections 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff984066f60 0x7ff984069420 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff9a4145040 0x7ff9a415f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff9a4071a20 0x7ff9a4142300 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 --2- 172.21.15.67:0/1176602262 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff9a406ef10 0x7ff9a413e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 >> 172.21.15.67:0/1176602262 conn(0x7ff9a406ac90 msgr2=0x7ff9a406b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:18.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 shutdown_connections 2024-04-17T10:39:18.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:18.062+0000 7ff9a8a00640 1 -- 172.21.15.67:0/1176602262 wait complete. 2024-04-17T10:39:18.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:18.475 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:19.476 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.638+0000 7f137d800640 1 Processor -- start 2024-04-17T10:39:19.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.638+0000 7f137d800640 1 -- start start 2024-04-17T10:39:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f137d800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f137806e6e0 0x7f13780ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f137d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806eac0 0x7f13780ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f137d800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f137815ae40 con 0x7f1378143ae0 2024-04-17T10:39:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f137d800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13780cf430 con 0x7f137806e6e0 2024-04-17T10:39:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f137d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1378071440 con 0x7f137806eac0 2024-04-17T10:39:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1377400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f1378143ae0 0x7f1378143f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:52128/0 (socket says 172.21.15.67:52128) 2024-04-17T10:39:19.631 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1377400640 1 -- 172.21.15.67:0/3399599157 learned_addr learned my addr 172.21.15.67:0/3399599157 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1376a00640 1 --2- 172.21.15.67:0/3399599157 >> v2:172.21.15.86:3300/0 conn(0x7f137806e6e0 0x7f13780ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f136fe00640 1 --2- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806eac0 0x7f13780ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1376a00640 1 -- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806eac0 msgr2=0x7f13780ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1376a00640 1 --2- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806eac0 0x7f13780ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.632 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1376a00640 1 -- 172.21.15.67:0/3399599157 >> v1:172.21.15.67:6789/0 conn(0x7f1378143ae0 legacy=0x7f1378143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:19.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1376000640 1 -- 172.21.15.67:0/3399599157 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890153623 0 0) 0x7f137815ae40 con 0x7f1378143ae0 2024-04-17T10:39:19.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.642+0000 7f1376a00640 1 -- 172.21.15.67:0/3399599157 --> v2:172.21.15.86:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1378071720 con 0x7f137806e6e0 2024-04-17T10:39:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f1376a00640 1 --2- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137806e6e0 0x7f13780ce9b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f13680097c0 tx=0x7f136801c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91482b626d12528a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f1376000640 1 -- 172.21.15.67:0/3399599157 <== mon.1 v2:172.21.15.86:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13680b0070 con 0x7f137806e6e0 2024-04-17T10:39:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f1376000640 1 -- 172.21.15.67:0/3399599157 <== mon.1 v2:172.21.15.86:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13680ab600 con 0x7f137806e6e0 2024-04-17T10:39:19.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f1376000640 1 -- 172.21.15.67:0/3399599157 <== mon.1 v2:172.21.15.86:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13680ab9d0 con 0x7f137806e6e0 2024-04-17T10:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 -- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137806e6e0 msgr2=0x7f13780ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 --2- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137806e6e0 0x7f13780ce9b0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f13680097c0 tx=0x7f136801c3e0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 -- 172.21.15.67:0/3399599157 shutdown_connections 2024-04-17T10:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 --2- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806eac0 0x7f13780ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 --2- 172.21.15.67:0/3399599157 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f137806e6e0 0x7f13780ce9b0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 -- 172.21.15.67:0/3399599157 >> 172.21.15.67:0/3399599157 conn(0x7f137806ac90 msgr2=0x7f137806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:19.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 -- 172.21.15.67:0/3399599157 shutdown_connections 2024-04-17T10:39:19.637 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 -- 172.21.15.67:0/3399599157 wait complete. 2024-04-17T10:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 Processor -- start 2024-04-17T10:39:19.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.646+0000 7f137d800640 1 -- start start 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806e6e0 0x7f1378157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f137806eac0 0x7f137813c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1378143ae0 0x7f137813ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137806dd00 con 0x7f137806eac0 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137806db80 con 0x7f1378143ae0 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137806de80 con 0x7f137806e6e0 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f137806eac0 0x7f137813c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f137806eac0 0x7f137813c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53416/0 (socket says 172.21.15.67:53416) 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 -- 172.21.15.67:0/2780654714 learned_addr learned my addr 172.21.15.67:0/2780654714 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f1376a00640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806e6e0 0x7f1378157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:19.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f1377400640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1378143ae0 0x7f137813ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 -- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806e6e0 msgr2=0x7f1378157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806e6e0 0x7f1378157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 -- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1378143ae0 msgr2=0x7f137813ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1378143ae0 0x7f137813ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 -- 172.21.15.67:0/2780654714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1378137d50 con 0x7f137806eac0 2024-04-17T10:39:19.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136fe00640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f137806eac0 0x7f137813c440 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f136000e930 tx=0x7f136000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136ea00640 1 -- 172.21.15.67:0/2780654714 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1360017070 con 0x7f137806eac0 2024-04-17T10:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136ea00640 1 -- 172.21.15.67:0/2780654714 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136000c9a0 con 0x7f137806eac0 2024-04-17T10:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136ea00640 1 -- 172.21.15.67:0/2780654714 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136000cd50 con 0x7f137806eac0 2024-04-17T10:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1378138030 con 0x7f137806eac0 2024-04-17T10:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1378140b50 con 0x7f137806eac0 2024-04-17T10:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1378072d50 con 0x7f137806eac0 2024-04-17T10:39:19.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.650+0000 7f136ea00640 1 -- 172.21.15.67:0/2780654714 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f136001a070 con 0x7f137806eac0 2024-04-17T10:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.654+0000 7f136ea00640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1354066f60 0x7f1354069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.654+0000 7f136ea00640 1 -- 172.21.15.67:0/2780654714 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1360085530 con 0x7f137806eac0 2024-04-17T10:39:19.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.654+0000 7f1376a00640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1354066f60 0x7f1354069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:19.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.654+0000 7f1376a00640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1354066f60 0x7f1354069420 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f13680ad000 tx=0x7f13680a6000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:19.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.658+0000 7f136ea00640 1 -- 172.21.15.67:0/2780654714 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1360055d80 con 0x7f137806eac0 2024-04-17T10:39:19.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.850+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1378074e30 con 0x7f137806eac0 2024-04-17T10:39:19.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.850+0000 7f136ea00640 1 -- 172.21.15.67:0/2780654714 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f136005c0d0 con 0x7f137806eac0 2024-04-17T10:39:19.842 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:19.842 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:19.842 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1354066f60 msgr2=0x7f1354069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1354066f60 0x7f1354069420 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f13680ad000 tx=0x7f13680a6000 comp rx=0 tx=0).stop 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f137806eac0 msgr2=0x7f137813c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f137806eac0 0x7f137813c440 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f136000e930 tx=0x7f136000edf0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 shutdown_connections 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1354066f60 0x7f1354069420 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f1378143ae0 0x7f137813ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f137806eac0 0x7f137813c440 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 --2- 172.21.15.67:0/2780654714 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f137806e6e0 0x7f1378157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 >> 172.21.15.67:0/2780654714 conn(0x7f137806ac90 msgr2=0x7f13780b1b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 shutdown_connections 2024-04-17T10:39:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:19.854+0000 7f137d800640 1 -- 172.21.15.67:0/2780654714 wait complete. 2024-04-17T10:39:20.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:20.268 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:21.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:21.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 Processor -- start 2024-04-17T10:39:21.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 -- start start 2024-04-17T10:39:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe200cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe2006b6a0 0x7fbe2006f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fbe200cf920 0x7fbe200cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe200cad20 con 0x7fbe2006b6a0 2024-04-17T10:39:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe200700a0 con 0x7fbe200cf920 2024-04-17T10:39:21.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe26400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe20070380 con 0x7fbe2015de50 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe2006b6a0 0x7fbe2006f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbe2006b6a0 0x7fbe2006f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38616/0 (socket says 172.21.15.67:38616) 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 -- 172.21.15.67:0/2389416138 learned_addr learned my addr 172.21.15.67:0/2389416138 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe1fe00640 1 --2- 172.21.15.67:0/2389416138 >> v2:172.21.15.86:3300/0 conn(0x7fbe200cf920 0x7fbe200cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe1f400640 1 --2- 172.21.15.67:0/2389416138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe200cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 -- 172.21.15.67:0/2389416138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 msgr2=0x7fbe200cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 --2- 172.21.15.67:0/2389416138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe200cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 -- 172.21.15.67:0/2389416138 >> v2:172.21.15.86:3300/0 conn(0x7fbe200cf920 msgr2=0x7fbe200cfdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 --2- 172.21.15.67:0/2389416138 >> v2:172.21.15.86:3300/0 conn(0x7fbe200cf920 0x7fbe200cfdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.434+0000 7fbe16a00640 1 -- 172.21.15.67:0/2389416138 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe200caf70 con 0x7fbe2006b6a0 2024-04-17T10:39:21.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe16a00640 1 --2- 172.21.15.67:0/2389416138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2006b6a0 0x7fbe2006f970 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fbe200b3340 tx=0x7fbe0801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44dcc70ee1af42ec server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:21.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe1ea00640 1 -- 172.21.15.67:0/2389416138 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080a55e0 con 0x7fbe2006b6a0 2024-04-17T10:39:21.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe1ea00640 1 -- 172.21.15.67:0/2389416138 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe080a5780 con 0x7fbe2006b6a0 2024-04-17T10:39:21.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe1ea00640 1 -- 172.21.15.67:0/2389416138 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080a5b50 con 0x7fbe2006b6a0 2024-04-17T10:39:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 -- 172.21.15.67:0/2389416138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2006b6a0 msgr2=0x7fbe2006f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 --2- 172.21.15.67:0/2389416138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2006b6a0 0x7fbe2006f970 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fbe200b3340 tx=0x7fbe0801c960 comp rx=0 tx=0).stop 2024-04-17T10:39:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 -- 172.21.15.67:0/2389416138 shutdown_connections 2024-04-17T10:39:21.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 --2- 172.21.15.67:0/2389416138 >> v2:172.21.15.86:3300/0 conn(0x7fbe200cf920 0x7fbe200cfdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 --2- 172.21.15.67:0/2389416138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe2006b6a0 0x7fbe2006f970 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 --2- 172.21.15.67:0/2389416138 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe200cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 -- 172.21.15.67:0/2389416138 >> 172.21.15.67:0/2389416138 conn(0x7fbe200b22a0 msgr2=0x7fbe200b26b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 -- 172.21.15.67:0/2389416138 shutdown_connections 2024-04-17T10:39:21.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.438+0000 7fbe26400640 1 -- 172.21.15.67:0/2389416138 wait complete. 2024-04-17T10:39:21.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 Processor -- start 2024-04-17T10:39:21.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 -- start start 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe2006b6a0 0x7fbe200ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe200cf920 0x7fbe20146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe201599b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe200cc2f0 con 0x7fbe200cf920 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe200cc170 con 0x7fbe2006b6a0 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2006e4a0 con 0x7fbe2015de50 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe200cf920 0x7fbe20146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe200cf920 0x7fbe20146b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38626/0 (socket says 172.21.15.67:38626) 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 -- 172.21.15.67:0/2744205031 learned_addr learned my addr 172.21.15.67:0/2744205031 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe1f400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe2006b6a0 0x7fbe200ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe1fe00640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe201599b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:21.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 -- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 msgr2=0x7fbe201599b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:21.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe201599b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 -- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe2006b6a0 msgr2=0x7fbe200ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:21.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe2006b6a0 0x7fbe200ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 -- 172.21.15.67:0/2744205031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe200c5380 con 0x7fbe200cf920 2024-04-17T10:39:21.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe16a00640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe200cf920 0x7fbe20146b30 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fbe0801d200 tx=0x7fbe08019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:21.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe17e00640 1 -- 172.21.15.67:0/2744205031 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe080bc020 con 0x7fbe200cf920 2024-04-17T10:39:21.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe17e00640 1 -- 172.21.15.67:0/2744205031 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe080b88a0 con 0x7fbe200cf920 2024-04-17T10:39:21.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe200c5660 con 0x7fbe200cf920 2024-04-17T10:39:21.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe17e00640 1 -- 172.21.15.67:0/2744205031 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe08009510 con 0x7fbe200cf920 2024-04-17T10:39:21.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe20159ef0 con 0x7fbe200cf920 2024-04-17T10:39:21.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.442+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2009e900 con 0x7fbe200cf920 2024-04-17T10:39:21.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.446+0000 7fbe17e00640 1 -- 172.21.15.67:0/2744205031 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fbe080bf020 con 0x7fbe200cf920 2024-04-17T10:39:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.446+0000 7fbe17e00640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdfc066f90 0x7fbdfc069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:21.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.446+0000 7fbe17e00640 1 -- 172.21.15.67:0/2744205031 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fbe08120100 con 0x7fbe200cf920 2024-04-17T10:39:21.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.446+0000 7fbe1f400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdfc066f90 0x7fbdfc069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:21.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.446+0000 7fbe1f400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdfc066f90 0x7fbdfc069450 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fbe20156d40 tx=0x7fbe10008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:21.441 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.450+0000 7fbe17e00640 1 -- 172.21.15.67:0/2744205031 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fbe080c6070 con 0x7fbe200cf920 2024-04-17T10:39:21.635 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe20074e30 con 0x7fbe200cf920 2024-04-17T10:39:21.636 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe17e00640 1 -- 172.21.15.67:0/2744205031 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fbe080165e0 con 0x7fbe200cf920 2024-04-17T10:39:21.636 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:21.636 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:21.636 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:21.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdfc066f90 msgr2=0x7fbdfc069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:21.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdfc066f90 0x7fbdfc069450 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fbe20156d40 tx=0x7fbe10008040 comp rx=0 tx=0).stop 2024-04-17T10:39:21.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe200cf920 msgr2=0x7fbe20146b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:21.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe200cf920 0x7fbe20146b30 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7fbe0801d200 tx=0x7fbe08019700 comp rx=0 tx=0).stop 2024-04-17T10:39:21.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 shutdown_connections 2024-04-17T10:39:21.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fbdfc066f90 0x7fbdfc069450 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fbe2015de50 0x7fbe201599b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe200cf920 0x7fbe20146b30 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 --2- 172.21.15.67:0/2744205031 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fbe2006b6a0 0x7fbe200ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:21.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 >> 172.21.15.67:0/2744205031 conn(0x7fbe200b22a0 msgr2=0x7fbe200b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:21.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 shutdown_connections 2024-04-17T10:39:21.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:21.646+0000 7fbe26400640 1 -- 172.21.15.67:0/2744205031 wait complete. 2024-04-17T10:39:22.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:22.060 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:23.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.222+0000 7f0f61000640 1 Processor -- start 2024-04-17T10:39:23.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f61000640 1 -- start start 2024-04-17T10:39:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f61000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c15dbc0 0x7f0f5c06f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f61000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f5c06db90 0x7f0f5c071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f61000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5c0ad8f0 con 0x7f0f5c0721a0 2024-04-17T10:39:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f61000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5c15dfa0 con 0x7f0f5c06db90 2024-04-17T10:39:23.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f61000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c06faa0 con 0x7f0f5c15dbc0 2024-04-17T10:39:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f51400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f5c06db90 0x7f0f5c071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f59e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c15dbc0 0x7f0f5c06f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f5a800640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0f5c0721a0 0x7f0f5c06b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.67:44604/0 (socket says 172.21.15.67:44604) 2024-04-17T10:39:23.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f51400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f0f5c06db90 0x7f0f5c071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.86:3300/0 says I am v2:172.21.15.67:40326/0 (socket says 172.21.15.67:40326) 2024-04-17T10:39:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f5a800640 1 -- 172.21.15.67:0/1333792332 learned_addr learned my addr 172.21.15.67:0/1333792332 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f59400640 1 -- 172.21.15.67:0/1333792332 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206842031 0 0) 0x7f0f5c0ad8f0 con 0x7f0f5c0721a0 2024-04-17T10:39:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f59e00640 1 -- 172.21.15.67:0/1333792332 >> v2:172.21.15.86:3300/0 conn(0x7f0f5c06db90 msgr2=0x7f0f5c071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f59e00640 1 --2- 172.21.15.67:0/1333792332 >> v2:172.21.15.86:3300/0 conn(0x7f0f5c06db90 0x7f0f5c071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f59e00640 1 -- 172.21.15.67:0/1333792332 >> v1:172.21.15.86:6789/0 conn(0x7f0f5c0721a0 legacy=0x7f0f5c06b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:23.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.226+0000 7f0f59e00640 1 -- 172.21.15.67:0/1333792332 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c0adbd0 con 0x7f0f5c15dbc0 2024-04-17T10:39:23.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f59e00640 1 --2- 172.21.15.67:0/1333792332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c15dbc0 0x7f0f5c06f560 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0097c0 tx=0x7f0f4c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5572380425e6ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:23.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f59400640 1 -- 172.21.15.67:0/1333792332 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c0b0070 con 0x7f0f5c15dbc0 2024-04-17T10:39:23.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f59400640 1 -- 172.21.15.67:0/1333792332 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c0ab600 con 0x7f0f5c15dbc0 2024-04-17T10:39:23.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f59400640 1 -- 172.21.15.67:0/1333792332 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c0ab9d0 con 0x7f0f5c15dbc0 2024-04-17T10:39:23.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 -- 172.21.15.67:0/1333792332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c15dbc0 msgr2=0x7f0f5c06f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:23.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 --2- 172.21.15.67:0/1333792332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c15dbc0 0x7f0f5c06f560 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0097c0 tx=0x7f0f4c01c410 comp rx=0 tx=0).stop 2024-04-17T10:39:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 -- 172.21.15.67:0/1333792332 shutdown_connections 2024-04-17T10:39:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 --2- 172.21.15.67:0/1333792332 >> v2:172.21.15.86:3300/0 conn(0x7f0f5c06db90 0x7f0f5c071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 --2- 172.21.15.67:0/1333792332 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c15dbc0 0x7f0f5c06f560 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 -- 172.21.15.67:0/1333792332 >> 172.21.15.67:0/1333792332 conn(0x7f0f5c0af670 msgr2=0x7f0f5c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 -- 172.21.15.67:0/1333792332 shutdown_connections 2024-04-17T10:39:23.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.230+0000 7f0f61000640 1 -- 172.21.15.67:0/1333792332 wait complete. 2024-04-17T10:39:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 Processor -- start 2024-04-17T10:39:23.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 -- start start 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f5c06db90 0x7f0f5c13a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c0721a0 0x7f0f5c0cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c15dbc0 0x7f0f5c14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c06c6f0 con 0x7f0f5c06db90 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c06c570 con 0x7f0f5c15dbc0 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c06af70 con 0x7f0f5c0721a0 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f5c06db90 0x7f0f5c13a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f5c06db90 0x7f0f5c13a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38634/0 (socket says 172.21.15.67:38634) 2024-04-17T10:39:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 -- 172.21.15.67:0/2590018836 learned_addr learned my addr 172.21.15.67:0/2590018836 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f51400640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c0721a0 0x7f0f5c0cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f5a800640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c15dbc0 0x7f0f5c14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 -- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c0721a0 msgr2=0x7f0f5c0cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c0721a0 0x7f0f5c0cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 -- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c15dbc0 msgr2=0x7f0f5c14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c15dbc0 0x7f0f5c14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 -- 172.21.15.67:0/2590018836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5c0cc4d0 con 0x7f0f5c06db90 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f5a800640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c15dbc0 0x7f0f5c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:39:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f59e00640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f5c06db90 0x7f0f5c13a7b0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c019020 tx=0x7f0f4c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f52a00640 1 -- 172.21.15.67:0/2590018836 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c0b9070 con 0x7f0f5c06db90 2024-04-17T10:39:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f52a00640 1 -- 172.21.15.67:0/2590018836 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4c0b5480 con 0x7f0f5c06db90 2024-04-17T10:39:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f52a00640 1 -- 172.21.15.67:0/2590018836 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c0b0040 con 0x7f0f5c06db90 2024-04-17T10:39:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c0cc7b0 con 0x7f0f5c06db90 2024-04-17T10:39:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.234+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c0ccd70 con 0x7f0f5c06db90 2024-04-17T10:39:23.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.238+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c09e260 con 0x7f0f5c06db90 2024-04-17T10:39:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.238+0000 7f0f52a00640 1 -- 172.21.15.67:0/2590018836 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f0f4c0bc070 con 0x7f0f5c06db90 2024-04-17T10:39:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.238+0000 7f0f52a00640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0f38066f60 0x7f0f38069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:23.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.238+0000 7f0f52a00640 1 -- 172.21.15.67:0/2590018836 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f0f4c11f4f0 con 0x7f0f5c06db90 2024-04-17T10:39:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.238+0000 7f0f51400640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0f38066f60 0x7f0f38069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:23.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.238+0000 7f0f51400640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0f38066f60 0x7f0f38069420 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c06af20 tx=0x7f0f4400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:23.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.242+0000 7f0f52a00640 1 -- 172.21.15.67:0/2590018836 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f0f4c0c3030 con 0x7f0f5c06db90 2024-04-17T10:39:23.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.434+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f5c0b1910 con 0x7f0f5c06db90 2024-04-17T10:39:23.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f52a00640 1 -- 172.21.15.67:0/2590018836 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f0f4c016090 con 0x7f0f5c06db90 2024-04-17T10:39:23.427 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:23.427 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:23.427 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0f38066f60 msgr2=0x7f0f38069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0f38066f60 0x7f0f38069420 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c06af20 tx=0x7f0f4400b040 comp rx=0 tx=0).stop 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f5c06db90 msgr2=0x7f0f5c13a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f5c06db90 0x7f0f5c13a7b0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c019020 tx=0x7f0f4c0191b0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 shutdown_connections 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f0f38066f60 0x7f0f38069420 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0f5c15dbc0 0x7f0f5c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f0f5c0721a0 0x7f0f5c0cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 --2- 172.21.15.67:0/2590018836 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0f5c06db90 0x7f0f5c13a7b0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 >> 172.21.15.67:0/2590018836 conn(0x7f0f5c0af670 msgr2=0x7f0f5c0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 shutdown_connections 2024-04-17T10:39:23.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:23.438+0000 7f0f61000640 1 -- 172.21.15.67:0/2590018836 wait complete. 2024-04-17T10:39:23.850 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:23.850 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:24.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.010+0000 7fd102600640 1 Processor -- start 2024-04-17T10:39:25.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.010+0000 7fd102600640 1 -- start start 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd102600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc145040 0x7fd0fc06e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd102600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0fc071a20 0x7fd0fc0cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd102600640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc06e2f0 con 0x7fd0fc06ef10 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd102600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc0a86c0 con 0x7fd0fc071a20 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd102600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc15bf10 con 0x7fd0fc145040 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0fc071a20 0x7fd0fc0cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fbe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd0fc071a20 0x7fd0fc0cc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38638/0 (socket says 172.21.15.67:38638) 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fbe00640 1 -- 172.21.15.67:0/2065258551 learned_addr learned my addr 172.21.15.67:0/2065258551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:25.004 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fb400640 1 --2- 172.21.15.67:0/2065258551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc145040 0x7fd0fc06e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:25.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fbe00640 1 -- 172.21.15.67:0/2065258551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc145040 msgr2=0x7fd0fc06e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:25.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fbe00640 1 --2- 172.21.15.67:0/2065258551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc145040 0x7fd0fc06e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fbe00640 1 -- 172.21.15.67:0/2065258551 >> v1:172.21.15.86:6789/0 conn(0x7fd0fc06ef10 legacy=0x7fd0fc071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:25.005 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.014+0000 7fd0fbe00640 1 -- 172.21.15.67:0/2065258551 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc06d6e0 con 0x7fd0fc071a20 2024-04-17T10:39:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd0fbe00640 1 --2- 172.21.15.67:0/2065258551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc071a20 0x7fd0fc0cc300 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f401f210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd895228aec124d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd0faa00640 1 -- 172.21.15.67:0/2065258551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40b4070 con 0x7fd0fc071a20 2024-04-17T10:39:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd0faa00640 1 -- 172.21.15.67:0/2065258551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f40af400 con 0x7fd0fc071a20 2024-04-17T10:39:25.007 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd0faa00640 1 -- 172.21.15.67:0/2065258551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40af7d0 con 0x7fd0fc071a20 2024-04-17T10:39:25.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 -- 172.21.15.67:0/2065258551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc071a20 msgr2=0x7fd0fc0cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:25.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 --2- 172.21.15.67:0/2065258551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc071a20 0x7fd0fc0cc300 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4009e50 tx=0x7fd0f401f210 comp rx=0 tx=0).stop 2024-04-17T10:39:25.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 -- 172.21.15.67:0/2065258551 shutdown_connections 2024-04-17T10:39:25.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 --2- 172.21.15.67:0/2065258551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc071a20 0x7fd0fc0cc300 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.008 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 --2- 172.21.15.67:0/2065258551 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc145040 0x7fd0fc06e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 -- 172.21.15.67:0/2065258551 >> 172.21.15.67:0/2065258551 conn(0x7fd0fc06ac90 msgr2=0x7fd0fc06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:25.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 -- 172.21.15.67:0/2065258551 shutdown_connections 2024-04-17T10:39:25.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 -- 172.21.15.67:0/2065258551 wait complete. 2024-04-17T10:39:25.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 Processor -- start 2024-04-17T10:39:25.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.018+0000 7fd102600640 1 -- start start 2024-04-17T10:39:25.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc06ef10 0x7fd0fc1f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:25.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0fc071a20 0x7fd0fc143630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:25.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc145040 0x7fd0fc143b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:25.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc06df80 con 0x7fd0fc145040 2024-04-17T10:39:25.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc06dca0 con 0x7fd0fc071a20 2024-04-17T10:39:25.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc06d370 con 0x7fd0fc06ef10 2024-04-17T10:39:25.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc145040 0x7fd0fc143b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc145040 0x7fd0fc143b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38644/0 (socket says 172.21.15.67:38644) 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 -- 172.21.15.67:0/715523985 learned_addr learned my addr 172.21.15.67:0/715523985 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fb400640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc06ef10 0x7fd0fc1f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0f2a00640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0fc071a20 0x7fd0fc143630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 -- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc06ef10 msgr2=0x7fd0fc1f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc06ef10 0x7fd0fc1f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 -- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0fc071a20 msgr2=0x7fd0fc143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0fc071a20 0x7fd0fc143630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 -- 172.21.15.67:0/715523985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc13b160 con 0x7fd0fc145040 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0fbe00640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc145040 0x7fd0fc143b70 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd0f401ce40 tx=0x7fd0f401bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0f3e00640 1 -- 172.21.15.67:0/715523985 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40b4040 con 0x7fd0fc145040 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0f3e00640 1 -- 172.21.15.67:0/715523985 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f40afd30 con 0x7fd0fc145040 2024-04-17T10:39:25.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0f3e00640 1 -- 172.21.15.67:0/715523985 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4009510 con 0x7fd0fc145040 2024-04-17T10:39:25.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc13b400 con 0x7fd0fc145040 2024-04-17T10:39:25.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc138cc0 con 0x7fd0fc145040 2024-04-17T10:39:25.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc072d50 con 0x7fd0fc145040 2024-04-17T10:39:25.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.022+0000 7fd0f3e00640 1 -- 172.21.15.67:0/715523985 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fd0f40c2020 con 0x7fd0fc145040 2024-04-17T10:39:25.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.026+0000 7fd0f3e00640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0d8066f90 0x7fd0d8069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:25.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.026+0000 7fd0f3e00640 1 -- 172.21.15.67:0/715523985 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fd0f4122b40 con 0x7fd0fc145040 2024-04-17T10:39:25.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.026+0000 7fd0fb400640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0d8066f90 0x7fd0d8069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:25.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.026+0000 7fd0fb400640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0d8066f90 0x7fd0d8069450 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0079c0 tx=0x7fd0ec008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:25.019 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.030+0000 7fd0f3e00640 1 -- 172.21.15.67:0/715523985 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fd0f40c9020 con 0x7fd0fc145040 2024-04-17T10:39:25.213 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.222+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0fc074e30 con 0x7fd0fc145040 2024-04-17T10:39:25.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.222+0000 7fd0f3e00640 1 -- 172.21.15.67:0/715523985 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fd0f4018d70 con 0x7fd0fc145040 2024-04-17T10:39:25.214 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:25.214 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:25.214 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0d8066f90 msgr2=0x7fd0d8069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0d8066f90 0x7fd0d8069450 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec0079c0 tx=0x7fd0ec008040 comp rx=0 tx=0).stop 2024-04-17T10:39:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc145040 msgr2=0x7fd0fc143b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc145040 0x7fd0fc143b70 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fd0f401ce40 tx=0x7fd0f401bfb0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 shutdown_connections 2024-04-17T10:39:25.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fd0d8066f90 0x7fd0d8069450 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0fc145040 0x7fd0fc143b70 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fd0fc071a20 0x7fd0fc143630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 --2- 172.21.15.67:0/715523985 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fd0fc06ef10 0x7fd0fc1f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 >> 172.21.15.67:0/715523985 conn(0x7fd0fc06ac90 msgr2=0x7fd0fc0b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 shutdown_connections 2024-04-17T10:39:25.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:25.226+0000 7fd102600640 1 -- 172.21.15.67:0/715523985 wait complete. 2024-04-17T10:39:25.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:25.638 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:26.639 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:26.791 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f7600640 1 Processor -- start 2024-04-17T10:39:26.792 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f7600640 1 -- start start 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f7600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0145040 0x7f64f006e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f7600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64f0071a20 0x7f64f00cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f7600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f006e2f0 con 0x7f64f006ef10 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f7600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f00a86c0 con 0x7f64f0071a20 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f7600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f015bf10 con 0x7f64f0145040 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f5600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f64f0071a20 0x7f64f00cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64e7e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f64f006ef10 0x7f64f0071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39320/0 (socket says 172.21.15.67:39320) 2024-04-17T10:39:26.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64e7e00640 1 -- 172.21.15.67:0/1361793269 learned_addr learned my addr 172.21.15.67:0/1361793269 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f4c00640 1 --2- 172.21.15.67:0/1361793269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0145040 0x7f64f006e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f5600640 1 -- 172.21.15.67:0/1361793269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0145040 msgr2=0x7f64f006e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f5600640 1 --2- 172.21.15.67:0/1361793269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0145040 0x7f64f006e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f5600640 1 -- 172.21.15.67:0/1361793269 >> v1:172.21.15.67:6789/0 conn(0x7f64f006ef10 legacy=0x7f64f0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64f5600640 1 -- 172.21.15.67:0/1361793269 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f006d6e0 con 0x7f64f0071a20 2024-04-17T10:39:26.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.802+0000 7f64efe00640 1 -- 172.21.15.67:0/1361793269 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950105580 0 0) 0x7f64f006e2f0 con 0x7f64f006ef10 2024-04-17T10:39:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f5600640 1 --2- 172.21.15.67:0/1361793269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f0071a20 0x7f64f00cc300 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009d90 tx=0x7f64e801f150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d302e46d79d7e2e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:26.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64efe00640 1 -- 172.21.15.67:0/1361793269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e80b4070 con 0x7f64f0071a20 2024-04-17T10:39:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64efe00640 1 -- 172.21.15.67:0/1361793269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e80af600 con 0x7f64f0071a20 2024-04-17T10:39:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64efe00640 1 -- 172.21.15.67:0/1361793269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e80af9d0 con 0x7f64f0071a20 2024-04-17T10:39:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 -- 172.21.15.67:0/1361793269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f0071a20 msgr2=0x7f64f00cc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 --2- 172.21.15.67:0/1361793269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f0071a20 0x7f64f00cc300 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009d90 tx=0x7f64e801f150 comp rx=0 tx=0).stop 2024-04-17T10:39:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 -- 172.21.15.67:0/1361793269 shutdown_connections 2024-04-17T10:39:26.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 --2- 172.21.15.67:0/1361793269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f0071a20 0x7f64f00cc300 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 --2- 172.21.15.67:0/1361793269 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0145040 0x7f64f006e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 -- 172.21.15.67:0/1361793269 >> 172.21.15.67:0/1361793269 conn(0x7f64f006ac90 msgr2=0x7f64f006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 -- 172.21.15.67:0/1361793269 shutdown_connections 2024-04-17T10:39:26.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.806+0000 7f64f7600640 1 -- 172.21.15.67:0/1361793269 wait complete. 2024-04-17T10:39:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 Processor -- start 2024-04-17T10:39:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 -- start start 2024-04-17T10:39:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f006ef10 0x7f64f0141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:26.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0071a20 0x7f64f015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f0145040 0x7f64f00af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f006df80 con 0x7f64f006ef10 2024-04-17T10:39:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f006dca0 con 0x7f64f0145040 2024-04-17T10:39:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f006d370 con 0x7f64f0071a20 2024-04-17T10:39:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f006ef10 0x7f64f0141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:26.805 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f006ef10 0x7f64f0141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38670/0 (socket says 172.21.15.67:38670) 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 -- 172.21.15.67:0/541467875 learned_addr learned my addr 172.21.15.67:0/541467875 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f5600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f0145040 0x7f64f00af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64e7e00640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0071a20 0x7f64f015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 -- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0071a20 msgr2=0x7f64f015b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0071a20 0x7f64f015b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 -- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f0145040 msgr2=0x7f64f00af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f0145040 0x7f64f00af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 -- 172.21.15.67:0/541467875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f00aeba0 con 0x7f64f006ef10 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f5600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f0145040 0x7f64f00af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64e7e00640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0071a20 0x7f64f015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f4c00640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f006ef10 0x7f64f0141bd0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f64e000d620 tx=0x7f64e000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64ed600640 1 -- 172.21.15.67:0/541467875 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e000b620 con 0x7f64f006ef10 2024-04-17T10:39:26.806 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64ed600640 1 -- 172.21.15.67:0/541467875 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e0019070 con 0x7f64f006ef10 2024-04-17T10:39:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64ed600640 1 -- 172.21.15.67:0/541467875 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e000b9e0 con 0x7f64f006ef10 2024-04-17T10:39:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f0142240 con 0x7f64f006ef10 2024-04-17T10:39:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.810+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f0143f60 con 0x7f64f006ef10 2024-04-17T10:39:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.814+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f0144390 con 0x7f64f006ef10 2024-04-17T10:39:26.807 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.814+0000 7f64ed600640 1 -- 172.21.15.67:0/541467875 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f64e000bb80 con 0x7f64f006ef10 2024-04-17T10:39:26.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.814+0000 7f64ed600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f64cc066f60 0x7f64cc069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:26.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.814+0000 7f64ed600640 1 -- 172.21.15.67:0/541467875 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f64e00855d0 con 0x7f64f006ef10 2024-04-17T10:39:26.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.814+0000 7f64e7e00640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f64cc066f60 0x7f64cc069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:26.808 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.814+0000 7f64e7e00640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f64cc066f60 0x7f64cc069420 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f64f01f48b0 tx=0x7f64d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:26.809 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:26.818+0000 7f64ed600640 1 -- 172.21.15.67:0/541467875 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f64e0056430 con 0x7f64f006ef10 2024-04-17T10:39:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.014+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64f0074e30 con 0x7f64f006ef10 2024-04-17T10:39:27.006 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.014+0000 7f64ed600640 1 -- 172.21.15.67:0/541467875 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f64e0055d30 con 0x7f64f006ef10 2024-04-17T10:39:27.007 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:27.007 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:27.007 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f64cc066f60 msgr2=0x7f64cc069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f64cc066f60 0x7f64cc069420 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f64f01f48b0 tx=0x7f64d8009040 comp rx=0 tx=0).stop 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f006ef10 msgr2=0x7f64f0141bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f006ef10 0x7f64f0141bd0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f64e000d620 tx=0x7f64e000dae0 comp rx=0 tx=0).stop 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 shutdown_connections 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f64cc066f60 0x7f64cc069420 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f64f0145040 0x7f64f00af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:27.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f64f0071a20 0x7f64f015b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:27.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 --2- 172.21.15.67:0/541467875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f64f006ef10 0x7f64f0141bd0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:27.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 >> 172.21.15.67:0/541467875 conn(0x7f64f006ac90 msgr2=0x7f64f006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:27.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 shutdown_connections 2024-04-17T10:39:27.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:27.018+0000 7f64f7600640 1 -- 172.21.15.67:0/541467875 wait complete. 2024-04-17T10:39:27.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:27.432 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:28.433 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:28.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d31800640 1 Processor -- start 2024-04-17T10:39:28.587 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d31800640 1 -- start start 2024-04-17T10:39:28.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d31800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d2c15b870 0x7f2d2c15fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:28.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d31800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f2d2c06f560 0x7f2d2c15bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:28.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d31800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c06aea0 con 0x7f2d2c15c190 2024-04-17T10:39:28.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d31800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c06b7d0 con 0x7f2d2c15b870 2024-04-17T10:39:28.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d31800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c06b180 con 0x7f2d2c06f560 2024-04-17T10:39:28.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2b400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2d2c15c190 0x7f2d2c071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:39332/0 (socket says 172.21.15.67:39332) 2024-04-17T10:39:28.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2aa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2d2c15b870 0x7f2d2c15fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:28.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2b400640 1 -- 172.21.15.67:0/1645713803 learned_addr learned my addr 172.21.15.67:0/1645713803 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:28.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d22000640 1 --2- 172.21.15.67:0/1645713803 >> v2:172.21.15.86:3300/0 conn(0x7f2d2c06f560 0x7f2d2c15bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:28.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2a000640 1 -- 172.21.15.67:0/1645713803 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592445424 0 0) 0x7f2d2c06aea0 con 0x7f2d2c15c190 2024-04-17T10:39:28.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2a000640 1 -- 172.21.15.67:0/1645713803 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d10003530 con 0x7f2d2c15c190 2024-04-17T10:39:28.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2aa00640 1 -- 172.21.15.67:0/1645713803 >> v2:172.21.15.86:3300/0 conn(0x7f2d2c06f560 msgr2=0x7f2d2c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:28.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2aa00640 1 --2- 172.21.15.67:0/1645713803 >> v2:172.21.15.86:3300/0 conn(0x7f2d2c06f560 0x7f2d2c15bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2aa00640 1 -- 172.21.15.67:0/1645713803 >> v1:172.21.15.67:6789/0 conn(0x7f2d2c15c190 legacy=0x7f2d2c071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:28.590 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.598+0000 7f2d2aa00640 1 -- 172.21.15.67:0/1645713803 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c06b460 con 0x7f2d2c15b870 2024-04-17T10:39:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d2aa00640 1 --2- 172.21.15.67:0/1645713803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c15b870 0x7f2d2c15fe10 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0ae410 tx=0x7f2d1c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8b838998a6c699 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d2a000640 1 -- 172.21.15.67:0/1645713803 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c0b2070 con 0x7f2d2c15b870 2024-04-17T10:39:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d2a000640 1 -- 172.21.15.67:0/1645713803 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0a5760 con 0x7f2d2c15b870 2024-04-17T10:39:28.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d2a000640 1 -- 172.21.15.67:0/1645713803 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c0a5b30 con 0x7f2d2c15b870 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 -- 172.21.15.67:0/1645713803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c15b870 msgr2=0x7f2d2c15fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 --2- 172.21.15.67:0/1645713803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c15b870 0x7f2d2c15fe10 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0ae410 tx=0x7f2d1c01c8a0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 -- 172.21.15.67:0/1645713803 shutdown_connections 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 --2- 172.21.15.67:0/1645713803 >> v2:172.21.15.86:3300/0 conn(0x7f2d2c06f560 0x7f2d2c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 --2- 172.21.15.67:0/1645713803 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c15b870 0x7f2d2c15fe10 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 -- 172.21.15.67:0/1645713803 >> 172.21.15.67:0/1645713803 conn(0x7f2d2c0af670 msgr2=0x7f2d2c0b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 -- 172.21.15.67:0/1645713803 shutdown_connections 2024-04-17T10:39:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 -- 172.21.15.67:0/1645713803 wait complete. 2024-04-17T10:39:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.602+0000 7f2d31800640 1 Processor -- start 2024-04-17T10:39:28.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 -- start start 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c06f560 0x7f2d2c14ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d2c15b870 0x7f2d2c165900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d2c15c190 0x7f2d2c165e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c06c470 con 0x7f2d2c06f560 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c072580 con 0x7f2d2c15c190 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c06c5f0 con 0x7f2d2c15b870 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c06f560 0x7f2d2c14ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c06f560 0x7f2d2c14ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38686/0 (socket says 172.21.15.67:38686) 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 -- 172.21.15.67:0/144721469 learned_addr learned my addr 172.21.15.67:0/144721469 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2b400640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d2c15c190 0x7f2d2c165e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:28.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d22000640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d2c15b870 0x7f2d2c165900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 -- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d2c15b870 msgr2=0x7f2d2c165900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d2c15b870 0x7f2d2c165900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 -- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d2c15c190 msgr2=0x7f2d2c165e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d2c15c190 0x7f2d2c165e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 -- 172.21.15.67:0/144721469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c1556d0 con 0x7f2d2c06f560 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d2aa00640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c06f560 0x7f2d2c14ec10 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0194b0 tx=0x7f2d1c019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d23400640 1 -- 172.21.15.67:0/144721469 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c0bd020 con 0x7f2d2c06f560 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d23400640 1 -- 172.21.15.67:0/144721469 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0a5e10 con 0x7f2d2c06f560 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d23400640 1 -- 172.21.15.67:0/144721469 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c0b2040 con 0x7f2d2c06f560 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c1559b0 con 0x7f2d2c06f560 2024-04-17T10:39:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c15df80 con 0x7f2d2c06f560 2024-04-17T10:39:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.606+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c09e260 con 0x7f2d2c06f560 2024-04-17T10:39:28.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.610+0000 7f2d23400640 1 -- 172.21.15.67:0/144721469 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f2d1c0c0020 con 0x7f2d2c06f560 2024-04-17T10:39:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.610+0000 7f2d23400640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2d08066f60 0x7f2d08069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:28.600 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.610+0000 7f2d23400640 1 -- 172.21.15.67:0/144721469 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f2d1c120250 con 0x7f2d2c06f560 2024-04-17T10:39:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.610+0000 7f2d22000640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2d08066f60 0x7f2d08069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.610+0000 7f2d22000640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2d08066f60 0x7f2d08069420 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c14fee0 tx=0x7f2d14008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:28.604 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.614+0000 7f2d23400640 1 -- 172.21.15.67:0/144721469 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f2d1c0c7030 con 0x7f2d2c06f560 2024-04-17T10:39:28.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d2c074e30 con 0x7f2d2c06f560 2024-04-17T10:39:28.800 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d23400640 1 -- 172.21.15.67:0/144721469 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f2d1c016520 con 0x7f2d2c06f560 2024-04-17T10:39:28.800 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:28.800 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:28.801 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2d08066f60 msgr2=0x7f2d08069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2d08066f60 0x7f2d08069420 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c14fee0 tx=0x7f2d14008040 comp rx=0 tx=0).stop 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c06f560 msgr2=0x7f2d2c14ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c06f560 0x7f2d2c14ec10 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0194b0 tx=0x7f2d1c019640 comp rx=0 tx=0).stop 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 shutdown_connections 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f2d08066f60 0x7f2d08069420 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f2d2c15c190 0x7f2d2c165e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f2d2c15b870 0x7f2d2c165900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 --2- 172.21.15.67:0/144721469 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2d2c06f560 0x7f2d2c14ec10 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:28.803 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 >> 172.21.15.67:0/144721469 conn(0x7f2d2c0af670 msgr2=0x7f2d2c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:28.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 shutdown_connections 2024-04-17T10:39:28.804 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:28.810+0000 7f2d31800640 1 -- 172.21.15.67:0/144721469 wait complete. 2024-04-17T10:39:28.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:39:28.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:39:29.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:29.224 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:30.225 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:30.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.386+0000 7f6eac400640 1 Processor -- start 2024-04-17T10:39:30.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.386+0000 7f6eac400640 1 -- start start 2024-04-17T10:39:30.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eac400640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6ea406c410 0x7f6ea406f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:30.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eac400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6ea4071c60 0x7f6ea40720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:30.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eac400640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea40ad8f0 con 0x7f6ea415dbc0 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eac400640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea415dfa0 con 0x7f6ea4071c60 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eac400640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea406faa0 con 0x7f6ea406c410 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6ea9a00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6ea415dbc0 0x7f6ea406db90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53814/0 (socket says 172.21.15.67:53814) 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6ea9a00640 1 -- 172.21.15.67:0/1170106223 learned_addr learned my addr 172.21.15.67:0/1170106223 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eaa400640 1 --2- 172.21.15.67:0/1170106223 >> v2:172.21.15.67:3300/0 conn(0x7f6ea4071c60 0x7f6ea40720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6ea9000640 1 --2- 172.21.15.67:0/1170106223 >> v2:172.21.15.86:3300/0 conn(0x7f6ea406c410 0x7f6ea406f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6ea3e00640 1 -- 172.21.15.67:0/1170106223 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673462746 0 0) 0x7f6ea40ad8f0 con 0x7f6ea415dbc0 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6ea3e00640 1 -- 172.21.15.67:0/1170106223 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c003530 con 0x7f6ea415dbc0 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eaa400640 1 -- 172.21.15.67:0/1170106223 >> v2:172.21.15.86:3300/0 conn(0x7f6ea406c410 msgr2=0x7f6ea406f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eaa400640 1 --2- 172.21.15.67:0/1170106223 >> v2:172.21.15.86:3300/0 conn(0x7f6ea406c410 0x7f6ea406f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eaa400640 1 -- 172.21.15.67:0/1170106223 >> v1:172.21.15.67:6789/0 conn(0x7f6ea415dbc0 legacy=0x7f6ea406db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:30.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eaa400640 1 -- 172.21.15.67:0/1170106223 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea40adbd0 con 0x7f6ea4071c60 2024-04-17T10:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.390+0000 7f6eaa400640 1 --2- 172.21.15.67:0/1170106223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea4071c60 0x7f6ea40720f0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4071c10 tx=0x7f6e9c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cd6e7b1eeafc0bf server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6ea3e00640 1 -- 172.21.15.67:0/1170106223 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0b2070 con 0x7f6ea4071c60 2024-04-17T10:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6ea3e00640 1 -- 172.21.15.67:0/1170106223 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c0a5760 con 0x7f6ea4071c60 2024-04-17T10:39:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6ea3e00640 1 -- 172.21.15.67:0/1170106223 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0a5b10 con 0x7f6ea4071c60 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- 172.21.15.67:0/1170106223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea4071c60 msgr2=0x7f6ea40720f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 --2- 172.21.15.67:0/1170106223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea4071c60 0x7f6ea40720f0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f6ea4071c10 tx=0x7f6e9c01c8a0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- 172.21.15.67:0/1170106223 shutdown_connections 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 --2- 172.21.15.67:0/1170106223 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea4071c60 0x7f6ea40720f0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 --2- 172.21.15.67:0/1170106223 >> v2:172.21.15.86:3300/0 conn(0x7f6ea406c410 0x7f6ea406f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- 172.21.15.67:0/1170106223 >> 172.21.15.67:0/1170106223 conn(0x7f6ea40af670 msgr2=0x7f6ea40b1c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- 172.21.15.67:0/1170106223 shutdown_connections 2024-04-17T10:39:30.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- 172.21.15.67:0/1170106223 wait complete. 2024-04-17T10:39:30.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 Processor -- start 2024-04-17T10:39:30.385 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- start start 2024-04-17T10:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea406c410 0x7f6ea40cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea4071c60 0x7f6ea414aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea415dbc0 0x7f6ea414d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea416a100 con 0x7f6ea415dbc0 2024-04-17T10:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea406af70 con 0x7f6ea4071c60 2024-04-17T10:39:30.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eac400640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea416a280 con 0x7f6ea406c410 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eaa400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea415dbc0 0x7f6ea414d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eaa400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea415dbc0 0x7f6ea414d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44840/0 (socket says 172.21.15.67:44840) 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6eaa400640 1 -- 172.21.15.67:0/736782634 learned_addr learned my addr 172.21.15.67:0/736782634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6ea9000640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea4071c60 0x7f6ea414aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.394+0000 7f6ea9a00640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea406c410 0x7f6ea40cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eaa400640 1 -- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea406c410 msgr2=0x7f6ea40cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eaa400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea406c410 0x7f6ea40cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eaa400640 1 -- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea4071c60 msgr2=0x7f6ea414aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eaa400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea4071c60 0x7f6ea414aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eaa400640 1 -- 172.21.15.67:0/736782634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea4157370 con 0x7f6ea415dbc0 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6ea9a00640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea406c410 0x7f6ea40cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eaa400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea415dbc0 0x7f6ea414d300 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c01a4d0 tx=0x7f6e9c019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6ea1600640 1 -- 172.21.15.67:0/736782634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0b2050 con 0x7f6ea415dbc0 2024-04-17T10:39:30.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6ea1600640 1 -- 172.21.15.67:0/736782634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9c0be070 con 0x7f6ea415dbc0 2024-04-17T10:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6ea1600640 1 -- 172.21.15.67:0/736782634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9c0ada70 con 0x7f6ea415dbc0 2024-04-17T10:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea414c5a0 con 0x7f6ea415dbc0 2024-04-17T10:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea414cde0 con 0x7f6ea415dbc0 2024-04-17T10:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea409e260 con 0x7f6ea415dbc0 2024-04-17T10:39:30.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.398+0000 7f6ea1600640 1 -- 172.21.15.67:0/736782634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6e9c0a55e0 con 0x7f6ea415dbc0 2024-04-17T10:39:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.402+0000 7f6ea1600640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e84066f60 0x7f6e84069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:30.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.402+0000 7f6ea1600640 1 -- 172.21.15.67:0/736782634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6e9c1204b0 con 0x7f6ea415dbc0 2024-04-17T10:39:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.402+0000 7f6ea9a00640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e84066f60 0x7f6e84069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:30.391 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.402+0000 7f6ea9a00640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e84066f60 0x7f6e84069420 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f6e98005890 tx=0x7f6e98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:30.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.406+0000 7f6ea1600640 1 -- 172.21.15.67:0/736782634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6ea414c5a0 con 0x7f6ea415dbc0 2024-04-17T10:39:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.602+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ea4074e30 con 0x7f6ea415dbc0 2024-04-17T10:39:30.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.602+0000 7f6ea1600640 1 -- 172.21.15.67:0/736782634 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6e9c016520 con 0x7f6ea415dbc0 2024-04-17T10:39:30.593 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:30.593 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:30.593 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.602+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e84066f60 msgr2=0x7f6e84069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e84066f60 0x7f6e84069420 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f6e98005890 tx=0x7f6e98009040 comp rx=0 tx=0).stop 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea415dbc0 msgr2=0x7f6ea414d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea415dbc0 0x7f6ea414d300 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f6e9c01a4d0 tx=0x7f6e9c019640 comp rx=0 tx=0).stop 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 shutdown_connections 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6e84066f60 0x7f6e84069420 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6ea415dbc0 0x7f6ea414d300 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.595 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6ea4071c60 0x7f6ea414aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 --2- 172.21.15.67:0/736782634 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6ea406c410 0x7f6ea40cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 >> 172.21.15.67:0/736782634 conn(0x7f6ea40af670 msgr2=0x7f6ea40d0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 shutdown_connections 2024-04-17T10:39:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:30.606+0000 7f6eac400640 1 -- 172.21.15.67:0/736782634 wait complete. 2024-04-17T10:39:31.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:31.017 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:32.018 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:32.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.182+0000 7ff0a7800640 1 Processor -- start 2024-04-17T10:39:32.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.182+0000 7ff0a7800640 1 -- start start 2024-04-17T10:39:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a7800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a0146da0 0x7ff0a0071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a7800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff0a0071850 0x7ff0a00adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a7800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a0161740 con 0x7ff0a01466e0 2024-04-17T10:39:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a7800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a0161a20 con 0x7ff0a0071850 2024-04-17T10:39:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a7800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a0161d00 con 0x7ff0a0146da0 2024-04-17T10:39:32.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a4e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff0a01466e0 0x7ff0a006eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53824/0 (socket says 172.21.15.67:53824) 2024-04-17T10:39:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a4e00640 1 -- 172.21.15.67:0/1158118623 learned_addr learned my addr 172.21.15.67:0/1158118623 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a5800640 1 --2- 172.21.15.67:0/1158118623 >> v2:172.21.15.67:3300/0 conn(0x7ff0a0071850 0x7ff0a00adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff097e00640 1 --2- 172.21.15.67:0/1158118623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a0146da0 0x7ff0a0071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff09fe00640 1 -- 172.21.15.67:0/1158118623 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298884341 0 0) 0x7ff0a0161740 con 0x7ff0a01466e0 2024-04-17T10:39:32.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff09fe00640 1 -- 172.21.15.67:0/1158118623 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff084003530 con 0x7ff0a01466e0 2024-04-17T10:39:32.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a5800640 1 -- 172.21.15.67:0/1158118623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a0146da0 msgr2=0x7ff0a0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:32.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a5800640 1 --2- 172.21.15.67:0/1158118623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a0146da0 0x7ff0a0071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a5800640 1 -- 172.21.15.67:0/1158118623 >> v1:172.21.15.67:6789/0 conn(0x7ff0a01466e0 legacy=0x7ff0a006eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:32.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.186+0000 7ff0a5800640 1 -- 172.21.15.67:0/1158118623 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a0161fe0 con 0x7ff0a0071850 2024-04-17T10:39:32.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a5800640 1 --2- 172.21.15.67:0/1158118623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0071850 0x7ff0a00adde0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff098009d90 tx=0x7ff09801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed0ac9e18a7084f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff09fe00640 1 -- 172.21.15.67:0/1158118623 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980b2070 con 0x7ff0a0071850 2024-04-17T10:39:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff09fe00640 1 -- 172.21.15.67:0/1158118623 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0980a57b0 con 0x7ff0a0071850 2024-04-17T10:39:32.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff09fe00640 1 -- 172.21.15.67:0/1158118623 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980a5b80 con 0x7ff0a0071850 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 -- 172.21.15.67:0/1158118623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0071850 msgr2=0x7ff0a00adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 --2- 172.21.15.67:0/1158118623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0071850 0x7ff0a00adde0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7ff098009d90 tx=0x7ff09801c970 comp rx=0 tx=0).stop 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 -- 172.21.15.67:0/1158118623 shutdown_connections 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 --2- 172.21.15.67:0/1158118623 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0071850 0x7ff0a00adde0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 --2- 172.21.15.67:0/1158118623 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a0146da0 0x7ff0a0071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 -- 172.21.15.67:0/1158118623 >> 172.21.15.67:0/1158118623 conn(0x7ff0a006ac90 msgr2=0x7ff0a006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 -- 172.21.15.67:0/1158118623 shutdown_connections 2024-04-17T10:39:32.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 -- 172.21.15.67:0/1158118623 wait complete. 2024-04-17T10:39:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 Processor -- start 2024-04-17T10:39:32.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.190+0000 7ff0a7800640 1 -- start start 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0a0071850 0x7ff0a015d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a01466e0 0x7ff0a00b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0146da0 0x7ff0a015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00aeec0 con 0x7ff0a0146da0 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00aed40 con 0x7ff0a0071850 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a00af040 con 0x7ff0a01466e0 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0146da0 0x7ff0a015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:32.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0146da0 0x7ff0a015d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44856/0 (socket says 172.21.15.67:44856) 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 -- 172.21.15.67:0/4010675795 learned_addr learned my addr 172.21.15.67:0/4010675795 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff097e00640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a01466e0 0x7ff0a00b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a4e00640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0a0071850 0x7ff0a015d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 -- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a01466e0 msgr2=0x7ff0a00b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a01466e0 0x7ff0a00b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 -- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0a0071850 msgr2=0x7ff0a015d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0a0071850 0x7ff0a015d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 -- 172.21.15.67:0/4010675795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a00b2280 con 0x7ff0a0146da0 2024-04-17T10:39:32.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a5800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0146da0 0x7ff0a015d880 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff0980b0040 tx=0x7ff098019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff09d600640 1 -- 172.21.15.67:0/4010675795 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980b2050 con 0x7ff0a0146da0 2024-04-17T10:39:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff09d600640 1 -- 172.21.15.67:0/4010675795 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0980be070 con 0x7ff0a0146da0 2024-04-17T10:39:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff09d600640 1 -- 172.21.15.67:0/4010675795 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098009450 con 0x7ff0a0146da0 2024-04-17T10:39:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a00b0ca0 con 0x7ff0a0146da0 2024-04-17T10:39:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a00b1260 con 0x7ff0a0146da0 2024-04-17T10:39:32.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.194+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a0072d50 con 0x7ff0a0146da0 2024-04-17T10:39:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.198+0000 7ff09d600640 1 -- 172.21.15.67:0/4010675795 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7ff0980adc40 con 0x7ff0a0146da0 2024-04-17T10:39:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.198+0000 7ff09d600640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff07c066f60 0x7ff07c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:32.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.198+0000 7ff09d600640 1 -- 172.21.15.67:0/4010675795 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7ff098120300 con 0x7ff0a0146da0 2024-04-17T10:39:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.198+0000 7ff0a4e00640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff07c066f60 0x7ff07c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:32.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.198+0000 7ff0a4e00640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff07c066f60 0x7ff07c069420 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff090005890 tx=0x7ff090009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:32.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.202+0000 7ff09d600640 1 -- 172.21.15.67:0/4010675795 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7ff098016560 con 0x7ff0a0146da0 2024-04-17T10:39:32.386 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.394+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0a00af660 con 0x7ff0a0146da0 2024-04-17T10:39:32.387 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff09d600640 1 -- 172.21.15.67:0/4010675795 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7ff0980f10e0 con 0x7ff0a0146da0 2024-04-17T10:39:32.387 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:32.387 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:32.387 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff07c066f60 msgr2=0x7ff07c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff07c066f60 0x7ff07c069420 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff090005890 tx=0x7ff090009040 comp rx=0 tx=0).stop 2024-04-17T10:39:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0146da0 msgr2=0x7ff0a015d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0146da0 0x7ff0a015d880 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7ff0980b0040 tx=0x7ff098019710 comp rx=0 tx=0).stop 2024-04-17T10:39:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 shutdown_connections 2024-04-17T10:39:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7ff07c066f60 0x7ff07c069420 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0a0146da0 0x7ff0a015d880 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7ff0a01466e0 0x7ff0a00b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 --2- 172.21.15.67:0/4010675795 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7ff0a0071850 0x7ff0a015d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 >> 172.21.15.67:0/4010675795 conn(0x7ff0a006ac90 msgr2=0x7ff0a00b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 shutdown_connections 2024-04-17T10:39:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:32.398+0000 7ff0a7800640 1 -- 172.21.15.67:0/4010675795 wait complete. 2024-04-17T10:39:32.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:32.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:33.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 Processor -- start 2024-04-17T10:39:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 -- start start 2024-04-17T10:39:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107c06e6e0 0x7f107c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f107c06eac0 0x7f107c0ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c143ae0 0x7f107c143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c0cf430 con 0x7f107c06e6e0 2024-04-17T10:39:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f107c15ae40 con 0x7f107c06eac0 2024-04-17T10:39:33.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1083000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c071440 con 0x7f107c143ae0 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f107be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107c06e6e0 0x7f107c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f107be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f107c06e6e0 0x7f107c0ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44862/0 (socket says 172.21.15.67:44862) 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f107be00640 1 -- 172.21.15.67:0/1541768502 learned_addr learned my addr 172.21.15.67:0/1541768502 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1073400640 1 --2- 172.21.15.67:0/1541768502 >> v2:172.21.15.86:3300/0 conn(0x7f107c06eac0 0x7f107c0ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.978+0000 7f1081000640 1 --2- 172.21.15.67:0/1541768502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c143ae0 0x7f107c143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107be00640 1 -- 172.21.15.67:0/1541768502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c143ae0 msgr2=0x7f107c143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107be00640 1 --2- 172.21.15.67:0/1541768502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c143ae0 0x7f107c143f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107be00640 1 -- 172.21.15.67:0/1541768502 >> v2:172.21.15.86:3300/0 conn(0x7f107c06eac0 msgr2=0x7f107c0ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107be00640 1 --2- 172.21.15.67:0/1541768502 >> v2:172.21.15.86:3300/0 conn(0x7f107c06eac0 0x7f107c0ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:33.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107be00640 1 -- 172.21.15.67:0/1541768502 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c071720 con 0x7f107c06e6e0 2024-04-17T10:39:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107be00640 1 --2- 172.21.15.67:0/1541768502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c06e6e0 0x7f107c0ce9b0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f107c06ef80 tx=0x7f106c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=984552007d025d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107b400640 1 -- 172.21.15.67:0/1541768502 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0ab620 con 0x7f107c06e6e0 2024-04-17T10:39:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107b400640 1 -- 172.21.15.67:0/1541768502 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106c0b0070 con 0x7f107c06e6e0 2024-04-17T10:39:33.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f107b400640 1 -- 172.21.15.67:0/1541768502 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106c0ab9e0 con 0x7f107c06e6e0 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f1083000640 1 -- 172.21.15.67:0/1541768502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c06e6e0 msgr2=0x7f107c0ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.982+0000 7f1083000640 1 --2- 172.21.15.67:0/1541768502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c06e6e0 0x7f107c0ce9b0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f107c06ef80 tx=0x7f106c01c460 comp rx=0 tx=0).stop 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- 172.21.15.67:0/1541768502 shutdown_connections 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 --2- 172.21.15.67:0/1541768502 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c143ae0 0x7f107c143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 --2- 172.21.15.67:0/1541768502 >> v2:172.21.15.86:3300/0 conn(0x7f107c06eac0 0x7f107c0ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 --2- 172.21.15.67:0/1541768502 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c06e6e0 0x7f107c0ce9b0 unknown :-1 s=CLOSED pgs=2180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- 172.21.15.67:0/1541768502 >> 172.21.15.67:0/1541768502 conn(0x7f107c06ac90 msgr2=0x7f107c06b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- 172.21.15.67:0/1541768502 shutdown_connections 2024-04-17T10:39:33.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- 172.21.15.67:0/1541768502 wait complete. 2024-04-17T10:39:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 Processor -- start 2024-04-17T10:39:33.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- start start 2024-04-17T10:39:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f107c06e6e0 0x7f107c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c06eac0 0x7f107c13d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c143ae0 0x7f107c13f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:33.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c15ebf0 con 0x7f107c143ae0 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107c1444b0 con 0x7f107c06e6e0 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1083000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f107c15ed70 con 0x7f107c06eac0 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c143ae0 0x7f107c13f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c143ae0 0x7f107c13f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44878/0 (socket says 172.21.15.67:44878) 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 -- 172.21.15.67:0/173725363 learned_addr learned my addr 172.21.15.67:0/173725363 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f107be00640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f107c06e6e0 0x7f107c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1073400640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c06eac0 0x7f107c13d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 -- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c06eac0 msgr2=0x7f107c13d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c06eac0 0x7f107c13d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 -- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f107c06e6e0 msgr2=0x7f107c159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f107c06e6e0 0x7f107c159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.986+0000 7f1081000640 1 -- 172.21.15.67:0/173725363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c15b5a0 con 0x7f107c143ae0 2024-04-17T10:39:33.979 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1081000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c143ae0 0x7f107c13f700 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d30 tx=0x7f107400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1078c00640 1 -- 172.21.15.67:0/173725363 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074018020 con 0x7f107c143ae0 2024-04-17T10:39:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1078c00640 1 -- 172.21.15.67:0/173725363 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1074014480 con 0x7f107c143ae0 2024-04-17T10:39:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1078c00640 1 -- 172.21.15.67:0/173725363 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1074014870 con 0x7f107c143ae0 2024-04-17T10:39:33.980 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c15b880 con 0x7f107c143ae0 2024-04-17T10:39:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c15be40 con 0x7f107c143ae0 2024-04-17T10:39:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f107c072d50 con 0x7f107c143ae0 2024-04-17T10:39:33.981 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1078c00640 1 -- 172.21.15.67:0/173725363 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f107401b020 con 0x7f107c143ae0 2024-04-17T10:39:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1078c00640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1058066f90 0x7f1058069450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f1078c00640 1 -- 172.21.15.67:0/173725363 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f1074085b80 con 0x7f107c143ae0 2024-04-17T10:39:33.982 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.990+0000 7f107be00640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1058066f90 0x7f1058069450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:33.983 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.994+0000 7f107be00640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1058066f90 0x7f1058069450 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f106c0af000 tx=0x7f106c0a7000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:33.986 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:33.994+0000 7f1078c00640 1 -- 172.21.15.67:0/173725363 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f1074022070 con 0x7f107c143ae0 2024-04-17T10:39:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.190+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f107c0b2c60 con 0x7f107c143ae0 2024-04-17T10:39:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1078c00640 1 -- 172.21.15.67:0/173725363 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f1074010040 con 0x7f107c143ae0 2024-04-17T10:39:34.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:34.183 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:34.183 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1058066f90 msgr2=0x7f1058069450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1058066f90 0x7f1058069450 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f106c0af000 tx=0x7f106c0a7000 comp rx=0 tx=0).stop 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c143ae0 msgr2=0x7f107c13f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c143ae0 0x7f107c13f700 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f1074009d30 tx=0x7f107400c490 comp rx=0 tx=0).stop 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 shutdown_connections 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f1058066f90 0x7f1058069450 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f107c143ae0 0x7f107c13f700 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f107c06eac0 0x7f107c13d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 --2- 172.21.15.67:0/173725363 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f107c06e6e0 0x7f107c159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 >> 172.21.15.67:0/173725363 conn(0x7f107c06ac90 msgr2=0x7f107c0cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 shutdown_connections 2024-04-17T10:39:34.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:34.194+0000 7f1083000640 1 -- 172.21.15.67:0/173725363 wait complete. 2024-04-17T10:39:34.606 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:34.607 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:35.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ffe00640 1 Processor -- start 2024-04-17T10:39:35.755 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ffe00640 1 -- start start 2024-04-17T10:39:35.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ffe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10015da90 0x7fe10015de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ffe00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe10006c3b0 0x7fe10015f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ffe00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10015e750 con 0x7fe100066d70 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ffe00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe10015ea30 con 0x7fe10006c3b0 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ffe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe10015ed10 con 0x7fe10015da90 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe10006c3b0 0x7fe10015f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ff400640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe10006c3b0 0x7fe10015f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44886/0 (socket says 172.21.15.67:44886) 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ff400640 1 -- 172.21.15.67:0/3876778128 learned_addr learned my addr 172.21.15.67:0/3876778128 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:35.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0fea00640 1 --2- 172.21.15.67:0/3876778128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10015da90 0x7fe10015de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:35.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ff400640 1 -- 172.21.15.67:0/3876778128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10015da90 msgr2=0x7fe10015de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:35.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ff400640 1 --2- 172.21.15.67:0/3876778128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10015da90 0x7fe10015de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ff400640 1 -- 172.21.15.67:0/3876778128 >> v1:172.21.15.86:6789/0 conn(0x7fe100066d70 legacy=0x7fe10006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:35.758 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.766+0000 7fe0ff400640 1 -- 172.21.15.67:0/3876778128 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10015e3b0 con 0x7fe10006c3b0 2024-04-17T10:39:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ff400640 1 --2- 172.21.15.67:0/3876778128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10006c3b0 0x7fe10015f7f0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009e50 tx=0x7fe0f801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3387baa659bfdfc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0fd600640 1 -- 172.21.15.67:0/3876778128 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80a5770 con 0x7fe10006c3b0 2024-04-17T10:39:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0fd600640 1 -- 172.21.15.67:0/3876778128 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f80a5910 con 0x7fe10006c3b0 2024-04-17T10:39:35.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0fd600640 1 -- 172.21.15.67:0/3876778128 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80a5ce0 con 0x7fe10006c3b0 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 -- 172.21.15.67:0/3876778128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10006c3b0 msgr2=0x7fe10015f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/3876778128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10006c3b0 0x7fe10015f7f0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009e50 tx=0x7fe0f801c9a0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 -- 172.21.15.67:0/3876778128 shutdown_connections 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/3876778128 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe10006c3b0 0x7fe10015f7f0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/3876778128 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10015da90 0x7fe10015de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 -- 172.21.15.67:0/3876778128 >> 172.21.15.67:0/3876778128 conn(0x7fe100071ad0 msgr2=0x7fe100071ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 -- 172.21.15.67:0/3876778128 shutdown_connections 2024-04-17T10:39:35.761 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 -- 172.21.15.67:0/3876778128 wait complete. 2024-04-17T10:39:35.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.770+0000 7fe0ffe00640 1 Processor -- start 2024-04-17T10:39:35.762 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 -- start start 2024-04-17T10:39:35.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe100066d70 0x7fe10014a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:35.764 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10006c3b0 0x7fe100152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10015da90 0x7fe100153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001608d0 con 0x7fe100066d70 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe100160750 con 0x7fe10015da90 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe100160a50 con 0x7fe10006c3b0 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe100066d70 0x7fe10014a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe100066d70 0x7fe10014a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44892/0 (socket says 172.21.15.67:44892) 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 -- 172.21.15.67:0/995737661 learned_addr learned my addr 172.21.15.67:0/995737661 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ff400640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10015da90 0x7fe100153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fe000640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10006c3b0 0x7fe100152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 -- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10006c3b0 msgr2=0x7fe100152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10006c3b0 0x7fe100152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 -- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10015da90 msgr2=0x7fe100153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10015da90 0x7fe100153020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 -- 172.21.15.67:0/995737661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1000ceb70 con 0x7fe100066d70 2024-04-17T10:39:35.765 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ff400640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10015da90 0x7fe100153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0fea00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe100066d70 0x7fe10014a2f0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000d5f0 tx=0x7fe0f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0f6a00640 1 -- 172.21.15.67:0/995737661 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f00125f0 con 0x7fe100066d70 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0f6a00640 1 -- 172.21.15.67:0/995737661 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f0019070 con 0x7fe100066d70 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0f6a00640 1 -- 172.21.15.67:0/995737661 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f00129b0 con 0x7fe100066d70 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1000ce890 con 0x7fe100066d70 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10006dde0 con 0x7fe100066d70 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.774+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe100074dc0 con 0x7fe100066d70 2024-04-17T10:39:35.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.778+0000 7fe0f6a00640 1 -- 172.21.15.67:0/995737661 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fe0f0012b50 con 0x7fe100066d70 2024-04-17T10:39:35.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.778+0000 7fe0f6a00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0e0066f60 0x7fe0e0069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:35.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.778+0000 7fe0f6a00640 1 -- 172.21.15.67:0/995737661 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fe0f00856a0 con 0x7fe100066d70 2024-04-17T10:39:35.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.778+0000 7fe0fe000640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0e0066f60 0x7fe0e0069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:35.768 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.778+0000 7fe0fe000640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0e0066f60 0x7fe0e0069420 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe10015c2c0 tx=0x7fe0ec009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:35.771 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.782+0000 7fe0f6a00640 1 -- 172.21.15.67:0/995737661 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fe0f0056480 con 0x7fe100066d70 2024-04-17T10:39:35.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1000b5240 con 0x7fe100066d70 2024-04-17T10:39:35.971 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0f6a00640 1 -- 172.21.15.67:0/995737661 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7fe0f0055d80 con 0x7fe100066d70 2024-04-17T10:39:35.971 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:35.971 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:35.972 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:35.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0e0066f60 msgr2=0x7fe0e0069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:35.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0e0066f60 0x7fe0e0069420 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe10015c2c0 tx=0x7fe0ec009040 comp rx=0 tx=0).stop 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe100066d70 msgr2=0x7fe10014a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe100066d70 0x7fe10014a2f0 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000d5f0 tx=0x7fe0f000dab0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 shutdown_connections 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fe0e0066f60 0x7fe0e0069420 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fe10015da90 0x7fe100153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fe10006c3b0 0x7fe100152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 --2- 172.21.15.67:0/995737661 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe100066d70 0x7fe10014a2f0 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 >> 172.21.15.67:0/995737661 conn(0x7fe100071ad0 msgr2=0x7fe10015a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 shutdown_connections 2024-04-17T10:39:35.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:35.982+0000 7fe0ffe00640 1 -- 172.21.15.67:0/995737661 wait complete. 2024-04-17T10:39:36.394 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:36.394 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:37.395 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:37.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.558+0000 7f6f1e800640 1 Processor -- start 2024-04-17T10:39:37.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.558+0000 7f6f1e800640 1 -- start start 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f1e800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f6f1806e6e0 0x7f6f180ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f1e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f1806eac0 0x7f6f180ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f1e800640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f18143ae0 0x7f6f18143f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f1e800640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f180cf430 con 0x7f6f18143ae0 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f1e800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1815ae40 con 0x7f6f1806e6e0 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f1e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18071440 con 0x7f6f1806eac0 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f18143ae0 0x7f6f18143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:37.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6f18143ae0 0x7f6f18143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44908/0 (socket says 172.21.15.67:44908) 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 -- 172.21.15.67:0/1345235276 learned_addr learned my addr 172.21.15.67:0/1345235276 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f0fe00640 1 --2- 172.21.15.67:0/1345235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f1806eac0 0x7f6f180ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17400640 1 --2- 172.21.15.67:0/1345235276 >> v2:172.21.15.86:3300/0 conn(0x7f6f1806e6e0 0x7f6f180ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 -- 172.21.15.67:0/1345235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f1806eac0 msgr2=0x7f6f180ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 --2- 172.21.15.67:0/1345235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f1806eac0 0x7f6f180ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 -- 172.21.15.67:0/1345235276 >> v2:172.21.15.86:3300/0 conn(0x7f6f1806e6e0 msgr2=0x7f6f180ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 --2- 172.21.15.67:0/1345235276 >> v2:172.21.15.86:3300/0 conn(0x7f6f1806e6e0 0x7f6f180ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.552 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.562+0000 7f6f17e00640 1 -- 172.21.15.67:0/1345235276 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f18071720 con 0x7f6f18143ae0 2024-04-17T10:39:37.554 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f17e00640 1 --2- 172.21.15.67:0/1345235276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f18143ae0 0x7f6f18143f70 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f6f180cf620 tx=0x7f6f1001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fbda62fb8de02ae server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f16a00640 1 -- 172.21.15.67:0/1345235276 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f100a57b0 con 0x7f6f18143ae0 2024-04-17T10:39:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f16a00640 1 -- 172.21.15.67:0/1345235276 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f100a5950 con 0x7f6f18143ae0 2024-04-17T10:39:37.555 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f16a00640 1 -- 172.21.15.67:0/1345235276 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f100a5d20 con 0x7f6f18143ae0 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- 172.21.15.67:0/1345235276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f18143ae0 msgr2=0x7f6f18143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 --2- 172.21.15.67:0/1345235276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f18143ae0 0x7f6f18143f70 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f6f180cf620 tx=0x7f6f1001c930 comp rx=0 tx=0).stop 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- 172.21.15.67:0/1345235276 shutdown_connections 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 --2- 172.21.15.67:0/1345235276 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f18143ae0 0x7f6f18143f70 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 --2- 172.21.15.67:0/1345235276 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f1806eac0 0x7f6f180ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 --2- 172.21.15.67:0/1345235276 >> v2:172.21.15.86:3300/0 conn(0x7f6f1806e6e0 0x7f6f180ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- 172.21.15.67:0/1345235276 >> 172.21.15.67:0/1345235276 conn(0x7f6f1806ac90 msgr2=0x7f6f1806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- 172.21.15.67:0/1345235276 shutdown_connections 2024-04-17T10:39:37.556 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- 172.21.15.67:0/1345235276 wait complete. 2024-04-17T10:39:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 Processor -- start 2024-04-17T10:39:37.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- start start 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f1806e6e0 0x7f6f1813b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1806eac0 0x7f6f1813ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f18143ae0 0x7f6f18142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1815ebf0 con 0x7f6f1806eac0 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f181444b0 con 0x7f6f1806e6e0 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f1e800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1815ed70 con 0x7f6f18143ae0 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1806eac0 0x7f6f1813ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f0fe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1806eac0 0x7f6f1813ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44920/0 (socket says 172.21.15.67:44920) 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f0fe00640 1 -- 172.21.15.67:0/3519497649 learned_addr learned my addr 172.21.15.67:0/3519497649 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f17e00640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f18143ae0 0x7f6f18142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.566+0000 7f6f17400640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f1806e6e0 0x7f6f1813b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:37.559 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0fe00640 1 -- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f18143ae0 msgr2=0x7f6f18142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0fe00640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f18143ae0 0x7f6f18142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0fe00640 1 -- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f1806e6e0 msgr2=0x7f6f1813b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0fe00640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f1806e6e0 0x7f6f1813b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0fe00640 1 -- 172.21.15.67:0/3519497649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f18139d70 con 0x7f6f1806eac0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0fe00640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1806eac0 0x7f6f1813ba40 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000e930 tx=0x7f6f0000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0f400640 1 -- 172.21.15.67:0/3519497649 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00017070 con 0x7f6f1806eac0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0f400640 1 -- 172.21.15.67:0/3519497649 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0000ca00 con 0x7f6f1806eac0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1813a050 con 0x7f6f1806eac0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0f400640 1 -- 172.21.15.67:0/3519497649 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0000cdb0 con 0x7f6f1806eac0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1813a610 con 0x7f6f1806eac0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f18072d50 con 0x7f6f1806eac0 2024-04-17T10:39:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0f400640 1 -- 172.21.15.67:0/3519497649 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f6f0001a070 con 0x7f6f1806eac0 2024-04-17T10:39:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0f400640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ef4066f60 0x7f6ef4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.570+0000 7f6f0f400640 1 -- 172.21.15.67:0/3519497649 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f6f000855d0 con 0x7f6f1806eac0 2024-04-17T10:39:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.574+0000 7f6f17400640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ef4066f60 0x7f6ef4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:37.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.574+0000 7f6f17400640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ef4066f60 0x7f6ef4069420 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6f18163f60 tx=0x7f6f08009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.574+0000 7f6f0f400640 1 -- 172.21.15.67:0/3519497649 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f6f00055d90 con 0x7f6f1806eac0 2024-04-17T10:39:37.766 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.774+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f1815da10 con 0x7f6f1806eac0 2024-04-17T10:39:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.774+0000 7f6f0f400640 1 -- 172.21.15.67:0/3519497649 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f6f00059e30 con 0x7f6f1806eac0 2024-04-17T10:39:37.767 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:37.767 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:37.767 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ef4066f60 msgr2=0x7f6ef4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ef4066f60 0x7f6ef4069420 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6f18163f60 tx=0x7f6f08009190 comp rx=0 tx=0).stop 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1806eac0 msgr2=0x7f6f1813ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1806eac0 0x7f6f1813ba40 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f6f0000e930 tx=0x7f6f0000edf0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 shutdown_connections 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f6ef4066f60 0x7f6ef4069420 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f6f18143ae0 0x7f6f18142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6f1806eac0 0x7f6f1813ba40 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 --2- 172.21.15.67:0/3519497649 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f6f1806e6e0 0x7f6f1813b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 >> 172.21.15.67:0/3519497649 conn(0x7f6f1806ac90 msgr2=0x7f6f180cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 shutdown_connections 2024-04-17T10:39:37.769 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:37.778+0000 7f6f1e800640 1 -- 172.21.15.67:0/3519497649 wait complete. 2024-04-17T10:39:38.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:38.190 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:39.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8556600640 1 Processor -- start 2024-04-17T10:39:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8556600640 1 -- start start 2024-04-17T10:39:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8556600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f85501466e0 0x7f855006eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8556600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550071a20 0x7f85500adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8556600640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8550161740 con 0x7f8550146da0 2024-04-17T10:39:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8556600640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8550161a20 con 0x7f85501466e0 2024-04-17T10:39:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8556600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8550161d00 con 0x7f8550071a20 2024-04-17T10:39:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8547e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f8550146da0 0x7f8550071310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:53836/0 (socket says 172.21.15.67:53836) 2024-04-17T10:39:39.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f8547e00640 1 -- 172.21.15.67:0/2865949964 learned_addr learned my addr 172.21.15.67:0/2865949964 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f854f400640 1 --2- 172.21.15.67:0/2865949964 >> v2:172.21.15.67:3300/0 conn(0x7f85501466e0 0x7f855006eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f854fe00640 1 --2- 172.21.15.67:0/2865949964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550071a20 0x7f85500adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f854f400640 1 -- 172.21.15.67:0/2865949964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550071a20 msgr2=0x7f85500adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f854f400640 1 --2- 172.21.15.67:0/2865949964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550071a20 0x7f85500adde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f854f400640 1 -- 172.21.15.67:0/2865949964 >> v1:172.21.15.67:6789/0 conn(0x7f8550146da0 legacy=0x7f8550071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f854f400640 1 -- 172.21.15.67:0/2865949964 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550161fe0 con 0x7f85501466e0 2024-04-17T10:39:39.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.354+0000 7f854ea00640 1 -- 172.21.15.67:0/2865949964 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505695737 0 0) 0x7f8550161740 con 0x7f8550146da0 2024-04-17T10:39:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f854f400640 1 --2- 172.21.15.67:0/2865949964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855006eb70 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f85400097c0 tx=0x7f854001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d68cfe102c7b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f854ea00640 1 -- 172.21.15.67:0/2865949964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400b0070 con 0x7f85501466e0 2024-04-17T10:39:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f854ea00640 1 -- 172.21.15.67:0/2865949964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85400ab600 con 0x7f85501466e0 2024-04-17T10:39:39.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f854ea00640 1 -- 172.21.15.67:0/2865949964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85400ab9d0 con 0x7f85501466e0 2024-04-17T10:39:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 -- 172.21.15.67:0/2865949964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 msgr2=0x7f855006eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 --2- 172.21.15.67:0/2865949964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855006eb70 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f85400097c0 tx=0x7f854001c4a0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 -- 172.21.15.67:0/2865949964 shutdown_connections 2024-04-17T10:39:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 --2- 172.21.15.67:0/2865949964 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550071a20 0x7f85500adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 --2- 172.21.15.67:0/2865949964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855006eb70 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 -- 172.21.15.67:0/2865949964 >> 172.21.15.67:0/2865949964 conn(0x7f855006ac90 msgr2=0x7f855006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 -- 172.21.15.67:0/2865949964 shutdown_connections 2024-04-17T10:39:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.358+0000 7f8556600640 1 -- 172.21.15.67:0/2865949964 wait complete. 2024-04-17T10:39:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 Processor -- start 2024-04-17T10:39:39.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 -- start start 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550071a20 0x7f85500cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855013e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550146da0 0x7f855013ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500aef40 con 0x7f85501466e0 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85500aedc0 con 0x7f8550071a20 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85500af0c0 con 0x7f8550146da0 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855013e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855013e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44936/0 (socket says 172.21.15.67:44936) 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 -- 172.21.15.67:0/1307783294 learned_addr learned my addr 172.21.15.67:0/1307783294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:39.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 -- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550146da0 msgr2=0x7f855013ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550146da0 0x7f855013ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 -- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550071a20 msgr2=0x7f85500cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550071a20 0x7f85500cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 -- 172.21.15.67:0/1307783294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500ccac0 con 0x7f85501466e0 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547e00640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855013e1a0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f853800c4b0 tx=0x7f853800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547400640 1 -- 172.21.15.67:0/1307783294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85380074e0 con 0x7f85501466e0 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547400640 1 -- 172.21.15.67:0/1307783294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85380047f0 con 0x7f85501466e0 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85500ccda0 con 0x7f85501466e0 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8547400640 1 -- 172.21.15.67:0/1307783294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538004be0 con 0x7f85501466e0 2024-04-17T10:39:39.355 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.362+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85500cc720 con 0x7f85501466e0 2024-04-17T10:39:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.366+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8550072d50 con 0x7f85501466e0 2024-04-17T10:39:39.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.366+0000 7f8547400640 1 -- 172.21.15.67:0/1307783294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f8538011070 con 0x7f85501466e0 2024-04-17T10:39:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.366+0000 7f8547400640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f852c066f60 0x7f852c069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.366+0000 7f8547400640 1 -- 172.21.15.67:0/1307783294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8538084ec0 con 0x7f85501466e0 2024-04-17T10:39:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.366+0000 7f854f400640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f852c066f60 0x7f852c069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:39.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.366+0000 7f854f400640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f852c066f60 0x7f852c069420 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f85400ad000 tx=0x7f8540004740 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:39.361 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.370+0000 7f8547400640 1 -- 172.21.15.67:0/1307783294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f853800e280 con 0x7f85501466e0 2024-04-17T10:39:39.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.566+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8550074e30 con 0x7f85501466e0 2024-04-17T10:39:39.557 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.566+0000 7f8547400640 1 -- 172.21.15.67:0/1307783294 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f8538055c20 con 0x7f85501466e0 2024-04-17T10:39:39.558 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:39.558 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:39.558 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f852c066f60 msgr2=0x7f852c069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f852c066f60 0x7f852c069420 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f85400ad000 tx=0x7f8540004740 comp rx=0 tx=0).stop 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 msgr2=0x7f855013e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855013e1a0 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f853800c4b0 tx=0x7f853800c970 comp rx=0 tx=0).stop 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 shutdown_connections 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f852c066f60 0x7f852c069420 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f8550146da0 0x7f855013ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f85501466e0 0x7f855013e1a0 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 --2- 172.21.15.67:0/1307783294 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f8550071a20 0x7f85500cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:39.560 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 >> 172.21.15.67:0/1307783294 conn(0x7f855006ac90 msgr2=0x7f85500b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 shutdown_connections 2024-04-17T10:39:39.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:39.570+0000 7f8556600640 1 -- 172.21.15.67:0/1307783294 wait complete. 2024-04-17T10:39:39.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:39.981 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:40.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-17T10:39:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.142+0000 7f1516800640 1 Processor -- start 2024-04-17T10:39:41.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.142+0000 7f1516800640 1 -- start start 2024-04-17T10:39:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f1516800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15100cf0c0 0x7f15101441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f1516800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f151015ae40 con 0x7f151006e6e0 2024-04-17T10:39:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f1516800640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1510071440 con 0x7f151006eac0 2024-04-17T10:39:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f1516800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1510071720 con 0x7f15100cf0c0 2024-04-17T10:39:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150f400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f151006e6e0 0x7f15100ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32882/0 (socket says 172.21.15.67:32882) 2024-04-17T10:39:41.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150f400640 1 -- 172.21.15.67:0/1418343782 learned_addr learned my addr 172.21.15.67:0/1418343782 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:41.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150fe00640 1 --2- 172.21.15.67:0/1418343782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15100cf0c0 0x7f15101441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:41.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786300369 0 0) 0x7f151015ae40 con 0x7f151006e6e0 2024-04-17T10:39:41.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f4005740 con 0x7f151006e6e0 2024-04-17T10:39:41.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536078849 0 0) 0x7f1510071440 con 0x7f151006eac0 2024-04-17T10:39:41.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151015ae40 con 0x7f151006eac0 2024-04-17T10:39:41.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 466885275 0 0) 0x7f14f4005740 con 0x7f151006e6e0 2024-04-17T10:39:41.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150fe00640 1 -- 172.21.15.67:0/1418343782 >> v1:172.21.15.86:6789/0 conn(0x7f151006eac0 legacy=0x7f1510143ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:41.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150fe00640 1 -- 172.21.15.67:0/1418343782 >> v1:172.21.15.67:6789/0 conn(0x7f151006e6e0 legacy=0x7f15100ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:41.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.146+0000 7f150fe00640 1 -- 172.21.15.67:0/1418343782 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1510071a00 con 0x7f15100cf0c0 2024-04-17T10:39:41.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f150fe00640 1 --2- 172.21.15.67:0/1418343782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15100cf0c0 0x7f15101441f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1508009dc0 tx=0x7f150801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=852a58b029a20703 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:41.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15080b2070 con 0x7f15100cf0c0 2024-04-17T10:39:41.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15080a5530 con 0x7f15100cf0c0 2024-04-17T10:39:41.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f150ea00640 1 -- 172.21.15.67:0/1418343782 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1500001680 con 0x7f15100cf0c0 2024-04-17T10:39:41.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 -- 172.21.15.67:0/1418343782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15100cf0c0 msgr2=0x7f15101441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:41.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 --2- 172.21.15.67:0/1418343782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15100cf0c0 0x7f15101441f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1508009dc0 tx=0x7f150801c7f0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 -- 172.21.15.67:0/1418343782 shutdown_connections 2024-04-17T10:39:41.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 --2- 172.21.15.67:0/1418343782 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f15100cf0c0 0x7f15101441f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 -- 172.21.15.67:0/1418343782 >> 172.21.15.67:0/1418343782 conn(0x7f151006ac90 msgr2=0x7f151006b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:41.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 -- 172.21.15.67:0/1418343782 shutdown_connections 2024-04-17T10:39:41.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 -- 172.21.15.67:0/1418343782 wait complete. 2024-04-17T10:39:41.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 Processor -- start 2024-04-17T10:39:41.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.150+0000 7f1516800640 1 -- start start 2024-04-17T10:39:41.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151006e6e0 0x7f1510159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:41.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f151006eac0 0x7f15100b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:41.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15100cf0c0 0x7f15100b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151006dd00 con 0x7f151006eac0 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f151006db80 con 0x7f15100cf0c0 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f151006de80 con 0x7f151006e6e0 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f151006eac0 0x7f15100b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f151006eac0 0x7f15100b16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52366/0 (socket says 172.21.15.67:52366) 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 -- 172.21.15.67:0/2393619653 learned_addr learned my addr 172.21.15.67:0/2393619653 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f150f400640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151006e6e0 0x7f1510159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f150fe00640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15100cf0c0 0x7f15100b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 -- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151006e6e0 msgr2=0x7f1510159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151006e6e0 0x7f1510159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 -- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15100cf0c0 msgr2=0x7f15100b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15100cf0c0 0x7f15100b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 -- 172.21.15.67:0/2393619653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1510137cd0 con 0x7f151006eac0 2024-04-17T10:39:41.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f150f400640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151006e6e0 0x7f1510159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:41.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f150fe00640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15100cf0c0 0x7f15100b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:41.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507e00640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f151006eac0 0x7f15100b16d0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f14f800c490 tx=0x7f14f800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:41.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507400640 1 -- 172.21.15.67:0/2393619653 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f80052d0 con 0x7f151006eac0 2024-04-17T10:39:41.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507400640 1 -- 172.21.15.67:0/2393619653 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8005470 con 0x7f151006eac0 2024-04-17T10:39:41.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507400640 1 -- 172.21.15.67:0/2393619653 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8012870 con 0x7f151006eac0 2024-04-17T10:39:41.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1510137fb0 con 0x7f151006eac0 2024-04-17T10:39:41.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1510138570 con 0x7f151006eac0 2024-04-17T10:39:41.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1510072d50 con 0x7f151006eac0 2024-04-17T10:39:41.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.154+0000 7f1507400640 1 -- 172.21.15.67:0/2393619653 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f14f8012a10 con 0x7f151006eac0 2024-04-17T10:39:41.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.158+0000 7f1507400640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14ec066f60 0x7f14ec069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:41.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.158+0000 7f1507400640 1 -- 172.21.15.67:0/2393619653 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f14f8084fa0 con 0x7f151006eac0 2024-04-17T10:39:41.147 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.158+0000 7f150f400640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14ec066f60 0x7f14ec069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.158+0000 7f150f400640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14ec066f60 0x7f14ec069420 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f15000011f0 tx=0x7f1500009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:41.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.162+0000 7f1507400640 1 -- 172.21.15.67:0/2393619653 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f14f8055cf0 con 0x7f151006eac0 2024-04-17T10:39:41.343 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1510074e30 con 0x7f151006eac0 2024-04-17T10:39:41.344 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1507400640 1 -- 172.21.15.67:0/2393619653 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4937 (secure 0 0 0) 0x7f14f80555f0 con 0x7f151006eac0 2024-04-17T10:39:41.344 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:41.344 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":106,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7976,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6837/1777746925","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6836","nonce":1777746925},{"type":"v1","addr":"172.21.15.86:6837","nonce":1777746925}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":7979,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/738606093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":738606093},{"type":"v1","addr":"172.21.15.67:6835","nonce":738606093}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98},{"gid":9055,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.86:6835/3497293124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6834","nonce":3497293124},{"type":"v1","addr":"172.21.15.86:6835","nonce":3497293124}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":97},{"gid":9939,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3606102270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3606102270},{"type":"v1","addr":"172.21.15.67:6834","nonce":3606102270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":106,"flags":51,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-17T10:35:36.470263+0000","modified":"2024-04-17T10:36:05.157126+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":142,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-04-17T10:39:41.344 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 106 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14ec066f60 msgr2=0x7f14ec069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14ec066f60 0x7f14ec069420 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f15000011f0 tx=0x7f1500009040 comp rx=0 tx=0).stop 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f151006eac0 msgr2=0x7f15100b16d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f151006eac0 0x7f15100b16d0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f14f800c490 tx=0x7f14f800c950 comp rx=0 tx=0).stop 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 shutdown_connections 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f14ec066f60 0x7f14ec069420 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f15100cf0c0 0x7f15100b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f151006eac0 0x7f15100b16d0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 --2- 172.21.15.67:0/2393619653 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f151006e6e0 0x7f1510159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 >> 172.21.15.67:0/2393619653 conn(0x7f151006ac90 msgr2=0x7f151006b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:41.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 shutdown_connections 2024-04-17T10:39:41.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:41.354+0000 7f1516800640 1 -- 172.21.15.67:0/2393619653 wait complete. 2024-04-17T10:39:41.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 106, 'flags': 51, 'flags_state': {'joinable': False, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-17T10:35:36.470263+0000', 'modified': '2024-04-17T10:36:05.157126+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 142, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-04-17T10:39:41.768 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-04-17T10:39:42.770 DEBUG:tasks.cephfs.filesystem:status = { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 10, "epoch": 106, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 10, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-17T10:35:36.470263+0000", "damaged": [], "data_pools": [ 22 ], "enabled": true, "epoch": 106, "ever_allowed_features": 32, "explicitly_allowed_features": 32, "failed": [ 0 ], "flags": 51, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": true, "balance_automate": false, "joinable": false, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 142, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 21, "modified": "2024-04-17T10:36:05.157126+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": {} } } ], "standbys": [ { "addr": "172.21.15.86:6837/1777746925", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6836", "nonce": 1777746925, "type": "v2" }, { "addr": "172.21.15.86:6837", "nonce": 1777746925, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7976, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.67:6835/738606093", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6833", "nonce": 738606093, "type": "v2" }, { "addr": "172.21.15.67:6835", "nonce": 738606093, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 98, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7979, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.86:6835/3497293124", "addrs": { "addrvec": [ { "addr": "172.21.15.86:6834", "nonce": 3497293124, "type": "v2" }, { "addr": "172.21.15.86:6835", "nonce": 3497293124, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 97, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9055, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.67:6834/3606102270", "addrs": { "addrvec": [ { "addr": "172.21.15.67:6832", "nonce": 3606102270, "type": "v2" }, { "addr": "172.21.15.67:6834", "nonce": 3606102270, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 106, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9939, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-17T10:39:42.770 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-17T10:39:42.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:39:42.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-17T10:39:42.921 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-17T10:39:42.921 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:39:42.921 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:39:42.931 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-04-17T10:39:42.931 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-17T10:39:42.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-17T10:39:43.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.190+0000 7fa74c800640 1 Processor -- start 2024-04-17T10:39:43.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.190+0000 7fa74c800640 1 -- start start 2024-04-17T10:39:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa74c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa744071310 0x7fa744071770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa74c800640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fa7440cc300 0x7fa7440cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa74c800640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa74406e2f0 con 0x7fa744145040 2024-04-17T10:39:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa74c800640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7440a86c0 con 0x7fa7440cc300 2024-04-17T10:39:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa74c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa744071cb0 con 0x7fa744071310 2024-04-17T10:39:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa749e00640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fa744145040 0x7fa74406e9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32890/0 (socket says 172.21.15.67:32890) 2024-04-17T10:39:43.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa749e00640 1 -- 172.21.15.67:0/4013868672 learned_addr learned my addr 172.21.15.67:0/4013868672 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa749400640 1 --2- 172.21.15.67:0/4013868672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa744071310 0x7fa744071770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa74a800640 1 --2- 172.21.15.67:0/4013868672 >> v2:172.21.15.86:3300/0 conn(0x7fa7440cc300 0x7fa7440cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa748a00640 1 -- 172.21.15.67:0/4013868672 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365499235 0 0) 0x7fa74406e2f0 con 0x7fa744145040 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa748a00640 1 -- 172.21.15.67:0/4013868672 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa72c003530 con 0x7fa744145040 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa749400640 1 -- 172.21.15.67:0/4013868672 >> v2:172.21.15.86:3300/0 conn(0x7fa7440cc300 msgr2=0x7fa7440cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa749400640 1 --2- 172.21.15.67:0/4013868672 >> v2:172.21.15.86:3300/0 conn(0x7fa7440cc300 0x7fa7440cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa749400640 1 -- 172.21.15.67:0/4013868672 >> v1:172.21.15.67:6789/0 conn(0x7fa744145040 legacy=0x7fa74406e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.194+0000 7fa749400640 1 -- 172.21.15.67:0/4013868672 --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74406d6e0 con 0x7fa744071310 2024-04-17T10:39:43.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa749400640 1 --2- 172.21.15.67:0/4013868672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa744071310 0x7fa744071770 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa7340097c0 tx=0x7fa73401ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cec9ee749b67efc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa748a00640 1 -- 172.21.15.67:0/4013868672 <== mon.2 v2:172.21.15.86:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340b5070 con 0x7fa744071310 2024-04-17T10:39:43.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa748a00640 1 -- 172.21.15.67:0/4013868672 <== mon.2 v2:172.21.15.86:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7340a8990 con 0x7fa744071310 2024-04-17T10:39:43.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa748a00640 1 -- 172.21.15.67:0/4013868672 <== mon.2 v2:172.21.15.86:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7340a8d40 con 0x7fa744071310 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 -- 172.21.15.67:0/4013868672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa744071310 msgr2=0x7fa744071770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 --2- 172.21.15.67:0/4013868672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa744071310 0x7fa744071770 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa7340097c0 tx=0x7fa73401ec90 comp rx=0 tx=0).stop 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 -- 172.21.15.67:0/4013868672 shutdown_connections 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 --2- 172.21.15.67:0/4013868672 >> v2:172.21.15.86:3300/0 conn(0x7fa7440cc300 0x7fa7440cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 --2- 172.21.15.67:0/4013868672 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa744071310 0x7fa744071770 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 -- 172.21.15.67:0/4013868672 >> 172.21.15.67:0/4013868672 conn(0x7fa74406ac90 msgr2=0x7fa74406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 -- 172.21.15.67:0/4013868672 shutdown_connections 2024-04-17T10:39:43.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 -- 172.21.15.67:0/4013868672 wait complete. 2024-04-17T10:39:43.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 Processor -- start 2024-04-17T10:39:43.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.198+0000 7fa74c800640 1 -- start start 2024-04-17T10:39:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa744071310 0x7fa74415b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7440cc300 0x7fa74415d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa744145040 0x7fa74415dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74406dca0 con 0x7fa744145040 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa74415bf10 con 0x7fa744071310 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa74406df80 con 0x7fa7440cc300 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa744145040 0x7fa74415dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa744145040 0x7fa74415dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52370/0 (socket says 172.21.15.67:52370) 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 -- 172.21.15.67:0/4055080883 learned_addr learned my addr 172.21.15.67:0/4055080883 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa749e00640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa744071310 0x7fa74415b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa749400640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7440cc300 0x7fa74415d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 -- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7440cc300 msgr2=0x7fa74415d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7440cc300 0x7fa74415d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 -- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa744071310 msgr2=0x7fa74415b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa744071310 0x7fa74415b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 -- 172.21.15.67:0/4055080883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa744134910 con 0x7fa744145040 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa749e00640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa744071310 0x7fa74415b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74a800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa744145040 0x7fa74415dbc0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fa740009dd0 tx=0x7fa74000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa73e000640 1 -- 172.21.15.67:0/4055080883 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740018020 con 0x7fa744145040 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa73e000640 1 -- 172.21.15.67:0/4055080883 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa740014480 con 0x7fa744145040 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa744131cd0 con 0x7fa744145040 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa73e000640 1 -- 172.21.15.67:0/4055080883 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa740014870 con 0x7fa744145040 2024-04-17T10:39:43.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa744133be0 con 0x7fa744145040 2024-04-17T10:39:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa744072d50 con 0x7fa744145040 2024-04-17T10:39:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.202+0000 7fa73e000640 1 -- 172.21.15.67:0/4055080883 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fa74001b020 con 0x7fa744145040 2024-04-17T10:39:43.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.206+0000 7fa73e000640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.206+0000 7fa73e000640 1 -- 172.21.15.67:0/4055080883 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fa740085cc0 con 0x7fa744145040 2024-04-17T10:39:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.206+0000 7fa749e00640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.206+0000 7fa749e00640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa738005890 tx=0x7fa738009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.210+0000 7fa73e000640 1 -- 172.21.15.67:0/4055080883 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fa74008c080 con 0x7fa744145040 2024-04-17T10:39:43.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.370+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa744074e30 con 0x7fa744145040 2024-04-17T10:39:43.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa73e000640 1 -- 172.21.15.67:0/4055080883 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+17131 (secure 0 0 0) 0x7fa740022070 con 0x7fa744145040 2024-04-17T10:39:43.363 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-04-17T10:39:43.363 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":144,"fsid":"8e8d1faf-cb51-4015-918b-f894d5d28e99","created":"2024-04-17T10:24:29.094599+0000","modified":"2024-04-17T10:36:37.981278+0000","last_up_change":"2024-04-17T10:24:41.511173+0000","last_in_change":"2024-04-17T10:24:31.631356+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-17T10:24:42.193580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2024-04-17T10:35:34.898907+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_data","create_time":"2024-04-17T10:35:35.464996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"141","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e7c33b47-11fd-4bd1-86a2-18d1179d12e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":143,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6801","nonce":3376065543}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6803","nonce":3376065543}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6807","nonce":3376065543}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":3376065543},{"type":"v1","addr":"172.21.15.67:6805","nonce":3376065543}]},"public_addr":"172.21.15.67:6801/3376065543","cluster_addr":"172.21.15.67:6803/3376065543","heartbeat_back_addr":"172.21.15.67:6807/3376065543","heartbeat_front_addr":"172.21.15.67:6805/3376065543","state":["exists","up"]},{"osd":1,"uuid":"526adc26-4eee-4b2c-86e3-6e55d021ca82","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6810","nonce":2092989796}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6814","nonce":2092989796}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6821","nonce":2092989796}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":2092989796},{"type":"v1","addr":"172.21.15.67:6817","nonce":2092989796}]},"public_addr":"172.21.15.67:6810/2092989796","cluster_addr":"172.21.15.67:6814/2092989796","heartbeat_back_addr":"172.21.15.67:6821/2092989796","heartbeat_front_addr":"172.21.15.67:6817/2092989796","state":["exists","up"]},{"osd":2,"uuid":"65a857d0-8abf-46ac-9056-b12b997ec561","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6811","nonce":2698936194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6813","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6815","nonce":2698936194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6823","nonce":2698936194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2698936194},{"type":"v1","addr":"172.21.15.67:6820","nonce":2698936194}]},"public_addr":"172.21.15.67:6811/2698936194","cluster_addr":"172.21.15.67:6815/2698936194","heartbeat_back_addr":"172.21.15.67:6823/2698936194","heartbeat_front_addr":"172.21.15.67:6820/2698936194","state":["exists","up"]},{"osd":3,"uuid":"7da79f1b-4b06-4704-9fbb-4100785acd2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6825","nonce":3611711736}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6827","nonce":3611711736}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6831","nonce":3611711736}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":3611711736},{"type":"v1","addr":"172.21.15.67:6829","nonce":3611711736}]},"public_addr":"172.21.15.67:6825/3611711736","cluster_addr":"172.21.15.67:6827/3611711736","heartbeat_back_addr":"172.21.15.67:6831/3611711736","heartbeat_front_addr":"172.21.15.67:6829/3611711736","state":["exists","up"]},{"osd":4,"uuid":"d77c3c57-bcdc-40aa-8383-1e284bf39505","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6801","nonce":710871084}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6803","nonce":710871084}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6807","nonce":710871084}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":710871084},{"type":"v1","addr":"172.21.15.86:6805","nonce":710871084}]},"public_addr":"172.21.15.86:6801/710871084","cluster_addr":"172.21.15.86:6803/710871084","heartbeat_back_addr":"172.21.15.86:6807/710871084","heartbeat_front_addr":"172.21.15.86:6805/710871084","state":["exists","up"]},{"osd":5,"uuid":"c2c739f2-870b-4bc2-8861-6f788264da32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6817","nonce":1567645357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6819","nonce":1567645357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6823","nonce":1567645357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1567645357},{"type":"v1","addr":"172.21.15.86:6821","nonce":1567645357}]},"public_addr":"172.21.15.86:6817/1567645357","cluster_addr":"172.21.15.86:6819/1567645357","heartbeat_back_addr":"172.21.15.86:6823/1567645357","heartbeat_front_addr":"172.21.15.86:6821/1567645357","state":["exists","up"]},{"osd":6,"uuid":"21849e19-c989-4e79-81fa-3a0c1b2d5656","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6825","nonce":4003922523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6827","nonce":4003922523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6831","nonce":4003922523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":4003922523},{"type":"v1","addr":"172.21.15.86:6829","nonce":4003922523}]},"public_addr":"172.21.15.86:6825/4003922523","cluster_addr":"172.21.15.86:6827/4003922523","heartbeat_back_addr":"172.21.15.86:6831/4003922523","heartbeat_front_addr":"172.21.15.86:6829/4003922523","state":["exists","up"]},{"osd":7,"uuid":"743537a0-cacc-4f52-8d54-524ded1ed61e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":139,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6809","nonce":2202166568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6811","nonce":2202166568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6815","nonce":2202166568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":2202166568},{"type":"v1","addr":"172.21.15.86:6813","nonce":2202166568}]},"public_addr":"172.21.15.86:6809/2202166568","cluster_addr":"172.21.15.86:6811/2202166568","heartbeat_back_addr":"172.21.15.86:6815/2202166568","heartbeat_front_addr":"172.21.15.86:6813/2202166568","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.207925+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.418097+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.333117+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.456035+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.629995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.307502+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.379624+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-17T10:24:37.659785+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"22.12","mappings":[{"from":3,"to":0}]},{"pgid":"22.3a","mappings":[{"from":3,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:6834/1952434688":"2024-04-18T10:36:04.768971+0000","172.21.15.67:6832/1952434688":"2024-04-18T10:36:04.768971+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-17T10:39:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 msgr2=0x7fa728069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fa738005890 tx=0x7fa738009040 comp rx=0 tx=0).stop 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa744145040 msgr2=0x7fa74415dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa744145040 0x7fa74415dbc0 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7fa740009dd0 tx=0x7fa74000c490 comp rx=0 tx=0).stop 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 shutdown_connections 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fa728066f60 0x7fa728069420 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa744145040 0x7fa74415dbc0 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fa7440cc300 0x7fa74415d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 --2- 172.21.15.67:0/4055080883 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa744071310 0x7fa74415b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 >> 172.21.15.67:0/4055080883 conn(0x7fa74406ac90 msgr2=0x7fa7440b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 shutdown_connections 2024-04-17T10:39:43.366 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.374+0000 7fa74c800640 1 -- 172.21.15.67:0/4055080883 wait complete. 2024-04-17T10:39:43.385 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-17T10:24:42.193580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-17T10:35:34.898907+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2024-04-17T10:35:35.464996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '141', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-17T10:39:43.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-17T10:39:43.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.538+0000 7f805a000640 1 Processor -- start 2024-04-17T10:39:43.530 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.538+0000 7f805a000640 1 -- start start 2024-04-17T10:39:43.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f805a000640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f805409a160 0x7f80540b50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f805a000640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f805415b9e0 0x7f80540b5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f805a000640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540ce1d0 con 0x7f805415b600 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f805a000640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80540b5cf0 con 0x7f805415b9e0 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f805a000640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80540ce490 con 0x7f805409a160 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8059600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f805415b9e0 0x7f80540b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8059600640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f805415b9e0 0x7f80540b5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52384/0 (socket says 172.21.15.67:52384) 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8059600640 1 -- 172.21.15.67:0/1209877007 learned_addr learned my addr 172.21.15.67:0/1209877007 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8053e00640 1 --2- 172.21.15.67:0/1209877007 >> v2:172.21.15.86:3300/0 conn(0x7f805409a160 0x7f80540b50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8059600640 1 -- 172.21.15.67:0/1209877007 >> v2:172.21.15.86:3300/0 conn(0x7f805409a160 msgr2=0x7f80540b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8059600640 1 --2- 172.21.15.67:0/1209877007 >> v2:172.21.15.86:3300/0 conn(0x7f805409a160 0x7f80540b50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8059600640 1 -- 172.21.15.67:0/1209877007 >> v1:172.21.15.86:6789/0 conn(0x7f805415b600 legacy=0x7f8054099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.532 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.542+0000 7f8059600640 1 -- 172.21.15.67:0/1209877007 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540ce770 con 0x7f805415b9e0 2024-04-17T10:39:43.534 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8059600640 1 --2- 172.21.15.67:0/1209877007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b9e0 0x7f80540b5620 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f804c009e80 tx=0x7f804c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c7b76b7e970d80e server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8053400640 1 -- 172.21.15.67:0/1209877007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0a5640 con 0x7f805415b9e0 2024-04-17T10:39:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8053400640 1 -- 172.21.15.67:0/1209877007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c0a57e0 con 0x7f805415b9e0 2024-04-17T10:39:43.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8053400640 1 -- 172.21.15.67:0/1209877007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c0a5bb0 con 0x7f805415b9e0 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- 172.21.15.67:0/1209877007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b9e0 msgr2=0x7f80540b5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 --2- 172.21.15.67:0/1209877007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b9e0 0x7f80540b5620 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f804c009e80 tx=0x7f804c01c870 comp rx=0 tx=0).stop 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- 172.21.15.67:0/1209877007 shutdown_connections 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 --2- 172.21.15.67:0/1209877007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b9e0 0x7f80540b5620 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 --2- 172.21.15.67:0/1209877007 >> v2:172.21.15.86:3300/0 conn(0x7f805409a160 0x7f80540b50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- 172.21.15.67:0/1209877007 >> 172.21.15.67:0/1209877007 conn(0x7f80540b31b0 msgr2=0x7f80540b35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- 172.21.15.67:0/1209877007 shutdown_connections 2024-04-17T10:39:43.536 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- 172.21.15.67:0/1209877007 wait complete. 2024-04-17T10:39:43.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 Processor -- start 2024-04-17T10:39:43.537 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- start start 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805409a160 0x7f8054149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b600 0x7f805415a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f805415b9e0 0x7f805413bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80540cfb10 con 0x7f805415b600 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80540cf990 con 0x7f805409a160 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f805a000640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80540cfc90 con 0x7f805415b9e0 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8053e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b600 0x7f805415a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8053e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b600 0x7f805415a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52386/0 (socket says 172.21.15.67:52386) 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8053e00640 1 -- 172.21.15.67:0/2436415229 learned_addr learned my addr 172.21.15.67:0/2436415229 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8058c00640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805409a160 0x7f8054149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.546+0000 7f8059600640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f805415b9e0 0x7f805413bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8053e00640 1 -- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f805415b9e0 msgr2=0x7f805413bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8053e00640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f805415b9e0 0x7f805413bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8053e00640 1 -- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805409a160 msgr2=0x7f8054149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8053e00640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805409a160 0x7f8054149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8053e00640 1 -- 172.21.15.67:0/2436415229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054159860 con 0x7f805415b600 2024-04-17T10:39:43.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8053e00640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b600 0x7f805415a790 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f804000d830 tx=0x7f804000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8050c00640 1 -- 172.21.15.67:0/2436415229 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804000b6c0 con 0x7f805415b600 2024-04-17T10:39:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8050c00640 1 -- 172.21.15.67:0/2436415229 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8040019070 con 0x7f805415b600 2024-04-17T10:39:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8050c00640 1 -- 172.21.15.67:0/2436415229 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804000ba80 con 0x7f805415b600 2024-04-17T10:39:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8054159b40 con 0x7f805415b600 2024-04-17T10:39:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8054159e20 con 0x7f805415b600 2024-04-17T10:39:43.541 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f805404f780 con 0x7f805415b600 2024-04-17T10:39:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8050c00640 1 -- 172.21.15.67:0/2436415229 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f804001e460 con 0x7f805415b600 2024-04-17T10:39:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8050c00640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8034066f60 0x7f8034069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8050c00640 1 -- 172.21.15.67:0/2436415229 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f8040085820 con 0x7f805415b600 2024-04-17T10:39:43.542 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.550+0000 7f8058c00640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8034066f60 0x7f8034069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.554+0000 7f8058c00640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8034066f60 0x7f8034069420 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8048007fd0 tx=0x7f8048009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.554+0000 7f8050c00640 1 -- 172.21.15.67:0/2436415229 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f8040055f20 con 0x7f805415b600 2024-04-17T10:39:43.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.718+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f80540712c0 con 0x7f805415b600 2024-04-17T10:39:43.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f8050c00640 1 -- 172.21.15.67:0/2436415229 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8040059fc0 con 0x7f805415b600 2024-04-17T10:39:43.711 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-04-17T10:39:43.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8034066f60 msgr2=0x7f8034069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8034066f60 0x7f8034069420 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8048007fd0 tx=0x7f8048009040 comp rx=0 tx=0).stop 2024-04-17T10:39:43.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b600 msgr2=0x7f805415a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b600 0x7f805415a790 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f804000d830 tx=0x7f804000dcf0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 shutdown_connections 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f8034066f60 0x7f8034069420 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f805415b9e0 0x7f805413bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f805415b600 0x7f805415a790 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 --2- 172.21.15.67:0/2436415229 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f805409a160 0x7f8054149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 >> 172.21.15.67:0/2436415229 conn(0x7f80540b31b0 msgr2=0x7f8054152b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 shutdown_connections 2024-04-17T10:39:43.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.722+0000 7f805a000640 1 -- 172.21.15.67:0/2436415229 wait complete. 2024-04-17T10:39:43.732 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-17T10:39:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98baa00640 1 Processor -- start 2024-04-17T10:39:43.877 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98baa00640 1 -- start start 2024-04-17T10:39:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98baa00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f98b406b3a0 0x7f98b40b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98baa00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98b40b3920 0x7f98b40d0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98baa00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b41616a0 con 0x7f98b41474c0 2024-04-17T10:39:43.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98baa00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b4161940 con 0x7f98b40b3920 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98baa00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b4161c20 con 0x7f98b406b3a0 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98b8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98b40b3920 0x7f98b40d0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98b8a00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f98b40b3920 0x7f98b40d0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52402/0 (socket says 172.21.15.67:52402) 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98b8a00640 1 -- 172.21.15.67:0/2806301628 learned_addr learned my addr 172.21.15.67:0/2806301628 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.886+0000 7f98abe00640 1 --2- 172.21.15.67:0/2806301628 >> v2:172.21.15.86:3300/0 conn(0x7f98b406b3a0 0x7f98b40b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b8a00640 1 -- 172.21.15.67:0/2806301628 >> v2:172.21.15.86:3300/0 conn(0x7f98b406b3a0 msgr2=0x7f98b40b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b8a00640 1 --2- 172.21.15.67:0/2806301628 >> v2:172.21.15.86:3300/0 conn(0x7f98b406b3a0 0x7f98b40b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b8a00640 1 -- 172.21.15.67:0/2806301628 >> v1:172.21.15.86:6789/0 conn(0x7f98b41474c0 legacy=0x7f98b406ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b8a00640 1 -- 172.21.15.67:0/2806301628 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b4161f00 con 0x7f98b40b3920 2024-04-17T10:39:43.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b8a00640 1 --2- 172.21.15.67:0/2806301628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40d0480 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f98b40b3500 tx=0x7f98ac01f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e30bd676b04801a server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b3400640 1 -- 172.21.15.67:0/2806301628 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0b4070 con 0x7f98b40b3920 2024-04-17T10:39:43.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b3400640 1 -- 172.21.15.67:0/2806301628 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac0af400 con 0x7f98b40b3920 2024-04-17T10:39:43.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.890+0000 7f98b3400640 1 -- 172.21.15.67:0/2806301628 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0af7d0 con 0x7f98b40b3920 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- 172.21.15.67:0/2806301628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 msgr2=0x7f98b40d0480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 --2- 172.21.15.67:0/2806301628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40d0480 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f98b40b3500 tx=0x7f98ac01f1e0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- 172.21.15.67:0/2806301628 shutdown_connections 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 --2- 172.21.15.67:0/2806301628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40d0480 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 --2- 172.21.15.67:0/2806301628 >> v2:172.21.15.86:3300/0 conn(0x7f98b406b3a0 0x7f98b40b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- 172.21.15.67:0/2806301628 >> 172.21.15.67:0/2806301628 conn(0x7f98b40b5ef0 msgr2=0x7f98b4099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- 172.21.15.67:0/2806301628 shutdown_connections 2024-04-17T10:39:43.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- 172.21.15.67:0/2806301628 wait complete. 2024-04-17T10:39:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 Processor -- start 2024-04-17T10:39:43.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- start start 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b406b3a0 0x7f98b415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b41474c0 0x7f98b415f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4162420 con 0x7f98b40b3920 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40b2250 con 0x7f98b41474c0 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98baa00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40d09c0 con 0x7f98b406b3a0 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52408/0 (socket says 172.21.15.67:52408) 2024-04-17T10:39:43.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 -- 172.21.15.67:0/1147981014 learned_addr learned my addr 172.21.15.67:0/1147981014 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98b3e00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b406b3a0 0x7f98b415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98b8a00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b41474c0 0x7f98b415f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 -- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b406b3a0 msgr2=0x7f98b415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b406b3a0 0x7f98b415eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 -- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b41474c0 msgr2=0x7f98b415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b41474c0 0x7f98b415f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 -- 172.21.15.67:0/1147981014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b40b05f0 con 0x7f98b40b3920 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98b8a00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b41474c0 0x7f98b415f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98b3e00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b406b3a0 0x7f98b415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.894+0000 7f98abe00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40b00b0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f989c00e930 tx=0x7f989c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98b0c00640 1 -- 172.21.15.67:0/1147981014 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c017070 con 0x7f98b40b3920 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98b0c00640 1 -- 172.21.15.67:0/1147981014 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c00ca00 con 0x7f98b40b3920 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98b0c00640 1 -- 172.21.15.67:0/1147981014 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c00cdb0 con 0x7f98b40b3920 2024-04-17T10:39:43.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b4160030 con 0x7f98b40b3920 2024-04-17T10:39:43.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b415e360 con 0x7f98b40b3920 2024-04-17T10:39:43.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b404ef30 con 0x7f98b40b3920 2024-04-17T10:39:43.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98b0c00640 1 -- 172.21.15.67:0/1147981014 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f989c01a070 con 0x7f98b40b3920 2024-04-17T10:39:43.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98b0c00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9890066f60 0x7f9890069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:43.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98b0c00640 1 -- 172.21.15.67:0/1147981014 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f989c085320 con 0x7f98b40b3920 2024-04-17T10:39:43.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.898+0000 7f98b3e00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9890066f60 0x7f9890069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:43.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.902+0000 7f98b3e00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9890066f60 0x7f9890069420 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f98a4001670 tx=0x7f98a4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:43.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:43.902+0000 7f98b0c00640 1 -- 172.21.15.67:0/1147981014 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f989c056120 con 0x7f98b40b3920 2024-04-17T10:39:44.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98b409a310 con 0x7f98b40b3920 2024-04-17T10:39:44.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98b0c00640 1 -- 172.21.15.67:0/1147981014 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v144) v1 ==== 104+0+11 (secure 0 0 0) 0x7f989c055a20 con 0x7f98b40b3920 2024-04-17T10:39:44.055 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 32 2024-04-17T10:39:44.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9890066f60 msgr2=0x7f9890069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9890066f60 0x7f9890069420 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f98a4001670 tx=0x7f98a4009040 comp rx=0 tx=0).stop 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 msgr2=0x7f98b40b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40b00b0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7f989c00e930 tx=0x7f989c00edf0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 shutdown_connections 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f9890066f60 0x7f9890069420 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f98b41474c0 0x7f98b415f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b40b3920 0x7f98b40b00b0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 --2- 172.21.15.67:0/1147981014 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f98b406b3a0 0x7f98b415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 >> 172.21.15.67:0/1147981014 conn(0x7f98b40b5ef0 msgr2=0x7f98b4143690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 shutdown_connections 2024-04-17T10:39:44.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.066+0000 7f98baa00640 1 -- 172.21.15.67:0/1147981014 wait complete. 2024-04-17T10:39:44.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-17T10:39:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae6be00640 1 Processor -- start 2024-04-17T10:39:44.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae6be00640 1 -- start start 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae6be00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7fae64071af0 0x7fae640ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae6be00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae6415be70 0x7fae6415c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae6be00640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6415cb30 con 0x7fae64159440 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae6be00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae640a8650 con 0x7fae6415be70 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae6be00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae6415ce10 con 0x7fae64071af0 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae69e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae6415be70 0x7fae6415c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae69e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fae6415be70 0x7fae6415c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52424/0 (socket says 172.21.15.67:52424) 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae69e00640 1 -- 172.21.15.67:0/1873952945 learned_addr learned my addr 172.21.15.67:0/1873952945 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae68a00640 1 --2- 172.21.15.67:0/1873952945 >> v2:172.21.15.86:3300/0 conn(0x7fae64071af0 0x7fae640ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae69e00640 1 -- 172.21.15.67:0/1873952945 >> v2:172.21.15.86:3300/0 conn(0x7fae64071af0 msgr2=0x7fae640ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:44.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae69e00640 1 --2- 172.21.15.67:0/1873952945 >> v2:172.21.15.86:3300/0 conn(0x7fae64071af0 0x7fae640ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae69e00640 1 -- 172.21.15.67:0/1873952945 >> v1:172.21.15.86:6789/0 conn(0x7fae64159440 legacy=0x7fae640713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.234+0000 7fae69e00640 1 -- 172.21.15.67:0/1873952945 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640cd610 con 0x7fae6415be70 2024-04-17T10:39:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae69e00640 1 --2- 172.21.15.67:0/1873952945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6415be70 0x7fae6415c2c0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fae640cf770 tx=0x7fae5c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4670801dee52369 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae63e00640 1 -- 172.21.15.67:0/1873952945 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c0a58b0 con 0x7fae6415be70 2024-04-17T10:39:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae63e00640 1 -- 172.21.15.67:0/1873952945 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c0a5a50 con 0x7fae6415be70 2024-04-17T10:39:44.229 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae63e00640 1 -- 172.21.15.67:0/1873952945 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c0a5e20 con 0x7fae6415be70 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 -- 172.21.15.67:0/1873952945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6415be70 msgr2=0x7fae6415c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 --2- 172.21.15.67:0/1873952945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6415be70 0x7fae6415c2c0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fae640cf770 tx=0x7fae5c01c9f0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 -- 172.21.15.67:0/1873952945 shutdown_connections 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 --2- 172.21.15.67:0/1873952945 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae6415be70 0x7fae6415c2c0 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 --2- 172.21.15.67:0/1873952945 >> v2:172.21.15.86:3300/0 conn(0x7fae64071af0 0x7fae640ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 -- 172.21.15.67:0/1873952945 >> 172.21.15.67:0/1873952945 conn(0x7fae640b1740 msgr2=0x7fae640b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 -- 172.21.15.67:0/1873952945 shutdown_connections 2024-04-17T10:39:44.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.238+0000 7fae6be00640 1 -- 172.21.15.67:0/1873952945 wait complete. 2024-04-17T10:39:44.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 Processor -- start 2024-04-17T10:39:44.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 -- start start 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae64071af0 0x7fae641410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae64159440 0x7fae640af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae6415be70 0x7fae640afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6415d840 con 0x7fae64071af0 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641598e0 con 0x7fae6415be70 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6415d9c0 con 0x7fae64159440 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae64071af0 0x7fae641410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae64071af0 0x7fae641410b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52430/0 (socket says 172.21.15.67:52430) 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 -- 172.21.15.67:0/3183675044 learned_addr learned my addr 172.21.15.67:0/3183675044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae68a00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae64159440 0x7fae640af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69e00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae6415be70 0x7fae640afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 -- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae64159440 msgr2=0x7fae640af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae64159440 0x7fae640af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 -- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae6415be70 msgr2=0x7fae640afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae6415be70 0x7fae640afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 -- 172.21.15.67:0/3183675044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640ad600 con 0x7fae64071af0 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae68a00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae64159440 0x7fae640af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69e00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae6415be70 0x7fae640afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae69400640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae64071af0 0x7fae641410b0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c500 tx=0x7fae5800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae61600640 1 -- 172.21.15.67:0/3183675044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58016020 con 0x7fae64071af0 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae61600640 1 -- 172.21.15.67:0/3183675044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae58005220 con 0x7fae64071af0 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae61600640 1 -- 172.21.15.67:0/3183675044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58007030 con 0x7fae64071af0 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae640ad8c0 con 0x7fae64071af0 2024-04-17T10:39:44.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae640ade80 con 0x7fae64071af0 2024-04-17T10:39:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.242+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae6404ef30 con 0x7fae64071af0 2024-04-17T10:39:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.246+0000 7fae61600640 1 -- 172.21.15.67:0/3183675044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7fae580071f0 con 0x7fae64071af0 2024-04-17T10:39:44.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.246+0000 7fae61600640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae44066f60 0x7fae44069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.246+0000 7fae61600640 1 -- 172.21.15.67:0/3183675044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7fae58084cb0 con 0x7fae64071af0 2024-04-17T10:39:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.246+0000 7fae68a00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae44066f60 0x7fae44069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.237 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.246+0000 7fae68a00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae44066f60 0x7fae44069420 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fae640ae880 tx=0x7fae50008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:44.241 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.250+0000 7fae61600640 1 -- 172.21.15.67:0/3183675044 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7fae58020020 con 0x7fae64071af0 2024-04-17T10:39:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.410+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae64152c30 con 0x7fae64071af0 2024-04-17T10:39:44.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.410+0000 7fae61600640 1 -- 172.21.15.67:0/3183675044 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v144) v1 ==== 100+0+11 (secure 0 0 0) 0x7fae58055a10 con 0x7fae64071af0 2024-04-17T10:39:44.403 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 64 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae44066f60 msgr2=0x7fae44069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae44066f60 0x7fae44069420 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fae640ae880 tx=0x7fae50008040 comp rx=0 tx=0).stop 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae64071af0 msgr2=0x7fae641410b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae64071af0 0x7fae641410b0 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fae5800c500 tx=0x7fae5800c9c0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 shutdown_connections 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7fae44066f60 0x7fae44069420 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fae6415be70 0x7fae640afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7fae64159440 0x7fae640af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 --2- 172.21.15.67:0/3183675044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae64071af0 0x7fae641410b0 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 >> 172.21.15.67:0/3183675044 conn(0x7fae640b1740 msgr2=0x7fae640713e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 shutdown_connections 2024-04-17T10:39:44.405 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.414+0000 7fae6be00640 1 -- 172.21.15.67:0/3183675044 wait complete. 2024-04-17T10:39:44.424 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-17T10:39:44.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.574+0000 7f92e7e00640 1 Processor -- start 2024-04-17T10:39:44.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.574+0000 7f92e7e00640 1 -- start start 2024-04-17T10:39:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.574+0000 7f92e7e00640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f92e806c7b0 0x7f92e806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.574+0000 7f92e7e00640 1 --2- >> v2:172.21.15.86:3300/0 conn(0x7f92e80d03e0 0x7f92e8099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e7e00640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92e809a7d0 con 0x7f92e80d0870 2024-04-17T10:39:44.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e7e00640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e80cf6d0 con 0x7f92e806c7b0 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e7e00640 1 -- --> v2:172.21.15.86:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e809a4f0 con 0x7f92e80d03e0 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e7400640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f92e80d0870 0x7f92e80cbc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:32898/0 (socket says 172.21.15.67:32898) 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e7400640 1 -- 172.21.15.67:0/1420882947 learned_addr learned my addr 172.21.15.67:0/1420882947 (peer_addr_for_me v1:172.21.15.67:0/0) 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6a00640 1 --2- 172.21.15.67:0/1420882947 >> v2:172.21.15.67:3300/0 conn(0x7f92e806c7b0 0x7f92e806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92dfe00640 1 --2- 172.21.15.67:0/1420882947 >> v2:172.21.15.86:3300/0 conn(0x7f92e80d03e0 0x7f92e8099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6000640 1 -- 172.21.15.67:0/1420882947 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220766423 0 0) 0x7f92e809a7d0 con 0x7f92e80d0870 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6000640 1 -- 172.21.15.67:0/1420882947 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92cc003530 con 0x7f92e80d0870 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6a00640 1 -- 172.21.15.67:0/1420882947 >> v2:172.21.15.86:3300/0 conn(0x7f92e80d03e0 msgr2=0x7f92e8099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6a00640 1 --2- 172.21.15.67:0/1420882947 >> v2:172.21.15.86:3300/0 conn(0x7f92e80d03e0 0x7f92e8099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6a00640 1 -- 172.21.15.67:0/1420882947 >> v1:172.21.15.67:6789/0 conn(0x7f92e80d0870 legacy=0x7f92e80cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6a00640 1 -- 172.21.15.67:0/1420882947 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e80cf900 con 0x7f92e806c7b0 2024-04-17T10:39:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6a00640 1 --2- 172.21.15.67:0/1420882947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e806cb90 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f92e80726f0 tx=0x7f92d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8904c8870e01641d server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6000640 1 -- 172.21.15.67:0/1420882947 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d80b0070 con 0x7f92e806c7b0 2024-04-17T10:39:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6000640 1 -- 172.21.15.67:0/1420882947 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d80ab600 con 0x7f92e806c7b0 2024-04-17T10:39:44.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.578+0000 7f92e6000640 1 -- 172.21.15.67:0/1420882947 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d80ab9d0 con 0x7f92e806c7b0 2024-04-17T10:39:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- 172.21.15.67:0/1420882947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 msgr2=0x7f92e806cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 --2- 172.21.15.67:0/1420882947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e806cb90 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f92e80726f0 tx=0x7f92d801c480 comp rx=0 tx=0).stop 2024-04-17T10:39:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- 172.21.15.67:0/1420882947 shutdown_connections 2024-04-17T10:39:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 --2- 172.21.15.67:0/1420882947 >> v2:172.21.15.86:3300/0 conn(0x7f92e80d03e0 0x7f92e8099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 --2- 172.21.15.67:0/1420882947 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e806cb90 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- 172.21.15.67:0/1420882947 >> 172.21.15.67:0/1420882947 conn(0x7f92e80d1620 msgr2=0x7f92e80d1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- 172.21.15.67:0/1420882947 shutdown_connections 2024-04-17T10:39:44.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- 172.21.15.67:0/1420882947 wait complete. 2024-04-17T10:39:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 Processor -- start 2024-04-17T10:39:44.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- start start 2024-04-17T10:39:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e8157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 --2- >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92e80d03e0 0x7f92e8142ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 --2- >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92e80d0870 0x7f92e815b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e816b210 con 0x7f92e806c7b0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- --> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e806d0d0 con 0x7f92e80d03e0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7e00640 1 -- --> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e816b390 con 0x7f92e80d0870 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e8157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e8157cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52450/0 (socket says 172.21.15.67:52450) 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 -- 172.21.15.67:0/3608043253 learned_addr learned my addr 172.21.15.67:0/3608043253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e7400640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92e80d0870 0x7f92e815b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92dfe00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92e80d03e0 0x7f92e8142ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 -- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92e80d0870 msgr2=0x7f92e815b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92e80d0870 0x7f92e815b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 -- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92e80d03e0 msgr2=0x7f92e8142ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92e80d03e0 0x7f92e8142ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.582+0000 7f92e6a00640 1 -- 172.21.15.67:0/3608043253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e815bff0 con 0x7f92e806c7b0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92e6a00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e8157cf0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f92d80ae040 tx=0x7f92d8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92df400640 1 -- 172.21.15.67:0/3608043253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d80b8020 con 0x7f92e806c7b0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92df400640 1 -- 172.21.15.67:0/3608043253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d80b4410 con 0x7f92e806c7b0 2024-04-17T10:39:44.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92df400640 1 -- 172.21.15.67:0/3608043253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d80b0040 con 0x7f92e806c7b0 2024-04-17T10:39:44.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e806f710 con 0x7f92e806c7b0 2024-04-17T10:39:44.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e806fcd0 con 0x7f92e806c7b0 2024-04-17T10:39:44.578 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e8074dc0 con 0x7f92e806c7b0 2024-04-17T10:39:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92df400640 1 -- 172.21.15.67:0/3608043253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 86727+0+0 (secure 0 0 0) 0x7f92d80bb020 con 0x7f92e806c7b0 2024-04-17T10:39:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92df400640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92c4066f60 0x7f92c4069420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-17T10:39:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92df400640 1 -- 172.21.15.67:0/3608043253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6245+0+0 (secure 0 0 0) 0x7f92d811ed40 con 0x7f92e806c7b0 2024-04-17T10:39:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.586+0000 7f92dfe00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92c4066f60 0x7f92c4069420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-17T10:39:44.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.590+0000 7f92dfe00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92c4066f60 0x7f92c4069420 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f92e815d130 tx=0x7f92d0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-17T10:39:44.581 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.590+0000 7f92df400640 1 -- 172.21.15.67:0/3608043253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175264 (secure 0 0 0) 0x7f92d80c2030 con 0x7f92e806c7b0 2024-04-17T10:39:44.756 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f92e80d0fd0 con 0x7f92e806c7b0 2024-04-17T10:39:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92df400640 1 -- 172.21.15.67:0/3608043253 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v144) v1 ==== 80+0+122 (secure 0 0 0) 0x7f92d8016070 con 0x7f92e806c7b0 2024-04-17T10:39:44.757 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6834/1952434688 2024-04-18T10:36:04.768971+0000 2024-04-17T10:39:44.757 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6832/1952434688 2024-04-18T10:36:04.768971+0000 2024-04-17T10:39:44.757 INFO:teuthology.orchestra.run.smithi067.stderr:listed 2 entries 2024-04-17T10:39:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92c4066f60 msgr2=0x7f92c4069420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92c4066f60 0x7f92c4069420 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f92e815d130 tx=0x7f92d0008040 comp rx=0 tx=0).stop 2024-04-17T10:39:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 msgr2=0x7f92e8157cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-17T10:39:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e8157cf0 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7f92d80ae040 tx=0x7f92d8019220 comp rx=0 tx=0).stop 2024-04-17T10:39:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 shutdown_connections 2024-04-17T10:39:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:6832/4206025909,v1:172.21.15.86:6833/4206025909] conn(0x7f92c4066f60 0x7f92c4069420 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.759 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3301/0,v1:172.21.15.86:6790/0] conn(0x7f92e80d0870 0x7f92e815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f92e80d03e0 0x7f92e8142ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 --2- 172.21.15.67:0/3608043253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f92e806c7b0 0x7f92e8157cf0 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-17T10:39:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.766+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 >> 172.21.15.67:0/3608043253 conn(0x7f92e80d1620 msgr2=0x7f92e80ca1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-17T10:39:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.770+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 shutdown_connections 2024-04-17T10:39:44.760 INFO:teuthology.orchestra.run.smithi067.stderr:2024-04-17T10:39:44.770+0000 7f92e7e00640 1 -- 172.21.15.67:0/3608043253 wait complete. 2024-04-17T10:39:44.778 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-17T10:39:44.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-17T10:39:44.778 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-17T10:39:58.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:39:58.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:39:59.987 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:39:59.997+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T10:40:28.950 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:40:28.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:40:59.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:40:59.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:41:29.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:41:29.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:41:59.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:41:59.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:42:29.201 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:42:29.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:42:59.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:42:59.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:43:29.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:43:29.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:43:59.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:43:59.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:44:29.490 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:44:29.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:44:44.783 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr raise socket.timeout() socket.timeout 2024-04-17T10:44:44.852 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-04-17T10:44:59.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:44:59.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:45:29.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:45:29.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:45:59.702 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:45:59.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:46:29.757 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:46:29.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:46:59.814 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:46:59.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:47:29.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:47:29.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:47:59.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:47:59.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:48:30.011 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:48:30.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:49:00.074 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:49:00.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:49:30.129 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:49:30.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:49:59.994 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:49:59.996+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T10:50:00.187 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:50:00.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:50:30.235 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:50:30.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:51:00.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:51:00.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:51:30.356 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:51:30.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:52:00.413 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:52:00.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:52:30.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:52:30.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:53:00.525 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:53:00.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:53:30.598 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:53:30.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:54:00.664 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:54:00.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:54:30.721 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:54:30.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:55:00.777 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:55:00.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:55:30.842 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:55:30.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:56:00.905 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:56:00.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:56:30.972 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:56:30.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:57:01.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:57:01.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:57:31.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:57:31.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:58:01.207 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:58:01.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:58:31.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:58:31.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:59:01.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:59:01.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:59:31.458 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:59:31.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T10:59:59.995 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T10:59:59.999+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T11:00:01.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:00:01.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:00:31.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:00:31.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:01:01.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:01:01.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:01:31.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:01:31.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:02:01.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:02:01.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:02:31.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:02:31.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:03:01.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:03:01.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:03:32.036 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:03:32.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:04:02.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:04:02.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:04:32.173 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:04:32.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:05:02.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:05:02.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:05:32.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:05:32.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:06:02.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:06:02.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:06:32.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:06:32.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:07:02.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:07:02.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:07:32.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:07:32.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:08:02.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:08:02.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:08:32.837 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:08:32.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:09:02.901 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:09:02.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:09:32.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:09:32.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:09:59.995 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T11:09:59.997+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T11:10:03.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:10:03.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:10:33.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:10:33.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:11:03.184 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:11:03.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:11:33.332 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:11:33.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:12:03.446 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:12:03.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:12:33.561 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:12:33.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:13:03.708 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:13:03.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:13:33.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:13:33.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:14:03.847 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:14:03.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:14:33.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:14:33.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:15:04.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:15:04.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:15:34.208 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:15:34.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:16:04.370 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:16:04.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:16:34.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:16:34.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:17:04.574 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:17:04.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:17:34.706 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:17:34.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:18:04.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:18:04.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:18:34.909 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:18:34.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:19:05.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:19:05.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:19:35.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:19:35.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:19:59.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T11:19:59.997+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T11:20:05.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:20:05.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:20:35.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:20:35.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:21:05.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:21:05.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:21:35.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:21:35.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:22:05.571 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:22:05.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:22:35.694 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:22:35.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:23:05.851 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:23:05.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:23:35.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:23:35.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:24:05.988 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:24:05.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:24:36.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:24:36.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:25:06.242 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:25:06.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:25:36.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:25:36.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:26:06.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:26:06.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:26:36.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:26:36.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:27:06.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:27:06.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:27:36.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:27:36.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:28:06.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:28:06.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:28:37.010 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:28:37.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:29:07.142 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:29:07.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:29:37.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:29:37.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:29:59.999 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T11:29:59.998+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T11:30:07.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:30:07.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:30:37.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:30:37.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:31:07.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:31:07.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:31:37.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:31:37.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:32:07.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:32:07.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:32:37.926 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:32:37.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:33:08.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:33:08.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:33:38.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:33:38.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:34:08.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:34:08.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:34:38.316 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:34:38.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:35:08.455 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:35:08.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:35:38.644 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:35:38.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:36:08.792 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:36:08.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:36:38.945 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:36:38.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:37:09.071 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:37:09.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:37:39.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:37:39.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:38:09.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:38:09.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:38:39.404 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:38:39.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:39:09.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:39:09.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:39:39.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:39:39.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:39:59.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T11:39:59.997+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T11:40:09.716 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:40:09.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:40:39.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:40:39.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:41:09.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:41:10.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:41:40.119 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:41:40.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:42:10.276 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:42:10.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:42:40.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:42:40.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:43:10.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:43:10.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:43:40.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:43:40.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:44:10.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:44:10.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:44:40.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:44:40.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:45:11.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:45:11.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:45:41.176 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:45:41.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:46:11.290 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:46:11.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:46:41.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:46:41.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:47:11.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:47:11.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:47:41.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:47:41.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:48:11.974 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:48:11.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:48:42.322 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:48:42.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:49:12.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:49:12.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:49:42.600 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:49:42.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:49:59.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T11:49:59.996+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T11:50:12.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:50:12.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:50:42.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:50:42.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:51:12.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:51:12.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:51:43.117 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:51:43.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:52:13.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:52:13.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:52:43.329 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:52:43.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:53:13.393 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:53:13.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:53:43.508 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:53:43.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:54:13.578 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:54:13.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:54:43.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:54:43.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:55:13.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:55:13.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:55:43.875 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:55:43.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:56:13.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:56:13.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:56:44.114 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:56:44.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:57:14.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:57:14.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:57:44.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:57:44.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:58:14.423 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:58:14.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:58:44.505 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:58:44.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:59:14.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:59:14.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:59:44.700 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:59:44.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T11:59:59.997 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T11:59:59.999+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T12:00:14.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:00:14.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:00:45.003 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:00:45.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:01:15.177 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:01:15.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:01:45.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:01:45.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:02:15.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:02:15.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:02:45.520 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:02:45.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:03:15.634 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:03:15.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:03:46.058 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:03:46.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:04:16.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:04:16.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:04:46.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:04:46.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:05:16.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:05:16.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:05:46.599 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:05:46.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:06:16.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:06:16.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:06:46.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:06:46.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:07:16.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:07:16.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:07:47.082 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:07:47.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:08:17.214 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:08:17.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:08:47.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:08:47.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:09:17.383 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:09:17.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:09:47.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:09:47.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:09:59.997 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T12:09:59.998+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T12:10:17.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:10:17.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:10:47.802 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:10:47.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:11:17.933 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:11:17.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:11:48.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:11:48.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:12:18.179 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:12:18.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:12:48.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:12:48.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:13:18.504 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:13:18.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:13:48.848 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:13:48.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:14:18.912 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:14:18.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:14:48.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:14:48.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:15:19.115 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:15:19.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:15:49.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:15:49.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:16:19.338 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:16:19.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:16:49.459 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:16:49.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:17:19.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:17:19.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:17:49.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:17:49.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:18:19.804 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:18:19.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:18:49.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:18:49.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:19:19.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:19:19.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:19:50.080 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:19:50.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:19:59.997 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T12:19:59.997+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T12:20:20.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:20:20.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:20:50.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:20:50.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:21:20.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:21:20.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:21:50.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:21:50.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:22:20.704 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:22:20.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:22:50.761 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:22:50.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:23:20.890 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:23:20.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:23:50.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:23:50.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:24:21.028 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:24:21.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:24:51.159 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:24:51.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:25:21.282 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:25:21.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:25:51.340 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:25:51.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:26:21.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:26:21.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:26:51.528 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:26:51.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:27:21.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:27:21.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:27:51.781 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:27:51.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:28:21.904 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:28:21.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:28:52.087 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:28:52.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:29:22.191 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:29:22.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:29:52.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:29:52.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:29:59.997 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T12:29:59.999+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T12:30:22.381 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:30:22.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:30:52.577 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:30:52.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:31:22.627 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:31:22.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:31:52.748 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:31:52.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:32:22.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:32:22.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:32:52.987 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:32:52.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:33:23.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:33:23.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:33:53.492 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:33:53.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:34:23.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:34:23.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:34:53.678 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:34:53.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:35:23.810 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:35:23.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:35:53.924 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:35:53.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:36:24.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:36:24.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:36:54.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:36:54.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:37:24.343 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:37:24.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:37:54.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:37:54.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:38:24.547 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:38:24.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:38:54.737 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:38:54.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:39:24.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:39:24.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:39:54.941 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:39:54.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:39:59.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T12:39:59.997+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T12:40:25.106 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:40:25.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:40:55.245 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:40:55.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:41:25.302 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:41:25.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:41:55.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:41:55.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:42:25.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:42:25.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:42:55.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:42:55.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:43:25.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:43:25.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:43:55.915 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:43:55.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:44:26.039 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:44:26.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:44:56.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:44:56.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:45:26.301 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:45:26.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:45:56.362 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:45:56.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:46:26.514 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:46:26.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:46:56.581 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:46:56.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:47:26.710 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:47:26.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:47:56.815 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:47:56.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:48:26.877 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:48:26.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:48:56.953 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:48:56.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:49:27.076 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:49:27.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:49:57.240 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:49:57.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:49:59.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T12:49:59.995+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T12:50:27.347 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:50:27.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:50:57.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:50:57.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:51:27.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:51:27.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:51:57.656 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:51:57.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:52:27.779 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:52:27.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:52:57.836 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:52:57.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:53:27.966 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:53:27.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:53:58.055 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:53:58.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:54:28.112 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:54:28.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:54:58.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:54:58.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:55:28.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:55:28.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:55:58.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:55:58.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:56:28.596 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:56:28.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:56:58.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:56:58.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:57:28.868 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:57:28.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:57:58.990 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:57:58.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:58:29.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:58:29.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:58:59.486 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:58:59.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:59:29.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:59:29.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:59:59.682 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:59:59.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T12:59:59.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T12:59:59.998+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T13:00:29.797 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:00:29.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:00:59.902 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:00:59.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:01:30.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:01:30.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:02:00.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:02:00.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:02:30.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:02:30.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:03:00.318 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:03:00.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:03:30.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:03:30.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:04:00.772 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:04:00.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:04:30.845 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:04:30.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:05:00.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:05:00.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:05:31.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:05:31.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:06:01.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:06:01.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:06:31.261 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:06:31.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:07:01.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:07:01.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:07:31.533 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:07:31.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:08:01.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:08:01.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:08:31.770 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:08:31.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:09:01.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:09:01.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:09:32.024 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:09:32.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:09:59.998 INFO:tasks.ceph.mon.a.smithi067.stderr:2024-04-17T13:09:59.996+0000 7ff642400640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline 2024-04-17T13:10:02.147 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:10:02.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:10:32.277 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:10:32.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:11:02.342 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:11:02.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:11:32.448 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:11:32.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:12:02.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:12:02.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:12:32.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:12:32.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:13:02.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:13:02.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:13:32.947 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:13:32.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:14:03.030 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:14:03.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:14:33.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:14:33.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:15:03.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:15:03.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:15:33.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:15:33.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:16:03.411 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:16:03.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:16:33.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:16:33.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:17:03.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:17:03.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:17:33.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:17:33.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:18:03.860 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:18:03.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:18:33.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:18:33.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:19:04.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:19:04.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-17T13:19:09.645 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2024-04-17T13:19:09.645 DEBUG:teuthology.task.console_log:Killing console logger for smithi067 2024-04-17T13:19:09.646 DEBUG:teuthology.task.console_log:Killing console logger for smithi086 2024-04-17T13:19:09.646 DEBUG:teuthology.exit:Finished running handlers